Jul 23 19:17:50.611343 ip-10-0-129-2 systemd[1]: kubelet.service: Failed to load environment files: No such file or directory Jul 23 19:17:50.611356 ip-10-0-129-2 systemd[1]: kubelet.service: Failed to run 'start-pre' task: No such file or directory Jul 23 19:17:50.611364 ip-10-0-129-2 systemd[1]: kubelet.service: Failed with result 'resources'. Jul 23 19:17:50.611861 ip-10-0-129-2 systemd[1]: Failed to start Kubernetes Kubelet. Jul 23 19:18:00.699096 ip-10-0-129-2 systemd[1]: kubelet.service: Failed to schedule restart job: Unit crio.service not found. Jul 23 19:18:00.699116 ip-10-0-129-2 systemd[1]: kubelet.service: Failed with result 'resources'. -- Boot b4366ba00f74472fa743286bfb1451fe -- Jul 23 19:19:34.022608 ip-10-0-129-2 systemd[1]: Starting Kubernetes Kubelet... Jul 23 19:19:34.459907 ip-10-0-129-2 kubenswrapper[2564]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Jul 23 19:19:34.459907 ip-10-0-129-2 kubenswrapper[2564]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version. Jul 23 19:19:34.459907 ip-10-0-129-2 kubenswrapper[2564]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Jul 23 19:19:34.460357 ip-10-0-129-2 kubenswrapper[2564]: Flag --pod-infra-container-image has been deprecated, will be removed in 1.35. Image garbage collector will get sandbox image information from CRI. Jul 23 19:19:34.460357 ip-10-0-129-2 kubenswrapper[2564]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Jul 23 19:19:34.462821 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:19:34.462745 2564 server.go:213] "--pod-infra-container-image will not be pruned by the image garbage collector in kubelet and should also be set in the remote runtime" Jul 23 19:19:34.469408 ip-10-0-129-2 kubenswrapper[2564]: W0723 19:19:34.469391 2564 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 23 19:19:34.469408 ip-10-0-129-2 kubenswrapper[2564]: W0723 19:19:34.469406 2564 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 23 19:19:34.469476 ip-10-0-129-2 kubenswrapper[2564]: W0723 19:19:34.469410 2564 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 23 19:19:34.469476 ip-10-0-129-2 kubenswrapper[2564]: W0723 19:19:34.469413 2564 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 23 19:19:34.469476 ip-10-0-129-2 kubenswrapper[2564]: W0723 19:19:34.469417 2564 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 23 19:19:34.469476 ip-10-0-129-2 kubenswrapper[2564]: W0723 19:19:34.469420 2564 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 23 19:19:34.469476 ip-10-0-129-2 kubenswrapper[2564]: W0723 19:19:34.469423 2564 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 23 19:19:34.469476 ip-10-0-129-2 kubenswrapper[2564]: W0723 19:19:34.469425 2564 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 23 19:19:34.469476 ip-10-0-129-2 kubenswrapper[2564]: W0723 19:19:34.469428 2564 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 23 19:19:34.469476 ip-10-0-129-2 kubenswrapper[2564]: W0723 19:19:34.469431 2564 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 23 19:19:34.469476 ip-10-0-129-2 kubenswrapper[2564]: W0723 19:19:34.469434 2564 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 23 19:19:34.469476 ip-10-0-129-2 kubenswrapper[2564]: W0723 19:19:34.469437 2564 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 23 19:19:34.469476 ip-10-0-129-2 kubenswrapper[2564]: W0723 19:19:34.469440 2564 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 23 19:19:34.469476 ip-10-0-129-2 kubenswrapper[2564]: W0723 19:19:34.469442 2564 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 23 19:19:34.469476 ip-10-0-129-2 kubenswrapper[2564]: W0723 19:19:34.469444 2564 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 23 19:19:34.469476 ip-10-0-129-2 kubenswrapper[2564]: W0723 19:19:34.469447 2564 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 23 19:19:34.469476 ip-10-0-129-2 kubenswrapper[2564]: W0723 19:19:34.469450 2564 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 23 19:19:34.469476 ip-10-0-129-2 kubenswrapper[2564]: W0723 19:19:34.469452 2564 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 23 19:19:34.469476 ip-10-0-129-2 kubenswrapper[2564]: W0723 19:19:34.469454 2564 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 23 19:19:34.469476 ip-10-0-129-2 kubenswrapper[2564]: W0723 19:19:34.469457 2564 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 23 19:19:34.469476 ip-10-0-129-2 kubenswrapper[2564]: W0723 19:19:34.469459 2564 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 23 19:19:34.469953 ip-10-0-129-2 kubenswrapper[2564]: W0723 19:19:34.469461 2564 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 23 19:19:34.469953 ip-10-0-129-2 kubenswrapper[2564]: W0723 19:19:34.469463 2564 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 23 19:19:34.469953 ip-10-0-129-2 kubenswrapper[2564]: W0723 19:19:34.469466 2564 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 23 19:19:34.469953 ip-10-0-129-2 kubenswrapper[2564]: W0723 19:19:34.469469 2564 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 23 19:19:34.469953 ip-10-0-129-2 kubenswrapper[2564]: W0723 19:19:34.469471 2564 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 23 19:19:34.469953 ip-10-0-129-2 kubenswrapper[2564]: W0723 19:19:34.469473 2564 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 23 19:19:34.469953 ip-10-0-129-2 kubenswrapper[2564]: W0723 19:19:34.469483 2564 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 23 19:19:34.469953 ip-10-0-129-2 kubenswrapper[2564]: W0723 19:19:34.469486 2564 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 23 19:19:34.469953 ip-10-0-129-2 kubenswrapper[2564]: W0723 19:19:34.469488 2564 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 23 19:19:34.469953 ip-10-0-129-2 kubenswrapper[2564]: W0723 19:19:34.469491 2564 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 23 19:19:34.469953 ip-10-0-129-2 kubenswrapper[2564]: W0723 19:19:34.469494 2564 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 23 19:19:34.469953 ip-10-0-129-2 kubenswrapper[2564]: W0723 19:19:34.469496 2564 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 23 19:19:34.469953 ip-10-0-129-2 kubenswrapper[2564]: W0723 19:19:34.469499 2564 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 23 19:19:34.469953 ip-10-0-129-2 kubenswrapper[2564]: W0723 19:19:34.469501 2564 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 23 19:19:34.469953 ip-10-0-129-2 kubenswrapper[2564]: W0723 19:19:34.469503 2564 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 23 19:19:34.469953 ip-10-0-129-2 kubenswrapper[2564]: W0723 19:19:34.469506 2564 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 23 19:19:34.469953 ip-10-0-129-2 kubenswrapper[2564]: W0723 19:19:34.469509 2564 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 23 19:19:34.469953 ip-10-0-129-2 kubenswrapper[2564]: W0723 19:19:34.469511 2564 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 23 19:19:34.469953 ip-10-0-129-2 kubenswrapper[2564]: W0723 19:19:34.469513 2564 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 23 19:19:34.469953 ip-10-0-129-2 kubenswrapper[2564]: W0723 19:19:34.469516 2564 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 23 19:19:34.470436 ip-10-0-129-2 kubenswrapper[2564]: W0723 19:19:34.469518 2564 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 23 19:19:34.470436 ip-10-0-129-2 kubenswrapper[2564]: W0723 19:19:34.469520 2564 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 23 19:19:34.470436 ip-10-0-129-2 kubenswrapper[2564]: W0723 19:19:34.469522 2564 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 23 19:19:34.470436 ip-10-0-129-2 kubenswrapper[2564]: W0723 19:19:34.469524 2564 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 23 19:19:34.470436 ip-10-0-129-2 kubenswrapper[2564]: W0723 19:19:34.469528 2564 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 23 19:19:34.470436 ip-10-0-129-2 kubenswrapper[2564]: W0723 19:19:34.469532 2564 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 23 19:19:34.470436 ip-10-0-129-2 kubenswrapper[2564]: W0723 19:19:34.469535 2564 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 23 19:19:34.470436 ip-10-0-129-2 kubenswrapper[2564]: W0723 19:19:34.469537 2564 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 23 19:19:34.470436 ip-10-0-129-2 kubenswrapper[2564]: W0723 19:19:34.469540 2564 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 23 19:19:34.470436 ip-10-0-129-2 kubenswrapper[2564]: W0723 19:19:34.469544 2564 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 23 19:19:34.470436 ip-10-0-129-2 kubenswrapper[2564]: W0723 19:19:34.469546 2564 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 23 19:19:34.470436 ip-10-0-129-2 kubenswrapper[2564]: W0723 19:19:34.469549 2564 feature_gate.go:328] unrecognized feature gate: Example Jul 23 19:19:34.470436 ip-10-0-129-2 kubenswrapper[2564]: W0723 19:19:34.469552 2564 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 23 19:19:34.470436 ip-10-0-129-2 kubenswrapper[2564]: W0723 19:19:34.469555 2564 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 23 19:19:34.470436 ip-10-0-129-2 kubenswrapper[2564]: W0723 19:19:34.469558 2564 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 23 19:19:34.470436 ip-10-0-129-2 kubenswrapper[2564]: W0723 19:19:34.469560 2564 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 23 19:19:34.470436 ip-10-0-129-2 kubenswrapper[2564]: W0723 19:19:34.469563 2564 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 23 19:19:34.470436 ip-10-0-129-2 kubenswrapper[2564]: W0723 19:19:34.469565 2564 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 23 19:19:34.470436 ip-10-0-129-2 kubenswrapper[2564]: W0723 19:19:34.469567 2564 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 23 19:19:34.470883 ip-10-0-129-2 kubenswrapper[2564]: W0723 19:19:34.469577 2564 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 23 19:19:34.470883 ip-10-0-129-2 kubenswrapper[2564]: W0723 19:19:34.469580 2564 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 23 19:19:34.470883 ip-10-0-129-2 kubenswrapper[2564]: W0723 19:19:34.469582 2564 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 23 19:19:34.470883 ip-10-0-129-2 kubenswrapper[2564]: W0723 19:19:34.469585 2564 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 23 19:19:34.470883 ip-10-0-129-2 kubenswrapper[2564]: W0723 19:19:34.469587 2564 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 23 19:19:34.470883 ip-10-0-129-2 kubenswrapper[2564]: W0723 19:19:34.469590 2564 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 23 19:19:34.470883 ip-10-0-129-2 kubenswrapper[2564]: W0723 19:19:34.469593 2564 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 23 19:19:34.470883 ip-10-0-129-2 kubenswrapper[2564]: W0723 19:19:34.469597 2564 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 23 19:19:34.470883 ip-10-0-129-2 kubenswrapper[2564]: W0723 19:19:34.469600 2564 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 23 19:19:34.470883 ip-10-0-129-2 kubenswrapper[2564]: W0723 19:19:34.469602 2564 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 23 19:19:34.470883 ip-10-0-129-2 kubenswrapper[2564]: W0723 19:19:34.469605 2564 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 23 19:19:34.470883 ip-10-0-129-2 kubenswrapper[2564]: W0723 19:19:34.469607 2564 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 23 19:19:34.470883 ip-10-0-129-2 kubenswrapper[2564]: W0723 19:19:34.469610 2564 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 23 19:19:34.470883 ip-10-0-129-2 kubenswrapper[2564]: W0723 19:19:34.469612 2564 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 23 19:19:34.470883 ip-10-0-129-2 kubenswrapper[2564]: W0723 19:19:34.469614 2564 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 23 19:19:34.470883 ip-10-0-129-2 kubenswrapper[2564]: W0723 19:19:34.469617 2564 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 23 19:19:34.470883 ip-10-0-129-2 kubenswrapper[2564]: W0723 19:19:34.469619 2564 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 23 19:19:34.470883 ip-10-0-129-2 kubenswrapper[2564]: W0723 19:19:34.469622 2564 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 23 19:19:34.470883 ip-10-0-129-2 kubenswrapper[2564]: W0723 19:19:34.469625 2564 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 23 19:19:34.471334 ip-10-0-129-2 kubenswrapper[2564]: W0723 19:19:34.469627 2564 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 23 19:19:34.471334 ip-10-0-129-2 kubenswrapper[2564]: W0723 19:19:34.469629 2564 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 23 19:19:34.471334 ip-10-0-129-2 kubenswrapper[2564]: W0723 19:19:34.469632 2564 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 23 19:19:34.471334 ip-10-0-129-2 kubenswrapper[2564]: W0723 19:19:34.469635 2564 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 23 19:19:34.471334 ip-10-0-129-2 kubenswrapper[2564]: W0723 19:19:34.469637 2564 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 23 19:19:34.471334 ip-10-0-129-2 kubenswrapper[2564]: W0723 19:19:34.469640 2564 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 23 19:19:34.471334 ip-10-0-129-2 kubenswrapper[2564]: W0723 19:19:34.469642 2564 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 23 19:19:34.471334 ip-10-0-129-2 kubenswrapper[2564]: W0723 19:19:34.469645 2564 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 23 19:19:34.471334 ip-10-0-129-2 kubenswrapper[2564]: W0723 19:19:34.469648 2564 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 23 19:19:34.471334 ip-10-0-129-2 kubenswrapper[2564]: W0723 19:19:34.469651 2564 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 23 19:19:34.471334 ip-10-0-129-2 kubenswrapper[2564]: W0723 19:19:34.469653 2564 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 23 19:19:34.471334 ip-10-0-129-2 kubenswrapper[2564]: W0723 19:19:34.469656 2564 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 23 19:19:34.471334 ip-10-0-129-2 kubenswrapper[2564]: W0723 19:19:34.469658 2564 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 23 19:19:34.471334 ip-10-0-129-2 kubenswrapper[2564]: W0723 19:19:34.469661 2564 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 23 19:19:34.471334 ip-10-0-129-2 kubenswrapper[2564]: W0723 19:19:34.469663 2564 feature_gate.go:328] unrecognized feature gate: Example2 Jul 23 19:19:34.471334 ip-10-0-129-2 kubenswrapper[2564]: W0723 19:19:34.469665 2564 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 23 19:19:34.471334 ip-10-0-129-2 kubenswrapper[2564]: W0723 19:19:34.469668 2564 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 23 19:19:34.471334 ip-10-0-129-2 kubenswrapper[2564]: W0723 19:19:34.469670 2564 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 23 19:19:34.471334 ip-10-0-129-2 kubenswrapper[2564]: W0723 19:19:34.469672 2564 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 23 19:19:34.471334 ip-10-0-129-2 kubenswrapper[2564]: W0723 19:19:34.469674 2564 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 23 19:19:34.471811 ip-10-0-129-2 kubenswrapper[2564]: W0723 19:19:34.471237 2564 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 23 19:19:34.471811 ip-10-0-129-2 kubenswrapper[2564]: W0723 19:19:34.471243 2564 feature_gate.go:328] unrecognized feature gate: Example2 Jul 23 19:19:34.471811 ip-10-0-129-2 kubenswrapper[2564]: W0723 19:19:34.471246 2564 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 23 19:19:34.471811 ip-10-0-129-2 kubenswrapper[2564]: W0723 19:19:34.471249 2564 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 23 19:19:34.471811 ip-10-0-129-2 kubenswrapper[2564]: W0723 19:19:34.471252 2564 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 23 19:19:34.471811 ip-10-0-129-2 kubenswrapper[2564]: W0723 19:19:34.471255 2564 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 23 19:19:34.471811 ip-10-0-129-2 kubenswrapper[2564]: W0723 19:19:34.471258 2564 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 23 19:19:34.471811 ip-10-0-129-2 kubenswrapper[2564]: W0723 19:19:34.471262 2564 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 23 19:19:34.471811 ip-10-0-129-2 kubenswrapper[2564]: W0723 19:19:34.471266 2564 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 23 19:19:34.471811 ip-10-0-129-2 kubenswrapper[2564]: W0723 19:19:34.471269 2564 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 23 19:19:34.471811 ip-10-0-129-2 kubenswrapper[2564]: W0723 19:19:34.471272 2564 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 23 19:19:34.471811 ip-10-0-129-2 kubenswrapper[2564]: W0723 19:19:34.471275 2564 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 23 19:19:34.471811 ip-10-0-129-2 kubenswrapper[2564]: W0723 19:19:34.471277 2564 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 23 19:19:34.471811 ip-10-0-129-2 kubenswrapper[2564]: W0723 19:19:34.471280 2564 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 23 19:19:34.471811 ip-10-0-129-2 kubenswrapper[2564]: W0723 19:19:34.471283 2564 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 23 19:19:34.471811 ip-10-0-129-2 kubenswrapper[2564]: W0723 19:19:34.471286 2564 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 23 19:19:34.471811 ip-10-0-129-2 kubenswrapper[2564]: W0723 19:19:34.471288 2564 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 23 19:19:34.471811 ip-10-0-129-2 kubenswrapper[2564]: W0723 19:19:34.471291 2564 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 23 19:19:34.471811 ip-10-0-129-2 kubenswrapper[2564]: W0723 19:19:34.471294 2564 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 23 19:19:34.472260 ip-10-0-129-2 kubenswrapper[2564]: W0723 19:19:34.471297 2564 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 23 19:19:34.472260 ip-10-0-129-2 kubenswrapper[2564]: W0723 19:19:34.471299 2564 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 23 19:19:34.472260 ip-10-0-129-2 kubenswrapper[2564]: W0723 19:19:34.471302 2564 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 23 19:19:34.472260 ip-10-0-129-2 kubenswrapper[2564]: W0723 19:19:34.471305 2564 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 23 19:19:34.472260 ip-10-0-129-2 kubenswrapper[2564]: W0723 19:19:34.471307 2564 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 23 19:19:34.472260 ip-10-0-129-2 kubenswrapper[2564]: W0723 19:19:34.471309 2564 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 23 19:19:34.472260 ip-10-0-129-2 kubenswrapper[2564]: W0723 19:19:34.471312 2564 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 23 19:19:34.472260 ip-10-0-129-2 kubenswrapper[2564]: W0723 19:19:34.471314 2564 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 23 19:19:34.472260 ip-10-0-129-2 kubenswrapper[2564]: W0723 19:19:34.471316 2564 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 23 19:19:34.472260 ip-10-0-129-2 kubenswrapper[2564]: W0723 19:19:34.471334 2564 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 23 19:19:34.472260 ip-10-0-129-2 kubenswrapper[2564]: W0723 19:19:34.471338 2564 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 23 19:19:34.472260 ip-10-0-129-2 kubenswrapper[2564]: W0723 19:19:34.471341 2564 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 23 19:19:34.472260 ip-10-0-129-2 kubenswrapper[2564]: W0723 19:19:34.471344 2564 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 23 19:19:34.472260 ip-10-0-129-2 kubenswrapper[2564]: W0723 19:19:34.471346 2564 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 23 19:19:34.472260 ip-10-0-129-2 kubenswrapper[2564]: W0723 19:19:34.471349 2564 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 23 19:19:34.472260 ip-10-0-129-2 kubenswrapper[2564]: W0723 19:19:34.471351 2564 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 23 19:19:34.472260 ip-10-0-129-2 kubenswrapper[2564]: W0723 19:19:34.471354 2564 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 23 19:19:34.472260 ip-10-0-129-2 kubenswrapper[2564]: W0723 19:19:34.471356 2564 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 23 19:19:34.472260 ip-10-0-129-2 kubenswrapper[2564]: W0723 19:19:34.471358 2564 feature_gate.go:328] unrecognized feature gate: Example Jul 23 19:19:34.472260 ip-10-0-129-2 kubenswrapper[2564]: W0723 19:19:34.471361 2564 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 23 19:19:34.472746 ip-10-0-129-2 kubenswrapper[2564]: W0723 19:19:34.471363 2564 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 23 19:19:34.472746 ip-10-0-129-2 kubenswrapper[2564]: W0723 19:19:34.471365 2564 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 23 19:19:34.472746 ip-10-0-129-2 kubenswrapper[2564]: W0723 19:19:34.471368 2564 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 23 19:19:34.472746 ip-10-0-129-2 kubenswrapper[2564]: W0723 19:19:34.471370 2564 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 23 19:19:34.472746 ip-10-0-129-2 kubenswrapper[2564]: W0723 19:19:34.471374 2564 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 23 19:19:34.472746 ip-10-0-129-2 kubenswrapper[2564]: W0723 19:19:34.471377 2564 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 23 19:19:34.472746 ip-10-0-129-2 kubenswrapper[2564]: W0723 19:19:34.471381 2564 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 23 19:19:34.472746 ip-10-0-129-2 kubenswrapper[2564]: W0723 19:19:34.471384 2564 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 23 19:19:34.472746 ip-10-0-129-2 kubenswrapper[2564]: W0723 19:19:34.471387 2564 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 23 19:19:34.472746 ip-10-0-129-2 kubenswrapper[2564]: W0723 19:19:34.471391 2564 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 23 19:19:34.472746 ip-10-0-129-2 kubenswrapper[2564]: W0723 19:19:34.471393 2564 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 23 19:19:34.472746 ip-10-0-129-2 kubenswrapper[2564]: W0723 19:19:34.471396 2564 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 23 19:19:34.472746 ip-10-0-129-2 kubenswrapper[2564]: W0723 19:19:34.471398 2564 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 23 19:19:34.472746 ip-10-0-129-2 kubenswrapper[2564]: W0723 19:19:34.471401 2564 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 23 19:19:34.472746 ip-10-0-129-2 kubenswrapper[2564]: W0723 19:19:34.471403 2564 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 23 19:19:34.472746 ip-10-0-129-2 kubenswrapper[2564]: W0723 19:19:34.471406 2564 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 23 19:19:34.472746 ip-10-0-129-2 kubenswrapper[2564]: W0723 19:19:34.471408 2564 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 23 19:19:34.472746 ip-10-0-129-2 kubenswrapper[2564]: W0723 19:19:34.471411 2564 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 23 19:19:34.472746 ip-10-0-129-2 kubenswrapper[2564]: W0723 19:19:34.471413 2564 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 23 19:19:34.472746 ip-10-0-129-2 kubenswrapper[2564]: W0723 19:19:34.471416 2564 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 23 19:19:34.473232 ip-10-0-129-2 kubenswrapper[2564]: W0723 19:19:34.471418 2564 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 23 19:19:34.473232 ip-10-0-129-2 kubenswrapper[2564]: W0723 19:19:34.471420 2564 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 23 19:19:34.473232 ip-10-0-129-2 kubenswrapper[2564]: W0723 19:19:34.471422 2564 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 23 19:19:34.473232 ip-10-0-129-2 kubenswrapper[2564]: W0723 19:19:34.471425 2564 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 23 19:19:34.473232 ip-10-0-129-2 kubenswrapper[2564]: W0723 19:19:34.471427 2564 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 23 19:19:34.473232 ip-10-0-129-2 kubenswrapper[2564]: W0723 19:19:34.471429 2564 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 23 19:19:34.473232 ip-10-0-129-2 kubenswrapper[2564]: W0723 19:19:34.471432 2564 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 23 19:19:34.473232 ip-10-0-129-2 kubenswrapper[2564]: W0723 19:19:34.471434 2564 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 23 19:19:34.473232 ip-10-0-129-2 kubenswrapper[2564]: W0723 19:19:34.471436 2564 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 23 19:19:34.473232 ip-10-0-129-2 kubenswrapper[2564]: W0723 19:19:34.471438 2564 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 23 19:19:34.473232 ip-10-0-129-2 kubenswrapper[2564]: W0723 19:19:34.471441 2564 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 23 19:19:34.473232 ip-10-0-129-2 kubenswrapper[2564]: W0723 19:19:34.471444 2564 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 23 19:19:34.473232 ip-10-0-129-2 kubenswrapper[2564]: W0723 19:19:34.471446 2564 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 23 19:19:34.473232 ip-10-0-129-2 kubenswrapper[2564]: W0723 19:19:34.471449 2564 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 23 19:19:34.473232 ip-10-0-129-2 kubenswrapper[2564]: W0723 19:19:34.471451 2564 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 23 19:19:34.473232 ip-10-0-129-2 kubenswrapper[2564]: W0723 19:19:34.471454 2564 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 23 19:19:34.473232 ip-10-0-129-2 kubenswrapper[2564]: W0723 19:19:34.471457 2564 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 23 19:19:34.473232 ip-10-0-129-2 kubenswrapper[2564]: W0723 19:19:34.471459 2564 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 23 19:19:34.473232 ip-10-0-129-2 kubenswrapper[2564]: W0723 19:19:34.471462 2564 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 23 19:19:34.473232 ip-10-0-129-2 kubenswrapper[2564]: W0723 19:19:34.471464 2564 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 23 19:19:34.473725 ip-10-0-129-2 kubenswrapper[2564]: W0723 19:19:34.471466 2564 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 23 19:19:34.473725 ip-10-0-129-2 kubenswrapper[2564]: W0723 19:19:34.471469 2564 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 23 19:19:34.473725 ip-10-0-129-2 kubenswrapper[2564]: W0723 19:19:34.471471 2564 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 23 19:19:34.473725 ip-10-0-129-2 kubenswrapper[2564]: W0723 19:19:34.471473 2564 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 23 19:19:34.473725 ip-10-0-129-2 kubenswrapper[2564]: W0723 19:19:34.471476 2564 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 23 19:19:34.473725 ip-10-0-129-2 kubenswrapper[2564]: W0723 19:19:34.471478 2564 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 23 19:19:34.473725 ip-10-0-129-2 kubenswrapper[2564]: W0723 19:19:34.471481 2564 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 23 19:19:34.473725 ip-10-0-129-2 kubenswrapper[2564]: W0723 19:19:34.471483 2564 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 23 19:19:34.473725 ip-10-0-129-2 kubenswrapper[2564]: W0723 19:19:34.471486 2564 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 23 19:19:34.473725 ip-10-0-129-2 kubenswrapper[2564]: W0723 19:19:34.471489 2564 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 23 19:19:34.473725 ip-10-0-129-2 kubenswrapper[2564]: W0723 19:19:34.471491 2564 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 23 19:19:34.473725 ip-10-0-129-2 kubenswrapper[2564]: W0723 19:19:34.471493 2564 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 23 19:19:34.473725 ip-10-0-129-2 kubenswrapper[2564]: W0723 19:19:34.471496 2564 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 23 19:19:34.473725 ip-10-0-129-2 kubenswrapper[2564]: W0723 19:19:34.471498 2564 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 23 19:19:34.473725 ip-10-0-129-2 kubenswrapper[2564]: W0723 19:19:34.471501 2564 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 23 19:19:34.473725 ip-10-0-129-2 kubenswrapper[2564]: W0723 19:19:34.471503 2564 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 23 19:19:34.473725 ip-10-0-129-2 kubenswrapper[2564]: W0723 19:19:34.471505 2564 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 23 19:19:34.473725 ip-10-0-129-2 kubenswrapper[2564]: W0723 19:19:34.471508 2564 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 23 19:19:34.473725 ip-10-0-129-2 kubenswrapper[2564]: W0723 19:19:34.471510 2564 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 23 19:19:34.473725 ip-10-0-129-2 kubenswrapper[2564]: W0723 19:19:34.471512 2564 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 23 19:19:34.474193 ip-10-0-129-2 kubenswrapper[2564]: W0723 19:19:34.471846 2564 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 23 19:19:34.474193 ip-10-0-129-2 kubenswrapper[2564]: W0723 19:19:34.471850 2564 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 23 19:19:34.474193 ip-10-0-129-2 kubenswrapper[2564]: W0723 19:19:34.471853 2564 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 23 19:19:34.474193 ip-10-0-129-2 kubenswrapper[2564]: W0723 19:19:34.471855 2564 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 23 19:19:34.474193 ip-10-0-129-2 kubenswrapper[2564]: W0723 19:19:34.471858 2564 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 23 19:19:34.474193 ip-10-0-129-2 kubenswrapper[2564]: W0723 19:19:34.471860 2564 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 23 19:19:34.474193 ip-10-0-129-2 kubenswrapper[2564]: W0723 19:19:34.471863 2564 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 23 19:19:34.474193 ip-10-0-129-2 kubenswrapper[2564]: W0723 19:19:34.471865 2564 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 23 19:19:34.474193 ip-10-0-129-2 kubenswrapper[2564]: W0723 19:19:34.471867 2564 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 23 19:19:34.474193 ip-10-0-129-2 kubenswrapper[2564]: W0723 19:19:34.471870 2564 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 23 19:19:34.474193 ip-10-0-129-2 kubenswrapper[2564]: W0723 19:19:34.471872 2564 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 23 19:19:34.474193 ip-10-0-129-2 kubenswrapper[2564]: W0723 19:19:34.471874 2564 feature_gate.go:328] unrecognized feature gate: Example Jul 23 19:19:34.474193 ip-10-0-129-2 kubenswrapper[2564]: W0723 19:19:34.471877 2564 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 23 19:19:34.474193 ip-10-0-129-2 kubenswrapper[2564]: W0723 19:19:34.471879 2564 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 23 19:19:34.474193 ip-10-0-129-2 kubenswrapper[2564]: W0723 19:19:34.471882 2564 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 23 19:19:34.474193 ip-10-0-129-2 kubenswrapper[2564]: W0723 19:19:34.471884 2564 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 23 19:19:34.474193 ip-10-0-129-2 kubenswrapper[2564]: W0723 19:19:34.471887 2564 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 23 19:19:34.474193 ip-10-0-129-2 kubenswrapper[2564]: W0723 19:19:34.471889 2564 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 23 19:19:34.474193 ip-10-0-129-2 kubenswrapper[2564]: W0723 19:19:34.471891 2564 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 23 19:19:34.474193 ip-10-0-129-2 kubenswrapper[2564]: W0723 19:19:34.471894 2564 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 23 19:19:34.474711 ip-10-0-129-2 kubenswrapper[2564]: W0723 19:19:34.471897 2564 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 23 19:19:34.474711 ip-10-0-129-2 kubenswrapper[2564]: W0723 19:19:34.471899 2564 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 23 19:19:34.474711 ip-10-0-129-2 kubenswrapper[2564]: W0723 19:19:34.471902 2564 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 23 19:19:34.474711 ip-10-0-129-2 kubenswrapper[2564]: W0723 19:19:34.471904 2564 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 23 19:19:34.474711 ip-10-0-129-2 kubenswrapper[2564]: W0723 19:19:34.471906 2564 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 23 19:19:34.474711 ip-10-0-129-2 kubenswrapper[2564]: W0723 19:19:34.471908 2564 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 23 19:19:34.474711 ip-10-0-129-2 kubenswrapper[2564]: W0723 19:19:34.471911 2564 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 23 19:19:34.474711 ip-10-0-129-2 kubenswrapper[2564]: W0723 19:19:34.471913 2564 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 23 19:19:34.474711 ip-10-0-129-2 kubenswrapper[2564]: W0723 19:19:34.471916 2564 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 23 19:19:34.474711 ip-10-0-129-2 kubenswrapper[2564]: W0723 19:19:34.471918 2564 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 23 19:19:34.474711 ip-10-0-129-2 kubenswrapper[2564]: W0723 19:19:34.471921 2564 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 23 19:19:34.474711 ip-10-0-129-2 kubenswrapper[2564]: W0723 19:19:34.471923 2564 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 23 19:19:34.474711 ip-10-0-129-2 kubenswrapper[2564]: W0723 19:19:34.471925 2564 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 23 19:19:34.474711 ip-10-0-129-2 kubenswrapper[2564]: W0723 19:19:34.471928 2564 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 23 19:19:34.474711 ip-10-0-129-2 kubenswrapper[2564]: W0723 19:19:34.471930 2564 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 23 19:19:34.474711 ip-10-0-129-2 kubenswrapper[2564]: W0723 19:19:34.471932 2564 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 23 19:19:34.474711 ip-10-0-129-2 kubenswrapper[2564]: W0723 19:19:34.471935 2564 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 23 19:19:34.474711 ip-10-0-129-2 kubenswrapper[2564]: W0723 19:19:34.471938 2564 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 23 19:19:34.474711 ip-10-0-129-2 kubenswrapper[2564]: W0723 19:19:34.471942 2564 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 23 19:19:34.475159 ip-10-0-129-2 kubenswrapper[2564]: W0723 19:19:34.471944 2564 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 23 19:19:34.475159 ip-10-0-129-2 kubenswrapper[2564]: W0723 19:19:34.471947 2564 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 23 19:19:34.475159 ip-10-0-129-2 kubenswrapper[2564]: W0723 19:19:34.471949 2564 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 23 19:19:34.475159 ip-10-0-129-2 kubenswrapper[2564]: W0723 19:19:34.471952 2564 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 23 19:19:34.475159 ip-10-0-129-2 kubenswrapper[2564]: W0723 19:19:34.471956 2564 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 23 19:19:34.475159 ip-10-0-129-2 kubenswrapper[2564]: W0723 19:19:34.471959 2564 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 23 19:19:34.475159 ip-10-0-129-2 kubenswrapper[2564]: W0723 19:19:34.471962 2564 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 23 19:19:34.475159 ip-10-0-129-2 kubenswrapper[2564]: W0723 19:19:34.471965 2564 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 23 19:19:34.475159 ip-10-0-129-2 kubenswrapper[2564]: W0723 19:19:34.471967 2564 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 23 19:19:34.475159 ip-10-0-129-2 kubenswrapper[2564]: W0723 19:19:34.471971 2564 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 23 19:19:34.475159 ip-10-0-129-2 kubenswrapper[2564]: W0723 19:19:34.471973 2564 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 23 19:19:34.475159 ip-10-0-129-2 kubenswrapper[2564]: W0723 19:19:34.471976 2564 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 23 19:19:34.475159 ip-10-0-129-2 kubenswrapper[2564]: W0723 19:19:34.471978 2564 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 23 19:19:34.475159 ip-10-0-129-2 kubenswrapper[2564]: W0723 19:19:34.471980 2564 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 23 19:19:34.475159 ip-10-0-129-2 kubenswrapper[2564]: W0723 19:19:34.471984 2564 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 23 19:19:34.475159 ip-10-0-129-2 kubenswrapper[2564]: W0723 19:19:34.471986 2564 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 23 19:19:34.475159 ip-10-0-129-2 kubenswrapper[2564]: W0723 19:19:34.471989 2564 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 23 19:19:34.475159 ip-10-0-129-2 kubenswrapper[2564]: W0723 19:19:34.471993 2564 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 23 19:19:34.475159 ip-10-0-129-2 kubenswrapper[2564]: W0723 19:19:34.471996 2564 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 23 19:19:34.475738 ip-10-0-129-2 kubenswrapper[2564]: W0723 19:19:34.471999 2564 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 23 19:19:34.475738 ip-10-0-129-2 kubenswrapper[2564]: W0723 19:19:34.472001 2564 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 23 19:19:34.475738 ip-10-0-129-2 kubenswrapper[2564]: W0723 19:19:34.472004 2564 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 23 19:19:34.475738 ip-10-0-129-2 kubenswrapper[2564]: W0723 19:19:34.472006 2564 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 23 19:19:34.475738 ip-10-0-129-2 kubenswrapper[2564]: W0723 19:19:34.472008 2564 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 23 19:19:34.475738 ip-10-0-129-2 kubenswrapper[2564]: W0723 19:19:34.472011 2564 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 23 19:19:34.475738 ip-10-0-129-2 kubenswrapper[2564]: W0723 19:19:34.472014 2564 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 23 19:19:34.475738 ip-10-0-129-2 kubenswrapper[2564]: W0723 19:19:34.472016 2564 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 23 19:19:34.475738 ip-10-0-129-2 kubenswrapper[2564]: W0723 19:19:34.472018 2564 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 23 19:19:34.475738 ip-10-0-129-2 kubenswrapper[2564]: W0723 19:19:34.472020 2564 feature_gate.go:328] unrecognized feature gate: Example2 Jul 23 19:19:34.475738 ip-10-0-129-2 kubenswrapper[2564]: W0723 19:19:34.472022 2564 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 23 19:19:34.475738 ip-10-0-129-2 kubenswrapper[2564]: W0723 19:19:34.472025 2564 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 23 19:19:34.475738 ip-10-0-129-2 kubenswrapper[2564]: W0723 19:19:34.472027 2564 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 23 19:19:34.475738 ip-10-0-129-2 kubenswrapper[2564]: W0723 19:19:34.472029 2564 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 23 19:19:34.475738 ip-10-0-129-2 kubenswrapper[2564]: W0723 19:19:34.472031 2564 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 23 19:19:34.475738 ip-10-0-129-2 kubenswrapper[2564]: W0723 19:19:34.472034 2564 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 23 19:19:34.475738 ip-10-0-129-2 kubenswrapper[2564]: W0723 19:19:34.472036 2564 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 23 19:19:34.475738 ip-10-0-129-2 kubenswrapper[2564]: W0723 19:19:34.472038 2564 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 23 19:19:34.475738 ip-10-0-129-2 kubenswrapper[2564]: W0723 19:19:34.472040 2564 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 23 19:19:34.476215 ip-10-0-129-2 kubenswrapper[2564]: W0723 19:19:34.472043 2564 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 23 19:19:34.476215 ip-10-0-129-2 kubenswrapper[2564]: W0723 19:19:34.472045 2564 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 23 19:19:34.476215 ip-10-0-129-2 kubenswrapper[2564]: W0723 19:19:34.472047 2564 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 23 19:19:34.476215 ip-10-0-129-2 kubenswrapper[2564]: W0723 19:19:34.472050 2564 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 23 19:19:34.476215 ip-10-0-129-2 kubenswrapper[2564]: W0723 19:19:34.472052 2564 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 23 19:19:34.476215 ip-10-0-129-2 kubenswrapper[2564]: W0723 19:19:34.472054 2564 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 23 19:19:34.476215 ip-10-0-129-2 kubenswrapper[2564]: W0723 19:19:34.472057 2564 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 23 19:19:34.476215 ip-10-0-129-2 kubenswrapper[2564]: W0723 19:19:34.472059 2564 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 23 19:19:34.476215 ip-10-0-129-2 kubenswrapper[2564]: W0723 19:19:34.472061 2564 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 23 19:19:34.476215 ip-10-0-129-2 kubenswrapper[2564]: W0723 19:19:34.472064 2564 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 23 19:19:34.476215 ip-10-0-129-2 kubenswrapper[2564]: W0723 19:19:34.472067 2564 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 23 19:19:34.476215 ip-10-0-129-2 kubenswrapper[2564]: W0723 19:19:34.472069 2564 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 23 19:19:34.476215 ip-10-0-129-2 kubenswrapper[2564]: W0723 19:19:34.472071 2564 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 23 19:19:34.476215 ip-10-0-129-2 kubenswrapper[2564]: W0723 19:19:34.472073 2564 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 23 19:19:34.476215 ip-10-0-129-2 kubenswrapper[2564]: W0723 19:19:34.472075 2564 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 23 19:19:34.476215 ip-10-0-129-2 kubenswrapper[2564]: W0723 19:19:34.472078 2564 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 23 19:19:34.476215 ip-10-0-129-2 kubenswrapper[2564]: W0723 19:19:34.472080 2564 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 23 19:19:34.476215 ip-10-0-129-2 kubenswrapper[2564]: W0723 19:19:34.472083 2564 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 23 19:19:34.476215 ip-10-0-129-2 kubenswrapper[2564]: W0723 19:19:34.472085 2564 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 23 19:19:34.476215 ip-10-0-129-2 kubenswrapper[2564]: W0723 19:19:34.472088 2564 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 23 19:19:34.476690 ip-10-0-129-2 kubenswrapper[2564]: W0723 19:19:34.472090 2564 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 23 19:19:34.476690 ip-10-0-129-2 kubenswrapper[2564]: W0723 19:19:34.472092 2564 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 23 19:19:34.476690 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:19:34.472156 2564 flags.go:64] FLAG: --address="0.0.0.0" Jul 23 19:19:34.476690 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:19:34.472163 2564 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]" Jul 23 19:19:34.476690 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:19:34.472168 2564 flags.go:64] FLAG: --anonymous-auth="true" Jul 23 19:19:34.476690 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:19:34.472174 2564 flags.go:64] FLAG: --application-metrics-count-limit="100" Jul 23 19:19:34.476690 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:19:34.472179 2564 flags.go:64] FLAG: --authentication-token-webhook="false" Jul 23 19:19:34.476690 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:19:34.472184 2564 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s" Jul 23 19:19:34.476690 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:19:34.472189 2564 flags.go:64] FLAG: --authorization-mode="AlwaysAllow" Jul 23 19:19:34.476690 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:19:34.472195 2564 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s" Jul 23 19:19:34.476690 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:19:34.472199 2564 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s" Jul 23 19:19:34.476690 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:19:34.472202 2564 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id" Jul 23 19:19:34.476690 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:19:34.472206 2564 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig" Jul 23 19:19:34.476690 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:19:34.472208 2564 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki" Jul 23 19:19:34.476690 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:19:34.472211 2564 flags.go:64] FLAG: --cgroup-driver="cgroupfs" Jul 23 19:19:34.476690 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:19:34.472214 2564 flags.go:64] FLAG: --cgroup-root="" Jul 23 19:19:34.476690 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:19:34.472216 2564 flags.go:64] FLAG: --cgroups-per-qos="true" Jul 23 19:19:34.476690 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:19:34.472219 2564 flags.go:64] FLAG: --client-ca-file="" Jul 23 19:19:34.476690 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:19:34.472222 2564 flags.go:64] FLAG: --cloud-provider="external" Jul 23 19:19:34.476690 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:19:34.472224 2564 flags.go:64] FLAG: --cluster-dns="[]" Jul 23 19:19:34.476690 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:19:34.472229 2564 flags.go:64] FLAG: --cluster-domain="" Jul 23 19:19:34.476690 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:19:34.472231 2564 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf" Jul 23 19:19:34.476690 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:19:34.472235 2564 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d" Jul 23 19:19:34.477222 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:19:34.472238 2564 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json" Jul 23 19:19:34.477222 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:19:34.472240 2564 flags.go:64] FLAG: --container-log-max-files="5" Jul 23 19:19:34.477222 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:19:34.472244 2564 flags.go:64] FLAG: --container-log-max-size="10Mi" Jul 23 19:19:34.477222 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:19:34.472246 2564 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock" Jul 23 19:19:34.477222 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:19:34.472249 2564 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock" Jul 23 19:19:34.477222 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:19:34.472252 2564 flags.go:64] FLAG: --containerd-namespace="k8s.io" Jul 23 19:19:34.477222 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:19:34.472254 2564 flags.go:64] FLAG: --contention-profiling="false" Jul 23 19:19:34.477222 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:19:34.472257 2564 flags.go:64] FLAG: --cpu-cfs-quota="true" Jul 23 19:19:34.477222 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:19:34.472259 2564 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms" Jul 23 19:19:34.477222 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:19:34.472262 2564 flags.go:64] FLAG: --cpu-manager-policy="none" Jul 23 19:19:34.477222 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:19:34.472266 2564 flags.go:64] FLAG: --cpu-manager-policy-options="" Jul 23 19:19:34.477222 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:19:34.472270 2564 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s" Jul 23 19:19:34.477222 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:19:34.472272 2564 flags.go:64] FLAG: --enable-controller-attach-detach="true" Jul 23 19:19:34.477222 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:19:34.472275 2564 flags.go:64] FLAG: --enable-debugging-handlers="true" Jul 23 19:19:34.477222 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:19:34.472278 2564 flags.go:64] FLAG: --enable-load-reader="false" Jul 23 19:19:34.477222 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:19:34.472280 2564 flags.go:64] FLAG: --enable-server="true" Jul 23 19:19:34.477222 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:19:34.472283 2564 flags.go:64] FLAG: --enforce-node-allocatable="[pods]" Jul 23 19:19:34.477222 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:19:34.472286 2564 flags.go:64] FLAG: --event-burst="100" Jul 23 19:19:34.477222 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:19:34.472289 2564 flags.go:64] FLAG: --event-qps="50" Jul 23 19:19:34.477222 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:19:34.472292 2564 flags.go:64] FLAG: --event-storage-age-limit="default=0" Jul 23 19:19:34.477222 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:19:34.472294 2564 flags.go:64] FLAG: --event-storage-event-limit="default=0" Jul 23 19:19:34.477222 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:19:34.472297 2564 flags.go:64] FLAG: --eviction-hard="" Jul 23 19:19:34.477222 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:19:34.472300 2564 flags.go:64] FLAG: --eviction-max-pod-grace-period="0" Jul 23 19:19:34.477222 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:19:34.472303 2564 flags.go:64] FLAG: --eviction-minimum-reclaim="" Jul 23 19:19:34.477806 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:19:34.472305 2564 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s" Jul 23 19:19:34.477806 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:19:34.472308 2564 flags.go:64] FLAG: --eviction-soft="" Jul 23 19:19:34.477806 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:19:34.472311 2564 flags.go:64] FLAG: --eviction-soft-grace-period="" Jul 23 19:19:34.477806 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:19:34.472313 2564 flags.go:64] FLAG: --exit-on-lock-contention="false" Jul 23 19:19:34.477806 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:19:34.472316 2564 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false" Jul 23 19:19:34.477806 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:19:34.472331 2564 flags.go:64] FLAG: --experimental-mounter-path="" Jul 23 19:19:34.477806 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:19:34.472334 2564 flags.go:64] FLAG: --fail-cgroupv1="false" Jul 23 19:19:34.477806 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:19:34.472336 2564 flags.go:64] FLAG: --fail-swap-on="true" Jul 23 19:19:34.477806 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:19:34.472339 2564 flags.go:64] FLAG: --feature-gates="" Jul 23 19:19:34.477806 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:19:34.472343 2564 flags.go:64] FLAG: --file-check-frequency="20s" Jul 23 19:19:34.477806 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:19:34.472345 2564 flags.go:64] FLAG: --global-housekeeping-interval="1m0s" Jul 23 19:19:34.477806 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:19:34.472348 2564 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge" Jul 23 19:19:34.477806 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:19:34.472351 2564 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1" Jul 23 19:19:34.477806 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:19:34.472353 2564 flags.go:64] FLAG: --healthz-port="10248" Jul 23 19:19:34.477806 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:19:34.472356 2564 flags.go:64] FLAG: --help="false" Jul 23 19:19:34.477806 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:19:34.472359 2564 flags.go:64] FLAG: --hostname-override="ip-10-0-129-2.ec2.internal" Jul 23 19:19:34.477806 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:19:34.472362 2564 flags.go:64] FLAG: --housekeeping-interval="10s" Jul 23 19:19:34.477806 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:19:34.472364 2564 flags.go:64] FLAG: --http-check-frequency="20s" Jul 23 19:19:34.477806 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:19:34.472367 2564 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins" Jul 23 19:19:34.477806 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:19:34.472370 2564 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml" Jul 23 19:19:34.477806 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:19:34.472373 2564 flags.go:64] FLAG: --image-gc-high-threshold="85" Jul 23 19:19:34.477806 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:19:34.472375 2564 flags.go:64] FLAG: --image-gc-low-threshold="80" Jul 23 19:19:34.477806 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:19:34.472378 2564 flags.go:64] FLAG: --image-service-endpoint="" Jul 23 19:19:34.478478 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:19:34.472380 2564 flags.go:64] FLAG: --kernel-memcg-notification="false" Jul 23 19:19:34.478478 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:19:34.472383 2564 flags.go:64] FLAG: --kube-api-burst="100" Jul 23 19:19:34.478478 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:19:34.472385 2564 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf" Jul 23 19:19:34.478478 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:19:34.472388 2564 flags.go:64] FLAG: --kube-api-qps="50" Jul 23 19:19:34.478478 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:19:34.472391 2564 flags.go:64] FLAG: --kube-reserved="" Jul 23 19:19:34.478478 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:19:34.472393 2564 flags.go:64] FLAG: --kube-reserved-cgroup="" Jul 23 19:19:34.478478 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:19:34.472395 2564 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig" Jul 23 19:19:34.478478 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:19:34.472398 2564 flags.go:64] FLAG: --kubelet-cgroups="" Jul 23 19:19:34.478478 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:19:34.472400 2564 flags.go:64] FLAG: --local-storage-capacity-isolation="true" Jul 23 19:19:34.478478 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:19:34.472406 2564 flags.go:64] FLAG: --lock-file="" Jul 23 19:19:34.478478 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:19:34.472409 2564 flags.go:64] FLAG: --log-cadvisor-usage="false" Jul 23 19:19:34.478478 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:19:34.472412 2564 flags.go:64] FLAG: --log-flush-frequency="5s" Jul 23 19:19:34.478478 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:19:34.472414 2564 flags.go:64] FLAG: --log-json-info-buffer-size="0" Jul 23 19:19:34.478478 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:19:34.472420 2564 flags.go:64] FLAG: --log-json-split-stream="false" Jul 23 19:19:34.478478 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:19:34.472422 2564 flags.go:64] FLAG: --log-text-info-buffer-size="0" Jul 23 19:19:34.478478 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:19:34.472425 2564 flags.go:64] FLAG: --log-text-split-stream="false" Jul 23 19:19:34.478478 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:19:34.472427 2564 flags.go:64] FLAG: --logging-format="text" Jul 23 19:19:34.478478 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:19:34.472430 2564 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id" Jul 23 19:19:34.478478 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:19:34.472434 2564 flags.go:64] FLAG: --make-iptables-util-chains="true" Jul 23 19:19:34.478478 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:19:34.472436 2564 flags.go:64] FLAG: --manifest-url="" Jul 23 19:19:34.478478 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:19:34.472439 2564 flags.go:64] FLAG: --manifest-url-header="" Jul 23 19:19:34.478478 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:19:34.472442 2564 flags.go:64] FLAG: --max-housekeeping-interval="15s" Jul 23 19:19:34.478478 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:19:34.472445 2564 flags.go:64] FLAG: --max-open-files="1000000" Jul 23 19:19:34.478478 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:19:34.472450 2564 flags.go:64] FLAG: --max-pods="110" Jul 23 19:19:34.478478 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:19:34.472453 2564 flags.go:64] FLAG: --maximum-dead-containers="-1" Jul 23 19:19:34.479436 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:19:34.472455 2564 flags.go:64] FLAG: --maximum-dead-containers-per-container="1" Jul 23 19:19:34.479436 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:19:34.472458 2564 flags.go:64] FLAG: --memory-manager-policy="None" Jul 23 19:19:34.479436 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:19:34.472461 2564 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s" Jul 23 19:19:34.479436 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:19:34.472464 2564 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s" Jul 23 19:19:34.479436 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:19:34.472466 2564 flags.go:64] FLAG: --node-ip="0.0.0.0" Jul 23 19:19:34.479436 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:19:34.472469 2564 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel" Jul 23 19:19:34.479436 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:19:34.472475 2564 flags.go:64] FLAG: --node-status-max-images="50" Jul 23 19:19:34.479436 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:19:34.472478 2564 flags.go:64] FLAG: --node-status-update-frequency="10s" Jul 23 19:19:34.479436 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:19:34.472481 2564 flags.go:64] FLAG: --oom-score-adj="-999" Jul 23 19:19:34.479436 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:19:34.472483 2564 flags.go:64] FLAG: --pod-cidr="" Jul 23 19:19:34.479436 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:19:34.472486 2564 flags.go:64] FLAG: --pod-infra-container-image="quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:4017e99d76d7b91598d3a3430cd0eba6eda8b33653addd3838fdfe3cca69c5a0" Jul 23 19:19:34.479436 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:19:34.472491 2564 flags.go:64] FLAG: --pod-manifest-path="" Jul 23 19:19:34.479436 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:19:34.472493 2564 flags.go:64] FLAG: --pod-max-pids="-1" Jul 23 19:19:34.479436 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:19:34.472496 2564 flags.go:64] FLAG: --pods-per-core="0" Jul 23 19:19:34.479436 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:19:34.472498 2564 flags.go:64] FLAG: --port="10250" Jul 23 19:19:34.479436 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:19:34.472501 2564 flags.go:64] FLAG: --protect-kernel-defaults="false" Jul 23 19:19:34.479436 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:19:34.472504 2564 flags.go:64] FLAG: --provider-id="aws:///us-east-1a/i-0caaffb7e66956c7d" Jul 23 19:19:34.479436 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:19:34.472509 2564 flags.go:64] FLAG: --qos-reserved="" Jul 23 19:19:34.479436 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:19:34.472511 2564 flags.go:64] FLAG: --read-only-port="10255" Jul 23 19:19:34.479436 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:19:34.472514 2564 flags.go:64] FLAG: --register-node="true" Jul 23 19:19:34.479436 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:19:34.472517 2564 flags.go:64] FLAG: --register-with-taints="" Jul 23 19:19:34.479436 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:19:34.472520 2564 flags.go:64] FLAG: --registry-burst="10" Jul 23 19:19:34.479436 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:19:34.472522 2564 flags.go:64] FLAG: --registry-qps="5" Jul 23 19:19:34.479436 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:19:34.472525 2564 flags.go:64] FLAG: --reserved-cpus="" Jul 23 19:19:34.480041 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:19:34.472527 2564 flags.go:64] FLAG: --reserved-memory="" Jul 23 19:19:34.480041 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:19:34.472531 2564 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf" Jul 23 19:19:34.480041 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:19:34.472534 2564 flags.go:64] FLAG: --root-dir="/var/lib/kubelet" Jul 23 19:19:34.480041 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:19:34.472536 2564 flags.go:64] FLAG: --rotate-certificates="false" Jul 23 19:19:34.480041 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:19:34.472539 2564 flags.go:64] FLAG: --rotate-server-certificates="false" Jul 23 19:19:34.480041 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:19:34.472542 2564 flags.go:64] FLAG: --runonce="false" Jul 23 19:19:34.480041 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:19:34.472547 2564 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service" Jul 23 19:19:34.480041 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:19:34.472550 2564 flags.go:64] FLAG: --runtime-request-timeout="2m0s" Jul 23 19:19:34.480041 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:19:34.472553 2564 flags.go:64] FLAG: --seccomp-default="false" Jul 23 19:19:34.480041 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:19:34.472555 2564 flags.go:64] FLAG: --serialize-image-pulls="true" Jul 23 19:19:34.480041 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:19:34.472558 2564 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s" Jul 23 19:19:34.480041 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:19:34.472561 2564 flags.go:64] FLAG: --storage-driver-db="cadvisor" Jul 23 19:19:34.480041 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:19:34.472563 2564 flags.go:64] FLAG: --storage-driver-host="localhost:8086" Jul 23 19:19:34.480041 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:19:34.472566 2564 flags.go:64] FLAG: --storage-driver-password="root" Jul 23 19:19:34.480041 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:19:34.472569 2564 flags.go:64] FLAG: --storage-driver-secure="false" Jul 23 19:19:34.480041 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:19:34.472571 2564 flags.go:64] FLAG: --storage-driver-table="stats" Jul 23 19:19:34.480041 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:19:34.472574 2564 flags.go:64] FLAG: --storage-driver-user="root" Jul 23 19:19:34.480041 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:19:34.472576 2564 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s" Jul 23 19:19:34.480041 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:19:34.472579 2564 flags.go:64] FLAG: --sync-frequency="1m0s" Jul 23 19:19:34.480041 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:19:34.472582 2564 flags.go:64] FLAG: --system-cgroups="" Jul 23 19:19:34.480041 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:19:34.472584 2564 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi" Jul 23 19:19:34.480041 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:19:34.472589 2564 flags.go:64] FLAG: --system-reserved-cgroup="" Jul 23 19:19:34.480041 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:19:34.472591 2564 flags.go:64] FLAG: --tls-cert-file="" Jul 23 19:19:34.480041 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:19:34.472594 2564 flags.go:64] FLAG: --tls-cipher-suites="[]" Jul 23 19:19:34.480041 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:19:34.472602 2564 flags.go:64] FLAG: --tls-min-version="" Jul 23 19:19:34.480645 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:19:34.472605 2564 flags.go:64] FLAG: --tls-private-key-file="" Jul 23 19:19:34.480645 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:19:34.472608 2564 flags.go:64] FLAG: --topology-manager-policy="none" Jul 23 19:19:34.480645 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:19:34.472610 2564 flags.go:64] FLAG: --topology-manager-policy-options="" Jul 23 19:19:34.480645 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:19:34.472613 2564 flags.go:64] FLAG: --topology-manager-scope="container" Jul 23 19:19:34.480645 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:19:34.472616 2564 flags.go:64] FLAG: --v="2" Jul 23 19:19:34.480645 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:19:34.472620 2564 flags.go:64] FLAG: --version="false" Jul 23 19:19:34.480645 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:19:34.472623 2564 flags.go:64] FLAG: --vmodule="" Jul 23 19:19:34.480645 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:19:34.472626 2564 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec" Jul 23 19:19:34.480645 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:19:34.472629 2564 flags.go:64] FLAG: --volume-stats-agg-period="1m0s" Jul 23 19:19:34.480645 ip-10-0-129-2 kubenswrapper[2564]: W0723 19:19:34.472741 2564 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 23 19:19:34.480645 ip-10-0-129-2 kubenswrapper[2564]: W0723 19:19:34.472745 2564 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 23 19:19:34.480645 ip-10-0-129-2 kubenswrapper[2564]: W0723 19:19:34.472748 2564 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 23 19:19:34.480645 ip-10-0-129-2 kubenswrapper[2564]: W0723 19:19:34.472753 2564 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 23 19:19:34.480645 ip-10-0-129-2 kubenswrapper[2564]: W0723 19:19:34.472757 2564 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 23 19:19:34.480645 ip-10-0-129-2 kubenswrapper[2564]: W0723 19:19:34.472760 2564 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 23 19:19:34.480645 ip-10-0-129-2 kubenswrapper[2564]: W0723 19:19:34.472763 2564 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 23 19:19:34.480645 ip-10-0-129-2 kubenswrapper[2564]: W0723 19:19:34.472767 2564 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 23 19:19:34.480645 ip-10-0-129-2 kubenswrapper[2564]: W0723 19:19:34.472770 2564 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 23 19:19:34.480645 ip-10-0-129-2 kubenswrapper[2564]: W0723 19:19:34.472773 2564 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 23 19:19:34.480645 ip-10-0-129-2 kubenswrapper[2564]: W0723 19:19:34.472777 2564 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 23 19:19:34.481120 ip-10-0-129-2 kubenswrapper[2564]: W0723 19:19:34.472780 2564 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 23 19:19:34.481120 ip-10-0-129-2 kubenswrapper[2564]: W0723 19:19:34.472782 2564 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 23 19:19:34.481120 ip-10-0-129-2 kubenswrapper[2564]: W0723 19:19:34.472785 2564 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 23 19:19:34.481120 ip-10-0-129-2 kubenswrapper[2564]: W0723 19:19:34.472787 2564 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 23 19:19:34.481120 ip-10-0-129-2 kubenswrapper[2564]: W0723 19:19:34.472789 2564 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 23 19:19:34.481120 ip-10-0-129-2 kubenswrapper[2564]: W0723 19:19:34.472792 2564 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 23 19:19:34.481120 ip-10-0-129-2 kubenswrapper[2564]: W0723 19:19:34.472795 2564 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 23 19:19:34.481120 ip-10-0-129-2 kubenswrapper[2564]: W0723 19:19:34.472797 2564 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 23 19:19:34.481120 ip-10-0-129-2 kubenswrapper[2564]: W0723 19:19:34.472800 2564 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 23 19:19:34.481120 ip-10-0-129-2 kubenswrapper[2564]: W0723 19:19:34.472802 2564 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 23 19:19:34.481120 ip-10-0-129-2 kubenswrapper[2564]: W0723 19:19:34.472804 2564 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 23 19:19:34.481120 ip-10-0-129-2 kubenswrapper[2564]: W0723 19:19:34.472807 2564 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 23 19:19:34.481120 ip-10-0-129-2 kubenswrapper[2564]: W0723 19:19:34.472810 2564 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 23 19:19:34.481120 ip-10-0-129-2 kubenswrapper[2564]: W0723 19:19:34.472813 2564 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 23 19:19:34.481120 ip-10-0-129-2 kubenswrapper[2564]: W0723 19:19:34.472815 2564 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 23 19:19:34.481120 ip-10-0-129-2 kubenswrapper[2564]: W0723 19:19:34.472817 2564 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 23 19:19:34.481120 ip-10-0-129-2 kubenswrapper[2564]: W0723 19:19:34.472820 2564 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 23 19:19:34.481120 ip-10-0-129-2 kubenswrapper[2564]: W0723 19:19:34.472822 2564 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 23 19:19:34.481120 ip-10-0-129-2 kubenswrapper[2564]: W0723 19:19:34.472824 2564 feature_gate.go:328] unrecognized feature gate: Example2 Jul 23 19:19:34.481120 ip-10-0-129-2 kubenswrapper[2564]: W0723 19:19:34.472827 2564 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 23 19:19:34.481675 ip-10-0-129-2 kubenswrapper[2564]: W0723 19:19:34.472829 2564 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 23 19:19:34.481675 ip-10-0-129-2 kubenswrapper[2564]: W0723 19:19:34.472831 2564 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 23 19:19:34.481675 ip-10-0-129-2 kubenswrapper[2564]: W0723 19:19:34.472833 2564 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 23 19:19:34.481675 ip-10-0-129-2 kubenswrapper[2564]: W0723 19:19:34.472836 2564 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 23 19:19:34.481675 ip-10-0-129-2 kubenswrapper[2564]: W0723 19:19:34.472839 2564 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 23 19:19:34.481675 ip-10-0-129-2 kubenswrapper[2564]: W0723 19:19:34.472842 2564 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 23 19:19:34.481675 ip-10-0-129-2 kubenswrapper[2564]: W0723 19:19:34.472844 2564 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 23 19:19:34.481675 ip-10-0-129-2 kubenswrapper[2564]: W0723 19:19:34.472846 2564 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 23 19:19:34.481675 ip-10-0-129-2 kubenswrapper[2564]: W0723 19:19:34.472849 2564 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 23 19:19:34.481675 ip-10-0-129-2 kubenswrapper[2564]: W0723 19:19:34.472851 2564 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 23 19:19:34.481675 ip-10-0-129-2 kubenswrapper[2564]: W0723 19:19:34.472854 2564 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 23 19:19:34.481675 ip-10-0-129-2 kubenswrapper[2564]: W0723 19:19:34.472856 2564 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 23 19:19:34.481675 ip-10-0-129-2 kubenswrapper[2564]: W0723 19:19:34.472859 2564 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 23 19:19:34.481675 ip-10-0-129-2 kubenswrapper[2564]: W0723 19:19:34.472861 2564 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 23 19:19:34.481675 ip-10-0-129-2 kubenswrapper[2564]: W0723 19:19:34.472863 2564 feature_gate.go:328] unrecognized feature gate: Example Jul 23 19:19:34.481675 ip-10-0-129-2 kubenswrapper[2564]: W0723 19:19:34.472865 2564 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 23 19:19:34.481675 ip-10-0-129-2 kubenswrapper[2564]: W0723 19:19:34.472868 2564 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 23 19:19:34.481675 ip-10-0-129-2 kubenswrapper[2564]: W0723 19:19:34.472870 2564 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 23 19:19:34.481675 ip-10-0-129-2 kubenswrapper[2564]: W0723 19:19:34.472872 2564 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 23 19:19:34.481675 ip-10-0-129-2 kubenswrapper[2564]: W0723 19:19:34.472875 2564 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 23 19:19:34.482152 ip-10-0-129-2 kubenswrapper[2564]: W0723 19:19:34.472877 2564 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 23 19:19:34.482152 ip-10-0-129-2 kubenswrapper[2564]: W0723 19:19:34.472879 2564 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 23 19:19:34.482152 ip-10-0-129-2 kubenswrapper[2564]: W0723 19:19:34.472882 2564 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 23 19:19:34.482152 ip-10-0-129-2 kubenswrapper[2564]: W0723 19:19:34.472884 2564 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 23 19:19:34.482152 ip-10-0-129-2 kubenswrapper[2564]: W0723 19:19:34.472887 2564 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 23 19:19:34.482152 ip-10-0-129-2 kubenswrapper[2564]: W0723 19:19:34.472890 2564 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 23 19:19:34.482152 ip-10-0-129-2 kubenswrapper[2564]: W0723 19:19:34.472892 2564 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 23 19:19:34.482152 ip-10-0-129-2 kubenswrapper[2564]: W0723 19:19:34.472894 2564 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 23 19:19:34.482152 ip-10-0-129-2 kubenswrapper[2564]: W0723 19:19:34.472897 2564 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 23 19:19:34.482152 ip-10-0-129-2 kubenswrapper[2564]: W0723 19:19:34.472899 2564 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 23 19:19:34.482152 ip-10-0-129-2 kubenswrapper[2564]: W0723 19:19:34.472901 2564 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 23 19:19:34.482152 ip-10-0-129-2 kubenswrapper[2564]: W0723 19:19:34.472903 2564 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 23 19:19:34.482152 ip-10-0-129-2 kubenswrapper[2564]: W0723 19:19:34.472906 2564 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 23 19:19:34.482152 ip-10-0-129-2 kubenswrapper[2564]: W0723 19:19:34.472908 2564 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 23 19:19:34.482152 ip-10-0-129-2 kubenswrapper[2564]: W0723 19:19:34.472910 2564 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 23 19:19:34.482152 ip-10-0-129-2 kubenswrapper[2564]: W0723 19:19:34.472913 2564 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 23 19:19:34.482152 ip-10-0-129-2 kubenswrapper[2564]: W0723 19:19:34.472916 2564 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 23 19:19:34.482152 ip-10-0-129-2 kubenswrapper[2564]: W0723 19:19:34.472919 2564 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 23 19:19:34.482152 ip-10-0-129-2 kubenswrapper[2564]: W0723 19:19:34.472921 2564 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 23 19:19:34.482152 ip-10-0-129-2 kubenswrapper[2564]: W0723 19:19:34.472924 2564 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 23 19:19:34.482644 ip-10-0-129-2 kubenswrapper[2564]: W0723 19:19:34.472926 2564 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 23 19:19:34.482644 ip-10-0-129-2 kubenswrapper[2564]: W0723 19:19:34.472928 2564 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 23 19:19:34.482644 ip-10-0-129-2 kubenswrapper[2564]: W0723 19:19:34.472931 2564 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 23 19:19:34.482644 ip-10-0-129-2 kubenswrapper[2564]: W0723 19:19:34.472933 2564 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 23 19:19:34.482644 ip-10-0-129-2 kubenswrapper[2564]: W0723 19:19:34.472935 2564 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 23 19:19:34.482644 ip-10-0-129-2 kubenswrapper[2564]: W0723 19:19:34.472938 2564 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 23 19:19:34.482644 ip-10-0-129-2 kubenswrapper[2564]: W0723 19:19:34.472940 2564 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 23 19:19:34.482644 ip-10-0-129-2 kubenswrapper[2564]: W0723 19:19:34.472943 2564 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 23 19:19:34.482644 ip-10-0-129-2 kubenswrapper[2564]: W0723 19:19:34.472945 2564 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 23 19:19:34.482644 ip-10-0-129-2 kubenswrapper[2564]: W0723 19:19:34.472947 2564 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 23 19:19:34.482644 ip-10-0-129-2 kubenswrapper[2564]: W0723 19:19:34.472949 2564 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 23 19:19:34.482644 ip-10-0-129-2 kubenswrapper[2564]: W0723 19:19:34.472952 2564 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 23 19:19:34.482644 ip-10-0-129-2 kubenswrapper[2564]: W0723 19:19:34.472954 2564 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 23 19:19:34.482644 ip-10-0-129-2 kubenswrapper[2564]: W0723 19:19:34.472956 2564 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 23 19:19:34.482644 ip-10-0-129-2 kubenswrapper[2564]: W0723 19:19:34.472959 2564 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 23 19:19:34.482644 ip-10-0-129-2 kubenswrapper[2564]: W0723 19:19:34.472961 2564 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 23 19:19:34.482644 ip-10-0-129-2 kubenswrapper[2564]: W0723 19:19:34.472964 2564 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 23 19:19:34.482644 ip-10-0-129-2 kubenswrapper[2564]: W0723 19:19:34.472966 2564 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 23 19:19:34.482644 ip-10-0-129-2 kubenswrapper[2564]: W0723 19:19:34.472969 2564 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 23 19:19:34.482644 ip-10-0-129-2 kubenswrapper[2564]: W0723 19:19:34.472971 2564 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 23 19:19:34.483111 ip-10-0-129-2 kubenswrapper[2564]: W0723 19:19:34.472973 2564 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 23 19:19:34.483111 ip-10-0-129-2 kubenswrapper[2564]: W0723 19:19:34.472975 2564 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 23 19:19:34.483111 ip-10-0-129-2 kubenswrapper[2564]: W0723 19:19:34.472978 2564 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 23 19:19:34.483111 ip-10-0-129-2 kubenswrapper[2564]: W0723 19:19:34.472980 2564 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 23 19:19:34.483111 ip-10-0-129-2 kubenswrapper[2564]: W0723 19:19:34.472982 2564 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 23 19:19:34.483111 ip-10-0-129-2 kubenswrapper[2564]: W0723 19:19:34.472985 2564 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 23 19:19:34.483111 ip-10-0-129-2 kubenswrapper[2564]: W0723 19:19:34.472987 2564 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 23 19:19:34.483111 ip-10-0-129-2 kubenswrapper[2564]: W0723 19:19:34.472989 2564 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 23 19:19:34.483111 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:19:34.472996 2564 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]} Jul 23 19:19:34.483111 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:19:34.480780 2564 server.go:529] "Kubelet version" kubeletVersion="v1.34.9" Jul 23 19:19:34.483111 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:19:34.480797 2564 server.go:531] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK="" Jul 23 19:19:34.483111 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:19:34.480815 2564 watchdog_linux.go:95] "Systemd watchdog is not enabled" Jul 23 19:19:34.483111 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:19:34.480820 2564 watchdog_linux.go:137] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started." Jul 23 19:19:34.483111 ip-10-0-129-2 kubenswrapper[2564]: W0723 19:19:34.480880 2564 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 23 19:19:34.483472 ip-10-0-129-2 kubenswrapper[2564]: W0723 19:19:34.480886 2564 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 23 19:19:34.483472 ip-10-0-129-2 kubenswrapper[2564]: W0723 19:19:34.480889 2564 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 23 19:19:34.483472 ip-10-0-129-2 kubenswrapper[2564]: W0723 19:19:34.480891 2564 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 23 19:19:34.483472 ip-10-0-129-2 kubenswrapper[2564]: W0723 19:19:34.480894 2564 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 23 19:19:34.483472 ip-10-0-129-2 kubenswrapper[2564]: W0723 19:19:34.480897 2564 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 23 19:19:34.483472 ip-10-0-129-2 kubenswrapper[2564]: W0723 19:19:34.480900 2564 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 23 19:19:34.483472 ip-10-0-129-2 kubenswrapper[2564]: W0723 19:19:34.480904 2564 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 23 19:19:34.483472 ip-10-0-129-2 kubenswrapper[2564]: W0723 19:19:34.480908 2564 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 23 19:19:34.483472 ip-10-0-129-2 kubenswrapper[2564]: W0723 19:19:34.480910 2564 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 23 19:19:34.483472 ip-10-0-129-2 kubenswrapper[2564]: W0723 19:19:34.480913 2564 feature_gate.go:328] unrecognized feature gate: Example Jul 23 19:19:34.483472 ip-10-0-129-2 kubenswrapper[2564]: W0723 19:19:34.480916 2564 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 23 19:19:34.483472 ip-10-0-129-2 kubenswrapper[2564]: W0723 19:19:34.480918 2564 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 23 19:19:34.483472 ip-10-0-129-2 kubenswrapper[2564]: W0723 19:19:34.480921 2564 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 23 19:19:34.483472 ip-10-0-129-2 kubenswrapper[2564]: W0723 19:19:34.480923 2564 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 23 19:19:34.483472 ip-10-0-129-2 kubenswrapper[2564]: W0723 19:19:34.480927 2564 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 23 19:19:34.483472 ip-10-0-129-2 kubenswrapper[2564]: W0723 19:19:34.480930 2564 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 23 19:19:34.483472 ip-10-0-129-2 kubenswrapper[2564]: W0723 19:19:34.480932 2564 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 23 19:19:34.483472 ip-10-0-129-2 kubenswrapper[2564]: W0723 19:19:34.480935 2564 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 23 19:19:34.483472 ip-10-0-129-2 kubenswrapper[2564]: W0723 19:19:34.480937 2564 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 23 19:19:34.483921 ip-10-0-129-2 kubenswrapper[2564]: W0723 19:19:34.480940 2564 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 23 19:19:34.483921 ip-10-0-129-2 kubenswrapper[2564]: W0723 19:19:34.480942 2564 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 23 19:19:34.483921 ip-10-0-129-2 kubenswrapper[2564]: W0723 19:19:34.480945 2564 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 23 19:19:34.483921 ip-10-0-129-2 kubenswrapper[2564]: W0723 19:19:34.480948 2564 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 23 19:19:34.483921 ip-10-0-129-2 kubenswrapper[2564]: W0723 19:19:34.480950 2564 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 23 19:19:34.483921 ip-10-0-129-2 kubenswrapper[2564]: W0723 19:19:34.480953 2564 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 23 19:19:34.483921 ip-10-0-129-2 kubenswrapper[2564]: W0723 19:19:34.480955 2564 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 23 19:19:34.483921 ip-10-0-129-2 kubenswrapper[2564]: W0723 19:19:34.480957 2564 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 23 19:19:34.483921 ip-10-0-129-2 kubenswrapper[2564]: W0723 19:19:34.480960 2564 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 23 19:19:34.483921 ip-10-0-129-2 kubenswrapper[2564]: W0723 19:19:34.480963 2564 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 23 19:19:34.483921 ip-10-0-129-2 kubenswrapper[2564]: W0723 19:19:34.480966 2564 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 23 19:19:34.483921 ip-10-0-129-2 kubenswrapper[2564]: W0723 19:19:34.480968 2564 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 23 19:19:34.483921 ip-10-0-129-2 kubenswrapper[2564]: W0723 19:19:34.480970 2564 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 23 19:19:34.483921 ip-10-0-129-2 kubenswrapper[2564]: W0723 19:19:34.480973 2564 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 23 19:19:34.483921 ip-10-0-129-2 kubenswrapper[2564]: W0723 19:19:34.480975 2564 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 23 19:19:34.483921 ip-10-0-129-2 kubenswrapper[2564]: W0723 19:19:34.480977 2564 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 23 19:19:34.483921 ip-10-0-129-2 kubenswrapper[2564]: W0723 19:19:34.480980 2564 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 23 19:19:34.483921 ip-10-0-129-2 kubenswrapper[2564]: W0723 19:19:34.480982 2564 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 23 19:19:34.483921 ip-10-0-129-2 kubenswrapper[2564]: W0723 19:19:34.480984 2564 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 23 19:19:34.483921 ip-10-0-129-2 kubenswrapper[2564]: W0723 19:19:34.480987 2564 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 23 19:19:34.484419 ip-10-0-129-2 kubenswrapper[2564]: W0723 19:19:34.480989 2564 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 23 19:19:34.484419 ip-10-0-129-2 kubenswrapper[2564]: W0723 19:19:34.480991 2564 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 23 19:19:34.484419 ip-10-0-129-2 kubenswrapper[2564]: W0723 19:19:34.480993 2564 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 23 19:19:34.484419 ip-10-0-129-2 kubenswrapper[2564]: W0723 19:19:34.480996 2564 feature_gate.go:328] unrecognized feature gate: Example2 Jul 23 19:19:34.484419 ip-10-0-129-2 kubenswrapper[2564]: W0723 19:19:34.480998 2564 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 23 19:19:34.484419 ip-10-0-129-2 kubenswrapper[2564]: W0723 19:19:34.481000 2564 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 23 19:19:34.484419 ip-10-0-129-2 kubenswrapper[2564]: W0723 19:19:34.481003 2564 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 23 19:19:34.484419 ip-10-0-129-2 kubenswrapper[2564]: W0723 19:19:34.481005 2564 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 23 19:19:34.484419 ip-10-0-129-2 kubenswrapper[2564]: W0723 19:19:34.481007 2564 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 23 19:19:34.484419 ip-10-0-129-2 kubenswrapper[2564]: W0723 19:19:34.481010 2564 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 23 19:19:34.484419 ip-10-0-129-2 kubenswrapper[2564]: W0723 19:19:34.481012 2564 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 23 19:19:34.484419 ip-10-0-129-2 kubenswrapper[2564]: W0723 19:19:34.481015 2564 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 23 19:19:34.484419 ip-10-0-129-2 kubenswrapper[2564]: W0723 19:19:34.481017 2564 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 23 19:19:34.484419 ip-10-0-129-2 kubenswrapper[2564]: W0723 19:19:34.481020 2564 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 23 19:19:34.484419 ip-10-0-129-2 kubenswrapper[2564]: W0723 19:19:34.481022 2564 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 23 19:19:34.484419 ip-10-0-129-2 kubenswrapper[2564]: W0723 19:19:34.481025 2564 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 23 19:19:34.484419 ip-10-0-129-2 kubenswrapper[2564]: W0723 19:19:34.481027 2564 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 23 19:19:34.484419 ip-10-0-129-2 kubenswrapper[2564]: W0723 19:19:34.481030 2564 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 23 19:19:34.484419 ip-10-0-129-2 kubenswrapper[2564]: W0723 19:19:34.481032 2564 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 23 19:19:34.484419 ip-10-0-129-2 kubenswrapper[2564]: W0723 19:19:34.481035 2564 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 23 19:19:34.484894 ip-10-0-129-2 kubenswrapper[2564]: W0723 19:19:34.481037 2564 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 23 19:19:34.484894 ip-10-0-129-2 kubenswrapper[2564]: W0723 19:19:34.481040 2564 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 23 19:19:34.484894 ip-10-0-129-2 kubenswrapper[2564]: W0723 19:19:34.481042 2564 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 23 19:19:34.484894 ip-10-0-129-2 kubenswrapper[2564]: W0723 19:19:34.481045 2564 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 23 19:19:34.484894 ip-10-0-129-2 kubenswrapper[2564]: W0723 19:19:34.481047 2564 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 23 19:19:34.484894 ip-10-0-129-2 kubenswrapper[2564]: W0723 19:19:34.481050 2564 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 23 19:19:34.484894 ip-10-0-129-2 kubenswrapper[2564]: W0723 19:19:34.481052 2564 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 23 19:19:34.484894 ip-10-0-129-2 kubenswrapper[2564]: W0723 19:19:34.481055 2564 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 23 19:19:34.484894 ip-10-0-129-2 kubenswrapper[2564]: W0723 19:19:34.481057 2564 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 23 19:19:34.484894 ip-10-0-129-2 kubenswrapper[2564]: W0723 19:19:34.481059 2564 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 23 19:19:34.484894 ip-10-0-129-2 kubenswrapper[2564]: W0723 19:19:34.481062 2564 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 23 19:19:34.484894 ip-10-0-129-2 kubenswrapper[2564]: W0723 19:19:34.481064 2564 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 23 19:19:34.484894 ip-10-0-129-2 kubenswrapper[2564]: W0723 19:19:34.481066 2564 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 23 19:19:34.484894 ip-10-0-129-2 kubenswrapper[2564]: W0723 19:19:34.481069 2564 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 23 19:19:34.484894 ip-10-0-129-2 kubenswrapper[2564]: W0723 19:19:34.481072 2564 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 23 19:19:34.484894 ip-10-0-129-2 kubenswrapper[2564]: W0723 19:19:34.481076 2564 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 23 19:19:34.484894 ip-10-0-129-2 kubenswrapper[2564]: W0723 19:19:34.481078 2564 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 23 19:19:34.484894 ip-10-0-129-2 kubenswrapper[2564]: W0723 19:19:34.481081 2564 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 23 19:19:34.484894 ip-10-0-129-2 kubenswrapper[2564]: W0723 19:19:34.481083 2564 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 23 19:19:34.485413 ip-10-0-129-2 kubenswrapper[2564]: W0723 19:19:34.481086 2564 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 23 19:19:34.485413 ip-10-0-129-2 kubenswrapper[2564]: W0723 19:19:34.481089 2564 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 23 19:19:34.485413 ip-10-0-129-2 kubenswrapper[2564]: W0723 19:19:34.481091 2564 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 23 19:19:34.485413 ip-10-0-129-2 kubenswrapper[2564]: W0723 19:19:34.481093 2564 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 23 19:19:34.485413 ip-10-0-129-2 kubenswrapper[2564]: W0723 19:19:34.481097 2564 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 23 19:19:34.485413 ip-10-0-129-2 kubenswrapper[2564]: W0723 19:19:34.481099 2564 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 23 19:19:34.485413 ip-10-0-129-2 kubenswrapper[2564]: W0723 19:19:34.481101 2564 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 23 19:19:34.485413 ip-10-0-129-2 kubenswrapper[2564]: W0723 19:19:34.481104 2564 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 23 19:19:34.485413 ip-10-0-129-2 kubenswrapper[2564]: W0723 19:19:34.481106 2564 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 23 19:19:34.485413 ip-10-0-129-2 kubenswrapper[2564]: W0723 19:19:34.481108 2564 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 23 19:19:34.485413 ip-10-0-129-2 kubenswrapper[2564]: W0723 19:19:34.481111 2564 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 23 19:19:34.485413 ip-10-0-129-2 kubenswrapper[2564]: W0723 19:19:34.481113 2564 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 23 19:19:34.485413 ip-10-0-129-2 kubenswrapper[2564]: W0723 19:19:34.481115 2564 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 23 19:19:34.485413 ip-10-0-129-2 kubenswrapper[2564]: W0723 19:19:34.481118 2564 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 23 19:19:34.485413 ip-10-0-129-2 kubenswrapper[2564]: W0723 19:19:34.481121 2564 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 23 19:19:34.485413 ip-10-0-129-2 kubenswrapper[2564]: W0723 19:19:34.481123 2564 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 23 19:19:34.485413 ip-10-0-129-2 kubenswrapper[2564]: W0723 19:19:34.481126 2564 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 23 19:19:34.485413 ip-10-0-129-2 kubenswrapper[2564]: W0723 19:19:34.481129 2564 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 23 19:19:34.485413 ip-10-0-129-2 kubenswrapper[2564]: W0723 19:19:34.481132 2564 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 23 19:19:34.485413 ip-10-0-129-2 kubenswrapper[2564]: W0723 19:19:34.481134 2564 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 23 19:19:34.485905 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:19:34.481139 2564 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]} Jul 23 19:19:34.485905 ip-10-0-129-2 kubenswrapper[2564]: W0723 19:19:34.481253 2564 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 23 19:19:34.485905 ip-10-0-129-2 kubenswrapper[2564]: W0723 19:19:34.481257 2564 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 23 19:19:34.485905 ip-10-0-129-2 kubenswrapper[2564]: W0723 19:19:34.481260 2564 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 23 19:19:34.485905 ip-10-0-129-2 kubenswrapper[2564]: W0723 19:19:34.481263 2564 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 23 19:19:34.485905 ip-10-0-129-2 kubenswrapper[2564]: W0723 19:19:34.481266 2564 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 23 19:19:34.485905 ip-10-0-129-2 kubenswrapper[2564]: W0723 19:19:34.481268 2564 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 23 19:19:34.485905 ip-10-0-129-2 kubenswrapper[2564]: W0723 19:19:34.481271 2564 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 23 19:19:34.485905 ip-10-0-129-2 kubenswrapper[2564]: W0723 19:19:34.481273 2564 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 23 19:19:34.485905 ip-10-0-129-2 kubenswrapper[2564]: W0723 19:19:34.481275 2564 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 23 19:19:34.485905 ip-10-0-129-2 kubenswrapper[2564]: W0723 19:19:34.481278 2564 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 23 19:19:34.485905 ip-10-0-129-2 kubenswrapper[2564]: W0723 19:19:34.481280 2564 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 23 19:19:34.485905 ip-10-0-129-2 kubenswrapper[2564]: W0723 19:19:34.481282 2564 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 23 19:19:34.485905 ip-10-0-129-2 kubenswrapper[2564]: W0723 19:19:34.481286 2564 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 23 19:19:34.485905 ip-10-0-129-2 kubenswrapper[2564]: W0723 19:19:34.481288 2564 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 23 19:19:34.486256 ip-10-0-129-2 kubenswrapper[2564]: W0723 19:19:34.481290 2564 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 23 19:19:34.486256 ip-10-0-129-2 kubenswrapper[2564]: W0723 19:19:34.481292 2564 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 23 19:19:34.486256 ip-10-0-129-2 kubenswrapper[2564]: W0723 19:19:34.481295 2564 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 23 19:19:34.486256 ip-10-0-129-2 kubenswrapper[2564]: W0723 19:19:34.481298 2564 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 23 19:19:34.486256 ip-10-0-129-2 kubenswrapper[2564]: W0723 19:19:34.481301 2564 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 23 19:19:34.486256 ip-10-0-129-2 kubenswrapper[2564]: W0723 19:19:34.481303 2564 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 23 19:19:34.486256 ip-10-0-129-2 kubenswrapper[2564]: W0723 19:19:34.481305 2564 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 23 19:19:34.486256 ip-10-0-129-2 kubenswrapper[2564]: W0723 19:19:34.481308 2564 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 23 19:19:34.486256 ip-10-0-129-2 kubenswrapper[2564]: W0723 19:19:34.481311 2564 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 23 19:19:34.486256 ip-10-0-129-2 kubenswrapper[2564]: W0723 19:19:34.481314 2564 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 23 19:19:34.486256 ip-10-0-129-2 kubenswrapper[2564]: W0723 19:19:34.481317 2564 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 23 19:19:34.486256 ip-10-0-129-2 kubenswrapper[2564]: W0723 19:19:34.481339 2564 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 23 19:19:34.486256 ip-10-0-129-2 kubenswrapper[2564]: W0723 19:19:34.481342 2564 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 23 19:19:34.486256 ip-10-0-129-2 kubenswrapper[2564]: W0723 19:19:34.481345 2564 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 23 19:19:34.486256 ip-10-0-129-2 kubenswrapper[2564]: W0723 19:19:34.481347 2564 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 23 19:19:34.486256 ip-10-0-129-2 kubenswrapper[2564]: W0723 19:19:34.481350 2564 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 23 19:19:34.486256 ip-10-0-129-2 kubenswrapper[2564]: W0723 19:19:34.481352 2564 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 23 19:19:34.486256 ip-10-0-129-2 kubenswrapper[2564]: W0723 19:19:34.481355 2564 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 23 19:19:34.486256 ip-10-0-129-2 kubenswrapper[2564]: W0723 19:19:34.481357 2564 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 23 19:19:34.486712 ip-10-0-129-2 kubenswrapper[2564]: W0723 19:19:34.481360 2564 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 23 19:19:34.486712 ip-10-0-129-2 kubenswrapper[2564]: W0723 19:19:34.481362 2564 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 23 19:19:34.486712 ip-10-0-129-2 kubenswrapper[2564]: W0723 19:19:34.481364 2564 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 23 19:19:34.486712 ip-10-0-129-2 kubenswrapper[2564]: W0723 19:19:34.481366 2564 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 23 19:19:34.486712 ip-10-0-129-2 kubenswrapper[2564]: W0723 19:19:34.481369 2564 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 23 19:19:34.486712 ip-10-0-129-2 kubenswrapper[2564]: W0723 19:19:34.481371 2564 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 23 19:19:34.486712 ip-10-0-129-2 kubenswrapper[2564]: W0723 19:19:34.481373 2564 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 23 19:19:34.486712 ip-10-0-129-2 kubenswrapper[2564]: W0723 19:19:34.481375 2564 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 23 19:19:34.486712 ip-10-0-129-2 kubenswrapper[2564]: W0723 19:19:34.481378 2564 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 23 19:19:34.486712 ip-10-0-129-2 kubenswrapper[2564]: W0723 19:19:34.481380 2564 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 23 19:19:34.486712 ip-10-0-129-2 kubenswrapper[2564]: W0723 19:19:34.481382 2564 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 23 19:19:34.486712 ip-10-0-129-2 kubenswrapper[2564]: W0723 19:19:34.481384 2564 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 23 19:19:34.486712 ip-10-0-129-2 kubenswrapper[2564]: W0723 19:19:34.481388 2564 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 23 19:19:34.486712 ip-10-0-129-2 kubenswrapper[2564]: W0723 19:19:34.481390 2564 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 23 19:19:34.486712 ip-10-0-129-2 kubenswrapper[2564]: W0723 19:19:34.481393 2564 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 23 19:19:34.486712 ip-10-0-129-2 kubenswrapper[2564]: W0723 19:19:34.481396 2564 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 23 19:19:34.486712 ip-10-0-129-2 kubenswrapper[2564]: W0723 19:19:34.481398 2564 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 23 19:19:34.486712 ip-10-0-129-2 kubenswrapper[2564]: W0723 19:19:34.481401 2564 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 23 19:19:34.486712 ip-10-0-129-2 kubenswrapper[2564]: W0723 19:19:34.481403 2564 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 23 19:19:34.486712 ip-10-0-129-2 kubenswrapper[2564]: W0723 19:19:34.481405 2564 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 23 19:19:34.487174 ip-10-0-129-2 kubenswrapper[2564]: W0723 19:19:34.481407 2564 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 23 19:19:34.487174 ip-10-0-129-2 kubenswrapper[2564]: W0723 19:19:34.481410 2564 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 23 19:19:34.487174 ip-10-0-129-2 kubenswrapper[2564]: W0723 19:19:34.481412 2564 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 23 19:19:34.487174 ip-10-0-129-2 kubenswrapper[2564]: W0723 19:19:34.481414 2564 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 23 19:19:34.487174 ip-10-0-129-2 kubenswrapper[2564]: W0723 19:19:34.481416 2564 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 23 19:19:34.487174 ip-10-0-129-2 kubenswrapper[2564]: W0723 19:19:34.481419 2564 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 23 19:19:34.487174 ip-10-0-129-2 kubenswrapper[2564]: W0723 19:19:34.481421 2564 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 23 19:19:34.487174 ip-10-0-129-2 kubenswrapper[2564]: W0723 19:19:34.481424 2564 feature_gate.go:328] unrecognized feature gate: Example Jul 23 19:19:34.487174 ip-10-0-129-2 kubenswrapper[2564]: W0723 19:19:34.481426 2564 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 23 19:19:34.487174 ip-10-0-129-2 kubenswrapper[2564]: W0723 19:19:34.481428 2564 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 23 19:19:34.487174 ip-10-0-129-2 kubenswrapper[2564]: W0723 19:19:34.481431 2564 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 23 19:19:34.487174 ip-10-0-129-2 kubenswrapper[2564]: W0723 19:19:34.481433 2564 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 23 19:19:34.487174 ip-10-0-129-2 kubenswrapper[2564]: W0723 19:19:34.481435 2564 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 23 19:19:34.487174 ip-10-0-129-2 kubenswrapper[2564]: W0723 19:19:34.481437 2564 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 23 19:19:34.487174 ip-10-0-129-2 kubenswrapper[2564]: W0723 19:19:34.481441 2564 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 23 19:19:34.487174 ip-10-0-129-2 kubenswrapper[2564]: W0723 19:19:34.481444 2564 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 23 19:19:34.487174 ip-10-0-129-2 kubenswrapper[2564]: W0723 19:19:34.481447 2564 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 23 19:19:34.487174 ip-10-0-129-2 kubenswrapper[2564]: W0723 19:19:34.481450 2564 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 23 19:19:34.487174 ip-10-0-129-2 kubenswrapper[2564]: W0723 19:19:34.481452 2564 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 23 19:19:34.487174 ip-10-0-129-2 kubenswrapper[2564]: W0723 19:19:34.481455 2564 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 23 19:19:34.487656 ip-10-0-129-2 kubenswrapper[2564]: W0723 19:19:34.481457 2564 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 23 19:19:34.487656 ip-10-0-129-2 kubenswrapper[2564]: W0723 19:19:34.481460 2564 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 23 19:19:34.487656 ip-10-0-129-2 kubenswrapper[2564]: W0723 19:19:34.481463 2564 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 23 19:19:34.487656 ip-10-0-129-2 kubenswrapper[2564]: W0723 19:19:34.481465 2564 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 23 19:19:34.487656 ip-10-0-129-2 kubenswrapper[2564]: W0723 19:19:34.481467 2564 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 23 19:19:34.487656 ip-10-0-129-2 kubenswrapper[2564]: W0723 19:19:34.481470 2564 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 23 19:19:34.487656 ip-10-0-129-2 kubenswrapper[2564]: W0723 19:19:34.481473 2564 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 23 19:19:34.487656 ip-10-0-129-2 kubenswrapper[2564]: W0723 19:19:34.481476 2564 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 23 19:19:34.487656 ip-10-0-129-2 kubenswrapper[2564]: W0723 19:19:34.481479 2564 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 23 19:19:34.487656 ip-10-0-129-2 kubenswrapper[2564]: W0723 19:19:34.481481 2564 feature_gate.go:328] unrecognized feature gate: Example2 Jul 23 19:19:34.487656 ip-10-0-129-2 kubenswrapper[2564]: W0723 19:19:34.481484 2564 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 23 19:19:34.487656 ip-10-0-129-2 kubenswrapper[2564]: W0723 19:19:34.481486 2564 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 23 19:19:34.487656 ip-10-0-129-2 kubenswrapper[2564]: W0723 19:19:34.481488 2564 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 23 19:19:34.487656 ip-10-0-129-2 kubenswrapper[2564]: W0723 19:19:34.481491 2564 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 23 19:19:34.487656 ip-10-0-129-2 kubenswrapper[2564]: W0723 19:19:34.481493 2564 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 23 19:19:34.487656 ip-10-0-129-2 kubenswrapper[2564]: W0723 19:19:34.481495 2564 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 23 19:19:34.487656 ip-10-0-129-2 kubenswrapper[2564]: W0723 19:19:34.481497 2564 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 23 19:19:34.487656 ip-10-0-129-2 kubenswrapper[2564]: W0723 19:19:34.481499 2564 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 23 19:19:34.487656 ip-10-0-129-2 kubenswrapper[2564]: W0723 19:19:34.481502 2564 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 23 19:19:34.487656 ip-10-0-129-2 kubenswrapper[2564]: W0723 19:19:34.481504 2564 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 23 19:19:34.488117 ip-10-0-129-2 kubenswrapper[2564]: W0723 19:19:34.481506 2564 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 23 19:19:34.488117 ip-10-0-129-2 kubenswrapper[2564]: W0723 19:19:34.481508 2564 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 23 19:19:34.488117 ip-10-0-129-2 kubenswrapper[2564]: W0723 19:19:34.481511 2564 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 23 19:19:34.488117 ip-10-0-129-2 kubenswrapper[2564]: W0723 19:19:34.481513 2564 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 23 19:19:34.488117 ip-10-0-129-2 kubenswrapper[2564]: W0723 19:19:34.481515 2564 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 23 19:19:34.488117 ip-10-0-129-2 kubenswrapper[2564]: W0723 19:19:34.481518 2564 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 23 19:19:34.488117 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:19:34.481522 2564 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]} Jul 23 19:19:34.488117 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:19:34.481637 2564 server.go:962] "Client rotation is on, will bootstrap in background" Jul 23 19:19:34.488117 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:19:34.485719 2564 bootstrap.go:101] "Use the bootstrap credentials to request a cert, and set kubeconfig to point to the certificate dir" Jul 23 19:19:34.488117 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:19:34.487452 2564 server.go:1019] "Starting client certificate rotation" Jul 23 19:19:34.488117 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:19:34.487548 2564 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet" Jul 23 19:19:34.488117 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:19:34.487583 2564 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kube-apiserver-client-kubelet" Jul 23 19:19:34.490354 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:19:34.490340 2564 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Jul 23 19:19:34.491581 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:19:34.491556 2564 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Jul 23 19:19:34.503250 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:19:34.503235 2564 log.go:25] "Validated CRI v1 runtime API" Jul 23 19:19:34.505931 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:19:34.505917 2564 log.go:25] "Validated CRI v1 image API" Jul 23 19:19:34.506657 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:19:34.506645 2564 server.go:1429] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd" Jul 23 19:19:34.509699 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:19:34.509679 2564 fs.go:135] Filesystem UUIDs: map[7B77-95E7:/dev/nvme0n1p2 8717dc15-6183-44dd-b7d8-4af16296d466:/dev/nvme0n1p4 d0a468ca-5f7f-4bc9-ad27-b7ba8ecf066e:/dev/nvme0n1p3] Jul 23 19:19:34.509772 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:19:34.509698 2564 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:22 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:24 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:31 fsType:tmpfs blockSize:0} composefs_0-33:{mountpoint:/ major:0 minor:33 fsType:overlay blockSize:0}] Jul 23 19:19:34.515184 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:19:34.515084 2564 manager.go:216] Machine: {Timestamp:2026-07-23 19:19:34.513612552 +0000 UTC m=+0.365388790 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3209762 MemoryCapacity:32812179456 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec22eeb9627142f642b260b3eea138ff SystemUUID:ec22eeb9-6271-42f6-42b2-60b3eea138ff BootID:b4366ba0-0f74-472f-a743-286bfb1451fe Filesystems:[{Device:/tmp DeviceMajor:0 DeviceMinor:31 Capacity:16406089728 Type:vfs Inodes:1048576 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true} {Device:/dev/shm DeviceMajor:0 DeviceMinor:22 Capacity:16406089728 Type:vfs Inodes:4005393 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:24 Capacity:6562439168 Type:vfs Inodes:819200 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:composefs_0-33 DeviceMajor:0 DeviceMinor:33 Capacity:6111232 Type:vfs Inodes:18446744073709551615 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:br-ex MacAddress:02:51:84:f8:95:fb Speed:0 Mtu:9001} {Name:ens5 MacAddress:02:51:84:f8:95:fb Speed:0 Mtu:9001} {Name:ovs-system MacAddress:3a:bd:0a:08:e5:ff Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:32812179456 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:34603008 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None} Jul 23 19:19:34.515184 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:19:34.515173 2564 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available. Jul 23 19:19:34.515372 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:19:34.515274 2564 manager.go:232] Version: {KernelVersion:5.14.0-570.128.1.el9_6.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.6.20260715-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:} Jul 23 19:19:34.517836 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:19:34.517820 2564 reflector.go:439] "Caches populated" logger="kubernetes.io/kube-apiserver-client-kubelet" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" Jul 23 19:19:34.518729 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:19:34.518700 2564 container_manager_linux.go:273] "Container manager verified user specified cgroup-root exists" cgroupRoot=[] Jul 23 19:19:34.518882 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:19:34.518736 2564 container_manager_linux.go:278] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-129-2.ec2.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2} Jul 23 19:19:34.519594 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:19:34.519580 2564 topology_manager.go:138] "Creating topology manager with none policy" Jul 23 19:19:34.519657 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:19:34.519599 2564 container_manager_linux.go:309] "Creating device plugin manager" Jul 23 19:19:34.519657 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:19:34.519609 2564 manager.go:141] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock" Jul 23 19:19:34.519657 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:19:34.519623 2564 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock" Jul 23 19:19:34.521596 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:19:34.521584 2564 container_manager_linux.go:318] "Creating Dynamic Resource Allocation (DRA) manager" Jul 23 19:19:34.526406 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:19:34.526393 2564 state_mem.go:36] "Initialized new in-memory state store" Jul 23 19:19:34.526526 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:19:34.526517 2564 server.go:1240] "Using root directory" path="/var/lib/kubelet" Jul 23 19:19:34.526555 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:19:34.526545 2564 kubelet.go:486] "Attempting to sync node with API server" Jul 23 19:19:34.526591 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:19:34.526556 2564 kubelet.go:387] "Adding static pod path" path="/etc/kubernetes/manifests" Jul 23 19:19:34.526591 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:19:34.526578 2564 file.go:69] "Watching path" path="/etc/kubernetes/manifests" Jul 23 19:19:34.526591 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:19:34.526587 2564 kubelet.go:398] "Adding apiserver pod source" Jul 23 19:19:34.526664 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:19:34.526598 2564 apiserver.go:42] "Waiting for node sync before watching apiserver pods" Jul 23 19:19:34.526910 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:19:34.526894 2564 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Jul 23 19:19:34.526910 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:19:34.526912 2564 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Jul 23 19:19:34.529089 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:19:34.529071 2564 kuberuntime_manager.go:301] "Container runtime initialized" containerRuntime="cri-o" version="1.34.10-2.rhaos4.21.git83ee25b.el9" apiVersion="v1" Jul 23 19:19:34.531424 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:19:34.531411 2564 kubelet.go:960] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled" Jul 23 19:19:34.531482 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:19:34.531440 2564 kubelet.go:984] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled" Jul 23 19:19:34.532963 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:19:34.532950 2564 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume" Jul 23 19:19:34.533020 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:19:34.532972 2564 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir" Jul 23 19:19:34.533020 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:19:34.532980 2564 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo" Jul 23 19:19:34.533020 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:19:34.532986 2564 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path" Jul 23 19:19:34.533020 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:19:34.532996 2564 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs" Jul 23 19:19:34.533020 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:19:34.533000 2564 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret" Jul 23 19:19:34.533020 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:19:34.533007 2564 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi" Jul 23 19:19:34.533020 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:19:34.533012 2564 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api" Jul 23 19:19:34.533020 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:19:34.533018 2564 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc" Jul 23 19:19:34.533020 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:19:34.533022 2564 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap" Jul 23 19:19:34.533239 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:19:34.533035 2564 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected" Jul 23 19:19:34.533239 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:19:34.533050 2564 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume" Jul 23 19:19:34.533920 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:19:34.533909 2564 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi" Jul 23 19:19:34.533958 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:19:34.533921 2564 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image" Jul 23 19:19:34.540995 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:19:34.540981 2564 server.go:1268] "Started kubelet" Jul 23 19:19:34.541302 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:19:34.541276 2564 server.go:180] "Starting to listen" address="0.0.0.0" port=10250 Jul 23 19:19:34.541581 ip-10-0-129-2 systemd[1]: Started Kubernetes Kubelet. Jul 23 19:19:34.542214 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:19:34.542204 2564 server.go:310] "Adding debug handlers to kubelet server" Jul 23 19:19:34.543038 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:19:34.543017 2564 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10 Jul 23 19:19:34.543084 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:19:34.543052 2564 server_v1.go:49] "podresources" method="list" useActivePods=true Jul 23 19:19:34.544590 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:19:34.544574 2564 server.go:249] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock" Jul 23 19:19:34.544847 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:19:34.544821 2564 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-129-2.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope Jul 23 19:19:34.544939 ip-10-0-129-2 kubenswrapper[2564]: E0723 19:19:34.544852 2564 reflector.go:205] "Failed to watch" err="failed to list *v1.Service: services is forbidden: User \"system:anonymous\" cannot list resource \"services\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Service" Jul 23 19:19:34.545379 ip-10-0-129-2 kubenswrapper[2564]: E0723 19:19:34.545353 2564 reflector.go:205] "Failed to watch" err="failed to list *v1.Node: nodes \"ip-10-0-129-2.ec2.internal\" is forbidden: User \"system:anonymous\" cannot list resource \"nodes\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Node" Jul 23 19:19:34.550039 ip-10-0-129-2 kubenswrapper[2564]: E0723 19:19:34.550018 2564 kubelet.go:1638] "Image garbage collection failed once. Stats initialization may not have completed yet" err="failed to get imageFs info: unable to find data in memory cache" Jul 23 19:19:34.550707 ip-10-0-129-2 kubenswrapper[2564]: E0723 19:19:34.549564 2564 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-129-2.ec2.internal.18c501683d6bb05f default 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-129-2.ec2.internal,UID:ip-10-0-129-2.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:Starting,Message:Starting kubelet.,Source:EventSource{Component:kubelet,Host:ip-10-0-129-2.ec2.internal,},FirstTimestamp:2026-07-23 19:19:34.540963935 +0000 UTC m=+0.392740173,LastTimestamp:2026-07-23 19:19:34.540963935 +0000 UTC m=+0.392740173,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-129-2.ec2.internal,}" Jul 23 19:19:34.553128 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:19:34.553097 2564 fs_resource_analyzer.go:67] "Starting FS ResourceAnalyzer" Jul 23 19:19:34.553128 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:19:34.553111 2564 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving" Jul 23 19:19:34.553590 ip-10-0-129-2 kubenswrapper[2564]: E0723 19:19:34.553559 2564 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-129-2.ec2.internal\" not found" Jul 23 19:19:34.554206 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:19:34.554189 2564 desired_state_of_world_populator.go:146] "Desired state populator starts to run" Jul 23 19:19:34.554370 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:19:34.554354 2564 volume_manager.go:311] "The desired_state_of_world populator starts" Jul 23 19:19:34.554449 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:19:34.554425 2564 volume_manager.go:313] "Starting Kubelet Volume Manager" Jul 23 19:19:34.554699 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:19:34.554607 2564 reconstruct.go:97] "Volume reconstruction finished" Jul 23 19:19:34.554699 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:19:34.554618 2564 reconciler.go:29] "Reconciler: start to sync state" Jul 23 19:19:34.554936 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:19:34.554920 2564 factory.go:55] Registering systemd factory Jul 23 19:19:34.554992 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:19:34.554938 2564 factory.go:223] Registration of the systemd container factory successfully Jul 23 19:19:34.555143 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:19:34.555127 2564 factory.go:153] Registering CRI-O factory Jul 23 19:19:34.555198 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:19:34.555149 2564 factory.go:223] Registration of the crio container factory successfully Jul 23 19:19:34.555242 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:19:34.555201 2564 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory Jul 23 19:19:34.555242 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:19:34.555222 2564 factory.go:103] Registering Raw factory Jul 23 19:19:34.555242 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:19:34.555236 2564 manager.go:1196] Started watching for new ooms in manager Jul 23 19:19:34.555799 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:19:34.555779 2564 manager.go:349] Starting recovery of all containers Jul 23 19:19:34.567044 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:19:34.567025 2564 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-69dm8" Jul 23 19:19:34.567044 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:19:34.567043 2564 manager.go:354] Recovery completed Jul 23 19:19:34.567572 ip-10-0-129-2 kubenswrapper[2564]: E0723 19:19:34.567552 2564 reflector.go:205] "Failed to watch" err="failed to list *v1.CSIDriver: csidrivers.storage.k8s.io is forbidden: User \"system:anonymous\" cannot list resource \"csidrivers\" in API group \"storage.k8s.io\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.CSIDriver" Jul 23 19:19:34.567639 ip-10-0-129-2 kubenswrapper[2564]: E0723 19:19:34.567592 2564 controller.go:145] "Failed to ensure lease exists, will retry" err="leases.coordination.k8s.io \"ip-10-0-129-2.ec2.internal\" is forbidden: User \"system:anonymous\" cannot get resource \"leases\" in API group \"coordination.k8s.io\" in the namespace \"kube-node-lease\"" interval="200ms" Jul 23 19:19:34.570861 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:19:34.570849 2564 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 23 19:19:34.572633 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:19:34.572616 2564 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-129-2.ec2.internal" event="NodeHasSufficientMemory" Jul 23 19:19:34.572688 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:19:34.572651 2564 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-129-2.ec2.internal" event="NodeHasNoDiskPressure" Jul 23 19:19:34.572688 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:19:34.572668 2564 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-129-2.ec2.internal" event="NodeHasSufficientPID" Jul 23 19:19:34.573219 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:19:34.573204 2564 cpu_manager.go:222] "Starting CPU manager" policy="none" Jul 23 19:19:34.573219 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:19:34.573216 2564 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 23 19:19:34.573219 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:19:34.573220 2564 cpu_manager.go:223] "Reconciling" reconcilePeriod="10s" Jul 23 19:19:34.573365 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:19:34.573236 2564 state_mem.go:36] "Initialized new in-memory state store" Jul 23 19:19:34.573887 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:19:34.573843 2564 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-129-2.ec2.internal" event="NodeHasSufficientMemory" Jul 23 19:19:34.573887 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:19:34.573875 2564 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-129-2.ec2.internal" event="NodeHasNoDiskPressure" Jul 23 19:19:34.573887 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:19:34.573888 2564 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-129-2.ec2.internal" event="NodeHasSufficientPID" Jul 23 19:19:34.574657 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:19:34.574643 2564 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-69dm8" Jul 23 19:19:34.575949 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:19:34.575937 2564 policy_none.go:49] "None policy: Start" Jul 23 19:19:34.575985 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:19:34.575954 2564 memory_manager.go:187] "Starting memorymanager" policy="None" Jul 23 19:19:34.575985 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:19:34.575965 2564 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint" Jul 23 19:19:34.577980 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:19:34.577969 2564 policy_none.go:47] "Start" Jul 23 19:19:34.580667 ip-10-0-129-2 kubenswrapper[2564]: E0723 19:19:34.580529 2564 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-129-2.ec2.internal.18c501683f4ef219 default 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-129-2.ec2.internal,UID:ip-10-0-129-2.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeHasSufficientMemory,Message:Node ip-10-0-129-2.ec2.internal status is now: NodeHasSufficientMemory,Source:EventSource{Component:kubelet,Host:ip-10-0-129-2.ec2.internal,},FirstTimestamp:2026-07-23 19:19:34.572634649 +0000 UTC m=+0.424410898,LastTimestamp:2026-07-23 19:19:34.572634649 +0000 UTC m=+0.424410898,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-129-2.ec2.internal,}" Jul 23 19:19:34.629182 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:19:34.615822 2564 manager.go:337] "Starting Device Plugin manager" Jul 23 19:19:34.629182 ip-10-0-129-2 kubenswrapper[2564]: E0723 19:19:34.615845 2564 manager.go:513] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint" Jul 23 19:19:34.629182 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:19:34.615853 2564 server.go:87] "Starting device plugin registration server" Jul 23 19:19:34.629182 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:19:34.616032 2564 eviction_manager.go:189] "Eviction manager: starting control loop" Jul 23 19:19:34.629182 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:19:34.616043 2564 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s" Jul 23 19:19:34.629182 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:19:34.616155 2564 plugin_watcher.go:51] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry" Jul 23 19:19:34.629182 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:19:34.616240 2564 plugin_manager.go:116] "The desired_state_of_world populator (plugin watcher) starts" Jul 23 19:19:34.629182 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:19:34.616249 2564 plugin_manager.go:118] "Starting Kubelet Plugin Manager" Jul 23 19:19:34.629182 ip-10-0-129-2 kubenswrapper[2564]: E0723 19:19:34.616645 2564 eviction_manager.go:267] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\"" Jul 23 19:19:34.629182 ip-10-0-129-2 kubenswrapper[2564]: E0723 19:19:34.616684 2564 eviction_manager.go:292] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-129-2.ec2.internal\" not found" Jul 23 19:19:34.650184 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:19:34.650158 2564 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4" Jul 23 19:19:34.651439 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:19:34.651422 2564 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6" Jul 23 19:19:34.651518 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:19:34.651455 2564 status_manager.go:244] "Starting to sync pod status with apiserver" Jul 23 19:19:34.651518 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:19:34.651471 2564 kubelet.go:2448] "Starting kubelet main sync loop" Jul 23 19:19:34.651518 ip-10-0-129-2 kubenswrapper[2564]: E0723 19:19:34.651499 2564 kubelet.go:2472] "Skipping pod synchronization" err="PLEG is not healthy: pleg has yet to be successful" Jul 23 19:19:34.653960 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:19:34.653939 2564 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160" Jul 23 19:19:34.716421 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:19:34.716378 2564 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 23 19:19:34.717126 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:19:34.717107 2564 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-129-2.ec2.internal" event="NodeHasSufficientMemory" Jul 23 19:19:34.717207 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:19:34.717134 2564 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-129-2.ec2.internal" event="NodeHasNoDiskPressure" Jul 23 19:19:34.717207 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:19:34.717149 2564 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-129-2.ec2.internal" event="NodeHasSufficientPID" Jul 23 19:19:34.717207 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:19:34.717174 2564 kubelet_node_status.go:78] "Attempting to register node" node="ip-10-0-129-2.ec2.internal" Jul 23 19:19:34.726273 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:19:34.726253 2564 kubelet_node_status.go:81] "Successfully registered node" node="ip-10-0-129-2.ec2.internal" Jul 23 19:19:34.726273 ip-10-0-129-2 kubenswrapper[2564]: E0723 19:19:34.726273 2564 kubelet_node_status.go:536] "Error updating node status, will retry" err="error getting node \"ip-10-0-129-2.ec2.internal\": node \"ip-10-0-129-2.ec2.internal\" not found" Jul 23 19:19:34.749880 ip-10-0-129-2 kubenswrapper[2564]: E0723 19:19:34.749861 2564 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-129-2.ec2.internal\" not found" Jul 23 19:19:34.752582 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:19:34.752567 2564 kubelet.go:2535] "SyncLoop ADD" source="file" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-2.ec2.internal","kube-system/kube-apiserver-proxy-ip-10-0-129-2.ec2.internal"] Jul 23 19:19:34.752625 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:19:34.752617 2564 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 23 19:19:34.753311 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:19:34.753298 2564 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-129-2.ec2.internal" event="NodeHasSufficientMemory" Jul 23 19:19:34.753405 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:19:34.753345 2564 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-129-2.ec2.internal" event="NodeHasNoDiskPressure" Jul 23 19:19:34.753405 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:19:34.753360 2564 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-129-2.ec2.internal" event="NodeHasSufficientPID" Jul 23 19:19:34.755464 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:19:34.755448 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/ee6c09f813ef3a4bd239061ba78fb650-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-129-2.ec2.internal\" (UID: \"ee6c09f813ef3a4bd239061ba78fb650\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-2.ec2.internal" Jul 23 19:19:34.755535 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:19:34.755470 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/ee6c09f813ef3a4bd239061ba78fb650-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-129-2.ec2.internal\" (UID: \"ee6c09f813ef3a4bd239061ba78fb650\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-2.ec2.internal" Jul 23 19:19:34.755711 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:19:34.755697 2564 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 23 19:19:34.755823 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:19:34.755809 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-2.ec2.internal" Jul 23 19:19:34.755864 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:19:34.755837 2564 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 23 19:19:34.756363 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:19:34.756345 2564 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-129-2.ec2.internal" event="NodeHasSufficientMemory" Jul 23 19:19:34.756458 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:19:34.756369 2564 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-129-2.ec2.internal" event="NodeHasNoDiskPressure" Jul 23 19:19:34.756458 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:19:34.756378 2564 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-129-2.ec2.internal" event="NodeHasSufficientPID" Jul 23 19:19:34.756458 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:19:34.756435 2564 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-129-2.ec2.internal" event="NodeHasSufficientMemory" Jul 23 19:19:34.756458 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:19:34.756458 2564 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-129-2.ec2.internal" event="NodeHasNoDiskPressure" Jul 23 19:19:34.756642 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:19:34.756471 2564 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-129-2.ec2.internal" event="NodeHasSufficientPID" Jul 23 19:19:34.758770 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:19:34.758754 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-129-2.ec2.internal" Jul 23 19:19:34.758850 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:19:34.758785 2564 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 23 19:19:34.759438 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:19:34.759426 2564 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-129-2.ec2.internal" event="NodeHasSufficientMemory" Jul 23 19:19:34.759510 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:19:34.759451 2564 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-129-2.ec2.internal" event="NodeHasNoDiskPressure" Jul 23 19:19:34.759510 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:19:34.759464 2564 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-129-2.ec2.internal" event="NodeHasSufficientPID" Jul 23 19:19:34.784894 ip-10-0-129-2 kubenswrapper[2564]: E0723 19:19:34.784871 2564 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-129-2.ec2.internal\" not found" node="ip-10-0-129-2.ec2.internal" Jul 23 19:19:34.789155 ip-10-0-129-2 kubenswrapper[2564]: E0723 19:19:34.789141 2564 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-129-2.ec2.internal\" not found" node="ip-10-0-129-2.ec2.internal" Jul 23 19:19:34.850182 ip-10-0-129-2 kubenswrapper[2564]: E0723 19:19:34.850166 2564 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-129-2.ec2.internal\" not found" Jul 23 19:19:34.856484 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:19:34.856466 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/f9c41c89c718c39ff54e57e416ba8933-config\") pod \"kube-apiserver-proxy-ip-10-0-129-2.ec2.internal\" (UID: \"f9c41c89c718c39ff54e57e416ba8933\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-129-2.ec2.internal" Jul 23 19:19:34.856573 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:19:34.856497 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/ee6c09f813ef3a4bd239061ba78fb650-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-129-2.ec2.internal\" (UID: \"ee6c09f813ef3a4bd239061ba78fb650\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-2.ec2.internal" Jul 23 19:19:34.856573 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:19:34.856513 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/ee6c09f813ef3a4bd239061ba78fb650-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-129-2.ec2.internal\" (UID: \"ee6c09f813ef3a4bd239061ba78fb650\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-2.ec2.internal" Jul 23 19:19:34.856573 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:19:34.856543 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/ee6c09f813ef3a4bd239061ba78fb650-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-129-2.ec2.internal\" (UID: \"ee6c09f813ef3a4bd239061ba78fb650\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-2.ec2.internal" Jul 23 19:19:34.856677 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:19:34.856580 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/ee6c09f813ef3a4bd239061ba78fb650-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-129-2.ec2.internal\" (UID: \"ee6c09f813ef3a4bd239061ba78fb650\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-2.ec2.internal" Jul 23 19:19:34.950637 ip-10-0-129-2 kubenswrapper[2564]: E0723 19:19:34.950612 2564 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-129-2.ec2.internal\" not found" Jul 23 19:19:34.956921 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:19:34.956907 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/f9c41c89c718c39ff54e57e416ba8933-config\") pod \"kube-apiserver-proxy-ip-10-0-129-2.ec2.internal\" (UID: \"f9c41c89c718c39ff54e57e416ba8933\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-129-2.ec2.internal" Jul 23 19:19:34.956965 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:19:34.956941 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/f9c41c89c718c39ff54e57e416ba8933-config\") pod \"kube-apiserver-proxy-ip-10-0-129-2.ec2.internal\" (UID: \"f9c41c89c718c39ff54e57e416ba8933\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-129-2.ec2.internal" Jul 23 19:19:35.051281 ip-10-0-129-2 kubenswrapper[2564]: E0723 19:19:35.051242 2564 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-129-2.ec2.internal\" not found" Jul 23 19:19:35.087672 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:19:35.087656 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-2.ec2.internal" Jul 23 19:19:35.092374 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:19:35.092360 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-129-2.ec2.internal" Jul 23 19:19:35.152285 ip-10-0-129-2 kubenswrapper[2564]: E0723 19:19:35.152263 2564 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-129-2.ec2.internal\" not found" Jul 23 19:19:35.252759 ip-10-0-129-2 kubenswrapper[2564]: E0723 19:19:35.252742 2564 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-129-2.ec2.internal\" not found" Jul 23 19:19:35.353298 ip-10-0-129-2 kubenswrapper[2564]: E0723 19:19:35.353262 2564 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-129-2.ec2.internal\" not found" Jul 23 19:19:35.453692 ip-10-0-129-2 kubenswrapper[2564]: E0723 19:19:35.453671 2564 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-129-2.ec2.internal\" not found" Jul 23 19:19:35.487045 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:19:35.487026 2564 transport.go:147] "Certificate rotation detected, shutting down client connections to start using new credentials" Jul 23 19:19:35.487379 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:19:35.487119 2564 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.RuntimeClass" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received" Jul 23 19:19:35.554358 ip-10-0-129-2 kubenswrapper[2564]: E0723 19:19:35.554337 2564 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-129-2.ec2.internal\" not found" Jul 23 19:19:35.580768 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:19:35.580740 2564 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-07-22 19:14:35 +0000 UTC" deadline="2028-03-12 12:39:01.117873752 +0000 UTC" Jul 23 19:19:35.580768 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:19:35.580767 2564 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="14345h19m25.537110742s" Jul 23 19:19:35.654428 ip-10-0-129-2 kubenswrapper[2564]: E0723 19:19:35.654410 2564 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-129-2.ec2.internal\" not found" Jul 23 19:19:35.654847 ip-10-0-129-2 kubenswrapper[2564]: W0723 19:19:35.654822 2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podee6c09f813ef3a4bd239061ba78fb650.slice/crio-6125ec56e00d5f11e16417f2f00a4cc216a74fc3cd13f8eefd35bde61a75ac64 WatchSource:0}: Error finding container 6125ec56e00d5f11e16417f2f00a4cc216a74fc3cd13f8eefd35bde61a75ac64: Status 404 returned error can't find the container with id 6125ec56e00d5f11e16417f2f00a4cc216a74fc3cd13f8eefd35bde61a75ac64 Jul 23 19:19:35.655258 ip-10-0-129-2 kubenswrapper[2564]: W0723 19:19:35.655238 2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf9c41c89c718c39ff54e57e416ba8933.slice/crio-dd6de12b7c4f3985d128c9f04f342e12c27b7bf30f4d3fc8ebd6787caca601c1 WatchSource:0}: Error finding container dd6de12b7c4f3985d128c9f04f342e12c27b7bf30f4d3fc8ebd6787caca601c1: Status 404 returned error can't find the container with id dd6de12b7c4f3985d128c9f04f342e12c27b7bf30f4d3fc8ebd6787caca601c1 Jul 23 19:19:35.659571 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:19:35.659557 2564 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Jul 23 19:19:35.717445 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:19:35.717431 2564 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 23 19:19:35.718283 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:19:35.718264 2564 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-129-2.ec2.internal" event="NodeHasSufficientMemory" Jul 23 19:19:35.718377 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:19:35.718291 2564 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-129-2.ec2.internal" event="NodeHasNoDiskPressure" Jul 23 19:19:35.718377 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:19:35.718306 2564 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-129-2.ec2.internal" event="NodeHasSufficientPID" Jul 23 19:19:35.754590 ip-10-0-129-2 kubenswrapper[2564]: E0723 19:19:35.754570 2564 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-129-2.ec2.internal\" not found" Jul 23 19:19:35.854972 ip-10-0-129-2 kubenswrapper[2564]: E0723 19:19:35.854950 2564 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-129-2.ec2.internal\" not found" Jul 23 19:19:35.901419 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:19:35.901380 2564 reflector.go:439] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:160" Jul 23 19:19:35.920176 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:19:35.920157 2564 reflector.go:439] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:160" Jul 23 19:19:35.955373 ip-10-0-129-2 kubenswrapper[2564]: E0723 19:19:35.955354 2564 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-129-2.ec2.internal\" not found" Jul 23 19:19:36.056095 ip-10-0-129-2 kubenswrapper[2564]: E0723 19:19:36.056074 2564 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-129-2.ec2.internal\" not found" Jul 23 19:19:36.118617 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:19:36.118594 2564 reflector.go:439] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:160" Jul 23 19:19:36.154592 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:19:36.154549 2564 kubelet.go:3248] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-2.ec2.internal" Jul 23 19:19:36.167520 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:19:36.167503 2564 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Jul 23 19:19:36.168507 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:19:36.168489 2564 kubelet.go:3248] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-129-2.ec2.internal" Jul 23 19:19:36.174746 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:19:36.174731 2564 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Jul 23 19:19:36.423656 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:19:36.423635 2564 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160" Jul 23 19:19:36.528574 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:19:36.528362 2564 apiserver.go:52] "Watching apiserver" Jul 23 19:19:36.537189 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:19:36.537030 2564 reflector.go:439] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66" Jul 23 19:19:36.538330 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:19:36.538306 2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-network-operator/iptables-alerter-gbqdl","openshift-ovn-kubernetes/ovnkube-node-8dtlh","kube-system/kube-apiserver-proxy-ip-10-0-129-2.ec2.internal","openshift-cluster-node-tuning-operator/tuned-74q7d","openshift-dns/node-resolver-6k8wr","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-2.ec2.internal","openshift-multus/multus-8967b","openshift-network-diagnostics/network-check-target-vxngp","kube-system/global-pull-secret-syncer-rtgbq","kube-system/konnectivity-agent-8rq79","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-l9rz7","openshift-image-registry/node-ca-9wwn6","openshift-multus/multus-additional-cni-plugins-sr96w","openshift-multus/network-metrics-daemon-qhrz5"] Jul 23 19:19:36.541041 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:19:36.541020 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-8967b" Jul 23 19:19:36.544052 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:19:36.544030 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-sr96w" Jul 23 19:19:36.545918 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:19:36.544738 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\"" Jul 23 19:19:36.545918 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:19:36.545007 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\"" Jul 23 19:19:36.545918 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:19:36.545263 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"default-dockercfg-mvkpp\"" Jul 23 19:19:36.545918 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:19:36.545537 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\"" Jul 23 19:19:36.545918 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:19:36.545756 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\"" Jul 23 19:19:36.546513 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:19:36.546491 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\"" Jul 23 19:19:36.546754 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:19:36.546741 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-pchw8\"" Jul 23 19:19:36.546793 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:19:36.546743 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\"" Jul 23 19:19:36.547008 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:19:36.546993 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-qhrz5" Jul 23 19:19:36.547089 ip-10-0-129-2 kubenswrapper[2564]: E0723 19:19:36.547067 2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-qhrz5" podUID="576f76ed-730c-49c2-a33e-1dc11370cad4" Jul 23 19:19:36.551131 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:19:36.549882 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-74q7d" Jul 23 19:19:36.552283 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:19:36.552257 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\"" Jul 23 19:19:36.552585 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:19:36.552564 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\"" Jul 23 19:19:36.552670 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:19:36.552597 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-vtkgp\"" Jul 23 19:19:36.553473 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:19:36.553338 2564 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kubelet-serving" Jul 23 19:19:36.554922 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:19:36.554549 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-6k8wr" Jul 23 19:19:36.556846 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:19:36.556827 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\"" Jul 23 19:19:36.557121 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:19:36.557105 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\"" Jul 23 19:19:36.557379 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:19:36.557360 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-mf6rp\"" Jul 23 19:19:36.557615 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:19:36.557599 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-vxngp" Jul 23 19:19:36.557678 ip-10-0-129-2 kubenswrapper[2564]: E0723 19:19:36.557662 2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-vxngp" podUID="8706aab3-bba6-4fff-b8bc-8890d14daa9c" Jul 23 19:19:36.560207 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:19:36.559627 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-rtgbq" Jul 23 19:19:36.560207 ip-10-0-129-2 kubenswrapper[2564]: E0723 19:19:36.559694 2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-rtgbq" podUID="40f386ec-607d-496f-b65b-03a1fd847b1f" Jul 23 19:19:36.562049 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:19:36.562017 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-gbqdl" Jul 23 19:19:36.565079 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:19:36.565058 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/7c4060f3-0a15-4661-803f-fa6a845fc1ac-cni-binary-copy\") pod \"multus-additional-cni-plugins-sr96w\" (UID: \"7c4060f3-0a15-4661-803f-fa6a845fc1ac\") " pod="openshift-multus/multus-additional-cni-plugins-sr96w" Jul 23 19:19:36.565170 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:19:36.565096 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jf5zp\" (UniqueName: \"kubernetes.io/projected/7c4060f3-0a15-4661-803f-fa6a845fc1ac-kube-api-access-jf5zp\") pod \"multus-additional-cni-plugins-sr96w\" (UID: \"7c4060f3-0a15-4661-803f-fa6a845fc1ac\") " pod="openshift-multus/multus-additional-cni-plugins-sr96w" Jul 23 19:19:36.565170 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:19:36.565122 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/3557bbdf-7f88-487d-b674-28db6ea9d156-host\") pod \"tuned-74q7d\" (UID: \"3557bbdf-7f88-487d-b674-28db6ea9d156\") " pod="openshift-cluster-node-tuning-operator/tuned-74q7d" Jul 23 19:19:36.565170 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:19:36.565146 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/23313c02-67bf-498f-a294-e16cdc7d4517-tmp-dir\") pod \"node-resolver-6k8wr\" (UID: \"23313c02-67bf-498f-a294-e16cdc7d4517\") " pod="openshift-dns/node-resolver-6k8wr" Jul 23 19:19:36.565324 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:19:36.565171 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/67269f87-53d8-4f3a-b396-4891d9326c38-system-cni-dir\") pod \"multus-8967b\" (UID: \"67269f87-53d8-4f3a-b396-4891d9326c38\") " pod="openshift-multus/multus-8967b" Jul 23 19:19:36.565324 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:19:36.565195 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/67269f87-53d8-4f3a-b396-4891d9326c38-host-run-multus-certs\") pod \"multus-8967b\" (UID: \"67269f87-53d8-4f3a-b396-4891d9326c38\") " pod="openshift-multus/multus-8967b" Jul 23 19:19:36.565324 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:19:36.565220 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/7c4060f3-0a15-4661-803f-fa6a845fc1ac-system-cni-dir\") pod \"multus-additional-cni-plugins-sr96w\" (UID: \"7c4060f3-0a15-4661-803f-fa6a845fc1ac\") " pod="openshift-multus/multus-additional-cni-plugins-sr96w" Jul 23 19:19:36.565324 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:19:36.565244 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/7c4060f3-0a15-4661-803f-fa6a845fc1ac-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-sr96w\" (UID: \"7c4060f3-0a15-4661-803f-fa6a845fc1ac\") " pod="openshift-multus/multus-additional-cni-plugins-sr96w" Jul 23 19:19:36.565324 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:19:36.565271 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/3557bbdf-7f88-487d-b674-28db6ea9d156-etc-sysctl-d\") pod \"tuned-74q7d\" (UID: \"3557bbdf-7f88-487d-b674-28db6ea9d156\") " pod="openshift-cluster-node-tuning-operator/tuned-74q7d" Jul 23 19:19:36.565324 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:19:36.565278 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\"" Jul 23 19:19:36.565324 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:19:36.565296 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/3557bbdf-7f88-487d-b674-28db6ea9d156-var-lib-kubelet\") pod \"tuned-74q7d\" (UID: \"3557bbdf-7f88-487d-b674-28db6ea9d156\") " pod="openshift-cluster-node-tuning-operator/tuned-74q7d" Jul 23 19:19:36.565732 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:19:36.565359 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/67269f87-53d8-4f3a-b396-4891d9326c38-multus-cni-dir\") pod \"multus-8967b\" (UID: \"67269f87-53d8-4f3a-b396-4891d9326c38\") " pod="openshift-multus/multus-8967b" Jul 23 19:19:36.565732 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:19:36.565384 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/67269f87-53d8-4f3a-b396-4891d9326c38-os-release\") pod \"multus-8967b\" (UID: \"67269f87-53d8-4f3a-b396-4891d9326c38\") " pod="openshift-multus/multus-8967b" Jul 23 19:19:36.565732 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:19:36.565359 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\"" Jul 23 19:19:36.565732 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:19:36.565430 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\"" Jul 23 19:19:36.565732 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:19:36.565429 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/67269f87-53d8-4f3a-b396-4891d9326c38-host-run-k8s-cni-cncf-io\") pod \"multus-8967b\" (UID: \"67269f87-53d8-4f3a-b396-4891d9326c38\") " pod="openshift-multus/multus-8967b" Jul 23 19:19:36.565732 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:19:36.565489 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/67269f87-53d8-4f3a-b396-4891d9326c38-host-var-lib-kubelet\") pod \"multus-8967b\" (UID: \"67269f87-53d8-4f3a-b396-4891d9326c38\") " pod="openshift-multus/multus-8967b" Jul 23 19:19:36.565732 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:19:36.565510 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-xhsrl\"" Jul 23 19:19:36.565732 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:19:36.565515 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/67269f87-53d8-4f3a-b396-4891d9326c38-multus-daemon-config\") pod \"multus-8967b\" (UID: \"67269f87-53d8-4f3a-b396-4891d9326c38\") " pod="openshift-multus/multus-8967b" Jul 23 19:19:36.565732 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:19:36.565539 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/67269f87-53d8-4f3a-b396-4891d9326c38-etc-kubernetes\") pod \"multus-8967b\" (UID: \"67269f87-53d8-4f3a-b396-4891d9326c38\") " pod="openshift-multus/multus-8967b" Jul 23 19:19:36.565732 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:19:36.565566 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/7c4060f3-0a15-4661-803f-fa6a845fc1ac-cnibin\") pod \"multus-additional-cni-plugins-sr96w\" (UID: \"7c4060f3-0a15-4661-803f-fa6a845fc1ac\") " pod="openshift-multus/multus-additional-cni-plugins-sr96w" Jul 23 19:19:36.565732 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:19:36.565590 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-54vhn\" (UniqueName: \"kubernetes.io/projected/576f76ed-730c-49c2-a33e-1dc11370cad4-kube-api-access-54vhn\") pod \"network-metrics-daemon-qhrz5\" (UID: \"576f76ed-730c-49c2-a33e-1dc11370cad4\") " pod="openshift-multus/network-metrics-daemon-qhrz5" Jul 23 19:19:36.565732 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:19:36.565614 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/3557bbdf-7f88-487d-b674-28db6ea9d156-etc-systemd\") pod \"tuned-74q7d\" (UID: \"3557bbdf-7f88-487d-b674-28db6ea9d156\") " pod="openshift-cluster-node-tuning-operator/tuned-74q7d" Jul 23 19:19:36.565732 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:19:36.565653 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/3557bbdf-7f88-487d-b674-28db6ea9d156-sys\") pod \"tuned-74q7d\" (UID: \"3557bbdf-7f88-487d-b674-28db6ea9d156\") " pod="openshift-cluster-node-tuning-operator/tuned-74q7d" Jul 23 19:19:36.565732 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:19:36.565685 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/3557bbdf-7f88-487d-b674-28db6ea9d156-tmp\") pod \"tuned-74q7d\" (UID: \"3557bbdf-7f88-487d-b674-28db6ea9d156\") " pod="openshift-cluster-node-tuning-operator/tuned-74q7d" Jul 23 19:19:36.565732 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:19:36.565715 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/67269f87-53d8-4f3a-b396-4891d9326c38-cnibin\") pod \"multus-8967b\" (UID: \"67269f87-53d8-4f3a-b396-4891d9326c38\") " pod="openshift-multus/multus-8967b" Jul 23 19:19:36.566491 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:19:36.565744 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rgpqz\" (UniqueName: \"kubernetes.io/projected/67269f87-53d8-4f3a-b396-4891d9326c38-kube-api-access-rgpqz\") pod \"multus-8967b\" (UID: \"67269f87-53d8-4f3a-b396-4891d9326c38\") " pod="openshift-multus/multus-8967b" Jul 23 19:19:36.566491 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:19:36.565798 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/576f76ed-730c-49c2-a33e-1dc11370cad4-metrics-certs\") pod \"network-metrics-daemon-qhrz5\" (UID: \"576f76ed-730c-49c2-a33e-1dc11370cad4\") " pod="openshift-multus/network-metrics-daemon-qhrz5" Jul 23 19:19:36.566491 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:19:36.565833 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/3557bbdf-7f88-487d-b674-28db6ea9d156-run\") pod \"tuned-74q7d\" (UID: \"3557bbdf-7f88-487d-b674-28db6ea9d156\") " pod="openshift-cluster-node-tuning-operator/tuned-74q7d" Jul 23 19:19:36.566491 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:19:36.565859 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hx49w\" (UniqueName: \"kubernetes.io/projected/23313c02-67bf-498f-a294-e16cdc7d4517-kube-api-access-hx49w\") pod \"node-resolver-6k8wr\" (UID: \"23313c02-67bf-498f-a294-e16cdc7d4517\") " pod="openshift-dns/node-resolver-6k8wr" Jul 23 19:19:36.566491 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:19:36.565884 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/67269f87-53d8-4f3a-b396-4891d9326c38-cni-binary-copy\") pod \"multus-8967b\" (UID: \"67269f87-53d8-4f3a-b396-4891d9326c38\") " pod="openshift-multus/multus-8967b" Jul 23 19:19:36.566491 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:19:36.565911 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/7c4060f3-0a15-4661-803f-fa6a845fc1ac-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-sr96w\" (UID: \"7c4060f3-0a15-4661-803f-fa6a845fc1ac\") " pod="openshift-multus/multus-additional-cni-plugins-sr96w" Jul 23 19:19:36.566491 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:19:36.565936 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/3557bbdf-7f88-487d-b674-28db6ea9d156-etc-modprobe-d\") pod \"tuned-74q7d\" (UID: \"3557bbdf-7f88-487d-b674-28db6ea9d156\") " pod="openshift-cluster-node-tuning-operator/tuned-74q7d" Jul 23 19:19:36.566491 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:19:36.565959 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/23313c02-67bf-498f-a294-e16cdc7d4517-hosts-file\") pod \"node-resolver-6k8wr\" (UID: \"23313c02-67bf-498f-a294-e16cdc7d4517\") " pod="openshift-dns/node-resolver-6k8wr" Jul 23 19:19:36.566491 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:19:36.565983 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cq8ll\" (UniqueName: \"kubernetes.io/projected/8706aab3-bba6-4fff-b8bc-8890d14daa9c-kube-api-access-cq8ll\") pod \"network-check-target-vxngp\" (UID: \"8706aab3-bba6-4fff-b8bc-8890d14daa9c\") " pod="openshift-network-diagnostics/network-check-target-vxngp" Jul 23 19:19:36.566491 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:19:36.566008 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/67269f87-53d8-4f3a-b396-4891d9326c38-host-var-lib-cni-bin\") pod \"multus-8967b\" (UID: \"67269f87-53d8-4f3a-b396-4891d9326c38\") " pod="openshift-multus/multus-8967b" Jul 23 19:19:36.566491 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:19:36.566032 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/7c4060f3-0a15-4661-803f-fa6a845fc1ac-os-release\") pod \"multus-additional-cni-plugins-sr96w\" (UID: \"7c4060f3-0a15-4661-803f-fa6a845fc1ac\") " pod="openshift-multus/multus-additional-cni-plugins-sr96w" Jul 23 19:19:36.566491 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:19:36.566057 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/3557bbdf-7f88-487d-b674-28db6ea9d156-etc-sysconfig\") pod \"tuned-74q7d\" (UID: \"3557bbdf-7f88-487d-b674-28db6ea9d156\") " pod="openshift-cluster-node-tuning-operator/tuned-74q7d" Jul 23 19:19:36.566491 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:19:36.566155 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/3557bbdf-7f88-487d-b674-28db6ea9d156-etc-sysctl-conf\") pod \"tuned-74q7d\" (UID: \"3557bbdf-7f88-487d-b674-28db6ea9d156\") " pod="openshift-cluster-node-tuning-operator/tuned-74q7d" Jul 23 19:19:36.566491 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:19:36.566159 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-8dtlh" Jul 23 19:19:36.566491 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:19:36.566196 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/3557bbdf-7f88-487d-b674-28db6ea9d156-lib-modules\") pod \"tuned-74q7d\" (UID: \"3557bbdf-7f88-487d-b674-28db6ea9d156\") " pod="openshift-cluster-node-tuning-operator/tuned-74q7d" Jul 23 19:19:36.566491 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:19:36.566244 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/3557bbdf-7f88-487d-b674-28db6ea9d156-etc-tuned\") pod \"tuned-74q7d\" (UID: \"3557bbdf-7f88-487d-b674-28db6ea9d156\") " pod="openshift-cluster-node-tuning-operator/tuned-74q7d" Jul 23 19:19:36.566491 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:19:36.566271 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/67269f87-53d8-4f3a-b396-4891d9326c38-hostroot\") pod \"multus-8967b\" (UID: \"67269f87-53d8-4f3a-b396-4891d9326c38\") " pod="openshift-multus/multus-8967b" Jul 23 19:19:36.567115 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:19:36.566299 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/67269f87-53d8-4f3a-b396-4891d9326c38-multus-conf-dir\") pod \"multus-8967b\" (UID: \"67269f87-53d8-4f3a-b396-4891d9326c38\") " pod="openshift-multus/multus-8967b" Jul 23 19:19:36.567115 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:19:36.566325 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/7c4060f3-0a15-4661-803f-fa6a845fc1ac-tuning-conf-dir\") pod \"multus-additional-cni-plugins-sr96w\" (UID: \"7c4060f3-0a15-4661-803f-fa6a845fc1ac\") " pod="openshift-multus/multus-additional-cni-plugins-sr96w" Jul 23 19:19:36.567115 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:19:36.566370 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/3557bbdf-7f88-487d-b674-28db6ea9d156-etc-kubernetes\") pod \"tuned-74q7d\" (UID: \"3557bbdf-7f88-487d-b674-28db6ea9d156\") " pod="openshift-cluster-node-tuning-operator/tuned-74q7d" Jul 23 19:19:36.567115 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:19:36.566394 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wlncm\" (UniqueName: \"kubernetes.io/projected/3557bbdf-7f88-487d-b674-28db6ea9d156-kube-api-access-wlncm\") pod \"tuned-74q7d\" (UID: \"3557bbdf-7f88-487d-b674-28db6ea9d156\") " pod="openshift-cluster-node-tuning-operator/tuned-74q7d" Jul 23 19:19:36.567115 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:19:36.566420 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/67269f87-53d8-4f3a-b396-4891d9326c38-multus-socket-dir-parent\") pod \"multus-8967b\" (UID: \"67269f87-53d8-4f3a-b396-4891d9326c38\") " pod="openshift-multus/multus-8967b" Jul 23 19:19:36.567115 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:19:36.566445 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/67269f87-53d8-4f3a-b396-4891d9326c38-host-run-netns\") pod \"multus-8967b\" (UID: \"67269f87-53d8-4f3a-b396-4891d9326c38\") " pod="openshift-multus/multus-8967b" Jul 23 19:19:36.567115 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:19:36.566471 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/67269f87-53d8-4f3a-b396-4891d9326c38-host-var-lib-cni-multus\") pod \"multus-8967b\" (UID: \"67269f87-53d8-4f3a-b396-4891d9326c38\") " pod="openshift-multus/multus-8967b" Jul 23 19:19:36.568640 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:19:36.568485 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\"" Jul 23 19:19:36.568640 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:19:36.568509 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\"" Jul 23 19:19:36.568941 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:19:36.568834 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-lvbpx\"" Jul 23 19:19:36.568941 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:19:36.568912 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\"" Jul 23 19:19:36.569089 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:19:36.569066 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\"" Jul 23 19:19:36.569214 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:19:36.569176 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\"" Jul 23 19:19:36.569379 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:19:36.569345 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\"" Jul 23 19:19:36.569379 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:19:36.569368 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-8rq79" Jul 23 19:19:36.569473 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:19:36.569389 2564 reflector.go:439] "Caches populated" logger="kubernetes.io/kubelet-serving" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" Jul 23 19:19:36.571718 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:19:36.571699 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\"" Jul 23 19:19:36.572215 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:19:36.571909 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\"" Jul 23 19:19:36.572215 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:19:36.571953 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"default-dockercfg-v5v2x\"" Jul 23 19:19:36.572879 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:19:36.572858 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-l9rz7" Jul 23 19:19:36.575668 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:19:36.575518 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\"" Jul 23 19:19:36.575785 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:19:36.575768 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\"" Jul 23 19:19:36.575845 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:19:36.575798 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\"" Jul 23 19:19:36.576063 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:19:36.576035 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-hg5hf\"" Jul 23 19:19:36.576881 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:19:36.576860 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-9wwn6" Jul 23 19:19:36.579115 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:19:36.579094 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\"" Jul 23 19:19:36.579218 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:19:36.579180 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-b8l8k\"" Jul 23 19:19:36.579521 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:19:36.579501 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\"" Jul 23 19:19:36.579629 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:19:36.579586 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\"" Jul 23 19:19:36.591534 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:19:36.591515 2564 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kubelet-serving" csr="csr-nh42t" Jul 23 19:19:36.599650 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:19:36.599624 2564 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kubelet-serving" csr="csr-nh42t" Jul 23 19:19:36.655778 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:19:36.655392 2564 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world" Jul 23 19:19:36.657385 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:19:36.657322 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-2.ec2.internal" event={"ID":"ee6c09f813ef3a4bd239061ba78fb650","Type":"ContainerStarted","Data":"6125ec56e00d5f11e16417f2f00a4cc216a74fc3cd13f8eefd35bde61a75ac64"} Jul 23 19:19:36.659194 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:19:36.659169 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-129-2.ec2.internal" event={"ID":"f9c41c89c718c39ff54e57e416ba8933","Type":"ContainerStarted","Data":"dd6de12b7c4f3985d128c9f04f342e12c27b7bf30f4d3fc8ebd6787caca601c1"} Jul 23 19:19:36.667521 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:19:36.667493 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/67269f87-53d8-4f3a-b396-4891d9326c38-host-run-k8s-cni-cncf-io\") pod \"multus-8967b\" (UID: \"67269f87-53d8-4f3a-b396-4891d9326c38\") " pod="openshift-multus/multus-8967b" Jul 23 19:19:36.667603 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:19:36.667532 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/67269f87-53d8-4f3a-b396-4891d9326c38-host-var-lib-kubelet\") pod \"multus-8967b\" (UID: \"67269f87-53d8-4f3a-b396-4891d9326c38\") " pod="openshift-multus/multus-8967b" Jul 23 19:19:36.667603 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:19:36.667552 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/67269f87-53d8-4f3a-b396-4891d9326c38-multus-daemon-config\") pod \"multus-8967b\" (UID: \"67269f87-53d8-4f3a-b396-4891d9326c38\") " pod="openshift-multus/multus-8967b" Jul 23 19:19:36.667603 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:19:36.667570 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/67269f87-53d8-4f3a-b396-4891d9326c38-etc-kubernetes\") pod \"multus-8967b\" (UID: \"67269f87-53d8-4f3a-b396-4891d9326c38\") " pod="openshift-multus/multus-8967b" Jul 23 19:19:36.667603 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:19:36.667588 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/7c4060f3-0a15-4661-803f-fa6a845fc1ac-cnibin\") pod \"multus-additional-cni-plugins-sr96w\" (UID: \"7c4060f3-0a15-4661-803f-fa6a845fc1ac\") " pod="openshift-multus/multus-additional-cni-plugins-sr96w" Jul 23 19:19:36.667874 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:19:36.667614 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/3557bbdf-7f88-487d-b674-28db6ea9d156-etc-systemd\") pod \"tuned-74q7d\" (UID: \"3557bbdf-7f88-487d-b674-28db6ea9d156\") " pod="openshift-cluster-node-tuning-operator/tuned-74q7d" Jul 23 19:19:36.667874 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:19:36.667625 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/67269f87-53d8-4f3a-b396-4891d9326c38-host-run-k8s-cni-cncf-io\") pod \"multus-8967b\" (UID: \"67269f87-53d8-4f3a-b396-4891d9326c38\") " pod="openshift-multus/multus-8967b" Jul 23 19:19:36.667874 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:19:36.667633 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/67269f87-53d8-4f3a-b396-4891d9326c38-host-var-lib-kubelet\") pod \"multus-8967b\" (UID: \"67269f87-53d8-4f3a-b396-4891d9326c38\") " pod="openshift-multus/multus-8967b" Jul 23 19:19:36.667874 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:19:36.667640 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/3557bbdf-7f88-487d-b674-28db6ea9d156-tmp\") pod \"tuned-74q7d\" (UID: \"3557bbdf-7f88-487d-b674-28db6ea9d156\") " pod="openshift-cluster-node-tuning-operator/tuned-74q7d" Jul 23 19:19:36.667874 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:19:36.667660 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/67269f87-53d8-4f3a-b396-4891d9326c38-etc-kubernetes\") pod \"multus-8967b\" (UID: \"67269f87-53d8-4f3a-b396-4891d9326c38\") " pod="openshift-multus/multus-8967b" Jul 23 19:19:36.667874 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:19:36.667689 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/3557bbdf-7f88-487d-b674-28db6ea9d156-sys\") pod \"tuned-74q7d\" (UID: \"3557bbdf-7f88-487d-b674-28db6ea9d156\") " pod="openshift-cluster-node-tuning-operator/tuned-74q7d" Jul 23 19:19:36.667874 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:19:36.667691 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/7c4060f3-0a15-4661-803f-fa6a845fc1ac-cnibin\") pod \"multus-additional-cni-plugins-sr96w\" (UID: \"7c4060f3-0a15-4661-803f-fa6a845fc1ac\") " pod="openshift-multus/multus-additional-cni-plugins-sr96w" Jul 23 19:19:36.667874 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:19:36.667720 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hczmj\" (UniqueName: \"kubernetes.io/projected/7951a728-910c-4631-b555-119d1ea12709-kube-api-access-hczmj\") pod \"iptables-alerter-gbqdl\" (UID: \"7951a728-910c-4631-b555-119d1ea12709\") " pod="openshift-network-operator/iptables-alerter-gbqdl" Jul 23 19:19:36.667874 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:19:36.667724 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/3557bbdf-7f88-487d-b674-28db6ea9d156-etc-systemd\") pod \"tuned-74q7d\" (UID: \"3557bbdf-7f88-487d-b674-28db6ea9d156\") " pod="openshift-cluster-node-tuning-operator/tuned-74q7d" Jul 23 19:19:36.667874 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:19:36.667747 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/3557bbdf-7f88-487d-b674-28db6ea9d156-sys\") pod \"tuned-74q7d\" (UID: \"3557bbdf-7f88-487d-b674-28db6ea9d156\") " pod="openshift-cluster-node-tuning-operator/tuned-74q7d" Jul 23 19:19:36.667874 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:19:36.667748 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/25ca5174-01a0-41fc-9293-77dea2d9ec1f-konnectivity-ca\") pod \"konnectivity-agent-8rq79\" (UID: \"25ca5174-01a0-41fc-9293-77dea2d9ec1f\") " pod="kube-system/konnectivity-agent-8rq79" Jul 23 19:19:36.667874 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:19:36.667806 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/40f386ec-607d-496f-b65b-03a1fd847b1f-kubelet-config\") pod \"global-pull-secret-syncer-rtgbq\" (UID: \"40f386ec-607d-496f-b65b-03a1fd847b1f\") " pod="kube-system/global-pull-secret-syncer-rtgbq" Jul 23 19:19:36.667874 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:19:36.667830 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/195ecf89-13ae-4311-bc72-fcdf78a5c4d6-host-run-ovn-kubernetes\") pod \"ovnkube-node-8dtlh\" (UID: \"195ecf89-13ae-4311-bc72-fcdf78a5c4d6\") " pod="openshift-ovn-kubernetes/ovnkube-node-8dtlh" Jul 23 19:19:36.667874 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:19:36.667850 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/195ecf89-13ae-4311-bc72-fcdf78a5c4d6-host-cni-bin\") pod \"ovnkube-node-8dtlh\" (UID: \"195ecf89-13ae-4311-bc72-fcdf78a5c4d6\") " pod="openshift-ovn-kubernetes/ovnkube-node-8dtlh" Jul 23 19:19:36.667874 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:19:36.667866 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/195ecf89-13ae-4311-bc72-fcdf78a5c4d6-ovnkube-script-lib\") pod \"ovnkube-node-8dtlh\" (UID: \"195ecf89-13ae-4311-bc72-fcdf78a5c4d6\") " pod="openshift-ovn-kubernetes/ovnkube-node-8dtlh" Jul 23 19:19:36.667874 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:19:36.667884 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rgpqz\" (UniqueName: \"kubernetes.io/projected/67269f87-53d8-4f3a-b396-4891d9326c38-kube-api-access-rgpqz\") pod \"multus-8967b\" (UID: \"67269f87-53d8-4f3a-b396-4891d9326c38\") " pod="openshift-multus/multus-8967b" Jul 23 19:19:36.668562 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:19:36.667908 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/576f76ed-730c-49c2-a33e-1dc11370cad4-metrics-certs\") pod \"network-metrics-daemon-qhrz5\" (UID: \"576f76ed-730c-49c2-a33e-1dc11370cad4\") " pod="openshift-multus/network-metrics-daemon-qhrz5" Jul 23 19:19:36.668562 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:19:36.667933 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/3557bbdf-7f88-487d-b674-28db6ea9d156-run\") pod \"tuned-74q7d\" (UID: \"3557bbdf-7f88-487d-b674-28db6ea9d156\") " pod="openshift-cluster-node-tuning-operator/tuned-74q7d" Jul 23 19:19:36.668562 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:19:36.667929 2564 swap_util.go:74] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory" Jul 23 19:19:36.668562 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:19:36.667960 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/195ecf89-13ae-4311-bc72-fcdf78a5c4d6-host-kubelet\") pod \"ovnkube-node-8dtlh\" (UID: \"195ecf89-13ae-4311-bc72-fcdf78a5c4d6\") " pod="openshift-ovn-kubernetes/ovnkube-node-8dtlh" Jul 23 19:19:36.668562 ip-10-0-129-2 kubenswrapper[2564]: E0723 19:19:36.668012 2564 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 23 19:19:36.668562 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:19:36.668020 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/3557bbdf-7f88-487d-b674-28db6ea9d156-run\") pod \"tuned-74q7d\" (UID: \"3557bbdf-7f88-487d-b674-28db6ea9d156\") " pod="openshift-cluster-node-tuning-operator/tuned-74q7d" Jul 23 19:19:36.668562 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:19:36.668039 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/195ecf89-13ae-4311-bc72-fcdf78a5c4d6-host-cni-netd\") pod \"ovnkube-node-8dtlh\" (UID: \"195ecf89-13ae-4311-bc72-fcdf78a5c4d6\") " pod="openshift-ovn-kubernetes/ovnkube-node-8dtlh" Jul 23 19:19:36.668562 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:19:36.668060 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/195ecf89-13ae-4311-bc72-fcdf78a5c4d6-env-overrides\") pod \"ovnkube-node-8dtlh\" (UID: \"195ecf89-13ae-4311-bc72-fcdf78a5c4d6\") " pod="openshift-ovn-kubernetes/ovnkube-node-8dtlh" Jul 23 19:19:36.668562 ip-10-0-129-2 kubenswrapper[2564]: E0723 19:19:36.668089 2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/576f76ed-730c-49c2-a33e-1dc11370cad4-metrics-certs podName:576f76ed-730c-49c2-a33e-1dc11370cad4 nodeName:}" failed. No retries permitted until 2026-07-23 19:19:37.168068662 +0000 UTC m=+3.019844911 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/576f76ed-730c-49c2-a33e-1dc11370cad4-metrics-certs") pod "network-metrics-daemon-qhrz5" (UID: "576f76ed-730c-49c2-a33e-1dc11370cad4") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 23 19:19:36.668562 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:19:36.668109 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/67269f87-53d8-4f3a-b396-4891d9326c38-cni-binary-copy\") pod \"multus-8967b\" (UID: \"67269f87-53d8-4f3a-b396-4891d9326c38\") " pod="openshift-multus/multus-8967b" Jul 23 19:19:36.668562 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:19:36.668132 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/3557bbdf-7f88-487d-b674-28db6ea9d156-etc-modprobe-d\") pod \"tuned-74q7d\" (UID: \"3557bbdf-7f88-487d-b674-28db6ea9d156\") " pod="openshift-cluster-node-tuning-operator/tuned-74q7d" Jul 23 19:19:36.668562 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:19:36.668150 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/23313c02-67bf-498f-a294-e16cdc7d4517-hosts-file\") pod \"node-resolver-6k8wr\" (UID: \"23313c02-67bf-498f-a294-e16cdc7d4517\") " pod="openshift-dns/node-resolver-6k8wr" Jul 23 19:19:36.668562 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:19:36.668170 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/7951a728-910c-4631-b555-119d1ea12709-host-slash\") pod \"iptables-alerter-gbqdl\" (UID: \"7951a728-910c-4631-b555-119d1ea12709\") " pod="openshift-network-operator/iptables-alerter-gbqdl" Jul 23 19:19:36.668562 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:19:36.668196 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/67269f87-53d8-4f3a-b396-4891d9326c38-host-var-lib-cni-bin\") pod \"multus-8967b\" (UID: \"67269f87-53d8-4f3a-b396-4891d9326c38\") " pod="openshift-multus/multus-8967b" Jul 23 19:19:36.668562 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:19:36.668218 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/7c4060f3-0a15-4661-803f-fa6a845fc1ac-os-release\") pod \"multus-additional-cni-plugins-sr96w\" (UID: \"7c4060f3-0a15-4661-803f-fa6a845fc1ac\") " pod="openshift-multus/multus-additional-cni-plugins-sr96w" Jul 23 19:19:36.668562 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:19:36.668215 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/67269f87-53d8-4f3a-b396-4891d9326c38-multus-daemon-config\") pod \"multus-8967b\" (UID: \"67269f87-53d8-4f3a-b396-4891d9326c38\") " pod="openshift-multus/multus-8967b" Jul 23 19:19:36.668562 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:19:36.668239 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/3557bbdf-7f88-487d-b674-28db6ea9d156-lib-modules\") pod \"tuned-74q7d\" (UID: \"3557bbdf-7f88-487d-b674-28db6ea9d156\") " pod="openshift-cluster-node-tuning-operator/tuned-74q7d" Jul 23 19:19:36.669292 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:19:36.668299 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/3557bbdf-7f88-487d-b674-28db6ea9d156-etc-modprobe-d\") pod \"tuned-74q7d\" (UID: \"3557bbdf-7f88-487d-b674-28db6ea9d156\") " pod="openshift-cluster-node-tuning-operator/tuned-74q7d" Jul 23 19:19:36.669292 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:19:36.668314 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/67269f87-53d8-4f3a-b396-4891d9326c38-host-var-lib-cni-bin\") pod \"multus-8967b\" (UID: \"67269f87-53d8-4f3a-b396-4891d9326c38\") " pod="openshift-multus/multus-8967b" Jul 23 19:19:36.669292 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:19:36.668313 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/7c4060f3-0a15-4661-803f-fa6a845fc1ac-os-release\") pod \"multus-additional-cni-plugins-sr96w\" (UID: \"7c4060f3-0a15-4661-803f-fa6a845fc1ac\") " pod="openshift-multus/multus-additional-cni-plugins-sr96w" Jul 23 19:19:36.669292 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:19:36.668343 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/3557bbdf-7f88-487d-b674-28db6ea9d156-lib-modules\") pod \"tuned-74q7d\" (UID: \"3557bbdf-7f88-487d-b674-28db6ea9d156\") " pod="openshift-cluster-node-tuning-operator/tuned-74q7d" Jul 23 19:19:36.669292 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:19:36.668381 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/23313c02-67bf-498f-a294-e16cdc7d4517-hosts-file\") pod \"node-resolver-6k8wr\" (UID: \"23313c02-67bf-498f-a294-e16cdc7d4517\") " pod="openshift-dns/node-resolver-6k8wr" Jul 23 19:19:36.669292 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:19:36.668421 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ldzrw\" (UniqueName: \"kubernetes.io/projected/dd466782-4d8e-42e2-83c2-ed5f1b48567b-kube-api-access-ldzrw\") pod \"node-ca-9wwn6\" (UID: \"dd466782-4d8e-42e2-83c2-ed5f1b48567b\") " pod="openshift-image-registry/node-ca-9wwn6" Jul 23 19:19:36.669292 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:19:36.668455 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/67269f87-53d8-4f3a-b396-4891d9326c38-hostroot\") pod \"multus-8967b\" (UID: \"67269f87-53d8-4f3a-b396-4891d9326c38\") " pod="openshift-multus/multus-8967b" Jul 23 19:19:36.669292 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:19:36.668484 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/7c4060f3-0a15-4661-803f-fa6a845fc1ac-tuning-conf-dir\") pod \"multus-additional-cni-plugins-sr96w\" (UID: \"7c4060f3-0a15-4661-803f-fa6a845fc1ac\") " pod="openshift-multus/multus-additional-cni-plugins-sr96w" Jul 23 19:19:36.669292 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:19:36.668508 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/67269f87-53d8-4f3a-b396-4891d9326c38-hostroot\") pod \"multus-8967b\" (UID: \"67269f87-53d8-4f3a-b396-4891d9326c38\") " pod="openshift-multus/multus-8967b" Jul 23 19:19:36.669292 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:19:36.668510 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wlncm\" (UniqueName: \"kubernetes.io/projected/3557bbdf-7f88-487d-b674-28db6ea9d156-kube-api-access-wlncm\") pod \"tuned-74q7d\" (UID: \"3557bbdf-7f88-487d-b674-28db6ea9d156\") " pod="openshift-cluster-node-tuning-operator/tuned-74q7d" Jul 23 19:19:36.669292 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:19:36.668553 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/195ecf89-13ae-4311-bc72-fcdf78a5c4d6-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-8dtlh\" (UID: \"195ecf89-13ae-4311-bc72-fcdf78a5c4d6\") " pod="openshift-ovn-kubernetes/ovnkube-node-8dtlh" Jul 23 19:19:36.669292 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:19:36.668576 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/67269f87-53d8-4f3a-b396-4891d9326c38-multus-socket-dir-parent\") pod \"multus-8967b\" (UID: \"67269f87-53d8-4f3a-b396-4891d9326c38\") " pod="openshift-multus/multus-8967b" Jul 23 19:19:36.669292 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:19:36.668593 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/3557bbdf-7f88-487d-b674-28db6ea9d156-host\") pod \"tuned-74q7d\" (UID: \"3557bbdf-7f88-487d-b674-28db6ea9d156\") " pod="openshift-cluster-node-tuning-operator/tuned-74q7d" Jul 23 19:19:36.669292 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:19:36.668611 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/23313c02-67bf-498f-a294-e16cdc7d4517-tmp-dir\") pod \"node-resolver-6k8wr\" (UID: \"23313c02-67bf-498f-a294-e16cdc7d4517\") " pod="openshift-dns/node-resolver-6k8wr" Jul 23 19:19:36.669292 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:19:36.668637 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/67269f87-53d8-4f3a-b396-4891d9326c38-system-cni-dir\") pod \"multus-8967b\" (UID: \"67269f87-53d8-4f3a-b396-4891d9326c38\") " pod="openshift-multus/multus-8967b" Jul 23 19:19:36.669292 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:19:36.668658 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/7c4060f3-0a15-4661-803f-fa6a845fc1ac-tuning-conf-dir\") pod \"multus-additional-cni-plugins-sr96w\" (UID: \"7c4060f3-0a15-4661-803f-fa6a845fc1ac\") " pod="openshift-multus/multus-additional-cni-plugins-sr96w" Jul 23 19:19:36.669292 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:19:36.668668 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/67269f87-53d8-4f3a-b396-4891d9326c38-multus-socket-dir-parent\") pod \"multus-8967b\" (UID: \"67269f87-53d8-4f3a-b396-4891d9326c38\") " pod="openshift-multus/multus-8967b" Jul 23 19:19:36.670145 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:19:36.668663 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/67269f87-53d8-4f3a-b396-4891d9326c38-host-run-multus-certs\") pod \"multus-8967b\" (UID: \"67269f87-53d8-4f3a-b396-4891d9326c38\") " pod="openshift-multus/multus-8967b" Jul 23 19:19:36.670145 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:19:36.668691 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/3557bbdf-7f88-487d-b674-28db6ea9d156-host\") pod \"tuned-74q7d\" (UID: \"3557bbdf-7f88-487d-b674-28db6ea9d156\") " pod="openshift-cluster-node-tuning-operator/tuned-74q7d" Jul 23 19:19:36.670145 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:19:36.668713 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/67269f87-53d8-4f3a-b396-4891d9326c38-host-run-multus-certs\") pod \"multus-8967b\" (UID: \"67269f87-53d8-4f3a-b396-4891d9326c38\") " pod="openshift-multus/multus-8967b" Jul 23 19:19:36.670145 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:19:36.668755 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/7c4060f3-0a15-4661-803f-fa6a845fc1ac-system-cni-dir\") pod \"multus-additional-cni-plugins-sr96w\" (UID: \"7c4060f3-0a15-4661-803f-fa6a845fc1ac\") " pod="openshift-multus/multus-additional-cni-plugins-sr96w" Jul 23 19:19:36.670145 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:19:36.668781 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/3557bbdf-7f88-487d-b674-28db6ea9d156-etc-sysctl-d\") pod \"tuned-74q7d\" (UID: \"3557bbdf-7f88-487d-b674-28db6ea9d156\") " pod="openshift-cluster-node-tuning-operator/tuned-74q7d" Jul 23 19:19:36.670145 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:19:36.668800 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/7c4060f3-0a15-4661-803f-fa6a845fc1ac-system-cni-dir\") pod \"multus-additional-cni-plugins-sr96w\" (UID: \"7c4060f3-0a15-4661-803f-fa6a845fc1ac\") " pod="openshift-multus/multus-additional-cni-plugins-sr96w" Jul 23 19:19:36.670145 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:19:36.668816 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/67269f87-53d8-4f3a-b396-4891d9326c38-cni-binary-copy\") pod \"multus-8967b\" (UID: \"67269f87-53d8-4f3a-b396-4891d9326c38\") " pod="openshift-multus/multus-8967b" Jul 23 19:19:36.670145 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:19:36.668838 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/3557bbdf-7f88-487d-b674-28db6ea9d156-var-lib-kubelet\") pod \"tuned-74q7d\" (UID: \"3557bbdf-7f88-487d-b674-28db6ea9d156\") " pod="openshift-cluster-node-tuning-operator/tuned-74q7d" Jul 23 19:19:36.670145 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:19:36.668864 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/195ecf89-13ae-4311-bc72-fcdf78a5c4d6-node-log\") pod \"ovnkube-node-8dtlh\" (UID: \"195ecf89-13ae-4311-bc72-fcdf78a5c4d6\") " pod="openshift-ovn-kubernetes/ovnkube-node-8dtlh" Jul 23 19:19:36.670145 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:19:36.668918 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/67269f87-53d8-4f3a-b396-4891d9326c38-system-cni-dir\") pod \"multus-8967b\" (UID: \"67269f87-53d8-4f3a-b396-4891d9326c38\") " pod="openshift-multus/multus-8967b" Jul 23 19:19:36.670145 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:19:36.668929 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/3557bbdf-7f88-487d-b674-28db6ea9d156-etc-sysctl-d\") pod \"tuned-74q7d\" (UID: \"3557bbdf-7f88-487d-b674-28db6ea9d156\") " pod="openshift-cluster-node-tuning-operator/tuned-74q7d" Jul 23 19:19:36.670145 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:19:36.668960 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/67269f87-53d8-4f3a-b396-4891d9326c38-multus-cni-dir\") pod \"multus-8967b\" (UID: \"67269f87-53d8-4f3a-b396-4891d9326c38\") " pod="openshift-multus/multus-8967b" Jul 23 19:19:36.670145 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:19:36.668992 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/23313c02-67bf-498f-a294-e16cdc7d4517-tmp-dir\") pod \"node-resolver-6k8wr\" (UID: \"23313c02-67bf-498f-a294-e16cdc7d4517\") " pod="openshift-dns/node-resolver-6k8wr" Jul 23 19:19:36.670145 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:19:36.669000 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/3557bbdf-7f88-487d-b674-28db6ea9d156-var-lib-kubelet\") pod \"tuned-74q7d\" (UID: \"3557bbdf-7f88-487d-b674-28db6ea9d156\") " pod="openshift-cluster-node-tuning-operator/tuned-74q7d" Jul 23 19:19:36.670145 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:19:36.669054 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/67269f87-53d8-4f3a-b396-4891d9326c38-os-release\") pod \"multus-8967b\" (UID: \"67269f87-53d8-4f3a-b396-4891d9326c38\") " pod="openshift-multus/multus-8967b" Jul 23 19:19:36.670145 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:19:36.669104 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-54vhn\" (UniqueName: \"kubernetes.io/projected/576f76ed-730c-49c2-a33e-1dc11370cad4-kube-api-access-54vhn\") pod \"network-metrics-daemon-qhrz5\" (UID: \"576f76ed-730c-49c2-a33e-1dc11370cad4\") " pod="openshift-multus/network-metrics-daemon-qhrz5" Jul 23 19:19:36.670145 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:19:36.669145 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/67269f87-53d8-4f3a-b396-4891d9326c38-multus-cni-dir\") pod \"multus-8967b\" (UID: \"67269f87-53d8-4f3a-b396-4891d9326c38\") " pod="openshift-multus/multus-8967b" Jul 23 19:19:36.670145 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:19:36.669147 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/195ecf89-13ae-4311-bc72-fcdf78a5c4d6-run-systemd\") pod \"ovnkube-node-8dtlh\" (UID: \"195ecf89-13ae-4311-bc72-fcdf78a5c4d6\") " pod="openshift-ovn-kubernetes/ovnkube-node-8dtlh" Jul 23 19:19:36.670891 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:19:36.669154 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/67269f87-53d8-4f3a-b396-4891d9326c38-os-release\") pod \"multus-8967b\" (UID: \"67269f87-53d8-4f3a-b396-4891d9326c38\") " pod="openshift-multus/multus-8967b" Jul 23 19:19:36.670891 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:19:36.669264 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fc8lq\" (UniqueName: \"kubernetes.io/projected/195ecf89-13ae-4311-bc72-fcdf78a5c4d6-kube-api-access-fc8lq\") pod \"ovnkube-node-8dtlh\" (UID: \"195ecf89-13ae-4311-bc72-fcdf78a5c4d6\") " pod="openshift-ovn-kubernetes/ovnkube-node-8dtlh" Jul 23 19:19:36.670891 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:19:36.669292 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/dd466782-4d8e-42e2-83c2-ed5f1b48567b-serviceca\") pod \"node-ca-9wwn6\" (UID: \"dd466782-4d8e-42e2-83c2-ed5f1b48567b\") " pod="openshift-image-registry/node-ca-9wwn6" Jul 23 19:19:36.670891 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:19:36.669352 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/e83347a0-7f9a-4a7a-8a71-b662b464cd6b-etc-selinux\") pod \"aws-ebs-csi-driver-node-l9rz7\" (UID: \"e83347a0-7f9a-4a7a-8a71-b662b464cd6b\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-l9rz7" Jul 23 19:19:36.670891 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:19:36.669388 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/195ecf89-13ae-4311-bc72-fcdf78a5c4d6-host-run-netns\") pod \"ovnkube-node-8dtlh\" (UID: \"195ecf89-13ae-4311-bc72-fcdf78a5c4d6\") " pod="openshift-ovn-kubernetes/ovnkube-node-8dtlh" Jul 23 19:19:36.670891 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:19:36.669410 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/195ecf89-13ae-4311-bc72-fcdf78a5c4d6-run-openvswitch\") pod \"ovnkube-node-8dtlh\" (UID: \"195ecf89-13ae-4311-bc72-fcdf78a5c4d6\") " pod="openshift-ovn-kubernetes/ovnkube-node-8dtlh" Jul 23 19:19:36.670891 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:19:36.669449 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/67269f87-53d8-4f3a-b396-4891d9326c38-cnibin\") pod \"multus-8967b\" (UID: \"67269f87-53d8-4f3a-b396-4891d9326c38\") " pod="openshift-multus/multus-8967b" Jul 23 19:19:36.670891 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:19:36.669486 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hx49w\" (UniqueName: \"kubernetes.io/projected/23313c02-67bf-498f-a294-e16cdc7d4517-kube-api-access-hx49w\") pod \"node-resolver-6k8wr\" (UID: \"23313c02-67bf-498f-a294-e16cdc7d4517\") " pod="openshift-dns/node-resolver-6k8wr" Jul 23 19:19:36.670891 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:19:36.669499 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/67269f87-53d8-4f3a-b396-4891d9326c38-cnibin\") pod \"multus-8967b\" (UID: \"67269f87-53d8-4f3a-b396-4891d9326c38\") " pod="openshift-multus/multus-8967b" Jul 23 19:19:36.670891 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:19:36.669517 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/195ecf89-13ae-4311-bc72-fcdf78a5c4d6-host-slash\") pod \"ovnkube-node-8dtlh\" (UID: \"195ecf89-13ae-4311-bc72-fcdf78a5c4d6\") " pod="openshift-ovn-kubernetes/ovnkube-node-8dtlh" Jul 23 19:19:36.670891 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:19:36.669551 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/195ecf89-13ae-4311-bc72-fcdf78a5c4d6-etc-openvswitch\") pod \"ovnkube-node-8dtlh\" (UID: \"195ecf89-13ae-4311-bc72-fcdf78a5c4d6\") " pod="openshift-ovn-kubernetes/ovnkube-node-8dtlh" Jul 23 19:19:36.670891 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:19:36.669576 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/7c4060f3-0a15-4661-803f-fa6a845fc1ac-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-sr96w\" (UID: \"7c4060f3-0a15-4661-803f-fa6a845fc1ac\") " pod="openshift-multus/multus-additional-cni-plugins-sr96w" Jul 23 19:19:36.670891 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:19:36.669600 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cq8ll\" (UniqueName: \"kubernetes.io/projected/8706aab3-bba6-4fff-b8bc-8890d14daa9c-kube-api-access-cq8ll\") pod \"network-check-target-vxngp\" (UID: \"8706aab3-bba6-4fff-b8bc-8890d14daa9c\") " pod="openshift-network-diagnostics/network-check-target-vxngp" Jul 23 19:19:36.670891 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:19:36.669622 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/e83347a0-7f9a-4a7a-8a71-b662b464cd6b-socket-dir\") pod \"aws-ebs-csi-driver-node-l9rz7\" (UID: \"e83347a0-7f9a-4a7a-8a71-b662b464cd6b\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-l9rz7" Jul 23 19:19:36.670891 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:19:36.669642 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xf6x4\" (UniqueName: \"kubernetes.io/projected/e83347a0-7f9a-4a7a-8a71-b662b464cd6b-kube-api-access-xf6x4\") pod \"aws-ebs-csi-driver-node-l9rz7\" (UID: \"e83347a0-7f9a-4a7a-8a71-b662b464cd6b\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-l9rz7" Jul 23 19:19:36.670891 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:19:36.669663 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/40f386ec-607d-496f-b65b-03a1fd847b1f-original-pull-secret\") pod \"global-pull-secret-syncer-rtgbq\" (UID: \"40f386ec-607d-496f-b65b-03a1fd847b1f\") " pod="kube-system/global-pull-secret-syncer-rtgbq" Jul 23 19:19:36.670891 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:19:36.669687 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/195ecf89-13ae-4311-bc72-fcdf78a5c4d6-log-socket\") pod \"ovnkube-node-8dtlh\" (UID: \"195ecf89-13ae-4311-bc72-fcdf78a5c4d6\") " pod="openshift-ovn-kubernetes/ovnkube-node-8dtlh" Jul 23 19:19:36.671465 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:19:36.669708 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/dd466782-4d8e-42e2-83c2-ed5f1b48567b-host\") pod \"node-ca-9wwn6\" (UID: \"dd466782-4d8e-42e2-83c2-ed5f1b48567b\") " pod="openshift-image-registry/node-ca-9wwn6" Jul 23 19:19:36.671465 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:19:36.669749 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/3557bbdf-7f88-487d-b674-28db6ea9d156-etc-sysconfig\") pod \"tuned-74q7d\" (UID: \"3557bbdf-7f88-487d-b674-28db6ea9d156\") " pod="openshift-cluster-node-tuning-operator/tuned-74q7d" Jul 23 19:19:36.671465 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:19:36.669770 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/3557bbdf-7f88-487d-b674-28db6ea9d156-etc-sysctl-conf\") pod \"tuned-74q7d\" (UID: \"3557bbdf-7f88-487d-b674-28db6ea9d156\") " pod="openshift-cluster-node-tuning-operator/tuned-74q7d" Jul 23 19:19:36.671465 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:19:36.669791 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/3557bbdf-7f88-487d-b674-28db6ea9d156-etc-tuned\") pod \"tuned-74q7d\" (UID: \"3557bbdf-7f88-487d-b674-28db6ea9d156\") " pod="openshift-cluster-node-tuning-operator/tuned-74q7d" Jul 23 19:19:36.671465 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:19:36.669825 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/e83347a0-7f9a-4a7a-8a71-b662b464cd6b-device-dir\") pod \"aws-ebs-csi-driver-node-l9rz7\" (UID: \"e83347a0-7f9a-4a7a-8a71-b662b464cd6b\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-l9rz7" Jul 23 19:19:36.671465 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:19:36.669857 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/40f386ec-607d-496f-b65b-03a1fd847b1f-dbus\") pod \"global-pull-secret-syncer-rtgbq\" (UID: \"40f386ec-607d-496f-b65b-03a1fd847b1f\") " pod="kube-system/global-pull-secret-syncer-rtgbq" Jul 23 19:19:36.671465 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:19:36.669913 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/3557bbdf-7f88-487d-b674-28db6ea9d156-etc-sysconfig\") pod \"tuned-74q7d\" (UID: \"3557bbdf-7f88-487d-b674-28db6ea9d156\") " pod="openshift-cluster-node-tuning-operator/tuned-74q7d" Jul 23 19:19:36.671465 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:19:36.669944 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/195ecf89-13ae-4311-bc72-fcdf78a5c4d6-run-ovn\") pod \"ovnkube-node-8dtlh\" (UID: \"195ecf89-13ae-4311-bc72-fcdf78a5c4d6\") " pod="openshift-ovn-kubernetes/ovnkube-node-8dtlh" Jul 23 19:19:36.671465 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:19:36.669954 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/3557bbdf-7f88-487d-b674-28db6ea9d156-etc-sysctl-conf\") pod \"tuned-74q7d\" (UID: \"3557bbdf-7f88-487d-b674-28db6ea9d156\") " pod="openshift-cluster-node-tuning-operator/tuned-74q7d" Jul 23 19:19:36.671465 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:19:36.669970 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/195ecf89-13ae-4311-bc72-fcdf78a5c4d6-ovnkube-config\") pod \"ovnkube-node-8dtlh\" (UID: \"195ecf89-13ae-4311-bc72-fcdf78a5c4d6\") " pod="openshift-ovn-kubernetes/ovnkube-node-8dtlh" Jul 23 19:19:36.671465 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:19:36.670007 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/67269f87-53d8-4f3a-b396-4891d9326c38-multus-conf-dir\") pod \"multus-8967b\" (UID: \"67269f87-53d8-4f3a-b396-4891d9326c38\") " pod="openshift-multus/multus-8967b" Jul 23 19:19:36.671465 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:19:36.670042 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/3557bbdf-7f88-487d-b674-28db6ea9d156-etc-kubernetes\") pod \"tuned-74q7d\" (UID: \"3557bbdf-7f88-487d-b674-28db6ea9d156\") " pod="openshift-cluster-node-tuning-operator/tuned-74q7d" Jul 23 19:19:36.671465 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:19:36.670070 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/e83347a0-7f9a-4a7a-8a71-b662b464cd6b-kubelet-dir\") pod \"aws-ebs-csi-driver-node-l9rz7\" (UID: \"e83347a0-7f9a-4a7a-8a71-b662b464cd6b\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-l9rz7" Jul 23 19:19:36.671465 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:19:36.670117 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/67269f87-53d8-4f3a-b396-4891d9326c38-multus-conf-dir\") pod \"multus-8967b\" (UID: \"67269f87-53d8-4f3a-b396-4891d9326c38\") " pod="openshift-multus/multus-8967b" Jul 23 19:19:36.671465 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:19:36.670119 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/3557bbdf-7f88-487d-b674-28db6ea9d156-etc-kubernetes\") pod \"tuned-74q7d\" (UID: \"3557bbdf-7f88-487d-b674-28db6ea9d156\") " pod="openshift-cluster-node-tuning-operator/tuned-74q7d" Jul 23 19:19:36.671465 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:19:36.670120 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/7c4060f3-0a15-4661-803f-fa6a845fc1ac-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-sr96w\" (UID: \"7c4060f3-0a15-4661-803f-fa6a845fc1ac\") " pod="openshift-multus/multus-additional-cni-plugins-sr96w" Jul 23 19:19:36.671465 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:19:36.670146 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/e83347a0-7f9a-4a7a-8a71-b662b464cd6b-registration-dir\") pod \"aws-ebs-csi-driver-node-l9rz7\" (UID: \"e83347a0-7f9a-4a7a-8a71-b662b464cd6b\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-l9rz7" Jul 23 19:19:36.672069 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:19:36.670189 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/195ecf89-13ae-4311-bc72-fcdf78a5c4d6-ovn-node-metrics-cert\") pod \"ovnkube-node-8dtlh\" (UID: \"195ecf89-13ae-4311-bc72-fcdf78a5c4d6\") " pod="openshift-ovn-kubernetes/ovnkube-node-8dtlh" Jul 23 19:19:36.672069 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:19:36.670218 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/67269f87-53d8-4f3a-b396-4891d9326c38-host-run-netns\") pod \"multus-8967b\" (UID: \"67269f87-53d8-4f3a-b396-4891d9326c38\") " pod="openshift-multus/multus-8967b" Jul 23 19:19:36.672069 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:19:36.670244 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/67269f87-53d8-4f3a-b396-4891d9326c38-host-var-lib-cni-multus\") pod \"multus-8967b\" (UID: \"67269f87-53d8-4f3a-b396-4891d9326c38\") " pod="openshift-multus/multus-8967b" Jul 23 19:19:36.672069 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:19:36.670269 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/7c4060f3-0a15-4661-803f-fa6a845fc1ac-cni-binary-copy\") pod \"multus-additional-cni-plugins-sr96w\" (UID: \"7c4060f3-0a15-4661-803f-fa6a845fc1ac\") " pod="openshift-multus/multus-additional-cni-plugins-sr96w" Jul 23 19:19:36.672069 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:19:36.670298 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/67269f87-53d8-4f3a-b396-4891d9326c38-host-var-lib-cni-multus\") pod \"multus-8967b\" (UID: \"67269f87-53d8-4f3a-b396-4891d9326c38\") " pod="openshift-multus/multus-8967b" Jul 23 19:19:36.672069 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:19:36.670299 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jf5zp\" (UniqueName: \"kubernetes.io/projected/7c4060f3-0a15-4661-803f-fa6a845fc1ac-kube-api-access-jf5zp\") pod \"multus-additional-cni-plugins-sr96w\" (UID: \"7c4060f3-0a15-4661-803f-fa6a845fc1ac\") " pod="openshift-multus/multus-additional-cni-plugins-sr96w" Jul 23 19:19:36.672069 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:19:36.670331 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/7951a728-910c-4631-b555-119d1ea12709-iptables-alerter-script\") pod \"iptables-alerter-gbqdl\" (UID: \"7951a728-910c-4631-b555-119d1ea12709\") " pod="openshift-network-operator/iptables-alerter-gbqdl" Jul 23 19:19:36.672069 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:19:36.670273 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/67269f87-53d8-4f3a-b396-4891d9326c38-host-run-netns\") pod \"multus-8967b\" (UID: \"67269f87-53d8-4f3a-b396-4891d9326c38\") " pod="openshift-multus/multus-8967b" Jul 23 19:19:36.672069 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:19:36.670357 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/25ca5174-01a0-41fc-9293-77dea2d9ec1f-agent-certs\") pod \"konnectivity-agent-8rq79\" (UID: \"25ca5174-01a0-41fc-9293-77dea2d9ec1f\") " pod="kube-system/konnectivity-agent-8rq79" Jul 23 19:19:36.672069 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:19:36.670378 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/e83347a0-7f9a-4a7a-8a71-b662b464cd6b-sys-fs\") pod \"aws-ebs-csi-driver-node-l9rz7\" (UID: \"e83347a0-7f9a-4a7a-8a71-b662b464cd6b\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-l9rz7" Jul 23 19:19:36.672069 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:19:36.670397 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/195ecf89-13ae-4311-bc72-fcdf78a5c4d6-systemd-units\") pod \"ovnkube-node-8dtlh\" (UID: \"195ecf89-13ae-4311-bc72-fcdf78a5c4d6\") " pod="openshift-ovn-kubernetes/ovnkube-node-8dtlh" Jul 23 19:19:36.672069 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:19:36.670439 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/7c4060f3-0a15-4661-803f-fa6a845fc1ac-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-sr96w\" (UID: \"7c4060f3-0a15-4661-803f-fa6a845fc1ac\") " pod="openshift-multus/multus-additional-cni-plugins-sr96w" Jul 23 19:19:36.672069 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:19:36.670454 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/195ecf89-13ae-4311-bc72-fcdf78a5c4d6-var-lib-openvswitch\") pod \"ovnkube-node-8dtlh\" (UID: \"195ecf89-13ae-4311-bc72-fcdf78a5c4d6\") " pod="openshift-ovn-kubernetes/ovnkube-node-8dtlh" Jul 23 19:19:36.672069 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:19:36.670791 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/7c4060f3-0a15-4661-803f-fa6a845fc1ac-cni-binary-copy\") pod \"multus-additional-cni-plugins-sr96w\" (UID: \"7c4060f3-0a15-4661-803f-fa6a845fc1ac\") " pod="openshift-multus/multus-additional-cni-plugins-sr96w" Jul 23 19:19:36.672069 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:19:36.670984 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/7c4060f3-0a15-4661-803f-fa6a845fc1ac-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-sr96w\" (UID: \"7c4060f3-0a15-4661-803f-fa6a845fc1ac\") " pod="openshift-multus/multus-additional-cni-plugins-sr96w" Jul 23 19:19:36.672762 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:19:36.672605 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/3557bbdf-7f88-487d-b674-28db6ea9d156-etc-tuned\") pod \"tuned-74q7d\" (UID: \"3557bbdf-7f88-487d-b674-28db6ea9d156\") " pod="openshift-cluster-node-tuning-operator/tuned-74q7d" Jul 23 19:19:36.672762 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:19:36.672626 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/3557bbdf-7f88-487d-b674-28db6ea9d156-tmp\") pod \"tuned-74q7d\" (UID: \"3557bbdf-7f88-487d-b674-28db6ea9d156\") " pod="openshift-cluster-node-tuning-operator/tuned-74q7d" Jul 23 19:19:36.679845 ip-10-0-129-2 kubenswrapper[2564]: E0723 19:19:36.679459 2564 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 23 19:19:36.679845 ip-10-0-129-2 kubenswrapper[2564]: E0723 19:19:36.679484 2564 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 23 19:19:36.679845 ip-10-0-129-2 kubenswrapper[2564]: E0723 19:19:36.679499 2564 projected.go:196] Error preparing data for projected volume kube-api-access-cq8ll for pod openshift-network-diagnostics/network-check-target-vxngp: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 23 19:19:36.679845 ip-10-0-129-2 kubenswrapper[2564]: E0723 19:19:36.679565 2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/8706aab3-bba6-4fff-b8bc-8890d14daa9c-kube-api-access-cq8ll podName:8706aab3-bba6-4fff-b8bc-8890d14daa9c nodeName:}" failed. No retries permitted until 2026-07-23 19:19:37.179546352 +0000 UTC m=+3.031322601 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-api-access-cq8ll" (UniqueName: "kubernetes.io/projected/8706aab3-bba6-4fff-b8bc-8890d14daa9c-kube-api-access-cq8ll") pod "network-check-target-vxngp" (UID: "8706aab3-bba6-4fff-b8bc-8890d14daa9c") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 23 19:19:36.680238 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:19:36.680215 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-wlncm\" (UniqueName: \"kubernetes.io/projected/3557bbdf-7f88-487d-b674-28db6ea9d156-kube-api-access-wlncm\") pod \"tuned-74q7d\" (UID: \"3557bbdf-7f88-487d-b674-28db6ea9d156\") " pod="openshift-cluster-node-tuning-operator/tuned-74q7d" Jul 23 19:19:36.680438 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:19:36.680411 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-54vhn\" (UniqueName: \"kubernetes.io/projected/576f76ed-730c-49c2-a33e-1dc11370cad4-kube-api-access-54vhn\") pod \"network-metrics-daemon-qhrz5\" (UID: \"576f76ed-730c-49c2-a33e-1dc11370cad4\") " pod="openshift-multus/network-metrics-daemon-qhrz5" Jul 23 19:19:36.680843 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:19:36.680824 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rgpqz\" (UniqueName: \"kubernetes.io/projected/67269f87-53d8-4f3a-b396-4891d9326c38-kube-api-access-rgpqz\") pod \"multus-8967b\" (UID: \"67269f87-53d8-4f3a-b396-4891d9326c38\") " pod="openshift-multus/multus-8967b" Jul 23 19:19:36.681562 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:19:36.681544 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jf5zp\" (UniqueName: \"kubernetes.io/projected/7c4060f3-0a15-4661-803f-fa6a845fc1ac-kube-api-access-jf5zp\") pod \"multus-additional-cni-plugins-sr96w\" (UID: \"7c4060f3-0a15-4661-803f-fa6a845fc1ac\") " pod="openshift-multus/multus-additional-cni-plugins-sr96w" Jul 23 19:19:36.681995 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:19:36.681971 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-hx49w\" (UniqueName: \"kubernetes.io/projected/23313c02-67bf-498f-a294-e16cdc7d4517-kube-api-access-hx49w\") pod \"node-resolver-6k8wr\" (UID: \"23313c02-67bf-498f-a294-e16cdc7d4517\") " pod="openshift-dns/node-resolver-6k8wr" Jul 23 19:19:36.770791 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:19:36.770758 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/195ecf89-13ae-4311-bc72-fcdf78a5c4d6-node-log\") pod \"ovnkube-node-8dtlh\" (UID: \"195ecf89-13ae-4311-bc72-fcdf78a5c4d6\") " pod="openshift-ovn-kubernetes/ovnkube-node-8dtlh" Jul 23 19:19:36.770898 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:19:36.770800 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/195ecf89-13ae-4311-bc72-fcdf78a5c4d6-run-systemd\") pod \"ovnkube-node-8dtlh\" (UID: \"195ecf89-13ae-4311-bc72-fcdf78a5c4d6\") " pod="openshift-ovn-kubernetes/ovnkube-node-8dtlh" Jul 23 19:19:36.770898 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:19:36.770830 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fc8lq\" (UniqueName: \"kubernetes.io/projected/195ecf89-13ae-4311-bc72-fcdf78a5c4d6-kube-api-access-fc8lq\") pod \"ovnkube-node-8dtlh\" (UID: \"195ecf89-13ae-4311-bc72-fcdf78a5c4d6\") " pod="openshift-ovn-kubernetes/ovnkube-node-8dtlh" Jul 23 19:19:36.770898 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:19:36.770838 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/195ecf89-13ae-4311-bc72-fcdf78a5c4d6-node-log\") pod \"ovnkube-node-8dtlh\" (UID: \"195ecf89-13ae-4311-bc72-fcdf78a5c4d6\") " pod="openshift-ovn-kubernetes/ovnkube-node-8dtlh" Jul 23 19:19:36.771055 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:19:36.770894 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/195ecf89-13ae-4311-bc72-fcdf78a5c4d6-run-systemd\") pod \"ovnkube-node-8dtlh\" (UID: \"195ecf89-13ae-4311-bc72-fcdf78a5c4d6\") " pod="openshift-ovn-kubernetes/ovnkube-node-8dtlh" Jul 23 19:19:36.771055 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:19:36.770943 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/dd466782-4d8e-42e2-83c2-ed5f1b48567b-serviceca\") pod \"node-ca-9wwn6\" (UID: \"dd466782-4d8e-42e2-83c2-ed5f1b48567b\") " pod="openshift-image-registry/node-ca-9wwn6" Jul 23 19:19:36.771055 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:19:36.770976 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/e83347a0-7f9a-4a7a-8a71-b662b464cd6b-etc-selinux\") pod \"aws-ebs-csi-driver-node-l9rz7\" (UID: \"e83347a0-7f9a-4a7a-8a71-b662b464cd6b\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-l9rz7" Jul 23 19:19:36.771055 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:19:36.771002 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/195ecf89-13ae-4311-bc72-fcdf78a5c4d6-host-run-netns\") pod \"ovnkube-node-8dtlh\" (UID: \"195ecf89-13ae-4311-bc72-fcdf78a5c4d6\") " pod="openshift-ovn-kubernetes/ovnkube-node-8dtlh" Jul 23 19:19:36.771055 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:19:36.771027 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/195ecf89-13ae-4311-bc72-fcdf78a5c4d6-run-openvswitch\") pod \"ovnkube-node-8dtlh\" (UID: \"195ecf89-13ae-4311-bc72-fcdf78a5c4d6\") " pod="openshift-ovn-kubernetes/ovnkube-node-8dtlh" Jul 23 19:19:36.771055 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:19:36.771054 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/195ecf89-13ae-4311-bc72-fcdf78a5c4d6-host-slash\") pod \"ovnkube-node-8dtlh\" (UID: \"195ecf89-13ae-4311-bc72-fcdf78a5c4d6\") " pod="openshift-ovn-kubernetes/ovnkube-node-8dtlh" Jul 23 19:19:36.771340 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:19:36.771078 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/195ecf89-13ae-4311-bc72-fcdf78a5c4d6-etc-openvswitch\") pod \"ovnkube-node-8dtlh\" (UID: \"195ecf89-13ae-4311-bc72-fcdf78a5c4d6\") " pod="openshift-ovn-kubernetes/ovnkube-node-8dtlh" Jul 23 19:19:36.771340 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:19:36.771084 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/195ecf89-13ae-4311-bc72-fcdf78a5c4d6-host-run-netns\") pod \"ovnkube-node-8dtlh\" (UID: \"195ecf89-13ae-4311-bc72-fcdf78a5c4d6\") " pod="openshift-ovn-kubernetes/ovnkube-node-8dtlh" Jul 23 19:19:36.771340 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:19:36.771118 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/e83347a0-7f9a-4a7a-8a71-b662b464cd6b-socket-dir\") pod \"aws-ebs-csi-driver-node-l9rz7\" (UID: \"e83347a0-7f9a-4a7a-8a71-b662b464cd6b\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-l9rz7" Jul 23 19:19:36.771340 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:19:36.771126 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/195ecf89-13ae-4311-bc72-fcdf78a5c4d6-run-openvswitch\") pod \"ovnkube-node-8dtlh\" (UID: \"195ecf89-13ae-4311-bc72-fcdf78a5c4d6\") " pod="openshift-ovn-kubernetes/ovnkube-node-8dtlh" Jul 23 19:19:36.771340 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:19:36.771142 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xf6x4\" (UniqueName: \"kubernetes.io/projected/e83347a0-7f9a-4a7a-8a71-b662b464cd6b-kube-api-access-xf6x4\") pod \"aws-ebs-csi-driver-node-l9rz7\" (UID: \"e83347a0-7f9a-4a7a-8a71-b662b464cd6b\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-l9rz7" Jul 23 19:19:36.771340 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:19:36.771168 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/40f386ec-607d-496f-b65b-03a1fd847b1f-original-pull-secret\") pod \"global-pull-secret-syncer-rtgbq\" (UID: \"40f386ec-607d-496f-b65b-03a1fd847b1f\") " pod="kube-system/global-pull-secret-syncer-rtgbq" Jul 23 19:19:36.771340 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:19:36.771187 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/195ecf89-13ae-4311-bc72-fcdf78a5c4d6-etc-openvswitch\") pod \"ovnkube-node-8dtlh\" (UID: \"195ecf89-13ae-4311-bc72-fcdf78a5c4d6\") " pod="openshift-ovn-kubernetes/ovnkube-node-8dtlh" Jul 23 19:19:36.771340 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:19:36.771192 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/195ecf89-13ae-4311-bc72-fcdf78a5c4d6-log-socket\") pod \"ovnkube-node-8dtlh\" (UID: \"195ecf89-13ae-4311-bc72-fcdf78a5c4d6\") " pod="openshift-ovn-kubernetes/ovnkube-node-8dtlh" Jul 23 19:19:36.771340 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:19:36.771216 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/dd466782-4d8e-42e2-83c2-ed5f1b48567b-host\") pod \"node-ca-9wwn6\" (UID: \"dd466782-4d8e-42e2-83c2-ed5f1b48567b\") " pod="openshift-image-registry/node-ca-9wwn6" Jul 23 19:19:36.771340 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:19:36.771227 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/195ecf89-13ae-4311-bc72-fcdf78a5c4d6-host-slash\") pod \"ovnkube-node-8dtlh\" (UID: \"195ecf89-13ae-4311-bc72-fcdf78a5c4d6\") " pod="openshift-ovn-kubernetes/ovnkube-node-8dtlh" Jul 23 19:19:36.771340 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:19:36.771242 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/e83347a0-7f9a-4a7a-8a71-b662b464cd6b-device-dir\") pod \"aws-ebs-csi-driver-node-l9rz7\" (UID: \"e83347a0-7f9a-4a7a-8a71-b662b464cd6b\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-l9rz7" Jul 23 19:19:36.771340 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:19:36.771265 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/40f386ec-607d-496f-b65b-03a1fd847b1f-dbus\") pod \"global-pull-secret-syncer-rtgbq\" (UID: \"40f386ec-607d-496f-b65b-03a1fd847b1f\") " pod="kube-system/global-pull-secret-syncer-rtgbq" Jul 23 19:19:36.771340 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:19:36.771268 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/195ecf89-13ae-4311-bc72-fcdf78a5c4d6-log-socket\") pod \"ovnkube-node-8dtlh\" (UID: \"195ecf89-13ae-4311-bc72-fcdf78a5c4d6\") " pod="openshift-ovn-kubernetes/ovnkube-node-8dtlh" Jul 23 19:19:36.771340 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:19:36.771288 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/195ecf89-13ae-4311-bc72-fcdf78a5c4d6-run-ovn\") pod \"ovnkube-node-8dtlh\" (UID: \"195ecf89-13ae-4311-bc72-fcdf78a5c4d6\") " pod="openshift-ovn-kubernetes/ovnkube-node-8dtlh" Jul 23 19:19:36.771340 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:19:36.771331 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/195ecf89-13ae-4311-bc72-fcdf78a5c4d6-ovnkube-config\") pod \"ovnkube-node-8dtlh\" (UID: \"195ecf89-13ae-4311-bc72-fcdf78a5c4d6\") " pod="openshift-ovn-kubernetes/ovnkube-node-8dtlh" Jul 23 19:19:36.772093 ip-10-0-129-2 kubenswrapper[2564]: E0723 19:19:36.771352 2564 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 23 19:19:36.772093 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:19:36.771357 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/e83347a0-7f9a-4a7a-8a71-b662b464cd6b-kubelet-dir\") pod \"aws-ebs-csi-driver-node-l9rz7\" (UID: \"e83347a0-7f9a-4a7a-8a71-b662b464cd6b\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-l9rz7" Jul 23 19:19:36.772093 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:19:36.771400 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/e83347a0-7f9a-4a7a-8a71-b662b464cd6b-registration-dir\") pod \"aws-ebs-csi-driver-node-l9rz7\" (UID: \"e83347a0-7f9a-4a7a-8a71-b662b464cd6b\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-l9rz7" Jul 23 19:19:36.772093 ip-10-0-129-2 kubenswrapper[2564]: E0723 19:19:36.771430 2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/40f386ec-607d-496f-b65b-03a1fd847b1f-original-pull-secret podName:40f386ec-607d-496f-b65b-03a1fd847b1f nodeName:}" failed. No retries permitted until 2026-07-23 19:19:37.271404364 +0000 UTC m=+3.123180590 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/40f386ec-607d-496f-b65b-03a1fd847b1f-original-pull-secret") pod "global-pull-secret-syncer-rtgbq" (UID: "40f386ec-607d-496f-b65b-03a1fd847b1f") : object "kube-system"/"original-pull-secret" not registered Jul 23 19:19:36.772093 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:19:36.771461 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/e83347a0-7f9a-4a7a-8a71-b662b464cd6b-registration-dir\") pod \"aws-ebs-csi-driver-node-l9rz7\" (UID: \"e83347a0-7f9a-4a7a-8a71-b662b464cd6b\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-l9rz7" Jul 23 19:19:36.772093 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:19:36.771465 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/195ecf89-13ae-4311-bc72-fcdf78a5c4d6-ovn-node-metrics-cert\") pod \"ovnkube-node-8dtlh\" (UID: \"195ecf89-13ae-4311-bc72-fcdf78a5c4d6\") " pod="openshift-ovn-kubernetes/ovnkube-node-8dtlh" Jul 23 19:19:36.772093 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:19:36.771491 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/dd466782-4d8e-42e2-83c2-ed5f1b48567b-host\") pod \"node-ca-9wwn6\" (UID: \"dd466782-4d8e-42e2-83c2-ed5f1b48567b\") " pod="openshift-image-registry/node-ca-9wwn6" Jul 23 19:19:36.772093 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:19:36.771500 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/7951a728-910c-4631-b555-119d1ea12709-iptables-alerter-script\") pod \"iptables-alerter-gbqdl\" (UID: \"7951a728-910c-4631-b555-119d1ea12709\") " pod="openshift-network-operator/iptables-alerter-gbqdl" Jul 23 19:19:36.772093 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:19:36.771520 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/e83347a0-7f9a-4a7a-8a71-b662b464cd6b-device-dir\") pod \"aws-ebs-csi-driver-node-l9rz7\" (UID: \"e83347a0-7f9a-4a7a-8a71-b662b464cd6b\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-l9rz7" Jul 23 19:19:36.772093 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:19:36.771527 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/25ca5174-01a0-41fc-9293-77dea2d9ec1f-agent-certs\") pod \"konnectivity-agent-8rq79\" (UID: \"25ca5174-01a0-41fc-9293-77dea2d9ec1f\") " pod="kube-system/konnectivity-agent-8rq79" Jul 23 19:19:36.772093 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:19:36.771541 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/dd466782-4d8e-42e2-83c2-ed5f1b48567b-serviceca\") pod \"node-ca-9wwn6\" (UID: \"dd466782-4d8e-42e2-83c2-ed5f1b48567b\") " pod="openshift-image-registry/node-ca-9wwn6" Jul 23 19:19:36.772093 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:19:36.771551 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/195ecf89-13ae-4311-bc72-fcdf78a5c4d6-run-ovn\") pod \"ovnkube-node-8dtlh\" (UID: \"195ecf89-13ae-4311-bc72-fcdf78a5c4d6\") " pod="openshift-ovn-kubernetes/ovnkube-node-8dtlh" Jul 23 19:19:36.772093 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:19:36.771602 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/e83347a0-7f9a-4a7a-8a71-b662b464cd6b-sys-fs\") pod \"aws-ebs-csi-driver-node-l9rz7\" (UID: \"e83347a0-7f9a-4a7a-8a71-b662b464cd6b\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-l9rz7" Jul 23 19:19:36.772093 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:19:36.771657 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/40f386ec-607d-496f-b65b-03a1fd847b1f-dbus\") pod \"global-pull-secret-syncer-rtgbq\" (UID: \"40f386ec-607d-496f-b65b-03a1fd847b1f\") " pod="kube-system/global-pull-secret-syncer-rtgbq" Jul 23 19:19:36.772093 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:19:36.771553 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/e83347a0-7f9a-4a7a-8a71-b662b464cd6b-sys-fs\") pod \"aws-ebs-csi-driver-node-l9rz7\" (UID: \"e83347a0-7f9a-4a7a-8a71-b662b464cd6b\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-l9rz7" Jul 23 19:19:36.772093 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:19:36.771699 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/195ecf89-13ae-4311-bc72-fcdf78a5c4d6-systemd-units\") pod \"ovnkube-node-8dtlh\" (UID: \"195ecf89-13ae-4311-bc72-fcdf78a5c4d6\") " pod="openshift-ovn-kubernetes/ovnkube-node-8dtlh" Jul 23 19:19:36.772093 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:19:36.771499 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/e83347a0-7f9a-4a7a-8a71-b662b464cd6b-socket-dir\") pod \"aws-ebs-csi-driver-node-l9rz7\" (UID: \"e83347a0-7f9a-4a7a-8a71-b662b464cd6b\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-l9rz7" Jul 23 19:19:36.772690 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:19:36.771728 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/195ecf89-13ae-4311-bc72-fcdf78a5c4d6-var-lib-openvswitch\") pod \"ovnkube-node-8dtlh\" (UID: \"195ecf89-13ae-4311-bc72-fcdf78a5c4d6\") " pod="openshift-ovn-kubernetes/ovnkube-node-8dtlh" Jul 23 19:19:36.772690 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:19:36.771763 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hczmj\" (UniqueName: \"kubernetes.io/projected/7951a728-910c-4631-b555-119d1ea12709-kube-api-access-hczmj\") pod \"iptables-alerter-gbqdl\" (UID: \"7951a728-910c-4631-b555-119d1ea12709\") " pod="openshift-network-operator/iptables-alerter-gbqdl" Jul 23 19:19:36.772690 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:19:36.771790 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/25ca5174-01a0-41fc-9293-77dea2d9ec1f-konnectivity-ca\") pod \"konnectivity-agent-8rq79\" (UID: \"25ca5174-01a0-41fc-9293-77dea2d9ec1f\") " pod="kube-system/konnectivity-agent-8rq79" Jul 23 19:19:36.772690 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:19:36.771816 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/40f386ec-607d-496f-b65b-03a1fd847b1f-kubelet-config\") pod \"global-pull-secret-syncer-rtgbq\" (UID: \"40f386ec-607d-496f-b65b-03a1fd847b1f\") " pod="kube-system/global-pull-secret-syncer-rtgbq" Jul 23 19:19:36.772690 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:19:36.771802 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/e83347a0-7f9a-4a7a-8a71-b662b464cd6b-kubelet-dir\") pod \"aws-ebs-csi-driver-node-l9rz7\" (UID: \"e83347a0-7f9a-4a7a-8a71-b662b464cd6b\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-l9rz7" Jul 23 19:19:36.772690 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:19:36.771847 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/195ecf89-13ae-4311-bc72-fcdf78a5c4d6-host-run-ovn-kubernetes\") pod \"ovnkube-node-8dtlh\" (UID: \"195ecf89-13ae-4311-bc72-fcdf78a5c4d6\") " pod="openshift-ovn-kubernetes/ovnkube-node-8dtlh" Jul 23 19:19:36.772690 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:19:36.771874 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/195ecf89-13ae-4311-bc72-fcdf78a5c4d6-host-cni-bin\") pod \"ovnkube-node-8dtlh\" (UID: \"195ecf89-13ae-4311-bc72-fcdf78a5c4d6\") " pod="openshift-ovn-kubernetes/ovnkube-node-8dtlh" Jul 23 19:19:36.772690 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:19:36.771901 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/195ecf89-13ae-4311-bc72-fcdf78a5c4d6-ovnkube-config\") pod \"ovnkube-node-8dtlh\" (UID: \"195ecf89-13ae-4311-bc72-fcdf78a5c4d6\") " pod="openshift-ovn-kubernetes/ovnkube-node-8dtlh" Jul 23 19:19:36.772690 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:19:36.771899 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/195ecf89-13ae-4311-bc72-fcdf78a5c4d6-ovnkube-script-lib\") pod \"ovnkube-node-8dtlh\" (UID: \"195ecf89-13ae-4311-bc72-fcdf78a5c4d6\") " pod="openshift-ovn-kubernetes/ovnkube-node-8dtlh" Jul 23 19:19:36.772690 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:19:36.771942 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/195ecf89-13ae-4311-bc72-fcdf78a5c4d6-host-kubelet\") pod \"ovnkube-node-8dtlh\" (UID: \"195ecf89-13ae-4311-bc72-fcdf78a5c4d6\") " pod="openshift-ovn-kubernetes/ovnkube-node-8dtlh" Jul 23 19:19:36.772690 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:19:36.771964 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/195ecf89-13ae-4311-bc72-fcdf78a5c4d6-host-cni-netd\") pod \"ovnkube-node-8dtlh\" (UID: \"195ecf89-13ae-4311-bc72-fcdf78a5c4d6\") " pod="openshift-ovn-kubernetes/ovnkube-node-8dtlh" Jul 23 19:19:36.772690 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:19:36.771979 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/195ecf89-13ae-4311-bc72-fcdf78a5c4d6-env-overrides\") pod \"ovnkube-node-8dtlh\" (UID: \"195ecf89-13ae-4311-bc72-fcdf78a5c4d6\") " pod="openshift-ovn-kubernetes/ovnkube-node-8dtlh" Jul 23 19:19:36.772690 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:19:36.771997 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/7951a728-910c-4631-b555-119d1ea12709-host-slash\") pod \"iptables-alerter-gbqdl\" (UID: \"7951a728-910c-4631-b555-119d1ea12709\") " pod="openshift-network-operator/iptables-alerter-gbqdl" Jul 23 19:19:36.772690 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:19:36.772019 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ldzrw\" (UniqueName: \"kubernetes.io/projected/dd466782-4d8e-42e2-83c2-ed5f1b48567b-kube-api-access-ldzrw\") pod \"node-ca-9wwn6\" (UID: \"dd466782-4d8e-42e2-83c2-ed5f1b48567b\") " pod="openshift-image-registry/node-ca-9wwn6" Jul 23 19:19:36.772690 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:19:36.772041 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/195ecf89-13ae-4311-bc72-fcdf78a5c4d6-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-8dtlh\" (UID: \"195ecf89-13ae-4311-bc72-fcdf78a5c4d6\") " pod="openshift-ovn-kubernetes/ovnkube-node-8dtlh" Jul 23 19:19:36.772690 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:19:36.772098 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/7951a728-910c-4631-b555-119d1ea12709-iptables-alerter-script\") pod \"iptables-alerter-gbqdl\" (UID: \"7951a728-910c-4631-b555-119d1ea12709\") " pod="openshift-network-operator/iptables-alerter-gbqdl" Jul 23 19:19:36.772690 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:19:36.772123 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/7951a728-910c-4631-b555-119d1ea12709-host-slash\") pod \"iptables-alerter-gbqdl\" (UID: \"7951a728-910c-4631-b555-119d1ea12709\") " pod="openshift-network-operator/iptables-alerter-gbqdl" Jul 23 19:19:36.773328 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:19:36.772099 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/195ecf89-13ae-4311-bc72-fcdf78a5c4d6-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-8dtlh\" (UID: \"195ecf89-13ae-4311-bc72-fcdf78a5c4d6\") " pod="openshift-ovn-kubernetes/ovnkube-node-8dtlh" Jul 23 19:19:36.773328 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:19:36.772164 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/195ecf89-13ae-4311-bc72-fcdf78a5c4d6-host-kubelet\") pod \"ovnkube-node-8dtlh\" (UID: \"195ecf89-13ae-4311-bc72-fcdf78a5c4d6\") " pod="openshift-ovn-kubernetes/ovnkube-node-8dtlh" Jul 23 19:19:36.773328 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:19:36.772205 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/195ecf89-13ae-4311-bc72-fcdf78a5c4d6-host-cni-netd\") pod \"ovnkube-node-8dtlh\" (UID: \"195ecf89-13ae-4311-bc72-fcdf78a5c4d6\") " pod="openshift-ovn-kubernetes/ovnkube-node-8dtlh" Jul 23 19:19:36.773328 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:19:36.772322 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/195ecf89-13ae-4311-bc72-fcdf78a5c4d6-systemd-units\") pod \"ovnkube-node-8dtlh\" (UID: \"195ecf89-13ae-4311-bc72-fcdf78a5c4d6\") " pod="openshift-ovn-kubernetes/ovnkube-node-8dtlh" Jul 23 19:19:36.773328 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:19:36.772346 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/195ecf89-13ae-4311-bc72-fcdf78a5c4d6-var-lib-openvswitch\") pod \"ovnkube-node-8dtlh\" (UID: \"195ecf89-13ae-4311-bc72-fcdf78a5c4d6\") " pod="openshift-ovn-kubernetes/ovnkube-node-8dtlh" Jul 23 19:19:36.773328 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:19:36.771187 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/e83347a0-7f9a-4a7a-8a71-b662b464cd6b-etc-selinux\") pod \"aws-ebs-csi-driver-node-l9rz7\" (UID: \"e83347a0-7f9a-4a7a-8a71-b662b464cd6b\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-l9rz7" Jul 23 19:19:36.773328 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:19:36.772379 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/195ecf89-13ae-4311-bc72-fcdf78a5c4d6-ovnkube-script-lib\") pod \"ovnkube-node-8dtlh\" (UID: \"195ecf89-13ae-4311-bc72-fcdf78a5c4d6\") " pod="openshift-ovn-kubernetes/ovnkube-node-8dtlh" Jul 23 19:19:36.773328 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:19:36.772464 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/195ecf89-13ae-4311-bc72-fcdf78a5c4d6-host-run-ovn-kubernetes\") pod \"ovnkube-node-8dtlh\" (UID: \"195ecf89-13ae-4311-bc72-fcdf78a5c4d6\") " pod="openshift-ovn-kubernetes/ovnkube-node-8dtlh" Jul 23 19:19:36.773328 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:19:36.772509 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/195ecf89-13ae-4311-bc72-fcdf78a5c4d6-env-overrides\") pod \"ovnkube-node-8dtlh\" (UID: \"195ecf89-13ae-4311-bc72-fcdf78a5c4d6\") " pod="openshift-ovn-kubernetes/ovnkube-node-8dtlh" Jul 23 19:19:36.773328 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:19:36.772513 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/195ecf89-13ae-4311-bc72-fcdf78a5c4d6-host-cni-bin\") pod \"ovnkube-node-8dtlh\" (UID: \"195ecf89-13ae-4311-bc72-fcdf78a5c4d6\") " pod="openshift-ovn-kubernetes/ovnkube-node-8dtlh" Jul 23 19:19:36.773328 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:19:36.772570 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/40f386ec-607d-496f-b65b-03a1fd847b1f-kubelet-config\") pod \"global-pull-secret-syncer-rtgbq\" (UID: \"40f386ec-607d-496f-b65b-03a1fd847b1f\") " pod="kube-system/global-pull-secret-syncer-rtgbq" Jul 23 19:19:36.773328 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:19:36.773044 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/25ca5174-01a0-41fc-9293-77dea2d9ec1f-konnectivity-ca\") pod \"konnectivity-agent-8rq79\" (UID: \"25ca5174-01a0-41fc-9293-77dea2d9ec1f\") " pod="kube-system/konnectivity-agent-8rq79" Jul 23 19:19:36.774654 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:19:36.774634 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/195ecf89-13ae-4311-bc72-fcdf78a5c4d6-ovn-node-metrics-cert\") pod \"ovnkube-node-8dtlh\" (UID: \"195ecf89-13ae-4311-bc72-fcdf78a5c4d6\") " pod="openshift-ovn-kubernetes/ovnkube-node-8dtlh" Jul 23 19:19:36.774862 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:19:36.774841 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/25ca5174-01a0-41fc-9293-77dea2d9ec1f-agent-certs\") pod \"konnectivity-agent-8rq79\" (UID: \"25ca5174-01a0-41fc-9293-77dea2d9ec1f\") " pod="kube-system/konnectivity-agent-8rq79" Jul 23 19:19:36.781545 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:19:36.781517 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xf6x4\" (UniqueName: \"kubernetes.io/projected/e83347a0-7f9a-4a7a-8a71-b662b464cd6b-kube-api-access-xf6x4\") pod \"aws-ebs-csi-driver-node-l9rz7\" (UID: \"e83347a0-7f9a-4a7a-8a71-b662b464cd6b\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-l9rz7" Jul 23 19:19:36.781971 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:19:36.781948 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ldzrw\" (UniqueName: \"kubernetes.io/projected/dd466782-4d8e-42e2-83c2-ed5f1b48567b-kube-api-access-ldzrw\") pod \"node-ca-9wwn6\" (UID: \"dd466782-4d8e-42e2-83c2-ed5f1b48567b\") " pod="openshift-image-registry/node-ca-9wwn6" Jul 23 19:19:36.782112 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:19:36.782091 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-fc8lq\" (UniqueName: \"kubernetes.io/projected/195ecf89-13ae-4311-bc72-fcdf78a5c4d6-kube-api-access-fc8lq\") pod \"ovnkube-node-8dtlh\" (UID: \"195ecf89-13ae-4311-bc72-fcdf78a5c4d6\") " pod="openshift-ovn-kubernetes/ovnkube-node-8dtlh" Jul 23 19:19:36.782467 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:19:36.782448 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-hczmj\" (UniqueName: \"kubernetes.io/projected/7951a728-910c-4631-b555-119d1ea12709-kube-api-access-hczmj\") pod \"iptables-alerter-gbqdl\" (UID: \"7951a728-910c-4631-b555-119d1ea12709\") " pod="openshift-network-operator/iptables-alerter-gbqdl" Jul 23 19:19:36.860244 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:19:36.860220 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-8967b" Jul 23 19:19:36.870979 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:19:36.870959 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-sr96w" Jul 23 19:19:36.880848 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:19:36.880830 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-74q7d" Jul 23 19:19:36.886747 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:19:36.886727 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-6k8wr" Jul 23 19:19:36.895659 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:19:36.895632 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-gbqdl" Jul 23 19:19:36.902661 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:19:36.902637 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-8dtlh" Jul 23 19:19:36.913552 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:19:36.913530 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-8rq79" Jul 23 19:19:36.920454 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:19:36.920432 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-l9rz7" Jul 23 19:19:36.926251 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:19:36.926226 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-9wwn6" Jul 23 19:19:37.175560 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:19:37.175532 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/576f76ed-730c-49c2-a33e-1dc11370cad4-metrics-certs\") pod \"network-metrics-daemon-qhrz5\" (UID: \"576f76ed-730c-49c2-a33e-1dc11370cad4\") " pod="openshift-multus/network-metrics-daemon-qhrz5" Jul 23 19:19:37.175774 ip-10-0-129-2 kubenswrapper[2564]: E0723 19:19:37.175704 2564 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 23 19:19:37.175774 ip-10-0-129-2 kubenswrapper[2564]: E0723 19:19:37.175770 2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/576f76ed-730c-49c2-a33e-1dc11370cad4-metrics-certs podName:576f76ed-730c-49c2-a33e-1dc11370cad4 nodeName:}" failed. No retries permitted until 2026-07-23 19:19:38.175749054 +0000 UTC m=+4.027525299 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/576f76ed-730c-49c2-a33e-1dc11370cad4-metrics-certs") pod "network-metrics-daemon-qhrz5" (UID: "576f76ed-730c-49c2-a33e-1dc11370cad4") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 23 19:19:37.276036 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:19:37.276012 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cq8ll\" (UniqueName: \"kubernetes.io/projected/8706aab3-bba6-4fff-b8bc-8890d14daa9c-kube-api-access-cq8ll\") pod \"network-check-target-vxngp\" (UID: \"8706aab3-bba6-4fff-b8bc-8890d14daa9c\") " pod="openshift-network-diagnostics/network-check-target-vxngp" Jul 23 19:19:37.276160 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:19:37.276046 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/40f386ec-607d-496f-b65b-03a1fd847b1f-original-pull-secret\") pod \"global-pull-secret-syncer-rtgbq\" (UID: \"40f386ec-607d-496f-b65b-03a1fd847b1f\") " pod="kube-system/global-pull-secret-syncer-rtgbq" Jul 23 19:19:37.276160 ip-10-0-129-2 kubenswrapper[2564]: E0723 19:19:37.276155 2564 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 23 19:19:37.276222 ip-10-0-129-2 kubenswrapper[2564]: E0723 19:19:37.276169 2564 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 23 19:19:37.276222 ip-10-0-129-2 kubenswrapper[2564]: E0723 19:19:37.276191 2564 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 23 19:19:37.276222 ip-10-0-129-2 kubenswrapper[2564]: E0723 19:19:37.276202 2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/40f386ec-607d-496f-b65b-03a1fd847b1f-original-pull-secret podName:40f386ec-607d-496f-b65b-03a1fd847b1f nodeName:}" failed. No retries permitted until 2026-07-23 19:19:38.276190091 +0000 UTC m=+4.127966318 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/40f386ec-607d-496f-b65b-03a1fd847b1f-original-pull-secret") pod "global-pull-secret-syncer-rtgbq" (UID: "40f386ec-607d-496f-b65b-03a1fd847b1f") : object "kube-system"/"original-pull-secret" not registered Jul 23 19:19:37.276222 ip-10-0-129-2 kubenswrapper[2564]: E0723 19:19:37.276203 2564 projected.go:196] Error preparing data for projected volume kube-api-access-cq8ll for pod openshift-network-diagnostics/network-check-target-vxngp: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 23 19:19:37.276349 ip-10-0-129-2 kubenswrapper[2564]: E0723 19:19:37.276246 2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/8706aab3-bba6-4fff-b8bc-8890d14daa9c-kube-api-access-cq8ll podName:8706aab3-bba6-4fff-b8bc-8890d14daa9c nodeName:}" failed. No retries permitted until 2026-07-23 19:19:38.276233891 +0000 UTC m=+4.128010117 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-api-access-cq8ll" (UniqueName: "kubernetes.io/projected/8706aab3-bba6-4fff-b8bc-8890d14daa9c-kube-api-access-cq8ll") pod "network-check-target-vxngp" (UID: "8706aab3-bba6-4fff-b8bc-8890d14daa9c") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 23 19:19:37.602576 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:19:37.602471 2564 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-07-22 19:14:37 +0000 UTC" deadline="2028-01-07 12:12:19.544558056 +0000 UTC" Jul 23 19:19:37.602576 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:19:37.602531 2564 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="12784h52m41.942029749s" Jul 23 19:19:37.669457 ip-10-0-129-2 kubenswrapper[2564]: W0723 19:19:37.669432 2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod23313c02_67bf_498f_a294_e16cdc7d4517.slice/crio-17540de1cfb98738a706457298702c40f94207e600db13f749882847e0b2d073 WatchSource:0}: Error finding container 17540de1cfb98738a706457298702c40f94207e600db13f749882847e0b2d073: Status 404 returned error can't find the container with id 17540de1cfb98738a706457298702c40f94207e600db13f749882847e0b2d073 Jul 23 19:19:37.670943 ip-10-0-129-2 kubenswrapper[2564]: W0723 19:19:37.670919 2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7951a728_910c_4631_b555_119d1ea12709.slice/crio-93b43477493eccce4f885c0a6d0e68953e2de0a337dcc66d9df9fda87ffd6fe2 WatchSource:0}: Error finding container 93b43477493eccce4f885c0a6d0e68953e2de0a337dcc66d9df9fda87ffd6fe2: Status 404 returned error can't find the container with id 93b43477493eccce4f885c0a6d0e68953e2de0a337dcc66d9df9fda87ffd6fe2 Jul 23 19:19:37.671701 ip-10-0-129-2 kubenswrapper[2564]: W0723 19:19:37.671678 2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode83347a0_7f9a_4a7a_8a71_b662b464cd6b.slice/crio-e168f8fcc612ffc2d76ec5491a8a5263b6160e106192e1046a920ade455c8784 WatchSource:0}: Error finding container e168f8fcc612ffc2d76ec5491a8a5263b6160e106192e1046a920ade455c8784: Status 404 returned error can't find the container with id e168f8fcc612ffc2d76ec5491a8a5263b6160e106192e1046a920ade455c8784 Jul 23 19:19:37.677178 ip-10-0-129-2 kubenswrapper[2564]: W0723 19:19:37.676631 2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7c4060f3_0a15_4661_803f_fa6a845fc1ac.slice/crio-4b64281246b8d93202b7c5396b3efd3bb9aca9ed4febc8a2c93c7db7f0b8eaa7 WatchSource:0}: Error finding container 4b64281246b8d93202b7c5396b3efd3bb9aca9ed4febc8a2c93c7db7f0b8eaa7: Status 404 returned error can't find the container with id 4b64281246b8d93202b7c5396b3efd3bb9aca9ed4febc8a2c93c7db7f0b8eaa7 Jul 23 19:19:37.698488 ip-10-0-129-2 kubenswrapper[2564]: W0723 19:19:37.698459 2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod67269f87_53d8_4f3a_b396_4891d9326c38.slice/crio-3b56230d6c134285bf6326e3b347776bd8fae85353d50e9dc239916e76883497 WatchSource:0}: Error finding container 3b56230d6c134285bf6326e3b347776bd8fae85353d50e9dc239916e76883497: Status 404 returned error can't find the container with id 3b56230d6c134285bf6326e3b347776bd8fae85353d50e9dc239916e76883497 Jul 23 19:19:37.700258 ip-10-0-129-2 kubenswrapper[2564]: W0723 19:19:37.700237 2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddd466782_4d8e_42e2_83c2_ed5f1b48567b.slice/crio-f0a950eff867bf08f68fbff790d7bedcd56525b20833b25fcf582fe8476df3fc WatchSource:0}: Error finding container f0a950eff867bf08f68fbff790d7bedcd56525b20833b25fcf582fe8476df3fc: Status 404 returned error can't find the container with id f0a950eff867bf08f68fbff790d7bedcd56525b20833b25fcf582fe8476df3fc Jul 23 19:19:37.700985 ip-10-0-129-2 kubenswrapper[2564]: W0723 19:19:37.700961 2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod25ca5174_01a0_41fc_9293_77dea2d9ec1f.slice/crio-543e2c18da789d152e7165451e266c9c7eb6912f13a13c9b3394721832863df5 WatchSource:0}: Error finding container 543e2c18da789d152e7165451e266c9c7eb6912f13a13c9b3394721832863df5: Status 404 returned error can't find the container with id 543e2c18da789d152e7165451e266c9c7eb6912f13a13c9b3394721832863df5 Jul 23 19:19:38.183439 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:19:38.182727 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/576f76ed-730c-49c2-a33e-1dc11370cad4-metrics-certs\") pod \"network-metrics-daemon-qhrz5\" (UID: \"576f76ed-730c-49c2-a33e-1dc11370cad4\") " pod="openshift-multus/network-metrics-daemon-qhrz5" Jul 23 19:19:38.183439 ip-10-0-129-2 kubenswrapper[2564]: E0723 19:19:38.182981 2564 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 23 19:19:38.183439 ip-10-0-129-2 kubenswrapper[2564]: E0723 19:19:38.183066 2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/576f76ed-730c-49c2-a33e-1dc11370cad4-metrics-certs podName:576f76ed-730c-49c2-a33e-1dc11370cad4 nodeName:}" failed. No retries permitted until 2026-07-23 19:19:40.183043022 +0000 UTC m=+6.034819264 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/576f76ed-730c-49c2-a33e-1dc11370cad4-metrics-certs") pod "network-metrics-daemon-qhrz5" (UID: "576f76ed-730c-49c2-a33e-1dc11370cad4") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 23 19:19:38.284152 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:19:38.284118 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cq8ll\" (UniqueName: \"kubernetes.io/projected/8706aab3-bba6-4fff-b8bc-8890d14daa9c-kube-api-access-cq8ll\") pod \"network-check-target-vxngp\" (UID: \"8706aab3-bba6-4fff-b8bc-8890d14daa9c\") " pod="openshift-network-diagnostics/network-check-target-vxngp" Jul 23 19:19:38.284303 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:19:38.284168 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/40f386ec-607d-496f-b65b-03a1fd847b1f-original-pull-secret\") pod \"global-pull-secret-syncer-rtgbq\" (UID: \"40f386ec-607d-496f-b65b-03a1fd847b1f\") " pod="kube-system/global-pull-secret-syncer-rtgbq" Jul 23 19:19:38.284396 ip-10-0-129-2 kubenswrapper[2564]: E0723 19:19:38.284307 2564 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 23 19:19:38.284396 ip-10-0-129-2 kubenswrapper[2564]: E0723 19:19:38.284305 2564 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 23 19:19:38.284396 ip-10-0-129-2 kubenswrapper[2564]: E0723 19:19:38.284352 2564 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 23 19:19:38.284396 ip-10-0-129-2 kubenswrapper[2564]: E0723 19:19:38.284365 2564 projected.go:196] Error preparing data for projected volume kube-api-access-cq8ll for pod openshift-network-diagnostics/network-check-target-vxngp: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 23 19:19:38.284396 ip-10-0-129-2 kubenswrapper[2564]: E0723 19:19:38.284375 2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/40f386ec-607d-496f-b65b-03a1fd847b1f-original-pull-secret podName:40f386ec-607d-496f-b65b-03a1fd847b1f nodeName:}" failed. No retries permitted until 2026-07-23 19:19:40.28434751 +0000 UTC m=+6.136123756 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/40f386ec-607d-496f-b65b-03a1fd847b1f-original-pull-secret") pod "global-pull-secret-syncer-rtgbq" (UID: "40f386ec-607d-496f-b65b-03a1fd847b1f") : object "kube-system"/"original-pull-secret" not registered Jul 23 19:19:38.284664 ip-10-0-129-2 kubenswrapper[2564]: E0723 19:19:38.284416 2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/8706aab3-bba6-4fff-b8bc-8890d14daa9c-kube-api-access-cq8ll podName:8706aab3-bba6-4fff-b8bc-8890d14daa9c nodeName:}" failed. No retries permitted until 2026-07-23 19:19:40.284398243 +0000 UTC m=+6.136174472 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-api-access-cq8ll" (UniqueName: "kubernetes.io/projected/8706aab3-bba6-4fff-b8bc-8890d14daa9c-kube-api-access-cq8ll") pod "network-check-target-vxngp" (UID: "8706aab3-bba6-4fff-b8bc-8890d14daa9c") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 23 19:19:38.603216 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:19:38.603129 2564 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-07-22 19:14:37 +0000 UTC" deadline="2028-02-16 21:52:04.333263833 +0000 UTC" Jul 23 19:19:38.603216 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:19:38.603167 2564 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="13754h32m25.730101909s" Jul 23 19:19:38.652430 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:19:38.652357 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-vxngp" Jul 23 19:19:38.652601 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:19:38.652357 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-rtgbq" Jul 23 19:19:38.652601 ip-10-0-129-2 kubenswrapper[2564]: E0723 19:19:38.652501 2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-vxngp" podUID="8706aab3-bba6-4fff-b8bc-8890d14daa9c" Jul 23 19:19:38.652601 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:19:38.652567 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-qhrz5" Jul 23 19:19:38.652782 ip-10-0-129-2 kubenswrapper[2564]: E0723 19:19:38.652703 2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-rtgbq" podUID="40f386ec-607d-496f-b65b-03a1fd847b1f" Jul 23 19:19:38.652835 ip-10-0-129-2 kubenswrapper[2564]: E0723 19:19:38.652791 2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-qhrz5" podUID="576f76ed-730c-49c2-a33e-1dc11370cad4" Jul 23 19:19:38.689059 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:19:38.689022 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-8dtlh" event={"ID":"195ecf89-13ae-4311-bc72-fcdf78a5c4d6","Type":"ContainerStarted","Data":"d008e9bf2125013af61097f2a0bb7c821569c890fb2ac9df6b07917cc4ee24f5"} Jul 23 19:19:38.693854 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:19:38.693718 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-6k8wr" event={"ID":"23313c02-67bf-498f-a294-e16cdc7d4517","Type":"ContainerStarted","Data":"17540de1cfb98738a706457298702c40f94207e600db13f749882847e0b2d073"} Jul 23 19:19:38.699102 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:19:38.699073 2564 generic.go:356] "Generic (PLEG): container finished" podID="ee6c09f813ef3a4bd239061ba78fb650" containerID="01b387019738b53694b9906492e077b36e750b89a243c769d13d6dbfeaec71ad" exitCode=0 Jul 23 19:19:38.699219 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:19:38.699153 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-2.ec2.internal" event={"ID":"ee6c09f813ef3a4bd239061ba78fb650","Type":"ContainerDied","Data":"01b387019738b53694b9906492e077b36e750b89a243c769d13d6dbfeaec71ad"} Jul 23 19:19:38.709852 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:19:38.709743 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-8rq79" event={"ID":"25ca5174-01a0-41fc-9293-77dea2d9ec1f","Type":"ContainerStarted","Data":"543e2c18da789d152e7165451e266c9c7eb6912f13a13c9b3394721832863df5"} Jul 23 19:19:38.713955 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:19:38.713886 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-9wwn6" event={"ID":"dd466782-4d8e-42e2-83c2-ed5f1b48567b","Type":"ContainerStarted","Data":"f0a950eff867bf08f68fbff790d7bedcd56525b20833b25fcf582fe8476df3fc"} Jul 23 19:19:38.724783 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:19:38.724743 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-sr96w" event={"ID":"7c4060f3-0a15-4661-803f-fa6a845fc1ac","Type":"ContainerStarted","Data":"4b64281246b8d93202b7c5396b3efd3bb9aca9ed4febc8a2c93c7db7f0b8eaa7"} Jul 23 19:19:38.740125 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:19:38.740097 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-74q7d" event={"ID":"3557bbdf-7f88-487d-b674-28db6ea9d156","Type":"ContainerStarted","Data":"625ef8c9ca035cabafa8fd8893da1a57bad65427035aee4f8e5339dfaffd9073"} Jul 23 19:19:38.764414 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:19:38.763443 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-gbqdl" event={"ID":"7951a728-910c-4631-b555-119d1ea12709","Type":"ContainerStarted","Data":"93b43477493eccce4f885c0a6d0e68953e2de0a337dcc66d9df9fda87ffd6fe2"} Jul 23 19:19:38.772006 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:19:38.771971 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-129-2.ec2.internal" event={"ID":"f9c41c89c718c39ff54e57e416ba8933","Type":"ContainerStarted","Data":"14e313cea84bd2b53b0ae0ea67838965eda01b1ea9959a959de4f4ce62739ea9"} Jul 23 19:19:38.790059 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:19:38.790036 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-8967b" event={"ID":"67269f87-53d8-4f3a-b396-4891d9326c38","Type":"ContainerStarted","Data":"3b56230d6c134285bf6326e3b347776bd8fae85353d50e9dc239916e76883497"} Jul 23 19:19:38.799668 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:19:38.799605 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-l9rz7" event={"ID":"e83347a0-7f9a-4a7a-8a71-b662b464cd6b","Type":"ContainerStarted","Data":"e168f8fcc612ffc2d76ec5491a8a5263b6160e106192e1046a920ade455c8784"} Jul 23 19:19:39.814805 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:19:39.814767 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-2.ec2.internal" event={"ID":"ee6c09f813ef3a4bd239061ba78fb650","Type":"ContainerStarted","Data":"7295f4211aba41cc3246b3e618fa444d4f00f5ca660e252a4707c86af649155c"} Jul 23 19:19:39.830856 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:19:39.830564 2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/kube-apiserver-proxy-ip-10-0-129-2.ec2.internal" podStartSLOduration=3.830545055 podStartE2EDuration="3.830545055s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-23 19:19:36 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-23 19:19:38.788200625 +0000 UTC m=+4.639976877" watchObservedRunningTime="2026-07-23 19:19:39.830545055 +0000 UTC m=+5.682321307" Jul 23 19:19:40.201462 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:19:40.201429 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/576f76ed-730c-49c2-a33e-1dc11370cad4-metrics-certs\") pod \"network-metrics-daemon-qhrz5\" (UID: \"576f76ed-730c-49c2-a33e-1dc11370cad4\") " pod="openshift-multus/network-metrics-daemon-qhrz5" Jul 23 19:19:40.201646 ip-10-0-129-2 kubenswrapper[2564]: E0723 19:19:40.201574 2564 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 23 19:19:40.201646 ip-10-0-129-2 kubenswrapper[2564]: E0723 19:19:40.201644 2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/576f76ed-730c-49c2-a33e-1dc11370cad4-metrics-certs podName:576f76ed-730c-49c2-a33e-1dc11370cad4 nodeName:}" failed. No retries permitted until 2026-07-23 19:19:44.201625499 +0000 UTC m=+10.053401731 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/576f76ed-730c-49c2-a33e-1dc11370cad4-metrics-certs") pod "network-metrics-daemon-qhrz5" (UID: "576f76ed-730c-49c2-a33e-1dc11370cad4") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 23 19:19:40.301798 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:19:40.301761 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cq8ll\" (UniqueName: \"kubernetes.io/projected/8706aab3-bba6-4fff-b8bc-8890d14daa9c-kube-api-access-cq8ll\") pod \"network-check-target-vxngp\" (UID: \"8706aab3-bba6-4fff-b8bc-8890d14daa9c\") " pod="openshift-network-diagnostics/network-check-target-vxngp" Jul 23 19:19:40.301963 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:19:40.301812 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/40f386ec-607d-496f-b65b-03a1fd847b1f-original-pull-secret\") pod \"global-pull-secret-syncer-rtgbq\" (UID: \"40f386ec-607d-496f-b65b-03a1fd847b1f\") " pod="kube-system/global-pull-secret-syncer-rtgbq" Jul 23 19:19:40.302043 ip-10-0-129-2 kubenswrapper[2564]: E0723 19:19:40.301994 2564 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 23 19:19:40.302104 ip-10-0-129-2 kubenswrapper[2564]: E0723 19:19:40.302093 2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/40f386ec-607d-496f-b65b-03a1fd847b1f-original-pull-secret podName:40f386ec-607d-496f-b65b-03a1fd847b1f nodeName:}" failed. No retries permitted until 2026-07-23 19:19:44.302070981 +0000 UTC m=+10.153847212 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/40f386ec-607d-496f-b65b-03a1fd847b1f-original-pull-secret") pod "global-pull-secret-syncer-rtgbq" (UID: "40f386ec-607d-496f-b65b-03a1fd847b1f") : object "kube-system"/"original-pull-secret" not registered Jul 23 19:19:40.302614 ip-10-0-129-2 kubenswrapper[2564]: E0723 19:19:40.302591 2564 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 23 19:19:40.302711 ip-10-0-129-2 kubenswrapper[2564]: E0723 19:19:40.302620 2564 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 23 19:19:40.302711 ip-10-0-129-2 kubenswrapper[2564]: E0723 19:19:40.302635 2564 projected.go:196] Error preparing data for projected volume kube-api-access-cq8ll for pod openshift-network-diagnostics/network-check-target-vxngp: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 23 19:19:40.302711 ip-10-0-129-2 kubenswrapper[2564]: E0723 19:19:40.302687 2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/8706aab3-bba6-4fff-b8bc-8890d14daa9c-kube-api-access-cq8ll podName:8706aab3-bba6-4fff-b8bc-8890d14daa9c nodeName:}" failed. No retries permitted until 2026-07-23 19:19:44.302669658 +0000 UTC m=+10.154445891 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-api-access-cq8ll" (UniqueName: "kubernetes.io/projected/8706aab3-bba6-4fff-b8bc-8890d14daa9c-kube-api-access-cq8ll") pod "network-check-target-vxngp" (UID: "8706aab3-bba6-4fff-b8bc-8890d14daa9c") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 23 19:19:40.654462 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:19:40.653566 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-rtgbq" Jul 23 19:19:40.654462 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:19:40.653597 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-vxngp" Jul 23 19:19:40.654462 ip-10-0-129-2 kubenswrapper[2564]: E0723 19:19:40.653705 2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-rtgbq" podUID="40f386ec-607d-496f-b65b-03a1fd847b1f" Jul 23 19:19:40.654462 ip-10-0-129-2 kubenswrapper[2564]: E0723 19:19:40.654151 2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-vxngp" podUID="8706aab3-bba6-4fff-b8bc-8890d14daa9c" Jul 23 19:19:40.654462 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:19:40.654225 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-qhrz5" Jul 23 19:19:40.654462 ip-10-0-129-2 kubenswrapper[2564]: E0723 19:19:40.654314 2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-qhrz5" podUID="576f76ed-730c-49c2-a33e-1dc11370cad4" Jul 23 19:19:42.652457 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:19:42.652424 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-vxngp" Jul 23 19:19:42.653013 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:19:42.652472 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-qhrz5" Jul 23 19:19:42.653013 ip-10-0-129-2 kubenswrapper[2564]: E0723 19:19:42.652553 2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-vxngp" podUID="8706aab3-bba6-4fff-b8bc-8890d14daa9c" Jul 23 19:19:42.653013 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:19:42.652424 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-rtgbq" Jul 23 19:19:42.653013 ip-10-0-129-2 kubenswrapper[2564]: E0723 19:19:42.652707 2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-qhrz5" podUID="576f76ed-730c-49c2-a33e-1dc11370cad4" Jul 23 19:19:42.653013 ip-10-0-129-2 kubenswrapper[2564]: E0723 19:19:42.652793 2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-rtgbq" podUID="40f386ec-607d-496f-b65b-03a1fd847b1f" Jul 23 19:19:44.234077 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:19:44.234033 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/576f76ed-730c-49c2-a33e-1dc11370cad4-metrics-certs\") pod \"network-metrics-daemon-qhrz5\" (UID: \"576f76ed-730c-49c2-a33e-1dc11370cad4\") " pod="openshift-multus/network-metrics-daemon-qhrz5" Jul 23 19:19:44.234618 ip-10-0-129-2 kubenswrapper[2564]: E0723 19:19:44.234196 2564 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 23 19:19:44.234618 ip-10-0-129-2 kubenswrapper[2564]: E0723 19:19:44.234248 2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/576f76ed-730c-49c2-a33e-1dc11370cad4-metrics-certs podName:576f76ed-730c-49c2-a33e-1dc11370cad4 nodeName:}" failed. No retries permitted until 2026-07-23 19:19:52.23423085 +0000 UTC m=+18.086007096 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/576f76ed-730c-49c2-a33e-1dc11370cad4-metrics-certs") pod "network-metrics-daemon-qhrz5" (UID: "576f76ed-730c-49c2-a33e-1dc11370cad4") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 23 19:19:44.335021 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:19:44.334985 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cq8ll\" (UniqueName: \"kubernetes.io/projected/8706aab3-bba6-4fff-b8bc-8890d14daa9c-kube-api-access-cq8ll\") pod \"network-check-target-vxngp\" (UID: \"8706aab3-bba6-4fff-b8bc-8890d14daa9c\") " pod="openshift-network-diagnostics/network-check-target-vxngp" Jul 23 19:19:44.335183 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:19:44.335033 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/40f386ec-607d-496f-b65b-03a1fd847b1f-original-pull-secret\") pod \"global-pull-secret-syncer-rtgbq\" (UID: \"40f386ec-607d-496f-b65b-03a1fd847b1f\") " pod="kube-system/global-pull-secret-syncer-rtgbq" Jul 23 19:19:44.335183 ip-10-0-129-2 kubenswrapper[2564]: E0723 19:19:44.335138 2564 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 23 19:19:44.335183 ip-10-0-129-2 kubenswrapper[2564]: E0723 19:19:44.335149 2564 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 23 19:19:44.335183 ip-10-0-129-2 kubenswrapper[2564]: E0723 19:19:44.335167 2564 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 23 19:19:44.335183 ip-10-0-129-2 kubenswrapper[2564]: E0723 19:19:44.335179 2564 projected.go:196] Error preparing data for projected volume kube-api-access-cq8ll for pod openshift-network-diagnostics/network-check-target-vxngp: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 23 19:19:44.335443 ip-10-0-129-2 kubenswrapper[2564]: E0723 19:19:44.335206 2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/40f386ec-607d-496f-b65b-03a1fd847b1f-original-pull-secret podName:40f386ec-607d-496f-b65b-03a1fd847b1f nodeName:}" failed. No retries permitted until 2026-07-23 19:19:52.335187265 +0000 UTC m=+18.186963490 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/40f386ec-607d-496f-b65b-03a1fd847b1f-original-pull-secret") pod "global-pull-secret-syncer-rtgbq" (UID: "40f386ec-607d-496f-b65b-03a1fd847b1f") : object "kube-system"/"original-pull-secret" not registered Jul 23 19:19:44.335443 ip-10-0-129-2 kubenswrapper[2564]: E0723 19:19:44.335222 2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/8706aab3-bba6-4fff-b8bc-8890d14daa9c-kube-api-access-cq8ll podName:8706aab3-bba6-4fff-b8bc-8890d14daa9c nodeName:}" failed. No retries permitted until 2026-07-23 19:19:52.335213881 +0000 UTC m=+18.186990106 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-api-access-cq8ll" (UniqueName: "kubernetes.io/projected/8706aab3-bba6-4fff-b8bc-8890d14daa9c-kube-api-access-cq8ll") pod "network-check-target-vxngp" (UID: "8706aab3-bba6-4fff-b8bc-8890d14daa9c") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 23 19:19:44.652679 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:19:44.652595 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-vxngp" Jul 23 19:19:44.652832 ip-10-0-129-2 kubenswrapper[2564]: E0723 19:19:44.652690 2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-vxngp" podUID="8706aab3-bba6-4fff-b8bc-8890d14daa9c" Jul 23 19:19:44.652832 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:19:44.652774 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-rtgbq" Jul 23 19:19:44.652957 ip-10-0-129-2 kubenswrapper[2564]: E0723 19:19:44.652883 2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-rtgbq" podUID="40f386ec-607d-496f-b65b-03a1fd847b1f" Jul 23 19:19:44.652957 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:19:44.652932 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-qhrz5" Jul 23 19:19:44.653068 ip-10-0-129-2 kubenswrapper[2564]: E0723 19:19:44.653018 2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-qhrz5" podUID="576f76ed-730c-49c2-a33e-1dc11370cad4" Jul 23 19:19:46.657006 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:19:46.656230 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-qhrz5" Jul 23 19:19:46.657006 ip-10-0-129-2 kubenswrapper[2564]: E0723 19:19:46.656379 2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-qhrz5" podUID="576f76ed-730c-49c2-a33e-1dc11370cad4" Jul 23 19:19:46.657006 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:19:46.656760 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-vxngp" Jul 23 19:19:46.657006 ip-10-0-129-2 kubenswrapper[2564]: E0723 19:19:46.656845 2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-vxngp" podUID="8706aab3-bba6-4fff-b8bc-8890d14daa9c" Jul 23 19:19:46.657006 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:19:46.656906 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-rtgbq" Jul 23 19:19:46.657006 ip-10-0-129-2 kubenswrapper[2564]: E0723 19:19:46.656969 2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-rtgbq" podUID="40f386ec-607d-496f-b65b-03a1fd847b1f" Jul 23 19:19:48.654990 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:19:48.654505 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-qhrz5" Jul 23 19:19:48.654990 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:19:48.654537 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-vxngp" Jul 23 19:19:48.654990 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:19:48.654547 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-rtgbq" Jul 23 19:19:48.654990 ip-10-0-129-2 kubenswrapper[2564]: E0723 19:19:48.654644 2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-qhrz5" podUID="576f76ed-730c-49c2-a33e-1dc11370cad4" Jul 23 19:19:48.654990 ip-10-0-129-2 kubenswrapper[2564]: E0723 19:19:48.654761 2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-vxngp" podUID="8706aab3-bba6-4fff-b8bc-8890d14daa9c" Jul 23 19:19:48.654990 ip-10-0-129-2 kubenswrapper[2564]: E0723 19:19:48.654826 2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-rtgbq" podUID="40f386ec-607d-496f-b65b-03a1fd847b1f" Jul 23 19:19:50.652507 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:19:50.652474 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-vxngp" Jul 23 19:19:50.652507 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:19:50.652501 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-rtgbq" Jul 23 19:19:50.653001 ip-10-0-129-2 kubenswrapper[2564]: E0723 19:19:50.652591 2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-vxngp" podUID="8706aab3-bba6-4fff-b8bc-8890d14daa9c" Jul 23 19:19:50.653055 ip-10-0-129-2 kubenswrapper[2564]: E0723 19:19:50.653024 2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-rtgbq" podUID="40f386ec-607d-496f-b65b-03a1fd847b1f" Jul 23 19:19:50.653110 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:19:50.653079 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-qhrz5" Jul 23 19:19:50.653200 ip-10-0-129-2 kubenswrapper[2564]: E0723 19:19:50.653174 2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-qhrz5" podUID="576f76ed-730c-49c2-a33e-1dc11370cad4" Jul 23 19:19:52.296917 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:19:52.296892 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/576f76ed-730c-49c2-a33e-1dc11370cad4-metrics-certs\") pod \"network-metrics-daemon-qhrz5\" (UID: \"576f76ed-730c-49c2-a33e-1dc11370cad4\") " pod="openshift-multus/network-metrics-daemon-qhrz5" Jul 23 19:19:52.297154 ip-10-0-129-2 kubenswrapper[2564]: E0723 19:19:52.296985 2564 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 23 19:19:52.297154 ip-10-0-129-2 kubenswrapper[2564]: E0723 19:19:52.297033 2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/576f76ed-730c-49c2-a33e-1dc11370cad4-metrics-certs podName:576f76ed-730c-49c2-a33e-1dc11370cad4 nodeName:}" failed. No retries permitted until 2026-07-23 19:20:08.297018906 +0000 UTC m=+34.148795130 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/576f76ed-730c-49c2-a33e-1dc11370cad4-metrics-certs") pod "network-metrics-daemon-qhrz5" (UID: "576f76ed-730c-49c2-a33e-1dc11370cad4") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 23 19:19:52.397750 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:19:52.397728 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cq8ll\" (UniqueName: \"kubernetes.io/projected/8706aab3-bba6-4fff-b8bc-8890d14daa9c-kube-api-access-cq8ll\") pod \"network-check-target-vxngp\" (UID: \"8706aab3-bba6-4fff-b8bc-8890d14daa9c\") " pod="openshift-network-diagnostics/network-check-target-vxngp" Jul 23 19:19:52.397849 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:19:52.397755 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/40f386ec-607d-496f-b65b-03a1fd847b1f-original-pull-secret\") pod \"global-pull-secret-syncer-rtgbq\" (UID: \"40f386ec-607d-496f-b65b-03a1fd847b1f\") " pod="kube-system/global-pull-secret-syncer-rtgbq" Jul 23 19:19:52.397887 ip-10-0-129-2 kubenswrapper[2564]: E0723 19:19:52.397852 2564 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 23 19:19:52.397887 ip-10-0-129-2 kubenswrapper[2564]: E0723 19:19:52.397864 2564 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 23 19:19:52.397887 ip-10-0-129-2 kubenswrapper[2564]: E0723 19:19:52.397886 2564 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 23 19:19:52.397972 ip-10-0-129-2 kubenswrapper[2564]: E0723 19:19:52.397890 2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/40f386ec-607d-496f-b65b-03a1fd847b1f-original-pull-secret podName:40f386ec-607d-496f-b65b-03a1fd847b1f nodeName:}" failed. No retries permitted until 2026-07-23 19:20:08.397879908 +0000 UTC m=+34.249656133 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/40f386ec-607d-496f-b65b-03a1fd847b1f-original-pull-secret") pod "global-pull-secret-syncer-rtgbq" (UID: "40f386ec-607d-496f-b65b-03a1fd847b1f") : object "kube-system"/"original-pull-secret" not registered Jul 23 19:19:52.397972 ip-10-0-129-2 kubenswrapper[2564]: E0723 19:19:52.397895 2564 projected.go:196] Error preparing data for projected volume kube-api-access-cq8ll for pod openshift-network-diagnostics/network-check-target-vxngp: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 23 19:19:52.397972 ip-10-0-129-2 kubenswrapper[2564]: E0723 19:19:52.397930 2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/8706aab3-bba6-4fff-b8bc-8890d14daa9c-kube-api-access-cq8ll podName:8706aab3-bba6-4fff-b8bc-8890d14daa9c nodeName:}" failed. No retries permitted until 2026-07-23 19:20:08.397918836 +0000 UTC m=+34.249695061 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "kube-api-access-cq8ll" (UniqueName: "kubernetes.io/projected/8706aab3-bba6-4fff-b8bc-8890d14daa9c-kube-api-access-cq8ll") pod "network-check-target-vxngp" (UID: "8706aab3-bba6-4fff-b8bc-8890d14daa9c") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 23 19:19:52.652539 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:19:52.652509 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-vxngp" Jul 23 19:19:52.652663 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:19:52.652513 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-rtgbq" Jul 23 19:19:52.652663 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:19:52.652626 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-qhrz5" Jul 23 19:19:52.652663 ip-10-0-129-2 kubenswrapper[2564]: E0723 19:19:52.652622 2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-vxngp" podUID="8706aab3-bba6-4fff-b8bc-8890d14daa9c" Jul 23 19:19:52.652786 ip-10-0-129-2 kubenswrapper[2564]: E0723 19:19:52.652708 2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-rtgbq" podUID="40f386ec-607d-496f-b65b-03a1fd847b1f" Jul 23 19:19:52.652838 ip-10-0-129-2 kubenswrapper[2564]: E0723 19:19:52.652811 2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-qhrz5" podUID="576f76ed-730c-49c2-a33e-1dc11370cad4" Jul 23 19:19:54.652296 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:19:54.652270 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-vxngp" Jul 23 19:19:54.652742 ip-10-0-129-2 kubenswrapper[2564]: E0723 19:19:54.652388 2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-vxngp" podUID="8706aab3-bba6-4fff-b8bc-8890d14daa9c" Jul 23 19:19:54.652742 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:19:54.652475 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-qhrz5" Jul 23 19:19:54.652742 ip-10-0-129-2 kubenswrapper[2564]: E0723 19:19:54.652576 2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-qhrz5" podUID="576f76ed-730c-49c2-a33e-1dc11370cad4" Jul 23 19:19:54.652742 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:19:54.652625 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-rtgbq" Jul 23 19:19:54.652742 ip-10-0-129-2 kubenswrapper[2564]: E0723 19:19:54.652689 2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-rtgbq" podUID="40f386ec-607d-496f-b65b-03a1fd847b1f" Jul 23 19:19:56.652678 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:19:56.652648 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-vxngp" Jul 23 19:19:56.652961 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:19:56.652654 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-rtgbq" Jul 23 19:19:56.652961 ip-10-0-129-2 kubenswrapper[2564]: E0723 19:19:56.652742 2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-vxngp" podUID="8706aab3-bba6-4fff-b8bc-8890d14daa9c" Jul 23 19:19:56.652961 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:19:56.652657 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-qhrz5" Jul 23 19:19:56.652961 ip-10-0-129-2 kubenswrapper[2564]: E0723 19:19:56.652827 2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-rtgbq" podUID="40f386ec-607d-496f-b65b-03a1fd847b1f" Jul 23 19:19:56.652961 ip-10-0-129-2 kubenswrapper[2564]: E0723 19:19:56.652903 2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-qhrz5" podUID="576f76ed-730c-49c2-a33e-1dc11370cad4" Jul 23 19:19:56.848312 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:19:56.848288 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-8rq79" event={"ID":"25ca5174-01a0-41fc-9293-77dea2d9ec1f","Type":"ContainerStarted","Data":"16866eb9b6b14b06c0bda30d48ef1b6cdf8a9e11e6af30af0996baf1a43aa1f6"} Jul 23 19:19:57.853190 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:19:57.852834 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-l9rz7" event={"ID":"e83347a0-7f9a-4a7a-8a71-b662b464cd6b","Type":"ContainerStarted","Data":"19cd80c1ff009ed43e65473f640faa691009efee3203f59d3725150a9cca9fc1"} Jul 23 19:19:57.857987 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:19:57.856995 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-8dtlh_195ecf89-13ae-4311-bc72-fcdf78a5c4d6/ovn-acl-logging/0.log" Jul 23 19:19:57.857987 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:19:57.857395 2564 generic.go:356] "Generic (PLEG): container finished" podID="195ecf89-13ae-4311-bc72-fcdf78a5c4d6" containerID="0102f8ce1553c6c06eb8a582ba5791f26242b94ebe11ac21bc51d4c8b89c60c3" exitCode=1 Jul 23 19:19:57.857987 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:19:57.857462 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-8dtlh" event={"ID":"195ecf89-13ae-4311-bc72-fcdf78a5c4d6","Type":"ContainerStarted","Data":"5c937942670a35f07e06810863cd977f882485e8b4467dadb2f80d2162d4a21f"} Jul 23 19:19:57.857987 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:19:57.857535 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-8dtlh" event={"ID":"195ecf89-13ae-4311-bc72-fcdf78a5c4d6","Type":"ContainerStarted","Data":"3801a6679aa80c5d652b9b1385b5f8bb6cc571c1b57bea5a63a1b28de32d232e"} Jul 23 19:19:57.857987 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:19:57.857580 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-8dtlh" event={"ID":"195ecf89-13ae-4311-bc72-fcdf78a5c4d6","Type":"ContainerStarted","Data":"573478089aa6a9c4ef64264edbd2c0397100343397177ad20a7c222b32232d6f"} Jul 23 19:19:57.857987 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:19:57.857622 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-8dtlh" event={"ID":"195ecf89-13ae-4311-bc72-fcdf78a5c4d6","Type":"ContainerStarted","Data":"4e51729beaeba8e851c7d72202e50490510e35f5cd7eb39b987bdb4660e0e9da"} Jul 23 19:19:57.857987 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:19:57.857661 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-8dtlh" event={"ID":"195ecf89-13ae-4311-bc72-fcdf78a5c4d6","Type":"ContainerDied","Data":"0102f8ce1553c6c06eb8a582ba5791f26242b94ebe11ac21bc51d4c8b89c60c3"} Jul 23 19:19:57.857987 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:19:57.857703 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-8dtlh" event={"ID":"195ecf89-13ae-4311-bc72-fcdf78a5c4d6","Type":"ContainerStarted","Data":"b01d1f5802f45bf14addd60cdbf0c7575c3ae9c60bb523001b76094287b2a78d"} Jul 23 19:19:57.859897 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:19:57.859873 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-6k8wr" event={"ID":"23313c02-67bf-498f-a294-e16cdc7d4517","Type":"ContainerStarted","Data":"869ac7fa0b24d0c0532a6bdfae7dfd83661f021a2e63fc67e9e4944c03e267b5"} Jul 23 19:19:57.861335 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:19:57.861302 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-9wwn6" event={"ID":"dd466782-4d8e-42e2-83c2-ed5f1b48567b","Type":"ContainerStarted","Data":"f23ad8b0e62d4d0bc65962e185fca3e4cd69d15342fd37febebebbba937e6ae5"} Jul 23 19:19:57.862731 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:19:57.862712 2564 generic.go:356] "Generic (PLEG): container finished" podID="7c4060f3-0a15-4661-803f-fa6a845fc1ac" containerID="8447aa9363fcd44eade481cf972154e61bf54114b9475202ceb3cacce1ed84b1" exitCode=0 Jul 23 19:19:57.862816 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:19:57.862780 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-sr96w" event={"ID":"7c4060f3-0a15-4661-803f-fa6a845fc1ac","Type":"ContainerDied","Data":"8447aa9363fcd44eade481cf972154e61bf54114b9475202ceb3cacce1ed84b1"} Jul 23 19:19:57.864359 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:19:57.864077 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-74q7d" event={"ID":"3557bbdf-7f88-487d-b674-28db6ea9d156","Type":"ContainerStarted","Data":"68330876fb4eeda6fde5076d37c439611ae94491eddd3d4e5955474313c73b6c"} Jul 23 19:19:57.865644 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:19:57.865624 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-8967b" event={"ID":"67269f87-53d8-4f3a-b396-4891d9326c38","Type":"ContainerStarted","Data":"98444e2e8c379b25f40f10cecd030936aa920aa5ed1dd2fc51f0eaf4285ac4f8"} Jul 23 19:19:57.884671 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:19:57.884629 2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/node-resolver-6k8wr" podStartSLOduration=3.847554713 podStartE2EDuration="22.884618402s" totalImagesPullingTime="19.037063689s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-23 19:19:35 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-23 19:19:57.884481718 +0000 UTC m=+23.736257965" watchObservedRunningTime="2026-07-23 19:19:57.884618402 +0000 UTC m=+23.736394649" Jul 23 19:19:57.884751 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:19:57.884699 2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-2.ec2.internal" podStartSLOduration=21.884694372 podStartE2EDuration="21.884694372s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-23 19:19:36 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-23 19:19:39.83003078 +0000 UTC m=+5.681807070" watchObservedRunningTime="2026-07-23 19:19:57.884694372 +0000 UTC m=+23.736470621" Jul 23 19:19:57.899201 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:19:57.899171 2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/konnectivity-agent-8rq79" podStartSLOduration=8.068833156 podStartE2EDuration="22.899160516s" totalImagesPullingTime="14.83032736s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-23 19:19:35 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-23 19:19:57.898947363 +0000 UTC m=+23.750723609" watchObservedRunningTime="2026-07-23 19:19:57.899160516 +0000 UTC m=+23.750936763" Jul 23 19:19:57.912648 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:19:57.912615 2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/node-ca-9wwn6" podStartSLOduration=3.972022103 podStartE2EDuration="22.912607273s" totalImagesPullingTime="18.94058517s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-23 19:19:35 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-23 19:19:57.912462368 +0000 UTC m=+23.764238614" watchObservedRunningTime="2026-07-23 19:19:57.912607273 +0000 UTC m=+23.764383535" Jul 23 19:19:57.930020 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:19:57.929990 2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-8967b" podStartSLOduration=3.836275318 podStartE2EDuration="22.929979868s" totalImagesPullingTime="19.09370455s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-23 19:19:35 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-23 19:19:57.929616642 +0000 UTC m=+23.781392889" watchObservedRunningTime="2026-07-23 19:19:57.929979868 +0000 UTC m=+23.781756114" Jul 23 19:19:57.973181 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:19:57.973150 2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-node-tuning-operator/tuned-74q7d" podStartSLOduration=3.960106117 podStartE2EDuration="22.973140615s" totalImagesPullingTime="19.013034498s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-23 19:19:35 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-23 19:19:57.972887104 +0000 UTC m=+23.824663350" watchObservedRunningTime="2026-07-23 19:19:57.973140615 +0000 UTC m=+23.824916861" Jul 23 19:19:58.292358 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:19:58.292338 2564 plugin_watcher.go:194] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock" Jul 23 19:19:58.629055 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:19:58.628949 2564 reconciler.go:161] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-07-23T19:19:58.292354183Z","UUID":"936bb71f-ee63-4a3d-a650-31b43edc2f4d","Handler":null,"Name":"","Endpoint":""} Jul 23 19:19:58.630439 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:19:58.630414 2564 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0 Jul 23 19:19:58.630540 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:19:58.630452 2564 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock Jul 23 19:19:58.652357 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:19:58.652308 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-vxngp" Jul 23 19:19:58.652479 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:19:58.652417 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-qhrz5" Jul 23 19:19:58.652479 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:19:58.652346 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-rtgbq" Jul 23 19:19:58.652639 ip-10-0-129-2 kubenswrapper[2564]: E0723 19:19:58.652615 2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-vxngp" podUID="8706aab3-bba6-4fff-b8bc-8890d14daa9c" Jul 23 19:19:58.652694 ip-10-0-129-2 kubenswrapper[2564]: E0723 19:19:58.652673 2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-qhrz5" podUID="576f76ed-730c-49c2-a33e-1dc11370cad4" Jul 23 19:19:58.652747 ip-10-0-129-2 kubenswrapper[2564]: E0723 19:19:58.652734 2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-rtgbq" podUID="40f386ec-607d-496f-b65b-03a1fd847b1f" Jul 23 19:19:58.870782 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:19:58.870752 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-gbqdl" event={"ID":"7951a728-910c-4631-b555-119d1ea12709","Type":"ContainerStarted","Data":"35d4b1fea33c0713711d0bbef268fc842703f81c3b2b763b7e0e4ed90ae4dd9d"} Jul 23 19:19:58.872932 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:19:58.872908 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-l9rz7" event={"ID":"e83347a0-7f9a-4a7a-8a71-b662b464cd6b","Type":"ContainerStarted","Data":"9f4b2bb4777d1f931fbd52ffc856529bafd5419f8dfc35c4c69060ea19e16492"} Jul 23 19:19:58.887241 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:19:58.887175 2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-operator/iptables-alerter-gbqdl" podStartSLOduration=4.852167121 podStartE2EDuration="23.887162712s" totalImagesPullingTime="19.034995591s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-23 19:19:35 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-23 19:19:58.886936886 +0000 UTC m=+24.738713135" watchObservedRunningTime="2026-07-23 19:19:58.887162712 +0000 UTC m=+24.738938958" Jul 23 19:19:59.850875 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:19:59.850846 2564 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kube-system/konnectivity-agent-8rq79" Jul 23 19:19:59.851397 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:19:59.851378 2564 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kube-system/konnectivity-agent-8rq79" Jul 23 19:19:59.877109 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:19:59.877089 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-l9rz7" event={"ID":"e83347a0-7f9a-4a7a-8a71-b662b464cd6b","Type":"ContainerStarted","Data":"7f9561631a041a997278562a7de1d3e35a0738f831c11232277bc8e1b4a3ab22"} Jul 23 19:19:59.880430 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:19:59.880413 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-8dtlh_195ecf89-13ae-4311-bc72-fcdf78a5c4d6/ovn-acl-logging/0.log" Jul 23 19:19:59.880778 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:19:59.880758 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-8dtlh" event={"ID":"195ecf89-13ae-4311-bc72-fcdf78a5c4d6","Type":"ContainerStarted","Data":"48421a08dff1d8404b5f9462eb6f3af876f323e63afc64e26781b734ae115507"} Jul 23 19:19:59.881012 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:19:59.880993 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kube-system/konnectivity-agent-8rq79" Jul 23 19:19:59.881367 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:19:59.881354 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-8rq79" Jul 23 19:19:59.894724 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:19:59.894682 2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-l9rz7" podStartSLOduration=3.607403207 podStartE2EDuration="24.89467063s" totalImagesPullingTime="21.287267423s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-23 19:19:35 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-23 19:19:59.894067539 +0000 UTC m=+25.745843788" watchObservedRunningTime="2026-07-23 19:19:59.89467063 +0000 UTC m=+25.746446876" Jul 23 19:20:00.652066 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:20:00.651860 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-vxngp" Jul 23 19:20:00.652240 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:20:00.651860 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-rtgbq" Jul 23 19:20:00.652240 ip-10-0-129-2 kubenswrapper[2564]: E0723 19:20:00.652185 2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-vxngp" podUID="8706aab3-bba6-4fff-b8bc-8890d14daa9c" Jul 23 19:20:00.652383 ip-10-0-129-2 kubenswrapper[2564]: E0723 19:20:00.652268 2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-rtgbq" podUID="40f386ec-607d-496f-b65b-03a1fd847b1f" Jul 23 19:20:00.652383 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:20:00.651909 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-qhrz5" Jul 23 19:20:00.652492 ip-10-0-129-2 kubenswrapper[2564]: E0723 19:20:00.652401 2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-qhrz5" podUID="576f76ed-730c-49c2-a33e-1dc11370cad4" Jul 23 19:20:02.652454 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:20:02.652279 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-vxngp" Jul 23 19:20:02.653024 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:20:02.652282 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-rtgbq" Jul 23 19:20:02.653024 ip-10-0-129-2 kubenswrapper[2564]: E0723 19:20:02.652581 2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-vxngp" podUID="8706aab3-bba6-4fff-b8bc-8890d14daa9c" Jul 23 19:20:02.653024 ip-10-0-129-2 kubenswrapper[2564]: E0723 19:20:02.652595 2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-rtgbq" podUID="40f386ec-607d-496f-b65b-03a1fd847b1f" Jul 23 19:20:02.653024 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:20:02.652299 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-qhrz5" Jul 23 19:20:02.653024 ip-10-0-129-2 kubenswrapper[2564]: E0723 19:20:02.652673 2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-qhrz5" podUID="576f76ed-730c-49c2-a33e-1dc11370cad4" Jul 23 19:20:02.890288 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:20:02.890268 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-8dtlh_195ecf89-13ae-4311-bc72-fcdf78a5c4d6/ovn-acl-logging/0.log" Jul 23 19:20:02.890679 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:20:02.890660 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-8dtlh" event={"ID":"195ecf89-13ae-4311-bc72-fcdf78a5c4d6","Type":"ContainerStarted","Data":"8b0c1132bbe1b5f6b1c7f4eb47dd54403f5862825b9a17bebe0a85467e7668dd"} Jul 23 19:20:02.890981 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:20:02.890833 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-8dtlh" Jul 23 19:20:02.890981 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:20:02.890887 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-8dtlh" Jul 23 19:20:02.890981 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:20:02.890929 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-8dtlh" Jul 23 19:20:02.891146 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:20:02.891011 2564 scope.go:117] "RemoveContainer" containerID="0102f8ce1553c6c06eb8a582ba5791f26242b94ebe11ac21bc51d4c8b89c60c3" Jul 23 19:20:02.894151 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:20:02.892707 2564 generic.go:356] "Generic (PLEG): container finished" podID="7c4060f3-0a15-4661-803f-fa6a845fc1ac" containerID="4e3da4b426dceb07922105f1d192e0cf6b426098de01a9a2e7bd3bd95b1ca542" exitCode=0 Jul 23 19:20:02.894151 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:20:02.892738 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-sr96w" event={"ID":"7c4060f3-0a15-4661-803f-fa6a845fc1ac","Type":"ContainerDied","Data":"4e3da4b426dceb07922105f1d192e0cf6b426098de01a9a2e7bd3bd95b1ca542"} Jul 23 19:20:02.906810 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:20:02.906788 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-8dtlh" Jul 23 19:20:02.907441 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:20:02.907426 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-8dtlh" Jul 23 19:20:03.899475 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:20:03.899451 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-8dtlh_195ecf89-13ae-4311-bc72-fcdf78a5c4d6/ovn-acl-logging/0.log" Jul 23 19:20:03.900088 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:20:03.899856 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-8dtlh" event={"ID":"195ecf89-13ae-4311-bc72-fcdf78a5c4d6","Type":"ContainerStarted","Data":"c400b15016847ef010dc9800d51508385b815ea6996e4a73f8136de857da8024"} Jul 23 19:20:03.932935 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:20:03.932894 2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ovn-kubernetes/ovnkube-node-8dtlh" podStartSLOduration=9.578831284 podStartE2EDuration="28.932877732s" totalImagesPullingTime="19.354046448s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-23 19:19:35 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-23 19:20:03.932879066 +0000 UTC m=+29.784655314" watchObservedRunningTime="2026-07-23 19:20:03.932877732 +0000 UTC m=+29.784653960" Jul 23 19:20:04.288458 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:20:04.288437 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-rtgbq"] Jul 23 19:20:04.288562 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:20:04.288543 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-rtgbq" Jul 23 19:20:04.288639 ip-10-0-129-2 kubenswrapper[2564]: E0723 19:20:04.288619 2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-rtgbq" podUID="40f386ec-607d-496f-b65b-03a1fd847b1f" Jul 23 19:20:04.295652 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:20:04.295625 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-vxngp"] Jul 23 19:20:04.295756 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:20:04.295731 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-vxngp" Jul 23 19:20:04.295840 ip-10-0-129-2 kubenswrapper[2564]: E0723 19:20:04.295815 2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-vxngp" podUID="8706aab3-bba6-4fff-b8bc-8890d14daa9c" Jul 23 19:20:04.299362 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:20:04.299307 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-qhrz5"] Jul 23 19:20:04.299468 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:20:04.299457 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-qhrz5" Jul 23 19:20:04.299562 ip-10-0-129-2 kubenswrapper[2564]: E0723 19:20:04.299545 2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-qhrz5" podUID="576f76ed-730c-49c2-a33e-1dc11370cad4" Jul 23 19:20:04.904751 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:20:04.904722 2564 generic.go:356] "Generic (PLEG): container finished" podID="7c4060f3-0a15-4661-803f-fa6a845fc1ac" containerID="c68f0ae8e6f10962e9054a5295b909dc02da2fd1d6d7ee5ab3f3df02922e68b4" exitCode=0 Jul 23 19:20:04.905090 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:20:04.904806 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-sr96w" event={"ID":"7c4060f3-0a15-4661-803f-fa6a845fc1ac","Type":"ContainerDied","Data":"c68f0ae8e6f10962e9054a5295b909dc02da2fd1d6d7ee5ab3f3df02922e68b4"} Jul 23 19:20:05.651841 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:20:05.651816 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-rtgbq" Jul 23 19:20:05.651841 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:20:05.651839 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-qhrz5" Jul 23 19:20:05.652003 ip-10-0-129-2 kubenswrapper[2564]: E0723 19:20:05.651903 2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-rtgbq" podUID="40f386ec-607d-496f-b65b-03a1fd847b1f" Jul 23 19:20:05.652039 ip-10-0-129-2 kubenswrapper[2564]: E0723 19:20:05.652016 2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-qhrz5" podUID="576f76ed-730c-49c2-a33e-1dc11370cad4" Jul 23 19:20:06.652028 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:20:06.651869 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-vxngp" Jul 23 19:20:06.652400 ip-10-0-129-2 kubenswrapper[2564]: E0723 19:20:06.652076 2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-vxngp" podUID="8706aab3-bba6-4fff-b8bc-8890d14daa9c" Jul 23 19:20:06.910613 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:20:06.910564 2564 generic.go:356] "Generic (PLEG): container finished" podID="7c4060f3-0a15-4661-803f-fa6a845fc1ac" containerID="b33220c08f98315296e3bcf088e3cbae6dea04e9e6570dc7d6575070e84b1955" exitCode=0 Jul 23 19:20:06.910706 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:20:06.910616 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-sr96w" event={"ID":"7c4060f3-0a15-4661-803f-fa6a845fc1ac","Type":"ContainerDied","Data":"b33220c08f98315296e3bcf088e3cbae6dea04e9e6570dc7d6575070e84b1955"} Jul 23 19:20:07.652335 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:20:07.652297 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-qhrz5" Jul 23 19:20:07.652335 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:20:07.652311 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-rtgbq" Jul 23 19:20:07.652754 ip-10-0-129-2 kubenswrapper[2564]: E0723 19:20:07.652412 2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-qhrz5" podUID="576f76ed-730c-49c2-a33e-1dc11370cad4" Jul 23 19:20:07.652754 ip-10-0-129-2 kubenswrapper[2564]: E0723 19:20:07.652538 2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-rtgbq" podUID="40f386ec-607d-496f-b65b-03a1fd847b1f" Jul 23 19:20:08.314533 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:20:08.314497 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/576f76ed-730c-49c2-a33e-1dc11370cad4-metrics-certs\") pod \"network-metrics-daemon-qhrz5\" (UID: \"576f76ed-730c-49c2-a33e-1dc11370cad4\") " pod="openshift-multus/network-metrics-daemon-qhrz5" Jul 23 19:20:08.314703 ip-10-0-129-2 kubenswrapper[2564]: E0723 19:20:08.314610 2564 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 23 19:20:08.314703 ip-10-0-129-2 kubenswrapper[2564]: E0723 19:20:08.314679 2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/576f76ed-730c-49c2-a33e-1dc11370cad4-metrics-certs podName:576f76ed-730c-49c2-a33e-1dc11370cad4 nodeName:}" failed. No retries permitted until 2026-07-23 19:20:40.314661797 +0000 UTC m=+66.166438040 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/576f76ed-730c-49c2-a33e-1dc11370cad4-metrics-certs") pod "network-metrics-daemon-qhrz5" (UID: "576f76ed-730c-49c2-a33e-1dc11370cad4") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 23 19:20:08.415546 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:20:08.415521 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cq8ll\" (UniqueName: \"kubernetes.io/projected/8706aab3-bba6-4fff-b8bc-8890d14daa9c-kube-api-access-cq8ll\") pod \"network-check-target-vxngp\" (UID: \"8706aab3-bba6-4fff-b8bc-8890d14daa9c\") " pod="openshift-network-diagnostics/network-check-target-vxngp" Jul 23 19:20:08.415696 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:20:08.415558 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/40f386ec-607d-496f-b65b-03a1fd847b1f-original-pull-secret\") pod \"global-pull-secret-syncer-rtgbq\" (UID: \"40f386ec-607d-496f-b65b-03a1fd847b1f\") " pod="kube-system/global-pull-secret-syncer-rtgbq" Jul 23 19:20:08.415696 ip-10-0-129-2 kubenswrapper[2564]: E0723 19:20:08.415652 2564 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 23 19:20:08.415696 ip-10-0-129-2 kubenswrapper[2564]: E0723 19:20:08.415670 2564 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 23 19:20:08.415696 ip-10-0-129-2 kubenswrapper[2564]: E0723 19:20:08.415680 2564 projected.go:196] Error preparing data for projected volume kube-api-access-cq8ll for pod openshift-network-diagnostics/network-check-target-vxngp: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 23 19:20:08.415822 ip-10-0-129-2 kubenswrapper[2564]: E0723 19:20:08.415712 2564 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 23 19:20:08.415822 ip-10-0-129-2 kubenswrapper[2564]: E0723 19:20:08.415723 2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/8706aab3-bba6-4fff-b8bc-8890d14daa9c-kube-api-access-cq8ll podName:8706aab3-bba6-4fff-b8bc-8890d14daa9c nodeName:}" failed. No retries permitted until 2026-07-23 19:20:40.415710132 +0000 UTC m=+66.267486360 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "kube-api-access-cq8ll" (UniqueName: "kubernetes.io/projected/8706aab3-bba6-4fff-b8bc-8890d14daa9c-kube-api-access-cq8ll") pod "network-check-target-vxngp" (UID: "8706aab3-bba6-4fff-b8bc-8890d14daa9c") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 23 19:20:08.415822 ip-10-0-129-2 kubenswrapper[2564]: E0723 19:20:08.415797 2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/40f386ec-607d-496f-b65b-03a1fd847b1f-original-pull-secret podName:40f386ec-607d-496f-b65b-03a1fd847b1f nodeName:}" failed. No retries permitted until 2026-07-23 19:20:40.415779118 +0000 UTC m=+66.267555346 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/40f386ec-607d-496f-b65b-03a1fd847b1f-original-pull-secret") pod "global-pull-secret-syncer-rtgbq" (UID: "40f386ec-607d-496f-b65b-03a1fd847b1f") : object "kube-system"/"original-pull-secret" not registered Jul 23 19:20:08.652422 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:20:08.652205 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-vxngp" Jul 23 19:20:08.652422 ip-10-0-129-2 kubenswrapper[2564]: E0723 19:20:08.652338 2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-vxngp" podUID="8706aab3-bba6-4fff-b8bc-8890d14daa9c" Jul 23 19:20:09.651856 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:20:09.651826 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-qhrz5" Jul 23 19:20:09.651856 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:20:09.651842 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-rtgbq" Jul 23 19:20:09.652058 ip-10-0-129-2 kubenswrapper[2564]: E0723 19:20:09.651947 2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-qhrz5" podUID="576f76ed-730c-49c2-a33e-1dc11370cad4" Jul 23 19:20:09.652117 ip-10-0-129-2 kubenswrapper[2564]: E0723 19:20:09.652094 2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-rtgbq" podUID="40f386ec-607d-496f-b65b-03a1fd847b1f" Jul 23 19:20:10.490906 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:20:10.490876 2564 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-129-2.ec2.internal" event="NodeReady" Jul 23 19:20:10.491425 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:20:10.490989 2564 kubelet_node_status.go:489] "Fast updating node status as it just became ready" Jul 23 19:20:10.528042 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:20:10.528017 2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-image-registry/image-registry-59b578868-9sb7b"] Jul 23 19:20:10.531799 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:20:10.531778 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-59b578868-9sb7b" Jul 23 19:20:10.536158 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:20:10.536137 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"installation-pull-secrets\"" Jul 23 19:20:10.536591 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:20:10.536571 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-tls\"" Jul 23 19:20:10.536711 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:20:10.536691 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-private-configuration\"" Jul 23 19:20:10.536852 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:20:10.536837 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"registry-dockercfg-zfpwt\"" Jul 23 19:20:10.543679 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:20:10.543651 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"trusted-ca\"" Jul 23 19:20:10.550150 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:20:10.550130 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-59b578868-9sb7b"] Jul 23 19:20:10.550464 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:20:10.550447 2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-dns/dns-default-vzws2"] Jul 23 19:20:10.552880 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:20:10.552859 2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress-canary/ingress-canary-wxwk8"] Jul 23 19:20:10.553023 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:20:10.553003 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-vzws2" Jul 23 19:20:10.555223 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:20:10.555205 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-j4csw\"" Jul 23 19:20:10.555384 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:20:10.555366 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-wxwk8" Jul 23 19:20:10.555454 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:20:10.555411 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\"" Jul 23 19:20:10.555454 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:20:10.555438 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\"" Jul 23 19:20:10.558102 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:20:10.558084 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\"" Jul 23 19:20:10.558102 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:20:10.558097 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\"" Jul 23 19:20:10.558478 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:20:10.558455 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\"" Jul 23 19:20:10.558569 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:20:10.558511 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"default-dockercfg-n4phc\"" Jul 23 19:20:10.563987 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:20:10.563970 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-wxwk8"] Jul 23 19:20:10.569368 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:20:10.569349 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-vzws2"] Jul 23 19:20:10.632072 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:20:10.632039 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/234ec34c-c9e5-42a6-b63c-c40f402dedbf-installation-pull-secrets\") pod \"image-registry-59b578868-9sb7b\" (UID: \"234ec34c-c9e5-42a6-b63c-c40f402dedbf\") " pod="openshift-image-registry/image-registry-59b578868-9sb7b" Jul 23 19:20:10.632195 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:20:10.632088 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/234ec34c-c9e5-42a6-b63c-c40f402dedbf-registry-certificates\") pod \"image-registry-59b578868-9sb7b\" (UID: \"234ec34c-c9e5-42a6-b63c-c40f402dedbf\") " pod="openshift-image-registry/image-registry-59b578868-9sb7b" Jul 23 19:20:10.632195 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:20:10.632120 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ftzrt\" (UniqueName: \"kubernetes.io/projected/234ec34c-c9e5-42a6-b63c-c40f402dedbf-kube-api-access-ftzrt\") pod \"image-registry-59b578868-9sb7b\" (UID: \"234ec34c-c9e5-42a6-b63c-c40f402dedbf\") " pod="openshift-image-registry/image-registry-59b578868-9sb7b" Jul 23 19:20:10.632195 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:20:10.632143 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/234ec34c-c9e5-42a6-b63c-c40f402dedbf-image-registry-private-configuration\") pod \"image-registry-59b578868-9sb7b\" (UID: \"234ec34c-c9e5-42a6-b63c-c40f402dedbf\") " pod="openshift-image-registry/image-registry-59b578868-9sb7b" Jul 23 19:20:10.632378 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:20:10.632196 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/234ec34c-c9e5-42a6-b63c-c40f402dedbf-trusted-ca\") pod \"image-registry-59b578868-9sb7b\" (UID: \"234ec34c-c9e5-42a6-b63c-c40f402dedbf\") " pod="openshift-image-registry/image-registry-59b578868-9sb7b" Jul 23 19:20:10.632378 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:20:10.632284 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/234ec34c-c9e5-42a6-b63c-c40f402dedbf-registry-tls\") pod \"image-registry-59b578868-9sb7b\" (UID: \"234ec34c-c9e5-42a6-b63c-c40f402dedbf\") " pod="openshift-image-registry/image-registry-59b578868-9sb7b" Jul 23 19:20:10.632479 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:20:10.632377 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/234ec34c-c9e5-42a6-b63c-c40f402dedbf-bound-sa-token\") pod \"image-registry-59b578868-9sb7b\" (UID: \"234ec34c-c9e5-42a6-b63c-c40f402dedbf\") " pod="openshift-image-registry/image-registry-59b578868-9sb7b" Jul 23 19:20:10.632479 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:20:10.632400 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/234ec34c-c9e5-42a6-b63c-c40f402dedbf-ca-trust-extracted\") pod \"image-registry-59b578868-9sb7b\" (UID: \"234ec34c-c9e5-42a6-b63c-c40f402dedbf\") " pod="openshift-image-registry/image-registry-59b578868-9sb7b" Jul 23 19:20:10.652437 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:20:10.652412 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-vxngp" Jul 23 19:20:10.654748 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:20:10.654717 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\"" Jul 23 19:20:10.654881 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:20:10.654861 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-t9qhx\"" Jul 23 19:20:10.654932 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:20:10.654906 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\"" Jul 23 19:20:10.733640 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:20:10.733607 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kdx42\" (UniqueName: \"kubernetes.io/projected/380698de-d0b8-426a-9e9b-708c0e36e818-kube-api-access-kdx42\") pod \"dns-default-vzws2\" (UID: \"380698de-d0b8-426a-9e9b-708c0e36e818\") " pod="openshift-dns/dns-default-vzws2" Jul 23 19:20:10.733771 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:20:10.733655 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/234ec34c-c9e5-42a6-b63c-c40f402dedbf-bound-sa-token\") pod \"image-registry-59b578868-9sb7b\" (UID: \"234ec34c-c9e5-42a6-b63c-c40f402dedbf\") " pod="openshift-image-registry/image-registry-59b578868-9sb7b" Jul 23 19:20:10.733771 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:20:10.733681 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/380698de-d0b8-426a-9e9b-708c0e36e818-tmp-dir\") pod \"dns-default-vzws2\" (UID: \"380698de-d0b8-426a-9e9b-708c0e36e818\") " pod="openshift-dns/dns-default-vzws2" Jul 23 19:20:10.733771 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:20:10.733706 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/234ec34c-c9e5-42a6-b63c-c40f402dedbf-ca-trust-extracted\") pod \"image-registry-59b578868-9sb7b\" (UID: \"234ec34c-c9e5-42a6-b63c-c40f402dedbf\") " pod="openshift-image-registry/image-registry-59b578868-9sb7b" Jul 23 19:20:10.733771 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:20:10.733721 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/e6d7c065-91d9-461c-8c0f-684ad4ea70e6-cert\") pod \"ingress-canary-wxwk8\" (UID: \"e6d7c065-91d9-461c-8c0f-684ad4ea70e6\") " pod="openshift-ingress-canary/ingress-canary-wxwk8" Jul 23 19:20:10.733771 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:20:10.733752 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/234ec34c-c9e5-42a6-b63c-c40f402dedbf-installation-pull-secrets\") pod \"image-registry-59b578868-9sb7b\" (UID: \"234ec34c-c9e5-42a6-b63c-c40f402dedbf\") " pod="openshift-image-registry/image-registry-59b578868-9sb7b" Jul 23 19:20:10.734018 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:20:10.733998 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/234ec34c-c9e5-42a6-b63c-c40f402dedbf-registry-certificates\") pod \"image-registry-59b578868-9sb7b\" (UID: \"234ec34c-c9e5-42a6-b63c-c40f402dedbf\") " pod="openshift-image-registry/image-registry-59b578868-9sb7b" Jul 23 19:20:10.734069 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:20:10.734041 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/380698de-d0b8-426a-9e9b-708c0e36e818-metrics-tls\") pod \"dns-default-vzws2\" (UID: \"380698de-d0b8-426a-9e9b-708c0e36e818\") " pod="openshift-dns/dns-default-vzws2" Jul 23 19:20:10.734120 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:20:10.734076 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ftzrt\" (UniqueName: \"kubernetes.io/projected/234ec34c-c9e5-42a6-b63c-c40f402dedbf-kube-api-access-ftzrt\") pod \"image-registry-59b578868-9sb7b\" (UID: \"234ec34c-c9e5-42a6-b63c-c40f402dedbf\") " pod="openshift-image-registry/image-registry-59b578868-9sb7b" Jul 23 19:20:10.734120 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:20:10.734098 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/234ec34c-c9e5-42a6-b63c-c40f402dedbf-ca-trust-extracted\") pod \"image-registry-59b578868-9sb7b\" (UID: \"234ec34c-c9e5-42a6-b63c-c40f402dedbf\") " pod="openshift-image-registry/image-registry-59b578868-9sb7b" Jul 23 19:20:10.734120 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:20:10.734105 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/234ec34c-c9e5-42a6-b63c-c40f402dedbf-image-registry-private-configuration\") pod \"image-registry-59b578868-9sb7b\" (UID: \"234ec34c-c9e5-42a6-b63c-c40f402dedbf\") " pod="openshift-image-registry/image-registry-59b578868-9sb7b" Jul 23 19:20:10.734257 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:20:10.734131 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/234ec34c-c9e5-42a6-b63c-c40f402dedbf-trusted-ca\") pod \"image-registry-59b578868-9sb7b\" (UID: \"234ec34c-c9e5-42a6-b63c-c40f402dedbf\") " pod="openshift-image-registry/image-registry-59b578868-9sb7b" Jul 23 19:20:10.734399 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:20:10.734366 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/380698de-d0b8-426a-9e9b-708c0e36e818-config-volume\") pod \"dns-default-vzws2\" (UID: \"380698de-d0b8-426a-9e9b-708c0e36e818\") " pod="openshift-dns/dns-default-vzws2" Jul 23 19:20:10.734486 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:20:10.734437 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/234ec34c-c9e5-42a6-b63c-c40f402dedbf-registry-tls\") pod \"image-registry-59b578868-9sb7b\" (UID: \"234ec34c-c9e5-42a6-b63c-c40f402dedbf\") " pod="openshift-image-registry/image-registry-59b578868-9sb7b" Jul 23 19:20:10.734486 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:20:10.734468 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jhg4v\" (UniqueName: \"kubernetes.io/projected/e6d7c065-91d9-461c-8c0f-684ad4ea70e6-kube-api-access-jhg4v\") pod \"ingress-canary-wxwk8\" (UID: \"e6d7c065-91d9-461c-8c0f-684ad4ea70e6\") " pod="openshift-ingress-canary/ingress-canary-wxwk8" Jul 23 19:20:10.734585 ip-10-0-129-2 kubenswrapper[2564]: E0723 19:20:10.734542 2564 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Jul 23 19:20:10.734585 ip-10-0-129-2 kubenswrapper[2564]: E0723 19:20:10.734559 2564 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-59b578868-9sb7b: secret "image-registry-tls" not found Jul 23 19:20:10.734677 ip-10-0-129-2 kubenswrapper[2564]: E0723 19:20:10.734622 2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/234ec34c-c9e5-42a6-b63c-c40f402dedbf-registry-tls podName:234ec34c-c9e5-42a6-b63c-c40f402dedbf nodeName:}" failed. No retries permitted until 2026-07-23 19:20:11.234604015 +0000 UTC m=+37.086380241 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/234ec34c-c9e5-42a6-b63c-c40f402dedbf-registry-tls") pod "image-registry-59b578868-9sb7b" (UID: "234ec34c-c9e5-42a6-b63c-c40f402dedbf") : secret "image-registry-tls" not found Jul 23 19:20:10.734677 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:20:10.734624 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/234ec34c-c9e5-42a6-b63c-c40f402dedbf-registry-certificates\") pod \"image-registry-59b578868-9sb7b\" (UID: \"234ec34c-c9e5-42a6-b63c-c40f402dedbf\") " pod="openshift-image-registry/image-registry-59b578868-9sb7b" Jul 23 19:20:10.734996 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:20:10.734977 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/234ec34c-c9e5-42a6-b63c-c40f402dedbf-trusted-ca\") pod \"image-registry-59b578868-9sb7b\" (UID: \"234ec34c-c9e5-42a6-b63c-c40f402dedbf\") " pod="openshift-image-registry/image-registry-59b578868-9sb7b" Jul 23 19:20:10.738740 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:20:10.738687 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/234ec34c-c9e5-42a6-b63c-c40f402dedbf-installation-pull-secrets\") pod \"image-registry-59b578868-9sb7b\" (UID: \"234ec34c-c9e5-42a6-b63c-c40f402dedbf\") " pod="openshift-image-registry/image-registry-59b578868-9sb7b" Jul 23 19:20:10.738740 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:20:10.738705 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/234ec34c-c9e5-42a6-b63c-c40f402dedbf-image-registry-private-configuration\") pod \"image-registry-59b578868-9sb7b\" (UID: \"234ec34c-c9e5-42a6-b63c-c40f402dedbf\") " pod="openshift-image-registry/image-registry-59b578868-9sb7b" Jul 23 19:20:10.742428 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:20:10.742406 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/234ec34c-c9e5-42a6-b63c-c40f402dedbf-bound-sa-token\") pod \"image-registry-59b578868-9sb7b\" (UID: \"234ec34c-c9e5-42a6-b63c-c40f402dedbf\") " pod="openshift-image-registry/image-registry-59b578868-9sb7b" Jul 23 19:20:10.742804 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:20:10.742785 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ftzrt\" (UniqueName: \"kubernetes.io/projected/234ec34c-c9e5-42a6-b63c-c40f402dedbf-kube-api-access-ftzrt\") pod \"image-registry-59b578868-9sb7b\" (UID: \"234ec34c-c9e5-42a6-b63c-c40f402dedbf\") " pod="openshift-image-registry/image-registry-59b578868-9sb7b" Jul 23 19:20:10.835520 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:20:10.835457 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kdx42\" (UniqueName: \"kubernetes.io/projected/380698de-d0b8-426a-9e9b-708c0e36e818-kube-api-access-kdx42\") pod \"dns-default-vzws2\" (UID: \"380698de-d0b8-426a-9e9b-708c0e36e818\") " pod="openshift-dns/dns-default-vzws2" Jul 23 19:20:10.835520 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:20:10.835499 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/380698de-d0b8-426a-9e9b-708c0e36e818-tmp-dir\") pod \"dns-default-vzws2\" (UID: \"380698de-d0b8-426a-9e9b-708c0e36e818\") " pod="openshift-dns/dns-default-vzws2" Jul 23 19:20:10.835670 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:20:10.835532 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/e6d7c065-91d9-461c-8c0f-684ad4ea70e6-cert\") pod \"ingress-canary-wxwk8\" (UID: \"e6d7c065-91d9-461c-8c0f-684ad4ea70e6\") " pod="openshift-ingress-canary/ingress-canary-wxwk8" Jul 23 19:20:10.835670 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:20:10.835587 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/380698de-d0b8-426a-9e9b-708c0e36e818-metrics-tls\") pod \"dns-default-vzws2\" (UID: \"380698de-d0b8-426a-9e9b-708c0e36e818\") " pod="openshift-dns/dns-default-vzws2" Jul 23 19:20:10.835670 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:20:10.835624 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/380698de-d0b8-426a-9e9b-708c0e36e818-config-volume\") pod \"dns-default-vzws2\" (UID: \"380698de-d0b8-426a-9e9b-708c0e36e818\") " pod="openshift-dns/dns-default-vzws2" Jul 23 19:20:10.835670 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:20:10.835666 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jhg4v\" (UniqueName: \"kubernetes.io/projected/e6d7c065-91d9-461c-8c0f-684ad4ea70e6-kube-api-access-jhg4v\") pod \"ingress-canary-wxwk8\" (UID: \"e6d7c065-91d9-461c-8c0f-684ad4ea70e6\") " pod="openshift-ingress-canary/ingress-canary-wxwk8" Jul 23 19:20:10.836502 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:20:10.836481 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/380698de-d0b8-426a-9e9b-708c0e36e818-tmp-dir\") pod \"dns-default-vzws2\" (UID: \"380698de-d0b8-426a-9e9b-708c0e36e818\") " pod="openshift-dns/dns-default-vzws2" Jul 23 19:20:10.836869 ip-10-0-129-2 kubenswrapper[2564]: E0723 19:20:10.836847 2564 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jul 23 19:20:10.836975 ip-10-0-129-2 kubenswrapper[2564]: E0723 19:20:10.836955 2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/380698de-d0b8-426a-9e9b-708c0e36e818-metrics-tls podName:380698de-d0b8-426a-9e9b-708c0e36e818 nodeName:}" failed. No retries permitted until 2026-07-23 19:20:11.336939105 +0000 UTC m=+37.188715332 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/380698de-d0b8-426a-9e9b-708c0e36e818-metrics-tls") pod "dns-default-vzws2" (UID: "380698de-d0b8-426a-9e9b-708c0e36e818") : secret "dns-default-metrics-tls" not found Jul 23 19:20:10.837304 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:20:10.837280 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/380698de-d0b8-426a-9e9b-708c0e36e818-config-volume\") pod \"dns-default-vzws2\" (UID: \"380698de-d0b8-426a-9e9b-708c0e36e818\") " pod="openshift-dns/dns-default-vzws2" Jul 23 19:20:10.839116 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:20:10.839094 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/e6d7c065-91d9-461c-8c0f-684ad4ea70e6-cert\") pod \"ingress-canary-wxwk8\" (UID: \"e6d7c065-91d9-461c-8c0f-684ad4ea70e6\") " pod="openshift-ingress-canary/ingress-canary-wxwk8" Jul 23 19:20:10.845009 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:20:10.844987 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-kdx42\" (UniqueName: \"kubernetes.io/projected/380698de-d0b8-426a-9e9b-708c0e36e818-kube-api-access-kdx42\") pod \"dns-default-vzws2\" (UID: \"380698de-d0b8-426a-9e9b-708c0e36e818\") " pod="openshift-dns/dns-default-vzws2" Jul 23 19:20:10.845277 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:20:10.845261 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jhg4v\" (UniqueName: \"kubernetes.io/projected/e6d7c065-91d9-461c-8c0f-684ad4ea70e6-kube-api-access-jhg4v\") pod \"ingress-canary-wxwk8\" (UID: \"e6d7c065-91d9-461c-8c0f-684ad4ea70e6\") " pod="openshift-ingress-canary/ingress-canary-wxwk8" Jul 23 19:20:10.870334 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:20:10.870298 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-wxwk8" Jul 23 19:20:11.191698 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:20:11.191573 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-wxwk8"] Jul 23 19:20:11.194461 ip-10-0-129-2 kubenswrapper[2564]: W0723 19:20:11.194439 2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode6d7c065_91d9_461c_8c0f_684ad4ea70e6.slice/crio-a6b6c56ea7ffc41c365bdbf3a7543fb58c3a3e5afb9f9762750ed9ccfbc1ad00 WatchSource:0}: Error finding container a6b6c56ea7ffc41c365bdbf3a7543fb58c3a3e5afb9f9762750ed9ccfbc1ad00: Status 404 returned error can't find the container with id a6b6c56ea7ffc41c365bdbf3a7543fb58c3a3e5afb9f9762750ed9ccfbc1ad00 Jul 23 19:20:11.237917 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:20:11.237896 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/234ec34c-c9e5-42a6-b63c-c40f402dedbf-registry-tls\") pod \"image-registry-59b578868-9sb7b\" (UID: \"234ec34c-c9e5-42a6-b63c-c40f402dedbf\") " pod="openshift-image-registry/image-registry-59b578868-9sb7b" Jul 23 19:20:11.238043 ip-10-0-129-2 kubenswrapper[2564]: E0723 19:20:11.238029 2564 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Jul 23 19:20:11.238094 ip-10-0-129-2 kubenswrapper[2564]: E0723 19:20:11.238046 2564 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-59b578868-9sb7b: secret "image-registry-tls" not found Jul 23 19:20:11.238129 ip-10-0-129-2 kubenswrapper[2564]: E0723 19:20:11.238096 2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/234ec34c-c9e5-42a6-b63c-c40f402dedbf-registry-tls podName:234ec34c-c9e5-42a6-b63c-c40f402dedbf nodeName:}" failed. No retries permitted until 2026-07-23 19:20:12.23808016 +0000 UTC m=+38.089856408 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/234ec34c-c9e5-42a6-b63c-c40f402dedbf-registry-tls") pod "image-registry-59b578868-9sb7b" (UID: "234ec34c-c9e5-42a6-b63c-c40f402dedbf") : secret "image-registry-tls" not found Jul 23 19:20:11.338808 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:20:11.338784 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/380698de-d0b8-426a-9e9b-708c0e36e818-metrics-tls\") pod \"dns-default-vzws2\" (UID: \"380698de-d0b8-426a-9e9b-708c0e36e818\") " pod="openshift-dns/dns-default-vzws2" Jul 23 19:20:11.338902 ip-10-0-129-2 kubenswrapper[2564]: E0723 19:20:11.338895 2564 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jul 23 19:20:11.338949 ip-10-0-129-2 kubenswrapper[2564]: E0723 19:20:11.338943 2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/380698de-d0b8-426a-9e9b-708c0e36e818-metrics-tls podName:380698de-d0b8-426a-9e9b-708c0e36e818 nodeName:}" failed. No retries permitted until 2026-07-23 19:20:12.338930178 +0000 UTC m=+38.190706403 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/380698de-d0b8-426a-9e9b-708c0e36e818-metrics-tls") pod "dns-default-vzws2" (UID: "380698de-d0b8-426a-9e9b-708c0e36e818") : secret "dns-default-metrics-tls" not found Jul 23 19:20:11.652309 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:20:11.652286 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-qhrz5" Jul 23 19:20:11.652697 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:20:11.652463 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-rtgbq" Jul 23 19:20:11.654935 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:20:11.654920 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\"" Jul 23 19:20:11.655060 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:20:11.654992 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\"" Jul 23 19:20:11.655060 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:20:11.655026 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-k9tbr\"" Jul 23 19:20:11.926069 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:20:11.926043 2564 generic.go:356] "Generic (PLEG): container finished" podID="7c4060f3-0a15-4661-803f-fa6a845fc1ac" containerID="2dfa0960841a02e2dedc11d9ff6237b0b548b2525424c7fac913e3297c6773de" exitCode=0 Jul 23 19:20:11.926179 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:20:11.926113 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-sr96w" event={"ID":"7c4060f3-0a15-4661-803f-fa6a845fc1ac","Type":"ContainerDied","Data":"2dfa0960841a02e2dedc11d9ff6237b0b548b2525424c7fac913e3297c6773de"} Jul 23 19:20:11.927600 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:20:11.927579 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-wxwk8" event={"ID":"e6d7c065-91d9-461c-8c0f-684ad4ea70e6","Type":"ContainerStarted","Data":"a6b6c56ea7ffc41c365bdbf3a7543fb58c3a3e5afb9f9762750ed9ccfbc1ad00"} Jul 23 19:20:12.249739 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:20:12.247232 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/234ec34c-c9e5-42a6-b63c-c40f402dedbf-registry-tls\") pod \"image-registry-59b578868-9sb7b\" (UID: \"234ec34c-c9e5-42a6-b63c-c40f402dedbf\") " pod="openshift-image-registry/image-registry-59b578868-9sb7b" Jul 23 19:20:12.249739 ip-10-0-129-2 kubenswrapper[2564]: E0723 19:20:12.247356 2564 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Jul 23 19:20:12.249739 ip-10-0-129-2 kubenswrapper[2564]: E0723 19:20:12.247378 2564 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-59b578868-9sb7b: secret "image-registry-tls" not found Jul 23 19:20:12.249739 ip-10-0-129-2 kubenswrapper[2564]: E0723 19:20:12.247451 2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/234ec34c-c9e5-42a6-b63c-c40f402dedbf-registry-tls podName:234ec34c-c9e5-42a6-b63c-c40f402dedbf nodeName:}" failed. No retries permitted until 2026-07-23 19:20:14.247432584 +0000 UTC m=+40.099208811 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/234ec34c-c9e5-42a6-b63c-c40f402dedbf-registry-tls") pod "image-registry-59b578868-9sb7b" (UID: "234ec34c-c9e5-42a6-b63c-c40f402dedbf") : secret "image-registry-tls" not found Jul 23 19:20:12.347948 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:20:12.347922 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/380698de-d0b8-426a-9e9b-708c0e36e818-metrics-tls\") pod \"dns-default-vzws2\" (UID: \"380698de-d0b8-426a-9e9b-708c0e36e818\") " pod="openshift-dns/dns-default-vzws2" Jul 23 19:20:12.348101 ip-10-0-129-2 kubenswrapper[2564]: E0723 19:20:12.348085 2564 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jul 23 19:20:12.348170 ip-10-0-129-2 kubenswrapper[2564]: E0723 19:20:12.348157 2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/380698de-d0b8-426a-9e9b-708c0e36e818-metrics-tls podName:380698de-d0b8-426a-9e9b-708c0e36e818 nodeName:}" failed. No retries permitted until 2026-07-23 19:20:14.348137999 +0000 UTC m=+40.199914241 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/380698de-d0b8-426a-9e9b-708c0e36e818-metrics-tls") pod "dns-default-vzws2" (UID: "380698de-d0b8-426a-9e9b-708c0e36e818") : secret "dns-default-metrics-tls" not found Jul 23 19:20:12.932723 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:20:12.932684 2564 generic.go:356] "Generic (PLEG): container finished" podID="7c4060f3-0a15-4661-803f-fa6a845fc1ac" containerID="18910280ad4ffddaea1f34b5b9043b9a42b51443f9ae66fc0d3abc70b59a5a80" exitCode=0 Jul 23 19:20:12.933209 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:20:12.932746 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-sr96w" event={"ID":"7c4060f3-0a15-4661-803f-fa6a845fc1ac","Type":"ContainerDied","Data":"18910280ad4ffddaea1f34b5b9043b9a42b51443f9ae66fc0d3abc70b59a5a80"} Jul 23 19:20:13.939828 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:20:13.939630 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-sr96w" event={"ID":"7c4060f3-0a15-4661-803f-fa6a845fc1ac","Type":"ContainerStarted","Data":"1759450a3f262da5de3125f6ff23ce299a841412427be737c9626d42c7ed1c76"} Jul 23 19:20:13.940949 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:20:13.940928 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-wxwk8" event={"ID":"e6d7c065-91d9-461c-8c0f-684ad4ea70e6","Type":"ContainerStarted","Data":"b59cb609bfbc4ff5c753b0470f804f31ccd0c8d000320b0b7556882105d7015c"} Jul 23 19:20:13.964117 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:20:13.964074 2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-additional-cni-plugins-sr96w" podStartSLOduration=9.076647059 podStartE2EDuration="38.964059724s" totalImagesPullingTime="29.887412665s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-23 19:19:35 +0000 UTC" imagePullSessionsCount=5 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-23 19:20:13.963478784 +0000 UTC m=+39.815255031" watchObservedRunningTime="2026-07-23 19:20:13.964059724 +0000 UTC m=+39.815835971" Jul 23 19:20:13.982622 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:20:13.982552 2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress-canary/ingress-canary-wxwk8" podStartSLOduration=1.45088667 podStartE2EDuration="3.982539747s" totalImagesPullingTime="2.531653077s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-23 19:20:10 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-23 19:20:13.981932181 +0000 UTC m=+39.833708426" watchObservedRunningTime="2026-07-23 19:20:13.982539747 +0000 UTC m=+39.834316023" Jul 23 19:20:14.128059 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:20:14.128030 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8rq79_25ca5174-01a0-41fc-9293-77dea2d9ec1f/konnectivity-agent/0.log" limit=1024 Jul 23 19:20:14.228427 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:20:14.228396 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8rq79_25ca5174-01a0-41fc-9293-77dea2d9ec1f/konnectivity-agent/0.log" limit=1024 Jul 23 19:20:14.264465 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:20:14.264411 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/234ec34c-c9e5-42a6-b63c-c40f402dedbf-registry-tls\") pod \"image-registry-59b578868-9sb7b\" (UID: \"234ec34c-c9e5-42a6-b63c-c40f402dedbf\") " pod="openshift-image-registry/image-registry-59b578868-9sb7b" Jul 23 19:20:14.264549 ip-10-0-129-2 kubenswrapper[2564]: E0723 19:20:14.264517 2564 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Jul 23 19:20:14.264549 ip-10-0-129-2 kubenswrapper[2564]: E0723 19:20:14.264527 2564 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-59b578868-9sb7b: secret "image-registry-tls" not found Jul 23 19:20:14.264613 ip-10-0-129-2 kubenswrapper[2564]: E0723 19:20:14.264572 2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/234ec34c-c9e5-42a6-b63c-c40f402dedbf-registry-tls podName:234ec34c-c9e5-42a6-b63c-c40f402dedbf nodeName:}" failed. No retries permitted until 2026-07-23 19:20:18.264556904 +0000 UTC m=+44.116333130 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/234ec34c-c9e5-42a6-b63c-c40f402dedbf-registry-tls") pod "image-registry-59b578868-9sb7b" (UID: "234ec34c-c9e5-42a6-b63c-c40f402dedbf") : secret "image-registry-tls" not found Jul 23 19:20:14.365392 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:20:14.365368 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/380698de-d0b8-426a-9e9b-708c0e36e818-metrics-tls\") pod \"dns-default-vzws2\" (UID: \"380698de-d0b8-426a-9e9b-708c0e36e818\") " pod="openshift-dns/dns-default-vzws2" Jul 23 19:20:14.365486 ip-10-0-129-2 kubenswrapper[2564]: E0723 19:20:14.365449 2564 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jul 23 19:20:14.365524 ip-10-0-129-2 kubenswrapper[2564]: E0723 19:20:14.365500 2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/380698de-d0b8-426a-9e9b-708c0e36e818-metrics-tls podName:380698de-d0b8-426a-9e9b-708c0e36e818 nodeName:}" failed. No retries permitted until 2026-07-23 19:20:18.36548678 +0000 UTC m=+44.217263007 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/380698de-d0b8-426a-9e9b-708c0e36e818-metrics-tls") pod "dns-default-vzws2" (UID: "380698de-d0b8-426a-9e9b-708c0e36e818") : secret "dns-default-metrics-tls" not found Jul 23 19:20:14.373681 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:20:14.373661 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8rq79_25ca5174-01a0-41fc-9293-77dea2d9ec1f/konnectivity-agent/0.log" limit=1024 Jul 23 19:20:14.470838 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:20:14.470810 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8rq79_25ca5174-01a0-41fc-9293-77dea2d9ec1f/konnectivity-agent/0.log" limit=1024 Jul 23 19:20:18.292295 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:20:18.292265 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/234ec34c-c9e5-42a6-b63c-c40f402dedbf-registry-tls\") pod \"image-registry-59b578868-9sb7b\" (UID: \"234ec34c-c9e5-42a6-b63c-c40f402dedbf\") " pod="openshift-image-registry/image-registry-59b578868-9sb7b" Jul 23 19:20:18.292737 ip-10-0-129-2 kubenswrapper[2564]: E0723 19:20:18.292406 2564 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Jul 23 19:20:18.292737 ip-10-0-129-2 kubenswrapper[2564]: E0723 19:20:18.292420 2564 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-59b578868-9sb7b: secret "image-registry-tls" not found Jul 23 19:20:18.292737 ip-10-0-129-2 kubenswrapper[2564]: E0723 19:20:18.292465 2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/234ec34c-c9e5-42a6-b63c-c40f402dedbf-registry-tls podName:234ec34c-c9e5-42a6-b63c-c40f402dedbf nodeName:}" failed. No retries permitted until 2026-07-23 19:20:26.292452964 +0000 UTC m=+52.144229189 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/234ec34c-c9e5-42a6-b63c-c40f402dedbf-registry-tls") pod "image-registry-59b578868-9sb7b" (UID: "234ec34c-c9e5-42a6-b63c-c40f402dedbf") : secret "image-registry-tls" not found Jul 23 19:20:18.393110 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:20:18.393087 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/380698de-d0b8-426a-9e9b-708c0e36e818-metrics-tls\") pod \"dns-default-vzws2\" (UID: \"380698de-d0b8-426a-9e9b-708c0e36e818\") " pod="openshift-dns/dns-default-vzws2" Jul 23 19:20:18.393205 ip-10-0-129-2 kubenswrapper[2564]: E0723 19:20:18.393172 2564 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jul 23 19:20:18.393234 ip-10-0-129-2 kubenswrapper[2564]: E0723 19:20:18.393210 2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/380698de-d0b8-426a-9e9b-708c0e36e818-metrics-tls podName:380698de-d0b8-426a-9e9b-708c0e36e818 nodeName:}" failed. No retries permitted until 2026-07-23 19:20:26.393199698 +0000 UTC m=+52.244975922 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/380698de-d0b8-426a-9e9b-708c0e36e818-metrics-tls") pod "dns-default-vzws2" (UID: "380698de-d0b8-426a-9e9b-708c0e36e818") : secret "dns-default-metrics-tls" not found Jul 23 19:20:21.849766 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:20:21.849741 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8rq79_25ca5174-01a0-41fc-9293-77dea2d9ec1f/konnectivity-agent/0.log" limit=1024 Jul 23 19:20:21.973946 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:20:21.973928 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8rq79_25ca5174-01a0-41fc-9293-77dea2d9ec1f/konnectivity-agent/0.log" limit=1024 Jul 23 19:20:22.075420 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:20:22.075394 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8rq79_25ca5174-01a0-41fc-9293-77dea2d9ec1f/konnectivity-agent/0.log" limit=1024 Jul 23 19:20:22.176101 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:20:22.176082 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8rq79_25ca5174-01a0-41fc-9293-77dea2d9ec1f/konnectivity-agent/0.log" limit=1024 Jul 23 19:20:22.896927 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:20:22.896897 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8rq79_25ca5174-01a0-41fc-9293-77dea2d9ec1f/konnectivity-agent/0.log" limit=1024 Jul 23 19:20:22.982193 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:20:22.982162 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8rq79_25ca5174-01a0-41fc-9293-77dea2d9ec1f/konnectivity-agent/0.log" limit=1024 Jul 23 19:20:23.059876 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:20:23.059850 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8rq79_25ca5174-01a0-41fc-9293-77dea2d9ec1f/konnectivity-agent/0.log" limit=1024 Jul 23 19:20:23.159765 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:20:23.159707 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8rq79_25ca5174-01a0-41fc-9293-77dea2d9ec1f/konnectivity-agent/0.log" limit=1024 Jul 23 19:20:23.262612 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:20:23.262590 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8rq79_25ca5174-01a0-41fc-9293-77dea2d9ec1f/konnectivity-agent/0.log" limit=1024 Jul 23 19:20:23.359931 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:20:23.359900 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8rq79_25ca5174-01a0-41fc-9293-77dea2d9ec1f/konnectivity-agent/0.log" limit=1024 Jul 23 19:20:23.468599 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:20:23.468565 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8rq79_25ca5174-01a0-41fc-9293-77dea2d9ec1f/konnectivity-agent/0.log" limit=1024 Jul 23 19:20:24.122409 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:20:24.122384 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8rq79_25ca5174-01a0-41fc-9293-77dea2d9ec1f/konnectivity-agent/0.log" limit=1024 Jul 23 19:20:24.189472 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:20:24.189446 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8rq79_25ca5174-01a0-41fc-9293-77dea2d9ec1f/konnectivity-agent/0.log" limit=1024 Jul 23 19:20:24.574763 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:20:24.574746 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8rq79_25ca5174-01a0-41fc-9293-77dea2d9ec1f/konnectivity-agent/0.log" limit=1024 Jul 23 19:20:24.659391 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:20:24.659368 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8rq79_25ca5174-01a0-41fc-9293-77dea2d9ec1f/konnectivity-agent/0.log" limit=1024 Jul 23 19:20:25.332824 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:20:25.332800 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8rq79_25ca5174-01a0-41fc-9293-77dea2d9ec1f/konnectivity-agent/0.log" limit=1024 Jul 23 19:20:25.721490 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:20:25.721467 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8rq79_25ca5174-01a0-41fc-9293-77dea2d9ec1f/konnectivity-agent/0.log" limit=1024 Jul 23 19:20:25.782655 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:20:25.782633 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8rq79_25ca5174-01a0-41fc-9293-77dea2d9ec1f/konnectivity-agent/0.log" limit=1024 Jul 23 19:20:25.869069 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:20:25.869045 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8rq79_25ca5174-01a0-41fc-9293-77dea2d9ec1f/konnectivity-agent/0.log" limit=1024 Jul 23 19:20:26.350869 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:20:26.350849 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/234ec34c-c9e5-42a6-b63c-c40f402dedbf-registry-tls\") pod \"image-registry-59b578868-9sb7b\" (UID: \"234ec34c-c9e5-42a6-b63c-c40f402dedbf\") " pod="openshift-image-registry/image-registry-59b578868-9sb7b" Jul 23 19:20:26.351111 ip-10-0-129-2 kubenswrapper[2564]: E0723 19:20:26.350941 2564 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Jul 23 19:20:26.351111 ip-10-0-129-2 kubenswrapper[2564]: E0723 19:20:26.350951 2564 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-59b578868-9sb7b: secret "image-registry-tls" not found Jul 23 19:20:26.351111 ip-10-0-129-2 kubenswrapper[2564]: E0723 19:20:26.351010 2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/234ec34c-c9e5-42a6-b63c-c40f402dedbf-registry-tls podName:234ec34c-c9e5-42a6-b63c-c40f402dedbf nodeName:}" failed. No retries permitted until 2026-07-23 19:20:42.350998053 +0000 UTC m=+68.202774277 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/234ec34c-c9e5-42a6-b63c-c40f402dedbf-registry-tls") pod "image-registry-59b578868-9sb7b" (UID: "234ec34c-c9e5-42a6-b63c-c40f402dedbf") : secret "image-registry-tls" not found Jul 23 19:20:26.451964 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:20:26.451945 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/380698de-d0b8-426a-9e9b-708c0e36e818-metrics-tls\") pod \"dns-default-vzws2\" (UID: \"380698de-d0b8-426a-9e9b-708c0e36e818\") " pod="openshift-dns/dns-default-vzws2" Jul 23 19:20:26.452080 ip-10-0-129-2 kubenswrapper[2564]: E0723 19:20:26.452067 2564 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jul 23 19:20:26.452123 ip-10-0-129-2 kubenswrapper[2564]: E0723 19:20:26.452114 2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/380698de-d0b8-426a-9e9b-708c0e36e818-metrics-tls podName:380698de-d0b8-426a-9e9b-708c0e36e818 nodeName:}" failed. No retries permitted until 2026-07-23 19:20:42.45210002 +0000 UTC m=+68.303876246 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/380698de-d0b8-426a-9e9b-708c0e36e818-metrics-tls") pod "dns-default-vzws2" (UID: "380698de-d0b8-426a-9e9b-708c0e36e818") : secret "dns-default-metrics-tls" not found Jul 23 19:20:28.044678 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:20:28.044657 2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-cluster-storage-operator/volume-data-source-validator-78cf4c6f6-zld8c"] Jul 23 19:20:28.066938 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:20:28.066920 2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-operator-555d9d7885-r8lv4"] Jul 23 19:20:28.067046 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:20:28.067031 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-storage-operator/volume-data-source-validator-78cf4c6f6-zld8c" Jul 23 19:20:28.069401 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:20:28.069379 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-storage-operator\"/\"volume-data-source-validator-dockercfg-5pnq5\"" Jul 23 19:20:28.069516 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:20:28.069403 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-storage-operator\"/\"openshift-service-ca.crt\"" Jul 23 19:20:28.069516 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:20:28.069385 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-storage-operator\"/\"kube-root-ca.crt\"" Jul 23 19:20:28.089031 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:20:28.089011 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-storage-operator/volume-data-source-validator-78cf4c6f6-zld8c"] Jul 23 19:20:28.089031 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:20:28.089041 2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-cluster-samples-operator/cluster-samples-operator-79b7c55f87-8l58m"] Jul 23 19:20:28.089166 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:20:28.089125 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-operator-555d9d7885-r8lv4" Jul 23 19:20:28.091569 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:20:28.091551 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\"" Jul 23 19:20:28.091670 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:20:28.091601 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"service-ca-bundle\"" Jul 23 19:20:28.091670 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:20:28.091633 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\"" Jul 23 19:20:28.091881 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:20:28.091868 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"openshift-insights-serving-cert\"" Jul 23 19:20:28.091935 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:20:28.091899 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"operator-dockercfg-vq8lr\"" Jul 23 19:20:28.096740 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:20:28.096724 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"trusted-ca-bundle\"" Jul 23 19:20:28.113017 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:20:28.113001 2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/router-default-7f64bc459d-9b6vg"] Jul 23 19:20:28.113126 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:20:28.113113 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-samples-operator/cluster-samples-operator-79b7c55f87-8l58m" Jul 23 19:20:28.115160 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:20:28.115141 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-samples-operator\"/\"openshift-service-ca.crt\"" Jul 23 19:20:28.115238 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:20:28.115171 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-samples-operator\"/\"samples-operator-tls\"" Jul 23 19:20:28.115289 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:20:28.115241 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-samples-operator\"/\"cluster-samples-operator-dockercfg-7qb62\"" Jul 23 19:20:28.115546 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:20:28.115533 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-samples-operator\"/\"kube-root-ca.crt\"" Jul 23 19:20:28.129150 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:20:28.129134 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-samples-operator/cluster-samples-operator-79b7c55f87-8l58m"] Jul 23 19:20:28.129216 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:20:28.129163 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-operator-555d9d7885-r8lv4"] Jul 23 19:20:28.129216 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:20:28.129187 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/router-default-7f64bc459d-9b6vg"] Jul 23 19:20:28.129288 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:20:28.129238 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/router-default-7f64bc459d-9b6vg" Jul 23 19:20:28.131371 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:20:28.131357 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-service-ca.crt\"" Jul 23 19:20:28.131482 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:20:28.131466 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"service-ca-bundle\"" Jul 23 19:20:28.131550 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:20:28.131468 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-stats-default\"" Jul 23 19:20:28.131700 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:20:28.131685 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"default-ingress-cert\"" Jul 23 19:20:28.131766 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:20:28.131700 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-dockercfg-fn7bv\"" Jul 23 19:20:28.131819 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:20:28.131690 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-metrics-certs-default\"" Jul 23 19:20:28.131919 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:20:28.131905 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"kube-root-ca.crt\"" Jul 23 19:20:28.140260 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:20:28.140244 2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-network-diagnostics/network-check-source-7bc5b6bf85-4lf8g"] Jul 23 19:20:28.161225 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:20:28.161206 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-source-7bc5b6bf85-4lf8g"] Jul 23 19:20:28.161300 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:20:28.161231 2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7658d9b749-6snc5"] Jul 23 19:20:28.161361 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:20:28.161309 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-source-7bc5b6bf85-4lf8g" Jul 23 19:20:28.164221 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:20:28.164208 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"network-diagnostics-dockercfg-t8jmp\"" Jul 23 19:20:28.185376 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:20:28.185297 2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console-operator/console-operator-565b7b569d-z5wf4"] Jul 23 19:20:28.185376 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:20:28.185370 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7658d9b749-6snc5" Jul 23 19:20:28.187855 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:20:28.187837 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"serving-cert\"" Jul 23 19:20:28.188038 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:20:28.188023 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"openshift-service-ca.crt\"" Jul 23 19:20:28.188159 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:20:28.188144 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"kube-storage-version-migrator-operator-dockercfg-txzfk\"" Jul 23 19:20:28.188406 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:20:28.188392 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"config\"" Jul 23 19:20:28.188472 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:20:28.188460 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"kube-root-ca.crt\"" Jul 23 19:20:28.207610 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:20:28.207590 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7658d9b749-6snc5"] Jul 23 19:20:28.207686 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:20:28.207616 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console-operator/console-operator-565b7b569d-z5wf4"] Jul 23 19:20:28.207686 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:20:28.207670 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console-operator/console-operator-565b7b569d-z5wf4" Jul 23 19:20:28.209713 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:20:28.209699 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console-operator\"/\"serving-cert\"" Jul 23 19:20:28.209935 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:20:28.209920 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"kube-root-ca.crt\"" Jul 23 19:20:28.210176 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:20:28.210162 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"openshift-service-ca.crt\"" Jul 23 19:20:28.210580 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:20:28.210559 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"console-operator-config\"" Jul 23 19:20:28.211150 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:20:28.211136 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console-operator\"/\"console-operator-dockercfg-mg95w\"" Jul 23 19:20:28.217767 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:20:28.217751 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"trusted-ca\"" Jul 23 19:20:28.227105 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:20:28.227079 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8rq79_25ca5174-01a0-41fc-9293-77dea2d9ec1f/konnectivity-agent/0.log" limit=1024 Jul 23 19:20:28.251669 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:20:28.251650 2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-network-console/networking-console-plugin-7bc7489cb5-4p45l"] Jul 23 19:20:28.262573 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:20:28.262549 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/00953f2a-6543-445a-8895-40525b4e3e25-tmp\") pod \"insights-operator-555d9d7885-r8lv4\" (UID: \"00953f2a-6543-445a-8895-40525b4e3e25\") " pod="openshift-insights/insights-operator-555d9d7885-r8lv4" Jul 23 19:20:28.262650 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:20:28.262587 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b7c6dedb-e5d3-41a0-b060-50a88ccca303-service-ca-bundle\") pod \"router-default-7f64bc459d-9b6vg\" (UID: \"b7c6dedb-e5d3-41a0-b060-50a88ccca303\") " pod="openshift-ingress/router-default-7f64bc459d-9b6vg" Jul 23 19:20:28.262650 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:20:28.262605 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/e8d01fa4-24e5-4b2b-bf87-5e5395933cee-samples-operator-tls\") pod \"cluster-samples-operator-79b7c55f87-8l58m\" (UID: \"e8d01fa4-24e5-4b2b-bf87-5e5395933cee\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-79b7c55f87-8l58m" Jul 23 19:20:28.262732 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:20:28.262675 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"snapshots\" (UniqueName: \"kubernetes.io/empty-dir/00953f2a-6543-445a-8895-40525b4e3e25-snapshots\") pod \"insights-operator-555d9d7885-r8lv4\" (UID: \"00953f2a-6543-445a-8895-40525b4e3e25\") " pod="openshift-insights/insights-operator-555d9d7885-r8lv4" Jul 23 19:20:28.262732 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:20:28.262697 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-l2j87\" (UniqueName: \"kubernetes.io/projected/b7c6dedb-e5d3-41a0-b060-50a88ccca303-kube-api-access-l2j87\") pod \"router-default-7f64bc459d-9b6vg\" (UID: \"b7c6dedb-e5d3-41a0-b060-50a88ccca303\") " pod="openshift-ingress/router-default-7f64bc459d-9b6vg" Jul 23 19:20:28.262732 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:20:28.262726 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pv6m4\" (UniqueName: \"kubernetes.io/projected/07b02e43-f95a-4173-b027-d0979fbbca4e-kube-api-access-pv6m4\") pod \"volume-data-source-validator-78cf4c6f6-zld8c\" (UID: \"07b02e43-f95a-4173-b027-d0979fbbca4e\") " pod="openshift-cluster-storage-operator/volume-data-source-validator-78cf4c6f6-zld8c" Jul 23 19:20:28.262831 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:20:28.262770 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b7c6dedb-e5d3-41a0-b060-50a88ccca303-metrics-certs\") pod \"router-default-7f64bc459d-9b6vg\" (UID: \"b7c6dedb-e5d3-41a0-b060-50a88ccca303\") " pod="openshift-ingress/router-default-7f64bc459d-9b6vg" Jul 23 19:20:28.262831 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:20:28.262787 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xvfk5\" (UniqueName: \"kubernetes.io/projected/00953f2a-6543-445a-8895-40525b4e3e25-kube-api-access-xvfk5\") pod \"insights-operator-555d9d7885-r8lv4\" (UID: \"00953f2a-6543-445a-8895-40525b4e3e25\") " pod="openshift-insights/insights-operator-555d9d7885-r8lv4" Jul 23 19:20:28.262831 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:20:28.262801 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zx6zl\" (UniqueName: \"kubernetes.io/projected/b76b29a4-cdf8-42d8-8694-ccbeab9a7908-kube-api-access-zx6zl\") pod \"network-check-source-7bc5b6bf85-4lf8g\" (UID: \"b76b29a4-cdf8-42d8-8694-ccbeab9a7908\") " pod="openshift-network-diagnostics/network-check-source-7bc5b6bf85-4lf8g" Jul 23 19:20:28.262831 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:20:28.262820 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/00953f2a-6543-445a-8895-40525b4e3e25-trusted-ca-bundle\") pod \"insights-operator-555d9d7885-r8lv4\" (UID: \"00953f2a-6543-445a-8895-40525b4e3e25\") " pod="openshift-insights/insights-operator-555d9d7885-r8lv4" Jul 23 19:20:28.262958 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:20:28.262858 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/00953f2a-6543-445a-8895-40525b4e3e25-service-ca-bundle\") pod \"insights-operator-555d9d7885-r8lv4\" (UID: \"00953f2a-6543-445a-8895-40525b4e3e25\") " pod="openshift-insights/insights-operator-555d9d7885-r8lv4" Jul 23 19:20:28.262958 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:20:28.262875 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"default-certificate\" (UniqueName: \"kubernetes.io/secret/b7c6dedb-e5d3-41a0-b060-50a88ccca303-default-certificate\") pod \"router-default-7f64bc459d-9b6vg\" (UID: \"b7c6dedb-e5d3-41a0-b060-50a88ccca303\") " pod="openshift-ingress/router-default-7f64bc459d-9b6vg" Jul 23 19:20:28.262958 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:20:28.262903 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"stats-auth\" (UniqueName: \"kubernetes.io/secret/b7c6dedb-e5d3-41a0-b060-50a88ccca303-stats-auth\") pod \"router-default-7f64bc459d-9b6vg\" (UID: \"b7c6dedb-e5d3-41a0-b060-50a88ccca303\") " pod="openshift-ingress/router-default-7f64bc459d-9b6vg" Jul 23 19:20:28.262958 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:20:28.262928 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-s5m9f\" (UniqueName: \"kubernetes.io/projected/e8d01fa4-24e5-4b2b-bf87-5e5395933cee-kube-api-access-s5m9f\") pod \"cluster-samples-operator-79b7c55f87-8l58m\" (UID: \"e8d01fa4-24e5-4b2b-bf87-5e5395933cee\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-79b7c55f87-8l58m" Jul 23 19:20:28.262958 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:20:28.262955 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/00953f2a-6543-445a-8895-40525b4e3e25-serving-cert\") pod \"insights-operator-555d9d7885-r8lv4\" (UID: \"00953f2a-6543-445a-8895-40525b4e3e25\") " pod="openshift-insights/insights-operator-555d9d7885-r8lv4" Jul 23 19:20:28.273694 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:20:28.273677 2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/cluster-monitoring-operator-76cf8966f8-vv2vj"] Jul 23 19:20:28.273785 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:20:28.273753 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-console/networking-console-plugin-7bc7489cb5-4p45l" Jul 23 19:20:28.276241 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:20:28.276224 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-console\"/\"networking-console-plugin\"" Jul 23 19:20:28.276419 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:20:28.276227 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-console\"/\"default-dockercfg-bgzbk\"" Jul 23 19:20:28.276506 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:20:28.276433 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-console\"/\"networking-console-plugin-cert\"" Jul 23 19:20:28.297824 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:20:28.297774 2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-service-ca-operator/service-ca-operator-5bcbc94c78-86qdl"] Jul 23 19:20:28.297896 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:20:28.297883 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/cluster-monitoring-operator-76cf8966f8-vv2vj" Jul 23 19:20:28.300989 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:20:28.300974 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"cluster-monitoring-operator-dockercfg-95j5j\"" Jul 23 19:20:28.301097 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:20:28.301071 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"cluster-monitoring-operator-tls\"" Jul 23 19:20:28.301097 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:20:28.301022 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\"" Jul 23 19:20:28.301212 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:20:28.301030 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"telemetry-config\"" Jul 23 19:20:28.301212 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:20:28.301020 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\"" Jul 23 19:20:28.308839 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:20:28.308818 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8rq79_25ca5174-01a0-41fc-9293-77dea2d9ec1f/konnectivity-agent/0.log" limit=1024 Jul 23 19:20:28.321861 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:20:28.321841 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-console/networking-console-plugin-7bc7489cb5-4p45l"] Jul 23 19:20:28.321861 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:20:28.321864 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/cluster-monitoring-operator-76cf8966f8-vv2vj"] Jul 23 19:20:28.321970 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:20:28.321877 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca-operator/service-ca-operator-5bcbc94c78-86qdl"] Jul 23 19:20:28.321970 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:20:28.321943 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca-operator/service-ca-operator-5bcbc94c78-86qdl" Jul 23 19:20:28.324520 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:20:28.324506 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca-operator\"/\"service-ca-operator-dockercfg-mg6kc\"" Jul 23 19:20:28.324905 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:20:28.324886 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"openshift-service-ca.crt\"" Jul 23 19:20:28.325059 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:20:28.324894 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"kube-root-ca.crt\"" Jul 23 19:20:28.325132 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:20:28.325085 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca-operator\"/\"serving-cert\"" Jul 23 19:20:28.325394 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:20:28.325382 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"service-ca-operator-config\"" Jul 23 19:20:28.363240 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:20:28.363216 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"snapshots\" (UniqueName: \"kubernetes.io/empty-dir/00953f2a-6543-445a-8895-40525b4e3e25-snapshots\") pod \"insights-operator-555d9d7885-r8lv4\" (UID: \"00953f2a-6543-445a-8895-40525b4e3e25\") " pod="openshift-insights/insights-operator-555d9d7885-r8lv4" Jul 23 19:20:28.363365 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:20:28.363245 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/a2c386d2-2c3a-4cf0-9a93-e18300e6905d-serving-cert\") pod \"kube-storage-version-migrator-operator-7658d9b749-6snc5\" (UID: \"a2c386d2-2c3a-4cf0-9a93-e18300e6905d\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7658d9b749-6snc5" Jul 23 19:20:28.363365 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:20:28.363262 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l2j87\" (UniqueName: \"kubernetes.io/projected/b7c6dedb-e5d3-41a0-b060-50a88ccca303-kube-api-access-l2j87\") pod \"router-default-7f64bc459d-9b6vg\" (UID: \"b7c6dedb-e5d3-41a0-b060-50a88ccca303\") " pod="openshift-ingress/router-default-7f64bc459d-9b6vg" Jul 23 19:20:28.363365 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:20:28.363279 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pv6m4\" (UniqueName: \"kubernetes.io/projected/07b02e43-f95a-4173-b027-d0979fbbca4e-kube-api-access-pv6m4\") pod \"volume-data-source-validator-78cf4c6f6-zld8c\" (UID: \"07b02e43-f95a-4173-b027-d0979fbbca4e\") " pod="openshift-cluster-storage-operator/volume-data-source-validator-78cf4c6f6-zld8c" Jul 23 19:20:28.363515 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:20:28.363361 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b7c6dedb-e5d3-41a0-b060-50a88ccca303-metrics-certs\") pod \"router-default-7f64bc459d-9b6vg\" (UID: \"b7c6dedb-e5d3-41a0-b060-50a88ccca303\") " pod="openshift-ingress/router-default-7f64bc459d-9b6vg" Jul 23 19:20:28.363515 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:20:28.363396 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xvfk5\" (UniqueName: \"kubernetes.io/projected/00953f2a-6543-445a-8895-40525b4e3e25-kube-api-access-xvfk5\") pod \"insights-operator-555d9d7885-r8lv4\" (UID: \"00953f2a-6543-445a-8895-40525b4e3e25\") " pod="openshift-insights/insights-operator-555d9d7885-r8lv4" Jul 23 19:20:28.363515 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:20:28.363421 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zx6zl\" (UniqueName: \"kubernetes.io/projected/b76b29a4-cdf8-42d8-8694-ccbeab9a7908-kube-api-access-zx6zl\") pod \"network-check-source-7bc5b6bf85-4lf8g\" (UID: \"b76b29a4-cdf8-42d8-8694-ccbeab9a7908\") " pod="openshift-network-diagnostics/network-check-source-7bc5b6bf85-4lf8g" Jul 23 19:20:28.363515 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:20:28.363453 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/00953f2a-6543-445a-8895-40525b4e3e25-trusted-ca-bundle\") pod \"insights-operator-555d9d7885-r8lv4\" (UID: \"00953f2a-6543-445a-8895-40525b4e3e25\") " pod="openshift-insights/insights-operator-555d9d7885-r8lv4" Jul 23 19:20:28.363515 ip-10-0-129-2 kubenswrapper[2564]: E0723 19:20:28.363471 2564 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found Jul 23 19:20:28.363515 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:20:28.363500 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bx9qw\" (UniqueName: \"kubernetes.io/projected/a2c386d2-2c3a-4cf0-9a93-e18300e6905d-kube-api-access-bx9qw\") pod \"kube-storage-version-migrator-operator-7658d9b749-6snc5\" (UID: \"a2c386d2-2c3a-4cf0-9a93-e18300e6905d\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7658d9b749-6snc5" Jul 23 19:20:28.363867 ip-10-0-129-2 kubenswrapper[2564]: E0723 19:20:28.363535 2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b7c6dedb-e5d3-41a0-b060-50a88ccca303-metrics-certs podName:b7c6dedb-e5d3-41a0-b060-50a88ccca303 nodeName:}" failed. No retries permitted until 2026-07-23 19:20:28.86351848 +0000 UTC m=+54.715294705 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/b7c6dedb-e5d3-41a0-b060-50a88ccca303-metrics-certs") pod "router-default-7f64bc459d-9b6vg" (UID: "b7c6dedb-e5d3-41a0-b060-50a88ccca303") : secret "router-metrics-certs-default" not found Jul 23 19:20:28.363867 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:20:28.363548 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/00953f2a-6543-445a-8895-40525b4e3e25-service-ca-bundle\") pod \"insights-operator-555d9d7885-r8lv4\" (UID: \"00953f2a-6543-445a-8895-40525b4e3e25\") " pod="openshift-insights/insights-operator-555d9d7885-r8lv4" Jul 23 19:20:28.363867 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:20:28.363575 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"default-certificate\" (UniqueName: \"kubernetes.io/secret/b7c6dedb-e5d3-41a0-b060-50a88ccca303-default-certificate\") pod \"router-default-7f64bc459d-9b6vg\" (UID: \"b7c6dedb-e5d3-41a0-b060-50a88ccca303\") " pod="openshift-ingress/router-default-7f64bc459d-9b6vg" Jul 23 19:20:28.363867 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:20:28.363608 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"stats-auth\" (UniqueName: \"kubernetes.io/secret/b7c6dedb-e5d3-41a0-b060-50a88ccca303-stats-auth\") pod \"router-default-7f64bc459d-9b6vg\" (UID: \"b7c6dedb-e5d3-41a0-b060-50a88ccca303\") " pod="openshift-ingress/router-default-7f64bc459d-9b6vg" Jul 23 19:20:28.363867 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:20:28.363631 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s5m9f\" (UniqueName: \"kubernetes.io/projected/e8d01fa4-24e5-4b2b-bf87-5e5395933cee-kube-api-access-s5m9f\") pod \"cluster-samples-operator-79b7c55f87-8l58m\" (UID: \"e8d01fa4-24e5-4b2b-bf87-5e5395933cee\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-79b7c55f87-8l58m" Jul 23 19:20:28.363867 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:20:28.363655 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/ded5b3af-c74c-4d5a-aa77-62fe72d1dd29-config\") pod \"console-operator-565b7b569d-z5wf4\" (UID: \"ded5b3af-c74c-4d5a-aa77-62fe72d1dd29\") " pod="openshift-console-operator/console-operator-565b7b569d-z5wf4" Jul 23 19:20:28.363867 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:20:28.363682 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/00953f2a-6543-445a-8895-40525b4e3e25-serving-cert\") pod \"insights-operator-555d9d7885-r8lv4\" (UID: \"00953f2a-6543-445a-8895-40525b4e3e25\") " pod="openshift-insights/insights-operator-555d9d7885-r8lv4" Jul 23 19:20:28.363867 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:20:28.363730 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/00953f2a-6543-445a-8895-40525b4e3e25-tmp\") pod \"insights-operator-555d9d7885-r8lv4\" (UID: \"00953f2a-6543-445a-8895-40525b4e3e25\") " pod="openshift-insights/insights-operator-555d9d7885-r8lv4" Jul 23 19:20:28.363867 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:20:28.363772 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/a2c386d2-2c3a-4cf0-9a93-e18300e6905d-config\") pod \"kube-storage-version-migrator-operator-7658d9b749-6snc5\" (UID: \"a2c386d2-2c3a-4cf0-9a93-e18300e6905d\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7658d9b749-6snc5" Jul 23 19:20:28.363867 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:20:28.363824 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b7c6dedb-e5d3-41a0-b060-50a88ccca303-service-ca-bundle\") pod \"router-default-7f64bc459d-9b6vg\" (UID: \"b7c6dedb-e5d3-41a0-b060-50a88ccca303\") " pod="openshift-ingress/router-default-7f64bc459d-9b6vg" Jul 23 19:20:28.363867 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:20:28.363852 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/e8d01fa4-24e5-4b2b-bf87-5e5395933cee-samples-operator-tls\") pod \"cluster-samples-operator-79b7c55f87-8l58m\" (UID: \"e8d01fa4-24e5-4b2b-bf87-5e5395933cee\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-79b7c55f87-8l58m" Jul 23 19:20:28.364368 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:20:28.363876 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/ded5b3af-c74c-4d5a-aa77-62fe72d1dd29-trusted-ca\") pod \"console-operator-565b7b569d-z5wf4\" (UID: \"ded5b3af-c74c-4d5a-aa77-62fe72d1dd29\") " pod="openshift-console-operator/console-operator-565b7b569d-z5wf4" Jul 23 19:20:28.364368 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:20:28.363900 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9hczd\" (UniqueName: \"kubernetes.io/projected/ded5b3af-c74c-4d5a-aa77-62fe72d1dd29-kube-api-access-9hczd\") pod \"console-operator-565b7b569d-z5wf4\" (UID: \"ded5b3af-c74c-4d5a-aa77-62fe72d1dd29\") " pod="openshift-console-operator/console-operator-565b7b569d-z5wf4" Jul 23 19:20:28.364368 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:20:28.363930 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/ded5b3af-c74c-4d5a-aa77-62fe72d1dd29-serving-cert\") pod \"console-operator-565b7b569d-z5wf4\" (UID: \"ded5b3af-c74c-4d5a-aa77-62fe72d1dd29\") " pod="openshift-console-operator/console-operator-565b7b569d-z5wf4" Jul 23 19:20:28.364368 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:20:28.363986 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/00953f2a-6543-445a-8895-40525b4e3e25-tmp\") pod \"insights-operator-555d9d7885-r8lv4\" (UID: \"00953f2a-6543-445a-8895-40525b4e3e25\") " pod="openshift-insights/insights-operator-555d9d7885-r8lv4" Jul 23 19:20:28.364368 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:20:28.364074 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/00953f2a-6543-445a-8895-40525b4e3e25-service-ca-bundle\") pod \"insights-operator-555d9d7885-r8lv4\" (UID: \"00953f2a-6543-445a-8895-40525b4e3e25\") " pod="openshift-insights/insights-operator-555d9d7885-r8lv4" Jul 23 19:20:28.364368 ip-10-0-129-2 kubenswrapper[2564]: E0723 19:20:28.364081 2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/b7c6dedb-e5d3-41a0-b060-50a88ccca303-service-ca-bundle podName:b7c6dedb-e5d3-41a0-b060-50a88ccca303 nodeName:}" failed. No retries permitted until 2026-07-23 19:20:28.864067047 +0000 UTC m=+54.715843292 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/b7c6dedb-e5d3-41a0-b060-50a88ccca303-service-ca-bundle") pod "router-default-7f64bc459d-9b6vg" (UID: "b7c6dedb-e5d3-41a0-b060-50a88ccca303") : configmap references non-existent config key: service-ca.crt Jul 23 19:20:28.364368 ip-10-0-129-2 kubenswrapper[2564]: E0723 19:20:28.364092 2564 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found Jul 23 19:20:28.364368 ip-10-0-129-2 kubenswrapper[2564]: E0723 19:20:28.364169 2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e8d01fa4-24e5-4b2b-bf87-5e5395933cee-samples-operator-tls podName:e8d01fa4-24e5-4b2b-bf87-5e5395933cee nodeName:}" failed. No retries permitted until 2026-07-23 19:20:28.864156441 +0000 UTC m=+54.715932669 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/e8d01fa4-24e5-4b2b-bf87-5e5395933cee-samples-operator-tls") pod "cluster-samples-operator-79b7c55f87-8l58m" (UID: "e8d01fa4-24e5-4b2b-bf87-5e5395933cee") : secret "samples-operator-tls" not found Jul 23 19:20:28.364368 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:20:28.364273 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"snapshots\" (UniqueName: \"kubernetes.io/empty-dir/00953f2a-6543-445a-8895-40525b4e3e25-snapshots\") pod \"insights-operator-555d9d7885-r8lv4\" (UID: \"00953f2a-6543-445a-8895-40525b4e3e25\") " pod="openshift-insights/insights-operator-555d9d7885-r8lv4" Jul 23 19:20:28.364748 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:20:28.364433 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/00953f2a-6543-445a-8895-40525b4e3e25-trusted-ca-bundle\") pod \"insights-operator-555d9d7885-r8lv4\" (UID: \"00953f2a-6543-445a-8895-40525b4e3e25\") " pod="openshift-insights/insights-operator-555d9d7885-r8lv4" Jul 23 19:20:28.367519 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:20:28.367500 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/00953f2a-6543-445a-8895-40525b4e3e25-serving-cert\") pod \"insights-operator-555d9d7885-r8lv4\" (UID: \"00953f2a-6543-445a-8895-40525b4e3e25\") " pod="openshift-insights/insights-operator-555d9d7885-r8lv4" Jul 23 19:20:28.367639 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:20:28.367625 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"stats-auth\" (UniqueName: \"kubernetes.io/secret/b7c6dedb-e5d3-41a0-b060-50a88ccca303-stats-auth\") pod \"router-default-7f64bc459d-9b6vg\" (UID: \"b7c6dedb-e5d3-41a0-b060-50a88ccca303\") " pod="openshift-ingress/router-default-7f64bc459d-9b6vg" Jul 23 19:20:28.367693 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:20:28.367677 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"default-certificate\" (UniqueName: \"kubernetes.io/secret/b7c6dedb-e5d3-41a0-b060-50a88ccca303-default-certificate\") pod \"router-default-7f64bc459d-9b6vg\" (UID: \"b7c6dedb-e5d3-41a0-b060-50a88ccca303\") " pod="openshift-ingress/router-default-7f64bc459d-9b6vg" Jul 23 19:20:28.375860 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:20:28.375842 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-pv6m4\" (UniqueName: \"kubernetes.io/projected/07b02e43-f95a-4173-b027-d0979fbbca4e-kube-api-access-pv6m4\") pod \"volume-data-source-validator-78cf4c6f6-zld8c\" (UID: \"07b02e43-f95a-4173-b027-d0979fbbca4e\") " pod="openshift-cluster-storage-operator/volume-data-source-validator-78cf4c6f6-zld8c" Jul 23 19:20:28.383432 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:20:28.383410 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-s5m9f\" (UniqueName: \"kubernetes.io/projected/e8d01fa4-24e5-4b2b-bf87-5e5395933cee-kube-api-access-s5m9f\") pod \"cluster-samples-operator-79b7c55f87-8l58m\" (UID: \"e8d01fa4-24e5-4b2b-bf87-5e5395933cee\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-79b7c55f87-8l58m" Jul 23 19:20:28.383930 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:20:28.383912 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-zx6zl\" (UniqueName: \"kubernetes.io/projected/b76b29a4-cdf8-42d8-8694-ccbeab9a7908-kube-api-access-zx6zl\") pod \"network-check-source-7bc5b6bf85-4lf8g\" (UID: \"b76b29a4-cdf8-42d8-8694-ccbeab9a7908\") " pod="openshift-network-diagnostics/network-check-source-7bc5b6bf85-4lf8g" Jul 23 19:20:28.384458 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:20:28.384437 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-l2j87\" (UniqueName: \"kubernetes.io/projected/b7c6dedb-e5d3-41a0-b060-50a88ccca303-kube-api-access-l2j87\") pod \"router-default-7f64bc459d-9b6vg\" (UID: \"b7c6dedb-e5d3-41a0-b060-50a88ccca303\") " pod="openshift-ingress/router-default-7f64bc459d-9b6vg" Jul 23 19:20:28.385486 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:20:28.385470 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xvfk5\" (UniqueName: \"kubernetes.io/projected/00953f2a-6543-445a-8895-40525b4e3e25-kube-api-access-xvfk5\") pod \"insights-operator-555d9d7885-r8lv4\" (UID: \"00953f2a-6543-445a-8895-40525b4e3e25\") " pod="openshift-insights/insights-operator-555d9d7885-r8lv4" Jul 23 19:20:28.397613 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:20:28.397598 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-operator-555d9d7885-r8lv4" Jul 23 19:20:28.398871 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:20:28.398849 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8rq79_25ca5174-01a0-41fc-9293-77dea2d9ec1f/konnectivity-agent/0.log" limit=1024 Jul 23 19:20:28.465264 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:20:28.465238 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bx9qw\" (UniqueName: \"kubernetes.io/projected/a2c386d2-2c3a-4cf0-9a93-e18300e6905d-kube-api-access-bx9qw\") pod \"kube-storage-version-migrator-operator-7658d9b749-6snc5\" (UID: \"a2c386d2-2c3a-4cf0-9a93-e18300e6905d\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7658d9b749-6snc5" Jul 23 19:20:28.465392 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:20:28.465285 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/678e0e44-d3bb-4269-99c9-43229d8f2291-config\") pod \"service-ca-operator-5bcbc94c78-86qdl\" (UID: \"678e0e44-d3bb-4269-99c9-43229d8f2291\") " pod="openshift-service-ca-operator/service-ca-operator-5bcbc94c78-86qdl" Jul 23 19:20:28.465392 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:20:28.465333 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/ded5b3af-c74c-4d5a-aa77-62fe72d1dd29-config\") pod \"console-operator-565b7b569d-z5wf4\" (UID: \"ded5b3af-c74c-4d5a-aa77-62fe72d1dd29\") " pod="openshift-console-operator/console-operator-565b7b569d-z5wf4" Jul 23 19:20:28.465392 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:20:28.465369 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zqd5s\" (UniqueName: \"kubernetes.io/projected/6f0a5836-d7de-49fc-b3a2-49830eaa468f-kube-api-access-zqd5s\") pod \"cluster-monitoring-operator-76cf8966f8-vv2vj\" (UID: \"6f0a5836-d7de-49fc-b3a2-49830eaa468f\") " pod="openshift-monitoring/cluster-monitoring-operator-76cf8966f8-vv2vj" Jul 23 19:20:28.465547 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:20:28.465417 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/a2c386d2-2c3a-4cf0-9a93-e18300e6905d-config\") pod \"kube-storage-version-migrator-operator-7658d9b749-6snc5\" (UID: \"a2c386d2-2c3a-4cf0-9a93-e18300e6905d\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7658d9b749-6snc5" Jul 23 19:20:28.465547 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:20:28.465479 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/ded5b3af-c74c-4d5a-aa77-62fe72d1dd29-trusted-ca\") pod \"console-operator-565b7b569d-z5wf4\" (UID: \"ded5b3af-c74c-4d5a-aa77-62fe72d1dd29\") " pod="openshift-console-operator/console-operator-565b7b569d-z5wf4" Jul 23 19:20:28.465547 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:20:28.465506 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9hczd\" (UniqueName: \"kubernetes.io/projected/ded5b3af-c74c-4d5a-aa77-62fe72d1dd29-kube-api-access-9hczd\") pod \"console-operator-565b7b569d-z5wf4\" (UID: \"ded5b3af-c74c-4d5a-aa77-62fe72d1dd29\") " pod="openshift-console-operator/console-operator-565b7b569d-z5wf4" Jul 23 19:20:28.465547 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:20:28.465538 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"telemetry-config\" (UniqueName: \"kubernetes.io/configmap/6f0a5836-d7de-49fc-b3a2-49830eaa468f-telemetry-config\") pod \"cluster-monitoring-operator-76cf8966f8-vv2vj\" (UID: \"6f0a5836-d7de-49fc-b3a2-49830eaa468f\") " pod="openshift-monitoring/cluster-monitoring-operator-76cf8966f8-vv2vj" Jul 23 19:20:28.465698 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:20:28.465568 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/6f0a5836-d7de-49fc-b3a2-49830eaa468f-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-76cf8966f8-vv2vj\" (UID: \"6f0a5836-d7de-49fc-b3a2-49830eaa468f\") " pod="openshift-monitoring/cluster-monitoring-operator-76cf8966f8-vv2vj" Jul 23 19:20:28.465753 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:20:28.465692 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/ded5b3af-c74c-4d5a-aa77-62fe72d1dd29-serving-cert\") pod \"console-operator-565b7b569d-z5wf4\" (UID: \"ded5b3af-c74c-4d5a-aa77-62fe72d1dd29\") " pod="openshift-console-operator/console-operator-565b7b569d-z5wf4" Jul 23 19:20:28.465753 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:20:28.465737 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/678e0e44-d3bb-4269-99c9-43229d8f2291-serving-cert\") pod \"service-ca-operator-5bcbc94c78-86qdl\" (UID: \"678e0e44-d3bb-4269-99c9-43229d8f2291\") " pod="openshift-service-ca-operator/service-ca-operator-5bcbc94c78-86qdl" Jul 23 19:20:28.465844 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:20:28.465765 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/a2c386d2-2c3a-4cf0-9a93-e18300e6905d-serving-cert\") pod \"kube-storage-version-migrator-operator-7658d9b749-6snc5\" (UID: \"a2c386d2-2c3a-4cf0-9a93-e18300e6905d\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7658d9b749-6snc5" Jul 23 19:20:28.465844 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:20:28.465825 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/7c2f37e4-a8b7-497e-ac12-7a6f5dd86619-nginx-conf\") pod \"networking-console-plugin-7bc7489cb5-4p45l\" (UID: \"7c2f37e4-a8b7-497e-ac12-7a6f5dd86619\") " pod="openshift-network-console/networking-console-plugin-7bc7489cb5-4p45l" Jul 23 19:20:28.465922 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:20:28.465854 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sqzm6\" (UniqueName: \"kubernetes.io/projected/678e0e44-d3bb-4269-99c9-43229d8f2291-kube-api-access-sqzm6\") pod \"service-ca-operator-5bcbc94c78-86qdl\" (UID: \"678e0e44-d3bb-4269-99c9-43229d8f2291\") " pod="openshift-service-ca-operator/service-ca-operator-5bcbc94c78-86qdl" Jul 23 19:20:28.465922 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:20:28.465892 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/7c2f37e4-a8b7-497e-ac12-7a6f5dd86619-networking-console-plugin-cert\") pod \"networking-console-plugin-7bc7489cb5-4p45l\" (UID: \"7c2f37e4-a8b7-497e-ac12-7a6f5dd86619\") " pod="openshift-network-console/networking-console-plugin-7bc7489cb5-4p45l" Jul 23 19:20:28.465992 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:20:28.465942 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/ded5b3af-c74c-4d5a-aa77-62fe72d1dd29-config\") pod \"console-operator-565b7b569d-z5wf4\" (UID: \"ded5b3af-c74c-4d5a-aa77-62fe72d1dd29\") " pod="openshift-console-operator/console-operator-565b7b569d-z5wf4" Jul 23 19:20:28.466345 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:20:28.466286 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/ded5b3af-c74c-4d5a-aa77-62fe72d1dd29-trusted-ca\") pod \"console-operator-565b7b569d-z5wf4\" (UID: \"ded5b3af-c74c-4d5a-aa77-62fe72d1dd29\") " pod="openshift-console-operator/console-operator-565b7b569d-z5wf4" Jul 23 19:20:28.467906 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:20:28.467884 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/ded5b3af-c74c-4d5a-aa77-62fe72d1dd29-serving-cert\") pod \"console-operator-565b7b569d-z5wf4\" (UID: \"ded5b3af-c74c-4d5a-aa77-62fe72d1dd29\") " pod="openshift-console-operator/console-operator-565b7b569d-z5wf4" Jul 23 19:20:28.468047 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:20:28.467954 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-source-7bc5b6bf85-4lf8g" Jul 23 19:20:28.473717 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:20:28.473694 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9hczd\" (UniqueName: \"kubernetes.io/projected/ded5b3af-c74c-4d5a-aa77-62fe72d1dd29-kube-api-access-9hczd\") pod \"console-operator-565b7b569d-z5wf4\" (UID: \"ded5b3af-c74c-4d5a-aa77-62fe72d1dd29\") " pod="openshift-console-operator/console-operator-565b7b569d-z5wf4" Jul 23 19:20:28.482293 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:20:28.482273 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/a2c386d2-2c3a-4cf0-9a93-e18300e6905d-config\") pod \"kube-storage-version-migrator-operator-7658d9b749-6snc5\" (UID: \"a2c386d2-2c3a-4cf0-9a93-e18300e6905d\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7658d9b749-6snc5" Jul 23 19:20:28.484235 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:20:28.484207 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/a2c386d2-2c3a-4cf0-9a93-e18300e6905d-serving-cert\") pod \"kube-storage-version-migrator-operator-7658d9b749-6snc5\" (UID: \"a2c386d2-2c3a-4cf0-9a93-e18300e6905d\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7658d9b749-6snc5" Jul 23 19:20:28.485713 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:20:28.485670 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-bx9qw\" (UniqueName: \"kubernetes.io/projected/a2c386d2-2c3a-4cf0-9a93-e18300e6905d-kube-api-access-bx9qw\") pod \"kube-storage-version-migrator-operator-7658d9b749-6snc5\" (UID: \"a2c386d2-2c3a-4cf0-9a93-e18300e6905d\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7658d9b749-6snc5" Jul 23 19:20:28.493471 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:20:28.493454 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7658d9b749-6snc5" Jul 23 19:20:28.498482 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:20:28.498435 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8rq79_25ca5174-01a0-41fc-9293-77dea2d9ec1f/konnectivity-agent/0.log" limit=1024 Jul 23 19:20:28.518029 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:20:28.518009 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console-operator/console-operator-565b7b569d-z5wf4" Jul 23 19:20:28.547987 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:20:28.547937 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-operator-555d9d7885-r8lv4"] Jul 23 19:20:28.551201 ip-10-0-129-2 kubenswrapper[2564]: W0723 19:20:28.551162 2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod00953f2a_6543_445a_8895_40525b4e3e25.slice/crio-1c0b7c061f5886c81d9c871a8462c55be4bac0c7338c79f8e40e41d71f604007 WatchSource:0}: Error finding container 1c0b7c061f5886c81d9c871a8462c55be4bac0c7338c79f8e40e41d71f604007: Status 404 returned error can't find the container with id 1c0b7c061f5886c81d9c871a8462c55be4bac0c7338c79f8e40e41d71f604007 Jul 23 19:20:28.566781 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:20:28.566755 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/678e0e44-d3bb-4269-99c9-43229d8f2291-serving-cert\") pod \"service-ca-operator-5bcbc94c78-86qdl\" (UID: \"678e0e44-d3bb-4269-99c9-43229d8f2291\") " pod="openshift-service-ca-operator/service-ca-operator-5bcbc94c78-86qdl" Jul 23 19:20:28.566954 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:20:28.566821 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/7c2f37e4-a8b7-497e-ac12-7a6f5dd86619-nginx-conf\") pod \"networking-console-plugin-7bc7489cb5-4p45l\" (UID: \"7c2f37e4-a8b7-497e-ac12-7a6f5dd86619\") " pod="openshift-network-console/networking-console-plugin-7bc7489cb5-4p45l" Jul 23 19:20:28.566954 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:20:28.566852 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sqzm6\" (UniqueName: \"kubernetes.io/projected/678e0e44-d3bb-4269-99c9-43229d8f2291-kube-api-access-sqzm6\") pod \"service-ca-operator-5bcbc94c78-86qdl\" (UID: \"678e0e44-d3bb-4269-99c9-43229d8f2291\") " pod="openshift-service-ca-operator/service-ca-operator-5bcbc94c78-86qdl" Jul 23 19:20:28.566954 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:20:28.566881 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/7c2f37e4-a8b7-497e-ac12-7a6f5dd86619-networking-console-plugin-cert\") pod \"networking-console-plugin-7bc7489cb5-4p45l\" (UID: \"7c2f37e4-a8b7-497e-ac12-7a6f5dd86619\") " pod="openshift-network-console/networking-console-plugin-7bc7489cb5-4p45l" Jul 23 19:20:28.566954 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:20:28.566944 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/678e0e44-d3bb-4269-99c9-43229d8f2291-config\") pod \"service-ca-operator-5bcbc94c78-86qdl\" (UID: \"678e0e44-d3bb-4269-99c9-43229d8f2291\") " pod="openshift-service-ca-operator/service-ca-operator-5bcbc94c78-86qdl" Jul 23 19:20:28.567260 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:20:28.566991 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zqd5s\" (UniqueName: \"kubernetes.io/projected/6f0a5836-d7de-49fc-b3a2-49830eaa468f-kube-api-access-zqd5s\") pod \"cluster-monitoring-operator-76cf8966f8-vv2vj\" (UID: \"6f0a5836-d7de-49fc-b3a2-49830eaa468f\") " pod="openshift-monitoring/cluster-monitoring-operator-76cf8966f8-vv2vj" Jul 23 19:20:28.567260 ip-10-0-129-2 kubenswrapper[2564]: E0723 19:20:28.567105 2564 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found Jul 23 19:20:28.567260 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:20:28.567118 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"telemetry-config\" (UniqueName: \"kubernetes.io/configmap/6f0a5836-d7de-49fc-b3a2-49830eaa468f-telemetry-config\") pod \"cluster-monitoring-operator-76cf8966f8-vv2vj\" (UID: \"6f0a5836-d7de-49fc-b3a2-49830eaa468f\") " pod="openshift-monitoring/cluster-monitoring-operator-76cf8966f8-vv2vj" Jul 23 19:20:28.567260 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:20:28.567155 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/6f0a5836-d7de-49fc-b3a2-49830eaa468f-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-76cf8966f8-vv2vj\" (UID: \"6f0a5836-d7de-49fc-b3a2-49830eaa468f\") " pod="openshift-monitoring/cluster-monitoring-operator-76cf8966f8-vv2vj" Jul 23 19:20:28.567260 ip-10-0-129-2 kubenswrapper[2564]: E0723 19:20:28.567177 2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7c2f37e4-a8b7-497e-ac12-7a6f5dd86619-networking-console-plugin-cert podName:7c2f37e4-a8b7-497e-ac12-7a6f5dd86619 nodeName:}" failed. No retries permitted until 2026-07-23 19:20:29.067158235 +0000 UTC m=+54.918934480 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/7c2f37e4-a8b7-497e-ac12-7a6f5dd86619-networking-console-plugin-cert") pod "networking-console-plugin-7bc7489cb5-4p45l" (UID: "7c2f37e4-a8b7-497e-ac12-7a6f5dd86619") : secret "networking-console-plugin-cert" not found Jul 23 19:20:28.567260 ip-10-0-129-2 kubenswrapper[2564]: E0723 19:20:28.567226 2564 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found Jul 23 19:20:28.567979 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:20:28.567957 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/7c2f37e4-a8b7-497e-ac12-7a6f5dd86619-nginx-conf\") pod \"networking-console-plugin-7bc7489cb5-4p45l\" (UID: \"7c2f37e4-a8b7-497e-ac12-7a6f5dd86619\") " pod="openshift-network-console/networking-console-plugin-7bc7489cb5-4p45l" Jul 23 19:20:28.568070 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:20:28.567966 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/678e0e44-d3bb-4269-99c9-43229d8f2291-config\") pod \"service-ca-operator-5bcbc94c78-86qdl\" (UID: \"678e0e44-d3bb-4269-99c9-43229d8f2291\") " pod="openshift-service-ca-operator/service-ca-operator-5bcbc94c78-86qdl" Jul 23 19:20:28.568070 ip-10-0-129-2 kubenswrapper[2564]: E0723 19:20:28.568048 2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6f0a5836-d7de-49fc-b3a2-49830eaa468f-cluster-monitoring-operator-tls podName:6f0a5836-d7de-49fc-b3a2-49830eaa468f nodeName:}" failed. No retries permitted until 2026-07-23 19:20:29.068016064 +0000 UTC m=+54.919792292 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/6f0a5836-d7de-49fc-b3a2-49830eaa468f-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-76cf8966f8-vv2vj" (UID: "6f0a5836-d7de-49fc-b3a2-49830eaa468f") : secret "cluster-monitoring-operator-tls" not found Jul 23 19:20:28.568269 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:20:28.568249 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"telemetry-config\" (UniqueName: \"kubernetes.io/configmap/6f0a5836-d7de-49fc-b3a2-49830eaa468f-telemetry-config\") pod \"cluster-monitoring-operator-76cf8966f8-vv2vj\" (UID: \"6f0a5836-d7de-49fc-b3a2-49830eaa468f\") " pod="openshift-monitoring/cluster-monitoring-operator-76cf8966f8-vv2vj" Jul 23 19:20:28.569397 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:20:28.569379 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/678e0e44-d3bb-4269-99c9-43229d8f2291-serving-cert\") pod \"service-ca-operator-5bcbc94c78-86qdl\" (UID: \"678e0e44-d3bb-4269-99c9-43229d8f2291\") " pod="openshift-service-ca-operator/service-ca-operator-5bcbc94c78-86qdl" Jul 23 19:20:28.576608 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:20:28.576586 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-zqd5s\" (UniqueName: \"kubernetes.io/projected/6f0a5836-d7de-49fc-b3a2-49830eaa468f-kube-api-access-zqd5s\") pod \"cluster-monitoring-operator-76cf8966f8-vv2vj\" (UID: \"6f0a5836-d7de-49fc-b3a2-49830eaa468f\") " pod="openshift-monitoring/cluster-monitoring-operator-76cf8966f8-vv2vj" Jul 23 19:20:28.577679 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:20:28.577657 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-sqzm6\" (UniqueName: \"kubernetes.io/projected/678e0e44-d3bb-4269-99c9-43229d8f2291-kube-api-access-sqzm6\") pod \"service-ca-operator-5bcbc94c78-86qdl\" (UID: \"678e0e44-d3bb-4269-99c9-43229d8f2291\") " pod="openshift-service-ca-operator/service-ca-operator-5bcbc94c78-86qdl" Jul 23 19:20:28.592199 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:20:28.592173 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8rq79_25ca5174-01a0-41fc-9293-77dea2d9ec1f/konnectivity-agent/0.log" limit=1024 Jul 23 19:20:28.638367 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:20:28.638264 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-source-7bc5b6bf85-4lf8g"] Jul 23 19:20:28.648117 ip-10-0-129-2 kubenswrapper[2564]: W0723 19:20:28.648095 2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb76b29a4_cdf8_42d8_8694_ccbeab9a7908.slice/crio-3b7dfffc90ce23f7cead8f23428246d7437655ba77fa921febd1e6f0d6e88878 WatchSource:0}: Error finding container 3b7dfffc90ce23f7cead8f23428246d7437655ba77fa921febd1e6f0d6e88878: Status 404 returned error can't find the container with id 3b7dfffc90ce23f7cead8f23428246d7437655ba77fa921febd1e6f0d6e88878 Jul 23 19:20:28.649317 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:20:28.649297 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca-operator/service-ca-operator-5bcbc94c78-86qdl" Jul 23 19:20:28.675438 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:20:28.675407 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-storage-operator/volume-data-source-validator-78cf4c6f6-zld8c" Jul 23 19:20:28.679761 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:20:28.679740 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7658d9b749-6snc5"] Jul 23 19:20:28.684912 ip-10-0-129-2 kubenswrapper[2564]: W0723 19:20:28.684879 2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda2c386d2_2c3a_4cf0_9a93_e18300e6905d.slice/crio-92610e2ea14832f1b39bdc5e368fee394429bc3688e9cb447ca3cc34d2199add WatchSource:0}: Error finding container 92610e2ea14832f1b39bdc5e368fee394429bc3688e9cb447ca3cc34d2199add: Status 404 returned error can't find the container with id 92610e2ea14832f1b39bdc5e368fee394429bc3688e9cb447ca3cc34d2199add Jul 23 19:20:28.710390 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:20:28.710363 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console-operator/console-operator-565b7b569d-z5wf4"] Jul 23 19:20:28.713534 ip-10-0-129-2 kubenswrapper[2564]: W0723 19:20:28.713476 2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podded5b3af_c74c_4d5a_aa77_62fe72d1dd29.slice/crio-b02d2230bafe1f930cba055f413e6d35802de9ed2b1b6981d7741e32f6ab2ab0 WatchSource:0}: Error finding container b02d2230bafe1f930cba055f413e6d35802de9ed2b1b6981d7741e32f6ab2ab0: Status 404 returned error can't find the container with id b02d2230bafe1f930cba055f413e6d35802de9ed2b1b6981d7741e32f6ab2ab0 Jul 23 19:20:28.717183 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:20:28.717133 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8rq79_25ca5174-01a0-41fc-9293-77dea2d9ec1f/konnectivity-agent/0.log" limit=1024 Jul 23 19:20:28.793405 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:20:28.793365 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca-operator/service-ca-operator-5bcbc94c78-86qdl"] Jul 23 19:20:28.795682 ip-10-0-129-2 kubenswrapper[2564]: W0723 19:20:28.795654 2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod678e0e44_d3bb_4269_99c9_43229d8f2291.slice/crio-b4660ba765507dfca367ee6701cdd418e2dfe200712738cecf32df2640a7e687 WatchSource:0}: Error finding container b4660ba765507dfca367ee6701cdd418e2dfe200712738cecf32df2640a7e687: Status 404 returned error can't find the container with id b4660ba765507dfca367ee6701cdd418e2dfe200712738cecf32df2640a7e687 Jul 23 19:20:28.805162 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:20:28.805139 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-storage-operator/volume-data-source-validator-78cf4c6f6-zld8c"] Jul 23 19:20:28.807846 ip-10-0-129-2 kubenswrapper[2564]: W0723 19:20:28.807826 2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod07b02e43_f95a_4173_b027_d0979fbbca4e.slice/crio-c7197308e3c8e39da7598035a9b61615b0e0510305cd6438112d8cecdb3dc3dd WatchSource:0}: Error finding container c7197308e3c8e39da7598035a9b61615b0e0510305cd6438112d8cecdb3dc3dd: Status 404 returned error can't find the container with id c7197308e3c8e39da7598035a9b61615b0e0510305cd6438112d8cecdb3dc3dd Jul 23 19:20:28.870460 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:20:28.870440 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b7c6dedb-e5d3-41a0-b060-50a88ccca303-service-ca-bundle\") pod \"router-default-7f64bc459d-9b6vg\" (UID: \"b7c6dedb-e5d3-41a0-b060-50a88ccca303\") " pod="openshift-ingress/router-default-7f64bc459d-9b6vg" Jul 23 19:20:28.870524 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:20:28.870469 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/e8d01fa4-24e5-4b2b-bf87-5e5395933cee-samples-operator-tls\") pod \"cluster-samples-operator-79b7c55f87-8l58m\" (UID: \"e8d01fa4-24e5-4b2b-bf87-5e5395933cee\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-79b7c55f87-8l58m" Jul 23 19:20:28.870556 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:20:28.870526 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b7c6dedb-e5d3-41a0-b060-50a88ccca303-metrics-certs\") pod \"router-default-7f64bc459d-9b6vg\" (UID: \"b7c6dedb-e5d3-41a0-b060-50a88ccca303\") " pod="openshift-ingress/router-default-7f64bc459d-9b6vg" Jul 23 19:20:28.870608 ip-10-0-129-2 kubenswrapper[2564]: E0723 19:20:28.870595 2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/b7c6dedb-e5d3-41a0-b060-50a88ccca303-service-ca-bundle podName:b7c6dedb-e5d3-41a0-b060-50a88ccca303 nodeName:}" failed. No retries permitted until 2026-07-23 19:20:29.870577579 +0000 UTC m=+55.722353827 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/b7c6dedb-e5d3-41a0-b060-50a88ccca303-service-ca-bundle") pod "router-default-7f64bc459d-9b6vg" (UID: "b7c6dedb-e5d3-41a0-b060-50a88ccca303") : configmap references non-existent config key: service-ca.crt Jul 23 19:20:28.870650 ip-10-0-129-2 kubenswrapper[2564]: E0723 19:20:28.870638 2564 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found Jul 23 19:20:28.870681 ip-10-0-129-2 kubenswrapper[2564]: E0723 19:20:28.870660 2564 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found Jul 23 19:20:28.870714 ip-10-0-129-2 kubenswrapper[2564]: E0723 19:20:28.870670 2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b7c6dedb-e5d3-41a0-b060-50a88ccca303-metrics-certs podName:b7c6dedb-e5d3-41a0-b060-50a88ccca303 nodeName:}" failed. No retries permitted until 2026-07-23 19:20:29.870660132 +0000 UTC m=+55.722436358 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/b7c6dedb-e5d3-41a0-b060-50a88ccca303-metrics-certs") pod "router-default-7f64bc459d-9b6vg" (UID: "b7c6dedb-e5d3-41a0-b060-50a88ccca303") : secret "router-metrics-certs-default" not found Jul 23 19:20:28.870747 ip-10-0-129-2 kubenswrapper[2564]: E0723 19:20:28.870725 2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e8d01fa4-24e5-4b2b-bf87-5e5395933cee-samples-operator-tls podName:e8d01fa4-24e5-4b2b-bf87-5e5395933cee nodeName:}" failed. No retries permitted until 2026-07-23 19:20:29.870710579 +0000 UTC m=+55.722486804 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/e8d01fa4-24e5-4b2b-bf87-5e5395933cee-samples-operator-tls") pod "cluster-samples-operator-79b7c55f87-8l58m" (UID: "e8d01fa4-24e5-4b2b-bf87-5e5395933cee") : secret "samples-operator-tls" not found Jul 23 19:20:28.970640 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:20:28.970618 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7658d9b749-6snc5" event={"ID":"a2c386d2-2c3a-4cf0-9a93-e18300e6905d","Type":"ContainerStarted","Data":"92610e2ea14832f1b39bdc5e368fee394429bc3688e9cb447ca3cc34d2199add"} Jul 23 19:20:28.971655 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:20:28.971635 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-555d9d7885-r8lv4" event={"ID":"00953f2a-6543-445a-8895-40525b4e3e25","Type":"ContainerStarted","Data":"1c0b7c061f5886c81d9c871a8462c55be4bac0c7338c79f8e40e41d71f604007"} Jul 23 19:20:28.972620 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:20:28.972601 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-565b7b569d-z5wf4" event={"ID":"ded5b3af-c74c-4d5a-aa77-62fe72d1dd29","Type":"ContainerStarted","Data":"b02d2230bafe1f930cba055f413e6d35802de9ed2b1b6981d7741e32f6ab2ab0"} Jul 23 19:20:28.973585 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:20:28.973555 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-source-7bc5b6bf85-4lf8g" event={"ID":"b76b29a4-cdf8-42d8-8694-ccbeab9a7908","Type":"ContainerStarted","Data":"3b7dfffc90ce23f7cead8f23428246d7437655ba77fa921febd1e6f0d6e88878"} Jul 23 19:20:28.974510 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:20:28.974494 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-storage-operator/volume-data-source-validator-78cf4c6f6-zld8c" event={"ID":"07b02e43-f95a-4173-b027-d0979fbbca4e","Type":"ContainerStarted","Data":"c7197308e3c8e39da7598035a9b61615b0e0510305cd6438112d8cecdb3dc3dd"} Jul 23 19:20:28.975454 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:20:28.975437 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-5bcbc94c78-86qdl" event={"ID":"678e0e44-d3bb-4269-99c9-43229d8f2291","Type":"ContainerStarted","Data":"b4660ba765507dfca367ee6701cdd418e2dfe200712738cecf32df2640a7e687"} Jul 23 19:20:29.072135 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:20:29.072088 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/6f0a5836-d7de-49fc-b3a2-49830eaa468f-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-76cf8966f8-vv2vj\" (UID: \"6f0a5836-d7de-49fc-b3a2-49830eaa468f\") " pod="openshift-monitoring/cluster-monitoring-operator-76cf8966f8-vv2vj" Jul 23 19:20:29.072419 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:20:29.072142 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/7c2f37e4-a8b7-497e-ac12-7a6f5dd86619-networking-console-plugin-cert\") pod \"networking-console-plugin-7bc7489cb5-4p45l\" (UID: \"7c2f37e4-a8b7-497e-ac12-7a6f5dd86619\") " pod="openshift-network-console/networking-console-plugin-7bc7489cb5-4p45l" Jul 23 19:20:29.072419 ip-10-0-129-2 kubenswrapper[2564]: E0723 19:20:29.072223 2564 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found Jul 23 19:20:29.072419 ip-10-0-129-2 kubenswrapper[2564]: E0723 19:20:29.072255 2564 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found Jul 23 19:20:29.072419 ip-10-0-129-2 kubenswrapper[2564]: E0723 19:20:29.072281 2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6f0a5836-d7de-49fc-b3a2-49830eaa468f-cluster-monitoring-operator-tls podName:6f0a5836-d7de-49fc-b3a2-49830eaa468f nodeName:}" failed. No retries permitted until 2026-07-23 19:20:30.07226655 +0000 UTC m=+55.924042797 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/6f0a5836-d7de-49fc-b3a2-49830eaa468f-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-76cf8966f8-vv2vj" (UID: "6f0a5836-d7de-49fc-b3a2-49830eaa468f") : secret "cluster-monitoring-operator-tls" not found Jul 23 19:20:29.072419 ip-10-0-129-2 kubenswrapper[2564]: E0723 19:20:29.072297 2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7c2f37e4-a8b7-497e-ac12-7a6f5dd86619-networking-console-plugin-cert podName:7c2f37e4-a8b7-497e-ac12-7a6f5dd86619 nodeName:}" failed. No retries permitted until 2026-07-23 19:20:30.072289225 +0000 UTC m=+55.924065450 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/7c2f37e4-a8b7-497e-ac12-7a6f5dd86619-networking-console-plugin-cert") pod "networking-console-plugin-7bc7489cb5-4p45l" (UID: "7c2f37e4-a8b7-497e-ac12-7a6f5dd86619") : secret "networking-console-plugin-cert" not found Jul 23 19:20:29.880805 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:20:29.880064 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b7c6dedb-e5d3-41a0-b060-50a88ccca303-service-ca-bundle\") pod \"router-default-7f64bc459d-9b6vg\" (UID: \"b7c6dedb-e5d3-41a0-b060-50a88ccca303\") " pod="openshift-ingress/router-default-7f64bc459d-9b6vg" Jul 23 19:20:29.880805 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:20:29.880114 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/e8d01fa4-24e5-4b2b-bf87-5e5395933cee-samples-operator-tls\") pod \"cluster-samples-operator-79b7c55f87-8l58m\" (UID: \"e8d01fa4-24e5-4b2b-bf87-5e5395933cee\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-79b7c55f87-8l58m" Jul 23 19:20:29.880805 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:20:29.880183 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b7c6dedb-e5d3-41a0-b060-50a88ccca303-metrics-certs\") pod \"router-default-7f64bc459d-9b6vg\" (UID: \"b7c6dedb-e5d3-41a0-b060-50a88ccca303\") " pod="openshift-ingress/router-default-7f64bc459d-9b6vg" Jul 23 19:20:29.880805 ip-10-0-129-2 kubenswrapper[2564]: E0723 19:20:29.880368 2564 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found Jul 23 19:20:29.880805 ip-10-0-129-2 kubenswrapper[2564]: E0723 19:20:29.880436 2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b7c6dedb-e5d3-41a0-b060-50a88ccca303-metrics-certs podName:b7c6dedb-e5d3-41a0-b060-50a88ccca303 nodeName:}" failed. No retries permitted until 2026-07-23 19:20:31.880416875 +0000 UTC m=+57.732193104 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/b7c6dedb-e5d3-41a0-b060-50a88ccca303-metrics-certs") pod "router-default-7f64bc459d-9b6vg" (UID: "b7c6dedb-e5d3-41a0-b060-50a88ccca303") : secret "router-metrics-certs-default" not found Jul 23 19:20:29.880805 ip-10-0-129-2 kubenswrapper[2564]: E0723 19:20:29.880570 2564 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found Jul 23 19:20:29.880805 ip-10-0-129-2 kubenswrapper[2564]: E0723 19:20:29.880612 2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/b7c6dedb-e5d3-41a0-b060-50a88ccca303-service-ca-bundle podName:b7c6dedb-e5d3-41a0-b060-50a88ccca303 nodeName:}" failed. No retries permitted until 2026-07-23 19:20:31.880592587 +0000 UTC m=+57.732368834 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/b7c6dedb-e5d3-41a0-b060-50a88ccca303-service-ca-bundle") pod "router-default-7f64bc459d-9b6vg" (UID: "b7c6dedb-e5d3-41a0-b060-50a88ccca303") : configmap references non-existent config key: service-ca.crt Jul 23 19:20:29.880805 ip-10-0-129-2 kubenswrapper[2564]: E0723 19:20:29.880642 2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e8d01fa4-24e5-4b2b-bf87-5e5395933cee-samples-operator-tls podName:e8d01fa4-24e5-4b2b-bf87-5e5395933cee nodeName:}" failed. No retries permitted until 2026-07-23 19:20:31.880632804 +0000 UTC m=+57.732409029 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/e8d01fa4-24e5-4b2b-bf87-5e5395933cee-samples-operator-tls") pod "cluster-samples-operator-79b7c55f87-8l58m" (UID: "e8d01fa4-24e5-4b2b-bf87-5e5395933cee") : secret "samples-operator-tls" not found Jul 23 19:20:30.082050 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:20:30.082013 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/7c2f37e4-a8b7-497e-ac12-7a6f5dd86619-networking-console-plugin-cert\") pod \"networking-console-plugin-7bc7489cb5-4p45l\" (UID: \"7c2f37e4-a8b7-497e-ac12-7a6f5dd86619\") " pod="openshift-network-console/networking-console-plugin-7bc7489cb5-4p45l" Jul 23 19:20:30.082513 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:20:30.082185 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/6f0a5836-d7de-49fc-b3a2-49830eaa468f-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-76cf8966f8-vv2vj\" (UID: \"6f0a5836-d7de-49fc-b3a2-49830eaa468f\") " pod="openshift-monitoring/cluster-monitoring-operator-76cf8966f8-vv2vj" Jul 23 19:20:30.082513 ip-10-0-129-2 kubenswrapper[2564]: E0723 19:20:30.082409 2564 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found Jul 23 19:20:30.082513 ip-10-0-129-2 kubenswrapper[2564]: E0723 19:20:30.082477 2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7c2f37e4-a8b7-497e-ac12-7a6f5dd86619-networking-console-plugin-cert podName:7c2f37e4-a8b7-497e-ac12-7a6f5dd86619 nodeName:}" failed. No retries permitted until 2026-07-23 19:20:32.082455823 +0000 UTC m=+57.934232052 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/7c2f37e4-a8b7-497e-ac12-7a6f5dd86619-networking-console-plugin-cert") pod "networking-console-plugin-7bc7489cb5-4p45l" (UID: "7c2f37e4-a8b7-497e-ac12-7a6f5dd86619") : secret "networking-console-plugin-cert" not found Jul 23 19:20:30.083050 ip-10-0-129-2 kubenswrapper[2564]: E0723 19:20:30.083028 2564 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found Jul 23 19:20:30.083158 ip-10-0-129-2 kubenswrapper[2564]: E0723 19:20:30.083093 2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6f0a5836-d7de-49fc-b3a2-49830eaa468f-cluster-monitoring-operator-tls podName:6f0a5836-d7de-49fc-b3a2-49830eaa468f nodeName:}" failed. No retries permitted until 2026-07-23 19:20:32.083076565 +0000 UTC m=+57.934852792 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/6f0a5836-d7de-49fc-b3a2-49830eaa468f-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-76cf8966f8-vv2vj" (UID: "6f0a5836-d7de-49fc-b3a2-49830eaa468f") : secret "cluster-monitoring-operator-tls" not found Jul 23 19:20:30.759532 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:20:30.759460 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8rq79_25ca5174-01a0-41fc-9293-77dea2d9ec1f/konnectivity-agent/0.log" limit=1024 Jul 23 19:20:30.985125 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:20:30.985089 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-storage-operator/volume-data-source-validator-78cf4c6f6-zld8c" event={"ID":"07b02e43-f95a-4173-b027-d0979fbbca4e","Type":"ContainerStarted","Data":"d1c63f842aeac627c621a4ce56d4b502533b2db17db55304f853a66bc4cc8e0c"} Jul 23 19:20:31.001931 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:20:31.001874 2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-storage-operator/volume-data-source-validator-78cf4c6f6-zld8c" podStartSLOduration=1.374159693 podStartE2EDuration="3.001854701s" totalImagesPullingTime="1.627695008s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-23 19:20:28 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-23 19:20:31.001815932 +0000 UTC m=+56.853592197" watchObservedRunningTime="2026-07-23 19:20:31.001854701 +0000 UTC m=+56.853630949" Jul 23 19:20:31.120175 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:20:31.120095 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8rq79_25ca5174-01a0-41fc-9293-77dea2d9ec1f/konnectivity-agent/0.log" limit=1024 Jul 23 19:20:31.221565 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:20:31.221528 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8rq79_25ca5174-01a0-41fc-9293-77dea2d9ec1f/konnectivity-agent/0.log" limit=1024 Jul 23 19:20:31.899083 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:20:31.899051 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b7c6dedb-e5d3-41a0-b060-50a88ccca303-service-ca-bundle\") pod \"router-default-7f64bc459d-9b6vg\" (UID: \"b7c6dedb-e5d3-41a0-b060-50a88ccca303\") " pod="openshift-ingress/router-default-7f64bc459d-9b6vg" Jul 23 19:20:31.899260 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:20:31.899090 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/e8d01fa4-24e5-4b2b-bf87-5e5395933cee-samples-operator-tls\") pod \"cluster-samples-operator-79b7c55f87-8l58m\" (UID: \"e8d01fa4-24e5-4b2b-bf87-5e5395933cee\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-79b7c55f87-8l58m" Jul 23 19:20:31.899260 ip-10-0-129-2 kubenswrapper[2564]: E0723 19:20:31.899224 2564 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found Jul 23 19:20:31.899260 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:20:31.899235 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b7c6dedb-e5d3-41a0-b060-50a88ccca303-metrics-certs\") pod \"router-default-7f64bc459d-9b6vg\" (UID: \"b7c6dedb-e5d3-41a0-b060-50a88ccca303\") " pod="openshift-ingress/router-default-7f64bc459d-9b6vg" Jul 23 19:20:31.899434 ip-10-0-129-2 kubenswrapper[2564]: E0723 19:20:31.899293 2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e8d01fa4-24e5-4b2b-bf87-5e5395933cee-samples-operator-tls podName:e8d01fa4-24e5-4b2b-bf87-5e5395933cee nodeName:}" failed. No retries permitted until 2026-07-23 19:20:35.899273814 +0000 UTC m=+61.751050044 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/e8d01fa4-24e5-4b2b-bf87-5e5395933cee-samples-operator-tls") pod "cluster-samples-operator-79b7c55f87-8l58m" (UID: "e8d01fa4-24e5-4b2b-bf87-5e5395933cee") : secret "samples-operator-tls" not found Jul 23 19:20:31.899434 ip-10-0-129-2 kubenswrapper[2564]: E0723 19:20:31.899301 2564 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found Jul 23 19:20:31.899434 ip-10-0-129-2 kubenswrapper[2564]: E0723 19:20:31.899314 2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/b7c6dedb-e5d3-41a0-b060-50a88ccca303-service-ca-bundle podName:b7c6dedb-e5d3-41a0-b060-50a88ccca303 nodeName:}" failed. No retries permitted until 2026-07-23 19:20:35.899304901 +0000 UTC m=+61.751081126 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/b7c6dedb-e5d3-41a0-b060-50a88ccca303-service-ca-bundle") pod "router-default-7f64bc459d-9b6vg" (UID: "b7c6dedb-e5d3-41a0-b060-50a88ccca303") : configmap references non-existent config key: service-ca.crt Jul 23 19:20:31.899434 ip-10-0-129-2 kubenswrapper[2564]: E0723 19:20:31.899359 2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b7c6dedb-e5d3-41a0-b060-50a88ccca303-metrics-certs podName:b7c6dedb-e5d3-41a0-b060-50a88ccca303 nodeName:}" failed. No retries permitted until 2026-07-23 19:20:35.899346301 +0000 UTC m=+61.751122527 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/b7c6dedb-e5d3-41a0-b060-50a88ccca303-metrics-certs") pod "router-default-7f64bc459d-9b6vg" (UID: "b7c6dedb-e5d3-41a0-b060-50a88ccca303") : secret "router-metrics-certs-default" not found Jul 23 19:20:32.102015 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:20:32.101981 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/6f0a5836-d7de-49fc-b3a2-49830eaa468f-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-76cf8966f8-vv2vj\" (UID: \"6f0a5836-d7de-49fc-b3a2-49830eaa468f\") " pod="openshift-monitoring/cluster-monitoring-operator-76cf8966f8-vv2vj" Jul 23 19:20:32.102182 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:20:32.102054 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/7c2f37e4-a8b7-497e-ac12-7a6f5dd86619-networking-console-plugin-cert\") pod \"networking-console-plugin-7bc7489cb5-4p45l\" (UID: \"7c2f37e4-a8b7-497e-ac12-7a6f5dd86619\") " pod="openshift-network-console/networking-console-plugin-7bc7489cb5-4p45l" Jul 23 19:20:32.102182 ip-10-0-129-2 kubenswrapper[2564]: E0723 19:20:32.102129 2564 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found Jul 23 19:20:32.102288 ip-10-0-129-2 kubenswrapper[2564]: E0723 19:20:32.102191 2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6f0a5836-d7de-49fc-b3a2-49830eaa468f-cluster-monitoring-operator-tls podName:6f0a5836-d7de-49fc-b3a2-49830eaa468f nodeName:}" failed. No retries permitted until 2026-07-23 19:20:36.102171443 +0000 UTC m=+61.953947690 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/6f0a5836-d7de-49fc-b3a2-49830eaa468f-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-76cf8966f8-vv2vj" (UID: "6f0a5836-d7de-49fc-b3a2-49830eaa468f") : secret "cluster-monitoring-operator-tls" not found Jul 23 19:20:32.102288 ip-10-0-129-2 kubenswrapper[2564]: E0723 19:20:32.102207 2564 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found Jul 23 19:20:32.102288 ip-10-0-129-2 kubenswrapper[2564]: E0723 19:20:32.102244 2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7c2f37e4-a8b7-497e-ac12-7a6f5dd86619-networking-console-plugin-cert podName:7c2f37e4-a8b7-497e-ac12-7a6f5dd86619 nodeName:}" failed. No retries permitted until 2026-07-23 19:20:36.102233513 +0000 UTC m=+61.954009741 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/7c2f37e4-a8b7-497e-ac12-7a6f5dd86619-networking-console-plugin-cert") pod "networking-console-plugin-7bc7489cb5-4p45l" (UID: "7c2f37e4-a8b7-497e-ac12-7a6f5dd86619") : secret "networking-console-plugin-cert" not found Jul 23 19:20:33.997343 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:20:33.997291 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-5bcbc94c78-86qdl" event={"ID":"678e0e44-d3bb-4269-99c9-43229d8f2291","Type":"ContainerStarted","Data":"97f6b99c91e2be492ba63b6543c5508e13570ebb4aade21c4c6f57c017bcdcf8"} Jul 23 19:20:33.998831 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:20:33.998804 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7658d9b749-6snc5" event={"ID":"a2c386d2-2c3a-4cf0-9a93-e18300e6905d","Type":"ContainerStarted","Data":"cf7d3650988b892d2d73179a08f88dfa49f72cc603d28c1ed2f1a371671d82c5"} Jul 23 19:20:34.000436 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:20:34.000414 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-555d9d7885-r8lv4" event={"ID":"00953f2a-6543-445a-8895-40525b4e3e25","Type":"ContainerStarted","Data":"cb1424ea8d08116b26738f3be39ba3dc94293d93ba47090549c11cbf6d17a6f2"} Jul 23 19:20:34.002146 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:20:34.002129 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-565b7b569d-z5wf4_ded5b3af-c74c-4d5a-aa77-62fe72d1dd29/console-operator/0.log" Jul 23 19:20:34.002246 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:20:34.002161 2564 generic.go:356] "Generic (PLEG): container finished" podID="ded5b3af-c74c-4d5a-aa77-62fe72d1dd29" containerID="9c80faf07a3a0e18a90a6c2eef9427f81269f51d5e9b2b213048936c7578abd0" exitCode=255 Jul 23 19:20:34.002246 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:20:34.002219 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-565b7b569d-z5wf4" event={"ID":"ded5b3af-c74c-4d5a-aa77-62fe72d1dd29","Type":"ContainerDied","Data":"9c80faf07a3a0e18a90a6c2eef9427f81269f51d5e9b2b213048936c7578abd0"} Jul 23 19:20:34.002563 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:20:34.002399 2564 scope.go:117] "RemoveContainer" containerID="9c80faf07a3a0e18a90a6c2eef9427f81269f51d5e9b2b213048936c7578abd0" Jul 23 19:20:34.004489 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:20:34.004468 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-source-7bc5b6bf85-4lf8g" event={"ID":"b76b29a4-cdf8-42d8-8694-ccbeab9a7908","Type":"ContainerStarted","Data":"ae0d52b10c895e3f955b7e3206c648e207620b91827715c8637307566ba0d82a"} Jul 23 19:20:34.015718 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:20:34.015675 2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-service-ca-operator/service-ca-operator-5bcbc94c78-86qdl" podStartSLOduration=1.146729908 podStartE2EDuration="6.015660347s" totalImagesPullingTime="4.868930439s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-23 19:20:28 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-23 19:20:34.015396081 +0000 UTC m=+59.867172343" watchObservedRunningTime="2026-07-23 19:20:34.015660347 +0000 UTC m=+59.867436577" Jul 23 19:20:34.045676 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:20:34.045620 2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-insights/insights-operator-555d9d7885-r8lv4" podStartSLOduration=0.937026223 podStartE2EDuration="6.045605603s" totalImagesPullingTime="5.10857938s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-23 19:20:28 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-23 19:20:34.044619199 +0000 UTC m=+59.896395484" watchObservedRunningTime="2026-07-23 19:20:34.045605603 +0000 UTC m=+59.897381849" Jul 23 19:20:34.075418 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:20:34.075172 2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7658d9b749-6snc5" podStartSLOduration=1.099828015 podStartE2EDuration="6.075157139s" totalImagesPullingTime="4.975329124s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-23 19:20:28 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-23 19:20:34.074593715 +0000 UTC m=+59.926369984" watchObservedRunningTime="2026-07-23 19:20:34.075157139 +0000 UTC m=+59.926933377" Jul 23 19:20:34.107464 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:20:34.107398 2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-source-7bc5b6bf85-4lf8g" podStartSLOduration=1.083240932 podStartE2EDuration="6.107383597s" totalImagesPullingTime="5.024142665s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-23 19:20:28 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-23 19:20:34.107360447 +0000 UTC m=+59.959136693" watchObservedRunningTime="2026-07-23 19:20:34.107383597 +0000 UTC m=+59.959159842" Jul 23 19:20:34.915942 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:20:34.915912 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-8dtlh" Jul 23 19:20:35.011192 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:20:35.011130 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-565b7b569d-z5wf4_ded5b3af-c74c-4d5a-aa77-62fe72d1dd29/console-operator/1.log" Jul 23 19:20:35.011694 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:20:35.011673 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-565b7b569d-z5wf4_ded5b3af-c74c-4d5a-aa77-62fe72d1dd29/console-operator/0.log" Jul 23 19:20:35.011770 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:20:35.011714 2564 generic.go:356] "Generic (PLEG): container finished" podID="ded5b3af-c74c-4d5a-aa77-62fe72d1dd29" containerID="892131f3b4a665a17c51f76de799ce69cec99ea7c1c3120611106c156f9bb714" exitCode=255 Jul 23 19:20:35.012288 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:20:35.012271 2564 scope.go:117] "RemoveContainer" containerID="892131f3b4a665a17c51f76de799ce69cec99ea7c1c3120611106c156f9bb714" Jul 23 19:20:35.012488 ip-10-0-129-2 kubenswrapper[2564]: E0723 19:20:35.012467 2564 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 10s restarting failed container=console-operator pod=console-operator-565b7b569d-z5wf4_openshift-console-operator(ded5b3af-c74c-4d5a-aa77-62fe72d1dd29)\"" pod="openshift-console-operator/console-operator-565b7b569d-z5wf4" podUID="ded5b3af-c74c-4d5a-aa77-62fe72d1dd29" Jul 23 19:20:35.012763 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:20:35.012744 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-565b7b569d-z5wf4" event={"ID":"ded5b3af-c74c-4d5a-aa77-62fe72d1dd29","Type":"ContainerDied","Data":"892131f3b4a665a17c51f76de799ce69cec99ea7c1c3120611106c156f9bb714"} Jul 23 19:20:35.012807 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:20:35.012793 2564 scope.go:117] "RemoveContainer" containerID="9c80faf07a3a0e18a90a6c2eef9427f81269f51d5e9b2b213048936c7578abd0" Jul 23 19:20:35.081887 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:20:35.081856 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8rq79_25ca5174-01a0-41fc-9293-77dea2d9ec1f/konnectivity-agent/0.log" limit=1024 Jul 23 19:20:35.139449 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:20:35.139427 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8rq79_25ca5174-01a0-41fc-9293-77dea2d9ec1f/konnectivity-agent/0.log" limit=1024 Jul 23 19:20:35.402152 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:20:35.402073 2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-runtime-extractor-8gqzv"] Jul 23 19:20:35.431011 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:20:35.430984 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-8gqzv"] Jul 23 19:20:35.431168 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:20:35.431098 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-8gqzv" Jul 23 19:20:35.433399 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:20:35.433379 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-v89gs\"" Jul 23 19:20:35.433533 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:20:35.433517 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\"" Jul 23 19:20:35.439005 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:20:35.438990 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\"" Jul 23 19:20:35.471557 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:20:35.471533 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8rq79_25ca5174-01a0-41fc-9293-77dea2d9ec1f/konnectivity-agent/0.log" limit=1024 Jul 23 19:20:35.533475 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:20:35.533448 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/e891cbfb-7697-43dd-9003-3190f8e8a844-crio-socket\") pod \"insights-runtime-extractor-8gqzv\" (UID: \"e891cbfb-7697-43dd-9003-3190f8e8a844\") " pod="openshift-insights/insights-runtime-extractor-8gqzv" Jul 23 19:20:35.533569 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:20:35.533517 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/e891cbfb-7697-43dd-9003-3190f8e8a844-data-volume\") pod \"insights-runtime-extractor-8gqzv\" (UID: \"e891cbfb-7697-43dd-9003-3190f8e8a844\") " pod="openshift-insights/insights-runtime-extractor-8gqzv" Jul 23 19:20:35.533569 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:20:35.533541 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/e891cbfb-7697-43dd-9003-3190f8e8a844-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-8gqzv\" (UID: \"e891cbfb-7697-43dd-9003-3190f8e8a844\") " pod="openshift-insights/insights-runtime-extractor-8gqzv" Jul 23 19:20:35.533569 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:20:35.533561 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-957rp\" (UniqueName: \"kubernetes.io/projected/e891cbfb-7697-43dd-9003-3190f8e8a844-kube-api-access-957rp\") pod \"insights-runtime-extractor-8gqzv\" (UID: \"e891cbfb-7697-43dd-9003-3190f8e8a844\") " pod="openshift-insights/insights-runtime-extractor-8gqzv" Jul 23 19:20:35.533665 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:20:35.533609 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/e891cbfb-7697-43dd-9003-3190f8e8a844-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-8gqzv\" (UID: \"e891cbfb-7697-43dd-9003-3190f8e8a844\") " pod="openshift-insights/insights-runtime-extractor-8gqzv" Jul 23 19:20:35.577553 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:20:35.577522 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8rq79_25ca5174-01a0-41fc-9293-77dea2d9ec1f/konnectivity-agent/0.log" limit=1024 Jul 23 19:20:35.634081 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:20:35.634058 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/e891cbfb-7697-43dd-9003-3190f8e8a844-crio-socket\") pod \"insights-runtime-extractor-8gqzv\" (UID: \"e891cbfb-7697-43dd-9003-3190f8e8a844\") " pod="openshift-insights/insights-runtime-extractor-8gqzv" Jul 23 19:20:35.634182 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:20:35.634141 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/e891cbfb-7697-43dd-9003-3190f8e8a844-data-volume\") pod \"insights-runtime-extractor-8gqzv\" (UID: \"e891cbfb-7697-43dd-9003-3190f8e8a844\") " pod="openshift-insights/insights-runtime-extractor-8gqzv" Jul 23 19:20:35.634238 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:20:35.634176 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/e891cbfb-7697-43dd-9003-3190f8e8a844-crio-socket\") pod \"insights-runtime-extractor-8gqzv\" (UID: \"e891cbfb-7697-43dd-9003-3190f8e8a844\") " pod="openshift-insights/insights-runtime-extractor-8gqzv" Jul 23 19:20:35.634238 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:20:35.634184 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/e891cbfb-7697-43dd-9003-3190f8e8a844-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-8gqzv\" (UID: \"e891cbfb-7697-43dd-9003-3190f8e8a844\") " pod="openshift-insights/insights-runtime-extractor-8gqzv" Jul 23 19:20:35.634361 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:20:35.634236 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-957rp\" (UniqueName: \"kubernetes.io/projected/e891cbfb-7697-43dd-9003-3190f8e8a844-kube-api-access-957rp\") pod \"insights-runtime-extractor-8gqzv\" (UID: \"e891cbfb-7697-43dd-9003-3190f8e8a844\") " pod="openshift-insights/insights-runtime-extractor-8gqzv" Jul 23 19:20:35.634361 ip-10-0-129-2 kubenswrapper[2564]: E0723 19:20:35.634281 2564 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Jul 23 19:20:35.634361 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:20:35.634303 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/e891cbfb-7697-43dd-9003-3190f8e8a844-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-8gqzv\" (UID: \"e891cbfb-7697-43dd-9003-3190f8e8a844\") " pod="openshift-insights/insights-runtime-extractor-8gqzv" Jul 23 19:20:35.634500 ip-10-0-129-2 kubenswrapper[2564]: E0723 19:20:35.634380 2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e891cbfb-7697-43dd-9003-3190f8e8a844-insights-runtime-extractor-tls podName:e891cbfb-7697-43dd-9003-3190f8e8a844 nodeName:}" failed. No retries permitted until 2026-07-23 19:20:36.134362041 +0000 UTC m=+61.986138273 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/e891cbfb-7697-43dd-9003-3190f8e8a844-insights-runtime-extractor-tls") pod "insights-runtime-extractor-8gqzv" (UID: "e891cbfb-7697-43dd-9003-3190f8e8a844") : secret "insights-runtime-extractor-tls" not found Jul 23 19:20:35.634500 ip-10-0-129-2 kubenswrapper[2564]: E0723 19:20:35.634441 2564 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Jul 23 19:20:35.634603 ip-10-0-129-2 kubenswrapper[2564]: E0723 19:20:35.634509 2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/e891cbfb-7697-43dd-9003-3190f8e8a844-kube-rbac-proxy-cm podName:e891cbfb-7697-43dd-9003-3190f8e8a844 nodeName:}" failed. No retries permitted until 2026-07-23 19:20:36.134495458 +0000 UTC m=+61.986271683 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/e891cbfb-7697-43dd-9003-3190f8e8a844-kube-rbac-proxy-cm") pod "insights-runtime-extractor-8gqzv" (UID: "e891cbfb-7697-43dd-9003-3190f8e8a844") : configmap "kube-rbac-proxy" not found Jul 23 19:20:35.634603 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:20:35.634513 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/e891cbfb-7697-43dd-9003-3190f8e8a844-data-volume\") pod \"insights-runtime-extractor-8gqzv\" (UID: \"e891cbfb-7697-43dd-9003-3190f8e8a844\") " pod="openshift-insights/insights-runtime-extractor-8gqzv" Jul 23 19:20:35.643182 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:20:35.643161 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-957rp\" (UniqueName: \"kubernetes.io/projected/e891cbfb-7697-43dd-9003-3190f8e8a844-kube-api-access-957rp\") pod \"insights-runtime-extractor-8gqzv\" (UID: \"e891cbfb-7697-43dd-9003-3190f8e8a844\") " pod="openshift-insights/insights-runtime-extractor-8gqzv" Jul 23 19:20:35.715875 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:20:35.715843 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8rq79_25ca5174-01a0-41fc-9293-77dea2d9ec1f/konnectivity-agent/0.log" limit=1024 Jul 23 19:20:35.902327 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:20:35.902303 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8rq79_25ca5174-01a0-41fc-9293-77dea2d9ec1f/konnectivity-agent/0.log" limit=1024 Jul 23 19:20:35.937094 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:20:35.937073 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b7c6dedb-e5d3-41a0-b060-50a88ccca303-service-ca-bundle\") pod \"router-default-7f64bc459d-9b6vg\" (UID: \"b7c6dedb-e5d3-41a0-b060-50a88ccca303\") " pod="openshift-ingress/router-default-7f64bc459d-9b6vg" Jul 23 19:20:35.937170 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:20:35.937106 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/e8d01fa4-24e5-4b2b-bf87-5e5395933cee-samples-operator-tls\") pod \"cluster-samples-operator-79b7c55f87-8l58m\" (UID: \"e8d01fa4-24e5-4b2b-bf87-5e5395933cee\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-79b7c55f87-8l58m" Jul 23 19:20:35.937207 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:20:35.937170 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b7c6dedb-e5d3-41a0-b060-50a88ccca303-metrics-certs\") pod \"router-default-7f64bc459d-9b6vg\" (UID: \"b7c6dedb-e5d3-41a0-b060-50a88ccca303\") " pod="openshift-ingress/router-default-7f64bc459d-9b6vg" Jul 23 19:20:35.937240 ip-10-0-129-2 kubenswrapper[2564]: E0723 19:20:35.937229 2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/b7c6dedb-e5d3-41a0-b060-50a88ccca303-service-ca-bundle podName:b7c6dedb-e5d3-41a0-b060-50a88ccca303 nodeName:}" failed. No retries permitted until 2026-07-23 19:20:43.937211478 +0000 UTC m=+69.788987720 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/b7c6dedb-e5d3-41a0-b060-50a88ccca303-service-ca-bundle") pod "router-default-7f64bc459d-9b6vg" (UID: "b7c6dedb-e5d3-41a0-b060-50a88ccca303") : configmap references non-existent config key: service-ca.crt Jul 23 19:20:35.937301 ip-10-0-129-2 kubenswrapper[2564]: E0723 19:20:35.937282 2564 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found Jul 23 19:20:35.937359 ip-10-0-129-2 kubenswrapper[2564]: E0723 19:20:35.937283 2564 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found Jul 23 19:20:35.937393 ip-10-0-129-2 kubenswrapper[2564]: E0723 19:20:35.937360 2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e8d01fa4-24e5-4b2b-bf87-5e5395933cee-samples-operator-tls podName:e8d01fa4-24e5-4b2b-bf87-5e5395933cee nodeName:}" failed. No retries permitted until 2026-07-23 19:20:43.937343447 +0000 UTC m=+69.789119677 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/e8d01fa4-24e5-4b2b-bf87-5e5395933cee-samples-operator-tls") pod "cluster-samples-operator-79b7c55f87-8l58m" (UID: "e8d01fa4-24e5-4b2b-bf87-5e5395933cee") : secret "samples-operator-tls" not found Jul 23 19:20:35.937429 ip-10-0-129-2 kubenswrapper[2564]: E0723 19:20:35.937412 2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b7c6dedb-e5d3-41a0-b060-50a88ccca303-metrics-certs podName:b7c6dedb-e5d3-41a0-b060-50a88ccca303 nodeName:}" failed. No retries permitted until 2026-07-23 19:20:43.937396229 +0000 UTC m=+69.789172458 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/b7c6dedb-e5d3-41a0-b060-50a88ccca303-metrics-certs") pod "router-default-7f64bc459d-9b6vg" (UID: "b7c6dedb-e5d3-41a0-b060-50a88ccca303") : secret "router-metrics-certs-default" not found Jul 23 19:20:36.002399 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:20:36.002345 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8rq79_25ca5174-01a0-41fc-9293-77dea2d9ec1f/konnectivity-agent/0.log" limit=1024 Jul 23 19:20:36.015957 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:20:36.015931 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-565b7b569d-z5wf4_ded5b3af-c74c-4d5a-aa77-62fe72d1dd29/console-operator/1.log" Jul 23 19:20:36.016256 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:20:36.016231 2564 scope.go:117] "RemoveContainer" containerID="892131f3b4a665a17c51f76de799ce69cec99ea7c1c3120611106c156f9bb714" Jul 23 19:20:36.016401 ip-10-0-129-2 kubenswrapper[2564]: E0723 19:20:36.016386 2564 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 10s restarting failed container=console-operator pod=console-operator-565b7b569d-z5wf4_openshift-console-operator(ded5b3af-c74c-4d5a-aa77-62fe72d1dd29)\"" pod="openshift-console-operator/console-operator-565b7b569d-z5wf4" podUID="ded5b3af-c74c-4d5a-aa77-62fe72d1dd29" Jul 23 19:20:36.139801 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:20:36.139773 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/6f0a5836-d7de-49fc-b3a2-49830eaa468f-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-76cf8966f8-vv2vj\" (UID: \"6f0a5836-d7de-49fc-b3a2-49830eaa468f\") " pod="openshift-monitoring/cluster-monitoring-operator-76cf8966f8-vv2vj" Jul 23 19:20:36.139895 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:20:36.139832 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/7c2f37e4-a8b7-497e-ac12-7a6f5dd86619-networking-console-plugin-cert\") pod \"networking-console-plugin-7bc7489cb5-4p45l\" (UID: \"7c2f37e4-a8b7-497e-ac12-7a6f5dd86619\") " pod="openshift-network-console/networking-console-plugin-7bc7489cb5-4p45l" Jul 23 19:20:36.139895 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:20:36.139882 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/e891cbfb-7697-43dd-9003-3190f8e8a844-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-8gqzv\" (UID: \"e891cbfb-7697-43dd-9003-3190f8e8a844\") " pod="openshift-insights/insights-runtime-extractor-8gqzv" Jul 23 19:20:36.139968 ip-10-0-129-2 kubenswrapper[2564]: E0723 19:20:36.139905 2564 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found Jul 23 19:20:36.139968 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:20:36.139931 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/e891cbfb-7697-43dd-9003-3190f8e8a844-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-8gqzv\" (UID: \"e891cbfb-7697-43dd-9003-3190f8e8a844\") " pod="openshift-insights/insights-runtime-extractor-8gqzv" Jul 23 19:20:36.139968 ip-10-0-129-2 kubenswrapper[2564]: E0723 19:20:36.139945 2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7c2f37e4-a8b7-497e-ac12-7a6f5dd86619-networking-console-plugin-cert podName:7c2f37e4-a8b7-497e-ac12-7a6f5dd86619 nodeName:}" failed. No retries permitted until 2026-07-23 19:20:44.139935756 +0000 UTC m=+69.991711981 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/7c2f37e4-a8b7-497e-ac12-7a6f5dd86619-networking-console-plugin-cert") pod "networking-console-plugin-7bc7489cb5-4p45l" (UID: "7c2f37e4-a8b7-497e-ac12-7a6f5dd86619") : secret "networking-console-plugin-cert" not found Jul 23 19:20:36.139968 ip-10-0-129-2 kubenswrapper[2564]: E0723 19:20:36.139964 2564 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Jul 23 19:20:36.140115 ip-10-0-129-2 kubenswrapper[2564]: E0723 19:20:36.140004 2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/e891cbfb-7697-43dd-9003-3190f8e8a844-kube-rbac-proxy-cm podName:e891cbfb-7697-43dd-9003-3190f8e8a844 nodeName:}" failed. No retries permitted until 2026-07-23 19:20:37.139990673 +0000 UTC m=+62.991766912 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/e891cbfb-7697-43dd-9003-3190f8e8a844-kube-rbac-proxy-cm") pod "insights-runtime-extractor-8gqzv" (UID: "e891cbfb-7697-43dd-9003-3190f8e8a844") : configmap "kube-rbac-proxy" not found Jul 23 19:20:36.140115 ip-10-0-129-2 kubenswrapper[2564]: E0723 19:20:36.140019 2564 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found Jul 23 19:20:36.140115 ip-10-0-129-2 kubenswrapper[2564]: E0723 19:20:36.140033 2564 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Jul 23 19:20:36.140115 ip-10-0-129-2 kubenswrapper[2564]: E0723 19:20:36.140060 2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6f0a5836-d7de-49fc-b3a2-49830eaa468f-cluster-monitoring-operator-tls podName:6f0a5836-d7de-49fc-b3a2-49830eaa468f nodeName:}" failed. No retries permitted until 2026-07-23 19:20:44.140048065 +0000 UTC m=+69.991824291 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/6f0a5836-d7de-49fc-b3a2-49830eaa468f-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-76cf8966f8-vv2vj" (UID: "6f0a5836-d7de-49fc-b3a2-49830eaa468f") : secret "cluster-monitoring-operator-tls" not found Jul 23 19:20:36.140115 ip-10-0-129-2 kubenswrapper[2564]: E0723 19:20:36.140078 2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e891cbfb-7697-43dd-9003-3190f8e8a844-insights-runtime-extractor-tls podName:e891cbfb-7697-43dd-9003-3190f8e8a844 nodeName:}" failed. No retries permitted until 2026-07-23 19:20:37.140069362 +0000 UTC m=+62.991845587 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/e891cbfb-7697-43dd-9003-3190f8e8a844-insights-runtime-extractor-tls") pod "insights-runtime-extractor-8gqzv" (UID: "e891cbfb-7697-43dd-9003-3190f8e8a844") : secret "insights-runtime-extractor-tls" not found Jul 23 19:20:36.183424 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:20:36.183400 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8rq79_25ca5174-01a0-41fc-9293-77dea2d9ec1f/konnectivity-agent/0.log" limit=1024 Jul 23 19:20:36.280631 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:20:36.280573 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8rq79_25ca5174-01a0-41fc-9293-77dea2d9ec1f/konnectivity-agent/0.log" limit=1024 Jul 23 19:20:36.792163 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:20:36.792145 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-6k8wr_23313c02-67bf-498f-a294-e16cdc7d4517/dns-node-resolver/0.log" Jul 23 19:20:37.149562 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:20:37.149513 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/e891cbfb-7697-43dd-9003-3190f8e8a844-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-8gqzv\" (UID: \"e891cbfb-7697-43dd-9003-3190f8e8a844\") " pod="openshift-insights/insights-runtime-extractor-8gqzv" Jul 23 19:20:37.149832 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:20:37.149564 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/e891cbfb-7697-43dd-9003-3190f8e8a844-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-8gqzv\" (UID: \"e891cbfb-7697-43dd-9003-3190f8e8a844\") " pod="openshift-insights/insights-runtime-extractor-8gqzv" Jul 23 19:20:37.149832 ip-10-0-129-2 kubenswrapper[2564]: E0723 19:20:37.149629 2564 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Jul 23 19:20:37.149832 ip-10-0-129-2 kubenswrapper[2564]: E0723 19:20:37.149656 2564 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Jul 23 19:20:37.149832 ip-10-0-129-2 kubenswrapper[2564]: E0723 19:20:37.149679 2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/e891cbfb-7697-43dd-9003-3190f8e8a844-kube-rbac-proxy-cm podName:e891cbfb-7697-43dd-9003-3190f8e8a844 nodeName:}" failed. No retries permitted until 2026-07-23 19:20:39.149667107 +0000 UTC m=+65.001443332 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/e891cbfb-7697-43dd-9003-3190f8e8a844-kube-rbac-proxy-cm") pod "insights-runtime-extractor-8gqzv" (UID: "e891cbfb-7697-43dd-9003-3190f8e8a844") : configmap "kube-rbac-proxy" not found Jul 23 19:20:37.149832 ip-10-0-129-2 kubenswrapper[2564]: E0723 19:20:37.149718 2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e891cbfb-7697-43dd-9003-3190f8e8a844-insights-runtime-extractor-tls podName:e891cbfb-7697-43dd-9003-3190f8e8a844 nodeName:}" failed. No retries permitted until 2026-07-23 19:20:39.149702975 +0000 UTC m=+65.001479206 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/e891cbfb-7697-43dd-9003-3190f8e8a844-insights-runtime-extractor-tls") pod "insights-runtime-extractor-8gqzv" (UID: "e891cbfb-7697-43dd-9003-3190f8e8a844") : secret "insights-runtime-extractor-tls" not found Jul 23 19:20:37.777233 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:20:37.777211 2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-service-ca/service-ca-58598695f7-zl8zg"] Jul 23 19:20:37.780368 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:20:37.780354 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca/service-ca-58598695f7-zl8zg" Jul 23 19:20:37.782641 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:20:37.782623 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca\"/\"signing-key\"" Jul 23 19:20:37.782641 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:20:37.782631 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"openshift-service-ca.crt\"" Jul 23 19:20:37.783413 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:20:37.783393 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"signing-cabundle\"" Jul 23 19:20:37.783509 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:20:37.783417 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"kube-root-ca.crt\"" Jul 23 19:20:37.783509 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:20:37.783401 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca\"/\"service-ca-dockercfg-vhsqs\"" Jul 23 19:20:37.790768 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:20:37.790748 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca/service-ca-58598695f7-zl8zg"] Jul 23 19:20:37.792502 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:20:37.792485 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-9wwn6_dd466782-4d8e-42e2-83c2-ed5f1b48567b/node-ca/0.log" Jul 23 19:20:37.811722 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:20:37.811696 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8rq79_25ca5174-01a0-41fc-9293-77dea2d9ec1f/konnectivity-agent/0.log" limit=1024 Jul 23 19:20:37.854962 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:20:37.854940 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tvm94\" (UniqueName: \"kubernetes.io/projected/bdeb485b-2ab7-4871-8f27-cdfb4f832c0d-kube-api-access-tvm94\") pod \"service-ca-58598695f7-zl8zg\" (UID: \"bdeb485b-2ab7-4871-8f27-cdfb4f832c0d\") " pod="openshift-service-ca/service-ca-58598695f7-zl8zg" Jul 23 19:20:37.855055 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:20:37.855040 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/bdeb485b-2ab7-4871-8f27-cdfb4f832c0d-signing-key\") pod \"service-ca-58598695f7-zl8zg\" (UID: \"bdeb485b-2ab7-4871-8f27-cdfb4f832c0d\") " pod="openshift-service-ca/service-ca-58598695f7-zl8zg" Jul 23 19:20:37.855103 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:20:37.855067 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/bdeb485b-2ab7-4871-8f27-cdfb4f832c0d-signing-cabundle\") pod \"service-ca-58598695f7-zl8zg\" (UID: \"bdeb485b-2ab7-4871-8f27-cdfb4f832c0d\") " pod="openshift-service-ca/service-ca-58598695f7-zl8zg" Jul 23 19:20:37.904999 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:20:37.904978 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8rq79_25ca5174-01a0-41fc-9293-77dea2d9ec1f/konnectivity-agent/0.log" limit=1024 Jul 23 19:20:37.956132 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:20:37.956109 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/bdeb485b-2ab7-4871-8f27-cdfb4f832c0d-signing-key\") pod \"service-ca-58598695f7-zl8zg\" (UID: \"bdeb485b-2ab7-4871-8f27-cdfb4f832c0d\") " pod="openshift-service-ca/service-ca-58598695f7-zl8zg" Jul 23 19:20:37.956209 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:20:37.956143 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/bdeb485b-2ab7-4871-8f27-cdfb4f832c0d-signing-cabundle\") pod \"service-ca-58598695f7-zl8zg\" (UID: \"bdeb485b-2ab7-4871-8f27-cdfb4f832c0d\") " pod="openshift-service-ca/service-ca-58598695f7-zl8zg" Jul 23 19:20:37.956209 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:20:37.956194 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tvm94\" (UniqueName: \"kubernetes.io/projected/bdeb485b-2ab7-4871-8f27-cdfb4f832c0d-kube-api-access-tvm94\") pod \"service-ca-58598695f7-zl8zg\" (UID: \"bdeb485b-2ab7-4871-8f27-cdfb4f832c0d\") " pod="openshift-service-ca/service-ca-58598695f7-zl8zg" Jul 23 19:20:37.956782 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:20:37.956767 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/bdeb485b-2ab7-4871-8f27-cdfb4f832c0d-signing-cabundle\") pod \"service-ca-58598695f7-zl8zg\" (UID: \"bdeb485b-2ab7-4871-8f27-cdfb4f832c0d\") " pod="openshift-service-ca/service-ca-58598695f7-zl8zg" Jul 23 19:20:37.958536 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:20:37.958520 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/bdeb485b-2ab7-4871-8f27-cdfb4f832c0d-signing-key\") pod \"service-ca-58598695f7-zl8zg\" (UID: \"bdeb485b-2ab7-4871-8f27-cdfb4f832c0d\") " pod="openshift-service-ca/service-ca-58598695f7-zl8zg" Jul 23 19:20:37.965176 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:20:37.965154 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-tvm94\" (UniqueName: \"kubernetes.io/projected/bdeb485b-2ab7-4871-8f27-cdfb4f832c0d-kube-api-access-tvm94\") pod \"service-ca-58598695f7-zl8zg\" (UID: \"bdeb485b-2ab7-4871-8f27-cdfb4f832c0d\") " pod="openshift-service-ca/service-ca-58598695f7-zl8zg" Jul 23 19:20:37.999256 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:20:37.999217 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8rq79_25ca5174-01a0-41fc-9293-77dea2d9ec1f/konnectivity-agent/0.log" limit=1024 Jul 23 19:20:38.088653 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:20:38.088596 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca/service-ca-58598695f7-zl8zg" Jul 23 19:20:38.095503 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:20:38.095474 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8rq79_25ca5174-01a0-41fc-9293-77dea2d9ec1f/konnectivity-agent/0.log" limit=1024 Jul 23 19:20:38.206366 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:20:38.206337 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca/service-ca-58598695f7-zl8zg"] Jul 23 19:20:38.209128 ip-10-0-129-2 kubenswrapper[2564]: W0723 19:20:38.209102 2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbdeb485b_2ab7_4871_8f27_cdfb4f832c0d.slice/crio-b3eaea54e72080cbf216903522f8dc3668a4ee8bb9d45a2c8cc284681529411c WatchSource:0}: Error finding container b3eaea54e72080cbf216903522f8dc3668a4ee8bb9d45a2c8cc284681529411c: Status 404 returned error can't find the container with id b3eaea54e72080cbf216903522f8dc3668a4ee8bb9d45a2c8cc284681529411c Jul 23 19:20:38.519115 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:20:38.519089 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console-operator/console-operator-565b7b569d-z5wf4" Jul 23 19:20:38.519221 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:20:38.519140 2564 kubelet.go:2654] "SyncLoop (probe)" probe="liveness" status="unhealthy" pod="openshift-console-operator/console-operator-565b7b569d-z5wf4" Jul 23 19:20:38.519462 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:20:38.519449 2564 scope.go:117] "RemoveContainer" containerID="892131f3b4a665a17c51f76de799ce69cec99ea7c1c3120611106c156f9bb714" Jul 23 19:20:38.519601 ip-10-0-129-2 kubenswrapper[2564]: E0723 19:20:38.519585 2564 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 10s restarting failed container=console-operator pod=console-operator-565b7b569d-z5wf4_openshift-console-operator(ded5b3af-c74c-4d5a-aa77-62fe72d1dd29)\"" pod="openshift-console-operator/console-operator-565b7b569d-z5wf4" podUID="ded5b3af-c74c-4d5a-aa77-62fe72d1dd29" Jul 23 19:20:38.718918 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:20:38.718892 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8rq79_25ca5174-01a0-41fc-9293-77dea2d9ec1f/konnectivity-agent/0.log" limit=1024 Jul 23 19:20:38.790665 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:20:38.790614 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-wxwk8_e6d7c065-91d9-461c-8c0f-684ad4ea70e6/serve-healthcheck-canary/0.log" Jul 23 19:20:38.827794 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:20:38.827767 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8rq79_25ca5174-01a0-41fc-9293-77dea2d9ec1f/konnectivity-agent/0.log" limit=1024 Jul 23 19:20:38.893366 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:20:38.893344 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8rq79_25ca5174-01a0-41fc-9293-77dea2d9ec1f/konnectivity-agent/0.log" limit=1024 Jul 23 19:20:39.031231 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:20:39.031202 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca/service-ca-58598695f7-zl8zg" event={"ID":"bdeb485b-2ab7-4871-8f27-cdfb4f832c0d","Type":"ContainerStarted","Data":"0168cdfba8d982a85cf5f7cd13034051368b2d7c85ddbaf5cb41fab061259c80"} Jul 23 19:20:39.031341 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:20:39.031293 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca/service-ca-58598695f7-zl8zg" event={"ID":"bdeb485b-2ab7-4871-8f27-cdfb4f832c0d","Type":"ContainerStarted","Data":"b3eaea54e72080cbf216903522f8dc3668a4ee8bb9d45a2c8cc284681529411c"} Jul 23 19:20:39.056138 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:20:39.056077 2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-service-ca/service-ca-58598695f7-zl8zg" podStartSLOduration=2.056065439 podStartE2EDuration="2.056065439s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-23 19:20:37 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-23 19:20:39.05447478 +0000 UTC m=+64.906251032" watchObservedRunningTime="2026-07-23 19:20:39.056065439 +0000 UTC m=+64.907841724" Jul 23 19:20:39.168272 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:20:39.168248 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/e891cbfb-7697-43dd-9003-3190f8e8a844-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-8gqzv\" (UID: \"e891cbfb-7697-43dd-9003-3190f8e8a844\") " pod="openshift-insights/insights-runtime-extractor-8gqzv" Jul 23 19:20:39.168383 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:20:39.168314 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/e891cbfb-7697-43dd-9003-3190f8e8a844-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-8gqzv\" (UID: \"e891cbfb-7697-43dd-9003-3190f8e8a844\") " pod="openshift-insights/insights-runtime-extractor-8gqzv" Jul 23 19:20:39.168441 ip-10-0-129-2 kubenswrapper[2564]: E0723 19:20:39.168396 2564 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Jul 23 19:20:39.168441 ip-10-0-129-2 kubenswrapper[2564]: E0723 19:20:39.168437 2564 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Jul 23 19:20:39.168504 ip-10-0-129-2 kubenswrapper[2564]: E0723 19:20:39.168462 2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e891cbfb-7697-43dd-9003-3190f8e8a844-insights-runtime-extractor-tls podName:e891cbfb-7697-43dd-9003-3190f8e8a844 nodeName:}" failed. No retries permitted until 2026-07-23 19:20:43.168445029 +0000 UTC m=+69.020221255 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/e891cbfb-7697-43dd-9003-3190f8e8a844-insights-runtime-extractor-tls") pod "insights-runtime-extractor-8gqzv" (UID: "e891cbfb-7697-43dd-9003-3190f8e8a844") : secret "insights-runtime-extractor-tls" not found Jul 23 19:20:39.168504 ip-10-0-129-2 kubenswrapper[2564]: E0723 19:20:39.168478 2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/e891cbfb-7697-43dd-9003-3190f8e8a844-kube-rbac-proxy-cm podName:e891cbfb-7697-43dd-9003-3190f8e8a844 nodeName:}" failed. No retries permitted until 2026-07-23 19:20:43.168470328 +0000 UTC m=+69.020246560 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/e891cbfb-7697-43dd-9003-3190f8e8a844-kube-rbac-proxy-cm") pod "insights-runtime-extractor-8gqzv" (UID: "e891cbfb-7697-43dd-9003-3190f8e8a844") : configmap "kube-rbac-proxy" not found Jul 23 19:20:39.534892 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:20:39.534856 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8rq79_25ca5174-01a0-41fc-9293-77dea2d9ec1f/konnectivity-agent/0.log" limit=1024 Jul 23 19:20:39.633175 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:20:39.633143 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8rq79_25ca5174-01a0-41fc-9293-77dea2d9ec1f/konnectivity-agent/0.log" limit=1024 Jul 23 19:20:39.777269 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:20:39.777238 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8rq79_25ca5174-01a0-41fc-9293-77dea2d9ec1f/konnectivity-agent/0.log" limit=1024 Jul 23 19:20:39.880367 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:20:39.880263 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8rq79_25ca5174-01a0-41fc-9293-77dea2d9ec1f/konnectivity-agent/0.log" limit=1024 Jul 23 19:20:39.976410 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:20:39.976385 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8rq79_25ca5174-01a0-41fc-9293-77dea2d9ec1f/konnectivity-agent/0.log" limit=1024 Jul 23 19:20:40.075071 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:20:40.075045 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8rq79_25ca5174-01a0-41fc-9293-77dea2d9ec1f/konnectivity-agent/0.log" limit=1024 Jul 23 19:20:40.249724 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:20:40.249695 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8rq79_25ca5174-01a0-41fc-9293-77dea2d9ec1f/konnectivity-agent/0.log" limit=1024 Jul 23 19:20:40.380516 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:20:40.380491 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/576f76ed-730c-49c2-a33e-1dc11370cad4-metrics-certs\") pod \"network-metrics-daemon-qhrz5\" (UID: \"576f76ed-730c-49c2-a33e-1dc11370cad4\") " pod="openshift-multus/network-metrics-daemon-qhrz5" Jul 23 19:20:40.383069 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:20:40.383053 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\"" Jul 23 19:20:40.391010 ip-10-0-129-2 kubenswrapper[2564]: E0723 19:20:40.390991 2564 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found Jul 23 19:20:40.391078 ip-10-0-129-2 kubenswrapper[2564]: E0723 19:20:40.391045 2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/576f76ed-730c-49c2-a33e-1dc11370cad4-metrics-certs podName:576f76ed-730c-49c2-a33e-1dc11370cad4 nodeName:}" failed. No retries permitted until 2026-07-23 19:21:44.391032001 +0000 UTC m=+130.242808226 (durationBeforeRetry 1m4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/576f76ed-730c-49c2-a33e-1dc11370cad4-metrics-certs") pod "network-metrics-daemon-qhrz5" (UID: "576f76ed-730c-49c2-a33e-1dc11370cad4") : secret "metrics-daemon-secret" not found Jul 23 19:20:40.481588 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:20:40.481567 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cq8ll\" (UniqueName: \"kubernetes.io/projected/8706aab3-bba6-4fff-b8bc-8890d14daa9c-kube-api-access-cq8ll\") pod \"network-check-target-vxngp\" (UID: \"8706aab3-bba6-4fff-b8bc-8890d14daa9c\") " pod="openshift-network-diagnostics/network-check-target-vxngp" Jul 23 19:20:40.481664 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:20:40.481594 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/40f386ec-607d-496f-b65b-03a1fd847b1f-original-pull-secret\") pod \"global-pull-secret-syncer-rtgbq\" (UID: \"40f386ec-607d-496f-b65b-03a1fd847b1f\") " pod="kube-system/global-pull-secret-syncer-rtgbq" Jul 23 19:20:40.483993 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:20:40.483978 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\"" Jul 23 19:20:40.484558 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:20:40.484541 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-cq8ll\" (UniqueName: \"kubernetes.io/projected/8706aab3-bba6-4fff-b8bc-8890d14daa9c-kube-api-access-cq8ll\") pod \"network-check-target-vxngp\" (UID: \"8706aab3-bba6-4fff-b8bc-8890d14daa9c\") " pod="openshift-network-diagnostics/network-check-target-vxngp" Jul 23 19:20:40.493978 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:20:40.493959 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/40f386ec-607d-496f-b65b-03a1fd847b1f-original-pull-secret\") pod \"global-pull-secret-syncer-rtgbq\" (UID: \"40f386ec-607d-496f-b65b-03a1fd847b1f\") " pod="kube-system/global-pull-secret-syncer-rtgbq" Jul 23 19:20:40.663916 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:20:40.663899 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-t9qhx\"" Jul 23 19:20:40.672720 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:20:40.672706 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-vxngp" Jul 23 19:20:40.765923 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:20:40.765894 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-rtgbq" Jul 23 19:20:40.795892 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:20:40.795858 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-vxngp"] Jul 23 19:20:40.797690 ip-10-0-129-2 kubenswrapper[2564]: W0723 19:20:40.797670 2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8706aab3_bba6_4fff_b8bc_8890d14daa9c.slice/crio-492f4b8fd3995d551f676f49d63efa22ad296300d02b2b63d7b796e8c08e026f WatchSource:0}: Error finding container 492f4b8fd3995d551f676f49d63efa22ad296300d02b2b63d7b796e8c08e026f: Status 404 returned error can't find the container with id 492f4b8fd3995d551f676f49d63efa22ad296300d02b2b63d7b796e8c08e026f Jul 23 19:20:40.813655 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:20:40.813629 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8rq79_25ca5174-01a0-41fc-9293-77dea2d9ec1f/konnectivity-agent/0.log" limit=1024 Jul 23 19:20:40.885982 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:20:40.885952 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8rq79_25ca5174-01a0-41fc-9293-77dea2d9ec1f/konnectivity-agent/0.log" limit=1024 Jul 23 19:20:40.892340 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:20:40.892301 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-rtgbq"] Jul 23 19:20:40.894802 ip-10-0-129-2 kubenswrapper[2564]: W0723 19:20:40.894782 2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod40f386ec_607d_496f_b65b_03a1fd847b1f.slice/crio-2bda7cd69636320254083db429d6c64500dcfa5a45308fa02cbc65d564d31c54 WatchSource:0}: Error finding container 2bda7cd69636320254083db429d6c64500dcfa5a45308fa02cbc65d564d31c54: Status 404 returned error can't find the container with id 2bda7cd69636320254083db429d6c64500dcfa5a45308fa02cbc65d564d31c54 Jul 23 19:20:41.040094 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:20:41.040042 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-vxngp" event={"ID":"8706aab3-bba6-4fff-b8bc-8890d14daa9c","Type":"ContainerStarted","Data":"0ebf0fdb6ed26abc6ec667eec1353f97862fa9b4ecfc0844ab70427a4794fbb9"} Jul 23 19:20:41.040094 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:20:41.040085 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-vxngp" event={"ID":"8706aab3-bba6-4fff-b8bc-8890d14daa9c","Type":"ContainerStarted","Data":"492f4b8fd3995d551f676f49d63efa22ad296300d02b2b63d7b796e8c08e026f"} Jul 23 19:20:41.040212 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:20:41.040144 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-vxngp" Jul 23 19:20:41.041211 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:20:41.041192 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-rtgbq" event={"ID":"40f386ec-607d-496f-b65b-03a1fd847b1f","Type":"ContainerStarted","Data":"2bda7cd69636320254083db429d6c64500dcfa5a45308fa02cbc65d564d31c54"} Jul 23 19:20:41.210787 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:20:41.210766 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8rq79_25ca5174-01a0-41fc-9293-77dea2d9ec1f/konnectivity-agent/0.log" limit=1024 Jul 23 19:20:41.447719 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:20:41.447687 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8rq79_25ca5174-01a0-41fc-9293-77dea2d9ec1f/konnectivity-agent/0.log" limit=1024 Jul 23 19:20:41.628873 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:20:41.628845 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8rq79_25ca5174-01a0-41fc-9293-77dea2d9ec1f/konnectivity-agent/0.log" limit=1024 Jul 23 19:20:42.022257 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:20:42.022230 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8rq79_25ca5174-01a0-41fc-9293-77dea2d9ec1f/konnectivity-agent/0.log" limit=1024 Jul 23 19:20:42.130783 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:20:42.130751 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8rq79_25ca5174-01a0-41fc-9293-77dea2d9ec1f/konnectivity-agent/0.log" limit=1024 Jul 23 19:20:42.400310 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:20:42.400239 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/234ec34c-c9e5-42a6-b63c-c40f402dedbf-registry-tls\") pod \"image-registry-59b578868-9sb7b\" (UID: \"234ec34c-c9e5-42a6-b63c-c40f402dedbf\") " pod="openshift-image-registry/image-registry-59b578868-9sb7b" Jul 23 19:20:42.400466 ip-10-0-129-2 kubenswrapper[2564]: E0723 19:20:42.400404 2564 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Jul 23 19:20:42.400466 ip-10-0-129-2 kubenswrapper[2564]: E0723 19:20:42.400420 2564 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-59b578868-9sb7b: secret "image-registry-tls" not found Jul 23 19:20:42.400573 ip-10-0-129-2 kubenswrapper[2564]: E0723 19:20:42.400480 2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/234ec34c-c9e5-42a6-b63c-c40f402dedbf-registry-tls podName:234ec34c-c9e5-42a6-b63c-c40f402dedbf nodeName:}" failed. No retries permitted until 2026-07-23 19:21:14.400461733 +0000 UTC m=+100.252237958 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/234ec34c-c9e5-42a6-b63c-c40f402dedbf-registry-tls") pod "image-registry-59b578868-9sb7b" (UID: "234ec34c-c9e5-42a6-b63c-c40f402dedbf") : secret "image-registry-tls" not found Jul 23 19:20:42.422097 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:20:42.422073 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8rq79_25ca5174-01a0-41fc-9293-77dea2d9ec1f/konnectivity-agent/0.log" limit=1024 Jul 23 19:20:42.500709 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:20:42.500682 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/380698de-d0b8-426a-9e9b-708c0e36e818-metrics-tls\") pod \"dns-default-vzws2\" (UID: \"380698de-d0b8-426a-9e9b-708c0e36e818\") " pod="openshift-dns/dns-default-vzws2" Jul 23 19:20:42.500832 ip-10-0-129-2 kubenswrapper[2564]: E0723 19:20:42.500816 2564 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jul 23 19:20:42.500878 ip-10-0-129-2 kubenswrapper[2564]: E0723 19:20:42.500868 2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/380698de-d0b8-426a-9e9b-708c0e36e818-metrics-tls podName:380698de-d0b8-426a-9e9b-708c0e36e818 nodeName:}" failed. No retries permitted until 2026-07-23 19:21:14.500855511 +0000 UTC m=+100.352631736 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/380698de-d0b8-426a-9e9b-708c0e36e818-metrics-tls") pod "dns-default-vzws2" (UID: "380698de-d0b8-426a-9e9b-708c0e36e818") : secret "dns-default-metrics-tls" not found Jul 23 19:20:43.205693 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:20:43.205657 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/e891cbfb-7697-43dd-9003-3190f8e8a844-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-8gqzv\" (UID: \"e891cbfb-7697-43dd-9003-3190f8e8a844\") " pod="openshift-insights/insights-runtime-extractor-8gqzv" Jul 23 19:20:43.206127 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:20:43.205740 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/e891cbfb-7697-43dd-9003-3190f8e8a844-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-8gqzv\" (UID: \"e891cbfb-7697-43dd-9003-3190f8e8a844\") " pod="openshift-insights/insights-runtime-extractor-8gqzv" Jul 23 19:20:43.206127 ip-10-0-129-2 kubenswrapper[2564]: E0723 19:20:43.205807 2564 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Jul 23 19:20:43.206127 ip-10-0-129-2 kubenswrapper[2564]: E0723 19:20:43.205868 2564 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Jul 23 19:20:43.206127 ip-10-0-129-2 kubenswrapper[2564]: E0723 19:20:43.205903 2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e891cbfb-7697-43dd-9003-3190f8e8a844-insights-runtime-extractor-tls podName:e891cbfb-7697-43dd-9003-3190f8e8a844 nodeName:}" failed. No retries permitted until 2026-07-23 19:20:51.20588734 +0000 UTC m=+77.057663576 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/e891cbfb-7697-43dd-9003-3190f8e8a844-insights-runtime-extractor-tls") pod "insights-runtime-extractor-8gqzv" (UID: "e891cbfb-7697-43dd-9003-3190f8e8a844") : secret "insights-runtime-extractor-tls" not found Jul 23 19:20:43.206127 ip-10-0-129-2 kubenswrapper[2564]: E0723 19:20:43.205925 2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/e891cbfb-7697-43dd-9003-3190f8e8a844-kube-rbac-proxy-cm podName:e891cbfb-7697-43dd-9003-3190f8e8a844 nodeName:}" failed. No retries permitted until 2026-07-23 19:20:51.205909876 +0000 UTC m=+77.057686105 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/e891cbfb-7697-43dd-9003-3190f8e8a844-kube-rbac-proxy-cm") pod "insights-runtime-extractor-8gqzv" (UID: "e891cbfb-7697-43dd-9003-3190f8e8a844") : configmap "kube-rbac-proxy" not found Jul 23 19:20:43.227034 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:20:43.227005 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8rq79_25ca5174-01a0-41fc-9293-77dea2d9ec1f/konnectivity-agent/0.log" limit=1024 Jul 23 19:20:43.330593 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:20:43.330566 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8rq79_25ca5174-01a0-41fc-9293-77dea2d9ec1f/konnectivity-agent/0.log" limit=1024 Jul 23 19:20:43.656657 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:20:43.656563 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8rq79_25ca5174-01a0-41fc-9293-77dea2d9ec1f/konnectivity-agent/0.log" limit=1024 Jul 23 19:20:43.869076 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:20:43.869046 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8rq79_25ca5174-01a0-41fc-9293-77dea2d9ec1f/konnectivity-agent/0.log" limit=1024 Jul 23 19:20:44.013083 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:20:44.013051 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b7c6dedb-e5d3-41a0-b060-50a88ccca303-metrics-certs\") pod \"router-default-7f64bc459d-9b6vg\" (UID: \"b7c6dedb-e5d3-41a0-b060-50a88ccca303\") " pod="openshift-ingress/router-default-7f64bc459d-9b6vg" Jul 23 19:20:44.013262 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:20:44.013184 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b7c6dedb-e5d3-41a0-b060-50a88ccca303-service-ca-bundle\") pod \"router-default-7f64bc459d-9b6vg\" (UID: \"b7c6dedb-e5d3-41a0-b060-50a88ccca303\") " pod="openshift-ingress/router-default-7f64bc459d-9b6vg" Jul 23 19:20:44.013262 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:20:44.013218 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/e8d01fa4-24e5-4b2b-bf87-5e5395933cee-samples-operator-tls\") pod \"cluster-samples-operator-79b7c55f87-8l58m\" (UID: \"e8d01fa4-24e5-4b2b-bf87-5e5395933cee\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-79b7c55f87-8l58m" Jul 23 19:20:44.013476 ip-10-0-129-2 kubenswrapper[2564]: E0723 19:20:44.013448 2564 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found Jul 23 19:20:44.013598 ip-10-0-129-2 kubenswrapper[2564]: E0723 19:20:44.013522 2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b7c6dedb-e5d3-41a0-b060-50a88ccca303-metrics-certs podName:b7c6dedb-e5d3-41a0-b060-50a88ccca303 nodeName:}" failed. No retries permitted until 2026-07-23 19:21:00.013498139 +0000 UTC m=+85.865274363 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/b7c6dedb-e5d3-41a0-b060-50a88ccca303-metrics-certs") pod "router-default-7f64bc459d-9b6vg" (UID: "b7c6dedb-e5d3-41a0-b060-50a88ccca303") : secret "router-metrics-certs-default" not found Jul 23 19:20:44.013598 ip-10-0-129-2 kubenswrapper[2564]: E0723 19:20:44.013541 2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/b7c6dedb-e5d3-41a0-b060-50a88ccca303-service-ca-bundle podName:b7c6dedb-e5d3-41a0-b060-50a88ccca303 nodeName:}" failed. No retries permitted until 2026-07-23 19:21:00.013530884 +0000 UTC m=+85.865307109 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/b7c6dedb-e5d3-41a0-b060-50a88ccca303-service-ca-bundle") pod "router-default-7f64bc459d-9b6vg" (UID: "b7c6dedb-e5d3-41a0-b060-50a88ccca303") : configmap references non-existent config key: service-ca.crt Jul 23 19:20:44.016516 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:20:44.016470 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/e8d01fa4-24e5-4b2b-bf87-5e5395933cee-samples-operator-tls\") pod \"cluster-samples-operator-79b7c55f87-8l58m\" (UID: \"e8d01fa4-24e5-4b2b-bf87-5e5395933cee\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-79b7c55f87-8l58m" Jul 23 19:20:44.027777 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:20:44.027757 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-samples-operator\"/\"cluster-samples-operator-dockercfg-7qb62\"" Jul 23 19:20:44.030178 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:20:44.030159 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-samples-operator/cluster-samples-operator-79b7c55f87-8l58m" Jul 23 19:20:44.049589 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:20:44.049564 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8rq79_25ca5174-01a0-41fc-9293-77dea2d9ec1f/konnectivity-agent/0.log" limit=1024 Jul 23 19:20:44.135878 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:20:44.135855 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8rq79_25ca5174-01a0-41fc-9293-77dea2d9ec1f/konnectivity-agent/0.log" limit=1024 Jul 23 19:20:44.216077 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:20:44.216046 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/6f0a5836-d7de-49fc-b3a2-49830eaa468f-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-76cf8966f8-vv2vj\" (UID: \"6f0a5836-d7de-49fc-b3a2-49830eaa468f\") " pod="openshift-monitoring/cluster-monitoring-operator-76cf8966f8-vv2vj" Jul 23 19:20:44.216472 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:20:44.216127 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/7c2f37e4-a8b7-497e-ac12-7a6f5dd86619-networking-console-plugin-cert\") pod \"networking-console-plugin-7bc7489cb5-4p45l\" (UID: \"7c2f37e4-a8b7-497e-ac12-7a6f5dd86619\") " pod="openshift-network-console/networking-console-plugin-7bc7489cb5-4p45l" Jul 23 19:20:44.216472 ip-10-0-129-2 kubenswrapper[2564]: E0723 19:20:44.216259 2564 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found Jul 23 19:20:44.216472 ip-10-0-129-2 kubenswrapper[2564]: E0723 19:20:44.216305 2564 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found Jul 23 19:20:44.216472 ip-10-0-129-2 kubenswrapper[2564]: E0723 19:20:44.216365 2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6f0a5836-d7de-49fc-b3a2-49830eaa468f-cluster-monitoring-operator-tls podName:6f0a5836-d7de-49fc-b3a2-49830eaa468f nodeName:}" failed. No retries permitted until 2026-07-23 19:21:00.216345081 +0000 UTC m=+86.068121322 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/6f0a5836-d7de-49fc-b3a2-49830eaa468f-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-76cf8966f8-vv2vj" (UID: "6f0a5836-d7de-49fc-b3a2-49830eaa468f") : secret "cluster-monitoring-operator-tls" not found Jul 23 19:20:44.216472 ip-10-0-129-2 kubenswrapper[2564]: E0723 19:20:44.216382 2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7c2f37e4-a8b7-497e-ac12-7a6f5dd86619-networking-console-plugin-cert podName:7c2f37e4-a8b7-497e-ac12-7a6f5dd86619 nodeName:}" failed. No retries permitted until 2026-07-23 19:21:00.216375333 +0000 UTC m=+86.068151558 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/7c2f37e4-a8b7-497e-ac12-7a6f5dd86619-networking-console-plugin-cert") pod "networking-console-plugin-7bc7489cb5-4p45l" (UID: "7c2f37e4-a8b7-497e-ac12-7a6f5dd86619") : secret "networking-console-plugin-cert" not found Jul 23 19:20:44.225288 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:20:44.225257 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8rq79_25ca5174-01a0-41fc-9293-77dea2d9ec1f/konnectivity-agent/0.log" limit=1024 Jul 23 19:20:44.974924 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:20:44.974868 2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-target-vxngp" podStartSLOduration=69.974848036 podStartE2EDuration="1m9.974848036s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-23 19:19:35 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-23 19:20:41.0565862 +0000 UTC m=+66.908362445" watchObservedRunningTime="2026-07-23 19:20:44.974848036 +0000 UTC m=+70.826624283" Jul 23 19:20:44.975074 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:20:44.975063 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-samples-operator/cluster-samples-operator-79b7c55f87-8l58m"] Jul 23 19:20:45.054893 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:20:45.054863 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-samples-operator/cluster-samples-operator-79b7c55f87-8l58m" event={"ID":"e8d01fa4-24e5-4b2b-bf87-5e5395933cee","Type":"ContainerStarted","Data":"4834d132dc10f3090faaf037a40702f292b32d3cd93a0c7c6fc76cc1275e88fa"} Jul 23 19:20:45.221376 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:20:45.221350 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8rq79_25ca5174-01a0-41fc-9293-77dea2d9ec1f/konnectivity-agent/0.log" limit=1024 Jul 23 19:20:45.344634 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:20:45.344597 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8rq79_25ca5174-01a0-41fc-9293-77dea2d9ec1f/konnectivity-agent/0.log" limit=1024 Jul 23 19:20:45.630518 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:20:45.630492 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8rq79_25ca5174-01a0-41fc-9293-77dea2d9ec1f/konnectivity-agent/0.log" limit=1024 Jul 23 19:20:46.060342 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:20:46.060294 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-rtgbq" event={"ID":"40f386ec-607d-496f-b65b-03a1fd847b1f","Type":"ContainerStarted","Data":"5e35e2f60a0cd8158a8a831f63a511259386a8c9bfe149475ed1174389628856"} Jul 23 19:20:46.076875 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:20:46.076827 2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/global-pull-secret-syncer-rtgbq" podStartSLOduration=66.833523477 podStartE2EDuration="1m11.07680922s" totalImagesPullingTime="4.243285743s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-23 19:19:35 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-23 19:20:46.075588297 +0000 UTC m=+71.927364546" watchObservedRunningTime="2026-07-23 19:20:46.07680922 +0000 UTC m=+71.928585468" Jul 23 19:20:47.065576 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:20:47.065551 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-samples-operator/cluster-samples-operator-79b7c55f87-8l58m" event={"ID":"e8d01fa4-24e5-4b2b-bf87-5e5395933cee","Type":"ContainerStarted","Data":"bfb36324d82950daf7fdd2387e909c37f9cad707fa0f89b1f75b700f8c95f379"} Jul 23 19:20:48.071414 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:20:48.071385 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-samples-operator/cluster-samples-operator-79b7c55f87-8l58m" event={"ID":"e8d01fa4-24e5-4b2b-bf87-5e5395933cee","Type":"ContainerStarted","Data":"a086d80cd5cb126dfdfb85503e1c3a0cde6e3609a57f91ff46181fd56953f105"} Jul 23 19:20:48.087493 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:20:48.087450 2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-samples-operator/cluster-samples-operator-79b7c55f87-8l58m" podStartSLOduration=18.213004646999998 podStartE2EDuration="20.087435773s" totalImagesPullingTime="1.874431126s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-23 19:20:28 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-23 19:20:48.087306668 +0000 UTC m=+73.939082913" watchObservedRunningTime="2026-07-23 19:20:48.087435773 +0000 UTC m=+73.939212018" Jul 23 19:20:51.273511 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:20:51.273482 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/e891cbfb-7697-43dd-9003-3190f8e8a844-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-8gqzv\" (UID: \"e891cbfb-7697-43dd-9003-3190f8e8a844\") " pod="openshift-insights/insights-runtime-extractor-8gqzv" Jul 23 19:20:51.273847 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:20:51.273538 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/e891cbfb-7697-43dd-9003-3190f8e8a844-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-8gqzv\" (UID: \"e891cbfb-7697-43dd-9003-3190f8e8a844\") " pod="openshift-insights/insights-runtime-extractor-8gqzv" Jul 23 19:20:51.273956 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:20:51.273941 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/e891cbfb-7697-43dd-9003-3190f8e8a844-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-8gqzv\" (UID: \"e891cbfb-7697-43dd-9003-3190f8e8a844\") " pod="openshift-insights/insights-runtime-extractor-8gqzv" Jul 23 19:20:51.275686 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:20:51.275666 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/e891cbfb-7697-43dd-9003-3190f8e8a844-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-8gqzv\" (UID: \"e891cbfb-7697-43dd-9003-3190f8e8a844\") " pod="openshift-insights/insights-runtime-extractor-8gqzv" Jul 23 19:20:51.340813 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:20:51.340787 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-8gqzv" Jul 23 19:20:51.480310 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:20:51.480288 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-8gqzv"] Jul 23 19:20:51.482673 ip-10-0-129-2 kubenswrapper[2564]: W0723 19:20:51.482646 2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode891cbfb_7697_43dd_9003_3190f8e8a844.slice/crio-1af3c375c5c1ab0cc0bd41ec7a443e2f68de3d74718883e07228b39c0a07248c WatchSource:0}: Error finding container 1af3c375c5c1ab0cc0bd41ec7a443e2f68de3d74718883e07228b39c0a07248c: Status 404 returned error can't find the container with id 1af3c375c5c1ab0cc0bd41ec7a443e2f68de3d74718883e07228b39c0a07248c Jul 23 19:20:52.087368 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:20:52.087307 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-8gqzv" event={"ID":"e891cbfb-7697-43dd-9003-3190f8e8a844","Type":"ContainerStarted","Data":"1af3c375c5c1ab0cc0bd41ec7a443e2f68de3d74718883e07228b39c0a07248c"} Jul 23 19:20:53.652127 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:20:53.652094 2564 scope.go:117] "RemoveContainer" containerID="892131f3b4a665a17c51f76de799ce69cec99ea7c1c3120611106c156f9bb714" Jul 23 19:20:56.101971 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:20:56.101939 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-8gqzv" event={"ID":"e891cbfb-7697-43dd-9003-3190f8e8a844","Type":"ContainerStarted","Data":"3abf5c8c2e6e949f60a53fb6ff3594ef5162a78d1b422f409527b098c27a7972"} Jul 23 19:20:56.103701 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:20:56.103682 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-565b7b569d-z5wf4_ded5b3af-c74c-4d5a-aa77-62fe72d1dd29/console-operator/1.log" Jul 23 19:20:56.103803 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:20:56.103732 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-565b7b569d-z5wf4" event={"ID":"ded5b3af-c74c-4d5a-aa77-62fe72d1dd29","Type":"ContainerStarted","Data":"a31d030d29f688a8e15c72c5f9003b1ea040776796ecaa29563c579e2b6faf3a"} Jul 23 19:20:56.104003 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:20:56.103982 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console-operator/console-operator-565b7b569d-z5wf4" Jul 23 19:20:56.121062 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:20:56.121023 2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console-operator/console-operator-565b7b569d-z5wf4" podStartSLOduration=23.175757517 podStartE2EDuration="28.121010472s" totalImagesPullingTime="4.945252955s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-23 19:20:28 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-23 19:20:56.120776598 +0000 UTC m=+81.972552844" watchObservedRunningTime="2026-07-23 19:20:56.121010472 +0000 UTC m=+81.972786717" Jul 23 19:20:56.905191 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:20:56.905161 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console-operator/console-operator-565b7b569d-z5wf4" Jul 23 19:20:57.106570 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:20:57.106537 2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/downloads-5bc67dcd46-4plfk"] Jul 23 19:20:57.109825 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:20:57.109806 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/downloads-5bc67dcd46-4plfk" Jul 23 19:20:57.112026 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:20:57.112005 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"openshift-service-ca.crt\"" Jul 23 19:20:57.112026 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:20:57.112020 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"kube-root-ca.crt\"" Jul 23 19:20:57.112211 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:20:57.112169 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"default-dockercfg-htfwr\"" Jul 23 19:20:57.120635 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:20:57.120603 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/downloads-5bc67dcd46-4plfk"] Jul 23 19:20:57.224962 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:20:57.224936 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2pp7r\" (UniqueName: \"kubernetes.io/projected/fff3e347-7210-4a43-a41d-2bf0dcb0edb0-kube-api-access-2pp7r\") pod \"downloads-5bc67dcd46-4plfk\" (UID: \"fff3e347-7210-4a43-a41d-2bf0dcb0edb0\") " pod="openshift-console/downloads-5bc67dcd46-4plfk" Jul 23 19:20:57.325928 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:20:57.325906 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2pp7r\" (UniqueName: \"kubernetes.io/projected/fff3e347-7210-4a43-a41d-2bf0dcb0edb0-kube-api-access-2pp7r\") pod \"downloads-5bc67dcd46-4plfk\" (UID: \"fff3e347-7210-4a43-a41d-2bf0dcb0edb0\") " pod="openshift-console/downloads-5bc67dcd46-4plfk" Jul 23 19:20:57.342679 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:20:57.342651 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2pp7r\" (UniqueName: \"kubernetes.io/projected/fff3e347-7210-4a43-a41d-2bf0dcb0edb0-kube-api-access-2pp7r\") pod \"downloads-5bc67dcd46-4plfk\" (UID: \"fff3e347-7210-4a43-a41d-2bf0dcb0edb0\") " pod="openshift-console/downloads-5bc67dcd46-4plfk" Jul 23 19:20:57.420413 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:20:57.420392 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/downloads-5bc67dcd46-4plfk" Jul 23 19:20:59.370541 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:20:59.370520 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/downloads-5bc67dcd46-4plfk"] Jul 23 19:20:59.373105 ip-10-0-129-2 kubenswrapper[2564]: W0723 19:20:59.373081 2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfff3e347_7210_4a43_a41d_2bf0dcb0edb0.slice/crio-b98ade1b2766983a2aeafbdb435b1f787872d7dba876c44f1a95143da10335dc WatchSource:0}: Error finding container b98ade1b2766983a2aeafbdb435b1f787872d7dba876c44f1a95143da10335dc: Status 404 returned error can't find the container with id b98ade1b2766983a2aeafbdb435b1f787872d7dba876c44f1a95143da10335dc Jul 23 19:21:00.050645 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:21:00.050619 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b7c6dedb-e5d3-41a0-b060-50a88ccca303-metrics-certs\") pod \"router-default-7f64bc459d-9b6vg\" (UID: \"b7c6dedb-e5d3-41a0-b060-50a88ccca303\") " pod="openshift-ingress/router-default-7f64bc459d-9b6vg" Jul 23 19:21:00.050825 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:21:00.050698 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b7c6dedb-e5d3-41a0-b060-50a88ccca303-service-ca-bundle\") pod \"router-default-7f64bc459d-9b6vg\" (UID: \"b7c6dedb-e5d3-41a0-b060-50a88ccca303\") " pod="openshift-ingress/router-default-7f64bc459d-9b6vg" Jul 23 19:21:00.051290 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:21:00.051265 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b7c6dedb-e5d3-41a0-b060-50a88ccca303-service-ca-bundle\") pod \"router-default-7f64bc459d-9b6vg\" (UID: \"b7c6dedb-e5d3-41a0-b060-50a88ccca303\") " pod="openshift-ingress/router-default-7f64bc459d-9b6vg" Jul 23 19:21:00.053194 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:21:00.053173 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b7c6dedb-e5d3-41a0-b060-50a88ccca303-metrics-certs\") pod \"router-default-7f64bc459d-9b6vg\" (UID: \"b7c6dedb-e5d3-41a0-b060-50a88ccca303\") " pod="openshift-ingress/router-default-7f64bc459d-9b6vg" Jul 23 19:21:00.120411 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:21:00.120317 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-8gqzv" event={"ID":"e891cbfb-7697-43dd-9003-3190f8e8a844","Type":"ContainerStarted","Data":"cf83042adebbe903166793986b1e882d1eef8ba615bc22428d8f2ecae096b0a5"} Jul 23 19:21:00.121739 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:21:00.121715 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/downloads-5bc67dcd46-4plfk" event={"ID":"fff3e347-7210-4a43-a41d-2bf0dcb0edb0","Type":"ContainerStarted","Data":"b98ade1b2766983a2aeafbdb435b1f787872d7dba876c44f1a95143da10335dc"} Jul 23 19:21:00.239605 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:21:00.239585 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-dockercfg-fn7bv\"" Jul 23 19:21:00.247475 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:21:00.247448 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/router-default-7f64bc459d-9b6vg" Jul 23 19:21:00.251733 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:21:00.251711 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/6f0a5836-d7de-49fc-b3a2-49830eaa468f-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-76cf8966f8-vv2vj\" (UID: \"6f0a5836-d7de-49fc-b3a2-49830eaa468f\") " pod="openshift-monitoring/cluster-monitoring-operator-76cf8966f8-vv2vj" Jul 23 19:21:00.251830 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:21:00.251761 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/7c2f37e4-a8b7-497e-ac12-7a6f5dd86619-networking-console-plugin-cert\") pod \"networking-console-plugin-7bc7489cb5-4p45l\" (UID: \"7c2f37e4-a8b7-497e-ac12-7a6f5dd86619\") " pod="openshift-network-console/networking-console-plugin-7bc7489cb5-4p45l" Jul 23 19:21:00.254518 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:21:00.254479 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/6f0a5836-d7de-49fc-b3a2-49830eaa468f-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-76cf8966f8-vv2vj\" (UID: \"6f0a5836-d7de-49fc-b3a2-49830eaa468f\") " pod="openshift-monitoring/cluster-monitoring-operator-76cf8966f8-vv2vj" Jul 23 19:21:00.254706 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:21:00.254682 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/7c2f37e4-a8b7-497e-ac12-7a6f5dd86619-networking-console-plugin-cert\") pod \"networking-console-plugin-7bc7489cb5-4p45l\" (UID: \"7c2f37e4-a8b7-497e-ac12-7a6f5dd86619\") " pod="openshift-network-console/networking-console-plugin-7bc7489cb5-4p45l" Jul 23 19:21:00.383594 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:21:00.383566 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-console\"/\"default-dockercfg-bgzbk\"" Jul 23 19:21:00.392539 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:21:00.392517 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-console/networking-console-plugin-7bc7489cb5-4p45l" Jul 23 19:21:00.394538 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:21:00.394516 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/router-default-7f64bc459d-9b6vg"] Jul 23 19:21:00.398862 ip-10-0-129-2 kubenswrapper[2564]: W0723 19:21:00.398833 2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb7c6dedb_e5d3_41a0_b060_50a88ccca303.slice/crio-d9f72cc046d417af48a1e64d0a78e881c1c9cec8f020eb22c6380f4c7795daa0 WatchSource:0}: Error finding container d9f72cc046d417af48a1e64d0a78e881c1c9cec8f020eb22c6380f4c7795daa0: Status 404 returned error can't find the container with id d9f72cc046d417af48a1e64d0a78e881c1c9cec8f020eb22c6380f4c7795daa0 Jul 23 19:21:00.407220 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:21:00.407133 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"cluster-monitoring-operator-dockercfg-95j5j\"" Jul 23 19:21:00.415593 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:21:00.415574 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/cluster-monitoring-operator-76cf8966f8-vv2vj" Jul 23 19:21:00.555619 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:21:00.555552 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-console/networking-console-plugin-7bc7489cb5-4p45l"] Jul 23 19:21:00.560884 ip-10-0-129-2 kubenswrapper[2564]: W0723 19:21:00.560861 2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7c2f37e4_a8b7_497e_ac12_7a6f5dd86619.slice/crio-b8f1aeba73890fabe7f346b9ccc795c0d42deba50e5058fff09a56af71774cd0 WatchSource:0}: Error finding container b8f1aeba73890fabe7f346b9ccc795c0d42deba50e5058fff09a56af71774cd0: Status 404 returned error can't find the container with id b8f1aeba73890fabe7f346b9ccc795c0d42deba50e5058fff09a56af71774cd0 Jul 23 19:21:00.582866 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:21:00.582844 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/cluster-monitoring-operator-76cf8966f8-vv2vj"] Jul 23 19:21:00.586573 ip-10-0-129-2 kubenswrapper[2564]: W0723 19:21:00.586548 2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6f0a5836_d7de_49fc_b3a2_49830eaa468f.slice/crio-4653486a2d7e8deeb2db28b33b40a9c06bac88c686d69e6ec7de4d225460df28 WatchSource:0}: Error finding container 4653486a2d7e8deeb2db28b33b40a9c06bac88c686d69e6ec7de4d225460df28: Status 404 returned error can't find the container with id 4653486a2d7e8deeb2db28b33b40a9c06bac88c686d69e6ec7de4d225460df28 Jul 23 19:21:01.130065 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:21:01.130030 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/cluster-monitoring-operator-76cf8966f8-vv2vj" event={"ID":"6f0a5836-d7de-49fc-b3a2-49830eaa468f","Type":"ContainerStarted","Data":"4653486a2d7e8deeb2db28b33b40a9c06bac88c686d69e6ec7de4d225460df28"} Jul 23 19:21:01.132403 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:21:01.132373 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-console/networking-console-plugin-7bc7489cb5-4p45l" event={"ID":"7c2f37e4-a8b7-497e-ac12-7a6f5dd86619","Type":"ContainerStarted","Data":"b8f1aeba73890fabe7f346b9ccc795c0d42deba50e5058fff09a56af71774cd0"} Jul 23 19:21:01.135167 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:21:01.135141 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/router-default-7f64bc459d-9b6vg" event={"ID":"b7c6dedb-e5d3-41a0-b060-50a88ccca303","Type":"ContainerStarted","Data":"dedb0d8b00e2649379bfbf70575a58252ab4c7408cbfab25de82b467f3e848c4"} Jul 23 19:21:01.135270 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:21:01.135189 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/router-default-7f64bc459d-9b6vg" event={"ID":"b7c6dedb-e5d3-41a0-b060-50a88ccca303","Type":"ContainerStarted","Data":"d9f72cc046d417af48a1e64d0a78e881c1c9cec8f020eb22c6380f4c7795daa0"} Jul 23 19:21:01.157605 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:21:01.157566 2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/router-default-7f64bc459d-9b6vg" podStartSLOduration=33.157553498 podStartE2EDuration="33.157553498s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-23 19:20:28 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-23 19:21:01.155100799 +0000 UTC m=+87.006877041" watchObservedRunningTime="2026-07-23 19:21:01.157553498 +0000 UTC m=+87.009329722" Jul 23 19:21:01.248044 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:21:01.248019 2564 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/router-default-7f64bc459d-9b6vg" Jul 23 19:21:01.251101 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:21:01.251072 2564 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/router-default-7f64bc459d-9b6vg" Jul 23 19:21:02.140116 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:21:02.140087 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/router-default-7f64bc459d-9b6vg" Jul 23 19:21:02.142182 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:21:02.142136 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/router-default-7f64bc459d-9b6vg" Jul 23 19:21:05.154427 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:21:05.154394 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-8gqzv" event={"ID":"e891cbfb-7697-43dd-9003-3190f8e8a844","Type":"ContainerStarted","Data":"d1a0042e428eeab1e9241b9be25abdc7ed2e00292af07aafbb2112bf944d719d"} Jul 23 19:21:05.156297 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:21:05.156246 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/cluster-monitoring-operator-76cf8966f8-vv2vj" event={"ID":"6f0a5836-d7de-49fc-b3a2-49830eaa468f","Type":"ContainerStarted","Data":"d43b6cbee47630d8de937d8b26003648fae7fe94ec31d40ec6770bb90d43e797"} Jul 23 19:21:05.158270 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:21:05.158245 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-console/networking-console-plugin-7bc7489cb5-4p45l" event={"ID":"7c2f37e4-a8b7-497e-ac12-7a6f5dd86619","Type":"ContainerStarted","Data":"42b5a094fa0f1b80ba878455cb26f8d6847dc66c6c4f36d13b151377f8c4476a"} Jul 23 19:21:05.175225 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:21:05.175181 2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-insights/insights-runtime-extractor-8gqzv" podStartSLOduration=17.527320979 podStartE2EDuration="30.175166247s" totalImagesPullingTime="12.647845268s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-23 19:20:35 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-23 19:21:05.173781891 +0000 UTC m=+91.025558159" watchObservedRunningTime="2026-07-23 19:21:05.175166247 +0000 UTC m=+91.026942495" Jul 23 19:21:05.191419 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:21:05.191379 2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-console/networking-console-plugin-7bc7489cb5-4p45l" podStartSLOduration=33.513207917 podStartE2EDuration="37.191368361s" totalImagesPullingTime="3.678160444s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-23 19:20:28 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-23 19:21:05.18937808 +0000 UTC m=+91.041154327" watchObservedRunningTime="2026-07-23 19:21:05.191368361 +0000 UTC m=+91.043144606" Jul 23 19:21:05.207123 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:21:05.207082 2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/cluster-monitoring-operator-76cf8966f8-vv2vj" podStartSLOduration=33.548749824 podStartE2EDuration="37.207064979s" totalImagesPullingTime="3.658315155s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-23 19:20:28 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-23 19:21:05.206394623 +0000 UTC m=+91.058170871" watchObservedRunningTime="2026-07-23 19:21:05.207064979 +0000 UTC m=+91.058841225" Jul 23 19:21:12.047490 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:21:12.047461 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-network-diagnostics/network-check-target-vxngp" Jul 23 19:21:14.466253 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:21:14.466219 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/234ec34c-c9e5-42a6-b63c-c40f402dedbf-registry-tls\") pod \"image-registry-59b578868-9sb7b\" (UID: \"234ec34c-c9e5-42a6-b63c-c40f402dedbf\") " pod="openshift-image-registry/image-registry-59b578868-9sb7b" Jul 23 19:21:14.468970 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:21:14.468941 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/234ec34c-c9e5-42a6-b63c-c40f402dedbf-registry-tls\") pod \"image-registry-59b578868-9sb7b\" (UID: \"234ec34c-c9e5-42a6-b63c-c40f402dedbf\") " pod="openshift-image-registry/image-registry-59b578868-9sb7b" Jul 23 19:21:14.567086 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:21:14.567057 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/380698de-d0b8-426a-9e9b-708c0e36e818-metrics-tls\") pod \"dns-default-vzws2\" (UID: \"380698de-d0b8-426a-9e9b-708c0e36e818\") " pod="openshift-dns/dns-default-vzws2" Jul 23 19:21:14.569561 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:21:14.569532 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/380698de-d0b8-426a-9e9b-708c0e36e818-metrics-tls\") pod \"dns-default-vzws2\" (UID: \"380698de-d0b8-426a-9e9b-708c0e36e818\") " pod="openshift-dns/dns-default-vzws2" Jul 23 19:21:14.745545 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:21:14.745480 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"registry-dockercfg-zfpwt\"" Jul 23 19:21:14.753627 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:21:14.753605 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-59b578868-9sb7b" Jul 23 19:21:14.767006 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:21:14.766984 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-j4csw\"" Jul 23 19:21:14.775761 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:21:14.775741 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-vzws2" Jul 23 19:21:16.234109 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:21:16.234083 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-59b578868-9sb7b"] Jul 23 19:21:16.237158 ip-10-0-129-2 kubenswrapper[2564]: W0723 19:21:16.237130 2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod234ec34c_c9e5_42a6_b63c_c40f402dedbf.slice/crio-260b87c0522101bdaf9bf753d5b3851bbd39a6fd40c2c12c3675138cb82bc59c WatchSource:0}: Error finding container 260b87c0522101bdaf9bf753d5b3851bbd39a6fd40c2c12c3675138cb82bc59c: Status 404 returned error can't find the container with id 260b87c0522101bdaf9bf753d5b3851bbd39a6fd40c2c12c3675138cb82bc59c Jul 23 19:21:16.260618 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:21:16.260590 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-vzws2"] Jul 23 19:21:16.264848 ip-10-0-129-2 kubenswrapper[2564]: W0723 19:21:16.264825 2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod380698de_d0b8_426a_9e9b_708c0e36e818.slice/crio-237daec5827b03def514a5a30994050c8de2f66aed7fe446414d716ac47eaf9b WatchSource:0}: Error finding container 237daec5827b03def514a5a30994050c8de2f66aed7fe446414d716ac47eaf9b: Status 404 returned error can't find the container with id 237daec5827b03def514a5a30994050c8de2f66aed7fe446414d716ac47eaf9b Jul 23 19:21:17.208255 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:21:17.208215 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/downloads-5bc67dcd46-4plfk" event={"ID":"fff3e347-7210-4a43-a41d-2bf0dcb0edb0","Type":"ContainerStarted","Data":"ba63e1fbb1aa7c2cc0a5aee666c215d5fa0a683d0d04289623d6742cf768fffa"} Jul 23 19:21:17.208903 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:21:17.208879 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/downloads-5bc67dcd46-4plfk" Jul 23 19:21:17.211005 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:21:17.210977 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-vzws2" event={"ID":"380698de-d0b8-426a-9e9b-708c0e36e818","Type":"ContainerStarted","Data":"237daec5827b03def514a5a30994050c8de2f66aed7fe446414d716ac47eaf9b"} Jul 23 19:21:17.213788 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:21:17.213137 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-59b578868-9sb7b" event={"ID":"234ec34c-c9e5-42a6-b63c-c40f402dedbf","Type":"ContainerStarted","Data":"2ab175450038740b85589f42b73f63738d8b4368f99b2a479c9276a198d89e9c"} Jul 23 19:21:17.213889 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:21:17.213857 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-image-registry/image-registry-59b578868-9sb7b" Jul 23 19:21:17.213889 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:21:17.213884 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-59b578868-9sb7b" event={"ID":"234ec34c-c9e5-42a6-b63c-c40f402dedbf","Type":"ContainerStarted","Data":"260b87c0522101bdaf9bf753d5b3851bbd39a6fd40c2c12c3675138cb82bc59c"} Jul 23 19:21:17.223314 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:21:17.223294 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/downloads-5bc67dcd46-4plfk" Jul 23 19:21:17.242979 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:21:17.242934 2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/downloads-5bc67dcd46-4plfk" podStartSLOduration=3.100096605 podStartE2EDuration="20.242921259s" totalImagesPullingTime="17.142824654s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-23 19:20:57 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-23 19:21:17.242374683 +0000 UTC m=+103.094150929" watchObservedRunningTime="2026-07-23 19:21:17.242921259 +0000 UTC m=+103.094697504" Jul 23 19:21:19.224532 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:21:19.224476 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-vzws2" event={"ID":"380698de-d0b8-426a-9e9b-708c0e36e818","Type":"ContainerStarted","Data":"4312e82ec367b28d1ab3c870139c704a07a2cf66c8db0f15c9c634a924a537ab"} Jul 23 19:21:19.225083 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:21:19.224839 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-dns/dns-default-vzws2" Jul 23 19:21:19.225083 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:21:19.224887 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-vzws2" event={"ID":"380698de-d0b8-426a-9e9b-708c0e36e818","Type":"ContainerStarted","Data":"776ccf7369d64a76b40e2c2569e8ddb5efe7d048e491f62b63d660d2e72cd839"} Jul 23 19:21:19.244049 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:21:19.243967 2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/image-registry-59b578868-9sb7b" podStartSLOduration=104.243953087 podStartE2EDuration="1m44.243953087s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-23 19:19:35 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-23 19:21:17.323982985 +0000 UTC m=+103.175759231" watchObservedRunningTime="2026-07-23 19:21:19.243953087 +0000 UTC m=+105.095729332" Jul 23 19:21:19.244553 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:21:19.244523 2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/dns-default-vzws2" podStartSLOduration=66.788231238 podStartE2EDuration="1m9.244514021s" totalImagesPullingTime="2.456282783s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-23 19:20:10 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-23 19:21:19.242523005 +0000 UTC m=+105.094299265" watchObservedRunningTime="2026-07-23 19:21:19.244514021 +0000 UTC m=+105.096290266" Jul 23 19:21:20.014524 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:21:20.014487 2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-59b578868-9sb7b"] Jul 23 19:21:25.538341 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:21:25.538287 2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/node-exporter-fhzlr"] Jul 23 19:21:25.571852 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:21:25.571827 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-fhzlr" Jul 23 19:21:25.576699 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:21:25.576677 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-q9wn7\"" Jul 23 19:21:25.576849 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:21:25.576756 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\"" Jul 23 19:21:25.576925 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:21:25.576799 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\"" Jul 23 19:21:25.577682 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:21:25.577663 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\"" Jul 23 19:21:25.577884 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:21:25.577688 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\"" Jul 23 19:21:25.661351 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:21:25.661313 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/06754a58-2f04-44d1-9618-2ce5a6d6468b-sys\") pod \"node-exporter-fhzlr\" (UID: \"06754a58-2f04-44d1-9618-2ce5a6d6468b\") " pod="openshift-monitoring/node-exporter-fhzlr" Jul 23 19:21:25.661440 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:21:25.661358 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/06754a58-2f04-44d1-9618-2ce5a6d6468b-metrics-client-ca\") pod \"node-exporter-fhzlr\" (UID: \"06754a58-2f04-44d1-9618-2ce5a6d6468b\") " pod="openshift-monitoring/node-exporter-fhzlr" Jul 23 19:21:25.661440 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:21:25.661381 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/06754a58-2f04-44d1-9618-2ce5a6d6468b-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-fhzlr\" (UID: \"06754a58-2f04-44d1-9618-2ce5a6d6468b\") " pod="openshift-monitoring/node-exporter-fhzlr" Jul 23 19:21:25.661440 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:21:25.661420 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7l7m9\" (UniqueName: \"kubernetes.io/projected/06754a58-2f04-44d1-9618-2ce5a6d6468b-kube-api-access-7l7m9\") pod \"node-exporter-fhzlr\" (UID: \"06754a58-2f04-44d1-9618-2ce5a6d6468b\") " pod="openshift-monitoring/node-exporter-fhzlr" Jul 23 19:21:25.661543 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:21:25.661449 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/06754a58-2f04-44d1-9618-2ce5a6d6468b-node-exporter-textfile\") pod \"node-exporter-fhzlr\" (UID: \"06754a58-2f04-44d1-9618-2ce5a6d6468b\") " pod="openshift-monitoring/node-exporter-fhzlr" Jul 23 19:21:25.661543 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:21:25.661477 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/06754a58-2f04-44d1-9618-2ce5a6d6468b-node-exporter-accelerators-collector-config\") pod \"node-exporter-fhzlr\" (UID: \"06754a58-2f04-44d1-9618-2ce5a6d6468b\") " pod="openshift-monitoring/node-exporter-fhzlr" Jul 23 19:21:25.661543 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:21:25.661524 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/06754a58-2f04-44d1-9618-2ce5a6d6468b-node-exporter-wtmp\") pod \"node-exporter-fhzlr\" (UID: \"06754a58-2f04-44d1-9618-2ce5a6d6468b\") " pod="openshift-monitoring/node-exporter-fhzlr" Jul 23 19:21:25.661637 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:21:25.661554 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/06754a58-2f04-44d1-9618-2ce5a6d6468b-root\") pod \"node-exporter-fhzlr\" (UID: \"06754a58-2f04-44d1-9618-2ce5a6d6468b\") " pod="openshift-monitoring/node-exporter-fhzlr" Jul 23 19:21:25.661637 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:21:25.661575 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/06754a58-2f04-44d1-9618-2ce5a6d6468b-node-exporter-tls\") pod \"node-exporter-fhzlr\" (UID: \"06754a58-2f04-44d1-9618-2ce5a6d6468b\") " pod="openshift-monitoring/node-exporter-fhzlr" Jul 23 19:21:25.762535 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:21:25.762512 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/06754a58-2f04-44d1-9618-2ce5a6d6468b-node-exporter-wtmp\") pod \"node-exporter-fhzlr\" (UID: \"06754a58-2f04-44d1-9618-2ce5a6d6468b\") " pod="openshift-monitoring/node-exporter-fhzlr" Jul 23 19:21:25.762644 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:21:25.762542 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/06754a58-2f04-44d1-9618-2ce5a6d6468b-root\") pod \"node-exporter-fhzlr\" (UID: \"06754a58-2f04-44d1-9618-2ce5a6d6468b\") " pod="openshift-monitoring/node-exporter-fhzlr" Jul 23 19:21:25.762644 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:21:25.762557 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/06754a58-2f04-44d1-9618-2ce5a6d6468b-node-exporter-tls\") pod \"node-exporter-fhzlr\" (UID: \"06754a58-2f04-44d1-9618-2ce5a6d6468b\") " pod="openshift-monitoring/node-exporter-fhzlr" Jul 23 19:21:25.762644 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:21:25.762585 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/06754a58-2f04-44d1-9618-2ce5a6d6468b-sys\") pod \"node-exporter-fhzlr\" (UID: \"06754a58-2f04-44d1-9618-2ce5a6d6468b\") " pod="openshift-monitoring/node-exporter-fhzlr" Jul 23 19:21:25.762644 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:21:25.762630 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/06754a58-2f04-44d1-9618-2ce5a6d6468b-sys\") pod \"node-exporter-fhzlr\" (UID: \"06754a58-2f04-44d1-9618-2ce5a6d6468b\") " pod="openshift-monitoring/node-exporter-fhzlr" Jul 23 19:21:25.762843 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:21:25.762648 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"root\" (UniqueName: \"kubernetes.io/host-path/06754a58-2f04-44d1-9618-2ce5a6d6468b-root\") pod \"node-exporter-fhzlr\" (UID: \"06754a58-2f04-44d1-9618-2ce5a6d6468b\") " pod="openshift-monitoring/node-exporter-fhzlr" Jul 23 19:21:25.762843 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:21:25.762690 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/06754a58-2f04-44d1-9618-2ce5a6d6468b-metrics-client-ca\") pod \"node-exporter-fhzlr\" (UID: \"06754a58-2f04-44d1-9618-2ce5a6d6468b\") " pod="openshift-monitoring/node-exporter-fhzlr" Jul 23 19:21:25.762843 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:21:25.762702 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/06754a58-2f04-44d1-9618-2ce5a6d6468b-node-exporter-wtmp\") pod \"node-exporter-fhzlr\" (UID: \"06754a58-2f04-44d1-9618-2ce5a6d6468b\") " pod="openshift-monitoring/node-exporter-fhzlr" Jul 23 19:21:25.762843 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:21:25.762724 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/06754a58-2f04-44d1-9618-2ce5a6d6468b-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-fhzlr\" (UID: \"06754a58-2f04-44d1-9618-2ce5a6d6468b\") " pod="openshift-monitoring/node-exporter-fhzlr" Jul 23 19:21:25.762843 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:21:25.762786 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7l7m9\" (UniqueName: \"kubernetes.io/projected/06754a58-2f04-44d1-9618-2ce5a6d6468b-kube-api-access-7l7m9\") pod \"node-exporter-fhzlr\" (UID: \"06754a58-2f04-44d1-9618-2ce5a6d6468b\") " pod="openshift-monitoring/node-exporter-fhzlr" Jul 23 19:21:25.762843 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:21:25.762807 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/06754a58-2f04-44d1-9618-2ce5a6d6468b-node-exporter-textfile\") pod \"node-exporter-fhzlr\" (UID: \"06754a58-2f04-44d1-9618-2ce5a6d6468b\") " pod="openshift-monitoring/node-exporter-fhzlr" Jul 23 19:21:25.763124 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:21:25.762987 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/06754a58-2f04-44d1-9618-2ce5a6d6468b-node-exporter-accelerators-collector-config\") pod \"node-exporter-fhzlr\" (UID: \"06754a58-2f04-44d1-9618-2ce5a6d6468b\") " pod="openshift-monitoring/node-exporter-fhzlr" Jul 23 19:21:25.763124 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:21:25.763101 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/06754a58-2f04-44d1-9618-2ce5a6d6468b-node-exporter-textfile\") pod \"node-exporter-fhzlr\" (UID: \"06754a58-2f04-44d1-9618-2ce5a6d6468b\") " pod="openshift-monitoring/node-exporter-fhzlr" Jul 23 19:21:25.763360 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:21:25.763317 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/06754a58-2f04-44d1-9618-2ce5a6d6468b-metrics-client-ca\") pod \"node-exporter-fhzlr\" (UID: \"06754a58-2f04-44d1-9618-2ce5a6d6468b\") " pod="openshift-monitoring/node-exporter-fhzlr" Jul 23 19:21:25.763475 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:21:25.763455 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/06754a58-2f04-44d1-9618-2ce5a6d6468b-node-exporter-accelerators-collector-config\") pod \"node-exporter-fhzlr\" (UID: \"06754a58-2f04-44d1-9618-2ce5a6d6468b\") " pod="openshift-monitoring/node-exporter-fhzlr" Jul 23 19:21:25.765210 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:21:25.765188 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/06754a58-2f04-44d1-9618-2ce5a6d6468b-node-exporter-tls\") pod \"node-exporter-fhzlr\" (UID: \"06754a58-2f04-44d1-9618-2ce5a6d6468b\") " pod="openshift-monitoring/node-exporter-fhzlr" Jul 23 19:21:25.765307 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:21:25.765213 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/06754a58-2f04-44d1-9618-2ce5a6d6468b-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-fhzlr\" (UID: \"06754a58-2f04-44d1-9618-2ce5a6d6468b\") " pod="openshift-monitoring/node-exporter-fhzlr" Jul 23 19:21:25.772583 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:21:25.772560 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7l7m9\" (UniqueName: \"kubernetes.io/projected/06754a58-2f04-44d1-9618-2ce5a6d6468b-kube-api-access-7l7m9\") pod \"node-exporter-fhzlr\" (UID: \"06754a58-2f04-44d1-9618-2ce5a6d6468b\") " pod="openshift-monitoring/node-exporter-fhzlr" Jul 23 19:21:25.883767 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:21:25.883723 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-fhzlr" Jul 23 19:21:25.918610 ip-10-0-129-2 kubenswrapper[2564]: W0723 19:21:25.918583 2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod06754a58_2f04_44d1_9618_2ce5a6d6468b.slice/crio-ac07cdb6ff6ea50ddacbeada6638a46098fea75c3a6c44dde0f97bd5735aed62 WatchSource:0}: Error finding container ac07cdb6ff6ea50ddacbeada6638a46098fea75c3a6c44dde0f97bd5735aed62: Status 404 returned error can't find the container with id ac07cdb6ff6ea50ddacbeada6638a46098fea75c3a6c44dde0f97bd5735aed62 Jul 23 19:21:26.254691 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:21:26.254665 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-fhzlr" event={"ID":"06754a58-2f04-44d1-9618-2ce5a6d6468b","Type":"ContainerStarted","Data":"ac07cdb6ff6ea50ddacbeada6638a46098fea75c3a6c44dde0f97bd5735aed62"} Jul 23 19:21:28.264953 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:21:28.264922 2564 generic.go:356] "Generic (PLEG): container finished" podID="06754a58-2f04-44d1-9618-2ce5a6d6468b" containerID="80df0c4b152bebd3ec4a93004da6451720fa30826f79d6389a1f18be55afb31c" exitCode=0 Jul 23 19:21:28.265279 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:21:28.264966 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-fhzlr" event={"ID":"06754a58-2f04-44d1-9618-2ce5a6d6468b","Type":"ContainerDied","Data":"80df0c4b152bebd3ec4a93004da6451720fa30826f79d6389a1f18be55afb31c"} Jul 23 19:21:29.230168 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:21:29.230140 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-dns/dns-default-vzws2" Jul 23 19:21:29.270827 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:21:29.270797 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-fhzlr" event={"ID":"06754a58-2f04-44d1-9618-2ce5a6d6468b","Type":"ContainerStarted","Data":"a47d3b0fa03ce7c57df05bf06d5bc979486babf620fe2e40727b85f74597e039"} Jul 23 19:21:29.271257 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:21:29.270854 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-fhzlr" event={"ID":"06754a58-2f04-44d1-9618-2ce5a6d6468b","Type":"ContainerStarted","Data":"4a6149196bec8d4650b6a45857d9d2c578860874cd10b3e5c687382c621e953b"} Jul 23 19:21:29.295272 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:21:29.295224 2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/node-exporter-fhzlr" podStartSLOduration=2.812693055 podStartE2EDuration="4.295207674s" totalImagesPullingTime="1.393050005s" totalInitContainerRuntime="89.464614ms" isStatefulPod=true podCreationTimestamp="2026-07-23 19:21:25 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-23 19:21:29.292817102 +0000 UTC m=+115.144593358" watchObservedRunningTime="2026-07-23 19:21:29.295207674 +0000 UTC m=+115.146983920" Jul 23 19:21:40.021154 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:21:40.021127 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-image-registry/image-registry-59b578868-9sb7b" Jul 23 19:21:40.310044 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:21:40.309989 2564 generic.go:356] "Generic (PLEG): container finished" podID="00953f2a-6543-445a-8895-40525b4e3e25" containerID="cb1424ea8d08116b26738f3be39ba3dc94293d93ba47090549c11cbf6d17a6f2" exitCode=0 Jul 23 19:21:40.310142 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:21:40.310047 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-555d9d7885-r8lv4" event={"ID":"00953f2a-6543-445a-8895-40525b4e3e25","Type":"ContainerDied","Data":"cb1424ea8d08116b26738f3be39ba3dc94293d93ba47090549c11cbf6d17a6f2"} Jul 23 19:21:40.310296 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:21:40.310284 2564 scope.go:117] "RemoveContainer" containerID="cb1424ea8d08116b26738f3be39ba3dc94293d93ba47090549c11cbf6d17a6f2" Jul 23 19:21:41.195639 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:21:41.195596 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_router-default-7f64bc459d-9b6vg_b7c6dedb-e5d3-41a0-b060-50a88ccca303/router/0.log" Jul 23 19:21:41.238432 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:21:41.238412 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-wxwk8_e6d7c065-91d9-461c-8c0f-684ad4ea70e6/serve-healthcheck-canary/0.log" Jul 23 19:21:41.315549 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:21:41.315527 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-555d9d7885-r8lv4" event={"ID":"00953f2a-6543-445a-8895-40525b4e3e25","Type":"ContainerStarted","Data":"2950666d907b5cd07afef69f0eeebdd2792963907452e7cca96f3b40c287155d"} Jul 23 19:21:44.395674 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:21:44.395639 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/576f76ed-730c-49c2-a33e-1dc11370cad4-metrics-certs\") pod \"network-metrics-daemon-qhrz5\" (UID: \"576f76ed-730c-49c2-a33e-1dc11370cad4\") " pod="openshift-multus/network-metrics-daemon-qhrz5" Jul 23 19:21:44.397836 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:21:44.397817 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/576f76ed-730c-49c2-a33e-1dc11370cad4-metrics-certs\") pod \"network-metrics-daemon-qhrz5\" (UID: \"576f76ed-730c-49c2-a33e-1dc11370cad4\") " pod="openshift-multus/network-metrics-daemon-qhrz5" Jul 23 19:21:44.664759 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:21:44.664740 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-k9tbr\"" Jul 23 19:21:44.671997 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:21:44.671981 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-qhrz5" Jul 23 19:21:44.797042 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:21:44.797010 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-qhrz5"] Jul 23 19:21:44.799737 ip-10-0-129-2 kubenswrapper[2564]: W0723 19:21:44.799710 2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod576f76ed_730c_49c2_a33e_1dc11370cad4.slice/crio-ed1e3eadaa2af4656d85582a50adfd7778248925ab91a9c3e2803c7af9fabe3e WatchSource:0}: Error finding container ed1e3eadaa2af4656d85582a50adfd7778248925ab91a9c3e2803c7af9fabe3e: Status 404 returned error can't find the container with id ed1e3eadaa2af4656d85582a50adfd7778248925ab91a9c3e2803c7af9fabe3e Jul 23 19:21:45.037181 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:21:45.037128 2564 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-image-registry/image-registry-59b578868-9sb7b" podUID="234ec34c-c9e5-42a6-b63c-c40f402dedbf" containerName="registry" containerID="cri-o://2ab175450038740b85589f42b73f63738d8b4368f99b2a479c9276a198d89e9c" gracePeriod=30 Jul 23 19:21:45.253455 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:21:45.253433 2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-59b578868-9sb7b" Jul 23 19:21:45.302024 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:21:45.301976 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/234ec34c-c9e5-42a6-b63c-c40f402dedbf-registry-certificates\") pod \"234ec34c-c9e5-42a6-b63c-c40f402dedbf\" (UID: \"234ec34c-c9e5-42a6-b63c-c40f402dedbf\") " Jul 23 19:21:45.302024 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:21:45.302009 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-ftzrt\" (UniqueName: \"kubernetes.io/projected/234ec34c-c9e5-42a6-b63c-c40f402dedbf-kube-api-access-ftzrt\") pod \"234ec34c-c9e5-42a6-b63c-c40f402dedbf\" (UID: \"234ec34c-c9e5-42a6-b63c-c40f402dedbf\") " Jul 23 19:21:45.302231 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:21:45.302037 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/234ec34c-c9e5-42a6-b63c-c40f402dedbf-ca-trust-extracted\") pod \"234ec34c-c9e5-42a6-b63c-c40f402dedbf\" (UID: \"234ec34c-c9e5-42a6-b63c-c40f402dedbf\") " Jul 23 19:21:45.302231 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:21:45.302074 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/234ec34c-c9e5-42a6-b63c-c40f402dedbf-installation-pull-secrets\") pod \"234ec34c-c9e5-42a6-b63c-c40f402dedbf\" (UID: \"234ec34c-c9e5-42a6-b63c-c40f402dedbf\") " Jul 23 19:21:45.302231 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:21:45.302121 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/234ec34c-c9e5-42a6-b63c-c40f402dedbf-bound-sa-token\") pod \"234ec34c-c9e5-42a6-b63c-c40f402dedbf\" (UID: \"234ec34c-c9e5-42a6-b63c-c40f402dedbf\") " Jul 23 19:21:45.302231 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:21:45.302170 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/234ec34c-c9e5-42a6-b63c-c40f402dedbf-trusted-ca\") pod \"234ec34c-c9e5-42a6-b63c-c40f402dedbf\" (UID: \"234ec34c-c9e5-42a6-b63c-c40f402dedbf\") " Jul 23 19:21:45.302231 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:21:45.302202 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/234ec34c-c9e5-42a6-b63c-c40f402dedbf-image-registry-private-configuration\") pod \"234ec34c-c9e5-42a6-b63c-c40f402dedbf\" (UID: \"234ec34c-c9e5-42a6-b63c-c40f402dedbf\") " Jul 23 19:21:45.302231 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:21:45.302229 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/234ec34c-c9e5-42a6-b63c-c40f402dedbf-registry-tls\") pod \"234ec34c-c9e5-42a6-b63c-c40f402dedbf\" (UID: \"234ec34c-c9e5-42a6-b63c-c40f402dedbf\") " Jul 23 19:21:45.302552 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:21:45.302453 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/234ec34c-c9e5-42a6-b63c-c40f402dedbf-registry-certificates" (OuterVolumeSpecName: "registry-certificates") pod "234ec34c-c9e5-42a6-b63c-c40f402dedbf" (UID: "234ec34c-c9e5-42a6-b63c-c40f402dedbf"). InnerVolumeSpecName "registry-certificates". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 23 19:21:45.302998 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:21:45.302968 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/234ec34c-c9e5-42a6-b63c-c40f402dedbf-trusted-ca" (OuterVolumeSpecName: "trusted-ca") pod "234ec34c-c9e5-42a6-b63c-c40f402dedbf" (UID: "234ec34c-c9e5-42a6-b63c-c40f402dedbf"). InnerVolumeSpecName "trusted-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 23 19:21:45.305111 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:21:45.305076 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/234ec34c-c9e5-42a6-b63c-c40f402dedbf-registry-tls" (OuterVolumeSpecName: "registry-tls") pod "234ec34c-c9e5-42a6-b63c-c40f402dedbf" (UID: "234ec34c-c9e5-42a6-b63c-c40f402dedbf"). InnerVolumeSpecName "registry-tls". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 23 19:21:45.305111 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:21:45.305079 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/234ec34c-c9e5-42a6-b63c-c40f402dedbf-image-registry-private-configuration" (OuterVolumeSpecName: "image-registry-private-configuration") pod "234ec34c-c9e5-42a6-b63c-c40f402dedbf" (UID: "234ec34c-c9e5-42a6-b63c-c40f402dedbf"). InnerVolumeSpecName "image-registry-private-configuration". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 23 19:21:45.305281 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:21:45.305177 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/234ec34c-c9e5-42a6-b63c-c40f402dedbf-installation-pull-secrets" (OuterVolumeSpecName: "installation-pull-secrets") pod "234ec34c-c9e5-42a6-b63c-c40f402dedbf" (UID: "234ec34c-c9e5-42a6-b63c-c40f402dedbf"). InnerVolumeSpecName "installation-pull-secrets". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 23 19:21:45.305571 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:21:45.305543 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/234ec34c-c9e5-42a6-b63c-c40f402dedbf-bound-sa-token" (OuterVolumeSpecName: "bound-sa-token") pod "234ec34c-c9e5-42a6-b63c-c40f402dedbf" (UID: "234ec34c-c9e5-42a6-b63c-c40f402dedbf"). InnerVolumeSpecName "bound-sa-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 23 19:21:45.305700 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:21:45.305680 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/234ec34c-c9e5-42a6-b63c-c40f402dedbf-kube-api-access-ftzrt" (OuterVolumeSpecName: "kube-api-access-ftzrt") pod "234ec34c-c9e5-42a6-b63c-c40f402dedbf" (UID: "234ec34c-c9e5-42a6-b63c-c40f402dedbf"). InnerVolumeSpecName "kube-api-access-ftzrt". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 23 19:21:45.311436 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:21:45.311416 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/234ec34c-c9e5-42a6-b63c-c40f402dedbf-ca-trust-extracted" (OuterVolumeSpecName: "ca-trust-extracted") pod "234ec34c-c9e5-42a6-b63c-c40f402dedbf" (UID: "234ec34c-c9e5-42a6-b63c-c40f402dedbf"). InnerVolumeSpecName "ca-trust-extracted". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 23 19:21:45.332556 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:21:45.332526 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-qhrz5" event={"ID":"576f76ed-730c-49c2-a33e-1dc11370cad4","Type":"ContainerStarted","Data":"ed1e3eadaa2af4656d85582a50adfd7778248925ab91a9c3e2803c7af9fabe3e"} Jul 23 19:21:45.334054 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:21:45.334030 2564 generic.go:356] "Generic (PLEG): container finished" podID="234ec34c-c9e5-42a6-b63c-c40f402dedbf" containerID="2ab175450038740b85589f42b73f63738d8b4368f99b2a479c9276a198d89e9c" exitCode=0 Jul 23 19:21:45.334152 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:21:45.334079 2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-59b578868-9sb7b" Jul 23 19:21:45.334152 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:21:45.334092 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-59b578868-9sb7b" event={"ID":"234ec34c-c9e5-42a6-b63c-c40f402dedbf","Type":"ContainerDied","Data":"2ab175450038740b85589f42b73f63738d8b4368f99b2a479c9276a198d89e9c"} Jul 23 19:21:45.334260 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:21:45.334165 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-59b578868-9sb7b" event={"ID":"234ec34c-c9e5-42a6-b63c-c40f402dedbf","Type":"ContainerDied","Data":"260b87c0522101bdaf9bf753d5b3851bbd39a6fd40c2c12c3675138cb82bc59c"} Jul 23 19:21:45.334260 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:21:45.334192 2564 scope.go:117] "RemoveContainer" containerID="2ab175450038740b85589f42b73f63738d8b4368f99b2a479c9276a198d89e9c" Jul 23 19:21:45.342995 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:21:45.342913 2564 scope.go:117] "RemoveContainer" containerID="2ab175450038740b85589f42b73f63738d8b4368f99b2a479c9276a198d89e9c" Jul 23 19:21:45.343243 ip-10-0-129-2 kubenswrapper[2564]: E0723 19:21:45.343214 2564 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2ab175450038740b85589f42b73f63738d8b4368f99b2a479c9276a198d89e9c\": container with ID starting with 2ab175450038740b85589f42b73f63738d8b4368f99b2a479c9276a198d89e9c not found: ID does not exist" containerID="2ab175450038740b85589f42b73f63738d8b4368f99b2a479c9276a198d89e9c" Jul 23 19:21:45.343314 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:21:45.343249 2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2ab175450038740b85589f42b73f63738d8b4368f99b2a479c9276a198d89e9c"} err="failed to get container status \"2ab175450038740b85589f42b73f63738d8b4368f99b2a479c9276a198d89e9c\": rpc error: code = NotFound desc = could not find container \"2ab175450038740b85589f42b73f63738d8b4368f99b2a479c9276a198d89e9c\": container with ID starting with 2ab175450038740b85589f42b73f63738d8b4368f99b2a479c9276a198d89e9c not found: ID does not exist" Jul 23 19:21:45.362019 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:21:45.361995 2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-59b578868-9sb7b"] Jul 23 19:21:45.372709 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:21:45.372685 2564 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-image-registry/image-registry-59b578868-9sb7b"] Jul 23 19:21:45.402982 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:21:45.402965 2564 reconciler_common.go:299] "Volume detached for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/234ec34c-c9e5-42a6-b63c-c40f402dedbf-trusted-ca\") on node \"ip-10-0-129-2.ec2.internal\" DevicePath \"\"" Jul 23 19:21:45.403255 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:21:45.403019 2564 reconciler_common.go:299] "Volume detached for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/234ec34c-c9e5-42a6-b63c-c40f402dedbf-image-registry-private-configuration\") on node \"ip-10-0-129-2.ec2.internal\" DevicePath \"\"" Jul 23 19:21:45.403255 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:21:45.403035 2564 reconciler_common.go:299] "Volume detached for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/234ec34c-c9e5-42a6-b63c-c40f402dedbf-registry-tls\") on node \"ip-10-0-129-2.ec2.internal\" DevicePath \"\"" Jul 23 19:21:45.403255 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:21:45.403049 2564 reconciler_common.go:299] "Volume detached for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/234ec34c-c9e5-42a6-b63c-c40f402dedbf-registry-certificates\") on node \"ip-10-0-129-2.ec2.internal\" DevicePath \"\"" Jul 23 19:21:45.403255 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:21:45.403071 2564 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ftzrt\" (UniqueName: \"kubernetes.io/projected/234ec34c-c9e5-42a6-b63c-c40f402dedbf-kube-api-access-ftzrt\") on node \"ip-10-0-129-2.ec2.internal\" DevicePath \"\"" Jul 23 19:21:45.403255 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:21:45.403097 2564 reconciler_common.go:299] "Volume detached for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/234ec34c-c9e5-42a6-b63c-c40f402dedbf-ca-trust-extracted\") on node \"ip-10-0-129-2.ec2.internal\" DevicePath \"\"" Jul 23 19:21:45.403255 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:21:45.403138 2564 reconciler_common.go:299] "Volume detached for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/234ec34c-c9e5-42a6-b63c-c40f402dedbf-installation-pull-secrets\") on node \"ip-10-0-129-2.ec2.internal\" DevicePath \"\"" Jul 23 19:21:45.403255 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:21:45.403172 2564 reconciler_common.go:299] "Volume detached for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/234ec34c-c9e5-42a6-b63c-c40f402dedbf-bound-sa-token\") on node \"ip-10-0-129-2.ec2.internal\" DevicePath \"\"" Jul 23 19:21:46.339586 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:21:46.339556 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-qhrz5" event={"ID":"576f76ed-730c-49c2-a33e-1dc11370cad4","Type":"ContainerStarted","Data":"1faf43dcafe8d352363dd0d4f99c748e781376d3407307b7284738d40211d268"} Jul 23 19:21:46.339739 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:21:46.339634 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-qhrz5" event={"ID":"576f76ed-730c-49c2-a33e-1dc11370cad4","Type":"ContainerStarted","Data":"bd4fdf9329335496df648cc5dcc713d755cca363fedc5cf26ba3fd48ffb4deb9"} Jul 23 19:21:46.354774 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:21:46.354731 2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/network-metrics-daemon-qhrz5" podStartSLOduration=130.39493129 podStartE2EDuration="2m11.354717381s" totalImagesPullingTime="959.786091ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-23 19:19:35 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-23 19:21:46.353757554 +0000 UTC m=+132.205533799" watchObservedRunningTime="2026-07-23 19:21:46.354717381 +0000 UTC m=+132.206493627" Jul 23 19:21:46.656111 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:21:46.656062 2564 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="234ec34c-c9e5-42a6-b63c-c40f402dedbf" path="/var/lib/kubelet/pods/234ec34c-c9e5-42a6-b63c-c40f402dedbf/volumes" Jul 23 19:21:50.357247 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:21:50.357221 2564 generic.go:356] "Generic (PLEG): container finished" podID="678e0e44-d3bb-4269-99c9-43229d8f2291" containerID="97f6b99c91e2be492ba63b6543c5508e13570ebb4aade21c4c6f57c017bcdcf8" exitCode=0 Jul 23 19:21:50.357551 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:21:50.357292 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-5bcbc94c78-86qdl" event={"ID":"678e0e44-d3bb-4269-99c9-43229d8f2291","Type":"ContainerDied","Data":"97f6b99c91e2be492ba63b6543c5508e13570ebb4aade21c4c6f57c017bcdcf8"} Jul 23 19:21:50.357619 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:21:50.357601 2564 scope.go:117] "RemoveContainer" containerID="97f6b99c91e2be492ba63b6543c5508e13570ebb4aade21c4c6f57c017bcdcf8" Jul 23 19:21:51.363555 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:21:51.363517 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-5bcbc94c78-86qdl" event={"ID":"678e0e44-d3bb-4269-99c9-43229d8f2291","Type":"ContainerStarted","Data":"6a6f624d630624beb3cdc26c6075bbb166b8a6cda8aa16d1e77e05b9b8fa8835"} Jul 23 19:21:55.379658 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:21:55.379632 2564 generic.go:356] "Generic (PLEG): container finished" podID="a2c386d2-2c3a-4cf0-9a93-e18300e6905d" containerID="cf7d3650988b892d2d73179a08f88dfa49f72cc603d28c1ed2f1a371671d82c5" exitCode=0 Jul 23 19:21:55.379953 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:21:55.379679 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7658d9b749-6snc5" event={"ID":"a2c386d2-2c3a-4cf0-9a93-e18300e6905d","Type":"ContainerDied","Data":"cf7d3650988b892d2d73179a08f88dfa49f72cc603d28c1ed2f1a371671d82c5"} Jul 23 19:21:55.379953 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:21:55.379919 2564 scope.go:117] "RemoveContainer" containerID="cf7d3650988b892d2d73179a08f88dfa49f72cc603d28c1ed2f1a371671d82c5" Jul 23 19:21:56.385421 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:21:56.385392 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7658d9b749-6snc5" event={"ID":"a2c386d2-2c3a-4cf0-9a93-e18300e6905d","Type":"ContainerStarted","Data":"6e66e68636b3fd93fe0e1c8c8d1986c04d9b4ef5c445ae02cc25806dbf9fcca6"} Jul 23 19:22:49.804949 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:22:49.804916 2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/telemeter-client-75f59688bc-p2725"] Jul 23 19:22:49.805432 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:22:49.805165 2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="234ec34c-c9e5-42a6-b63c-c40f402dedbf" containerName="registry" Jul 23 19:22:49.805432 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:22:49.805175 2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="234ec34c-c9e5-42a6-b63c-c40f402dedbf" containerName="registry" Jul 23 19:22:49.805432 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:22:49.805225 2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="234ec34c-c9e5-42a6-b63c-c40f402dedbf" containerName="registry" Jul 23 19:22:49.809474 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:22:49.809457 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/telemeter-client-75f59688bc-p2725" Jul 23 19:22:49.811701 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:22:49.811667 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client-tls\"" Jul 23 19:22:49.811840 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:22:49.811800 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client-dockercfg-5748d\"" Jul 23 19:22:49.811840 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:22:49.811800 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client\"" Jul 23 19:22:49.812010 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:22:49.811817 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"federate-client-certs\"" Jul 23 19:22:49.812066 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:22:49.812006 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"telemeter-client-serving-certs-ca-bundle\"" Jul 23 19:22:49.812230 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:22:49.812213 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client-kube-rbac-proxy-config\"" Jul 23 19:22:49.817667 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:22:49.817647 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"telemeter-trusted-ca-bundle-8i12ta5c71j38\"" Jul 23 19:22:49.822532 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:22:49.822513 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/telemeter-client-75f59688bc-p2725"] Jul 23 19:22:49.939687 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:22:49.939667 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-b585q\" (UniqueName: \"kubernetes.io/projected/cbe5aee7-9398-489a-a1db-4b2c8e1d7b1f-kube-api-access-b585q\") pod \"telemeter-client-75f59688bc-p2725\" (UID: \"cbe5aee7-9398-489a-a1db-4b2c8e1d7b1f\") " pod="openshift-monitoring/telemeter-client-75f59688bc-p2725" Jul 23 19:22:49.939789 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:22:49.939703 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-telemeter-client\" (UniqueName: \"kubernetes.io/secret/cbe5aee7-9398-489a-a1db-4b2c8e1d7b1f-secret-telemeter-client\") pod \"telemeter-client-75f59688bc-p2725\" (UID: \"cbe5aee7-9398-489a-a1db-4b2c8e1d7b1f\") " pod="openshift-monitoring/telemeter-client-75f59688bc-p2725" Jul 23 19:22:49.939789 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:22:49.939727 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"telemeter-client-tls\" (UniqueName: \"kubernetes.io/secret/cbe5aee7-9398-489a-a1db-4b2c8e1d7b1f-telemeter-client-tls\") pod \"telemeter-client-75f59688bc-p2725\" (UID: \"cbe5aee7-9398-489a-a1db-4b2c8e1d7b1f\") " pod="openshift-monitoring/telemeter-client-75f59688bc-p2725" Jul 23 19:22:49.939789 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:22:49.939741 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"telemeter-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/cbe5aee7-9398-489a-a1db-4b2c8e1d7b1f-telemeter-trusted-ca-bundle\") pod \"telemeter-client-75f59688bc-p2725\" (UID: \"cbe5aee7-9398-489a-a1db-4b2c8e1d7b1f\") " pod="openshift-monitoring/telemeter-client-75f59688bc-p2725" Jul 23 19:22:49.939919 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:22:49.939804 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/cbe5aee7-9398-489a-a1db-4b2c8e1d7b1f-metrics-client-ca\") pod \"telemeter-client-75f59688bc-p2725\" (UID: \"cbe5aee7-9398-489a-a1db-4b2c8e1d7b1f\") " pod="openshift-monitoring/telemeter-client-75f59688bc-p2725" Jul 23 19:22:49.939919 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:22:49.939832 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/cbe5aee7-9398-489a-a1db-4b2c8e1d7b1f-serving-certs-ca-bundle\") pod \"telemeter-client-75f59688bc-p2725\" (UID: \"cbe5aee7-9398-489a-a1db-4b2c8e1d7b1f\") " pod="openshift-monitoring/telemeter-client-75f59688bc-p2725" Jul 23 19:22:49.939919 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:22:49.939862 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-telemeter-client-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/cbe5aee7-9398-489a-a1db-4b2c8e1d7b1f-secret-telemeter-client-kube-rbac-proxy-config\") pod \"telemeter-client-75f59688bc-p2725\" (UID: \"cbe5aee7-9398-489a-a1db-4b2c8e1d7b1f\") " pod="openshift-monitoring/telemeter-client-75f59688bc-p2725" Jul 23 19:22:49.939919 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:22:49.939878 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"federate-client-tls\" (UniqueName: \"kubernetes.io/secret/cbe5aee7-9398-489a-a1db-4b2c8e1d7b1f-federate-client-tls\") pod \"telemeter-client-75f59688bc-p2725\" (UID: \"cbe5aee7-9398-489a-a1db-4b2c8e1d7b1f\") " pod="openshift-monitoring/telemeter-client-75f59688bc-p2725" Jul 23 19:22:50.040563 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:22:50.040541 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b585q\" (UniqueName: \"kubernetes.io/projected/cbe5aee7-9398-489a-a1db-4b2c8e1d7b1f-kube-api-access-b585q\") pod \"telemeter-client-75f59688bc-p2725\" (UID: \"cbe5aee7-9398-489a-a1db-4b2c8e1d7b1f\") " pod="openshift-monitoring/telemeter-client-75f59688bc-p2725" Jul 23 19:22:50.040648 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:22:50.040566 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-telemeter-client\" (UniqueName: \"kubernetes.io/secret/cbe5aee7-9398-489a-a1db-4b2c8e1d7b1f-secret-telemeter-client\") pod \"telemeter-client-75f59688bc-p2725\" (UID: \"cbe5aee7-9398-489a-a1db-4b2c8e1d7b1f\") " pod="openshift-monitoring/telemeter-client-75f59688bc-p2725" Jul 23 19:22:50.040648 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:22:50.040589 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"telemeter-client-tls\" (UniqueName: \"kubernetes.io/secret/cbe5aee7-9398-489a-a1db-4b2c8e1d7b1f-telemeter-client-tls\") pod \"telemeter-client-75f59688bc-p2725\" (UID: \"cbe5aee7-9398-489a-a1db-4b2c8e1d7b1f\") " pod="openshift-monitoring/telemeter-client-75f59688bc-p2725" Jul 23 19:22:50.040648 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:22:50.040604 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"telemeter-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/cbe5aee7-9398-489a-a1db-4b2c8e1d7b1f-telemeter-trusted-ca-bundle\") pod \"telemeter-client-75f59688bc-p2725\" (UID: \"cbe5aee7-9398-489a-a1db-4b2c8e1d7b1f\") " pod="openshift-monitoring/telemeter-client-75f59688bc-p2725" Jul 23 19:22:50.040648 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:22:50.040623 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/cbe5aee7-9398-489a-a1db-4b2c8e1d7b1f-metrics-client-ca\") pod \"telemeter-client-75f59688bc-p2725\" (UID: \"cbe5aee7-9398-489a-a1db-4b2c8e1d7b1f\") " pod="openshift-monitoring/telemeter-client-75f59688bc-p2725" Jul 23 19:22:50.040832 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:22:50.040679 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/cbe5aee7-9398-489a-a1db-4b2c8e1d7b1f-serving-certs-ca-bundle\") pod \"telemeter-client-75f59688bc-p2725\" (UID: \"cbe5aee7-9398-489a-a1db-4b2c8e1d7b1f\") " pod="openshift-monitoring/telemeter-client-75f59688bc-p2725" Jul 23 19:22:50.040832 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:22:50.040743 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-telemeter-client-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/cbe5aee7-9398-489a-a1db-4b2c8e1d7b1f-secret-telemeter-client-kube-rbac-proxy-config\") pod \"telemeter-client-75f59688bc-p2725\" (UID: \"cbe5aee7-9398-489a-a1db-4b2c8e1d7b1f\") " pod="openshift-monitoring/telemeter-client-75f59688bc-p2725" Jul 23 19:22:50.040832 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:22:50.040773 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"federate-client-tls\" (UniqueName: \"kubernetes.io/secret/cbe5aee7-9398-489a-a1db-4b2c8e1d7b1f-federate-client-tls\") pod \"telemeter-client-75f59688bc-p2725\" (UID: \"cbe5aee7-9398-489a-a1db-4b2c8e1d7b1f\") " pod="openshift-monitoring/telemeter-client-75f59688bc-p2725" Jul 23 19:22:50.041540 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:22:50.041483 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/cbe5aee7-9398-489a-a1db-4b2c8e1d7b1f-serving-certs-ca-bundle\") pod \"telemeter-client-75f59688bc-p2725\" (UID: \"cbe5aee7-9398-489a-a1db-4b2c8e1d7b1f\") " pod="openshift-monitoring/telemeter-client-75f59688bc-p2725" Jul 23 19:22:50.041672 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:22:50.041654 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"telemeter-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/cbe5aee7-9398-489a-a1db-4b2c8e1d7b1f-telemeter-trusted-ca-bundle\") pod \"telemeter-client-75f59688bc-p2725\" (UID: \"cbe5aee7-9398-489a-a1db-4b2c8e1d7b1f\") " pod="openshift-monitoring/telemeter-client-75f59688bc-p2725" Jul 23 19:22:50.041672 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:22:50.041663 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/cbe5aee7-9398-489a-a1db-4b2c8e1d7b1f-metrics-client-ca\") pod \"telemeter-client-75f59688bc-p2725\" (UID: \"cbe5aee7-9398-489a-a1db-4b2c8e1d7b1f\") " pod="openshift-monitoring/telemeter-client-75f59688bc-p2725" Jul 23 19:22:50.043092 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:22:50.043070 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-telemeter-client\" (UniqueName: \"kubernetes.io/secret/cbe5aee7-9398-489a-a1db-4b2c8e1d7b1f-secret-telemeter-client\") pod \"telemeter-client-75f59688bc-p2725\" (UID: \"cbe5aee7-9398-489a-a1db-4b2c8e1d7b1f\") " pod="openshift-monitoring/telemeter-client-75f59688bc-p2725" Jul 23 19:22:50.043185 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:22:50.043140 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"telemeter-client-tls\" (UniqueName: \"kubernetes.io/secret/cbe5aee7-9398-489a-a1db-4b2c8e1d7b1f-telemeter-client-tls\") pod \"telemeter-client-75f59688bc-p2725\" (UID: \"cbe5aee7-9398-489a-a1db-4b2c8e1d7b1f\") " pod="openshift-monitoring/telemeter-client-75f59688bc-p2725" Jul 23 19:22:50.043633 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:22:50.043613 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-telemeter-client-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/cbe5aee7-9398-489a-a1db-4b2c8e1d7b1f-secret-telemeter-client-kube-rbac-proxy-config\") pod \"telemeter-client-75f59688bc-p2725\" (UID: \"cbe5aee7-9398-489a-a1db-4b2c8e1d7b1f\") " pod="openshift-monitoring/telemeter-client-75f59688bc-p2725" Jul 23 19:22:50.043697 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:22:50.043657 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"federate-client-tls\" (UniqueName: \"kubernetes.io/secret/cbe5aee7-9398-489a-a1db-4b2c8e1d7b1f-federate-client-tls\") pod \"telemeter-client-75f59688bc-p2725\" (UID: \"cbe5aee7-9398-489a-a1db-4b2c8e1d7b1f\") " pod="openshift-monitoring/telemeter-client-75f59688bc-p2725" Jul 23 19:22:50.049562 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:22:50.049542 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-b585q\" (UniqueName: \"kubernetes.io/projected/cbe5aee7-9398-489a-a1db-4b2c8e1d7b1f-kube-api-access-b585q\") pod \"telemeter-client-75f59688bc-p2725\" (UID: \"cbe5aee7-9398-489a-a1db-4b2c8e1d7b1f\") " pod="openshift-monitoring/telemeter-client-75f59688bc-p2725" Jul 23 19:22:50.119512 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:22:50.119465 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/telemeter-client-75f59688bc-p2725" Jul 23 19:22:50.255989 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:22:50.255954 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/telemeter-client-75f59688bc-p2725"] Jul 23 19:22:50.259630 ip-10-0-129-2 kubenswrapper[2564]: W0723 19:22:50.259605 2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcbe5aee7_9398_489a_a1db_4b2c8e1d7b1f.slice/crio-a458042be0ad1b68c7c9e52d6be4040f78156bdcbb2ed003214ee42d4599f168 WatchSource:0}: Error finding container a458042be0ad1b68c7c9e52d6be4040f78156bdcbb2ed003214ee42d4599f168: Status 404 returned error can't find the container with id a458042be0ad1b68c7c9e52d6be4040f78156bdcbb2ed003214ee42d4599f168 Jul 23 19:22:50.558452 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:22:50.558422 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-75f59688bc-p2725" event={"ID":"cbe5aee7-9398-489a-a1db-4b2c8e1d7b1f","Type":"ContainerStarted","Data":"a458042be0ad1b68c7c9e52d6be4040f78156bdcbb2ed003214ee42d4599f168"} Jul 23 19:22:52.569001 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:22:52.568937 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-75f59688bc-p2725" event={"ID":"cbe5aee7-9398-489a-a1db-4b2c8e1d7b1f","Type":"ContainerStarted","Data":"d07d91652f85d7f39c42232f09081baec73c1c5c855e6c009d06d18834d4edf8"} Jul 23 19:22:53.574620 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:22:53.574553 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-75f59688bc-p2725" event={"ID":"cbe5aee7-9398-489a-a1db-4b2c8e1d7b1f","Type":"ContainerStarted","Data":"a3e4aeffb6820da56fede517d61cdd7fe34fa7c35dfb0dd258c87ae31d5ba0de"} Jul 23 19:22:53.574620 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:22:53.574602 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-75f59688bc-p2725" event={"ID":"cbe5aee7-9398-489a-a1db-4b2c8e1d7b1f","Type":"ContainerStarted","Data":"8a3356c6c23b537af6c1bc570daf30b9420dca2617fa5f41b98a025e6bf3ebb6"} Jul 23 19:22:53.599246 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:22:53.599203 2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/telemeter-client-75f59688bc-p2725" podStartSLOduration=1.664831608 podStartE2EDuration="4.599187759s" totalImagesPullingTime="2.934356151s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-23 19:22:49 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-23 19:22:53.597258136 +0000 UTC m=+199.449034403" watchObservedRunningTime="2026-07-23 19:22:53.599187759 +0000 UTC m=+199.450964004" Jul 23 19:24:34.562644 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:24:34.562610 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-565b7b569d-z5wf4_ded5b3af-c74c-4d5a-aa77-62fe72d1dd29/console-operator/1.log" Jul 23 19:24:34.567340 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:24:34.567306 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-565b7b569d-z5wf4_ded5b3af-c74c-4d5a-aa77-62fe72d1dd29/console-operator/1.log" Jul 23 19:24:34.569406 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:24:34.569386 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-8dtlh_195ecf89-13ae-4311-bc72-fcdf78a5c4d6/ovn-acl-logging/0.log" Jul 23 19:24:34.573276 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:24:34.573255 2564 kubelet.go:1648] "Image garbage collection succeeded" Jul 23 19:24:34.575135 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:24:34.575115 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-8dtlh_195ecf89-13ae-4311-bc72-fcdf78a5c4d6/ovn-acl-logging/0.log" Jul 23 19:24:47.274968 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:24:47.274941 2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540zc8kg"] Jul 23 19:24:47.278118 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:24:47.278102 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540zc8kg" Jul 23 19:24:47.281875 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:24:47.281849 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"openshift-service-ca.crt\"" Jul 23 19:24:47.282001 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:24:47.281861 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"default-dockercfg-lx4sn\"" Jul 23 19:24:47.282001 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:24:47.281858 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"kube-root-ca.crt\"" Jul 23 19:24:47.313958 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:24:47.313934 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540zc8kg"] Jul 23 19:24:47.401206 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:24:47.401183 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/ea3c2156-e89b-4b60-9f6d-195b20a15024-util\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540zc8kg\" (UID: \"ea3c2156-e89b-4b60-9f6d-195b20a15024\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540zc8kg" Jul 23 19:24:47.401307 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:24:47.401223 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-r2zn9\" (UniqueName: \"kubernetes.io/projected/ea3c2156-e89b-4b60-9f6d-195b20a15024-kube-api-access-r2zn9\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540zc8kg\" (UID: \"ea3c2156-e89b-4b60-9f6d-195b20a15024\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540zc8kg" Jul 23 19:24:47.401307 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:24:47.401247 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/ea3c2156-e89b-4b60-9f6d-195b20a15024-bundle\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540zc8kg\" (UID: \"ea3c2156-e89b-4b60-9f6d-195b20a15024\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540zc8kg" Jul 23 19:24:47.502476 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:24:47.502456 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/ea3c2156-e89b-4b60-9f6d-195b20a15024-util\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540zc8kg\" (UID: \"ea3c2156-e89b-4b60-9f6d-195b20a15024\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540zc8kg" Jul 23 19:24:47.502553 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:24:47.502480 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r2zn9\" (UniqueName: \"kubernetes.io/projected/ea3c2156-e89b-4b60-9f6d-195b20a15024-kube-api-access-r2zn9\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540zc8kg\" (UID: \"ea3c2156-e89b-4b60-9f6d-195b20a15024\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540zc8kg" Jul 23 19:24:47.502553 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:24:47.502502 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/ea3c2156-e89b-4b60-9f6d-195b20a15024-bundle\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540zc8kg\" (UID: \"ea3c2156-e89b-4b60-9f6d-195b20a15024\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540zc8kg" Jul 23 19:24:47.502777 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:24:47.502763 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/ea3c2156-e89b-4b60-9f6d-195b20a15024-bundle\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540zc8kg\" (UID: \"ea3c2156-e89b-4b60-9f6d-195b20a15024\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540zc8kg" Jul 23 19:24:47.502826 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:24:47.502809 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/ea3c2156-e89b-4b60-9f6d-195b20a15024-util\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540zc8kg\" (UID: \"ea3c2156-e89b-4b60-9f6d-195b20a15024\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540zc8kg" Jul 23 19:24:47.511120 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:24:47.511100 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-r2zn9\" (UniqueName: \"kubernetes.io/projected/ea3c2156-e89b-4b60-9f6d-195b20a15024-kube-api-access-r2zn9\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540zc8kg\" (UID: \"ea3c2156-e89b-4b60-9f6d-195b20a15024\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540zc8kg" Jul 23 19:24:47.586724 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:24:47.586681 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540zc8kg" Jul 23 19:24:47.714208 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:24:47.714185 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540zc8kg"] Jul 23 19:24:47.716120 ip-10-0-129-2 kubenswrapper[2564]: W0723 19:24:47.716090 2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podea3c2156_e89b_4b60_9f6d_195b20a15024.slice/crio-45790117c189b95ce723d02094b43d951b019d4a22b9e64b02eec2701203d965 WatchSource:0}: Error finding container 45790117c189b95ce723d02094b43d951b019d4a22b9e64b02eec2701203d965: Status 404 returned error can't find the container with id 45790117c189b95ce723d02094b43d951b019d4a22b9e64b02eec2701203d965 Jul 23 19:24:47.718058 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:24:47.718040 2564 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Jul 23 19:24:47.946215 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:24:47.946186 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540zc8kg" event={"ID":"ea3c2156-e89b-4b60-9f6d-195b20a15024","Type":"ContainerStarted","Data":"45790117c189b95ce723d02094b43d951b019d4a22b9e64b02eec2701203d965"} Jul 23 19:24:53.972908 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:24:53.972851 2564 generic.go:356] "Generic (PLEG): container finished" podID="ea3c2156-e89b-4b60-9f6d-195b20a15024" containerID="51fae951e318405f8ef25a99d9f08cd22b5fac213c3a0ab9025ef6f899a3f612" exitCode=0 Jul 23 19:24:53.973226 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:24:53.972934 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540zc8kg" event={"ID":"ea3c2156-e89b-4b60-9f6d-195b20a15024","Type":"ContainerDied","Data":"51fae951e318405f8ef25a99d9f08cd22b5fac213c3a0ab9025ef6f899a3f612"} Jul 23 19:24:55.983161 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:24:55.983126 2564 generic.go:356] "Generic (PLEG): container finished" podID="ea3c2156-e89b-4b60-9f6d-195b20a15024" containerID="acae6c822eee84de2641bc3d4794262c6d07383c2c342de9273d56a2f099e51f" exitCode=0 Jul 23 19:24:55.983488 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:24:55.983218 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540zc8kg" event={"ID":"ea3c2156-e89b-4b60-9f6d-195b20a15024","Type":"ContainerDied","Data":"acae6c822eee84de2641bc3d4794262c6d07383c2c342de9273d56a2f099e51f"} Jul 23 19:25:04.014892 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:25:04.014863 2564 generic.go:356] "Generic (PLEG): container finished" podID="ea3c2156-e89b-4b60-9f6d-195b20a15024" containerID="8339048eb8deebcb356ffdf1a43f6210014a8ac578a4e601e64b7ad37760d951" exitCode=0 Jul 23 19:25:04.015205 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:25:04.014949 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540zc8kg" event={"ID":"ea3c2156-e89b-4b60-9f6d-195b20a15024","Type":"ContainerDied","Data":"8339048eb8deebcb356ffdf1a43f6210014a8ac578a4e601e64b7ad37760d951"} Jul 23 19:25:05.136156 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:25:05.136132 2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540zc8kg" Jul 23 19:25:05.232924 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:25:05.232903 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/ea3c2156-e89b-4b60-9f6d-195b20a15024-bundle\") pod \"ea3c2156-e89b-4b60-9f6d-195b20a15024\" (UID: \"ea3c2156-e89b-4b60-9f6d-195b20a15024\") " Jul 23 19:25:05.233036 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:25:05.232944 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/ea3c2156-e89b-4b60-9f6d-195b20a15024-util\") pod \"ea3c2156-e89b-4b60-9f6d-195b20a15024\" (UID: \"ea3c2156-e89b-4b60-9f6d-195b20a15024\") " Jul 23 19:25:05.233036 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:25:05.232989 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-r2zn9\" (UniqueName: \"kubernetes.io/projected/ea3c2156-e89b-4b60-9f6d-195b20a15024-kube-api-access-r2zn9\") pod \"ea3c2156-e89b-4b60-9f6d-195b20a15024\" (UID: \"ea3c2156-e89b-4b60-9f6d-195b20a15024\") " Jul 23 19:25:05.233521 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:25:05.233498 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ea3c2156-e89b-4b60-9f6d-195b20a15024-bundle" (OuterVolumeSpecName: "bundle") pod "ea3c2156-e89b-4b60-9f6d-195b20a15024" (UID: "ea3c2156-e89b-4b60-9f6d-195b20a15024"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 23 19:25:05.235081 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:25:05.235053 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ea3c2156-e89b-4b60-9f6d-195b20a15024-kube-api-access-r2zn9" (OuterVolumeSpecName: "kube-api-access-r2zn9") pod "ea3c2156-e89b-4b60-9f6d-195b20a15024" (UID: "ea3c2156-e89b-4b60-9f6d-195b20a15024"). InnerVolumeSpecName "kube-api-access-r2zn9". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 23 19:25:05.237060 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:25:05.237041 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ea3c2156-e89b-4b60-9f6d-195b20a15024-util" (OuterVolumeSpecName: "util") pod "ea3c2156-e89b-4b60-9f6d-195b20a15024" (UID: "ea3c2156-e89b-4b60-9f6d-195b20a15024"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 23 19:25:05.334077 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:25:05.334033 2564 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/ea3c2156-e89b-4b60-9f6d-195b20a15024-util\") on node \"ip-10-0-129-2.ec2.internal\" DevicePath \"\"" Jul 23 19:25:05.334077 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:25:05.334052 2564 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-r2zn9\" (UniqueName: \"kubernetes.io/projected/ea3c2156-e89b-4b60-9f6d-195b20a15024-kube-api-access-r2zn9\") on node \"ip-10-0-129-2.ec2.internal\" DevicePath \"\"" Jul 23 19:25:05.334077 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:25:05.334061 2564 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/ea3c2156-e89b-4b60-9f6d-195b20a15024-bundle\") on node \"ip-10-0-129-2.ec2.internal\" DevicePath \"\"" Jul 23 19:25:06.024138 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:25:06.024111 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540zc8kg" event={"ID":"ea3c2156-e89b-4b60-9f6d-195b20a15024","Type":"ContainerDied","Data":"45790117c189b95ce723d02094b43d951b019d4a22b9e64b02eec2701203d965"} Jul 23 19:25:06.024138 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:25:06.024127 2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540zc8kg" Jul 23 19:25:06.024281 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:25:06.024150 2564 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="45790117c189b95ce723d02094b43d951b019d4a22b9e64b02eec2701203d965" Jul 23 19:25:10.198613 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:25:10.198583 2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-skzl2"] Jul 23 19:25:10.198958 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:25:10.198855 2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ea3c2156-e89b-4b60-9f6d-195b20a15024" containerName="pull" Jul 23 19:25:10.198958 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:25:10.198869 2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="ea3c2156-e89b-4b60-9f6d-195b20a15024" containerName="pull" Jul 23 19:25:10.198958 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:25:10.198884 2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ea3c2156-e89b-4b60-9f6d-195b20a15024" containerName="extract" Jul 23 19:25:10.198958 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:25:10.198890 2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="ea3c2156-e89b-4b60-9f6d-195b20a15024" containerName="extract" Jul 23 19:25:10.198958 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:25:10.198909 2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ea3c2156-e89b-4b60-9f6d-195b20a15024" containerName="util" Jul 23 19:25:10.198958 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:25:10.198914 2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="ea3c2156-e89b-4b60-9f6d-195b20a15024" containerName="util" Jul 23 19:25:10.199127 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:25:10.198962 2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="ea3c2156-e89b-4b60-9f6d-195b20a15024" containerName="extract" Jul 23 19:25:10.253712 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:25:10.253685 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-skzl2"] Jul 23 19:25:10.253841 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:25:10.253773 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-skzl2" Jul 23 19:25:10.256494 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:25:10.256478 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager-operator\"/\"openshift-service-ca.crt\"" Jul 23 19:25:10.256494 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:25:10.256485 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"cert-manager-operator\"/\"cert-manager-operator-controller-manager-dockercfg-mj4hs\"" Jul 23 19:25:10.256753 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:25:10.256741 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager-operator\"/\"kube-root-ca.crt\"" Jul 23 19:25:10.365898 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:25:10.365875 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/f2491db7-9e57-4217-88c7-712d6163fa9b-tmp\") pod \"cert-manager-operator-controller-manager-55c444bf86-skzl2\" (UID: \"f2491db7-9e57-4217-88c7-712d6163fa9b\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-skzl2" Jul 23 19:25:10.365988 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:25:10.365927 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2m8m9\" (UniqueName: \"kubernetes.io/projected/f2491db7-9e57-4217-88c7-712d6163fa9b-kube-api-access-2m8m9\") pod \"cert-manager-operator-controller-manager-55c444bf86-skzl2\" (UID: \"f2491db7-9e57-4217-88c7-712d6163fa9b\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-skzl2" Jul 23 19:25:10.466479 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:25:10.466426 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2m8m9\" (UniqueName: \"kubernetes.io/projected/f2491db7-9e57-4217-88c7-712d6163fa9b-kube-api-access-2m8m9\") pod \"cert-manager-operator-controller-manager-55c444bf86-skzl2\" (UID: \"f2491db7-9e57-4217-88c7-712d6163fa9b\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-skzl2" Jul 23 19:25:10.466479 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:25:10.466460 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/f2491db7-9e57-4217-88c7-712d6163fa9b-tmp\") pod \"cert-manager-operator-controller-manager-55c444bf86-skzl2\" (UID: \"f2491db7-9e57-4217-88c7-712d6163fa9b\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-skzl2" Jul 23 19:25:10.466751 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:25:10.466738 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/f2491db7-9e57-4217-88c7-712d6163fa9b-tmp\") pod \"cert-manager-operator-controller-manager-55c444bf86-skzl2\" (UID: \"f2491db7-9e57-4217-88c7-712d6163fa9b\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-skzl2" Jul 23 19:25:10.476010 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:25:10.475991 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2m8m9\" (UniqueName: \"kubernetes.io/projected/f2491db7-9e57-4217-88c7-712d6163fa9b-kube-api-access-2m8m9\") pod \"cert-manager-operator-controller-manager-55c444bf86-skzl2\" (UID: \"f2491db7-9e57-4217-88c7-712d6163fa9b\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-skzl2" Jul 23 19:25:10.562085 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:25:10.562069 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-skzl2" Jul 23 19:25:10.711126 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:25:10.711103 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-skzl2"] Jul 23 19:25:10.713350 ip-10-0-129-2 kubenswrapper[2564]: W0723 19:25:10.713306 2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf2491db7_9e57_4217_88c7_712d6163fa9b.slice/crio-27719a39eacd41b46605330998283f84b1829ed20f5b91f0490cf9b58450064b WatchSource:0}: Error finding container 27719a39eacd41b46605330998283f84b1829ed20f5b91f0490cf9b58450064b: Status 404 returned error can't find the container with id 27719a39eacd41b46605330998283f84b1829ed20f5b91f0490cf9b58450064b Jul 23 19:25:11.041191 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:25:11.041136 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-skzl2" event={"ID":"f2491db7-9e57-4217-88c7-712d6163fa9b","Type":"ContainerStarted","Data":"27719a39eacd41b46605330998283f84b1829ed20f5b91f0490cf9b58450064b"} Jul 23 19:25:14.055271 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:25:14.055198 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-skzl2" event={"ID":"f2491db7-9e57-4217-88c7-712d6163fa9b","Type":"ContainerStarted","Data":"7ac9f6dcf38f85649deff700d1c9420b0ae687daa148274bf01d842d200fdb90"} Jul 23 19:25:14.081082 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:25:14.081035 2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-skzl2" podStartSLOduration=1.709217853 podStartE2EDuration="4.081021623s" totalImagesPullingTime="2.37180377s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-23 19:25:10 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-23 19:25:14.080265816 +0000 UTC m=+339.932042062" watchObservedRunningTime="2026-07-23 19:25:14.081021623 +0000 UTC m=+339.932797868" Jul 23 19:25:14.867625 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:25:14.867600 2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f5gzc7"] Jul 23 19:25:14.884780 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:25:14.884759 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f5gzc7" Jul 23 19:25:14.892541 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:25:14.892520 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f5gzc7"] Jul 23 19:25:14.894379 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:25:14.894363 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"default-dockercfg-lx4sn\"" Jul 23 19:25:14.895753 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:25:14.895735 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"openshift-service-ca.crt\"" Jul 23 19:25:14.898076 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:25:14.898058 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"kube-root-ca.crt\"" Jul 23 19:25:15.003360 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:25:15.003306 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9hsbk\" (UniqueName: \"kubernetes.io/projected/da80b5e5-a87f-4093-b6d5-b89f2ed11b9f-kube-api-access-9hsbk\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f5gzc7\" (UID: \"da80b5e5-a87f-4093-b6d5-b89f2ed11b9f\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f5gzc7" Jul 23 19:25:15.003442 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:25:15.003418 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/da80b5e5-a87f-4093-b6d5-b89f2ed11b9f-util\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f5gzc7\" (UID: \"da80b5e5-a87f-4093-b6d5-b89f2ed11b9f\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f5gzc7" Jul 23 19:25:15.003490 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:25:15.003476 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/da80b5e5-a87f-4093-b6d5-b89f2ed11b9f-bundle\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f5gzc7\" (UID: \"da80b5e5-a87f-4093-b6d5-b89f2ed11b9f\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f5gzc7" Jul 23 19:25:15.104108 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:25:15.104088 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/da80b5e5-a87f-4093-b6d5-b89f2ed11b9f-bundle\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f5gzc7\" (UID: \"da80b5e5-a87f-4093-b6d5-b89f2ed11b9f\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f5gzc7" Jul 23 19:25:15.104395 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:25:15.104127 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9hsbk\" (UniqueName: \"kubernetes.io/projected/da80b5e5-a87f-4093-b6d5-b89f2ed11b9f-kube-api-access-9hsbk\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f5gzc7\" (UID: \"da80b5e5-a87f-4093-b6d5-b89f2ed11b9f\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f5gzc7" Jul 23 19:25:15.104395 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:25:15.104153 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/da80b5e5-a87f-4093-b6d5-b89f2ed11b9f-util\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f5gzc7\" (UID: \"da80b5e5-a87f-4093-b6d5-b89f2ed11b9f\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f5gzc7" Jul 23 19:25:15.104465 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:25:15.104421 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/da80b5e5-a87f-4093-b6d5-b89f2ed11b9f-bundle\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f5gzc7\" (UID: \"da80b5e5-a87f-4093-b6d5-b89f2ed11b9f\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f5gzc7" Jul 23 19:25:15.104506 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:25:15.104491 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/da80b5e5-a87f-4093-b6d5-b89f2ed11b9f-util\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f5gzc7\" (UID: \"da80b5e5-a87f-4093-b6d5-b89f2ed11b9f\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f5gzc7" Jul 23 19:25:15.127423 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:25:15.127381 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9hsbk\" (UniqueName: \"kubernetes.io/projected/da80b5e5-a87f-4093-b6d5-b89f2ed11b9f-kube-api-access-9hsbk\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f5gzc7\" (UID: \"da80b5e5-a87f-4093-b6d5-b89f2ed11b9f\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f5gzc7" Jul 23 19:25:15.193243 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:25:15.193221 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f5gzc7" Jul 23 19:25:15.337395 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:25:15.337364 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f5gzc7"] Jul 23 19:25:15.339905 ip-10-0-129-2 kubenswrapper[2564]: W0723 19:25:15.339866 2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podda80b5e5_a87f_4093_b6d5_b89f2ed11b9f.slice/crio-c006726d59e577db295400e37483644bc9f16208cfd306e5d42d7c45c7144c50 WatchSource:0}: Error finding container c006726d59e577db295400e37483644bc9f16208cfd306e5d42d7c45c7144c50: Status 404 returned error can't find the container with id c006726d59e577db295400e37483644bc9f16208cfd306e5d42d7c45c7144c50 Jul 23 19:25:16.063927 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:25:16.063899 2564 generic.go:356] "Generic (PLEG): container finished" podID="da80b5e5-a87f-4093-b6d5-b89f2ed11b9f" containerID="b6fe9f1ef9887041e8a3d7a06e5d7a72caf8bfa5942853922494611e98946929" exitCode=0 Jul 23 19:25:16.064047 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:25:16.063930 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f5gzc7" event={"ID":"da80b5e5-a87f-4093-b6d5-b89f2ed11b9f","Type":"ContainerDied","Data":"b6fe9f1ef9887041e8a3d7a06e5d7a72caf8bfa5942853922494611e98946929"} Jul 23 19:25:16.064047 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:25:16.063966 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f5gzc7" event={"ID":"da80b5e5-a87f-4093-b6d5-b89f2ed11b9f","Type":"ContainerStarted","Data":"c006726d59e577db295400e37483644bc9f16208cfd306e5d42d7c45c7144c50"} Jul 23 19:25:17.285526 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:25:17.285499 2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["cert-manager/cert-manager-cainjector-559cdd8c87-vl4dm"] Jul 23 19:25:17.288181 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:25:17.288161 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-cainjector-559cdd8c87-vl4dm" Jul 23 19:25:17.290633 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:25:17.290603 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager\"/\"kube-root-ca.crt\"" Jul 23 19:25:17.291489 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:25:17.291470 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"cert-manager\"/\"cert-manager-cainjector-dockercfg-zsgl9\"" Jul 23 19:25:17.291566 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:25:17.291470 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager\"/\"openshift-service-ca.crt\"" Jul 23 19:25:17.299067 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:25:17.299045 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-cainjector-559cdd8c87-vl4dm"] Jul 23 19:25:17.419661 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:25:17.419640 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/b9a993b2-9410-4bdf-89ae-785caf999fac-bound-sa-token\") pod \"cert-manager-cainjector-559cdd8c87-vl4dm\" (UID: \"b9a993b2-9410-4bdf-89ae-785caf999fac\") " pod="cert-manager/cert-manager-cainjector-559cdd8c87-vl4dm" Jul 23 19:25:17.419740 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:25:17.419689 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gsl6z\" (UniqueName: \"kubernetes.io/projected/b9a993b2-9410-4bdf-89ae-785caf999fac-kube-api-access-gsl6z\") pod \"cert-manager-cainjector-559cdd8c87-vl4dm\" (UID: \"b9a993b2-9410-4bdf-89ae-785caf999fac\") " pod="cert-manager/cert-manager-cainjector-559cdd8c87-vl4dm" Jul 23 19:25:17.520894 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:25:17.520875 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gsl6z\" (UniqueName: \"kubernetes.io/projected/b9a993b2-9410-4bdf-89ae-785caf999fac-kube-api-access-gsl6z\") pod \"cert-manager-cainjector-559cdd8c87-vl4dm\" (UID: \"b9a993b2-9410-4bdf-89ae-785caf999fac\") " pod="cert-manager/cert-manager-cainjector-559cdd8c87-vl4dm" Jul 23 19:25:17.520982 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:25:17.520917 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/b9a993b2-9410-4bdf-89ae-785caf999fac-bound-sa-token\") pod \"cert-manager-cainjector-559cdd8c87-vl4dm\" (UID: \"b9a993b2-9410-4bdf-89ae-785caf999fac\") " pod="cert-manager/cert-manager-cainjector-559cdd8c87-vl4dm" Jul 23 19:25:17.528800 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:25:17.528772 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/b9a993b2-9410-4bdf-89ae-785caf999fac-bound-sa-token\") pod \"cert-manager-cainjector-559cdd8c87-vl4dm\" (UID: \"b9a993b2-9410-4bdf-89ae-785caf999fac\") " pod="cert-manager/cert-manager-cainjector-559cdd8c87-vl4dm" Jul 23 19:25:17.528881 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:25:17.528866 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-gsl6z\" (UniqueName: \"kubernetes.io/projected/b9a993b2-9410-4bdf-89ae-785caf999fac-kube-api-access-gsl6z\") pod \"cert-manager-cainjector-559cdd8c87-vl4dm\" (UID: \"b9a993b2-9410-4bdf-89ae-785caf999fac\") " pod="cert-manager/cert-manager-cainjector-559cdd8c87-vl4dm" Jul 23 19:25:17.613450 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:25:17.613403 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-cainjector-559cdd8c87-vl4dm" Jul 23 19:25:17.735580 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:25:17.735545 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-cainjector-559cdd8c87-vl4dm"] Jul 23 19:25:17.737729 ip-10-0-129-2 kubenswrapper[2564]: W0723 19:25:17.737704 2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podb9a993b2_9410_4bdf_89ae_785caf999fac.slice/crio-e4c1d158232b9fc733d7a9b7b96aea74f37e6e8e73d06af9665345577114586c WatchSource:0}: Error finding container e4c1d158232b9fc733d7a9b7b96aea74f37e6e8e73d06af9665345577114586c: Status 404 returned error can't find the container with id e4c1d158232b9fc733d7a9b7b96aea74f37e6e8e73d06af9665345577114586c Jul 23 19:25:18.073650 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:25:18.073614 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-cainjector-559cdd8c87-vl4dm" event={"ID":"b9a993b2-9410-4bdf-89ae-785caf999fac","Type":"ContainerStarted","Data":"e4c1d158232b9fc733d7a9b7b96aea74f37e6e8e73d06af9665345577114586c"} Jul 23 19:25:20.085932 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:25:20.085903 2564 generic.go:356] "Generic (PLEG): container finished" podID="da80b5e5-a87f-4093-b6d5-b89f2ed11b9f" containerID="55a0e0a19095ccad0ae8122e620fd65d17fa14613d38f8d64e5e5eeb401b58a9" exitCode=0 Jul 23 19:25:20.086421 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:25:20.085946 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f5gzc7" event={"ID":"da80b5e5-a87f-4093-b6d5-b89f2ed11b9f","Type":"ContainerDied","Data":"55a0e0a19095ccad0ae8122e620fd65d17fa14613d38f8d64e5e5eeb401b58a9"} Jul 23 19:25:21.091948 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:25:21.091916 2564 generic.go:356] "Generic (PLEG): container finished" podID="da80b5e5-a87f-4093-b6d5-b89f2ed11b9f" containerID="d73bd321f5a8833f6982f482096f1ffc9c25643cba89cddd99f94a17eda7cf11" exitCode=0 Jul 23 19:25:21.092315 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:25:21.091949 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f5gzc7" event={"ID":"da80b5e5-a87f-4093-b6d5-b89f2ed11b9f","Type":"ContainerDied","Data":"d73bd321f5a8833f6982f482096f1ffc9c25643cba89cddd99f94a17eda7cf11"} Jul 23 19:25:21.093412 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:25:21.093392 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-cainjector-559cdd8c87-vl4dm" event={"ID":"b9a993b2-9410-4bdf-89ae-785caf999fac","Type":"ContainerStarted","Data":"c32a25a817aa95bfc6a10cc7c74a2272c861cbb1a968098f2793bd81400e7e5d"} Jul 23 19:25:21.131121 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:25:21.131051 2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="cert-manager/cert-manager-cainjector-559cdd8c87-vl4dm" podStartSLOduration=0.916943684 podStartE2EDuration="4.131040055s" totalImagesPullingTime="3.214096371s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-23 19:25:17 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-23 19:25:21.130087823 +0000 UTC m=+346.981864080" watchObservedRunningTime="2026-07-23 19:25:21.131040055 +0000 UTC m=+346.982816300" Jul 23 19:25:21.356500 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:25:21.356468 2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["cert-manager/cert-manager-webhook-6486ffd997-cszws"] Jul 23 19:25:21.360236 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:25:21.360216 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-webhook-6486ffd997-cszws" Jul 23 19:25:21.362781 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:25:21.362763 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"cert-manager\"/\"cert-manager-webhook-dockercfg-zz5rt\"" Jul 23 19:25:21.368470 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:25:21.368446 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-webhook-6486ffd997-cszws"] Jul 23 19:25:21.556132 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:25:21.556100 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/02a19335-f416-4cce-bdd3-b696f93609b9-bound-sa-token\") pod \"cert-manager-webhook-6486ffd997-cszws\" (UID: \"02a19335-f416-4cce-bdd3-b696f93609b9\") " pod="cert-manager/cert-manager-webhook-6486ffd997-cszws" Jul 23 19:25:21.556259 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:25:21.556177 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-98hpn\" (UniqueName: \"kubernetes.io/projected/02a19335-f416-4cce-bdd3-b696f93609b9-kube-api-access-98hpn\") pod \"cert-manager-webhook-6486ffd997-cszws\" (UID: \"02a19335-f416-4cce-bdd3-b696f93609b9\") " pod="cert-manager/cert-manager-webhook-6486ffd997-cszws" Jul 23 19:25:21.656475 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:25:21.656456 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-98hpn\" (UniqueName: \"kubernetes.io/projected/02a19335-f416-4cce-bdd3-b696f93609b9-kube-api-access-98hpn\") pod \"cert-manager-webhook-6486ffd997-cszws\" (UID: \"02a19335-f416-4cce-bdd3-b696f93609b9\") " pod="cert-manager/cert-manager-webhook-6486ffd997-cszws" Jul 23 19:25:21.656577 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:25:21.656500 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/02a19335-f416-4cce-bdd3-b696f93609b9-bound-sa-token\") pod \"cert-manager-webhook-6486ffd997-cszws\" (UID: \"02a19335-f416-4cce-bdd3-b696f93609b9\") " pod="cert-manager/cert-manager-webhook-6486ffd997-cszws" Jul 23 19:25:21.668887 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:25:21.668864 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/02a19335-f416-4cce-bdd3-b696f93609b9-bound-sa-token\") pod \"cert-manager-webhook-6486ffd997-cszws\" (UID: \"02a19335-f416-4cce-bdd3-b696f93609b9\") " pod="cert-manager/cert-manager-webhook-6486ffd997-cszws" Jul 23 19:25:21.669011 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:25:21.668926 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-98hpn\" (UniqueName: \"kubernetes.io/projected/02a19335-f416-4cce-bdd3-b696f93609b9-kube-api-access-98hpn\") pod \"cert-manager-webhook-6486ffd997-cszws\" (UID: \"02a19335-f416-4cce-bdd3-b696f93609b9\") " pod="cert-manager/cert-manager-webhook-6486ffd997-cszws" Jul 23 19:25:21.670648 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:25:21.670623 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-webhook-6486ffd997-cszws" Jul 23 19:25:21.793138 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:25:21.793117 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-webhook-6486ffd997-cszws"] Jul 23 19:25:21.795480 ip-10-0-129-2 kubenswrapper[2564]: W0723 19:25:21.795457 2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod02a19335_f416_4cce_bdd3_b696f93609b9.slice/crio-7e2904036592aea37f2946902ddfe9433a255ba9ad0853865a6cc009435cabc9 WatchSource:0}: Error finding container 7e2904036592aea37f2946902ddfe9433a255ba9ad0853865a6cc009435cabc9: Status 404 returned error can't find the container with id 7e2904036592aea37f2946902ddfe9433a255ba9ad0853865a6cc009435cabc9 Jul 23 19:25:22.098705 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:25:22.098636 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-webhook-6486ffd997-cszws" event={"ID":"02a19335-f416-4cce-bdd3-b696f93609b9","Type":"ContainerStarted","Data":"f784492185092cb2720ffc5dd934b41fb1f8cdd06236607274f65669b1184925"} Jul 23 19:25:22.098705 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:25:22.098674 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-webhook-6486ffd997-cszws" event={"ID":"02a19335-f416-4cce-bdd3-b696f93609b9","Type":"ContainerStarted","Data":"7e2904036592aea37f2946902ddfe9433a255ba9ad0853865a6cc009435cabc9"} Jul 23 19:25:22.099093 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:25:22.099018 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="cert-manager/cert-manager-webhook-6486ffd997-cszws" Jul 23 19:25:22.116912 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:25:22.116866 2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="cert-manager/cert-manager-webhook-6486ffd997-cszws" podStartSLOduration=1.116850951 podStartE2EDuration="1.116850951s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-23 19:25:21 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-23 19:25:22.114934348 +0000 UTC m=+347.966710593" watchObservedRunningTime="2026-07-23 19:25:22.116850951 +0000 UTC m=+347.968627196" Jul 23 19:25:22.223341 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:25:22.223304 2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f5gzc7" Jul 23 19:25:22.261608 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:25:22.261582 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-9hsbk\" (UniqueName: \"kubernetes.io/projected/da80b5e5-a87f-4093-b6d5-b89f2ed11b9f-kube-api-access-9hsbk\") pod \"da80b5e5-a87f-4093-b6d5-b89f2ed11b9f\" (UID: \"da80b5e5-a87f-4093-b6d5-b89f2ed11b9f\") " Jul 23 19:25:22.261727 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:25:22.261682 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/da80b5e5-a87f-4093-b6d5-b89f2ed11b9f-util\") pod \"da80b5e5-a87f-4093-b6d5-b89f2ed11b9f\" (UID: \"da80b5e5-a87f-4093-b6d5-b89f2ed11b9f\") " Jul 23 19:25:22.261727 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:25:22.261714 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/da80b5e5-a87f-4093-b6d5-b89f2ed11b9f-bundle\") pod \"da80b5e5-a87f-4093-b6d5-b89f2ed11b9f\" (UID: \"da80b5e5-a87f-4093-b6d5-b89f2ed11b9f\") " Jul 23 19:25:22.262108 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:25:22.262085 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/da80b5e5-a87f-4093-b6d5-b89f2ed11b9f-bundle" (OuterVolumeSpecName: "bundle") pod "da80b5e5-a87f-4093-b6d5-b89f2ed11b9f" (UID: "da80b5e5-a87f-4093-b6d5-b89f2ed11b9f"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 23 19:25:22.263595 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:25:22.263577 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/da80b5e5-a87f-4093-b6d5-b89f2ed11b9f-kube-api-access-9hsbk" (OuterVolumeSpecName: "kube-api-access-9hsbk") pod "da80b5e5-a87f-4093-b6d5-b89f2ed11b9f" (UID: "da80b5e5-a87f-4093-b6d5-b89f2ed11b9f"). InnerVolumeSpecName "kube-api-access-9hsbk". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 23 19:25:22.267589 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:25:22.267567 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/da80b5e5-a87f-4093-b6d5-b89f2ed11b9f-util" (OuterVolumeSpecName: "util") pod "da80b5e5-a87f-4093-b6d5-b89f2ed11b9f" (UID: "da80b5e5-a87f-4093-b6d5-b89f2ed11b9f"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 23 19:25:22.362304 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:25:22.362251 2564 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/da80b5e5-a87f-4093-b6d5-b89f2ed11b9f-util\") on node \"ip-10-0-129-2.ec2.internal\" DevicePath \"\"" Jul 23 19:25:22.362304 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:25:22.362273 2564 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/da80b5e5-a87f-4093-b6d5-b89f2ed11b9f-bundle\") on node \"ip-10-0-129-2.ec2.internal\" DevicePath \"\"" Jul 23 19:25:22.362304 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:25:22.362287 2564 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9hsbk\" (UniqueName: \"kubernetes.io/projected/da80b5e5-a87f-4093-b6d5-b89f2ed11b9f-kube-api-access-9hsbk\") on node \"ip-10-0-129-2.ec2.internal\" DevicePath \"\"" Jul 23 19:25:23.104186 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:25:23.104131 2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f5gzc7" Jul 23 19:25:23.104186 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:25:23.104147 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f5gzc7" event={"ID":"da80b5e5-a87f-4093-b6d5-b89f2ed11b9f","Type":"ContainerDied","Data":"c006726d59e577db295400e37483644bc9f16208cfd306e5d42d7c45c7144c50"} Jul 23 19:25:23.104186 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:25:23.104188 2564 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="c006726d59e577db295400e37483644bc9f16208cfd306e5d42d7c45c7144c50" Jul 23 19:25:29.110852 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:25:29.110772 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="cert-manager/cert-manager-webhook-6486ffd997-cszws" Jul 23 19:25:40.770052 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:25:40.770023 2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afgdhjb"] Jul 23 19:25:40.770413 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:25:40.770296 2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="da80b5e5-a87f-4093-b6d5-b89f2ed11b9f" containerName="util" Jul 23 19:25:40.770413 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:25:40.770310 2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="da80b5e5-a87f-4093-b6d5-b89f2ed11b9f" containerName="util" Jul 23 19:25:40.770413 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:25:40.770334 2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="da80b5e5-a87f-4093-b6d5-b89f2ed11b9f" containerName="pull" Jul 23 19:25:40.770413 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:25:40.770340 2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="da80b5e5-a87f-4093-b6d5-b89f2ed11b9f" containerName="pull" Jul 23 19:25:40.770413 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:25:40.770361 2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="da80b5e5-a87f-4093-b6d5-b89f2ed11b9f" containerName="extract" Jul 23 19:25:40.770413 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:25:40.770366 2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="da80b5e5-a87f-4093-b6d5-b89f2ed11b9f" containerName="extract" Jul 23 19:25:40.770584 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:25:40.770423 2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="da80b5e5-a87f-4093-b6d5-b89f2ed11b9f" containerName="extract" Jul 23 19:25:40.773476 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:25:40.773461 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afgdhjb" Jul 23 19:25:40.775819 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:25:40.775789 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"default-dockercfg-lx4sn\"" Jul 23 19:25:40.776052 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:25:40.776039 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"kube-root-ca.crt\"" Jul 23 19:25:40.776472 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:25:40.776459 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"openshift-service-ca.crt\"" Jul 23 19:25:40.781951 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:25:40.781930 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afgdhjb"] Jul 23 19:25:40.882767 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:25:40.882749 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/89ea6c23-db5b-43b2-aeaa-71b9645bfbac-util\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afgdhjb\" (UID: \"89ea6c23-db5b-43b2-aeaa-71b9645bfbac\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afgdhjb" Jul 23 19:25:40.882870 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:25:40.882782 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wlbqs\" (UniqueName: \"kubernetes.io/projected/89ea6c23-db5b-43b2-aeaa-71b9645bfbac-kube-api-access-wlbqs\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afgdhjb\" (UID: \"89ea6c23-db5b-43b2-aeaa-71b9645bfbac\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afgdhjb" Jul 23 19:25:40.882870 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:25:40.882806 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/89ea6c23-db5b-43b2-aeaa-71b9645bfbac-bundle\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afgdhjb\" (UID: \"89ea6c23-db5b-43b2-aeaa-71b9645bfbac\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afgdhjb" Jul 23 19:25:40.983430 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:25:40.983409 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/89ea6c23-db5b-43b2-aeaa-71b9645bfbac-util\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afgdhjb\" (UID: \"89ea6c23-db5b-43b2-aeaa-71b9645bfbac\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afgdhjb" Jul 23 19:25:40.983519 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:25:40.983439 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wlbqs\" (UniqueName: \"kubernetes.io/projected/89ea6c23-db5b-43b2-aeaa-71b9645bfbac-kube-api-access-wlbqs\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afgdhjb\" (UID: \"89ea6c23-db5b-43b2-aeaa-71b9645bfbac\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afgdhjb" Jul 23 19:25:40.983519 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:25:40.983461 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/89ea6c23-db5b-43b2-aeaa-71b9645bfbac-bundle\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afgdhjb\" (UID: \"89ea6c23-db5b-43b2-aeaa-71b9645bfbac\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afgdhjb" Jul 23 19:25:40.983748 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:25:40.983730 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/89ea6c23-db5b-43b2-aeaa-71b9645bfbac-util\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afgdhjb\" (UID: \"89ea6c23-db5b-43b2-aeaa-71b9645bfbac\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afgdhjb" Jul 23 19:25:40.983748 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:25:40.983743 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/89ea6c23-db5b-43b2-aeaa-71b9645bfbac-bundle\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afgdhjb\" (UID: \"89ea6c23-db5b-43b2-aeaa-71b9645bfbac\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afgdhjb" Jul 23 19:25:40.992521 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:25:40.992497 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-wlbqs\" (UniqueName: \"kubernetes.io/projected/89ea6c23-db5b-43b2-aeaa-71b9645bfbac-kube-api-access-wlbqs\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afgdhjb\" (UID: \"89ea6c23-db5b-43b2-aeaa-71b9645bfbac\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afgdhjb" Jul 23 19:25:41.082586 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:25:41.082539 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afgdhjb" Jul 23 19:25:41.208493 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:25:41.208473 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afgdhjb"] Jul 23 19:25:41.210446 ip-10-0-129-2 kubenswrapper[2564]: W0723 19:25:41.210423 2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod89ea6c23_db5b_43b2_aeaa_71b9645bfbac.slice/crio-b3c54ea8dbc658742e84fe0b21aa44c3557b6122ac92c1c812b5c338e4bf3c86 WatchSource:0}: Error finding container b3c54ea8dbc658742e84fe0b21aa44c3557b6122ac92c1c812b5c338e4bf3c86: Status 404 returned error can't find the container with id b3c54ea8dbc658742e84fe0b21aa44c3557b6122ac92c1c812b5c338e4bf3c86 Jul 23 19:25:42.186645 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:25:42.186618 2564 generic.go:356] "Generic (PLEG): container finished" podID="89ea6c23-db5b-43b2-aeaa-71b9645bfbac" containerID="ba6d27657bb215befe2544ad7142e1d1c2e32c3592abf8f67fc38eb5ffac48e8" exitCode=0 Jul 23 19:25:42.186971 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:25:42.186650 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afgdhjb" event={"ID":"89ea6c23-db5b-43b2-aeaa-71b9645bfbac","Type":"ContainerDied","Data":"ba6d27657bb215befe2544ad7142e1d1c2e32c3592abf8f67fc38eb5ffac48e8"} Jul 23 19:25:42.186971 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:25:42.186683 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afgdhjb" event={"ID":"89ea6c23-db5b-43b2-aeaa-71b9645bfbac","Type":"ContainerStarted","Data":"b3c54ea8dbc658742e84fe0b21aa44c3557b6122ac92c1c812b5c338e4bf3c86"} Jul 23 19:25:43.192838 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:25:43.192814 2564 generic.go:356] "Generic (PLEG): container finished" podID="89ea6c23-db5b-43b2-aeaa-71b9645bfbac" containerID="fe67f1eb407b9cb4dc055a185c5be3c23d7ff922997bd0f749caa26ad546e063" exitCode=0 Jul 23 19:25:43.193146 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:25:43.192860 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afgdhjb" event={"ID":"89ea6c23-db5b-43b2-aeaa-71b9645bfbac","Type":"ContainerDied","Data":"fe67f1eb407b9cb4dc055a185c5be3c23d7ff922997bd0f749caa26ad546e063"} Jul 23 19:25:44.198974 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:25:44.198941 2564 generic.go:356] "Generic (PLEG): container finished" podID="89ea6c23-db5b-43b2-aeaa-71b9645bfbac" containerID="5adaef497d7e680e9799aaa9cef1dae0bea20a2d93e8c956088f3404b3408632" exitCode=0 Jul 23 19:25:44.199303 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:25:44.199023 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afgdhjb" event={"ID":"89ea6c23-db5b-43b2-aeaa-71b9645bfbac","Type":"ContainerDied","Data":"5adaef497d7e680e9799aaa9cef1dae0bea20a2d93e8c956088f3404b3408632"} Jul 23 19:25:45.318054 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:25:45.318034 2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afgdhjb" Jul 23 19:25:45.416205 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:25:45.416184 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-wlbqs\" (UniqueName: \"kubernetes.io/projected/89ea6c23-db5b-43b2-aeaa-71b9645bfbac-kube-api-access-wlbqs\") pod \"89ea6c23-db5b-43b2-aeaa-71b9645bfbac\" (UID: \"89ea6c23-db5b-43b2-aeaa-71b9645bfbac\") " Jul 23 19:25:45.416308 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:25:45.416229 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/89ea6c23-db5b-43b2-aeaa-71b9645bfbac-util\") pod \"89ea6c23-db5b-43b2-aeaa-71b9645bfbac\" (UID: \"89ea6c23-db5b-43b2-aeaa-71b9645bfbac\") " Jul 23 19:25:45.416308 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:25:45.416245 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/89ea6c23-db5b-43b2-aeaa-71b9645bfbac-bundle\") pod \"89ea6c23-db5b-43b2-aeaa-71b9645bfbac\" (UID: \"89ea6c23-db5b-43b2-aeaa-71b9645bfbac\") " Jul 23 19:25:45.416894 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:25:45.416868 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/89ea6c23-db5b-43b2-aeaa-71b9645bfbac-bundle" (OuterVolumeSpecName: "bundle") pod "89ea6c23-db5b-43b2-aeaa-71b9645bfbac" (UID: "89ea6c23-db5b-43b2-aeaa-71b9645bfbac"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 23 19:25:45.418254 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:25:45.418230 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/89ea6c23-db5b-43b2-aeaa-71b9645bfbac-kube-api-access-wlbqs" (OuterVolumeSpecName: "kube-api-access-wlbqs") pod "89ea6c23-db5b-43b2-aeaa-71b9645bfbac" (UID: "89ea6c23-db5b-43b2-aeaa-71b9645bfbac"). InnerVolumeSpecName "kube-api-access-wlbqs". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 23 19:25:45.421421 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:25:45.421393 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/89ea6c23-db5b-43b2-aeaa-71b9645bfbac-util" (OuterVolumeSpecName: "util") pod "89ea6c23-db5b-43b2-aeaa-71b9645bfbac" (UID: "89ea6c23-db5b-43b2-aeaa-71b9645bfbac"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 23 19:25:45.517636 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:25:45.517586 2564 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wlbqs\" (UniqueName: \"kubernetes.io/projected/89ea6c23-db5b-43b2-aeaa-71b9645bfbac-kube-api-access-wlbqs\") on node \"ip-10-0-129-2.ec2.internal\" DevicePath \"\"" Jul 23 19:25:45.517636 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:25:45.517606 2564 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/89ea6c23-db5b-43b2-aeaa-71b9645bfbac-util\") on node \"ip-10-0-129-2.ec2.internal\" DevicePath \"\"" Jul 23 19:25:45.517636 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:25:45.517615 2564 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/89ea6c23-db5b-43b2-aeaa-71b9645bfbac-bundle\") on node \"ip-10-0-129-2.ec2.internal\" DevicePath \"\"" Jul 23 19:25:46.209095 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:25:46.209077 2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afgdhjb" Jul 23 19:25:46.209095 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:25:46.209090 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afgdhjb" event={"ID":"89ea6c23-db5b-43b2-aeaa-71b9645bfbac","Type":"ContainerDied","Data":"b3c54ea8dbc658742e84fe0b21aa44c3557b6122ac92c1c812b5c338e4bf3c86"} Jul 23 19:25:46.209223 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:25:46.209124 2564 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="b3c54ea8dbc658742e84fe0b21aa44c3557b6122ac92c1c812b5c338e4bf3c86" Jul 23 19:25:52.718204 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:25:52.718172 2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-559f567dc-sc2mg"] Jul 23 19:25:52.718652 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:25:52.718468 2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="89ea6c23-db5b-43b2-aeaa-71b9645bfbac" containerName="extract" Jul 23 19:25:52.718652 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:25:52.718481 2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="89ea6c23-db5b-43b2-aeaa-71b9645bfbac" containerName="extract" Jul 23 19:25:52.718652 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:25:52.718490 2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="89ea6c23-db5b-43b2-aeaa-71b9645bfbac" containerName="util" Jul 23 19:25:52.718652 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:25:52.718496 2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="89ea6c23-db5b-43b2-aeaa-71b9645bfbac" containerName="util" Jul 23 19:25:52.718652 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:25:52.718503 2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="89ea6c23-db5b-43b2-aeaa-71b9645bfbac" containerName="pull" Jul 23 19:25:52.718652 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:25:52.718508 2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="89ea6c23-db5b-43b2-aeaa-71b9645bfbac" containerName="pull" Jul 23 19:25:52.718652 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:25:52.718556 2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="89ea6c23-db5b-43b2-aeaa-71b9645bfbac" containerName="extract" Jul 23 19:25:52.722785 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:25:52.722769 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/opendatahub-operator-controller-manager-559f567dc-sc2mg" Jul 23 19:25:52.727080 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:25:52.727061 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-manager-dockercfg-rtrs9\"" Jul 23 19:25:52.727363 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:25:52.727348 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-manager-service-cert\"" Jul 23 19:25:52.727483 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:25:52.727461 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"kube-root-ca.crt\"" Jul 23 19:25:52.727546 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:25:52.727488 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-webhook-cert\"" Jul 23 19:25:52.727731 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:25:52.727714 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"openshift-service-ca.crt\"" Jul 23 19:25:52.753368 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:25:52.753347 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-559f567dc-sc2mg"] Jul 23 19:25:52.765687 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:25:52.765669 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/3deb6cad-ea6c-4be3-95ad-6b4b5fbad490-tmp\") pod \"opendatahub-operator-controller-manager-559f567dc-sc2mg\" (UID: \"3deb6cad-ea6c-4be3-95ad-6b4b5fbad490\") " pod="opendatahub/opendatahub-operator-controller-manager-559f567dc-sc2mg" Jul 23 19:25:52.765776 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:25:52.765703 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/3deb6cad-ea6c-4be3-95ad-6b4b5fbad490-apiservice-cert\") pod \"opendatahub-operator-controller-manager-559f567dc-sc2mg\" (UID: \"3deb6cad-ea6c-4be3-95ad-6b4b5fbad490\") " pod="opendatahub/opendatahub-operator-controller-manager-559f567dc-sc2mg" Jul 23 19:25:52.765776 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:25:52.765736 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/3deb6cad-ea6c-4be3-95ad-6b4b5fbad490-webhook-cert\") pod \"opendatahub-operator-controller-manager-559f567dc-sc2mg\" (UID: \"3deb6cad-ea6c-4be3-95ad-6b4b5fbad490\") " pod="opendatahub/opendatahub-operator-controller-manager-559f567dc-sc2mg" Jul 23 19:25:52.765848 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:25:52.765811 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/3deb6cad-ea6c-4be3-95ad-6b4b5fbad490-manifests\") pod \"opendatahub-operator-controller-manager-559f567dc-sc2mg\" (UID: \"3deb6cad-ea6c-4be3-95ad-6b4b5fbad490\") " pod="opendatahub/opendatahub-operator-controller-manager-559f567dc-sc2mg" Jul 23 19:25:52.765848 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:25:52.765838 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nmgrk\" (UniqueName: \"kubernetes.io/projected/3deb6cad-ea6c-4be3-95ad-6b4b5fbad490-kube-api-access-nmgrk\") pod \"opendatahub-operator-controller-manager-559f567dc-sc2mg\" (UID: \"3deb6cad-ea6c-4be3-95ad-6b4b5fbad490\") " pod="opendatahub/opendatahub-operator-controller-manager-559f567dc-sc2mg" Jul 23 19:25:52.866702 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:25:52.866679 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/3deb6cad-ea6c-4be3-95ad-6b4b5fbad490-tmp\") pod \"opendatahub-operator-controller-manager-559f567dc-sc2mg\" (UID: \"3deb6cad-ea6c-4be3-95ad-6b4b5fbad490\") " pod="opendatahub/opendatahub-operator-controller-manager-559f567dc-sc2mg" Jul 23 19:25:52.866787 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:25:52.866716 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/3deb6cad-ea6c-4be3-95ad-6b4b5fbad490-apiservice-cert\") pod \"opendatahub-operator-controller-manager-559f567dc-sc2mg\" (UID: \"3deb6cad-ea6c-4be3-95ad-6b4b5fbad490\") " pod="opendatahub/opendatahub-operator-controller-manager-559f567dc-sc2mg" Jul 23 19:25:52.866787 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:25:52.866746 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/3deb6cad-ea6c-4be3-95ad-6b4b5fbad490-webhook-cert\") pod \"opendatahub-operator-controller-manager-559f567dc-sc2mg\" (UID: \"3deb6cad-ea6c-4be3-95ad-6b4b5fbad490\") " pod="opendatahub/opendatahub-operator-controller-manager-559f567dc-sc2mg" Jul 23 19:25:52.866787 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:25:52.866773 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/3deb6cad-ea6c-4be3-95ad-6b4b5fbad490-manifests\") pod \"opendatahub-operator-controller-manager-559f567dc-sc2mg\" (UID: \"3deb6cad-ea6c-4be3-95ad-6b4b5fbad490\") " pod="opendatahub/opendatahub-operator-controller-manager-559f567dc-sc2mg" Jul 23 19:25:52.866894 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:25:52.866800 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nmgrk\" (UniqueName: \"kubernetes.io/projected/3deb6cad-ea6c-4be3-95ad-6b4b5fbad490-kube-api-access-nmgrk\") pod \"opendatahub-operator-controller-manager-559f567dc-sc2mg\" (UID: \"3deb6cad-ea6c-4be3-95ad-6b4b5fbad490\") " pod="opendatahub/opendatahub-operator-controller-manager-559f567dc-sc2mg" Jul 23 19:25:52.867129 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:25:52.867113 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/3deb6cad-ea6c-4be3-95ad-6b4b5fbad490-manifests\") pod \"opendatahub-operator-controller-manager-559f567dc-sc2mg\" (UID: \"3deb6cad-ea6c-4be3-95ad-6b4b5fbad490\") " pod="opendatahub/opendatahub-operator-controller-manager-559f567dc-sc2mg" Jul 23 19:25:52.868984 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:25:52.868954 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/3deb6cad-ea6c-4be3-95ad-6b4b5fbad490-tmp\") pod \"opendatahub-operator-controller-manager-559f567dc-sc2mg\" (UID: \"3deb6cad-ea6c-4be3-95ad-6b4b5fbad490\") " pod="opendatahub/opendatahub-operator-controller-manager-559f567dc-sc2mg" Jul 23 19:25:52.869092 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:25:52.869076 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/3deb6cad-ea6c-4be3-95ad-6b4b5fbad490-apiservice-cert\") pod \"opendatahub-operator-controller-manager-559f567dc-sc2mg\" (UID: \"3deb6cad-ea6c-4be3-95ad-6b4b5fbad490\") " pod="opendatahub/opendatahub-operator-controller-manager-559f567dc-sc2mg" Jul 23 19:25:52.869146 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:25:52.869093 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/3deb6cad-ea6c-4be3-95ad-6b4b5fbad490-webhook-cert\") pod \"opendatahub-operator-controller-manager-559f567dc-sc2mg\" (UID: \"3deb6cad-ea6c-4be3-95ad-6b4b5fbad490\") " pod="opendatahub/opendatahub-operator-controller-manager-559f567dc-sc2mg" Jul 23 19:25:52.878974 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:25:52.878954 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-nmgrk\" (UniqueName: \"kubernetes.io/projected/3deb6cad-ea6c-4be3-95ad-6b4b5fbad490-kube-api-access-nmgrk\") pod \"opendatahub-operator-controller-manager-559f567dc-sc2mg\" (UID: \"3deb6cad-ea6c-4be3-95ad-6b4b5fbad490\") " pod="opendatahub/opendatahub-operator-controller-manager-559f567dc-sc2mg" Jul 23 19:25:53.032397 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:25:53.032343 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/opendatahub-operator-controller-manager-559f567dc-sc2mg" Jul 23 19:25:53.166799 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:25:53.166772 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-559f567dc-sc2mg"] Jul 23 19:25:53.167808 ip-10-0-129-2 kubenswrapper[2564]: W0723 19:25:53.167778 2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3deb6cad_ea6c_4be3_95ad_6b4b5fbad490.slice/crio-d7067238804ef4e17382135f4779a1289e73ea52678f9893341d6a97c30f1708 WatchSource:0}: Error finding container d7067238804ef4e17382135f4779a1289e73ea52678f9893341d6a97c30f1708: Status 404 returned error can't find the container with id d7067238804ef4e17382135f4779a1289e73ea52678f9893341d6a97c30f1708 Jul 23 19:25:53.237234 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:25:53.237210 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-559f567dc-sc2mg" event={"ID":"3deb6cad-ea6c-4be3-95ad-6b4b5fbad490","Type":"ContainerStarted","Data":"d7067238804ef4e17382135f4779a1289e73ea52678f9893341d6a97c30f1708"} Jul 23 19:25:57.257891 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:25:57.257849 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-559f567dc-sc2mg" event={"ID":"3deb6cad-ea6c-4be3-95ad-6b4b5fbad490","Type":"ContainerStarted","Data":"34b738efc4562b1166cf0c373c045db3ada89fa06c36dfa1e2b06dfb8aa6e39c"} Jul 23 19:25:58.263306 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:25:58.263281 2564 generic.go:356] "Generic (PLEG): container finished" podID="3deb6cad-ea6c-4be3-95ad-6b4b5fbad490" containerID="34b738efc4562b1166cf0c373c045db3ada89fa06c36dfa1e2b06dfb8aa6e39c" exitCode=0 Jul 23 19:25:58.263660 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:25:58.263364 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-559f567dc-sc2mg" event={"ID":"3deb6cad-ea6c-4be3-95ad-6b4b5fbad490","Type":"ContainerDied","Data":"34b738efc4562b1166cf0c373c045db3ada89fa06c36dfa1e2b06dfb8aa6e39c"} Jul 23 19:25:59.268936 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:25:59.268903 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-559f567dc-sc2mg" event={"ID":"3deb6cad-ea6c-4be3-95ad-6b4b5fbad490","Type":"ContainerStarted","Data":"2adb725c2c28a14ce1591a1ad9fa42b80427edde3637bd9ae45639b0569ce632"} Jul 23 19:25:59.269298 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:25:59.269016 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/opendatahub-operator-controller-manager-559f567dc-sc2mg" Jul 23 19:25:59.292157 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:25:59.292118 2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/opendatahub-operator-controller-manager-559f567dc-sc2mg" podStartSLOduration=2.952135064 podStartE2EDuration="7.292105746s" totalImagesPullingTime="3.425646517s" totalInitContainerRuntime="914.324165ms" isStatefulPod=false podCreationTimestamp="2026-07-23 19:25:52 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-23 19:25:59.290072215 +0000 UTC m=+385.141848480" watchObservedRunningTime="2026-07-23 19:25:59.292105746 +0000 UTC m=+385.143882061" Jul 23 19:26:10.274395 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:26:10.274366 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/opendatahub-operator-controller-manager-559f567dc-sc2mg" Jul 23 19:26:12.795599 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:26:12.795562 2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/odh-model-controller-7998c67dc4-52fhl"] Jul 23 19:26:12.804374 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:26:12.804347 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/odh-model-controller-7998c67dc4-52fhl" Jul 23 19:26:12.808935 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:26:12.808917 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"odh-model-controller-dockercfg-z6sdz\"" Jul 23 19:26:12.809084 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:26:12.809057 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"odh-model-controller-webhook-cert\"" Jul 23 19:26:12.830367 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:26:12.830346 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/odh-model-controller-7998c67dc4-52fhl"] Jul 23 19:26:12.910872 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:26:12.910844 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ktsq5\" (UniqueName: \"kubernetes.io/projected/a88408a9-0854-4017-8248-4fa9c183712d-kube-api-access-ktsq5\") pod \"odh-model-controller-7998c67dc4-52fhl\" (UID: \"a88408a9-0854-4017-8248-4fa9c183712d\") " pod="opendatahub/odh-model-controller-7998c67dc4-52fhl" Jul 23 19:26:12.911018 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:26:12.910953 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/a88408a9-0854-4017-8248-4fa9c183712d-cert\") pod \"odh-model-controller-7998c67dc4-52fhl\" (UID: \"a88408a9-0854-4017-8248-4fa9c183712d\") " pod="opendatahub/odh-model-controller-7998c67dc4-52fhl" Jul 23 19:26:13.012274 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:26:13.012250 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/a88408a9-0854-4017-8248-4fa9c183712d-cert\") pod \"odh-model-controller-7998c67dc4-52fhl\" (UID: \"a88408a9-0854-4017-8248-4fa9c183712d\") " pod="opendatahub/odh-model-controller-7998c67dc4-52fhl" Jul 23 19:26:13.012398 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:26:13.012292 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ktsq5\" (UniqueName: \"kubernetes.io/projected/a88408a9-0854-4017-8248-4fa9c183712d-kube-api-access-ktsq5\") pod \"odh-model-controller-7998c67dc4-52fhl\" (UID: \"a88408a9-0854-4017-8248-4fa9c183712d\") " pod="opendatahub/odh-model-controller-7998c67dc4-52fhl" Jul 23 19:26:13.012439 ip-10-0-129-2 kubenswrapper[2564]: E0723 19:26:13.012420 2564 secret.go:189] Couldn't get secret opendatahub/odh-model-controller-webhook-cert: secret "odh-model-controller-webhook-cert" not found Jul 23 19:26:13.012497 ip-10-0-129-2 kubenswrapper[2564]: E0723 19:26:13.012487 2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a88408a9-0854-4017-8248-4fa9c183712d-cert podName:a88408a9-0854-4017-8248-4fa9c183712d nodeName:}" failed. No retries permitted until 2026-07-23 19:26:13.512466649 +0000 UTC m=+399.364242881 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "cert" (UniqueName: "kubernetes.io/secret/a88408a9-0854-4017-8248-4fa9c183712d-cert") pod "odh-model-controller-7998c67dc4-52fhl" (UID: "a88408a9-0854-4017-8248-4fa9c183712d") : secret "odh-model-controller-webhook-cert" not found Jul 23 19:26:13.028225 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:26:13.028204 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ktsq5\" (UniqueName: \"kubernetes.io/projected/a88408a9-0854-4017-8248-4fa9c183712d-kube-api-access-ktsq5\") pod \"odh-model-controller-7998c67dc4-52fhl\" (UID: \"a88408a9-0854-4017-8248-4fa9c183712d\") " pod="opendatahub/odh-model-controller-7998c67dc4-52fhl" Jul 23 19:26:13.516552 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:26:13.516519 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/a88408a9-0854-4017-8248-4fa9c183712d-cert\") pod \"odh-model-controller-7998c67dc4-52fhl\" (UID: \"a88408a9-0854-4017-8248-4fa9c183712d\") " pod="opendatahub/odh-model-controller-7998c67dc4-52fhl" Jul 23 19:26:13.518721 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:26:13.518703 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/a88408a9-0854-4017-8248-4fa9c183712d-cert\") pod \"odh-model-controller-7998c67dc4-52fhl\" (UID: \"a88408a9-0854-4017-8248-4fa9c183712d\") " pod="opendatahub/odh-model-controller-7998c67dc4-52fhl" Jul 23 19:26:13.713832 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:26:13.713809 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/odh-model-controller-7998c67dc4-52fhl" Jul 23 19:26:13.813518 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:26:13.813487 2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/model-serving-api-54d89bd5fc-bhjvm"] Jul 23 19:26:13.818614 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:26:13.818592 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/model-serving-api-54d89bd5fc-bhjvm" Jul 23 19:26:13.822847 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:26:13.822826 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"model-serving-api-tls\"" Jul 23 19:26:13.836016 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:26:13.835993 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/model-serving-api-54d89bd5fc-bhjvm"] Jul 23 19:26:13.855257 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:26:13.855240 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/odh-model-controller-7998c67dc4-52fhl"] Jul 23 19:26:13.857708 ip-10-0-129-2 kubenswrapper[2564]: W0723 19:26:13.857685 2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda88408a9_0854_4017_8248_4fa9c183712d.slice/crio-6388fa39f5aff9b03e88d8c4b8feacd12c88fd7f2890d97c858020f4037f4dbb WatchSource:0}: Error finding container 6388fa39f5aff9b03e88d8c4b8feacd12c88fd7f2890d97c858020f4037f4dbb: Status 404 returned error can't find the container with id 6388fa39f5aff9b03e88d8c4b8feacd12c88fd7f2890d97c858020f4037f4dbb Jul 23 19:26:13.919213 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:26:13.919194 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/946f6fb8-32c2-406c-819e-8bf5b690971b-tls-certs\") pod \"model-serving-api-54d89bd5fc-bhjvm\" (UID: \"946f6fb8-32c2-406c-819e-8bf5b690971b\") " pod="opendatahub/model-serving-api-54d89bd5fc-bhjvm" Jul 23 19:26:13.919316 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:26:13.919252 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pjhnk\" (UniqueName: \"kubernetes.io/projected/946f6fb8-32c2-406c-819e-8bf5b690971b-kube-api-access-pjhnk\") pod \"model-serving-api-54d89bd5fc-bhjvm\" (UID: \"946f6fb8-32c2-406c-819e-8bf5b690971b\") " pod="opendatahub/model-serving-api-54d89bd5fc-bhjvm" Jul 23 19:26:14.020558 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:26:14.020536 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pjhnk\" (UniqueName: \"kubernetes.io/projected/946f6fb8-32c2-406c-819e-8bf5b690971b-kube-api-access-pjhnk\") pod \"model-serving-api-54d89bd5fc-bhjvm\" (UID: \"946f6fb8-32c2-406c-819e-8bf5b690971b\") " pod="opendatahub/model-serving-api-54d89bd5fc-bhjvm" Jul 23 19:26:14.020650 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:26:14.020594 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/946f6fb8-32c2-406c-819e-8bf5b690971b-tls-certs\") pod \"model-serving-api-54d89bd5fc-bhjvm\" (UID: \"946f6fb8-32c2-406c-819e-8bf5b690971b\") " pod="opendatahub/model-serving-api-54d89bd5fc-bhjvm" Jul 23 19:26:14.020738 ip-10-0-129-2 kubenswrapper[2564]: E0723 19:26:14.020724 2564 secret.go:189] Couldn't get secret opendatahub/model-serving-api-tls: secret "model-serving-api-tls" not found Jul 23 19:26:14.020796 ip-10-0-129-2 kubenswrapper[2564]: E0723 19:26:14.020779 2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/946f6fb8-32c2-406c-819e-8bf5b690971b-tls-certs podName:946f6fb8-32c2-406c-819e-8bf5b690971b nodeName:}" failed. No retries permitted until 2026-07-23 19:26:14.520759756 +0000 UTC m=+400.372535997 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "tls-certs" (UniqueName: "kubernetes.io/secret/946f6fb8-32c2-406c-819e-8bf5b690971b-tls-certs") pod "model-serving-api-54d89bd5fc-bhjvm" (UID: "946f6fb8-32c2-406c-819e-8bf5b690971b") : secret "model-serving-api-tls" not found Jul 23 19:26:14.031839 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:26:14.031816 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-pjhnk\" (UniqueName: \"kubernetes.io/projected/946f6fb8-32c2-406c-819e-8bf5b690971b-kube-api-access-pjhnk\") pod \"model-serving-api-54d89bd5fc-bhjvm\" (UID: \"946f6fb8-32c2-406c-819e-8bf5b690971b\") " pod="opendatahub/model-serving-api-54d89bd5fc-bhjvm" Jul 23 19:26:14.329734 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:26:14.329709 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/odh-model-controller-7998c67dc4-52fhl" event={"ID":"a88408a9-0854-4017-8248-4fa9c183712d","Type":"ContainerStarted","Data":"6388fa39f5aff9b03e88d8c4b8feacd12c88fd7f2890d97c858020f4037f4dbb"} Jul 23 19:26:14.524200 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:26:14.524181 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/946f6fb8-32c2-406c-819e-8bf5b690971b-tls-certs\") pod \"model-serving-api-54d89bd5fc-bhjvm\" (UID: \"946f6fb8-32c2-406c-819e-8bf5b690971b\") " pod="opendatahub/model-serving-api-54d89bd5fc-bhjvm" Jul 23 19:26:14.526971 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:26:14.526952 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/946f6fb8-32c2-406c-819e-8bf5b690971b-tls-certs\") pod \"model-serving-api-54d89bd5fc-bhjvm\" (UID: \"946f6fb8-32c2-406c-819e-8bf5b690971b\") " pod="opendatahub/model-serving-api-54d89bd5fc-bhjvm" Jul 23 19:26:14.730119 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:26:14.730090 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/model-serving-api-54d89bd5fc-bhjvm" Jul 23 19:26:14.889479 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:26:14.888539 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/model-serving-api-54d89bd5fc-bhjvm"] Jul 23 19:26:14.890690 ip-10-0-129-2 kubenswrapper[2564]: W0723 19:26:14.890529 2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod946f6fb8_32c2_406c_819e_8bf5b690971b.slice/crio-e1e870df4df1af8718685125e7841ef88d57b80e5284634e6958f40c67e66155 WatchSource:0}: Error finding container e1e870df4df1af8718685125e7841ef88d57b80e5284634e6958f40c67e66155: Status 404 returned error can't find the container with id e1e870df4df1af8718685125e7841ef88d57b80e5284634e6958f40c67e66155 Jul 23 19:26:15.335043 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:26:15.335013 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/model-serving-api-54d89bd5fc-bhjvm" event={"ID":"946f6fb8-32c2-406c-819e-8bf5b690971b","Type":"ContainerStarted","Data":"e1e870df4df1af8718685125e7841ef88d57b80e5284634e6958f40c67e66155"} Jul 23 19:26:17.346271 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:26:17.346228 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/odh-model-controller-7998c67dc4-52fhl" event={"ID":"a88408a9-0854-4017-8248-4fa9c183712d","Type":"ContainerStarted","Data":"d6313a4df58ff8d863e261ad63ef188e17e12ac391295859ed040333a5bc0f7a"} Jul 23 19:26:17.346722 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:26:17.346700 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/odh-model-controller-7998c67dc4-52fhl" Jul 23 19:26:17.365430 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:26:17.365383 2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/odh-model-controller-7998c67dc4-52fhl" podStartSLOduration=2.672759662 podStartE2EDuration="5.365370138s" totalImagesPullingTime="2.692610476s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-23 19:26:12 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-23 19:26:17.36272037 +0000 UTC m=+403.214496615" watchObservedRunningTime="2026-07-23 19:26:17.365370138 +0000 UTC m=+403.217146393" Jul 23 19:26:18.352208 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:26:18.352172 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/model-serving-api-54d89bd5fc-bhjvm" event={"ID":"946f6fb8-32c2-406c-819e-8bf5b690971b","Type":"ContainerStarted","Data":"b8719b36ce034c031ea927f7d6c12b98745da9f8370ff295991efbc24feab287"} Jul 23 19:26:18.352560 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:26:18.352382 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/model-serving-api-54d89bd5fc-bhjvm" Jul 23 19:26:18.372367 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:26:18.372306 2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/model-serving-api-54d89bd5fc-bhjvm" podStartSLOduration=2.4678244080000002 podStartE2EDuration="5.372292382s" totalImagesPullingTime="2.904467974s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-23 19:26:13 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-23 19:26:18.370482464 +0000 UTC m=+404.222258703" watchObservedRunningTime="2026-07-23 19:26:18.372292382 +0000 UTC m=+404.224068627" Jul 23 19:26:22.792415 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:26:22.792386 2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/kserve-controller-manager-d6f4b65b8-d6kk8"] Jul 23 19:26:22.796037 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:26:22.796019 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/kserve-controller-manager-d6f4b65b8-d6kk8" Jul 23 19:26:22.800983 ip-10-0-129-2 kubenswrapper[2564]: E0723 19:26:22.800955 2564 status_manager.go:1018] "Failed to get status for pod" err="pods \"kserve-controller-manager-d6f4b65b8-d6kk8\" is forbidden: User \"system:node:ip-10-0-129-2.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-129-2.ec2.internal' and this object" podUID="f6e838ac-b62f-45ee-8a24-937c0ad19b09" pod="opendatahub/kserve-controller-manager-d6f4b65b8-d6kk8" Jul 23 19:26:22.800983 ip-10-0-129-2 kubenswrapper[2564]: E0723 19:26:22.800964 2564 reflector.go:205] "Failed to watch" err="failed to list *v1.Secret: secrets \"kserve-controller-manager-dockercfg-2gv4z\" is forbidden: User \"system:node:ip-10-0-129-2.ec2.internal\" cannot list resource \"secrets\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-129-2.ec2.internal' and this object" logger="UnhandledError" reflector="object-\"opendatahub\"/\"kserve-controller-manager-dockercfg-2gv4z\"" type="*v1.Secret" Jul 23 19:26:22.800983 ip-10-0-129-2 kubenswrapper[2564]: E0723 19:26:22.800964 2564 reflector.go:205] "Failed to watch" err="failed to list *v1.Secret: secrets \"kserve-webhook-server-cert\" is forbidden: User \"system:node:ip-10-0-129-2.ec2.internal\" cannot list resource \"secrets\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-129-2.ec2.internal' and this object" logger="UnhandledError" reflector="object-\"opendatahub\"/\"kserve-webhook-server-cert\"" type="*v1.Secret" Jul 23 19:26:22.807004 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:26:22.806979 2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/llmisvc-controller-manager-777c6bd7d5-7qz4p"] Jul 23 19:26:22.810107 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:26:22.810092 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-7qz4p" Jul 23 19:26:22.814872 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:26:22.814850 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"llmisvc-controller-manager-dockercfg-vllf7\"" Jul 23 19:26:22.815207 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:26:22.815195 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"llmisvc-webhook-server-cert\"" Jul 23 19:26:22.822175 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:26:22.822152 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/kserve-controller-manager-d6f4b65b8-d6kk8"] Jul 23 19:26:22.863473 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:26:22.863449 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/llmisvc-controller-manager-777c6bd7d5-7qz4p"] Jul 23 19:26:22.892670 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:26:22.892647 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-m4zcg\" (UniqueName: \"kubernetes.io/projected/bddb241b-f968-4140-86c1-9aeddc995590-kube-api-access-m4zcg\") pod \"llmisvc-controller-manager-777c6bd7d5-7qz4p\" (UID: \"bddb241b-f968-4140-86c1-9aeddc995590\") " pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-7qz4p" Jul 23 19:26:22.892779 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:26:22.892720 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/bddb241b-f968-4140-86c1-9aeddc995590-cert\") pod \"llmisvc-controller-manager-777c6bd7d5-7qz4p\" (UID: \"bddb241b-f968-4140-86c1-9aeddc995590\") " pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-7qz4p" Jul 23 19:26:22.892779 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:26:22.892756 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/f6e838ac-b62f-45ee-8a24-937c0ad19b09-cert\") pod \"kserve-controller-manager-d6f4b65b8-d6kk8\" (UID: \"f6e838ac-b62f-45ee-8a24-937c0ad19b09\") " pod="opendatahub/kserve-controller-manager-d6f4b65b8-d6kk8" Jul 23 19:26:22.892889 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:26:22.892810 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lfvb8\" (UniqueName: \"kubernetes.io/projected/f6e838ac-b62f-45ee-8a24-937c0ad19b09-kube-api-access-lfvb8\") pod \"kserve-controller-manager-d6f4b65b8-d6kk8\" (UID: \"f6e838ac-b62f-45ee-8a24-937c0ad19b09\") " pod="opendatahub/kserve-controller-manager-d6f4b65b8-d6kk8" Jul 23 19:26:22.993661 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:26:22.993625 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lfvb8\" (UniqueName: \"kubernetes.io/projected/f6e838ac-b62f-45ee-8a24-937c0ad19b09-kube-api-access-lfvb8\") pod \"kserve-controller-manager-d6f4b65b8-d6kk8\" (UID: \"f6e838ac-b62f-45ee-8a24-937c0ad19b09\") " pod="opendatahub/kserve-controller-manager-d6f4b65b8-d6kk8" Jul 23 19:26:22.993796 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:26:22.993700 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m4zcg\" (UniqueName: \"kubernetes.io/projected/bddb241b-f968-4140-86c1-9aeddc995590-kube-api-access-m4zcg\") pod \"llmisvc-controller-manager-777c6bd7d5-7qz4p\" (UID: \"bddb241b-f968-4140-86c1-9aeddc995590\") " pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-7qz4p" Jul 23 19:26:22.993796 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:26:22.993776 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/bddb241b-f968-4140-86c1-9aeddc995590-cert\") pod \"llmisvc-controller-manager-777c6bd7d5-7qz4p\" (UID: \"bddb241b-f968-4140-86c1-9aeddc995590\") " pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-7qz4p" Jul 23 19:26:22.993900 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:26:22.993811 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/f6e838ac-b62f-45ee-8a24-937c0ad19b09-cert\") pod \"kserve-controller-manager-d6f4b65b8-d6kk8\" (UID: \"f6e838ac-b62f-45ee-8a24-937c0ad19b09\") " pod="opendatahub/kserve-controller-manager-d6f4b65b8-d6kk8" Jul 23 19:26:22.996160 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:26:22.996140 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/bddb241b-f968-4140-86c1-9aeddc995590-cert\") pod \"llmisvc-controller-manager-777c6bd7d5-7qz4p\" (UID: \"bddb241b-f968-4140-86c1-9aeddc995590\") " pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-7qz4p" Jul 23 19:26:23.009765 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:26:23.009746 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-lfvb8\" (UniqueName: \"kubernetes.io/projected/f6e838ac-b62f-45ee-8a24-937c0ad19b09-kube-api-access-lfvb8\") pod \"kserve-controller-manager-d6f4b65b8-d6kk8\" (UID: \"f6e838ac-b62f-45ee-8a24-937c0ad19b09\") " pod="opendatahub/kserve-controller-manager-d6f4b65b8-d6kk8" Jul 23 19:26:23.011532 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:26:23.011516 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-m4zcg\" (UniqueName: \"kubernetes.io/projected/bddb241b-f968-4140-86c1-9aeddc995590-kube-api-access-m4zcg\") pod \"llmisvc-controller-manager-777c6bd7d5-7qz4p\" (UID: \"bddb241b-f968-4140-86c1-9aeddc995590\") " pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-7qz4p" Jul 23 19:26:23.112279 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:26:23.112225 2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/kube-auth-proxy-5fdd4658f-lkckb"] Jul 23 19:26:23.115439 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:26:23.115418 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/kube-auth-proxy-5fdd4658f-lkckb" Jul 23 19:26:23.120566 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:26:23.120547 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-7qz4p" Jul 23 19:26:23.121227 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:26:23.121208 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-tls\"" Jul 23 19:26:23.121537 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:26:23.121520 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-dockercfg-mz5fj\"" Jul 23 19:26:23.122451 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:26:23.122432 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-creds\"" Jul 23 19:26:23.150693 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:26:23.143245 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/kube-auth-proxy-5fdd4658f-lkckb"] Jul 23 19:26:23.195130 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:26:23.195099 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e20f2247-2981-4d9b-83d2-eeb094ceb1f6-tls-certs\") pod \"kube-auth-proxy-5fdd4658f-lkckb\" (UID: \"e20f2247-2981-4d9b-83d2-eeb094ceb1f6\") " pod="openshift-ingress/kube-auth-proxy-5fdd4658f-lkckb" Jul 23 19:26:23.195289 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:26:23.195263 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/e20f2247-2981-4d9b-83d2-eeb094ceb1f6-tmp\") pod \"kube-auth-proxy-5fdd4658f-lkckb\" (UID: \"e20f2247-2981-4d9b-83d2-eeb094ceb1f6\") " pod="openshift-ingress/kube-auth-proxy-5fdd4658f-lkckb" Jul 23 19:26:23.195377 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:26:23.195346 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fm9c8\" (UniqueName: \"kubernetes.io/projected/e20f2247-2981-4d9b-83d2-eeb094ceb1f6-kube-api-access-fm9c8\") pod \"kube-auth-proxy-5fdd4658f-lkckb\" (UID: \"e20f2247-2981-4d9b-83d2-eeb094ceb1f6\") " pod="openshift-ingress/kube-auth-proxy-5fdd4658f-lkckb" Jul 23 19:26:23.289442 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:26:23.289415 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/llmisvc-controller-manager-777c6bd7d5-7qz4p"] Jul 23 19:26:23.295950 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:26:23.295925 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/e20f2247-2981-4d9b-83d2-eeb094ceb1f6-tmp\") pod \"kube-auth-proxy-5fdd4658f-lkckb\" (UID: \"e20f2247-2981-4d9b-83d2-eeb094ceb1f6\") " pod="openshift-ingress/kube-auth-proxy-5fdd4658f-lkckb" Jul 23 19:26:23.296084 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:26:23.295966 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fm9c8\" (UniqueName: \"kubernetes.io/projected/e20f2247-2981-4d9b-83d2-eeb094ceb1f6-kube-api-access-fm9c8\") pod \"kube-auth-proxy-5fdd4658f-lkckb\" (UID: \"e20f2247-2981-4d9b-83d2-eeb094ceb1f6\") " pod="openshift-ingress/kube-auth-proxy-5fdd4658f-lkckb" Jul 23 19:26:23.296084 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:26:23.295996 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e20f2247-2981-4d9b-83d2-eeb094ceb1f6-tls-certs\") pod \"kube-auth-proxy-5fdd4658f-lkckb\" (UID: \"e20f2247-2981-4d9b-83d2-eeb094ceb1f6\") " pod="openshift-ingress/kube-auth-proxy-5fdd4658f-lkckb" Jul 23 19:26:23.296192 ip-10-0-129-2 kubenswrapper[2564]: W0723 19:26:23.296125 2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-podbddb241b_f968_4140_86c1_9aeddc995590.slice/crio-f95900299a21d79798103219dfadb2b0bf49e4a9570d68c6cc0d5b1a1c0de17f WatchSource:0}: Error finding container f95900299a21d79798103219dfadb2b0bf49e4a9570d68c6cc0d5b1a1c0de17f: Status 404 returned error can't find the container with id f95900299a21d79798103219dfadb2b0bf49e4a9570d68c6cc0d5b1a1c0de17f Jul 23 19:26:23.298221 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:26:23.298199 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/e20f2247-2981-4d9b-83d2-eeb094ceb1f6-tmp\") pod \"kube-auth-proxy-5fdd4658f-lkckb\" (UID: \"e20f2247-2981-4d9b-83d2-eeb094ceb1f6\") " pod="openshift-ingress/kube-auth-proxy-5fdd4658f-lkckb" Jul 23 19:26:23.298602 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:26:23.298586 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e20f2247-2981-4d9b-83d2-eeb094ceb1f6-tls-certs\") pod \"kube-auth-proxy-5fdd4658f-lkckb\" (UID: \"e20f2247-2981-4d9b-83d2-eeb094ceb1f6\") " pod="openshift-ingress/kube-auth-proxy-5fdd4658f-lkckb" Jul 23 19:26:23.323041 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:26:23.323020 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-fm9c8\" (UniqueName: \"kubernetes.io/projected/e20f2247-2981-4d9b-83d2-eeb094ceb1f6-kube-api-access-fm9c8\") pod \"kube-auth-proxy-5fdd4658f-lkckb\" (UID: \"e20f2247-2981-4d9b-83d2-eeb094ceb1f6\") " pod="openshift-ingress/kube-auth-proxy-5fdd4658f-lkckb" Jul 23 19:26:23.375448 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:26:23.375397 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-7qz4p" event={"ID":"bddb241b-f968-4140-86c1-9aeddc995590","Type":"ContainerStarted","Data":"f95900299a21d79798103219dfadb2b0bf49e4a9570d68c6cc0d5b1a1c0de17f"} Jul 23 19:26:23.424461 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:26:23.424445 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/kube-auth-proxy-5fdd4658f-lkckb" Jul 23 19:26:23.568775 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:26:23.568747 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/kube-auth-proxy-5fdd4658f-lkckb"] Jul 23 19:26:23.569377 ip-10-0-129-2 kubenswrapper[2564]: W0723 19:26:23.569352 2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode20f2247_2981_4d9b_83d2_eeb094ceb1f6.slice/crio-f79919cc9b6c39dc3308c67f33e4d339dc786d2379272a001b17f98af1fe2ce3 WatchSource:0}: Error finding container f79919cc9b6c39dc3308c67f33e4d339dc786d2379272a001b17f98af1fe2ce3: Status 404 returned error can't find the container with id f79919cc9b6c39dc3308c67f33e4d339dc786d2379272a001b17f98af1fe2ce3 Jul 23 19:26:23.994739 ip-10-0-129-2 kubenswrapper[2564]: E0723 19:26:23.994719 2564 secret.go:189] Couldn't get secret opendatahub/kserve-webhook-server-cert: failed to sync secret cache: timed out waiting for the condition Jul 23 19:26:23.995029 ip-10-0-129-2 kubenswrapper[2564]: E0723 19:26:23.994777 2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f6e838ac-b62f-45ee-8a24-937c0ad19b09-cert podName:f6e838ac-b62f-45ee-8a24-937c0ad19b09 nodeName:}" failed. No retries permitted until 2026-07-23 19:26:24.494761528 +0000 UTC m=+410.346537755 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "cert" (UniqueName: "kubernetes.io/secret/f6e838ac-b62f-45ee-8a24-937c0ad19b09-cert") pod "kserve-controller-manager-d6f4b65b8-d6kk8" (UID: "f6e838ac-b62f-45ee-8a24-937c0ad19b09") : failed to sync secret cache: timed out waiting for the condition Jul 23 19:26:24.158159 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:26:24.158132 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"kserve-controller-manager-dockercfg-2gv4z\"" Jul 23 19:26:24.201554 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:26:24.201531 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"kserve-webhook-server-cert\"" Jul 23 19:26:24.382480 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:26:24.382404 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/kube-auth-proxy-5fdd4658f-lkckb" event={"ID":"e20f2247-2981-4d9b-83d2-eeb094ceb1f6","Type":"ContainerStarted","Data":"f79919cc9b6c39dc3308c67f33e4d339dc786d2379272a001b17f98af1fe2ce3"} Jul 23 19:26:24.505962 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:26:24.505925 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/f6e838ac-b62f-45ee-8a24-937c0ad19b09-cert\") pod \"kserve-controller-manager-d6f4b65b8-d6kk8\" (UID: \"f6e838ac-b62f-45ee-8a24-937c0ad19b09\") " pod="opendatahub/kserve-controller-manager-d6f4b65b8-d6kk8" Jul 23 19:26:24.508638 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:26:24.508608 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/f6e838ac-b62f-45ee-8a24-937c0ad19b09-cert\") pod \"kserve-controller-manager-d6f4b65b8-d6kk8\" (UID: \"f6e838ac-b62f-45ee-8a24-937c0ad19b09\") " pod="opendatahub/kserve-controller-manager-d6f4b65b8-d6kk8" Jul 23 19:26:24.607658 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:26:24.607580 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/kserve-controller-manager-d6f4b65b8-d6kk8" Jul 23 19:26:24.935147 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:26:24.935121 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/kserve-controller-manager-d6f4b65b8-d6kk8"] Jul 23 19:26:24.937504 ip-10-0-129-2 kubenswrapper[2564]: W0723 19:26:24.937463 2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf6e838ac_b62f_45ee_8a24_937c0ad19b09.slice/crio-29c91ab2558e299fb97be5a156c2ee54e8a31c636c8908d23e7747f806895e4b WatchSource:0}: Error finding container 29c91ab2558e299fb97be5a156c2ee54e8a31c636c8908d23e7747f806895e4b: Status 404 returned error can't find the container with id 29c91ab2558e299fb97be5a156c2ee54e8a31c636c8908d23e7747f806895e4b Jul 23 19:26:25.389679 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:26:25.389634 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/kserve-controller-manager-d6f4b65b8-d6kk8" event={"ID":"f6e838ac-b62f-45ee-8a24-937c0ad19b09","Type":"ContainerStarted","Data":"29c91ab2558e299fb97be5a156c2ee54e8a31c636c8908d23e7747f806895e4b"} Jul 23 19:26:25.662675 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:26:25.662266 2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/istiod-openshift-gateway-94bb8fbfd-9pnfp"] Jul 23 19:26:25.668653 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:26:25.668627 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-9pnfp" Jul 23 19:26:25.672471 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:26:25.672453 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istiod-tls\"" Jul 23 19:26:25.672645 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:26:25.672625 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istiod-openshift-gateway-dockercfg-gbkkc\"" Jul 23 19:26:25.672710 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:26:25.672656 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-gw-ca-root-cert\"" Jul 23 19:26:25.673048 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:26:25.673033 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istio-kubeconfig\"" Jul 23 19:26:25.680729 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:26:25.680709 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"cacerts\"" Jul 23 19:26:25.697979 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:26:25.697947 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-94bb8fbfd-9pnfp"] Jul 23 19:26:25.817390 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:26:25.817361 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/c983ed8c-9773-4fc8-b029-fb1892487cc6-local-certs\") pod \"istiod-openshift-gateway-94bb8fbfd-9pnfp\" (UID: \"c983ed8c-9773-4fc8-b029-fb1892487cc6\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-9pnfp" Jul 23 19:26:25.817507 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:26:25.817407 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c983ed8c-9773-4fc8-b029-fb1892487cc6-istio-token\") pod \"istiod-openshift-gateway-94bb8fbfd-9pnfp\" (UID: \"c983ed8c-9773-4fc8-b029-fb1892487cc6\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-9pnfp" Jul 23 19:26:25.817507 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:26:25.817443 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/c983ed8c-9773-4fc8-b029-fb1892487cc6-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-94bb8fbfd-9pnfp\" (UID: \"c983ed8c-9773-4fc8-b029-fb1892487cc6\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-9pnfp" Jul 23 19:26:25.817507 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:26:25.817470 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/c983ed8c-9773-4fc8-b029-fb1892487cc6-cacerts\") pod \"istiod-openshift-gateway-94bb8fbfd-9pnfp\" (UID: \"c983ed8c-9773-4fc8-b029-fb1892487cc6\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-9pnfp" Jul 23 19:26:25.817619 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:26:25.817508 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/c983ed8c-9773-4fc8-b029-fb1892487cc6-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-94bb8fbfd-9pnfp\" (UID: \"c983ed8c-9773-4fc8-b029-fb1892487cc6\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-9pnfp" Jul 23 19:26:25.817619 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:26:25.817563 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/c983ed8c-9773-4fc8-b029-fb1892487cc6-istio-kubeconfig\") pod \"istiod-openshift-gateway-94bb8fbfd-9pnfp\" (UID: \"c983ed8c-9773-4fc8-b029-fb1892487cc6\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-9pnfp" Jul 23 19:26:25.817619 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:26:25.817584 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xz8qp\" (UniqueName: \"kubernetes.io/projected/c983ed8c-9773-4fc8-b029-fb1892487cc6-kube-api-access-xz8qp\") pod \"istiod-openshift-gateway-94bb8fbfd-9pnfp\" (UID: \"c983ed8c-9773-4fc8-b029-fb1892487cc6\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-9pnfp" Jul 23 19:26:25.918511 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:26:25.918435 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/c983ed8c-9773-4fc8-b029-fb1892487cc6-istio-kubeconfig\") pod \"istiod-openshift-gateway-94bb8fbfd-9pnfp\" (UID: \"c983ed8c-9773-4fc8-b029-fb1892487cc6\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-9pnfp" Jul 23 19:26:25.918511 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:26:25.918478 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xz8qp\" (UniqueName: \"kubernetes.io/projected/c983ed8c-9773-4fc8-b029-fb1892487cc6-kube-api-access-xz8qp\") pod \"istiod-openshift-gateway-94bb8fbfd-9pnfp\" (UID: \"c983ed8c-9773-4fc8-b029-fb1892487cc6\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-9pnfp" Jul 23 19:26:25.918815 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:26:25.918540 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/c983ed8c-9773-4fc8-b029-fb1892487cc6-local-certs\") pod \"istiod-openshift-gateway-94bb8fbfd-9pnfp\" (UID: \"c983ed8c-9773-4fc8-b029-fb1892487cc6\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-9pnfp" Jul 23 19:26:25.918815 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:26:25.918576 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c983ed8c-9773-4fc8-b029-fb1892487cc6-istio-token\") pod \"istiod-openshift-gateway-94bb8fbfd-9pnfp\" (UID: \"c983ed8c-9773-4fc8-b029-fb1892487cc6\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-9pnfp" Jul 23 19:26:25.918815 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:26:25.918610 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/c983ed8c-9773-4fc8-b029-fb1892487cc6-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-94bb8fbfd-9pnfp\" (UID: \"c983ed8c-9773-4fc8-b029-fb1892487cc6\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-9pnfp" Jul 23 19:26:25.918815 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:26:25.918642 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/c983ed8c-9773-4fc8-b029-fb1892487cc6-cacerts\") pod \"istiod-openshift-gateway-94bb8fbfd-9pnfp\" (UID: \"c983ed8c-9773-4fc8-b029-fb1892487cc6\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-9pnfp" Jul 23 19:26:25.918815 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:26:25.918687 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/c983ed8c-9773-4fc8-b029-fb1892487cc6-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-94bb8fbfd-9pnfp\" (UID: \"c983ed8c-9773-4fc8-b029-fb1892487cc6\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-9pnfp" Jul 23 19:26:25.919297 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:26:25.919272 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/c983ed8c-9773-4fc8-b029-fb1892487cc6-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-94bb8fbfd-9pnfp\" (UID: \"c983ed8c-9773-4fc8-b029-fb1892487cc6\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-9pnfp" Jul 23 19:26:25.921388 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:26:25.921345 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/c983ed8c-9773-4fc8-b029-fb1892487cc6-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-94bb8fbfd-9pnfp\" (UID: \"c983ed8c-9773-4fc8-b029-fb1892487cc6\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-9pnfp" Jul 23 19:26:25.922269 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:26:25.922227 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/c983ed8c-9773-4fc8-b029-fb1892487cc6-local-certs\") pod \"istiod-openshift-gateway-94bb8fbfd-9pnfp\" (UID: \"c983ed8c-9773-4fc8-b029-fb1892487cc6\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-9pnfp" Jul 23 19:26:25.922381 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:26:25.922286 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/c983ed8c-9773-4fc8-b029-fb1892487cc6-cacerts\") pod \"istiod-openshift-gateway-94bb8fbfd-9pnfp\" (UID: \"c983ed8c-9773-4fc8-b029-fb1892487cc6\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-9pnfp" Jul 23 19:26:25.922481 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:26:25.922458 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/c983ed8c-9773-4fc8-b029-fb1892487cc6-istio-kubeconfig\") pod \"istiod-openshift-gateway-94bb8fbfd-9pnfp\" (UID: \"c983ed8c-9773-4fc8-b029-fb1892487cc6\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-9pnfp" Jul 23 19:26:25.928131 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:26:25.928105 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c983ed8c-9773-4fc8-b029-fb1892487cc6-istio-token\") pod \"istiod-openshift-gateway-94bb8fbfd-9pnfp\" (UID: \"c983ed8c-9773-4fc8-b029-fb1892487cc6\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-9pnfp" Jul 23 19:26:25.928426 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:26:25.928402 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xz8qp\" (UniqueName: \"kubernetes.io/projected/c983ed8c-9773-4fc8-b029-fb1892487cc6-kube-api-access-xz8qp\") pod \"istiod-openshift-gateway-94bb8fbfd-9pnfp\" (UID: \"c983ed8c-9773-4fc8-b029-fb1892487cc6\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-9pnfp" Jul 23 19:26:25.979540 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:26:25.979519 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-9pnfp" Jul 23 19:26:28.025472 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:26:28.025449 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-94bb8fbfd-9pnfp"] Jul 23 19:26:28.026892 ip-10-0-129-2 kubenswrapper[2564]: W0723 19:26:28.026870 2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc983ed8c_9773_4fc8_b029_fb1892487cc6.slice/crio-8d00ae3ee517bafe9ab94b42f9f514d7710da8d2af5fdb614727a8e1dee41c5e WatchSource:0}: Error finding container 8d00ae3ee517bafe9ab94b42f9f514d7710da8d2af5fdb614727a8e1dee41c5e: Status 404 returned error can't find the container with id 8d00ae3ee517bafe9ab94b42f9f514d7710da8d2af5fdb614727a8e1dee41c5e Jul 23 19:26:28.353917 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:26:28.353892 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/odh-model-controller-7998c67dc4-52fhl" Jul 23 19:26:28.408482 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:26:28.408451 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-7qz4p" event={"ID":"bddb241b-f968-4140-86c1-9aeddc995590","Type":"ContainerStarted","Data":"bc69e0e740072e307e2f81bf54f9235ea62ece817b4f3701ace10c10e6f28eaf"} Jul 23 19:26:28.408667 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:26:28.408566 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-7qz4p" Jul 23 19:26:28.409881 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:26:28.409853 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-9pnfp" event={"ID":"c983ed8c-9773-4fc8-b029-fb1892487cc6","Type":"ContainerStarted","Data":"8d00ae3ee517bafe9ab94b42f9f514d7710da8d2af5fdb614727a8e1dee41c5e"} Jul 23 19:26:28.411316 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:26:28.411292 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/kube-auth-proxy-5fdd4658f-lkckb" event={"ID":"e20f2247-2981-4d9b-83d2-eeb094ceb1f6","Type":"ContainerStarted","Data":"102e9caeaff5ffff5d3c2c3a9dc490b321984ad7321bd9cf35294a78b10c180f"} Jul 23 19:26:28.412735 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:26:28.412711 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/kserve-controller-manager-d6f4b65b8-d6kk8" event={"ID":"f6e838ac-b62f-45ee-8a24-937c0ad19b09","Type":"ContainerStarted","Data":"a494c624e634197c9af515cb275ed194380d5f884f90b1e5718ce0768e05d806"} Jul 23 19:26:28.412863 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:26:28.412850 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/kserve-controller-manager-d6f4b65b8-d6kk8" Jul 23 19:26:28.425598 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:26:28.425556 2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-7qz4p" podStartSLOduration=1.846770386 podStartE2EDuration="6.425540397s" totalImagesPullingTime="4.578770011s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-23 19:26:22 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-23 19:26:28.423673756 +0000 UTC m=+414.275450002" watchObservedRunningTime="2026-07-23 19:26:28.425540397 +0000 UTC m=+414.277316643" Jul 23 19:26:28.448878 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:26:28.448845 2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/kube-auth-proxy-5fdd4658f-lkckb" podStartSLOduration=1.147758771 podStartE2EDuration="5.44883362s" totalImagesPullingTime="4.301074849s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-23 19:26:23 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-23 19:26:28.447412507 +0000 UTC m=+414.299188752" watchObservedRunningTime="2026-07-23 19:26:28.44883362 +0000 UTC m=+414.300609865" Jul 23 19:26:28.466149 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:26:28.466101 2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/kserve-controller-manager-d6f4b65b8-d6kk8" podStartSLOduration=3.484892147 podStartE2EDuration="6.466086054s" totalImagesPullingTime="2.981193907s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-23 19:26:22 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-23 19:26:28.463043514 +0000 UTC m=+414.314819761" watchObservedRunningTime="2026-07-23 19:26:28.466086054 +0000 UTC m=+414.317862311" Jul 23 19:26:29.361307 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:26:29.361277 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/model-serving-api-54d89bd5fc-bhjvm" Jul 23 19:26:33.607873 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:26:33.607840 2564 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892168Ki","pods":"250"} Jul 23 19:26:33.608121 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:26:33.607908 2564 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892168Ki","pods":"250"} Jul 23 19:26:34.444979 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:26:34.444947 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-9pnfp" event={"ID":"c983ed8c-9773-4fc8-b029-fb1892487cc6","Type":"ContainerStarted","Data":"e40579a3d0c2f8f3dd8b89a77108faac513f391e9168bff583915b85f46c3829"} Jul 23 19:26:34.445181 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:26:34.445164 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-9pnfp" Jul 23 19:26:34.446931 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:26:34.446908 2564 patch_prober.go:28] interesting pod/istiod-openshift-gateway-94bb8fbfd-9pnfp container/discovery namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Jul 23 19:26:34.447049 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:26:34.446970 2564 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-9pnfp" podUID="c983ed8c-9773-4fc8-b029-fb1892487cc6" containerName="discovery" probeResult="failure" output="HTTP probe failed with statuscode: 503" Jul 23 19:26:34.471272 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:26:34.471233 2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-9pnfp" podStartSLOduration=3.892564709 podStartE2EDuration="9.471220529s" totalImagesPullingTime="5.57865582s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-23 19:26:25 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-23 19:26:34.470207417 +0000 UTC m=+420.321983680" watchObservedRunningTime="2026-07-23 19:26:34.471220529 +0000 UTC m=+420.322996775" Jul 23 19:26:35.449446 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:26:35.449423 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-9pnfp" Jul 23 19:26:59.419966 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:26:59.419874 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-7qz4p" Jul 23 19:26:59.422968 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:26:59.422949 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/kserve-controller-manager-d6f4b65b8-d6kk8" Jul 23 19:27:33.176023 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:27:33.175992 2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e02zdvq"] Jul 23 19:27:33.179451 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:27:33.179434 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e02zdvq" Jul 23 19:27:33.182205 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:27:33.182185 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"openshift-service-ca.crt\"" Jul 23 19:27:33.182346 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:27:33.182286 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kube-root-ca.crt\"" Jul 23 19:27:33.183054 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:27:33.183034 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"default-dockercfg-mcgx7\"" Jul 23 19:27:33.192637 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:27:33.192611 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e02zdvq"] Jul 23 19:27:33.241988 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:27:33.241965 2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b75982c7j"] Jul 23 19:27:33.245267 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:27:33.245253 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b75982c7j" Jul 23 19:27:33.253836 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:27:33.253814 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b75982c7j"] Jul 23 19:27:33.293924 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:27:33.293905 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/7326cf84-a21c-4c92-a262-fbff97a47bbd-bundle\") pod \"19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e02zdvq\" (UID: \"7326cf84-a21c-4c92-a262-fbff97a47bbd\") " pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e02zdvq" Jul 23 19:27:33.294020 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:27:33.293928 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/fd160cf4-96c6-410f-a5b3-7dc83bb993c7-util\") pod \"0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b75982c7j\" (UID: \"fd160cf4-96c6-410f-a5b3-7dc83bb993c7\") " pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b75982c7j" Jul 23 19:27:33.294020 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:27:33.293947 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4kpfk\" (UniqueName: \"kubernetes.io/projected/fd160cf4-96c6-410f-a5b3-7dc83bb993c7-kube-api-access-4kpfk\") pod \"0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b75982c7j\" (UID: \"fd160cf4-96c6-410f-a5b3-7dc83bb993c7\") " pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b75982c7j" Jul 23 19:27:33.294093 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:27:33.294057 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-64gpc\" (UniqueName: \"kubernetes.io/projected/7326cf84-a21c-4c92-a262-fbff97a47bbd-kube-api-access-64gpc\") pod \"19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e02zdvq\" (UID: \"7326cf84-a21c-4c92-a262-fbff97a47bbd\") " pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e02zdvq" Jul 23 19:27:33.294093 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:27:33.294084 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/fd160cf4-96c6-410f-a5b3-7dc83bb993c7-bundle\") pod \"0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b75982c7j\" (UID: \"fd160cf4-96c6-410f-a5b3-7dc83bb993c7\") " pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b75982c7j" Jul 23 19:27:33.294154 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:27:33.294114 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/7326cf84-a21c-4c92-a262-fbff97a47bbd-util\") pod \"19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e02zdvq\" (UID: \"7326cf84-a21c-4c92-a262-fbff97a47bbd\") " pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e02zdvq" Jul 23 19:27:33.394788 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:27:33.394768 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-64gpc\" (UniqueName: \"kubernetes.io/projected/7326cf84-a21c-4c92-a262-fbff97a47bbd-kube-api-access-64gpc\") pod \"19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e02zdvq\" (UID: \"7326cf84-a21c-4c92-a262-fbff97a47bbd\") " pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e02zdvq" Jul 23 19:27:33.394869 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:27:33.394793 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/fd160cf4-96c6-410f-a5b3-7dc83bb993c7-bundle\") pod \"0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b75982c7j\" (UID: \"fd160cf4-96c6-410f-a5b3-7dc83bb993c7\") " pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b75982c7j" Jul 23 19:27:33.394869 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:27:33.394817 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/7326cf84-a21c-4c92-a262-fbff97a47bbd-util\") pod \"19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e02zdvq\" (UID: \"7326cf84-a21c-4c92-a262-fbff97a47bbd\") " pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e02zdvq" Jul 23 19:27:33.394869 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:27:33.394851 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/7326cf84-a21c-4c92-a262-fbff97a47bbd-bundle\") pod \"19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e02zdvq\" (UID: \"7326cf84-a21c-4c92-a262-fbff97a47bbd\") " pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e02zdvq" Jul 23 19:27:33.395006 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:27:33.394872 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/fd160cf4-96c6-410f-a5b3-7dc83bb993c7-util\") pod \"0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b75982c7j\" (UID: \"fd160cf4-96c6-410f-a5b3-7dc83bb993c7\") " pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b75982c7j" Jul 23 19:27:33.395006 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:27:33.394899 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4kpfk\" (UniqueName: \"kubernetes.io/projected/fd160cf4-96c6-410f-a5b3-7dc83bb993c7-kube-api-access-4kpfk\") pod \"0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b75982c7j\" (UID: \"fd160cf4-96c6-410f-a5b3-7dc83bb993c7\") " pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b75982c7j" Jul 23 19:27:33.395185 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:27:33.395164 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/7326cf84-a21c-4c92-a262-fbff97a47bbd-util\") pod \"19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e02zdvq\" (UID: \"7326cf84-a21c-4c92-a262-fbff97a47bbd\") " pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e02zdvq" Jul 23 19:27:33.395247 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:27:33.395185 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/fd160cf4-96c6-410f-a5b3-7dc83bb993c7-bundle\") pod \"0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b75982c7j\" (UID: \"fd160cf4-96c6-410f-a5b3-7dc83bb993c7\") " pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b75982c7j" Jul 23 19:27:33.395247 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:27:33.395196 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/fd160cf4-96c6-410f-a5b3-7dc83bb993c7-util\") pod \"0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b75982c7j\" (UID: \"fd160cf4-96c6-410f-a5b3-7dc83bb993c7\") " pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b75982c7j" Jul 23 19:27:33.395247 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:27:33.395221 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/7326cf84-a21c-4c92-a262-fbff97a47bbd-bundle\") pod \"19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e02zdvq\" (UID: \"7326cf84-a21c-4c92-a262-fbff97a47bbd\") " pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e02zdvq" Jul 23 19:27:33.403705 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:27:33.403677 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4kpfk\" (UniqueName: \"kubernetes.io/projected/fd160cf4-96c6-410f-a5b3-7dc83bb993c7-kube-api-access-4kpfk\") pod \"0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b75982c7j\" (UID: \"fd160cf4-96c6-410f-a5b3-7dc83bb993c7\") " pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b75982c7j" Jul 23 19:27:33.403705 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:27:33.403687 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-64gpc\" (UniqueName: \"kubernetes.io/projected/7326cf84-a21c-4c92-a262-fbff97a47bbd-kube-api-access-64gpc\") pod \"19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e02zdvq\" (UID: \"7326cf84-a21c-4c92-a262-fbff97a47bbd\") " pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e02zdvq" Jul 23 19:27:33.488627 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:27:33.488584 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e02zdvq" Jul 23 19:27:33.525691 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:27:33.525666 2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73nmtfx"] Jul 23 19:27:33.530231 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:27:33.530211 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73nmtfx" Jul 23 19:27:33.541969 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:27:33.541946 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73nmtfx"] Jul 23 19:27:33.553845 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:27:33.553801 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b75982c7j" Jul 23 19:27:33.597005 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:27:33.596971 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/d1495985-7d1d-486a-894a-9af5eedb393e-bundle\") pod \"5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73nmtfx\" (UID: \"d1495985-7d1d-486a-894a-9af5eedb393e\") " pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73nmtfx" Jul 23 19:27:33.597151 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:27:33.597037 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/d1495985-7d1d-486a-894a-9af5eedb393e-util\") pod \"5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73nmtfx\" (UID: \"d1495985-7d1d-486a-894a-9af5eedb393e\") " pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73nmtfx" Jul 23 19:27:33.597151 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:27:33.597093 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dp7mw\" (UniqueName: \"kubernetes.io/projected/d1495985-7d1d-486a-894a-9af5eedb393e-kube-api-access-dp7mw\") pod \"5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73nmtfx\" (UID: \"d1495985-7d1d-486a-894a-9af5eedb393e\") " pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73nmtfx" Jul 23 19:27:33.626593 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:27:33.626549 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e02zdvq"] Jul 23 19:27:33.627729 ip-10-0-129-2 kubenswrapper[2564]: W0723 19:27:33.627699 2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7326cf84_a21c_4c92_a262_fbff97a47bbd.slice/crio-f7efb7eb3bbe35b37e7f67f0f6ede12b6aa0ad1953434df984ae65a0af7c3728 WatchSource:0}: Error finding container f7efb7eb3bbe35b37e7f67f0f6ede12b6aa0ad1953434df984ae65a0af7c3728: Status 404 returned error can't find the container with id f7efb7eb3bbe35b37e7f67f0f6ede12b6aa0ad1953434df984ae65a0af7c3728 Jul 23 19:27:33.692013 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:27:33.691992 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b75982c7j"] Jul 23 19:27:33.693752 ip-10-0-129-2 kubenswrapper[2564]: W0723 19:27:33.693731 2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfd160cf4_96c6_410f_a5b3_7dc83bb993c7.slice/crio-f5a86702da05b0ce69a2e8d6d44267c12df94a47ce93d355fb8cea4edf1a4258 WatchSource:0}: Error finding container f5a86702da05b0ce69a2e8d6d44267c12df94a47ce93d355fb8cea4edf1a4258: Status 404 returned error can't find the container with id f5a86702da05b0ce69a2e8d6d44267c12df94a47ce93d355fb8cea4edf1a4258 Jul 23 19:27:33.698109 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:27:33.698093 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/d1495985-7d1d-486a-894a-9af5eedb393e-bundle\") pod \"5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73nmtfx\" (UID: \"d1495985-7d1d-486a-894a-9af5eedb393e\") " pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73nmtfx" Jul 23 19:27:33.698184 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:27:33.698126 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/d1495985-7d1d-486a-894a-9af5eedb393e-util\") pod \"5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73nmtfx\" (UID: \"d1495985-7d1d-486a-894a-9af5eedb393e\") " pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73nmtfx" Jul 23 19:27:33.698184 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:27:33.698155 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dp7mw\" (UniqueName: \"kubernetes.io/projected/d1495985-7d1d-486a-894a-9af5eedb393e-kube-api-access-dp7mw\") pod \"5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73nmtfx\" (UID: \"d1495985-7d1d-486a-894a-9af5eedb393e\") " pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73nmtfx" Jul 23 19:27:33.698490 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:27:33.698472 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/d1495985-7d1d-486a-894a-9af5eedb393e-util\") pod \"5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73nmtfx\" (UID: \"d1495985-7d1d-486a-894a-9af5eedb393e\") " pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73nmtfx" Jul 23 19:27:33.698554 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:27:33.698541 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/d1495985-7d1d-486a-894a-9af5eedb393e-bundle\") pod \"5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73nmtfx\" (UID: \"d1495985-7d1d-486a-894a-9af5eedb393e\") " pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73nmtfx" Jul 23 19:27:33.704276 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:27:33.704252 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e02zdvq" event={"ID":"7326cf84-a21c-4c92-a262-fbff97a47bbd","Type":"ContainerStarted","Data":"2ebaa94cdd05790e72f941b48fa10f0945876dbe7049903d74a523341b3b87e9"} Jul 23 19:27:33.704386 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:27:33.704303 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e02zdvq" event={"ID":"7326cf84-a21c-4c92-a262-fbff97a47bbd","Type":"ContainerStarted","Data":"f7efb7eb3bbe35b37e7f67f0f6ede12b6aa0ad1953434df984ae65a0af7c3728"} Jul 23 19:27:33.705495 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:27:33.705478 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b75982c7j" event={"ID":"fd160cf4-96c6-410f-a5b3-7dc83bb993c7","Type":"ContainerStarted","Data":"f5a86702da05b0ce69a2e8d6d44267c12df94a47ce93d355fb8cea4edf1a4258"} Jul 23 19:27:33.707155 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:27:33.707133 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-dp7mw\" (UniqueName: \"kubernetes.io/projected/d1495985-7d1d-486a-894a-9af5eedb393e-kube-api-access-dp7mw\") pod \"5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73nmtfx\" (UID: \"d1495985-7d1d-486a-894a-9af5eedb393e\") " pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73nmtfx" Jul 23 19:27:33.844206 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:27:33.844186 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73nmtfx" Jul 23 19:27:33.933859 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:27:33.933834 2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1m88qh"] Jul 23 19:27:33.938393 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:27:33.938374 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1m88qh" Jul 23 19:27:33.948641 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:27:33.948620 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1m88qh"] Jul 23 19:27:33.974865 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:27:33.974843 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73nmtfx"] Jul 23 19:27:33.976568 ip-10-0-129-2 kubenswrapper[2564]: W0723 19:27:33.976545 2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd1495985_7d1d_486a_894a_9af5eedb393e.slice/crio-8f66b8b68dfd7144c1e3bd455cee0e350e0f2c5381bab383c408042975cbbd78 WatchSource:0}: Error finding container 8f66b8b68dfd7144c1e3bd455cee0e350e0f2c5381bab383c408042975cbbd78: Status 404 returned error can't find the container with id 8f66b8b68dfd7144c1e3bd455cee0e350e0f2c5381bab383c408042975cbbd78 Jul 23 19:27:34.001127 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:27:34.001108 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/acf936f7-9ae5-4c32-af67-8f92692382a7-bundle\") pod \"9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1m88qh\" (UID: \"acf936f7-9ae5-4c32-af67-8f92692382a7\") " pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1m88qh" Jul 23 19:27:34.001204 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:27:34.001143 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2vfg7\" (UniqueName: \"kubernetes.io/projected/acf936f7-9ae5-4c32-af67-8f92692382a7-kube-api-access-2vfg7\") pod \"9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1m88qh\" (UID: \"acf936f7-9ae5-4c32-af67-8f92692382a7\") " pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1m88qh" Jul 23 19:27:34.001241 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:27:34.001205 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/acf936f7-9ae5-4c32-af67-8f92692382a7-util\") pod \"9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1m88qh\" (UID: \"acf936f7-9ae5-4c32-af67-8f92692382a7\") " pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1m88qh" Jul 23 19:27:34.102581 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:27:34.102560 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/acf936f7-9ae5-4c32-af67-8f92692382a7-util\") pod \"9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1m88qh\" (UID: \"acf936f7-9ae5-4c32-af67-8f92692382a7\") " pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1m88qh" Jul 23 19:27:34.102675 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:27:34.102649 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/acf936f7-9ae5-4c32-af67-8f92692382a7-bundle\") pod \"9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1m88qh\" (UID: \"acf936f7-9ae5-4c32-af67-8f92692382a7\") " pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1m88qh" Jul 23 19:27:34.102714 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:27:34.102689 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2vfg7\" (UniqueName: \"kubernetes.io/projected/acf936f7-9ae5-4c32-af67-8f92692382a7-kube-api-access-2vfg7\") pod \"9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1m88qh\" (UID: \"acf936f7-9ae5-4c32-af67-8f92692382a7\") " pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1m88qh" Jul 23 19:27:34.102955 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:27:34.102935 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/acf936f7-9ae5-4c32-af67-8f92692382a7-bundle\") pod \"9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1m88qh\" (UID: \"acf936f7-9ae5-4c32-af67-8f92692382a7\") " pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1m88qh" Jul 23 19:27:34.102992 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:27:34.102977 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/acf936f7-9ae5-4c32-af67-8f92692382a7-util\") pod \"9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1m88qh\" (UID: \"acf936f7-9ae5-4c32-af67-8f92692382a7\") " pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1m88qh" Jul 23 19:27:34.113251 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:27:34.113234 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2vfg7\" (UniqueName: \"kubernetes.io/projected/acf936f7-9ae5-4c32-af67-8f92692382a7-kube-api-access-2vfg7\") pod \"9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1m88qh\" (UID: \"acf936f7-9ae5-4c32-af67-8f92692382a7\") " pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1m88qh" Jul 23 19:27:34.250523 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:27:34.250477 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1m88qh" Jul 23 19:27:34.376727 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:27:34.376702 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1m88qh"] Jul 23 19:27:34.378108 ip-10-0-129-2 kubenswrapper[2564]: W0723 19:27:34.378088 2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podacf936f7_9ae5_4c32_af67_8f92692382a7.slice/crio-abc40a7cbd0472afc520246e812428e848ad4128c36dd0d53c3fc9faafd91f5a WatchSource:0}: Error finding container abc40a7cbd0472afc520246e812428e848ad4128c36dd0d53c3fc9faafd91f5a: Status 404 returned error can't find the container with id abc40a7cbd0472afc520246e812428e848ad4128c36dd0d53c3fc9faafd91f5a Jul 23 19:27:34.710358 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:27:34.710305 2564 generic.go:356] "Generic (PLEG): container finished" podID="acf936f7-9ae5-4c32-af67-8f92692382a7" containerID="e701602efe1ad29566b6e09970ae7b843d9aeba116b5e6a6a90b637945123646" exitCode=0 Jul 23 19:27:34.710479 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:27:34.710389 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1m88qh" event={"ID":"acf936f7-9ae5-4c32-af67-8f92692382a7","Type":"ContainerDied","Data":"e701602efe1ad29566b6e09970ae7b843d9aeba116b5e6a6a90b637945123646"} Jul 23 19:27:34.710479 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:27:34.710435 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1m88qh" event={"ID":"acf936f7-9ae5-4c32-af67-8f92692382a7","Type":"ContainerStarted","Data":"abc40a7cbd0472afc520246e812428e848ad4128c36dd0d53c3fc9faafd91f5a"} Jul 23 19:27:34.717109 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:27:34.717079 2564 generic.go:356] "Generic (PLEG): container finished" podID="d1495985-7d1d-486a-894a-9af5eedb393e" containerID="a69165fe6c4873bc401ed5a7645957454a3bd3235bc49a36933125ea0da082c9" exitCode=0 Jul 23 19:27:34.717443 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:27:34.717249 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73nmtfx" event={"ID":"d1495985-7d1d-486a-894a-9af5eedb393e","Type":"ContainerDied","Data":"a69165fe6c4873bc401ed5a7645957454a3bd3235bc49a36933125ea0da082c9"} Jul 23 19:27:34.717443 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:27:34.717368 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73nmtfx" event={"ID":"d1495985-7d1d-486a-894a-9af5eedb393e","Type":"ContainerStarted","Data":"8f66b8b68dfd7144c1e3bd455cee0e350e0f2c5381bab383c408042975cbbd78"} Jul 23 19:27:34.719391 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:27:34.719370 2564 generic.go:356] "Generic (PLEG): container finished" podID="fd160cf4-96c6-410f-a5b3-7dc83bb993c7" containerID="c675cb261aa8905fa8a63d2ead33ca5afeb03cdeb770e88763f70ed0a863eb67" exitCode=0 Jul 23 19:27:34.719478 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:27:34.719429 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b75982c7j" event={"ID":"fd160cf4-96c6-410f-a5b3-7dc83bb993c7","Type":"ContainerDied","Data":"c675cb261aa8905fa8a63d2ead33ca5afeb03cdeb770e88763f70ed0a863eb67"} Jul 23 19:27:34.721106 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:27:34.721083 2564 generic.go:356] "Generic (PLEG): container finished" podID="7326cf84-a21c-4c92-a262-fbff97a47bbd" containerID="2ebaa94cdd05790e72f941b48fa10f0945876dbe7049903d74a523341b3b87e9" exitCode=0 Jul 23 19:27:34.721202 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:27:34.721168 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e02zdvq" event={"ID":"7326cf84-a21c-4c92-a262-fbff97a47bbd","Type":"ContainerDied","Data":"2ebaa94cdd05790e72f941b48fa10f0945876dbe7049903d74a523341b3b87e9"} Jul 23 19:27:35.728105 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:27:35.728082 2564 generic.go:356] "Generic (PLEG): container finished" podID="7326cf84-a21c-4c92-a262-fbff97a47bbd" containerID="b784abbdf63f39923e96f7dfca6073c49577b4f314f18da540cb0a33892cfdf5" exitCode=0 Jul 23 19:27:35.728434 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:27:35.728165 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e02zdvq" event={"ID":"7326cf84-a21c-4c92-a262-fbff97a47bbd","Type":"ContainerDied","Data":"b784abbdf63f39923e96f7dfca6073c49577b4f314f18da540cb0a33892cfdf5"} Jul 23 19:27:36.733837 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:27:36.733788 2564 generic.go:356] "Generic (PLEG): container finished" podID="acf936f7-9ae5-4c32-af67-8f92692382a7" containerID="779dbddb2d90bcca94e2e3587d3c6f99f67ee1062a3e1e96c6df959aea215ead" exitCode=0 Jul 23 19:27:36.734173 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:27:36.733863 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1m88qh" event={"ID":"acf936f7-9ae5-4c32-af67-8f92692382a7","Type":"ContainerDied","Data":"779dbddb2d90bcca94e2e3587d3c6f99f67ee1062a3e1e96c6df959aea215ead"} Jul 23 19:27:36.735726 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:27:36.735677 2564 generic.go:356] "Generic (PLEG): container finished" podID="d1495985-7d1d-486a-894a-9af5eedb393e" containerID="b502b45d3761e2a8aa8b7e1ec3415ed1c05ff25e32c03e8680a158fbe240bdfc" exitCode=0 Jul 23 19:27:36.735810 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:27:36.735751 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73nmtfx" event={"ID":"d1495985-7d1d-486a-894a-9af5eedb393e","Type":"ContainerDied","Data":"b502b45d3761e2a8aa8b7e1ec3415ed1c05ff25e32c03e8680a158fbe240bdfc"} Jul 23 19:27:36.737555 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:27:36.737536 2564 generic.go:356] "Generic (PLEG): container finished" podID="fd160cf4-96c6-410f-a5b3-7dc83bb993c7" containerID="17d9c2c59aa87c73b28145c092d33a6c66a157dcb0516bd5e7fee33c1c262758" exitCode=0 Jul 23 19:27:36.737633 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:27:36.737604 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b75982c7j" event={"ID":"fd160cf4-96c6-410f-a5b3-7dc83bb993c7","Type":"ContainerDied","Data":"17d9c2c59aa87c73b28145c092d33a6c66a157dcb0516bd5e7fee33c1c262758"} Jul 23 19:27:36.739842 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:27:36.739795 2564 generic.go:356] "Generic (PLEG): container finished" podID="7326cf84-a21c-4c92-a262-fbff97a47bbd" containerID="5d89b366c3c1a2082704e3fa71ffd6121c6e112652d1e8da72bc0e3222629dc6" exitCode=0 Jul 23 19:27:36.739919 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:27:36.739873 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e02zdvq" event={"ID":"7326cf84-a21c-4c92-a262-fbff97a47bbd","Type":"ContainerDied","Data":"5d89b366c3c1a2082704e3fa71ffd6121c6e112652d1e8da72bc0e3222629dc6"} Jul 23 19:27:37.747458 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:27:37.747403 2564 generic.go:356] "Generic (PLEG): container finished" podID="acf936f7-9ae5-4c32-af67-8f92692382a7" containerID="3270d923f35cbb5b417c65e86ab5d2f381d245bdcdb7391d17a6b71c8b61cefc" exitCode=0 Jul 23 19:27:37.747762 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:27:37.747483 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1m88qh" event={"ID":"acf936f7-9ae5-4c32-af67-8f92692382a7","Type":"ContainerDied","Data":"3270d923f35cbb5b417c65e86ab5d2f381d245bdcdb7391d17a6b71c8b61cefc"} Jul 23 19:27:37.749475 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:27:37.749456 2564 generic.go:356] "Generic (PLEG): container finished" podID="d1495985-7d1d-486a-894a-9af5eedb393e" containerID="204bf816a51ec73e38d430721364842f19ba3c4343f8f80d584b670cfefbb286" exitCode=0 Jul 23 19:27:37.749565 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:27:37.749533 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73nmtfx" event={"ID":"d1495985-7d1d-486a-894a-9af5eedb393e","Type":"ContainerDied","Data":"204bf816a51ec73e38d430721364842f19ba3c4343f8f80d584b670cfefbb286"} Jul 23 19:27:37.751429 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:27:37.751410 2564 generic.go:356] "Generic (PLEG): container finished" podID="fd160cf4-96c6-410f-a5b3-7dc83bb993c7" containerID="983fc39d8780c21eb81e00f948bd0bb38dd02aa282fcf65f075ea76964bb31ca" exitCode=0 Jul 23 19:27:37.751501 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:27:37.751489 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b75982c7j" event={"ID":"fd160cf4-96c6-410f-a5b3-7dc83bb993c7","Type":"ContainerDied","Data":"983fc39d8780c21eb81e00f948bd0bb38dd02aa282fcf65f075ea76964bb31ca"} Jul 23 19:27:37.879288 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:27:37.879265 2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e02zdvq" Jul 23 19:27:37.931954 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:27:37.931929 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/7326cf84-a21c-4c92-a262-fbff97a47bbd-bundle\") pod \"7326cf84-a21c-4c92-a262-fbff97a47bbd\" (UID: \"7326cf84-a21c-4c92-a262-fbff97a47bbd\") " Jul 23 19:27:37.932057 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:27:37.931975 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/7326cf84-a21c-4c92-a262-fbff97a47bbd-util\") pod \"7326cf84-a21c-4c92-a262-fbff97a47bbd\" (UID: \"7326cf84-a21c-4c92-a262-fbff97a47bbd\") " Jul 23 19:27:37.932057 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:27:37.932037 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-64gpc\" (UniqueName: \"kubernetes.io/projected/7326cf84-a21c-4c92-a262-fbff97a47bbd-kube-api-access-64gpc\") pod \"7326cf84-a21c-4c92-a262-fbff97a47bbd\" (UID: \"7326cf84-a21c-4c92-a262-fbff97a47bbd\") " Jul 23 19:27:37.932397 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:27:37.932367 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7326cf84-a21c-4c92-a262-fbff97a47bbd-bundle" (OuterVolumeSpecName: "bundle") pod "7326cf84-a21c-4c92-a262-fbff97a47bbd" (UID: "7326cf84-a21c-4c92-a262-fbff97a47bbd"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 23 19:27:37.934181 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:27:37.934153 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7326cf84-a21c-4c92-a262-fbff97a47bbd-kube-api-access-64gpc" (OuterVolumeSpecName: "kube-api-access-64gpc") pod "7326cf84-a21c-4c92-a262-fbff97a47bbd" (UID: "7326cf84-a21c-4c92-a262-fbff97a47bbd"). InnerVolumeSpecName "kube-api-access-64gpc". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 23 19:27:37.937159 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:27:37.937135 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7326cf84-a21c-4c92-a262-fbff97a47bbd-util" (OuterVolumeSpecName: "util") pod "7326cf84-a21c-4c92-a262-fbff97a47bbd" (UID: "7326cf84-a21c-4c92-a262-fbff97a47bbd"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 23 19:27:38.033160 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:27:38.033113 2564 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-64gpc\" (UniqueName: \"kubernetes.io/projected/7326cf84-a21c-4c92-a262-fbff97a47bbd-kube-api-access-64gpc\") on node \"ip-10-0-129-2.ec2.internal\" DevicePath \"\"" Jul 23 19:27:38.033160 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:27:38.033132 2564 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/7326cf84-a21c-4c92-a262-fbff97a47bbd-bundle\") on node \"ip-10-0-129-2.ec2.internal\" DevicePath \"\"" Jul 23 19:27:38.033160 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:27:38.033140 2564 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/7326cf84-a21c-4c92-a262-fbff97a47bbd-util\") on node \"ip-10-0-129-2.ec2.internal\" DevicePath \"\"" Jul 23 19:27:38.758073 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:27:38.758053 2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e02zdvq" Jul 23 19:27:38.758485 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:27:38.758095 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e02zdvq" event={"ID":"7326cf84-a21c-4c92-a262-fbff97a47bbd","Type":"ContainerDied","Data":"f7efb7eb3bbe35b37e7f67f0f6ede12b6aa0ad1953434df984ae65a0af7c3728"} Jul 23 19:27:38.758485 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:27:38.758146 2564 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="f7efb7eb3bbe35b37e7f67f0f6ede12b6aa0ad1953434df984ae65a0af7c3728" Jul 23 19:27:38.903352 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:27:38.903315 2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1m88qh" Jul 23 19:27:38.938254 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:27:38.938221 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/acf936f7-9ae5-4c32-af67-8f92692382a7-util\") pod \"acf936f7-9ae5-4c32-af67-8f92692382a7\" (UID: \"acf936f7-9ae5-4c32-af67-8f92692382a7\") " Jul 23 19:27:38.938380 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:27:38.938301 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-2vfg7\" (UniqueName: \"kubernetes.io/projected/acf936f7-9ae5-4c32-af67-8f92692382a7-kube-api-access-2vfg7\") pod \"acf936f7-9ae5-4c32-af67-8f92692382a7\" (UID: \"acf936f7-9ae5-4c32-af67-8f92692382a7\") " Jul 23 19:27:38.938455 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:27:38.938424 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/acf936f7-9ae5-4c32-af67-8f92692382a7-bundle\") pod \"acf936f7-9ae5-4c32-af67-8f92692382a7\" (UID: \"acf936f7-9ae5-4c32-af67-8f92692382a7\") " Jul 23 19:27:38.938985 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:27:38.938911 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/acf936f7-9ae5-4c32-af67-8f92692382a7-bundle" (OuterVolumeSpecName: "bundle") pod "acf936f7-9ae5-4c32-af67-8f92692382a7" (UID: "acf936f7-9ae5-4c32-af67-8f92692382a7"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 23 19:27:38.939195 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:27:38.939176 2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b75982c7j" Jul 23 19:27:38.940661 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:27:38.940631 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/acf936f7-9ae5-4c32-af67-8f92692382a7-kube-api-access-2vfg7" (OuterVolumeSpecName: "kube-api-access-2vfg7") pod "acf936f7-9ae5-4c32-af67-8f92692382a7" (UID: "acf936f7-9ae5-4c32-af67-8f92692382a7"). InnerVolumeSpecName "kube-api-access-2vfg7". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 23 19:27:38.942882 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:27:38.942862 2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73nmtfx" Jul 23 19:27:38.943961 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:27:38.943939 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/acf936f7-9ae5-4c32-af67-8f92692382a7-util" (OuterVolumeSpecName: "util") pod "acf936f7-9ae5-4c32-af67-8f92692382a7" (UID: "acf936f7-9ae5-4c32-af67-8f92692382a7"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 23 19:27:39.039649 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:27:39.039628 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/fd160cf4-96c6-410f-a5b3-7dc83bb993c7-bundle\") pod \"fd160cf4-96c6-410f-a5b3-7dc83bb993c7\" (UID: \"fd160cf4-96c6-410f-a5b3-7dc83bb993c7\") " Jul 23 19:27:39.039767 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:27:39.039663 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-4kpfk\" (UniqueName: \"kubernetes.io/projected/fd160cf4-96c6-410f-a5b3-7dc83bb993c7-kube-api-access-4kpfk\") pod \"fd160cf4-96c6-410f-a5b3-7dc83bb993c7\" (UID: \"fd160cf4-96c6-410f-a5b3-7dc83bb993c7\") " Jul 23 19:27:39.039767 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:27:39.039698 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-dp7mw\" (UniqueName: \"kubernetes.io/projected/d1495985-7d1d-486a-894a-9af5eedb393e-kube-api-access-dp7mw\") pod \"d1495985-7d1d-486a-894a-9af5eedb393e\" (UID: \"d1495985-7d1d-486a-894a-9af5eedb393e\") " Jul 23 19:27:39.039767 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:27:39.039726 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/fd160cf4-96c6-410f-a5b3-7dc83bb993c7-util\") pod \"fd160cf4-96c6-410f-a5b3-7dc83bb993c7\" (UID: \"fd160cf4-96c6-410f-a5b3-7dc83bb993c7\") " Jul 23 19:27:39.039767 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:27:39.039741 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/d1495985-7d1d-486a-894a-9af5eedb393e-bundle\") pod \"d1495985-7d1d-486a-894a-9af5eedb393e\" (UID: \"d1495985-7d1d-486a-894a-9af5eedb393e\") " Jul 23 19:27:39.039767 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:27:39.039762 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/d1495985-7d1d-486a-894a-9af5eedb393e-util\") pod \"d1495985-7d1d-486a-894a-9af5eedb393e\" (UID: \"d1495985-7d1d-486a-894a-9af5eedb393e\") " Jul 23 19:27:39.040017 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:27:39.039973 2564 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/acf936f7-9ae5-4c32-af67-8f92692382a7-bundle\") on node \"ip-10-0-129-2.ec2.internal\" DevicePath \"\"" Jul 23 19:27:39.040017 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:27:39.039989 2564 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/acf936f7-9ae5-4c32-af67-8f92692382a7-util\") on node \"ip-10-0-129-2.ec2.internal\" DevicePath \"\"" Jul 23 19:27:39.040017 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:27:39.040002 2564 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2vfg7\" (UniqueName: \"kubernetes.io/projected/acf936f7-9ae5-4c32-af67-8f92692382a7-kube-api-access-2vfg7\") on node \"ip-10-0-129-2.ec2.internal\" DevicePath \"\"" Jul 23 19:27:39.040207 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:27:39.040167 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fd160cf4-96c6-410f-a5b3-7dc83bb993c7-bundle" (OuterVolumeSpecName: "bundle") pod "fd160cf4-96c6-410f-a5b3-7dc83bb993c7" (UID: "fd160cf4-96c6-410f-a5b3-7dc83bb993c7"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 23 19:27:39.040427 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:27:39.040406 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d1495985-7d1d-486a-894a-9af5eedb393e-bundle" (OuterVolumeSpecName: "bundle") pod "d1495985-7d1d-486a-894a-9af5eedb393e" (UID: "d1495985-7d1d-486a-894a-9af5eedb393e"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 23 19:27:39.041827 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:27:39.041805 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d1495985-7d1d-486a-894a-9af5eedb393e-kube-api-access-dp7mw" (OuterVolumeSpecName: "kube-api-access-dp7mw") pod "d1495985-7d1d-486a-894a-9af5eedb393e" (UID: "d1495985-7d1d-486a-894a-9af5eedb393e"). InnerVolumeSpecName "kube-api-access-dp7mw". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 23 19:27:39.041907 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:27:39.041847 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fd160cf4-96c6-410f-a5b3-7dc83bb993c7-kube-api-access-4kpfk" (OuterVolumeSpecName: "kube-api-access-4kpfk") pod "fd160cf4-96c6-410f-a5b3-7dc83bb993c7" (UID: "fd160cf4-96c6-410f-a5b3-7dc83bb993c7"). InnerVolumeSpecName "kube-api-access-4kpfk". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 23 19:27:39.048118 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:27:39.048094 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fd160cf4-96c6-410f-a5b3-7dc83bb993c7-util" (OuterVolumeSpecName: "util") pod "fd160cf4-96c6-410f-a5b3-7dc83bb993c7" (UID: "fd160cf4-96c6-410f-a5b3-7dc83bb993c7"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 23 19:27:39.048427 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:27:39.048411 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d1495985-7d1d-486a-894a-9af5eedb393e-util" (OuterVolumeSpecName: "util") pod "d1495985-7d1d-486a-894a-9af5eedb393e" (UID: "d1495985-7d1d-486a-894a-9af5eedb393e"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 23 19:27:39.141026 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:27:39.141005 2564 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/d1495985-7d1d-486a-894a-9af5eedb393e-util\") on node \"ip-10-0-129-2.ec2.internal\" DevicePath \"\"" Jul 23 19:27:39.141026 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:27:39.141024 2564 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/fd160cf4-96c6-410f-a5b3-7dc83bb993c7-bundle\") on node \"ip-10-0-129-2.ec2.internal\" DevicePath \"\"" Jul 23 19:27:39.141132 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:27:39.141033 2564 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4kpfk\" (UniqueName: \"kubernetes.io/projected/fd160cf4-96c6-410f-a5b3-7dc83bb993c7-kube-api-access-4kpfk\") on node \"ip-10-0-129-2.ec2.internal\" DevicePath \"\"" Jul 23 19:27:39.141132 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:27:39.141042 2564 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dp7mw\" (UniqueName: \"kubernetes.io/projected/d1495985-7d1d-486a-894a-9af5eedb393e-kube-api-access-dp7mw\") on node \"ip-10-0-129-2.ec2.internal\" DevicePath \"\"" Jul 23 19:27:39.141132 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:27:39.141050 2564 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/fd160cf4-96c6-410f-a5b3-7dc83bb993c7-util\") on node \"ip-10-0-129-2.ec2.internal\" DevicePath \"\"" Jul 23 19:27:39.141132 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:27:39.141057 2564 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/d1495985-7d1d-486a-894a-9af5eedb393e-bundle\") on node \"ip-10-0-129-2.ec2.internal\" DevicePath \"\"" Jul 23 19:27:39.764979 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:27:39.764954 2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b75982c7j" Jul 23 19:27:39.765299 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:27:39.764995 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b75982c7j" event={"ID":"fd160cf4-96c6-410f-a5b3-7dc83bb993c7","Type":"ContainerDied","Data":"f5a86702da05b0ce69a2e8d6d44267c12df94a47ce93d355fb8cea4edf1a4258"} Jul 23 19:27:39.765299 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:27:39.765029 2564 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="f5a86702da05b0ce69a2e8d6d44267c12df94a47ce93d355fb8cea4edf1a4258" Jul 23 19:27:39.767058 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:27:39.767038 2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1m88qh" Jul 23 19:27:39.767148 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:27:39.767071 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1m88qh" event={"ID":"acf936f7-9ae5-4c32-af67-8f92692382a7","Type":"ContainerDied","Data":"abc40a7cbd0472afc520246e812428e848ad4128c36dd0d53c3fc9faafd91f5a"} Jul 23 19:27:39.767148 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:27:39.767105 2564 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="abc40a7cbd0472afc520246e812428e848ad4128c36dd0d53c3fc9faafd91f5a" Jul 23 19:27:39.769048 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:27:39.769023 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73nmtfx" event={"ID":"d1495985-7d1d-486a-894a-9af5eedb393e","Type":"ContainerDied","Data":"8f66b8b68dfd7144c1e3bd455cee0e350e0f2c5381bab383c408042975cbbd78"} Jul 23 19:27:39.769125 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:27:39.769057 2564 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="8f66b8b68dfd7144c1e3bd455cee0e350e0f2c5381bab383c408042975cbbd78" Jul 23 19:27:39.769125 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:27:39.769078 2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73nmtfx" Jul 23 19:27:52.828453 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:27:52.828427 2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-6wvl9"] Jul 23 19:27:52.828949 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:27:52.828722 2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="fd160cf4-96c6-410f-a5b3-7dc83bb993c7" containerName="pull" Jul 23 19:27:52.828949 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:27:52.828734 2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="fd160cf4-96c6-410f-a5b3-7dc83bb993c7" containerName="pull" Jul 23 19:27:52.828949 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:27:52.828746 2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="acf936f7-9ae5-4c32-af67-8f92692382a7" containerName="pull" Jul 23 19:27:52.828949 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:27:52.828753 2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="acf936f7-9ae5-4c32-af67-8f92692382a7" containerName="pull" Jul 23 19:27:52.828949 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:27:52.828763 2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="acf936f7-9ae5-4c32-af67-8f92692382a7" containerName="extract" Jul 23 19:27:52.828949 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:27:52.828768 2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="acf936f7-9ae5-4c32-af67-8f92692382a7" containerName="extract" Jul 23 19:27:52.828949 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:27:52.828777 2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7326cf84-a21c-4c92-a262-fbff97a47bbd" containerName="util" Jul 23 19:27:52.828949 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:27:52.828781 2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="7326cf84-a21c-4c92-a262-fbff97a47bbd" containerName="util" Jul 23 19:27:52.828949 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:27:52.828791 2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d1495985-7d1d-486a-894a-9af5eedb393e" containerName="pull" Jul 23 19:27:52.828949 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:27:52.828795 2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="d1495985-7d1d-486a-894a-9af5eedb393e" containerName="pull" Jul 23 19:27:52.828949 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:27:52.828804 2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7326cf84-a21c-4c92-a262-fbff97a47bbd" containerName="extract" Jul 23 19:27:52.828949 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:27:52.828810 2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="7326cf84-a21c-4c92-a262-fbff97a47bbd" containerName="extract" Jul 23 19:27:52.828949 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:27:52.828821 2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7326cf84-a21c-4c92-a262-fbff97a47bbd" containerName="pull" Jul 23 19:27:52.828949 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:27:52.828828 2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="7326cf84-a21c-4c92-a262-fbff97a47bbd" containerName="pull" Jul 23 19:27:52.828949 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:27:52.828833 2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d1495985-7d1d-486a-894a-9af5eedb393e" containerName="util" Jul 23 19:27:52.828949 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:27:52.828838 2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="d1495985-7d1d-486a-894a-9af5eedb393e" containerName="util" Jul 23 19:27:52.828949 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:27:52.828843 2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d1495985-7d1d-486a-894a-9af5eedb393e" containerName="extract" Jul 23 19:27:52.828949 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:27:52.828848 2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="d1495985-7d1d-486a-894a-9af5eedb393e" containerName="extract" Jul 23 19:27:52.828949 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:27:52.828855 2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="fd160cf4-96c6-410f-a5b3-7dc83bb993c7" containerName="util" Jul 23 19:27:52.828949 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:27:52.828860 2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="fd160cf4-96c6-410f-a5b3-7dc83bb993c7" containerName="util" Jul 23 19:27:52.828949 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:27:52.828865 2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="acf936f7-9ae5-4c32-af67-8f92692382a7" containerName="util" Jul 23 19:27:52.828949 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:27:52.828870 2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="acf936f7-9ae5-4c32-af67-8f92692382a7" containerName="util" Jul 23 19:27:52.828949 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:27:52.828876 2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="fd160cf4-96c6-410f-a5b3-7dc83bb993c7" containerName="extract" Jul 23 19:27:52.828949 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:27:52.828880 2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="fd160cf4-96c6-410f-a5b3-7dc83bb993c7" containerName="extract" Jul 23 19:27:52.828949 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:27:52.828947 2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="acf936f7-9ae5-4c32-af67-8f92692382a7" containerName="extract" Jul 23 19:27:52.828949 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:27:52.828955 2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="d1495985-7d1d-486a-894a-9af5eedb393e" containerName="extract" Jul 23 19:27:52.828949 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:27:52.828962 2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="fd160cf4-96c6-410f-a5b3-7dc83bb993c7" containerName="extract" Jul 23 19:27:52.828949 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:27:52.828967 2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="7326cf84-a21c-4c92-a262-fbff97a47bbd" containerName="extract" Jul 23 19:27:52.833179 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:27:52.833161 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-6wvl9" Jul 23 19:27:52.835791 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:27:52.835773 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"openshift-service-ca.crt\"" Jul 23 19:27:52.835979 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:27:52.835965 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"kuadrant-operator-controller-manager-dockercfg-44wf7\"" Jul 23 19:27:52.838162 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:27:52.838144 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kube-root-ca.crt\"" Jul 23 19:27:52.858432 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:27:52.858410 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-6wvl9"] Jul 23 19:27:52.939943 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:27:52.939915 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/cc7f516e-0208-4831-82a7-290db149d2c4-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-8684c9ff49-6wvl9\" (UID: \"cc7f516e-0208-4831-82a7-290db149d2c4\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-6wvl9" Jul 23 19:27:52.940059 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:27:52.939957 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-k28w7\" (UniqueName: \"kubernetes.io/projected/cc7f516e-0208-4831-82a7-290db149d2c4-kube-api-access-k28w7\") pod \"kuadrant-operator-controller-manager-8684c9ff49-6wvl9\" (UID: \"cc7f516e-0208-4831-82a7-290db149d2c4\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-6wvl9" Jul 23 19:27:53.041306 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:27:53.041278 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/cc7f516e-0208-4831-82a7-290db149d2c4-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-8684c9ff49-6wvl9\" (UID: \"cc7f516e-0208-4831-82a7-290db149d2c4\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-6wvl9" Jul 23 19:27:53.041457 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:27:53.041342 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k28w7\" (UniqueName: \"kubernetes.io/projected/cc7f516e-0208-4831-82a7-290db149d2c4-kube-api-access-k28w7\") pod \"kuadrant-operator-controller-manager-8684c9ff49-6wvl9\" (UID: \"cc7f516e-0208-4831-82a7-290db149d2c4\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-6wvl9" Jul 23 19:27:53.041711 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:27:53.041694 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/cc7f516e-0208-4831-82a7-290db149d2c4-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-8684c9ff49-6wvl9\" (UID: \"cc7f516e-0208-4831-82a7-290db149d2c4\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-6wvl9" Jul 23 19:27:53.077592 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:27:53.077567 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-k28w7\" (UniqueName: \"kubernetes.io/projected/cc7f516e-0208-4831-82a7-290db149d2c4-kube-api-access-k28w7\") pod \"kuadrant-operator-controller-manager-8684c9ff49-6wvl9\" (UID: \"cc7f516e-0208-4831-82a7-290db149d2c4\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-6wvl9" Jul 23 19:27:53.142099 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:27:53.142049 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-6wvl9" Jul 23 19:27:53.284464 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:27:53.284432 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-6wvl9"] Jul 23 19:27:53.288454 ip-10-0-129-2 kubenswrapper[2564]: W0723 19:27:53.288424 2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcc7f516e_0208_4831_82a7_290db149d2c4.slice/crio-b2bbaccca9849fa8048509e65bc71f67e81d80a6f6ebb167fdaa43ff36eab41a WatchSource:0}: Error finding container b2bbaccca9849fa8048509e65bc71f67e81d80a6f6ebb167fdaa43ff36eab41a: Status 404 returned error can't find the container with id b2bbaccca9849fa8048509e65bc71f67e81d80a6f6ebb167fdaa43ff36eab41a Jul 23 19:27:53.838749 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:27:53.838718 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-6wvl9" event={"ID":"cc7f516e-0208-4831-82a7-290db149d2c4","Type":"ContainerStarted","Data":"b2bbaccca9849fa8048509e65bc71f67e81d80a6f6ebb167fdaa43ff36eab41a"} Jul 23 19:27:59.875877 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:27:59.875806 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-6wvl9" event={"ID":"cc7f516e-0208-4831-82a7-290db149d2c4","Type":"ContainerStarted","Data":"d15088362816e1c28f5286adbad57fb274437095d15fe9febc1e46db0d4f93a2"} Jul 23 19:27:59.876288 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:27:59.875936 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-6wvl9" Jul 23 19:27:59.895439 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:27:59.895396 2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-6wvl9" podStartSLOduration=1.7622336920000001 podStartE2EDuration="7.895384118s" totalImagesPullingTime="6.133150426s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-23 19:27:52 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-23 19:27:59.893189071 +0000 UTC m=+505.744965308" watchObservedRunningTime="2026-07-23 19:27:59.895384118 +0000 UTC m=+505.747160363" Jul 23 19:28:10.882925 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:28:10.882900 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-6wvl9" Jul 23 19:28:12.877633 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:28:12.877600 2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-6wvl9"] Jul 23 19:28:12.878107 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:28:12.877803 2564 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-6wvl9" podUID="cc7f516e-0208-4831-82a7-290db149d2c4" containerName="manager" containerID="cri-o://d15088362816e1c28f5286adbad57fb274437095d15fe9febc1e46db0d4f93a2" gracePeriod=2 Jul 23 19:28:12.879875 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:28:12.879853 2564 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-6wvl9"] Jul 23 19:28:12.882556 ip-10-0-129-2 kubenswrapper[2564]: E0723 19:28:12.882525 2564 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-8684c9ff49-6wvl9\" is forbidden: User \"system:node:ip-10-0-129-2.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-129-2.ec2.internal' and this object" podUID="cc7f516e-0208-4831-82a7-290db149d2c4" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-6wvl9" Jul 23 19:28:12.937605 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:28:12.937573 2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-s4nck"] Jul 23 19:28:12.938136 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:28:12.938120 2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="cc7f516e-0208-4831-82a7-290db149d2c4" containerName="manager" Jul 23 19:28:12.938177 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:28:12.938141 2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="cc7f516e-0208-4831-82a7-290db149d2c4" containerName="manager" Jul 23 19:28:12.938247 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:28:12.938233 2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="cc7f516e-0208-4831-82a7-290db149d2c4" containerName="manager" Jul 23 19:28:12.941630 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:28:12.941610 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-s4nck" Jul 23 19:28:12.948694 ip-10-0-129-2 kubenswrapper[2564]: E0723 19:28:12.948670 2564 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-8684c9ff49-6wvl9\" is forbidden: User \"system:node:ip-10-0-129-2.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-129-2.ec2.internal' and this object" podUID="cc7f516e-0208-4831-82a7-290db149d2c4" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-6wvl9" Jul 23 19:28:12.964828 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:28:12.964784 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-s4nck"] Jul 23 19:28:13.090867 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:28:13.090839 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kshd4\" (UniqueName: \"kubernetes.io/projected/c66588cd-b9f6-4ccb-b82b-60501df6b656-kube-api-access-kshd4\") pod \"kuadrant-operator-controller-manager-8684c9ff49-s4nck\" (UID: \"c66588cd-b9f6-4ccb-b82b-60501df6b656\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-s4nck" Jul 23 19:28:13.090986 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:28:13.090892 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/c66588cd-b9f6-4ccb-b82b-60501df6b656-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-8684c9ff49-s4nck\" (UID: \"c66588cd-b9f6-4ccb-b82b-60501df6b656\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-s4nck" Jul 23 19:28:13.127520 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:28:13.127499 2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-6wvl9" Jul 23 19:28:13.133203 ip-10-0-129-2 kubenswrapper[2564]: E0723 19:28:13.133154 2564 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-8684c9ff49-6wvl9\" is forbidden: User \"system:node:ip-10-0-129-2.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-129-2.ec2.internal' and this object" podUID="cc7f516e-0208-4831-82a7-290db149d2c4" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-6wvl9" Jul 23 19:28:13.191790 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:28:13.191768 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kshd4\" (UniqueName: \"kubernetes.io/projected/c66588cd-b9f6-4ccb-b82b-60501df6b656-kube-api-access-kshd4\") pod \"kuadrant-operator-controller-manager-8684c9ff49-s4nck\" (UID: \"c66588cd-b9f6-4ccb-b82b-60501df6b656\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-s4nck" Jul 23 19:28:13.191864 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:28:13.191804 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/c66588cd-b9f6-4ccb-b82b-60501df6b656-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-8684c9ff49-s4nck\" (UID: \"c66588cd-b9f6-4ccb-b82b-60501df6b656\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-s4nck" Jul 23 19:28:13.192069 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:28:13.192055 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/c66588cd-b9f6-4ccb-b82b-60501df6b656-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-8684c9ff49-s4nck\" (UID: \"c66588cd-b9f6-4ccb-b82b-60501df6b656\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-s4nck" Jul 23 19:28:13.201850 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:28:13.201823 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-kshd4\" (UniqueName: \"kubernetes.io/projected/c66588cd-b9f6-4ccb-b82b-60501df6b656-kube-api-access-kshd4\") pod \"kuadrant-operator-controller-manager-8684c9ff49-s4nck\" (UID: \"c66588cd-b9f6-4ccb-b82b-60501df6b656\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-s4nck" Jul 23 19:28:13.258537 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:28:13.258518 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-s4nck" Jul 23 19:28:13.292468 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:28:13.292439 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-k28w7\" (UniqueName: \"kubernetes.io/projected/cc7f516e-0208-4831-82a7-290db149d2c4-kube-api-access-k28w7\") pod \"cc7f516e-0208-4831-82a7-290db149d2c4\" (UID: \"cc7f516e-0208-4831-82a7-290db149d2c4\") " Jul 23 19:28:13.292563 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:28:13.292505 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/cc7f516e-0208-4831-82a7-290db149d2c4-extensions-socket-volume\") pod \"cc7f516e-0208-4831-82a7-290db149d2c4\" (UID: \"cc7f516e-0208-4831-82a7-290db149d2c4\") " Jul 23 19:28:13.293074 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:28:13.293051 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cc7f516e-0208-4831-82a7-290db149d2c4-extensions-socket-volume" (OuterVolumeSpecName: "extensions-socket-volume") pod "cc7f516e-0208-4831-82a7-290db149d2c4" (UID: "cc7f516e-0208-4831-82a7-290db149d2c4"). InnerVolumeSpecName "extensions-socket-volume". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 23 19:28:13.294470 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:28:13.294445 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cc7f516e-0208-4831-82a7-290db149d2c4-kube-api-access-k28w7" (OuterVolumeSpecName: "kube-api-access-k28w7") pod "cc7f516e-0208-4831-82a7-290db149d2c4" (UID: "cc7f516e-0208-4831-82a7-290db149d2c4"). InnerVolumeSpecName "kube-api-access-k28w7". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 23 19:28:13.393487 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:28:13.393428 2564 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-k28w7\" (UniqueName: \"kubernetes.io/projected/cc7f516e-0208-4831-82a7-290db149d2c4-kube-api-access-k28w7\") on node \"ip-10-0-129-2.ec2.internal\" DevicePath \"\"" Jul 23 19:28:13.393487 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:28:13.393451 2564 reconciler_common.go:299] "Volume detached for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/cc7f516e-0208-4831-82a7-290db149d2c4-extensions-socket-volume\") on node \"ip-10-0-129-2.ec2.internal\" DevicePath \"\"" Jul 23 19:28:13.477601 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:28:13.477581 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-s4nck"] Jul 23 19:28:13.479607 ip-10-0-129-2 kubenswrapper[2564]: W0723 19:28:13.479579 2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc66588cd_b9f6_4ccb_b82b_60501df6b656.slice/crio-0e3069761d749d8c7263ce72d9cb2cd6316541ea27f1a379dbe121e7624945aa WatchSource:0}: Error finding container 0e3069761d749d8c7263ce72d9cb2cd6316541ea27f1a379dbe121e7624945aa: Status 404 returned error can't find the container with id 0e3069761d749d8c7263ce72d9cb2cd6316541ea27f1a379dbe121e7624945aa Jul 23 19:28:13.943470 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:28:13.943440 2564 generic.go:356] "Generic (PLEG): container finished" podID="cc7f516e-0208-4831-82a7-290db149d2c4" containerID="d15088362816e1c28f5286adbad57fb274437095d15fe9febc1e46db0d4f93a2" exitCode=0 Jul 23 19:28:13.943866 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:28:13.943486 2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-6wvl9" Jul 23 19:28:13.943866 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:28:13.943543 2564 scope.go:117] "RemoveContainer" containerID="d15088362816e1c28f5286adbad57fb274437095d15fe9febc1e46db0d4f93a2" Jul 23 19:28:13.945712 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:28:13.945674 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-s4nck" event={"ID":"c66588cd-b9f6-4ccb-b82b-60501df6b656","Type":"ContainerStarted","Data":"fe6fd85d394149a981c1c9a081e8aeba573dab17801c39ea936be236c2370045"} Jul 23 19:28:13.945817 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:28:13.945763 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-s4nck" event={"ID":"c66588cd-b9f6-4ccb-b82b-60501df6b656","Type":"ContainerStarted","Data":"0e3069761d749d8c7263ce72d9cb2cd6316541ea27f1a379dbe121e7624945aa"} Jul 23 19:28:13.945817 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:28:13.945800 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-s4nck" Jul 23 19:28:13.946540 ip-10-0-129-2 kubenswrapper[2564]: E0723 19:28:13.946516 2564 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-8684c9ff49-6wvl9\" is forbidden: User \"system:node:ip-10-0-129-2.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-129-2.ec2.internal' and this object" podUID="cc7f516e-0208-4831-82a7-290db149d2c4" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-6wvl9" Jul 23 19:28:13.949470 ip-10-0-129-2 kubenswrapper[2564]: E0723 19:28:13.949443 2564 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-8684c9ff49-6wvl9\" is forbidden: User \"system:node:ip-10-0-129-2.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-129-2.ec2.internal' and this object" podUID="cc7f516e-0208-4831-82a7-290db149d2c4" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-6wvl9" Jul 23 19:28:13.955624 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:28:13.955608 2564 scope.go:117] "RemoveContainer" containerID="d15088362816e1c28f5286adbad57fb274437095d15fe9febc1e46db0d4f93a2" Jul 23 19:28:13.955850 ip-10-0-129-2 kubenswrapper[2564]: E0723 19:28:13.955832 2564 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d15088362816e1c28f5286adbad57fb274437095d15fe9febc1e46db0d4f93a2\": container with ID starting with d15088362816e1c28f5286adbad57fb274437095d15fe9febc1e46db0d4f93a2 not found: ID does not exist" containerID="d15088362816e1c28f5286adbad57fb274437095d15fe9febc1e46db0d4f93a2" Jul 23 19:28:13.955941 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:28:13.955857 2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d15088362816e1c28f5286adbad57fb274437095d15fe9febc1e46db0d4f93a2"} err="failed to get container status \"d15088362816e1c28f5286adbad57fb274437095d15fe9febc1e46db0d4f93a2\": rpc error: code = NotFound desc = could not find container \"d15088362816e1c28f5286adbad57fb274437095d15fe9febc1e46db0d4f93a2\": container with ID starting with d15088362816e1c28f5286adbad57fb274437095d15fe9febc1e46db0d4f93a2 not found: ID does not exist" Jul 23 19:28:14.005873 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:28:14.005833 2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-s4nck" podStartSLOduration=2.00582178 podStartE2EDuration="2.00582178s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-23 19:28:12 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-23 19:28:14.005273768 +0000 UTC m=+519.857050013" watchObservedRunningTime="2026-07-23 19:28:14.00582178 +0000 UTC m=+519.857598025" Jul 23 19:28:14.007406 ip-10-0-129-2 kubenswrapper[2564]: E0723 19:28:14.007388 2564 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-8684c9ff49-6wvl9\" is forbidden: User \"system:node:ip-10-0-129-2.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-129-2.ec2.internal' and this object" podUID="cc7f516e-0208-4831-82a7-290db149d2c4" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-6wvl9" Jul 23 19:28:14.657797 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:28:14.657769 2564 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="cc7f516e-0208-4831-82a7-290db149d2c4" path="/var/lib/kubelet/pods/cc7f516e-0208-4831-82a7-290db149d2c4/volumes" Jul 23 19:28:14.658155 ip-10-0-129-2 kubenswrapper[2564]: E0723 19:28:14.658138 2564 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-8684c9ff49-6wvl9\" is forbidden: User \"system:node:ip-10-0-129-2.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-129-2.ec2.internal' and this object" podUID="cc7f516e-0208-4831-82a7-290db149d2c4" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-6wvl9" Jul 23 19:28:24.953200 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:28:24.953170 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-s4nck" Jul 23 19:28:40.615497 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:28:40.615467 2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-s4nck"] Jul 23 19:28:40.615873 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:28:40.615625 2564 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-s4nck" podUID="c66588cd-b9f6-4ccb-b82b-60501df6b656" containerName="manager" containerID="cri-o://fe6fd85d394149a981c1c9a081e8aeba573dab17801c39ea936be236c2370045" gracePeriod=10 Jul 23 19:28:40.861870 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:28:40.861844 2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-s4nck" Jul 23 19:28:40.983064 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:28:40.983040 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/c66588cd-b9f6-4ccb-b82b-60501df6b656-extensions-socket-volume\") pod \"c66588cd-b9f6-4ccb-b82b-60501df6b656\" (UID: \"c66588cd-b9f6-4ccb-b82b-60501df6b656\") " Jul 23 19:28:40.983194 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:28:40.983101 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-kshd4\" (UniqueName: \"kubernetes.io/projected/c66588cd-b9f6-4ccb-b82b-60501df6b656-kube-api-access-kshd4\") pod \"c66588cd-b9f6-4ccb-b82b-60501df6b656\" (UID: \"c66588cd-b9f6-4ccb-b82b-60501df6b656\") " Jul 23 19:28:40.983473 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:28:40.983444 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c66588cd-b9f6-4ccb-b82b-60501df6b656-extensions-socket-volume" (OuterVolumeSpecName: "extensions-socket-volume") pod "c66588cd-b9f6-4ccb-b82b-60501df6b656" (UID: "c66588cd-b9f6-4ccb-b82b-60501df6b656"). InnerVolumeSpecName "extensions-socket-volume". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 23 19:28:40.985077 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:28:40.985062 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c66588cd-b9f6-4ccb-b82b-60501df6b656-kube-api-access-kshd4" (OuterVolumeSpecName: "kube-api-access-kshd4") pod "c66588cd-b9f6-4ccb-b82b-60501df6b656" (UID: "c66588cd-b9f6-4ccb-b82b-60501df6b656"). InnerVolumeSpecName "kube-api-access-kshd4". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 23 19:28:41.077208 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:28:41.077182 2564 generic.go:356] "Generic (PLEG): container finished" podID="c66588cd-b9f6-4ccb-b82b-60501df6b656" containerID="fe6fd85d394149a981c1c9a081e8aeba573dab17801c39ea936be236c2370045" exitCode=0 Jul 23 19:28:41.077293 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:28:41.077232 2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-s4nck" Jul 23 19:28:41.077293 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:28:41.077237 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-s4nck" event={"ID":"c66588cd-b9f6-4ccb-b82b-60501df6b656","Type":"ContainerDied","Data":"fe6fd85d394149a981c1c9a081e8aeba573dab17801c39ea936be236c2370045"} Jul 23 19:28:41.077387 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:28:41.077341 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-s4nck" event={"ID":"c66588cd-b9f6-4ccb-b82b-60501df6b656","Type":"ContainerDied","Data":"0e3069761d749d8c7263ce72d9cb2cd6316541ea27f1a379dbe121e7624945aa"} Jul 23 19:28:41.077387 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:28:41.077373 2564 scope.go:117] "RemoveContainer" containerID="fe6fd85d394149a981c1c9a081e8aeba573dab17801c39ea936be236c2370045" Jul 23 19:28:41.083768 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:28:41.083750 2564 reconciler_common.go:299] "Volume detached for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/c66588cd-b9f6-4ccb-b82b-60501df6b656-extensions-socket-volume\") on node \"ip-10-0-129-2.ec2.internal\" DevicePath \"\"" Jul 23 19:28:41.083877 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:28:41.083770 2564 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kshd4\" (UniqueName: \"kubernetes.io/projected/c66588cd-b9f6-4ccb-b82b-60501df6b656-kube-api-access-kshd4\") on node \"ip-10-0-129-2.ec2.internal\" DevicePath \"\"" Jul 23 19:28:41.086385 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:28:41.086368 2564 scope.go:117] "RemoveContainer" containerID="fe6fd85d394149a981c1c9a081e8aeba573dab17801c39ea936be236c2370045" Jul 23 19:28:41.086613 ip-10-0-129-2 kubenswrapper[2564]: E0723 19:28:41.086597 2564 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fe6fd85d394149a981c1c9a081e8aeba573dab17801c39ea936be236c2370045\": container with ID starting with fe6fd85d394149a981c1c9a081e8aeba573dab17801c39ea936be236c2370045 not found: ID does not exist" containerID="fe6fd85d394149a981c1c9a081e8aeba573dab17801c39ea936be236c2370045" Jul 23 19:28:41.086678 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:28:41.086620 2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fe6fd85d394149a981c1c9a081e8aeba573dab17801c39ea936be236c2370045"} err="failed to get container status \"fe6fd85d394149a981c1c9a081e8aeba573dab17801c39ea936be236c2370045\": rpc error: code = NotFound desc = could not find container \"fe6fd85d394149a981c1c9a081e8aeba573dab17801c39ea936be236c2370045\": container with ID starting with fe6fd85d394149a981c1c9a081e8aeba573dab17801c39ea936be236c2370045 not found: ID does not exist" Jul 23 19:28:41.104422 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:28:41.104400 2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-s4nck"] Jul 23 19:28:41.107826 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:28:41.107808 2564 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-s4nck"] Jul 23 19:28:42.657599 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:28:42.657576 2564 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="c66588cd-b9f6-4ccb-b82b-60501df6b656" path="/var/lib/kubelet/pods/c66588cd-b9f6-4ccb-b82b-60501df6b656/volumes" Jul 23 19:28:57.145243 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:28:57.145216 2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-pp577"] Jul 23 19:28:57.145777 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:28:57.145755 2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c66588cd-b9f6-4ccb-b82b-60501df6b656" containerName="manager" Jul 23 19:28:57.145777 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:28:57.145774 2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="c66588cd-b9f6-4ccb-b82b-60501df6b656" containerName="manager" Jul 23 19:28:57.145907 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:28:57.145841 2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="c66588cd-b9f6-4ccb-b82b-60501df6b656" containerName="manager" Jul 23 19:28:57.148928 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:28:57.148911 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-pp577" Jul 23 19:28:57.151536 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:28:57.151513 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"maas-default-gateway-openshift-default-dockercfg-2znxf\"" Jul 23 19:28:57.165080 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:28:57.165059 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-pp577"] Jul 23 19:28:57.192289 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:28:57.192263 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/01866086-e44f-4c2d-9f3d-5b5a7cf79029-istio-data\") pod \"maas-default-gateway-openshift-default-56654dcf7-pp577\" (UID: \"01866086-e44f-4c2d-9f3d-5b5a7cf79029\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-pp577" Jul 23 19:28:57.192418 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:28:57.192304 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/01866086-e44f-4c2d-9f3d-5b5a7cf79029-credential-socket\") pod \"maas-default-gateway-openshift-default-56654dcf7-pp577\" (UID: \"01866086-e44f-4c2d-9f3d-5b5a7cf79029\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-pp577" Jul 23 19:28:57.192418 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:28:57.192355 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/01866086-e44f-4c2d-9f3d-5b5a7cf79029-istio-token\") pod \"maas-default-gateway-openshift-default-56654dcf7-pp577\" (UID: \"01866086-e44f-4c2d-9f3d-5b5a7cf79029\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-pp577" Jul 23 19:28:57.192418 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:28:57.192385 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/01866086-e44f-4c2d-9f3d-5b5a7cf79029-istio-envoy\") pod \"maas-default-gateway-openshift-default-56654dcf7-pp577\" (UID: \"01866086-e44f-4c2d-9f3d-5b5a7cf79029\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-pp577" Jul 23 19:28:57.192418 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:28:57.192409 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/01866086-e44f-4c2d-9f3d-5b5a7cf79029-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-56654dcf7-pp577\" (UID: \"01866086-e44f-4c2d-9f3d-5b5a7cf79029\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-pp577" Jul 23 19:28:57.192609 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:28:57.192443 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/01866086-e44f-4c2d-9f3d-5b5a7cf79029-workload-certs\") pod \"maas-default-gateway-openshift-default-56654dcf7-pp577\" (UID: \"01866086-e44f-4c2d-9f3d-5b5a7cf79029\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-pp577" Jul 23 19:28:57.192609 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:28:57.192467 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/01866086-e44f-4c2d-9f3d-5b5a7cf79029-istio-podinfo\") pod \"maas-default-gateway-openshift-default-56654dcf7-pp577\" (UID: \"01866086-e44f-4c2d-9f3d-5b5a7cf79029\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-pp577" Jul 23 19:28:57.192609 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:28:57.192513 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/01866086-e44f-4c2d-9f3d-5b5a7cf79029-workload-socket\") pod \"maas-default-gateway-openshift-default-56654dcf7-pp577\" (UID: \"01866086-e44f-4c2d-9f3d-5b5a7cf79029\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-pp577" Jul 23 19:28:57.192609 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:28:57.192558 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8b25k\" (UniqueName: \"kubernetes.io/projected/01866086-e44f-4c2d-9f3d-5b5a7cf79029-kube-api-access-8b25k\") pod \"maas-default-gateway-openshift-default-56654dcf7-pp577\" (UID: \"01866086-e44f-4c2d-9f3d-5b5a7cf79029\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-pp577" Jul 23 19:28:57.292886 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:28:57.292865 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/01866086-e44f-4c2d-9f3d-5b5a7cf79029-istio-data\") pod \"maas-default-gateway-openshift-default-56654dcf7-pp577\" (UID: \"01866086-e44f-4c2d-9f3d-5b5a7cf79029\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-pp577" Jul 23 19:28:57.293017 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:28:57.292903 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/01866086-e44f-4c2d-9f3d-5b5a7cf79029-credential-socket\") pod \"maas-default-gateway-openshift-default-56654dcf7-pp577\" (UID: \"01866086-e44f-4c2d-9f3d-5b5a7cf79029\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-pp577" Jul 23 19:28:57.293017 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:28:57.292926 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/01866086-e44f-4c2d-9f3d-5b5a7cf79029-istio-token\") pod \"maas-default-gateway-openshift-default-56654dcf7-pp577\" (UID: \"01866086-e44f-4c2d-9f3d-5b5a7cf79029\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-pp577" Jul 23 19:28:57.293017 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:28:57.292954 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/01866086-e44f-4c2d-9f3d-5b5a7cf79029-istio-envoy\") pod \"maas-default-gateway-openshift-default-56654dcf7-pp577\" (UID: \"01866086-e44f-4c2d-9f3d-5b5a7cf79029\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-pp577" Jul 23 19:28:57.293017 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:28:57.292976 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/01866086-e44f-4c2d-9f3d-5b5a7cf79029-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-56654dcf7-pp577\" (UID: \"01866086-e44f-4c2d-9f3d-5b5a7cf79029\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-pp577" Jul 23 19:28:57.293017 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:28:57.293010 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/01866086-e44f-4c2d-9f3d-5b5a7cf79029-workload-certs\") pod \"maas-default-gateway-openshift-default-56654dcf7-pp577\" (UID: \"01866086-e44f-4c2d-9f3d-5b5a7cf79029\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-pp577" Jul 23 19:28:57.293246 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:28:57.293034 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/01866086-e44f-4c2d-9f3d-5b5a7cf79029-istio-podinfo\") pod \"maas-default-gateway-openshift-default-56654dcf7-pp577\" (UID: \"01866086-e44f-4c2d-9f3d-5b5a7cf79029\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-pp577" Jul 23 19:28:57.293246 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:28:57.293076 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/01866086-e44f-4c2d-9f3d-5b5a7cf79029-workload-socket\") pod \"maas-default-gateway-openshift-default-56654dcf7-pp577\" (UID: \"01866086-e44f-4c2d-9f3d-5b5a7cf79029\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-pp577" Jul 23 19:28:57.293246 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:28:57.293141 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8b25k\" (UniqueName: \"kubernetes.io/projected/01866086-e44f-4c2d-9f3d-5b5a7cf79029-kube-api-access-8b25k\") pod \"maas-default-gateway-openshift-default-56654dcf7-pp577\" (UID: \"01866086-e44f-4c2d-9f3d-5b5a7cf79029\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-pp577" Jul 23 19:28:57.293411 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:28:57.293277 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/01866086-e44f-4c2d-9f3d-5b5a7cf79029-istio-data\") pod \"maas-default-gateway-openshift-default-56654dcf7-pp577\" (UID: \"01866086-e44f-4c2d-9f3d-5b5a7cf79029\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-pp577" Jul 23 19:28:57.293411 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:28:57.293294 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/01866086-e44f-4c2d-9f3d-5b5a7cf79029-credential-socket\") pod \"maas-default-gateway-openshift-default-56654dcf7-pp577\" (UID: \"01866086-e44f-4c2d-9f3d-5b5a7cf79029\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-pp577" Jul 23 19:28:57.293514 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:28:57.293490 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/01866086-e44f-4c2d-9f3d-5b5a7cf79029-workload-socket\") pod \"maas-default-gateway-openshift-default-56654dcf7-pp577\" (UID: \"01866086-e44f-4c2d-9f3d-5b5a7cf79029\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-pp577" Jul 23 19:28:57.293570 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:28:57.293541 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/01866086-e44f-4c2d-9f3d-5b5a7cf79029-workload-certs\") pod \"maas-default-gateway-openshift-default-56654dcf7-pp577\" (UID: \"01866086-e44f-4c2d-9f3d-5b5a7cf79029\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-pp577" Jul 23 19:28:57.293746 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:28:57.293726 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/01866086-e44f-4c2d-9f3d-5b5a7cf79029-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-56654dcf7-pp577\" (UID: \"01866086-e44f-4c2d-9f3d-5b5a7cf79029\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-pp577" Jul 23 19:28:57.295443 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:28:57.295424 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/01866086-e44f-4c2d-9f3d-5b5a7cf79029-istio-envoy\") pod \"maas-default-gateway-openshift-default-56654dcf7-pp577\" (UID: \"01866086-e44f-4c2d-9f3d-5b5a7cf79029\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-pp577" Jul 23 19:28:57.295691 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:28:57.295676 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/01866086-e44f-4c2d-9f3d-5b5a7cf79029-istio-podinfo\") pod \"maas-default-gateway-openshift-default-56654dcf7-pp577\" (UID: \"01866086-e44f-4c2d-9f3d-5b5a7cf79029\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-pp577" Jul 23 19:28:57.301508 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:28:57.301485 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8b25k\" (UniqueName: \"kubernetes.io/projected/01866086-e44f-4c2d-9f3d-5b5a7cf79029-kube-api-access-8b25k\") pod \"maas-default-gateway-openshift-default-56654dcf7-pp577\" (UID: \"01866086-e44f-4c2d-9f3d-5b5a7cf79029\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-pp577" Jul 23 19:28:57.301674 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:28:57.301655 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/01866086-e44f-4c2d-9f3d-5b5a7cf79029-istio-token\") pod \"maas-default-gateway-openshift-default-56654dcf7-pp577\" (UID: \"01866086-e44f-4c2d-9f3d-5b5a7cf79029\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-pp577" Jul 23 19:28:57.461848 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:28:57.461826 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-pp577" Jul 23 19:28:57.597044 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:28:57.596410 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-pp577"] Jul 23 19:28:57.601401 ip-10-0-129-2 kubenswrapper[2564]: W0723 19:28:57.601367 2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod01866086_e44f_4c2d_9f3d_5b5a7cf79029.slice/crio-f41f149588a5a7129f57ba4b7c1e8ede5163c67c2f7588d7657d218b8647e818 WatchSource:0}: Error finding container f41f149588a5a7129f57ba4b7c1e8ede5163c67c2f7588d7657d218b8647e818: Status 404 returned error can't find the container with id f41f149588a5a7129f57ba4b7c1e8ede5163c67c2f7588d7657d218b8647e818 Jul 23 19:28:58.158001 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:28:58.157975 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-pp577" event={"ID":"01866086-e44f-4c2d-9f3d-5b5a7cf79029","Type":"ContainerStarted","Data":"f41f149588a5a7129f57ba4b7c1e8ede5163c67c2f7588d7657d218b8647e818"} Jul 23 19:29:00.551057 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:29:00.551025 2564 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892168Ki","pods":"250"} Jul 23 19:29:00.551301 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:29:00.551092 2564 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892168Ki","pods":"250"} Jul 23 19:29:00.551301 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:29:00.551119 2564 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892168Ki","pods":"250"} Jul 23 19:29:01.174861 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:29:01.174827 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-pp577" event={"ID":"01866086-e44f-4c2d-9f3d-5b5a7cf79029","Type":"ContainerStarted","Data":"ed1af698aa02984002f965f71c8825a3fe93d809673d29691dbc9ed14627a285"} Jul 23 19:29:01.196819 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:29:01.196779 2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-pp577" podStartSLOduration=1.249367972 podStartE2EDuration="4.196762458s" totalImagesPullingTime="2.947394486s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-23 19:28:57 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-23 19:29:01.194555555 +0000 UTC m=+567.046331802" watchObservedRunningTime="2026-07-23 19:29:01.196762458 +0000 UTC m=+567.048538703" Jul 23 19:29:01.462852 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:29:01.462777 2564 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-pp577" Jul 23 19:29:01.467415 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:29:01.467392 2564 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-pp577" Jul 23 19:29:02.179534 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:29:02.179505 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-pp577" Jul 23 19:29:02.180226 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:29:02.180208 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-pp577" Jul 23 19:29:02.300151 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:29:02.300122 2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/limitador-limitador-64c94fd64f-x5bxt"] Jul 23 19:29:02.303674 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:29:02.303658 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-64c94fd64f-x5bxt" Jul 23 19:29:02.306153 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:29:02.306126 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"default-dockercfg-mcgx7\"" Jul 23 19:29:02.306277 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:29:02.306226 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kube-root-ca.crt\"" Jul 23 19:29:02.306490 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:29:02.306461 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"limitador-limits-config-limitador\"" Jul 23 19:29:02.306593 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:29:02.306505 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"openshift-service-ca.crt\"" Jul 23 19:29:02.313370 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:29:02.313348 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-64c94fd64f-x5bxt"] Jul 23 19:29:02.329004 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:29:02.328983 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/d2112873-68ce-4b17-b641-daf202b55aff-config-file\") pod \"limitador-limitador-64c94fd64f-x5bxt\" (UID: \"d2112873-68ce-4b17-b641-daf202b55aff\") " pod="kuadrant-system/limitador-limitador-64c94fd64f-x5bxt" Jul 23 19:29:02.329111 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:29:02.329092 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hptsx\" (UniqueName: \"kubernetes.io/projected/d2112873-68ce-4b17-b641-daf202b55aff-kube-api-access-hptsx\") pod \"limitador-limitador-64c94fd64f-x5bxt\" (UID: \"d2112873-68ce-4b17-b641-daf202b55aff\") " pod="kuadrant-system/limitador-limitador-64c94fd64f-x5bxt" Jul 23 19:29:02.420099 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:29:02.420066 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-64c94fd64f-x5bxt"] Jul 23 19:29:02.429846 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:29:02.429787 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hptsx\" (UniqueName: \"kubernetes.io/projected/d2112873-68ce-4b17-b641-daf202b55aff-kube-api-access-hptsx\") pod \"limitador-limitador-64c94fd64f-x5bxt\" (UID: \"d2112873-68ce-4b17-b641-daf202b55aff\") " pod="kuadrant-system/limitador-limitador-64c94fd64f-x5bxt" Jul 23 19:29:02.429846 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:29:02.429828 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/d2112873-68ce-4b17-b641-daf202b55aff-config-file\") pod \"limitador-limitador-64c94fd64f-x5bxt\" (UID: \"d2112873-68ce-4b17-b641-daf202b55aff\") " pod="kuadrant-system/limitador-limitador-64c94fd64f-x5bxt" Jul 23 19:29:02.430373 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:29:02.430356 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/d2112873-68ce-4b17-b641-daf202b55aff-config-file\") pod \"limitador-limitador-64c94fd64f-x5bxt\" (UID: \"d2112873-68ce-4b17-b641-daf202b55aff\") " pod="kuadrant-system/limitador-limitador-64c94fd64f-x5bxt" Jul 23 19:29:02.449831 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:29:02.449808 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-hptsx\" (UniqueName: \"kubernetes.io/projected/d2112873-68ce-4b17-b641-daf202b55aff-kube-api-access-hptsx\") pod \"limitador-limitador-64c94fd64f-x5bxt\" (UID: \"d2112873-68ce-4b17-b641-daf202b55aff\") " pod="kuadrant-system/limitador-limitador-64c94fd64f-x5bxt" Jul 23 19:29:02.614092 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:29:02.614066 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-64c94fd64f-x5bxt" Jul 23 19:29:02.713491 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:29:02.713457 2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-2dgmk"] Jul 23 19:29:02.718553 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:29:02.718272 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-54d6d988d5-2dgmk" Jul 23 19:29:02.734839 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:29:02.734816 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-2dgmk"] Jul 23 19:29:02.768414 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:29:02.768394 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-64c94fd64f-x5bxt"] Jul 23 19:29:02.770473 ip-10-0-129-2 kubenswrapper[2564]: W0723 19:29:02.770451 2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd2112873_68ce_4b17_b641_daf202b55aff.slice/crio-ea376d03b0653718f0de08b823543c0ab009f197ce95195b22b310b743bb2cde WatchSource:0}: Error finding container ea376d03b0653718f0de08b823543c0ab009f197ce95195b22b310b743bb2cde: Status 404 returned error can't find the container with id ea376d03b0653718f0de08b823543c0ab009f197ce95195b22b310b743bb2cde Jul 23 19:29:02.780144 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:29:02.780125 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-2dgmk"] Jul 23 19:29:02.833879 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:29:02.833860 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4l27c\" (UniqueName: \"kubernetes.io/projected/ebafc262-cd0e-4e53-b4ce-def03413139e-kube-api-access-4l27c\") pod \"limitador-limitador-54d6d988d5-2dgmk\" (UID: \"ebafc262-cd0e-4e53-b4ce-def03413139e\") " pod="kuadrant-system/limitador-limitador-54d6d988d5-2dgmk" Jul 23 19:29:02.833978 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:29:02.833900 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/ebafc262-cd0e-4e53-b4ce-def03413139e-config-file\") pod \"limitador-limitador-54d6d988d5-2dgmk\" (UID: \"ebafc262-cd0e-4e53-b4ce-def03413139e\") " pod="kuadrant-system/limitador-limitador-54d6d988d5-2dgmk" Jul 23 19:29:02.934585 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:29:02.934562 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4l27c\" (UniqueName: \"kubernetes.io/projected/ebafc262-cd0e-4e53-b4ce-def03413139e-kube-api-access-4l27c\") pod \"limitador-limitador-54d6d988d5-2dgmk\" (UID: \"ebafc262-cd0e-4e53-b4ce-def03413139e\") " pod="kuadrant-system/limitador-limitador-54d6d988d5-2dgmk" Jul 23 19:29:02.934674 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:29:02.934610 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/ebafc262-cd0e-4e53-b4ce-def03413139e-config-file\") pod \"limitador-limitador-54d6d988d5-2dgmk\" (UID: \"ebafc262-cd0e-4e53-b4ce-def03413139e\") " pod="kuadrant-system/limitador-limitador-54d6d988d5-2dgmk" Jul 23 19:29:02.935127 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:29:02.935110 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/ebafc262-cd0e-4e53-b4ce-def03413139e-config-file\") pod \"limitador-limitador-54d6d988d5-2dgmk\" (UID: \"ebafc262-cd0e-4e53-b4ce-def03413139e\") " pod="kuadrant-system/limitador-limitador-54d6d988d5-2dgmk" Jul 23 19:29:02.944041 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:29:02.944015 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4l27c\" (UniqueName: \"kubernetes.io/projected/ebafc262-cd0e-4e53-b4ce-def03413139e-kube-api-access-4l27c\") pod \"limitador-limitador-54d6d988d5-2dgmk\" (UID: \"ebafc262-cd0e-4e53-b4ce-def03413139e\") " pod="kuadrant-system/limitador-limitador-54d6d988d5-2dgmk" Jul 23 19:29:03.029146 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:29:03.029093 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-54d6d988d5-2dgmk" Jul 23 19:29:03.161524 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:29:03.161465 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-2dgmk"] Jul 23 19:29:03.164093 ip-10-0-129-2 kubenswrapper[2564]: W0723 19:29:03.164060 2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podebafc262_cd0e_4e53_b4ce_def03413139e.slice/crio-16b3d61a37c3a554269d319b18c34de89998ebf09b4805af99fc0b6f5e13d3f2 WatchSource:0}: Error finding container 16b3d61a37c3a554269d319b18c34de89998ebf09b4805af99fc0b6f5e13d3f2: Status 404 returned error can't find the container with id 16b3d61a37c3a554269d319b18c34de89998ebf09b4805af99fc0b6f5e13d3f2 Jul 23 19:29:03.186311 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:29:03.186286 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-54d6d988d5-2dgmk" event={"ID":"ebafc262-cd0e-4e53-b4ce-def03413139e","Type":"ContainerStarted","Data":"16b3d61a37c3a554269d319b18c34de89998ebf09b4805af99fc0b6f5e13d3f2"} Jul 23 19:29:03.187524 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:29:03.187506 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-64c94fd64f-x5bxt" event={"ID":"d2112873-68ce-4b17-b641-daf202b55aff","Type":"ContainerStarted","Data":"ea376d03b0653718f0de08b823543c0ab009f197ce95195b22b310b743bb2cde"} Jul 23 19:29:06.207775 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:29:06.207736 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-54d6d988d5-2dgmk" event={"ID":"ebafc262-cd0e-4e53-b4ce-def03413139e","Type":"ContainerStarted","Data":"04378dca38a59be8ef6eeedd18a5668660a3cbc2b5aab6ed695829a17fe4b2b9"} Jul 23 19:29:06.208156 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:29:06.207894 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/limitador-limitador-54d6d988d5-2dgmk" Jul 23 19:29:06.209254 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:29:06.209234 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-64c94fd64f-x5bxt" event={"ID":"d2112873-68ce-4b17-b641-daf202b55aff","Type":"ContainerStarted","Data":"0866912db9ecf42b25aaebb29d82b93aa7969c801f7a2257ee7094c641e8575c"} Jul 23 19:29:06.209368 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:29:06.209283 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/limitador-limitador-64c94fd64f-x5bxt" Jul 23 19:29:06.226964 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:29:06.226922 2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/limitador-limitador-54d6d988d5-2dgmk" podStartSLOduration=1.736321152 podStartE2EDuration="4.22691249s" totalImagesPullingTime="2.490591338s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-23 19:29:02 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-23 19:29:06.225823553 +0000 UTC m=+572.077599798" watchObservedRunningTime="2026-07-23 19:29:06.22691249 +0000 UTC m=+572.078688735" Jul 23 19:29:06.246718 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:29:06.246676 2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/limitador-limitador-64c94fd64f-x5bxt" podStartSLOduration=1.355712033 podStartE2EDuration="4.246665412s" totalImagesPullingTime="2.890953379s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-23 19:29:02 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-23 19:29:06.245586914 +0000 UTC m=+572.097363159" watchObservedRunningTime="2026-07-23 19:29:06.246665412 +0000 UTC m=+572.098441657" Jul 23 19:29:17.214761 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:29:17.214731 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/limitador-limitador-64c94fd64f-x5bxt" Jul 23 19:29:17.215140 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:29:17.214790 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/limitador-limitador-54d6d988d5-2dgmk" Jul 23 19:29:17.290409 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:29:17.290381 2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/limitador-limitador-64c94fd64f-x5bxt"] Jul 23 19:29:17.290554 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:29:17.290524 2564 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/limitador-limitador-64c94fd64f-x5bxt" podUID="d2112873-68ce-4b17-b641-daf202b55aff" containerName="limitador" containerID="cri-o://0866912db9ecf42b25aaebb29d82b93aa7969c801f7a2257ee7094c641e8575c" gracePeriod=30 Jul 23 19:29:17.834917 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:29:17.834898 2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-64c94fd64f-x5bxt" Jul 23 19:29:17.929581 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:29:17.929558 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/d2112873-68ce-4b17-b641-daf202b55aff-config-file\") pod \"d2112873-68ce-4b17-b641-daf202b55aff\" (UID: \"d2112873-68ce-4b17-b641-daf202b55aff\") " Jul 23 19:29:17.929690 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:29:17.929600 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-hptsx\" (UniqueName: \"kubernetes.io/projected/d2112873-68ce-4b17-b641-daf202b55aff-kube-api-access-hptsx\") pod \"d2112873-68ce-4b17-b641-daf202b55aff\" (UID: \"d2112873-68ce-4b17-b641-daf202b55aff\") " Jul 23 19:29:17.929942 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:29:17.929917 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d2112873-68ce-4b17-b641-daf202b55aff-config-file" (OuterVolumeSpecName: "config-file") pod "d2112873-68ce-4b17-b641-daf202b55aff" (UID: "d2112873-68ce-4b17-b641-daf202b55aff"). InnerVolumeSpecName "config-file". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 23 19:29:17.931632 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:29:17.931610 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d2112873-68ce-4b17-b641-daf202b55aff-kube-api-access-hptsx" (OuterVolumeSpecName: "kube-api-access-hptsx") pod "d2112873-68ce-4b17-b641-daf202b55aff" (UID: "d2112873-68ce-4b17-b641-daf202b55aff"). InnerVolumeSpecName "kube-api-access-hptsx". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 23 19:29:18.030945 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:29:18.030922 2564 reconciler_common.go:299] "Volume detached for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/d2112873-68ce-4b17-b641-daf202b55aff-config-file\") on node \"ip-10-0-129-2.ec2.internal\" DevicePath \"\"" Jul 23 19:29:18.030945 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:29:18.030943 2564 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hptsx\" (UniqueName: \"kubernetes.io/projected/d2112873-68ce-4b17-b641-daf202b55aff-kube-api-access-hptsx\") on node \"ip-10-0-129-2.ec2.internal\" DevicePath \"\"" Jul 23 19:29:18.281769 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:29:18.281715 2564 generic.go:356] "Generic (PLEG): container finished" podID="d2112873-68ce-4b17-b641-daf202b55aff" containerID="0866912db9ecf42b25aaebb29d82b93aa7969c801f7a2257ee7094c641e8575c" exitCode=0 Jul 23 19:29:18.281769 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:29:18.281761 2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-64c94fd64f-x5bxt" Jul 23 19:29:18.282113 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:29:18.281798 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-64c94fd64f-x5bxt" event={"ID":"d2112873-68ce-4b17-b641-daf202b55aff","Type":"ContainerDied","Data":"0866912db9ecf42b25aaebb29d82b93aa7969c801f7a2257ee7094c641e8575c"} Jul 23 19:29:18.282113 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:29:18.281842 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-64c94fd64f-x5bxt" event={"ID":"d2112873-68ce-4b17-b641-daf202b55aff","Type":"ContainerDied","Data":"ea376d03b0653718f0de08b823543c0ab009f197ce95195b22b310b743bb2cde"} Jul 23 19:29:18.282113 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:29:18.281863 2564 scope.go:117] "RemoveContainer" containerID="0866912db9ecf42b25aaebb29d82b93aa7969c801f7a2257ee7094c641e8575c" Jul 23 19:29:18.291218 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:29:18.291084 2564 scope.go:117] "RemoveContainer" containerID="0866912db9ecf42b25aaebb29d82b93aa7969c801f7a2257ee7094c641e8575c" Jul 23 19:29:18.291351 ip-10-0-129-2 kubenswrapper[2564]: E0723 19:29:18.291315 2564 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0866912db9ecf42b25aaebb29d82b93aa7969c801f7a2257ee7094c641e8575c\": container with ID starting with 0866912db9ecf42b25aaebb29d82b93aa7969c801f7a2257ee7094c641e8575c not found: ID does not exist" containerID="0866912db9ecf42b25aaebb29d82b93aa7969c801f7a2257ee7094c641e8575c" Jul 23 19:29:18.291440 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:29:18.291361 2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0866912db9ecf42b25aaebb29d82b93aa7969c801f7a2257ee7094c641e8575c"} err="failed to get container status \"0866912db9ecf42b25aaebb29d82b93aa7969c801f7a2257ee7094c641e8575c\": rpc error: code = NotFound desc = could not find container \"0866912db9ecf42b25aaebb29d82b93aa7969c801f7a2257ee7094c641e8575c\": container with ID starting with 0866912db9ecf42b25aaebb29d82b93aa7969c801f7a2257ee7094c641e8575c not found: ID does not exist" Jul 23 19:29:18.303678 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:29:18.303659 2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/limitador-limitador-64c94fd64f-x5bxt"] Jul 23 19:29:18.307628 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:29:18.307611 2564 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/limitador-limitador-64c94fd64f-x5bxt"] Jul 23 19:29:18.657367 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:29:18.657289 2564 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="d2112873-68ce-4b17-b641-daf202b55aff" path="/var/lib/kubelet/pods/d2112873-68ce-4b17-b641-daf202b55aff/volumes" Jul 23 19:29:32.142924 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:29:32.142897 2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-5c6965cf8f-ccqwf"] Jul 23 19:29:32.143400 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:29:32.143383 2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d2112873-68ce-4b17-b641-daf202b55aff" containerName="limitador" Jul 23 19:29:32.143478 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:29:32.143402 2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="d2112873-68ce-4b17-b641-daf202b55aff" containerName="limitador" Jul 23 19:29:32.143540 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:29:32.143526 2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="d2112873-68ce-4b17-b641-daf202b55aff" containerName="limitador" Jul 23 19:29:32.146614 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:29:32.146596 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-5c6965cf8f-ccqwf" Jul 23 19:29:32.149569 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:29:32.149550 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-authorino-dockercfg-rwdhb\"" Jul 23 19:29:32.149667 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:29:32.149574 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-server-cert\"" Jul 23 19:29:32.154623 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:29:32.154603 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-5c6965cf8f-ccqwf"] Jul 23 19:29:32.230845 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:29:32.230814 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8lqdr\" (UniqueName: \"kubernetes.io/projected/97fdc4d6-7a0a-41ed-a6a9-684aa3fae641-kube-api-access-8lqdr\") pod \"authorino-5c6965cf8f-ccqwf\" (UID: \"97fdc4d6-7a0a-41ed-a6a9-684aa3fae641\") " pod="kuadrant-system/authorino-5c6965cf8f-ccqwf" Jul 23 19:29:32.230942 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:29:32.230915 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/97fdc4d6-7a0a-41ed-a6a9-684aa3fae641-tls-cert\") pod \"authorino-5c6965cf8f-ccqwf\" (UID: \"97fdc4d6-7a0a-41ed-a6a9-684aa3fae641\") " pod="kuadrant-system/authorino-5c6965cf8f-ccqwf" Jul 23 19:29:32.331622 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:29:32.331599 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/97fdc4d6-7a0a-41ed-a6a9-684aa3fae641-tls-cert\") pod \"authorino-5c6965cf8f-ccqwf\" (UID: \"97fdc4d6-7a0a-41ed-a6a9-684aa3fae641\") " pod="kuadrant-system/authorino-5c6965cf8f-ccqwf" Jul 23 19:29:32.331719 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:29:32.331629 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8lqdr\" (UniqueName: \"kubernetes.io/projected/97fdc4d6-7a0a-41ed-a6a9-684aa3fae641-kube-api-access-8lqdr\") pod \"authorino-5c6965cf8f-ccqwf\" (UID: \"97fdc4d6-7a0a-41ed-a6a9-684aa3fae641\") " pod="kuadrant-system/authorino-5c6965cf8f-ccqwf" Jul 23 19:29:32.334036 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:29:32.334018 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/97fdc4d6-7a0a-41ed-a6a9-684aa3fae641-tls-cert\") pod \"authorino-5c6965cf8f-ccqwf\" (UID: \"97fdc4d6-7a0a-41ed-a6a9-684aa3fae641\") " pod="kuadrant-system/authorino-5c6965cf8f-ccqwf" Jul 23 19:29:32.342912 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:29:32.342895 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8lqdr\" (UniqueName: \"kubernetes.io/projected/97fdc4d6-7a0a-41ed-a6a9-684aa3fae641-kube-api-access-8lqdr\") pod \"authorino-5c6965cf8f-ccqwf\" (UID: \"97fdc4d6-7a0a-41ed-a6a9-684aa3fae641\") " pod="kuadrant-system/authorino-5c6965cf8f-ccqwf" Jul 23 19:29:32.456439 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:29:32.456417 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-5c6965cf8f-ccqwf" Jul 23 19:29:32.583045 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:29:32.582981 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-5c6965cf8f-ccqwf"] Jul 23 19:29:32.585418 ip-10-0-129-2 kubenswrapper[2564]: W0723 19:29:32.585392 2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod97fdc4d6_7a0a_41ed_a6a9_684aa3fae641.slice/crio-7b6314e18bb710bd06da5ef5abc822ab26666c9695097701d7315733a1be6e85 WatchSource:0}: Error finding container 7b6314e18bb710bd06da5ef5abc822ab26666c9695097701d7315733a1be6e85: Status 404 returned error can't find the container with id 7b6314e18bb710bd06da5ef5abc822ab26666c9695097701d7315733a1be6e85 Jul 23 19:29:33.360630 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:29:33.360570 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-5c6965cf8f-ccqwf" event={"ID":"97fdc4d6-7a0a-41ed-a6a9-684aa3fae641","Type":"ContainerStarted","Data":"7b6314e18bb710bd06da5ef5abc822ab26666c9695097701d7315733a1be6e85"} Jul 23 19:29:35.366265 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:29:35.366242 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-565b7b569d-z5wf4_ded5b3af-c74c-4d5a-aa77-62fe72d1dd29/console-operator/1.log" Jul 23 19:29:35.366665 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:29:35.366242 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-565b7b569d-z5wf4_ded5b3af-c74c-4d5a-aa77-62fe72d1dd29/console-operator/1.log" Jul 23 19:29:35.373092 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:29:35.373072 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-8dtlh_195ecf89-13ae-4311-bc72-fcdf78a5c4d6/ovn-acl-logging/0.log" Jul 23 19:29:35.373495 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:29:35.373476 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-8dtlh_195ecf89-13ae-4311-bc72-fcdf78a5c4d6/ovn-acl-logging/0.log" Jul 23 19:29:36.377279 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:29:36.377253 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-5c6965cf8f-ccqwf" event={"ID":"97fdc4d6-7a0a-41ed-a6a9-684aa3fae641","Type":"ContainerStarted","Data":"9bfeb10ba604b785e80a098e430f87972f60c01816eb05f45ac19fc84bea67c1"} Jul 23 19:29:36.394965 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:29:36.394924 2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/authorino-5c6965cf8f-ccqwf" podStartSLOduration=1.584055483 podStartE2EDuration="4.394910897s" totalImagesPullingTime="2.810855414s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-23 19:29:32 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-23 19:29:36.393241514 +0000 UTC m=+602.245017761" watchObservedRunningTime="2026-07-23 19:29:36.394910897 +0000 UTC m=+602.246687194" Jul 23 19:30:16.607468 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:30:16.607435 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-2dgmk"] Jul 23 19:30:16.754044 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:30:16.754007 2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-5bf47cd988-rwnsj"] Jul 23 19:30:16.762765 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:30:16.762736 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-5bf47cd988-rwnsj" Jul 23 19:30:16.766379 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:30:16.766359 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-dockercfg-5kbrn\"" Jul 23 19:30:16.766548 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:30:16.766530 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins\"" Jul 23 19:30:16.778965 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:30:16.778943 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-5bf47cd988-rwnsj"] Jul 23 19:30:16.854985 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:30:16.854960 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-w2m4f\" (UniqueName: \"kubernetes.io/projected/45506cd4-c775-4185-888e-607405dd6675-kube-api-access-w2m4f\") pod \"payload-processing-5bf47cd988-rwnsj\" (UID: \"45506cd4-c775-4185-888e-607405dd6675\") " pod="openshift-ingress/payload-processing-5bf47cd988-rwnsj" Jul 23 19:30:16.855112 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:30:16.854999 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/45506cd4-c775-4185-888e-607405dd6675-plugins-config-volume\") pod \"payload-processing-5bf47cd988-rwnsj\" (UID: \"45506cd4-c775-4185-888e-607405dd6675\") " pod="openshift-ingress/payload-processing-5bf47cd988-rwnsj" Jul 23 19:30:16.955451 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:30:16.955428 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w2m4f\" (UniqueName: \"kubernetes.io/projected/45506cd4-c775-4185-888e-607405dd6675-kube-api-access-w2m4f\") pod \"payload-processing-5bf47cd988-rwnsj\" (UID: \"45506cd4-c775-4185-888e-607405dd6675\") " pod="openshift-ingress/payload-processing-5bf47cd988-rwnsj" Jul 23 19:30:16.955588 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:30:16.955466 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/45506cd4-c775-4185-888e-607405dd6675-plugins-config-volume\") pod \"payload-processing-5bf47cd988-rwnsj\" (UID: \"45506cd4-c775-4185-888e-607405dd6675\") " pod="openshift-ingress/payload-processing-5bf47cd988-rwnsj" Jul 23 19:30:16.956181 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:30:16.956161 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/45506cd4-c775-4185-888e-607405dd6675-plugins-config-volume\") pod \"payload-processing-5bf47cd988-rwnsj\" (UID: \"45506cd4-c775-4185-888e-607405dd6675\") " pod="openshift-ingress/payload-processing-5bf47cd988-rwnsj" Jul 23 19:30:16.964112 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:30:16.964086 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-w2m4f\" (UniqueName: \"kubernetes.io/projected/45506cd4-c775-4185-888e-607405dd6675-kube-api-access-w2m4f\") pod \"payload-processing-5bf47cd988-rwnsj\" (UID: \"45506cd4-c775-4185-888e-607405dd6675\") " pod="openshift-ingress/payload-processing-5bf47cd988-rwnsj" Jul 23 19:30:17.073706 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:30:17.073674 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-5bf47cd988-rwnsj" Jul 23 19:30:17.255398 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:30:17.255376 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-5bf47cd988-rwnsj"] Jul 23 19:30:17.274346 ip-10-0-129-2 kubenswrapper[2564]: W0723 19:30:17.269597 2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod45506cd4_c775_4185_888e_607405dd6675.slice/crio-619ddb3ae217aed7752ad626d7fab396ca80657994aa5d2f7181239689072159 WatchSource:0}: Error finding container 619ddb3ae217aed7752ad626d7fab396ca80657994aa5d2f7181239689072159: Status 404 returned error can't find the container with id 619ddb3ae217aed7752ad626d7fab396ca80657994aa5d2f7181239689072159 Jul 23 19:30:17.285003 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:30:17.284981 2564 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Jul 23 19:30:17.590943 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:30:17.590850 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-5bf47cd988-rwnsj" event={"ID":"45506cd4-c775-4185-888e-607405dd6675","Type":"ContainerStarted","Data":"619ddb3ae217aed7752ad626d7fab396ca80657994aa5d2f7181239689072159"} Jul 23 19:30:19.602037 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:30:19.602005 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-5bf47cd988-rwnsj" event={"ID":"45506cd4-c775-4185-888e-607405dd6675","Type":"ContainerStarted","Data":"a4b57c63d1d9b4ee8673fd17377a50b0e4c6c9b814b95a8907596fd7a603bd16"} Jul 23 19:30:19.602611 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:30:19.602596 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-5bf47cd988-rwnsj" Jul 23 19:30:19.630348 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:30:19.630295 2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-5bf47cd988-rwnsj" podStartSLOduration=1.883415601 podStartE2EDuration="3.63028193s" totalImagesPullingTime="1.746866329s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-23 19:30:16 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-23 19:30:19.628198791 +0000 UTC m=+645.479975038" watchObservedRunningTime="2026-07-23 19:30:19.63028193 +0000 UTC m=+645.482058181" Jul 23 19:30:24.629281 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:30:24.629255 2564 generic.go:356] "Generic (PLEG): container finished" podID="01866086-e44f-4c2d-9f3d-5b5a7cf79029" containerID="ed1af698aa02984002f965f71c8825a3fe93d809673d29691dbc9ed14627a285" exitCode=0 Jul 23 19:30:24.629672 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:30:24.629344 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-pp577" event={"ID":"01866086-e44f-4c2d-9f3d-5b5a7cf79029","Type":"ContainerDied","Data":"ed1af698aa02984002f965f71c8825a3fe93d809673d29691dbc9ed14627a285"} Jul 23 19:30:24.629716 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:30:24.629703 2564 scope.go:117] "RemoveContainer" containerID="ed1af698aa02984002f965f71c8825a3fe93d809673d29691dbc9ed14627a285" Jul 23 19:30:24.630674 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:30:24.630647 2564 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892168Ki","pods":"250"} Jul 23 19:30:24.630729 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:30:24.630710 2564 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892168Ki","pods":"250"} Jul 23 19:30:24.630769 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:30:24.630740 2564 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892168Ki","pods":"250"} Jul 23 19:30:25.636980 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:30:25.636948 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-pp577" event={"ID":"01866086-e44f-4c2d-9f3d-5b5a7cf79029","Type":"ContainerStarted","Data":"9e624f3c1d10773ead9006d059995ecfc77ec51bc2d61d144975bea495f79851"} Jul 23 19:30:26.463047 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:30:26.463017 2564 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-pp577" Jul 23 19:30:26.463415 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:30:26.463392 2564 patch_prober.go:28] interesting pod/maas-default-gateway-openshift-default-56654dcf7-pp577 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.42:15021/healthz/ready\": dial tcp 10.132.0.42:15021: connect: connection refused" start-of-body= Jul 23 19:30:26.463602 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:30:26.463578 2564 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-pp577" podUID="01866086-e44f-4c2d-9f3d-5b5a7cf79029" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.42:15021/healthz/ready\": dial tcp 10.132.0.42:15021: connect: connection refused" Jul 23 19:30:27.462811 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:30:27.462362 2564 patch_prober.go:28] interesting pod/maas-default-gateway-openshift-default-56654dcf7-pp577 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.42:15021/healthz/ready\": dial tcp 10.132.0.42:15021: connect: connection refused" start-of-body= Jul 23 19:30:27.462811 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:30:27.462439 2564 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-pp577" podUID="01866086-e44f-4c2d-9f3d-5b5a7cf79029" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.42:15021/healthz/ready\": dial tcp 10.132.0.42:15021: connect: connection refused" Jul 23 19:30:28.463020 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:30:28.462986 2564 patch_prober.go:28] interesting pod/maas-default-gateway-openshift-default-56654dcf7-pp577 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.42:15021/healthz/ready\": dial tcp 10.132.0.42:15021: connect: connection refused" start-of-body= Jul 23 19:30:28.463585 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:30:28.463557 2564 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-pp577" podUID="01866086-e44f-4c2d-9f3d-5b5a7cf79029" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.42:15021/healthz/ready\": dial tcp 10.132.0.42:15021: connect: connection refused" Jul 23 19:30:29.465816 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:30:29.465783 2564 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-pp577" Jul 23 19:30:29.466199 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:30:29.466145 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-pp577" Jul 23 19:30:29.466835 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:30:29.466816 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-pp577" Jul 23 19:30:31.614474 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:30:31.614449 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-5bf47cd988-rwnsj" Jul 23 19:30:47.124354 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:30:47.124295 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-2dgmk"] Jul 23 19:30:52.936950 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:30:52.936919 2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/e2e-distinct-simulated-kserve-59975d69f9-7w7zq"] Jul 23 19:30:52.941565 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:30:52.941538 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-7w7zq" Jul 23 19:30:52.945266 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:30:52.945244 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-distinct-simulated-kserve-self-signed-certs\"" Jul 23 19:30:52.945402 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:30:52.945278 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"default-dockercfg-tbbll\"" Jul 23 19:30:52.945477 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:30:52.945451 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"kube-root-ca.crt\"" Jul 23 19:30:52.945539 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:30:52.945489 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"openshift-service-ca.crt\"" Jul 23 19:30:52.946923 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:30:52.946868 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f2240a8a-5e11-48fe-97e3-874df0e79f7e-dshm\") pod \"e2e-distinct-simulated-kserve-59975d69f9-7w7zq\" (UID: \"f2240a8a-5e11-48fe-97e3-874df0e79f7e\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-7w7zq" Jul 23 19:30:52.946923 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:30:52.946901 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f2240a8a-5e11-48fe-97e3-874df0e79f7e-model-cache\") pod \"e2e-distinct-simulated-kserve-59975d69f9-7w7zq\" (UID: \"f2240a8a-5e11-48fe-97e3-874df0e79f7e\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-7w7zq" Jul 23 19:30:52.947076 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:30:52.946966 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f2240a8a-5e11-48fe-97e3-874df0e79f7e-home\") pod \"e2e-distinct-simulated-kserve-59975d69f9-7w7zq\" (UID: \"f2240a8a-5e11-48fe-97e3-874df0e79f7e\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-7w7zq" Jul 23 19:30:52.947076 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:30:52.947000 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f2240a8a-5e11-48fe-97e3-874df0e79f7e-tls-certs\") pod \"e2e-distinct-simulated-kserve-59975d69f9-7w7zq\" (UID: \"f2240a8a-5e11-48fe-97e3-874df0e79f7e\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-7w7zq" Jul 23 19:30:52.947177 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:30:52.947079 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fvtv4\" (UniqueName: \"kubernetes.io/projected/f2240a8a-5e11-48fe-97e3-874df0e79f7e-kube-api-access-fvtv4\") pod \"e2e-distinct-simulated-kserve-59975d69f9-7w7zq\" (UID: \"f2240a8a-5e11-48fe-97e3-874df0e79f7e\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-7w7zq" Jul 23 19:30:52.947177 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:30:52.947146 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f2240a8a-5e11-48fe-97e3-874df0e79f7e-tmp-dir\") pod \"e2e-distinct-simulated-kserve-59975d69f9-7w7zq\" (UID: \"f2240a8a-5e11-48fe-97e3-874df0e79f7e\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-7w7zq" Jul 23 19:30:52.953812 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:30:52.953792 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-distinct-simulated-kserve-59975d69f9-7w7zq"] Jul 23 19:30:53.048125 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:30:53.048099 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fvtv4\" (UniqueName: \"kubernetes.io/projected/f2240a8a-5e11-48fe-97e3-874df0e79f7e-kube-api-access-fvtv4\") pod \"e2e-distinct-simulated-kserve-59975d69f9-7w7zq\" (UID: \"f2240a8a-5e11-48fe-97e3-874df0e79f7e\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-7w7zq" Jul 23 19:30:53.048125 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:30:53.048129 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f2240a8a-5e11-48fe-97e3-874df0e79f7e-tmp-dir\") pod \"e2e-distinct-simulated-kserve-59975d69f9-7w7zq\" (UID: \"f2240a8a-5e11-48fe-97e3-874df0e79f7e\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-7w7zq" Jul 23 19:30:53.048373 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:30:53.048174 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f2240a8a-5e11-48fe-97e3-874df0e79f7e-dshm\") pod \"e2e-distinct-simulated-kserve-59975d69f9-7w7zq\" (UID: \"f2240a8a-5e11-48fe-97e3-874df0e79f7e\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-7w7zq" Jul 23 19:30:53.048373 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:30:53.048189 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f2240a8a-5e11-48fe-97e3-874df0e79f7e-model-cache\") pod \"e2e-distinct-simulated-kserve-59975d69f9-7w7zq\" (UID: \"f2240a8a-5e11-48fe-97e3-874df0e79f7e\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-7w7zq" Jul 23 19:30:53.048373 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:30:53.048212 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f2240a8a-5e11-48fe-97e3-874df0e79f7e-home\") pod \"e2e-distinct-simulated-kserve-59975d69f9-7w7zq\" (UID: \"f2240a8a-5e11-48fe-97e3-874df0e79f7e\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-7w7zq" Jul 23 19:30:53.048373 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:30:53.048225 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f2240a8a-5e11-48fe-97e3-874df0e79f7e-tls-certs\") pod \"e2e-distinct-simulated-kserve-59975d69f9-7w7zq\" (UID: \"f2240a8a-5e11-48fe-97e3-874df0e79f7e\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-7w7zq" Jul 23 19:30:53.048689 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:30:53.048652 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f2240a8a-5e11-48fe-97e3-874df0e79f7e-model-cache\") pod \"e2e-distinct-simulated-kserve-59975d69f9-7w7zq\" (UID: \"f2240a8a-5e11-48fe-97e3-874df0e79f7e\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-7w7zq" Jul 23 19:30:53.048689 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:30:53.048670 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f2240a8a-5e11-48fe-97e3-874df0e79f7e-tmp-dir\") pod \"e2e-distinct-simulated-kserve-59975d69f9-7w7zq\" (UID: \"f2240a8a-5e11-48fe-97e3-874df0e79f7e\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-7w7zq" Jul 23 19:30:53.048833 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:30:53.048711 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f2240a8a-5e11-48fe-97e3-874df0e79f7e-home\") pod \"e2e-distinct-simulated-kserve-59975d69f9-7w7zq\" (UID: \"f2240a8a-5e11-48fe-97e3-874df0e79f7e\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-7w7zq" Jul 23 19:30:53.050429 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:30:53.050408 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f2240a8a-5e11-48fe-97e3-874df0e79f7e-dshm\") pod \"e2e-distinct-simulated-kserve-59975d69f9-7w7zq\" (UID: \"f2240a8a-5e11-48fe-97e3-874df0e79f7e\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-7w7zq" Jul 23 19:30:53.050683 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:30:53.050666 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f2240a8a-5e11-48fe-97e3-874df0e79f7e-tls-certs\") pod \"e2e-distinct-simulated-kserve-59975d69f9-7w7zq\" (UID: \"f2240a8a-5e11-48fe-97e3-874df0e79f7e\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-7w7zq" Jul 23 19:30:53.056502 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:30:53.056483 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-fvtv4\" (UniqueName: \"kubernetes.io/projected/f2240a8a-5e11-48fe-97e3-874df0e79f7e-kube-api-access-fvtv4\") pod \"e2e-distinct-simulated-kserve-59975d69f9-7w7zq\" (UID: \"f2240a8a-5e11-48fe-97e3-874df0e79f7e\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-7w7zq" Jul 23 19:30:53.251666 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:30:53.251589 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-7w7zq" Jul 23 19:30:53.388445 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:30:53.388419 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-distinct-simulated-kserve-59975d69f9-7w7zq"] Jul 23 19:30:53.390416 ip-10-0-129-2 kubenswrapper[2564]: W0723 19:30:53.390389 2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf2240a8a_5e11_48fe_97e3_874df0e79f7e.slice/crio-f0387f5d4bb293483368862b47086a3810e8f7a7532b94d562116626fd84532b WatchSource:0}: Error finding container f0387f5d4bb293483368862b47086a3810e8f7a7532b94d562116626fd84532b: Status 404 returned error can't find the container with id f0387f5d4bb293483368862b47086a3810e8f7a7532b94d562116626fd84532b Jul 23 19:30:53.733540 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:30:53.733418 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-2dgmk"] Jul 23 19:30:53.776421 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:30:53.776205 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-7w7zq" event={"ID":"f2240a8a-5e11-48fe-97e3-874df0e79f7e","Type":"ContainerStarted","Data":"f0387f5d4bb293483368862b47086a3810e8f7a7532b94d562116626fd84532b"} Jul 23 19:30:54.783286 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:30:54.783191 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-7w7zq" event={"ID":"f2240a8a-5e11-48fe-97e3-874df0e79f7e","Type":"ContainerStarted","Data":"97c1ef59b6d3f5b61b00d1118f7c351e064fcd561a8796fabd4bc31bde78ca5a"} Jul 23 19:30:54.817746 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:30:54.817700 2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-7w7zq" podStartSLOduration=1.6964549679999998 podStartE2EDuration="2.817685612s" totalImagesPullingTime="1.121230644s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-23 19:30:52 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-23 19:30:54.817147285 +0000 UTC m=+680.668923544" watchObservedRunningTime="2026-07-23 19:30:54.817685612 +0000 UTC m=+680.669461859" Jul 23 19:31:03.252782 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:31:03.252701 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-7w7zq" Jul 23 19:31:03.253255 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:31:03.252797 2564 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-7w7zq" Jul 23 19:31:03.264667 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:31:03.264646 2564 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-7w7zq" Jul 23 19:31:03.841057 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:31:03.841031 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-7w7zq" Jul 23 19:31:05.301741 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:31:05.301708 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-2dgmk"] Jul 23 19:31:08.310577 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:31:08.310542 2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f8wfwr"] Jul 23 19:31:08.314238 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:31:08.314219 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f8wfwr" Jul 23 19:31:08.316521 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:31:08.316504 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-unab60ef4d3a239b5143b412cab04acac3-kserve-self-signed-certs\"" Jul 23 19:31:08.322853 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:31:08.322832 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f8wfwr"] Jul 23 19:31:08.362654 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:31:08.362633 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/dfb3b12c-9e0d-460f-a7ba-66239eaf52e9-tls-certs\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f8wfwr\" (UID: \"dfb3b12c-9e0d-460f-a7ba-66239eaf52e9\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f8wfwr" Jul 23 19:31:08.362751 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:31:08.362678 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/dfb3b12c-9e0d-460f-a7ba-66239eaf52e9-tmp-dir\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f8wfwr\" (UID: \"dfb3b12c-9e0d-460f-a7ba-66239eaf52e9\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f8wfwr" Jul 23 19:31:08.362751 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:31:08.362704 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/dfb3b12c-9e0d-460f-a7ba-66239eaf52e9-home\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f8wfwr\" (UID: \"dfb3b12c-9e0d-460f-a7ba-66239eaf52e9\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f8wfwr" Jul 23 19:31:08.362751 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:31:08.362720 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/dfb3b12c-9e0d-460f-a7ba-66239eaf52e9-model-cache\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f8wfwr\" (UID: \"dfb3b12c-9e0d-460f-a7ba-66239eaf52e9\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f8wfwr" Jul 23 19:31:08.362873 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:31:08.362827 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/dfb3b12c-9e0d-460f-a7ba-66239eaf52e9-dshm\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f8wfwr\" (UID: \"dfb3b12c-9e0d-460f-a7ba-66239eaf52e9\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f8wfwr" Jul 23 19:31:08.362873 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:31:08.362858 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wm8lt\" (UniqueName: \"kubernetes.io/projected/dfb3b12c-9e0d-460f-a7ba-66239eaf52e9-kube-api-access-wm8lt\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f8wfwr\" (UID: \"dfb3b12c-9e0d-460f-a7ba-66239eaf52e9\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f8wfwr" Jul 23 19:31:08.463207 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:31:08.463180 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/dfb3b12c-9e0d-460f-a7ba-66239eaf52e9-tls-certs\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f8wfwr\" (UID: \"dfb3b12c-9e0d-460f-a7ba-66239eaf52e9\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f8wfwr" Jul 23 19:31:08.463297 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:31:08.463264 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/dfb3b12c-9e0d-460f-a7ba-66239eaf52e9-tmp-dir\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f8wfwr\" (UID: \"dfb3b12c-9e0d-460f-a7ba-66239eaf52e9\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f8wfwr" Jul 23 19:31:08.463369 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:31:08.463307 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/dfb3b12c-9e0d-460f-a7ba-66239eaf52e9-home\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f8wfwr\" (UID: \"dfb3b12c-9e0d-460f-a7ba-66239eaf52e9\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f8wfwr" Jul 23 19:31:08.463369 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:31:08.463358 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/dfb3b12c-9e0d-460f-a7ba-66239eaf52e9-model-cache\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f8wfwr\" (UID: \"dfb3b12c-9e0d-460f-a7ba-66239eaf52e9\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f8wfwr" Jul 23 19:31:08.463445 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:31:08.463401 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/dfb3b12c-9e0d-460f-a7ba-66239eaf52e9-dshm\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f8wfwr\" (UID: \"dfb3b12c-9e0d-460f-a7ba-66239eaf52e9\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f8wfwr" Jul 23 19:31:08.463445 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:31:08.463435 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wm8lt\" (UniqueName: \"kubernetes.io/projected/dfb3b12c-9e0d-460f-a7ba-66239eaf52e9-kube-api-access-wm8lt\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f8wfwr\" (UID: \"dfb3b12c-9e0d-460f-a7ba-66239eaf52e9\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f8wfwr" Jul 23 19:31:08.463714 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:31:08.463691 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/dfb3b12c-9e0d-460f-a7ba-66239eaf52e9-tmp-dir\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f8wfwr\" (UID: \"dfb3b12c-9e0d-460f-a7ba-66239eaf52e9\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f8wfwr" Jul 23 19:31:08.463780 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:31:08.463696 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/dfb3b12c-9e0d-460f-a7ba-66239eaf52e9-home\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f8wfwr\" (UID: \"dfb3b12c-9e0d-460f-a7ba-66239eaf52e9\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f8wfwr" Jul 23 19:31:08.463780 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:31:08.463769 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/dfb3b12c-9e0d-460f-a7ba-66239eaf52e9-model-cache\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f8wfwr\" (UID: \"dfb3b12c-9e0d-460f-a7ba-66239eaf52e9\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f8wfwr" Jul 23 19:31:08.465570 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:31:08.465546 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/dfb3b12c-9e0d-460f-a7ba-66239eaf52e9-dshm\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f8wfwr\" (UID: \"dfb3b12c-9e0d-460f-a7ba-66239eaf52e9\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f8wfwr" Jul 23 19:31:08.465733 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:31:08.465717 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/dfb3b12c-9e0d-460f-a7ba-66239eaf52e9-tls-certs\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f8wfwr\" (UID: \"dfb3b12c-9e0d-460f-a7ba-66239eaf52e9\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f8wfwr" Jul 23 19:31:08.471853 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:31:08.471827 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-wm8lt\" (UniqueName: \"kubernetes.io/projected/dfb3b12c-9e0d-460f-a7ba-66239eaf52e9-kube-api-access-wm8lt\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f8wfwr\" (UID: \"dfb3b12c-9e0d-460f-a7ba-66239eaf52e9\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f8wfwr" Jul 23 19:31:08.624698 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:31:08.624623 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f8wfwr" Jul 23 19:31:08.762763 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:31:08.762736 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f8wfwr"] Jul 23 19:31:08.765042 ip-10-0-129-2 kubenswrapper[2564]: W0723 19:31:08.765014 2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddfb3b12c_9e0d_460f_a7ba_66239eaf52e9.slice/crio-068626332d2a773d7e226f00bf9be99b1efda8b416a1a986443f54d11075f9a6 WatchSource:0}: Error finding container 068626332d2a773d7e226f00bf9be99b1efda8b416a1a986443f54d11075f9a6: Status 404 returned error can't find the container with id 068626332d2a773d7e226f00bf9be99b1efda8b416a1a986443f54d11075f9a6 Jul 23 19:31:08.855627 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:31:08.855598 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f8wfwr" event={"ID":"dfb3b12c-9e0d-460f-a7ba-66239eaf52e9","Type":"ContainerStarted","Data":"068626332d2a773d7e226f00bf9be99b1efda8b416a1a986443f54d11075f9a6"} Jul 23 19:31:09.405045 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:31:09.404961 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-2dgmk"] Jul 23 19:31:09.863049 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:31:09.863009 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f8wfwr" event={"ID":"dfb3b12c-9e0d-460f-a7ba-66239eaf52e9","Type":"ContainerStarted","Data":"47472dd8cad38dfc74e9bf4ea427e338ac88c82963a6482c76ac7984a6abb4b3"} Jul 23 19:31:09.893269 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:31:09.893216 2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f8wfwr" podStartSLOduration=1.690423393 podStartE2EDuration="1.893201254s" totalImagesPullingTime="202.777861ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-23 19:31:08 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-23 19:31:09.891038722 +0000 UTC m=+695.742814969" watchObservedRunningTime="2026-07-23 19:31:09.893201254 +0000 UTC m=+695.744977500" Jul 23 19:31:18.625724 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:31:18.625685 2564 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f8wfwr" Jul 23 19:31:18.626210 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:31:18.625757 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f8wfwr" Jul 23 19:31:18.638782 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:31:18.638755 2564 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f8wfwr" Jul 23 19:31:18.704978 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:31:18.704940 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-2dgmk"] Jul 23 19:31:18.922017 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:31:18.921994 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f8wfwr" Jul 23 19:31:27.806631 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:31:27.806545 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-2dgmk"] Jul 23 19:32:41.285722 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:32:41.285680 2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-5c6965cf8f-ccqwf"] Jul 23 19:32:41.286206 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:32:41.285922 2564 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/authorino-5c6965cf8f-ccqwf" podUID="97fdc4d6-7a0a-41ed-a6a9-684aa3fae641" containerName="authorino" containerID="cri-o://9bfeb10ba604b785e80a098e430f87972f60c01816eb05f45ac19fc84bea67c1" gracePeriod=30 Jul 23 19:32:41.541254 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:32:41.541184 2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-5c6965cf8f-ccqwf" Jul 23 19:32:41.625076 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:32:41.625037 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/97fdc4d6-7a0a-41ed-a6a9-684aa3fae641-tls-cert\") pod \"97fdc4d6-7a0a-41ed-a6a9-684aa3fae641\" (UID: \"97fdc4d6-7a0a-41ed-a6a9-684aa3fae641\") " Jul 23 19:32:41.625274 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:32:41.625087 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-8lqdr\" (UniqueName: \"kubernetes.io/projected/97fdc4d6-7a0a-41ed-a6a9-684aa3fae641-kube-api-access-8lqdr\") pod \"97fdc4d6-7a0a-41ed-a6a9-684aa3fae641\" (UID: \"97fdc4d6-7a0a-41ed-a6a9-684aa3fae641\") " Jul 23 19:32:41.627222 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:32:41.627186 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/97fdc4d6-7a0a-41ed-a6a9-684aa3fae641-kube-api-access-8lqdr" (OuterVolumeSpecName: "kube-api-access-8lqdr") pod "97fdc4d6-7a0a-41ed-a6a9-684aa3fae641" (UID: "97fdc4d6-7a0a-41ed-a6a9-684aa3fae641"). InnerVolumeSpecName "kube-api-access-8lqdr". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 23 19:32:41.635838 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:32:41.635814 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/97fdc4d6-7a0a-41ed-a6a9-684aa3fae641-tls-cert" (OuterVolumeSpecName: "tls-cert") pod "97fdc4d6-7a0a-41ed-a6a9-684aa3fae641" (UID: "97fdc4d6-7a0a-41ed-a6a9-684aa3fae641"). InnerVolumeSpecName "tls-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 23 19:32:41.727175 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:32:41.727130 2564 reconciler_common.go:299] "Volume detached for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/97fdc4d6-7a0a-41ed-a6a9-684aa3fae641-tls-cert\") on node \"ip-10-0-129-2.ec2.internal\" DevicePath \"\"" Jul 23 19:32:41.727175 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:32:41.727158 2564 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8lqdr\" (UniqueName: \"kubernetes.io/projected/97fdc4d6-7a0a-41ed-a6a9-684aa3fae641-kube-api-access-8lqdr\") on node \"ip-10-0-129-2.ec2.internal\" DevicePath \"\"" Jul 23 19:32:42.341982 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:32:42.341944 2564 generic.go:356] "Generic (PLEG): container finished" podID="97fdc4d6-7a0a-41ed-a6a9-684aa3fae641" containerID="9bfeb10ba604b785e80a098e430f87972f60c01816eb05f45ac19fc84bea67c1" exitCode=0 Jul 23 19:32:42.342415 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:32:42.342000 2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-5c6965cf8f-ccqwf" Jul 23 19:32:42.342415 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:32:42.342037 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-5c6965cf8f-ccqwf" event={"ID":"97fdc4d6-7a0a-41ed-a6a9-684aa3fae641","Type":"ContainerDied","Data":"9bfeb10ba604b785e80a098e430f87972f60c01816eb05f45ac19fc84bea67c1"} Jul 23 19:32:42.342415 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:32:42.342082 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-5c6965cf8f-ccqwf" event={"ID":"97fdc4d6-7a0a-41ed-a6a9-684aa3fae641","Type":"ContainerDied","Data":"7b6314e18bb710bd06da5ef5abc822ab26666c9695097701d7315733a1be6e85"} Jul 23 19:32:42.342415 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:32:42.342100 2564 scope.go:117] "RemoveContainer" containerID="9bfeb10ba604b785e80a098e430f87972f60c01816eb05f45ac19fc84bea67c1" Jul 23 19:32:42.351945 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:32:42.351927 2564 scope.go:117] "RemoveContainer" containerID="9bfeb10ba604b785e80a098e430f87972f60c01816eb05f45ac19fc84bea67c1" Jul 23 19:32:42.352223 ip-10-0-129-2 kubenswrapper[2564]: E0723 19:32:42.352203 2564 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9bfeb10ba604b785e80a098e430f87972f60c01816eb05f45ac19fc84bea67c1\": container with ID starting with 9bfeb10ba604b785e80a098e430f87972f60c01816eb05f45ac19fc84bea67c1 not found: ID does not exist" containerID="9bfeb10ba604b785e80a098e430f87972f60c01816eb05f45ac19fc84bea67c1" Jul 23 19:32:42.352290 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:32:42.352235 2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9bfeb10ba604b785e80a098e430f87972f60c01816eb05f45ac19fc84bea67c1"} err="failed to get container status \"9bfeb10ba604b785e80a098e430f87972f60c01816eb05f45ac19fc84bea67c1\": rpc error: code = NotFound desc = could not find container \"9bfeb10ba604b785e80a098e430f87972f60c01816eb05f45ac19fc84bea67c1\": container with ID starting with 9bfeb10ba604b785e80a098e430f87972f60c01816eb05f45ac19fc84bea67c1 not found: ID does not exist" Jul 23 19:32:42.362514 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:32:42.362474 2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-5c6965cf8f-ccqwf"] Jul 23 19:32:42.364729 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:32:42.364700 2564 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-5c6965cf8f-ccqwf"] Jul 23 19:32:42.658592 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:32:42.658514 2564 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="97fdc4d6-7a0a-41ed-a6a9-684aa3fae641" path="/var/lib/kubelet/pods/97fdc4d6-7a0a-41ed-a6a9-684aa3fae641/volumes" Jul 23 19:33:09.494450 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:33:09.494359 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-2dgmk"] Jul 23 19:33:18.885996 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:33:18.885964 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-2dgmk"] Jul 23 19:33:28.703232 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:33:28.703206 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-2dgmk"] Jul 23 19:34:35.409453 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:34:35.409429 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-565b7b569d-z5wf4_ded5b3af-c74c-4d5a-aa77-62fe72d1dd29/console-operator/1.log" Jul 23 19:34:35.409942 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:34:35.409785 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-565b7b569d-z5wf4_ded5b3af-c74c-4d5a-aa77-62fe72d1dd29/console-operator/1.log" Jul 23 19:34:35.416467 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:34:35.416451 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-8dtlh_195ecf89-13ae-4311-bc72-fcdf78a5c4d6/ovn-acl-logging/0.log" Jul 23 19:34:35.417094 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:34:35.417079 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-8dtlh_195ecf89-13ae-4311-bc72-fcdf78a5c4d6/ovn-acl-logging/0.log" Jul 23 19:35:08.895096 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:35:08.895063 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-2dgmk"] Jul 23 19:35:19.990417 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:35:19.990384 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-2dgmk"] Jul 23 19:35:54.394235 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:35:54.394193 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-2dgmk"] Jul 23 19:36:01.995259 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:36:01.995176 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-2dgmk"] Jul 23 19:36:11.091010 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:36:11.090979 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-2dgmk"] Jul 23 19:36:16.688294 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:36:16.688264 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-2dgmk"] Jul 23 19:36:28.087954 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:36:28.087920 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-2dgmk"] Jul 23 19:36:40.392565 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:36:40.392532 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-2dgmk"] Jul 23 19:36:50.985742 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:36:50.985709 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-2dgmk"] Jul 23 19:37:01.593402 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:37:01.593367 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-2dgmk"] Jul 23 19:37:22.901919 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:37:22.901877 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-2dgmk"] Jul 23 19:37:35.793033 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:37:35.792995 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-2dgmk"] Jul 23 19:37:58.092507 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:37:58.092464 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-2dgmk"] Jul 23 19:38:09.092987 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:38:09.092939 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-2dgmk"] Jul 23 19:38:23.421303 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:38:23.421268 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-2dgmk"] Jul 23 19:38:34.987649 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:38:34.987614 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-2dgmk"] Jul 23 19:38:44.300261 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:38:44.300223 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-2dgmk"] Jul 23 19:38:53.792272 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:38:53.792241 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-2dgmk"] Jul 23 19:39:03.111513 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:39:03.111435 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-2dgmk"] Jul 23 19:39:35.449893 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:39:35.449864 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-565b7b569d-z5wf4_ded5b3af-c74c-4d5a-aa77-62fe72d1dd29/console-operator/1.log" Jul 23 19:39:35.450638 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:39:35.450621 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-565b7b569d-z5wf4_ded5b3af-c74c-4d5a-aa77-62fe72d1dd29/console-operator/1.log" Jul 23 19:39:35.457257 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:39:35.457238 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-8dtlh_195ecf89-13ae-4311-bc72-fcdf78a5c4d6/ovn-acl-logging/0.log" Jul 23 19:39:35.457956 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:39:35.457942 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-8dtlh_195ecf89-13ae-4311-bc72-fcdf78a5c4d6/ovn-acl-logging/0.log" Jul 23 19:40:51.587643 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:40:51.587608 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-2dgmk"] Jul 23 19:41:02.606101 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:41:02.606068 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-2dgmk"] Jul 23 19:41:13.597055 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:41:13.597017 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-2dgmk"] Jul 23 19:41:24.406099 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:41:24.406071 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-2dgmk"] Jul 23 19:41:35.695023 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:41:35.694994 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-2dgmk"] Jul 23 19:41:46.135947 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:41:46.135918 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-2dgmk"] Jul 23 19:41:54.990402 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:41:54.990374 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-2dgmk"] Jul 23 19:42:05.843303 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:42:05.843223 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-2dgmk"] Jul 23 19:42:17.896907 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:42:17.896874 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-2dgmk"] Jul 23 19:42:26.090882 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:42:26.090850 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-2dgmk"] Jul 23 19:42:36.292265 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:42:36.292238 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-2dgmk"] Jul 23 19:43:00.895166 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:43:00.895132 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-2dgmk"] Jul 23 19:43:08.392157 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:43:08.392125 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-2dgmk"] Jul 23 19:43:21.422955 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:43:21.422923 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-2dgmk"] Jul 23 19:43:54.912072 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:43:54.912036 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-2dgmk"] Jul 23 19:44:09.095926 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:44:09.095890 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-2dgmk"] Jul 23 19:44:15.100374 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:44:15.100340 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-2dgmk"] Jul 23 19:44:32.805135 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:44:32.805106 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-2dgmk"] Jul 23 19:44:35.484654 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:44:35.484624 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-565b7b569d-z5wf4_ded5b3af-c74c-4d5a-aa77-62fe72d1dd29/console-operator/1.log" Jul 23 19:44:35.486478 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:44:35.486457 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-565b7b569d-z5wf4_ded5b3af-c74c-4d5a-aa77-62fe72d1dd29/console-operator/1.log" Jul 23 19:44:35.491641 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:44:35.491623 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-8dtlh_195ecf89-13ae-4311-bc72-fcdf78a5c4d6/ovn-acl-logging/0.log" Jul 23 19:44:35.493571 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:44:35.493552 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-8dtlh_195ecf89-13ae-4311-bc72-fcdf78a5c4d6/ovn-acl-logging/0.log" Jul 23 19:44:42.194723 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:44:42.194685 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-2dgmk"] Jul 23 19:44:52.398656 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:44:52.398618 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-2dgmk"] Jul 23 19:45:00.187974 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:45:00.187895 2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29747265-qd6f8"] Jul 23 19:45:00.188311 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:45:00.188281 2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="97fdc4d6-7a0a-41ed-a6a9-684aa3fae641" containerName="authorino" Jul 23 19:45:00.188311 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:45:00.188294 2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="97fdc4d6-7a0a-41ed-a6a9-684aa3fae641" containerName="authorino" Jul 23 19:45:00.188421 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:45:00.188382 2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="97fdc4d6-7a0a-41ed-a6a9-684aa3fae641" containerName="authorino" Jul 23 19:45:00.191446 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:45:00.191432 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29747265-qd6f8" Jul 23 19:45:00.196947 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:45:00.196926 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-gcspk\"" Jul 23 19:45:00.201542 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:45:00.201521 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29747265-qd6f8" Jul 23 19:45:00.202623 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:45:00.202606 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29747265-qd6f8"] Jul 23 19:45:00.343346 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:45:00.343308 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29747265-qd6f8"] Jul 23 19:45:00.344266 ip-10-0-129-2 kubenswrapper[2564]: W0723 19:45:00.344236 2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod00515ff4_26d7_4e68_ba28_d8254b4f3e5c.slice/crio-d62a173665586d405729575cbbc5beab5ff8442a774f985e1059950566ae822d WatchSource:0}: Error finding container d62a173665586d405729575cbbc5beab5ff8442a774f985e1059950566ae822d: Status 404 returned error can't find the container with id d62a173665586d405729575cbbc5beab5ff8442a774f985e1059950566ae822d Jul 23 19:45:00.346270 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:45:00.346249 2564 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Jul 23 19:45:01.176353 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:45:01.176314 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29747265-qd6f8" event={"ID":"00515ff4-26d7-4e68-ba28-d8254b4f3e5c","Type":"ContainerStarted","Data":"d62a173665586d405729575cbbc5beab5ff8442a774f985e1059950566ae822d"} Jul 23 19:45:02.897370 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:45:02.897342 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-2dgmk"] Jul 23 19:45:03.189212 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:45:03.189179 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29747265-qd6f8" event={"ID":"00515ff4-26d7-4e68-ba28-d8254b4f3e5c","Type":"ContainerStarted","Data":"2a2e433d10f9ccd025de2c3b52f8f2299a0d3116d5d5e0f3aebeed2a46f7f89c"} Jul 23 19:45:03.205209 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:45:03.205167 2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29747265-qd6f8" podStartSLOduration=0.643733213 podStartE2EDuration="3.205153432s" totalImagesPullingTime="2.561420219s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-23 19:45:00 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-23 19:45:03.202964309 +0000 UTC m=+1529.054740593" watchObservedRunningTime="2026-07-23 19:45:03.205153432 +0000 UTC m=+1529.056929674" Jul 23 19:45:04.196060 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:45:04.196024 2564 generic.go:356] "Generic (PLEG): container finished" podID="00515ff4-26d7-4e68-ba28-d8254b4f3e5c" containerID="2a2e433d10f9ccd025de2c3b52f8f2299a0d3116d5d5e0f3aebeed2a46f7f89c" exitCode=0 Jul 23 19:45:04.196521 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:45:04.196111 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29747265-qd6f8" event={"ID":"00515ff4-26d7-4e68-ba28-d8254b4f3e5c","Type":"ContainerDied","Data":"2a2e433d10f9ccd025de2c3b52f8f2299a0d3116d5d5e0f3aebeed2a46f7f89c"} Jul 23 19:45:05.336379 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:45:05.336353 2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29747265-qd6f8" Jul 23 19:45:06.208596 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:45:06.208573 2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29747265-qd6f8" Jul 23 19:45:06.208596 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:45:06.208583 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29747265-qd6f8" event={"ID":"00515ff4-26d7-4e68-ba28-d8254b4f3e5c","Type":"ContainerDied","Data":"d62a173665586d405729575cbbc5beab5ff8442a774f985e1059950566ae822d"} Jul 23 19:45:06.208781 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:45:06.208629 2564 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="d62a173665586d405729575cbbc5beab5ff8442a774f985e1059950566ae822d" Jul 23 19:45:11.698355 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:45:11.698309 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-2dgmk"] Jul 23 19:45:21.400255 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:45:21.400224 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-2dgmk"] Jul 23 19:45:34.195625 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:45:34.195594 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-2dgmk"] Jul 23 19:45:44.091559 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:45:44.091531 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-2dgmk"] Jul 23 19:45:53.795931 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:45:53.795897 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-2dgmk"] Jul 23 19:46:02.996027 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:46:02.995997 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-2dgmk"] Jul 23 19:46:16.194786 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:46:16.194753 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-2dgmk"] Jul 23 19:46:24.801277 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:46:24.801240 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-2dgmk"] Jul 23 19:46:37.490791 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:46:37.490752 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-2dgmk"] Jul 23 19:46:50.501298 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:46:50.501270 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-2dgmk"] Jul 23 19:46:59.288839 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:46:59.288801 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-2dgmk"] Jul 23 19:47:13.197872 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:47:13.197838 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-2dgmk"] Jul 23 19:47:22.690040 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:47:22.690008 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-2dgmk"] Jul 23 19:47:32.092519 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:47:32.092485 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-2dgmk"] Jul 23 19:47:35.905737 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:47:35.905706 2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-12f58144-566c57875c-xlj7x"] Jul 23 19:47:35.906244 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:47:35.906229 2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="00515ff4-26d7-4e68-ba28-d8254b4f3e5c" containerName="cleanup" Jul 23 19:47:35.906347 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:47:35.906248 2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="00515ff4-26d7-4e68-ba28-d8254b4f3e5c" containerName="cleanup" Jul 23 19:47:35.906420 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:47:35.906380 2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="00515ff4-26d7-4e68-ba28-d8254b4f3e5c" containerName="cleanup" Jul 23 19:47:35.909740 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:47:35.909715 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-12f58144-566c57875c-xlj7x" Jul 23 19:47:35.911883 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:47:35.911864 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ait-12f58144-serving-cert\"" Jul 23 19:47:35.912099 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:47:35.912066 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Jul 23 19:47:35.912099 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:47:35.912084 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-7m48c\"" Jul 23 19:47:35.912715 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:47:35.912698 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Jul 23 19:47:35.918885 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:47:35.918862 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-12f58144-566c57875c-xlj7x"] Jul 23 19:47:36.022380 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:47:36.022345 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-scmrl\" (UniqueName: \"kubernetes.io/projected/9f859de2-d870-493d-bb66-7c6cea9d4aa6-kube-api-access-scmrl\") pod \"maas-api-e2e-ait-12f58144-566c57875c-xlj7x\" (UID: \"9f859de2-d870-493d-bb66-7c6cea9d4aa6\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-12f58144-566c57875c-xlj7x" Jul 23 19:47:36.022550 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:47:36.022418 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9f859de2-d870-493d-bb66-7c6cea9d4aa6-maas-api-tls\") pod \"maas-api-e2e-ait-12f58144-566c57875c-xlj7x\" (UID: \"9f859de2-d870-493d-bb66-7c6cea9d4aa6\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-12f58144-566c57875c-xlj7x" Jul 23 19:47:36.022550 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:47:36.022511 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/9f859de2-d870-493d-bb66-7c6cea9d4aa6-openshift-service-ca\") pod \"maas-api-e2e-ait-12f58144-566c57875c-xlj7x\" (UID: \"9f859de2-d870-493d-bb66-7c6cea9d4aa6\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-12f58144-566c57875c-xlj7x" Jul 23 19:47:36.123396 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:47:36.123369 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9f859de2-d870-493d-bb66-7c6cea9d4aa6-maas-api-tls\") pod \"maas-api-e2e-ait-12f58144-566c57875c-xlj7x\" (UID: \"9f859de2-d870-493d-bb66-7c6cea9d4aa6\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-12f58144-566c57875c-xlj7x" Jul 23 19:47:36.123528 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:47:36.123452 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/9f859de2-d870-493d-bb66-7c6cea9d4aa6-openshift-service-ca\") pod \"maas-api-e2e-ait-12f58144-566c57875c-xlj7x\" (UID: \"9f859de2-d870-493d-bb66-7c6cea9d4aa6\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-12f58144-566c57875c-xlj7x" Jul 23 19:47:36.123528 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:47:36.123518 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-scmrl\" (UniqueName: \"kubernetes.io/projected/9f859de2-d870-493d-bb66-7c6cea9d4aa6-kube-api-access-scmrl\") pod \"maas-api-e2e-ait-12f58144-566c57875c-xlj7x\" (UID: \"9f859de2-d870-493d-bb66-7c6cea9d4aa6\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-12f58144-566c57875c-xlj7x" Jul 23 19:47:36.124070 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:47:36.124012 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/9f859de2-d870-493d-bb66-7c6cea9d4aa6-openshift-service-ca\") pod \"maas-api-e2e-ait-12f58144-566c57875c-xlj7x\" (UID: \"9f859de2-d870-493d-bb66-7c6cea9d4aa6\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-12f58144-566c57875c-xlj7x" Jul 23 19:47:36.125814 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:47:36.125793 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9f859de2-d870-493d-bb66-7c6cea9d4aa6-maas-api-tls\") pod \"maas-api-e2e-ait-12f58144-566c57875c-xlj7x\" (UID: \"9f859de2-d870-493d-bb66-7c6cea9d4aa6\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-12f58144-566c57875c-xlj7x" Jul 23 19:47:36.131172 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:47:36.131155 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-scmrl\" (UniqueName: \"kubernetes.io/projected/9f859de2-d870-493d-bb66-7c6cea9d4aa6-kube-api-access-scmrl\") pod \"maas-api-e2e-ait-12f58144-566c57875c-xlj7x\" (UID: \"9f859de2-d870-493d-bb66-7c6cea9d4aa6\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-12f58144-566c57875c-xlj7x" Jul 23 19:47:36.221932 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:47:36.221911 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-12f58144-566c57875c-xlj7x" Jul 23 19:47:36.349203 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:47:36.349176 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-12f58144-566c57875c-xlj7x"] Jul 23 19:47:36.350680 ip-10-0-129-2 kubenswrapper[2564]: W0723 19:47:36.350658 2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9f859de2_d870_493d_bb66_7c6cea9d4aa6.slice/crio-64ea742fc290f4f52d8a0592a1a82c54fc953457eaa40b28af663094d22d87aa WatchSource:0}: Error finding container 64ea742fc290f4f52d8a0592a1a82c54fc953457eaa40b28af663094d22d87aa: Status 404 returned error can't find the container with id 64ea742fc290f4f52d8a0592a1a82c54fc953457eaa40b28af663094d22d87aa Jul 23 19:47:36.964978 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:47:36.964938 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-12f58144-566c57875c-xlj7x" event={"ID":"9f859de2-d870-493d-bb66-7c6cea9d4aa6","Type":"ContainerStarted","Data":"64ea742fc290f4f52d8a0592a1a82c54fc953457eaa40b28af663094d22d87aa"} Jul 23 19:47:38.977345 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:47:38.977299 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-12f58144-566c57875c-xlj7x" event={"ID":"9f859de2-d870-493d-bb66-7c6cea9d4aa6","Type":"ContainerStarted","Data":"dda8c5117e9de7f0e075e3f241060bd6376700ba65c1803d947a8abd80ffe1c2"} Jul 23 19:47:38.977691 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:47:38.977381 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-12f58144-566c57875c-xlj7x" Jul 23 19:47:38.997106 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:47:38.997061 2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ait-12f58144-566c57875c-xlj7x" podStartSLOduration=1.804901851 podStartE2EDuration="3.997049274s" totalImagesPullingTime="2.192147423s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-23 19:47:35 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-23 19:47:38.995204656 +0000 UTC m=+1684.846980919" watchObservedRunningTime="2026-07-23 19:47:38.997049274 +0000 UTC m=+1684.848825519" Jul 23 19:47:42.396521 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:47:42.396489 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-2dgmk"] Jul 23 19:47:44.987703 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:47:44.987672 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-12f58144-566c57875c-xlj7x" Jul 23 19:48:01.243395 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:48:01.243298 2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-12f58144-mwdk5"] Jul 23 19:48:01.252534 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:48:01.252506 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-12f58144-mwdk5" Jul 23 19:48:01.254913 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:48:01.254885 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-12f58144-mwdk5"] Jul 23 19:48:01.255032 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:48:01.254997 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-gcspk\"" Jul 23 19:48:01.326461 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:48:01.326427 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ef74adaa-205b-4a00-897c-ae8d1b4980fe-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-12f58144-mwdk5\" (UID: \"ef74adaa-205b-4a00-897c-ae8d1b4980fe\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-12f58144-mwdk5" Jul 23 19:48:01.427052 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:48:01.427011 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ef74adaa-205b-4a00-897c-ae8d1b4980fe-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-12f58144-mwdk5\" (UID: \"ef74adaa-205b-4a00-897c-ae8d1b4980fe\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-12f58144-mwdk5" Jul 23 19:48:01.427691 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:48:01.427671 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ef74adaa-205b-4a00-897c-ae8d1b4980fe-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-12f58144-mwdk5\" (UID: \"ef74adaa-205b-4a00-897c-ae8d1b4980fe\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-12f58144-mwdk5" Jul 23 19:48:01.564663 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:48:01.564581 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-12f58144-mwdk5" Jul 23 19:48:01.695230 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:48:01.695202 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-12f58144-mwdk5"] Jul 23 19:48:01.697360 ip-10-0-129-2 kubenswrapper[2564]: W0723 19:48:01.697304 2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podef74adaa_205b_4a00_897c_ae8d1b4980fe.slice/crio-cfa5f95c34e80d7abd748bf95b658fe14e28fa65740bf8042e263b89715bae0b WatchSource:0}: Error finding container cfa5f95c34e80d7abd748bf95b658fe14e28fa65740bf8042e263b89715bae0b: Status 404 returned error can't find the container with id cfa5f95c34e80d7abd748bf95b658fe14e28fa65740bf8042e263b89715bae0b Jul 23 19:48:02.099738 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:48:02.099704 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-12f58144-mwdk5" event={"ID":"ef74adaa-205b-4a00-897c-ae8d1b4980fe","Type":"ContainerStarted","Data":"291e3434fecb1eee5f7b300c2812aaf9e75af1bbde0e206b92bb587157f303f8"} Jul 23 19:48:02.099856 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:48:02.099768 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-12f58144-mwdk5" event={"ID":"ef74adaa-205b-4a00-897c-ae8d1b4980fe","Type":"ContainerStarted","Data":"cfa5f95c34e80d7abd748bf95b658fe14e28fa65740bf8042e263b89715bae0b"} Jul 23 19:48:02.116095 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:48:02.116051 2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-12f58144-mwdk5" podStartSLOduration=1.116038039 podStartE2EDuration="1.116038039s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-23 19:48:01 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-23 19:48:02.114093352 +0000 UTC m=+1707.965869601" watchObservedRunningTime="2026-07-23 19:48:02.116038039 +0000 UTC m=+1707.967814264" Jul 23 19:48:03.106770 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:48:03.106739 2564 generic.go:356] "Generic (PLEG): container finished" podID="ef74adaa-205b-4a00-897c-ae8d1b4980fe" containerID="291e3434fecb1eee5f7b300c2812aaf9e75af1bbde0e206b92bb587157f303f8" exitCode=0 Jul 23 19:48:03.107138 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:48:03.106810 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-12f58144-mwdk5" event={"ID":"ef74adaa-205b-4a00-897c-ae8d1b4980fe","Type":"ContainerDied","Data":"291e3434fecb1eee5f7b300c2812aaf9e75af1bbde0e206b92bb587157f303f8"} Jul 23 19:48:04.245895 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:48:04.245871 2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-12f58144-mwdk5" Jul 23 19:48:04.353010 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:48:04.352980 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ef74adaa-205b-4a00-897c-ae8d1b4980fe-maas-api-service-ca\") pod \"ef74adaa-205b-4a00-897c-ae8d1b4980fe\" (UID: \"ef74adaa-205b-4a00-897c-ae8d1b4980fe\") " Jul 23 19:48:04.353315 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:48:04.353293 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ef74adaa-205b-4a00-897c-ae8d1b4980fe-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "ef74adaa-205b-4a00-897c-ae8d1b4980fe" (UID: "ef74adaa-205b-4a00-897c-ae8d1b4980fe"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 23 19:48:04.453819 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:48:04.453780 2564 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ef74adaa-205b-4a00-897c-ae8d1b4980fe-maas-api-service-ca\") on node \"ip-10-0-129-2.ec2.internal\" DevicePath \"\"" Jul 23 19:48:05.117969 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:48:05.117944 2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-12f58144-mwdk5" Jul 23 19:48:05.118145 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:48:05.117978 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-12f58144-mwdk5" event={"ID":"ef74adaa-205b-4a00-897c-ae8d1b4980fe","Type":"ContainerDied","Data":"cfa5f95c34e80d7abd748bf95b658fe14e28fa65740bf8042e263b89715bae0b"} Jul 23 19:48:05.118145 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:48:05.118018 2564 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="cfa5f95c34e80d7abd748bf95b658fe14e28fa65740bf8042e263b89715bae0b" Jul 23 19:48:11.677482 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:48:11.677451 2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-12f58144-566c57875c-xlj7x"] Jul 23 19:48:11.678376 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:48:11.678312 2564 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ait-12f58144-566c57875c-xlj7x" podUID="9f859de2-d870-493d-bb66-7c6cea9d4aa6" containerName="maas-api" containerID="cri-o://dda8c5117e9de7f0e075e3f241060bd6376700ba65c1803d947a8abd80ffe1c2" gracePeriod=30 Jul 23 19:48:11.935316 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:48:11.935262 2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-12f58144-566c57875c-xlj7x" Jul 23 19:48:12.014665 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:48:12.014638 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-scmrl\" (UniqueName: \"kubernetes.io/projected/9f859de2-d870-493d-bb66-7c6cea9d4aa6-kube-api-access-scmrl\") pod \"9f859de2-d870-493d-bb66-7c6cea9d4aa6\" (UID: \"9f859de2-d870-493d-bb66-7c6cea9d4aa6\") " Jul 23 19:48:12.014787 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:48:12.014703 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/9f859de2-d870-493d-bb66-7c6cea9d4aa6-openshift-service-ca\") pod \"9f859de2-d870-493d-bb66-7c6cea9d4aa6\" (UID: \"9f859de2-d870-493d-bb66-7c6cea9d4aa6\") " Jul 23 19:48:12.014787 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:48:12.014758 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9f859de2-d870-493d-bb66-7c6cea9d4aa6-maas-api-tls\") pod \"9f859de2-d870-493d-bb66-7c6cea9d4aa6\" (UID: \"9f859de2-d870-493d-bb66-7c6cea9d4aa6\") " Jul 23 19:48:12.015022 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:48:12.015001 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9f859de2-d870-493d-bb66-7c6cea9d4aa6-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "9f859de2-d870-493d-bb66-7c6cea9d4aa6" (UID: "9f859de2-d870-493d-bb66-7c6cea9d4aa6"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 23 19:48:12.016875 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:48:12.016854 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9f859de2-d870-493d-bb66-7c6cea9d4aa6-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "9f859de2-d870-493d-bb66-7c6cea9d4aa6" (UID: "9f859de2-d870-493d-bb66-7c6cea9d4aa6"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 23 19:48:12.016987 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:48:12.016920 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9f859de2-d870-493d-bb66-7c6cea9d4aa6-kube-api-access-scmrl" (OuterVolumeSpecName: "kube-api-access-scmrl") pod "9f859de2-d870-493d-bb66-7c6cea9d4aa6" (UID: "9f859de2-d870-493d-bb66-7c6cea9d4aa6"). InnerVolumeSpecName "kube-api-access-scmrl". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 23 19:48:12.116148 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:48:12.116125 2564 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-scmrl\" (UniqueName: \"kubernetes.io/projected/9f859de2-d870-493d-bb66-7c6cea9d4aa6-kube-api-access-scmrl\") on node \"ip-10-0-129-2.ec2.internal\" DevicePath \"\"" Jul 23 19:48:12.116148 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:48:12.116147 2564 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/9f859de2-d870-493d-bb66-7c6cea9d4aa6-openshift-service-ca\") on node \"ip-10-0-129-2.ec2.internal\" DevicePath \"\"" Jul 23 19:48:12.116274 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:48:12.116156 2564 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9f859de2-d870-493d-bb66-7c6cea9d4aa6-maas-api-tls\") on node \"ip-10-0-129-2.ec2.internal\" DevicePath \"\"" Jul 23 19:48:12.152611 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:48:12.152584 2564 generic.go:356] "Generic (PLEG): container finished" podID="9f859de2-d870-493d-bb66-7c6cea9d4aa6" containerID="dda8c5117e9de7f0e075e3f241060bd6376700ba65c1803d947a8abd80ffe1c2" exitCode=0 Jul 23 19:48:12.152701 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:48:12.152626 2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-12f58144-566c57875c-xlj7x" Jul 23 19:48:12.152701 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:48:12.152659 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-12f58144-566c57875c-xlj7x" event={"ID":"9f859de2-d870-493d-bb66-7c6cea9d4aa6","Type":"ContainerDied","Data":"dda8c5117e9de7f0e075e3f241060bd6376700ba65c1803d947a8abd80ffe1c2"} Jul 23 19:48:12.152772 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:48:12.152710 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-12f58144-566c57875c-xlj7x" event={"ID":"9f859de2-d870-493d-bb66-7c6cea9d4aa6","Type":"ContainerDied","Data":"64ea742fc290f4f52d8a0592a1a82c54fc953457eaa40b28af663094d22d87aa"} Jul 23 19:48:12.152772 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:48:12.152734 2564 scope.go:117] "RemoveContainer" containerID="dda8c5117e9de7f0e075e3f241060bd6376700ba65c1803d947a8abd80ffe1c2" Jul 23 19:48:12.161837 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:48:12.161814 2564 scope.go:117] "RemoveContainer" containerID="dda8c5117e9de7f0e075e3f241060bd6376700ba65c1803d947a8abd80ffe1c2" Jul 23 19:48:12.162117 ip-10-0-129-2 kubenswrapper[2564]: E0723 19:48:12.162099 2564 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"dda8c5117e9de7f0e075e3f241060bd6376700ba65c1803d947a8abd80ffe1c2\": container with ID starting with dda8c5117e9de7f0e075e3f241060bd6376700ba65c1803d947a8abd80ffe1c2 not found: ID does not exist" containerID="dda8c5117e9de7f0e075e3f241060bd6376700ba65c1803d947a8abd80ffe1c2" Jul 23 19:48:12.162187 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:48:12.162129 2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"dda8c5117e9de7f0e075e3f241060bd6376700ba65c1803d947a8abd80ffe1c2"} err="failed to get container status \"dda8c5117e9de7f0e075e3f241060bd6376700ba65c1803d947a8abd80ffe1c2\": rpc error: code = NotFound desc = could not find container \"dda8c5117e9de7f0e075e3f241060bd6376700ba65c1803d947a8abd80ffe1c2\": container with ID starting with dda8c5117e9de7f0e075e3f241060bd6376700ba65c1803d947a8abd80ffe1c2 not found: ID does not exist" Jul 23 19:48:12.177869 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:48:12.177846 2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-12f58144-566c57875c-xlj7x"] Jul 23 19:48:12.181478 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:48:12.181459 2564 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-12f58144-566c57875c-xlj7x"] Jul 23 19:48:12.658538 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:48:12.658502 2564 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="9f859de2-d870-493d-bb66-7c6cea9d4aa6" path="/var/lib/kubelet/pods/9f859de2-d870-493d-bb66-7c6cea9d4aa6/volumes" Jul 23 19:48:14.392813 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:48:14.392776 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-2dgmk"] Jul 23 19:48:16.328616 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:48:16.328578 2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-12f58144-mwdk5"] Jul 23 19:48:16.331710 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:48:16.331684 2564 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-12f58144-mwdk5"] Jul 23 19:48:16.657816 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:48:16.657738 2564 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="ef74adaa-205b-4a00-897c-ae8d1b4980fe" path="/var/lib/kubelet/pods/ef74adaa-205b-4a00-897c-ae8d1b4980fe/volumes" Jul 23 19:48:20.297117 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:48:20.297080 2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-7d09e9f8-7f6b7849c5-p9pf5"] Jul 23 19:48:20.297582 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:48:20.297566 2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9f859de2-d870-493d-bb66-7c6cea9d4aa6" containerName="maas-api" Jul 23 19:48:20.297637 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:48:20.297585 2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="9f859de2-d870-493d-bb66-7c6cea9d4aa6" containerName="maas-api" Jul 23 19:48:20.297637 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:48:20.297602 2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ef74adaa-205b-4a00-897c-ae8d1b4980fe" containerName="revoke-keys" Jul 23 19:48:20.297637 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:48:20.297611 2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="ef74adaa-205b-4a00-897c-ae8d1b4980fe" containerName="revoke-keys" Jul 23 19:48:20.297731 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:48:20.297684 2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="9f859de2-d870-493d-bb66-7c6cea9d4aa6" containerName="maas-api" Jul 23 19:48:20.297731 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:48:20.297698 2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="ef74adaa-205b-4a00-897c-ae8d1b4980fe" containerName="revoke-keys" Jul 23 19:48:20.302082 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:48:20.302063 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-7d09e9f8-7f6b7849c5-p9pf5" Jul 23 19:48:20.305051 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:48:20.305029 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-7m48c\"" Jul 23 19:48:20.305163 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:48:20.305092 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Jul 23 19:48:20.305707 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:48:20.305686 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Jul 23 19:48:20.305809 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:48:20.305685 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-migrate-7d09e9f8-serving-cert\"" Jul 23 19:48:20.313481 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:48:20.313460 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-7d09e9f8-7f6b7849c5-p9pf5"] Jul 23 19:48:20.390185 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:48:20.390150 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f1187ab3-3260-4816-8cad-7375bc85a33a-maas-api-tls\") pod \"maas-api-e2e-migrate-7d09e9f8-7f6b7849c5-p9pf5\" (UID: \"f1187ab3-3260-4816-8cad-7375bc85a33a\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-7d09e9f8-7f6b7849c5-p9pf5" Jul 23 19:48:20.390408 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:48:20.390199 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f1187ab3-3260-4816-8cad-7375bc85a33a-openshift-service-ca\") pod \"maas-api-e2e-migrate-7d09e9f8-7f6b7849c5-p9pf5\" (UID: \"f1187ab3-3260-4816-8cad-7375bc85a33a\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-7d09e9f8-7f6b7849c5-p9pf5" Jul 23 19:48:20.390408 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:48:20.390262 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dlfsd\" (UniqueName: \"kubernetes.io/projected/f1187ab3-3260-4816-8cad-7375bc85a33a-kube-api-access-dlfsd\") pod \"maas-api-e2e-migrate-7d09e9f8-7f6b7849c5-p9pf5\" (UID: \"f1187ab3-3260-4816-8cad-7375bc85a33a\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-7d09e9f8-7f6b7849c5-p9pf5" Jul 23 19:48:20.491276 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:48:20.491239 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dlfsd\" (UniqueName: \"kubernetes.io/projected/f1187ab3-3260-4816-8cad-7375bc85a33a-kube-api-access-dlfsd\") pod \"maas-api-e2e-migrate-7d09e9f8-7f6b7849c5-p9pf5\" (UID: \"f1187ab3-3260-4816-8cad-7375bc85a33a\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-7d09e9f8-7f6b7849c5-p9pf5" Jul 23 19:48:20.491543 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:48:20.491404 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f1187ab3-3260-4816-8cad-7375bc85a33a-maas-api-tls\") pod \"maas-api-e2e-migrate-7d09e9f8-7f6b7849c5-p9pf5\" (UID: \"f1187ab3-3260-4816-8cad-7375bc85a33a\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-7d09e9f8-7f6b7849c5-p9pf5" Jul 23 19:48:20.491543 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:48:20.491513 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f1187ab3-3260-4816-8cad-7375bc85a33a-openshift-service-ca\") pod \"maas-api-e2e-migrate-7d09e9f8-7f6b7849c5-p9pf5\" (UID: \"f1187ab3-3260-4816-8cad-7375bc85a33a\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-7d09e9f8-7f6b7849c5-p9pf5" Jul 23 19:48:20.491543 ip-10-0-129-2 kubenswrapper[2564]: E0723 19:48:20.491534 2564 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-migrate-7d09e9f8-serving-cert: secret "maas-api-e2e-migrate-7d09e9f8-serving-cert" not found Jul 23 19:48:20.491748 ip-10-0-129-2 kubenswrapper[2564]: E0723 19:48:20.491730 2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f1187ab3-3260-4816-8cad-7375bc85a33a-maas-api-tls podName:f1187ab3-3260-4816-8cad-7375bc85a33a nodeName:}" failed. No retries permitted until 2026-07-23 19:48:20.99170511 +0000 UTC m=+1726.843481346 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/f1187ab3-3260-4816-8cad-7375bc85a33a-maas-api-tls") pod "maas-api-e2e-migrate-7d09e9f8-7f6b7849c5-p9pf5" (UID: "f1187ab3-3260-4816-8cad-7375bc85a33a") : secret "maas-api-e2e-migrate-7d09e9f8-serving-cert" not found Jul 23 19:48:20.492224 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:48:20.492210 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f1187ab3-3260-4816-8cad-7375bc85a33a-openshift-service-ca\") pod \"maas-api-e2e-migrate-7d09e9f8-7f6b7849c5-p9pf5\" (UID: \"f1187ab3-3260-4816-8cad-7375bc85a33a\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-7d09e9f8-7f6b7849c5-p9pf5" Jul 23 19:48:20.500308 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:48:20.500286 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-dlfsd\" (UniqueName: \"kubernetes.io/projected/f1187ab3-3260-4816-8cad-7375bc85a33a-kube-api-access-dlfsd\") pod \"maas-api-e2e-migrate-7d09e9f8-7f6b7849c5-p9pf5\" (UID: \"f1187ab3-3260-4816-8cad-7375bc85a33a\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-7d09e9f8-7f6b7849c5-p9pf5" Jul 23 19:48:20.995841 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:48:20.995807 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f1187ab3-3260-4816-8cad-7375bc85a33a-maas-api-tls\") pod \"maas-api-e2e-migrate-7d09e9f8-7f6b7849c5-p9pf5\" (UID: \"f1187ab3-3260-4816-8cad-7375bc85a33a\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-7d09e9f8-7f6b7849c5-p9pf5" Jul 23 19:48:20.998408 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:48:20.998382 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f1187ab3-3260-4816-8cad-7375bc85a33a-maas-api-tls\") pod \"maas-api-e2e-migrate-7d09e9f8-7f6b7849c5-p9pf5\" (UID: \"f1187ab3-3260-4816-8cad-7375bc85a33a\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-7d09e9f8-7f6b7849c5-p9pf5" Jul 23 19:48:21.212769 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:48:21.212737 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-7d09e9f8-7f6b7849c5-p9pf5" Jul 23 19:48:21.374830 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:48:21.374779 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-7d09e9f8-7f6b7849c5-p9pf5"] Jul 23 19:48:21.377133 ip-10-0-129-2 kubenswrapper[2564]: W0723 19:48:21.377105 2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf1187ab3_3260_4816_8cad_7375bc85a33a.slice/crio-0b442ff65dc5d24916be9b57d52b45ba62979a07e330ee3e5a8c27a77c1a2aaf WatchSource:0}: Error finding container 0b442ff65dc5d24916be9b57d52b45ba62979a07e330ee3e5a8c27a77c1a2aaf: Status 404 returned error can't find the container with id 0b442ff65dc5d24916be9b57d52b45ba62979a07e330ee3e5a8c27a77c1a2aaf Jul 23 19:48:22.206394 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:48:22.206361 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-7d09e9f8-7f6b7849c5-p9pf5" event={"ID":"f1187ab3-3260-4816-8cad-7375bc85a33a","Type":"ContainerStarted","Data":"5f9772cc963c75b314088174911418bcb920c4f2a6fa8352875014c5698b6f76"} Jul 23 19:48:22.206559 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:48:22.206417 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-7d09e9f8-7f6b7849c5-p9pf5" event={"ID":"f1187ab3-3260-4816-8cad-7375bc85a33a","Type":"ContainerStarted","Data":"0b442ff65dc5d24916be9b57d52b45ba62979a07e330ee3e5a8c27a77c1a2aaf"} Jul 23 19:48:22.206559 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:48:22.206458 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-7d09e9f8-7f6b7849c5-p9pf5" Jul 23 19:48:22.223053 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:48:22.223004 2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-7d09e9f8-7f6b7849c5-p9pf5" podStartSLOduration=1.8486792749999998 podStartE2EDuration="2.222990086s" totalImagesPullingTime="374.310811ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-23 19:48:20 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-23 19:48:22.221220064 +0000 UTC m=+1728.072996310" watchObservedRunningTime="2026-07-23 19:48:22.222990086 +0000 UTC m=+1728.074766312" Jul 23 19:48:27.100270 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:48:27.100236 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-2dgmk"] Jul 23 19:48:28.217386 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:48:28.217360 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-7d09e9f8-7f6b7849c5-p9pf5" Jul 23 19:48:45.022276 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:48:45.022245 2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-7d09e9f8-hbvnv"] Jul 23 19:48:45.026940 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:48:45.026922 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-7d09e9f8-hbvnv" Jul 23 19:48:45.029857 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:48:45.029640 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-gcspk\"" Jul 23 19:48:45.031924 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:48:45.031902 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-7d09e9f8-hbvnv"] Jul 23 19:48:45.087251 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:48:45.087232 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/308e68aa-b7f2-497d-ad01-017cce437d3c-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-7d09e9f8-hbvnv\" (UID: \"308e68aa-b7f2-497d-ad01-017cce437d3c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-7d09e9f8-hbvnv" Jul 23 19:48:45.187646 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:48:45.187628 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/308e68aa-b7f2-497d-ad01-017cce437d3c-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-7d09e9f8-hbvnv\" (UID: \"308e68aa-b7f2-497d-ad01-017cce437d3c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-7d09e9f8-hbvnv" Jul 23 19:48:45.188117 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:48:45.188101 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/308e68aa-b7f2-497d-ad01-017cce437d3c-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-7d09e9f8-hbvnv\" (UID: \"308e68aa-b7f2-497d-ad01-017cce437d3c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-7d09e9f8-hbvnv" Jul 23 19:48:45.339232 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:48:45.339178 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-7d09e9f8-hbvnv" Jul 23 19:48:45.469130 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:48:45.469105 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-7d09e9f8-hbvnv"] Jul 23 19:48:45.471693 ip-10-0-129-2 kubenswrapper[2564]: W0723 19:48:45.471666 2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod308e68aa_b7f2_497d_ad01_017cce437d3c.slice/crio-bda5ef528386699df417e4b4dde24738ca11a44fa798e54a00ccc98bcbfb8975 WatchSource:0}: Error finding container bda5ef528386699df417e4b4dde24738ca11a44fa798e54a00ccc98bcbfb8975: Status 404 returned error can't find the container with id bda5ef528386699df417e4b4dde24738ca11a44fa798e54a00ccc98bcbfb8975 Jul 23 19:48:46.344185 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:48:46.344129 2564 generic.go:356] "Generic (PLEG): container finished" podID="308e68aa-b7f2-497d-ad01-017cce437d3c" containerID="e646662c71a5a3b1e91340d71ba1171a313fb8310a459291bce2670343727c35" exitCode=0 Jul 23 19:48:46.344485 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:48:46.344212 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-7d09e9f8-hbvnv" event={"ID":"308e68aa-b7f2-497d-ad01-017cce437d3c","Type":"ContainerDied","Data":"e646662c71a5a3b1e91340d71ba1171a313fb8310a459291bce2670343727c35"} Jul 23 19:48:46.344485 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:48:46.344260 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-7d09e9f8-hbvnv" event={"ID":"308e68aa-b7f2-497d-ad01-017cce437d3c","Type":"ContainerStarted","Data":"bda5ef528386699df417e4b4dde24738ca11a44fa798e54a00ccc98bcbfb8975"} Jul 23 19:48:47.482512 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:48:47.482482 2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-7d09e9f8-hbvnv" Jul 23 19:48:47.504834 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:48:47.504813 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/308e68aa-b7f2-497d-ad01-017cce437d3c-maas-api-service-ca\") pod \"308e68aa-b7f2-497d-ad01-017cce437d3c\" (UID: \"308e68aa-b7f2-497d-ad01-017cce437d3c\") " Jul 23 19:48:47.505197 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:48:47.505172 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/308e68aa-b7f2-497d-ad01-017cce437d3c-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "308e68aa-b7f2-497d-ad01-017cce437d3c" (UID: "308e68aa-b7f2-497d-ad01-017cce437d3c"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 23 19:48:47.605608 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:48:47.605545 2564 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/308e68aa-b7f2-497d-ad01-017cce437d3c-maas-api-service-ca\") on node \"ip-10-0-129-2.ec2.internal\" DevicePath \"\"" Jul 23 19:48:48.355478 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:48:48.355457 2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-7d09e9f8-hbvnv" Jul 23 19:48:48.355604 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:48:48.355496 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-7d09e9f8-hbvnv" event={"ID":"308e68aa-b7f2-497d-ad01-017cce437d3c","Type":"ContainerDied","Data":"bda5ef528386699df417e4b4dde24738ca11a44fa798e54a00ccc98bcbfb8975"} Jul 23 19:48:48.355604 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:48:48.355533 2564 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="bda5ef528386699df417e4b4dde24738ca11a44fa798e54a00ccc98bcbfb8975" Jul 23 19:48:55.442822 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:48:55.442789 2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-7d09e9f8-7f6b7849c5-p9pf5"] Jul 23 19:48:55.443524 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:48:55.443494 2564 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-7d09e9f8-7f6b7849c5-p9pf5" podUID="f1187ab3-3260-4816-8cad-7375bc85a33a" containerName="maas-api" containerID="cri-o://5f9772cc963c75b314088174911418bcb920c4f2a6fa8352875014c5698b6f76" gracePeriod=30 Jul 23 19:48:55.699677 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:48:55.699615 2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-7d09e9f8-7f6b7849c5-p9pf5" Jul 23 19:48:55.768689 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:48:55.768666 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-dlfsd\" (UniqueName: \"kubernetes.io/projected/f1187ab3-3260-4816-8cad-7375bc85a33a-kube-api-access-dlfsd\") pod \"f1187ab3-3260-4816-8cad-7375bc85a33a\" (UID: \"f1187ab3-3260-4816-8cad-7375bc85a33a\") " Jul 23 19:48:55.768803 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:48:55.768745 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f1187ab3-3260-4816-8cad-7375bc85a33a-openshift-service-ca\") pod \"f1187ab3-3260-4816-8cad-7375bc85a33a\" (UID: \"f1187ab3-3260-4816-8cad-7375bc85a33a\") " Jul 23 19:48:55.768803 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:48:55.768780 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f1187ab3-3260-4816-8cad-7375bc85a33a-maas-api-tls\") pod \"f1187ab3-3260-4816-8cad-7375bc85a33a\" (UID: \"f1187ab3-3260-4816-8cad-7375bc85a33a\") " Jul 23 19:48:55.769046 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:48:55.769021 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f1187ab3-3260-4816-8cad-7375bc85a33a-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "f1187ab3-3260-4816-8cad-7375bc85a33a" (UID: "f1187ab3-3260-4816-8cad-7375bc85a33a"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 23 19:48:55.770807 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:48:55.770779 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f1187ab3-3260-4816-8cad-7375bc85a33a-kube-api-access-dlfsd" (OuterVolumeSpecName: "kube-api-access-dlfsd") pod "f1187ab3-3260-4816-8cad-7375bc85a33a" (UID: "f1187ab3-3260-4816-8cad-7375bc85a33a"). InnerVolumeSpecName "kube-api-access-dlfsd". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 23 19:48:55.770896 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:48:55.770844 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f1187ab3-3260-4816-8cad-7375bc85a33a-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "f1187ab3-3260-4816-8cad-7375bc85a33a" (UID: "f1187ab3-3260-4816-8cad-7375bc85a33a"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 23 19:48:55.870141 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:48:55.870119 2564 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f1187ab3-3260-4816-8cad-7375bc85a33a-maas-api-tls\") on node \"ip-10-0-129-2.ec2.internal\" DevicePath \"\"" Jul 23 19:48:55.870141 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:48:55.870142 2564 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dlfsd\" (UniqueName: \"kubernetes.io/projected/f1187ab3-3260-4816-8cad-7375bc85a33a-kube-api-access-dlfsd\") on node \"ip-10-0-129-2.ec2.internal\" DevicePath \"\"" Jul 23 19:48:55.870256 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:48:55.870151 2564 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f1187ab3-3260-4816-8cad-7375bc85a33a-openshift-service-ca\") on node \"ip-10-0-129-2.ec2.internal\" DevicePath \"\"" Jul 23 19:48:56.405846 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:48:56.405822 2564 generic.go:356] "Generic (PLEG): container finished" podID="f1187ab3-3260-4816-8cad-7375bc85a33a" containerID="5f9772cc963c75b314088174911418bcb920c4f2a6fa8352875014c5698b6f76" exitCode=0 Jul 23 19:48:56.405981 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:48:56.405865 2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-7d09e9f8-7f6b7849c5-p9pf5" Jul 23 19:48:56.405981 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:48:56.405920 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-7d09e9f8-7f6b7849c5-p9pf5" event={"ID":"f1187ab3-3260-4816-8cad-7375bc85a33a","Type":"ContainerDied","Data":"5f9772cc963c75b314088174911418bcb920c4f2a6fa8352875014c5698b6f76"} Jul 23 19:48:56.405981 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:48:56.405970 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-7d09e9f8-7f6b7849c5-p9pf5" event={"ID":"f1187ab3-3260-4816-8cad-7375bc85a33a","Type":"ContainerDied","Data":"0b442ff65dc5d24916be9b57d52b45ba62979a07e330ee3e5a8c27a77c1a2aaf"} Jul 23 19:48:56.406088 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:48:56.405992 2564 scope.go:117] "RemoveContainer" containerID="5f9772cc963c75b314088174911418bcb920c4f2a6fa8352875014c5698b6f76" Jul 23 19:48:56.415594 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:48:56.415580 2564 scope.go:117] "RemoveContainer" containerID="5f9772cc963c75b314088174911418bcb920c4f2a6fa8352875014c5698b6f76" Jul 23 19:48:56.415841 ip-10-0-129-2 kubenswrapper[2564]: E0723 19:48:56.415824 2564 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5f9772cc963c75b314088174911418bcb920c4f2a6fa8352875014c5698b6f76\": container with ID starting with 5f9772cc963c75b314088174911418bcb920c4f2a6fa8352875014c5698b6f76 not found: ID does not exist" containerID="5f9772cc963c75b314088174911418bcb920c4f2a6fa8352875014c5698b6f76" Jul 23 19:48:56.415910 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:48:56.415849 2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5f9772cc963c75b314088174911418bcb920c4f2a6fa8352875014c5698b6f76"} err="failed to get container status \"5f9772cc963c75b314088174911418bcb920c4f2a6fa8352875014c5698b6f76\": rpc error: code = NotFound desc = could not find container \"5f9772cc963c75b314088174911418bcb920c4f2a6fa8352875014c5698b6f76\": container with ID starting with 5f9772cc963c75b314088174911418bcb920c4f2a6fa8352875014c5698b6f76 not found: ID does not exist" Jul 23 19:48:56.426193 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:48:56.426171 2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-7d09e9f8-7f6b7849c5-p9pf5"] Jul 23 19:48:56.431857 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:48:56.431839 2564 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-7d09e9f8-7f6b7849c5-p9pf5"] Jul 23 19:48:56.661859 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:48:56.661805 2564 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="f1187ab3-3260-4816-8cad-7375bc85a33a" path="/var/lib/kubelet/pods/f1187ab3-3260-4816-8cad-7375bc85a33a/volumes" Jul 23 19:48:58.297397 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:48:58.297369 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-2dgmk"] Jul 23 19:49:00.106120 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:49:00.106084 2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-7d09e9f8-hbvnv"] Jul 23 19:49:00.108440 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:49:00.108415 2564 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-7d09e9f8-hbvnv"] Jul 23 19:49:00.657739 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:49:00.657714 2564 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="308e68aa-b7f2-497d-ad01-017cce437d3c" path="/var/lib/kubelet/pods/308e68aa-b7f2-497d-ad01-017cce437d3c/volumes" Jul 23 19:49:03.452199 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:49:03.452156 2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-132311c3-699d679fc9-ps99d"] Jul 23 19:49:03.452727 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:49:03.452707 2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f1187ab3-3260-4816-8cad-7375bc85a33a" containerName="maas-api" Jul 23 19:49:03.452808 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:49:03.452731 2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="f1187ab3-3260-4816-8cad-7375bc85a33a" containerName="maas-api" Jul 23 19:49:03.452808 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:49:03.452743 2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="308e68aa-b7f2-497d-ad01-017cce437d3c" containerName="revoke-keys" Jul 23 19:49:03.452808 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:49:03.452752 2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="308e68aa-b7f2-497d-ad01-017cce437d3c" containerName="revoke-keys" Jul 23 19:49:03.452957 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:49:03.452857 2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="f1187ab3-3260-4816-8cad-7375bc85a33a" containerName="maas-api" Jul 23 19:49:03.452957 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:49:03.452876 2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="308e68aa-b7f2-497d-ad01-017cce437d3c" containerName="revoke-keys" Jul 23 19:49:03.457627 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:49:03.457597 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-132311c3-699d679fc9-ps99d" Jul 23 19:49:03.460199 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:49:03.460140 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ait-132311c3-serving-cert\"" Jul 23 19:49:03.460300 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:49:03.460228 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Jul 23 19:49:03.460300 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:49:03.460251 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Jul 23 19:49:03.460991 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:49:03.460972 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-7m48c\"" Jul 23 19:49:03.476838 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:49:03.476819 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-132311c3-699d679fc9-ps99d"] Jul 23 19:49:03.529962 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:49:03.529938 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/848a13a9-75ed-4781-be94-cd21ddea41b9-openshift-service-ca\") pod \"maas-api-e2e-ait-132311c3-699d679fc9-ps99d\" (UID: \"848a13a9-75ed-4781-be94-cd21ddea41b9\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-132311c3-699d679fc9-ps99d" Jul 23 19:49:03.530049 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:49:03.529968 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/848a13a9-75ed-4781-be94-cd21ddea41b9-maas-api-tls\") pod \"maas-api-e2e-ait-132311c3-699d679fc9-ps99d\" (UID: \"848a13a9-75ed-4781-be94-cd21ddea41b9\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-132311c3-699d679fc9-ps99d" Jul 23 19:49:03.530049 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:49:03.529992 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bzdcf\" (UniqueName: \"kubernetes.io/projected/848a13a9-75ed-4781-be94-cd21ddea41b9-kube-api-access-bzdcf\") pod \"maas-api-e2e-ait-132311c3-699d679fc9-ps99d\" (UID: \"848a13a9-75ed-4781-be94-cd21ddea41b9\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-132311c3-699d679fc9-ps99d" Jul 23 19:49:03.630635 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:49:03.630610 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/848a13a9-75ed-4781-be94-cd21ddea41b9-openshift-service-ca\") pod \"maas-api-e2e-ait-132311c3-699d679fc9-ps99d\" (UID: \"848a13a9-75ed-4781-be94-cd21ddea41b9\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-132311c3-699d679fc9-ps99d" Jul 23 19:49:03.630751 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:49:03.630644 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/848a13a9-75ed-4781-be94-cd21ddea41b9-maas-api-tls\") pod \"maas-api-e2e-ait-132311c3-699d679fc9-ps99d\" (UID: \"848a13a9-75ed-4781-be94-cd21ddea41b9\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-132311c3-699d679fc9-ps99d" Jul 23 19:49:03.630751 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:49:03.630667 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bzdcf\" (UniqueName: \"kubernetes.io/projected/848a13a9-75ed-4781-be94-cd21ddea41b9-kube-api-access-bzdcf\") pod \"maas-api-e2e-ait-132311c3-699d679fc9-ps99d\" (UID: \"848a13a9-75ed-4781-be94-cd21ddea41b9\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-132311c3-699d679fc9-ps99d" Jul 23 19:49:03.630834 ip-10-0-129-2 kubenswrapper[2564]: E0723 19:49:03.630788 2564 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-ait-132311c3-serving-cert: secret "maas-api-e2e-ait-132311c3-serving-cert" not found Jul 23 19:49:03.630920 ip-10-0-129-2 kubenswrapper[2564]: E0723 19:49:03.630864 2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/848a13a9-75ed-4781-be94-cd21ddea41b9-maas-api-tls podName:848a13a9-75ed-4781-be94-cd21ddea41b9 nodeName:}" failed. No retries permitted until 2026-07-23 19:49:04.13084523 +0000 UTC m=+1769.982621455 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/848a13a9-75ed-4781-be94-cd21ddea41b9-maas-api-tls") pod "maas-api-e2e-ait-132311c3-699d679fc9-ps99d" (UID: "848a13a9-75ed-4781-be94-cd21ddea41b9") : secret "maas-api-e2e-ait-132311c3-serving-cert" not found Jul 23 19:49:03.631215 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:49:03.631198 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/848a13a9-75ed-4781-be94-cd21ddea41b9-openshift-service-ca\") pod \"maas-api-e2e-ait-132311c3-699d679fc9-ps99d\" (UID: \"848a13a9-75ed-4781-be94-cd21ddea41b9\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-132311c3-699d679fc9-ps99d" Jul 23 19:49:03.643315 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:49:03.643293 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-bzdcf\" (UniqueName: \"kubernetes.io/projected/848a13a9-75ed-4781-be94-cd21ddea41b9-kube-api-access-bzdcf\") pod \"maas-api-e2e-ait-132311c3-699d679fc9-ps99d\" (UID: \"848a13a9-75ed-4781-be94-cd21ddea41b9\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-132311c3-699d679fc9-ps99d" Jul 23 19:49:04.134203 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:49:04.134177 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/848a13a9-75ed-4781-be94-cd21ddea41b9-maas-api-tls\") pod \"maas-api-e2e-ait-132311c3-699d679fc9-ps99d\" (UID: \"848a13a9-75ed-4781-be94-cd21ddea41b9\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-132311c3-699d679fc9-ps99d" Jul 23 19:49:04.136646 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:49:04.136623 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/848a13a9-75ed-4781-be94-cd21ddea41b9-maas-api-tls\") pod \"maas-api-e2e-ait-132311c3-699d679fc9-ps99d\" (UID: \"848a13a9-75ed-4781-be94-cd21ddea41b9\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-132311c3-699d679fc9-ps99d" Jul 23 19:49:04.368169 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:49:04.368140 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-132311c3-699d679fc9-ps99d" Jul 23 19:49:04.508888 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:49:04.508865 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-132311c3-699d679fc9-ps99d"] Jul 23 19:49:04.511251 ip-10-0-129-2 kubenswrapper[2564]: W0723 19:49:04.511231 2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod848a13a9_75ed_4781_be94_cd21ddea41b9.slice/crio-ef429efb30c6747182b9dec422ac1fef4fa0bbeb6c56518995829f495fb850ea WatchSource:0}: Error finding container ef429efb30c6747182b9dec422ac1fef4fa0bbeb6c56518995829f495fb850ea: Status 404 returned error can't find the container with id ef429efb30c6747182b9dec422ac1fef4fa0bbeb6c56518995829f495fb850ea Jul 23 19:49:05.457485 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:49:05.457459 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-132311c3-699d679fc9-ps99d" event={"ID":"848a13a9-75ed-4781-be94-cd21ddea41b9","Type":"ContainerStarted","Data":"9489b6907ec02d28f1c20b2c0c7fc8895d88c06d726af683fe4d8ea533351778"} Jul 23 19:49:05.457664 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:49:05.457507 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-132311c3-699d679fc9-ps99d" event={"ID":"848a13a9-75ed-4781-be94-cd21ddea41b9","Type":"ContainerStarted","Data":"ef429efb30c6747182b9dec422ac1fef4fa0bbeb6c56518995829f495fb850ea"} Jul 23 19:49:05.457664 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:49:05.457574 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-132311c3-699d679fc9-ps99d" Jul 23 19:49:05.477599 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:49:05.477550 2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ait-132311c3-699d679fc9-ps99d" podStartSLOduration=2.099562285 podStartE2EDuration="2.477538228s" totalImagesPullingTime="377.975943ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-23 19:49:03 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-23 19:49:05.475483244 +0000 UTC m=+1771.327259492" watchObservedRunningTime="2026-07-23 19:49:05.477538228 +0000 UTC m=+1771.329314475" Jul 23 19:49:10.098913 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:49:10.098877 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-2dgmk"] Jul 23 19:49:11.467397 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:49:11.467371 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-132311c3-699d679fc9-ps99d" Jul 23 19:49:28.939616 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:49:28.939541 2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-132311c3-2x5dl"] Jul 23 19:49:28.943242 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:49:28.943224 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-132311c3-2x5dl" Jul 23 19:49:28.945555 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:49:28.945536 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-gcspk\"" Jul 23 19:49:28.954234 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:49:28.954209 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-132311c3-2x5dl"] Jul 23 19:49:29.018195 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:49:29.018171 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/69cfb701-7993-4386-a60e-f6fa0d274a36-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-132311c3-2x5dl\" (UID: \"69cfb701-7993-4386-a60e-f6fa0d274a36\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-132311c3-2x5dl" Jul 23 19:49:29.118552 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:49:29.118529 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/69cfb701-7993-4386-a60e-f6fa0d274a36-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-132311c3-2x5dl\" (UID: \"69cfb701-7993-4386-a60e-f6fa0d274a36\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-132311c3-2x5dl" Jul 23 19:49:29.119277 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:49:29.119256 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/69cfb701-7993-4386-a60e-f6fa0d274a36-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-132311c3-2x5dl\" (UID: \"69cfb701-7993-4386-a60e-f6fa0d274a36\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-132311c3-2x5dl" Jul 23 19:49:29.253633 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:49:29.253572 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-132311c3-2x5dl" Jul 23 19:49:29.592658 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:49:29.592613 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-132311c3-2x5dl"] Jul 23 19:49:29.593809 ip-10-0-129-2 kubenswrapper[2564]: W0723 19:49:29.593773 2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod69cfb701_7993_4386_a60e_f6fa0d274a36.slice/crio-ae3e214a953b84b9f748ca92d498b14b5492c1e48244ffb73a7f8167084fe7bb WatchSource:0}: Error finding container ae3e214a953b84b9f748ca92d498b14b5492c1e48244ffb73a7f8167084fe7bb: Status 404 returned error can't find the container with id ae3e214a953b84b9f748ca92d498b14b5492c1e48244ffb73a7f8167084fe7bb Jul 23 19:49:30.593452 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:49:30.593386 2564 generic.go:356] "Generic (PLEG): container finished" podID="69cfb701-7993-4386-a60e-f6fa0d274a36" containerID="c770abe5b74d6ba95c15a15cf56f2e7ae0c1ed8be05eab2d552c9d038a4ebd29" exitCode=0 Jul 23 19:49:30.593452 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:49:30.593437 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-132311c3-2x5dl" event={"ID":"69cfb701-7993-4386-a60e-f6fa0d274a36","Type":"ContainerDied","Data":"c770abe5b74d6ba95c15a15cf56f2e7ae0c1ed8be05eab2d552c9d038a4ebd29"} Jul 23 19:49:30.593775 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:49:30.593500 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-132311c3-2x5dl" event={"ID":"69cfb701-7993-4386-a60e-f6fa0d274a36","Type":"ContainerStarted","Data":"ae3e214a953b84b9f748ca92d498b14b5492c1e48244ffb73a7f8167084fe7bb"} Jul 23 19:49:31.731886 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:49:31.731865 2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-132311c3-2x5dl" Jul 23 19:49:31.843037 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:49:31.843014 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/69cfb701-7993-4386-a60e-f6fa0d274a36-maas-api-service-ca\") pod \"69cfb701-7993-4386-a60e-f6fa0d274a36\" (UID: \"69cfb701-7993-4386-a60e-f6fa0d274a36\") " Jul 23 19:49:31.843343 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:49:31.843307 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/69cfb701-7993-4386-a60e-f6fa0d274a36-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "69cfb701-7993-4386-a60e-f6fa0d274a36" (UID: "69cfb701-7993-4386-a60e-f6fa0d274a36"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 23 19:49:31.944380 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:49:31.944357 2564 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/69cfb701-7993-4386-a60e-f6fa0d274a36-maas-api-service-ca\") on node \"ip-10-0-129-2.ec2.internal\" DevicePath \"\"" Jul 23 19:49:32.605091 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:49:32.605069 2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-132311c3-2x5dl" Jul 23 19:49:32.605209 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:49:32.605091 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-132311c3-2x5dl" event={"ID":"69cfb701-7993-4386-a60e-f6fa0d274a36","Type":"ContainerDied","Data":"ae3e214a953b84b9f748ca92d498b14b5492c1e48244ffb73a7f8167084fe7bb"} Jul 23 19:49:32.605209 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:49:32.605137 2564 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="ae3e214a953b84b9f748ca92d498b14b5492c1e48244ffb73a7f8167084fe7bb" Jul 23 19:49:35.528069 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:49:35.528044 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-565b7b569d-z5wf4_ded5b3af-c74c-4d5a-aa77-62fe72d1dd29/console-operator/1.log" Jul 23 19:49:35.531944 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:49:35.531921 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-565b7b569d-z5wf4_ded5b3af-c74c-4d5a-aa77-62fe72d1dd29/console-operator/1.log" Jul 23 19:49:35.535271 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:49:35.535249 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-8dtlh_195ecf89-13ae-4311-bc72-fcdf78a5c4d6/ovn-acl-logging/0.log" Jul 23 19:49:35.538787 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:49:35.538772 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-8dtlh_195ecf89-13ae-4311-bc72-fcdf78a5c4d6/ovn-acl-logging/0.log" Jul 23 19:49:39.466411 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:49:39.466378 2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-132311c3-699d679fc9-ps99d"] Jul 23 19:49:39.467264 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:49:39.467230 2564 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ait-132311c3-699d679fc9-ps99d" podUID="848a13a9-75ed-4781-be94-cd21ddea41b9" containerName="maas-api" containerID="cri-o://9489b6907ec02d28f1c20b2c0c7fc8895d88c06d726af683fe4d8ea533351778" gracePeriod=30 Jul 23 19:49:39.644004 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:49:39.643959 2564 generic.go:356] "Generic (PLEG): container finished" podID="848a13a9-75ed-4781-be94-cd21ddea41b9" containerID="9489b6907ec02d28f1c20b2c0c7fc8895d88c06d726af683fe4d8ea533351778" exitCode=0 Jul 23 19:49:39.644135 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:49:39.644107 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-132311c3-699d679fc9-ps99d" event={"ID":"848a13a9-75ed-4781-be94-cd21ddea41b9","Type":"ContainerDied","Data":"9489b6907ec02d28f1c20b2c0c7fc8895d88c06d726af683fe4d8ea533351778"} Jul 23 19:49:39.716208 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:49:39.716185 2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-132311c3-699d679fc9-ps99d" Jul 23 19:49:39.804238 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:49:39.804176 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/848a13a9-75ed-4781-be94-cd21ddea41b9-maas-api-tls\") pod \"848a13a9-75ed-4781-be94-cd21ddea41b9\" (UID: \"848a13a9-75ed-4781-be94-cd21ddea41b9\") " Jul 23 19:49:39.804238 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:49:39.804209 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-bzdcf\" (UniqueName: \"kubernetes.io/projected/848a13a9-75ed-4781-be94-cd21ddea41b9-kube-api-access-bzdcf\") pod \"848a13a9-75ed-4781-be94-cd21ddea41b9\" (UID: \"848a13a9-75ed-4781-be94-cd21ddea41b9\") " Jul 23 19:49:39.804419 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:49:39.804288 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/848a13a9-75ed-4781-be94-cd21ddea41b9-openshift-service-ca\") pod \"848a13a9-75ed-4781-be94-cd21ddea41b9\" (UID: \"848a13a9-75ed-4781-be94-cd21ddea41b9\") " Jul 23 19:49:39.804687 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:49:39.804666 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/848a13a9-75ed-4781-be94-cd21ddea41b9-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "848a13a9-75ed-4781-be94-cd21ddea41b9" (UID: "848a13a9-75ed-4781-be94-cd21ddea41b9"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 23 19:49:39.806397 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:49:39.806376 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/848a13a9-75ed-4781-be94-cd21ddea41b9-kube-api-access-bzdcf" (OuterVolumeSpecName: "kube-api-access-bzdcf") pod "848a13a9-75ed-4781-be94-cd21ddea41b9" (UID: "848a13a9-75ed-4781-be94-cd21ddea41b9"). InnerVolumeSpecName "kube-api-access-bzdcf". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 23 19:49:39.806397 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:49:39.806388 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/848a13a9-75ed-4781-be94-cd21ddea41b9-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "848a13a9-75ed-4781-be94-cd21ddea41b9" (UID: "848a13a9-75ed-4781-be94-cd21ddea41b9"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 23 19:49:39.905643 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:49:39.905611 2564 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/848a13a9-75ed-4781-be94-cd21ddea41b9-openshift-service-ca\") on node \"ip-10-0-129-2.ec2.internal\" DevicePath \"\"" Jul 23 19:49:39.905739 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:49:39.905658 2564 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/848a13a9-75ed-4781-be94-cd21ddea41b9-maas-api-tls\") on node \"ip-10-0-129-2.ec2.internal\" DevicePath \"\"" Jul 23 19:49:39.905739 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:49:39.905668 2564 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bzdcf\" (UniqueName: \"kubernetes.io/projected/848a13a9-75ed-4781-be94-cd21ddea41b9-kube-api-access-bzdcf\") on node \"ip-10-0-129-2.ec2.internal\" DevicePath \"\"" Jul 23 19:49:40.650490 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:49:40.650467 2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-132311c3-699d679fc9-ps99d" Jul 23 19:49:40.650883 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:49:40.650497 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-132311c3-699d679fc9-ps99d" event={"ID":"848a13a9-75ed-4781-be94-cd21ddea41b9","Type":"ContainerDied","Data":"ef429efb30c6747182b9dec422ac1fef4fa0bbeb6c56518995829f495fb850ea"} Jul 23 19:49:40.650883 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:49:40.650576 2564 scope.go:117] "RemoveContainer" containerID="9489b6907ec02d28f1c20b2c0c7fc8895d88c06d726af683fe4d8ea533351778" Jul 23 19:49:40.673373 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:49:40.673343 2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-132311c3-699d679fc9-ps99d"] Jul 23 19:49:40.680379 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:49:40.680358 2564 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-132311c3-699d679fc9-ps99d"] Jul 23 19:49:42.286615 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:49:42.286588 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-2dgmk"] Jul 23 19:49:42.658276 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:49:42.658203 2564 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="848a13a9-75ed-4781-be94-cd21ddea41b9" path="/var/lib/kubelet/pods/848a13a9-75ed-4781-be94-cd21ddea41b9/volumes" Jul 23 19:49:44.012421 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:49:44.012393 2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-132311c3-2x5dl"] Jul 23 19:49:44.015534 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:49:44.015513 2564 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-132311c3-2x5dl"] Jul 23 19:49:44.657935 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:49:44.657909 2564 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="69cfb701-7993-4386-a60e-f6fa0d274a36" path="/var/lib/kubelet/pods/69cfb701-7993-4386-a60e-f6fa0d274a36/volumes" Jul 23 19:49:48.596215 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:49:48.596175 2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-f7b99782-5bb7499f97-8m8xb"] Jul 23 19:49:48.596708 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:49:48.596595 2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="848a13a9-75ed-4781-be94-cd21ddea41b9" containerName="maas-api" Jul 23 19:49:48.596708 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:49:48.596607 2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="848a13a9-75ed-4781-be94-cd21ddea41b9" containerName="maas-api" Jul 23 19:49:48.596708 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:49:48.596621 2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="69cfb701-7993-4386-a60e-f6fa0d274a36" containerName="revoke-keys" Jul 23 19:49:48.596708 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:49:48.596630 2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="69cfb701-7993-4386-a60e-f6fa0d274a36" containerName="revoke-keys" Jul 23 19:49:48.596708 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:49:48.596693 2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="848a13a9-75ed-4781-be94-cd21ddea41b9" containerName="maas-api" Jul 23 19:49:48.596708 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:49:48.596709 2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="69cfb701-7993-4386-a60e-f6fa0d274a36" containerName="revoke-keys" Jul 23 19:49:48.601005 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:49:48.600986 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-derive-f7b99782-5bb7499f97-8m8xb" Jul 23 19:49:48.603182 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:49:48.603156 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-derive-f7b99782-serving-cert\"" Jul 23 19:49:48.603344 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:49:48.603238 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Jul 23 19:49:48.604001 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:49:48.603981 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Jul 23 19:49:48.604125 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:49:48.603981 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-7m48c\"" Jul 23 19:49:48.611632 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:49:48.611609 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-f7b99782-5bb7499f97-8m8xb"] Jul 23 19:49:48.674655 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:49:48.674635 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mftlt\" (UniqueName: \"kubernetes.io/projected/81bbf865-1114-4dd5-bd21-034aebaed0e6-kube-api-access-mftlt\") pod \"maas-api-e2e-derive-f7b99782-5bb7499f97-8m8xb\" (UID: \"81bbf865-1114-4dd5-bd21-034aebaed0e6\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-f7b99782-5bb7499f97-8m8xb" Jul 23 19:49:48.674771 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:49:48.674668 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/81bbf865-1114-4dd5-bd21-034aebaed0e6-openshift-service-ca\") pod \"maas-api-e2e-derive-f7b99782-5bb7499f97-8m8xb\" (UID: \"81bbf865-1114-4dd5-bd21-034aebaed0e6\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-f7b99782-5bb7499f97-8m8xb" Jul 23 19:49:48.674771 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:49:48.674728 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/81bbf865-1114-4dd5-bd21-034aebaed0e6-maas-api-tls\") pod \"maas-api-e2e-derive-f7b99782-5bb7499f97-8m8xb\" (UID: \"81bbf865-1114-4dd5-bd21-034aebaed0e6\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-f7b99782-5bb7499f97-8m8xb" Jul 23 19:49:48.775695 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:49:48.775670 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/81bbf865-1114-4dd5-bd21-034aebaed0e6-maas-api-tls\") pod \"maas-api-e2e-derive-f7b99782-5bb7499f97-8m8xb\" (UID: \"81bbf865-1114-4dd5-bd21-034aebaed0e6\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-f7b99782-5bb7499f97-8m8xb" Jul 23 19:49:48.775843 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:49:48.775741 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mftlt\" (UniqueName: \"kubernetes.io/projected/81bbf865-1114-4dd5-bd21-034aebaed0e6-kube-api-access-mftlt\") pod \"maas-api-e2e-derive-f7b99782-5bb7499f97-8m8xb\" (UID: \"81bbf865-1114-4dd5-bd21-034aebaed0e6\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-f7b99782-5bb7499f97-8m8xb" Jul 23 19:49:48.775843 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:49:48.775792 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/81bbf865-1114-4dd5-bd21-034aebaed0e6-openshift-service-ca\") pod \"maas-api-e2e-derive-f7b99782-5bb7499f97-8m8xb\" (UID: \"81bbf865-1114-4dd5-bd21-034aebaed0e6\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-f7b99782-5bb7499f97-8m8xb" Jul 23 19:49:48.775843 ip-10-0-129-2 kubenswrapper[2564]: E0723 19:49:48.775820 2564 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-derive-f7b99782-serving-cert: secret "maas-api-e2e-derive-f7b99782-serving-cert" not found Jul 23 19:49:48.776040 ip-10-0-129-2 kubenswrapper[2564]: E0723 19:49:48.775894 2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/81bbf865-1114-4dd5-bd21-034aebaed0e6-maas-api-tls podName:81bbf865-1114-4dd5-bd21-034aebaed0e6 nodeName:}" failed. No retries permitted until 2026-07-23 19:49:49.275875364 +0000 UTC m=+1815.127651593 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/81bbf865-1114-4dd5-bd21-034aebaed0e6-maas-api-tls") pod "maas-api-e2e-derive-f7b99782-5bb7499f97-8m8xb" (UID: "81bbf865-1114-4dd5-bd21-034aebaed0e6") : secret "maas-api-e2e-derive-f7b99782-serving-cert" not found Jul 23 19:49:48.776511 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:49:48.776489 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/81bbf865-1114-4dd5-bd21-034aebaed0e6-openshift-service-ca\") pod \"maas-api-e2e-derive-f7b99782-5bb7499f97-8m8xb\" (UID: \"81bbf865-1114-4dd5-bd21-034aebaed0e6\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-f7b99782-5bb7499f97-8m8xb" Jul 23 19:49:48.784239 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:49:48.784212 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-mftlt\" (UniqueName: \"kubernetes.io/projected/81bbf865-1114-4dd5-bd21-034aebaed0e6-kube-api-access-mftlt\") pod \"maas-api-e2e-derive-f7b99782-5bb7499f97-8m8xb\" (UID: \"81bbf865-1114-4dd5-bd21-034aebaed0e6\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-f7b99782-5bb7499f97-8m8xb" Jul 23 19:49:49.279472 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:49:49.279433 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/81bbf865-1114-4dd5-bd21-034aebaed0e6-maas-api-tls\") pod \"maas-api-e2e-derive-f7b99782-5bb7499f97-8m8xb\" (UID: \"81bbf865-1114-4dd5-bd21-034aebaed0e6\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-f7b99782-5bb7499f97-8m8xb" Jul 23 19:49:49.281930 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:49:49.281913 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/81bbf865-1114-4dd5-bd21-034aebaed0e6-maas-api-tls\") pod \"maas-api-e2e-derive-f7b99782-5bb7499f97-8m8xb\" (UID: \"81bbf865-1114-4dd5-bd21-034aebaed0e6\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-f7b99782-5bb7499f97-8m8xb" Jul 23 19:49:49.513984 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:49:49.513951 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-derive-f7b99782-5bb7499f97-8m8xb" Jul 23 19:49:49.649012 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:49:49.648985 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-f7b99782-5bb7499f97-8m8xb"] Jul 23 19:49:49.650781 ip-10-0-129-2 kubenswrapper[2564]: W0723 19:49:49.650756 2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod81bbf865_1114_4dd5_bd21_034aebaed0e6.slice/crio-6ae42a5d61c06adb8ed4ed4573fd1e1704a452a1ea52a5691efebe807a4d9537 WatchSource:0}: Error finding container 6ae42a5d61c06adb8ed4ed4573fd1e1704a452a1ea52a5691efebe807a4d9537: Status 404 returned error can't find the container with id 6ae42a5d61c06adb8ed4ed4573fd1e1704a452a1ea52a5691efebe807a4d9537 Jul 23 19:49:49.699672 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:49:49.699640 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-derive-f7b99782-5bb7499f97-8m8xb" event={"ID":"81bbf865-1114-4dd5-bd21-034aebaed0e6","Type":"ContainerStarted","Data":"6ae42a5d61c06adb8ed4ed4573fd1e1704a452a1ea52a5691efebe807a4d9537"} Jul 23 19:49:50.705739 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:49:50.705707 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-derive-f7b99782-5bb7499f97-8m8xb" event={"ID":"81bbf865-1114-4dd5-bd21-034aebaed0e6","Type":"ContainerStarted","Data":"a86148d92cddf4bff82972d5343f94f51f08167019a4443dc3dcb6292f338383"} Jul 23 19:49:50.706256 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:49:50.705821 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-derive-f7b99782-5bb7499f97-8m8xb" Jul 23 19:49:50.726473 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:49:50.726434 2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-derive-f7b99782-5bb7499f97-8m8xb" podStartSLOduration=2.289392489 podStartE2EDuration="2.726420783s" totalImagesPullingTime="437.028294ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-23 19:49:48 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-23 19:49:50.72356137 +0000 UTC m=+1816.575337616" watchObservedRunningTime="2026-07-23 19:49:50.726420783 +0000 UTC m=+1816.578197030" Jul 23 19:49:54.798698 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:49:54.798665 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-2dgmk"] Jul 23 19:49:56.715711 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:49:56.715683 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-derive-f7b99782-5bb7499f97-8m8xb" Jul 23 19:50:13.195717 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:50:13.195684 2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-f7b99782-kqwww"] Jul 23 19:50:13.199128 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:50:13.199110 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-f7b99782-kqwww" Jul 23 19:50:13.201489 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:50:13.201458 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-gcspk\"" Jul 23 19:50:13.205604 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:50:13.205581 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-f7b99782-kqwww"] Jul 23 19:50:13.262173 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:50:13.262147 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/cdb08c30-21f0-447e-ba46-b610f6234b96-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-derive-f7b99782-kqwww\" (UID: \"cdb08c30-21f0-447e-ba46-b610f6234b96\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-f7b99782-kqwww" Jul 23 19:50:13.363041 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:50:13.363018 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/cdb08c30-21f0-447e-ba46-b610f6234b96-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-derive-f7b99782-kqwww\" (UID: \"cdb08c30-21f0-447e-ba46-b610f6234b96\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-f7b99782-kqwww" Jul 23 19:50:13.363591 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:50:13.363574 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/cdb08c30-21f0-447e-ba46-b610f6234b96-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-derive-f7b99782-kqwww\" (UID: \"cdb08c30-21f0-447e-ba46-b610f6234b96\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-f7b99782-kqwww" Jul 23 19:50:13.510416 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:50:13.510366 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-f7b99782-kqwww" Jul 23 19:50:13.843097 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:50:13.843071 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-f7b99782-kqwww"] Jul 23 19:50:13.844495 ip-10-0-129-2 kubenswrapper[2564]: W0723 19:50:13.844464 2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcdb08c30_21f0_447e_ba46_b610f6234b96.slice/crio-d45df9689fe400da408b4fb5615d2d5a89b8a15da25f4e2d36b0e3d855279ea6 WatchSource:0}: Error finding container d45df9689fe400da408b4fb5615d2d5a89b8a15da25f4e2d36b0e3d855279ea6: Status 404 returned error can't find the container with id d45df9689fe400da408b4fb5615d2d5a89b8a15da25f4e2d36b0e3d855279ea6 Jul 23 19:50:14.836187 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:50:14.836118 2564 generic.go:356] "Generic (PLEG): container finished" podID="cdb08c30-21f0-447e-ba46-b610f6234b96" containerID="bb2c26148286d0b2354cf76b788f061f4776ee19bb1f851201b56684b2f0718c" exitCode=0 Jul 23 19:50:14.836187 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:50:14.836165 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-f7b99782-kqwww" event={"ID":"cdb08c30-21f0-447e-ba46-b610f6234b96","Type":"ContainerDied","Data":"bb2c26148286d0b2354cf76b788f061f4776ee19bb1f851201b56684b2f0718c"} Jul 23 19:50:14.836558 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:50:14.836209 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-f7b99782-kqwww" event={"ID":"cdb08c30-21f0-447e-ba46-b610f6234b96","Type":"ContainerStarted","Data":"d45df9689fe400da408b4fb5615d2d5a89b8a15da25f4e2d36b0e3d855279ea6"} Jul 23 19:50:15.971555 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:50:15.971533 2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-f7b99782-kqwww" Jul 23 19:50:15.982334 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:50:15.982300 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/cdb08c30-21f0-447e-ba46-b610f6234b96-maas-api-service-ca\") pod \"cdb08c30-21f0-447e-ba46-b610f6234b96\" (UID: \"cdb08c30-21f0-447e-ba46-b610f6234b96\") " Jul 23 19:50:15.982567 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:50:15.982547 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cdb08c30-21f0-447e-ba46-b610f6234b96-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "cdb08c30-21f0-447e-ba46-b610f6234b96" (UID: "cdb08c30-21f0-447e-ba46-b610f6234b96"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 23 19:50:16.083003 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:50:16.082980 2564 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/cdb08c30-21f0-447e-ba46-b610f6234b96-maas-api-service-ca\") on node \"ip-10-0-129-2.ec2.internal\" DevicePath \"\"" Jul 23 19:50:16.848335 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:50:16.848295 2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-f7b99782-kqwww" Jul 23 19:50:16.848444 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:50:16.848346 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-f7b99782-kqwww" event={"ID":"cdb08c30-21f0-447e-ba46-b610f6234b96","Type":"ContainerDied","Data":"d45df9689fe400da408b4fb5615d2d5a89b8a15da25f4e2d36b0e3d855279ea6"} Jul 23 19:50:16.848444 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:50:16.848393 2564 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="d45df9689fe400da408b4fb5615d2d5a89b8a15da25f4e2d36b0e3d855279ea6" Jul 23 19:50:23.751851 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:50:23.751815 2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-f7b99782-5bb7499f97-8m8xb"] Jul 23 19:50:23.752428 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:50:23.752019 2564 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-derive-f7b99782-5bb7499f97-8m8xb" podUID="81bbf865-1114-4dd5-bd21-034aebaed0e6" containerName="maas-api" containerID="cri-o://a86148d92cddf4bff82972d5343f94f51f08167019a4443dc3dcb6292f338383" gracePeriod=30 Jul 23 19:50:23.890989 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:50:23.890940 2564 generic.go:356] "Generic (PLEG): container finished" podID="81bbf865-1114-4dd5-bd21-034aebaed0e6" containerID="a86148d92cddf4bff82972d5343f94f51f08167019a4443dc3dcb6292f338383" exitCode=0 Jul 23 19:50:23.891151 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:50:23.890974 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-derive-f7b99782-5bb7499f97-8m8xb" event={"ID":"81bbf865-1114-4dd5-bd21-034aebaed0e6","Type":"ContainerDied","Data":"a86148d92cddf4bff82972d5343f94f51f08167019a4443dc3dcb6292f338383"} Jul 23 19:50:24.006628 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:50:24.006574 2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-derive-f7b99782-5bb7499f97-8m8xb" Jul 23 19:50:24.040782 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:50:24.040762 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/81bbf865-1114-4dd5-bd21-034aebaed0e6-maas-api-tls\") pod \"81bbf865-1114-4dd5-bd21-034aebaed0e6\" (UID: \"81bbf865-1114-4dd5-bd21-034aebaed0e6\") " Jul 23 19:50:24.040895 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:50:24.040798 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/81bbf865-1114-4dd5-bd21-034aebaed0e6-openshift-service-ca\") pod \"81bbf865-1114-4dd5-bd21-034aebaed0e6\" (UID: \"81bbf865-1114-4dd5-bd21-034aebaed0e6\") " Jul 23 19:50:24.040895 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:50:24.040840 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-mftlt\" (UniqueName: \"kubernetes.io/projected/81bbf865-1114-4dd5-bd21-034aebaed0e6-kube-api-access-mftlt\") pod \"81bbf865-1114-4dd5-bd21-034aebaed0e6\" (UID: \"81bbf865-1114-4dd5-bd21-034aebaed0e6\") " Jul 23 19:50:24.041156 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:50:24.041134 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/81bbf865-1114-4dd5-bd21-034aebaed0e6-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "81bbf865-1114-4dd5-bd21-034aebaed0e6" (UID: "81bbf865-1114-4dd5-bd21-034aebaed0e6"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 23 19:50:24.043012 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:50:24.042983 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/81bbf865-1114-4dd5-bd21-034aebaed0e6-kube-api-access-mftlt" (OuterVolumeSpecName: "kube-api-access-mftlt") pod "81bbf865-1114-4dd5-bd21-034aebaed0e6" (UID: "81bbf865-1114-4dd5-bd21-034aebaed0e6"). InnerVolumeSpecName "kube-api-access-mftlt". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 23 19:50:24.043120 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:50:24.042989 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/81bbf865-1114-4dd5-bd21-034aebaed0e6-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "81bbf865-1114-4dd5-bd21-034aebaed0e6" (UID: "81bbf865-1114-4dd5-bd21-034aebaed0e6"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 23 19:50:24.141793 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:50:24.141771 2564 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/81bbf865-1114-4dd5-bd21-034aebaed0e6-maas-api-tls\") on node \"ip-10-0-129-2.ec2.internal\" DevicePath \"\"" Jul 23 19:50:24.141793 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:50:24.141792 2564 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/81bbf865-1114-4dd5-bd21-034aebaed0e6-openshift-service-ca\") on node \"ip-10-0-129-2.ec2.internal\" DevicePath \"\"" Jul 23 19:50:24.141923 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:50:24.141802 2564 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mftlt\" (UniqueName: \"kubernetes.io/projected/81bbf865-1114-4dd5-bd21-034aebaed0e6-kube-api-access-mftlt\") on node \"ip-10-0-129-2.ec2.internal\" DevicePath \"\"" Jul 23 19:50:24.897610 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:50:24.897584 2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-derive-f7b99782-5bb7499f97-8m8xb" Jul 23 19:50:24.897947 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:50:24.897617 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-derive-f7b99782-5bb7499f97-8m8xb" event={"ID":"81bbf865-1114-4dd5-bd21-034aebaed0e6","Type":"ContainerDied","Data":"6ae42a5d61c06adb8ed4ed4573fd1e1704a452a1ea52a5691efebe807a4d9537"} Jul 23 19:50:24.897947 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:50:24.897678 2564 scope.go:117] "RemoveContainer" containerID="a86148d92cddf4bff82972d5343f94f51f08167019a4443dc3dcb6292f338383" Jul 23 19:50:24.913661 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:50:24.913641 2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-f7b99782-5bb7499f97-8m8xb"] Jul 23 19:50:24.917650 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:50:24.917592 2564 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-f7b99782-5bb7499f97-8m8xb"] Jul 23 19:50:26.595678 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:50:26.595648 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-2dgmk"] Jul 23 19:50:26.657945 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:50:26.657918 2564 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="81bbf865-1114-4dd5-bd21-034aebaed0e6" path="/var/lib/kubelet/pods/81bbf865-1114-4dd5-bd21-034aebaed0e6/volumes" Jul 23 19:50:28.270470 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:50:28.270440 2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-f7b99782-kqwww"] Jul 23 19:50:28.272601 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:50:28.272578 2564 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-f7b99782-kqwww"] Jul 23 19:50:28.658340 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:50:28.658252 2564 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="cdb08c30-21f0-447e-ba46-b610f6234b96" path="/var/lib/kubelet/pods/cdb08c30-21f0-447e-ba46-b610f6234b96/volumes" Jul 23 19:51:53.304000 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:51:53.303960 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-2dgmk"] Jul 23 19:52:02.987484 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:52:02.987443 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-2dgmk"] Jul 23 19:52:13.138364 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:52:13.138315 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-2dgmk"] Jul 23 19:52:20.273891 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:52:20.273857 2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-536fc4c1-6hmwc"] Jul 23 19:52:20.274256 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:52:20.274241 2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="cdb08c30-21f0-447e-ba46-b610f6234b96" containerName="revoke-keys" Jul 23 19:52:20.274300 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:52:20.274258 2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="cdb08c30-21f0-447e-ba46-b610f6234b96" containerName="revoke-keys" Jul 23 19:52:20.274300 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:52:20.274270 2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="81bbf865-1114-4dd5-bd21-034aebaed0e6" containerName="maas-api" Jul 23 19:52:20.274300 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:52:20.274276 2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="81bbf865-1114-4dd5-bd21-034aebaed0e6" containerName="maas-api" Jul 23 19:52:20.274417 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:52:20.274367 2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="cdb08c30-21f0-447e-ba46-b610f6234b96" containerName="revoke-keys" Jul 23 19:52:20.274417 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:52:20.274376 2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="81bbf865-1114-4dd5-bd21-034aebaed0e6" containerName="maas-api" Jul 23 19:52:20.277287 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:52:20.277272 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-536fc4c1-6hmwc" Jul 23 19:52:20.279910 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:52:20.279886 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-gcspk\"" Jul 23 19:52:20.279910 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:52:20.279905 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Jul 23 19:52:20.285757 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:52:20.285735 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-536fc4c1-6hmwc"] Jul 23 19:52:20.306007 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:52:20.305982 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/435e7a19-84db-498c-8087-cfe2e65ca747-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-mt-536fc4c1-6hmwc\" (UID: \"435e7a19-84db-498c-8087-cfe2e65ca747\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-536fc4c1-6hmwc" Jul 23 19:52:20.407229 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:52:20.407206 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/435e7a19-84db-498c-8087-cfe2e65ca747-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-mt-536fc4c1-6hmwc\" (UID: \"435e7a19-84db-498c-8087-cfe2e65ca747\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-536fc4c1-6hmwc" Jul 23 19:52:20.407766 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:52:20.407749 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/435e7a19-84db-498c-8087-cfe2e65ca747-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-mt-536fc4c1-6hmwc\" (UID: \"435e7a19-84db-498c-8087-cfe2e65ca747\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-536fc4c1-6hmwc" Jul 23 19:52:20.588852 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:52:20.588800 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-536fc4c1-6hmwc" Jul 23 19:52:20.722017 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:52:20.721986 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-536fc4c1-6hmwc"] Jul 23 19:52:20.723216 ip-10-0-129-2 kubenswrapper[2564]: W0723 19:52:20.723190 2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod435e7a19_84db_498c_8087_cfe2e65ca747.slice/crio-6a10d79f2961f72c91920fe75bf7e9b43dd3affb44b13b77f9e24ccb5c72078c WatchSource:0}: Error finding container 6a10d79f2961f72c91920fe75bf7e9b43dd3affb44b13b77f9e24ccb5c72078c: Status 404 returned error can't find the container with id 6a10d79f2961f72c91920fe75bf7e9b43dd3affb44b13b77f9e24ccb5c72078c Jul 23 19:52:21.547331 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:52:21.547287 2564 generic.go:356] "Generic (PLEG): container finished" podID="435e7a19-84db-498c-8087-cfe2e65ca747" containerID="fdb891652400427214ac9a8410fdec3c6561b44bf2f3a49df72c8f2c41c16fab" exitCode=0 Jul 23 19:52:21.547644 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:52:21.547377 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-536fc4c1-6hmwc" event={"ID":"435e7a19-84db-498c-8087-cfe2e65ca747","Type":"ContainerDied","Data":"fdb891652400427214ac9a8410fdec3c6561b44bf2f3a49df72c8f2c41c16fab"} Jul 23 19:52:21.547644 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:52:21.547441 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-536fc4c1-6hmwc" event={"ID":"435e7a19-84db-498c-8087-cfe2e65ca747","Type":"ContainerStarted","Data":"6a10d79f2961f72c91920fe75bf7e9b43dd3affb44b13b77f9e24ccb5c72078c"} Jul 23 19:52:22.682071 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:52:22.682042 2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-536fc4c1-6hmwc" Jul 23 19:52:22.727039 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:52:22.727013 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/435e7a19-84db-498c-8087-cfe2e65ca747-maas-api-service-ca\") pod \"435e7a19-84db-498c-8087-cfe2e65ca747\" (UID: \"435e7a19-84db-498c-8087-cfe2e65ca747\") " Jul 23 19:52:22.727344 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:52:22.727298 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/435e7a19-84db-498c-8087-cfe2e65ca747-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "435e7a19-84db-498c-8087-cfe2e65ca747" (UID: "435e7a19-84db-498c-8087-cfe2e65ca747"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 23 19:52:22.828420 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:52:22.828370 2564 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/435e7a19-84db-498c-8087-cfe2e65ca747-maas-api-service-ca\") on node \"ip-10-0-129-2.ec2.internal\" DevicePath \"\"" Jul 23 19:52:23.559848 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:52:23.559822 2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-536fc4c1-6hmwc" Jul 23 19:52:23.560011 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:52:23.559855 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-536fc4c1-6hmwc" event={"ID":"435e7a19-84db-498c-8087-cfe2e65ca747","Type":"ContainerDied","Data":"6a10d79f2961f72c91920fe75bf7e9b43dd3affb44b13b77f9e24ccb5c72078c"} Jul 23 19:52:23.560011 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:52:23.559910 2564 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="6a10d79f2961f72c91920fe75bf7e9b43dd3affb44b13b77f9e24ccb5c72078c" Jul 23 19:52:33.394814 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:52:33.394786 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-2dgmk"] Jul 23 19:52:35.343599 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:52:35.343565 2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-536fc4c1-6hmwc"] Jul 23 19:52:35.345783 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:52:35.345761 2564 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-536fc4c1-6hmwc"] Jul 23 19:52:36.657221 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:52:36.657190 2564 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="435e7a19-84db-498c-8087-cfe2e65ca747" path="/var/lib/kubelet/pods/435e7a19-84db-498c-8087-cfe2e65ca747/volumes" Jul 23 19:53:04.914434 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:53:04.914402 2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-9051a48a-78b8d56cc8-xspk7"] Jul 23 19:53:04.914997 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:53:04.914974 2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="435e7a19-84db-498c-8087-cfe2e65ca747" containerName="revoke-keys" Jul 23 19:53:04.914997 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:53:04.914992 2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="435e7a19-84db-498c-8087-cfe2e65ca747" containerName="revoke-keys" Jul 23 19:53:04.915113 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:53:04.915096 2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="435e7a19-84db-498c-8087-cfe2e65ca747" containerName="revoke-keys" Jul 23 19:53:04.918275 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:53:04.918255 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-9051a48a-78b8d56cc8-xspk7" Jul 23 19:53:04.920912 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:53:04.920894 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Jul 23 19:53:04.920912 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:53:04.920902 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Jul 23 19:53:04.921566 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:53:04.921553 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-7m48c\"" Jul 23 19:53:04.921621 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:53:04.921572 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-mt-9051a48a-serving-cert\"" Jul 23 19:53:04.928695 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:53:04.928666 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-9051a48a-78b8d56cc8-xspk7"] Jul 23 19:53:05.042032 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:53:05.042009 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2ded52a5-d008-4036-a9de-26aecf019832-maas-api-tls\") pod \"maas-api-e2e-mt-9051a48a-78b8d56cc8-xspk7\" (UID: \"2ded52a5-d008-4036-a9de-26aecf019832\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-9051a48a-78b8d56cc8-xspk7" Jul 23 19:53:05.042187 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:53:05.042040 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/2ded52a5-d008-4036-a9de-26aecf019832-openshift-service-ca\") pod \"maas-api-e2e-mt-9051a48a-78b8d56cc8-xspk7\" (UID: \"2ded52a5-d008-4036-a9de-26aecf019832\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-9051a48a-78b8d56cc8-xspk7" Jul 23 19:53:05.042187 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:53:05.042065 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6gsls\" (UniqueName: \"kubernetes.io/projected/2ded52a5-d008-4036-a9de-26aecf019832-kube-api-access-6gsls\") pod \"maas-api-e2e-mt-9051a48a-78b8d56cc8-xspk7\" (UID: \"2ded52a5-d008-4036-a9de-26aecf019832\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-9051a48a-78b8d56cc8-xspk7" Jul 23 19:53:05.143416 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:53:05.143388 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2ded52a5-d008-4036-a9de-26aecf019832-maas-api-tls\") pod \"maas-api-e2e-mt-9051a48a-78b8d56cc8-xspk7\" (UID: \"2ded52a5-d008-4036-a9de-26aecf019832\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-9051a48a-78b8d56cc8-xspk7" Jul 23 19:53:05.143550 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:53:05.143421 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/2ded52a5-d008-4036-a9de-26aecf019832-openshift-service-ca\") pod \"maas-api-e2e-mt-9051a48a-78b8d56cc8-xspk7\" (UID: \"2ded52a5-d008-4036-a9de-26aecf019832\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-9051a48a-78b8d56cc8-xspk7" Jul 23 19:53:05.143550 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:53:05.143444 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6gsls\" (UniqueName: \"kubernetes.io/projected/2ded52a5-d008-4036-a9de-26aecf019832-kube-api-access-6gsls\") pod \"maas-api-e2e-mt-9051a48a-78b8d56cc8-xspk7\" (UID: \"2ded52a5-d008-4036-a9de-26aecf019832\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-9051a48a-78b8d56cc8-xspk7" Jul 23 19:53:05.143550 ip-10-0-129-2 kubenswrapper[2564]: E0723 19:53:05.143538 2564 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-mt-9051a48a-serving-cert: secret "maas-api-e2e-mt-9051a48a-serving-cert" not found Jul 23 19:53:05.143656 ip-10-0-129-2 kubenswrapper[2564]: E0723 19:53:05.143613 2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2ded52a5-d008-4036-a9de-26aecf019832-maas-api-tls podName:2ded52a5-d008-4036-a9de-26aecf019832 nodeName:}" failed. No retries permitted until 2026-07-23 19:53:05.643593975 +0000 UTC m=+2011.495370217 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/2ded52a5-d008-4036-a9de-26aecf019832-maas-api-tls") pod "maas-api-e2e-mt-9051a48a-78b8d56cc8-xspk7" (UID: "2ded52a5-d008-4036-a9de-26aecf019832") : secret "maas-api-e2e-mt-9051a48a-serving-cert" not found Jul 23 19:53:05.144062 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:53:05.144044 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/2ded52a5-d008-4036-a9de-26aecf019832-openshift-service-ca\") pod \"maas-api-e2e-mt-9051a48a-78b8d56cc8-xspk7\" (UID: \"2ded52a5-d008-4036-a9de-26aecf019832\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-9051a48a-78b8d56cc8-xspk7" Jul 23 19:53:05.159512 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:53:05.159486 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6gsls\" (UniqueName: \"kubernetes.io/projected/2ded52a5-d008-4036-a9de-26aecf019832-kube-api-access-6gsls\") pod \"maas-api-e2e-mt-9051a48a-78b8d56cc8-xspk7\" (UID: \"2ded52a5-d008-4036-a9de-26aecf019832\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-9051a48a-78b8d56cc8-xspk7" Jul 23 19:53:05.647371 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:53:05.647343 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2ded52a5-d008-4036-a9de-26aecf019832-maas-api-tls\") pod \"maas-api-e2e-mt-9051a48a-78b8d56cc8-xspk7\" (UID: \"2ded52a5-d008-4036-a9de-26aecf019832\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-9051a48a-78b8d56cc8-xspk7" Jul 23 19:53:05.649772 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:53:05.649747 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2ded52a5-d008-4036-a9de-26aecf019832-maas-api-tls\") pod \"maas-api-e2e-mt-9051a48a-78b8d56cc8-xspk7\" (UID: \"2ded52a5-d008-4036-a9de-26aecf019832\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-9051a48a-78b8d56cc8-xspk7" Jul 23 19:53:05.828153 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:53:05.828126 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-9051a48a-78b8d56cc8-xspk7" Jul 23 19:53:05.978266 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:53:05.978239 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-9051a48a-78b8d56cc8-xspk7"] Jul 23 19:53:05.980027 ip-10-0-129-2 kubenswrapper[2564]: W0723 19:53:05.980002 2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2ded52a5_d008_4036_a9de_26aecf019832.slice/crio-ba4d77a003ea967952e8aa3c3ba52d8d226729f17cd83b0099cbd443ac95bc9e WatchSource:0}: Error finding container ba4d77a003ea967952e8aa3c3ba52d8d226729f17cd83b0099cbd443ac95bc9e: Status 404 returned error can't find the container with id ba4d77a003ea967952e8aa3c3ba52d8d226729f17cd83b0099cbd443ac95bc9e Jul 23 19:53:05.981708 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:53:05.981690 2564 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Jul 23 19:53:06.794773 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:53:06.794695 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-9051a48a-78b8d56cc8-xspk7" event={"ID":"2ded52a5-d008-4036-a9de-26aecf019832","Type":"ContainerStarted","Data":"c6a36580166a7e0c2383eb0bd42cbe03a4bc688052e6ceec6862918f070ce5d4"} Jul 23 19:53:06.794773 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:53:06.794757 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-9051a48a-78b8d56cc8-xspk7" event={"ID":"2ded52a5-d008-4036-a9de-26aecf019832","Type":"ContainerStarted","Data":"ba4d77a003ea967952e8aa3c3ba52d8d226729f17cd83b0099cbd443ac95bc9e"} Jul 23 19:53:06.794977 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:53:06.794778 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-mt-9051a48a-78b8d56cc8-xspk7" Jul 23 19:53:06.817420 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:53:06.817368 2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-mt-9051a48a-78b8d56cc8-xspk7" podStartSLOduration=2.337208957 podStartE2EDuration="2.8173491s" totalImagesPullingTime="480.140143ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-23 19:53:04 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-23 19:53:06.815244816 +0000 UTC m=+2012.667021118" watchObservedRunningTime="2026-07-23 19:53:06.8173491 +0000 UTC m=+2012.669125349" Jul 23 19:53:11.391945 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:53:11.391910 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-2dgmk"] Jul 23 19:53:12.805828 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:53:12.805798 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-mt-9051a48a-78b8d56cc8-xspk7" Jul 23 19:53:26.592790 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:53:26.592756 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-2dgmk"] Jul 23 19:53:33.290256 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:53:33.290222 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-2dgmk"] Jul 23 19:53:37.455702 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:53:37.455669 2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-9051a48a-9jpq4"] Jul 23 19:53:37.459295 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:53:37.459275 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-9051a48a-9jpq4" Jul 23 19:53:37.461339 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:53:37.461300 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-gcspk\"" Jul 23 19:53:37.466222 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:53:37.466203 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-9051a48a-9jpq4"] Jul 23 19:53:37.508696 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:53:37.508676 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c2a52a14-2942-416a-9ec0-d3ebcbd342b9-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-mt-9051a48a-9jpq4\" (UID: \"c2a52a14-2942-416a-9ec0-d3ebcbd342b9\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-9051a48a-9jpq4" Jul 23 19:53:37.609563 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:53:37.609542 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c2a52a14-2942-416a-9ec0-d3ebcbd342b9-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-mt-9051a48a-9jpq4\" (UID: \"c2a52a14-2942-416a-9ec0-d3ebcbd342b9\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-9051a48a-9jpq4" Jul 23 19:53:37.610050 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:53:37.610035 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c2a52a14-2942-416a-9ec0-d3ebcbd342b9-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-mt-9051a48a-9jpq4\" (UID: \"c2a52a14-2942-416a-9ec0-d3ebcbd342b9\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-9051a48a-9jpq4" Jul 23 19:53:37.770318 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:53:37.770271 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-9051a48a-9jpq4" Jul 23 19:53:37.901912 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:53:37.901883 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-9051a48a-9jpq4"] Jul 23 19:53:37.904780 ip-10-0-129-2 kubenswrapper[2564]: W0723 19:53:37.904755 2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc2a52a14_2942_416a_9ec0_d3ebcbd342b9.slice/crio-3bc0623e40bc6987f6ed244a4350e4f3ac9bc1c4956b38e0d6b77937a45e95ac WatchSource:0}: Error finding container 3bc0623e40bc6987f6ed244a4350e4f3ac9bc1c4956b38e0d6b77937a45e95ac: Status 404 returned error can't find the container with id 3bc0623e40bc6987f6ed244a4350e4f3ac9bc1c4956b38e0d6b77937a45e95ac Jul 23 19:53:37.961868 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:53:37.961840 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-9051a48a-9jpq4" event={"ID":"c2a52a14-2942-416a-9ec0-d3ebcbd342b9","Type":"ContainerStarted","Data":"3bc0623e40bc6987f6ed244a4350e4f3ac9bc1c4956b38e0d6b77937a45e95ac"} Jul 23 19:53:38.968405 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:53:38.968381 2564 generic.go:356] "Generic (PLEG): container finished" podID="c2a52a14-2942-416a-9ec0-d3ebcbd342b9" containerID="84b1562677062dbd234fd52418433cf3ea380756805a7dc680e4094e1b332e93" exitCode=0 Jul 23 19:53:38.968704 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:53:38.968453 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-9051a48a-9jpq4" event={"ID":"c2a52a14-2942-416a-9ec0-d3ebcbd342b9","Type":"ContainerDied","Data":"84b1562677062dbd234fd52418433cf3ea380756805a7dc680e4094e1b332e93"} Jul 23 19:53:40.104616 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:53:40.104596 2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-9051a48a-9jpq4" Jul 23 19:53:40.127860 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:53:40.127836 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c2a52a14-2942-416a-9ec0-d3ebcbd342b9-maas-api-service-ca\") pod \"c2a52a14-2942-416a-9ec0-d3ebcbd342b9\" (UID: \"c2a52a14-2942-416a-9ec0-d3ebcbd342b9\") " Jul 23 19:53:40.128289 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:53:40.128260 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c2a52a14-2942-416a-9ec0-d3ebcbd342b9-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "c2a52a14-2942-416a-9ec0-d3ebcbd342b9" (UID: "c2a52a14-2942-416a-9ec0-d3ebcbd342b9"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 23 19:53:40.228940 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:53:40.228890 2564 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c2a52a14-2942-416a-9ec0-d3ebcbd342b9-maas-api-service-ca\") on node \"ip-10-0-129-2.ec2.internal\" DevicePath \"\"" Jul 23 19:53:40.979434 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:53:40.979386 2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-9051a48a-9jpq4" Jul 23 19:53:40.979434 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:53:40.979408 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-9051a48a-9jpq4" event={"ID":"c2a52a14-2942-416a-9ec0-d3ebcbd342b9","Type":"ContainerDied","Data":"3bc0623e40bc6987f6ed244a4350e4f3ac9bc1c4956b38e0d6b77937a45e95ac"} Jul 23 19:53:40.979560 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:53:40.979463 2564 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="3bc0623e40bc6987f6ed244a4350e4f3ac9bc1c4956b38e0d6b77937a45e95ac" Jul 23 19:53:51.695198 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:53:51.695168 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-2dgmk"] Jul 23 19:53:52.999374 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:53:52.999345 2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-9051a48a-78b8d56cc8-xspk7"] Jul 23 19:53:52.999810 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:53:52.999590 2564 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-mt-9051a48a-78b8d56cc8-xspk7" podUID="2ded52a5-d008-4036-a9de-26aecf019832" containerName="maas-api" containerID="cri-o://c6a36580166a7e0c2383eb0bd42cbe03a4bc688052e6ceec6862918f070ce5d4" gracePeriod=30 Jul 23 19:53:53.246681 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:53:53.246659 2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-9051a48a-78b8d56cc8-xspk7" Jul 23 19:53:53.324494 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:53:53.324437 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/2ded52a5-d008-4036-a9de-26aecf019832-openshift-service-ca\") pod \"2ded52a5-d008-4036-a9de-26aecf019832\" (UID: \"2ded52a5-d008-4036-a9de-26aecf019832\") " Jul 23 19:53:53.324494 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:53:53.324480 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2ded52a5-d008-4036-a9de-26aecf019832-maas-api-tls\") pod \"2ded52a5-d008-4036-a9de-26aecf019832\" (UID: \"2ded52a5-d008-4036-a9de-26aecf019832\") " Jul 23 19:53:53.324674 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:53:53.324540 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-6gsls\" (UniqueName: \"kubernetes.io/projected/2ded52a5-d008-4036-a9de-26aecf019832-kube-api-access-6gsls\") pod \"2ded52a5-d008-4036-a9de-26aecf019832\" (UID: \"2ded52a5-d008-4036-a9de-26aecf019832\") " Jul 23 19:53:53.324797 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:53:53.324780 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2ded52a5-d008-4036-a9de-26aecf019832-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "2ded52a5-d008-4036-a9de-26aecf019832" (UID: "2ded52a5-d008-4036-a9de-26aecf019832"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 23 19:53:53.326580 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:53:53.326552 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2ded52a5-d008-4036-a9de-26aecf019832-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "2ded52a5-d008-4036-a9de-26aecf019832" (UID: "2ded52a5-d008-4036-a9de-26aecf019832"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 23 19:53:53.326681 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:53:53.326646 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2ded52a5-d008-4036-a9de-26aecf019832-kube-api-access-6gsls" (OuterVolumeSpecName: "kube-api-access-6gsls") pod "2ded52a5-d008-4036-a9de-26aecf019832" (UID: "2ded52a5-d008-4036-a9de-26aecf019832"). InnerVolumeSpecName "kube-api-access-6gsls". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 23 19:53:53.425302 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:53:53.425279 2564 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/2ded52a5-d008-4036-a9de-26aecf019832-openshift-service-ca\") on node \"ip-10-0-129-2.ec2.internal\" DevicePath \"\"" Jul 23 19:53:53.425302 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:53:53.425301 2564 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2ded52a5-d008-4036-a9de-26aecf019832-maas-api-tls\") on node \"ip-10-0-129-2.ec2.internal\" DevicePath \"\"" Jul 23 19:53:53.425457 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:53:53.425310 2564 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6gsls\" (UniqueName: \"kubernetes.io/projected/2ded52a5-d008-4036-a9de-26aecf019832-kube-api-access-6gsls\") on node \"ip-10-0-129-2.ec2.internal\" DevicePath \"\"" Jul 23 19:53:54.051239 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:53:54.051212 2564 generic.go:356] "Generic (PLEG): container finished" podID="2ded52a5-d008-4036-a9de-26aecf019832" containerID="c6a36580166a7e0c2383eb0bd42cbe03a4bc688052e6ceec6862918f070ce5d4" exitCode=0 Jul 23 19:53:54.051607 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:53:54.051254 2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-9051a48a-78b8d56cc8-xspk7" Jul 23 19:53:54.051607 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:53:54.051270 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-9051a48a-78b8d56cc8-xspk7" event={"ID":"2ded52a5-d008-4036-a9de-26aecf019832","Type":"ContainerDied","Data":"c6a36580166a7e0c2383eb0bd42cbe03a4bc688052e6ceec6862918f070ce5d4"} Jul 23 19:53:54.051607 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:53:54.051314 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-9051a48a-78b8d56cc8-xspk7" event={"ID":"2ded52a5-d008-4036-a9de-26aecf019832","Type":"ContainerDied","Data":"ba4d77a003ea967952e8aa3c3ba52d8d226729f17cd83b0099cbd443ac95bc9e"} Jul 23 19:53:54.051607 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:53:54.051364 2564 scope.go:117] "RemoveContainer" containerID="c6a36580166a7e0c2383eb0bd42cbe03a4bc688052e6ceec6862918f070ce5d4" Jul 23 19:53:54.061432 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:53:54.061409 2564 scope.go:117] "RemoveContainer" containerID="c6a36580166a7e0c2383eb0bd42cbe03a4bc688052e6ceec6862918f070ce5d4" Jul 23 19:53:54.061712 ip-10-0-129-2 kubenswrapper[2564]: E0723 19:53:54.061693 2564 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c6a36580166a7e0c2383eb0bd42cbe03a4bc688052e6ceec6862918f070ce5d4\": container with ID starting with c6a36580166a7e0c2383eb0bd42cbe03a4bc688052e6ceec6862918f070ce5d4 not found: ID does not exist" containerID="c6a36580166a7e0c2383eb0bd42cbe03a4bc688052e6ceec6862918f070ce5d4" Jul 23 19:53:54.061780 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:53:54.061719 2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c6a36580166a7e0c2383eb0bd42cbe03a4bc688052e6ceec6862918f070ce5d4"} err="failed to get container status \"c6a36580166a7e0c2383eb0bd42cbe03a4bc688052e6ceec6862918f070ce5d4\": rpc error: code = NotFound desc = could not find container \"c6a36580166a7e0c2383eb0bd42cbe03a4bc688052e6ceec6862918f070ce5d4\": container with ID starting with c6a36580166a7e0c2383eb0bd42cbe03a4bc688052e6ceec6862918f070ce5d4 not found: ID does not exist" Jul 23 19:53:54.084060 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:53:54.084039 2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-9051a48a-78b8d56cc8-xspk7"] Jul 23 19:53:54.093389 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:53:54.090919 2564 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-9051a48a-78b8d56cc8-xspk7"] Jul 23 19:53:54.658086 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:53:54.658062 2564 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="2ded52a5-d008-4036-a9de-26aecf019832" path="/var/lib/kubelet/pods/2ded52a5-d008-4036-a9de-26aecf019832/volumes" Jul 23 19:53:57.551712 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:53:57.551625 2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-9051a48a-9jpq4"] Jul 23 19:53:57.553822 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:53:57.553789 2564 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-9051a48a-9jpq4"] Jul 23 19:53:58.657598 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:53:58.657564 2564 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="c2a52a14-2942-416a-9ec0-d3ebcbd342b9" path="/var/lib/kubelet/pods/c2a52a14-2942-416a-9ec0-d3ebcbd342b9/volumes" Jul 23 19:54:02.502153 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:54:02.502122 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-2dgmk"] Jul 23 19:54:35.565858 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:54:35.565832 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-565b7b569d-z5wf4_ded5b3af-c74c-4d5a-aa77-62fe72d1dd29/console-operator/1.log" Jul 23 19:54:35.571228 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:54:35.571204 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-565b7b569d-z5wf4_ded5b3af-c74c-4d5a-aa77-62fe72d1dd29/console-operator/1.log" Jul 23 19:54:35.572870 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:54:35.572839 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-8dtlh_195ecf89-13ae-4311-bc72-fcdf78a5c4d6/ovn-acl-logging/0.log" Jul 23 19:54:35.573911 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:54:35.573887 2564 scope.go:117] "RemoveContainer" containerID="291e3434fecb1eee5f7b300c2812aaf9e75af1bbde0e206b92bb587157f303f8" Jul 23 19:54:35.578769 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:54:35.578752 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-8dtlh_195ecf89-13ae-4311-bc72-fcdf78a5c4d6/ovn-acl-logging/0.log" Jul 23 19:54:57.896339 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:54:57.896295 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-2dgmk"] Jul 23 19:55:07.191535 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:55:07.191503 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-2dgmk"] Jul 23 19:55:35.602428 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:55:35.602394 2564 scope.go:117] "RemoveContainer" containerID="c770abe5b74d6ba95c15a15cf56f2e7ae0c1ed8be05eab2d552c9d038a4ebd29" Jul 23 19:55:35.610757 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:55:35.610739 2564 scope.go:117] "RemoveContainer" containerID="e646662c71a5a3b1e91340d71ba1171a313fb8310a459291bce2670343727c35" Jul 23 19:56:00.290022 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:56:00.289985 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-2dgmk"] Jul 23 19:56:00.688462 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:56:00.688429 2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-111c5d-f69fc6cc7-sjmgv"] Jul 23 19:56:00.689013 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:56:00.688994 2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2ded52a5-d008-4036-a9de-26aecf019832" containerName="maas-api" Jul 23 19:56:00.689100 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:56:00.689021 2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="2ded52a5-d008-4036-a9de-26aecf019832" containerName="maas-api" Jul 23 19:56:00.689100 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:56:00.689066 2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c2a52a14-2942-416a-9ec0-d3ebcbd342b9" containerName="revoke-keys" Jul 23 19:56:00.689100 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:56:00.689075 2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="c2a52a14-2942-416a-9ec0-d3ebcbd342b9" containerName="revoke-keys" Jul 23 19:56:00.689259 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:56:00.689161 2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="c2a52a14-2942-416a-9ec0-d3ebcbd342b9" containerName="revoke-keys" Jul 23 19:56:00.689259 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:56:00.689181 2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="2ded52a5-d008-4036-a9de-26aecf019832" containerName="maas-api" Jul 23 19:56:00.692917 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:56:00.692891 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-111c5d-f69fc6cc7-sjmgv" Jul 23 19:56:00.695304 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:56:00.695281 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-inf-b-111c5d-serving-cert\"" Jul 23 19:56:00.695478 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:56:00.695355 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Jul 23 19:56:00.695876 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:56:00.695857 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-7m48c\"" Jul 23 19:56:00.695988 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:56:00.695877 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Jul 23 19:56:00.702018 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:56:00.701992 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-111c5d-f69fc6cc7-sjmgv"] Jul 23 19:56:00.726178 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:56:00.726148 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e2de19ca-1aeb-4a39-b40f-cb08d1e9c6b4-maas-api-tls\") pod \"maas-api-e2e-inf-b-111c5d-f69fc6cc7-sjmgv\" (UID: \"e2de19ca-1aeb-4a39-b40f-cb08d1e9c6b4\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-111c5d-f69fc6cc7-sjmgv" Jul 23 19:56:00.726302 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:56:00.726229 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xcphz\" (UniqueName: \"kubernetes.io/projected/e2de19ca-1aeb-4a39-b40f-cb08d1e9c6b4-kube-api-access-xcphz\") pod \"maas-api-e2e-inf-b-111c5d-f69fc6cc7-sjmgv\" (UID: \"e2de19ca-1aeb-4a39-b40f-cb08d1e9c6b4\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-111c5d-f69fc6cc7-sjmgv" Jul 23 19:56:00.726388 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:56:00.726303 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e2de19ca-1aeb-4a39-b40f-cb08d1e9c6b4-openshift-service-ca\") pod \"maas-api-e2e-inf-b-111c5d-f69fc6cc7-sjmgv\" (UID: \"e2de19ca-1aeb-4a39-b40f-cb08d1e9c6b4\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-111c5d-f69fc6cc7-sjmgv" Jul 23 19:56:00.827047 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:56:00.826802 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e2de19ca-1aeb-4a39-b40f-cb08d1e9c6b4-openshift-service-ca\") pod \"maas-api-e2e-inf-b-111c5d-f69fc6cc7-sjmgv\" (UID: \"e2de19ca-1aeb-4a39-b40f-cb08d1e9c6b4\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-111c5d-f69fc6cc7-sjmgv" Jul 23 19:56:00.827169 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:56:00.827054 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e2de19ca-1aeb-4a39-b40f-cb08d1e9c6b4-maas-api-tls\") pod \"maas-api-e2e-inf-b-111c5d-f69fc6cc7-sjmgv\" (UID: \"e2de19ca-1aeb-4a39-b40f-cb08d1e9c6b4\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-111c5d-f69fc6cc7-sjmgv" Jul 23 19:56:00.827169 ip-10-0-129-2 kubenswrapper[2564]: E0723 19:56:00.827152 2564 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-inf-b-111c5d-serving-cert: secret "maas-api-e2e-inf-b-111c5d-serving-cert" not found Jul 23 19:56:00.827264 ip-10-0-129-2 kubenswrapper[2564]: E0723 19:56:00.827228 2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e2de19ca-1aeb-4a39-b40f-cb08d1e9c6b4-maas-api-tls podName:e2de19ca-1aeb-4a39-b40f-cb08d1e9c6b4 nodeName:}" failed. No retries permitted until 2026-07-23 19:56:01.327211866 +0000 UTC m=+2187.178988091 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/e2de19ca-1aeb-4a39-b40f-cb08d1e9c6b4-maas-api-tls") pod "maas-api-e2e-inf-b-111c5d-f69fc6cc7-sjmgv" (UID: "e2de19ca-1aeb-4a39-b40f-cb08d1e9c6b4") : secret "maas-api-e2e-inf-b-111c5d-serving-cert" not found Jul 23 19:56:00.827314 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:56:00.827259 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xcphz\" (UniqueName: \"kubernetes.io/projected/e2de19ca-1aeb-4a39-b40f-cb08d1e9c6b4-kube-api-access-xcphz\") pod \"maas-api-e2e-inf-b-111c5d-f69fc6cc7-sjmgv\" (UID: \"e2de19ca-1aeb-4a39-b40f-cb08d1e9c6b4\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-111c5d-f69fc6cc7-sjmgv" Jul 23 19:56:00.827484 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:56:00.827464 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e2de19ca-1aeb-4a39-b40f-cb08d1e9c6b4-openshift-service-ca\") pod \"maas-api-e2e-inf-b-111c5d-f69fc6cc7-sjmgv\" (UID: \"e2de19ca-1aeb-4a39-b40f-cb08d1e9c6b4\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-111c5d-f69fc6cc7-sjmgv" Jul 23 19:56:00.835822 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:56:00.835794 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xcphz\" (UniqueName: \"kubernetes.io/projected/e2de19ca-1aeb-4a39-b40f-cb08d1e9c6b4-kube-api-access-xcphz\") pod \"maas-api-e2e-inf-b-111c5d-f69fc6cc7-sjmgv\" (UID: \"e2de19ca-1aeb-4a39-b40f-cb08d1e9c6b4\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-111c5d-f69fc6cc7-sjmgv" Jul 23 19:56:01.330926 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:56:01.330889 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e2de19ca-1aeb-4a39-b40f-cb08d1e9c6b4-maas-api-tls\") pod \"maas-api-e2e-inf-b-111c5d-f69fc6cc7-sjmgv\" (UID: \"e2de19ca-1aeb-4a39-b40f-cb08d1e9c6b4\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-111c5d-f69fc6cc7-sjmgv" Jul 23 19:56:01.333423 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:56:01.333396 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e2de19ca-1aeb-4a39-b40f-cb08d1e9c6b4-maas-api-tls\") pod \"maas-api-e2e-inf-b-111c5d-f69fc6cc7-sjmgv\" (UID: \"e2de19ca-1aeb-4a39-b40f-cb08d1e9c6b4\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-111c5d-f69fc6cc7-sjmgv" Jul 23 19:56:01.605513 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:56:01.605431 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-111c5d-f69fc6cc7-sjmgv" Jul 23 19:56:01.746294 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:56:01.746233 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-111c5d-f69fc6cc7-sjmgv"] Jul 23 19:56:01.747601 ip-10-0-129-2 kubenswrapper[2564]: W0723 19:56:01.747575 2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode2de19ca_1aeb_4a39_b40f_cb08d1e9c6b4.slice/crio-88ef5dd78dfc427eb49a5927aace20a5981ecec966e1e6e537bffd99b71a3030 WatchSource:0}: Error finding container 88ef5dd78dfc427eb49a5927aace20a5981ecec966e1e6e537bffd99b71a3030: Status 404 returned error can't find the container with id 88ef5dd78dfc427eb49a5927aace20a5981ecec966e1e6e537bffd99b71a3030 Jul 23 19:56:02.758586 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:56:02.758554 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-111c5d-f69fc6cc7-sjmgv" event={"ID":"e2de19ca-1aeb-4a39-b40f-cb08d1e9c6b4","Type":"ContainerStarted","Data":"4131fa551345045b4fea51ea5296d72b4be494f071dd01fcd259543b515d3312"} Jul 23 19:56:02.758982 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:56:02.758605 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-111c5d-f69fc6cc7-sjmgv" event={"ID":"e2de19ca-1aeb-4a39-b40f-cb08d1e9c6b4","Type":"ContainerStarted","Data":"88ef5dd78dfc427eb49a5927aace20a5981ecec966e1e6e537bffd99b71a3030"} Jul 23 19:56:02.758982 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:56:02.758661 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-111c5d-f69fc6cc7-sjmgv" Jul 23 19:56:02.778272 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:56:02.778234 2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-111c5d-f69fc6cc7-sjmgv" podStartSLOduration=2.4456592759999998 podStartE2EDuration="2.778221941s" totalImagesPullingTime="332.562665ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-23 19:56:00 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-23 19:56:02.775573248 +0000 UTC m=+2188.627349494" watchObservedRunningTime="2026-07-23 19:56:02.778221941 +0000 UTC m=+2188.629998182" Jul 23 19:56:08.770014 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:56:08.769986 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-111c5d-f69fc6cc7-sjmgv" Jul 23 19:56:14.595338 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:56:14.595287 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-2dgmk"] Jul 23 19:56:24.096873 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:56:24.096837 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-2dgmk"] Jul 23 19:56:35.631959 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:56:35.631931 2564 scope.go:117] "RemoveContainer" containerID="bb2c26148286d0b2354cf76b788f061f4776ee19bb1f851201b56684b2f0718c" Jul 23 19:56:38.289393 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:56:38.289361 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-2dgmk"] Jul 23 19:56:55.890970 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:56:55.890885 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-2dgmk"] Jul 23 19:57:25.412013 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:57:25.411980 2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-4223ab-z6wdb"] Jul 23 19:57:25.415439 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:57:25.415414 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-4223ab-z6wdb" Jul 23 19:57:25.417647 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:57:25.417628 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-gcspk\"" Jul 23 19:57:25.423787 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:57:25.423380 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-4223ab-z6wdb"] Jul 23 19:57:25.490065 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:57:25.490039 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/0dbc8c77-0e58-4a8f-aac0-606b1005f4ba-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-4223ab-z6wdb\" (UID: \"0dbc8c77-0e58-4a8f-aac0-606b1005f4ba\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-4223ab-z6wdb" Jul 23 19:57:25.591132 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:57:25.591112 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/0dbc8c77-0e58-4a8f-aac0-606b1005f4ba-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-4223ab-z6wdb\" (UID: \"0dbc8c77-0e58-4a8f-aac0-606b1005f4ba\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-4223ab-z6wdb" Jul 23 19:57:25.591721 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:57:25.591702 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/0dbc8c77-0e58-4a8f-aac0-606b1005f4ba-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-4223ab-z6wdb\" (UID: \"0dbc8c77-0e58-4a8f-aac0-606b1005f4ba\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-4223ab-z6wdb" Jul 23 19:57:25.727963 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:57:25.727905 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-4223ab-z6wdb" Jul 23 19:57:25.856816 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:57:25.856785 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-4223ab-z6wdb"] Jul 23 19:57:25.859605 ip-10-0-129-2 kubenswrapper[2564]: W0723 19:57:25.859580 2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0dbc8c77_0e58_4a8f_aac0_606b1005f4ba.slice/crio-69670a16b4f227fca8ec3a250e7e5380fef1bad8e64a53b4409ada0410025c71 WatchSource:0}: Error finding container 69670a16b4f227fca8ec3a250e7e5380fef1bad8e64a53b4409ada0410025c71: Status 404 returned error can't find the container with id 69670a16b4f227fca8ec3a250e7e5380fef1bad8e64a53b4409ada0410025c71 Jul 23 19:57:26.214507 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:57:26.214472 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-4223ab-z6wdb" event={"ID":"0dbc8c77-0e58-4a8f-aac0-606b1005f4ba","Type":"ContainerStarted","Data":"2f30e39042b92bb964c88cf36e52b8165f924be96eb8b800fb0bf7afc9e8bbaa"} Jul 23 19:57:26.214619 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:57:26.214521 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-4223ab-z6wdb" event={"ID":"0dbc8c77-0e58-4a8f-aac0-606b1005f4ba","Type":"ContainerStarted","Data":"69670a16b4f227fca8ec3a250e7e5380fef1bad8e64a53b4409ada0410025c71"} Jul 23 19:57:26.230144 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:57:26.230098 2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-4223ab-z6wdb" podStartSLOduration=1.230083886 podStartE2EDuration="1.230083886s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-23 19:57:25 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-23 19:57:26.229004833 +0000 UTC m=+2272.080781081" watchObservedRunningTime="2026-07-23 19:57:26.230083886 +0000 UTC m=+2272.081860132" Jul 23 19:57:27.220933 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:57:27.220905 2564 generic.go:356] "Generic (PLEG): container finished" podID="0dbc8c77-0e58-4a8f-aac0-606b1005f4ba" containerID="2f30e39042b92bb964c88cf36e52b8165f924be96eb8b800fb0bf7afc9e8bbaa" exitCode=0 Jul 23 19:57:27.221285 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:57:27.220985 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-4223ab-z6wdb" event={"ID":"0dbc8c77-0e58-4a8f-aac0-606b1005f4ba","Type":"ContainerDied","Data":"2f30e39042b92bb964c88cf36e52b8165f924be96eb8b800fb0bf7afc9e8bbaa"} Jul 23 19:57:27.645154 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:57:27.645084 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-2dgmk"] Jul 23 19:57:28.353235 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:57:28.353213 2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-4223ab-z6wdb" Jul 23 19:57:28.415791 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:57:28.415765 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/0dbc8c77-0e58-4a8f-aac0-606b1005f4ba-maas-api-service-ca\") pod \"0dbc8c77-0e58-4a8f-aac0-606b1005f4ba\" (UID: \"0dbc8c77-0e58-4a8f-aac0-606b1005f4ba\") " Jul 23 19:57:28.416091 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:57:28.416069 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0dbc8c77-0e58-4a8f-aac0-606b1005f4ba-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "0dbc8c77-0e58-4a8f-aac0-606b1005f4ba" (UID: "0dbc8c77-0e58-4a8f-aac0-606b1005f4ba"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 23 19:57:28.516492 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:57:28.516437 2564 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/0dbc8c77-0e58-4a8f-aac0-606b1005f4ba-maas-api-service-ca\") on node \"ip-10-0-129-2.ec2.internal\" DevicePath \"\"" Jul 23 19:57:29.232875 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:57:29.232850 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-4223ab-z6wdb" event={"ID":"0dbc8c77-0e58-4a8f-aac0-606b1005f4ba","Type":"ContainerDied","Data":"69670a16b4f227fca8ec3a250e7e5380fef1bad8e64a53b4409ada0410025c71"} Jul 23 19:57:29.233010 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:57:29.232894 2564 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="69670a16b4f227fca8ec3a250e7e5380fef1bad8e64a53b4409ada0410025c71" Jul 23 19:57:29.233010 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:57:29.232859 2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-4223ab-z6wdb" Jul 23 19:57:39.888999 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:57:39.888965 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-2dgmk"] Jul 23 19:57:40.501191 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:57:40.501151 2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-4223ab-z6wdb"] Jul 23 19:57:40.503469 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:57:40.503448 2564 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-4223ab-z6wdb"] Jul 23 19:57:40.658449 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:57:40.658422 2564 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="0dbc8c77-0e58-4a8f-aac0-606b1005f4ba" path="/var/lib/kubelet/pods/0dbc8c77-0e58-4a8f-aac0-606b1005f4ba/volumes" Jul 23 19:57:47.230900 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:57:47.230872 2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-111c5d-978xn"] Jul 23 19:57:47.231279 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:57:47.231263 2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0dbc8c77-0e58-4a8f-aac0-606b1005f4ba" containerName="revoke-keys" Jul 23 19:57:47.231340 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:57:47.231281 2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="0dbc8c77-0e58-4a8f-aac0-606b1005f4ba" containerName="revoke-keys" Jul 23 19:57:47.231388 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:57:47.231378 2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="0dbc8c77-0e58-4a8f-aac0-606b1005f4ba" containerName="revoke-keys" Jul 23 19:57:47.234627 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:57:47.234613 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-111c5d-978xn" Jul 23 19:57:47.237025 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:57:47.237003 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-gcspk\"" Jul 23 19:57:47.242242 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:57:47.242219 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-111c5d-978xn"] Jul 23 19:57:47.355667 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:57:47.355645 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/9201d5a5-222e-432f-b51d-629adfda16bf-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-111c5d-978xn\" (UID: \"9201d5a5-222e-432f-b51d-629adfda16bf\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-111c5d-978xn" Jul 23 19:57:47.456266 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:57:47.456239 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/9201d5a5-222e-432f-b51d-629adfda16bf-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-111c5d-978xn\" (UID: \"9201d5a5-222e-432f-b51d-629adfda16bf\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-111c5d-978xn" Jul 23 19:57:47.456814 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:57:47.456797 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/9201d5a5-222e-432f-b51d-629adfda16bf-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-111c5d-978xn\" (UID: \"9201d5a5-222e-432f-b51d-629adfda16bf\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-111c5d-978xn" Jul 23 19:57:47.546593 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:57:47.546544 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-111c5d-978xn" Jul 23 19:57:47.711725 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:57:47.711698 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-111c5d-978xn"] Jul 23 19:57:47.712552 ip-10-0-129-2 kubenswrapper[2564]: W0723 19:57:47.712523 2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9201d5a5_222e_432f_b51d_629adfda16bf.slice/crio-0e89a295838a03eaa1a666a754c39f15bc84e4c515786b56f4d4e7fd246aa27e WatchSource:0}: Error finding container 0e89a295838a03eaa1a666a754c39f15bc84e4c515786b56f4d4e7fd246aa27e: Status 404 returned error can't find the container with id 0e89a295838a03eaa1a666a754c39f15bc84e4c515786b56f4d4e7fd246aa27e Jul 23 19:57:48.337943 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:57:48.337912 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-111c5d-978xn" event={"ID":"9201d5a5-222e-432f-b51d-629adfda16bf","Type":"ContainerStarted","Data":"1735a2910690c68a61b9cd732b3790df7a95f3a813a304b16468094623b00e59"} Jul 23 19:57:48.338347 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:57:48.337960 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-111c5d-978xn" event={"ID":"9201d5a5-222e-432f-b51d-629adfda16bf","Type":"ContainerStarted","Data":"0e89a295838a03eaa1a666a754c39f15bc84e4c515786b56f4d4e7fd246aa27e"} Jul 23 19:57:48.354447 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:57:48.354399 2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-111c5d-978xn" podStartSLOduration=1.354385255 podStartE2EDuration="1.354385255s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-23 19:57:47 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-23 19:57:48.353415191 +0000 UTC m=+2294.205191429" watchObservedRunningTime="2026-07-23 19:57:48.354385255 +0000 UTC m=+2294.206161501" Jul 23 19:57:49.344419 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:57:49.344385 2564 generic.go:356] "Generic (PLEG): container finished" podID="9201d5a5-222e-432f-b51d-629adfda16bf" containerID="1735a2910690c68a61b9cd732b3790df7a95f3a813a304b16468094623b00e59" exitCode=0 Jul 23 19:57:49.344775 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:57:49.344475 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-111c5d-978xn" event={"ID":"9201d5a5-222e-432f-b51d-629adfda16bf","Type":"ContainerDied","Data":"1735a2910690c68a61b9cd732b3790df7a95f3a813a304b16468094623b00e59"} Jul 23 19:57:50.486499 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:57:50.486473 2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-111c5d-978xn" Jul 23 19:57:50.582450 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:57:50.582422 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/9201d5a5-222e-432f-b51d-629adfda16bf-maas-api-service-ca\") pod \"9201d5a5-222e-432f-b51d-629adfda16bf\" (UID: \"9201d5a5-222e-432f-b51d-629adfda16bf\") " Jul 23 19:57:50.582733 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:57:50.582713 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9201d5a5-222e-432f-b51d-629adfda16bf-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "9201d5a5-222e-432f-b51d-629adfda16bf" (UID: "9201d5a5-222e-432f-b51d-629adfda16bf"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 23 19:57:50.684106 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:57:50.684087 2564 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/9201d5a5-222e-432f-b51d-629adfda16bf-maas-api-service-ca\") on node \"ip-10-0-129-2.ec2.internal\" DevicePath \"\"" Jul 23 19:57:51.356049 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:57:51.356022 2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-111c5d-978xn" Jul 23 19:57:51.356049 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:57:51.356036 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-111c5d-978xn" event={"ID":"9201d5a5-222e-432f-b51d-629adfda16bf","Type":"ContainerDied","Data":"0e89a295838a03eaa1a666a754c39f15bc84e4c515786b56f4d4e7fd246aa27e"} Jul 23 19:57:51.356310 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:57:51.356080 2564 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="0e89a295838a03eaa1a666a754c39f15bc84e4c515786b56f4d4e7fd246aa27e" Jul 23 19:57:57.746583 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:57:57.746555 2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-111c5d-f69fc6cc7-sjmgv"] Jul 23 19:57:57.746974 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:57:57.746752 2564 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-111c5d-f69fc6cc7-sjmgv" podUID="e2de19ca-1aeb-4a39-b40f-cb08d1e9c6b4" containerName="maas-api" containerID="cri-o://4131fa551345045b4fea51ea5296d72b4be494f071dd01fcd259543b515d3312" gracePeriod=30 Jul 23 19:57:57.997863 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:57:57.997808 2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-111c5d-f69fc6cc7-sjmgv" Jul 23 19:57:58.147373 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:57:58.147346 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e2de19ca-1aeb-4a39-b40f-cb08d1e9c6b4-openshift-service-ca\") pod \"e2de19ca-1aeb-4a39-b40f-cb08d1e9c6b4\" (UID: \"e2de19ca-1aeb-4a39-b40f-cb08d1e9c6b4\") " Jul 23 19:57:58.147501 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:57:58.147400 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e2de19ca-1aeb-4a39-b40f-cb08d1e9c6b4-maas-api-tls\") pod \"e2de19ca-1aeb-4a39-b40f-cb08d1e9c6b4\" (UID: \"e2de19ca-1aeb-4a39-b40f-cb08d1e9c6b4\") " Jul 23 19:57:58.147546 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:57:58.147511 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-xcphz\" (UniqueName: \"kubernetes.io/projected/e2de19ca-1aeb-4a39-b40f-cb08d1e9c6b4-kube-api-access-xcphz\") pod \"e2de19ca-1aeb-4a39-b40f-cb08d1e9c6b4\" (UID: \"e2de19ca-1aeb-4a39-b40f-cb08d1e9c6b4\") " Jul 23 19:57:58.147682 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:57:58.147661 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e2de19ca-1aeb-4a39-b40f-cb08d1e9c6b4-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "e2de19ca-1aeb-4a39-b40f-cb08d1e9c6b4" (UID: "e2de19ca-1aeb-4a39-b40f-cb08d1e9c6b4"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 23 19:57:58.147767 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:57:58.147755 2564 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e2de19ca-1aeb-4a39-b40f-cb08d1e9c6b4-openshift-service-ca\") on node \"ip-10-0-129-2.ec2.internal\" DevicePath \"\"" Jul 23 19:57:58.149525 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:57:58.149497 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e2de19ca-1aeb-4a39-b40f-cb08d1e9c6b4-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "e2de19ca-1aeb-4a39-b40f-cb08d1e9c6b4" (UID: "e2de19ca-1aeb-4a39-b40f-cb08d1e9c6b4"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 23 19:57:58.149628 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:57:58.149496 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e2de19ca-1aeb-4a39-b40f-cb08d1e9c6b4-kube-api-access-xcphz" (OuterVolumeSpecName: "kube-api-access-xcphz") pod "e2de19ca-1aeb-4a39-b40f-cb08d1e9c6b4" (UID: "e2de19ca-1aeb-4a39-b40f-cb08d1e9c6b4"). InnerVolumeSpecName "kube-api-access-xcphz". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 23 19:57:58.248422 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:57:58.248370 2564 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e2de19ca-1aeb-4a39-b40f-cb08d1e9c6b4-maas-api-tls\") on node \"ip-10-0-129-2.ec2.internal\" DevicePath \"\"" Jul 23 19:57:58.248422 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:57:58.248391 2564 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xcphz\" (UniqueName: \"kubernetes.io/projected/e2de19ca-1aeb-4a39-b40f-cb08d1e9c6b4-kube-api-access-xcphz\") on node \"ip-10-0-129-2.ec2.internal\" DevicePath \"\"" Jul 23 19:57:58.392794 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:57:58.392765 2564 generic.go:356] "Generic (PLEG): container finished" podID="e2de19ca-1aeb-4a39-b40f-cb08d1e9c6b4" containerID="4131fa551345045b4fea51ea5296d72b4be494f071dd01fcd259543b515d3312" exitCode=0 Jul 23 19:57:58.392909 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:57:58.392822 2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-111c5d-f69fc6cc7-sjmgv" Jul 23 19:57:58.392909 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:57:58.392862 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-111c5d-f69fc6cc7-sjmgv" event={"ID":"e2de19ca-1aeb-4a39-b40f-cb08d1e9c6b4","Type":"ContainerDied","Data":"4131fa551345045b4fea51ea5296d72b4be494f071dd01fcd259543b515d3312"} Jul 23 19:57:58.392993 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:57:58.392914 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-111c5d-f69fc6cc7-sjmgv" event={"ID":"e2de19ca-1aeb-4a39-b40f-cb08d1e9c6b4","Type":"ContainerDied","Data":"88ef5dd78dfc427eb49a5927aace20a5981ecec966e1e6e537bffd99b71a3030"} Jul 23 19:57:58.392993 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:57:58.392938 2564 scope.go:117] "RemoveContainer" containerID="4131fa551345045b4fea51ea5296d72b4be494f071dd01fcd259543b515d3312" Jul 23 19:57:58.402056 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:57:58.402042 2564 scope.go:117] "RemoveContainer" containerID="4131fa551345045b4fea51ea5296d72b4be494f071dd01fcd259543b515d3312" Jul 23 19:57:58.402308 ip-10-0-129-2 kubenswrapper[2564]: E0723 19:57:58.402290 2564 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4131fa551345045b4fea51ea5296d72b4be494f071dd01fcd259543b515d3312\": container with ID starting with 4131fa551345045b4fea51ea5296d72b4be494f071dd01fcd259543b515d3312 not found: ID does not exist" containerID="4131fa551345045b4fea51ea5296d72b4be494f071dd01fcd259543b515d3312" Jul 23 19:57:58.402367 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:57:58.402315 2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4131fa551345045b4fea51ea5296d72b4be494f071dd01fcd259543b515d3312"} err="failed to get container status \"4131fa551345045b4fea51ea5296d72b4be494f071dd01fcd259543b515d3312\": rpc error: code = NotFound desc = could not find container \"4131fa551345045b4fea51ea5296d72b4be494f071dd01fcd259543b515d3312\": container with ID starting with 4131fa551345045b4fea51ea5296d72b4be494f071dd01fcd259543b515d3312 not found: ID does not exist" Jul 23 19:57:58.413689 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:57:58.413663 2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-111c5d-f69fc6cc7-sjmgv"] Jul 23 19:57:58.415924 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:57:58.415908 2564 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-111c5d-f69fc6cc7-sjmgv"] Jul 23 19:57:58.658620 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:57:58.658569 2564 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="e2de19ca-1aeb-4a39-b40f-cb08d1e9c6b4" path="/var/lib/kubelet/pods/e2de19ca-1aeb-4a39-b40f-cb08d1e9c6b4/volumes" Jul 23 19:58:00.694592 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:58:00.694564 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-2dgmk"] Jul 23 19:58:02.306137 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:58:02.306105 2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-111c5d-978xn"] Jul 23 19:58:02.310983 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:58:02.310957 2564 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-111c5d-978xn"] Jul 23 19:58:02.657984 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:58:02.657909 2564 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="9201d5a5-222e-432f-b51d-629adfda16bf" path="/var/lib/kubelet/pods/9201d5a5-222e-432f-b51d-629adfda16bf/volumes" Jul 23 19:58:12.236741 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:58:12.236708 2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-a8f21c-6d87bfbc6f-pqz9w"] Jul 23 19:58:12.237174 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:58:12.237069 2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e2de19ca-1aeb-4a39-b40f-cb08d1e9c6b4" containerName="maas-api" Jul 23 19:58:12.237174 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:58:12.237079 2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="e2de19ca-1aeb-4a39-b40f-cb08d1e9c6b4" containerName="maas-api" Jul 23 19:58:12.237174 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:58:12.237097 2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9201d5a5-222e-432f-b51d-629adfda16bf" containerName="revoke-keys" Jul 23 19:58:12.237174 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:58:12.237103 2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="9201d5a5-222e-432f-b51d-629adfda16bf" containerName="revoke-keys" Jul 23 19:58:12.237174 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:58:12.237163 2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="9201d5a5-222e-432f-b51d-629adfda16bf" containerName="revoke-keys" Jul 23 19:58:12.237174 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:58:12.237174 2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="e2de19ca-1aeb-4a39-b40f-cb08d1e9c6b4" containerName="maas-api" Jul 23 19:58:12.240369 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:58:12.240353 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-a8f21c-6d87bfbc6f-pqz9w" Jul 23 19:58:12.242738 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:58:12.242712 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-api-a-a8f21c-serving-cert\"" Jul 23 19:58:12.243038 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:58:12.243022 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-7m48c\"" Jul 23 19:58:12.243239 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:58:12.243226 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Jul 23 19:58:12.243821 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:58:12.243806 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Jul 23 19:58:12.253052 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:58:12.253029 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-a8f21c-6d87bfbc6f-pqz9w"] Jul 23 19:58:12.352669 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:58:12.352643 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lsf8m\" (UniqueName: \"kubernetes.io/projected/da86ae6b-59a2-4591-b2a2-37ca04bc0ccd-kube-api-access-lsf8m\") pod \"maas-api-e2e-api-a-a8f21c-6d87bfbc6f-pqz9w\" (UID: \"da86ae6b-59a2-4591-b2a2-37ca04bc0ccd\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-a8f21c-6d87bfbc6f-pqz9w" Jul 23 19:58:12.352822 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:58:12.352675 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/da86ae6b-59a2-4591-b2a2-37ca04bc0ccd-maas-api-tls\") pod \"maas-api-e2e-api-a-a8f21c-6d87bfbc6f-pqz9w\" (UID: \"da86ae6b-59a2-4591-b2a2-37ca04bc0ccd\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-a8f21c-6d87bfbc6f-pqz9w" Jul 23 19:58:12.352822 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:58:12.352700 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/da86ae6b-59a2-4591-b2a2-37ca04bc0ccd-openshift-service-ca\") pod \"maas-api-e2e-api-a-a8f21c-6d87bfbc6f-pqz9w\" (UID: \"da86ae6b-59a2-4591-b2a2-37ca04bc0ccd\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-a8f21c-6d87bfbc6f-pqz9w" Jul 23 19:58:12.453965 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:58:12.453931 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lsf8m\" (UniqueName: \"kubernetes.io/projected/da86ae6b-59a2-4591-b2a2-37ca04bc0ccd-kube-api-access-lsf8m\") pod \"maas-api-e2e-api-a-a8f21c-6d87bfbc6f-pqz9w\" (UID: \"da86ae6b-59a2-4591-b2a2-37ca04bc0ccd\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-a8f21c-6d87bfbc6f-pqz9w" Jul 23 19:58:12.454119 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:58:12.453974 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/da86ae6b-59a2-4591-b2a2-37ca04bc0ccd-maas-api-tls\") pod \"maas-api-e2e-api-a-a8f21c-6d87bfbc6f-pqz9w\" (UID: \"da86ae6b-59a2-4591-b2a2-37ca04bc0ccd\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-a8f21c-6d87bfbc6f-pqz9w" Jul 23 19:58:12.454119 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:58:12.454005 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/da86ae6b-59a2-4591-b2a2-37ca04bc0ccd-openshift-service-ca\") pod \"maas-api-e2e-api-a-a8f21c-6d87bfbc6f-pqz9w\" (UID: \"da86ae6b-59a2-4591-b2a2-37ca04bc0ccd\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-a8f21c-6d87bfbc6f-pqz9w" Jul 23 19:58:12.454195 ip-10-0-129-2 kubenswrapper[2564]: E0723 19:58:12.454152 2564 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-api-a-a8f21c-serving-cert: secret "maas-api-e2e-api-a-a8f21c-serving-cert" not found Jul 23 19:58:12.454245 ip-10-0-129-2 kubenswrapper[2564]: E0723 19:58:12.454234 2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/da86ae6b-59a2-4591-b2a2-37ca04bc0ccd-maas-api-tls podName:da86ae6b-59a2-4591-b2a2-37ca04bc0ccd nodeName:}" failed. No retries permitted until 2026-07-23 19:58:12.954212568 +0000 UTC m=+2318.805988798 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/da86ae6b-59a2-4591-b2a2-37ca04bc0ccd-maas-api-tls") pod "maas-api-e2e-api-a-a8f21c-6d87bfbc6f-pqz9w" (UID: "da86ae6b-59a2-4591-b2a2-37ca04bc0ccd") : secret "maas-api-e2e-api-a-a8f21c-serving-cert" not found Jul 23 19:58:12.454693 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:58:12.454671 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/da86ae6b-59a2-4591-b2a2-37ca04bc0ccd-openshift-service-ca\") pod \"maas-api-e2e-api-a-a8f21c-6d87bfbc6f-pqz9w\" (UID: \"da86ae6b-59a2-4591-b2a2-37ca04bc0ccd\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-a8f21c-6d87bfbc6f-pqz9w" Jul 23 19:58:12.463149 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:58:12.463128 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-lsf8m\" (UniqueName: \"kubernetes.io/projected/da86ae6b-59a2-4591-b2a2-37ca04bc0ccd-kube-api-access-lsf8m\") pod \"maas-api-e2e-api-a-a8f21c-6d87bfbc6f-pqz9w\" (UID: \"da86ae6b-59a2-4591-b2a2-37ca04bc0ccd\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-a8f21c-6d87bfbc6f-pqz9w" Jul 23 19:58:12.958943 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:58:12.958913 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/da86ae6b-59a2-4591-b2a2-37ca04bc0ccd-maas-api-tls\") pod \"maas-api-e2e-api-a-a8f21c-6d87bfbc6f-pqz9w\" (UID: \"da86ae6b-59a2-4591-b2a2-37ca04bc0ccd\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-a8f21c-6d87bfbc6f-pqz9w" Jul 23 19:58:12.959128 ip-10-0-129-2 kubenswrapper[2564]: E0723 19:58:12.959065 2564 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-api-a-a8f21c-serving-cert: secret "maas-api-e2e-api-a-a8f21c-serving-cert" not found Jul 23 19:58:12.959128 ip-10-0-129-2 kubenswrapper[2564]: E0723 19:58:12.959120 2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/da86ae6b-59a2-4591-b2a2-37ca04bc0ccd-maas-api-tls podName:da86ae6b-59a2-4591-b2a2-37ca04bc0ccd nodeName:}" failed. No retries permitted until 2026-07-23 19:58:13.959105809 +0000 UTC m=+2319.810882034 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/da86ae6b-59a2-4591-b2a2-37ca04bc0ccd-maas-api-tls") pod "maas-api-e2e-api-a-a8f21c-6d87bfbc6f-pqz9w" (UID: "da86ae6b-59a2-4591-b2a2-37ca04bc0ccd") : secret "maas-api-e2e-api-a-a8f21c-serving-cert" not found Jul 23 19:58:13.969242 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:58:13.969213 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/da86ae6b-59a2-4591-b2a2-37ca04bc0ccd-maas-api-tls\") pod \"maas-api-e2e-api-a-a8f21c-6d87bfbc6f-pqz9w\" (UID: \"da86ae6b-59a2-4591-b2a2-37ca04bc0ccd\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-a8f21c-6d87bfbc6f-pqz9w" Jul 23 19:58:13.971662 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:58:13.971638 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/da86ae6b-59a2-4591-b2a2-37ca04bc0ccd-maas-api-tls\") pod \"maas-api-e2e-api-a-a8f21c-6d87bfbc6f-pqz9w\" (UID: \"da86ae6b-59a2-4591-b2a2-37ca04bc0ccd\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-a8f21c-6d87bfbc6f-pqz9w" Jul 23 19:58:14.053029 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:58:14.053005 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-a8f21c-6d87bfbc6f-pqz9w" Jul 23 19:58:14.181047 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:58:14.181018 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-a8f21c-6d87bfbc6f-pqz9w"] Jul 23 19:58:14.182263 ip-10-0-129-2 kubenswrapper[2564]: W0723 19:58:14.182239 2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podda86ae6b_59a2_4591_b2a2_37ca04bc0ccd.slice/crio-4cf1311bb0f5f4a9d28a11d6909a1cad53c45cad0800559d8cf4c8658d358973 WatchSource:0}: Error finding container 4cf1311bb0f5f4a9d28a11d6909a1cad53c45cad0800559d8cf4c8658d358973: Status 404 returned error can't find the container with id 4cf1311bb0f5f4a9d28a11d6909a1cad53c45cad0800559d8cf4c8658d358973 Jul 23 19:58:14.183614 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:58:14.183597 2564 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Jul 23 19:58:14.486356 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:58:14.486304 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-a8f21c-6d87bfbc6f-pqz9w" event={"ID":"da86ae6b-59a2-4591-b2a2-37ca04bc0ccd","Type":"ContainerStarted","Data":"4cf1311bb0f5f4a9d28a11d6909a1cad53c45cad0800559d8cf4c8658d358973"} Jul 23 19:58:15.493066 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:58:15.493031 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-a8f21c-6d87bfbc6f-pqz9w" event={"ID":"da86ae6b-59a2-4591-b2a2-37ca04bc0ccd","Type":"ContainerStarted","Data":"9f91edc99b31856730ea1da26d7ce58b3d75c91adcfa16e78636cd1ca24b4567"} Jul 23 19:58:15.493463 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:58:15.493102 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-a8f21c-6d87bfbc6f-pqz9w" Jul 23 19:58:15.509716 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:58:15.509672 2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-a8f21c-6d87bfbc6f-pqz9w" podStartSLOduration=3.121511273 podStartE2EDuration="3.509660808s" totalImagesPullingTime="388.149535ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-23 19:58:12 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-23 19:58:15.507406665 +0000 UTC m=+2321.359182943" watchObservedRunningTime="2026-07-23 19:58:15.509660808 +0000 UTC m=+2321.361437076" Jul 23 19:58:19.690194 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:58:19.690163 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-2dgmk"] Jul 23 19:58:20.385292 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:58:20.385261 2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-1d04d2-69bdd998f6-jgxpl"] Jul 23 19:58:20.390615 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:58:20.390592 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-1d04d2-69bdd998f6-jgxpl" Jul 23 19:58:20.392962 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:58:20.392939 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-api-b-1d04d2-serving-cert\"" Jul 23 19:58:20.399131 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:58:20.399109 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-1d04d2-69bdd998f6-jgxpl"] Jul 23 19:58:20.522251 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:58:20.522225 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/6b810ffd-523a-4f5e-9ac6-74336267d9f0-openshift-service-ca\") pod \"maas-api-e2e-api-b-1d04d2-69bdd998f6-jgxpl\" (UID: \"6b810ffd-523a-4f5e-9ac6-74336267d9f0\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-1d04d2-69bdd998f6-jgxpl" Jul 23 19:58:20.522436 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:58:20.522271 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hgbrx\" (UniqueName: \"kubernetes.io/projected/6b810ffd-523a-4f5e-9ac6-74336267d9f0-kube-api-access-hgbrx\") pod \"maas-api-e2e-api-b-1d04d2-69bdd998f6-jgxpl\" (UID: \"6b810ffd-523a-4f5e-9ac6-74336267d9f0\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-1d04d2-69bdd998f6-jgxpl" Jul 23 19:58:20.522436 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:58:20.522383 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6b810ffd-523a-4f5e-9ac6-74336267d9f0-maas-api-tls\") pod \"maas-api-e2e-api-b-1d04d2-69bdd998f6-jgxpl\" (UID: \"6b810ffd-523a-4f5e-9ac6-74336267d9f0\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-1d04d2-69bdd998f6-jgxpl" Jul 23 19:58:20.623644 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:58:20.623613 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hgbrx\" (UniqueName: \"kubernetes.io/projected/6b810ffd-523a-4f5e-9ac6-74336267d9f0-kube-api-access-hgbrx\") pod \"maas-api-e2e-api-b-1d04d2-69bdd998f6-jgxpl\" (UID: \"6b810ffd-523a-4f5e-9ac6-74336267d9f0\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-1d04d2-69bdd998f6-jgxpl" Jul 23 19:58:20.623761 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:58:20.623672 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6b810ffd-523a-4f5e-9ac6-74336267d9f0-maas-api-tls\") pod \"maas-api-e2e-api-b-1d04d2-69bdd998f6-jgxpl\" (UID: \"6b810ffd-523a-4f5e-9ac6-74336267d9f0\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-1d04d2-69bdd998f6-jgxpl" Jul 23 19:58:20.623761 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:58:20.623742 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/6b810ffd-523a-4f5e-9ac6-74336267d9f0-openshift-service-ca\") pod \"maas-api-e2e-api-b-1d04d2-69bdd998f6-jgxpl\" (UID: \"6b810ffd-523a-4f5e-9ac6-74336267d9f0\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-1d04d2-69bdd998f6-jgxpl" Jul 23 19:58:20.624388 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:58:20.624348 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/6b810ffd-523a-4f5e-9ac6-74336267d9f0-openshift-service-ca\") pod \"maas-api-e2e-api-b-1d04d2-69bdd998f6-jgxpl\" (UID: \"6b810ffd-523a-4f5e-9ac6-74336267d9f0\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-1d04d2-69bdd998f6-jgxpl" Jul 23 19:58:20.626097 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:58:20.626079 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6b810ffd-523a-4f5e-9ac6-74336267d9f0-maas-api-tls\") pod \"maas-api-e2e-api-b-1d04d2-69bdd998f6-jgxpl\" (UID: \"6b810ffd-523a-4f5e-9ac6-74336267d9f0\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-1d04d2-69bdd998f6-jgxpl" Jul 23 19:58:20.639741 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:58:20.639689 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-hgbrx\" (UniqueName: \"kubernetes.io/projected/6b810ffd-523a-4f5e-9ac6-74336267d9f0-kube-api-access-hgbrx\") pod \"maas-api-e2e-api-b-1d04d2-69bdd998f6-jgxpl\" (UID: \"6b810ffd-523a-4f5e-9ac6-74336267d9f0\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-1d04d2-69bdd998f6-jgxpl" Jul 23 19:58:20.703810 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:58:20.703789 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-1d04d2-69bdd998f6-jgxpl" Jul 23 19:58:20.836948 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:58:20.836909 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-1d04d2-69bdd998f6-jgxpl"] Jul 23 19:58:20.839716 ip-10-0-129-2 kubenswrapper[2564]: W0723 19:58:20.839687 2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6b810ffd_523a_4f5e_9ac6_74336267d9f0.slice/crio-e05e8284e63efe023bdaf07c6c38ebcd8f859b78534046fc5e549fa4fba99709 WatchSource:0}: Error finding container e05e8284e63efe023bdaf07c6c38ebcd8f859b78534046fc5e549fa4fba99709: Status 404 returned error can't find the container with id e05e8284e63efe023bdaf07c6c38ebcd8f859b78534046fc5e549fa4fba99709 Jul 23 19:58:21.504425 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:58:21.504402 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-a8f21c-6d87bfbc6f-pqz9w" Jul 23 19:58:21.527560 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:58:21.527527 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-1d04d2-69bdd998f6-jgxpl" event={"ID":"6b810ffd-523a-4f5e-9ac6-74336267d9f0","Type":"ContainerStarted","Data":"50cd716660261aeaf08d7d8d88c77c5204e6375379d95f6e96fd864fb5f8dae1"} Jul 23 19:58:21.527689 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:58:21.527578 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-1d04d2-69bdd998f6-jgxpl" event={"ID":"6b810ffd-523a-4f5e-9ac6-74336267d9f0","Type":"ContainerStarted","Data":"e05e8284e63efe023bdaf07c6c38ebcd8f859b78534046fc5e549fa4fba99709"} Jul 23 19:58:21.527689 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:58:21.527631 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-1d04d2-69bdd998f6-jgxpl" Jul 23 19:58:21.549422 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:58:21.549376 2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-1d04d2-69bdd998f6-jgxpl" podStartSLOduration=1.227849639 podStartE2EDuration="1.54936001s" totalImagesPullingTime="321.510371ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-23 19:58:20 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-23 19:58:21.549133193 +0000 UTC m=+2327.400909467" watchObservedRunningTime="2026-07-23 19:58:21.54936001 +0000 UTC m=+2327.401136253" Jul 23 19:58:27.538699 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:58:27.538620 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-1d04d2-69bdd998f6-jgxpl" Jul 23 19:58:32.678393 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:58:32.678363 2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-a8f21c-smpl8"] Jul 23 19:58:32.681988 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:58:32.681972 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-a8f21c-smpl8" Jul 23 19:58:32.684080 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:58:32.684064 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-gcspk\"" Jul 23 19:58:32.689879 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:58:32.689855 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-a8f21c-smpl8"] Jul 23 19:58:32.716707 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:58:32.716674 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/5265b78a-bee3-4147-aa9b-0be306cd00c9-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-api-a-a8f21c-smpl8\" (UID: \"5265b78a-bee3-4147-aa9b-0be306cd00c9\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-a8f21c-smpl8" Jul 23 19:58:32.817653 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:58:32.817628 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/5265b78a-bee3-4147-aa9b-0be306cd00c9-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-api-a-a8f21c-smpl8\" (UID: \"5265b78a-bee3-4147-aa9b-0be306cd00c9\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-a8f21c-smpl8" Jul 23 19:58:32.818116 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:58:32.818100 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/5265b78a-bee3-4147-aa9b-0be306cd00c9-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-api-a-a8f21c-smpl8\" (UID: \"5265b78a-bee3-4147-aa9b-0be306cd00c9\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-a8f21c-smpl8" Jul 23 19:58:32.994369 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:58:32.994296 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-a8f21c-smpl8" Jul 23 19:58:33.123183 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:58:33.123158 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-a8f21c-smpl8"] Jul 23 19:58:33.124879 ip-10-0-129-2 kubenswrapper[2564]: W0723 19:58:33.124854 2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5265b78a_bee3_4147_aa9b_0be306cd00c9.slice/crio-0b06558d989717fa2b32cdaed441deb597c092bd8e96878018ec40aef73a6f7e WatchSource:0}: Error finding container 0b06558d989717fa2b32cdaed441deb597c092bd8e96878018ec40aef73a6f7e: Status 404 returned error can't find the container with id 0b06558d989717fa2b32cdaed441deb597c092bd8e96878018ec40aef73a6f7e Jul 23 19:58:33.593951 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:58:33.593915 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-a8f21c-smpl8" event={"ID":"5265b78a-bee3-4147-aa9b-0be306cd00c9","Type":"ContainerStarted","Data":"b243218bd90e53ba6e7e4fe757f9301ebe371135e27b734fabd9f87bd25204d7"} Jul 23 19:58:33.594088 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:58:33.593959 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-a8f21c-smpl8" event={"ID":"5265b78a-bee3-4147-aa9b-0be306cd00c9","Type":"ContainerStarted","Data":"0b06558d989717fa2b32cdaed441deb597c092bd8e96878018ec40aef73a6f7e"} Jul 23 19:58:33.610424 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:58:33.610381 2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-a8f21c-smpl8" podStartSLOduration=1.610367048 podStartE2EDuration="1.610367048s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-23 19:58:32 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-23 19:58:33.608382473 +0000 UTC m=+2339.460158718" watchObservedRunningTime="2026-07-23 19:58:33.610367048 +0000 UTC m=+2339.462143293" Jul 23 19:58:34.600861 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:58:34.600832 2564 generic.go:356] "Generic (PLEG): container finished" podID="5265b78a-bee3-4147-aa9b-0be306cd00c9" containerID="b243218bd90e53ba6e7e4fe757f9301ebe371135e27b734fabd9f87bd25204d7" exitCode=0 Jul 23 19:58:34.601204 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:58:34.600903 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-a8f21c-smpl8" event={"ID":"5265b78a-bee3-4147-aa9b-0be306cd00c9","Type":"ContainerDied","Data":"b243218bd90e53ba6e7e4fe757f9301ebe371135e27b734fabd9f87bd25204d7"} Jul 23 19:58:35.667369 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:58:35.667344 2564 scope.go:117] "RemoveContainer" containerID="fdb891652400427214ac9a8410fdec3c6561b44bf2f3a49df72c8f2c41c16fab" Jul 23 19:58:35.740730 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:58:35.740709 2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-a8f21c-smpl8" Jul 23 19:58:35.840950 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:58:35.840928 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/5265b78a-bee3-4147-aa9b-0be306cd00c9-maas-api-service-ca\") pod \"5265b78a-bee3-4147-aa9b-0be306cd00c9\" (UID: \"5265b78a-bee3-4147-aa9b-0be306cd00c9\") " Jul 23 19:58:35.841236 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:58:35.841215 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5265b78a-bee3-4147-aa9b-0be306cd00c9-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "5265b78a-bee3-4147-aa9b-0be306cd00c9" (UID: "5265b78a-bee3-4147-aa9b-0be306cd00c9"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 23 19:58:35.942040 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:58:35.942007 2564 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/5265b78a-bee3-4147-aa9b-0be306cd00c9-maas-api-service-ca\") on node \"ip-10-0-129-2.ec2.internal\" DevicePath \"\"" Jul 23 19:58:36.619773 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:58:36.619751 2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-a8f21c-smpl8" Jul 23 19:58:36.619877 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:58:36.619789 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-a8f21c-smpl8" event={"ID":"5265b78a-bee3-4147-aa9b-0be306cd00c9","Type":"ContainerDied","Data":"0b06558d989717fa2b32cdaed441deb597c092bd8e96878018ec40aef73a6f7e"} Jul 23 19:58:36.619877 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:58:36.619859 2564 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="0b06558d989717fa2b32cdaed441deb597c092bd8e96878018ec40aef73a6f7e" Jul 23 19:58:37.893275 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:58:37.893240 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-2dgmk"] Jul 23 19:58:48.509496 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:58:48.509465 2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-a8f21c-6d87bfbc6f-pqz9w"] Jul 23 19:58:48.510066 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:58:48.509709 2564 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-a8f21c-6d87bfbc6f-pqz9w" podUID="da86ae6b-59a2-4591-b2a2-37ca04bc0ccd" containerName="maas-api" containerID="cri-o://9f91edc99b31856730ea1da26d7ce58b3d75c91adcfa16e78636cd1ca24b4567" gracePeriod=30 Jul 23 19:58:48.681829 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:58:48.681799 2564 generic.go:356] "Generic (PLEG): container finished" podID="da86ae6b-59a2-4591-b2a2-37ca04bc0ccd" containerID="9f91edc99b31856730ea1da26d7ce58b3d75c91adcfa16e78636cd1ca24b4567" exitCode=0 Jul 23 19:58:48.681983 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:58:48.681859 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-a8f21c-6d87bfbc6f-pqz9w" event={"ID":"da86ae6b-59a2-4591-b2a2-37ca04bc0ccd","Type":"ContainerDied","Data":"9f91edc99b31856730ea1da26d7ce58b3d75c91adcfa16e78636cd1ca24b4567"} Jul 23 19:58:48.766843 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:58:48.766780 2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-a8f21c-6d87bfbc6f-pqz9w" Jul 23 19:58:48.840582 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:58:48.840552 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-lsf8m\" (UniqueName: \"kubernetes.io/projected/da86ae6b-59a2-4591-b2a2-37ca04bc0ccd-kube-api-access-lsf8m\") pod \"da86ae6b-59a2-4591-b2a2-37ca04bc0ccd\" (UID: \"da86ae6b-59a2-4591-b2a2-37ca04bc0ccd\") " Jul 23 19:58:48.840730 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:58:48.840611 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/da86ae6b-59a2-4591-b2a2-37ca04bc0ccd-openshift-service-ca\") pod \"da86ae6b-59a2-4591-b2a2-37ca04bc0ccd\" (UID: \"da86ae6b-59a2-4591-b2a2-37ca04bc0ccd\") " Jul 23 19:58:48.840730 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:58:48.840652 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/da86ae6b-59a2-4591-b2a2-37ca04bc0ccd-maas-api-tls\") pod \"da86ae6b-59a2-4591-b2a2-37ca04bc0ccd\" (UID: \"da86ae6b-59a2-4591-b2a2-37ca04bc0ccd\") " Jul 23 19:58:48.841007 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:58:48.840980 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/da86ae6b-59a2-4591-b2a2-37ca04bc0ccd-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "da86ae6b-59a2-4591-b2a2-37ca04bc0ccd" (UID: "da86ae6b-59a2-4591-b2a2-37ca04bc0ccd"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 23 19:58:48.842623 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:58:48.842601 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/da86ae6b-59a2-4591-b2a2-37ca04bc0ccd-kube-api-access-lsf8m" (OuterVolumeSpecName: "kube-api-access-lsf8m") pod "da86ae6b-59a2-4591-b2a2-37ca04bc0ccd" (UID: "da86ae6b-59a2-4591-b2a2-37ca04bc0ccd"). InnerVolumeSpecName "kube-api-access-lsf8m". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 23 19:58:48.842740 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:58:48.842725 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/da86ae6b-59a2-4591-b2a2-37ca04bc0ccd-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "da86ae6b-59a2-4591-b2a2-37ca04bc0ccd" (UID: "da86ae6b-59a2-4591-b2a2-37ca04bc0ccd"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 23 19:58:48.941383 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:58:48.941354 2564 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lsf8m\" (UniqueName: \"kubernetes.io/projected/da86ae6b-59a2-4591-b2a2-37ca04bc0ccd-kube-api-access-lsf8m\") on node \"ip-10-0-129-2.ec2.internal\" DevicePath \"\"" Jul 23 19:58:48.941383 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:58:48.941383 2564 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/da86ae6b-59a2-4591-b2a2-37ca04bc0ccd-openshift-service-ca\") on node \"ip-10-0-129-2.ec2.internal\" DevicePath \"\"" Jul 23 19:58:48.941383 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:58:48.941392 2564 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/da86ae6b-59a2-4591-b2a2-37ca04bc0ccd-maas-api-tls\") on node \"ip-10-0-129-2.ec2.internal\" DevicePath \"\"" Jul 23 19:58:49.688512 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:58:49.688483 2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-a8f21c-6d87bfbc6f-pqz9w" Jul 23 19:58:49.688951 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:58:49.688521 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-a8f21c-6d87bfbc6f-pqz9w" event={"ID":"da86ae6b-59a2-4591-b2a2-37ca04bc0ccd","Type":"ContainerDied","Data":"4cf1311bb0f5f4a9d28a11d6909a1cad53c45cad0800559d8cf4c8658d358973"} Jul 23 19:58:49.688951 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:58:49.688583 2564 scope.go:117] "RemoveContainer" containerID="9f91edc99b31856730ea1da26d7ce58b3d75c91adcfa16e78636cd1ca24b4567" Jul 23 19:58:49.727713 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:58:49.727692 2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-a8f21c-6d87bfbc6f-pqz9w"] Jul 23 19:58:49.729797 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:58:49.729778 2564 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-a8f21c-6d87bfbc6f-pqz9w"] Jul 23 19:58:50.657764 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:58:50.657743 2564 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="da86ae6b-59a2-4591-b2a2-37ca04bc0ccd" path="/var/lib/kubelet/pods/da86ae6b-59a2-4591-b2a2-37ca04bc0ccd/volumes" Jul 23 19:58:52.798557 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:58:52.798527 2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-a8f21c-smpl8"] Jul 23 19:58:52.801518 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:58:52.801497 2564 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-a8f21c-smpl8"] Jul 23 19:58:53.791214 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:58:53.791176 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-2dgmk"] Jul 23 19:58:54.664617 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:58:54.664584 2564 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="5265b78a-bee3-4147-aa9b-0be306cd00c9" path="/var/lib/kubelet/pods/5265b78a-bee3-4147-aa9b-0be306cd00c9/volumes" Jul 23 19:58:59.677800 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:58:59.677769 2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-1d04d2-rhmsq"] Jul 23 19:58:59.678189 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:58:59.678141 2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5265b78a-bee3-4147-aa9b-0be306cd00c9" containerName="revoke-keys" Jul 23 19:58:59.678189 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:58:59.678153 2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="5265b78a-bee3-4147-aa9b-0be306cd00c9" containerName="revoke-keys" Jul 23 19:58:59.678189 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:58:59.678170 2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="da86ae6b-59a2-4591-b2a2-37ca04bc0ccd" containerName="maas-api" Jul 23 19:58:59.678189 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:58:59.678175 2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="da86ae6b-59a2-4591-b2a2-37ca04bc0ccd" containerName="maas-api" Jul 23 19:58:59.678376 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:58:59.678235 2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="da86ae6b-59a2-4591-b2a2-37ca04bc0ccd" containerName="maas-api" Jul 23 19:58:59.678376 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:58:59.678245 2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="5265b78a-bee3-4147-aa9b-0be306cd00c9" containerName="revoke-keys" Jul 23 19:58:59.682554 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:58:59.682536 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-1d04d2-rhmsq" Jul 23 19:58:59.684920 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:58:59.684894 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-gcspk\"" Jul 23 19:58:59.688339 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:58:59.688293 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-1d04d2-rhmsq"] Jul 23 19:58:59.721309 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:58:59.721288 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/45499ae4-d0ae-4cdf-b214-9075f62c27ba-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-api-b-1d04d2-rhmsq\" (UID: \"45499ae4-d0ae-4cdf-b214-9075f62c27ba\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-1d04d2-rhmsq" Jul 23 19:58:59.822357 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:58:59.822313 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/45499ae4-d0ae-4cdf-b214-9075f62c27ba-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-api-b-1d04d2-rhmsq\" (UID: \"45499ae4-d0ae-4cdf-b214-9075f62c27ba\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-1d04d2-rhmsq" Jul 23 19:58:59.822817 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:58:59.822801 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/45499ae4-d0ae-4cdf-b214-9075f62c27ba-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-api-b-1d04d2-rhmsq\" (UID: \"45499ae4-d0ae-4cdf-b214-9075f62c27ba\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-1d04d2-rhmsq" Jul 23 19:58:59.994249 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:58:59.994198 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-1d04d2-rhmsq" Jul 23 19:59:00.144267 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:59:00.144239 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-1d04d2-rhmsq"] Jul 23 19:59:00.145754 ip-10-0-129-2 kubenswrapper[2564]: W0723 19:59:00.145726 2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod45499ae4_d0ae_4cdf_b214_9075f62c27ba.slice/crio-969d84bda70a4662c0fe2f78c3551806086786d4addd668e8f3142dee56c33b7 WatchSource:0}: Error finding container 969d84bda70a4662c0fe2f78c3551806086786d4addd668e8f3142dee56c33b7: Status 404 returned error can't find the container with id 969d84bda70a4662c0fe2f78c3551806086786d4addd668e8f3142dee56c33b7 Jul 23 19:59:00.750347 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:59:00.750286 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-1d04d2-rhmsq" event={"ID":"45499ae4-d0ae-4cdf-b214-9075f62c27ba","Type":"ContainerStarted","Data":"f40a25df307a3d2fc204b0d2d88b81b62e4b35df742003902f2c40bbbfce7711"} Jul 23 19:59:00.750686 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:59:00.750386 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-1d04d2-rhmsq" event={"ID":"45499ae4-d0ae-4cdf-b214-9075f62c27ba","Type":"ContainerStarted","Data":"969d84bda70a4662c0fe2f78c3551806086786d4addd668e8f3142dee56c33b7"} Jul 23 19:59:00.768462 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:59:00.768414 2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-1d04d2-rhmsq" podStartSLOduration=1.76840033 podStartE2EDuration="1.76840033s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-23 19:58:59 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-23 19:59:00.765931205 +0000 UTC m=+2366.617707453" watchObservedRunningTime="2026-07-23 19:59:00.76840033 +0000 UTC m=+2366.620176555" Jul 23 19:59:01.757060 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:59:01.757031 2564 generic.go:356] "Generic (PLEG): container finished" podID="45499ae4-d0ae-4cdf-b214-9075f62c27ba" containerID="f40a25df307a3d2fc204b0d2d88b81b62e4b35df742003902f2c40bbbfce7711" exitCode=0 Jul 23 19:59:01.757456 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:59:01.757108 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-1d04d2-rhmsq" event={"ID":"45499ae4-d0ae-4cdf-b214-9075f62c27ba","Type":"ContainerDied","Data":"f40a25df307a3d2fc204b0d2d88b81b62e4b35df742003902f2c40bbbfce7711"} Jul 23 19:59:02.893392 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:59:02.893361 2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-1d04d2-rhmsq" Jul 23 19:59:02.949552 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:59:02.949526 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/45499ae4-d0ae-4cdf-b214-9075f62c27ba-maas-api-service-ca\") pod \"45499ae4-d0ae-4cdf-b214-9075f62c27ba\" (UID: \"45499ae4-d0ae-4cdf-b214-9075f62c27ba\") " Jul 23 19:59:02.949830 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:59:02.949810 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/45499ae4-d0ae-4cdf-b214-9075f62c27ba-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "45499ae4-d0ae-4cdf-b214-9075f62c27ba" (UID: "45499ae4-d0ae-4cdf-b214-9075f62c27ba"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 23 19:59:03.050784 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:59:03.050731 2564 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/45499ae4-d0ae-4cdf-b214-9075f62c27ba-maas-api-service-ca\") on node \"ip-10-0-129-2.ec2.internal\" DevicePath \"\"" Jul 23 19:59:03.769095 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:59:03.769072 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-1d04d2-rhmsq" event={"ID":"45499ae4-d0ae-4cdf-b214-9075f62c27ba","Type":"ContainerDied","Data":"969d84bda70a4662c0fe2f78c3551806086786d4addd668e8f3142dee56c33b7"} Jul 23 19:59:03.769245 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:59:03.769117 2564 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="969d84bda70a4662c0fe2f78c3551806086786d4addd668e8f3142dee56c33b7" Jul 23 19:59:03.769245 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:59:03.769077 2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-1d04d2-rhmsq" Jul 23 19:59:15.252698 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:59:15.252663 2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-1d04d2-69bdd998f6-jgxpl"] Jul 23 19:59:15.253140 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:59:15.252918 2564 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-1d04d2-69bdd998f6-jgxpl" podUID="6b810ffd-523a-4f5e-9ac6-74336267d9f0" containerName="maas-api" containerID="cri-o://50cd716660261aeaf08d7d8d88c77c5204e6375379d95f6e96fd864fb5f8dae1" gracePeriod=30 Jul 23 19:59:15.487397 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:59:15.487370 2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-1d04d2-69bdd998f6-jgxpl" Jul 23 19:59:15.540862 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:59:15.540800 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/6b810ffd-523a-4f5e-9ac6-74336267d9f0-openshift-service-ca\") pod \"6b810ffd-523a-4f5e-9ac6-74336267d9f0\" (UID: \"6b810ffd-523a-4f5e-9ac6-74336267d9f0\") " Jul 23 19:59:15.540862 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:59:15.540854 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6b810ffd-523a-4f5e-9ac6-74336267d9f0-maas-api-tls\") pod \"6b810ffd-523a-4f5e-9ac6-74336267d9f0\" (UID: \"6b810ffd-523a-4f5e-9ac6-74336267d9f0\") " Jul 23 19:59:15.540999 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:59:15.540894 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-hgbrx\" (UniqueName: \"kubernetes.io/projected/6b810ffd-523a-4f5e-9ac6-74336267d9f0-kube-api-access-hgbrx\") pod \"6b810ffd-523a-4f5e-9ac6-74336267d9f0\" (UID: \"6b810ffd-523a-4f5e-9ac6-74336267d9f0\") " Jul 23 19:59:15.541157 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:59:15.541135 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6b810ffd-523a-4f5e-9ac6-74336267d9f0-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "6b810ffd-523a-4f5e-9ac6-74336267d9f0" (UID: "6b810ffd-523a-4f5e-9ac6-74336267d9f0"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 23 19:59:15.542925 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:59:15.542902 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6b810ffd-523a-4f5e-9ac6-74336267d9f0-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "6b810ffd-523a-4f5e-9ac6-74336267d9f0" (UID: "6b810ffd-523a-4f5e-9ac6-74336267d9f0"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 23 19:59:15.543029 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:59:15.543016 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6b810ffd-523a-4f5e-9ac6-74336267d9f0-kube-api-access-hgbrx" (OuterVolumeSpecName: "kube-api-access-hgbrx") pod "6b810ffd-523a-4f5e-9ac6-74336267d9f0" (UID: "6b810ffd-523a-4f5e-9ac6-74336267d9f0"). InnerVolumeSpecName "kube-api-access-hgbrx". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 23 19:59:15.641971 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:59:15.641948 2564 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hgbrx\" (UniqueName: \"kubernetes.io/projected/6b810ffd-523a-4f5e-9ac6-74336267d9f0-kube-api-access-hgbrx\") on node \"ip-10-0-129-2.ec2.internal\" DevicePath \"\"" Jul 23 19:59:15.641971 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:59:15.641969 2564 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/6b810ffd-523a-4f5e-9ac6-74336267d9f0-openshift-service-ca\") on node \"ip-10-0-129-2.ec2.internal\" DevicePath \"\"" Jul 23 19:59:15.642101 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:59:15.641978 2564 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6b810ffd-523a-4f5e-9ac6-74336267d9f0-maas-api-tls\") on node \"ip-10-0-129-2.ec2.internal\" DevicePath \"\"" Jul 23 19:59:15.835589 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:59:15.835527 2564 generic.go:356] "Generic (PLEG): container finished" podID="6b810ffd-523a-4f5e-9ac6-74336267d9f0" containerID="50cd716660261aeaf08d7d8d88c77c5204e6375379d95f6e96fd864fb5f8dae1" exitCode=0 Jul 23 19:59:15.835589 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:59:15.835572 2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-1d04d2-69bdd998f6-jgxpl" Jul 23 19:59:15.835715 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:59:15.835606 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-1d04d2-69bdd998f6-jgxpl" event={"ID":"6b810ffd-523a-4f5e-9ac6-74336267d9f0","Type":"ContainerDied","Data":"50cd716660261aeaf08d7d8d88c77c5204e6375379d95f6e96fd864fb5f8dae1"} Jul 23 19:59:15.835715 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:59:15.835656 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-1d04d2-69bdd998f6-jgxpl" event={"ID":"6b810ffd-523a-4f5e-9ac6-74336267d9f0","Type":"ContainerDied","Data":"e05e8284e63efe023bdaf07c6c38ebcd8f859b78534046fc5e549fa4fba99709"} Jul 23 19:59:15.835715 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:59:15.835678 2564 scope.go:117] "RemoveContainer" containerID="50cd716660261aeaf08d7d8d88c77c5204e6375379d95f6e96fd864fb5f8dae1" Jul 23 19:59:15.844939 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:59:15.844923 2564 scope.go:117] "RemoveContainer" containerID="50cd716660261aeaf08d7d8d88c77c5204e6375379d95f6e96fd864fb5f8dae1" Jul 23 19:59:15.845193 ip-10-0-129-2 kubenswrapper[2564]: E0723 19:59:15.845174 2564 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"50cd716660261aeaf08d7d8d88c77c5204e6375379d95f6e96fd864fb5f8dae1\": container with ID starting with 50cd716660261aeaf08d7d8d88c77c5204e6375379d95f6e96fd864fb5f8dae1 not found: ID does not exist" containerID="50cd716660261aeaf08d7d8d88c77c5204e6375379d95f6e96fd864fb5f8dae1" Jul 23 19:59:15.845236 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:59:15.845201 2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"50cd716660261aeaf08d7d8d88c77c5204e6375379d95f6e96fd864fb5f8dae1"} err="failed to get container status \"50cd716660261aeaf08d7d8d88c77c5204e6375379d95f6e96fd864fb5f8dae1\": rpc error: code = NotFound desc = could not find container \"50cd716660261aeaf08d7d8d88c77c5204e6375379d95f6e96fd864fb5f8dae1\": container with ID starting with 50cd716660261aeaf08d7d8d88c77c5204e6375379d95f6e96fd864fb5f8dae1 not found: ID does not exist" Jul 23 19:59:15.856625 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:59:15.856603 2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-1d04d2-69bdd998f6-jgxpl"] Jul 23 19:59:15.861973 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:59:15.861953 2564 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-1d04d2-69bdd998f6-jgxpl"] Jul 23 19:59:16.657745 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:59:16.657724 2564 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="6b810ffd-523a-4f5e-9ac6-74336267d9f0" path="/var/lib/kubelet/pods/6b810ffd-523a-4f5e-9ac6-74336267d9f0/volumes" Jul 23 19:59:19.756664 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:59:19.756635 2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-1d04d2-rhmsq"] Jul 23 19:59:19.762698 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:59:19.762672 2564 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-1d04d2-rhmsq"] Jul 23 19:59:19.886634 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:59:19.886573 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-2dgmk"] Jul 23 19:59:20.658780 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:59:20.658750 2564 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="45499ae4-d0ae-4cdf-b214-9075f62c27ba" path="/var/lib/kubelet/pods/45499ae4-d0ae-4cdf-b214-9075f62c27ba/volumes" Jul 23 19:59:29.617057 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:59:29.617020 2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-fb64e5-7699755b7c-gjzt9"] Jul 23 19:59:29.617639 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:59:29.617621 2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="45499ae4-d0ae-4cdf-b214-9075f62c27ba" containerName="revoke-keys" Jul 23 19:59:29.617685 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:59:29.617642 2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="45499ae4-d0ae-4cdf-b214-9075f62c27ba" containerName="revoke-keys" Jul 23 19:59:29.617685 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:59:29.617657 2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6b810ffd-523a-4f5e-9ac6-74336267d9f0" containerName="maas-api" Jul 23 19:59:29.617685 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:59:29.617666 2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="6b810ffd-523a-4f5e-9ac6-74336267d9f0" containerName="maas-api" Jul 23 19:59:29.617781 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:59:29.617760 2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="6b810ffd-523a-4f5e-9ac6-74336267d9f0" containerName="maas-api" Jul 23 19:59:29.617781 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:59:29.617774 2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="45499ae4-d0ae-4cdf-b214-9075f62c27ba" containerName="revoke-keys" Jul 23 19:59:29.622410 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:59:29.622392 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-fb64e5-7699755b7c-gjzt9" Jul 23 19:59:29.624894 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:59:29.624870 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Jul 23 19:59:29.625129 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:59:29.625107 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-7m48c\"" Jul 23 19:59:29.625290 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:59:29.625266 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Jul 23 19:59:29.625406 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:59:29.625273 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-shared-a-fb64e5-serving-cert\"" Jul 23 19:59:29.633258 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:59:29.633236 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-fb64e5-7699755b7c-gjzt9"] Jul 23 19:59:29.754625 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:59:29.754591 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/19811d4d-4066-47ed-9ae3-0c43d8fb6b93-maas-api-tls\") pod \"maas-api-e2e-shared-a-fb64e5-7699755b7c-gjzt9\" (UID: \"19811d4d-4066-47ed-9ae3-0c43d8fb6b93\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-fb64e5-7699755b7c-gjzt9" Jul 23 19:59:29.754625 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:59:29.754627 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/19811d4d-4066-47ed-9ae3-0c43d8fb6b93-openshift-service-ca\") pod \"maas-api-e2e-shared-a-fb64e5-7699755b7c-gjzt9\" (UID: \"19811d4d-4066-47ed-9ae3-0c43d8fb6b93\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-fb64e5-7699755b7c-gjzt9" Jul 23 19:59:29.754839 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:59:29.754733 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-khkkz\" (UniqueName: \"kubernetes.io/projected/19811d4d-4066-47ed-9ae3-0c43d8fb6b93-kube-api-access-khkkz\") pod \"maas-api-e2e-shared-a-fb64e5-7699755b7c-gjzt9\" (UID: \"19811d4d-4066-47ed-9ae3-0c43d8fb6b93\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-fb64e5-7699755b7c-gjzt9" Jul 23 19:59:29.856159 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:59:29.856121 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-khkkz\" (UniqueName: \"kubernetes.io/projected/19811d4d-4066-47ed-9ae3-0c43d8fb6b93-kube-api-access-khkkz\") pod \"maas-api-e2e-shared-a-fb64e5-7699755b7c-gjzt9\" (UID: \"19811d4d-4066-47ed-9ae3-0c43d8fb6b93\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-fb64e5-7699755b7c-gjzt9" Jul 23 19:59:29.856369 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:59:29.856170 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/19811d4d-4066-47ed-9ae3-0c43d8fb6b93-maas-api-tls\") pod \"maas-api-e2e-shared-a-fb64e5-7699755b7c-gjzt9\" (UID: \"19811d4d-4066-47ed-9ae3-0c43d8fb6b93\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-fb64e5-7699755b7c-gjzt9" Jul 23 19:59:29.856369 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:59:29.856193 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/19811d4d-4066-47ed-9ae3-0c43d8fb6b93-openshift-service-ca\") pod \"maas-api-e2e-shared-a-fb64e5-7699755b7c-gjzt9\" (UID: \"19811d4d-4066-47ed-9ae3-0c43d8fb6b93\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-fb64e5-7699755b7c-gjzt9" Jul 23 19:59:29.856369 ip-10-0-129-2 kubenswrapper[2564]: E0723 19:59:29.856275 2564 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-shared-a-fb64e5-serving-cert: secret "maas-api-e2e-shared-a-fb64e5-serving-cert" not found Jul 23 19:59:29.856369 ip-10-0-129-2 kubenswrapper[2564]: E0723 19:59:29.856359 2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/19811d4d-4066-47ed-9ae3-0c43d8fb6b93-maas-api-tls podName:19811d4d-4066-47ed-9ae3-0c43d8fb6b93 nodeName:}" failed. No retries permitted until 2026-07-23 19:59:30.356343666 +0000 UTC m=+2396.208119893 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/19811d4d-4066-47ed-9ae3-0c43d8fb6b93-maas-api-tls") pod "maas-api-e2e-shared-a-fb64e5-7699755b7c-gjzt9" (UID: "19811d4d-4066-47ed-9ae3-0c43d8fb6b93") : secret "maas-api-e2e-shared-a-fb64e5-serving-cert" not found Jul 23 19:59:29.856812 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:59:29.856796 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/19811d4d-4066-47ed-9ae3-0c43d8fb6b93-openshift-service-ca\") pod \"maas-api-e2e-shared-a-fb64e5-7699755b7c-gjzt9\" (UID: \"19811d4d-4066-47ed-9ae3-0c43d8fb6b93\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-fb64e5-7699755b7c-gjzt9" Jul 23 19:59:29.864848 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:59:29.864826 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-khkkz\" (UniqueName: \"kubernetes.io/projected/19811d4d-4066-47ed-9ae3-0c43d8fb6b93-kube-api-access-khkkz\") pod \"maas-api-e2e-shared-a-fb64e5-7699755b7c-gjzt9\" (UID: \"19811d4d-4066-47ed-9ae3-0c43d8fb6b93\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-fb64e5-7699755b7c-gjzt9" Jul 23 19:59:30.361983 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:59:30.361951 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/19811d4d-4066-47ed-9ae3-0c43d8fb6b93-maas-api-tls\") pod \"maas-api-e2e-shared-a-fb64e5-7699755b7c-gjzt9\" (UID: \"19811d4d-4066-47ed-9ae3-0c43d8fb6b93\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-fb64e5-7699755b7c-gjzt9" Jul 23 19:59:30.364496 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:59:30.364467 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/19811d4d-4066-47ed-9ae3-0c43d8fb6b93-maas-api-tls\") pod \"maas-api-e2e-shared-a-fb64e5-7699755b7c-gjzt9\" (UID: \"19811d4d-4066-47ed-9ae3-0c43d8fb6b93\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-fb64e5-7699755b7c-gjzt9" Jul 23 19:59:30.534710 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:59:30.534674 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-fb64e5-7699755b7c-gjzt9" Jul 23 19:59:30.669913 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:59:30.669885 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-fb64e5-7699755b7c-gjzt9"] Jul 23 19:59:30.671609 ip-10-0-129-2 kubenswrapper[2564]: W0723 19:59:30.671575 2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod19811d4d_4066_47ed_9ae3_0c43d8fb6b93.slice/crio-9e9c53fe3f3f948f4b05a95cc9e1a746a3c34e49168afab68e711055a2eabebf WatchSource:0}: Error finding container 9e9c53fe3f3f948f4b05a95cc9e1a746a3c34e49168afab68e711055a2eabebf: Status 404 returned error can't find the container with id 9e9c53fe3f3f948f4b05a95cc9e1a746a3c34e49168afab68e711055a2eabebf Jul 23 19:59:30.920370 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:59:30.920340 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-fb64e5-7699755b7c-gjzt9" event={"ID":"19811d4d-4066-47ed-9ae3-0c43d8fb6b93","Type":"ContainerStarted","Data":"9e9c53fe3f3f948f4b05a95cc9e1a746a3c34e49168afab68e711055a2eabebf"} Jul 23 19:59:31.927795 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:59:31.927758 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-fb64e5-7699755b7c-gjzt9" event={"ID":"19811d4d-4066-47ed-9ae3-0c43d8fb6b93","Type":"ContainerStarted","Data":"bf6dd07cd7a72071cadb699d6763690688de9fb80737df4136ed19eec23b4a7f"} Jul 23 19:59:31.928202 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:59:31.927860 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-fb64e5-7699755b7c-gjzt9" Jul 23 19:59:31.946031 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:59:31.945994 2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-fb64e5-7699755b7c-gjzt9" podStartSLOduration=2.537718494 podStartE2EDuration="2.945980904s" totalImagesPullingTime="408.26241ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-23 19:59:29 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-23 19:59:31.945058034 +0000 UTC m=+2397.796834305" watchObservedRunningTime="2026-07-23 19:59:31.945980904 +0000 UTC m=+2397.797757197" Jul 23 19:59:35.602060 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:59:35.602038 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-565b7b569d-z5wf4_ded5b3af-c74c-4d5a-aa77-62fe72d1dd29/console-operator/1.log" Jul 23 19:59:35.608941 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:59:35.608922 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-8dtlh_195ecf89-13ae-4311-bc72-fcdf78a5c4d6/ovn-acl-logging/0.log" Jul 23 19:59:35.611563 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:59:35.611545 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-565b7b569d-z5wf4_ded5b3af-c74c-4d5a-aa77-62fe72d1dd29/console-operator/1.log" Jul 23 19:59:35.618123 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:59:35.618105 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-8dtlh_195ecf89-13ae-4311-bc72-fcdf78a5c4d6/ovn-acl-logging/0.log" Jul 23 19:59:36.893136 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:59:36.893106 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-2dgmk"] Jul 23 19:59:37.939655 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:59:37.939287 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-fb64e5-7699755b7c-gjzt9" Jul 23 19:59:51.398457 ip-10-0-129-2 kubenswrapper[2564]: I0723 19:59:51.398424 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-2dgmk"] Jul 23 20:00:00.137702 ip-10-0-129-2 kubenswrapper[2564]: I0723 20:00:00.137628 2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-470e58-29747280-hrrmv"] Jul 23 20:00:00.144660 ip-10-0-129-2 kubenswrapper[2564]: I0723 20:00:00.144640 2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29747280-vw88h"] Jul 23 20:00:00.144779 ip-10-0-129-2 kubenswrapper[2564]: I0723 20:00:00.144747 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-470e58-29747280-hrrmv" Jul 23 20:00:00.147544 ip-10-0-129-2 kubenswrapper[2564]: I0723 20:00:00.147525 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-gcspk\"" Jul 23 20:00:00.147840 ip-10-0-129-2 kubenswrapper[2564]: I0723 20:00:00.147827 2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-fb64e5-29747280-b52r7"] Jul 23 20:00:00.147963 ip-10-0-129-2 kubenswrapper[2564]: I0723 20:00:00.147948 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29747280-vw88h" Jul 23 20:00:00.151527 ip-10-0-129-2 kubenswrapper[2564]: I0723 20:00:00.151507 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-fb64e5-29747280-b52r7" Jul 23 20:00:00.152171 ip-10-0-129-2 kubenswrapper[2564]: I0723 20:00:00.152151 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-470e58-29747280-hrrmv"] Jul 23 20:00:00.155506 ip-10-0-129-2 kubenswrapper[2564]: I0723 20:00:00.155484 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29747280-vw88h"] Jul 23 20:00:00.158950 ip-10-0-129-2 kubenswrapper[2564]: I0723 20:00:00.158927 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-fb64e5-29747280-b52r7"] Jul 23 20:00:00.159827 ip-10-0-129-2 kubenswrapper[2564]: I0723 20:00:00.159806 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-470e58-29747280-hrrmv" Jul 23 20:00:00.170358 ip-10-0-129-2 kubenswrapper[2564]: I0723 20:00:00.168691 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29747280-vw88h" Jul 23 20:00:00.207794 ip-10-0-129-2 kubenswrapper[2564]: I0723 20:00:00.207771 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-fb64e5-29747280-b52r7" Jul 23 20:00:00.323511 ip-10-0-129-2 kubenswrapper[2564]: I0723 20:00:00.323486 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-470e58-29747280-hrrmv"] Jul 23 20:00:00.326449 ip-10-0-129-2 kubenswrapper[2564]: W0723 20:00:00.326421 2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod22392289_8b04_48dd_8a18_43a79fcb5dda.slice/crio-5ed3df800dc61df085c9f4797c5e16bd757bb01bec2fb7c7b3a8e6fa7fb94c59 WatchSource:0}: Error finding container 5ed3df800dc61df085c9f4797c5e16bd757bb01bec2fb7c7b3a8e6fa7fb94c59: Status 404 returned error can't find the container with id 5ed3df800dc61df085c9f4797c5e16bd757bb01bec2fb7c7b3a8e6fa7fb94c59 Jul 23 20:00:00.379709 ip-10-0-129-2 kubenswrapper[2564]: I0723 20:00:00.379682 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-fb64e5-29747280-b52r7"] Jul 23 20:00:00.380066 ip-10-0-129-2 kubenswrapper[2564]: W0723 20:00:00.380039 2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podde7f66c4_068a_4f4a_9f10_fde3c0bac17c.slice/crio-e38c0f9de619f4822ba661e3c64b443c81eae761f7e54fbc327d661d3fec409f WatchSource:0}: Error finding container e38c0f9de619f4822ba661e3c64b443c81eae761f7e54fbc327d661d3fec409f: Status 404 returned error can't find the container with id e38c0f9de619f4822ba661e3c64b443c81eae761f7e54fbc327d661d3fec409f Jul 23 20:00:00.561631 ip-10-0-129-2 kubenswrapper[2564]: W0723 20:00:00.561580 2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2432a745_c8d5_4505_a20a_4d82fe439720.slice/crio-c9e1f41e048b9a6951815b816d0c57a94daacf48d58fb1c3da289a78d7dca0d9 WatchSource:0}: Error finding container c9e1f41e048b9a6951815b816d0c57a94daacf48d58fb1c3da289a78d7dca0d9: Status 404 returned error can't find the container with id c9e1f41e048b9a6951815b816d0c57a94daacf48d58fb1c3da289a78d7dca0d9 Jul 23 20:00:00.561916 ip-10-0-129-2 kubenswrapper[2564]: I0723 20:00:00.561893 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29747280-vw88h"] Jul 23 20:00:01.086341 ip-10-0-129-2 kubenswrapper[2564]: I0723 20:00:01.086249 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-fb64e5-29747280-b52r7" event={"ID":"de7f66c4-068a-4f4a-9f10-fde3c0bac17c","Type":"ContainerStarted","Data":"97ff3937d8a8a97b25b2e80df47f68b2e514ab7f4dcf6b67c03be6143ffaa334"} Jul 23 20:00:01.086341 ip-10-0-129-2 kubenswrapper[2564]: I0723 20:00:01.086317 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-fb64e5-29747280-b52r7" event={"ID":"de7f66c4-068a-4f4a-9f10-fde3c0bac17c","Type":"ContainerStarted","Data":"e38c0f9de619f4822ba661e3c64b443c81eae761f7e54fbc327d661d3fec409f"} Jul 23 20:00:01.087826 ip-10-0-129-2 kubenswrapper[2564]: I0723 20:00:01.087793 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29747280-vw88h" event={"ID":"2432a745-c8d5-4505-a20a-4d82fe439720","Type":"ContainerStarted","Data":"aa1ffbff30021be7c38c18df3da3a24b8376bc8c1981316b766cf9f7d6d2a915"} Jul 23 20:00:01.087944 ip-10-0-129-2 kubenswrapper[2564]: I0723 20:00:01.087848 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29747280-vw88h" event={"ID":"2432a745-c8d5-4505-a20a-4d82fe439720","Type":"ContainerStarted","Data":"c9e1f41e048b9a6951815b816d0c57a94daacf48d58fb1c3da289a78d7dca0d9"} Jul 23 20:00:01.089292 ip-10-0-129-2 kubenswrapper[2564]: I0723 20:00:01.089272 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-470e58-29747280-hrrmv" event={"ID":"22392289-8b04-48dd-8a18-43a79fcb5dda","Type":"ContainerStarted","Data":"3bbc962944c2c3400b6aa87601f4859d68f3963398a77688d305289812c7d90e"} Jul 23 20:00:01.089398 ip-10-0-129-2 kubenswrapper[2564]: I0723 20:00:01.089308 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-470e58-29747280-hrrmv" event={"ID":"22392289-8b04-48dd-8a18-43a79fcb5dda","Type":"ContainerStarted","Data":"5ed3df800dc61df085c9f4797c5e16bd757bb01bec2fb7c7b3a8e6fa7fb94c59"} Jul 23 20:00:01.097989 ip-10-0-129-2 kubenswrapper[2564]: I0723 20:00:01.097968 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-2dgmk"] Jul 23 20:00:01.102264 ip-10-0-129-2 kubenswrapper[2564]: I0723 20:00:01.102223 2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-fb64e5-29747280-b52r7" podStartSLOduration=1.102209507 podStartE2EDuration="1.102209507s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-23 20:00:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-23 20:00:01.100887713 +0000 UTC m=+2426.952663961" watchObservedRunningTime="2026-07-23 20:00:01.102209507 +0000 UTC m=+2426.953985732" Jul 23 20:00:01.117141 ip-10-0-129-2 kubenswrapper[2564]: I0723 20:00:01.117108 2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29747280-vw88h" podStartSLOduration=1.117095978 podStartE2EDuration="1.117095978s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-23 20:00:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-23 20:00:01.116125564 +0000 UTC m=+2426.967901813" watchObservedRunningTime="2026-07-23 20:00:01.117095978 +0000 UTC m=+2426.968872225" Jul 23 20:00:01.132641 ip-10-0-129-2 kubenswrapper[2564]: I0723 20:00:01.132605 2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-470e58-29747280-hrrmv" podStartSLOduration=1.1325898859999999 podStartE2EDuration="1.132589886s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-23 20:00:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-23 20:00:01.132174718 +0000 UTC m=+2426.983950982" watchObservedRunningTime="2026-07-23 20:00:01.132589886 +0000 UTC m=+2426.984366133" Jul 23 20:00:02.096575 ip-10-0-129-2 kubenswrapper[2564]: I0723 20:00:02.096543 2564 generic.go:356] "Generic (PLEG): container finished" podID="22392289-8b04-48dd-8a18-43a79fcb5dda" containerID="3bbc962944c2c3400b6aa87601f4859d68f3963398a77688d305289812c7d90e" exitCode=0 Jul 23 20:00:02.096984 ip-10-0-129-2 kubenswrapper[2564]: I0723 20:00:02.096620 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-470e58-29747280-hrrmv" event={"ID":"22392289-8b04-48dd-8a18-43a79fcb5dda","Type":"ContainerDied","Data":"3bbc962944c2c3400b6aa87601f4859d68f3963398a77688d305289812c7d90e"} Jul 23 20:00:02.098158 ip-10-0-129-2 kubenswrapper[2564]: I0723 20:00:02.098139 2564 generic.go:356] "Generic (PLEG): container finished" podID="de7f66c4-068a-4f4a-9f10-fde3c0bac17c" containerID="97ff3937d8a8a97b25b2e80df47f68b2e514ab7f4dcf6b67c03be6143ffaa334" exitCode=0 Jul 23 20:00:02.098278 ip-10-0-129-2 kubenswrapper[2564]: I0723 20:00:02.098199 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-fb64e5-29747280-b52r7" event={"ID":"de7f66c4-068a-4f4a-9f10-fde3c0bac17c","Type":"ContainerDied","Data":"97ff3937d8a8a97b25b2e80df47f68b2e514ab7f4dcf6b67c03be6143ffaa334"} Jul 23 20:00:02.099725 ip-10-0-129-2 kubenswrapper[2564]: I0723 20:00:02.099708 2564 generic.go:356] "Generic (PLEG): container finished" podID="2432a745-c8d5-4505-a20a-4d82fe439720" containerID="aa1ffbff30021be7c38c18df3da3a24b8376bc8c1981316b766cf9f7d6d2a915" exitCode=0 Jul 23 20:00:02.099804 ip-10-0-129-2 kubenswrapper[2564]: I0723 20:00:02.099731 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29747280-vw88h" event={"ID":"2432a745-c8d5-4505-a20a-4d82fe439720","Type":"ContainerDied","Data":"aa1ffbff30021be7c38c18df3da3a24b8376bc8c1981316b766cf9f7d6d2a915"} Jul 23 20:00:03.259786 ip-10-0-129-2 kubenswrapper[2564]: I0723 20:00:03.259760 2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-470e58-29747280-hrrmv" Jul 23 20:00:03.300030 ip-10-0-129-2 kubenswrapper[2564]: I0723 20:00:03.300006 2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29747280-vw88h" Jul 23 20:00:03.302899 ip-10-0-129-2 kubenswrapper[2564]: I0723 20:00:03.302880 2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-fb64e5-29747280-b52r7" Jul 23 20:00:04.113632 ip-10-0-129-2 kubenswrapper[2564]: I0723 20:00:04.113602 2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-470e58-29747280-hrrmv" Jul 23 20:00:04.113808 ip-10-0-129-2 kubenswrapper[2564]: I0723 20:00:04.113598 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-470e58-29747280-hrrmv" event={"ID":"22392289-8b04-48dd-8a18-43a79fcb5dda","Type":"ContainerDied","Data":"5ed3df800dc61df085c9f4797c5e16bd757bb01bec2fb7c7b3a8e6fa7fb94c59"} Jul 23 20:00:04.113808 ip-10-0-129-2 kubenswrapper[2564]: I0723 20:00:04.113730 2564 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="5ed3df800dc61df085c9f4797c5e16bd757bb01bec2fb7c7b3a8e6fa7fb94c59" Jul 23 20:00:04.114971 ip-10-0-129-2 kubenswrapper[2564]: I0723 20:00:04.114952 2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-fb64e5-29747280-b52r7" Jul 23 20:00:04.115082 ip-10-0-129-2 kubenswrapper[2564]: I0723 20:00:04.114975 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-fb64e5-29747280-b52r7" event={"ID":"de7f66c4-068a-4f4a-9f10-fde3c0bac17c","Type":"ContainerDied","Data":"e38c0f9de619f4822ba661e3c64b443c81eae761f7e54fbc327d661d3fec409f"} Jul 23 20:00:04.115082 ip-10-0-129-2 kubenswrapper[2564]: I0723 20:00:04.115015 2564 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="e38c0f9de619f4822ba661e3c64b443c81eae761f7e54fbc327d661d3fec409f" Jul 23 20:00:04.116385 ip-10-0-129-2 kubenswrapper[2564]: I0723 20:00:04.116368 2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29747280-vw88h" Jul 23 20:00:04.116490 ip-10-0-129-2 kubenswrapper[2564]: I0723 20:00:04.116389 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29747280-vw88h" event={"ID":"2432a745-c8d5-4505-a20a-4d82fe439720","Type":"ContainerDied","Data":"c9e1f41e048b9a6951815b816d0c57a94daacf48d58fb1c3da289a78d7dca0d9"} Jul 23 20:00:04.116490 ip-10-0-129-2 kubenswrapper[2564]: I0723 20:00:04.116427 2564 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="c9e1f41e048b9a6951815b816d0c57a94daacf48d58fb1c3da289a78d7dca0d9" Jul 23 20:00:16.295401 ip-10-0-129-2 kubenswrapper[2564]: I0723 20:00:16.295371 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-2dgmk"] Jul 23 20:00:30.110303 ip-10-0-129-2 kubenswrapper[2564]: I0723 20:00:30.110272 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-2dgmk"] Jul 23 20:00:35.711545 ip-10-0-129-2 kubenswrapper[2564]: I0723 20:00:35.711513 2564 scope.go:117] "RemoveContainer" containerID="84b1562677062dbd234fd52418433cf3ea380756805a7dc680e4094e1b332e93" Jul 23 20:00:48.102358 ip-10-0-129-2 kubenswrapper[2564]: I0723 20:00:48.102312 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-2dgmk"] Jul 23 20:01:02.025230 ip-10-0-129-2 kubenswrapper[2564]: I0723 20:01:02.025196 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-2dgmk"] Jul 23 20:01:21.188562 ip-10-0-129-2 kubenswrapper[2564]: I0723 20:01:21.188526 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-2dgmk"] Jul 23 20:01:40.398017 ip-10-0-129-2 kubenswrapper[2564]: I0723 20:01:40.397985 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-2dgmk"] Jul 23 20:01:59.693832 ip-10-0-129-2 kubenswrapper[2564]: I0723 20:01:59.693791 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-2dgmk"] Jul 23 20:02:18.592169 ip-10-0-129-2 kubenswrapper[2564]: I0723 20:02:18.592131 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-2dgmk"] Jul 23 20:02:39.692931 ip-10-0-129-2 kubenswrapper[2564]: I0723 20:02:39.692896 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-2dgmk"] Jul 23 20:02:53.696403 ip-10-0-129-2 kubenswrapper[2564]: I0723 20:02:53.696365 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-2dgmk"] Jul 23 20:03:12.000107 ip-10-0-129-2 kubenswrapper[2564]: I0723 20:03:12.000022 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-2dgmk"] Jul 23 20:03:29.196340 ip-10-0-129-2 kubenswrapper[2564]: I0723 20:03:29.196287 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-2dgmk"] Jul 23 20:03:35.749183 ip-10-0-129-2 kubenswrapper[2564]: I0723 20:03:35.749140 2564 scope.go:117] "RemoveContainer" containerID="2f30e39042b92bb964c88cf36e52b8165f924be96eb8b800fb0bf7afc9e8bbaa" Jul 23 20:03:57.692127 ip-10-0-129-2 kubenswrapper[2564]: I0723 20:03:57.692085 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-2dgmk"] Jul 23 20:04:14.703441 ip-10-0-129-2 kubenswrapper[2564]: I0723 20:04:14.703398 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-2dgmk"] Jul 23 20:04:29.792453 ip-10-0-129-2 kubenswrapper[2564]: I0723 20:04:29.792348 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-2dgmk"] Jul 23 20:04:35.641486 ip-10-0-129-2 kubenswrapper[2564]: I0723 20:04:35.641458 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-565b7b569d-z5wf4_ded5b3af-c74c-4d5a-aa77-62fe72d1dd29/console-operator/1.log" Jul 23 20:04:35.649357 ip-10-0-129-2 kubenswrapper[2564]: I0723 20:04:35.649315 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-8dtlh_195ecf89-13ae-4311-bc72-fcdf78a5c4d6/ovn-acl-logging/0.log" Jul 23 20:04:35.652684 ip-10-0-129-2 kubenswrapper[2564]: I0723 20:04:35.652662 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-565b7b569d-z5wf4_ded5b3af-c74c-4d5a-aa77-62fe72d1dd29/console-operator/1.log" Jul 23 20:04:35.660067 ip-10-0-129-2 kubenswrapper[2564]: I0723 20:04:35.660045 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-8dtlh_195ecf89-13ae-4311-bc72-fcdf78a5c4d6/ovn-acl-logging/0.log" Jul 23 20:04:35.774578 ip-10-0-129-2 kubenswrapper[2564]: I0723 20:04:35.774543 2564 scope.go:117] "RemoveContainer" containerID="1735a2910690c68a61b9cd732b3790df7a95f3a813a304b16468094623b00e59" Jul 23 20:04:35.783923 ip-10-0-129-2 kubenswrapper[2564]: I0723 20:04:35.783901 2564 scope.go:117] "RemoveContainer" containerID="b243218bd90e53ba6e7e4fe757f9301ebe371135e27b734fabd9f87bd25204d7" Jul 23 20:04:38.725631 ip-10-0-129-2 kubenswrapper[2564]: I0723 20:04:38.725577 2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7456-139672225534512"] Jul 23 20:04:38.726046 ip-10-0-129-2 kubenswrapper[2564]: I0723 20:04:38.725997 2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2432a745-c8d5-4505-a20a-4d82fe439720" containerName="cleanup" Jul 23 20:04:38.726046 ip-10-0-129-2 kubenswrapper[2564]: I0723 20:04:38.726009 2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="2432a745-c8d5-4505-a20a-4d82fe439720" containerName="cleanup" Jul 23 20:04:38.726046 ip-10-0-129-2 kubenswrapper[2564]: I0723 20:04:38.726019 2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="22392289-8b04-48dd-8a18-43a79fcb5dda" containerName="cleanup" Jul 23 20:04:38.726046 ip-10-0-129-2 kubenswrapper[2564]: I0723 20:04:38.726024 2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="22392289-8b04-48dd-8a18-43a79fcb5dda" containerName="cleanup" Jul 23 20:04:38.726046 ip-10-0-129-2 kubenswrapper[2564]: I0723 20:04:38.726030 2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="de7f66c4-068a-4f4a-9f10-fde3c0bac17c" containerName="cleanup" Jul 23 20:04:38.726046 ip-10-0-129-2 kubenswrapper[2564]: I0723 20:04:38.726036 2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="de7f66c4-068a-4f4a-9f10-fde3c0bac17c" containerName="cleanup" Jul 23 20:04:38.726244 ip-10-0-129-2 kubenswrapper[2564]: I0723 20:04:38.726095 2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="2432a745-c8d5-4505-a20a-4d82fe439720" containerName="cleanup" Jul 23 20:04:38.726244 ip-10-0-129-2 kubenswrapper[2564]: I0723 20:04:38.726104 2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="de7f66c4-068a-4f4a-9f10-fde3c0bac17c" containerName="cleanup" Jul 23 20:04:38.726244 ip-10-0-129-2 kubenswrapper[2564]: I0723 20:04:38.726112 2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="22392289-8b04-48dd-8a18-43a79fcb5dda" containerName="cleanup" Jul 23 20:04:38.730725 ip-10-0-129-2 kubenswrapper[2564]: I0723 20:04:38.730703 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7456-139672225534512" Jul 23 20:04:38.733296 ip-10-0-129-2 kubenswrapper[2564]: I0723 20:04:38.733267 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"default-dockercfg-7hsc4\"" Jul 23 20:04:38.735829 ip-10-0-129-2 kubenswrapper[2564]: I0723 20:04:38.735798 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7456-139672225534512"] Jul 23 20:04:38.770911 ip-10-0-129-2 kubenswrapper[2564]: I0723 20:04:38.770866 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-n9flf\" (UniqueName: \"kubernetes.io/projected/646f3741-d8cf-49aa-b3fb-2e4165fb0b2b-kube-api-access-n9flf\") pod \"test-curl-7456-139672225534512\" (UID: \"646f3741-d8cf-49aa-b3fb-2e4165fb0b2b\") " pod="opendatahub/test-curl-7456-139672225534512" Jul 23 20:04:38.872337 ip-10-0-129-2 kubenswrapper[2564]: I0723 20:04:38.872297 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n9flf\" (UniqueName: \"kubernetes.io/projected/646f3741-d8cf-49aa-b3fb-2e4165fb0b2b-kube-api-access-n9flf\") pod \"test-curl-7456-139672225534512\" (UID: \"646f3741-d8cf-49aa-b3fb-2e4165fb0b2b\") " pod="opendatahub/test-curl-7456-139672225534512" Jul 23 20:04:38.880756 ip-10-0-129-2 kubenswrapper[2564]: I0723 20:04:38.880722 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-n9flf\" (UniqueName: \"kubernetes.io/projected/646f3741-d8cf-49aa-b3fb-2e4165fb0b2b-kube-api-access-n9flf\") pod \"test-curl-7456-139672225534512\" (UID: \"646f3741-d8cf-49aa-b3fb-2e4165fb0b2b\") " pod="opendatahub/test-curl-7456-139672225534512" Jul 23 20:04:39.043113 ip-10-0-129-2 kubenswrapper[2564]: I0723 20:04:39.043011 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7456-139672225534512" Jul 23 20:04:39.187723 ip-10-0-129-2 kubenswrapper[2564]: I0723 20:04:39.187695 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7456-139672225534512"] Jul 23 20:04:39.190779 ip-10-0-129-2 kubenswrapper[2564]: W0723 20:04:39.190752 2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod646f3741_d8cf_49aa_b3fb_2e4165fb0b2b.slice/crio-250f212c404b8c53e68208bbf0d573ae54d96e211d0c784016cf17157d08e8ac WatchSource:0}: Error finding container 250f212c404b8c53e68208bbf0d573ae54d96e211d0c784016cf17157d08e8ac: Status 404 returned error can't find the container with id 250f212c404b8c53e68208bbf0d573ae54d96e211d0c784016cf17157d08e8ac Jul 23 20:04:39.192566 ip-10-0-129-2 kubenswrapper[2564]: I0723 20:04:39.192548 2564 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Jul 23 20:04:39.697166 ip-10-0-129-2 kubenswrapper[2564]: I0723 20:04:39.697128 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7456-139672225534512" event={"ID":"646f3741-d8cf-49aa-b3fb-2e4165fb0b2b","Type":"ContainerStarted","Data":"250f212c404b8c53e68208bbf0d573ae54d96e211d0c784016cf17157d08e8ac"} Jul 23 20:04:40.704379 ip-10-0-129-2 kubenswrapper[2564]: I0723 20:04:40.704341 2564 generic.go:356] "Generic (PLEG): container finished" podID="646f3741-d8cf-49aa-b3fb-2e4165fb0b2b" containerID="f417bca3627b1c9e4b80c814bc5e722acce4a2c4867b74031b85d7ce2a9e8663" exitCode=0 Jul 23 20:04:40.704813 ip-10-0-129-2 kubenswrapper[2564]: I0723 20:04:40.704420 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7456-139672225534512" event={"ID":"646f3741-d8cf-49aa-b3fb-2e4165fb0b2b","Type":"ContainerDied","Data":"f417bca3627b1c9e4b80c814bc5e722acce4a2c4867b74031b85d7ce2a9e8663"} Jul 23 20:04:41.854496 ip-10-0-129-2 kubenswrapper[2564]: I0723 20:04:41.854470 2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7456-139672225534512" Jul 23 20:04:41.891314 ip-10-0-129-2 kubenswrapper[2564]: I0723 20:04:41.891285 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_test-curl-7456-139672225534512_646f3741-d8cf-49aa-b3fb-2e4165fb0b2b/test-curl-7456-139672225534512/0.log" Jul 23 20:04:41.903637 ip-10-0-129-2 kubenswrapper[2564]: I0723 20:04:41.903609 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-n9flf\" (UniqueName: \"kubernetes.io/projected/646f3741-d8cf-49aa-b3fb-2e4165fb0b2b-kube-api-access-n9flf\") pod \"646f3741-d8cf-49aa-b3fb-2e4165fb0b2b\" (UID: \"646f3741-d8cf-49aa-b3fb-2e4165fb0b2b\") " Jul 23 20:04:41.906027 ip-10-0-129-2 kubenswrapper[2564]: I0723 20:04:41.905994 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/646f3741-d8cf-49aa-b3fb-2e4165fb0b2b-kube-api-access-n9flf" (OuterVolumeSpecName: "kube-api-access-n9flf") pod "646f3741-d8cf-49aa-b3fb-2e4165fb0b2b" (UID: "646f3741-d8cf-49aa-b3fb-2e4165fb0b2b"). InnerVolumeSpecName "kube-api-access-n9flf". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 23 20:04:41.914553 ip-10-0-129-2 kubenswrapper[2564]: I0723 20:04:41.914513 2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7456-139672225534512"] Jul 23 20:04:41.918058 ip-10-0-129-2 kubenswrapper[2564]: I0723 20:04:41.918034 2564 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7456-139672225534512"] Jul 23 20:04:42.005177 ip-10-0-129-2 kubenswrapper[2564]: I0723 20:04:42.005085 2564 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-n9flf\" (UniqueName: \"kubernetes.io/projected/646f3741-d8cf-49aa-b3fb-2e4165fb0b2b-kube-api-access-n9flf\") on node \"ip-10-0-129-2.ec2.internal\" DevicePath \"\"" Jul 23 20:04:42.035343 ip-10-0-129-2 kubenswrapper[2564]: I0723 20:04:42.035288 2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7456-139672225673520"] Jul 23 20:04:42.035726 ip-10-0-129-2 kubenswrapper[2564]: I0723 20:04:42.035712 2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="646f3741-d8cf-49aa-b3fb-2e4165fb0b2b" containerName="test-curl-7456-139672225534512" Jul 23 20:04:42.035772 ip-10-0-129-2 kubenswrapper[2564]: I0723 20:04:42.035727 2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="646f3741-d8cf-49aa-b3fb-2e4165fb0b2b" containerName="test-curl-7456-139672225534512" Jul 23 20:04:42.035809 ip-10-0-129-2 kubenswrapper[2564]: I0723 20:04:42.035794 2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="646f3741-d8cf-49aa-b3fb-2e4165fb0b2b" containerName="test-curl-7456-139672225534512" Jul 23 20:04:42.038876 ip-10-0-129-2 kubenswrapper[2564]: I0723 20:04:42.038858 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7456-139672225673520" Jul 23 20:04:42.043763 ip-10-0-129-2 kubenswrapper[2564]: I0723 20:04:42.043732 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7456-139672225673520"] Jul 23 20:04:42.105894 ip-10-0-129-2 kubenswrapper[2564]: I0723 20:04:42.105851 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tjmm7\" (UniqueName: \"kubernetes.io/projected/fd1d9bde-c710-4aef-a6b0-60afb50b1243-kube-api-access-tjmm7\") pod \"test-curl-7456-139672225673520\" (UID: \"fd1d9bde-c710-4aef-a6b0-60afb50b1243\") " pod="opendatahub/test-curl-7456-139672225673520" Jul 23 20:04:42.207559 ip-10-0-129-2 kubenswrapper[2564]: I0723 20:04:42.207520 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tjmm7\" (UniqueName: \"kubernetes.io/projected/fd1d9bde-c710-4aef-a6b0-60afb50b1243-kube-api-access-tjmm7\") pod \"test-curl-7456-139672225673520\" (UID: \"fd1d9bde-c710-4aef-a6b0-60afb50b1243\") " pod="opendatahub/test-curl-7456-139672225673520" Jul 23 20:04:42.216255 ip-10-0-129-2 kubenswrapper[2564]: I0723 20:04:42.216215 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-tjmm7\" (UniqueName: \"kubernetes.io/projected/fd1d9bde-c710-4aef-a6b0-60afb50b1243-kube-api-access-tjmm7\") pod \"test-curl-7456-139672225673520\" (UID: \"fd1d9bde-c710-4aef-a6b0-60afb50b1243\") " pod="opendatahub/test-curl-7456-139672225673520" Jul 23 20:04:42.351543 ip-10-0-129-2 kubenswrapper[2564]: I0723 20:04:42.351446 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7456-139672225673520" Jul 23 20:04:42.498041 ip-10-0-129-2 kubenswrapper[2564]: I0723 20:04:42.498013 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7456-139672225673520"] Jul 23 20:04:42.500182 ip-10-0-129-2 kubenswrapper[2564]: W0723 20:04:42.500155 2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podfd1d9bde_c710_4aef_a6b0_60afb50b1243.slice/crio-c1e47808b0cc2a11e41c5765562a3974d8c384b5cd96771aac8ff58b57f93f13 WatchSource:0}: Error finding container c1e47808b0cc2a11e41c5765562a3974d8c384b5cd96771aac8ff58b57f93f13: Status 404 returned error can't find the container with id c1e47808b0cc2a11e41c5765562a3974d8c384b5cd96771aac8ff58b57f93f13 Jul 23 20:04:42.658810 ip-10-0-129-2 kubenswrapper[2564]: I0723 20:04:42.658718 2564 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="646f3741-d8cf-49aa-b3fb-2e4165fb0b2b" path="/var/lib/kubelet/pods/646f3741-d8cf-49aa-b3fb-2e4165fb0b2b/volumes" Jul 23 20:04:42.720434 ip-10-0-129-2 kubenswrapper[2564]: I0723 20:04:42.720401 2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7456-139672225534512" Jul 23 20:04:42.720640 ip-10-0-129-2 kubenswrapper[2564]: I0723 20:04:42.720471 2564 scope.go:117] "RemoveContainer" containerID="f417bca3627b1c9e4b80c814bc5e722acce4a2c4867b74031b85d7ce2a9e8663" Jul 23 20:04:42.721887 ip-10-0-129-2 kubenswrapper[2564]: I0723 20:04:42.721773 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7456-139672225673520" event={"ID":"fd1d9bde-c710-4aef-a6b0-60afb50b1243","Type":"ContainerStarted","Data":"c1e47808b0cc2a11e41c5765562a3974d8c384b5cd96771aac8ff58b57f93f13"} Jul 23 20:04:43.730867 ip-10-0-129-2 kubenswrapper[2564]: I0723 20:04:43.730826 2564 generic.go:356] "Generic (PLEG): container finished" podID="fd1d9bde-c710-4aef-a6b0-60afb50b1243" containerID="d85c6bcd5e0ad8a8e86f1e5ec3f2e0ce9dc678acdc9651783118b4a616270f0a" exitCode=0 Jul 23 20:04:43.731351 ip-10-0-129-2 kubenswrapper[2564]: I0723 20:04:43.730914 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7456-139672225673520" event={"ID":"fd1d9bde-c710-4aef-a6b0-60afb50b1243","Type":"ContainerDied","Data":"d85c6bcd5e0ad8a8e86f1e5ec3f2e0ce9dc678acdc9651783118b4a616270f0a"} Jul 23 20:04:43.794535 ip-10-0-129-2 kubenswrapper[2564]: I0723 20:04:43.794490 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-2dgmk"] Jul 23 20:04:44.885890 ip-10-0-129-2 kubenswrapper[2564]: I0723 20:04:44.885860 2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7456-139672225673520" Jul 23 20:04:44.905008 ip-10-0-129-2 kubenswrapper[2564]: I0723 20:04:44.904978 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_test-curl-7456-139672225673520_fd1d9bde-c710-4aef-a6b0-60afb50b1243/test-curl-7456-139672225673520/0.log" Jul 23 20:04:44.929438 ip-10-0-129-2 kubenswrapper[2564]: I0723 20:04:44.929386 2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7456-139672225673520"] Jul 23 20:04:44.933763 ip-10-0-129-2 kubenswrapper[2564]: I0723 20:04:44.933733 2564 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7456-139672225673520"] Jul 23 20:04:44.935988 ip-10-0-129-2 kubenswrapper[2564]: I0723 20:04:44.935967 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-tjmm7\" (UniqueName: \"kubernetes.io/projected/fd1d9bde-c710-4aef-a6b0-60afb50b1243-kube-api-access-tjmm7\") pod \"fd1d9bde-c710-4aef-a6b0-60afb50b1243\" (UID: \"fd1d9bde-c710-4aef-a6b0-60afb50b1243\") " Jul 23 20:04:44.938164 ip-10-0-129-2 kubenswrapper[2564]: I0723 20:04:44.938140 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fd1d9bde-c710-4aef-a6b0-60afb50b1243-kube-api-access-tjmm7" (OuterVolumeSpecName: "kube-api-access-tjmm7") pod "fd1d9bde-c710-4aef-a6b0-60afb50b1243" (UID: "fd1d9bde-c710-4aef-a6b0-60afb50b1243"). InnerVolumeSpecName "kube-api-access-tjmm7". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 23 20:04:45.037064 ip-10-0-129-2 kubenswrapper[2564]: I0723 20:04:45.036956 2564 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tjmm7\" (UniqueName: \"kubernetes.io/projected/fd1d9bde-c710-4aef-a6b0-60afb50b1243-kube-api-access-tjmm7\") on node \"ip-10-0-129-2.ec2.internal\" DevicePath \"\"" Jul 23 20:04:45.092401 ip-10-0-129-2 kubenswrapper[2564]: I0723 20:04:45.092359 2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7456-139672225592784"] Jul 23 20:04:45.092911 ip-10-0-129-2 kubenswrapper[2564]: I0723 20:04:45.092895 2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="fd1d9bde-c710-4aef-a6b0-60afb50b1243" containerName="test-curl-7456-139672225673520" Jul 23 20:04:45.092966 ip-10-0-129-2 kubenswrapper[2564]: I0723 20:04:45.092914 2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="fd1d9bde-c710-4aef-a6b0-60afb50b1243" containerName="test-curl-7456-139672225673520" Jul 23 20:04:45.093028 ip-10-0-129-2 kubenswrapper[2564]: I0723 20:04:45.093017 2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="fd1d9bde-c710-4aef-a6b0-60afb50b1243" containerName="test-curl-7456-139672225673520" Jul 23 20:04:45.096524 ip-10-0-129-2 kubenswrapper[2564]: I0723 20:04:45.096506 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7456-139672225592784" Jul 23 20:04:45.101597 ip-10-0-129-2 kubenswrapper[2564]: I0723 20:04:45.101556 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7456-139672225592784"] Jul 23 20:04:45.239297 ip-10-0-129-2 kubenswrapper[2564]: I0723 20:04:45.239257 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bmk5l\" (UniqueName: \"kubernetes.io/projected/6afd1294-e5ea-4832-a2f8-9d5d71059a9a-kube-api-access-bmk5l\") pod \"test-curl-7456-139672225592784\" (UID: \"6afd1294-e5ea-4832-a2f8-9d5d71059a9a\") " pod="opendatahub/test-curl-7456-139672225592784" Jul 23 20:04:45.340828 ip-10-0-129-2 kubenswrapper[2564]: I0723 20:04:45.340710 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bmk5l\" (UniqueName: \"kubernetes.io/projected/6afd1294-e5ea-4832-a2f8-9d5d71059a9a-kube-api-access-bmk5l\") pod \"test-curl-7456-139672225592784\" (UID: \"6afd1294-e5ea-4832-a2f8-9d5d71059a9a\") " pod="opendatahub/test-curl-7456-139672225592784" Jul 23 20:04:45.369936 ip-10-0-129-2 kubenswrapper[2564]: I0723 20:04:45.369908 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-bmk5l\" (UniqueName: \"kubernetes.io/projected/6afd1294-e5ea-4832-a2f8-9d5d71059a9a-kube-api-access-bmk5l\") pod \"test-curl-7456-139672225592784\" (UID: \"6afd1294-e5ea-4832-a2f8-9d5d71059a9a\") " pod="opendatahub/test-curl-7456-139672225592784" Jul 23 20:04:45.410229 ip-10-0-129-2 kubenswrapper[2564]: I0723 20:04:45.410191 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7456-139672225592784" Jul 23 20:04:45.560462 ip-10-0-129-2 kubenswrapper[2564]: I0723 20:04:45.560433 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7456-139672225592784"] Jul 23 20:04:45.562488 ip-10-0-129-2 kubenswrapper[2564]: W0723 20:04:45.562454 2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod6afd1294_e5ea_4832_a2f8_9d5d71059a9a.slice/crio-bf37e90d4719858d92fd3d2618bc04683fb26364be7070b1e789ac084f64e179 WatchSource:0}: Error finding container bf37e90d4719858d92fd3d2618bc04683fb26364be7070b1e789ac084f64e179: Status 404 returned error can't find the container with id bf37e90d4719858d92fd3d2618bc04683fb26364be7070b1e789ac084f64e179 Jul 23 20:04:45.744427 ip-10-0-129-2 kubenswrapper[2564]: I0723 20:04:45.744390 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7456-139672225592784" event={"ID":"6afd1294-e5ea-4832-a2f8-9d5d71059a9a","Type":"ContainerStarted","Data":"bf37e90d4719858d92fd3d2618bc04683fb26364be7070b1e789ac084f64e179"} Jul 23 20:04:45.745683 ip-10-0-129-2 kubenswrapper[2564]: I0723 20:04:45.745662 2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7456-139672225673520" Jul 23 20:04:45.745804 ip-10-0-129-2 kubenswrapper[2564]: I0723 20:04:45.745683 2564 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="c1e47808b0cc2a11e41c5765562a3974d8c384b5cd96771aac8ff58b57f93f13" Jul 23 20:04:45.755851 ip-10-0-129-2 kubenswrapper[2564]: E0723 20:04:45.755823 2564 status_manager.go:1018] "Failed to get status for pod" err="pods \"test-curl-7456-139672225673520\" is forbidden: User \"system:node:ip-10-0-129-2.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-129-2.ec2.internal' and this object" podUID="fd1d9bde-c710-4aef-a6b0-60afb50b1243" pod="opendatahub/test-curl-7456-139672225673520" Jul 23 20:04:46.658391 ip-10-0-129-2 kubenswrapper[2564]: I0723 20:04:46.658353 2564 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="fd1d9bde-c710-4aef-a6b0-60afb50b1243" path="/var/lib/kubelet/pods/fd1d9bde-c710-4aef-a6b0-60afb50b1243/volumes" Jul 23 20:04:46.752948 ip-10-0-129-2 kubenswrapper[2564]: I0723 20:04:46.752915 2564 generic.go:356] "Generic (PLEG): container finished" podID="6afd1294-e5ea-4832-a2f8-9d5d71059a9a" containerID="4828cf00d34e0fe545e37ccd1b31d3134fb79052c798c806c2f9f180944b2a01" exitCode=0 Jul 23 20:04:46.753133 ip-10-0-129-2 kubenswrapper[2564]: I0723 20:04:46.753004 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7456-139672225592784" event={"ID":"6afd1294-e5ea-4832-a2f8-9d5d71059a9a","Type":"ContainerDied","Data":"4828cf00d34e0fe545e37ccd1b31d3134fb79052c798c806c2f9f180944b2a01"} Jul 23 20:04:47.906407 ip-10-0-129-2 kubenswrapper[2564]: I0723 20:04:47.906375 2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7456-139672225592784" Jul 23 20:04:47.928076 ip-10-0-129-2 kubenswrapper[2564]: I0723 20:04:47.928039 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_test-curl-7456-139672225592784_6afd1294-e5ea-4832-a2f8-9d5d71059a9a/test-curl-7456-139672225592784/0.log" Jul 23 20:04:47.952721 ip-10-0-129-2 kubenswrapper[2564]: I0723 20:04:47.952675 2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7456-139672225592784"] Jul 23 20:04:47.956455 ip-10-0-129-2 kubenswrapper[2564]: I0723 20:04:47.956421 2564 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7456-139672225592784"] Jul 23 20:04:47.967120 ip-10-0-129-2 kubenswrapper[2564]: I0723 20:04:47.967090 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-bmk5l\" (UniqueName: \"kubernetes.io/projected/6afd1294-e5ea-4832-a2f8-9d5d71059a9a-kube-api-access-bmk5l\") pod \"6afd1294-e5ea-4832-a2f8-9d5d71059a9a\" (UID: \"6afd1294-e5ea-4832-a2f8-9d5d71059a9a\") " Jul 23 20:04:47.969410 ip-10-0-129-2 kubenswrapper[2564]: I0723 20:04:47.969384 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6afd1294-e5ea-4832-a2f8-9d5d71059a9a-kube-api-access-bmk5l" (OuterVolumeSpecName: "kube-api-access-bmk5l") pod "6afd1294-e5ea-4832-a2f8-9d5d71059a9a" (UID: "6afd1294-e5ea-4832-a2f8-9d5d71059a9a"). InnerVolumeSpecName "kube-api-access-bmk5l". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 23 20:04:48.067930 ip-10-0-129-2 kubenswrapper[2564]: I0723 20:04:48.067827 2564 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bmk5l\" (UniqueName: \"kubernetes.io/projected/6afd1294-e5ea-4832-a2f8-9d5d71059a9a-kube-api-access-bmk5l\") on node \"ip-10-0-129-2.ec2.internal\" DevicePath \"\"" Jul 23 20:04:48.071512 ip-10-0-129-2 kubenswrapper[2564]: I0723 20:04:48.071479 2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7456-139672225592784"] Jul 23 20:04:48.071902 ip-10-0-129-2 kubenswrapper[2564]: I0723 20:04:48.071889 2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6afd1294-e5ea-4832-a2f8-9d5d71059a9a" containerName="test-curl-7456-139672225592784" Jul 23 20:04:48.071953 ip-10-0-129-2 kubenswrapper[2564]: I0723 20:04:48.071904 2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="6afd1294-e5ea-4832-a2f8-9d5d71059a9a" containerName="test-curl-7456-139672225592784" Jul 23 20:04:48.071993 ip-10-0-129-2 kubenswrapper[2564]: I0723 20:04:48.071969 2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="6afd1294-e5ea-4832-a2f8-9d5d71059a9a" containerName="test-curl-7456-139672225592784" Jul 23 20:04:48.075097 ip-10-0-129-2 kubenswrapper[2564]: I0723 20:04:48.075079 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7456-139672225592784" Jul 23 20:04:48.082238 ip-10-0-129-2 kubenswrapper[2564]: I0723 20:04:48.082204 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7456-139672225592784"] Jul 23 20:04:48.168529 ip-10-0-129-2 kubenswrapper[2564]: I0723 20:04:48.168489 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kplp2\" (UniqueName: \"kubernetes.io/projected/6a31cc1c-d524-4303-9e60-029f032893ad-kube-api-access-kplp2\") pod \"test-curl-7456-139672225592784\" (UID: \"6a31cc1c-d524-4303-9e60-029f032893ad\") " pod="opendatahub/test-curl-7456-139672225592784" Jul 23 20:04:48.269268 ip-10-0-129-2 kubenswrapper[2564]: I0723 20:04:48.269204 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kplp2\" (UniqueName: \"kubernetes.io/projected/6a31cc1c-d524-4303-9e60-029f032893ad-kube-api-access-kplp2\") pod \"test-curl-7456-139672225592784\" (UID: \"6a31cc1c-d524-4303-9e60-029f032893ad\") " pod="opendatahub/test-curl-7456-139672225592784" Jul 23 20:04:48.278471 ip-10-0-129-2 kubenswrapper[2564]: I0723 20:04:48.278445 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-kplp2\" (UniqueName: \"kubernetes.io/projected/6a31cc1c-d524-4303-9e60-029f032893ad-kube-api-access-kplp2\") pod \"test-curl-7456-139672225592784\" (UID: \"6a31cc1c-d524-4303-9e60-029f032893ad\") " pod="opendatahub/test-curl-7456-139672225592784" Jul 23 20:04:48.385924 ip-10-0-129-2 kubenswrapper[2564]: I0723 20:04:48.385788 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7456-139672225592784" Jul 23 20:04:48.530664 ip-10-0-129-2 kubenswrapper[2564]: I0723 20:04:48.530637 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7456-139672225592784"] Jul 23 20:04:48.532536 ip-10-0-129-2 kubenswrapper[2564]: W0723 20:04:48.532508 2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod6a31cc1c_d524_4303_9e60_029f032893ad.slice/crio-78482acafeeb3baafd133aba0fa0249cc06622e9c6add45520507a7e7251dd75 WatchSource:0}: Error finding container 78482acafeeb3baafd133aba0fa0249cc06622e9c6add45520507a7e7251dd75: Status 404 returned error can't find the container with id 78482acafeeb3baafd133aba0fa0249cc06622e9c6add45520507a7e7251dd75 Jul 23 20:04:48.659424 ip-10-0-129-2 kubenswrapper[2564]: I0723 20:04:48.659310 2564 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="6afd1294-e5ea-4832-a2f8-9d5d71059a9a" path="/var/lib/kubelet/pods/6afd1294-e5ea-4832-a2f8-9d5d71059a9a/volumes" Jul 23 20:04:48.765912 ip-10-0-129-2 kubenswrapper[2564]: I0723 20:04:48.765884 2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7456-139672225592784" Jul 23 20:04:48.765912 ip-10-0-129-2 kubenswrapper[2564]: I0723 20:04:48.765895 2564 scope.go:117] "RemoveContainer" containerID="4828cf00d34e0fe545e37ccd1b31d3134fb79052c798c806c2f9f180944b2a01" Jul 23 20:04:48.767382 ip-10-0-129-2 kubenswrapper[2564]: I0723 20:04:48.767358 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7456-139672225592784" event={"ID":"6a31cc1c-d524-4303-9e60-029f032893ad","Type":"ContainerStarted","Data":"78482acafeeb3baafd133aba0fa0249cc06622e9c6add45520507a7e7251dd75"} Jul 23 20:04:49.774907 ip-10-0-129-2 kubenswrapper[2564]: I0723 20:04:49.774871 2564 generic.go:356] "Generic (PLEG): container finished" podID="6a31cc1c-d524-4303-9e60-029f032893ad" containerID="c8cad1fca4901ffe50a4e0237136d2ddf34212d6d289a30af8cb5752a1e5515b" exitCode=0 Jul 23 20:04:49.775586 ip-10-0-129-2 kubenswrapper[2564]: I0723 20:04:49.774960 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7456-139672225592784" event={"ID":"6a31cc1c-d524-4303-9e60-029f032893ad","Type":"ContainerDied","Data":"c8cad1fca4901ffe50a4e0237136d2ddf34212d6d289a30af8cb5752a1e5515b"} Jul 23 20:04:50.940246 ip-10-0-129-2 kubenswrapper[2564]: I0723 20:04:50.940218 2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7456-139672225592784" Jul 23 20:04:50.962542 ip-10-0-129-2 kubenswrapper[2564]: I0723 20:04:50.962496 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_test-curl-7456-139672225592784_6a31cc1c-d524-4303-9e60-029f032893ad/test-curl-7456-139672225592784/0.log" Jul 23 20:04:50.985057 ip-10-0-129-2 kubenswrapper[2564]: I0723 20:04:50.985019 2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7456-139672225592784"] Jul 23 20:04:50.989354 ip-10-0-129-2 kubenswrapper[2564]: I0723 20:04:50.989296 2564 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7456-139672225592784"] Jul 23 20:04:51.095352 ip-10-0-129-2 kubenswrapper[2564]: I0723 20:04:51.095226 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-kplp2\" (UniqueName: \"kubernetes.io/projected/6a31cc1c-d524-4303-9e60-029f032893ad-kube-api-access-kplp2\") pod \"6a31cc1c-d524-4303-9e60-029f032893ad\" (UID: \"6a31cc1c-d524-4303-9e60-029f032893ad\") " Jul 23 20:04:51.097704 ip-10-0-129-2 kubenswrapper[2564]: I0723 20:04:51.097664 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6a31cc1c-d524-4303-9e60-029f032893ad-kube-api-access-kplp2" (OuterVolumeSpecName: "kube-api-access-kplp2") pod "6a31cc1c-d524-4303-9e60-029f032893ad" (UID: "6a31cc1c-d524-4303-9e60-029f032893ad"). InnerVolumeSpecName "kube-api-access-kplp2". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 23 20:04:51.100588 ip-10-0-129-2 kubenswrapper[2564]: I0723 20:04:51.100557 2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7456-139672225593792"] Jul 23 20:04:51.100998 ip-10-0-129-2 kubenswrapper[2564]: I0723 20:04:51.100984 2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6a31cc1c-d524-4303-9e60-029f032893ad" containerName="test-curl-7456-139672225592784" Jul 23 20:04:51.101042 ip-10-0-129-2 kubenswrapper[2564]: I0723 20:04:51.101000 2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="6a31cc1c-d524-4303-9e60-029f032893ad" containerName="test-curl-7456-139672225592784" Jul 23 20:04:51.101084 ip-10-0-129-2 kubenswrapper[2564]: I0723 20:04:51.101074 2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="6a31cc1c-d524-4303-9e60-029f032893ad" containerName="test-curl-7456-139672225592784" Jul 23 20:04:51.104279 ip-10-0-129-2 kubenswrapper[2564]: I0723 20:04:51.104261 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7456-139672225593792" Jul 23 20:04:51.110124 ip-10-0-129-2 kubenswrapper[2564]: I0723 20:04:51.110092 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7456-139672225593792"] Jul 23 20:04:51.196800 ip-10-0-129-2 kubenswrapper[2564]: I0723 20:04:51.196754 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bjjt8\" (UniqueName: \"kubernetes.io/projected/978fa211-d9e3-4f24-b90d-6618522259a0-kube-api-access-bjjt8\") pod \"test-curl-7456-139672225593792\" (UID: \"978fa211-d9e3-4f24-b90d-6618522259a0\") " pod="opendatahub/test-curl-7456-139672225593792" Jul 23 20:04:51.196993 ip-10-0-129-2 kubenswrapper[2564]: I0723 20:04:51.196863 2564 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kplp2\" (UniqueName: \"kubernetes.io/projected/6a31cc1c-d524-4303-9e60-029f032893ad-kube-api-access-kplp2\") on node \"ip-10-0-129-2.ec2.internal\" DevicePath \"\"" Jul 23 20:04:51.297745 ip-10-0-129-2 kubenswrapper[2564]: I0723 20:04:51.297699 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bjjt8\" (UniqueName: \"kubernetes.io/projected/978fa211-d9e3-4f24-b90d-6618522259a0-kube-api-access-bjjt8\") pod \"test-curl-7456-139672225593792\" (UID: \"978fa211-d9e3-4f24-b90d-6618522259a0\") " pod="opendatahub/test-curl-7456-139672225593792" Jul 23 20:04:51.306267 ip-10-0-129-2 kubenswrapper[2564]: I0723 20:04:51.306235 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-bjjt8\" (UniqueName: \"kubernetes.io/projected/978fa211-d9e3-4f24-b90d-6618522259a0-kube-api-access-bjjt8\") pod \"test-curl-7456-139672225593792\" (UID: \"978fa211-d9e3-4f24-b90d-6618522259a0\") " pod="opendatahub/test-curl-7456-139672225593792" Jul 23 20:04:51.416307 ip-10-0-129-2 kubenswrapper[2564]: I0723 20:04:51.416270 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7456-139672225593792" Jul 23 20:04:51.566731 ip-10-0-129-2 kubenswrapper[2564]: I0723 20:04:51.566702 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7456-139672225593792"] Jul 23 20:04:51.568204 ip-10-0-129-2 kubenswrapper[2564]: W0723 20:04:51.568172 2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod978fa211_d9e3_4f24_b90d_6618522259a0.slice/crio-1e64ba9bfb8d21694bedbb19396e50265bde1605fdb6f8df89f49a48198e230d WatchSource:0}: Error finding container 1e64ba9bfb8d21694bedbb19396e50265bde1605fdb6f8df89f49a48198e230d: Status 404 returned error can't find the container with id 1e64ba9bfb8d21694bedbb19396e50265bde1605fdb6f8df89f49a48198e230d Jul 23 20:04:51.790998 ip-10-0-129-2 kubenswrapper[2564]: I0723 20:04:51.790909 2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7456-139672225592784" Jul 23 20:04:51.790998 ip-10-0-129-2 kubenswrapper[2564]: I0723 20:04:51.790955 2564 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="78482acafeeb3baafd133aba0fa0249cc06622e9c6add45520507a7e7251dd75" Jul 23 20:04:51.792387 ip-10-0-129-2 kubenswrapper[2564]: I0723 20:04:51.792359 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7456-139672225593792" event={"ID":"978fa211-d9e3-4f24-b90d-6618522259a0","Type":"ContainerStarted","Data":"1e64ba9bfb8d21694bedbb19396e50265bde1605fdb6f8df89f49a48198e230d"} Jul 23 20:04:51.801437 ip-10-0-129-2 kubenswrapper[2564]: E0723 20:04:51.801402 2564 status_manager.go:1018] "Failed to get status for pod" err="pods \"test-curl-7456-139672225592784\" is forbidden: User \"system:node:ip-10-0-129-2.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-129-2.ec2.internal' and this object" podUID="6a31cc1c-d524-4303-9e60-029f032893ad" pod="opendatahub/test-curl-7456-139672225592784" Jul 23 20:04:52.659011 ip-10-0-129-2 kubenswrapper[2564]: I0723 20:04:52.658973 2564 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="6a31cc1c-d524-4303-9e60-029f032893ad" path="/var/lib/kubelet/pods/6a31cc1c-d524-4303-9e60-029f032893ad/volumes" Jul 23 20:04:52.800547 ip-10-0-129-2 kubenswrapper[2564]: I0723 20:04:52.800517 2564 generic.go:356] "Generic (PLEG): container finished" podID="978fa211-d9e3-4f24-b90d-6618522259a0" containerID="59b9574934029df079cf56eda61cc10906de6540767dc1290ad3d549e0b73321" exitCode=0 Jul 23 20:04:52.800717 ip-10-0-129-2 kubenswrapper[2564]: I0723 20:04:52.800563 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7456-139672225593792" event={"ID":"978fa211-d9e3-4f24-b90d-6618522259a0","Type":"ContainerDied","Data":"59b9574934029df079cf56eda61cc10906de6540767dc1290ad3d549e0b73321"} Jul 23 20:04:53.952407 ip-10-0-129-2 kubenswrapper[2564]: I0723 20:04:53.952376 2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7456-139672225593792" Jul 23 20:04:53.972958 ip-10-0-129-2 kubenswrapper[2564]: I0723 20:04:53.972930 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_test-curl-7456-139672225593792_978fa211-d9e3-4f24-b90d-6618522259a0/test-curl-7456-139672225593792/0.log" Jul 23 20:04:53.995226 ip-10-0-129-2 kubenswrapper[2564]: I0723 20:04:53.995182 2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7456-139672225593792"] Jul 23 20:04:53.997793 ip-10-0-129-2 kubenswrapper[2564]: I0723 20:04:53.997763 2564 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7456-139672225593792"] Jul 23 20:04:54.024758 ip-10-0-129-2 kubenswrapper[2564]: I0723 20:04:54.024720 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-bjjt8\" (UniqueName: \"kubernetes.io/projected/978fa211-d9e3-4f24-b90d-6618522259a0-kube-api-access-bjjt8\") pod \"978fa211-d9e3-4f24-b90d-6618522259a0\" (UID: \"978fa211-d9e3-4f24-b90d-6618522259a0\") " Jul 23 20:04:54.027065 ip-10-0-129-2 kubenswrapper[2564]: I0723 20:04:54.027032 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/978fa211-d9e3-4f24-b90d-6618522259a0-kube-api-access-bjjt8" (OuterVolumeSpecName: "kube-api-access-bjjt8") pod "978fa211-d9e3-4f24-b90d-6618522259a0" (UID: "978fa211-d9e3-4f24-b90d-6618522259a0"). InnerVolumeSpecName "kube-api-access-bjjt8". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 23 20:04:54.126351 ip-10-0-129-2 kubenswrapper[2564]: I0723 20:04:54.126238 2564 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bjjt8\" (UniqueName: \"kubernetes.io/projected/978fa211-d9e3-4f24-b90d-6618522259a0-kube-api-access-bjjt8\") on node \"ip-10-0-129-2.ec2.internal\" DevicePath \"\"" Jul 23 20:04:54.659967 ip-10-0-129-2 kubenswrapper[2564]: I0723 20:04:54.659929 2564 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="978fa211-d9e3-4f24-b90d-6618522259a0" path="/var/lib/kubelet/pods/978fa211-d9e3-4f24-b90d-6618522259a0/volumes" Jul 23 20:04:54.814552 ip-10-0-129-2 kubenswrapper[2564]: I0723 20:04:54.814521 2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7456-139672225593792" Jul 23 20:04:54.814748 ip-10-0-129-2 kubenswrapper[2564]: I0723 20:04:54.814557 2564 scope.go:117] "RemoveContainer" containerID="59b9574934029df079cf56eda61cc10906de6540767dc1290ad3d549e0b73321" Jul 23 20:04:57.224041 ip-10-0-129-2 kubenswrapper[2564]: I0723 20:04:57.223993 2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-fb64e5-4l47h"] Jul 23 20:04:57.224646 ip-10-0-129-2 kubenswrapper[2564]: I0723 20:04:57.224624 2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="978fa211-d9e3-4f24-b90d-6618522259a0" containerName="test-curl-7456-139672225593792" Jul 23 20:04:57.224731 ip-10-0-129-2 kubenswrapper[2564]: I0723 20:04:57.224649 2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="978fa211-d9e3-4f24-b90d-6618522259a0" containerName="test-curl-7456-139672225593792" Jul 23 20:04:57.224786 ip-10-0-129-2 kubenswrapper[2564]: I0723 20:04:57.224758 2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="978fa211-d9e3-4f24-b90d-6618522259a0" containerName="test-curl-7456-139672225593792" Jul 23 20:04:57.229424 ip-10-0-129-2 kubenswrapper[2564]: I0723 20:04:57.229398 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-fb64e5-4l47h" Jul 23 20:04:57.231906 ip-10-0-129-2 kubenswrapper[2564]: I0723 20:04:57.231874 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-gcspk\"" Jul 23 20:04:57.235862 ip-10-0-129-2 kubenswrapper[2564]: I0723 20:04:57.235827 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-fb64e5-4l47h"] Jul 23 20:04:57.357216 ip-10-0-129-2 kubenswrapper[2564]: I0723 20:04:57.357176 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/cb9e26a7-a0e6-403d-a67a-4ff71960e57a-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-fb64e5-4l47h\" (UID: \"cb9e26a7-a0e6-403d-a67a-4ff71960e57a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-fb64e5-4l47h" Jul 23 20:04:57.458613 ip-10-0-129-2 kubenswrapper[2564]: I0723 20:04:57.458575 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/cb9e26a7-a0e6-403d-a67a-4ff71960e57a-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-fb64e5-4l47h\" (UID: \"cb9e26a7-a0e6-403d-a67a-4ff71960e57a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-fb64e5-4l47h" Jul 23 20:04:57.459252 ip-10-0-129-2 kubenswrapper[2564]: I0723 20:04:57.459225 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/cb9e26a7-a0e6-403d-a67a-4ff71960e57a-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-fb64e5-4l47h\" (UID: \"cb9e26a7-a0e6-403d-a67a-4ff71960e57a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-fb64e5-4l47h" Jul 23 20:04:57.542785 ip-10-0-129-2 kubenswrapper[2564]: I0723 20:04:57.542683 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-fb64e5-4l47h" Jul 23 20:04:57.689373 ip-10-0-129-2 kubenswrapper[2564]: I0723 20:04:57.689344 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-fb64e5-4l47h"] Jul 23 20:04:57.691882 ip-10-0-129-2 kubenswrapper[2564]: W0723 20:04:57.691852 2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcb9e26a7_a0e6_403d_a67a_4ff71960e57a.slice/crio-29f52edf2191178852e17cd855250c0a97ff71f810b5bb7e7254ce2bd3adb8ae WatchSource:0}: Error finding container 29f52edf2191178852e17cd855250c0a97ff71f810b5bb7e7254ce2bd3adb8ae: Status 404 returned error can't find the container with id 29f52edf2191178852e17cd855250c0a97ff71f810b5bb7e7254ce2bd3adb8ae Jul 23 20:04:57.835825 ip-10-0-129-2 kubenswrapper[2564]: I0723 20:04:57.835720 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-fb64e5-4l47h" event={"ID":"cb9e26a7-a0e6-403d-a67a-4ff71960e57a","Type":"ContainerStarted","Data":"29f52edf2191178852e17cd855250c0a97ff71f810b5bb7e7254ce2bd3adb8ae"} Jul 23 20:04:58.843433 ip-10-0-129-2 kubenswrapper[2564]: I0723 20:04:58.843396 2564 generic.go:356] "Generic (PLEG): container finished" podID="cb9e26a7-a0e6-403d-a67a-4ff71960e57a" containerID="f2b78d992596e8072eaeac818ff3e7e16ab0c26a42b0ef5c9ec57e4abcddffd0" exitCode=0 Jul 23 20:04:58.843814 ip-10-0-129-2 kubenswrapper[2564]: I0723 20:04:58.843482 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-fb64e5-4l47h" event={"ID":"cb9e26a7-a0e6-403d-a67a-4ff71960e57a","Type":"ContainerDied","Data":"f2b78d992596e8072eaeac818ff3e7e16ab0c26a42b0ef5c9ec57e4abcddffd0"} Jul 23 20:05:00.017979 ip-10-0-129-2 kubenswrapper[2564]: I0723 20:05:00.017951 2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-fb64e5-4l47h" Jul 23 20:05:00.184400 ip-10-0-129-2 kubenswrapper[2564]: I0723 20:05:00.184316 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/cb9e26a7-a0e6-403d-a67a-4ff71960e57a-maas-api-service-ca\") pod \"cb9e26a7-a0e6-403d-a67a-4ff71960e57a\" (UID: \"cb9e26a7-a0e6-403d-a67a-4ff71960e57a\") " Jul 23 20:05:00.184699 ip-10-0-129-2 kubenswrapper[2564]: I0723 20:05:00.184677 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cb9e26a7-a0e6-403d-a67a-4ff71960e57a-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "cb9e26a7-a0e6-403d-a67a-4ff71960e57a" (UID: "cb9e26a7-a0e6-403d-a67a-4ff71960e57a"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 23 20:05:00.285696 ip-10-0-129-2 kubenswrapper[2564]: I0723 20:05:00.285655 2564 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/cb9e26a7-a0e6-403d-a67a-4ff71960e57a-maas-api-service-ca\") on node \"ip-10-0-129-2.ec2.internal\" DevicePath \"\"" Jul 23 20:05:00.857410 ip-10-0-129-2 kubenswrapper[2564]: I0723 20:05:00.857367 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-fb64e5-4l47h" event={"ID":"cb9e26a7-a0e6-403d-a67a-4ff71960e57a","Type":"ContainerDied","Data":"29f52edf2191178852e17cd855250c0a97ff71f810b5bb7e7254ce2bd3adb8ae"} Jul 23 20:05:00.857410 ip-10-0-129-2 kubenswrapper[2564]: I0723 20:05:00.857405 2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-fb64e5-4l47h" Jul 23 20:05:00.857740 ip-10-0-129-2 kubenswrapper[2564]: I0723 20:05:00.857432 2564 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="29f52edf2191178852e17cd855250c0a97ff71f810b5bb7e7254ce2bd3adb8ae" Jul 23 20:05:07.990731 ip-10-0-129-2 kubenswrapper[2564]: I0723 20:05:07.990689 2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-fb64e5-7699755b7c-gjzt9"] Jul 23 20:05:07.991270 ip-10-0-129-2 kubenswrapper[2564]: I0723 20:05:07.991044 2564 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-fb64e5-7699755b7c-gjzt9" podUID="19811d4d-4066-47ed-9ae3-0c43d8fb6b93" containerName="maas-api" containerID="cri-o://bf6dd07cd7a72071cadb699d6763690688de9fb80737df4136ed19eec23b4a7f" gracePeriod=30 Jul 23 20:05:08.020531 ip-10-0-129-2 kubenswrapper[2564]: I0723 20:05:08.020485 2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-fb64e5-29747280-b52r7"] Jul 23 20:05:08.023469 ip-10-0-129-2 kubenswrapper[2564]: I0723 20:05:08.023440 2564 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-fb64e5-29747280-b52r7"] Jul 23 20:05:08.256829 ip-10-0-129-2 kubenswrapper[2564]: I0723 20:05:08.256737 2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-fb64e5-7699755b7c-gjzt9" Jul 23 20:05:08.361685 ip-10-0-129-2 kubenswrapper[2564]: I0723 20:05:08.361646 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/19811d4d-4066-47ed-9ae3-0c43d8fb6b93-openshift-service-ca\") pod \"19811d4d-4066-47ed-9ae3-0c43d8fb6b93\" (UID: \"19811d4d-4066-47ed-9ae3-0c43d8fb6b93\") " Jul 23 20:05:08.361685 ip-10-0-129-2 kubenswrapper[2564]: I0723 20:05:08.361696 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/19811d4d-4066-47ed-9ae3-0c43d8fb6b93-maas-api-tls\") pod \"19811d4d-4066-47ed-9ae3-0c43d8fb6b93\" (UID: \"19811d4d-4066-47ed-9ae3-0c43d8fb6b93\") " Jul 23 20:05:08.361977 ip-10-0-129-2 kubenswrapper[2564]: I0723 20:05:08.361747 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-khkkz\" (UniqueName: \"kubernetes.io/projected/19811d4d-4066-47ed-9ae3-0c43d8fb6b93-kube-api-access-khkkz\") pod \"19811d4d-4066-47ed-9ae3-0c43d8fb6b93\" (UID: \"19811d4d-4066-47ed-9ae3-0c43d8fb6b93\") " Jul 23 20:05:08.362112 ip-10-0-129-2 kubenswrapper[2564]: I0723 20:05:08.362086 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/19811d4d-4066-47ed-9ae3-0c43d8fb6b93-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "19811d4d-4066-47ed-9ae3-0c43d8fb6b93" (UID: "19811d4d-4066-47ed-9ae3-0c43d8fb6b93"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 23 20:05:08.364146 ip-10-0-129-2 kubenswrapper[2564]: I0723 20:05:08.364113 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/19811d4d-4066-47ed-9ae3-0c43d8fb6b93-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "19811d4d-4066-47ed-9ae3-0c43d8fb6b93" (UID: "19811d4d-4066-47ed-9ae3-0c43d8fb6b93"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 23 20:05:08.364285 ip-10-0-129-2 kubenswrapper[2564]: I0723 20:05:08.364221 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/19811d4d-4066-47ed-9ae3-0c43d8fb6b93-kube-api-access-khkkz" (OuterVolumeSpecName: "kube-api-access-khkkz") pod "19811d4d-4066-47ed-9ae3-0c43d8fb6b93" (UID: "19811d4d-4066-47ed-9ae3-0c43d8fb6b93"). InnerVolumeSpecName "kube-api-access-khkkz". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 23 20:05:08.463230 ip-10-0-129-2 kubenswrapper[2564]: I0723 20:05:08.463184 2564 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/19811d4d-4066-47ed-9ae3-0c43d8fb6b93-openshift-service-ca\") on node \"ip-10-0-129-2.ec2.internal\" DevicePath \"\"" Jul 23 20:05:08.463230 ip-10-0-129-2 kubenswrapper[2564]: I0723 20:05:08.463221 2564 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/19811d4d-4066-47ed-9ae3-0c43d8fb6b93-maas-api-tls\") on node \"ip-10-0-129-2.ec2.internal\" DevicePath \"\"" Jul 23 20:05:08.463230 ip-10-0-129-2 kubenswrapper[2564]: I0723 20:05:08.463232 2564 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-khkkz\" (UniqueName: \"kubernetes.io/projected/19811d4d-4066-47ed-9ae3-0c43d8fb6b93-kube-api-access-khkkz\") on node \"ip-10-0-129-2.ec2.internal\" DevicePath \"\"" Jul 23 20:05:08.659191 ip-10-0-129-2 kubenswrapper[2564]: I0723 20:05:08.659102 2564 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="de7f66c4-068a-4f4a-9f10-fde3c0bac17c" path="/var/lib/kubelet/pods/de7f66c4-068a-4f4a-9f10-fde3c0bac17c/volumes" Jul 23 20:05:08.907492 ip-10-0-129-2 kubenswrapper[2564]: I0723 20:05:08.907456 2564 generic.go:356] "Generic (PLEG): container finished" podID="19811d4d-4066-47ed-9ae3-0c43d8fb6b93" containerID="bf6dd07cd7a72071cadb699d6763690688de9fb80737df4136ed19eec23b4a7f" exitCode=0 Jul 23 20:05:08.907648 ip-10-0-129-2 kubenswrapper[2564]: I0723 20:05:08.907501 2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-fb64e5-7699755b7c-gjzt9" Jul 23 20:05:08.907648 ip-10-0-129-2 kubenswrapper[2564]: I0723 20:05:08.907536 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-fb64e5-7699755b7c-gjzt9" event={"ID":"19811d4d-4066-47ed-9ae3-0c43d8fb6b93","Type":"ContainerDied","Data":"bf6dd07cd7a72071cadb699d6763690688de9fb80737df4136ed19eec23b4a7f"} Jul 23 20:05:08.907648 ip-10-0-129-2 kubenswrapper[2564]: I0723 20:05:08.907594 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-fb64e5-7699755b7c-gjzt9" event={"ID":"19811d4d-4066-47ed-9ae3-0c43d8fb6b93","Type":"ContainerDied","Data":"9e9c53fe3f3f948f4b05a95cc9e1a746a3c34e49168afab68e711055a2eabebf"} Jul 23 20:05:08.907648 ip-10-0-129-2 kubenswrapper[2564]: I0723 20:05:08.907617 2564 scope.go:117] "RemoveContainer" containerID="bf6dd07cd7a72071cadb699d6763690688de9fb80737df4136ed19eec23b4a7f" Jul 23 20:05:08.918221 ip-10-0-129-2 kubenswrapper[2564]: I0723 20:05:08.918200 2564 scope.go:117] "RemoveContainer" containerID="bf6dd07cd7a72071cadb699d6763690688de9fb80737df4136ed19eec23b4a7f" Jul 23 20:05:08.918601 ip-10-0-129-2 kubenswrapper[2564]: E0723 20:05:08.918574 2564 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"bf6dd07cd7a72071cadb699d6763690688de9fb80737df4136ed19eec23b4a7f\": container with ID starting with bf6dd07cd7a72071cadb699d6763690688de9fb80737df4136ed19eec23b4a7f not found: ID does not exist" containerID="bf6dd07cd7a72071cadb699d6763690688de9fb80737df4136ed19eec23b4a7f" Jul 23 20:05:08.918697 ip-10-0-129-2 kubenswrapper[2564]: I0723 20:05:08.918615 2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bf6dd07cd7a72071cadb699d6763690688de9fb80737df4136ed19eec23b4a7f"} err="failed to get container status \"bf6dd07cd7a72071cadb699d6763690688de9fb80737df4136ed19eec23b4a7f\": rpc error: code = NotFound desc = could not find container \"bf6dd07cd7a72071cadb699d6763690688de9fb80737df4136ed19eec23b4a7f\": container with ID starting with bf6dd07cd7a72071cadb699d6763690688de9fb80737df4136ed19eec23b4a7f not found: ID does not exist" Jul 23 20:05:08.925523 ip-10-0-129-2 kubenswrapper[2564]: I0723 20:05:08.925494 2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-fb64e5-7699755b7c-gjzt9"] Jul 23 20:05:08.932032 ip-10-0-129-2 kubenswrapper[2564]: I0723 20:05:08.932004 2564 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-fb64e5-7699755b7c-gjzt9"] Jul 23 20:05:10.658519 ip-10-0-129-2 kubenswrapper[2564]: I0723 20:05:10.658483 2564 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="19811d4d-4066-47ed-9ae3-0c43d8fb6b93" path="/var/lib/kubelet/pods/19811d4d-4066-47ed-9ae3-0c43d8fb6b93/volumes" Jul 23 20:05:10.998305 ip-10-0-129-2 kubenswrapper[2564]: I0723 20:05:10.998272 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-2dgmk"] Jul 23 20:05:12.328292 ip-10-0-129-2 kubenswrapper[2564]: I0723 20:05:12.328241 2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-fb64e5-4l47h"] Jul 23 20:05:12.331439 ip-10-0-129-2 kubenswrapper[2564]: I0723 20:05:12.331406 2564 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-fb64e5-4l47h"] Jul 23 20:05:12.662112 ip-10-0-129-2 kubenswrapper[2564]: I0723 20:05:12.662086 2564 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="cb9e26a7-a0e6-403d-a67a-4ff71960e57a" path="/var/lib/kubelet/pods/cb9e26a7-a0e6-403d-a67a-4ff71960e57a/volumes" Jul 23 20:05:19.117760 ip-10-0-129-2 kubenswrapper[2564]: I0723 20:05:19.117715 2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-470e58-rbjdr"] Jul 23 20:05:19.118407 ip-10-0-129-2 kubenswrapper[2564]: I0723 20:05:19.118312 2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="cb9e26a7-a0e6-403d-a67a-4ff71960e57a" containerName="revoke-keys" Jul 23 20:05:19.118407 ip-10-0-129-2 kubenswrapper[2564]: I0723 20:05:19.118355 2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="cb9e26a7-a0e6-403d-a67a-4ff71960e57a" containerName="revoke-keys" Jul 23 20:05:19.118407 ip-10-0-129-2 kubenswrapper[2564]: I0723 20:05:19.118397 2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="19811d4d-4066-47ed-9ae3-0c43d8fb6b93" containerName="maas-api" Jul 23 20:05:19.118407 ip-10-0-129-2 kubenswrapper[2564]: I0723 20:05:19.118406 2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="19811d4d-4066-47ed-9ae3-0c43d8fb6b93" containerName="maas-api" Jul 23 20:05:19.118637 ip-10-0-129-2 kubenswrapper[2564]: I0723 20:05:19.118499 2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="19811d4d-4066-47ed-9ae3-0c43d8fb6b93" containerName="maas-api" Jul 23 20:05:19.118637 ip-10-0-129-2 kubenswrapper[2564]: I0723 20:05:19.118511 2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="cb9e26a7-a0e6-403d-a67a-4ff71960e57a" containerName="revoke-keys" Jul 23 20:05:19.122982 ip-10-0-129-2 kubenswrapper[2564]: I0723 20:05:19.122959 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-470e58-rbjdr" Jul 23 20:05:19.125503 ip-10-0-129-2 kubenswrapper[2564]: I0723 20:05:19.125479 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Jul 23 20:05:19.126087 ip-10-0-129-2 kubenswrapper[2564]: I0723 20:05:19.126064 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-gcspk\"" Jul 23 20:05:19.129606 ip-10-0-129-2 kubenswrapper[2564]: I0723 20:05:19.129563 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-470e58-rbjdr"] Jul 23 20:05:19.159491 ip-10-0-129-2 kubenswrapper[2564]: I0723 20:05:19.159456 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e118b593-f5c2-40e3-9eef-9faefbde1aa8-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-470e58-rbjdr\" (UID: \"e118b593-f5c2-40e3-9eef-9faefbde1aa8\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-470e58-rbjdr" Jul 23 20:05:19.260809 ip-10-0-129-2 kubenswrapper[2564]: I0723 20:05:19.260768 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e118b593-f5c2-40e3-9eef-9faefbde1aa8-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-470e58-rbjdr\" (UID: \"e118b593-f5c2-40e3-9eef-9faefbde1aa8\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-470e58-rbjdr" Jul 23 20:05:19.261522 ip-10-0-129-2 kubenswrapper[2564]: I0723 20:05:19.261497 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e118b593-f5c2-40e3-9eef-9faefbde1aa8-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-470e58-rbjdr\" (UID: \"e118b593-f5c2-40e3-9eef-9faefbde1aa8\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-470e58-rbjdr" Jul 23 20:05:19.436357 ip-10-0-129-2 kubenswrapper[2564]: I0723 20:05:19.436303 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-470e58-rbjdr" Jul 23 20:05:19.587391 ip-10-0-129-2 kubenswrapper[2564]: I0723 20:05:19.587359 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-470e58-rbjdr"] Jul 23 20:05:19.589392 ip-10-0-129-2 kubenswrapper[2564]: W0723 20:05:19.589363 2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode118b593_f5c2_40e3_9eef_9faefbde1aa8.slice/crio-e776dc8d7d2ffd2304bfabbb875c3fd69a0015d62b24f300369e17fcbb4407d6 WatchSource:0}: Error finding container e776dc8d7d2ffd2304bfabbb875c3fd69a0015d62b24f300369e17fcbb4407d6: Status 404 returned error can't find the container with id e776dc8d7d2ffd2304bfabbb875c3fd69a0015d62b24f300369e17fcbb4407d6 Jul 23 20:05:19.975703 ip-10-0-129-2 kubenswrapper[2564]: I0723 20:05:19.975669 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-470e58-rbjdr" event={"ID":"e118b593-f5c2-40e3-9eef-9faefbde1aa8","Type":"ContainerStarted","Data":"68dba2e5a45196bca6fe86c51700619263225e66ab7c4b3d1723650e66547ec4"} Jul 23 20:05:19.975886 ip-10-0-129-2 kubenswrapper[2564]: I0723 20:05:19.975750 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-470e58-rbjdr" event={"ID":"e118b593-f5c2-40e3-9eef-9faefbde1aa8","Type":"ContainerStarted","Data":"e776dc8d7d2ffd2304bfabbb875c3fd69a0015d62b24f300369e17fcbb4407d6"} Jul 23 20:05:19.994236 ip-10-0-129-2 kubenswrapper[2564]: I0723 20:05:19.994160 2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-470e58-rbjdr" podStartSLOduration=0.994138403 podStartE2EDuration="994.138403ms" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-23 20:05:19 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-23 20:05:19.991518583 +0000 UTC m=+2745.843294835" watchObservedRunningTime="2026-07-23 20:05:19.994138403 +0000 UTC m=+2745.845914653" Jul 23 20:05:20.983634 ip-10-0-129-2 kubenswrapper[2564]: I0723 20:05:20.983605 2564 generic.go:356] "Generic (PLEG): container finished" podID="e118b593-f5c2-40e3-9eef-9faefbde1aa8" containerID="68dba2e5a45196bca6fe86c51700619263225e66ab7c4b3d1723650e66547ec4" exitCode=0 Jul 23 20:05:20.984059 ip-10-0-129-2 kubenswrapper[2564]: I0723 20:05:20.983754 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-470e58-rbjdr" event={"ID":"e118b593-f5c2-40e3-9eef-9faefbde1aa8","Type":"ContainerDied","Data":"68dba2e5a45196bca6fe86c51700619263225e66ab7c4b3d1723650e66547ec4"} Jul 23 20:05:22.139107 ip-10-0-129-2 kubenswrapper[2564]: I0723 20:05:22.139073 2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-470e58-rbjdr" Jul 23 20:05:22.189939 ip-10-0-129-2 kubenswrapper[2564]: I0723 20:05:22.189903 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e118b593-f5c2-40e3-9eef-9faefbde1aa8-maas-api-service-ca\") pod \"e118b593-f5c2-40e3-9eef-9faefbde1aa8\" (UID: \"e118b593-f5c2-40e3-9eef-9faefbde1aa8\") " Jul 23 20:05:22.190260 ip-10-0-129-2 kubenswrapper[2564]: I0723 20:05:22.190238 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e118b593-f5c2-40e3-9eef-9faefbde1aa8-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "e118b593-f5c2-40e3-9eef-9faefbde1aa8" (UID: "e118b593-f5c2-40e3-9eef-9faefbde1aa8"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 23 20:05:22.291299 ip-10-0-129-2 kubenswrapper[2564]: I0723 20:05:22.291205 2564 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e118b593-f5c2-40e3-9eef-9faefbde1aa8-maas-api-service-ca\") on node \"ip-10-0-129-2.ec2.internal\" DevicePath \"\"" Jul 23 20:05:22.998587 ip-10-0-129-2 kubenswrapper[2564]: I0723 20:05:22.998494 2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-470e58-rbjdr" Jul 23 20:05:22.998587 ip-10-0-129-2 kubenswrapper[2564]: I0723 20:05:22.998530 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-470e58-rbjdr" event={"ID":"e118b593-f5c2-40e3-9eef-9faefbde1aa8","Type":"ContainerDied","Data":"e776dc8d7d2ffd2304bfabbb875c3fd69a0015d62b24f300369e17fcbb4407d6"} Jul 23 20:05:22.998587 ip-10-0-129-2 kubenswrapper[2564]: I0723 20:05:22.998586 2564 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="e776dc8d7d2ffd2304bfabbb875c3fd69a0015d62b24f300369e17fcbb4407d6" Jul 23 20:05:29.692513 ip-10-0-129-2 kubenswrapper[2564]: I0723 20:05:29.692472 2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-470e58-29747280-hrrmv"] Jul 23 20:05:29.697429 ip-10-0-129-2 kubenswrapper[2564]: I0723 20:05:29.697398 2564 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-470e58-29747280-hrrmv"] Jul 23 20:05:30.658738 ip-10-0-129-2 kubenswrapper[2564]: I0723 20:05:30.658706 2564 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="22392289-8b04-48dd-8a18-43a79fcb5dda" path="/var/lib/kubelet/pods/22392289-8b04-48dd-8a18-43a79fcb5dda/volumes" Jul 23 20:05:32.599471 ip-10-0-129-2 kubenswrapper[2564]: I0723 20:05:32.599430 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-2dgmk"] Jul 23 20:05:34.234996 ip-10-0-129-2 kubenswrapper[2564]: I0723 20:05:34.234955 2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-470e58-rbjdr"] Jul 23 20:05:34.237281 ip-10-0-129-2 kubenswrapper[2564]: I0723 20:05:34.237259 2564 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-470e58-rbjdr"] Jul 23 20:05:34.659541 ip-10-0-129-2 kubenswrapper[2564]: I0723 20:05:34.659446 2564 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="e118b593-f5c2-40e3-9eef-9faefbde1aa8" path="/var/lib/kubelet/pods/e118b593-f5c2-40e3-9eef-9faefbde1aa8/volumes" Jul 23 20:05:35.810209 ip-10-0-129-2 kubenswrapper[2564]: I0723 20:05:35.810174 2564 scope.go:117] "RemoveContainer" containerID="f40a25df307a3d2fc204b0d2d88b81b62e4b35df742003902f2c40bbbfce7711" Jul 23 20:05:46.510844 ip-10-0-129-2 kubenswrapper[2564]: I0723 20:05:46.510811 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_kserve-controller-manager-d6f4b65b8-d6kk8_f6e838ac-b62f-45ee-8a24-937c0ad19b09/manager/0.log" Jul 23 20:05:46.638893 ip-10-0-129-2 kubenswrapper[2564]: I0723 20:05:46.638857 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_llmisvc-controller-manager-777c6bd7d5-7qz4p_bddb241b-f968-4140-86c1-9aeddc995590/manager/0.log" Jul 23 20:05:46.882840 ip-10-0-129-2 kubenswrapper[2564]: I0723 20:05:46.882748 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_model-serving-api-54d89bd5fc-bhjvm_946f6fb8-32c2-406c-819e-8bf5b690971b/server/0.log" Jul 23 20:05:47.013344 ip-10-0-129-2 kubenswrapper[2564]: I0723 20:05:47.013284 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_odh-model-controller-7998c67dc4-52fhl_a88408a9-0854-4017-8248-4fa9c183712d/manager/0.log" Jul 23 20:05:47.386177 ip-10-0-129-2 kubenswrapper[2564]: I0723 20:05:47.386153 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-559f567dc-sc2mg_3deb6cad-ea6c-4be3-95ad-6b4b5fbad490/copy-manifests/0.log" Jul 23 20:05:47.393552 ip-10-0-129-2 kubenswrapper[2564]: I0723 20:05:47.393524 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-559f567dc-sc2mg_3deb6cad-ea6c-4be3-95ad-6b4b5fbad490/manager/0.log" Jul 23 20:05:47.842016 ip-10-0-129-2 kubenswrapper[2564]: I0723 20:05:47.841990 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-key-cleanup-29747265-qd6f8_00515ff4-26d7-4e68-ba28-d8254b4f3e5c/cleanup/0.log" Jul 23 20:05:47.953910 ip-10-0-129-2 kubenswrapper[2564]: I0723 20:05:47.953878 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-key-cleanup-29747280-vw88h_2432a745-c8d5-4505-a20a-4d82fe439720/cleanup/0.log" Jul 23 20:05:48.874342 ip-10-0-129-2 kubenswrapper[2564]: I0723 20:05:48.874292 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b75982c7j_fd160cf4-96c6-410f-a5b3-7dc83bb993c7/util/0.log" Jul 23 20:05:48.881701 ip-10-0-129-2 kubenswrapper[2564]: I0723 20:05:48.881677 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b75982c7j_fd160cf4-96c6-410f-a5b3-7dc83bb993c7/pull/0.log" Jul 23 20:05:48.888664 ip-10-0-129-2 kubenswrapper[2564]: I0723 20:05:48.888640 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b75982c7j_fd160cf4-96c6-410f-a5b3-7dc83bb993c7/extract/0.log" Jul 23 20:05:49.004437 ip-10-0-129-2 kubenswrapper[2564]: I0723 20:05:49.004403 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e02zdvq_7326cf84-a21c-4c92-a262-fbff97a47bbd/util/0.log" Jul 23 20:05:49.011432 ip-10-0-129-2 kubenswrapper[2564]: I0723 20:05:49.011403 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e02zdvq_7326cf84-a21c-4c92-a262-fbff97a47bbd/pull/0.log" Jul 23 20:05:49.018785 ip-10-0-129-2 kubenswrapper[2564]: I0723 20:05:49.018753 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e02zdvq_7326cf84-a21c-4c92-a262-fbff97a47bbd/extract/0.log" Jul 23 20:05:49.127218 ip-10-0-129-2 kubenswrapper[2564]: I0723 20:05:49.127129 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73nmtfx_d1495985-7d1d-486a-894a-9af5eedb393e/util/0.log" Jul 23 20:05:49.133523 ip-10-0-129-2 kubenswrapper[2564]: I0723 20:05:49.133488 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73nmtfx_d1495985-7d1d-486a-894a-9af5eedb393e/pull/0.log" Jul 23 20:05:49.139535 ip-10-0-129-2 kubenswrapper[2564]: I0723 20:05:49.139507 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73nmtfx_d1495985-7d1d-486a-894a-9af5eedb393e/extract/0.log" Jul 23 20:05:49.253567 ip-10-0-129-2 kubenswrapper[2564]: I0723 20:05:49.253534 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1m88qh_acf936f7-9ae5-4c32-af67-8f92692382a7/util/0.log" Jul 23 20:05:49.260783 ip-10-0-129-2 kubenswrapper[2564]: I0723 20:05:49.260749 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1m88qh_acf936f7-9ae5-4c32-af67-8f92692382a7/pull/0.log" Jul 23 20:05:49.267831 ip-10-0-129-2 kubenswrapper[2564]: I0723 20:05:49.267803 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1m88qh_acf936f7-9ae5-4c32-af67-8f92692382a7/extract/0.log" Jul 23 20:05:50.062692 ip-10-0-129-2 kubenswrapper[2564]: I0723 20:05:50.062668 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-limitador-54d6d988d5-2dgmk_ebafc262-cd0e-4e53-b4ce-def03413139e/limitador/0.log" Jul 23 20:05:50.659180 ip-10-0-129-2 kubenswrapper[2564]: I0723 20:05:50.659150 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_istiod-openshift-gateway-94bb8fbfd-9pnfp_c983ed8c-9773-4fc8-b029-fb1892487cc6/discovery/0.log" Jul 23 20:05:50.878856 ip-10-0-129-2 kubenswrapper[2564]: I0723 20:05:50.878819 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-5fdd4658f-lkckb_e20f2247-2981-4d9b-83d2-eeb094ceb1f6/kube-auth-proxy/0.log" Jul 23 20:05:51.003499 ip-10-0-129-2 kubenswrapper[2564]: I0723 20:05:51.003457 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-56654dcf7-pp577_01866086-e44f-4c2d-9f3d-5b5a7cf79029/istio-proxy/1.log" Jul 23 20:05:51.241221 ip-10-0-129-2 kubenswrapper[2564]: I0723 20:05:51.241177 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-5bf47cd988-rwnsj_45506cd4-c775-4185-888e-607405dd6675/payload-processing/0.log" Jul 23 20:05:51.358218 ip-10-0-129-2 kubenswrapper[2564]: I0723 20:05:51.358133 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_router-default-7f64bc459d-9b6vg_b7c6dedb-e5d3-41a0-b060-50a88ccca303/router/0.log" Jul 23 20:05:51.806674 ip-10-0-129-2 kubenswrapper[2564]: I0723 20:05:51.806642 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-distinct-simulated-kserve-59975d69f9-7w7zq_f2240a8a-5e11-48fe-97e3-874df0e79f7e/main/0.log" Jul 23 20:05:52.025934 ip-10-0-129-2 kubenswrapper[2564]: I0723 20:05:52.025900 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f8wfwr_dfb3b12c-9e0d-460f-a7ba-66239eaf52e9/main/0.log" Jul 23 20:06:01.216410 ip-10-0-129-2 kubenswrapper[2564]: I0723 20:06:01.216301 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_global-pull-secret-syncer-rtgbq_40f386ec-607d-496f-b65b-03a1fd847b1f/global-pull-secret-syncer/0.log" Jul 23 20:06:01.319743 ip-10-0-129-2 kubenswrapper[2564]: I0723 20:06:01.319713 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_konnectivity-agent-8rq79_25ca5174-01a0-41fc-9293-77dea2d9ec1f/konnectivity-agent/0.log" Jul 23 20:06:01.380784 ip-10-0-129-2 kubenswrapper[2564]: I0723 20:06:01.380754 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kube-apiserver-proxy-ip-10-0-129-2.ec2.internal_f9c41c89c718c39ff54e57e416ba8933/haproxy/0.log" Jul 23 20:06:06.061937 ip-10-0-129-2 kubenswrapper[2564]: I0723 20:06:06.061895 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b75982c7j_fd160cf4-96c6-410f-a5b3-7dc83bb993c7/extract/0.log" Jul 23 20:06:06.086521 ip-10-0-129-2 kubenswrapper[2564]: I0723 20:06:06.086493 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b75982c7j_fd160cf4-96c6-410f-a5b3-7dc83bb993c7/util/0.log" Jul 23 20:06:06.111225 ip-10-0-129-2 kubenswrapper[2564]: I0723 20:06:06.111190 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b75982c7j_fd160cf4-96c6-410f-a5b3-7dc83bb993c7/pull/0.log" Jul 23 20:06:06.139176 ip-10-0-129-2 kubenswrapper[2564]: I0723 20:06:06.139142 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e02zdvq_7326cf84-a21c-4c92-a262-fbff97a47bbd/extract/0.log" Jul 23 20:06:06.161768 ip-10-0-129-2 kubenswrapper[2564]: I0723 20:06:06.161728 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e02zdvq_7326cf84-a21c-4c92-a262-fbff97a47bbd/util/0.log" Jul 23 20:06:06.188979 ip-10-0-129-2 kubenswrapper[2564]: I0723 20:06:06.188948 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e02zdvq_7326cf84-a21c-4c92-a262-fbff97a47bbd/pull/0.log" Jul 23 20:06:06.219306 ip-10-0-129-2 kubenswrapper[2564]: I0723 20:06:06.219278 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73nmtfx_d1495985-7d1d-486a-894a-9af5eedb393e/extract/0.log" Jul 23 20:06:06.242773 ip-10-0-129-2 kubenswrapper[2564]: I0723 20:06:06.242741 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73nmtfx_d1495985-7d1d-486a-894a-9af5eedb393e/util/0.log" Jul 23 20:06:06.265058 ip-10-0-129-2 kubenswrapper[2564]: I0723 20:06:06.265022 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73nmtfx_d1495985-7d1d-486a-894a-9af5eedb393e/pull/0.log" Jul 23 20:06:06.295993 ip-10-0-129-2 kubenswrapper[2564]: I0723 20:06:06.295960 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1m88qh_acf936f7-9ae5-4c32-af67-8f92692382a7/extract/0.log" Jul 23 20:06:06.319682 ip-10-0-129-2 kubenswrapper[2564]: I0723 20:06:06.319583 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1m88qh_acf936f7-9ae5-4c32-af67-8f92692382a7/util/0.log" Jul 23 20:06:06.343763 ip-10-0-129-2 kubenswrapper[2564]: I0723 20:06:06.343730 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1m88qh_acf936f7-9ae5-4c32-af67-8f92692382a7/pull/0.log" Jul 23 20:06:07.488551 ip-10-0-129-2 kubenswrapper[2564]: I0723 20:06:07.488519 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-limitador-54d6d988d5-2dgmk_ebafc262-cd0e-4e53-b4ce-def03413139e/limitador/0.log" Jul 23 20:06:09.490014 ip-10-0-129-2 kubenswrapper[2564]: I0723 20:06:09.489972 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_cluster-monitoring-operator-76cf8966f8-vv2vj_6f0a5836-d7de-49fc-b3a2-49830eaa468f/cluster-monitoring-operator/0.log" Jul 23 20:06:09.795248 ip-10-0-129-2 kubenswrapper[2564]: I0723 20:06:09.795153 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-fhzlr_06754a58-2f04-44d1-9618-2ce5a6d6468b/node-exporter/0.log" Jul 23 20:06:09.821244 ip-10-0-129-2 kubenswrapper[2564]: I0723 20:06:09.821212 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-fhzlr_06754a58-2f04-44d1-9618-2ce5a6d6468b/kube-rbac-proxy/0.log" Jul 23 20:06:09.843502 ip-10-0-129-2 kubenswrapper[2564]: I0723 20:06:09.843476 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-fhzlr_06754a58-2f04-44d1-9618-2ce5a6d6468b/init-textfile/0.log" Jul 23 20:06:10.186001 ip-10-0-129-2 kubenswrapper[2564]: I0723 20:06:10.185965 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_telemeter-client-75f59688bc-p2725_cbe5aee7-9398-489a-a1db-4b2c8e1d7b1f/telemeter-client/0.log" Jul 23 20:06:10.212634 ip-10-0-129-2 kubenswrapper[2564]: I0723 20:06:10.212597 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_telemeter-client-75f59688bc-p2725_cbe5aee7-9398-489a-a1db-4b2c8e1d7b1f/reload/0.log" Jul 23 20:06:10.237065 ip-10-0-129-2 kubenswrapper[2564]: I0723 20:06:10.237028 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_telemeter-client-75f59688bc-p2725_cbe5aee7-9398-489a-a1db-4b2c8e1d7b1f/kube-rbac-proxy/0.log" Jul 23 20:06:10.889293 ip-10-0-129-2 kubenswrapper[2564]: I0723 20:06:10.889243 2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-must-gather-tjlxs/perf-node-gather-daemonset-sr4mn"] Jul 23 20:06:10.889773 ip-10-0-129-2 kubenswrapper[2564]: I0723 20:06:10.889743 2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e118b593-f5c2-40e3-9eef-9faefbde1aa8" containerName="revoke-keys" Jul 23 20:06:10.889773 ip-10-0-129-2 kubenswrapper[2564]: I0723 20:06:10.889760 2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="e118b593-f5c2-40e3-9eef-9faefbde1aa8" containerName="revoke-keys" Jul 23 20:06:10.889855 ip-10-0-129-2 kubenswrapper[2564]: I0723 20:06:10.889837 2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="e118b593-f5c2-40e3-9eef-9faefbde1aa8" containerName="revoke-keys" Jul 23 20:06:10.894481 ip-10-0-129-2 kubenswrapper[2564]: I0723 20:06:10.894457 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-tjlxs/perf-node-gather-daemonset-sr4mn" Jul 23 20:06:10.899711 ip-10-0-129-2 kubenswrapper[2564]: I0723 20:06:10.899688 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-tjlxs\"/\"openshift-service-ca.crt\"" Jul 23 20:06:10.900844 ip-10-0-129-2 kubenswrapper[2564]: I0723 20:06:10.900821 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-tjlxs\"/\"kube-root-ca.crt\"" Jul 23 20:06:10.900979 ip-10-0-129-2 kubenswrapper[2564]: I0723 20:06:10.900827 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-tjlxs\"/\"default-dockercfg-sm7kz\"" Jul 23 20:06:10.909434 ip-10-0-129-2 kubenswrapper[2564]: I0723 20:06:10.909406 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-tjlxs/perf-node-gather-daemonset-sr4mn"] Jul 23 20:06:10.940191 ip-10-0-129-2 kubenswrapper[2564]: I0723 20:06:10.940155 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/faccb774-2773-42ba-a281-904afc16c307-lib-modules\") pod \"perf-node-gather-daemonset-sr4mn\" (UID: \"faccb774-2773-42ba-a281-904afc16c307\") " pod="openshift-must-gather-tjlxs/perf-node-gather-daemonset-sr4mn" Jul 23 20:06:10.940191 ip-10-0-129-2 kubenswrapper[2564]: I0723 20:06:10.940199 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-plkwc\" (UniqueName: \"kubernetes.io/projected/faccb774-2773-42ba-a281-904afc16c307-kube-api-access-plkwc\") pod \"perf-node-gather-daemonset-sr4mn\" (UID: \"faccb774-2773-42ba-a281-904afc16c307\") " pod="openshift-must-gather-tjlxs/perf-node-gather-daemonset-sr4mn" Jul 23 20:06:10.940511 ip-10-0-129-2 kubenswrapper[2564]: I0723 20:06:10.940224 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/faccb774-2773-42ba-a281-904afc16c307-podres\") pod \"perf-node-gather-daemonset-sr4mn\" (UID: \"faccb774-2773-42ba-a281-904afc16c307\") " pod="openshift-must-gather-tjlxs/perf-node-gather-daemonset-sr4mn" Jul 23 20:06:10.940511 ip-10-0-129-2 kubenswrapper[2564]: I0723 20:06:10.940297 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/faccb774-2773-42ba-a281-904afc16c307-sys\") pod \"perf-node-gather-daemonset-sr4mn\" (UID: \"faccb774-2773-42ba-a281-904afc16c307\") " pod="openshift-must-gather-tjlxs/perf-node-gather-daemonset-sr4mn" Jul 23 20:06:10.940511 ip-10-0-129-2 kubenswrapper[2564]: I0723 20:06:10.940369 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/faccb774-2773-42ba-a281-904afc16c307-proc\") pod \"perf-node-gather-daemonset-sr4mn\" (UID: \"faccb774-2773-42ba-a281-904afc16c307\") " pod="openshift-must-gather-tjlxs/perf-node-gather-daemonset-sr4mn" Jul 23 20:06:11.041915 ip-10-0-129-2 kubenswrapper[2564]: I0723 20:06:11.041853 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/faccb774-2773-42ba-a281-904afc16c307-lib-modules\") pod \"perf-node-gather-daemonset-sr4mn\" (UID: \"faccb774-2773-42ba-a281-904afc16c307\") " pod="openshift-must-gather-tjlxs/perf-node-gather-daemonset-sr4mn" Jul 23 20:06:11.041915 ip-10-0-129-2 kubenswrapper[2564]: I0723 20:06:11.041923 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-plkwc\" (UniqueName: \"kubernetes.io/projected/faccb774-2773-42ba-a281-904afc16c307-kube-api-access-plkwc\") pod \"perf-node-gather-daemonset-sr4mn\" (UID: \"faccb774-2773-42ba-a281-904afc16c307\") " pod="openshift-must-gather-tjlxs/perf-node-gather-daemonset-sr4mn" Jul 23 20:06:11.042204 ip-10-0-129-2 kubenswrapper[2564]: I0723 20:06:11.041949 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/faccb774-2773-42ba-a281-904afc16c307-podres\") pod \"perf-node-gather-daemonset-sr4mn\" (UID: \"faccb774-2773-42ba-a281-904afc16c307\") " pod="openshift-must-gather-tjlxs/perf-node-gather-daemonset-sr4mn" Jul 23 20:06:11.042204 ip-10-0-129-2 kubenswrapper[2564]: I0723 20:06:11.041979 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/faccb774-2773-42ba-a281-904afc16c307-sys\") pod \"perf-node-gather-daemonset-sr4mn\" (UID: \"faccb774-2773-42ba-a281-904afc16c307\") " pod="openshift-must-gather-tjlxs/perf-node-gather-daemonset-sr4mn" Jul 23 20:06:11.042204 ip-10-0-129-2 kubenswrapper[2564]: I0723 20:06:11.042014 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/faccb774-2773-42ba-a281-904afc16c307-proc\") pod \"perf-node-gather-daemonset-sr4mn\" (UID: \"faccb774-2773-42ba-a281-904afc16c307\") " pod="openshift-must-gather-tjlxs/perf-node-gather-daemonset-sr4mn" Jul 23 20:06:11.042204 ip-10-0-129-2 kubenswrapper[2564]: I0723 20:06:11.042064 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/faccb774-2773-42ba-a281-904afc16c307-lib-modules\") pod \"perf-node-gather-daemonset-sr4mn\" (UID: \"faccb774-2773-42ba-a281-904afc16c307\") " pod="openshift-must-gather-tjlxs/perf-node-gather-daemonset-sr4mn" Jul 23 20:06:11.042204 ip-10-0-129-2 kubenswrapper[2564]: I0723 20:06:11.042087 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/faccb774-2773-42ba-a281-904afc16c307-podres\") pod \"perf-node-gather-daemonset-sr4mn\" (UID: \"faccb774-2773-42ba-a281-904afc16c307\") " pod="openshift-must-gather-tjlxs/perf-node-gather-daemonset-sr4mn" Jul 23 20:06:11.042204 ip-10-0-129-2 kubenswrapper[2564]: I0723 20:06:11.042098 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/faccb774-2773-42ba-a281-904afc16c307-sys\") pod \"perf-node-gather-daemonset-sr4mn\" (UID: \"faccb774-2773-42ba-a281-904afc16c307\") " pod="openshift-must-gather-tjlxs/perf-node-gather-daemonset-sr4mn" Jul 23 20:06:11.042204 ip-10-0-129-2 kubenswrapper[2564]: I0723 20:06:11.042092 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/faccb774-2773-42ba-a281-904afc16c307-proc\") pod \"perf-node-gather-daemonset-sr4mn\" (UID: \"faccb774-2773-42ba-a281-904afc16c307\") " pod="openshift-must-gather-tjlxs/perf-node-gather-daemonset-sr4mn" Jul 23 20:06:11.054632 ip-10-0-129-2 kubenswrapper[2564]: I0723 20:06:11.054604 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-plkwc\" (UniqueName: \"kubernetes.io/projected/faccb774-2773-42ba-a281-904afc16c307-kube-api-access-plkwc\") pod \"perf-node-gather-daemonset-sr4mn\" (UID: \"faccb774-2773-42ba-a281-904afc16c307\") " pod="openshift-must-gather-tjlxs/perf-node-gather-daemonset-sr4mn" Jul 23 20:06:11.205921 ip-10-0-129-2 kubenswrapper[2564]: I0723 20:06:11.205876 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-tjlxs/perf-node-gather-daemonset-sr4mn" Jul 23 20:06:11.359959 ip-10-0-129-2 kubenswrapper[2564]: I0723 20:06:11.359928 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-tjlxs/perf-node-gather-daemonset-sr4mn"] Jul 23 20:06:11.364312 ip-10-0-129-2 kubenswrapper[2564]: W0723 20:06:11.364257 2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-podfaccb774_2773_42ba_a281_904afc16c307.slice/crio-97705d5f13afb14a4f729ebe00ff6d69a1108375906c785385980d51ec551b01 WatchSource:0}: Error finding container 97705d5f13afb14a4f729ebe00ff6d69a1108375906c785385980d51ec551b01: Status 404 returned error can't find the container with id 97705d5f13afb14a4f729ebe00ff6d69a1108375906c785385980d51ec551b01 Jul 23 20:06:11.634542 ip-10-0-129-2 kubenswrapper[2564]: I0723 20:06:11.634497 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-network-console_networking-console-plugin-7bc7489cb5-4p45l_7c2f37e4-a8b7-497e-ac12-7a6f5dd86619/networking-console-plugin/0.log" Jul 23 20:06:12.232426 ip-10-0-129-2 kubenswrapper[2564]: I0723 20:06:12.232390 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-565b7b569d-z5wf4_ded5b3af-c74c-4d5a-aa77-62fe72d1dd29/console-operator/1.log" Jul 23 20:06:12.241737 ip-10-0-129-2 kubenswrapper[2564]: I0723 20:06:12.241699 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-565b7b569d-z5wf4_ded5b3af-c74c-4d5a-aa77-62fe72d1dd29/console-operator/2.log" Jul 23 20:06:12.301883 ip-10-0-129-2 kubenswrapper[2564]: I0723 20:06:12.301845 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-tjlxs/perf-node-gather-daemonset-sr4mn" event={"ID":"faccb774-2773-42ba-a281-904afc16c307","Type":"ContainerStarted","Data":"1852c1b4c62eb21f41a51a3e88e5a14f083b9b6a852ae771061ac3eb7960c62f"} Jul 23 20:06:12.302063 ip-10-0-129-2 kubenswrapper[2564]: I0723 20:06:12.301897 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-tjlxs/perf-node-gather-daemonset-sr4mn" event={"ID":"faccb774-2773-42ba-a281-904afc16c307","Type":"ContainerStarted","Data":"97705d5f13afb14a4f729ebe00ff6d69a1108375906c785385980d51ec551b01"} Jul 23 20:06:12.302063 ip-10-0-129-2 kubenswrapper[2564]: I0723 20:06:12.301925 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-must-gather-tjlxs/perf-node-gather-daemonset-sr4mn" Jul 23 20:06:12.325997 ip-10-0-129-2 kubenswrapper[2564]: I0723 20:06:12.325938 2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-must-gather-tjlxs/perf-node-gather-daemonset-sr4mn" podStartSLOduration=2.325919456 podStartE2EDuration="2.325919456s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-23 20:06:10 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-23 20:06:12.323749124 +0000 UTC m=+2798.175525374" watchObservedRunningTime="2026-07-23 20:06:12.325919456 +0000 UTC m=+2798.177695703" Jul 23 20:06:12.726398 ip-10-0-129-2 kubenswrapper[2564]: I0723 20:06:12.726360 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console_downloads-5bc67dcd46-4plfk_fff3e347-7210-4a43-a41d-2bf0dcb0edb0/download-server/0.log" Jul 23 20:06:13.308337 ip-10-0-129-2 kubenswrapper[2564]: I0723 20:06:13.308293 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-cluster-storage-operator_volume-data-source-validator-78cf4c6f6-zld8c_07b02e43-f95a-4173-b027-d0979fbbca4e/volume-data-source-validator/0.log" Jul 23 20:06:14.484190 ip-10-0-129-2 kubenswrapper[2564]: I0723 20:06:14.484156 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-vzws2_380698de-d0b8-426a-9e9b-708c0e36e818/dns/0.log" Jul 23 20:06:14.545817 ip-10-0-129-2 kubenswrapper[2564]: I0723 20:06:14.545780 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-vzws2_380698de-d0b8-426a-9e9b-708c0e36e818/kube-rbac-proxy/0.log" Jul 23 20:06:14.573218 ip-10-0-129-2 kubenswrapper[2564]: I0723 20:06:14.573185 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-6k8wr_23313c02-67bf-498f-a294-e16cdc7d4517/dns-node-resolver/0.log" Jul 23 20:06:15.193423 ip-10-0-129-2 kubenswrapper[2564]: I0723 20:06:15.193388 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-9wwn6_dd466782-4d8e-42e2-83c2-ed5f1b48567b/node-ca/0.log" Jul 23 20:06:16.518079 ip-10-0-129-2 kubenswrapper[2564]: I0723 20:06:16.518051 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_istiod-openshift-gateway-94bb8fbfd-9pnfp_c983ed8c-9773-4fc8-b029-fb1892487cc6/discovery/0.log" Jul 23 20:06:16.568102 ip-10-0-129-2 kubenswrapper[2564]: I0723 20:06:16.568072 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-5fdd4658f-lkckb_e20f2247-2981-4d9b-83d2-eeb094ceb1f6/kube-auth-proxy/0.log" Jul 23 20:06:16.600950 ip-10-0-129-2 kubenswrapper[2564]: I0723 20:06:16.600914 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-56654dcf7-pp577_01866086-e44f-4c2d-9f3d-5b5a7cf79029/istio-proxy/0.log" Jul 23 20:06:16.656588 ip-10-0-129-2 kubenswrapper[2564]: I0723 20:06:16.656531 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-56654dcf7-pp577_01866086-e44f-4c2d-9f3d-5b5a7cf79029/istio-proxy/1.log" Jul 23 20:06:16.739371 ip-10-0-129-2 kubenswrapper[2564]: I0723 20:06:16.739339 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-5bf47cd988-rwnsj_45506cd4-c775-4185-888e-607405dd6675/payload-processing/0.log" Jul 23 20:06:16.770723 ip-10-0-129-2 kubenswrapper[2564]: I0723 20:06:16.770635 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_router-default-7f64bc459d-9b6vg_b7c6dedb-e5d3-41a0-b060-50a88ccca303/router/0.log" Jul 23 20:06:17.587506 ip-10-0-129-2 kubenswrapper[2564]: I0723 20:06:17.587478 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-wxwk8_e6d7c065-91d9-461c-8c0f-684ad4ea70e6/serve-healthcheck-canary/0.log" Jul 23 20:06:18.084684 ip-10-0-129-2 kubenswrapper[2564]: I0723 20:06:18.084644 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-operator-555d9d7885-r8lv4_00953f2a-6543-445a-8895-40525b4e3e25/insights-operator/0.log" Jul 23 20:06:18.086284 ip-10-0-129-2 kubenswrapper[2564]: I0723 20:06:18.086259 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-operator-555d9d7885-r8lv4_00953f2a-6543-445a-8895-40525b4e3e25/insights-operator/1.log" Jul 23 20:06:18.108524 ip-10-0-129-2 kubenswrapper[2564]: I0723 20:06:18.108495 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-8gqzv_e891cbfb-7697-43dd-9003-3190f8e8a844/kube-rbac-proxy/0.log" Jul 23 20:06:18.131478 ip-10-0-129-2 kubenswrapper[2564]: I0723 20:06:18.131450 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-8gqzv_e891cbfb-7697-43dd-9003-3190f8e8a844/exporter/0.log" Jul 23 20:06:18.155087 ip-10-0-129-2 kubenswrapper[2564]: I0723 20:06:18.155059 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-8gqzv_e891cbfb-7697-43dd-9003-3190f8e8a844/extractor/0.log" Jul 23 20:06:18.319518 ip-10-0-129-2 kubenswrapper[2564]: I0723 20:06:18.319490 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-must-gather-tjlxs/perf-node-gather-daemonset-sr4mn" Jul 23 20:06:20.644490 ip-10-0-129-2 kubenswrapper[2564]: I0723 20:06:20.644401 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_kserve-controller-manager-d6f4b65b8-d6kk8_f6e838ac-b62f-45ee-8a24-937c0ad19b09/manager/0.log" Jul 23 20:06:21.009005 ip-10-0-129-2 kubenswrapper[2564]: I0723 20:06:21.008966 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_llmisvc-controller-manager-777c6bd7d5-7qz4p_bddb241b-f968-4140-86c1-9aeddc995590/manager/0.log" Jul 23 20:06:21.185412 ip-10-0-129-2 kubenswrapper[2564]: I0723 20:06:21.185296 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_model-serving-api-54d89bd5fc-bhjvm_946f6fb8-32c2-406c-819e-8bf5b690971b/server/0.log" Jul 23 20:06:21.303663 ip-10-0-129-2 kubenswrapper[2564]: I0723 20:06:21.303628 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_odh-model-controller-7998c67dc4-52fhl_a88408a9-0854-4017-8248-4fa9c183712d/manager/0.log" Jul 23 20:06:21.491564 ip-10-0-129-2 kubenswrapper[2564]: I0723 20:06:21.491469 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-559f567dc-sc2mg_3deb6cad-ea6c-4be3-95ad-6b4b5fbad490/manager/0.log" Jul 23 20:06:21.513231 ip-10-0-129-2 kubenswrapper[2564]: I0723 20:06:21.513192 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-559f567dc-sc2mg_3deb6cad-ea6c-4be3-95ad-6b4b5fbad490/copy-manifests/0.log" Jul 23 20:06:21.685485 ip-10-0-129-2 kubenswrapper[2564]: I0723 20:06:21.685450 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_istiod-openshift-gateway-94bb8fbfd-9pnfp_c983ed8c-9773-4fc8-b029-fb1892487cc6/discovery/0.log" Jul 23 20:06:21.716584 ip-10-0-129-2 kubenswrapper[2564]: I0723 20:06:21.716545 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-5fdd4658f-lkckb_e20f2247-2981-4d9b-83d2-eeb094ceb1f6/kube-auth-proxy/0.log" Jul 23 20:06:21.737960 ip-10-0-129-2 kubenswrapper[2564]: I0723 20:06:21.737906 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-56654dcf7-pp577_01866086-e44f-4c2d-9f3d-5b5a7cf79029/istio-proxy/0.log" Jul 23 20:06:21.793912 ip-10-0-129-2 kubenswrapper[2564]: I0723 20:06:21.793796 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-56654dcf7-pp577_01866086-e44f-4c2d-9f3d-5b5a7cf79029/istio-proxy/1.log" Jul 23 20:06:21.853649 ip-10-0-129-2 kubenswrapper[2564]: I0723 20:06:21.853601 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-5bf47cd988-rwnsj_45506cd4-c775-4185-888e-607405dd6675/payload-processing/0.log" Jul 23 20:06:21.868780 ip-10-0-129-2 kubenswrapper[2564]: I0723 20:06:21.868748 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_router-default-7f64bc459d-9b6vg_b7c6dedb-e5d3-41a0-b060-50a88ccca303/router/0.log" Jul 23 20:06:23.285567 ip-10-0-129-2 kubenswrapper[2564]: I0723 20:06:23.285532 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_istiod-openshift-gateway-94bb8fbfd-9pnfp_c983ed8c-9773-4fc8-b029-fb1892487cc6/discovery/0.log" Jul 23 20:06:23.311925 ip-10-0-129-2 kubenswrapper[2564]: I0723 20:06:23.311891 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-5fdd4658f-lkckb_e20f2247-2981-4d9b-83d2-eeb094ceb1f6/kube-auth-proxy/0.log" Jul 23 20:06:23.333245 ip-10-0-129-2 kubenswrapper[2564]: I0723 20:06:23.333211 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-56654dcf7-pp577_01866086-e44f-4c2d-9f3d-5b5a7cf79029/istio-proxy/0.log" Jul 23 20:06:23.390093 ip-10-0-129-2 kubenswrapper[2564]: I0723 20:06:23.390046 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-56654dcf7-pp577_01866086-e44f-4c2d-9f3d-5b5a7cf79029/istio-proxy/1.log" Jul 23 20:06:23.455741 ip-10-0-129-2 kubenswrapper[2564]: I0723 20:06:23.455706 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-5bf47cd988-rwnsj_45506cd4-c775-4185-888e-607405dd6675/payload-processing/0.log" Jul 23 20:06:23.469917 ip-10-0-129-2 kubenswrapper[2564]: I0723 20:06:23.469889 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_router-default-7f64bc459d-9b6vg_b7c6dedb-e5d3-41a0-b060-50a88ccca303/router/0.log" Jul 23 20:06:27.793950 ip-10-0-129-2 kubenswrapper[2564]: I0723 20:06:27.793916 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator-operator_kube-storage-version-migrator-operator-7658d9b749-6snc5_a2c386d2-2c3a-4cf0-9a93-e18300e6905d/kube-storage-version-migrator-operator/1.log" Jul 23 20:06:27.795512 ip-10-0-129-2 kubenswrapper[2564]: I0723 20:06:27.795483 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator-operator_kube-storage-version-migrator-operator-7658d9b749-6snc5_a2c386d2-2c3a-4cf0-9a93-e18300e6905d/kube-storage-version-migrator-operator/0.log"