Jul 07 17:38:30.082334 ip-10-0-129-39 systemd[1]: kubelet.service: Failed to load environment files: No such file or directory Jul 07 17:38:30.082347 ip-10-0-129-39 systemd[1]: kubelet.service: Failed to run 'start-pre' task: No such file or directory Jul 07 17:38:30.082357 ip-10-0-129-39 systemd[1]: kubelet.service: Failed with result 'resources'. Jul 07 17:38:30.082625 ip-10-0-129-39 systemd[1]: Failed to start Kubernetes Kubelet. Jul 07 17:38:40.214039 ip-10-0-129-39 systemd[1]: kubelet.service: Failed to schedule restart job: Unit crio.service not found. Jul 07 17:38:40.214055 ip-10-0-129-39 systemd[1]: kubelet.service: Failed with result 'resources'. -- Boot 25b7db8697f84479a590ae0a9c91dfd5 -- Jul 07 17:40:38.272615 ip-10-0-129-39 systemd[1]: Starting Kubernetes Kubelet... Jul 07 17:40:39.828387 ip-10-0-129-39 kubenswrapper[2575]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Jul 07 17:40:39.828387 ip-10-0-129-39 kubenswrapper[2575]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version. Jul 07 17:40:39.828387 ip-10-0-129-39 kubenswrapper[2575]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Jul 07 17:40:39.828387 ip-10-0-129-39 kubenswrapper[2575]: Flag --pod-infra-container-image has been deprecated, will be removed in 1.35. Image garbage collector will get sandbox image information from CRI. Jul 07 17:40:39.828387 ip-10-0-129-39 kubenswrapper[2575]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Jul 07 17:40:39.828387 ip-10-0-129-39 kubenswrapper[2575]: I0707 17:40:39.827601 2575 server.go:213] "--pod-infra-container-image will not be pruned by the image garbage collector in kubelet and should also be set in the remote runtime" Jul 07 17:40:39.842383 ip-10-0-129-39 kubenswrapper[2575]: W0707 17:40:39.833399 2575 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 07 17:40:39.842383 ip-10-0-129-39 kubenswrapper[2575]: W0707 17:40:39.833414 2575 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 07 17:40:39.842383 ip-10-0-129-39 kubenswrapper[2575]: W0707 17:40:39.833418 2575 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 07 17:40:39.842383 ip-10-0-129-39 kubenswrapper[2575]: W0707 17:40:39.833422 2575 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 07 17:40:39.842383 ip-10-0-129-39 kubenswrapper[2575]: W0707 17:40:39.833427 2575 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 07 17:40:39.842383 ip-10-0-129-39 kubenswrapper[2575]: W0707 17:40:39.833430 2575 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 07 17:40:39.842383 ip-10-0-129-39 kubenswrapper[2575]: W0707 17:40:39.833433 2575 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 07 17:40:39.842383 ip-10-0-129-39 kubenswrapper[2575]: W0707 17:40:39.833436 2575 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 07 17:40:39.842383 ip-10-0-129-39 kubenswrapper[2575]: W0707 17:40:39.833439 2575 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 07 17:40:39.842383 ip-10-0-129-39 kubenswrapper[2575]: W0707 17:40:39.833442 2575 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 07 17:40:39.842383 ip-10-0-129-39 kubenswrapper[2575]: W0707 17:40:39.833445 2575 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 07 17:40:39.842383 ip-10-0-129-39 kubenswrapper[2575]: W0707 17:40:39.833448 2575 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 07 17:40:39.842383 ip-10-0-129-39 kubenswrapper[2575]: W0707 17:40:39.833452 2575 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 07 17:40:39.842383 ip-10-0-129-39 kubenswrapper[2575]: W0707 17:40:39.833455 2575 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 07 17:40:39.842383 ip-10-0-129-39 kubenswrapper[2575]: W0707 17:40:39.833457 2575 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 07 17:40:39.842383 ip-10-0-129-39 kubenswrapper[2575]: W0707 17:40:39.833460 2575 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 07 17:40:39.842383 ip-10-0-129-39 kubenswrapper[2575]: W0707 17:40:39.833462 2575 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 07 17:40:39.842383 ip-10-0-129-39 kubenswrapper[2575]: W0707 17:40:39.833465 2575 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 07 17:40:39.842383 ip-10-0-129-39 kubenswrapper[2575]: W0707 17:40:39.833467 2575 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 07 17:40:39.842383 ip-10-0-129-39 kubenswrapper[2575]: W0707 17:40:39.833470 2575 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 07 17:40:39.842892 ip-10-0-129-39 kubenswrapper[2575]: W0707 17:40:39.833472 2575 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 07 17:40:39.842892 ip-10-0-129-39 kubenswrapper[2575]: W0707 17:40:39.833475 2575 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 07 17:40:39.842892 ip-10-0-129-39 kubenswrapper[2575]: W0707 17:40:39.833477 2575 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 07 17:40:39.842892 ip-10-0-129-39 kubenswrapper[2575]: W0707 17:40:39.833480 2575 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 07 17:40:39.842892 ip-10-0-129-39 kubenswrapper[2575]: W0707 17:40:39.833482 2575 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 07 17:40:39.842892 ip-10-0-129-39 kubenswrapper[2575]: W0707 17:40:39.833485 2575 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 07 17:40:39.842892 ip-10-0-129-39 kubenswrapper[2575]: W0707 17:40:39.833487 2575 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 07 17:40:39.842892 ip-10-0-129-39 kubenswrapper[2575]: W0707 17:40:39.833490 2575 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 07 17:40:39.842892 ip-10-0-129-39 kubenswrapper[2575]: W0707 17:40:39.833493 2575 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 07 17:40:39.842892 ip-10-0-129-39 kubenswrapper[2575]: W0707 17:40:39.833495 2575 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 07 17:40:39.842892 ip-10-0-129-39 kubenswrapper[2575]: W0707 17:40:39.833498 2575 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 07 17:40:39.842892 ip-10-0-129-39 kubenswrapper[2575]: W0707 17:40:39.833517 2575 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 07 17:40:39.842892 ip-10-0-129-39 kubenswrapper[2575]: W0707 17:40:39.833520 2575 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 07 17:40:39.842892 ip-10-0-129-39 kubenswrapper[2575]: W0707 17:40:39.833525 2575 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 07 17:40:39.842892 ip-10-0-129-39 kubenswrapper[2575]: W0707 17:40:39.833528 2575 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 07 17:40:39.842892 ip-10-0-129-39 kubenswrapper[2575]: W0707 17:40:39.833530 2575 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 07 17:40:39.842892 ip-10-0-129-39 kubenswrapper[2575]: W0707 17:40:39.833533 2575 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 07 17:40:39.842892 ip-10-0-129-39 kubenswrapper[2575]: W0707 17:40:39.833536 2575 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 07 17:40:39.842892 ip-10-0-129-39 kubenswrapper[2575]: W0707 17:40:39.833538 2575 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 07 17:40:39.842892 ip-10-0-129-39 kubenswrapper[2575]: W0707 17:40:39.833541 2575 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 07 17:40:39.843396 ip-10-0-129-39 kubenswrapper[2575]: W0707 17:40:39.833543 2575 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 07 17:40:39.843396 ip-10-0-129-39 kubenswrapper[2575]: W0707 17:40:39.833546 2575 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 07 17:40:39.843396 ip-10-0-129-39 kubenswrapper[2575]: W0707 17:40:39.833549 2575 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 07 17:40:39.843396 ip-10-0-129-39 kubenswrapper[2575]: W0707 17:40:39.833552 2575 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 07 17:40:39.843396 ip-10-0-129-39 kubenswrapper[2575]: W0707 17:40:39.833554 2575 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 07 17:40:39.843396 ip-10-0-129-39 kubenswrapper[2575]: W0707 17:40:39.833557 2575 feature_gate.go:328] unrecognized feature gate: Example Jul 07 17:40:39.843396 ip-10-0-129-39 kubenswrapper[2575]: W0707 17:40:39.833559 2575 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 07 17:40:39.843396 ip-10-0-129-39 kubenswrapper[2575]: W0707 17:40:39.833562 2575 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 07 17:40:39.843396 ip-10-0-129-39 kubenswrapper[2575]: W0707 17:40:39.833564 2575 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 07 17:40:39.843396 ip-10-0-129-39 kubenswrapper[2575]: W0707 17:40:39.833567 2575 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 07 17:40:39.843396 ip-10-0-129-39 kubenswrapper[2575]: W0707 17:40:39.833569 2575 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 07 17:40:39.843396 ip-10-0-129-39 kubenswrapper[2575]: W0707 17:40:39.833571 2575 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 07 17:40:39.843396 ip-10-0-129-39 kubenswrapper[2575]: W0707 17:40:39.833574 2575 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 07 17:40:39.843396 ip-10-0-129-39 kubenswrapper[2575]: W0707 17:40:39.833576 2575 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 07 17:40:39.843396 ip-10-0-129-39 kubenswrapper[2575]: W0707 17:40:39.833579 2575 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 07 17:40:39.843396 ip-10-0-129-39 kubenswrapper[2575]: W0707 17:40:39.833581 2575 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 07 17:40:39.843396 ip-10-0-129-39 kubenswrapper[2575]: W0707 17:40:39.833584 2575 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 07 17:40:39.843396 ip-10-0-129-39 kubenswrapper[2575]: W0707 17:40:39.833587 2575 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 07 17:40:39.843396 ip-10-0-129-39 kubenswrapper[2575]: W0707 17:40:39.833589 2575 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 07 17:40:39.843396 ip-10-0-129-39 kubenswrapper[2575]: W0707 17:40:39.833592 2575 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 07 17:40:39.843886 ip-10-0-129-39 kubenswrapper[2575]: W0707 17:40:39.833596 2575 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 07 17:40:39.843886 ip-10-0-129-39 kubenswrapper[2575]: W0707 17:40:39.833600 2575 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 07 17:40:39.843886 ip-10-0-129-39 kubenswrapper[2575]: W0707 17:40:39.833602 2575 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 07 17:40:39.843886 ip-10-0-129-39 kubenswrapper[2575]: W0707 17:40:39.833605 2575 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 07 17:40:39.843886 ip-10-0-129-39 kubenswrapper[2575]: W0707 17:40:39.833608 2575 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 07 17:40:39.843886 ip-10-0-129-39 kubenswrapper[2575]: W0707 17:40:39.833612 2575 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 07 17:40:39.843886 ip-10-0-129-39 kubenswrapper[2575]: W0707 17:40:39.833615 2575 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 07 17:40:39.843886 ip-10-0-129-39 kubenswrapper[2575]: W0707 17:40:39.833617 2575 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 07 17:40:39.843886 ip-10-0-129-39 kubenswrapper[2575]: W0707 17:40:39.833620 2575 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 07 17:40:39.843886 ip-10-0-129-39 kubenswrapper[2575]: W0707 17:40:39.833622 2575 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 07 17:40:39.843886 ip-10-0-129-39 kubenswrapper[2575]: W0707 17:40:39.833625 2575 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 07 17:40:39.843886 ip-10-0-129-39 kubenswrapper[2575]: W0707 17:40:39.833627 2575 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 07 17:40:39.843886 ip-10-0-129-39 kubenswrapper[2575]: W0707 17:40:39.833630 2575 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 07 17:40:39.843886 ip-10-0-129-39 kubenswrapper[2575]: W0707 17:40:39.833632 2575 feature_gate.go:328] unrecognized feature gate: Example2 Jul 07 17:40:39.843886 ip-10-0-129-39 kubenswrapper[2575]: W0707 17:40:39.833635 2575 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 07 17:40:39.843886 ip-10-0-129-39 kubenswrapper[2575]: W0707 17:40:39.833639 2575 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 07 17:40:39.843886 ip-10-0-129-39 kubenswrapper[2575]: W0707 17:40:39.833643 2575 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 07 17:40:39.843886 ip-10-0-129-39 kubenswrapper[2575]: W0707 17:40:39.833646 2575 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 07 17:40:39.843886 ip-10-0-129-39 kubenswrapper[2575]: W0707 17:40:39.833648 2575 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 07 17:40:39.844321 ip-10-0-129-39 kubenswrapper[2575]: W0707 17:40:39.833651 2575 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 07 17:40:39.844321 ip-10-0-129-39 kubenswrapper[2575]: W0707 17:40:39.833654 2575 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 07 17:40:39.844321 ip-10-0-129-39 kubenswrapper[2575]: W0707 17:40:39.833656 2575 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 07 17:40:39.844321 ip-10-0-129-39 kubenswrapper[2575]: W0707 17:40:39.833659 2575 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 07 17:40:39.844321 ip-10-0-129-39 kubenswrapper[2575]: W0707 17:40:39.833663 2575 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 07 17:40:39.844321 ip-10-0-129-39 kubenswrapper[2575]: W0707 17:40:39.833666 2575 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 07 17:40:39.844321 ip-10-0-129-39 kubenswrapper[2575]: W0707 17:40:39.833669 2575 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 07 17:40:39.844321 ip-10-0-129-39 kubenswrapper[2575]: W0707 17:40:39.833672 2575 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 07 17:40:39.844321 ip-10-0-129-39 kubenswrapper[2575]: W0707 17:40:39.833674 2575 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 07 17:40:39.844321 ip-10-0-129-39 kubenswrapper[2575]: W0707 17:40:39.833676 2575 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 07 17:40:39.844321 ip-10-0-129-39 kubenswrapper[2575]: W0707 17:40:39.833679 2575 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 07 17:40:39.844321 ip-10-0-129-39 kubenswrapper[2575]: W0707 17:40:39.833681 2575 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 07 17:40:39.844321 ip-10-0-129-39 kubenswrapper[2575]: W0707 17:40:39.833684 2575 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 07 17:40:39.844321 ip-10-0-129-39 kubenswrapper[2575]: W0707 17:40:39.833686 2575 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 07 17:40:39.844321 ip-10-0-129-39 kubenswrapper[2575]: W0707 17:40:39.833689 2575 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 07 17:40:39.844321 ip-10-0-129-39 kubenswrapper[2575]: W0707 17:40:39.833691 2575 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 07 17:40:39.844321 ip-10-0-129-39 kubenswrapper[2575]: W0707 17:40:39.833694 2575 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 07 17:40:39.844321 ip-10-0-129-39 kubenswrapper[2575]: W0707 17:40:39.833697 2575 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 07 17:40:39.844321 ip-10-0-129-39 kubenswrapper[2575]: W0707 17:40:39.833700 2575 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 07 17:40:39.844878 ip-10-0-129-39 kubenswrapper[2575]: W0707 17:40:39.833703 2575 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 07 17:40:39.844878 ip-10-0-129-39 kubenswrapper[2575]: W0707 17:40:39.836851 2575 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 07 17:40:39.844878 ip-10-0-129-39 kubenswrapper[2575]: W0707 17:40:39.836858 2575 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 07 17:40:39.844878 ip-10-0-129-39 kubenswrapper[2575]: W0707 17:40:39.836862 2575 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 07 17:40:39.844878 ip-10-0-129-39 kubenswrapper[2575]: W0707 17:40:39.836867 2575 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 07 17:40:39.844878 ip-10-0-129-39 kubenswrapper[2575]: W0707 17:40:39.836871 2575 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 07 17:40:39.844878 ip-10-0-129-39 kubenswrapper[2575]: W0707 17:40:39.836874 2575 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 07 17:40:39.844878 ip-10-0-129-39 kubenswrapper[2575]: W0707 17:40:39.836877 2575 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 07 17:40:39.844878 ip-10-0-129-39 kubenswrapper[2575]: W0707 17:40:39.836880 2575 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 07 17:40:39.844878 ip-10-0-129-39 kubenswrapper[2575]: W0707 17:40:39.836883 2575 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 07 17:40:39.844878 ip-10-0-129-39 kubenswrapper[2575]: W0707 17:40:39.836886 2575 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 07 17:40:39.844878 ip-10-0-129-39 kubenswrapper[2575]: W0707 17:40:39.836888 2575 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 07 17:40:39.844878 ip-10-0-129-39 kubenswrapper[2575]: W0707 17:40:39.836891 2575 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 07 17:40:39.844878 ip-10-0-129-39 kubenswrapper[2575]: W0707 17:40:39.836895 2575 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 07 17:40:39.844878 ip-10-0-129-39 kubenswrapper[2575]: W0707 17:40:39.836899 2575 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 07 17:40:39.844878 ip-10-0-129-39 kubenswrapper[2575]: W0707 17:40:39.836902 2575 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 07 17:40:39.844878 ip-10-0-129-39 kubenswrapper[2575]: W0707 17:40:39.836904 2575 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 07 17:40:39.844878 ip-10-0-129-39 kubenswrapper[2575]: W0707 17:40:39.836907 2575 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 07 17:40:39.844878 ip-10-0-129-39 kubenswrapper[2575]: W0707 17:40:39.836909 2575 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 07 17:40:39.845372 ip-10-0-129-39 kubenswrapper[2575]: W0707 17:40:39.836912 2575 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 07 17:40:39.845372 ip-10-0-129-39 kubenswrapper[2575]: W0707 17:40:39.836914 2575 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 07 17:40:39.845372 ip-10-0-129-39 kubenswrapper[2575]: W0707 17:40:39.836917 2575 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 07 17:40:39.845372 ip-10-0-129-39 kubenswrapper[2575]: W0707 17:40:39.836919 2575 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 07 17:40:39.845372 ip-10-0-129-39 kubenswrapper[2575]: W0707 17:40:39.836922 2575 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 07 17:40:39.845372 ip-10-0-129-39 kubenswrapper[2575]: W0707 17:40:39.836925 2575 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 07 17:40:39.845372 ip-10-0-129-39 kubenswrapper[2575]: W0707 17:40:39.836928 2575 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 07 17:40:39.845372 ip-10-0-129-39 kubenswrapper[2575]: W0707 17:40:39.836931 2575 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 07 17:40:39.845372 ip-10-0-129-39 kubenswrapper[2575]: W0707 17:40:39.836933 2575 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 07 17:40:39.845372 ip-10-0-129-39 kubenswrapper[2575]: W0707 17:40:39.836936 2575 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 07 17:40:39.845372 ip-10-0-129-39 kubenswrapper[2575]: W0707 17:40:39.836939 2575 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 07 17:40:39.845372 ip-10-0-129-39 kubenswrapper[2575]: W0707 17:40:39.836941 2575 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 07 17:40:39.845372 ip-10-0-129-39 kubenswrapper[2575]: W0707 17:40:39.836944 2575 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 07 17:40:39.845372 ip-10-0-129-39 kubenswrapper[2575]: W0707 17:40:39.836946 2575 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 07 17:40:39.845372 ip-10-0-129-39 kubenswrapper[2575]: W0707 17:40:39.836949 2575 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 07 17:40:39.845372 ip-10-0-129-39 kubenswrapper[2575]: W0707 17:40:39.836951 2575 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 07 17:40:39.845372 ip-10-0-129-39 kubenswrapper[2575]: W0707 17:40:39.836954 2575 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 07 17:40:39.845372 ip-10-0-129-39 kubenswrapper[2575]: W0707 17:40:39.836956 2575 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 07 17:40:39.845372 ip-10-0-129-39 kubenswrapper[2575]: W0707 17:40:39.836959 2575 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 07 17:40:39.846083 ip-10-0-129-39 kubenswrapper[2575]: W0707 17:40:39.836961 2575 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 07 17:40:39.846083 ip-10-0-129-39 kubenswrapper[2575]: W0707 17:40:39.836964 2575 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 07 17:40:39.846083 ip-10-0-129-39 kubenswrapper[2575]: W0707 17:40:39.836966 2575 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 07 17:40:39.846083 ip-10-0-129-39 kubenswrapper[2575]: W0707 17:40:39.836969 2575 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 07 17:40:39.846083 ip-10-0-129-39 kubenswrapper[2575]: W0707 17:40:39.836971 2575 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 07 17:40:39.846083 ip-10-0-129-39 kubenswrapper[2575]: W0707 17:40:39.836977 2575 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 07 17:40:39.846083 ip-10-0-129-39 kubenswrapper[2575]: W0707 17:40:39.836979 2575 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 07 17:40:39.846083 ip-10-0-129-39 kubenswrapper[2575]: W0707 17:40:39.836982 2575 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 07 17:40:39.846083 ip-10-0-129-39 kubenswrapper[2575]: W0707 17:40:39.836984 2575 feature_gate.go:328] unrecognized feature gate: Example2 Jul 07 17:40:39.846083 ip-10-0-129-39 kubenswrapper[2575]: W0707 17:40:39.836987 2575 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 07 17:40:39.846083 ip-10-0-129-39 kubenswrapper[2575]: W0707 17:40:39.836989 2575 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 07 17:40:39.846083 ip-10-0-129-39 kubenswrapper[2575]: W0707 17:40:39.836992 2575 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 07 17:40:39.846083 ip-10-0-129-39 kubenswrapper[2575]: W0707 17:40:39.836995 2575 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 07 17:40:39.846083 ip-10-0-129-39 kubenswrapper[2575]: W0707 17:40:39.836997 2575 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 07 17:40:39.846083 ip-10-0-129-39 kubenswrapper[2575]: W0707 17:40:39.837000 2575 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 07 17:40:39.846083 ip-10-0-129-39 kubenswrapper[2575]: W0707 17:40:39.837002 2575 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 07 17:40:39.846083 ip-10-0-129-39 kubenswrapper[2575]: W0707 17:40:39.837004 2575 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 07 17:40:39.846083 ip-10-0-129-39 kubenswrapper[2575]: W0707 17:40:39.837007 2575 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 07 17:40:39.846083 ip-10-0-129-39 kubenswrapper[2575]: W0707 17:40:39.837010 2575 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 07 17:40:39.846083 ip-10-0-129-39 kubenswrapper[2575]: W0707 17:40:39.837012 2575 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 07 17:40:39.846846 ip-10-0-129-39 kubenswrapper[2575]: W0707 17:40:39.837015 2575 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 07 17:40:39.846846 ip-10-0-129-39 kubenswrapper[2575]: W0707 17:40:39.837017 2575 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 07 17:40:39.846846 ip-10-0-129-39 kubenswrapper[2575]: W0707 17:40:39.837020 2575 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 07 17:40:39.846846 ip-10-0-129-39 kubenswrapper[2575]: W0707 17:40:39.837022 2575 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 07 17:40:39.846846 ip-10-0-129-39 kubenswrapper[2575]: W0707 17:40:39.837027 2575 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 07 17:40:39.846846 ip-10-0-129-39 kubenswrapper[2575]: W0707 17:40:39.837030 2575 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 07 17:40:39.846846 ip-10-0-129-39 kubenswrapper[2575]: W0707 17:40:39.837032 2575 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 07 17:40:39.846846 ip-10-0-129-39 kubenswrapper[2575]: W0707 17:40:39.837035 2575 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 07 17:40:39.846846 ip-10-0-129-39 kubenswrapper[2575]: W0707 17:40:39.837037 2575 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 07 17:40:39.846846 ip-10-0-129-39 kubenswrapper[2575]: W0707 17:40:39.837040 2575 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 07 17:40:39.846846 ip-10-0-129-39 kubenswrapper[2575]: W0707 17:40:39.837042 2575 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 07 17:40:39.846846 ip-10-0-129-39 kubenswrapper[2575]: W0707 17:40:39.837046 2575 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 07 17:40:39.846846 ip-10-0-129-39 kubenswrapper[2575]: W0707 17:40:39.837050 2575 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 07 17:40:39.846846 ip-10-0-129-39 kubenswrapper[2575]: W0707 17:40:39.837053 2575 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 07 17:40:39.846846 ip-10-0-129-39 kubenswrapper[2575]: W0707 17:40:39.837056 2575 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 07 17:40:39.846846 ip-10-0-129-39 kubenswrapper[2575]: W0707 17:40:39.837058 2575 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 07 17:40:39.846846 ip-10-0-129-39 kubenswrapper[2575]: W0707 17:40:39.837061 2575 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 07 17:40:39.846846 ip-10-0-129-39 kubenswrapper[2575]: W0707 17:40:39.837064 2575 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 07 17:40:39.846846 ip-10-0-129-39 kubenswrapper[2575]: W0707 17:40:39.837067 2575 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 07 17:40:39.847301 ip-10-0-129-39 kubenswrapper[2575]: W0707 17:40:39.837071 2575 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 07 17:40:39.847301 ip-10-0-129-39 kubenswrapper[2575]: W0707 17:40:39.837073 2575 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 07 17:40:39.847301 ip-10-0-129-39 kubenswrapper[2575]: W0707 17:40:39.837076 2575 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 07 17:40:39.847301 ip-10-0-129-39 kubenswrapper[2575]: W0707 17:40:39.837079 2575 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 07 17:40:39.847301 ip-10-0-129-39 kubenswrapper[2575]: W0707 17:40:39.837082 2575 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 07 17:40:39.847301 ip-10-0-129-39 kubenswrapper[2575]: W0707 17:40:39.837086 2575 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 07 17:40:39.847301 ip-10-0-129-39 kubenswrapper[2575]: W0707 17:40:39.837088 2575 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 07 17:40:39.847301 ip-10-0-129-39 kubenswrapper[2575]: W0707 17:40:39.837092 2575 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 07 17:40:39.847301 ip-10-0-129-39 kubenswrapper[2575]: W0707 17:40:39.837094 2575 feature_gate.go:328] unrecognized feature gate: Example Jul 07 17:40:39.847301 ip-10-0-129-39 kubenswrapper[2575]: W0707 17:40:39.837097 2575 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 07 17:40:39.847301 ip-10-0-129-39 kubenswrapper[2575]: W0707 17:40:39.837100 2575 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 07 17:40:39.847301 ip-10-0-129-39 kubenswrapper[2575]: W0707 17:40:39.837103 2575 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 07 17:40:39.847301 ip-10-0-129-39 kubenswrapper[2575]: W0707 17:40:39.837105 2575 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 07 17:40:39.847301 ip-10-0-129-39 kubenswrapper[2575]: W0707 17:40:39.837109 2575 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 07 17:40:39.847301 ip-10-0-129-39 kubenswrapper[2575]: W0707 17:40:39.837111 2575 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 07 17:40:39.847301 ip-10-0-129-39 kubenswrapper[2575]: W0707 17:40:39.837114 2575 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 07 17:40:39.847301 ip-10-0-129-39 kubenswrapper[2575]: W0707 17:40:39.837117 2575 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 07 17:40:39.847301 ip-10-0-129-39 kubenswrapper[2575]: W0707 17:40:39.837120 2575 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 07 17:40:39.847301 ip-10-0-129-39 kubenswrapper[2575]: W0707 17:40:39.837123 2575 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 07 17:40:39.847301 ip-10-0-129-39 kubenswrapper[2575]: W0707 17:40:39.837125 2575 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 07 17:40:39.847808 ip-10-0-129-39 kubenswrapper[2575]: W0707 17:40:39.837128 2575 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 07 17:40:39.847808 ip-10-0-129-39 kubenswrapper[2575]: W0707 17:40:39.837130 2575 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 07 17:40:39.847808 ip-10-0-129-39 kubenswrapper[2575]: W0707 17:40:39.837133 2575 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 07 17:40:39.847808 ip-10-0-129-39 kubenswrapper[2575]: W0707 17:40:39.837478 2575 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 07 17:40:39.847808 ip-10-0-129-39 kubenswrapper[2575]: W0707 17:40:39.837483 2575 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 07 17:40:39.847808 ip-10-0-129-39 kubenswrapper[2575]: W0707 17:40:39.837486 2575 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 07 17:40:39.847808 ip-10-0-129-39 kubenswrapper[2575]: W0707 17:40:39.837489 2575 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 07 17:40:39.847808 ip-10-0-129-39 kubenswrapper[2575]: W0707 17:40:39.837491 2575 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 07 17:40:39.847808 ip-10-0-129-39 kubenswrapper[2575]: W0707 17:40:39.837494 2575 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 07 17:40:39.847808 ip-10-0-129-39 kubenswrapper[2575]: W0707 17:40:39.837497 2575 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 07 17:40:39.847808 ip-10-0-129-39 kubenswrapper[2575]: W0707 17:40:39.837515 2575 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 07 17:40:39.847808 ip-10-0-129-39 kubenswrapper[2575]: W0707 17:40:39.837520 2575 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 07 17:40:39.847808 ip-10-0-129-39 kubenswrapper[2575]: W0707 17:40:39.837523 2575 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 07 17:40:39.847808 ip-10-0-129-39 kubenswrapper[2575]: W0707 17:40:39.837525 2575 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 07 17:40:39.847808 ip-10-0-129-39 kubenswrapper[2575]: W0707 17:40:39.837528 2575 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 07 17:40:39.847808 ip-10-0-129-39 kubenswrapper[2575]: W0707 17:40:39.837531 2575 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 07 17:40:39.847808 ip-10-0-129-39 kubenswrapper[2575]: W0707 17:40:39.837533 2575 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 07 17:40:39.847808 ip-10-0-129-39 kubenswrapper[2575]: W0707 17:40:39.837536 2575 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 07 17:40:39.847808 ip-10-0-129-39 kubenswrapper[2575]: W0707 17:40:39.837539 2575 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 07 17:40:39.847808 ip-10-0-129-39 kubenswrapper[2575]: W0707 17:40:39.837541 2575 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 07 17:40:39.848277 ip-10-0-129-39 kubenswrapper[2575]: W0707 17:40:39.837544 2575 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 07 17:40:39.848277 ip-10-0-129-39 kubenswrapper[2575]: W0707 17:40:39.837547 2575 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 07 17:40:39.848277 ip-10-0-129-39 kubenswrapper[2575]: W0707 17:40:39.837549 2575 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 07 17:40:39.848277 ip-10-0-129-39 kubenswrapper[2575]: W0707 17:40:39.837552 2575 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 07 17:40:39.848277 ip-10-0-129-39 kubenswrapper[2575]: W0707 17:40:39.837554 2575 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 07 17:40:39.848277 ip-10-0-129-39 kubenswrapper[2575]: W0707 17:40:39.837557 2575 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 07 17:40:39.848277 ip-10-0-129-39 kubenswrapper[2575]: W0707 17:40:39.837559 2575 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 07 17:40:39.848277 ip-10-0-129-39 kubenswrapper[2575]: W0707 17:40:39.837562 2575 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 07 17:40:39.848277 ip-10-0-129-39 kubenswrapper[2575]: W0707 17:40:39.837565 2575 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 07 17:40:39.848277 ip-10-0-129-39 kubenswrapper[2575]: W0707 17:40:39.837569 2575 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 07 17:40:39.848277 ip-10-0-129-39 kubenswrapper[2575]: W0707 17:40:39.837571 2575 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 07 17:40:39.848277 ip-10-0-129-39 kubenswrapper[2575]: W0707 17:40:39.837574 2575 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 07 17:40:39.848277 ip-10-0-129-39 kubenswrapper[2575]: W0707 17:40:39.837576 2575 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 07 17:40:39.848277 ip-10-0-129-39 kubenswrapper[2575]: W0707 17:40:39.837579 2575 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 07 17:40:39.848277 ip-10-0-129-39 kubenswrapper[2575]: W0707 17:40:39.837581 2575 feature_gate.go:328] unrecognized feature gate: Example2 Jul 07 17:40:39.848277 ip-10-0-129-39 kubenswrapper[2575]: W0707 17:40:39.837584 2575 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 07 17:40:39.848277 ip-10-0-129-39 kubenswrapper[2575]: W0707 17:40:39.837586 2575 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 07 17:40:39.848277 ip-10-0-129-39 kubenswrapper[2575]: W0707 17:40:39.837588 2575 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 07 17:40:39.848277 ip-10-0-129-39 kubenswrapper[2575]: W0707 17:40:39.837591 2575 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 07 17:40:39.848277 ip-10-0-129-39 kubenswrapper[2575]: W0707 17:40:39.837595 2575 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 07 17:40:39.848856 ip-10-0-129-39 kubenswrapper[2575]: W0707 17:40:39.837598 2575 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 07 17:40:39.848856 ip-10-0-129-39 kubenswrapper[2575]: W0707 17:40:39.837600 2575 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 07 17:40:39.848856 ip-10-0-129-39 kubenswrapper[2575]: W0707 17:40:39.837602 2575 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 07 17:40:39.848856 ip-10-0-129-39 kubenswrapper[2575]: W0707 17:40:39.837605 2575 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 07 17:40:39.848856 ip-10-0-129-39 kubenswrapper[2575]: W0707 17:40:39.837608 2575 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 07 17:40:39.848856 ip-10-0-129-39 kubenswrapper[2575]: W0707 17:40:39.837610 2575 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 07 17:40:39.848856 ip-10-0-129-39 kubenswrapper[2575]: W0707 17:40:39.837612 2575 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 07 17:40:39.848856 ip-10-0-129-39 kubenswrapper[2575]: W0707 17:40:39.837615 2575 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 07 17:40:39.848856 ip-10-0-129-39 kubenswrapper[2575]: W0707 17:40:39.837617 2575 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 07 17:40:39.848856 ip-10-0-129-39 kubenswrapper[2575]: W0707 17:40:39.837620 2575 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 07 17:40:39.848856 ip-10-0-129-39 kubenswrapper[2575]: W0707 17:40:39.837622 2575 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 07 17:40:39.848856 ip-10-0-129-39 kubenswrapper[2575]: W0707 17:40:39.837625 2575 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 07 17:40:39.848856 ip-10-0-129-39 kubenswrapper[2575]: W0707 17:40:39.837627 2575 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 07 17:40:39.848856 ip-10-0-129-39 kubenswrapper[2575]: W0707 17:40:39.837631 2575 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 07 17:40:39.848856 ip-10-0-129-39 kubenswrapper[2575]: W0707 17:40:39.837634 2575 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 07 17:40:39.848856 ip-10-0-129-39 kubenswrapper[2575]: W0707 17:40:39.837637 2575 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 07 17:40:39.848856 ip-10-0-129-39 kubenswrapper[2575]: W0707 17:40:39.837639 2575 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 07 17:40:39.848856 ip-10-0-129-39 kubenswrapper[2575]: W0707 17:40:39.837642 2575 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 07 17:40:39.848856 ip-10-0-129-39 kubenswrapper[2575]: W0707 17:40:39.837644 2575 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 07 17:40:39.848856 ip-10-0-129-39 kubenswrapper[2575]: W0707 17:40:39.837647 2575 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 07 17:40:39.849712 ip-10-0-129-39 kubenswrapper[2575]: W0707 17:40:39.837649 2575 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 07 17:40:39.849712 ip-10-0-129-39 kubenswrapper[2575]: W0707 17:40:39.837653 2575 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 07 17:40:39.849712 ip-10-0-129-39 kubenswrapper[2575]: W0707 17:40:39.837655 2575 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 07 17:40:39.849712 ip-10-0-129-39 kubenswrapper[2575]: W0707 17:40:39.837658 2575 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 07 17:40:39.849712 ip-10-0-129-39 kubenswrapper[2575]: W0707 17:40:39.837660 2575 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 07 17:40:39.849712 ip-10-0-129-39 kubenswrapper[2575]: W0707 17:40:39.837663 2575 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 07 17:40:39.849712 ip-10-0-129-39 kubenswrapper[2575]: W0707 17:40:39.837666 2575 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 07 17:40:39.849712 ip-10-0-129-39 kubenswrapper[2575]: W0707 17:40:39.837668 2575 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 07 17:40:39.849712 ip-10-0-129-39 kubenswrapper[2575]: W0707 17:40:39.837671 2575 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 07 17:40:39.849712 ip-10-0-129-39 kubenswrapper[2575]: W0707 17:40:39.837673 2575 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 07 17:40:39.849712 ip-10-0-129-39 kubenswrapper[2575]: W0707 17:40:39.837676 2575 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 07 17:40:39.849712 ip-10-0-129-39 kubenswrapper[2575]: W0707 17:40:39.837679 2575 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 07 17:40:39.849712 ip-10-0-129-39 kubenswrapper[2575]: W0707 17:40:39.837682 2575 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 07 17:40:39.849712 ip-10-0-129-39 kubenswrapper[2575]: W0707 17:40:39.837684 2575 feature_gate.go:328] unrecognized feature gate: Example Jul 07 17:40:39.849712 ip-10-0-129-39 kubenswrapper[2575]: W0707 17:40:39.837687 2575 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 07 17:40:39.849712 ip-10-0-129-39 kubenswrapper[2575]: W0707 17:40:39.837689 2575 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 07 17:40:39.849712 ip-10-0-129-39 kubenswrapper[2575]: W0707 17:40:39.837691 2575 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 07 17:40:39.849712 ip-10-0-129-39 kubenswrapper[2575]: W0707 17:40:39.837694 2575 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 07 17:40:39.849712 ip-10-0-129-39 kubenswrapper[2575]: W0707 17:40:39.837697 2575 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 07 17:40:39.850545 ip-10-0-129-39 kubenswrapper[2575]: W0707 17:40:39.837699 2575 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 07 17:40:39.850545 ip-10-0-129-39 kubenswrapper[2575]: W0707 17:40:39.837702 2575 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 07 17:40:39.850545 ip-10-0-129-39 kubenswrapper[2575]: W0707 17:40:39.837704 2575 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 07 17:40:39.850545 ip-10-0-129-39 kubenswrapper[2575]: W0707 17:40:39.837709 2575 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 07 17:40:39.850545 ip-10-0-129-39 kubenswrapper[2575]: W0707 17:40:39.837712 2575 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 07 17:40:39.850545 ip-10-0-129-39 kubenswrapper[2575]: W0707 17:40:39.837715 2575 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 07 17:40:39.850545 ip-10-0-129-39 kubenswrapper[2575]: W0707 17:40:39.837718 2575 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 07 17:40:39.850545 ip-10-0-129-39 kubenswrapper[2575]: W0707 17:40:39.837720 2575 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 07 17:40:39.850545 ip-10-0-129-39 kubenswrapper[2575]: W0707 17:40:39.837723 2575 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 07 17:40:39.850545 ip-10-0-129-39 kubenswrapper[2575]: W0707 17:40:39.837726 2575 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 07 17:40:39.850545 ip-10-0-129-39 kubenswrapper[2575]: W0707 17:40:39.837728 2575 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 07 17:40:39.850545 ip-10-0-129-39 kubenswrapper[2575]: W0707 17:40:39.837731 2575 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 07 17:40:39.850545 ip-10-0-129-39 kubenswrapper[2575]: W0707 17:40:39.837735 2575 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 07 17:40:39.850545 ip-10-0-129-39 kubenswrapper[2575]: W0707 17:40:39.837738 2575 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 07 17:40:39.850545 ip-10-0-129-39 kubenswrapper[2575]: W0707 17:40:39.837743 2575 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 07 17:40:39.850545 ip-10-0-129-39 kubenswrapper[2575]: W0707 17:40:39.837746 2575 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 07 17:40:39.850545 ip-10-0-129-39 kubenswrapper[2575]: W0707 17:40:39.837749 2575 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 07 17:40:39.850545 ip-10-0-129-39 kubenswrapper[2575]: W0707 17:40:39.837752 2575 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 07 17:40:39.850545 ip-10-0-129-39 kubenswrapper[2575]: W0707 17:40:39.837754 2575 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 07 17:40:39.851183 ip-10-0-129-39 kubenswrapper[2575]: W0707 17:40:39.837757 2575 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 07 17:40:39.851183 ip-10-0-129-39 kubenswrapper[2575]: W0707 17:40:39.837760 2575 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 07 17:40:39.851183 ip-10-0-129-39 kubenswrapper[2575]: W0707 17:40:39.837763 2575 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 07 17:40:39.851183 ip-10-0-129-39 kubenswrapper[2575]: W0707 17:40:39.837765 2575 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 07 17:40:39.851183 ip-10-0-129-39 kubenswrapper[2575]: I0707 17:40:39.837843 2575 flags.go:64] FLAG: --address="0.0.0.0" Jul 07 17:40:39.851183 ip-10-0-129-39 kubenswrapper[2575]: I0707 17:40:39.837854 2575 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]" Jul 07 17:40:39.851183 ip-10-0-129-39 kubenswrapper[2575]: I0707 17:40:39.837863 2575 flags.go:64] FLAG: --anonymous-auth="true" Jul 07 17:40:39.851183 ip-10-0-129-39 kubenswrapper[2575]: I0707 17:40:39.837869 2575 flags.go:64] FLAG: --application-metrics-count-limit="100" Jul 07 17:40:39.851183 ip-10-0-129-39 kubenswrapper[2575]: I0707 17:40:39.837875 2575 flags.go:64] FLAG: --authentication-token-webhook="false" Jul 07 17:40:39.851183 ip-10-0-129-39 kubenswrapper[2575]: I0707 17:40:39.837880 2575 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s" Jul 07 17:40:39.851183 ip-10-0-129-39 kubenswrapper[2575]: I0707 17:40:39.837889 2575 flags.go:64] FLAG: --authorization-mode="AlwaysAllow" Jul 07 17:40:39.851183 ip-10-0-129-39 kubenswrapper[2575]: I0707 17:40:39.837898 2575 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s" Jul 07 17:40:39.851183 ip-10-0-129-39 kubenswrapper[2575]: I0707 17:40:39.837907 2575 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s" Jul 07 17:40:39.851183 ip-10-0-129-39 kubenswrapper[2575]: I0707 17:40:39.837915 2575 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id" Jul 07 17:40:39.851183 ip-10-0-129-39 kubenswrapper[2575]: I0707 17:40:39.837922 2575 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig" Jul 07 17:40:39.851183 ip-10-0-129-39 kubenswrapper[2575]: I0707 17:40:39.837930 2575 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki" Jul 07 17:40:39.851183 ip-10-0-129-39 kubenswrapper[2575]: I0707 17:40:39.837937 2575 flags.go:64] FLAG: --cgroup-driver="cgroupfs" Jul 07 17:40:39.851183 ip-10-0-129-39 kubenswrapper[2575]: I0707 17:40:39.837944 2575 flags.go:64] FLAG: --cgroup-root="" Jul 07 17:40:39.851183 ip-10-0-129-39 kubenswrapper[2575]: I0707 17:40:39.837951 2575 flags.go:64] FLAG: --cgroups-per-qos="true" Jul 07 17:40:39.851183 ip-10-0-129-39 kubenswrapper[2575]: I0707 17:40:39.837959 2575 flags.go:64] FLAG: --client-ca-file="" Jul 07 17:40:39.851183 ip-10-0-129-39 kubenswrapper[2575]: I0707 17:40:39.837966 2575 flags.go:64] FLAG: --cloud-provider="external" Jul 07 17:40:39.851183 ip-10-0-129-39 kubenswrapper[2575]: I0707 17:40:39.837972 2575 flags.go:64] FLAG: --cluster-dns="[]" Jul 07 17:40:39.851183 ip-10-0-129-39 kubenswrapper[2575]: I0707 17:40:39.837981 2575 flags.go:64] FLAG: --cluster-domain="" Jul 07 17:40:39.851183 ip-10-0-129-39 kubenswrapper[2575]: I0707 17:40:39.837988 2575 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf" Jul 07 17:40:39.851878 ip-10-0-129-39 kubenswrapper[2575]: I0707 17:40:39.837995 2575 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d" Jul 07 17:40:39.851878 ip-10-0-129-39 kubenswrapper[2575]: I0707 17:40:39.838003 2575 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json" Jul 07 17:40:39.851878 ip-10-0-129-39 kubenswrapper[2575]: I0707 17:40:39.838010 2575 flags.go:64] FLAG: --container-log-max-files="5" Jul 07 17:40:39.851878 ip-10-0-129-39 kubenswrapper[2575]: I0707 17:40:39.838022 2575 flags.go:64] FLAG: --container-log-max-size="10Mi" Jul 07 17:40:39.851878 ip-10-0-129-39 kubenswrapper[2575]: I0707 17:40:39.838029 2575 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock" Jul 07 17:40:39.851878 ip-10-0-129-39 kubenswrapper[2575]: I0707 17:40:39.838037 2575 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock" Jul 07 17:40:39.851878 ip-10-0-129-39 kubenswrapper[2575]: I0707 17:40:39.838045 2575 flags.go:64] FLAG: --containerd-namespace="k8s.io" Jul 07 17:40:39.851878 ip-10-0-129-39 kubenswrapper[2575]: I0707 17:40:39.838052 2575 flags.go:64] FLAG: --contention-profiling="false" Jul 07 17:40:39.851878 ip-10-0-129-39 kubenswrapper[2575]: I0707 17:40:39.838059 2575 flags.go:64] FLAG: --cpu-cfs-quota="true" Jul 07 17:40:39.851878 ip-10-0-129-39 kubenswrapper[2575]: I0707 17:40:39.838066 2575 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms" Jul 07 17:40:39.851878 ip-10-0-129-39 kubenswrapper[2575]: I0707 17:40:39.838074 2575 flags.go:64] FLAG: --cpu-manager-policy="none" Jul 07 17:40:39.851878 ip-10-0-129-39 kubenswrapper[2575]: I0707 17:40:39.838081 2575 flags.go:64] FLAG: --cpu-manager-policy-options="" Jul 07 17:40:39.851878 ip-10-0-129-39 kubenswrapper[2575]: I0707 17:40:39.838091 2575 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s" Jul 07 17:40:39.851878 ip-10-0-129-39 kubenswrapper[2575]: I0707 17:40:39.838104 2575 flags.go:64] FLAG: --enable-controller-attach-detach="true" Jul 07 17:40:39.851878 ip-10-0-129-39 kubenswrapper[2575]: I0707 17:40:39.838111 2575 flags.go:64] FLAG: --enable-debugging-handlers="true" Jul 07 17:40:39.851878 ip-10-0-129-39 kubenswrapper[2575]: I0707 17:40:39.838119 2575 flags.go:64] FLAG: --enable-load-reader="false" Jul 07 17:40:39.851878 ip-10-0-129-39 kubenswrapper[2575]: I0707 17:40:39.838126 2575 flags.go:64] FLAG: --enable-server="true" Jul 07 17:40:39.851878 ip-10-0-129-39 kubenswrapper[2575]: I0707 17:40:39.838134 2575 flags.go:64] FLAG: --enforce-node-allocatable="[pods]" Jul 07 17:40:39.851878 ip-10-0-129-39 kubenswrapper[2575]: I0707 17:40:39.838144 2575 flags.go:64] FLAG: --event-burst="100" Jul 07 17:40:39.851878 ip-10-0-129-39 kubenswrapper[2575]: I0707 17:40:39.838152 2575 flags.go:64] FLAG: --event-qps="50" Jul 07 17:40:39.851878 ip-10-0-129-39 kubenswrapper[2575]: I0707 17:40:39.838159 2575 flags.go:64] FLAG: --event-storage-age-limit="default=0" Jul 07 17:40:39.851878 ip-10-0-129-39 kubenswrapper[2575]: I0707 17:40:39.838167 2575 flags.go:64] FLAG: --event-storage-event-limit="default=0" Jul 07 17:40:39.851878 ip-10-0-129-39 kubenswrapper[2575]: I0707 17:40:39.838175 2575 flags.go:64] FLAG: --eviction-hard="" Jul 07 17:40:39.851878 ip-10-0-129-39 kubenswrapper[2575]: I0707 17:40:39.838183 2575 flags.go:64] FLAG: --eviction-max-pod-grace-period="0" Jul 07 17:40:39.852610 ip-10-0-129-39 kubenswrapper[2575]: I0707 17:40:39.838190 2575 flags.go:64] FLAG: --eviction-minimum-reclaim="" Jul 07 17:40:39.852610 ip-10-0-129-39 kubenswrapper[2575]: I0707 17:40:39.838197 2575 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s" Jul 07 17:40:39.852610 ip-10-0-129-39 kubenswrapper[2575]: I0707 17:40:39.838205 2575 flags.go:64] FLAG: --eviction-soft="" Jul 07 17:40:39.852610 ip-10-0-129-39 kubenswrapper[2575]: I0707 17:40:39.838213 2575 flags.go:64] FLAG: --eviction-soft-grace-period="" Jul 07 17:40:39.852610 ip-10-0-129-39 kubenswrapper[2575]: I0707 17:40:39.838219 2575 flags.go:64] FLAG: --exit-on-lock-contention="false" Jul 07 17:40:39.852610 ip-10-0-129-39 kubenswrapper[2575]: I0707 17:40:39.838226 2575 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false" Jul 07 17:40:39.852610 ip-10-0-129-39 kubenswrapper[2575]: I0707 17:40:39.838234 2575 flags.go:64] FLAG: --experimental-mounter-path="" Jul 07 17:40:39.852610 ip-10-0-129-39 kubenswrapper[2575]: I0707 17:40:39.838241 2575 flags.go:64] FLAG: --fail-cgroupv1="false" Jul 07 17:40:39.852610 ip-10-0-129-39 kubenswrapper[2575]: I0707 17:40:39.838249 2575 flags.go:64] FLAG: --fail-swap-on="true" Jul 07 17:40:39.852610 ip-10-0-129-39 kubenswrapper[2575]: I0707 17:40:39.838257 2575 flags.go:64] FLAG: --feature-gates="" Jul 07 17:40:39.852610 ip-10-0-129-39 kubenswrapper[2575]: I0707 17:40:39.838265 2575 flags.go:64] FLAG: --file-check-frequency="20s" Jul 07 17:40:39.852610 ip-10-0-129-39 kubenswrapper[2575]: I0707 17:40:39.838278 2575 flags.go:64] FLAG: --global-housekeeping-interval="1m0s" Jul 07 17:40:39.852610 ip-10-0-129-39 kubenswrapper[2575]: I0707 17:40:39.838285 2575 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge" Jul 07 17:40:39.852610 ip-10-0-129-39 kubenswrapper[2575]: I0707 17:40:39.838294 2575 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1" Jul 07 17:40:39.852610 ip-10-0-129-39 kubenswrapper[2575]: I0707 17:40:39.838301 2575 flags.go:64] FLAG: --healthz-port="10248" Jul 07 17:40:39.852610 ip-10-0-129-39 kubenswrapper[2575]: I0707 17:40:39.838309 2575 flags.go:64] FLAG: --help="false" Jul 07 17:40:39.852610 ip-10-0-129-39 kubenswrapper[2575]: I0707 17:40:39.838315 2575 flags.go:64] FLAG: --hostname-override="ip-10-0-129-39.ec2.internal" Jul 07 17:40:39.852610 ip-10-0-129-39 kubenswrapper[2575]: I0707 17:40:39.838320 2575 flags.go:64] FLAG: --housekeeping-interval="10s" Jul 07 17:40:39.852610 ip-10-0-129-39 kubenswrapper[2575]: I0707 17:40:39.838325 2575 flags.go:64] FLAG: --http-check-frequency="20s" Jul 07 17:40:39.852610 ip-10-0-129-39 kubenswrapper[2575]: I0707 17:40:39.838328 2575 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins" Jul 07 17:40:39.852610 ip-10-0-129-39 kubenswrapper[2575]: I0707 17:40:39.838331 2575 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml" Jul 07 17:40:39.852610 ip-10-0-129-39 kubenswrapper[2575]: I0707 17:40:39.838336 2575 flags.go:64] FLAG: --image-gc-high-threshold="85" Jul 07 17:40:39.852610 ip-10-0-129-39 kubenswrapper[2575]: I0707 17:40:39.838339 2575 flags.go:64] FLAG: --image-gc-low-threshold="80" Jul 07 17:40:39.852610 ip-10-0-129-39 kubenswrapper[2575]: I0707 17:40:39.838342 2575 flags.go:64] FLAG: --image-service-endpoint="" Jul 07 17:40:39.853360 ip-10-0-129-39 kubenswrapper[2575]: I0707 17:40:39.838345 2575 flags.go:64] FLAG: --kernel-memcg-notification="false" Jul 07 17:40:39.853360 ip-10-0-129-39 kubenswrapper[2575]: I0707 17:40:39.838348 2575 flags.go:64] FLAG: --kube-api-burst="100" Jul 07 17:40:39.853360 ip-10-0-129-39 kubenswrapper[2575]: I0707 17:40:39.838351 2575 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf" Jul 07 17:40:39.853360 ip-10-0-129-39 kubenswrapper[2575]: I0707 17:40:39.838354 2575 flags.go:64] FLAG: --kube-api-qps="50" Jul 07 17:40:39.853360 ip-10-0-129-39 kubenswrapper[2575]: I0707 17:40:39.838357 2575 flags.go:64] FLAG: --kube-reserved="" Jul 07 17:40:39.853360 ip-10-0-129-39 kubenswrapper[2575]: I0707 17:40:39.838360 2575 flags.go:64] FLAG: --kube-reserved-cgroup="" Jul 07 17:40:39.853360 ip-10-0-129-39 kubenswrapper[2575]: I0707 17:40:39.838363 2575 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig" Jul 07 17:40:39.853360 ip-10-0-129-39 kubenswrapper[2575]: I0707 17:40:39.838366 2575 flags.go:64] FLAG: --kubelet-cgroups="" Jul 07 17:40:39.853360 ip-10-0-129-39 kubenswrapper[2575]: I0707 17:40:39.838369 2575 flags.go:64] FLAG: --local-storage-capacity-isolation="true" Jul 07 17:40:39.853360 ip-10-0-129-39 kubenswrapper[2575]: I0707 17:40:39.838372 2575 flags.go:64] FLAG: --lock-file="" Jul 07 17:40:39.853360 ip-10-0-129-39 kubenswrapper[2575]: I0707 17:40:39.838375 2575 flags.go:64] FLAG: --log-cadvisor-usage="false" Jul 07 17:40:39.853360 ip-10-0-129-39 kubenswrapper[2575]: I0707 17:40:39.838377 2575 flags.go:64] FLAG: --log-flush-frequency="5s" Jul 07 17:40:39.853360 ip-10-0-129-39 kubenswrapper[2575]: I0707 17:40:39.838380 2575 flags.go:64] FLAG: --log-json-info-buffer-size="0" Jul 07 17:40:39.853360 ip-10-0-129-39 kubenswrapper[2575]: I0707 17:40:39.838385 2575 flags.go:64] FLAG: --log-json-split-stream="false" Jul 07 17:40:39.853360 ip-10-0-129-39 kubenswrapper[2575]: I0707 17:40:39.838388 2575 flags.go:64] FLAG: --log-text-info-buffer-size="0" Jul 07 17:40:39.853360 ip-10-0-129-39 kubenswrapper[2575]: I0707 17:40:39.838391 2575 flags.go:64] FLAG: --log-text-split-stream="false" Jul 07 17:40:39.853360 ip-10-0-129-39 kubenswrapper[2575]: I0707 17:40:39.838394 2575 flags.go:64] FLAG: --logging-format="text" Jul 07 17:40:39.853360 ip-10-0-129-39 kubenswrapper[2575]: I0707 17:40:39.838397 2575 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id" Jul 07 17:40:39.853360 ip-10-0-129-39 kubenswrapper[2575]: I0707 17:40:39.838400 2575 flags.go:64] FLAG: --make-iptables-util-chains="true" Jul 07 17:40:39.853360 ip-10-0-129-39 kubenswrapper[2575]: I0707 17:40:39.838404 2575 flags.go:64] FLAG: --manifest-url="" Jul 07 17:40:39.853360 ip-10-0-129-39 kubenswrapper[2575]: I0707 17:40:39.838407 2575 flags.go:64] FLAG: --manifest-url-header="" Jul 07 17:40:39.853360 ip-10-0-129-39 kubenswrapper[2575]: I0707 17:40:39.838411 2575 flags.go:64] FLAG: --max-housekeeping-interval="15s" Jul 07 17:40:39.853360 ip-10-0-129-39 kubenswrapper[2575]: I0707 17:40:39.838414 2575 flags.go:64] FLAG: --max-open-files="1000000" Jul 07 17:40:39.853360 ip-10-0-129-39 kubenswrapper[2575]: I0707 17:40:39.838418 2575 flags.go:64] FLAG: --max-pods="110" Jul 07 17:40:39.853360 ip-10-0-129-39 kubenswrapper[2575]: I0707 17:40:39.838421 2575 flags.go:64] FLAG: --maximum-dead-containers="-1" Jul 07 17:40:39.854091 ip-10-0-129-39 kubenswrapper[2575]: I0707 17:40:39.838424 2575 flags.go:64] FLAG: --maximum-dead-containers-per-container="1" Jul 07 17:40:39.854091 ip-10-0-129-39 kubenswrapper[2575]: I0707 17:40:39.838427 2575 flags.go:64] FLAG: --memory-manager-policy="None" Jul 07 17:40:39.854091 ip-10-0-129-39 kubenswrapper[2575]: I0707 17:40:39.838429 2575 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s" Jul 07 17:40:39.854091 ip-10-0-129-39 kubenswrapper[2575]: I0707 17:40:39.838432 2575 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s" Jul 07 17:40:39.854091 ip-10-0-129-39 kubenswrapper[2575]: I0707 17:40:39.838436 2575 flags.go:64] FLAG: --node-ip="0.0.0.0" Jul 07 17:40:39.854091 ip-10-0-129-39 kubenswrapper[2575]: I0707 17:40:39.838439 2575 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel" Jul 07 17:40:39.854091 ip-10-0-129-39 kubenswrapper[2575]: I0707 17:40:39.838446 2575 flags.go:64] FLAG: --node-status-max-images="50" Jul 07 17:40:39.854091 ip-10-0-129-39 kubenswrapper[2575]: I0707 17:40:39.838449 2575 flags.go:64] FLAG: --node-status-update-frequency="10s" Jul 07 17:40:39.854091 ip-10-0-129-39 kubenswrapper[2575]: I0707 17:40:39.838452 2575 flags.go:64] FLAG: --oom-score-adj="-999" Jul 07 17:40:39.854091 ip-10-0-129-39 kubenswrapper[2575]: I0707 17:40:39.838455 2575 flags.go:64] FLAG: --pod-cidr="" Jul 07 17:40:39.854091 ip-10-0-129-39 kubenswrapper[2575]: I0707 17:40:39.838458 2575 flags.go:64] FLAG: --pod-infra-container-image="quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:ce93f6b4197b9c62a26a3131f8057ffe9c9379c5079a4f62e5919b75151394e5" Jul 07 17:40:39.854091 ip-10-0-129-39 kubenswrapper[2575]: I0707 17:40:39.838463 2575 flags.go:64] FLAG: --pod-manifest-path="" Jul 07 17:40:39.854091 ip-10-0-129-39 kubenswrapper[2575]: I0707 17:40:39.838466 2575 flags.go:64] FLAG: --pod-max-pids="-1" Jul 07 17:40:39.854091 ip-10-0-129-39 kubenswrapper[2575]: I0707 17:40:39.838469 2575 flags.go:64] FLAG: --pods-per-core="0" Jul 07 17:40:39.854091 ip-10-0-129-39 kubenswrapper[2575]: I0707 17:40:39.838473 2575 flags.go:64] FLAG: --port="10250" Jul 07 17:40:39.854091 ip-10-0-129-39 kubenswrapper[2575]: I0707 17:40:39.838476 2575 flags.go:64] FLAG: --protect-kernel-defaults="false" Jul 07 17:40:39.854091 ip-10-0-129-39 kubenswrapper[2575]: I0707 17:40:39.838479 2575 flags.go:64] FLAG: --provider-id="aws:///us-east-1a/i-09cee8e3efa485774" Jul 07 17:40:39.854091 ip-10-0-129-39 kubenswrapper[2575]: I0707 17:40:39.838482 2575 flags.go:64] FLAG: --qos-reserved="" Jul 07 17:40:39.854091 ip-10-0-129-39 kubenswrapper[2575]: I0707 17:40:39.838485 2575 flags.go:64] FLAG: --read-only-port="10255" Jul 07 17:40:39.854091 ip-10-0-129-39 kubenswrapper[2575]: I0707 17:40:39.838488 2575 flags.go:64] FLAG: --register-node="true" Jul 07 17:40:39.854091 ip-10-0-129-39 kubenswrapper[2575]: I0707 17:40:39.838491 2575 flags.go:64] FLAG: --register-with-taints="" Jul 07 17:40:39.854091 ip-10-0-129-39 kubenswrapper[2575]: I0707 17:40:39.838494 2575 flags.go:64] FLAG: --registry-burst="10" Jul 07 17:40:39.854091 ip-10-0-129-39 kubenswrapper[2575]: I0707 17:40:39.838497 2575 flags.go:64] FLAG: --registry-qps="5" Jul 07 17:40:39.854091 ip-10-0-129-39 kubenswrapper[2575]: I0707 17:40:39.838516 2575 flags.go:64] FLAG: --reserved-cpus="" Jul 07 17:40:39.854789 ip-10-0-129-39 kubenswrapper[2575]: I0707 17:40:39.838520 2575 flags.go:64] FLAG: --reserved-memory="" Jul 07 17:40:39.854789 ip-10-0-129-39 kubenswrapper[2575]: I0707 17:40:39.838524 2575 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf" Jul 07 17:40:39.854789 ip-10-0-129-39 kubenswrapper[2575]: I0707 17:40:39.838527 2575 flags.go:64] FLAG: --root-dir="/var/lib/kubelet" Jul 07 17:40:39.854789 ip-10-0-129-39 kubenswrapper[2575]: I0707 17:40:39.838531 2575 flags.go:64] FLAG: --rotate-certificates="false" Jul 07 17:40:39.854789 ip-10-0-129-39 kubenswrapper[2575]: I0707 17:40:39.838534 2575 flags.go:64] FLAG: --rotate-server-certificates="false" Jul 07 17:40:39.854789 ip-10-0-129-39 kubenswrapper[2575]: I0707 17:40:39.838537 2575 flags.go:64] FLAG: --runonce="false" Jul 07 17:40:39.854789 ip-10-0-129-39 kubenswrapper[2575]: I0707 17:40:39.838540 2575 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service" Jul 07 17:40:39.854789 ip-10-0-129-39 kubenswrapper[2575]: I0707 17:40:39.838543 2575 flags.go:64] FLAG: --runtime-request-timeout="2m0s" Jul 07 17:40:39.854789 ip-10-0-129-39 kubenswrapper[2575]: I0707 17:40:39.838546 2575 flags.go:64] FLAG: --seccomp-default="false" Jul 07 17:40:39.854789 ip-10-0-129-39 kubenswrapper[2575]: I0707 17:40:39.838549 2575 flags.go:64] FLAG: --serialize-image-pulls="true" Jul 07 17:40:39.854789 ip-10-0-129-39 kubenswrapper[2575]: I0707 17:40:39.838551 2575 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s" Jul 07 17:40:39.854789 ip-10-0-129-39 kubenswrapper[2575]: I0707 17:40:39.838554 2575 flags.go:64] FLAG: --storage-driver-db="cadvisor" Jul 07 17:40:39.854789 ip-10-0-129-39 kubenswrapper[2575]: I0707 17:40:39.838559 2575 flags.go:64] FLAG: --storage-driver-host="localhost:8086" Jul 07 17:40:39.854789 ip-10-0-129-39 kubenswrapper[2575]: I0707 17:40:39.838562 2575 flags.go:64] FLAG: --storage-driver-password="root" Jul 07 17:40:39.854789 ip-10-0-129-39 kubenswrapper[2575]: I0707 17:40:39.838565 2575 flags.go:64] FLAG: --storage-driver-secure="false" Jul 07 17:40:39.854789 ip-10-0-129-39 kubenswrapper[2575]: I0707 17:40:39.838568 2575 flags.go:64] FLAG: --storage-driver-table="stats" Jul 07 17:40:39.854789 ip-10-0-129-39 kubenswrapper[2575]: I0707 17:40:39.838571 2575 flags.go:64] FLAG: --storage-driver-user="root" Jul 07 17:40:39.854789 ip-10-0-129-39 kubenswrapper[2575]: I0707 17:40:39.838573 2575 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s" Jul 07 17:40:39.854789 ip-10-0-129-39 kubenswrapper[2575]: I0707 17:40:39.838576 2575 flags.go:64] FLAG: --sync-frequency="1m0s" Jul 07 17:40:39.854789 ip-10-0-129-39 kubenswrapper[2575]: I0707 17:40:39.838579 2575 flags.go:64] FLAG: --system-cgroups="" Jul 07 17:40:39.854789 ip-10-0-129-39 kubenswrapper[2575]: I0707 17:40:39.838582 2575 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi" Jul 07 17:40:39.854789 ip-10-0-129-39 kubenswrapper[2575]: I0707 17:40:39.838587 2575 flags.go:64] FLAG: --system-reserved-cgroup="" Jul 07 17:40:39.854789 ip-10-0-129-39 kubenswrapper[2575]: I0707 17:40:39.838590 2575 flags.go:64] FLAG: --tls-cert-file="" Jul 07 17:40:39.854789 ip-10-0-129-39 kubenswrapper[2575]: I0707 17:40:39.838593 2575 flags.go:64] FLAG: --tls-cipher-suites="[]" Jul 07 17:40:39.854789 ip-10-0-129-39 kubenswrapper[2575]: I0707 17:40:39.838597 2575 flags.go:64] FLAG: --tls-min-version="" Jul 07 17:40:39.855497 ip-10-0-129-39 kubenswrapper[2575]: I0707 17:40:39.838600 2575 flags.go:64] FLAG: --tls-private-key-file="" Jul 07 17:40:39.855497 ip-10-0-129-39 kubenswrapper[2575]: I0707 17:40:39.838603 2575 flags.go:64] FLAG: --topology-manager-policy="none" Jul 07 17:40:39.855497 ip-10-0-129-39 kubenswrapper[2575]: I0707 17:40:39.838605 2575 flags.go:64] FLAG: --topology-manager-policy-options="" Jul 07 17:40:39.855497 ip-10-0-129-39 kubenswrapper[2575]: I0707 17:40:39.838608 2575 flags.go:64] FLAG: --topology-manager-scope="container" Jul 07 17:40:39.855497 ip-10-0-129-39 kubenswrapper[2575]: I0707 17:40:39.838611 2575 flags.go:64] FLAG: --v="2" Jul 07 17:40:39.855497 ip-10-0-129-39 kubenswrapper[2575]: I0707 17:40:39.838615 2575 flags.go:64] FLAG: --version="false" Jul 07 17:40:39.855497 ip-10-0-129-39 kubenswrapper[2575]: I0707 17:40:39.838619 2575 flags.go:64] FLAG: --vmodule="" Jul 07 17:40:39.855497 ip-10-0-129-39 kubenswrapper[2575]: I0707 17:40:39.838623 2575 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec" Jul 07 17:40:39.855497 ip-10-0-129-39 kubenswrapper[2575]: I0707 17:40:39.838626 2575 flags.go:64] FLAG: --volume-stats-agg-period="1m0s" Jul 07 17:40:39.855497 ip-10-0-129-39 kubenswrapper[2575]: W0707 17:40:39.838742 2575 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 07 17:40:39.855497 ip-10-0-129-39 kubenswrapper[2575]: W0707 17:40:39.838748 2575 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 07 17:40:39.855497 ip-10-0-129-39 kubenswrapper[2575]: W0707 17:40:39.838751 2575 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 07 17:40:39.855497 ip-10-0-129-39 kubenswrapper[2575]: W0707 17:40:39.838754 2575 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 07 17:40:39.855497 ip-10-0-129-39 kubenswrapper[2575]: W0707 17:40:39.838758 2575 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 07 17:40:39.855497 ip-10-0-129-39 kubenswrapper[2575]: W0707 17:40:39.838761 2575 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 07 17:40:39.855497 ip-10-0-129-39 kubenswrapper[2575]: W0707 17:40:39.838764 2575 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 07 17:40:39.855497 ip-10-0-129-39 kubenswrapper[2575]: W0707 17:40:39.838767 2575 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 07 17:40:39.855497 ip-10-0-129-39 kubenswrapper[2575]: W0707 17:40:39.838770 2575 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 07 17:40:39.855497 ip-10-0-129-39 kubenswrapper[2575]: W0707 17:40:39.838773 2575 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 07 17:40:39.855497 ip-10-0-129-39 kubenswrapper[2575]: W0707 17:40:39.838777 2575 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 07 17:40:39.855497 ip-10-0-129-39 kubenswrapper[2575]: W0707 17:40:39.838779 2575 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 07 17:40:39.856114 ip-10-0-129-39 kubenswrapper[2575]: W0707 17:40:39.838782 2575 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 07 17:40:39.856114 ip-10-0-129-39 kubenswrapper[2575]: W0707 17:40:39.838785 2575 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 07 17:40:39.856114 ip-10-0-129-39 kubenswrapper[2575]: W0707 17:40:39.838787 2575 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 07 17:40:39.856114 ip-10-0-129-39 kubenswrapper[2575]: W0707 17:40:39.838790 2575 feature_gate.go:328] unrecognized feature gate: Example2 Jul 07 17:40:39.856114 ip-10-0-129-39 kubenswrapper[2575]: W0707 17:40:39.838792 2575 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 07 17:40:39.856114 ip-10-0-129-39 kubenswrapper[2575]: W0707 17:40:39.838795 2575 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 07 17:40:39.856114 ip-10-0-129-39 kubenswrapper[2575]: W0707 17:40:39.838797 2575 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 07 17:40:39.856114 ip-10-0-129-39 kubenswrapper[2575]: W0707 17:40:39.838800 2575 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 07 17:40:39.856114 ip-10-0-129-39 kubenswrapper[2575]: W0707 17:40:39.838802 2575 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 07 17:40:39.856114 ip-10-0-129-39 kubenswrapper[2575]: W0707 17:40:39.838805 2575 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 07 17:40:39.856114 ip-10-0-129-39 kubenswrapper[2575]: W0707 17:40:39.838808 2575 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 07 17:40:39.856114 ip-10-0-129-39 kubenswrapper[2575]: W0707 17:40:39.838810 2575 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 07 17:40:39.856114 ip-10-0-129-39 kubenswrapper[2575]: W0707 17:40:39.838813 2575 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 07 17:40:39.856114 ip-10-0-129-39 kubenswrapper[2575]: W0707 17:40:39.838815 2575 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 07 17:40:39.856114 ip-10-0-129-39 kubenswrapper[2575]: W0707 17:40:39.838818 2575 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 07 17:40:39.856114 ip-10-0-129-39 kubenswrapper[2575]: W0707 17:40:39.838820 2575 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 07 17:40:39.856114 ip-10-0-129-39 kubenswrapper[2575]: W0707 17:40:39.838823 2575 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 07 17:40:39.856114 ip-10-0-129-39 kubenswrapper[2575]: W0707 17:40:39.838826 2575 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 07 17:40:39.856114 ip-10-0-129-39 kubenswrapper[2575]: W0707 17:40:39.838828 2575 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 07 17:40:39.856764 ip-10-0-129-39 kubenswrapper[2575]: W0707 17:40:39.838831 2575 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 07 17:40:39.856764 ip-10-0-129-39 kubenswrapper[2575]: W0707 17:40:39.838833 2575 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 07 17:40:39.856764 ip-10-0-129-39 kubenswrapper[2575]: W0707 17:40:39.838837 2575 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 07 17:40:39.856764 ip-10-0-129-39 kubenswrapper[2575]: W0707 17:40:39.838841 2575 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 07 17:40:39.856764 ip-10-0-129-39 kubenswrapper[2575]: W0707 17:40:39.838844 2575 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 07 17:40:39.856764 ip-10-0-129-39 kubenswrapper[2575]: W0707 17:40:39.838847 2575 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 07 17:40:39.856764 ip-10-0-129-39 kubenswrapper[2575]: W0707 17:40:39.838850 2575 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 07 17:40:39.856764 ip-10-0-129-39 kubenswrapper[2575]: W0707 17:40:39.838852 2575 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 07 17:40:39.856764 ip-10-0-129-39 kubenswrapper[2575]: W0707 17:40:39.838855 2575 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 07 17:40:39.856764 ip-10-0-129-39 kubenswrapper[2575]: W0707 17:40:39.838857 2575 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 07 17:40:39.856764 ip-10-0-129-39 kubenswrapper[2575]: W0707 17:40:39.838860 2575 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 07 17:40:39.856764 ip-10-0-129-39 kubenswrapper[2575]: W0707 17:40:39.838863 2575 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 07 17:40:39.856764 ip-10-0-129-39 kubenswrapper[2575]: W0707 17:40:39.838866 2575 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 07 17:40:39.856764 ip-10-0-129-39 kubenswrapper[2575]: W0707 17:40:39.838868 2575 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 07 17:40:39.856764 ip-10-0-129-39 kubenswrapper[2575]: W0707 17:40:39.838871 2575 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 07 17:40:39.856764 ip-10-0-129-39 kubenswrapper[2575]: W0707 17:40:39.838873 2575 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 07 17:40:39.856764 ip-10-0-129-39 kubenswrapper[2575]: W0707 17:40:39.838875 2575 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 07 17:40:39.856764 ip-10-0-129-39 kubenswrapper[2575]: W0707 17:40:39.838878 2575 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 07 17:40:39.856764 ip-10-0-129-39 kubenswrapper[2575]: W0707 17:40:39.838880 2575 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 07 17:40:39.857339 ip-10-0-129-39 kubenswrapper[2575]: W0707 17:40:39.838883 2575 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 07 17:40:39.857339 ip-10-0-129-39 kubenswrapper[2575]: W0707 17:40:39.838885 2575 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 07 17:40:39.857339 ip-10-0-129-39 kubenswrapper[2575]: W0707 17:40:39.838888 2575 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 07 17:40:39.857339 ip-10-0-129-39 kubenswrapper[2575]: W0707 17:40:39.838890 2575 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 07 17:40:39.857339 ip-10-0-129-39 kubenswrapper[2575]: W0707 17:40:39.838893 2575 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 07 17:40:39.857339 ip-10-0-129-39 kubenswrapper[2575]: W0707 17:40:39.838896 2575 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 07 17:40:39.857339 ip-10-0-129-39 kubenswrapper[2575]: W0707 17:40:39.838898 2575 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 07 17:40:39.857339 ip-10-0-129-39 kubenswrapper[2575]: W0707 17:40:39.838900 2575 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 07 17:40:39.857339 ip-10-0-129-39 kubenswrapper[2575]: W0707 17:40:39.838903 2575 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 07 17:40:39.857339 ip-10-0-129-39 kubenswrapper[2575]: W0707 17:40:39.838905 2575 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 07 17:40:39.857339 ip-10-0-129-39 kubenswrapper[2575]: W0707 17:40:39.838911 2575 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 07 17:40:39.857339 ip-10-0-129-39 kubenswrapper[2575]: W0707 17:40:39.838914 2575 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 07 17:40:39.857339 ip-10-0-129-39 kubenswrapper[2575]: W0707 17:40:39.838916 2575 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 07 17:40:39.857339 ip-10-0-129-39 kubenswrapper[2575]: W0707 17:40:39.838919 2575 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 07 17:40:39.857339 ip-10-0-129-39 kubenswrapper[2575]: W0707 17:40:39.838921 2575 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 07 17:40:39.857339 ip-10-0-129-39 kubenswrapper[2575]: W0707 17:40:39.838926 2575 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 07 17:40:39.857339 ip-10-0-129-39 kubenswrapper[2575]: W0707 17:40:39.838929 2575 feature_gate.go:328] unrecognized feature gate: Example Jul 07 17:40:39.857339 ip-10-0-129-39 kubenswrapper[2575]: W0707 17:40:39.838932 2575 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 07 17:40:39.857339 ip-10-0-129-39 kubenswrapper[2575]: W0707 17:40:39.838936 2575 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 07 17:40:39.857339 ip-10-0-129-39 kubenswrapper[2575]: W0707 17:40:39.838938 2575 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 07 17:40:39.857936 ip-10-0-129-39 kubenswrapper[2575]: W0707 17:40:39.838941 2575 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 07 17:40:39.857936 ip-10-0-129-39 kubenswrapper[2575]: W0707 17:40:39.838944 2575 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 07 17:40:39.857936 ip-10-0-129-39 kubenswrapper[2575]: W0707 17:40:39.838947 2575 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 07 17:40:39.857936 ip-10-0-129-39 kubenswrapper[2575]: W0707 17:40:39.838950 2575 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 07 17:40:39.857936 ip-10-0-129-39 kubenswrapper[2575]: W0707 17:40:39.838953 2575 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 07 17:40:39.857936 ip-10-0-129-39 kubenswrapper[2575]: W0707 17:40:39.838955 2575 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 07 17:40:39.857936 ip-10-0-129-39 kubenswrapper[2575]: W0707 17:40:39.838958 2575 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 07 17:40:39.857936 ip-10-0-129-39 kubenswrapper[2575]: W0707 17:40:39.838961 2575 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 07 17:40:39.857936 ip-10-0-129-39 kubenswrapper[2575]: W0707 17:40:39.838963 2575 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 07 17:40:39.857936 ip-10-0-129-39 kubenswrapper[2575]: W0707 17:40:39.838966 2575 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 07 17:40:39.857936 ip-10-0-129-39 kubenswrapper[2575]: W0707 17:40:39.838968 2575 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 07 17:40:39.857936 ip-10-0-129-39 kubenswrapper[2575]: W0707 17:40:39.838971 2575 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 07 17:40:39.857936 ip-10-0-129-39 kubenswrapper[2575]: W0707 17:40:39.838974 2575 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 07 17:40:39.857936 ip-10-0-129-39 kubenswrapper[2575]: W0707 17:40:39.838976 2575 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 07 17:40:39.857936 ip-10-0-129-39 kubenswrapper[2575]: W0707 17:40:39.838979 2575 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 07 17:40:39.857936 ip-10-0-129-39 kubenswrapper[2575]: W0707 17:40:39.838981 2575 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 07 17:40:39.857936 ip-10-0-129-39 kubenswrapper[2575]: W0707 17:40:39.838984 2575 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 07 17:40:39.857936 ip-10-0-129-39 kubenswrapper[2575]: W0707 17:40:39.838986 2575 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 07 17:40:39.857936 ip-10-0-129-39 kubenswrapper[2575]: W0707 17:40:39.838989 2575 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 07 17:40:39.857936 ip-10-0-129-39 kubenswrapper[2575]: W0707 17:40:39.838992 2575 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 07 17:40:39.857936 ip-10-0-129-39 kubenswrapper[2575]: W0707 17:40:39.838995 2575 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 07 17:40:39.858540 ip-10-0-129-39 kubenswrapper[2575]: W0707 17:40:39.838997 2575 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 07 17:40:39.858540 ip-10-0-129-39 kubenswrapper[2575]: W0707 17:40:39.839000 2575 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 07 17:40:39.858540 ip-10-0-129-39 kubenswrapper[2575]: W0707 17:40:39.839002 2575 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 07 17:40:39.858540 ip-10-0-129-39 kubenswrapper[2575]: W0707 17:40:39.839005 2575 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 07 17:40:39.858540 ip-10-0-129-39 kubenswrapper[2575]: W0707 17:40:39.839007 2575 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 07 17:40:39.858540 ip-10-0-129-39 kubenswrapper[2575]: W0707 17:40:39.839009 2575 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 07 17:40:39.858540 ip-10-0-129-39 kubenswrapper[2575]: W0707 17:40:39.839013 2575 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 07 17:40:39.858540 ip-10-0-129-39 kubenswrapper[2575]: W0707 17:40:39.839016 2575 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 07 17:40:39.858540 ip-10-0-129-39 kubenswrapper[2575]: I0707 17:40:39.839618 2575 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]} Jul 07 17:40:39.858540 ip-10-0-129-39 kubenswrapper[2575]: I0707 17:40:39.847118 2575 server.go:529] "Kubelet version" kubeletVersion="v1.34.8" Jul 07 17:40:39.858540 ip-10-0-129-39 kubenswrapper[2575]: I0707 17:40:39.847132 2575 server.go:531] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK="" Jul 07 17:40:39.858540 ip-10-0-129-39 kubenswrapper[2575]: I0707 17:40:39.848052 2575 watchdog_linux.go:95] "Systemd watchdog is not enabled" Jul 07 17:40:39.858540 ip-10-0-129-39 kubenswrapper[2575]: I0707 17:40:39.848059 2575 watchdog_linux.go:137] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started." Jul 07 17:40:39.858540 ip-10-0-129-39 kubenswrapper[2575]: W0707 17:40:39.848117 2575 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 07 17:40:39.858952 ip-10-0-129-39 kubenswrapper[2575]: W0707 17:40:39.848121 2575 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 07 17:40:39.858952 ip-10-0-129-39 kubenswrapper[2575]: W0707 17:40:39.848126 2575 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 07 17:40:39.858952 ip-10-0-129-39 kubenswrapper[2575]: W0707 17:40:39.848129 2575 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 07 17:40:39.858952 ip-10-0-129-39 kubenswrapper[2575]: W0707 17:40:39.848132 2575 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 07 17:40:39.858952 ip-10-0-129-39 kubenswrapper[2575]: W0707 17:40:39.848134 2575 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 07 17:40:39.858952 ip-10-0-129-39 kubenswrapper[2575]: W0707 17:40:39.848137 2575 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 07 17:40:39.858952 ip-10-0-129-39 kubenswrapper[2575]: W0707 17:40:39.848141 2575 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 07 17:40:39.858952 ip-10-0-129-39 kubenswrapper[2575]: W0707 17:40:39.848143 2575 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 07 17:40:39.858952 ip-10-0-129-39 kubenswrapper[2575]: W0707 17:40:39.848146 2575 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 07 17:40:39.858952 ip-10-0-129-39 kubenswrapper[2575]: W0707 17:40:39.848148 2575 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 07 17:40:39.858952 ip-10-0-129-39 kubenswrapper[2575]: W0707 17:40:39.848151 2575 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 07 17:40:39.858952 ip-10-0-129-39 kubenswrapper[2575]: W0707 17:40:39.848154 2575 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 07 17:40:39.858952 ip-10-0-129-39 kubenswrapper[2575]: W0707 17:40:39.848156 2575 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 07 17:40:39.858952 ip-10-0-129-39 kubenswrapper[2575]: W0707 17:40:39.848159 2575 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 07 17:40:39.858952 ip-10-0-129-39 kubenswrapper[2575]: W0707 17:40:39.848162 2575 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 07 17:40:39.858952 ip-10-0-129-39 kubenswrapper[2575]: W0707 17:40:39.848164 2575 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 07 17:40:39.858952 ip-10-0-129-39 kubenswrapper[2575]: W0707 17:40:39.848167 2575 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 07 17:40:39.858952 ip-10-0-129-39 kubenswrapper[2575]: W0707 17:40:39.848169 2575 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 07 17:40:39.858952 ip-10-0-129-39 kubenswrapper[2575]: W0707 17:40:39.848172 2575 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 07 17:40:39.858952 ip-10-0-129-39 kubenswrapper[2575]: W0707 17:40:39.848175 2575 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 07 17:40:39.859530 ip-10-0-129-39 kubenswrapper[2575]: W0707 17:40:39.848178 2575 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 07 17:40:39.859530 ip-10-0-129-39 kubenswrapper[2575]: W0707 17:40:39.848181 2575 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 07 17:40:39.859530 ip-10-0-129-39 kubenswrapper[2575]: W0707 17:40:39.848183 2575 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 07 17:40:39.859530 ip-10-0-129-39 kubenswrapper[2575]: W0707 17:40:39.848186 2575 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 07 17:40:39.859530 ip-10-0-129-39 kubenswrapper[2575]: W0707 17:40:39.848189 2575 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 07 17:40:39.859530 ip-10-0-129-39 kubenswrapper[2575]: W0707 17:40:39.848192 2575 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 07 17:40:39.859530 ip-10-0-129-39 kubenswrapper[2575]: W0707 17:40:39.848195 2575 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 07 17:40:39.859530 ip-10-0-129-39 kubenswrapper[2575]: W0707 17:40:39.848198 2575 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 07 17:40:39.859530 ip-10-0-129-39 kubenswrapper[2575]: W0707 17:40:39.848201 2575 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 07 17:40:39.859530 ip-10-0-129-39 kubenswrapper[2575]: W0707 17:40:39.848203 2575 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 07 17:40:39.859530 ip-10-0-129-39 kubenswrapper[2575]: W0707 17:40:39.848206 2575 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 07 17:40:39.859530 ip-10-0-129-39 kubenswrapper[2575]: W0707 17:40:39.848208 2575 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 07 17:40:39.859530 ip-10-0-129-39 kubenswrapper[2575]: W0707 17:40:39.848211 2575 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 07 17:40:39.859530 ip-10-0-129-39 kubenswrapper[2575]: W0707 17:40:39.848214 2575 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 07 17:40:39.859530 ip-10-0-129-39 kubenswrapper[2575]: W0707 17:40:39.848217 2575 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 07 17:40:39.859530 ip-10-0-129-39 kubenswrapper[2575]: W0707 17:40:39.848220 2575 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 07 17:40:39.859530 ip-10-0-129-39 kubenswrapper[2575]: W0707 17:40:39.848222 2575 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 07 17:40:39.859530 ip-10-0-129-39 kubenswrapper[2575]: W0707 17:40:39.848226 2575 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 07 17:40:39.859530 ip-10-0-129-39 kubenswrapper[2575]: W0707 17:40:39.848230 2575 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 07 17:40:39.859530 ip-10-0-129-39 kubenswrapper[2575]: W0707 17:40:39.848233 2575 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 07 17:40:39.860094 ip-10-0-129-39 kubenswrapper[2575]: W0707 17:40:39.848236 2575 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 07 17:40:39.860094 ip-10-0-129-39 kubenswrapper[2575]: W0707 17:40:39.848239 2575 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 07 17:40:39.860094 ip-10-0-129-39 kubenswrapper[2575]: W0707 17:40:39.848242 2575 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 07 17:40:39.860094 ip-10-0-129-39 kubenswrapper[2575]: W0707 17:40:39.848245 2575 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 07 17:40:39.860094 ip-10-0-129-39 kubenswrapper[2575]: W0707 17:40:39.848248 2575 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 07 17:40:39.860094 ip-10-0-129-39 kubenswrapper[2575]: W0707 17:40:39.848251 2575 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 07 17:40:39.860094 ip-10-0-129-39 kubenswrapper[2575]: W0707 17:40:39.848254 2575 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 07 17:40:39.860094 ip-10-0-129-39 kubenswrapper[2575]: W0707 17:40:39.848257 2575 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 07 17:40:39.860094 ip-10-0-129-39 kubenswrapper[2575]: W0707 17:40:39.848259 2575 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 07 17:40:39.860094 ip-10-0-129-39 kubenswrapper[2575]: W0707 17:40:39.848262 2575 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 07 17:40:39.860094 ip-10-0-129-39 kubenswrapper[2575]: W0707 17:40:39.848265 2575 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 07 17:40:39.860094 ip-10-0-129-39 kubenswrapper[2575]: W0707 17:40:39.848268 2575 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 07 17:40:39.860094 ip-10-0-129-39 kubenswrapper[2575]: W0707 17:40:39.848272 2575 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 07 17:40:39.860094 ip-10-0-129-39 kubenswrapper[2575]: W0707 17:40:39.848275 2575 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 07 17:40:39.860094 ip-10-0-129-39 kubenswrapper[2575]: W0707 17:40:39.848277 2575 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 07 17:40:39.860094 ip-10-0-129-39 kubenswrapper[2575]: W0707 17:40:39.848280 2575 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 07 17:40:39.860094 ip-10-0-129-39 kubenswrapper[2575]: W0707 17:40:39.848283 2575 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 07 17:40:39.860094 ip-10-0-129-39 kubenswrapper[2575]: W0707 17:40:39.848286 2575 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 07 17:40:39.860094 ip-10-0-129-39 kubenswrapper[2575]: W0707 17:40:39.848288 2575 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 07 17:40:39.860643 ip-10-0-129-39 kubenswrapper[2575]: W0707 17:40:39.848291 2575 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 07 17:40:39.860643 ip-10-0-129-39 kubenswrapper[2575]: W0707 17:40:39.848294 2575 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 07 17:40:39.860643 ip-10-0-129-39 kubenswrapper[2575]: W0707 17:40:39.848297 2575 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 07 17:40:39.860643 ip-10-0-129-39 kubenswrapper[2575]: W0707 17:40:39.848299 2575 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 07 17:40:39.860643 ip-10-0-129-39 kubenswrapper[2575]: W0707 17:40:39.848302 2575 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 07 17:40:39.860643 ip-10-0-129-39 kubenswrapper[2575]: W0707 17:40:39.848304 2575 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 07 17:40:39.860643 ip-10-0-129-39 kubenswrapper[2575]: W0707 17:40:39.848307 2575 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 07 17:40:39.860643 ip-10-0-129-39 kubenswrapper[2575]: W0707 17:40:39.848310 2575 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 07 17:40:39.860643 ip-10-0-129-39 kubenswrapper[2575]: W0707 17:40:39.848313 2575 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 07 17:40:39.860643 ip-10-0-129-39 kubenswrapper[2575]: W0707 17:40:39.848315 2575 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 07 17:40:39.860643 ip-10-0-129-39 kubenswrapper[2575]: W0707 17:40:39.848318 2575 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 07 17:40:39.860643 ip-10-0-129-39 kubenswrapper[2575]: W0707 17:40:39.848320 2575 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 07 17:40:39.860643 ip-10-0-129-39 kubenswrapper[2575]: W0707 17:40:39.848323 2575 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 07 17:40:39.860643 ip-10-0-129-39 kubenswrapper[2575]: W0707 17:40:39.848325 2575 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 07 17:40:39.860643 ip-10-0-129-39 kubenswrapper[2575]: W0707 17:40:39.848328 2575 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 07 17:40:39.860643 ip-10-0-129-39 kubenswrapper[2575]: W0707 17:40:39.848331 2575 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 07 17:40:39.860643 ip-10-0-129-39 kubenswrapper[2575]: W0707 17:40:39.848333 2575 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 07 17:40:39.860643 ip-10-0-129-39 kubenswrapper[2575]: W0707 17:40:39.848337 2575 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 07 17:40:39.860643 ip-10-0-129-39 kubenswrapper[2575]: W0707 17:40:39.848340 2575 feature_gate.go:328] unrecognized feature gate: Example2 Jul 07 17:40:39.860643 ip-10-0-129-39 kubenswrapper[2575]: W0707 17:40:39.848343 2575 feature_gate.go:328] unrecognized feature gate: Example Jul 07 17:40:39.861388 ip-10-0-129-39 kubenswrapper[2575]: W0707 17:40:39.848346 2575 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 07 17:40:39.861388 ip-10-0-129-39 kubenswrapper[2575]: W0707 17:40:39.848349 2575 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 07 17:40:39.861388 ip-10-0-129-39 kubenswrapper[2575]: W0707 17:40:39.848352 2575 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 07 17:40:39.861388 ip-10-0-129-39 kubenswrapper[2575]: W0707 17:40:39.848355 2575 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 07 17:40:39.861388 ip-10-0-129-39 kubenswrapper[2575]: W0707 17:40:39.848358 2575 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 07 17:40:39.861388 ip-10-0-129-39 kubenswrapper[2575]: W0707 17:40:39.848360 2575 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 07 17:40:39.861388 ip-10-0-129-39 kubenswrapper[2575]: W0707 17:40:39.848362 2575 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 07 17:40:39.861388 ip-10-0-129-39 kubenswrapper[2575]: W0707 17:40:39.848365 2575 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 07 17:40:39.861388 ip-10-0-129-39 kubenswrapper[2575]: W0707 17:40:39.848367 2575 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 07 17:40:39.861388 ip-10-0-129-39 kubenswrapper[2575]: W0707 17:40:39.848370 2575 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 07 17:40:39.861388 ip-10-0-129-39 kubenswrapper[2575]: W0707 17:40:39.848373 2575 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 07 17:40:39.861388 ip-10-0-129-39 kubenswrapper[2575]: W0707 17:40:39.848376 2575 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 07 17:40:39.861388 ip-10-0-129-39 kubenswrapper[2575]: W0707 17:40:39.848379 2575 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 07 17:40:39.861388 ip-10-0-129-39 kubenswrapper[2575]: W0707 17:40:39.848381 2575 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 07 17:40:39.861388 ip-10-0-129-39 kubenswrapper[2575]: W0707 17:40:39.848384 2575 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 07 17:40:39.861388 ip-10-0-129-39 kubenswrapper[2575]: W0707 17:40:39.848386 2575 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 07 17:40:39.861388 ip-10-0-129-39 kubenswrapper[2575]: W0707 17:40:39.848389 2575 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 07 17:40:39.861388 ip-10-0-129-39 kubenswrapper[2575]: W0707 17:40:39.848391 2575 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 07 17:40:39.861388 ip-10-0-129-39 kubenswrapper[2575]: W0707 17:40:39.848394 2575 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 07 17:40:39.861967 ip-10-0-129-39 kubenswrapper[2575]: I0707 17:40:39.848399 2575 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]} Jul 07 17:40:39.861967 ip-10-0-129-39 kubenswrapper[2575]: W0707 17:40:39.848531 2575 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 07 17:40:39.861967 ip-10-0-129-39 kubenswrapper[2575]: W0707 17:40:39.848540 2575 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 07 17:40:39.861967 ip-10-0-129-39 kubenswrapper[2575]: W0707 17:40:39.848544 2575 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 07 17:40:39.861967 ip-10-0-129-39 kubenswrapper[2575]: W0707 17:40:39.848550 2575 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 07 17:40:39.861967 ip-10-0-129-39 kubenswrapper[2575]: W0707 17:40:39.848554 2575 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 07 17:40:39.861967 ip-10-0-129-39 kubenswrapper[2575]: W0707 17:40:39.848559 2575 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 07 17:40:39.861967 ip-10-0-129-39 kubenswrapper[2575]: W0707 17:40:39.848563 2575 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 07 17:40:39.861967 ip-10-0-129-39 kubenswrapper[2575]: W0707 17:40:39.848566 2575 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 07 17:40:39.861967 ip-10-0-129-39 kubenswrapper[2575]: W0707 17:40:39.848568 2575 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 07 17:40:39.861967 ip-10-0-129-39 kubenswrapper[2575]: W0707 17:40:39.848571 2575 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 07 17:40:39.861967 ip-10-0-129-39 kubenswrapper[2575]: W0707 17:40:39.848574 2575 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 07 17:40:39.861967 ip-10-0-129-39 kubenswrapper[2575]: W0707 17:40:39.848577 2575 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 07 17:40:39.861967 ip-10-0-129-39 kubenswrapper[2575]: W0707 17:40:39.848583 2575 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 07 17:40:39.861967 ip-10-0-129-39 kubenswrapper[2575]: W0707 17:40:39.848586 2575 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 07 17:40:39.862454 ip-10-0-129-39 kubenswrapper[2575]: W0707 17:40:39.848589 2575 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 07 17:40:39.862454 ip-10-0-129-39 kubenswrapper[2575]: W0707 17:40:39.848591 2575 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 07 17:40:39.862454 ip-10-0-129-39 kubenswrapper[2575]: W0707 17:40:39.848594 2575 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 07 17:40:39.862454 ip-10-0-129-39 kubenswrapper[2575]: W0707 17:40:39.848597 2575 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 07 17:40:39.862454 ip-10-0-129-39 kubenswrapper[2575]: W0707 17:40:39.848599 2575 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 07 17:40:39.862454 ip-10-0-129-39 kubenswrapper[2575]: W0707 17:40:39.848602 2575 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 07 17:40:39.862454 ip-10-0-129-39 kubenswrapper[2575]: W0707 17:40:39.848605 2575 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 07 17:40:39.862454 ip-10-0-129-39 kubenswrapper[2575]: W0707 17:40:39.848607 2575 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 07 17:40:39.862454 ip-10-0-129-39 kubenswrapper[2575]: W0707 17:40:39.848610 2575 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 07 17:40:39.862454 ip-10-0-129-39 kubenswrapper[2575]: W0707 17:40:39.848612 2575 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 07 17:40:39.862454 ip-10-0-129-39 kubenswrapper[2575]: W0707 17:40:39.848615 2575 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 07 17:40:39.862454 ip-10-0-129-39 kubenswrapper[2575]: W0707 17:40:39.848617 2575 feature_gate.go:328] unrecognized feature gate: Example Jul 07 17:40:39.862454 ip-10-0-129-39 kubenswrapper[2575]: W0707 17:40:39.848620 2575 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 07 17:40:39.862454 ip-10-0-129-39 kubenswrapper[2575]: W0707 17:40:39.848623 2575 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 07 17:40:39.862454 ip-10-0-129-39 kubenswrapper[2575]: W0707 17:40:39.848625 2575 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 07 17:40:39.862454 ip-10-0-129-39 kubenswrapper[2575]: W0707 17:40:39.848628 2575 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 07 17:40:39.862454 ip-10-0-129-39 kubenswrapper[2575]: W0707 17:40:39.848630 2575 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 07 17:40:39.862454 ip-10-0-129-39 kubenswrapper[2575]: W0707 17:40:39.848633 2575 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 07 17:40:39.862454 ip-10-0-129-39 kubenswrapper[2575]: W0707 17:40:39.848636 2575 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 07 17:40:39.862454 ip-10-0-129-39 kubenswrapper[2575]: W0707 17:40:39.848638 2575 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 07 17:40:39.863077 ip-10-0-129-39 kubenswrapper[2575]: W0707 17:40:39.848641 2575 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 07 17:40:39.863077 ip-10-0-129-39 kubenswrapper[2575]: W0707 17:40:39.848643 2575 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 07 17:40:39.863077 ip-10-0-129-39 kubenswrapper[2575]: W0707 17:40:39.848646 2575 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 07 17:40:39.863077 ip-10-0-129-39 kubenswrapper[2575]: W0707 17:40:39.848648 2575 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 07 17:40:39.863077 ip-10-0-129-39 kubenswrapper[2575]: W0707 17:40:39.848651 2575 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 07 17:40:39.863077 ip-10-0-129-39 kubenswrapper[2575]: W0707 17:40:39.848653 2575 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 07 17:40:39.863077 ip-10-0-129-39 kubenswrapper[2575]: W0707 17:40:39.848656 2575 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 07 17:40:39.863077 ip-10-0-129-39 kubenswrapper[2575]: W0707 17:40:39.848658 2575 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 07 17:40:39.863077 ip-10-0-129-39 kubenswrapper[2575]: W0707 17:40:39.848661 2575 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 07 17:40:39.863077 ip-10-0-129-39 kubenswrapper[2575]: W0707 17:40:39.848664 2575 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 07 17:40:39.863077 ip-10-0-129-39 kubenswrapper[2575]: W0707 17:40:39.848668 2575 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 07 17:40:39.863077 ip-10-0-129-39 kubenswrapper[2575]: W0707 17:40:39.848672 2575 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 07 17:40:39.863077 ip-10-0-129-39 kubenswrapper[2575]: W0707 17:40:39.848675 2575 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 07 17:40:39.863077 ip-10-0-129-39 kubenswrapper[2575]: W0707 17:40:39.848678 2575 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 07 17:40:39.863077 ip-10-0-129-39 kubenswrapper[2575]: W0707 17:40:39.848681 2575 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 07 17:40:39.863077 ip-10-0-129-39 kubenswrapper[2575]: W0707 17:40:39.848683 2575 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 07 17:40:39.863077 ip-10-0-129-39 kubenswrapper[2575]: W0707 17:40:39.848686 2575 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 07 17:40:39.863077 ip-10-0-129-39 kubenswrapper[2575]: W0707 17:40:39.848689 2575 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 07 17:40:39.863077 ip-10-0-129-39 kubenswrapper[2575]: W0707 17:40:39.848691 2575 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 07 17:40:39.863679 ip-10-0-129-39 kubenswrapper[2575]: W0707 17:40:39.848694 2575 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 07 17:40:39.863679 ip-10-0-129-39 kubenswrapper[2575]: W0707 17:40:39.848697 2575 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 07 17:40:39.863679 ip-10-0-129-39 kubenswrapper[2575]: W0707 17:40:39.848709 2575 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 07 17:40:39.863679 ip-10-0-129-39 kubenswrapper[2575]: W0707 17:40:39.848714 2575 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 07 17:40:39.863679 ip-10-0-129-39 kubenswrapper[2575]: W0707 17:40:39.848716 2575 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 07 17:40:39.863679 ip-10-0-129-39 kubenswrapper[2575]: W0707 17:40:39.848719 2575 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 07 17:40:39.863679 ip-10-0-129-39 kubenswrapper[2575]: W0707 17:40:39.848721 2575 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 07 17:40:39.863679 ip-10-0-129-39 kubenswrapper[2575]: W0707 17:40:39.848724 2575 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 07 17:40:39.863679 ip-10-0-129-39 kubenswrapper[2575]: W0707 17:40:39.848727 2575 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 07 17:40:39.863679 ip-10-0-129-39 kubenswrapper[2575]: W0707 17:40:39.848729 2575 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 07 17:40:39.863679 ip-10-0-129-39 kubenswrapper[2575]: W0707 17:40:39.848732 2575 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 07 17:40:39.863679 ip-10-0-129-39 kubenswrapper[2575]: W0707 17:40:39.848734 2575 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 07 17:40:39.863679 ip-10-0-129-39 kubenswrapper[2575]: W0707 17:40:39.848737 2575 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 07 17:40:39.863679 ip-10-0-129-39 kubenswrapper[2575]: W0707 17:40:39.848740 2575 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 07 17:40:39.863679 ip-10-0-129-39 kubenswrapper[2575]: W0707 17:40:39.848742 2575 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 07 17:40:39.863679 ip-10-0-129-39 kubenswrapper[2575]: W0707 17:40:39.848745 2575 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 07 17:40:39.863679 ip-10-0-129-39 kubenswrapper[2575]: W0707 17:40:39.848748 2575 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 07 17:40:39.863679 ip-10-0-129-39 kubenswrapper[2575]: W0707 17:40:39.848750 2575 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 07 17:40:39.863679 ip-10-0-129-39 kubenswrapper[2575]: W0707 17:40:39.848753 2575 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 07 17:40:39.863679 ip-10-0-129-39 kubenswrapper[2575]: W0707 17:40:39.848756 2575 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 07 17:40:39.864321 ip-10-0-129-39 kubenswrapper[2575]: W0707 17:40:39.848758 2575 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 07 17:40:39.864321 ip-10-0-129-39 kubenswrapper[2575]: W0707 17:40:39.848762 2575 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 07 17:40:39.864321 ip-10-0-129-39 kubenswrapper[2575]: W0707 17:40:39.848765 2575 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 07 17:40:39.864321 ip-10-0-129-39 kubenswrapper[2575]: W0707 17:40:39.848768 2575 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 07 17:40:39.864321 ip-10-0-129-39 kubenswrapper[2575]: W0707 17:40:39.848771 2575 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 07 17:40:39.864321 ip-10-0-129-39 kubenswrapper[2575]: W0707 17:40:39.848774 2575 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 07 17:40:39.864321 ip-10-0-129-39 kubenswrapper[2575]: W0707 17:40:39.848777 2575 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 07 17:40:39.864321 ip-10-0-129-39 kubenswrapper[2575]: W0707 17:40:39.848780 2575 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 07 17:40:39.864321 ip-10-0-129-39 kubenswrapper[2575]: W0707 17:40:39.848782 2575 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 07 17:40:39.864321 ip-10-0-129-39 kubenswrapper[2575]: W0707 17:40:39.848785 2575 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 07 17:40:39.864321 ip-10-0-129-39 kubenswrapper[2575]: W0707 17:40:39.848788 2575 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 07 17:40:39.864321 ip-10-0-129-39 kubenswrapper[2575]: W0707 17:40:39.848791 2575 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 07 17:40:39.864321 ip-10-0-129-39 kubenswrapper[2575]: W0707 17:40:39.848793 2575 feature_gate.go:328] unrecognized feature gate: Example2 Jul 07 17:40:39.864321 ip-10-0-129-39 kubenswrapper[2575]: W0707 17:40:39.848795 2575 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 07 17:40:39.864321 ip-10-0-129-39 kubenswrapper[2575]: W0707 17:40:39.848798 2575 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 07 17:40:39.864321 ip-10-0-129-39 kubenswrapper[2575]: W0707 17:40:39.848802 2575 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 07 17:40:39.864321 ip-10-0-129-39 kubenswrapper[2575]: W0707 17:40:39.848804 2575 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 07 17:40:39.864321 ip-10-0-129-39 kubenswrapper[2575]: W0707 17:40:39.848807 2575 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 07 17:40:39.864321 ip-10-0-129-39 kubenswrapper[2575]: W0707 17:40:39.848809 2575 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 07 17:40:39.864321 ip-10-0-129-39 kubenswrapper[2575]: W0707 17:40:39.848812 2575 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 07 17:40:39.868452 ip-10-0-129-39 kubenswrapper[2575]: W0707 17:40:39.848815 2575 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 07 17:40:39.868452 ip-10-0-129-39 kubenswrapper[2575]: W0707 17:40:39.848817 2575 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 07 17:40:39.868452 ip-10-0-129-39 kubenswrapper[2575]: W0707 17:40:39.848820 2575 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 07 17:40:39.868452 ip-10-0-129-39 kubenswrapper[2575]: W0707 17:40:39.848822 2575 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 07 17:40:39.868452 ip-10-0-129-39 kubenswrapper[2575]: W0707 17:40:39.848824 2575 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 07 17:40:39.868452 ip-10-0-129-39 kubenswrapper[2575]: W0707 17:40:39.848828 2575 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 07 17:40:39.868452 ip-10-0-129-39 kubenswrapper[2575]: I0707 17:40:39.848833 2575 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]} Jul 07 17:40:39.868452 ip-10-0-129-39 kubenswrapper[2575]: I0707 17:40:39.848933 2575 server.go:962] "Client rotation is on, will bootstrap in background" Jul 07 17:40:39.868452 ip-10-0-129-39 kubenswrapper[2575]: I0707 17:40:39.850940 2575 bootstrap.go:101] "Use the bootstrap credentials to request a cert, and set kubeconfig to point to the certificate dir" Jul 07 17:40:39.868452 ip-10-0-129-39 kubenswrapper[2575]: I0707 17:40:39.852552 2575 server.go:1019] "Starting client certificate rotation" Jul 07 17:40:39.868452 ip-10-0-129-39 kubenswrapper[2575]: I0707 17:40:39.852658 2575 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet" Jul 07 17:40:39.868452 ip-10-0-129-39 kubenswrapper[2575]: I0707 17:40:39.852688 2575 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kube-apiserver-client-kubelet" Jul 07 17:40:39.868452 ip-10-0-129-39 kubenswrapper[2575]: I0707 17:40:39.856391 2575 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Jul 07 17:40:39.868876 ip-10-0-129-39 kubenswrapper[2575]: I0707 17:40:39.857739 2575 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Jul 07 17:40:39.868876 ip-10-0-129-39 kubenswrapper[2575]: I0707 17:40:39.865975 2575 log.go:25] "Validated CRI v1 runtime API" Jul 07 17:40:39.870433 ip-10-0-129-39 kubenswrapper[2575]: I0707 17:40:39.870416 2575 log.go:25] "Validated CRI v1 image API" Jul 07 17:40:39.871311 ip-10-0-129-39 kubenswrapper[2575]: I0707 17:40:39.871271 2575 server.go:1429] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd" Jul 07 17:40:39.874796 ip-10-0-129-39 kubenswrapper[2575]: I0707 17:40:39.874773 2575 fs.go:135] Filesystem UUIDs: map[17a35e75-0950-4f06-975a-1c96fc32fb09:/dev/nvme0n1p3 7B77-95E7:/dev/nvme0n1p2 7a7fb344-10bf-4949-900f-a189e2d00465:/dev/nvme0n1p4] Jul 07 17:40:39.874861 ip-10-0-129-39 kubenswrapper[2575]: I0707 17:40:39.874794 2575 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:22 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:24 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:31 fsType:tmpfs blockSize:0} composefs_0-33:{mountpoint:/ major:0 minor:33 fsType:overlay blockSize:0}] Jul 07 17:40:39.881830 ip-10-0-129-39 kubenswrapper[2575]: I0707 17:40:39.881725 2575 manager.go:216] Machine: {Timestamp:2026-07-07 17:40:39.880336638 +0000 UTC m=+0.386143295 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3106230 MemoryCapacity:33164500992 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec2f4d8e067465f3a402be2f93a7b7d5 SystemUUID:ec2f4d8e-0674-65f3-a402-be2f93a7b7d5 BootID:25b7db86-97f8-4479-a590-ae0a9c91dfd5 Filesystems:[{Device:/tmp DeviceMajor:0 DeviceMinor:31 Capacity:16582250496 Type:vfs Inodes:1048576 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true} {Device:/dev/shm DeviceMajor:0 DeviceMinor:22 Capacity:16582250496 Type:vfs Inodes:4048401 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:24 Capacity:6632902656 Type:vfs Inodes:819200 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:composefs_0-33 DeviceMajor:0 DeviceMinor:33 Capacity:6107136 Type:vfs Inodes:18446744073709551615 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:br-ex MacAddress:02:c8:e9:b3:cd:d3 Speed:0 Mtu:9001} {Name:ens5 MacAddress:02:c8:e9:b3:cd:d3 Speed:0 Mtu:9001} {Name:ovs-system MacAddress:ce:12:7d:41:4b:85 Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:33164500992 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:37486592 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None} Jul 07 17:40:39.881830 ip-10-0-129-39 kubenswrapper[2575]: I0707 17:40:39.881825 2575 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available. Jul 07 17:40:39.881951 ip-10-0-129-39 kubenswrapper[2575]: I0707 17:40:39.881929 2575 manager.go:232] Version: {KernelVersion:5.14.0-570.125.1.el9_6.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.6.20260630-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:} Jul 07 17:40:39.882614 ip-10-0-129-39 kubenswrapper[2575]: I0707 17:40:39.882595 2575 reflector.go:439] "Caches populated" logger="kubernetes.io/kube-apiserver-client-kubelet" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" Jul 07 17:40:39.883930 ip-10-0-129-39 kubenswrapper[2575]: I0707 17:40:39.883903 2575 container_manager_linux.go:273] "Container manager verified user specified cgroup-root exists" cgroupRoot=[] Jul 07 17:40:39.884054 ip-10-0-129-39 kubenswrapper[2575]: I0707 17:40:39.883934 2575 container_manager_linux.go:278] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-129-39.ec2.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2} Jul 07 17:40:39.884732 ip-10-0-129-39 kubenswrapper[2575]: I0707 17:40:39.884721 2575 topology_manager.go:138] "Creating topology manager with none policy" Jul 07 17:40:39.884765 ip-10-0-129-39 kubenswrapper[2575]: I0707 17:40:39.884735 2575 container_manager_linux.go:309] "Creating device plugin manager" Jul 07 17:40:39.884765 ip-10-0-129-39 kubenswrapper[2575]: I0707 17:40:39.884742 2575 manager.go:141] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock" Jul 07 17:40:39.886425 ip-10-0-129-39 kubenswrapper[2575]: I0707 17:40:39.886415 2575 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock" Jul 07 17:40:39.887634 ip-10-0-129-39 kubenswrapper[2575]: I0707 17:40:39.887624 2575 container_manager_linux.go:318] "Creating Dynamic Resource Allocation (DRA) manager" Jul 07 17:40:39.890716 ip-10-0-129-39 kubenswrapper[2575]: I0707 17:40:39.890705 2575 state_mem.go:36] "Initialized new in-memory state store" Jul 07 17:40:39.890818 ip-10-0-129-39 kubenswrapper[2575]: I0707 17:40:39.890808 2575 server.go:1240] "Using root directory" path="/var/lib/kubelet" Jul 07 17:40:39.890855 ip-10-0-129-39 kubenswrapper[2575]: I0707 17:40:39.890836 2575 kubelet.go:486] "Attempting to sync node with API server" Jul 07 17:40:39.890855 ip-10-0-129-39 kubenswrapper[2575]: I0707 17:40:39.890845 2575 kubelet.go:387] "Adding static pod path" path="/etc/kubernetes/manifests" Jul 07 17:40:39.890924 ip-10-0-129-39 kubenswrapper[2575]: I0707 17:40:39.890858 2575 file.go:69] "Watching path" path="/etc/kubernetes/manifests" Jul 07 17:40:39.890924 ip-10-0-129-39 kubenswrapper[2575]: I0707 17:40:39.890866 2575 kubelet.go:398] "Adding apiserver pod source" Jul 07 17:40:39.890924 ip-10-0-129-39 kubenswrapper[2575]: I0707 17:40:39.890874 2575 apiserver.go:42] "Waiting for node sync before watching apiserver pods" Jul 07 17:40:39.891159 ip-10-0-129-39 kubenswrapper[2575]: I0707 17:40:39.891148 2575 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Jul 07 17:40:39.891198 ip-10-0-129-39 kubenswrapper[2575]: I0707 17:40:39.891165 2575 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Jul 07 17:40:39.893359 ip-10-0-129-39 kubenswrapper[2575]: I0707 17:40:39.893344 2575 kuberuntime_manager.go:301] "Container runtime initialized" containerRuntime="cri-o" version="1.34.8-3.rhaos4.21.git91b4b0b.el9" apiVersion="v1" Jul 07 17:40:39.895112 ip-10-0-129-39 kubenswrapper[2575]: I0707 17:40:39.895100 2575 kubelet.go:960] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled" Jul 07 17:40:39.895154 ip-10-0-129-39 kubenswrapper[2575]: I0707 17:40:39.895130 2575 kubelet.go:984] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled" Jul 07 17:40:39.896182 ip-10-0-129-39 kubenswrapper[2575]: I0707 17:40:39.896171 2575 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume" Jul 07 17:40:39.896240 ip-10-0-129-39 kubenswrapper[2575]: I0707 17:40:39.896186 2575 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir" Jul 07 17:40:39.896240 ip-10-0-129-39 kubenswrapper[2575]: I0707 17:40:39.896192 2575 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo" Jul 07 17:40:39.896240 ip-10-0-129-39 kubenswrapper[2575]: I0707 17:40:39.896197 2575 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path" Jul 07 17:40:39.896240 ip-10-0-129-39 kubenswrapper[2575]: I0707 17:40:39.896206 2575 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs" Jul 07 17:40:39.896240 ip-10-0-129-39 kubenswrapper[2575]: I0707 17:40:39.896212 2575 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret" Jul 07 17:40:39.896240 ip-10-0-129-39 kubenswrapper[2575]: I0707 17:40:39.896220 2575 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi" Jul 07 17:40:39.896240 ip-10-0-129-39 kubenswrapper[2575]: I0707 17:40:39.896226 2575 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api" Jul 07 17:40:39.896240 ip-10-0-129-39 kubenswrapper[2575]: I0707 17:40:39.896231 2575 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc" Jul 07 17:40:39.896240 ip-10-0-129-39 kubenswrapper[2575]: I0707 17:40:39.896236 2575 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap" Jul 07 17:40:39.896240 ip-10-0-129-39 kubenswrapper[2575]: I0707 17:40:39.896244 2575 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected" Jul 07 17:40:39.896482 ip-10-0-129-39 kubenswrapper[2575]: I0707 17:40:39.896250 2575 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume" Jul 07 17:40:39.897128 ip-10-0-129-39 kubenswrapper[2575]: I0707 17:40:39.897114 2575 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi" Jul 07 17:40:39.897128 ip-10-0-129-39 kubenswrapper[2575]: I0707 17:40:39.897126 2575 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image" Jul 07 17:40:39.900065 ip-10-0-129-39 kubenswrapper[2575]: I0707 17:40:39.900053 2575 server.go:1268] "Started kubelet" Jul 07 17:40:39.900162 ip-10-0-129-39 kubenswrapper[2575]: I0707 17:40:39.900134 2575 server.go:180] "Starting to listen" address="0.0.0.0" port=10250 Jul 07 17:40:39.900281 ip-10-0-129-39 kubenswrapper[2575]: I0707 17:40:39.900167 2575 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10 Jul 07 17:40:39.900359 ip-10-0-129-39 kubenswrapper[2575]: I0707 17:40:39.900341 2575 server_v1.go:49] "podresources" method="list" useActivePods=true Jul 07 17:40:39.900920 ip-10-0-129-39 systemd[1]: Started Kubernetes Kubelet. Jul 07 17:40:39.901366 ip-10-0-129-39 kubenswrapper[2575]: I0707 17:40:39.901347 2575 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-129-39.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope Jul 07 17:40:39.901461 ip-10-0-129-39 kubenswrapper[2575]: E0707 17:40:39.901402 2575 reflector.go:205] "Failed to watch" err="failed to list *v1.Node: nodes \"ip-10-0-129-39.ec2.internal\" is forbidden: User \"system:anonymous\" cannot list resource \"nodes\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Node" Jul 07 17:40:39.901641 ip-10-0-129-39 kubenswrapper[2575]: E0707 17:40:39.901461 2575 reflector.go:205] "Failed to watch" err="failed to list *v1.Service: services is forbidden: User \"system:anonymous\" cannot list resource \"services\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Service" Jul 07 17:40:39.902035 ip-10-0-129-39 kubenswrapper[2575]: I0707 17:40:39.901893 2575 server.go:249] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock" Jul 07 17:40:39.902240 ip-10-0-129-39 kubenswrapper[2575]: I0707 17:40:39.902222 2575 server.go:310] "Adding debug handlers to kubelet server" Jul 07 17:40:39.905169 ip-10-0-129-39 kubenswrapper[2575]: E0707 17:40:39.905139 2575 kubelet.go:1638] "Image garbage collection failed once. Stats initialization may not have completed yet" err="failed to get imageFs info: unable to find data in memory cache" Jul 07 17:40:39.905896 ip-10-0-129-39 kubenswrapper[2575]: E0707 17:40:39.904864 2575 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-129-39.ec2.internal.18c012b964517b28 default 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-129-39.ec2.internal,UID:ip-10-0-129-39.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:Starting,Message:Starting kubelet.,Source:EventSource{Component:kubelet,Host:ip-10-0-129-39.ec2.internal,},FirstTimestamp:2026-07-07 17:40:39.900027688 +0000 UTC m=+0.405834344,LastTimestamp:2026-07-07 17:40:39.900027688 +0000 UTC m=+0.405834344,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-129-39.ec2.internal,}" Jul 07 17:40:39.907460 ip-10-0-129-39 kubenswrapper[2575]: I0707 17:40:39.907442 2575 fs_resource_analyzer.go:67] "Starting FS ResourceAnalyzer" Jul 07 17:40:39.907569 ip-10-0-129-39 kubenswrapper[2575]: I0707 17:40:39.907448 2575 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving" Jul 07 17:40:39.907631 ip-10-0-129-39 kubenswrapper[2575]: E0707 17:40:39.907577 2575 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-129-39.ec2.internal\" not found" Jul 07 17:40:39.908162 ip-10-0-129-39 kubenswrapper[2575]: I0707 17:40:39.908141 2575 desired_state_of_world_populator.go:146] "Desired state populator starts to run" Jul 07 17:40:39.908246 ip-10-0-129-39 kubenswrapper[2575]: I0707 17:40:39.908202 2575 volume_manager.go:311] "The desired_state_of_world populator starts" Jul 07 17:40:39.908246 ip-10-0-129-39 kubenswrapper[2575]: I0707 17:40:39.908223 2575 volume_manager.go:313] "Starting Kubelet Volume Manager" Jul 07 17:40:39.908348 ip-10-0-129-39 kubenswrapper[2575]: I0707 17:40:39.908294 2575 factory.go:55] Registering systemd factory Jul 07 17:40:39.908348 ip-10-0-129-39 kubenswrapper[2575]: I0707 17:40:39.908310 2575 factory.go:223] Registration of the systemd container factory successfully Jul 07 17:40:39.908348 ip-10-0-129-39 kubenswrapper[2575]: I0707 17:40:39.908347 2575 reconstruct.go:97] "Volume reconstruction finished" Jul 07 17:40:39.908474 ip-10-0-129-39 kubenswrapper[2575]: I0707 17:40:39.908358 2575 reconciler.go:29] "Reconciler: start to sync state" Jul 07 17:40:39.908537 ip-10-0-129-39 kubenswrapper[2575]: I0707 17:40:39.908497 2575 factory.go:153] Registering CRI-O factory Jul 07 17:40:39.908537 ip-10-0-129-39 kubenswrapper[2575]: I0707 17:40:39.908524 2575 factory.go:223] Registration of the crio container factory successfully Jul 07 17:40:39.908598 ip-10-0-129-39 kubenswrapper[2575]: I0707 17:40:39.908566 2575 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory Jul 07 17:40:39.908598 ip-10-0-129-39 kubenswrapper[2575]: I0707 17:40:39.908588 2575 factory.go:103] Registering Raw factory Jul 07 17:40:39.908663 ip-10-0-129-39 kubenswrapper[2575]: I0707 17:40:39.908624 2575 manager.go:1196] Started watching for new ooms in manager Jul 07 17:40:39.909416 ip-10-0-129-39 kubenswrapper[2575]: I0707 17:40:39.909402 2575 manager.go:349] Starting recovery of all containers Jul 07 17:40:39.910187 ip-10-0-129-39 kubenswrapper[2575]: E0707 17:40:39.910162 2575 controller.go:145] "Failed to ensure lease exists, will retry" err="leases.coordination.k8s.io \"ip-10-0-129-39.ec2.internal\" is forbidden: User \"system:anonymous\" cannot get resource \"leases\" in API group \"coordination.k8s.io\" in the namespace \"kube-node-lease\"" interval="200ms" Jul 07 17:40:39.910439 ip-10-0-129-39 kubenswrapper[2575]: E0707 17:40:39.910408 2575 reflector.go:205] "Failed to watch" err="failed to list *v1.CSIDriver: csidrivers.storage.k8s.io is forbidden: User \"system:anonymous\" cannot list resource \"csidrivers\" in API group \"storage.k8s.io\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.CSIDriver" Jul 07 17:40:39.912714 ip-10-0-129-39 kubenswrapper[2575]: I0707 17:40:39.912655 2575 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-v7c5c" Jul 07 17:40:39.920155 ip-10-0-129-39 kubenswrapper[2575]: I0707 17:40:39.920134 2575 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-v7c5c" Jul 07 17:40:39.922691 ip-10-0-129-39 kubenswrapper[2575]: I0707 17:40:39.922662 2575 manager.go:354] Recovery completed Jul 07 17:40:39.926998 ip-10-0-129-39 kubenswrapper[2575]: I0707 17:40:39.926986 2575 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 07 17:40:39.928533 ip-10-0-129-39 kubenswrapper[2575]: I0707 17:40:39.928516 2575 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-129-39.ec2.internal" event="NodeHasSufficientMemory" Jul 07 17:40:39.928608 ip-10-0-129-39 kubenswrapper[2575]: I0707 17:40:39.928547 2575 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-129-39.ec2.internal" event="NodeHasNoDiskPressure" Jul 07 17:40:39.928608 ip-10-0-129-39 kubenswrapper[2575]: I0707 17:40:39.928561 2575 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-129-39.ec2.internal" event="NodeHasSufficientPID" Jul 07 17:40:39.929087 ip-10-0-129-39 kubenswrapper[2575]: I0707 17:40:39.929074 2575 cpu_manager.go:222] "Starting CPU manager" policy="none" Jul 07 17:40:39.929087 ip-10-0-129-39 kubenswrapper[2575]: I0707 17:40:39.929086 2575 cpu_manager.go:223] "Reconciling" reconcilePeriod="10s" Jul 07 17:40:39.929153 ip-10-0-129-39 kubenswrapper[2575]: I0707 17:40:39.929100 2575 state_mem.go:36] "Initialized new in-memory state store" Jul 07 17:40:39.929153 ip-10-0-129-39 kubenswrapper[2575]: I0707 17:40:39.929127 2575 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 07 17:40:39.929761 ip-10-0-129-39 kubenswrapper[2575]: I0707 17:40:39.929749 2575 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-129-39.ec2.internal" event="NodeHasSufficientMemory" Jul 07 17:40:39.929813 ip-10-0-129-39 kubenswrapper[2575]: I0707 17:40:39.929773 2575 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-129-39.ec2.internal" event="NodeHasNoDiskPressure" Jul 07 17:40:39.929813 ip-10-0-129-39 kubenswrapper[2575]: I0707 17:40:39.929782 2575 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-129-39.ec2.internal" event="NodeHasSufficientPID" Jul 07 17:40:39.930261 ip-10-0-129-39 kubenswrapper[2575]: I0707 17:40:39.930250 2575 policy_none.go:49] "None policy: Start" Jul 07 17:40:39.930301 ip-10-0-129-39 kubenswrapper[2575]: I0707 17:40:39.930265 2575 memory_manager.go:187] "Starting memorymanager" policy="None" Jul 07 17:40:39.930301 ip-10-0-129-39 kubenswrapper[2575]: I0707 17:40:39.930275 2575 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint" Jul 07 17:40:39.930823 ip-10-0-129-39 kubenswrapper[2575]: E0707 17:40:39.930757 2575 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-129-39.ec2.internal.18c012b966046be7 default 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-129-39.ec2.internal,UID:ip-10-0-129-39.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeHasSufficientMemory,Message:Node ip-10-0-129-39.ec2.internal status is now: NodeHasSufficientMemory,Source:EventSource{Component:kubelet,Host:ip-10-0-129-39.ec2.internal,},FirstTimestamp:2026-07-07 17:40:39.928531943 +0000 UTC m=+0.434338605,LastTimestamp:2026-07-07 17:40:39.928531943 +0000 UTC m=+0.434338605,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-129-39.ec2.internal,}" Jul 07 17:40:39.931312 ip-10-0-129-39 kubenswrapper[2575]: I0707 17:40:39.931301 2575 policy_none.go:47] "Start" Jul 07 17:40:39.969104 ip-10-0-129-39 kubenswrapper[2575]: I0707 17:40:39.969087 2575 manager.go:337] "Starting Device Plugin manager" Jul 07 17:40:39.969180 ip-10-0-129-39 kubenswrapper[2575]: E0707 17:40:39.969121 2575 manager.go:513] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint" Jul 07 17:40:39.969180 ip-10-0-129-39 kubenswrapper[2575]: I0707 17:40:39.969133 2575 server.go:87] "Starting device plugin registration server" Jul 07 17:40:39.969378 ip-10-0-129-39 kubenswrapper[2575]: I0707 17:40:39.969366 2575 eviction_manager.go:189] "Eviction manager: starting control loop" Jul 07 17:40:39.969450 ip-10-0-129-39 kubenswrapper[2575]: I0707 17:40:39.969380 2575 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s" Jul 07 17:40:39.969547 ip-10-0-129-39 kubenswrapper[2575]: I0707 17:40:39.969529 2575 plugin_watcher.go:51] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry" Jul 07 17:40:39.969626 ip-10-0-129-39 kubenswrapper[2575]: I0707 17:40:39.969614 2575 plugin_manager.go:116] "The desired_state_of_world populator (plugin watcher) starts" Jul 07 17:40:39.969685 ip-10-0-129-39 kubenswrapper[2575]: I0707 17:40:39.969627 2575 plugin_manager.go:118] "Starting Kubelet Plugin Manager" Jul 07 17:40:39.969970 ip-10-0-129-39 kubenswrapper[2575]: E0707 17:40:39.969953 2575 eviction_manager.go:267] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\"" Jul 07 17:40:39.970042 ip-10-0-129-39 kubenswrapper[2575]: E0707 17:40:39.969990 2575 eviction_manager.go:292] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-129-39.ec2.internal\" not found" Jul 07 17:40:40.065914 ip-10-0-129-39 kubenswrapper[2575]: I0707 17:40:40.065881 2575 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4" Jul 07 17:40:40.067157 ip-10-0-129-39 kubenswrapper[2575]: I0707 17:40:40.067143 2575 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6" Jul 07 17:40:40.067231 ip-10-0-129-39 kubenswrapper[2575]: I0707 17:40:40.067175 2575 status_manager.go:244] "Starting to sync pod status with apiserver" Jul 07 17:40:40.067231 ip-10-0-129-39 kubenswrapper[2575]: I0707 17:40:40.067194 2575 kubelet.go:2448] "Starting kubelet main sync loop" Jul 07 17:40:40.067231 ip-10-0-129-39 kubenswrapper[2575]: E0707 17:40:40.067221 2575 kubelet.go:2472] "Skipping pod synchronization" err="PLEG is not healthy: pleg has yet to be successful" Jul 07 17:40:40.069495 ip-10-0-129-39 kubenswrapper[2575]: I0707 17:40:40.069476 2575 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 07 17:40:40.069605 ip-10-0-129-39 kubenswrapper[2575]: I0707 17:40:40.069543 2575 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160" Jul 07 17:40:40.070316 ip-10-0-129-39 kubenswrapper[2575]: I0707 17:40:40.070301 2575 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-129-39.ec2.internal" event="NodeHasSufficientMemory" Jul 07 17:40:40.070390 ip-10-0-129-39 kubenswrapper[2575]: I0707 17:40:40.070330 2575 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-129-39.ec2.internal" event="NodeHasNoDiskPressure" Jul 07 17:40:40.070390 ip-10-0-129-39 kubenswrapper[2575]: I0707 17:40:40.070340 2575 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-129-39.ec2.internal" event="NodeHasSufficientPID" Jul 07 17:40:40.070390 ip-10-0-129-39 kubenswrapper[2575]: I0707 17:40:40.070359 2575 kubelet_node_status.go:78] "Attempting to register node" node="ip-10-0-129-39.ec2.internal" Jul 07 17:40:40.078637 ip-10-0-129-39 kubenswrapper[2575]: I0707 17:40:40.078618 2575 kubelet_node_status.go:81] "Successfully registered node" node="ip-10-0-129-39.ec2.internal" Jul 07 17:40:40.078711 ip-10-0-129-39 kubenswrapper[2575]: E0707 17:40:40.078640 2575 kubelet_node_status.go:536] "Error updating node status, will retry" err="error getting node \"ip-10-0-129-39.ec2.internal\": node \"ip-10-0-129-39.ec2.internal\" not found" Jul 07 17:40:40.090464 ip-10-0-129-39 kubenswrapper[2575]: E0707 17:40:40.090446 2575 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-129-39.ec2.internal\" not found" Jul 07 17:40:40.168113 ip-10-0-129-39 kubenswrapper[2575]: I0707 17:40:40.168094 2575 kubelet.go:2535] "SyncLoop ADD" source="file" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-39.ec2.internal","kube-system/kube-apiserver-proxy-ip-10-0-129-39.ec2.internal"] Jul 07 17:40:40.168204 ip-10-0-129-39 kubenswrapper[2575]: I0707 17:40:40.168160 2575 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 07 17:40:40.169729 ip-10-0-129-39 kubenswrapper[2575]: I0707 17:40:40.169714 2575 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-129-39.ec2.internal" event="NodeHasSufficientMemory" Jul 07 17:40:40.169786 ip-10-0-129-39 kubenswrapper[2575]: I0707 17:40:40.169741 2575 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-129-39.ec2.internal" event="NodeHasNoDiskPressure" Jul 07 17:40:40.169786 ip-10-0-129-39 kubenswrapper[2575]: I0707 17:40:40.169751 2575 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-129-39.ec2.internal" event="NodeHasSufficientPID" Jul 07 17:40:40.170854 ip-10-0-129-39 kubenswrapper[2575]: I0707 17:40:40.170843 2575 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 07 17:40:40.170969 ip-10-0-129-39 kubenswrapper[2575]: I0707 17:40:40.170954 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-39.ec2.internal" Jul 07 17:40:40.171014 ip-10-0-129-39 kubenswrapper[2575]: I0707 17:40:40.170990 2575 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 07 17:40:40.171515 ip-10-0-129-39 kubenswrapper[2575]: I0707 17:40:40.171484 2575 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-129-39.ec2.internal" event="NodeHasSufficientMemory" Jul 07 17:40:40.171610 ip-10-0-129-39 kubenswrapper[2575]: I0707 17:40:40.171523 2575 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-129-39.ec2.internal" event="NodeHasNoDiskPressure" Jul 07 17:40:40.171610 ip-10-0-129-39 kubenswrapper[2575]: I0707 17:40:40.171538 2575 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-129-39.ec2.internal" event="NodeHasSufficientPID" Jul 07 17:40:40.171715 ip-10-0-129-39 kubenswrapper[2575]: I0707 17:40:40.171522 2575 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-129-39.ec2.internal" event="NodeHasSufficientMemory" Jul 07 17:40:40.171715 ip-10-0-129-39 kubenswrapper[2575]: I0707 17:40:40.171666 2575 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-129-39.ec2.internal" event="NodeHasNoDiskPressure" Jul 07 17:40:40.171715 ip-10-0-129-39 kubenswrapper[2575]: I0707 17:40:40.171685 2575 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-129-39.ec2.internal" event="NodeHasSufficientPID" Jul 07 17:40:40.172654 ip-10-0-129-39 kubenswrapper[2575]: I0707 17:40:40.172638 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-129-39.ec2.internal" Jul 07 17:40:40.172742 ip-10-0-129-39 kubenswrapper[2575]: I0707 17:40:40.172669 2575 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 07 17:40:40.173723 ip-10-0-129-39 kubenswrapper[2575]: I0707 17:40:40.173708 2575 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-129-39.ec2.internal" event="NodeHasSufficientMemory" Jul 07 17:40:40.173811 ip-10-0-129-39 kubenswrapper[2575]: I0707 17:40:40.173738 2575 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-129-39.ec2.internal" event="NodeHasNoDiskPressure" Jul 07 17:40:40.173811 ip-10-0-129-39 kubenswrapper[2575]: I0707 17:40:40.173752 2575 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-129-39.ec2.internal" event="NodeHasSufficientPID" Jul 07 17:40:40.190524 ip-10-0-129-39 kubenswrapper[2575]: E0707 17:40:40.190495 2575 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-129-39.ec2.internal\" not found" Jul 07 17:40:40.204186 ip-10-0-129-39 kubenswrapper[2575]: E0707 17:40:40.204167 2575 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-129-39.ec2.internal\" not found" node="ip-10-0-129-39.ec2.internal" Jul 07 17:40:40.209870 ip-10-0-129-39 kubenswrapper[2575]: I0707 17:40:40.209850 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/2f2a5517e65a2e6eea49dbce93563b2c-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-129-39.ec2.internal\" (UID: \"2f2a5517e65a2e6eea49dbce93563b2c\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-39.ec2.internal" Jul 07 17:40:40.209948 ip-10-0-129-39 kubenswrapper[2575]: I0707 17:40:40.209875 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/2f2a5517e65a2e6eea49dbce93563b2c-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-129-39.ec2.internal\" (UID: \"2f2a5517e65a2e6eea49dbce93563b2c\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-39.ec2.internal" Jul 07 17:40:40.209948 ip-10-0-129-39 kubenswrapper[2575]: I0707 17:40:40.209890 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/9178fa6a4c12c534b2a4b3275d3b9775-config\") pod \"kube-apiserver-proxy-ip-10-0-129-39.ec2.internal\" (UID: \"9178fa6a4c12c534b2a4b3275d3b9775\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-129-39.ec2.internal" Jul 07 17:40:40.210368 ip-10-0-129-39 kubenswrapper[2575]: E0707 17:40:40.210348 2575 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-129-39.ec2.internal\" not found" node="ip-10-0-129-39.ec2.internal" Jul 07 17:40:40.291359 ip-10-0-129-39 kubenswrapper[2575]: E0707 17:40:40.291324 2575 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-129-39.ec2.internal\" not found" Jul 07 17:40:40.310666 ip-10-0-129-39 kubenswrapper[2575]: I0707 17:40:40.310640 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/2f2a5517e65a2e6eea49dbce93563b2c-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-129-39.ec2.internal\" (UID: \"2f2a5517e65a2e6eea49dbce93563b2c\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-39.ec2.internal" Jul 07 17:40:40.310769 ip-10-0-129-39 kubenswrapper[2575]: I0707 17:40:40.310695 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/2f2a5517e65a2e6eea49dbce93563b2c-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-129-39.ec2.internal\" (UID: \"2f2a5517e65a2e6eea49dbce93563b2c\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-39.ec2.internal" Jul 07 17:40:40.310769 ip-10-0-129-39 kubenswrapper[2575]: I0707 17:40:40.310724 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/2f2a5517e65a2e6eea49dbce93563b2c-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-129-39.ec2.internal\" (UID: \"2f2a5517e65a2e6eea49dbce93563b2c\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-39.ec2.internal" Jul 07 17:40:40.310769 ip-10-0-129-39 kubenswrapper[2575]: I0707 17:40:40.310745 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/9178fa6a4c12c534b2a4b3275d3b9775-config\") pod \"kube-apiserver-proxy-ip-10-0-129-39.ec2.internal\" (UID: \"9178fa6a4c12c534b2a4b3275d3b9775\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-129-39.ec2.internal" Jul 07 17:40:40.310878 ip-10-0-129-39 kubenswrapper[2575]: I0707 17:40:40.310777 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/9178fa6a4c12c534b2a4b3275d3b9775-config\") pod \"kube-apiserver-proxy-ip-10-0-129-39.ec2.internal\" (UID: \"9178fa6a4c12c534b2a4b3275d3b9775\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-129-39.ec2.internal" Jul 07 17:40:40.310878 ip-10-0-129-39 kubenswrapper[2575]: I0707 17:40:40.310800 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/2f2a5517e65a2e6eea49dbce93563b2c-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-129-39.ec2.internal\" (UID: \"2f2a5517e65a2e6eea49dbce93563b2c\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-39.ec2.internal" Jul 07 17:40:40.391992 ip-10-0-129-39 kubenswrapper[2575]: E0707 17:40:40.391933 2575 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-129-39.ec2.internal\" not found" Jul 07 17:40:40.492497 ip-10-0-129-39 kubenswrapper[2575]: E0707 17:40:40.492461 2575 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-129-39.ec2.internal\" not found" Jul 07 17:40:40.507614 ip-10-0-129-39 kubenswrapper[2575]: I0707 17:40:40.507595 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-39.ec2.internal" Jul 07 17:40:40.513376 ip-10-0-129-39 kubenswrapper[2575]: I0707 17:40:40.513355 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-129-39.ec2.internal" Jul 07 17:40:40.593164 ip-10-0-129-39 kubenswrapper[2575]: E0707 17:40:40.593129 2575 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-129-39.ec2.internal\" not found" Jul 07 17:40:40.693586 ip-10-0-129-39 kubenswrapper[2575]: E0707 17:40:40.693526 2575 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-129-39.ec2.internal\" not found" Jul 07 17:40:40.793993 ip-10-0-129-39 kubenswrapper[2575]: E0707 17:40:40.793974 2575 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-129-39.ec2.internal\" not found" Jul 07 17:40:40.851475 ip-10-0-129-39 kubenswrapper[2575]: I0707 17:40:40.851454 2575 transport.go:147] "Certificate rotation detected, shutting down client connections to start using new credentials" Jul 07 17:40:40.851848 ip-10-0-129-39 kubenswrapper[2575]: I0707 17:40:40.851601 2575 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.RuntimeClass" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received" Jul 07 17:40:40.894897 ip-10-0-129-39 kubenswrapper[2575]: E0707 17:40:40.894877 2575 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-129-39.ec2.internal\" not found" Jul 07 17:40:40.922456 ip-10-0-129-39 kubenswrapper[2575]: I0707 17:40:40.922425 2575 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-07-06 17:35:39 +0000 UTC" deadline="2028-03-09 10:27:35.822903039 +0000 UTC" Jul 07 17:40:40.922456 ip-10-0-129-39 kubenswrapper[2575]: I0707 17:40:40.922453 2575 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="14656h46m54.900452708s" Jul 07 17:40:40.995763 ip-10-0-129-39 kubenswrapper[2575]: E0707 17:40:40.995724 2575 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-129-39.ec2.internal\" not found" Jul 07 17:40:41.068309 ip-10-0-129-39 kubenswrapper[2575]: W0707 17:40:41.068266 2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2f2a5517e65a2e6eea49dbce93563b2c.slice/crio-9e8cea172e4e89937539ea6d5874a8ece100d3f5c09aeee76724ce5f1da87f7d WatchSource:0}: Error finding container 9e8cea172e4e89937539ea6d5874a8ece100d3f5c09aeee76724ce5f1da87f7d: Status 404 returned error can't find the container with id 9e8cea172e4e89937539ea6d5874a8ece100d3f5c09aeee76724ce5f1da87f7d Jul 07 17:40:41.072870 ip-10-0-129-39 kubenswrapper[2575]: I0707 17:40:41.072855 2575 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Jul 07 17:40:41.076342 ip-10-0-129-39 kubenswrapper[2575]: W0707 17:40:41.076321 2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9178fa6a4c12c534b2a4b3275d3b9775.slice/crio-6f1fe83224870054bbc02f0b0ec20d51b89bf7d51e45c9b2025f0ed6e97e518f WatchSource:0}: Error finding container 6f1fe83224870054bbc02f0b0ec20d51b89bf7d51e45c9b2025f0ed6e97e518f: Status 404 returned error can't find the container with id 6f1fe83224870054bbc02f0b0ec20d51b89bf7d51e45c9b2025f0ed6e97e518f Jul 07 17:40:41.078451 ip-10-0-129-39 kubenswrapper[2575]: I0707 17:40:41.078432 2575 reflector.go:439] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:160" Jul 07 17:40:41.096464 ip-10-0-129-39 kubenswrapper[2575]: E0707 17:40:41.096442 2575 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-129-39.ec2.internal\" not found" Jul 07 17:40:41.196958 ip-10-0-129-39 kubenswrapper[2575]: E0707 17:40:41.196931 2575 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-129-39.ec2.internal\" not found" Jul 07 17:40:41.297521 ip-10-0-129-39 kubenswrapper[2575]: E0707 17:40:41.297450 2575 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-129-39.ec2.internal\" not found" Jul 07 17:40:41.355294 ip-10-0-129-39 kubenswrapper[2575]: I0707 17:40:41.355276 2575 reflector.go:439] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:160" Jul 07 17:40:41.398330 ip-10-0-129-39 kubenswrapper[2575]: E0707 17:40:41.398309 2575 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-129-39.ec2.internal\" not found" Jul 07 17:40:41.410496 ip-10-0-129-39 kubenswrapper[2575]: I0707 17:40:41.410477 2575 reflector.go:439] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:160" Jul 07 17:40:41.508491 ip-10-0-129-39 kubenswrapper[2575]: I0707 17:40:41.508470 2575 kubelet.go:3248] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-129-39.ec2.internal" Jul 07 17:40:41.518288 ip-10-0-129-39 kubenswrapper[2575]: I0707 17:40:41.518273 2575 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Jul 07 17:40:41.519439 ip-10-0-129-39 kubenswrapper[2575]: I0707 17:40:41.519427 2575 kubelet.go:3248] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-39.ec2.internal" Jul 07 17:40:41.532569 ip-10-0-129-39 kubenswrapper[2575]: I0707 17:40:41.532554 2575 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Jul 07 17:40:41.892211 ip-10-0-129-39 kubenswrapper[2575]: I0707 17:40:41.892189 2575 apiserver.go:52] "Watching apiserver" Jul 07 17:40:41.898176 ip-10-0-129-39 kubenswrapper[2575]: I0707 17:40:41.898147 2575 reflector.go:439] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66" Jul 07 17:40:41.898591 ip-10-0-129-39 kubenswrapper[2575]: I0707 17:40:41.898560 2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-dns/node-resolver-fc7qv","openshift-multus/multus-additional-cni-plugins-9m48z","openshift-multus/multus-rs8sg","openshift-ovn-kubernetes/ovnkube-node-qgbk5","kube-system/konnectivity-agent-hr726","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-btqqg","openshift-cluster-node-tuning-operator/tuned-7hrdq","openshift-image-registry/node-ca-l6pnn","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-39.ec2.internal","openshift-multus/network-metrics-daemon-gz97j","openshift-network-diagnostics/network-check-target-vzf4m","openshift-network-operator/iptables-alerter-5vc77","kube-system/global-pull-secret-syncer-76r79","kube-system/kube-apiserver-proxy-ip-10-0-129-39.ec2.internal"] Jul 07 17:40:41.900014 ip-10-0-129-39 kubenswrapper[2575]: I0707 17:40:41.899949 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-7hrdq" Jul 07 17:40:41.901126 ip-10-0-129-39 kubenswrapper[2575]: I0707 17:40:41.901067 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-76r79" Jul 07 17:40:41.901258 ip-10-0-129-39 kubenswrapper[2575]: E0707 17:40:41.901123 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-76r79" podUID="8344b558-69d0-43b3-9aac-0cea0d5379f3" Jul 07 17:40:41.902213 ip-10-0-129-39 kubenswrapper[2575]: I0707 17:40:41.901902 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-4kmsj\"" Jul 07 17:40:41.902322 ip-10-0-129-39 kubenswrapper[2575]: I0707 17:40:41.902219 2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\"" Jul 07 17:40:41.902428 ip-10-0-129-39 kubenswrapper[2575]: I0707 17:40:41.902413 2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\"" Jul 07 17:40:41.902597 ip-10-0-129-39 kubenswrapper[2575]: I0707 17:40:41.902578 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-rs8sg" Jul 07 17:40:41.904144 ip-10-0-129-39 kubenswrapper[2575]: I0707 17:40:41.903735 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-qgbk5" Jul 07 17:40:41.904938 ip-10-0-129-39 kubenswrapper[2575]: I0707 17:40:41.904767 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"default-dockercfg-khgq9\"" Jul 07 17:40:41.905039 ip-10-0-129-39 kubenswrapper[2575]: I0707 17:40:41.905010 2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\"" Jul 07 17:40:41.907492 ip-10-0-129-39 kubenswrapper[2575]: I0707 17:40:41.906742 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-hr726" Jul 07 17:40:41.908452 ip-10-0-129-39 kubenswrapper[2575]: I0707 17:40:41.908432 2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\"" Jul 07 17:40:41.908567 ip-10-0-129-39 kubenswrapper[2575]: I0707 17:40:41.908475 2575 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kubelet-serving" Jul 07 17:40:41.908623 ip-10-0-129-39 kubenswrapper[2575]: I0707 17:40:41.908569 2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\"" Jul 07 17:40:41.908679 ip-10-0-129-39 kubenswrapper[2575]: I0707 17:40:41.908629 2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\"" Jul 07 17:40:41.908679 ip-10-0-129-39 kubenswrapper[2575]: I0707 17:40:41.908657 2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\"" Jul 07 17:40:41.908781 ip-10-0-129-39 kubenswrapper[2575]: I0707 17:40:41.908738 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\"" Jul 07 17:40:41.908867 ip-10-0-129-39 kubenswrapper[2575]: I0707 17:40:41.908841 2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\"" Jul 07 17:40:41.908934 ip-10-0-129-39 kubenswrapper[2575]: I0707 17:40:41.908905 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-c7nr6\"" Jul 07 17:40:41.908988 ip-10-0-129-39 kubenswrapper[2575]: I0707 17:40:41.908965 2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\"" Jul 07 17:40:41.908988 ip-10-0-129-39 kubenswrapper[2575]: I0707 17:40:41.908972 2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\"" Jul 07 17:40:41.909084 ip-10-0-129-39 kubenswrapper[2575]: I0707 17:40:41.909003 2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\"" Jul 07 17:40:41.910711 ip-10-0-129-39 kubenswrapper[2575]: I0707 17:40:41.910588 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-btqqg" Jul 07 17:40:41.911488 ip-10-0-129-39 kubenswrapper[2575]: I0707 17:40:41.911407 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"default-dockercfg-kl6jn\"" Jul 07 17:40:41.911720 ip-10-0-129-39 kubenswrapper[2575]: I0707 17:40:41.911665 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\"" Jul 07 17:40:41.911812 ip-10-0-129-39 kubenswrapper[2575]: I0707 17:40:41.911756 2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\"" Jul 07 17:40:41.911915 ip-10-0-129-39 kubenswrapper[2575]: I0707 17:40:41.911896 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-l6pnn" Jul 07 17:40:41.912605 ip-10-0-129-39 kubenswrapper[2575]: I0707 17:40:41.912583 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\"" Jul 07 17:40:41.912841 ip-10-0-129-39 kubenswrapper[2575]: I0707 17:40:41.912819 2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\"" Jul 07 17:40:41.912841 ip-10-0-129-39 kubenswrapper[2575]: I0707 17:40:41.912838 2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\"" Jul 07 17:40:41.912963 ip-10-0-129-39 kubenswrapper[2575]: I0707 17:40:41.912822 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-c4jsc\"" Jul 07 17:40:41.913577 ip-10-0-129-39 kubenswrapper[2575]: I0707 17:40:41.913402 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-fc7qv" Jul 07 17:40:41.913980 ip-10-0-129-39 kubenswrapper[2575]: I0707 17:40:41.913962 2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\"" Jul 07 17:40:41.914210 ip-10-0-129-39 kubenswrapper[2575]: I0707 17:40:41.913968 2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\"" Jul 07 17:40:41.914210 ip-10-0-129-39 kubenswrapper[2575]: I0707 17:40:41.914090 2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\"" Jul 07 17:40:41.914210 ip-10-0-129-39 kubenswrapper[2575]: I0707 17:40:41.914158 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-t4tpc\"" Jul 07 17:40:41.914591 ip-10-0-129-39 kubenswrapper[2575]: I0707 17:40:41.914572 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-9m48z" Jul 07 17:40:41.915811 ip-10-0-129-39 kubenswrapper[2575]: I0707 17:40:41.915794 2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\"" Jul 07 17:40:41.916062 ip-10-0-129-39 kubenswrapper[2575]: I0707 17:40:41.916045 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-xmmjs\"" Jul 07 17:40:41.916362 ip-10-0-129-39 kubenswrapper[2575]: I0707 17:40:41.916348 2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\"" Jul 07 17:40:41.916595 ip-10-0-129-39 kubenswrapper[2575]: I0707 17:40:41.916579 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-gz97j" Jul 07 17:40:41.916666 ip-10-0-129-39 kubenswrapper[2575]: I0707 17:40:41.916614 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-4zfhw\"" Jul 07 17:40:41.916666 ip-10-0-129-39 kubenswrapper[2575]: E0707 17:40:41.916647 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-gz97j" podUID="d587896d-9156-4b75-aa70-27962a977051" Jul 07 17:40:41.916900 ip-10-0-129-39 kubenswrapper[2575]: I0707 17:40:41.916882 2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\"" Jul 07 17:40:41.916958 ip-10-0-129-39 kubenswrapper[2575]: I0707 17:40:41.916909 2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\"" Jul 07 17:40:41.917337 ip-10-0-129-39 kubenswrapper[2575]: I0707 17:40:41.917316 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-vzf4m" Jul 07 17:40:41.917454 ip-10-0-129-39 kubenswrapper[2575]: E0707 17:40:41.917432 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-vzf4m" podUID="5d33c1cb-fcda-490d-9694-b1d1eb16b8da" Jul 07 17:40:41.918826 ip-10-0-129-39 kubenswrapper[2575]: I0707 17:40:41.918808 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-5vc77" Jul 07 17:40:41.919025 ip-10-0-129-39 kubenswrapper[2575]: I0707 17:40:41.919009 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/2d51e575-aadb-4cdd-a6cf-1ef119b49029-lib-modules\") pod \"tuned-7hrdq\" (UID: \"2d51e575-aadb-4cdd-a6cf-1ef119b49029\") " pod="openshift-cluster-node-tuning-operator/tuned-7hrdq" Jul 07 17:40:41.919100 ip-10-0-129-39 kubenswrapper[2575]: I0707 17:40:41.919039 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/4f2c6f8d-3b44-4f4f-8d6b-433f013a867f-konnectivity-ca\") pod \"konnectivity-agent-hr726\" (UID: \"4f2c6f8d-3b44-4f4f-8d6b-433f013a867f\") " pod="kube-system/konnectivity-agent-hr726" Jul 07 17:40:41.919100 ip-10-0-129-39 kubenswrapper[2575]: I0707 17:40:41.919065 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/a5765746-152c-4b60-b664-2dc9c75bd013-kubelet-dir\") pod \"aws-ebs-csi-driver-node-btqqg\" (UID: \"a5765746-152c-4b60-b664-2dc9c75bd013\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-btqqg" Jul 07 17:40:41.919100 ip-10-0-129-39 kubenswrapper[2575]: I0707 17:40:41.919088 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/2d51e575-aadb-4cdd-a6cf-1ef119b49029-run\") pod \"tuned-7hrdq\" (UID: \"2d51e575-aadb-4cdd-a6cf-1ef119b49029\") " pod="openshift-cluster-node-tuning-operator/tuned-7hrdq" Jul 07 17:40:41.919242 ip-10-0-129-39 kubenswrapper[2575]: I0707 17:40:41.919110 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/ad91fa94-7ae7-4245-90ee-78701b164dd2-host-run-netns\") pod \"multus-rs8sg\" (UID: \"ad91fa94-7ae7-4245-90ee-78701b164dd2\") " pod="openshift-multus/multus-rs8sg" Jul 07 17:40:41.919242 ip-10-0-129-39 kubenswrapper[2575]: I0707 17:40:41.919132 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/ad91fa94-7ae7-4245-90ee-78701b164dd2-host-var-lib-kubelet\") pod \"multus-rs8sg\" (UID: \"ad91fa94-7ae7-4245-90ee-78701b164dd2\") " pod="openshift-multus/multus-rs8sg" Jul 07 17:40:41.919242 ip-10-0-129-39 kubenswrapper[2575]: I0707 17:40:41.919157 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/b8eb06e8-1aea-4b07-801e-fe9f5f86224f-etc-openvswitch\") pod \"ovnkube-node-qgbk5\" (UID: \"b8eb06e8-1aea-4b07-801e-fe9f5f86224f\") " pod="openshift-ovn-kubernetes/ovnkube-node-qgbk5" Jul 07 17:40:41.919242 ip-10-0-129-39 kubenswrapper[2575]: I0707 17:40:41.919180 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/b8eb06e8-1aea-4b07-801e-fe9f5f86224f-log-socket\") pod \"ovnkube-node-qgbk5\" (UID: \"b8eb06e8-1aea-4b07-801e-fe9f5f86224f\") " pod="openshift-ovn-kubernetes/ovnkube-node-qgbk5" Jul 07 17:40:41.919242 ip-10-0-129-39 kubenswrapper[2575]: I0707 17:40:41.919202 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/2d51e575-aadb-4cdd-a6cf-1ef119b49029-sys\") pod \"tuned-7hrdq\" (UID: \"2d51e575-aadb-4cdd-a6cf-1ef119b49029\") " pod="openshift-cluster-node-tuning-operator/tuned-7hrdq" Jul 07 17:40:41.919242 ip-10-0-129-39 kubenswrapper[2575]: I0707 17:40:41.919225 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/b8eb06e8-1aea-4b07-801e-fe9f5f86224f-var-lib-openvswitch\") pod \"ovnkube-node-qgbk5\" (UID: \"b8eb06e8-1aea-4b07-801e-fe9f5f86224f\") " pod="openshift-ovn-kubernetes/ovnkube-node-qgbk5" Jul 07 17:40:41.919534 ip-10-0-129-39 kubenswrapper[2575]: I0707 17:40:41.919247 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/b8eb06e8-1aea-4b07-801e-fe9f5f86224f-host-cni-bin\") pod \"ovnkube-node-qgbk5\" (UID: \"b8eb06e8-1aea-4b07-801e-fe9f5f86224f\") " pod="openshift-ovn-kubernetes/ovnkube-node-qgbk5" Jul 07 17:40:41.919534 ip-10-0-129-39 kubenswrapper[2575]: I0707 17:40:41.919269 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/4f2c6f8d-3b44-4f4f-8d6b-433f013a867f-agent-certs\") pod \"konnectivity-agent-hr726\" (UID: \"4f2c6f8d-3b44-4f4f-8d6b-433f013a867f\") " pod="kube-system/konnectivity-agent-hr726" Jul 07 17:40:41.919534 ip-10-0-129-39 kubenswrapper[2575]: I0707 17:40:41.919308 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/ad91fa94-7ae7-4245-90ee-78701b164dd2-os-release\") pod \"multus-rs8sg\" (UID: \"ad91fa94-7ae7-4245-90ee-78701b164dd2\") " pod="openshift-multus/multus-rs8sg" Jul 07 17:40:41.919534 ip-10-0-129-39 kubenswrapper[2575]: I0707 17:40:41.919366 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/b8eb06e8-1aea-4b07-801e-fe9f5f86224f-node-log\") pod \"ovnkube-node-qgbk5\" (UID: \"b8eb06e8-1aea-4b07-801e-fe9f5f86224f\") " pod="openshift-ovn-kubernetes/ovnkube-node-qgbk5" Jul 07 17:40:41.919534 ip-10-0-129-39 kubenswrapper[2575]: I0707 17:40:41.919399 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/a5765746-152c-4b60-b664-2dc9c75bd013-socket-dir\") pod \"aws-ebs-csi-driver-node-btqqg\" (UID: \"a5765746-152c-4b60-b664-2dc9c75bd013\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-btqqg" Jul 07 17:40:41.919534 ip-10-0-129-39 kubenswrapper[2575]: I0707 17:40:41.919438 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-r4vhp\" (UniqueName: \"kubernetes.io/projected/a5765746-152c-4b60-b664-2dc9c75bd013-kube-api-access-r4vhp\") pod \"aws-ebs-csi-driver-node-btqqg\" (UID: \"a5765746-152c-4b60-b664-2dc9c75bd013\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-btqqg" Jul 07 17:40:41.919534 ip-10-0-129-39 kubenswrapper[2575]: I0707 17:40:41.919473 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/8344b558-69d0-43b3-9aac-0cea0d5379f3-kubelet-config\") pod \"global-pull-secret-syncer-76r79\" (UID: \"8344b558-69d0-43b3-9aac-0cea0d5379f3\") " pod="kube-system/global-pull-secret-syncer-76r79" Jul 07 17:40:41.919534 ip-10-0-129-39 kubenswrapper[2575]: I0707 17:40:41.919497 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/8344b558-69d0-43b3-9aac-0cea0d5379f3-original-pull-secret\") pod \"global-pull-secret-syncer-76r79\" (UID: \"8344b558-69d0-43b3-9aac-0cea0d5379f3\") " pod="kube-system/global-pull-secret-syncer-76r79" Jul 07 17:40:41.919877 ip-10-0-129-39 kubenswrapper[2575]: I0707 17:40:41.919541 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/ad91fa94-7ae7-4245-90ee-78701b164dd2-system-cni-dir\") pod \"multus-rs8sg\" (UID: \"ad91fa94-7ae7-4245-90ee-78701b164dd2\") " pod="openshift-multus/multus-rs8sg" Jul 07 17:40:41.919877 ip-10-0-129-39 kubenswrapper[2575]: I0707 17:40:41.919563 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/ad91fa94-7ae7-4245-90ee-78701b164dd2-cni-binary-copy\") pod \"multus-rs8sg\" (UID: \"ad91fa94-7ae7-4245-90ee-78701b164dd2\") " pod="openshift-multus/multus-rs8sg" Jul 07 17:40:41.919877 ip-10-0-129-39 kubenswrapper[2575]: I0707 17:40:41.919586 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/ad91fa94-7ae7-4245-90ee-78701b164dd2-host-run-k8s-cni-cncf-io\") pod \"multus-rs8sg\" (UID: \"ad91fa94-7ae7-4245-90ee-78701b164dd2\") " pod="openshift-multus/multus-rs8sg" Jul 07 17:40:41.919877 ip-10-0-129-39 kubenswrapper[2575]: I0707 17:40:41.919624 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/ad91fa94-7ae7-4245-90ee-78701b164dd2-host-var-lib-cni-bin\") pod \"multus-rs8sg\" (UID: \"ad91fa94-7ae7-4245-90ee-78701b164dd2\") " pod="openshift-multus/multus-rs8sg" Jul 07 17:40:41.919877 ip-10-0-129-39 kubenswrapper[2575]: I0707 17:40:41.919647 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/ad91fa94-7ae7-4245-90ee-78701b164dd2-host-run-multus-certs\") pod \"multus-rs8sg\" (UID: \"ad91fa94-7ae7-4245-90ee-78701b164dd2\") " pod="openshift-multus/multus-rs8sg" Jul 07 17:40:41.919877 ip-10-0-129-39 kubenswrapper[2575]: I0707 17:40:41.919671 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/b8eb06e8-1aea-4b07-801e-fe9f5f86224f-host-run-netns\") pod \"ovnkube-node-qgbk5\" (UID: \"b8eb06e8-1aea-4b07-801e-fe9f5f86224f\") " pod="openshift-ovn-kubernetes/ovnkube-node-qgbk5" Jul 07 17:40:41.919877 ip-10-0-129-39 kubenswrapper[2575]: I0707 17:40:41.919703 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/2d51e575-aadb-4cdd-a6cf-1ef119b49029-etc-tuned\") pod \"tuned-7hrdq\" (UID: \"2d51e575-aadb-4cdd-a6cf-1ef119b49029\") " pod="openshift-cluster-node-tuning-operator/tuned-7hrdq" Jul 07 17:40:41.919877 ip-10-0-129-39 kubenswrapper[2575]: I0707 17:40:41.919764 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/ad91fa94-7ae7-4245-90ee-78701b164dd2-cnibin\") pod \"multus-rs8sg\" (UID: \"ad91fa94-7ae7-4245-90ee-78701b164dd2\") " pod="openshift-multus/multus-rs8sg" Jul 07 17:40:41.919877 ip-10-0-129-39 kubenswrapper[2575]: I0707 17:40:41.919792 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/b8eb06e8-1aea-4b07-801e-fe9f5f86224f-host-slash\") pod \"ovnkube-node-qgbk5\" (UID: \"b8eb06e8-1aea-4b07-801e-fe9f5f86224f\") " pod="openshift-ovn-kubernetes/ovnkube-node-qgbk5" Jul 07 17:40:41.919877 ip-10-0-129-39 kubenswrapper[2575]: I0707 17:40:41.919812 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/b8eb06e8-1aea-4b07-801e-fe9f5f86224f-host-run-ovn-kubernetes\") pod \"ovnkube-node-qgbk5\" (UID: \"b8eb06e8-1aea-4b07-801e-fe9f5f86224f\") " pod="openshift-ovn-kubernetes/ovnkube-node-qgbk5" Jul 07 17:40:41.919877 ip-10-0-129-39 kubenswrapper[2575]: I0707 17:40:41.919823 2575 reflector.go:439] "Caches populated" logger="kubernetes.io/kubelet-serving" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" Jul 07 17:40:41.919877 ip-10-0-129-39 kubenswrapper[2575]: I0707 17:40:41.919834 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/b8eb06e8-1aea-4b07-801e-fe9f5f86224f-ovnkube-script-lib\") pod \"ovnkube-node-qgbk5\" (UID: \"b8eb06e8-1aea-4b07-801e-fe9f5f86224f\") " pod="openshift-ovn-kubernetes/ovnkube-node-qgbk5" Jul 07 17:40:41.919877 ip-10-0-129-39 kubenswrapper[2575]: I0707 17:40:41.919857 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wh2cb\" (UniqueName: \"kubernetes.io/projected/b8eb06e8-1aea-4b07-801e-fe9f5f86224f-kube-api-access-wh2cb\") pod \"ovnkube-node-qgbk5\" (UID: \"b8eb06e8-1aea-4b07-801e-fe9f5f86224f\") " pod="openshift-ovn-kubernetes/ovnkube-node-qgbk5" Jul 07 17:40:41.920587 ip-10-0-129-39 kubenswrapper[2575]: I0707 17:40:41.919897 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/2d51e575-aadb-4cdd-a6cf-1ef119b49029-etc-sysconfig\") pod \"tuned-7hrdq\" (UID: \"2d51e575-aadb-4cdd-a6cf-1ef119b49029\") " pod="openshift-cluster-node-tuning-operator/tuned-7hrdq" Jul 07 17:40:41.920587 ip-10-0-129-39 kubenswrapper[2575]: I0707 17:40:41.919923 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/ad91fa94-7ae7-4245-90ee-78701b164dd2-multus-cni-dir\") pod \"multus-rs8sg\" (UID: \"ad91fa94-7ae7-4245-90ee-78701b164dd2\") " pod="openshift-multus/multus-rs8sg" Jul 07 17:40:41.920587 ip-10-0-129-39 kubenswrapper[2575]: I0707 17:40:41.919945 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/a5765746-152c-4b60-b664-2dc9c75bd013-etc-selinux\") pod \"aws-ebs-csi-driver-node-btqqg\" (UID: \"a5765746-152c-4b60-b664-2dc9c75bd013\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-btqqg" Jul 07 17:40:41.920587 ip-10-0-129-39 kubenswrapper[2575]: I0707 17:40:41.919967 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/b8eb06e8-1aea-4b07-801e-fe9f5f86224f-env-overrides\") pod \"ovnkube-node-qgbk5\" (UID: \"b8eb06e8-1aea-4b07-801e-fe9f5f86224f\") " pod="openshift-ovn-kubernetes/ovnkube-node-qgbk5" Jul 07 17:40:41.920587 ip-10-0-129-39 kubenswrapper[2575]: I0707 17:40:41.919998 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/2d51e575-aadb-4cdd-a6cf-1ef119b49029-etc-modprobe-d\") pod \"tuned-7hrdq\" (UID: \"2d51e575-aadb-4cdd-a6cf-1ef119b49029\") " pod="openshift-cluster-node-tuning-operator/tuned-7hrdq" Jul 07 17:40:41.920587 ip-10-0-129-39 kubenswrapper[2575]: I0707 17:40:41.920020 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/ad91fa94-7ae7-4245-90ee-78701b164dd2-multus-conf-dir\") pod \"multus-rs8sg\" (UID: \"ad91fa94-7ae7-4245-90ee-78701b164dd2\") " pod="openshift-multus/multus-rs8sg" Jul 07 17:40:41.920587 ip-10-0-129-39 kubenswrapper[2575]: I0707 17:40:41.920043 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/b8eb06e8-1aea-4b07-801e-fe9f5f86224f-host-kubelet\") pod \"ovnkube-node-qgbk5\" (UID: \"b8eb06e8-1aea-4b07-801e-fe9f5f86224f\") " pod="openshift-ovn-kubernetes/ovnkube-node-qgbk5" Jul 07 17:40:41.920587 ip-10-0-129-39 kubenswrapper[2575]: I0707 17:40:41.920065 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vtmvs\" (UniqueName: \"kubernetes.io/projected/2d51e575-aadb-4cdd-a6cf-1ef119b49029-kube-api-access-vtmvs\") pod \"tuned-7hrdq\" (UID: \"2d51e575-aadb-4cdd-a6cf-1ef119b49029\") " pod="openshift-cluster-node-tuning-operator/tuned-7hrdq" Jul 07 17:40:41.920587 ip-10-0-129-39 kubenswrapper[2575]: I0707 17:40:41.920097 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/8344b558-69d0-43b3-9aac-0cea0d5379f3-dbus\") pod \"global-pull-secret-syncer-76r79\" (UID: \"8344b558-69d0-43b3-9aac-0cea0d5379f3\") " pod="kube-system/global-pull-secret-syncer-76r79" Jul 07 17:40:41.920587 ip-10-0-129-39 kubenswrapper[2575]: I0707 17:40:41.920151 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/ad91fa94-7ae7-4245-90ee-78701b164dd2-host-var-lib-cni-multus\") pod \"multus-rs8sg\" (UID: \"ad91fa94-7ae7-4245-90ee-78701b164dd2\") " pod="openshift-multus/multus-rs8sg" Jul 07 17:40:41.920587 ip-10-0-129-39 kubenswrapper[2575]: I0707 17:40:41.920182 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/ad91fa94-7ae7-4245-90ee-78701b164dd2-hostroot\") pod \"multus-rs8sg\" (UID: \"ad91fa94-7ae7-4245-90ee-78701b164dd2\") " pod="openshift-multus/multus-rs8sg" Jul 07 17:40:41.920587 ip-10-0-129-39 kubenswrapper[2575]: I0707 17:40:41.920213 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/b8eb06e8-1aea-4b07-801e-fe9f5f86224f-run-openvswitch\") pod \"ovnkube-node-qgbk5\" (UID: \"b8eb06e8-1aea-4b07-801e-fe9f5f86224f\") " pod="openshift-ovn-kubernetes/ovnkube-node-qgbk5" Jul 07 17:40:41.920587 ip-10-0-129-39 kubenswrapper[2575]: I0707 17:40:41.920236 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/b8eb06e8-1aea-4b07-801e-fe9f5f86224f-host-cni-netd\") pod \"ovnkube-node-qgbk5\" (UID: \"b8eb06e8-1aea-4b07-801e-fe9f5f86224f\") " pod="openshift-ovn-kubernetes/ovnkube-node-qgbk5" Jul 07 17:40:41.920587 ip-10-0-129-39 kubenswrapper[2575]: I0707 17:40:41.920277 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/b8eb06e8-1aea-4b07-801e-fe9f5f86224f-ovn-node-metrics-cert\") pod \"ovnkube-node-qgbk5\" (UID: \"b8eb06e8-1aea-4b07-801e-fe9f5f86224f\") " pod="openshift-ovn-kubernetes/ovnkube-node-qgbk5" Jul 07 17:40:41.920587 ip-10-0-129-39 kubenswrapper[2575]: I0707 17:40:41.920330 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/2d51e575-aadb-4cdd-a6cf-1ef119b49029-etc-sysctl-d\") pod \"tuned-7hrdq\" (UID: \"2d51e575-aadb-4cdd-a6cf-1ef119b49029\") " pod="openshift-cluster-node-tuning-operator/tuned-7hrdq" Jul 07 17:40:41.920587 ip-10-0-129-39 kubenswrapper[2575]: I0707 17:40:41.920370 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/2d51e575-aadb-4cdd-a6cf-1ef119b49029-etc-sysctl-conf\") pod \"tuned-7hrdq\" (UID: \"2d51e575-aadb-4cdd-a6cf-1ef119b49029\") " pod="openshift-cluster-node-tuning-operator/tuned-7hrdq" Jul 07 17:40:41.920587 ip-10-0-129-39 kubenswrapper[2575]: I0707 17:40:41.920402 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/2d51e575-aadb-4cdd-a6cf-1ef119b49029-etc-systemd\") pod \"tuned-7hrdq\" (UID: \"2d51e575-aadb-4cdd-a6cf-1ef119b49029\") " pod="openshift-cluster-node-tuning-operator/tuned-7hrdq" Jul 07 17:40:41.921364 ip-10-0-129-39 kubenswrapper[2575]: I0707 17:40:41.920422 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/2d51e575-aadb-4cdd-a6cf-1ef119b49029-tmp\") pod \"tuned-7hrdq\" (UID: \"2d51e575-aadb-4cdd-a6cf-1ef119b49029\") " pod="openshift-cluster-node-tuning-operator/tuned-7hrdq" Jul 07 17:40:41.921364 ip-10-0-129-39 kubenswrapper[2575]: I0707 17:40:41.920436 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/ad91fa94-7ae7-4245-90ee-78701b164dd2-etc-kubernetes\") pod \"multus-rs8sg\" (UID: \"ad91fa94-7ae7-4245-90ee-78701b164dd2\") " pod="openshift-multus/multus-rs8sg" Jul 07 17:40:41.921364 ip-10-0-129-39 kubenswrapper[2575]: I0707 17:40:41.920450 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8kv7w\" (UniqueName: \"kubernetes.io/projected/ad91fa94-7ae7-4245-90ee-78701b164dd2-kube-api-access-8kv7w\") pod \"multus-rs8sg\" (UID: \"ad91fa94-7ae7-4245-90ee-78701b164dd2\") " pod="openshift-multus/multus-rs8sg" Jul 07 17:40:41.921364 ip-10-0-129-39 kubenswrapper[2575]: I0707 17:40:41.920464 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/b8eb06e8-1aea-4b07-801e-fe9f5f86224f-run-ovn\") pod \"ovnkube-node-qgbk5\" (UID: \"b8eb06e8-1aea-4b07-801e-fe9f5f86224f\") " pod="openshift-ovn-kubernetes/ovnkube-node-qgbk5" Jul 07 17:40:41.921364 ip-10-0-129-39 kubenswrapper[2575]: I0707 17:40:41.920527 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/b8eb06e8-1aea-4b07-801e-fe9f5f86224f-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-qgbk5\" (UID: \"b8eb06e8-1aea-4b07-801e-fe9f5f86224f\") " pod="openshift-ovn-kubernetes/ovnkube-node-qgbk5" Jul 07 17:40:41.921364 ip-10-0-129-39 kubenswrapper[2575]: I0707 17:40:41.920553 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/b8eb06e8-1aea-4b07-801e-fe9f5f86224f-ovnkube-config\") pod \"ovnkube-node-qgbk5\" (UID: \"b8eb06e8-1aea-4b07-801e-fe9f5f86224f\") " pod="openshift-ovn-kubernetes/ovnkube-node-qgbk5" Jul 07 17:40:41.921364 ip-10-0-129-39 kubenswrapper[2575]: I0707 17:40:41.920603 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/a5765746-152c-4b60-b664-2dc9c75bd013-registration-dir\") pod \"aws-ebs-csi-driver-node-btqqg\" (UID: \"a5765746-152c-4b60-b664-2dc9c75bd013\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-btqqg" Jul 07 17:40:41.921364 ip-10-0-129-39 kubenswrapper[2575]: I0707 17:40:41.920639 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/2d51e575-aadb-4cdd-a6cf-1ef119b49029-host\") pod \"tuned-7hrdq\" (UID: \"2d51e575-aadb-4cdd-a6cf-1ef119b49029\") " pod="openshift-cluster-node-tuning-operator/tuned-7hrdq" Jul 07 17:40:41.921364 ip-10-0-129-39 kubenswrapper[2575]: I0707 17:40:41.920666 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/a5765746-152c-4b60-b664-2dc9c75bd013-sys-fs\") pod \"aws-ebs-csi-driver-node-btqqg\" (UID: \"a5765746-152c-4b60-b664-2dc9c75bd013\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-btqqg" Jul 07 17:40:41.921364 ip-10-0-129-39 kubenswrapper[2575]: I0707 17:40:41.920720 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/2d51e575-aadb-4cdd-a6cf-1ef119b49029-etc-kubernetes\") pod \"tuned-7hrdq\" (UID: \"2d51e575-aadb-4cdd-a6cf-1ef119b49029\") " pod="openshift-cluster-node-tuning-operator/tuned-7hrdq" Jul 07 17:40:41.921364 ip-10-0-129-39 kubenswrapper[2575]: I0707 17:40:41.920776 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/2d51e575-aadb-4cdd-a6cf-1ef119b49029-var-lib-kubelet\") pod \"tuned-7hrdq\" (UID: \"2d51e575-aadb-4cdd-a6cf-1ef119b49029\") " pod="openshift-cluster-node-tuning-operator/tuned-7hrdq" Jul 07 17:40:41.921364 ip-10-0-129-39 kubenswrapper[2575]: I0707 17:40:41.920823 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/ad91fa94-7ae7-4245-90ee-78701b164dd2-multus-daemon-config\") pod \"multus-rs8sg\" (UID: \"ad91fa94-7ae7-4245-90ee-78701b164dd2\") " pod="openshift-multus/multus-rs8sg" Jul 07 17:40:41.921364 ip-10-0-129-39 kubenswrapper[2575]: I0707 17:40:41.920850 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/a5765746-152c-4b60-b664-2dc9c75bd013-device-dir\") pod \"aws-ebs-csi-driver-node-btqqg\" (UID: \"a5765746-152c-4b60-b664-2dc9c75bd013\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-btqqg" Jul 07 17:40:41.921364 ip-10-0-129-39 kubenswrapper[2575]: I0707 17:40:41.920862 2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\"" Jul 07 17:40:41.921364 ip-10-0-129-39 kubenswrapper[2575]: I0707 17:40:41.920879 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/ad91fa94-7ae7-4245-90ee-78701b164dd2-multus-socket-dir-parent\") pod \"multus-rs8sg\" (UID: \"ad91fa94-7ae7-4245-90ee-78701b164dd2\") " pod="openshift-multus/multus-rs8sg" Jul 07 17:40:41.921364 ip-10-0-129-39 kubenswrapper[2575]: I0707 17:40:41.920905 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/b8eb06e8-1aea-4b07-801e-fe9f5f86224f-systemd-units\") pod \"ovnkube-node-qgbk5\" (UID: \"b8eb06e8-1aea-4b07-801e-fe9f5f86224f\") " pod="openshift-ovn-kubernetes/ovnkube-node-qgbk5" Jul 07 17:40:41.921364 ip-10-0-129-39 kubenswrapper[2575]: I0707 17:40:41.920927 2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\"" Jul 07 17:40:41.921364 ip-10-0-129-39 kubenswrapper[2575]: I0707 17:40:41.920998 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-dllg7\"" Jul 07 17:40:41.922267 ip-10-0-129-39 kubenswrapper[2575]: I0707 17:40:41.920928 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/b8eb06e8-1aea-4b07-801e-fe9f5f86224f-run-systemd\") pod \"ovnkube-node-qgbk5\" (UID: \"b8eb06e8-1aea-4b07-801e-fe9f5f86224f\") " pod="openshift-ovn-kubernetes/ovnkube-node-qgbk5" Jul 07 17:40:41.922267 ip-10-0-129-39 kubenswrapper[2575]: I0707 17:40:41.921152 2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\"" Jul 07 17:40:41.963558 ip-10-0-129-39 kubenswrapper[2575]: I0707 17:40:41.963536 2575 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kubelet-serving" csr="csr-j684j" Jul 07 17:40:41.971263 ip-10-0-129-39 kubenswrapper[2575]: I0707 17:40:41.971231 2575 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kubelet-serving" csr="csr-j684j" Jul 07 17:40:42.009215 ip-10-0-129-39 kubenswrapper[2575]: I0707 17:40:42.009193 2575 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world" Jul 07 17:40:42.022152 ip-10-0-129-39 kubenswrapper[2575]: I0707 17:40:42.022125 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/2d51e575-aadb-4cdd-a6cf-1ef119b49029-run\") pod \"tuned-7hrdq\" (UID: \"2d51e575-aadb-4cdd-a6cf-1ef119b49029\") " pod="openshift-cluster-node-tuning-operator/tuned-7hrdq" Jul 07 17:40:42.022246 ip-10-0-129-39 kubenswrapper[2575]: I0707 17:40:42.022155 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/ad91fa94-7ae7-4245-90ee-78701b164dd2-host-run-netns\") pod \"multus-rs8sg\" (UID: \"ad91fa94-7ae7-4245-90ee-78701b164dd2\") " pod="openshift-multus/multus-rs8sg" Jul 07 17:40:42.022246 ip-10-0-129-39 kubenswrapper[2575]: I0707 17:40:42.022172 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/ad91fa94-7ae7-4245-90ee-78701b164dd2-host-var-lib-kubelet\") pod \"multus-rs8sg\" (UID: \"ad91fa94-7ae7-4245-90ee-78701b164dd2\") " pod="openshift-multus/multus-rs8sg" Jul 07 17:40:42.022246 ip-10-0-129-39 kubenswrapper[2575]: I0707 17:40:42.022187 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/b8eb06e8-1aea-4b07-801e-fe9f5f86224f-etc-openvswitch\") pod \"ovnkube-node-qgbk5\" (UID: \"b8eb06e8-1aea-4b07-801e-fe9f5f86224f\") " pod="openshift-ovn-kubernetes/ovnkube-node-qgbk5" Jul 07 17:40:42.022246 ip-10-0-129-39 kubenswrapper[2575]: I0707 17:40:42.022204 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/b8eb06e8-1aea-4b07-801e-fe9f5f86224f-log-socket\") pod \"ovnkube-node-qgbk5\" (UID: \"b8eb06e8-1aea-4b07-801e-fe9f5f86224f\") " pod="openshift-ovn-kubernetes/ovnkube-node-qgbk5" Jul 07 17:40:42.022246 ip-10-0-129-39 kubenswrapper[2575]: I0707 17:40:42.022231 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gzrcq\" (UniqueName: \"kubernetes.io/projected/1233b6d4-571b-4319-b523-436909681456-kube-api-access-gzrcq\") pod \"iptables-alerter-5vc77\" (UID: \"1233b6d4-571b-4319-b523-436909681456\") " pod="openshift-network-operator/iptables-alerter-5vc77" Jul 07 17:40:42.022478 ip-10-0-129-39 kubenswrapper[2575]: I0707 17:40:42.022253 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/ad91fa94-7ae7-4245-90ee-78701b164dd2-host-run-netns\") pod \"multus-rs8sg\" (UID: \"ad91fa94-7ae7-4245-90ee-78701b164dd2\") " pod="openshift-multus/multus-rs8sg" Jul 07 17:40:42.022478 ip-10-0-129-39 kubenswrapper[2575]: I0707 17:40:42.022255 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/ff237df8-d435-4d8c-88ec-8f31795bf0ef-host\") pod \"node-ca-l6pnn\" (UID: \"ff237df8-d435-4d8c-88ec-8f31795bf0ef\") " pod="openshift-image-registry/node-ca-l6pnn" Jul 07 17:40:42.022478 ip-10-0-129-39 kubenswrapper[2575]: I0707 17:40:42.022276 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/b8eb06e8-1aea-4b07-801e-fe9f5f86224f-etc-openvswitch\") pod \"ovnkube-node-qgbk5\" (UID: \"b8eb06e8-1aea-4b07-801e-fe9f5f86224f\") " pod="openshift-ovn-kubernetes/ovnkube-node-qgbk5" Jul 07 17:40:42.022478 ip-10-0-129-39 kubenswrapper[2575]: I0707 17:40:42.022262 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/b8eb06e8-1aea-4b07-801e-fe9f5f86224f-log-socket\") pod \"ovnkube-node-qgbk5\" (UID: \"b8eb06e8-1aea-4b07-801e-fe9f5f86224f\") " pod="openshift-ovn-kubernetes/ovnkube-node-qgbk5" Jul 07 17:40:42.022478 ip-10-0-129-39 kubenswrapper[2575]: I0707 17:40:42.022305 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/ad91fa94-7ae7-4245-90ee-78701b164dd2-host-var-lib-kubelet\") pod \"multus-rs8sg\" (UID: \"ad91fa94-7ae7-4245-90ee-78701b164dd2\") " pod="openshift-multus/multus-rs8sg" Jul 07 17:40:42.022478 ip-10-0-129-39 kubenswrapper[2575]: I0707 17:40:42.022305 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/59b94723-b7c1-4eb7-be5f-4ec4254b915d-tuning-conf-dir\") pod \"multus-additional-cni-plugins-9m48z\" (UID: \"59b94723-b7c1-4eb7-be5f-4ec4254b915d\") " pod="openshift-multus/multus-additional-cni-plugins-9m48z" Jul 07 17:40:42.022478 ip-10-0-129-39 kubenswrapper[2575]: I0707 17:40:42.022351 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/2d51e575-aadb-4cdd-a6cf-1ef119b49029-run\") pod \"tuned-7hrdq\" (UID: \"2d51e575-aadb-4cdd-a6cf-1ef119b49029\") " pod="openshift-cluster-node-tuning-operator/tuned-7hrdq" Jul 07 17:40:42.022478 ip-10-0-129-39 kubenswrapper[2575]: I0707 17:40:42.022362 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/2d51e575-aadb-4cdd-a6cf-1ef119b49029-sys\") pod \"tuned-7hrdq\" (UID: \"2d51e575-aadb-4cdd-a6cf-1ef119b49029\") " pod="openshift-cluster-node-tuning-operator/tuned-7hrdq" Jul 07 17:40:42.022478 ip-10-0-129-39 kubenswrapper[2575]: I0707 17:40:42.022398 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/b8eb06e8-1aea-4b07-801e-fe9f5f86224f-var-lib-openvswitch\") pod \"ovnkube-node-qgbk5\" (UID: \"b8eb06e8-1aea-4b07-801e-fe9f5f86224f\") " pod="openshift-ovn-kubernetes/ovnkube-node-qgbk5" Jul 07 17:40:42.022478 ip-10-0-129-39 kubenswrapper[2575]: I0707 17:40:42.022423 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/b8eb06e8-1aea-4b07-801e-fe9f5f86224f-host-cni-bin\") pod \"ovnkube-node-qgbk5\" (UID: \"b8eb06e8-1aea-4b07-801e-fe9f5f86224f\") " pod="openshift-ovn-kubernetes/ovnkube-node-qgbk5" Jul 07 17:40:42.022478 ip-10-0-129-39 kubenswrapper[2575]: I0707 17:40:42.022444 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/2d51e575-aadb-4cdd-a6cf-1ef119b49029-sys\") pod \"tuned-7hrdq\" (UID: \"2d51e575-aadb-4cdd-a6cf-1ef119b49029\") " pod="openshift-cluster-node-tuning-operator/tuned-7hrdq" Jul 07 17:40:42.022478 ip-10-0-129-39 kubenswrapper[2575]: I0707 17:40:42.022453 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/b8eb06e8-1aea-4b07-801e-fe9f5f86224f-var-lib-openvswitch\") pod \"ovnkube-node-qgbk5\" (UID: \"b8eb06e8-1aea-4b07-801e-fe9f5f86224f\") " pod="openshift-ovn-kubernetes/ovnkube-node-qgbk5" Jul 07 17:40:42.022478 ip-10-0-129-39 kubenswrapper[2575]: I0707 17:40:42.022446 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/4f2c6f8d-3b44-4f4f-8d6b-433f013a867f-agent-certs\") pod \"konnectivity-agent-hr726\" (UID: \"4f2c6f8d-3b44-4f4f-8d6b-433f013a867f\") " pod="kube-system/konnectivity-agent-hr726" Jul 07 17:40:42.022478 ip-10-0-129-39 kubenswrapper[2575]: I0707 17:40:42.022480 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4962z\" (UniqueName: \"kubernetes.io/projected/ff237df8-d435-4d8c-88ec-8f31795bf0ef-kube-api-access-4962z\") pod \"node-ca-l6pnn\" (UID: \"ff237df8-d435-4d8c-88ec-8f31795bf0ef\") " pod="openshift-image-registry/node-ca-l6pnn" Jul 07 17:40:42.022478 ip-10-0-129-39 kubenswrapper[2575]: I0707 17:40:42.022485 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/b8eb06e8-1aea-4b07-801e-fe9f5f86224f-host-cni-bin\") pod \"ovnkube-node-qgbk5\" (UID: \"b8eb06e8-1aea-4b07-801e-fe9f5f86224f\") " pod="openshift-ovn-kubernetes/ovnkube-node-qgbk5" Jul 07 17:40:42.023143 ip-10-0-129-39 kubenswrapper[2575]: I0707 17:40:42.022498 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/ad91fa94-7ae7-4245-90ee-78701b164dd2-os-release\") pod \"multus-rs8sg\" (UID: \"ad91fa94-7ae7-4245-90ee-78701b164dd2\") " pod="openshift-multus/multus-rs8sg" Jul 07 17:40:42.023143 ip-10-0-129-39 kubenswrapper[2575]: I0707 17:40:42.022530 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/b8eb06e8-1aea-4b07-801e-fe9f5f86224f-node-log\") pod \"ovnkube-node-qgbk5\" (UID: \"b8eb06e8-1aea-4b07-801e-fe9f5f86224f\") " pod="openshift-ovn-kubernetes/ovnkube-node-qgbk5" Jul 07 17:40:42.023143 ip-10-0-129-39 kubenswrapper[2575]: I0707 17:40:42.022552 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/a5765746-152c-4b60-b664-2dc9c75bd013-socket-dir\") pod \"aws-ebs-csi-driver-node-btqqg\" (UID: \"a5765746-152c-4b60-b664-2dc9c75bd013\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-btqqg" Jul 07 17:40:42.023143 ip-10-0-129-39 kubenswrapper[2575]: I0707 17:40:42.022568 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r4vhp\" (UniqueName: \"kubernetes.io/projected/a5765746-152c-4b60-b664-2dc9c75bd013-kube-api-access-r4vhp\") pod \"aws-ebs-csi-driver-node-btqqg\" (UID: \"a5765746-152c-4b60-b664-2dc9c75bd013\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-btqqg" Jul 07 17:40:42.023143 ip-10-0-129-39 kubenswrapper[2575]: I0707 17:40:42.022585 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/59b94723-b7c1-4eb7-be5f-4ec4254b915d-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-9m48z\" (UID: \"59b94723-b7c1-4eb7-be5f-4ec4254b915d\") " pod="openshift-multus/multus-additional-cni-plugins-9m48z" Jul 07 17:40:42.023143 ip-10-0-129-39 kubenswrapper[2575]: I0707 17:40:42.022609 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d587896d-9156-4b75-aa70-27962a977051-metrics-certs\") pod \"network-metrics-daemon-gz97j\" (UID: \"d587896d-9156-4b75-aa70-27962a977051\") " pod="openshift-multus/network-metrics-daemon-gz97j" Jul 07 17:40:42.023143 ip-10-0-129-39 kubenswrapper[2575]: I0707 17:40:42.022624 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/8344b558-69d0-43b3-9aac-0cea0d5379f3-kubelet-config\") pod \"global-pull-secret-syncer-76r79\" (UID: \"8344b558-69d0-43b3-9aac-0cea0d5379f3\") " pod="kube-system/global-pull-secret-syncer-76r79" Jul 07 17:40:42.023143 ip-10-0-129-39 kubenswrapper[2575]: I0707 17:40:42.022624 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/b8eb06e8-1aea-4b07-801e-fe9f5f86224f-node-log\") pod \"ovnkube-node-qgbk5\" (UID: \"b8eb06e8-1aea-4b07-801e-fe9f5f86224f\") " pod="openshift-ovn-kubernetes/ovnkube-node-qgbk5" Jul 07 17:40:42.023143 ip-10-0-129-39 kubenswrapper[2575]: I0707 17:40:42.022639 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/8344b558-69d0-43b3-9aac-0cea0d5379f3-original-pull-secret\") pod \"global-pull-secret-syncer-76r79\" (UID: \"8344b558-69d0-43b3-9aac-0cea0d5379f3\") " pod="kube-system/global-pull-secret-syncer-76r79" Jul 07 17:40:42.023143 ip-10-0-129-39 kubenswrapper[2575]: I0707 17:40:42.022656 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/ad91fa94-7ae7-4245-90ee-78701b164dd2-system-cni-dir\") pod \"multus-rs8sg\" (UID: \"ad91fa94-7ae7-4245-90ee-78701b164dd2\") " pod="openshift-multus/multus-rs8sg" Jul 07 17:40:42.023143 ip-10-0-129-39 kubenswrapper[2575]: I0707 17:40:42.022673 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/ad91fa94-7ae7-4245-90ee-78701b164dd2-cni-binary-copy\") pod \"multus-rs8sg\" (UID: \"ad91fa94-7ae7-4245-90ee-78701b164dd2\") " pod="openshift-multus/multus-rs8sg" Jul 07 17:40:42.023143 ip-10-0-129-39 kubenswrapper[2575]: I0707 17:40:42.022694 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/ad91fa94-7ae7-4245-90ee-78701b164dd2-host-run-k8s-cni-cncf-io\") pod \"multus-rs8sg\" (UID: \"ad91fa94-7ae7-4245-90ee-78701b164dd2\") " pod="openshift-multus/multus-rs8sg" Jul 07 17:40:42.023143 ip-10-0-129-39 kubenswrapper[2575]: I0707 17:40:42.022683 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/8344b558-69d0-43b3-9aac-0cea0d5379f3-kubelet-config\") pod \"global-pull-secret-syncer-76r79\" (UID: \"8344b558-69d0-43b3-9aac-0cea0d5379f3\") " pod="kube-system/global-pull-secret-syncer-76r79" Jul 07 17:40:42.023143 ip-10-0-129-39 kubenswrapper[2575]: I0707 17:40:42.022710 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/ad91fa94-7ae7-4245-90ee-78701b164dd2-host-var-lib-cni-bin\") pod \"multus-rs8sg\" (UID: \"ad91fa94-7ae7-4245-90ee-78701b164dd2\") " pod="openshift-multus/multus-rs8sg" Jul 07 17:40:42.023143 ip-10-0-129-39 kubenswrapper[2575]: I0707 17:40:42.022740 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/ad91fa94-7ae7-4245-90ee-78701b164dd2-host-var-lib-cni-bin\") pod \"multus-rs8sg\" (UID: \"ad91fa94-7ae7-4245-90ee-78701b164dd2\") " pod="openshift-multus/multus-rs8sg" Jul 07 17:40:42.023143 ip-10-0-129-39 kubenswrapper[2575]: I0707 17:40:42.022764 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/ad91fa94-7ae7-4245-90ee-78701b164dd2-system-cni-dir\") pod \"multus-rs8sg\" (UID: \"ad91fa94-7ae7-4245-90ee-78701b164dd2\") " pod="openshift-multus/multus-rs8sg" Jul 07 17:40:42.023143 ip-10-0-129-39 kubenswrapper[2575]: I0707 17:40:42.022797 2575 swap_util.go:74] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory" Jul 07 17:40:42.023143 ip-10-0-129-39 kubenswrapper[2575]: E0707 17:40:42.022855 2575 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 07 17:40:42.024019 ip-10-0-129-39 kubenswrapper[2575]: I0707 17:40:42.022877 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/a5765746-152c-4b60-b664-2dc9c75bd013-socket-dir\") pod \"aws-ebs-csi-driver-node-btqqg\" (UID: \"a5765746-152c-4b60-b664-2dc9c75bd013\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-btqqg" Jul 07 17:40:42.024019 ip-10-0-129-39 kubenswrapper[2575]: I0707 17:40:42.022915 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/ad91fa94-7ae7-4245-90ee-78701b164dd2-host-run-multus-certs\") pod \"multus-rs8sg\" (UID: \"ad91fa94-7ae7-4245-90ee-78701b164dd2\") " pod="openshift-multus/multus-rs8sg" Jul 07 17:40:42.024019 ip-10-0-129-39 kubenswrapper[2575]: E0707 17:40:42.022944 2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8344b558-69d0-43b3-9aac-0cea0d5379f3-original-pull-secret podName:8344b558-69d0-43b3-9aac-0cea0d5379f3 nodeName:}" failed. No retries permitted until 2026-07-07 17:40:42.522920865 +0000 UTC m=+3.028727532 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/8344b558-69d0-43b3-9aac-0cea0d5379f3-original-pull-secret") pod "global-pull-secret-syncer-76r79" (UID: "8344b558-69d0-43b3-9aac-0cea0d5379f3") : object "kube-system"/"original-pull-secret" not registered Jul 07 17:40:42.024019 ip-10-0-129-39 kubenswrapper[2575]: I0707 17:40:42.022948 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/ad91fa94-7ae7-4245-90ee-78701b164dd2-host-run-multus-certs\") pod \"multus-rs8sg\" (UID: \"ad91fa94-7ae7-4245-90ee-78701b164dd2\") " pod="openshift-multus/multus-rs8sg" Jul 07 17:40:42.024019 ip-10-0-129-39 kubenswrapper[2575]: I0707 17:40:42.022975 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/b8eb06e8-1aea-4b07-801e-fe9f5f86224f-host-run-netns\") pod \"ovnkube-node-qgbk5\" (UID: \"b8eb06e8-1aea-4b07-801e-fe9f5f86224f\") " pod="openshift-ovn-kubernetes/ovnkube-node-qgbk5" Jul 07 17:40:42.024019 ip-10-0-129-39 kubenswrapper[2575]: I0707 17:40:42.023001 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/2d51e575-aadb-4cdd-a6cf-1ef119b49029-etc-tuned\") pod \"tuned-7hrdq\" (UID: \"2d51e575-aadb-4cdd-a6cf-1ef119b49029\") " pod="openshift-cluster-node-tuning-operator/tuned-7hrdq" Jul 07 17:40:42.024019 ip-10-0-129-39 kubenswrapper[2575]: I0707 17:40:42.023026 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/ad91fa94-7ae7-4245-90ee-78701b164dd2-cnibin\") pod \"multus-rs8sg\" (UID: \"ad91fa94-7ae7-4245-90ee-78701b164dd2\") " pod="openshift-multus/multus-rs8sg" Jul 07 17:40:42.024019 ip-10-0-129-39 kubenswrapper[2575]: I0707 17:40:42.023048 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/b8eb06e8-1aea-4b07-801e-fe9f5f86224f-host-slash\") pod \"ovnkube-node-qgbk5\" (UID: \"b8eb06e8-1aea-4b07-801e-fe9f5f86224f\") " pod="openshift-ovn-kubernetes/ovnkube-node-qgbk5" Jul 07 17:40:42.024019 ip-10-0-129-39 kubenswrapper[2575]: I0707 17:40:42.023070 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/ad91fa94-7ae7-4245-90ee-78701b164dd2-os-release\") pod \"multus-rs8sg\" (UID: \"ad91fa94-7ae7-4245-90ee-78701b164dd2\") " pod="openshift-multus/multus-rs8sg" Jul 07 17:40:42.024019 ip-10-0-129-39 kubenswrapper[2575]: I0707 17:40:42.023112 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/b8eb06e8-1aea-4b07-801e-fe9f5f86224f-host-run-ovn-kubernetes\") pod \"ovnkube-node-qgbk5\" (UID: \"b8eb06e8-1aea-4b07-801e-fe9f5f86224f\") " pod="openshift-ovn-kubernetes/ovnkube-node-qgbk5" Jul 07 17:40:42.024019 ip-10-0-129-39 kubenswrapper[2575]: I0707 17:40:42.023074 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/b8eb06e8-1aea-4b07-801e-fe9f5f86224f-host-run-ovn-kubernetes\") pod \"ovnkube-node-qgbk5\" (UID: \"b8eb06e8-1aea-4b07-801e-fe9f5f86224f\") " pod="openshift-ovn-kubernetes/ovnkube-node-qgbk5" Jul 07 17:40:42.024019 ip-10-0-129-39 kubenswrapper[2575]: I0707 17:40:42.023152 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/b8eb06e8-1aea-4b07-801e-fe9f5f86224f-ovnkube-script-lib\") pod \"ovnkube-node-qgbk5\" (UID: \"b8eb06e8-1aea-4b07-801e-fe9f5f86224f\") " pod="openshift-ovn-kubernetes/ovnkube-node-qgbk5" Jul 07 17:40:42.024019 ip-10-0-129-39 kubenswrapper[2575]: I0707 17:40:42.023179 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wh2cb\" (UniqueName: \"kubernetes.io/projected/b8eb06e8-1aea-4b07-801e-fe9f5f86224f-kube-api-access-wh2cb\") pod \"ovnkube-node-qgbk5\" (UID: \"b8eb06e8-1aea-4b07-801e-fe9f5f86224f\") " pod="openshift-ovn-kubernetes/ovnkube-node-qgbk5" Jul 07 17:40:42.024019 ip-10-0-129-39 kubenswrapper[2575]: I0707 17:40:42.023185 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/ad91fa94-7ae7-4245-90ee-78701b164dd2-cni-binary-copy\") pod \"multus-rs8sg\" (UID: \"ad91fa94-7ae7-4245-90ee-78701b164dd2\") " pod="openshift-multus/multus-rs8sg" Jul 07 17:40:42.024019 ip-10-0-129-39 kubenswrapper[2575]: I0707 17:40:42.023183 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/ad91fa94-7ae7-4245-90ee-78701b164dd2-host-run-k8s-cni-cncf-io\") pod \"multus-rs8sg\" (UID: \"ad91fa94-7ae7-4245-90ee-78701b164dd2\") " pod="openshift-multus/multus-rs8sg" Jul 07 17:40:42.024019 ip-10-0-129-39 kubenswrapper[2575]: I0707 17:40:42.023223 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/b8eb06e8-1aea-4b07-801e-fe9f5f86224f-host-run-netns\") pod \"ovnkube-node-qgbk5\" (UID: \"b8eb06e8-1aea-4b07-801e-fe9f5f86224f\") " pod="openshift-ovn-kubernetes/ovnkube-node-qgbk5" Jul 07 17:40:42.024019 ip-10-0-129-39 kubenswrapper[2575]: I0707 17:40:42.023223 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/ad91fa94-7ae7-4245-90ee-78701b164dd2-cnibin\") pod \"multus-rs8sg\" (UID: \"ad91fa94-7ae7-4245-90ee-78701b164dd2\") " pod="openshift-multus/multus-rs8sg" Jul 07 17:40:42.024851 ip-10-0-129-39 kubenswrapper[2575]: I0707 17:40:42.023259 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/b8eb06e8-1aea-4b07-801e-fe9f5f86224f-host-slash\") pod \"ovnkube-node-qgbk5\" (UID: \"b8eb06e8-1aea-4b07-801e-fe9f5f86224f\") " pod="openshift-ovn-kubernetes/ovnkube-node-qgbk5" Jul 07 17:40:42.024851 ip-10-0-129-39 kubenswrapper[2575]: I0707 17:40:42.023285 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/59b94723-b7c1-4eb7-be5f-4ec4254b915d-system-cni-dir\") pod \"multus-additional-cni-plugins-9m48z\" (UID: \"59b94723-b7c1-4eb7-be5f-4ec4254b915d\") " pod="openshift-multus/multus-additional-cni-plugins-9m48z" Jul 07 17:40:42.024851 ip-10-0-129-39 kubenswrapper[2575]: I0707 17:40:42.023314 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-78bqt\" (UniqueName: \"kubernetes.io/projected/5d33c1cb-fcda-490d-9694-b1d1eb16b8da-kube-api-access-78bqt\") pod \"network-check-target-vzf4m\" (UID: \"5d33c1cb-fcda-490d-9694-b1d1eb16b8da\") " pod="openshift-network-diagnostics/network-check-target-vzf4m" Jul 07 17:40:42.024851 ip-10-0-129-39 kubenswrapper[2575]: I0707 17:40:42.023340 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/2d51e575-aadb-4cdd-a6cf-1ef119b49029-etc-sysconfig\") pod \"tuned-7hrdq\" (UID: \"2d51e575-aadb-4cdd-a6cf-1ef119b49029\") " pod="openshift-cluster-node-tuning-operator/tuned-7hrdq" Jul 07 17:40:42.024851 ip-10-0-129-39 kubenswrapper[2575]: I0707 17:40:42.023364 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/ad91fa94-7ae7-4245-90ee-78701b164dd2-multus-cni-dir\") pod \"multus-rs8sg\" (UID: \"ad91fa94-7ae7-4245-90ee-78701b164dd2\") " pod="openshift-multus/multus-rs8sg" Jul 07 17:40:42.024851 ip-10-0-129-39 kubenswrapper[2575]: I0707 17:40:42.023390 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/a5765746-152c-4b60-b664-2dc9c75bd013-etc-selinux\") pod \"aws-ebs-csi-driver-node-btqqg\" (UID: \"a5765746-152c-4b60-b664-2dc9c75bd013\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-btqqg" Jul 07 17:40:42.024851 ip-10-0-129-39 kubenswrapper[2575]: I0707 17:40:42.023392 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/2d51e575-aadb-4cdd-a6cf-1ef119b49029-etc-sysconfig\") pod \"tuned-7hrdq\" (UID: \"2d51e575-aadb-4cdd-a6cf-1ef119b49029\") " pod="openshift-cluster-node-tuning-operator/tuned-7hrdq" Jul 07 17:40:42.024851 ip-10-0-129-39 kubenswrapper[2575]: I0707 17:40:42.023414 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/b8eb06e8-1aea-4b07-801e-fe9f5f86224f-env-overrides\") pod \"ovnkube-node-qgbk5\" (UID: \"b8eb06e8-1aea-4b07-801e-fe9f5f86224f\") " pod="openshift-ovn-kubernetes/ovnkube-node-qgbk5" Jul 07 17:40:42.024851 ip-10-0-129-39 kubenswrapper[2575]: I0707 17:40:42.023449 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/1233b6d4-571b-4319-b523-436909681456-host-slash\") pod \"iptables-alerter-5vc77\" (UID: \"1233b6d4-571b-4319-b523-436909681456\") " pod="openshift-network-operator/iptables-alerter-5vc77" Jul 07 17:40:42.024851 ip-10-0-129-39 kubenswrapper[2575]: I0707 17:40:42.023475 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4zlmc\" (UniqueName: \"kubernetes.io/projected/d587896d-9156-4b75-aa70-27962a977051-kube-api-access-4zlmc\") pod \"network-metrics-daemon-gz97j\" (UID: \"d587896d-9156-4b75-aa70-27962a977051\") " pod="openshift-multus/network-metrics-daemon-gz97j" Jul 07 17:40:42.024851 ip-10-0-129-39 kubenswrapper[2575]: I0707 17:40:42.023482 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/a5765746-152c-4b60-b664-2dc9c75bd013-etc-selinux\") pod \"aws-ebs-csi-driver-node-btqqg\" (UID: \"a5765746-152c-4b60-b664-2dc9c75bd013\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-btqqg" Jul 07 17:40:42.024851 ip-10-0-129-39 kubenswrapper[2575]: I0707 17:40:42.023526 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/2d51e575-aadb-4cdd-a6cf-1ef119b49029-etc-modprobe-d\") pod \"tuned-7hrdq\" (UID: \"2d51e575-aadb-4cdd-a6cf-1ef119b49029\") " pod="openshift-cluster-node-tuning-operator/tuned-7hrdq" Jul 07 17:40:42.024851 ip-10-0-129-39 kubenswrapper[2575]: I0707 17:40:42.023573 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/ad91fa94-7ae7-4245-90ee-78701b164dd2-multus-conf-dir\") pod \"multus-rs8sg\" (UID: \"ad91fa94-7ae7-4245-90ee-78701b164dd2\") " pod="openshift-multus/multus-rs8sg" Jul 07 17:40:42.024851 ip-10-0-129-39 kubenswrapper[2575]: I0707 17:40:42.023606 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/b8eb06e8-1aea-4b07-801e-fe9f5f86224f-host-kubelet\") pod \"ovnkube-node-qgbk5\" (UID: \"b8eb06e8-1aea-4b07-801e-fe9f5f86224f\") " pod="openshift-ovn-kubernetes/ovnkube-node-qgbk5" Jul 07 17:40:42.024851 ip-10-0-129-39 kubenswrapper[2575]: I0707 17:40:42.023614 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/ad91fa94-7ae7-4245-90ee-78701b164dd2-multus-cni-dir\") pod \"multus-rs8sg\" (UID: \"ad91fa94-7ae7-4245-90ee-78701b164dd2\") " pod="openshift-multus/multus-rs8sg" Jul 07 17:40:42.024851 ip-10-0-129-39 kubenswrapper[2575]: I0707 17:40:42.023629 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/2d51e575-aadb-4cdd-a6cf-1ef119b49029-etc-modprobe-d\") pod \"tuned-7hrdq\" (UID: \"2d51e575-aadb-4cdd-a6cf-1ef119b49029\") " pod="openshift-cluster-node-tuning-operator/tuned-7hrdq" Jul 07 17:40:42.024851 ip-10-0-129-39 kubenswrapper[2575]: I0707 17:40:42.023633 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vtmvs\" (UniqueName: \"kubernetes.io/projected/2d51e575-aadb-4cdd-a6cf-1ef119b49029-kube-api-access-vtmvs\") pod \"tuned-7hrdq\" (UID: \"2d51e575-aadb-4cdd-a6cf-1ef119b49029\") " pod="openshift-cluster-node-tuning-operator/tuned-7hrdq" Jul 07 17:40:42.025708 ip-10-0-129-39 kubenswrapper[2575]: I0707 17:40:42.023653 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/ad91fa94-7ae7-4245-90ee-78701b164dd2-multus-conf-dir\") pod \"multus-rs8sg\" (UID: \"ad91fa94-7ae7-4245-90ee-78701b164dd2\") " pod="openshift-multus/multus-rs8sg" Jul 07 17:40:42.025708 ip-10-0-129-39 kubenswrapper[2575]: I0707 17:40:42.023665 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/8344b558-69d0-43b3-9aac-0cea0d5379f3-dbus\") pod \"global-pull-secret-syncer-76r79\" (UID: \"8344b558-69d0-43b3-9aac-0cea0d5379f3\") " pod="kube-system/global-pull-secret-syncer-76r79" Jul 07 17:40:42.025708 ip-10-0-129-39 kubenswrapper[2575]: I0707 17:40:42.023684 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/b8eb06e8-1aea-4b07-801e-fe9f5f86224f-host-kubelet\") pod \"ovnkube-node-qgbk5\" (UID: \"b8eb06e8-1aea-4b07-801e-fe9f5f86224f\") " pod="openshift-ovn-kubernetes/ovnkube-node-qgbk5" Jul 07 17:40:42.025708 ip-10-0-129-39 kubenswrapper[2575]: I0707 17:40:42.023690 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/ad91fa94-7ae7-4245-90ee-78701b164dd2-host-var-lib-cni-multus\") pod \"multus-rs8sg\" (UID: \"ad91fa94-7ae7-4245-90ee-78701b164dd2\") " pod="openshift-multus/multus-rs8sg" Jul 07 17:40:42.025708 ip-10-0-129-39 kubenswrapper[2575]: I0707 17:40:42.023713 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/ad91fa94-7ae7-4245-90ee-78701b164dd2-hostroot\") pod \"multus-rs8sg\" (UID: \"ad91fa94-7ae7-4245-90ee-78701b164dd2\") " pod="openshift-multus/multus-rs8sg" Jul 07 17:40:42.025708 ip-10-0-129-39 kubenswrapper[2575]: I0707 17:40:42.023723 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/8344b558-69d0-43b3-9aac-0cea0d5379f3-dbus\") pod \"global-pull-secret-syncer-76r79\" (UID: \"8344b558-69d0-43b3-9aac-0cea0d5379f3\") " pod="kube-system/global-pull-secret-syncer-76r79" Jul 07 17:40:42.025708 ip-10-0-129-39 kubenswrapper[2575]: I0707 17:40:42.023736 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/b8eb06e8-1aea-4b07-801e-fe9f5f86224f-run-openvswitch\") pod \"ovnkube-node-qgbk5\" (UID: \"b8eb06e8-1aea-4b07-801e-fe9f5f86224f\") " pod="openshift-ovn-kubernetes/ovnkube-node-qgbk5" Jul 07 17:40:42.025708 ip-10-0-129-39 kubenswrapper[2575]: I0707 17:40:42.023760 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/ad91fa94-7ae7-4245-90ee-78701b164dd2-host-var-lib-cni-multus\") pod \"multus-rs8sg\" (UID: \"ad91fa94-7ae7-4245-90ee-78701b164dd2\") " pod="openshift-multus/multus-rs8sg" Jul 07 17:40:42.025708 ip-10-0-129-39 kubenswrapper[2575]: I0707 17:40:42.023761 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/59b94723-b7c1-4eb7-be5f-4ec4254b915d-cnibin\") pod \"multus-additional-cni-plugins-9m48z\" (UID: \"59b94723-b7c1-4eb7-be5f-4ec4254b915d\") " pod="openshift-multus/multus-additional-cni-plugins-9m48z" Jul 07 17:40:42.025708 ip-10-0-129-39 kubenswrapper[2575]: I0707 17:40:42.023790 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lm2wr\" (UniqueName: \"kubernetes.io/projected/59b94723-b7c1-4eb7-be5f-4ec4254b915d-kube-api-access-lm2wr\") pod \"multus-additional-cni-plugins-9m48z\" (UID: \"59b94723-b7c1-4eb7-be5f-4ec4254b915d\") " pod="openshift-multus/multus-additional-cni-plugins-9m48z" Jul 07 17:40:42.025708 ip-10-0-129-39 kubenswrapper[2575]: I0707 17:40:42.023802 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/b8eb06e8-1aea-4b07-801e-fe9f5f86224f-env-overrides\") pod \"ovnkube-node-qgbk5\" (UID: \"b8eb06e8-1aea-4b07-801e-fe9f5f86224f\") " pod="openshift-ovn-kubernetes/ovnkube-node-qgbk5" Jul 07 17:40:42.025708 ip-10-0-129-39 kubenswrapper[2575]: I0707 17:40:42.023812 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/b8eb06e8-1aea-4b07-801e-fe9f5f86224f-ovnkube-script-lib\") pod \"ovnkube-node-qgbk5\" (UID: \"b8eb06e8-1aea-4b07-801e-fe9f5f86224f\") " pod="openshift-ovn-kubernetes/ovnkube-node-qgbk5" Jul 07 17:40:42.025708 ip-10-0-129-39 kubenswrapper[2575]: I0707 17:40:42.023792 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/ad91fa94-7ae7-4245-90ee-78701b164dd2-hostroot\") pod \"multus-rs8sg\" (UID: \"ad91fa94-7ae7-4245-90ee-78701b164dd2\") " pod="openshift-multus/multus-rs8sg" Jul 07 17:40:42.025708 ip-10-0-129-39 kubenswrapper[2575]: I0707 17:40:42.023814 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/d4c44345-d6bf-4778-8911-09c7a1a0740a-hosts-file\") pod \"node-resolver-fc7qv\" (UID: \"d4c44345-d6bf-4778-8911-09c7a1a0740a\") " pod="openshift-dns/node-resolver-fc7qv" Jul 07 17:40:42.025708 ip-10-0-129-39 kubenswrapper[2575]: I0707 17:40:42.023816 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/b8eb06e8-1aea-4b07-801e-fe9f5f86224f-run-openvswitch\") pod \"ovnkube-node-qgbk5\" (UID: \"b8eb06e8-1aea-4b07-801e-fe9f5f86224f\") " pod="openshift-ovn-kubernetes/ovnkube-node-qgbk5" Jul 07 17:40:42.025708 ip-10-0-129-39 kubenswrapper[2575]: I0707 17:40:42.023864 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/b8eb06e8-1aea-4b07-801e-fe9f5f86224f-host-cni-netd\") pod \"ovnkube-node-qgbk5\" (UID: \"b8eb06e8-1aea-4b07-801e-fe9f5f86224f\") " pod="openshift-ovn-kubernetes/ovnkube-node-qgbk5" Jul 07 17:40:42.025708 ip-10-0-129-39 kubenswrapper[2575]: I0707 17:40:42.023907 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/b8eb06e8-1aea-4b07-801e-fe9f5f86224f-ovn-node-metrics-cert\") pod \"ovnkube-node-qgbk5\" (UID: \"b8eb06e8-1aea-4b07-801e-fe9f5f86224f\") " pod="openshift-ovn-kubernetes/ovnkube-node-qgbk5" Jul 07 17:40:42.025708 ip-10-0-129-39 kubenswrapper[2575]: I0707 17:40:42.023933 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/2d51e575-aadb-4cdd-a6cf-1ef119b49029-etc-sysctl-d\") pod \"tuned-7hrdq\" (UID: \"2d51e575-aadb-4cdd-a6cf-1ef119b49029\") " pod="openshift-cluster-node-tuning-operator/tuned-7hrdq" Jul 07 17:40:42.026601 ip-10-0-129-39 kubenswrapper[2575]: I0707 17:40:42.023937 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/b8eb06e8-1aea-4b07-801e-fe9f5f86224f-host-cni-netd\") pod \"ovnkube-node-qgbk5\" (UID: \"b8eb06e8-1aea-4b07-801e-fe9f5f86224f\") " pod="openshift-ovn-kubernetes/ovnkube-node-qgbk5" Jul 07 17:40:42.026601 ip-10-0-129-39 kubenswrapper[2575]: I0707 17:40:42.023954 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/2d51e575-aadb-4cdd-a6cf-1ef119b49029-etc-sysctl-conf\") pod \"tuned-7hrdq\" (UID: \"2d51e575-aadb-4cdd-a6cf-1ef119b49029\") " pod="openshift-cluster-node-tuning-operator/tuned-7hrdq" Jul 07 17:40:42.026601 ip-10-0-129-39 kubenswrapper[2575]: I0707 17:40:42.023983 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d4c44345-d6bf-4778-8911-09c7a1a0740a-tmp-dir\") pod \"node-resolver-fc7qv\" (UID: \"d4c44345-d6bf-4778-8911-09c7a1a0740a\") " pod="openshift-dns/node-resolver-fc7qv" Jul 07 17:40:42.026601 ip-10-0-129-39 kubenswrapper[2575]: I0707 17:40:42.024006 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/2d51e575-aadb-4cdd-a6cf-1ef119b49029-etc-systemd\") pod \"tuned-7hrdq\" (UID: \"2d51e575-aadb-4cdd-a6cf-1ef119b49029\") " pod="openshift-cluster-node-tuning-operator/tuned-7hrdq" Jul 07 17:40:42.026601 ip-10-0-129-39 kubenswrapper[2575]: I0707 17:40:42.024027 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/2d51e575-aadb-4cdd-a6cf-1ef119b49029-tmp\") pod \"tuned-7hrdq\" (UID: \"2d51e575-aadb-4cdd-a6cf-1ef119b49029\") " pod="openshift-cluster-node-tuning-operator/tuned-7hrdq" Jul 07 17:40:42.026601 ip-10-0-129-39 kubenswrapper[2575]: I0707 17:40:42.024061 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/ad91fa94-7ae7-4245-90ee-78701b164dd2-etc-kubernetes\") pod \"multus-rs8sg\" (UID: \"ad91fa94-7ae7-4245-90ee-78701b164dd2\") " pod="openshift-multus/multus-rs8sg" Jul 07 17:40:42.026601 ip-10-0-129-39 kubenswrapper[2575]: I0707 17:40:42.024078 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/2d51e575-aadb-4cdd-a6cf-1ef119b49029-etc-sysctl-d\") pod \"tuned-7hrdq\" (UID: \"2d51e575-aadb-4cdd-a6cf-1ef119b49029\") " pod="openshift-cluster-node-tuning-operator/tuned-7hrdq" Jul 07 17:40:42.026601 ip-10-0-129-39 kubenswrapper[2575]: I0707 17:40:42.024084 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8kv7w\" (UniqueName: \"kubernetes.io/projected/ad91fa94-7ae7-4245-90ee-78701b164dd2-kube-api-access-8kv7w\") pod \"multus-rs8sg\" (UID: \"ad91fa94-7ae7-4245-90ee-78701b164dd2\") " pod="openshift-multus/multus-rs8sg" Jul 07 17:40:42.026601 ip-10-0-129-39 kubenswrapper[2575]: I0707 17:40:42.024126 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/b8eb06e8-1aea-4b07-801e-fe9f5f86224f-run-ovn\") pod \"ovnkube-node-qgbk5\" (UID: \"b8eb06e8-1aea-4b07-801e-fe9f5f86224f\") " pod="openshift-ovn-kubernetes/ovnkube-node-qgbk5" Jul 07 17:40:42.026601 ip-10-0-129-39 kubenswrapper[2575]: I0707 17:40:42.024152 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/b8eb06e8-1aea-4b07-801e-fe9f5f86224f-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-qgbk5\" (UID: \"b8eb06e8-1aea-4b07-801e-fe9f5f86224f\") " pod="openshift-ovn-kubernetes/ovnkube-node-qgbk5" Jul 07 17:40:42.026601 ip-10-0-129-39 kubenswrapper[2575]: I0707 17:40:42.024179 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/b8eb06e8-1aea-4b07-801e-fe9f5f86224f-ovnkube-config\") pod \"ovnkube-node-qgbk5\" (UID: \"b8eb06e8-1aea-4b07-801e-fe9f5f86224f\") " pod="openshift-ovn-kubernetes/ovnkube-node-qgbk5" Jul 07 17:40:42.026601 ip-10-0-129-39 kubenswrapper[2575]: I0707 17:40:42.024204 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/a5765746-152c-4b60-b664-2dc9c75bd013-registration-dir\") pod \"aws-ebs-csi-driver-node-btqqg\" (UID: \"a5765746-152c-4b60-b664-2dc9c75bd013\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-btqqg" Jul 07 17:40:42.026601 ip-10-0-129-39 kubenswrapper[2575]: I0707 17:40:42.024229 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/2d51e575-aadb-4cdd-a6cf-1ef119b49029-host\") pod \"tuned-7hrdq\" (UID: \"2d51e575-aadb-4cdd-a6cf-1ef119b49029\") " pod="openshift-cluster-node-tuning-operator/tuned-7hrdq" Jul 07 17:40:42.026601 ip-10-0-129-39 kubenswrapper[2575]: I0707 17:40:42.024253 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/a5765746-152c-4b60-b664-2dc9c75bd013-sys-fs\") pod \"aws-ebs-csi-driver-node-btqqg\" (UID: \"a5765746-152c-4b60-b664-2dc9c75bd013\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-btqqg" Jul 07 17:40:42.026601 ip-10-0-129-39 kubenswrapper[2575]: I0707 17:40:42.024303 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/a5765746-152c-4b60-b664-2dc9c75bd013-sys-fs\") pod \"aws-ebs-csi-driver-node-btqqg\" (UID: \"a5765746-152c-4b60-b664-2dc9c75bd013\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-btqqg" Jul 07 17:40:42.026601 ip-10-0-129-39 kubenswrapper[2575]: I0707 17:40:42.024351 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/a5765746-152c-4b60-b664-2dc9c75bd013-registration-dir\") pod \"aws-ebs-csi-driver-node-btqqg\" (UID: \"a5765746-152c-4b60-b664-2dc9c75bd013\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-btqqg" Jul 07 17:40:42.026601 ip-10-0-129-39 kubenswrapper[2575]: I0707 17:40:42.024388 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/2d51e575-aadb-4cdd-a6cf-1ef119b49029-etc-sysctl-conf\") pod \"tuned-7hrdq\" (UID: \"2d51e575-aadb-4cdd-a6cf-1ef119b49029\") " pod="openshift-cluster-node-tuning-operator/tuned-7hrdq" Jul 07 17:40:42.027394 ip-10-0-129-39 kubenswrapper[2575]: I0707 17:40:42.024433 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/2d51e575-aadb-4cdd-a6cf-1ef119b49029-etc-systemd\") pod \"tuned-7hrdq\" (UID: \"2d51e575-aadb-4cdd-a6cf-1ef119b49029\") " pod="openshift-cluster-node-tuning-operator/tuned-7hrdq" Jul 07 17:40:42.027394 ip-10-0-129-39 kubenswrapper[2575]: I0707 17:40:42.024476 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/2d51e575-aadb-4cdd-a6cf-1ef119b49029-host\") pod \"tuned-7hrdq\" (UID: \"2d51e575-aadb-4cdd-a6cf-1ef119b49029\") " pod="openshift-cluster-node-tuning-operator/tuned-7hrdq" Jul 07 17:40:42.027394 ip-10-0-129-39 kubenswrapper[2575]: I0707 17:40:42.024525 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/b8eb06e8-1aea-4b07-801e-fe9f5f86224f-run-ovn\") pod \"ovnkube-node-qgbk5\" (UID: \"b8eb06e8-1aea-4b07-801e-fe9f5f86224f\") " pod="openshift-ovn-kubernetes/ovnkube-node-qgbk5" Jul 07 17:40:42.027394 ip-10-0-129-39 kubenswrapper[2575]: I0707 17:40:42.024530 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/ad91fa94-7ae7-4245-90ee-78701b164dd2-etc-kubernetes\") pod \"multus-rs8sg\" (UID: \"ad91fa94-7ae7-4245-90ee-78701b164dd2\") " pod="openshift-multus/multus-rs8sg" Jul 07 17:40:42.027394 ip-10-0-129-39 kubenswrapper[2575]: I0707 17:40:42.024534 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/b8eb06e8-1aea-4b07-801e-fe9f5f86224f-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-qgbk5\" (UID: \"b8eb06e8-1aea-4b07-801e-fe9f5f86224f\") " pod="openshift-ovn-kubernetes/ovnkube-node-qgbk5" Jul 07 17:40:42.027394 ip-10-0-129-39 kubenswrapper[2575]: I0707 17:40:42.024576 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/59b94723-b7c1-4eb7-be5f-4ec4254b915d-os-release\") pod \"multus-additional-cni-plugins-9m48z\" (UID: \"59b94723-b7c1-4eb7-be5f-4ec4254b915d\") " pod="openshift-multus/multus-additional-cni-plugins-9m48z" Jul 07 17:40:42.027394 ip-10-0-129-39 kubenswrapper[2575]: I0707 17:40:42.024603 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/59b94723-b7c1-4eb7-be5f-4ec4254b915d-cni-binary-copy\") pod \"multus-additional-cni-plugins-9m48z\" (UID: \"59b94723-b7c1-4eb7-be5f-4ec4254b915d\") " pod="openshift-multus/multus-additional-cni-plugins-9m48z" Jul 07 17:40:42.027394 ip-10-0-129-39 kubenswrapper[2575]: I0707 17:40:42.024630 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/59b94723-b7c1-4eb7-be5f-4ec4254b915d-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-9m48z\" (UID: \"59b94723-b7c1-4eb7-be5f-4ec4254b915d\") " pod="openshift-multus/multus-additional-cni-plugins-9m48z" Jul 07 17:40:42.027394 ip-10-0-129-39 kubenswrapper[2575]: I0707 17:40:42.024660 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/2d51e575-aadb-4cdd-a6cf-1ef119b49029-etc-kubernetes\") pod \"tuned-7hrdq\" (UID: \"2d51e575-aadb-4cdd-a6cf-1ef119b49029\") " pod="openshift-cluster-node-tuning-operator/tuned-7hrdq" Jul 07 17:40:42.027394 ip-10-0-129-39 kubenswrapper[2575]: I0707 17:40:42.024685 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/2d51e575-aadb-4cdd-a6cf-1ef119b49029-var-lib-kubelet\") pod \"tuned-7hrdq\" (UID: \"2d51e575-aadb-4cdd-a6cf-1ef119b49029\") " pod="openshift-cluster-node-tuning-operator/tuned-7hrdq" Jul 07 17:40:42.027394 ip-10-0-129-39 kubenswrapper[2575]: I0707 17:40:42.024719 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/ad91fa94-7ae7-4245-90ee-78701b164dd2-multus-daemon-config\") pod \"multus-rs8sg\" (UID: \"ad91fa94-7ae7-4245-90ee-78701b164dd2\") " pod="openshift-multus/multus-rs8sg" Jul 07 17:40:42.027394 ip-10-0-129-39 kubenswrapper[2575]: I0707 17:40:42.024722 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/b8eb06e8-1aea-4b07-801e-fe9f5f86224f-ovnkube-config\") pod \"ovnkube-node-qgbk5\" (UID: \"b8eb06e8-1aea-4b07-801e-fe9f5f86224f\") " pod="openshift-ovn-kubernetes/ovnkube-node-qgbk5" Jul 07 17:40:42.027394 ip-10-0-129-39 kubenswrapper[2575]: I0707 17:40:42.024742 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/a5765746-152c-4b60-b664-2dc9c75bd013-device-dir\") pod \"aws-ebs-csi-driver-node-btqqg\" (UID: \"a5765746-152c-4b60-b664-2dc9c75bd013\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-btqqg" Jul 07 17:40:42.027394 ip-10-0-129-39 kubenswrapper[2575]: I0707 17:40:42.024792 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/2d51e575-aadb-4cdd-a6cf-1ef119b49029-var-lib-kubelet\") pod \"tuned-7hrdq\" (UID: \"2d51e575-aadb-4cdd-a6cf-1ef119b49029\") " pod="openshift-cluster-node-tuning-operator/tuned-7hrdq" Jul 07 17:40:42.027394 ip-10-0-129-39 kubenswrapper[2575]: I0707 17:40:42.024798 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/a5765746-152c-4b60-b664-2dc9c75bd013-device-dir\") pod \"aws-ebs-csi-driver-node-btqqg\" (UID: \"a5765746-152c-4b60-b664-2dc9c75bd013\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-btqqg" Jul 07 17:40:42.027394 ip-10-0-129-39 kubenswrapper[2575]: I0707 17:40:42.024790 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/2d51e575-aadb-4cdd-a6cf-1ef119b49029-etc-kubernetes\") pod \"tuned-7hrdq\" (UID: \"2d51e575-aadb-4cdd-a6cf-1ef119b49029\") " pod="openshift-cluster-node-tuning-operator/tuned-7hrdq" Jul 07 17:40:42.027394 ip-10-0-129-39 kubenswrapper[2575]: I0707 17:40:42.024834 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/ad91fa94-7ae7-4245-90ee-78701b164dd2-multus-socket-dir-parent\") pod \"multus-rs8sg\" (UID: \"ad91fa94-7ae7-4245-90ee-78701b164dd2\") " pod="openshift-multus/multus-rs8sg" Jul 07 17:40:42.027971 ip-10-0-129-39 kubenswrapper[2575]: I0707 17:40:42.024904 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/b8eb06e8-1aea-4b07-801e-fe9f5f86224f-systemd-units\") pod \"ovnkube-node-qgbk5\" (UID: \"b8eb06e8-1aea-4b07-801e-fe9f5f86224f\") " pod="openshift-ovn-kubernetes/ovnkube-node-qgbk5" Jul 07 17:40:42.027971 ip-10-0-129-39 kubenswrapper[2575]: I0707 17:40:42.024921 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/ad91fa94-7ae7-4245-90ee-78701b164dd2-multus-socket-dir-parent\") pod \"multus-rs8sg\" (UID: \"ad91fa94-7ae7-4245-90ee-78701b164dd2\") " pod="openshift-multus/multus-rs8sg" Jul 07 17:40:42.027971 ip-10-0-129-39 kubenswrapper[2575]: I0707 17:40:42.024960 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/b8eb06e8-1aea-4b07-801e-fe9f5f86224f-run-systemd\") pod \"ovnkube-node-qgbk5\" (UID: \"b8eb06e8-1aea-4b07-801e-fe9f5f86224f\") " pod="openshift-ovn-kubernetes/ovnkube-node-qgbk5" Jul 07 17:40:42.027971 ip-10-0-129-39 kubenswrapper[2575]: I0707 17:40:42.024993 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/1233b6d4-571b-4319-b523-436909681456-iptables-alerter-script\") pod \"iptables-alerter-5vc77\" (UID: \"1233b6d4-571b-4319-b523-436909681456\") " pod="openshift-network-operator/iptables-alerter-5vc77" Jul 07 17:40:42.027971 ip-10-0-129-39 kubenswrapper[2575]: I0707 17:40:42.025084 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/ff237df8-d435-4d8c-88ec-8f31795bf0ef-serviceca\") pod \"node-ca-l6pnn\" (UID: \"ff237df8-d435-4d8c-88ec-8f31795bf0ef\") " pod="openshift-image-registry/node-ca-l6pnn" Jul 07 17:40:42.027971 ip-10-0-129-39 kubenswrapper[2575]: I0707 17:40:42.025085 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/b8eb06e8-1aea-4b07-801e-fe9f5f86224f-systemd-units\") pod \"ovnkube-node-qgbk5\" (UID: \"b8eb06e8-1aea-4b07-801e-fe9f5f86224f\") " pod="openshift-ovn-kubernetes/ovnkube-node-qgbk5" Jul 07 17:40:42.027971 ip-10-0-129-39 kubenswrapper[2575]: I0707 17:40:42.025121 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/b8eb06e8-1aea-4b07-801e-fe9f5f86224f-run-systemd\") pod \"ovnkube-node-qgbk5\" (UID: \"b8eb06e8-1aea-4b07-801e-fe9f5f86224f\") " pod="openshift-ovn-kubernetes/ovnkube-node-qgbk5" Jul 07 17:40:42.027971 ip-10-0-129-39 kubenswrapper[2575]: I0707 17:40:42.025164 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/2d51e575-aadb-4cdd-a6cf-1ef119b49029-lib-modules\") pod \"tuned-7hrdq\" (UID: \"2d51e575-aadb-4cdd-a6cf-1ef119b49029\") " pod="openshift-cluster-node-tuning-operator/tuned-7hrdq" Jul 07 17:40:42.027971 ip-10-0-129-39 kubenswrapper[2575]: I0707 17:40:42.025199 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/4f2c6f8d-3b44-4f4f-8d6b-433f013a867f-konnectivity-ca\") pod \"konnectivity-agent-hr726\" (UID: \"4f2c6f8d-3b44-4f4f-8d6b-433f013a867f\") " pod="kube-system/konnectivity-agent-hr726" Jul 07 17:40:42.027971 ip-10-0-129-39 kubenswrapper[2575]: I0707 17:40:42.025238 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/a5765746-152c-4b60-b664-2dc9c75bd013-kubelet-dir\") pod \"aws-ebs-csi-driver-node-btqqg\" (UID: \"a5765746-152c-4b60-b664-2dc9c75bd013\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-btqqg" Jul 07 17:40:42.027971 ip-10-0-129-39 kubenswrapper[2575]: I0707 17:40:42.025242 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/ad91fa94-7ae7-4245-90ee-78701b164dd2-multus-daemon-config\") pod \"multus-rs8sg\" (UID: \"ad91fa94-7ae7-4245-90ee-78701b164dd2\") " pod="openshift-multus/multus-rs8sg" Jul 07 17:40:42.027971 ip-10-0-129-39 kubenswrapper[2575]: I0707 17:40:42.025256 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/2d51e575-aadb-4cdd-a6cf-1ef119b49029-lib-modules\") pod \"tuned-7hrdq\" (UID: \"2d51e575-aadb-4cdd-a6cf-1ef119b49029\") " pod="openshift-cluster-node-tuning-operator/tuned-7hrdq" Jul 07 17:40:42.027971 ip-10-0-129-39 kubenswrapper[2575]: I0707 17:40:42.025262 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hlsmf\" (UniqueName: \"kubernetes.io/projected/d4c44345-d6bf-4778-8911-09c7a1a0740a-kube-api-access-hlsmf\") pod \"node-resolver-fc7qv\" (UID: \"d4c44345-d6bf-4778-8911-09c7a1a0740a\") " pod="openshift-dns/node-resolver-fc7qv" Jul 07 17:40:42.027971 ip-10-0-129-39 kubenswrapper[2575]: I0707 17:40:42.025298 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/a5765746-152c-4b60-b664-2dc9c75bd013-kubelet-dir\") pod \"aws-ebs-csi-driver-node-btqqg\" (UID: \"a5765746-152c-4b60-b664-2dc9c75bd013\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-btqqg" Jul 07 17:40:42.027971 ip-10-0-129-39 kubenswrapper[2575]: I0707 17:40:42.025750 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/4f2c6f8d-3b44-4f4f-8d6b-433f013a867f-konnectivity-ca\") pod \"konnectivity-agent-hr726\" (UID: \"4f2c6f8d-3b44-4f4f-8d6b-433f013a867f\") " pod="kube-system/konnectivity-agent-hr726" Jul 07 17:40:42.027971 ip-10-0-129-39 kubenswrapper[2575]: I0707 17:40:42.026731 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/2d51e575-aadb-4cdd-a6cf-1ef119b49029-etc-tuned\") pod \"tuned-7hrdq\" (UID: \"2d51e575-aadb-4cdd-a6cf-1ef119b49029\") " pod="openshift-cluster-node-tuning-operator/tuned-7hrdq" Jul 07 17:40:42.027971 ip-10-0-129-39 kubenswrapper[2575]: I0707 17:40:42.027064 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/4f2c6f8d-3b44-4f4f-8d6b-433f013a867f-agent-certs\") pod \"konnectivity-agent-hr726\" (UID: \"4f2c6f8d-3b44-4f4f-8d6b-433f013a867f\") " pod="kube-system/konnectivity-agent-hr726" Jul 07 17:40:42.028420 ip-10-0-129-39 kubenswrapper[2575]: I0707 17:40:42.027600 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/2d51e575-aadb-4cdd-a6cf-1ef119b49029-tmp\") pod \"tuned-7hrdq\" (UID: \"2d51e575-aadb-4cdd-a6cf-1ef119b49029\") " pod="openshift-cluster-node-tuning-operator/tuned-7hrdq" Jul 07 17:40:42.028420 ip-10-0-129-39 kubenswrapper[2575]: I0707 17:40:42.028322 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/b8eb06e8-1aea-4b07-801e-fe9f5f86224f-ovn-node-metrics-cert\") pod \"ovnkube-node-qgbk5\" (UID: \"b8eb06e8-1aea-4b07-801e-fe9f5f86224f\") " pod="openshift-ovn-kubernetes/ovnkube-node-qgbk5" Jul 07 17:40:42.031429 ip-10-0-129-39 kubenswrapper[2575]: I0707 17:40:42.031403 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wh2cb\" (UniqueName: \"kubernetes.io/projected/b8eb06e8-1aea-4b07-801e-fe9f5f86224f-kube-api-access-wh2cb\") pod \"ovnkube-node-qgbk5\" (UID: \"b8eb06e8-1aea-4b07-801e-fe9f5f86224f\") " pod="openshift-ovn-kubernetes/ovnkube-node-qgbk5" Jul 07 17:40:42.031842 ip-10-0-129-39 kubenswrapper[2575]: I0707 17:40:42.031821 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-r4vhp\" (UniqueName: \"kubernetes.io/projected/a5765746-152c-4b60-b664-2dc9c75bd013-kube-api-access-r4vhp\") pod \"aws-ebs-csi-driver-node-btqqg\" (UID: \"a5765746-152c-4b60-b664-2dc9c75bd013\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-btqqg" Jul 07 17:40:42.031932 ip-10-0-129-39 kubenswrapper[2575]: I0707 17:40:42.031879 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vtmvs\" (UniqueName: \"kubernetes.io/projected/2d51e575-aadb-4cdd-a6cf-1ef119b49029-kube-api-access-vtmvs\") pod \"tuned-7hrdq\" (UID: \"2d51e575-aadb-4cdd-a6cf-1ef119b49029\") " pod="openshift-cluster-node-tuning-operator/tuned-7hrdq" Jul 07 17:40:42.032351 ip-10-0-129-39 kubenswrapper[2575]: I0707 17:40:42.032322 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8kv7w\" (UniqueName: \"kubernetes.io/projected/ad91fa94-7ae7-4245-90ee-78701b164dd2-kube-api-access-8kv7w\") pod \"multus-rs8sg\" (UID: \"ad91fa94-7ae7-4245-90ee-78701b164dd2\") " pod="openshift-multus/multus-rs8sg" Jul 07 17:40:42.072517 ip-10-0-129-39 kubenswrapper[2575]: I0707 17:40:42.072464 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-129-39.ec2.internal" event={"ID":"9178fa6a4c12c534b2a4b3275d3b9775","Type":"ContainerStarted","Data":"6f1fe83224870054bbc02f0b0ec20d51b89bf7d51e45c9b2025f0ed6e97e518f"} Jul 07 17:40:42.073589 ip-10-0-129-39 kubenswrapper[2575]: I0707 17:40:42.073568 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-39.ec2.internal" event={"ID":"2f2a5517e65a2e6eea49dbce93563b2c","Type":"ContainerStarted","Data":"9e8cea172e4e89937539ea6d5874a8ece100d3f5c09aeee76724ce5f1da87f7d"} Jul 07 17:40:42.125871 ip-10-0-129-39 kubenswrapper[2575]: I0707 17:40:42.125843 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/1233b6d4-571b-4319-b523-436909681456-host-slash\") pod \"iptables-alerter-5vc77\" (UID: \"1233b6d4-571b-4319-b523-436909681456\") " pod="openshift-network-operator/iptables-alerter-5vc77" Jul 07 17:40:42.125986 ip-10-0-129-39 kubenswrapper[2575]: I0707 17:40:42.125886 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4zlmc\" (UniqueName: \"kubernetes.io/projected/d587896d-9156-4b75-aa70-27962a977051-kube-api-access-4zlmc\") pod \"network-metrics-daemon-gz97j\" (UID: \"d587896d-9156-4b75-aa70-27962a977051\") " pod="openshift-multus/network-metrics-daemon-gz97j" Jul 07 17:40:42.125986 ip-10-0-129-39 kubenswrapper[2575]: I0707 17:40:42.125916 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/59b94723-b7c1-4eb7-be5f-4ec4254b915d-cnibin\") pod \"multus-additional-cni-plugins-9m48z\" (UID: \"59b94723-b7c1-4eb7-be5f-4ec4254b915d\") " pod="openshift-multus/multus-additional-cni-plugins-9m48z" Jul 07 17:40:42.125986 ip-10-0-129-39 kubenswrapper[2575]: I0707 17:40:42.125940 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lm2wr\" (UniqueName: \"kubernetes.io/projected/59b94723-b7c1-4eb7-be5f-4ec4254b915d-kube-api-access-lm2wr\") pod \"multus-additional-cni-plugins-9m48z\" (UID: \"59b94723-b7c1-4eb7-be5f-4ec4254b915d\") " pod="openshift-multus/multus-additional-cni-plugins-9m48z" Jul 07 17:40:42.125986 ip-10-0-129-39 kubenswrapper[2575]: I0707 17:40:42.125949 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/1233b6d4-571b-4319-b523-436909681456-host-slash\") pod \"iptables-alerter-5vc77\" (UID: \"1233b6d4-571b-4319-b523-436909681456\") " pod="openshift-network-operator/iptables-alerter-5vc77" Jul 07 17:40:42.125986 ip-10-0-129-39 kubenswrapper[2575]: I0707 17:40:42.125965 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/d4c44345-d6bf-4778-8911-09c7a1a0740a-hosts-file\") pod \"node-resolver-fc7qv\" (UID: \"d4c44345-d6bf-4778-8911-09c7a1a0740a\") " pod="openshift-dns/node-resolver-fc7qv" Jul 07 17:40:42.126241 ip-10-0-129-39 kubenswrapper[2575]: I0707 17:40:42.126033 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/d4c44345-d6bf-4778-8911-09c7a1a0740a-hosts-file\") pod \"node-resolver-fc7qv\" (UID: \"d4c44345-d6bf-4778-8911-09c7a1a0740a\") " pod="openshift-dns/node-resolver-fc7qv" Jul 07 17:40:42.126241 ip-10-0-129-39 kubenswrapper[2575]: I0707 17:40:42.126031 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/59b94723-b7c1-4eb7-be5f-4ec4254b915d-cnibin\") pod \"multus-additional-cni-plugins-9m48z\" (UID: \"59b94723-b7c1-4eb7-be5f-4ec4254b915d\") " pod="openshift-multus/multus-additional-cni-plugins-9m48z" Jul 07 17:40:42.126241 ip-10-0-129-39 kubenswrapper[2575]: I0707 17:40:42.126142 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d4c44345-d6bf-4778-8911-09c7a1a0740a-tmp-dir\") pod \"node-resolver-fc7qv\" (UID: \"d4c44345-d6bf-4778-8911-09c7a1a0740a\") " pod="openshift-dns/node-resolver-fc7qv" Jul 07 17:40:42.126241 ip-10-0-129-39 kubenswrapper[2575]: I0707 17:40:42.126176 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/59b94723-b7c1-4eb7-be5f-4ec4254b915d-os-release\") pod \"multus-additional-cni-plugins-9m48z\" (UID: \"59b94723-b7c1-4eb7-be5f-4ec4254b915d\") " pod="openshift-multus/multus-additional-cni-plugins-9m48z" Jul 07 17:40:42.126241 ip-10-0-129-39 kubenswrapper[2575]: I0707 17:40:42.126200 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/59b94723-b7c1-4eb7-be5f-4ec4254b915d-cni-binary-copy\") pod \"multus-additional-cni-plugins-9m48z\" (UID: \"59b94723-b7c1-4eb7-be5f-4ec4254b915d\") " pod="openshift-multus/multus-additional-cni-plugins-9m48z" Jul 07 17:40:42.126241 ip-10-0-129-39 kubenswrapper[2575]: I0707 17:40:42.126226 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/59b94723-b7c1-4eb7-be5f-4ec4254b915d-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-9m48z\" (UID: \"59b94723-b7c1-4eb7-be5f-4ec4254b915d\") " pod="openshift-multus/multus-additional-cni-plugins-9m48z" Jul 07 17:40:42.126590 ip-10-0-129-39 kubenswrapper[2575]: I0707 17:40:42.126257 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/1233b6d4-571b-4319-b523-436909681456-iptables-alerter-script\") pod \"iptables-alerter-5vc77\" (UID: \"1233b6d4-571b-4319-b523-436909681456\") " pod="openshift-network-operator/iptables-alerter-5vc77" Jul 07 17:40:42.126590 ip-10-0-129-39 kubenswrapper[2575]: I0707 17:40:42.126275 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/59b94723-b7c1-4eb7-be5f-4ec4254b915d-os-release\") pod \"multus-additional-cni-plugins-9m48z\" (UID: \"59b94723-b7c1-4eb7-be5f-4ec4254b915d\") " pod="openshift-multus/multus-additional-cni-plugins-9m48z" Jul 07 17:40:42.126590 ip-10-0-129-39 kubenswrapper[2575]: I0707 17:40:42.126280 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/ff237df8-d435-4d8c-88ec-8f31795bf0ef-serviceca\") pod \"node-ca-l6pnn\" (UID: \"ff237df8-d435-4d8c-88ec-8f31795bf0ef\") " pod="openshift-image-registry/node-ca-l6pnn" Jul 07 17:40:42.126590 ip-10-0-129-39 kubenswrapper[2575]: I0707 17:40:42.126342 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hlsmf\" (UniqueName: \"kubernetes.io/projected/d4c44345-d6bf-4778-8911-09c7a1a0740a-kube-api-access-hlsmf\") pod \"node-resolver-fc7qv\" (UID: \"d4c44345-d6bf-4778-8911-09c7a1a0740a\") " pod="openshift-dns/node-resolver-fc7qv" Jul 07 17:40:42.126590 ip-10-0-129-39 kubenswrapper[2575]: I0707 17:40:42.126373 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gzrcq\" (UniqueName: \"kubernetes.io/projected/1233b6d4-571b-4319-b523-436909681456-kube-api-access-gzrcq\") pod \"iptables-alerter-5vc77\" (UID: \"1233b6d4-571b-4319-b523-436909681456\") " pod="openshift-network-operator/iptables-alerter-5vc77" Jul 07 17:40:42.126590 ip-10-0-129-39 kubenswrapper[2575]: I0707 17:40:42.126397 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/ff237df8-d435-4d8c-88ec-8f31795bf0ef-host\") pod \"node-ca-l6pnn\" (UID: \"ff237df8-d435-4d8c-88ec-8f31795bf0ef\") " pod="openshift-image-registry/node-ca-l6pnn" Jul 07 17:40:42.126590 ip-10-0-129-39 kubenswrapper[2575]: I0707 17:40:42.126421 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/59b94723-b7c1-4eb7-be5f-4ec4254b915d-tuning-conf-dir\") pod \"multus-additional-cni-plugins-9m48z\" (UID: \"59b94723-b7c1-4eb7-be5f-4ec4254b915d\") " pod="openshift-multus/multus-additional-cni-plugins-9m48z" Jul 07 17:40:42.126590 ip-10-0-129-39 kubenswrapper[2575]: I0707 17:40:42.126448 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4962z\" (UniqueName: \"kubernetes.io/projected/ff237df8-d435-4d8c-88ec-8f31795bf0ef-kube-api-access-4962z\") pod \"node-ca-l6pnn\" (UID: \"ff237df8-d435-4d8c-88ec-8f31795bf0ef\") " pod="openshift-image-registry/node-ca-l6pnn" Jul 07 17:40:42.126590 ip-10-0-129-39 kubenswrapper[2575]: I0707 17:40:42.126477 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/59b94723-b7c1-4eb7-be5f-4ec4254b915d-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-9m48z\" (UID: \"59b94723-b7c1-4eb7-be5f-4ec4254b915d\") " pod="openshift-multus/multus-additional-cni-plugins-9m48z" Jul 07 17:40:42.126590 ip-10-0-129-39 kubenswrapper[2575]: I0707 17:40:42.126478 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d4c44345-d6bf-4778-8911-09c7a1a0740a-tmp-dir\") pod \"node-resolver-fc7qv\" (UID: \"d4c44345-d6bf-4778-8911-09c7a1a0740a\") " pod="openshift-dns/node-resolver-fc7qv" Jul 07 17:40:42.126590 ip-10-0-129-39 kubenswrapper[2575]: I0707 17:40:42.126520 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d587896d-9156-4b75-aa70-27962a977051-metrics-certs\") pod \"network-metrics-daemon-gz97j\" (UID: \"d587896d-9156-4b75-aa70-27962a977051\") " pod="openshift-multus/network-metrics-daemon-gz97j" Jul 07 17:40:42.126590 ip-10-0-129-39 kubenswrapper[2575]: I0707 17:40:42.126571 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/59b94723-b7c1-4eb7-be5f-4ec4254b915d-system-cni-dir\") pod \"multus-additional-cni-plugins-9m48z\" (UID: \"59b94723-b7c1-4eb7-be5f-4ec4254b915d\") " pod="openshift-multus/multus-additional-cni-plugins-9m48z" Jul 07 17:40:42.126590 ip-10-0-129-39 kubenswrapper[2575]: I0707 17:40:42.126594 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-78bqt\" (UniqueName: \"kubernetes.io/projected/5d33c1cb-fcda-490d-9694-b1d1eb16b8da-kube-api-access-78bqt\") pod \"network-check-target-vzf4m\" (UID: \"5d33c1cb-fcda-490d-9694-b1d1eb16b8da\") " pod="openshift-network-diagnostics/network-check-target-vzf4m" Jul 07 17:40:42.127192 ip-10-0-129-39 kubenswrapper[2575]: I0707 17:40:42.126620 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/59b94723-b7c1-4eb7-be5f-4ec4254b915d-tuning-conf-dir\") pod \"multus-additional-cni-plugins-9m48z\" (UID: \"59b94723-b7c1-4eb7-be5f-4ec4254b915d\") " pod="openshift-multus/multus-additional-cni-plugins-9m48z" Jul 07 17:40:42.127192 ip-10-0-129-39 kubenswrapper[2575]: I0707 17:40:42.126668 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/ff237df8-d435-4d8c-88ec-8f31795bf0ef-host\") pod \"node-ca-l6pnn\" (UID: \"ff237df8-d435-4d8c-88ec-8f31795bf0ef\") " pod="openshift-image-registry/node-ca-l6pnn" Jul 07 17:40:42.127192 ip-10-0-129-39 kubenswrapper[2575]: E0707 17:40:42.126706 2575 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 07 17:40:42.127192 ip-10-0-129-39 kubenswrapper[2575]: I0707 17:40:42.126737 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/ff237df8-d435-4d8c-88ec-8f31795bf0ef-serviceca\") pod \"node-ca-l6pnn\" (UID: \"ff237df8-d435-4d8c-88ec-8f31795bf0ef\") " pod="openshift-image-registry/node-ca-l6pnn" Jul 07 17:40:42.127192 ip-10-0-129-39 kubenswrapper[2575]: I0707 17:40:42.126792 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/59b94723-b7c1-4eb7-be5f-4ec4254b915d-cni-binary-copy\") pod \"multus-additional-cni-plugins-9m48z\" (UID: \"59b94723-b7c1-4eb7-be5f-4ec4254b915d\") " pod="openshift-multus/multus-additional-cni-plugins-9m48z" Jul 07 17:40:42.127192 ip-10-0-129-39 kubenswrapper[2575]: E0707 17:40:42.126802 2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d587896d-9156-4b75-aa70-27962a977051-metrics-certs podName:d587896d-9156-4b75-aa70-27962a977051 nodeName:}" failed. No retries permitted until 2026-07-07 17:40:42.626784592 +0000 UTC m=+3.132591243 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/d587896d-9156-4b75-aa70-27962a977051-metrics-certs") pod "network-metrics-daemon-gz97j" (UID: "d587896d-9156-4b75-aa70-27962a977051") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 07 17:40:42.127192 ip-10-0-129-39 kubenswrapper[2575]: I0707 17:40:42.126843 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/59b94723-b7c1-4eb7-be5f-4ec4254b915d-system-cni-dir\") pod \"multus-additional-cni-plugins-9m48z\" (UID: \"59b94723-b7c1-4eb7-be5f-4ec4254b915d\") " pod="openshift-multus/multus-additional-cni-plugins-9m48z" Jul 07 17:40:42.127534 ip-10-0-129-39 kubenswrapper[2575]: I0707 17:40:42.127320 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/59b94723-b7c1-4eb7-be5f-4ec4254b915d-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-9m48z\" (UID: \"59b94723-b7c1-4eb7-be5f-4ec4254b915d\") " pod="openshift-multus/multus-additional-cni-plugins-9m48z" Jul 07 17:40:42.127534 ip-10-0-129-39 kubenswrapper[2575]: I0707 17:40:42.127334 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/59b94723-b7c1-4eb7-be5f-4ec4254b915d-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-9m48z\" (UID: \"59b94723-b7c1-4eb7-be5f-4ec4254b915d\") " pod="openshift-multus/multus-additional-cni-plugins-9m48z" Jul 07 17:40:42.127534 ip-10-0-129-39 kubenswrapper[2575]: I0707 17:40:42.127414 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/1233b6d4-571b-4319-b523-436909681456-iptables-alerter-script\") pod \"iptables-alerter-5vc77\" (UID: \"1233b6d4-571b-4319-b523-436909681456\") " pod="openshift-network-operator/iptables-alerter-5vc77" Jul 07 17:40:42.134000 ip-10-0-129-39 kubenswrapper[2575]: E0707 17:40:42.133980 2575 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 07 17:40:42.134108 ip-10-0-129-39 kubenswrapper[2575]: E0707 17:40:42.134006 2575 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 07 17:40:42.134108 ip-10-0-129-39 kubenswrapper[2575]: E0707 17:40:42.134019 2575 projected.go:196] Error preparing data for projected volume kube-api-access-78bqt for pod openshift-network-diagnostics/network-check-target-vzf4m: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 07 17:40:42.134108 ip-10-0-129-39 kubenswrapper[2575]: E0707 17:40:42.134089 2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/5d33c1cb-fcda-490d-9694-b1d1eb16b8da-kube-api-access-78bqt podName:5d33c1cb-fcda-490d-9694-b1d1eb16b8da nodeName:}" failed. No retries permitted until 2026-07-07 17:40:42.634072812 +0000 UTC m=+3.139879475 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-api-access-78bqt" (UniqueName: "kubernetes.io/projected/5d33c1cb-fcda-490d-9694-b1d1eb16b8da-kube-api-access-78bqt") pod "network-check-target-vzf4m" (UID: "5d33c1cb-fcda-490d-9694-b1d1eb16b8da") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 07 17:40:42.136620 ip-10-0-129-39 kubenswrapper[2575]: I0707 17:40:42.136599 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hlsmf\" (UniqueName: \"kubernetes.io/projected/d4c44345-d6bf-4778-8911-09c7a1a0740a-kube-api-access-hlsmf\") pod \"node-resolver-fc7qv\" (UID: \"d4c44345-d6bf-4778-8911-09c7a1a0740a\") " pod="openshift-dns/node-resolver-fc7qv" Jul 07 17:40:42.136930 ip-10-0-129-39 kubenswrapper[2575]: I0707 17:40:42.136911 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4962z\" (UniqueName: \"kubernetes.io/projected/ff237df8-d435-4d8c-88ec-8f31795bf0ef-kube-api-access-4962z\") pod \"node-ca-l6pnn\" (UID: \"ff237df8-d435-4d8c-88ec-8f31795bf0ef\") " pod="openshift-image-registry/node-ca-l6pnn" Jul 07 17:40:42.137014 ip-10-0-129-39 kubenswrapper[2575]: I0707 17:40:42.136934 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gzrcq\" (UniqueName: \"kubernetes.io/projected/1233b6d4-571b-4319-b523-436909681456-kube-api-access-gzrcq\") pod \"iptables-alerter-5vc77\" (UID: \"1233b6d4-571b-4319-b523-436909681456\") " pod="openshift-network-operator/iptables-alerter-5vc77" Jul 07 17:40:42.137067 ip-10-0-129-39 kubenswrapper[2575]: I0707 17:40:42.137030 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4zlmc\" (UniqueName: \"kubernetes.io/projected/d587896d-9156-4b75-aa70-27962a977051-kube-api-access-4zlmc\") pod \"network-metrics-daemon-gz97j\" (UID: \"d587896d-9156-4b75-aa70-27962a977051\") " pod="openshift-multus/network-metrics-daemon-gz97j" Jul 07 17:40:42.137376 ip-10-0-129-39 kubenswrapper[2575]: I0707 17:40:42.137353 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lm2wr\" (UniqueName: \"kubernetes.io/projected/59b94723-b7c1-4eb7-be5f-4ec4254b915d-kube-api-access-lm2wr\") pod \"multus-additional-cni-plugins-9m48z\" (UID: \"59b94723-b7c1-4eb7-be5f-4ec4254b915d\") " pod="openshift-multus/multus-additional-cni-plugins-9m48z" Jul 07 17:40:42.215777 ip-10-0-129-39 kubenswrapper[2575]: I0707 17:40:42.215699 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-7hrdq" Jul 07 17:40:42.224625 ip-10-0-129-39 kubenswrapper[2575]: I0707 17:40:42.224605 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-rs8sg" Jul 07 17:40:42.233644 ip-10-0-129-39 kubenswrapper[2575]: I0707 17:40:42.233624 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-qgbk5" Jul 07 17:40:42.240477 ip-10-0-129-39 kubenswrapper[2575]: I0707 17:40:42.240451 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-hr726" Jul 07 17:40:42.248105 ip-10-0-129-39 kubenswrapper[2575]: I0707 17:40:42.248084 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-btqqg" Jul 07 17:40:42.255829 ip-10-0-129-39 kubenswrapper[2575]: I0707 17:40:42.255808 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-l6pnn" Jul 07 17:40:42.270652 ip-10-0-129-39 kubenswrapper[2575]: I0707 17:40:42.270636 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-fc7qv" Jul 07 17:40:42.277286 ip-10-0-129-39 kubenswrapper[2575]: I0707 17:40:42.277268 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-9m48z" Jul 07 17:40:42.284834 ip-10-0-129-39 kubenswrapper[2575]: I0707 17:40:42.284818 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-5vc77" Jul 07 17:40:42.441238 ip-10-0-129-39 kubenswrapper[2575]: I0707 17:40:42.441208 2575 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160" Jul 07 17:40:42.529820 ip-10-0-129-39 kubenswrapper[2575]: I0707 17:40:42.529735 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/8344b558-69d0-43b3-9aac-0cea0d5379f3-original-pull-secret\") pod \"global-pull-secret-syncer-76r79\" (UID: \"8344b558-69d0-43b3-9aac-0cea0d5379f3\") " pod="kube-system/global-pull-secret-syncer-76r79" Jul 07 17:40:42.529978 ip-10-0-129-39 kubenswrapper[2575]: E0707 17:40:42.529871 2575 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 07 17:40:42.529978 ip-10-0-129-39 kubenswrapper[2575]: E0707 17:40:42.529929 2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8344b558-69d0-43b3-9aac-0cea0d5379f3-original-pull-secret podName:8344b558-69d0-43b3-9aac-0cea0d5379f3 nodeName:}" failed. No retries permitted until 2026-07-07 17:40:43.529913623 +0000 UTC m=+4.035720268 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/8344b558-69d0-43b3-9aac-0cea0d5379f3-original-pull-secret") pod "global-pull-secret-syncer-76r79" (UID: "8344b558-69d0-43b3-9aac-0cea0d5379f3") : object "kube-system"/"original-pull-secret" not registered Jul 07 17:40:42.630835 ip-10-0-129-39 kubenswrapper[2575]: I0707 17:40:42.630798 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d587896d-9156-4b75-aa70-27962a977051-metrics-certs\") pod \"network-metrics-daemon-gz97j\" (UID: \"d587896d-9156-4b75-aa70-27962a977051\") " pod="openshift-multus/network-metrics-daemon-gz97j" Jul 07 17:40:42.630994 ip-10-0-129-39 kubenswrapper[2575]: E0707 17:40:42.630974 2575 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 07 17:40:42.631053 ip-10-0-129-39 kubenswrapper[2575]: E0707 17:40:42.631041 2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d587896d-9156-4b75-aa70-27962a977051-metrics-certs podName:d587896d-9156-4b75-aa70-27962a977051 nodeName:}" failed. No retries permitted until 2026-07-07 17:40:43.631026041 +0000 UTC m=+4.136832685 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/d587896d-9156-4b75-aa70-27962a977051-metrics-certs") pod "network-metrics-daemon-gz97j" (UID: "d587896d-9156-4b75-aa70-27962a977051") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 07 17:40:42.732021 ip-10-0-129-39 kubenswrapper[2575]: I0707 17:40:42.731984 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-78bqt\" (UniqueName: \"kubernetes.io/projected/5d33c1cb-fcda-490d-9694-b1d1eb16b8da-kube-api-access-78bqt\") pod \"network-check-target-vzf4m\" (UID: \"5d33c1cb-fcda-490d-9694-b1d1eb16b8da\") " pod="openshift-network-diagnostics/network-check-target-vzf4m" Jul 07 17:40:42.732169 ip-10-0-129-39 kubenswrapper[2575]: E0707 17:40:42.732103 2575 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 07 17:40:42.732169 ip-10-0-129-39 kubenswrapper[2575]: E0707 17:40:42.732116 2575 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 07 17:40:42.732169 ip-10-0-129-39 kubenswrapper[2575]: E0707 17:40:42.732124 2575 projected.go:196] Error preparing data for projected volume kube-api-access-78bqt for pod openshift-network-diagnostics/network-check-target-vzf4m: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 07 17:40:42.732169 ip-10-0-129-39 kubenswrapper[2575]: E0707 17:40:42.732169 2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/5d33c1cb-fcda-490d-9694-b1d1eb16b8da-kube-api-access-78bqt podName:5d33c1cb-fcda-490d-9694-b1d1eb16b8da nodeName:}" failed. No retries permitted until 2026-07-07 17:40:43.732156589 +0000 UTC m=+4.237963233 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-api-access-78bqt" (UniqueName: "kubernetes.io/projected/5d33c1cb-fcda-490d-9694-b1d1eb16b8da-kube-api-access-78bqt") pod "network-check-target-vzf4m" (UID: "5d33c1cb-fcda-490d-9694-b1d1eb16b8da") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 07 17:40:42.973995 ip-10-0-129-39 kubenswrapper[2575]: I0707 17:40:42.973905 2575 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-07-06 17:35:41 +0000 UTC" deadline="2027-12-11 22:40:51.733458445 +0000 UTC" Jul 07 17:40:42.973995 ip-10-0-129-39 kubenswrapper[2575]: I0707 17:40:42.973951 2575 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="12533h0m8.759512013s" Jul 07 17:40:43.042558 ip-10-0-129-39 kubenswrapper[2575]: W0707 17:40:43.042529 2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4f2c6f8d_3b44_4f4f_8d6b_433f013a867f.slice/crio-359f75cd8994b65c1f20ca501d8de8bbb79ea9a6914689498b9cc2459048a8a4 WatchSource:0}: Error finding container 359f75cd8994b65c1f20ca501d8de8bbb79ea9a6914689498b9cc2459048a8a4: Status 404 returned error can't find the container with id 359f75cd8994b65c1f20ca501d8de8bbb79ea9a6914689498b9cc2459048a8a4 Jul 07 17:40:43.043731 ip-10-0-129-39 kubenswrapper[2575]: W0707 17:40:43.043669 2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd4c44345_d6bf_4778_8911_09c7a1a0740a.slice/crio-ce5bb0c102a54ae4ad67ac8997ad10c77c8e990680f8980860f5962c9a86c528 WatchSource:0}: Error finding container ce5bb0c102a54ae4ad67ac8997ad10c77c8e990680f8980860f5962c9a86c528: Status 404 returned error can't find the container with id ce5bb0c102a54ae4ad67ac8997ad10c77c8e990680f8980860f5962c9a86c528 Jul 07 17:40:43.046779 ip-10-0-129-39 kubenswrapper[2575]: W0707 17:40:43.046760 2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podff237df8_d435_4d8c_88ec_8f31795bf0ef.slice/crio-00efb01bac34f2ac54c0c8cf4871c9f1b14da936874352cb8a1364fde681390d WatchSource:0}: Error finding container 00efb01bac34f2ac54c0c8cf4871c9f1b14da936874352cb8a1364fde681390d: Status 404 returned error can't find the container with id 00efb01bac34f2ac54c0c8cf4871c9f1b14da936874352cb8a1364fde681390d Jul 07 17:40:43.047377 ip-10-0-129-39 kubenswrapper[2575]: W0707 17:40:43.047356 2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb8eb06e8_1aea_4b07_801e_fe9f5f86224f.slice/crio-0530c8c7a662c6825b76218710196e9f2295c611bb8745342ddf5923227ad3f5 WatchSource:0}: Error finding container 0530c8c7a662c6825b76218710196e9f2295c611bb8745342ddf5923227ad3f5: Status 404 returned error can't find the container with id 0530c8c7a662c6825b76218710196e9f2295c611bb8745342ddf5923227ad3f5 Jul 07 17:40:43.048059 ip-10-0-129-39 kubenswrapper[2575]: W0707 17:40:43.048035 2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda5765746_152c_4b60_b664_2dc9c75bd013.slice/crio-ba98f9861f1e4b6573a8b45f4eb53084450e24dd11330e5475ae80eea42519d7 WatchSource:0}: Error finding container ba98f9861f1e4b6573a8b45f4eb53084450e24dd11330e5475ae80eea42519d7: Status 404 returned error can't find the container with id ba98f9861f1e4b6573a8b45f4eb53084450e24dd11330e5475ae80eea42519d7 Jul 07 17:40:43.048844 ip-10-0-129-39 kubenswrapper[2575]: W0707 17:40:43.048819 2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2d51e575_aadb_4cdd_a6cf_1ef119b49029.slice/crio-2a6614647d3956eb20fd3646c597383e40cfab7a3fd0880ef931f09e2fad438b WatchSource:0}: Error finding container 2a6614647d3956eb20fd3646c597383e40cfab7a3fd0880ef931f09e2fad438b: Status 404 returned error can't find the container with id 2a6614647d3956eb20fd3646c597383e40cfab7a3fd0880ef931f09e2fad438b Jul 07 17:40:43.049779 ip-10-0-129-39 kubenswrapper[2575]: W0707 17:40:43.049759 2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podad91fa94_7ae7_4245_90ee_78701b164dd2.slice/crio-92ff74f9eaaeb368de475224014e14c6f681cd5ae64e440e85f5d162901d413d WatchSource:0}: Error finding container 92ff74f9eaaeb368de475224014e14c6f681cd5ae64e440e85f5d162901d413d: Status 404 returned error can't find the container with id 92ff74f9eaaeb368de475224014e14c6f681cd5ae64e440e85f5d162901d413d Jul 07 17:40:43.052584 ip-10-0-129-39 kubenswrapper[2575]: W0707 17:40:43.051941 2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1233b6d4_571b_4319_b523_436909681456.slice/crio-55b3d17581ac490c5e76517c27a9cbe03290d9f78c56318794b329209ff17e16 WatchSource:0}: Error finding container 55b3d17581ac490c5e76517c27a9cbe03290d9f78c56318794b329209ff17e16: Status 404 returned error can't find the container with id 55b3d17581ac490c5e76517c27a9cbe03290d9f78c56318794b329209ff17e16 Jul 07 17:40:43.076079 ip-10-0-129-39 kubenswrapper[2575]: I0707 17:40:43.076055 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-9m48z" event={"ID":"59b94723-b7c1-4eb7-be5f-4ec4254b915d","Type":"ContainerStarted","Data":"8ec15a710e614c65a0f3280c989afbf28666f16ba43f3eed0f15339ec712cb46"} Jul 07 17:40:43.077265 ip-10-0-129-39 kubenswrapper[2575]: I0707 17:40:43.077243 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-5vc77" event={"ID":"1233b6d4-571b-4319-b523-436909681456","Type":"ContainerStarted","Data":"55b3d17581ac490c5e76517c27a9cbe03290d9f78c56318794b329209ff17e16"} Jul 07 17:40:43.078533 ip-10-0-129-39 kubenswrapper[2575]: I0707 17:40:43.078490 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-rs8sg" event={"ID":"ad91fa94-7ae7-4245-90ee-78701b164dd2","Type":"ContainerStarted","Data":"92ff74f9eaaeb368de475224014e14c6f681cd5ae64e440e85f5d162901d413d"} Jul 07 17:40:43.080430 ip-10-0-129-39 kubenswrapper[2575]: I0707 17:40:43.080406 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-btqqg" event={"ID":"a5765746-152c-4b60-b664-2dc9c75bd013","Type":"ContainerStarted","Data":"ba98f9861f1e4b6573a8b45f4eb53084450e24dd11330e5475ae80eea42519d7"} Jul 07 17:40:43.085307 ip-10-0-129-39 kubenswrapper[2575]: I0707 17:40:43.085288 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-fc7qv" event={"ID":"d4c44345-d6bf-4778-8911-09c7a1a0740a","Type":"ContainerStarted","Data":"ce5bb0c102a54ae4ad67ac8997ad10c77c8e990680f8980860f5962c9a86c528"} Jul 07 17:40:43.089155 ip-10-0-129-39 kubenswrapper[2575]: I0707 17:40:43.089135 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-7hrdq" event={"ID":"2d51e575-aadb-4cdd-a6cf-1ef119b49029","Type":"ContainerStarted","Data":"2a6614647d3956eb20fd3646c597383e40cfab7a3fd0880ef931f09e2fad438b"} Jul 07 17:40:43.090123 ip-10-0-129-39 kubenswrapper[2575]: I0707 17:40:43.090103 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-qgbk5" event={"ID":"b8eb06e8-1aea-4b07-801e-fe9f5f86224f","Type":"ContainerStarted","Data":"0530c8c7a662c6825b76218710196e9f2295c611bb8745342ddf5923227ad3f5"} Jul 07 17:40:43.091158 ip-10-0-129-39 kubenswrapper[2575]: I0707 17:40:43.091141 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-l6pnn" event={"ID":"ff237df8-d435-4d8c-88ec-8f31795bf0ef","Type":"ContainerStarted","Data":"00efb01bac34f2ac54c0c8cf4871c9f1b14da936874352cb8a1364fde681390d"} Jul 07 17:40:43.092200 ip-10-0-129-39 kubenswrapper[2575]: I0707 17:40:43.092180 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-hr726" event={"ID":"4f2c6f8d-3b44-4f4f-8d6b-433f013a867f","Type":"ContainerStarted","Data":"359f75cd8994b65c1f20ca501d8de8bbb79ea9a6914689498b9cc2459048a8a4"} Jul 07 17:40:43.538310 ip-10-0-129-39 kubenswrapper[2575]: I0707 17:40:43.538026 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/8344b558-69d0-43b3-9aac-0cea0d5379f3-original-pull-secret\") pod \"global-pull-secret-syncer-76r79\" (UID: \"8344b558-69d0-43b3-9aac-0cea0d5379f3\") " pod="kube-system/global-pull-secret-syncer-76r79" Jul 07 17:40:43.538310 ip-10-0-129-39 kubenswrapper[2575]: E0707 17:40:43.538257 2575 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 07 17:40:43.538552 ip-10-0-129-39 kubenswrapper[2575]: E0707 17:40:43.538322 2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8344b558-69d0-43b3-9aac-0cea0d5379f3-original-pull-secret podName:8344b558-69d0-43b3-9aac-0cea0d5379f3 nodeName:}" failed. No retries permitted until 2026-07-07 17:40:45.538302629 +0000 UTC m=+6.044109294 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/8344b558-69d0-43b3-9aac-0cea0d5379f3-original-pull-secret") pod "global-pull-secret-syncer-76r79" (UID: "8344b558-69d0-43b3-9aac-0cea0d5379f3") : object "kube-system"/"original-pull-secret" not registered Jul 07 17:40:43.639521 ip-10-0-129-39 kubenswrapper[2575]: I0707 17:40:43.639479 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d587896d-9156-4b75-aa70-27962a977051-metrics-certs\") pod \"network-metrics-daemon-gz97j\" (UID: \"d587896d-9156-4b75-aa70-27962a977051\") " pod="openshift-multus/network-metrics-daemon-gz97j" Jul 07 17:40:43.639736 ip-10-0-129-39 kubenswrapper[2575]: E0707 17:40:43.639619 2575 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 07 17:40:43.639816 ip-10-0-129-39 kubenswrapper[2575]: E0707 17:40:43.639780 2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d587896d-9156-4b75-aa70-27962a977051-metrics-certs podName:d587896d-9156-4b75-aa70-27962a977051 nodeName:}" failed. No retries permitted until 2026-07-07 17:40:45.639761714 +0000 UTC m=+6.145568359 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/d587896d-9156-4b75-aa70-27962a977051-metrics-certs") pod "network-metrics-daemon-gz97j" (UID: "d587896d-9156-4b75-aa70-27962a977051") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 07 17:40:43.740122 ip-10-0-129-39 kubenswrapper[2575]: I0707 17:40:43.740088 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-78bqt\" (UniqueName: \"kubernetes.io/projected/5d33c1cb-fcda-490d-9694-b1d1eb16b8da-kube-api-access-78bqt\") pod \"network-check-target-vzf4m\" (UID: \"5d33c1cb-fcda-490d-9694-b1d1eb16b8da\") " pod="openshift-network-diagnostics/network-check-target-vzf4m" Jul 07 17:40:43.740291 ip-10-0-129-39 kubenswrapper[2575]: E0707 17:40:43.740245 2575 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 07 17:40:43.740291 ip-10-0-129-39 kubenswrapper[2575]: E0707 17:40:43.740262 2575 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 07 17:40:43.740291 ip-10-0-129-39 kubenswrapper[2575]: E0707 17:40:43.740274 2575 projected.go:196] Error preparing data for projected volume kube-api-access-78bqt for pod openshift-network-diagnostics/network-check-target-vzf4m: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 07 17:40:43.740442 ip-10-0-129-39 kubenswrapper[2575]: E0707 17:40:43.740326 2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/5d33c1cb-fcda-490d-9694-b1d1eb16b8da-kube-api-access-78bqt podName:5d33c1cb-fcda-490d-9694-b1d1eb16b8da nodeName:}" failed. No retries permitted until 2026-07-07 17:40:45.740308961 +0000 UTC m=+6.246115611 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-api-access-78bqt" (UniqueName: "kubernetes.io/projected/5d33c1cb-fcda-490d-9694-b1d1eb16b8da-kube-api-access-78bqt") pod "network-check-target-vzf4m" (UID: "5d33c1cb-fcda-490d-9694-b1d1eb16b8da") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 07 17:40:43.974836 ip-10-0-129-39 kubenswrapper[2575]: I0707 17:40:43.974734 2575 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-07-06 17:35:41 +0000 UTC" deadline="2027-12-18 08:48:47.768914777 +0000 UTC" Jul 07 17:40:43.974836 ip-10-0-129-39 kubenswrapper[2575]: I0707 17:40:43.974793 2575 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="12687h8m3.794125707s" Jul 07 17:40:44.069100 ip-10-0-129-39 kubenswrapper[2575]: I0707 17:40:44.068629 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-vzf4m" Jul 07 17:40:44.069100 ip-10-0-129-39 kubenswrapper[2575]: I0707 17:40:44.068754 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-gz97j" Jul 07 17:40:44.069100 ip-10-0-129-39 kubenswrapper[2575]: E0707 17:40:44.068873 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-gz97j" podUID="d587896d-9156-4b75-aa70-27962a977051" Jul 07 17:40:44.069100 ip-10-0-129-39 kubenswrapper[2575]: E0707 17:40:44.068752 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-vzf4m" podUID="5d33c1cb-fcda-490d-9694-b1d1eb16b8da" Jul 07 17:40:44.069100 ip-10-0-129-39 kubenswrapper[2575]: I0707 17:40:44.068629 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-76r79" Jul 07 17:40:44.069100 ip-10-0-129-39 kubenswrapper[2575]: E0707 17:40:44.068972 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-76r79" podUID="8344b558-69d0-43b3-9aac-0cea0d5379f3" Jul 07 17:40:44.118525 ip-10-0-129-39 kubenswrapper[2575]: I0707 17:40:44.117726 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-129-39.ec2.internal" event={"ID":"9178fa6a4c12c534b2a4b3275d3b9775","Type":"ContainerStarted","Data":"a8bed1d58049f25b0661f79ed49d07b47dd96ef7b578afa35774c8669f68689f"} Jul 07 17:40:44.132107 ip-10-0-129-39 kubenswrapper[2575]: I0707 17:40:44.131541 2575 generic.go:356] "Generic (PLEG): container finished" podID="2f2a5517e65a2e6eea49dbce93563b2c" containerID="a9dc0bedf0ef6936b42ba8bfda3a9ab1c74f5fda7f2bbdfd6ecefda0fddde9ce" exitCode=0 Jul 07 17:40:44.132107 ip-10-0-129-39 kubenswrapper[2575]: I0707 17:40:44.131588 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-39.ec2.internal" event={"ID":"2f2a5517e65a2e6eea49dbce93563b2c","Type":"ContainerDied","Data":"a9dc0bedf0ef6936b42ba8bfda3a9ab1c74f5fda7f2bbdfd6ecefda0fddde9ce"} Jul 07 17:40:44.134748 ip-10-0-129-39 kubenswrapper[2575]: I0707 17:40:44.133580 2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/kube-apiserver-proxy-ip-10-0-129-39.ec2.internal" podStartSLOduration=3.133562328 podStartE2EDuration="3.133562328s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-07 17:40:41 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 17:40:44.133239295 +0000 UTC m=+4.639045963" watchObservedRunningTime="2026-07-07 17:40:44.133562328 +0000 UTC m=+4.639368995" Jul 07 17:40:45.141731 ip-10-0-129-39 kubenswrapper[2575]: I0707 17:40:45.141697 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-39.ec2.internal" event={"ID":"2f2a5517e65a2e6eea49dbce93563b2c","Type":"ContainerStarted","Data":"729d12f419a5b2bf93b1bbf1edddad53894648a7dc5f1d824c690c1524fd37d1"} Jul 07 17:40:45.557538 ip-10-0-129-39 kubenswrapper[2575]: I0707 17:40:45.556918 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/8344b558-69d0-43b3-9aac-0cea0d5379f3-original-pull-secret\") pod \"global-pull-secret-syncer-76r79\" (UID: \"8344b558-69d0-43b3-9aac-0cea0d5379f3\") " pod="kube-system/global-pull-secret-syncer-76r79" Jul 07 17:40:45.557538 ip-10-0-129-39 kubenswrapper[2575]: E0707 17:40:45.557076 2575 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 07 17:40:45.557538 ip-10-0-129-39 kubenswrapper[2575]: E0707 17:40:45.557127 2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8344b558-69d0-43b3-9aac-0cea0d5379f3-original-pull-secret podName:8344b558-69d0-43b3-9aac-0cea0d5379f3 nodeName:}" failed. No retries permitted until 2026-07-07 17:40:49.557110746 +0000 UTC m=+10.062917395 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/8344b558-69d0-43b3-9aac-0cea0d5379f3-original-pull-secret") pod "global-pull-secret-syncer-76r79" (UID: "8344b558-69d0-43b3-9aac-0cea0d5379f3") : object "kube-system"/"original-pull-secret" not registered Jul 07 17:40:45.657491 ip-10-0-129-39 kubenswrapper[2575]: I0707 17:40:45.657441 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d587896d-9156-4b75-aa70-27962a977051-metrics-certs\") pod \"network-metrics-daemon-gz97j\" (UID: \"d587896d-9156-4b75-aa70-27962a977051\") " pod="openshift-multus/network-metrics-daemon-gz97j" Jul 07 17:40:45.657666 ip-10-0-129-39 kubenswrapper[2575]: E0707 17:40:45.657612 2575 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 07 17:40:45.657738 ip-10-0-129-39 kubenswrapper[2575]: E0707 17:40:45.657681 2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d587896d-9156-4b75-aa70-27962a977051-metrics-certs podName:d587896d-9156-4b75-aa70-27962a977051 nodeName:}" failed. No retries permitted until 2026-07-07 17:40:49.657661637 +0000 UTC m=+10.163468283 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/d587896d-9156-4b75-aa70-27962a977051-metrics-certs") pod "network-metrics-daemon-gz97j" (UID: "d587896d-9156-4b75-aa70-27962a977051") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 07 17:40:45.758219 ip-10-0-129-39 kubenswrapper[2575]: I0707 17:40:45.758189 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-78bqt\" (UniqueName: \"kubernetes.io/projected/5d33c1cb-fcda-490d-9694-b1d1eb16b8da-kube-api-access-78bqt\") pod \"network-check-target-vzf4m\" (UID: \"5d33c1cb-fcda-490d-9694-b1d1eb16b8da\") " pod="openshift-network-diagnostics/network-check-target-vzf4m" Jul 07 17:40:45.758389 ip-10-0-129-39 kubenswrapper[2575]: E0707 17:40:45.758351 2575 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 07 17:40:45.758389 ip-10-0-129-39 kubenswrapper[2575]: E0707 17:40:45.758370 2575 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 07 17:40:45.758389 ip-10-0-129-39 kubenswrapper[2575]: E0707 17:40:45.758381 2575 projected.go:196] Error preparing data for projected volume kube-api-access-78bqt for pod openshift-network-diagnostics/network-check-target-vzf4m: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 07 17:40:45.758563 ip-10-0-129-39 kubenswrapper[2575]: E0707 17:40:45.758435 2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/5d33c1cb-fcda-490d-9694-b1d1eb16b8da-kube-api-access-78bqt podName:5d33c1cb-fcda-490d-9694-b1d1eb16b8da nodeName:}" failed. No retries permitted until 2026-07-07 17:40:49.75841587 +0000 UTC m=+10.264222530 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-api-access-78bqt" (UniqueName: "kubernetes.io/projected/5d33c1cb-fcda-490d-9694-b1d1eb16b8da-kube-api-access-78bqt") pod "network-check-target-vzf4m" (UID: "5d33c1cb-fcda-490d-9694-b1d1eb16b8da") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 07 17:40:46.070269 ip-10-0-129-39 kubenswrapper[2575]: I0707 17:40:46.069592 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-vzf4m" Jul 07 17:40:46.070269 ip-10-0-129-39 kubenswrapper[2575]: E0707 17:40:46.069706 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-vzf4m" podUID="5d33c1cb-fcda-490d-9694-b1d1eb16b8da" Jul 07 17:40:46.070269 ip-10-0-129-39 kubenswrapper[2575]: I0707 17:40:46.070093 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-76r79" Jul 07 17:40:46.070269 ip-10-0-129-39 kubenswrapper[2575]: E0707 17:40:46.070193 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-76r79" podUID="8344b558-69d0-43b3-9aac-0cea0d5379f3" Jul 07 17:40:46.070934 ip-10-0-129-39 kubenswrapper[2575]: I0707 17:40:46.070593 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-gz97j" Jul 07 17:40:46.070934 ip-10-0-129-39 kubenswrapper[2575]: E0707 17:40:46.070699 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-gz97j" podUID="d587896d-9156-4b75-aa70-27962a977051" Jul 07 17:40:48.072211 ip-10-0-129-39 kubenswrapper[2575]: I0707 17:40:48.071396 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-gz97j" Jul 07 17:40:48.072211 ip-10-0-129-39 kubenswrapper[2575]: E0707 17:40:48.071561 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-gz97j" podUID="d587896d-9156-4b75-aa70-27962a977051" Jul 07 17:40:48.072211 ip-10-0-129-39 kubenswrapper[2575]: I0707 17:40:48.071949 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-vzf4m" Jul 07 17:40:48.072211 ip-10-0-129-39 kubenswrapper[2575]: E0707 17:40:48.072040 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-vzf4m" podUID="5d33c1cb-fcda-490d-9694-b1d1eb16b8da" Jul 07 17:40:48.072211 ip-10-0-129-39 kubenswrapper[2575]: I0707 17:40:48.072101 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-76r79" Jul 07 17:40:48.072211 ip-10-0-129-39 kubenswrapper[2575]: E0707 17:40:48.072173 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-76r79" podUID="8344b558-69d0-43b3-9aac-0cea0d5379f3" Jul 07 17:40:49.591089 ip-10-0-129-39 kubenswrapper[2575]: I0707 17:40:49.591053 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/8344b558-69d0-43b3-9aac-0cea0d5379f3-original-pull-secret\") pod \"global-pull-secret-syncer-76r79\" (UID: \"8344b558-69d0-43b3-9aac-0cea0d5379f3\") " pod="kube-system/global-pull-secret-syncer-76r79" Jul 07 17:40:49.591498 ip-10-0-129-39 kubenswrapper[2575]: E0707 17:40:49.591208 2575 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 07 17:40:49.591498 ip-10-0-129-39 kubenswrapper[2575]: E0707 17:40:49.591273 2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8344b558-69d0-43b3-9aac-0cea0d5379f3-original-pull-secret podName:8344b558-69d0-43b3-9aac-0cea0d5379f3 nodeName:}" failed. No retries permitted until 2026-07-07 17:40:57.591256595 +0000 UTC m=+18.097063238 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/8344b558-69d0-43b3-9aac-0cea0d5379f3-original-pull-secret") pod "global-pull-secret-syncer-76r79" (UID: "8344b558-69d0-43b3-9aac-0cea0d5379f3") : object "kube-system"/"original-pull-secret" not registered Jul 07 17:40:49.692297 ip-10-0-129-39 kubenswrapper[2575]: I0707 17:40:49.692232 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d587896d-9156-4b75-aa70-27962a977051-metrics-certs\") pod \"network-metrics-daemon-gz97j\" (UID: \"d587896d-9156-4b75-aa70-27962a977051\") " pod="openshift-multus/network-metrics-daemon-gz97j" Jul 07 17:40:49.692466 ip-10-0-129-39 kubenswrapper[2575]: E0707 17:40:49.692407 2575 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 07 17:40:49.692545 ip-10-0-129-39 kubenswrapper[2575]: E0707 17:40:49.692484 2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d587896d-9156-4b75-aa70-27962a977051-metrics-certs podName:d587896d-9156-4b75-aa70-27962a977051 nodeName:}" failed. No retries permitted until 2026-07-07 17:40:57.692464261 +0000 UTC m=+18.198270907 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/d587896d-9156-4b75-aa70-27962a977051-metrics-certs") pod "network-metrics-daemon-gz97j" (UID: "d587896d-9156-4b75-aa70-27962a977051") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 07 17:40:49.793597 ip-10-0-129-39 kubenswrapper[2575]: I0707 17:40:49.793558 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-78bqt\" (UniqueName: \"kubernetes.io/projected/5d33c1cb-fcda-490d-9694-b1d1eb16b8da-kube-api-access-78bqt\") pod \"network-check-target-vzf4m\" (UID: \"5d33c1cb-fcda-490d-9694-b1d1eb16b8da\") " pod="openshift-network-diagnostics/network-check-target-vzf4m" Jul 07 17:40:49.793783 ip-10-0-129-39 kubenswrapper[2575]: E0707 17:40:49.793722 2575 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 07 17:40:49.793783 ip-10-0-129-39 kubenswrapper[2575]: E0707 17:40:49.793751 2575 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 07 17:40:49.793783 ip-10-0-129-39 kubenswrapper[2575]: E0707 17:40:49.793764 2575 projected.go:196] Error preparing data for projected volume kube-api-access-78bqt for pod openshift-network-diagnostics/network-check-target-vzf4m: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 07 17:40:49.793952 ip-10-0-129-39 kubenswrapper[2575]: E0707 17:40:49.793828 2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/5d33c1cb-fcda-490d-9694-b1d1eb16b8da-kube-api-access-78bqt podName:5d33c1cb-fcda-490d-9694-b1d1eb16b8da nodeName:}" failed. No retries permitted until 2026-07-07 17:40:57.793807491 +0000 UTC m=+18.299614141 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-api-access-78bqt" (UniqueName: "kubernetes.io/projected/5d33c1cb-fcda-490d-9694-b1d1eb16b8da-kube-api-access-78bqt") pod "network-check-target-vzf4m" (UID: "5d33c1cb-fcda-490d-9694-b1d1eb16b8da") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 07 17:40:50.071854 ip-10-0-129-39 kubenswrapper[2575]: I0707 17:40:50.071780 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-gz97j" Jul 07 17:40:50.071998 ip-10-0-129-39 kubenswrapper[2575]: E0707 17:40:50.071955 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-gz97j" podUID="d587896d-9156-4b75-aa70-27962a977051" Jul 07 17:40:50.072424 ip-10-0-129-39 kubenswrapper[2575]: I0707 17:40:50.072403 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-vzf4m" Jul 07 17:40:50.072481 ip-10-0-129-39 kubenswrapper[2575]: I0707 17:40:50.072414 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-76r79" Jul 07 17:40:50.073692 ip-10-0-129-39 kubenswrapper[2575]: E0707 17:40:50.072549 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-vzf4m" podUID="5d33c1cb-fcda-490d-9694-b1d1eb16b8da" Jul 07 17:40:50.073692 ip-10-0-129-39 kubenswrapper[2575]: E0707 17:40:50.072668 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-76r79" podUID="8344b558-69d0-43b3-9aac-0cea0d5379f3" Jul 07 17:40:52.068470 ip-10-0-129-39 kubenswrapper[2575]: I0707 17:40:52.068428 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-vzf4m" Jul 07 17:40:52.068925 ip-10-0-129-39 kubenswrapper[2575]: E0707 17:40:52.068584 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-vzf4m" podUID="5d33c1cb-fcda-490d-9694-b1d1eb16b8da" Jul 07 17:40:52.068925 ip-10-0-129-39 kubenswrapper[2575]: I0707 17:40:52.068748 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-76r79" Jul 07 17:40:52.068925 ip-10-0-129-39 kubenswrapper[2575]: E0707 17:40:52.068837 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-76r79" podUID="8344b558-69d0-43b3-9aac-0cea0d5379f3" Jul 07 17:40:52.069087 ip-10-0-129-39 kubenswrapper[2575]: I0707 17:40:52.068980 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-gz97j" Jul 07 17:40:52.069140 ip-10-0-129-39 kubenswrapper[2575]: E0707 17:40:52.069108 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-gz97j" podUID="d587896d-9156-4b75-aa70-27962a977051" Jul 07 17:40:54.068667 ip-10-0-129-39 kubenswrapper[2575]: I0707 17:40:54.068386 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-vzf4m" Jul 07 17:40:54.069097 ip-10-0-129-39 kubenswrapper[2575]: I0707 17:40:54.068404 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-gz97j" Jul 07 17:40:54.069097 ip-10-0-129-39 kubenswrapper[2575]: E0707 17:40:54.068731 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-vzf4m" podUID="5d33c1cb-fcda-490d-9694-b1d1eb16b8da" Jul 07 17:40:54.069097 ip-10-0-129-39 kubenswrapper[2575]: I0707 17:40:54.068454 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-76r79" Jul 07 17:40:54.069097 ip-10-0-129-39 kubenswrapper[2575]: E0707 17:40:54.068808 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-gz97j" podUID="d587896d-9156-4b75-aa70-27962a977051" Jul 07 17:40:54.069097 ip-10-0-129-39 kubenswrapper[2575]: E0707 17:40:54.068856 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-76r79" podUID="8344b558-69d0-43b3-9aac-0cea0d5379f3" Jul 07 17:40:54.174101 ip-10-0-129-39 kubenswrapper[2575]: I0707 17:40:54.174057 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-l6pnn" event={"ID":"ff237df8-d435-4d8c-88ec-8f31795bf0ef","Type":"ContainerStarted","Data":"3fb00751809d026821e9597a86dee014f264dbf3baa54634aca2c76f78006b96"} Jul 07 17:40:54.175952 ip-10-0-129-39 kubenswrapper[2575]: I0707 17:40:54.175926 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-hr726" event={"ID":"4f2c6f8d-3b44-4f4f-8d6b-433f013a867f","Type":"ContainerStarted","Data":"df70d2ee19e4918e2932e83c2813f6e68a4f1c4b3a1d9c9522a46f848798463d"} Jul 07 17:40:54.177988 ip-10-0-129-39 kubenswrapper[2575]: I0707 17:40:54.177949 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-9m48z" event={"ID":"59b94723-b7c1-4eb7-be5f-4ec4254b915d","Type":"ContainerStarted","Data":"263aa63c6426dd569715dfac6f7d035742b8253159b3ec501146beb1f08490af"} Jul 07 17:40:54.179806 ip-10-0-129-39 kubenswrapper[2575]: I0707 17:40:54.179775 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-btqqg" event={"ID":"a5765746-152c-4b60-b664-2dc9c75bd013","Type":"ContainerStarted","Data":"c81a1ad298e413483828d8ffe5fa3716089b62f7c1870df83d1932bd849d3864"} Jul 07 17:40:54.182438 ip-10-0-129-39 kubenswrapper[2575]: I0707 17:40:54.182412 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-fc7qv" event={"ID":"d4c44345-d6bf-4778-8911-09c7a1a0740a","Type":"ContainerStarted","Data":"cd7f102c47675d734bbcc073f44b72233bafb05fe4cbae049c50112edf22d412"} Jul 07 17:40:54.184324 ip-10-0-129-39 kubenswrapper[2575]: I0707 17:40:54.184300 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-7hrdq" event={"ID":"2d51e575-aadb-4cdd-a6cf-1ef119b49029","Type":"ContainerStarted","Data":"873f3778323aa21dfb14aa779a10808a868860e21e3769867b3e4cf780cfee63"} Jul 07 17:40:54.191305 ip-10-0-129-39 kubenswrapper[2575]: I0707 17:40:54.191259 2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/node-ca-l6pnn" podStartSLOduration=3.990384918 podStartE2EDuration="14.191242997s" totalImagesPullingTime="10.200858079s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-07 17:40:40 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 17:40:54.190555241 +0000 UTC m=+14.696361906" watchObservedRunningTime="2026-07-07 17:40:54.191242997 +0000 UTC m=+14.697049667" Jul 07 17:40:54.191420 ip-10-0-129-39 kubenswrapper[2575]: I0707 17:40:54.191350 2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-39.ec2.internal" podStartSLOduration=13.191344337 podStartE2EDuration="13.191344337s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-07 17:40:41 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 17:40:45.158370637 +0000 UTC m=+5.664177305" watchObservedRunningTime="2026-07-07 17:40:54.191344337 +0000 UTC m=+14.697151004" Jul 07 17:40:54.208972 ip-10-0-129-39 kubenswrapper[2575]: I0707 17:40:54.208935 2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-node-tuning-operator/tuned-7hrdq" podStartSLOduration=3.769828461 podStartE2EDuration="14.208919003s" totalImagesPullingTime="10.439090542s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-07 17:40:40 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 17:40:54.208448073 +0000 UTC m=+14.714254736" watchObservedRunningTime="2026-07-07 17:40:54.208919003 +0000 UTC m=+14.714725670" Jul 07 17:40:54.223922 ip-10-0-129-39 kubenswrapper[2575]: I0707 17:40:54.223885 2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/node-resolver-fc7qv" podStartSLOduration=4.013869041 podStartE2EDuration="14.223873817s" totalImagesPullingTime="10.210004776s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-07 17:40:40 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 17:40:54.22345124 +0000 UTC m=+14.729257918" watchObservedRunningTime="2026-07-07 17:40:54.223873817 +0000 UTC m=+14.729680482" Jul 07 17:40:54.262549 ip-10-0-129-39 kubenswrapper[2575]: I0707 17:40:54.262490 2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/konnectivity-agent-hr726" podStartSLOduration=4.072593749 podStartE2EDuration="14.262474646s" totalImagesPullingTime="10.189880897s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-07 17:40:40 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 17:40:54.262279921 +0000 UTC m=+14.768086586" watchObservedRunningTime="2026-07-07 17:40:54.262474646 +0000 UTC m=+14.768281347" Jul 07 17:40:55.187843 ip-10-0-129-39 kubenswrapper[2575]: I0707 17:40:55.187785 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-5vc77" event={"ID":"1233b6d4-571b-4319-b523-436909681456","Type":"ContainerStarted","Data":"5cc56a630f18b8a6cbdaf3e03a89388149716781d6c6cfdb194e95c521e89c10"} Jul 07 17:40:55.215632 ip-10-0-129-39 kubenswrapper[2575]: I0707 17:40:55.215575 2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-operator/iptables-alerter-5vc77" podStartSLOduration=4.798290025 podStartE2EDuration="15.215555628s" totalImagesPullingTime="10.417265603s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-07 17:40:40 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 17:40:55.215381764 +0000 UTC m=+15.721188432" watchObservedRunningTime="2026-07-07 17:40:55.215555628 +0000 UTC m=+15.721362307" Jul 07 17:40:55.394587 ip-10-0-129-39 kubenswrapper[2575]: I0707 17:40:55.394498 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hr726_4f2c6f8d-3b44-4f4f-8d6b-433f013a867f/konnectivity-agent/0.log" limit=1024 Jul 07 17:40:55.469945 ip-10-0-129-39 kubenswrapper[2575]: I0707 17:40:55.469913 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hr726_4f2c6f8d-3b44-4f4f-8d6b-433f013a867f/konnectivity-agent/0.log" limit=1024 Jul 07 17:40:56.071247 ip-10-0-129-39 kubenswrapper[2575]: I0707 17:40:56.071215 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-vzf4m" Jul 07 17:40:56.071432 ip-10-0-129-39 kubenswrapper[2575]: E0707 17:40:56.071324 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-vzf4m" podUID="5d33c1cb-fcda-490d-9694-b1d1eb16b8da" Jul 07 17:40:56.071432 ip-10-0-129-39 kubenswrapper[2575]: I0707 17:40:56.071402 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-76r79" Jul 07 17:40:56.071559 ip-10-0-129-39 kubenswrapper[2575]: E0707 17:40:56.071522 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-76r79" podUID="8344b558-69d0-43b3-9aac-0cea0d5379f3" Jul 07 17:40:56.071559 ip-10-0-129-39 kubenswrapper[2575]: I0707 17:40:56.071556 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-gz97j" Jul 07 17:40:56.071661 ip-10-0-129-39 kubenswrapper[2575]: E0707 17:40:56.071641 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-gz97j" podUID="d587896d-9156-4b75-aa70-27962a977051" Jul 07 17:40:57.655789 ip-10-0-129-39 kubenswrapper[2575]: I0707 17:40:57.655747 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/8344b558-69d0-43b3-9aac-0cea0d5379f3-original-pull-secret\") pod \"global-pull-secret-syncer-76r79\" (UID: \"8344b558-69d0-43b3-9aac-0cea0d5379f3\") " pod="kube-system/global-pull-secret-syncer-76r79" Jul 07 17:40:57.656429 ip-10-0-129-39 kubenswrapper[2575]: E0707 17:40:57.655865 2575 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 07 17:40:57.656429 ip-10-0-129-39 kubenswrapper[2575]: E0707 17:40:57.655941 2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8344b558-69d0-43b3-9aac-0cea0d5379f3-original-pull-secret podName:8344b558-69d0-43b3-9aac-0cea0d5379f3 nodeName:}" failed. No retries permitted until 2026-07-07 17:41:13.655922895 +0000 UTC m=+34.161729539 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/8344b558-69d0-43b3-9aac-0cea0d5379f3-original-pull-secret") pod "global-pull-secret-syncer-76r79" (UID: "8344b558-69d0-43b3-9aac-0cea0d5379f3") : object "kube-system"/"original-pull-secret" not registered Jul 07 17:40:57.756344 ip-10-0-129-39 kubenswrapper[2575]: I0707 17:40:57.756309 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d587896d-9156-4b75-aa70-27962a977051-metrics-certs\") pod \"network-metrics-daemon-gz97j\" (UID: \"d587896d-9156-4b75-aa70-27962a977051\") " pod="openshift-multus/network-metrics-daemon-gz97j" Jul 07 17:40:57.756489 ip-10-0-129-39 kubenswrapper[2575]: E0707 17:40:57.756473 2575 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 07 17:40:57.756553 ip-10-0-129-39 kubenswrapper[2575]: E0707 17:40:57.756546 2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d587896d-9156-4b75-aa70-27962a977051-metrics-certs podName:d587896d-9156-4b75-aa70-27962a977051 nodeName:}" failed. No retries permitted until 2026-07-07 17:41:13.756532336 +0000 UTC m=+34.262338980 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/d587896d-9156-4b75-aa70-27962a977051-metrics-certs") pod "network-metrics-daemon-gz97j" (UID: "d587896d-9156-4b75-aa70-27962a977051") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 07 17:40:57.857517 ip-10-0-129-39 kubenswrapper[2575]: I0707 17:40:57.857471 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-78bqt\" (UniqueName: \"kubernetes.io/projected/5d33c1cb-fcda-490d-9694-b1d1eb16b8da-kube-api-access-78bqt\") pod \"network-check-target-vzf4m\" (UID: \"5d33c1cb-fcda-490d-9694-b1d1eb16b8da\") " pod="openshift-network-diagnostics/network-check-target-vzf4m" Jul 07 17:40:57.857656 ip-10-0-129-39 kubenswrapper[2575]: E0707 17:40:57.857643 2575 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 07 17:40:57.857694 ip-10-0-129-39 kubenswrapper[2575]: E0707 17:40:57.857663 2575 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 07 17:40:57.857694 ip-10-0-129-39 kubenswrapper[2575]: E0707 17:40:57.857673 2575 projected.go:196] Error preparing data for projected volume kube-api-access-78bqt for pod openshift-network-diagnostics/network-check-target-vzf4m: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 07 17:40:57.857770 ip-10-0-129-39 kubenswrapper[2575]: E0707 17:40:57.857721 2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/5d33c1cb-fcda-490d-9694-b1d1eb16b8da-kube-api-access-78bqt podName:5d33c1cb-fcda-490d-9694-b1d1eb16b8da nodeName:}" failed. No retries permitted until 2026-07-07 17:41:13.857707793 +0000 UTC m=+34.363514437 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "kube-api-access-78bqt" (UniqueName: "kubernetes.io/projected/5d33c1cb-fcda-490d-9694-b1d1eb16b8da-kube-api-access-78bqt") pod "network-check-target-vzf4m" (UID: "5d33c1cb-fcda-490d-9694-b1d1eb16b8da") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 07 17:40:58.070922 ip-10-0-129-39 kubenswrapper[2575]: I0707 17:40:58.070898 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-gz97j" Jul 07 17:40:58.071113 ip-10-0-129-39 kubenswrapper[2575]: I0707 17:40:58.071000 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-76r79" Jul 07 17:40:58.071113 ip-10-0-129-39 kubenswrapper[2575]: E0707 17:40:58.071097 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-76r79" podUID="8344b558-69d0-43b3-9aac-0cea0d5379f3" Jul 07 17:40:58.071113 ip-10-0-129-39 kubenswrapper[2575]: E0707 17:40:58.071002 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-gz97j" podUID="d587896d-9156-4b75-aa70-27962a977051" Jul 07 17:40:58.071272 ip-10-0-129-39 kubenswrapper[2575]: I0707 17:40:58.071006 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-vzf4m" Jul 07 17:40:58.071272 ip-10-0-129-39 kubenswrapper[2575]: E0707 17:40:58.071176 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-vzf4m" podUID="5d33c1cb-fcda-490d-9694-b1d1eb16b8da" Jul 07 17:40:58.194952 ip-10-0-129-39 kubenswrapper[2575]: I0707 17:40:58.194886 2575 generic.go:356] "Generic (PLEG): container finished" podID="59b94723-b7c1-4eb7-be5f-4ec4254b915d" containerID="263aa63c6426dd569715dfac6f7d035742b8253159b3ec501146beb1f08490af" exitCode=0 Jul 07 17:40:58.194952 ip-10-0-129-39 kubenswrapper[2575]: I0707 17:40:58.194938 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-9m48z" event={"ID":"59b94723-b7c1-4eb7-be5f-4ec4254b915d","Type":"ContainerDied","Data":"263aa63c6426dd569715dfac6f7d035742b8253159b3ec501146beb1f08490af"} Jul 07 17:40:58.426695 ip-10-0-129-39 kubenswrapper[2575]: I0707 17:40:58.426657 2575 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kube-system/konnectivity-agent-hr726" Jul 07 17:40:58.427193 ip-10-0-129-39 kubenswrapper[2575]: I0707 17:40:58.427168 2575 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kube-system/konnectivity-agent-hr726" Jul 07 17:40:59.196889 ip-10-0-129-39 kubenswrapper[2575]: I0707 17:40:59.196862 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kube-system/konnectivity-agent-hr726" Jul 07 17:40:59.197497 ip-10-0-129-39 kubenswrapper[2575]: I0707 17:40:59.197484 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-hr726" Jul 07 17:40:59.271710 ip-10-0-129-39 kubenswrapper[2575]: I0707 17:40:59.270663 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hr726_4f2c6f8d-3b44-4f4f-8d6b-433f013a867f/konnectivity-agent/0.log" limit=1024 Jul 07 17:41:00.071653 ip-10-0-129-39 kubenswrapper[2575]: I0707 17:41:00.071624 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-gz97j" Jul 07 17:41:00.071810 ip-10-0-129-39 kubenswrapper[2575]: I0707 17:41:00.071738 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-vzf4m" Jul 07 17:41:00.071810 ip-10-0-129-39 kubenswrapper[2575]: E0707 17:41:00.071748 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-gz97j" podUID="d587896d-9156-4b75-aa70-27962a977051" Jul 07 17:41:00.071810 ip-10-0-129-39 kubenswrapper[2575]: I0707 17:41:00.071770 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-76r79" Jul 07 17:41:00.071976 ip-10-0-129-39 kubenswrapper[2575]: E0707 17:41:00.071856 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-vzf4m" podUID="5d33c1cb-fcda-490d-9694-b1d1eb16b8da" Jul 07 17:41:00.072015 ip-10-0-129-39 kubenswrapper[2575]: E0707 17:41:00.071970 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-76r79" podUID="8344b558-69d0-43b3-9aac-0cea0d5379f3" Jul 07 17:41:02.068537 ip-10-0-129-39 kubenswrapper[2575]: I0707 17:41:02.068386 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-76r79" Jul 07 17:41:02.068945 ip-10-0-129-39 kubenswrapper[2575]: I0707 17:41:02.068388 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-vzf4m" Jul 07 17:41:02.068945 ip-10-0-129-39 kubenswrapper[2575]: E0707 17:41:02.068634 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-76r79" podUID="8344b558-69d0-43b3-9aac-0cea0d5379f3" Jul 07 17:41:02.068945 ip-10-0-129-39 kubenswrapper[2575]: E0707 17:41:02.068736 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-vzf4m" podUID="5d33c1cb-fcda-490d-9694-b1d1eb16b8da" Jul 07 17:41:02.068945 ip-10-0-129-39 kubenswrapper[2575]: I0707 17:41:02.068443 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-gz97j" Jul 07 17:41:02.068945 ip-10-0-129-39 kubenswrapper[2575]: E0707 17:41:02.068848 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-gz97j" podUID="d587896d-9156-4b75-aa70-27962a977051" Jul 07 17:41:02.211066 ip-10-0-129-39 kubenswrapper[2575]: I0707 17:41:02.211010 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-qgbk5" event={"ID":"b8eb06e8-1aea-4b07-801e-fe9f5f86224f","Type":"ContainerStarted","Data":"20dc5be87f52d3d54510c9974a33605556a526f17175c1c3440246c3aadfce9a"} Jul 07 17:41:02.213117 ip-10-0-129-39 kubenswrapper[2575]: I0707 17:41:02.213090 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-rs8sg" event={"ID":"ad91fa94-7ae7-4245-90ee-78701b164dd2","Type":"ContainerStarted","Data":"c61ea8268cfcc2f31537bb50ac477c475370523768fdf4e1e17c6f7a83adc64a"} Jul 07 17:41:02.346478 ip-10-0-129-39 kubenswrapper[2575]: I0707 17:41:02.346329 2575 plugin_watcher.go:194] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock" Jul 07 17:41:02.362710 ip-10-0-129-39 kubenswrapper[2575]: I0707 17:41:02.362678 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hr726_4f2c6f8d-3b44-4f4f-8d6b-433f013a867f/konnectivity-agent/0.log" limit=1024 Jul 07 17:41:02.462482 ip-10-0-129-39 kubenswrapper[2575]: I0707 17:41:02.462449 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hr726_4f2c6f8d-3b44-4f4f-8d6b-433f013a867f/konnectivity-agent/0.log" limit=1024 Jul 07 17:41:02.984171 ip-10-0-129-39 kubenswrapper[2575]: I0707 17:41:02.984064 2575 reconciler.go:161] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-07-07T17:41:02.346474613Z","UUID":"4b85770a-8a1c-4577-8c5b-45449b1317e8","Handler":null,"Name":"","Endpoint":""} Jul 07 17:41:02.985750 ip-10-0-129-39 kubenswrapper[2575]: I0707 17:41:02.985728 2575 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0 Jul 07 17:41:02.985896 ip-10-0-129-39 kubenswrapper[2575]: I0707 17:41:02.985771 2575 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock Jul 07 17:41:03.218256 ip-10-0-129-39 kubenswrapper[2575]: I0707 17:41:03.218173 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-btqqg" event={"ID":"a5765746-152c-4b60-b664-2dc9c75bd013","Type":"ContainerStarted","Data":"e7058dd63b98b31b72530eb749c65b0a188dfb7bfe13018d12b618d76ed14d1b"} Jul 07 17:41:03.222443 ip-10-0-129-39 kubenswrapper[2575]: I0707 17:41:03.222175 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-qgbk5" event={"ID":"b8eb06e8-1aea-4b07-801e-fe9f5f86224f","Type":"ContainerStarted","Data":"7c373e0e47380ff7af66f5436b820ae4b224b54a5bcfa6203746e6616771fe95"} Jul 07 17:41:03.222443 ip-10-0-129-39 kubenswrapper[2575]: I0707 17:41:03.222263 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-qgbk5" event={"ID":"b8eb06e8-1aea-4b07-801e-fe9f5f86224f","Type":"ContainerStarted","Data":"39cb8c8f440ecae23de7787d86beae633256c354786847631a62df3d3f7a5645"} Jul 07 17:41:03.222443 ip-10-0-129-39 kubenswrapper[2575]: I0707 17:41:03.222306 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-qgbk5" event={"ID":"b8eb06e8-1aea-4b07-801e-fe9f5f86224f","Type":"ContainerStarted","Data":"ba4721e563cfc85c5e1047efa235de5fe1ab06f90ac916821d9ea7073415df7c"} Jul 07 17:41:03.222443 ip-10-0-129-39 kubenswrapper[2575]: I0707 17:41:03.222342 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-qgbk5" event={"ID":"b8eb06e8-1aea-4b07-801e-fe9f5f86224f","Type":"ContainerStarted","Data":"33b30d6ccd39aeb0ddd78bbc65f268e414cc30e7c3db642455fe8d08fb966acf"} Jul 07 17:41:03.222443 ip-10-0-129-39 kubenswrapper[2575]: I0707 17:41:03.222382 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-qgbk5" event={"ID":"b8eb06e8-1aea-4b07-801e-fe9f5f86224f","Type":"ContainerStarted","Data":"74a59b05f6719136b987701af2e1cddc0143bba942c1a9b2f25ce519436a242b"} Jul 07 17:41:04.068086 ip-10-0-129-39 kubenswrapper[2575]: I0707 17:41:04.068015 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-76r79" Jul 07 17:41:04.068086 ip-10-0-129-39 kubenswrapper[2575]: I0707 17:41:04.068046 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-vzf4m" Jul 07 17:41:04.068263 ip-10-0-129-39 kubenswrapper[2575]: I0707 17:41:04.068134 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-gz97j" Jul 07 17:41:04.068263 ip-10-0-129-39 kubenswrapper[2575]: E0707 17:41:04.068135 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-76r79" podUID="8344b558-69d0-43b3-9aac-0cea0d5379f3" Jul 07 17:41:04.068263 ip-10-0-129-39 kubenswrapper[2575]: E0707 17:41:04.068199 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-vzf4m" podUID="5d33c1cb-fcda-490d-9694-b1d1eb16b8da" Jul 07 17:41:04.068379 ip-10-0-129-39 kubenswrapper[2575]: E0707 17:41:04.068303 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-gz97j" podUID="d587896d-9156-4b75-aa70-27962a977051" Jul 07 17:41:04.231412 ip-10-0-129-39 kubenswrapper[2575]: I0707 17:41:04.231372 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-btqqg" event={"ID":"a5765746-152c-4b60-b664-2dc9c75bd013","Type":"ContainerStarted","Data":"8940f547e243405720b911647283fc5c4ec5a54da5a113a0b2b18fdba4e38edd"} Jul 07 17:41:04.251205 ip-10-0-129-39 kubenswrapper[2575]: I0707 17:41:04.251154 2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-rs8sg" podStartSLOduration=5.220108679 podStartE2EDuration="24.251138222s" totalImagesPullingTime="19.031029543s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-07 17:40:40 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 17:41:02.237962712 +0000 UTC m=+22.743769379" watchObservedRunningTime="2026-07-07 17:41:04.251138222 +0000 UTC m=+24.756944888" Jul 07 17:41:04.251613 ip-10-0-129-39 kubenswrapper[2575]: I0707 17:41:04.251572 2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-btqqg" podStartSLOduration=4.284423129 podStartE2EDuration="24.251561397s" totalImagesPullingTime="19.967138268s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-07 17:40:40 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 17:41:04.250868124 +0000 UTC m=+24.756674814" watchObservedRunningTime="2026-07-07 17:41:04.251561397 +0000 UTC m=+24.757368056" Jul 07 17:41:06.067919 ip-10-0-129-39 kubenswrapper[2575]: I0707 17:41:06.067893 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-vzf4m" Jul 07 17:41:06.067919 ip-10-0-129-39 kubenswrapper[2575]: I0707 17:41:06.067909 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-76r79" Jul 07 17:41:06.068334 ip-10-0-129-39 kubenswrapper[2575]: E0707 17:41:06.067987 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-vzf4m" podUID="5d33c1cb-fcda-490d-9694-b1d1eb16b8da" Jul 07 17:41:06.068334 ip-10-0-129-39 kubenswrapper[2575]: I0707 17:41:06.068077 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-gz97j" Jul 07 17:41:06.068334 ip-10-0-129-39 kubenswrapper[2575]: E0707 17:41:06.068073 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-76r79" podUID="8344b558-69d0-43b3-9aac-0cea0d5379f3" Jul 07 17:41:06.068334 ip-10-0-129-39 kubenswrapper[2575]: E0707 17:41:06.068173 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-gz97j" podUID="d587896d-9156-4b75-aa70-27962a977051" Jul 07 17:41:06.239439 ip-10-0-129-39 kubenswrapper[2575]: I0707 17:41:06.239378 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-qgbk5" event={"ID":"b8eb06e8-1aea-4b07-801e-fe9f5f86224f","Type":"ContainerStarted","Data":"9d1107f92efde19da9a728135e24ba8ae8b1b21f2aa04e022d20ea642b0c3cd2"} Jul 07 17:41:07.243560 ip-10-0-129-39 kubenswrapper[2575]: I0707 17:41:07.243527 2575 generic.go:356] "Generic (PLEG): container finished" podID="59b94723-b7c1-4eb7-be5f-4ec4254b915d" containerID="dc7ac0ffd1cc15376a77a6d1e04fffa7892cc132067c6713fdf1f5a27362d045" exitCode=0 Jul 07 17:41:07.244073 ip-10-0-129-39 kubenswrapper[2575]: I0707 17:41:07.243577 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-9m48z" event={"ID":"59b94723-b7c1-4eb7-be5f-4ec4254b915d","Type":"ContainerDied","Data":"dc7ac0ffd1cc15376a77a6d1e04fffa7892cc132067c6713fdf1f5a27362d045"} Jul 07 17:41:08.068341 ip-10-0-129-39 kubenswrapper[2575]: I0707 17:41:08.068316 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-76r79" Jul 07 17:41:08.068445 ip-10-0-129-39 kubenswrapper[2575]: E0707 17:41:08.068412 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-76r79" podUID="8344b558-69d0-43b3-9aac-0cea0d5379f3" Jul 07 17:41:08.068492 ip-10-0-129-39 kubenswrapper[2575]: I0707 17:41:08.068444 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-gz97j" Jul 07 17:41:08.068551 ip-10-0-129-39 kubenswrapper[2575]: I0707 17:41:08.068318 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-vzf4m" Jul 07 17:41:08.068601 ip-10-0-129-39 kubenswrapper[2575]: E0707 17:41:08.068559 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-gz97j" podUID="d587896d-9156-4b75-aa70-27962a977051" Jul 07 17:41:08.068663 ip-10-0-129-39 kubenswrapper[2575]: E0707 17:41:08.068640 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-vzf4m" podUID="5d33c1cb-fcda-490d-9694-b1d1eb16b8da" Jul 07 17:41:08.249519 ip-10-0-129-39 kubenswrapper[2575]: I0707 17:41:08.249480 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-qgbk5" event={"ID":"b8eb06e8-1aea-4b07-801e-fe9f5f86224f","Type":"ContainerStarted","Data":"87a2e5a306200442477e2d940851607f869d0e14aad362c97f22934564c1c51f"} Jul 07 17:41:08.249984 ip-10-0-129-39 kubenswrapper[2575]: I0707 17:41:08.249744 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-qgbk5" Jul 07 17:41:08.249984 ip-10-0-129-39 kubenswrapper[2575]: I0707 17:41:08.249800 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-qgbk5" Jul 07 17:41:08.264202 ip-10-0-129-39 kubenswrapper[2575]: I0707 17:41:08.264186 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-qgbk5" Jul 07 17:41:08.281838 ip-10-0-129-39 kubenswrapper[2575]: I0707 17:41:08.281699 2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ovn-kubernetes/ovnkube-node-qgbk5" podStartSLOduration=9.290822282 podStartE2EDuration="28.281687095s" totalImagesPullingTime="18.990864813s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-07 17:40:40 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 17:41:08.281280942 +0000 UTC m=+28.787087625" watchObservedRunningTime="2026-07-07 17:41:08.281687095 +0000 UTC m=+28.787493761" Jul 07 17:41:09.253593 ip-10-0-129-39 kubenswrapper[2575]: I0707 17:41:09.253559 2575 generic.go:356] "Generic (PLEG): container finished" podID="59b94723-b7c1-4eb7-be5f-4ec4254b915d" containerID="5912b79941c905292e6152e7f4b39e311eaae45f50a431d0be138e38a8c7af4e" exitCode=0 Jul 07 17:41:09.253929 ip-10-0-129-39 kubenswrapper[2575]: I0707 17:41:09.253628 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-9m48z" event={"ID":"59b94723-b7c1-4eb7-be5f-4ec4254b915d","Type":"ContainerDied","Data":"5912b79941c905292e6152e7f4b39e311eaae45f50a431d0be138e38a8c7af4e"} Jul 07 17:41:09.254108 ip-10-0-129-39 kubenswrapper[2575]: I0707 17:41:09.254078 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-qgbk5" Jul 07 17:41:09.268763 ip-10-0-129-39 kubenswrapper[2575]: I0707 17:41:09.268744 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-qgbk5" Jul 07 17:41:10.068342 ip-10-0-129-39 kubenswrapper[2575]: I0707 17:41:10.068317 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-vzf4m" Jul 07 17:41:10.068521 ip-10-0-129-39 kubenswrapper[2575]: I0707 17:41:10.068488 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-gz97j" Jul 07 17:41:10.068602 ip-10-0-129-39 kubenswrapper[2575]: E0707 17:41:10.068520 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-vzf4m" podUID="5d33c1cb-fcda-490d-9694-b1d1eb16b8da" Jul 07 17:41:10.068656 ip-10-0-129-39 kubenswrapper[2575]: E0707 17:41:10.068620 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-gz97j" podUID="d587896d-9156-4b75-aa70-27962a977051" Jul 07 17:41:10.068715 ip-10-0-129-39 kubenswrapper[2575]: I0707 17:41:10.068654 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-76r79" Jul 07 17:41:10.068762 ip-10-0-129-39 kubenswrapper[2575]: E0707 17:41:10.068737 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-76r79" podUID="8344b558-69d0-43b3-9aac-0cea0d5379f3" Jul 07 17:41:10.136466 ip-10-0-129-39 kubenswrapper[2575]: I0707 17:41:10.136439 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-76r79"] Jul 07 17:41:10.139400 ip-10-0-129-39 kubenswrapper[2575]: I0707 17:41:10.139371 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-vzf4m"] Jul 07 17:41:10.140056 ip-10-0-129-39 kubenswrapper[2575]: I0707 17:41:10.140032 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-gz97j"] Jul 07 17:41:10.256670 ip-10-0-129-39 kubenswrapper[2575]: I0707 17:41:10.256646 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-76r79" Jul 07 17:41:10.257016 ip-10-0-129-39 kubenswrapper[2575]: E0707 17:41:10.256773 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-76r79" podUID="8344b558-69d0-43b3-9aac-0cea0d5379f3" Jul 07 17:41:10.258091 ip-10-0-129-39 kubenswrapper[2575]: I0707 17:41:10.258069 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-gz97j" Jul 07 17:41:10.258197 ip-10-0-129-39 kubenswrapper[2575]: I0707 17:41:10.258179 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-vzf4m" Jul 07 17:41:10.258240 ip-10-0-129-39 kubenswrapper[2575]: E0707 17:41:10.258190 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-gz97j" podUID="d587896d-9156-4b75-aa70-27962a977051" Jul 07 17:41:10.258314 ip-10-0-129-39 kubenswrapper[2575]: E0707 17:41:10.258292 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-vzf4m" podUID="5d33c1cb-fcda-490d-9694-b1d1eb16b8da" Jul 07 17:41:11.261292 ip-10-0-129-39 kubenswrapper[2575]: I0707 17:41:11.261086 2575 generic.go:356] "Generic (PLEG): container finished" podID="59b94723-b7c1-4eb7-be5f-4ec4254b915d" containerID="bc210e689f1c2312a85817b646d6ee6eb41775a3fccbd2ba66558d04843a8313" exitCode=0 Jul 07 17:41:11.261740 ip-10-0-129-39 kubenswrapper[2575]: I0707 17:41:11.261151 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-9m48z" event={"ID":"59b94723-b7c1-4eb7-be5f-4ec4254b915d","Type":"ContainerDied","Data":"bc210e689f1c2312a85817b646d6ee6eb41775a3fccbd2ba66558d04843a8313"} Jul 07 17:41:12.067887 ip-10-0-129-39 kubenswrapper[2575]: I0707 17:41:12.067853 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-gz97j" Jul 07 17:41:12.068024 ip-10-0-129-39 kubenswrapper[2575]: I0707 17:41:12.067862 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-76r79" Jul 07 17:41:12.068024 ip-10-0-129-39 kubenswrapper[2575]: E0707 17:41:12.067986 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-gz97j" podUID="d587896d-9156-4b75-aa70-27962a977051" Jul 07 17:41:12.068125 ip-10-0-129-39 kubenswrapper[2575]: I0707 17:41:12.067862 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-vzf4m" Jul 07 17:41:12.068125 ip-10-0-129-39 kubenswrapper[2575]: E0707 17:41:12.068033 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-76r79" podUID="8344b558-69d0-43b3-9aac-0cea0d5379f3" Jul 07 17:41:12.068125 ip-10-0-129-39 kubenswrapper[2575]: E0707 17:41:12.068088 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-vzf4m" podUID="5d33c1cb-fcda-490d-9694-b1d1eb16b8da" Jul 07 17:41:13.678202 ip-10-0-129-39 kubenswrapper[2575]: I0707 17:41:13.678173 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/8344b558-69d0-43b3-9aac-0cea0d5379f3-original-pull-secret\") pod \"global-pull-secret-syncer-76r79\" (UID: \"8344b558-69d0-43b3-9aac-0cea0d5379f3\") " pod="kube-system/global-pull-secret-syncer-76r79" Jul 07 17:41:13.678654 ip-10-0-129-39 kubenswrapper[2575]: E0707 17:41:13.678318 2575 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 07 17:41:13.678654 ip-10-0-129-39 kubenswrapper[2575]: E0707 17:41:13.678376 2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8344b558-69d0-43b3-9aac-0cea0d5379f3-original-pull-secret podName:8344b558-69d0-43b3-9aac-0cea0d5379f3 nodeName:}" failed. No retries permitted until 2026-07-07 17:41:45.678362767 +0000 UTC m=+66.184169411 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/8344b558-69d0-43b3-9aac-0cea0d5379f3-original-pull-secret") pod "global-pull-secret-syncer-76r79" (UID: "8344b558-69d0-43b3-9aac-0cea0d5379f3") : object "kube-system"/"original-pull-secret" not registered Jul 07 17:41:13.683188 ip-10-0-129-39 kubenswrapper[2575]: I0707 17:41:13.683167 2575 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-129-39.ec2.internal" event="NodeReady" Jul 07 17:41:13.683293 ip-10-0-129-39 kubenswrapper[2575]: I0707 17:41:13.683271 2575 kubelet_node_status.go:489] "Fast updating node status as it just became ready" Jul 07 17:41:13.738411 ip-10-0-129-39 kubenswrapper[2575]: I0707 17:41:13.738389 2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-dns/dns-default-b2mnn"] Jul 07 17:41:13.751728 ip-10-0-129-39 kubenswrapper[2575]: I0707 17:41:13.751689 2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress-canary/ingress-canary-x7kdh"] Jul 07 17:41:13.751842 ip-10-0-129-39 kubenswrapper[2575]: I0707 17:41:13.751799 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-b2mnn" Jul 07 17:41:13.755058 ip-10-0-129-39 kubenswrapper[2575]: I0707 17:41:13.755036 2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\"" Jul 07 17:41:13.755623 ip-10-0-129-39 kubenswrapper[2575]: I0707 17:41:13.755582 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-gg8rv\"" Jul 07 17:41:13.755734 ip-10-0-129-39 kubenswrapper[2575]: I0707 17:41:13.755716 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\"" Jul 07 17:41:13.765781 ip-10-0-129-39 kubenswrapper[2575]: I0707 17:41:13.765729 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-x7kdh" Jul 07 17:41:13.769856 ip-10-0-129-39 kubenswrapper[2575]: I0707 17:41:13.769710 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"default-dockercfg-l7zp2\"" Jul 07 17:41:13.769856 ip-10-0-129-39 kubenswrapper[2575]: I0707 17:41:13.769812 2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\"" Jul 07 17:41:13.770001 ip-10-0-129-39 kubenswrapper[2575]: I0707 17:41:13.769721 2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\"" Jul 07 17:41:13.770001 ip-10-0-129-39 kubenswrapper[2575]: I0707 17:41:13.769774 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\"" Jul 07 17:41:13.770737 ip-10-0-129-39 kubenswrapper[2575]: I0707 17:41:13.770679 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-b2mnn"] Jul 07 17:41:13.771795 ip-10-0-129-39 kubenswrapper[2575]: I0707 17:41:13.771774 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-x7kdh"] Jul 07 17:41:13.779383 ip-10-0-129-39 kubenswrapper[2575]: I0707 17:41:13.779362 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d587896d-9156-4b75-aa70-27962a977051-metrics-certs\") pod \"network-metrics-daemon-gz97j\" (UID: \"d587896d-9156-4b75-aa70-27962a977051\") " pod="openshift-multus/network-metrics-daemon-gz97j" Jul 07 17:41:13.779547 ip-10-0-129-39 kubenswrapper[2575]: E0707 17:41:13.779531 2575 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 07 17:41:13.779598 ip-10-0-129-39 kubenswrapper[2575]: E0707 17:41:13.779589 2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d587896d-9156-4b75-aa70-27962a977051-metrics-certs podName:d587896d-9156-4b75-aa70-27962a977051 nodeName:}" failed. No retries permitted until 2026-07-07 17:41:45.77957435 +0000 UTC m=+66.285381013 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/d587896d-9156-4b75-aa70-27962a977051-metrics-certs") pod "network-metrics-daemon-gz97j" (UID: "d587896d-9156-4b75-aa70-27962a977051") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 07 17:41:13.879928 ip-10-0-129-39 kubenswrapper[2575]: I0707 17:41:13.879896 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-78bqt\" (UniqueName: \"kubernetes.io/projected/5d33c1cb-fcda-490d-9694-b1d1eb16b8da-kube-api-access-78bqt\") pod \"network-check-target-vzf4m\" (UID: \"5d33c1cb-fcda-490d-9694-b1d1eb16b8da\") " pod="openshift-network-diagnostics/network-check-target-vzf4m" Jul 07 17:41:13.880078 ip-10-0-129-39 kubenswrapper[2575]: I0707 17:41:13.879940 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vmvwr\" (UniqueName: \"kubernetes.io/projected/b2656ecc-1db3-4850-b2f6-58b0a14eb683-kube-api-access-vmvwr\") pod \"dns-default-b2mnn\" (UID: \"b2656ecc-1db3-4850-b2f6-58b0a14eb683\") " pod="openshift-dns/dns-default-b2mnn" Jul 07 17:41:13.880078 ip-10-0-129-39 kubenswrapper[2575]: I0707 17:41:13.879972 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b2656ecc-1db3-4850-b2f6-58b0a14eb683-tmp-dir\") pod \"dns-default-b2mnn\" (UID: \"b2656ecc-1db3-4850-b2f6-58b0a14eb683\") " pod="openshift-dns/dns-default-b2mnn" Jul 07 17:41:13.880078 ip-10-0-129-39 kubenswrapper[2575]: I0707 17:41:13.880061 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/b2656ecc-1db3-4850-b2f6-58b0a14eb683-metrics-tls\") pod \"dns-default-b2mnn\" (UID: \"b2656ecc-1db3-4850-b2f6-58b0a14eb683\") " pod="openshift-dns/dns-default-b2mnn" Jul 07 17:41:13.880219 ip-10-0-129-39 kubenswrapper[2575]: E0707 17:41:13.880078 2575 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 07 17:41:13.880219 ip-10-0-129-39 kubenswrapper[2575]: E0707 17:41:13.880094 2575 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 07 17:41:13.880219 ip-10-0-129-39 kubenswrapper[2575]: E0707 17:41:13.880103 2575 projected.go:196] Error preparing data for projected volume kube-api-access-78bqt for pod openshift-network-diagnostics/network-check-target-vzf4m: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 07 17:41:13.880219 ip-10-0-129-39 kubenswrapper[2575]: I0707 17:41:13.880101 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/1935a145-8811-413c-ad46-ae93176ffd7c-cert\") pod \"ingress-canary-x7kdh\" (UID: \"1935a145-8811-413c-ad46-ae93176ffd7c\") " pod="openshift-ingress-canary/ingress-canary-x7kdh" Jul 07 17:41:13.880219 ip-10-0-129-39 kubenswrapper[2575]: I0707 17:41:13.880138 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/b2656ecc-1db3-4850-b2f6-58b0a14eb683-config-volume\") pod \"dns-default-b2mnn\" (UID: \"b2656ecc-1db3-4850-b2f6-58b0a14eb683\") " pod="openshift-dns/dns-default-b2mnn" Jul 07 17:41:13.880219 ip-10-0-129-39 kubenswrapper[2575]: E0707 17:41:13.880150 2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/5d33c1cb-fcda-490d-9694-b1d1eb16b8da-kube-api-access-78bqt podName:5d33c1cb-fcda-490d-9694-b1d1eb16b8da nodeName:}" failed. No retries permitted until 2026-07-07 17:41:45.880133099 +0000 UTC m=+66.385939754 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "kube-api-access-78bqt" (UniqueName: "kubernetes.io/projected/5d33c1cb-fcda-490d-9694-b1d1eb16b8da-kube-api-access-78bqt") pod "network-check-target-vzf4m" (UID: "5d33c1cb-fcda-490d-9694-b1d1eb16b8da") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 07 17:41:13.880219 ip-10-0-129-39 kubenswrapper[2575]: I0707 17:41:13.880190 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-k24cj\" (UniqueName: \"kubernetes.io/projected/1935a145-8811-413c-ad46-ae93176ffd7c-kube-api-access-k24cj\") pod \"ingress-canary-x7kdh\" (UID: \"1935a145-8811-413c-ad46-ae93176ffd7c\") " pod="openshift-ingress-canary/ingress-canary-x7kdh" Jul 07 17:41:13.980956 ip-10-0-129-39 kubenswrapper[2575]: I0707 17:41:13.980928 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/b2656ecc-1db3-4850-b2f6-58b0a14eb683-config-volume\") pod \"dns-default-b2mnn\" (UID: \"b2656ecc-1db3-4850-b2f6-58b0a14eb683\") " pod="openshift-dns/dns-default-b2mnn" Jul 07 17:41:13.980956 ip-10-0-129-39 kubenswrapper[2575]: I0707 17:41:13.980958 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k24cj\" (UniqueName: \"kubernetes.io/projected/1935a145-8811-413c-ad46-ae93176ffd7c-kube-api-access-k24cj\") pod \"ingress-canary-x7kdh\" (UID: \"1935a145-8811-413c-ad46-ae93176ffd7c\") " pod="openshift-ingress-canary/ingress-canary-x7kdh" Jul 07 17:41:13.981132 ip-10-0-129-39 kubenswrapper[2575]: I0707 17:41:13.981102 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vmvwr\" (UniqueName: \"kubernetes.io/projected/b2656ecc-1db3-4850-b2f6-58b0a14eb683-kube-api-access-vmvwr\") pod \"dns-default-b2mnn\" (UID: \"b2656ecc-1db3-4850-b2f6-58b0a14eb683\") " pod="openshift-dns/dns-default-b2mnn" Jul 07 17:41:13.981167 ip-10-0-129-39 kubenswrapper[2575]: I0707 17:41:13.981142 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b2656ecc-1db3-4850-b2f6-58b0a14eb683-tmp-dir\") pod \"dns-default-b2mnn\" (UID: \"b2656ecc-1db3-4850-b2f6-58b0a14eb683\") " pod="openshift-dns/dns-default-b2mnn" Jul 07 17:41:13.981208 ip-10-0-129-39 kubenswrapper[2575]: I0707 17:41:13.981184 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/b2656ecc-1db3-4850-b2f6-58b0a14eb683-metrics-tls\") pod \"dns-default-b2mnn\" (UID: \"b2656ecc-1db3-4850-b2f6-58b0a14eb683\") " pod="openshift-dns/dns-default-b2mnn" Jul 07 17:41:13.981244 ip-10-0-129-39 kubenswrapper[2575]: I0707 17:41:13.981220 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/1935a145-8811-413c-ad46-ae93176ffd7c-cert\") pod \"ingress-canary-x7kdh\" (UID: \"1935a145-8811-413c-ad46-ae93176ffd7c\") " pod="openshift-ingress-canary/ingress-canary-x7kdh" Jul 07 17:41:13.981320 ip-10-0-129-39 kubenswrapper[2575]: E0707 17:41:13.981301 2575 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jul 07 17:41:13.981378 ip-10-0-129-39 kubenswrapper[2575]: E0707 17:41:13.981363 2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b2656ecc-1db3-4850-b2f6-58b0a14eb683-metrics-tls podName:b2656ecc-1db3-4850-b2f6-58b0a14eb683 nodeName:}" failed. No retries permitted until 2026-07-07 17:41:14.481346747 +0000 UTC m=+34.987153392 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/b2656ecc-1db3-4850-b2f6-58b0a14eb683-metrics-tls") pod "dns-default-b2mnn" (UID: "b2656ecc-1db3-4850-b2f6-58b0a14eb683") : secret "dns-default-metrics-tls" not found Jul 07 17:41:13.981494 ip-10-0-129-39 kubenswrapper[2575]: I0707 17:41:13.981475 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b2656ecc-1db3-4850-b2f6-58b0a14eb683-tmp-dir\") pod \"dns-default-b2mnn\" (UID: \"b2656ecc-1db3-4850-b2f6-58b0a14eb683\") " pod="openshift-dns/dns-default-b2mnn" Jul 07 17:41:13.981645 ip-10-0-129-39 kubenswrapper[2575]: I0707 17:41:13.981626 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/b2656ecc-1db3-4850-b2f6-58b0a14eb683-config-volume\") pod \"dns-default-b2mnn\" (UID: \"b2656ecc-1db3-4850-b2f6-58b0a14eb683\") " pod="openshift-dns/dns-default-b2mnn" Jul 07 17:41:13.985305 ip-10-0-129-39 kubenswrapper[2575]: I0707 17:41:13.985287 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/1935a145-8811-413c-ad46-ae93176ffd7c-cert\") pod \"ingress-canary-x7kdh\" (UID: \"1935a145-8811-413c-ad46-ae93176ffd7c\") " pod="openshift-ingress-canary/ingress-canary-x7kdh" Jul 07 17:41:13.990959 ip-10-0-129-39 kubenswrapper[2575]: I0707 17:41:13.990930 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vmvwr\" (UniqueName: \"kubernetes.io/projected/b2656ecc-1db3-4850-b2f6-58b0a14eb683-kube-api-access-vmvwr\") pod \"dns-default-b2mnn\" (UID: \"b2656ecc-1db3-4850-b2f6-58b0a14eb683\") " pod="openshift-dns/dns-default-b2mnn" Jul 07 17:41:13.991042 ip-10-0-129-39 kubenswrapper[2575]: I0707 17:41:13.990986 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-k24cj\" (UniqueName: \"kubernetes.io/projected/1935a145-8811-413c-ad46-ae93176ffd7c-kube-api-access-k24cj\") pod \"ingress-canary-x7kdh\" (UID: \"1935a145-8811-413c-ad46-ae93176ffd7c\") " pod="openshift-ingress-canary/ingress-canary-x7kdh" Jul 07 17:41:14.068391 ip-10-0-129-39 kubenswrapper[2575]: I0707 17:41:14.068331 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-76r79" Jul 07 17:41:14.068494 ip-10-0-129-39 kubenswrapper[2575]: I0707 17:41:14.068331 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-vzf4m" Jul 07 17:41:14.068494 ip-10-0-129-39 kubenswrapper[2575]: I0707 17:41:14.068473 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-gz97j" Jul 07 17:41:14.070931 ip-10-0-129-39 kubenswrapper[2575]: I0707 17:41:14.070907 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\"" Jul 07 17:41:14.071115 ip-10-0-129-39 kubenswrapper[2575]: I0707 17:41:14.071097 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-92crh\"" Jul 07 17:41:14.071179 ip-10-0-129-39 kubenswrapper[2575]: I0707 17:41:14.071168 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\"" Jul 07 17:41:14.071232 ip-10-0-129-39 kubenswrapper[2575]: I0707 17:41:14.071212 2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\"" Jul 07 17:41:14.071303 ip-10-0-129-39 kubenswrapper[2575]: I0707 17:41:14.071219 2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\"" Jul 07 17:41:14.071574 ip-10-0-129-39 kubenswrapper[2575]: I0707 17:41:14.071519 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-njwcl\"" Jul 07 17:41:14.075357 ip-10-0-129-39 kubenswrapper[2575]: I0707 17:41:14.075339 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-x7kdh" Jul 07 17:41:14.233917 ip-10-0-129-39 kubenswrapper[2575]: I0707 17:41:14.233712 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-x7kdh"] Jul 07 17:41:14.238206 ip-10-0-129-39 kubenswrapper[2575]: W0707 17:41:14.238181 2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1935a145_8811_413c_ad46_ae93176ffd7c.slice/crio-b349fd026859debbb61f4eb67e94f87b05c9fbffc83c6cfa08545d8379ae9303 WatchSource:0}: Error finding container b349fd026859debbb61f4eb67e94f87b05c9fbffc83c6cfa08545d8379ae9303: Status 404 returned error can't find the container with id b349fd026859debbb61f4eb67e94f87b05c9fbffc83c6cfa08545d8379ae9303 Jul 07 17:41:14.268726 ip-10-0-129-39 kubenswrapper[2575]: I0707 17:41:14.268680 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-x7kdh" event={"ID":"1935a145-8811-413c-ad46-ae93176ffd7c","Type":"ContainerStarted","Data":"b349fd026859debbb61f4eb67e94f87b05c9fbffc83c6cfa08545d8379ae9303"} Jul 07 17:41:14.485388 ip-10-0-129-39 kubenswrapper[2575]: I0707 17:41:14.485353 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/b2656ecc-1db3-4850-b2f6-58b0a14eb683-metrics-tls\") pod \"dns-default-b2mnn\" (UID: \"b2656ecc-1db3-4850-b2f6-58b0a14eb683\") " pod="openshift-dns/dns-default-b2mnn" Jul 07 17:41:14.485571 ip-10-0-129-39 kubenswrapper[2575]: E0707 17:41:14.485521 2575 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jul 07 17:41:14.485629 ip-10-0-129-39 kubenswrapper[2575]: E0707 17:41:14.485593 2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b2656ecc-1db3-4850-b2f6-58b0a14eb683-metrics-tls podName:b2656ecc-1db3-4850-b2f6-58b0a14eb683 nodeName:}" failed. No retries permitted until 2026-07-07 17:41:15.485576648 +0000 UTC m=+35.991383293 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/b2656ecc-1db3-4850-b2f6-58b0a14eb683-metrics-tls") pod "dns-default-b2mnn" (UID: "b2656ecc-1db3-4850-b2f6-58b0a14eb683") : secret "dns-default-metrics-tls" not found Jul 07 17:41:15.492319 ip-10-0-129-39 kubenswrapper[2575]: I0707 17:41:15.492242 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/b2656ecc-1db3-4850-b2f6-58b0a14eb683-metrics-tls\") pod \"dns-default-b2mnn\" (UID: \"b2656ecc-1db3-4850-b2f6-58b0a14eb683\") " pod="openshift-dns/dns-default-b2mnn" Jul 07 17:41:15.492980 ip-10-0-129-39 kubenswrapper[2575]: E0707 17:41:15.492414 2575 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jul 07 17:41:15.492980 ip-10-0-129-39 kubenswrapper[2575]: E0707 17:41:15.492491 2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b2656ecc-1db3-4850-b2f6-58b0a14eb683-metrics-tls podName:b2656ecc-1db3-4850-b2f6-58b0a14eb683 nodeName:}" failed. No retries permitted until 2026-07-07 17:41:17.492470707 +0000 UTC m=+37.998277357 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/b2656ecc-1db3-4850-b2f6-58b0a14eb683-metrics-tls") pod "dns-default-b2mnn" (UID: "b2656ecc-1db3-4850-b2f6-58b0a14eb683") : secret "dns-default-metrics-tls" not found Jul 07 17:41:17.507935 ip-10-0-129-39 kubenswrapper[2575]: I0707 17:41:17.507901 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/b2656ecc-1db3-4850-b2f6-58b0a14eb683-metrics-tls\") pod \"dns-default-b2mnn\" (UID: \"b2656ecc-1db3-4850-b2f6-58b0a14eb683\") " pod="openshift-dns/dns-default-b2mnn" Jul 07 17:41:17.508273 ip-10-0-129-39 kubenswrapper[2575]: E0707 17:41:17.508014 2575 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jul 07 17:41:17.508273 ip-10-0-129-39 kubenswrapper[2575]: E0707 17:41:17.508063 2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b2656ecc-1db3-4850-b2f6-58b0a14eb683-metrics-tls podName:b2656ecc-1db3-4850-b2f6-58b0a14eb683 nodeName:}" failed. No retries permitted until 2026-07-07 17:41:21.508051042 +0000 UTC m=+42.013857685 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/b2656ecc-1db3-4850-b2f6-58b0a14eb683-metrics-tls") pod "dns-default-b2mnn" (UID: "b2656ecc-1db3-4850-b2f6-58b0a14eb683") : secret "dns-default-metrics-tls" not found Jul 07 17:41:18.278758 ip-10-0-129-39 kubenswrapper[2575]: I0707 17:41:18.278723 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-x7kdh" event={"ID":"1935a145-8811-413c-ad46-ae93176ffd7c","Type":"ContainerStarted","Data":"a1505c9dba9a52e976c069f91c666115fdcb7426200bdb16f27704d4e96b66c6"} Jul 07 17:41:18.281550 ip-10-0-129-39 kubenswrapper[2575]: I0707 17:41:18.281524 2575 generic.go:356] "Generic (PLEG): container finished" podID="59b94723-b7c1-4eb7-be5f-4ec4254b915d" containerID="ae0510e2fda037bb34d1eae484d5972b9eb4d0b2849787b4f987b46a291ccfd2" exitCode=0 Jul 07 17:41:18.281645 ip-10-0-129-39 kubenswrapper[2575]: I0707 17:41:18.281574 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-9m48z" event={"ID":"59b94723-b7c1-4eb7-be5f-4ec4254b915d","Type":"ContainerDied","Data":"ae0510e2fda037bb34d1eae484d5972b9eb4d0b2849787b4f987b46a291ccfd2"} Jul 07 17:41:18.294977 ip-10-0-129-39 kubenswrapper[2575]: I0707 17:41:18.294823 2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress-canary/ingress-canary-x7kdh" podStartSLOduration=1.936589613 podStartE2EDuration="5.294811792s" totalImagesPullingTime="3.358222179s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-07 17:41:13 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 17:41:18.294781296 +0000 UTC m=+38.800587965" watchObservedRunningTime="2026-07-07 17:41:18.294811792 +0000 UTC m=+38.800618457" Jul 07 17:41:19.286811 ip-10-0-129-39 kubenswrapper[2575]: I0707 17:41:19.286783 2575 generic.go:356] "Generic (PLEG): container finished" podID="59b94723-b7c1-4eb7-be5f-4ec4254b915d" containerID="399fd1ae8497f28527e6c7467a2ac4ebf6ac7f2045b013102df92e75d5471bd7" exitCode=0 Jul 07 17:41:19.287234 ip-10-0-129-39 kubenswrapper[2575]: I0707 17:41:19.286850 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-9m48z" event={"ID":"59b94723-b7c1-4eb7-be5f-4ec4254b915d","Type":"ContainerDied","Data":"399fd1ae8497f28527e6c7467a2ac4ebf6ac7f2045b013102df92e75d5471bd7"} Jul 07 17:41:20.294781 ip-10-0-129-39 kubenswrapper[2575]: I0707 17:41:20.294607 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-9m48z" event={"ID":"59b94723-b7c1-4eb7-be5f-4ec4254b915d","Type":"ContainerStarted","Data":"7bdf57fca7a3a380d597c51e966a8abaefaad7248e9fd79904435fe0d362521c"} Jul 07 17:41:20.318658 ip-10-0-129-39 kubenswrapper[2575]: I0707 17:41:20.318614 2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-additional-cni-plugins-9m48z" podStartSLOduration=13.644039666 podStartE2EDuration="40.318600019s" totalImagesPullingTime="26.674560353s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-07 17:40:40 +0000 UTC" imagePullSessionsCount=5 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 17:41:20.317457889 +0000 UTC m=+40.823264555" watchObservedRunningTime="2026-07-07 17:41:20.318600019 +0000 UTC m=+40.824406682" Jul 07 17:41:21.536353 ip-10-0-129-39 kubenswrapper[2575]: I0707 17:41:21.536316 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/b2656ecc-1db3-4850-b2f6-58b0a14eb683-metrics-tls\") pod \"dns-default-b2mnn\" (UID: \"b2656ecc-1db3-4850-b2f6-58b0a14eb683\") " pod="openshift-dns/dns-default-b2mnn" Jul 07 17:41:21.536858 ip-10-0-129-39 kubenswrapper[2575]: E0707 17:41:21.536451 2575 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jul 07 17:41:21.536858 ip-10-0-129-39 kubenswrapper[2575]: E0707 17:41:21.536528 2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b2656ecc-1db3-4850-b2f6-58b0a14eb683-metrics-tls podName:b2656ecc-1db3-4850-b2f6-58b0a14eb683 nodeName:}" failed. No retries permitted until 2026-07-07 17:41:29.536497815 +0000 UTC m=+50.042304459 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/b2656ecc-1db3-4850-b2f6-58b0a14eb683-metrics-tls") pod "dns-default-b2mnn" (UID: "b2656ecc-1db3-4850-b2f6-58b0a14eb683") : secret "dns-default-metrics-tls" not found Jul 07 17:41:24.862335 ip-10-0-129-39 kubenswrapper[2575]: I0707 17:41:24.862299 2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/cluster-monitoring-operator-64786d9cbb-4fqsh"] Jul 07 17:41:24.889966 ip-10-0-129-39 kubenswrapper[2575]: I0707 17:41:24.889938 2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-cluster-storage-operator/volume-data-source-validator-5d456fdbcd-4knqs"] Jul 07 17:41:24.890091 ip-10-0-129-39 kubenswrapper[2575]: I0707 17:41:24.890051 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/cluster-monitoring-operator-64786d9cbb-4fqsh" Jul 07 17:41:24.892310 ip-10-0-129-39 kubenswrapper[2575]: I0707 17:41:24.892284 2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\"" Jul 07 17:41:24.892535 ip-10-0-129-39 kubenswrapper[2575]: I0707 17:41:24.892517 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"cluster-monitoring-operator-tls\"" Jul 07 17:41:24.892591 ip-10-0-129-39 kubenswrapper[2575]: I0707 17:41:24.892540 2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\"" Jul 07 17:41:24.893484 ip-10-0-129-39 kubenswrapper[2575]: I0707 17:41:24.893469 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"cluster-monitoring-operator-dockercfg-6vlhx\"" Jul 07 17:41:24.893803 ip-10-0-129-39 kubenswrapper[2575]: I0707 17:41:24.893790 2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"telemetry-config\"" Jul 07 17:41:24.903907 ip-10-0-129-39 kubenswrapper[2575]: I0707 17:41:24.903889 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-storage-operator/volume-data-source-validator-5d456fdbcd-4knqs"] Jul 07 17:41:24.903973 ip-10-0-129-39 kubenswrapper[2575]: I0707 17:41:24.903918 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/cluster-monitoring-operator-64786d9cbb-4fqsh"] Jul 07 17:41:24.904014 ip-10-0-129-39 kubenswrapper[2575]: I0707 17:41:24.903989 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-storage-operator/volume-data-source-validator-5d456fdbcd-4knqs" Jul 07 17:41:24.907390 ip-10-0-129-39 kubenswrapper[2575]: I0707 17:41:24.907369 2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-storage-operator\"/\"kube-root-ca.crt\"" Jul 07 17:41:24.907390 ip-10-0-129-39 kubenswrapper[2575]: I0707 17:41:24.907384 2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-storage-operator\"/\"openshift-service-ca.crt\"" Jul 07 17:41:24.907591 ip-10-0-129-39 kubenswrapper[2575]: I0707 17:41:24.907577 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-storage-operator\"/\"volume-data-source-validator-dockercfg-tsc7t\"" Jul 07 17:41:24.980959 ip-10-0-129-39 kubenswrapper[2575]: I0707 17:41:24.980928 2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/router-default-5d884d4557-nd85p"] Jul 07 17:41:25.001670 ip-10-0-129-39 kubenswrapper[2575]: I0707 17:41:25.001648 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/router-default-5d884d4557-nd85p"] Jul 07 17:41:25.001781 ip-10-0-129-39 kubenswrapper[2575]: I0707 17:41:25.001743 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/router-default-5d884d4557-nd85p" Jul 07 17:41:25.004325 ip-10-0-129-39 kubenswrapper[2575]: I0707 17:41:25.004303 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"default-ingress-cert\"" Jul 07 17:41:25.004429 ip-10-0-129-39 kubenswrapper[2575]: I0707 17:41:25.004312 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-metrics-certs-default\"" Jul 07 17:41:25.004527 ip-10-0-129-39 kubenswrapper[2575]: I0707 17:41:25.004494 2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"kube-root-ca.crt\"" Jul 07 17:41:25.004635 ip-10-0-129-39 kubenswrapper[2575]: I0707 17:41:25.004619 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-dockercfg-6vm4z\"" Jul 07 17:41:25.004721 ip-10-0-129-39 kubenswrapper[2575]: I0707 17:41:25.004707 2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-service-ca.crt\"" Jul 07 17:41:25.004780 ip-10-0-129-39 kubenswrapper[2575]: I0707 17:41:25.004741 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-stats-default\"" Jul 07 17:41:25.004836 ip-10-0-129-39 kubenswrapper[2575]: I0707 17:41:25.004741 2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"service-ca-bundle\"" Jul 07 17:41:25.060282 ip-10-0-129-39 kubenswrapper[2575]: I0707 17:41:25.060260 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"telemetry-config\" (UniqueName: \"kubernetes.io/configmap/bab28ccd-f54a-4691-add4-8c665b4e1124-telemetry-config\") pod \"cluster-monitoring-operator-64786d9cbb-4fqsh\" (UID: \"bab28ccd-f54a-4691-add4-8c665b4e1124\") " pod="openshift-monitoring/cluster-monitoring-operator-64786d9cbb-4fqsh" Jul 07 17:41:25.060371 ip-10-0-129-39 kubenswrapper[2575]: I0707 17:41:25.060286 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bls6j\" (UniqueName: \"kubernetes.io/projected/bab28ccd-f54a-4691-add4-8c665b4e1124-kube-api-access-bls6j\") pod \"cluster-monitoring-operator-64786d9cbb-4fqsh\" (UID: \"bab28ccd-f54a-4691-add4-8c665b4e1124\") " pod="openshift-monitoring/cluster-monitoring-operator-64786d9cbb-4fqsh" Jul 07 17:41:25.060371 ip-10-0-129-39 kubenswrapper[2575]: I0707 17:41:25.060304 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8nbg2\" (UniqueName: \"kubernetes.io/projected/e43b188b-652c-42b7-bb10-16092a59b829-kube-api-access-8nbg2\") pod \"volume-data-source-validator-5d456fdbcd-4knqs\" (UID: \"e43b188b-652c-42b7-bb10-16092a59b829\") " pod="openshift-cluster-storage-operator/volume-data-source-validator-5d456fdbcd-4knqs" Jul 07 17:41:25.060371 ip-10-0-129-39 kubenswrapper[2575]: I0707 17:41:25.060325 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/bab28ccd-f54a-4691-add4-8c665b4e1124-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-64786d9cbb-4fqsh\" (UID: \"bab28ccd-f54a-4691-add4-8c665b4e1124\") " pod="openshift-monitoring/cluster-monitoring-operator-64786d9cbb-4fqsh" Jul 07 17:41:25.124170 ip-10-0-129-39 kubenswrapper[2575]: I0707 17:41:25.124110 2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-network-diagnostics/network-check-source-989f8b8dc-8ddzc"] Jul 07 17:41:25.145326 ip-10-0-129-39 kubenswrapper[2575]: I0707 17:41:25.145310 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-source-989f8b8dc-8ddzc" Jul 07 17:41:25.145412 ip-10-0-129-39 kubenswrapper[2575]: I0707 17:41:25.145323 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-source-989f8b8dc-8ddzc"] Jul 07 17:41:25.147258 ip-10-0-129-39 kubenswrapper[2575]: I0707 17:41:25.147243 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"network-diagnostics-dockercfg-8xxdf\"" Jul 07 17:41:25.160659 ip-10-0-129-39 kubenswrapper[2575]: I0707 17:41:25.160638 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"telemetry-config\" (UniqueName: \"kubernetes.io/configmap/bab28ccd-f54a-4691-add4-8c665b4e1124-telemetry-config\") pod \"cluster-monitoring-operator-64786d9cbb-4fqsh\" (UID: \"bab28ccd-f54a-4691-add4-8c665b4e1124\") " pod="openshift-monitoring/cluster-monitoring-operator-64786d9cbb-4fqsh" Jul 07 17:41:25.160763 ip-10-0-129-39 kubenswrapper[2575]: I0707 17:41:25.160681 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bls6j\" (UniqueName: \"kubernetes.io/projected/bab28ccd-f54a-4691-add4-8c665b4e1124-kube-api-access-bls6j\") pod \"cluster-monitoring-operator-64786d9cbb-4fqsh\" (UID: \"bab28ccd-f54a-4691-add4-8c665b4e1124\") " pod="openshift-monitoring/cluster-monitoring-operator-64786d9cbb-4fqsh" Jul 07 17:41:25.160763 ip-10-0-129-39 kubenswrapper[2575]: I0707 17:41:25.160712 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/080f0a04-d8b0-45c9-98f7-db87cbc18520-service-ca-bundle\") pod \"router-default-5d884d4557-nd85p\" (UID: \"080f0a04-d8b0-45c9-98f7-db87cbc18520\") " pod="openshift-ingress/router-default-5d884d4557-nd85p" Jul 07 17:41:25.160877 ip-10-0-129-39 kubenswrapper[2575]: I0707 17:41:25.160828 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8nbg2\" (UniqueName: \"kubernetes.io/projected/e43b188b-652c-42b7-bb10-16092a59b829-kube-api-access-8nbg2\") pod \"volume-data-source-validator-5d456fdbcd-4knqs\" (UID: \"e43b188b-652c-42b7-bb10-16092a59b829\") " pod="openshift-cluster-storage-operator/volume-data-source-validator-5d456fdbcd-4knqs" Jul 07 17:41:25.160877 ip-10-0-129-39 kubenswrapper[2575]: I0707 17:41:25.160856 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/bab28ccd-f54a-4691-add4-8c665b4e1124-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-64786d9cbb-4fqsh\" (UID: \"bab28ccd-f54a-4691-add4-8c665b4e1124\") " pod="openshift-monitoring/cluster-monitoring-operator-64786d9cbb-4fqsh" Jul 07 17:41:25.160972 ip-10-0-129-39 kubenswrapper[2575]: I0707 17:41:25.160883 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"stats-auth\" (UniqueName: \"kubernetes.io/secret/080f0a04-d8b0-45c9-98f7-db87cbc18520-stats-auth\") pod \"router-default-5d884d4557-nd85p\" (UID: \"080f0a04-d8b0-45c9-98f7-db87cbc18520\") " pod="openshift-ingress/router-default-5d884d4557-nd85p" Jul 07 17:41:25.160972 ip-10-0-129-39 kubenswrapper[2575]: I0707 17:41:25.160915 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"default-certificate\" (UniqueName: \"kubernetes.io/secret/080f0a04-d8b0-45c9-98f7-db87cbc18520-default-certificate\") pod \"router-default-5d884d4557-nd85p\" (UID: \"080f0a04-d8b0-45c9-98f7-db87cbc18520\") " pod="openshift-ingress/router-default-5d884d4557-nd85p" Jul 07 17:41:25.161046 ip-10-0-129-39 kubenswrapper[2575]: I0707 17:41:25.160965 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-s9tvg\" (UniqueName: \"kubernetes.io/projected/080f0a04-d8b0-45c9-98f7-db87cbc18520-kube-api-access-s9tvg\") pod \"router-default-5d884d4557-nd85p\" (UID: \"080f0a04-d8b0-45c9-98f7-db87cbc18520\") " pod="openshift-ingress/router-default-5d884d4557-nd85p" Jul 07 17:41:25.161046 ip-10-0-129-39 kubenswrapper[2575]: E0707 17:41:25.161012 2575 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found Jul 07 17:41:25.161130 ip-10-0-129-39 kubenswrapper[2575]: E0707 17:41:25.161073 2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/bab28ccd-f54a-4691-add4-8c665b4e1124-cluster-monitoring-operator-tls podName:bab28ccd-f54a-4691-add4-8c665b4e1124 nodeName:}" failed. No retries permitted until 2026-07-07 17:41:25.661054067 +0000 UTC m=+46.166860715 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/bab28ccd-f54a-4691-add4-8c665b4e1124-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-64786d9cbb-4fqsh" (UID: "bab28ccd-f54a-4691-add4-8c665b4e1124") : secret "cluster-monitoring-operator-tls" not found Jul 07 17:41:25.161186 ip-10-0-129-39 kubenswrapper[2575]: I0707 17:41:25.161148 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/080f0a04-d8b0-45c9-98f7-db87cbc18520-metrics-certs\") pod \"router-default-5d884d4557-nd85p\" (UID: \"080f0a04-d8b0-45c9-98f7-db87cbc18520\") " pod="openshift-ingress/router-default-5d884d4557-nd85p" Jul 07 17:41:25.161484 ip-10-0-129-39 kubenswrapper[2575]: I0707 17:41:25.161466 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"telemetry-config\" (UniqueName: \"kubernetes.io/configmap/bab28ccd-f54a-4691-add4-8c665b4e1124-telemetry-config\") pod \"cluster-monitoring-operator-64786d9cbb-4fqsh\" (UID: \"bab28ccd-f54a-4691-add4-8c665b4e1124\") " pod="openshift-monitoring/cluster-monitoring-operator-64786d9cbb-4fqsh" Jul 07 17:41:25.170643 ip-10-0-129-39 kubenswrapper[2575]: I0707 17:41:25.170621 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bls6j\" (UniqueName: \"kubernetes.io/projected/bab28ccd-f54a-4691-add4-8c665b4e1124-kube-api-access-bls6j\") pod \"cluster-monitoring-operator-64786d9cbb-4fqsh\" (UID: \"bab28ccd-f54a-4691-add4-8c665b4e1124\") " pod="openshift-monitoring/cluster-monitoring-operator-64786d9cbb-4fqsh" Jul 07 17:41:25.170803 ip-10-0-129-39 kubenswrapper[2575]: I0707 17:41:25.170783 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8nbg2\" (UniqueName: \"kubernetes.io/projected/e43b188b-652c-42b7-bb10-16092a59b829-kube-api-access-8nbg2\") pod \"volume-data-source-validator-5d456fdbcd-4knqs\" (UID: \"e43b188b-652c-42b7-bb10-16092a59b829\") " pod="openshift-cluster-storage-operator/volume-data-source-validator-5d456fdbcd-4knqs" Jul 07 17:41:25.211899 ip-10-0-129-39 kubenswrapper[2575]: I0707 17:41:25.211882 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-storage-operator/volume-data-source-validator-5d456fdbcd-4knqs" Jul 07 17:41:25.262015 ip-10-0-129-39 kubenswrapper[2575]: I0707 17:41:25.261992 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2khzt\" (UniqueName: \"kubernetes.io/projected/bf901975-3175-4cd5-b4f4-0e822bc66a1a-kube-api-access-2khzt\") pod \"network-check-source-989f8b8dc-8ddzc\" (UID: \"bf901975-3175-4cd5-b4f4-0e822bc66a1a\") " pod="openshift-network-diagnostics/network-check-source-989f8b8dc-8ddzc" Jul 07 17:41:25.262121 ip-10-0-129-39 kubenswrapper[2575]: I0707 17:41:25.262100 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/080f0a04-d8b0-45c9-98f7-db87cbc18520-metrics-certs\") pod \"router-default-5d884d4557-nd85p\" (UID: \"080f0a04-d8b0-45c9-98f7-db87cbc18520\") " pod="openshift-ingress/router-default-5d884d4557-nd85p" Jul 07 17:41:25.262169 ip-10-0-129-39 kubenswrapper[2575]: I0707 17:41:25.262157 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/080f0a04-d8b0-45c9-98f7-db87cbc18520-service-ca-bundle\") pod \"router-default-5d884d4557-nd85p\" (UID: \"080f0a04-d8b0-45c9-98f7-db87cbc18520\") " pod="openshift-ingress/router-default-5d884d4557-nd85p" Jul 07 17:41:25.262213 ip-10-0-129-39 kubenswrapper[2575]: I0707 17:41:25.262186 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"stats-auth\" (UniqueName: \"kubernetes.io/secret/080f0a04-d8b0-45c9-98f7-db87cbc18520-stats-auth\") pod \"router-default-5d884d4557-nd85p\" (UID: \"080f0a04-d8b0-45c9-98f7-db87cbc18520\") " pod="openshift-ingress/router-default-5d884d4557-nd85p" Jul 07 17:41:25.262248 ip-10-0-129-39 kubenswrapper[2575]: I0707 17:41:25.262219 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"default-certificate\" (UniqueName: \"kubernetes.io/secret/080f0a04-d8b0-45c9-98f7-db87cbc18520-default-certificate\") pod \"router-default-5d884d4557-nd85p\" (UID: \"080f0a04-d8b0-45c9-98f7-db87cbc18520\") " pod="openshift-ingress/router-default-5d884d4557-nd85p" Jul 07 17:41:25.262248 ip-10-0-129-39 kubenswrapper[2575]: I0707 17:41:25.262243 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s9tvg\" (UniqueName: \"kubernetes.io/projected/080f0a04-d8b0-45c9-98f7-db87cbc18520-kube-api-access-s9tvg\") pod \"router-default-5d884d4557-nd85p\" (UID: \"080f0a04-d8b0-45c9-98f7-db87cbc18520\") " pod="openshift-ingress/router-default-5d884d4557-nd85p" Jul 07 17:41:25.262363 ip-10-0-129-39 kubenswrapper[2575]: E0707 17:41:25.262274 2575 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found Jul 07 17:41:25.262363 ip-10-0-129-39 kubenswrapper[2575]: E0707 17:41:25.262314 2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/080f0a04-d8b0-45c9-98f7-db87cbc18520-service-ca-bundle podName:080f0a04-d8b0-45c9-98f7-db87cbc18520 nodeName:}" failed. No retries permitted until 2026-07-07 17:41:25.76229449 +0000 UTC m=+46.268101176 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/080f0a04-d8b0-45c9-98f7-db87cbc18520-service-ca-bundle") pod "router-default-5d884d4557-nd85p" (UID: "080f0a04-d8b0-45c9-98f7-db87cbc18520") : configmap references non-existent config key: service-ca.crt Jul 07 17:41:25.262363 ip-10-0-129-39 kubenswrapper[2575]: E0707 17:41:25.262332 2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/080f0a04-d8b0-45c9-98f7-db87cbc18520-metrics-certs podName:080f0a04-d8b0-45c9-98f7-db87cbc18520 nodeName:}" failed. No retries permitted until 2026-07-07 17:41:25.762325315 +0000 UTC m=+46.268131959 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/080f0a04-d8b0-45c9-98f7-db87cbc18520-metrics-certs") pod "router-default-5d884d4557-nd85p" (UID: "080f0a04-d8b0-45c9-98f7-db87cbc18520") : secret "router-metrics-certs-default" not found Jul 07 17:41:25.264742 ip-10-0-129-39 kubenswrapper[2575]: I0707 17:41:25.264721 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"default-certificate\" (UniqueName: \"kubernetes.io/secret/080f0a04-d8b0-45c9-98f7-db87cbc18520-default-certificate\") pod \"router-default-5d884d4557-nd85p\" (UID: \"080f0a04-d8b0-45c9-98f7-db87cbc18520\") " pod="openshift-ingress/router-default-5d884d4557-nd85p" Jul 07 17:41:25.264838 ip-10-0-129-39 kubenswrapper[2575]: I0707 17:41:25.264782 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"stats-auth\" (UniqueName: \"kubernetes.io/secret/080f0a04-d8b0-45c9-98f7-db87cbc18520-stats-auth\") pod \"router-default-5d884d4557-nd85p\" (UID: \"080f0a04-d8b0-45c9-98f7-db87cbc18520\") " pod="openshift-ingress/router-default-5d884d4557-nd85p" Jul 07 17:41:25.271751 ip-10-0-129-39 kubenswrapper[2575]: I0707 17:41:25.271727 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-s9tvg\" (UniqueName: \"kubernetes.io/projected/080f0a04-d8b0-45c9-98f7-db87cbc18520-kube-api-access-s9tvg\") pod \"router-default-5d884d4557-nd85p\" (UID: \"080f0a04-d8b0-45c9-98f7-db87cbc18520\") " pod="openshift-ingress/router-default-5d884d4557-nd85p" Jul 07 17:41:25.354871 ip-10-0-129-39 kubenswrapper[2575]: I0707 17:41:25.354707 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-storage-operator/volume-data-source-validator-5d456fdbcd-4knqs"] Jul 07 17:41:25.358437 ip-10-0-129-39 kubenswrapper[2575]: W0707 17:41:25.358410 2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode43b188b_652c_42b7_bb10_16092a59b829.slice/crio-0acfcd89f8dd8cfca3dd9dfa85c3043d54a3948dec686969506ecaf634c93353 WatchSource:0}: Error finding container 0acfcd89f8dd8cfca3dd9dfa85c3043d54a3948dec686969506ecaf634c93353: Status 404 returned error can't find the container with id 0acfcd89f8dd8cfca3dd9dfa85c3043d54a3948dec686969506ecaf634c93353 Jul 07 17:41:25.363393 ip-10-0-129-39 kubenswrapper[2575]: I0707 17:41:25.363368 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2khzt\" (UniqueName: \"kubernetes.io/projected/bf901975-3175-4cd5-b4f4-0e822bc66a1a-kube-api-access-2khzt\") pod \"network-check-source-989f8b8dc-8ddzc\" (UID: \"bf901975-3175-4cd5-b4f4-0e822bc66a1a\") " pod="openshift-network-diagnostics/network-check-source-989f8b8dc-8ddzc" Jul 07 17:41:25.372322 ip-10-0-129-39 kubenswrapper[2575]: I0707 17:41:25.372302 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2khzt\" (UniqueName: \"kubernetes.io/projected/bf901975-3175-4cd5-b4f4-0e822bc66a1a-kube-api-access-2khzt\") pod \"network-check-source-989f8b8dc-8ddzc\" (UID: \"bf901975-3175-4cd5-b4f4-0e822bc66a1a\") " pod="openshift-network-diagnostics/network-check-source-989f8b8dc-8ddzc" Jul 07 17:41:25.453351 ip-10-0-129-39 kubenswrapper[2575]: I0707 17:41:25.453285 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-source-989f8b8dc-8ddzc" Jul 07 17:41:25.574113 ip-10-0-129-39 kubenswrapper[2575]: I0707 17:41:25.574089 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-source-989f8b8dc-8ddzc"] Jul 07 17:41:25.577309 ip-10-0-129-39 kubenswrapper[2575]: W0707 17:41:25.577284 2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbf901975_3175_4cd5_b4f4_0e822bc66a1a.slice/crio-eff4656821928100b7e1d72838316a84ad2c9deb290ef4a77661e40012c7231e WatchSource:0}: Error finding container eff4656821928100b7e1d72838316a84ad2c9deb290ef4a77661e40012c7231e: Status 404 returned error can't find the container with id eff4656821928100b7e1d72838316a84ad2c9deb290ef4a77661e40012c7231e Jul 07 17:41:25.665280 ip-10-0-129-39 kubenswrapper[2575]: I0707 17:41:25.665255 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/bab28ccd-f54a-4691-add4-8c665b4e1124-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-64786d9cbb-4fqsh\" (UID: \"bab28ccd-f54a-4691-add4-8c665b4e1124\") " pod="openshift-monitoring/cluster-monitoring-operator-64786d9cbb-4fqsh" Jul 07 17:41:25.665415 ip-10-0-129-39 kubenswrapper[2575]: E0707 17:41:25.665398 2575 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found Jul 07 17:41:25.665470 ip-10-0-129-39 kubenswrapper[2575]: E0707 17:41:25.665456 2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/bab28ccd-f54a-4691-add4-8c665b4e1124-cluster-monitoring-operator-tls podName:bab28ccd-f54a-4691-add4-8c665b4e1124 nodeName:}" failed. No retries permitted until 2026-07-07 17:41:26.665442503 +0000 UTC m=+47.171249147 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/bab28ccd-f54a-4691-add4-8c665b4e1124-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-64786d9cbb-4fqsh" (UID: "bab28ccd-f54a-4691-add4-8c665b4e1124") : secret "cluster-monitoring-operator-tls" not found Jul 07 17:41:25.766565 ip-10-0-129-39 kubenswrapper[2575]: I0707 17:41:25.766483 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/080f0a04-d8b0-45c9-98f7-db87cbc18520-metrics-certs\") pod \"router-default-5d884d4557-nd85p\" (UID: \"080f0a04-d8b0-45c9-98f7-db87cbc18520\") " pod="openshift-ingress/router-default-5d884d4557-nd85p" Jul 07 17:41:25.766565 ip-10-0-129-39 kubenswrapper[2575]: I0707 17:41:25.766554 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/080f0a04-d8b0-45c9-98f7-db87cbc18520-service-ca-bundle\") pod \"router-default-5d884d4557-nd85p\" (UID: \"080f0a04-d8b0-45c9-98f7-db87cbc18520\") " pod="openshift-ingress/router-default-5d884d4557-nd85p" Jul 07 17:41:25.766750 ip-10-0-129-39 kubenswrapper[2575]: E0707 17:41:25.766645 2575 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found Jul 07 17:41:25.766750 ip-10-0-129-39 kubenswrapper[2575]: E0707 17:41:25.766672 2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/080f0a04-d8b0-45c9-98f7-db87cbc18520-service-ca-bundle podName:080f0a04-d8b0-45c9-98f7-db87cbc18520 nodeName:}" failed. No retries permitted until 2026-07-07 17:41:26.766659591 +0000 UTC m=+47.272466235 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/080f0a04-d8b0-45c9-98f7-db87cbc18520-service-ca-bundle") pod "router-default-5d884d4557-nd85p" (UID: "080f0a04-d8b0-45c9-98f7-db87cbc18520") : configmap references non-existent config key: service-ca.crt Jul 07 17:41:25.766750 ip-10-0-129-39 kubenswrapper[2575]: E0707 17:41:25.766708 2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/080f0a04-d8b0-45c9-98f7-db87cbc18520-metrics-certs podName:080f0a04-d8b0-45c9-98f7-db87cbc18520 nodeName:}" failed. No retries permitted until 2026-07-07 17:41:26.766689011 +0000 UTC m=+47.272495654 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/080f0a04-d8b0-45c9-98f7-db87cbc18520-metrics-certs") pod "router-default-5d884d4557-nd85p" (UID: "080f0a04-d8b0-45c9-98f7-db87cbc18520") : secret "router-metrics-certs-default" not found Jul 07 17:41:26.310565 ip-10-0-129-39 kubenswrapper[2575]: I0707 17:41:26.310527 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-source-989f8b8dc-8ddzc" event={"ID":"bf901975-3175-4cd5-b4f4-0e822bc66a1a","Type":"ContainerStarted","Data":"eff4656821928100b7e1d72838316a84ad2c9deb290ef4a77661e40012c7231e"} Jul 07 17:41:26.312020 ip-10-0-129-39 kubenswrapper[2575]: I0707 17:41:26.311992 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-storage-operator/volume-data-source-validator-5d456fdbcd-4knqs" event={"ID":"e43b188b-652c-42b7-bb10-16092a59b829","Type":"ContainerStarted","Data":"0acfcd89f8dd8cfca3dd9dfa85c3043d54a3948dec686969506ecaf634c93353"} Jul 07 17:41:26.685663 ip-10-0-129-39 kubenswrapper[2575]: I0707 17:41:26.685624 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/bab28ccd-f54a-4691-add4-8c665b4e1124-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-64786d9cbb-4fqsh\" (UID: \"bab28ccd-f54a-4691-add4-8c665b4e1124\") " pod="openshift-monitoring/cluster-monitoring-operator-64786d9cbb-4fqsh" Jul 07 17:41:26.685889 ip-10-0-129-39 kubenswrapper[2575]: E0707 17:41:26.685778 2575 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found Jul 07 17:41:26.685889 ip-10-0-129-39 kubenswrapper[2575]: E0707 17:41:26.685846 2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/bab28ccd-f54a-4691-add4-8c665b4e1124-cluster-monitoring-operator-tls podName:bab28ccd-f54a-4691-add4-8c665b4e1124 nodeName:}" failed. No retries permitted until 2026-07-07 17:41:28.685830255 +0000 UTC m=+49.191636899 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/bab28ccd-f54a-4691-add4-8c665b4e1124-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-64786d9cbb-4fqsh" (UID: "bab28ccd-f54a-4691-add4-8c665b4e1124") : secret "cluster-monitoring-operator-tls" not found Jul 07 17:41:26.787173 ip-10-0-129-39 kubenswrapper[2575]: I0707 17:41:26.787135 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/080f0a04-d8b0-45c9-98f7-db87cbc18520-metrics-certs\") pod \"router-default-5d884d4557-nd85p\" (UID: \"080f0a04-d8b0-45c9-98f7-db87cbc18520\") " pod="openshift-ingress/router-default-5d884d4557-nd85p" Jul 07 17:41:26.787314 ip-10-0-129-39 kubenswrapper[2575]: I0707 17:41:26.787209 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/080f0a04-d8b0-45c9-98f7-db87cbc18520-service-ca-bundle\") pod \"router-default-5d884d4557-nd85p\" (UID: \"080f0a04-d8b0-45c9-98f7-db87cbc18520\") " pod="openshift-ingress/router-default-5d884d4557-nd85p" Jul 07 17:41:26.787314 ip-10-0-129-39 kubenswrapper[2575]: E0707 17:41:26.787298 2575 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found Jul 07 17:41:26.787426 ip-10-0-129-39 kubenswrapper[2575]: E0707 17:41:26.787365 2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/080f0a04-d8b0-45c9-98f7-db87cbc18520-service-ca-bundle podName:080f0a04-d8b0-45c9-98f7-db87cbc18520 nodeName:}" failed. No retries permitted until 2026-07-07 17:41:28.787347086 +0000 UTC m=+49.293153734 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/080f0a04-d8b0-45c9-98f7-db87cbc18520-service-ca-bundle") pod "router-default-5d884d4557-nd85p" (UID: "080f0a04-d8b0-45c9-98f7-db87cbc18520") : configmap references non-existent config key: service-ca.crt Jul 07 17:41:26.787426 ip-10-0-129-39 kubenswrapper[2575]: E0707 17:41:26.787388 2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/080f0a04-d8b0-45c9-98f7-db87cbc18520-metrics-certs podName:080f0a04-d8b0-45c9-98f7-db87cbc18520 nodeName:}" failed. No retries permitted until 2026-07-07 17:41:28.787378285 +0000 UTC m=+49.293184933 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/080f0a04-d8b0-45c9-98f7-db87cbc18520-metrics-certs") pod "router-default-5d884d4557-nd85p" (UID: "080f0a04-d8b0-45c9-98f7-db87cbc18520") : secret "router-metrics-certs-default" not found Jul 07 17:41:28.318106 ip-10-0-129-39 kubenswrapper[2575]: I0707 17:41:28.318068 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-storage-operator/volume-data-source-validator-5d456fdbcd-4knqs" event={"ID":"e43b188b-652c-42b7-bb10-16092a59b829","Type":"ContainerStarted","Data":"b6a145a63b6ce0686bc8e0f3f196bc1367ea5bea01100b82eef62f2b11a71998"} Jul 07 17:41:28.333173 ip-10-0-129-39 kubenswrapper[2575]: I0707 17:41:28.333126 2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-storage-operator/volume-data-source-validator-5d456fdbcd-4knqs" podStartSLOduration=2.279657294 podStartE2EDuration="4.333111137s" totalImagesPullingTime="2.053453843s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-07 17:41:24 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 17:41:28.332663555 +0000 UTC m=+48.838470218" watchObservedRunningTime="2026-07-07 17:41:28.333111137 +0000 UTC m=+48.838917803" Jul 07 17:41:28.705167 ip-10-0-129-39 kubenswrapper[2575]: I0707 17:41:28.705123 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/bab28ccd-f54a-4691-add4-8c665b4e1124-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-64786d9cbb-4fqsh\" (UID: \"bab28ccd-f54a-4691-add4-8c665b4e1124\") " pod="openshift-monitoring/cluster-monitoring-operator-64786d9cbb-4fqsh" Jul 07 17:41:28.705428 ip-10-0-129-39 kubenswrapper[2575]: E0707 17:41:28.705403 2575 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found Jul 07 17:41:28.705534 ip-10-0-129-39 kubenswrapper[2575]: E0707 17:41:28.705490 2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/bab28ccd-f54a-4691-add4-8c665b4e1124-cluster-monitoring-operator-tls podName:bab28ccd-f54a-4691-add4-8c665b4e1124 nodeName:}" failed. No retries permitted until 2026-07-07 17:41:32.70546883 +0000 UTC m=+53.211275477 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/bab28ccd-f54a-4691-add4-8c665b4e1124-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-64786d9cbb-4fqsh" (UID: "bab28ccd-f54a-4691-add4-8c665b4e1124") : secret "cluster-monitoring-operator-tls" not found Jul 07 17:41:28.806122 ip-10-0-129-39 kubenswrapper[2575]: I0707 17:41:28.806082 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/080f0a04-d8b0-45c9-98f7-db87cbc18520-metrics-certs\") pod \"router-default-5d884d4557-nd85p\" (UID: \"080f0a04-d8b0-45c9-98f7-db87cbc18520\") " pod="openshift-ingress/router-default-5d884d4557-nd85p" Jul 07 17:41:28.806304 ip-10-0-129-39 kubenswrapper[2575]: I0707 17:41:28.806217 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/080f0a04-d8b0-45c9-98f7-db87cbc18520-service-ca-bundle\") pod \"router-default-5d884d4557-nd85p\" (UID: \"080f0a04-d8b0-45c9-98f7-db87cbc18520\") " pod="openshift-ingress/router-default-5d884d4557-nd85p" Jul 07 17:41:28.806535 ip-10-0-129-39 kubenswrapper[2575]: E0707 17:41:28.806322 2575 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found Jul 07 17:41:28.806535 ip-10-0-129-39 kubenswrapper[2575]: E0707 17:41:28.806400 2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/080f0a04-d8b0-45c9-98f7-db87cbc18520-metrics-certs podName:080f0a04-d8b0-45c9-98f7-db87cbc18520 nodeName:}" failed. No retries permitted until 2026-07-07 17:41:32.806379775 +0000 UTC m=+53.312186424 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/080f0a04-d8b0-45c9-98f7-db87cbc18520-metrics-certs") pod "router-default-5d884d4557-nd85p" (UID: "080f0a04-d8b0-45c9-98f7-db87cbc18520") : secret "router-metrics-certs-default" not found Jul 07 17:41:28.806535 ip-10-0-129-39 kubenswrapper[2575]: E0707 17:41:28.806419 2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/080f0a04-d8b0-45c9-98f7-db87cbc18520-service-ca-bundle podName:080f0a04-d8b0-45c9-98f7-db87cbc18520 nodeName:}" failed. No retries permitted until 2026-07-07 17:41:32.806409194 +0000 UTC m=+53.312215839 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/080f0a04-d8b0-45c9-98f7-db87cbc18520-service-ca-bundle") pod "router-default-5d884d4557-nd85p" (UID: "080f0a04-d8b0-45c9-98f7-db87cbc18520") : configmap references non-existent config key: service-ca.crt Jul 07 17:41:29.614212 ip-10-0-129-39 kubenswrapper[2575]: I0707 17:41:29.614168 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/b2656ecc-1db3-4850-b2f6-58b0a14eb683-metrics-tls\") pod \"dns-default-b2mnn\" (UID: \"b2656ecc-1db3-4850-b2f6-58b0a14eb683\") " pod="openshift-dns/dns-default-b2mnn" Jul 07 17:41:29.614705 ip-10-0-129-39 kubenswrapper[2575]: E0707 17:41:29.614295 2575 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jul 07 17:41:29.614705 ip-10-0-129-39 kubenswrapper[2575]: E0707 17:41:29.614363 2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b2656ecc-1db3-4850-b2f6-58b0a14eb683-metrics-tls podName:b2656ecc-1db3-4850-b2f6-58b0a14eb683 nodeName:}" failed. No retries permitted until 2026-07-07 17:41:45.614346297 +0000 UTC m=+66.120152964 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/b2656ecc-1db3-4850-b2f6-58b0a14eb683-metrics-tls") pod "dns-default-b2mnn" (UID: "b2656ecc-1db3-4850-b2f6-58b0a14eb683") : secret "dns-default-metrics-tls" not found Jul 07 17:41:29.809652 ip-10-0-129-39 kubenswrapper[2575]: I0707 17:41:29.809619 2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-runtime-extractor-fmk64"] Jul 07 17:41:29.823724 ip-10-0-129-39 kubenswrapper[2575]: I0707 17:41:29.823700 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-fmk64" Jul 07 17:41:29.825971 ip-10-0-129-39 kubenswrapper[2575]: I0707 17:41:29.825947 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-fmk64"] Jul 07 17:41:29.826570 ip-10-0-129-39 kubenswrapper[2575]: I0707 17:41:29.826540 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\"" Jul 07 17:41:29.827568 ip-10-0-129-39 kubenswrapper[2575]: I0707 17:41:29.827541 2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\"" Jul 07 17:41:29.827666 ip-10-0-129-39 kubenswrapper[2575]: I0707 17:41:29.827603 2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\"" Jul 07 17:41:29.827666 ip-10-0-129-39 kubenswrapper[2575]: I0707 17:41:29.827625 2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\"" Jul 07 17:41:29.827775 ip-10-0-129-39 kubenswrapper[2575]: I0707 17:41:29.827690 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-6sssg\"" Jul 07 17:41:29.917086 ip-10-0-129-39 kubenswrapper[2575]: I0707 17:41:29.917030 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5fm2j\" (UniqueName: \"kubernetes.io/projected/0c1d1c87-6251-49e9-ba58-47cb22ee80ca-kube-api-access-5fm2j\") pod \"insights-runtime-extractor-fmk64\" (UID: \"0c1d1c87-6251-49e9-ba58-47cb22ee80ca\") " pod="openshift-insights/insights-runtime-extractor-fmk64" Jul 07 17:41:29.917190 ip-10-0-129-39 kubenswrapper[2575]: I0707 17:41:29.917090 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/0c1d1c87-6251-49e9-ba58-47cb22ee80ca-data-volume\") pod \"insights-runtime-extractor-fmk64\" (UID: \"0c1d1c87-6251-49e9-ba58-47cb22ee80ca\") " pod="openshift-insights/insights-runtime-extractor-fmk64" Jul 07 17:41:29.917190 ip-10-0-129-39 kubenswrapper[2575]: I0707 17:41:29.917121 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/0c1d1c87-6251-49e9-ba58-47cb22ee80ca-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-fmk64\" (UID: \"0c1d1c87-6251-49e9-ba58-47cb22ee80ca\") " pod="openshift-insights/insights-runtime-extractor-fmk64" Jul 07 17:41:29.917257 ip-10-0-129-39 kubenswrapper[2575]: I0707 17:41:29.917190 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/0c1d1c87-6251-49e9-ba58-47cb22ee80ca-crio-socket\") pod \"insights-runtime-extractor-fmk64\" (UID: \"0c1d1c87-6251-49e9-ba58-47cb22ee80ca\") " pod="openshift-insights/insights-runtime-extractor-fmk64" Jul 07 17:41:29.917257 ip-10-0-129-39 kubenswrapper[2575]: I0707 17:41:29.917206 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/0c1d1c87-6251-49e9-ba58-47cb22ee80ca-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-fmk64\" (UID: \"0c1d1c87-6251-49e9-ba58-47cb22ee80ca\") " pod="openshift-insights/insights-runtime-extractor-fmk64" Jul 07 17:41:30.018187 ip-10-0-129-39 kubenswrapper[2575]: I0707 17:41:30.018167 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/0c1d1c87-6251-49e9-ba58-47cb22ee80ca-crio-socket\") pod \"insights-runtime-extractor-fmk64\" (UID: \"0c1d1c87-6251-49e9-ba58-47cb22ee80ca\") " pod="openshift-insights/insights-runtime-extractor-fmk64" Jul 07 17:41:30.018295 ip-10-0-129-39 kubenswrapper[2575]: I0707 17:41:30.018195 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/0c1d1c87-6251-49e9-ba58-47cb22ee80ca-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-fmk64\" (UID: \"0c1d1c87-6251-49e9-ba58-47cb22ee80ca\") " pod="openshift-insights/insights-runtime-extractor-fmk64" Jul 07 17:41:30.018295 ip-10-0-129-39 kubenswrapper[2575]: I0707 17:41:30.018224 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5fm2j\" (UniqueName: \"kubernetes.io/projected/0c1d1c87-6251-49e9-ba58-47cb22ee80ca-kube-api-access-5fm2j\") pod \"insights-runtime-extractor-fmk64\" (UID: \"0c1d1c87-6251-49e9-ba58-47cb22ee80ca\") " pod="openshift-insights/insights-runtime-extractor-fmk64" Jul 07 17:41:30.018295 ip-10-0-129-39 kubenswrapper[2575]: I0707 17:41:30.018273 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/0c1d1c87-6251-49e9-ba58-47cb22ee80ca-data-volume\") pod \"insights-runtime-extractor-fmk64\" (UID: \"0c1d1c87-6251-49e9-ba58-47cb22ee80ca\") " pod="openshift-insights/insights-runtime-extractor-fmk64" Jul 07 17:41:30.018433 ip-10-0-129-39 kubenswrapper[2575]: I0707 17:41:30.018294 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/0c1d1c87-6251-49e9-ba58-47cb22ee80ca-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-fmk64\" (UID: \"0c1d1c87-6251-49e9-ba58-47cb22ee80ca\") " pod="openshift-insights/insights-runtime-extractor-fmk64" Jul 07 17:41:30.018433 ip-10-0-129-39 kubenswrapper[2575]: E0707 17:41:30.018315 2575 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Jul 07 17:41:30.018433 ip-10-0-129-39 kubenswrapper[2575]: E0707 17:41:30.018357 2575 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Jul 07 17:41:30.018433 ip-10-0-129-39 kubenswrapper[2575]: E0707 17:41:30.018376 2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0c1d1c87-6251-49e9-ba58-47cb22ee80ca-insights-runtime-extractor-tls podName:0c1d1c87-6251-49e9-ba58-47cb22ee80ca nodeName:}" failed. No retries permitted until 2026-07-07 17:41:30.518357521 +0000 UTC m=+51.024164168 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/0c1d1c87-6251-49e9-ba58-47cb22ee80ca-insights-runtime-extractor-tls") pod "insights-runtime-extractor-fmk64" (UID: "0c1d1c87-6251-49e9-ba58-47cb22ee80ca") : secret "insights-runtime-extractor-tls" not found Jul 07 17:41:30.018433 ip-10-0-129-39 kubenswrapper[2575]: I0707 17:41:30.018388 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/0c1d1c87-6251-49e9-ba58-47cb22ee80ca-crio-socket\") pod \"insights-runtime-extractor-fmk64\" (UID: \"0c1d1c87-6251-49e9-ba58-47cb22ee80ca\") " pod="openshift-insights/insights-runtime-extractor-fmk64" Jul 07 17:41:30.018433 ip-10-0-129-39 kubenswrapper[2575]: E0707 17:41:30.018407 2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/0c1d1c87-6251-49e9-ba58-47cb22ee80ca-kube-rbac-proxy-cm podName:0c1d1c87-6251-49e9-ba58-47cb22ee80ca nodeName:}" failed. No retries permitted until 2026-07-07 17:41:30.518396928 +0000 UTC m=+51.024203571 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/0c1d1c87-6251-49e9-ba58-47cb22ee80ca-kube-rbac-proxy-cm") pod "insights-runtime-extractor-fmk64" (UID: "0c1d1c87-6251-49e9-ba58-47cb22ee80ca") : configmap "kube-rbac-proxy" not found Jul 07 17:41:30.018645 ip-10-0-129-39 kubenswrapper[2575]: I0707 17:41:30.018630 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/0c1d1c87-6251-49e9-ba58-47cb22ee80ca-data-volume\") pod \"insights-runtime-extractor-fmk64\" (UID: \"0c1d1c87-6251-49e9-ba58-47cb22ee80ca\") " pod="openshift-insights/insights-runtime-extractor-fmk64" Jul 07 17:41:30.027014 ip-10-0-129-39 kubenswrapper[2575]: I0707 17:41:30.026991 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5fm2j\" (UniqueName: \"kubernetes.io/projected/0c1d1c87-6251-49e9-ba58-47cb22ee80ca-kube-api-access-5fm2j\") pod \"insights-runtime-extractor-fmk64\" (UID: \"0c1d1c87-6251-49e9-ba58-47cb22ee80ca\") " pod="openshift-insights/insights-runtime-extractor-fmk64" Jul 07 17:41:30.324034 ip-10-0-129-39 kubenswrapper[2575]: I0707 17:41:30.324008 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-source-989f8b8dc-8ddzc" event={"ID":"bf901975-3175-4cd5-b4f4-0e822bc66a1a","Type":"ContainerStarted","Data":"5ae73dc9034a8f6ccb90d7931d6b5179cd1ddd0b3643f9b4e2c97ac73f02a77a"} Jul 07 17:41:30.340658 ip-10-0-129-39 kubenswrapper[2575]: I0707 17:41:30.340614 2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-source-989f8b8dc-8ddzc" podStartSLOduration=0.968157444 podStartE2EDuration="5.340599191s" totalImagesPullingTime="4.372441747s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-07 17:41:25 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 17:41:30.339651449 +0000 UTC m=+50.845458116" watchObservedRunningTime="2026-07-07 17:41:30.340599191 +0000 UTC m=+50.846406022" Jul 07 17:41:30.520467 ip-10-0-129-39 kubenswrapper[2575]: I0707 17:41:30.520437 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/0c1d1c87-6251-49e9-ba58-47cb22ee80ca-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-fmk64\" (UID: \"0c1d1c87-6251-49e9-ba58-47cb22ee80ca\") " pod="openshift-insights/insights-runtime-extractor-fmk64" Jul 07 17:41:30.520637 ip-10-0-129-39 kubenswrapper[2575]: I0707 17:41:30.520534 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/0c1d1c87-6251-49e9-ba58-47cb22ee80ca-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-fmk64\" (UID: \"0c1d1c87-6251-49e9-ba58-47cb22ee80ca\") " pod="openshift-insights/insights-runtime-extractor-fmk64" Jul 07 17:41:30.520637 ip-10-0-129-39 kubenswrapper[2575]: E0707 17:41:30.520557 2575 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Jul 07 17:41:30.520637 ip-10-0-129-39 kubenswrapper[2575]: E0707 17:41:30.520612 2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/0c1d1c87-6251-49e9-ba58-47cb22ee80ca-kube-rbac-proxy-cm podName:0c1d1c87-6251-49e9-ba58-47cb22ee80ca nodeName:}" failed. No retries permitted until 2026-07-07 17:41:31.520596499 +0000 UTC m=+52.026403143 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/0c1d1c87-6251-49e9-ba58-47cb22ee80ca-kube-rbac-proxy-cm") pod "insights-runtime-extractor-fmk64" (UID: "0c1d1c87-6251-49e9-ba58-47cb22ee80ca") : configmap "kube-rbac-proxy" not found Jul 07 17:41:30.520637 ip-10-0-129-39 kubenswrapper[2575]: E0707 17:41:30.520625 2575 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Jul 07 17:41:30.520799 ip-10-0-129-39 kubenswrapper[2575]: E0707 17:41:30.520681 2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0c1d1c87-6251-49e9-ba58-47cb22ee80ca-insights-runtime-extractor-tls podName:0c1d1c87-6251-49e9-ba58-47cb22ee80ca nodeName:}" failed. No retries permitted until 2026-07-07 17:41:31.52066193 +0000 UTC m=+52.026468575 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/0c1d1c87-6251-49e9-ba58-47cb22ee80ca-insights-runtime-extractor-tls") pod "insights-runtime-extractor-fmk64" (UID: "0c1d1c87-6251-49e9-ba58-47cb22ee80ca") : secret "insights-runtime-extractor-tls" not found Jul 07 17:41:31.025654 ip-10-0-129-39 kubenswrapper[2575]: I0707 17:41:31.025622 2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-kube-storage-version-migrator/migrator-d57d58cc9-j2bs6"] Jul 07 17:41:31.041161 ip-10-0-129-39 kubenswrapper[2575]: I0707 17:41:31.041131 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator/migrator-d57d58cc9-j2bs6"] Jul 07 17:41:31.041293 ip-10-0-129-39 kubenswrapper[2575]: I0707 17:41:31.041139 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator/migrator-d57d58cc9-j2bs6" Jul 07 17:41:31.045341 ip-10-0-129-39 kubenswrapper[2575]: I0707 17:41:31.043906 2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator\"/\"kube-root-ca.crt\"" Jul 07 17:41:31.045341 ip-10-0-129-39 kubenswrapper[2575]: I0707 17:41:31.043942 2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator\"/\"openshift-service-ca.crt\"" Jul 07 17:41:31.045341 ip-10-0-129-39 kubenswrapper[2575]: I0707 17:41:31.044797 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator\"/\"kube-storage-version-migrator-sa-dockercfg-2v749\"" Jul 07 17:41:31.125003 ip-10-0-129-39 kubenswrapper[2575]: I0707 17:41:31.124966 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-d5bwg\" (UniqueName: \"kubernetes.io/projected/b7301c91-bb20-4577-9536-2673f662f3fd-kube-api-access-d5bwg\") pod \"migrator-d57d58cc9-j2bs6\" (UID: \"b7301c91-bb20-4577-9536-2673f662f3fd\") " pod="openshift-kube-storage-version-migrator/migrator-d57d58cc9-j2bs6" Jul 07 17:41:31.225573 ip-10-0-129-39 kubenswrapper[2575]: I0707 17:41:31.225538 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d5bwg\" (UniqueName: \"kubernetes.io/projected/b7301c91-bb20-4577-9536-2673f662f3fd-kube-api-access-d5bwg\") pod \"migrator-d57d58cc9-j2bs6\" (UID: \"b7301c91-bb20-4577-9536-2673f662f3fd\") " pod="openshift-kube-storage-version-migrator/migrator-d57d58cc9-j2bs6" Jul 07 17:41:31.234260 ip-10-0-129-39 kubenswrapper[2575]: I0707 17:41:31.234237 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-d5bwg\" (UniqueName: \"kubernetes.io/projected/b7301c91-bb20-4577-9536-2673f662f3fd-kube-api-access-d5bwg\") pod \"migrator-d57d58cc9-j2bs6\" (UID: \"b7301c91-bb20-4577-9536-2673f662f3fd\") " pod="openshift-kube-storage-version-migrator/migrator-d57d58cc9-j2bs6" Jul 07 17:41:31.355593 ip-10-0-129-39 kubenswrapper[2575]: I0707 17:41:31.355569 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator/migrator-d57d58cc9-j2bs6" Jul 07 17:41:31.493104 ip-10-0-129-39 kubenswrapper[2575]: I0707 17:41:31.493075 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator/migrator-d57d58cc9-j2bs6"] Jul 07 17:41:31.496033 ip-10-0-129-39 kubenswrapper[2575]: W0707 17:41:31.496009 2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb7301c91_bb20_4577_9536_2673f662f3fd.slice/crio-2a159343312d20bbdac980118816d2c7787ce85eb9db0101873c448227e98b19 WatchSource:0}: Error finding container 2a159343312d20bbdac980118816d2c7787ce85eb9db0101873c448227e98b19: Status 404 returned error can't find the container with id 2a159343312d20bbdac980118816d2c7787ce85eb9db0101873c448227e98b19 Jul 07 17:41:31.527242 ip-10-0-129-39 kubenswrapper[2575]: I0707 17:41:31.527217 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/0c1d1c87-6251-49e9-ba58-47cb22ee80ca-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-fmk64\" (UID: \"0c1d1c87-6251-49e9-ba58-47cb22ee80ca\") " pod="openshift-insights/insights-runtime-extractor-fmk64" Jul 07 17:41:31.527338 ip-10-0-129-39 kubenswrapper[2575]: I0707 17:41:31.527287 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/0c1d1c87-6251-49e9-ba58-47cb22ee80ca-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-fmk64\" (UID: \"0c1d1c87-6251-49e9-ba58-47cb22ee80ca\") " pod="openshift-insights/insights-runtime-extractor-fmk64" Jul 07 17:41:31.527402 ip-10-0-129-39 kubenswrapper[2575]: E0707 17:41:31.527349 2575 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Jul 07 17:41:31.527402 ip-10-0-129-39 kubenswrapper[2575]: E0707 17:41:31.527382 2575 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Jul 07 17:41:31.527495 ip-10-0-129-39 kubenswrapper[2575]: E0707 17:41:31.527420 2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/0c1d1c87-6251-49e9-ba58-47cb22ee80ca-kube-rbac-proxy-cm podName:0c1d1c87-6251-49e9-ba58-47cb22ee80ca nodeName:}" failed. No retries permitted until 2026-07-07 17:41:33.527399875 +0000 UTC m=+54.033206529 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/0c1d1c87-6251-49e9-ba58-47cb22ee80ca-kube-rbac-proxy-cm") pod "insights-runtime-extractor-fmk64" (UID: "0c1d1c87-6251-49e9-ba58-47cb22ee80ca") : configmap "kube-rbac-proxy" not found Jul 07 17:41:31.527495 ip-10-0-129-39 kubenswrapper[2575]: E0707 17:41:31.527442 2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0c1d1c87-6251-49e9-ba58-47cb22ee80ca-insights-runtime-extractor-tls podName:0c1d1c87-6251-49e9-ba58-47cb22ee80ca nodeName:}" failed. No retries permitted until 2026-07-07 17:41:33.527428445 +0000 UTC m=+54.033235109 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/0c1d1c87-6251-49e9-ba58-47cb22ee80ca-insights-runtime-extractor-tls") pod "insights-runtime-extractor-fmk64" (UID: "0c1d1c87-6251-49e9-ba58-47cb22ee80ca") : secret "insights-runtime-extractor-tls" not found Jul 07 17:41:31.788593 ip-10-0-129-39 kubenswrapper[2575]: I0707 17:41:31.788527 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-fc7qv_d4c44345-d6bf-4778-8911-09c7a1a0740a/dns-node-resolver/0.log" Jul 07 17:41:32.329828 ip-10-0-129-39 kubenswrapper[2575]: I0707 17:41:32.329794 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator/migrator-d57d58cc9-j2bs6" event={"ID":"b7301c91-bb20-4577-9536-2673f662f3fd","Type":"ContainerStarted","Data":"2a159343312d20bbdac980118816d2c7787ce85eb9db0101873c448227e98b19"} Jul 07 17:41:32.736899 ip-10-0-129-39 kubenswrapper[2575]: I0707 17:41:32.736821 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/bab28ccd-f54a-4691-add4-8c665b4e1124-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-64786d9cbb-4fqsh\" (UID: \"bab28ccd-f54a-4691-add4-8c665b4e1124\") " pod="openshift-monitoring/cluster-monitoring-operator-64786d9cbb-4fqsh" Jul 07 17:41:32.737096 ip-10-0-129-39 kubenswrapper[2575]: E0707 17:41:32.736971 2575 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found Jul 07 17:41:32.737096 ip-10-0-129-39 kubenswrapper[2575]: E0707 17:41:32.737045 2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/bab28ccd-f54a-4691-add4-8c665b4e1124-cluster-monitoring-operator-tls podName:bab28ccd-f54a-4691-add4-8c665b4e1124 nodeName:}" failed. No retries permitted until 2026-07-07 17:41:40.737027981 +0000 UTC m=+61.242834638 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/bab28ccd-f54a-4691-add4-8c665b4e1124-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-64786d9cbb-4fqsh" (UID: "bab28ccd-f54a-4691-add4-8c665b4e1124") : secret "cluster-monitoring-operator-tls" not found Jul 07 17:41:32.788383 ip-10-0-129-39 kubenswrapper[2575]: I0707 17:41:32.788355 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-l6pnn_ff237df8-d435-4d8c-88ec-8f31795bf0ef/node-ca/0.log" Jul 07 17:41:32.837902 ip-10-0-129-39 kubenswrapper[2575]: I0707 17:41:32.837878 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/080f0a04-d8b0-45c9-98f7-db87cbc18520-service-ca-bundle\") pod \"router-default-5d884d4557-nd85p\" (UID: \"080f0a04-d8b0-45c9-98f7-db87cbc18520\") " pod="openshift-ingress/router-default-5d884d4557-nd85p" Jul 07 17:41:32.838067 ip-10-0-129-39 kubenswrapper[2575]: I0707 17:41:32.838018 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/080f0a04-d8b0-45c9-98f7-db87cbc18520-metrics-certs\") pod \"router-default-5d884d4557-nd85p\" (UID: \"080f0a04-d8b0-45c9-98f7-db87cbc18520\") " pod="openshift-ingress/router-default-5d884d4557-nd85p" Jul 07 17:41:32.838067 ip-10-0-129-39 kubenswrapper[2575]: E0707 17:41:32.838034 2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/080f0a04-d8b0-45c9-98f7-db87cbc18520-service-ca-bundle podName:080f0a04-d8b0-45c9-98f7-db87cbc18520 nodeName:}" failed. No retries permitted until 2026-07-07 17:41:40.838014733 +0000 UTC m=+61.343821395 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/080f0a04-d8b0-45c9-98f7-db87cbc18520-service-ca-bundle") pod "router-default-5d884d4557-nd85p" (UID: "080f0a04-d8b0-45c9-98f7-db87cbc18520") : configmap references non-existent config key: service-ca.crt Jul 07 17:41:32.838193 ip-10-0-129-39 kubenswrapper[2575]: E0707 17:41:32.838117 2575 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found Jul 07 17:41:32.838193 ip-10-0-129-39 kubenswrapper[2575]: E0707 17:41:32.838175 2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/080f0a04-d8b0-45c9-98f7-db87cbc18520-metrics-certs podName:080f0a04-d8b0-45c9-98f7-db87cbc18520 nodeName:}" failed. No retries permitted until 2026-07-07 17:41:40.838159601 +0000 UTC m=+61.343966250 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/080f0a04-d8b0-45c9-98f7-db87cbc18520-metrics-certs") pod "router-default-5d884d4557-nd85p" (UID: "080f0a04-d8b0-45c9-98f7-db87cbc18520") : secret "router-metrics-certs-default" not found Jul 07 17:41:33.544551 ip-10-0-129-39 kubenswrapper[2575]: I0707 17:41:33.544518 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/0c1d1c87-6251-49e9-ba58-47cb22ee80ca-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-fmk64\" (UID: \"0c1d1c87-6251-49e9-ba58-47cb22ee80ca\") " pod="openshift-insights/insights-runtime-extractor-fmk64" Jul 07 17:41:33.544955 ip-10-0-129-39 kubenswrapper[2575]: E0707 17:41:33.544638 2575 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Jul 07 17:41:33.544955 ip-10-0-129-39 kubenswrapper[2575]: E0707 17:41:33.544706 2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/0c1d1c87-6251-49e9-ba58-47cb22ee80ca-kube-rbac-proxy-cm podName:0c1d1c87-6251-49e9-ba58-47cb22ee80ca nodeName:}" failed. No retries permitted until 2026-07-07 17:41:37.544690308 +0000 UTC m=+58.050496955 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/0c1d1c87-6251-49e9-ba58-47cb22ee80ca-kube-rbac-proxy-cm") pod "insights-runtime-extractor-fmk64" (UID: "0c1d1c87-6251-49e9-ba58-47cb22ee80ca") : configmap "kube-rbac-proxy" not found Jul 07 17:41:33.544955 ip-10-0-129-39 kubenswrapper[2575]: I0707 17:41:33.544704 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/0c1d1c87-6251-49e9-ba58-47cb22ee80ca-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-fmk64\" (UID: \"0c1d1c87-6251-49e9-ba58-47cb22ee80ca\") " pod="openshift-insights/insights-runtime-extractor-fmk64" Jul 07 17:41:33.544955 ip-10-0-129-39 kubenswrapper[2575]: E0707 17:41:33.544826 2575 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Jul 07 17:41:33.544955 ip-10-0-129-39 kubenswrapper[2575]: E0707 17:41:33.544869 2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0c1d1c87-6251-49e9-ba58-47cb22ee80ca-insights-runtime-extractor-tls podName:0c1d1c87-6251-49e9-ba58-47cb22ee80ca nodeName:}" failed. No retries permitted until 2026-07-07 17:41:37.544858998 +0000 UTC m=+58.050665642 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/0c1d1c87-6251-49e9-ba58-47cb22ee80ca-insights-runtime-extractor-tls") pod "insights-runtime-extractor-fmk64" (UID: "0c1d1c87-6251-49e9-ba58-47cb22ee80ca") : secret "insights-runtime-extractor-tls" not found Jul 07 17:41:33.793976 ip-10-0-129-39 kubenswrapper[2575]: I0707 17:41:33.793949 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-x7kdh_1935a145-8811-413c-ad46-ae93176ffd7c/serve-healthcheck-canary/0.log" Jul 07 17:41:34.337496 ip-10-0-129-39 kubenswrapper[2575]: I0707 17:41:34.337458 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator/migrator-d57d58cc9-j2bs6" event={"ID":"b7301c91-bb20-4577-9536-2673f662f3fd","Type":"ContainerStarted","Data":"c57d1dfffc9b14c3cb42831107e1892122033249598616d24041d253e01d4886"} Jul 07 17:41:34.337670 ip-10-0-129-39 kubenswrapper[2575]: I0707 17:41:34.337524 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator/migrator-d57d58cc9-j2bs6" event={"ID":"b7301c91-bb20-4577-9536-2673f662f3fd","Type":"ContainerStarted","Data":"6a59abe609d9fae92cfb1585cb0b2a9bc57bacb057ac8b95433b7edd99a34d3b"} Jul 07 17:41:34.355795 ip-10-0-129-39 kubenswrapper[2575]: I0707 17:41:34.355753 2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-kube-storage-version-migrator/migrator-d57d58cc9-j2bs6" podStartSLOduration=1.055103912 podStartE2EDuration="3.355737249s" totalImagesPullingTime="2.300633337s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-07 17:41:31 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 17:41:34.354558358 +0000 UTC m=+54.860365025" watchObservedRunningTime="2026-07-07 17:41:34.355737249 +0000 UTC m=+54.861543915" Jul 07 17:41:37.573081 ip-10-0-129-39 kubenswrapper[2575]: I0707 17:41:37.573044 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/0c1d1c87-6251-49e9-ba58-47cb22ee80ca-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-fmk64\" (UID: \"0c1d1c87-6251-49e9-ba58-47cb22ee80ca\") " pod="openshift-insights/insights-runtime-extractor-fmk64" Jul 07 17:41:37.573540 ip-10-0-129-39 kubenswrapper[2575]: I0707 17:41:37.573126 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/0c1d1c87-6251-49e9-ba58-47cb22ee80ca-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-fmk64\" (UID: \"0c1d1c87-6251-49e9-ba58-47cb22ee80ca\") " pod="openshift-insights/insights-runtime-extractor-fmk64" Jul 07 17:41:37.573540 ip-10-0-129-39 kubenswrapper[2575]: E0707 17:41:37.573209 2575 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Jul 07 17:41:37.573540 ip-10-0-129-39 kubenswrapper[2575]: E0707 17:41:37.573223 2575 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Jul 07 17:41:37.573540 ip-10-0-129-39 kubenswrapper[2575]: E0707 17:41:37.573279 2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0c1d1c87-6251-49e9-ba58-47cb22ee80ca-insights-runtime-extractor-tls podName:0c1d1c87-6251-49e9-ba58-47cb22ee80ca nodeName:}" failed. No retries permitted until 2026-07-07 17:41:45.573260819 +0000 UTC m=+66.079067482 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/0c1d1c87-6251-49e9-ba58-47cb22ee80ca-insights-runtime-extractor-tls") pod "insights-runtime-extractor-fmk64" (UID: "0c1d1c87-6251-49e9-ba58-47cb22ee80ca") : secret "insights-runtime-extractor-tls" not found Jul 07 17:41:37.573540 ip-10-0-129-39 kubenswrapper[2575]: E0707 17:41:37.573300 2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/0c1d1c87-6251-49e9-ba58-47cb22ee80ca-kube-rbac-proxy-cm podName:0c1d1c87-6251-49e9-ba58-47cb22ee80ca nodeName:}" failed. No retries permitted until 2026-07-07 17:41:45.573290632 +0000 UTC m=+66.079097276 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/0c1d1c87-6251-49e9-ba58-47cb22ee80ca-kube-rbac-proxy-cm") pod "insights-runtime-extractor-fmk64" (UID: "0c1d1c87-6251-49e9-ba58-47cb22ee80ca") : configmap "kube-rbac-proxy" not found Jul 07 17:41:40.796165 ip-10-0-129-39 kubenswrapper[2575]: I0707 17:41:40.796124 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/bab28ccd-f54a-4691-add4-8c665b4e1124-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-64786d9cbb-4fqsh\" (UID: \"bab28ccd-f54a-4691-add4-8c665b4e1124\") " pod="openshift-monitoring/cluster-monitoring-operator-64786d9cbb-4fqsh" Jul 07 17:41:40.798652 ip-10-0-129-39 kubenswrapper[2575]: I0707 17:41:40.798630 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/bab28ccd-f54a-4691-add4-8c665b4e1124-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-64786d9cbb-4fqsh\" (UID: \"bab28ccd-f54a-4691-add4-8c665b4e1124\") " pod="openshift-monitoring/cluster-monitoring-operator-64786d9cbb-4fqsh" Jul 07 17:41:40.800913 ip-10-0-129-39 kubenswrapper[2575]: I0707 17:41:40.800895 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"cluster-monitoring-operator-dockercfg-6vlhx\"" Jul 07 17:41:40.809820 ip-10-0-129-39 kubenswrapper[2575]: I0707 17:41:40.809797 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/cluster-monitoring-operator-64786d9cbb-4fqsh" Jul 07 17:41:40.896957 ip-10-0-129-39 kubenswrapper[2575]: I0707 17:41:40.896930 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/080f0a04-d8b0-45c9-98f7-db87cbc18520-metrics-certs\") pod \"router-default-5d884d4557-nd85p\" (UID: \"080f0a04-d8b0-45c9-98f7-db87cbc18520\") " pod="openshift-ingress/router-default-5d884d4557-nd85p" Jul 07 17:41:40.897129 ip-10-0-129-39 kubenswrapper[2575]: I0707 17:41:40.896975 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/080f0a04-d8b0-45c9-98f7-db87cbc18520-service-ca-bundle\") pod \"router-default-5d884d4557-nd85p\" (UID: \"080f0a04-d8b0-45c9-98f7-db87cbc18520\") " pod="openshift-ingress/router-default-5d884d4557-nd85p" Jul 07 17:41:40.897589 ip-10-0-129-39 kubenswrapper[2575]: I0707 17:41:40.897562 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/080f0a04-d8b0-45c9-98f7-db87cbc18520-service-ca-bundle\") pod \"router-default-5d884d4557-nd85p\" (UID: \"080f0a04-d8b0-45c9-98f7-db87cbc18520\") " pod="openshift-ingress/router-default-5d884d4557-nd85p" Jul 07 17:41:40.899061 ip-10-0-129-39 kubenswrapper[2575]: I0707 17:41:40.899037 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/080f0a04-d8b0-45c9-98f7-db87cbc18520-metrics-certs\") pod \"router-default-5d884d4557-nd85p\" (UID: \"080f0a04-d8b0-45c9-98f7-db87cbc18520\") " pod="openshift-ingress/router-default-5d884d4557-nd85p" Jul 07 17:41:40.911953 ip-10-0-129-39 kubenswrapper[2575]: I0707 17:41:40.911929 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-dockercfg-6vm4z\"" Jul 07 17:41:40.920156 ip-10-0-129-39 kubenswrapper[2575]: I0707 17:41:40.920139 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/router-default-5d884d4557-nd85p" Jul 07 17:41:40.957343 ip-10-0-129-39 kubenswrapper[2575]: I0707 17:41:40.957318 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/cluster-monitoring-operator-64786d9cbb-4fqsh"] Jul 07 17:41:40.960013 ip-10-0-129-39 kubenswrapper[2575]: W0707 17:41:40.959990 2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbab28ccd_f54a_4691_add4_8c665b4e1124.slice/crio-56b88996071452e65467efcf2ec7308906e8af34ab71aafdcad34bbf813bae33 WatchSource:0}: Error finding container 56b88996071452e65467efcf2ec7308906e8af34ab71aafdcad34bbf813bae33: Status 404 returned error can't find the container with id 56b88996071452e65467efcf2ec7308906e8af34ab71aafdcad34bbf813bae33 Jul 07 17:41:41.062882 ip-10-0-129-39 kubenswrapper[2575]: I0707 17:41:41.062821 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/router-default-5d884d4557-nd85p"] Jul 07 17:41:41.065645 ip-10-0-129-39 kubenswrapper[2575]: W0707 17:41:41.065621 2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod080f0a04_d8b0_45c9_98f7_db87cbc18520.slice/crio-4e6766859512ede41a9348fa8197fec7c93d881480728626efa67bc5a5960046 WatchSource:0}: Error finding container 4e6766859512ede41a9348fa8197fec7c93d881480728626efa67bc5a5960046: Status 404 returned error can't find the container with id 4e6766859512ede41a9348fa8197fec7c93d881480728626efa67bc5a5960046 Jul 07 17:41:41.271917 ip-10-0-129-39 kubenswrapper[2575]: I0707 17:41:41.271892 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-qgbk5" Jul 07 17:41:41.356410 ip-10-0-129-39 kubenswrapper[2575]: I0707 17:41:41.356374 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/router-default-5d884d4557-nd85p" event={"ID":"080f0a04-d8b0-45c9-98f7-db87cbc18520","Type":"ContainerStarted","Data":"c25d25ea31dc39f71af49bf44a3200c3a47b5bd0856ba5715d9bd60a0a7898be"} Jul 07 17:41:41.356608 ip-10-0-129-39 kubenswrapper[2575]: I0707 17:41:41.356430 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/router-default-5d884d4557-nd85p" event={"ID":"080f0a04-d8b0-45c9-98f7-db87cbc18520","Type":"ContainerStarted","Data":"4e6766859512ede41a9348fa8197fec7c93d881480728626efa67bc5a5960046"} Jul 07 17:41:41.357558 ip-10-0-129-39 kubenswrapper[2575]: I0707 17:41:41.357536 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/cluster-monitoring-operator-64786d9cbb-4fqsh" event={"ID":"bab28ccd-f54a-4691-add4-8c665b4e1124","Type":"ContainerStarted","Data":"56b88996071452e65467efcf2ec7308906e8af34ab71aafdcad34bbf813bae33"} Jul 07 17:41:41.375676 ip-10-0-129-39 kubenswrapper[2575]: I0707 17:41:41.375639 2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/router-default-5d884d4557-nd85p" podStartSLOduration=17.375626334 podStartE2EDuration="17.375626334s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-07 17:41:24 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 17:41:41.374746719 +0000 UTC m=+61.880553385" watchObservedRunningTime="2026-07-07 17:41:41.375626334 +0000 UTC m=+61.881433000" Jul 07 17:41:41.920543 ip-10-0-129-39 kubenswrapper[2575]: I0707 17:41:41.920496 2575 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/router-default-5d884d4557-nd85p" Jul 07 17:41:41.922930 ip-10-0-129-39 kubenswrapper[2575]: I0707 17:41:41.922907 2575 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/router-default-5d884d4557-nd85p" Jul 07 17:41:42.366125 ip-10-0-129-39 kubenswrapper[2575]: I0707 17:41:42.366094 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/router-default-5d884d4557-nd85p" Jul 07 17:41:42.367434 ip-10-0-129-39 kubenswrapper[2575]: I0707 17:41:42.367409 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/router-default-5d884d4557-nd85p" Jul 07 17:41:43.370782 ip-10-0-129-39 kubenswrapper[2575]: I0707 17:41:43.370740 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/cluster-monitoring-operator-64786d9cbb-4fqsh" event={"ID":"bab28ccd-f54a-4691-add4-8c665b4e1124","Type":"ContainerStarted","Data":"25f0604bc3af4c677484078a73e5586614a7c3d4d5bbf886c39d5d36fac2cf06"} Jul 07 17:41:43.390213 ip-10-0-129-39 kubenswrapper[2575]: I0707 17:41:43.390163 2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/cluster-monitoring-operator-64786d9cbb-4fqsh" podStartSLOduration=17.206250268 podStartE2EDuration="19.390144487s" totalImagesPullingTime="2.183894219s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-07 17:41:24 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 17:41:43.389607995 +0000 UTC m=+63.895414661" watchObservedRunningTime="2026-07-07 17:41:43.390144487 +0000 UTC m=+63.895951155" Jul 07 17:41:45.634745 ip-10-0-129-39 kubenswrapper[2575]: I0707 17:41:45.634704 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/0c1d1c87-6251-49e9-ba58-47cb22ee80ca-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-fmk64\" (UID: \"0c1d1c87-6251-49e9-ba58-47cb22ee80ca\") " pod="openshift-insights/insights-runtime-extractor-fmk64" Jul 07 17:41:45.634745 ip-10-0-129-39 kubenswrapper[2575]: I0707 17:41:45.634749 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/b2656ecc-1db3-4850-b2f6-58b0a14eb683-metrics-tls\") pod \"dns-default-b2mnn\" (UID: \"b2656ecc-1db3-4850-b2f6-58b0a14eb683\") " pod="openshift-dns/dns-default-b2mnn" Jul 07 17:41:45.635183 ip-10-0-129-39 kubenswrapper[2575]: I0707 17:41:45.634776 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/0c1d1c87-6251-49e9-ba58-47cb22ee80ca-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-fmk64\" (UID: \"0c1d1c87-6251-49e9-ba58-47cb22ee80ca\") " pod="openshift-insights/insights-runtime-extractor-fmk64" Jul 07 17:41:45.635252 ip-10-0-129-39 kubenswrapper[2575]: I0707 17:41:45.635232 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/0c1d1c87-6251-49e9-ba58-47cb22ee80ca-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-fmk64\" (UID: \"0c1d1c87-6251-49e9-ba58-47cb22ee80ca\") " pod="openshift-insights/insights-runtime-extractor-fmk64" Jul 07 17:41:45.637195 ip-10-0-129-39 kubenswrapper[2575]: I0707 17:41:45.637174 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/0c1d1c87-6251-49e9-ba58-47cb22ee80ca-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-fmk64\" (UID: \"0c1d1c87-6251-49e9-ba58-47cb22ee80ca\") " pod="openshift-insights/insights-runtime-extractor-fmk64" Jul 07 17:41:45.637267 ip-10-0-129-39 kubenswrapper[2575]: I0707 17:41:45.637230 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/b2656ecc-1db3-4850-b2f6-58b0a14eb683-metrics-tls\") pod \"dns-default-b2mnn\" (UID: \"b2656ecc-1db3-4850-b2f6-58b0a14eb683\") " pod="openshift-dns/dns-default-b2mnn" Jul 07 17:41:45.735139 ip-10-0-129-39 kubenswrapper[2575]: I0707 17:41:45.735107 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/8344b558-69d0-43b3-9aac-0cea0d5379f3-original-pull-secret\") pod \"global-pull-secret-syncer-76r79\" (UID: \"8344b558-69d0-43b3-9aac-0cea0d5379f3\") " pod="kube-system/global-pull-secret-syncer-76r79" Jul 07 17:41:45.735139 ip-10-0-129-39 kubenswrapper[2575]: I0707 17:41:45.735120 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-6sssg\"" Jul 07 17:41:45.737136 ip-10-0-129-39 kubenswrapper[2575]: I0707 17:41:45.737112 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\"" Jul 07 17:41:45.742761 ip-10-0-129-39 kubenswrapper[2575]: I0707 17:41:45.742746 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-fmk64" Jul 07 17:41:45.747542 ip-10-0-129-39 kubenswrapper[2575]: I0707 17:41:45.747524 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/8344b558-69d0-43b3-9aac-0cea0d5379f3-original-pull-secret\") pod \"global-pull-secret-syncer-76r79\" (UID: \"8344b558-69d0-43b3-9aac-0cea0d5379f3\") " pod="kube-system/global-pull-secret-syncer-76r79" Jul 07 17:41:45.836096 ip-10-0-129-39 kubenswrapper[2575]: I0707 17:41:45.836064 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d587896d-9156-4b75-aa70-27962a977051-metrics-certs\") pod \"network-metrics-daemon-gz97j\" (UID: \"d587896d-9156-4b75-aa70-27962a977051\") " pod="openshift-multus/network-metrics-daemon-gz97j" Jul 07 17:41:45.838569 ip-10-0-129-39 kubenswrapper[2575]: I0707 17:41:45.838496 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\"" Jul 07 17:41:45.848604 ip-10-0-129-39 kubenswrapper[2575]: I0707 17:41:45.848579 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d587896d-9156-4b75-aa70-27962a977051-metrics-certs\") pod \"network-metrics-daemon-gz97j\" (UID: \"d587896d-9156-4b75-aa70-27962a977051\") " pod="openshift-multus/network-metrics-daemon-gz97j" Jul 07 17:41:45.865624 ip-10-0-129-39 kubenswrapper[2575]: I0707 17:41:45.865601 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-fmk64"] Jul 07 17:41:45.866322 ip-10-0-129-39 kubenswrapper[2575]: I0707 17:41:45.866309 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-gg8rv\"" Jul 07 17:41:45.868681 ip-10-0-129-39 kubenswrapper[2575]: W0707 17:41:45.868660 2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0c1d1c87_6251_49e9_ba58_47cb22ee80ca.slice/crio-d61aea2b0c945994d17b0b1fd9229ca1683b80a330955fec0207bce4c9e11d62 WatchSource:0}: Error finding container d61aea2b0c945994d17b0b1fd9229ca1683b80a330955fec0207bce4c9e11d62: Status 404 returned error can't find the container with id d61aea2b0c945994d17b0b1fd9229ca1683b80a330955fec0207bce4c9e11d62 Jul 07 17:41:45.874250 ip-10-0-129-39 kubenswrapper[2575]: I0707 17:41:45.874235 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-b2mnn" Jul 07 17:41:45.879628 ip-10-0-129-39 kubenswrapper[2575]: I0707 17:41:45.879612 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-76r79" Jul 07 17:41:45.896450 ip-10-0-129-39 kubenswrapper[2575]: I0707 17:41:45.896428 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-92crh\"" Jul 07 17:41:45.904677 ip-10-0-129-39 kubenswrapper[2575]: I0707 17:41:45.904643 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-gz97j" Jul 07 17:41:45.936927 ip-10-0-129-39 kubenswrapper[2575]: I0707 17:41:45.936825 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-78bqt\" (UniqueName: \"kubernetes.io/projected/5d33c1cb-fcda-490d-9694-b1d1eb16b8da-kube-api-access-78bqt\") pod \"network-check-target-vzf4m\" (UID: \"5d33c1cb-fcda-490d-9694-b1d1eb16b8da\") " pod="openshift-network-diagnostics/network-check-target-vzf4m" Jul 07 17:41:45.943235 ip-10-0-129-39 kubenswrapper[2575]: I0707 17:41:45.943210 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-78bqt\" (UniqueName: \"kubernetes.io/projected/5d33c1cb-fcda-490d-9694-b1d1eb16b8da-kube-api-access-78bqt\") pod \"network-check-target-vzf4m\" (UID: \"5d33c1cb-fcda-490d-9694-b1d1eb16b8da\") " pod="openshift-network-diagnostics/network-check-target-vzf4m" Jul 07 17:41:46.017637 ip-10-0-129-39 kubenswrapper[2575]: I0707 17:41:46.017604 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-b2mnn"] Jul 07 17:41:46.020280 ip-10-0-129-39 kubenswrapper[2575]: W0707 17:41:46.020257 2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb2656ecc_1db3_4850_b2f6_58b0a14eb683.slice/crio-d394e3e2368a2e79eff2c5a6cf39ea8358296af2e716280a1fe32016bf83b6f8 WatchSource:0}: Error finding container d394e3e2368a2e79eff2c5a6cf39ea8358296af2e716280a1fe32016bf83b6f8: Status 404 returned error can't find the container with id d394e3e2368a2e79eff2c5a6cf39ea8358296af2e716280a1fe32016bf83b6f8 Jul 07 17:41:46.028044 ip-10-0-129-39 kubenswrapper[2575]: I0707 17:41:46.028017 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-76r79"] Jul 07 17:41:46.030994 ip-10-0-129-39 kubenswrapper[2575]: W0707 17:41:46.030973 2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8344b558_69d0_43b3_9aac_0cea0d5379f3.slice/crio-291c6d9c07763362b979998cdaeca4a0d0f52e6481bdf9c303dc70078f962dc5 WatchSource:0}: Error finding container 291c6d9c07763362b979998cdaeca4a0d0f52e6481bdf9c303dc70078f962dc5: Status 404 returned error can't find the container with id 291c6d9c07763362b979998cdaeca4a0d0f52e6481bdf9c303dc70078f962dc5 Jul 07 17:41:46.055213 ip-10-0-129-39 kubenswrapper[2575]: I0707 17:41:46.055178 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-gz97j"] Jul 07 17:41:46.060230 ip-10-0-129-39 kubenswrapper[2575]: W0707 17:41:46.060210 2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd587896d_9156_4b75_aa70_27962a977051.slice/crio-ede0b4e06db2f0fed000cd7f360b2840bc96541fc5ccf3ce2976100d2809653d WatchSource:0}: Error finding container ede0b4e06db2f0fed000cd7f360b2840bc96541fc5ccf3ce2976100d2809653d: Status 404 returned error can't find the container with id ede0b4e06db2f0fed000cd7f360b2840bc96541fc5ccf3ce2976100d2809653d Jul 07 17:41:46.190296 ip-10-0-129-39 kubenswrapper[2575]: I0707 17:41:46.190240 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-njwcl\"" Jul 07 17:41:46.198105 ip-10-0-129-39 kubenswrapper[2575]: I0707 17:41:46.198086 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-vzf4m" Jul 07 17:41:46.313364 ip-10-0-129-39 kubenswrapper[2575]: I0707 17:41:46.313335 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-vzf4m"] Jul 07 17:41:46.317402 ip-10-0-129-39 kubenswrapper[2575]: W0707 17:41:46.317372 2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5d33c1cb_fcda_490d_9694_b1d1eb16b8da.slice/crio-48fd34dfa34663913ee241c6a32ff952e349e69d5c6697cddc8f14c539948058 WatchSource:0}: Error finding container 48fd34dfa34663913ee241c6a32ff952e349e69d5c6697cddc8f14c539948058: Status 404 returned error can't find the container with id 48fd34dfa34663913ee241c6a32ff952e349e69d5c6697cddc8f14c539948058 Jul 07 17:41:46.386660 ip-10-0-129-39 kubenswrapper[2575]: I0707 17:41:46.386624 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-gz97j" event={"ID":"d587896d-9156-4b75-aa70-27962a977051","Type":"ContainerStarted","Data":"ede0b4e06db2f0fed000cd7f360b2840bc96541fc5ccf3ce2976100d2809653d"} Jul 07 17:41:46.392566 ip-10-0-129-39 kubenswrapper[2575]: I0707 17:41:46.390963 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-fmk64" event={"ID":"0c1d1c87-6251-49e9-ba58-47cb22ee80ca","Type":"ContainerStarted","Data":"d61aea2b0c945994d17b0b1fd9229ca1683b80a330955fec0207bce4c9e11d62"} Jul 07 17:41:46.394787 ip-10-0-129-39 kubenswrapper[2575]: I0707 17:41:46.393231 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-vzf4m" event={"ID":"5d33c1cb-fcda-490d-9694-b1d1eb16b8da","Type":"ContainerStarted","Data":"48fd34dfa34663913ee241c6a32ff952e349e69d5c6697cddc8f14c539948058"} Jul 07 17:41:46.394787 ip-10-0-129-39 kubenswrapper[2575]: I0707 17:41:46.393925 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-vzf4m" Jul 07 17:41:46.395967 ip-10-0-129-39 kubenswrapper[2575]: I0707 17:41:46.395945 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-b2mnn" event={"ID":"b2656ecc-1db3-4850-b2f6-58b0a14eb683","Type":"ContainerStarted","Data":"d394e3e2368a2e79eff2c5a6cf39ea8358296af2e716280a1fe32016bf83b6f8"} Jul 07 17:41:46.399360 ip-10-0-129-39 kubenswrapper[2575]: I0707 17:41:46.399335 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-76r79" event={"ID":"8344b558-69d0-43b3-9aac-0cea0d5379f3","Type":"ContainerStarted","Data":"291c6d9c07763362b979998cdaeca4a0d0f52e6481bdf9c303dc70078f962dc5"} Jul 07 17:41:46.409004 ip-10-0-129-39 kubenswrapper[2575]: I0707 17:41:46.408892 2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-target-vzf4m" podStartSLOduration=66.408876568 podStartE2EDuration="1m6.408876568s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-07 17:40:40 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 17:41:46.408652478 +0000 UTC m=+66.914459145" watchObservedRunningTime="2026-07-07 17:41:46.408876568 +0000 UTC m=+66.914683229" Jul 07 17:41:47.406496 ip-10-0-129-39 kubenswrapper[2575]: I0707 17:41:47.405978 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-vzf4m" event={"ID":"5d33c1cb-fcda-490d-9694-b1d1eb16b8da","Type":"ContainerStarted","Data":"bc5f6d752cae36ab1e30cd5c0a3a7f5288e6eeeb89a3f6395d35e2e0a2649aa4"} Jul 07 17:41:50.518019 ip-10-0-129-39 kubenswrapper[2575]: I0707 17:41:50.517986 2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["open-cluster-management-agent-addon/cluster-proxy-proxy-agent-b5997f658-vr99g"] Jul 07 17:41:50.525089 ip-10-0-129-39 kubenswrapper[2575]: I0707 17:41:50.523580 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-b5997f658-vr99g" Jul 07 17:41:50.526934 ip-10-0-129-39 kubenswrapper[2575]: I0707 17:41:50.526871 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"cluster-proxy-ca\"" Jul 07 17:41:50.527045 ip-10-0-129-39 kubenswrapper[2575]: I0707 17:41:50.526944 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"cluster-proxy-open-cluster-management.io-proxy-agent-signer-client-cert\"" Jul 07 17:41:50.527104 ip-10-0-129-39 kubenswrapper[2575]: I0707 17:41:50.527085 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"open-cluster-management-image-pull-credentials\"" Jul 07 17:41:50.527195 ip-10-0-129-39 kubenswrapper[2575]: I0707 17:41:50.527170 2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"open-cluster-management-agent-addon\"/\"kube-root-ca.crt\"" Jul 07 17:41:50.527826 ip-10-0-129-39 kubenswrapper[2575]: I0707 17:41:50.527808 2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"open-cluster-management-agent-addon\"/\"openshift-service-ca.crt\"" Jul 07 17:41:50.527922 ip-10-0-129-39 kubenswrapper[2575]: I0707 17:41:50.527810 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"cluster-proxy-service-proxy-server-certificates\"" Jul 07 17:41:50.528079 ip-10-0-129-39 kubenswrapper[2575]: I0707 17:41:50.528060 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"cluster-proxy-hub-kubeconfig\"" Jul 07 17:41:50.536857 ip-10-0-129-39 kubenswrapper[2575]: I0707 17:41:50.536838 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/cluster-proxy-proxy-agent-b5997f658-vr99g"] Jul 07 17:41:50.575381 ip-10-0-129-39 kubenswrapper[2575]: I0707 17:41:50.575355 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/9dbed709-3d53-4bfc-9350-e92e8b8144ca-hub-kubeconfig\") pod \"cluster-proxy-proxy-agent-b5997f658-vr99g\" (UID: \"9dbed709-3d53-4bfc-9350-e92e8b8144ca\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-b5997f658-vr99g" Jul 07 17:41:50.575527 ip-10-0-129-39 kubenswrapper[2575]: I0707 17:41:50.575397 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-proxy-server-cert\" (UniqueName: \"kubernetes.io/secret/9dbed709-3d53-4bfc-9350-e92e8b8144ca-service-proxy-server-cert\") pod \"cluster-proxy-proxy-agent-b5997f658-vr99g\" (UID: \"9dbed709-3d53-4bfc-9350-e92e8b8144ca\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-b5997f658-vr99g" Jul 07 17:41:50.575527 ip-10-0-129-39 kubenswrapper[2575]: I0707 17:41:50.575434 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"additional-service-ca\" (UniqueName: \"kubernetes.io/configmap/9dbed709-3d53-4bfc-9350-e92e8b8144ca-additional-service-ca\") pod \"cluster-proxy-proxy-agent-b5997f658-vr99g\" (UID: \"9dbed709-3d53-4bfc-9350-e92e8b8144ca\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-b5997f658-vr99g" Jul 07 17:41:50.575527 ip-10-0-129-39 kubenswrapper[2575]: I0707 17:41:50.575467 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca\" (UniqueName: \"kubernetes.io/secret/9dbed709-3d53-4bfc-9350-e92e8b8144ca-ca\") pod \"cluster-proxy-proxy-agent-b5997f658-vr99g\" (UID: \"9dbed709-3d53-4bfc-9350-e92e8b8144ca\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-b5997f658-vr99g" Jul 07 17:41:50.575527 ip-10-0-129-39 kubenswrapper[2575]: I0707 17:41:50.575517 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xrmz6\" (UniqueName: \"kubernetes.io/projected/9dbed709-3d53-4bfc-9350-e92e8b8144ca-kube-api-access-xrmz6\") pod \"cluster-proxy-proxy-agent-b5997f658-vr99g\" (UID: \"9dbed709-3d53-4bfc-9350-e92e8b8144ca\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-b5997f658-vr99g" Jul 07 17:41:50.575700 ip-10-0-129-39 kubenswrapper[2575]: I0707 17:41:50.575597 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hub\" (UniqueName: \"kubernetes.io/secret/9dbed709-3d53-4bfc-9350-e92e8b8144ca-hub\") pod \"cluster-proxy-proxy-agent-b5997f658-vr99g\" (UID: \"9dbed709-3d53-4bfc-9350-e92e8b8144ca\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-b5997f658-vr99g" Jul 07 17:41:50.676756 ip-10-0-129-39 kubenswrapper[2575]: I0707 17:41:50.676719 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xrmz6\" (UniqueName: \"kubernetes.io/projected/9dbed709-3d53-4bfc-9350-e92e8b8144ca-kube-api-access-xrmz6\") pod \"cluster-proxy-proxy-agent-b5997f658-vr99g\" (UID: \"9dbed709-3d53-4bfc-9350-e92e8b8144ca\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-b5997f658-vr99g" Jul 07 17:41:50.676925 ip-10-0-129-39 kubenswrapper[2575]: I0707 17:41:50.676782 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hub\" (UniqueName: \"kubernetes.io/secret/9dbed709-3d53-4bfc-9350-e92e8b8144ca-hub\") pod \"cluster-proxy-proxy-agent-b5997f658-vr99g\" (UID: \"9dbed709-3d53-4bfc-9350-e92e8b8144ca\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-b5997f658-vr99g" Jul 07 17:41:50.676925 ip-10-0-129-39 kubenswrapper[2575]: I0707 17:41:50.676834 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/9dbed709-3d53-4bfc-9350-e92e8b8144ca-hub-kubeconfig\") pod \"cluster-proxy-proxy-agent-b5997f658-vr99g\" (UID: \"9dbed709-3d53-4bfc-9350-e92e8b8144ca\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-b5997f658-vr99g" Jul 07 17:41:50.676925 ip-10-0-129-39 kubenswrapper[2575]: I0707 17:41:50.676863 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-proxy-server-cert\" (UniqueName: \"kubernetes.io/secret/9dbed709-3d53-4bfc-9350-e92e8b8144ca-service-proxy-server-cert\") pod \"cluster-proxy-proxy-agent-b5997f658-vr99g\" (UID: \"9dbed709-3d53-4bfc-9350-e92e8b8144ca\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-b5997f658-vr99g" Jul 07 17:41:50.676925 ip-10-0-129-39 kubenswrapper[2575]: I0707 17:41:50.676895 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"additional-service-ca\" (UniqueName: \"kubernetes.io/configmap/9dbed709-3d53-4bfc-9350-e92e8b8144ca-additional-service-ca\") pod \"cluster-proxy-proxy-agent-b5997f658-vr99g\" (UID: \"9dbed709-3d53-4bfc-9350-e92e8b8144ca\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-b5997f658-vr99g" Jul 07 17:41:50.677123 ip-10-0-129-39 kubenswrapper[2575]: I0707 17:41:50.676930 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca\" (UniqueName: \"kubernetes.io/secret/9dbed709-3d53-4bfc-9350-e92e8b8144ca-ca\") pod \"cluster-proxy-proxy-agent-b5997f658-vr99g\" (UID: \"9dbed709-3d53-4bfc-9350-e92e8b8144ca\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-b5997f658-vr99g" Jul 07 17:41:50.677784 ip-10-0-129-39 kubenswrapper[2575]: I0707 17:41:50.677754 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"additional-service-ca\" (UniqueName: \"kubernetes.io/configmap/9dbed709-3d53-4bfc-9350-e92e8b8144ca-additional-service-ca\") pod \"cluster-proxy-proxy-agent-b5997f658-vr99g\" (UID: \"9dbed709-3d53-4bfc-9350-e92e8b8144ca\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-b5997f658-vr99g" Jul 07 17:41:50.679850 ip-10-0-129-39 kubenswrapper[2575]: I0707 17:41:50.679827 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hub\" (UniqueName: \"kubernetes.io/secret/9dbed709-3d53-4bfc-9350-e92e8b8144ca-hub\") pod \"cluster-proxy-proxy-agent-b5997f658-vr99g\" (UID: \"9dbed709-3d53-4bfc-9350-e92e8b8144ca\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-b5997f658-vr99g" Jul 07 17:41:50.679958 ip-10-0-129-39 kubenswrapper[2575]: I0707 17:41:50.679904 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ca\" (UniqueName: \"kubernetes.io/secret/9dbed709-3d53-4bfc-9350-e92e8b8144ca-ca\") pod \"cluster-proxy-proxy-agent-b5997f658-vr99g\" (UID: \"9dbed709-3d53-4bfc-9350-e92e8b8144ca\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-b5997f658-vr99g" Jul 07 17:41:50.679958 ip-10-0-129-39 kubenswrapper[2575]: I0707 17:41:50.679914 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-proxy-server-cert\" (UniqueName: \"kubernetes.io/secret/9dbed709-3d53-4bfc-9350-e92e8b8144ca-service-proxy-server-cert\") pod \"cluster-proxy-proxy-agent-b5997f658-vr99g\" (UID: \"9dbed709-3d53-4bfc-9350-e92e8b8144ca\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-b5997f658-vr99g" Jul 07 17:41:50.680595 ip-10-0-129-39 kubenswrapper[2575]: I0707 17:41:50.680572 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/9dbed709-3d53-4bfc-9350-e92e8b8144ca-hub-kubeconfig\") pod \"cluster-proxy-proxy-agent-b5997f658-vr99g\" (UID: \"9dbed709-3d53-4bfc-9350-e92e8b8144ca\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-b5997f658-vr99g" Jul 07 17:41:50.688440 ip-10-0-129-39 kubenswrapper[2575]: I0707 17:41:50.688415 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xrmz6\" (UniqueName: \"kubernetes.io/projected/9dbed709-3d53-4bfc-9350-e92e8b8144ca-kube-api-access-xrmz6\") pod \"cluster-proxy-proxy-agent-b5997f658-vr99g\" (UID: \"9dbed709-3d53-4bfc-9350-e92e8b8144ca\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-b5997f658-vr99g" Jul 07 17:41:50.846385 ip-10-0-129-39 kubenswrapper[2575]: I0707 17:41:50.846358 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-b5997f658-vr99g" Jul 07 17:41:52.719241 ip-10-0-129-39 kubenswrapper[2575]: I0707 17:41:52.719182 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/cluster-proxy-proxy-agent-b5997f658-vr99g"] Jul 07 17:41:52.729257 ip-10-0-129-39 kubenswrapper[2575]: W0707 17:41:52.729222 2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod9dbed709_3d53_4bfc_9350_e92e8b8144ca.slice/crio-de1fd6fe77fe38bf984b89c55f91cc79bcc3b2668f105f63234fdda9078d3209 WatchSource:0}: Error finding container de1fd6fe77fe38bf984b89c55f91cc79bcc3b2668f105f63234fdda9078d3209: Status 404 returned error can't find the container with id de1fd6fe77fe38bf984b89c55f91cc79bcc3b2668f105f63234fdda9078d3209 Jul 07 17:41:53.430048 ip-10-0-129-39 kubenswrapper[2575]: I0707 17:41:53.430021 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-76r79" event={"ID":"8344b558-69d0-43b3-9aac-0cea0d5379f3","Type":"ContainerStarted","Data":"c914347bac79f0176e1e1ed6be693f1d220f60777e59d2d6c0c39385aead3744"} Jul 07 17:41:53.431206 ip-10-0-129-39 kubenswrapper[2575]: I0707 17:41:53.431182 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-b5997f658-vr99g" event={"ID":"9dbed709-3d53-4bfc-9350-e92e8b8144ca","Type":"ContainerStarted","Data":"de1fd6fe77fe38bf984b89c55f91cc79bcc3b2668f105f63234fdda9078d3209"} Jul 07 17:41:53.432942 ip-10-0-129-39 kubenswrapper[2575]: I0707 17:41:53.432914 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-gz97j" event={"ID":"d587896d-9156-4b75-aa70-27962a977051","Type":"ContainerStarted","Data":"c82ae8cfec47b497d32a4022ce15d7710f3186f08b6d5e8a9b23f9647488fcb6"} Jul 07 17:41:53.433032 ip-10-0-129-39 kubenswrapper[2575]: I0707 17:41:53.432961 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-gz97j" event={"ID":"d587896d-9156-4b75-aa70-27962a977051","Type":"ContainerStarted","Data":"44471bcf8142fd165590633824854e571ef8c45e37586b6efc837dcce85e8987"} Jul 07 17:41:53.434651 ip-10-0-129-39 kubenswrapper[2575]: I0707 17:41:53.434629 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-fmk64" event={"ID":"0c1d1c87-6251-49e9-ba58-47cb22ee80ca","Type":"ContainerStarted","Data":"8724c9f3094bf95c7b59e7aca05440b4bbdbc1742a3152e7f11baa729cd8e99c"} Jul 07 17:41:53.434840 ip-10-0-129-39 kubenswrapper[2575]: I0707 17:41:53.434677 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-fmk64" event={"ID":"0c1d1c87-6251-49e9-ba58-47cb22ee80ca","Type":"ContainerStarted","Data":"b32dbcb68da853684cb7e6dc4dcb4d7103ecad42b1f0c4763ff97b9298597356"} Jul 07 17:41:53.436374 ip-10-0-129-39 kubenswrapper[2575]: I0707 17:41:53.436356 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-b2mnn" event={"ID":"b2656ecc-1db3-4850-b2f6-58b0a14eb683","Type":"ContainerStarted","Data":"e87d37fcc78dfa32cc5caac109a3fdacf3a56e7e5de4cf412754e707a1f4e850"} Jul 07 17:41:53.436440 ip-10-0-129-39 kubenswrapper[2575]: I0707 17:41:53.436390 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-b2mnn" event={"ID":"b2656ecc-1db3-4850-b2f6-58b0a14eb683","Type":"ContainerStarted","Data":"ee99e1b3f46cdd1916d0f3bde8bdf1c712137936ef8e9bef2eebdc66c7df553b"} Jul 07 17:41:53.436523 ip-10-0-129-39 kubenswrapper[2575]: I0707 17:41:53.436494 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-dns/dns-default-b2mnn" Jul 07 17:41:53.451095 ip-10-0-129-39 kubenswrapper[2575]: I0707 17:41:53.451053 2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/global-pull-secret-syncer-76r79" podStartSLOduration=66.885519896 podStartE2EDuration="1m13.45103826s" totalImagesPullingTime="6.565518364s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-07 17:40:40 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 17:41:53.449859194 +0000 UTC m=+73.955665859" watchObservedRunningTime="2026-07-07 17:41:53.45103826 +0000 UTC m=+73.956844926" Jul 07 17:41:53.475534 ip-10-0-129-39 kubenswrapper[2575]: I0707 17:41:53.475457 2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/network-metrics-daemon-gz97j" podStartSLOduration=66.972127995 podStartE2EDuration="1m13.475439844s" totalImagesPullingTime="6.503311849s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-07 17:40:40 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 17:41:53.4694045 +0000 UTC m=+73.975211166" watchObservedRunningTime="2026-07-07 17:41:53.475439844 +0000 UTC m=+73.981246515" Jul 07 17:41:56.450251 ip-10-0-129-39 kubenswrapper[2575]: I0707 17:41:56.450217 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-fmk64" event={"ID":"0c1d1c87-6251-49e9-ba58-47cb22ee80ca","Type":"ContainerStarted","Data":"aa16f426930b6ae8b218a9e18fd0107e39514eb9c6b0b127d1c1a383a58fb20e"} Jul 07 17:41:56.470178 ip-10-0-129-39 kubenswrapper[2575]: I0707 17:41:56.470134 2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/dns-default-b2mnn" podStartSLOduration=36.923131261 podStartE2EDuration="43.470119122s" totalImagesPullingTime="6.546987861s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-07 17:41:13 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 17:41:53.495094813 +0000 UTC m=+74.000901478" watchObservedRunningTime="2026-07-07 17:41:56.470119122 +0000 UTC m=+76.975925787" Jul 07 17:41:56.470528 ip-10-0-129-39 kubenswrapper[2575]: I0707 17:41:56.470490 2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-insights/insights-runtime-extractor-fmk64" podStartSLOduration=17.651941784 podStartE2EDuration="27.470482944s" totalImagesPullingTime="9.81854116s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-07 17:41:29 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 17:41:56.468852696 +0000 UTC m=+76.974659362" watchObservedRunningTime="2026-07-07 17:41:56.470482944 +0000 UTC m=+76.976289609" Jul 07 17:42:00.128284 ip-10-0-129-39 kubenswrapper[2575]: I0707 17:42:00.128252 2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/node-exporter-nbqxr"] Jul 07 17:42:00.130479 ip-10-0-129-39 kubenswrapper[2575]: I0707 17:42:00.130457 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-nbqxr" Jul 07 17:42:00.134780 ip-10-0-129-39 kubenswrapper[2575]: I0707 17:42:00.134757 2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\"" Jul 07 17:42:00.134902 ip-10-0-129-39 kubenswrapper[2575]: I0707 17:42:00.134869 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\"" Jul 07 17:42:00.134965 ip-10-0-129-39 kubenswrapper[2575]: I0707 17:42:00.134869 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\"" Jul 07 17:42:00.134965 ip-10-0-129-39 kubenswrapper[2575]: I0707 17:42:00.134917 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-fcg8q\"" Jul 07 17:42:00.135588 ip-10-0-129-39 kubenswrapper[2575]: I0707 17:42:00.135571 2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\"" Jul 07 17:42:00.245429 ip-10-0-129-39 kubenswrapper[2575]: I0707 17:42:00.245391 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/ba95b45f-7620-472e-b924-83a1ed9ba25c-root\") pod \"node-exporter-nbqxr\" (UID: \"ba95b45f-7620-472e-b924-83a1ed9ba25c\") " pod="openshift-monitoring/node-exporter-nbqxr" Jul 07 17:42:00.245429 ip-10-0-129-39 kubenswrapper[2575]: I0707 17:42:00.245429 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/ba95b45f-7620-472e-b924-83a1ed9ba25c-node-exporter-tls\") pod \"node-exporter-nbqxr\" (UID: \"ba95b45f-7620-472e-b924-83a1ed9ba25c\") " pod="openshift-monitoring/node-exporter-nbqxr" Jul 07 17:42:00.245645 ip-10-0-129-39 kubenswrapper[2575]: I0707 17:42:00.245446 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dmfg2\" (UniqueName: \"kubernetes.io/projected/ba95b45f-7620-472e-b924-83a1ed9ba25c-kube-api-access-dmfg2\") pod \"node-exporter-nbqxr\" (UID: \"ba95b45f-7620-472e-b924-83a1ed9ba25c\") " pod="openshift-monitoring/node-exporter-nbqxr" Jul 07 17:42:00.245645 ip-10-0-129-39 kubenswrapper[2575]: I0707 17:42:00.245471 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/ba95b45f-7620-472e-b924-83a1ed9ba25c-node-exporter-textfile\") pod \"node-exporter-nbqxr\" (UID: \"ba95b45f-7620-472e-b924-83a1ed9ba25c\") " pod="openshift-monitoring/node-exporter-nbqxr" Jul 07 17:42:00.245645 ip-10-0-129-39 kubenswrapper[2575]: I0707 17:42:00.245536 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/ba95b45f-7620-472e-b924-83a1ed9ba25c-node-exporter-wtmp\") pod \"node-exporter-nbqxr\" (UID: \"ba95b45f-7620-472e-b924-83a1ed9ba25c\") " pod="openshift-monitoring/node-exporter-nbqxr" Jul 07 17:42:00.245645 ip-10-0-129-39 kubenswrapper[2575]: I0707 17:42:00.245587 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/ba95b45f-7620-472e-b924-83a1ed9ba25c-metrics-client-ca\") pod \"node-exporter-nbqxr\" (UID: \"ba95b45f-7620-472e-b924-83a1ed9ba25c\") " pod="openshift-monitoring/node-exporter-nbqxr" Jul 07 17:42:00.245645 ip-10-0-129-39 kubenswrapper[2575]: I0707 17:42:00.245632 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/ba95b45f-7620-472e-b924-83a1ed9ba25c-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-nbqxr\" (UID: \"ba95b45f-7620-472e-b924-83a1ed9ba25c\") " pod="openshift-monitoring/node-exporter-nbqxr" Jul 07 17:42:00.245793 ip-10-0-129-39 kubenswrapper[2575]: I0707 17:42:00.245655 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/ba95b45f-7620-472e-b924-83a1ed9ba25c-node-exporter-accelerators-collector-config\") pod \"node-exporter-nbqxr\" (UID: \"ba95b45f-7620-472e-b924-83a1ed9ba25c\") " pod="openshift-monitoring/node-exporter-nbqxr" Jul 07 17:42:00.245793 ip-10-0-129-39 kubenswrapper[2575]: I0707 17:42:00.245680 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/ba95b45f-7620-472e-b924-83a1ed9ba25c-sys\") pod \"node-exporter-nbqxr\" (UID: \"ba95b45f-7620-472e-b924-83a1ed9ba25c\") " pod="openshift-monitoring/node-exporter-nbqxr" Jul 07 17:42:00.346814 ip-10-0-129-39 kubenswrapper[2575]: I0707 17:42:00.346784 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/ba95b45f-7620-472e-b924-83a1ed9ba25c-root\") pod \"node-exporter-nbqxr\" (UID: \"ba95b45f-7620-472e-b924-83a1ed9ba25c\") " pod="openshift-monitoring/node-exporter-nbqxr" Jul 07 17:42:00.346814 ip-10-0-129-39 kubenswrapper[2575]: I0707 17:42:00.346815 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/ba95b45f-7620-472e-b924-83a1ed9ba25c-node-exporter-tls\") pod \"node-exporter-nbqxr\" (UID: \"ba95b45f-7620-472e-b924-83a1ed9ba25c\") " pod="openshift-monitoring/node-exporter-nbqxr" Jul 07 17:42:00.346980 ip-10-0-129-39 kubenswrapper[2575]: I0707 17:42:00.346837 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dmfg2\" (UniqueName: \"kubernetes.io/projected/ba95b45f-7620-472e-b924-83a1ed9ba25c-kube-api-access-dmfg2\") pod \"node-exporter-nbqxr\" (UID: \"ba95b45f-7620-472e-b924-83a1ed9ba25c\") " pod="openshift-monitoring/node-exporter-nbqxr" Jul 07 17:42:00.346980 ip-10-0-129-39 kubenswrapper[2575]: I0707 17:42:00.346864 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/ba95b45f-7620-472e-b924-83a1ed9ba25c-node-exporter-textfile\") pod \"node-exporter-nbqxr\" (UID: \"ba95b45f-7620-472e-b924-83a1ed9ba25c\") " pod="openshift-monitoring/node-exporter-nbqxr" Jul 07 17:42:00.346980 ip-10-0-129-39 kubenswrapper[2575]: I0707 17:42:00.346888 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/ba95b45f-7620-472e-b924-83a1ed9ba25c-node-exporter-wtmp\") pod \"node-exporter-nbqxr\" (UID: \"ba95b45f-7620-472e-b924-83a1ed9ba25c\") " pod="openshift-monitoring/node-exporter-nbqxr" Jul 07 17:42:00.346980 ip-10-0-129-39 kubenswrapper[2575]: I0707 17:42:00.346927 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/ba95b45f-7620-472e-b924-83a1ed9ba25c-metrics-client-ca\") pod \"node-exporter-nbqxr\" (UID: \"ba95b45f-7620-472e-b924-83a1ed9ba25c\") " pod="openshift-monitoring/node-exporter-nbqxr" Jul 07 17:42:00.346980 ip-10-0-129-39 kubenswrapper[2575]: I0707 17:42:00.346968 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/ba95b45f-7620-472e-b924-83a1ed9ba25c-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-nbqxr\" (UID: \"ba95b45f-7620-472e-b924-83a1ed9ba25c\") " pod="openshift-monitoring/node-exporter-nbqxr" Jul 07 17:42:00.346980 ip-10-0-129-39 kubenswrapper[2575]: I0707 17:42:00.346889 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"root\" (UniqueName: \"kubernetes.io/host-path/ba95b45f-7620-472e-b924-83a1ed9ba25c-root\") pod \"node-exporter-nbqxr\" (UID: \"ba95b45f-7620-472e-b924-83a1ed9ba25c\") " pod="openshift-monitoring/node-exporter-nbqxr" Jul 07 17:42:00.347325 ip-10-0-129-39 kubenswrapper[2575]: I0707 17:42:00.346998 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/ba95b45f-7620-472e-b924-83a1ed9ba25c-node-exporter-accelerators-collector-config\") pod \"node-exporter-nbqxr\" (UID: \"ba95b45f-7620-472e-b924-83a1ed9ba25c\") " pod="openshift-monitoring/node-exporter-nbqxr" Jul 07 17:42:00.347325 ip-10-0-129-39 kubenswrapper[2575]: I0707 17:42:00.347009 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/ba95b45f-7620-472e-b924-83a1ed9ba25c-node-exporter-wtmp\") pod \"node-exporter-nbqxr\" (UID: \"ba95b45f-7620-472e-b924-83a1ed9ba25c\") " pod="openshift-monitoring/node-exporter-nbqxr" Jul 07 17:42:00.347325 ip-10-0-129-39 kubenswrapper[2575]: I0707 17:42:00.347025 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/ba95b45f-7620-472e-b924-83a1ed9ba25c-sys\") pod \"node-exporter-nbqxr\" (UID: \"ba95b45f-7620-472e-b924-83a1ed9ba25c\") " pod="openshift-monitoring/node-exporter-nbqxr" Jul 07 17:42:00.347325 ip-10-0-129-39 kubenswrapper[2575]: E0707 17:42:00.347027 2575 secret.go:189] Couldn't get secret openshift-monitoring/node-exporter-tls: secret "node-exporter-tls" not found Jul 07 17:42:00.347325 ip-10-0-129-39 kubenswrapper[2575]: I0707 17:42:00.347073 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/ba95b45f-7620-472e-b924-83a1ed9ba25c-sys\") pod \"node-exporter-nbqxr\" (UID: \"ba95b45f-7620-472e-b924-83a1ed9ba25c\") " pod="openshift-monitoring/node-exporter-nbqxr" Jul 07 17:42:00.347325 ip-10-0-129-39 kubenswrapper[2575]: E0707 17:42:00.347111 2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ba95b45f-7620-472e-b924-83a1ed9ba25c-node-exporter-tls podName:ba95b45f-7620-472e-b924-83a1ed9ba25c nodeName:}" failed. No retries permitted until 2026-07-07 17:42:00.847091137 +0000 UTC m=+81.352897783 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "node-exporter-tls" (UniqueName: "kubernetes.io/secret/ba95b45f-7620-472e-b924-83a1ed9ba25c-node-exporter-tls") pod "node-exporter-nbqxr" (UID: "ba95b45f-7620-472e-b924-83a1ed9ba25c") : secret "node-exporter-tls" not found Jul 07 17:42:00.347325 ip-10-0-129-39 kubenswrapper[2575]: I0707 17:42:00.347205 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/ba95b45f-7620-472e-b924-83a1ed9ba25c-node-exporter-textfile\") pod \"node-exporter-nbqxr\" (UID: \"ba95b45f-7620-472e-b924-83a1ed9ba25c\") " pod="openshift-monitoring/node-exporter-nbqxr" Jul 07 17:42:00.347714 ip-10-0-129-39 kubenswrapper[2575]: I0707 17:42:00.347480 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/ba95b45f-7620-472e-b924-83a1ed9ba25c-node-exporter-accelerators-collector-config\") pod \"node-exporter-nbqxr\" (UID: \"ba95b45f-7620-472e-b924-83a1ed9ba25c\") " pod="openshift-monitoring/node-exporter-nbqxr" Jul 07 17:42:00.347714 ip-10-0-129-39 kubenswrapper[2575]: I0707 17:42:00.347626 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/ba95b45f-7620-472e-b924-83a1ed9ba25c-metrics-client-ca\") pod \"node-exporter-nbqxr\" (UID: \"ba95b45f-7620-472e-b924-83a1ed9ba25c\") " pod="openshift-monitoring/node-exporter-nbqxr" Jul 07 17:42:00.349235 ip-10-0-129-39 kubenswrapper[2575]: I0707 17:42:00.349214 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/ba95b45f-7620-472e-b924-83a1ed9ba25c-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-nbqxr\" (UID: \"ba95b45f-7620-472e-b924-83a1ed9ba25c\") " pod="openshift-monitoring/node-exporter-nbqxr" Jul 07 17:42:00.357991 ip-10-0-129-39 kubenswrapper[2575]: I0707 17:42:00.357966 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dmfg2\" (UniqueName: \"kubernetes.io/projected/ba95b45f-7620-472e-b924-83a1ed9ba25c-kube-api-access-dmfg2\") pod \"node-exporter-nbqxr\" (UID: \"ba95b45f-7620-472e-b924-83a1ed9ba25c\") " pod="openshift-monitoring/node-exporter-nbqxr" Jul 07 17:42:00.849833 ip-10-0-129-39 kubenswrapper[2575]: I0707 17:42:00.849787 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/ba95b45f-7620-472e-b924-83a1ed9ba25c-node-exporter-tls\") pod \"node-exporter-nbqxr\" (UID: \"ba95b45f-7620-472e-b924-83a1ed9ba25c\") " pod="openshift-monitoring/node-exporter-nbqxr" Jul 07 17:42:00.852049 ip-10-0-129-39 kubenswrapper[2575]: I0707 17:42:00.852024 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/ba95b45f-7620-472e-b924-83a1ed9ba25c-node-exporter-tls\") pod \"node-exporter-nbqxr\" (UID: \"ba95b45f-7620-472e-b924-83a1ed9ba25c\") " pod="openshift-monitoring/node-exporter-nbqxr" Jul 07 17:42:01.039588 ip-10-0-129-39 kubenswrapper[2575]: I0707 17:42:01.039561 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-nbqxr" Jul 07 17:42:01.055129 ip-10-0-129-39 kubenswrapper[2575]: W0707 17:42:01.055099 2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podba95b45f_7620_472e_b924_83a1ed9ba25c.slice/crio-cb24f2df2ebbdd301be5ad620746bc8c17b7b02a52282fea17ab446f3f4675bd WatchSource:0}: Error finding container cb24f2df2ebbdd301be5ad620746bc8c17b7b02a52282fea17ab446f3f4675bd: Status 404 returned error can't find the container with id cb24f2df2ebbdd301be5ad620746bc8c17b7b02a52282fea17ab446f3f4675bd Jul 07 17:42:01.467814 ip-10-0-129-39 kubenswrapper[2575]: I0707 17:42:01.467773 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-nbqxr" event={"ID":"ba95b45f-7620-472e-b924-83a1ed9ba25c","Type":"ContainerStarted","Data":"cb24f2df2ebbdd301be5ad620746bc8c17b7b02a52282fea17ab446f3f4675bd"} Jul 07 17:42:02.472755 ip-10-0-129-39 kubenswrapper[2575]: I0707 17:42:02.472718 2575 generic.go:356] "Generic (PLEG): container finished" podID="ba95b45f-7620-472e-b924-83a1ed9ba25c" containerID="101a97e047c8139c62e5bea5861064ead21cc64b6f50d057a395ae29b4e783e7" exitCode=0 Jul 07 17:42:02.473030 ip-10-0-129-39 kubenswrapper[2575]: I0707 17:42:02.472782 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-nbqxr" event={"ID":"ba95b45f-7620-472e-b924-83a1ed9ba25c","Type":"ContainerDied","Data":"101a97e047c8139c62e5bea5861064ead21cc64b6f50d057a395ae29b4e783e7"} Jul 07 17:42:03.442888 ip-10-0-129-39 kubenswrapper[2575]: I0707 17:42:03.442853 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-dns/dns-default-b2mnn" Jul 07 17:42:03.479088 ip-10-0-129-39 kubenswrapper[2575]: I0707 17:42:03.479059 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-nbqxr" event={"ID":"ba95b45f-7620-472e-b924-83a1ed9ba25c","Type":"ContainerStarted","Data":"0098e83a10e7f56bc539d87f54168c0c4bbcbd62253e5200b01b46a4c62a73dd"} Jul 07 17:42:03.479466 ip-10-0-129-39 kubenswrapper[2575]: I0707 17:42:03.479111 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-nbqxr" event={"ID":"ba95b45f-7620-472e-b924-83a1ed9ba25c","Type":"ContainerStarted","Data":"5b9883b14034b8b1beb8b6fae589adca82516757f55e5e1402ee7e8047b94314"} Jul 07 17:42:04.523790 ip-10-0-129-39 kubenswrapper[2575]: I0707 17:42:04.523735 2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/node-exporter-nbqxr" podStartSLOduration=3.271215733 podStartE2EDuration="4.523712785s" totalImagesPullingTime="1.219584459s" totalInitContainerRuntime="32.912593ms" isStatefulPod=true podCreationTimestamp="2026-07-07 17:42:00 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 17:42:03.502213627 +0000 UTC m=+84.008020293" watchObservedRunningTime="2026-07-07 17:42:04.523712785 +0000 UTC m=+85.029519453" Jul 07 17:42:04.524912 ip-10-0-129-39 kubenswrapper[2575]: I0707 17:42:04.524843 2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/metrics-server-7df964d657-xdrrj"] Jul 07 17:42:04.527303 ip-10-0-129-39 kubenswrapper[2575]: I0707 17:42:04.527257 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/metrics-server-7df964d657-xdrrj" Jul 07 17:42:04.531326 ip-10-0-129-39 kubenswrapper[2575]: I0707 17:42:04.531204 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-tls\"" Jul 07 17:42:04.531326 ip-10-0-129-39 kubenswrapper[2575]: I0707 17:42:04.531250 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-bgetg17lm8jvk\"" Jul 07 17:42:04.531521 ip-10-0-129-39 kubenswrapper[2575]: I0707 17:42:04.531475 2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-server-audit-profiles\"" Jul 07 17:42:04.531521 ip-10-0-129-39 kubenswrapper[2575]: I0707 17:42:04.531490 2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kubelet-serving-ca-bundle\"" Jul 07 17:42:04.531941 ip-10-0-129-39 kubenswrapper[2575]: I0707 17:42:04.531759 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-client-certs\"" Jul 07 17:42:04.531941 ip-10-0-129-39 kubenswrapper[2575]: I0707 17:42:04.531773 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-dockercfg-94922\"" Jul 07 17:42:04.543433 ip-10-0-129-39 kubenswrapper[2575]: I0707 17:42:04.543411 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/metrics-server-7df964d657-xdrrj"] Jul 07 17:42:04.679303 ip-10-0-129-39 kubenswrapper[2575]: I0707 17:42:04.679273 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-server-tls\" (UniqueName: \"kubernetes.io/secret/7406f82e-b601-4f1a-beec-dd77a9b835d4-secret-metrics-server-tls\") pod \"metrics-server-7df964d657-xdrrj\" (UID: \"7406f82e-b601-4f1a-beec-dd77a9b835d4\") " pod="openshift-monitoring/metrics-server-7df964d657-xdrrj" Jul 07 17:42:04.679549 ip-10-0-129-39 kubenswrapper[2575]: I0707 17:42:04.679311 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-server-client-certs\" (UniqueName: \"kubernetes.io/secret/7406f82e-b601-4f1a-beec-dd77a9b835d4-secret-metrics-server-client-certs\") pod \"metrics-server-7df964d657-xdrrj\" (UID: \"7406f82e-b601-4f1a-beec-dd77a9b835d4\") " pod="openshift-monitoring/metrics-server-7df964d657-xdrrj" Jul 07 17:42:04.679549 ip-10-0-129-39 kubenswrapper[2575]: I0707 17:42:04.679340 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-server-audit-profiles\" (UniqueName: \"kubernetes.io/configmap/7406f82e-b601-4f1a-beec-dd77a9b835d4-metrics-server-audit-profiles\") pod \"metrics-server-7df964d657-xdrrj\" (UID: \"7406f82e-b601-4f1a-beec-dd77a9b835d4\") " pod="openshift-monitoring/metrics-server-7df964d657-xdrrj" Jul 07 17:42:04.679549 ip-10-0-129-39 kubenswrapper[2575]: I0707 17:42:04.679428 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pl4lz\" (UniqueName: \"kubernetes.io/projected/7406f82e-b601-4f1a-beec-dd77a9b835d4-kube-api-access-pl4lz\") pod \"metrics-server-7df964d657-xdrrj\" (UID: \"7406f82e-b601-4f1a-beec-dd77a9b835d4\") " pod="openshift-monitoring/metrics-server-7df964d657-xdrrj" Jul 07 17:42:04.679549 ip-10-0-129-39 kubenswrapper[2575]: I0707 17:42:04.679478 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"audit-log\" (UniqueName: \"kubernetes.io/empty-dir/7406f82e-b601-4f1a-beec-dd77a9b835d4-audit-log\") pod \"metrics-server-7df964d657-xdrrj\" (UID: \"7406f82e-b601-4f1a-beec-dd77a9b835d4\") " pod="openshift-monitoring/metrics-server-7df964d657-xdrrj" Jul 07 17:42:04.679549 ip-10-0-129-39 kubenswrapper[2575]: I0707 17:42:04.679526 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"client-ca-bundle\" (UniqueName: \"kubernetes.io/secret/7406f82e-b601-4f1a-beec-dd77a9b835d4-client-ca-bundle\") pod \"metrics-server-7df964d657-xdrrj\" (UID: \"7406f82e-b601-4f1a-beec-dd77a9b835d4\") " pod="openshift-monitoring/metrics-server-7df964d657-xdrrj" Jul 07 17:42:04.679801 ip-10-0-129-39 kubenswrapper[2575]: I0707 17:42:04.679554 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/7406f82e-b601-4f1a-beec-dd77a9b835d4-configmap-kubelet-serving-ca-bundle\") pod \"metrics-server-7df964d657-xdrrj\" (UID: \"7406f82e-b601-4f1a-beec-dd77a9b835d4\") " pod="openshift-monitoring/metrics-server-7df964d657-xdrrj" Jul 07 17:42:04.780713 ip-10-0-129-39 kubenswrapper[2575]: I0707 17:42:04.780603 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pl4lz\" (UniqueName: \"kubernetes.io/projected/7406f82e-b601-4f1a-beec-dd77a9b835d4-kube-api-access-pl4lz\") pod \"metrics-server-7df964d657-xdrrj\" (UID: \"7406f82e-b601-4f1a-beec-dd77a9b835d4\") " pod="openshift-monitoring/metrics-server-7df964d657-xdrrj" Jul 07 17:42:04.780713 ip-10-0-129-39 kubenswrapper[2575]: I0707 17:42:04.780665 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"audit-log\" (UniqueName: \"kubernetes.io/empty-dir/7406f82e-b601-4f1a-beec-dd77a9b835d4-audit-log\") pod \"metrics-server-7df964d657-xdrrj\" (UID: \"7406f82e-b601-4f1a-beec-dd77a9b835d4\") " pod="openshift-monitoring/metrics-server-7df964d657-xdrrj" Jul 07 17:42:04.780713 ip-10-0-129-39 kubenswrapper[2575]: I0707 17:42:04.780702 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"client-ca-bundle\" (UniqueName: \"kubernetes.io/secret/7406f82e-b601-4f1a-beec-dd77a9b835d4-client-ca-bundle\") pod \"metrics-server-7df964d657-xdrrj\" (UID: \"7406f82e-b601-4f1a-beec-dd77a9b835d4\") " pod="openshift-monitoring/metrics-server-7df964d657-xdrrj" Jul 07 17:42:04.781058 ip-10-0-129-39 kubenswrapper[2575]: I0707 17:42:04.780742 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/7406f82e-b601-4f1a-beec-dd77a9b835d4-configmap-kubelet-serving-ca-bundle\") pod \"metrics-server-7df964d657-xdrrj\" (UID: \"7406f82e-b601-4f1a-beec-dd77a9b835d4\") " pod="openshift-monitoring/metrics-server-7df964d657-xdrrj" Jul 07 17:42:04.781058 ip-10-0-129-39 kubenswrapper[2575]: I0707 17:42:04.780777 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-server-tls\" (UniqueName: \"kubernetes.io/secret/7406f82e-b601-4f1a-beec-dd77a9b835d4-secret-metrics-server-tls\") pod \"metrics-server-7df964d657-xdrrj\" (UID: \"7406f82e-b601-4f1a-beec-dd77a9b835d4\") " pod="openshift-monitoring/metrics-server-7df964d657-xdrrj" Jul 07 17:42:04.781058 ip-10-0-129-39 kubenswrapper[2575]: I0707 17:42:04.780812 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-server-client-certs\" (UniqueName: \"kubernetes.io/secret/7406f82e-b601-4f1a-beec-dd77a9b835d4-secret-metrics-server-client-certs\") pod \"metrics-server-7df964d657-xdrrj\" (UID: \"7406f82e-b601-4f1a-beec-dd77a9b835d4\") " pod="openshift-monitoring/metrics-server-7df964d657-xdrrj" Jul 07 17:42:04.781058 ip-10-0-129-39 kubenswrapper[2575]: I0707 17:42:04.780841 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-server-audit-profiles\" (UniqueName: \"kubernetes.io/configmap/7406f82e-b601-4f1a-beec-dd77a9b835d4-metrics-server-audit-profiles\") pod \"metrics-server-7df964d657-xdrrj\" (UID: \"7406f82e-b601-4f1a-beec-dd77a9b835d4\") " pod="openshift-monitoring/metrics-server-7df964d657-xdrrj" Jul 07 17:42:04.781709 ip-10-0-129-39 kubenswrapper[2575]: I0707 17:42:04.781683 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/7406f82e-b601-4f1a-beec-dd77a9b835d4-configmap-kubelet-serving-ca-bundle\") pod \"metrics-server-7df964d657-xdrrj\" (UID: \"7406f82e-b601-4f1a-beec-dd77a9b835d4\") " pod="openshift-monitoring/metrics-server-7df964d657-xdrrj" Jul 07 17:42:04.782279 ip-10-0-129-39 kubenswrapper[2575]: I0707 17:42:04.782226 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-server-audit-profiles\" (UniqueName: \"kubernetes.io/configmap/7406f82e-b601-4f1a-beec-dd77a9b835d4-metrics-server-audit-profiles\") pod \"metrics-server-7df964d657-xdrrj\" (UID: \"7406f82e-b601-4f1a-beec-dd77a9b835d4\") " pod="openshift-monitoring/metrics-server-7df964d657-xdrrj" Jul 07 17:42:04.782540 ip-10-0-129-39 kubenswrapper[2575]: I0707 17:42:04.782480 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"audit-log\" (UniqueName: \"kubernetes.io/empty-dir/7406f82e-b601-4f1a-beec-dd77a9b835d4-audit-log\") pod \"metrics-server-7df964d657-xdrrj\" (UID: \"7406f82e-b601-4f1a-beec-dd77a9b835d4\") " pod="openshift-monitoring/metrics-server-7df964d657-xdrrj" Jul 07 17:42:04.784198 ip-10-0-129-39 kubenswrapper[2575]: I0707 17:42:04.784153 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-metrics-server-client-certs\" (UniqueName: \"kubernetes.io/secret/7406f82e-b601-4f1a-beec-dd77a9b835d4-secret-metrics-server-client-certs\") pod \"metrics-server-7df964d657-xdrrj\" (UID: \"7406f82e-b601-4f1a-beec-dd77a9b835d4\") " pod="openshift-monitoring/metrics-server-7df964d657-xdrrj" Jul 07 17:42:04.784296 ip-10-0-129-39 kubenswrapper[2575]: I0707 17:42:04.784220 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-metrics-server-tls\" (UniqueName: \"kubernetes.io/secret/7406f82e-b601-4f1a-beec-dd77a9b835d4-secret-metrics-server-tls\") pod \"metrics-server-7df964d657-xdrrj\" (UID: \"7406f82e-b601-4f1a-beec-dd77a9b835d4\") " pod="openshift-monitoring/metrics-server-7df964d657-xdrrj" Jul 07 17:42:04.786558 ip-10-0-129-39 kubenswrapper[2575]: I0707 17:42:04.786492 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"client-ca-bundle\" (UniqueName: \"kubernetes.io/secret/7406f82e-b601-4f1a-beec-dd77a9b835d4-client-ca-bundle\") pod \"metrics-server-7df964d657-xdrrj\" (UID: \"7406f82e-b601-4f1a-beec-dd77a9b835d4\") " pod="openshift-monitoring/metrics-server-7df964d657-xdrrj" Jul 07 17:42:04.794557 ip-10-0-129-39 kubenswrapper[2575]: I0707 17:42:04.794535 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pl4lz\" (UniqueName: \"kubernetes.io/projected/7406f82e-b601-4f1a-beec-dd77a9b835d4-kube-api-access-pl4lz\") pod \"metrics-server-7df964d657-xdrrj\" (UID: \"7406f82e-b601-4f1a-beec-dd77a9b835d4\") " pod="openshift-monitoring/metrics-server-7df964d657-xdrrj" Jul 07 17:42:04.839083 ip-10-0-129-39 kubenswrapper[2575]: I0707 17:42:04.839043 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/metrics-server-7df964d657-xdrrj" Jul 07 17:42:04.997206 ip-10-0-129-39 kubenswrapper[2575]: I0707 17:42:04.997139 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/metrics-server-7df964d657-xdrrj"] Jul 07 17:42:05.000713 ip-10-0-129-39 kubenswrapper[2575]: W0707 17:42:05.000686 2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7406f82e_b601_4f1a_beec_dd77a9b835d4.slice/crio-bc45c9ac1b7f4e580c0ed500fe301dc94337b3414c9a8c356d2c476cb4622e24 WatchSource:0}: Error finding container bc45c9ac1b7f4e580c0ed500fe301dc94337b3414c9a8c356d2c476cb4622e24: Status 404 returned error can't find the container with id bc45c9ac1b7f4e580c0ed500fe301dc94337b3414c9a8c356d2c476cb4622e24 Jul 07 17:42:05.493867 ip-10-0-129-39 kubenswrapper[2575]: I0707 17:42:05.493824 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/metrics-server-7df964d657-xdrrj" event={"ID":"7406f82e-b601-4f1a-beec-dd77a9b835d4","Type":"ContainerStarted","Data":"bc45c9ac1b7f4e580c0ed500fe301dc94337b3414c9a8c356d2c476cb4622e24"} Jul 07 17:42:08.506170 ip-10-0-129-39 kubenswrapper[2575]: I0707 17:42:08.506137 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/metrics-server-7df964d657-xdrrj" event={"ID":"7406f82e-b601-4f1a-beec-dd77a9b835d4","Type":"ContainerStarted","Data":"3eeb36249a154afe1e81f5b8e3c30f90c26a8c6bd489990060b1fd23738c0d4d"} Jul 07 17:42:08.508281 ip-10-0-129-39 kubenswrapper[2575]: I0707 17:42:08.508247 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-b5997f658-vr99g" event={"ID":"9dbed709-3d53-4bfc-9350-e92e8b8144ca","Type":"ContainerStarted","Data":"4d96b63751c1239d880b84d2c71df6f9429f68392770df832522a356654211ea"} Jul 07 17:42:08.508281 ip-10-0-129-39 kubenswrapper[2575]: I0707 17:42:08.508283 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-b5997f658-vr99g" event={"ID":"9dbed709-3d53-4bfc-9350-e92e8b8144ca","Type":"ContainerStarted","Data":"a704d2d07e97a8cdfc7be851c240d61a7666430a4c8510ffd7cbffc37edee100"} Jul 07 17:42:08.508430 ip-10-0-129-39 kubenswrapper[2575]: I0707 17:42:08.508300 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-b5997f658-vr99g" event={"ID":"9dbed709-3d53-4bfc-9350-e92e8b8144ca","Type":"ContainerStarted","Data":"13639f87df7a15823586cff43d2ed7ab3a85a5fb47f53bab1522540f37ffbe04"} Jul 07 17:42:08.525930 ip-10-0-129-39 kubenswrapper[2575]: I0707 17:42:08.525891 2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/metrics-server-7df964d657-xdrrj" podStartSLOduration=1.848448898 podStartE2EDuration="4.525876412s" totalImagesPullingTime="2.677427514s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-07 17:42:04 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 17:42:08.524717399 +0000 UTC m=+89.030524065" watchObservedRunningTime="2026-07-07 17:42:08.525876412 +0000 UTC m=+89.031683077" Jul 07 17:42:08.542418 ip-10-0-129-39 kubenswrapper[2575]: I0707 17:42:08.542385 2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-b5997f658-vr99g" podStartSLOduration=3.582830396 podStartE2EDuration="18.542371456s" totalImagesPullingTime="14.95954106s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-07 17:41:50 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 17:42:08.542254669 +0000 UTC m=+89.048061335" watchObservedRunningTime="2026-07-07 17:42:08.542371456 +0000 UTC m=+89.048178123" Jul 07 17:42:12.465468 ip-10-0-129-39 kubenswrapper[2575]: I0707 17:42:12.465433 2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-cdbcbbc7-6m8p7"] Jul 07 17:42:12.468033 ip-10-0-129-39 kubenswrapper[2575]: I0707 17:42:12.468018 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-cdbcbbc7-6m8p7" Jul 07 17:42:12.470672 ip-10-0-129-39 kubenswrapper[2575]: I0707 17:42:12.470650 2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"service-ca\"" Jul 07 17:42:12.470788 ip-10-0-129-39 kubenswrapper[2575]: I0707 17:42:12.470748 2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"openshift-service-ca.crt\"" Jul 07 17:42:12.471722 ip-10-0-129-39 kubenswrapper[2575]: I0707 17:42:12.471690 2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"console-config\"" Jul 07 17:42:12.471864 ip-10-0-129-39 kubenswrapper[2575]: I0707 17:42:12.471848 2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"oauth-serving-cert\"" Jul 07 17:42:12.472263 ip-10-0-129-39 kubenswrapper[2575]: I0707 17:42:12.472243 2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"kube-root-ca.crt\"" Jul 07 17:42:12.476259 ip-10-0-129-39 kubenswrapper[2575]: I0707 17:42:12.472727 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-oauth-config\"" Jul 07 17:42:12.476259 ip-10-0-129-39 kubenswrapper[2575]: I0707 17:42:12.473363 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-dockercfg-l6245\"" Jul 07 17:42:12.476259 ip-10-0-129-39 kubenswrapper[2575]: I0707 17:42:12.473450 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-serving-cert\"" Jul 07 17:42:12.479857 ip-10-0-129-39 kubenswrapper[2575]: I0707 17:42:12.479840 2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"trusted-ca-bundle\"" Jul 07 17:42:12.484882 ip-10-0-129-39 kubenswrapper[2575]: I0707 17:42:12.484862 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-cdbcbbc7-6m8p7"] Jul 07 17:42:12.544032 ip-10-0-129-39 kubenswrapper[2575]: I0707 17:42:12.544003 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-f2jr7\" (UniqueName: \"kubernetes.io/projected/0279222d-ac83-46c5-9973-9e2c562cdccd-kube-api-access-f2jr7\") pod \"console-cdbcbbc7-6m8p7\" (UID: \"0279222d-ac83-46c5-9973-9e2c562cdccd\") " pod="openshift-console/console-cdbcbbc7-6m8p7" Jul 07 17:42:12.544126 ip-10-0-129-39 kubenswrapper[2575]: I0707 17:42:12.544033 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/0279222d-ac83-46c5-9973-9e2c562cdccd-trusted-ca-bundle\") pod \"console-cdbcbbc7-6m8p7\" (UID: \"0279222d-ac83-46c5-9973-9e2c562cdccd\") " pod="openshift-console/console-cdbcbbc7-6m8p7" Jul 07 17:42:12.544126 ip-10-0-129-39 kubenswrapper[2575]: I0707 17:42:12.544056 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/0279222d-ac83-46c5-9973-9e2c562cdccd-oauth-serving-cert\") pod \"console-cdbcbbc7-6m8p7\" (UID: \"0279222d-ac83-46c5-9973-9e2c562cdccd\") " pod="openshift-console/console-cdbcbbc7-6m8p7" Jul 07 17:42:12.544126 ip-10-0-129-39 kubenswrapper[2575]: I0707 17:42:12.544073 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/0279222d-ac83-46c5-9973-9e2c562cdccd-console-config\") pod \"console-cdbcbbc7-6m8p7\" (UID: \"0279222d-ac83-46c5-9973-9e2c562cdccd\") " pod="openshift-console/console-cdbcbbc7-6m8p7" Jul 07 17:42:12.544223 ip-10-0-129-39 kubenswrapper[2575]: I0707 17:42:12.544127 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/0279222d-ac83-46c5-9973-9e2c562cdccd-console-oauth-config\") pod \"console-cdbcbbc7-6m8p7\" (UID: \"0279222d-ac83-46c5-9973-9e2c562cdccd\") " pod="openshift-console/console-cdbcbbc7-6m8p7" Jul 07 17:42:12.544223 ip-10-0-129-39 kubenswrapper[2575]: I0707 17:42:12.544188 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/0279222d-ac83-46c5-9973-9e2c562cdccd-console-serving-cert\") pod \"console-cdbcbbc7-6m8p7\" (UID: \"0279222d-ac83-46c5-9973-9e2c562cdccd\") " pod="openshift-console/console-cdbcbbc7-6m8p7" Jul 07 17:42:12.544223 ip-10-0-129-39 kubenswrapper[2575]: I0707 17:42:12.544216 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/0279222d-ac83-46c5-9973-9e2c562cdccd-service-ca\") pod \"console-cdbcbbc7-6m8p7\" (UID: \"0279222d-ac83-46c5-9973-9e2c562cdccd\") " pod="openshift-console/console-cdbcbbc7-6m8p7" Jul 07 17:42:12.645462 ip-10-0-129-39 kubenswrapper[2575]: I0707 17:42:12.645440 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f2jr7\" (UniqueName: \"kubernetes.io/projected/0279222d-ac83-46c5-9973-9e2c562cdccd-kube-api-access-f2jr7\") pod \"console-cdbcbbc7-6m8p7\" (UID: \"0279222d-ac83-46c5-9973-9e2c562cdccd\") " pod="openshift-console/console-cdbcbbc7-6m8p7" Jul 07 17:42:12.645578 ip-10-0-129-39 kubenswrapper[2575]: I0707 17:42:12.645471 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/0279222d-ac83-46c5-9973-9e2c562cdccd-trusted-ca-bundle\") pod \"console-cdbcbbc7-6m8p7\" (UID: \"0279222d-ac83-46c5-9973-9e2c562cdccd\") " pod="openshift-console/console-cdbcbbc7-6m8p7" Jul 07 17:42:12.645578 ip-10-0-129-39 kubenswrapper[2575]: I0707 17:42:12.645494 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/0279222d-ac83-46c5-9973-9e2c562cdccd-oauth-serving-cert\") pod \"console-cdbcbbc7-6m8p7\" (UID: \"0279222d-ac83-46c5-9973-9e2c562cdccd\") " pod="openshift-console/console-cdbcbbc7-6m8p7" Jul 07 17:42:12.645578 ip-10-0-129-39 kubenswrapper[2575]: I0707 17:42:12.645538 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/0279222d-ac83-46c5-9973-9e2c562cdccd-console-config\") pod \"console-cdbcbbc7-6m8p7\" (UID: \"0279222d-ac83-46c5-9973-9e2c562cdccd\") " pod="openshift-console/console-cdbcbbc7-6m8p7" Jul 07 17:42:12.645578 ip-10-0-129-39 kubenswrapper[2575]: I0707 17:42:12.645559 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/0279222d-ac83-46c5-9973-9e2c562cdccd-console-oauth-config\") pod \"console-cdbcbbc7-6m8p7\" (UID: \"0279222d-ac83-46c5-9973-9e2c562cdccd\") " pod="openshift-console/console-cdbcbbc7-6m8p7" Jul 07 17:42:12.645578 ip-10-0-129-39 kubenswrapper[2575]: I0707 17:42:12.645576 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/0279222d-ac83-46c5-9973-9e2c562cdccd-console-serving-cert\") pod \"console-cdbcbbc7-6m8p7\" (UID: \"0279222d-ac83-46c5-9973-9e2c562cdccd\") " pod="openshift-console/console-cdbcbbc7-6m8p7" Jul 07 17:42:12.645806 ip-10-0-129-39 kubenswrapper[2575]: I0707 17:42:12.645590 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/0279222d-ac83-46c5-9973-9e2c562cdccd-service-ca\") pod \"console-cdbcbbc7-6m8p7\" (UID: \"0279222d-ac83-46c5-9973-9e2c562cdccd\") " pod="openshift-console/console-cdbcbbc7-6m8p7" Jul 07 17:42:12.646258 ip-10-0-129-39 kubenswrapper[2575]: I0707 17:42:12.646226 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/0279222d-ac83-46c5-9973-9e2c562cdccd-oauth-serving-cert\") pod \"console-cdbcbbc7-6m8p7\" (UID: \"0279222d-ac83-46c5-9973-9e2c562cdccd\") " pod="openshift-console/console-cdbcbbc7-6m8p7" Jul 07 17:42:12.646364 ip-10-0-129-39 kubenswrapper[2575]: I0707 17:42:12.646336 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/0279222d-ac83-46c5-9973-9e2c562cdccd-service-ca\") pod \"console-cdbcbbc7-6m8p7\" (UID: \"0279222d-ac83-46c5-9973-9e2c562cdccd\") " pod="openshift-console/console-cdbcbbc7-6m8p7" Jul 07 17:42:12.646364 ip-10-0-129-39 kubenswrapper[2575]: I0707 17:42:12.646360 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/0279222d-ac83-46c5-9973-9e2c562cdccd-console-config\") pod \"console-cdbcbbc7-6m8p7\" (UID: \"0279222d-ac83-46c5-9973-9e2c562cdccd\") " pod="openshift-console/console-cdbcbbc7-6m8p7" Jul 07 17:42:12.646562 ip-10-0-129-39 kubenswrapper[2575]: I0707 17:42:12.646551 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/0279222d-ac83-46c5-9973-9e2c562cdccd-trusted-ca-bundle\") pod \"console-cdbcbbc7-6m8p7\" (UID: \"0279222d-ac83-46c5-9973-9e2c562cdccd\") " pod="openshift-console/console-cdbcbbc7-6m8p7" Jul 07 17:42:12.647921 ip-10-0-129-39 kubenswrapper[2575]: I0707 17:42:12.647902 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/0279222d-ac83-46c5-9973-9e2c562cdccd-console-oauth-config\") pod \"console-cdbcbbc7-6m8p7\" (UID: \"0279222d-ac83-46c5-9973-9e2c562cdccd\") " pod="openshift-console/console-cdbcbbc7-6m8p7" Jul 07 17:42:12.648085 ip-10-0-129-39 kubenswrapper[2575]: I0707 17:42:12.648066 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/0279222d-ac83-46c5-9973-9e2c562cdccd-console-serving-cert\") pod \"console-cdbcbbc7-6m8p7\" (UID: \"0279222d-ac83-46c5-9973-9e2c562cdccd\") " pod="openshift-console/console-cdbcbbc7-6m8p7" Jul 07 17:42:12.653912 ip-10-0-129-39 kubenswrapper[2575]: I0707 17:42:12.653888 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-f2jr7\" (UniqueName: \"kubernetes.io/projected/0279222d-ac83-46c5-9973-9e2c562cdccd-kube-api-access-f2jr7\") pod \"console-cdbcbbc7-6m8p7\" (UID: \"0279222d-ac83-46c5-9973-9e2c562cdccd\") " pod="openshift-console/console-cdbcbbc7-6m8p7" Jul 07 17:42:12.779462 ip-10-0-129-39 kubenswrapper[2575]: I0707 17:42:12.779400 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-cdbcbbc7-6m8p7" Jul 07 17:42:12.904689 ip-10-0-129-39 kubenswrapper[2575]: I0707 17:42:12.904667 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-cdbcbbc7-6m8p7"] Jul 07 17:42:12.907049 ip-10-0-129-39 kubenswrapper[2575]: W0707 17:42:12.907022 2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0279222d_ac83_46c5_9973_9e2c562cdccd.slice/crio-97ea0970ba6ad034d662dba3b1685876bb28f0ce916bb53dfa16bb00d9c10bb7 WatchSource:0}: Error finding container 97ea0970ba6ad034d662dba3b1685876bb28f0ce916bb53dfa16bb00d9c10bb7: Status 404 returned error can't find the container with id 97ea0970ba6ad034d662dba3b1685876bb28f0ce916bb53dfa16bb00d9c10bb7 Jul 07 17:42:13.526356 ip-10-0-129-39 kubenswrapper[2575]: I0707 17:42:13.526320 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-cdbcbbc7-6m8p7" event={"ID":"0279222d-ac83-46c5-9973-9e2c562cdccd","Type":"ContainerStarted","Data":"97ea0970ba6ad034d662dba3b1685876bb28f0ce916bb53dfa16bb00d9c10bb7"} Jul 07 17:42:16.538736 ip-10-0-129-39 kubenswrapper[2575]: I0707 17:42:16.538697 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-cdbcbbc7-6m8p7" event={"ID":"0279222d-ac83-46c5-9973-9e2c562cdccd","Type":"ContainerStarted","Data":"d0e55aa0b1d91b46658b61ea94febc8876f31100065e3fdfc32eef38ec150c70"} Jul 07 17:42:16.560871 ip-10-0-129-39 kubenswrapper[2575]: I0707 17:42:16.560721 2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-cdbcbbc7-6m8p7" podStartSLOduration=1.663958492 podStartE2EDuration="4.560702832s" totalImagesPullingTime="2.89674434s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-07 17:42:12 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 17:42:16.560616803 +0000 UTC m=+97.066423471" watchObservedRunningTime="2026-07-07 17:42:16.560702832 +0000 UTC m=+97.066509498" Jul 07 17:42:18.412767 ip-10-0-129-39 kubenswrapper[2575]: I0707 17:42:18.412724 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-network-diagnostics/network-check-target-vzf4m" Jul 07 17:42:22.780167 ip-10-0-129-39 kubenswrapper[2575]: I0707 17:42:22.780131 2575 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-cdbcbbc7-6m8p7" Jul 07 17:42:22.780571 ip-10-0-129-39 kubenswrapper[2575]: I0707 17:42:22.780198 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-cdbcbbc7-6m8p7" Jul 07 17:42:22.785494 ip-10-0-129-39 kubenswrapper[2575]: I0707 17:42:22.785473 2575 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-cdbcbbc7-6m8p7" Jul 07 17:42:23.567069 ip-10-0-129-39 kubenswrapper[2575]: I0707 17:42:23.567042 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-cdbcbbc7-6m8p7" Jul 07 17:42:24.840022 ip-10-0-129-39 kubenswrapper[2575]: I0707 17:42:24.839989 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/metrics-server-7df964d657-xdrrj" Jul 07 17:42:24.840529 ip-10-0-129-39 kubenswrapper[2575]: I0707 17:42:24.840073 2575 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-monitoring/metrics-server-7df964d657-xdrrj" Jul 07 17:42:44.844645 ip-10-0-129-39 kubenswrapper[2575]: I0707 17:42:44.844617 2575 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-monitoring/metrics-server-7df964d657-xdrrj" Jul 07 17:42:44.848628 ip-10-0-129-39 kubenswrapper[2575]: I0707 17:42:44.848601 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/metrics-server-7df964d657-xdrrj" Jul 07 17:43:35.336101 ip-10-0-129-39 kubenswrapper[2575]: I0707 17:43:35.336017 2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-cdbcbbc7-6m8p7"] Jul 07 17:44:00.356108 ip-10-0-129-39 kubenswrapper[2575]: I0707 17:44:00.356052 2575 kuberuntime_container.go:872] "Killing container with a grace period" pod="openshift-console/console-cdbcbbc7-6m8p7" podUID="0279222d-ac83-46c5-9973-9e2c562cdccd" containerName="console" containerID="cri-o://d0e55aa0b1d91b46658b61ea94febc8876f31100065e3fdfc32eef38ec150c70" gracePeriod=15 Jul 07 17:44:00.594287 ip-10-0-129-39 kubenswrapper[2575]: I0707 17:44:00.594264 2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-cdbcbbc7-6m8p7" Jul 07 17:44:00.715254 ip-10-0-129-39 kubenswrapper[2575]: I0707 17:44:00.715175 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/0279222d-ac83-46c5-9973-9e2c562cdccd-trusted-ca-bundle\") pod \"0279222d-ac83-46c5-9973-9e2c562cdccd\" (UID: \"0279222d-ac83-46c5-9973-9e2c562cdccd\") " Jul 07 17:44:00.715254 ip-10-0-129-39 kubenswrapper[2575]: I0707 17:44:00.715224 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/0279222d-ac83-46c5-9973-9e2c562cdccd-console-oauth-config\") pod \"0279222d-ac83-46c5-9973-9e2c562cdccd\" (UID: \"0279222d-ac83-46c5-9973-9e2c562cdccd\") " Jul 07 17:44:00.715254 ip-10-0-129-39 kubenswrapper[2575]: I0707 17:44:00.715240 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/0279222d-ac83-46c5-9973-9e2c562cdccd-service-ca\") pod \"0279222d-ac83-46c5-9973-9e2c562cdccd\" (UID: \"0279222d-ac83-46c5-9973-9e2c562cdccd\") " Jul 07 17:44:00.715540 ip-10-0-129-39 kubenswrapper[2575]: I0707 17:44:00.715267 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/0279222d-ac83-46c5-9973-9e2c562cdccd-console-config\") pod \"0279222d-ac83-46c5-9973-9e2c562cdccd\" (UID: \"0279222d-ac83-46c5-9973-9e2c562cdccd\") " Jul 07 17:44:00.715540 ip-10-0-129-39 kubenswrapper[2575]: I0707 17:44:00.715466 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-f2jr7\" (UniqueName: \"kubernetes.io/projected/0279222d-ac83-46c5-9973-9e2c562cdccd-kube-api-access-f2jr7\") pod \"0279222d-ac83-46c5-9973-9e2c562cdccd\" (UID: \"0279222d-ac83-46c5-9973-9e2c562cdccd\") " Jul 07 17:44:00.715666 ip-10-0-129-39 kubenswrapper[2575]: I0707 17:44:00.715552 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/0279222d-ac83-46c5-9973-9e2c562cdccd-console-serving-cert\") pod \"0279222d-ac83-46c5-9973-9e2c562cdccd\" (UID: \"0279222d-ac83-46c5-9973-9e2c562cdccd\") " Jul 07 17:44:00.715666 ip-10-0-129-39 kubenswrapper[2575]: I0707 17:44:00.715561 2575 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0279222d-ac83-46c5-9973-9e2c562cdccd-console-config" (OuterVolumeSpecName: "console-config") pod "0279222d-ac83-46c5-9973-9e2c562cdccd" (UID: "0279222d-ac83-46c5-9973-9e2c562cdccd"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 07 17:44:00.715666 ip-10-0-129-39 kubenswrapper[2575]: I0707 17:44:00.715584 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/0279222d-ac83-46c5-9973-9e2c562cdccd-oauth-serving-cert\") pod \"0279222d-ac83-46c5-9973-9e2c562cdccd\" (UID: \"0279222d-ac83-46c5-9973-9e2c562cdccd\") " Jul 07 17:44:00.715666 ip-10-0-129-39 kubenswrapper[2575]: I0707 17:44:00.715625 2575 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0279222d-ac83-46c5-9973-9e2c562cdccd-trusted-ca-bundle" (OuterVolumeSpecName: "trusted-ca-bundle") pod "0279222d-ac83-46c5-9973-9e2c562cdccd" (UID: "0279222d-ac83-46c5-9973-9e2c562cdccd"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 07 17:44:00.715855 ip-10-0-129-39 kubenswrapper[2575]: I0707 17:44:00.715718 2575 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0279222d-ac83-46c5-9973-9e2c562cdccd-service-ca" (OuterVolumeSpecName: "service-ca") pod "0279222d-ac83-46c5-9973-9e2c562cdccd" (UID: "0279222d-ac83-46c5-9973-9e2c562cdccd"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 07 17:44:00.715937 ip-10-0-129-39 kubenswrapper[2575]: I0707 17:44:00.715917 2575 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/0279222d-ac83-46c5-9973-9e2c562cdccd-service-ca\") on node \"ip-10-0-129-39.ec2.internal\" DevicePath \"\"" Jul 07 17:44:00.715992 ip-10-0-129-39 kubenswrapper[2575]: I0707 17:44:00.715945 2575 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/0279222d-ac83-46c5-9973-9e2c562cdccd-console-config\") on node \"ip-10-0-129-39.ec2.internal\" DevicePath \"\"" Jul 07 17:44:00.715992 ip-10-0-129-39 kubenswrapper[2575]: I0707 17:44:00.715960 2575 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/0279222d-ac83-46c5-9973-9e2c562cdccd-trusted-ca-bundle\") on node \"ip-10-0-129-39.ec2.internal\" DevicePath \"\"" Jul 07 17:44:00.715992 ip-10-0-129-39 kubenswrapper[2575]: I0707 17:44:00.715963 2575 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0279222d-ac83-46c5-9973-9e2c562cdccd-oauth-serving-cert" (OuterVolumeSpecName: "oauth-serving-cert") pod "0279222d-ac83-46c5-9973-9e2c562cdccd" (UID: "0279222d-ac83-46c5-9973-9e2c562cdccd"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 07 17:44:00.717608 ip-10-0-129-39 kubenswrapper[2575]: I0707 17:44:00.717581 2575 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0279222d-ac83-46c5-9973-9e2c562cdccd-console-serving-cert" (OuterVolumeSpecName: "console-serving-cert") pod "0279222d-ac83-46c5-9973-9e2c562cdccd" (UID: "0279222d-ac83-46c5-9973-9e2c562cdccd"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 07 17:44:00.717701 ip-10-0-129-39 kubenswrapper[2575]: I0707 17:44:00.717674 2575 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0279222d-ac83-46c5-9973-9e2c562cdccd-kube-api-access-f2jr7" (OuterVolumeSpecName: "kube-api-access-f2jr7") pod "0279222d-ac83-46c5-9973-9e2c562cdccd" (UID: "0279222d-ac83-46c5-9973-9e2c562cdccd"). InnerVolumeSpecName "kube-api-access-f2jr7". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 07 17:44:00.717739 ip-10-0-129-39 kubenswrapper[2575]: I0707 17:44:00.717703 2575 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0279222d-ac83-46c5-9973-9e2c562cdccd-console-oauth-config" (OuterVolumeSpecName: "console-oauth-config") pod "0279222d-ac83-46c5-9973-9e2c562cdccd" (UID: "0279222d-ac83-46c5-9973-9e2c562cdccd"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 07 17:44:00.817134 ip-10-0-129-39 kubenswrapper[2575]: I0707 17:44:00.817110 2575 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/0279222d-ac83-46c5-9973-9e2c562cdccd-console-oauth-config\") on node \"ip-10-0-129-39.ec2.internal\" DevicePath \"\"" Jul 07 17:44:00.817134 ip-10-0-129-39 kubenswrapper[2575]: I0707 17:44:00.817129 2575 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-f2jr7\" (UniqueName: \"kubernetes.io/projected/0279222d-ac83-46c5-9973-9e2c562cdccd-kube-api-access-f2jr7\") on node \"ip-10-0-129-39.ec2.internal\" DevicePath \"\"" Jul 07 17:44:00.817134 ip-10-0-129-39 kubenswrapper[2575]: I0707 17:44:00.817137 2575 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/0279222d-ac83-46c5-9973-9e2c562cdccd-console-serving-cert\") on node \"ip-10-0-129-39.ec2.internal\" DevicePath \"\"" Jul 07 17:44:00.817298 ip-10-0-129-39 kubenswrapper[2575]: I0707 17:44:00.817146 2575 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/0279222d-ac83-46c5-9973-9e2c562cdccd-oauth-serving-cert\") on node \"ip-10-0-129-39.ec2.internal\" DevicePath \"\"" Jul 07 17:44:00.890272 ip-10-0-129-39 kubenswrapper[2575]: I0707 17:44:00.890241 2575 generic.go:356] "Generic (PLEG): container finished" podID="0279222d-ac83-46c5-9973-9e2c562cdccd" containerID="d0e55aa0b1d91b46658b61ea94febc8876f31100065e3fdfc32eef38ec150c70" exitCode=0 Jul 07 17:44:00.890383 ip-10-0-129-39 kubenswrapper[2575]: I0707 17:44:00.890287 2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-cdbcbbc7-6m8p7" Jul 07 17:44:00.890383 ip-10-0-129-39 kubenswrapper[2575]: I0707 17:44:00.890323 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-cdbcbbc7-6m8p7" event={"ID":"0279222d-ac83-46c5-9973-9e2c562cdccd","Type":"ContainerDied","Data":"d0e55aa0b1d91b46658b61ea94febc8876f31100065e3fdfc32eef38ec150c70"} Jul 07 17:44:00.890383 ip-10-0-129-39 kubenswrapper[2575]: I0707 17:44:00.890368 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-cdbcbbc7-6m8p7" event={"ID":"0279222d-ac83-46c5-9973-9e2c562cdccd","Type":"ContainerDied","Data":"97ea0970ba6ad034d662dba3b1685876bb28f0ce916bb53dfa16bb00d9c10bb7"} Jul 07 17:44:00.890498 ip-10-0-129-39 kubenswrapper[2575]: I0707 17:44:00.890388 2575 scope.go:117] "RemoveContainer" containerID="d0e55aa0b1d91b46658b61ea94febc8876f31100065e3fdfc32eef38ec150c70" Jul 07 17:44:00.899364 ip-10-0-129-39 kubenswrapper[2575]: I0707 17:44:00.899349 2575 scope.go:117] "RemoveContainer" containerID="d0e55aa0b1d91b46658b61ea94febc8876f31100065e3fdfc32eef38ec150c70" Jul 07 17:44:00.899606 ip-10-0-129-39 kubenswrapper[2575]: E0707 17:44:00.899588 2575 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d0e55aa0b1d91b46658b61ea94febc8876f31100065e3fdfc32eef38ec150c70\": container with ID starting with d0e55aa0b1d91b46658b61ea94febc8876f31100065e3fdfc32eef38ec150c70 not found: ID does not exist" containerID="d0e55aa0b1d91b46658b61ea94febc8876f31100065e3fdfc32eef38ec150c70" Jul 07 17:44:00.899665 ip-10-0-129-39 kubenswrapper[2575]: I0707 17:44:00.899613 2575 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d0e55aa0b1d91b46658b61ea94febc8876f31100065e3fdfc32eef38ec150c70"} err="failed to get container status \"d0e55aa0b1d91b46658b61ea94febc8876f31100065e3fdfc32eef38ec150c70\": rpc error: code = NotFound desc = could not find container \"d0e55aa0b1d91b46658b61ea94febc8876f31100065e3fdfc32eef38ec150c70\": container with ID starting with d0e55aa0b1d91b46658b61ea94febc8876f31100065e3fdfc32eef38ec150c70 not found: ID does not exist" Jul 07 17:44:00.916055 ip-10-0-129-39 kubenswrapper[2575]: I0707 17:44:00.916025 2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-cdbcbbc7-6m8p7"] Jul 07 17:44:00.917151 ip-10-0-129-39 kubenswrapper[2575]: I0707 17:44:00.917130 2575 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-cdbcbbc7-6m8p7"] Jul 07 17:44:02.073217 ip-10-0-129-39 kubenswrapper[2575]: I0707 17:44:02.073185 2575 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="0279222d-ac83-46c5-9973-9e2c562cdccd" path="/var/lib/kubelet/pods/0279222d-ac83-46c5-9973-9e2c562cdccd/volumes" Jul 07 17:44:41.565479 ip-10-0-129-39 kubenswrapper[2575]: I0707 17:44:41.565445 2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["cert-manager/cert-manager-cainjector-559cdd8c87-7cqx7"] Jul 07 17:44:41.566001 ip-10-0-129-39 kubenswrapper[2575]: I0707 17:44:41.565813 2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0279222d-ac83-46c5-9973-9e2c562cdccd" containerName="console" Jul 07 17:44:41.566001 ip-10-0-129-39 kubenswrapper[2575]: I0707 17:44:41.565831 2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="0279222d-ac83-46c5-9973-9e2c562cdccd" containerName="console" Jul 07 17:44:41.566001 ip-10-0-129-39 kubenswrapper[2575]: I0707 17:44:41.565892 2575 memory_manager.go:368] "RemoveStaleState removing state" podUID="0279222d-ac83-46c5-9973-9e2c562cdccd" containerName="console" Jul 07 17:44:41.569862 ip-10-0-129-39 kubenswrapper[2575]: I0707 17:44:41.569840 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-cainjector-559cdd8c87-7cqx7" Jul 07 17:44:41.575122 ip-10-0-129-39 kubenswrapper[2575]: I0707 17:44:41.575095 2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager\"/\"kube-root-ca.crt\"" Jul 07 17:44:41.575242 ip-10-0-129-39 kubenswrapper[2575]: I0707 17:44:41.575223 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"cert-manager\"/\"cert-manager-cainjector-dockercfg-v4hrb\"" Jul 07 17:44:41.575689 ip-10-0-129-39 kubenswrapper[2575]: I0707 17:44:41.575673 2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager\"/\"openshift-service-ca.crt\"" Jul 07 17:44:41.578627 ip-10-0-129-39 kubenswrapper[2575]: I0707 17:44:41.578608 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-cainjector-559cdd8c87-7cqx7"] Jul 07 17:44:41.690096 ip-10-0-129-39 kubenswrapper[2575]: I0707 17:44:41.690071 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/de0882d0-2d02-403b-a06d-ecd466b074d9-bound-sa-token\") pod \"cert-manager-cainjector-559cdd8c87-7cqx7\" (UID: \"de0882d0-2d02-403b-a06d-ecd466b074d9\") " pod="cert-manager/cert-manager-cainjector-559cdd8c87-7cqx7" Jul 07 17:44:41.690096 ip-10-0-129-39 kubenswrapper[2575]: I0707 17:44:41.690095 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zds5g\" (UniqueName: \"kubernetes.io/projected/de0882d0-2d02-403b-a06d-ecd466b074d9-kube-api-access-zds5g\") pod \"cert-manager-cainjector-559cdd8c87-7cqx7\" (UID: \"de0882d0-2d02-403b-a06d-ecd466b074d9\") " pod="cert-manager/cert-manager-cainjector-559cdd8c87-7cqx7" Jul 07 17:44:41.791241 ip-10-0-129-39 kubenswrapper[2575]: I0707 17:44:41.791212 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/de0882d0-2d02-403b-a06d-ecd466b074d9-bound-sa-token\") pod \"cert-manager-cainjector-559cdd8c87-7cqx7\" (UID: \"de0882d0-2d02-403b-a06d-ecd466b074d9\") " pod="cert-manager/cert-manager-cainjector-559cdd8c87-7cqx7" Jul 07 17:44:41.791241 ip-10-0-129-39 kubenswrapper[2575]: I0707 17:44:41.791240 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zds5g\" (UniqueName: \"kubernetes.io/projected/de0882d0-2d02-403b-a06d-ecd466b074d9-kube-api-access-zds5g\") pod \"cert-manager-cainjector-559cdd8c87-7cqx7\" (UID: \"de0882d0-2d02-403b-a06d-ecd466b074d9\") " pod="cert-manager/cert-manager-cainjector-559cdd8c87-7cqx7" Jul 07 17:44:41.799680 ip-10-0-129-39 kubenswrapper[2575]: I0707 17:44:41.799659 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zds5g\" (UniqueName: \"kubernetes.io/projected/de0882d0-2d02-403b-a06d-ecd466b074d9-kube-api-access-zds5g\") pod \"cert-manager-cainjector-559cdd8c87-7cqx7\" (UID: \"de0882d0-2d02-403b-a06d-ecd466b074d9\") " pod="cert-manager/cert-manager-cainjector-559cdd8c87-7cqx7" Jul 07 17:44:41.799995 ip-10-0-129-39 kubenswrapper[2575]: I0707 17:44:41.799980 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/de0882d0-2d02-403b-a06d-ecd466b074d9-bound-sa-token\") pod \"cert-manager-cainjector-559cdd8c87-7cqx7\" (UID: \"de0882d0-2d02-403b-a06d-ecd466b074d9\") " pod="cert-manager/cert-manager-cainjector-559cdd8c87-7cqx7" Jul 07 17:44:41.883661 ip-10-0-129-39 kubenswrapper[2575]: I0707 17:44:41.883638 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-cainjector-559cdd8c87-7cqx7" Jul 07 17:44:42.004758 ip-10-0-129-39 kubenswrapper[2575]: I0707 17:44:42.004722 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-cainjector-559cdd8c87-7cqx7"] Jul 07 17:44:42.007606 ip-10-0-129-39 kubenswrapper[2575]: W0707 17:44:42.007583 2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podde0882d0_2d02_403b_a06d_ecd466b074d9.slice/crio-f53cbc1306054cff0783542660e96d51a42177e305e25723c0d590c81bfde302 WatchSource:0}: Error finding container f53cbc1306054cff0783542660e96d51a42177e305e25723c0d590c81bfde302: Status 404 returned error can't find the container with id f53cbc1306054cff0783542660e96d51a42177e305e25723c0d590c81bfde302 Jul 07 17:44:42.028001 ip-10-0-129-39 kubenswrapper[2575]: I0707 17:44:42.027973 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-cainjector-559cdd8c87-7cqx7" event={"ID":"de0882d0-2d02-403b-a06d-ecd466b074d9","Type":"ContainerStarted","Data":"f53cbc1306054cff0783542660e96d51a42177e305e25723c0d590c81bfde302"} Jul 07 17:44:51.563888 ip-10-0-129-39 kubenswrapper[2575]: I0707 17:44:51.563857 2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-lws-operator/openshift-lws-operator-fd8ccff4c-rrdm6"] Jul 07 17:44:51.566775 ip-10-0-129-39 kubenswrapper[2575]: I0707 17:44:51.566759 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-rrdm6" Jul 07 17:44:51.569030 ip-10-0-129-39 kubenswrapper[2575]: I0707 17:44:51.569009 2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"kube-root-ca.crt\"" Jul 07 17:44:51.569134 ip-10-0-129-39 kubenswrapper[2575]: I0707 17:44:51.569094 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"openshift-lws-operator-dockercfg-xvnbs\"" Jul 07 17:44:51.570032 ip-10-0-129-39 kubenswrapper[2575]: I0707 17:44:51.570014 2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"openshift-service-ca.crt\"" Jul 07 17:44:51.577629 ip-10-0-129-39 kubenswrapper[2575]: I0707 17:44:51.577609 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/openshift-lws-operator-fd8ccff4c-rrdm6"] Jul 07 17:44:51.663368 ip-10-0-129-39 kubenswrapper[2575]: I0707 17:44:51.663345 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/2afbc361-c420-4739-a8fb-5c7912cdd14e-tmp\") pod \"openshift-lws-operator-fd8ccff4c-rrdm6\" (UID: \"2afbc361-c420-4739-a8fb-5c7912cdd14e\") " pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-rrdm6" Jul 07 17:44:51.663459 ip-10-0-129-39 kubenswrapper[2575]: I0707 17:44:51.663398 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ndxbp\" (UniqueName: \"kubernetes.io/projected/2afbc361-c420-4739-a8fb-5c7912cdd14e-kube-api-access-ndxbp\") pod \"openshift-lws-operator-fd8ccff4c-rrdm6\" (UID: \"2afbc361-c420-4739-a8fb-5c7912cdd14e\") " pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-rrdm6" Jul 07 17:44:51.764548 ip-10-0-129-39 kubenswrapper[2575]: I0707 17:44:51.764521 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/2afbc361-c420-4739-a8fb-5c7912cdd14e-tmp\") pod \"openshift-lws-operator-fd8ccff4c-rrdm6\" (UID: \"2afbc361-c420-4739-a8fb-5c7912cdd14e\") " pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-rrdm6" Jul 07 17:44:51.764641 ip-10-0-129-39 kubenswrapper[2575]: I0707 17:44:51.764564 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ndxbp\" (UniqueName: \"kubernetes.io/projected/2afbc361-c420-4739-a8fb-5c7912cdd14e-kube-api-access-ndxbp\") pod \"openshift-lws-operator-fd8ccff4c-rrdm6\" (UID: \"2afbc361-c420-4739-a8fb-5c7912cdd14e\") " pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-rrdm6" Jul 07 17:44:51.764857 ip-10-0-129-39 kubenswrapper[2575]: I0707 17:44:51.764841 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/2afbc361-c420-4739-a8fb-5c7912cdd14e-tmp\") pod \"openshift-lws-operator-fd8ccff4c-rrdm6\" (UID: \"2afbc361-c420-4739-a8fb-5c7912cdd14e\") " pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-rrdm6" Jul 07 17:44:51.782886 ip-10-0-129-39 kubenswrapper[2575]: I0707 17:44:51.782860 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ndxbp\" (UniqueName: \"kubernetes.io/projected/2afbc361-c420-4739-a8fb-5c7912cdd14e-kube-api-access-ndxbp\") pod \"openshift-lws-operator-fd8ccff4c-rrdm6\" (UID: \"2afbc361-c420-4739-a8fb-5c7912cdd14e\") " pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-rrdm6" Jul 07 17:44:51.876081 ip-10-0-129-39 kubenswrapper[2575]: I0707 17:44:51.876057 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-rrdm6" Jul 07 17:44:52.005802 ip-10-0-129-39 kubenswrapper[2575]: I0707 17:44:52.005776 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/openshift-lws-operator-fd8ccff4c-rrdm6"] Jul 07 17:44:52.008706 ip-10-0-129-39 kubenswrapper[2575]: W0707 17:44:52.008681 2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod2afbc361_c420_4739_a8fb_5c7912cdd14e.slice/crio-440ba517a739ec125cce8fd1c6b6db049d88ce359977bca5fa052cfccc03001a WatchSource:0}: Error finding container 440ba517a739ec125cce8fd1c6b6db049d88ce359977bca5fa052cfccc03001a: Status 404 returned error can't find the container with id 440ba517a739ec125cce8fd1c6b6db049d88ce359977bca5fa052cfccc03001a Jul 07 17:44:52.063868 ip-10-0-129-39 kubenswrapper[2575]: I0707 17:44:52.063844 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-rrdm6" event={"ID":"2afbc361-c420-4739-a8fb-5c7912cdd14e","Type":"ContainerStarted","Data":"440ba517a739ec125cce8fd1c6b6db049d88ce359977bca5fa052cfccc03001a"} Jul 07 17:44:55.077119 ip-10-0-129-39 kubenswrapper[2575]: I0707 17:44:55.077078 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-rrdm6" event={"ID":"2afbc361-c420-4739-a8fb-5c7912cdd14e","Type":"ContainerStarted","Data":"8812d28d90ffa2039163929eda884d1868d708dade9b6b30d10231ef7edc10c3"} Jul 07 17:44:55.098913 ip-10-0-129-39 kubenswrapper[2575]: I0707 17:44:55.098866 2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-rrdm6" podStartSLOduration=1.792603738 podStartE2EDuration="4.098850753s" totalImagesPullingTime="2.306247015s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-07 17:44:51 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 17:44:55.097893225 +0000 UTC m=+255.603699888" watchObservedRunningTime="2026-07-07 17:44:55.098850753 +0000 UTC m=+255.604657396" Jul 07 17:45:10.857298 ip-10-0-129-39 kubenswrapper[2575]: I0707 17:45:10.857223 2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-d698b7b5d-7hn2s"] Jul 07 17:45:10.864108 ip-10-0-129-39 kubenswrapper[2575]: I0707 17:45:10.864087 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/opendatahub-operator-controller-manager-d698b7b5d-7hn2s" Jul 07 17:45:10.867722 ip-10-0-129-39 kubenswrapper[2575]: I0707 17:45:10.867696 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-webhook-cert\"" Jul 07 17:45:10.868548 ip-10-0-129-39 kubenswrapper[2575]: I0707 17:45:10.868533 2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"openshift-service-ca.crt\"" Jul 07 17:45:10.868786 ip-10-0-129-39 kubenswrapper[2575]: I0707 17:45:10.868772 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-manager-service-cert\"" Jul 07 17:45:10.868869 ip-10-0-129-39 kubenswrapper[2575]: I0707 17:45:10.868775 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-manager-dockercfg-lvxjn\"" Jul 07 17:45:10.874571 ip-10-0-129-39 kubenswrapper[2575]: I0707 17:45:10.874555 2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"kube-root-ca.crt\"" Jul 07 17:45:10.885518 ip-10-0-129-39 kubenswrapper[2575]: I0707 17:45:10.885482 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-d698b7b5d-7hn2s"] Jul 07 17:45:10.993800 ip-10-0-129-39 kubenswrapper[2575]: I0707 17:45:10.993771 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/9cb279fb-33ab-4bbf-bf04-886e5e82b247-webhook-cert\") pod \"opendatahub-operator-controller-manager-d698b7b5d-7hn2s\" (UID: \"9cb279fb-33ab-4bbf-bf04-886e5e82b247\") " pod="opendatahub/opendatahub-operator-controller-manager-d698b7b5d-7hn2s" Jul 07 17:45:10.993800 ip-10-0-129-39 kubenswrapper[2575]: I0707 17:45:10.993804 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/9cb279fb-33ab-4bbf-bf04-886e5e82b247-apiservice-cert\") pod \"opendatahub-operator-controller-manager-d698b7b5d-7hn2s\" (UID: \"9cb279fb-33ab-4bbf-bf04-886e5e82b247\") " pod="opendatahub/opendatahub-operator-controller-manager-d698b7b5d-7hn2s" Jul 07 17:45:10.993962 ip-10-0-129-39 kubenswrapper[2575]: I0707 17:45:10.993826 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/9cb279fb-33ab-4bbf-bf04-886e5e82b247-manifests\") pod \"opendatahub-operator-controller-manager-d698b7b5d-7hn2s\" (UID: \"9cb279fb-33ab-4bbf-bf04-886e5e82b247\") " pod="opendatahub/opendatahub-operator-controller-manager-d698b7b5d-7hn2s" Jul 07 17:45:10.993962 ip-10-0-129-39 kubenswrapper[2575]: I0707 17:45:10.993840 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/9cb279fb-33ab-4bbf-bf04-886e5e82b247-tmp\") pod \"opendatahub-operator-controller-manager-d698b7b5d-7hn2s\" (UID: \"9cb279fb-33ab-4bbf-bf04-886e5e82b247\") " pod="opendatahub/opendatahub-operator-controller-manager-d698b7b5d-7hn2s" Jul 07 17:45:10.993962 ip-10-0-129-39 kubenswrapper[2575]: I0707 17:45:10.993897 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-k9flz\" (UniqueName: \"kubernetes.io/projected/9cb279fb-33ab-4bbf-bf04-886e5e82b247-kube-api-access-k9flz\") pod \"opendatahub-operator-controller-manager-d698b7b5d-7hn2s\" (UID: \"9cb279fb-33ab-4bbf-bf04-886e5e82b247\") " pod="opendatahub/opendatahub-operator-controller-manager-d698b7b5d-7hn2s" Jul 07 17:45:11.094903 ip-10-0-129-39 kubenswrapper[2575]: I0707 17:45:11.094866 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/9cb279fb-33ab-4bbf-bf04-886e5e82b247-webhook-cert\") pod \"opendatahub-operator-controller-manager-d698b7b5d-7hn2s\" (UID: \"9cb279fb-33ab-4bbf-bf04-886e5e82b247\") " pod="opendatahub/opendatahub-operator-controller-manager-d698b7b5d-7hn2s" Jul 07 17:45:11.094903 ip-10-0-129-39 kubenswrapper[2575]: I0707 17:45:11.094910 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/9cb279fb-33ab-4bbf-bf04-886e5e82b247-apiservice-cert\") pod \"opendatahub-operator-controller-manager-d698b7b5d-7hn2s\" (UID: \"9cb279fb-33ab-4bbf-bf04-886e5e82b247\") " pod="opendatahub/opendatahub-operator-controller-manager-d698b7b5d-7hn2s" Jul 07 17:45:11.095118 ip-10-0-129-39 kubenswrapper[2575]: I0707 17:45:11.094934 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/9cb279fb-33ab-4bbf-bf04-886e5e82b247-manifests\") pod \"opendatahub-operator-controller-manager-d698b7b5d-7hn2s\" (UID: \"9cb279fb-33ab-4bbf-bf04-886e5e82b247\") " pod="opendatahub/opendatahub-operator-controller-manager-d698b7b5d-7hn2s" Jul 07 17:45:11.095118 ip-10-0-129-39 kubenswrapper[2575]: I0707 17:45:11.094950 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/9cb279fb-33ab-4bbf-bf04-886e5e82b247-tmp\") pod \"opendatahub-operator-controller-manager-d698b7b5d-7hn2s\" (UID: \"9cb279fb-33ab-4bbf-bf04-886e5e82b247\") " pod="opendatahub/opendatahub-operator-controller-manager-d698b7b5d-7hn2s" Jul 07 17:45:11.095118 ip-10-0-129-39 kubenswrapper[2575]: I0707 17:45:11.094968 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k9flz\" (UniqueName: \"kubernetes.io/projected/9cb279fb-33ab-4bbf-bf04-886e5e82b247-kube-api-access-k9flz\") pod \"opendatahub-operator-controller-manager-d698b7b5d-7hn2s\" (UID: \"9cb279fb-33ab-4bbf-bf04-886e5e82b247\") " pod="opendatahub/opendatahub-operator-controller-manager-d698b7b5d-7hn2s" Jul 07 17:45:11.095270 ip-10-0-129-39 kubenswrapper[2575]: I0707 17:45:11.095246 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/9cb279fb-33ab-4bbf-bf04-886e5e82b247-manifests\") pod \"opendatahub-operator-controller-manager-d698b7b5d-7hn2s\" (UID: \"9cb279fb-33ab-4bbf-bf04-886e5e82b247\") " pod="opendatahub/opendatahub-operator-controller-manager-d698b7b5d-7hn2s" Jul 07 17:45:11.097345 ip-10-0-129-39 kubenswrapper[2575]: I0707 17:45:11.097307 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/9cb279fb-33ab-4bbf-bf04-886e5e82b247-tmp\") pod \"opendatahub-operator-controller-manager-d698b7b5d-7hn2s\" (UID: \"9cb279fb-33ab-4bbf-bf04-886e5e82b247\") " pod="opendatahub/opendatahub-operator-controller-manager-d698b7b5d-7hn2s" Jul 07 17:45:11.097345 ip-10-0-129-39 kubenswrapper[2575]: I0707 17:45:11.097327 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/9cb279fb-33ab-4bbf-bf04-886e5e82b247-apiservice-cert\") pod \"opendatahub-operator-controller-manager-d698b7b5d-7hn2s\" (UID: \"9cb279fb-33ab-4bbf-bf04-886e5e82b247\") " pod="opendatahub/opendatahub-operator-controller-manager-d698b7b5d-7hn2s" Jul 07 17:45:11.097486 ip-10-0-129-39 kubenswrapper[2575]: I0707 17:45:11.097463 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/9cb279fb-33ab-4bbf-bf04-886e5e82b247-webhook-cert\") pod \"opendatahub-operator-controller-manager-d698b7b5d-7hn2s\" (UID: \"9cb279fb-33ab-4bbf-bf04-886e5e82b247\") " pod="opendatahub/opendatahub-operator-controller-manager-d698b7b5d-7hn2s" Jul 07 17:45:11.108489 ip-10-0-129-39 kubenswrapper[2575]: I0707 17:45:11.108435 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-k9flz\" (UniqueName: \"kubernetes.io/projected/9cb279fb-33ab-4bbf-bf04-886e5e82b247-kube-api-access-k9flz\") pod \"opendatahub-operator-controller-manager-d698b7b5d-7hn2s\" (UID: \"9cb279fb-33ab-4bbf-bf04-886e5e82b247\") " pod="opendatahub/opendatahub-operator-controller-manager-d698b7b5d-7hn2s" Jul 07 17:45:11.174469 ip-10-0-129-39 kubenswrapper[2575]: I0707 17:45:11.174447 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/opendatahub-operator-controller-manager-d698b7b5d-7hn2s" Jul 07 17:45:11.360173 ip-10-0-129-39 kubenswrapper[2575]: I0707 17:45:11.360105 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-d698b7b5d-7hn2s"] Jul 07 17:45:11.363453 ip-10-0-129-39 kubenswrapper[2575]: W0707 17:45:11.363421 2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9cb279fb_33ab_4bbf_bf04_886e5e82b247.slice/crio-aa042d9281e18df43d1138ece422c311c7c67fa44d319bbef161cebfde43f1c5 WatchSource:0}: Error finding container aa042d9281e18df43d1138ece422c311c7c67fa44d319bbef161cebfde43f1c5: Status 404 returned error can't find the container with id aa042d9281e18df43d1138ece422c311c7c67fa44d319bbef161cebfde43f1c5 Jul 07 17:45:12.136569 ip-10-0-129-39 kubenswrapper[2575]: I0707 17:45:12.136539 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-d698b7b5d-7hn2s" event={"ID":"9cb279fb-33ab-4bbf-bf04-886e5e82b247","Type":"ContainerStarted","Data":"aa042d9281e18df43d1138ece422c311c7c67fa44d319bbef161cebfde43f1c5"} Jul 07 17:45:16.153326 ip-10-0-129-39 kubenswrapper[2575]: I0707 17:45:16.153288 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-cainjector-559cdd8c87-7cqx7" event={"ID":"de0882d0-2d02-403b-a06d-ecd466b074d9","Type":"ContainerStarted","Data":"19f36a5aa6a34b825206d9bc662b7d364ab6f0f49dc201450e0ba263fb67913e"} Jul 07 17:45:16.154867 ip-10-0-129-39 kubenswrapper[2575]: I0707 17:45:16.154842 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-d698b7b5d-7hn2s" event={"ID":"9cb279fb-33ab-4bbf-bf04-886e5e82b247","Type":"ContainerStarted","Data":"87628b5c8f925672976ea03ec747485ed57ab912a10e7321c6306566a4467bed"} Jul 07 17:45:16.172191 ip-10-0-129-39 kubenswrapper[2575]: I0707 17:45:16.172146 2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="cert-manager/cert-manager-cainjector-559cdd8c87-7cqx7" podStartSLOduration=1.762880885 podStartE2EDuration="35.172130575s" totalImagesPullingTime="33.40924969s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-07 17:44:41 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 17:45:16.171618918 +0000 UTC m=+276.677425595" watchObservedRunningTime="2026-07-07 17:45:16.172130575 +0000 UTC m=+276.677937240" Jul 07 17:45:17.160877 ip-10-0-129-39 kubenswrapper[2575]: I0707 17:45:17.160845 2575 generic.go:356] "Generic (PLEG): container finished" podID="9cb279fb-33ab-4bbf-bf04-886e5e82b247" containerID="87628b5c8f925672976ea03ec747485ed57ab912a10e7321c6306566a4467bed" exitCode=0 Jul 07 17:45:17.161271 ip-10-0-129-39 kubenswrapper[2575]: I0707 17:45:17.160929 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-d698b7b5d-7hn2s" event={"ID":"9cb279fb-33ab-4bbf-bf04-886e5e82b247","Type":"ContainerDied","Data":"87628b5c8f925672976ea03ec747485ed57ab912a10e7321c6306566a4467bed"} Jul 07 17:45:18.166026 ip-10-0-129-39 kubenswrapper[2575]: I0707 17:45:18.165996 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-d698b7b5d-7hn2s" event={"ID":"9cb279fb-33ab-4bbf-bf04-886e5e82b247","Type":"ContainerStarted","Data":"88201b77746b335589fe28cb7716259d0469d08e6a2fa77d970494cf9f7b310b"} Jul 07 17:45:18.166447 ip-10-0-129-39 kubenswrapper[2575]: I0707 17:45:18.166133 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/opendatahub-operator-controller-manager-d698b7b5d-7hn2s" Jul 07 17:45:18.206159 ip-10-0-129-39 kubenswrapper[2575]: I0707 17:45:18.206121 2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/opendatahub-operator-controller-manager-d698b7b5d-7hn2s" podStartSLOduration=2.614809419 podStartE2EDuration="8.206105139s" totalImagesPullingTime="4.54577584s" totalInitContainerRuntime="1.04551988s" isStatefulPod=false podCreationTimestamp="2026-07-07 17:45:10 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 17:45:18.201581458 +0000 UTC m=+278.707388124" watchObservedRunningTime="2026-07-07 17:45:18.206105139 +0000 UTC m=+278.711911799" Jul 07 17:45:29.172020 ip-10-0-129-39 kubenswrapper[2575]: I0707 17:45:29.171988 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/opendatahub-operator-controller-manager-d698b7b5d-7hn2s" Jul 07 17:45:31.800419 ip-10-0-129-39 kubenswrapper[2575]: I0707 17:45:31.800386 2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/kube-auth-proxy-544cc96c69-v9zdn"] Jul 07 17:45:31.802412 ip-10-0-129-39 kubenswrapper[2575]: I0707 17:45:31.802397 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/kube-auth-proxy-544cc96c69-v9zdn" Jul 07 17:45:31.805605 ip-10-0-129-39 kubenswrapper[2575]: I0707 17:45:31.805587 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-creds\"" Jul 07 17:45:31.805721 ip-10-0-129-39 kubenswrapper[2575]: I0707 17:45:31.805595 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-dockercfg-27bmn\"" Jul 07 17:45:31.806257 ip-10-0-129-39 kubenswrapper[2575]: I0707 17:45:31.806241 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-tls\"" Jul 07 17:45:31.817981 ip-10-0-129-39 kubenswrapper[2575]: I0707 17:45:31.817959 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/kube-auth-proxy-544cc96c69-v9zdn"] Jul 07 17:45:31.933827 ip-10-0-129-39 kubenswrapper[2575]: I0707 17:45:31.933799 2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/odh-model-controller-7998c67dc4-km2nt"] Jul 07 17:45:31.935982 ip-10-0-129-39 kubenswrapper[2575]: I0707 17:45:31.935966 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/odh-model-controller-7998c67dc4-km2nt" Jul 07 17:45:31.939309 ip-10-0-129-39 kubenswrapper[2575]: I0707 17:45:31.939293 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"odh-model-controller-webhook-cert\"" Jul 07 17:45:31.939615 ip-10-0-129-39 kubenswrapper[2575]: I0707 17:45:31.939600 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"odh-model-controller-dockercfg-rvw7l\"" Jul 07 17:45:31.941389 ip-10-0-129-39 kubenswrapper[2575]: I0707 17:45:31.941372 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/5fda3a91-211d-4d8c-8237-df8287d92b46-tmp\") pod \"kube-auth-proxy-544cc96c69-v9zdn\" (UID: \"5fda3a91-211d-4d8c-8237-df8287d92b46\") " pod="openshift-ingress/kube-auth-proxy-544cc96c69-v9zdn" Jul 07 17:45:31.941464 ip-10-0-129-39 kubenswrapper[2575]: I0707 17:45:31.941401 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5fda3a91-211d-4d8c-8237-df8287d92b46-tls-certs\") pod \"kube-auth-proxy-544cc96c69-v9zdn\" (UID: \"5fda3a91-211d-4d8c-8237-df8287d92b46\") " pod="openshift-ingress/kube-auth-proxy-544cc96c69-v9zdn" Jul 07 17:45:31.941534 ip-10-0-129-39 kubenswrapper[2575]: I0707 17:45:31.941518 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xmx62\" (UniqueName: \"kubernetes.io/projected/5fda3a91-211d-4d8c-8237-df8287d92b46-kube-api-access-xmx62\") pod \"kube-auth-proxy-544cc96c69-v9zdn\" (UID: \"5fda3a91-211d-4d8c-8237-df8287d92b46\") " pod="openshift-ingress/kube-auth-proxy-544cc96c69-v9zdn" Jul 07 17:45:31.954303 ip-10-0-129-39 kubenswrapper[2575]: I0707 17:45:31.954281 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/odh-model-controller-7998c67dc4-km2nt"] Jul 07 17:45:32.042128 ip-10-0-129-39 kubenswrapper[2575]: I0707 17:45:32.042101 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/5fda3a91-211d-4d8c-8237-df8287d92b46-tmp\") pod \"kube-auth-proxy-544cc96c69-v9zdn\" (UID: \"5fda3a91-211d-4d8c-8237-df8287d92b46\") " pod="openshift-ingress/kube-auth-proxy-544cc96c69-v9zdn" Jul 07 17:45:32.042269 ip-10-0-129-39 kubenswrapper[2575]: I0707 17:45:32.042133 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5fda3a91-211d-4d8c-8237-df8287d92b46-tls-certs\") pod \"kube-auth-proxy-544cc96c69-v9zdn\" (UID: \"5fda3a91-211d-4d8c-8237-df8287d92b46\") " pod="openshift-ingress/kube-auth-proxy-544cc96c69-v9zdn" Jul 07 17:45:32.042269 ip-10-0-129-39 kubenswrapper[2575]: I0707 17:45:32.042157 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nkm8c\" (UniqueName: \"kubernetes.io/projected/a9875c1b-dca8-4038-b6e0-2ce204c410a0-kube-api-access-nkm8c\") pod \"odh-model-controller-7998c67dc4-km2nt\" (UID: \"a9875c1b-dca8-4038-b6e0-2ce204c410a0\") " pod="opendatahub/odh-model-controller-7998c67dc4-km2nt" Jul 07 17:45:32.042269 ip-10-0-129-39 kubenswrapper[2575]: I0707 17:45:32.042212 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xmx62\" (UniqueName: \"kubernetes.io/projected/5fda3a91-211d-4d8c-8237-df8287d92b46-kube-api-access-xmx62\") pod \"kube-auth-proxy-544cc96c69-v9zdn\" (UID: \"5fda3a91-211d-4d8c-8237-df8287d92b46\") " pod="openshift-ingress/kube-auth-proxy-544cc96c69-v9zdn" Jul 07 17:45:32.042269 ip-10-0-129-39 kubenswrapper[2575]: I0707 17:45:32.042231 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/a9875c1b-dca8-4038-b6e0-2ce204c410a0-cert\") pod \"odh-model-controller-7998c67dc4-km2nt\" (UID: \"a9875c1b-dca8-4038-b6e0-2ce204c410a0\") " pod="opendatahub/odh-model-controller-7998c67dc4-km2nt" Jul 07 17:45:32.044294 ip-10-0-129-39 kubenswrapper[2575]: I0707 17:45:32.044268 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/5fda3a91-211d-4d8c-8237-df8287d92b46-tmp\") pod \"kube-auth-proxy-544cc96c69-v9zdn\" (UID: \"5fda3a91-211d-4d8c-8237-df8287d92b46\") " pod="openshift-ingress/kube-auth-proxy-544cc96c69-v9zdn" Jul 07 17:45:32.044559 ip-10-0-129-39 kubenswrapper[2575]: I0707 17:45:32.044538 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5fda3a91-211d-4d8c-8237-df8287d92b46-tls-certs\") pod \"kube-auth-proxy-544cc96c69-v9zdn\" (UID: \"5fda3a91-211d-4d8c-8237-df8287d92b46\") " pod="openshift-ingress/kube-auth-proxy-544cc96c69-v9zdn" Jul 07 17:45:32.053017 ip-10-0-129-39 kubenswrapper[2575]: I0707 17:45:32.052951 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xmx62\" (UniqueName: \"kubernetes.io/projected/5fda3a91-211d-4d8c-8237-df8287d92b46-kube-api-access-xmx62\") pod \"kube-auth-proxy-544cc96c69-v9zdn\" (UID: \"5fda3a91-211d-4d8c-8237-df8287d92b46\") " pod="openshift-ingress/kube-auth-proxy-544cc96c69-v9zdn" Jul 07 17:45:32.111760 ip-10-0-129-39 kubenswrapper[2575]: I0707 17:45:32.111731 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/kube-auth-proxy-544cc96c69-v9zdn" Jul 07 17:45:32.143224 ip-10-0-129-39 kubenswrapper[2575]: I0707 17:45:32.143198 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/a9875c1b-dca8-4038-b6e0-2ce204c410a0-cert\") pod \"odh-model-controller-7998c67dc4-km2nt\" (UID: \"a9875c1b-dca8-4038-b6e0-2ce204c410a0\") " pod="opendatahub/odh-model-controller-7998c67dc4-km2nt" Jul 07 17:45:32.143333 ip-10-0-129-39 kubenswrapper[2575]: I0707 17:45:32.143236 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nkm8c\" (UniqueName: \"kubernetes.io/projected/a9875c1b-dca8-4038-b6e0-2ce204c410a0-kube-api-access-nkm8c\") pod \"odh-model-controller-7998c67dc4-km2nt\" (UID: \"a9875c1b-dca8-4038-b6e0-2ce204c410a0\") " pod="opendatahub/odh-model-controller-7998c67dc4-km2nt" Jul 07 17:45:32.143375 ip-10-0-129-39 kubenswrapper[2575]: E0707 17:45:32.143337 2575 secret.go:189] Couldn't get secret opendatahub/odh-model-controller-webhook-cert: secret "odh-model-controller-webhook-cert" not found Jul 07 17:45:32.143417 ip-10-0-129-39 kubenswrapper[2575]: E0707 17:45:32.143406 2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a9875c1b-dca8-4038-b6e0-2ce204c410a0-cert podName:a9875c1b-dca8-4038-b6e0-2ce204c410a0 nodeName:}" failed. No retries permitted until 2026-07-07 17:45:32.643389076 +0000 UTC m=+293.149195720 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "cert" (UniqueName: "kubernetes.io/secret/a9875c1b-dca8-4038-b6e0-2ce204c410a0-cert") pod "odh-model-controller-7998c67dc4-km2nt" (UID: "a9875c1b-dca8-4038-b6e0-2ce204c410a0") : secret "odh-model-controller-webhook-cert" not found Jul 07 17:45:32.154746 ip-10-0-129-39 kubenswrapper[2575]: I0707 17:45:32.154726 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nkm8c\" (UniqueName: \"kubernetes.io/projected/a9875c1b-dca8-4038-b6e0-2ce204c410a0-kube-api-access-nkm8c\") pod \"odh-model-controller-7998c67dc4-km2nt\" (UID: \"a9875c1b-dca8-4038-b6e0-2ce204c410a0\") " pod="opendatahub/odh-model-controller-7998c67dc4-km2nt" Jul 07 17:45:32.256926 ip-10-0-129-39 kubenswrapper[2575]: I0707 17:45:32.256890 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/kube-auth-proxy-544cc96c69-v9zdn"] Jul 07 17:45:32.259549 ip-10-0-129-39 kubenswrapper[2575]: W0707 17:45:32.259523 2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5fda3a91_211d_4d8c_8237_df8287d92b46.slice/crio-007d5be970a59153c16d9f4c126628c997b8eb8f26efc08dfc4e4e283c610529 WatchSource:0}: Error finding container 007d5be970a59153c16d9f4c126628c997b8eb8f26efc08dfc4e4e283c610529: Status 404 returned error can't find the container with id 007d5be970a59153c16d9f4c126628c997b8eb8f26efc08dfc4e4e283c610529 Jul 07 17:45:32.647134 ip-10-0-129-39 kubenswrapper[2575]: I0707 17:45:32.647101 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/a9875c1b-dca8-4038-b6e0-2ce204c410a0-cert\") pod \"odh-model-controller-7998c67dc4-km2nt\" (UID: \"a9875c1b-dca8-4038-b6e0-2ce204c410a0\") " pod="opendatahub/odh-model-controller-7998c67dc4-km2nt" Jul 07 17:45:32.649919 ip-10-0-129-39 kubenswrapper[2575]: I0707 17:45:32.649890 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/a9875c1b-dca8-4038-b6e0-2ce204c410a0-cert\") pod \"odh-model-controller-7998c67dc4-km2nt\" (UID: \"a9875c1b-dca8-4038-b6e0-2ce204c410a0\") " pod="opendatahub/odh-model-controller-7998c67dc4-km2nt" Jul 07 17:45:32.845728 ip-10-0-129-39 kubenswrapper[2575]: I0707 17:45:32.845687 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/odh-model-controller-7998c67dc4-km2nt" Jul 07 17:45:32.866352 ip-10-0-129-39 kubenswrapper[2575]: I0707 17:45:32.866323 2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/model-serving-api-54d89bd5fc-pb6bw"] Jul 07 17:45:32.868619 ip-10-0-129-39 kubenswrapper[2575]: I0707 17:45:32.868603 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/model-serving-api-54d89bd5fc-pb6bw" Jul 07 17:45:32.873385 ip-10-0-129-39 kubenswrapper[2575]: I0707 17:45:32.873368 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"model-serving-api-tls\"" Jul 07 17:45:32.881701 ip-10-0-129-39 kubenswrapper[2575]: I0707 17:45:32.881683 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"model-serving-api-dockercfg-2zzhn\"" Jul 07 17:45:32.895579 ip-10-0-129-39 kubenswrapper[2575]: I0707 17:45:32.895531 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/model-serving-api-54d89bd5fc-pb6bw"] Jul 07 17:45:32.994945 ip-10-0-129-39 kubenswrapper[2575]: I0707 17:45:32.994918 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/odh-model-controller-7998c67dc4-km2nt"] Jul 07 17:45:32.998690 ip-10-0-129-39 kubenswrapper[2575]: W0707 17:45:32.998665 2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda9875c1b_dca8_4038_b6e0_2ce204c410a0.slice/crio-88d2994e0db291f21b2a218ba392a74fcffb0a50f2af0bbb1424819742d6f95c WatchSource:0}: Error finding container 88d2994e0db291f21b2a218ba392a74fcffb0a50f2af0bbb1424819742d6f95c: Status 404 returned error can't find the container with id 88d2994e0db291f21b2a218ba392a74fcffb0a50f2af0bbb1424819742d6f95c Jul 07 17:45:33.050348 ip-10-0-129-39 kubenswrapper[2575]: I0707 17:45:33.050321 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/549a1f02-41ef-4b55-914b-f34bbc8cda0c-tls-certs\") pod \"model-serving-api-54d89bd5fc-pb6bw\" (UID: \"549a1f02-41ef-4b55-914b-f34bbc8cda0c\") " pod="opendatahub/model-serving-api-54d89bd5fc-pb6bw" Jul 07 17:45:33.050462 ip-10-0-129-39 kubenswrapper[2575]: I0707 17:45:33.050376 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zj8v6\" (UniqueName: \"kubernetes.io/projected/549a1f02-41ef-4b55-914b-f34bbc8cda0c-kube-api-access-zj8v6\") pod \"model-serving-api-54d89bd5fc-pb6bw\" (UID: \"549a1f02-41ef-4b55-914b-f34bbc8cda0c\") " pod="opendatahub/model-serving-api-54d89bd5fc-pb6bw" Jul 07 17:45:33.151542 ip-10-0-129-39 kubenswrapper[2575]: I0707 17:45:33.151457 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zj8v6\" (UniqueName: \"kubernetes.io/projected/549a1f02-41ef-4b55-914b-f34bbc8cda0c-kube-api-access-zj8v6\") pod \"model-serving-api-54d89bd5fc-pb6bw\" (UID: \"549a1f02-41ef-4b55-914b-f34bbc8cda0c\") " pod="opendatahub/model-serving-api-54d89bd5fc-pb6bw" Jul 07 17:45:33.151657 ip-10-0-129-39 kubenswrapper[2575]: I0707 17:45:33.151583 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/549a1f02-41ef-4b55-914b-f34bbc8cda0c-tls-certs\") pod \"model-serving-api-54d89bd5fc-pb6bw\" (UID: \"549a1f02-41ef-4b55-914b-f34bbc8cda0c\") " pod="opendatahub/model-serving-api-54d89bd5fc-pb6bw" Jul 07 17:45:33.154217 ip-10-0-129-39 kubenswrapper[2575]: I0707 17:45:33.154195 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/549a1f02-41ef-4b55-914b-f34bbc8cda0c-tls-certs\") pod \"model-serving-api-54d89bd5fc-pb6bw\" (UID: \"549a1f02-41ef-4b55-914b-f34bbc8cda0c\") " pod="opendatahub/model-serving-api-54d89bd5fc-pb6bw" Jul 07 17:45:33.168605 ip-10-0-129-39 kubenswrapper[2575]: I0707 17:45:33.168575 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zj8v6\" (UniqueName: \"kubernetes.io/projected/549a1f02-41ef-4b55-914b-f34bbc8cda0c-kube-api-access-zj8v6\") pod \"model-serving-api-54d89bd5fc-pb6bw\" (UID: \"549a1f02-41ef-4b55-914b-f34bbc8cda0c\") " pod="opendatahub/model-serving-api-54d89bd5fc-pb6bw" Jul 07 17:45:33.182325 ip-10-0-129-39 kubenswrapper[2575]: I0707 17:45:33.182295 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/model-serving-api-54d89bd5fc-pb6bw" Jul 07 17:45:33.223901 ip-10-0-129-39 kubenswrapper[2575]: I0707 17:45:33.223869 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/odh-model-controller-7998c67dc4-km2nt" event={"ID":"a9875c1b-dca8-4038-b6e0-2ce204c410a0","Type":"ContainerStarted","Data":"88d2994e0db291f21b2a218ba392a74fcffb0a50f2af0bbb1424819742d6f95c"} Jul 07 17:45:33.225353 ip-10-0-129-39 kubenswrapper[2575]: I0707 17:45:33.225321 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/kube-auth-proxy-544cc96c69-v9zdn" event={"ID":"5fda3a91-211d-4d8c-8237-df8287d92b46","Type":"ContainerStarted","Data":"007d5be970a59153c16d9f4c126628c997b8eb8f26efc08dfc4e4e283c610529"} Jul 07 17:45:33.343583 ip-10-0-129-39 kubenswrapper[2575]: I0707 17:45:33.343550 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/model-serving-api-54d89bd5fc-pb6bw"] Jul 07 17:45:33.347197 ip-10-0-129-39 kubenswrapper[2575]: W0707 17:45:33.347171 2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod549a1f02_41ef_4b55_914b_f34bbc8cda0c.slice/crio-f406e99db97d337c8587754386006303a721db39e9481684482ad23da351b154 WatchSource:0}: Error finding container f406e99db97d337c8587754386006303a721db39e9481684482ad23da351b154: Status 404 returned error can't find the container with id f406e99db97d337c8587754386006303a721db39e9481684482ad23da351b154 Jul 07 17:45:33.828785 ip-10-0-129-39 kubenswrapper[2575]: I0707 17:45:33.828753 2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7574459656-mpfz4"] Jul 07 17:45:33.831546 ip-10-0-129-39 kubenswrapper[2575]: I0707 17:45:33.831528 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7574459656-mpfz4" Jul 07 17:45:33.834195 ip-10-0-129-39 kubenswrapper[2575]: I0707 17:45:33.834174 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"cacerts\"" Jul 07 17:45:33.834521 ip-10-0-129-39 kubenswrapper[2575]: I0707 17:45:33.834480 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istiod-tls\"" Jul 07 17:45:33.834604 ip-10-0-129-39 kubenswrapper[2575]: I0707 17:45:33.834593 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istio-kubeconfig\"" Jul 07 17:45:33.834740 ip-10-0-129-39 kubenswrapper[2575]: I0707 17:45:33.834725 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istiod-openshift-gateway-dockercfg-jmpgc\"" Jul 07 17:45:33.834931 ip-10-0-129-39 kubenswrapper[2575]: I0707 17:45:33.834907 2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-gw-ca-root-cert\"" Jul 07 17:45:33.847594 ip-10-0-129-39 kubenswrapper[2575]: I0707 17:45:33.847572 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7574459656-mpfz4"] Jul 07 17:45:33.958838 ip-10-0-129-39 kubenswrapper[2575]: I0707 17:45:33.958809 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/c1a4828b-4d80-41ae-b008-722cf9f308bc-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7574459656-mpfz4\" (UID: \"c1a4828b-4d80-41ae-b008-722cf9f308bc\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-mpfz4" Jul 07 17:45:33.958996 ip-10-0-129-39 kubenswrapper[2575]: I0707 17:45:33.958848 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/c1a4828b-4d80-41ae-b008-722cf9f308bc-istio-kubeconfig\") pod \"istiod-openshift-gateway-7574459656-mpfz4\" (UID: \"c1a4828b-4d80-41ae-b008-722cf9f308bc\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-mpfz4" Jul 07 17:45:33.958996 ip-10-0-129-39 kubenswrapper[2575]: I0707 17:45:33.958870 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/c1a4828b-4d80-41ae-b008-722cf9f308bc-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7574459656-mpfz4\" (UID: \"c1a4828b-4d80-41ae-b008-722cf9f308bc\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-mpfz4" Jul 07 17:45:33.958996 ip-10-0-129-39 kubenswrapper[2575]: I0707 17:45:33.958958 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-c9qvq\" (UniqueName: \"kubernetes.io/projected/c1a4828b-4d80-41ae-b008-722cf9f308bc-kube-api-access-c9qvq\") pod \"istiod-openshift-gateway-7574459656-mpfz4\" (UID: \"c1a4828b-4d80-41ae-b008-722cf9f308bc\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-mpfz4" Jul 07 17:45:33.959157 ip-10-0-129-39 kubenswrapper[2575]: I0707 17:45:33.959062 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c1a4828b-4d80-41ae-b008-722cf9f308bc-istio-token\") pod \"istiod-openshift-gateway-7574459656-mpfz4\" (UID: \"c1a4828b-4d80-41ae-b008-722cf9f308bc\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-mpfz4" Jul 07 17:45:33.959157 ip-10-0-129-39 kubenswrapper[2575]: I0707 17:45:33.959106 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/c1a4828b-4d80-41ae-b008-722cf9f308bc-cacerts\") pod \"istiod-openshift-gateway-7574459656-mpfz4\" (UID: \"c1a4828b-4d80-41ae-b008-722cf9f308bc\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-mpfz4" Jul 07 17:45:33.959157 ip-10-0-129-39 kubenswrapper[2575]: I0707 17:45:33.959136 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/c1a4828b-4d80-41ae-b008-722cf9f308bc-local-certs\") pod \"istiod-openshift-gateway-7574459656-mpfz4\" (UID: \"c1a4828b-4d80-41ae-b008-722cf9f308bc\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-mpfz4" Jul 07 17:45:34.059712 ip-10-0-129-39 kubenswrapper[2575]: I0707 17:45:34.059682 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c1a4828b-4d80-41ae-b008-722cf9f308bc-istio-token\") pod \"istiod-openshift-gateway-7574459656-mpfz4\" (UID: \"c1a4828b-4d80-41ae-b008-722cf9f308bc\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-mpfz4" Jul 07 17:45:34.059869 ip-10-0-129-39 kubenswrapper[2575]: I0707 17:45:34.059728 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/c1a4828b-4d80-41ae-b008-722cf9f308bc-cacerts\") pod \"istiod-openshift-gateway-7574459656-mpfz4\" (UID: \"c1a4828b-4d80-41ae-b008-722cf9f308bc\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-mpfz4" Jul 07 17:45:34.059869 ip-10-0-129-39 kubenswrapper[2575]: I0707 17:45:34.059755 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/c1a4828b-4d80-41ae-b008-722cf9f308bc-local-certs\") pod \"istiod-openshift-gateway-7574459656-mpfz4\" (UID: \"c1a4828b-4d80-41ae-b008-722cf9f308bc\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-mpfz4" Jul 07 17:45:34.059869 ip-10-0-129-39 kubenswrapper[2575]: I0707 17:45:34.059785 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/c1a4828b-4d80-41ae-b008-722cf9f308bc-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7574459656-mpfz4\" (UID: \"c1a4828b-4d80-41ae-b008-722cf9f308bc\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-mpfz4" Jul 07 17:45:34.059869 ip-10-0-129-39 kubenswrapper[2575]: I0707 17:45:34.059825 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/c1a4828b-4d80-41ae-b008-722cf9f308bc-istio-kubeconfig\") pod \"istiod-openshift-gateway-7574459656-mpfz4\" (UID: \"c1a4828b-4d80-41ae-b008-722cf9f308bc\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-mpfz4" Jul 07 17:45:34.059869 ip-10-0-129-39 kubenswrapper[2575]: I0707 17:45:34.059853 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/c1a4828b-4d80-41ae-b008-722cf9f308bc-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7574459656-mpfz4\" (UID: \"c1a4828b-4d80-41ae-b008-722cf9f308bc\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-mpfz4" Jul 07 17:45:34.060115 ip-10-0-129-39 kubenswrapper[2575]: I0707 17:45:34.059909 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c9qvq\" (UniqueName: \"kubernetes.io/projected/c1a4828b-4d80-41ae-b008-722cf9f308bc-kube-api-access-c9qvq\") pod \"istiod-openshift-gateway-7574459656-mpfz4\" (UID: \"c1a4828b-4d80-41ae-b008-722cf9f308bc\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-mpfz4" Jul 07 17:45:34.060802 ip-10-0-129-39 kubenswrapper[2575]: I0707 17:45:34.060755 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/c1a4828b-4d80-41ae-b008-722cf9f308bc-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7574459656-mpfz4\" (UID: \"c1a4828b-4d80-41ae-b008-722cf9f308bc\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-mpfz4" Jul 07 17:45:34.062867 ip-10-0-129-39 kubenswrapper[2575]: I0707 17:45:34.062844 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/c1a4828b-4d80-41ae-b008-722cf9f308bc-local-certs\") pod \"istiod-openshift-gateway-7574459656-mpfz4\" (UID: \"c1a4828b-4d80-41ae-b008-722cf9f308bc\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-mpfz4" Jul 07 17:45:34.063526 ip-10-0-129-39 kubenswrapper[2575]: I0707 17:45:34.063434 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/c1a4828b-4d80-41ae-b008-722cf9f308bc-istio-kubeconfig\") pod \"istiod-openshift-gateway-7574459656-mpfz4\" (UID: \"c1a4828b-4d80-41ae-b008-722cf9f308bc\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-mpfz4" Jul 07 17:45:34.063526 ip-10-0-129-39 kubenswrapper[2575]: I0707 17:45:34.063485 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/c1a4828b-4d80-41ae-b008-722cf9f308bc-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7574459656-mpfz4\" (UID: \"c1a4828b-4d80-41ae-b008-722cf9f308bc\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-mpfz4" Jul 07 17:45:34.065414 ip-10-0-129-39 kubenswrapper[2575]: I0707 17:45:34.065389 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/c1a4828b-4d80-41ae-b008-722cf9f308bc-cacerts\") pod \"istiod-openshift-gateway-7574459656-mpfz4\" (UID: \"c1a4828b-4d80-41ae-b008-722cf9f308bc\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-mpfz4" Jul 07 17:45:34.070621 ip-10-0-129-39 kubenswrapper[2575]: I0707 17:45:34.070599 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c1a4828b-4d80-41ae-b008-722cf9f308bc-istio-token\") pod \"istiod-openshift-gateway-7574459656-mpfz4\" (UID: \"c1a4828b-4d80-41ae-b008-722cf9f308bc\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-mpfz4" Jul 07 17:45:34.072349 ip-10-0-129-39 kubenswrapper[2575]: I0707 17:45:34.072330 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-c9qvq\" (UniqueName: \"kubernetes.io/projected/c1a4828b-4d80-41ae-b008-722cf9f308bc-kube-api-access-c9qvq\") pod \"istiod-openshift-gateway-7574459656-mpfz4\" (UID: \"c1a4828b-4d80-41ae-b008-722cf9f308bc\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-mpfz4" Jul 07 17:45:34.141207 ip-10-0-129-39 kubenswrapper[2575]: I0707 17:45:34.141139 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7574459656-mpfz4" Jul 07 17:45:34.252266 ip-10-0-129-39 kubenswrapper[2575]: I0707 17:45:34.252152 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/model-serving-api-54d89bd5fc-pb6bw" event={"ID":"549a1f02-41ef-4b55-914b-f34bbc8cda0c","Type":"ContainerStarted","Data":"f406e99db97d337c8587754386006303a721db39e9481684482ad23da351b154"} Jul 07 17:45:34.346344 ip-10-0-129-39 kubenswrapper[2575]: I0707 17:45:34.345973 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7574459656-mpfz4"] Jul 07 17:45:34.349891 ip-10-0-129-39 kubenswrapper[2575]: W0707 17:45:34.349851 2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc1a4828b_4d80_41ae_b008_722cf9f308bc.slice/crio-c49a7cb3fe98ca0c69d9cb6590a0f4af32f0232139f580cc6c5754c4b411e89d WatchSource:0}: Error finding container c49a7cb3fe98ca0c69d9cb6590a0f4af32f0232139f580cc6c5754c4b411e89d: Status 404 returned error can't find the container with id c49a7cb3fe98ca0c69d9cb6590a0f4af32f0232139f580cc6c5754c4b411e89d Jul 07 17:45:35.257653 ip-10-0-129-39 kubenswrapper[2575]: I0707 17:45:35.257613 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7574459656-mpfz4" event={"ID":"c1a4828b-4d80-41ae-b008-722cf9f308bc","Type":"ContainerStarted","Data":"c49a7cb3fe98ca0c69d9cb6590a0f4af32f0232139f580cc6c5754c4b411e89d"} Jul 07 17:45:37.269253 ip-10-0-129-39 kubenswrapper[2575]: I0707 17:45:37.269147 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/kube-auth-proxy-544cc96c69-v9zdn" event={"ID":"5fda3a91-211d-4d8c-8237-df8287d92b46","Type":"ContainerStarted","Data":"3515176d4a2971aa9cd57d6750a22bd75be35952d5bee0e503c3efc2cc5226ea"} Jul 07 17:45:37.270858 ip-10-0-129-39 kubenswrapper[2575]: I0707 17:45:37.270822 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/model-serving-api-54d89bd5fc-pb6bw" event={"ID":"549a1f02-41ef-4b55-914b-f34bbc8cda0c","Type":"ContainerStarted","Data":"e29d6291976deeb2045aec346e450abe3ebf8c8041e5e55ec91dd4f442981359"} Jul 07 17:45:37.270982 ip-10-0-129-39 kubenswrapper[2575]: I0707 17:45:37.270888 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/model-serving-api-54d89bd5fc-pb6bw" Jul 07 17:45:37.272697 ip-10-0-129-39 kubenswrapper[2575]: I0707 17:45:37.272657 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/odh-model-controller-7998c67dc4-km2nt" event={"ID":"a9875c1b-dca8-4038-b6e0-2ce204c410a0","Type":"ContainerStarted","Data":"564959e1defac95e52a22769495f65f97de1cb4ba8788c41973d63767e701110"} Jul 07 17:45:37.272796 ip-10-0-129-39 kubenswrapper[2575]: I0707 17:45:37.272720 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/odh-model-controller-7998c67dc4-km2nt" Jul 07 17:45:37.295047 ip-10-0-129-39 kubenswrapper[2575]: I0707 17:45:37.294998 2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/kube-auth-proxy-544cc96c69-v9zdn" podStartSLOduration=1.449159373 podStartE2EDuration="6.294979169s" totalImagesPullingTime="4.845819796s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-07 17:45:31 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 17:45:37.29292161 +0000 UTC m=+297.798728277" watchObservedRunningTime="2026-07-07 17:45:37.294979169 +0000 UTC m=+297.800785836" Jul 07 17:45:37.325064 ip-10-0-129-39 kubenswrapper[2575]: I0707 17:45:37.325024 2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/odh-model-controller-7998c67dc4-km2nt" podStartSLOduration=2.18246379 podStartE2EDuration="6.32501031s" totalImagesPullingTime="4.14254652s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-07 17:45:31 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 17:45:37.323874231 +0000 UTC m=+297.829680910" watchObservedRunningTime="2026-07-07 17:45:37.32501031 +0000 UTC m=+297.830816976" Jul 07 17:45:37.356661 ip-10-0-129-39 kubenswrapper[2575]: I0707 17:45:37.356560 2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/model-serving-api-54d89bd5fc-pb6bw" podStartSLOduration=1.600620229 podStartE2EDuration="5.35653629s" totalImagesPullingTime="3.755916061s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-07 17:45:32 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 17:45:37.354483606 +0000 UTC m=+297.860290273" watchObservedRunningTime="2026-07-07 17:45:37.35653629 +0000 UTC m=+297.862342961" Jul 07 17:45:38.401554 ip-10-0-129-39 kubenswrapper[2575]: I0707 17:45:38.401489 2575 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"} Jul 07 17:45:38.401832 ip-10-0-129-39 kubenswrapper[2575]: I0707 17:45:38.401596 2575 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"} Jul 07 17:45:39.282717 ip-10-0-129-39 kubenswrapper[2575]: I0707 17:45:39.282684 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7574459656-mpfz4" event={"ID":"c1a4828b-4d80-41ae-b008-722cf9f308bc","Type":"ContainerStarted","Data":"7164ae30f91e08fba08e7850359a49c3b8eb113fb26c4d3dd2ae35344d64e0d3"} Jul 07 17:45:39.282922 ip-10-0-129-39 kubenswrapper[2575]: I0707 17:45:39.282897 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/istiod-openshift-gateway-7574459656-mpfz4" Jul 07 17:45:39.285114 ip-10-0-129-39 kubenswrapper[2575]: I0707 17:45:39.284838 2575 patch_prober.go:28] interesting pod/istiod-openshift-gateway-7574459656-mpfz4 container/discovery namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Jul 07 17:45:39.285114 ip-10-0-129-39 kubenswrapper[2575]: I0707 17:45:39.284889 2575 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/istiod-openshift-gateway-7574459656-mpfz4" podUID="c1a4828b-4d80-41ae-b008-722cf9f308bc" containerName="discovery" probeResult="failure" output="HTTP probe failed with statuscode: 503" Jul 07 17:45:39.309575 ip-10-0-129-39 kubenswrapper[2575]: I0707 17:45:39.309528 2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/istiod-openshift-gateway-7574459656-mpfz4" podStartSLOduration=2.260880668 podStartE2EDuration="6.309491558s" totalImagesPullingTime="4.04861089s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-07 17:45:33 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 17:45:39.307593283 +0000 UTC m=+299.813399949" watchObservedRunningTime="2026-07-07 17:45:39.309491558 +0000 UTC m=+299.815298225" Jul 07 17:45:39.935050 ip-10-0-129-39 kubenswrapper[2575]: I0707 17:45:39.935027 2575 kubelet.go:1648] "Image garbage collection succeeded" Jul 07 17:45:40.287777 ip-10-0-129-39 kubenswrapper[2575]: I0707 17:45:40.287703 2575 patch_prober.go:28] interesting pod/istiod-openshift-gateway-7574459656-mpfz4 container/discovery namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Jul 07 17:45:40.287777 ip-10-0-129-39 kubenswrapper[2575]: I0707 17:45:40.287755 2575 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/istiod-openshift-gateway-7574459656-mpfz4" podUID="c1a4828b-4d80-41ae-b008-722cf9f308bc" containerName="discovery" probeResult="failure" output="HTTP probe failed with statuscode: 503" Jul 07 17:45:40.695848 ip-10-0-129-39 kubenswrapper[2575]: I0707 17:45:40.695814 2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4q24g5"] Jul 07 17:45:40.698119 ip-10-0-129-39 kubenswrapper[2575]: I0707 17:45:40.698103 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4q24g5" Jul 07 17:45:40.701561 ip-10-0-129-39 kubenswrapper[2575]: I0707 17:45:40.701540 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"data-science-gateway-data-science-gateway-class-dockercfg-rgwv2\"" Jul 07 17:45:40.728591 ip-10-0-129-39 kubenswrapper[2575]: I0707 17:45:40.728570 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4q24g5"] Jul 07 17:45:40.817724 ip-10-0-129-39 kubenswrapper[2575]: I0707 17:45:40.817696 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cac31641-45b4-464e-ba63-65ce20b7cfdb-istio-data\") pod \"data-science-gateway-data-science-gateway-class-6576859cf4q24g5\" (UID: \"cac31641-45b4-464e-ba63-65ce20b7cfdb\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4q24g5" Jul 07 17:45:40.817851 ip-10-0-129-39 kubenswrapper[2575]: I0707 17:45:40.817728 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6rxsl\" (UniqueName: \"kubernetes.io/projected/cac31641-45b4-464e-ba63-65ce20b7cfdb-kube-api-access-6rxsl\") pod \"data-science-gateway-data-science-gateway-class-6576859cf4q24g5\" (UID: \"cac31641-45b4-464e-ba63-65ce20b7cfdb\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4q24g5" Jul 07 17:45:40.817851 ip-10-0-129-39 kubenswrapper[2575]: I0707 17:45:40.817768 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cac31641-45b4-464e-ba63-65ce20b7cfdb-workload-socket\") pod \"data-science-gateway-data-science-gateway-class-6576859cf4q24g5\" (UID: \"cac31641-45b4-464e-ba63-65ce20b7cfdb\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4q24g5" Jul 07 17:45:40.817923 ip-10-0-129-39 kubenswrapper[2575]: I0707 17:45:40.817865 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cac31641-45b4-464e-ba63-65ce20b7cfdb-istio-podinfo\") pod \"data-science-gateway-data-science-gateway-class-6576859cf4q24g5\" (UID: \"cac31641-45b4-464e-ba63-65ce20b7cfdb\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4q24g5" Jul 07 17:45:40.817923 ip-10-0-129-39 kubenswrapper[2575]: I0707 17:45:40.817894 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cac31641-45b4-464e-ba63-65ce20b7cfdb-workload-certs\") pod \"data-science-gateway-data-science-gateway-class-6576859cf4q24g5\" (UID: \"cac31641-45b4-464e-ba63-65ce20b7cfdb\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4q24g5" Jul 07 17:45:40.817923 ip-10-0-129-39 kubenswrapper[2575]: I0707 17:45:40.817912 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cac31641-45b4-464e-ba63-65ce20b7cfdb-credential-socket\") pod \"data-science-gateway-data-science-gateway-class-6576859cf4q24g5\" (UID: \"cac31641-45b4-464e-ba63-65ce20b7cfdb\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4q24g5" Jul 07 17:45:40.818035 ip-10-0-129-39 kubenswrapper[2575]: I0707 17:45:40.817928 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cac31641-45b4-464e-ba63-65ce20b7cfdb-istio-envoy\") pod \"data-science-gateway-data-science-gateway-class-6576859cf4q24g5\" (UID: \"cac31641-45b4-464e-ba63-65ce20b7cfdb\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4q24g5" Jul 07 17:45:40.818035 ip-10-0-129-39 kubenswrapper[2575]: I0707 17:45:40.817960 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cac31641-45b4-464e-ba63-65ce20b7cfdb-istio-token\") pod \"data-science-gateway-data-science-gateway-class-6576859cf4q24g5\" (UID: \"cac31641-45b4-464e-ba63-65ce20b7cfdb\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4q24g5" Jul 07 17:45:40.818035 ip-10-0-129-39 kubenswrapper[2575]: I0707 17:45:40.818012 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cac31641-45b4-464e-ba63-65ce20b7cfdb-istiod-ca-cert\") pod \"data-science-gateway-data-science-gateway-class-6576859cf4q24g5\" (UID: \"cac31641-45b4-464e-ba63-65ce20b7cfdb\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4q24g5" Jul 07 17:45:40.918740 ip-10-0-129-39 kubenswrapper[2575]: I0707 17:45:40.918708 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cac31641-45b4-464e-ba63-65ce20b7cfdb-istio-data\") pod \"data-science-gateway-data-science-gateway-class-6576859cf4q24g5\" (UID: \"cac31641-45b4-464e-ba63-65ce20b7cfdb\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4q24g5" Jul 07 17:45:40.918740 ip-10-0-129-39 kubenswrapper[2575]: I0707 17:45:40.918742 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6rxsl\" (UniqueName: \"kubernetes.io/projected/cac31641-45b4-464e-ba63-65ce20b7cfdb-kube-api-access-6rxsl\") pod \"data-science-gateway-data-science-gateway-class-6576859cf4q24g5\" (UID: \"cac31641-45b4-464e-ba63-65ce20b7cfdb\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4q24g5" Jul 07 17:45:40.919008 ip-10-0-129-39 kubenswrapper[2575]: I0707 17:45:40.918772 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cac31641-45b4-464e-ba63-65ce20b7cfdb-workload-socket\") pod \"data-science-gateway-data-science-gateway-class-6576859cf4q24g5\" (UID: \"cac31641-45b4-464e-ba63-65ce20b7cfdb\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4q24g5" Jul 07 17:45:40.919008 ip-10-0-129-39 kubenswrapper[2575]: I0707 17:45:40.918810 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cac31641-45b4-464e-ba63-65ce20b7cfdb-istio-podinfo\") pod \"data-science-gateway-data-science-gateway-class-6576859cf4q24g5\" (UID: \"cac31641-45b4-464e-ba63-65ce20b7cfdb\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4q24g5" Jul 07 17:45:40.919008 ip-10-0-129-39 kubenswrapper[2575]: I0707 17:45:40.918828 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cac31641-45b4-464e-ba63-65ce20b7cfdb-workload-certs\") pod \"data-science-gateway-data-science-gateway-class-6576859cf4q24g5\" (UID: \"cac31641-45b4-464e-ba63-65ce20b7cfdb\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4q24g5" Jul 07 17:45:40.919008 ip-10-0-129-39 kubenswrapper[2575]: I0707 17:45:40.918846 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cac31641-45b4-464e-ba63-65ce20b7cfdb-credential-socket\") pod \"data-science-gateway-data-science-gateway-class-6576859cf4q24g5\" (UID: \"cac31641-45b4-464e-ba63-65ce20b7cfdb\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4q24g5" Jul 07 17:45:40.919008 ip-10-0-129-39 kubenswrapper[2575]: I0707 17:45:40.918860 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cac31641-45b4-464e-ba63-65ce20b7cfdb-istio-envoy\") pod \"data-science-gateway-data-science-gateway-class-6576859cf4q24g5\" (UID: \"cac31641-45b4-464e-ba63-65ce20b7cfdb\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4q24g5" Jul 07 17:45:40.919008 ip-10-0-129-39 kubenswrapper[2575]: I0707 17:45:40.918877 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cac31641-45b4-464e-ba63-65ce20b7cfdb-istio-token\") pod \"data-science-gateway-data-science-gateway-class-6576859cf4q24g5\" (UID: \"cac31641-45b4-464e-ba63-65ce20b7cfdb\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4q24g5" Jul 07 17:45:40.919008 ip-10-0-129-39 kubenswrapper[2575]: I0707 17:45:40.918931 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cac31641-45b4-464e-ba63-65ce20b7cfdb-istiod-ca-cert\") pod \"data-science-gateway-data-science-gateway-class-6576859cf4q24g5\" (UID: \"cac31641-45b4-464e-ba63-65ce20b7cfdb\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4q24g5" Jul 07 17:45:40.919349 ip-10-0-129-39 kubenswrapper[2575]: I0707 17:45:40.919171 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cac31641-45b4-464e-ba63-65ce20b7cfdb-istio-data\") pod \"data-science-gateway-data-science-gateway-class-6576859cf4q24g5\" (UID: \"cac31641-45b4-464e-ba63-65ce20b7cfdb\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4q24g5" Jul 07 17:45:40.919349 ip-10-0-129-39 kubenswrapper[2575]: I0707 17:45:40.919281 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cac31641-45b4-464e-ba63-65ce20b7cfdb-credential-socket\") pod \"data-science-gateway-data-science-gateway-class-6576859cf4q24g5\" (UID: \"cac31641-45b4-464e-ba63-65ce20b7cfdb\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4q24g5" Jul 07 17:45:40.919349 ip-10-0-129-39 kubenswrapper[2575]: I0707 17:45:40.919326 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cac31641-45b4-464e-ba63-65ce20b7cfdb-workload-socket\") pod \"data-science-gateway-data-science-gateway-class-6576859cf4q24g5\" (UID: \"cac31641-45b4-464e-ba63-65ce20b7cfdb\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4q24g5" Jul 07 17:45:40.919579 ip-10-0-129-39 kubenswrapper[2575]: I0707 17:45:40.919558 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cac31641-45b4-464e-ba63-65ce20b7cfdb-istiod-ca-cert\") pod \"data-science-gateway-data-science-gateway-class-6576859cf4q24g5\" (UID: \"cac31641-45b4-464e-ba63-65ce20b7cfdb\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4q24g5" Jul 07 17:45:40.919658 ip-10-0-129-39 kubenswrapper[2575]: I0707 17:45:40.919607 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cac31641-45b4-464e-ba63-65ce20b7cfdb-workload-certs\") pod \"data-science-gateway-data-science-gateway-class-6576859cf4q24g5\" (UID: \"cac31641-45b4-464e-ba63-65ce20b7cfdb\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4q24g5" Jul 07 17:45:40.921210 ip-10-0-129-39 kubenswrapper[2575]: I0707 17:45:40.921182 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cac31641-45b4-464e-ba63-65ce20b7cfdb-istio-envoy\") pod \"data-science-gateway-data-science-gateway-class-6576859cf4q24g5\" (UID: \"cac31641-45b4-464e-ba63-65ce20b7cfdb\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4q24g5" Jul 07 17:45:40.921338 ip-10-0-129-39 kubenswrapper[2575]: I0707 17:45:40.921307 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cac31641-45b4-464e-ba63-65ce20b7cfdb-istio-podinfo\") pod \"data-science-gateway-data-science-gateway-class-6576859cf4q24g5\" (UID: \"cac31641-45b4-464e-ba63-65ce20b7cfdb\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4q24g5" Jul 07 17:45:40.932748 ip-10-0-129-39 kubenswrapper[2575]: I0707 17:45:40.932724 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cac31641-45b4-464e-ba63-65ce20b7cfdb-istio-token\") pod \"data-science-gateway-data-science-gateway-class-6576859cf4q24g5\" (UID: \"cac31641-45b4-464e-ba63-65ce20b7cfdb\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4q24g5" Jul 07 17:45:40.942822 ip-10-0-129-39 kubenswrapper[2575]: I0707 17:45:40.942802 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6rxsl\" (UniqueName: \"kubernetes.io/projected/cac31641-45b4-464e-ba63-65ce20b7cfdb-kube-api-access-6rxsl\") pod \"data-science-gateway-data-science-gateway-class-6576859cf4q24g5\" (UID: \"cac31641-45b4-464e-ba63-65ce20b7cfdb\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4q24g5" Jul 07 17:45:41.007735 ip-10-0-129-39 kubenswrapper[2575]: I0707 17:45:41.007681 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4q24g5" Jul 07 17:45:41.232897 ip-10-0-129-39 kubenswrapper[2575]: I0707 17:45:41.232866 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4q24g5"] Jul 07 17:45:41.235894 ip-10-0-129-39 kubenswrapper[2575]: W0707 17:45:41.235866 2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcac31641_45b4_464e_ba63_65ce20b7cfdb.slice/crio-a01098e115a68a34a71294f620bfc64398b54e62be11a578fb65c1bc0def93fd WatchSource:0}: Error finding container a01098e115a68a34a71294f620bfc64398b54e62be11a578fb65c1bc0def93fd: Status 404 returned error can't find the container with id a01098e115a68a34a71294f620bfc64398b54e62be11a578fb65c1bc0def93fd Jul 07 17:45:41.237899 ip-10-0-129-39 kubenswrapper[2575]: I0707 17:45:41.237885 2575 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Jul 07 17:45:41.291376 ip-10-0-129-39 kubenswrapper[2575]: I0707 17:45:41.291319 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4q24g5" event={"ID":"cac31641-45b4-464e-ba63-65ce20b7cfdb","Type":"ContainerStarted","Data":"a01098e115a68a34a71294f620bfc64398b54e62be11a578fb65c1bc0def93fd"} Jul 07 17:45:43.288671 ip-10-0-129-39 kubenswrapper[2575]: I0707 17:45:43.288632 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/istiod-openshift-gateway-7574459656-mpfz4" Jul 07 17:45:44.162447 ip-10-0-129-39 kubenswrapper[2575]: I0707 17:45:44.162404 2575 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"} Jul 07 17:45:44.162539 ip-10-0-129-39 kubenswrapper[2575]: I0707 17:45:44.162474 2575 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"} Jul 07 17:45:44.162539 ip-10-0-129-39 kubenswrapper[2575]: I0707 17:45:44.162517 2575 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"} Jul 07 17:45:44.307844 ip-10-0-129-39 kubenswrapper[2575]: I0707 17:45:44.307809 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4q24g5" event={"ID":"cac31641-45b4-464e-ba63-65ce20b7cfdb","Type":"ContainerStarted","Data":"bea02eae9360d22389f7fbe7f727f27d76a7e3d5fc5395e5b3e96c66a85a83ef"} Jul 07 17:45:44.361804 ip-10-0-129-39 kubenswrapper[2575]: I0707 17:45:44.361749 2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4q24g5" podStartSLOduration=1.437567952 podStartE2EDuration="4.361727048s" totalImagesPullingTime="2.924159096s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-07 17:45:40 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 17:45:44.360566188 +0000 UTC m=+304.866372856" watchObservedRunningTime="2026-07-07 17:45:44.361727048 +0000 UTC m=+304.867533715" Jul 07 17:45:44.674297 ip-10-0-129-39 kubenswrapper[2575]: I0707 17:45:44.674261 2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/kserve-controller-manager-d6f4b65b8-d2drf"] Jul 07 17:45:44.677639 ip-10-0-129-39 kubenswrapper[2575]: I0707 17:45:44.677623 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/kserve-controller-manager-d6f4b65b8-d2drf" Jul 07 17:45:44.689126 ip-10-0-129-39 kubenswrapper[2575]: I0707 17:45:44.689103 2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/llmisvc-controller-manager-777c6bd7d5-8nrlc"] Jul 07 17:45:44.695114 ip-10-0-129-39 kubenswrapper[2575]: I0707 17:45:44.695092 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"kserve-webhook-server-cert\"" Jul 07 17:45:44.695228 ip-10-0-129-39 kubenswrapper[2575]: I0707 17:45:44.695208 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"kserve-controller-manager-dockercfg-thzxr\"" Jul 07 17:45:44.697463 ip-10-0-129-39 kubenswrapper[2575]: I0707 17:45:44.697447 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-8nrlc" Jul 07 17:45:44.714278 ip-10-0-129-39 kubenswrapper[2575]: I0707 17:45:44.714257 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"llmisvc-webhook-server-cert\"" Jul 07 17:45:44.714365 ip-10-0-129-39 kubenswrapper[2575]: I0707 17:45:44.714283 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"llmisvc-controller-manager-dockercfg-r2ddc\"" Jul 07 17:45:44.747209 ip-10-0-129-39 kubenswrapper[2575]: I0707 17:45:44.747185 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/kserve-controller-manager-d6f4b65b8-d2drf"] Jul 07 17:45:44.751310 ip-10-0-129-39 kubenswrapper[2575]: I0707 17:45:44.751280 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/5f66a7de-f49a-4ff3-ab89-b72709548dbf-cert\") pod \"llmisvc-controller-manager-777c6bd7d5-8nrlc\" (UID: \"5f66a7de-f49a-4ff3-ab89-b72709548dbf\") " pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-8nrlc" Jul 07 17:45:44.751452 ip-10-0-129-39 kubenswrapper[2575]: I0707 17:45:44.751370 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vbgn9\" (UniqueName: \"kubernetes.io/projected/5f66a7de-f49a-4ff3-ab89-b72709548dbf-kube-api-access-vbgn9\") pod \"llmisvc-controller-manager-777c6bd7d5-8nrlc\" (UID: \"5f66a7de-f49a-4ff3-ab89-b72709548dbf\") " pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-8nrlc" Jul 07 17:45:44.751452 ip-10-0-129-39 kubenswrapper[2575]: I0707 17:45:44.751415 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/8d9db521-9487-4db9-9d32-9870e1739492-cert\") pod \"kserve-controller-manager-d6f4b65b8-d2drf\" (UID: \"8d9db521-9487-4db9-9d32-9870e1739492\") " pod="opendatahub/kserve-controller-manager-d6f4b65b8-d2drf" Jul 07 17:45:44.751593 ip-10-0-129-39 kubenswrapper[2575]: I0707 17:45:44.751448 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gfzq6\" (UniqueName: \"kubernetes.io/projected/8d9db521-9487-4db9-9d32-9870e1739492-kube-api-access-gfzq6\") pod \"kserve-controller-manager-d6f4b65b8-d2drf\" (UID: \"8d9db521-9487-4db9-9d32-9870e1739492\") " pod="opendatahub/kserve-controller-manager-d6f4b65b8-d2drf" Jul 07 17:45:44.852035 ip-10-0-129-39 kubenswrapper[2575]: I0707 17:45:44.852002 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vbgn9\" (UniqueName: \"kubernetes.io/projected/5f66a7de-f49a-4ff3-ab89-b72709548dbf-kube-api-access-vbgn9\") pod \"llmisvc-controller-manager-777c6bd7d5-8nrlc\" (UID: \"5f66a7de-f49a-4ff3-ab89-b72709548dbf\") " pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-8nrlc" Jul 07 17:45:44.852180 ip-10-0-129-39 kubenswrapper[2575]: I0707 17:45:44.852050 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/8d9db521-9487-4db9-9d32-9870e1739492-cert\") pod \"kserve-controller-manager-d6f4b65b8-d2drf\" (UID: \"8d9db521-9487-4db9-9d32-9870e1739492\") " pod="opendatahub/kserve-controller-manager-d6f4b65b8-d2drf" Jul 07 17:45:44.852180 ip-10-0-129-39 kubenswrapper[2575]: I0707 17:45:44.852080 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gfzq6\" (UniqueName: \"kubernetes.io/projected/8d9db521-9487-4db9-9d32-9870e1739492-kube-api-access-gfzq6\") pod \"kserve-controller-manager-d6f4b65b8-d2drf\" (UID: \"8d9db521-9487-4db9-9d32-9870e1739492\") " pod="opendatahub/kserve-controller-manager-d6f4b65b8-d2drf" Jul 07 17:45:44.852180 ip-10-0-129-39 kubenswrapper[2575]: I0707 17:45:44.852107 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/5f66a7de-f49a-4ff3-ab89-b72709548dbf-cert\") pod \"llmisvc-controller-manager-777c6bd7d5-8nrlc\" (UID: \"5f66a7de-f49a-4ff3-ab89-b72709548dbf\") " pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-8nrlc" Jul 07 17:45:44.852358 ip-10-0-129-39 kubenswrapper[2575]: I0707 17:45:44.852320 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/llmisvc-controller-manager-777c6bd7d5-8nrlc"] Jul 07 17:45:44.854769 ip-10-0-129-39 kubenswrapper[2575]: I0707 17:45:44.854743 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/8d9db521-9487-4db9-9d32-9870e1739492-cert\") pod \"kserve-controller-manager-d6f4b65b8-d2drf\" (UID: \"8d9db521-9487-4db9-9d32-9870e1739492\") " pod="opendatahub/kserve-controller-manager-d6f4b65b8-d2drf" Jul 07 17:45:44.854892 ip-10-0-129-39 kubenswrapper[2575]: I0707 17:45:44.854746 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/5f66a7de-f49a-4ff3-ab89-b72709548dbf-cert\") pod \"llmisvc-controller-manager-777c6bd7d5-8nrlc\" (UID: \"5f66a7de-f49a-4ff3-ab89-b72709548dbf\") " pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-8nrlc" Jul 07 17:45:44.871732 ip-10-0-129-39 kubenswrapper[2575]: I0707 17:45:44.871701 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gfzq6\" (UniqueName: \"kubernetes.io/projected/8d9db521-9487-4db9-9d32-9870e1739492-kube-api-access-gfzq6\") pod \"kserve-controller-manager-d6f4b65b8-d2drf\" (UID: \"8d9db521-9487-4db9-9d32-9870e1739492\") " pod="opendatahub/kserve-controller-manager-d6f4b65b8-d2drf" Jul 07 17:45:44.873548 ip-10-0-129-39 kubenswrapper[2575]: I0707 17:45:44.873528 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vbgn9\" (UniqueName: \"kubernetes.io/projected/5f66a7de-f49a-4ff3-ab89-b72709548dbf-kube-api-access-vbgn9\") pod \"llmisvc-controller-manager-777c6bd7d5-8nrlc\" (UID: \"5f66a7de-f49a-4ff3-ab89-b72709548dbf\") " pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-8nrlc" Jul 07 17:45:44.988407 ip-10-0-129-39 kubenswrapper[2575]: I0707 17:45:44.988338 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/kserve-controller-manager-d6f4b65b8-d2drf" Jul 07 17:45:45.006492 ip-10-0-129-39 kubenswrapper[2575]: I0707 17:45:45.006472 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-8nrlc" Jul 07 17:45:45.008662 ip-10-0-129-39 kubenswrapper[2575]: I0707 17:45:45.008631 2575 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4q24g5" Jul 07 17:45:45.013936 ip-10-0-129-39 kubenswrapper[2575]: I0707 17:45:45.013917 2575 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4q24g5" Jul 07 17:45:45.143262 ip-10-0-129-39 kubenswrapper[2575]: I0707 17:45:45.143185 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/kserve-controller-manager-d6f4b65b8-d2drf"] Jul 07 17:45:45.145478 ip-10-0-129-39 kubenswrapper[2575]: W0707 17:45:45.145447 2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8d9db521_9487_4db9_9d32_9870e1739492.slice/crio-44debc2ebf192aa93d715bd1a279838fc219b01ae15e89890d7d41c1324f7de6 WatchSource:0}: Error finding container 44debc2ebf192aa93d715bd1a279838fc219b01ae15e89890d7d41c1324f7de6: Status 404 returned error can't find the container with id 44debc2ebf192aa93d715bd1a279838fc219b01ae15e89890d7d41c1324f7de6 Jul 07 17:45:45.174625 ip-10-0-129-39 kubenswrapper[2575]: I0707 17:45:45.174600 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/llmisvc-controller-manager-777c6bd7d5-8nrlc"] Jul 07 17:45:45.178189 ip-10-0-129-39 kubenswrapper[2575]: W0707 17:45:45.178161 2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-pod5f66a7de_f49a_4ff3_ab89_b72709548dbf.slice/crio-c83073ba90a456d72eb480f1e9c2ad72ea7bca9ce5e7283be13a96247871de13 WatchSource:0}: Error finding container c83073ba90a456d72eb480f1e9c2ad72ea7bca9ce5e7283be13a96247871de13: Status 404 returned error can't find the container with id c83073ba90a456d72eb480f1e9c2ad72ea7bca9ce5e7283be13a96247871de13 Jul 07 17:45:45.312634 ip-10-0-129-39 kubenswrapper[2575]: I0707 17:45:45.312570 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/kserve-controller-manager-d6f4b65b8-d2drf" event={"ID":"8d9db521-9487-4db9-9d32-9870e1739492","Type":"ContainerStarted","Data":"44debc2ebf192aa93d715bd1a279838fc219b01ae15e89890d7d41c1324f7de6"} Jul 07 17:45:45.313730 ip-10-0-129-39 kubenswrapper[2575]: I0707 17:45:45.313708 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-8nrlc" event={"ID":"5f66a7de-f49a-4ff3-ab89-b72709548dbf","Type":"ContainerStarted","Data":"c83073ba90a456d72eb480f1e9c2ad72ea7bca9ce5e7283be13a96247871de13"} Jul 07 17:45:45.313889 ip-10-0-129-39 kubenswrapper[2575]: I0707 17:45:45.313873 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4q24g5" Jul 07 17:45:45.314726 ip-10-0-129-39 kubenswrapper[2575]: I0707 17:45:45.314711 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4q24g5" Jul 07 17:45:47.107875 ip-10-0-129-39 kubenswrapper[2575]: I0707 17:45:47.107846 2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7574459656-mpfz4"] Jul 07 17:45:47.108290 ip-10-0-129-39 kubenswrapper[2575]: I0707 17:45:47.108048 2575 kuberuntime_container.go:872] "Killing container with a grace period" pod="openshift-ingress/istiod-openshift-gateway-7574459656-mpfz4" podUID="c1a4828b-4d80-41ae-b008-722cf9f308bc" containerName="discovery" containerID="cri-o://7164ae30f91e08fba08e7850359a49c3b8eb113fb26c4d3dd2ae35344d64e0d3" gracePeriod=30 Jul 07 17:45:48.194237 ip-10-0-129-39 kubenswrapper[2575]: I0707 17:45:48.194215 2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7574459656-mpfz4" Jul 07 17:45:48.277750 ip-10-0-129-39 kubenswrapper[2575]: I0707 17:45:48.277726 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/c1a4828b-4d80-41ae-b008-722cf9f308bc-istio-kubeconfig\") pod \"c1a4828b-4d80-41ae-b008-722cf9f308bc\" (UID: \"c1a4828b-4d80-41ae-b008-722cf9f308bc\") " Jul 07 17:45:48.277876 ip-10-0-129-39 kubenswrapper[2575]: I0707 17:45:48.277756 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c1a4828b-4d80-41ae-b008-722cf9f308bc-istio-token\") pod \"c1a4828b-4d80-41ae-b008-722cf9f308bc\" (UID: \"c1a4828b-4d80-41ae-b008-722cf9f308bc\") " Jul 07 17:45:48.277876 ip-10-0-129-39 kubenswrapper[2575]: I0707 17:45:48.277784 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-c9qvq\" (UniqueName: \"kubernetes.io/projected/c1a4828b-4d80-41ae-b008-722cf9f308bc-kube-api-access-c9qvq\") pod \"c1a4828b-4d80-41ae-b008-722cf9f308bc\" (UID: \"c1a4828b-4d80-41ae-b008-722cf9f308bc\") " Jul 07 17:45:48.277876 ip-10-0-129-39 kubenswrapper[2575]: I0707 17:45:48.277810 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/c1a4828b-4d80-41ae-b008-722cf9f308bc-istio-csr-dns-cert\") pod \"c1a4828b-4d80-41ae-b008-722cf9f308bc\" (UID: \"c1a4828b-4d80-41ae-b008-722cf9f308bc\") " Jul 07 17:45:48.278225 ip-10-0-129-39 kubenswrapper[2575]: I0707 17:45:48.278199 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/c1a4828b-4d80-41ae-b008-722cf9f308bc-istio-csr-ca-configmap\") pod \"c1a4828b-4d80-41ae-b008-722cf9f308bc\" (UID: \"c1a4828b-4d80-41ae-b008-722cf9f308bc\") " Jul 07 17:45:48.278317 ip-10-0-129-39 kubenswrapper[2575]: I0707 17:45:48.278261 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/c1a4828b-4d80-41ae-b008-722cf9f308bc-cacerts\") pod \"c1a4828b-4d80-41ae-b008-722cf9f308bc\" (UID: \"c1a4828b-4d80-41ae-b008-722cf9f308bc\") " Jul 07 17:45:48.278317 ip-10-0-129-39 kubenswrapper[2575]: I0707 17:45:48.278311 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/c1a4828b-4d80-41ae-b008-722cf9f308bc-local-certs\") pod \"c1a4828b-4d80-41ae-b008-722cf9f308bc\" (UID: \"c1a4828b-4d80-41ae-b008-722cf9f308bc\") " Jul 07 17:45:48.278931 ip-10-0-129-39 kubenswrapper[2575]: I0707 17:45:48.278898 2575 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c1a4828b-4d80-41ae-b008-722cf9f308bc-istio-csr-ca-configmap" (OuterVolumeSpecName: "istio-csr-ca-configmap") pod "c1a4828b-4d80-41ae-b008-722cf9f308bc" (UID: "c1a4828b-4d80-41ae-b008-722cf9f308bc"). InnerVolumeSpecName "istio-csr-ca-configmap". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 07 17:45:48.280137 ip-10-0-129-39 kubenswrapper[2575]: I0707 17:45:48.279397 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/odh-model-controller-7998c67dc4-km2nt" Jul 07 17:45:48.282495 ip-10-0-129-39 kubenswrapper[2575]: I0707 17:45:48.282479 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/model-serving-api-54d89bd5fc-pb6bw" Jul 07 17:45:48.322113 ip-10-0-129-39 kubenswrapper[2575]: I0707 17:45:48.322079 2575 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c1a4828b-4d80-41ae-b008-722cf9f308bc-kube-api-access-c9qvq" (OuterVolumeSpecName: "kube-api-access-c9qvq") pod "c1a4828b-4d80-41ae-b008-722cf9f308bc" (UID: "c1a4828b-4d80-41ae-b008-722cf9f308bc"). InnerVolumeSpecName "kube-api-access-c9qvq". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 07 17:45:48.322229 ip-10-0-129-39 kubenswrapper[2575]: I0707 17:45:48.322165 2575 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c1a4828b-4d80-41ae-b008-722cf9f308bc-local-certs" (OuterVolumeSpecName: "local-certs") pod "c1a4828b-4d80-41ae-b008-722cf9f308bc" (UID: "c1a4828b-4d80-41ae-b008-722cf9f308bc"). InnerVolumeSpecName "local-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 07 17:45:48.322298 ip-10-0-129-39 kubenswrapper[2575]: I0707 17:45:48.322233 2575 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c1a4828b-4d80-41ae-b008-722cf9f308bc-istio-kubeconfig" (OuterVolumeSpecName: "istio-kubeconfig") pod "c1a4828b-4d80-41ae-b008-722cf9f308bc" (UID: "c1a4828b-4d80-41ae-b008-722cf9f308bc"). InnerVolumeSpecName "istio-kubeconfig". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 07 17:45:48.322298 ip-10-0-129-39 kubenswrapper[2575]: I0707 17:45:48.322237 2575 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c1a4828b-4d80-41ae-b008-722cf9f308bc-istio-token" (OuterVolumeSpecName: "istio-token") pod "c1a4828b-4d80-41ae-b008-722cf9f308bc" (UID: "c1a4828b-4d80-41ae-b008-722cf9f308bc"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 07 17:45:48.322298 ip-10-0-129-39 kubenswrapper[2575]: I0707 17:45:48.322284 2575 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c1a4828b-4d80-41ae-b008-722cf9f308bc-istio-csr-dns-cert" (OuterVolumeSpecName: "istio-csr-dns-cert") pod "c1a4828b-4d80-41ae-b008-722cf9f308bc" (UID: "c1a4828b-4d80-41ae-b008-722cf9f308bc"). InnerVolumeSpecName "istio-csr-dns-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 07 17:45:48.322406 ip-10-0-129-39 kubenswrapper[2575]: I0707 17:45:48.322331 2575 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c1a4828b-4d80-41ae-b008-722cf9f308bc-cacerts" (OuterVolumeSpecName: "cacerts") pod "c1a4828b-4d80-41ae-b008-722cf9f308bc" (UID: "c1a4828b-4d80-41ae-b008-722cf9f308bc"). InnerVolumeSpecName "cacerts". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 07 17:45:48.330583 ip-10-0-129-39 kubenswrapper[2575]: I0707 17:45:48.330564 2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7574459656-mpfz4" Jul 07 17:45:48.330583 ip-10-0-129-39 kubenswrapper[2575]: I0707 17:45:48.330563 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7574459656-mpfz4" event={"ID":"c1a4828b-4d80-41ae-b008-722cf9f308bc","Type":"ContainerDied","Data":"7164ae30f91e08fba08e7850359a49c3b8eb113fb26c4d3dd2ae35344d64e0d3"} Jul 07 17:45:48.330713 ip-10-0-129-39 kubenswrapper[2575]: I0707 17:45:48.330619 2575 scope.go:117] "RemoveContainer" containerID="7164ae30f91e08fba08e7850359a49c3b8eb113fb26c4d3dd2ae35344d64e0d3" Jul 07 17:45:48.330713 ip-10-0-129-39 kubenswrapper[2575]: I0707 17:45:48.330496 2575 generic.go:356] "Generic (PLEG): container finished" podID="c1a4828b-4d80-41ae-b008-722cf9f308bc" containerID="7164ae30f91e08fba08e7850359a49c3b8eb113fb26c4d3dd2ae35344d64e0d3" exitCode=0 Jul 07 17:45:48.330805 ip-10-0-129-39 kubenswrapper[2575]: I0707 17:45:48.330743 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7574459656-mpfz4" event={"ID":"c1a4828b-4d80-41ae-b008-722cf9f308bc","Type":"ContainerDied","Data":"c49a7cb3fe98ca0c69d9cb6590a0f4af32f0232139f580cc6c5754c4b411e89d"} Jul 07 17:45:48.339461 ip-10-0-129-39 kubenswrapper[2575]: I0707 17:45:48.339396 2575 scope.go:117] "RemoveContainer" containerID="7164ae30f91e08fba08e7850359a49c3b8eb113fb26c4d3dd2ae35344d64e0d3" Jul 07 17:45:48.339862 ip-10-0-129-39 kubenswrapper[2575]: E0707 17:45:48.339822 2575 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7164ae30f91e08fba08e7850359a49c3b8eb113fb26c4d3dd2ae35344d64e0d3\": container with ID starting with 7164ae30f91e08fba08e7850359a49c3b8eb113fb26c4d3dd2ae35344d64e0d3 not found: ID does not exist" containerID="7164ae30f91e08fba08e7850359a49c3b8eb113fb26c4d3dd2ae35344d64e0d3" Jul 07 17:45:48.339964 ip-10-0-129-39 kubenswrapper[2575]: I0707 17:45:48.339865 2575 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7164ae30f91e08fba08e7850359a49c3b8eb113fb26c4d3dd2ae35344d64e0d3"} err="failed to get container status \"7164ae30f91e08fba08e7850359a49c3b8eb113fb26c4d3dd2ae35344d64e0d3\": rpc error: code = NotFound desc = could not find container \"7164ae30f91e08fba08e7850359a49c3b8eb113fb26c4d3dd2ae35344d64e0d3\": container with ID starting with 7164ae30f91e08fba08e7850359a49c3b8eb113fb26c4d3dd2ae35344d64e0d3 not found: ID does not exist" Jul 07 17:45:48.379784 ip-10-0-129-39 kubenswrapper[2575]: I0707 17:45:48.379755 2575 reconciler_common.go:299] "Volume detached for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/c1a4828b-4d80-41ae-b008-722cf9f308bc-istio-kubeconfig\") on node \"ip-10-0-129-39.ec2.internal\" DevicePath \"\"" Jul 07 17:45:48.379784 ip-10-0-129-39 kubenswrapper[2575]: I0707 17:45:48.379786 2575 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c1a4828b-4d80-41ae-b008-722cf9f308bc-istio-token\") on node \"ip-10-0-129-39.ec2.internal\" DevicePath \"\"" Jul 07 17:45:48.380025 ip-10-0-129-39 kubenswrapper[2575]: I0707 17:45:48.379799 2575 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-c9qvq\" (UniqueName: \"kubernetes.io/projected/c1a4828b-4d80-41ae-b008-722cf9f308bc-kube-api-access-c9qvq\") on node \"ip-10-0-129-39.ec2.internal\" DevicePath \"\"" Jul 07 17:45:48.380025 ip-10-0-129-39 kubenswrapper[2575]: I0707 17:45:48.379812 2575 reconciler_common.go:299] "Volume detached for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/c1a4828b-4d80-41ae-b008-722cf9f308bc-istio-csr-dns-cert\") on node \"ip-10-0-129-39.ec2.internal\" DevicePath \"\"" Jul 07 17:45:48.380025 ip-10-0-129-39 kubenswrapper[2575]: I0707 17:45:48.379826 2575 reconciler_common.go:299] "Volume detached for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/c1a4828b-4d80-41ae-b008-722cf9f308bc-istio-csr-ca-configmap\") on node \"ip-10-0-129-39.ec2.internal\" DevicePath \"\"" Jul 07 17:45:48.380025 ip-10-0-129-39 kubenswrapper[2575]: I0707 17:45:48.379838 2575 reconciler_common.go:299] "Volume detached for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/c1a4828b-4d80-41ae-b008-722cf9f308bc-cacerts\") on node \"ip-10-0-129-39.ec2.internal\" DevicePath \"\"" Jul 07 17:45:48.380025 ip-10-0-129-39 kubenswrapper[2575]: I0707 17:45:48.379849 2575 reconciler_common.go:299] "Volume detached for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/c1a4828b-4d80-41ae-b008-722cf9f308bc-local-certs\") on node \"ip-10-0-129-39.ec2.internal\" DevicePath \"\"" Jul 07 17:45:48.423307 ip-10-0-129-39 kubenswrapper[2575]: I0707 17:45:48.423277 2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7574459656-mpfz4"] Jul 07 17:45:48.438914 ip-10-0-129-39 kubenswrapper[2575]: I0707 17:45:48.438840 2575 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7574459656-mpfz4"] Jul 07 17:45:49.337224 ip-10-0-129-39 kubenswrapper[2575]: I0707 17:45:49.337190 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/kserve-controller-manager-d6f4b65b8-d2drf" event={"ID":"8d9db521-9487-4db9-9d32-9870e1739492","Type":"ContainerStarted","Data":"2fc1f211e7100eb8a6568877a2d6f00294ba9f56da73d335a31d745c75609da0"} Jul 07 17:45:49.337689 ip-10-0-129-39 kubenswrapper[2575]: I0707 17:45:49.337289 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/kserve-controller-manager-d6f4b65b8-d2drf" Jul 07 17:45:49.338739 ip-10-0-129-39 kubenswrapper[2575]: I0707 17:45:49.338715 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-8nrlc" event={"ID":"5f66a7de-f49a-4ff3-ab89-b72709548dbf","Type":"ContainerStarted","Data":"4b36a78d2208bb57361456b0917f5db99b65f8aa69a8146d5e9aad8622262fc9"} Jul 07 17:45:49.338848 ip-10-0-129-39 kubenswrapper[2575]: I0707 17:45:49.338769 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-8nrlc" Jul 07 17:45:49.381057 ip-10-0-129-39 kubenswrapper[2575]: I0707 17:45:49.381014 2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/kserve-controller-manager-d6f4b65b8-d2drf" podStartSLOduration=2.319279042 podStartE2EDuration="5.381001203s" totalImagesPullingTime="3.061722161s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-07 17:45:44 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 17:45:49.3764115 +0000 UTC m=+309.882218166" watchObservedRunningTime="2026-07-07 17:45:49.381001203 +0000 UTC m=+309.886807869" Jul 07 17:45:49.477184 ip-10-0-129-39 kubenswrapper[2575]: I0707 17:45:49.477134 2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-8nrlc" podStartSLOduration=2.443802453 podStartE2EDuration="5.477117591s" totalImagesPullingTime="3.033315138s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-07 17:45:44 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 17:45:49.470035022 +0000 UTC m=+309.975841687" watchObservedRunningTime="2026-07-07 17:45:49.477117591 +0000 UTC m=+309.982924257" Jul 07 17:45:50.073418 ip-10-0-129-39 kubenswrapper[2575]: I0707 17:45:50.073385 2575 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="c1a4828b-4d80-41ae-b008-722cf9f308bc" path="/var/lib/kubelet/pods/c1a4828b-4d80-41ae-b008-722cf9f308bc/volumes" Jul 07 17:46:20.346498 ip-10-0-129-39 kubenswrapper[2575]: I0707 17:46:20.346467 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-8nrlc" Jul 07 17:46:20.349543 ip-10-0-129-39 kubenswrapper[2575]: I0707 17:46:20.349522 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/kserve-controller-manager-d6f4b65b8-d2drf" Jul 07 17:46:32.659014 ip-10-0-129-39 kubenswrapper[2575]: I0707 17:46:32.658940 2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-catalog-jxnfb"] Jul 07 17:46:32.659406 ip-10-0-129-39 kubenswrapper[2575]: I0707 17:46:32.659300 2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c1a4828b-4d80-41ae-b008-722cf9f308bc" containerName="discovery" Jul 07 17:46:32.659406 ip-10-0-129-39 kubenswrapper[2575]: I0707 17:46:32.659313 2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="c1a4828b-4d80-41ae-b008-722cf9f308bc" containerName="discovery" Jul 07 17:46:32.659406 ip-10-0-129-39 kubenswrapper[2575]: I0707 17:46:32.659372 2575 memory_manager.go:368] "RemoveStaleState removing state" podUID="c1a4828b-4d80-41ae-b008-722cf9f308bc" containerName="discovery" Jul 07 17:46:32.661205 ip-10-0-129-39 kubenswrapper[2575]: I0707 17:46:32.661189 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-catalog-jxnfb" Jul 07 17:46:32.664698 ip-10-0-129-39 kubenswrapper[2575]: I0707 17:46:32.664675 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"kuadrant-operator-catalog-dockercfg-htgwt\"" Jul 07 17:46:32.665581 ip-10-0-129-39 kubenswrapper[2575]: I0707 17:46:32.665544 2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kube-root-ca.crt\"" Jul 07 17:46:32.665777 ip-10-0-129-39 kubenswrapper[2575]: I0707 17:46:32.665642 2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"openshift-service-ca.crt\"" Jul 07 17:46:32.672780 ip-10-0-129-39 kubenswrapper[2575]: I0707 17:46:32.672756 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-catalog-jxnfb"] Jul 07 17:46:32.808684 ip-10-0-129-39 kubenswrapper[2575]: I0707 17:46:32.808646 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2gtdv\" (UniqueName: \"kubernetes.io/projected/cc60a9ce-e894-4b81-be80-ff08e4534017-kube-api-access-2gtdv\") pod \"kuadrant-operator-catalog-jxnfb\" (UID: \"cc60a9ce-e894-4b81-be80-ff08e4534017\") " pod="kuadrant-system/kuadrant-operator-catalog-jxnfb" Jul 07 17:46:32.909583 ip-10-0-129-39 kubenswrapper[2575]: I0707 17:46:32.909490 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2gtdv\" (UniqueName: \"kubernetes.io/projected/cc60a9ce-e894-4b81-be80-ff08e4534017-kube-api-access-2gtdv\") pod \"kuadrant-operator-catalog-jxnfb\" (UID: \"cc60a9ce-e894-4b81-be80-ff08e4534017\") " pod="kuadrant-system/kuadrant-operator-catalog-jxnfb" Jul 07 17:46:32.918875 ip-10-0-129-39 kubenswrapper[2575]: I0707 17:46:32.918849 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2gtdv\" (UniqueName: \"kubernetes.io/projected/cc60a9ce-e894-4b81-be80-ff08e4534017-kube-api-access-2gtdv\") pod \"kuadrant-operator-catalog-jxnfb\" (UID: \"cc60a9ce-e894-4b81-be80-ff08e4534017\") " pod="kuadrant-system/kuadrant-operator-catalog-jxnfb" Jul 07 17:46:32.972617 ip-10-0-129-39 kubenswrapper[2575]: I0707 17:46:32.972584 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-catalog-jxnfb" Jul 07 17:46:33.027036 ip-10-0-129-39 kubenswrapper[2575]: I0707 17:46:33.027000 2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-catalog-jxnfb"] Jul 07 17:46:33.106173 ip-10-0-129-39 kubenswrapper[2575]: I0707 17:46:33.106143 2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-catalog-jxnfb"] Jul 07 17:46:33.110246 ip-10-0-129-39 kubenswrapper[2575]: W0707 17:46:33.110221 2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcc60a9ce_e894_4b81_be80_ff08e4534017.slice/crio-77aab6dbe6265373b84fa13bd77c41e3977472b1198bda1ab59e7650700eef5f WatchSource:0}: Error finding container 77aab6dbe6265373b84fa13bd77c41e3977472b1198bda1ab59e7650700eef5f: Status 404 returned error can't find the container with id 77aab6dbe6265373b84fa13bd77c41e3977472b1198bda1ab59e7650700eef5f Jul 07 17:46:33.240093 ip-10-0-129-39 kubenswrapper[2575]: I0707 17:46:33.240020 2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-catalog-sf58p"] Jul 07 17:46:33.242921 ip-10-0-129-39 kubenswrapper[2575]: I0707 17:46:33.242902 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-catalog-sf58p" Jul 07 17:46:33.253409 ip-10-0-129-39 kubenswrapper[2575]: I0707 17:46:33.253385 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-catalog-sf58p"] Jul 07 17:46:33.313278 ip-10-0-129-39 kubenswrapper[2575]: I0707 17:46:33.313253 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-x6lln\" (UniqueName: \"kubernetes.io/projected/1ac64042-f41a-4ebb-a26b-e12f4c356c19-kube-api-access-x6lln\") pod \"kuadrant-operator-catalog-sf58p\" (UID: \"1ac64042-f41a-4ebb-a26b-e12f4c356c19\") " pod="kuadrant-system/kuadrant-operator-catalog-sf58p" Jul 07 17:46:33.414259 ip-10-0-129-39 kubenswrapper[2575]: I0707 17:46:33.414236 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x6lln\" (UniqueName: \"kubernetes.io/projected/1ac64042-f41a-4ebb-a26b-e12f4c356c19-kube-api-access-x6lln\") pod \"kuadrant-operator-catalog-sf58p\" (UID: \"1ac64042-f41a-4ebb-a26b-e12f4c356c19\") " pod="kuadrant-system/kuadrant-operator-catalog-sf58p" Jul 07 17:46:33.428445 ip-10-0-129-39 kubenswrapper[2575]: I0707 17:46:33.428415 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-x6lln\" (UniqueName: \"kubernetes.io/projected/1ac64042-f41a-4ebb-a26b-e12f4c356c19-kube-api-access-x6lln\") pod \"kuadrant-operator-catalog-sf58p\" (UID: \"1ac64042-f41a-4ebb-a26b-e12f4c356c19\") " pod="kuadrant-system/kuadrant-operator-catalog-sf58p" Jul 07 17:46:33.519674 ip-10-0-129-39 kubenswrapper[2575]: I0707 17:46:33.519602 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-catalog-jxnfb" event={"ID":"cc60a9ce-e894-4b81-be80-ff08e4534017","Type":"ContainerStarted","Data":"77aab6dbe6265373b84fa13bd77c41e3977472b1198bda1ab59e7650700eef5f"} Jul 07 17:46:33.551997 ip-10-0-129-39 kubenswrapper[2575]: I0707 17:46:33.551972 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-catalog-sf58p" Jul 07 17:46:33.685992 ip-10-0-129-39 kubenswrapper[2575]: I0707 17:46:33.685970 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-catalog-sf58p"] Jul 07 17:46:33.688441 ip-10-0-129-39 kubenswrapper[2575]: W0707 17:46:33.688417 2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1ac64042_f41a_4ebb_a26b_e12f4c356c19.slice/crio-a36fafe1e5b66ee43ee79f364a1a91978175ec7e1479ac73c1241809e053b7fe WatchSource:0}: Error finding container a36fafe1e5b66ee43ee79f364a1a91978175ec7e1479ac73c1241809e053b7fe: Status 404 returned error can't find the container with id a36fafe1e5b66ee43ee79f364a1a91978175ec7e1479ac73c1241809e053b7fe Jul 07 17:46:34.526592 ip-10-0-129-39 kubenswrapper[2575]: I0707 17:46:34.526553 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-catalog-sf58p" event={"ID":"1ac64042-f41a-4ebb-a26b-e12f4c356c19","Type":"ContainerStarted","Data":"a36fafe1e5b66ee43ee79f364a1a91978175ec7e1479ac73c1241809e053b7fe"} Jul 07 17:46:36.537757 ip-10-0-129-39 kubenswrapper[2575]: I0707 17:46:36.537720 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-catalog-jxnfb" event={"ID":"cc60a9ce-e894-4b81-be80-ff08e4534017","Type":"ContainerStarted","Data":"e2719e8bc1114dcf7b18f69e919f1c5732bbe860ca282a9ab968feecf7226e4c"} Jul 07 17:46:36.538213 ip-10-0-129-39 kubenswrapper[2575]: I0707 17:46:36.537779 2575 kuberuntime_container.go:872] "Killing container with a grace period" pod="kuadrant-system/kuadrant-operator-catalog-jxnfb" podUID="cc60a9ce-e894-4b81-be80-ff08e4534017" containerName="registry-server" containerID="cri-o://e2719e8bc1114dcf7b18f69e919f1c5732bbe860ca282a9ab968feecf7226e4c" gracePeriod=2 Jul 07 17:46:36.539187 ip-10-0-129-39 kubenswrapper[2575]: I0707 17:46:36.539162 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-catalog-sf58p" event={"ID":"1ac64042-f41a-4ebb-a26b-e12f4c356c19","Type":"ContainerStarted","Data":"b4670db2b5cdc7d089a9e85c6b45b2675d629ed9efc2266982083eec731e21ec"} Jul 07 17:46:36.554341 ip-10-0-129-39 kubenswrapper[2575]: I0707 17:46:36.554280 2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-catalog-jxnfb" podStartSLOduration=2.107897629 podStartE2EDuration="4.554264853s" totalImagesPullingTime="2.446367224s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-07 17:46:32 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 17:46:36.553906117 +0000 UTC m=+357.059712784" watchObservedRunningTime="2026-07-07 17:46:36.554264853 +0000 UTC m=+357.060071522" Jul 07 17:46:36.571805 ip-10-0-129-39 kubenswrapper[2575]: I0707 17:46:36.571767 2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-catalog-sf58p" podStartSLOduration=1.701157968 podStartE2EDuration="3.571754777s" totalImagesPullingTime="1.870596809s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-07 17:46:33 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 17:46:36.570281451 +0000 UTC m=+357.076088119" watchObservedRunningTime="2026-07-07 17:46:36.571754777 +0000 UTC m=+357.077561443" Jul 07 17:46:36.783402 ip-10-0-129-39 kubenswrapper[2575]: I0707 17:46:36.783379 2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-catalog-jxnfb" Jul 07 17:46:36.940440 ip-10-0-129-39 kubenswrapper[2575]: I0707 17:46:36.940413 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-2gtdv\" (UniqueName: \"kubernetes.io/projected/cc60a9ce-e894-4b81-be80-ff08e4534017-kube-api-access-2gtdv\") pod \"cc60a9ce-e894-4b81-be80-ff08e4534017\" (UID: \"cc60a9ce-e894-4b81-be80-ff08e4534017\") " Jul 07 17:46:36.942602 ip-10-0-129-39 kubenswrapper[2575]: I0707 17:46:36.942580 2575 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cc60a9ce-e894-4b81-be80-ff08e4534017-kube-api-access-2gtdv" (OuterVolumeSpecName: "kube-api-access-2gtdv") pod "cc60a9ce-e894-4b81-be80-ff08e4534017" (UID: "cc60a9ce-e894-4b81-be80-ff08e4534017"). InnerVolumeSpecName "kube-api-access-2gtdv". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 07 17:46:37.041141 ip-10-0-129-39 kubenswrapper[2575]: I0707 17:46:37.041116 2575 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2gtdv\" (UniqueName: \"kubernetes.io/projected/cc60a9ce-e894-4b81-be80-ff08e4534017-kube-api-access-2gtdv\") on node \"ip-10-0-129-39.ec2.internal\" DevicePath \"\"" Jul 07 17:46:37.544368 ip-10-0-129-39 kubenswrapper[2575]: I0707 17:46:37.544329 2575 generic.go:356] "Generic (PLEG): container finished" podID="cc60a9ce-e894-4b81-be80-ff08e4534017" containerID="e2719e8bc1114dcf7b18f69e919f1c5732bbe860ca282a9ab968feecf7226e4c" exitCode=0 Jul 07 17:46:37.544825 ip-10-0-129-39 kubenswrapper[2575]: I0707 17:46:37.544382 2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-catalog-jxnfb" Jul 07 17:46:37.544825 ip-10-0-129-39 kubenswrapper[2575]: I0707 17:46:37.544420 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-catalog-jxnfb" event={"ID":"cc60a9ce-e894-4b81-be80-ff08e4534017","Type":"ContainerDied","Data":"e2719e8bc1114dcf7b18f69e919f1c5732bbe860ca282a9ab968feecf7226e4c"} Jul 07 17:46:37.544825 ip-10-0-129-39 kubenswrapper[2575]: I0707 17:46:37.544468 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-catalog-jxnfb" event={"ID":"cc60a9ce-e894-4b81-be80-ff08e4534017","Type":"ContainerDied","Data":"77aab6dbe6265373b84fa13bd77c41e3977472b1198bda1ab59e7650700eef5f"} Jul 07 17:46:37.544825 ip-10-0-129-39 kubenswrapper[2575]: I0707 17:46:37.544495 2575 scope.go:117] "RemoveContainer" containerID="e2719e8bc1114dcf7b18f69e919f1c5732bbe860ca282a9ab968feecf7226e4c" Jul 07 17:46:37.553382 ip-10-0-129-39 kubenswrapper[2575]: I0707 17:46:37.553365 2575 scope.go:117] "RemoveContainer" containerID="e2719e8bc1114dcf7b18f69e919f1c5732bbe860ca282a9ab968feecf7226e4c" Jul 07 17:46:37.553666 ip-10-0-129-39 kubenswrapper[2575]: E0707 17:46:37.553645 2575 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e2719e8bc1114dcf7b18f69e919f1c5732bbe860ca282a9ab968feecf7226e4c\": container with ID starting with e2719e8bc1114dcf7b18f69e919f1c5732bbe860ca282a9ab968feecf7226e4c not found: ID does not exist" containerID="e2719e8bc1114dcf7b18f69e919f1c5732bbe860ca282a9ab968feecf7226e4c" Jul 07 17:46:37.553722 ip-10-0-129-39 kubenswrapper[2575]: I0707 17:46:37.553674 2575 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e2719e8bc1114dcf7b18f69e919f1c5732bbe860ca282a9ab968feecf7226e4c"} err="failed to get container status \"e2719e8bc1114dcf7b18f69e919f1c5732bbe860ca282a9ab968feecf7226e4c\": rpc error: code = NotFound desc = could not find container \"e2719e8bc1114dcf7b18f69e919f1c5732bbe860ca282a9ab968feecf7226e4c\": container with ID starting with e2719e8bc1114dcf7b18f69e919f1c5732bbe860ca282a9ab968feecf7226e4c not found: ID does not exist" Jul 07 17:46:37.567184 ip-10-0-129-39 kubenswrapper[2575]: I0707 17:46:37.567159 2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-catalog-jxnfb"] Jul 07 17:46:37.574675 ip-10-0-129-39 kubenswrapper[2575]: I0707 17:46:37.574650 2575 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/kuadrant-operator-catalog-jxnfb"] Jul 07 17:46:38.073668 ip-10-0-129-39 kubenswrapper[2575]: I0707 17:46:38.073637 2575 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="cc60a9ce-e894-4b81-be80-ff08e4534017" path="/var/lib/kubelet/pods/cc60a9ce-e894-4b81-be80-ff08e4534017/volumes" Jul 07 17:46:43.552771 ip-10-0-129-39 kubenswrapper[2575]: I0707 17:46:43.552728 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-catalog-sf58p" Jul 07 17:46:43.553223 ip-10-0-129-39 kubenswrapper[2575]: I0707 17:46:43.552828 2575 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kuadrant-system/kuadrant-operator-catalog-sf58p" Jul 07 17:46:43.574787 ip-10-0-129-39 kubenswrapper[2575]: I0707 17:46:43.574761 2575 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kuadrant-system/kuadrant-operator-catalog-sf58p" Jul 07 17:46:43.596063 ip-10-0-129-39 kubenswrapper[2575]: I0707 17:46:43.596030 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-catalog-sf58p" Jul 07 17:47:03.170967 ip-10-0-129-39 kubenswrapper[2575]: I0707 17:47:03.170937 2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-dc76n"] Jul 07 17:47:03.171479 ip-10-0-129-39 kubenswrapper[2575]: I0707 17:47:03.171242 2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="cc60a9ce-e894-4b81-be80-ff08e4534017" containerName="registry-server" Jul 07 17:47:03.171479 ip-10-0-129-39 kubenswrapper[2575]: I0707 17:47:03.171253 2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="cc60a9ce-e894-4b81-be80-ff08e4534017" containerName="registry-server" Jul 07 17:47:03.171479 ip-10-0-129-39 kubenswrapper[2575]: I0707 17:47:03.171320 2575 memory_manager.go:368] "RemoveStaleState removing state" podUID="cc60a9ce-e894-4b81-be80-ff08e4534017" containerName="registry-server" Jul 07 17:47:03.173108 ip-10-0-129-39 kubenswrapper[2575]: I0707 17:47:03.173092 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-dc76n" Jul 07 17:47:03.187952 ip-10-0-129-39 kubenswrapper[2575]: I0707 17:47:03.187931 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"kuadrant-operator-controller-manager-dockercfg-6kr82\"" Jul 07 17:47:03.218681 ip-10-0-129-39 kubenswrapper[2575]: I0707 17:47:03.218655 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-dc76n"] Jul 07 17:47:03.337037 ip-10-0-129-39 kubenswrapper[2575]: I0707 17:47:03.337006 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/419be36b-c6a1-4709-90da-3622dab0bfee-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-8684c9ff49-dc76n\" (UID: \"419be36b-c6a1-4709-90da-3622dab0bfee\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-dc76n" Jul 07 17:47:03.337037 ip-10-0-129-39 kubenswrapper[2575]: I0707 17:47:03.337042 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hrnpk\" (UniqueName: \"kubernetes.io/projected/419be36b-c6a1-4709-90da-3622dab0bfee-kube-api-access-hrnpk\") pod \"kuadrant-operator-controller-manager-8684c9ff49-dc76n\" (UID: \"419be36b-c6a1-4709-90da-3622dab0bfee\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-dc76n" Jul 07 17:47:03.437709 ip-10-0-129-39 kubenswrapper[2575]: I0707 17:47:03.437637 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/419be36b-c6a1-4709-90da-3622dab0bfee-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-8684c9ff49-dc76n\" (UID: \"419be36b-c6a1-4709-90da-3622dab0bfee\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-dc76n" Jul 07 17:47:03.437709 ip-10-0-129-39 kubenswrapper[2575]: I0707 17:47:03.437673 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hrnpk\" (UniqueName: \"kubernetes.io/projected/419be36b-c6a1-4709-90da-3622dab0bfee-kube-api-access-hrnpk\") pod \"kuadrant-operator-controller-manager-8684c9ff49-dc76n\" (UID: \"419be36b-c6a1-4709-90da-3622dab0bfee\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-dc76n" Jul 07 17:47:03.438025 ip-10-0-129-39 kubenswrapper[2575]: I0707 17:47:03.438005 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/419be36b-c6a1-4709-90da-3622dab0bfee-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-8684c9ff49-dc76n\" (UID: \"419be36b-c6a1-4709-90da-3622dab0bfee\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-dc76n" Jul 07 17:47:03.449330 ip-10-0-129-39 kubenswrapper[2575]: I0707 17:47:03.449302 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hrnpk\" (UniqueName: \"kubernetes.io/projected/419be36b-c6a1-4709-90da-3622dab0bfee-kube-api-access-hrnpk\") pod \"kuadrant-operator-controller-manager-8684c9ff49-dc76n\" (UID: \"419be36b-c6a1-4709-90da-3622dab0bfee\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-dc76n" Jul 07 17:47:03.482877 ip-10-0-129-39 kubenswrapper[2575]: I0707 17:47:03.482851 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-dc76n" Jul 07 17:47:03.689609 ip-10-0-129-39 kubenswrapper[2575]: I0707 17:47:03.689588 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-dc76n"] Jul 07 17:47:03.692120 ip-10-0-129-39 kubenswrapper[2575]: W0707 17:47:03.692087 2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod419be36b_c6a1_4709_90da_3622dab0bfee.slice/crio-4cec7a02a0e7fd807bfe4ebcf5e29770d39a0c0dce8ffcb36320ff4909a4791b WatchSource:0}: Error finding container 4cec7a02a0e7fd807bfe4ebcf5e29770d39a0c0dce8ffcb36320ff4909a4791b: Status 404 returned error can't find the container with id 4cec7a02a0e7fd807bfe4ebcf5e29770d39a0c0dce8ffcb36320ff4909a4791b Jul 07 17:47:04.655960 ip-10-0-129-39 kubenswrapper[2575]: I0707 17:47:04.655928 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-dc76n" event={"ID":"419be36b-c6a1-4709-90da-3622dab0bfee","Type":"ContainerStarted","Data":"4cec7a02a0e7fd807bfe4ebcf5e29770d39a0c0dce8ffcb36320ff4909a4791b"} Jul 07 17:47:06.008914 ip-10-0-129-39 kubenswrapper[2575]: I0707 17:47:06.008880 2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-operator-75cc8d5cb7-hlvlz"] Jul 07 17:47:06.012515 ip-10-0-129-39 kubenswrapper[2575]: I0707 17:47:06.012487 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-operator-75cc8d5cb7-hlvlz" Jul 07 17:47:06.015107 ip-10-0-129-39 kubenswrapper[2575]: I0707 17:47:06.015086 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-operator-dockercfg-frf6x\"" Jul 07 17:47:06.024286 ip-10-0-129-39 kubenswrapper[2575]: I0707 17:47:06.024265 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-operator-75cc8d5cb7-hlvlz"] Jul 07 17:47:06.057616 ip-10-0-129-39 kubenswrapper[2575]: I0707 17:47:06.057593 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kcfrw\" (UniqueName: \"kubernetes.io/projected/69ae37f2-4d15-44b6-a34d-ea5073901d1a-kube-api-access-kcfrw\") pod \"authorino-operator-75cc8d5cb7-hlvlz\" (UID: \"69ae37f2-4d15-44b6-a34d-ea5073901d1a\") " pod="kuadrant-system/authorino-operator-75cc8d5cb7-hlvlz" Jul 07 17:47:06.158297 ip-10-0-129-39 kubenswrapper[2575]: I0707 17:47:06.158264 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kcfrw\" (UniqueName: \"kubernetes.io/projected/69ae37f2-4d15-44b6-a34d-ea5073901d1a-kube-api-access-kcfrw\") pod \"authorino-operator-75cc8d5cb7-hlvlz\" (UID: \"69ae37f2-4d15-44b6-a34d-ea5073901d1a\") " pod="kuadrant-system/authorino-operator-75cc8d5cb7-hlvlz" Jul 07 17:47:06.173518 ip-10-0-129-39 kubenswrapper[2575]: I0707 17:47:06.173476 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kcfrw\" (UniqueName: \"kubernetes.io/projected/69ae37f2-4d15-44b6-a34d-ea5073901d1a-kube-api-access-kcfrw\") pod \"authorino-operator-75cc8d5cb7-hlvlz\" (UID: \"69ae37f2-4d15-44b6-a34d-ea5073901d1a\") " pod="kuadrant-system/authorino-operator-75cc8d5cb7-hlvlz" Jul 07 17:47:06.323243 ip-10-0-129-39 kubenswrapper[2575]: I0707 17:47:06.323219 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-operator-75cc8d5cb7-hlvlz" Jul 07 17:47:06.472184 ip-10-0-129-39 kubenswrapper[2575]: I0707 17:47:06.472165 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-operator-75cc8d5cb7-hlvlz"] Jul 07 17:47:06.474133 ip-10-0-129-39 kubenswrapper[2575]: W0707 17:47:06.474104 2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod69ae37f2_4d15_44b6_a34d_ea5073901d1a.slice/crio-c415261c7b728e5c68a6b14684a59ee9cf318627d04117646c691a21477aac24 WatchSource:0}: Error finding container c415261c7b728e5c68a6b14684a59ee9cf318627d04117646c691a21477aac24: Status 404 returned error can't find the container with id c415261c7b728e5c68a6b14684a59ee9cf318627d04117646c691a21477aac24 Jul 07 17:47:06.665208 ip-10-0-129-39 kubenswrapper[2575]: I0707 17:47:06.665131 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-operator-75cc8d5cb7-hlvlz" event={"ID":"69ae37f2-4d15-44b6-a34d-ea5073901d1a","Type":"ContainerStarted","Data":"c415261c7b728e5c68a6b14684a59ee9cf318627d04117646c691a21477aac24"} Jul 07 17:47:10.632526 ip-10-0-129-39 kubenswrapper[2575]: I0707 17:47:10.632477 2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-859d7b78bd-h2v4h"] Jul 07 17:47:10.635910 ip-10-0-129-39 kubenswrapper[2575]: I0707 17:47:10.635888 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-h2v4h" Jul 07 17:47:10.638304 ip-10-0-129-39 kubenswrapper[2575]: I0707 17:47:10.638282 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"limitador-operator-controller-manager-dockercfg-z6zdb\"" Jul 07 17:47:10.648647 ip-10-0-129-39 kubenswrapper[2575]: I0707 17:47:10.648627 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-859d7b78bd-h2v4h"] Jul 07 17:47:10.695272 ip-10-0-129-39 kubenswrapper[2575]: I0707 17:47:10.695238 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-operator-75cc8d5cb7-hlvlz" event={"ID":"69ae37f2-4d15-44b6-a34d-ea5073901d1a","Type":"ContainerStarted","Data":"90b3ca8275252740b5cbfeaf4a70bda9776e13f576a1db015568bba353601ce5"} Jul 07 17:47:10.695396 ip-10-0-129-39 kubenswrapper[2575]: I0707 17:47:10.695370 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/authorino-operator-75cc8d5cb7-hlvlz" Jul 07 17:47:10.696315 ip-10-0-129-39 kubenswrapper[2575]: I0707 17:47:10.696296 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ftkgn\" (UniqueName: \"kubernetes.io/projected/31101d8f-1952-4356-838c-7bde0aad908a-kube-api-access-ftkgn\") pod \"limitador-operator-controller-manager-859d7b78bd-h2v4h\" (UID: \"31101d8f-1952-4356-838c-7bde0aad908a\") " pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-h2v4h" Jul 07 17:47:10.696835 ip-10-0-129-39 kubenswrapper[2575]: I0707 17:47:10.696818 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-dc76n" event={"ID":"419be36b-c6a1-4709-90da-3622dab0bfee","Type":"ContainerStarted","Data":"f19839a92d2f25d689463ffd7df42b1e187f2d4461e2dace46a267e6ced3b51e"} Jul 07 17:47:10.696958 ip-10-0-129-39 kubenswrapper[2575]: I0707 17:47:10.696947 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-dc76n" Jul 07 17:47:10.717779 ip-10-0-129-39 kubenswrapper[2575]: I0707 17:47:10.717739 2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/authorino-operator-75cc8d5cb7-hlvlz" podStartSLOduration=2.423597862 podStartE2EDuration="5.717725662s" totalImagesPullingTime="3.2941278s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-07 17:47:05 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 17:47:10.717044297 +0000 UTC m=+391.222850962" watchObservedRunningTime="2026-07-07 17:47:10.717725662 +0000 UTC m=+391.223532327" Jul 07 17:47:10.735966 ip-10-0-129-39 kubenswrapper[2575]: I0707 17:47:10.735928 2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-dc76n" podStartSLOduration=1.694493578 podStartE2EDuration="7.735915205s" totalImagesPullingTime="6.041421627s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-07 17:47:03 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 17:47:10.734991561 +0000 UTC m=+391.240798227" watchObservedRunningTime="2026-07-07 17:47:10.735915205 +0000 UTC m=+391.241721871" Jul 07 17:47:10.797214 ip-10-0-129-39 kubenswrapper[2575]: I0707 17:47:10.797181 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ftkgn\" (UniqueName: \"kubernetes.io/projected/31101d8f-1952-4356-838c-7bde0aad908a-kube-api-access-ftkgn\") pod \"limitador-operator-controller-manager-859d7b78bd-h2v4h\" (UID: \"31101d8f-1952-4356-838c-7bde0aad908a\") " pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-h2v4h" Jul 07 17:47:10.810372 ip-10-0-129-39 kubenswrapper[2575]: I0707 17:47:10.810349 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ftkgn\" (UniqueName: \"kubernetes.io/projected/31101d8f-1952-4356-838c-7bde0aad908a-kube-api-access-ftkgn\") pod \"limitador-operator-controller-manager-859d7b78bd-h2v4h\" (UID: \"31101d8f-1952-4356-838c-7bde0aad908a\") " pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-h2v4h" Jul 07 17:47:10.946416 ip-10-0-129-39 kubenswrapper[2575]: I0707 17:47:10.946344 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-h2v4h" Jul 07 17:47:11.086729 ip-10-0-129-39 kubenswrapper[2575]: I0707 17:47:11.086705 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-859d7b78bd-h2v4h"] Jul 07 17:47:11.089180 ip-10-0-129-39 kubenswrapper[2575]: W0707 17:47:11.089154 2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod31101d8f_1952_4356_838c_7bde0aad908a.slice/crio-7dd50819fdee49452329118c86d230968523f8165daa87f444c9e2ba66a5916d WatchSource:0}: Error finding container 7dd50819fdee49452329118c86d230968523f8165daa87f444c9e2ba66a5916d: Status 404 returned error can't find the container with id 7dd50819fdee49452329118c86d230968523f8165daa87f444c9e2ba66a5916d Jul 07 17:47:11.705814 ip-10-0-129-39 kubenswrapper[2575]: I0707 17:47:11.705782 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-h2v4h" event={"ID":"31101d8f-1952-4356-838c-7bde0aad908a","Type":"ContainerStarted","Data":"7dd50819fdee49452329118c86d230968523f8165daa87f444c9e2ba66a5916d"} Jul 07 17:47:13.715551 ip-10-0-129-39 kubenswrapper[2575]: I0707 17:47:13.715498 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-h2v4h" event={"ID":"31101d8f-1952-4356-838c-7bde0aad908a","Type":"ContainerStarted","Data":"ac9e7c7a92ebaf2542dea697f54735fde9c6d96f6f88865eb5c03bdf7ce0d565"} Jul 07 17:47:13.715940 ip-10-0-129-39 kubenswrapper[2575]: I0707 17:47:13.715623 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-h2v4h" Jul 07 17:47:13.737796 ip-10-0-129-39 kubenswrapper[2575]: I0707 17:47:13.737747 2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-h2v4h" podStartSLOduration=1.877798956 podStartE2EDuration="3.737727981s" totalImagesPullingTime="1.859929025s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-07 17:47:10 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 17:47:13.737472676 +0000 UTC m=+394.243279344" watchObservedRunningTime="2026-07-07 17:47:13.737727981 +0000 UTC m=+394.243534648" Jul 07 17:47:21.708779 ip-10-0-129-39 kubenswrapper[2575]: I0707 17:47:21.708751 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/authorino-operator-75cc8d5cb7-hlvlz" Jul 07 17:47:21.709165 ip-10-0-129-39 kubenswrapper[2575]: I0707 17:47:21.708833 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-dc76n" Jul 07 17:47:23.746181 ip-10-0-129-39 kubenswrapper[2575]: I0707 17:47:23.746146 2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-dc76n"] Jul 07 17:47:23.746599 ip-10-0-129-39 kubenswrapper[2575]: I0707 17:47:23.746325 2575 kuberuntime_container.go:872] "Killing container with a grace period" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-dc76n" podUID="419be36b-c6a1-4709-90da-3622dab0bfee" containerName="manager" containerID="cri-o://f19839a92d2f25d689463ffd7df42b1e187f2d4461e2dace46a267e6ced3b51e" gracePeriod=2 Jul 07 17:47:23.753818 ip-10-0-129-39 kubenswrapper[2575]: I0707 17:47:23.753792 2575 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-dc76n"] Jul 07 17:47:23.757463 ip-10-0-129-39 kubenswrapper[2575]: E0707 17:47:23.757433 2575 status_manager.go:1041] "Failed to update status for pod" err="failed to patch status \"{\\\"metadata\\\":{\\\"uid\\\":\\\"419be36b-c6a1-4709-90da-3622dab0bfee\\\"},\\\"status\\\":{\\\"$setElementOrder/conditions\\\":[{\\\"type\\\":\\\"PodReadyToStartContainers\\\"},{\\\"type\\\":\\\"Initialized\\\"},{\\\"type\\\":\\\"Ready\\\"},{\\\"type\\\":\\\"ContainersReady\\\"},{\\\"type\\\":\\\"PodScheduled\\\"}],\\\"conditions\\\":[{\\\"observedGeneration\\\":2,\\\"type\\\":\\\"PodReadyToStartContainers\\\"},{\\\"observedGeneration\\\":2,\\\"type\\\":\\\"Initialized\\\"},{\\\"observedGeneration\\\":2,\\\"type\\\":\\\"Ready\\\"},{\\\"observedGeneration\\\":2,\\\"type\\\":\\\"ContainersReady\\\"},{\\\"observedGeneration\\\":2,\\\"type\\\":\\\"PodScheduled\\\"}],\\\"observedGeneration\\\":2}}\" for pod \"kuadrant-system\"/\"kuadrant-operator-controller-manager-8684c9ff49-dc76n\": pods \"kuadrant-operator-controller-manager-8684c9ff49-dc76n\" not found" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-dc76n" Jul 07 17:47:23.770623 ip-10-0-129-39 kubenswrapper[2575]: I0707 17:47:23.770600 2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-859d7b78bd-h2v4h"] Jul 07 17:47:23.770793 ip-10-0-129-39 kubenswrapper[2575]: I0707 17:47:23.770770 2575 kuberuntime_container.go:872] "Killing container with a grace period" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-h2v4h" podUID="31101d8f-1952-4356-838c-7bde0aad908a" containerName="manager" containerID="cri-o://ac9e7c7a92ebaf2542dea697f54735fde9c6d96f6f88865eb5c03bdf7ce0d565" gracePeriod=2 Jul 07 17:47:23.772622 ip-10-0-129-39 kubenswrapper[2575]: I0707 17:47:23.772570 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-h2v4h" Jul 07 17:47:23.781199 ip-10-0-129-39 kubenswrapper[2575]: E0707 17:47:23.781161 2575 status_manager.go:1018] "Failed to get status for pod" err="pods \"limitador-operator-controller-manager-859d7b78bd-h2v4h\" is forbidden: User \"system:node:ip-10-0-129-39.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-129-39.ec2.internal' and this object" podUID="31101d8f-1952-4356-838c-7bde0aad908a" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-h2v4h" Jul 07 17:47:23.781344 ip-10-0-129-39 kubenswrapper[2575]: I0707 17:47:23.781310 2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-rg267"] Jul 07 17:47:23.782004 ip-10-0-129-39 kubenswrapper[2575]: I0707 17:47:23.781824 2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="419be36b-c6a1-4709-90da-3622dab0bfee" containerName="manager" Jul 07 17:47:23.782004 ip-10-0-129-39 kubenswrapper[2575]: I0707 17:47:23.781846 2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="419be36b-c6a1-4709-90da-3622dab0bfee" containerName="manager" Jul 07 17:47:23.782004 ip-10-0-129-39 kubenswrapper[2575]: I0707 17:47:23.781959 2575 memory_manager.go:368] "RemoveStaleState removing state" podUID="419be36b-c6a1-4709-90da-3622dab0bfee" containerName="manager" Jul 07 17:47:23.783915 ip-10-0-129-39 kubenswrapper[2575]: E0707 17:47:23.783882 2575 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-8684c9ff49-dc76n\" is forbidden: User \"system:node:ip-10-0-129-39.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-129-39.ec2.internal' and this object" podUID="419be36b-c6a1-4709-90da-3622dab0bfee" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-dc76n" Jul 07 17:47:23.785220 ip-10-0-129-39 kubenswrapper[2575]: I0707 17:47:23.785201 2575 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-859d7b78bd-h2v4h"] Jul 07 17:47:23.785294 ip-10-0-129-39 kubenswrapper[2575]: I0707 17:47:23.785281 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-rg267" Jul 07 17:47:23.786275 ip-10-0-129-39 kubenswrapper[2575]: E0707 17:47:23.786242 2575 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-8684c9ff49-dc76n\" is forbidden: User \"system:node:ip-10-0-129-39.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-129-39.ec2.internal' and this object" podUID="419be36b-c6a1-4709-90da-3622dab0bfee" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-dc76n" Jul 07 17:47:23.789091 ip-10-0-129-39 kubenswrapper[2575]: E0707 17:47:23.789067 2575 status_manager.go:1018] "Failed to get status for pod" err="pods \"limitador-operator-controller-manager-859d7b78bd-h2v4h\" is forbidden: User \"system:node:ip-10-0-129-39.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-129-39.ec2.internal' and this object" podUID="31101d8f-1952-4356-838c-7bde0aad908a" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-h2v4h" Jul 07 17:47:23.791279 ip-10-0-129-39 kubenswrapper[2575]: E0707 17:47:23.791257 2575 status_manager.go:1018] "Failed to get status for pod" err="pods \"limitador-operator-controller-manager-859d7b78bd-h2v4h\" is forbidden: User \"system:node:ip-10-0-129-39.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-129-39.ec2.internal' and this object" podUID="31101d8f-1952-4356-838c-7bde0aad908a" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-h2v4h" Jul 07 17:47:23.799262 ip-10-0-129-39 kubenswrapper[2575]: I0707 17:47:23.799236 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-rg267"] Jul 07 17:47:23.821395 ip-10-0-129-39 kubenswrapper[2575]: E0707 17:47:23.821360 2575 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-8684c9ff49-dc76n\" is forbidden: User \"system:node:ip-10-0-129-39.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-129-39.ec2.internal' and this object" podUID="419be36b-c6a1-4709-90da-3622dab0bfee" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-dc76n" Jul 07 17:47:23.903391 ip-10-0-129-39 kubenswrapper[2575]: I0707 17:47:23.903362 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8m7pq\" (UniqueName: \"kubernetes.io/projected/f4ec98a5-a47b-46b3-a9c4-8ea072fb219c-kube-api-access-8m7pq\") pod \"kuadrant-operator-controller-manager-8684c9ff49-rg267\" (UID: \"f4ec98a5-a47b-46b3-a9c4-8ea072fb219c\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-rg267" Jul 07 17:47:23.903590 ip-10-0-129-39 kubenswrapper[2575]: I0707 17:47:23.903405 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/f4ec98a5-a47b-46b3-a9c4-8ea072fb219c-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-8684c9ff49-rg267\" (UID: \"f4ec98a5-a47b-46b3-a9c4-8ea072fb219c\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-rg267" Jul 07 17:47:24.004731 ip-10-0-129-39 kubenswrapper[2575]: I0707 17:47:24.004706 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8m7pq\" (UniqueName: \"kubernetes.io/projected/f4ec98a5-a47b-46b3-a9c4-8ea072fb219c-kube-api-access-8m7pq\") pod \"kuadrant-operator-controller-manager-8684c9ff49-rg267\" (UID: \"f4ec98a5-a47b-46b3-a9c4-8ea072fb219c\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-rg267" Jul 07 17:47:24.004810 ip-10-0-129-39 kubenswrapper[2575]: I0707 17:47:24.004769 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/f4ec98a5-a47b-46b3-a9c4-8ea072fb219c-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-8684c9ff49-rg267\" (UID: \"f4ec98a5-a47b-46b3-a9c4-8ea072fb219c\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-rg267" Jul 07 17:47:24.005181 ip-10-0-129-39 kubenswrapper[2575]: I0707 17:47:24.005156 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/f4ec98a5-a47b-46b3-a9c4-8ea072fb219c-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-8684c9ff49-rg267\" (UID: \"f4ec98a5-a47b-46b3-a9c4-8ea072fb219c\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-rg267" Jul 07 17:47:24.015734 ip-10-0-129-39 kubenswrapper[2575]: I0707 17:47:24.015712 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8m7pq\" (UniqueName: \"kubernetes.io/projected/f4ec98a5-a47b-46b3-a9c4-8ea072fb219c-kube-api-access-8m7pq\") pod \"kuadrant-operator-controller-manager-8684c9ff49-rg267\" (UID: \"f4ec98a5-a47b-46b3-a9c4-8ea072fb219c\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-rg267" Jul 07 17:47:24.019307 ip-10-0-129-39 kubenswrapper[2575]: I0707 17:47:24.019290 2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-dc76n" Jul 07 17:47:24.021381 ip-10-0-129-39 kubenswrapper[2575]: I0707 17:47:24.021365 2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-h2v4h" Jul 07 17:47:24.021991 ip-10-0-129-39 kubenswrapper[2575]: E0707 17:47:24.021967 2575 status_manager.go:1018] "Failed to get status for pod" err="pods \"limitador-operator-controller-manager-859d7b78bd-h2v4h\" is forbidden: User \"system:node:ip-10-0-129-39.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-129-39.ec2.internal' and this object" podUID="31101d8f-1952-4356-838c-7bde0aad908a" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-h2v4h" Jul 07 17:47:24.024236 ip-10-0-129-39 kubenswrapper[2575]: E0707 17:47:24.024214 2575 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-8684c9ff49-dc76n\" is forbidden: User \"system:node:ip-10-0-129-39.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-129-39.ec2.internal' and this object" podUID="419be36b-c6a1-4709-90da-3622dab0bfee" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-dc76n" Jul 07 17:47:24.026341 ip-10-0-129-39 kubenswrapper[2575]: E0707 17:47:24.026322 2575 status_manager.go:1018] "Failed to get status for pod" err="pods \"limitador-operator-controller-manager-859d7b78bd-h2v4h\" is forbidden: User \"system:node:ip-10-0-129-39.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-129-39.ec2.internal' and this object" podUID="31101d8f-1952-4356-838c-7bde0aad908a" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-h2v4h" Jul 07 17:47:24.028041 ip-10-0-129-39 kubenswrapper[2575]: E0707 17:47:24.028024 2575 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-8684c9ff49-dc76n\" is forbidden: User \"system:node:ip-10-0-129-39.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-129-39.ec2.internal' and this object" podUID="419be36b-c6a1-4709-90da-3622dab0bfee" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-dc76n" Jul 07 17:47:24.057320 ip-10-0-129-39 kubenswrapper[2575]: I0707 17:47:24.057302 2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-86878c769f-x85t4"] Jul 07 17:47:24.057647 ip-10-0-129-39 kubenswrapper[2575]: I0707 17:47:24.057634 2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="31101d8f-1952-4356-838c-7bde0aad908a" containerName="manager" Jul 07 17:47:24.057696 ip-10-0-129-39 kubenswrapper[2575]: I0707 17:47:24.057649 2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="31101d8f-1952-4356-838c-7bde0aad908a" containerName="manager" Jul 07 17:47:24.057730 ip-10-0-129-39 kubenswrapper[2575]: I0707 17:47:24.057704 2575 memory_manager.go:368] "RemoveStaleState removing state" podUID="31101d8f-1952-4356-838c-7bde0aad908a" containerName="manager" Jul 07 17:47:24.060366 ip-10-0-129-39 kubenswrapper[2575]: I0707 17:47:24.060354 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-x85t4" Jul 07 17:47:24.065880 ip-10-0-129-39 kubenswrapper[2575]: E0707 17:47:24.065857 2575 status_manager.go:1018] "Failed to get status for pod" err="pods \"limitador-operator-controller-manager-859d7b78bd-h2v4h\" is forbidden: User \"system:node:ip-10-0-129-39.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-129-39.ec2.internal' and this object" podUID="31101d8f-1952-4356-838c-7bde0aad908a" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-h2v4h" Jul 07 17:47:24.068464 ip-10-0-129-39 kubenswrapper[2575]: E0707 17:47:24.068438 2575 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-8684c9ff49-dc76n\" is forbidden: User \"system:node:ip-10-0-129-39.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-129-39.ec2.internal' and this object" podUID="419be36b-c6a1-4709-90da-3622dab0bfee" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-dc76n" Jul 07 17:47:24.078018 ip-10-0-129-39 kubenswrapper[2575]: I0707 17:47:24.077999 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-86878c769f-x85t4"] Jul 07 17:47:24.105109 ip-10-0-129-39 kubenswrapper[2575]: I0707 17:47:24.105084 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/419be36b-c6a1-4709-90da-3622dab0bfee-extensions-socket-volume\") pod \"419be36b-c6a1-4709-90da-3622dab0bfee\" (UID: \"419be36b-c6a1-4709-90da-3622dab0bfee\") " Jul 07 17:47:24.105218 ip-10-0-129-39 kubenswrapper[2575]: I0707 17:47:24.105125 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-ftkgn\" (UniqueName: \"kubernetes.io/projected/31101d8f-1952-4356-838c-7bde0aad908a-kube-api-access-ftkgn\") pod \"31101d8f-1952-4356-838c-7bde0aad908a\" (UID: \"31101d8f-1952-4356-838c-7bde0aad908a\") " Jul 07 17:47:24.105218 ip-10-0-129-39 kubenswrapper[2575]: I0707 17:47:24.105173 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-hrnpk\" (UniqueName: \"kubernetes.io/projected/419be36b-c6a1-4709-90da-3622dab0bfee-kube-api-access-hrnpk\") pod \"419be36b-c6a1-4709-90da-3622dab0bfee\" (UID: \"419be36b-c6a1-4709-90da-3622dab0bfee\") " Jul 07 17:47:24.105326 ip-10-0-129-39 kubenswrapper[2575]: I0707 17:47:24.105286 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zm4q9\" (UniqueName: \"kubernetes.io/projected/d270ae63-e153-4532-8963-16397c54c338-kube-api-access-zm4q9\") pod \"kuadrant-operator-controller-manager-86878c769f-x85t4\" (UID: \"d270ae63-e153-4532-8963-16397c54c338\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-x85t4" Jul 07 17:47:24.105385 ip-10-0-129-39 kubenswrapper[2575]: I0707 17:47:24.105372 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/d270ae63-e153-4532-8963-16397c54c338-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-86878c769f-x85t4\" (UID: \"d270ae63-e153-4532-8963-16397c54c338\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-x85t4" Jul 07 17:47:24.105691 ip-10-0-129-39 kubenswrapper[2575]: I0707 17:47:24.105662 2575 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/419be36b-c6a1-4709-90da-3622dab0bfee-extensions-socket-volume" (OuterVolumeSpecName: "extensions-socket-volume") pod "419be36b-c6a1-4709-90da-3622dab0bfee" (UID: "419be36b-c6a1-4709-90da-3622dab0bfee"). InnerVolumeSpecName "extensions-socket-volume". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 07 17:47:24.107251 ip-10-0-129-39 kubenswrapper[2575]: I0707 17:47:24.107229 2575 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/31101d8f-1952-4356-838c-7bde0aad908a-kube-api-access-ftkgn" (OuterVolumeSpecName: "kube-api-access-ftkgn") pod "31101d8f-1952-4356-838c-7bde0aad908a" (UID: "31101d8f-1952-4356-838c-7bde0aad908a"). InnerVolumeSpecName "kube-api-access-ftkgn". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 07 17:47:24.107518 ip-10-0-129-39 kubenswrapper[2575]: I0707 17:47:24.107484 2575 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/419be36b-c6a1-4709-90da-3622dab0bfee-kube-api-access-hrnpk" (OuterVolumeSpecName: "kube-api-access-hrnpk") pod "419be36b-c6a1-4709-90da-3622dab0bfee" (UID: "419be36b-c6a1-4709-90da-3622dab0bfee"). InnerVolumeSpecName "kube-api-access-hrnpk". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 07 17:47:24.185487 ip-10-0-129-39 kubenswrapper[2575]: I0707 17:47:24.185463 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-rg267" Jul 07 17:47:24.205865 ip-10-0-129-39 kubenswrapper[2575]: I0707 17:47:24.205844 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zm4q9\" (UniqueName: \"kubernetes.io/projected/d270ae63-e153-4532-8963-16397c54c338-kube-api-access-zm4q9\") pod \"kuadrant-operator-controller-manager-86878c769f-x85t4\" (UID: \"d270ae63-e153-4532-8963-16397c54c338\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-x85t4" Jul 07 17:47:24.205953 ip-10-0-129-39 kubenswrapper[2575]: I0707 17:47:24.205889 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/d270ae63-e153-4532-8963-16397c54c338-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-86878c769f-x85t4\" (UID: \"d270ae63-e153-4532-8963-16397c54c338\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-x85t4" Jul 07 17:47:24.205953 ip-10-0-129-39 kubenswrapper[2575]: I0707 17:47:24.205932 2575 reconciler_common.go:299] "Volume detached for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/419be36b-c6a1-4709-90da-3622dab0bfee-extensions-socket-volume\") on node \"ip-10-0-129-39.ec2.internal\" DevicePath \"\"" Jul 07 17:47:24.205953 ip-10-0-129-39 kubenswrapper[2575]: I0707 17:47:24.205942 2575 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ftkgn\" (UniqueName: \"kubernetes.io/projected/31101d8f-1952-4356-838c-7bde0aad908a-kube-api-access-ftkgn\") on node \"ip-10-0-129-39.ec2.internal\" DevicePath \"\"" Jul 07 17:47:24.205953 ip-10-0-129-39 kubenswrapper[2575]: I0707 17:47:24.205951 2575 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hrnpk\" (UniqueName: \"kubernetes.io/projected/419be36b-c6a1-4709-90da-3622dab0bfee-kube-api-access-hrnpk\") on node \"ip-10-0-129-39.ec2.internal\" DevicePath \"\"" Jul 07 17:47:24.206216 ip-10-0-129-39 kubenswrapper[2575]: I0707 17:47:24.206197 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/d270ae63-e153-4532-8963-16397c54c338-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-86878c769f-x85t4\" (UID: \"d270ae63-e153-4532-8963-16397c54c338\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-x85t4" Jul 07 17:47:24.222060 ip-10-0-129-39 kubenswrapper[2575]: I0707 17:47:24.222035 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zm4q9\" (UniqueName: \"kubernetes.io/projected/d270ae63-e153-4532-8963-16397c54c338-kube-api-access-zm4q9\") pod \"kuadrant-operator-controller-manager-86878c769f-x85t4\" (UID: \"d270ae63-e153-4532-8963-16397c54c338\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-x85t4" Jul 07 17:47:24.318989 ip-10-0-129-39 kubenswrapper[2575]: I0707 17:47:24.318968 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-rg267"] Jul 07 17:47:24.320799 ip-10-0-129-39 kubenswrapper[2575]: W0707 17:47:24.320774 2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf4ec98a5_a47b_46b3_a9c4_8ea072fb219c.slice/crio-c95bcab3cf5339287e2db9eb2eafe2b0b3a464d6720f44e131c13d927b5d3bcb WatchSource:0}: Error finding container c95bcab3cf5339287e2db9eb2eafe2b0b3a464d6720f44e131c13d927b5d3bcb: Status 404 returned error can't find the container with id c95bcab3cf5339287e2db9eb2eafe2b0b3a464d6720f44e131c13d927b5d3bcb Jul 07 17:47:24.369597 ip-10-0-129-39 kubenswrapper[2575]: I0707 17:47:24.369579 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-x85t4" Jul 07 17:47:24.526455 ip-10-0-129-39 kubenswrapper[2575]: I0707 17:47:24.526376 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-86878c769f-x85t4"] Jul 07 17:47:24.530117 ip-10-0-129-39 kubenswrapper[2575]: W0707 17:47:24.530093 2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd270ae63_e153_4532_8963_16397c54c338.slice/crio-ae236fa1f198a7dc46d6b06bb49d5d9d702f79e619ba099cff3deee15c76180f WatchSource:0}: Error finding container ae236fa1f198a7dc46d6b06bb49d5d9d702f79e619ba099cff3deee15c76180f: Status 404 returned error can't find the container with id ae236fa1f198a7dc46d6b06bb49d5d9d702f79e619ba099cff3deee15c76180f Jul 07 17:47:24.764514 ip-10-0-129-39 kubenswrapper[2575]: I0707 17:47:24.764458 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-x85t4" event={"ID":"d270ae63-e153-4532-8963-16397c54c338","Type":"ContainerStarted","Data":"5bf41ae976b1b9de65b3d9543875097f3d625f98f56ff7f7b24b680a80791386"} Jul 07 17:47:24.764941 ip-10-0-129-39 kubenswrapper[2575]: I0707 17:47:24.764581 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-x85t4" Jul 07 17:47:24.764941 ip-10-0-129-39 kubenswrapper[2575]: I0707 17:47:24.764603 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-x85t4" event={"ID":"d270ae63-e153-4532-8963-16397c54c338","Type":"ContainerStarted","Data":"ae236fa1f198a7dc46d6b06bb49d5d9d702f79e619ba099cff3deee15c76180f"} Jul 07 17:47:24.765888 ip-10-0-129-39 kubenswrapper[2575]: I0707 17:47:24.765860 2575 generic.go:356] "Generic (PLEG): container finished" podID="419be36b-c6a1-4709-90da-3622dab0bfee" containerID="f19839a92d2f25d689463ffd7df42b1e187f2d4461e2dace46a267e6ced3b51e" exitCode=0 Jul 07 17:47:24.766045 ip-10-0-129-39 kubenswrapper[2575]: I0707 17:47:24.765901 2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-dc76n" Jul 07 17:47:24.766786 ip-10-0-129-39 kubenswrapper[2575]: I0707 17:47:24.765959 2575 scope.go:117] "RemoveContainer" containerID="f19839a92d2f25d689463ffd7df42b1e187f2d4461e2dace46a267e6ced3b51e" Jul 07 17:47:24.767705 ip-10-0-129-39 kubenswrapper[2575]: I0707 17:47:24.767678 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-rg267" event={"ID":"f4ec98a5-a47b-46b3-a9c4-8ea072fb219c","Type":"ContainerStarted","Data":"4a5f99af1cfc84ce0f4db260bb7497e182be3ee7854bfa26dfb4065127842156"} Jul 07 17:47:24.767805 ip-10-0-129-39 kubenswrapper[2575]: I0707 17:47:24.767727 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-rg267" Jul 07 17:47:24.767805 ip-10-0-129-39 kubenswrapper[2575]: I0707 17:47:24.767753 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-rg267" event={"ID":"f4ec98a5-a47b-46b3-a9c4-8ea072fb219c","Type":"ContainerStarted","Data":"c95bcab3cf5339287e2db9eb2eafe2b0b3a464d6720f44e131c13d927b5d3bcb"} Jul 07 17:47:24.769269 ip-10-0-129-39 kubenswrapper[2575]: I0707 17:47:24.769243 2575 generic.go:356] "Generic (PLEG): container finished" podID="31101d8f-1952-4356-838c-7bde0aad908a" containerID="ac9e7c7a92ebaf2542dea697f54735fde9c6d96f6f88865eb5c03bdf7ce0d565" exitCode=0 Jul 07 17:47:24.769388 ip-10-0-129-39 kubenswrapper[2575]: I0707 17:47:24.769288 2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-h2v4h" Jul 07 17:47:24.779943 ip-10-0-129-39 kubenswrapper[2575]: I0707 17:47:24.779929 2575 scope.go:117] "RemoveContainer" containerID="f19839a92d2f25d689463ffd7df42b1e187f2d4461e2dace46a267e6ced3b51e" Jul 07 17:47:24.780195 ip-10-0-129-39 kubenswrapper[2575]: E0707 17:47:24.780178 2575 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f19839a92d2f25d689463ffd7df42b1e187f2d4461e2dace46a267e6ced3b51e\": container with ID starting with f19839a92d2f25d689463ffd7df42b1e187f2d4461e2dace46a267e6ced3b51e not found: ID does not exist" containerID="f19839a92d2f25d689463ffd7df42b1e187f2d4461e2dace46a267e6ced3b51e" Jul 07 17:47:24.780245 ip-10-0-129-39 kubenswrapper[2575]: I0707 17:47:24.780203 2575 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f19839a92d2f25d689463ffd7df42b1e187f2d4461e2dace46a267e6ced3b51e"} err="failed to get container status \"f19839a92d2f25d689463ffd7df42b1e187f2d4461e2dace46a267e6ced3b51e\": rpc error: code = NotFound desc = could not find container \"f19839a92d2f25d689463ffd7df42b1e187f2d4461e2dace46a267e6ced3b51e\": container with ID starting with f19839a92d2f25d689463ffd7df42b1e187f2d4461e2dace46a267e6ced3b51e not found: ID does not exist" Jul 07 17:47:24.780245 ip-10-0-129-39 kubenswrapper[2575]: I0707 17:47:24.780219 2575 scope.go:117] "RemoveContainer" containerID="ac9e7c7a92ebaf2542dea697f54735fde9c6d96f6f88865eb5c03bdf7ce0d565" Jul 07 17:47:24.787263 ip-10-0-129-39 kubenswrapper[2575]: I0707 17:47:24.787246 2575 scope.go:117] "RemoveContainer" containerID="ac9e7c7a92ebaf2542dea697f54735fde9c6d96f6f88865eb5c03bdf7ce0d565" Jul 07 17:47:24.787534 ip-10-0-129-39 kubenswrapper[2575]: E0707 17:47:24.787493 2575 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ac9e7c7a92ebaf2542dea697f54735fde9c6d96f6f88865eb5c03bdf7ce0d565\": container with ID starting with ac9e7c7a92ebaf2542dea697f54735fde9c6d96f6f88865eb5c03bdf7ce0d565 not found: ID does not exist" containerID="ac9e7c7a92ebaf2542dea697f54735fde9c6d96f6f88865eb5c03bdf7ce0d565" Jul 07 17:47:24.787586 ip-10-0-129-39 kubenswrapper[2575]: I0707 17:47:24.787539 2575 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ac9e7c7a92ebaf2542dea697f54735fde9c6d96f6f88865eb5c03bdf7ce0d565"} err="failed to get container status \"ac9e7c7a92ebaf2542dea697f54735fde9c6d96f6f88865eb5c03bdf7ce0d565\": rpc error: code = NotFound desc = could not find container \"ac9e7c7a92ebaf2542dea697f54735fde9c6d96f6f88865eb5c03bdf7ce0d565\": container with ID starting with ac9e7c7a92ebaf2542dea697f54735fde9c6d96f6f88865eb5c03bdf7ce0d565 not found: ID does not exist" Jul 07 17:47:24.829533 ip-10-0-129-39 kubenswrapper[2575]: I0707 17:47:24.829477 2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-x85t4" podStartSLOduration=0.829462087 podStartE2EDuration="829.462087ms" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-07 17:47:24 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 17:47:24.826035983 +0000 UTC m=+405.331842650" watchObservedRunningTime="2026-07-07 17:47:24.829462087 +0000 UTC m=+405.335268753" Jul 07 17:47:24.946350 ip-10-0-129-39 kubenswrapper[2575]: I0707 17:47:24.946297 2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-rg267" podStartSLOduration=1.946281775 podStartE2EDuration="1.946281775s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-07 17:47:23 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 17:47:24.945322665 +0000 UTC m=+405.451129331" watchObservedRunningTime="2026-07-07 17:47:24.946281775 +0000 UTC m=+405.452088440" Jul 07 17:47:26.073552 ip-10-0-129-39 kubenswrapper[2575]: I0707 17:47:26.073523 2575 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="31101d8f-1952-4356-838c-7bde0aad908a" path="/var/lib/kubelet/pods/31101d8f-1952-4356-838c-7bde0aad908a/volumes" Jul 07 17:47:26.073922 ip-10-0-129-39 kubenswrapper[2575]: I0707 17:47:26.073866 2575 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="419be36b-c6a1-4709-90da-3622dab0bfee" path="/var/lib/kubelet/pods/419be36b-c6a1-4709-90da-3622dab0bfee/volumes" Jul 07 17:47:35.779640 ip-10-0-129-39 kubenswrapper[2575]: I0707 17:47:35.779615 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-rg267" Jul 07 17:47:35.780087 ip-10-0-129-39 kubenswrapper[2575]: I0707 17:47:35.779682 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-x85t4" Jul 07 17:47:35.905686 ip-10-0-129-39 kubenswrapper[2575]: I0707 17:47:35.905657 2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-rg267"] Jul 07 17:47:35.905854 ip-10-0-129-39 kubenswrapper[2575]: I0707 17:47:35.905829 2575 kuberuntime_container.go:872] "Killing container with a grace period" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-rg267" podUID="f4ec98a5-a47b-46b3-a9c4-8ea072fb219c" containerName="manager" containerID="cri-o://4a5f99af1cfc84ce0f4db260bb7497e182be3ee7854bfa26dfb4065127842156" gracePeriod=10 Jul 07 17:47:36.149794 ip-10-0-129-39 kubenswrapper[2575]: I0707 17:47:36.149768 2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-rg267" Jul 07 17:47:36.203556 ip-10-0-129-39 kubenswrapper[2575]: I0707 17:47:36.203532 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-8m7pq\" (UniqueName: \"kubernetes.io/projected/f4ec98a5-a47b-46b3-a9c4-8ea072fb219c-kube-api-access-8m7pq\") pod \"f4ec98a5-a47b-46b3-a9c4-8ea072fb219c\" (UID: \"f4ec98a5-a47b-46b3-a9c4-8ea072fb219c\") " Jul 07 17:47:36.203682 ip-10-0-129-39 kubenswrapper[2575]: I0707 17:47:36.203569 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/f4ec98a5-a47b-46b3-a9c4-8ea072fb219c-extensions-socket-volume\") pod \"f4ec98a5-a47b-46b3-a9c4-8ea072fb219c\" (UID: \"f4ec98a5-a47b-46b3-a9c4-8ea072fb219c\") " Jul 07 17:47:36.203981 ip-10-0-129-39 kubenswrapper[2575]: I0707 17:47:36.203960 2575 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f4ec98a5-a47b-46b3-a9c4-8ea072fb219c-extensions-socket-volume" (OuterVolumeSpecName: "extensions-socket-volume") pod "f4ec98a5-a47b-46b3-a9c4-8ea072fb219c" (UID: "f4ec98a5-a47b-46b3-a9c4-8ea072fb219c"). InnerVolumeSpecName "extensions-socket-volume". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 07 17:47:36.205540 ip-10-0-129-39 kubenswrapper[2575]: I0707 17:47:36.205492 2575 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f4ec98a5-a47b-46b3-a9c4-8ea072fb219c-kube-api-access-8m7pq" (OuterVolumeSpecName: "kube-api-access-8m7pq") pod "f4ec98a5-a47b-46b3-a9c4-8ea072fb219c" (UID: "f4ec98a5-a47b-46b3-a9c4-8ea072fb219c"). InnerVolumeSpecName "kube-api-access-8m7pq". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 07 17:47:36.304758 ip-10-0-129-39 kubenswrapper[2575]: I0707 17:47:36.304730 2575 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8m7pq\" (UniqueName: \"kubernetes.io/projected/f4ec98a5-a47b-46b3-a9c4-8ea072fb219c-kube-api-access-8m7pq\") on node \"ip-10-0-129-39.ec2.internal\" DevicePath \"\"" Jul 07 17:47:36.304758 ip-10-0-129-39 kubenswrapper[2575]: I0707 17:47:36.304754 2575 reconciler_common.go:299] "Volume detached for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/f4ec98a5-a47b-46b3-a9c4-8ea072fb219c-extensions-socket-volume\") on node \"ip-10-0-129-39.ec2.internal\" DevicePath \"\"" Jul 07 17:47:36.827568 ip-10-0-129-39 kubenswrapper[2575]: I0707 17:47:36.827494 2575 generic.go:356] "Generic (PLEG): container finished" podID="f4ec98a5-a47b-46b3-a9c4-8ea072fb219c" containerID="4a5f99af1cfc84ce0f4db260bb7497e182be3ee7854bfa26dfb4065127842156" exitCode=0 Jul 07 17:47:36.827568 ip-10-0-129-39 kubenswrapper[2575]: I0707 17:47:36.827563 2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-rg267" Jul 07 17:47:36.828090 ip-10-0-129-39 kubenswrapper[2575]: I0707 17:47:36.827582 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-rg267" event={"ID":"f4ec98a5-a47b-46b3-a9c4-8ea072fb219c","Type":"ContainerDied","Data":"4a5f99af1cfc84ce0f4db260bb7497e182be3ee7854bfa26dfb4065127842156"} Jul 07 17:47:36.828090 ip-10-0-129-39 kubenswrapper[2575]: I0707 17:47:36.827663 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-rg267" event={"ID":"f4ec98a5-a47b-46b3-a9c4-8ea072fb219c","Type":"ContainerDied","Data":"c95bcab3cf5339287e2db9eb2eafe2b0b3a464d6720f44e131c13d927b5d3bcb"} Jul 07 17:47:36.828090 ip-10-0-129-39 kubenswrapper[2575]: I0707 17:47:36.827686 2575 scope.go:117] "RemoveContainer" containerID="4a5f99af1cfc84ce0f4db260bb7497e182be3ee7854bfa26dfb4065127842156" Jul 07 17:47:36.835981 ip-10-0-129-39 kubenswrapper[2575]: I0707 17:47:36.835959 2575 scope.go:117] "RemoveContainer" containerID="4a5f99af1cfc84ce0f4db260bb7497e182be3ee7854bfa26dfb4065127842156" Jul 07 17:47:36.836236 ip-10-0-129-39 kubenswrapper[2575]: E0707 17:47:36.836218 2575 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4a5f99af1cfc84ce0f4db260bb7497e182be3ee7854bfa26dfb4065127842156\": container with ID starting with 4a5f99af1cfc84ce0f4db260bb7497e182be3ee7854bfa26dfb4065127842156 not found: ID does not exist" containerID="4a5f99af1cfc84ce0f4db260bb7497e182be3ee7854bfa26dfb4065127842156" Jul 07 17:47:36.836341 ip-10-0-129-39 kubenswrapper[2575]: I0707 17:47:36.836248 2575 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4a5f99af1cfc84ce0f4db260bb7497e182be3ee7854bfa26dfb4065127842156"} err="failed to get container status \"4a5f99af1cfc84ce0f4db260bb7497e182be3ee7854bfa26dfb4065127842156\": rpc error: code = NotFound desc = could not find container \"4a5f99af1cfc84ce0f4db260bb7497e182be3ee7854bfa26dfb4065127842156\": container with ID starting with 4a5f99af1cfc84ce0f4db260bb7497e182be3ee7854bfa26dfb4065127842156 not found: ID does not exist" Jul 07 17:47:36.862803 ip-10-0-129-39 kubenswrapper[2575]: I0707 17:47:36.862773 2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-rg267"] Jul 07 17:47:36.872593 ip-10-0-129-39 kubenswrapper[2575]: I0707 17:47:36.872568 2575 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-rg267"] Jul 07 17:47:38.072884 ip-10-0-129-39 kubenswrapper[2575]: I0707 17:47:38.072850 2575 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="f4ec98a5-a47b-46b3-a9c4-8ea072fb219c" path="/var/lib/kubelet/pods/f4ec98a5-a47b-46b3-a9c4-8ea072fb219c/volumes" Jul 07 17:47:52.637186 ip-10-0-129-39 kubenswrapper[2575]: I0707 17:47:52.637152 2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-q6xht"] Jul 07 17:47:52.637582 ip-10-0-129-39 kubenswrapper[2575]: I0707 17:47:52.637495 2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f4ec98a5-a47b-46b3-a9c4-8ea072fb219c" containerName="manager" Jul 07 17:47:52.637582 ip-10-0-129-39 kubenswrapper[2575]: I0707 17:47:52.637522 2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="f4ec98a5-a47b-46b3-a9c4-8ea072fb219c" containerName="manager" Jul 07 17:47:52.637582 ip-10-0-129-39 kubenswrapper[2575]: I0707 17:47:52.637574 2575 memory_manager.go:368] "RemoveStaleState removing state" podUID="f4ec98a5-a47b-46b3-a9c4-8ea072fb219c" containerName="manager" Jul 07 17:47:52.642266 ip-10-0-129-39 kubenswrapper[2575]: I0707 17:47:52.642245 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-q6xht" Jul 07 17:47:52.645141 ip-10-0-129-39 kubenswrapper[2575]: I0707 17:47:52.645123 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"maas-default-gateway-openshift-default-dockercfg-hprw2\"" Jul 07 17:47:52.674430 ip-10-0-129-39 kubenswrapper[2575]: I0707 17:47:52.674400 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-q6xht"] Jul 07 17:47:52.744263 ip-10-0-129-39 kubenswrapper[2575]: I0707 17:47:52.744223 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/45428b46-dfbb-4c68-8a6c-ceffc91e8506-istio-token\") pod \"maas-default-gateway-openshift-default-56654dcf7-q6xht\" (UID: \"45428b46-dfbb-4c68-8a6c-ceffc91e8506\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-q6xht" Jul 07 17:47:52.744263 ip-10-0-129-39 kubenswrapper[2575]: I0707 17:47:52.744268 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/45428b46-dfbb-4c68-8a6c-ceffc91e8506-credential-socket\") pod \"maas-default-gateway-openshift-default-56654dcf7-q6xht\" (UID: \"45428b46-dfbb-4c68-8a6c-ceffc91e8506\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-q6xht" Jul 07 17:47:52.744490 ip-10-0-129-39 kubenswrapper[2575]: I0707 17:47:52.744331 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hxx77\" (UniqueName: \"kubernetes.io/projected/45428b46-dfbb-4c68-8a6c-ceffc91e8506-kube-api-access-hxx77\") pod \"maas-default-gateway-openshift-default-56654dcf7-q6xht\" (UID: \"45428b46-dfbb-4c68-8a6c-ceffc91e8506\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-q6xht" Jul 07 17:47:52.744490 ip-10-0-129-39 kubenswrapper[2575]: I0707 17:47:52.744401 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/45428b46-dfbb-4c68-8a6c-ceffc91e8506-workload-socket\") pod \"maas-default-gateway-openshift-default-56654dcf7-q6xht\" (UID: \"45428b46-dfbb-4c68-8a6c-ceffc91e8506\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-q6xht" Jul 07 17:47:52.744490 ip-10-0-129-39 kubenswrapper[2575]: I0707 17:47:52.744429 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/45428b46-dfbb-4c68-8a6c-ceffc91e8506-istio-data\") pod \"maas-default-gateway-openshift-default-56654dcf7-q6xht\" (UID: \"45428b46-dfbb-4c68-8a6c-ceffc91e8506\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-q6xht" Jul 07 17:47:52.744490 ip-10-0-129-39 kubenswrapper[2575]: I0707 17:47:52.744466 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/45428b46-dfbb-4c68-8a6c-ceffc91e8506-workload-certs\") pod \"maas-default-gateway-openshift-default-56654dcf7-q6xht\" (UID: \"45428b46-dfbb-4c68-8a6c-ceffc91e8506\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-q6xht" Jul 07 17:47:52.744490 ip-10-0-129-39 kubenswrapper[2575]: I0707 17:47:52.744487 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/45428b46-dfbb-4c68-8a6c-ceffc91e8506-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-56654dcf7-q6xht\" (UID: \"45428b46-dfbb-4c68-8a6c-ceffc91e8506\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-q6xht" Jul 07 17:47:52.744708 ip-10-0-129-39 kubenswrapper[2575]: I0707 17:47:52.744531 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/45428b46-dfbb-4c68-8a6c-ceffc91e8506-istio-envoy\") pod \"maas-default-gateway-openshift-default-56654dcf7-q6xht\" (UID: \"45428b46-dfbb-4c68-8a6c-ceffc91e8506\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-q6xht" Jul 07 17:47:52.744708 ip-10-0-129-39 kubenswrapper[2575]: I0707 17:47:52.744556 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/45428b46-dfbb-4c68-8a6c-ceffc91e8506-istio-podinfo\") pod \"maas-default-gateway-openshift-default-56654dcf7-q6xht\" (UID: \"45428b46-dfbb-4c68-8a6c-ceffc91e8506\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-q6xht" Jul 07 17:47:52.845590 ip-10-0-129-39 kubenswrapper[2575]: I0707 17:47:52.845555 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/45428b46-dfbb-4c68-8a6c-ceffc91e8506-istio-podinfo\") pod \"maas-default-gateway-openshift-default-56654dcf7-q6xht\" (UID: \"45428b46-dfbb-4c68-8a6c-ceffc91e8506\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-q6xht" Jul 07 17:47:52.845779 ip-10-0-129-39 kubenswrapper[2575]: I0707 17:47:52.845602 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/45428b46-dfbb-4c68-8a6c-ceffc91e8506-istio-token\") pod \"maas-default-gateway-openshift-default-56654dcf7-q6xht\" (UID: \"45428b46-dfbb-4c68-8a6c-ceffc91e8506\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-q6xht" Jul 07 17:47:52.845779 ip-10-0-129-39 kubenswrapper[2575]: I0707 17:47:52.845640 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/45428b46-dfbb-4c68-8a6c-ceffc91e8506-credential-socket\") pod \"maas-default-gateway-openshift-default-56654dcf7-q6xht\" (UID: \"45428b46-dfbb-4c68-8a6c-ceffc91e8506\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-q6xht" Jul 07 17:47:52.845779 ip-10-0-129-39 kubenswrapper[2575]: I0707 17:47:52.845668 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hxx77\" (UniqueName: \"kubernetes.io/projected/45428b46-dfbb-4c68-8a6c-ceffc91e8506-kube-api-access-hxx77\") pod \"maas-default-gateway-openshift-default-56654dcf7-q6xht\" (UID: \"45428b46-dfbb-4c68-8a6c-ceffc91e8506\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-q6xht" Jul 07 17:47:52.845779 ip-10-0-129-39 kubenswrapper[2575]: I0707 17:47:52.845717 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/45428b46-dfbb-4c68-8a6c-ceffc91e8506-workload-socket\") pod \"maas-default-gateway-openshift-default-56654dcf7-q6xht\" (UID: \"45428b46-dfbb-4c68-8a6c-ceffc91e8506\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-q6xht" Jul 07 17:47:52.845779 ip-10-0-129-39 kubenswrapper[2575]: I0707 17:47:52.845742 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/45428b46-dfbb-4c68-8a6c-ceffc91e8506-istio-data\") pod \"maas-default-gateway-openshift-default-56654dcf7-q6xht\" (UID: \"45428b46-dfbb-4c68-8a6c-ceffc91e8506\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-q6xht" Jul 07 17:47:52.846030 ip-10-0-129-39 kubenswrapper[2575]: I0707 17:47:52.845782 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/45428b46-dfbb-4c68-8a6c-ceffc91e8506-workload-certs\") pod \"maas-default-gateway-openshift-default-56654dcf7-q6xht\" (UID: \"45428b46-dfbb-4c68-8a6c-ceffc91e8506\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-q6xht" Jul 07 17:47:52.846030 ip-10-0-129-39 kubenswrapper[2575]: I0707 17:47:52.845813 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/45428b46-dfbb-4c68-8a6c-ceffc91e8506-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-56654dcf7-q6xht\" (UID: \"45428b46-dfbb-4c68-8a6c-ceffc91e8506\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-q6xht" Jul 07 17:47:52.846030 ip-10-0-129-39 kubenswrapper[2575]: I0707 17:47:52.845849 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/45428b46-dfbb-4c68-8a6c-ceffc91e8506-istio-envoy\") pod \"maas-default-gateway-openshift-default-56654dcf7-q6xht\" (UID: \"45428b46-dfbb-4c68-8a6c-ceffc91e8506\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-q6xht" Jul 07 17:47:52.846195 ip-10-0-129-39 kubenswrapper[2575]: I0707 17:47:52.846166 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/45428b46-dfbb-4c68-8a6c-ceffc91e8506-workload-socket\") pod \"maas-default-gateway-openshift-default-56654dcf7-q6xht\" (UID: \"45428b46-dfbb-4c68-8a6c-ceffc91e8506\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-q6xht" Jul 07 17:47:52.846254 ip-10-0-129-39 kubenswrapper[2575]: I0707 17:47:52.846198 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/45428b46-dfbb-4c68-8a6c-ceffc91e8506-credential-socket\") pod \"maas-default-gateway-openshift-default-56654dcf7-q6xht\" (UID: \"45428b46-dfbb-4c68-8a6c-ceffc91e8506\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-q6xht" Jul 07 17:47:52.846299 ip-10-0-129-39 kubenswrapper[2575]: I0707 17:47:52.846253 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/45428b46-dfbb-4c68-8a6c-ceffc91e8506-workload-certs\") pod \"maas-default-gateway-openshift-default-56654dcf7-q6xht\" (UID: \"45428b46-dfbb-4c68-8a6c-ceffc91e8506\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-q6xht" Jul 07 17:47:52.846356 ip-10-0-129-39 kubenswrapper[2575]: I0707 17:47:52.846332 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/45428b46-dfbb-4c68-8a6c-ceffc91e8506-istio-data\") pod \"maas-default-gateway-openshift-default-56654dcf7-q6xht\" (UID: \"45428b46-dfbb-4c68-8a6c-ceffc91e8506\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-q6xht" Jul 07 17:47:52.846662 ip-10-0-129-39 kubenswrapper[2575]: I0707 17:47:52.846644 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/45428b46-dfbb-4c68-8a6c-ceffc91e8506-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-56654dcf7-q6xht\" (UID: \"45428b46-dfbb-4c68-8a6c-ceffc91e8506\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-q6xht" Jul 07 17:47:52.848226 ip-10-0-129-39 kubenswrapper[2575]: I0707 17:47:52.848201 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/45428b46-dfbb-4c68-8a6c-ceffc91e8506-istio-envoy\") pod \"maas-default-gateway-openshift-default-56654dcf7-q6xht\" (UID: \"45428b46-dfbb-4c68-8a6c-ceffc91e8506\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-q6xht" Jul 07 17:47:52.848293 ip-10-0-129-39 kubenswrapper[2575]: I0707 17:47:52.848257 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/45428b46-dfbb-4c68-8a6c-ceffc91e8506-istio-podinfo\") pod \"maas-default-gateway-openshift-default-56654dcf7-q6xht\" (UID: \"45428b46-dfbb-4c68-8a6c-ceffc91e8506\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-q6xht" Jul 07 17:47:52.857189 ip-10-0-129-39 kubenswrapper[2575]: I0707 17:47:52.857163 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/45428b46-dfbb-4c68-8a6c-ceffc91e8506-istio-token\") pod \"maas-default-gateway-openshift-default-56654dcf7-q6xht\" (UID: \"45428b46-dfbb-4c68-8a6c-ceffc91e8506\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-q6xht" Jul 07 17:47:52.863140 ip-10-0-129-39 kubenswrapper[2575]: I0707 17:47:52.863115 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hxx77\" (UniqueName: \"kubernetes.io/projected/45428b46-dfbb-4c68-8a6c-ceffc91e8506-kube-api-access-hxx77\") pod \"maas-default-gateway-openshift-default-56654dcf7-q6xht\" (UID: \"45428b46-dfbb-4c68-8a6c-ceffc91e8506\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-q6xht" Jul 07 17:47:52.954083 ip-10-0-129-39 kubenswrapper[2575]: I0707 17:47:52.953989 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-q6xht" Jul 07 17:47:53.107312 ip-10-0-129-39 kubenswrapper[2575]: I0707 17:47:53.107286 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-q6xht"] Jul 07 17:47:53.109788 ip-10-0-129-39 kubenswrapper[2575]: W0707 17:47:53.109762 2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod45428b46_dfbb_4c68_8a6c_ceffc91e8506.slice/crio-87257f216433308b1250939dbd8aba8061844101eea09028b9dfe26bebca4710 WatchSource:0}: Error finding container 87257f216433308b1250939dbd8aba8061844101eea09028b9dfe26bebca4710: Status 404 returned error can't find the container with id 87257f216433308b1250939dbd8aba8061844101eea09028b9dfe26bebca4710 Jul 07 17:47:53.111812 ip-10-0-129-39 kubenswrapper[2575]: I0707 17:47:53.111777 2575 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"} Jul 07 17:47:53.111920 ip-10-0-129-39 kubenswrapper[2575]: I0707 17:47:53.111851 2575 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"} Jul 07 17:47:53.111920 ip-10-0-129-39 kubenswrapper[2575]: I0707 17:47:53.111880 2575 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"} Jul 07 17:47:53.901043 ip-10-0-129-39 kubenswrapper[2575]: I0707 17:47:53.901007 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-q6xht" event={"ID":"45428b46-dfbb-4c68-8a6c-ceffc91e8506","Type":"ContainerStarted","Data":"4563a5f91fc589e8d82e42909c180b33a4a0784314e301ded5b1415186c2b18e"} Jul 07 17:47:53.901399 ip-10-0-129-39 kubenswrapper[2575]: I0707 17:47:53.901078 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-q6xht" event={"ID":"45428b46-dfbb-4c68-8a6c-ceffc91e8506","Type":"ContainerStarted","Data":"87257f216433308b1250939dbd8aba8061844101eea09028b9dfe26bebca4710"} Jul 07 17:47:53.954416 ip-10-0-129-39 kubenswrapper[2575]: I0707 17:47:53.954392 2575 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-q6xht" Jul 07 17:47:53.971256 ip-10-0-129-39 kubenswrapper[2575]: I0707 17:47:53.971205 2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-q6xht" podStartSLOduration=1.971187622 podStartE2EDuration="1.971187622s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-07 17:47:52 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 17:47:53.970582016 +0000 UTC m=+434.476388683" watchObservedRunningTime="2026-07-07 17:47:53.971187622 +0000 UTC m=+434.476994289" Jul 07 17:47:54.958812 ip-10-0-129-39 kubenswrapper[2575]: I0707 17:47:54.958783 2575 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-q6xht" Jul 07 17:47:55.911002 ip-10-0-129-39 kubenswrapper[2575]: I0707 17:47:55.910971 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-q6xht" Jul 07 17:47:55.911929 ip-10-0-129-39 kubenswrapper[2575]: I0707 17:47:55.911906 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-q6xht" Jul 07 17:47:56.726155 ip-10-0-129-39 kubenswrapper[2575]: I0707 17:47:56.726121 2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-58cd955675-g7njm"] Jul 07 17:47:56.728323 ip-10-0-129-39 kubenswrapper[2575]: I0707 17:47:56.728308 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-58cd955675-g7njm" Jul 07 17:47:56.734307 ip-10-0-129-39 kubenswrapper[2575]: I0707 17:47:56.734285 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-authorino-dockercfg-vx6wg\"" Jul 07 17:47:56.745364 ip-10-0-129-39 kubenswrapper[2575]: I0707 17:47:56.745341 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-58cd955675-g7njm"] Jul 07 17:47:56.880298 ip-10-0-129-39 kubenswrapper[2575]: I0707 17:47:56.880267 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-88mgg\" (UniqueName: \"kubernetes.io/projected/cc16dd26-15fc-4465-8f03-cf16ac77d9cf-kube-api-access-88mgg\") pod \"authorino-58cd955675-g7njm\" (UID: \"cc16dd26-15fc-4465-8f03-cf16ac77d9cf\") " pod="kuadrant-system/authorino-58cd955675-g7njm" Jul 07 17:47:56.981468 ip-10-0-129-39 kubenswrapper[2575]: I0707 17:47:56.981396 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-88mgg\" (UniqueName: \"kubernetes.io/projected/cc16dd26-15fc-4465-8f03-cf16ac77d9cf-kube-api-access-88mgg\") pod \"authorino-58cd955675-g7njm\" (UID: \"cc16dd26-15fc-4465-8f03-cf16ac77d9cf\") " pod="kuadrant-system/authorino-58cd955675-g7njm" Jul 07 17:47:57.006845 ip-10-0-129-39 kubenswrapper[2575]: I0707 17:47:57.006819 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-88mgg\" (UniqueName: \"kubernetes.io/projected/cc16dd26-15fc-4465-8f03-cf16ac77d9cf-kube-api-access-88mgg\") pod \"authorino-58cd955675-g7njm\" (UID: \"cc16dd26-15fc-4465-8f03-cf16ac77d9cf\") " pod="kuadrant-system/authorino-58cd955675-g7njm" Jul 07 17:47:57.037374 ip-10-0-129-39 kubenswrapper[2575]: I0707 17:47:57.037352 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-58cd955675-g7njm" Jul 07 17:47:57.184853 ip-10-0-129-39 kubenswrapper[2575]: I0707 17:47:57.184827 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-58cd955675-g7njm"] Jul 07 17:47:57.187087 ip-10-0-129-39 kubenswrapper[2575]: W0707 17:47:57.187059 2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podcc16dd26_15fc_4465_8f03_cf16ac77d9cf.slice/crio-0a1284aa7dfe2d6c4dce5968d04a1c74612268ed62699993d383b661ed6a04e6 WatchSource:0}: Error finding container 0a1284aa7dfe2d6c4dce5968d04a1c74612268ed62699993d383b661ed6a04e6: Status 404 returned error can't find the container with id 0a1284aa7dfe2d6c4dce5968d04a1c74612268ed62699993d383b661ed6a04e6 Jul 07 17:47:57.925662 ip-10-0-129-39 kubenswrapper[2575]: I0707 17:47:57.925579 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-58cd955675-g7njm" event={"ID":"cc16dd26-15fc-4465-8f03-cf16ac77d9cf","Type":"ContainerStarted","Data":"0a1284aa7dfe2d6c4dce5968d04a1c74612268ed62699993d383b661ed6a04e6"} Jul 07 17:47:58.538711 ip-10-0-129-39 kubenswrapper[2575]: I0707 17:47:58.538658 2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-m825v"] Jul 07 17:47:58.541376 ip-10-0-129-39 kubenswrapper[2575]: I0707 17:47:58.541352 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-54d6d988d5-m825v" Jul 07 17:47:58.548168 ip-10-0-129-39 kubenswrapper[2575]: I0707 17:47:58.548141 2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"limitador-limits-config-limitador\"" Jul 07 17:47:58.548751 ip-10-0-129-39 kubenswrapper[2575]: I0707 17:47:58.548392 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"default-dockercfg-zbn5v\"" Jul 07 17:47:58.561216 ip-10-0-129-39 kubenswrapper[2575]: I0707 17:47:58.561193 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-m825v"] Jul 07 17:47:58.573920 ip-10-0-129-39 kubenswrapper[2575]: I0707 17:47:58.573864 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-m825v"] Jul 07 17:47:58.695065 ip-10-0-129-39 kubenswrapper[2575]: I0707 17:47:58.694998 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-n58g2\" (UniqueName: \"kubernetes.io/projected/f8a96214-b316-454a-9889-3c962c1ea035-kube-api-access-n58g2\") pod \"limitador-limitador-54d6d988d5-m825v\" (UID: \"f8a96214-b316-454a-9889-3c962c1ea035\") " pod="kuadrant-system/limitador-limitador-54d6d988d5-m825v" Jul 07 17:47:58.695237 ip-10-0-129-39 kubenswrapper[2575]: I0707 17:47:58.695142 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/f8a96214-b316-454a-9889-3c962c1ea035-config-file\") pod \"limitador-limitador-54d6d988d5-m825v\" (UID: \"f8a96214-b316-454a-9889-3c962c1ea035\") " pod="kuadrant-system/limitador-limitador-54d6d988d5-m825v" Jul 07 17:47:58.796476 ip-10-0-129-39 kubenswrapper[2575]: I0707 17:47:58.796399 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n58g2\" (UniqueName: \"kubernetes.io/projected/f8a96214-b316-454a-9889-3c962c1ea035-kube-api-access-n58g2\") pod \"limitador-limitador-54d6d988d5-m825v\" (UID: \"f8a96214-b316-454a-9889-3c962c1ea035\") " pod="kuadrant-system/limitador-limitador-54d6d988d5-m825v" Jul 07 17:47:58.796656 ip-10-0-129-39 kubenswrapper[2575]: I0707 17:47:58.796483 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/f8a96214-b316-454a-9889-3c962c1ea035-config-file\") pod \"limitador-limitador-54d6d988d5-m825v\" (UID: \"f8a96214-b316-454a-9889-3c962c1ea035\") " pod="kuadrant-system/limitador-limitador-54d6d988d5-m825v" Jul 07 17:47:58.797205 ip-10-0-129-39 kubenswrapper[2575]: I0707 17:47:58.797179 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/f8a96214-b316-454a-9889-3c962c1ea035-config-file\") pod \"limitador-limitador-54d6d988d5-m825v\" (UID: \"f8a96214-b316-454a-9889-3c962c1ea035\") " pod="kuadrant-system/limitador-limitador-54d6d988d5-m825v" Jul 07 17:47:58.808444 ip-10-0-129-39 kubenswrapper[2575]: I0707 17:47:58.808415 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-n58g2\" (UniqueName: \"kubernetes.io/projected/f8a96214-b316-454a-9889-3c962c1ea035-kube-api-access-n58g2\") pod \"limitador-limitador-54d6d988d5-m825v\" (UID: \"f8a96214-b316-454a-9889-3c962c1ea035\") " pod="kuadrant-system/limitador-limitador-54d6d988d5-m825v" Jul 07 17:47:58.856831 ip-10-0-129-39 kubenswrapper[2575]: I0707 17:47:58.856807 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-54d6d988d5-m825v" Jul 07 17:48:00.373790 ip-10-0-129-39 kubenswrapper[2575]: I0707 17:48:00.373661 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-m825v"] Jul 07 17:48:00.376477 ip-10-0-129-39 kubenswrapper[2575]: W0707 17:48:00.376448 2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf8a96214_b316_454a_9889_3c962c1ea035.slice/crio-aa13cc89673e71f33ef869f36277c5070ff9190b7f905459ec5a31aec458d172 WatchSource:0}: Error finding container aa13cc89673e71f33ef869f36277c5070ff9190b7f905459ec5a31aec458d172: Status 404 returned error can't find the container with id aa13cc89673e71f33ef869f36277c5070ff9190b7f905459ec5a31aec458d172 Jul 07 17:48:00.748681 ip-10-0-129-39 kubenswrapper[2575]: I0707 17:48:00.748646 2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-58cd955675-g7njm"] Jul 07 17:48:00.945401 ip-10-0-129-39 kubenswrapper[2575]: I0707 17:48:00.945359 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-54d6d988d5-m825v" event={"ID":"f8a96214-b316-454a-9889-3c962c1ea035","Type":"ContainerStarted","Data":"aa13cc89673e71f33ef869f36277c5070ff9190b7f905459ec5a31aec458d172"} Jul 07 17:48:00.947996 ip-10-0-129-39 kubenswrapper[2575]: I0707 17:48:00.947937 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-58cd955675-g7njm" event={"ID":"cc16dd26-15fc-4465-8f03-cf16ac77d9cf","Type":"ContainerStarted","Data":"e2f8ff73ab2c067a4a54b82b8f9892b07f007faac7fcf485eb18d970685b17bf"} Jul 07 17:48:01.952591 ip-10-0-129-39 kubenswrapper[2575]: I0707 17:48:01.952488 2575 kuberuntime_container.go:872] "Killing container with a grace period" pod="kuadrant-system/authorino-58cd955675-g7njm" podUID="cc16dd26-15fc-4465-8f03-cf16ac77d9cf" containerName="authorino" containerID="cri-o://e2f8ff73ab2c067a4a54b82b8f9892b07f007faac7fcf485eb18d970685b17bf" gracePeriod=30 Jul 07 17:48:01.981925 ip-10-0-129-39 kubenswrapper[2575]: I0707 17:48:01.981863 2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/authorino-58cd955675-g7njm" podStartSLOduration=2.916578582 podStartE2EDuration="5.98184344s" totalImagesPullingTime="3.065264858s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-07 17:47:56 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 17:48:00.992527998 +0000 UTC m=+441.498334667" watchObservedRunningTime="2026-07-07 17:48:01.98184344 +0000 UTC m=+442.487650107" Jul 07 17:48:02.436421 ip-10-0-129-39 kubenswrapper[2575]: I0707 17:48:02.436400 2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-58cd955675-g7njm" Jul 07 17:48:02.529359 ip-10-0-129-39 kubenswrapper[2575]: I0707 17:48:02.529328 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-88mgg\" (UniqueName: \"kubernetes.io/projected/cc16dd26-15fc-4465-8f03-cf16ac77d9cf-kube-api-access-88mgg\") pod \"cc16dd26-15fc-4465-8f03-cf16ac77d9cf\" (UID: \"cc16dd26-15fc-4465-8f03-cf16ac77d9cf\") " Jul 07 17:48:02.531556 ip-10-0-129-39 kubenswrapper[2575]: I0707 17:48:02.531522 2575 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cc16dd26-15fc-4465-8f03-cf16ac77d9cf-kube-api-access-88mgg" (OuterVolumeSpecName: "kube-api-access-88mgg") pod "cc16dd26-15fc-4465-8f03-cf16ac77d9cf" (UID: "cc16dd26-15fc-4465-8f03-cf16ac77d9cf"). InnerVolumeSpecName "kube-api-access-88mgg". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 07 17:48:02.630434 ip-10-0-129-39 kubenswrapper[2575]: I0707 17:48:02.630401 2575 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-88mgg\" (UniqueName: \"kubernetes.io/projected/cc16dd26-15fc-4465-8f03-cf16ac77d9cf-kube-api-access-88mgg\") on node \"ip-10-0-129-39.ec2.internal\" DevicePath \"\"" Jul 07 17:48:02.957984 ip-10-0-129-39 kubenswrapper[2575]: I0707 17:48:02.957960 2575 generic.go:356] "Generic (PLEG): container finished" podID="cc16dd26-15fc-4465-8f03-cf16ac77d9cf" containerID="e2f8ff73ab2c067a4a54b82b8f9892b07f007faac7fcf485eb18d970685b17bf" exitCode=0 Jul 07 17:48:02.958346 ip-10-0-129-39 kubenswrapper[2575]: I0707 17:48:02.958001 2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-58cd955675-g7njm" Jul 07 17:48:02.958346 ip-10-0-129-39 kubenswrapper[2575]: I0707 17:48:02.958043 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-58cd955675-g7njm" event={"ID":"cc16dd26-15fc-4465-8f03-cf16ac77d9cf","Type":"ContainerDied","Data":"e2f8ff73ab2c067a4a54b82b8f9892b07f007faac7fcf485eb18d970685b17bf"} Jul 07 17:48:02.958346 ip-10-0-129-39 kubenswrapper[2575]: I0707 17:48:02.958080 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-58cd955675-g7njm" event={"ID":"cc16dd26-15fc-4465-8f03-cf16ac77d9cf","Type":"ContainerDied","Data":"0a1284aa7dfe2d6c4dce5968d04a1c74612268ed62699993d383b661ed6a04e6"} Jul 07 17:48:02.958346 ip-10-0-129-39 kubenswrapper[2575]: I0707 17:48:02.958096 2575 scope.go:117] "RemoveContainer" containerID="e2f8ff73ab2c067a4a54b82b8f9892b07f007faac7fcf485eb18d970685b17bf" Jul 07 17:48:02.972610 ip-10-0-129-39 kubenswrapper[2575]: I0707 17:48:02.972591 2575 scope.go:117] "RemoveContainer" containerID="e2f8ff73ab2c067a4a54b82b8f9892b07f007faac7fcf485eb18d970685b17bf" Jul 07 17:48:02.972982 ip-10-0-129-39 kubenswrapper[2575]: E0707 17:48:02.972908 2575 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e2f8ff73ab2c067a4a54b82b8f9892b07f007faac7fcf485eb18d970685b17bf\": container with ID starting with e2f8ff73ab2c067a4a54b82b8f9892b07f007faac7fcf485eb18d970685b17bf not found: ID does not exist" containerID="e2f8ff73ab2c067a4a54b82b8f9892b07f007faac7fcf485eb18d970685b17bf" Jul 07 17:48:02.972982 ip-10-0-129-39 kubenswrapper[2575]: I0707 17:48:02.972933 2575 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e2f8ff73ab2c067a4a54b82b8f9892b07f007faac7fcf485eb18d970685b17bf"} err="failed to get container status \"e2f8ff73ab2c067a4a54b82b8f9892b07f007faac7fcf485eb18d970685b17bf\": rpc error: code = NotFound desc = could not find container \"e2f8ff73ab2c067a4a54b82b8f9892b07f007faac7fcf485eb18d970685b17bf\": container with ID starting with e2f8ff73ab2c067a4a54b82b8f9892b07f007faac7fcf485eb18d970685b17bf not found: ID does not exist" Jul 07 17:48:02.989880 ip-10-0-129-39 kubenswrapper[2575]: I0707 17:48:02.989855 2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-58cd955675-g7njm"] Jul 07 17:48:02.997985 ip-10-0-129-39 kubenswrapper[2575]: I0707 17:48:02.997966 2575 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-58cd955675-g7njm"] Jul 07 17:48:03.963942 ip-10-0-129-39 kubenswrapper[2575]: I0707 17:48:03.963905 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-54d6d988d5-m825v" event={"ID":"f8a96214-b316-454a-9889-3c962c1ea035","Type":"ContainerStarted","Data":"29200154caa8dadca2c0546d632dea8fb1aa16b1c205ebab8d041c6884959d8c"} Jul 07 17:48:03.964390 ip-10-0-129-39 kubenswrapper[2575]: I0707 17:48:03.963993 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/limitador-limitador-54d6d988d5-m825v" Jul 07 17:48:03.994302 ip-10-0-129-39 kubenswrapper[2575]: I0707 17:48:03.994257 2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/limitador-limitador-54d6d988d5-m825v" podStartSLOduration=3.464832638 podStartE2EDuration="5.994243666s" totalImagesPullingTime="2.529411028s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-07 17:47:58 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 17:48:03.99329321 +0000 UTC m=+444.499099876" watchObservedRunningTime="2026-07-07 17:48:03.994243666 +0000 UTC m=+444.500050332" Jul 07 17:48:04.073110 ip-10-0-129-39 kubenswrapper[2575]: I0707 17:48:04.073085 2575 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="cc16dd26-15fc-4465-8f03-cf16ac77d9cf" path="/var/lib/kubelet/pods/cc16dd26-15fc-4465-8f03-cf16ac77d9cf/volumes" Jul 07 17:48:14.969951 ip-10-0-129-39 kubenswrapper[2575]: I0707 17:48:14.969921 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/limitador-limitador-54d6d988d5-m825v" Jul 07 17:48:30.313495 ip-10-0-129-39 kubenswrapper[2575]: I0707 17:48:30.313462 2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-5d4b67c446-jqsgz"] Jul 07 17:48:30.313902 ip-10-0-129-39 kubenswrapper[2575]: I0707 17:48:30.313839 2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="cc16dd26-15fc-4465-8f03-cf16ac77d9cf" containerName="authorino" Jul 07 17:48:30.313902 ip-10-0-129-39 kubenswrapper[2575]: I0707 17:48:30.313850 2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="cc16dd26-15fc-4465-8f03-cf16ac77d9cf" containerName="authorino" Jul 07 17:48:30.313902 ip-10-0-129-39 kubenswrapper[2575]: I0707 17:48:30.313903 2575 memory_manager.go:368] "RemoveStaleState removing state" podUID="cc16dd26-15fc-4465-8f03-cf16ac77d9cf" containerName="authorino" Jul 07 17:48:30.339485 ip-10-0-129-39 kubenswrapper[2575]: I0707 17:48:30.339453 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-5d4b67c446-jqsgz"] Jul 07 17:48:30.339637 ip-10-0-129-39 kubenswrapper[2575]: I0707 17:48:30.339575 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-5d4b67c446-jqsgz" Jul 07 17:48:30.342414 ip-10-0-129-39 kubenswrapper[2575]: I0707 17:48:30.342391 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-authorino-dockercfg-vx6wg\"" Jul 07 17:48:30.441231 ip-10-0-129-39 kubenswrapper[2575]: I0707 17:48:30.441204 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9qnqs\" (UniqueName: \"kubernetes.io/projected/c1f603c0-4888-4489-859c-b973ca7d5406-kube-api-access-9qnqs\") pod \"authorino-5d4b67c446-jqsgz\" (UID: \"c1f603c0-4888-4489-859c-b973ca7d5406\") " pod="kuadrant-system/authorino-5d4b67c446-jqsgz" Jul 07 17:48:30.542642 ip-10-0-129-39 kubenswrapper[2575]: I0707 17:48:30.542608 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9qnqs\" (UniqueName: \"kubernetes.io/projected/c1f603c0-4888-4489-859c-b973ca7d5406-kube-api-access-9qnqs\") pod \"authorino-5d4b67c446-jqsgz\" (UID: \"c1f603c0-4888-4489-859c-b973ca7d5406\") " pod="kuadrant-system/authorino-5d4b67c446-jqsgz" Jul 07 17:48:30.569447 ip-10-0-129-39 kubenswrapper[2575]: I0707 17:48:30.569380 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9qnqs\" (UniqueName: \"kubernetes.io/projected/c1f603c0-4888-4489-859c-b973ca7d5406-kube-api-access-9qnqs\") pod \"authorino-5d4b67c446-jqsgz\" (UID: \"c1f603c0-4888-4489-859c-b973ca7d5406\") " pod="kuadrant-system/authorino-5d4b67c446-jqsgz" Jul 07 17:48:30.582276 ip-10-0-129-39 kubenswrapper[2575]: I0707 17:48:30.582250 2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-5d4b67c446-jqsgz"] Jul 07 17:48:30.582477 ip-10-0-129-39 kubenswrapper[2575]: I0707 17:48:30.582466 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-5d4b67c446-jqsgz" Jul 07 17:48:30.734573 ip-10-0-129-39 kubenswrapper[2575]: I0707 17:48:30.734525 2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-5d4b67c446-jqsgz"] Jul 07 17:48:30.737215 ip-10-0-129-39 kubenswrapper[2575]: W0707 17:48:30.737189 2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podc1f603c0_4888_4489_859c_b973ca7d5406.slice/crio-6e4f877aa67071955a92b1be7fec5b6797612f7459e470326fc5f302226c642f WatchSource:0}: Error finding container 6e4f877aa67071955a92b1be7fec5b6797612f7459e470326fc5f302226c642f: Status 404 returned error can't find the container with id 6e4f877aa67071955a92b1be7fec5b6797612f7459e470326fc5f302226c642f Jul 07 17:48:31.087728 ip-10-0-129-39 kubenswrapper[2575]: I0707 17:48:31.087697 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-5d4b67c446-jqsgz" event={"ID":"c1f603c0-4888-4489-859c-b973ca7d5406","Type":"ContainerStarted","Data":"6e4f877aa67071955a92b1be7fec5b6797612f7459e470326fc5f302226c642f"} Jul 07 17:48:32.093997 ip-10-0-129-39 kubenswrapper[2575]: I0707 17:48:32.093961 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-5d4b67c446-jqsgz" event={"ID":"c1f603c0-4888-4489-859c-b973ca7d5406","Type":"ContainerStarted","Data":"20d2cce85f64949c74d009bcc36dc22532f8f0d52c37694f5d46332c05987522"} Jul 07 17:48:32.094424 ip-10-0-129-39 kubenswrapper[2575]: I0707 17:48:32.094056 2575 kuberuntime_container.go:872] "Killing container with a grace period" pod="kuadrant-system/authorino-5d4b67c446-jqsgz" podUID="c1f603c0-4888-4489-859c-b973ca7d5406" containerName="authorino" containerID="cri-o://20d2cce85f64949c74d009bcc36dc22532f8f0d52c37694f5d46332c05987522" gracePeriod=30 Jul 07 17:48:32.136689 ip-10-0-129-39 kubenswrapper[2575]: I0707 17:48:32.136631 2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/authorino-5d4b67c446-jqsgz" podStartSLOduration=1.701593279 podStartE2EDuration="2.136600648s" totalImagesPullingTime="435.007369ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-07 17:48:30 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 17:48:32.134993706 +0000 UTC m=+472.640800372" watchObservedRunningTime="2026-07-07 17:48:32.136600648 +0000 UTC m=+472.642407315" Jul 07 17:48:32.334991 ip-10-0-129-39 kubenswrapper[2575]: I0707 17:48:32.334967 2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-5d4b67c446-jqsgz" Jul 07 17:48:32.460403 ip-10-0-129-39 kubenswrapper[2575]: I0707 17:48:32.460324 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-9qnqs\" (UniqueName: \"kubernetes.io/projected/c1f603c0-4888-4489-859c-b973ca7d5406-kube-api-access-9qnqs\") pod \"c1f603c0-4888-4489-859c-b973ca7d5406\" (UID: \"c1f603c0-4888-4489-859c-b973ca7d5406\") " Jul 07 17:48:32.462301 ip-10-0-129-39 kubenswrapper[2575]: I0707 17:48:32.462267 2575 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c1f603c0-4888-4489-859c-b973ca7d5406-kube-api-access-9qnqs" (OuterVolumeSpecName: "kube-api-access-9qnqs") pod "c1f603c0-4888-4489-859c-b973ca7d5406" (UID: "c1f603c0-4888-4489-859c-b973ca7d5406"). InnerVolumeSpecName "kube-api-access-9qnqs". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 07 17:48:32.561282 ip-10-0-129-39 kubenswrapper[2575]: I0707 17:48:32.561255 2575 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9qnqs\" (UniqueName: \"kubernetes.io/projected/c1f603c0-4888-4489-859c-b973ca7d5406-kube-api-access-9qnqs\") on node \"ip-10-0-129-39.ec2.internal\" DevicePath \"\"" Jul 07 17:48:33.099275 ip-10-0-129-39 kubenswrapper[2575]: I0707 17:48:33.099233 2575 generic.go:356] "Generic (PLEG): container finished" podID="c1f603c0-4888-4489-859c-b973ca7d5406" containerID="20d2cce85f64949c74d009bcc36dc22532f8f0d52c37694f5d46332c05987522" exitCode=0 Jul 07 17:48:33.099708 ip-10-0-129-39 kubenswrapper[2575]: I0707 17:48:33.099291 2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-5d4b67c446-jqsgz" Jul 07 17:48:33.099708 ip-10-0-129-39 kubenswrapper[2575]: I0707 17:48:33.099316 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-5d4b67c446-jqsgz" event={"ID":"c1f603c0-4888-4489-859c-b973ca7d5406","Type":"ContainerDied","Data":"20d2cce85f64949c74d009bcc36dc22532f8f0d52c37694f5d46332c05987522"} Jul 07 17:48:33.099708 ip-10-0-129-39 kubenswrapper[2575]: I0707 17:48:33.099350 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-5d4b67c446-jqsgz" event={"ID":"c1f603c0-4888-4489-859c-b973ca7d5406","Type":"ContainerDied","Data":"6e4f877aa67071955a92b1be7fec5b6797612f7459e470326fc5f302226c642f"} Jul 07 17:48:33.099708 ip-10-0-129-39 kubenswrapper[2575]: I0707 17:48:33.099369 2575 scope.go:117] "RemoveContainer" containerID="20d2cce85f64949c74d009bcc36dc22532f8f0d52c37694f5d46332c05987522" Jul 07 17:48:33.108376 ip-10-0-129-39 kubenswrapper[2575]: I0707 17:48:33.108356 2575 scope.go:117] "RemoveContainer" containerID="20d2cce85f64949c74d009bcc36dc22532f8f0d52c37694f5d46332c05987522" Jul 07 17:48:33.108660 ip-10-0-129-39 kubenswrapper[2575]: E0707 17:48:33.108641 2575 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"20d2cce85f64949c74d009bcc36dc22532f8f0d52c37694f5d46332c05987522\": container with ID starting with 20d2cce85f64949c74d009bcc36dc22532f8f0d52c37694f5d46332c05987522 not found: ID does not exist" containerID="20d2cce85f64949c74d009bcc36dc22532f8f0d52c37694f5d46332c05987522" Jul 07 17:48:33.108730 ip-10-0-129-39 kubenswrapper[2575]: I0707 17:48:33.108667 2575 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"20d2cce85f64949c74d009bcc36dc22532f8f0d52c37694f5d46332c05987522"} err="failed to get container status \"20d2cce85f64949c74d009bcc36dc22532f8f0d52c37694f5d46332c05987522\": rpc error: code = NotFound desc = could not find container \"20d2cce85f64949c74d009bcc36dc22532f8f0d52c37694f5d46332c05987522\": container with ID starting with 20d2cce85f64949c74d009bcc36dc22532f8f0d52c37694f5d46332c05987522 not found: ID does not exist" Jul 07 17:48:33.127740 ip-10-0-129-39 kubenswrapper[2575]: I0707 17:48:33.127720 2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-5d4b67c446-jqsgz"] Jul 07 17:48:33.131303 ip-10-0-129-39 kubenswrapper[2575]: I0707 17:48:33.131280 2575 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-5d4b67c446-jqsgz"] Jul 07 17:48:34.073460 ip-10-0-129-39 kubenswrapper[2575]: I0707 17:48:34.073427 2575 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="c1f603c0-4888-4489-859c-b973ca7d5406" path="/var/lib/kubelet/pods/c1f603c0-4888-4489-859c-b973ca7d5406/volumes" Jul 07 17:49:16.076964 ip-10-0-129-39 kubenswrapper[2575]: I0707 17:49:16.076930 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-m825v"] Jul 07 17:49:23.323035 ip-10-0-129-39 kubenswrapper[2575]: I0707 17:49:23.323008 2575 generic.go:356] "Generic (PLEG): container finished" podID="45428b46-dfbb-4c68-8a6c-ceffc91e8506" containerID="4563a5f91fc589e8d82e42909c180b33a4a0784314e301ded5b1415186c2b18e" exitCode=0 Jul 07 17:49:23.323451 ip-10-0-129-39 kubenswrapper[2575]: I0707 17:49:23.323050 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-q6xht" event={"ID":"45428b46-dfbb-4c68-8a6c-ceffc91e8506","Type":"ContainerDied","Data":"4563a5f91fc589e8d82e42909c180b33a4a0784314e301ded5b1415186c2b18e"} Jul 07 17:49:23.323539 ip-10-0-129-39 kubenswrapper[2575]: I0707 17:49:23.323466 2575 scope.go:117] "RemoveContainer" containerID="4563a5f91fc589e8d82e42909c180b33a4a0784314e301ded5b1415186c2b18e" Jul 07 17:49:23.324449 ip-10-0-129-39 kubenswrapper[2575]: I0707 17:49:23.324420 2575 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"} Jul 07 17:49:23.324566 ip-10-0-129-39 kubenswrapper[2575]: I0707 17:49:23.324519 2575 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"} Jul 07 17:49:23.324638 ip-10-0-129-39 kubenswrapper[2575]: I0707 17:49:23.324568 2575 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"} Jul 07 17:49:24.332053 ip-10-0-129-39 kubenswrapper[2575]: I0707 17:49:24.332009 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-q6xht" event={"ID":"45428b46-dfbb-4c68-8a6c-ceffc91e8506","Type":"ContainerStarted","Data":"9a4b264ba23ffa35b40a1d9945aef17ab5234b64f6532dcf2f14980eb8c98cf1"} Jul 07 17:49:24.954365 ip-10-0-129-39 kubenswrapper[2575]: I0707 17:49:24.954327 2575 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-q6xht" Jul 07 17:49:24.955417 ip-10-0-129-39 kubenswrapper[2575]: I0707 17:49:24.955394 2575 patch_prober.go:28] interesting pod/maas-default-gateway-openshift-default-56654dcf7-q6xht container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.34:15021/healthz/ready\": dial tcp 10.133.0.34:15021: connect: connection refused" start-of-body= Jul 07 17:49:24.955479 ip-10-0-129-39 kubenswrapper[2575]: I0707 17:49:24.955454 2575 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-q6xht" podUID="45428b46-dfbb-4c68-8a6c-ceffc91e8506" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.34:15021/healthz/ready\": dial tcp 10.133.0.34:15021: connect: connection refused" Jul 07 17:49:25.911817 ip-10-0-129-39 kubenswrapper[2575]: I0707 17:49:25.911784 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-q6xht" Jul 07 17:49:25.955325 ip-10-0-129-39 kubenswrapper[2575]: I0707 17:49:25.955296 2575 patch_prober.go:28] interesting pod/maas-default-gateway-openshift-default-56654dcf7-q6xht container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.34:15021/healthz/ready\": dial tcp 10.133.0.34:15021: connect: connection refused" start-of-body= Jul 07 17:49:25.955478 ip-10-0-129-39 kubenswrapper[2575]: I0707 17:49:25.955346 2575 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-q6xht" podUID="45428b46-dfbb-4c68-8a6c-ceffc91e8506" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.34:15021/healthz/ready\": dial tcp 10.133.0.34:15021: connect: connection refused" Jul 07 17:49:26.954570 ip-10-0-129-39 kubenswrapper[2575]: I0707 17:49:26.954532 2575 patch_prober.go:28] interesting pod/maas-default-gateway-openshift-default-56654dcf7-q6xht container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.34:15021/healthz/ready\": dial tcp 10.133.0.34:15021: connect: connection refused" start-of-body= Jul 07 17:49:26.954956 ip-10-0-129-39 kubenswrapper[2575]: I0707 17:49:26.954589 2575 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-q6xht" podUID="45428b46-dfbb-4c68-8a6c-ceffc91e8506" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.34:15021/healthz/ready\": dial tcp 10.133.0.34:15021: connect: connection refused" Jul 07 17:49:27.954853 ip-10-0-129-39 kubenswrapper[2575]: I0707 17:49:27.954822 2575 patch_prober.go:28] interesting pod/maas-default-gateway-openshift-default-56654dcf7-q6xht container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.34:15021/healthz/ready\": dial tcp 10.133.0.34:15021: connect: connection refused" start-of-body= Jul 07 17:49:27.955231 ip-10-0-129-39 kubenswrapper[2575]: I0707 17:49:27.954891 2575 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-q6xht" podUID="45428b46-dfbb-4c68-8a6c-ceffc91e8506" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.34:15021/healthz/ready\": dial tcp 10.133.0.34:15021: connect: connection refused" Jul 07 17:49:28.958254 ip-10-0-129-39 kubenswrapper[2575]: I0707 17:49:28.958223 2575 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-q6xht" Jul 07 17:49:28.959179 ip-10-0-129-39 kubenswrapper[2575]: I0707 17:49:28.959160 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-q6xht" Jul 07 17:49:46.784419 ip-10-0-129-39 kubenswrapper[2575]: I0707 17:49:46.784340 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-m825v"] Jul 07 17:50:01.480215 ip-10-0-129-39 kubenswrapper[2575]: I0707 17:50:01.480180 2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/e2e-distinct-simulated-kserve-59975d69f9-npxxz"] Jul 07 17:50:01.480694 ip-10-0-129-39 kubenswrapper[2575]: I0707 17:50:01.480528 2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c1f603c0-4888-4489-859c-b973ca7d5406" containerName="authorino" Jul 07 17:50:01.480694 ip-10-0-129-39 kubenswrapper[2575]: I0707 17:50:01.480540 2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="c1f603c0-4888-4489-859c-b973ca7d5406" containerName="authorino" Jul 07 17:50:01.480694 ip-10-0-129-39 kubenswrapper[2575]: I0707 17:50:01.480590 2575 memory_manager.go:368] "RemoveStaleState removing state" podUID="c1f603c0-4888-4489-859c-b973ca7d5406" containerName="authorino" Jul 07 17:50:01.483832 ip-10-0-129-39 kubenswrapper[2575]: I0707 17:50:01.483806 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-npxxz" Jul 07 17:50:01.487357 ip-10-0-129-39 kubenswrapper[2575]: I0707 17:50:01.487332 2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"openshift-service-ca.crt\"" Jul 07 17:50:01.487493 ip-10-0-129-39 kubenswrapper[2575]: I0707 17:50:01.487332 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"default-dockercfg-rgwkq\"" Jul 07 17:50:01.487493 ip-10-0-129-39 kubenswrapper[2575]: I0707 17:50:01.487409 2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"kube-root-ca.crt\"" Jul 07 17:50:01.487493 ip-10-0-129-39 kubenswrapper[2575]: I0707 17:50:01.487394 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-distinct-simulated-kserve-self-signed-certs\"" Jul 07 17:50:01.493452 ip-10-0-129-39 kubenswrapper[2575]: I0707 17:50:01.493431 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-distinct-simulated-kserve-59975d69f9-npxxz"] Jul 07 17:50:01.548605 ip-10-0-129-39 kubenswrapper[2575]: I0707 17:50:01.548573 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/570486ae-48e8-470b-bdc2-075cebe69e8e-tmp-dir\") pod \"e2e-distinct-simulated-kserve-59975d69f9-npxxz\" (UID: \"570486ae-48e8-470b-bdc2-075cebe69e8e\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-npxxz" Jul 07 17:50:01.548766 ip-10-0-129-39 kubenswrapper[2575]: I0707 17:50:01.548613 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/570486ae-48e8-470b-bdc2-075cebe69e8e-model-cache\") pod \"e2e-distinct-simulated-kserve-59975d69f9-npxxz\" (UID: \"570486ae-48e8-470b-bdc2-075cebe69e8e\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-npxxz" Jul 07 17:50:01.548766 ip-10-0-129-39 kubenswrapper[2575]: I0707 17:50:01.548675 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/570486ae-48e8-470b-bdc2-075cebe69e8e-home\") pod \"e2e-distinct-simulated-kserve-59975d69f9-npxxz\" (UID: \"570486ae-48e8-470b-bdc2-075cebe69e8e\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-npxxz" Jul 07 17:50:01.548766 ip-10-0-129-39 kubenswrapper[2575]: I0707 17:50:01.548699 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-whfsb\" (UniqueName: \"kubernetes.io/projected/570486ae-48e8-470b-bdc2-075cebe69e8e-kube-api-access-whfsb\") pod \"e2e-distinct-simulated-kserve-59975d69f9-npxxz\" (UID: \"570486ae-48e8-470b-bdc2-075cebe69e8e\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-npxxz" Jul 07 17:50:01.548882 ip-10-0-129-39 kubenswrapper[2575]: I0707 17:50:01.548777 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/570486ae-48e8-470b-bdc2-075cebe69e8e-dshm\") pod \"e2e-distinct-simulated-kserve-59975d69f9-npxxz\" (UID: \"570486ae-48e8-470b-bdc2-075cebe69e8e\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-npxxz" Jul 07 17:50:01.548882 ip-10-0-129-39 kubenswrapper[2575]: I0707 17:50:01.548809 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/570486ae-48e8-470b-bdc2-075cebe69e8e-tls-certs\") pod \"e2e-distinct-simulated-kserve-59975d69f9-npxxz\" (UID: \"570486ae-48e8-470b-bdc2-075cebe69e8e\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-npxxz" Jul 07 17:50:01.650004 ip-10-0-129-39 kubenswrapper[2575]: I0707 17:50:01.649970 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/570486ae-48e8-470b-bdc2-075cebe69e8e-dshm\") pod \"e2e-distinct-simulated-kserve-59975d69f9-npxxz\" (UID: \"570486ae-48e8-470b-bdc2-075cebe69e8e\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-npxxz" Jul 07 17:50:01.650154 ip-10-0-129-39 kubenswrapper[2575]: I0707 17:50:01.650010 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/570486ae-48e8-470b-bdc2-075cebe69e8e-tls-certs\") pod \"e2e-distinct-simulated-kserve-59975d69f9-npxxz\" (UID: \"570486ae-48e8-470b-bdc2-075cebe69e8e\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-npxxz" Jul 07 17:50:01.650154 ip-10-0-129-39 kubenswrapper[2575]: I0707 17:50:01.650051 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/570486ae-48e8-470b-bdc2-075cebe69e8e-tmp-dir\") pod \"e2e-distinct-simulated-kserve-59975d69f9-npxxz\" (UID: \"570486ae-48e8-470b-bdc2-075cebe69e8e\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-npxxz" Jul 07 17:50:01.650154 ip-10-0-129-39 kubenswrapper[2575]: I0707 17:50:01.650087 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/570486ae-48e8-470b-bdc2-075cebe69e8e-model-cache\") pod \"e2e-distinct-simulated-kserve-59975d69f9-npxxz\" (UID: \"570486ae-48e8-470b-bdc2-075cebe69e8e\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-npxxz" Jul 07 17:50:01.650307 ip-10-0-129-39 kubenswrapper[2575]: I0707 17:50:01.650157 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/570486ae-48e8-470b-bdc2-075cebe69e8e-home\") pod \"e2e-distinct-simulated-kserve-59975d69f9-npxxz\" (UID: \"570486ae-48e8-470b-bdc2-075cebe69e8e\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-npxxz" Jul 07 17:50:01.650307 ip-10-0-129-39 kubenswrapper[2575]: I0707 17:50:01.650203 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-whfsb\" (UniqueName: \"kubernetes.io/projected/570486ae-48e8-470b-bdc2-075cebe69e8e-kube-api-access-whfsb\") pod \"e2e-distinct-simulated-kserve-59975d69f9-npxxz\" (UID: \"570486ae-48e8-470b-bdc2-075cebe69e8e\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-npxxz" Jul 07 17:50:01.650489 ip-10-0-129-39 kubenswrapper[2575]: I0707 17:50:01.650464 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/570486ae-48e8-470b-bdc2-075cebe69e8e-model-cache\") pod \"e2e-distinct-simulated-kserve-59975d69f9-npxxz\" (UID: \"570486ae-48e8-470b-bdc2-075cebe69e8e\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-npxxz" Jul 07 17:50:01.650636 ip-10-0-129-39 kubenswrapper[2575]: I0707 17:50:01.650583 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/570486ae-48e8-470b-bdc2-075cebe69e8e-home\") pod \"e2e-distinct-simulated-kserve-59975d69f9-npxxz\" (UID: \"570486ae-48e8-470b-bdc2-075cebe69e8e\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-npxxz" Jul 07 17:50:01.650636 ip-10-0-129-39 kubenswrapper[2575]: I0707 17:50:01.650623 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/570486ae-48e8-470b-bdc2-075cebe69e8e-tmp-dir\") pod \"e2e-distinct-simulated-kserve-59975d69f9-npxxz\" (UID: \"570486ae-48e8-470b-bdc2-075cebe69e8e\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-npxxz" Jul 07 17:50:01.652289 ip-10-0-129-39 kubenswrapper[2575]: I0707 17:50:01.652260 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/570486ae-48e8-470b-bdc2-075cebe69e8e-dshm\") pod \"e2e-distinct-simulated-kserve-59975d69f9-npxxz\" (UID: \"570486ae-48e8-470b-bdc2-075cebe69e8e\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-npxxz" Jul 07 17:50:01.652540 ip-10-0-129-39 kubenswrapper[2575]: I0707 17:50:01.652495 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/570486ae-48e8-470b-bdc2-075cebe69e8e-tls-certs\") pod \"e2e-distinct-simulated-kserve-59975d69f9-npxxz\" (UID: \"570486ae-48e8-470b-bdc2-075cebe69e8e\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-npxxz" Jul 07 17:50:01.663585 ip-10-0-129-39 kubenswrapper[2575]: I0707 17:50:01.663561 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-whfsb\" (UniqueName: \"kubernetes.io/projected/570486ae-48e8-470b-bdc2-075cebe69e8e-kube-api-access-whfsb\") pod \"e2e-distinct-simulated-kserve-59975d69f9-npxxz\" (UID: \"570486ae-48e8-470b-bdc2-075cebe69e8e\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-npxxz" Jul 07 17:50:01.795283 ip-10-0-129-39 kubenswrapper[2575]: I0707 17:50:01.795195 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-npxxz" Jul 07 17:50:01.936132 ip-10-0-129-39 kubenswrapper[2575]: I0707 17:50:01.936111 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-distinct-simulated-kserve-59975d69f9-npxxz"] Jul 07 17:50:01.938169 ip-10-0-129-39 kubenswrapper[2575]: W0707 17:50:01.938139 2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod570486ae_48e8_470b_bdc2_075cebe69e8e.slice/crio-2caa3ff3d90cd822d1973149ad99d1f2eee075f71a5f97d452b6969121dd4d50 WatchSource:0}: Error finding container 2caa3ff3d90cd822d1973149ad99d1f2eee075f71a5f97d452b6969121dd4d50: Status 404 returned error can't find the container with id 2caa3ff3d90cd822d1973149ad99d1f2eee075f71a5f97d452b6969121dd4d50 Jul 07 17:50:02.292636 ip-10-0-129-39 kubenswrapper[2575]: I0707 17:50:02.292587 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-m825v"] Jul 07 17:50:02.507175 ip-10-0-129-39 kubenswrapper[2575]: I0707 17:50:02.507122 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-npxxz" event={"ID":"570486ae-48e8-470b-bdc2-075cebe69e8e","Type":"ContainerStarted","Data":"2caa3ff3d90cd822d1973149ad99d1f2eee075f71a5f97d452b6969121dd4d50"} Jul 07 17:50:04.518623 ip-10-0-129-39 kubenswrapper[2575]: I0707 17:50:04.518587 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-npxxz" event={"ID":"570486ae-48e8-470b-bdc2-075cebe69e8e","Type":"ContainerStarted","Data":"cbfe4e0e37eb93b3da9abea7077deadcb282044c4daf02f96fa425323390901c"} Jul 07 17:50:04.540002 ip-10-0-129-39 kubenswrapper[2575]: I0707 17:50:04.539954 2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-npxxz" podStartSLOduration=1.847929889 podStartE2EDuration="3.539939539s" totalImagesPullingTime="1.69200965s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-07 17:50:01 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 17:50:04.537890355 +0000 UTC m=+565.043697020" watchObservedRunningTime="2026-07-07 17:50:04.539939539 +0000 UTC m=+565.045746183" Jul 07 17:50:05.190366 ip-10-0-129-39 kubenswrapper[2575]: I0707 17:50:05.190337 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-m825v"] Jul 07 17:50:11.795582 ip-10-0-129-39 kubenswrapper[2575]: I0707 17:50:11.795549 2575 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-npxxz" Jul 07 17:50:11.795977 ip-10-0-129-39 kubenswrapper[2575]: I0707 17:50:11.795598 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-npxxz" Jul 07 17:50:11.808253 ip-10-0-129-39 kubenswrapper[2575]: I0707 17:50:11.808226 2575 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-npxxz" Jul 07 17:50:12.565711 ip-10-0-129-39 kubenswrapper[2575]: I0707 17:50:12.565681 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-npxxz" Jul 07 17:50:15.984938 ip-10-0-129-39 kubenswrapper[2575]: I0707 17:50:15.984901 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-m825v"] Jul 07 17:50:34.481667 ip-10-0-129-39 kubenswrapper[2575]: I0707 17:50:34.481629 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-m825v"] Jul 07 17:50:45.776166 ip-10-0-129-39 kubenswrapper[2575]: I0707 17:50:45.776136 2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-q49ch"] Jul 07 17:50:45.779999 ip-10-0-129-39 kubenswrapper[2575]: I0707 17:50:45.779979 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-q49ch" Jul 07 17:50:45.782745 ip-10-0-129-39 kubenswrapper[2575]: I0707 17:50:45.782724 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"premium-simulated-simulated-premium-kserve-self-signed-certs\"" Jul 07 17:50:45.791549 ip-10-0-129-39 kubenswrapper[2575]: I0707 17:50:45.791523 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-q49ch"] Jul 07 17:50:45.806880 ip-10-0-129-39 kubenswrapper[2575]: I0707 17:50:45.806854 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lnjwp\" (UniqueName: \"kubernetes.io/projected/a5ebeb37-0413-4f97-9115-7cb0708d1641-kube-api-access-lnjwp\") pod \"premium-simulated-simulated-premium-kserve-5fc6fb456f-q49ch\" (UID: \"a5ebeb37-0413-4f97-9115-7cb0708d1641\") " pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-q49ch" Jul 07 17:50:45.807002 ip-10-0-129-39 kubenswrapper[2575]: I0707 17:50:45.806899 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a5ebeb37-0413-4f97-9115-7cb0708d1641-model-cache\") pod \"premium-simulated-simulated-premium-kserve-5fc6fb456f-q49ch\" (UID: \"a5ebeb37-0413-4f97-9115-7cb0708d1641\") " pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-q49ch" Jul 07 17:50:45.807002 ip-10-0-129-39 kubenswrapper[2575]: I0707 17:50:45.806928 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a5ebeb37-0413-4f97-9115-7cb0708d1641-tmp-dir\") pod \"premium-simulated-simulated-premium-kserve-5fc6fb456f-q49ch\" (UID: \"a5ebeb37-0413-4f97-9115-7cb0708d1641\") " pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-q49ch" Jul 07 17:50:45.807002 ip-10-0-129-39 kubenswrapper[2575]: I0707 17:50:45.806953 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a5ebeb37-0413-4f97-9115-7cb0708d1641-home\") pod \"premium-simulated-simulated-premium-kserve-5fc6fb456f-q49ch\" (UID: \"a5ebeb37-0413-4f97-9115-7cb0708d1641\") " pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-q49ch" Jul 07 17:50:45.807002 ip-10-0-129-39 kubenswrapper[2575]: I0707 17:50:45.806972 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a5ebeb37-0413-4f97-9115-7cb0708d1641-dshm\") pod \"premium-simulated-simulated-premium-kserve-5fc6fb456f-q49ch\" (UID: \"a5ebeb37-0413-4f97-9115-7cb0708d1641\") " pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-q49ch" Jul 07 17:50:45.807208 ip-10-0-129-39 kubenswrapper[2575]: I0707 17:50:45.807015 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a5ebeb37-0413-4f97-9115-7cb0708d1641-tls-certs\") pod \"premium-simulated-simulated-premium-kserve-5fc6fb456f-q49ch\" (UID: \"a5ebeb37-0413-4f97-9115-7cb0708d1641\") " pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-q49ch" Jul 07 17:50:45.907812 ip-10-0-129-39 kubenswrapper[2575]: I0707 17:50:45.907720 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a5ebeb37-0413-4f97-9115-7cb0708d1641-model-cache\") pod \"premium-simulated-simulated-premium-kserve-5fc6fb456f-q49ch\" (UID: \"a5ebeb37-0413-4f97-9115-7cb0708d1641\") " pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-q49ch" Jul 07 17:50:45.907954 ip-10-0-129-39 kubenswrapper[2575]: I0707 17:50:45.907853 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a5ebeb37-0413-4f97-9115-7cb0708d1641-tmp-dir\") pod \"premium-simulated-simulated-premium-kserve-5fc6fb456f-q49ch\" (UID: \"a5ebeb37-0413-4f97-9115-7cb0708d1641\") " pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-q49ch" Jul 07 17:50:45.907954 ip-10-0-129-39 kubenswrapper[2575]: I0707 17:50:45.907878 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a5ebeb37-0413-4f97-9115-7cb0708d1641-home\") pod \"premium-simulated-simulated-premium-kserve-5fc6fb456f-q49ch\" (UID: \"a5ebeb37-0413-4f97-9115-7cb0708d1641\") " pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-q49ch" Jul 07 17:50:45.907954 ip-10-0-129-39 kubenswrapper[2575]: I0707 17:50:45.907902 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a5ebeb37-0413-4f97-9115-7cb0708d1641-dshm\") pod \"premium-simulated-simulated-premium-kserve-5fc6fb456f-q49ch\" (UID: \"a5ebeb37-0413-4f97-9115-7cb0708d1641\") " pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-q49ch" Jul 07 17:50:45.908108 ip-10-0-129-39 kubenswrapper[2575]: I0707 17:50:45.907955 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a5ebeb37-0413-4f97-9115-7cb0708d1641-tls-certs\") pod \"premium-simulated-simulated-premium-kserve-5fc6fb456f-q49ch\" (UID: \"a5ebeb37-0413-4f97-9115-7cb0708d1641\") " pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-q49ch" Jul 07 17:50:45.908108 ip-10-0-129-39 kubenswrapper[2575]: I0707 17:50:45.907985 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lnjwp\" (UniqueName: \"kubernetes.io/projected/a5ebeb37-0413-4f97-9115-7cb0708d1641-kube-api-access-lnjwp\") pod \"premium-simulated-simulated-premium-kserve-5fc6fb456f-q49ch\" (UID: \"a5ebeb37-0413-4f97-9115-7cb0708d1641\") " pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-q49ch" Jul 07 17:50:45.908108 ip-10-0-129-39 kubenswrapper[2575]: I0707 17:50:45.908089 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a5ebeb37-0413-4f97-9115-7cb0708d1641-model-cache\") pod \"premium-simulated-simulated-premium-kserve-5fc6fb456f-q49ch\" (UID: \"a5ebeb37-0413-4f97-9115-7cb0708d1641\") " pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-q49ch" Jul 07 17:50:45.908256 ip-10-0-129-39 kubenswrapper[2575]: I0707 17:50:45.908157 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a5ebeb37-0413-4f97-9115-7cb0708d1641-tmp-dir\") pod \"premium-simulated-simulated-premium-kserve-5fc6fb456f-q49ch\" (UID: \"a5ebeb37-0413-4f97-9115-7cb0708d1641\") " pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-q49ch" Jul 07 17:50:45.908256 ip-10-0-129-39 kubenswrapper[2575]: I0707 17:50:45.908216 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a5ebeb37-0413-4f97-9115-7cb0708d1641-home\") pod \"premium-simulated-simulated-premium-kserve-5fc6fb456f-q49ch\" (UID: \"a5ebeb37-0413-4f97-9115-7cb0708d1641\") " pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-q49ch" Jul 07 17:50:45.910129 ip-10-0-129-39 kubenswrapper[2575]: I0707 17:50:45.910101 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a5ebeb37-0413-4f97-9115-7cb0708d1641-dshm\") pod \"premium-simulated-simulated-premium-kserve-5fc6fb456f-q49ch\" (UID: \"a5ebeb37-0413-4f97-9115-7cb0708d1641\") " pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-q49ch" Jul 07 17:50:45.910396 ip-10-0-129-39 kubenswrapper[2575]: I0707 17:50:45.910376 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a5ebeb37-0413-4f97-9115-7cb0708d1641-tls-certs\") pod \"premium-simulated-simulated-premium-kserve-5fc6fb456f-q49ch\" (UID: \"a5ebeb37-0413-4f97-9115-7cb0708d1641\") " pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-q49ch" Jul 07 17:50:45.917012 ip-10-0-129-39 kubenswrapper[2575]: I0707 17:50:45.916987 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lnjwp\" (UniqueName: \"kubernetes.io/projected/a5ebeb37-0413-4f97-9115-7cb0708d1641-kube-api-access-lnjwp\") pod \"premium-simulated-simulated-premium-kserve-5fc6fb456f-q49ch\" (UID: \"a5ebeb37-0413-4f97-9115-7cb0708d1641\") " pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-q49ch" Jul 07 17:50:46.089984 ip-10-0-129-39 kubenswrapper[2575]: I0707 17:50:46.089953 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-q49ch" Jul 07 17:50:46.221219 ip-10-0-129-39 kubenswrapper[2575]: I0707 17:50:46.221192 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-q49ch"] Jul 07 17:50:46.223648 ip-10-0-129-39 kubenswrapper[2575]: W0707 17:50:46.223623 2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-poda5ebeb37_0413_4f97_9115_7cb0708d1641.slice/crio-e916a3dfa803f5f819224b61177aad556ebcc5daa5d0b8de6597aeb2797a8d98 WatchSource:0}: Error finding container e916a3dfa803f5f819224b61177aad556ebcc5daa5d0b8de6597aeb2797a8d98: Status 404 returned error can't find the container with id e916a3dfa803f5f819224b61177aad556ebcc5daa5d0b8de6597aeb2797a8d98 Jul 07 17:50:46.225526 ip-10-0-129-39 kubenswrapper[2575]: I0707 17:50:46.225487 2575 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Jul 07 17:50:46.714330 ip-10-0-129-39 kubenswrapper[2575]: I0707 17:50:46.714247 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-q49ch" event={"ID":"a5ebeb37-0413-4f97-9115-7cb0708d1641","Type":"ContainerStarted","Data":"c5bf0524ca9a4c94143a55cc3c662f0fef00dacb3caa0c02b29fca4660ab03ca"} Jul 07 17:50:46.714330 ip-10-0-129-39 kubenswrapper[2575]: I0707 17:50:46.714284 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-q49ch" event={"ID":"a5ebeb37-0413-4f97-9115-7cb0708d1641","Type":"ContainerStarted","Data":"e916a3dfa803f5f819224b61177aad556ebcc5daa5d0b8de6597aeb2797a8d98"} Jul 07 17:50:46.733394 ip-10-0-129-39 kubenswrapper[2575]: I0707 17:50:46.733342 2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-q49ch" podStartSLOduration=1.5501413670000002 podStartE2EDuration="1.733329719s" totalImagesPullingTime="183.188352ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-07 17:50:45 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 17:50:46.732352936 +0000 UTC m=+607.238159606" watchObservedRunningTime="2026-07-07 17:50:46.733329719 +0000 UTC m=+607.239136384" Jul 07 17:50:48.388140 ip-10-0-129-39 kubenswrapper[2575]: I0707 17:50:48.388106 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-m825v"] Jul 07 17:50:56.090273 ip-10-0-129-39 kubenswrapper[2575]: I0707 17:50:56.090239 2575 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-q49ch" Jul 07 17:50:56.090765 ip-10-0-129-39 kubenswrapper[2575]: I0707 17:50:56.090283 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-q49ch" Jul 07 17:50:56.102596 ip-10-0-129-39 kubenswrapper[2575]: I0707 17:50:56.102575 2575 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-q49ch" Jul 07 17:50:56.780254 ip-10-0-129-39 kubenswrapper[2575]: I0707 17:50:56.780219 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-q49ch" Jul 07 17:52:46.487678 ip-10-0-129-39 kubenswrapper[2575]: I0707 17:52:46.487601 2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-65dc978fbf-6sh24"] Jul 07 17:52:46.490985 ip-10-0-129-39 kubenswrapper[2575]: I0707 17:52:46.490968 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-65dc978fbf-6sh24" Jul 07 17:52:46.494354 ip-10-0-129-39 kubenswrapper[2575]: I0707 17:52:46.494332 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-server-cert\"" Jul 07 17:52:46.494354 ip-10-0-129-39 kubenswrapper[2575]: I0707 17:52:46.494349 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-authorino-dockercfg-vx6wg\"" Jul 07 17:52:46.508297 ip-10-0-129-39 kubenswrapper[2575]: I0707 17:52:46.508279 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-65dc978fbf-6sh24"] Jul 07 17:52:46.573964 ip-10-0-129-39 kubenswrapper[2575]: I0707 17:52:46.573932 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fm9fh\" (UniqueName: \"kubernetes.io/projected/db6a34f4-5292-455c-a3ad-b0d1d72dabee-kube-api-access-fm9fh\") pod \"authorino-65dc978fbf-6sh24\" (UID: \"db6a34f4-5292-455c-a3ad-b0d1d72dabee\") " pod="kuadrant-system/authorino-65dc978fbf-6sh24" Jul 07 17:52:46.574102 ip-10-0-129-39 kubenswrapper[2575]: I0707 17:52:46.574059 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/db6a34f4-5292-455c-a3ad-b0d1d72dabee-tls-cert\") pod \"authorino-65dc978fbf-6sh24\" (UID: \"db6a34f4-5292-455c-a3ad-b0d1d72dabee\") " pod="kuadrant-system/authorino-65dc978fbf-6sh24" Jul 07 17:52:46.675374 ip-10-0-129-39 kubenswrapper[2575]: I0707 17:52:46.675337 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/db6a34f4-5292-455c-a3ad-b0d1d72dabee-tls-cert\") pod \"authorino-65dc978fbf-6sh24\" (UID: \"db6a34f4-5292-455c-a3ad-b0d1d72dabee\") " pod="kuadrant-system/authorino-65dc978fbf-6sh24" Jul 07 17:52:46.675538 ip-10-0-129-39 kubenswrapper[2575]: I0707 17:52:46.675381 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fm9fh\" (UniqueName: \"kubernetes.io/projected/db6a34f4-5292-455c-a3ad-b0d1d72dabee-kube-api-access-fm9fh\") pod \"authorino-65dc978fbf-6sh24\" (UID: \"db6a34f4-5292-455c-a3ad-b0d1d72dabee\") " pod="kuadrant-system/authorino-65dc978fbf-6sh24" Jul 07 17:52:46.677902 ip-10-0-129-39 kubenswrapper[2575]: I0707 17:52:46.677877 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/db6a34f4-5292-455c-a3ad-b0d1d72dabee-tls-cert\") pod \"authorino-65dc978fbf-6sh24\" (UID: \"db6a34f4-5292-455c-a3ad-b0d1d72dabee\") " pod="kuadrant-system/authorino-65dc978fbf-6sh24" Jul 07 17:52:46.686069 ip-10-0-129-39 kubenswrapper[2575]: I0707 17:52:46.686048 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fm9fh\" (UniqueName: \"kubernetes.io/projected/db6a34f4-5292-455c-a3ad-b0d1d72dabee-kube-api-access-fm9fh\") pod \"authorino-65dc978fbf-6sh24\" (UID: \"db6a34f4-5292-455c-a3ad-b0d1d72dabee\") " pod="kuadrant-system/authorino-65dc978fbf-6sh24" Jul 07 17:52:46.799908 ip-10-0-129-39 kubenswrapper[2575]: I0707 17:52:46.799835 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-65dc978fbf-6sh24" Jul 07 17:52:46.928110 ip-10-0-129-39 kubenswrapper[2575]: I0707 17:52:46.928081 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-65dc978fbf-6sh24"] Jul 07 17:52:46.929805 ip-10-0-129-39 kubenswrapper[2575]: W0707 17:52:46.929779 2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-poddb6a34f4_5292_455c_a3ad_b0d1d72dabee.slice/crio-6e63f5aab9d05692832cb3eae98c0bc8f8542058fd5753c4a475c9cdb163bf31 WatchSource:0}: Error finding container 6e63f5aab9d05692832cb3eae98c0bc8f8542058fd5753c4a475c9cdb163bf31: Status 404 returned error can't find the container with id 6e63f5aab9d05692832cb3eae98c0bc8f8542058fd5753c4a475c9cdb163bf31 Jul 07 17:52:47.267225 ip-10-0-129-39 kubenswrapper[2575]: I0707 17:52:47.267189 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-65dc978fbf-6sh24" event={"ID":"db6a34f4-5292-455c-a3ad-b0d1d72dabee","Type":"ContainerStarted","Data":"6e63f5aab9d05692832cb3eae98c0bc8f8542058fd5753c4a475c9cdb163bf31"} Jul 07 17:52:48.273013 ip-10-0-129-39 kubenswrapper[2575]: I0707 17:52:48.272966 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-65dc978fbf-6sh24" event={"ID":"db6a34f4-5292-455c-a3ad-b0d1d72dabee","Type":"ContainerStarted","Data":"e75bee3d76073327dbfbf47847f0bda64ac0c65cb85c0bba2542b0701775458a"} Jul 07 17:52:48.293996 ip-10-0-129-39 kubenswrapper[2575]: I0707 17:52:48.293951 2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/authorino-65dc978fbf-6sh24" podStartSLOduration=1.7565648870000001 podStartE2EDuration="2.29393724s" totalImagesPullingTime="537.372353ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-07 17:52:46 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 17:52:48.293027669 +0000 UTC m=+728.798834328" watchObservedRunningTime="2026-07-07 17:52:48.29393724 +0000 UTC m=+728.799743902" Jul 07 17:53:15.591051 ip-10-0-129-39 kubenswrapper[2575]: I0707 17:53:15.591010 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-m825v"] Jul 07 17:53:24.485192 ip-10-0-129-39 kubenswrapper[2575]: I0707 17:53:24.485155 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-m825v"] Jul 07 17:53:36.289941 ip-10-0-129-39 kubenswrapper[2575]: I0707 17:53:36.289906 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-m825v"] Jul 07 17:53:44.487406 ip-10-0-129-39 kubenswrapper[2575]: I0707 17:53:44.487359 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-m825v"] Jul 07 17:53:54.883411 ip-10-0-129-39 kubenswrapper[2575]: I0707 17:53:54.883378 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-m825v"] Jul 07 17:54:22.390386 ip-10-0-129-39 kubenswrapper[2575]: I0707 17:54:22.390347 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-m825v"] Jul 07 17:54:33.392750 ip-10-0-129-39 kubenswrapper[2575]: I0707 17:54:33.392705 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-m825v"] Jul 07 17:55:09.589419 ip-10-0-129-39 kubenswrapper[2575]: I0707 17:55:09.589385 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-m825v"] Jul 07 17:55:19.200299 ip-10-0-129-39 kubenswrapper[2575]: I0707 17:55:19.200265 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-m825v"] Jul 07 17:55:28.224623 ip-10-0-129-39 kubenswrapper[2575]: I0707 17:55:28.224587 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-m825v"] Jul 07 17:55:39.517460 ip-10-0-129-39 kubenswrapper[2575]: I0707 17:55:39.517382 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-m825v"] Jul 07 17:55:48.889358 ip-10-0-129-39 kubenswrapper[2575]: I0707 17:55:48.889322 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-m825v"] Jul 07 17:56:04.393683 ip-10-0-129-39 kubenswrapper[2575]: I0707 17:56:04.393649 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-m825v"] Jul 07 17:56:10.401390 ip-10-0-129-39 kubenswrapper[2575]: I0707 17:56:10.401356 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-m825v"] Jul 07 17:56:20.197842 ip-10-0-129-39 kubenswrapper[2575]: I0707 17:56:20.197810 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-m825v"] Jul 07 17:56:41.793831 ip-10-0-129-39 kubenswrapper[2575]: I0707 17:56:41.793795 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-m825v"] Jul 07 17:56:53.098179 ip-10-0-129-39 kubenswrapper[2575]: I0707 17:56:53.098141 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-m825v"] Jul 07 17:57:17.893288 ip-10-0-129-39 kubenswrapper[2575]: I0707 17:57:17.893190 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-m825v"] Jul 07 17:57:28.493771 ip-10-0-129-39 kubenswrapper[2575]: I0707 17:57:28.493736 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-m825v"] Jul 07 17:57:42.998130 ip-10-0-129-39 kubenswrapper[2575]: I0707 17:57:42.998096 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-m825v"] Jul 07 17:57:53.694791 ip-10-0-129-39 kubenswrapper[2575]: I0707 17:57:53.694763 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-m825v"] Jul 07 17:58:02.801877 ip-10-0-129-39 kubenswrapper[2575]: I0707 17:58:02.801843 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-m825v"] Jul 07 17:58:14.092224 ip-10-0-129-39 kubenswrapper[2575]: I0707 17:58:14.092191 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-m825v"] Jul 07 17:58:22.994524 ip-10-0-129-39 kubenswrapper[2575]: I0707 17:58:22.994477 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-m825v"] Jul 07 17:58:50.876860 ip-10-0-129-39 kubenswrapper[2575]: I0707 17:58:50.876779 2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-86878c769f-x85t4"] Jul 07 17:58:50.877458 ip-10-0-129-39 kubenswrapper[2575]: I0707 17:58:50.876996 2575 kuberuntime_container.go:872] "Killing container with a grace period" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-x85t4" podUID="d270ae63-e153-4532-8963-16397c54c338" containerName="manager" containerID="cri-o://5bf41ae976b1b9de65b3d9543875097f3d625f98f56ff7f7b24b680a80791386" gracePeriod=10 Jul 07 17:58:53.967264 ip-10-0-129-39 kubenswrapper[2575]: I0707 17:58:53.967185 2575 generic.go:356] "Generic (PLEG): container finished" podID="d270ae63-e153-4532-8963-16397c54c338" containerID="5bf41ae976b1b9de65b3d9543875097f3d625f98f56ff7f7b24b680a80791386" exitCode=0 Jul 07 17:58:53.967639 ip-10-0-129-39 kubenswrapper[2575]: I0707 17:58:53.967262 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-x85t4" event={"ID":"d270ae63-e153-4532-8963-16397c54c338","Type":"ContainerDied","Data":"5bf41ae976b1b9de65b3d9543875097f3d625f98f56ff7f7b24b680a80791386"} Jul 07 17:58:54.224877 ip-10-0-129-39 kubenswrapper[2575]: I0707 17:58:54.224826 2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-x85t4" Jul 07 17:58:54.386143 ip-10-0-129-39 kubenswrapper[2575]: I0707 17:58:54.386116 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/d270ae63-e153-4532-8963-16397c54c338-extensions-socket-volume\") pod \"d270ae63-e153-4532-8963-16397c54c338\" (UID: \"d270ae63-e153-4532-8963-16397c54c338\") " Jul 07 17:58:54.386315 ip-10-0-129-39 kubenswrapper[2575]: I0707 17:58:54.386210 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-zm4q9\" (UniqueName: \"kubernetes.io/projected/d270ae63-e153-4532-8963-16397c54c338-kube-api-access-zm4q9\") pod \"d270ae63-e153-4532-8963-16397c54c338\" (UID: \"d270ae63-e153-4532-8963-16397c54c338\") " Jul 07 17:58:54.386480 ip-10-0-129-39 kubenswrapper[2575]: I0707 17:58:54.386457 2575 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d270ae63-e153-4532-8963-16397c54c338-extensions-socket-volume" (OuterVolumeSpecName: "extensions-socket-volume") pod "d270ae63-e153-4532-8963-16397c54c338" (UID: "d270ae63-e153-4532-8963-16397c54c338"). InnerVolumeSpecName "extensions-socket-volume". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 07 17:58:54.388168 ip-10-0-129-39 kubenswrapper[2575]: I0707 17:58:54.388142 2575 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d270ae63-e153-4532-8963-16397c54c338-kube-api-access-zm4q9" (OuterVolumeSpecName: "kube-api-access-zm4q9") pod "d270ae63-e153-4532-8963-16397c54c338" (UID: "d270ae63-e153-4532-8963-16397c54c338"). InnerVolumeSpecName "kube-api-access-zm4q9". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 07 17:58:54.487114 ip-10-0-129-39 kubenswrapper[2575]: I0707 17:58:54.487048 2575 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zm4q9\" (UniqueName: \"kubernetes.io/projected/d270ae63-e153-4532-8963-16397c54c338-kube-api-access-zm4q9\") on node \"ip-10-0-129-39.ec2.internal\" DevicePath \"\"" Jul 07 17:58:54.487114 ip-10-0-129-39 kubenswrapper[2575]: I0707 17:58:54.487073 2575 reconciler_common.go:299] "Volume detached for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/d270ae63-e153-4532-8963-16397c54c338-extensions-socket-volume\") on node \"ip-10-0-129-39.ec2.internal\" DevicePath \"\"" Jul 07 17:58:54.972766 ip-10-0-129-39 kubenswrapper[2575]: I0707 17:58:54.972727 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-x85t4" event={"ID":"d270ae63-e153-4532-8963-16397c54c338","Type":"ContainerDied","Data":"ae236fa1f198a7dc46d6b06bb49d5d9d702f79e619ba099cff3deee15c76180f"} Jul 07 17:58:54.972766 ip-10-0-129-39 kubenswrapper[2575]: I0707 17:58:54.972759 2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-x85t4" Jul 07 17:58:54.973211 ip-10-0-129-39 kubenswrapper[2575]: I0707 17:58:54.972797 2575 scope.go:117] "RemoveContainer" containerID="5bf41ae976b1b9de65b3d9543875097f3d625f98f56ff7f7b24b680a80791386" Jul 07 17:58:54.997114 ip-10-0-129-39 kubenswrapper[2575]: I0707 17:58:54.997091 2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-86878c769f-x85t4"] Jul 07 17:58:55.001951 ip-10-0-129-39 kubenswrapper[2575]: I0707 17:58:55.001930 2575 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-86878c769f-x85t4"] Jul 07 17:58:56.073590 ip-10-0-129-39 kubenswrapper[2575]: I0707 17:58:56.073560 2575 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="d270ae63-e153-4532-8963-16397c54c338" path="/var/lib/kubelet/pods/d270ae63-e153-4532-8963-16397c54c338/volumes" Jul 07 17:59:52.025393 ip-10-0-129-39 kubenswrapper[2575]: I0707 17:59:52.025360 2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-86878c769f-tw877"] Jul 07 17:59:52.025808 ip-10-0-129-39 kubenswrapper[2575]: I0707 17:59:52.025700 2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d270ae63-e153-4532-8963-16397c54c338" containerName="manager" Jul 07 17:59:52.025808 ip-10-0-129-39 kubenswrapper[2575]: I0707 17:59:52.025711 2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="d270ae63-e153-4532-8963-16397c54c338" containerName="manager" Jul 07 17:59:52.025808 ip-10-0-129-39 kubenswrapper[2575]: I0707 17:59:52.025772 2575 memory_manager.go:368] "RemoveStaleState removing state" podUID="d270ae63-e153-4532-8963-16397c54c338" containerName="manager" Jul 07 17:59:52.029761 ip-10-0-129-39 kubenswrapper[2575]: I0707 17:59:52.029746 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-tw877" Jul 07 17:59:52.033344 ip-10-0-129-39 kubenswrapper[2575]: I0707 17:59:52.033323 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"kuadrant-operator-controller-manager-dockercfg-6kr82\"" Jul 07 17:59:52.045441 ip-10-0-129-39 kubenswrapper[2575]: I0707 17:59:52.045421 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-86878c769f-tw877"] Jul 07 17:59:52.139082 ip-10-0-129-39 kubenswrapper[2575]: I0707 17:59:52.139052 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/06c4d2f4-68a1-4e31-b426-2bd937d5433e-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-86878c769f-tw877\" (UID: \"06c4d2f4-68a1-4e31-b426-2bd937d5433e\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-tw877" Jul 07 17:59:52.139225 ip-10-0-129-39 kubenswrapper[2575]: I0707 17:59:52.139086 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-r84h4\" (UniqueName: \"kubernetes.io/projected/06c4d2f4-68a1-4e31-b426-2bd937d5433e-kube-api-access-r84h4\") pod \"kuadrant-operator-controller-manager-86878c769f-tw877\" (UID: \"06c4d2f4-68a1-4e31-b426-2bd937d5433e\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-tw877" Jul 07 17:59:52.240649 ip-10-0-129-39 kubenswrapper[2575]: I0707 17:59:52.240623 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/06c4d2f4-68a1-4e31-b426-2bd937d5433e-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-86878c769f-tw877\" (UID: \"06c4d2f4-68a1-4e31-b426-2bd937d5433e\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-tw877" Jul 07 17:59:52.240808 ip-10-0-129-39 kubenswrapper[2575]: I0707 17:59:52.240659 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r84h4\" (UniqueName: \"kubernetes.io/projected/06c4d2f4-68a1-4e31-b426-2bd937d5433e-kube-api-access-r84h4\") pod \"kuadrant-operator-controller-manager-86878c769f-tw877\" (UID: \"06c4d2f4-68a1-4e31-b426-2bd937d5433e\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-tw877" Jul 07 17:59:52.240985 ip-10-0-129-39 kubenswrapper[2575]: I0707 17:59:52.240964 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/06c4d2f4-68a1-4e31-b426-2bd937d5433e-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-86878c769f-tw877\" (UID: \"06c4d2f4-68a1-4e31-b426-2bd937d5433e\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-tw877" Jul 07 17:59:52.255207 ip-10-0-129-39 kubenswrapper[2575]: I0707 17:59:52.255175 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-r84h4\" (UniqueName: \"kubernetes.io/projected/06c4d2f4-68a1-4e31-b426-2bd937d5433e-kube-api-access-r84h4\") pod \"kuadrant-operator-controller-manager-86878c769f-tw877\" (UID: \"06c4d2f4-68a1-4e31-b426-2bd937d5433e\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-tw877" Jul 07 17:59:52.339371 ip-10-0-129-39 kubenswrapper[2575]: I0707 17:59:52.339343 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-tw877" Jul 07 17:59:52.475789 ip-10-0-129-39 kubenswrapper[2575]: I0707 17:59:52.475751 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-86878c769f-tw877"] Jul 07 17:59:52.479324 ip-10-0-129-39 kubenswrapper[2575]: W0707 17:59:52.479299 2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod06c4d2f4_68a1_4e31_b426_2bd937d5433e.slice/crio-b0a3a4d8e757b64553f9e3cda866287afa90513189591c12a7c45b18121c22c1 WatchSource:0}: Error finding container b0a3a4d8e757b64553f9e3cda866287afa90513189591c12a7c45b18121c22c1: Status 404 returned error can't find the container with id b0a3a4d8e757b64553f9e3cda866287afa90513189591c12a7c45b18121c22c1 Jul 07 17:59:53.233339 ip-10-0-129-39 kubenswrapper[2575]: I0707 17:59:53.233301 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-tw877" event={"ID":"06c4d2f4-68a1-4e31-b426-2bd937d5433e","Type":"ContainerStarted","Data":"958285de5672442b6c29ea34f44909d9515d181195614c610baffc4a60250db9"} Jul 07 17:59:53.233737 ip-10-0-129-39 kubenswrapper[2575]: I0707 17:59:53.233357 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-tw877" event={"ID":"06c4d2f4-68a1-4e31-b426-2bd937d5433e","Type":"ContainerStarted","Data":"b0a3a4d8e757b64553f9e3cda866287afa90513189591c12a7c45b18121c22c1"} Jul 07 17:59:53.233737 ip-10-0-129-39 kubenswrapper[2575]: I0707 17:59:53.233396 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-tw877" Jul 07 17:59:53.258302 ip-10-0-129-39 kubenswrapper[2575]: I0707 17:59:53.258252 2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-tw877" podStartSLOduration=2.258222568 podStartE2EDuration="2.258222568s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-07 17:59:51 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 17:59:53.256994125 +0000 UTC m=+1153.762800791" watchObservedRunningTime="2026-07-07 17:59:53.258222568 +0000 UTC m=+1153.764029234" Jul 07 18:00:04.239998 ip-10-0-129-39 kubenswrapper[2575]: I0707 18:00:04.239971 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-tw877" Jul 07 18:00:12.711951 ip-10-0-129-39 kubenswrapper[2575]: I0707 18:00:12.711914 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-m825v"] Jul 07 18:00:23.594056 ip-10-0-129-39 kubenswrapper[2575]: I0707 18:00:23.594022 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-m825v"] Jul 07 18:00:32.599241 ip-10-0-129-39 kubenswrapper[2575]: I0707 18:00:32.599205 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-m825v"] Jul 07 18:00:44.663245 ip-10-0-129-39 kubenswrapper[2575]: I0707 18:00:44.663209 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-m825v"] Jul 07 18:00:56.842421 ip-10-0-129-39 kubenswrapper[2575]: I0707 18:00:56.842387 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-m825v"] Jul 07 18:01:06.633636 ip-10-0-129-39 kubenswrapper[2575]: I0707 18:01:06.633605 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-m825v"] Jul 07 18:01:14.924092 ip-10-0-129-39 kubenswrapper[2575]: I0707 18:01:14.924054 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-m825v"] Jul 07 18:01:25.823966 ip-10-0-129-39 kubenswrapper[2575]: I0707 18:01:25.823928 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-m825v"] Jul 07 18:01:37.611918 ip-10-0-129-39 kubenswrapper[2575]: I0707 18:01:37.611833 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-m825v"] Jul 07 18:01:46.196937 ip-10-0-129-39 kubenswrapper[2575]: I0707 18:01:46.196903 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-m825v"] Jul 07 18:01:58.491170 ip-10-0-129-39 kubenswrapper[2575]: I0707 18:01:58.491137 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-m825v"] Jul 07 18:02:16.800652 ip-10-0-129-39 kubenswrapper[2575]: I0707 18:02:16.800618 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-m825v"] Jul 07 18:02:25.531136 ip-10-0-129-39 kubenswrapper[2575]: I0707 18:02:25.531092 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-m825v"] Jul 07 18:02:35.020185 ip-10-0-129-39 kubenswrapper[2575]: I0707 18:02:35.020151 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-m825v"] Jul 07 18:02:46.443745 ip-10-0-129-39 kubenswrapper[2575]: I0707 18:02:46.443709 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-m825v"] Jul 07 18:03:04.253012 ip-10-0-129-39 kubenswrapper[2575]: I0707 18:03:04.252931 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-m825v"] Jul 07 18:03:17.885494 ip-10-0-129-39 kubenswrapper[2575]: I0707 18:03:17.885458 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-m825v"] Jul 07 18:03:27.003618 ip-10-0-129-39 kubenswrapper[2575]: I0707 18:03:27.003582 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-m825v"] Jul 07 18:03:41.689857 ip-10-0-129-39 kubenswrapper[2575]: I0707 18:03:41.689818 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-m825v"] Jul 07 18:03:50.496456 ip-10-0-129-39 kubenswrapper[2575]: I0707 18:03:50.496415 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-m825v"] Jul 07 18:04:03.404029 ip-10-0-129-39 kubenswrapper[2575]: I0707 18:04:03.403998 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-m825v"] Jul 07 18:04:12.690981 ip-10-0-129-39 kubenswrapper[2575]: I0707 18:04:12.690944 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-m825v"] Jul 07 18:04:22.701951 ip-10-0-129-39 kubenswrapper[2575]: I0707 18:04:22.701914 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-m825v"] Jul 07 18:04:34.000926 ip-10-0-129-39 kubenswrapper[2575]: I0707 18:04:34.000846 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-m825v"] Jul 07 18:04:46.584815 ip-10-0-129-39 kubenswrapper[2575]: I0707 18:04:46.584777 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-m825v"] Jul 07 18:04:54.187158 ip-10-0-129-39 kubenswrapper[2575]: I0707 18:04:54.187122 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-m825v"] Jul 07 18:05:06.890552 ip-10-0-129-39 kubenswrapper[2575]: I0707 18:05:06.890519 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-m825v"] Jul 07 18:05:18.998398 ip-10-0-129-39 kubenswrapper[2575]: I0707 18:05:18.998367 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-m825v"] Jul 07 18:05:31.790257 ip-10-0-129-39 kubenswrapper[2575]: I0707 18:05:31.790227 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-m825v"] Jul 07 18:05:39.403998 ip-10-0-129-39 kubenswrapper[2575]: I0707 18:05:39.403959 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-m825v"] Jul 07 18:05:49.300712 ip-10-0-129-39 kubenswrapper[2575]: I0707 18:05:49.300672 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-m825v"] Jul 07 18:05:58.305389 ip-10-0-129-39 kubenswrapper[2575]: I0707 18:05:58.305357 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-m825v"] Jul 07 18:06:10.797426 ip-10-0-129-39 kubenswrapper[2575]: I0707 18:06:10.797346 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-m825v"] Jul 07 18:06:19.147577 ip-10-0-129-39 kubenswrapper[2575]: I0707 18:06:19.147529 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-m825v"] Jul 07 18:06:42.116606 ip-10-0-129-39 kubenswrapper[2575]: I0707 18:06:42.116576 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-m825v"] Jul 07 18:06:49.786228 ip-10-0-129-39 kubenswrapper[2575]: I0707 18:06:49.786194 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-m825v"] Jul 07 18:08:15.587320 ip-10-0-129-39 kubenswrapper[2575]: I0707 18:08:15.587289 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-m825v"] Jul 07 18:08:22.889376 ip-10-0-129-39 kubenswrapper[2575]: I0707 18:08:22.889340 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-m825v"] Jul 07 18:08:30.207301 ip-10-0-129-39 kubenswrapper[2575]: I0707 18:08:30.207266 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-m825v"] Jul 07 18:08:37.835049 ip-10-0-129-39 kubenswrapper[2575]: I0707 18:08:37.835007 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-m825v"] Jul 07 18:08:49.578389 ip-10-0-129-39 kubenswrapper[2575]: I0707 18:08:49.578356 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-m825v"] Jul 07 18:09:21.810519 ip-10-0-129-39 kubenswrapper[2575]: I0707 18:09:21.810416 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-m825v"] Jul 07 18:09:36.488253 ip-10-0-129-39 kubenswrapper[2575]: I0707 18:09:36.488220 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-m825v"] Jul 07 18:10:01.189735 ip-10-0-129-39 kubenswrapper[2575]: I0707 18:10:01.189700 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-m825v"] Jul 07 18:10:14.608897 ip-10-0-129-39 kubenswrapper[2575]: I0707 18:10:14.608860 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-m825v"] Jul 07 18:10:54.375517 ip-10-0-129-39 kubenswrapper[2575]: I0707 18:10:54.375470 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-m825v"] Jul 07 18:11:26.087674 ip-10-0-129-39 kubenswrapper[2575]: I0707 18:11:26.087639 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-m825v"] Jul 07 18:11:39.824227 ip-10-0-129-39 kubenswrapper[2575]: I0707 18:11:39.824191 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-m825v"] Jul 07 18:12:27.383021 ip-10-0-129-39 kubenswrapper[2575]: I0707 18:12:27.382942 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-m825v"] Jul 07 18:12:38.590902 ip-10-0-129-39 kubenswrapper[2575]: I0707 18:12:38.590866 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-m825v"] Jul 07 18:12:48.707669 ip-10-0-129-39 kubenswrapper[2575]: I0707 18:12:48.707625 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-m825v"] Jul 07 18:13:10.680561 ip-10-0-129-39 kubenswrapper[2575]: I0707 18:13:10.680528 2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-inf-b-54d21f/test-model-54d21f-kserve-5597656ddf-9klbc"] Jul 07 18:13:10.684233 ip-10-0-129-39 kubenswrapper[2575]: I0707 18:13:10.684210 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-54d21f/test-model-54d21f-kserve-5597656ddf-9klbc" Jul 07 18:13:10.686824 ip-10-0-129-39 kubenswrapper[2575]: I0707 18:13:10.686800 2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-b-54d21f\"/\"openshift-service-ca.crt\"" Jul 07 18:13:10.686940 ip-10-0-129-39 kubenswrapper[2575]: I0707 18:13:10.686839 2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-b-54d21f\"/\"kube-root-ca.crt\"" Jul 07 18:13:10.687829 ip-10-0-129-39 kubenswrapper[2575]: I0707 18:13:10.687808 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-b-54d21f\"/\"test-model-54d21f-kserve-self-signed-certs\"" Jul 07 18:13:10.687936 ip-10-0-129-39 kubenswrapper[2575]: I0707 18:13:10.687844 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-b-54d21f\"/\"default-dockercfg-6jcn6\"" Jul 07 18:13:10.695770 ip-10-0-129-39 kubenswrapper[2575]: I0707 18:13:10.695748 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-b-54d21f/test-model-54d21f-kserve-5597656ddf-9klbc"] Jul 07 18:13:10.699235 ip-10-0-129-39 kubenswrapper[2575]: I0707 18:13:10.699216 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c3a0afbf-2da3-4f25-9703-dbca958bbac2-model-cache\") pod \"test-model-54d21f-kserve-5597656ddf-9klbc\" (UID: \"c3a0afbf-2da3-4f25-9703-dbca958bbac2\") " pod="ai-tenant-e2e-inf-b-54d21f/test-model-54d21f-kserve-5597656ddf-9klbc" Jul 07 18:13:10.699350 ip-10-0-129-39 kubenswrapper[2575]: I0707 18:13:10.699260 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c3a0afbf-2da3-4f25-9703-dbca958bbac2-tls-certs\") pod \"test-model-54d21f-kserve-5597656ddf-9klbc\" (UID: \"c3a0afbf-2da3-4f25-9703-dbca958bbac2\") " pod="ai-tenant-e2e-inf-b-54d21f/test-model-54d21f-kserve-5597656ddf-9klbc" Jul 07 18:13:10.699411 ip-10-0-129-39 kubenswrapper[2575]: I0707 18:13:10.699362 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c3a0afbf-2da3-4f25-9703-dbca958bbac2-home\") pod \"test-model-54d21f-kserve-5597656ddf-9klbc\" (UID: \"c3a0afbf-2da3-4f25-9703-dbca958bbac2\") " pod="ai-tenant-e2e-inf-b-54d21f/test-model-54d21f-kserve-5597656ddf-9klbc" Jul 07 18:13:10.699411 ip-10-0-129-39 kubenswrapper[2575]: I0707 18:13:10.699393 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c3a0afbf-2da3-4f25-9703-dbca958bbac2-tmp-dir\") pod \"test-model-54d21f-kserve-5597656ddf-9klbc\" (UID: \"c3a0afbf-2da3-4f25-9703-dbca958bbac2\") " pod="ai-tenant-e2e-inf-b-54d21f/test-model-54d21f-kserve-5597656ddf-9klbc" Jul 07 18:13:10.699521 ip-10-0-129-39 kubenswrapper[2575]: I0707 18:13:10.699419 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c3a0afbf-2da3-4f25-9703-dbca958bbac2-dshm\") pod \"test-model-54d21f-kserve-5597656ddf-9klbc\" (UID: \"c3a0afbf-2da3-4f25-9703-dbca958bbac2\") " pod="ai-tenant-e2e-inf-b-54d21f/test-model-54d21f-kserve-5597656ddf-9klbc" Jul 07 18:13:10.699521 ip-10-0-129-39 kubenswrapper[2575]: I0707 18:13:10.699440 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8bdpx\" (UniqueName: \"kubernetes.io/projected/c3a0afbf-2da3-4f25-9703-dbca958bbac2-kube-api-access-8bdpx\") pod \"test-model-54d21f-kserve-5597656ddf-9klbc\" (UID: \"c3a0afbf-2da3-4f25-9703-dbca958bbac2\") " pod="ai-tenant-e2e-inf-b-54d21f/test-model-54d21f-kserve-5597656ddf-9klbc" Jul 07 18:13:10.800037 ip-10-0-129-39 kubenswrapper[2575]: I0707 18:13:10.799995 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c3a0afbf-2da3-4f25-9703-dbca958bbac2-model-cache\") pod \"test-model-54d21f-kserve-5597656ddf-9klbc\" (UID: \"c3a0afbf-2da3-4f25-9703-dbca958bbac2\") " pod="ai-tenant-e2e-inf-b-54d21f/test-model-54d21f-kserve-5597656ddf-9klbc" Jul 07 18:13:10.800202 ip-10-0-129-39 kubenswrapper[2575]: I0707 18:13:10.800061 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c3a0afbf-2da3-4f25-9703-dbca958bbac2-tls-certs\") pod \"test-model-54d21f-kserve-5597656ddf-9klbc\" (UID: \"c3a0afbf-2da3-4f25-9703-dbca958bbac2\") " pod="ai-tenant-e2e-inf-b-54d21f/test-model-54d21f-kserve-5597656ddf-9klbc" Jul 07 18:13:10.800202 ip-10-0-129-39 kubenswrapper[2575]: I0707 18:13:10.800113 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c3a0afbf-2da3-4f25-9703-dbca958bbac2-home\") pod \"test-model-54d21f-kserve-5597656ddf-9klbc\" (UID: \"c3a0afbf-2da3-4f25-9703-dbca958bbac2\") " pod="ai-tenant-e2e-inf-b-54d21f/test-model-54d21f-kserve-5597656ddf-9klbc" Jul 07 18:13:10.800202 ip-10-0-129-39 kubenswrapper[2575]: I0707 18:13:10.800128 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c3a0afbf-2da3-4f25-9703-dbca958bbac2-tmp-dir\") pod \"test-model-54d21f-kserve-5597656ddf-9klbc\" (UID: \"c3a0afbf-2da3-4f25-9703-dbca958bbac2\") " pod="ai-tenant-e2e-inf-b-54d21f/test-model-54d21f-kserve-5597656ddf-9klbc" Jul 07 18:13:10.800202 ip-10-0-129-39 kubenswrapper[2575]: I0707 18:13:10.800145 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c3a0afbf-2da3-4f25-9703-dbca958bbac2-dshm\") pod \"test-model-54d21f-kserve-5597656ddf-9klbc\" (UID: \"c3a0afbf-2da3-4f25-9703-dbca958bbac2\") " pod="ai-tenant-e2e-inf-b-54d21f/test-model-54d21f-kserve-5597656ddf-9klbc" Jul 07 18:13:10.800202 ip-10-0-129-39 kubenswrapper[2575]: I0707 18:13:10.800160 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8bdpx\" (UniqueName: \"kubernetes.io/projected/c3a0afbf-2da3-4f25-9703-dbca958bbac2-kube-api-access-8bdpx\") pod \"test-model-54d21f-kserve-5597656ddf-9klbc\" (UID: \"c3a0afbf-2da3-4f25-9703-dbca958bbac2\") " pod="ai-tenant-e2e-inf-b-54d21f/test-model-54d21f-kserve-5597656ddf-9klbc" Jul 07 18:13:10.800498 ip-10-0-129-39 kubenswrapper[2575]: I0707 18:13:10.800451 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c3a0afbf-2da3-4f25-9703-dbca958bbac2-model-cache\") pod \"test-model-54d21f-kserve-5597656ddf-9klbc\" (UID: \"c3a0afbf-2da3-4f25-9703-dbca958bbac2\") " pod="ai-tenant-e2e-inf-b-54d21f/test-model-54d21f-kserve-5597656ddf-9klbc" Jul 07 18:13:10.800498 ip-10-0-129-39 kubenswrapper[2575]: I0707 18:13:10.800482 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c3a0afbf-2da3-4f25-9703-dbca958bbac2-tmp-dir\") pod \"test-model-54d21f-kserve-5597656ddf-9klbc\" (UID: \"c3a0afbf-2da3-4f25-9703-dbca958bbac2\") " pod="ai-tenant-e2e-inf-b-54d21f/test-model-54d21f-kserve-5597656ddf-9klbc" Jul 07 18:13:10.800610 ip-10-0-129-39 kubenswrapper[2575]: I0707 18:13:10.800593 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c3a0afbf-2da3-4f25-9703-dbca958bbac2-home\") pod \"test-model-54d21f-kserve-5597656ddf-9klbc\" (UID: \"c3a0afbf-2da3-4f25-9703-dbca958bbac2\") " pod="ai-tenant-e2e-inf-b-54d21f/test-model-54d21f-kserve-5597656ddf-9klbc" Jul 07 18:13:10.802373 ip-10-0-129-39 kubenswrapper[2575]: I0707 18:13:10.802350 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c3a0afbf-2da3-4f25-9703-dbca958bbac2-dshm\") pod \"test-model-54d21f-kserve-5597656ddf-9klbc\" (UID: \"c3a0afbf-2da3-4f25-9703-dbca958bbac2\") " pod="ai-tenant-e2e-inf-b-54d21f/test-model-54d21f-kserve-5597656ddf-9klbc" Jul 07 18:13:10.802632 ip-10-0-129-39 kubenswrapper[2575]: I0707 18:13:10.802613 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c3a0afbf-2da3-4f25-9703-dbca958bbac2-tls-certs\") pod \"test-model-54d21f-kserve-5597656ddf-9klbc\" (UID: \"c3a0afbf-2da3-4f25-9703-dbca958bbac2\") " pod="ai-tenant-e2e-inf-b-54d21f/test-model-54d21f-kserve-5597656ddf-9klbc" Jul 07 18:13:10.808945 ip-10-0-129-39 kubenswrapper[2575]: I0707 18:13:10.808924 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8bdpx\" (UniqueName: \"kubernetes.io/projected/c3a0afbf-2da3-4f25-9703-dbca958bbac2-kube-api-access-8bdpx\") pod \"test-model-54d21f-kserve-5597656ddf-9klbc\" (UID: \"c3a0afbf-2da3-4f25-9703-dbca958bbac2\") " pod="ai-tenant-e2e-inf-b-54d21f/test-model-54d21f-kserve-5597656ddf-9klbc" Jul 07 18:13:10.994252 ip-10-0-129-39 kubenswrapper[2575]: I0707 18:13:10.994165 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-54d21f/test-model-54d21f-kserve-5597656ddf-9klbc" Jul 07 18:13:11.123374 ip-10-0-129-39 kubenswrapper[2575]: I0707 18:13:11.123342 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-b-54d21f/test-model-54d21f-kserve-5597656ddf-9klbc"] Jul 07 18:13:11.127212 ip-10-0-129-39 kubenswrapper[2575]: W0707 18:13:11.127181 2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podc3a0afbf_2da3_4f25_9703_dbca958bbac2.slice/crio-c281c396afdf1d0d78b901e2264911ab27d436250f5ac219351197a74feb533c WatchSource:0}: Error finding container c281c396afdf1d0d78b901e2264911ab27d436250f5ac219351197a74feb533c: Status 404 returned error can't find the container with id c281c396afdf1d0d78b901e2264911ab27d436250f5ac219351197a74feb533c Jul 07 18:13:11.891566 ip-10-0-129-39 kubenswrapper[2575]: I0707 18:13:11.891525 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-54d21f/test-model-54d21f-kserve-5597656ddf-9klbc" event={"ID":"c3a0afbf-2da3-4f25-9703-dbca958bbac2","Type":"ContainerStarted","Data":"bef427ba9b483dbcf8561e4ae500f0ded9e13ad1260f56bd0f9380b0a84ef644"} Jul 07 18:13:11.891946 ip-10-0-129-39 kubenswrapper[2575]: I0707 18:13:11.891574 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-54d21f/test-model-54d21f-kserve-5597656ddf-9klbc" event={"ID":"c3a0afbf-2da3-4f25-9703-dbca958bbac2","Type":"ContainerStarted","Data":"c281c396afdf1d0d78b901e2264911ab27d436250f5ac219351197a74feb533c"} Jul 07 18:13:11.917016 ip-10-0-129-39 kubenswrapper[2575]: I0707 18:13:11.916972 2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-inf-b-54d21f/test-model-54d21f-kserve-5597656ddf-9klbc" podStartSLOduration=1.9169579049999999 podStartE2EDuration="1.916957905s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-07 18:13:10 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 18:13:11.916129915 +0000 UTC m=+1952.421936587" watchObservedRunningTime="2026-07-07 18:13:11.916957905 +0000 UTC m=+1952.422764570" Jul 07 18:13:14.593323 ip-10-0-129-39 kubenswrapper[2575]: I0707 18:13:14.593287 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-m825v"] Jul 07 18:13:20.994869 ip-10-0-129-39 kubenswrapper[2575]: I0707 18:13:20.994840 2575 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-inf-b-54d21f/test-model-54d21f-kserve-5597656ddf-9klbc" Jul 07 18:13:20.995271 ip-10-0-129-39 kubenswrapper[2575]: I0707 18:13:20.994887 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-inf-b-54d21f/test-model-54d21f-kserve-5597656ddf-9klbc" Jul 07 18:13:21.007331 ip-10-0-129-39 kubenswrapper[2575]: I0707 18:13:21.007315 2575 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-inf-b-54d21f/test-model-54d21f-kserve-5597656ddf-9klbc" Jul 07 18:13:21.949590 ip-10-0-129-39 kubenswrapper[2575]: I0707 18:13:21.949562 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-inf-b-54d21f/test-model-54d21f-kserve-5597656ddf-9klbc" Jul 07 18:13:33.480454 ip-10-0-129-39 kubenswrapper[2575]: I0707 18:13:33.480368 2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-b-54d21f/test-model-54d21f-kserve-5597656ddf-9klbc"] Jul 07 18:13:37.586299 ip-10-0-129-39 kubenswrapper[2575]: I0707 18:13:37.586262 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-m825v"] Jul 07 18:13:48.500570 ip-10-0-129-39 kubenswrapper[2575]: I0707 18:13:48.500484 2575 kuberuntime_container.go:872] "Killing container with a grace period" pod="ai-tenant-e2e-inf-b-54d21f/test-model-54d21f-kserve-5597656ddf-9klbc" podUID="c3a0afbf-2da3-4f25-9703-dbca958bbac2" containerName="main" containerID="cri-o://bef427ba9b483dbcf8561e4ae500f0ded9e13ad1260f56bd0f9380b0a84ef644" gracePeriod=45 Jul 07 18:13:48.750467 ip-10-0-129-39 kubenswrapper[2575]: I0707 18:13:48.750444 2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-54d21f/test-model-54d21f-kserve-5597656ddf-9klbc" Jul 07 18:13:48.910482 ip-10-0-129-39 kubenswrapper[2575]: I0707 18:13:48.910454 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c3a0afbf-2da3-4f25-9703-dbca958bbac2-dshm\") pod \"c3a0afbf-2da3-4f25-9703-dbca958bbac2\" (UID: \"c3a0afbf-2da3-4f25-9703-dbca958bbac2\") " Jul 07 18:13:48.910662 ip-10-0-129-39 kubenswrapper[2575]: I0707 18:13:48.910518 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c3a0afbf-2da3-4f25-9703-dbca958bbac2-tls-certs\") pod \"c3a0afbf-2da3-4f25-9703-dbca958bbac2\" (UID: \"c3a0afbf-2da3-4f25-9703-dbca958bbac2\") " Jul 07 18:13:48.910662 ip-10-0-129-39 kubenswrapper[2575]: I0707 18:13:48.910561 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-8bdpx\" (UniqueName: \"kubernetes.io/projected/c3a0afbf-2da3-4f25-9703-dbca958bbac2-kube-api-access-8bdpx\") pod \"c3a0afbf-2da3-4f25-9703-dbca958bbac2\" (UID: \"c3a0afbf-2da3-4f25-9703-dbca958bbac2\") " Jul 07 18:13:48.910662 ip-10-0-129-39 kubenswrapper[2575]: I0707 18:13:48.910579 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c3a0afbf-2da3-4f25-9703-dbca958bbac2-home\") pod \"c3a0afbf-2da3-4f25-9703-dbca958bbac2\" (UID: \"c3a0afbf-2da3-4f25-9703-dbca958bbac2\") " Jul 07 18:13:48.910662 ip-10-0-129-39 kubenswrapper[2575]: I0707 18:13:48.910619 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c3a0afbf-2da3-4f25-9703-dbca958bbac2-model-cache\") pod \"c3a0afbf-2da3-4f25-9703-dbca958bbac2\" (UID: \"c3a0afbf-2da3-4f25-9703-dbca958bbac2\") " Jul 07 18:13:48.910875 ip-10-0-129-39 kubenswrapper[2575]: I0707 18:13:48.910706 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c3a0afbf-2da3-4f25-9703-dbca958bbac2-tmp-dir\") pod \"c3a0afbf-2da3-4f25-9703-dbca958bbac2\" (UID: \"c3a0afbf-2da3-4f25-9703-dbca958bbac2\") " Jul 07 18:13:48.910932 ip-10-0-129-39 kubenswrapper[2575]: I0707 18:13:48.910868 2575 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c3a0afbf-2da3-4f25-9703-dbca958bbac2-model-cache" (OuterVolumeSpecName: "model-cache") pod "c3a0afbf-2da3-4f25-9703-dbca958bbac2" (UID: "c3a0afbf-2da3-4f25-9703-dbca958bbac2"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 07 18:13:48.910932 ip-10-0-129-39 kubenswrapper[2575]: I0707 18:13:48.910889 2575 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c3a0afbf-2da3-4f25-9703-dbca958bbac2-home" (OuterVolumeSpecName: "home") pod "c3a0afbf-2da3-4f25-9703-dbca958bbac2" (UID: "c3a0afbf-2da3-4f25-9703-dbca958bbac2"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 07 18:13:48.911023 ip-10-0-129-39 kubenswrapper[2575]: I0707 18:13:48.910955 2575 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c3a0afbf-2da3-4f25-9703-dbca958bbac2-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "c3a0afbf-2da3-4f25-9703-dbca958bbac2" (UID: "c3a0afbf-2da3-4f25-9703-dbca958bbac2"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 07 18:13:48.911147 ip-10-0-129-39 kubenswrapper[2575]: I0707 18:13:48.911130 2575 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c3a0afbf-2da3-4f25-9703-dbca958bbac2-model-cache\") on node \"ip-10-0-129-39.ec2.internal\" DevicePath \"\"" Jul 07 18:13:48.911195 ip-10-0-129-39 kubenswrapper[2575]: I0707 18:13:48.911155 2575 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c3a0afbf-2da3-4f25-9703-dbca958bbac2-tmp-dir\") on node \"ip-10-0-129-39.ec2.internal\" DevicePath \"\"" Jul 07 18:13:48.911195 ip-10-0-129-39 kubenswrapper[2575]: I0707 18:13:48.911168 2575 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c3a0afbf-2da3-4f25-9703-dbca958bbac2-home\") on node \"ip-10-0-129-39.ec2.internal\" DevicePath \"\"" Jul 07 18:13:48.912584 ip-10-0-129-39 kubenswrapper[2575]: I0707 18:13:48.912561 2575 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c3a0afbf-2da3-4f25-9703-dbca958bbac2-dshm" (OuterVolumeSpecName: "dshm") pod "c3a0afbf-2da3-4f25-9703-dbca958bbac2" (UID: "c3a0afbf-2da3-4f25-9703-dbca958bbac2"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 07 18:13:48.912694 ip-10-0-129-39 kubenswrapper[2575]: I0707 18:13:48.912657 2575 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c3a0afbf-2da3-4f25-9703-dbca958bbac2-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "c3a0afbf-2da3-4f25-9703-dbca958bbac2" (UID: "c3a0afbf-2da3-4f25-9703-dbca958bbac2"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 07 18:13:48.912817 ip-10-0-129-39 kubenswrapper[2575]: I0707 18:13:48.912798 2575 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c3a0afbf-2da3-4f25-9703-dbca958bbac2-kube-api-access-8bdpx" (OuterVolumeSpecName: "kube-api-access-8bdpx") pod "c3a0afbf-2da3-4f25-9703-dbca958bbac2" (UID: "c3a0afbf-2da3-4f25-9703-dbca958bbac2"). InnerVolumeSpecName "kube-api-access-8bdpx". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 07 18:13:49.012581 ip-10-0-129-39 kubenswrapper[2575]: I0707 18:13:49.012549 2575 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c3a0afbf-2da3-4f25-9703-dbca958bbac2-dshm\") on node \"ip-10-0-129-39.ec2.internal\" DevicePath \"\"" Jul 07 18:13:49.012581 ip-10-0-129-39 kubenswrapper[2575]: I0707 18:13:49.012573 2575 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c3a0afbf-2da3-4f25-9703-dbca958bbac2-tls-certs\") on node \"ip-10-0-129-39.ec2.internal\" DevicePath \"\"" Jul 07 18:13:49.012581 ip-10-0-129-39 kubenswrapper[2575]: I0707 18:13:49.012584 2575 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8bdpx\" (UniqueName: \"kubernetes.io/projected/c3a0afbf-2da3-4f25-9703-dbca958bbac2-kube-api-access-8bdpx\") on node \"ip-10-0-129-39.ec2.internal\" DevicePath \"\"" Jul 07 18:13:49.062057 ip-10-0-129-39 kubenswrapper[2575]: I0707 18:13:49.062022 2575 generic.go:356] "Generic (PLEG): container finished" podID="c3a0afbf-2da3-4f25-9703-dbca958bbac2" containerID="bef427ba9b483dbcf8561e4ae500f0ded9e13ad1260f56bd0f9380b0a84ef644" exitCode=0 Jul 07 18:13:49.062224 ip-10-0-129-39 kubenswrapper[2575]: I0707 18:13:49.062068 2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-54d21f/test-model-54d21f-kserve-5597656ddf-9klbc" Jul 07 18:13:49.062224 ip-10-0-129-39 kubenswrapper[2575]: I0707 18:13:49.062102 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-54d21f/test-model-54d21f-kserve-5597656ddf-9klbc" event={"ID":"c3a0afbf-2da3-4f25-9703-dbca958bbac2","Type":"ContainerDied","Data":"bef427ba9b483dbcf8561e4ae500f0ded9e13ad1260f56bd0f9380b0a84ef644"} Jul 07 18:13:49.062224 ip-10-0-129-39 kubenswrapper[2575]: I0707 18:13:49.062155 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-54d21f/test-model-54d21f-kserve-5597656ddf-9klbc" event={"ID":"c3a0afbf-2da3-4f25-9703-dbca958bbac2","Type":"ContainerDied","Data":"c281c396afdf1d0d78b901e2264911ab27d436250f5ac219351197a74feb533c"} Jul 07 18:13:49.062224 ip-10-0-129-39 kubenswrapper[2575]: I0707 18:13:49.062176 2575 scope.go:117] "RemoveContainer" containerID="bef427ba9b483dbcf8561e4ae500f0ded9e13ad1260f56bd0f9380b0a84ef644" Jul 07 18:13:49.070881 ip-10-0-129-39 kubenswrapper[2575]: I0707 18:13:49.070865 2575 scope.go:117] "RemoveContainer" containerID="bef427ba9b483dbcf8561e4ae500f0ded9e13ad1260f56bd0f9380b0a84ef644" Jul 07 18:13:49.071146 ip-10-0-129-39 kubenswrapper[2575]: E0707 18:13:49.071125 2575 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"bef427ba9b483dbcf8561e4ae500f0ded9e13ad1260f56bd0f9380b0a84ef644\": container with ID starting with bef427ba9b483dbcf8561e4ae500f0ded9e13ad1260f56bd0f9380b0a84ef644 not found: ID does not exist" containerID="bef427ba9b483dbcf8561e4ae500f0ded9e13ad1260f56bd0f9380b0a84ef644" Jul 07 18:13:49.071217 ip-10-0-129-39 kubenswrapper[2575]: I0707 18:13:49.071157 2575 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bef427ba9b483dbcf8561e4ae500f0ded9e13ad1260f56bd0f9380b0a84ef644"} err="failed to get container status \"bef427ba9b483dbcf8561e4ae500f0ded9e13ad1260f56bd0f9380b0a84ef644\": rpc error: code = NotFound desc = could not find container \"bef427ba9b483dbcf8561e4ae500f0ded9e13ad1260f56bd0f9380b0a84ef644\": container with ID starting with bef427ba9b483dbcf8561e4ae500f0ded9e13ad1260f56bd0f9380b0a84ef644 not found: ID does not exist" Jul 07 18:13:49.084191 ip-10-0-129-39 kubenswrapper[2575]: I0707 18:13:49.084169 2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-b-54d21f/test-model-54d21f-kserve-5597656ddf-9klbc"] Jul 07 18:13:49.087904 ip-10-0-129-39 kubenswrapper[2575]: I0707 18:13:49.087881 2575 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-inf-b-54d21f/test-model-54d21f-kserve-5597656ddf-9klbc"] Jul 07 18:13:50.073190 ip-10-0-129-39 kubenswrapper[2575]: I0707 18:13:50.073144 2575 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="c3a0afbf-2da3-4f25-9703-dbca958bbac2" path="/var/lib/kubelet/pods/c3a0afbf-2da3-4f25-9703-dbca958bbac2/volumes" Jul 07 18:14:22.191695 ip-10-0-129-39 kubenswrapper[2575]: I0707 18:14:22.191660 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-m825v"] Jul 07 18:14:34.891576 ip-10-0-129-39 kubenswrapper[2575]: I0707 18:14:34.891544 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-m825v"] Jul 07 18:14:43.939114 ip-10-0-129-39 kubenswrapper[2575]: I0707 18:14:43.939077 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-m825v"] Jul 07 18:14:54.083019 ip-10-0-129-39 kubenswrapper[2575]: I0707 18:14:54.082985 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-m825v"] Jul 07 18:15:07.696524 ip-10-0-129-39 kubenswrapper[2575]: I0707 18:15:07.694565 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-m825v"] Jul 07 18:15:18.993306 ip-10-0-129-39 kubenswrapper[2575]: I0707 18:15:18.993271 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-m825v"] Jul 07 18:15:29.674952 ip-10-0-129-39 kubenswrapper[2575]: I0707 18:15:29.674917 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-m825v"] Jul 07 18:15:38.692078 ip-10-0-129-39 kubenswrapper[2575]: I0707 18:15:38.692039 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-m825v"] Jul 07 18:15:55.620941 ip-10-0-129-39 kubenswrapper[2575]: I0707 18:15:55.620906 2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-b-fad388/auth-test-model-fad388-kserve-654f46b44c-85szm"] Jul 07 18:15:55.621302 ip-10-0-129-39 kubenswrapper[2575]: I0707 18:15:55.621286 2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c3a0afbf-2da3-4f25-9703-dbca958bbac2" containerName="main" Jul 07 18:15:55.621302 ip-10-0-129-39 kubenswrapper[2575]: I0707 18:15:55.621298 2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="c3a0afbf-2da3-4f25-9703-dbca958bbac2" containerName="main" Jul 07 18:15:55.621388 ip-10-0-129-39 kubenswrapper[2575]: I0707 18:15:55.621377 2575 memory_manager.go:368] "RemoveStaleState removing state" podUID="c3a0afbf-2da3-4f25-9703-dbca958bbac2" containerName="main" Jul 07 18:15:55.624345 ip-10-0-129-39 kubenswrapper[2575]: I0707 18:15:55.624325 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-fad388/auth-test-model-fad388-kserve-654f46b44c-85szm" Jul 07 18:15:55.628189 ip-10-0-129-39 kubenswrapper[2575]: I0707 18:15:55.628164 2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-fad388\"/\"openshift-service-ca.crt\"" Jul 07 18:15:55.628337 ip-10-0-129-39 kubenswrapper[2575]: I0707 18:15:55.628316 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-fad388\"/\"default-dockercfg-qpg98\"" Jul 07 18:15:55.629124 ip-10-0-129-39 kubenswrapper[2575]: I0707 18:15:55.629104 2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-fad388\"/\"kube-root-ca.crt\"" Jul 07 18:15:55.629224 ip-10-0-129-39 kubenswrapper[2575]: I0707 18:15:55.629193 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-fad388\"/\"auth-test-model-fad388-kserve-self-signed-certs\"" Jul 07 18:15:55.640726 ip-10-0-129-39 kubenswrapper[2575]: I0707 18:15:55.640693 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-fad388/auth-test-model-fad388-kserve-654f46b44c-85szm"] Jul 07 18:15:55.741947 ip-10-0-129-39 kubenswrapper[2575]: I0707 18:15:55.741910 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0bab5041-28ce-4a5b-85e9-92cf9acb6ccc-model-cache\") pod \"auth-test-model-fad388-kserve-654f46b44c-85szm\" (UID: \"0bab5041-28ce-4a5b-85e9-92cf9acb6ccc\") " pod="ai-tenant-e2e-shared-b-fad388/auth-test-model-fad388-kserve-654f46b44c-85szm" Jul 07 18:15:55.742123 ip-10-0-129-39 kubenswrapper[2575]: I0707 18:15:55.741982 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hbcrj\" (UniqueName: \"kubernetes.io/projected/0bab5041-28ce-4a5b-85e9-92cf9acb6ccc-kube-api-access-hbcrj\") pod \"auth-test-model-fad388-kserve-654f46b44c-85szm\" (UID: \"0bab5041-28ce-4a5b-85e9-92cf9acb6ccc\") " pod="ai-tenant-e2e-shared-b-fad388/auth-test-model-fad388-kserve-654f46b44c-85szm" Jul 07 18:15:55.742123 ip-10-0-129-39 kubenswrapper[2575]: I0707 18:15:55.742010 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0bab5041-28ce-4a5b-85e9-92cf9acb6ccc-home\") pod \"auth-test-model-fad388-kserve-654f46b44c-85szm\" (UID: \"0bab5041-28ce-4a5b-85e9-92cf9acb6ccc\") " pod="ai-tenant-e2e-shared-b-fad388/auth-test-model-fad388-kserve-654f46b44c-85szm" Jul 07 18:15:55.742123 ip-10-0-129-39 kubenswrapper[2575]: I0707 18:15:55.742040 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0bab5041-28ce-4a5b-85e9-92cf9acb6ccc-tmp-dir\") pod \"auth-test-model-fad388-kserve-654f46b44c-85szm\" (UID: \"0bab5041-28ce-4a5b-85e9-92cf9acb6ccc\") " pod="ai-tenant-e2e-shared-b-fad388/auth-test-model-fad388-kserve-654f46b44c-85szm" Jul 07 18:15:55.742123 ip-10-0-129-39 kubenswrapper[2575]: I0707 18:15:55.742091 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0bab5041-28ce-4a5b-85e9-92cf9acb6ccc-dshm\") pod \"auth-test-model-fad388-kserve-654f46b44c-85szm\" (UID: \"0bab5041-28ce-4a5b-85e9-92cf9acb6ccc\") " pod="ai-tenant-e2e-shared-b-fad388/auth-test-model-fad388-kserve-654f46b44c-85szm" Jul 07 18:15:55.742262 ip-10-0-129-39 kubenswrapper[2575]: I0707 18:15:55.742161 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0bab5041-28ce-4a5b-85e9-92cf9acb6ccc-tls-certs\") pod \"auth-test-model-fad388-kserve-654f46b44c-85szm\" (UID: \"0bab5041-28ce-4a5b-85e9-92cf9acb6ccc\") " pod="ai-tenant-e2e-shared-b-fad388/auth-test-model-fad388-kserve-654f46b44c-85szm" Jul 07 18:15:55.843374 ip-10-0-129-39 kubenswrapper[2575]: I0707 18:15:55.843344 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0bab5041-28ce-4a5b-85e9-92cf9acb6ccc-home\") pod \"auth-test-model-fad388-kserve-654f46b44c-85szm\" (UID: \"0bab5041-28ce-4a5b-85e9-92cf9acb6ccc\") " pod="ai-tenant-e2e-shared-b-fad388/auth-test-model-fad388-kserve-654f46b44c-85szm" Jul 07 18:15:55.843595 ip-10-0-129-39 kubenswrapper[2575]: I0707 18:15:55.843389 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0bab5041-28ce-4a5b-85e9-92cf9acb6ccc-tmp-dir\") pod \"auth-test-model-fad388-kserve-654f46b44c-85szm\" (UID: \"0bab5041-28ce-4a5b-85e9-92cf9acb6ccc\") " pod="ai-tenant-e2e-shared-b-fad388/auth-test-model-fad388-kserve-654f46b44c-85szm" Jul 07 18:15:55.843595 ip-10-0-129-39 kubenswrapper[2575]: I0707 18:15:55.843421 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0bab5041-28ce-4a5b-85e9-92cf9acb6ccc-dshm\") pod \"auth-test-model-fad388-kserve-654f46b44c-85szm\" (UID: \"0bab5041-28ce-4a5b-85e9-92cf9acb6ccc\") " pod="ai-tenant-e2e-shared-b-fad388/auth-test-model-fad388-kserve-654f46b44c-85szm" Jul 07 18:15:55.843595 ip-10-0-129-39 kubenswrapper[2575]: I0707 18:15:55.843464 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0bab5041-28ce-4a5b-85e9-92cf9acb6ccc-tls-certs\") pod \"auth-test-model-fad388-kserve-654f46b44c-85szm\" (UID: \"0bab5041-28ce-4a5b-85e9-92cf9acb6ccc\") " pod="ai-tenant-e2e-shared-b-fad388/auth-test-model-fad388-kserve-654f46b44c-85szm" Jul 07 18:15:55.843595 ip-10-0-129-39 kubenswrapper[2575]: I0707 18:15:55.843566 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0bab5041-28ce-4a5b-85e9-92cf9acb6ccc-model-cache\") pod \"auth-test-model-fad388-kserve-654f46b44c-85szm\" (UID: \"0bab5041-28ce-4a5b-85e9-92cf9acb6ccc\") " pod="ai-tenant-e2e-shared-b-fad388/auth-test-model-fad388-kserve-654f46b44c-85szm" Jul 07 18:15:55.843816 ip-10-0-129-39 kubenswrapper[2575]: I0707 18:15:55.843622 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hbcrj\" (UniqueName: \"kubernetes.io/projected/0bab5041-28ce-4a5b-85e9-92cf9acb6ccc-kube-api-access-hbcrj\") pod \"auth-test-model-fad388-kserve-654f46b44c-85szm\" (UID: \"0bab5041-28ce-4a5b-85e9-92cf9acb6ccc\") " pod="ai-tenant-e2e-shared-b-fad388/auth-test-model-fad388-kserve-654f46b44c-85szm" Jul 07 18:15:55.843816 ip-10-0-129-39 kubenswrapper[2575]: I0707 18:15:55.843776 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0bab5041-28ce-4a5b-85e9-92cf9acb6ccc-home\") pod \"auth-test-model-fad388-kserve-654f46b44c-85szm\" (UID: \"0bab5041-28ce-4a5b-85e9-92cf9acb6ccc\") " pod="ai-tenant-e2e-shared-b-fad388/auth-test-model-fad388-kserve-654f46b44c-85szm" Jul 07 18:15:55.843916 ip-10-0-129-39 kubenswrapper[2575]: I0707 18:15:55.843826 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0bab5041-28ce-4a5b-85e9-92cf9acb6ccc-tmp-dir\") pod \"auth-test-model-fad388-kserve-654f46b44c-85szm\" (UID: \"0bab5041-28ce-4a5b-85e9-92cf9acb6ccc\") " pod="ai-tenant-e2e-shared-b-fad388/auth-test-model-fad388-kserve-654f46b44c-85szm" Jul 07 18:15:55.844038 ip-10-0-129-39 kubenswrapper[2575]: I0707 18:15:55.844017 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0bab5041-28ce-4a5b-85e9-92cf9acb6ccc-model-cache\") pod \"auth-test-model-fad388-kserve-654f46b44c-85szm\" (UID: \"0bab5041-28ce-4a5b-85e9-92cf9acb6ccc\") " pod="ai-tenant-e2e-shared-b-fad388/auth-test-model-fad388-kserve-654f46b44c-85szm" Jul 07 18:15:55.845775 ip-10-0-129-39 kubenswrapper[2575]: I0707 18:15:55.845753 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0bab5041-28ce-4a5b-85e9-92cf9acb6ccc-dshm\") pod \"auth-test-model-fad388-kserve-654f46b44c-85szm\" (UID: \"0bab5041-28ce-4a5b-85e9-92cf9acb6ccc\") " pod="ai-tenant-e2e-shared-b-fad388/auth-test-model-fad388-kserve-654f46b44c-85szm" Jul 07 18:15:55.846016 ip-10-0-129-39 kubenswrapper[2575]: I0707 18:15:55.846000 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0bab5041-28ce-4a5b-85e9-92cf9acb6ccc-tls-certs\") pod \"auth-test-model-fad388-kserve-654f46b44c-85szm\" (UID: \"0bab5041-28ce-4a5b-85e9-92cf9acb6ccc\") " pod="ai-tenant-e2e-shared-b-fad388/auth-test-model-fad388-kserve-654f46b44c-85szm" Jul 07 18:15:55.857781 ip-10-0-129-39 kubenswrapper[2575]: I0707 18:15:55.857757 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hbcrj\" (UniqueName: \"kubernetes.io/projected/0bab5041-28ce-4a5b-85e9-92cf9acb6ccc-kube-api-access-hbcrj\") pod \"auth-test-model-fad388-kserve-654f46b44c-85szm\" (UID: \"0bab5041-28ce-4a5b-85e9-92cf9acb6ccc\") " pod="ai-tenant-e2e-shared-b-fad388/auth-test-model-fad388-kserve-654f46b44c-85szm" Jul 07 18:15:55.933390 ip-10-0-129-39 kubenswrapper[2575]: I0707 18:15:55.933334 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-fad388/auth-test-model-fad388-kserve-654f46b44c-85szm" Jul 07 18:15:56.074540 ip-10-0-129-39 kubenswrapper[2575]: W0707 18:15:56.074496 2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod0bab5041_28ce_4a5b_85e9_92cf9acb6ccc.slice/crio-284f4f10a6df2342cdc2f72799d1f1aa4d906fade5e6578c3b2a10511bb09e43 WatchSource:0}: Error finding container 284f4f10a6df2342cdc2f72799d1f1aa4d906fade5e6578c3b2a10511bb09e43: Status 404 returned error can't find the container with id 284f4f10a6df2342cdc2f72799d1f1aa4d906fade5e6578c3b2a10511bb09e43 Jul 07 18:15:56.075196 ip-10-0-129-39 kubenswrapper[2575]: I0707 18:15:56.075174 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-fad388/auth-test-model-fad388-kserve-654f46b44c-85szm"] Jul 07 18:15:56.659896 ip-10-0-129-39 kubenswrapper[2575]: I0707 18:15:56.659858 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-fad388/auth-test-model-fad388-kserve-654f46b44c-85szm" event={"ID":"0bab5041-28ce-4a5b-85e9-92cf9acb6ccc","Type":"ContainerStarted","Data":"0f38d3beb94c079df2a205539f91e164ce112bd8e5c8c667c7081d0408983180"} Jul 07 18:15:56.659896 ip-10-0-129-39 kubenswrapper[2575]: I0707 18:15:56.659901 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-fad388/auth-test-model-fad388-kserve-654f46b44c-85szm" event={"ID":"0bab5041-28ce-4a5b-85e9-92cf9acb6ccc","Type":"ContainerStarted","Data":"284f4f10a6df2342cdc2f72799d1f1aa4d906fade5e6578c3b2a10511bb09e43"} Jul 07 18:15:56.677848 ip-10-0-129-39 kubenswrapper[2575]: I0707 18:15:56.677802 2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-b-fad388/auth-test-model-fad388-kserve-654f46b44c-85szm" podStartSLOduration=1.677787056 podStartE2EDuration="1.677787056s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-07 18:15:55 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 18:15:56.676374933 +0000 UTC m=+2117.182181599" watchObservedRunningTime="2026-07-07 18:15:56.677787056 +0000 UTC m=+2117.183593736" Jul 07 18:15:59.085445 ip-10-0-129-39 kubenswrapper[2575]: I0707 18:15:59.085413 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-m825v"] Jul 07 18:16:05.933485 ip-10-0-129-39 kubenswrapper[2575]: I0707 18:16:05.933447 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-b-fad388/auth-test-model-fad388-kserve-654f46b44c-85szm" Jul 07 18:16:05.933961 ip-10-0-129-39 kubenswrapper[2575]: I0707 18:16:05.933553 2575 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-b-fad388/auth-test-model-fad388-kserve-654f46b44c-85szm" Jul 07 18:16:05.951462 ip-10-0-129-39 kubenswrapper[2575]: I0707 18:16:05.951440 2575 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-b-fad388/auth-test-model-fad388-kserve-654f46b44c-85szm" Jul 07 18:16:06.720030 ip-10-0-129-39 kubenswrapper[2575]: I0707 18:16:06.720004 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-b-fad388/auth-test-model-fad388-kserve-654f46b44c-85szm" Jul 07 18:16:13.434138 ip-10-0-129-39 kubenswrapper[2575]: I0707 18:16:13.434103 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-m825v"] Jul 07 18:16:31.024048 ip-10-0-129-39 kubenswrapper[2575]: I0707 18:16:31.024008 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-m825v"] Jul 07 18:16:45.195960 ip-10-0-129-39 kubenswrapper[2575]: I0707 18:16:45.195925 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-m825v"] Jul 07 18:17:01.110318 ip-10-0-129-39 kubenswrapper[2575]: I0707 18:17:01.110284 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-m825v"] Jul 07 18:17:16.443655 ip-10-0-129-39 kubenswrapper[2575]: I0707 18:17:16.443615 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-m825v"] Jul 07 18:17:31.684176 ip-10-0-129-39 kubenswrapper[2575]: I0707 18:17:31.684137 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-m825v"] Jul 07 18:17:43.925662 ip-10-0-129-39 kubenswrapper[2575]: I0707 18:17:43.925624 2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-fad388/auth-test-model-fad388-kserve-654f46b44c-85szm"] Jul 07 18:17:45.594274 ip-10-0-129-39 kubenswrapper[2575]: I0707 18:17:45.594245 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-m825v"] Jul 07 18:17:49.395795 ip-10-0-129-39 kubenswrapper[2575]: I0707 18:17:49.395757 2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-a-21f62c/sub-test-model-21f62c-kserve-dd8db588c-h5vrj"] Jul 07 18:17:49.399293 ip-10-0-129-39 kubenswrapper[2575]: I0707 18:17:49.399273 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-21f62c/sub-test-model-21f62c-kserve-dd8db588c-h5vrj" Jul 07 18:17:49.401879 ip-10-0-129-39 kubenswrapper[2575]: I0707 18:17:49.401855 2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-21f62c\"/\"openshift-service-ca.crt\"" Jul 07 18:17:49.403102 ip-10-0-129-39 kubenswrapper[2575]: I0707 18:17:49.403072 2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-21f62c\"/\"kube-root-ca.crt\"" Jul 07 18:17:49.403102 ip-10-0-129-39 kubenswrapper[2575]: I0707 18:17:49.403081 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-21f62c\"/\"default-dockercfg-xvp6n\"" Jul 07 18:17:49.403232 ip-10-0-129-39 kubenswrapper[2575]: I0707 18:17:49.403153 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-21f62c\"/\"sub-test-model-21f62c-kserve-self-signed-certs\"" Jul 07 18:17:49.407138 ip-10-0-129-39 kubenswrapper[2575]: I0707 18:17:49.407116 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-21f62c/sub-test-model-21f62c-kserve-dd8db588c-h5vrj"] Jul 07 18:17:49.492700 ip-10-0-129-39 kubenswrapper[2575]: I0707 18:17:49.492665 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0319c93a-e08d-4677-9150-96dce21fb5d9-home\") pod \"sub-test-model-21f62c-kserve-dd8db588c-h5vrj\" (UID: \"0319c93a-e08d-4677-9150-96dce21fb5d9\") " pod="ai-tenant-e2e-shared-a-21f62c/sub-test-model-21f62c-kserve-dd8db588c-h5vrj" Jul 07 18:17:49.492871 ip-10-0-129-39 kubenswrapper[2575]: I0707 18:17:49.492764 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0319c93a-e08d-4677-9150-96dce21fb5d9-tls-certs\") pod \"sub-test-model-21f62c-kserve-dd8db588c-h5vrj\" (UID: \"0319c93a-e08d-4677-9150-96dce21fb5d9\") " pod="ai-tenant-e2e-shared-a-21f62c/sub-test-model-21f62c-kserve-dd8db588c-h5vrj" Jul 07 18:17:49.492871 ip-10-0-129-39 kubenswrapper[2575]: I0707 18:17:49.492815 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0319c93a-e08d-4677-9150-96dce21fb5d9-tmp-dir\") pod \"sub-test-model-21f62c-kserve-dd8db588c-h5vrj\" (UID: \"0319c93a-e08d-4677-9150-96dce21fb5d9\") " pod="ai-tenant-e2e-shared-a-21f62c/sub-test-model-21f62c-kserve-dd8db588c-h5vrj" Jul 07 18:17:49.492993 ip-10-0-129-39 kubenswrapper[2575]: I0707 18:17:49.492915 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8ghct\" (UniqueName: \"kubernetes.io/projected/0319c93a-e08d-4677-9150-96dce21fb5d9-kube-api-access-8ghct\") pod \"sub-test-model-21f62c-kserve-dd8db588c-h5vrj\" (UID: \"0319c93a-e08d-4677-9150-96dce21fb5d9\") " pod="ai-tenant-e2e-shared-a-21f62c/sub-test-model-21f62c-kserve-dd8db588c-h5vrj" Jul 07 18:17:49.492993 ip-10-0-129-39 kubenswrapper[2575]: I0707 18:17:49.492960 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0319c93a-e08d-4677-9150-96dce21fb5d9-model-cache\") pod \"sub-test-model-21f62c-kserve-dd8db588c-h5vrj\" (UID: \"0319c93a-e08d-4677-9150-96dce21fb5d9\") " pod="ai-tenant-e2e-shared-a-21f62c/sub-test-model-21f62c-kserve-dd8db588c-h5vrj" Jul 07 18:17:49.493105 ip-10-0-129-39 kubenswrapper[2575]: I0707 18:17:49.492994 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0319c93a-e08d-4677-9150-96dce21fb5d9-dshm\") pod \"sub-test-model-21f62c-kserve-dd8db588c-h5vrj\" (UID: \"0319c93a-e08d-4677-9150-96dce21fb5d9\") " pod="ai-tenant-e2e-shared-a-21f62c/sub-test-model-21f62c-kserve-dd8db588c-h5vrj" Jul 07 18:17:49.594046 ip-10-0-129-39 kubenswrapper[2575]: I0707 18:17:49.594004 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8ghct\" (UniqueName: \"kubernetes.io/projected/0319c93a-e08d-4677-9150-96dce21fb5d9-kube-api-access-8ghct\") pod \"sub-test-model-21f62c-kserve-dd8db588c-h5vrj\" (UID: \"0319c93a-e08d-4677-9150-96dce21fb5d9\") " pod="ai-tenant-e2e-shared-a-21f62c/sub-test-model-21f62c-kserve-dd8db588c-h5vrj" Jul 07 18:17:49.594244 ip-10-0-129-39 kubenswrapper[2575]: I0707 18:17:49.594050 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0319c93a-e08d-4677-9150-96dce21fb5d9-model-cache\") pod \"sub-test-model-21f62c-kserve-dd8db588c-h5vrj\" (UID: \"0319c93a-e08d-4677-9150-96dce21fb5d9\") " pod="ai-tenant-e2e-shared-a-21f62c/sub-test-model-21f62c-kserve-dd8db588c-h5vrj" Jul 07 18:17:49.594244 ip-10-0-129-39 kubenswrapper[2575]: I0707 18:17:49.594075 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0319c93a-e08d-4677-9150-96dce21fb5d9-dshm\") pod \"sub-test-model-21f62c-kserve-dd8db588c-h5vrj\" (UID: \"0319c93a-e08d-4677-9150-96dce21fb5d9\") " pod="ai-tenant-e2e-shared-a-21f62c/sub-test-model-21f62c-kserve-dd8db588c-h5vrj" Jul 07 18:17:49.594244 ip-10-0-129-39 kubenswrapper[2575]: I0707 18:17:49.594109 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0319c93a-e08d-4677-9150-96dce21fb5d9-home\") pod \"sub-test-model-21f62c-kserve-dd8db588c-h5vrj\" (UID: \"0319c93a-e08d-4677-9150-96dce21fb5d9\") " pod="ai-tenant-e2e-shared-a-21f62c/sub-test-model-21f62c-kserve-dd8db588c-h5vrj" Jul 07 18:17:49.594244 ip-10-0-129-39 kubenswrapper[2575]: I0707 18:17:49.594165 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0319c93a-e08d-4677-9150-96dce21fb5d9-tls-certs\") pod \"sub-test-model-21f62c-kserve-dd8db588c-h5vrj\" (UID: \"0319c93a-e08d-4677-9150-96dce21fb5d9\") " pod="ai-tenant-e2e-shared-a-21f62c/sub-test-model-21f62c-kserve-dd8db588c-h5vrj" Jul 07 18:17:49.594244 ip-10-0-129-39 kubenswrapper[2575]: I0707 18:17:49.594190 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0319c93a-e08d-4677-9150-96dce21fb5d9-tmp-dir\") pod \"sub-test-model-21f62c-kserve-dd8db588c-h5vrj\" (UID: \"0319c93a-e08d-4677-9150-96dce21fb5d9\") " pod="ai-tenant-e2e-shared-a-21f62c/sub-test-model-21f62c-kserve-dd8db588c-h5vrj" Jul 07 18:17:49.594699 ip-10-0-129-39 kubenswrapper[2575]: I0707 18:17:49.594674 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0319c93a-e08d-4677-9150-96dce21fb5d9-tmp-dir\") pod \"sub-test-model-21f62c-kserve-dd8db588c-h5vrj\" (UID: \"0319c93a-e08d-4677-9150-96dce21fb5d9\") " pod="ai-tenant-e2e-shared-a-21f62c/sub-test-model-21f62c-kserve-dd8db588c-h5vrj" Jul 07 18:17:49.594791 ip-10-0-129-39 kubenswrapper[2575]: I0707 18:17:49.594714 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0319c93a-e08d-4677-9150-96dce21fb5d9-home\") pod \"sub-test-model-21f62c-kserve-dd8db588c-h5vrj\" (UID: \"0319c93a-e08d-4677-9150-96dce21fb5d9\") " pod="ai-tenant-e2e-shared-a-21f62c/sub-test-model-21f62c-kserve-dd8db588c-h5vrj" Jul 07 18:17:49.595014 ip-10-0-129-39 kubenswrapper[2575]: I0707 18:17:49.594988 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0319c93a-e08d-4677-9150-96dce21fb5d9-model-cache\") pod \"sub-test-model-21f62c-kserve-dd8db588c-h5vrj\" (UID: \"0319c93a-e08d-4677-9150-96dce21fb5d9\") " pod="ai-tenant-e2e-shared-a-21f62c/sub-test-model-21f62c-kserve-dd8db588c-h5vrj" Jul 07 18:17:49.597140 ip-10-0-129-39 kubenswrapper[2575]: I0707 18:17:49.597117 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0319c93a-e08d-4677-9150-96dce21fb5d9-dshm\") pod \"sub-test-model-21f62c-kserve-dd8db588c-h5vrj\" (UID: \"0319c93a-e08d-4677-9150-96dce21fb5d9\") " pod="ai-tenant-e2e-shared-a-21f62c/sub-test-model-21f62c-kserve-dd8db588c-h5vrj" Jul 07 18:17:49.597906 ip-10-0-129-39 kubenswrapper[2575]: I0707 18:17:49.597883 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0319c93a-e08d-4677-9150-96dce21fb5d9-tls-certs\") pod \"sub-test-model-21f62c-kserve-dd8db588c-h5vrj\" (UID: \"0319c93a-e08d-4677-9150-96dce21fb5d9\") " pod="ai-tenant-e2e-shared-a-21f62c/sub-test-model-21f62c-kserve-dd8db588c-h5vrj" Jul 07 18:17:49.602549 ip-10-0-129-39 kubenswrapper[2575]: I0707 18:17:49.602526 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8ghct\" (UniqueName: \"kubernetes.io/projected/0319c93a-e08d-4677-9150-96dce21fb5d9-kube-api-access-8ghct\") pod \"sub-test-model-21f62c-kserve-dd8db588c-h5vrj\" (UID: \"0319c93a-e08d-4677-9150-96dce21fb5d9\") " pod="ai-tenant-e2e-shared-a-21f62c/sub-test-model-21f62c-kserve-dd8db588c-h5vrj" Jul 07 18:17:49.709817 ip-10-0-129-39 kubenswrapper[2575]: I0707 18:17:49.709751 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-21f62c/sub-test-model-21f62c-kserve-dd8db588c-h5vrj" Jul 07 18:17:49.845638 ip-10-0-129-39 kubenswrapper[2575]: I0707 18:17:49.845614 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-21f62c/sub-test-model-21f62c-kserve-dd8db588c-h5vrj"] Jul 07 18:17:49.847527 ip-10-0-129-39 kubenswrapper[2575]: W0707 18:17:49.847490 2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod0319c93a_e08d_4677_9150_96dce21fb5d9.slice/crio-3e2d06c0d1b013fe9fcf72670a4a196c856b830b530edaf6c01da2f9c8f3b50a WatchSource:0}: Error finding container 3e2d06c0d1b013fe9fcf72670a4a196c856b830b530edaf6c01da2f9c8f3b50a: Status 404 returned error can't find the container with id 3e2d06c0d1b013fe9fcf72670a4a196c856b830b530edaf6c01da2f9c8f3b50a Jul 07 18:17:50.195790 ip-10-0-129-39 kubenswrapper[2575]: I0707 18:17:50.195756 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-21f62c/sub-test-model-21f62c-kserve-dd8db588c-h5vrj" event={"ID":"0319c93a-e08d-4677-9150-96dce21fb5d9","Type":"ContainerStarted","Data":"9bf7fcd23eec35cea7e1e973dbfdab5da4a5b873ed3ce11c3dcba9756d57f7fe"} Jul 07 18:17:50.195946 ip-10-0-129-39 kubenswrapper[2575]: I0707 18:17:50.195797 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-21f62c/sub-test-model-21f62c-kserve-dd8db588c-h5vrj" event={"ID":"0319c93a-e08d-4677-9150-96dce21fb5d9","Type":"ContainerStarted","Data":"3e2d06c0d1b013fe9fcf72670a4a196c856b830b530edaf6c01da2f9c8f3b50a"} Jul 07 18:17:50.215863 ip-10-0-129-39 kubenswrapper[2575]: I0707 18:17:50.215824 2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-a-21f62c/sub-test-model-21f62c-kserve-dd8db588c-h5vrj" podStartSLOduration=1.215810756 podStartE2EDuration="1.215810756s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-07 18:17:49 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 18:17:50.213460902 +0000 UTC m=+2230.719267569" watchObservedRunningTime="2026-07-07 18:17:50.215810756 +0000 UTC m=+2230.721617419" Jul 07 18:17:58.945866 ip-10-0-129-39 kubenswrapper[2575]: I0707 18:17:58.945817 2575 kuberuntime_container.go:872] "Killing container with a grace period" pod="ai-tenant-e2e-shared-b-fad388/auth-test-model-fad388-kserve-654f46b44c-85szm" podUID="0bab5041-28ce-4a5b-85e9-92cf9acb6ccc" containerName="main" containerID="cri-o://0f38d3beb94c079df2a205539f91e164ce112bd8e5c8c667c7081d0408983180" gracePeriod=45 Jul 07 18:17:59.202048 ip-10-0-129-39 kubenswrapper[2575]: I0707 18:17:59.201989 2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-fad388/auth-test-model-fad388-kserve-654f46b44c-85szm" Jul 07 18:17:59.240981 ip-10-0-129-39 kubenswrapper[2575]: I0707 18:17:59.240951 2575 generic.go:356] "Generic (PLEG): container finished" podID="0bab5041-28ce-4a5b-85e9-92cf9acb6ccc" containerID="0f38d3beb94c079df2a205539f91e164ce112bd8e5c8c667c7081d0408983180" exitCode=0 Jul 07 18:17:59.241139 ip-10-0-129-39 kubenswrapper[2575]: I0707 18:17:59.240989 2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-fad388/auth-test-model-fad388-kserve-654f46b44c-85szm" Jul 07 18:17:59.241139 ip-10-0-129-39 kubenswrapper[2575]: I0707 18:17:59.241022 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-fad388/auth-test-model-fad388-kserve-654f46b44c-85szm" event={"ID":"0bab5041-28ce-4a5b-85e9-92cf9acb6ccc","Type":"ContainerDied","Data":"0f38d3beb94c079df2a205539f91e164ce112bd8e5c8c667c7081d0408983180"} Jul 07 18:17:59.241139 ip-10-0-129-39 kubenswrapper[2575]: I0707 18:17:59.241060 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-fad388/auth-test-model-fad388-kserve-654f46b44c-85szm" event={"ID":"0bab5041-28ce-4a5b-85e9-92cf9acb6ccc","Type":"ContainerDied","Data":"284f4f10a6df2342cdc2f72799d1f1aa4d906fade5e6578c3b2a10511bb09e43"} Jul 07 18:17:59.241139 ip-10-0-129-39 kubenswrapper[2575]: I0707 18:17:59.241082 2575 scope.go:117] "RemoveContainer" containerID="0f38d3beb94c079df2a205539f91e164ce112bd8e5c8c667c7081d0408983180" Jul 07 18:17:59.251463 ip-10-0-129-39 kubenswrapper[2575]: I0707 18:17:59.251437 2575 scope.go:117] "RemoveContainer" containerID="0f38d3beb94c079df2a205539f91e164ce112bd8e5c8c667c7081d0408983180" Jul 07 18:17:59.251923 ip-10-0-129-39 kubenswrapper[2575]: E0707 18:17:59.251894 2575 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0f38d3beb94c079df2a205539f91e164ce112bd8e5c8c667c7081d0408983180\": container with ID starting with 0f38d3beb94c079df2a205539f91e164ce112bd8e5c8c667c7081d0408983180 not found: ID does not exist" containerID="0f38d3beb94c079df2a205539f91e164ce112bd8e5c8c667c7081d0408983180" Jul 07 18:17:59.252022 ip-10-0-129-39 kubenswrapper[2575]: I0707 18:17:59.251930 2575 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0f38d3beb94c079df2a205539f91e164ce112bd8e5c8c667c7081d0408983180"} err="failed to get container status \"0f38d3beb94c079df2a205539f91e164ce112bd8e5c8c667c7081d0408983180\": rpc error: code = NotFound desc = could not find container \"0f38d3beb94c079df2a205539f91e164ce112bd8e5c8c667c7081d0408983180\": container with ID starting with 0f38d3beb94c079df2a205539f91e164ce112bd8e5c8c667c7081d0408983180 not found: ID does not exist" Jul 07 18:17:59.284950 ip-10-0-129-39 kubenswrapper[2575]: I0707 18:17:59.284930 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0bab5041-28ce-4a5b-85e9-92cf9acb6ccc-tls-certs\") pod \"0bab5041-28ce-4a5b-85e9-92cf9acb6ccc\" (UID: \"0bab5041-28ce-4a5b-85e9-92cf9acb6ccc\") " Jul 07 18:17:59.285059 ip-10-0-129-39 kubenswrapper[2575]: I0707 18:17:59.284974 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0bab5041-28ce-4a5b-85e9-92cf9acb6ccc-home\") pod \"0bab5041-28ce-4a5b-85e9-92cf9acb6ccc\" (UID: \"0bab5041-28ce-4a5b-85e9-92cf9acb6ccc\") " Jul 07 18:17:59.285059 ip-10-0-129-39 kubenswrapper[2575]: I0707 18:17:59.285019 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0bab5041-28ce-4a5b-85e9-92cf9acb6ccc-dshm\") pod \"0bab5041-28ce-4a5b-85e9-92cf9acb6ccc\" (UID: \"0bab5041-28ce-4a5b-85e9-92cf9acb6ccc\") " Jul 07 18:17:59.285059 ip-10-0-129-39 kubenswrapper[2575]: I0707 18:17:59.285040 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-hbcrj\" (UniqueName: \"kubernetes.io/projected/0bab5041-28ce-4a5b-85e9-92cf9acb6ccc-kube-api-access-hbcrj\") pod \"0bab5041-28ce-4a5b-85e9-92cf9acb6ccc\" (UID: \"0bab5041-28ce-4a5b-85e9-92cf9acb6ccc\") " Jul 07 18:17:59.285212 ip-10-0-129-39 kubenswrapper[2575]: I0707 18:17:59.285063 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0bab5041-28ce-4a5b-85e9-92cf9acb6ccc-tmp-dir\") pod \"0bab5041-28ce-4a5b-85e9-92cf9acb6ccc\" (UID: \"0bab5041-28ce-4a5b-85e9-92cf9acb6ccc\") " Jul 07 18:17:59.285212 ip-10-0-129-39 kubenswrapper[2575]: I0707 18:17:59.285089 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0bab5041-28ce-4a5b-85e9-92cf9acb6ccc-model-cache\") pod \"0bab5041-28ce-4a5b-85e9-92cf9acb6ccc\" (UID: \"0bab5041-28ce-4a5b-85e9-92cf9acb6ccc\") " Jul 07 18:17:59.285362 ip-10-0-129-39 kubenswrapper[2575]: I0707 18:17:59.285333 2575 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0bab5041-28ce-4a5b-85e9-92cf9acb6ccc-home" (OuterVolumeSpecName: "home") pod "0bab5041-28ce-4a5b-85e9-92cf9acb6ccc" (UID: "0bab5041-28ce-4a5b-85e9-92cf9acb6ccc"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 07 18:17:59.285462 ip-10-0-129-39 kubenswrapper[2575]: I0707 18:17:59.285347 2575 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0bab5041-28ce-4a5b-85e9-92cf9acb6ccc-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "0bab5041-28ce-4a5b-85e9-92cf9acb6ccc" (UID: "0bab5041-28ce-4a5b-85e9-92cf9acb6ccc"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 07 18:17:59.285462 ip-10-0-129-39 kubenswrapper[2575]: I0707 18:17:59.285428 2575 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0bab5041-28ce-4a5b-85e9-92cf9acb6ccc-model-cache" (OuterVolumeSpecName: "model-cache") pod "0bab5041-28ce-4a5b-85e9-92cf9acb6ccc" (UID: "0bab5041-28ce-4a5b-85e9-92cf9acb6ccc"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 07 18:17:59.287001 ip-10-0-129-39 kubenswrapper[2575]: I0707 18:17:59.286973 2575 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0bab5041-28ce-4a5b-85e9-92cf9acb6ccc-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "0bab5041-28ce-4a5b-85e9-92cf9acb6ccc" (UID: "0bab5041-28ce-4a5b-85e9-92cf9acb6ccc"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 07 18:17:59.287102 ip-10-0-129-39 kubenswrapper[2575]: I0707 18:17:59.287078 2575 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0bab5041-28ce-4a5b-85e9-92cf9acb6ccc-kube-api-access-hbcrj" (OuterVolumeSpecName: "kube-api-access-hbcrj") pod "0bab5041-28ce-4a5b-85e9-92cf9acb6ccc" (UID: "0bab5041-28ce-4a5b-85e9-92cf9acb6ccc"). InnerVolumeSpecName "kube-api-access-hbcrj". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 07 18:17:59.287102 ip-10-0-129-39 kubenswrapper[2575]: I0707 18:17:59.287090 2575 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0bab5041-28ce-4a5b-85e9-92cf9acb6ccc-dshm" (OuterVolumeSpecName: "dshm") pod "0bab5041-28ce-4a5b-85e9-92cf9acb6ccc" (UID: "0bab5041-28ce-4a5b-85e9-92cf9acb6ccc"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 07 18:17:59.385711 ip-10-0-129-39 kubenswrapper[2575]: I0707 18:17:59.385683 2575 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0bab5041-28ce-4a5b-85e9-92cf9acb6ccc-dshm\") on node \"ip-10-0-129-39.ec2.internal\" DevicePath \"\"" Jul 07 18:17:59.385711 ip-10-0-129-39 kubenswrapper[2575]: I0707 18:17:59.385708 2575 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hbcrj\" (UniqueName: \"kubernetes.io/projected/0bab5041-28ce-4a5b-85e9-92cf9acb6ccc-kube-api-access-hbcrj\") on node \"ip-10-0-129-39.ec2.internal\" DevicePath \"\"" Jul 07 18:17:59.385869 ip-10-0-129-39 kubenswrapper[2575]: I0707 18:17:59.385722 2575 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0bab5041-28ce-4a5b-85e9-92cf9acb6ccc-tmp-dir\") on node \"ip-10-0-129-39.ec2.internal\" DevicePath \"\"" Jul 07 18:17:59.385869 ip-10-0-129-39 kubenswrapper[2575]: I0707 18:17:59.385734 2575 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0bab5041-28ce-4a5b-85e9-92cf9acb6ccc-model-cache\") on node \"ip-10-0-129-39.ec2.internal\" DevicePath \"\"" Jul 07 18:17:59.385869 ip-10-0-129-39 kubenswrapper[2575]: I0707 18:17:59.385744 2575 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0bab5041-28ce-4a5b-85e9-92cf9acb6ccc-tls-certs\") on node \"ip-10-0-129-39.ec2.internal\" DevicePath \"\"" Jul 07 18:17:59.385869 ip-10-0-129-39 kubenswrapper[2575]: I0707 18:17:59.385754 2575 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0bab5041-28ce-4a5b-85e9-92cf9acb6ccc-home\") on node \"ip-10-0-129-39.ec2.internal\" DevicePath \"\"" Jul 07 18:17:59.563786 ip-10-0-129-39 kubenswrapper[2575]: I0707 18:17:59.563750 2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-fad388/auth-test-model-fad388-kserve-654f46b44c-85szm"] Jul 07 18:17:59.567984 ip-10-0-129-39 kubenswrapper[2575]: I0707 18:17:59.567963 2575 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-b-fad388/auth-test-model-fad388-kserve-654f46b44c-85szm"] Jul 07 18:17:59.709868 ip-10-0-129-39 kubenswrapper[2575]: I0707 18:17:59.709836 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-a-21f62c/sub-test-model-21f62c-kserve-dd8db588c-h5vrj" Jul 07 18:17:59.710005 ip-10-0-129-39 kubenswrapper[2575]: I0707 18:17:59.709874 2575 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-a-21f62c/sub-test-model-21f62c-kserve-dd8db588c-h5vrj" Jul 07 18:17:59.722338 ip-10-0-129-39 kubenswrapper[2575]: I0707 18:17:59.722319 2575 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-a-21f62c/sub-test-model-21f62c-kserve-dd8db588c-h5vrj" Jul 07 18:18:00.074546 ip-10-0-129-39 kubenswrapper[2575]: I0707 18:18:00.074492 2575 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="0bab5041-28ce-4a5b-85e9-92cf9acb6ccc" path="/var/lib/kubelet/pods/0bab5041-28ce-4a5b-85e9-92cf9acb6ccc/volumes" Jul 07 18:18:00.259538 ip-10-0-129-39 kubenswrapper[2575]: I0707 18:18:00.259244 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-a-21f62c/sub-test-model-21f62c-kserve-dd8db588c-h5vrj" Jul 07 18:18:00.786769 ip-10-0-129-39 kubenswrapper[2575]: I0707 18:18:00.786736 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-m825v"] Jul 07 18:18:26.993175 ip-10-0-129-39 kubenswrapper[2575]: I0707 18:18:26.993096 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-m825v"] Jul 07 18:18:41.585257 ip-10-0-129-39 kubenswrapper[2575]: I0707 18:18:41.585225 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-m825v"] Jul 07 18:18:55.465119 ip-10-0-129-39 kubenswrapper[2575]: I0707 18:18:55.465089 2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-21f62c/sub-test-model-21f62c-kserve-dd8db588c-h5vrj"] Jul 07 18:18:59.583048 ip-10-0-129-39 kubenswrapper[2575]: I0707 18:18:59.583015 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-m825v"] Jul 07 18:19:10.485201 ip-10-0-129-39 kubenswrapper[2575]: I0707 18:19:10.485134 2575 kuberuntime_container.go:872] "Killing container with a grace period" pod="ai-tenant-e2e-shared-a-21f62c/sub-test-model-21f62c-kserve-dd8db588c-h5vrj" podUID="0319c93a-e08d-4677-9150-96dce21fb5d9" containerName="main" containerID="cri-o://9bf7fcd23eec35cea7e1e973dbfdab5da4a5b873ed3ce11c3dcba9756d57f7fe" gracePeriod=45 Jul 07 18:19:10.740139 ip-10-0-129-39 kubenswrapper[2575]: I0707 18:19:10.740084 2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-21f62c/sub-test-model-21f62c-kserve-dd8db588c-h5vrj" Jul 07 18:19:10.779536 ip-10-0-129-39 kubenswrapper[2575]: I0707 18:19:10.779488 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0319c93a-e08d-4677-9150-96dce21fb5d9-tls-certs\") pod \"0319c93a-e08d-4677-9150-96dce21fb5d9\" (UID: \"0319c93a-e08d-4677-9150-96dce21fb5d9\") " Jul 07 18:19:10.779684 ip-10-0-129-39 kubenswrapper[2575]: I0707 18:19:10.779603 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-8ghct\" (UniqueName: \"kubernetes.io/projected/0319c93a-e08d-4677-9150-96dce21fb5d9-kube-api-access-8ghct\") pod \"0319c93a-e08d-4677-9150-96dce21fb5d9\" (UID: \"0319c93a-e08d-4677-9150-96dce21fb5d9\") " Jul 07 18:19:10.779684 ip-10-0-129-39 kubenswrapper[2575]: I0707 18:19:10.779635 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0319c93a-e08d-4677-9150-96dce21fb5d9-home\") pod \"0319c93a-e08d-4677-9150-96dce21fb5d9\" (UID: \"0319c93a-e08d-4677-9150-96dce21fb5d9\") " Jul 07 18:19:10.779684 ip-10-0-129-39 kubenswrapper[2575]: I0707 18:19:10.779673 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0319c93a-e08d-4677-9150-96dce21fb5d9-model-cache\") pod \"0319c93a-e08d-4677-9150-96dce21fb5d9\" (UID: \"0319c93a-e08d-4677-9150-96dce21fb5d9\") " Jul 07 18:19:10.779845 ip-10-0-129-39 kubenswrapper[2575]: I0707 18:19:10.779700 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0319c93a-e08d-4677-9150-96dce21fb5d9-dshm\") pod \"0319c93a-e08d-4677-9150-96dce21fb5d9\" (UID: \"0319c93a-e08d-4677-9150-96dce21fb5d9\") " Jul 07 18:19:10.779845 ip-10-0-129-39 kubenswrapper[2575]: I0707 18:19:10.779730 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0319c93a-e08d-4677-9150-96dce21fb5d9-tmp-dir\") pod \"0319c93a-e08d-4677-9150-96dce21fb5d9\" (UID: \"0319c93a-e08d-4677-9150-96dce21fb5d9\") " Jul 07 18:19:10.779935 ip-10-0-129-39 kubenswrapper[2575]: I0707 18:19:10.779915 2575 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0319c93a-e08d-4677-9150-96dce21fb5d9-home" (OuterVolumeSpecName: "home") pod "0319c93a-e08d-4677-9150-96dce21fb5d9" (UID: "0319c93a-e08d-4677-9150-96dce21fb5d9"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 07 18:19:10.779980 ip-10-0-129-39 kubenswrapper[2575]: I0707 18:19:10.779930 2575 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0319c93a-e08d-4677-9150-96dce21fb5d9-model-cache" (OuterVolumeSpecName: "model-cache") pod "0319c93a-e08d-4677-9150-96dce21fb5d9" (UID: "0319c93a-e08d-4677-9150-96dce21fb5d9"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 07 18:19:10.780097 ip-10-0-129-39 kubenswrapper[2575]: I0707 18:19:10.780078 2575 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0319c93a-e08d-4677-9150-96dce21fb5d9-home\") on node \"ip-10-0-129-39.ec2.internal\" DevicePath \"\"" Jul 07 18:19:10.780144 ip-10-0-129-39 kubenswrapper[2575]: I0707 18:19:10.780104 2575 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0319c93a-e08d-4677-9150-96dce21fb5d9-model-cache\") on node \"ip-10-0-129-39.ec2.internal\" DevicePath \"\"" Jul 07 18:19:10.780144 ip-10-0-129-39 kubenswrapper[2575]: I0707 18:19:10.780112 2575 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0319c93a-e08d-4677-9150-96dce21fb5d9-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "0319c93a-e08d-4677-9150-96dce21fb5d9" (UID: "0319c93a-e08d-4677-9150-96dce21fb5d9"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 07 18:19:10.781798 ip-10-0-129-39 kubenswrapper[2575]: I0707 18:19:10.781771 2575 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0319c93a-e08d-4677-9150-96dce21fb5d9-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "0319c93a-e08d-4677-9150-96dce21fb5d9" (UID: "0319c93a-e08d-4677-9150-96dce21fb5d9"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 07 18:19:10.781798 ip-10-0-129-39 kubenswrapper[2575]: I0707 18:19:10.781770 2575 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0319c93a-e08d-4677-9150-96dce21fb5d9-dshm" (OuterVolumeSpecName: "dshm") pod "0319c93a-e08d-4677-9150-96dce21fb5d9" (UID: "0319c93a-e08d-4677-9150-96dce21fb5d9"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 07 18:19:10.781977 ip-10-0-129-39 kubenswrapper[2575]: I0707 18:19:10.781886 2575 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0319c93a-e08d-4677-9150-96dce21fb5d9-kube-api-access-8ghct" (OuterVolumeSpecName: "kube-api-access-8ghct") pod "0319c93a-e08d-4677-9150-96dce21fb5d9" (UID: "0319c93a-e08d-4677-9150-96dce21fb5d9"). InnerVolumeSpecName "kube-api-access-8ghct". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 07 18:19:10.881257 ip-10-0-129-39 kubenswrapper[2575]: I0707 18:19:10.881231 2575 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0319c93a-e08d-4677-9150-96dce21fb5d9-dshm\") on node \"ip-10-0-129-39.ec2.internal\" DevicePath \"\"" Jul 07 18:19:10.881257 ip-10-0-129-39 kubenswrapper[2575]: I0707 18:19:10.881255 2575 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0319c93a-e08d-4677-9150-96dce21fb5d9-tmp-dir\") on node \"ip-10-0-129-39.ec2.internal\" DevicePath \"\"" Jul 07 18:19:10.881418 ip-10-0-129-39 kubenswrapper[2575]: I0707 18:19:10.881264 2575 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0319c93a-e08d-4677-9150-96dce21fb5d9-tls-certs\") on node \"ip-10-0-129-39.ec2.internal\" DevicePath \"\"" Jul 07 18:19:10.881418 ip-10-0-129-39 kubenswrapper[2575]: I0707 18:19:10.881278 2575 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8ghct\" (UniqueName: \"kubernetes.io/projected/0319c93a-e08d-4677-9150-96dce21fb5d9-kube-api-access-8ghct\") on node \"ip-10-0-129-39.ec2.internal\" DevicePath \"\"" Jul 07 18:19:11.584776 ip-10-0-129-39 kubenswrapper[2575]: I0707 18:19:11.584745 2575 generic.go:356] "Generic (PLEG): container finished" podID="0319c93a-e08d-4677-9150-96dce21fb5d9" containerID="9bf7fcd23eec35cea7e1e973dbfdab5da4a5b873ed3ce11c3dcba9756d57f7fe" exitCode=0 Jul 07 18:19:11.585235 ip-10-0-129-39 kubenswrapper[2575]: I0707 18:19:11.584808 2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-21f62c/sub-test-model-21f62c-kserve-dd8db588c-h5vrj" Jul 07 18:19:11.585235 ip-10-0-129-39 kubenswrapper[2575]: I0707 18:19:11.584839 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-21f62c/sub-test-model-21f62c-kserve-dd8db588c-h5vrj" event={"ID":"0319c93a-e08d-4677-9150-96dce21fb5d9","Type":"ContainerDied","Data":"9bf7fcd23eec35cea7e1e973dbfdab5da4a5b873ed3ce11c3dcba9756d57f7fe"} Jul 07 18:19:11.585235 ip-10-0-129-39 kubenswrapper[2575]: I0707 18:19:11.584894 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-21f62c/sub-test-model-21f62c-kserve-dd8db588c-h5vrj" event={"ID":"0319c93a-e08d-4677-9150-96dce21fb5d9","Type":"ContainerDied","Data":"3e2d06c0d1b013fe9fcf72670a4a196c856b830b530edaf6c01da2f9c8f3b50a"} Jul 07 18:19:11.585235 ip-10-0-129-39 kubenswrapper[2575]: I0707 18:19:11.584920 2575 scope.go:117] "RemoveContainer" containerID="9bf7fcd23eec35cea7e1e973dbfdab5da4a5b873ed3ce11c3dcba9756d57f7fe" Jul 07 18:19:11.593644 ip-10-0-129-39 kubenswrapper[2575]: I0707 18:19:11.593625 2575 scope.go:117] "RemoveContainer" containerID="9bf7fcd23eec35cea7e1e973dbfdab5da4a5b873ed3ce11c3dcba9756d57f7fe" Jul 07 18:19:11.593900 ip-10-0-129-39 kubenswrapper[2575]: E0707 18:19:11.593881 2575 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9bf7fcd23eec35cea7e1e973dbfdab5da4a5b873ed3ce11c3dcba9756d57f7fe\": container with ID starting with 9bf7fcd23eec35cea7e1e973dbfdab5da4a5b873ed3ce11c3dcba9756d57f7fe not found: ID does not exist" containerID="9bf7fcd23eec35cea7e1e973dbfdab5da4a5b873ed3ce11c3dcba9756d57f7fe" Jul 07 18:19:11.593967 ip-10-0-129-39 kubenswrapper[2575]: I0707 18:19:11.593911 2575 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9bf7fcd23eec35cea7e1e973dbfdab5da4a5b873ed3ce11c3dcba9756d57f7fe"} err="failed to get container status \"9bf7fcd23eec35cea7e1e973dbfdab5da4a5b873ed3ce11c3dcba9756d57f7fe\": rpc error: code = NotFound desc = could not find container \"9bf7fcd23eec35cea7e1e973dbfdab5da4a5b873ed3ce11c3dcba9756d57f7fe\": container with ID starting with 9bf7fcd23eec35cea7e1e973dbfdab5da4a5b873ed3ce11c3dcba9756d57f7fe not found: ID does not exist" Jul 07 18:19:11.609754 ip-10-0-129-39 kubenswrapper[2575]: I0707 18:19:11.609733 2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-21f62c/sub-test-model-21f62c-kserve-dd8db588c-h5vrj"] Jul 07 18:19:11.614062 ip-10-0-129-39 kubenswrapper[2575]: I0707 18:19:11.614045 2575 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-a-21f62c/sub-test-model-21f62c-kserve-dd8db588c-h5vrj"] Jul 07 18:19:12.073852 ip-10-0-129-39 kubenswrapper[2575]: I0707 18:19:12.073826 2575 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="0319c93a-e08d-4677-9150-96dce21fb5d9" path="/var/lib/kubelet/pods/0319c93a-e08d-4677-9150-96dce21fb5d9/volumes" Jul 07 18:19:19.987662 ip-10-0-129-39 kubenswrapper[2575]: I0707 18:19:19.987629 2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-a-21f62c/rate-test-model-21f62c-kserve-6954b87d59-hjwbw"] Jul 07 18:19:19.988154 ip-10-0-129-39 kubenswrapper[2575]: I0707 18:19:19.988134 2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0319c93a-e08d-4677-9150-96dce21fb5d9" containerName="main" Jul 07 18:19:19.988199 ip-10-0-129-39 kubenswrapper[2575]: I0707 18:19:19.988158 2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="0319c93a-e08d-4677-9150-96dce21fb5d9" containerName="main" Jul 07 18:19:19.988199 ip-10-0-129-39 kubenswrapper[2575]: I0707 18:19:19.988181 2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0bab5041-28ce-4a5b-85e9-92cf9acb6ccc" containerName="main" Jul 07 18:19:19.988199 ip-10-0-129-39 kubenswrapper[2575]: I0707 18:19:19.988190 2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="0bab5041-28ce-4a5b-85e9-92cf9acb6ccc" containerName="main" Jul 07 18:19:19.988293 ip-10-0-129-39 kubenswrapper[2575]: I0707 18:19:19.988284 2575 memory_manager.go:368] "RemoveStaleState removing state" podUID="0319c93a-e08d-4677-9150-96dce21fb5d9" containerName="main" Jul 07 18:19:19.988326 ip-10-0-129-39 kubenswrapper[2575]: I0707 18:19:19.988302 2575 memory_manager.go:368] "RemoveStaleState removing state" podUID="0bab5041-28ce-4a5b-85e9-92cf9acb6ccc" containerName="main" Jul 07 18:19:19.991450 ip-10-0-129-39 kubenswrapper[2575]: I0707 18:19:19.991432 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-21f62c/rate-test-model-21f62c-kserve-6954b87d59-hjwbw" Jul 07 18:19:19.994878 ip-10-0-129-39 kubenswrapper[2575]: I0707 18:19:19.994857 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-21f62c\"/\"rate-test-model-21f62c-kserve-self-signed-certs\"" Jul 07 18:19:19.995813 ip-10-0-129-39 kubenswrapper[2575]: I0707 18:19:19.995790 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-21f62c\"/\"default-dockercfg-xvp6n\"" Jul 07 18:19:19.995919 ip-10-0-129-39 kubenswrapper[2575]: I0707 18:19:19.995794 2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-21f62c\"/\"openshift-service-ca.crt\"" Jul 07 18:19:19.995919 ip-10-0-129-39 kubenswrapper[2575]: I0707 18:19:19.995861 2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-21f62c\"/\"kube-root-ca.crt\"" Jul 07 18:19:20.001486 ip-10-0-129-39 kubenswrapper[2575]: I0707 18:19:20.001467 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-21f62c/rate-test-model-21f62c-kserve-6954b87d59-hjwbw"] Jul 07 18:19:20.054553 ip-10-0-129-39 kubenswrapper[2575]: I0707 18:19:20.054496 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/28e6fd0e-6029-4bc7-87b7-f220986dd689-dshm\") pod \"rate-test-model-21f62c-kserve-6954b87d59-hjwbw\" (UID: \"28e6fd0e-6029-4bc7-87b7-f220986dd689\") " pod="ai-tenant-e2e-shared-a-21f62c/rate-test-model-21f62c-kserve-6954b87d59-hjwbw" Jul 07 18:19:20.054691 ip-10-0-129-39 kubenswrapper[2575]: I0707 18:19:20.054557 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/28e6fd0e-6029-4bc7-87b7-f220986dd689-home\") pod \"rate-test-model-21f62c-kserve-6954b87d59-hjwbw\" (UID: \"28e6fd0e-6029-4bc7-87b7-f220986dd689\") " pod="ai-tenant-e2e-shared-a-21f62c/rate-test-model-21f62c-kserve-6954b87d59-hjwbw" Jul 07 18:19:20.054691 ip-10-0-129-39 kubenswrapper[2575]: I0707 18:19:20.054670 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/28e6fd0e-6029-4bc7-87b7-f220986dd689-tls-certs\") pod \"rate-test-model-21f62c-kserve-6954b87d59-hjwbw\" (UID: \"28e6fd0e-6029-4bc7-87b7-f220986dd689\") " pod="ai-tenant-e2e-shared-a-21f62c/rate-test-model-21f62c-kserve-6954b87d59-hjwbw" Jul 07 18:19:20.054768 ip-10-0-129-39 kubenswrapper[2575]: I0707 18:19:20.054699 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-slqxl\" (UniqueName: \"kubernetes.io/projected/28e6fd0e-6029-4bc7-87b7-f220986dd689-kube-api-access-slqxl\") pod \"rate-test-model-21f62c-kserve-6954b87d59-hjwbw\" (UID: \"28e6fd0e-6029-4bc7-87b7-f220986dd689\") " pod="ai-tenant-e2e-shared-a-21f62c/rate-test-model-21f62c-kserve-6954b87d59-hjwbw" Jul 07 18:19:20.054807 ip-10-0-129-39 kubenswrapper[2575]: I0707 18:19:20.054793 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/28e6fd0e-6029-4bc7-87b7-f220986dd689-model-cache\") pod \"rate-test-model-21f62c-kserve-6954b87d59-hjwbw\" (UID: \"28e6fd0e-6029-4bc7-87b7-f220986dd689\") " pod="ai-tenant-e2e-shared-a-21f62c/rate-test-model-21f62c-kserve-6954b87d59-hjwbw" Jul 07 18:19:20.054843 ip-10-0-129-39 kubenswrapper[2575]: I0707 18:19:20.054822 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/28e6fd0e-6029-4bc7-87b7-f220986dd689-tmp-dir\") pod \"rate-test-model-21f62c-kserve-6954b87d59-hjwbw\" (UID: \"28e6fd0e-6029-4bc7-87b7-f220986dd689\") " pod="ai-tenant-e2e-shared-a-21f62c/rate-test-model-21f62c-kserve-6954b87d59-hjwbw" Jul 07 18:19:20.155421 ip-10-0-129-39 kubenswrapper[2575]: I0707 18:19:20.155386 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/28e6fd0e-6029-4bc7-87b7-f220986dd689-tls-certs\") pod \"rate-test-model-21f62c-kserve-6954b87d59-hjwbw\" (UID: \"28e6fd0e-6029-4bc7-87b7-f220986dd689\") " pod="ai-tenant-e2e-shared-a-21f62c/rate-test-model-21f62c-kserve-6954b87d59-hjwbw" Jul 07 18:19:20.155421 ip-10-0-129-39 kubenswrapper[2575]: I0707 18:19:20.155434 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-slqxl\" (UniqueName: \"kubernetes.io/projected/28e6fd0e-6029-4bc7-87b7-f220986dd689-kube-api-access-slqxl\") pod \"rate-test-model-21f62c-kserve-6954b87d59-hjwbw\" (UID: \"28e6fd0e-6029-4bc7-87b7-f220986dd689\") " pod="ai-tenant-e2e-shared-a-21f62c/rate-test-model-21f62c-kserve-6954b87d59-hjwbw" Jul 07 18:19:20.155716 ip-10-0-129-39 kubenswrapper[2575]: I0707 18:19:20.155495 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/28e6fd0e-6029-4bc7-87b7-f220986dd689-model-cache\") pod \"rate-test-model-21f62c-kserve-6954b87d59-hjwbw\" (UID: \"28e6fd0e-6029-4bc7-87b7-f220986dd689\") " pod="ai-tenant-e2e-shared-a-21f62c/rate-test-model-21f62c-kserve-6954b87d59-hjwbw" Jul 07 18:19:20.155716 ip-10-0-129-39 kubenswrapper[2575]: I0707 18:19:20.155547 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/28e6fd0e-6029-4bc7-87b7-f220986dd689-tmp-dir\") pod \"rate-test-model-21f62c-kserve-6954b87d59-hjwbw\" (UID: \"28e6fd0e-6029-4bc7-87b7-f220986dd689\") " pod="ai-tenant-e2e-shared-a-21f62c/rate-test-model-21f62c-kserve-6954b87d59-hjwbw" Jul 07 18:19:20.155716 ip-10-0-129-39 kubenswrapper[2575]: I0707 18:19:20.155582 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/28e6fd0e-6029-4bc7-87b7-f220986dd689-dshm\") pod \"rate-test-model-21f62c-kserve-6954b87d59-hjwbw\" (UID: \"28e6fd0e-6029-4bc7-87b7-f220986dd689\") " pod="ai-tenant-e2e-shared-a-21f62c/rate-test-model-21f62c-kserve-6954b87d59-hjwbw" Jul 07 18:19:20.155716 ip-10-0-129-39 kubenswrapper[2575]: I0707 18:19:20.155614 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/28e6fd0e-6029-4bc7-87b7-f220986dd689-home\") pod \"rate-test-model-21f62c-kserve-6954b87d59-hjwbw\" (UID: \"28e6fd0e-6029-4bc7-87b7-f220986dd689\") " pod="ai-tenant-e2e-shared-a-21f62c/rate-test-model-21f62c-kserve-6954b87d59-hjwbw" Jul 07 18:19:20.156163 ip-10-0-129-39 kubenswrapper[2575]: I0707 18:19:20.156104 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/28e6fd0e-6029-4bc7-87b7-f220986dd689-home\") pod \"rate-test-model-21f62c-kserve-6954b87d59-hjwbw\" (UID: \"28e6fd0e-6029-4bc7-87b7-f220986dd689\") " pod="ai-tenant-e2e-shared-a-21f62c/rate-test-model-21f62c-kserve-6954b87d59-hjwbw" Jul 07 18:19:20.156414 ip-10-0-129-39 kubenswrapper[2575]: I0707 18:19:20.156390 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/28e6fd0e-6029-4bc7-87b7-f220986dd689-tmp-dir\") pod \"rate-test-model-21f62c-kserve-6954b87d59-hjwbw\" (UID: \"28e6fd0e-6029-4bc7-87b7-f220986dd689\") " pod="ai-tenant-e2e-shared-a-21f62c/rate-test-model-21f62c-kserve-6954b87d59-hjwbw" Jul 07 18:19:20.156616 ip-10-0-129-39 kubenswrapper[2575]: I0707 18:19:20.156593 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/28e6fd0e-6029-4bc7-87b7-f220986dd689-model-cache\") pod \"rate-test-model-21f62c-kserve-6954b87d59-hjwbw\" (UID: \"28e6fd0e-6029-4bc7-87b7-f220986dd689\") " pod="ai-tenant-e2e-shared-a-21f62c/rate-test-model-21f62c-kserve-6954b87d59-hjwbw" Jul 07 18:19:20.159290 ip-10-0-129-39 kubenswrapper[2575]: I0707 18:19:20.159261 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/28e6fd0e-6029-4bc7-87b7-f220986dd689-tls-certs\") pod \"rate-test-model-21f62c-kserve-6954b87d59-hjwbw\" (UID: \"28e6fd0e-6029-4bc7-87b7-f220986dd689\") " pod="ai-tenant-e2e-shared-a-21f62c/rate-test-model-21f62c-kserve-6954b87d59-hjwbw" Jul 07 18:19:20.159531 ip-10-0-129-39 kubenswrapper[2575]: I0707 18:19:20.159491 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/28e6fd0e-6029-4bc7-87b7-f220986dd689-dshm\") pod \"rate-test-model-21f62c-kserve-6954b87d59-hjwbw\" (UID: \"28e6fd0e-6029-4bc7-87b7-f220986dd689\") " pod="ai-tenant-e2e-shared-a-21f62c/rate-test-model-21f62c-kserve-6954b87d59-hjwbw" Jul 07 18:19:20.164262 ip-10-0-129-39 kubenswrapper[2575]: I0707 18:19:20.164243 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-slqxl\" (UniqueName: \"kubernetes.io/projected/28e6fd0e-6029-4bc7-87b7-f220986dd689-kube-api-access-slqxl\") pod \"rate-test-model-21f62c-kserve-6954b87d59-hjwbw\" (UID: \"28e6fd0e-6029-4bc7-87b7-f220986dd689\") " pod="ai-tenant-e2e-shared-a-21f62c/rate-test-model-21f62c-kserve-6954b87d59-hjwbw" Jul 07 18:19:20.302775 ip-10-0-129-39 kubenswrapper[2575]: I0707 18:19:20.302702 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-21f62c/rate-test-model-21f62c-kserve-6954b87d59-hjwbw" Jul 07 18:19:20.439336 ip-10-0-129-39 kubenswrapper[2575]: I0707 18:19:20.439303 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-21f62c/rate-test-model-21f62c-kserve-6954b87d59-hjwbw"] Jul 07 18:19:20.440251 ip-10-0-129-39 kubenswrapper[2575]: W0707 18:19:20.440223 2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod28e6fd0e_6029_4bc7_87b7_f220986dd689.slice/crio-07c741e07cac1cf91dfc643eb6bb5f69355aee1d8dbff3e1c8cda1ce6d5a21be WatchSource:0}: Error finding container 07c741e07cac1cf91dfc643eb6bb5f69355aee1d8dbff3e1c8cda1ce6d5a21be: Status 404 returned error can't find the container with id 07c741e07cac1cf91dfc643eb6bb5f69355aee1d8dbff3e1c8cda1ce6d5a21be Jul 07 18:19:20.629577 ip-10-0-129-39 kubenswrapper[2575]: I0707 18:19:20.629543 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-21f62c/rate-test-model-21f62c-kserve-6954b87d59-hjwbw" event={"ID":"28e6fd0e-6029-4bc7-87b7-f220986dd689","Type":"ContainerStarted","Data":"2d6b17c37ae406a191254c7e7c441669d0c6301d2cbefdbada5bf351623494a8"} Jul 07 18:19:20.629735 ip-10-0-129-39 kubenswrapper[2575]: I0707 18:19:20.629584 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-21f62c/rate-test-model-21f62c-kserve-6954b87d59-hjwbw" event={"ID":"28e6fd0e-6029-4bc7-87b7-f220986dd689","Type":"ContainerStarted","Data":"07c741e07cac1cf91dfc643eb6bb5f69355aee1d8dbff3e1c8cda1ce6d5a21be"} Jul 07 18:19:20.647441 ip-10-0-129-39 kubenswrapper[2575]: I0707 18:19:20.647398 2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-a-21f62c/rate-test-model-21f62c-kserve-6954b87d59-hjwbw" podStartSLOduration=1.6473844579999999 podStartE2EDuration="1.647384458s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-07 18:19:19 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 18:19:20.646343502 +0000 UTC m=+2321.152150169" watchObservedRunningTime="2026-07-07 18:19:20.647384458 +0000 UTC m=+2321.153191122" Jul 07 18:19:23.190430 ip-10-0-129-39 kubenswrapper[2575]: I0707 18:19:23.190394 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-m825v"] Jul 07 18:19:30.303268 ip-10-0-129-39 kubenswrapper[2575]: I0707 18:19:30.303236 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-a-21f62c/rate-test-model-21f62c-kserve-6954b87d59-hjwbw" Jul 07 18:19:30.303689 ip-10-0-129-39 kubenswrapper[2575]: I0707 18:19:30.303283 2575 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-a-21f62c/rate-test-model-21f62c-kserve-6954b87d59-hjwbw" Jul 07 18:19:30.315683 ip-10-0-129-39 kubenswrapper[2575]: I0707 18:19:30.315659 2575 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-a-21f62c/rate-test-model-21f62c-kserve-6954b87d59-hjwbw" Jul 07 18:19:30.687011 ip-10-0-129-39 kubenswrapper[2575]: I0707 18:19:30.686984 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-a-21f62c/rate-test-model-21f62c-kserve-6954b87d59-hjwbw" Jul 07 18:19:40.892931 ip-10-0-129-39 kubenswrapper[2575]: I0707 18:19:40.892900 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-m825v"] Jul 07 18:20:03.804382 ip-10-0-129-39 kubenswrapper[2575]: I0707 18:20:03.804347 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-m825v"] Jul 07 18:20:17.582652 ip-10-0-129-39 kubenswrapper[2575]: I0707 18:20:17.582607 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-m825v"] Jul 07 18:20:23.529323 ip-10-0-129-39 kubenswrapper[2575]: I0707 18:20:23.529291 2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-21f62c/rate-test-model-21f62c-kserve-6954b87d59-hjwbw"] Jul 07 18:20:35.489403 ip-10-0-129-39 kubenswrapper[2575]: I0707 18:20:35.489368 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-m825v"] Jul 07 18:20:38.548889 ip-10-0-129-39 kubenswrapper[2575]: I0707 18:20:38.548848 2575 kuberuntime_container.go:872] "Killing container with a grace period" pod="ai-tenant-e2e-shared-a-21f62c/rate-test-model-21f62c-kserve-6954b87d59-hjwbw" podUID="28e6fd0e-6029-4bc7-87b7-f220986dd689" containerName="main" containerID="cri-o://2d6b17c37ae406a191254c7e7c441669d0c6301d2cbefdbada5bf351623494a8" gracePeriod=45 Jul 07 18:20:38.806971 ip-10-0-129-39 kubenswrapper[2575]: I0707 18:20:38.806910 2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-21f62c/rate-test-model-21f62c-kserve-6954b87d59-hjwbw" Jul 07 18:20:38.927374 ip-10-0-129-39 kubenswrapper[2575]: I0707 18:20:38.927346 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/28e6fd0e-6029-4bc7-87b7-f220986dd689-dshm\") pod \"28e6fd0e-6029-4bc7-87b7-f220986dd689\" (UID: \"28e6fd0e-6029-4bc7-87b7-f220986dd689\") " Jul 07 18:20:38.927575 ip-10-0-129-39 kubenswrapper[2575]: I0707 18:20:38.927387 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/28e6fd0e-6029-4bc7-87b7-f220986dd689-model-cache\") pod \"28e6fd0e-6029-4bc7-87b7-f220986dd689\" (UID: \"28e6fd0e-6029-4bc7-87b7-f220986dd689\") " Jul 07 18:20:38.927575 ip-10-0-129-39 kubenswrapper[2575]: I0707 18:20:38.927429 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/28e6fd0e-6029-4bc7-87b7-f220986dd689-home\") pod \"28e6fd0e-6029-4bc7-87b7-f220986dd689\" (UID: \"28e6fd0e-6029-4bc7-87b7-f220986dd689\") " Jul 07 18:20:38.927575 ip-10-0-129-39 kubenswrapper[2575]: I0707 18:20:38.927455 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/28e6fd0e-6029-4bc7-87b7-f220986dd689-tls-certs\") pod \"28e6fd0e-6029-4bc7-87b7-f220986dd689\" (UID: \"28e6fd0e-6029-4bc7-87b7-f220986dd689\") " Jul 07 18:20:38.927575 ip-10-0-129-39 kubenswrapper[2575]: I0707 18:20:38.927567 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-slqxl\" (UniqueName: \"kubernetes.io/projected/28e6fd0e-6029-4bc7-87b7-f220986dd689-kube-api-access-slqxl\") pod \"28e6fd0e-6029-4bc7-87b7-f220986dd689\" (UID: \"28e6fd0e-6029-4bc7-87b7-f220986dd689\") " Jul 07 18:20:38.927815 ip-10-0-129-39 kubenswrapper[2575]: I0707 18:20:38.927622 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/28e6fd0e-6029-4bc7-87b7-f220986dd689-tmp-dir\") pod \"28e6fd0e-6029-4bc7-87b7-f220986dd689\" (UID: \"28e6fd0e-6029-4bc7-87b7-f220986dd689\") " Jul 07 18:20:38.927815 ip-10-0-129-39 kubenswrapper[2575]: I0707 18:20:38.927689 2575 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/28e6fd0e-6029-4bc7-87b7-f220986dd689-model-cache" (OuterVolumeSpecName: "model-cache") pod "28e6fd0e-6029-4bc7-87b7-f220986dd689" (UID: "28e6fd0e-6029-4bc7-87b7-f220986dd689"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 07 18:20:38.927815 ip-10-0-129-39 kubenswrapper[2575]: I0707 18:20:38.927702 2575 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/28e6fd0e-6029-4bc7-87b7-f220986dd689-home" (OuterVolumeSpecName: "home") pod "28e6fd0e-6029-4bc7-87b7-f220986dd689" (UID: "28e6fd0e-6029-4bc7-87b7-f220986dd689"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 07 18:20:38.927969 ip-10-0-129-39 kubenswrapper[2575]: I0707 18:20:38.927903 2575 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/28e6fd0e-6029-4bc7-87b7-f220986dd689-model-cache\") on node \"ip-10-0-129-39.ec2.internal\" DevicePath \"\"" Jul 07 18:20:38.927969 ip-10-0-129-39 kubenswrapper[2575]: I0707 18:20:38.927911 2575 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/28e6fd0e-6029-4bc7-87b7-f220986dd689-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "28e6fd0e-6029-4bc7-87b7-f220986dd689" (UID: "28e6fd0e-6029-4bc7-87b7-f220986dd689"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 07 18:20:38.927969 ip-10-0-129-39 kubenswrapper[2575]: I0707 18:20:38.927920 2575 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/28e6fd0e-6029-4bc7-87b7-f220986dd689-home\") on node \"ip-10-0-129-39.ec2.internal\" DevicePath \"\"" Jul 07 18:20:38.929518 ip-10-0-129-39 kubenswrapper[2575]: I0707 18:20:38.929482 2575 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/28e6fd0e-6029-4bc7-87b7-f220986dd689-dshm" (OuterVolumeSpecName: "dshm") pod "28e6fd0e-6029-4bc7-87b7-f220986dd689" (UID: "28e6fd0e-6029-4bc7-87b7-f220986dd689"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 07 18:20:38.929692 ip-10-0-129-39 kubenswrapper[2575]: I0707 18:20:38.929675 2575 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/28e6fd0e-6029-4bc7-87b7-f220986dd689-kube-api-access-slqxl" (OuterVolumeSpecName: "kube-api-access-slqxl") pod "28e6fd0e-6029-4bc7-87b7-f220986dd689" (UID: "28e6fd0e-6029-4bc7-87b7-f220986dd689"). InnerVolumeSpecName "kube-api-access-slqxl". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 07 18:20:38.929812 ip-10-0-129-39 kubenswrapper[2575]: I0707 18:20:38.929787 2575 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/28e6fd0e-6029-4bc7-87b7-f220986dd689-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "28e6fd0e-6029-4bc7-87b7-f220986dd689" (UID: "28e6fd0e-6029-4bc7-87b7-f220986dd689"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 07 18:20:38.999873 ip-10-0-129-39 kubenswrapper[2575]: I0707 18:20:38.999842 2575 generic.go:356] "Generic (PLEG): container finished" podID="28e6fd0e-6029-4bc7-87b7-f220986dd689" containerID="2d6b17c37ae406a191254c7e7c441669d0c6301d2cbefdbada5bf351623494a8" exitCode=0 Jul 07 18:20:38.999989 ip-10-0-129-39 kubenswrapper[2575]: I0707 18:20:38.999883 2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-21f62c/rate-test-model-21f62c-kserve-6954b87d59-hjwbw" Jul 07 18:20:38.999989 ip-10-0-129-39 kubenswrapper[2575]: I0707 18:20:38.999929 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-21f62c/rate-test-model-21f62c-kserve-6954b87d59-hjwbw" event={"ID":"28e6fd0e-6029-4bc7-87b7-f220986dd689","Type":"ContainerDied","Data":"2d6b17c37ae406a191254c7e7c441669d0c6301d2cbefdbada5bf351623494a8"} Jul 07 18:20:38.999989 ip-10-0-129-39 kubenswrapper[2575]: I0707 18:20:38.999967 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-21f62c/rate-test-model-21f62c-kserve-6954b87d59-hjwbw" event={"ID":"28e6fd0e-6029-4bc7-87b7-f220986dd689","Type":"ContainerDied","Data":"07c741e07cac1cf91dfc643eb6bb5f69355aee1d8dbff3e1c8cda1ce6d5a21be"} Jul 07 18:20:38.999989 ip-10-0-129-39 kubenswrapper[2575]: I0707 18:20:38.999983 2575 scope.go:117] "RemoveContainer" containerID="2d6b17c37ae406a191254c7e7c441669d0c6301d2cbefdbada5bf351623494a8" Jul 07 18:20:39.008550 ip-10-0-129-39 kubenswrapper[2575]: I0707 18:20:39.008533 2575 scope.go:117] "RemoveContainer" containerID="2d6b17c37ae406a191254c7e7c441669d0c6301d2cbefdbada5bf351623494a8" Jul 07 18:20:39.008841 ip-10-0-129-39 kubenswrapper[2575]: E0707 18:20:39.008785 2575 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2d6b17c37ae406a191254c7e7c441669d0c6301d2cbefdbada5bf351623494a8\": container with ID starting with 2d6b17c37ae406a191254c7e7c441669d0c6301d2cbefdbada5bf351623494a8 not found: ID does not exist" containerID="2d6b17c37ae406a191254c7e7c441669d0c6301d2cbefdbada5bf351623494a8" Jul 07 18:20:39.008841 ip-10-0-129-39 kubenswrapper[2575]: I0707 18:20:39.008810 2575 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2d6b17c37ae406a191254c7e7c441669d0c6301d2cbefdbada5bf351623494a8"} err="failed to get container status \"2d6b17c37ae406a191254c7e7c441669d0c6301d2cbefdbada5bf351623494a8\": rpc error: code = NotFound desc = could not find container \"2d6b17c37ae406a191254c7e7c441669d0c6301d2cbefdbada5bf351623494a8\": container with ID starting with 2d6b17c37ae406a191254c7e7c441669d0c6301d2cbefdbada5bf351623494a8 not found: ID does not exist" Jul 07 18:20:39.022328 ip-10-0-129-39 kubenswrapper[2575]: I0707 18:20:39.022303 2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-21f62c/rate-test-model-21f62c-kserve-6954b87d59-hjwbw"] Jul 07 18:20:39.027169 ip-10-0-129-39 kubenswrapper[2575]: I0707 18:20:39.027148 2575 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-a-21f62c/rate-test-model-21f62c-kserve-6954b87d59-hjwbw"] Jul 07 18:20:39.028343 ip-10-0-129-39 kubenswrapper[2575]: I0707 18:20:39.028320 2575 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/28e6fd0e-6029-4bc7-87b7-f220986dd689-tls-certs\") on node \"ip-10-0-129-39.ec2.internal\" DevicePath \"\"" Jul 07 18:20:39.028438 ip-10-0-129-39 kubenswrapper[2575]: I0707 18:20:39.028346 2575 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-slqxl\" (UniqueName: \"kubernetes.io/projected/28e6fd0e-6029-4bc7-87b7-f220986dd689-kube-api-access-slqxl\") on node \"ip-10-0-129-39.ec2.internal\" DevicePath \"\"" Jul 07 18:20:39.028438 ip-10-0-129-39 kubenswrapper[2575]: I0707 18:20:39.028362 2575 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/28e6fd0e-6029-4bc7-87b7-f220986dd689-tmp-dir\") on node \"ip-10-0-129-39.ec2.internal\" DevicePath \"\"" Jul 07 18:20:39.028438 ip-10-0-129-39 kubenswrapper[2575]: I0707 18:20:39.028375 2575 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/28e6fd0e-6029-4bc7-87b7-f220986dd689-dshm\") on node \"ip-10-0-129-39.ec2.internal\" DevicePath \"\"" Jul 07 18:20:40.073243 ip-10-0-129-39 kubenswrapper[2575]: I0707 18:20:40.073215 2575 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="28e6fd0e-6029-4bc7-87b7-f220986dd689" path="/var/lib/kubelet/pods/28e6fd0e-6029-4bc7-87b7-f220986dd689/volumes" Jul 07 18:21:11.198133 ip-10-0-129-39 kubenswrapper[2575]: I0707 18:21:11.198052 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-m825v"] Jul 07 18:21:17.120844 ip-10-0-129-39 kubenswrapper[2575]: I0707 18:21:17.120817 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-65dc978fbf-6sh24_db6a34f4-5292-455c-a3ad-b0d1d72dabee/authorino/0.log" Jul 07 18:21:21.853560 ip-10-0-129-39 kubenswrapper[2575]: I0707 18:21:21.853530 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_kserve-controller-manager-d6f4b65b8-d2drf_8d9db521-9487-4db9-9d32-9870e1739492/manager/0.log" Jul 07 18:21:21.980128 ip-10-0-129-39 kubenswrapper[2575]: I0707 18:21:21.980102 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_llmisvc-controller-manager-777c6bd7d5-8nrlc_5f66a7de-f49a-4ff3-ab89-b72709548dbf/manager/0.log" Jul 07 18:21:22.215222 ip-10-0-129-39 kubenswrapper[2575]: I0707 18:21:22.215138 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_model-serving-api-54d89bd5fc-pb6bw_549a1f02-41ef-4b55-914b-f34bbc8cda0c/server/0.log" Jul 07 18:21:22.333952 ip-10-0-129-39 kubenswrapper[2575]: I0707 18:21:22.333923 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_odh-model-controller-7998c67dc4-km2nt_a9875c1b-dca8-4038-b6e0-2ce204c410a0/manager/0.log" Jul 07 18:21:22.597727 ip-10-0-129-39 kubenswrapper[2575]: I0707 18:21:22.597702 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-d698b7b5d-7hn2s_9cb279fb-33ab-4bbf-bf04-886e5e82b247/copy-manifests/0.log" Jul 07 18:21:22.606391 ip-10-0-129-39 kubenswrapper[2575]: I0707 18:21:22.606349 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-d698b7b5d-7hn2s_9cb279fb-33ab-4bbf-bf04-886e5e82b247/manager/0.log" Jul 07 18:21:24.094441 ip-10-0-129-39 kubenswrapper[2575]: I0707 18:21:24.094411 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-65dc978fbf-6sh24_db6a34f4-5292-455c-a3ad-b0d1d72dabee/authorino/0.log" Jul 07 18:21:24.217317 ip-10-0-129-39 kubenswrapper[2575]: I0707 18:21:24.217292 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-operator-75cc8d5cb7-hlvlz_69ae37f2-4d15-44b6-a34d-ea5073901d1a/manager/0.log" Jul 07 18:21:24.569919 ip-10-0-129-39 kubenswrapper[2575]: I0707 18:21:24.569828 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-operator-catalog-sf58p_1ac64042-f41a-4ebb-a26b-e12f4c356c19/registry-server/0.log" Jul 07 18:21:24.708920 ip-10-0-129-39 kubenswrapper[2575]: I0707 18:21:24.708893 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-operator-controller-manager-86878c769f-tw877_06c4d2f4-68a1-4e31-b426-2bd937d5433e/manager/0.log" Jul 07 18:21:24.832907 ip-10-0-129-39 kubenswrapper[2575]: I0707 18:21:24.832880 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-limitador-54d6d988d5-m825v_f8a96214-b316-454a-9889-3c962c1ea035/limitador/0.log" Jul 07 18:21:25.308474 ip-10-0-129-39 kubenswrapper[2575]: I0707 18:21:25.308393 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_data-science-gateway-data-science-gateway-class-6576859cf4q24g5_cac31641-45b4-464e-ba63-65ce20b7cfdb/istio-proxy/0.log" Jul 07 18:21:25.685998 ip-10-0-129-39 kubenswrapper[2575]: I0707 18:21:25.685972 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-544cc96c69-v9zdn_5fda3a91-211d-4d8c-8237-df8287d92b46/kube-auth-proxy/0.log" Jul 07 18:21:25.811901 ip-10-0-129-39 kubenswrapper[2575]: I0707 18:21:25.811869 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-56654dcf7-q6xht_45428b46-dfbb-4c68-8a6c-ceffc91e8506/istio-proxy/1.log" Jul 07 18:21:26.184431 ip-10-0-129-39 kubenswrapper[2575]: I0707 18:21:26.184204 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_router-default-5d884d4557-nd85p_080f0a04-d8b0-45c9-98f7-db87cbc18520/router/0.log" Jul 07 18:21:26.684404 ip-10-0-129-39 kubenswrapper[2575]: I0707 18:21:26.684364 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-distinct-simulated-kserve-59975d69f9-npxxz_570486ae-48e8-470b-bdc2-075cebe69e8e/main/0.log" Jul 07 18:21:27.198467 ip-10-0-129-39 kubenswrapper[2575]: I0707 18:21:27.198436 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_premium-simulated-simulated-premium-kserve-5fc6fb456f-q49ch_a5ebeb37-0413-4f97-9115-7cb0708d1641/main/0.log" Jul 07 18:21:34.498392 ip-10-0-129-39 kubenswrapper[2575]: I0707 18:21:34.498367 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_global-pull-secret-syncer-76r79_8344b558-69d0-43b3-9aac-0cea0d5379f3/global-pull-secret-syncer/0.log" Jul 07 18:21:34.647395 ip-10-0-129-39 kubenswrapper[2575]: I0707 18:21:34.647363 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_konnectivity-agent-hr726_4f2c6f8d-3b44-4f4f-8d6b-433f013a867f/konnectivity-agent/0.log" Jul 07 18:21:34.722124 ip-10-0-129-39 kubenswrapper[2575]: I0707 18:21:34.722095 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kube-apiserver-proxy-ip-10-0-129-39.ec2.internal_9178fa6a4c12c534b2a4b3275d3b9775/haproxy/0.log" Jul 07 18:21:40.306060 ip-10-0-129-39 kubenswrapper[2575]: I0707 18:21:40.306024 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-65dc978fbf-6sh24_db6a34f4-5292-455c-a3ad-b0d1d72dabee/authorino/0.log" Jul 07 18:21:40.338845 ip-10-0-129-39 kubenswrapper[2575]: I0707 18:21:40.338820 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-operator-75cc8d5cb7-hlvlz_69ae37f2-4d15-44b6-a34d-ea5073901d1a/manager/0.log" Jul 07 18:21:40.478599 ip-10-0-129-39 kubenswrapper[2575]: I0707 18:21:40.478575 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-operator-catalog-sf58p_1ac64042-f41a-4ebb-a26b-e12f4c356c19/registry-server/0.log" Jul 07 18:21:40.668822 ip-10-0-129-39 kubenswrapper[2575]: I0707 18:21:40.668791 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-operator-controller-manager-86878c769f-tw877_06c4d2f4-68a1-4e31-b426-2bd937d5433e/manager/0.log" Jul 07 18:21:40.693401 ip-10-0-129-39 kubenswrapper[2575]: I0707 18:21:40.693380 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-limitador-54d6d988d5-m825v_f8a96214-b316-454a-9889-3c962c1ea035/limitador/0.log" Jul 07 18:21:42.656996 ip-10-0-129-39 kubenswrapper[2575]: I0707 18:21:42.656961 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_cluster-monitoring-operator-64786d9cbb-4fqsh_bab28ccd-f54a-4691-add4-8c665b4e1124/cluster-monitoring-operator/0.log" Jul 07 18:21:42.796148 ip-10-0-129-39 kubenswrapper[2575]: I0707 18:21:42.796123 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_metrics-server-7df964d657-xdrrj_7406f82e-b601-4f1a-beec-dd77a9b835d4/metrics-server/0.log" Jul 07 18:21:43.096450 ip-10-0-129-39 kubenswrapper[2575]: I0707 18:21:43.096418 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-nbqxr_ba95b45f-7620-472e-b924-83a1ed9ba25c/node-exporter/0.log" Jul 07 18:21:43.123278 ip-10-0-129-39 kubenswrapper[2575]: I0707 18:21:43.123254 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-nbqxr_ba95b45f-7620-472e-b924-83a1ed9ba25c/kube-rbac-proxy/0.log" Jul 07 18:21:43.150620 ip-10-0-129-39 kubenswrapper[2575]: I0707 18:21:43.150599 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-nbqxr_ba95b45f-7620-472e-b924-83a1ed9ba25c/init-textfile/0.log" Jul 07 18:21:44.465950 ip-10-0-129-39 kubenswrapper[2575]: I0707 18:21:44.465913 2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-must-gather-99zpf/perf-node-gather-daemonset-lz9x7"] Jul 07 18:21:44.466322 ip-10-0-129-39 kubenswrapper[2575]: I0707 18:21:44.466249 2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="28e6fd0e-6029-4bc7-87b7-f220986dd689" containerName="main" Jul 07 18:21:44.466322 ip-10-0-129-39 kubenswrapper[2575]: I0707 18:21:44.466260 2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="28e6fd0e-6029-4bc7-87b7-f220986dd689" containerName="main" Jul 07 18:21:44.466402 ip-10-0-129-39 kubenswrapper[2575]: I0707 18:21:44.466331 2575 memory_manager.go:368] "RemoveStaleState removing state" podUID="28e6fd0e-6029-4bc7-87b7-f220986dd689" containerName="main" Jul 07 18:21:44.469610 ip-10-0-129-39 kubenswrapper[2575]: I0707 18:21:44.469589 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-99zpf/perf-node-gather-daemonset-lz9x7" Jul 07 18:21:44.474800 ip-10-0-129-39 kubenswrapper[2575]: E0707 18:21:44.474765 2575 reflector.go:205] "Failed to watch" err="failed to list *v1.ConfigMap: configmaps \"kube-root-ca.crt\" is forbidden: User \"system:node:ip-10-0-129-39.ec2.internal\" cannot list resource \"configmaps\" in API group \"\" in the namespace \"openshift-must-gather-99zpf\": no relationship found between node 'ip-10-0-129-39.ec2.internal' and this object" logger="UnhandledError" reflector="object-\"openshift-must-gather-99zpf\"/\"kube-root-ca.crt\"" type="*v1.ConfigMap" Jul 07 18:21:44.474964 ip-10-0-129-39 kubenswrapper[2575]: E0707 18:21:44.474820 2575 status_manager.go:1018] "Failed to get status for pod" err="pods \"perf-node-gather-daemonset-lz9x7\" is forbidden: User \"system:node:ip-10-0-129-39.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"openshift-must-gather-99zpf\": no relationship found between node 'ip-10-0-129-39.ec2.internal' and this object" podUID="bf6dbeb5-9d95-452e-8c5e-f36ef6faecdb" pod="openshift-must-gather-99zpf/perf-node-gather-daemonset-lz9x7" Jul 07 18:21:44.475112 ip-10-0-129-39 kubenswrapper[2575]: E0707 18:21:44.474904 2575 reflector.go:205] "Failed to watch" err="failed to list *v1.ConfigMap: configmaps \"openshift-service-ca.crt\" is forbidden: User \"system:node:ip-10-0-129-39.ec2.internal\" cannot list resource \"configmaps\" in API group \"\" in the namespace \"openshift-must-gather-99zpf\": no relationship found between node 'ip-10-0-129-39.ec2.internal' and this object" logger="UnhandledError" reflector="object-\"openshift-must-gather-99zpf\"/\"openshift-service-ca.crt\"" type="*v1.ConfigMap" Jul 07 18:21:44.477012 ip-10-0-129-39 kubenswrapper[2575]: E0707 18:21:44.475468 2575 reflector.go:205] "Failed to watch" err="failed to list *v1.Secret: secrets \"default-dockercfg-bkmgk\" is forbidden: User \"system:node:ip-10-0-129-39.ec2.internal\" cannot list resource \"secrets\" in API group \"\" in the namespace \"openshift-must-gather-99zpf\": no relationship found between node 'ip-10-0-129-39.ec2.internal' and this object" logger="UnhandledError" reflector="object-\"openshift-must-gather-99zpf\"/\"default-dockercfg-bkmgk\"" type="*v1.Secret" Jul 07 18:21:44.482218 ip-10-0-129-39 kubenswrapper[2575]: I0707 18:21:44.482189 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-99zpf/perf-node-gather-daemonset-lz9x7"] Jul 07 18:21:44.550074 ip-10-0-129-39 kubenswrapper[2575]: I0707 18:21:44.550039 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/bf6dbeb5-9d95-452e-8c5e-f36ef6faecdb-proc\") pod \"perf-node-gather-daemonset-lz9x7\" (UID: \"bf6dbeb5-9d95-452e-8c5e-f36ef6faecdb\") " pod="openshift-must-gather-99zpf/perf-node-gather-daemonset-lz9x7" Jul 07 18:21:44.550201 ip-10-0-129-39 kubenswrapper[2575]: I0707 18:21:44.550086 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/bf6dbeb5-9d95-452e-8c5e-f36ef6faecdb-podres\") pod \"perf-node-gather-daemonset-lz9x7\" (UID: \"bf6dbeb5-9d95-452e-8c5e-f36ef6faecdb\") " pod="openshift-must-gather-99zpf/perf-node-gather-daemonset-lz9x7" Jul 07 18:21:44.550201 ip-10-0-129-39 kubenswrapper[2575]: I0707 18:21:44.550105 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/bf6dbeb5-9d95-452e-8c5e-f36ef6faecdb-sys\") pod \"perf-node-gather-daemonset-lz9x7\" (UID: \"bf6dbeb5-9d95-452e-8c5e-f36ef6faecdb\") " pod="openshift-must-gather-99zpf/perf-node-gather-daemonset-lz9x7" Jul 07 18:21:44.550201 ip-10-0-129-39 kubenswrapper[2575]: I0707 18:21:44.550166 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/bf6dbeb5-9d95-452e-8c5e-f36ef6faecdb-lib-modules\") pod \"perf-node-gather-daemonset-lz9x7\" (UID: \"bf6dbeb5-9d95-452e-8c5e-f36ef6faecdb\") " pod="openshift-must-gather-99zpf/perf-node-gather-daemonset-lz9x7" Jul 07 18:21:44.550305 ip-10-0-129-39 kubenswrapper[2575]: I0707 18:21:44.550230 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mrwvk\" (UniqueName: \"kubernetes.io/projected/bf6dbeb5-9d95-452e-8c5e-f36ef6faecdb-kube-api-access-mrwvk\") pod \"perf-node-gather-daemonset-lz9x7\" (UID: \"bf6dbeb5-9d95-452e-8c5e-f36ef6faecdb\") " pod="openshift-must-gather-99zpf/perf-node-gather-daemonset-lz9x7" Jul 07 18:21:44.651397 ip-10-0-129-39 kubenswrapper[2575]: I0707 18:21:44.651372 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/bf6dbeb5-9d95-452e-8c5e-f36ef6faecdb-sys\") pod \"perf-node-gather-daemonset-lz9x7\" (UID: \"bf6dbeb5-9d95-452e-8c5e-f36ef6faecdb\") " pod="openshift-must-gather-99zpf/perf-node-gather-daemonset-lz9x7" Jul 07 18:21:44.651496 ip-10-0-129-39 kubenswrapper[2575]: I0707 18:21:44.651400 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/bf6dbeb5-9d95-452e-8c5e-f36ef6faecdb-lib-modules\") pod \"perf-node-gather-daemonset-lz9x7\" (UID: \"bf6dbeb5-9d95-452e-8c5e-f36ef6faecdb\") " pod="openshift-must-gather-99zpf/perf-node-gather-daemonset-lz9x7" Jul 07 18:21:44.651496 ip-10-0-129-39 kubenswrapper[2575]: I0707 18:21:44.651424 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mrwvk\" (UniqueName: \"kubernetes.io/projected/bf6dbeb5-9d95-452e-8c5e-f36ef6faecdb-kube-api-access-mrwvk\") pod \"perf-node-gather-daemonset-lz9x7\" (UID: \"bf6dbeb5-9d95-452e-8c5e-f36ef6faecdb\") " pod="openshift-must-gather-99zpf/perf-node-gather-daemonset-lz9x7" Jul 07 18:21:44.651496 ip-10-0-129-39 kubenswrapper[2575]: I0707 18:21:44.651494 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/bf6dbeb5-9d95-452e-8c5e-f36ef6faecdb-proc\") pod \"perf-node-gather-daemonset-lz9x7\" (UID: \"bf6dbeb5-9d95-452e-8c5e-f36ef6faecdb\") " pod="openshift-must-gather-99zpf/perf-node-gather-daemonset-lz9x7" Jul 07 18:21:44.651648 ip-10-0-129-39 kubenswrapper[2575]: I0707 18:21:44.651549 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/bf6dbeb5-9d95-452e-8c5e-f36ef6faecdb-podres\") pod \"perf-node-gather-daemonset-lz9x7\" (UID: \"bf6dbeb5-9d95-452e-8c5e-f36ef6faecdb\") " pod="openshift-must-gather-99zpf/perf-node-gather-daemonset-lz9x7" Jul 07 18:21:44.651648 ip-10-0-129-39 kubenswrapper[2575]: I0707 18:21:44.651555 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/bf6dbeb5-9d95-452e-8c5e-f36ef6faecdb-proc\") pod \"perf-node-gather-daemonset-lz9x7\" (UID: \"bf6dbeb5-9d95-452e-8c5e-f36ef6faecdb\") " pod="openshift-must-gather-99zpf/perf-node-gather-daemonset-lz9x7" Jul 07 18:21:44.651648 ip-10-0-129-39 kubenswrapper[2575]: I0707 18:21:44.651495 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/bf6dbeb5-9d95-452e-8c5e-f36ef6faecdb-sys\") pod \"perf-node-gather-daemonset-lz9x7\" (UID: \"bf6dbeb5-9d95-452e-8c5e-f36ef6faecdb\") " pod="openshift-must-gather-99zpf/perf-node-gather-daemonset-lz9x7" Jul 07 18:21:44.651648 ip-10-0-129-39 kubenswrapper[2575]: I0707 18:21:44.651607 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/bf6dbeb5-9d95-452e-8c5e-f36ef6faecdb-lib-modules\") pod \"perf-node-gather-daemonset-lz9x7\" (UID: \"bf6dbeb5-9d95-452e-8c5e-f36ef6faecdb\") " pod="openshift-must-gather-99zpf/perf-node-gather-daemonset-lz9x7" Jul 07 18:21:44.651786 ip-10-0-129-39 kubenswrapper[2575]: I0707 18:21:44.651652 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/bf6dbeb5-9d95-452e-8c5e-f36ef6faecdb-podres\") pod \"perf-node-gather-daemonset-lz9x7\" (UID: \"bf6dbeb5-9d95-452e-8c5e-f36ef6faecdb\") " pod="openshift-must-gather-99zpf/perf-node-gather-daemonset-lz9x7" Jul 07 18:21:45.660243 ip-10-0-129-39 kubenswrapper[2575]: E0707 18:21:45.660211 2575 projected.go:291] Couldn't get configMap openshift-must-gather-99zpf/kube-root-ca.crt: failed to sync configmap cache: timed out waiting for the condition Jul 07 18:21:45.709696 ip-10-0-129-39 kubenswrapper[2575]: I0707 18:21:45.709669 2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-99zpf\"/\"kube-root-ca.crt\"" Jul 07 18:21:45.788871 ip-10-0-129-39 kubenswrapper[2575]: I0707 18:21:45.788845 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-99zpf\"/\"default-dockercfg-bkmgk\"" Jul 07 18:21:45.879919 ip-10-0-129-39 kubenswrapper[2575]: I0707 18:21:45.879898 2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-99zpf\"/\"openshift-service-ca.crt\"" Jul 07 18:21:45.880901 ip-10-0-129-39 kubenswrapper[2575]: E0707 18:21:45.880885 2575 projected.go:196] Error preparing data for projected volume kube-api-access-mrwvk for pod openshift-must-gather-99zpf/perf-node-gather-daemonset-lz9x7: failed to sync configmap cache: timed out waiting for the condition Jul 07 18:21:45.880960 ip-10-0-129-39 kubenswrapper[2575]: E0707 18:21:45.880947 2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/bf6dbeb5-9d95-452e-8c5e-f36ef6faecdb-kube-api-access-mrwvk podName:bf6dbeb5-9d95-452e-8c5e-f36ef6faecdb nodeName:}" failed. No retries permitted until 2026-07-07 18:21:46.380931357 +0000 UTC m=+2466.886738004 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-api-access-mrwvk" (UniqueName: "kubernetes.io/projected/bf6dbeb5-9d95-452e-8c5e-f36ef6faecdb-kube-api-access-mrwvk") pod "perf-node-gather-daemonset-lz9x7" (UID: "bf6dbeb5-9d95-452e-8c5e-f36ef6faecdb") : failed to sync configmap cache: timed out waiting for the condition Jul 07 18:21:46.466882 ip-10-0-129-39 kubenswrapper[2575]: I0707 18:21:46.466847 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mrwvk\" (UniqueName: \"kubernetes.io/projected/bf6dbeb5-9d95-452e-8c5e-f36ef6faecdb-kube-api-access-mrwvk\") pod \"perf-node-gather-daemonset-lz9x7\" (UID: \"bf6dbeb5-9d95-452e-8c5e-f36ef6faecdb\") " pod="openshift-must-gather-99zpf/perf-node-gather-daemonset-lz9x7" Jul 07 18:21:46.469217 ip-10-0-129-39 kubenswrapper[2575]: I0707 18:21:46.469195 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mrwvk\" (UniqueName: \"kubernetes.io/projected/bf6dbeb5-9d95-452e-8c5e-f36ef6faecdb-kube-api-access-mrwvk\") pod \"perf-node-gather-daemonset-lz9x7\" (UID: \"bf6dbeb5-9d95-452e-8c5e-f36ef6faecdb\") " pod="openshift-must-gather-99zpf/perf-node-gather-daemonset-lz9x7" Jul 07 18:21:46.583814 ip-10-0-129-39 kubenswrapper[2575]: I0707 18:21:46.583784 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-99zpf/perf-node-gather-daemonset-lz9x7" Jul 07 18:21:46.731806 ip-10-0-129-39 kubenswrapper[2575]: I0707 18:21:46.731775 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-99zpf/perf-node-gather-daemonset-lz9x7"] Jul 07 18:21:46.732604 ip-10-0-129-39 kubenswrapper[2575]: W0707 18:21:46.732575 2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-podbf6dbeb5_9d95_452e_8c5e_f36ef6faecdb.slice/crio-0f2417f03bd34f21c0677932bdbe01b6de128fd29e179959d111e7cdeec43031 WatchSource:0}: Error finding container 0f2417f03bd34f21c0677932bdbe01b6de128fd29e179959d111e7cdeec43031: Status 404 returned error can't find the container with id 0f2417f03bd34f21c0677932bdbe01b6de128fd29e179959d111e7cdeec43031 Jul 07 18:21:47.291214 ip-10-0-129-39 kubenswrapper[2575]: I0707 18:21:47.291185 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-cluster-storage-operator_volume-data-source-validator-5d456fdbcd-4knqs_e43b188b-652c-42b7-bb10-16092a59b829/volume-data-source-validator/0.log" Jul 07 18:21:47.299385 ip-10-0-129-39 kubenswrapper[2575]: I0707 18:21:47.299360 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-99zpf/perf-node-gather-daemonset-lz9x7" event={"ID":"bf6dbeb5-9d95-452e-8c5e-f36ef6faecdb","Type":"ContainerStarted","Data":"126c7e5a8e48c4219594b44bd433a1cb70b7914262cf7c0c629758432cde2031"} Jul 07 18:21:47.299526 ip-10-0-129-39 kubenswrapper[2575]: I0707 18:21:47.299405 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-99zpf/perf-node-gather-daemonset-lz9x7" event={"ID":"bf6dbeb5-9d95-452e-8c5e-f36ef6faecdb","Type":"ContainerStarted","Data":"0f2417f03bd34f21c0677932bdbe01b6de128fd29e179959d111e7cdeec43031"} Jul 07 18:21:47.299526 ip-10-0-129-39 kubenswrapper[2575]: I0707 18:21:47.299489 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-must-gather-99zpf/perf-node-gather-daemonset-lz9x7" Jul 07 18:21:47.316972 ip-10-0-129-39 kubenswrapper[2575]: I0707 18:21:47.316934 2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-must-gather-99zpf/perf-node-gather-daemonset-lz9x7" podStartSLOduration=3.316921613 podStartE2EDuration="3.316921613s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-07 18:21:44 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 18:21:47.31583449 +0000 UTC m=+2467.821641165" watchObservedRunningTime="2026-07-07 18:21:47.316921613 +0000 UTC m=+2467.822728279" Jul 07 18:21:48.559116 ip-10-0-129-39 kubenswrapper[2575]: I0707 18:21:48.559083 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-b2mnn_b2656ecc-1db3-4850-b2f6-58b0a14eb683/dns/0.log" Jul 07 18:21:48.586932 ip-10-0-129-39 kubenswrapper[2575]: I0707 18:21:48.586911 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-b2mnn_b2656ecc-1db3-4850-b2f6-58b0a14eb683/kube-rbac-proxy/0.log" Jul 07 18:21:48.732700 ip-10-0-129-39 kubenswrapper[2575]: I0707 18:21:48.732677 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-fc7qv_d4c44345-d6bf-4778-8911-09c7a1a0740a/dns-node-resolver/0.log" Jul 07 18:21:49.421241 ip-10-0-129-39 kubenswrapper[2575]: I0707 18:21:49.421209 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-l6pnn_ff237df8-d435-4d8c-88ec-8f31795bf0ef/node-ca/0.log" Jul 07 18:21:50.482138 ip-10-0-129-39 kubenswrapper[2575]: I0707 18:21:50.482104 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_data-science-gateway-data-science-gateway-class-6576859cf4q24g5_cac31641-45b4-464e-ba63-65ce20b7cfdb/istio-proxy/0.log" Jul 07 18:21:50.661940 ip-10-0-129-39 kubenswrapper[2575]: I0707 18:21:50.661913 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-544cc96c69-v9zdn_5fda3a91-211d-4d8c-8237-df8287d92b46/kube-auth-proxy/0.log" Jul 07 18:21:50.696137 ip-10-0-129-39 kubenswrapper[2575]: I0707 18:21:50.696106 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-56654dcf7-q6xht_45428b46-dfbb-4c68-8a6c-ceffc91e8506/istio-proxy/0.log" Jul 07 18:21:50.746999 ip-10-0-129-39 kubenswrapper[2575]: I0707 18:21:50.746915 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-56654dcf7-q6xht_45428b46-dfbb-4c68-8a6c-ceffc91e8506/istio-proxy/1.log" Jul 07 18:21:50.859414 ip-10-0-129-39 kubenswrapper[2575]: I0707 18:21:50.859385 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_router-default-5d884d4557-nd85p_080f0a04-d8b0-45c9-98f7-db87cbc18520/router/0.log" Jul 07 18:21:51.658403 ip-10-0-129-39 kubenswrapper[2575]: I0707 18:21:51.658372 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-x7kdh_1935a145-8811-413c-ad46-ae93176ffd7c/serve-healthcheck-canary/0.log" Jul 07 18:21:52.371815 ip-10-0-129-39 kubenswrapper[2575]: I0707 18:21:52.371781 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-fmk64_0c1d1c87-6251-49e9-ba58-47cb22ee80ca/kube-rbac-proxy/0.log" Jul 07 18:21:52.398931 ip-10-0-129-39 kubenswrapper[2575]: I0707 18:21:52.398903 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-fmk64_0c1d1c87-6251-49e9-ba58-47cb22ee80ca/exporter/0.log" Jul 07 18:21:52.423928 ip-10-0-129-39 kubenswrapper[2575]: I0707 18:21:52.423902 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-fmk64_0c1d1c87-6251-49e9-ba58-47cb22ee80ca/extractor/0.log" Jul 07 18:21:53.313680 ip-10-0-129-39 kubenswrapper[2575]: I0707 18:21:53.313614 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-must-gather-99zpf/perf-node-gather-daemonset-lz9x7" Jul 07 18:21:55.063325 ip-10-0-129-39 kubenswrapper[2575]: I0707 18:21:55.063294 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_kserve-controller-manager-d6f4b65b8-d2drf_8d9db521-9487-4db9-9d32-9870e1739492/manager/0.log" Jul 07 18:21:55.213326 ip-10-0-129-39 kubenswrapper[2575]: I0707 18:21:55.213295 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_data-science-gateway-data-science-gateway-class-6576859cf4q24g5_cac31641-45b4-464e-ba63-65ce20b7cfdb/istio-proxy/0.log" Jul 07 18:21:55.287688 ip-10-0-129-39 kubenswrapper[2575]: I0707 18:21:55.287614 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_llmisvc-controller-manager-777c6bd7d5-8nrlc_5f66a7de-f49a-4ff3-ab89-b72709548dbf/manager/0.log" Jul 07 18:21:55.364747 ip-10-0-129-39 kubenswrapper[2575]: I0707 18:21:55.364717 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-544cc96c69-v9zdn_5fda3a91-211d-4d8c-8237-df8287d92b46/kube-auth-proxy/0.log" Jul 07 18:21:55.390543 ip-10-0-129-39 kubenswrapper[2575]: I0707 18:21:55.390521 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-56654dcf7-q6xht_45428b46-dfbb-4c68-8a6c-ceffc91e8506/istio-proxy/0.log" Jul 07 18:21:55.413257 ip-10-0-129-39 kubenswrapper[2575]: I0707 18:21:55.413238 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_model-serving-api-54d89bd5fc-pb6bw_549a1f02-41ef-4b55-914b-f34bbc8cda0c/server/0.log" Jul 07 18:21:55.446863 ip-10-0-129-39 kubenswrapper[2575]: I0707 18:21:55.446841 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-56654dcf7-q6xht_45428b46-dfbb-4c68-8a6c-ceffc91e8506/istio-proxy/1.log" Jul 07 18:21:55.524705 ip-10-0-129-39 kubenswrapper[2575]: I0707 18:21:55.524684 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_router-default-5d884d4557-nd85p_080f0a04-d8b0-45c9-98f7-db87cbc18520/router/0.log" Jul 07 18:21:55.527625 ip-10-0-129-39 kubenswrapper[2575]: I0707 18:21:55.527598 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_odh-model-controller-7998c67dc4-km2nt_a9875c1b-dca8-4038-b6e0-2ce204c410a0/manager/0.log" Jul 07 18:21:55.619900 ip-10-0-129-39 kubenswrapper[2575]: I0707 18:21:55.619882 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-d698b7b5d-7hn2s_9cb279fb-33ab-4bbf-bf04-886e5e82b247/manager/0.log" Jul 07 18:21:55.644476 ip-10-0-129-39 kubenswrapper[2575]: I0707 18:21:55.644459 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-d698b7b5d-7hn2s_9cb279fb-33ab-4bbf-bf04-886e5e82b247/copy-manifests/0.log" Jul 07 18:21:56.723231 ip-10-0-129-39 kubenswrapper[2575]: I0707 18:21:56.723195 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_data-science-gateway-data-science-gateway-class-6576859cf4q24g5_cac31641-45b4-464e-ba63-65ce20b7cfdb/istio-proxy/0.log" Jul 07 18:21:56.870162 ip-10-0-129-39 kubenswrapper[2575]: I0707 18:21:56.870132 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-544cc96c69-v9zdn_5fda3a91-211d-4d8c-8237-df8287d92b46/kube-auth-proxy/0.log" Jul 07 18:21:56.901368 ip-10-0-129-39 kubenswrapper[2575]: I0707 18:21:56.901342 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-56654dcf7-q6xht_45428b46-dfbb-4c68-8a6c-ceffc91e8506/istio-proxy/0.log" Jul 07 18:21:56.957132 ip-10-0-129-39 kubenswrapper[2575]: I0707 18:21:56.957106 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-56654dcf7-q6xht_45428b46-dfbb-4c68-8a6c-ceffc91e8506/istio-proxy/1.log" Jul 07 18:21:57.041906 ip-10-0-129-39 kubenswrapper[2575]: I0707 18:21:57.041876 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_router-default-5d884d4557-nd85p_080f0a04-d8b0-45c9-98f7-db87cbc18520/router/0.log" Jul 07 18:21:57.132492 ip-10-0-129-39 kubenswrapper[2575]: I0707 18:21:57.132423 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-lws-operator_openshift-lws-operator-fd8ccff4c-rrdm6_2afbc361-c420-4739-a8fb-5c7912cdd14e/openshift-lws-operator/0.log" Jul 07 18:22:02.417797 ip-10-0-129-39 kubenswrapper[2575]: I0707 18:22:02.417766 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator_migrator-d57d58cc9-j2bs6_b7301c91-bb20-4577-9536-2673f662f3fd/migrator/0.log" Jul 07 18:22:02.454254 ip-10-0-129-39 kubenswrapper[2575]: I0707 18:22:02.454220 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator_migrator-d57d58cc9-j2bs6_b7301c91-bb20-4577-9536-2673f662f3fd/graceful-termination/0.log"