Jul 10 00:26:20.857337 ip-10-0-130-146 systemd[1]: Starting Kubernetes Kubelet... Jul 10 00:26:21.161821 ip-10-0-130-146 kubenswrapper[2570]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Jul 10 00:26:21.161821 ip-10-0-130-146 kubenswrapper[2570]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version. Jul 10 00:26:21.161821 ip-10-0-130-146 kubenswrapper[2570]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Jul 10 00:26:21.161821 ip-10-0-130-146 kubenswrapper[2570]: Flag --pod-infra-container-image has been deprecated, will be removed in 1.35. Image garbage collector will get sandbox image information from CRI. Jul 10 00:26:21.161821 ip-10-0-130-146 kubenswrapper[2570]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Jul 10 00:26:21.163860 ip-10-0-130-146 kubenswrapper[2570]: I0710 00:26:21.163765 2570 server.go:213] "--pod-infra-container-image will not be pruned by the image garbage collector in kubelet and should also be set in the remote runtime" Jul 10 00:26:21.168509 ip-10-0-130-146 kubenswrapper[2570]: W0710 00:26:21.168492 2570 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 10 00:26:21.168509 ip-10-0-130-146 kubenswrapper[2570]: W0710 00:26:21.168509 2570 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 10 00:26:21.168573 ip-10-0-130-146 kubenswrapper[2570]: W0710 00:26:21.168513 2570 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 10 00:26:21.168573 ip-10-0-130-146 kubenswrapper[2570]: W0710 00:26:21.168516 2570 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 10 00:26:21.168573 ip-10-0-130-146 kubenswrapper[2570]: W0710 00:26:21.168520 2570 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 10 00:26:21.168573 ip-10-0-130-146 kubenswrapper[2570]: W0710 00:26:21.168523 2570 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 10 00:26:21.168573 ip-10-0-130-146 kubenswrapper[2570]: W0710 00:26:21.168526 2570 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 10 00:26:21.168573 ip-10-0-130-146 kubenswrapper[2570]: W0710 00:26:21.168529 2570 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 10 00:26:21.168573 ip-10-0-130-146 kubenswrapper[2570]: W0710 00:26:21.168532 2570 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 10 00:26:21.168573 ip-10-0-130-146 kubenswrapper[2570]: W0710 00:26:21.168535 2570 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 10 00:26:21.168573 ip-10-0-130-146 kubenswrapper[2570]: W0710 00:26:21.168538 2570 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 10 00:26:21.168573 ip-10-0-130-146 kubenswrapper[2570]: W0710 00:26:21.168541 2570 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 10 00:26:21.168573 ip-10-0-130-146 kubenswrapper[2570]: W0710 00:26:21.168544 2570 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 10 00:26:21.168573 ip-10-0-130-146 kubenswrapper[2570]: W0710 00:26:21.168546 2570 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 10 00:26:21.168573 ip-10-0-130-146 kubenswrapper[2570]: W0710 00:26:21.168549 2570 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 10 00:26:21.168573 ip-10-0-130-146 kubenswrapper[2570]: W0710 00:26:21.168552 2570 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 10 00:26:21.168573 ip-10-0-130-146 kubenswrapper[2570]: W0710 00:26:21.168555 2570 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 10 00:26:21.168573 ip-10-0-130-146 kubenswrapper[2570]: W0710 00:26:21.168558 2570 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 10 00:26:21.168573 ip-10-0-130-146 kubenswrapper[2570]: W0710 00:26:21.168560 2570 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 10 00:26:21.168573 ip-10-0-130-146 kubenswrapper[2570]: W0710 00:26:21.168563 2570 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 10 00:26:21.168573 ip-10-0-130-146 kubenswrapper[2570]: W0710 00:26:21.168568 2570 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 10 00:26:21.168573 ip-10-0-130-146 kubenswrapper[2570]: W0710 00:26:21.168573 2570 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 10 00:26:21.169068 ip-10-0-130-146 kubenswrapper[2570]: W0710 00:26:21.168576 2570 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 10 00:26:21.169068 ip-10-0-130-146 kubenswrapper[2570]: W0710 00:26:21.168581 2570 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 10 00:26:21.169068 ip-10-0-130-146 kubenswrapper[2570]: W0710 00:26:21.168586 2570 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 10 00:26:21.169068 ip-10-0-130-146 kubenswrapper[2570]: W0710 00:26:21.168590 2570 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 10 00:26:21.169068 ip-10-0-130-146 kubenswrapper[2570]: W0710 00:26:21.168593 2570 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 10 00:26:21.169068 ip-10-0-130-146 kubenswrapper[2570]: W0710 00:26:21.168596 2570 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 10 00:26:21.169068 ip-10-0-130-146 kubenswrapper[2570]: W0710 00:26:21.168600 2570 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 10 00:26:21.169068 ip-10-0-130-146 kubenswrapper[2570]: W0710 00:26:21.168603 2570 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 10 00:26:21.169068 ip-10-0-130-146 kubenswrapper[2570]: W0710 00:26:21.168606 2570 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 10 00:26:21.169068 ip-10-0-130-146 kubenswrapper[2570]: W0710 00:26:21.168609 2570 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 10 00:26:21.169068 ip-10-0-130-146 kubenswrapper[2570]: W0710 00:26:21.168612 2570 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 10 00:26:21.169068 ip-10-0-130-146 kubenswrapper[2570]: W0710 00:26:21.168615 2570 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 10 00:26:21.169068 ip-10-0-130-146 kubenswrapper[2570]: W0710 00:26:21.168617 2570 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 10 00:26:21.169068 ip-10-0-130-146 kubenswrapper[2570]: W0710 00:26:21.168621 2570 feature_gate.go:328] unrecognized feature gate: Example Jul 10 00:26:21.169068 ip-10-0-130-146 kubenswrapper[2570]: W0710 00:26:21.168623 2570 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 10 00:26:21.169068 ip-10-0-130-146 kubenswrapper[2570]: W0710 00:26:21.168626 2570 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 10 00:26:21.169068 ip-10-0-130-146 kubenswrapper[2570]: W0710 00:26:21.168630 2570 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 10 00:26:21.169068 ip-10-0-130-146 kubenswrapper[2570]: W0710 00:26:21.168633 2570 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 10 00:26:21.169068 ip-10-0-130-146 kubenswrapper[2570]: W0710 00:26:21.168636 2570 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 10 00:26:21.169068 ip-10-0-130-146 kubenswrapper[2570]: W0710 00:26:21.168638 2570 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 10 00:26:21.169590 ip-10-0-130-146 kubenswrapper[2570]: W0710 00:26:21.168641 2570 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 10 00:26:21.169590 ip-10-0-130-146 kubenswrapper[2570]: W0710 00:26:21.168644 2570 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 10 00:26:21.169590 ip-10-0-130-146 kubenswrapper[2570]: W0710 00:26:21.168647 2570 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 10 00:26:21.169590 ip-10-0-130-146 kubenswrapper[2570]: W0710 00:26:21.168649 2570 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 10 00:26:21.169590 ip-10-0-130-146 kubenswrapper[2570]: W0710 00:26:21.168652 2570 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 10 00:26:21.169590 ip-10-0-130-146 kubenswrapper[2570]: W0710 00:26:21.168655 2570 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 10 00:26:21.169590 ip-10-0-130-146 kubenswrapper[2570]: W0710 00:26:21.168657 2570 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 10 00:26:21.169590 ip-10-0-130-146 kubenswrapper[2570]: W0710 00:26:21.168660 2570 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 10 00:26:21.169590 ip-10-0-130-146 kubenswrapper[2570]: W0710 00:26:21.168665 2570 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 10 00:26:21.169590 ip-10-0-130-146 kubenswrapper[2570]: W0710 00:26:21.168668 2570 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 10 00:26:21.169590 ip-10-0-130-146 kubenswrapper[2570]: W0710 00:26:21.168671 2570 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 10 00:26:21.169590 ip-10-0-130-146 kubenswrapper[2570]: W0710 00:26:21.168674 2570 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 10 00:26:21.169590 ip-10-0-130-146 kubenswrapper[2570]: W0710 00:26:21.168677 2570 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 10 00:26:21.169590 ip-10-0-130-146 kubenswrapper[2570]: W0710 00:26:21.168682 2570 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 10 00:26:21.169590 ip-10-0-130-146 kubenswrapper[2570]: W0710 00:26:21.168685 2570 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 10 00:26:21.169590 ip-10-0-130-146 kubenswrapper[2570]: W0710 00:26:21.168688 2570 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 10 00:26:21.169590 ip-10-0-130-146 kubenswrapper[2570]: W0710 00:26:21.168691 2570 feature_gate.go:328] unrecognized feature gate: Example2 Jul 10 00:26:21.169590 ip-10-0-130-146 kubenswrapper[2570]: W0710 00:26:21.168694 2570 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 10 00:26:21.169590 ip-10-0-130-146 kubenswrapper[2570]: W0710 00:26:21.168697 2570 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 10 00:26:21.170071 ip-10-0-130-146 kubenswrapper[2570]: W0710 00:26:21.168700 2570 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 10 00:26:21.170071 ip-10-0-130-146 kubenswrapper[2570]: W0710 00:26:21.168703 2570 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 10 00:26:21.170071 ip-10-0-130-146 kubenswrapper[2570]: W0710 00:26:21.168706 2570 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 10 00:26:21.170071 ip-10-0-130-146 kubenswrapper[2570]: W0710 00:26:21.168708 2570 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 10 00:26:21.170071 ip-10-0-130-146 kubenswrapper[2570]: W0710 00:26:21.168711 2570 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 10 00:26:21.170071 ip-10-0-130-146 kubenswrapper[2570]: W0710 00:26:21.168714 2570 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 10 00:26:21.170071 ip-10-0-130-146 kubenswrapper[2570]: W0710 00:26:21.168717 2570 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 10 00:26:21.170071 ip-10-0-130-146 kubenswrapper[2570]: W0710 00:26:21.168720 2570 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 10 00:26:21.170071 ip-10-0-130-146 kubenswrapper[2570]: W0710 00:26:21.168723 2570 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 10 00:26:21.170071 ip-10-0-130-146 kubenswrapper[2570]: W0710 00:26:21.168725 2570 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 10 00:26:21.170071 ip-10-0-130-146 kubenswrapper[2570]: W0710 00:26:21.168729 2570 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 10 00:26:21.170071 ip-10-0-130-146 kubenswrapper[2570]: W0710 00:26:21.168731 2570 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 10 00:26:21.170071 ip-10-0-130-146 kubenswrapper[2570]: W0710 00:26:21.168734 2570 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 10 00:26:21.170071 ip-10-0-130-146 kubenswrapper[2570]: W0710 00:26:21.168737 2570 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 10 00:26:21.170071 ip-10-0-130-146 kubenswrapper[2570]: W0710 00:26:21.168740 2570 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 10 00:26:21.170071 ip-10-0-130-146 kubenswrapper[2570]: W0710 00:26:21.168744 2570 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 10 00:26:21.170071 ip-10-0-130-146 kubenswrapper[2570]: W0710 00:26:21.168746 2570 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 10 00:26:21.170071 ip-10-0-130-146 kubenswrapper[2570]: W0710 00:26:21.168749 2570 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 10 00:26:21.170071 ip-10-0-130-146 kubenswrapper[2570]: W0710 00:26:21.168752 2570 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 10 00:26:21.170071 ip-10-0-130-146 kubenswrapper[2570]: W0710 00:26:21.168755 2570 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 10 00:26:21.170603 ip-10-0-130-146 kubenswrapper[2570]: W0710 00:26:21.168758 2570 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 10 00:26:21.170603 ip-10-0-130-146 kubenswrapper[2570]: W0710 00:26:21.168760 2570 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 10 00:26:21.170603 ip-10-0-130-146 kubenswrapper[2570]: W0710 00:26:21.168763 2570 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 10 00:26:21.170603 ip-10-0-130-146 kubenswrapper[2570]: W0710 00:26:21.168765 2570 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 10 00:26:21.170603 ip-10-0-130-146 kubenswrapper[2570]: W0710 00:26:21.168768 2570 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 10 00:26:21.170603 ip-10-0-130-146 kubenswrapper[2570]: W0710 00:26:21.168771 2570 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 10 00:26:21.170603 ip-10-0-130-146 kubenswrapper[2570]: W0710 00:26:21.168775 2570 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 10 00:26:21.170603 ip-10-0-130-146 kubenswrapper[2570]: W0710 00:26:21.168778 2570 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 10 00:26:21.170603 ip-10-0-130-146 kubenswrapper[2570]: W0710 00:26:21.168780 2570 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 10 00:26:21.170603 ip-10-0-130-146 kubenswrapper[2570]: W0710 00:26:21.168783 2570 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 10 00:26:21.170603 ip-10-0-130-146 kubenswrapper[2570]: W0710 00:26:21.168786 2570 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 10 00:26:21.170603 ip-10-0-130-146 kubenswrapper[2570]: W0710 00:26:21.168788 2570 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 10 00:26:21.170603 ip-10-0-130-146 kubenswrapper[2570]: W0710 00:26:21.168791 2570 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 10 00:26:21.170603 ip-10-0-130-146 kubenswrapper[2570]: W0710 00:26:21.168794 2570 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 10 00:26:21.170603 ip-10-0-130-146 kubenswrapper[2570]: W0710 00:26:21.168797 2570 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 10 00:26:21.170603 ip-10-0-130-146 kubenswrapper[2570]: W0710 00:26:21.168799 2570 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 10 00:26:21.170603 ip-10-0-130-146 kubenswrapper[2570]: W0710 00:26:21.168802 2570 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 10 00:26:21.170603 ip-10-0-130-146 kubenswrapper[2570]: W0710 00:26:21.168805 2570 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 10 00:26:21.170603 ip-10-0-130-146 kubenswrapper[2570]: W0710 00:26:21.169545 2570 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 10 00:26:21.170603 ip-10-0-130-146 kubenswrapper[2570]: W0710 00:26:21.169553 2570 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 10 00:26:21.171166 ip-10-0-130-146 kubenswrapper[2570]: W0710 00:26:21.169556 2570 feature_gate.go:328] unrecognized feature gate: Example Jul 10 00:26:21.171166 ip-10-0-130-146 kubenswrapper[2570]: W0710 00:26:21.169559 2570 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 10 00:26:21.171166 ip-10-0-130-146 kubenswrapper[2570]: W0710 00:26:21.169562 2570 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 10 00:26:21.171166 ip-10-0-130-146 kubenswrapper[2570]: W0710 00:26:21.169566 2570 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 10 00:26:21.171166 ip-10-0-130-146 kubenswrapper[2570]: W0710 00:26:21.169569 2570 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 10 00:26:21.171166 ip-10-0-130-146 kubenswrapper[2570]: W0710 00:26:21.169572 2570 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 10 00:26:21.171166 ip-10-0-130-146 kubenswrapper[2570]: W0710 00:26:21.169575 2570 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 10 00:26:21.171166 ip-10-0-130-146 kubenswrapper[2570]: W0710 00:26:21.169577 2570 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 10 00:26:21.171166 ip-10-0-130-146 kubenswrapper[2570]: W0710 00:26:21.169580 2570 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 10 00:26:21.171166 ip-10-0-130-146 kubenswrapper[2570]: W0710 00:26:21.169583 2570 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 10 00:26:21.171166 ip-10-0-130-146 kubenswrapper[2570]: W0710 00:26:21.169585 2570 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 10 00:26:21.171166 ip-10-0-130-146 kubenswrapper[2570]: W0710 00:26:21.169588 2570 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 10 00:26:21.171166 ip-10-0-130-146 kubenswrapper[2570]: W0710 00:26:21.169591 2570 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 10 00:26:21.171166 ip-10-0-130-146 kubenswrapper[2570]: W0710 00:26:21.169594 2570 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 10 00:26:21.171166 ip-10-0-130-146 kubenswrapper[2570]: W0710 00:26:21.169597 2570 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 10 00:26:21.171166 ip-10-0-130-146 kubenswrapper[2570]: W0710 00:26:21.169600 2570 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 10 00:26:21.171166 ip-10-0-130-146 kubenswrapper[2570]: W0710 00:26:21.169603 2570 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 10 00:26:21.171166 ip-10-0-130-146 kubenswrapper[2570]: W0710 00:26:21.169606 2570 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 10 00:26:21.171166 ip-10-0-130-146 kubenswrapper[2570]: W0710 00:26:21.169610 2570 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 10 00:26:21.171166 ip-10-0-130-146 kubenswrapper[2570]: W0710 00:26:21.169613 2570 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 10 00:26:21.171690 ip-10-0-130-146 kubenswrapper[2570]: W0710 00:26:21.169616 2570 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 10 00:26:21.171690 ip-10-0-130-146 kubenswrapper[2570]: W0710 00:26:21.169619 2570 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 10 00:26:21.171690 ip-10-0-130-146 kubenswrapper[2570]: W0710 00:26:21.169622 2570 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 10 00:26:21.171690 ip-10-0-130-146 kubenswrapper[2570]: W0710 00:26:21.169625 2570 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 10 00:26:21.171690 ip-10-0-130-146 kubenswrapper[2570]: W0710 00:26:21.169628 2570 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 10 00:26:21.171690 ip-10-0-130-146 kubenswrapper[2570]: W0710 00:26:21.169631 2570 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 10 00:26:21.171690 ip-10-0-130-146 kubenswrapper[2570]: W0710 00:26:21.169633 2570 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 10 00:26:21.171690 ip-10-0-130-146 kubenswrapper[2570]: W0710 00:26:21.169636 2570 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 10 00:26:21.171690 ip-10-0-130-146 kubenswrapper[2570]: W0710 00:26:21.169639 2570 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 10 00:26:21.171690 ip-10-0-130-146 kubenswrapper[2570]: W0710 00:26:21.169642 2570 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 10 00:26:21.171690 ip-10-0-130-146 kubenswrapper[2570]: W0710 00:26:21.169645 2570 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 10 00:26:21.171690 ip-10-0-130-146 kubenswrapper[2570]: W0710 00:26:21.169647 2570 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 10 00:26:21.171690 ip-10-0-130-146 kubenswrapper[2570]: W0710 00:26:21.169650 2570 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 10 00:26:21.171690 ip-10-0-130-146 kubenswrapper[2570]: W0710 00:26:21.169653 2570 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 10 00:26:21.171690 ip-10-0-130-146 kubenswrapper[2570]: W0710 00:26:21.169656 2570 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 10 00:26:21.171690 ip-10-0-130-146 kubenswrapper[2570]: W0710 00:26:21.169659 2570 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 10 00:26:21.171690 ip-10-0-130-146 kubenswrapper[2570]: W0710 00:26:21.169662 2570 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 10 00:26:21.171690 ip-10-0-130-146 kubenswrapper[2570]: W0710 00:26:21.169665 2570 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 10 00:26:21.171690 ip-10-0-130-146 kubenswrapper[2570]: W0710 00:26:21.169667 2570 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 10 00:26:21.172198 ip-10-0-130-146 kubenswrapper[2570]: W0710 00:26:21.169672 2570 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 10 00:26:21.172198 ip-10-0-130-146 kubenswrapper[2570]: W0710 00:26:21.169676 2570 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 10 00:26:21.172198 ip-10-0-130-146 kubenswrapper[2570]: W0710 00:26:21.169679 2570 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 10 00:26:21.172198 ip-10-0-130-146 kubenswrapper[2570]: W0710 00:26:21.169682 2570 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 10 00:26:21.172198 ip-10-0-130-146 kubenswrapper[2570]: W0710 00:26:21.169686 2570 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 10 00:26:21.172198 ip-10-0-130-146 kubenswrapper[2570]: W0710 00:26:21.169689 2570 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 10 00:26:21.172198 ip-10-0-130-146 kubenswrapper[2570]: W0710 00:26:21.169692 2570 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 10 00:26:21.172198 ip-10-0-130-146 kubenswrapper[2570]: W0710 00:26:21.169694 2570 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 10 00:26:21.172198 ip-10-0-130-146 kubenswrapper[2570]: W0710 00:26:21.169697 2570 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 10 00:26:21.172198 ip-10-0-130-146 kubenswrapper[2570]: W0710 00:26:21.169700 2570 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 10 00:26:21.172198 ip-10-0-130-146 kubenswrapper[2570]: W0710 00:26:21.169702 2570 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 10 00:26:21.172198 ip-10-0-130-146 kubenswrapper[2570]: W0710 00:26:21.169706 2570 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 10 00:26:21.172198 ip-10-0-130-146 kubenswrapper[2570]: W0710 00:26:21.169709 2570 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 10 00:26:21.172198 ip-10-0-130-146 kubenswrapper[2570]: W0710 00:26:21.169712 2570 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 10 00:26:21.172198 ip-10-0-130-146 kubenswrapper[2570]: W0710 00:26:21.169715 2570 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 10 00:26:21.172198 ip-10-0-130-146 kubenswrapper[2570]: W0710 00:26:21.169717 2570 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 10 00:26:21.172198 ip-10-0-130-146 kubenswrapper[2570]: W0710 00:26:21.169720 2570 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 10 00:26:21.172198 ip-10-0-130-146 kubenswrapper[2570]: W0710 00:26:21.169723 2570 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 10 00:26:21.172198 ip-10-0-130-146 kubenswrapper[2570]: W0710 00:26:21.169725 2570 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 10 00:26:21.172687 ip-10-0-130-146 kubenswrapper[2570]: W0710 00:26:21.169729 2570 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 10 00:26:21.172687 ip-10-0-130-146 kubenswrapper[2570]: W0710 00:26:21.169732 2570 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 10 00:26:21.172687 ip-10-0-130-146 kubenswrapper[2570]: W0710 00:26:21.169734 2570 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 10 00:26:21.172687 ip-10-0-130-146 kubenswrapper[2570]: W0710 00:26:21.169737 2570 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 10 00:26:21.172687 ip-10-0-130-146 kubenswrapper[2570]: W0710 00:26:21.169740 2570 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 10 00:26:21.172687 ip-10-0-130-146 kubenswrapper[2570]: W0710 00:26:21.169742 2570 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 10 00:26:21.172687 ip-10-0-130-146 kubenswrapper[2570]: W0710 00:26:21.169745 2570 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 10 00:26:21.172687 ip-10-0-130-146 kubenswrapper[2570]: W0710 00:26:21.169747 2570 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 10 00:26:21.172687 ip-10-0-130-146 kubenswrapper[2570]: W0710 00:26:21.169750 2570 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 10 00:26:21.172687 ip-10-0-130-146 kubenswrapper[2570]: W0710 00:26:21.169753 2570 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 10 00:26:21.172687 ip-10-0-130-146 kubenswrapper[2570]: W0710 00:26:21.169755 2570 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 10 00:26:21.172687 ip-10-0-130-146 kubenswrapper[2570]: W0710 00:26:21.169759 2570 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 10 00:26:21.172687 ip-10-0-130-146 kubenswrapper[2570]: W0710 00:26:21.169761 2570 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 10 00:26:21.172687 ip-10-0-130-146 kubenswrapper[2570]: W0710 00:26:21.169764 2570 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 10 00:26:21.172687 ip-10-0-130-146 kubenswrapper[2570]: W0710 00:26:21.169766 2570 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 10 00:26:21.172687 ip-10-0-130-146 kubenswrapper[2570]: W0710 00:26:21.169769 2570 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 10 00:26:21.172687 ip-10-0-130-146 kubenswrapper[2570]: W0710 00:26:21.169772 2570 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 10 00:26:21.172687 ip-10-0-130-146 kubenswrapper[2570]: W0710 00:26:21.169775 2570 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 10 00:26:21.172687 ip-10-0-130-146 kubenswrapper[2570]: W0710 00:26:21.169777 2570 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 10 00:26:21.173206 ip-10-0-130-146 kubenswrapper[2570]: W0710 00:26:21.169782 2570 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 10 00:26:21.173206 ip-10-0-130-146 kubenswrapper[2570]: W0710 00:26:21.169785 2570 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 10 00:26:21.173206 ip-10-0-130-146 kubenswrapper[2570]: W0710 00:26:21.169788 2570 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 10 00:26:21.173206 ip-10-0-130-146 kubenswrapper[2570]: W0710 00:26:21.169791 2570 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 10 00:26:21.173206 ip-10-0-130-146 kubenswrapper[2570]: W0710 00:26:21.169794 2570 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 10 00:26:21.173206 ip-10-0-130-146 kubenswrapper[2570]: W0710 00:26:21.169798 2570 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 10 00:26:21.173206 ip-10-0-130-146 kubenswrapper[2570]: W0710 00:26:21.169801 2570 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 10 00:26:21.173206 ip-10-0-130-146 kubenswrapper[2570]: W0710 00:26:21.169804 2570 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 10 00:26:21.173206 ip-10-0-130-146 kubenswrapper[2570]: W0710 00:26:21.169806 2570 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 10 00:26:21.173206 ip-10-0-130-146 kubenswrapper[2570]: W0710 00:26:21.169809 2570 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 10 00:26:21.173206 ip-10-0-130-146 kubenswrapper[2570]: W0710 00:26:21.169812 2570 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 10 00:26:21.173206 ip-10-0-130-146 kubenswrapper[2570]: W0710 00:26:21.169815 2570 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 10 00:26:21.173206 ip-10-0-130-146 kubenswrapper[2570]: W0710 00:26:21.169817 2570 feature_gate.go:328] unrecognized feature gate: Example2 Jul 10 00:26:21.173206 ip-10-0-130-146 kubenswrapper[2570]: W0710 00:26:21.169820 2570 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 10 00:26:21.173206 ip-10-0-130-146 kubenswrapper[2570]: W0710 00:26:21.169823 2570 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 10 00:26:21.173206 ip-10-0-130-146 kubenswrapper[2570]: W0710 00:26:21.169826 2570 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 10 00:26:21.173206 ip-10-0-130-146 kubenswrapper[2570]: W0710 00:26:21.169829 2570 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 10 00:26:21.173206 ip-10-0-130-146 kubenswrapper[2570]: W0710 00:26:21.169833 2570 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 10 00:26:21.173206 ip-10-0-130-146 kubenswrapper[2570]: W0710 00:26:21.169837 2570 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 10 00:26:21.173709 ip-10-0-130-146 kubenswrapper[2570]: W0710 00:26:21.169840 2570 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 10 00:26:21.173709 ip-10-0-130-146 kubenswrapper[2570]: W0710 00:26:21.170254 2570 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 10 00:26:21.173709 ip-10-0-130-146 kubenswrapper[2570]: W0710 00:26:21.170260 2570 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 10 00:26:21.173709 ip-10-0-130-146 kubenswrapper[2570]: W0710 00:26:21.170263 2570 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 10 00:26:21.173709 ip-10-0-130-146 kubenswrapper[2570]: W0710 00:26:21.170267 2570 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 10 00:26:21.173709 ip-10-0-130-146 kubenswrapper[2570]: W0710 00:26:21.170278 2570 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 10 00:26:21.173709 ip-10-0-130-146 kubenswrapper[2570]: W0710 00:26:21.170281 2570 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 10 00:26:21.173709 ip-10-0-130-146 kubenswrapper[2570]: W0710 00:26:21.170284 2570 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 10 00:26:21.173709 ip-10-0-130-146 kubenswrapper[2570]: W0710 00:26:21.170287 2570 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 10 00:26:21.173709 ip-10-0-130-146 kubenswrapper[2570]: W0710 00:26:21.170289 2570 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 10 00:26:21.173709 ip-10-0-130-146 kubenswrapper[2570]: W0710 00:26:21.170292 2570 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 10 00:26:21.173709 ip-10-0-130-146 kubenswrapper[2570]: W0710 00:26:21.170295 2570 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 10 00:26:21.173709 ip-10-0-130-146 kubenswrapper[2570]: W0710 00:26:21.170297 2570 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 10 00:26:21.173709 ip-10-0-130-146 kubenswrapper[2570]: W0710 00:26:21.170301 2570 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 10 00:26:21.173709 ip-10-0-130-146 kubenswrapper[2570]: W0710 00:26:21.170304 2570 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 10 00:26:21.173709 ip-10-0-130-146 kubenswrapper[2570]: W0710 00:26:21.170306 2570 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 10 00:26:21.173709 ip-10-0-130-146 kubenswrapper[2570]: W0710 00:26:21.170309 2570 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 10 00:26:21.173709 ip-10-0-130-146 kubenswrapper[2570]: W0710 00:26:21.170312 2570 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 10 00:26:21.173709 ip-10-0-130-146 kubenswrapper[2570]: W0710 00:26:21.170315 2570 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 10 00:26:21.173709 ip-10-0-130-146 kubenswrapper[2570]: W0710 00:26:21.170318 2570 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 10 00:26:21.174262 ip-10-0-130-146 kubenswrapper[2570]: W0710 00:26:21.170321 2570 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 10 00:26:21.174262 ip-10-0-130-146 kubenswrapper[2570]: W0710 00:26:21.170324 2570 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 10 00:26:21.174262 ip-10-0-130-146 kubenswrapper[2570]: W0710 00:26:21.170327 2570 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 10 00:26:21.174262 ip-10-0-130-146 kubenswrapper[2570]: W0710 00:26:21.170330 2570 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 10 00:26:21.174262 ip-10-0-130-146 kubenswrapper[2570]: W0710 00:26:21.170333 2570 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 10 00:26:21.174262 ip-10-0-130-146 kubenswrapper[2570]: W0710 00:26:21.170335 2570 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 10 00:26:21.174262 ip-10-0-130-146 kubenswrapper[2570]: W0710 00:26:21.170338 2570 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 10 00:26:21.174262 ip-10-0-130-146 kubenswrapper[2570]: W0710 00:26:21.170342 2570 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 10 00:26:21.174262 ip-10-0-130-146 kubenswrapper[2570]: W0710 00:26:21.170345 2570 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 10 00:26:21.174262 ip-10-0-130-146 kubenswrapper[2570]: W0710 00:26:21.170347 2570 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 10 00:26:21.174262 ip-10-0-130-146 kubenswrapper[2570]: W0710 00:26:21.170350 2570 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 10 00:26:21.174262 ip-10-0-130-146 kubenswrapper[2570]: W0710 00:26:21.170353 2570 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 10 00:26:21.174262 ip-10-0-130-146 kubenswrapper[2570]: W0710 00:26:21.170355 2570 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 10 00:26:21.174262 ip-10-0-130-146 kubenswrapper[2570]: W0710 00:26:21.170359 2570 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 10 00:26:21.174262 ip-10-0-130-146 kubenswrapper[2570]: W0710 00:26:21.170363 2570 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 10 00:26:21.174262 ip-10-0-130-146 kubenswrapper[2570]: W0710 00:26:21.170366 2570 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 10 00:26:21.174262 ip-10-0-130-146 kubenswrapper[2570]: W0710 00:26:21.170370 2570 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 10 00:26:21.174262 ip-10-0-130-146 kubenswrapper[2570]: W0710 00:26:21.170373 2570 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 10 00:26:21.174262 ip-10-0-130-146 kubenswrapper[2570]: W0710 00:26:21.170382 2570 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 10 00:26:21.174262 ip-10-0-130-146 kubenswrapper[2570]: W0710 00:26:21.170385 2570 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 10 00:26:21.174752 ip-10-0-130-146 kubenswrapper[2570]: W0710 00:26:21.170388 2570 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 10 00:26:21.174752 ip-10-0-130-146 kubenswrapper[2570]: W0710 00:26:21.170390 2570 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 10 00:26:21.174752 ip-10-0-130-146 kubenswrapper[2570]: W0710 00:26:21.170393 2570 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 10 00:26:21.174752 ip-10-0-130-146 kubenswrapper[2570]: W0710 00:26:21.170395 2570 feature_gate.go:328] unrecognized feature gate: Example Jul 10 00:26:21.174752 ip-10-0-130-146 kubenswrapper[2570]: W0710 00:26:21.170398 2570 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 10 00:26:21.174752 ip-10-0-130-146 kubenswrapper[2570]: W0710 00:26:21.170401 2570 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 10 00:26:21.174752 ip-10-0-130-146 kubenswrapper[2570]: W0710 00:26:21.170403 2570 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 10 00:26:21.174752 ip-10-0-130-146 kubenswrapper[2570]: W0710 00:26:21.170406 2570 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 10 00:26:21.174752 ip-10-0-130-146 kubenswrapper[2570]: W0710 00:26:21.170409 2570 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 10 00:26:21.174752 ip-10-0-130-146 kubenswrapper[2570]: W0710 00:26:21.170411 2570 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 10 00:26:21.174752 ip-10-0-130-146 kubenswrapper[2570]: W0710 00:26:21.170414 2570 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 10 00:26:21.174752 ip-10-0-130-146 kubenswrapper[2570]: W0710 00:26:21.170417 2570 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 10 00:26:21.174752 ip-10-0-130-146 kubenswrapper[2570]: W0710 00:26:21.170420 2570 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 10 00:26:21.174752 ip-10-0-130-146 kubenswrapper[2570]: W0710 00:26:21.170423 2570 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 10 00:26:21.174752 ip-10-0-130-146 kubenswrapper[2570]: W0710 00:26:21.170426 2570 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 10 00:26:21.174752 ip-10-0-130-146 kubenswrapper[2570]: W0710 00:26:21.170429 2570 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 10 00:26:21.174752 ip-10-0-130-146 kubenswrapper[2570]: W0710 00:26:21.170432 2570 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 10 00:26:21.174752 ip-10-0-130-146 kubenswrapper[2570]: W0710 00:26:21.170434 2570 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 10 00:26:21.174752 ip-10-0-130-146 kubenswrapper[2570]: W0710 00:26:21.170437 2570 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 10 00:26:21.174752 ip-10-0-130-146 kubenswrapper[2570]: W0710 00:26:21.170440 2570 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 10 00:26:21.175321 ip-10-0-130-146 kubenswrapper[2570]: W0710 00:26:21.170443 2570 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 10 00:26:21.175321 ip-10-0-130-146 kubenswrapper[2570]: W0710 00:26:21.170446 2570 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 10 00:26:21.175321 ip-10-0-130-146 kubenswrapper[2570]: W0710 00:26:21.170448 2570 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 10 00:26:21.175321 ip-10-0-130-146 kubenswrapper[2570]: W0710 00:26:21.170452 2570 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 10 00:26:21.175321 ip-10-0-130-146 kubenswrapper[2570]: W0710 00:26:21.170456 2570 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 10 00:26:21.175321 ip-10-0-130-146 kubenswrapper[2570]: W0710 00:26:21.170459 2570 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 10 00:26:21.175321 ip-10-0-130-146 kubenswrapper[2570]: W0710 00:26:21.170462 2570 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 10 00:26:21.175321 ip-10-0-130-146 kubenswrapper[2570]: W0710 00:26:21.170465 2570 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 10 00:26:21.175321 ip-10-0-130-146 kubenswrapper[2570]: W0710 00:26:21.170467 2570 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 10 00:26:21.175321 ip-10-0-130-146 kubenswrapper[2570]: W0710 00:26:21.170470 2570 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 10 00:26:21.175321 ip-10-0-130-146 kubenswrapper[2570]: W0710 00:26:21.170472 2570 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 10 00:26:21.175321 ip-10-0-130-146 kubenswrapper[2570]: W0710 00:26:21.170483 2570 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 10 00:26:21.175321 ip-10-0-130-146 kubenswrapper[2570]: W0710 00:26:21.170486 2570 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 10 00:26:21.175321 ip-10-0-130-146 kubenswrapper[2570]: W0710 00:26:21.170490 2570 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 10 00:26:21.175321 ip-10-0-130-146 kubenswrapper[2570]: W0710 00:26:21.170492 2570 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 10 00:26:21.175321 ip-10-0-130-146 kubenswrapper[2570]: W0710 00:26:21.170495 2570 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 10 00:26:21.175321 ip-10-0-130-146 kubenswrapper[2570]: W0710 00:26:21.170498 2570 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 10 00:26:21.175321 ip-10-0-130-146 kubenswrapper[2570]: W0710 00:26:21.170501 2570 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 10 00:26:21.175321 ip-10-0-130-146 kubenswrapper[2570]: W0710 00:26:21.170504 2570 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 10 00:26:21.175849 ip-10-0-130-146 kubenswrapper[2570]: W0710 00:26:21.170506 2570 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 10 00:26:21.175849 ip-10-0-130-146 kubenswrapper[2570]: W0710 00:26:21.170509 2570 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 10 00:26:21.175849 ip-10-0-130-146 kubenswrapper[2570]: W0710 00:26:21.170512 2570 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 10 00:26:21.175849 ip-10-0-130-146 kubenswrapper[2570]: W0710 00:26:21.170514 2570 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 10 00:26:21.175849 ip-10-0-130-146 kubenswrapper[2570]: W0710 00:26:21.170519 2570 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 10 00:26:21.175849 ip-10-0-130-146 kubenswrapper[2570]: W0710 00:26:21.170522 2570 feature_gate.go:328] unrecognized feature gate: Example2 Jul 10 00:26:21.175849 ip-10-0-130-146 kubenswrapper[2570]: W0710 00:26:21.170525 2570 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 10 00:26:21.175849 ip-10-0-130-146 kubenswrapper[2570]: W0710 00:26:21.170528 2570 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 10 00:26:21.175849 ip-10-0-130-146 kubenswrapper[2570]: W0710 00:26:21.170531 2570 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 10 00:26:21.175849 ip-10-0-130-146 kubenswrapper[2570]: W0710 00:26:21.170536 2570 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 10 00:26:21.175849 ip-10-0-130-146 kubenswrapper[2570]: W0710 00:26:21.170539 2570 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 10 00:26:21.175849 ip-10-0-130-146 kubenswrapper[2570]: W0710 00:26:21.170548 2570 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 10 00:26:21.175849 ip-10-0-130-146 kubenswrapper[2570]: W0710 00:26:21.170551 2570 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 10 00:26:21.175849 ip-10-0-130-146 kubenswrapper[2570]: W0710 00:26:21.170555 2570 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 10 00:26:21.175849 ip-10-0-130-146 kubenswrapper[2570]: W0710 00:26:21.170558 2570 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 10 00:26:21.175849 ip-10-0-130-146 kubenswrapper[2570]: W0710 00:26:21.170561 2570 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 10 00:26:21.175849 ip-10-0-130-146 kubenswrapper[2570]: W0710 00:26:21.170564 2570 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 10 00:26:21.175849 ip-10-0-130-146 kubenswrapper[2570]: W0710 00:26:21.170567 2570 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 10 00:26:21.175849 ip-10-0-130-146 kubenswrapper[2570]: W0710 00:26:21.170570 2570 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 10 00:26:21.175849 ip-10-0-130-146 kubenswrapper[2570]: W0710 00:26:21.170573 2570 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 10 00:26:21.176393 ip-10-0-130-146 kubenswrapper[2570]: W0710 00:26:21.170576 2570 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 10 00:26:21.176393 ip-10-0-130-146 kubenswrapper[2570]: I0710 00:26:21.170644 2570 flags.go:64] FLAG: --address="0.0.0.0" Jul 10 00:26:21.176393 ip-10-0-130-146 kubenswrapper[2570]: I0710 00:26:21.170651 2570 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]" Jul 10 00:26:21.176393 ip-10-0-130-146 kubenswrapper[2570]: I0710 00:26:21.170658 2570 flags.go:64] FLAG: --anonymous-auth="true" Jul 10 00:26:21.176393 ip-10-0-130-146 kubenswrapper[2570]: I0710 00:26:21.170663 2570 flags.go:64] FLAG: --application-metrics-count-limit="100" Jul 10 00:26:21.176393 ip-10-0-130-146 kubenswrapper[2570]: I0710 00:26:21.170678 2570 flags.go:64] FLAG: --authentication-token-webhook="false" Jul 10 00:26:21.176393 ip-10-0-130-146 kubenswrapper[2570]: I0710 00:26:21.170682 2570 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s" Jul 10 00:26:21.176393 ip-10-0-130-146 kubenswrapper[2570]: I0710 00:26:21.170695 2570 flags.go:64] FLAG: --authorization-mode="AlwaysAllow" Jul 10 00:26:21.176393 ip-10-0-130-146 kubenswrapper[2570]: I0710 00:26:21.170703 2570 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s" Jul 10 00:26:21.176393 ip-10-0-130-146 kubenswrapper[2570]: I0710 00:26:21.170708 2570 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s" Jul 10 00:26:21.176393 ip-10-0-130-146 kubenswrapper[2570]: I0710 00:26:21.170713 2570 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id" Jul 10 00:26:21.176393 ip-10-0-130-146 kubenswrapper[2570]: I0710 00:26:21.170717 2570 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig" Jul 10 00:26:21.176393 ip-10-0-130-146 kubenswrapper[2570]: I0710 00:26:21.170721 2570 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki" Jul 10 00:26:21.176393 ip-10-0-130-146 kubenswrapper[2570]: I0710 00:26:21.170724 2570 flags.go:64] FLAG: --cgroup-driver="cgroupfs" Jul 10 00:26:21.176393 ip-10-0-130-146 kubenswrapper[2570]: I0710 00:26:21.170727 2570 flags.go:64] FLAG: --cgroup-root="" Jul 10 00:26:21.176393 ip-10-0-130-146 kubenswrapper[2570]: I0710 00:26:21.170731 2570 flags.go:64] FLAG: --cgroups-per-qos="true" Jul 10 00:26:21.176393 ip-10-0-130-146 kubenswrapper[2570]: I0710 00:26:21.170734 2570 flags.go:64] FLAG: --client-ca-file="" Jul 10 00:26:21.176393 ip-10-0-130-146 kubenswrapper[2570]: I0710 00:26:21.170737 2570 flags.go:64] FLAG: --cloud-provider="external" Jul 10 00:26:21.176393 ip-10-0-130-146 kubenswrapper[2570]: I0710 00:26:21.170740 2570 flags.go:64] FLAG: --cluster-dns="[]" Jul 10 00:26:21.176393 ip-10-0-130-146 kubenswrapper[2570]: I0710 00:26:21.170744 2570 flags.go:64] FLAG: --cluster-domain="" Jul 10 00:26:21.176393 ip-10-0-130-146 kubenswrapper[2570]: I0710 00:26:21.170747 2570 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf" Jul 10 00:26:21.176393 ip-10-0-130-146 kubenswrapper[2570]: I0710 00:26:21.170751 2570 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d" Jul 10 00:26:21.176393 ip-10-0-130-146 kubenswrapper[2570]: I0710 00:26:21.170755 2570 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json" Jul 10 00:26:21.176393 ip-10-0-130-146 kubenswrapper[2570]: I0710 00:26:21.170758 2570 flags.go:64] FLAG: --container-log-max-files="5" Jul 10 00:26:21.177018 ip-10-0-130-146 kubenswrapper[2570]: I0710 00:26:21.170762 2570 flags.go:64] FLAG: --container-log-max-size="10Mi" Jul 10 00:26:21.177018 ip-10-0-130-146 kubenswrapper[2570]: I0710 00:26:21.170765 2570 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock" Jul 10 00:26:21.177018 ip-10-0-130-146 kubenswrapper[2570]: I0710 00:26:21.170769 2570 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock" Jul 10 00:26:21.177018 ip-10-0-130-146 kubenswrapper[2570]: I0710 00:26:21.170772 2570 flags.go:64] FLAG: --containerd-namespace="k8s.io" Jul 10 00:26:21.177018 ip-10-0-130-146 kubenswrapper[2570]: I0710 00:26:21.170775 2570 flags.go:64] FLAG: --contention-profiling="false" Jul 10 00:26:21.177018 ip-10-0-130-146 kubenswrapper[2570]: I0710 00:26:21.170778 2570 flags.go:64] FLAG: --cpu-cfs-quota="true" Jul 10 00:26:21.177018 ip-10-0-130-146 kubenswrapper[2570]: I0710 00:26:21.170782 2570 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms" Jul 10 00:26:21.177018 ip-10-0-130-146 kubenswrapper[2570]: I0710 00:26:21.170785 2570 flags.go:64] FLAG: --cpu-manager-policy="none" Jul 10 00:26:21.177018 ip-10-0-130-146 kubenswrapper[2570]: I0710 00:26:21.170788 2570 flags.go:64] FLAG: --cpu-manager-policy-options="" Jul 10 00:26:21.177018 ip-10-0-130-146 kubenswrapper[2570]: I0710 00:26:21.170796 2570 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s" Jul 10 00:26:21.177018 ip-10-0-130-146 kubenswrapper[2570]: I0710 00:26:21.170801 2570 flags.go:64] FLAG: --enable-controller-attach-detach="true" Jul 10 00:26:21.177018 ip-10-0-130-146 kubenswrapper[2570]: I0710 00:26:21.170804 2570 flags.go:64] FLAG: --enable-debugging-handlers="true" Jul 10 00:26:21.177018 ip-10-0-130-146 kubenswrapper[2570]: I0710 00:26:21.170807 2570 flags.go:64] FLAG: --enable-load-reader="false" Jul 10 00:26:21.177018 ip-10-0-130-146 kubenswrapper[2570]: I0710 00:26:21.170810 2570 flags.go:64] FLAG: --enable-server="true" Jul 10 00:26:21.177018 ip-10-0-130-146 kubenswrapper[2570]: I0710 00:26:21.170820 2570 flags.go:64] FLAG: --enforce-node-allocatable="[pods]" Jul 10 00:26:21.177018 ip-10-0-130-146 kubenswrapper[2570]: I0710 00:26:21.170828 2570 flags.go:64] FLAG: --event-burst="100" Jul 10 00:26:21.177018 ip-10-0-130-146 kubenswrapper[2570]: I0710 00:26:21.170832 2570 flags.go:64] FLAG: --event-qps="50" Jul 10 00:26:21.177018 ip-10-0-130-146 kubenswrapper[2570]: I0710 00:26:21.170835 2570 flags.go:64] FLAG: --event-storage-age-limit="default=0" Jul 10 00:26:21.177018 ip-10-0-130-146 kubenswrapper[2570]: I0710 00:26:21.170838 2570 flags.go:64] FLAG: --event-storage-event-limit="default=0" Jul 10 00:26:21.177018 ip-10-0-130-146 kubenswrapper[2570]: I0710 00:26:21.170841 2570 flags.go:64] FLAG: --eviction-hard="" Jul 10 00:26:21.177018 ip-10-0-130-146 kubenswrapper[2570]: I0710 00:26:21.170845 2570 flags.go:64] FLAG: --eviction-max-pod-grace-period="0" Jul 10 00:26:21.177018 ip-10-0-130-146 kubenswrapper[2570]: I0710 00:26:21.170848 2570 flags.go:64] FLAG: --eviction-minimum-reclaim="" Jul 10 00:26:21.177018 ip-10-0-130-146 kubenswrapper[2570]: I0710 00:26:21.170851 2570 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s" Jul 10 00:26:21.177018 ip-10-0-130-146 kubenswrapper[2570]: I0710 00:26:21.170854 2570 flags.go:64] FLAG: --eviction-soft="" Jul 10 00:26:21.177018 ip-10-0-130-146 kubenswrapper[2570]: I0710 00:26:21.170857 2570 flags.go:64] FLAG: --eviction-soft-grace-period="" Jul 10 00:26:21.177676 ip-10-0-130-146 kubenswrapper[2570]: I0710 00:26:21.170860 2570 flags.go:64] FLAG: --exit-on-lock-contention="false" Jul 10 00:26:21.177676 ip-10-0-130-146 kubenswrapper[2570]: I0710 00:26:21.170864 2570 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false" Jul 10 00:26:21.177676 ip-10-0-130-146 kubenswrapper[2570]: I0710 00:26:21.170867 2570 flags.go:64] FLAG: --experimental-mounter-path="" Jul 10 00:26:21.177676 ip-10-0-130-146 kubenswrapper[2570]: I0710 00:26:21.170870 2570 flags.go:64] FLAG: --fail-cgroupv1="false" Jul 10 00:26:21.177676 ip-10-0-130-146 kubenswrapper[2570]: I0710 00:26:21.170874 2570 flags.go:64] FLAG: --fail-swap-on="true" Jul 10 00:26:21.177676 ip-10-0-130-146 kubenswrapper[2570]: I0710 00:26:21.170877 2570 flags.go:64] FLAG: --feature-gates="" Jul 10 00:26:21.177676 ip-10-0-130-146 kubenswrapper[2570]: I0710 00:26:21.170881 2570 flags.go:64] FLAG: --file-check-frequency="20s" Jul 10 00:26:21.177676 ip-10-0-130-146 kubenswrapper[2570]: I0710 00:26:21.170885 2570 flags.go:64] FLAG: --global-housekeeping-interval="1m0s" Jul 10 00:26:21.177676 ip-10-0-130-146 kubenswrapper[2570]: I0710 00:26:21.170888 2570 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge" Jul 10 00:26:21.177676 ip-10-0-130-146 kubenswrapper[2570]: I0710 00:26:21.170891 2570 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1" Jul 10 00:26:21.177676 ip-10-0-130-146 kubenswrapper[2570]: I0710 00:26:21.170894 2570 flags.go:64] FLAG: --healthz-port="10248" Jul 10 00:26:21.177676 ip-10-0-130-146 kubenswrapper[2570]: I0710 00:26:21.170897 2570 flags.go:64] FLAG: --help="false" Jul 10 00:26:21.177676 ip-10-0-130-146 kubenswrapper[2570]: I0710 00:26:21.170900 2570 flags.go:64] FLAG: --hostname-override="ip-10-0-130-146.ec2.internal" Jul 10 00:26:21.177676 ip-10-0-130-146 kubenswrapper[2570]: I0710 00:26:21.170904 2570 flags.go:64] FLAG: --housekeeping-interval="10s" Jul 10 00:26:21.177676 ip-10-0-130-146 kubenswrapper[2570]: I0710 00:26:21.170907 2570 flags.go:64] FLAG: --http-check-frequency="20s" Jul 10 00:26:21.177676 ip-10-0-130-146 kubenswrapper[2570]: I0710 00:26:21.170910 2570 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins" Jul 10 00:26:21.177676 ip-10-0-130-146 kubenswrapper[2570]: I0710 00:26:21.170914 2570 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml" Jul 10 00:26:21.177676 ip-10-0-130-146 kubenswrapper[2570]: I0710 00:26:21.170919 2570 flags.go:64] FLAG: --image-gc-high-threshold="85" Jul 10 00:26:21.177676 ip-10-0-130-146 kubenswrapper[2570]: I0710 00:26:21.170922 2570 flags.go:64] FLAG: --image-gc-low-threshold="80" Jul 10 00:26:21.177676 ip-10-0-130-146 kubenswrapper[2570]: I0710 00:26:21.170925 2570 flags.go:64] FLAG: --image-service-endpoint="" Jul 10 00:26:21.177676 ip-10-0-130-146 kubenswrapper[2570]: I0710 00:26:21.170928 2570 flags.go:64] FLAG: --kernel-memcg-notification="false" Jul 10 00:26:21.177676 ip-10-0-130-146 kubenswrapper[2570]: I0710 00:26:21.170931 2570 flags.go:64] FLAG: --kube-api-burst="100" Jul 10 00:26:21.177676 ip-10-0-130-146 kubenswrapper[2570]: I0710 00:26:21.170935 2570 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf" Jul 10 00:26:21.178423 ip-10-0-130-146 kubenswrapper[2570]: I0710 00:26:21.170938 2570 flags.go:64] FLAG: --kube-api-qps="50" Jul 10 00:26:21.178423 ip-10-0-130-146 kubenswrapper[2570]: I0710 00:26:21.170941 2570 flags.go:64] FLAG: --kube-reserved="" Jul 10 00:26:21.178423 ip-10-0-130-146 kubenswrapper[2570]: I0710 00:26:21.170944 2570 flags.go:64] FLAG: --kube-reserved-cgroup="" Jul 10 00:26:21.178423 ip-10-0-130-146 kubenswrapper[2570]: I0710 00:26:21.170948 2570 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig" Jul 10 00:26:21.178423 ip-10-0-130-146 kubenswrapper[2570]: I0710 00:26:21.170951 2570 flags.go:64] FLAG: --kubelet-cgroups="" Jul 10 00:26:21.178423 ip-10-0-130-146 kubenswrapper[2570]: I0710 00:26:21.170954 2570 flags.go:64] FLAG: --local-storage-capacity-isolation="true" Jul 10 00:26:21.178423 ip-10-0-130-146 kubenswrapper[2570]: I0710 00:26:21.170957 2570 flags.go:64] FLAG: --lock-file="" Jul 10 00:26:21.178423 ip-10-0-130-146 kubenswrapper[2570]: I0710 00:26:21.170960 2570 flags.go:64] FLAG: --log-cadvisor-usage="false" Jul 10 00:26:21.178423 ip-10-0-130-146 kubenswrapper[2570]: I0710 00:26:21.170963 2570 flags.go:64] FLAG: --log-flush-frequency="5s" Jul 10 00:26:21.178423 ip-10-0-130-146 kubenswrapper[2570]: I0710 00:26:21.170966 2570 flags.go:64] FLAG: --log-json-info-buffer-size="0" Jul 10 00:26:21.178423 ip-10-0-130-146 kubenswrapper[2570]: I0710 00:26:21.170972 2570 flags.go:64] FLAG: --log-json-split-stream="false" Jul 10 00:26:21.178423 ip-10-0-130-146 kubenswrapper[2570]: I0710 00:26:21.170975 2570 flags.go:64] FLAG: --log-text-info-buffer-size="0" Jul 10 00:26:21.178423 ip-10-0-130-146 kubenswrapper[2570]: I0710 00:26:21.170978 2570 flags.go:64] FLAG: --log-text-split-stream="false" Jul 10 00:26:21.178423 ip-10-0-130-146 kubenswrapper[2570]: I0710 00:26:21.170982 2570 flags.go:64] FLAG: --logging-format="text" Jul 10 00:26:21.178423 ip-10-0-130-146 kubenswrapper[2570]: I0710 00:26:21.170986 2570 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id" Jul 10 00:26:21.178423 ip-10-0-130-146 kubenswrapper[2570]: I0710 00:26:21.170990 2570 flags.go:64] FLAG: --make-iptables-util-chains="true" Jul 10 00:26:21.178423 ip-10-0-130-146 kubenswrapper[2570]: I0710 00:26:21.170993 2570 flags.go:64] FLAG: --manifest-url="" Jul 10 00:26:21.178423 ip-10-0-130-146 kubenswrapper[2570]: I0710 00:26:21.170996 2570 flags.go:64] FLAG: --manifest-url-header="" Jul 10 00:26:21.178423 ip-10-0-130-146 kubenswrapper[2570]: I0710 00:26:21.171000 2570 flags.go:64] FLAG: --max-housekeeping-interval="15s" Jul 10 00:26:21.178423 ip-10-0-130-146 kubenswrapper[2570]: I0710 00:26:21.171003 2570 flags.go:64] FLAG: --max-open-files="1000000" Jul 10 00:26:21.178423 ip-10-0-130-146 kubenswrapper[2570]: I0710 00:26:21.171007 2570 flags.go:64] FLAG: --max-pods="110" Jul 10 00:26:21.178423 ip-10-0-130-146 kubenswrapper[2570]: I0710 00:26:21.171011 2570 flags.go:64] FLAG: --maximum-dead-containers="-1" Jul 10 00:26:21.178423 ip-10-0-130-146 kubenswrapper[2570]: I0710 00:26:21.171014 2570 flags.go:64] FLAG: --maximum-dead-containers-per-container="1" Jul 10 00:26:21.178423 ip-10-0-130-146 kubenswrapper[2570]: I0710 00:26:21.171017 2570 flags.go:64] FLAG: --memory-manager-policy="None" Jul 10 00:26:21.178423 ip-10-0-130-146 kubenswrapper[2570]: I0710 00:26:21.171020 2570 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s" Jul 10 00:26:21.179070 ip-10-0-130-146 kubenswrapper[2570]: I0710 00:26:21.171023 2570 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s" Jul 10 00:26:21.179070 ip-10-0-130-146 kubenswrapper[2570]: I0710 00:26:21.171029 2570 flags.go:64] FLAG: --node-ip="0.0.0.0" Jul 10 00:26:21.179070 ip-10-0-130-146 kubenswrapper[2570]: I0710 00:26:21.171032 2570 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel" Jul 10 00:26:21.179070 ip-10-0-130-146 kubenswrapper[2570]: I0710 00:26:21.171039 2570 flags.go:64] FLAG: --node-status-max-images="50" Jul 10 00:26:21.179070 ip-10-0-130-146 kubenswrapper[2570]: I0710 00:26:21.171043 2570 flags.go:64] FLAG: --node-status-update-frequency="10s" Jul 10 00:26:21.179070 ip-10-0-130-146 kubenswrapper[2570]: I0710 00:26:21.171046 2570 flags.go:64] FLAG: --oom-score-adj="-999" Jul 10 00:26:21.179070 ip-10-0-130-146 kubenswrapper[2570]: I0710 00:26:21.171049 2570 flags.go:64] FLAG: --pod-cidr="" Jul 10 00:26:21.179070 ip-10-0-130-146 kubenswrapper[2570]: I0710 00:26:21.171053 2570 flags.go:64] FLAG: --pod-infra-container-image="quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:b04c0b1f73af4d021dd8f0b51bfeb302c24482c093de5f2c43654ab56c506926" Jul 10 00:26:21.179070 ip-10-0-130-146 kubenswrapper[2570]: I0710 00:26:21.171059 2570 flags.go:64] FLAG: --pod-manifest-path="" Jul 10 00:26:21.179070 ip-10-0-130-146 kubenswrapper[2570]: I0710 00:26:21.171062 2570 flags.go:64] FLAG: --pod-max-pids="-1" Jul 10 00:26:21.179070 ip-10-0-130-146 kubenswrapper[2570]: I0710 00:26:21.171065 2570 flags.go:64] FLAG: --pods-per-core="0" Jul 10 00:26:21.179070 ip-10-0-130-146 kubenswrapper[2570]: I0710 00:26:21.171069 2570 flags.go:64] FLAG: --port="10250" Jul 10 00:26:21.179070 ip-10-0-130-146 kubenswrapper[2570]: I0710 00:26:21.171072 2570 flags.go:64] FLAG: --protect-kernel-defaults="false" Jul 10 00:26:21.179070 ip-10-0-130-146 kubenswrapper[2570]: I0710 00:26:21.171075 2570 flags.go:64] FLAG: --provider-id="aws:///us-east-1a/i-0e18a645b6d36d631" Jul 10 00:26:21.179070 ip-10-0-130-146 kubenswrapper[2570]: I0710 00:26:21.171078 2570 flags.go:64] FLAG: --qos-reserved="" Jul 10 00:26:21.179070 ip-10-0-130-146 kubenswrapper[2570]: I0710 00:26:21.171081 2570 flags.go:64] FLAG: --read-only-port="10255" Jul 10 00:26:21.179070 ip-10-0-130-146 kubenswrapper[2570]: I0710 00:26:21.171085 2570 flags.go:64] FLAG: --register-node="true" Jul 10 00:26:21.179070 ip-10-0-130-146 kubenswrapper[2570]: I0710 00:26:21.171088 2570 flags.go:64] FLAG: --register-with-taints="" Jul 10 00:26:21.179070 ip-10-0-130-146 kubenswrapper[2570]: I0710 00:26:21.171091 2570 flags.go:64] FLAG: --registry-burst="10" Jul 10 00:26:21.179070 ip-10-0-130-146 kubenswrapper[2570]: I0710 00:26:21.171094 2570 flags.go:64] FLAG: --registry-qps="5" Jul 10 00:26:21.179070 ip-10-0-130-146 kubenswrapper[2570]: I0710 00:26:21.171099 2570 flags.go:64] FLAG: --reserved-cpus="" Jul 10 00:26:21.179070 ip-10-0-130-146 kubenswrapper[2570]: I0710 00:26:21.171102 2570 flags.go:64] FLAG: --reserved-memory="" Jul 10 00:26:21.179070 ip-10-0-130-146 kubenswrapper[2570]: I0710 00:26:21.171106 2570 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf" Jul 10 00:26:21.179070 ip-10-0-130-146 kubenswrapper[2570]: I0710 00:26:21.171110 2570 flags.go:64] FLAG: --root-dir="/var/lib/kubelet" Jul 10 00:26:21.179070 ip-10-0-130-146 kubenswrapper[2570]: I0710 00:26:21.171113 2570 flags.go:64] FLAG: --rotate-certificates="false" Jul 10 00:26:21.179733 ip-10-0-130-146 kubenswrapper[2570]: I0710 00:26:21.171116 2570 flags.go:64] FLAG: --rotate-server-certificates="false" Jul 10 00:26:21.179733 ip-10-0-130-146 kubenswrapper[2570]: I0710 00:26:21.171119 2570 flags.go:64] FLAG: --runonce="false" Jul 10 00:26:21.179733 ip-10-0-130-146 kubenswrapper[2570]: I0710 00:26:21.171142 2570 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service" Jul 10 00:26:21.179733 ip-10-0-130-146 kubenswrapper[2570]: I0710 00:26:21.171146 2570 flags.go:64] FLAG: --runtime-request-timeout="2m0s" Jul 10 00:26:21.179733 ip-10-0-130-146 kubenswrapper[2570]: I0710 00:26:21.171149 2570 flags.go:64] FLAG: --seccomp-default="false" Jul 10 00:26:21.179733 ip-10-0-130-146 kubenswrapper[2570]: I0710 00:26:21.171152 2570 flags.go:64] FLAG: --serialize-image-pulls="true" Jul 10 00:26:21.179733 ip-10-0-130-146 kubenswrapper[2570]: I0710 00:26:21.171155 2570 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s" Jul 10 00:26:21.179733 ip-10-0-130-146 kubenswrapper[2570]: I0710 00:26:21.171158 2570 flags.go:64] FLAG: --storage-driver-db="cadvisor" Jul 10 00:26:21.179733 ip-10-0-130-146 kubenswrapper[2570]: I0710 00:26:21.171162 2570 flags.go:64] FLAG: --storage-driver-host="localhost:8086" Jul 10 00:26:21.179733 ip-10-0-130-146 kubenswrapper[2570]: I0710 00:26:21.171166 2570 flags.go:64] FLAG: --storage-driver-password="root" Jul 10 00:26:21.179733 ip-10-0-130-146 kubenswrapper[2570]: I0710 00:26:21.171170 2570 flags.go:64] FLAG: --storage-driver-secure="false" Jul 10 00:26:21.179733 ip-10-0-130-146 kubenswrapper[2570]: I0710 00:26:21.171173 2570 flags.go:64] FLAG: --storage-driver-table="stats" Jul 10 00:26:21.179733 ip-10-0-130-146 kubenswrapper[2570]: I0710 00:26:21.171176 2570 flags.go:64] FLAG: --storage-driver-user="root" Jul 10 00:26:21.179733 ip-10-0-130-146 kubenswrapper[2570]: I0710 00:26:21.171179 2570 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s" Jul 10 00:26:21.179733 ip-10-0-130-146 kubenswrapper[2570]: I0710 00:26:21.171183 2570 flags.go:64] FLAG: --sync-frequency="1m0s" Jul 10 00:26:21.179733 ip-10-0-130-146 kubenswrapper[2570]: I0710 00:26:21.171187 2570 flags.go:64] FLAG: --system-cgroups="" Jul 10 00:26:21.179733 ip-10-0-130-146 kubenswrapper[2570]: I0710 00:26:21.171190 2570 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi" Jul 10 00:26:21.179733 ip-10-0-130-146 kubenswrapper[2570]: I0710 00:26:21.171196 2570 flags.go:64] FLAG: --system-reserved-cgroup="" Jul 10 00:26:21.179733 ip-10-0-130-146 kubenswrapper[2570]: I0710 00:26:21.171199 2570 flags.go:64] FLAG: --tls-cert-file="" Jul 10 00:26:21.179733 ip-10-0-130-146 kubenswrapper[2570]: I0710 00:26:21.171202 2570 flags.go:64] FLAG: --tls-cipher-suites="[]" Jul 10 00:26:21.179733 ip-10-0-130-146 kubenswrapper[2570]: I0710 00:26:21.171206 2570 flags.go:64] FLAG: --tls-min-version="" Jul 10 00:26:21.179733 ip-10-0-130-146 kubenswrapper[2570]: I0710 00:26:21.171209 2570 flags.go:64] FLAG: --tls-private-key-file="" Jul 10 00:26:21.179733 ip-10-0-130-146 kubenswrapper[2570]: I0710 00:26:21.171212 2570 flags.go:64] FLAG: --topology-manager-policy="none" Jul 10 00:26:21.179733 ip-10-0-130-146 kubenswrapper[2570]: I0710 00:26:21.171215 2570 flags.go:64] FLAG: --topology-manager-policy-options="" Jul 10 00:26:21.179733 ip-10-0-130-146 kubenswrapper[2570]: I0710 00:26:21.171218 2570 flags.go:64] FLAG: --topology-manager-scope="container" Jul 10 00:26:21.180460 ip-10-0-130-146 kubenswrapper[2570]: I0710 00:26:21.171221 2570 flags.go:64] FLAG: --v="2" Jul 10 00:26:21.180460 ip-10-0-130-146 kubenswrapper[2570]: I0710 00:26:21.171225 2570 flags.go:64] FLAG: --version="false" Jul 10 00:26:21.180460 ip-10-0-130-146 kubenswrapper[2570]: I0710 00:26:21.171231 2570 flags.go:64] FLAG: --vmodule="" Jul 10 00:26:21.180460 ip-10-0-130-146 kubenswrapper[2570]: I0710 00:26:21.171235 2570 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec" Jul 10 00:26:21.180460 ip-10-0-130-146 kubenswrapper[2570]: I0710 00:26:21.171238 2570 flags.go:64] FLAG: --volume-stats-agg-period="1m0s" Jul 10 00:26:21.180460 ip-10-0-130-146 kubenswrapper[2570]: W0710 00:26:21.171379 2570 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 10 00:26:21.180460 ip-10-0-130-146 kubenswrapper[2570]: W0710 00:26:21.171384 2570 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 10 00:26:21.180460 ip-10-0-130-146 kubenswrapper[2570]: W0710 00:26:21.171387 2570 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 10 00:26:21.180460 ip-10-0-130-146 kubenswrapper[2570]: W0710 00:26:21.171391 2570 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 10 00:26:21.180460 ip-10-0-130-146 kubenswrapper[2570]: W0710 00:26:21.171394 2570 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 10 00:26:21.180460 ip-10-0-130-146 kubenswrapper[2570]: W0710 00:26:21.171397 2570 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 10 00:26:21.180460 ip-10-0-130-146 kubenswrapper[2570]: W0710 00:26:21.171399 2570 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 10 00:26:21.180460 ip-10-0-130-146 kubenswrapper[2570]: W0710 00:26:21.171402 2570 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 10 00:26:21.180460 ip-10-0-130-146 kubenswrapper[2570]: W0710 00:26:21.171405 2570 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 10 00:26:21.180460 ip-10-0-130-146 kubenswrapper[2570]: W0710 00:26:21.171410 2570 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 10 00:26:21.180460 ip-10-0-130-146 kubenswrapper[2570]: W0710 00:26:21.171413 2570 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 10 00:26:21.180460 ip-10-0-130-146 kubenswrapper[2570]: W0710 00:26:21.171418 2570 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 10 00:26:21.180460 ip-10-0-130-146 kubenswrapper[2570]: W0710 00:26:21.171421 2570 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 10 00:26:21.180460 ip-10-0-130-146 kubenswrapper[2570]: W0710 00:26:21.171424 2570 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 10 00:26:21.180460 ip-10-0-130-146 kubenswrapper[2570]: W0710 00:26:21.171427 2570 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 10 00:26:21.180460 ip-10-0-130-146 kubenswrapper[2570]: W0710 00:26:21.171430 2570 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 10 00:26:21.181025 ip-10-0-130-146 kubenswrapper[2570]: W0710 00:26:21.171432 2570 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 10 00:26:21.181025 ip-10-0-130-146 kubenswrapper[2570]: W0710 00:26:21.171435 2570 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 10 00:26:21.181025 ip-10-0-130-146 kubenswrapper[2570]: W0710 00:26:21.171439 2570 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 10 00:26:21.181025 ip-10-0-130-146 kubenswrapper[2570]: W0710 00:26:21.171441 2570 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 10 00:26:21.181025 ip-10-0-130-146 kubenswrapper[2570]: W0710 00:26:21.171444 2570 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 10 00:26:21.181025 ip-10-0-130-146 kubenswrapper[2570]: W0710 00:26:21.171447 2570 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 10 00:26:21.181025 ip-10-0-130-146 kubenswrapper[2570]: W0710 00:26:21.171450 2570 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 10 00:26:21.181025 ip-10-0-130-146 kubenswrapper[2570]: W0710 00:26:21.171453 2570 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 10 00:26:21.181025 ip-10-0-130-146 kubenswrapper[2570]: W0710 00:26:21.171455 2570 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 10 00:26:21.181025 ip-10-0-130-146 kubenswrapper[2570]: W0710 00:26:21.171458 2570 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 10 00:26:21.181025 ip-10-0-130-146 kubenswrapper[2570]: W0710 00:26:21.171461 2570 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 10 00:26:21.181025 ip-10-0-130-146 kubenswrapper[2570]: W0710 00:26:21.171463 2570 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 10 00:26:21.181025 ip-10-0-130-146 kubenswrapper[2570]: W0710 00:26:21.171466 2570 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 10 00:26:21.181025 ip-10-0-130-146 kubenswrapper[2570]: W0710 00:26:21.171471 2570 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 10 00:26:21.181025 ip-10-0-130-146 kubenswrapper[2570]: W0710 00:26:21.171474 2570 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 10 00:26:21.181025 ip-10-0-130-146 kubenswrapper[2570]: W0710 00:26:21.171477 2570 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 10 00:26:21.181025 ip-10-0-130-146 kubenswrapper[2570]: W0710 00:26:21.171480 2570 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 10 00:26:21.181025 ip-10-0-130-146 kubenswrapper[2570]: W0710 00:26:21.171483 2570 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 10 00:26:21.181025 ip-10-0-130-146 kubenswrapper[2570]: W0710 00:26:21.171486 2570 feature_gate.go:328] unrecognized feature gate: Example2 Jul 10 00:26:21.181025 ip-10-0-130-146 kubenswrapper[2570]: W0710 00:26:21.171488 2570 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 10 00:26:21.181564 ip-10-0-130-146 kubenswrapper[2570]: W0710 00:26:21.171491 2570 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 10 00:26:21.181564 ip-10-0-130-146 kubenswrapper[2570]: W0710 00:26:21.171494 2570 feature_gate.go:328] unrecognized feature gate: Example Jul 10 00:26:21.181564 ip-10-0-130-146 kubenswrapper[2570]: W0710 00:26:21.171496 2570 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 10 00:26:21.181564 ip-10-0-130-146 kubenswrapper[2570]: W0710 00:26:21.171499 2570 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 10 00:26:21.181564 ip-10-0-130-146 kubenswrapper[2570]: W0710 00:26:21.171502 2570 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 10 00:26:21.181564 ip-10-0-130-146 kubenswrapper[2570]: W0710 00:26:21.171506 2570 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 10 00:26:21.181564 ip-10-0-130-146 kubenswrapper[2570]: W0710 00:26:21.171509 2570 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 10 00:26:21.181564 ip-10-0-130-146 kubenswrapper[2570]: W0710 00:26:21.171514 2570 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 10 00:26:21.181564 ip-10-0-130-146 kubenswrapper[2570]: W0710 00:26:21.171517 2570 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 10 00:26:21.181564 ip-10-0-130-146 kubenswrapper[2570]: W0710 00:26:21.171520 2570 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 10 00:26:21.181564 ip-10-0-130-146 kubenswrapper[2570]: W0710 00:26:21.171522 2570 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 10 00:26:21.181564 ip-10-0-130-146 kubenswrapper[2570]: W0710 00:26:21.171525 2570 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 10 00:26:21.181564 ip-10-0-130-146 kubenswrapper[2570]: W0710 00:26:21.171528 2570 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 10 00:26:21.181564 ip-10-0-130-146 kubenswrapper[2570]: W0710 00:26:21.171531 2570 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 10 00:26:21.181564 ip-10-0-130-146 kubenswrapper[2570]: W0710 00:26:21.171533 2570 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 10 00:26:21.181564 ip-10-0-130-146 kubenswrapper[2570]: W0710 00:26:21.171537 2570 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 10 00:26:21.181564 ip-10-0-130-146 kubenswrapper[2570]: W0710 00:26:21.171547 2570 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 10 00:26:21.181564 ip-10-0-130-146 kubenswrapper[2570]: W0710 00:26:21.171550 2570 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 10 00:26:21.181564 ip-10-0-130-146 kubenswrapper[2570]: W0710 00:26:21.171553 2570 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 10 00:26:21.182047 ip-10-0-130-146 kubenswrapper[2570]: W0710 00:26:21.171556 2570 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 10 00:26:21.182047 ip-10-0-130-146 kubenswrapper[2570]: W0710 00:26:21.171559 2570 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 10 00:26:21.182047 ip-10-0-130-146 kubenswrapper[2570]: W0710 00:26:21.171561 2570 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 10 00:26:21.182047 ip-10-0-130-146 kubenswrapper[2570]: W0710 00:26:21.171564 2570 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 10 00:26:21.182047 ip-10-0-130-146 kubenswrapper[2570]: W0710 00:26:21.171567 2570 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 10 00:26:21.182047 ip-10-0-130-146 kubenswrapper[2570]: W0710 00:26:21.171569 2570 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 10 00:26:21.182047 ip-10-0-130-146 kubenswrapper[2570]: W0710 00:26:21.171574 2570 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 10 00:26:21.182047 ip-10-0-130-146 kubenswrapper[2570]: W0710 00:26:21.171577 2570 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 10 00:26:21.182047 ip-10-0-130-146 kubenswrapper[2570]: W0710 00:26:21.171579 2570 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 10 00:26:21.182047 ip-10-0-130-146 kubenswrapper[2570]: W0710 00:26:21.171582 2570 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 10 00:26:21.182047 ip-10-0-130-146 kubenswrapper[2570]: W0710 00:26:21.171585 2570 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 10 00:26:21.182047 ip-10-0-130-146 kubenswrapper[2570]: W0710 00:26:21.171587 2570 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 10 00:26:21.182047 ip-10-0-130-146 kubenswrapper[2570]: W0710 00:26:21.171590 2570 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 10 00:26:21.182047 ip-10-0-130-146 kubenswrapper[2570]: W0710 00:26:21.171593 2570 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 10 00:26:21.182047 ip-10-0-130-146 kubenswrapper[2570]: W0710 00:26:21.171596 2570 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 10 00:26:21.182047 ip-10-0-130-146 kubenswrapper[2570]: W0710 00:26:21.171599 2570 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 10 00:26:21.182047 ip-10-0-130-146 kubenswrapper[2570]: W0710 00:26:21.171602 2570 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 10 00:26:21.182047 ip-10-0-130-146 kubenswrapper[2570]: W0710 00:26:21.171604 2570 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 10 00:26:21.182047 ip-10-0-130-146 kubenswrapper[2570]: W0710 00:26:21.171607 2570 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 10 00:26:21.182047 ip-10-0-130-146 kubenswrapper[2570]: W0710 00:26:21.171610 2570 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 10 00:26:21.182047 ip-10-0-130-146 kubenswrapper[2570]: W0710 00:26:21.171614 2570 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 10 00:26:21.182596 ip-10-0-130-146 kubenswrapper[2570]: W0710 00:26:21.171617 2570 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 10 00:26:21.182596 ip-10-0-130-146 kubenswrapper[2570]: W0710 00:26:21.171619 2570 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 10 00:26:21.182596 ip-10-0-130-146 kubenswrapper[2570]: W0710 00:26:21.171622 2570 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 10 00:26:21.182596 ip-10-0-130-146 kubenswrapper[2570]: W0710 00:26:21.171625 2570 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 10 00:26:21.182596 ip-10-0-130-146 kubenswrapper[2570]: W0710 00:26:21.171628 2570 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 10 00:26:21.182596 ip-10-0-130-146 kubenswrapper[2570]: W0710 00:26:21.171630 2570 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 10 00:26:21.182596 ip-10-0-130-146 kubenswrapper[2570]: W0710 00:26:21.171635 2570 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 10 00:26:21.182596 ip-10-0-130-146 kubenswrapper[2570]: W0710 00:26:21.171638 2570 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 10 00:26:21.182596 ip-10-0-130-146 kubenswrapper[2570]: W0710 00:26:21.171642 2570 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 10 00:26:21.182596 ip-10-0-130-146 kubenswrapper[2570]: W0710 00:26:21.171645 2570 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 10 00:26:21.182596 ip-10-0-130-146 kubenswrapper[2570]: W0710 00:26:21.171649 2570 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 10 00:26:21.182596 ip-10-0-130-146 kubenswrapper[2570]: W0710 00:26:21.171652 2570 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 10 00:26:21.182596 ip-10-0-130-146 kubenswrapper[2570]: W0710 00:26:21.171655 2570 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 10 00:26:21.182596 ip-10-0-130-146 kubenswrapper[2570]: W0710 00:26:21.171658 2570 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 10 00:26:21.182596 ip-10-0-130-146 kubenswrapper[2570]: W0710 00:26:21.171661 2570 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 10 00:26:21.182596 ip-10-0-130-146 kubenswrapper[2570]: W0710 00:26:21.171663 2570 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 10 00:26:21.182596 ip-10-0-130-146 kubenswrapper[2570]: W0710 00:26:21.171666 2570 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 10 00:26:21.182596 ip-10-0-130-146 kubenswrapper[2570]: W0710 00:26:21.171670 2570 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 10 00:26:21.182596 ip-10-0-130-146 kubenswrapper[2570]: W0710 00:26:21.171673 2570 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 10 00:26:21.183081 ip-10-0-130-146 kubenswrapper[2570]: W0710 00:26:21.171676 2570 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 10 00:26:21.183081 ip-10-0-130-146 kubenswrapper[2570]: W0710 00:26:21.171679 2570 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 10 00:26:21.183081 ip-10-0-130-146 kubenswrapper[2570]: W0710 00:26:21.171681 2570 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 10 00:26:21.183081 ip-10-0-130-146 kubenswrapper[2570]: W0710 00:26:21.171684 2570 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 10 00:26:21.183081 ip-10-0-130-146 kubenswrapper[2570]: I0710 00:26:21.172191 2570 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]} Jul 10 00:26:21.183081 ip-10-0-130-146 kubenswrapper[2570]: I0710 00:26:21.178884 2570 server.go:529] "Kubelet version" kubeletVersion="v1.34.9" Jul 10 00:26:21.183081 ip-10-0-130-146 kubenswrapper[2570]: I0710 00:26:21.178900 2570 server.go:531] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK="" Jul 10 00:26:21.183081 ip-10-0-130-146 kubenswrapper[2570]: I0710 00:26:21.178923 2570 watchdog_linux.go:95] "Systemd watchdog is not enabled" Jul 10 00:26:21.183081 ip-10-0-130-146 kubenswrapper[2570]: I0710 00:26:21.178929 2570 watchdog_linux.go:137] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started." Jul 10 00:26:21.183081 ip-10-0-130-146 kubenswrapper[2570]: W0710 00:26:21.178993 2570 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 10 00:26:21.183081 ip-10-0-130-146 kubenswrapper[2570]: W0710 00:26:21.178999 2570 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 10 00:26:21.183081 ip-10-0-130-146 kubenswrapper[2570]: W0710 00:26:21.179003 2570 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 10 00:26:21.183081 ip-10-0-130-146 kubenswrapper[2570]: W0710 00:26:21.179007 2570 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 10 00:26:21.183081 ip-10-0-130-146 kubenswrapper[2570]: W0710 00:26:21.179010 2570 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 10 00:26:21.183081 ip-10-0-130-146 kubenswrapper[2570]: W0710 00:26:21.179013 2570 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 10 00:26:21.183487 ip-10-0-130-146 kubenswrapper[2570]: W0710 00:26:21.179016 2570 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 10 00:26:21.183487 ip-10-0-130-146 kubenswrapper[2570]: W0710 00:26:21.179020 2570 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 10 00:26:21.183487 ip-10-0-130-146 kubenswrapper[2570]: W0710 00:26:21.179023 2570 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 10 00:26:21.183487 ip-10-0-130-146 kubenswrapper[2570]: W0710 00:26:21.179025 2570 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 10 00:26:21.183487 ip-10-0-130-146 kubenswrapper[2570]: W0710 00:26:21.179028 2570 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 10 00:26:21.183487 ip-10-0-130-146 kubenswrapper[2570]: W0710 00:26:21.179032 2570 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 10 00:26:21.183487 ip-10-0-130-146 kubenswrapper[2570]: W0710 00:26:21.179036 2570 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 10 00:26:21.183487 ip-10-0-130-146 kubenswrapper[2570]: W0710 00:26:21.179040 2570 feature_gate.go:328] unrecognized feature gate: Example Jul 10 00:26:21.183487 ip-10-0-130-146 kubenswrapper[2570]: W0710 00:26:21.179043 2570 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 10 00:26:21.183487 ip-10-0-130-146 kubenswrapper[2570]: W0710 00:26:21.179045 2570 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 10 00:26:21.183487 ip-10-0-130-146 kubenswrapper[2570]: W0710 00:26:21.179048 2570 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 10 00:26:21.183487 ip-10-0-130-146 kubenswrapper[2570]: W0710 00:26:21.179051 2570 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 10 00:26:21.183487 ip-10-0-130-146 kubenswrapper[2570]: W0710 00:26:21.179054 2570 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 10 00:26:21.183487 ip-10-0-130-146 kubenswrapper[2570]: W0710 00:26:21.179057 2570 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 10 00:26:21.183487 ip-10-0-130-146 kubenswrapper[2570]: W0710 00:26:21.179060 2570 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 10 00:26:21.183487 ip-10-0-130-146 kubenswrapper[2570]: W0710 00:26:21.179063 2570 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 10 00:26:21.183487 ip-10-0-130-146 kubenswrapper[2570]: W0710 00:26:21.179066 2570 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 10 00:26:21.183487 ip-10-0-130-146 kubenswrapper[2570]: W0710 00:26:21.179068 2570 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 10 00:26:21.183487 ip-10-0-130-146 kubenswrapper[2570]: W0710 00:26:21.179071 2570 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 10 00:26:21.183487 ip-10-0-130-146 kubenswrapper[2570]: W0710 00:26:21.179074 2570 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 10 00:26:21.183988 ip-10-0-130-146 kubenswrapper[2570]: W0710 00:26:21.179077 2570 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 10 00:26:21.183988 ip-10-0-130-146 kubenswrapper[2570]: W0710 00:26:21.179080 2570 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 10 00:26:21.183988 ip-10-0-130-146 kubenswrapper[2570]: W0710 00:26:21.179083 2570 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 10 00:26:21.183988 ip-10-0-130-146 kubenswrapper[2570]: W0710 00:26:21.179088 2570 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 10 00:26:21.183988 ip-10-0-130-146 kubenswrapper[2570]: W0710 00:26:21.179091 2570 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 10 00:26:21.183988 ip-10-0-130-146 kubenswrapper[2570]: W0710 00:26:21.179093 2570 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 10 00:26:21.183988 ip-10-0-130-146 kubenswrapper[2570]: W0710 00:26:21.179096 2570 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 10 00:26:21.183988 ip-10-0-130-146 kubenswrapper[2570]: W0710 00:26:21.179099 2570 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 10 00:26:21.183988 ip-10-0-130-146 kubenswrapper[2570]: W0710 00:26:21.179102 2570 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 10 00:26:21.183988 ip-10-0-130-146 kubenswrapper[2570]: W0710 00:26:21.179105 2570 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 10 00:26:21.183988 ip-10-0-130-146 kubenswrapper[2570]: W0710 00:26:21.179108 2570 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 10 00:26:21.183988 ip-10-0-130-146 kubenswrapper[2570]: W0710 00:26:21.179111 2570 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 10 00:26:21.183988 ip-10-0-130-146 kubenswrapper[2570]: W0710 00:26:21.179114 2570 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 10 00:26:21.183988 ip-10-0-130-146 kubenswrapper[2570]: W0710 00:26:21.179116 2570 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 10 00:26:21.183988 ip-10-0-130-146 kubenswrapper[2570]: W0710 00:26:21.179119 2570 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 10 00:26:21.183988 ip-10-0-130-146 kubenswrapper[2570]: W0710 00:26:21.179136 2570 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 10 00:26:21.183988 ip-10-0-130-146 kubenswrapper[2570]: W0710 00:26:21.179139 2570 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 10 00:26:21.183988 ip-10-0-130-146 kubenswrapper[2570]: W0710 00:26:21.179142 2570 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 10 00:26:21.183988 ip-10-0-130-146 kubenswrapper[2570]: W0710 00:26:21.179145 2570 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 10 00:26:21.183988 ip-10-0-130-146 kubenswrapper[2570]: W0710 00:26:21.179149 2570 feature_gate.go:328] unrecognized feature gate: Example2 Jul 10 00:26:21.184510 ip-10-0-130-146 kubenswrapper[2570]: W0710 00:26:21.179152 2570 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 10 00:26:21.184510 ip-10-0-130-146 kubenswrapper[2570]: W0710 00:26:21.179155 2570 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 10 00:26:21.184510 ip-10-0-130-146 kubenswrapper[2570]: W0710 00:26:21.179158 2570 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 10 00:26:21.184510 ip-10-0-130-146 kubenswrapper[2570]: W0710 00:26:21.179161 2570 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 10 00:26:21.184510 ip-10-0-130-146 kubenswrapper[2570]: W0710 00:26:21.179164 2570 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 10 00:26:21.184510 ip-10-0-130-146 kubenswrapper[2570]: W0710 00:26:21.179167 2570 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 10 00:26:21.184510 ip-10-0-130-146 kubenswrapper[2570]: W0710 00:26:21.179170 2570 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 10 00:26:21.184510 ip-10-0-130-146 kubenswrapper[2570]: W0710 00:26:21.179173 2570 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 10 00:26:21.184510 ip-10-0-130-146 kubenswrapper[2570]: W0710 00:26:21.179176 2570 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 10 00:26:21.184510 ip-10-0-130-146 kubenswrapper[2570]: W0710 00:26:21.179179 2570 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 10 00:26:21.184510 ip-10-0-130-146 kubenswrapper[2570]: W0710 00:26:21.179183 2570 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 10 00:26:21.184510 ip-10-0-130-146 kubenswrapper[2570]: W0710 00:26:21.179187 2570 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 10 00:26:21.184510 ip-10-0-130-146 kubenswrapper[2570]: W0710 00:26:21.179190 2570 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 10 00:26:21.184510 ip-10-0-130-146 kubenswrapper[2570]: W0710 00:26:21.179194 2570 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 10 00:26:21.184510 ip-10-0-130-146 kubenswrapper[2570]: W0710 00:26:21.179199 2570 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 10 00:26:21.184510 ip-10-0-130-146 kubenswrapper[2570]: W0710 00:26:21.179202 2570 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 10 00:26:21.184510 ip-10-0-130-146 kubenswrapper[2570]: W0710 00:26:21.179205 2570 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 10 00:26:21.184510 ip-10-0-130-146 kubenswrapper[2570]: W0710 00:26:21.179209 2570 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 10 00:26:21.184510 ip-10-0-130-146 kubenswrapper[2570]: W0710 00:26:21.179212 2570 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 10 00:26:21.185003 ip-10-0-130-146 kubenswrapper[2570]: W0710 00:26:21.179215 2570 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 10 00:26:21.185003 ip-10-0-130-146 kubenswrapper[2570]: W0710 00:26:21.179218 2570 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 10 00:26:21.185003 ip-10-0-130-146 kubenswrapper[2570]: W0710 00:26:21.179221 2570 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 10 00:26:21.185003 ip-10-0-130-146 kubenswrapper[2570]: W0710 00:26:21.179224 2570 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 10 00:26:21.185003 ip-10-0-130-146 kubenswrapper[2570]: W0710 00:26:21.179227 2570 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 10 00:26:21.185003 ip-10-0-130-146 kubenswrapper[2570]: W0710 00:26:21.179231 2570 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 10 00:26:21.185003 ip-10-0-130-146 kubenswrapper[2570]: W0710 00:26:21.179234 2570 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 10 00:26:21.185003 ip-10-0-130-146 kubenswrapper[2570]: W0710 00:26:21.179237 2570 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 10 00:26:21.185003 ip-10-0-130-146 kubenswrapper[2570]: W0710 00:26:21.179240 2570 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 10 00:26:21.185003 ip-10-0-130-146 kubenswrapper[2570]: W0710 00:26:21.179243 2570 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 10 00:26:21.185003 ip-10-0-130-146 kubenswrapper[2570]: W0710 00:26:21.179246 2570 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 10 00:26:21.185003 ip-10-0-130-146 kubenswrapper[2570]: W0710 00:26:21.179249 2570 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 10 00:26:21.185003 ip-10-0-130-146 kubenswrapper[2570]: W0710 00:26:21.179252 2570 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 10 00:26:21.185003 ip-10-0-130-146 kubenswrapper[2570]: W0710 00:26:21.179256 2570 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 10 00:26:21.185003 ip-10-0-130-146 kubenswrapper[2570]: W0710 00:26:21.179258 2570 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 10 00:26:21.185003 ip-10-0-130-146 kubenswrapper[2570]: W0710 00:26:21.179263 2570 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 10 00:26:21.185003 ip-10-0-130-146 kubenswrapper[2570]: W0710 00:26:21.179267 2570 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 10 00:26:21.185003 ip-10-0-130-146 kubenswrapper[2570]: W0710 00:26:21.179270 2570 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 10 00:26:21.185003 ip-10-0-130-146 kubenswrapper[2570]: W0710 00:26:21.179273 2570 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 10 00:26:21.185508 ip-10-0-130-146 kubenswrapper[2570]: W0710 00:26:21.179276 2570 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 10 00:26:21.185508 ip-10-0-130-146 kubenswrapper[2570]: W0710 00:26:21.179279 2570 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 10 00:26:21.185508 ip-10-0-130-146 kubenswrapper[2570]: W0710 00:26:21.179282 2570 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 10 00:26:21.185508 ip-10-0-130-146 kubenswrapper[2570]: W0710 00:26:21.179285 2570 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 10 00:26:21.185508 ip-10-0-130-146 kubenswrapper[2570]: W0710 00:26:21.179288 2570 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 10 00:26:21.185508 ip-10-0-130-146 kubenswrapper[2570]: W0710 00:26:21.179291 2570 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 10 00:26:21.185508 ip-10-0-130-146 kubenswrapper[2570]: W0710 00:26:21.179294 2570 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 10 00:26:21.185508 ip-10-0-130-146 kubenswrapper[2570]: W0710 00:26:21.179298 2570 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 10 00:26:21.185508 ip-10-0-130-146 kubenswrapper[2570]: W0710 00:26:21.179302 2570 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 10 00:26:21.185508 ip-10-0-130-146 kubenswrapper[2570]: W0710 00:26:21.179305 2570 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 10 00:26:21.185508 ip-10-0-130-146 kubenswrapper[2570]: W0710 00:26:21.179308 2570 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 10 00:26:21.185508 ip-10-0-130-146 kubenswrapper[2570]: W0710 00:26:21.179311 2570 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 10 00:26:21.185508 ip-10-0-130-146 kubenswrapper[2570]: W0710 00:26:21.179314 2570 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 10 00:26:21.185508 ip-10-0-130-146 kubenswrapper[2570]: W0710 00:26:21.179318 2570 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 10 00:26:21.185508 ip-10-0-130-146 kubenswrapper[2570]: W0710 00:26:21.179321 2570 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 10 00:26:21.185898 ip-10-0-130-146 kubenswrapper[2570]: I0710 00:26:21.179326 2570 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]} Jul 10 00:26:21.185898 ip-10-0-130-146 kubenswrapper[2570]: W0710 00:26:21.179447 2570 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 10 00:26:21.185898 ip-10-0-130-146 kubenswrapper[2570]: W0710 00:26:21.179452 2570 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 10 00:26:21.185898 ip-10-0-130-146 kubenswrapper[2570]: W0710 00:26:21.179455 2570 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 10 00:26:21.185898 ip-10-0-130-146 kubenswrapper[2570]: W0710 00:26:21.179458 2570 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 10 00:26:21.185898 ip-10-0-130-146 kubenswrapper[2570]: W0710 00:26:21.179460 2570 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 10 00:26:21.185898 ip-10-0-130-146 kubenswrapper[2570]: W0710 00:26:21.179463 2570 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 10 00:26:21.185898 ip-10-0-130-146 kubenswrapper[2570]: W0710 00:26:21.179468 2570 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 10 00:26:21.185898 ip-10-0-130-146 kubenswrapper[2570]: W0710 00:26:21.179472 2570 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 10 00:26:21.185898 ip-10-0-130-146 kubenswrapper[2570]: W0710 00:26:21.179475 2570 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 10 00:26:21.185898 ip-10-0-130-146 kubenswrapper[2570]: W0710 00:26:21.179479 2570 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 10 00:26:21.185898 ip-10-0-130-146 kubenswrapper[2570]: W0710 00:26:21.179482 2570 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 10 00:26:21.185898 ip-10-0-130-146 kubenswrapper[2570]: W0710 00:26:21.179486 2570 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 10 00:26:21.185898 ip-10-0-130-146 kubenswrapper[2570]: W0710 00:26:21.179489 2570 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 10 00:26:21.186364 ip-10-0-130-146 kubenswrapper[2570]: W0710 00:26:21.179492 2570 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 10 00:26:21.186364 ip-10-0-130-146 kubenswrapper[2570]: W0710 00:26:21.179495 2570 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 10 00:26:21.186364 ip-10-0-130-146 kubenswrapper[2570]: W0710 00:26:21.179498 2570 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 10 00:26:21.186364 ip-10-0-130-146 kubenswrapper[2570]: W0710 00:26:21.179501 2570 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 10 00:26:21.186364 ip-10-0-130-146 kubenswrapper[2570]: W0710 00:26:21.179503 2570 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 10 00:26:21.186364 ip-10-0-130-146 kubenswrapper[2570]: W0710 00:26:21.179506 2570 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 10 00:26:21.186364 ip-10-0-130-146 kubenswrapper[2570]: W0710 00:26:21.179509 2570 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 10 00:26:21.186364 ip-10-0-130-146 kubenswrapper[2570]: W0710 00:26:21.179512 2570 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 10 00:26:21.186364 ip-10-0-130-146 kubenswrapper[2570]: W0710 00:26:21.179515 2570 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 10 00:26:21.186364 ip-10-0-130-146 kubenswrapper[2570]: W0710 00:26:21.179518 2570 feature_gate.go:328] unrecognized feature gate: Example Jul 10 00:26:21.186364 ip-10-0-130-146 kubenswrapper[2570]: W0710 00:26:21.179520 2570 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 10 00:26:21.186364 ip-10-0-130-146 kubenswrapper[2570]: W0710 00:26:21.179524 2570 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 10 00:26:21.186364 ip-10-0-130-146 kubenswrapper[2570]: W0710 00:26:21.179526 2570 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 10 00:26:21.186364 ip-10-0-130-146 kubenswrapper[2570]: W0710 00:26:21.179529 2570 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 10 00:26:21.186364 ip-10-0-130-146 kubenswrapper[2570]: W0710 00:26:21.179532 2570 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 10 00:26:21.186364 ip-10-0-130-146 kubenswrapper[2570]: W0710 00:26:21.179535 2570 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 10 00:26:21.186364 ip-10-0-130-146 kubenswrapper[2570]: W0710 00:26:21.179539 2570 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 10 00:26:21.186364 ip-10-0-130-146 kubenswrapper[2570]: W0710 00:26:21.179543 2570 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 10 00:26:21.186364 ip-10-0-130-146 kubenswrapper[2570]: W0710 00:26:21.179546 2570 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 10 00:26:21.187012 ip-10-0-130-146 kubenswrapper[2570]: W0710 00:26:21.179549 2570 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 10 00:26:21.187012 ip-10-0-130-146 kubenswrapper[2570]: W0710 00:26:21.179552 2570 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 10 00:26:21.187012 ip-10-0-130-146 kubenswrapper[2570]: W0710 00:26:21.179554 2570 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 10 00:26:21.187012 ip-10-0-130-146 kubenswrapper[2570]: W0710 00:26:21.179557 2570 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 10 00:26:21.187012 ip-10-0-130-146 kubenswrapper[2570]: W0710 00:26:21.179560 2570 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 10 00:26:21.187012 ip-10-0-130-146 kubenswrapper[2570]: W0710 00:26:21.179565 2570 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 10 00:26:21.187012 ip-10-0-130-146 kubenswrapper[2570]: W0710 00:26:21.179568 2570 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 10 00:26:21.187012 ip-10-0-130-146 kubenswrapper[2570]: W0710 00:26:21.179572 2570 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 10 00:26:21.187012 ip-10-0-130-146 kubenswrapper[2570]: W0710 00:26:21.179575 2570 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 10 00:26:21.187012 ip-10-0-130-146 kubenswrapper[2570]: W0710 00:26:21.179578 2570 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 10 00:26:21.187012 ip-10-0-130-146 kubenswrapper[2570]: W0710 00:26:21.179581 2570 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 10 00:26:21.187012 ip-10-0-130-146 kubenswrapper[2570]: W0710 00:26:21.179584 2570 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 10 00:26:21.187012 ip-10-0-130-146 kubenswrapper[2570]: W0710 00:26:21.179587 2570 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 10 00:26:21.187012 ip-10-0-130-146 kubenswrapper[2570]: W0710 00:26:21.179590 2570 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 10 00:26:21.187012 ip-10-0-130-146 kubenswrapper[2570]: W0710 00:26:21.179593 2570 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 10 00:26:21.187012 ip-10-0-130-146 kubenswrapper[2570]: W0710 00:26:21.179596 2570 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 10 00:26:21.187012 ip-10-0-130-146 kubenswrapper[2570]: W0710 00:26:21.179599 2570 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 10 00:26:21.187012 ip-10-0-130-146 kubenswrapper[2570]: W0710 00:26:21.179602 2570 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 10 00:26:21.187012 ip-10-0-130-146 kubenswrapper[2570]: W0710 00:26:21.179605 2570 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 10 00:26:21.187012 ip-10-0-130-146 kubenswrapper[2570]: W0710 00:26:21.179607 2570 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 10 00:26:21.187653 ip-10-0-130-146 kubenswrapper[2570]: W0710 00:26:21.179610 2570 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 10 00:26:21.187653 ip-10-0-130-146 kubenswrapper[2570]: W0710 00:26:21.179613 2570 feature_gate.go:328] unrecognized feature gate: Example2 Jul 10 00:26:21.187653 ip-10-0-130-146 kubenswrapper[2570]: W0710 00:26:21.179616 2570 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 10 00:26:21.187653 ip-10-0-130-146 kubenswrapper[2570]: W0710 00:26:21.179619 2570 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 10 00:26:21.187653 ip-10-0-130-146 kubenswrapper[2570]: W0710 00:26:21.179622 2570 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 10 00:26:21.187653 ip-10-0-130-146 kubenswrapper[2570]: W0710 00:26:21.179624 2570 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 10 00:26:21.187653 ip-10-0-130-146 kubenswrapper[2570]: W0710 00:26:21.179627 2570 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 10 00:26:21.187653 ip-10-0-130-146 kubenswrapper[2570]: W0710 00:26:21.179630 2570 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 10 00:26:21.187653 ip-10-0-130-146 kubenswrapper[2570]: W0710 00:26:21.179632 2570 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 10 00:26:21.187653 ip-10-0-130-146 kubenswrapper[2570]: W0710 00:26:21.179635 2570 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 10 00:26:21.187653 ip-10-0-130-146 kubenswrapper[2570]: W0710 00:26:21.179638 2570 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 10 00:26:21.187653 ip-10-0-130-146 kubenswrapper[2570]: W0710 00:26:21.179641 2570 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 10 00:26:21.187653 ip-10-0-130-146 kubenswrapper[2570]: W0710 00:26:21.179643 2570 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 10 00:26:21.187653 ip-10-0-130-146 kubenswrapper[2570]: W0710 00:26:21.179646 2570 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 10 00:26:21.187653 ip-10-0-130-146 kubenswrapper[2570]: W0710 00:26:21.179649 2570 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 10 00:26:21.187653 ip-10-0-130-146 kubenswrapper[2570]: W0710 00:26:21.179651 2570 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 10 00:26:21.187653 ip-10-0-130-146 kubenswrapper[2570]: W0710 00:26:21.179654 2570 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 10 00:26:21.187653 ip-10-0-130-146 kubenswrapper[2570]: W0710 00:26:21.179657 2570 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 10 00:26:21.187653 ip-10-0-130-146 kubenswrapper[2570]: W0710 00:26:21.179659 2570 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 10 00:26:21.187653 ip-10-0-130-146 kubenswrapper[2570]: W0710 00:26:21.179662 2570 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 10 00:26:21.188195 ip-10-0-130-146 kubenswrapper[2570]: W0710 00:26:21.179665 2570 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 10 00:26:21.188195 ip-10-0-130-146 kubenswrapper[2570]: W0710 00:26:21.179668 2570 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 10 00:26:21.188195 ip-10-0-130-146 kubenswrapper[2570]: W0710 00:26:21.179671 2570 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 10 00:26:21.188195 ip-10-0-130-146 kubenswrapper[2570]: W0710 00:26:21.179673 2570 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 10 00:26:21.188195 ip-10-0-130-146 kubenswrapper[2570]: W0710 00:26:21.179676 2570 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 10 00:26:21.188195 ip-10-0-130-146 kubenswrapper[2570]: W0710 00:26:21.179679 2570 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 10 00:26:21.188195 ip-10-0-130-146 kubenswrapper[2570]: W0710 00:26:21.179682 2570 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 10 00:26:21.188195 ip-10-0-130-146 kubenswrapper[2570]: W0710 00:26:21.179684 2570 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 10 00:26:21.188195 ip-10-0-130-146 kubenswrapper[2570]: W0710 00:26:21.179688 2570 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 10 00:26:21.188195 ip-10-0-130-146 kubenswrapper[2570]: W0710 00:26:21.179690 2570 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 10 00:26:21.188195 ip-10-0-130-146 kubenswrapper[2570]: W0710 00:26:21.179693 2570 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 10 00:26:21.188195 ip-10-0-130-146 kubenswrapper[2570]: W0710 00:26:21.179696 2570 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 10 00:26:21.188195 ip-10-0-130-146 kubenswrapper[2570]: W0710 00:26:21.179698 2570 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 10 00:26:21.188195 ip-10-0-130-146 kubenswrapper[2570]: W0710 00:26:21.179701 2570 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 10 00:26:21.188195 ip-10-0-130-146 kubenswrapper[2570]: W0710 00:26:21.179704 2570 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 10 00:26:21.188195 ip-10-0-130-146 kubenswrapper[2570]: W0710 00:26:21.179706 2570 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 10 00:26:21.188195 ip-10-0-130-146 kubenswrapper[2570]: W0710 00:26:21.179709 2570 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 10 00:26:21.188195 ip-10-0-130-146 kubenswrapper[2570]: W0710 00:26:21.179712 2570 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 10 00:26:21.188195 ip-10-0-130-146 kubenswrapper[2570]: W0710 00:26:21.179715 2570 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 10 00:26:21.188195 ip-10-0-130-146 kubenswrapper[2570]: W0710 00:26:21.179718 2570 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 10 00:26:21.188706 ip-10-0-130-146 kubenswrapper[2570]: W0710 00:26:21.179720 2570 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 10 00:26:21.188706 ip-10-0-130-146 kubenswrapper[2570]: W0710 00:26:21.179723 2570 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 10 00:26:21.188706 ip-10-0-130-146 kubenswrapper[2570]: W0710 00:26:21.179726 2570 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 10 00:26:21.188706 ip-10-0-130-146 kubenswrapper[2570]: W0710 00:26:21.179728 2570 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 10 00:26:21.188706 ip-10-0-130-146 kubenswrapper[2570]: W0710 00:26:21.179731 2570 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 10 00:26:21.188706 ip-10-0-130-146 kubenswrapper[2570]: W0710 00:26:21.179734 2570 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 10 00:26:21.188706 ip-10-0-130-146 kubenswrapper[2570]: W0710 00:26:21.179736 2570 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 10 00:26:21.188706 ip-10-0-130-146 kubenswrapper[2570]: I0710 00:26:21.179742 2570 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]} Jul 10 00:26:21.188706 ip-10-0-130-146 kubenswrapper[2570]: I0710 00:26:21.179849 2570 server.go:962] "Client rotation is on, will bootstrap in background" Jul 10 00:26:21.188706 ip-10-0-130-146 kubenswrapper[2570]: I0710 00:26:21.181970 2570 bootstrap.go:101] "Use the bootstrap credentials to request a cert, and set kubeconfig to point to the certificate dir" Jul 10 00:26:21.188706 ip-10-0-130-146 kubenswrapper[2570]: I0710 00:26:21.183139 2570 server.go:1019] "Starting client certificate rotation" Jul 10 00:26:21.188706 ip-10-0-130-146 kubenswrapper[2570]: I0710 00:26:21.183245 2570 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet" Jul 10 00:26:21.188706 ip-10-0-130-146 kubenswrapper[2570]: I0710 00:26:21.183803 2570 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kube-apiserver-client-kubelet" Jul 10 00:26:21.189042 ip-10-0-130-146 kubenswrapper[2570]: I0710 00:26:21.185406 2570 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Jul 10 00:26:21.192538 ip-10-0-130-146 kubenswrapper[2570]: I0710 00:26:21.192516 2570 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Jul 10 00:26:21.203228 ip-10-0-130-146 kubenswrapper[2570]: I0710 00:26:21.203207 2570 log.go:25] "Validated CRI v1 runtime API" Jul 10 00:26:21.205175 ip-10-0-130-146 kubenswrapper[2570]: I0710 00:26:21.205162 2570 log.go:25] "Validated CRI v1 image API" Jul 10 00:26:21.206715 ip-10-0-130-146 kubenswrapper[2570]: I0710 00:26:21.206693 2570 server.go:1429] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd" Jul 10 00:26:21.209241 ip-10-0-130-146 kubenswrapper[2570]: I0710 00:26:21.209222 2570 fs.go:135] Filesystem UUIDs: map[7B77-95E7:/dev/nvme0n1p2 96a49108-b4b4-498a-b668-193ee5206766:/dev/nvme0n1p4 d2781641-a5a4-49a5-99ea-82e1e5d6705b:/dev/nvme0n1p3] Jul 10 00:26:21.209301 ip-10-0-130-146 kubenswrapper[2570]: I0710 00:26:21.209243 2570 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:22 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:24 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:31 fsType:tmpfs blockSize:0} composefs_0-33:{mountpoint:/ major:0 minor:33 fsType:overlay blockSize:0}] Jul 10 00:26:21.211773 ip-10-0-130-146 kubenswrapper[2570]: I0710 00:26:21.211755 2570 reflector.go:439] "Caches populated" logger="kubernetes.io/kube-apiserver-client-kubelet" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" Jul 10 00:26:21.213739 ip-10-0-130-146 kubenswrapper[2570]: I0710 00:26:21.213634 2570 manager.go:216] Machine: {Timestamp:2026-07-10 00:26:21.213134678 +0000 UTC m=+0.261513393 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3100211 MemoryCapacity:33164500992 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec2aa0eeb8b34dae8e33241d5a76c0fc SystemUUID:ec2aa0ee-b8b3-4dae-8e33-241d5a76c0fc BootID:316aeaa4-e062-4a8b-bc00-e6ce9a158570 Filesystems:[{Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:composefs_0-33 DeviceMajor:0 DeviceMinor:33 Capacity:6107136 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:31 Capacity:16582250496 Type:vfs Inodes:1048576 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true} {Device:/dev/shm DeviceMajor:0 DeviceMinor:22 Capacity:16582250496 Type:vfs Inodes:4048401 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:24 Capacity:6632902656 Type:vfs Inodes:819200 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:br-ex MacAddress:02:25:a5:29:9d:71 Speed:0 Mtu:9001} {Name:ens5 MacAddress:02:25:a5:29:9d:71 Speed:0 Mtu:9001} {Name:ovs-system MacAddress:be:bb:19:12:95:15 Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:33164500992 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:37486592 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None} Jul 10 00:26:21.213739 ip-10-0-130-146 kubenswrapper[2570]: I0710 00:26:21.213735 2570 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available. Jul 10 00:26:21.213854 ip-10-0-130-146 kubenswrapper[2570]: I0710 00:26:21.213846 2570 manager.go:232] Version: {KernelVersion:5.14.0-570.126.1.el9_6.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.6.20260702-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:} Jul 10 00:26:21.216566 ip-10-0-130-146 kubenswrapper[2570]: I0710 00:26:21.216539 2570 container_manager_linux.go:273] "Container manager verified user specified cgroup-root exists" cgroupRoot=[] Jul 10 00:26:21.216714 ip-10-0-130-146 kubenswrapper[2570]: I0710 00:26:21.216570 2570 container_manager_linux.go:278] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-130-146.ec2.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2} Jul 10 00:26:21.217254 ip-10-0-130-146 kubenswrapper[2570]: I0710 00:26:21.217244 2570 topology_manager.go:138] "Creating topology manager with none policy" Jul 10 00:26:21.217288 ip-10-0-130-146 kubenswrapper[2570]: I0710 00:26:21.217257 2570 container_manager_linux.go:309] "Creating device plugin manager" Jul 10 00:26:21.217288 ip-10-0-130-146 kubenswrapper[2570]: I0710 00:26:21.217265 2570 manager.go:141] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock" Jul 10 00:26:21.217908 ip-10-0-130-146 kubenswrapper[2570]: I0710 00:26:21.217897 2570 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock" Jul 10 00:26:21.218814 ip-10-0-130-146 kubenswrapper[2570]: I0710 00:26:21.218804 2570 container_manager_linux.go:318] "Creating Dynamic Resource Allocation (DRA) manager" Jul 10 00:26:21.222713 ip-10-0-130-146 kubenswrapper[2570]: I0710 00:26:21.222701 2570 state_mem.go:36] "Initialized new in-memory state store" Jul 10 00:26:21.222863 ip-10-0-130-146 kubenswrapper[2570]: I0710 00:26:21.222853 2570 server.go:1240] "Using root directory" path="/var/lib/kubelet" Jul 10 00:26:21.222905 ip-10-0-130-146 kubenswrapper[2570]: I0710 00:26:21.222897 2570 kubelet.go:486] "Attempting to sync node with API server" Jul 10 00:26:21.222940 ip-10-0-130-146 kubenswrapper[2570]: I0710 00:26:21.222918 2570 kubelet.go:387] "Adding static pod path" path="/etc/kubernetes/manifests" Jul 10 00:26:21.222940 ip-10-0-130-146 kubenswrapper[2570]: I0710 00:26:21.222934 2570 file.go:69] "Watching path" path="/etc/kubernetes/manifests" Jul 10 00:26:21.222998 ip-10-0-130-146 kubenswrapper[2570]: I0710 00:26:21.222943 2570 kubelet.go:398] "Adding apiserver pod source" Jul 10 00:26:21.222998 ip-10-0-130-146 kubenswrapper[2570]: I0710 00:26:21.222962 2570 apiserver.go:42] "Waiting for node sync before watching apiserver pods" Jul 10 00:26:21.223495 ip-10-0-130-146 kubenswrapper[2570]: I0710 00:26:21.223480 2570 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Jul 10 00:26:21.223571 ip-10-0-130-146 kubenswrapper[2570]: I0710 00:26:21.223510 2570 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Jul 10 00:26:21.225943 ip-10-0-130-146 kubenswrapper[2570]: I0710 00:26:21.225927 2570 kuberuntime_manager.go:301] "Container runtime initialized" containerRuntime="cri-o" version="1.34.10-2.rhaos4.21.git83ee25b.el9" apiVersion="v1" Jul 10 00:26:21.227419 ip-10-0-130-146 kubenswrapper[2570]: I0710 00:26:21.227405 2570 kubelet.go:960] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled" Jul 10 00:26:21.227463 ip-10-0-130-146 kubenswrapper[2570]: I0710 00:26:21.227437 2570 kubelet.go:984] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled" Jul 10 00:26:21.228081 ip-10-0-130-146 kubenswrapper[2570]: I0710 00:26:21.228064 2570 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume" Jul 10 00:26:21.228081 ip-10-0-130-146 kubenswrapper[2570]: I0710 00:26:21.228085 2570 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir" Jul 10 00:26:21.228281 ip-10-0-130-146 kubenswrapper[2570]: I0710 00:26:21.228094 2570 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo" Jul 10 00:26:21.228281 ip-10-0-130-146 kubenswrapper[2570]: I0710 00:26:21.228102 2570 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path" Jul 10 00:26:21.228281 ip-10-0-130-146 kubenswrapper[2570]: I0710 00:26:21.228115 2570 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs" Jul 10 00:26:21.228281 ip-10-0-130-146 kubenswrapper[2570]: I0710 00:26:21.228147 2570 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret" Jul 10 00:26:21.228281 ip-10-0-130-146 kubenswrapper[2570]: I0710 00:26:21.228161 2570 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi" Jul 10 00:26:21.228281 ip-10-0-130-146 kubenswrapper[2570]: I0710 00:26:21.228169 2570 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api" Jul 10 00:26:21.228281 ip-10-0-130-146 kubenswrapper[2570]: I0710 00:26:21.228178 2570 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc" Jul 10 00:26:21.228281 ip-10-0-130-146 kubenswrapper[2570]: I0710 00:26:21.228186 2570 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap" Jul 10 00:26:21.228281 ip-10-0-130-146 kubenswrapper[2570]: I0710 00:26:21.228199 2570 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected" Jul 10 00:26:21.228281 ip-10-0-130-146 kubenswrapper[2570]: I0710 00:26:21.228207 2570 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume" Jul 10 00:26:21.229412 ip-10-0-130-146 kubenswrapper[2570]: I0710 00:26:21.229399 2570 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi" Jul 10 00:26:21.229468 ip-10-0-130-146 kubenswrapper[2570]: I0710 00:26:21.229416 2570 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image" Jul 10 00:26:21.233040 ip-10-0-130-146 kubenswrapper[2570]: I0710 00:26:21.232825 2570 server.go:1268] "Started kubelet" Jul 10 00:26:21.233147 ip-10-0-130-146 kubenswrapper[2570]: I0710 00:26:21.232927 2570 server.go:180] "Starting to listen" address="0.0.0.0" port=10250 Jul 10 00:26:21.233271 ip-10-0-130-146 kubenswrapper[2570]: I0710 00:26:21.232936 2570 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10 Jul 10 00:26:21.233439 ip-10-0-130-146 kubenswrapper[2570]: I0710 00:26:21.233294 2570 server_v1.go:49] "podresources" method="list" useActivePods=true Jul 10 00:26:21.234257 ip-10-0-130-146 systemd[1]: Started Kubernetes Kubelet. Jul 10 00:26:21.236955 ip-10-0-130-146 kubenswrapper[2570]: I0710 00:26:21.236887 2570 server.go:249] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock" Jul 10 00:26:21.236955 ip-10-0-130-146 kubenswrapper[2570]: I0710 00:26:21.236928 2570 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-130-146.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope Jul 10 00:26:21.237235 ip-10-0-130-146 kubenswrapper[2570]: E0710 00:26:21.237212 2570 reflector.go:205] "Failed to watch" err="failed to list *v1.Node: nodes \"ip-10-0-130-146.ec2.internal\" is forbidden: User \"system:anonymous\" cannot list resource \"nodes\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Node" Jul 10 00:26:21.237462 ip-10-0-130-146 kubenswrapper[2570]: E0710 00:26:21.237436 2570 reflector.go:205] "Failed to watch" err="failed to list *v1.Service: services is forbidden: User \"system:anonymous\" cannot list resource \"services\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Service" Jul 10 00:26:21.238629 ip-10-0-130-146 kubenswrapper[2570]: I0710 00:26:21.238613 2570 server.go:310] "Adding debug handlers to kubelet server" Jul 10 00:26:21.238728 ip-10-0-130-146 kubenswrapper[2570]: I0710 00:26:21.238657 2570 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-cgf5q" Jul 10 00:26:21.238877 ip-10-0-130-146 kubenswrapper[2570]: E0710 00:26:21.238855 2570 kubelet.go:1638] "Image garbage collection failed once. Stats initialization may not have completed yet" err="failed to get imageFs info: unable to find data in memory cache" Jul 10 00:26:21.241034 ip-10-0-130-146 kubenswrapper[2570]: E0710 00:26:21.240270 2570 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-130-146.ec2.internal.18c0c605ef6dae52 default 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-130-146.ec2.internal,UID:ip-10-0-130-146.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:Starting,Message:Starting kubelet.,Source:EventSource{Component:kubelet,Host:ip-10-0-130-146.ec2.internal,},FirstTimestamp:2026-07-10 00:26:21.232795218 +0000 UTC m=+0.281173932,LastTimestamp:2026-07-10 00:26:21.232795218 +0000 UTC m=+0.281173932,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-130-146.ec2.internal,}" Jul 10 00:26:21.243197 ip-10-0-130-146 kubenswrapper[2570]: I0710 00:26:21.243178 2570 fs_resource_analyzer.go:67] "Starting FS ResourceAnalyzer" Jul 10 00:26:21.243281 ip-10-0-130-146 kubenswrapper[2570]: I0710 00:26:21.243221 2570 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving" Jul 10 00:26:21.243281 ip-10-0-130-146 kubenswrapper[2570]: E0710 00:26:21.243268 2570 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-130-146.ec2.internal\" not found" Jul 10 00:26:21.243993 ip-10-0-130-146 kubenswrapper[2570]: I0710 00:26:21.243705 2570 desired_state_of_world_populator.go:146] "Desired state populator starts to run" Jul 10 00:26:21.243993 ip-10-0-130-146 kubenswrapper[2570]: I0710 00:26:21.243708 2570 volume_manager.go:311] "The desired_state_of_world populator starts" Jul 10 00:26:21.243993 ip-10-0-130-146 kubenswrapper[2570]: I0710 00:26:21.243738 2570 volume_manager.go:313] "Starting Kubelet Volume Manager" Jul 10 00:26:21.243993 ip-10-0-130-146 kubenswrapper[2570]: I0710 00:26:21.243875 2570 reconstruct.go:97] "Volume reconstruction finished" Jul 10 00:26:21.243993 ip-10-0-130-146 kubenswrapper[2570]: I0710 00:26:21.243902 2570 reconciler.go:29] "Reconciler: start to sync state" Jul 10 00:26:21.244335 ip-10-0-130-146 kubenswrapper[2570]: I0710 00:26:21.244295 2570 factory.go:55] Registering systemd factory Jul 10 00:26:21.244335 ip-10-0-130-146 kubenswrapper[2570]: I0710 00:26:21.244313 2570 factory.go:223] Registration of the systemd container factory successfully Jul 10 00:26:21.244651 ip-10-0-130-146 kubenswrapper[2570]: I0710 00:26:21.244571 2570 factory.go:153] Registering CRI-O factory Jul 10 00:26:21.244651 ip-10-0-130-146 kubenswrapper[2570]: I0710 00:26:21.244590 2570 factory.go:223] Registration of the crio container factory successfully Jul 10 00:26:21.244799 ip-10-0-130-146 kubenswrapper[2570]: I0710 00:26:21.244657 2570 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory Jul 10 00:26:21.244799 ip-10-0-130-146 kubenswrapper[2570]: I0710 00:26:21.244683 2570 factory.go:103] Registering Raw factory Jul 10 00:26:21.244799 ip-10-0-130-146 kubenswrapper[2570]: I0710 00:26:21.244700 2570 manager.go:1196] Started watching for new ooms in manager Jul 10 00:26:21.245194 ip-10-0-130-146 kubenswrapper[2570]: I0710 00:26:21.245178 2570 manager.go:349] Starting recovery of all containers Jul 10 00:26:21.246642 ip-10-0-130-146 kubenswrapper[2570]: E0710 00:26:21.246611 2570 controller.go:145] "Failed to ensure lease exists, will retry" err="leases.coordination.k8s.io \"ip-10-0-130-146.ec2.internal\" is forbidden: User \"system:anonymous\" cannot get resource \"leases\" in API group \"coordination.k8s.io\" in the namespace \"kube-node-lease\"" interval="200ms" Jul 10 00:26:21.246820 ip-10-0-130-146 kubenswrapper[2570]: E0710 00:26:21.246795 2570 reflector.go:205] "Failed to watch" err="failed to list *v1.CSIDriver: csidrivers.storage.k8s.io is forbidden: User \"system:anonymous\" cannot list resource \"csidrivers\" in API group \"storage.k8s.io\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.CSIDriver" Jul 10 00:26:21.248938 ip-10-0-130-146 kubenswrapper[2570]: I0710 00:26:21.248910 2570 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-cgf5q" Jul 10 00:26:21.254199 ip-10-0-130-146 kubenswrapper[2570]: I0710 00:26:21.254184 2570 manager.go:354] Recovery completed Jul 10 00:26:21.258732 ip-10-0-130-146 kubenswrapper[2570]: I0710 00:26:21.258718 2570 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 10 00:26:21.260480 ip-10-0-130-146 kubenswrapper[2570]: I0710 00:26:21.260463 2570 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-130-146.ec2.internal" event="NodeHasSufficientMemory" Jul 10 00:26:21.260550 ip-10-0-130-146 kubenswrapper[2570]: I0710 00:26:21.260490 2570 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-130-146.ec2.internal" event="NodeHasNoDiskPressure" Jul 10 00:26:21.260550 ip-10-0-130-146 kubenswrapper[2570]: I0710 00:26:21.260501 2570 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-130-146.ec2.internal" event="NodeHasSufficientPID" Jul 10 00:26:21.260975 ip-10-0-130-146 kubenswrapper[2570]: I0710 00:26:21.260962 2570 cpu_manager.go:222] "Starting CPU manager" policy="none" Jul 10 00:26:21.260975 ip-10-0-130-146 kubenswrapper[2570]: I0710 00:26:21.260975 2570 cpu_manager.go:223] "Reconciling" reconcilePeriod="10s" Jul 10 00:26:21.261047 ip-10-0-130-146 kubenswrapper[2570]: I0710 00:26:21.260990 2570 state_mem.go:36] "Initialized new in-memory state store" Jul 10 00:26:21.261047 ip-10-0-130-146 kubenswrapper[2570]: I0710 00:26:21.261027 2570 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 10 00:26:21.261674 ip-10-0-130-146 kubenswrapper[2570]: I0710 00:26:21.261661 2570 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-130-146.ec2.internal" event="NodeHasSufficientMemory" Jul 10 00:26:21.261720 ip-10-0-130-146 kubenswrapper[2570]: I0710 00:26:21.261686 2570 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-130-146.ec2.internal" event="NodeHasNoDiskPressure" Jul 10 00:26:21.261720 ip-10-0-130-146 kubenswrapper[2570]: I0710 00:26:21.261699 2570 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-130-146.ec2.internal" event="NodeHasSufficientPID" Jul 10 00:26:21.262822 ip-10-0-130-146 kubenswrapper[2570]: E0710 00:26:21.262734 2570 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-130-146.ec2.internal.18c0c605f1141faa default 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-130-146.ec2.internal,UID:ip-10-0-130-146.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeHasSufficientMemory,Message:Node ip-10-0-130-146.ec2.internal status is now: NodeHasSufficientMemory,Source:EventSource{Component:kubelet,Host:ip-10-0-130-146.ec2.internal,},FirstTimestamp:2026-07-10 00:26:21.260480426 +0000 UTC m=+0.308859141,LastTimestamp:2026-07-10 00:26:21.260480426 +0000 UTC m=+0.308859141,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-130-146.ec2.internal,}" Jul 10 00:26:21.263217 ip-10-0-130-146 kubenswrapper[2570]: I0710 00:26:21.263205 2570 policy_none.go:49] "None policy: Start" Jul 10 00:26:21.263260 ip-10-0-130-146 kubenswrapper[2570]: I0710 00:26:21.263223 2570 memory_manager.go:187] "Starting memorymanager" policy="None" Jul 10 00:26:21.263260 ip-10-0-130-146 kubenswrapper[2570]: I0710 00:26:21.263234 2570 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint" Jul 10 00:26:21.265345 ip-10-0-130-146 kubenswrapper[2570]: I0710 00:26:21.265334 2570 policy_none.go:47] "Start" Jul 10 00:26:21.295732 ip-10-0-130-146 kubenswrapper[2570]: I0710 00:26:21.295714 2570 manager.go:337] "Starting Device Plugin manager" Jul 10 00:26:21.295835 ip-10-0-130-146 kubenswrapper[2570]: E0710 00:26:21.295802 2570 manager.go:513] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint" Jul 10 00:26:21.295835 ip-10-0-130-146 kubenswrapper[2570]: I0710 00:26:21.295817 2570 server.go:87] "Starting device plugin registration server" Jul 10 00:26:21.296078 ip-10-0-130-146 kubenswrapper[2570]: I0710 00:26:21.296064 2570 eviction_manager.go:189] "Eviction manager: starting control loop" Jul 10 00:26:21.296158 ip-10-0-130-146 kubenswrapper[2570]: I0710 00:26:21.296098 2570 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s" Jul 10 00:26:21.296218 ip-10-0-130-146 kubenswrapper[2570]: I0710 00:26:21.296186 2570 plugin_watcher.go:51] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry" Jul 10 00:26:21.296308 ip-10-0-130-146 kubenswrapper[2570]: I0710 00:26:21.296291 2570 plugin_manager.go:116] "The desired_state_of_world populator (plugin watcher) starts" Jul 10 00:26:21.296488 ip-10-0-130-146 kubenswrapper[2570]: I0710 00:26:21.296317 2570 plugin_manager.go:118] "Starting Kubelet Plugin Manager" Jul 10 00:26:21.296811 ip-10-0-130-146 kubenswrapper[2570]: E0710 00:26:21.296788 2570 eviction_manager.go:267] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\"" Jul 10 00:26:21.296916 ip-10-0-130-146 kubenswrapper[2570]: E0710 00:26:21.296843 2570 eviction_manager.go:292] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-130-146.ec2.internal\" not found" Jul 10 00:26:21.397097 ip-10-0-130-146 kubenswrapper[2570]: I0710 00:26:21.396998 2570 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 10 00:26:21.399308 ip-10-0-130-146 kubenswrapper[2570]: I0710 00:26:21.398191 2570 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-130-146.ec2.internal" event="NodeHasSufficientMemory" Jul 10 00:26:21.399308 ip-10-0-130-146 kubenswrapper[2570]: I0710 00:26:21.398222 2570 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-130-146.ec2.internal" event="NodeHasNoDiskPressure" Jul 10 00:26:21.399308 ip-10-0-130-146 kubenswrapper[2570]: I0710 00:26:21.398233 2570 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-130-146.ec2.internal" event="NodeHasSufficientPID" Jul 10 00:26:21.399308 ip-10-0-130-146 kubenswrapper[2570]: I0710 00:26:21.398268 2570 kubelet_node_status.go:78] "Attempting to register node" node="ip-10-0-130-146.ec2.internal" Jul 10 00:26:21.408654 ip-10-0-130-146 kubenswrapper[2570]: I0710 00:26:21.408636 2570 kubelet_node_status.go:81] "Successfully registered node" node="ip-10-0-130-146.ec2.internal" Jul 10 00:26:21.408706 ip-10-0-130-146 kubenswrapper[2570]: E0710 00:26:21.408659 2570 kubelet_node_status.go:536] "Error updating node status, will retry" err="error getting node \"ip-10-0-130-146.ec2.internal\": node \"ip-10-0-130-146.ec2.internal\" not found" Jul 10 00:26:21.427336 ip-10-0-130-146 kubenswrapper[2570]: E0710 00:26:21.427314 2570 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-130-146.ec2.internal\" not found" Jul 10 00:26:21.446175 ip-10-0-130-146 kubenswrapper[2570]: I0710 00:26:21.446153 2570 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4" Jul 10 00:26:21.447491 ip-10-0-130-146 kubenswrapper[2570]: I0710 00:26:21.447466 2570 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6" Jul 10 00:26:21.447585 ip-10-0-130-146 kubenswrapper[2570]: I0710 00:26:21.447510 2570 status_manager.go:244] "Starting to sync pod status with apiserver" Jul 10 00:26:21.447585 ip-10-0-130-146 kubenswrapper[2570]: I0710 00:26:21.447535 2570 kubelet.go:2448] "Starting kubelet main sync loop" Jul 10 00:26:21.447585 ip-10-0-130-146 kubenswrapper[2570]: E0710 00:26:21.447574 2570 kubelet.go:2472] "Skipping pod synchronization" err="PLEG is not healthy: pleg has yet to be successful" Jul 10 00:26:21.449827 ip-10-0-130-146 kubenswrapper[2570]: I0710 00:26:21.449810 2570 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160" Jul 10 00:26:21.528362 ip-10-0-130-146 kubenswrapper[2570]: E0710 00:26:21.528331 2570 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-130-146.ec2.internal\" not found" Jul 10 00:26:21.547754 ip-10-0-130-146 kubenswrapper[2570]: I0710 00:26:21.547730 2570 kubelet.go:2535] "SyncLoop ADD" source="file" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-146.ec2.internal","kube-system/kube-apiserver-proxy-ip-10-0-130-146.ec2.internal"] Jul 10 00:26:21.547813 ip-10-0-130-146 kubenswrapper[2570]: I0710 00:26:21.547797 2570 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 10 00:26:21.548701 ip-10-0-130-146 kubenswrapper[2570]: I0710 00:26:21.548683 2570 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-130-146.ec2.internal" event="NodeHasSufficientMemory" Jul 10 00:26:21.548793 ip-10-0-130-146 kubenswrapper[2570]: I0710 00:26:21.548718 2570 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-130-146.ec2.internal" event="NodeHasNoDiskPressure" Jul 10 00:26:21.548793 ip-10-0-130-146 kubenswrapper[2570]: I0710 00:26:21.548748 2570 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-130-146.ec2.internal" event="NodeHasSufficientPID" Jul 10 00:26:21.551175 ip-10-0-130-146 kubenswrapper[2570]: I0710 00:26:21.551161 2570 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 10 00:26:21.551308 ip-10-0-130-146 kubenswrapper[2570]: I0710 00:26:21.551292 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-146.ec2.internal" Jul 10 00:26:21.551348 ip-10-0-130-146 kubenswrapper[2570]: I0710 00:26:21.551329 2570 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 10 00:26:21.551974 ip-10-0-130-146 kubenswrapper[2570]: I0710 00:26:21.551956 2570 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-130-146.ec2.internal" event="NodeHasSufficientMemory" Jul 10 00:26:21.552076 ip-10-0-130-146 kubenswrapper[2570]: I0710 00:26:21.551983 2570 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-130-146.ec2.internal" event="NodeHasNoDiskPressure" Jul 10 00:26:21.552076 ip-10-0-130-146 kubenswrapper[2570]: I0710 00:26:21.551996 2570 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-130-146.ec2.internal" event="NodeHasSufficientPID" Jul 10 00:26:21.552076 ip-10-0-130-146 kubenswrapper[2570]: I0710 00:26:21.552001 2570 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-130-146.ec2.internal" event="NodeHasSufficientMemory" Jul 10 00:26:21.552076 ip-10-0-130-146 kubenswrapper[2570]: I0710 00:26:21.552021 2570 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-130-146.ec2.internal" event="NodeHasNoDiskPressure" Jul 10 00:26:21.552076 ip-10-0-130-146 kubenswrapper[2570]: I0710 00:26:21.552033 2570 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-130-146.ec2.internal" event="NodeHasSufficientPID" Jul 10 00:26:21.554394 ip-10-0-130-146 kubenswrapper[2570]: I0710 00:26:21.554377 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-130-146.ec2.internal" Jul 10 00:26:21.554487 ip-10-0-130-146 kubenswrapper[2570]: I0710 00:26:21.554403 2570 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 10 00:26:21.555102 ip-10-0-130-146 kubenswrapper[2570]: I0710 00:26:21.555088 2570 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-130-146.ec2.internal" event="NodeHasSufficientMemory" Jul 10 00:26:21.555190 ip-10-0-130-146 kubenswrapper[2570]: I0710 00:26:21.555114 2570 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-130-146.ec2.internal" event="NodeHasNoDiskPressure" Jul 10 00:26:21.555190 ip-10-0-130-146 kubenswrapper[2570]: I0710 00:26:21.555130 2570 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-130-146.ec2.internal" event="NodeHasSufficientPID" Jul 10 00:26:21.568707 ip-10-0-130-146 kubenswrapper[2570]: E0710 00:26:21.568684 2570 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-130-146.ec2.internal\" not found" node="ip-10-0-130-146.ec2.internal" Jul 10 00:26:21.573271 ip-10-0-130-146 kubenswrapper[2570]: E0710 00:26:21.573256 2570 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-130-146.ec2.internal\" not found" node="ip-10-0-130-146.ec2.internal" Jul 10 00:26:21.628840 ip-10-0-130-146 kubenswrapper[2570]: E0710 00:26:21.628815 2570 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-130-146.ec2.internal\" not found" Jul 10 00:26:21.646328 ip-10-0-130-146 kubenswrapper[2570]: I0710 00:26:21.646302 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/277746f2e3b301c16ca7c3a5af84dd5d-config\") pod \"kube-apiserver-proxy-ip-10-0-130-146.ec2.internal\" (UID: \"277746f2e3b301c16ca7c3a5af84dd5d\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-130-146.ec2.internal" Jul 10 00:26:21.646422 ip-10-0-130-146 kubenswrapper[2570]: I0710 00:26:21.646337 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/0ddb9c9bc924902db48bbad9696eddc0-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-130-146.ec2.internal\" (UID: \"0ddb9c9bc924902db48bbad9696eddc0\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-146.ec2.internal" Jul 10 00:26:21.646422 ip-10-0-130-146 kubenswrapper[2570]: I0710 00:26:21.646365 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/0ddb9c9bc924902db48bbad9696eddc0-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-130-146.ec2.internal\" (UID: \"0ddb9c9bc924902db48bbad9696eddc0\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-146.ec2.internal" Jul 10 00:26:21.729166 ip-10-0-130-146 kubenswrapper[2570]: E0710 00:26:21.729094 2570 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-130-146.ec2.internal\" not found" Jul 10 00:26:21.746681 ip-10-0-130-146 kubenswrapper[2570]: I0710 00:26:21.746656 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/0ddb9c9bc924902db48bbad9696eddc0-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-130-146.ec2.internal\" (UID: \"0ddb9c9bc924902db48bbad9696eddc0\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-146.ec2.internal" Jul 10 00:26:21.746780 ip-10-0-130-146 kubenswrapper[2570]: I0710 00:26:21.746691 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/0ddb9c9bc924902db48bbad9696eddc0-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-130-146.ec2.internal\" (UID: \"0ddb9c9bc924902db48bbad9696eddc0\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-146.ec2.internal" Jul 10 00:26:21.746780 ip-10-0-130-146 kubenswrapper[2570]: I0710 00:26:21.746716 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/277746f2e3b301c16ca7c3a5af84dd5d-config\") pod \"kube-apiserver-proxy-ip-10-0-130-146.ec2.internal\" (UID: \"277746f2e3b301c16ca7c3a5af84dd5d\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-130-146.ec2.internal" Jul 10 00:26:21.746780 ip-10-0-130-146 kubenswrapper[2570]: I0710 00:26:21.746730 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/0ddb9c9bc924902db48bbad9696eddc0-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-130-146.ec2.internal\" (UID: \"0ddb9c9bc924902db48bbad9696eddc0\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-146.ec2.internal" Jul 10 00:26:21.746780 ip-10-0-130-146 kubenswrapper[2570]: I0710 00:26:21.746772 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/277746f2e3b301c16ca7c3a5af84dd5d-config\") pod \"kube-apiserver-proxy-ip-10-0-130-146.ec2.internal\" (UID: \"277746f2e3b301c16ca7c3a5af84dd5d\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-130-146.ec2.internal" Jul 10 00:26:21.746927 ip-10-0-130-146 kubenswrapper[2570]: I0710 00:26:21.746775 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/0ddb9c9bc924902db48bbad9696eddc0-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-130-146.ec2.internal\" (UID: \"0ddb9c9bc924902db48bbad9696eddc0\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-146.ec2.internal" Jul 10 00:26:21.834796 ip-10-0-130-146 kubenswrapper[2570]: E0710 00:26:21.834753 2570 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-130-146.ec2.internal\" not found" Jul 10 00:26:21.872474 ip-10-0-130-146 kubenswrapper[2570]: I0710 00:26:21.872442 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-146.ec2.internal" Jul 10 00:26:21.875805 ip-10-0-130-146 kubenswrapper[2570]: I0710 00:26:21.875785 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-130-146.ec2.internal" Jul 10 00:26:21.934961 ip-10-0-130-146 kubenswrapper[2570]: E0710 00:26:21.934918 2570 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-130-146.ec2.internal\" not found" Jul 10 00:26:22.041608 ip-10-0-130-146 kubenswrapper[2570]: E0710 00:26:22.041531 2570 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-130-146.ec2.internal\" not found" Jul 10 00:26:22.141719 ip-10-0-130-146 kubenswrapper[2570]: E0710 00:26:22.141694 2570 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-130-146.ec2.internal\" not found" Jul 10 00:26:22.182566 ip-10-0-130-146 kubenswrapper[2570]: I0710 00:26:22.182534 2570 transport.go:147] "Certificate rotation detected, shutting down client connections to start using new credentials" Jul 10 00:26:22.183247 ip-10-0-130-146 kubenswrapper[2570]: E0710 00:26:22.182637 2570 request.go:1196] "Unexpected error when reading response body" err="read tcp 10.0.130.146:49878->18.235.93.85:6443: use of closed network connection" Jul 10 00:26:22.183247 ip-10-0-130-146 kubenswrapper[2570]: E0710 00:26:22.182692 2570 reflector.go:205] "Failed to watch" err="failed to list *v1.Node: unexpected error when reading response body. Please retry. Original error: read tcp 10.0.130.146:49878->18.235.93.85:6443: use of closed network connection" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Node" Jul 10 00:26:22.183247 ip-10-0-130-146 kubenswrapper[2570]: I0710 00:26:22.182704 2570 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.RuntimeClass" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received" Jul 10 00:26:22.241841 ip-10-0-130-146 kubenswrapper[2570]: E0710 00:26:22.241815 2570 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-130-146.ec2.internal\" not found" Jul 10 00:26:22.256543 ip-10-0-130-146 kubenswrapper[2570]: I0710 00:26:22.256513 2570 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-07-09 00:21:21 +0000 UTC" deadline="2028-03-24 01:43:33.575557756 +0000 UTC" Jul 10 00:26:22.256543 ip-10-0-130-146 kubenswrapper[2570]: I0710 00:26:22.256540 2570 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="14953h17m11.319020811s" Jul 10 00:26:22.342770 ip-10-0-130-146 kubenswrapper[2570]: E0710 00:26:22.342702 2570 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-130-146.ec2.internal\" not found" Jul 10 00:26:22.443542 ip-10-0-130-146 kubenswrapper[2570]: E0710 00:26:22.443507 2570 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-130-146.ec2.internal\" not found" Jul 10 00:26:22.447051 ip-10-0-130-146 kubenswrapper[2570]: I0710 00:26:22.447032 2570 reflector.go:439] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:160" Jul 10 00:26:22.483455 ip-10-0-130-146 kubenswrapper[2570]: W0710 00:26:22.483419 2570 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0ddb9c9bc924902db48bbad9696eddc0.slice/crio-8d15e4c9f4a27c838d6de8d07d50ec327b5ea6d579cb7831aa156bf24c16fa30 WatchSource:0}: Error finding container 8d15e4c9f4a27c838d6de8d07d50ec327b5ea6d579cb7831aa156bf24c16fa30: Status 404 returned error can't find the container with id 8d15e4c9f4a27c838d6de8d07d50ec327b5ea6d579cb7831aa156bf24c16fa30 Jul 10 00:26:22.492641 ip-10-0-130-146 kubenswrapper[2570]: I0710 00:26:22.492625 2570 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Jul 10 00:26:22.500156 ip-10-0-130-146 kubenswrapper[2570]: W0710 00:26:22.500110 2570 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod277746f2e3b301c16ca7c3a5af84dd5d.slice/crio-d92e8c97d3d62bff8b01d0ccb719d9ac0a8113857429d36115601cd837ae06bc WatchSource:0}: Error finding container d92e8c97d3d62bff8b01d0ccb719d9ac0a8113857429d36115601cd837ae06bc: Status 404 returned error can't find the container with id d92e8c97d3d62bff8b01d0ccb719d9ac0a8113857429d36115601cd837ae06bc Jul 10 00:26:22.544630 ip-10-0-130-146 kubenswrapper[2570]: E0710 00:26:22.544605 2570 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-130-146.ec2.internal\" not found" Jul 10 00:26:22.584326 ip-10-0-130-146 kubenswrapper[2570]: I0710 00:26:22.584310 2570 reflector.go:439] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:160" Jul 10 00:26:22.645188 ip-10-0-130-146 kubenswrapper[2570]: E0710 00:26:22.645159 2570 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-130-146.ec2.internal\" not found" Jul 10 00:26:22.745559 ip-10-0-130-146 kubenswrapper[2570]: E0710 00:26:22.745520 2570 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-130-146.ec2.internal\" not found" Jul 10 00:26:22.846481 ip-10-0-130-146 kubenswrapper[2570]: E0710 00:26:22.846451 2570 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-130-146.ec2.internal\" not found" Jul 10 00:26:22.947474 ip-10-0-130-146 kubenswrapper[2570]: E0710 00:26:22.947391 2570 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-130-146.ec2.internal\" not found" Jul 10 00:26:23.048133 ip-10-0-130-146 kubenswrapper[2570]: E0710 00:26:23.048091 2570 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-130-146.ec2.internal\" not found" Jul 10 00:26:23.148766 ip-10-0-130-146 kubenswrapper[2570]: E0710 00:26:23.148722 2570 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-130-146.ec2.internal\" not found" Jul 10 00:26:23.249174 ip-10-0-130-146 kubenswrapper[2570]: E0710 00:26:23.249070 2570 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-130-146.ec2.internal\" not found" Jul 10 00:26:23.350144 ip-10-0-130-146 kubenswrapper[2570]: E0710 00:26:23.350112 2570 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-130-146.ec2.internal\" not found" Jul 10 00:26:23.420193 ip-10-0-130-146 kubenswrapper[2570]: I0710 00:26:23.420157 2570 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160" Jul 10 00:26:23.450287 ip-10-0-130-146 kubenswrapper[2570]: E0710 00:26:23.450242 2570 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-130-146.ec2.internal\" not found" Jul 10 00:26:23.452560 ip-10-0-130-146 kubenswrapper[2570]: I0710 00:26:23.452507 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-146.ec2.internal" event={"ID":"0ddb9c9bc924902db48bbad9696eddc0","Type":"ContainerStarted","Data":"8d15e4c9f4a27c838d6de8d07d50ec327b5ea6d579cb7831aa156bf24c16fa30"} Jul 10 00:26:23.453856 ip-10-0-130-146 kubenswrapper[2570]: I0710 00:26:23.453833 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-130-146.ec2.internal" event={"ID":"277746f2e3b301c16ca7c3a5af84dd5d","Type":"ContainerStarted","Data":"d92e8c97d3d62bff8b01d0ccb719d9ac0a8113857429d36115601cd837ae06bc"} Jul 10 00:26:23.550604 ip-10-0-130-146 kubenswrapper[2570]: E0710 00:26:23.550503 2570 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-130-146.ec2.internal\" not found" Jul 10 00:26:23.651309 ip-10-0-130-146 kubenswrapper[2570]: E0710 00:26:23.651268 2570 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-130-146.ec2.internal\" not found" Jul 10 00:26:23.752294 ip-10-0-130-146 kubenswrapper[2570]: E0710 00:26:23.752253 2570 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-130-146.ec2.internal\" not found" Jul 10 00:26:23.853218 ip-10-0-130-146 kubenswrapper[2570]: E0710 00:26:23.853128 2570 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-130-146.ec2.internal\" not found" Jul 10 00:26:23.953881 ip-10-0-130-146 kubenswrapper[2570]: E0710 00:26:23.953847 2570 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-130-146.ec2.internal\" not found" Jul 10 00:26:24.054930 ip-10-0-130-146 kubenswrapper[2570]: E0710 00:26:24.054894 2570 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-130-146.ec2.internal\" not found" Jul 10 00:26:24.155402 ip-10-0-130-146 kubenswrapper[2570]: E0710 00:26:24.155372 2570 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-130-146.ec2.internal\" not found" Jul 10 00:26:24.256336 ip-10-0-130-146 kubenswrapper[2570]: E0710 00:26:24.256300 2570 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-130-146.ec2.internal\" not found" Jul 10 00:26:24.357125 ip-10-0-130-146 kubenswrapper[2570]: E0710 00:26:24.357090 2570 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-130-146.ec2.internal\" not found" Jul 10 00:26:24.457783 ip-10-0-130-146 kubenswrapper[2570]: E0710 00:26:24.457707 2570 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-130-146.ec2.internal\" not found" Jul 10 00:26:24.558248 ip-10-0-130-146 kubenswrapper[2570]: E0710 00:26:24.558211 2570 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-130-146.ec2.internal\" not found" Jul 10 00:26:24.659464 ip-10-0-130-146 kubenswrapper[2570]: E0710 00:26:24.659117 2570 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-130-146.ec2.internal\" not found" Jul 10 00:26:24.760431 ip-10-0-130-146 kubenswrapper[2570]: E0710 00:26:24.760340 2570 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-130-146.ec2.internal\" not found" Jul 10 00:26:24.861124 ip-10-0-130-146 kubenswrapper[2570]: E0710 00:26:24.861096 2570 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-130-146.ec2.internal\" not found" Jul 10 00:26:24.961938 ip-10-0-130-146 kubenswrapper[2570]: E0710 00:26:24.961908 2570 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-130-146.ec2.internal\" not found" Jul 10 00:26:25.062460 ip-10-0-130-146 kubenswrapper[2570]: E0710 00:26:25.062375 2570 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-130-146.ec2.internal\" not found" Jul 10 00:26:25.100283 ip-10-0-130-146 kubenswrapper[2570]: I0710 00:26:25.100256 2570 reflector.go:439] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:160" Jul 10 00:26:25.144338 ip-10-0-130-146 kubenswrapper[2570]: I0710 00:26:25.144314 2570 kubelet.go:3248] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-146.ec2.internal" Jul 10 00:26:25.154448 ip-10-0-130-146 kubenswrapper[2570]: I0710 00:26:25.154432 2570 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Jul 10 00:26:25.155237 ip-10-0-130-146 kubenswrapper[2570]: I0710 00:26:25.155224 2570 kubelet.go:3248] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-130-146.ec2.internal" Jul 10 00:26:25.162862 ip-10-0-130-146 kubenswrapper[2570]: I0710 00:26:25.162842 2570 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Jul 10 00:26:25.223994 ip-10-0-130-146 kubenswrapper[2570]: I0710 00:26:25.223972 2570 apiserver.go:52] "Watching apiserver" Jul 10 00:26:25.231896 ip-10-0-130-146 kubenswrapper[2570]: I0710 00:26:25.231881 2570 reflector.go:439] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66" Jul 10 00:26:25.232679 ip-10-0-130-146 kubenswrapper[2570]: I0710 00:26:25.232661 2570 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ovn-kubernetes/ovnkube-node-mbs58","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-68tgb","openshift-cluster-node-tuning-operator/tuned-7hmqd","openshift-dns/node-resolver-7tn6x","openshift-image-registry/node-ca-t526b","openshift-multus/network-metrics-daemon-d8t56","openshift-network-operator/iptables-alerter-bw9kx","kube-system/global-pull-secret-syncer-6bqvb","kube-system/konnectivity-agent-995ch","kube-system/kube-apiserver-proxy-ip-10-0-130-146.ec2.internal","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-146.ec2.internal","openshift-multus/multus-2pdkh","openshift-multus/multus-additional-cni-plugins-7qfbg","openshift-network-diagnostics/network-check-target-5hs2b"] Jul 10 00:26:25.235059 ip-10-0-130-146 kubenswrapper[2570]: I0710 00:26:25.235036 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-bw9kx" Jul 10 00:26:25.237119 ip-10-0-130-146 kubenswrapper[2570]: I0710 00:26:25.237096 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-7qfbg" Jul 10 00:26:25.238022 ip-10-0-130-146 kubenswrapper[2570]: I0710 00:26:25.238007 2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-8kdfn\"" Jul 10 00:26:25.238145 ip-10-0-130-146 kubenswrapper[2570]: I0710 00:26:25.238124 2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\"" Jul 10 00:26:25.238230 ip-10-0-130-146 kubenswrapper[2570]: I0710 00:26:25.238189 2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\"" Jul 10 00:26:25.238230 ip-10-0-130-146 kubenswrapper[2570]: I0710 00:26:25.238189 2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\"" Jul 10 00:26:25.239130 ip-10-0-130-146 kubenswrapper[2570]: I0710 00:26:25.239116 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-7hmqd" Jul 10 00:26:25.239831 ip-10-0-130-146 kubenswrapper[2570]: I0710 00:26:25.239815 2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-cls26\"" Jul 10 00:26:25.239912 ip-10-0-130-146 kubenswrapper[2570]: I0710 00:26:25.239832 2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\"" Jul 10 00:26:25.239912 ip-10-0-130-146 kubenswrapper[2570]: I0710 00:26:25.239871 2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\"" Jul 10 00:26:25.240154 ip-10-0-130-146 kubenswrapper[2570]: I0710 00:26:25.240133 2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\"" Jul 10 00:26:25.240154 ip-10-0-130-146 kubenswrapper[2570]: I0710 00:26:25.240133 2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\"" Jul 10 00:26:25.240284 ip-10-0-130-146 kubenswrapper[2570]: I0710 00:26:25.240171 2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\"" Jul 10 00:26:25.241338 ip-10-0-130-146 kubenswrapper[2570]: I0710 00:26:25.241319 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-7tn6x" Jul 10 00:26:25.241729 ip-10-0-130-146 kubenswrapper[2570]: I0710 00:26:25.241712 2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\"" Jul 10 00:26:25.241792 ip-10-0-130-146 kubenswrapper[2570]: I0710 00:26:25.241753 2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\"" Jul 10 00:26:25.241792 ip-10-0-130-146 kubenswrapper[2570]: I0710 00:26:25.241777 2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-hhs5p\"" Jul 10 00:26:25.243553 ip-10-0-130-146 kubenswrapper[2570]: I0710 00:26:25.243508 2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\"" Jul 10 00:26:25.243553 ip-10-0-130-146 kubenswrapper[2570]: I0710 00:26:25.243537 2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\"" Jul 10 00:26:25.243553 ip-10-0-130-146 kubenswrapper[2570]: I0710 00:26:25.243546 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-t526b" Jul 10 00:26:25.243749 ip-10-0-130-146 kubenswrapper[2570]: I0710 00:26:25.243594 2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-2dvwh\"" Jul 10 00:26:25.245570 ip-10-0-130-146 kubenswrapper[2570]: I0710 00:26:25.245555 2570 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kubelet-serving" Jul 10 00:26:25.245809 ip-10-0-130-146 kubenswrapper[2570]: I0710 00:26:25.245791 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-d8t56" Jul 10 00:26:25.245877 ip-10-0-130-146 kubenswrapper[2570]: I0710 00:26:25.245831 2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\"" Jul 10 00:26:25.245877 ip-10-0-130-146 kubenswrapper[2570]: I0710 00:26:25.245847 2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\"" Jul 10 00:26:25.245977 ip-10-0-130-146 kubenswrapper[2570]: I0710 00:26:25.245875 2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-gdfvl\"" Jul 10 00:26:25.245977 ip-10-0-130-146 kubenswrapper[2570]: E0710 00:26:25.245878 2570 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-d8t56" podUID="b7aab806-00a2-491e-a997-ed3174bd547a" Jul 10 00:26:25.246150 ip-10-0-130-146 kubenswrapper[2570]: I0710 00:26:25.246134 2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\"" Jul 10 00:26:25.248063 ip-10-0-130-146 kubenswrapper[2570]: I0710 00:26:25.248029 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-995ch" Jul 10 00:26:25.250522 ip-10-0-130-146 kubenswrapper[2570]: I0710 00:26:25.250503 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-mbs58" Jul 10 00:26:25.251062 ip-10-0-130-146 kubenswrapper[2570]: I0710 00:26:25.250745 2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"default-dockercfg-drcnp\"" Jul 10 00:26:25.251062 ip-10-0-130-146 kubenswrapper[2570]: I0710 00:26:25.250916 2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\"" Jul 10 00:26:25.251062 ip-10-0-130-146 kubenswrapper[2570]: I0710 00:26:25.250924 2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\"" Jul 10 00:26:25.252997 ip-10-0-130-146 kubenswrapper[2570]: I0710 00:26:25.252975 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-68tgb" Jul 10 00:26:25.253238 ip-10-0-130-146 kubenswrapper[2570]: I0710 00:26:25.253219 2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\"" Jul 10 00:26:25.253305 ip-10-0-130-146 kubenswrapper[2570]: I0710 00:26:25.253279 2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\"" Jul 10 00:26:25.253449 ip-10-0-130-146 kubenswrapper[2570]: I0710 00:26:25.253432 2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\"" Jul 10 00:26:25.254488 ip-10-0-130-146 kubenswrapper[2570]: I0710 00:26:25.254209 2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\"" Jul 10 00:26:25.254592 ip-10-0-130-146 kubenswrapper[2570]: I0710 00:26:25.254560 2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\"" Jul 10 00:26:25.254701 ip-10-0-130-146 kubenswrapper[2570]: I0710 00:26:25.254685 2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-pfxd2\"" Jul 10 00:26:25.254762 ip-10-0-130-146 kubenswrapper[2570]: I0710 00:26:25.254697 2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\"" Jul 10 00:26:25.255554 ip-10-0-130-146 kubenswrapper[2570]: I0710 00:26:25.255439 2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\"" Jul 10 00:26:25.257101 ip-10-0-130-146 kubenswrapper[2570]: I0710 00:26:25.255989 2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\"" Jul 10 00:26:25.257101 ip-10-0-130-146 kubenswrapper[2570]: I0710 00:26:25.256413 2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\"" Jul 10 00:26:25.257101 ip-10-0-130-146 kubenswrapper[2570]: I0710 00:26:25.256570 2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-kvs8f\"" Jul 10 00:26:25.257101 ip-10-0-130-146 kubenswrapper[2570]: I0710 00:26:25.256815 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-5hs2b" Jul 10 00:26:25.257101 ip-10-0-130-146 kubenswrapper[2570]: E0710 00:26:25.256894 2570 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-5hs2b" podUID="78717d89-03f2-4824-942a-48892a2bfb99" Jul 10 00:26:25.259613 ip-10-0-130-146 kubenswrapper[2570]: I0710 00:26:25.259597 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-6bqvb" Jul 10 00:26:25.259689 ip-10-0-130-146 kubenswrapper[2570]: E0710 00:26:25.259667 2570 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-6bqvb" podUID="5667efb4-6019-4bde-827a-0f1e68096b93" Jul 10 00:26:25.261898 ip-10-0-130-146 kubenswrapper[2570]: I0710 00:26:25.261879 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-2pdkh" Jul 10 00:26:25.263067 ip-10-0-130-146 kubenswrapper[2570]: I0710 00:26:25.263032 2570 reflector.go:439] "Caches populated" logger="kubernetes.io/kubelet-serving" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" Jul 10 00:26:25.264235 ip-10-0-130-146 kubenswrapper[2570]: I0710 00:26:25.264216 2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\"" Jul 10 00:26:25.264350 ip-10-0-130-146 kubenswrapper[2570]: I0710 00:26:25.264332 2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"default-dockercfg-b9778\"" Jul 10 00:26:25.271765 ip-10-0-130-146 kubenswrapper[2570]: I0710 00:26:25.271748 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/78f9f66c-5c2f-4cee-bf34-39961a78dbcc-cnibin\") pod \"multus-additional-cni-plugins-7qfbg\" (UID: \"78f9f66c-5c2f-4cee-bf34-39961a78dbcc\") " pod="openshift-multus/multus-additional-cni-plugins-7qfbg" Jul 10 00:26:25.271835 ip-10-0-130-146 kubenswrapper[2570]: I0710 00:26:25.271778 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/173d0656-f60a-40da-8ff5-2e92a0ca8a22-etc-sysctl-d\") pod \"tuned-7hmqd\" (UID: \"173d0656-f60a-40da-8ff5-2e92a0ca8a22\") " pod="openshift-cluster-node-tuning-operator/tuned-7hmqd" Jul 10 00:26:25.271835 ip-10-0-130-146 kubenswrapper[2570]: I0710 00:26:25.271801 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vf7hx\" (UniqueName: \"kubernetes.io/projected/006ba44e-264c-42cd-9891-ad8d769386c8-kube-api-access-vf7hx\") pod \"node-resolver-7tn6x\" (UID: \"006ba44e-264c-42cd-9891-ad8d769386c8\") " pod="openshift-dns/node-resolver-7tn6x" Jul 10 00:26:25.271899 ip-10-0-130-146 kubenswrapper[2570]: I0710 00:26:25.271840 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jsds9\" (UniqueName: \"kubernetes.io/projected/b7aab806-00a2-491e-a997-ed3174bd547a-kube-api-access-jsds9\") pod \"network-metrics-daemon-d8t56\" (UID: \"b7aab806-00a2-491e-a997-ed3174bd547a\") " pod="openshift-multus/network-metrics-daemon-d8t56" Jul 10 00:26:25.271899 ip-10-0-130-146 kubenswrapper[2570]: I0710 00:26:25.271866 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/78f9f66c-5c2f-4cee-bf34-39961a78dbcc-os-release\") pod \"multus-additional-cni-plugins-7qfbg\" (UID: \"78f9f66c-5c2f-4cee-bf34-39961a78dbcc\") " pod="openshift-multus/multus-additional-cni-plugins-7qfbg" Jul 10 00:26:25.271899 ip-10-0-130-146 kubenswrapper[2570]: I0710 00:26:25.271882 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/78f9f66c-5c2f-4cee-bf34-39961a78dbcc-tuning-conf-dir\") pod \"multus-additional-cni-plugins-7qfbg\" (UID: \"78f9f66c-5c2f-4cee-bf34-39961a78dbcc\") " pod="openshift-multus/multus-additional-cni-plugins-7qfbg" Jul 10 00:26:25.271988 ip-10-0-130-146 kubenswrapper[2570]: I0710 00:26:25.271917 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fjxt7\" (UniqueName: \"kubernetes.io/projected/173d0656-f60a-40da-8ff5-2e92a0ca8a22-kube-api-access-fjxt7\") pod \"tuned-7hmqd\" (UID: \"173d0656-f60a-40da-8ff5-2e92a0ca8a22\") " pod="openshift-cluster-node-tuning-operator/tuned-7hmqd" Jul 10 00:26:25.271988 ip-10-0-130-146 kubenswrapper[2570]: I0710 00:26:25.271958 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/78f9f66c-5c2f-4cee-bf34-39961a78dbcc-system-cni-dir\") pod \"multus-additional-cni-plugins-7qfbg\" (UID: \"78f9f66c-5c2f-4cee-bf34-39961a78dbcc\") " pod="openshift-multus/multus-additional-cni-plugins-7qfbg" Jul 10 00:26:25.271988 ip-10-0-130-146 kubenswrapper[2570]: I0710 00:26:25.271983 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/173d0656-f60a-40da-8ff5-2e92a0ca8a22-host\") pod \"tuned-7hmqd\" (UID: \"173d0656-f60a-40da-8ff5-2e92a0ca8a22\") " pod="openshift-cluster-node-tuning-operator/tuned-7hmqd" Jul 10 00:26:25.272096 ip-10-0-130-146 kubenswrapper[2570]: I0710 00:26:25.272005 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/173d0656-f60a-40da-8ff5-2e92a0ca8a22-tmp\") pod \"tuned-7hmqd\" (UID: \"173d0656-f60a-40da-8ff5-2e92a0ca8a22\") " pod="openshift-cluster-node-tuning-operator/tuned-7hmqd" Jul 10 00:26:25.272096 ip-10-0-130-146 kubenswrapper[2570]: I0710 00:26:25.272027 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/52179d08-5861-4a23-9c8d-653c13bed339-host\") pod \"node-ca-t526b\" (UID: \"52179d08-5861-4a23-9c8d-653c13bed339\") " pod="openshift-image-registry/node-ca-t526b" Jul 10 00:26:25.272096 ip-10-0-130-146 kubenswrapper[2570]: I0710 00:26:25.272068 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gp6ct\" (UniqueName: \"kubernetes.io/projected/52179d08-5861-4a23-9c8d-653c13bed339-kube-api-access-gp6ct\") pod \"node-ca-t526b\" (UID: \"52179d08-5861-4a23-9c8d-653c13bed339\") " pod="openshift-image-registry/node-ca-t526b" Jul 10 00:26:25.272096 ip-10-0-130-146 kubenswrapper[2570]: I0710 00:26:25.272091 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/173d0656-f60a-40da-8ff5-2e92a0ca8a22-etc-tuned\") pod \"tuned-7hmqd\" (UID: \"173d0656-f60a-40da-8ff5-2e92a0ca8a22\") " pod="openshift-cluster-node-tuning-operator/tuned-7hmqd" Jul 10 00:26:25.272232 ip-10-0-130-146 kubenswrapper[2570]: I0710 00:26:25.272115 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/5dcf6a9a-85e6-4f25-85b3-7b8a9eaa8654-host-slash\") pod \"iptables-alerter-bw9kx\" (UID: \"5dcf6a9a-85e6-4f25-85b3-7b8a9eaa8654\") " pod="openshift-network-operator/iptables-alerter-bw9kx" Jul 10 00:26:25.272232 ip-10-0-130-146 kubenswrapper[2570]: I0710 00:26:25.272148 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rtmcc\" (UniqueName: \"kubernetes.io/projected/78f9f66c-5c2f-4cee-bf34-39961a78dbcc-kube-api-access-rtmcc\") pod \"multus-additional-cni-plugins-7qfbg\" (UID: \"78f9f66c-5c2f-4cee-bf34-39961a78dbcc\") " pod="openshift-multus/multus-additional-cni-plugins-7qfbg" Jul 10 00:26:25.272232 ip-10-0-130-146 kubenswrapper[2570]: I0710 00:26:25.272167 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/173d0656-f60a-40da-8ff5-2e92a0ca8a22-etc-sysctl-conf\") pod \"tuned-7hmqd\" (UID: \"173d0656-f60a-40da-8ff5-2e92a0ca8a22\") " pod="openshift-cluster-node-tuning-operator/tuned-7hmqd" Jul 10 00:26:25.272232 ip-10-0-130-146 kubenswrapper[2570]: I0710 00:26:25.272181 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/173d0656-f60a-40da-8ff5-2e92a0ca8a22-etc-systemd\") pod \"tuned-7hmqd\" (UID: \"173d0656-f60a-40da-8ff5-2e92a0ca8a22\") " pod="openshift-cluster-node-tuning-operator/tuned-7hmqd" Jul 10 00:26:25.272232 ip-10-0-130-146 kubenswrapper[2570]: I0710 00:26:25.272196 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/173d0656-f60a-40da-8ff5-2e92a0ca8a22-sys\") pod \"tuned-7hmqd\" (UID: \"173d0656-f60a-40da-8ff5-2e92a0ca8a22\") " pod="openshift-cluster-node-tuning-operator/tuned-7hmqd" Jul 10 00:26:25.272232 ip-10-0-130-146 kubenswrapper[2570]: I0710 00:26:25.272211 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b7aab806-00a2-491e-a997-ed3174bd547a-metrics-certs\") pod \"network-metrics-daemon-d8t56\" (UID: \"b7aab806-00a2-491e-a997-ed3174bd547a\") " pod="openshift-multus/network-metrics-daemon-d8t56" Jul 10 00:26:25.272495 ip-10-0-130-146 kubenswrapper[2570]: I0710 00:26:25.272234 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sp2wx\" (UniqueName: \"kubernetes.io/projected/5dcf6a9a-85e6-4f25-85b3-7b8a9eaa8654-kube-api-access-sp2wx\") pod \"iptables-alerter-bw9kx\" (UID: \"5dcf6a9a-85e6-4f25-85b3-7b8a9eaa8654\") " pod="openshift-network-operator/iptables-alerter-bw9kx" Jul 10 00:26:25.272495 ip-10-0-130-146 kubenswrapper[2570]: I0710 00:26:25.272256 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/78f9f66c-5c2f-4cee-bf34-39961a78dbcc-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-7qfbg\" (UID: \"78f9f66c-5c2f-4cee-bf34-39961a78dbcc\") " pod="openshift-multus/multus-additional-cni-plugins-7qfbg" Jul 10 00:26:25.272495 ip-10-0-130-146 kubenswrapper[2570]: I0710 00:26:25.272270 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/173d0656-f60a-40da-8ff5-2e92a0ca8a22-lib-modules\") pod \"tuned-7hmqd\" (UID: \"173d0656-f60a-40da-8ff5-2e92a0ca8a22\") " pod="openshift-cluster-node-tuning-operator/tuned-7hmqd" Jul 10 00:26:25.272495 ip-10-0-130-146 kubenswrapper[2570]: I0710 00:26:25.272287 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/78f9f66c-5c2f-4cee-bf34-39961a78dbcc-cni-binary-copy\") pod \"multus-additional-cni-plugins-7qfbg\" (UID: \"78f9f66c-5c2f-4cee-bf34-39961a78dbcc\") " pod="openshift-multus/multus-additional-cni-plugins-7qfbg" Jul 10 00:26:25.272495 ip-10-0-130-146 kubenswrapper[2570]: I0710 00:26:25.272315 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/52179d08-5861-4a23-9c8d-653c13bed339-serviceca\") pod \"node-ca-t526b\" (UID: \"52179d08-5861-4a23-9c8d-653c13bed339\") " pod="openshift-image-registry/node-ca-t526b" Jul 10 00:26:25.272495 ip-10-0-130-146 kubenswrapper[2570]: I0710 00:26:25.272355 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/78f9f66c-5c2f-4cee-bf34-39961a78dbcc-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-7qfbg\" (UID: \"78f9f66c-5c2f-4cee-bf34-39961a78dbcc\") " pod="openshift-multus/multus-additional-cni-plugins-7qfbg" Jul 10 00:26:25.272495 ip-10-0-130-146 kubenswrapper[2570]: I0710 00:26:25.272374 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/173d0656-f60a-40da-8ff5-2e92a0ca8a22-etc-modprobe-d\") pod \"tuned-7hmqd\" (UID: \"173d0656-f60a-40da-8ff5-2e92a0ca8a22\") " pod="openshift-cluster-node-tuning-operator/tuned-7hmqd" Jul 10 00:26:25.272495 ip-10-0-130-146 kubenswrapper[2570]: I0710 00:26:25.272408 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/173d0656-f60a-40da-8ff5-2e92a0ca8a22-run\") pod \"tuned-7hmqd\" (UID: \"173d0656-f60a-40da-8ff5-2e92a0ca8a22\") " pod="openshift-cluster-node-tuning-operator/tuned-7hmqd" Jul 10 00:26:25.272495 ip-10-0-130-146 kubenswrapper[2570]: I0710 00:26:25.272452 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/006ba44e-264c-42cd-9891-ad8d769386c8-hosts-file\") pod \"node-resolver-7tn6x\" (UID: \"006ba44e-264c-42cd-9891-ad8d769386c8\") " pod="openshift-dns/node-resolver-7tn6x" Jul 10 00:26:25.272495 ip-10-0-130-146 kubenswrapper[2570]: I0710 00:26:25.272488 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/006ba44e-264c-42cd-9891-ad8d769386c8-tmp-dir\") pod \"node-resolver-7tn6x\" (UID: \"006ba44e-264c-42cd-9891-ad8d769386c8\") " pod="openshift-dns/node-resolver-7tn6x" Jul 10 00:26:25.272821 ip-10-0-130-146 kubenswrapper[2570]: I0710 00:26:25.272509 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/173d0656-f60a-40da-8ff5-2e92a0ca8a22-var-lib-kubelet\") pod \"tuned-7hmqd\" (UID: \"173d0656-f60a-40da-8ff5-2e92a0ca8a22\") " pod="openshift-cluster-node-tuning-operator/tuned-7hmqd" Jul 10 00:26:25.272821 ip-10-0-130-146 kubenswrapper[2570]: I0710 00:26:25.272533 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/5dcf6a9a-85e6-4f25-85b3-7b8a9eaa8654-iptables-alerter-script\") pod \"iptables-alerter-bw9kx\" (UID: \"5dcf6a9a-85e6-4f25-85b3-7b8a9eaa8654\") " pod="openshift-network-operator/iptables-alerter-bw9kx" Jul 10 00:26:25.272821 ip-10-0-130-146 kubenswrapper[2570]: I0710 00:26:25.272557 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/173d0656-f60a-40da-8ff5-2e92a0ca8a22-etc-sysconfig\") pod \"tuned-7hmqd\" (UID: \"173d0656-f60a-40da-8ff5-2e92a0ca8a22\") " pod="openshift-cluster-node-tuning-operator/tuned-7hmqd" Jul 10 00:26:25.272821 ip-10-0-130-146 kubenswrapper[2570]: I0710 00:26:25.272596 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/173d0656-f60a-40da-8ff5-2e92a0ca8a22-etc-kubernetes\") pod \"tuned-7hmqd\" (UID: \"173d0656-f60a-40da-8ff5-2e92a0ca8a22\") " pod="openshift-cluster-node-tuning-operator/tuned-7hmqd" Jul 10 00:26:25.292895 ip-10-0-130-146 kubenswrapper[2570]: I0710 00:26:25.292879 2570 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kubelet-serving" csr="csr-ltbl9" Jul 10 00:26:25.298862 ip-10-0-130-146 kubenswrapper[2570]: I0710 00:26:25.298848 2570 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kubelet-serving" csr="csr-ltbl9" Jul 10 00:26:25.345374 ip-10-0-130-146 kubenswrapper[2570]: I0710 00:26:25.345317 2570 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world" Jul 10 00:26:25.372853 ip-10-0-130-146 kubenswrapper[2570]: I0710 00:26:25.372824 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gw9kb\" (UniqueName: \"kubernetes.io/projected/f726a6ab-3c92-403e-ad1b-bdaf7e37c5e3-kube-api-access-gw9kb\") pod \"multus-2pdkh\" (UID: \"f726a6ab-3c92-403e-ad1b-bdaf7e37c5e3\") " pod="openshift-multus/multus-2pdkh" Jul 10 00:26:25.372968 ip-10-0-130-146 kubenswrapper[2570]: I0710 00:26:25.372863 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/78f9f66c-5c2f-4cee-bf34-39961a78dbcc-cni-binary-copy\") pod \"multus-additional-cni-plugins-7qfbg\" (UID: \"78f9f66c-5c2f-4cee-bf34-39961a78dbcc\") " pod="openshift-multus/multus-additional-cni-plugins-7qfbg" Jul 10 00:26:25.372968 ip-10-0-130-146 kubenswrapper[2570]: I0710 00:26:25.372886 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/52179d08-5861-4a23-9c8d-653c13bed339-serviceca\") pod \"node-ca-t526b\" (UID: \"52179d08-5861-4a23-9c8d-653c13bed339\") " pod="openshift-image-registry/node-ca-t526b" Jul 10 00:26:25.372968 ip-10-0-130-146 kubenswrapper[2570]: I0710 00:26:25.372922 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/c4ef019a-364a-4933-ab85-de86d86167f1-run-ovn\") pod \"ovnkube-node-mbs58\" (UID: \"c4ef019a-364a-4933-ab85-de86d86167f1\") " pod="openshift-ovn-kubernetes/ovnkube-node-mbs58" Jul 10 00:26:25.373137 ip-10-0-130-146 kubenswrapper[2570]: I0710 00:26:25.372999 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/c4ef019a-364a-4933-ab85-de86d86167f1-host-run-ovn-kubernetes\") pod \"ovnkube-node-mbs58\" (UID: \"c4ef019a-364a-4933-ab85-de86d86167f1\") " pod="openshift-ovn-kubernetes/ovnkube-node-mbs58" Jul 10 00:26:25.373137 ip-10-0-130-146 kubenswrapper[2570]: I0710 00:26:25.373029 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/c4ef019a-364a-4933-ab85-de86d86167f1-ovnkube-config\") pod \"ovnkube-node-mbs58\" (UID: \"c4ef019a-364a-4933-ab85-de86d86167f1\") " pod="openshift-ovn-kubernetes/ovnkube-node-mbs58" Jul 10 00:26:25.373137 ip-10-0-130-146 kubenswrapper[2570]: I0710 00:26:25.373081 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/f726a6ab-3c92-403e-ad1b-bdaf7e37c5e3-host-var-lib-cni-bin\") pod \"multus-2pdkh\" (UID: \"f726a6ab-3c92-403e-ad1b-bdaf7e37c5e3\") " pod="openshift-multus/multus-2pdkh" Jul 10 00:26:25.373137 ip-10-0-130-146 kubenswrapper[2570]: I0710 00:26:25.373111 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/78f9f66c-5c2f-4cee-bf34-39961a78dbcc-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-7qfbg\" (UID: \"78f9f66c-5c2f-4cee-bf34-39961a78dbcc\") " pod="openshift-multus/multus-additional-cni-plugins-7qfbg" Jul 10 00:26:25.373137 ip-10-0-130-146 kubenswrapper[2570]: I0710 00:26:25.373136 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/173d0656-f60a-40da-8ff5-2e92a0ca8a22-etc-modprobe-d\") pod \"tuned-7hmqd\" (UID: \"173d0656-f60a-40da-8ff5-2e92a0ca8a22\") " pod="openshift-cluster-node-tuning-operator/tuned-7hmqd" Jul 10 00:26:25.373688 ip-10-0-130-146 kubenswrapper[2570]: I0710 00:26:25.373221 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/173d0656-f60a-40da-8ff5-2e92a0ca8a22-run\") pod \"tuned-7hmqd\" (UID: \"173d0656-f60a-40da-8ff5-2e92a0ca8a22\") " pod="openshift-cluster-node-tuning-operator/tuned-7hmqd" Jul 10 00:26:25.373688 ip-10-0-130-146 kubenswrapper[2570]: I0710 00:26:25.373265 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/006ba44e-264c-42cd-9891-ad8d769386c8-hosts-file\") pod \"node-resolver-7tn6x\" (UID: \"006ba44e-264c-42cd-9891-ad8d769386c8\") " pod="openshift-dns/node-resolver-7tn6x" Jul 10 00:26:25.373688 ip-10-0-130-146 kubenswrapper[2570]: I0710 00:26:25.373280 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sst8k\" (UniqueName: \"kubernetes.io/projected/c44f0289-9f01-4548-ab98-74a17ec2b150-kube-api-access-sst8k\") pod \"aws-ebs-csi-driver-node-68tgb\" (UID: \"c44f0289-9f01-4548-ab98-74a17ec2b150\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-68tgb" Jul 10 00:26:25.373688 ip-10-0-130-146 kubenswrapper[2570]: I0710 00:26:25.373296 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/c4ef019a-364a-4933-ab85-de86d86167f1-systemd-units\") pod \"ovnkube-node-mbs58\" (UID: \"c4ef019a-364a-4933-ab85-de86d86167f1\") " pod="openshift-ovn-kubernetes/ovnkube-node-mbs58" Jul 10 00:26:25.373688 ip-10-0-130-146 kubenswrapper[2570]: I0710 00:26:25.373314 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/c4ef019a-364a-4933-ab85-de86d86167f1-run-systemd\") pod \"ovnkube-node-mbs58\" (UID: \"c4ef019a-364a-4933-ab85-de86d86167f1\") " pod="openshift-ovn-kubernetes/ovnkube-node-mbs58" Jul 10 00:26:25.373688 ip-10-0-130-146 kubenswrapper[2570]: I0710 00:26:25.373321 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/173d0656-f60a-40da-8ff5-2e92a0ca8a22-etc-modprobe-d\") pod \"tuned-7hmqd\" (UID: \"173d0656-f60a-40da-8ff5-2e92a0ca8a22\") " pod="openshift-cluster-node-tuning-operator/tuned-7hmqd" Jul 10 00:26:25.373688 ip-10-0-130-146 kubenswrapper[2570]: I0710 00:26:25.373337 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/173d0656-f60a-40da-8ff5-2e92a0ca8a22-etc-sysctl-d\") pod \"tuned-7hmqd\" (UID: \"173d0656-f60a-40da-8ff5-2e92a0ca8a22\") " pod="openshift-cluster-node-tuning-operator/tuned-7hmqd" Jul 10 00:26:25.373688 ip-10-0-130-146 kubenswrapper[2570]: I0710 00:26:25.373370 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/006ba44e-264c-42cd-9891-ad8d769386c8-hosts-file\") pod \"node-resolver-7tn6x\" (UID: \"006ba44e-264c-42cd-9891-ad8d769386c8\") " pod="openshift-dns/node-resolver-7tn6x" Jul 10 00:26:25.373688 ip-10-0-130-146 kubenswrapper[2570]: I0710 00:26:25.373376 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jsds9\" (UniqueName: \"kubernetes.io/projected/b7aab806-00a2-491e-a997-ed3174bd547a-kube-api-access-jsds9\") pod \"network-metrics-daemon-d8t56\" (UID: \"b7aab806-00a2-491e-a997-ed3174bd547a\") " pod="openshift-multus/network-metrics-daemon-d8t56" Jul 10 00:26:25.373688 ip-10-0-130-146 kubenswrapper[2570]: I0710 00:26:25.373388 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/52179d08-5861-4a23-9c8d-653c13bed339-serviceca\") pod \"node-ca-t526b\" (UID: \"52179d08-5861-4a23-9c8d-653c13bed339\") " pod="openshift-image-registry/node-ca-t526b" Jul 10 00:26:25.373688 ip-10-0-130-146 kubenswrapper[2570]: I0710 00:26:25.373393 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/173d0656-f60a-40da-8ff5-2e92a0ca8a22-run\") pod \"tuned-7hmqd\" (UID: \"173d0656-f60a-40da-8ff5-2e92a0ca8a22\") " pod="openshift-cluster-node-tuning-operator/tuned-7hmqd" Jul 10 00:26:25.373688 ip-10-0-130-146 kubenswrapper[2570]: I0710 00:26:25.373405 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/c4ef019a-364a-4933-ab85-de86d86167f1-ovnkube-script-lib\") pod \"ovnkube-node-mbs58\" (UID: \"c4ef019a-364a-4933-ab85-de86d86167f1\") " pod="openshift-ovn-kubernetes/ovnkube-node-mbs58" Jul 10 00:26:25.373688 ip-10-0-130-146 kubenswrapper[2570]: I0710 00:26:25.373443 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/f726a6ab-3c92-403e-ad1b-bdaf7e37c5e3-os-release\") pod \"multus-2pdkh\" (UID: \"f726a6ab-3c92-403e-ad1b-bdaf7e37c5e3\") " pod="openshift-multus/multus-2pdkh" Jul 10 00:26:25.373688 ip-10-0-130-146 kubenswrapper[2570]: I0710 00:26:25.373481 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/173d0656-f60a-40da-8ff5-2e92a0ca8a22-etc-sysctl-d\") pod \"tuned-7hmqd\" (UID: \"173d0656-f60a-40da-8ff5-2e92a0ca8a22\") " pod="openshift-cluster-node-tuning-operator/tuned-7hmqd" Jul 10 00:26:25.373688 ip-10-0-130-146 kubenswrapper[2570]: I0710 00:26:25.373496 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/f726a6ab-3c92-403e-ad1b-bdaf7e37c5e3-host-var-lib-kubelet\") pod \"multus-2pdkh\" (UID: \"f726a6ab-3c92-403e-ad1b-bdaf7e37c5e3\") " pod="openshift-multus/multus-2pdkh" Jul 10 00:26:25.373688 ip-10-0-130-146 kubenswrapper[2570]: I0710 00:26:25.373537 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/c4ef019a-364a-4933-ab85-de86d86167f1-run-openvswitch\") pod \"ovnkube-node-mbs58\" (UID: \"c4ef019a-364a-4933-ab85-de86d86167f1\") " pod="openshift-ovn-kubernetes/ovnkube-node-mbs58" Jul 10 00:26:25.373688 ip-10-0-130-146 kubenswrapper[2570]: I0710 00:26:25.373564 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/c4ef019a-364a-4933-ab85-de86d86167f1-host-cni-bin\") pod \"ovnkube-node-mbs58\" (UID: \"c4ef019a-364a-4933-ab85-de86d86167f1\") " pod="openshift-ovn-kubernetes/ovnkube-node-mbs58" Jul 10 00:26:25.374352 ip-10-0-130-146 kubenswrapper[2570]: I0710 00:26:25.373585 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/78f9f66c-5c2f-4cee-bf34-39961a78dbcc-cni-binary-copy\") pod \"multus-additional-cni-plugins-7qfbg\" (UID: \"78f9f66c-5c2f-4cee-bf34-39961a78dbcc\") " pod="openshift-multus/multus-additional-cni-plugins-7qfbg" Jul 10 00:26:25.374352 ip-10-0-130-146 kubenswrapper[2570]: I0710 00:26:25.373590 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/f726a6ab-3c92-403e-ad1b-bdaf7e37c5e3-cnibin\") pod \"multus-2pdkh\" (UID: \"f726a6ab-3c92-403e-ad1b-bdaf7e37c5e3\") " pod="openshift-multus/multus-2pdkh" Jul 10 00:26:25.374352 ip-10-0-130-146 kubenswrapper[2570]: I0710 00:26:25.373584 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/78f9f66c-5c2f-4cee-bf34-39961a78dbcc-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-7qfbg\" (UID: \"78f9f66c-5c2f-4cee-bf34-39961a78dbcc\") " pod="openshift-multus/multus-additional-cni-plugins-7qfbg" Jul 10 00:26:25.374352 ip-10-0-130-146 kubenswrapper[2570]: I0710 00:26:25.373615 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/173d0656-f60a-40da-8ff5-2e92a0ca8a22-etc-tuned\") pod \"tuned-7hmqd\" (UID: \"173d0656-f60a-40da-8ff5-2e92a0ca8a22\") " pod="openshift-cluster-node-tuning-operator/tuned-7hmqd" Jul 10 00:26:25.374352 ip-10-0-130-146 kubenswrapper[2570]: I0710 00:26:25.373669 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/5dcf6a9a-85e6-4f25-85b3-7b8a9eaa8654-host-slash\") pod \"iptables-alerter-bw9kx\" (UID: \"5dcf6a9a-85e6-4f25-85b3-7b8a9eaa8654\") " pod="openshift-network-operator/iptables-alerter-bw9kx" Jul 10 00:26:25.374352 ip-10-0-130-146 kubenswrapper[2570]: I0710 00:26:25.373710 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/5dcf6a9a-85e6-4f25-85b3-7b8a9eaa8654-host-slash\") pod \"iptables-alerter-bw9kx\" (UID: \"5dcf6a9a-85e6-4f25-85b3-7b8a9eaa8654\") " pod="openshift-network-operator/iptables-alerter-bw9kx" Jul 10 00:26:25.374352 ip-10-0-130-146 kubenswrapper[2570]: I0710 00:26:25.373724 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/173d0656-f60a-40da-8ff5-2e92a0ca8a22-etc-systemd\") pod \"tuned-7hmqd\" (UID: \"173d0656-f60a-40da-8ff5-2e92a0ca8a22\") " pod="openshift-cluster-node-tuning-operator/tuned-7hmqd" Jul 10 00:26:25.374352 ip-10-0-130-146 kubenswrapper[2570]: I0710 00:26:25.373751 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/173d0656-f60a-40da-8ff5-2e92a0ca8a22-sys\") pod \"tuned-7hmqd\" (UID: \"173d0656-f60a-40da-8ff5-2e92a0ca8a22\") " pod="openshift-cluster-node-tuning-operator/tuned-7hmqd" Jul 10 00:26:25.374352 ip-10-0-130-146 kubenswrapper[2570]: I0710 00:26:25.373774 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/5e419a12-f1a1-4b3d-8c72-7c6af574948f-konnectivity-ca\") pod \"konnectivity-agent-995ch\" (UID: \"5e419a12-f1a1-4b3d-8c72-7c6af574948f\") " pod="kube-system/konnectivity-agent-995ch" Jul 10 00:26:25.374352 ip-10-0-130-146 kubenswrapper[2570]: I0710 00:26:25.373784 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/173d0656-f60a-40da-8ff5-2e92a0ca8a22-etc-systemd\") pod \"tuned-7hmqd\" (UID: \"173d0656-f60a-40da-8ff5-2e92a0ca8a22\") " pod="openshift-cluster-node-tuning-operator/tuned-7hmqd" Jul 10 00:26:25.374352 ip-10-0-130-146 kubenswrapper[2570]: I0710 00:26:25.373799 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/f726a6ab-3c92-403e-ad1b-bdaf7e37c5e3-system-cni-dir\") pod \"multus-2pdkh\" (UID: \"f726a6ab-3c92-403e-ad1b-bdaf7e37c5e3\") " pod="openshift-multus/multus-2pdkh" Jul 10 00:26:25.374352 ip-10-0-130-146 kubenswrapper[2570]: I0710 00:26:25.373824 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/173d0656-f60a-40da-8ff5-2e92a0ca8a22-sys\") pod \"tuned-7hmqd\" (UID: \"173d0656-f60a-40da-8ff5-2e92a0ca8a22\") " pod="openshift-cluster-node-tuning-operator/tuned-7hmqd" Jul 10 00:26:25.374352 ip-10-0-130-146 kubenswrapper[2570]: I0710 00:26:25.373822 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/f726a6ab-3c92-403e-ad1b-bdaf7e37c5e3-host-var-lib-cni-multus\") pod \"multus-2pdkh\" (UID: \"f726a6ab-3c92-403e-ad1b-bdaf7e37c5e3\") " pod="openshift-multus/multus-2pdkh" Jul 10 00:26:25.374352 ip-10-0-130-146 kubenswrapper[2570]: I0710 00:26:25.373862 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/f726a6ab-3c92-403e-ad1b-bdaf7e37c5e3-hostroot\") pod \"multus-2pdkh\" (UID: \"f726a6ab-3c92-403e-ad1b-bdaf7e37c5e3\") " pod="openshift-multus/multus-2pdkh" Jul 10 00:26:25.374352 ip-10-0-130-146 kubenswrapper[2570]: I0710 00:26:25.373886 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/78f9f66c-5c2f-4cee-bf34-39961a78dbcc-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-7qfbg\" (UID: \"78f9f66c-5c2f-4cee-bf34-39961a78dbcc\") " pod="openshift-multus/multus-additional-cni-plugins-7qfbg" Jul 10 00:26:25.374352 ip-10-0-130-146 kubenswrapper[2570]: I0710 00:26:25.373909 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/173d0656-f60a-40da-8ff5-2e92a0ca8a22-lib-modules\") pod \"tuned-7hmqd\" (UID: \"173d0656-f60a-40da-8ff5-2e92a0ca8a22\") " pod="openshift-cluster-node-tuning-operator/tuned-7hmqd" Jul 10 00:26:25.374352 ip-10-0-130-146 kubenswrapper[2570]: I0710 00:26:25.373925 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/c4ef019a-364a-4933-ab85-de86d86167f1-host-kubelet\") pod \"ovnkube-node-mbs58\" (UID: \"c4ef019a-364a-4933-ab85-de86d86167f1\") " pod="openshift-ovn-kubernetes/ovnkube-node-mbs58" Jul 10 00:26:25.375069 ip-10-0-130-146 kubenswrapper[2570]: I0710 00:26:25.373940 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/c44f0289-9f01-4548-ab98-74a17ec2b150-registration-dir\") pod \"aws-ebs-csi-driver-node-68tgb\" (UID: \"c44f0289-9f01-4548-ab98-74a17ec2b150\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-68tgb" Jul 10 00:26:25.375069 ip-10-0-130-146 kubenswrapper[2570]: I0710 00:26:25.373972 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/f726a6ab-3c92-403e-ad1b-bdaf7e37c5e3-cni-binary-copy\") pod \"multus-2pdkh\" (UID: \"f726a6ab-3c92-403e-ad1b-bdaf7e37c5e3\") " pod="openshift-multus/multus-2pdkh" Jul 10 00:26:25.375069 ip-10-0-130-146 kubenswrapper[2570]: I0710 00:26:25.374002 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/f726a6ab-3c92-403e-ad1b-bdaf7e37c5e3-multus-conf-dir\") pod \"multus-2pdkh\" (UID: \"f726a6ab-3c92-403e-ad1b-bdaf7e37c5e3\") " pod="openshift-multus/multus-2pdkh" Jul 10 00:26:25.375069 ip-10-0-130-146 kubenswrapper[2570]: I0710 00:26:25.374008 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/173d0656-f60a-40da-8ff5-2e92a0ca8a22-lib-modules\") pod \"tuned-7hmqd\" (UID: \"173d0656-f60a-40da-8ff5-2e92a0ca8a22\") " pod="openshift-cluster-node-tuning-operator/tuned-7hmqd" Jul 10 00:26:25.375069 ip-10-0-130-146 kubenswrapper[2570]: I0710 00:26:25.374007 2570 swap_util.go:74] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory" Jul 10 00:26:25.375069 ip-10-0-130-146 kubenswrapper[2570]: I0710 00:26:25.374029 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/c4ef019a-364a-4933-ab85-de86d86167f1-env-overrides\") pod \"ovnkube-node-mbs58\" (UID: \"c4ef019a-364a-4933-ab85-de86d86167f1\") " pod="openshift-ovn-kubernetes/ovnkube-node-mbs58" Jul 10 00:26:25.375069 ip-10-0-130-146 kubenswrapper[2570]: I0710 00:26:25.374094 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kl6s9\" (UniqueName: \"kubernetes.io/projected/78717d89-03f2-4824-942a-48892a2bfb99-kube-api-access-kl6s9\") pod \"network-check-target-5hs2b\" (UID: \"78717d89-03f2-4824-942a-48892a2bfb99\") " pod="openshift-network-diagnostics/network-check-target-5hs2b" Jul 10 00:26:25.375069 ip-10-0-130-146 kubenswrapper[2570]: I0710 00:26:25.374112 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/f726a6ab-3c92-403e-ad1b-bdaf7e37c5e3-multus-cni-dir\") pod \"multus-2pdkh\" (UID: \"f726a6ab-3c92-403e-ad1b-bdaf7e37c5e3\") " pod="openshift-multus/multus-2pdkh" Jul 10 00:26:25.375069 ip-10-0-130-146 kubenswrapper[2570]: I0710 00:26:25.374130 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/006ba44e-264c-42cd-9891-ad8d769386c8-tmp-dir\") pod \"node-resolver-7tn6x\" (UID: \"006ba44e-264c-42cd-9891-ad8d769386c8\") " pod="openshift-dns/node-resolver-7tn6x" Jul 10 00:26:25.375069 ip-10-0-130-146 kubenswrapper[2570]: I0710 00:26:25.374146 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/c4ef019a-364a-4933-ab85-de86d86167f1-host-slash\") pod \"ovnkube-node-mbs58\" (UID: \"c4ef019a-364a-4933-ab85-de86d86167f1\") " pod="openshift-ovn-kubernetes/ovnkube-node-mbs58" Jul 10 00:26:25.375069 ip-10-0-130-146 kubenswrapper[2570]: I0710 00:26:25.374160 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/c4ef019a-364a-4933-ab85-de86d86167f1-host-run-netns\") pod \"ovnkube-node-mbs58\" (UID: \"c4ef019a-364a-4933-ab85-de86d86167f1\") " pod="openshift-ovn-kubernetes/ovnkube-node-mbs58" Jul 10 00:26:25.375069 ip-10-0-130-146 kubenswrapper[2570]: I0710 00:26:25.374185 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/f726a6ab-3c92-403e-ad1b-bdaf7e37c5e3-multus-socket-dir-parent\") pod \"multus-2pdkh\" (UID: \"f726a6ab-3c92-403e-ad1b-bdaf7e37c5e3\") " pod="openshift-multus/multus-2pdkh" Jul 10 00:26:25.375069 ip-10-0-130-146 kubenswrapper[2570]: I0710 00:26:25.374218 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/173d0656-f60a-40da-8ff5-2e92a0ca8a22-var-lib-kubelet\") pod \"tuned-7hmqd\" (UID: \"173d0656-f60a-40da-8ff5-2e92a0ca8a22\") " pod="openshift-cluster-node-tuning-operator/tuned-7hmqd" Jul 10 00:26:25.375069 ip-10-0-130-146 kubenswrapper[2570]: I0710 00:26:25.374249 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/5dcf6a9a-85e6-4f25-85b3-7b8a9eaa8654-iptables-alerter-script\") pod \"iptables-alerter-bw9kx\" (UID: \"5dcf6a9a-85e6-4f25-85b3-7b8a9eaa8654\") " pod="openshift-network-operator/iptables-alerter-bw9kx" Jul 10 00:26:25.375069 ip-10-0-130-146 kubenswrapper[2570]: I0710 00:26:25.374275 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/173d0656-f60a-40da-8ff5-2e92a0ca8a22-etc-sysconfig\") pod \"tuned-7hmqd\" (UID: \"173d0656-f60a-40da-8ff5-2e92a0ca8a22\") " pod="openshift-cluster-node-tuning-operator/tuned-7hmqd" Jul 10 00:26:25.375069 ip-10-0-130-146 kubenswrapper[2570]: I0710 00:26:25.374294 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/173d0656-f60a-40da-8ff5-2e92a0ca8a22-var-lib-kubelet\") pod \"tuned-7hmqd\" (UID: \"173d0656-f60a-40da-8ff5-2e92a0ca8a22\") " pod="openshift-cluster-node-tuning-operator/tuned-7hmqd" Jul 10 00:26:25.375069 ip-10-0-130-146 kubenswrapper[2570]: I0710 00:26:25.374300 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/173d0656-f60a-40da-8ff5-2e92a0ca8a22-etc-kubernetes\") pod \"tuned-7hmqd\" (UID: \"173d0656-f60a-40da-8ff5-2e92a0ca8a22\") " pod="openshift-cluster-node-tuning-operator/tuned-7hmqd" Jul 10 00:26:25.375786 ip-10-0-130-146 kubenswrapper[2570]: I0710 00:26:25.374324 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/78f9f66c-5c2f-4cee-bf34-39961a78dbcc-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-7qfbg\" (UID: \"78f9f66c-5c2f-4cee-bf34-39961a78dbcc\") " pod="openshift-multus/multus-additional-cni-plugins-7qfbg" Jul 10 00:26:25.375786 ip-10-0-130-146 kubenswrapper[2570]: I0710 00:26:25.374332 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/c4ef019a-364a-4933-ab85-de86d86167f1-var-lib-openvswitch\") pod \"ovnkube-node-mbs58\" (UID: \"c4ef019a-364a-4933-ab85-de86d86167f1\") " pod="openshift-ovn-kubernetes/ovnkube-node-mbs58" Jul 10 00:26:25.375786 ip-10-0-130-146 kubenswrapper[2570]: I0710 00:26:25.374358 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/c4ef019a-364a-4933-ab85-de86d86167f1-log-socket\") pod \"ovnkube-node-mbs58\" (UID: \"c4ef019a-364a-4933-ab85-de86d86167f1\") " pod="openshift-ovn-kubernetes/ovnkube-node-mbs58" Jul 10 00:26:25.375786 ip-10-0-130-146 kubenswrapper[2570]: I0710 00:26:25.374372 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/173d0656-f60a-40da-8ff5-2e92a0ca8a22-etc-sysconfig\") pod \"tuned-7hmqd\" (UID: \"173d0656-f60a-40da-8ff5-2e92a0ca8a22\") " pod="openshift-cluster-node-tuning-operator/tuned-7hmqd" Jul 10 00:26:25.375786 ip-10-0-130-146 kubenswrapper[2570]: I0710 00:26:25.374381 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/173d0656-f60a-40da-8ff5-2e92a0ca8a22-etc-kubernetes\") pod \"tuned-7hmqd\" (UID: \"173d0656-f60a-40da-8ff5-2e92a0ca8a22\") " pod="openshift-cluster-node-tuning-operator/tuned-7hmqd" Jul 10 00:26:25.375786 ip-10-0-130-146 kubenswrapper[2570]: I0710 00:26:25.374395 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/c4ef019a-364a-4933-ab85-de86d86167f1-ovn-node-metrics-cert\") pod \"ovnkube-node-mbs58\" (UID: \"c4ef019a-364a-4933-ab85-de86d86167f1\") " pod="openshift-ovn-kubernetes/ovnkube-node-mbs58" Jul 10 00:26:25.375786 ip-10-0-130-146 kubenswrapper[2570]: I0710 00:26:25.374420 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/c44f0289-9f01-4548-ab98-74a17ec2b150-kubelet-dir\") pod \"aws-ebs-csi-driver-node-68tgb\" (UID: \"c44f0289-9f01-4548-ab98-74a17ec2b150\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-68tgb" Jul 10 00:26:25.375786 ip-10-0-130-146 kubenswrapper[2570]: I0710 00:26:25.374445 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/78f9f66c-5c2f-4cee-bf34-39961a78dbcc-cnibin\") pod \"multus-additional-cni-plugins-7qfbg\" (UID: \"78f9f66c-5c2f-4cee-bf34-39961a78dbcc\") " pod="openshift-multus/multus-additional-cni-plugins-7qfbg" Jul 10 00:26:25.375786 ip-10-0-130-146 kubenswrapper[2570]: I0710 00:26:25.374473 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vf7hx\" (UniqueName: \"kubernetes.io/projected/006ba44e-264c-42cd-9891-ad8d769386c8-kube-api-access-vf7hx\") pod \"node-resolver-7tn6x\" (UID: \"006ba44e-264c-42cd-9891-ad8d769386c8\") " pod="openshift-dns/node-resolver-7tn6x" Jul 10 00:26:25.375786 ip-10-0-130-146 kubenswrapper[2570]: I0710 00:26:25.374484 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/78f9f66c-5c2f-4cee-bf34-39961a78dbcc-cnibin\") pod \"multus-additional-cni-plugins-7qfbg\" (UID: \"78f9f66c-5c2f-4cee-bf34-39961a78dbcc\") " pod="openshift-multus/multus-additional-cni-plugins-7qfbg" Jul 10 00:26:25.375786 ip-10-0-130-146 kubenswrapper[2570]: I0710 00:26:25.374494 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/c4ef019a-364a-4933-ab85-de86d86167f1-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-mbs58\" (UID: \"c4ef019a-364a-4933-ab85-de86d86167f1\") " pod="openshift-ovn-kubernetes/ovnkube-node-mbs58" Jul 10 00:26:25.375786 ip-10-0-130-146 kubenswrapper[2570]: I0710 00:26:25.374516 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5m6sc\" (UniqueName: \"kubernetes.io/projected/c4ef019a-364a-4933-ab85-de86d86167f1-kube-api-access-5m6sc\") pod \"ovnkube-node-mbs58\" (UID: \"c4ef019a-364a-4933-ab85-de86d86167f1\") " pod="openshift-ovn-kubernetes/ovnkube-node-mbs58" Jul 10 00:26:25.375786 ip-10-0-130-146 kubenswrapper[2570]: I0710 00:26:25.374536 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/5667efb4-6019-4bde-827a-0f1e68096b93-original-pull-secret\") pod \"global-pull-secret-syncer-6bqvb\" (UID: \"5667efb4-6019-4bde-827a-0f1e68096b93\") " pod="kube-system/global-pull-secret-syncer-6bqvb" Jul 10 00:26:25.375786 ip-10-0-130-146 kubenswrapper[2570]: I0710 00:26:25.374568 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/f726a6ab-3c92-403e-ad1b-bdaf7e37c5e3-host-run-k8s-cni-cncf-io\") pod \"multus-2pdkh\" (UID: \"f726a6ab-3c92-403e-ad1b-bdaf7e37c5e3\") " pod="openshift-multus/multus-2pdkh" Jul 10 00:26:25.375786 ip-10-0-130-146 kubenswrapper[2570]: I0710 00:26:25.374610 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/f726a6ab-3c92-403e-ad1b-bdaf7e37c5e3-host-run-netns\") pod \"multus-2pdkh\" (UID: \"f726a6ab-3c92-403e-ad1b-bdaf7e37c5e3\") " pod="openshift-multus/multus-2pdkh" Jul 10 00:26:25.375786 ip-10-0-130-146 kubenswrapper[2570]: I0710 00:26:25.374650 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/78f9f66c-5c2f-4cee-bf34-39961a78dbcc-os-release\") pod \"multus-additional-cni-plugins-7qfbg\" (UID: \"78f9f66c-5c2f-4cee-bf34-39961a78dbcc\") " pod="openshift-multus/multus-additional-cni-plugins-7qfbg" Jul 10 00:26:25.376355 ip-10-0-130-146 kubenswrapper[2570]: I0710 00:26:25.374676 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/78f9f66c-5c2f-4cee-bf34-39961a78dbcc-tuning-conf-dir\") pod \"multus-additional-cni-plugins-7qfbg\" (UID: \"78f9f66c-5c2f-4cee-bf34-39961a78dbcc\") " pod="openshift-multus/multus-additional-cni-plugins-7qfbg" Jul 10 00:26:25.376355 ip-10-0-130-146 kubenswrapper[2570]: I0710 00:26:25.374706 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fjxt7\" (UniqueName: \"kubernetes.io/projected/173d0656-f60a-40da-8ff5-2e92a0ca8a22-kube-api-access-fjxt7\") pod \"tuned-7hmqd\" (UID: \"173d0656-f60a-40da-8ff5-2e92a0ca8a22\") " pod="openshift-cluster-node-tuning-operator/tuned-7hmqd" Jul 10 00:26:25.376355 ip-10-0-130-146 kubenswrapper[2570]: I0710 00:26:25.374731 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/5dcf6a9a-85e6-4f25-85b3-7b8a9eaa8654-iptables-alerter-script\") pod \"iptables-alerter-bw9kx\" (UID: \"5dcf6a9a-85e6-4f25-85b3-7b8a9eaa8654\") " pod="openshift-network-operator/iptables-alerter-bw9kx" Jul 10 00:26:25.376355 ip-10-0-130-146 kubenswrapper[2570]: I0710 00:26:25.374786 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/78f9f66c-5c2f-4cee-bf34-39961a78dbcc-os-release\") pod \"multus-additional-cni-plugins-7qfbg\" (UID: \"78f9f66c-5c2f-4cee-bf34-39961a78dbcc\") " pod="openshift-multus/multus-additional-cni-plugins-7qfbg" Jul 10 00:26:25.376355 ip-10-0-130-146 kubenswrapper[2570]: I0710 00:26:25.374800 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/c4ef019a-364a-4933-ab85-de86d86167f1-node-log\") pod \"ovnkube-node-mbs58\" (UID: \"c4ef019a-364a-4933-ab85-de86d86167f1\") " pod="openshift-ovn-kubernetes/ovnkube-node-mbs58" Jul 10 00:26:25.376355 ip-10-0-130-146 kubenswrapper[2570]: I0710 00:26:25.374829 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/c4ef019a-364a-4933-ab85-de86d86167f1-host-cni-netd\") pod \"ovnkube-node-mbs58\" (UID: \"c4ef019a-364a-4933-ab85-de86d86167f1\") " pod="openshift-ovn-kubernetes/ovnkube-node-mbs58" Jul 10 00:26:25.376355 ip-10-0-130-146 kubenswrapper[2570]: I0710 00:26:25.374843 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/78f9f66c-5c2f-4cee-bf34-39961a78dbcc-tuning-conf-dir\") pod \"multus-additional-cni-plugins-7qfbg\" (UID: \"78f9f66c-5c2f-4cee-bf34-39961a78dbcc\") " pod="openshift-multus/multus-additional-cni-plugins-7qfbg" Jul 10 00:26:25.376355 ip-10-0-130-146 kubenswrapper[2570]: I0710 00:26:25.374852 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/c44f0289-9f01-4548-ab98-74a17ec2b150-sys-fs\") pod \"aws-ebs-csi-driver-node-68tgb\" (UID: \"c44f0289-9f01-4548-ab98-74a17ec2b150\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-68tgb" Jul 10 00:26:25.376355 ip-10-0-130-146 kubenswrapper[2570]: I0710 00:26:25.374887 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/f726a6ab-3c92-403e-ad1b-bdaf7e37c5e3-multus-daemon-config\") pod \"multus-2pdkh\" (UID: \"f726a6ab-3c92-403e-ad1b-bdaf7e37c5e3\") " pod="openshift-multus/multus-2pdkh" Jul 10 00:26:25.376355 ip-10-0-130-146 kubenswrapper[2570]: I0710 00:26:25.374922 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/f726a6ab-3c92-403e-ad1b-bdaf7e37c5e3-etc-kubernetes\") pod \"multus-2pdkh\" (UID: \"f726a6ab-3c92-403e-ad1b-bdaf7e37c5e3\") " pod="openshift-multus/multus-2pdkh" Jul 10 00:26:25.376355 ip-10-0-130-146 kubenswrapper[2570]: I0710 00:26:25.374935 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/006ba44e-264c-42cd-9891-ad8d769386c8-tmp-dir\") pod \"node-resolver-7tn6x\" (UID: \"006ba44e-264c-42cd-9891-ad8d769386c8\") " pod="openshift-dns/node-resolver-7tn6x" Jul 10 00:26:25.376355 ip-10-0-130-146 kubenswrapper[2570]: I0710 00:26:25.374949 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/78f9f66c-5c2f-4cee-bf34-39961a78dbcc-system-cni-dir\") pod \"multus-additional-cni-plugins-7qfbg\" (UID: \"78f9f66c-5c2f-4cee-bf34-39961a78dbcc\") " pod="openshift-multus/multus-additional-cni-plugins-7qfbg" Jul 10 00:26:25.376355 ip-10-0-130-146 kubenswrapper[2570]: I0710 00:26:25.374989 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/173d0656-f60a-40da-8ff5-2e92a0ca8a22-host\") pod \"tuned-7hmqd\" (UID: \"173d0656-f60a-40da-8ff5-2e92a0ca8a22\") " pod="openshift-cluster-node-tuning-operator/tuned-7hmqd" Jul 10 00:26:25.376355 ip-10-0-130-146 kubenswrapper[2570]: I0710 00:26:25.374992 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/78f9f66c-5c2f-4cee-bf34-39961a78dbcc-system-cni-dir\") pod \"multus-additional-cni-plugins-7qfbg\" (UID: \"78f9f66c-5c2f-4cee-bf34-39961a78dbcc\") " pod="openshift-multus/multus-additional-cni-plugins-7qfbg" Jul 10 00:26:25.376355 ip-10-0-130-146 kubenswrapper[2570]: I0710 00:26:25.375025 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/173d0656-f60a-40da-8ff5-2e92a0ca8a22-tmp\") pod \"tuned-7hmqd\" (UID: \"173d0656-f60a-40da-8ff5-2e92a0ca8a22\") " pod="openshift-cluster-node-tuning-operator/tuned-7hmqd" Jul 10 00:26:25.376355 ip-10-0-130-146 kubenswrapper[2570]: I0710 00:26:25.375034 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/173d0656-f60a-40da-8ff5-2e92a0ca8a22-host\") pod \"tuned-7hmqd\" (UID: \"173d0656-f60a-40da-8ff5-2e92a0ca8a22\") " pod="openshift-cluster-node-tuning-operator/tuned-7hmqd" Jul 10 00:26:25.376355 ip-10-0-130-146 kubenswrapper[2570]: I0710 00:26:25.375088 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/52179d08-5861-4a23-9c8d-653c13bed339-host\") pod \"node-ca-t526b\" (UID: \"52179d08-5861-4a23-9c8d-653c13bed339\") " pod="openshift-image-registry/node-ca-t526b" Jul 10 00:26:25.376828 ip-10-0-130-146 kubenswrapper[2570]: I0710 00:26:25.375114 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gp6ct\" (UniqueName: \"kubernetes.io/projected/52179d08-5861-4a23-9c8d-653c13bed339-kube-api-access-gp6ct\") pod \"node-ca-t526b\" (UID: \"52179d08-5861-4a23-9c8d-653c13bed339\") " pod="openshift-image-registry/node-ca-t526b" Jul 10 00:26:25.376828 ip-10-0-130-146 kubenswrapper[2570]: I0710 00:26:25.375173 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/52179d08-5861-4a23-9c8d-653c13bed339-host\") pod \"node-ca-t526b\" (UID: \"52179d08-5861-4a23-9c8d-653c13bed339\") " pod="openshift-image-registry/node-ca-t526b" Jul 10 00:26:25.376828 ip-10-0-130-146 kubenswrapper[2570]: I0710 00:26:25.375132 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/5e419a12-f1a1-4b3d-8c72-7c6af574948f-agent-certs\") pod \"konnectivity-agent-995ch\" (UID: \"5e419a12-f1a1-4b3d-8c72-7c6af574948f\") " pod="kube-system/konnectivity-agent-995ch" Jul 10 00:26:25.376828 ip-10-0-130-146 kubenswrapper[2570]: I0710 00:26:25.375585 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/c44f0289-9f01-4548-ab98-74a17ec2b150-socket-dir\") pod \"aws-ebs-csi-driver-node-68tgb\" (UID: \"c44f0289-9f01-4548-ab98-74a17ec2b150\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-68tgb" Jul 10 00:26:25.376828 ip-10-0-130-146 kubenswrapper[2570]: I0710 00:26:25.375601 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/c44f0289-9f01-4548-ab98-74a17ec2b150-etc-selinux\") pod \"aws-ebs-csi-driver-node-68tgb\" (UID: \"c44f0289-9f01-4548-ab98-74a17ec2b150\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-68tgb" Jul 10 00:26:25.376828 ip-10-0-130-146 kubenswrapper[2570]: I0710 00:26:25.375620 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rtmcc\" (UniqueName: \"kubernetes.io/projected/78f9f66c-5c2f-4cee-bf34-39961a78dbcc-kube-api-access-rtmcc\") pod \"multus-additional-cni-plugins-7qfbg\" (UID: \"78f9f66c-5c2f-4cee-bf34-39961a78dbcc\") " pod="openshift-multus/multus-additional-cni-plugins-7qfbg" Jul 10 00:26:25.376828 ip-10-0-130-146 kubenswrapper[2570]: I0710 00:26:25.375636 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/173d0656-f60a-40da-8ff5-2e92a0ca8a22-etc-sysctl-conf\") pod \"tuned-7hmqd\" (UID: \"173d0656-f60a-40da-8ff5-2e92a0ca8a22\") " pod="openshift-cluster-node-tuning-operator/tuned-7hmqd" Jul 10 00:26:25.376828 ip-10-0-130-146 kubenswrapper[2570]: I0710 00:26:25.375654 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/c4ef019a-364a-4933-ab85-de86d86167f1-etc-openvswitch\") pod \"ovnkube-node-mbs58\" (UID: \"c4ef019a-364a-4933-ab85-de86d86167f1\") " pod="openshift-ovn-kubernetes/ovnkube-node-mbs58" Jul 10 00:26:25.376828 ip-10-0-130-146 kubenswrapper[2570]: I0710 00:26:25.375676 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/5667efb4-6019-4bde-827a-0f1e68096b93-kubelet-config\") pod \"global-pull-secret-syncer-6bqvb\" (UID: \"5667efb4-6019-4bde-827a-0f1e68096b93\") " pod="kube-system/global-pull-secret-syncer-6bqvb" Jul 10 00:26:25.376828 ip-10-0-130-146 kubenswrapper[2570]: I0710 00:26:25.375700 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b7aab806-00a2-491e-a997-ed3174bd547a-metrics-certs\") pod \"network-metrics-daemon-d8t56\" (UID: \"b7aab806-00a2-491e-a997-ed3174bd547a\") " pod="openshift-multus/network-metrics-daemon-d8t56" Jul 10 00:26:25.376828 ip-10-0-130-146 kubenswrapper[2570]: I0710 00:26:25.375743 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sp2wx\" (UniqueName: \"kubernetes.io/projected/5dcf6a9a-85e6-4f25-85b3-7b8a9eaa8654-kube-api-access-sp2wx\") pod \"iptables-alerter-bw9kx\" (UID: \"5dcf6a9a-85e6-4f25-85b3-7b8a9eaa8654\") " pod="openshift-network-operator/iptables-alerter-bw9kx" Jul 10 00:26:25.376828 ip-10-0-130-146 kubenswrapper[2570]: I0710 00:26:25.375766 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/173d0656-f60a-40da-8ff5-2e92a0ca8a22-etc-sysctl-conf\") pod \"tuned-7hmqd\" (UID: \"173d0656-f60a-40da-8ff5-2e92a0ca8a22\") " pod="openshift-cluster-node-tuning-operator/tuned-7hmqd" Jul 10 00:26:25.376828 ip-10-0-130-146 kubenswrapper[2570]: I0710 00:26:25.375871 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/c44f0289-9f01-4548-ab98-74a17ec2b150-device-dir\") pod \"aws-ebs-csi-driver-node-68tgb\" (UID: \"c44f0289-9f01-4548-ab98-74a17ec2b150\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-68tgb" Jul 10 00:26:25.376828 ip-10-0-130-146 kubenswrapper[2570]: I0710 00:26:25.375927 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/5667efb4-6019-4bde-827a-0f1e68096b93-dbus\") pod \"global-pull-secret-syncer-6bqvb\" (UID: \"5667efb4-6019-4bde-827a-0f1e68096b93\") " pod="kube-system/global-pull-secret-syncer-6bqvb" Jul 10 00:26:25.376828 ip-10-0-130-146 kubenswrapper[2570]: I0710 00:26:25.375971 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/f726a6ab-3c92-403e-ad1b-bdaf7e37c5e3-host-run-multus-certs\") pod \"multus-2pdkh\" (UID: \"f726a6ab-3c92-403e-ad1b-bdaf7e37c5e3\") " pod="openshift-multus/multus-2pdkh" Jul 10 00:26:25.376828 ip-10-0-130-146 kubenswrapper[2570]: E0710 00:26:25.376124 2570 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 10 00:26:25.377285 ip-10-0-130-146 kubenswrapper[2570]: E0710 00:26:25.376198 2570 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b7aab806-00a2-491e-a997-ed3174bd547a-metrics-certs podName:b7aab806-00a2-491e-a997-ed3174bd547a nodeName:}" failed. No retries permitted until 2026-07-10 00:26:25.876175991 +0000 UTC m=+4.924554709 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/b7aab806-00a2-491e-a997-ed3174bd547a-metrics-certs") pod "network-metrics-daemon-d8t56" (UID: "b7aab806-00a2-491e-a997-ed3174bd547a") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 10 00:26:25.377628 ip-10-0-130-146 kubenswrapper[2570]: I0710 00:26:25.377609 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/173d0656-f60a-40da-8ff5-2e92a0ca8a22-tmp\") pod \"tuned-7hmqd\" (UID: \"173d0656-f60a-40da-8ff5-2e92a0ca8a22\") " pod="openshift-cluster-node-tuning-operator/tuned-7hmqd" Jul 10 00:26:25.377661 ip-10-0-130-146 kubenswrapper[2570]: I0710 00:26:25.377620 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/173d0656-f60a-40da-8ff5-2e92a0ca8a22-etc-tuned\") pod \"tuned-7hmqd\" (UID: \"173d0656-f60a-40da-8ff5-2e92a0ca8a22\") " pod="openshift-cluster-node-tuning-operator/tuned-7hmqd" Jul 10 00:26:25.384933 ip-10-0-130-146 kubenswrapper[2570]: I0710 00:26:25.384910 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rtmcc\" (UniqueName: \"kubernetes.io/projected/78f9f66c-5c2f-4cee-bf34-39961a78dbcc-kube-api-access-rtmcc\") pod \"multus-additional-cni-plugins-7qfbg\" (UID: \"78f9f66c-5c2f-4cee-bf34-39961a78dbcc\") " pod="openshift-multus/multus-additional-cni-plugins-7qfbg" Jul 10 00:26:25.385233 ip-10-0-130-146 kubenswrapper[2570]: I0710 00:26:25.385213 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-sp2wx\" (UniqueName: \"kubernetes.io/projected/5dcf6a9a-85e6-4f25-85b3-7b8a9eaa8654-kube-api-access-sp2wx\") pod \"iptables-alerter-bw9kx\" (UID: \"5dcf6a9a-85e6-4f25-85b3-7b8a9eaa8654\") " pod="openshift-network-operator/iptables-alerter-bw9kx" Jul 10 00:26:25.385281 ip-10-0-130-146 kubenswrapper[2570]: I0710 00:26:25.385248 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-gp6ct\" (UniqueName: \"kubernetes.io/projected/52179d08-5861-4a23-9c8d-653c13bed339-kube-api-access-gp6ct\") pod \"node-ca-t526b\" (UID: \"52179d08-5861-4a23-9c8d-653c13bed339\") " pod="openshift-image-registry/node-ca-t526b" Jul 10 00:26:25.385372 ip-10-0-130-146 kubenswrapper[2570]: I0710 00:26:25.385352 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vf7hx\" (UniqueName: \"kubernetes.io/projected/006ba44e-264c-42cd-9891-ad8d769386c8-kube-api-access-vf7hx\") pod \"node-resolver-7tn6x\" (UID: \"006ba44e-264c-42cd-9891-ad8d769386c8\") " pod="openshift-dns/node-resolver-7tn6x" Jul 10 00:26:25.386073 ip-10-0-130-146 kubenswrapper[2570]: I0710 00:26:25.386041 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-fjxt7\" (UniqueName: \"kubernetes.io/projected/173d0656-f60a-40da-8ff5-2e92a0ca8a22-kube-api-access-fjxt7\") pod \"tuned-7hmqd\" (UID: \"173d0656-f60a-40da-8ff5-2e92a0ca8a22\") " pod="openshift-cluster-node-tuning-operator/tuned-7hmqd" Jul 10 00:26:25.386828 ip-10-0-130-146 kubenswrapper[2570]: I0710 00:26:25.386813 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jsds9\" (UniqueName: \"kubernetes.io/projected/b7aab806-00a2-491e-a997-ed3174bd547a-kube-api-access-jsds9\") pod \"network-metrics-daemon-d8t56\" (UID: \"b7aab806-00a2-491e-a997-ed3174bd547a\") " pod="openshift-multus/network-metrics-daemon-d8t56" Jul 10 00:26:25.458955 ip-10-0-130-146 kubenswrapper[2570]: I0710 00:26:25.458924 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-130-146.ec2.internal" event={"ID":"277746f2e3b301c16ca7c3a5af84dd5d","Type":"ContainerStarted","Data":"d0e0b1c3601677df74af63579d9388f626bba9418edfca645009bae40125c8c8"} Jul 10 00:26:25.459105 ip-10-0-130-146 kubenswrapper[2570]: I0710 00:26:25.459094 2570 kubelet.go:3248] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-130-146.ec2.internal" Jul 10 00:26:25.460112 ip-10-0-130-146 kubenswrapper[2570]: I0710 00:26:25.460093 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-146.ec2.internal" event={"ID":"0ddb9c9bc924902db48bbad9696eddc0","Type":"ContainerStarted","Data":"58668167646ec43b1e79a568487892f07a3a1007a1eb4579dd96a270e27a737b"} Jul 10 00:26:25.460267 ip-10-0-130-146 kubenswrapper[2570]: I0710 00:26:25.460250 2570 kubelet.go:3248] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-146.ec2.internal" Jul 10 00:26:25.471976 ip-10-0-130-146 kubenswrapper[2570]: I0710 00:26:25.471959 2570 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Jul 10 00:26:25.472096 ip-10-0-130-146 kubenswrapper[2570]: E0710 00:26:25.472001 2570 kubelet.go:3250] "Failed creating a mirror pod" err="pods \"kube-apiserver-proxy-ip-10-0-130-146.ec2.internal\" already exists" pod="kube-system/kube-apiserver-proxy-ip-10-0-130-146.ec2.internal" Jul 10 00:26:25.472379 ip-10-0-130-146 kubenswrapper[2570]: I0710 00:26:25.472365 2570 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Jul 10 00:26:25.472460 ip-10-0-130-146 kubenswrapper[2570]: E0710 00:26:25.472400 2570 kubelet.go:3250] "Failed creating a mirror pod" err="pods \"kube-rbac-proxy-crio-ip-10-0-130-146.ec2.internal\" already exists" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-146.ec2.internal" Jul 10 00:26:25.476189 ip-10-0-130-146 kubenswrapper[2570]: I0710 00:26:25.476174 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/f726a6ab-3c92-403e-ad1b-bdaf7e37c5e3-system-cni-dir\") pod \"multus-2pdkh\" (UID: \"f726a6ab-3c92-403e-ad1b-bdaf7e37c5e3\") " pod="openshift-multus/multus-2pdkh" Jul 10 00:26:25.476264 ip-10-0-130-146 kubenswrapper[2570]: I0710 00:26:25.476197 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/f726a6ab-3c92-403e-ad1b-bdaf7e37c5e3-host-var-lib-cni-multus\") pod \"multus-2pdkh\" (UID: \"f726a6ab-3c92-403e-ad1b-bdaf7e37c5e3\") " pod="openshift-multus/multus-2pdkh" Jul 10 00:26:25.476264 ip-10-0-130-146 kubenswrapper[2570]: I0710 00:26:25.476214 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/f726a6ab-3c92-403e-ad1b-bdaf7e37c5e3-hostroot\") pod \"multus-2pdkh\" (UID: \"f726a6ab-3c92-403e-ad1b-bdaf7e37c5e3\") " pod="openshift-multus/multus-2pdkh" Jul 10 00:26:25.476264 ip-10-0-130-146 kubenswrapper[2570]: I0710 00:26:25.476230 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/c4ef019a-364a-4933-ab85-de86d86167f1-host-kubelet\") pod \"ovnkube-node-mbs58\" (UID: \"c4ef019a-364a-4933-ab85-de86d86167f1\") " pod="openshift-ovn-kubernetes/ovnkube-node-mbs58" Jul 10 00:26:25.476264 ip-10-0-130-146 kubenswrapper[2570]: I0710 00:26:25.476245 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/c44f0289-9f01-4548-ab98-74a17ec2b150-registration-dir\") pod \"aws-ebs-csi-driver-node-68tgb\" (UID: \"c44f0289-9f01-4548-ab98-74a17ec2b150\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-68tgb" Jul 10 00:26:25.476264 ip-10-0-130-146 kubenswrapper[2570]: I0710 00:26:25.476259 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/f726a6ab-3c92-403e-ad1b-bdaf7e37c5e3-cni-binary-copy\") pod \"multus-2pdkh\" (UID: \"f726a6ab-3c92-403e-ad1b-bdaf7e37c5e3\") " pod="openshift-multus/multus-2pdkh" Jul 10 00:26:25.476485 ip-10-0-130-146 kubenswrapper[2570]: I0710 00:26:25.476273 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/f726a6ab-3c92-403e-ad1b-bdaf7e37c5e3-multus-conf-dir\") pod \"multus-2pdkh\" (UID: \"f726a6ab-3c92-403e-ad1b-bdaf7e37c5e3\") " pod="openshift-multus/multus-2pdkh" Jul 10 00:26:25.476485 ip-10-0-130-146 kubenswrapper[2570]: I0710 00:26:25.476288 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/c4ef019a-364a-4933-ab85-de86d86167f1-env-overrides\") pod \"ovnkube-node-mbs58\" (UID: \"c4ef019a-364a-4933-ab85-de86d86167f1\") " pod="openshift-ovn-kubernetes/ovnkube-node-mbs58" Jul 10 00:26:25.476485 ip-10-0-130-146 kubenswrapper[2570]: I0710 00:26:25.476292 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/f726a6ab-3c92-403e-ad1b-bdaf7e37c5e3-hostroot\") pod \"multus-2pdkh\" (UID: \"f726a6ab-3c92-403e-ad1b-bdaf7e37c5e3\") " pod="openshift-multus/multus-2pdkh" Jul 10 00:26:25.476485 ip-10-0-130-146 kubenswrapper[2570]: I0710 00:26:25.476278 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/f726a6ab-3c92-403e-ad1b-bdaf7e37c5e3-host-var-lib-cni-multus\") pod \"multus-2pdkh\" (UID: \"f726a6ab-3c92-403e-ad1b-bdaf7e37c5e3\") " pod="openshift-multus/multus-2pdkh" Jul 10 00:26:25.476485 ip-10-0-130-146 kubenswrapper[2570]: I0710 00:26:25.476304 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/c4ef019a-364a-4933-ab85-de86d86167f1-host-kubelet\") pod \"ovnkube-node-mbs58\" (UID: \"c4ef019a-364a-4933-ab85-de86d86167f1\") " pod="openshift-ovn-kubernetes/ovnkube-node-mbs58" Jul 10 00:26:25.476485 ip-10-0-130-146 kubenswrapper[2570]: I0710 00:26:25.476311 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kl6s9\" (UniqueName: \"kubernetes.io/projected/78717d89-03f2-4824-942a-48892a2bfb99-kube-api-access-kl6s9\") pod \"network-check-target-5hs2b\" (UID: \"78717d89-03f2-4824-942a-48892a2bfb99\") " pod="openshift-network-diagnostics/network-check-target-5hs2b" Jul 10 00:26:25.476485 ip-10-0-130-146 kubenswrapper[2570]: I0710 00:26:25.476333 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/c44f0289-9f01-4548-ab98-74a17ec2b150-registration-dir\") pod \"aws-ebs-csi-driver-node-68tgb\" (UID: \"c44f0289-9f01-4548-ab98-74a17ec2b150\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-68tgb" Jul 10 00:26:25.476485 ip-10-0-130-146 kubenswrapper[2570]: I0710 00:26:25.476349 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/f726a6ab-3c92-403e-ad1b-bdaf7e37c5e3-multus-cni-dir\") pod \"multus-2pdkh\" (UID: \"f726a6ab-3c92-403e-ad1b-bdaf7e37c5e3\") " pod="openshift-multus/multus-2pdkh" Jul 10 00:26:25.476485 ip-10-0-130-146 kubenswrapper[2570]: I0710 00:26:25.476336 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/f726a6ab-3c92-403e-ad1b-bdaf7e37c5e3-multus-conf-dir\") pod \"multus-2pdkh\" (UID: \"f726a6ab-3c92-403e-ad1b-bdaf7e37c5e3\") " pod="openshift-multus/multus-2pdkh" Jul 10 00:26:25.476485 ip-10-0-130-146 kubenswrapper[2570]: I0710 00:26:25.476347 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/f726a6ab-3c92-403e-ad1b-bdaf7e37c5e3-system-cni-dir\") pod \"multus-2pdkh\" (UID: \"f726a6ab-3c92-403e-ad1b-bdaf7e37c5e3\") " pod="openshift-multus/multus-2pdkh" Jul 10 00:26:25.476485 ip-10-0-130-146 kubenswrapper[2570]: I0710 00:26:25.476395 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/c4ef019a-364a-4933-ab85-de86d86167f1-host-slash\") pod \"ovnkube-node-mbs58\" (UID: \"c4ef019a-364a-4933-ab85-de86d86167f1\") " pod="openshift-ovn-kubernetes/ovnkube-node-mbs58" Jul 10 00:26:25.476485 ip-10-0-130-146 kubenswrapper[2570]: I0710 00:26:25.476411 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/f726a6ab-3c92-403e-ad1b-bdaf7e37c5e3-multus-cni-dir\") pod \"multus-2pdkh\" (UID: \"f726a6ab-3c92-403e-ad1b-bdaf7e37c5e3\") " pod="openshift-multus/multus-2pdkh" Jul 10 00:26:25.476485 ip-10-0-130-146 kubenswrapper[2570]: I0710 00:26:25.476418 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/c4ef019a-364a-4933-ab85-de86d86167f1-host-run-netns\") pod \"ovnkube-node-mbs58\" (UID: \"c4ef019a-364a-4933-ab85-de86d86167f1\") " pod="openshift-ovn-kubernetes/ovnkube-node-mbs58" Jul 10 00:26:25.476485 ip-10-0-130-146 kubenswrapper[2570]: I0710 00:26:25.476440 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/f726a6ab-3c92-403e-ad1b-bdaf7e37c5e3-multus-socket-dir-parent\") pod \"multus-2pdkh\" (UID: \"f726a6ab-3c92-403e-ad1b-bdaf7e37c5e3\") " pod="openshift-multus/multus-2pdkh" Jul 10 00:26:25.476485 ip-10-0-130-146 kubenswrapper[2570]: I0710 00:26:25.476446 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/c4ef019a-364a-4933-ab85-de86d86167f1-host-slash\") pod \"ovnkube-node-mbs58\" (UID: \"c4ef019a-364a-4933-ab85-de86d86167f1\") " pod="openshift-ovn-kubernetes/ovnkube-node-mbs58" Jul 10 00:26:25.476485 ip-10-0-130-146 kubenswrapper[2570]: I0710 00:26:25.476464 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/c4ef019a-364a-4933-ab85-de86d86167f1-var-lib-openvswitch\") pod \"ovnkube-node-mbs58\" (UID: \"c4ef019a-364a-4933-ab85-de86d86167f1\") " pod="openshift-ovn-kubernetes/ovnkube-node-mbs58" Jul 10 00:26:25.476485 ip-10-0-130-146 kubenswrapper[2570]: I0710 00:26:25.476488 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/c4ef019a-364a-4933-ab85-de86d86167f1-host-run-netns\") pod \"ovnkube-node-mbs58\" (UID: \"c4ef019a-364a-4933-ab85-de86d86167f1\") " pod="openshift-ovn-kubernetes/ovnkube-node-mbs58" Jul 10 00:26:25.477207 ip-10-0-130-146 kubenswrapper[2570]: I0710 00:26:25.476498 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/c4ef019a-364a-4933-ab85-de86d86167f1-log-socket\") pod \"ovnkube-node-mbs58\" (UID: \"c4ef019a-364a-4933-ab85-de86d86167f1\") " pod="openshift-ovn-kubernetes/ovnkube-node-mbs58" Jul 10 00:26:25.477207 ip-10-0-130-146 kubenswrapper[2570]: I0710 00:26:25.476531 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/c4ef019a-364a-4933-ab85-de86d86167f1-log-socket\") pod \"ovnkube-node-mbs58\" (UID: \"c4ef019a-364a-4933-ab85-de86d86167f1\") " pod="openshift-ovn-kubernetes/ovnkube-node-mbs58" Jul 10 00:26:25.477207 ip-10-0-130-146 kubenswrapper[2570]: I0710 00:26:25.476533 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/f726a6ab-3c92-403e-ad1b-bdaf7e37c5e3-multus-socket-dir-parent\") pod \"multus-2pdkh\" (UID: \"f726a6ab-3c92-403e-ad1b-bdaf7e37c5e3\") " pod="openshift-multus/multus-2pdkh" Jul 10 00:26:25.477207 ip-10-0-130-146 kubenswrapper[2570]: I0710 00:26:25.476529 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/c4ef019a-364a-4933-ab85-de86d86167f1-var-lib-openvswitch\") pod \"ovnkube-node-mbs58\" (UID: \"c4ef019a-364a-4933-ab85-de86d86167f1\") " pod="openshift-ovn-kubernetes/ovnkube-node-mbs58" Jul 10 00:26:25.477207 ip-10-0-130-146 kubenswrapper[2570]: I0710 00:26:25.476533 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/c4ef019a-364a-4933-ab85-de86d86167f1-ovn-node-metrics-cert\") pod \"ovnkube-node-mbs58\" (UID: \"c4ef019a-364a-4933-ab85-de86d86167f1\") " pod="openshift-ovn-kubernetes/ovnkube-node-mbs58" Jul 10 00:26:25.477207 ip-10-0-130-146 kubenswrapper[2570]: I0710 00:26:25.476575 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/c44f0289-9f01-4548-ab98-74a17ec2b150-kubelet-dir\") pod \"aws-ebs-csi-driver-node-68tgb\" (UID: \"c44f0289-9f01-4548-ab98-74a17ec2b150\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-68tgb" Jul 10 00:26:25.477207 ip-10-0-130-146 kubenswrapper[2570]: I0710 00:26:25.476599 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/c4ef019a-364a-4933-ab85-de86d86167f1-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-mbs58\" (UID: \"c4ef019a-364a-4933-ab85-de86d86167f1\") " pod="openshift-ovn-kubernetes/ovnkube-node-mbs58" Jul 10 00:26:25.477207 ip-10-0-130-146 kubenswrapper[2570]: I0710 00:26:25.476617 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5m6sc\" (UniqueName: \"kubernetes.io/projected/c4ef019a-364a-4933-ab85-de86d86167f1-kube-api-access-5m6sc\") pod \"ovnkube-node-mbs58\" (UID: \"c4ef019a-364a-4933-ab85-de86d86167f1\") " pod="openshift-ovn-kubernetes/ovnkube-node-mbs58" Jul 10 00:26:25.477207 ip-10-0-130-146 kubenswrapper[2570]: I0710 00:26:25.476632 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/5667efb4-6019-4bde-827a-0f1e68096b93-original-pull-secret\") pod \"global-pull-secret-syncer-6bqvb\" (UID: \"5667efb4-6019-4bde-827a-0f1e68096b93\") " pod="kube-system/global-pull-secret-syncer-6bqvb" Jul 10 00:26:25.477207 ip-10-0-130-146 kubenswrapper[2570]: I0710 00:26:25.476646 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/f726a6ab-3c92-403e-ad1b-bdaf7e37c5e3-host-run-k8s-cni-cncf-io\") pod \"multus-2pdkh\" (UID: \"f726a6ab-3c92-403e-ad1b-bdaf7e37c5e3\") " pod="openshift-multus/multus-2pdkh" Jul 10 00:26:25.477207 ip-10-0-130-146 kubenswrapper[2570]: I0710 00:26:25.476667 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/f726a6ab-3c92-403e-ad1b-bdaf7e37c5e3-host-run-netns\") pod \"multus-2pdkh\" (UID: \"f726a6ab-3c92-403e-ad1b-bdaf7e37c5e3\") " pod="openshift-multus/multus-2pdkh" Jul 10 00:26:25.477207 ip-10-0-130-146 kubenswrapper[2570]: I0710 00:26:25.476690 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/c4ef019a-364a-4933-ab85-de86d86167f1-node-log\") pod \"ovnkube-node-mbs58\" (UID: \"c4ef019a-364a-4933-ab85-de86d86167f1\") " pod="openshift-ovn-kubernetes/ovnkube-node-mbs58" Jul 10 00:26:25.477207 ip-10-0-130-146 kubenswrapper[2570]: I0710 00:26:25.476725 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/c4ef019a-364a-4933-ab85-de86d86167f1-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-mbs58\" (UID: \"c4ef019a-364a-4933-ab85-de86d86167f1\") " pod="openshift-ovn-kubernetes/ovnkube-node-mbs58" Jul 10 00:26:25.477207 ip-10-0-130-146 kubenswrapper[2570]: I0710 00:26:25.476734 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/c4ef019a-364a-4933-ab85-de86d86167f1-node-log\") pod \"ovnkube-node-mbs58\" (UID: \"c4ef019a-364a-4933-ab85-de86d86167f1\") " pod="openshift-ovn-kubernetes/ovnkube-node-mbs58" Jul 10 00:26:25.477207 ip-10-0-130-146 kubenswrapper[2570]: I0710 00:26:25.476690 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/c44f0289-9f01-4548-ab98-74a17ec2b150-kubelet-dir\") pod \"aws-ebs-csi-driver-node-68tgb\" (UID: \"c44f0289-9f01-4548-ab98-74a17ec2b150\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-68tgb" Jul 10 00:26:25.477207 ip-10-0-130-146 kubenswrapper[2570]: I0710 00:26:25.476715 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/c4ef019a-364a-4933-ab85-de86d86167f1-host-cni-netd\") pod \"ovnkube-node-mbs58\" (UID: \"c4ef019a-364a-4933-ab85-de86d86167f1\") " pod="openshift-ovn-kubernetes/ovnkube-node-mbs58" Jul 10 00:26:25.477207 ip-10-0-130-146 kubenswrapper[2570]: I0710 00:26:25.476760 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/f726a6ab-3c92-403e-ad1b-bdaf7e37c5e3-host-run-netns\") pod \"multus-2pdkh\" (UID: \"f726a6ab-3c92-403e-ad1b-bdaf7e37c5e3\") " pod="openshift-multus/multus-2pdkh" Jul 10 00:26:25.477984 ip-10-0-130-146 kubenswrapper[2570]: I0710 00:26:25.476730 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/f726a6ab-3c92-403e-ad1b-bdaf7e37c5e3-host-run-k8s-cni-cncf-io\") pod \"multus-2pdkh\" (UID: \"f726a6ab-3c92-403e-ad1b-bdaf7e37c5e3\") " pod="openshift-multus/multus-2pdkh" Jul 10 00:26:25.477984 ip-10-0-130-146 kubenswrapper[2570]: E0710 00:26:25.476756 2570 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 10 00:26:25.477984 ip-10-0-130-146 kubenswrapper[2570]: I0710 00:26:25.476790 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/c44f0289-9f01-4548-ab98-74a17ec2b150-sys-fs\") pod \"aws-ebs-csi-driver-node-68tgb\" (UID: \"c44f0289-9f01-4548-ab98-74a17ec2b150\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-68tgb" Jul 10 00:26:25.477984 ip-10-0-130-146 kubenswrapper[2570]: I0710 00:26:25.476816 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/f726a6ab-3c92-403e-ad1b-bdaf7e37c5e3-multus-daemon-config\") pod \"multus-2pdkh\" (UID: \"f726a6ab-3c92-403e-ad1b-bdaf7e37c5e3\") " pod="openshift-multus/multus-2pdkh" Jul 10 00:26:25.477984 ip-10-0-130-146 kubenswrapper[2570]: I0710 00:26:25.476829 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/c44f0289-9f01-4548-ab98-74a17ec2b150-sys-fs\") pod \"aws-ebs-csi-driver-node-68tgb\" (UID: \"c44f0289-9f01-4548-ab98-74a17ec2b150\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-68tgb" Jul 10 00:26:25.477984 ip-10-0-130-146 kubenswrapper[2570]: I0710 00:26:25.476791 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/c4ef019a-364a-4933-ab85-de86d86167f1-host-cni-netd\") pod \"ovnkube-node-mbs58\" (UID: \"c4ef019a-364a-4933-ab85-de86d86167f1\") " pod="openshift-ovn-kubernetes/ovnkube-node-mbs58" Jul 10 00:26:25.477984 ip-10-0-130-146 kubenswrapper[2570]: E0710 00:26:25.476852 2570 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5667efb4-6019-4bde-827a-0f1e68096b93-original-pull-secret podName:5667efb4-6019-4bde-827a-0f1e68096b93 nodeName:}" failed. No retries permitted until 2026-07-10 00:26:25.97681514 +0000 UTC m=+5.025193854 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/5667efb4-6019-4bde-827a-0f1e68096b93-original-pull-secret") pod "global-pull-secret-syncer-6bqvb" (UID: "5667efb4-6019-4bde-827a-0f1e68096b93") : object "kube-system"/"original-pull-secret" not registered Jul 10 00:26:25.477984 ip-10-0-130-146 kubenswrapper[2570]: I0710 00:26:25.476882 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/f726a6ab-3c92-403e-ad1b-bdaf7e37c5e3-cni-binary-copy\") pod \"multus-2pdkh\" (UID: \"f726a6ab-3c92-403e-ad1b-bdaf7e37c5e3\") " pod="openshift-multus/multus-2pdkh" Jul 10 00:26:25.477984 ip-10-0-130-146 kubenswrapper[2570]: I0710 00:26:25.476885 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/f726a6ab-3c92-403e-ad1b-bdaf7e37c5e3-etc-kubernetes\") pod \"multus-2pdkh\" (UID: \"f726a6ab-3c92-403e-ad1b-bdaf7e37c5e3\") " pod="openshift-multus/multus-2pdkh" Jul 10 00:26:25.477984 ip-10-0-130-146 kubenswrapper[2570]: I0710 00:26:25.476933 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/5e419a12-f1a1-4b3d-8c72-7c6af574948f-agent-certs\") pod \"konnectivity-agent-995ch\" (UID: \"5e419a12-f1a1-4b3d-8c72-7c6af574948f\") " pod="kube-system/konnectivity-agent-995ch" Jul 10 00:26:25.477984 ip-10-0-130-146 kubenswrapper[2570]: I0710 00:26:25.476972 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/f726a6ab-3c92-403e-ad1b-bdaf7e37c5e3-etc-kubernetes\") pod \"multus-2pdkh\" (UID: \"f726a6ab-3c92-403e-ad1b-bdaf7e37c5e3\") " pod="openshift-multus/multus-2pdkh" Jul 10 00:26:25.477984 ip-10-0-130-146 kubenswrapper[2570]: I0710 00:26:25.476985 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/c44f0289-9f01-4548-ab98-74a17ec2b150-socket-dir\") pod \"aws-ebs-csi-driver-node-68tgb\" (UID: \"c44f0289-9f01-4548-ab98-74a17ec2b150\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-68tgb" Jul 10 00:26:25.477984 ip-10-0-130-146 kubenswrapper[2570]: I0710 00:26:25.477010 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/c44f0289-9f01-4548-ab98-74a17ec2b150-etc-selinux\") pod \"aws-ebs-csi-driver-node-68tgb\" (UID: \"c44f0289-9f01-4548-ab98-74a17ec2b150\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-68tgb" Jul 10 00:26:25.477984 ip-10-0-130-146 kubenswrapper[2570]: I0710 00:26:25.477029 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/c4ef019a-364a-4933-ab85-de86d86167f1-etc-openvswitch\") pod \"ovnkube-node-mbs58\" (UID: \"c4ef019a-364a-4933-ab85-de86d86167f1\") " pod="openshift-ovn-kubernetes/ovnkube-node-mbs58" Jul 10 00:26:25.477984 ip-10-0-130-146 kubenswrapper[2570]: I0710 00:26:25.477059 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/5667efb4-6019-4bde-827a-0f1e68096b93-kubelet-config\") pod \"global-pull-secret-syncer-6bqvb\" (UID: \"5667efb4-6019-4bde-827a-0f1e68096b93\") " pod="kube-system/global-pull-secret-syncer-6bqvb" Jul 10 00:26:25.477984 ip-10-0-130-146 kubenswrapper[2570]: I0710 00:26:25.477096 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/c44f0289-9f01-4548-ab98-74a17ec2b150-device-dir\") pod \"aws-ebs-csi-driver-node-68tgb\" (UID: \"c44f0289-9f01-4548-ab98-74a17ec2b150\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-68tgb" Jul 10 00:26:25.477984 ip-10-0-130-146 kubenswrapper[2570]: I0710 00:26:25.477100 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/c44f0289-9f01-4548-ab98-74a17ec2b150-socket-dir\") pod \"aws-ebs-csi-driver-node-68tgb\" (UID: \"c44f0289-9f01-4548-ab98-74a17ec2b150\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-68tgb" Jul 10 00:26:25.478807 ip-10-0-130-146 kubenswrapper[2570]: I0710 00:26:25.477110 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/c4ef019a-364a-4933-ab85-de86d86167f1-etc-openvswitch\") pod \"ovnkube-node-mbs58\" (UID: \"c4ef019a-364a-4933-ab85-de86d86167f1\") " pod="openshift-ovn-kubernetes/ovnkube-node-mbs58" Jul 10 00:26:25.478807 ip-10-0-130-146 kubenswrapper[2570]: I0710 00:26:25.477111 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/c44f0289-9f01-4548-ab98-74a17ec2b150-etc-selinux\") pod \"aws-ebs-csi-driver-node-68tgb\" (UID: \"c44f0289-9f01-4548-ab98-74a17ec2b150\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-68tgb" Jul 10 00:26:25.478807 ip-10-0-130-146 kubenswrapper[2570]: I0710 00:26:25.477120 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/5667efb4-6019-4bde-827a-0f1e68096b93-dbus\") pod \"global-pull-secret-syncer-6bqvb\" (UID: \"5667efb4-6019-4bde-827a-0f1e68096b93\") " pod="kube-system/global-pull-secret-syncer-6bqvb" Jul 10 00:26:25.478807 ip-10-0-130-146 kubenswrapper[2570]: I0710 00:26:25.477143 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/f726a6ab-3c92-403e-ad1b-bdaf7e37c5e3-host-run-multus-certs\") pod \"multus-2pdkh\" (UID: \"f726a6ab-3c92-403e-ad1b-bdaf7e37c5e3\") " pod="openshift-multus/multus-2pdkh" Jul 10 00:26:25.478807 ip-10-0-130-146 kubenswrapper[2570]: I0710 00:26:25.477151 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/5667efb4-6019-4bde-827a-0f1e68096b93-kubelet-config\") pod \"global-pull-secret-syncer-6bqvb\" (UID: \"5667efb4-6019-4bde-827a-0f1e68096b93\") " pod="kube-system/global-pull-secret-syncer-6bqvb" Jul 10 00:26:25.478807 ip-10-0-130-146 kubenswrapper[2570]: I0710 00:26:25.477166 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gw9kb\" (UniqueName: \"kubernetes.io/projected/f726a6ab-3c92-403e-ad1b-bdaf7e37c5e3-kube-api-access-gw9kb\") pod \"multus-2pdkh\" (UID: \"f726a6ab-3c92-403e-ad1b-bdaf7e37c5e3\") " pod="openshift-multus/multus-2pdkh" Jul 10 00:26:25.478807 ip-10-0-130-146 kubenswrapper[2570]: I0710 00:26:25.477191 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/c4ef019a-364a-4933-ab85-de86d86167f1-run-ovn\") pod \"ovnkube-node-mbs58\" (UID: \"c4ef019a-364a-4933-ab85-de86d86167f1\") " pod="openshift-ovn-kubernetes/ovnkube-node-mbs58" Jul 10 00:26:25.478807 ip-10-0-130-146 kubenswrapper[2570]: I0710 00:26:25.477200 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/f726a6ab-3c92-403e-ad1b-bdaf7e37c5e3-host-run-multus-certs\") pod \"multus-2pdkh\" (UID: \"f726a6ab-3c92-403e-ad1b-bdaf7e37c5e3\") " pod="openshift-multus/multus-2pdkh" Jul 10 00:26:25.478807 ip-10-0-130-146 kubenswrapper[2570]: I0710 00:26:25.477168 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/c44f0289-9f01-4548-ab98-74a17ec2b150-device-dir\") pod \"aws-ebs-csi-driver-node-68tgb\" (UID: \"c44f0289-9f01-4548-ab98-74a17ec2b150\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-68tgb" Jul 10 00:26:25.478807 ip-10-0-130-146 kubenswrapper[2570]: I0710 00:26:25.477216 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/c4ef019a-364a-4933-ab85-de86d86167f1-host-run-ovn-kubernetes\") pod \"ovnkube-node-mbs58\" (UID: \"c4ef019a-364a-4933-ab85-de86d86167f1\") " pod="openshift-ovn-kubernetes/ovnkube-node-mbs58" Jul 10 00:26:25.478807 ip-10-0-130-146 kubenswrapper[2570]: I0710 00:26:25.477240 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/c4ef019a-364a-4933-ab85-de86d86167f1-ovnkube-config\") pod \"ovnkube-node-mbs58\" (UID: \"c4ef019a-364a-4933-ab85-de86d86167f1\") " pod="openshift-ovn-kubernetes/ovnkube-node-mbs58" Jul 10 00:26:25.478807 ip-10-0-130-146 kubenswrapper[2570]: I0710 00:26:25.477249 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/c4ef019a-364a-4933-ab85-de86d86167f1-run-ovn\") pod \"ovnkube-node-mbs58\" (UID: \"c4ef019a-364a-4933-ab85-de86d86167f1\") " pod="openshift-ovn-kubernetes/ovnkube-node-mbs58" Jul 10 00:26:25.478807 ip-10-0-130-146 kubenswrapper[2570]: I0710 00:26:25.477263 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/5667efb4-6019-4bde-827a-0f1e68096b93-dbus\") pod \"global-pull-secret-syncer-6bqvb\" (UID: \"5667efb4-6019-4bde-827a-0f1e68096b93\") " pod="kube-system/global-pull-secret-syncer-6bqvb" Jul 10 00:26:25.478807 ip-10-0-130-146 kubenswrapper[2570]: I0710 00:26:25.477262 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/f726a6ab-3c92-403e-ad1b-bdaf7e37c5e3-host-var-lib-cni-bin\") pod \"multus-2pdkh\" (UID: \"f726a6ab-3c92-403e-ad1b-bdaf7e37c5e3\") " pod="openshift-multus/multus-2pdkh" Jul 10 00:26:25.478807 ip-10-0-130-146 kubenswrapper[2570]: I0710 00:26:25.477289 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/c4ef019a-364a-4933-ab85-de86d86167f1-host-run-ovn-kubernetes\") pod \"ovnkube-node-mbs58\" (UID: \"c4ef019a-364a-4933-ab85-de86d86167f1\") " pod="openshift-ovn-kubernetes/ovnkube-node-mbs58" Jul 10 00:26:25.478807 ip-10-0-130-146 kubenswrapper[2570]: I0710 00:26:25.477302 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/f726a6ab-3c92-403e-ad1b-bdaf7e37c5e3-host-var-lib-cni-bin\") pod \"multus-2pdkh\" (UID: \"f726a6ab-3c92-403e-ad1b-bdaf7e37c5e3\") " pod="openshift-multus/multus-2pdkh" Jul 10 00:26:25.478807 ip-10-0-130-146 kubenswrapper[2570]: I0710 00:26:25.477321 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sst8k\" (UniqueName: \"kubernetes.io/projected/c44f0289-9f01-4548-ab98-74a17ec2b150-kube-api-access-sst8k\") pod \"aws-ebs-csi-driver-node-68tgb\" (UID: \"c44f0289-9f01-4548-ab98-74a17ec2b150\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-68tgb" Jul 10 00:26:25.479442 ip-10-0-130-146 kubenswrapper[2570]: I0710 00:26:25.477349 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/c4ef019a-364a-4933-ab85-de86d86167f1-systemd-units\") pod \"ovnkube-node-mbs58\" (UID: \"c4ef019a-364a-4933-ab85-de86d86167f1\") " pod="openshift-ovn-kubernetes/ovnkube-node-mbs58" Jul 10 00:26:25.479442 ip-10-0-130-146 kubenswrapper[2570]: I0710 00:26:25.477371 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/c4ef019a-364a-4933-ab85-de86d86167f1-run-systemd\") pod \"ovnkube-node-mbs58\" (UID: \"c4ef019a-364a-4933-ab85-de86d86167f1\") " pod="openshift-ovn-kubernetes/ovnkube-node-mbs58" Jul 10 00:26:25.479442 ip-10-0-130-146 kubenswrapper[2570]: I0710 00:26:25.477393 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/c4ef019a-364a-4933-ab85-de86d86167f1-ovnkube-script-lib\") pod \"ovnkube-node-mbs58\" (UID: \"c4ef019a-364a-4933-ab85-de86d86167f1\") " pod="openshift-ovn-kubernetes/ovnkube-node-mbs58" Jul 10 00:26:25.479442 ip-10-0-130-146 kubenswrapper[2570]: I0710 00:26:25.477416 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/c4ef019a-364a-4933-ab85-de86d86167f1-systemd-units\") pod \"ovnkube-node-mbs58\" (UID: \"c4ef019a-364a-4933-ab85-de86d86167f1\") " pod="openshift-ovn-kubernetes/ovnkube-node-mbs58" Jul 10 00:26:25.479442 ip-10-0-130-146 kubenswrapper[2570]: I0710 00:26:25.477417 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/f726a6ab-3c92-403e-ad1b-bdaf7e37c5e3-os-release\") pod \"multus-2pdkh\" (UID: \"f726a6ab-3c92-403e-ad1b-bdaf7e37c5e3\") " pod="openshift-multus/multus-2pdkh" Jul 10 00:26:25.479442 ip-10-0-130-146 kubenswrapper[2570]: I0710 00:26:25.477451 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/f726a6ab-3c92-403e-ad1b-bdaf7e37c5e3-host-var-lib-kubelet\") pod \"multus-2pdkh\" (UID: \"f726a6ab-3c92-403e-ad1b-bdaf7e37c5e3\") " pod="openshift-multus/multus-2pdkh" Jul 10 00:26:25.479442 ip-10-0-130-146 kubenswrapper[2570]: I0710 00:26:25.477480 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/f726a6ab-3c92-403e-ad1b-bdaf7e37c5e3-os-release\") pod \"multus-2pdkh\" (UID: \"f726a6ab-3c92-403e-ad1b-bdaf7e37c5e3\") " pod="openshift-multus/multus-2pdkh" Jul 10 00:26:25.479442 ip-10-0-130-146 kubenswrapper[2570]: I0710 00:26:25.477502 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/c4ef019a-364a-4933-ab85-de86d86167f1-run-openvswitch\") pod \"ovnkube-node-mbs58\" (UID: \"c4ef019a-364a-4933-ab85-de86d86167f1\") " pod="openshift-ovn-kubernetes/ovnkube-node-mbs58" Jul 10 00:26:25.479442 ip-10-0-130-146 kubenswrapper[2570]: I0710 00:26:25.477526 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/c4ef019a-364a-4933-ab85-de86d86167f1-run-systemd\") pod \"ovnkube-node-mbs58\" (UID: \"c4ef019a-364a-4933-ab85-de86d86167f1\") " pod="openshift-ovn-kubernetes/ovnkube-node-mbs58" Jul 10 00:26:25.479442 ip-10-0-130-146 kubenswrapper[2570]: I0710 00:26:25.477529 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/c4ef019a-364a-4933-ab85-de86d86167f1-host-cni-bin\") pod \"ovnkube-node-mbs58\" (UID: \"c4ef019a-364a-4933-ab85-de86d86167f1\") " pod="openshift-ovn-kubernetes/ovnkube-node-mbs58" Jul 10 00:26:25.479442 ip-10-0-130-146 kubenswrapper[2570]: I0710 00:26:25.477554 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/f726a6ab-3c92-403e-ad1b-bdaf7e37c5e3-host-var-lib-kubelet\") pod \"multus-2pdkh\" (UID: \"f726a6ab-3c92-403e-ad1b-bdaf7e37c5e3\") " pod="openshift-multus/multus-2pdkh" Jul 10 00:26:25.479442 ip-10-0-130-146 kubenswrapper[2570]: I0710 00:26:25.477564 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/c4ef019a-364a-4933-ab85-de86d86167f1-host-cni-bin\") pod \"ovnkube-node-mbs58\" (UID: \"c4ef019a-364a-4933-ab85-de86d86167f1\") " pod="openshift-ovn-kubernetes/ovnkube-node-mbs58" Jul 10 00:26:25.479442 ip-10-0-130-146 kubenswrapper[2570]: I0710 00:26:25.477590 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/f726a6ab-3c92-403e-ad1b-bdaf7e37c5e3-cnibin\") pod \"multus-2pdkh\" (UID: \"f726a6ab-3c92-403e-ad1b-bdaf7e37c5e3\") " pod="openshift-multus/multus-2pdkh" Jul 10 00:26:25.479442 ip-10-0-130-146 kubenswrapper[2570]: I0710 00:26:25.477600 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/c4ef019a-364a-4933-ab85-de86d86167f1-run-openvswitch\") pod \"ovnkube-node-mbs58\" (UID: \"c4ef019a-364a-4933-ab85-de86d86167f1\") " pod="openshift-ovn-kubernetes/ovnkube-node-mbs58" Jul 10 00:26:25.479442 ip-10-0-130-146 kubenswrapper[2570]: I0710 00:26:25.477625 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/5e419a12-f1a1-4b3d-8c72-7c6af574948f-konnectivity-ca\") pod \"konnectivity-agent-995ch\" (UID: \"5e419a12-f1a1-4b3d-8c72-7c6af574948f\") " pod="kube-system/konnectivity-agent-995ch" Jul 10 00:26:25.479442 ip-10-0-130-146 kubenswrapper[2570]: I0710 00:26:25.477642 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/f726a6ab-3c92-403e-ad1b-bdaf7e37c5e3-cnibin\") pod \"multus-2pdkh\" (UID: \"f726a6ab-3c92-403e-ad1b-bdaf7e37c5e3\") " pod="openshift-multus/multus-2pdkh" Jul 10 00:26:25.479442 ip-10-0-130-146 kubenswrapper[2570]: I0710 00:26:25.477732 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/c4ef019a-364a-4933-ab85-de86d86167f1-ovnkube-config\") pod \"ovnkube-node-mbs58\" (UID: \"c4ef019a-364a-4933-ab85-de86d86167f1\") " pod="openshift-ovn-kubernetes/ovnkube-node-mbs58" Jul 10 00:26:25.479442 ip-10-0-130-146 kubenswrapper[2570]: I0710 00:26:25.477713 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/c4ef019a-364a-4933-ab85-de86d86167f1-env-overrides\") pod \"ovnkube-node-mbs58\" (UID: \"c4ef019a-364a-4933-ab85-de86d86167f1\") " pod="openshift-ovn-kubernetes/ovnkube-node-mbs58" Jul 10 00:26:25.479901 ip-10-0-130-146 kubenswrapper[2570]: I0710 00:26:25.477961 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/c4ef019a-364a-4933-ab85-de86d86167f1-ovnkube-script-lib\") pod \"ovnkube-node-mbs58\" (UID: \"c4ef019a-364a-4933-ab85-de86d86167f1\") " pod="openshift-ovn-kubernetes/ovnkube-node-mbs58" Jul 10 00:26:25.479901 ip-10-0-130-146 kubenswrapper[2570]: I0710 00:26:25.477972 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/f726a6ab-3c92-403e-ad1b-bdaf7e37c5e3-multus-daemon-config\") pod \"multus-2pdkh\" (UID: \"f726a6ab-3c92-403e-ad1b-bdaf7e37c5e3\") " pod="openshift-multus/multus-2pdkh" Jul 10 00:26:25.479901 ip-10-0-130-146 kubenswrapper[2570]: I0710 00:26:25.478122 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/5e419a12-f1a1-4b3d-8c72-7c6af574948f-konnectivity-ca\") pod \"konnectivity-agent-995ch\" (UID: \"5e419a12-f1a1-4b3d-8c72-7c6af574948f\") " pod="kube-system/konnectivity-agent-995ch" Jul 10 00:26:25.479901 ip-10-0-130-146 kubenswrapper[2570]: I0710 00:26:25.479155 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/5e419a12-f1a1-4b3d-8c72-7c6af574948f-agent-certs\") pod \"konnectivity-agent-995ch\" (UID: \"5e419a12-f1a1-4b3d-8c72-7c6af574948f\") " pod="kube-system/konnectivity-agent-995ch" Jul 10 00:26:25.479901 ip-10-0-130-146 kubenswrapper[2570]: I0710 00:26:25.479395 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/c4ef019a-364a-4933-ab85-de86d86167f1-ovn-node-metrics-cert\") pod \"ovnkube-node-mbs58\" (UID: \"c4ef019a-364a-4933-ab85-de86d86167f1\") " pod="openshift-ovn-kubernetes/ovnkube-node-mbs58" Jul 10 00:26:25.486699 ip-10-0-130-146 kubenswrapper[2570]: E0710 00:26:25.486685 2570 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 10 00:26:25.486746 ip-10-0-130-146 kubenswrapper[2570]: E0710 00:26:25.486707 2570 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 10 00:26:25.486746 ip-10-0-130-146 kubenswrapper[2570]: E0710 00:26:25.486718 2570 projected.go:196] Error preparing data for projected volume kube-api-access-kl6s9 for pod openshift-network-diagnostics/network-check-target-5hs2b: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 10 00:26:25.486815 ip-10-0-130-146 kubenswrapper[2570]: E0710 00:26:25.486762 2570 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/78717d89-03f2-4824-942a-48892a2bfb99-kube-api-access-kl6s9 podName:78717d89-03f2-4824-942a-48892a2bfb99 nodeName:}" failed. No retries permitted until 2026-07-10 00:26:25.986750767 +0000 UTC m=+5.035129495 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-api-access-kl6s9" (UniqueName: "kubernetes.io/projected/78717d89-03f2-4824-942a-48892a2bfb99-kube-api-access-kl6s9") pod "network-check-target-5hs2b" (UID: "78717d89-03f2-4824-942a-48892a2bfb99") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 10 00:26:25.489259 ip-10-0-130-146 kubenswrapper[2570]: I0710 00:26:25.489243 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5m6sc\" (UniqueName: \"kubernetes.io/projected/c4ef019a-364a-4933-ab85-de86d86167f1-kube-api-access-5m6sc\") pod \"ovnkube-node-mbs58\" (UID: \"c4ef019a-364a-4933-ab85-de86d86167f1\") " pod="openshift-ovn-kubernetes/ovnkube-node-mbs58" Jul 10 00:26:25.489458 ip-10-0-130-146 kubenswrapper[2570]: I0710 00:26:25.489442 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-gw9kb\" (UniqueName: \"kubernetes.io/projected/f726a6ab-3c92-403e-ad1b-bdaf7e37c5e3-kube-api-access-gw9kb\") pod \"multus-2pdkh\" (UID: \"f726a6ab-3c92-403e-ad1b-bdaf7e37c5e3\") " pod="openshift-multus/multus-2pdkh" Jul 10 00:26:25.489634 ip-10-0-130-146 kubenswrapper[2570]: I0710 00:26:25.489618 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-sst8k\" (UniqueName: \"kubernetes.io/projected/c44f0289-9f01-4548-ab98-74a17ec2b150-kube-api-access-sst8k\") pod \"aws-ebs-csi-driver-node-68tgb\" (UID: \"c44f0289-9f01-4548-ab98-74a17ec2b150\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-68tgb" Jul 10 00:26:25.530278 ip-10-0-130-146 kubenswrapper[2570]: I0710 00:26:25.530239 2570 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/kube-apiserver-proxy-ip-10-0-130-146.ec2.internal" podStartSLOduration=0.530227705 podStartE2EDuration="530.227705ms" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-10 00:26:25 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 00:26:25.530108969 +0000 UTC m=+4.578487691" watchObservedRunningTime="2026-07-10 00:26:25.530227705 +0000 UTC m=+4.578606426" Jul 10 00:26:25.546249 ip-10-0-130-146 kubenswrapper[2570]: I0710 00:26:25.546230 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-bw9kx" Jul 10 00:26:25.552484 ip-10-0-130-146 kubenswrapper[2570]: I0710 00:26:25.552468 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-7qfbg" Jul 10 00:26:25.556274 ip-10-0-130-146 kubenswrapper[2570]: W0710 00:26:25.556258 2570 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5dcf6a9a_85e6_4f25_85b3_7b8a9eaa8654.slice/crio-7906e138aee2d2d8ba1933f47d030491ca4478994cda330682df6b62090d247a WatchSource:0}: Error finding container 7906e138aee2d2d8ba1933f47d030491ca4478994cda330682df6b62090d247a: Status 404 returned error can't find the container with id 7906e138aee2d2d8ba1933f47d030491ca4478994cda330682df6b62090d247a Jul 10 00:26:25.560308 ip-10-0-130-146 kubenswrapper[2570]: I0710 00:26:25.560290 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-7hmqd" Jul 10 00:26:25.560962 ip-10-0-130-146 kubenswrapper[2570]: W0710 00:26:25.560943 2570 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod78f9f66c_5c2f_4cee_bf34_39961a78dbcc.slice/crio-4d0975739fb9db56815af49d3c74dd7fb5718d0fce1b25c1f72c403e8d024e76 WatchSource:0}: Error finding container 4d0975739fb9db56815af49d3c74dd7fb5718d0fce1b25c1f72c403e8d024e76: Status 404 returned error can't find the container with id 4d0975739fb9db56815af49d3c74dd7fb5718d0fce1b25c1f72c403e8d024e76 Jul 10 00:26:25.567084 ip-10-0-130-146 kubenswrapper[2570]: I0710 00:26:25.567062 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-68tgb" Jul 10 00:26:25.568788 ip-10-0-130-146 kubenswrapper[2570]: W0710 00:26:25.568766 2570 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod173d0656_f60a_40da_8ff5_2e92a0ca8a22.slice/crio-9c6a6d17d327562b94b4d9b05715a85d49b0c8ae3d58eb20b471759969d2a506 WatchSource:0}: Error finding container 9c6a6d17d327562b94b4d9b05715a85d49b0c8ae3d58eb20b471759969d2a506: Status 404 returned error can't find the container with id 9c6a6d17d327562b94b4d9b05715a85d49b0c8ae3d58eb20b471759969d2a506 Jul 10 00:26:25.572246 ip-10-0-130-146 kubenswrapper[2570]: I0710 00:26:25.572231 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-7tn6x" Jul 10 00:26:25.575556 ip-10-0-130-146 kubenswrapper[2570]: W0710 00:26:25.575537 2570 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc44f0289_9f01_4548_ab98_74a17ec2b150.slice/crio-fc907ceb1feab5dad5951510e30150b638d74e7438a43d5a6e8d98bc062b62e2 WatchSource:0}: Error finding container fc907ceb1feab5dad5951510e30150b638d74e7438a43d5a6e8d98bc062b62e2: Status 404 returned error can't find the container with id fc907ceb1feab5dad5951510e30150b638d74e7438a43d5a6e8d98bc062b62e2 Jul 10 00:26:25.577095 ip-10-0-130-146 kubenswrapper[2570]: I0710 00:26:25.577072 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-t526b" Jul 10 00:26:25.579805 ip-10-0-130-146 kubenswrapper[2570]: W0710 00:26:25.579783 2570 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod006ba44e_264c_42cd_9891_ad8d769386c8.slice/crio-39e75f9dfcd83c8ff5f4bc5b5e2df33465207c0bc3cad4df84c032c3c26f6aac WatchSource:0}: Error finding container 39e75f9dfcd83c8ff5f4bc5b5e2df33465207c0bc3cad4df84c032c3c26f6aac: Status 404 returned error can't find the container with id 39e75f9dfcd83c8ff5f4bc5b5e2df33465207c0bc3cad4df84c032c3c26f6aac Jul 10 00:26:25.581796 ip-10-0-130-146 kubenswrapper[2570]: I0710 00:26:25.581783 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-995ch" Jul 10 00:26:25.584741 ip-10-0-130-146 kubenswrapper[2570]: W0710 00:26:25.584717 2570 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod52179d08_5861_4a23_9c8d_653c13bed339.slice/crio-6d2daf9a6ef58499347cacc05366a360f07b9f67445183881cc6a18a333a5c25 WatchSource:0}: Error finding container 6d2daf9a6ef58499347cacc05366a360f07b9f67445183881cc6a18a333a5c25: Status 404 returned error can't find the container with id 6d2daf9a6ef58499347cacc05366a360f07b9f67445183881cc6a18a333a5c25 Jul 10 00:26:25.586562 ip-10-0-130-146 kubenswrapper[2570]: I0710 00:26:25.586503 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-mbs58" Jul 10 00:26:25.590251 ip-10-0-130-146 kubenswrapper[2570]: I0710 00:26:25.590219 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-2pdkh" Jul 10 00:26:25.591468 ip-10-0-130-146 kubenswrapper[2570]: W0710 00:26:25.591442 2570 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5e419a12_f1a1_4b3d_8c72_7c6af574948f.slice/crio-6e409a4143d20c76cf6a3a8107af7d2f971c966f11f895cf30fa8ee52013c56f WatchSource:0}: Error finding container 6e409a4143d20c76cf6a3a8107af7d2f971c966f11f895cf30fa8ee52013c56f: Status 404 returned error can't find the container with id 6e409a4143d20c76cf6a3a8107af7d2f971c966f11f895cf30fa8ee52013c56f Jul 10 00:26:25.597808 ip-10-0-130-146 kubenswrapper[2570]: W0710 00:26:25.597791 2570 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc4ef019a_364a_4933_ab85_de86d86167f1.slice/crio-a7967f600ed527059b1f7b08d3fa6b2b7bc96d8813b07a2ff58cec3cf0c41774 WatchSource:0}: Error finding container a7967f600ed527059b1f7b08d3fa6b2b7bc96d8813b07a2ff58cec3cf0c41774: Status 404 returned error can't find the container with id a7967f600ed527059b1f7b08d3fa6b2b7bc96d8813b07a2ff58cec3cf0c41774 Jul 10 00:26:25.600222 ip-10-0-130-146 kubenswrapper[2570]: W0710 00:26:25.600202 2570 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf726a6ab_3c92_403e_ad1b_bdaf7e37c5e3.slice/crio-e222248d3937ee386be4e520d3bbcab5a84d1416bb2d764377f07b9edefebef5 WatchSource:0}: Error finding container e222248d3937ee386be4e520d3bbcab5a84d1416bb2d764377f07b9edefebef5: Status 404 returned error can't find the container with id e222248d3937ee386be4e520d3bbcab5a84d1416bb2d764377f07b9edefebef5 Jul 10 00:26:25.879891 ip-10-0-130-146 kubenswrapper[2570]: I0710 00:26:25.879863 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b7aab806-00a2-491e-a997-ed3174bd547a-metrics-certs\") pod \"network-metrics-daemon-d8t56\" (UID: \"b7aab806-00a2-491e-a997-ed3174bd547a\") " pod="openshift-multus/network-metrics-daemon-d8t56" Jul 10 00:26:25.879995 ip-10-0-130-146 kubenswrapper[2570]: E0710 00:26:25.879983 2570 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 10 00:26:25.880041 ip-10-0-130-146 kubenswrapper[2570]: E0710 00:26:25.880036 2570 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b7aab806-00a2-491e-a997-ed3174bd547a-metrics-certs podName:b7aab806-00a2-491e-a997-ed3174bd547a nodeName:}" failed. No retries permitted until 2026-07-10 00:26:26.88002058 +0000 UTC m=+5.928399292 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/b7aab806-00a2-491e-a997-ed3174bd547a-metrics-certs") pod "network-metrics-daemon-d8t56" (UID: "b7aab806-00a2-491e-a997-ed3174bd547a") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 10 00:26:25.981062 ip-10-0-130-146 kubenswrapper[2570]: I0710 00:26:25.981016 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/5667efb4-6019-4bde-827a-0f1e68096b93-original-pull-secret\") pod \"global-pull-secret-syncer-6bqvb\" (UID: \"5667efb4-6019-4bde-827a-0f1e68096b93\") " pod="kube-system/global-pull-secret-syncer-6bqvb" Jul 10 00:26:25.981238 ip-10-0-130-146 kubenswrapper[2570]: E0710 00:26:25.981215 2570 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 10 00:26:25.981312 ip-10-0-130-146 kubenswrapper[2570]: E0710 00:26:25.981296 2570 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5667efb4-6019-4bde-827a-0f1e68096b93-original-pull-secret podName:5667efb4-6019-4bde-827a-0f1e68096b93 nodeName:}" failed. No retries permitted until 2026-07-10 00:26:26.981276594 +0000 UTC m=+6.029655299 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/5667efb4-6019-4bde-827a-0f1e68096b93-original-pull-secret") pod "global-pull-secret-syncer-6bqvb" (UID: "5667efb4-6019-4bde-827a-0f1e68096b93") : object "kube-system"/"original-pull-secret" not registered Jul 10 00:26:26.081467 ip-10-0-130-146 kubenswrapper[2570]: I0710 00:26:26.081434 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kl6s9\" (UniqueName: \"kubernetes.io/projected/78717d89-03f2-4824-942a-48892a2bfb99-kube-api-access-kl6s9\") pod \"network-check-target-5hs2b\" (UID: \"78717d89-03f2-4824-942a-48892a2bfb99\") " pod="openshift-network-diagnostics/network-check-target-5hs2b" Jul 10 00:26:26.081634 ip-10-0-130-146 kubenswrapper[2570]: E0710 00:26:26.081624 2570 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 10 00:26:26.081701 ip-10-0-130-146 kubenswrapper[2570]: E0710 00:26:26.081644 2570 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 10 00:26:26.081701 ip-10-0-130-146 kubenswrapper[2570]: E0710 00:26:26.081657 2570 projected.go:196] Error preparing data for projected volume kube-api-access-kl6s9 for pod openshift-network-diagnostics/network-check-target-5hs2b: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 10 00:26:26.081803 ip-10-0-130-146 kubenswrapper[2570]: E0710 00:26:26.081716 2570 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/78717d89-03f2-4824-942a-48892a2bfb99-kube-api-access-kl6s9 podName:78717d89-03f2-4824-942a-48892a2bfb99 nodeName:}" failed. No retries permitted until 2026-07-10 00:26:27.081698938 +0000 UTC m=+6.130077655 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-api-access-kl6s9" (UniqueName: "kubernetes.io/projected/78717d89-03f2-4824-942a-48892a2bfb99-kube-api-access-kl6s9") pod "network-check-target-5hs2b" (UID: "78717d89-03f2-4824-942a-48892a2bfb99") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 10 00:26:26.302369 ip-10-0-130-146 kubenswrapper[2570]: I0710 00:26:26.302274 2570 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-07-09 00:21:25 +0000 UTC" deadline="2028-01-30 09:15:47.357042054 +0000 UTC" Jul 10 00:26:26.302369 ip-10-0-130-146 kubenswrapper[2570]: I0710 00:26:26.302321 2570 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="13664h49m21.054725658s" Jul 10 00:26:26.448396 ip-10-0-130-146 kubenswrapper[2570]: I0710 00:26:26.447853 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-d8t56" Jul 10 00:26:26.448396 ip-10-0-130-146 kubenswrapper[2570]: I0710 00:26:26.447895 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-5hs2b" Jul 10 00:26:26.448396 ip-10-0-130-146 kubenswrapper[2570]: E0710 00:26:26.448005 2570 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-d8t56" podUID="b7aab806-00a2-491e-a997-ed3174bd547a" Jul 10 00:26:26.448396 ip-10-0-130-146 kubenswrapper[2570]: E0710 00:26:26.448155 2570 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-5hs2b" podUID="78717d89-03f2-4824-942a-48892a2bfb99" Jul 10 00:26:26.474426 ip-10-0-130-146 kubenswrapper[2570]: I0710 00:26:26.474393 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-mbs58" event={"ID":"c4ef019a-364a-4933-ab85-de86d86167f1","Type":"ContainerStarted","Data":"a7967f600ed527059b1f7b08d3fa6b2b7bc96d8813b07a2ff58cec3cf0c41774"} Jul 10 00:26:26.485124 ip-10-0-130-146 kubenswrapper[2570]: I0710 00:26:26.485091 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-995ch" event={"ID":"5e419a12-f1a1-4b3d-8c72-7c6af574948f","Type":"ContainerStarted","Data":"6e409a4143d20c76cf6a3a8107af7d2f971c966f11f895cf30fa8ee52013c56f"} Jul 10 00:26:26.499446 ip-10-0-130-146 kubenswrapper[2570]: I0710 00:26:26.499404 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-7qfbg" event={"ID":"78f9f66c-5c2f-4cee-bf34-39961a78dbcc","Type":"ContainerStarted","Data":"4d0975739fb9db56815af49d3c74dd7fb5718d0fce1b25c1f72c403e8d024e76"} Jul 10 00:26:26.508070 ip-10-0-130-146 kubenswrapper[2570]: I0710 00:26:26.507532 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-bw9kx" event={"ID":"5dcf6a9a-85e6-4f25-85b3-7b8a9eaa8654","Type":"ContainerStarted","Data":"7906e138aee2d2d8ba1933f47d030491ca4478994cda330682df6b62090d247a"} Jul 10 00:26:26.520193 ip-10-0-130-146 kubenswrapper[2570]: I0710 00:26:26.519790 2570 generic.go:356] "Generic (PLEG): container finished" podID="0ddb9c9bc924902db48bbad9696eddc0" containerID="58668167646ec43b1e79a568487892f07a3a1007a1eb4579dd96a270e27a737b" exitCode=0 Jul 10 00:26:26.520193 ip-10-0-130-146 kubenswrapper[2570]: I0710 00:26:26.519972 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-146.ec2.internal" event={"ID":"0ddb9c9bc924902db48bbad9696eddc0","Type":"ContainerDied","Data":"58668167646ec43b1e79a568487892f07a3a1007a1eb4579dd96a270e27a737b"} Jul 10 00:26:26.533073 ip-10-0-130-146 kubenswrapper[2570]: I0710 00:26:26.532984 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-2pdkh" event={"ID":"f726a6ab-3c92-403e-ad1b-bdaf7e37c5e3","Type":"ContainerStarted","Data":"e222248d3937ee386be4e520d3bbcab5a84d1416bb2d764377f07b9edefebef5"} Jul 10 00:26:26.564976 ip-10-0-130-146 kubenswrapper[2570]: I0710 00:26:26.564906 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-t526b" event={"ID":"52179d08-5861-4a23-9c8d-653c13bed339","Type":"ContainerStarted","Data":"6d2daf9a6ef58499347cacc05366a360f07b9f67445183881cc6a18a333a5c25"} Jul 10 00:26:26.574306 ip-10-0-130-146 kubenswrapper[2570]: I0710 00:26:26.574276 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-7tn6x" event={"ID":"006ba44e-264c-42cd-9891-ad8d769386c8","Type":"ContainerStarted","Data":"39e75f9dfcd83c8ff5f4bc5b5e2df33465207c0bc3cad4df84c032c3c26f6aac"} Jul 10 00:26:26.592211 ip-10-0-130-146 kubenswrapper[2570]: I0710 00:26:26.592125 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-68tgb" event={"ID":"c44f0289-9f01-4548-ab98-74a17ec2b150","Type":"ContainerStarted","Data":"fc907ceb1feab5dad5951510e30150b638d74e7438a43d5a6e8d98bc062b62e2"} Jul 10 00:26:26.595358 ip-10-0-130-146 kubenswrapper[2570]: I0710 00:26:26.595308 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-7hmqd" event={"ID":"173d0656-f60a-40da-8ff5-2e92a0ca8a22","Type":"ContainerStarted","Data":"9c6a6d17d327562b94b4d9b05715a85d49b0c8ae3d58eb20b471759969d2a506"} Jul 10 00:26:26.889543 ip-10-0-130-146 kubenswrapper[2570]: I0710 00:26:26.889291 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b7aab806-00a2-491e-a997-ed3174bd547a-metrics-certs\") pod \"network-metrics-daemon-d8t56\" (UID: \"b7aab806-00a2-491e-a997-ed3174bd547a\") " pod="openshift-multus/network-metrics-daemon-d8t56" Jul 10 00:26:26.889729 ip-10-0-130-146 kubenswrapper[2570]: E0710 00:26:26.889579 2570 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 10 00:26:26.889729 ip-10-0-130-146 kubenswrapper[2570]: E0710 00:26:26.889659 2570 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b7aab806-00a2-491e-a997-ed3174bd547a-metrics-certs podName:b7aab806-00a2-491e-a997-ed3174bd547a nodeName:}" failed. No retries permitted until 2026-07-10 00:26:28.889638098 +0000 UTC m=+7.938016814 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/b7aab806-00a2-491e-a997-ed3174bd547a-metrics-certs") pod "network-metrics-daemon-d8t56" (UID: "b7aab806-00a2-491e-a997-ed3174bd547a") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 10 00:26:26.990897 ip-10-0-130-146 kubenswrapper[2570]: I0710 00:26:26.990845 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/5667efb4-6019-4bde-827a-0f1e68096b93-original-pull-secret\") pod \"global-pull-secret-syncer-6bqvb\" (UID: \"5667efb4-6019-4bde-827a-0f1e68096b93\") " pod="kube-system/global-pull-secret-syncer-6bqvb" Jul 10 00:26:26.991102 ip-10-0-130-146 kubenswrapper[2570]: E0710 00:26:26.991012 2570 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 10 00:26:26.991102 ip-10-0-130-146 kubenswrapper[2570]: E0710 00:26:26.991095 2570 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5667efb4-6019-4bde-827a-0f1e68096b93-original-pull-secret podName:5667efb4-6019-4bde-827a-0f1e68096b93 nodeName:}" failed. No retries permitted until 2026-07-10 00:26:28.991074613 +0000 UTC m=+8.039453317 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/5667efb4-6019-4bde-827a-0f1e68096b93-original-pull-secret") pod "global-pull-secret-syncer-6bqvb" (UID: "5667efb4-6019-4bde-827a-0f1e68096b93") : object "kube-system"/"original-pull-secret" not registered Jul 10 00:26:27.091650 ip-10-0-130-146 kubenswrapper[2570]: I0710 00:26:27.091613 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kl6s9\" (UniqueName: \"kubernetes.io/projected/78717d89-03f2-4824-942a-48892a2bfb99-kube-api-access-kl6s9\") pod \"network-check-target-5hs2b\" (UID: \"78717d89-03f2-4824-942a-48892a2bfb99\") " pod="openshift-network-diagnostics/network-check-target-5hs2b" Jul 10 00:26:27.091839 ip-10-0-130-146 kubenswrapper[2570]: E0710 00:26:27.091820 2570 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 10 00:26:27.091904 ip-10-0-130-146 kubenswrapper[2570]: E0710 00:26:27.091851 2570 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 10 00:26:27.091904 ip-10-0-130-146 kubenswrapper[2570]: E0710 00:26:27.091864 2570 projected.go:196] Error preparing data for projected volume kube-api-access-kl6s9 for pod openshift-network-diagnostics/network-check-target-5hs2b: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 10 00:26:27.091994 ip-10-0-130-146 kubenswrapper[2570]: E0710 00:26:27.091928 2570 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/78717d89-03f2-4824-942a-48892a2bfb99-kube-api-access-kl6s9 podName:78717d89-03f2-4824-942a-48892a2bfb99 nodeName:}" failed. No retries permitted until 2026-07-10 00:26:29.091908569 +0000 UTC m=+8.140287271 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-api-access-kl6s9" (UniqueName: "kubernetes.io/projected/78717d89-03f2-4824-942a-48892a2bfb99-kube-api-access-kl6s9") pod "network-check-target-5hs2b" (UID: "78717d89-03f2-4824-942a-48892a2bfb99") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 10 00:26:27.303444 ip-10-0-130-146 kubenswrapper[2570]: I0710 00:26:27.303324 2570 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-07-09 00:21:25 +0000 UTC" deadline="2027-12-30 07:36:26.6661045 +0000 UTC" Jul 10 00:26:27.303444 ip-10-0-130-146 kubenswrapper[2570]: I0710 00:26:27.303384 2570 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="12919h9m59.362725093s" Jul 10 00:26:27.450241 ip-10-0-130-146 kubenswrapper[2570]: I0710 00:26:27.450201 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-6bqvb" Jul 10 00:26:27.450397 ip-10-0-130-146 kubenswrapper[2570]: E0710 00:26:27.450324 2570 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-6bqvb" podUID="5667efb4-6019-4bde-827a-0f1e68096b93" Jul 10 00:26:27.614548 ip-10-0-130-146 kubenswrapper[2570]: I0710 00:26:27.614468 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-146.ec2.internal" event={"ID":"0ddb9c9bc924902db48bbad9696eddc0","Type":"ContainerStarted","Data":"04ed44bb165455213f64650506a3fba5baa5cac8f076597ddd03434705e2fe62"} Jul 10 00:26:28.448643 ip-10-0-130-146 kubenswrapper[2570]: I0710 00:26:28.448606 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-5hs2b" Jul 10 00:26:28.449129 ip-10-0-130-146 kubenswrapper[2570]: I0710 00:26:28.448629 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-d8t56" Jul 10 00:26:28.449129 ip-10-0-130-146 kubenswrapper[2570]: E0710 00:26:28.448743 2570 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-5hs2b" podUID="78717d89-03f2-4824-942a-48892a2bfb99" Jul 10 00:26:28.449129 ip-10-0-130-146 kubenswrapper[2570]: E0710 00:26:28.448779 2570 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-d8t56" podUID="b7aab806-00a2-491e-a997-ed3174bd547a" Jul 10 00:26:28.909069 ip-10-0-130-146 kubenswrapper[2570]: I0710 00:26:28.909006 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b7aab806-00a2-491e-a997-ed3174bd547a-metrics-certs\") pod \"network-metrics-daemon-d8t56\" (UID: \"b7aab806-00a2-491e-a997-ed3174bd547a\") " pod="openshift-multus/network-metrics-daemon-d8t56" Jul 10 00:26:28.909249 ip-10-0-130-146 kubenswrapper[2570]: E0710 00:26:28.909142 2570 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 10 00:26:28.909249 ip-10-0-130-146 kubenswrapper[2570]: E0710 00:26:28.909212 2570 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b7aab806-00a2-491e-a997-ed3174bd547a-metrics-certs podName:b7aab806-00a2-491e-a997-ed3174bd547a nodeName:}" failed. No retries permitted until 2026-07-10 00:26:32.909192693 +0000 UTC m=+11.957571394 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/b7aab806-00a2-491e-a997-ed3174bd547a-metrics-certs") pod "network-metrics-daemon-d8t56" (UID: "b7aab806-00a2-491e-a997-ed3174bd547a") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 10 00:26:29.009402 ip-10-0-130-146 kubenswrapper[2570]: I0710 00:26:29.009343 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/5667efb4-6019-4bde-827a-0f1e68096b93-original-pull-secret\") pod \"global-pull-secret-syncer-6bqvb\" (UID: \"5667efb4-6019-4bde-827a-0f1e68096b93\") " pod="kube-system/global-pull-secret-syncer-6bqvb" Jul 10 00:26:29.009569 ip-10-0-130-146 kubenswrapper[2570]: E0710 00:26:29.009550 2570 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 10 00:26:29.009640 ip-10-0-130-146 kubenswrapper[2570]: E0710 00:26:29.009623 2570 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5667efb4-6019-4bde-827a-0f1e68096b93-original-pull-secret podName:5667efb4-6019-4bde-827a-0f1e68096b93 nodeName:}" failed. No retries permitted until 2026-07-10 00:26:33.009603501 +0000 UTC m=+12.057982214 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/5667efb4-6019-4bde-827a-0f1e68096b93-original-pull-secret") pod "global-pull-secret-syncer-6bqvb" (UID: "5667efb4-6019-4bde-827a-0f1e68096b93") : object "kube-system"/"original-pull-secret" not registered Jul 10 00:26:29.110837 ip-10-0-130-146 kubenswrapper[2570]: I0710 00:26:29.109840 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kl6s9\" (UniqueName: \"kubernetes.io/projected/78717d89-03f2-4824-942a-48892a2bfb99-kube-api-access-kl6s9\") pod \"network-check-target-5hs2b\" (UID: \"78717d89-03f2-4824-942a-48892a2bfb99\") " pod="openshift-network-diagnostics/network-check-target-5hs2b" Jul 10 00:26:29.110837 ip-10-0-130-146 kubenswrapper[2570]: E0710 00:26:29.109999 2570 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 10 00:26:29.110837 ip-10-0-130-146 kubenswrapper[2570]: E0710 00:26:29.110016 2570 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 10 00:26:29.110837 ip-10-0-130-146 kubenswrapper[2570]: E0710 00:26:29.110030 2570 projected.go:196] Error preparing data for projected volume kube-api-access-kl6s9 for pod openshift-network-diagnostics/network-check-target-5hs2b: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 10 00:26:29.110837 ip-10-0-130-146 kubenswrapper[2570]: E0710 00:26:29.110115 2570 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/78717d89-03f2-4824-942a-48892a2bfb99-kube-api-access-kl6s9 podName:78717d89-03f2-4824-942a-48892a2bfb99 nodeName:}" failed. No retries permitted until 2026-07-10 00:26:33.110095234 +0000 UTC m=+12.158473950 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-api-access-kl6s9" (UniqueName: "kubernetes.io/projected/78717d89-03f2-4824-942a-48892a2bfb99-kube-api-access-kl6s9") pod "network-check-target-5hs2b" (UID: "78717d89-03f2-4824-942a-48892a2bfb99") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 10 00:26:29.449181 ip-10-0-130-146 kubenswrapper[2570]: I0710 00:26:29.449151 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-6bqvb" Jul 10 00:26:29.449621 ip-10-0-130-146 kubenswrapper[2570]: E0710 00:26:29.449289 2570 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-6bqvb" podUID="5667efb4-6019-4bde-827a-0f1e68096b93" Jul 10 00:26:30.448102 ip-10-0-130-146 kubenswrapper[2570]: I0710 00:26:30.448068 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-d8t56" Jul 10 00:26:30.448280 ip-10-0-130-146 kubenswrapper[2570]: I0710 00:26:30.448121 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-5hs2b" Jul 10 00:26:30.448280 ip-10-0-130-146 kubenswrapper[2570]: E0710 00:26:30.448207 2570 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-d8t56" podUID="b7aab806-00a2-491e-a997-ed3174bd547a" Jul 10 00:26:30.448393 ip-10-0-130-146 kubenswrapper[2570]: E0710 00:26:30.448340 2570 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-5hs2b" podUID="78717d89-03f2-4824-942a-48892a2bfb99" Jul 10 00:26:31.449690 ip-10-0-130-146 kubenswrapper[2570]: I0710 00:26:31.449654 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-6bqvb" Jul 10 00:26:31.450190 ip-10-0-130-146 kubenswrapper[2570]: E0710 00:26:31.449775 2570 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-6bqvb" podUID="5667efb4-6019-4bde-827a-0f1e68096b93" Jul 10 00:26:32.448538 ip-10-0-130-146 kubenswrapper[2570]: I0710 00:26:32.448507 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-5hs2b" Jul 10 00:26:32.448711 ip-10-0-130-146 kubenswrapper[2570]: E0710 00:26:32.448637 2570 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-5hs2b" podUID="78717d89-03f2-4824-942a-48892a2bfb99" Jul 10 00:26:32.449011 ip-10-0-130-146 kubenswrapper[2570]: I0710 00:26:32.448992 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-d8t56" Jul 10 00:26:32.449145 ip-10-0-130-146 kubenswrapper[2570]: E0710 00:26:32.449119 2570 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-d8t56" podUID="b7aab806-00a2-491e-a997-ed3174bd547a" Jul 10 00:26:32.940455 ip-10-0-130-146 kubenswrapper[2570]: I0710 00:26:32.940361 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b7aab806-00a2-491e-a997-ed3174bd547a-metrics-certs\") pod \"network-metrics-daemon-d8t56\" (UID: \"b7aab806-00a2-491e-a997-ed3174bd547a\") " pod="openshift-multus/network-metrics-daemon-d8t56" Jul 10 00:26:32.940920 ip-10-0-130-146 kubenswrapper[2570]: E0710 00:26:32.940635 2570 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 10 00:26:32.940920 ip-10-0-130-146 kubenswrapper[2570]: E0710 00:26:32.940745 2570 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b7aab806-00a2-491e-a997-ed3174bd547a-metrics-certs podName:b7aab806-00a2-491e-a997-ed3174bd547a nodeName:}" failed. No retries permitted until 2026-07-10 00:26:40.940695763 +0000 UTC m=+19.989074464 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/b7aab806-00a2-491e-a997-ed3174bd547a-metrics-certs") pod "network-metrics-daemon-d8t56" (UID: "b7aab806-00a2-491e-a997-ed3174bd547a") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 10 00:26:33.040771 ip-10-0-130-146 kubenswrapper[2570]: I0710 00:26:33.040733 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/5667efb4-6019-4bde-827a-0f1e68096b93-original-pull-secret\") pod \"global-pull-secret-syncer-6bqvb\" (UID: \"5667efb4-6019-4bde-827a-0f1e68096b93\") " pod="kube-system/global-pull-secret-syncer-6bqvb" Jul 10 00:26:33.040958 ip-10-0-130-146 kubenswrapper[2570]: E0710 00:26:33.040913 2570 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 10 00:26:33.041015 ip-10-0-130-146 kubenswrapper[2570]: E0710 00:26:33.040989 2570 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5667efb4-6019-4bde-827a-0f1e68096b93-original-pull-secret podName:5667efb4-6019-4bde-827a-0f1e68096b93 nodeName:}" failed. No retries permitted until 2026-07-10 00:26:41.04096529 +0000 UTC m=+20.089344007 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/5667efb4-6019-4bde-827a-0f1e68096b93-original-pull-secret") pod "global-pull-secret-syncer-6bqvb" (UID: "5667efb4-6019-4bde-827a-0f1e68096b93") : object "kube-system"/"original-pull-secret" not registered Jul 10 00:26:33.141881 ip-10-0-130-146 kubenswrapper[2570]: I0710 00:26:33.141849 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kl6s9\" (UniqueName: \"kubernetes.io/projected/78717d89-03f2-4824-942a-48892a2bfb99-kube-api-access-kl6s9\") pod \"network-check-target-5hs2b\" (UID: \"78717d89-03f2-4824-942a-48892a2bfb99\") " pod="openshift-network-diagnostics/network-check-target-5hs2b" Jul 10 00:26:33.142081 ip-10-0-130-146 kubenswrapper[2570]: E0710 00:26:33.142058 2570 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 10 00:26:33.142133 ip-10-0-130-146 kubenswrapper[2570]: E0710 00:26:33.142090 2570 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 10 00:26:33.142133 ip-10-0-130-146 kubenswrapper[2570]: E0710 00:26:33.142104 2570 projected.go:196] Error preparing data for projected volume kube-api-access-kl6s9 for pod openshift-network-diagnostics/network-check-target-5hs2b: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 10 00:26:33.142200 ip-10-0-130-146 kubenswrapper[2570]: E0710 00:26:33.142168 2570 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/78717d89-03f2-4824-942a-48892a2bfb99-kube-api-access-kl6s9 podName:78717d89-03f2-4824-942a-48892a2bfb99 nodeName:}" failed. No retries permitted until 2026-07-10 00:26:41.142149511 +0000 UTC m=+20.190528218 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-api-access-kl6s9" (UniqueName: "kubernetes.io/projected/78717d89-03f2-4824-942a-48892a2bfb99-kube-api-access-kl6s9") pod "network-check-target-5hs2b" (UID: "78717d89-03f2-4824-942a-48892a2bfb99") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 10 00:26:33.448832 ip-10-0-130-146 kubenswrapper[2570]: I0710 00:26:33.448353 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-6bqvb" Jul 10 00:26:33.448832 ip-10-0-130-146 kubenswrapper[2570]: E0710 00:26:33.448489 2570 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-6bqvb" podUID="5667efb4-6019-4bde-827a-0f1e68096b93" Jul 10 00:26:34.448343 ip-10-0-130-146 kubenswrapper[2570]: I0710 00:26:34.448308 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-5hs2b" Jul 10 00:26:34.448343 ip-10-0-130-146 kubenswrapper[2570]: I0710 00:26:34.448330 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-d8t56" Jul 10 00:26:34.448722 ip-10-0-130-146 kubenswrapper[2570]: E0710 00:26:34.448431 2570 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-d8t56" podUID="b7aab806-00a2-491e-a997-ed3174bd547a" Jul 10 00:26:34.448722 ip-10-0-130-146 kubenswrapper[2570]: E0710 00:26:34.448538 2570 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-5hs2b" podUID="78717d89-03f2-4824-942a-48892a2bfb99" Jul 10 00:26:35.448712 ip-10-0-130-146 kubenswrapper[2570]: I0710 00:26:35.448691 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-6bqvb" Jul 10 00:26:35.449002 ip-10-0-130-146 kubenswrapper[2570]: E0710 00:26:35.448799 2570 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-6bqvb" podUID="5667efb4-6019-4bde-827a-0f1e68096b93" Jul 10 00:26:35.640467 ip-10-0-130-146 kubenswrapper[2570]: I0710 00:26:35.640437 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-7hmqd" event={"ID":"173d0656-f60a-40da-8ff5-2e92a0ca8a22","Type":"ContainerStarted","Data":"867de9b3e029498292bf60518a55b4c06700ea721f60f2abbb1f707e6c833155"} Jul 10 00:26:35.642840 ip-10-0-130-146 kubenswrapper[2570]: I0710 00:26:35.642496 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-995ch" event={"ID":"5e419a12-f1a1-4b3d-8c72-7c6af574948f","Type":"ContainerStarted","Data":"e38a0a54248d0e3561950d65878fef96e0e88a14b8ccef94f5dcfa3f31ff6a6a"} Jul 10 00:26:35.657552 ip-10-0-130-146 kubenswrapper[2570]: I0710 00:26:35.657500 2570 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-146.ec2.internal" podStartSLOduration=10.65748073 podStartE2EDuration="10.65748073s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-10 00:26:25 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 00:26:27.62997662 +0000 UTC m=+6.678355342" watchObservedRunningTime="2026-07-10 00:26:35.65748073 +0000 UTC m=+14.705859452" Jul 10 00:26:35.658212 ip-10-0-130-146 kubenswrapper[2570]: I0710 00:26:35.658171 2570 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-node-tuning-operator/tuned-7hmqd" podStartSLOduration=4.736914833 podStartE2EDuration="14.658154149s" totalImagesPullingTime="9.921239316s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-10 00:26:21 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 00:26:35.657110238 +0000 UTC m=+14.705488962" watchObservedRunningTime="2026-07-10 00:26:35.658154149 +0000 UTC m=+14.706532870" Jul 10 00:26:35.711747 ip-10-0-130-146 kubenswrapper[2570]: I0710 00:26:35.711689 2570 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/konnectivity-agent-995ch" podStartSLOduration=4.880175321 podStartE2EDuration="14.711670016s" totalImagesPullingTime="9.831494695s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-10 00:26:21 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 00:26:35.711605913 +0000 UTC m=+14.759984636" watchObservedRunningTime="2026-07-10 00:26:35.711670016 +0000 UTC m=+14.760048738" Jul 10 00:26:35.712548 ip-10-0-130-146 kubenswrapper[2570]: I0710 00:26:35.712509 2570 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/node-ca-t526b" podStartSLOduration=4.868351337 podStartE2EDuration="14.712500765s" totalImagesPullingTime="9.844149428s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-10 00:26:21 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 00:26:35.697915129 +0000 UTC m=+14.746293874" watchObservedRunningTime="2026-07-10 00:26:35.712500765 +0000 UTC m=+14.760879488" Jul 10 00:26:36.448405 ip-10-0-130-146 kubenswrapper[2570]: I0710 00:26:36.448329 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-5hs2b" Jul 10 00:26:36.448548 ip-10-0-130-146 kubenswrapper[2570]: E0710 00:26:36.448447 2570 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-5hs2b" podUID="78717d89-03f2-4824-942a-48892a2bfb99" Jul 10 00:26:36.448732 ip-10-0-130-146 kubenswrapper[2570]: I0710 00:26:36.448678 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-d8t56" Jul 10 00:26:36.449543 ip-10-0-130-146 kubenswrapper[2570]: E0710 00:26:36.448799 2570 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-d8t56" podUID="b7aab806-00a2-491e-a997-ed3174bd547a" Jul 10 00:26:36.651613 ip-10-0-130-146 kubenswrapper[2570]: I0710 00:26:36.651576 2570 generic.go:356] "Generic (PLEG): container finished" podID="78f9f66c-5c2f-4cee-bf34-39961a78dbcc" containerID="c06ebbc756b6ee80a2bc704ba9100a2f8b6e12c9a4a8306a3cb96fd9f01edb6f" exitCode=0 Jul 10 00:26:36.651777 ip-10-0-130-146 kubenswrapper[2570]: I0710 00:26:36.651655 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-7qfbg" event={"ID":"78f9f66c-5c2f-4cee-bf34-39961a78dbcc","Type":"ContainerDied","Data":"c06ebbc756b6ee80a2bc704ba9100a2f8b6e12c9a4a8306a3cb96fd9f01edb6f"} Jul 10 00:26:36.653515 ip-10-0-130-146 kubenswrapper[2570]: I0710 00:26:36.653489 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-t526b" event={"ID":"52179d08-5861-4a23-9c8d-653c13bed339","Type":"ContainerStarted","Data":"acc8651008652d4997271b1efa7bdf6972a964934fe04cc34e58d6f7f2e18201"} Jul 10 00:26:36.655183 ip-10-0-130-146 kubenswrapper[2570]: I0710 00:26:36.655157 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-7tn6x" event={"ID":"006ba44e-264c-42cd-9891-ad8d769386c8","Type":"ContainerStarted","Data":"ea49bee3771e8b911167feaac79cb237c23baa6c659421467085d618aba636bb"} Jul 10 00:26:36.657211 ip-10-0-130-146 kubenswrapper[2570]: I0710 00:26:36.657180 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-68tgb" event={"ID":"c44f0289-9f01-4548-ab98-74a17ec2b150","Type":"ContainerStarted","Data":"095be7970cc52cd5757f3ecee23ba6a61117903869e65c1ec0f98de4b8e038b3"} Jul 10 00:26:36.685065 ip-10-0-130-146 kubenswrapper[2570]: I0710 00:26:36.685008 2570 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/node-resolver-7tn6x" podStartSLOduration=5.835309712 podStartE2EDuration="15.684990149s" totalImagesPullingTime="9.849680437s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-10 00:26:21 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 00:26:36.68472939 +0000 UTC m=+15.733108113" watchObservedRunningTime="2026-07-10 00:26:36.684990149 +0000 UTC m=+15.733368896" Jul 10 00:26:37.448756 ip-10-0-130-146 kubenswrapper[2570]: I0710 00:26:37.448581 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-6bqvb" Jul 10 00:26:37.449131 ip-10-0-130-146 kubenswrapper[2570]: E0710 00:26:37.448837 2570 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-6bqvb" podUID="5667efb4-6019-4bde-827a-0f1e68096b93" Jul 10 00:26:37.661195 ip-10-0-130-146 kubenswrapper[2570]: I0710 00:26:37.661157 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-bw9kx" event={"ID":"5dcf6a9a-85e6-4f25-85b3-7b8a9eaa8654","Type":"ContainerStarted","Data":"ce1ea8f0e098df728383efcf9adfd915109bd06547507012040bbc937bd39bdf"} Jul 10 00:26:37.674947 ip-10-0-130-146 kubenswrapper[2570]: I0710 00:26:37.674906 2570 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-operator/iptables-alerter-bw9kx" podStartSLOduration=6.775593083 podStartE2EDuration="16.674886681s" totalImagesPullingTime="9.899293598s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-10 00:26:21 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 00:26:37.674870275 +0000 UTC m=+16.723248998" watchObservedRunningTime="2026-07-10 00:26:37.674886681 +0000 UTC m=+16.723265383" Jul 10 00:26:38.283791 ip-10-0-130-146 kubenswrapper[2570]: I0710 00:26:38.283753 2570 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kube-system/konnectivity-agent-995ch" Jul 10 00:26:38.284412 ip-10-0-130-146 kubenswrapper[2570]: I0710 00:26:38.284389 2570 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kube-system/konnectivity-agent-995ch" Jul 10 00:26:38.447865 ip-10-0-130-146 kubenswrapper[2570]: I0710 00:26:38.447832 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-5hs2b" Jul 10 00:26:38.448036 ip-10-0-130-146 kubenswrapper[2570]: E0710 00:26:38.447935 2570 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-5hs2b" podUID="78717d89-03f2-4824-942a-48892a2bfb99" Jul 10 00:26:38.448036 ip-10-0-130-146 kubenswrapper[2570]: I0710 00:26:38.448003 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-d8t56" Jul 10 00:26:38.448197 ip-10-0-130-146 kubenswrapper[2570]: E0710 00:26:38.448175 2570 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-d8t56" podUID="b7aab806-00a2-491e-a997-ed3174bd547a" Jul 10 00:26:38.663641 ip-10-0-130-146 kubenswrapper[2570]: I0710 00:26:38.663610 2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kube-system/konnectivity-agent-995ch" Jul 10 00:26:38.664283 ip-10-0-130-146 kubenswrapper[2570]: I0710 00:26:38.664065 2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-995ch" Jul 10 00:26:39.448138 ip-10-0-130-146 kubenswrapper[2570]: I0710 00:26:39.448108 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-6bqvb" Jul 10 00:26:39.448291 ip-10-0-130-146 kubenswrapper[2570]: E0710 00:26:39.448234 2570 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-6bqvb" podUID="5667efb4-6019-4bde-827a-0f1e68096b93" Jul 10 00:26:40.448130 ip-10-0-130-146 kubenswrapper[2570]: I0710 00:26:40.448095 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-d8t56" Jul 10 00:26:40.448130 ip-10-0-130-146 kubenswrapper[2570]: I0710 00:26:40.448115 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-5hs2b" Jul 10 00:26:40.448588 ip-10-0-130-146 kubenswrapper[2570]: E0710 00:26:40.448243 2570 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-d8t56" podUID="b7aab806-00a2-491e-a997-ed3174bd547a" Jul 10 00:26:40.448588 ip-10-0-130-146 kubenswrapper[2570]: E0710 00:26:40.448373 2570 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-5hs2b" podUID="78717d89-03f2-4824-942a-48892a2bfb99" Jul 10 00:26:40.999824 ip-10-0-130-146 kubenswrapper[2570]: I0710 00:26:40.999779 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b7aab806-00a2-491e-a997-ed3174bd547a-metrics-certs\") pod \"network-metrics-daemon-d8t56\" (UID: \"b7aab806-00a2-491e-a997-ed3174bd547a\") " pod="openshift-multus/network-metrics-daemon-d8t56" Jul 10 00:26:41.000002 ip-10-0-130-146 kubenswrapper[2570]: E0710 00:26:40.999977 2570 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 10 00:26:41.000155 ip-10-0-130-146 kubenswrapper[2570]: E0710 00:26:41.000062 2570 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b7aab806-00a2-491e-a997-ed3174bd547a-metrics-certs podName:b7aab806-00a2-491e-a997-ed3174bd547a nodeName:}" failed. No retries permitted until 2026-07-10 00:26:57.000029612 +0000 UTC m=+36.048408359 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/b7aab806-00a2-491e-a997-ed3174bd547a-metrics-certs") pod "network-metrics-daemon-d8t56" (UID: "b7aab806-00a2-491e-a997-ed3174bd547a") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 10 00:26:41.100342 ip-10-0-130-146 kubenswrapper[2570]: I0710 00:26:41.100306 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/5667efb4-6019-4bde-827a-0f1e68096b93-original-pull-secret\") pod \"global-pull-secret-syncer-6bqvb\" (UID: \"5667efb4-6019-4bde-827a-0f1e68096b93\") " pod="kube-system/global-pull-secret-syncer-6bqvb" Jul 10 00:26:41.100541 ip-10-0-130-146 kubenswrapper[2570]: E0710 00:26:41.100421 2570 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 10 00:26:41.100541 ip-10-0-130-146 kubenswrapper[2570]: E0710 00:26:41.100502 2570 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5667efb4-6019-4bde-827a-0f1e68096b93-original-pull-secret podName:5667efb4-6019-4bde-827a-0f1e68096b93 nodeName:}" failed. No retries permitted until 2026-07-10 00:26:57.100481467 +0000 UTC m=+36.148860167 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/5667efb4-6019-4bde-827a-0f1e68096b93-original-pull-secret") pod "global-pull-secret-syncer-6bqvb" (UID: "5667efb4-6019-4bde-827a-0f1e68096b93") : object "kube-system"/"original-pull-secret" not registered Jul 10 00:26:41.200874 ip-10-0-130-146 kubenswrapper[2570]: I0710 00:26:41.200834 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kl6s9\" (UniqueName: \"kubernetes.io/projected/78717d89-03f2-4824-942a-48892a2bfb99-kube-api-access-kl6s9\") pod \"network-check-target-5hs2b\" (UID: \"78717d89-03f2-4824-942a-48892a2bfb99\") " pod="openshift-network-diagnostics/network-check-target-5hs2b" Jul 10 00:26:41.201084 ip-10-0-130-146 kubenswrapper[2570]: E0710 00:26:41.201023 2570 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 10 00:26:41.201084 ip-10-0-130-146 kubenswrapper[2570]: E0710 00:26:41.201064 2570 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 10 00:26:41.201084 ip-10-0-130-146 kubenswrapper[2570]: E0710 00:26:41.201080 2570 projected.go:196] Error preparing data for projected volume kube-api-access-kl6s9 for pod openshift-network-diagnostics/network-check-target-5hs2b: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 10 00:26:41.201237 ip-10-0-130-146 kubenswrapper[2570]: E0710 00:26:41.201147 2570 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/78717d89-03f2-4824-942a-48892a2bfb99-kube-api-access-kl6s9 podName:78717d89-03f2-4824-942a-48892a2bfb99 nodeName:}" failed. No retries permitted until 2026-07-10 00:26:57.201127681 +0000 UTC m=+36.249506380 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "kube-api-access-kl6s9" (UniqueName: "kubernetes.io/projected/78717d89-03f2-4824-942a-48892a2bfb99-kube-api-access-kl6s9") pod "network-check-target-5hs2b" (UID: "78717d89-03f2-4824-942a-48892a2bfb99") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 10 00:26:41.449209 ip-10-0-130-146 kubenswrapper[2570]: I0710 00:26:41.449171 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-6bqvb" Jul 10 00:26:41.449684 ip-10-0-130-146 kubenswrapper[2570]: E0710 00:26:41.449296 2570 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-6bqvb" podUID="5667efb4-6019-4bde-827a-0f1e68096b93" Jul 10 00:26:42.448081 ip-10-0-130-146 kubenswrapper[2570]: I0710 00:26:42.448041 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-5hs2b" Jul 10 00:26:42.448252 ip-10-0-130-146 kubenswrapper[2570]: E0710 00:26:42.448166 2570 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-5hs2b" podUID="78717d89-03f2-4824-942a-48892a2bfb99" Jul 10 00:26:42.448252 ip-10-0-130-146 kubenswrapper[2570]: I0710 00:26:42.448225 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-d8t56" Jul 10 00:26:42.448381 ip-10-0-130-146 kubenswrapper[2570]: E0710 00:26:42.448361 2570 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-d8t56" podUID="b7aab806-00a2-491e-a997-ed3174bd547a" Jul 10 00:26:43.448069 ip-10-0-130-146 kubenswrapper[2570]: I0710 00:26:43.448020 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-6bqvb" Jul 10 00:26:43.448627 ip-10-0-130-146 kubenswrapper[2570]: E0710 00:26:43.448198 2570 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-6bqvb" podUID="5667efb4-6019-4bde-827a-0f1e68096b93" Jul 10 00:26:44.448039 ip-10-0-130-146 kubenswrapper[2570]: I0710 00:26:44.447995 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-5hs2b" Jul 10 00:26:44.448358 ip-10-0-130-146 kubenswrapper[2570]: I0710 00:26:44.447995 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-d8t56" Jul 10 00:26:44.448358 ip-10-0-130-146 kubenswrapper[2570]: E0710 00:26:44.448141 2570 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-5hs2b" podUID="78717d89-03f2-4824-942a-48892a2bfb99" Jul 10 00:26:44.448358 ip-10-0-130-146 kubenswrapper[2570]: E0710 00:26:44.448183 2570 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-d8t56" podUID="b7aab806-00a2-491e-a997-ed3174bd547a" Jul 10 00:26:45.448467 ip-10-0-130-146 kubenswrapper[2570]: I0710 00:26:45.448437 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-6bqvb" Jul 10 00:26:45.448905 ip-10-0-130-146 kubenswrapper[2570]: E0710 00:26:45.448565 2570 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-6bqvb" podUID="5667efb4-6019-4bde-827a-0f1e68096b93" Jul 10 00:26:46.447879 ip-10-0-130-146 kubenswrapper[2570]: I0710 00:26:46.447839 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-5hs2b" Jul 10 00:26:46.447879 ip-10-0-130-146 kubenswrapper[2570]: I0710 00:26:46.447882 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-d8t56" Jul 10 00:26:46.448148 ip-10-0-130-146 kubenswrapper[2570]: E0710 00:26:46.447971 2570 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-5hs2b" podUID="78717d89-03f2-4824-942a-48892a2bfb99" Jul 10 00:26:46.448148 ip-10-0-130-146 kubenswrapper[2570]: E0710 00:26:46.448114 2570 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-d8t56" podUID="b7aab806-00a2-491e-a997-ed3174bd547a" Jul 10 00:26:46.858891 ip-10-0-130-146 kubenswrapper[2570]: I0710 00:26:46.858865 2570 plugin_watcher.go:194] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock" Jul 10 00:26:47.313429 ip-10-0-130-146 kubenswrapper[2570]: I0710 00:26:47.313125 2570 reconciler.go:161] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-07-10T00:26:46.858886943Z","UUID":"d0b72f32-a4e1-4926-bb07-398b578f3b7d","Handler":null,"Name":"","Endpoint":""} Jul 10 00:26:47.315463 ip-10-0-130-146 kubenswrapper[2570]: I0710 00:26:47.315440 2570 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0 Jul 10 00:26:47.315595 ip-10-0-130-146 kubenswrapper[2570]: I0710 00:26:47.315479 2570 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock Jul 10 00:26:47.447836 ip-10-0-130-146 kubenswrapper[2570]: I0710 00:26:47.447805 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-6bqvb" Jul 10 00:26:47.448115 ip-10-0-130-146 kubenswrapper[2570]: E0710 00:26:47.447935 2570 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-6bqvb" podUID="5667efb4-6019-4bde-827a-0f1e68096b93" Jul 10 00:26:47.685211 ip-10-0-130-146 kubenswrapper[2570]: I0710 00:26:47.685180 2570 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-mbs58_c4ef019a-364a-4933-ab85-de86d86167f1/ovn-acl-logging/0.log" Jul 10 00:26:47.685594 ip-10-0-130-146 kubenswrapper[2570]: I0710 00:26:47.685562 2570 generic.go:356] "Generic (PLEG): container finished" podID="c4ef019a-364a-4933-ab85-de86d86167f1" containerID="308484e8b366c83a2a86b90558cf6c3c0ddd93cced3b36b7af0bd3c3e5e18f5b" exitCode=1 Jul 10 00:26:47.685704 ip-10-0-130-146 kubenswrapper[2570]: I0710 00:26:47.685610 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-mbs58" event={"ID":"c4ef019a-364a-4933-ab85-de86d86167f1","Type":"ContainerStarted","Data":"45da8ad9cf191c0e458a15617c6bb1869533dfc3f0b82cb25327be346f3c6002"} Jul 10 00:26:47.685704 ip-10-0-130-146 kubenswrapper[2570]: I0710 00:26:47.685693 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-mbs58" event={"ID":"c4ef019a-364a-4933-ab85-de86d86167f1","Type":"ContainerStarted","Data":"1959b7cf4451800e17dea6c39625d1930789efd48f051a3ab8b5cb487ed91820"} Jul 10 00:26:47.685812 ip-10-0-130-146 kubenswrapper[2570]: I0710 00:26:47.685733 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-mbs58" event={"ID":"c4ef019a-364a-4933-ab85-de86d86167f1","Type":"ContainerStarted","Data":"71f3000f7fb936d932c166bff5c17acdaed2e34821d6a6132106cf2e9477a978"} Jul 10 00:26:47.685812 ip-10-0-130-146 kubenswrapper[2570]: I0710 00:26:47.685775 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-mbs58" event={"ID":"c4ef019a-364a-4933-ab85-de86d86167f1","Type":"ContainerStarted","Data":"31deb5e27a31e37cdef585bcb2118a39f759520074f7ced27928a7340d1d49b2"} Jul 10 00:26:47.685894 ip-10-0-130-146 kubenswrapper[2570]: I0710 00:26:47.685812 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-mbs58" event={"ID":"c4ef019a-364a-4933-ab85-de86d86167f1","Type":"ContainerDied","Data":"308484e8b366c83a2a86b90558cf6c3c0ddd93cced3b36b7af0bd3c3e5e18f5b"} Jul 10 00:26:47.685894 ip-10-0-130-146 kubenswrapper[2570]: I0710 00:26:47.685847 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-mbs58" event={"ID":"c4ef019a-364a-4933-ab85-de86d86167f1","Type":"ContainerStarted","Data":"85ba14ddd8ecf24abf6ffe9641ce5cbb6a1baab2299d285595def54f6d4109e6"} Jul 10 00:26:47.687594 ip-10-0-130-146 kubenswrapper[2570]: I0710 00:26:47.687571 2570 generic.go:356] "Generic (PLEG): container finished" podID="78f9f66c-5c2f-4cee-bf34-39961a78dbcc" containerID="cb05bbb5fbd6d115229719b2b3e026cb8ac2e68b6fa02e7df3f045728d0b5aaf" exitCode=0 Jul 10 00:26:47.687702 ip-10-0-130-146 kubenswrapper[2570]: I0710 00:26:47.687640 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-7qfbg" event={"ID":"78f9f66c-5c2f-4cee-bf34-39961a78dbcc","Type":"ContainerDied","Data":"cb05bbb5fbd6d115229719b2b3e026cb8ac2e68b6fa02e7df3f045728d0b5aaf"} Jul 10 00:26:47.689634 ip-10-0-130-146 kubenswrapper[2570]: I0710 00:26:47.689438 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-2pdkh" event={"ID":"f726a6ab-3c92-403e-ad1b-bdaf7e37c5e3","Type":"ContainerStarted","Data":"b0cbbca5f417274f26c305f86b79cc02326899e8758dadc12d68db74eaa180ed"} Jul 10 00:26:47.691321 ip-10-0-130-146 kubenswrapper[2570]: I0710 00:26:47.691297 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-68tgb" event={"ID":"c44f0289-9f01-4548-ab98-74a17ec2b150","Type":"ContainerStarted","Data":"d99990cf90a4ed6c4f60ff51443a8aa5622a2b28fd159db9dc41070ecab0410d"} Jul 10 00:26:47.724710 ip-10-0-130-146 kubenswrapper[2570]: I0710 00:26:47.724662 2570 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-2pdkh" podStartSLOduration=5.572837735 podStartE2EDuration="26.724648252s" totalImagesPullingTime="21.151810517s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-10 00:26:21 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 00:26:47.724575761 +0000 UTC m=+26.772954483" watchObservedRunningTime="2026-07-10 00:26:47.724648252 +0000 UTC m=+26.773026972" Jul 10 00:26:48.448813 ip-10-0-130-146 kubenswrapper[2570]: I0710 00:26:48.448744 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-5hs2b" Jul 10 00:26:48.449467 ip-10-0-130-146 kubenswrapper[2570]: E0710 00:26:48.448852 2570 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-5hs2b" podUID="78717d89-03f2-4824-942a-48892a2bfb99" Jul 10 00:26:48.449467 ip-10-0-130-146 kubenswrapper[2570]: I0710 00:26:48.448907 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-d8t56" Jul 10 00:26:48.449467 ip-10-0-130-146 kubenswrapper[2570]: E0710 00:26:48.449007 2570 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-d8t56" podUID="b7aab806-00a2-491e-a997-ed3174bd547a" Jul 10 00:26:48.695226 ip-10-0-130-146 kubenswrapper[2570]: I0710 00:26:48.695185 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-68tgb" event={"ID":"c44f0289-9f01-4548-ab98-74a17ec2b150","Type":"ContainerStarted","Data":"858fce787a809fc7f8d64ad8203edc5b2d9aad050470c3070ef1e968b0d02fed"} Jul 10 00:26:48.712325 ip-10-0-130-146 kubenswrapper[2570]: I0710 00:26:48.712242 2570 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-68tgb" podStartSLOduration=5.485763766 podStartE2EDuration="27.712228097s" totalImagesPullingTime="22.226464331s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-10 00:26:21 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 00:26:48.711784283 +0000 UTC m=+27.760163004" watchObservedRunningTime="2026-07-10 00:26:48.712228097 +0000 UTC m=+27.760606817" Jul 10 00:26:49.448391 ip-10-0-130-146 kubenswrapper[2570]: I0710 00:26:49.448365 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-6bqvb" Jul 10 00:26:49.448552 ip-10-0-130-146 kubenswrapper[2570]: E0710 00:26:49.448494 2570 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-6bqvb" podUID="5667efb4-6019-4bde-827a-0f1e68096b93" Jul 10 00:26:49.699591 ip-10-0-130-146 kubenswrapper[2570]: I0710 00:26:49.699557 2570 generic.go:356] "Generic (PLEG): container finished" podID="78f9f66c-5c2f-4cee-bf34-39961a78dbcc" containerID="b33d860dc6e97b6c59442e864d80323a46cd2f45917c682a2413dabcd39e0d8d" exitCode=0 Jul 10 00:26:49.700030 ip-10-0-130-146 kubenswrapper[2570]: I0710 00:26:49.699630 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-7qfbg" event={"ID":"78f9f66c-5c2f-4cee-bf34-39961a78dbcc","Type":"ContainerDied","Data":"b33d860dc6e97b6c59442e864d80323a46cd2f45917c682a2413dabcd39e0d8d"} Jul 10 00:26:49.702711 ip-10-0-130-146 kubenswrapper[2570]: I0710 00:26:49.702695 2570 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-mbs58_c4ef019a-364a-4933-ab85-de86d86167f1/ovn-acl-logging/0.log" Jul 10 00:26:49.703081 ip-10-0-130-146 kubenswrapper[2570]: I0710 00:26:49.703043 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-mbs58" event={"ID":"c4ef019a-364a-4933-ab85-de86d86167f1","Type":"ContainerStarted","Data":"5acca7c70e2b7bbe4cca16b0eb04fd8ff8e8595a1e38a2003ac2853157fe0c07"} Jul 10 00:26:50.448473 ip-10-0-130-146 kubenswrapper[2570]: I0710 00:26:50.448452 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-5hs2b" Jul 10 00:26:50.448566 ip-10-0-130-146 kubenswrapper[2570]: E0710 00:26:50.448549 2570 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-5hs2b" podUID="78717d89-03f2-4824-942a-48892a2bfb99" Jul 10 00:26:50.448621 ip-10-0-130-146 kubenswrapper[2570]: I0710 00:26:50.448579 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-d8t56" Jul 10 00:26:50.448677 ip-10-0-130-146 kubenswrapper[2570]: E0710 00:26:50.448655 2570 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-d8t56" podUID="b7aab806-00a2-491e-a997-ed3174bd547a" Jul 10 00:26:50.707486 ip-10-0-130-146 kubenswrapper[2570]: I0710 00:26:50.707403 2570 generic.go:356] "Generic (PLEG): container finished" podID="78f9f66c-5c2f-4cee-bf34-39961a78dbcc" containerID="e089d38cf305669ad000b093efb4c97f53c3246822c76073cebe75a303f5db32" exitCode=0 Jul 10 00:26:50.707486 ip-10-0-130-146 kubenswrapper[2570]: I0710 00:26:50.707454 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-7qfbg" event={"ID":"78f9f66c-5c2f-4cee-bf34-39961a78dbcc","Type":"ContainerDied","Data":"e089d38cf305669ad000b093efb4c97f53c3246822c76073cebe75a303f5db32"} Jul 10 00:26:51.449272 ip-10-0-130-146 kubenswrapper[2570]: I0710 00:26:51.449091 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-6bqvb" Jul 10 00:26:51.449451 ip-10-0-130-146 kubenswrapper[2570]: E0710 00:26:51.449323 2570 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-6bqvb" podUID="5667efb4-6019-4bde-827a-0f1e68096b93" Jul 10 00:26:51.712542 ip-10-0-130-146 kubenswrapper[2570]: I0710 00:26:51.712450 2570 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-mbs58_c4ef019a-364a-4933-ab85-de86d86167f1/ovn-acl-logging/0.log" Jul 10 00:26:51.712958 ip-10-0-130-146 kubenswrapper[2570]: I0710 00:26:51.712816 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-mbs58" event={"ID":"c4ef019a-364a-4933-ab85-de86d86167f1","Type":"ContainerStarted","Data":"55dde61e00ce00059f000422a12ca801296cb4b9592a548006647910e47c680f"} Jul 10 00:26:51.713114 ip-10-0-130-146 kubenswrapper[2570]: I0710 00:26:51.713097 2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-mbs58" Jul 10 00:26:51.713178 ip-10-0-130-146 kubenswrapper[2570]: I0710 00:26:51.713166 2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-mbs58" Jul 10 00:26:51.713240 ip-10-0-130-146 kubenswrapper[2570]: I0710 00:26:51.713226 2570 scope.go:117] "RemoveContainer" containerID="308484e8b366c83a2a86b90558cf6c3c0ddd93cced3b36b7af0bd3c3e5e18f5b" Jul 10 00:26:51.729307 ip-10-0-130-146 kubenswrapper[2570]: I0710 00:26:51.729289 2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-mbs58" Jul 10 00:26:52.448453 ip-10-0-130-146 kubenswrapper[2570]: I0710 00:26:52.448421 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-d8t56" Jul 10 00:26:52.448620 ip-10-0-130-146 kubenswrapper[2570]: I0710 00:26:52.448420 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-5hs2b" Jul 10 00:26:52.448620 ip-10-0-130-146 kubenswrapper[2570]: E0710 00:26:52.448542 2570 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-d8t56" podUID="b7aab806-00a2-491e-a997-ed3174bd547a" Jul 10 00:26:52.448620 ip-10-0-130-146 kubenswrapper[2570]: E0710 00:26:52.448605 2570 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-5hs2b" podUID="78717d89-03f2-4824-942a-48892a2bfb99" Jul 10 00:26:52.721170 ip-10-0-130-146 kubenswrapper[2570]: I0710 00:26:52.720790 2570 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-mbs58_c4ef019a-364a-4933-ab85-de86d86167f1/ovn-acl-logging/0.log" Jul 10 00:26:52.721585 ip-10-0-130-146 kubenswrapper[2570]: I0710 00:26:52.721253 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-mbs58" event={"ID":"c4ef019a-364a-4933-ab85-de86d86167f1","Type":"ContainerStarted","Data":"5dbc89ed6d97c55f3d9cb89432ec9659dcc2ad025b12825d8ccb7d9345a58be7"} Jul 10 00:26:52.722501 ip-10-0-130-146 kubenswrapper[2570]: I0710 00:26:52.722398 2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-mbs58" Jul 10 00:26:52.744101 ip-10-0-130-146 kubenswrapper[2570]: I0710 00:26:52.744059 2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-mbs58" Jul 10 00:26:52.750905 ip-10-0-130-146 kubenswrapper[2570]: I0710 00:26:52.750855 2570 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ovn-kubernetes/ovnkube-node-mbs58" podStartSLOduration=10.596697855 podStartE2EDuration="31.750836032s" totalImagesPullingTime="21.154138177s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-10 00:26:21 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 00:26:52.749295332 +0000 UTC m=+31.797674101" watchObservedRunningTime="2026-07-10 00:26:52.750836032 +0000 UTC m=+31.799214754" Jul 10 00:26:53.452556 ip-10-0-130-146 kubenswrapper[2570]: I0710 00:26:53.452400 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-6bqvb" Jul 10 00:26:53.452688 ip-10-0-130-146 kubenswrapper[2570]: E0710 00:26:53.452640 2570 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-6bqvb" podUID="5667efb4-6019-4bde-827a-0f1e68096b93" Jul 10 00:26:53.461230 ip-10-0-130-146 kubenswrapper[2570]: I0710 00:26:53.461196 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-5hs2b"] Jul 10 00:26:53.461333 ip-10-0-130-146 kubenswrapper[2570]: I0710 00:26:53.461302 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-5hs2b" Jul 10 00:26:53.461415 ip-10-0-130-146 kubenswrapper[2570]: E0710 00:26:53.461394 2570 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-5hs2b" podUID="78717d89-03f2-4824-942a-48892a2bfb99" Jul 10 00:26:53.464291 ip-10-0-130-146 kubenswrapper[2570]: I0710 00:26:53.464270 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-6bqvb"] Jul 10 00:26:53.465128 ip-10-0-130-146 kubenswrapper[2570]: I0710 00:26:53.465086 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-d8t56"] Jul 10 00:26:53.465270 ip-10-0-130-146 kubenswrapper[2570]: I0710 00:26:53.465189 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-d8t56" Jul 10 00:26:53.465328 ip-10-0-130-146 kubenswrapper[2570]: E0710 00:26:53.465296 2570 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-d8t56" podUID="b7aab806-00a2-491e-a997-ed3174bd547a" Jul 10 00:26:53.723528 ip-10-0-130-146 kubenswrapper[2570]: I0710 00:26:53.723450 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-6bqvb" Jul 10 00:26:53.724002 ip-10-0-130-146 kubenswrapper[2570]: E0710 00:26:53.723575 2570 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-6bqvb" podUID="5667efb4-6019-4bde-827a-0f1e68096b93" Jul 10 00:26:55.448033 ip-10-0-130-146 kubenswrapper[2570]: I0710 00:26:55.448004 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-5hs2b" Jul 10 00:26:55.448523 ip-10-0-130-146 kubenswrapper[2570]: I0710 00:26:55.448003 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-d8t56" Jul 10 00:26:55.448523 ip-10-0-130-146 kubenswrapper[2570]: I0710 00:26:55.448008 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-6bqvb" Jul 10 00:26:55.448523 ip-10-0-130-146 kubenswrapper[2570]: E0710 00:26:55.448156 2570 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-5hs2b" podUID="78717d89-03f2-4824-942a-48892a2bfb99" Jul 10 00:26:55.448523 ip-10-0-130-146 kubenswrapper[2570]: E0710 00:26:55.448296 2570 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-d8t56" podUID="b7aab806-00a2-491e-a997-ed3174bd547a" Jul 10 00:26:55.448523 ip-10-0-130-146 kubenswrapper[2570]: E0710 00:26:55.448412 2570 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-6bqvb" podUID="5667efb4-6019-4bde-827a-0f1e68096b93" Jul 10 00:26:56.732138 ip-10-0-130-146 kubenswrapper[2570]: I0710 00:26:56.732110 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-7qfbg" event={"ID":"78f9f66c-5c2f-4cee-bf34-39961a78dbcc","Type":"ContainerStarted","Data":"94352caac0ca7ed852c151afcf8ef1d24cf5e483ea0d3afb108a619ead572ea3"} Jul 10 00:26:57.021500 ip-10-0-130-146 kubenswrapper[2570]: I0710 00:26:57.021474 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b7aab806-00a2-491e-a997-ed3174bd547a-metrics-certs\") pod \"network-metrics-daemon-d8t56\" (UID: \"b7aab806-00a2-491e-a997-ed3174bd547a\") " pod="openshift-multus/network-metrics-daemon-d8t56" Jul 10 00:26:57.021695 ip-10-0-130-146 kubenswrapper[2570]: E0710 00:26:57.021561 2570 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 10 00:26:57.021695 ip-10-0-130-146 kubenswrapper[2570]: E0710 00:26:57.021608 2570 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b7aab806-00a2-491e-a997-ed3174bd547a-metrics-certs podName:b7aab806-00a2-491e-a997-ed3174bd547a nodeName:}" failed. No retries permitted until 2026-07-10 00:27:29.021593446 +0000 UTC m=+68.069972144 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/b7aab806-00a2-491e-a997-ed3174bd547a-metrics-certs") pod "network-metrics-daemon-d8t56" (UID: "b7aab806-00a2-491e-a997-ed3174bd547a") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 10 00:26:57.122134 ip-10-0-130-146 kubenswrapper[2570]: I0710 00:26:57.122104 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/5667efb4-6019-4bde-827a-0f1e68096b93-original-pull-secret\") pod \"global-pull-secret-syncer-6bqvb\" (UID: \"5667efb4-6019-4bde-827a-0f1e68096b93\") " pod="kube-system/global-pull-secret-syncer-6bqvb" Jul 10 00:26:57.122250 ip-10-0-130-146 kubenswrapper[2570]: E0710 00:26:57.122231 2570 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 10 00:26:57.122296 ip-10-0-130-146 kubenswrapper[2570]: E0710 00:26:57.122286 2570 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5667efb4-6019-4bde-827a-0f1e68096b93-original-pull-secret podName:5667efb4-6019-4bde-827a-0f1e68096b93 nodeName:}" failed. No retries permitted until 2026-07-10 00:27:29.12226932 +0000 UTC m=+68.170648034 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/5667efb4-6019-4bde-827a-0f1e68096b93-original-pull-secret") pod "global-pull-secret-syncer-6bqvb" (UID: "5667efb4-6019-4bde-827a-0f1e68096b93") : object "kube-system"/"original-pull-secret" not registered Jul 10 00:26:57.223367 ip-10-0-130-146 kubenswrapper[2570]: I0710 00:26:57.223335 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kl6s9\" (UniqueName: \"kubernetes.io/projected/78717d89-03f2-4824-942a-48892a2bfb99-kube-api-access-kl6s9\") pod \"network-check-target-5hs2b\" (UID: \"78717d89-03f2-4824-942a-48892a2bfb99\") " pod="openshift-network-diagnostics/network-check-target-5hs2b" Jul 10 00:26:57.223494 ip-10-0-130-146 kubenswrapper[2570]: E0710 00:26:57.223469 2570 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 10 00:26:57.223494 ip-10-0-130-146 kubenswrapper[2570]: E0710 00:26:57.223488 2570 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 10 00:26:57.223559 ip-10-0-130-146 kubenswrapper[2570]: E0710 00:26:57.223499 2570 projected.go:196] Error preparing data for projected volume kube-api-access-kl6s9 for pod openshift-network-diagnostics/network-check-target-5hs2b: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 10 00:26:57.223559 ip-10-0-130-146 kubenswrapper[2570]: E0710 00:26:57.223545 2570 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/78717d89-03f2-4824-942a-48892a2bfb99-kube-api-access-kl6s9 podName:78717d89-03f2-4824-942a-48892a2bfb99 nodeName:}" failed. No retries permitted until 2026-07-10 00:27:29.223530238 +0000 UTC m=+68.271908938 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "kube-api-access-kl6s9" (UniqueName: "kubernetes.io/projected/78717d89-03f2-4824-942a-48892a2bfb99-kube-api-access-kl6s9") pod "network-check-target-5hs2b" (UID: "78717d89-03f2-4824-942a-48892a2bfb99") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 10 00:26:57.451044 ip-10-0-130-146 kubenswrapper[2570]: I0710 00:26:57.451009 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-6bqvb" Jul 10 00:26:57.451231 ip-10-0-130-146 kubenswrapper[2570]: I0710 00:26:57.451134 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-d8t56" Jul 10 00:26:57.451231 ip-10-0-130-146 kubenswrapper[2570]: E0710 00:26:57.451146 2570 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-6bqvb" podUID="5667efb4-6019-4bde-827a-0f1e68096b93" Jul 10 00:26:57.451231 ip-10-0-130-146 kubenswrapper[2570]: I0710 00:26:57.451162 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-5hs2b" Jul 10 00:26:57.451374 ip-10-0-130-146 kubenswrapper[2570]: E0710 00:26:57.451247 2570 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-d8t56" podUID="b7aab806-00a2-491e-a997-ed3174bd547a" Jul 10 00:26:57.451374 ip-10-0-130-146 kubenswrapper[2570]: E0710 00:26:57.451309 2570 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-5hs2b" podUID="78717d89-03f2-4824-942a-48892a2bfb99" Jul 10 00:26:57.736473 ip-10-0-130-146 kubenswrapper[2570]: I0710 00:26:57.736403 2570 generic.go:356] "Generic (PLEG): container finished" podID="78f9f66c-5c2f-4cee-bf34-39961a78dbcc" containerID="94352caac0ca7ed852c151afcf8ef1d24cf5e483ea0d3afb108a619ead572ea3" exitCode=0 Jul 10 00:26:57.736473 ip-10-0-130-146 kubenswrapper[2570]: I0710 00:26:57.736452 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-7qfbg" event={"ID":"78f9f66c-5c2f-4cee-bf34-39961a78dbcc","Type":"ContainerDied","Data":"94352caac0ca7ed852c151afcf8ef1d24cf5e483ea0d3afb108a619ead572ea3"} Jul 10 00:26:58.741668 ip-10-0-130-146 kubenswrapper[2570]: I0710 00:26:58.741639 2570 generic.go:356] "Generic (PLEG): container finished" podID="78f9f66c-5c2f-4cee-bf34-39961a78dbcc" containerID="833deeb84d6414665164ad950756cd2bf889a719e1e55380e5816c74867ffb09" exitCode=0 Jul 10 00:26:58.742127 ip-10-0-130-146 kubenswrapper[2570]: I0710 00:26:58.741682 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-7qfbg" event={"ID":"78f9f66c-5c2f-4cee-bf34-39961a78dbcc","Type":"ContainerDied","Data":"833deeb84d6414665164ad950756cd2bf889a719e1e55380e5816c74867ffb09"} Jul 10 00:26:59.448161 ip-10-0-130-146 kubenswrapper[2570]: I0710 00:26:59.448119 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-5hs2b" Jul 10 00:26:59.448318 ip-10-0-130-146 kubenswrapper[2570]: I0710 00:26:59.448141 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-6bqvb" Jul 10 00:26:59.448318 ip-10-0-130-146 kubenswrapper[2570]: E0710 00:26:59.448257 2570 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-5hs2b" podUID="78717d89-03f2-4824-942a-48892a2bfb99" Jul 10 00:26:59.448318 ip-10-0-130-146 kubenswrapper[2570]: I0710 00:26:59.448285 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-d8t56" Jul 10 00:26:59.448434 ip-10-0-130-146 kubenswrapper[2570]: E0710 00:26:59.448412 2570 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-6bqvb" podUID="5667efb4-6019-4bde-827a-0f1e68096b93" Jul 10 00:26:59.448529 ip-10-0-130-146 kubenswrapper[2570]: E0710 00:26:59.448513 2570 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-d8t56" podUID="b7aab806-00a2-491e-a997-ed3174bd547a" Jul 10 00:26:59.746241 ip-10-0-130-146 kubenswrapper[2570]: I0710 00:26:59.746167 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-7qfbg" event={"ID":"78f9f66c-5c2f-4cee-bf34-39961a78dbcc","Type":"ContainerStarted","Data":"ca5aa56133f08ff56acae3233bf900aea30fa29788c1094bc94ea8740aa32261"} Jul 10 00:26:59.770302 ip-10-0-130-146 kubenswrapper[2570]: I0710 00:26:59.770258 2570 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-additional-cni-plugins-7qfbg" podStartSLOduration=11.134565751 podStartE2EDuration="38.77024392s" totalImagesPullingTime="27.635678169s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-10 00:26:21 +0000 UTC" imagePullSessionsCount=5 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 00:26:59.768891091 +0000 UTC m=+38.817269813" watchObservedRunningTime="2026-07-10 00:26:59.77024392 +0000 UTC m=+38.818622640" Jul 10 00:27:00.320950 ip-10-0-130-146 kubenswrapper[2570]: I0710 00:27:00.320922 2570 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-130-146.ec2.internal" event="NodeReady" Jul 10 00:27:00.321143 ip-10-0-130-146 kubenswrapper[2570]: I0710 00:27:00.321095 2570 kubelet_node_status.go:489] "Fast updating node status as it just became ready" Jul 10 00:27:00.363661 ip-10-0-130-146 kubenswrapper[2570]: I0710 00:27:00.363629 2570 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-dns/dns-default-fcndn"] Jul 10 00:27:00.385465 ip-10-0-130-146 kubenswrapper[2570]: I0710 00:27:00.385440 2570 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress-canary/ingress-canary-ck6gk"] Jul 10 00:27:00.385612 ip-10-0-130-146 kubenswrapper[2570]: I0710 00:27:00.385548 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-fcndn" Jul 10 00:27:00.388625 ip-10-0-130-146 kubenswrapper[2570]: I0710 00:27:00.388593 2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\"" Jul 10 00:27:00.388764 ip-10-0-130-146 kubenswrapper[2570]: I0710 00:27:00.388593 2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\"" Jul 10 00:27:00.388764 ip-10-0-130-146 kubenswrapper[2570]: I0710 00:27:00.388602 2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-2cgcq\"" Jul 10 00:27:00.405598 ip-10-0-130-146 kubenswrapper[2570]: I0710 00:27:00.405430 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-fcndn"] Jul 10 00:27:00.405598 ip-10-0-130-146 kubenswrapper[2570]: I0710 00:27:00.405588 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-ck6gk"] Jul 10 00:27:00.405707 ip-10-0-130-146 kubenswrapper[2570]: I0710 00:27:00.405541 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-ck6gk" Jul 10 00:27:00.408335 ip-10-0-130-146 kubenswrapper[2570]: I0710 00:27:00.408319 2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"default-dockercfg-5rckt\"" Jul 10 00:27:00.408416 ip-10-0-130-146 kubenswrapper[2570]: I0710 00:27:00.408320 2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\"" Jul 10 00:27:00.408416 ip-10-0-130-146 kubenswrapper[2570]: I0710 00:27:00.408371 2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\"" Jul 10 00:27:00.408416 ip-10-0-130-146 kubenswrapper[2570]: I0710 00:27:00.408371 2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\"" Jul 10 00:27:00.547516 ip-10-0-130-146 kubenswrapper[2570]: I0710 00:27:00.547488 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wfp44\" (UniqueName: \"kubernetes.io/projected/3ec525d5-c820-42fe-a411-99faa6ef373d-kube-api-access-wfp44\") pod \"ingress-canary-ck6gk\" (UID: \"3ec525d5-c820-42fe-a411-99faa6ef373d\") " pod="openshift-ingress-canary/ingress-canary-ck6gk" Jul 10 00:27:00.547691 ip-10-0-130-146 kubenswrapper[2570]: I0710 00:27:00.547531 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bbv7j\" (UniqueName: \"kubernetes.io/projected/e2252bd4-69bf-4c02-a315-d9cbcf4ed91e-kube-api-access-bbv7j\") pod \"dns-default-fcndn\" (UID: \"e2252bd4-69bf-4c02-a315-d9cbcf4ed91e\") " pod="openshift-dns/dns-default-fcndn" Jul 10 00:27:00.547691 ip-10-0-130-146 kubenswrapper[2570]: I0710 00:27:00.547572 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/3ec525d5-c820-42fe-a411-99faa6ef373d-cert\") pod \"ingress-canary-ck6gk\" (UID: \"3ec525d5-c820-42fe-a411-99faa6ef373d\") " pod="openshift-ingress-canary/ingress-canary-ck6gk" Jul 10 00:27:00.547691 ip-10-0-130-146 kubenswrapper[2570]: I0710 00:27:00.547630 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e2252bd4-69bf-4c02-a315-d9cbcf4ed91e-tmp-dir\") pod \"dns-default-fcndn\" (UID: \"e2252bd4-69bf-4c02-a315-d9cbcf4ed91e\") " pod="openshift-dns/dns-default-fcndn" Jul 10 00:27:00.547691 ip-10-0-130-146 kubenswrapper[2570]: I0710 00:27:00.547664 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/e2252bd4-69bf-4c02-a315-d9cbcf4ed91e-config-volume\") pod \"dns-default-fcndn\" (UID: \"e2252bd4-69bf-4c02-a315-d9cbcf4ed91e\") " pod="openshift-dns/dns-default-fcndn" Jul 10 00:27:00.547842 ip-10-0-130-146 kubenswrapper[2570]: I0710 00:27:00.547697 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/e2252bd4-69bf-4c02-a315-d9cbcf4ed91e-metrics-tls\") pod \"dns-default-fcndn\" (UID: \"e2252bd4-69bf-4c02-a315-d9cbcf4ed91e\") " pod="openshift-dns/dns-default-fcndn" Jul 10 00:27:00.648833 ip-10-0-130-146 kubenswrapper[2570]: I0710 00:27:00.648799 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e2252bd4-69bf-4c02-a315-d9cbcf4ed91e-tmp-dir\") pod \"dns-default-fcndn\" (UID: \"e2252bd4-69bf-4c02-a315-d9cbcf4ed91e\") " pod="openshift-dns/dns-default-fcndn" Jul 10 00:27:00.649003 ip-10-0-130-146 kubenswrapper[2570]: I0710 00:27:00.648840 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/e2252bd4-69bf-4c02-a315-d9cbcf4ed91e-config-volume\") pod \"dns-default-fcndn\" (UID: \"e2252bd4-69bf-4c02-a315-d9cbcf4ed91e\") " pod="openshift-dns/dns-default-fcndn" Jul 10 00:27:00.649003 ip-10-0-130-146 kubenswrapper[2570]: I0710 00:27:00.648878 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/e2252bd4-69bf-4c02-a315-d9cbcf4ed91e-metrics-tls\") pod \"dns-default-fcndn\" (UID: \"e2252bd4-69bf-4c02-a315-d9cbcf4ed91e\") " pod="openshift-dns/dns-default-fcndn" Jul 10 00:27:00.649003 ip-10-0-130-146 kubenswrapper[2570]: I0710 00:27:00.648913 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wfp44\" (UniqueName: \"kubernetes.io/projected/3ec525d5-c820-42fe-a411-99faa6ef373d-kube-api-access-wfp44\") pod \"ingress-canary-ck6gk\" (UID: \"3ec525d5-c820-42fe-a411-99faa6ef373d\") " pod="openshift-ingress-canary/ingress-canary-ck6gk" Jul 10 00:27:00.649003 ip-10-0-130-146 kubenswrapper[2570]: I0710 00:27:00.648951 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bbv7j\" (UniqueName: \"kubernetes.io/projected/e2252bd4-69bf-4c02-a315-d9cbcf4ed91e-kube-api-access-bbv7j\") pod \"dns-default-fcndn\" (UID: \"e2252bd4-69bf-4c02-a315-d9cbcf4ed91e\") " pod="openshift-dns/dns-default-fcndn" Jul 10 00:27:00.649248 ip-10-0-130-146 kubenswrapper[2570]: E0710 00:27:00.649043 2570 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jul 10 00:27:00.649248 ip-10-0-130-146 kubenswrapper[2570]: E0710 00:27:00.649126 2570 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e2252bd4-69bf-4c02-a315-d9cbcf4ed91e-metrics-tls podName:e2252bd4-69bf-4c02-a315-d9cbcf4ed91e nodeName:}" failed. No retries permitted until 2026-07-10 00:27:01.149107707 +0000 UTC m=+40.197486408 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/e2252bd4-69bf-4c02-a315-d9cbcf4ed91e-metrics-tls") pod "dns-default-fcndn" (UID: "e2252bd4-69bf-4c02-a315-d9cbcf4ed91e") : secret "dns-default-metrics-tls" not found Jul 10 00:27:00.649248 ip-10-0-130-146 kubenswrapper[2570]: I0710 00:27:00.649117 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/3ec525d5-c820-42fe-a411-99faa6ef373d-cert\") pod \"ingress-canary-ck6gk\" (UID: \"3ec525d5-c820-42fe-a411-99faa6ef373d\") " pod="openshift-ingress-canary/ingress-canary-ck6gk" Jul 10 00:27:00.649248 ip-10-0-130-146 kubenswrapper[2570]: I0710 00:27:00.649197 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e2252bd4-69bf-4c02-a315-d9cbcf4ed91e-tmp-dir\") pod \"dns-default-fcndn\" (UID: \"e2252bd4-69bf-4c02-a315-d9cbcf4ed91e\") " pod="openshift-dns/dns-default-fcndn" Jul 10 00:27:00.649589 ip-10-0-130-146 kubenswrapper[2570]: I0710 00:27:00.649566 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/e2252bd4-69bf-4c02-a315-d9cbcf4ed91e-config-volume\") pod \"dns-default-fcndn\" (UID: \"e2252bd4-69bf-4c02-a315-d9cbcf4ed91e\") " pod="openshift-dns/dns-default-fcndn" Jul 10 00:27:00.652978 ip-10-0-130-146 kubenswrapper[2570]: I0710 00:27:00.652961 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/3ec525d5-c820-42fe-a411-99faa6ef373d-cert\") pod \"ingress-canary-ck6gk\" (UID: \"3ec525d5-c820-42fe-a411-99faa6ef373d\") " pod="openshift-ingress-canary/ingress-canary-ck6gk" Jul 10 00:27:00.659387 ip-10-0-130-146 kubenswrapper[2570]: I0710 00:27:00.659316 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-bbv7j\" (UniqueName: \"kubernetes.io/projected/e2252bd4-69bf-4c02-a315-d9cbcf4ed91e-kube-api-access-bbv7j\") pod \"dns-default-fcndn\" (UID: \"e2252bd4-69bf-4c02-a315-d9cbcf4ed91e\") " pod="openshift-dns/dns-default-fcndn" Jul 10 00:27:00.659387 ip-10-0-130-146 kubenswrapper[2570]: I0710 00:27:00.659362 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-wfp44\" (UniqueName: \"kubernetes.io/projected/3ec525d5-c820-42fe-a411-99faa6ef373d-kube-api-access-wfp44\") pod \"ingress-canary-ck6gk\" (UID: \"3ec525d5-c820-42fe-a411-99faa6ef373d\") " pod="openshift-ingress-canary/ingress-canary-ck6gk" Jul 10 00:27:00.713003 ip-10-0-130-146 kubenswrapper[2570]: I0710 00:27:00.712980 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-ck6gk" Jul 10 00:27:00.895191 ip-10-0-130-146 kubenswrapper[2570]: I0710 00:27:00.895163 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-ck6gk"] Jul 10 00:27:00.898690 ip-10-0-130-146 kubenswrapper[2570]: W0710 00:27:00.898664 2570 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3ec525d5_c820_42fe_a411_99faa6ef373d.slice/crio-fa1ce5faf6eae95f2b076bf97b9b7e277f24ded0870ebbb2c6278de63bc4fab0 WatchSource:0}: Error finding container fa1ce5faf6eae95f2b076bf97b9b7e277f24ded0870ebbb2c6278de63bc4fab0: Status 404 returned error can't find the container with id fa1ce5faf6eae95f2b076bf97b9b7e277f24ded0870ebbb2c6278de63bc4fab0 Jul 10 00:27:01.151909 ip-10-0-130-146 kubenswrapper[2570]: I0710 00:27:01.151871 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/e2252bd4-69bf-4c02-a315-d9cbcf4ed91e-metrics-tls\") pod \"dns-default-fcndn\" (UID: \"e2252bd4-69bf-4c02-a315-d9cbcf4ed91e\") " pod="openshift-dns/dns-default-fcndn" Jul 10 00:27:01.152111 ip-10-0-130-146 kubenswrapper[2570]: E0710 00:27:01.152022 2570 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jul 10 00:27:01.152111 ip-10-0-130-146 kubenswrapper[2570]: E0710 00:27:01.152102 2570 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e2252bd4-69bf-4c02-a315-d9cbcf4ed91e-metrics-tls podName:e2252bd4-69bf-4c02-a315-d9cbcf4ed91e nodeName:}" failed. No retries permitted until 2026-07-10 00:27:02.152084659 +0000 UTC m=+41.200463361 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/e2252bd4-69bf-4c02-a315-d9cbcf4ed91e-metrics-tls") pod "dns-default-fcndn" (UID: "e2252bd4-69bf-4c02-a315-d9cbcf4ed91e") : secret "dns-default-metrics-tls" not found Jul 10 00:27:01.449225 ip-10-0-130-146 kubenswrapper[2570]: I0710 00:27:01.449140 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-d8t56" Jul 10 00:27:01.449387 ip-10-0-130-146 kubenswrapper[2570]: I0710 00:27:01.449230 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-5hs2b" Jul 10 00:27:01.449387 ip-10-0-130-146 kubenswrapper[2570]: I0710 00:27:01.449274 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-6bqvb" Jul 10 00:27:01.452097 ip-10-0-130-146 kubenswrapper[2570]: I0710 00:27:01.452073 2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\"" Jul 10 00:27:01.453450 ip-10-0-130-146 kubenswrapper[2570]: I0710 00:27:01.453429 2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-d5zcx\"" Jul 10 00:27:01.453615 ip-10-0-130-146 kubenswrapper[2570]: I0710 00:27:01.453426 2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\"" Jul 10 00:27:01.453615 ip-10-0-130-146 kubenswrapper[2570]: I0710 00:27:01.453487 2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-zkg6s\"" Jul 10 00:27:01.453615 ip-10-0-130-146 kubenswrapper[2570]: I0710 00:27:01.453426 2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\"" Jul 10 00:27:01.453781 ip-10-0-130-146 kubenswrapper[2570]: I0710 00:27:01.453767 2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\"" Jul 10 00:27:01.752860 ip-10-0-130-146 kubenswrapper[2570]: I0710 00:27:01.752783 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-ck6gk" event={"ID":"3ec525d5-c820-42fe-a411-99faa6ef373d","Type":"ContainerStarted","Data":"fa1ce5faf6eae95f2b076bf97b9b7e277f24ded0870ebbb2c6278de63bc4fab0"} Jul 10 00:27:02.157696 ip-10-0-130-146 kubenswrapper[2570]: I0710 00:27:02.157660 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/e2252bd4-69bf-4c02-a315-d9cbcf4ed91e-metrics-tls\") pod \"dns-default-fcndn\" (UID: \"e2252bd4-69bf-4c02-a315-d9cbcf4ed91e\") " pod="openshift-dns/dns-default-fcndn" Jul 10 00:27:02.158085 ip-10-0-130-146 kubenswrapper[2570]: E0710 00:27:02.157793 2570 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jul 10 00:27:02.158085 ip-10-0-130-146 kubenswrapper[2570]: E0710 00:27:02.157846 2570 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e2252bd4-69bf-4c02-a315-d9cbcf4ed91e-metrics-tls podName:e2252bd4-69bf-4c02-a315-d9cbcf4ed91e nodeName:}" failed. No retries permitted until 2026-07-10 00:27:04.157831175 +0000 UTC m=+43.206209875 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/e2252bd4-69bf-4c02-a315-d9cbcf4ed91e-metrics-tls") pod "dns-default-fcndn" (UID: "e2252bd4-69bf-4c02-a315-d9cbcf4ed91e") : secret "dns-default-metrics-tls" not found Jul 10 00:27:04.172293 ip-10-0-130-146 kubenswrapper[2570]: I0710 00:27:04.172253 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/e2252bd4-69bf-4c02-a315-d9cbcf4ed91e-metrics-tls\") pod \"dns-default-fcndn\" (UID: \"e2252bd4-69bf-4c02-a315-d9cbcf4ed91e\") " pod="openshift-dns/dns-default-fcndn" Jul 10 00:27:04.172890 ip-10-0-130-146 kubenswrapper[2570]: E0710 00:27:04.172384 2570 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jul 10 00:27:04.172890 ip-10-0-130-146 kubenswrapper[2570]: E0710 00:27:04.172437 2570 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e2252bd4-69bf-4c02-a315-d9cbcf4ed91e-metrics-tls podName:e2252bd4-69bf-4c02-a315-d9cbcf4ed91e nodeName:}" failed. No retries permitted until 2026-07-10 00:27:08.17242239 +0000 UTC m=+47.220801089 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/e2252bd4-69bf-4c02-a315-d9cbcf4ed91e-metrics-tls") pod "dns-default-fcndn" (UID: "e2252bd4-69bf-4c02-a315-d9cbcf4ed91e") : secret "dns-default-metrics-tls" not found Jul 10 00:27:04.759573 ip-10-0-130-146 kubenswrapper[2570]: I0710 00:27:04.759541 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-ck6gk" event={"ID":"3ec525d5-c820-42fe-a411-99faa6ef373d","Type":"ContainerStarted","Data":"699e1c8dec8c9aa1fa85189caba02cc9275ab1c26b4c22b30784bcfdcc1c8db4"} Jul 10 00:27:04.775610 ip-10-0-130-146 kubenswrapper[2570]: I0710 00:27:04.775547 2570 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress-canary/ingress-canary-ck6gk" podStartSLOduration=1.725276195 podStartE2EDuration="4.775535319s" totalImagesPullingTime="3.050259124s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-10 00:27:00 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 00:27:04.774798106 +0000 UTC m=+43.823176829" watchObservedRunningTime="2026-07-10 00:27:04.775535319 +0000 UTC m=+43.823914040" Jul 10 00:27:08.201487 ip-10-0-130-146 kubenswrapper[2570]: I0710 00:27:08.201446 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/e2252bd4-69bf-4c02-a315-d9cbcf4ed91e-metrics-tls\") pod \"dns-default-fcndn\" (UID: \"e2252bd4-69bf-4c02-a315-d9cbcf4ed91e\") " pod="openshift-dns/dns-default-fcndn" Jul 10 00:27:08.202033 ip-10-0-130-146 kubenswrapper[2570]: E0710 00:27:08.201571 2570 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jul 10 00:27:08.202033 ip-10-0-130-146 kubenswrapper[2570]: E0710 00:27:08.201640 2570 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e2252bd4-69bf-4c02-a315-d9cbcf4ed91e-metrics-tls podName:e2252bd4-69bf-4c02-a315-d9cbcf4ed91e nodeName:}" failed. No retries permitted until 2026-07-10 00:27:16.201623641 +0000 UTC m=+55.250002341 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/e2252bd4-69bf-4c02-a315-d9cbcf4ed91e-metrics-tls") pod "dns-default-fcndn" (UID: "e2252bd4-69bf-4c02-a315-d9cbcf4ed91e") : secret "dns-default-metrics-tls" not found Jul 10 00:27:16.257034 ip-10-0-130-146 kubenswrapper[2570]: I0710 00:27:16.256990 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/e2252bd4-69bf-4c02-a315-d9cbcf4ed91e-metrics-tls\") pod \"dns-default-fcndn\" (UID: \"e2252bd4-69bf-4c02-a315-d9cbcf4ed91e\") " pod="openshift-dns/dns-default-fcndn" Jul 10 00:27:16.257606 ip-10-0-130-146 kubenswrapper[2570]: E0710 00:27:16.257124 2570 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jul 10 00:27:16.257606 ip-10-0-130-146 kubenswrapper[2570]: E0710 00:27:16.257190 2570 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e2252bd4-69bf-4c02-a315-d9cbcf4ed91e-metrics-tls podName:e2252bd4-69bf-4c02-a315-d9cbcf4ed91e nodeName:}" failed. No retries permitted until 2026-07-10 00:27:32.257174909 +0000 UTC m=+71.305553607 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/e2252bd4-69bf-4c02-a315-d9cbcf4ed91e-metrics-tls") pod "dns-default-fcndn" (UID: "e2252bd4-69bf-4c02-a315-d9cbcf4ed91e") : secret "dns-default-metrics-tls" not found Jul 10 00:27:24.735675 ip-10-0-130-146 kubenswrapper[2570]: I0710 00:27:24.735647 2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-mbs58" Jul 10 00:27:29.046266 ip-10-0-130-146 kubenswrapper[2570]: I0710 00:27:29.046226 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b7aab806-00a2-491e-a997-ed3174bd547a-metrics-certs\") pod \"network-metrics-daemon-d8t56\" (UID: \"b7aab806-00a2-491e-a997-ed3174bd547a\") " pod="openshift-multus/network-metrics-daemon-d8t56" Jul 10 00:27:29.048882 ip-10-0-130-146 kubenswrapper[2570]: I0710 00:27:29.048866 2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\"" Jul 10 00:27:29.056915 ip-10-0-130-146 kubenswrapper[2570]: E0710 00:27:29.056895 2570 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found Jul 10 00:27:29.057009 ip-10-0-130-146 kubenswrapper[2570]: E0710 00:27:29.056955 2570 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b7aab806-00a2-491e-a997-ed3174bd547a-metrics-certs podName:b7aab806-00a2-491e-a997-ed3174bd547a nodeName:}" failed. No retries permitted until 2026-07-10 00:28:33.056937728 +0000 UTC m=+132.105316428 (durationBeforeRetry 1m4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/b7aab806-00a2-491e-a997-ed3174bd547a-metrics-certs") pod "network-metrics-daemon-d8t56" (UID: "b7aab806-00a2-491e-a997-ed3174bd547a") : secret "metrics-daemon-secret" not found Jul 10 00:27:29.147448 ip-10-0-130-146 kubenswrapper[2570]: I0710 00:27:29.147403 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/5667efb4-6019-4bde-827a-0f1e68096b93-original-pull-secret\") pod \"global-pull-secret-syncer-6bqvb\" (UID: \"5667efb4-6019-4bde-827a-0f1e68096b93\") " pod="kube-system/global-pull-secret-syncer-6bqvb" Jul 10 00:27:29.150433 ip-10-0-130-146 kubenswrapper[2570]: I0710 00:27:29.150410 2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\"" Jul 10 00:27:29.161772 ip-10-0-130-146 kubenswrapper[2570]: I0710 00:27:29.161749 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/5667efb4-6019-4bde-827a-0f1e68096b93-original-pull-secret\") pod \"global-pull-secret-syncer-6bqvb\" (UID: \"5667efb4-6019-4bde-827a-0f1e68096b93\") " pod="kube-system/global-pull-secret-syncer-6bqvb" Jul 10 00:27:29.248265 ip-10-0-130-146 kubenswrapper[2570]: I0710 00:27:29.248226 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kl6s9\" (UniqueName: \"kubernetes.io/projected/78717d89-03f2-4824-942a-48892a2bfb99-kube-api-access-kl6s9\") pod \"network-check-target-5hs2b\" (UID: \"78717d89-03f2-4824-942a-48892a2bfb99\") " pod="openshift-network-diagnostics/network-check-target-5hs2b" Jul 10 00:27:29.250894 ip-10-0-130-146 kubenswrapper[2570]: I0710 00:27:29.250876 2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\"" Jul 10 00:27:29.261367 ip-10-0-130-146 kubenswrapper[2570]: I0710 00:27:29.261345 2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\"" Jul 10 00:27:29.275242 ip-10-0-130-146 kubenswrapper[2570]: I0710 00:27:29.275218 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-kl6s9\" (UniqueName: \"kubernetes.io/projected/78717d89-03f2-4824-942a-48892a2bfb99-kube-api-access-kl6s9\") pod \"network-check-target-5hs2b\" (UID: \"78717d89-03f2-4824-942a-48892a2bfb99\") " pod="openshift-network-diagnostics/network-check-target-5hs2b" Jul 10 00:27:29.361301 ip-10-0-130-146 kubenswrapper[2570]: I0710 00:27:29.361203 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-6bqvb" Jul 10 00:27:29.377337 ip-10-0-130-146 kubenswrapper[2570]: I0710 00:27:29.377312 2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-zkg6s\"" Jul 10 00:27:29.385234 ip-10-0-130-146 kubenswrapper[2570]: I0710 00:27:29.385214 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-5hs2b" Jul 10 00:27:29.509714 ip-10-0-130-146 kubenswrapper[2570]: I0710 00:27:29.509681 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-6bqvb"] Jul 10 00:27:29.513752 ip-10-0-130-146 kubenswrapper[2570]: W0710 00:27:29.513723 2570 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5667efb4_6019_4bde_827a_0f1e68096b93.slice/crio-ca1601131938b78119957d55073ca1c1cab8b23eef320ce8bffee151267e1893 WatchSource:0}: Error finding container ca1601131938b78119957d55073ca1c1cab8b23eef320ce8bffee151267e1893: Status 404 returned error can't find the container with id ca1601131938b78119957d55073ca1c1cab8b23eef320ce8bffee151267e1893 Jul 10 00:27:29.544994 ip-10-0-130-146 kubenswrapper[2570]: I0710 00:27:29.544963 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-5hs2b"] Jul 10 00:27:29.564535 ip-10-0-130-146 kubenswrapper[2570]: W0710 00:27:29.564508 2570 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod78717d89_03f2_4824_942a_48892a2bfb99.slice/crio-08810234e15dfcae7c906895ec7b3afcd48ef07f5563bcc75c0e54040ab4ad9f WatchSource:0}: Error finding container 08810234e15dfcae7c906895ec7b3afcd48ef07f5563bcc75c0e54040ab4ad9f: Status 404 returned error can't find the container with id 08810234e15dfcae7c906895ec7b3afcd48ef07f5563bcc75c0e54040ab4ad9f Jul 10 00:27:29.818529 ip-10-0-130-146 kubenswrapper[2570]: I0710 00:27:29.818495 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-6bqvb" event={"ID":"5667efb4-6019-4bde-827a-0f1e68096b93","Type":"ContainerStarted","Data":"ca1601131938b78119957d55073ca1c1cab8b23eef320ce8bffee151267e1893"} Jul 10 00:27:29.819477 ip-10-0-130-146 kubenswrapper[2570]: I0710 00:27:29.819453 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-5hs2b" event={"ID":"78717d89-03f2-4824-942a-48892a2bfb99","Type":"ContainerStarted","Data":"08810234e15dfcae7c906895ec7b3afcd48ef07f5563bcc75c0e54040ab4ad9f"} Jul 10 00:27:32.270176 ip-10-0-130-146 kubenswrapper[2570]: I0710 00:27:32.270125 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/e2252bd4-69bf-4c02-a315-d9cbcf4ed91e-metrics-tls\") pod \"dns-default-fcndn\" (UID: \"e2252bd4-69bf-4c02-a315-d9cbcf4ed91e\") " pod="openshift-dns/dns-default-fcndn" Jul 10 00:27:32.270612 ip-10-0-130-146 kubenswrapper[2570]: E0710 00:27:32.270257 2570 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jul 10 00:27:32.270612 ip-10-0-130-146 kubenswrapper[2570]: E0710 00:27:32.270327 2570 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e2252bd4-69bf-4c02-a315-d9cbcf4ed91e-metrics-tls podName:e2252bd4-69bf-4c02-a315-d9cbcf4ed91e nodeName:}" failed. No retries permitted until 2026-07-10 00:28:04.270308264 +0000 UTC m=+103.318686986 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/e2252bd4-69bf-4c02-a315-d9cbcf4ed91e-metrics-tls") pod "dns-default-fcndn" (UID: "e2252bd4-69bf-4c02-a315-d9cbcf4ed91e") : secret "dns-default-metrics-tls" not found Jul 10 00:27:34.834278 ip-10-0-130-146 kubenswrapper[2570]: I0710 00:27:34.834236 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-5hs2b" event={"ID":"78717d89-03f2-4824-942a-48892a2bfb99","Type":"ContainerStarted","Data":"7a77b5034f5af035a86dd8d38173f7c009fcc7213c4fd140472694f3780a7b02"} Jul 10 00:27:34.834758 ip-10-0-130-146 kubenswrapper[2570]: I0710 00:27:34.834307 2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-5hs2b" Jul 10 00:27:34.835726 ip-10-0-130-146 kubenswrapper[2570]: I0710 00:27:34.835692 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-6bqvb" event={"ID":"5667efb4-6019-4bde-827a-0f1e68096b93","Type":"ContainerStarted","Data":"81c27275367ec3efa000999c9f4e91fe67bd24a8bc74a5427357acb6eebcf6ce"} Jul 10 00:27:34.849490 ip-10-0-130-146 kubenswrapper[2570]: I0710 00:27:34.849442 2570 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-target-5hs2b" podStartSLOduration=69.539916079 podStartE2EDuration="1m13.849428362s" totalImagesPullingTime="4.309512283s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-10 00:26:21 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 00:27:34.848555017 +0000 UTC m=+73.896933741" watchObservedRunningTime="2026-07-10 00:27:34.849428362 +0000 UTC m=+73.897807083" Jul 10 00:27:34.861628 ip-10-0-130-146 kubenswrapper[2570]: I0710 00:27:34.861584 2570 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/global-pull-secret-syncer-6bqvb" podStartSLOduration=69.497007941 podStartE2EDuration="1m13.861570972s" totalImagesPullingTime="4.364563031s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-10 00:26:21 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 00:27:34.861149238 +0000 UTC m=+73.909527959" watchObservedRunningTime="2026-07-10 00:27:34.861570972 +0000 UTC m=+73.909949692" Jul 10 00:28:04.283636 ip-10-0-130-146 kubenswrapper[2570]: I0710 00:28:04.283590 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/e2252bd4-69bf-4c02-a315-d9cbcf4ed91e-metrics-tls\") pod \"dns-default-fcndn\" (UID: \"e2252bd4-69bf-4c02-a315-d9cbcf4ed91e\") " pod="openshift-dns/dns-default-fcndn" Jul 10 00:28:04.284015 ip-10-0-130-146 kubenswrapper[2570]: E0710 00:28:04.283701 2570 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jul 10 00:28:04.284015 ip-10-0-130-146 kubenswrapper[2570]: E0710 00:28:04.283761 2570 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e2252bd4-69bf-4c02-a315-d9cbcf4ed91e-metrics-tls podName:e2252bd4-69bf-4c02-a315-d9cbcf4ed91e nodeName:}" failed. No retries permitted until 2026-07-10 00:29:08.283747502 +0000 UTC m=+167.332126201 (durationBeforeRetry 1m4s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/e2252bd4-69bf-4c02-a315-d9cbcf4ed91e-metrics-tls") pod "dns-default-fcndn" (UID: "e2252bd4-69bf-4c02-a315-d9cbcf4ed91e") : secret "dns-default-metrics-tls" not found Jul 10 00:28:05.841181 ip-10-0-130-146 kubenswrapper[2570]: I0710 00:28:05.841151 2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-network-diagnostics/network-check-target-5hs2b" Jul 10 00:28:33.079172 ip-10-0-130-146 kubenswrapper[2570]: I0710 00:28:33.079137 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b7aab806-00a2-491e-a997-ed3174bd547a-metrics-certs\") pod \"network-metrics-daemon-d8t56\" (UID: \"b7aab806-00a2-491e-a997-ed3174bd547a\") " pod="openshift-multus/network-metrics-daemon-d8t56" Jul 10 00:28:33.079575 ip-10-0-130-146 kubenswrapper[2570]: E0710 00:28:33.079257 2570 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found Jul 10 00:28:33.079575 ip-10-0-130-146 kubenswrapper[2570]: E0710 00:28:33.079317 2570 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b7aab806-00a2-491e-a997-ed3174bd547a-metrics-certs podName:b7aab806-00a2-491e-a997-ed3174bd547a nodeName:}" failed. No retries permitted until 2026-07-10 00:30:35.079303011 +0000 UTC m=+254.127681710 (durationBeforeRetry 2m2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/b7aab806-00a2-491e-a997-ed3174bd547a-metrics-certs") pod "network-metrics-daemon-d8t56" (UID: "b7aab806-00a2-491e-a997-ed3174bd547a") : secret "metrics-daemon-secret" not found Jul 10 00:29:03.395919 ip-10-0-130-146 kubenswrapper[2570]: E0710 00:29:03.395874 2570 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[metrics-tls], unattached volumes=[], failed to process volumes=[]: context deadline exceeded" pod="openshift-dns/dns-default-fcndn" podUID="e2252bd4-69bf-4c02-a315-d9cbcf4ed91e" Jul 10 00:29:04.047506 ip-10-0-130-146 kubenswrapper[2570]: I0710 00:29:04.047479 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-fcndn" Jul 10 00:29:04.468110 ip-10-0-130-146 kubenswrapper[2570]: E0710 00:29:04.468063 2570 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[metrics-certs], unattached volumes=[], failed to process volumes=[]: context deadline exceeded" pod="openshift-multus/network-metrics-daemon-d8t56" podUID="b7aab806-00a2-491e-a997-ed3174bd547a" Jul 10 00:29:08.314144 ip-10-0-130-146 kubenswrapper[2570]: I0710 00:29:08.314108 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/e2252bd4-69bf-4c02-a315-d9cbcf4ed91e-metrics-tls\") pod \"dns-default-fcndn\" (UID: \"e2252bd4-69bf-4c02-a315-d9cbcf4ed91e\") " pod="openshift-dns/dns-default-fcndn" Jul 10 00:29:08.314556 ip-10-0-130-146 kubenswrapper[2570]: E0710 00:29:08.314217 2570 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jul 10 00:29:08.314556 ip-10-0-130-146 kubenswrapper[2570]: E0710 00:29:08.314277 2570 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e2252bd4-69bf-4c02-a315-d9cbcf4ed91e-metrics-tls podName:e2252bd4-69bf-4c02-a315-d9cbcf4ed91e nodeName:}" failed. No retries permitted until 2026-07-10 00:31:10.314261599 +0000 UTC m=+289.362640298 (durationBeforeRetry 2m2s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/e2252bd4-69bf-4c02-a315-d9cbcf4ed91e-metrics-tls") pod "dns-default-fcndn" (UID: "e2252bd4-69bf-4c02-a315-d9cbcf4ed91e") : secret "dns-default-metrics-tls" not found Jul 10 00:29:17.448668 ip-10-0-130-146 kubenswrapper[2570]: I0710 00:29:17.448630 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-d8t56" Jul 10 00:30:20.638421 ip-10-0-130-146 kubenswrapper[2570]: I0710 00:30:20.638389 2570 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-network-diagnostics/network-check-source-869dcfcdcd-4vnvp"] Jul 10 00:30:20.641249 ip-10-0-130-146 kubenswrapper[2570]: I0710 00:30:20.641226 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-source-869dcfcdcd-4vnvp" Jul 10 00:30:20.645151 ip-10-0-130-146 kubenswrapper[2570]: I0710 00:30:20.645133 2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"network-diagnostics-dockercfg-jzrmj\"" Jul 10 00:30:20.650415 ip-10-0-130-146 kubenswrapper[2570]: I0710 00:30:20.650396 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-source-869dcfcdcd-4vnvp"] Jul 10 00:30:20.656031 ip-10-0-130-146 kubenswrapper[2570]: I0710 00:30:20.656009 2570 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console-operator/console-operator-7866497494-d6fcd"] Jul 10 00:30:20.658666 ip-10-0-130-146 kubenswrapper[2570]: I0710 00:30:20.658650 2570 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-operator-68d9dd94cf-7rq8g"] Jul 10 00:30:20.658801 ip-10-0-130-146 kubenswrapper[2570]: I0710 00:30:20.658785 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console-operator/console-operator-7866497494-d6fcd" Jul 10 00:30:20.661294 ip-10-0-130-146 kubenswrapper[2570]: I0710 00:30:20.661266 2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"console-operator-config\"" Jul 10 00:30:20.661400 ip-10-0-130-146 kubenswrapper[2570]: I0710 00:30:20.661328 2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console-operator\"/\"serving-cert\"" Jul 10 00:30:20.661400 ip-10-0-130-146 kubenswrapper[2570]: I0710 00:30:20.661341 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-operator-68d9dd94cf-7rq8g" Jul 10 00:30:20.661400 ip-10-0-130-146 kubenswrapper[2570]: I0710 00:30:20.661381 2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"openshift-service-ca.crt\"" Jul 10 00:30:20.661565 ip-10-0-130-146 kubenswrapper[2570]: I0710 00:30:20.661427 2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"kube-root-ca.crt\"" Jul 10 00:30:20.661565 ip-10-0-130-146 kubenswrapper[2570]: I0710 00:30:20.661448 2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console-operator\"/\"console-operator-dockercfg-lxskd\"" Jul 10 00:30:20.663842 ip-10-0-130-146 kubenswrapper[2570]: I0710 00:30:20.663825 2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\"" Jul 10 00:30:20.665162 ip-10-0-130-146 kubenswrapper[2570]: I0710 00:30:20.665142 2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"service-ca-bundle\"" Jul 10 00:30:20.665674 ip-10-0-130-146 kubenswrapper[2570]: I0710 00:30:20.665577 2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"openshift-insights-serving-cert\"" Jul 10 00:30:20.665674 ip-10-0-130-146 kubenswrapper[2570]: I0710 00:30:20.665627 2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"operator-dockercfg-d5s4b\"" Jul 10 00:30:20.666197 ip-10-0-130-146 kubenswrapper[2570]: I0710 00:30:20.666181 2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\"" Jul 10 00:30:20.668346 ip-10-0-130-146 kubenswrapper[2570]: I0710 00:30:20.668326 2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"trusted-ca\"" Jul 10 00:30:20.670686 ip-10-0-130-146 kubenswrapper[2570]: I0710 00:30:20.670669 2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"trusted-ca-bundle\"" Jul 10 00:30:20.670798 ip-10-0-130-146 kubenswrapper[2570]: I0710 00:30:20.670784 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console-operator/console-operator-7866497494-d6fcd"] Jul 10 00:30:20.671664 ip-10-0-130-146 kubenswrapper[2570]: I0710 00:30:20.671646 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-operator-68d9dd94cf-7rq8g"] Jul 10 00:30:20.691640 ip-10-0-130-146 kubenswrapper[2570]: I0710 00:30:20.691618 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rlhm7\" (UniqueName: \"kubernetes.io/projected/28eaf961-fb7d-495a-8733-abd273807b37-kube-api-access-rlhm7\") pod \"network-check-source-869dcfcdcd-4vnvp\" (UID: \"28eaf961-fb7d-495a-8733-abd273807b37\") " pod="openshift-network-diagnostics/network-check-source-869dcfcdcd-4vnvp" Jul 10 00:30:20.735151 ip-10-0-130-146 kubenswrapper[2570]: I0710 00:30:20.735124 2570 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-799cb7f958-p44ms"] Jul 10 00:30:20.737825 ip-10-0-130-146 kubenswrapper[2570]: I0710 00:30:20.737806 2570 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-network-console/networking-console-plugin-6cf856758c-45x56"] Jul 10 00:30:20.737938 ip-10-0-130-146 kubenswrapper[2570]: I0710 00:30:20.737923 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-799cb7f958-p44ms" Jul 10 00:30:20.740380 ip-10-0-130-146 kubenswrapper[2570]: I0710 00:30:20.740361 2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"config\"" Jul 10 00:30:20.740478 ip-10-0-130-146 kubenswrapper[2570]: I0710 00:30:20.740367 2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"kube-storage-version-migrator-operator-dockercfg-p8b2f\"" Jul 10 00:30:20.740478 ip-10-0-130-146 kubenswrapper[2570]: I0710 00:30:20.740439 2570 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-cluster-samples-operator/cluster-samples-operator-65db858894-bzzfq"] Jul 10 00:30:20.740604 ip-10-0-130-146 kubenswrapper[2570]: I0710 00:30:20.740589 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-console/networking-console-plugin-6cf856758c-45x56" Jul 10 00:30:20.740652 ip-10-0-130-146 kubenswrapper[2570]: I0710 00:30:20.740603 2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"openshift-service-ca.crt\"" Jul 10 00:30:20.740652 ip-10-0-130-146 kubenswrapper[2570]: I0710 00:30:20.740620 2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"serving-cert\"" Jul 10 00:30:20.740652 ip-10-0-130-146 kubenswrapper[2570]: I0710 00:30:20.740603 2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"kube-root-ca.crt\"" Jul 10 00:30:20.743035 ip-10-0-130-146 kubenswrapper[2570]: I0710 00:30:20.742979 2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-console\"/\"networking-console-plugin-cert\"" Jul 10 00:30:20.743150 ip-10-0-130-146 kubenswrapper[2570]: I0710 00:30:20.743073 2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-console\"/\"networking-console-plugin\"" Jul 10 00:30:20.743150 ip-10-0-130-146 kubenswrapper[2570]: I0710 00:30:20.743111 2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-console\"/\"default-dockercfg-rtz98\"" Jul 10 00:30:20.743318 ip-10-0-130-146 kubenswrapper[2570]: I0710 00:30:20.743286 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-samples-operator/cluster-samples-operator-65db858894-bzzfq" Jul 10 00:30:20.743423 ip-10-0-130-146 kubenswrapper[2570]: I0710 00:30:20.743395 2570 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-image-registry/image-registry-b9fdbf76c-kptvw"] Jul 10 00:30:20.746070 ip-10-0-130-146 kubenswrapper[2570]: I0710 00:30:20.746038 2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-samples-operator\"/\"cluster-samples-operator-dockercfg-g478g\"" Jul 10 00:30:20.746070 ip-10-0-130-146 kubenswrapper[2570]: I0710 00:30:20.746065 2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-samples-operator\"/\"kube-root-ca.crt\"" Jul 10 00:30:20.746211 ip-10-0-130-146 kubenswrapper[2570]: I0710 00:30:20.746080 2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-samples-operator\"/\"samples-operator-tls\"" Jul 10 00:30:20.746211 ip-10-0-130-146 kubenswrapper[2570]: I0710 00:30:20.746113 2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-samples-operator\"/\"openshift-service-ca.crt\"" Jul 10 00:30:20.746211 ip-10-0-130-146 kubenswrapper[2570]: I0710 00:30:20.746132 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-b9fdbf76c-kptvw" Jul 10 00:30:20.749029 ip-10-0-130-146 kubenswrapper[2570]: I0710 00:30:20.749014 2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"registry-dockercfg-6nz9c\"" Jul 10 00:30:20.749137 ip-10-0-130-146 kubenswrapper[2570]: I0710 00:30:20.749014 2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-tls\"" Jul 10 00:30:20.749300 ip-10-0-130-146 kubenswrapper[2570]: I0710 00:30:20.749285 2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-private-configuration\"" Jul 10 00:30:20.749764 ip-10-0-130-146 kubenswrapper[2570]: I0710 00:30:20.749747 2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"installation-pull-secrets\"" Jul 10 00:30:20.750242 ip-10-0-130-146 kubenswrapper[2570]: I0710 00:30:20.750227 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-799cb7f958-p44ms"] Jul 10 00:30:20.753856 ip-10-0-130-146 kubenswrapper[2570]: I0710 00:30:20.753838 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-samples-operator/cluster-samples-operator-65db858894-bzzfq"] Jul 10 00:30:20.755185 ip-10-0-130-146 kubenswrapper[2570]: I0710 00:30:20.755165 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-console/networking-console-plugin-6cf856758c-45x56"] Jul 10 00:30:20.757009 ip-10-0-130-146 kubenswrapper[2570]: I0710 00:30:20.756965 2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"trusted-ca\"" Jul 10 00:30:20.763170 ip-10-0-130-146 kubenswrapper[2570]: I0710 00:30:20.763152 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-b9fdbf76c-kptvw"] Jul 10 00:30:20.792546 ip-10-0-130-146 kubenswrapper[2570]: I0710 00:30:20.792523 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/f612c430-f495-4773-9ca6-82f7417e3a0c-tmp\") pod \"insights-operator-68d9dd94cf-7rq8g\" (UID: \"f612c430-f495-4773-9ca6-82f7417e3a0c\") " pod="openshift-insights/insights-operator-68d9dd94cf-7rq8g" Jul 10 00:30:20.792637 ip-10-0-130-146 kubenswrapper[2570]: I0710 00:30:20.792549 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-86mjf\" (UniqueName: \"kubernetes.io/projected/6f87cbd8-9d72-4708-a230-2818fa5a836d-kube-api-access-86mjf\") pod \"kube-storage-version-migrator-operator-799cb7f958-p44ms\" (UID: \"6f87cbd8-9d72-4708-a230-2818fa5a836d\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-799cb7f958-p44ms" Jul 10 00:30:20.792637 ip-10-0-130-146 kubenswrapper[2570]: I0710 00:30:20.792568 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/2b6cb4ca-79ff-42f0-b080-66af842f22d3-trusted-ca\") pod \"console-operator-7866497494-d6fcd\" (UID: \"2b6cb4ca-79ff-42f0-b080-66af842f22d3\") " pod="openshift-console-operator/console-operator-7866497494-d6fcd" Jul 10 00:30:20.792637 ip-10-0-130-146 kubenswrapper[2570]: I0710 00:30:20.792583 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/d4b4eb0f-539c-4193-907e-cbff010aae7b-nginx-conf\") pod \"networking-console-plugin-6cf856758c-45x56\" (UID: \"d4b4eb0f-539c-4193-907e-cbff010aae7b\") " pod="openshift-network-console/networking-console-plugin-6cf856758c-45x56" Jul 10 00:30:20.792637 ip-10-0-130-146 kubenswrapper[2570]: I0710 00:30:20.792607 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rlhm7\" (UniqueName: \"kubernetes.io/projected/28eaf961-fb7d-495a-8733-abd273807b37-kube-api-access-rlhm7\") pod \"network-check-source-869dcfcdcd-4vnvp\" (UID: \"28eaf961-fb7d-495a-8733-abd273807b37\") " pod="openshift-network-diagnostics/network-check-source-869dcfcdcd-4vnvp" Jul 10 00:30:20.792808 ip-10-0-130-146 kubenswrapper[2570]: I0710 00:30:20.792644 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/57c018ee-b976-4f56-a33b-b41cb28c4417-installation-pull-secrets\") pod \"image-registry-b9fdbf76c-kptvw\" (UID: \"57c018ee-b976-4f56-a33b-b41cb28c4417\") " pod="openshift-image-registry/image-registry-b9fdbf76c-kptvw" Jul 10 00:30:20.792808 ip-10-0-130-146 kubenswrapper[2570]: I0710 00:30:20.792674 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qpsq9\" (UniqueName: \"kubernetes.io/projected/57c018ee-b976-4f56-a33b-b41cb28c4417-kube-api-access-qpsq9\") pod \"image-registry-b9fdbf76c-kptvw\" (UID: \"57c018ee-b976-4f56-a33b-b41cb28c4417\") " pod="openshift-image-registry/image-registry-b9fdbf76c-kptvw" Jul 10 00:30:20.792808 ip-10-0-130-146 kubenswrapper[2570]: I0710 00:30:20.792711 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/6f87cbd8-9d72-4708-a230-2818fa5a836d-serving-cert\") pod \"kube-storage-version-migrator-operator-799cb7f958-p44ms\" (UID: \"6f87cbd8-9d72-4708-a230-2818fa5a836d\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-799cb7f958-p44ms" Jul 10 00:30:20.792808 ip-10-0-130-146 kubenswrapper[2570]: I0710 00:30:20.792735 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-l5tsr\" (UniqueName: \"kubernetes.io/projected/2b6cb4ca-79ff-42f0-b080-66af842f22d3-kube-api-access-l5tsr\") pod \"console-operator-7866497494-d6fcd\" (UID: \"2b6cb4ca-79ff-42f0-b080-66af842f22d3\") " pod="openshift-console-operator/console-operator-7866497494-d6fcd" Jul 10 00:30:20.792808 ip-10-0-130-146 kubenswrapper[2570]: I0710 00:30:20.792778 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/57c018ee-b976-4f56-a33b-b41cb28c4417-image-registry-private-configuration\") pod \"image-registry-b9fdbf76c-kptvw\" (UID: \"57c018ee-b976-4f56-a33b-b41cb28c4417\") " pod="openshift-image-registry/image-registry-b9fdbf76c-kptvw" Jul 10 00:30:20.793028 ip-10-0-130-146 kubenswrapper[2570]: I0710 00:30:20.792815 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"snapshots\" (UniqueName: \"kubernetes.io/empty-dir/f612c430-f495-4773-9ca6-82f7417e3a0c-snapshots\") pod \"insights-operator-68d9dd94cf-7rq8g\" (UID: \"f612c430-f495-4773-9ca6-82f7417e3a0c\") " pod="openshift-insights/insights-operator-68d9dd94cf-7rq8g" Jul 10 00:30:20.793028 ip-10-0-130-146 kubenswrapper[2570]: I0710 00:30:20.792839 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/57c018ee-b976-4f56-a33b-b41cb28c4417-registry-certificates\") pod \"image-registry-b9fdbf76c-kptvw\" (UID: \"57c018ee-b976-4f56-a33b-b41cb28c4417\") " pod="openshift-image-registry/image-registry-b9fdbf76c-kptvw" Jul 10 00:30:20.793028 ip-10-0-130-146 kubenswrapper[2570]: I0710 00:30:20.792858 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/57c018ee-b976-4f56-a33b-b41cb28c4417-trusted-ca\") pod \"image-registry-b9fdbf76c-kptvw\" (UID: \"57c018ee-b976-4f56-a33b-b41cb28c4417\") " pod="openshift-image-registry/image-registry-b9fdbf76c-kptvw" Jul 10 00:30:20.793028 ip-10-0-130-146 kubenswrapper[2570]: I0710 00:30:20.792873 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/57c018ee-b976-4f56-a33b-b41cb28c4417-ca-trust-extracted\") pod \"image-registry-b9fdbf76c-kptvw\" (UID: \"57c018ee-b976-4f56-a33b-b41cb28c4417\") " pod="openshift-image-registry/image-registry-b9fdbf76c-kptvw" Jul 10 00:30:20.793028 ip-10-0-130-146 kubenswrapper[2570]: I0710 00:30:20.792889 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f612c430-f495-4773-9ca6-82f7417e3a0c-trusted-ca-bundle\") pod \"insights-operator-68d9dd94cf-7rq8g\" (UID: \"f612c430-f495-4773-9ca6-82f7417e3a0c\") " pod="openshift-insights/insights-operator-68d9dd94cf-7rq8g" Jul 10 00:30:20.793028 ip-10-0-130-146 kubenswrapper[2570]: I0710 00:30:20.792905 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5rx5k\" (UniqueName: \"kubernetes.io/projected/f612c430-f495-4773-9ca6-82f7417e3a0c-kube-api-access-5rx5k\") pod \"insights-operator-68d9dd94cf-7rq8g\" (UID: \"f612c430-f495-4773-9ca6-82f7417e3a0c\") " pod="openshift-insights/insights-operator-68d9dd94cf-7rq8g" Jul 10 00:30:20.793028 ip-10-0-130-146 kubenswrapper[2570]: I0710 00:30:20.792920 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/4e3a154b-8ba7-4c18-b75b-93444d0de385-samples-operator-tls\") pod \"cluster-samples-operator-65db858894-bzzfq\" (UID: \"4e3a154b-8ba7-4c18-b75b-93444d0de385\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-65db858894-bzzfq" Jul 10 00:30:20.793028 ip-10-0-130-146 kubenswrapper[2570]: I0710 00:30:20.792944 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/2b6cb4ca-79ff-42f0-b080-66af842f22d3-config\") pod \"console-operator-7866497494-d6fcd\" (UID: \"2b6cb4ca-79ff-42f0-b080-66af842f22d3\") " pod="openshift-console-operator/console-operator-7866497494-d6fcd" Jul 10 00:30:20.793028 ip-10-0-130-146 kubenswrapper[2570]: I0710 00:30:20.792983 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/2b6cb4ca-79ff-42f0-b080-66af842f22d3-serving-cert\") pod \"console-operator-7866497494-d6fcd\" (UID: \"2b6cb4ca-79ff-42f0-b080-66af842f22d3\") " pod="openshift-console-operator/console-operator-7866497494-d6fcd" Jul 10 00:30:20.793028 ip-10-0-130-146 kubenswrapper[2570]: I0710 00:30:20.793022 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/6f87cbd8-9d72-4708-a230-2818fa5a836d-config\") pod \"kube-storage-version-migrator-operator-799cb7f958-p44ms\" (UID: \"6f87cbd8-9d72-4708-a230-2818fa5a836d\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-799cb7f958-p44ms" Jul 10 00:30:20.793382 ip-10-0-130-146 kubenswrapper[2570]: I0710 00:30:20.793041 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/57c018ee-b976-4f56-a33b-b41cb28c4417-bound-sa-token\") pod \"image-registry-b9fdbf76c-kptvw\" (UID: \"57c018ee-b976-4f56-a33b-b41cb28c4417\") " pod="openshift-image-registry/image-registry-b9fdbf76c-kptvw" Jul 10 00:30:20.793382 ip-10-0-130-146 kubenswrapper[2570]: I0710 00:30:20.793079 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f612c430-f495-4773-9ca6-82f7417e3a0c-service-ca-bundle\") pod \"insights-operator-68d9dd94cf-7rq8g\" (UID: \"f612c430-f495-4773-9ca6-82f7417e3a0c\") " pod="openshift-insights/insights-operator-68d9dd94cf-7rq8g" Jul 10 00:30:20.793382 ip-10-0-130-146 kubenswrapper[2570]: I0710 00:30:20.793094 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/f612c430-f495-4773-9ca6-82f7417e3a0c-serving-cert\") pod \"insights-operator-68d9dd94cf-7rq8g\" (UID: \"f612c430-f495-4773-9ca6-82f7417e3a0c\") " pod="openshift-insights/insights-operator-68d9dd94cf-7rq8g" Jul 10 00:30:20.793382 ip-10-0-130-146 kubenswrapper[2570]: I0710 00:30:20.793115 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/57c018ee-b976-4f56-a33b-b41cb28c4417-registry-tls\") pod \"image-registry-b9fdbf76c-kptvw\" (UID: \"57c018ee-b976-4f56-a33b-b41cb28c4417\") " pod="openshift-image-registry/image-registry-b9fdbf76c-kptvw" Jul 10 00:30:20.793382 ip-10-0-130-146 kubenswrapper[2570]: I0710 00:30:20.793172 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/d4b4eb0f-539c-4193-907e-cbff010aae7b-networking-console-plugin-cert\") pod \"networking-console-plugin-6cf856758c-45x56\" (UID: \"d4b4eb0f-539c-4193-907e-cbff010aae7b\") " pod="openshift-network-console/networking-console-plugin-6cf856758c-45x56" Jul 10 00:30:20.793382 ip-10-0-130-146 kubenswrapper[2570]: I0710 00:30:20.793207 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jgg4z\" (UniqueName: \"kubernetes.io/projected/4e3a154b-8ba7-4c18-b75b-93444d0de385-kube-api-access-jgg4z\") pod \"cluster-samples-operator-65db858894-bzzfq\" (UID: \"4e3a154b-8ba7-4c18-b75b-93444d0de385\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-65db858894-bzzfq" Jul 10 00:30:20.803236 ip-10-0-130-146 kubenswrapper[2570]: I0710 00:30:20.803218 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rlhm7\" (UniqueName: \"kubernetes.io/projected/28eaf961-fb7d-495a-8733-abd273807b37-kube-api-access-rlhm7\") pod \"network-check-source-869dcfcdcd-4vnvp\" (UID: \"28eaf961-fb7d-495a-8733-abd273807b37\") " pod="openshift-network-diagnostics/network-check-source-869dcfcdcd-4vnvp" Jul 10 00:30:20.893926 ip-10-0-130-146 kubenswrapper[2570]: I0710 00:30:20.893855 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jgg4z\" (UniqueName: \"kubernetes.io/projected/4e3a154b-8ba7-4c18-b75b-93444d0de385-kube-api-access-jgg4z\") pod \"cluster-samples-operator-65db858894-bzzfq\" (UID: \"4e3a154b-8ba7-4c18-b75b-93444d0de385\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-65db858894-bzzfq" Jul 10 00:30:20.893926 ip-10-0-130-146 kubenswrapper[2570]: I0710 00:30:20.893890 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/f612c430-f495-4773-9ca6-82f7417e3a0c-tmp\") pod \"insights-operator-68d9dd94cf-7rq8g\" (UID: \"f612c430-f495-4773-9ca6-82f7417e3a0c\") " pod="openshift-insights/insights-operator-68d9dd94cf-7rq8g" Jul 10 00:30:20.893926 ip-10-0-130-146 kubenswrapper[2570]: I0710 00:30:20.893911 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-86mjf\" (UniqueName: \"kubernetes.io/projected/6f87cbd8-9d72-4708-a230-2818fa5a836d-kube-api-access-86mjf\") pod \"kube-storage-version-migrator-operator-799cb7f958-p44ms\" (UID: \"6f87cbd8-9d72-4708-a230-2818fa5a836d\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-799cb7f958-p44ms" Jul 10 00:30:20.894146 ip-10-0-130-146 kubenswrapper[2570]: I0710 00:30:20.893934 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/2b6cb4ca-79ff-42f0-b080-66af842f22d3-trusted-ca\") pod \"console-operator-7866497494-d6fcd\" (UID: \"2b6cb4ca-79ff-42f0-b080-66af842f22d3\") " pod="openshift-console-operator/console-operator-7866497494-d6fcd" Jul 10 00:30:20.894146 ip-10-0-130-146 kubenswrapper[2570]: I0710 00:30:20.894090 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/d4b4eb0f-539c-4193-907e-cbff010aae7b-nginx-conf\") pod \"networking-console-plugin-6cf856758c-45x56\" (UID: \"d4b4eb0f-539c-4193-907e-cbff010aae7b\") " pod="openshift-network-console/networking-console-plugin-6cf856758c-45x56" Jul 10 00:30:20.894146 ip-10-0-130-146 kubenswrapper[2570]: I0710 00:30:20.894135 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/57c018ee-b976-4f56-a33b-b41cb28c4417-installation-pull-secrets\") pod \"image-registry-b9fdbf76c-kptvw\" (UID: \"57c018ee-b976-4f56-a33b-b41cb28c4417\") " pod="openshift-image-registry/image-registry-b9fdbf76c-kptvw" Jul 10 00:30:20.894411 ip-10-0-130-146 kubenswrapper[2570]: I0710 00:30:20.894162 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qpsq9\" (UniqueName: \"kubernetes.io/projected/57c018ee-b976-4f56-a33b-b41cb28c4417-kube-api-access-qpsq9\") pod \"image-registry-b9fdbf76c-kptvw\" (UID: \"57c018ee-b976-4f56-a33b-b41cb28c4417\") " pod="openshift-image-registry/image-registry-b9fdbf76c-kptvw" Jul 10 00:30:20.894411 ip-10-0-130-146 kubenswrapper[2570]: I0710 00:30:20.894192 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/6f87cbd8-9d72-4708-a230-2818fa5a836d-serving-cert\") pod \"kube-storage-version-migrator-operator-799cb7f958-p44ms\" (UID: \"6f87cbd8-9d72-4708-a230-2818fa5a836d\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-799cb7f958-p44ms" Jul 10 00:30:20.894411 ip-10-0-130-146 kubenswrapper[2570]: I0710 00:30:20.894283 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/f612c430-f495-4773-9ca6-82f7417e3a0c-tmp\") pod \"insights-operator-68d9dd94cf-7rq8g\" (UID: \"f612c430-f495-4773-9ca6-82f7417e3a0c\") " pod="openshift-insights/insights-operator-68d9dd94cf-7rq8g" Jul 10 00:30:20.894411 ip-10-0-130-146 kubenswrapper[2570]: I0710 00:30:20.894327 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l5tsr\" (UniqueName: \"kubernetes.io/projected/2b6cb4ca-79ff-42f0-b080-66af842f22d3-kube-api-access-l5tsr\") pod \"console-operator-7866497494-d6fcd\" (UID: \"2b6cb4ca-79ff-42f0-b080-66af842f22d3\") " pod="openshift-console-operator/console-operator-7866497494-d6fcd" Jul 10 00:30:20.894411 ip-10-0-130-146 kubenswrapper[2570]: I0710 00:30:20.894361 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/57c018ee-b976-4f56-a33b-b41cb28c4417-image-registry-private-configuration\") pod \"image-registry-b9fdbf76c-kptvw\" (UID: \"57c018ee-b976-4f56-a33b-b41cb28c4417\") " pod="openshift-image-registry/image-registry-b9fdbf76c-kptvw" Jul 10 00:30:20.894411 ip-10-0-130-146 kubenswrapper[2570]: I0710 00:30:20.894402 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"snapshots\" (UniqueName: \"kubernetes.io/empty-dir/f612c430-f495-4773-9ca6-82f7417e3a0c-snapshots\") pod \"insights-operator-68d9dd94cf-7rq8g\" (UID: \"f612c430-f495-4773-9ca6-82f7417e3a0c\") " pod="openshift-insights/insights-operator-68d9dd94cf-7rq8g" Jul 10 00:30:20.894720 ip-10-0-130-146 kubenswrapper[2570]: I0710 00:30:20.894426 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/57c018ee-b976-4f56-a33b-b41cb28c4417-registry-certificates\") pod \"image-registry-b9fdbf76c-kptvw\" (UID: \"57c018ee-b976-4f56-a33b-b41cb28c4417\") " pod="openshift-image-registry/image-registry-b9fdbf76c-kptvw" Jul 10 00:30:20.894720 ip-10-0-130-146 kubenswrapper[2570]: I0710 00:30:20.894452 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/57c018ee-b976-4f56-a33b-b41cb28c4417-trusted-ca\") pod \"image-registry-b9fdbf76c-kptvw\" (UID: \"57c018ee-b976-4f56-a33b-b41cb28c4417\") " pod="openshift-image-registry/image-registry-b9fdbf76c-kptvw" Jul 10 00:30:20.894720 ip-10-0-130-146 kubenswrapper[2570]: I0710 00:30:20.894476 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/57c018ee-b976-4f56-a33b-b41cb28c4417-ca-trust-extracted\") pod \"image-registry-b9fdbf76c-kptvw\" (UID: \"57c018ee-b976-4f56-a33b-b41cb28c4417\") " pod="openshift-image-registry/image-registry-b9fdbf76c-kptvw" Jul 10 00:30:20.894720 ip-10-0-130-146 kubenswrapper[2570]: I0710 00:30:20.894503 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f612c430-f495-4773-9ca6-82f7417e3a0c-trusted-ca-bundle\") pod \"insights-operator-68d9dd94cf-7rq8g\" (UID: \"f612c430-f495-4773-9ca6-82f7417e3a0c\") " pod="openshift-insights/insights-operator-68d9dd94cf-7rq8g" Jul 10 00:30:20.894720 ip-10-0-130-146 kubenswrapper[2570]: I0710 00:30:20.894527 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5rx5k\" (UniqueName: \"kubernetes.io/projected/f612c430-f495-4773-9ca6-82f7417e3a0c-kube-api-access-5rx5k\") pod \"insights-operator-68d9dd94cf-7rq8g\" (UID: \"f612c430-f495-4773-9ca6-82f7417e3a0c\") " pod="openshift-insights/insights-operator-68d9dd94cf-7rq8g" Jul 10 00:30:20.894720 ip-10-0-130-146 kubenswrapper[2570]: I0710 00:30:20.894550 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/4e3a154b-8ba7-4c18-b75b-93444d0de385-samples-operator-tls\") pod \"cluster-samples-operator-65db858894-bzzfq\" (UID: \"4e3a154b-8ba7-4c18-b75b-93444d0de385\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-65db858894-bzzfq" Jul 10 00:30:20.894720 ip-10-0-130-146 kubenswrapper[2570]: I0710 00:30:20.894582 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/2b6cb4ca-79ff-42f0-b080-66af842f22d3-config\") pod \"console-operator-7866497494-d6fcd\" (UID: \"2b6cb4ca-79ff-42f0-b080-66af842f22d3\") " pod="openshift-console-operator/console-operator-7866497494-d6fcd" Jul 10 00:30:20.894720 ip-10-0-130-146 kubenswrapper[2570]: I0710 00:30:20.894604 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/2b6cb4ca-79ff-42f0-b080-66af842f22d3-serving-cert\") pod \"console-operator-7866497494-d6fcd\" (UID: \"2b6cb4ca-79ff-42f0-b080-66af842f22d3\") " pod="openshift-console-operator/console-operator-7866497494-d6fcd" Jul 10 00:30:20.894720 ip-10-0-130-146 kubenswrapper[2570]: I0710 00:30:20.894631 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/6f87cbd8-9d72-4708-a230-2818fa5a836d-config\") pod \"kube-storage-version-migrator-operator-799cb7f958-p44ms\" (UID: \"6f87cbd8-9d72-4708-a230-2818fa5a836d\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-799cb7f958-p44ms" Jul 10 00:30:20.894720 ip-10-0-130-146 kubenswrapper[2570]: I0710 00:30:20.894657 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/57c018ee-b976-4f56-a33b-b41cb28c4417-bound-sa-token\") pod \"image-registry-b9fdbf76c-kptvw\" (UID: \"57c018ee-b976-4f56-a33b-b41cb28c4417\") " pod="openshift-image-registry/image-registry-b9fdbf76c-kptvw" Jul 10 00:30:20.894720 ip-10-0-130-146 kubenswrapper[2570]: I0710 00:30:20.894681 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f612c430-f495-4773-9ca6-82f7417e3a0c-service-ca-bundle\") pod \"insights-operator-68d9dd94cf-7rq8g\" (UID: \"f612c430-f495-4773-9ca6-82f7417e3a0c\") " pod="openshift-insights/insights-operator-68d9dd94cf-7rq8g" Jul 10 00:30:20.894720 ip-10-0-130-146 kubenswrapper[2570]: I0710 00:30:20.894705 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/f612c430-f495-4773-9ca6-82f7417e3a0c-serving-cert\") pod \"insights-operator-68d9dd94cf-7rq8g\" (UID: \"f612c430-f495-4773-9ca6-82f7417e3a0c\") " pod="openshift-insights/insights-operator-68d9dd94cf-7rq8g" Jul 10 00:30:20.895344 ip-10-0-130-146 kubenswrapper[2570]: I0710 00:30:20.894741 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/57c018ee-b976-4f56-a33b-b41cb28c4417-registry-tls\") pod \"image-registry-b9fdbf76c-kptvw\" (UID: \"57c018ee-b976-4f56-a33b-b41cb28c4417\") " pod="openshift-image-registry/image-registry-b9fdbf76c-kptvw" Jul 10 00:30:20.895344 ip-10-0-130-146 kubenswrapper[2570]: I0710 00:30:20.894778 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/d4b4eb0f-539c-4193-907e-cbff010aae7b-networking-console-plugin-cert\") pod \"networking-console-plugin-6cf856758c-45x56\" (UID: \"d4b4eb0f-539c-4193-907e-cbff010aae7b\") " pod="openshift-network-console/networking-console-plugin-6cf856758c-45x56" Jul 10 00:30:20.895344 ip-10-0-130-146 kubenswrapper[2570]: E0710 00:30:20.894903 2570 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found Jul 10 00:30:20.895344 ip-10-0-130-146 kubenswrapper[2570]: E0710 00:30:20.894952 2570 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d4b4eb0f-539c-4193-907e-cbff010aae7b-networking-console-plugin-cert podName:d4b4eb0f-539c-4193-907e-cbff010aae7b nodeName:}" failed. No retries permitted until 2026-07-10 00:30:21.394937999 +0000 UTC m=+240.443316712 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/d4b4eb0f-539c-4193-907e-cbff010aae7b-networking-console-plugin-cert") pod "networking-console-plugin-6cf856758c-45x56" (UID: "d4b4eb0f-539c-4193-907e-cbff010aae7b") : secret "networking-console-plugin-cert" not found Jul 10 00:30:20.895344 ip-10-0-130-146 kubenswrapper[2570]: I0710 00:30:20.894998 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/d4b4eb0f-539c-4193-907e-cbff010aae7b-nginx-conf\") pod \"networking-console-plugin-6cf856758c-45x56\" (UID: \"d4b4eb0f-539c-4193-907e-cbff010aae7b\") " pod="openshift-network-console/networking-console-plugin-6cf856758c-45x56" Jul 10 00:30:20.895344 ip-10-0-130-146 kubenswrapper[2570]: I0710 00:30:20.895009 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/2b6cb4ca-79ff-42f0-b080-66af842f22d3-trusted-ca\") pod \"console-operator-7866497494-d6fcd\" (UID: \"2b6cb4ca-79ff-42f0-b080-66af842f22d3\") " pod="openshift-console-operator/console-operator-7866497494-d6fcd" Jul 10 00:30:20.895344 ip-10-0-130-146 kubenswrapper[2570]: E0710 00:30:20.895143 2570 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Jul 10 00:30:20.895344 ip-10-0-130-146 kubenswrapper[2570]: E0710 00:30:20.895183 2570 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-b9fdbf76c-kptvw: secret "image-registry-tls" not found Jul 10 00:30:20.895344 ip-10-0-130-146 kubenswrapper[2570]: E0710 00:30:20.895241 2570 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/57c018ee-b976-4f56-a33b-b41cb28c4417-registry-tls podName:57c018ee-b976-4f56-a33b-b41cb28c4417 nodeName:}" failed. No retries permitted until 2026-07-10 00:30:21.395223732 +0000 UTC m=+240.443602446 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/57c018ee-b976-4f56-a33b-b41cb28c4417-registry-tls") pod "image-registry-b9fdbf76c-kptvw" (UID: "57c018ee-b976-4f56-a33b-b41cb28c4417") : secret "image-registry-tls" not found Jul 10 00:30:20.895859 ip-10-0-130-146 kubenswrapper[2570]: I0710 00:30:20.895451 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/2b6cb4ca-79ff-42f0-b080-66af842f22d3-config\") pod \"console-operator-7866497494-d6fcd\" (UID: \"2b6cb4ca-79ff-42f0-b080-66af842f22d3\") " pod="openshift-console-operator/console-operator-7866497494-d6fcd" Jul 10 00:30:20.895859 ip-10-0-130-146 kubenswrapper[2570]: E0710 00:30:20.895539 2570 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found Jul 10 00:30:20.895859 ip-10-0-130-146 kubenswrapper[2570]: E0710 00:30:20.895590 2570 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4e3a154b-8ba7-4c18-b75b-93444d0de385-samples-operator-tls podName:4e3a154b-8ba7-4c18-b75b-93444d0de385 nodeName:}" failed. No retries permitted until 2026-07-10 00:30:21.39557671 +0000 UTC m=+240.443955429 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/4e3a154b-8ba7-4c18-b75b-93444d0de385-samples-operator-tls") pod "cluster-samples-operator-65db858894-bzzfq" (UID: "4e3a154b-8ba7-4c18-b75b-93444d0de385") : secret "samples-operator-tls" not found Jul 10 00:30:20.895859 ip-10-0-130-146 kubenswrapper[2570]: I0710 00:30:20.895663 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/57c018ee-b976-4f56-a33b-b41cb28c4417-registry-certificates\") pod \"image-registry-b9fdbf76c-kptvw\" (UID: \"57c018ee-b976-4f56-a33b-b41cb28c4417\") " pod="openshift-image-registry/image-registry-b9fdbf76c-kptvw" Jul 10 00:30:20.896081 ip-10-0-130-146 kubenswrapper[2570]: I0710 00:30:20.895977 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f612c430-f495-4773-9ca6-82f7417e3a0c-trusted-ca-bundle\") pod \"insights-operator-68d9dd94cf-7rq8g\" (UID: \"f612c430-f495-4773-9ca6-82f7417e3a0c\") " pod="openshift-insights/insights-operator-68d9dd94cf-7rq8g" Jul 10 00:30:20.896173 ip-10-0-130-146 kubenswrapper[2570]: I0710 00:30:20.896152 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f612c430-f495-4773-9ca6-82f7417e3a0c-service-ca-bundle\") pod \"insights-operator-68d9dd94cf-7rq8g\" (UID: \"f612c430-f495-4773-9ca6-82f7417e3a0c\") " pod="openshift-insights/insights-operator-68d9dd94cf-7rq8g" Jul 10 00:30:20.896333 ip-10-0-130-146 kubenswrapper[2570]: I0710 00:30:20.896285 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/57c018ee-b976-4f56-a33b-b41cb28c4417-ca-trust-extracted\") pod \"image-registry-b9fdbf76c-kptvw\" (UID: \"57c018ee-b976-4f56-a33b-b41cb28c4417\") " pod="openshift-image-registry/image-registry-b9fdbf76c-kptvw" Jul 10 00:30:20.896410 ip-10-0-130-146 kubenswrapper[2570]: I0710 00:30:20.896332 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"snapshots\" (UniqueName: \"kubernetes.io/empty-dir/f612c430-f495-4773-9ca6-82f7417e3a0c-snapshots\") pod \"insights-operator-68d9dd94cf-7rq8g\" (UID: \"f612c430-f495-4773-9ca6-82f7417e3a0c\") " pod="openshift-insights/insights-operator-68d9dd94cf-7rq8g" Jul 10 00:30:20.896410 ip-10-0-130-146 kubenswrapper[2570]: I0710 00:30:20.896345 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/6f87cbd8-9d72-4708-a230-2818fa5a836d-config\") pod \"kube-storage-version-migrator-operator-799cb7f958-p44ms\" (UID: \"6f87cbd8-9d72-4708-a230-2818fa5a836d\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-799cb7f958-p44ms" Jul 10 00:30:20.896595 ip-10-0-130-146 kubenswrapper[2570]: I0710 00:30:20.896545 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/57c018ee-b976-4f56-a33b-b41cb28c4417-trusted-ca\") pod \"image-registry-b9fdbf76c-kptvw\" (UID: \"57c018ee-b976-4f56-a33b-b41cb28c4417\") " pod="openshift-image-registry/image-registry-b9fdbf76c-kptvw" Jul 10 00:30:20.897037 ip-10-0-130-146 kubenswrapper[2570]: I0710 00:30:20.897011 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/6f87cbd8-9d72-4708-a230-2818fa5a836d-serving-cert\") pod \"kube-storage-version-migrator-operator-799cb7f958-p44ms\" (UID: \"6f87cbd8-9d72-4708-a230-2818fa5a836d\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-799cb7f958-p44ms" Jul 10 00:30:20.897144 ip-10-0-130-146 kubenswrapper[2570]: I0710 00:30:20.897016 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/57c018ee-b976-4f56-a33b-b41cb28c4417-installation-pull-secrets\") pod \"image-registry-b9fdbf76c-kptvw\" (UID: \"57c018ee-b976-4f56-a33b-b41cb28c4417\") " pod="openshift-image-registry/image-registry-b9fdbf76c-kptvw" Jul 10 00:30:20.897940 ip-10-0-130-146 kubenswrapper[2570]: I0710 00:30:20.897921 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/2b6cb4ca-79ff-42f0-b080-66af842f22d3-serving-cert\") pod \"console-operator-7866497494-d6fcd\" (UID: \"2b6cb4ca-79ff-42f0-b080-66af842f22d3\") " pod="openshift-console-operator/console-operator-7866497494-d6fcd" Jul 10 00:30:20.898111 ip-10-0-130-146 kubenswrapper[2570]: I0710 00:30:20.898095 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/f612c430-f495-4773-9ca6-82f7417e3a0c-serving-cert\") pod \"insights-operator-68d9dd94cf-7rq8g\" (UID: \"f612c430-f495-4773-9ca6-82f7417e3a0c\") " pod="openshift-insights/insights-operator-68d9dd94cf-7rq8g" Jul 10 00:30:20.898255 ip-10-0-130-146 kubenswrapper[2570]: I0710 00:30:20.898236 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/57c018ee-b976-4f56-a33b-b41cb28c4417-image-registry-private-configuration\") pod \"image-registry-b9fdbf76c-kptvw\" (UID: \"57c018ee-b976-4f56-a33b-b41cb28c4417\") " pod="openshift-image-registry/image-registry-b9fdbf76c-kptvw" Jul 10 00:30:20.901542 ip-10-0-130-146 kubenswrapper[2570]: I0710 00:30:20.901522 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qpsq9\" (UniqueName: \"kubernetes.io/projected/57c018ee-b976-4f56-a33b-b41cb28c4417-kube-api-access-qpsq9\") pod \"image-registry-b9fdbf76c-kptvw\" (UID: \"57c018ee-b976-4f56-a33b-b41cb28c4417\") " pod="openshift-image-registry/image-registry-b9fdbf76c-kptvw" Jul 10 00:30:20.901839 ip-10-0-130-146 kubenswrapper[2570]: I0710 00:30:20.901814 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jgg4z\" (UniqueName: \"kubernetes.io/projected/4e3a154b-8ba7-4c18-b75b-93444d0de385-kube-api-access-jgg4z\") pod \"cluster-samples-operator-65db858894-bzzfq\" (UID: \"4e3a154b-8ba7-4c18-b75b-93444d0de385\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-65db858894-bzzfq" Jul 10 00:30:20.902744 ip-10-0-130-146 kubenswrapper[2570]: I0710 00:30:20.902715 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-86mjf\" (UniqueName: \"kubernetes.io/projected/6f87cbd8-9d72-4708-a230-2818fa5a836d-kube-api-access-86mjf\") pod \"kube-storage-version-migrator-operator-799cb7f958-p44ms\" (UID: \"6f87cbd8-9d72-4708-a230-2818fa5a836d\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-799cb7f958-p44ms" Jul 10 00:30:20.903430 ip-10-0-130-146 kubenswrapper[2570]: I0710 00:30:20.903407 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5rx5k\" (UniqueName: \"kubernetes.io/projected/f612c430-f495-4773-9ca6-82f7417e3a0c-kube-api-access-5rx5k\") pod \"insights-operator-68d9dd94cf-7rq8g\" (UID: \"f612c430-f495-4773-9ca6-82f7417e3a0c\") " pod="openshift-insights/insights-operator-68d9dd94cf-7rq8g" Jul 10 00:30:20.903562 ip-10-0-130-146 kubenswrapper[2570]: I0710 00:30:20.903547 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-l5tsr\" (UniqueName: \"kubernetes.io/projected/2b6cb4ca-79ff-42f0-b080-66af842f22d3-kube-api-access-l5tsr\") pod \"console-operator-7866497494-d6fcd\" (UID: \"2b6cb4ca-79ff-42f0-b080-66af842f22d3\") " pod="openshift-console-operator/console-operator-7866497494-d6fcd" Jul 10 00:30:20.903748 ip-10-0-130-146 kubenswrapper[2570]: I0710 00:30:20.903729 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/57c018ee-b976-4f56-a33b-b41cb28c4417-bound-sa-token\") pod \"image-registry-b9fdbf76c-kptvw\" (UID: \"57c018ee-b976-4f56-a33b-b41cb28c4417\") " pod="openshift-image-registry/image-registry-b9fdbf76c-kptvw" Jul 10 00:30:20.949725 ip-10-0-130-146 kubenswrapper[2570]: I0710 00:30:20.949702 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-source-869dcfcdcd-4vnvp" Jul 10 00:30:20.970471 ip-10-0-130-146 kubenswrapper[2570]: I0710 00:30:20.970448 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console-operator/console-operator-7866497494-d6fcd" Jul 10 00:30:20.975417 ip-10-0-130-146 kubenswrapper[2570]: I0710 00:30:20.975392 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-operator-68d9dd94cf-7rq8g" Jul 10 00:30:21.048502 ip-10-0-130-146 kubenswrapper[2570]: I0710 00:30:21.048287 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-799cb7f958-p44ms" Jul 10 00:30:21.095249 ip-10-0-130-146 kubenswrapper[2570]: I0710 00:30:21.095165 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-source-869dcfcdcd-4vnvp"] Jul 10 00:30:21.101228 ip-10-0-130-146 kubenswrapper[2570]: W0710 00:30:21.100442 2570 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod28eaf961_fb7d_495a_8733_abd273807b37.slice/crio-396d5ef73158fefde119070978429815f65a297c76b337f62281bfb688b30336 WatchSource:0}: Error finding container 396d5ef73158fefde119070978429815f65a297c76b337f62281bfb688b30336: Status 404 returned error can't find the container with id 396d5ef73158fefde119070978429815f65a297c76b337f62281bfb688b30336 Jul 10 00:30:21.110314 ip-10-0-130-146 kubenswrapper[2570]: I0710 00:30:21.110259 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console-operator/console-operator-7866497494-d6fcd"] Jul 10 00:30:21.130814 ip-10-0-130-146 kubenswrapper[2570]: I0710 00:30:21.130792 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-operator-68d9dd94cf-7rq8g"] Jul 10 00:30:21.133086 ip-10-0-130-146 kubenswrapper[2570]: W0710 00:30:21.133040 2570 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf612c430_f495_4773_9ca6_82f7417e3a0c.slice/crio-6a0cc285b3fda61219cc20465305bc3a88abdeaff06dd6a59500d160f31acfe8 WatchSource:0}: Error finding container 6a0cc285b3fda61219cc20465305bc3a88abdeaff06dd6a59500d160f31acfe8: Status 404 returned error can't find the container with id 6a0cc285b3fda61219cc20465305bc3a88abdeaff06dd6a59500d160f31acfe8 Jul 10 00:30:21.180901 ip-10-0-130-146 kubenswrapper[2570]: I0710 00:30:21.180877 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-799cb7f958-p44ms"] Jul 10 00:30:21.183556 ip-10-0-130-146 kubenswrapper[2570]: W0710 00:30:21.183531 2570 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6f87cbd8_9d72_4708_a230_2818fa5a836d.slice/crio-381c72128eaa17be99ce4024fc3ffaa45960f41239c27308005a196019e3da8f WatchSource:0}: Error finding container 381c72128eaa17be99ce4024fc3ffaa45960f41239c27308005a196019e3da8f: Status 404 returned error can't find the container with id 381c72128eaa17be99ce4024fc3ffaa45960f41239c27308005a196019e3da8f Jul 10 00:30:21.227939 ip-10-0-130-146 kubenswrapper[2570]: I0710 00:30:21.227903 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-799cb7f958-p44ms" event={"ID":"6f87cbd8-9d72-4708-a230-2818fa5a836d","Type":"ContainerStarted","Data":"381c72128eaa17be99ce4024fc3ffaa45960f41239c27308005a196019e3da8f"} Jul 10 00:30:21.228909 ip-10-0-130-146 kubenswrapper[2570]: I0710 00:30:21.228876 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-68d9dd94cf-7rq8g" event={"ID":"f612c430-f495-4773-9ca6-82f7417e3a0c","Type":"ContainerStarted","Data":"6a0cc285b3fda61219cc20465305bc3a88abdeaff06dd6a59500d160f31acfe8"} Jul 10 00:30:21.229876 ip-10-0-130-146 kubenswrapper[2570]: I0710 00:30:21.229845 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-7866497494-d6fcd" event={"ID":"2b6cb4ca-79ff-42f0-b080-66af842f22d3","Type":"ContainerStarted","Data":"1e167301f60d0378e4d2f09d565a98c0cd7869fa1c1c98f29bd05d27a4aa3764"} Jul 10 00:30:21.231209 ip-10-0-130-146 kubenswrapper[2570]: I0710 00:30:21.231188 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-source-869dcfcdcd-4vnvp" event={"ID":"28eaf961-fb7d-495a-8733-abd273807b37","Type":"ContainerStarted","Data":"227ee58eae318e2754d2f1733442743ff03434d968586fe5f24f7766908eb732"} Jul 10 00:30:21.231299 ip-10-0-130-146 kubenswrapper[2570]: I0710 00:30:21.231220 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-source-869dcfcdcd-4vnvp" event={"ID":"28eaf961-fb7d-495a-8733-abd273807b37","Type":"ContainerStarted","Data":"396d5ef73158fefde119070978429815f65a297c76b337f62281bfb688b30336"} Jul 10 00:30:21.245999 ip-10-0-130-146 kubenswrapper[2570]: I0710 00:30:21.245961 2570 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-source-869dcfcdcd-4vnvp" podStartSLOduration=1.2459495 podStartE2EDuration="1.2459495s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-10 00:30:20 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 00:30:21.244887473 +0000 UTC m=+240.293266191" watchObservedRunningTime="2026-07-10 00:30:21.2459495 +0000 UTC m=+240.294328220" Jul 10 00:30:21.399248 ip-10-0-130-146 kubenswrapper[2570]: I0710 00:30:21.399217 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/4e3a154b-8ba7-4c18-b75b-93444d0de385-samples-operator-tls\") pod \"cluster-samples-operator-65db858894-bzzfq\" (UID: \"4e3a154b-8ba7-4c18-b75b-93444d0de385\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-65db858894-bzzfq" Jul 10 00:30:21.399414 ip-10-0-130-146 kubenswrapper[2570]: I0710 00:30:21.399260 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/57c018ee-b976-4f56-a33b-b41cb28c4417-registry-tls\") pod \"image-registry-b9fdbf76c-kptvw\" (UID: \"57c018ee-b976-4f56-a33b-b41cb28c4417\") " pod="openshift-image-registry/image-registry-b9fdbf76c-kptvw" Jul 10 00:30:21.399414 ip-10-0-130-146 kubenswrapper[2570]: I0710 00:30:21.399283 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/d4b4eb0f-539c-4193-907e-cbff010aae7b-networking-console-plugin-cert\") pod \"networking-console-plugin-6cf856758c-45x56\" (UID: \"d4b4eb0f-539c-4193-907e-cbff010aae7b\") " pod="openshift-network-console/networking-console-plugin-6cf856758c-45x56" Jul 10 00:30:21.399414 ip-10-0-130-146 kubenswrapper[2570]: E0710 00:30:21.399378 2570 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Jul 10 00:30:21.399414 ip-10-0-130-146 kubenswrapper[2570]: E0710 00:30:21.399385 2570 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found Jul 10 00:30:21.399414 ip-10-0-130-146 kubenswrapper[2570]: E0710 00:30:21.399400 2570 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-b9fdbf76c-kptvw: secret "image-registry-tls" not found Jul 10 00:30:21.399571 ip-10-0-130-146 kubenswrapper[2570]: E0710 00:30:21.399381 2570 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found Jul 10 00:30:21.399571 ip-10-0-130-146 kubenswrapper[2570]: E0710 00:30:21.399448 2570 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/57c018ee-b976-4f56-a33b-b41cb28c4417-registry-tls podName:57c018ee-b976-4f56-a33b-b41cb28c4417 nodeName:}" failed. No retries permitted until 2026-07-10 00:30:22.399433137 +0000 UTC m=+241.447811840 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/57c018ee-b976-4f56-a33b-b41cb28c4417-registry-tls") pod "image-registry-b9fdbf76c-kptvw" (UID: "57c018ee-b976-4f56-a33b-b41cb28c4417") : secret "image-registry-tls" not found Jul 10 00:30:21.399571 ip-10-0-130-146 kubenswrapper[2570]: E0710 00:30:21.399460 2570 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4e3a154b-8ba7-4c18-b75b-93444d0de385-samples-operator-tls podName:4e3a154b-8ba7-4c18-b75b-93444d0de385 nodeName:}" failed. No retries permitted until 2026-07-10 00:30:22.399454659 +0000 UTC m=+241.447833358 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/4e3a154b-8ba7-4c18-b75b-93444d0de385-samples-operator-tls") pod "cluster-samples-operator-65db858894-bzzfq" (UID: "4e3a154b-8ba7-4c18-b75b-93444d0de385") : secret "samples-operator-tls" not found Jul 10 00:30:21.399571 ip-10-0-130-146 kubenswrapper[2570]: E0710 00:30:21.399481 2570 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d4b4eb0f-539c-4193-907e-cbff010aae7b-networking-console-plugin-cert podName:d4b4eb0f-539c-4193-907e-cbff010aae7b nodeName:}" failed. No retries permitted until 2026-07-10 00:30:22.399464429 +0000 UTC m=+241.447843131 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/d4b4eb0f-539c-4193-907e-cbff010aae7b-networking-console-plugin-cert") pod "networking-console-plugin-6cf856758c-45x56" (UID: "d4b4eb0f-539c-4193-907e-cbff010aae7b") : secret "networking-console-plugin-cert" not found Jul 10 00:30:22.407392 ip-10-0-130-146 kubenswrapper[2570]: I0710 00:30:22.407360 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/4e3a154b-8ba7-4c18-b75b-93444d0de385-samples-operator-tls\") pod \"cluster-samples-operator-65db858894-bzzfq\" (UID: \"4e3a154b-8ba7-4c18-b75b-93444d0de385\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-65db858894-bzzfq" Jul 10 00:30:22.407825 ip-10-0-130-146 kubenswrapper[2570]: I0710 00:30:22.407418 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/57c018ee-b976-4f56-a33b-b41cb28c4417-registry-tls\") pod \"image-registry-b9fdbf76c-kptvw\" (UID: \"57c018ee-b976-4f56-a33b-b41cb28c4417\") " pod="openshift-image-registry/image-registry-b9fdbf76c-kptvw" Jul 10 00:30:22.407825 ip-10-0-130-146 kubenswrapper[2570]: I0710 00:30:22.407450 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/d4b4eb0f-539c-4193-907e-cbff010aae7b-networking-console-plugin-cert\") pod \"networking-console-plugin-6cf856758c-45x56\" (UID: \"d4b4eb0f-539c-4193-907e-cbff010aae7b\") " pod="openshift-network-console/networking-console-plugin-6cf856758c-45x56" Jul 10 00:30:22.407825 ip-10-0-130-146 kubenswrapper[2570]: E0710 00:30:22.407569 2570 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found Jul 10 00:30:22.407825 ip-10-0-130-146 kubenswrapper[2570]: E0710 00:30:22.407629 2570 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d4b4eb0f-539c-4193-907e-cbff010aae7b-networking-console-plugin-cert podName:d4b4eb0f-539c-4193-907e-cbff010aae7b nodeName:}" failed. No retries permitted until 2026-07-10 00:30:24.407611209 +0000 UTC m=+243.455989909 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/d4b4eb0f-539c-4193-907e-cbff010aae7b-networking-console-plugin-cert") pod "networking-console-plugin-6cf856758c-45x56" (UID: "d4b4eb0f-539c-4193-907e-cbff010aae7b") : secret "networking-console-plugin-cert" not found Jul 10 00:30:22.408073 ip-10-0-130-146 kubenswrapper[2570]: E0710 00:30:22.407934 2570 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found Jul 10 00:30:22.408073 ip-10-0-130-146 kubenswrapper[2570]: E0710 00:30:22.407971 2570 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4e3a154b-8ba7-4c18-b75b-93444d0de385-samples-operator-tls podName:4e3a154b-8ba7-4c18-b75b-93444d0de385 nodeName:}" failed. No retries permitted until 2026-07-10 00:30:24.407960958 +0000 UTC m=+243.456339657 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/4e3a154b-8ba7-4c18-b75b-93444d0de385-samples-operator-tls") pod "cluster-samples-operator-65db858894-bzzfq" (UID: "4e3a154b-8ba7-4c18-b75b-93444d0de385") : secret "samples-operator-tls" not found Jul 10 00:30:22.408073 ip-10-0-130-146 kubenswrapper[2570]: E0710 00:30:22.408030 2570 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Jul 10 00:30:22.408073 ip-10-0-130-146 kubenswrapper[2570]: E0710 00:30:22.408041 2570 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-b9fdbf76c-kptvw: secret "image-registry-tls" not found Jul 10 00:30:22.408277 ip-10-0-130-146 kubenswrapper[2570]: E0710 00:30:22.408096 2570 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/57c018ee-b976-4f56-a33b-b41cb28c4417-registry-tls podName:57c018ee-b976-4f56-a33b-b41cb28c4417 nodeName:}" failed. No retries permitted until 2026-07-10 00:30:24.40808361 +0000 UTC m=+243.456462308 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/57c018ee-b976-4f56-a33b-b41cb28c4417-registry-tls") pod "image-registry-b9fdbf76c-kptvw" (UID: "57c018ee-b976-4f56-a33b-b41cb28c4417") : secret "image-registry-tls" not found Jul 10 00:30:24.423998 ip-10-0-130-146 kubenswrapper[2570]: I0710 00:30:24.423968 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/d4b4eb0f-539c-4193-907e-cbff010aae7b-networking-console-plugin-cert\") pod \"networking-console-plugin-6cf856758c-45x56\" (UID: \"d4b4eb0f-539c-4193-907e-cbff010aae7b\") " pod="openshift-network-console/networking-console-plugin-6cf856758c-45x56" Jul 10 00:30:24.424387 ip-10-0-130-146 kubenswrapper[2570]: I0710 00:30:24.424089 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/4e3a154b-8ba7-4c18-b75b-93444d0de385-samples-operator-tls\") pod \"cluster-samples-operator-65db858894-bzzfq\" (UID: \"4e3a154b-8ba7-4c18-b75b-93444d0de385\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-65db858894-bzzfq" Jul 10 00:30:24.424387 ip-10-0-130-146 kubenswrapper[2570]: E0710 00:30:24.424123 2570 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found Jul 10 00:30:24.424387 ip-10-0-130-146 kubenswrapper[2570]: I0710 00:30:24.424145 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/57c018ee-b976-4f56-a33b-b41cb28c4417-registry-tls\") pod \"image-registry-b9fdbf76c-kptvw\" (UID: \"57c018ee-b976-4f56-a33b-b41cb28c4417\") " pod="openshift-image-registry/image-registry-b9fdbf76c-kptvw" Jul 10 00:30:24.424387 ip-10-0-130-146 kubenswrapper[2570]: E0710 00:30:24.424186 2570 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d4b4eb0f-539c-4193-907e-cbff010aae7b-networking-console-plugin-cert podName:d4b4eb0f-539c-4193-907e-cbff010aae7b nodeName:}" failed. No retries permitted until 2026-07-10 00:30:28.424169254 +0000 UTC m=+247.472547971 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/d4b4eb0f-539c-4193-907e-cbff010aae7b-networking-console-plugin-cert") pod "networking-console-plugin-6cf856758c-45x56" (UID: "d4b4eb0f-539c-4193-907e-cbff010aae7b") : secret "networking-console-plugin-cert" not found Jul 10 00:30:24.424387 ip-10-0-130-146 kubenswrapper[2570]: E0710 00:30:24.424240 2570 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found Jul 10 00:30:24.424387 ip-10-0-130-146 kubenswrapper[2570]: E0710 00:30:24.424253 2570 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Jul 10 00:30:24.424387 ip-10-0-130-146 kubenswrapper[2570]: E0710 00:30:24.424267 2570 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-b9fdbf76c-kptvw: secret "image-registry-tls" not found Jul 10 00:30:24.424387 ip-10-0-130-146 kubenswrapper[2570]: E0710 00:30:24.424305 2570 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4e3a154b-8ba7-4c18-b75b-93444d0de385-samples-operator-tls podName:4e3a154b-8ba7-4c18-b75b-93444d0de385 nodeName:}" failed. No retries permitted until 2026-07-10 00:30:28.424288855 +0000 UTC m=+247.472667557 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/4e3a154b-8ba7-4c18-b75b-93444d0de385-samples-operator-tls") pod "cluster-samples-operator-65db858894-bzzfq" (UID: "4e3a154b-8ba7-4c18-b75b-93444d0de385") : secret "samples-operator-tls" not found Jul 10 00:30:24.424387 ip-10-0-130-146 kubenswrapper[2570]: E0710 00:30:24.424322 2570 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/57c018ee-b976-4f56-a33b-b41cb28c4417-registry-tls podName:57c018ee-b976-4f56-a33b-b41cb28c4417 nodeName:}" failed. No retries permitted until 2026-07-10 00:30:28.424312424 +0000 UTC m=+247.472691129 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/57c018ee-b976-4f56-a33b-b41cb28c4417-registry-tls") pod "image-registry-b9fdbf76c-kptvw" (UID: "57c018ee-b976-4f56-a33b-b41cb28c4417") : secret "image-registry-tls" not found Jul 10 00:30:25.246991 ip-10-0-130-146 kubenswrapper[2570]: I0710 00:30:25.246955 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-799cb7f958-p44ms" event={"ID":"6f87cbd8-9d72-4708-a230-2818fa5a836d","Type":"ContainerStarted","Data":"e13fc7802c777f81a25e1d8dba4ec33a9812570cf4b99ab56537eedeb360ceb7"} Jul 10 00:30:25.248511 ip-10-0-130-146 kubenswrapper[2570]: I0710 00:30:25.248483 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-68d9dd94cf-7rq8g" event={"ID":"f612c430-f495-4773-9ca6-82f7417e3a0c","Type":"ContainerStarted","Data":"ab2af976c38ea3ebd24a288184d7937ee4fefc1368113b6f9cb5b19e1f981cb1"} Jul 10 00:30:25.250019 ip-10-0-130-146 kubenswrapper[2570]: I0710 00:30:25.250000 2570 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7866497494-d6fcd_2b6cb4ca-79ff-42f0-b080-66af842f22d3/console-operator/0.log" Jul 10 00:30:25.250154 ip-10-0-130-146 kubenswrapper[2570]: I0710 00:30:25.250035 2570 generic.go:356] "Generic (PLEG): container finished" podID="2b6cb4ca-79ff-42f0-b080-66af842f22d3" containerID="eb83d8e035a06863701e57aaf2de56c4b99e299226ca5eb8963a658af5aeab0a" exitCode=255 Jul 10 00:30:25.250154 ip-10-0-130-146 kubenswrapper[2570]: I0710 00:30:25.250085 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-7866497494-d6fcd" event={"ID":"2b6cb4ca-79ff-42f0-b080-66af842f22d3","Type":"ContainerDied","Data":"eb83d8e035a06863701e57aaf2de56c4b99e299226ca5eb8963a658af5aeab0a"} Jul 10 00:30:25.250267 ip-10-0-130-146 kubenswrapper[2570]: I0710 00:30:25.250252 2570 scope.go:117] "RemoveContainer" containerID="eb83d8e035a06863701e57aaf2de56c4b99e299226ca5eb8963a658af5aeab0a" Jul 10 00:30:25.262395 ip-10-0-130-146 kubenswrapper[2570]: I0710 00:30:25.262354 2570 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-799cb7f958-p44ms" podStartSLOduration=2.165081835 podStartE2EDuration="5.262340624s" totalImagesPullingTime="3.097258789s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-10 00:30:20 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 00:30:25.261590879 +0000 UTC m=+244.309969600" watchObservedRunningTime="2026-07-10 00:30:25.262340624 +0000 UTC m=+244.310719346" Jul 10 00:30:25.301330 ip-10-0-130-146 kubenswrapper[2570]: I0710 00:30:25.301288 2570 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-insights/insights-operator-68d9dd94cf-7rq8g" podStartSLOduration=2.15906889 podStartE2EDuration="5.301273666s" totalImagesPullingTime="3.142204776s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-10 00:30:20 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 00:30:25.300262951 +0000 UTC m=+244.348641690" watchObservedRunningTime="2026-07-10 00:30:25.301273666 +0000 UTC m=+244.349652421" Jul 10 00:30:25.483386 ip-10-0-130-146 kubenswrapper[2570]: I0710 00:30:25.483351 2570 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-runtime-extractor-sp47j"] Jul 10 00:30:25.486667 ip-10-0-130-146 kubenswrapper[2570]: I0710 00:30:25.486643 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-sp47j" Jul 10 00:30:25.489926 ip-10-0-130-146 kubenswrapper[2570]: I0710 00:30:25.489908 2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\"" Jul 10 00:30:25.490037 ip-10-0-130-146 kubenswrapper[2570]: I0710 00:30:25.489909 2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\"" Jul 10 00:30:25.490459 ip-10-0-130-146 kubenswrapper[2570]: I0710 00:30:25.490421 2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-g2hvb\"" Jul 10 00:30:25.500417 ip-10-0-130-146 kubenswrapper[2570]: I0710 00:30:25.500368 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-sp47j"] Jul 10 00:30:25.633487 ip-10-0-130-146 kubenswrapper[2570]: I0710 00:30:25.633457 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/95631388-25a0-4948-a49d-2950902d0998-data-volume\") pod \"insights-runtime-extractor-sp47j\" (UID: \"95631388-25a0-4948-a49d-2950902d0998\") " pod="openshift-insights/insights-runtime-extractor-sp47j" Jul 10 00:30:25.633632 ip-10-0-130-146 kubenswrapper[2570]: I0710 00:30:25.633524 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/95631388-25a0-4948-a49d-2950902d0998-crio-socket\") pod \"insights-runtime-extractor-sp47j\" (UID: \"95631388-25a0-4948-a49d-2950902d0998\") " pod="openshift-insights/insights-runtime-extractor-sp47j" Jul 10 00:30:25.633632 ip-10-0-130-146 kubenswrapper[2570]: I0710 00:30:25.633543 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rd5nx\" (UniqueName: \"kubernetes.io/projected/95631388-25a0-4948-a49d-2950902d0998-kube-api-access-rd5nx\") pod \"insights-runtime-extractor-sp47j\" (UID: \"95631388-25a0-4948-a49d-2950902d0998\") " pod="openshift-insights/insights-runtime-extractor-sp47j" Jul 10 00:30:25.633632 ip-10-0-130-146 kubenswrapper[2570]: I0710 00:30:25.633576 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/95631388-25a0-4948-a49d-2950902d0998-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-sp47j\" (UID: \"95631388-25a0-4948-a49d-2950902d0998\") " pod="openshift-insights/insights-runtime-extractor-sp47j" Jul 10 00:30:25.633632 ip-10-0-130-146 kubenswrapper[2570]: I0710 00:30:25.633602 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/95631388-25a0-4948-a49d-2950902d0998-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-sp47j\" (UID: \"95631388-25a0-4948-a49d-2950902d0998\") " pod="openshift-insights/insights-runtime-extractor-sp47j" Jul 10 00:30:25.734822 ip-10-0-130-146 kubenswrapper[2570]: I0710 00:30:25.734783 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/95631388-25a0-4948-a49d-2950902d0998-crio-socket\") pod \"insights-runtime-extractor-sp47j\" (UID: \"95631388-25a0-4948-a49d-2950902d0998\") " pod="openshift-insights/insights-runtime-extractor-sp47j" Jul 10 00:30:25.734822 ip-10-0-130-146 kubenswrapper[2570]: I0710 00:30:25.734821 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rd5nx\" (UniqueName: \"kubernetes.io/projected/95631388-25a0-4948-a49d-2950902d0998-kube-api-access-rd5nx\") pod \"insights-runtime-extractor-sp47j\" (UID: \"95631388-25a0-4948-a49d-2950902d0998\") " pod="openshift-insights/insights-runtime-extractor-sp47j" Jul 10 00:30:25.735013 ip-10-0-130-146 kubenswrapper[2570]: I0710 00:30:25.734847 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/95631388-25a0-4948-a49d-2950902d0998-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-sp47j\" (UID: \"95631388-25a0-4948-a49d-2950902d0998\") " pod="openshift-insights/insights-runtime-extractor-sp47j" Jul 10 00:30:25.735013 ip-10-0-130-146 kubenswrapper[2570]: I0710 00:30:25.734887 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/95631388-25a0-4948-a49d-2950902d0998-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-sp47j\" (UID: \"95631388-25a0-4948-a49d-2950902d0998\") " pod="openshift-insights/insights-runtime-extractor-sp47j" Jul 10 00:30:25.735013 ip-10-0-130-146 kubenswrapper[2570]: I0710 00:30:25.734923 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/95631388-25a0-4948-a49d-2950902d0998-crio-socket\") pod \"insights-runtime-extractor-sp47j\" (UID: \"95631388-25a0-4948-a49d-2950902d0998\") " pod="openshift-insights/insights-runtime-extractor-sp47j" Jul 10 00:30:25.735013 ip-10-0-130-146 kubenswrapper[2570]: I0710 00:30:25.734946 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/95631388-25a0-4948-a49d-2950902d0998-data-volume\") pod \"insights-runtime-extractor-sp47j\" (UID: \"95631388-25a0-4948-a49d-2950902d0998\") " pod="openshift-insights/insights-runtime-extractor-sp47j" Jul 10 00:30:25.735013 ip-10-0-130-146 kubenswrapper[2570]: E0710 00:30:25.735002 2570 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Jul 10 00:30:25.735199 ip-10-0-130-146 kubenswrapper[2570]: E0710 00:30:25.735026 2570 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Jul 10 00:30:25.735199 ip-10-0-130-146 kubenswrapper[2570]: E0710 00:30:25.735087 2570 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/95631388-25a0-4948-a49d-2950902d0998-kube-rbac-proxy-cm podName:95631388-25a0-4948-a49d-2950902d0998 nodeName:}" failed. No retries permitted until 2026-07-10 00:30:26.235044546 +0000 UTC m=+245.283423250 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/95631388-25a0-4948-a49d-2950902d0998-kube-rbac-proxy-cm") pod "insights-runtime-extractor-sp47j" (UID: "95631388-25a0-4948-a49d-2950902d0998") : configmap "kube-rbac-proxy" not found Jul 10 00:30:25.735199 ip-10-0-130-146 kubenswrapper[2570]: E0710 00:30:25.735104 2570 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/95631388-25a0-4948-a49d-2950902d0998-insights-runtime-extractor-tls podName:95631388-25a0-4948-a49d-2950902d0998 nodeName:}" failed. No retries permitted until 2026-07-10 00:30:26.235096213 +0000 UTC m=+245.283474912 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/95631388-25a0-4948-a49d-2950902d0998-insights-runtime-extractor-tls") pod "insights-runtime-extractor-sp47j" (UID: "95631388-25a0-4948-a49d-2950902d0998") : secret "insights-runtime-extractor-tls" not found Jul 10 00:30:25.735300 ip-10-0-130-146 kubenswrapper[2570]: I0710 00:30:25.735276 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/95631388-25a0-4948-a49d-2950902d0998-data-volume\") pod \"insights-runtime-extractor-sp47j\" (UID: \"95631388-25a0-4948-a49d-2950902d0998\") " pod="openshift-insights/insights-runtime-extractor-sp47j" Jul 10 00:30:25.743514 ip-10-0-130-146 kubenswrapper[2570]: I0710 00:30:25.743496 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rd5nx\" (UniqueName: \"kubernetes.io/projected/95631388-25a0-4948-a49d-2950902d0998-kube-api-access-rd5nx\") pod \"insights-runtime-extractor-sp47j\" (UID: \"95631388-25a0-4948-a49d-2950902d0998\") " pod="openshift-insights/insights-runtime-extractor-sp47j" Jul 10 00:30:26.239198 ip-10-0-130-146 kubenswrapper[2570]: I0710 00:30:26.239156 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/95631388-25a0-4948-a49d-2950902d0998-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-sp47j\" (UID: \"95631388-25a0-4948-a49d-2950902d0998\") " pod="openshift-insights/insights-runtime-extractor-sp47j" Jul 10 00:30:26.239198 ip-10-0-130-146 kubenswrapper[2570]: I0710 00:30:26.239212 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/95631388-25a0-4948-a49d-2950902d0998-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-sp47j\" (UID: \"95631388-25a0-4948-a49d-2950902d0998\") " pod="openshift-insights/insights-runtime-extractor-sp47j" Jul 10 00:30:26.239431 ip-10-0-130-146 kubenswrapper[2570]: E0710 00:30:26.239279 2570 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Jul 10 00:30:26.239431 ip-10-0-130-146 kubenswrapper[2570]: E0710 00:30:26.239318 2570 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Jul 10 00:30:26.239431 ip-10-0-130-146 kubenswrapper[2570]: E0710 00:30:26.239374 2570 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/95631388-25a0-4948-a49d-2950902d0998-kube-rbac-proxy-cm podName:95631388-25a0-4948-a49d-2950902d0998 nodeName:}" failed. No retries permitted until 2026-07-10 00:30:27.239352757 +0000 UTC m=+246.287731460 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/95631388-25a0-4948-a49d-2950902d0998-kube-rbac-proxy-cm") pod "insights-runtime-extractor-sp47j" (UID: "95631388-25a0-4948-a49d-2950902d0998") : configmap "kube-rbac-proxy" not found Jul 10 00:30:26.239431 ip-10-0-130-146 kubenswrapper[2570]: E0710 00:30:26.239393 2570 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/95631388-25a0-4948-a49d-2950902d0998-insights-runtime-extractor-tls podName:95631388-25a0-4948-a49d-2950902d0998 nodeName:}" failed. No retries permitted until 2026-07-10 00:30:27.239386295 +0000 UTC m=+246.287764993 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/95631388-25a0-4948-a49d-2950902d0998-insights-runtime-extractor-tls") pod "insights-runtime-extractor-sp47j" (UID: "95631388-25a0-4948-a49d-2950902d0998") : secret "insights-runtime-extractor-tls" not found Jul 10 00:30:26.255057 ip-10-0-130-146 kubenswrapper[2570]: I0710 00:30:26.255035 2570 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7866497494-d6fcd_2b6cb4ca-79ff-42f0-b080-66af842f22d3/console-operator/1.log" Jul 10 00:30:26.255472 ip-10-0-130-146 kubenswrapper[2570]: I0710 00:30:26.255458 2570 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7866497494-d6fcd_2b6cb4ca-79ff-42f0-b080-66af842f22d3/console-operator/0.log" Jul 10 00:30:26.255530 ip-10-0-130-146 kubenswrapper[2570]: I0710 00:30:26.255491 2570 generic.go:356] "Generic (PLEG): container finished" podID="2b6cb4ca-79ff-42f0-b080-66af842f22d3" containerID="d70906c40620c67eb02e63b34e6af43ec4e4415cda15c67b868c4c53492778f1" exitCode=255 Jul 10 00:30:26.255575 ip-10-0-130-146 kubenswrapper[2570]: I0710 00:30:26.255559 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-7866497494-d6fcd" event={"ID":"2b6cb4ca-79ff-42f0-b080-66af842f22d3","Type":"ContainerDied","Data":"d70906c40620c67eb02e63b34e6af43ec4e4415cda15c67b868c4c53492778f1"} Jul 10 00:30:26.256152 ip-10-0-130-146 kubenswrapper[2570]: I0710 00:30:26.256135 2570 scope.go:117] "RemoveContainer" containerID="eb83d8e035a06863701e57aaf2de56c4b99e299226ca5eb8963a658af5aeab0a" Jul 10 00:30:26.256244 ip-10-0-130-146 kubenswrapper[2570]: I0710 00:30:26.256232 2570 scope.go:117] "RemoveContainer" containerID="d70906c40620c67eb02e63b34e6af43ec4e4415cda15c67b868c4c53492778f1" Jul 10 00:30:26.256373 ip-10-0-130-146 kubenswrapper[2570]: E0710 00:30:26.256355 2570 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 10s restarting failed container=console-operator pod=console-operator-7866497494-d6fcd_openshift-console-operator(2b6cb4ca-79ff-42f0-b080-66af842f22d3)\"" pod="openshift-console-operator/console-operator-7866497494-d6fcd" podUID="2b6cb4ca-79ff-42f0-b080-66af842f22d3" Jul 10 00:30:27.247698 ip-10-0-130-146 kubenswrapper[2570]: I0710 00:30:27.247659 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/95631388-25a0-4948-a49d-2950902d0998-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-sp47j\" (UID: \"95631388-25a0-4948-a49d-2950902d0998\") " pod="openshift-insights/insights-runtime-extractor-sp47j" Jul 10 00:30:27.248087 ip-10-0-130-146 kubenswrapper[2570]: I0710 00:30:27.247710 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/95631388-25a0-4948-a49d-2950902d0998-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-sp47j\" (UID: \"95631388-25a0-4948-a49d-2950902d0998\") " pod="openshift-insights/insights-runtime-extractor-sp47j" Jul 10 00:30:27.248087 ip-10-0-130-146 kubenswrapper[2570]: E0710 00:30:27.247815 2570 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Jul 10 00:30:27.248087 ip-10-0-130-146 kubenswrapper[2570]: E0710 00:30:27.247884 2570 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/95631388-25a0-4948-a49d-2950902d0998-insights-runtime-extractor-tls podName:95631388-25a0-4948-a49d-2950902d0998 nodeName:}" failed. No retries permitted until 2026-07-10 00:30:29.247867698 +0000 UTC m=+248.296246400 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/95631388-25a0-4948-a49d-2950902d0998-insights-runtime-extractor-tls") pod "insights-runtime-extractor-sp47j" (UID: "95631388-25a0-4948-a49d-2950902d0998") : secret "insights-runtime-extractor-tls" not found Jul 10 00:30:27.248087 ip-10-0-130-146 kubenswrapper[2570]: E0710 00:30:27.247819 2570 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Jul 10 00:30:27.248087 ip-10-0-130-146 kubenswrapper[2570]: E0710 00:30:27.247945 2570 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/95631388-25a0-4948-a49d-2950902d0998-kube-rbac-proxy-cm podName:95631388-25a0-4948-a49d-2950902d0998 nodeName:}" failed. No retries permitted until 2026-07-10 00:30:29.247933809 +0000 UTC m=+248.296312509 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/95631388-25a0-4948-a49d-2950902d0998-kube-rbac-proxy-cm") pod "insights-runtime-extractor-sp47j" (UID: "95631388-25a0-4948-a49d-2950902d0998") : configmap "kube-rbac-proxy" not found Jul 10 00:30:27.260126 ip-10-0-130-146 kubenswrapper[2570]: I0710 00:30:27.260102 2570 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7866497494-d6fcd_2b6cb4ca-79ff-42f0-b080-66af842f22d3/console-operator/1.log" Jul 10 00:30:27.260445 ip-10-0-130-146 kubenswrapper[2570]: I0710 00:30:27.260429 2570 scope.go:117] "RemoveContainer" containerID="d70906c40620c67eb02e63b34e6af43ec4e4415cda15c67b868c4c53492778f1" Jul 10 00:30:27.260611 ip-10-0-130-146 kubenswrapper[2570]: E0710 00:30:27.260592 2570 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 10s restarting failed container=console-operator pod=console-operator-7866497494-d6fcd_openshift-console-operator(2b6cb4ca-79ff-42f0-b080-66af842f22d3)\"" pod="openshift-console-operator/console-operator-7866497494-d6fcd" podUID="2b6cb4ca-79ff-42f0-b080-66af842f22d3" Jul 10 00:30:27.346484 ip-10-0-130-146 kubenswrapper[2570]: I0710 00:30:27.346456 2570 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-service-ca/service-ca-79f68b9f4f-x4nls"] Jul 10 00:30:27.350747 ip-10-0-130-146 kubenswrapper[2570]: I0710 00:30:27.350727 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca/service-ca-79f68b9f4f-x4nls" Jul 10 00:30:27.353442 ip-10-0-130-146 kubenswrapper[2570]: I0710 00:30:27.353409 2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"kube-root-ca.crt\"" Jul 10 00:30:27.353442 ip-10-0-130-146 kubenswrapper[2570]: I0710 00:30:27.353427 2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"openshift-service-ca.crt\"" Jul 10 00:30:27.353599 ip-10-0-130-146 kubenswrapper[2570]: I0710 00:30:27.353454 2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca\"/\"signing-key\"" Jul 10 00:30:27.353599 ip-10-0-130-146 kubenswrapper[2570]: I0710 00:30:27.353427 2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"signing-cabundle\"" Jul 10 00:30:27.353599 ip-10-0-130-146 kubenswrapper[2570]: I0710 00:30:27.353427 2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca\"/\"service-ca-dockercfg-sfpl2\"" Jul 10 00:30:27.357809 ip-10-0-130-146 kubenswrapper[2570]: I0710 00:30:27.357791 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca/service-ca-79f68b9f4f-x4nls"] Jul 10 00:30:27.449865 ip-10-0-130-146 kubenswrapper[2570]: I0710 00:30:27.449828 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/ee9cbd3f-21bc-4cce-be6c-a98b8024e6c1-signing-key\") pod \"service-ca-79f68b9f4f-x4nls\" (UID: \"ee9cbd3f-21bc-4cce-be6c-a98b8024e6c1\") " pod="openshift-service-ca/service-ca-79f68b9f4f-x4nls" Jul 10 00:30:27.450005 ip-10-0-130-146 kubenswrapper[2570]: I0710 00:30:27.449958 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/ee9cbd3f-21bc-4cce-be6c-a98b8024e6c1-signing-cabundle\") pod \"service-ca-79f68b9f4f-x4nls\" (UID: \"ee9cbd3f-21bc-4cce-be6c-a98b8024e6c1\") " pod="openshift-service-ca/service-ca-79f68b9f4f-x4nls" Jul 10 00:30:27.450099 ip-10-0-130-146 kubenswrapper[2570]: I0710 00:30:27.450017 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jkt9n\" (UniqueName: \"kubernetes.io/projected/ee9cbd3f-21bc-4cce-be6c-a98b8024e6c1-kube-api-access-jkt9n\") pod \"service-ca-79f68b9f4f-x4nls\" (UID: \"ee9cbd3f-21bc-4cce-be6c-a98b8024e6c1\") " pod="openshift-service-ca/service-ca-79f68b9f4f-x4nls" Jul 10 00:30:27.488758 ip-10-0-130-146 kubenswrapper[2570]: I0710 00:30:27.488729 2570 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-7tn6x_006ba44e-264c-42cd-9891-ad8d769386c8/dns-node-resolver/0.log" Jul 10 00:30:27.550915 ip-10-0-130-146 kubenswrapper[2570]: I0710 00:30:27.550846 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/ee9cbd3f-21bc-4cce-be6c-a98b8024e6c1-signing-key\") pod \"service-ca-79f68b9f4f-x4nls\" (UID: \"ee9cbd3f-21bc-4cce-be6c-a98b8024e6c1\") " pod="openshift-service-ca/service-ca-79f68b9f4f-x4nls" Jul 10 00:30:27.550915 ip-10-0-130-146 kubenswrapper[2570]: I0710 00:30:27.550904 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/ee9cbd3f-21bc-4cce-be6c-a98b8024e6c1-signing-cabundle\") pod \"service-ca-79f68b9f4f-x4nls\" (UID: \"ee9cbd3f-21bc-4cce-be6c-a98b8024e6c1\") " pod="openshift-service-ca/service-ca-79f68b9f4f-x4nls" Jul 10 00:30:27.551068 ip-10-0-130-146 kubenswrapper[2570]: I0710 00:30:27.550938 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jkt9n\" (UniqueName: \"kubernetes.io/projected/ee9cbd3f-21bc-4cce-be6c-a98b8024e6c1-kube-api-access-jkt9n\") pod \"service-ca-79f68b9f4f-x4nls\" (UID: \"ee9cbd3f-21bc-4cce-be6c-a98b8024e6c1\") " pod="openshift-service-ca/service-ca-79f68b9f4f-x4nls" Jul 10 00:30:27.551628 ip-10-0-130-146 kubenswrapper[2570]: I0710 00:30:27.551611 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/ee9cbd3f-21bc-4cce-be6c-a98b8024e6c1-signing-cabundle\") pod \"service-ca-79f68b9f4f-x4nls\" (UID: \"ee9cbd3f-21bc-4cce-be6c-a98b8024e6c1\") " pod="openshift-service-ca/service-ca-79f68b9f4f-x4nls" Jul 10 00:30:27.553216 ip-10-0-130-146 kubenswrapper[2570]: I0710 00:30:27.553197 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/ee9cbd3f-21bc-4cce-be6c-a98b8024e6c1-signing-key\") pod \"service-ca-79f68b9f4f-x4nls\" (UID: \"ee9cbd3f-21bc-4cce-be6c-a98b8024e6c1\") " pod="openshift-service-ca/service-ca-79f68b9f4f-x4nls" Jul 10 00:30:27.559007 ip-10-0-130-146 kubenswrapper[2570]: I0710 00:30:27.558988 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jkt9n\" (UniqueName: \"kubernetes.io/projected/ee9cbd3f-21bc-4cce-be6c-a98b8024e6c1-kube-api-access-jkt9n\") pod \"service-ca-79f68b9f4f-x4nls\" (UID: \"ee9cbd3f-21bc-4cce-be6c-a98b8024e6c1\") " pod="openshift-service-ca/service-ca-79f68b9f4f-x4nls" Jul 10 00:30:27.660021 ip-10-0-130-146 kubenswrapper[2570]: I0710 00:30:27.659995 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca/service-ca-79f68b9f4f-x4nls" Jul 10 00:30:27.779217 ip-10-0-130-146 kubenswrapper[2570]: I0710 00:30:27.779200 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca/service-ca-79f68b9f4f-x4nls"] Jul 10 00:30:27.781196 ip-10-0-130-146 kubenswrapper[2570]: W0710 00:30:27.781170 2570 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podee9cbd3f_21bc_4cce_be6c_a98b8024e6c1.slice/crio-d9da2accc2832810193c981bef529daa0b3c736fbdf56a128159c9b3a3405e0b WatchSource:0}: Error finding container d9da2accc2832810193c981bef529daa0b3c736fbdf56a128159c9b3a3405e0b: Status 404 returned error can't find the container with id d9da2accc2832810193c981bef529daa0b3c736fbdf56a128159c9b3a3405e0b Jul 10 00:30:28.264573 ip-10-0-130-146 kubenswrapper[2570]: I0710 00:30:28.264538 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca/service-ca-79f68b9f4f-x4nls" event={"ID":"ee9cbd3f-21bc-4cce-be6c-a98b8024e6c1","Type":"ContainerStarted","Data":"d9da2accc2832810193c981bef529daa0b3c736fbdf56a128159c9b3a3405e0b"} Jul 10 00:30:28.457968 ip-10-0-130-146 kubenswrapper[2570]: I0710 00:30:28.457931 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/57c018ee-b976-4f56-a33b-b41cb28c4417-registry-tls\") pod \"image-registry-b9fdbf76c-kptvw\" (UID: \"57c018ee-b976-4f56-a33b-b41cb28c4417\") " pod="openshift-image-registry/image-registry-b9fdbf76c-kptvw" Jul 10 00:30:28.458175 ip-10-0-130-146 kubenswrapper[2570]: I0710 00:30:28.457978 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/d4b4eb0f-539c-4193-907e-cbff010aae7b-networking-console-plugin-cert\") pod \"networking-console-plugin-6cf856758c-45x56\" (UID: \"d4b4eb0f-539c-4193-907e-cbff010aae7b\") " pod="openshift-network-console/networking-console-plugin-6cf856758c-45x56" Jul 10 00:30:28.458175 ip-10-0-130-146 kubenswrapper[2570]: E0710 00:30:28.458108 2570 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Jul 10 00:30:28.458175 ip-10-0-130-146 kubenswrapper[2570]: E0710 00:30:28.458126 2570 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-b9fdbf76c-kptvw: secret "image-registry-tls" not found Jul 10 00:30:28.458175 ip-10-0-130-146 kubenswrapper[2570]: E0710 00:30:28.458165 2570 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found Jul 10 00:30:28.458393 ip-10-0-130-146 kubenswrapper[2570]: E0710 00:30:28.458179 2570 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found Jul 10 00:30:28.458393 ip-10-0-130-146 kubenswrapper[2570]: I0710 00:30:28.458107 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/4e3a154b-8ba7-4c18-b75b-93444d0de385-samples-operator-tls\") pod \"cluster-samples-operator-65db858894-bzzfq\" (UID: \"4e3a154b-8ba7-4c18-b75b-93444d0de385\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-65db858894-bzzfq" Jul 10 00:30:28.458393 ip-10-0-130-146 kubenswrapper[2570]: E0710 00:30:28.458185 2570 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/57c018ee-b976-4f56-a33b-b41cb28c4417-registry-tls podName:57c018ee-b976-4f56-a33b-b41cb28c4417 nodeName:}" failed. No retries permitted until 2026-07-10 00:30:36.458166868 +0000 UTC m=+255.506545571 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/57c018ee-b976-4f56-a33b-b41cb28c4417-registry-tls") pod "image-registry-b9fdbf76c-kptvw" (UID: "57c018ee-b976-4f56-a33b-b41cb28c4417") : secret "image-registry-tls" not found Jul 10 00:30:28.458393 ip-10-0-130-146 kubenswrapper[2570]: E0710 00:30:28.458278 2570 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4e3a154b-8ba7-4c18-b75b-93444d0de385-samples-operator-tls podName:4e3a154b-8ba7-4c18-b75b-93444d0de385 nodeName:}" failed. No retries permitted until 2026-07-10 00:30:36.458259653 +0000 UTC m=+255.506638353 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/4e3a154b-8ba7-4c18-b75b-93444d0de385-samples-operator-tls") pod "cluster-samples-operator-65db858894-bzzfq" (UID: "4e3a154b-8ba7-4c18-b75b-93444d0de385") : secret "samples-operator-tls" not found Jul 10 00:30:28.458393 ip-10-0-130-146 kubenswrapper[2570]: E0710 00:30:28.458298 2570 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d4b4eb0f-539c-4193-907e-cbff010aae7b-networking-console-plugin-cert podName:d4b4eb0f-539c-4193-907e-cbff010aae7b nodeName:}" failed. No retries permitted until 2026-07-10 00:30:36.45828788 +0000 UTC m=+255.506666584 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/d4b4eb0f-539c-4193-907e-cbff010aae7b-networking-console-plugin-cert") pod "networking-console-plugin-6cf856758c-45x56" (UID: "d4b4eb0f-539c-4193-907e-cbff010aae7b") : secret "networking-console-plugin-cert" not found Jul 10 00:30:28.488968 ip-10-0-130-146 kubenswrapper[2570]: I0710 00:30:28.488943 2570 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-t526b_52179d08-5861-4a23-9c8d-653c13bed339/node-ca/0.log" Jul 10 00:30:29.089540 ip-10-0-130-146 kubenswrapper[2570]: I0710 00:30:29.089501 2570 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-ck6gk_3ec525d5-c820-42fe-a411-99faa6ef373d/serve-healthcheck-canary/0.log" Jul 10 00:30:29.263563 ip-10-0-130-146 kubenswrapper[2570]: I0710 00:30:29.263529 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/95631388-25a0-4948-a49d-2950902d0998-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-sp47j\" (UID: \"95631388-25a0-4948-a49d-2950902d0998\") " pod="openshift-insights/insights-runtime-extractor-sp47j" Jul 10 00:30:29.263736 ip-10-0-130-146 kubenswrapper[2570]: I0710 00:30:29.263576 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/95631388-25a0-4948-a49d-2950902d0998-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-sp47j\" (UID: \"95631388-25a0-4948-a49d-2950902d0998\") " pod="openshift-insights/insights-runtime-extractor-sp47j" Jul 10 00:30:29.263736 ip-10-0-130-146 kubenswrapper[2570]: E0710 00:30:29.263644 2570 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Jul 10 00:30:29.263736 ip-10-0-130-146 kubenswrapper[2570]: E0710 00:30:29.263690 2570 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/95631388-25a0-4948-a49d-2950902d0998-kube-rbac-proxy-cm podName:95631388-25a0-4948-a49d-2950902d0998 nodeName:}" failed. No retries permitted until 2026-07-10 00:30:33.263677216 +0000 UTC m=+252.312055914 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/95631388-25a0-4948-a49d-2950902d0998-kube-rbac-proxy-cm") pod "insights-runtime-extractor-sp47j" (UID: "95631388-25a0-4948-a49d-2950902d0998") : configmap "kube-rbac-proxy" not found Jul 10 00:30:29.263736 ip-10-0-130-146 kubenswrapper[2570]: E0710 00:30:29.263687 2570 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Jul 10 00:30:29.263882 ip-10-0-130-146 kubenswrapper[2570]: E0710 00:30:29.263757 2570 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/95631388-25a0-4948-a49d-2950902d0998-insights-runtime-extractor-tls podName:95631388-25a0-4948-a49d-2950902d0998 nodeName:}" failed. No retries permitted until 2026-07-10 00:30:33.263738548 +0000 UTC m=+252.312117247 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/95631388-25a0-4948-a49d-2950902d0998-insights-runtime-extractor-tls") pod "insights-runtime-extractor-sp47j" (UID: "95631388-25a0-4948-a49d-2950902d0998") : secret "insights-runtime-extractor-tls" not found Jul 10 00:30:30.272413 ip-10-0-130-146 kubenswrapper[2570]: I0710 00:30:30.272375 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca/service-ca-79f68b9f4f-x4nls" event={"ID":"ee9cbd3f-21bc-4cce-be6c-a98b8024e6c1","Type":"ContainerStarted","Data":"98cc098e92994895faf1678f95e46c8a9c8f85cd159735f8e7ecbc3c5c8a1478"} Jul 10 00:30:30.287582 ip-10-0-130-146 kubenswrapper[2570]: I0710 00:30:30.287537 2570 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-service-ca/service-ca-79f68b9f4f-x4nls" podStartSLOduration=0.944360604 podStartE2EDuration="3.287523415s" totalImagesPullingTime="2.343162811s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-10 00:30:27 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 00:30:30.286293238 +0000 UTC m=+249.334671959" watchObservedRunningTime="2026-07-10 00:30:30.287523415 +0000 UTC m=+249.335902137" Jul 10 00:30:30.971302 ip-10-0-130-146 kubenswrapper[2570]: I0710 00:30:30.971222 2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console-operator/console-operator-7866497494-d6fcd" Jul 10 00:30:30.971302 ip-10-0-130-146 kubenswrapper[2570]: I0710 00:30:30.971293 2570 kubelet.go:2654] "SyncLoop (probe)" probe="liveness" status="unhealthy" pod="openshift-console-operator/console-operator-7866497494-d6fcd" Jul 10 00:30:30.971623 ip-10-0-130-146 kubenswrapper[2570]: I0710 00:30:30.971611 2570 scope.go:117] "RemoveContainer" containerID="d70906c40620c67eb02e63b34e6af43ec4e4415cda15c67b868c4c53492778f1" Jul 10 00:30:30.971778 ip-10-0-130-146 kubenswrapper[2570]: E0710 00:30:30.971759 2570 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 10s restarting failed container=console-operator pod=console-operator-7866497494-d6fcd_openshift-console-operator(2b6cb4ca-79ff-42f0-b080-66af842f22d3)\"" pod="openshift-console-operator/console-operator-7866497494-d6fcd" podUID="2b6cb4ca-79ff-42f0-b080-66af842f22d3" Jul 10 00:30:33.301198 ip-10-0-130-146 kubenswrapper[2570]: I0710 00:30:33.301163 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/95631388-25a0-4948-a49d-2950902d0998-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-sp47j\" (UID: \"95631388-25a0-4948-a49d-2950902d0998\") " pod="openshift-insights/insights-runtime-extractor-sp47j" Jul 10 00:30:33.301590 ip-10-0-130-146 kubenswrapper[2570]: I0710 00:30:33.301211 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/95631388-25a0-4948-a49d-2950902d0998-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-sp47j\" (UID: \"95631388-25a0-4948-a49d-2950902d0998\") " pod="openshift-insights/insights-runtime-extractor-sp47j" Jul 10 00:30:33.301590 ip-10-0-130-146 kubenswrapper[2570]: E0710 00:30:33.301325 2570 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Jul 10 00:30:33.301590 ip-10-0-130-146 kubenswrapper[2570]: E0710 00:30:33.301330 2570 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Jul 10 00:30:33.301590 ip-10-0-130-146 kubenswrapper[2570]: E0710 00:30:33.301375 2570 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/95631388-25a0-4948-a49d-2950902d0998-kube-rbac-proxy-cm podName:95631388-25a0-4948-a49d-2950902d0998 nodeName:}" failed. No retries permitted until 2026-07-10 00:30:41.301360329 +0000 UTC m=+260.349739028 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/95631388-25a0-4948-a49d-2950902d0998-kube-rbac-proxy-cm") pod "insights-runtime-extractor-sp47j" (UID: "95631388-25a0-4948-a49d-2950902d0998") : configmap "kube-rbac-proxy" not found Jul 10 00:30:33.301590 ip-10-0-130-146 kubenswrapper[2570]: E0710 00:30:33.301387 2570 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/95631388-25a0-4948-a49d-2950902d0998-insights-runtime-extractor-tls podName:95631388-25a0-4948-a49d-2950902d0998 nodeName:}" failed. No retries permitted until 2026-07-10 00:30:41.301381159 +0000 UTC m=+260.349759858 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/95631388-25a0-4948-a49d-2950902d0998-insights-runtime-extractor-tls") pod "insights-runtime-extractor-sp47j" (UID: "95631388-25a0-4948-a49d-2950902d0998") : secret "insights-runtime-extractor-tls" not found Jul 10 00:30:35.116352 ip-10-0-130-146 kubenswrapper[2570]: I0710 00:30:35.116316 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b7aab806-00a2-491e-a997-ed3174bd547a-metrics-certs\") pod \"network-metrics-daemon-d8t56\" (UID: \"b7aab806-00a2-491e-a997-ed3174bd547a\") " pod="openshift-multus/network-metrics-daemon-d8t56" Jul 10 00:30:35.116794 ip-10-0-130-146 kubenswrapper[2570]: E0710 00:30:35.116494 2570 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found Jul 10 00:30:35.116794 ip-10-0-130-146 kubenswrapper[2570]: E0710 00:30:35.116578 2570 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b7aab806-00a2-491e-a997-ed3174bd547a-metrics-certs podName:b7aab806-00a2-491e-a997-ed3174bd547a nodeName:}" failed. No retries permitted until 2026-07-10 00:32:37.116557215 +0000 UTC m=+376.164935926 (durationBeforeRetry 2m2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/b7aab806-00a2-491e-a997-ed3174bd547a-metrics-certs") pod "network-metrics-daemon-d8t56" (UID: "b7aab806-00a2-491e-a997-ed3174bd547a") : secret "metrics-daemon-secret" not found Jul 10 00:30:36.526775 ip-10-0-130-146 kubenswrapper[2570]: I0710 00:30:36.526735 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/57c018ee-b976-4f56-a33b-b41cb28c4417-registry-tls\") pod \"image-registry-b9fdbf76c-kptvw\" (UID: \"57c018ee-b976-4f56-a33b-b41cb28c4417\") " pod="openshift-image-registry/image-registry-b9fdbf76c-kptvw" Jul 10 00:30:36.526775 ip-10-0-130-146 kubenswrapper[2570]: I0710 00:30:36.526781 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/d4b4eb0f-539c-4193-907e-cbff010aae7b-networking-console-plugin-cert\") pod \"networking-console-plugin-6cf856758c-45x56\" (UID: \"d4b4eb0f-539c-4193-907e-cbff010aae7b\") " pod="openshift-network-console/networking-console-plugin-6cf856758c-45x56" Jul 10 00:30:36.527311 ip-10-0-130-146 kubenswrapper[2570]: E0710 00:30:36.526865 2570 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found Jul 10 00:30:36.527311 ip-10-0-130-146 kubenswrapper[2570]: E0710 00:30:36.526912 2570 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d4b4eb0f-539c-4193-907e-cbff010aae7b-networking-console-plugin-cert podName:d4b4eb0f-539c-4193-907e-cbff010aae7b nodeName:}" failed. No retries permitted until 2026-07-10 00:30:52.526898909 +0000 UTC m=+271.575277607 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/d4b4eb0f-539c-4193-907e-cbff010aae7b-networking-console-plugin-cert") pod "networking-console-plugin-6cf856758c-45x56" (UID: "d4b4eb0f-539c-4193-907e-cbff010aae7b") : secret "networking-console-plugin-cert" not found Jul 10 00:30:36.527311 ip-10-0-130-146 kubenswrapper[2570]: I0710 00:30:36.527090 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/4e3a154b-8ba7-4c18-b75b-93444d0de385-samples-operator-tls\") pod \"cluster-samples-operator-65db858894-bzzfq\" (UID: \"4e3a154b-8ba7-4c18-b75b-93444d0de385\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-65db858894-bzzfq" Jul 10 00:30:36.529273 ip-10-0-130-146 kubenswrapper[2570]: I0710 00:30:36.529250 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/4e3a154b-8ba7-4c18-b75b-93444d0de385-samples-operator-tls\") pod \"cluster-samples-operator-65db858894-bzzfq\" (UID: \"4e3a154b-8ba7-4c18-b75b-93444d0de385\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-65db858894-bzzfq" Jul 10 00:30:36.529376 ip-10-0-130-146 kubenswrapper[2570]: I0710 00:30:36.529341 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/57c018ee-b976-4f56-a33b-b41cb28c4417-registry-tls\") pod \"image-registry-b9fdbf76c-kptvw\" (UID: \"57c018ee-b976-4f56-a33b-b41cb28c4417\") " pod="openshift-image-registry/image-registry-b9fdbf76c-kptvw" Jul 10 00:30:36.665112 ip-10-0-130-146 kubenswrapper[2570]: I0710 00:30:36.665080 2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-samples-operator\"/\"cluster-samples-operator-dockercfg-g478g\"" Jul 10 00:30:36.670095 ip-10-0-130-146 kubenswrapper[2570]: I0710 00:30:36.670069 2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"registry-dockercfg-6nz9c\"" Jul 10 00:30:36.672849 ip-10-0-130-146 kubenswrapper[2570]: I0710 00:30:36.672831 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-samples-operator/cluster-samples-operator-65db858894-bzzfq" Jul 10 00:30:36.678680 ip-10-0-130-146 kubenswrapper[2570]: I0710 00:30:36.678656 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-b9fdbf76c-kptvw" Jul 10 00:30:36.827584 ip-10-0-130-146 kubenswrapper[2570]: I0710 00:30:36.827542 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-samples-operator/cluster-samples-operator-65db858894-bzzfq"] Jul 10 00:30:36.832490 ip-10-0-130-146 kubenswrapper[2570]: I0710 00:30:36.832459 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-b9fdbf76c-kptvw"] Jul 10 00:30:36.835033 ip-10-0-130-146 kubenswrapper[2570]: W0710 00:30:36.835008 2570 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod57c018ee_b976_4f56_a33b_b41cb28c4417.slice/crio-a08461e74e45d3f3ba334adc4149f969361c0262d9133e74506bddaea6f3f11b WatchSource:0}: Error finding container a08461e74e45d3f3ba334adc4149f969361c0262d9133e74506bddaea6f3f11b: Status 404 returned error can't find the container with id a08461e74e45d3f3ba334adc4149f969361c0262d9133e74506bddaea6f3f11b Jul 10 00:30:37.294582 ip-10-0-130-146 kubenswrapper[2570]: I0710 00:30:37.294544 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-b9fdbf76c-kptvw" event={"ID":"57c018ee-b976-4f56-a33b-b41cb28c4417","Type":"ContainerStarted","Data":"ba389df5506b155942234611c7bfd5a44637aa550bc6cae57914c6a083d7c053"} Jul 10 00:30:37.294767 ip-10-0-130-146 kubenswrapper[2570]: I0710 00:30:37.294615 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-b9fdbf76c-kptvw" event={"ID":"57c018ee-b976-4f56-a33b-b41cb28c4417","Type":"ContainerStarted","Data":"a08461e74e45d3f3ba334adc4149f969361c0262d9133e74506bddaea6f3f11b"} Jul 10 00:30:37.294767 ip-10-0-130-146 kubenswrapper[2570]: I0710 00:30:37.294647 2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-image-registry/image-registry-b9fdbf76c-kptvw" Jul 10 00:30:37.295601 ip-10-0-130-146 kubenswrapper[2570]: I0710 00:30:37.295572 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-samples-operator/cluster-samples-operator-65db858894-bzzfq" event={"ID":"4e3a154b-8ba7-4c18-b75b-93444d0de385","Type":"ContainerStarted","Data":"18455d833417daf3f938a2c5b8d91de6d7700d9732d4ad86cc58c2bdb264d468"} Jul 10 00:30:37.313124 ip-10-0-130-146 kubenswrapper[2570]: I0710 00:30:37.313069 2570 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/image-registry-b9fdbf76c-kptvw" podStartSLOduration=17.313033891 podStartE2EDuration="17.313033891s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-10 00:30:20 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 00:30:37.312117375 +0000 UTC m=+256.360496097" watchObservedRunningTime="2026-07-10 00:30:37.313033891 +0000 UTC m=+256.361412594" Jul 10 00:30:39.309358 ip-10-0-130-146 kubenswrapper[2570]: I0710 00:30:39.309319 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-samples-operator/cluster-samples-operator-65db858894-bzzfq" event={"ID":"4e3a154b-8ba7-4c18-b75b-93444d0de385","Type":"ContainerStarted","Data":"f4b99829a9302259fa2544e5e5885ebe41f2a76ce36f09889833dcb88098b0cd"} Jul 10 00:30:39.309733 ip-10-0-130-146 kubenswrapper[2570]: I0710 00:30:39.309416 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-samples-operator/cluster-samples-operator-65db858894-bzzfq" event={"ID":"4e3a154b-8ba7-4c18-b75b-93444d0de385","Type":"ContainerStarted","Data":"d6bf4fa26c6a20009c393cd4e8e0f8f76b21d01b3a6f0de3a43b56f53a5eae6b"} Jul 10 00:30:39.325029 ip-10-0-130-146 kubenswrapper[2570]: I0710 00:30:39.324982 2570 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-samples-operator/cluster-samples-operator-65db858894-bzzfq" podStartSLOduration=17.583050062 podStartE2EDuration="19.324968414s" totalImagesPullingTime="1.741918352s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-10 00:30:20 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 00:30:39.323715632 +0000 UTC m=+258.372094353" watchObservedRunningTime="2026-07-10 00:30:39.324968414 +0000 UTC m=+258.373347134" Jul 10 00:30:41.369475 ip-10-0-130-146 kubenswrapper[2570]: I0710 00:30:41.369439 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/95631388-25a0-4948-a49d-2950902d0998-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-sp47j\" (UID: \"95631388-25a0-4948-a49d-2950902d0998\") " pod="openshift-insights/insights-runtime-extractor-sp47j" Jul 10 00:30:41.369977 ip-10-0-130-146 kubenswrapper[2570]: I0710 00:30:41.369487 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/95631388-25a0-4948-a49d-2950902d0998-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-sp47j\" (UID: \"95631388-25a0-4948-a49d-2950902d0998\") " pod="openshift-insights/insights-runtime-extractor-sp47j" Jul 10 00:30:41.369977 ip-10-0-130-146 kubenswrapper[2570]: I0710 00:30:41.369973 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/95631388-25a0-4948-a49d-2950902d0998-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-sp47j\" (UID: \"95631388-25a0-4948-a49d-2950902d0998\") " pod="openshift-insights/insights-runtime-extractor-sp47j" Jul 10 00:30:41.371831 ip-10-0-130-146 kubenswrapper[2570]: I0710 00:30:41.371809 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/95631388-25a0-4948-a49d-2950902d0998-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-sp47j\" (UID: \"95631388-25a0-4948-a49d-2950902d0998\") " pod="openshift-insights/insights-runtime-extractor-sp47j" Jul 10 00:30:41.398486 ip-10-0-130-146 kubenswrapper[2570]: I0710 00:30:41.398462 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-sp47j" Jul 10 00:30:41.520502 ip-10-0-130-146 kubenswrapper[2570]: I0710 00:30:41.520470 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-sp47j"] Jul 10 00:30:41.523118 ip-10-0-130-146 kubenswrapper[2570]: W0710 00:30:41.523092 2570 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod95631388_25a0_4948_a49d_2950902d0998.slice/crio-148e1bfe86ac9a82932eff0df51457dd92a8cf34214c79703ddb099ea615eb7c WatchSource:0}: Error finding container 148e1bfe86ac9a82932eff0df51457dd92a8cf34214c79703ddb099ea615eb7c: Status 404 returned error can't find the container with id 148e1bfe86ac9a82932eff0df51457dd92a8cf34214c79703ddb099ea615eb7c Jul 10 00:30:42.321902 ip-10-0-130-146 kubenswrapper[2570]: I0710 00:30:42.321865 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-sp47j" event={"ID":"95631388-25a0-4948-a49d-2950902d0998","Type":"ContainerStarted","Data":"148e1bfe86ac9a82932eff0df51457dd92a8cf34214c79703ddb099ea615eb7c"} Jul 10 00:30:45.448606 ip-10-0-130-146 kubenswrapper[2570]: I0710 00:30:45.448584 2570 scope.go:117] "RemoveContainer" containerID="d70906c40620c67eb02e63b34e6af43ec4e4415cda15c67b868c4c53492778f1" Jul 10 00:30:46.000774 ip-10-0-130-146 kubenswrapper[2570]: I0710 00:30:46.000744 2570 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-b9fdbf76c-kptvw"] Jul 10 00:30:46.337109 ip-10-0-130-146 kubenswrapper[2570]: I0710 00:30:46.337032 2570 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7866497494-d6fcd_2b6cb4ca-79ff-42f0-b080-66af842f22d3/console-operator/2.log" Jul 10 00:30:46.337450 ip-10-0-130-146 kubenswrapper[2570]: I0710 00:30:46.337436 2570 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7866497494-d6fcd_2b6cb4ca-79ff-42f0-b080-66af842f22d3/console-operator/1.log" Jul 10 00:30:46.337492 ip-10-0-130-146 kubenswrapper[2570]: I0710 00:30:46.337469 2570 generic.go:356] "Generic (PLEG): container finished" podID="2b6cb4ca-79ff-42f0-b080-66af842f22d3" containerID="23d074c40a057b5a145a3b2987dad227ee87a1f1a689a785235038ad300edf66" exitCode=255 Jul 10 00:30:46.337565 ip-10-0-130-146 kubenswrapper[2570]: I0710 00:30:46.337545 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-7866497494-d6fcd" event={"ID":"2b6cb4ca-79ff-42f0-b080-66af842f22d3","Type":"ContainerDied","Data":"23d074c40a057b5a145a3b2987dad227ee87a1f1a689a785235038ad300edf66"} Jul 10 00:30:46.337621 ip-10-0-130-146 kubenswrapper[2570]: I0710 00:30:46.337603 2570 scope.go:117] "RemoveContainer" containerID="d70906c40620c67eb02e63b34e6af43ec4e4415cda15c67b868c4c53492778f1" Jul 10 00:30:46.337955 ip-10-0-130-146 kubenswrapper[2570]: I0710 00:30:46.337935 2570 scope.go:117] "RemoveContainer" containerID="23d074c40a057b5a145a3b2987dad227ee87a1f1a689a785235038ad300edf66" Jul 10 00:30:46.338149 ip-10-0-130-146 kubenswrapper[2570]: E0710 00:30:46.338126 2570 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 20s restarting failed container=console-operator pod=console-operator-7866497494-d6fcd_openshift-console-operator(2b6cb4ca-79ff-42f0-b080-66af842f22d3)\"" pod="openshift-console-operator/console-operator-7866497494-d6fcd" podUID="2b6cb4ca-79ff-42f0-b080-66af842f22d3" Jul 10 00:30:46.339383 ip-10-0-130-146 kubenswrapper[2570]: I0710 00:30:46.339364 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-sp47j" event={"ID":"95631388-25a0-4948-a49d-2950902d0998","Type":"ContainerStarted","Data":"170887f16214cbd8518508bb62dfb16ca3e364b488b82fc8c9f6d7e89116e413"} Jul 10 00:30:46.339462 ip-10-0-130-146 kubenswrapper[2570]: I0710 00:30:46.339403 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-sp47j" event={"ID":"95631388-25a0-4948-a49d-2950902d0998","Type":"ContainerStarted","Data":"43e26445da2eb4b69a2772bf0ee01db7c3e3c6bb2b7525dccea66b63f3d783b1"} Jul 10 00:30:47.344901 ip-10-0-130-146 kubenswrapper[2570]: I0710 00:30:47.344872 2570 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7866497494-d6fcd_2b6cb4ca-79ff-42f0-b080-66af842f22d3/console-operator/2.log" Jul 10 00:30:49.355615 ip-10-0-130-146 kubenswrapper[2570]: I0710 00:30:49.355580 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-sp47j" event={"ID":"95631388-25a0-4948-a49d-2950902d0998","Type":"ContainerStarted","Data":"04cf2e776a30556c137b3e0c9270023793c3708eadaaf5163d8ace5dc8f78202"} Jul 10 00:30:49.378356 ip-10-0-130-146 kubenswrapper[2570]: I0710 00:30:49.378305 2570 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-insights/insights-runtime-extractor-sp47j" podStartSLOduration=17.480491336 podStartE2EDuration="24.378289185s" totalImagesPullingTime="6.897797849s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-10 00:30:25 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 00:30:49.376464225 +0000 UTC m=+268.424842948" watchObservedRunningTime="2026-07-10 00:30:49.378289185 +0000 UTC m=+268.426667906" Jul 10 00:30:50.971098 ip-10-0-130-146 kubenswrapper[2570]: I0710 00:30:50.971041 2570 kubelet.go:2654] "SyncLoop (probe)" probe="liveness" status="unhealthy" pod="openshift-console-operator/console-operator-7866497494-d6fcd" Jul 10 00:30:50.971498 ip-10-0-130-146 kubenswrapper[2570]: I0710 00:30:50.971117 2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console-operator/console-operator-7866497494-d6fcd" Jul 10 00:30:50.971498 ip-10-0-130-146 kubenswrapper[2570]: I0710 00:30:50.971416 2570 scope.go:117] "RemoveContainer" containerID="23d074c40a057b5a145a3b2987dad227ee87a1f1a689a785235038ad300edf66" Jul 10 00:30:50.971569 ip-10-0-130-146 kubenswrapper[2570]: E0710 00:30:50.971548 2570 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 20s restarting failed container=console-operator pod=console-operator-7866497494-d6fcd_openshift-console-operator(2b6cb4ca-79ff-42f0-b080-66af842f22d3)\"" pod="openshift-console-operator/console-operator-7866497494-d6fcd" podUID="2b6cb4ca-79ff-42f0-b080-66af842f22d3" Jul 10 00:30:52.559297 ip-10-0-130-146 kubenswrapper[2570]: I0710 00:30:52.559251 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/d4b4eb0f-539c-4193-907e-cbff010aae7b-networking-console-plugin-cert\") pod \"networking-console-plugin-6cf856758c-45x56\" (UID: \"d4b4eb0f-539c-4193-907e-cbff010aae7b\") " pod="openshift-network-console/networking-console-plugin-6cf856758c-45x56" Jul 10 00:30:52.561518 ip-10-0-130-146 kubenswrapper[2570]: I0710 00:30:52.561496 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/d4b4eb0f-539c-4193-907e-cbff010aae7b-networking-console-plugin-cert\") pod \"networking-console-plugin-6cf856758c-45x56\" (UID: \"d4b4eb0f-539c-4193-907e-cbff010aae7b\") " pod="openshift-network-console/networking-console-plugin-6cf856758c-45x56" Jul 10 00:30:52.858665 ip-10-0-130-146 kubenswrapper[2570]: I0710 00:30:52.858578 2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-console\"/\"default-dockercfg-rtz98\"" Jul 10 00:30:52.866930 ip-10-0-130-146 kubenswrapper[2570]: I0710 00:30:52.866910 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-console/networking-console-plugin-6cf856758c-45x56" Jul 10 00:30:52.997580 ip-10-0-130-146 kubenswrapper[2570]: I0710 00:30:52.997551 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-console/networking-console-plugin-6cf856758c-45x56"] Jul 10 00:30:53.000309 ip-10-0-130-146 kubenswrapper[2570]: W0710 00:30:53.000271 2570 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd4b4eb0f_539c_4193_907e_cbff010aae7b.slice/crio-ef0b6cf6beadacc8f9cfd0314af5c725a332fb0e27944198261d34ec78613ada WatchSource:0}: Error finding container ef0b6cf6beadacc8f9cfd0314af5c725a332fb0e27944198261d34ec78613ada: Status 404 returned error can't find the container with id ef0b6cf6beadacc8f9cfd0314af5c725a332fb0e27944198261d34ec78613ada Jul 10 00:30:53.369173 ip-10-0-130-146 kubenswrapper[2570]: I0710 00:30:53.369138 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-console/networking-console-plugin-6cf856758c-45x56" event={"ID":"d4b4eb0f-539c-4193-907e-cbff010aae7b","Type":"ContainerStarted","Data":"ef0b6cf6beadacc8f9cfd0314af5c725a332fb0e27944198261d34ec78613ada"} Jul 10 00:30:54.373542 ip-10-0-130-146 kubenswrapper[2570]: I0710 00:30:54.373498 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-console/networking-console-plugin-6cf856758c-45x56" event={"ID":"d4b4eb0f-539c-4193-907e-cbff010aae7b","Type":"ContainerStarted","Data":"be55f23bba6b6f2b19269ab4823e88e7d75f11592c1f277a4c10cc013e3e0ffc"} Jul 10 00:30:54.388127 ip-10-0-130-146 kubenswrapper[2570]: I0710 00:30:54.388071 2570 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-console/networking-console-plugin-6cf856758c-45x56" podStartSLOduration=33.18636874 podStartE2EDuration="34.38803816s" totalImagesPullingTime="1.20166942s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-10 00:30:20 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 00:30:54.387572208 +0000 UTC m=+273.435950930" watchObservedRunningTime="2026-07-10 00:30:54.38803816 +0000 UTC m=+273.436416895" Jul 10 00:30:55.535671 ip-10-0-130-146 kubenswrapper[2570]: I0710 00:30:55.535639 2570 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-operator-admission-webhook-77d98bc786-7br7w"] Jul 10 00:30:55.539111 ip-10-0-130-146 kubenswrapper[2570]: I0710 00:30:55.539095 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-admission-webhook-77d98bc786-7br7w" Jul 10 00:30:55.541592 ip-10-0-130-146 kubenswrapper[2570]: I0710 00:30:55.541572 2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-admission-webhook-dockercfg-tvf9t\"" Jul 10 00:30:55.541703 ip-10-0-130-146 kubenswrapper[2570]: I0710 00:30:55.541624 2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-admission-webhook-tls\"" Jul 10 00:30:55.548623 ip-10-0-130-146 kubenswrapper[2570]: I0710 00:30:55.548603 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-admission-webhook-77d98bc786-7br7w"] Jul 10 00:30:55.582985 ip-10-0-130-146 kubenswrapper[2570]: I0710 00:30:55.582961 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/be57dd6d-f83f-443f-9211-dc40cea396cc-tls-certificates\") pod \"prometheus-operator-admission-webhook-77d98bc786-7br7w\" (UID: \"be57dd6d-f83f-443f-9211-dc40cea396cc\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-77d98bc786-7br7w" Jul 10 00:30:55.683955 ip-10-0-130-146 kubenswrapper[2570]: I0710 00:30:55.683930 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/be57dd6d-f83f-443f-9211-dc40cea396cc-tls-certificates\") pod \"prometheus-operator-admission-webhook-77d98bc786-7br7w\" (UID: \"be57dd6d-f83f-443f-9211-dc40cea396cc\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-77d98bc786-7br7w" Jul 10 00:30:55.686311 ip-10-0-130-146 kubenswrapper[2570]: I0710 00:30:55.686294 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/be57dd6d-f83f-443f-9211-dc40cea396cc-tls-certificates\") pod \"prometheus-operator-admission-webhook-77d98bc786-7br7w\" (UID: \"be57dd6d-f83f-443f-9211-dc40cea396cc\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-77d98bc786-7br7w" Jul 10 00:30:55.848244 ip-10-0-130-146 kubenswrapper[2570]: I0710 00:30:55.848163 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-admission-webhook-77d98bc786-7br7w" Jul 10 00:30:55.964562 ip-10-0-130-146 kubenswrapper[2570]: I0710 00:30:55.964539 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-admission-webhook-77d98bc786-7br7w"] Jul 10 00:30:55.966864 ip-10-0-130-146 kubenswrapper[2570]: W0710 00:30:55.966839 2570 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbe57dd6d_f83f_443f_9211_dc40cea396cc.slice/crio-cd7c5d2538b18ec1a97f40868b6cd5eb31e47060e4147f84de4dfa71ccd85189 WatchSource:0}: Error finding container cd7c5d2538b18ec1a97f40868b6cd5eb31e47060e4147f84de4dfa71ccd85189: Status 404 returned error can't find the container with id cd7c5d2538b18ec1a97f40868b6cd5eb31e47060e4147f84de4dfa71ccd85189 Jul 10 00:30:56.006767 ip-10-0-130-146 kubenswrapper[2570]: I0710 00:30:56.006739 2570 patch_prober.go:28] interesting pod/image-registry-b9fdbf76c-kptvw container/registry namespace/openshift-image-registry: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body={"errors":[{"code":"UNAVAILABLE","message":"service unavailable","detail":"health check failed: please see /debug/health"}]} Jul 10 00:30:56.006874 ip-10-0-130-146 kubenswrapper[2570]: I0710 00:30:56.006790 2570 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-image-registry/image-registry-b9fdbf76c-kptvw" podUID="57c018ee-b976-4f56-a33b-b41cb28c4417" containerName="registry" probeResult="failure" output="HTTP probe failed with statuscode: 503" Jul 10 00:30:56.380538 ip-10-0-130-146 kubenswrapper[2570]: I0710 00:30:56.380506 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-admission-webhook-77d98bc786-7br7w" event={"ID":"be57dd6d-f83f-443f-9211-dc40cea396cc","Type":"ContainerStarted","Data":"cd7c5d2538b18ec1a97f40868b6cd5eb31e47060e4147f84de4dfa71ccd85189"} Jul 10 00:30:57.385094 ip-10-0-130-146 kubenswrapper[2570]: I0710 00:30:57.385034 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-admission-webhook-77d98bc786-7br7w" event={"ID":"be57dd6d-f83f-443f-9211-dc40cea396cc","Type":"ContainerStarted","Data":"10d53c4f71f531a62a9512f1d86d89ab951deb0f743b667ba54238e0dfb22beb"} Jul 10 00:30:57.385556 ip-10-0-130-146 kubenswrapper[2570]: I0710 00:30:57.385256 2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/prometheus-operator-admission-webhook-77d98bc786-7br7w" Jul 10 00:30:57.389845 ip-10-0-130-146 kubenswrapper[2570]: I0710 00:30:57.389824 2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/prometheus-operator-admission-webhook-77d98bc786-7br7w" Jul 10 00:30:57.399567 ip-10-0-130-146 kubenswrapper[2570]: I0710 00:30:57.399526 2570 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/prometheus-operator-admission-webhook-77d98bc786-7br7w" podStartSLOduration=1.213928064 podStartE2EDuration="2.399511831s" totalImagesPullingTime="1.185583767s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-10 00:30:55 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 00:30:57.398315552 +0000 UTC m=+276.446694275" watchObservedRunningTime="2026-07-10 00:30:57.399511831 +0000 UTC m=+276.447890587" Jul 10 00:30:57.593687 ip-10-0-130-146 kubenswrapper[2570]: I0710 00:30:57.593603 2570 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-operator-64785b6675-499ct"] Jul 10 00:30:57.596715 ip-10-0-130-146 kubenswrapper[2570]: I0710 00:30:57.596697 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-64785b6675-499ct" Jul 10 00:30:57.599511 ip-10-0-130-146 kubenswrapper[2570]: I0710 00:30:57.599479 2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\"" Jul 10 00:30:57.599511 ip-10-0-130-146 kubenswrapper[2570]: I0710 00:30:57.599502 2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\"" Jul 10 00:30:57.599757 ip-10-0-130-146 kubenswrapper[2570]: I0710 00:30:57.599741 2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-kube-rbac-proxy-config\"" Jul 10 00:30:57.599834 ip-10-0-130-146 kubenswrapper[2570]: I0710 00:30:57.599743 2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-dockercfg-qjtm6\"" Jul 10 00:30:57.599890 ip-10-0-130-146 kubenswrapper[2570]: I0710 00:30:57.599822 2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-tls\"" Jul 10 00:30:57.599890 ip-10-0-130-146 kubenswrapper[2570]: I0710 00:30:57.599874 2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\"" Jul 10 00:30:57.604586 ip-10-0-130-146 kubenswrapper[2570]: I0710 00:30:57.604569 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-64785b6675-499ct"] Jul 10 00:30:57.701022 ip-10-0-130-146 kubenswrapper[2570]: I0710 00:30:57.700993 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-operator-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/15c29606-9069-42e0-8db8-2dae9df7f5a2-prometheus-operator-kube-rbac-proxy-config\") pod \"prometheus-operator-64785b6675-499ct\" (UID: \"15c29606-9069-42e0-8db8-2dae9df7f5a2\") " pod="openshift-monitoring/prometheus-operator-64785b6675-499ct" Jul 10 00:30:57.701153 ip-10-0-130-146 kubenswrapper[2570]: I0710 00:30:57.701027 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/15c29606-9069-42e0-8db8-2dae9df7f5a2-prometheus-operator-tls\") pod \"prometheus-operator-64785b6675-499ct\" (UID: \"15c29606-9069-42e0-8db8-2dae9df7f5a2\") " pod="openshift-monitoring/prometheus-operator-64785b6675-499ct" Jul 10 00:30:57.701153 ip-10-0-130-146 kubenswrapper[2570]: I0710 00:30:57.701090 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5pp7h\" (UniqueName: \"kubernetes.io/projected/15c29606-9069-42e0-8db8-2dae9df7f5a2-kube-api-access-5pp7h\") pod \"prometheus-operator-64785b6675-499ct\" (UID: \"15c29606-9069-42e0-8db8-2dae9df7f5a2\") " pod="openshift-monitoring/prometheus-operator-64785b6675-499ct" Jul 10 00:30:57.701153 ip-10-0-130-146 kubenswrapper[2570]: I0710 00:30:57.701128 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/15c29606-9069-42e0-8db8-2dae9df7f5a2-metrics-client-ca\") pod \"prometheus-operator-64785b6675-499ct\" (UID: \"15c29606-9069-42e0-8db8-2dae9df7f5a2\") " pod="openshift-monitoring/prometheus-operator-64785b6675-499ct" Jul 10 00:30:57.801979 ip-10-0-130-146 kubenswrapper[2570]: I0710 00:30:57.801949 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5pp7h\" (UniqueName: \"kubernetes.io/projected/15c29606-9069-42e0-8db8-2dae9df7f5a2-kube-api-access-5pp7h\") pod \"prometheus-operator-64785b6675-499ct\" (UID: \"15c29606-9069-42e0-8db8-2dae9df7f5a2\") " pod="openshift-monitoring/prometheus-operator-64785b6675-499ct" Jul 10 00:30:57.802147 ip-10-0-130-146 kubenswrapper[2570]: I0710 00:30:57.801994 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/15c29606-9069-42e0-8db8-2dae9df7f5a2-metrics-client-ca\") pod \"prometheus-operator-64785b6675-499ct\" (UID: \"15c29606-9069-42e0-8db8-2dae9df7f5a2\") " pod="openshift-monitoring/prometheus-operator-64785b6675-499ct" Jul 10 00:30:57.802147 ip-10-0-130-146 kubenswrapper[2570]: I0710 00:30:57.802082 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-operator-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/15c29606-9069-42e0-8db8-2dae9df7f5a2-prometheus-operator-kube-rbac-proxy-config\") pod \"prometheus-operator-64785b6675-499ct\" (UID: \"15c29606-9069-42e0-8db8-2dae9df7f5a2\") " pod="openshift-monitoring/prometheus-operator-64785b6675-499ct" Jul 10 00:30:57.802147 ip-10-0-130-146 kubenswrapper[2570]: I0710 00:30:57.802108 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/15c29606-9069-42e0-8db8-2dae9df7f5a2-prometheus-operator-tls\") pod \"prometheus-operator-64785b6675-499ct\" (UID: \"15c29606-9069-42e0-8db8-2dae9df7f5a2\") " pod="openshift-monitoring/prometheus-operator-64785b6675-499ct" Jul 10 00:30:57.802817 ip-10-0-130-146 kubenswrapper[2570]: I0710 00:30:57.802798 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/15c29606-9069-42e0-8db8-2dae9df7f5a2-metrics-client-ca\") pod \"prometheus-operator-64785b6675-499ct\" (UID: \"15c29606-9069-42e0-8db8-2dae9df7f5a2\") " pod="openshift-monitoring/prometheus-operator-64785b6675-499ct" Jul 10 00:30:57.804612 ip-10-0-130-146 kubenswrapper[2570]: I0710 00:30:57.804583 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-operator-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/15c29606-9069-42e0-8db8-2dae9df7f5a2-prometheus-operator-kube-rbac-proxy-config\") pod \"prometheus-operator-64785b6675-499ct\" (UID: \"15c29606-9069-42e0-8db8-2dae9df7f5a2\") " pod="openshift-monitoring/prometheus-operator-64785b6675-499ct" Jul 10 00:30:57.804704 ip-10-0-130-146 kubenswrapper[2570]: I0710 00:30:57.804678 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/15c29606-9069-42e0-8db8-2dae9df7f5a2-prometheus-operator-tls\") pod \"prometheus-operator-64785b6675-499ct\" (UID: \"15c29606-9069-42e0-8db8-2dae9df7f5a2\") " pod="openshift-monitoring/prometheus-operator-64785b6675-499ct" Jul 10 00:30:57.809418 ip-10-0-130-146 kubenswrapper[2570]: I0710 00:30:57.809397 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5pp7h\" (UniqueName: \"kubernetes.io/projected/15c29606-9069-42e0-8db8-2dae9df7f5a2-kube-api-access-5pp7h\") pod \"prometheus-operator-64785b6675-499ct\" (UID: \"15c29606-9069-42e0-8db8-2dae9df7f5a2\") " pod="openshift-monitoring/prometheus-operator-64785b6675-499ct" Jul 10 00:30:57.906537 ip-10-0-130-146 kubenswrapper[2570]: I0710 00:30:57.906512 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-64785b6675-499ct" Jul 10 00:30:58.023792 ip-10-0-130-146 kubenswrapper[2570]: I0710 00:30:58.023663 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-64785b6675-499ct"] Jul 10 00:30:58.026211 ip-10-0-130-146 kubenswrapper[2570]: W0710 00:30:58.026186 2570 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod15c29606_9069_42e0_8db8_2dae9df7f5a2.slice/crio-fb601be12b0730f12aa560450ccee12221135cba130738b454eff6d24af70477 WatchSource:0}: Error finding container fb601be12b0730f12aa560450ccee12221135cba130738b454eff6d24af70477: Status 404 returned error can't find the container with id fb601be12b0730f12aa560450ccee12221135cba130738b454eff6d24af70477 Jul 10 00:30:58.388884 ip-10-0-130-146 kubenswrapper[2570]: I0710 00:30:58.388855 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-64785b6675-499ct" event={"ID":"15c29606-9069-42e0-8db8-2dae9df7f5a2","Type":"ContainerStarted","Data":"fb601be12b0730f12aa560450ccee12221135cba130738b454eff6d24af70477"} Jul 10 00:31:00.396436 ip-10-0-130-146 kubenswrapper[2570]: I0710 00:31:00.396402 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-64785b6675-499ct" event={"ID":"15c29606-9069-42e0-8db8-2dae9df7f5a2","Type":"ContainerStarted","Data":"5a2b51fccff8a8d2a82a2b7fd93fb6910551323a438ebbcf4aac6b9a24901f4e"} Jul 10 00:31:00.396810 ip-10-0-130-146 kubenswrapper[2570]: I0710 00:31:00.396454 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-64785b6675-499ct" event={"ID":"15c29606-9069-42e0-8db8-2dae9df7f5a2","Type":"ContainerStarted","Data":"a3b3ef04f6bf727a7f8bde29c0e9968d1f51094d558123c56ea280039e99da35"} Jul 10 00:31:00.415330 ip-10-0-130-146 kubenswrapper[2570]: I0710 00:31:00.415269 2570 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/prometheus-operator-64785b6675-499ct" podStartSLOduration=1.703249384 podStartE2EDuration="3.415254019s" totalImagesPullingTime="1.712004635s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-10 00:30:57 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 00:31:00.414197897 +0000 UTC m=+279.462576618" watchObservedRunningTime="2026-07-10 00:31:00.415254019 +0000 UTC m=+279.463632739" Jul 10 00:31:01.937224 ip-10-0-130-146 kubenswrapper[2570]: I0710 00:31:01.937194 2570 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/openshift-state-metrics-84bd68b896-wh4d2"] Jul 10 00:31:01.940544 ip-10-0-130-146 kubenswrapper[2570]: I0710 00:31:01.940527 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/openshift-state-metrics-84bd68b896-wh4d2" Jul 10 00:31:01.943944 ip-10-0-130-146 kubenswrapper[2570]: I0710 00:31:01.943926 2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-tls\"" Jul 10 00:31:01.944741 ip-10-0-130-146 kubenswrapper[2570]: I0710 00:31:01.944717 2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-kube-rbac-proxy-config\"" Jul 10 00:31:01.946022 ip-10-0-130-146 kubenswrapper[2570]: I0710 00:31:01.946006 2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-dockercfg-bth8f\"" Jul 10 00:31:01.954115 ip-10-0-130-146 kubenswrapper[2570]: I0710 00:31:01.954096 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/openshift-state-metrics-84bd68b896-wh4d2"] Jul 10 00:31:01.957561 ip-10-0-130-146 kubenswrapper[2570]: I0710 00:31:01.957538 2570 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/kube-state-metrics-785778ff74-xxgzv"] Jul 10 00:31:01.961169 ip-10-0-130-146 kubenswrapper[2570]: I0710 00:31:01.961152 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/kube-state-metrics-785778ff74-xxgzv" Jul 10 00:31:01.963889 ip-10-0-130-146 kubenswrapper[2570]: I0710 00:31:01.963867 2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-tls\"" Jul 10 00:31:01.964038 ip-10-0-130-146 kubenswrapper[2570]: I0710 00:31:01.963991 2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-custom-resource-state-configmap\"" Jul 10 00:31:01.964272 ip-10-0-130-146 kubenswrapper[2570]: I0710 00:31:01.964089 2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-kube-rbac-proxy-config\"" Jul 10 00:31:01.964272 ip-10-0-130-146 kubenswrapper[2570]: I0710 00:31:01.964126 2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-dockercfg-25vtq\"" Jul 10 00:31:01.973565 ip-10-0-130-146 kubenswrapper[2570]: I0710 00:31:01.973543 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/kube-state-metrics-785778ff74-xxgzv"] Jul 10 00:31:01.977777 ip-10-0-130-146 kubenswrapper[2570]: I0710 00:31:01.977759 2570 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/node-exporter-pfjbn"] Jul 10 00:31:01.980857 ip-10-0-130-146 kubenswrapper[2570]: I0710 00:31:01.980841 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-pfjbn" Jul 10 00:31:01.983786 ip-10-0-130-146 kubenswrapper[2570]: I0710 00:31:01.983590 2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\"" Jul 10 00:31:01.983786 ip-10-0-130-146 kubenswrapper[2570]: I0710 00:31:01.983593 2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\"" Jul 10 00:31:01.983786 ip-10-0-130-146 kubenswrapper[2570]: I0710 00:31:01.983658 2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\"" Jul 10 00:31:01.983786 ip-10-0-130-146 kubenswrapper[2570]: I0710 00:31:01.983681 2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-qzs5v\"" Jul 10 00:31:02.031485 ip-10-0-130-146 kubenswrapper[2570]: I0710 00:31:02.031457 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/d2f81651-c77b-4ce3-b388-5753557860e3-node-exporter-tls\") pod \"node-exporter-pfjbn\" (UID: \"d2f81651-c77b-4ce3-b388-5753557860e3\") " pod="openshift-monitoring/node-exporter-pfjbn" Jul 10 00:31:02.031485 ip-10-0-130-146 kubenswrapper[2570]: I0710 00:31:02.031489 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/8d7a11c5-d953-4dd7-8d43-dd1e0e4582ff-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-785778ff74-xxgzv\" (UID: \"8d7a11c5-d953-4dd7-8d43-dd1e0e4582ff\") " pod="openshift-monitoring/kube-state-metrics-785778ff74-xxgzv" Jul 10 00:31:02.031690 ip-10-0-130-146 kubenswrapper[2570]: I0710 00:31:02.031512 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/d2f81651-c77b-4ce3-b388-5753557860e3-sys\") pod \"node-exporter-pfjbn\" (UID: \"d2f81651-c77b-4ce3-b388-5753557860e3\") " pod="openshift-monitoring/node-exporter-pfjbn" Jul 10 00:31:02.031690 ip-10-0-130-146 kubenswrapper[2570]: I0710 00:31:02.031527 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/d2f81651-c77b-4ce3-b388-5753557860e3-node-exporter-accelerators-collector-config\") pod \"node-exporter-pfjbn\" (UID: \"d2f81651-c77b-4ce3-b388-5753557860e3\") " pod="openshift-monitoring/node-exporter-pfjbn" Jul 10 00:31:02.031690 ip-10-0-130-146 kubenswrapper[2570]: I0710 00:31:02.031608 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/d2f81651-c77b-4ce3-b388-5753557860e3-node-exporter-textfile\") pod \"node-exporter-pfjbn\" (UID: \"d2f81651-c77b-4ce3-b388-5753557860e3\") " pod="openshift-monitoring/node-exporter-pfjbn" Jul 10 00:31:02.031690 ip-10-0-130-146 kubenswrapper[2570]: I0710 00:31:02.031659 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/8d7a11c5-d953-4dd7-8d43-dd1e0e4582ff-volume-directive-shadow\") pod \"kube-state-metrics-785778ff74-xxgzv\" (UID: \"8d7a11c5-d953-4dd7-8d43-dd1e0e4582ff\") " pod="openshift-monitoring/kube-state-metrics-785778ff74-xxgzv" Jul 10 00:31:02.031690 ip-10-0-130-146 kubenswrapper[2570]: I0710 00:31:02.031684 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kv5c7\" (UniqueName: \"kubernetes.io/projected/d2f81651-c77b-4ce3-b388-5753557860e3-kube-api-access-kv5c7\") pod \"node-exporter-pfjbn\" (UID: \"d2f81651-c77b-4ce3-b388-5753557860e3\") " pod="openshift-monitoring/node-exporter-pfjbn" Jul 10 00:31:02.031921 ip-10-0-130-146 kubenswrapper[2570]: I0710 00:31:02.031702 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/2ffcf64d-9a08-46c6-9302-c20df48e965b-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-84bd68b896-wh4d2\" (UID: \"2ffcf64d-9a08-46c6-9302-c20df48e965b\") " pod="openshift-monitoring/openshift-state-metrics-84bd68b896-wh4d2" Jul 10 00:31:02.031921 ip-10-0-130-146 kubenswrapper[2570]: I0710 00:31:02.031723 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-n5tfq\" (UniqueName: \"kubernetes.io/projected/2ffcf64d-9a08-46c6-9302-c20df48e965b-kube-api-access-n5tfq\") pod \"openshift-state-metrics-84bd68b896-wh4d2\" (UID: \"2ffcf64d-9a08-46c6-9302-c20df48e965b\") " pod="openshift-monitoring/openshift-state-metrics-84bd68b896-wh4d2" Jul 10 00:31:02.031921 ip-10-0-130-146 kubenswrapper[2570]: I0710 00:31:02.031785 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/8d7a11c5-d953-4dd7-8d43-dd1e0e4582ff-metrics-client-ca\") pod \"kube-state-metrics-785778ff74-xxgzv\" (UID: \"8d7a11c5-d953-4dd7-8d43-dd1e0e4582ff\") " pod="openshift-monitoring/kube-state-metrics-785778ff74-xxgzv" Jul 10 00:31:02.031921 ip-10-0-130-146 kubenswrapper[2570]: I0710 00:31:02.031805 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/8d7a11c5-d953-4dd7-8d43-dd1e0e4582ff-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-785778ff74-xxgzv\" (UID: \"8d7a11c5-d953-4dd7-8d43-dd1e0e4582ff\") " pod="openshift-monitoring/kube-state-metrics-785778ff74-xxgzv" Jul 10 00:31:02.031921 ip-10-0-130-146 kubenswrapper[2570]: I0710 00:31:02.031837 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fczfd\" (UniqueName: \"kubernetes.io/projected/8d7a11c5-d953-4dd7-8d43-dd1e0e4582ff-kube-api-access-fczfd\") pod \"kube-state-metrics-785778ff74-xxgzv\" (UID: \"8d7a11c5-d953-4dd7-8d43-dd1e0e4582ff\") " pod="openshift-monitoring/kube-state-metrics-785778ff74-xxgzv" Jul 10 00:31:02.031921 ip-10-0-130-146 kubenswrapper[2570]: I0710 00:31:02.031885 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/d2f81651-c77b-4ce3-b388-5753557860e3-root\") pod \"node-exporter-pfjbn\" (UID: \"d2f81651-c77b-4ce3-b388-5753557860e3\") " pod="openshift-monitoring/node-exporter-pfjbn" Jul 10 00:31:02.031921 ip-10-0-130-146 kubenswrapper[2570]: I0710 00:31:02.031910 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/2ffcf64d-9a08-46c6-9302-c20df48e965b-metrics-client-ca\") pod \"openshift-state-metrics-84bd68b896-wh4d2\" (UID: \"2ffcf64d-9a08-46c6-9302-c20df48e965b\") " pod="openshift-monitoring/openshift-state-metrics-84bd68b896-wh4d2" Jul 10 00:31:02.032331 ip-10-0-130-146 kubenswrapper[2570]: I0710 00:31:02.031939 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/d2f81651-c77b-4ce3-b388-5753557860e3-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-pfjbn\" (UID: \"d2f81651-c77b-4ce3-b388-5753557860e3\") " pod="openshift-monitoring/node-exporter-pfjbn" Jul 10 00:31:02.032331 ip-10-0-130-146 kubenswrapper[2570]: I0710 00:31:02.031967 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/2ffcf64d-9a08-46c6-9302-c20df48e965b-openshift-state-metrics-tls\") pod \"openshift-state-metrics-84bd68b896-wh4d2\" (UID: \"2ffcf64d-9a08-46c6-9302-c20df48e965b\") " pod="openshift-monitoring/openshift-state-metrics-84bd68b896-wh4d2" Jul 10 00:31:02.032331 ip-10-0-130-146 kubenswrapper[2570]: I0710 00:31:02.032019 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/8d7a11c5-d953-4dd7-8d43-dd1e0e4582ff-kube-state-metrics-tls\") pod \"kube-state-metrics-785778ff74-xxgzv\" (UID: \"8d7a11c5-d953-4dd7-8d43-dd1e0e4582ff\") " pod="openshift-monitoring/kube-state-metrics-785778ff74-xxgzv" Jul 10 00:31:02.032331 ip-10-0-130-146 kubenswrapper[2570]: I0710 00:31:02.032066 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/d2f81651-c77b-4ce3-b388-5753557860e3-metrics-client-ca\") pod \"node-exporter-pfjbn\" (UID: \"d2f81651-c77b-4ce3-b388-5753557860e3\") " pod="openshift-monitoring/node-exporter-pfjbn" Jul 10 00:31:02.032331 ip-10-0-130-146 kubenswrapper[2570]: I0710 00:31:02.032092 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/d2f81651-c77b-4ce3-b388-5753557860e3-node-exporter-wtmp\") pod \"node-exporter-pfjbn\" (UID: \"d2f81651-c77b-4ce3-b388-5753557860e3\") " pod="openshift-monitoring/node-exporter-pfjbn" Jul 10 00:31:02.132539 ip-10-0-130-146 kubenswrapper[2570]: I0710 00:31:02.132503 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n5tfq\" (UniqueName: \"kubernetes.io/projected/2ffcf64d-9a08-46c6-9302-c20df48e965b-kube-api-access-n5tfq\") pod \"openshift-state-metrics-84bd68b896-wh4d2\" (UID: \"2ffcf64d-9a08-46c6-9302-c20df48e965b\") " pod="openshift-monitoring/openshift-state-metrics-84bd68b896-wh4d2" Jul 10 00:31:02.132539 ip-10-0-130-146 kubenswrapper[2570]: I0710 00:31:02.132541 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/8d7a11c5-d953-4dd7-8d43-dd1e0e4582ff-metrics-client-ca\") pod \"kube-state-metrics-785778ff74-xxgzv\" (UID: \"8d7a11c5-d953-4dd7-8d43-dd1e0e4582ff\") " pod="openshift-monitoring/kube-state-metrics-785778ff74-xxgzv" Jul 10 00:31:02.132777 ip-10-0-130-146 kubenswrapper[2570]: I0710 00:31:02.132559 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/8d7a11c5-d953-4dd7-8d43-dd1e0e4582ff-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-785778ff74-xxgzv\" (UID: \"8d7a11c5-d953-4dd7-8d43-dd1e0e4582ff\") " pod="openshift-monitoring/kube-state-metrics-785778ff74-xxgzv" Jul 10 00:31:02.132777 ip-10-0-130-146 kubenswrapper[2570]: I0710 00:31:02.132583 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fczfd\" (UniqueName: \"kubernetes.io/projected/8d7a11c5-d953-4dd7-8d43-dd1e0e4582ff-kube-api-access-fczfd\") pod \"kube-state-metrics-785778ff74-xxgzv\" (UID: \"8d7a11c5-d953-4dd7-8d43-dd1e0e4582ff\") " pod="openshift-monitoring/kube-state-metrics-785778ff74-xxgzv" Jul 10 00:31:02.132777 ip-10-0-130-146 kubenswrapper[2570]: I0710 00:31:02.132607 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/d2f81651-c77b-4ce3-b388-5753557860e3-root\") pod \"node-exporter-pfjbn\" (UID: \"d2f81651-c77b-4ce3-b388-5753557860e3\") " pod="openshift-monitoring/node-exporter-pfjbn" Jul 10 00:31:02.132777 ip-10-0-130-146 kubenswrapper[2570]: I0710 00:31:02.132623 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/2ffcf64d-9a08-46c6-9302-c20df48e965b-metrics-client-ca\") pod \"openshift-state-metrics-84bd68b896-wh4d2\" (UID: \"2ffcf64d-9a08-46c6-9302-c20df48e965b\") " pod="openshift-monitoring/openshift-state-metrics-84bd68b896-wh4d2" Jul 10 00:31:02.132777 ip-10-0-130-146 kubenswrapper[2570]: I0710 00:31:02.132642 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/d2f81651-c77b-4ce3-b388-5753557860e3-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-pfjbn\" (UID: \"d2f81651-c77b-4ce3-b388-5753557860e3\") " pod="openshift-monitoring/node-exporter-pfjbn" Jul 10 00:31:02.132777 ip-10-0-130-146 kubenswrapper[2570]: I0710 00:31:02.132670 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/2ffcf64d-9a08-46c6-9302-c20df48e965b-openshift-state-metrics-tls\") pod \"openshift-state-metrics-84bd68b896-wh4d2\" (UID: \"2ffcf64d-9a08-46c6-9302-c20df48e965b\") " pod="openshift-monitoring/openshift-state-metrics-84bd68b896-wh4d2" Jul 10 00:31:02.132777 ip-10-0-130-146 kubenswrapper[2570]: I0710 00:31:02.132700 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/8d7a11c5-d953-4dd7-8d43-dd1e0e4582ff-kube-state-metrics-tls\") pod \"kube-state-metrics-785778ff74-xxgzv\" (UID: \"8d7a11c5-d953-4dd7-8d43-dd1e0e4582ff\") " pod="openshift-monitoring/kube-state-metrics-785778ff74-xxgzv" Jul 10 00:31:02.132777 ip-10-0-130-146 kubenswrapper[2570]: I0710 00:31:02.132716 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"root\" (UniqueName: \"kubernetes.io/host-path/d2f81651-c77b-4ce3-b388-5753557860e3-root\") pod \"node-exporter-pfjbn\" (UID: \"d2f81651-c77b-4ce3-b388-5753557860e3\") " pod="openshift-monitoring/node-exporter-pfjbn" Jul 10 00:31:02.132777 ip-10-0-130-146 kubenswrapper[2570]: I0710 00:31:02.132740 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/d2f81651-c77b-4ce3-b388-5753557860e3-metrics-client-ca\") pod \"node-exporter-pfjbn\" (UID: \"d2f81651-c77b-4ce3-b388-5753557860e3\") " pod="openshift-monitoring/node-exporter-pfjbn" Jul 10 00:31:02.132777 ip-10-0-130-146 kubenswrapper[2570]: I0710 00:31:02.132766 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/d2f81651-c77b-4ce3-b388-5753557860e3-node-exporter-wtmp\") pod \"node-exporter-pfjbn\" (UID: \"d2f81651-c77b-4ce3-b388-5753557860e3\") " pod="openshift-monitoring/node-exporter-pfjbn" Jul 10 00:31:02.133298 ip-10-0-130-146 kubenswrapper[2570]: I0710 00:31:02.132809 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/d2f81651-c77b-4ce3-b388-5753557860e3-node-exporter-tls\") pod \"node-exporter-pfjbn\" (UID: \"d2f81651-c77b-4ce3-b388-5753557860e3\") " pod="openshift-monitoring/node-exporter-pfjbn" Jul 10 00:31:02.133298 ip-10-0-130-146 kubenswrapper[2570]: I0710 00:31:02.132843 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/8d7a11c5-d953-4dd7-8d43-dd1e0e4582ff-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-785778ff74-xxgzv\" (UID: \"8d7a11c5-d953-4dd7-8d43-dd1e0e4582ff\") " pod="openshift-monitoring/kube-state-metrics-785778ff74-xxgzv" Jul 10 00:31:02.133298 ip-10-0-130-146 kubenswrapper[2570]: I0710 00:31:02.132871 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/d2f81651-c77b-4ce3-b388-5753557860e3-sys\") pod \"node-exporter-pfjbn\" (UID: \"d2f81651-c77b-4ce3-b388-5753557860e3\") " pod="openshift-monitoring/node-exporter-pfjbn" Jul 10 00:31:02.133298 ip-10-0-130-146 kubenswrapper[2570]: I0710 00:31:02.132896 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/d2f81651-c77b-4ce3-b388-5753557860e3-node-exporter-accelerators-collector-config\") pod \"node-exporter-pfjbn\" (UID: \"d2f81651-c77b-4ce3-b388-5753557860e3\") " pod="openshift-monitoring/node-exporter-pfjbn" Jul 10 00:31:02.133298 ip-10-0-130-146 kubenswrapper[2570]: I0710 00:31:02.132923 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/d2f81651-c77b-4ce3-b388-5753557860e3-node-exporter-textfile\") pod \"node-exporter-pfjbn\" (UID: \"d2f81651-c77b-4ce3-b388-5753557860e3\") " pod="openshift-monitoring/node-exporter-pfjbn" Jul 10 00:31:02.133298 ip-10-0-130-146 kubenswrapper[2570]: I0710 00:31:02.133019 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/d2f81651-c77b-4ce3-b388-5753557860e3-sys\") pod \"node-exporter-pfjbn\" (UID: \"d2f81651-c77b-4ce3-b388-5753557860e3\") " pod="openshift-monitoring/node-exporter-pfjbn" Jul 10 00:31:02.133298 ip-10-0-130-146 kubenswrapper[2570]: I0710 00:31:02.133028 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/d2f81651-c77b-4ce3-b388-5753557860e3-node-exporter-wtmp\") pod \"node-exporter-pfjbn\" (UID: \"d2f81651-c77b-4ce3-b388-5753557860e3\") " pod="openshift-monitoring/node-exporter-pfjbn" Jul 10 00:31:02.133298 ip-10-0-130-146 kubenswrapper[2570]: E0710 00:31:02.133142 2570 secret.go:189] Couldn't get secret openshift-monitoring/node-exporter-tls: secret "node-exporter-tls" not found Jul 10 00:31:02.133298 ip-10-0-130-146 kubenswrapper[2570]: E0710 00:31:02.133207 2570 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d2f81651-c77b-4ce3-b388-5753557860e3-node-exporter-tls podName:d2f81651-c77b-4ce3-b388-5753557860e3 nodeName:}" failed. No retries permitted until 2026-07-10 00:31:02.633189802 +0000 UTC m=+281.681568501 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "node-exporter-tls" (UniqueName: "kubernetes.io/secret/d2f81651-c77b-4ce3-b388-5753557860e3-node-exporter-tls") pod "node-exporter-pfjbn" (UID: "d2f81651-c77b-4ce3-b388-5753557860e3") : secret "node-exporter-tls" not found Jul 10 00:31:02.133739 ip-10-0-130-146 kubenswrapper[2570]: I0710 00:31:02.133611 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/d2f81651-c77b-4ce3-b388-5753557860e3-metrics-client-ca\") pod \"node-exporter-pfjbn\" (UID: \"d2f81651-c77b-4ce3-b388-5753557860e3\") " pod="openshift-monitoring/node-exporter-pfjbn" Jul 10 00:31:02.133793 ip-10-0-130-146 kubenswrapper[2570]: I0710 00:31:02.133751 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/8d7a11c5-d953-4dd7-8d43-dd1e0e4582ff-metrics-client-ca\") pod \"kube-state-metrics-785778ff74-xxgzv\" (UID: \"8d7a11c5-d953-4dd7-8d43-dd1e0e4582ff\") " pod="openshift-monitoring/kube-state-metrics-785778ff74-xxgzv" Jul 10 00:31:02.133845 ip-10-0-130-146 kubenswrapper[2570]: I0710 00:31:02.132961 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/8d7a11c5-d953-4dd7-8d43-dd1e0e4582ff-volume-directive-shadow\") pod \"kube-state-metrics-785778ff74-xxgzv\" (UID: \"8d7a11c5-d953-4dd7-8d43-dd1e0e4582ff\") " pod="openshift-monitoring/kube-state-metrics-785778ff74-xxgzv" Jul 10 00:31:02.133896 ip-10-0-130-146 kubenswrapper[2570]: I0710 00:31:02.133846 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/8d7a11c5-d953-4dd7-8d43-dd1e0e4582ff-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-785778ff74-xxgzv\" (UID: \"8d7a11c5-d953-4dd7-8d43-dd1e0e4582ff\") " pod="openshift-monitoring/kube-state-metrics-785778ff74-xxgzv" Jul 10 00:31:02.133896 ip-10-0-130-146 kubenswrapper[2570]: I0710 00:31:02.133883 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/d2f81651-c77b-4ce3-b388-5753557860e3-node-exporter-textfile\") pod \"node-exporter-pfjbn\" (UID: \"d2f81651-c77b-4ce3-b388-5753557860e3\") " pod="openshift-monitoring/node-exporter-pfjbn" Jul 10 00:31:02.134163 ip-10-0-130-146 kubenswrapper[2570]: I0710 00:31:02.134134 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/8d7a11c5-d953-4dd7-8d43-dd1e0e4582ff-volume-directive-shadow\") pod \"kube-state-metrics-785778ff74-xxgzv\" (UID: \"8d7a11c5-d953-4dd7-8d43-dd1e0e4582ff\") " pod="openshift-monitoring/kube-state-metrics-785778ff74-xxgzv" Jul 10 00:31:02.134261 ip-10-0-130-146 kubenswrapper[2570]: I0710 00:31:02.134176 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kv5c7\" (UniqueName: \"kubernetes.io/projected/d2f81651-c77b-4ce3-b388-5753557860e3-kube-api-access-kv5c7\") pod \"node-exporter-pfjbn\" (UID: \"d2f81651-c77b-4ce3-b388-5753557860e3\") " pod="openshift-monitoring/node-exporter-pfjbn" Jul 10 00:31:02.134261 ip-10-0-130-146 kubenswrapper[2570]: I0710 00:31:02.134213 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/2ffcf64d-9a08-46c6-9302-c20df48e965b-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-84bd68b896-wh4d2\" (UID: \"2ffcf64d-9a08-46c6-9302-c20df48e965b\") " pod="openshift-monitoring/openshift-state-metrics-84bd68b896-wh4d2" Jul 10 00:31:02.134261 ip-10-0-130-146 kubenswrapper[2570]: I0710 00:31:02.134230 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/d2f81651-c77b-4ce3-b388-5753557860e3-node-exporter-accelerators-collector-config\") pod \"node-exporter-pfjbn\" (UID: \"d2f81651-c77b-4ce3-b388-5753557860e3\") " pod="openshift-monitoring/node-exporter-pfjbn" Jul 10 00:31:02.134476 ip-10-0-130-146 kubenswrapper[2570]: I0710 00:31:02.134444 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/2ffcf64d-9a08-46c6-9302-c20df48e965b-metrics-client-ca\") pod \"openshift-state-metrics-84bd68b896-wh4d2\" (UID: \"2ffcf64d-9a08-46c6-9302-c20df48e965b\") " pod="openshift-monitoring/openshift-state-metrics-84bd68b896-wh4d2" Jul 10 00:31:02.135716 ip-10-0-130-146 kubenswrapper[2570]: I0710 00:31:02.135674 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/d2f81651-c77b-4ce3-b388-5753557860e3-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-pfjbn\" (UID: \"d2f81651-c77b-4ce3-b388-5753557860e3\") " pod="openshift-monitoring/node-exporter-pfjbn" Jul 10 00:31:02.135820 ip-10-0-130-146 kubenswrapper[2570]: I0710 00:31:02.135734 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/8d7a11c5-d953-4dd7-8d43-dd1e0e4582ff-kube-state-metrics-tls\") pod \"kube-state-metrics-785778ff74-xxgzv\" (UID: \"8d7a11c5-d953-4dd7-8d43-dd1e0e4582ff\") " pod="openshift-monitoring/kube-state-metrics-785778ff74-xxgzv" Jul 10 00:31:02.136580 ip-10-0-130-146 kubenswrapper[2570]: I0710 00:31:02.136558 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/8d7a11c5-d953-4dd7-8d43-dd1e0e4582ff-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-785778ff74-xxgzv\" (UID: \"8d7a11c5-d953-4dd7-8d43-dd1e0e4582ff\") " pod="openshift-monitoring/kube-state-metrics-785778ff74-xxgzv" Jul 10 00:31:02.136688 ip-10-0-130-146 kubenswrapper[2570]: I0710 00:31:02.136666 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/2ffcf64d-9a08-46c6-9302-c20df48e965b-openshift-state-metrics-tls\") pod \"openshift-state-metrics-84bd68b896-wh4d2\" (UID: \"2ffcf64d-9a08-46c6-9302-c20df48e965b\") " pod="openshift-monitoring/openshift-state-metrics-84bd68b896-wh4d2" Jul 10 00:31:02.136852 ip-10-0-130-146 kubenswrapper[2570]: I0710 00:31:02.136836 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/2ffcf64d-9a08-46c6-9302-c20df48e965b-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-84bd68b896-wh4d2\" (UID: \"2ffcf64d-9a08-46c6-9302-c20df48e965b\") " pod="openshift-monitoring/openshift-state-metrics-84bd68b896-wh4d2" Jul 10 00:31:02.142109 ip-10-0-130-146 kubenswrapper[2570]: I0710 00:31:02.142044 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-n5tfq\" (UniqueName: \"kubernetes.io/projected/2ffcf64d-9a08-46c6-9302-c20df48e965b-kube-api-access-n5tfq\") pod \"openshift-state-metrics-84bd68b896-wh4d2\" (UID: \"2ffcf64d-9a08-46c6-9302-c20df48e965b\") " pod="openshift-monitoring/openshift-state-metrics-84bd68b896-wh4d2" Jul 10 00:31:02.142197 ip-10-0-130-146 kubenswrapper[2570]: I0710 00:31:02.142149 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-fczfd\" (UniqueName: \"kubernetes.io/projected/8d7a11c5-d953-4dd7-8d43-dd1e0e4582ff-kube-api-access-fczfd\") pod \"kube-state-metrics-785778ff74-xxgzv\" (UID: \"8d7a11c5-d953-4dd7-8d43-dd1e0e4582ff\") " pod="openshift-monitoring/kube-state-metrics-785778ff74-xxgzv" Jul 10 00:31:02.142735 ip-10-0-130-146 kubenswrapper[2570]: I0710 00:31:02.142714 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-kv5c7\" (UniqueName: \"kubernetes.io/projected/d2f81651-c77b-4ce3-b388-5753557860e3-kube-api-access-kv5c7\") pod \"node-exporter-pfjbn\" (UID: \"d2f81651-c77b-4ce3-b388-5753557860e3\") " pod="openshift-monitoring/node-exporter-pfjbn" Jul 10 00:31:02.251721 ip-10-0-130-146 kubenswrapper[2570]: I0710 00:31:02.251644 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/openshift-state-metrics-84bd68b896-wh4d2" Jul 10 00:31:02.271857 ip-10-0-130-146 kubenswrapper[2570]: I0710 00:31:02.271830 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/kube-state-metrics-785778ff74-xxgzv" Jul 10 00:31:02.394235 ip-10-0-130-146 kubenswrapper[2570]: I0710 00:31:02.394210 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/openshift-state-metrics-84bd68b896-wh4d2"] Jul 10 00:31:02.396388 ip-10-0-130-146 kubenswrapper[2570]: W0710 00:31:02.396363 2570 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2ffcf64d_9a08_46c6_9302_c20df48e965b.slice/crio-390cf9a77c0e703c93dea5345cb50b6af4e82977bfcfaf1fd98a664eaf022e0a WatchSource:0}: Error finding container 390cf9a77c0e703c93dea5345cb50b6af4e82977bfcfaf1fd98a664eaf022e0a: Status 404 returned error can't find the container with id 390cf9a77c0e703c93dea5345cb50b6af4e82977bfcfaf1fd98a664eaf022e0a Jul 10 00:31:02.404487 ip-10-0-130-146 kubenswrapper[2570]: I0710 00:31:02.404457 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-84bd68b896-wh4d2" event={"ID":"2ffcf64d-9a08-46c6-9302-c20df48e965b","Type":"ContainerStarted","Data":"390cf9a77c0e703c93dea5345cb50b6af4e82977bfcfaf1fd98a664eaf022e0a"} Jul 10 00:31:02.413740 ip-10-0-130-146 kubenswrapper[2570]: I0710 00:31:02.413717 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/kube-state-metrics-785778ff74-xxgzv"] Jul 10 00:31:02.415532 ip-10-0-130-146 kubenswrapper[2570]: W0710 00:31:02.415506 2570 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8d7a11c5_d953_4dd7_8d43_dd1e0e4582ff.slice/crio-24adf81df56a841c25947875345ac23250ffc7a920946b2da4628b7adb8e6a8d WatchSource:0}: Error finding container 24adf81df56a841c25947875345ac23250ffc7a920946b2da4628b7adb8e6a8d: Status 404 returned error can't find the container with id 24adf81df56a841c25947875345ac23250ffc7a920946b2da4628b7adb8e6a8d Jul 10 00:31:02.448141 ip-10-0-130-146 kubenswrapper[2570]: I0710 00:31:02.448116 2570 scope.go:117] "RemoveContainer" containerID="23d074c40a057b5a145a3b2987dad227ee87a1f1a689a785235038ad300edf66" Jul 10 00:31:02.448272 ip-10-0-130-146 kubenswrapper[2570]: E0710 00:31:02.448254 2570 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 20s restarting failed container=console-operator pod=console-operator-7866497494-d6fcd_openshift-console-operator(2b6cb4ca-79ff-42f0-b080-66af842f22d3)\"" pod="openshift-console-operator/console-operator-7866497494-d6fcd" podUID="2b6cb4ca-79ff-42f0-b080-66af842f22d3" Jul 10 00:31:02.639015 ip-10-0-130-146 kubenswrapper[2570]: I0710 00:31:02.638980 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/d2f81651-c77b-4ce3-b388-5753557860e3-node-exporter-tls\") pod \"node-exporter-pfjbn\" (UID: \"d2f81651-c77b-4ce3-b388-5753557860e3\") " pod="openshift-monitoring/node-exporter-pfjbn" Jul 10 00:31:02.641183 ip-10-0-130-146 kubenswrapper[2570]: I0710 00:31:02.641160 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/d2f81651-c77b-4ce3-b388-5753557860e3-node-exporter-tls\") pod \"node-exporter-pfjbn\" (UID: \"d2f81651-c77b-4ce3-b388-5753557860e3\") " pod="openshift-monitoring/node-exporter-pfjbn" Jul 10 00:31:02.890593 ip-10-0-130-146 kubenswrapper[2570]: I0710 00:31:02.890563 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-pfjbn" Jul 10 00:31:02.902820 ip-10-0-130-146 kubenswrapper[2570]: W0710 00:31:02.902790 2570 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd2f81651_c77b_4ce3_b388_5753557860e3.slice/crio-d50aa047ad315f8337a6e543e2b099ba3837698cf757bf0ac83df0a1797a22a7 WatchSource:0}: Error finding container d50aa047ad315f8337a6e543e2b099ba3837698cf757bf0ac83df0a1797a22a7: Status 404 returned error can't find the container with id d50aa047ad315f8337a6e543e2b099ba3837698cf757bf0ac83df0a1797a22a7 Jul 10 00:31:03.411060 ip-10-0-130-146 kubenswrapper[2570]: I0710 00:31:03.411018 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-pfjbn" event={"ID":"d2f81651-c77b-4ce3-b388-5753557860e3","Type":"ContainerStarted","Data":"d50aa047ad315f8337a6e543e2b099ba3837698cf757bf0ac83df0a1797a22a7"} Jul 10 00:31:03.412612 ip-10-0-130-146 kubenswrapper[2570]: I0710 00:31:03.412583 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-785778ff74-xxgzv" event={"ID":"8d7a11c5-d953-4dd7-8d43-dd1e0e4582ff","Type":"ContainerStarted","Data":"24adf81df56a841c25947875345ac23250ffc7a920946b2da4628b7adb8e6a8d"} Jul 10 00:31:03.414746 ip-10-0-130-146 kubenswrapper[2570]: I0710 00:31:03.414717 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-84bd68b896-wh4d2" event={"ID":"2ffcf64d-9a08-46c6-9302-c20df48e965b","Type":"ContainerStarted","Data":"6541db8cd463943a8169a1f059279db0774e0f11b128a6a58dcfec8e19b8d7f7"} Jul 10 00:31:03.414827 ip-10-0-130-146 kubenswrapper[2570]: I0710 00:31:03.414773 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-84bd68b896-wh4d2" event={"ID":"2ffcf64d-9a08-46c6-9302-c20df48e965b","Type":"ContainerStarted","Data":"052bebd1c52f016819cb77dc9d96643de0ff41f23c2184c315e1fb3e271f74c8"} Jul 10 00:31:05.427761 ip-10-0-130-146 kubenswrapper[2570]: I0710 00:31:05.427725 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-84bd68b896-wh4d2" event={"ID":"2ffcf64d-9a08-46c6-9302-c20df48e965b","Type":"ContainerStarted","Data":"c0d88318eaba022b217a94938d85d8d91dad9f1363342a7d4e30b0e07204a00c"} Jul 10 00:31:05.429248 ip-10-0-130-146 kubenswrapper[2570]: I0710 00:31:05.429226 2570 generic.go:356] "Generic (PLEG): container finished" podID="d2f81651-c77b-4ce3-b388-5753557860e3" containerID="cc7cd601888e843fcf7f07b9a3aea0b721715a93396ed6e8b8bc3dab0b319847" exitCode=0 Jul 10 00:31:05.429466 ip-10-0-130-146 kubenswrapper[2570]: I0710 00:31:05.429444 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-pfjbn" event={"ID":"d2f81651-c77b-4ce3-b388-5753557860e3","Type":"ContainerDied","Data":"cc7cd601888e843fcf7f07b9a3aea0b721715a93396ed6e8b8bc3dab0b319847"} Jul 10 00:31:05.431627 ip-10-0-130-146 kubenswrapper[2570]: I0710 00:31:05.431604 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-785778ff74-xxgzv" event={"ID":"8d7a11c5-d953-4dd7-8d43-dd1e0e4582ff","Type":"ContainerStarted","Data":"dc84e744ec4054d3b34c9fd8a7a2484cf1968343810afa105ed90cb4193ff933"} Jul 10 00:31:05.431742 ip-10-0-130-146 kubenswrapper[2570]: I0710 00:31:05.431647 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-785778ff74-xxgzv" event={"ID":"8d7a11c5-d953-4dd7-8d43-dd1e0e4582ff","Type":"ContainerStarted","Data":"cf15ed7a4113b583431e766b81b144794523a95c97ee1e84463140145c38cdaa"} Jul 10 00:31:05.431742 ip-10-0-130-146 kubenswrapper[2570]: I0710 00:31:05.431668 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-785778ff74-xxgzv" event={"ID":"8d7a11c5-d953-4dd7-8d43-dd1e0e4582ff","Type":"ContainerStarted","Data":"ebc535ba8cc6befe5d41fbb47d7d9982258ef144579252acaf6b5fa220a00cdf"} Jul 10 00:31:05.448786 ip-10-0-130-146 kubenswrapper[2570]: I0710 00:31:05.448741 2570 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/openshift-state-metrics-84bd68b896-wh4d2" podStartSLOduration=2.628500757 podStartE2EDuration="4.448723944s" totalImagesPullingTime="1.820223187s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-10 00:31:01 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 00:31:05.447160077 +0000 UTC m=+284.495538800" watchObservedRunningTime="2026-07-10 00:31:05.448723944 +0000 UTC m=+284.497102667" Jul 10 00:31:05.470845 ip-10-0-130-146 kubenswrapper[2570]: I0710 00:31:05.470804 2570 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/kube-state-metrics-785778ff74-xxgzv" podStartSLOduration=2.546560048 podStartE2EDuration="4.470789588s" totalImagesPullingTime="1.92422954s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-10 00:31:01 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 00:31:05.469332428 +0000 UTC m=+284.517711178" watchObservedRunningTime="2026-07-10 00:31:05.470789588 +0000 UTC m=+284.519168309" Jul 10 00:31:06.006380 ip-10-0-130-146 kubenswrapper[2570]: I0710 00:31:06.006344 2570 patch_prober.go:28] interesting pod/image-registry-b9fdbf76c-kptvw container/registry namespace/openshift-image-registry: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body={"errors":[{"code":"UNAVAILABLE","message":"service unavailable","detail":"health check failed: please see /debug/health"}]} Jul 10 00:31:06.006524 ip-10-0-130-146 kubenswrapper[2570]: I0710 00:31:06.006400 2570 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-image-registry/image-registry-b9fdbf76c-kptvw" podUID="57c018ee-b976-4f56-a33b-b41cb28c4417" containerName="registry" probeResult="failure" output="HTTP probe failed with statuscode: 503" Jul 10 00:31:06.436987 ip-10-0-130-146 kubenswrapper[2570]: I0710 00:31:06.436947 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-pfjbn" event={"ID":"d2f81651-c77b-4ce3-b388-5753557860e3","Type":"ContainerStarted","Data":"ecfacbd4658b176276c6cdd99b1be021d30db094be2095a9874d36b72b7dd42a"} Jul 10 00:31:06.437406 ip-10-0-130-146 kubenswrapper[2570]: I0710 00:31:06.437014 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-pfjbn" event={"ID":"d2f81651-c77b-4ce3-b388-5753557860e3","Type":"ContainerStarted","Data":"dd5db1955b36b5ff78ddf80ea38a006fd667abc562ef942cc803c498cd2ad92c"} Jul 10 00:31:06.456309 ip-10-0-130-146 kubenswrapper[2570]: I0710 00:31:06.456263 2570 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/node-exporter-pfjbn" podStartSLOduration=3.8922648300000002 podStartE2EDuration="5.456248027s" totalImagesPullingTime="1.482942537s" totalInitContainerRuntime="81.04066ms" isStatefulPod=true podCreationTimestamp="2026-07-10 00:31:01 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 00:31:06.45536939 +0000 UTC m=+285.503748112" watchObservedRunningTime="2026-07-10 00:31:06.456248027 +0000 UTC m=+285.504626726" Jul 10 00:31:07.048027 ip-10-0-130-146 kubenswrapper[2570]: E0710 00:31:07.047983 2570 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[metrics-tls], unattached volumes=[], failed to process volumes=[]: context deadline exceeded" pod="openshift-dns/dns-default-fcndn" podUID="e2252bd4-69bf-4c02-a315-d9cbcf4ed91e" Jul 10 00:31:07.441429 ip-10-0-130-146 kubenswrapper[2570]: I0710 00:31:07.441401 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-fcndn" Jul 10 00:31:10.407560 ip-10-0-130-146 kubenswrapper[2570]: I0710 00:31:10.407526 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/e2252bd4-69bf-4c02-a315-d9cbcf4ed91e-metrics-tls\") pod \"dns-default-fcndn\" (UID: \"e2252bd4-69bf-4c02-a315-d9cbcf4ed91e\") " pod="openshift-dns/dns-default-fcndn" Jul 10 00:31:10.409742 ip-10-0-130-146 kubenswrapper[2570]: I0710 00:31:10.409721 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/e2252bd4-69bf-4c02-a315-d9cbcf4ed91e-metrics-tls\") pod \"dns-default-fcndn\" (UID: \"e2252bd4-69bf-4c02-a315-d9cbcf4ed91e\") " pod="openshift-dns/dns-default-fcndn" Jul 10 00:31:10.444899 ip-10-0-130-146 kubenswrapper[2570]: I0710 00:31:10.444873 2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-2cgcq\"" Jul 10 00:31:10.453304 ip-10-0-130-146 kubenswrapper[2570]: I0710 00:31:10.453287 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-fcndn" Jul 10 00:31:10.577954 ip-10-0-130-146 kubenswrapper[2570]: I0710 00:31:10.577923 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-fcndn"] Jul 10 00:31:10.580603 ip-10-0-130-146 kubenswrapper[2570]: W0710 00:31:10.580579 2570 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode2252bd4_69bf_4c02_a315_d9cbcf4ed91e.slice/crio-a70cbf62f3d627d0ab95d9d2267b1adc8e5aa65978e6020c3b1363a1e70dfd86 WatchSource:0}: Error finding container a70cbf62f3d627d0ab95d9d2267b1adc8e5aa65978e6020c3b1363a1e70dfd86: Status 404 returned error can't find the container with id a70cbf62f3d627d0ab95d9d2267b1adc8e5aa65978e6020c3b1363a1e70dfd86 Jul 10 00:31:11.022906 ip-10-0-130-146 kubenswrapper[2570]: I0710 00:31:11.022868 2570 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-image-registry/image-registry-b9fdbf76c-kptvw" podUID="57c018ee-b976-4f56-a33b-b41cb28c4417" containerName="registry" containerID="cri-o://ba389df5506b155942234611c7bfd5a44637aa550bc6cae57914c6a083d7c053" gracePeriod=30 Jul 10 00:31:11.253064 ip-10-0-130-146 kubenswrapper[2570]: I0710 00:31:11.253025 2570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-b9fdbf76c-kptvw" Jul 10 00:31:11.315520 ip-10-0-130-146 kubenswrapper[2570]: I0710 00:31:11.315444 2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/57c018ee-b976-4f56-a33b-b41cb28c4417-image-registry-private-configuration\") pod \"57c018ee-b976-4f56-a33b-b41cb28c4417\" (UID: \"57c018ee-b976-4f56-a33b-b41cb28c4417\") " Jul 10 00:31:11.315520 ip-10-0-130-146 kubenswrapper[2570]: I0710 00:31:11.315491 2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/57c018ee-b976-4f56-a33b-b41cb28c4417-bound-sa-token\") pod \"57c018ee-b976-4f56-a33b-b41cb28c4417\" (UID: \"57c018ee-b976-4f56-a33b-b41cb28c4417\") " Jul 10 00:31:11.315721 ip-10-0-130-146 kubenswrapper[2570]: I0710 00:31:11.315539 2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/57c018ee-b976-4f56-a33b-b41cb28c4417-trusted-ca\") pod \"57c018ee-b976-4f56-a33b-b41cb28c4417\" (UID: \"57c018ee-b976-4f56-a33b-b41cb28c4417\") " Jul 10 00:31:11.315721 ip-10-0-130-146 kubenswrapper[2570]: I0710 00:31:11.315566 2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/57c018ee-b976-4f56-a33b-b41cb28c4417-installation-pull-secrets\") pod \"57c018ee-b976-4f56-a33b-b41cb28c4417\" (UID: \"57c018ee-b976-4f56-a33b-b41cb28c4417\") " Jul 10 00:31:11.315721 ip-10-0-130-146 kubenswrapper[2570]: I0710 00:31:11.315605 2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/57c018ee-b976-4f56-a33b-b41cb28c4417-registry-certificates\") pod \"57c018ee-b976-4f56-a33b-b41cb28c4417\" (UID: \"57c018ee-b976-4f56-a33b-b41cb28c4417\") " Jul 10 00:31:11.315862 ip-10-0-130-146 kubenswrapper[2570]: I0710 00:31:11.315806 2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-qpsq9\" (UniqueName: \"kubernetes.io/projected/57c018ee-b976-4f56-a33b-b41cb28c4417-kube-api-access-qpsq9\") pod \"57c018ee-b976-4f56-a33b-b41cb28c4417\" (UID: \"57c018ee-b976-4f56-a33b-b41cb28c4417\") " Jul 10 00:31:11.315911 ip-10-0-130-146 kubenswrapper[2570]: I0710 00:31:11.315859 2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/57c018ee-b976-4f56-a33b-b41cb28c4417-registry-tls\") pod \"57c018ee-b976-4f56-a33b-b41cb28c4417\" (UID: \"57c018ee-b976-4f56-a33b-b41cb28c4417\") " Jul 10 00:31:11.315911 ip-10-0-130-146 kubenswrapper[2570]: I0710 00:31:11.315892 2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/57c018ee-b976-4f56-a33b-b41cb28c4417-ca-trust-extracted\") pod \"57c018ee-b976-4f56-a33b-b41cb28c4417\" (UID: \"57c018ee-b976-4f56-a33b-b41cb28c4417\") " Jul 10 00:31:11.316115 ip-10-0-130-146 kubenswrapper[2570]: I0710 00:31:11.316088 2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/57c018ee-b976-4f56-a33b-b41cb28c4417-registry-certificates" (OuterVolumeSpecName: "registry-certificates") pod "57c018ee-b976-4f56-a33b-b41cb28c4417" (UID: "57c018ee-b976-4f56-a33b-b41cb28c4417"). InnerVolumeSpecName "registry-certificates". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 10 00:31:11.316186 ip-10-0-130-146 kubenswrapper[2570]: I0710 00:31:11.316122 2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/57c018ee-b976-4f56-a33b-b41cb28c4417-trusted-ca" (OuterVolumeSpecName: "trusted-ca") pod "57c018ee-b976-4f56-a33b-b41cb28c4417" (UID: "57c018ee-b976-4f56-a33b-b41cb28c4417"). InnerVolumeSpecName "trusted-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 10 00:31:11.316374 ip-10-0-130-146 kubenswrapper[2570]: I0710 00:31:11.316328 2570 reconciler_common.go:299] "Volume detached for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/57c018ee-b976-4f56-a33b-b41cb28c4417-trusted-ca\") on node \"ip-10-0-130-146.ec2.internal\" DevicePath \"\"" Jul 10 00:31:11.316374 ip-10-0-130-146 kubenswrapper[2570]: I0710 00:31:11.316355 2570 reconciler_common.go:299] "Volume detached for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/57c018ee-b976-4f56-a33b-b41cb28c4417-registry-certificates\") on node \"ip-10-0-130-146.ec2.internal\" DevicePath \"\"" Jul 10 00:31:11.318701 ip-10-0-130-146 kubenswrapper[2570]: I0710 00:31:11.318642 2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/57c018ee-b976-4f56-a33b-b41cb28c4417-installation-pull-secrets" (OuterVolumeSpecName: "installation-pull-secrets") pod "57c018ee-b976-4f56-a33b-b41cb28c4417" (UID: "57c018ee-b976-4f56-a33b-b41cb28c4417"). InnerVolumeSpecName "installation-pull-secrets". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 10 00:31:11.318798 ip-10-0-130-146 kubenswrapper[2570]: I0710 00:31:11.318764 2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/57c018ee-b976-4f56-a33b-b41cb28c4417-registry-tls" (OuterVolumeSpecName: "registry-tls") pod "57c018ee-b976-4f56-a33b-b41cb28c4417" (UID: "57c018ee-b976-4f56-a33b-b41cb28c4417"). InnerVolumeSpecName "registry-tls". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 10 00:31:11.319331 ip-10-0-130-146 kubenswrapper[2570]: I0710 00:31:11.318925 2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/57c018ee-b976-4f56-a33b-b41cb28c4417-image-registry-private-configuration" (OuterVolumeSpecName: "image-registry-private-configuration") pod "57c018ee-b976-4f56-a33b-b41cb28c4417" (UID: "57c018ee-b976-4f56-a33b-b41cb28c4417"). InnerVolumeSpecName "image-registry-private-configuration". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 10 00:31:11.319331 ip-10-0-130-146 kubenswrapper[2570]: I0710 00:31:11.319260 2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/57c018ee-b976-4f56-a33b-b41cb28c4417-bound-sa-token" (OuterVolumeSpecName: "bound-sa-token") pod "57c018ee-b976-4f56-a33b-b41cb28c4417" (UID: "57c018ee-b976-4f56-a33b-b41cb28c4417"). InnerVolumeSpecName "bound-sa-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 10 00:31:11.319927 ip-10-0-130-146 kubenswrapper[2570]: I0710 00:31:11.319893 2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/57c018ee-b976-4f56-a33b-b41cb28c4417-kube-api-access-qpsq9" (OuterVolumeSpecName: "kube-api-access-qpsq9") pod "57c018ee-b976-4f56-a33b-b41cb28c4417" (UID: "57c018ee-b976-4f56-a33b-b41cb28c4417"). InnerVolumeSpecName "kube-api-access-qpsq9". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 10 00:31:11.331901 ip-10-0-130-146 kubenswrapper[2570]: I0710 00:31:11.331863 2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/57c018ee-b976-4f56-a33b-b41cb28c4417-ca-trust-extracted" (OuterVolumeSpecName: "ca-trust-extracted") pod "57c018ee-b976-4f56-a33b-b41cb28c4417" (UID: "57c018ee-b976-4f56-a33b-b41cb28c4417"). InnerVolumeSpecName "ca-trust-extracted". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 10 00:31:11.417401 ip-10-0-130-146 kubenswrapper[2570]: I0710 00:31:11.417287 2570 reconciler_common.go:299] "Volume detached for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/57c018ee-b976-4f56-a33b-b41cb28c4417-installation-pull-secrets\") on node \"ip-10-0-130-146.ec2.internal\" DevicePath \"\"" Jul 10 00:31:11.417401 ip-10-0-130-146 kubenswrapper[2570]: I0710 00:31:11.417320 2570 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qpsq9\" (UniqueName: \"kubernetes.io/projected/57c018ee-b976-4f56-a33b-b41cb28c4417-kube-api-access-qpsq9\") on node \"ip-10-0-130-146.ec2.internal\" DevicePath \"\"" Jul 10 00:31:11.417401 ip-10-0-130-146 kubenswrapper[2570]: I0710 00:31:11.417336 2570 reconciler_common.go:299] "Volume detached for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/57c018ee-b976-4f56-a33b-b41cb28c4417-registry-tls\") on node \"ip-10-0-130-146.ec2.internal\" DevicePath \"\"" Jul 10 00:31:11.417401 ip-10-0-130-146 kubenswrapper[2570]: I0710 00:31:11.417350 2570 reconciler_common.go:299] "Volume detached for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/57c018ee-b976-4f56-a33b-b41cb28c4417-ca-trust-extracted\") on node \"ip-10-0-130-146.ec2.internal\" DevicePath \"\"" Jul 10 00:31:11.417401 ip-10-0-130-146 kubenswrapper[2570]: I0710 00:31:11.417365 2570 reconciler_common.go:299] "Volume detached for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/57c018ee-b976-4f56-a33b-b41cb28c4417-image-registry-private-configuration\") on node \"ip-10-0-130-146.ec2.internal\" DevicePath \"\"" Jul 10 00:31:11.417401 ip-10-0-130-146 kubenswrapper[2570]: I0710 00:31:11.417380 2570 reconciler_common.go:299] "Volume detached for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/57c018ee-b976-4f56-a33b-b41cb28c4417-bound-sa-token\") on node \"ip-10-0-130-146.ec2.internal\" DevicePath \"\"" Jul 10 00:31:11.458992 ip-10-0-130-146 kubenswrapper[2570]: I0710 00:31:11.458952 2570 generic.go:356] "Generic (PLEG): container finished" podID="57c018ee-b976-4f56-a33b-b41cb28c4417" containerID="ba389df5506b155942234611c7bfd5a44637aa550bc6cae57914c6a083d7c053" exitCode=0 Jul 10 00:31:11.459167 ip-10-0-130-146 kubenswrapper[2570]: I0710 00:31:11.459013 2570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-b9fdbf76c-kptvw" Jul 10 00:31:11.459167 ip-10-0-130-146 kubenswrapper[2570]: I0710 00:31:11.459043 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-b9fdbf76c-kptvw" event={"ID":"57c018ee-b976-4f56-a33b-b41cb28c4417","Type":"ContainerDied","Data":"ba389df5506b155942234611c7bfd5a44637aa550bc6cae57914c6a083d7c053"} Jul 10 00:31:11.459167 ip-10-0-130-146 kubenswrapper[2570]: I0710 00:31:11.459112 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-b9fdbf76c-kptvw" event={"ID":"57c018ee-b976-4f56-a33b-b41cb28c4417","Type":"ContainerDied","Data":"a08461e74e45d3f3ba334adc4149f969361c0262d9133e74506bddaea6f3f11b"} Jul 10 00:31:11.459167 ip-10-0-130-146 kubenswrapper[2570]: I0710 00:31:11.459136 2570 scope.go:117] "RemoveContainer" containerID="ba389df5506b155942234611c7bfd5a44637aa550bc6cae57914c6a083d7c053" Jul 10 00:31:11.460852 ip-10-0-130-146 kubenswrapper[2570]: I0710 00:31:11.460824 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-fcndn" event={"ID":"e2252bd4-69bf-4c02-a315-d9cbcf4ed91e","Type":"ContainerStarted","Data":"a70cbf62f3d627d0ab95d9d2267b1adc8e5aa65978e6020c3b1363a1e70dfd86"} Jul 10 00:31:11.470552 ip-10-0-130-146 kubenswrapper[2570]: I0710 00:31:11.470536 2570 scope.go:117] "RemoveContainer" containerID="ba389df5506b155942234611c7bfd5a44637aa550bc6cae57914c6a083d7c053" Jul 10 00:31:11.470904 ip-10-0-130-146 kubenswrapper[2570]: E0710 00:31:11.470880 2570 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ba389df5506b155942234611c7bfd5a44637aa550bc6cae57914c6a083d7c053\": container with ID starting with ba389df5506b155942234611c7bfd5a44637aa550bc6cae57914c6a083d7c053 not found: ID does not exist" containerID="ba389df5506b155942234611c7bfd5a44637aa550bc6cae57914c6a083d7c053" Jul 10 00:31:11.470965 ip-10-0-130-146 kubenswrapper[2570]: I0710 00:31:11.470916 2570 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ba389df5506b155942234611c7bfd5a44637aa550bc6cae57914c6a083d7c053"} err="failed to get container status \"ba389df5506b155942234611c7bfd5a44637aa550bc6cae57914c6a083d7c053\": rpc error: code = NotFound desc = could not find container \"ba389df5506b155942234611c7bfd5a44637aa550bc6cae57914c6a083d7c053\": container with ID starting with ba389df5506b155942234611c7bfd5a44637aa550bc6cae57914c6a083d7c053 not found: ID does not exist" Jul 10 00:31:11.480072 ip-10-0-130-146 kubenswrapper[2570]: I0710 00:31:11.480026 2570 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-b9fdbf76c-kptvw"] Jul 10 00:31:11.483575 ip-10-0-130-146 kubenswrapper[2570]: I0710 00:31:11.483550 2570 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-image-registry/image-registry-b9fdbf76c-kptvw"] Jul 10 00:31:13.452696 ip-10-0-130-146 kubenswrapper[2570]: I0710 00:31:13.452665 2570 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="57c018ee-b976-4f56-a33b-b41cb28c4417" path="/var/lib/kubelet/pods/57c018ee-b976-4f56-a33b-b41cb28c4417/volumes" Jul 10 00:31:13.470524 ip-10-0-130-146 kubenswrapper[2570]: I0710 00:31:13.470495 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-fcndn" event={"ID":"e2252bd4-69bf-4c02-a315-d9cbcf4ed91e","Type":"ContainerStarted","Data":"5f543649df0faefb26145893c2ea23d9fe4fed232daec4207ff6379d1281cc9b"} Jul 10 00:31:13.470680 ip-10-0-130-146 kubenswrapper[2570]: I0710 00:31:13.470538 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-fcndn" event={"ID":"e2252bd4-69bf-4c02-a315-d9cbcf4ed91e","Type":"ContainerStarted","Data":"174af6c21912fbe44e9d00450bcdf024ca7e8fdf15f032404704fa3d1cb91bf8"} Jul 10 00:31:13.470680 ip-10-0-130-146 kubenswrapper[2570]: I0710 00:31:13.470637 2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-dns/dns-default-fcndn" Jul 10 00:31:13.486933 ip-10-0-130-146 kubenswrapper[2570]: I0710 00:31:13.486889 2570 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/dns-default-fcndn" podStartSLOduration=251.64315294 podStartE2EDuration="4m13.486875528s" totalImagesPullingTime="1.843722588s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-10 00:27:00 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 00:31:13.485250744 +0000 UTC m=+292.533629466" watchObservedRunningTime="2026-07-10 00:31:13.486875528 +0000 UTC m=+292.535254249" Jul 10 00:31:17.448530 ip-10-0-130-146 kubenswrapper[2570]: I0710 00:31:17.448492 2570 scope.go:117] "RemoveContainer" containerID="23d074c40a057b5a145a3b2987dad227ee87a1f1a689a785235038ad300edf66" Jul 10 00:31:18.248992 ip-10-0-130-146 kubenswrapper[2570]: I0710 00:31:18.248953 2570 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/downloads-68c9cbb497-4prsj"] Jul 10 00:31:18.249330 ip-10-0-130-146 kubenswrapper[2570]: I0710 00:31:18.249313 2570 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="57c018ee-b976-4f56-a33b-b41cb28c4417" containerName="registry" Jul 10 00:31:18.249408 ip-10-0-130-146 kubenswrapper[2570]: I0710 00:31:18.249333 2570 state_mem.go:107] "Deleted CPUSet assignment" podUID="57c018ee-b976-4f56-a33b-b41cb28c4417" containerName="registry" Jul 10 00:31:18.249463 ip-10-0-130-146 kubenswrapper[2570]: I0710 00:31:18.249406 2570 memory_manager.go:368] "RemoveStaleState removing state" podUID="57c018ee-b976-4f56-a33b-b41cb28c4417" containerName="registry" Jul 10 00:31:18.251191 ip-10-0-130-146 kubenswrapper[2570]: I0710 00:31:18.251171 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/downloads-68c9cbb497-4prsj" Jul 10 00:31:18.253397 ip-10-0-130-146 kubenswrapper[2570]: I0710 00:31:18.253376 2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"kube-root-ca.crt\"" Jul 10 00:31:18.253490 ip-10-0-130-146 kubenswrapper[2570]: I0710 00:31:18.253454 2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"default-dockercfg-kjh7k\"" Jul 10 00:31:18.253490 ip-10-0-130-146 kubenswrapper[2570]: I0710 00:31:18.253459 2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"openshift-service-ca.crt\"" Jul 10 00:31:18.261962 ip-10-0-130-146 kubenswrapper[2570]: I0710 00:31:18.261941 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/downloads-68c9cbb497-4prsj"] Jul 10 00:31:18.372917 ip-10-0-130-146 kubenswrapper[2570]: I0710 00:31:18.372881 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-b4b45\" (UniqueName: \"kubernetes.io/projected/e05c8204-bfbd-4bce-8cda-9a56ef492fcb-kube-api-access-b4b45\") pod \"downloads-68c9cbb497-4prsj\" (UID: \"e05c8204-bfbd-4bce-8cda-9a56ef492fcb\") " pod="openshift-console/downloads-68c9cbb497-4prsj" Jul 10 00:31:18.473880 ip-10-0-130-146 kubenswrapper[2570]: I0710 00:31:18.473847 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b4b45\" (UniqueName: \"kubernetes.io/projected/e05c8204-bfbd-4bce-8cda-9a56ef492fcb-kube-api-access-b4b45\") pod \"downloads-68c9cbb497-4prsj\" (UID: \"e05c8204-bfbd-4bce-8cda-9a56ef492fcb\") " pod="openshift-console/downloads-68c9cbb497-4prsj" Jul 10 00:31:18.481805 ip-10-0-130-146 kubenswrapper[2570]: I0710 00:31:18.481779 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-b4b45\" (UniqueName: \"kubernetes.io/projected/e05c8204-bfbd-4bce-8cda-9a56ef492fcb-kube-api-access-b4b45\") pod \"downloads-68c9cbb497-4prsj\" (UID: \"e05c8204-bfbd-4bce-8cda-9a56ef492fcb\") " pod="openshift-console/downloads-68c9cbb497-4prsj" Jul 10 00:31:18.488800 ip-10-0-130-146 kubenswrapper[2570]: I0710 00:31:18.488781 2570 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7866497494-d6fcd_2b6cb4ca-79ff-42f0-b080-66af842f22d3/console-operator/2.log" Jul 10 00:31:18.488910 ip-10-0-130-146 kubenswrapper[2570]: I0710 00:31:18.488891 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-7866497494-d6fcd" event={"ID":"2b6cb4ca-79ff-42f0-b080-66af842f22d3","Type":"ContainerStarted","Data":"bae051e5f09af41ee75a3ecfff3baedd9c6cb85cb4ab5bc1c352334acd0847df"} Jul 10 00:31:18.489172 ip-10-0-130-146 kubenswrapper[2570]: I0710 00:31:18.489158 2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console-operator/console-operator-7866497494-d6fcd" Jul 10 00:31:18.493871 ip-10-0-130-146 kubenswrapper[2570]: I0710 00:31:18.493850 2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console-operator/console-operator-7866497494-d6fcd" Jul 10 00:31:18.506797 ip-10-0-130-146 kubenswrapper[2570]: I0710 00:31:18.506726 2570 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console-operator/console-operator-7866497494-d6fcd" podStartSLOduration=55.343804242 podStartE2EDuration="58.506712286s" totalImagesPullingTime="3.162908044s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-10 00:30:20 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 00:31:18.506111123 +0000 UTC m=+297.554489857" watchObservedRunningTime="2026-07-10 00:31:18.506712286 +0000 UTC m=+297.555090986" Jul 10 00:31:18.560312 ip-10-0-130-146 kubenswrapper[2570]: I0710 00:31:18.560290 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/downloads-68c9cbb497-4prsj" Jul 10 00:31:18.684760 ip-10-0-130-146 kubenswrapper[2570]: I0710 00:31:18.684738 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/downloads-68c9cbb497-4prsj"] Jul 10 00:31:18.687330 ip-10-0-130-146 kubenswrapper[2570]: W0710 00:31:18.687302 2570 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode05c8204_bfbd_4bce_8cda_9a56ef492fcb.slice/crio-2a6136a855d18cdc18e74fc8537a4d03e6c85c26095e732bc5a07252bf83f8ce WatchSource:0}: Error finding container 2a6136a855d18cdc18e74fc8537a4d03e6c85c26095e732bc5a07252bf83f8ce: Status 404 returned error can't find the container with id 2a6136a855d18cdc18e74fc8537a4d03e6c85c26095e732bc5a07252bf83f8ce Jul 10 00:31:19.494942 ip-10-0-130-146 kubenswrapper[2570]: I0710 00:31:19.494902 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/downloads-68c9cbb497-4prsj" event={"ID":"e05c8204-bfbd-4bce-8cda-9a56ef492fcb","Type":"ContainerStarted","Data":"2a6136a855d18cdc18e74fc8537a4d03e6c85c26095e732bc5a07252bf83f8ce"} Jul 10 00:31:20.449988 ip-10-0-130-146 kubenswrapper[2570]: E0710 00:31:20.449948 2570 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[metrics-certs], unattached volumes=[], failed to process volumes=[]: context deadline exceeded" pod="openshift-multus/network-metrics-daemon-d8t56" podUID="b7aab806-00a2-491e-a997-ed3174bd547a" Jul 10 00:31:21.259904 ip-10-0-130-146 kubenswrapper[2570]: I0710 00:31:21.259876 2570 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7866497494-d6fcd_2b6cb4ca-79ff-42f0-b080-66af842f22d3/console-operator/2.log" Jul 10 00:31:21.260312 ip-10-0-130-146 kubenswrapper[2570]: I0710 00:31:21.259936 2570 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7866497494-d6fcd_2b6cb4ca-79ff-42f0-b080-66af842f22d3/console-operator/2.log" Jul 10 00:31:21.263940 ip-10-0-130-146 kubenswrapper[2570]: I0710 00:31:21.263923 2570 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-mbs58_c4ef019a-364a-4933-ab85-de86d86167f1/ovn-acl-logging/0.log" Jul 10 00:31:21.264066 ip-10-0-130-146 kubenswrapper[2570]: I0710 00:31:21.263925 2570 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-mbs58_c4ef019a-364a-4933-ab85-de86d86167f1/ovn-acl-logging/0.log" Jul 10 00:31:21.270888 ip-10-0-130-146 kubenswrapper[2570]: I0710 00:31:21.270870 2570 kubelet.go:1648] "Image garbage collection succeeded" Jul 10 00:31:23.476396 ip-10-0-130-146 kubenswrapper[2570]: I0710 00:31:23.476363 2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-dns/dns-default-fcndn" Jul 10 00:31:28.156108 ip-10-0-130-146 kubenswrapper[2570]: I0710 00:31:28.156070 2570 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-56ccc595db-rn7fp"] Jul 10 00:31:28.163431 ip-10-0-130-146 kubenswrapper[2570]: I0710 00:31:28.163405 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-56ccc595db-rn7fp" Jul 10 00:31:28.165960 ip-10-0-130-146 kubenswrapper[2570]: I0710 00:31:28.165925 2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"console-config\"" Jul 10 00:31:28.166378 ip-10-0-130-146 kubenswrapper[2570]: I0710 00:31:28.166190 2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"service-ca\"" Jul 10 00:31:28.166378 ip-10-0-130-146 kubenswrapper[2570]: I0710 00:31:28.165957 2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"oauth-serving-cert\"" Jul 10 00:31:28.166378 ip-10-0-130-146 kubenswrapper[2570]: I0710 00:31:28.166004 2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-oauth-config\"" Jul 10 00:31:28.166378 ip-10-0-130-146 kubenswrapper[2570]: I0710 00:31:28.166014 2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-serving-cert\"" Jul 10 00:31:28.166378 ip-10-0-130-146 kubenswrapper[2570]: I0710 00:31:28.165925 2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-dockercfg-vg244\"" Jul 10 00:31:28.168708 ip-10-0-130-146 kubenswrapper[2570]: I0710 00:31:28.168686 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-56ccc595db-rn7fp"] Jul 10 00:31:28.254538 ip-10-0-130-146 kubenswrapper[2570]: I0710 00:31:28.254504 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/c0223a0c-8f9d-4a17-b9cc-557b1abba0e8-console-oauth-config\") pod \"console-56ccc595db-rn7fp\" (UID: \"c0223a0c-8f9d-4a17-b9cc-557b1abba0e8\") " pod="openshift-console/console-56ccc595db-rn7fp" Jul 10 00:31:28.254708 ip-10-0-130-146 kubenswrapper[2570]: I0710 00:31:28.254574 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/c0223a0c-8f9d-4a17-b9cc-557b1abba0e8-console-serving-cert\") pod \"console-56ccc595db-rn7fp\" (UID: \"c0223a0c-8f9d-4a17-b9cc-557b1abba0e8\") " pod="openshift-console/console-56ccc595db-rn7fp" Jul 10 00:31:28.254708 ip-10-0-130-146 kubenswrapper[2570]: I0710 00:31:28.254620 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/c0223a0c-8f9d-4a17-b9cc-557b1abba0e8-console-config\") pod \"console-56ccc595db-rn7fp\" (UID: \"c0223a0c-8f9d-4a17-b9cc-557b1abba0e8\") " pod="openshift-console/console-56ccc595db-rn7fp" Jul 10 00:31:28.254823 ip-10-0-130-146 kubenswrapper[2570]: I0710 00:31:28.254744 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-t5ndm\" (UniqueName: \"kubernetes.io/projected/c0223a0c-8f9d-4a17-b9cc-557b1abba0e8-kube-api-access-t5ndm\") pod \"console-56ccc595db-rn7fp\" (UID: \"c0223a0c-8f9d-4a17-b9cc-557b1abba0e8\") " pod="openshift-console/console-56ccc595db-rn7fp" Jul 10 00:31:28.254823 ip-10-0-130-146 kubenswrapper[2570]: I0710 00:31:28.254786 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/c0223a0c-8f9d-4a17-b9cc-557b1abba0e8-service-ca\") pod \"console-56ccc595db-rn7fp\" (UID: \"c0223a0c-8f9d-4a17-b9cc-557b1abba0e8\") " pod="openshift-console/console-56ccc595db-rn7fp" Jul 10 00:31:28.254913 ip-10-0-130-146 kubenswrapper[2570]: I0710 00:31:28.254830 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/c0223a0c-8f9d-4a17-b9cc-557b1abba0e8-oauth-serving-cert\") pod \"console-56ccc595db-rn7fp\" (UID: \"c0223a0c-8f9d-4a17-b9cc-557b1abba0e8\") " pod="openshift-console/console-56ccc595db-rn7fp" Jul 10 00:31:28.355859 ip-10-0-130-146 kubenswrapper[2570]: I0710 00:31:28.355820 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/c0223a0c-8f9d-4a17-b9cc-557b1abba0e8-oauth-serving-cert\") pod \"console-56ccc595db-rn7fp\" (UID: \"c0223a0c-8f9d-4a17-b9cc-557b1abba0e8\") " pod="openshift-console/console-56ccc595db-rn7fp" Jul 10 00:31:28.356026 ip-10-0-130-146 kubenswrapper[2570]: I0710 00:31:28.355885 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/c0223a0c-8f9d-4a17-b9cc-557b1abba0e8-console-oauth-config\") pod \"console-56ccc595db-rn7fp\" (UID: \"c0223a0c-8f9d-4a17-b9cc-557b1abba0e8\") " pod="openshift-console/console-56ccc595db-rn7fp" Jul 10 00:31:28.356026 ip-10-0-130-146 kubenswrapper[2570]: I0710 00:31:28.355945 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/c0223a0c-8f9d-4a17-b9cc-557b1abba0e8-console-serving-cert\") pod \"console-56ccc595db-rn7fp\" (UID: \"c0223a0c-8f9d-4a17-b9cc-557b1abba0e8\") " pod="openshift-console/console-56ccc595db-rn7fp" Jul 10 00:31:28.356026 ip-10-0-130-146 kubenswrapper[2570]: I0710 00:31:28.355968 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/c0223a0c-8f9d-4a17-b9cc-557b1abba0e8-console-config\") pod \"console-56ccc595db-rn7fp\" (UID: \"c0223a0c-8f9d-4a17-b9cc-557b1abba0e8\") " pod="openshift-console/console-56ccc595db-rn7fp" Jul 10 00:31:28.356026 ip-10-0-130-146 kubenswrapper[2570]: I0710 00:31:28.356001 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t5ndm\" (UniqueName: \"kubernetes.io/projected/c0223a0c-8f9d-4a17-b9cc-557b1abba0e8-kube-api-access-t5ndm\") pod \"console-56ccc595db-rn7fp\" (UID: \"c0223a0c-8f9d-4a17-b9cc-557b1abba0e8\") " pod="openshift-console/console-56ccc595db-rn7fp" Jul 10 00:31:28.356258 ip-10-0-130-146 kubenswrapper[2570]: I0710 00:31:28.356036 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/c0223a0c-8f9d-4a17-b9cc-557b1abba0e8-service-ca\") pod \"console-56ccc595db-rn7fp\" (UID: \"c0223a0c-8f9d-4a17-b9cc-557b1abba0e8\") " pod="openshift-console/console-56ccc595db-rn7fp" Jul 10 00:31:28.356724 ip-10-0-130-146 kubenswrapper[2570]: I0710 00:31:28.356701 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/c0223a0c-8f9d-4a17-b9cc-557b1abba0e8-service-ca\") pod \"console-56ccc595db-rn7fp\" (UID: \"c0223a0c-8f9d-4a17-b9cc-557b1abba0e8\") " pod="openshift-console/console-56ccc595db-rn7fp" Jul 10 00:31:28.356819 ip-10-0-130-146 kubenswrapper[2570]: I0710 00:31:28.356788 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/c0223a0c-8f9d-4a17-b9cc-557b1abba0e8-console-config\") pod \"console-56ccc595db-rn7fp\" (UID: \"c0223a0c-8f9d-4a17-b9cc-557b1abba0e8\") " pod="openshift-console/console-56ccc595db-rn7fp" Jul 10 00:31:28.357446 ip-10-0-130-146 kubenswrapper[2570]: I0710 00:31:28.357397 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/c0223a0c-8f9d-4a17-b9cc-557b1abba0e8-oauth-serving-cert\") pod \"console-56ccc595db-rn7fp\" (UID: \"c0223a0c-8f9d-4a17-b9cc-557b1abba0e8\") " pod="openshift-console/console-56ccc595db-rn7fp" Jul 10 00:31:28.359364 ip-10-0-130-146 kubenswrapper[2570]: I0710 00:31:28.359333 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/c0223a0c-8f9d-4a17-b9cc-557b1abba0e8-console-oauth-config\") pod \"console-56ccc595db-rn7fp\" (UID: \"c0223a0c-8f9d-4a17-b9cc-557b1abba0e8\") " pod="openshift-console/console-56ccc595db-rn7fp" Jul 10 00:31:28.359675 ip-10-0-130-146 kubenswrapper[2570]: I0710 00:31:28.359655 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/c0223a0c-8f9d-4a17-b9cc-557b1abba0e8-console-serving-cert\") pod \"console-56ccc595db-rn7fp\" (UID: \"c0223a0c-8f9d-4a17-b9cc-557b1abba0e8\") " pod="openshift-console/console-56ccc595db-rn7fp" Jul 10 00:31:28.365725 ip-10-0-130-146 kubenswrapper[2570]: I0710 00:31:28.365702 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-t5ndm\" (UniqueName: \"kubernetes.io/projected/c0223a0c-8f9d-4a17-b9cc-557b1abba0e8-kube-api-access-t5ndm\") pod \"console-56ccc595db-rn7fp\" (UID: \"c0223a0c-8f9d-4a17-b9cc-557b1abba0e8\") " pod="openshift-console/console-56ccc595db-rn7fp" Jul 10 00:31:28.475226 ip-10-0-130-146 kubenswrapper[2570]: I0710 00:31:28.475146 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-56ccc595db-rn7fp" Jul 10 00:31:28.613153 ip-10-0-130-146 kubenswrapper[2570]: I0710 00:31:28.612969 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-56ccc595db-rn7fp"] Jul 10 00:31:28.615523 ip-10-0-130-146 kubenswrapper[2570]: W0710 00:31:28.615496 2570 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc0223a0c_8f9d_4a17_b9cc_557b1abba0e8.slice/crio-e4d83b90397a11627cd5ee284762c824eda2da6e5f60a29ac881937e9ab4cbd5 WatchSource:0}: Error finding container e4d83b90397a11627cd5ee284762c824eda2da6e5f60a29ac881937e9ab4cbd5: Status 404 returned error can't find the container with id e4d83b90397a11627cd5ee284762c824eda2da6e5f60a29ac881937e9ab4cbd5 Jul 10 00:31:28.617944 ip-10-0-130-146 kubenswrapper[2570]: I0710 00:31:28.617924 2570 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Jul 10 00:31:29.537913 ip-10-0-130-146 kubenswrapper[2570]: I0710 00:31:29.537875 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-56ccc595db-rn7fp" event={"ID":"c0223a0c-8f9d-4a17-b9cc-557b1abba0e8","Type":"ContainerStarted","Data":"e4d83b90397a11627cd5ee284762c824eda2da6e5f60a29ac881937e9ab4cbd5"} Jul 10 00:31:30.544711 ip-10-0-130-146 kubenswrapper[2570]: I0710 00:31:30.544677 2570 generic.go:356] "Generic (PLEG): container finished" podID="f612c430-f495-4773-9ca6-82f7417e3a0c" containerID="ab2af976c38ea3ebd24a288184d7937ee4fefc1368113b6f9cb5b19e1f981cb1" exitCode=0 Jul 10 00:31:30.545155 ip-10-0-130-146 kubenswrapper[2570]: I0710 00:31:30.544754 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-68d9dd94cf-7rq8g" event={"ID":"f612c430-f495-4773-9ca6-82f7417e3a0c","Type":"ContainerDied","Data":"ab2af976c38ea3ebd24a288184d7937ee4fefc1368113b6f9cb5b19e1f981cb1"} Jul 10 00:31:30.545230 ip-10-0-130-146 kubenswrapper[2570]: I0710 00:31:30.545182 2570 scope.go:117] "RemoveContainer" containerID="ab2af976c38ea3ebd24a288184d7937ee4fefc1368113b6f9cb5b19e1f981cb1" Jul 10 00:31:31.450893 ip-10-0-130-146 kubenswrapper[2570]: I0710 00:31:31.450855 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-d8t56" Jul 10 00:31:31.552785 ip-10-0-130-146 kubenswrapper[2570]: I0710 00:31:31.552746 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-68d9dd94cf-7rq8g" event={"ID":"f612c430-f495-4773-9ca6-82f7417e3a0c","Type":"ContainerStarted","Data":"f942f6581878a588dbf893c6aa151b8099e4434609f619c01002c230d806256b"} Jul 10 00:31:32.559461 ip-10-0-130-146 kubenswrapper[2570]: I0710 00:31:32.559414 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-56ccc595db-rn7fp" event={"ID":"c0223a0c-8f9d-4a17-b9cc-557b1abba0e8","Type":"ContainerStarted","Data":"a4242be6af1c02a07b23a0cae810e0f03832534ba9e9cb654911ae090b7de002"} Jul 10 00:31:32.577067 ip-10-0-130-146 kubenswrapper[2570]: I0710 00:31:32.576994 2570 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-56ccc595db-rn7fp" podStartSLOduration=1.245968658 podStartE2EDuration="4.576975559s" totalImagesPullingTime="3.331006901s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-10 00:31:28 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 00:31:32.574985793 +0000 UTC m=+311.623364729" watchObservedRunningTime="2026-07-10 00:31:32.576975559 +0000 UTC m=+311.625354281" Jul 10 00:31:36.938712 ip-10-0-130-146 kubenswrapper[2570]: I0710 00:31:36.938676 2570 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-847dc9597b-l8kgk"] Jul 10 00:31:36.942584 ip-10-0-130-146 kubenswrapper[2570]: I0710 00:31:36.942558 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-847dc9597b-l8kgk" Jul 10 00:31:36.951902 ip-10-0-130-146 kubenswrapper[2570]: I0710 00:31:36.951873 2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"trusted-ca-bundle\"" Jul 10 00:31:36.952638 ip-10-0-130-146 kubenswrapper[2570]: I0710 00:31:36.952616 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-847dc9597b-l8kgk"] Jul 10 00:31:37.034861 ip-10-0-130-146 kubenswrapper[2570]: I0710 00:31:37.034808 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/f36e7fdf-3a6c-473e-a83e-452b22b2c96c-service-ca\") pod \"console-847dc9597b-l8kgk\" (UID: \"f36e7fdf-3a6c-473e-a83e-452b22b2c96c\") " pod="openshift-console/console-847dc9597b-l8kgk" Jul 10 00:31:37.034861 ip-10-0-130-146 kubenswrapper[2570]: I0710 00:31:37.034866 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f36e7fdf-3a6c-473e-a83e-452b22b2c96c-trusted-ca-bundle\") pod \"console-847dc9597b-l8kgk\" (UID: \"f36e7fdf-3a6c-473e-a83e-452b22b2c96c\") " pod="openshift-console/console-847dc9597b-l8kgk" Jul 10 00:31:37.035130 ip-10-0-130-146 kubenswrapper[2570]: I0710 00:31:37.034969 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/f36e7fdf-3a6c-473e-a83e-452b22b2c96c-console-config\") pod \"console-847dc9597b-l8kgk\" (UID: \"f36e7fdf-3a6c-473e-a83e-452b22b2c96c\") " pod="openshift-console/console-847dc9597b-l8kgk" Jul 10 00:31:37.035130 ip-10-0-130-146 kubenswrapper[2570]: I0710 00:31:37.035041 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/f36e7fdf-3a6c-473e-a83e-452b22b2c96c-console-oauth-config\") pod \"console-847dc9597b-l8kgk\" (UID: \"f36e7fdf-3a6c-473e-a83e-452b22b2c96c\") " pod="openshift-console/console-847dc9597b-l8kgk" Jul 10 00:31:37.035130 ip-10-0-130-146 kubenswrapper[2570]: I0710 00:31:37.035090 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4kg77\" (UniqueName: \"kubernetes.io/projected/f36e7fdf-3a6c-473e-a83e-452b22b2c96c-kube-api-access-4kg77\") pod \"console-847dc9597b-l8kgk\" (UID: \"f36e7fdf-3a6c-473e-a83e-452b22b2c96c\") " pod="openshift-console/console-847dc9597b-l8kgk" Jul 10 00:31:37.035255 ip-10-0-130-146 kubenswrapper[2570]: I0710 00:31:37.035130 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/f36e7fdf-3a6c-473e-a83e-452b22b2c96c-console-serving-cert\") pod \"console-847dc9597b-l8kgk\" (UID: \"f36e7fdf-3a6c-473e-a83e-452b22b2c96c\") " pod="openshift-console/console-847dc9597b-l8kgk" Jul 10 00:31:37.035255 ip-10-0-130-146 kubenswrapper[2570]: I0710 00:31:37.035162 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/f36e7fdf-3a6c-473e-a83e-452b22b2c96c-oauth-serving-cert\") pod \"console-847dc9597b-l8kgk\" (UID: \"f36e7fdf-3a6c-473e-a83e-452b22b2c96c\") " pod="openshift-console/console-847dc9597b-l8kgk" Jul 10 00:31:37.135866 ip-10-0-130-146 kubenswrapper[2570]: I0710 00:31:37.135831 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/f36e7fdf-3a6c-473e-a83e-452b22b2c96c-console-oauth-config\") pod \"console-847dc9597b-l8kgk\" (UID: \"f36e7fdf-3a6c-473e-a83e-452b22b2c96c\") " pod="openshift-console/console-847dc9597b-l8kgk" Jul 10 00:31:37.135866 ip-10-0-130-146 kubenswrapper[2570]: I0710 00:31:37.135867 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4kg77\" (UniqueName: \"kubernetes.io/projected/f36e7fdf-3a6c-473e-a83e-452b22b2c96c-kube-api-access-4kg77\") pod \"console-847dc9597b-l8kgk\" (UID: \"f36e7fdf-3a6c-473e-a83e-452b22b2c96c\") " pod="openshift-console/console-847dc9597b-l8kgk" Jul 10 00:31:37.136154 ip-10-0-130-146 kubenswrapper[2570]: I0710 00:31:37.135894 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/f36e7fdf-3a6c-473e-a83e-452b22b2c96c-console-serving-cert\") pod \"console-847dc9597b-l8kgk\" (UID: \"f36e7fdf-3a6c-473e-a83e-452b22b2c96c\") " pod="openshift-console/console-847dc9597b-l8kgk" Jul 10 00:31:37.136154 ip-10-0-130-146 kubenswrapper[2570]: I0710 00:31:37.135916 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/f36e7fdf-3a6c-473e-a83e-452b22b2c96c-oauth-serving-cert\") pod \"console-847dc9597b-l8kgk\" (UID: \"f36e7fdf-3a6c-473e-a83e-452b22b2c96c\") " pod="openshift-console/console-847dc9597b-l8kgk" Jul 10 00:31:37.136154 ip-10-0-130-146 kubenswrapper[2570]: I0710 00:31:37.135970 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/f36e7fdf-3a6c-473e-a83e-452b22b2c96c-service-ca\") pod \"console-847dc9597b-l8kgk\" (UID: \"f36e7fdf-3a6c-473e-a83e-452b22b2c96c\") " pod="openshift-console/console-847dc9597b-l8kgk" Jul 10 00:31:37.136154 ip-10-0-130-146 kubenswrapper[2570]: I0710 00:31:37.136001 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f36e7fdf-3a6c-473e-a83e-452b22b2c96c-trusted-ca-bundle\") pod \"console-847dc9597b-l8kgk\" (UID: \"f36e7fdf-3a6c-473e-a83e-452b22b2c96c\") " pod="openshift-console/console-847dc9597b-l8kgk" Jul 10 00:31:37.136154 ip-10-0-130-146 kubenswrapper[2570]: I0710 00:31:37.136063 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/f36e7fdf-3a6c-473e-a83e-452b22b2c96c-console-config\") pod \"console-847dc9597b-l8kgk\" (UID: \"f36e7fdf-3a6c-473e-a83e-452b22b2c96c\") " pod="openshift-console/console-847dc9597b-l8kgk" Jul 10 00:31:37.136813 ip-10-0-130-146 kubenswrapper[2570]: I0710 00:31:37.136769 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/f36e7fdf-3a6c-473e-a83e-452b22b2c96c-console-config\") pod \"console-847dc9597b-l8kgk\" (UID: \"f36e7fdf-3a6c-473e-a83e-452b22b2c96c\") " pod="openshift-console/console-847dc9597b-l8kgk" Jul 10 00:31:37.136948 ip-10-0-130-146 kubenswrapper[2570]: I0710 00:31:37.136870 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/f36e7fdf-3a6c-473e-a83e-452b22b2c96c-oauth-serving-cert\") pod \"console-847dc9597b-l8kgk\" (UID: \"f36e7fdf-3a6c-473e-a83e-452b22b2c96c\") " pod="openshift-console/console-847dc9597b-l8kgk" Jul 10 00:31:37.136948 ip-10-0-130-146 kubenswrapper[2570]: I0710 00:31:37.136937 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/f36e7fdf-3a6c-473e-a83e-452b22b2c96c-service-ca\") pod \"console-847dc9597b-l8kgk\" (UID: \"f36e7fdf-3a6c-473e-a83e-452b22b2c96c\") " pod="openshift-console/console-847dc9597b-l8kgk" Jul 10 00:31:37.137316 ip-10-0-130-146 kubenswrapper[2570]: I0710 00:31:37.137291 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f36e7fdf-3a6c-473e-a83e-452b22b2c96c-trusted-ca-bundle\") pod \"console-847dc9597b-l8kgk\" (UID: \"f36e7fdf-3a6c-473e-a83e-452b22b2c96c\") " pod="openshift-console/console-847dc9597b-l8kgk" Jul 10 00:31:37.138786 ip-10-0-130-146 kubenswrapper[2570]: I0710 00:31:37.138761 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/f36e7fdf-3a6c-473e-a83e-452b22b2c96c-console-oauth-config\") pod \"console-847dc9597b-l8kgk\" (UID: \"f36e7fdf-3a6c-473e-a83e-452b22b2c96c\") " pod="openshift-console/console-847dc9597b-l8kgk" Jul 10 00:31:37.139015 ip-10-0-130-146 kubenswrapper[2570]: I0710 00:31:37.138998 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/f36e7fdf-3a6c-473e-a83e-452b22b2c96c-console-serving-cert\") pod \"console-847dc9597b-l8kgk\" (UID: \"f36e7fdf-3a6c-473e-a83e-452b22b2c96c\") " pod="openshift-console/console-847dc9597b-l8kgk" Jul 10 00:31:37.143686 ip-10-0-130-146 kubenswrapper[2570]: I0710 00:31:37.143660 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4kg77\" (UniqueName: \"kubernetes.io/projected/f36e7fdf-3a6c-473e-a83e-452b22b2c96c-kube-api-access-4kg77\") pod \"console-847dc9597b-l8kgk\" (UID: \"f36e7fdf-3a6c-473e-a83e-452b22b2c96c\") " pod="openshift-console/console-847dc9597b-l8kgk" Jul 10 00:31:37.254104 ip-10-0-130-146 kubenswrapper[2570]: I0710 00:31:37.254005 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-847dc9597b-l8kgk" Jul 10 00:31:38.475297 ip-10-0-130-146 kubenswrapper[2570]: I0710 00:31:38.475263 2570 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-56ccc595db-rn7fp" Jul 10 00:31:38.475711 ip-10-0-130-146 kubenswrapper[2570]: I0710 00:31:38.475325 2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-56ccc595db-rn7fp" Jul 10 00:31:38.481599 ip-10-0-130-146 kubenswrapper[2570]: I0710 00:31:38.481568 2570 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-56ccc595db-rn7fp" Jul 10 00:31:38.589184 ip-10-0-130-146 kubenswrapper[2570]: I0710 00:31:38.589155 2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-56ccc595db-rn7fp" Jul 10 00:31:41.600680 ip-10-0-130-146 kubenswrapper[2570]: I0710 00:31:41.600601 2570 generic.go:356] "Generic (PLEG): container finished" podID="6f87cbd8-9d72-4708-a230-2818fa5a836d" containerID="e13fc7802c777f81a25e1d8dba4ec33a9812570cf4b99ab56537eedeb360ceb7" exitCode=0 Jul 10 00:31:41.600680 ip-10-0-130-146 kubenswrapper[2570]: I0710 00:31:41.600667 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-799cb7f958-p44ms" event={"ID":"6f87cbd8-9d72-4708-a230-2818fa5a836d","Type":"ContainerDied","Data":"e13fc7802c777f81a25e1d8dba4ec33a9812570cf4b99ab56537eedeb360ceb7"} Jul 10 00:31:41.601160 ip-10-0-130-146 kubenswrapper[2570]: I0710 00:31:41.601004 2570 scope.go:117] "RemoveContainer" containerID="e13fc7802c777f81a25e1d8dba4ec33a9812570cf4b99ab56537eedeb360ceb7" Jul 10 00:31:41.639671 ip-10-0-130-146 kubenswrapper[2570]: I0710 00:31:41.639645 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-847dc9597b-l8kgk"] Jul 10 00:31:41.642657 ip-10-0-130-146 kubenswrapper[2570]: W0710 00:31:41.642631 2570 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf36e7fdf_3a6c_473e_a83e_452b22b2c96c.slice/crio-b85fcc48dfb789222bfb6f5a86e004947943a445b98bf6ebc7621683bcf0ba01 WatchSource:0}: Error finding container b85fcc48dfb789222bfb6f5a86e004947943a445b98bf6ebc7621683bcf0ba01: Status 404 returned error can't find the container with id b85fcc48dfb789222bfb6f5a86e004947943a445b98bf6ebc7621683bcf0ba01 Jul 10 00:31:42.608305 ip-10-0-130-146 kubenswrapper[2570]: I0710 00:31:42.608264 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-799cb7f958-p44ms" event={"ID":"6f87cbd8-9d72-4708-a230-2818fa5a836d","Type":"ContainerStarted","Data":"833d9473b008f9fce59181175096966ec42b175bfcd200f7b73bfac06dad7d48"} Jul 10 00:31:42.610219 ip-10-0-130-146 kubenswrapper[2570]: I0710 00:31:42.610033 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/downloads-68c9cbb497-4prsj" event={"ID":"e05c8204-bfbd-4bce-8cda-9a56ef492fcb","Type":"ContainerStarted","Data":"1e08ff3b49eb72b7e7008f01b55fbdccc51b4f94360f7c4fec120e08d7ec3ae4"} Jul 10 00:31:42.610219 ip-10-0-130-146 kubenswrapper[2570]: I0710 00:31:42.610102 2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/downloads-68c9cbb497-4prsj" Jul 10 00:31:42.612648 ip-10-0-130-146 kubenswrapper[2570]: I0710 00:31:42.612607 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-847dc9597b-l8kgk" event={"ID":"f36e7fdf-3a6c-473e-a83e-452b22b2c96c","Type":"ContainerStarted","Data":"a77c6bf8521b0a680ec619e1b38d748db350c92535990433bc52f3860af3b507"} Jul 10 00:31:42.612773 ip-10-0-130-146 kubenswrapper[2570]: I0710 00:31:42.612656 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-847dc9597b-l8kgk" event={"ID":"f36e7fdf-3a6c-473e-a83e-452b22b2c96c","Type":"ContainerStarted","Data":"b85fcc48dfb789222bfb6f5a86e004947943a445b98bf6ebc7621683bcf0ba01"} Jul 10 00:31:42.616994 ip-10-0-130-146 kubenswrapper[2570]: I0710 00:31:42.616972 2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/downloads-68c9cbb497-4prsj" Jul 10 00:31:42.642489 ip-10-0-130-146 kubenswrapper[2570]: I0710 00:31:42.642414 2570 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-847dc9597b-l8kgk" podStartSLOduration=6.642394761 podStartE2EDuration="6.642394761s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-10 00:31:36 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 00:31:42.641403062 +0000 UTC m=+321.689781791" watchObservedRunningTime="2026-07-10 00:31:42.642394761 +0000 UTC m=+321.690773483" Jul 10 00:31:42.657625 ip-10-0-130-146 kubenswrapper[2570]: I0710 00:31:42.657571 2570 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/downloads-68c9cbb497-4prsj" podStartSLOduration=1.728028996 podStartE2EDuration="24.657551829s" totalImagesPullingTime="22.929522833s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-10 00:31:18 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 00:31:42.656180101 +0000 UTC m=+321.704558843" watchObservedRunningTime="2026-07-10 00:31:42.657551829 +0000 UTC m=+321.705930566" Jul 10 00:31:47.254953 ip-10-0-130-146 kubenswrapper[2570]: I0710 00:31:47.254911 2570 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-847dc9597b-l8kgk" Jul 10 00:31:47.255526 ip-10-0-130-146 kubenswrapper[2570]: I0710 00:31:47.255031 2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-847dc9597b-l8kgk" Jul 10 00:31:47.260695 ip-10-0-130-146 kubenswrapper[2570]: I0710 00:31:47.260670 2570 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-847dc9597b-l8kgk" Jul 10 00:31:47.640144 ip-10-0-130-146 kubenswrapper[2570]: I0710 00:31:47.640114 2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-847dc9597b-l8kgk" Jul 10 00:31:47.685879 ip-10-0-130-146 kubenswrapper[2570]: I0710 00:31:47.685848 2570 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-56ccc595db-rn7fp"] Jul 10 00:32:12.728238 ip-10-0-130-146 kubenswrapper[2570]: I0710 00:32:12.728194 2570 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-console/console-56ccc595db-rn7fp" podUID="c0223a0c-8f9d-4a17-b9cc-557b1abba0e8" containerName="console" containerID="cri-o://a4242be6af1c02a07b23a0cae810e0f03832534ba9e9cb654911ae090b7de002" gracePeriod=15 Jul 10 00:32:13.000015 ip-10-0-130-146 kubenswrapper[2570]: I0710 00:32:12.999954 2570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-56ccc595db-rn7fp" Jul 10 00:32:13.139245 ip-10-0-130-146 kubenswrapper[2570]: I0710 00:32:13.139213 2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/c0223a0c-8f9d-4a17-b9cc-557b1abba0e8-console-oauth-config\") pod \"c0223a0c-8f9d-4a17-b9cc-557b1abba0e8\" (UID: \"c0223a0c-8f9d-4a17-b9cc-557b1abba0e8\") " Jul 10 00:32:13.139245 ip-10-0-130-146 kubenswrapper[2570]: I0710 00:32:13.139247 2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/c0223a0c-8f9d-4a17-b9cc-557b1abba0e8-console-config\") pod \"c0223a0c-8f9d-4a17-b9cc-557b1abba0e8\" (UID: \"c0223a0c-8f9d-4a17-b9cc-557b1abba0e8\") " Jul 10 00:32:13.139449 ip-10-0-130-146 kubenswrapper[2570]: I0710 00:32:13.139269 2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/c0223a0c-8f9d-4a17-b9cc-557b1abba0e8-service-ca\") pod \"c0223a0c-8f9d-4a17-b9cc-557b1abba0e8\" (UID: \"c0223a0c-8f9d-4a17-b9cc-557b1abba0e8\") " Jul 10 00:32:13.139449 ip-10-0-130-146 kubenswrapper[2570]: I0710 00:32:13.139293 2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/c0223a0c-8f9d-4a17-b9cc-557b1abba0e8-oauth-serving-cert\") pod \"c0223a0c-8f9d-4a17-b9cc-557b1abba0e8\" (UID: \"c0223a0c-8f9d-4a17-b9cc-557b1abba0e8\") " Jul 10 00:32:13.139449 ip-10-0-130-146 kubenswrapper[2570]: I0710 00:32:13.139338 2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-t5ndm\" (UniqueName: \"kubernetes.io/projected/c0223a0c-8f9d-4a17-b9cc-557b1abba0e8-kube-api-access-t5ndm\") pod \"c0223a0c-8f9d-4a17-b9cc-557b1abba0e8\" (UID: \"c0223a0c-8f9d-4a17-b9cc-557b1abba0e8\") " Jul 10 00:32:13.139449 ip-10-0-130-146 kubenswrapper[2570]: I0710 00:32:13.139399 2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/c0223a0c-8f9d-4a17-b9cc-557b1abba0e8-console-serving-cert\") pod \"c0223a0c-8f9d-4a17-b9cc-557b1abba0e8\" (UID: \"c0223a0c-8f9d-4a17-b9cc-557b1abba0e8\") " Jul 10 00:32:13.139804 ip-10-0-130-146 kubenswrapper[2570]: I0710 00:32:13.139679 2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c0223a0c-8f9d-4a17-b9cc-557b1abba0e8-service-ca" (OuterVolumeSpecName: "service-ca") pod "c0223a0c-8f9d-4a17-b9cc-557b1abba0e8" (UID: "c0223a0c-8f9d-4a17-b9cc-557b1abba0e8"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 10 00:32:13.139804 ip-10-0-130-146 kubenswrapper[2570]: I0710 00:32:13.139695 2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c0223a0c-8f9d-4a17-b9cc-557b1abba0e8-oauth-serving-cert" (OuterVolumeSpecName: "oauth-serving-cert") pod "c0223a0c-8f9d-4a17-b9cc-557b1abba0e8" (UID: "c0223a0c-8f9d-4a17-b9cc-557b1abba0e8"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 10 00:32:13.139993 ip-10-0-130-146 kubenswrapper[2570]: I0710 00:32:13.139846 2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c0223a0c-8f9d-4a17-b9cc-557b1abba0e8-console-config" (OuterVolumeSpecName: "console-config") pod "c0223a0c-8f9d-4a17-b9cc-557b1abba0e8" (UID: "c0223a0c-8f9d-4a17-b9cc-557b1abba0e8"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 10 00:32:13.141480 ip-10-0-130-146 kubenswrapper[2570]: I0710 00:32:13.141451 2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c0223a0c-8f9d-4a17-b9cc-557b1abba0e8-console-serving-cert" (OuterVolumeSpecName: "console-serving-cert") pod "c0223a0c-8f9d-4a17-b9cc-557b1abba0e8" (UID: "c0223a0c-8f9d-4a17-b9cc-557b1abba0e8"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 10 00:32:13.141579 ip-10-0-130-146 kubenswrapper[2570]: I0710 00:32:13.141514 2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c0223a0c-8f9d-4a17-b9cc-557b1abba0e8-kube-api-access-t5ndm" (OuterVolumeSpecName: "kube-api-access-t5ndm") pod "c0223a0c-8f9d-4a17-b9cc-557b1abba0e8" (UID: "c0223a0c-8f9d-4a17-b9cc-557b1abba0e8"). InnerVolumeSpecName "kube-api-access-t5ndm". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 10 00:32:13.141579 ip-10-0-130-146 kubenswrapper[2570]: I0710 00:32:13.141521 2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c0223a0c-8f9d-4a17-b9cc-557b1abba0e8-console-oauth-config" (OuterVolumeSpecName: "console-oauth-config") pod "c0223a0c-8f9d-4a17-b9cc-557b1abba0e8" (UID: "c0223a0c-8f9d-4a17-b9cc-557b1abba0e8"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 10 00:32:13.240236 ip-10-0-130-146 kubenswrapper[2570]: I0710 00:32:13.240207 2570 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-t5ndm\" (UniqueName: \"kubernetes.io/projected/c0223a0c-8f9d-4a17-b9cc-557b1abba0e8-kube-api-access-t5ndm\") on node \"ip-10-0-130-146.ec2.internal\" DevicePath \"\"" Jul 10 00:32:13.240236 ip-10-0-130-146 kubenswrapper[2570]: I0710 00:32:13.240231 2570 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/c0223a0c-8f9d-4a17-b9cc-557b1abba0e8-console-serving-cert\") on node \"ip-10-0-130-146.ec2.internal\" DevicePath \"\"" Jul 10 00:32:13.240236 ip-10-0-130-146 kubenswrapper[2570]: I0710 00:32:13.240241 2570 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/c0223a0c-8f9d-4a17-b9cc-557b1abba0e8-console-oauth-config\") on node \"ip-10-0-130-146.ec2.internal\" DevicePath \"\"" Jul 10 00:32:13.240449 ip-10-0-130-146 kubenswrapper[2570]: I0710 00:32:13.240250 2570 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/c0223a0c-8f9d-4a17-b9cc-557b1abba0e8-console-config\") on node \"ip-10-0-130-146.ec2.internal\" DevicePath \"\"" Jul 10 00:32:13.240449 ip-10-0-130-146 kubenswrapper[2570]: I0710 00:32:13.240259 2570 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/c0223a0c-8f9d-4a17-b9cc-557b1abba0e8-service-ca\") on node \"ip-10-0-130-146.ec2.internal\" DevicePath \"\"" Jul 10 00:32:13.240449 ip-10-0-130-146 kubenswrapper[2570]: I0710 00:32:13.240267 2570 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/c0223a0c-8f9d-4a17-b9cc-557b1abba0e8-oauth-serving-cert\") on node \"ip-10-0-130-146.ec2.internal\" DevicePath \"\"" Jul 10 00:32:13.733661 ip-10-0-130-146 kubenswrapper[2570]: I0710 00:32:13.733577 2570 generic.go:356] "Generic (PLEG): container finished" podID="c0223a0c-8f9d-4a17-b9cc-557b1abba0e8" containerID="a4242be6af1c02a07b23a0cae810e0f03832534ba9e9cb654911ae090b7de002" exitCode=0 Jul 10 00:32:13.733661 ip-10-0-130-146 kubenswrapper[2570]: I0710 00:32:13.733620 2570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-56ccc595db-rn7fp" Jul 10 00:32:13.733661 ip-10-0-130-146 kubenswrapper[2570]: I0710 00:32:13.733626 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-56ccc595db-rn7fp" event={"ID":"c0223a0c-8f9d-4a17-b9cc-557b1abba0e8","Type":"ContainerDied","Data":"a4242be6af1c02a07b23a0cae810e0f03832534ba9e9cb654911ae090b7de002"} Jul 10 00:32:13.734143 ip-10-0-130-146 kubenswrapper[2570]: I0710 00:32:13.733664 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-56ccc595db-rn7fp" event={"ID":"c0223a0c-8f9d-4a17-b9cc-557b1abba0e8","Type":"ContainerDied","Data":"e4d83b90397a11627cd5ee284762c824eda2da6e5f60a29ac881937e9ab4cbd5"} Jul 10 00:32:13.734143 ip-10-0-130-146 kubenswrapper[2570]: I0710 00:32:13.733685 2570 scope.go:117] "RemoveContainer" containerID="a4242be6af1c02a07b23a0cae810e0f03832534ba9e9cb654911ae090b7de002" Jul 10 00:32:13.741423 ip-10-0-130-146 kubenswrapper[2570]: I0710 00:32:13.741409 2570 scope.go:117] "RemoveContainer" containerID="a4242be6af1c02a07b23a0cae810e0f03832534ba9e9cb654911ae090b7de002" Jul 10 00:32:13.741646 ip-10-0-130-146 kubenswrapper[2570]: E0710 00:32:13.741630 2570 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a4242be6af1c02a07b23a0cae810e0f03832534ba9e9cb654911ae090b7de002\": container with ID starting with a4242be6af1c02a07b23a0cae810e0f03832534ba9e9cb654911ae090b7de002 not found: ID does not exist" containerID="a4242be6af1c02a07b23a0cae810e0f03832534ba9e9cb654911ae090b7de002" Jul 10 00:32:13.741683 ip-10-0-130-146 kubenswrapper[2570]: I0710 00:32:13.741653 2570 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a4242be6af1c02a07b23a0cae810e0f03832534ba9e9cb654911ae090b7de002"} err="failed to get container status \"a4242be6af1c02a07b23a0cae810e0f03832534ba9e9cb654911ae090b7de002\": rpc error: code = NotFound desc = could not find container \"a4242be6af1c02a07b23a0cae810e0f03832534ba9e9cb654911ae090b7de002\": container with ID starting with a4242be6af1c02a07b23a0cae810e0f03832534ba9e9cb654911ae090b7de002 not found: ID does not exist" Jul 10 00:32:13.749283 ip-10-0-130-146 kubenswrapper[2570]: I0710 00:32:13.749262 2570 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-56ccc595db-rn7fp"] Jul 10 00:32:13.752899 ip-10-0-130-146 kubenswrapper[2570]: I0710 00:32:13.752880 2570 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-56ccc595db-rn7fp"] Jul 10 00:32:15.453224 ip-10-0-130-146 kubenswrapper[2570]: I0710 00:32:15.453194 2570 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="c0223a0c-8f9d-4a17-b9cc-557b1abba0e8" path="/var/lib/kubelet/pods/c0223a0c-8f9d-4a17-b9cc-557b1abba0e8/volumes" Jul 10 00:32:26.226600 ip-10-0-130-146 kubenswrapper[2570]: I0710 00:32:26.226559 2570 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/telemeter-client-68b5f47d8-swcn5"] Jul 10 00:32:26.227027 ip-10-0-130-146 kubenswrapper[2570]: I0710 00:32:26.226913 2570 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c0223a0c-8f9d-4a17-b9cc-557b1abba0e8" containerName="console" Jul 10 00:32:26.227027 ip-10-0-130-146 kubenswrapper[2570]: I0710 00:32:26.226929 2570 state_mem.go:107] "Deleted CPUSet assignment" podUID="c0223a0c-8f9d-4a17-b9cc-557b1abba0e8" containerName="console" Jul 10 00:32:26.227027 ip-10-0-130-146 kubenswrapper[2570]: I0710 00:32:26.226998 2570 memory_manager.go:368] "RemoveStaleState removing state" podUID="c0223a0c-8f9d-4a17-b9cc-557b1abba0e8" containerName="console" Jul 10 00:32:26.261221 ip-10-0-130-146 kubenswrapper[2570]: I0710 00:32:26.261189 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/telemeter-client-68b5f47d8-swcn5"] Jul 10 00:32:26.261363 ip-10-0-130-146 kubenswrapper[2570]: I0710 00:32:26.261261 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/telemeter-client-68b5f47d8-swcn5" Jul 10 00:32:26.264322 ip-10-0-130-146 kubenswrapper[2570]: I0710 00:32:26.264302 2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client-dockercfg-l5c7z\"" Jul 10 00:32:26.264827 ip-10-0-130-146 kubenswrapper[2570]: I0710 00:32:26.264809 2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client-tls\"" Jul 10 00:32:26.264987 ip-10-0-130-146 kubenswrapper[2570]: I0710 00:32:26.264814 2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"federate-client-certs\"" Jul 10 00:32:26.265122 ip-10-0-130-146 kubenswrapper[2570]: I0710 00:32:26.264864 2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client-kube-rbac-proxy-config\"" Jul 10 00:32:26.265122 ip-10-0-130-146 kubenswrapper[2570]: I0710 00:32:26.264909 2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client\"" Jul 10 00:32:26.265226 ip-10-0-130-146 kubenswrapper[2570]: I0710 00:32:26.264898 2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"telemeter-client-serving-certs-ca-bundle\"" Jul 10 00:32:26.269408 ip-10-0-130-146 kubenswrapper[2570]: I0710 00:32:26.269388 2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"telemeter-trusted-ca-bundle-8i12ta5c71j38\"" Jul 10 00:32:26.453805 ip-10-0-130-146 kubenswrapper[2570]: I0710 00:32:26.453769 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/a60c4e5b-3332-4a4b-8d32-c94b0f823695-metrics-client-ca\") pod \"telemeter-client-68b5f47d8-swcn5\" (UID: \"a60c4e5b-3332-4a4b-8d32-c94b0f823695\") " pod="openshift-monitoring/telemeter-client-68b5f47d8-swcn5" Jul 10 00:32:26.453993 ip-10-0-130-146 kubenswrapper[2570]: I0710 00:32:26.453898 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-telemeter-client-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/a60c4e5b-3332-4a4b-8d32-c94b0f823695-secret-telemeter-client-kube-rbac-proxy-config\") pod \"telemeter-client-68b5f47d8-swcn5\" (UID: \"a60c4e5b-3332-4a4b-8d32-c94b0f823695\") " pod="openshift-monitoring/telemeter-client-68b5f47d8-swcn5" Jul 10 00:32:26.453993 ip-10-0-130-146 kubenswrapper[2570]: I0710 00:32:26.453968 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-telemeter-client\" (UniqueName: \"kubernetes.io/secret/a60c4e5b-3332-4a4b-8d32-c94b0f823695-secret-telemeter-client\") pod \"telemeter-client-68b5f47d8-swcn5\" (UID: \"a60c4e5b-3332-4a4b-8d32-c94b0f823695\") " pod="openshift-monitoring/telemeter-client-68b5f47d8-swcn5" Jul 10 00:32:26.453993 ip-10-0-130-146 kubenswrapper[2570]: I0710 00:32:26.453991 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"federate-client-tls\" (UniqueName: \"kubernetes.io/secret/a60c4e5b-3332-4a4b-8d32-c94b0f823695-federate-client-tls\") pod \"telemeter-client-68b5f47d8-swcn5\" (UID: \"a60c4e5b-3332-4a4b-8d32-c94b0f823695\") " pod="openshift-monitoring/telemeter-client-68b5f47d8-swcn5" Jul 10 00:32:26.454164 ip-10-0-130-146 kubenswrapper[2570]: I0710 00:32:26.454006 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"telemeter-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/a60c4e5b-3332-4a4b-8d32-c94b0f823695-telemeter-trusted-ca-bundle\") pod \"telemeter-client-68b5f47d8-swcn5\" (UID: \"a60c4e5b-3332-4a4b-8d32-c94b0f823695\") " pod="openshift-monitoring/telemeter-client-68b5f47d8-swcn5" Jul 10 00:32:26.454164 ip-10-0-130-146 kubenswrapper[2570]: I0710 00:32:26.454024 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wxqvc\" (UniqueName: \"kubernetes.io/projected/a60c4e5b-3332-4a4b-8d32-c94b0f823695-kube-api-access-wxqvc\") pod \"telemeter-client-68b5f47d8-swcn5\" (UID: \"a60c4e5b-3332-4a4b-8d32-c94b0f823695\") " pod="openshift-monitoring/telemeter-client-68b5f47d8-swcn5" Jul 10 00:32:26.454164 ip-10-0-130-146 kubenswrapper[2570]: I0710 00:32:26.454039 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/a60c4e5b-3332-4a4b-8d32-c94b0f823695-serving-certs-ca-bundle\") pod \"telemeter-client-68b5f47d8-swcn5\" (UID: \"a60c4e5b-3332-4a4b-8d32-c94b0f823695\") " pod="openshift-monitoring/telemeter-client-68b5f47d8-swcn5" Jul 10 00:32:26.454164 ip-10-0-130-146 kubenswrapper[2570]: I0710 00:32:26.454073 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"telemeter-client-tls\" (UniqueName: \"kubernetes.io/secret/a60c4e5b-3332-4a4b-8d32-c94b0f823695-telemeter-client-tls\") pod \"telemeter-client-68b5f47d8-swcn5\" (UID: \"a60c4e5b-3332-4a4b-8d32-c94b0f823695\") " pod="openshift-monitoring/telemeter-client-68b5f47d8-swcn5" Jul 10 00:32:26.555440 ip-10-0-130-146 kubenswrapper[2570]: I0710 00:32:26.555350 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-telemeter-client-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/a60c4e5b-3332-4a4b-8d32-c94b0f823695-secret-telemeter-client-kube-rbac-proxy-config\") pod \"telemeter-client-68b5f47d8-swcn5\" (UID: \"a60c4e5b-3332-4a4b-8d32-c94b0f823695\") " pod="openshift-monitoring/telemeter-client-68b5f47d8-swcn5" Jul 10 00:32:26.555694 ip-10-0-130-146 kubenswrapper[2570]: I0710 00:32:26.555676 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-telemeter-client\" (UniqueName: \"kubernetes.io/secret/a60c4e5b-3332-4a4b-8d32-c94b0f823695-secret-telemeter-client\") pod \"telemeter-client-68b5f47d8-swcn5\" (UID: \"a60c4e5b-3332-4a4b-8d32-c94b0f823695\") " pod="openshift-monitoring/telemeter-client-68b5f47d8-swcn5" Jul 10 00:32:26.556184 ip-10-0-130-146 kubenswrapper[2570]: I0710 00:32:26.556164 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"federate-client-tls\" (UniqueName: \"kubernetes.io/secret/a60c4e5b-3332-4a4b-8d32-c94b0f823695-federate-client-tls\") pod \"telemeter-client-68b5f47d8-swcn5\" (UID: \"a60c4e5b-3332-4a4b-8d32-c94b0f823695\") " pod="openshift-monitoring/telemeter-client-68b5f47d8-swcn5" Jul 10 00:32:26.556786 ip-10-0-130-146 kubenswrapper[2570]: I0710 00:32:26.556761 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"telemeter-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/a60c4e5b-3332-4a4b-8d32-c94b0f823695-telemeter-trusted-ca-bundle\") pod \"telemeter-client-68b5f47d8-swcn5\" (UID: \"a60c4e5b-3332-4a4b-8d32-c94b0f823695\") " pod="openshift-monitoring/telemeter-client-68b5f47d8-swcn5" Jul 10 00:32:26.556914 ip-10-0-130-146 kubenswrapper[2570]: I0710 00:32:26.556801 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wxqvc\" (UniqueName: \"kubernetes.io/projected/a60c4e5b-3332-4a4b-8d32-c94b0f823695-kube-api-access-wxqvc\") pod \"telemeter-client-68b5f47d8-swcn5\" (UID: \"a60c4e5b-3332-4a4b-8d32-c94b0f823695\") " pod="openshift-monitoring/telemeter-client-68b5f47d8-swcn5" Jul 10 00:32:26.556914 ip-10-0-130-146 kubenswrapper[2570]: I0710 00:32:26.556831 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/a60c4e5b-3332-4a4b-8d32-c94b0f823695-serving-certs-ca-bundle\") pod \"telemeter-client-68b5f47d8-swcn5\" (UID: \"a60c4e5b-3332-4a4b-8d32-c94b0f823695\") " pod="openshift-monitoring/telemeter-client-68b5f47d8-swcn5" Jul 10 00:32:26.557414 ip-10-0-130-146 kubenswrapper[2570]: I0710 00:32:26.557390 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"telemeter-client-tls\" (UniqueName: \"kubernetes.io/secret/a60c4e5b-3332-4a4b-8d32-c94b0f823695-telemeter-client-tls\") pod \"telemeter-client-68b5f47d8-swcn5\" (UID: \"a60c4e5b-3332-4a4b-8d32-c94b0f823695\") " pod="openshift-monitoring/telemeter-client-68b5f47d8-swcn5" Jul 10 00:32:26.557596 ip-10-0-130-146 kubenswrapper[2570]: I0710 00:32:26.557579 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/a60c4e5b-3332-4a4b-8d32-c94b0f823695-metrics-client-ca\") pod \"telemeter-client-68b5f47d8-swcn5\" (UID: \"a60c4e5b-3332-4a4b-8d32-c94b0f823695\") " pod="openshift-monitoring/telemeter-client-68b5f47d8-swcn5" Jul 10 00:32:26.558353 ip-10-0-130-146 kubenswrapper[2570]: I0710 00:32:26.557930 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/a60c4e5b-3332-4a4b-8d32-c94b0f823695-serving-certs-ca-bundle\") pod \"telemeter-client-68b5f47d8-swcn5\" (UID: \"a60c4e5b-3332-4a4b-8d32-c94b0f823695\") " pod="openshift-monitoring/telemeter-client-68b5f47d8-swcn5" Jul 10 00:32:26.558446 ip-10-0-130-146 kubenswrapper[2570]: I0710 00:32:26.558023 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"telemeter-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/a60c4e5b-3332-4a4b-8d32-c94b0f823695-telemeter-trusted-ca-bundle\") pod \"telemeter-client-68b5f47d8-swcn5\" (UID: \"a60c4e5b-3332-4a4b-8d32-c94b0f823695\") " pod="openshift-monitoring/telemeter-client-68b5f47d8-swcn5" Jul 10 00:32:26.559008 ip-10-0-130-146 kubenswrapper[2570]: I0710 00:32:26.558633 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-telemeter-client-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/a60c4e5b-3332-4a4b-8d32-c94b0f823695-secret-telemeter-client-kube-rbac-proxy-config\") pod \"telemeter-client-68b5f47d8-swcn5\" (UID: \"a60c4e5b-3332-4a4b-8d32-c94b0f823695\") " pod="openshift-monitoring/telemeter-client-68b5f47d8-swcn5" Jul 10 00:32:26.559008 ip-10-0-130-146 kubenswrapper[2570]: I0710 00:32:26.558971 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"federate-client-tls\" (UniqueName: \"kubernetes.io/secret/a60c4e5b-3332-4a4b-8d32-c94b0f823695-federate-client-tls\") pod \"telemeter-client-68b5f47d8-swcn5\" (UID: \"a60c4e5b-3332-4a4b-8d32-c94b0f823695\") " pod="openshift-monitoring/telemeter-client-68b5f47d8-swcn5" Jul 10 00:32:26.559166 ip-10-0-130-146 kubenswrapper[2570]: I0710 00:32:26.559020 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/a60c4e5b-3332-4a4b-8d32-c94b0f823695-metrics-client-ca\") pod \"telemeter-client-68b5f47d8-swcn5\" (UID: \"a60c4e5b-3332-4a4b-8d32-c94b0f823695\") " pod="openshift-monitoring/telemeter-client-68b5f47d8-swcn5" Jul 10 00:32:26.559640 ip-10-0-130-146 kubenswrapper[2570]: I0710 00:32:26.559620 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-telemeter-client\" (UniqueName: \"kubernetes.io/secret/a60c4e5b-3332-4a4b-8d32-c94b0f823695-secret-telemeter-client\") pod \"telemeter-client-68b5f47d8-swcn5\" (UID: \"a60c4e5b-3332-4a4b-8d32-c94b0f823695\") " pod="openshift-monitoring/telemeter-client-68b5f47d8-swcn5" Jul 10 00:32:26.560196 ip-10-0-130-146 kubenswrapper[2570]: I0710 00:32:26.560176 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"telemeter-client-tls\" (UniqueName: \"kubernetes.io/secret/a60c4e5b-3332-4a4b-8d32-c94b0f823695-telemeter-client-tls\") pod \"telemeter-client-68b5f47d8-swcn5\" (UID: \"a60c4e5b-3332-4a4b-8d32-c94b0f823695\") " pod="openshift-monitoring/telemeter-client-68b5f47d8-swcn5" Jul 10 00:32:26.570663 ip-10-0-130-146 kubenswrapper[2570]: I0710 00:32:26.564996 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-wxqvc\" (UniqueName: \"kubernetes.io/projected/a60c4e5b-3332-4a4b-8d32-c94b0f823695-kube-api-access-wxqvc\") pod \"telemeter-client-68b5f47d8-swcn5\" (UID: \"a60c4e5b-3332-4a4b-8d32-c94b0f823695\") " pod="openshift-monitoring/telemeter-client-68b5f47d8-swcn5" Jul 10 00:32:26.573434 ip-10-0-130-146 kubenswrapper[2570]: I0710 00:32:26.573399 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/telemeter-client-68b5f47d8-swcn5" Jul 10 00:32:26.745299 ip-10-0-130-146 kubenswrapper[2570]: I0710 00:32:26.744983 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/telemeter-client-68b5f47d8-swcn5"] Jul 10 00:32:26.748254 ip-10-0-130-146 kubenswrapper[2570]: W0710 00:32:26.748228 2570 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda60c4e5b_3332_4a4b_8d32_c94b0f823695.slice/crio-3d315bde1a2eefd5188bb9ea8d5773b46b21129070871d041999c2396f97e21e WatchSource:0}: Error finding container 3d315bde1a2eefd5188bb9ea8d5773b46b21129070871d041999c2396f97e21e: Status 404 returned error can't find the container with id 3d315bde1a2eefd5188bb9ea8d5773b46b21129070871d041999c2396f97e21e Jul 10 00:32:26.783095 ip-10-0-130-146 kubenswrapper[2570]: I0710 00:32:26.783064 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-68b5f47d8-swcn5" event={"ID":"a60c4e5b-3332-4a4b-8d32-c94b0f823695","Type":"ContainerStarted","Data":"3d315bde1a2eefd5188bb9ea8d5773b46b21129070871d041999c2396f97e21e"} Jul 10 00:32:29.799125 ip-10-0-130-146 kubenswrapper[2570]: I0710 00:32:29.799093 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-68b5f47d8-swcn5" event={"ID":"a60c4e5b-3332-4a4b-8d32-c94b0f823695","Type":"ContainerStarted","Data":"065700697ca641f1df1d15b6bf36de596b042a635dd984feed82ff479145f8e8"} Jul 10 00:32:30.804931 ip-10-0-130-146 kubenswrapper[2570]: I0710 00:32:30.804894 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-68b5f47d8-swcn5" event={"ID":"a60c4e5b-3332-4a4b-8d32-c94b0f823695","Type":"ContainerStarted","Data":"d435caaa72be2156af45c5e3be941f62f7b00bc0ec031387be36b2c0aeadca24"} Jul 10 00:32:30.805324 ip-10-0-130-146 kubenswrapper[2570]: I0710 00:32:30.804959 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-68b5f47d8-swcn5" event={"ID":"a60c4e5b-3332-4a4b-8d32-c94b0f823695","Type":"ContainerStarted","Data":"42bc4b10e9d9c09022f01f7c744c62178d4dbff69b1db392fdd0c0c367fe2c38"} Jul 10 00:32:30.826957 ip-10-0-130-146 kubenswrapper[2570]: I0710 00:32:30.826909 2570 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/telemeter-client-68b5f47d8-swcn5" podStartSLOduration=1.282721543 podStartE2EDuration="4.826892603s" totalImagesPullingTime="3.54417106s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-10 00:32:26 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 00:32:30.824180082 +0000 UTC m=+369.872558802" watchObservedRunningTime="2026-07-10 00:32:30.826892603 +0000 UTC m=+369.875271324" Jul 10 00:32:31.864632 ip-10-0-130-146 kubenswrapper[2570]: I0710 00:32:31.864590 2570 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-f9cbff9ff-qmtfl"] Jul 10 00:32:31.867043 ip-10-0-130-146 kubenswrapper[2570]: I0710 00:32:31.867017 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-f9cbff9ff-qmtfl" Jul 10 00:32:31.878234 ip-10-0-130-146 kubenswrapper[2570]: I0710 00:32:31.878210 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-f9cbff9ff-qmtfl"] Jul 10 00:32:31.897311 ip-10-0-130-146 kubenswrapper[2570]: I0710 00:32:31.897289 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/87908ff5-db99-4f3c-813b-162ae40820e7-trusted-ca-bundle\") pod \"console-f9cbff9ff-qmtfl\" (UID: \"87908ff5-db99-4f3c-813b-162ae40820e7\") " pod="openshift-console/console-f9cbff9ff-qmtfl" Jul 10 00:32:31.897416 ip-10-0-130-146 kubenswrapper[2570]: I0710 00:32:31.897338 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/87908ff5-db99-4f3c-813b-162ae40820e7-console-oauth-config\") pod \"console-f9cbff9ff-qmtfl\" (UID: \"87908ff5-db99-4f3c-813b-162ae40820e7\") " pod="openshift-console/console-f9cbff9ff-qmtfl" Jul 10 00:32:31.897457 ip-10-0-130-146 kubenswrapper[2570]: I0710 00:32:31.897427 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9xwjl\" (UniqueName: \"kubernetes.io/projected/87908ff5-db99-4f3c-813b-162ae40820e7-kube-api-access-9xwjl\") pod \"console-f9cbff9ff-qmtfl\" (UID: \"87908ff5-db99-4f3c-813b-162ae40820e7\") " pod="openshift-console/console-f9cbff9ff-qmtfl" Jul 10 00:32:31.897530 ip-10-0-130-146 kubenswrapper[2570]: I0710 00:32:31.897518 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/87908ff5-db99-4f3c-813b-162ae40820e7-console-config\") pod \"console-f9cbff9ff-qmtfl\" (UID: \"87908ff5-db99-4f3c-813b-162ae40820e7\") " pod="openshift-console/console-f9cbff9ff-qmtfl" Jul 10 00:32:31.897576 ip-10-0-130-146 kubenswrapper[2570]: I0710 00:32:31.897545 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/87908ff5-db99-4f3c-813b-162ae40820e7-service-ca\") pod \"console-f9cbff9ff-qmtfl\" (UID: \"87908ff5-db99-4f3c-813b-162ae40820e7\") " pod="openshift-console/console-f9cbff9ff-qmtfl" Jul 10 00:32:31.897701 ip-10-0-130-146 kubenswrapper[2570]: I0710 00:32:31.897684 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/87908ff5-db99-4f3c-813b-162ae40820e7-oauth-serving-cert\") pod \"console-f9cbff9ff-qmtfl\" (UID: \"87908ff5-db99-4f3c-813b-162ae40820e7\") " pod="openshift-console/console-f9cbff9ff-qmtfl" Jul 10 00:32:31.897766 ip-10-0-130-146 kubenswrapper[2570]: I0710 00:32:31.897738 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/87908ff5-db99-4f3c-813b-162ae40820e7-console-serving-cert\") pod \"console-f9cbff9ff-qmtfl\" (UID: \"87908ff5-db99-4f3c-813b-162ae40820e7\") " pod="openshift-console/console-f9cbff9ff-qmtfl" Jul 10 00:32:31.998057 ip-10-0-130-146 kubenswrapper[2570]: I0710 00:32:31.998014 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/87908ff5-db99-4f3c-813b-162ae40820e7-service-ca\") pod \"console-f9cbff9ff-qmtfl\" (UID: \"87908ff5-db99-4f3c-813b-162ae40820e7\") " pod="openshift-console/console-f9cbff9ff-qmtfl" Jul 10 00:32:31.998197 ip-10-0-130-146 kubenswrapper[2570]: I0710 00:32:31.998075 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/87908ff5-db99-4f3c-813b-162ae40820e7-oauth-serving-cert\") pod \"console-f9cbff9ff-qmtfl\" (UID: \"87908ff5-db99-4f3c-813b-162ae40820e7\") " pod="openshift-console/console-f9cbff9ff-qmtfl" Jul 10 00:32:31.998197 ip-10-0-130-146 kubenswrapper[2570]: I0710 00:32:31.998108 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/87908ff5-db99-4f3c-813b-162ae40820e7-console-serving-cert\") pod \"console-f9cbff9ff-qmtfl\" (UID: \"87908ff5-db99-4f3c-813b-162ae40820e7\") " pod="openshift-console/console-f9cbff9ff-qmtfl" Jul 10 00:32:31.998197 ip-10-0-130-146 kubenswrapper[2570]: I0710 00:32:31.998127 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/87908ff5-db99-4f3c-813b-162ae40820e7-trusted-ca-bundle\") pod \"console-f9cbff9ff-qmtfl\" (UID: \"87908ff5-db99-4f3c-813b-162ae40820e7\") " pod="openshift-console/console-f9cbff9ff-qmtfl" Jul 10 00:32:31.998197 ip-10-0-130-146 kubenswrapper[2570]: I0710 00:32:31.998156 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/87908ff5-db99-4f3c-813b-162ae40820e7-console-oauth-config\") pod \"console-f9cbff9ff-qmtfl\" (UID: \"87908ff5-db99-4f3c-813b-162ae40820e7\") " pod="openshift-console/console-f9cbff9ff-qmtfl" Jul 10 00:32:31.998197 ip-10-0-130-146 kubenswrapper[2570]: I0710 00:32:31.998179 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9xwjl\" (UniqueName: \"kubernetes.io/projected/87908ff5-db99-4f3c-813b-162ae40820e7-kube-api-access-9xwjl\") pod \"console-f9cbff9ff-qmtfl\" (UID: \"87908ff5-db99-4f3c-813b-162ae40820e7\") " pod="openshift-console/console-f9cbff9ff-qmtfl" Jul 10 00:32:31.998452 ip-10-0-130-146 kubenswrapper[2570]: I0710 00:32:31.998211 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/87908ff5-db99-4f3c-813b-162ae40820e7-console-config\") pod \"console-f9cbff9ff-qmtfl\" (UID: \"87908ff5-db99-4f3c-813b-162ae40820e7\") " pod="openshift-console/console-f9cbff9ff-qmtfl" Jul 10 00:32:31.998790 ip-10-0-130-146 kubenswrapper[2570]: I0710 00:32:31.998766 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/87908ff5-db99-4f3c-813b-162ae40820e7-service-ca\") pod \"console-f9cbff9ff-qmtfl\" (UID: \"87908ff5-db99-4f3c-813b-162ae40820e7\") " pod="openshift-console/console-f9cbff9ff-qmtfl" Jul 10 00:32:31.998888 ip-10-0-130-146 kubenswrapper[2570]: I0710 00:32:31.998869 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/87908ff5-db99-4f3c-813b-162ae40820e7-oauth-serving-cert\") pod \"console-f9cbff9ff-qmtfl\" (UID: \"87908ff5-db99-4f3c-813b-162ae40820e7\") " pod="openshift-console/console-f9cbff9ff-qmtfl" Jul 10 00:32:31.998983 ip-10-0-130-146 kubenswrapper[2570]: I0710 00:32:31.998965 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/87908ff5-db99-4f3c-813b-162ae40820e7-trusted-ca-bundle\") pod \"console-f9cbff9ff-qmtfl\" (UID: \"87908ff5-db99-4f3c-813b-162ae40820e7\") " pod="openshift-console/console-f9cbff9ff-qmtfl" Jul 10 00:32:31.999066 ip-10-0-130-146 kubenswrapper[2570]: I0710 00:32:31.998991 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/87908ff5-db99-4f3c-813b-162ae40820e7-console-config\") pod \"console-f9cbff9ff-qmtfl\" (UID: \"87908ff5-db99-4f3c-813b-162ae40820e7\") " pod="openshift-console/console-f9cbff9ff-qmtfl" Jul 10 00:32:32.000656 ip-10-0-130-146 kubenswrapper[2570]: I0710 00:32:32.000627 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/87908ff5-db99-4f3c-813b-162ae40820e7-console-oauth-config\") pod \"console-f9cbff9ff-qmtfl\" (UID: \"87908ff5-db99-4f3c-813b-162ae40820e7\") " pod="openshift-console/console-f9cbff9ff-qmtfl" Jul 10 00:32:32.000741 ip-10-0-130-146 kubenswrapper[2570]: I0710 00:32:32.000671 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/87908ff5-db99-4f3c-813b-162ae40820e7-console-serving-cert\") pod \"console-f9cbff9ff-qmtfl\" (UID: \"87908ff5-db99-4f3c-813b-162ae40820e7\") " pod="openshift-console/console-f9cbff9ff-qmtfl" Jul 10 00:32:32.005979 ip-10-0-130-146 kubenswrapper[2570]: I0710 00:32:32.005958 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9xwjl\" (UniqueName: \"kubernetes.io/projected/87908ff5-db99-4f3c-813b-162ae40820e7-kube-api-access-9xwjl\") pod \"console-f9cbff9ff-qmtfl\" (UID: \"87908ff5-db99-4f3c-813b-162ae40820e7\") " pod="openshift-console/console-f9cbff9ff-qmtfl" Jul 10 00:32:32.176814 ip-10-0-130-146 kubenswrapper[2570]: I0710 00:32:32.176775 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-f9cbff9ff-qmtfl" Jul 10 00:32:32.303480 ip-10-0-130-146 kubenswrapper[2570]: I0710 00:32:32.303457 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-f9cbff9ff-qmtfl"] Jul 10 00:32:32.305379 ip-10-0-130-146 kubenswrapper[2570]: W0710 00:32:32.305357 2570 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod87908ff5_db99_4f3c_813b_162ae40820e7.slice/crio-382ec788a6d5ffb01d1818465921b09fbad71dcf3c0c17a6ea158e9ac19e076b WatchSource:0}: Error finding container 382ec788a6d5ffb01d1818465921b09fbad71dcf3c0c17a6ea158e9ac19e076b: Status 404 returned error can't find the container with id 382ec788a6d5ffb01d1818465921b09fbad71dcf3c0c17a6ea158e9ac19e076b Jul 10 00:32:32.814339 ip-10-0-130-146 kubenswrapper[2570]: I0710 00:32:32.814306 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-f9cbff9ff-qmtfl" event={"ID":"87908ff5-db99-4f3c-813b-162ae40820e7","Type":"ContainerStarted","Data":"13bf8ba5a096fdbe12291d8354cc531bd02db860d8f340c945530eac926dec00"} Jul 10 00:32:32.814526 ip-10-0-130-146 kubenswrapper[2570]: I0710 00:32:32.814358 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-f9cbff9ff-qmtfl" event={"ID":"87908ff5-db99-4f3c-813b-162ae40820e7","Type":"ContainerStarted","Data":"382ec788a6d5ffb01d1818465921b09fbad71dcf3c0c17a6ea158e9ac19e076b"} Jul 10 00:32:32.831524 ip-10-0-130-146 kubenswrapper[2570]: I0710 00:32:32.831481 2570 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-f9cbff9ff-qmtfl" podStartSLOduration=1.831463126 podStartE2EDuration="1.831463126s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-10 00:32:31 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 00:32:32.829531744 +0000 UTC m=+371.877910466" watchObservedRunningTime="2026-07-10 00:32:32.831463126 +0000 UTC m=+371.879841849" Jul 10 00:32:37.148740 ip-10-0-130-146 kubenswrapper[2570]: I0710 00:32:37.148703 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b7aab806-00a2-491e-a997-ed3174bd547a-metrics-certs\") pod \"network-metrics-daemon-d8t56\" (UID: \"b7aab806-00a2-491e-a997-ed3174bd547a\") " pod="openshift-multus/network-metrics-daemon-d8t56" Jul 10 00:32:37.150990 ip-10-0-130-146 kubenswrapper[2570]: I0710 00:32:37.150965 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b7aab806-00a2-491e-a997-ed3174bd547a-metrics-certs\") pod \"network-metrics-daemon-d8t56\" (UID: \"b7aab806-00a2-491e-a997-ed3174bd547a\") " pod="openshift-multus/network-metrics-daemon-d8t56" Jul 10 00:32:37.154235 ip-10-0-130-146 kubenswrapper[2570]: I0710 00:32:37.154216 2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-d5zcx\"" Jul 10 00:32:37.162276 ip-10-0-130-146 kubenswrapper[2570]: I0710 00:32:37.162250 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-d8t56" Jul 10 00:32:37.285092 ip-10-0-130-146 kubenswrapper[2570]: I0710 00:32:37.285043 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-d8t56"] Jul 10 00:32:37.288126 ip-10-0-130-146 kubenswrapper[2570]: W0710 00:32:37.288100 2570 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb7aab806_00a2_491e_a997_ed3174bd547a.slice/crio-4ec53eaf7700f1f6fab38f4a67308273fbef4fe0ad70052eb416fc31a4c33b15 WatchSource:0}: Error finding container 4ec53eaf7700f1f6fab38f4a67308273fbef4fe0ad70052eb416fc31a4c33b15: Status 404 returned error can't find the container with id 4ec53eaf7700f1f6fab38f4a67308273fbef4fe0ad70052eb416fc31a4c33b15 Jul 10 00:32:37.840740 ip-10-0-130-146 kubenswrapper[2570]: I0710 00:32:37.840702 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-d8t56" event={"ID":"b7aab806-00a2-491e-a997-ed3174bd547a","Type":"ContainerStarted","Data":"4ec53eaf7700f1f6fab38f4a67308273fbef4fe0ad70052eb416fc31a4c33b15"} Jul 10 00:32:38.845880 ip-10-0-130-146 kubenswrapper[2570]: I0710 00:32:38.845796 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-d8t56" event={"ID":"b7aab806-00a2-491e-a997-ed3174bd547a","Type":"ContainerStarted","Data":"7cc53fd9269b321f725ddd707fd7a3bdf4d3e5c0d63dd06204b7ff361855a999"} Jul 10 00:32:38.845880 ip-10-0-130-146 kubenswrapper[2570]: I0710 00:32:38.845848 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-d8t56" event={"ID":"b7aab806-00a2-491e-a997-ed3174bd547a","Type":"ContainerStarted","Data":"62448e3504332a7cad10bce9b3dd63c87bc4dd6605a120360ce6321ea3c112d6"} Jul 10 00:32:38.865348 ip-10-0-130-146 kubenswrapper[2570]: I0710 00:32:38.862225 2570 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/network-metrics-daemon-d8t56" podStartSLOduration=376.644996989 podStartE2EDuration="6m17.862204245s" totalImagesPullingTime="1.217207256s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-10 00:26:21 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 00:32:38.860616619 +0000 UTC m=+377.908995341" watchObservedRunningTime="2026-07-10 00:32:38.862204245 +0000 UTC m=+377.910582967" Jul 10 00:32:42.177424 ip-10-0-130-146 kubenswrapper[2570]: I0710 00:32:42.177382 2570 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-f9cbff9ff-qmtfl" Jul 10 00:32:42.177794 ip-10-0-130-146 kubenswrapper[2570]: I0710 00:32:42.177444 2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-f9cbff9ff-qmtfl" Jul 10 00:32:42.182090 ip-10-0-130-146 kubenswrapper[2570]: I0710 00:32:42.182066 2570 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-f9cbff9ff-qmtfl" Jul 10 00:32:42.866623 ip-10-0-130-146 kubenswrapper[2570]: I0710 00:32:42.866589 2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-f9cbff9ff-qmtfl" Jul 10 00:32:42.907922 ip-10-0-130-146 kubenswrapper[2570]: I0710 00:32:42.907894 2570 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-847dc9597b-l8kgk"] Jul 10 00:33:07.929896 ip-10-0-130-146 kubenswrapper[2570]: I0710 00:33:07.929836 2570 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-console/console-847dc9597b-l8kgk" podUID="f36e7fdf-3a6c-473e-a83e-452b22b2c96c" containerName="console" containerID="cri-o://a77c6bf8521b0a680ec619e1b38d748db350c92535990433bc52f3860af3b507" gracePeriod=15 Jul 10 00:33:08.178653 ip-10-0-130-146 kubenswrapper[2570]: I0710 00:33:08.178629 2570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-847dc9597b-l8kgk" Jul 10 00:33:08.305755 ip-10-0-130-146 kubenswrapper[2570]: I0710 00:33:08.305647 2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/f36e7fdf-3a6c-473e-a83e-452b22b2c96c-console-oauth-config\") pod \"f36e7fdf-3a6c-473e-a83e-452b22b2c96c\" (UID: \"f36e7fdf-3a6c-473e-a83e-452b22b2c96c\") " Jul 10 00:33:08.305755 ip-10-0-130-146 kubenswrapper[2570]: I0710 00:33:08.305705 2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/f36e7fdf-3a6c-473e-a83e-452b22b2c96c-console-serving-cert\") pod \"f36e7fdf-3a6c-473e-a83e-452b22b2c96c\" (UID: \"f36e7fdf-3a6c-473e-a83e-452b22b2c96c\") " Jul 10 00:33:08.305755 ip-10-0-130-146 kubenswrapper[2570]: I0710 00:33:08.305742 2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f36e7fdf-3a6c-473e-a83e-452b22b2c96c-trusted-ca-bundle\") pod \"f36e7fdf-3a6c-473e-a83e-452b22b2c96c\" (UID: \"f36e7fdf-3a6c-473e-a83e-452b22b2c96c\") " Jul 10 00:33:08.306023 ip-10-0-130-146 kubenswrapper[2570]: I0710 00:33:08.305768 2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/f36e7fdf-3a6c-473e-a83e-452b22b2c96c-console-config\") pod \"f36e7fdf-3a6c-473e-a83e-452b22b2c96c\" (UID: \"f36e7fdf-3a6c-473e-a83e-452b22b2c96c\") " Jul 10 00:33:08.306023 ip-10-0-130-146 kubenswrapper[2570]: I0710 00:33:08.305797 2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/f36e7fdf-3a6c-473e-a83e-452b22b2c96c-service-ca\") pod \"f36e7fdf-3a6c-473e-a83e-452b22b2c96c\" (UID: \"f36e7fdf-3a6c-473e-a83e-452b22b2c96c\") " Jul 10 00:33:08.306023 ip-10-0-130-146 kubenswrapper[2570]: I0710 00:33:08.305852 2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-4kg77\" (UniqueName: \"kubernetes.io/projected/f36e7fdf-3a6c-473e-a83e-452b22b2c96c-kube-api-access-4kg77\") pod \"f36e7fdf-3a6c-473e-a83e-452b22b2c96c\" (UID: \"f36e7fdf-3a6c-473e-a83e-452b22b2c96c\") " Jul 10 00:33:08.306023 ip-10-0-130-146 kubenswrapper[2570]: I0710 00:33:08.305895 2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/f36e7fdf-3a6c-473e-a83e-452b22b2c96c-oauth-serving-cert\") pod \"f36e7fdf-3a6c-473e-a83e-452b22b2c96c\" (UID: \"f36e7fdf-3a6c-473e-a83e-452b22b2c96c\") " Jul 10 00:33:08.306266 ip-10-0-130-146 kubenswrapper[2570]: I0710 00:33:08.306232 2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f36e7fdf-3a6c-473e-a83e-452b22b2c96c-trusted-ca-bundle" (OuterVolumeSpecName: "trusted-ca-bundle") pod "f36e7fdf-3a6c-473e-a83e-452b22b2c96c" (UID: "f36e7fdf-3a6c-473e-a83e-452b22b2c96c"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 10 00:33:08.306319 ip-10-0-130-146 kubenswrapper[2570]: I0710 00:33:08.306232 2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f36e7fdf-3a6c-473e-a83e-452b22b2c96c-console-config" (OuterVolumeSpecName: "console-config") pod "f36e7fdf-3a6c-473e-a83e-452b22b2c96c" (UID: "f36e7fdf-3a6c-473e-a83e-452b22b2c96c"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 10 00:33:08.306430 ip-10-0-130-146 kubenswrapper[2570]: I0710 00:33:08.306391 2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f36e7fdf-3a6c-473e-a83e-452b22b2c96c-service-ca" (OuterVolumeSpecName: "service-ca") pod "f36e7fdf-3a6c-473e-a83e-452b22b2c96c" (UID: "f36e7fdf-3a6c-473e-a83e-452b22b2c96c"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 10 00:33:08.306506 ip-10-0-130-146 kubenswrapper[2570]: I0710 00:33:08.306450 2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f36e7fdf-3a6c-473e-a83e-452b22b2c96c-oauth-serving-cert" (OuterVolumeSpecName: "oauth-serving-cert") pod "f36e7fdf-3a6c-473e-a83e-452b22b2c96c" (UID: "f36e7fdf-3a6c-473e-a83e-452b22b2c96c"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 10 00:33:08.308068 ip-10-0-130-146 kubenswrapper[2570]: I0710 00:33:08.308028 2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f36e7fdf-3a6c-473e-a83e-452b22b2c96c-kube-api-access-4kg77" (OuterVolumeSpecName: "kube-api-access-4kg77") pod "f36e7fdf-3a6c-473e-a83e-452b22b2c96c" (UID: "f36e7fdf-3a6c-473e-a83e-452b22b2c96c"). InnerVolumeSpecName "kube-api-access-4kg77". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 10 00:33:08.308314 ip-10-0-130-146 kubenswrapper[2570]: I0710 00:33:08.308296 2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f36e7fdf-3a6c-473e-a83e-452b22b2c96c-console-serving-cert" (OuterVolumeSpecName: "console-serving-cert") pod "f36e7fdf-3a6c-473e-a83e-452b22b2c96c" (UID: "f36e7fdf-3a6c-473e-a83e-452b22b2c96c"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 10 00:33:08.308378 ip-10-0-130-146 kubenswrapper[2570]: I0710 00:33:08.308317 2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f36e7fdf-3a6c-473e-a83e-452b22b2c96c-console-oauth-config" (OuterVolumeSpecName: "console-oauth-config") pod "f36e7fdf-3a6c-473e-a83e-452b22b2c96c" (UID: "f36e7fdf-3a6c-473e-a83e-452b22b2c96c"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 10 00:33:08.407328 ip-10-0-130-146 kubenswrapper[2570]: I0710 00:33:08.407292 2570 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/f36e7fdf-3a6c-473e-a83e-452b22b2c96c-console-oauth-config\") on node \"ip-10-0-130-146.ec2.internal\" DevicePath \"\"" Jul 10 00:33:08.407328 ip-10-0-130-146 kubenswrapper[2570]: I0710 00:33:08.407319 2570 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/f36e7fdf-3a6c-473e-a83e-452b22b2c96c-console-serving-cert\") on node \"ip-10-0-130-146.ec2.internal\" DevicePath \"\"" Jul 10 00:33:08.407328 ip-10-0-130-146 kubenswrapper[2570]: I0710 00:33:08.407328 2570 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f36e7fdf-3a6c-473e-a83e-452b22b2c96c-trusted-ca-bundle\") on node \"ip-10-0-130-146.ec2.internal\" DevicePath \"\"" Jul 10 00:33:08.407328 ip-10-0-130-146 kubenswrapper[2570]: I0710 00:33:08.407336 2570 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/f36e7fdf-3a6c-473e-a83e-452b22b2c96c-console-config\") on node \"ip-10-0-130-146.ec2.internal\" DevicePath \"\"" Jul 10 00:33:08.407571 ip-10-0-130-146 kubenswrapper[2570]: I0710 00:33:08.407344 2570 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/f36e7fdf-3a6c-473e-a83e-452b22b2c96c-service-ca\") on node \"ip-10-0-130-146.ec2.internal\" DevicePath \"\"" Jul 10 00:33:08.407571 ip-10-0-130-146 kubenswrapper[2570]: I0710 00:33:08.407353 2570 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4kg77\" (UniqueName: \"kubernetes.io/projected/f36e7fdf-3a6c-473e-a83e-452b22b2c96c-kube-api-access-4kg77\") on node \"ip-10-0-130-146.ec2.internal\" DevicePath \"\"" Jul 10 00:33:08.407571 ip-10-0-130-146 kubenswrapper[2570]: I0710 00:33:08.407361 2570 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/f36e7fdf-3a6c-473e-a83e-452b22b2c96c-oauth-serving-cert\") on node \"ip-10-0-130-146.ec2.internal\" DevicePath \"\"" Jul 10 00:33:08.966657 ip-10-0-130-146 kubenswrapper[2570]: I0710 00:33:08.966625 2570 generic.go:356] "Generic (PLEG): container finished" podID="f36e7fdf-3a6c-473e-a83e-452b22b2c96c" containerID="a77c6bf8521b0a680ec619e1b38d748db350c92535990433bc52f3860af3b507" exitCode=0 Jul 10 00:33:08.967210 ip-10-0-130-146 kubenswrapper[2570]: I0710 00:33:08.966683 2570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-847dc9597b-l8kgk" Jul 10 00:33:08.967210 ip-10-0-130-146 kubenswrapper[2570]: I0710 00:33:08.966712 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-847dc9597b-l8kgk" event={"ID":"f36e7fdf-3a6c-473e-a83e-452b22b2c96c","Type":"ContainerDied","Data":"a77c6bf8521b0a680ec619e1b38d748db350c92535990433bc52f3860af3b507"} Jul 10 00:33:08.967210 ip-10-0-130-146 kubenswrapper[2570]: I0710 00:33:08.966760 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-847dc9597b-l8kgk" event={"ID":"f36e7fdf-3a6c-473e-a83e-452b22b2c96c","Type":"ContainerDied","Data":"b85fcc48dfb789222bfb6f5a86e004947943a445b98bf6ebc7621683bcf0ba01"} Jul 10 00:33:08.967210 ip-10-0-130-146 kubenswrapper[2570]: I0710 00:33:08.966779 2570 scope.go:117] "RemoveContainer" containerID="a77c6bf8521b0a680ec619e1b38d748db350c92535990433bc52f3860af3b507" Jul 10 00:33:08.981655 ip-10-0-130-146 kubenswrapper[2570]: I0710 00:33:08.981633 2570 scope.go:117] "RemoveContainer" containerID="a77c6bf8521b0a680ec619e1b38d748db350c92535990433bc52f3860af3b507" Jul 10 00:33:08.982035 ip-10-0-130-146 kubenswrapper[2570]: E0710 00:33:08.982006 2570 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a77c6bf8521b0a680ec619e1b38d748db350c92535990433bc52f3860af3b507\": container with ID starting with a77c6bf8521b0a680ec619e1b38d748db350c92535990433bc52f3860af3b507 not found: ID does not exist" containerID="a77c6bf8521b0a680ec619e1b38d748db350c92535990433bc52f3860af3b507" Jul 10 00:33:08.982142 ip-10-0-130-146 kubenswrapper[2570]: I0710 00:33:08.982042 2570 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a77c6bf8521b0a680ec619e1b38d748db350c92535990433bc52f3860af3b507"} err="failed to get container status \"a77c6bf8521b0a680ec619e1b38d748db350c92535990433bc52f3860af3b507\": rpc error: code = NotFound desc = could not find container \"a77c6bf8521b0a680ec619e1b38d748db350c92535990433bc52f3860af3b507\": container with ID starting with a77c6bf8521b0a680ec619e1b38d748db350c92535990433bc52f3860af3b507 not found: ID does not exist" Jul 10 00:33:08.991440 ip-10-0-130-146 kubenswrapper[2570]: I0710 00:33:08.991413 2570 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-847dc9597b-l8kgk"] Jul 10 00:33:08.995041 ip-10-0-130-146 kubenswrapper[2570]: I0710 00:33:08.995021 2570 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-847dc9597b-l8kgk"] Jul 10 00:33:09.453606 ip-10-0-130-146 kubenswrapper[2570]: I0710 00:33:09.453577 2570 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="f36e7fdf-3a6c-473e-a83e-452b22b2c96c" path="/var/lib/kubelet/pods/f36e7fdf-3a6c-473e-a83e-452b22b2c96c/volumes" Jul 10 00:33:40.485256 ip-10-0-130-146 kubenswrapper[2570]: I0710 00:33:40.485226 2570 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-6dc8d7758c-tl2pk"] Jul 10 00:33:40.485776 ip-10-0-130-146 kubenswrapper[2570]: I0710 00:33:40.485548 2570 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f36e7fdf-3a6c-473e-a83e-452b22b2c96c" containerName="console" Jul 10 00:33:40.485776 ip-10-0-130-146 kubenswrapper[2570]: I0710 00:33:40.485559 2570 state_mem.go:107] "Deleted CPUSet assignment" podUID="f36e7fdf-3a6c-473e-a83e-452b22b2c96c" containerName="console" Jul 10 00:33:40.485776 ip-10-0-130-146 kubenswrapper[2570]: I0710 00:33:40.485617 2570 memory_manager.go:368] "RemoveStaleState removing state" podUID="f36e7fdf-3a6c-473e-a83e-452b22b2c96c" containerName="console" Jul 10 00:33:40.490223 ip-10-0-130-146 kubenswrapper[2570]: I0710 00:33:40.490201 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-6dc8d7758c-tl2pk" Jul 10 00:33:40.500717 ip-10-0-130-146 kubenswrapper[2570]: I0710 00:33:40.500693 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-6dc8d7758c-tl2pk"] Jul 10 00:33:40.561292 ip-10-0-130-146 kubenswrapper[2570]: I0710 00:33:40.561262 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/2c2ca2d2-9dc3-45f7-a27a-e10da918748d-console-oauth-config\") pod \"console-6dc8d7758c-tl2pk\" (UID: \"2c2ca2d2-9dc3-45f7-a27a-e10da918748d\") " pod="openshift-console/console-6dc8d7758c-tl2pk" Jul 10 00:33:40.561450 ip-10-0-130-146 kubenswrapper[2570]: I0710 00:33:40.561300 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/2c2ca2d2-9dc3-45f7-a27a-e10da918748d-console-config\") pod \"console-6dc8d7758c-tl2pk\" (UID: \"2c2ca2d2-9dc3-45f7-a27a-e10da918748d\") " pod="openshift-console/console-6dc8d7758c-tl2pk" Jul 10 00:33:40.561450 ip-10-0-130-146 kubenswrapper[2570]: I0710 00:33:40.561319 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/2c2ca2d2-9dc3-45f7-a27a-e10da918748d-console-serving-cert\") pod \"console-6dc8d7758c-tl2pk\" (UID: \"2c2ca2d2-9dc3-45f7-a27a-e10da918748d\") " pod="openshift-console/console-6dc8d7758c-tl2pk" Jul 10 00:33:40.561450 ip-10-0-130-146 kubenswrapper[2570]: I0710 00:33:40.561406 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/2c2ca2d2-9dc3-45f7-a27a-e10da918748d-oauth-serving-cert\") pod \"console-6dc8d7758c-tl2pk\" (UID: \"2c2ca2d2-9dc3-45f7-a27a-e10da918748d\") " pod="openshift-console/console-6dc8d7758c-tl2pk" Jul 10 00:33:40.561584 ip-10-0-130-146 kubenswrapper[2570]: I0710 00:33:40.561462 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/2c2ca2d2-9dc3-45f7-a27a-e10da918748d-trusted-ca-bundle\") pod \"console-6dc8d7758c-tl2pk\" (UID: \"2c2ca2d2-9dc3-45f7-a27a-e10da918748d\") " pod="openshift-console/console-6dc8d7758c-tl2pk" Jul 10 00:33:40.561584 ip-10-0-130-146 kubenswrapper[2570]: I0710 00:33:40.561490 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zxc2v\" (UniqueName: \"kubernetes.io/projected/2c2ca2d2-9dc3-45f7-a27a-e10da918748d-kube-api-access-zxc2v\") pod \"console-6dc8d7758c-tl2pk\" (UID: \"2c2ca2d2-9dc3-45f7-a27a-e10da918748d\") " pod="openshift-console/console-6dc8d7758c-tl2pk" Jul 10 00:33:40.561584 ip-10-0-130-146 kubenswrapper[2570]: I0710 00:33:40.561517 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/2c2ca2d2-9dc3-45f7-a27a-e10da918748d-service-ca\") pod \"console-6dc8d7758c-tl2pk\" (UID: \"2c2ca2d2-9dc3-45f7-a27a-e10da918748d\") " pod="openshift-console/console-6dc8d7758c-tl2pk" Jul 10 00:33:40.662389 ip-10-0-130-146 kubenswrapper[2570]: I0710 00:33:40.662352 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/2c2ca2d2-9dc3-45f7-a27a-e10da918748d-service-ca\") pod \"console-6dc8d7758c-tl2pk\" (UID: \"2c2ca2d2-9dc3-45f7-a27a-e10da918748d\") " pod="openshift-console/console-6dc8d7758c-tl2pk" Jul 10 00:33:40.662557 ip-10-0-130-146 kubenswrapper[2570]: I0710 00:33:40.662413 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/2c2ca2d2-9dc3-45f7-a27a-e10da918748d-console-oauth-config\") pod \"console-6dc8d7758c-tl2pk\" (UID: \"2c2ca2d2-9dc3-45f7-a27a-e10da918748d\") " pod="openshift-console/console-6dc8d7758c-tl2pk" Jul 10 00:33:40.662557 ip-10-0-130-146 kubenswrapper[2570]: I0710 00:33:40.662442 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/2c2ca2d2-9dc3-45f7-a27a-e10da918748d-console-config\") pod \"console-6dc8d7758c-tl2pk\" (UID: \"2c2ca2d2-9dc3-45f7-a27a-e10da918748d\") " pod="openshift-console/console-6dc8d7758c-tl2pk" Jul 10 00:33:40.662557 ip-10-0-130-146 kubenswrapper[2570]: I0710 00:33:40.662465 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/2c2ca2d2-9dc3-45f7-a27a-e10da918748d-console-serving-cert\") pod \"console-6dc8d7758c-tl2pk\" (UID: \"2c2ca2d2-9dc3-45f7-a27a-e10da918748d\") " pod="openshift-console/console-6dc8d7758c-tl2pk" Jul 10 00:33:40.662557 ip-10-0-130-146 kubenswrapper[2570]: I0710 00:33:40.662499 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/2c2ca2d2-9dc3-45f7-a27a-e10da918748d-oauth-serving-cert\") pod \"console-6dc8d7758c-tl2pk\" (UID: \"2c2ca2d2-9dc3-45f7-a27a-e10da918748d\") " pod="openshift-console/console-6dc8d7758c-tl2pk" Jul 10 00:33:40.662557 ip-10-0-130-146 kubenswrapper[2570]: I0710 00:33:40.662537 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/2c2ca2d2-9dc3-45f7-a27a-e10da918748d-trusted-ca-bundle\") pod \"console-6dc8d7758c-tl2pk\" (UID: \"2c2ca2d2-9dc3-45f7-a27a-e10da918748d\") " pod="openshift-console/console-6dc8d7758c-tl2pk" Jul 10 00:33:40.662803 ip-10-0-130-146 kubenswrapper[2570]: I0710 00:33:40.662582 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zxc2v\" (UniqueName: \"kubernetes.io/projected/2c2ca2d2-9dc3-45f7-a27a-e10da918748d-kube-api-access-zxc2v\") pod \"console-6dc8d7758c-tl2pk\" (UID: \"2c2ca2d2-9dc3-45f7-a27a-e10da918748d\") " pod="openshift-console/console-6dc8d7758c-tl2pk" Jul 10 00:33:40.663211 ip-10-0-130-146 kubenswrapper[2570]: I0710 00:33:40.663180 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/2c2ca2d2-9dc3-45f7-a27a-e10da918748d-service-ca\") pod \"console-6dc8d7758c-tl2pk\" (UID: \"2c2ca2d2-9dc3-45f7-a27a-e10da918748d\") " pod="openshift-console/console-6dc8d7758c-tl2pk" Jul 10 00:33:40.663211 ip-10-0-130-146 kubenswrapper[2570]: I0710 00:33:40.663195 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/2c2ca2d2-9dc3-45f7-a27a-e10da918748d-console-config\") pod \"console-6dc8d7758c-tl2pk\" (UID: \"2c2ca2d2-9dc3-45f7-a27a-e10da918748d\") " pod="openshift-console/console-6dc8d7758c-tl2pk" Jul 10 00:33:40.663406 ip-10-0-130-146 kubenswrapper[2570]: I0710 00:33:40.663223 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/2c2ca2d2-9dc3-45f7-a27a-e10da918748d-oauth-serving-cert\") pod \"console-6dc8d7758c-tl2pk\" (UID: \"2c2ca2d2-9dc3-45f7-a27a-e10da918748d\") " pod="openshift-console/console-6dc8d7758c-tl2pk" Jul 10 00:33:40.663468 ip-10-0-130-146 kubenswrapper[2570]: I0710 00:33:40.663412 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/2c2ca2d2-9dc3-45f7-a27a-e10da918748d-trusted-ca-bundle\") pod \"console-6dc8d7758c-tl2pk\" (UID: \"2c2ca2d2-9dc3-45f7-a27a-e10da918748d\") " pod="openshift-console/console-6dc8d7758c-tl2pk" Jul 10 00:33:40.664890 ip-10-0-130-146 kubenswrapper[2570]: I0710 00:33:40.664870 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/2c2ca2d2-9dc3-45f7-a27a-e10da918748d-console-serving-cert\") pod \"console-6dc8d7758c-tl2pk\" (UID: \"2c2ca2d2-9dc3-45f7-a27a-e10da918748d\") " pod="openshift-console/console-6dc8d7758c-tl2pk" Jul 10 00:33:40.664994 ip-10-0-130-146 kubenswrapper[2570]: I0710 00:33:40.664945 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/2c2ca2d2-9dc3-45f7-a27a-e10da918748d-console-oauth-config\") pod \"console-6dc8d7758c-tl2pk\" (UID: \"2c2ca2d2-9dc3-45f7-a27a-e10da918748d\") " pod="openshift-console/console-6dc8d7758c-tl2pk" Jul 10 00:33:40.669320 ip-10-0-130-146 kubenswrapper[2570]: I0710 00:33:40.669302 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-zxc2v\" (UniqueName: \"kubernetes.io/projected/2c2ca2d2-9dc3-45f7-a27a-e10da918748d-kube-api-access-zxc2v\") pod \"console-6dc8d7758c-tl2pk\" (UID: \"2c2ca2d2-9dc3-45f7-a27a-e10da918748d\") " pod="openshift-console/console-6dc8d7758c-tl2pk" Jul 10 00:33:40.801385 ip-10-0-130-146 kubenswrapper[2570]: I0710 00:33:40.801332 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-6dc8d7758c-tl2pk" Jul 10 00:33:40.932887 ip-10-0-130-146 kubenswrapper[2570]: I0710 00:33:40.932863 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-6dc8d7758c-tl2pk"] Jul 10 00:33:40.935030 ip-10-0-130-146 kubenswrapper[2570]: W0710 00:33:40.935005 2570 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2c2ca2d2_9dc3_45f7_a27a_e10da918748d.slice/crio-53b6d02eb18fc06d3dc0fc1b716c7078174944ec3dd8b483277f268d46cf63f7 WatchSource:0}: Error finding container 53b6d02eb18fc06d3dc0fc1b716c7078174944ec3dd8b483277f268d46cf63f7: Status 404 returned error can't find the container with id 53b6d02eb18fc06d3dc0fc1b716c7078174944ec3dd8b483277f268d46cf63f7 Jul 10 00:33:41.090816 ip-10-0-130-146 kubenswrapper[2570]: I0710 00:33:41.090717 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-6dc8d7758c-tl2pk" event={"ID":"2c2ca2d2-9dc3-45f7-a27a-e10da918748d","Type":"ContainerStarted","Data":"78fdc68b523c2721ee456f9280f2d35e349a8e830ec30e102e715487f09a44ca"} Jul 10 00:33:41.090816 ip-10-0-130-146 kubenswrapper[2570]: I0710 00:33:41.090793 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-6dc8d7758c-tl2pk" event={"ID":"2c2ca2d2-9dc3-45f7-a27a-e10da918748d","Type":"ContainerStarted","Data":"53b6d02eb18fc06d3dc0fc1b716c7078174944ec3dd8b483277f268d46cf63f7"} Jul 10 00:33:41.118100 ip-10-0-130-146 kubenswrapper[2570]: I0710 00:33:41.118033 2570 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-6dc8d7758c-tl2pk" podStartSLOduration=1.118016659 podStartE2EDuration="1.118016659s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-10 00:33:40 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 00:33:41.117146437 +0000 UTC m=+440.165525179" watchObservedRunningTime="2026-07-10 00:33:41.118016659 +0000 UTC m=+440.166395380" Jul 10 00:33:50.801600 ip-10-0-130-146 kubenswrapper[2570]: I0710 00:33:50.801511 2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-6dc8d7758c-tl2pk" Jul 10 00:33:50.802044 ip-10-0-130-146 kubenswrapper[2570]: I0710 00:33:50.801781 2570 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-6dc8d7758c-tl2pk" Jul 10 00:33:50.806330 ip-10-0-130-146 kubenswrapper[2570]: I0710 00:33:50.806312 2570 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-6dc8d7758c-tl2pk" Jul 10 00:33:51.131266 ip-10-0-130-146 kubenswrapper[2570]: I0710 00:33:51.131234 2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-6dc8d7758c-tl2pk" Jul 10 00:33:51.172723 ip-10-0-130-146 kubenswrapper[2570]: I0710 00:33:51.172694 2570 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-f9cbff9ff-qmtfl"] Jul 10 00:34:16.194883 ip-10-0-130-146 kubenswrapper[2570]: I0710 00:34:16.194820 2570 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-console/console-f9cbff9ff-qmtfl" podUID="87908ff5-db99-4f3c-813b-162ae40820e7" containerName="console" containerID="cri-o://13bf8ba5a096fdbe12291d8354cc531bd02db860d8f340c945530eac926dec00" gracePeriod=15 Jul 10 00:34:16.436007 ip-10-0-130-146 kubenswrapper[2570]: I0710 00:34:16.435983 2570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-f9cbff9ff-qmtfl" Jul 10 00:34:16.470178 ip-10-0-130-146 kubenswrapper[2570]: I0710 00:34:16.470102 2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/87908ff5-db99-4f3c-813b-162ae40820e7-service-ca\") pod \"87908ff5-db99-4f3c-813b-162ae40820e7\" (UID: \"87908ff5-db99-4f3c-813b-162ae40820e7\") " Jul 10 00:34:16.470178 ip-10-0-130-146 kubenswrapper[2570]: I0710 00:34:16.470142 2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/87908ff5-db99-4f3c-813b-162ae40820e7-oauth-serving-cert\") pod \"87908ff5-db99-4f3c-813b-162ae40820e7\" (UID: \"87908ff5-db99-4f3c-813b-162ae40820e7\") " Jul 10 00:34:16.470390 ip-10-0-130-146 kubenswrapper[2570]: I0710 00:34:16.470180 2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/87908ff5-db99-4f3c-813b-162ae40820e7-console-serving-cert\") pod \"87908ff5-db99-4f3c-813b-162ae40820e7\" (UID: \"87908ff5-db99-4f3c-813b-162ae40820e7\") " Jul 10 00:34:16.470390 ip-10-0-130-146 kubenswrapper[2570]: I0710 00:34:16.470209 2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/87908ff5-db99-4f3c-813b-162ae40820e7-console-oauth-config\") pod \"87908ff5-db99-4f3c-813b-162ae40820e7\" (UID: \"87908ff5-db99-4f3c-813b-162ae40820e7\") " Jul 10 00:34:16.470390 ip-10-0-130-146 kubenswrapper[2570]: I0710 00:34:16.470229 2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/87908ff5-db99-4f3c-813b-162ae40820e7-trusted-ca-bundle\") pod \"87908ff5-db99-4f3c-813b-162ae40820e7\" (UID: \"87908ff5-db99-4f3c-813b-162ae40820e7\") " Jul 10 00:34:16.470390 ip-10-0-130-146 kubenswrapper[2570]: I0710 00:34:16.470260 2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-9xwjl\" (UniqueName: \"kubernetes.io/projected/87908ff5-db99-4f3c-813b-162ae40820e7-kube-api-access-9xwjl\") pod \"87908ff5-db99-4f3c-813b-162ae40820e7\" (UID: \"87908ff5-db99-4f3c-813b-162ae40820e7\") " Jul 10 00:34:16.470390 ip-10-0-130-146 kubenswrapper[2570]: I0710 00:34:16.470292 2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/87908ff5-db99-4f3c-813b-162ae40820e7-console-config\") pod \"87908ff5-db99-4f3c-813b-162ae40820e7\" (UID: \"87908ff5-db99-4f3c-813b-162ae40820e7\") " Jul 10 00:34:16.470635 ip-10-0-130-146 kubenswrapper[2570]: I0710 00:34:16.470568 2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/87908ff5-db99-4f3c-813b-162ae40820e7-service-ca" (OuterVolumeSpecName: "service-ca") pod "87908ff5-db99-4f3c-813b-162ae40820e7" (UID: "87908ff5-db99-4f3c-813b-162ae40820e7"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 10 00:34:16.470696 ip-10-0-130-146 kubenswrapper[2570]: I0710 00:34:16.470648 2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/87908ff5-db99-4f3c-813b-162ae40820e7-trusted-ca-bundle" (OuterVolumeSpecName: "trusted-ca-bundle") pod "87908ff5-db99-4f3c-813b-162ae40820e7" (UID: "87908ff5-db99-4f3c-813b-162ae40820e7"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 10 00:34:16.470749 ip-10-0-130-146 kubenswrapper[2570]: I0710 00:34:16.470732 2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/87908ff5-db99-4f3c-813b-162ae40820e7-oauth-serving-cert" (OuterVolumeSpecName: "oauth-serving-cert") pod "87908ff5-db99-4f3c-813b-162ae40820e7" (UID: "87908ff5-db99-4f3c-813b-162ae40820e7"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 10 00:34:16.470827 ip-10-0-130-146 kubenswrapper[2570]: I0710 00:34:16.470799 2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/87908ff5-db99-4f3c-813b-162ae40820e7-console-config" (OuterVolumeSpecName: "console-config") pod "87908ff5-db99-4f3c-813b-162ae40820e7" (UID: "87908ff5-db99-4f3c-813b-162ae40820e7"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 10 00:34:16.472467 ip-10-0-130-146 kubenswrapper[2570]: I0710 00:34:16.472442 2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/87908ff5-db99-4f3c-813b-162ae40820e7-console-serving-cert" (OuterVolumeSpecName: "console-serving-cert") pod "87908ff5-db99-4f3c-813b-162ae40820e7" (UID: "87908ff5-db99-4f3c-813b-162ae40820e7"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 10 00:34:16.472550 ip-10-0-130-146 kubenswrapper[2570]: I0710 00:34:16.472469 2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/87908ff5-db99-4f3c-813b-162ae40820e7-console-oauth-config" (OuterVolumeSpecName: "console-oauth-config") pod "87908ff5-db99-4f3c-813b-162ae40820e7" (UID: "87908ff5-db99-4f3c-813b-162ae40820e7"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 10 00:34:16.472550 ip-10-0-130-146 kubenswrapper[2570]: I0710 00:34:16.472517 2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/87908ff5-db99-4f3c-813b-162ae40820e7-kube-api-access-9xwjl" (OuterVolumeSpecName: "kube-api-access-9xwjl") pod "87908ff5-db99-4f3c-813b-162ae40820e7" (UID: "87908ff5-db99-4f3c-813b-162ae40820e7"). InnerVolumeSpecName "kube-api-access-9xwjl". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 10 00:34:16.571485 ip-10-0-130-146 kubenswrapper[2570]: I0710 00:34:16.571450 2570 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/87908ff5-db99-4f3c-813b-162ae40820e7-console-oauth-config\") on node \"ip-10-0-130-146.ec2.internal\" DevicePath \"\"" Jul 10 00:34:16.571485 ip-10-0-130-146 kubenswrapper[2570]: I0710 00:34:16.571478 2570 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/87908ff5-db99-4f3c-813b-162ae40820e7-trusted-ca-bundle\") on node \"ip-10-0-130-146.ec2.internal\" DevicePath \"\"" Jul 10 00:34:16.571485 ip-10-0-130-146 kubenswrapper[2570]: I0710 00:34:16.571488 2570 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9xwjl\" (UniqueName: \"kubernetes.io/projected/87908ff5-db99-4f3c-813b-162ae40820e7-kube-api-access-9xwjl\") on node \"ip-10-0-130-146.ec2.internal\" DevicePath \"\"" Jul 10 00:34:16.571707 ip-10-0-130-146 kubenswrapper[2570]: I0710 00:34:16.571498 2570 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/87908ff5-db99-4f3c-813b-162ae40820e7-console-config\") on node \"ip-10-0-130-146.ec2.internal\" DevicePath \"\"" Jul 10 00:34:16.571707 ip-10-0-130-146 kubenswrapper[2570]: I0710 00:34:16.571508 2570 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/87908ff5-db99-4f3c-813b-162ae40820e7-service-ca\") on node \"ip-10-0-130-146.ec2.internal\" DevicePath \"\"" Jul 10 00:34:16.571707 ip-10-0-130-146 kubenswrapper[2570]: I0710 00:34:16.571516 2570 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/87908ff5-db99-4f3c-813b-162ae40820e7-oauth-serving-cert\") on node \"ip-10-0-130-146.ec2.internal\" DevicePath \"\"" Jul 10 00:34:16.571707 ip-10-0-130-146 kubenswrapper[2570]: I0710 00:34:16.571524 2570 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/87908ff5-db99-4f3c-813b-162ae40820e7-console-serving-cert\") on node \"ip-10-0-130-146.ec2.internal\" DevicePath \"\"" Jul 10 00:34:17.227505 ip-10-0-130-146 kubenswrapper[2570]: I0710 00:34:17.227469 2570 generic.go:356] "Generic (PLEG): container finished" podID="87908ff5-db99-4f3c-813b-162ae40820e7" containerID="13bf8ba5a096fdbe12291d8354cc531bd02db860d8f340c945530eac926dec00" exitCode=0 Jul 10 00:34:17.227904 ip-10-0-130-146 kubenswrapper[2570]: I0710 00:34:17.227515 2570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-f9cbff9ff-qmtfl" Jul 10 00:34:17.227904 ip-10-0-130-146 kubenswrapper[2570]: I0710 00:34:17.227514 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-f9cbff9ff-qmtfl" event={"ID":"87908ff5-db99-4f3c-813b-162ae40820e7","Type":"ContainerDied","Data":"13bf8ba5a096fdbe12291d8354cc531bd02db860d8f340c945530eac926dec00"} Jul 10 00:34:17.227904 ip-10-0-130-146 kubenswrapper[2570]: I0710 00:34:17.227569 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-f9cbff9ff-qmtfl" event={"ID":"87908ff5-db99-4f3c-813b-162ae40820e7","Type":"ContainerDied","Data":"382ec788a6d5ffb01d1818465921b09fbad71dcf3c0c17a6ea158e9ac19e076b"} Jul 10 00:34:17.227904 ip-10-0-130-146 kubenswrapper[2570]: I0710 00:34:17.227588 2570 scope.go:117] "RemoveContainer" containerID="13bf8ba5a096fdbe12291d8354cc531bd02db860d8f340c945530eac926dec00" Jul 10 00:34:17.236297 ip-10-0-130-146 kubenswrapper[2570]: I0710 00:34:17.236280 2570 scope.go:117] "RemoveContainer" containerID="13bf8ba5a096fdbe12291d8354cc531bd02db860d8f340c945530eac926dec00" Jul 10 00:34:17.236532 ip-10-0-130-146 kubenswrapper[2570]: E0710 00:34:17.236509 2570 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"13bf8ba5a096fdbe12291d8354cc531bd02db860d8f340c945530eac926dec00\": container with ID starting with 13bf8ba5a096fdbe12291d8354cc531bd02db860d8f340c945530eac926dec00 not found: ID does not exist" containerID="13bf8ba5a096fdbe12291d8354cc531bd02db860d8f340c945530eac926dec00" Jul 10 00:34:17.236574 ip-10-0-130-146 kubenswrapper[2570]: I0710 00:34:17.236540 2570 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"13bf8ba5a096fdbe12291d8354cc531bd02db860d8f340c945530eac926dec00"} err="failed to get container status \"13bf8ba5a096fdbe12291d8354cc531bd02db860d8f340c945530eac926dec00\": rpc error: code = NotFound desc = could not find container \"13bf8ba5a096fdbe12291d8354cc531bd02db860d8f340c945530eac926dec00\": container with ID starting with 13bf8ba5a096fdbe12291d8354cc531bd02db860d8f340c945530eac926dec00 not found: ID does not exist" Jul 10 00:34:17.247650 ip-10-0-130-146 kubenswrapper[2570]: I0710 00:34:17.247628 2570 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-f9cbff9ff-qmtfl"] Jul 10 00:34:17.251203 ip-10-0-130-146 kubenswrapper[2570]: I0710 00:34:17.251182 2570 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-f9cbff9ff-qmtfl"] Jul 10 00:34:17.453712 ip-10-0-130-146 kubenswrapper[2570]: I0710 00:34:17.453682 2570 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="87908ff5-db99-4f3c-813b-162ae40820e7" path="/var/lib/kubelet/pods/87908ff5-db99-4f3c-813b-162ae40820e7/volumes" Jul 10 00:35:03.668042 ip-10-0-130-146 kubenswrapper[2570]: I0710 00:35:03.668002 2570 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540h467v"] Jul 10 00:35:03.668638 ip-10-0-130-146 kubenswrapper[2570]: I0710 00:35:03.668471 2570 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="87908ff5-db99-4f3c-813b-162ae40820e7" containerName="console" Jul 10 00:35:03.668638 ip-10-0-130-146 kubenswrapper[2570]: I0710 00:35:03.668491 2570 state_mem.go:107] "Deleted CPUSet assignment" podUID="87908ff5-db99-4f3c-813b-162ae40820e7" containerName="console" Jul 10 00:35:03.668638 ip-10-0-130-146 kubenswrapper[2570]: I0710 00:35:03.668577 2570 memory_manager.go:368] "RemoveStaleState removing state" podUID="87908ff5-db99-4f3c-813b-162ae40820e7" containerName="console" Jul 10 00:35:03.671705 ip-10-0-130-146 kubenswrapper[2570]: I0710 00:35:03.671682 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540h467v" Jul 10 00:35:03.674254 ip-10-0-130-146 kubenswrapper[2570]: I0710 00:35:03.674235 2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"kube-root-ca.crt\"" Jul 10 00:35:03.674345 ip-10-0-130-146 kubenswrapper[2570]: I0710 00:35:03.674323 2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"default-dockercfg-qclk2\"" Jul 10 00:35:03.675264 ip-10-0-130-146 kubenswrapper[2570]: I0710 00:35:03.675246 2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"openshift-service-ca.crt\"" Jul 10 00:35:03.681461 ip-10-0-130-146 kubenswrapper[2570]: I0710 00:35:03.681442 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540h467v"] Jul 10 00:35:03.842948 ip-10-0-130-146 kubenswrapper[2570]: I0710 00:35:03.842915 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/cfc76edd-6fc8-4645-a536-9e3fc22b1e21-bundle\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540h467v\" (UID: \"cfc76edd-6fc8-4645-a536-9e3fc22b1e21\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540h467v" Jul 10 00:35:03.843133 ip-10-0-130-146 kubenswrapper[2570]: I0710 00:35:03.842996 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/cfc76edd-6fc8-4645-a536-9e3fc22b1e21-util\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540h467v\" (UID: \"cfc76edd-6fc8-4645-a536-9e3fc22b1e21\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540h467v" Jul 10 00:35:03.843133 ip-10-0-130-146 kubenswrapper[2570]: I0710 00:35:03.843022 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cfwdd\" (UniqueName: \"kubernetes.io/projected/cfc76edd-6fc8-4645-a536-9e3fc22b1e21-kube-api-access-cfwdd\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540h467v\" (UID: \"cfc76edd-6fc8-4645-a536-9e3fc22b1e21\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540h467v" Jul 10 00:35:03.943886 ip-10-0-130-146 kubenswrapper[2570]: I0710 00:35:03.943808 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/cfc76edd-6fc8-4645-a536-9e3fc22b1e21-util\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540h467v\" (UID: \"cfc76edd-6fc8-4645-a536-9e3fc22b1e21\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540h467v" Jul 10 00:35:03.943886 ip-10-0-130-146 kubenswrapper[2570]: I0710 00:35:03.943850 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cfwdd\" (UniqueName: \"kubernetes.io/projected/cfc76edd-6fc8-4645-a536-9e3fc22b1e21-kube-api-access-cfwdd\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540h467v\" (UID: \"cfc76edd-6fc8-4645-a536-9e3fc22b1e21\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540h467v" Jul 10 00:35:03.944071 ip-10-0-130-146 kubenswrapper[2570]: I0710 00:35:03.943894 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/cfc76edd-6fc8-4645-a536-9e3fc22b1e21-bundle\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540h467v\" (UID: \"cfc76edd-6fc8-4645-a536-9e3fc22b1e21\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540h467v" Jul 10 00:35:03.944295 ip-10-0-130-146 kubenswrapper[2570]: I0710 00:35:03.944275 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/cfc76edd-6fc8-4645-a536-9e3fc22b1e21-bundle\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540h467v\" (UID: \"cfc76edd-6fc8-4645-a536-9e3fc22b1e21\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540h467v" Jul 10 00:35:03.944337 ip-10-0-130-146 kubenswrapper[2570]: I0710 00:35:03.944275 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/cfc76edd-6fc8-4645-a536-9e3fc22b1e21-util\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540h467v\" (UID: \"cfc76edd-6fc8-4645-a536-9e3fc22b1e21\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540h467v" Jul 10 00:35:03.952304 ip-10-0-130-146 kubenswrapper[2570]: I0710 00:35:03.952286 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-cfwdd\" (UniqueName: \"kubernetes.io/projected/cfc76edd-6fc8-4645-a536-9e3fc22b1e21-kube-api-access-cfwdd\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540h467v\" (UID: \"cfc76edd-6fc8-4645-a536-9e3fc22b1e21\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540h467v" Jul 10 00:35:03.981159 ip-10-0-130-146 kubenswrapper[2570]: I0710 00:35:03.981131 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540h467v" Jul 10 00:35:04.106973 ip-10-0-130-146 kubenswrapper[2570]: I0710 00:35:04.106944 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540h467v"] Jul 10 00:35:04.110150 ip-10-0-130-146 kubenswrapper[2570]: W0710 00:35:04.110124 2570 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcfc76edd_6fc8_4645_a536_9e3fc22b1e21.slice/crio-3410bf9ceab5293e3ddac8f5fddf2960aaa63ac9cae59179ea3e73d010def731 WatchSource:0}: Error finding container 3410bf9ceab5293e3ddac8f5fddf2960aaa63ac9cae59179ea3e73d010def731: Status 404 returned error can't find the container with id 3410bf9ceab5293e3ddac8f5fddf2960aaa63ac9cae59179ea3e73d010def731 Jul 10 00:35:04.409367 ip-10-0-130-146 kubenswrapper[2570]: I0710 00:35:04.409334 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540h467v" event={"ID":"cfc76edd-6fc8-4645-a536-9e3fc22b1e21","Type":"ContainerStarted","Data":"3410bf9ceab5293e3ddac8f5fddf2960aaa63ac9cae59179ea3e73d010def731"} Jul 10 00:35:13.452967 ip-10-0-130-146 kubenswrapper[2570]: I0710 00:35:13.452933 2570 generic.go:356] "Generic (PLEG): container finished" podID="cfc76edd-6fc8-4645-a536-9e3fc22b1e21" containerID="0a34014fcc606486f8d5ae29b6b8e0116a531efbba447259b8b0ee32edc56b2b" exitCode=0 Jul 10 00:35:13.453345 ip-10-0-130-146 kubenswrapper[2570]: I0710 00:35:13.452989 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540h467v" event={"ID":"cfc76edd-6fc8-4645-a536-9e3fc22b1e21","Type":"ContainerDied","Data":"0a34014fcc606486f8d5ae29b6b8e0116a531efbba447259b8b0ee32edc56b2b"} Jul 10 00:35:15.463733 ip-10-0-130-146 kubenswrapper[2570]: I0710 00:35:15.463703 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540h467v" event={"ID":"cfc76edd-6fc8-4645-a536-9e3fc22b1e21","Type":"ContainerStarted","Data":"678055f9a6a4b9e73ffdba295284cc3ac833d2db9fbfbd382c65867ea5abf776"} Jul 10 00:35:16.469533 ip-10-0-130-146 kubenswrapper[2570]: I0710 00:35:16.469500 2570 generic.go:356] "Generic (PLEG): container finished" podID="cfc76edd-6fc8-4645-a536-9e3fc22b1e21" containerID="678055f9a6a4b9e73ffdba295284cc3ac833d2db9fbfbd382c65867ea5abf776" exitCode=0 Jul 10 00:35:16.469533 ip-10-0-130-146 kubenswrapper[2570]: I0710 00:35:16.469537 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540h467v" event={"ID":"cfc76edd-6fc8-4645-a536-9e3fc22b1e21","Type":"ContainerDied","Data":"678055f9a6a4b9e73ffdba295284cc3ac833d2db9fbfbd382c65867ea5abf776"} Jul 10 00:35:26.510569 ip-10-0-130-146 kubenswrapper[2570]: I0710 00:35:26.510536 2570 generic.go:356] "Generic (PLEG): container finished" podID="cfc76edd-6fc8-4645-a536-9e3fc22b1e21" containerID="3d4f04e6ed3207e16279026e4717bb5ffb540f55af3ddc1a7b10d6c38cbd2408" exitCode=0 Jul 10 00:35:26.511002 ip-10-0-130-146 kubenswrapper[2570]: I0710 00:35:26.510615 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540h467v" event={"ID":"cfc76edd-6fc8-4645-a536-9e3fc22b1e21","Type":"ContainerDied","Data":"3d4f04e6ed3207e16279026e4717bb5ffb540f55af3ddc1a7b10d6c38cbd2408"} Jul 10 00:35:27.651659 ip-10-0-130-146 kubenswrapper[2570]: I0710 00:35:27.651635 2570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540h467v" Jul 10 00:35:27.734316 ip-10-0-130-146 kubenswrapper[2570]: I0710 00:35:27.734289 2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/cfc76edd-6fc8-4645-a536-9e3fc22b1e21-bundle\") pod \"cfc76edd-6fc8-4645-a536-9e3fc22b1e21\" (UID: \"cfc76edd-6fc8-4645-a536-9e3fc22b1e21\") " Jul 10 00:35:27.734316 ip-10-0-130-146 kubenswrapper[2570]: I0710 00:35:27.734318 2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/cfc76edd-6fc8-4645-a536-9e3fc22b1e21-util\") pod \"cfc76edd-6fc8-4645-a536-9e3fc22b1e21\" (UID: \"cfc76edd-6fc8-4645-a536-9e3fc22b1e21\") " Jul 10 00:35:27.734486 ip-10-0-130-146 kubenswrapper[2570]: I0710 00:35:27.734337 2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-cfwdd\" (UniqueName: \"kubernetes.io/projected/cfc76edd-6fc8-4645-a536-9e3fc22b1e21-kube-api-access-cfwdd\") pod \"cfc76edd-6fc8-4645-a536-9e3fc22b1e21\" (UID: \"cfc76edd-6fc8-4645-a536-9e3fc22b1e21\") " Jul 10 00:35:27.734939 ip-10-0-130-146 kubenswrapper[2570]: I0710 00:35:27.734907 2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cfc76edd-6fc8-4645-a536-9e3fc22b1e21-bundle" (OuterVolumeSpecName: "bundle") pod "cfc76edd-6fc8-4645-a536-9e3fc22b1e21" (UID: "cfc76edd-6fc8-4645-a536-9e3fc22b1e21"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 10 00:35:27.736523 ip-10-0-130-146 kubenswrapper[2570]: I0710 00:35:27.736492 2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cfc76edd-6fc8-4645-a536-9e3fc22b1e21-kube-api-access-cfwdd" (OuterVolumeSpecName: "kube-api-access-cfwdd") pod "cfc76edd-6fc8-4645-a536-9e3fc22b1e21" (UID: "cfc76edd-6fc8-4645-a536-9e3fc22b1e21"). InnerVolumeSpecName "kube-api-access-cfwdd". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 10 00:35:27.738376 ip-10-0-130-146 kubenswrapper[2570]: I0710 00:35:27.738356 2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cfc76edd-6fc8-4645-a536-9e3fc22b1e21-util" (OuterVolumeSpecName: "util") pod "cfc76edd-6fc8-4645-a536-9e3fc22b1e21" (UID: "cfc76edd-6fc8-4645-a536-9e3fc22b1e21"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 10 00:35:27.835090 ip-10-0-130-146 kubenswrapper[2570]: I0710 00:35:27.834992 2570 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/cfc76edd-6fc8-4645-a536-9e3fc22b1e21-bundle\") on node \"ip-10-0-130-146.ec2.internal\" DevicePath \"\"" Jul 10 00:35:27.835090 ip-10-0-130-146 kubenswrapper[2570]: I0710 00:35:27.835019 2570 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/cfc76edd-6fc8-4645-a536-9e3fc22b1e21-util\") on node \"ip-10-0-130-146.ec2.internal\" DevicePath \"\"" Jul 10 00:35:27.835090 ip-10-0-130-146 kubenswrapper[2570]: I0710 00:35:27.835031 2570 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cfwdd\" (UniqueName: \"kubernetes.io/projected/cfc76edd-6fc8-4645-a536-9e3fc22b1e21-kube-api-access-cfwdd\") on node \"ip-10-0-130-146.ec2.internal\" DevicePath \"\"" Jul 10 00:35:28.520679 ip-10-0-130-146 kubenswrapper[2570]: I0710 00:35:28.520651 2570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540h467v" Jul 10 00:35:28.520844 ip-10-0-130-146 kubenswrapper[2570]: I0710 00:35:28.520650 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540h467v" event={"ID":"cfc76edd-6fc8-4645-a536-9e3fc22b1e21","Type":"ContainerDied","Data":"3410bf9ceab5293e3ddac8f5fddf2960aaa63ac9cae59179ea3e73d010def731"} Jul 10 00:35:28.520844 ip-10-0-130-146 kubenswrapper[2570]: I0710 00:35:28.520774 2570 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="3410bf9ceab5293e3ddac8f5fddf2960aaa63ac9cae59179ea3e73d010def731" Jul 10 00:35:31.642223 ip-10-0-130-146 kubenswrapper[2570]: I0710 00:35:31.642191 2570 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-nk2l4"] Jul 10 00:35:31.642677 ip-10-0-130-146 kubenswrapper[2570]: I0710 00:35:31.642625 2570 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="cfc76edd-6fc8-4645-a536-9e3fc22b1e21" containerName="pull" Jul 10 00:35:31.642677 ip-10-0-130-146 kubenswrapper[2570]: I0710 00:35:31.642642 2570 state_mem.go:107] "Deleted CPUSet assignment" podUID="cfc76edd-6fc8-4645-a536-9e3fc22b1e21" containerName="pull" Jul 10 00:35:31.642794 ip-10-0-130-146 kubenswrapper[2570]: I0710 00:35:31.642680 2570 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="cfc76edd-6fc8-4645-a536-9e3fc22b1e21" containerName="util" Jul 10 00:35:31.642794 ip-10-0-130-146 kubenswrapper[2570]: I0710 00:35:31.642690 2570 state_mem.go:107] "Deleted CPUSet assignment" podUID="cfc76edd-6fc8-4645-a536-9e3fc22b1e21" containerName="util" Jul 10 00:35:31.642794 ip-10-0-130-146 kubenswrapper[2570]: I0710 00:35:31.642713 2570 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="cfc76edd-6fc8-4645-a536-9e3fc22b1e21" containerName="extract" Jul 10 00:35:31.642794 ip-10-0-130-146 kubenswrapper[2570]: I0710 00:35:31.642721 2570 state_mem.go:107] "Deleted CPUSet assignment" podUID="cfc76edd-6fc8-4645-a536-9e3fc22b1e21" containerName="extract" Jul 10 00:35:31.642975 ip-10-0-130-146 kubenswrapper[2570]: I0710 00:35:31.642805 2570 memory_manager.go:368] "RemoveStaleState removing state" podUID="cfc76edd-6fc8-4645-a536-9e3fc22b1e21" containerName="extract" Jul 10 00:35:31.644615 ip-10-0-130-146 kubenswrapper[2570]: I0710 00:35:31.644595 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-nk2l4" Jul 10 00:35:31.647115 ip-10-0-130-146 kubenswrapper[2570]: I0710 00:35:31.647096 2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager-operator\"/\"kube-root-ca.crt\"" Jul 10 00:35:31.647242 ip-10-0-130-146 kubenswrapper[2570]: I0710 00:35:31.647225 2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"cert-manager-operator\"/\"cert-manager-operator-controller-manager-dockercfg-5cdmt\"" Jul 10 00:35:31.647292 ip-10-0-130-146 kubenswrapper[2570]: I0710 00:35:31.647280 2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager-operator\"/\"openshift-service-ca.crt\"" Jul 10 00:35:31.659242 ip-10-0-130-146 kubenswrapper[2570]: I0710 00:35:31.659218 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-nk2l4"] Jul 10 00:35:31.768447 ip-10-0-130-146 kubenswrapper[2570]: I0710 00:35:31.768419 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9cvnn\" (UniqueName: \"kubernetes.io/projected/09bdd361-3d6f-4abc-8122-d48949eb92cf-kube-api-access-9cvnn\") pod \"cert-manager-operator-controller-manager-55c444bf86-nk2l4\" (UID: \"09bdd361-3d6f-4abc-8122-d48949eb92cf\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-nk2l4" Jul 10 00:35:31.768614 ip-10-0-130-146 kubenswrapper[2570]: I0710 00:35:31.768458 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/09bdd361-3d6f-4abc-8122-d48949eb92cf-tmp\") pod \"cert-manager-operator-controller-manager-55c444bf86-nk2l4\" (UID: \"09bdd361-3d6f-4abc-8122-d48949eb92cf\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-nk2l4" Jul 10 00:35:31.869383 ip-10-0-130-146 kubenswrapper[2570]: I0710 00:35:31.869348 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9cvnn\" (UniqueName: \"kubernetes.io/projected/09bdd361-3d6f-4abc-8122-d48949eb92cf-kube-api-access-9cvnn\") pod \"cert-manager-operator-controller-manager-55c444bf86-nk2l4\" (UID: \"09bdd361-3d6f-4abc-8122-d48949eb92cf\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-nk2l4" Jul 10 00:35:31.869383 ip-10-0-130-146 kubenswrapper[2570]: I0710 00:35:31.869391 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/09bdd361-3d6f-4abc-8122-d48949eb92cf-tmp\") pod \"cert-manager-operator-controller-manager-55c444bf86-nk2l4\" (UID: \"09bdd361-3d6f-4abc-8122-d48949eb92cf\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-nk2l4" Jul 10 00:35:31.869746 ip-10-0-130-146 kubenswrapper[2570]: I0710 00:35:31.869729 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/09bdd361-3d6f-4abc-8122-d48949eb92cf-tmp\") pod \"cert-manager-operator-controller-manager-55c444bf86-nk2l4\" (UID: \"09bdd361-3d6f-4abc-8122-d48949eb92cf\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-nk2l4" Jul 10 00:35:31.878917 ip-10-0-130-146 kubenswrapper[2570]: I0710 00:35:31.878894 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9cvnn\" (UniqueName: \"kubernetes.io/projected/09bdd361-3d6f-4abc-8122-d48949eb92cf-kube-api-access-9cvnn\") pod \"cert-manager-operator-controller-manager-55c444bf86-nk2l4\" (UID: \"09bdd361-3d6f-4abc-8122-d48949eb92cf\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-nk2l4" Jul 10 00:35:31.953924 ip-10-0-130-146 kubenswrapper[2570]: I0710 00:35:31.953834 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-nk2l4" Jul 10 00:35:32.087777 ip-10-0-130-146 kubenswrapper[2570]: I0710 00:35:32.087752 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-nk2l4"] Jul 10 00:35:32.090693 ip-10-0-130-146 kubenswrapper[2570]: W0710 00:35:32.090658 2570 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod09bdd361_3d6f_4abc_8122_d48949eb92cf.slice/crio-b337c67d3489d074a515c610929ce31f4f08a155da8de13cffcb65d2f2d193b6 WatchSource:0}: Error finding container b337c67d3489d074a515c610929ce31f4f08a155da8de13cffcb65d2f2d193b6: Status 404 returned error can't find the container with id b337c67d3489d074a515c610929ce31f4f08a155da8de13cffcb65d2f2d193b6 Jul 10 00:35:32.540388 ip-10-0-130-146 kubenswrapper[2570]: I0710 00:35:32.540352 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-nk2l4" event={"ID":"09bdd361-3d6f-4abc-8122-d48949eb92cf","Type":"ContainerStarted","Data":"b337c67d3489d074a515c610929ce31f4f08a155da8de13cffcb65d2f2d193b6"} Jul 10 00:35:34.550939 ip-10-0-130-146 kubenswrapper[2570]: I0710 00:35:34.550901 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-nk2l4" event={"ID":"09bdd361-3d6f-4abc-8122-d48949eb92cf","Type":"ContainerStarted","Data":"5ac9b8ebaf16d182ae158ec1bb9a0ac20b86306c9f5e3e6fa0e4e5f0553f7b52"} Jul 10 00:35:34.574115 ip-10-0-130-146 kubenswrapper[2570]: I0710 00:35:34.574020 2570 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-nk2l4" podStartSLOduration=1.837302779 podStartE2EDuration="3.574004705s" totalImagesPullingTime="1.736701926s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-10 00:35:31 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 00:35:34.571307138 +0000 UTC m=+553.619685861" watchObservedRunningTime="2026-07-10 00:35:34.574004705 +0000 UTC m=+553.622383425" Jul 10 00:35:35.585213 ip-10-0-130-146 kubenswrapper[2570]: I0710 00:35:35.585180 2570 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fx8vff"] Jul 10 00:35:35.587402 ip-10-0-130-146 kubenswrapper[2570]: I0710 00:35:35.587387 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fx8vff" Jul 10 00:35:35.589962 ip-10-0-130-146 kubenswrapper[2570]: I0710 00:35:35.589929 2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"kube-root-ca.crt\"" Jul 10 00:35:35.590193 ip-10-0-130-146 kubenswrapper[2570]: I0710 00:35:35.590178 2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"default-dockercfg-qclk2\"" Jul 10 00:35:35.591158 ip-10-0-130-146 kubenswrapper[2570]: I0710 00:35:35.591138 2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"openshift-service-ca.crt\"" Jul 10 00:35:35.598707 ip-10-0-130-146 kubenswrapper[2570]: I0710 00:35:35.598684 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fx8vff"] Jul 10 00:35:35.701098 ip-10-0-130-146 kubenswrapper[2570]: I0710 00:35:35.701030 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-v4szh\" (UniqueName: \"kubernetes.io/projected/13a06374-f603-4d7a-b613-733145f08cab-kube-api-access-v4szh\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fx8vff\" (UID: \"13a06374-f603-4d7a-b613-733145f08cab\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fx8vff" Jul 10 00:35:35.701299 ip-10-0-130-146 kubenswrapper[2570]: I0710 00:35:35.701155 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/13a06374-f603-4d7a-b613-733145f08cab-bundle\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fx8vff\" (UID: \"13a06374-f603-4d7a-b613-733145f08cab\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fx8vff" Jul 10 00:35:35.701299 ip-10-0-130-146 kubenswrapper[2570]: I0710 00:35:35.701182 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/13a06374-f603-4d7a-b613-733145f08cab-util\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fx8vff\" (UID: \"13a06374-f603-4d7a-b613-733145f08cab\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fx8vff" Jul 10 00:35:35.802416 ip-10-0-130-146 kubenswrapper[2570]: I0710 00:35:35.802377 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v4szh\" (UniqueName: \"kubernetes.io/projected/13a06374-f603-4d7a-b613-733145f08cab-kube-api-access-v4szh\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fx8vff\" (UID: \"13a06374-f603-4d7a-b613-733145f08cab\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fx8vff" Jul 10 00:35:35.802603 ip-10-0-130-146 kubenswrapper[2570]: I0710 00:35:35.802433 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/13a06374-f603-4d7a-b613-733145f08cab-bundle\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fx8vff\" (UID: \"13a06374-f603-4d7a-b613-733145f08cab\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fx8vff" Jul 10 00:35:35.802603 ip-10-0-130-146 kubenswrapper[2570]: I0710 00:35:35.802460 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/13a06374-f603-4d7a-b613-733145f08cab-util\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fx8vff\" (UID: \"13a06374-f603-4d7a-b613-733145f08cab\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fx8vff" Jul 10 00:35:35.802864 ip-10-0-130-146 kubenswrapper[2570]: I0710 00:35:35.802843 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/13a06374-f603-4d7a-b613-733145f08cab-bundle\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fx8vff\" (UID: \"13a06374-f603-4d7a-b613-733145f08cab\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fx8vff" Jul 10 00:35:35.802939 ip-10-0-130-146 kubenswrapper[2570]: I0710 00:35:35.802925 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/13a06374-f603-4d7a-b613-733145f08cab-util\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fx8vff\" (UID: \"13a06374-f603-4d7a-b613-733145f08cab\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fx8vff" Jul 10 00:35:35.808033 ip-10-0-130-146 kubenswrapper[2570]: I0710 00:35:35.808005 2570 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["cert-manager/cert-manager-webhook-6486ffd997-w2nzw"] Jul 10 00:35:35.810425 ip-10-0-130-146 kubenswrapper[2570]: I0710 00:35:35.810401 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-v4szh\" (UniqueName: \"kubernetes.io/projected/13a06374-f603-4d7a-b613-733145f08cab-kube-api-access-v4szh\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fx8vff\" (UID: \"13a06374-f603-4d7a-b613-733145f08cab\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fx8vff" Jul 10 00:35:35.810712 ip-10-0-130-146 kubenswrapper[2570]: I0710 00:35:35.810695 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-webhook-6486ffd997-w2nzw" Jul 10 00:35:35.813252 ip-10-0-130-146 kubenswrapper[2570]: I0710 00:35:35.813229 2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager\"/\"openshift-service-ca.crt\"" Jul 10 00:35:35.813361 ip-10-0-130-146 kubenswrapper[2570]: I0710 00:35:35.813258 2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"cert-manager\"/\"cert-manager-webhook-dockercfg-hv6gn\"" Jul 10 00:35:35.813361 ip-10-0-130-146 kubenswrapper[2570]: I0710 00:35:35.813292 2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager\"/\"kube-root-ca.crt\"" Jul 10 00:35:35.819482 ip-10-0-130-146 kubenswrapper[2570]: I0710 00:35:35.819456 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-webhook-6486ffd997-w2nzw"] Jul 10 00:35:35.896011 ip-10-0-130-146 kubenswrapper[2570]: I0710 00:35:35.895969 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fx8vff" Jul 10 00:35:35.902995 ip-10-0-130-146 kubenswrapper[2570]: I0710 00:35:35.902972 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-g5q2j\" (UniqueName: \"kubernetes.io/projected/8b1498dd-0962-4774-916e-c8c80beed234-kube-api-access-g5q2j\") pod \"cert-manager-webhook-6486ffd997-w2nzw\" (UID: \"8b1498dd-0962-4774-916e-c8c80beed234\") " pod="cert-manager/cert-manager-webhook-6486ffd997-w2nzw" Jul 10 00:35:35.903136 ip-10-0-130-146 kubenswrapper[2570]: I0710 00:35:35.903037 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/8b1498dd-0962-4774-916e-c8c80beed234-bound-sa-token\") pod \"cert-manager-webhook-6486ffd997-w2nzw\" (UID: \"8b1498dd-0962-4774-916e-c8c80beed234\") " pod="cert-manager/cert-manager-webhook-6486ffd997-w2nzw" Jul 10 00:35:36.003736 ip-10-0-130-146 kubenswrapper[2570]: I0710 00:35:36.003705 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/8b1498dd-0962-4774-916e-c8c80beed234-bound-sa-token\") pod \"cert-manager-webhook-6486ffd997-w2nzw\" (UID: \"8b1498dd-0962-4774-916e-c8c80beed234\") " pod="cert-manager/cert-manager-webhook-6486ffd997-w2nzw" Jul 10 00:35:36.003892 ip-10-0-130-146 kubenswrapper[2570]: I0710 00:35:36.003786 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g5q2j\" (UniqueName: \"kubernetes.io/projected/8b1498dd-0962-4774-916e-c8c80beed234-kube-api-access-g5q2j\") pod \"cert-manager-webhook-6486ffd997-w2nzw\" (UID: \"8b1498dd-0962-4774-916e-c8c80beed234\") " pod="cert-manager/cert-manager-webhook-6486ffd997-w2nzw" Jul 10 00:35:36.011689 ip-10-0-130-146 kubenswrapper[2570]: I0710 00:35:36.011662 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/8b1498dd-0962-4774-916e-c8c80beed234-bound-sa-token\") pod \"cert-manager-webhook-6486ffd997-w2nzw\" (UID: \"8b1498dd-0962-4774-916e-c8c80beed234\") " pod="cert-manager/cert-manager-webhook-6486ffd997-w2nzw" Jul 10 00:35:36.011806 ip-10-0-130-146 kubenswrapper[2570]: I0710 00:35:36.011720 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-g5q2j\" (UniqueName: \"kubernetes.io/projected/8b1498dd-0962-4774-916e-c8c80beed234-kube-api-access-g5q2j\") pod \"cert-manager-webhook-6486ffd997-w2nzw\" (UID: \"8b1498dd-0962-4774-916e-c8c80beed234\") " pod="cert-manager/cert-manager-webhook-6486ffd997-w2nzw" Jul 10 00:35:36.020156 ip-10-0-130-146 kubenswrapper[2570]: I0710 00:35:36.020135 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fx8vff"] Jul 10 00:35:36.022515 ip-10-0-130-146 kubenswrapper[2570]: W0710 00:35:36.022493 2570 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod13a06374_f603_4d7a_b613_733145f08cab.slice/crio-9aad1054755ca6014966d9e1c3792b20722d6cccef1bb013d59e531718f614b6 WatchSource:0}: Error finding container 9aad1054755ca6014966d9e1c3792b20722d6cccef1bb013d59e531718f614b6: Status 404 returned error can't find the container with id 9aad1054755ca6014966d9e1c3792b20722d6cccef1bb013d59e531718f614b6 Jul 10 00:35:36.147772 ip-10-0-130-146 kubenswrapper[2570]: I0710 00:35:36.147709 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-webhook-6486ffd997-w2nzw" Jul 10 00:35:36.273657 ip-10-0-130-146 kubenswrapper[2570]: I0710 00:35:36.273626 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-webhook-6486ffd997-w2nzw"] Jul 10 00:35:36.276394 ip-10-0-130-146 kubenswrapper[2570]: W0710 00:35:36.276365 2570 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod8b1498dd_0962_4774_916e_c8c80beed234.slice/crio-ea9de03bf992daabbe6d097a0ab6c76dc961a642d7434952ac55e913b4bf26fe WatchSource:0}: Error finding container ea9de03bf992daabbe6d097a0ab6c76dc961a642d7434952ac55e913b4bf26fe: Status 404 returned error can't find the container with id ea9de03bf992daabbe6d097a0ab6c76dc961a642d7434952ac55e913b4bf26fe Jul 10 00:35:36.560810 ip-10-0-130-146 kubenswrapper[2570]: I0710 00:35:36.560713 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-webhook-6486ffd997-w2nzw" event={"ID":"8b1498dd-0962-4774-916e-c8c80beed234","Type":"ContainerStarted","Data":"ea9de03bf992daabbe6d097a0ab6c76dc961a642d7434952ac55e913b4bf26fe"} Jul 10 00:35:36.562044 ip-10-0-130-146 kubenswrapper[2570]: I0710 00:35:36.562020 2570 generic.go:356] "Generic (PLEG): container finished" podID="13a06374-f603-4d7a-b613-733145f08cab" containerID="730bb015ce974999d1c99e378bdce990541a1f47c3969b62a80e18998721b695" exitCode=0 Jul 10 00:35:36.562220 ip-10-0-130-146 kubenswrapper[2570]: I0710 00:35:36.562117 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fx8vff" event={"ID":"13a06374-f603-4d7a-b613-733145f08cab","Type":"ContainerDied","Data":"730bb015ce974999d1c99e378bdce990541a1f47c3969b62a80e18998721b695"} Jul 10 00:35:36.562220 ip-10-0-130-146 kubenswrapper[2570]: I0710 00:35:36.562173 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fx8vff" event={"ID":"13a06374-f603-4d7a-b613-733145f08cab","Type":"ContainerStarted","Data":"9aad1054755ca6014966d9e1c3792b20722d6cccef1bb013d59e531718f614b6"} Jul 10 00:35:38.107967 ip-10-0-130-146 kubenswrapper[2570]: I0710 00:35:38.107913 2570 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["cert-manager/cert-manager-cainjector-559cdd8c87-57gwd"] Jul 10 00:35:38.114060 ip-10-0-130-146 kubenswrapper[2570]: I0710 00:35:38.114033 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-cainjector-559cdd8c87-57gwd" Jul 10 00:35:38.116539 ip-10-0-130-146 kubenswrapper[2570]: I0710 00:35:38.116519 2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"cert-manager\"/\"cert-manager-cainjector-dockercfg-zk95g\"" Jul 10 00:35:38.120539 ip-10-0-130-146 kubenswrapper[2570]: I0710 00:35:38.120515 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-cainjector-559cdd8c87-57gwd"] Jul 10 00:35:38.220993 ip-10-0-130-146 kubenswrapper[2570]: I0710 00:35:38.220947 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ff56c\" (UniqueName: \"kubernetes.io/projected/b5eb0cd5-6324-4dc7-8aea-7e446ee19af4-kube-api-access-ff56c\") pod \"cert-manager-cainjector-559cdd8c87-57gwd\" (UID: \"b5eb0cd5-6324-4dc7-8aea-7e446ee19af4\") " pod="cert-manager/cert-manager-cainjector-559cdd8c87-57gwd" Jul 10 00:35:38.221208 ip-10-0-130-146 kubenswrapper[2570]: I0710 00:35:38.221008 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/b5eb0cd5-6324-4dc7-8aea-7e446ee19af4-bound-sa-token\") pod \"cert-manager-cainjector-559cdd8c87-57gwd\" (UID: \"b5eb0cd5-6324-4dc7-8aea-7e446ee19af4\") " pod="cert-manager/cert-manager-cainjector-559cdd8c87-57gwd" Jul 10 00:35:38.321955 ip-10-0-130-146 kubenswrapper[2570]: I0710 00:35:38.321910 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ff56c\" (UniqueName: \"kubernetes.io/projected/b5eb0cd5-6324-4dc7-8aea-7e446ee19af4-kube-api-access-ff56c\") pod \"cert-manager-cainjector-559cdd8c87-57gwd\" (UID: \"b5eb0cd5-6324-4dc7-8aea-7e446ee19af4\") " pod="cert-manager/cert-manager-cainjector-559cdd8c87-57gwd" Jul 10 00:35:38.322143 ip-10-0-130-146 kubenswrapper[2570]: I0710 00:35:38.321966 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/b5eb0cd5-6324-4dc7-8aea-7e446ee19af4-bound-sa-token\") pod \"cert-manager-cainjector-559cdd8c87-57gwd\" (UID: \"b5eb0cd5-6324-4dc7-8aea-7e446ee19af4\") " pod="cert-manager/cert-manager-cainjector-559cdd8c87-57gwd" Jul 10 00:35:38.330373 ip-10-0-130-146 kubenswrapper[2570]: I0710 00:35:38.330338 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/b5eb0cd5-6324-4dc7-8aea-7e446ee19af4-bound-sa-token\") pod \"cert-manager-cainjector-559cdd8c87-57gwd\" (UID: \"b5eb0cd5-6324-4dc7-8aea-7e446ee19af4\") " pod="cert-manager/cert-manager-cainjector-559cdd8c87-57gwd" Jul 10 00:35:38.330528 ip-10-0-130-146 kubenswrapper[2570]: I0710 00:35:38.330504 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ff56c\" (UniqueName: \"kubernetes.io/projected/b5eb0cd5-6324-4dc7-8aea-7e446ee19af4-kube-api-access-ff56c\") pod \"cert-manager-cainjector-559cdd8c87-57gwd\" (UID: \"b5eb0cd5-6324-4dc7-8aea-7e446ee19af4\") " pod="cert-manager/cert-manager-cainjector-559cdd8c87-57gwd" Jul 10 00:35:38.427527 ip-10-0-130-146 kubenswrapper[2570]: I0710 00:35:38.427490 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-cainjector-559cdd8c87-57gwd" Jul 10 00:35:38.795446 ip-10-0-130-146 kubenswrapper[2570]: I0710 00:35:38.795414 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-cainjector-559cdd8c87-57gwd"] Jul 10 00:35:38.798698 ip-10-0-130-146 kubenswrapper[2570]: W0710 00:35:38.798661 2570 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podb5eb0cd5_6324_4dc7_8aea_7e446ee19af4.slice/crio-1307ca691a92f9e37f0a604ae367ecc81f5c2ba9425f093275d24ea502127460 WatchSource:0}: Error finding container 1307ca691a92f9e37f0a604ae367ecc81f5c2ba9425f093275d24ea502127460: Status 404 returned error can't find the container with id 1307ca691a92f9e37f0a604ae367ecc81f5c2ba9425f093275d24ea502127460 Jul 10 00:35:39.583615 ip-10-0-130-146 kubenswrapper[2570]: I0710 00:35:39.583580 2570 generic.go:356] "Generic (PLEG): container finished" podID="13a06374-f603-4d7a-b613-733145f08cab" containerID="dd0f9d4f78bcae2f693a538427bb0df125dad2b1865d6fe8148bcdeedd69ca81" exitCode=0 Jul 10 00:35:39.584074 ip-10-0-130-146 kubenswrapper[2570]: I0710 00:35:39.583667 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fx8vff" event={"ID":"13a06374-f603-4d7a-b613-733145f08cab","Type":"ContainerDied","Data":"dd0f9d4f78bcae2f693a538427bb0df125dad2b1865d6fe8148bcdeedd69ca81"} Jul 10 00:35:39.585036 ip-10-0-130-146 kubenswrapper[2570]: I0710 00:35:39.585008 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-cainjector-559cdd8c87-57gwd" event={"ID":"b5eb0cd5-6324-4dc7-8aea-7e446ee19af4","Type":"ContainerStarted","Data":"1307ca691a92f9e37f0a604ae367ecc81f5c2ba9425f093275d24ea502127460"} Jul 10 00:35:40.591171 ip-10-0-130-146 kubenswrapper[2570]: I0710 00:35:40.591137 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-webhook-6486ffd997-w2nzw" event={"ID":"8b1498dd-0962-4774-916e-c8c80beed234","Type":"ContainerStarted","Data":"f06d9571c4d047eafdc438445618180048c2ca8984fa66e09ab9f06d20e200f8"} Jul 10 00:35:40.591649 ip-10-0-130-146 kubenswrapper[2570]: I0710 00:35:40.591215 2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="cert-manager/cert-manager-webhook-6486ffd997-w2nzw" Jul 10 00:35:40.593343 ip-10-0-130-146 kubenswrapper[2570]: I0710 00:35:40.593318 2570 generic.go:356] "Generic (PLEG): container finished" podID="13a06374-f603-4d7a-b613-733145f08cab" containerID="edb44b5b851b09e94dd258a20e0cd247223980519e0787eec3382ee067100565" exitCode=0 Jul 10 00:35:40.593460 ip-10-0-130-146 kubenswrapper[2570]: I0710 00:35:40.593409 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fx8vff" event={"ID":"13a06374-f603-4d7a-b613-733145f08cab","Type":"ContainerDied","Data":"edb44b5b851b09e94dd258a20e0cd247223980519e0787eec3382ee067100565"} Jul 10 00:35:40.594780 ip-10-0-130-146 kubenswrapper[2570]: I0710 00:35:40.594751 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-cainjector-559cdd8c87-57gwd" event={"ID":"b5eb0cd5-6324-4dc7-8aea-7e446ee19af4","Type":"ContainerStarted","Data":"3dfb5f8d66ff1cba15528d0d87cb1386f3ff49e3b2ef9b0121620abd177564ca"} Jul 10 00:35:40.608027 ip-10-0-130-146 kubenswrapper[2570]: I0710 00:35:40.607982 2570 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="cert-manager/cert-manager-webhook-6486ffd997-w2nzw" podStartSLOduration=2.011237444 podStartE2EDuration="5.607967207s" totalImagesPullingTime="3.596729763s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-10 00:35:35 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 00:35:40.605686199 +0000 UTC m=+559.654064949" watchObservedRunningTime="2026-07-10 00:35:40.607967207 +0000 UTC m=+559.656345927" Jul 10 00:35:40.619990 ip-10-0-130-146 kubenswrapper[2570]: I0710 00:35:40.619947 2570 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="cert-manager/cert-manager-cainjector-559cdd8c87-57gwd" podStartSLOduration=1.545052415 podStartE2EDuration="2.619934018s" totalImagesPullingTime="1.074881603s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-10 00:35:38 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 00:35:40.618431908 +0000 UTC m=+559.666810629" watchObservedRunningTime="2026-07-10 00:35:40.619934018 +0000 UTC m=+559.668312739" Jul 10 00:35:41.723760 ip-10-0-130-146 kubenswrapper[2570]: I0710 00:35:41.723730 2570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fx8vff" Jul 10 00:35:41.855541 ip-10-0-130-146 kubenswrapper[2570]: I0710 00:35:41.854460 2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/13a06374-f603-4d7a-b613-733145f08cab-bundle\") pod \"13a06374-f603-4d7a-b613-733145f08cab\" (UID: \"13a06374-f603-4d7a-b613-733145f08cab\") " Jul 10 00:35:41.855541 ip-10-0-130-146 kubenswrapper[2570]: I0710 00:35:41.854531 2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/13a06374-f603-4d7a-b613-733145f08cab-util\") pod \"13a06374-f603-4d7a-b613-733145f08cab\" (UID: \"13a06374-f603-4d7a-b613-733145f08cab\") " Jul 10 00:35:41.855541 ip-10-0-130-146 kubenswrapper[2570]: I0710 00:35:41.854581 2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-v4szh\" (UniqueName: \"kubernetes.io/projected/13a06374-f603-4d7a-b613-733145f08cab-kube-api-access-v4szh\") pod \"13a06374-f603-4d7a-b613-733145f08cab\" (UID: \"13a06374-f603-4d7a-b613-733145f08cab\") " Jul 10 00:35:41.856702 ip-10-0-130-146 kubenswrapper[2570]: I0710 00:35:41.856458 2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/13a06374-f603-4d7a-b613-733145f08cab-bundle" (OuterVolumeSpecName: "bundle") pod "13a06374-f603-4d7a-b613-733145f08cab" (UID: "13a06374-f603-4d7a-b613-733145f08cab"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 10 00:35:41.860942 ip-10-0-130-146 kubenswrapper[2570]: I0710 00:35:41.860913 2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/13a06374-f603-4d7a-b613-733145f08cab-kube-api-access-v4szh" (OuterVolumeSpecName: "kube-api-access-v4szh") pod "13a06374-f603-4d7a-b613-733145f08cab" (UID: "13a06374-f603-4d7a-b613-733145f08cab"). InnerVolumeSpecName "kube-api-access-v4szh". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 10 00:35:41.863230 ip-10-0-130-146 kubenswrapper[2570]: I0710 00:35:41.863198 2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/13a06374-f603-4d7a-b613-733145f08cab-util" (OuterVolumeSpecName: "util") pod "13a06374-f603-4d7a-b613-733145f08cab" (UID: "13a06374-f603-4d7a-b613-733145f08cab"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 10 00:35:41.955493 ip-10-0-130-146 kubenswrapper[2570]: I0710 00:35:41.955460 2570 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/13a06374-f603-4d7a-b613-733145f08cab-bundle\") on node \"ip-10-0-130-146.ec2.internal\" DevicePath \"\"" Jul 10 00:35:41.955493 ip-10-0-130-146 kubenswrapper[2570]: I0710 00:35:41.955490 2570 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/13a06374-f603-4d7a-b613-733145f08cab-util\") on node \"ip-10-0-130-146.ec2.internal\" DevicePath \"\"" Jul 10 00:35:41.955493 ip-10-0-130-146 kubenswrapper[2570]: I0710 00:35:41.955500 2570 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-v4szh\" (UniqueName: \"kubernetes.io/projected/13a06374-f603-4d7a-b613-733145f08cab-kube-api-access-v4szh\") on node \"ip-10-0-130-146.ec2.internal\" DevicePath \"\"" Jul 10 00:35:42.605403 ip-10-0-130-146 kubenswrapper[2570]: I0710 00:35:42.605368 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fx8vff" event={"ID":"13a06374-f603-4d7a-b613-733145f08cab","Type":"ContainerDied","Data":"9aad1054755ca6014966d9e1c3792b20722d6cccef1bb013d59e531718f614b6"} Jul 10 00:35:42.605403 ip-10-0-130-146 kubenswrapper[2570]: I0710 00:35:42.605386 2570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fx8vff" Jul 10 00:35:42.605605 ip-10-0-130-146 kubenswrapper[2570]: I0710 00:35:42.605418 2570 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="9aad1054755ca6014966d9e1c3792b20722d6cccef1bb013d59e531718f614b6" Jul 10 00:35:46.602281 ip-10-0-130-146 kubenswrapper[2570]: I0710 00:35:46.602253 2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="cert-manager/cert-manager-webhook-6486ffd997-w2nzw" Jul 10 00:35:48.568638 ip-10-0-130-146 kubenswrapper[2570]: I0710 00:35:48.568604 2570 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-lws-operator/openshift-lws-operator-fd8ccff4c-4d5fm"] Jul 10 00:35:48.568979 ip-10-0-130-146 kubenswrapper[2570]: I0710 00:35:48.568941 2570 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="13a06374-f603-4d7a-b613-733145f08cab" containerName="extract" Jul 10 00:35:48.568979 ip-10-0-130-146 kubenswrapper[2570]: I0710 00:35:48.568954 2570 state_mem.go:107] "Deleted CPUSet assignment" podUID="13a06374-f603-4d7a-b613-733145f08cab" containerName="extract" Jul 10 00:35:48.568979 ip-10-0-130-146 kubenswrapper[2570]: I0710 00:35:48.568972 2570 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="13a06374-f603-4d7a-b613-733145f08cab" containerName="pull" Jul 10 00:35:48.568979 ip-10-0-130-146 kubenswrapper[2570]: I0710 00:35:48.568977 2570 state_mem.go:107] "Deleted CPUSet assignment" podUID="13a06374-f603-4d7a-b613-733145f08cab" containerName="pull" Jul 10 00:35:48.569138 ip-10-0-130-146 kubenswrapper[2570]: I0710 00:35:48.568990 2570 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="13a06374-f603-4d7a-b613-733145f08cab" containerName="util" Jul 10 00:35:48.569138 ip-10-0-130-146 kubenswrapper[2570]: I0710 00:35:48.568997 2570 state_mem.go:107] "Deleted CPUSet assignment" podUID="13a06374-f603-4d7a-b613-733145f08cab" containerName="util" Jul 10 00:35:48.569138 ip-10-0-130-146 kubenswrapper[2570]: I0710 00:35:48.569062 2570 memory_manager.go:368] "RemoveStaleState removing state" podUID="13a06374-f603-4d7a-b613-733145f08cab" containerName="extract" Jul 10 00:35:48.571878 ip-10-0-130-146 kubenswrapper[2570]: I0710 00:35:48.571861 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-4d5fm" Jul 10 00:35:48.574149 ip-10-0-130-146 kubenswrapper[2570]: I0710 00:35:48.574117 2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"openshift-service-ca.crt\"" Jul 10 00:35:48.574296 ip-10-0-130-146 kubenswrapper[2570]: I0710 00:35:48.574282 2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"kube-root-ca.crt\"" Jul 10 00:35:48.575236 ip-10-0-130-146 kubenswrapper[2570]: I0710 00:35:48.575219 2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"openshift-lws-operator-dockercfg-jpbzf\"" Jul 10 00:35:48.580551 ip-10-0-130-146 kubenswrapper[2570]: I0710 00:35:48.580530 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/openshift-lws-operator-fd8ccff4c-4d5fm"] Jul 10 00:35:48.716314 ip-10-0-130-146 kubenswrapper[2570]: I0710 00:35:48.716283 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jtnck\" (UniqueName: \"kubernetes.io/projected/31700c7c-d83c-4d84-a9be-07bfecdf60f6-kube-api-access-jtnck\") pod \"openshift-lws-operator-fd8ccff4c-4d5fm\" (UID: \"31700c7c-d83c-4d84-a9be-07bfecdf60f6\") " pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-4d5fm" Jul 10 00:35:48.716314 ip-10-0-130-146 kubenswrapper[2570]: I0710 00:35:48.716319 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/31700c7c-d83c-4d84-a9be-07bfecdf60f6-tmp\") pod \"openshift-lws-operator-fd8ccff4c-4d5fm\" (UID: \"31700c7c-d83c-4d84-a9be-07bfecdf60f6\") " pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-4d5fm" Jul 10 00:35:48.817263 ip-10-0-130-146 kubenswrapper[2570]: I0710 00:35:48.817229 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jtnck\" (UniqueName: \"kubernetes.io/projected/31700c7c-d83c-4d84-a9be-07bfecdf60f6-kube-api-access-jtnck\") pod \"openshift-lws-operator-fd8ccff4c-4d5fm\" (UID: \"31700c7c-d83c-4d84-a9be-07bfecdf60f6\") " pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-4d5fm" Jul 10 00:35:48.817263 ip-10-0-130-146 kubenswrapper[2570]: I0710 00:35:48.817266 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/31700c7c-d83c-4d84-a9be-07bfecdf60f6-tmp\") pod \"openshift-lws-operator-fd8ccff4c-4d5fm\" (UID: \"31700c7c-d83c-4d84-a9be-07bfecdf60f6\") " pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-4d5fm" Jul 10 00:35:48.817590 ip-10-0-130-146 kubenswrapper[2570]: I0710 00:35:48.817560 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/31700c7c-d83c-4d84-a9be-07bfecdf60f6-tmp\") pod \"openshift-lws-operator-fd8ccff4c-4d5fm\" (UID: \"31700c7c-d83c-4d84-a9be-07bfecdf60f6\") " pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-4d5fm" Jul 10 00:35:48.830377 ip-10-0-130-146 kubenswrapper[2570]: I0710 00:35:48.825209 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jtnck\" (UniqueName: \"kubernetes.io/projected/31700c7c-d83c-4d84-a9be-07bfecdf60f6-kube-api-access-jtnck\") pod \"openshift-lws-operator-fd8ccff4c-4d5fm\" (UID: \"31700c7c-d83c-4d84-a9be-07bfecdf60f6\") " pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-4d5fm" Jul 10 00:35:48.881149 ip-10-0-130-146 kubenswrapper[2570]: I0710 00:35:48.881122 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-4d5fm" Jul 10 00:35:49.006600 ip-10-0-130-146 kubenswrapper[2570]: I0710 00:35:49.006519 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/openshift-lws-operator-fd8ccff4c-4d5fm"] Jul 10 00:35:49.008914 ip-10-0-130-146 kubenswrapper[2570]: W0710 00:35:49.008881 2570 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod31700c7c_d83c_4d84_a9be_07bfecdf60f6.slice/crio-eb33afba7827062c0f76119b9bc505807aa8ae8936447b066050136ad7539a5e WatchSource:0}: Error finding container eb33afba7827062c0f76119b9bc505807aa8ae8936447b066050136ad7539a5e: Status 404 returned error can't find the container with id eb33afba7827062c0f76119b9bc505807aa8ae8936447b066050136ad7539a5e Jul 10 00:35:49.636910 ip-10-0-130-146 kubenswrapper[2570]: I0710 00:35:49.636875 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-4d5fm" event={"ID":"31700c7c-d83c-4d84-a9be-07bfecdf60f6","Type":"ContainerStarted","Data":"eb33afba7827062c0f76119b9bc505807aa8ae8936447b066050136ad7539a5e"} Jul 10 00:35:51.651078 ip-10-0-130-146 kubenswrapper[2570]: I0710 00:35:51.651024 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-4d5fm" event={"ID":"31700c7c-d83c-4d84-a9be-07bfecdf60f6","Type":"ContainerStarted","Data":"97f94fb6f2a0ac1de25ce1075fa3e119b1bf0dbdac0b7002e608d3fe97da1714"} Jul 10 00:35:51.671924 ip-10-0-130-146 kubenswrapper[2570]: I0710 00:35:51.671872 2570 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-4d5fm" podStartSLOduration=1.206175612 podStartE2EDuration="3.671851867s" totalImagesPullingTime="2.465676255s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-10 00:35:48 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 00:35:51.669408095 +0000 UTC m=+570.717786826" watchObservedRunningTime="2026-07-10 00:35:51.671851867 +0000 UTC m=+570.720230590" Jul 10 00:35:56.374069 ip-10-0-130-146 kubenswrapper[2570]: I0710 00:35:56.374014 2570 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afhmjhx"] Jul 10 00:35:56.376415 ip-10-0-130-146 kubenswrapper[2570]: I0710 00:35:56.376388 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afhmjhx" Jul 10 00:35:56.378956 ip-10-0-130-146 kubenswrapper[2570]: I0710 00:35:56.378935 2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"openshift-service-ca.crt\"" Jul 10 00:35:56.379086 ip-10-0-130-146 kubenswrapper[2570]: I0710 00:35:56.378938 2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"default-dockercfg-qclk2\"" Jul 10 00:35:56.379086 ip-10-0-130-146 kubenswrapper[2570]: I0710 00:35:56.378991 2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"kube-root-ca.crt\"" Jul 10 00:35:56.385804 ip-10-0-130-146 kubenswrapper[2570]: I0710 00:35:56.385782 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afhmjhx"] Jul 10 00:35:56.485375 ip-10-0-130-146 kubenswrapper[2570]: I0710 00:35:56.485341 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kqsdg\" (UniqueName: \"kubernetes.io/projected/8a995fba-0e88-49e1-a7b1-e29698304ae2-kube-api-access-kqsdg\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afhmjhx\" (UID: \"8a995fba-0e88-49e1-a7b1-e29698304ae2\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afhmjhx" Jul 10 00:35:56.485533 ip-10-0-130-146 kubenswrapper[2570]: I0710 00:35:56.485393 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/8a995fba-0e88-49e1-a7b1-e29698304ae2-util\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afhmjhx\" (UID: \"8a995fba-0e88-49e1-a7b1-e29698304ae2\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afhmjhx" Jul 10 00:35:56.485533 ip-10-0-130-146 kubenswrapper[2570]: I0710 00:35:56.485456 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/8a995fba-0e88-49e1-a7b1-e29698304ae2-bundle\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afhmjhx\" (UID: \"8a995fba-0e88-49e1-a7b1-e29698304ae2\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afhmjhx" Jul 10 00:35:56.586262 ip-10-0-130-146 kubenswrapper[2570]: I0710 00:35:56.586228 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/8a995fba-0e88-49e1-a7b1-e29698304ae2-util\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afhmjhx\" (UID: \"8a995fba-0e88-49e1-a7b1-e29698304ae2\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afhmjhx" Jul 10 00:35:56.586428 ip-10-0-130-146 kubenswrapper[2570]: I0710 00:35:56.586283 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/8a995fba-0e88-49e1-a7b1-e29698304ae2-bundle\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afhmjhx\" (UID: \"8a995fba-0e88-49e1-a7b1-e29698304ae2\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afhmjhx" Jul 10 00:35:56.586428 ip-10-0-130-146 kubenswrapper[2570]: I0710 00:35:56.586339 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kqsdg\" (UniqueName: \"kubernetes.io/projected/8a995fba-0e88-49e1-a7b1-e29698304ae2-kube-api-access-kqsdg\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afhmjhx\" (UID: \"8a995fba-0e88-49e1-a7b1-e29698304ae2\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afhmjhx" Jul 10 00:35:56.586723 ip-10-0-130-146 kubenswrapper[2570]: I0710 00:35:56.586704 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/8a995fba-0e88-49e1-a7b1-e29698304ae2-util\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afhmjhx\" (UID: \"8a995fba-0e88-49e1-a7b1-e29698304ae2\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afhmjhx" Jul 10 00:35:56.586802 ip-10-0-130-146 kubenswrapper[2570]: I0710 00:35:56.586738 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/8a995fba-0e88-49e1-a7b1-e29698304ae2-bundle\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afhmjhx\" (UID: \"8a995fba-0e88-49e1-a7b1-e29698304ae2\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afhmjhx" Jul 10 00:35:56.594266 ip-10-0-130-146 kubenswrapper[2570]: I0710 00:35:56.594239 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-kqsdg\" (UniqueName: \"kubernetes.io/projected/8a995fba-0e88-49e1-a7b1-e29698304ae2-kube-api-access-kqsdg\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afhmjhx\" (UID: \"8a995fba-0e88-49e1-a7b1-e29698304ae2\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afhmjhx" Jul 10 00:35:56.686122 ip-10-0-130-146 kubenswrapper[2570]: I0710 00:35:56.686022 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afhmjhx" Jul 10 00:35:56.813510 ip-10-0-130-146 kubenswrapper[2570]: I0710 00:35:56.813486 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afhmjhx"] Jul 10 00:35:56.815312 ip-10-0-130-146 kubenswrapper[2570]: W0710 00:35:56.815284 2570 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8a995fba_0e88_49e1_a7b1_e29698304ae2.slice/crio-c553038c1b1aa87a680a2f0de7531a778545be68540d4ef4ab00c0477cbad52f WatchSource:0}: Error finding container c553038c1b1aa87a680a2f0de7531a778545be68540d4ef4ab00c0477cbad52f: Status 404 returned error can't find the container with id c553038c1b1aa87a680a2f0de7531a778545be68540d4ef4ab00c0477cbad52f Jul 10 00:35:57.680295 ip-10-0-130-146 kubenswrapper[2570]: I0710 00:35:57.680262 2570 generic.go:356] "Generic (PLEG): container finished" podID="8a995fba-0e88-49e1-a7b1-e29698304ae2" containerID="ba313f5cb4f8868ead6770cd3ac424ef73ecb6def47d295f701e59f7f52fb526" exitCode=0 Jul 10 00:35:57.680642 ip-10-0-130-146 kubenswrapper[2570]: I0710 00:35:57.680352 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afhmjhx" event={"ID":"8a995fba-0e88-49e1-a7b1-e29698304ae2","Type":"ContainerDied","Data":"ba313f5cb4f8868ead6770cd3ac424ef73ecb6def47d295f701e59f7f52fb526"} Jul 10 00:35:57.680642 ip-10-0-130-146 kubenswrapper[2570]: I0710 00:35:57.680408 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afhmjhx" event={"ID":"8a995fba-0e88-49e1-a7b1-e29698304ae2","Type":"ContainerStarted","Data":"c553038c1b1aa87a680a2f0de7531a778545be68540d4ef4ab00c0477cbad52f"} Jul 10 00:35:58.687177 ip-10-0-130-146 kubenswrapper[2570]: I0710 00:35:58.687138 2570 generic.go:356] "Generic (PLEG): container finished" podID="8a995fba-0e88-49e1-a7b1-e29698304ae2" containerID="9b5461076ef78acf5ca0b6e9ee927b1504ba3c036cdac725375938ec4fd38cea" exitCode=0 Jul 10 00:35:58.687561 ip-10-0-130-146 kubenswrapper[2570]: I0710 00:35:58.687227 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afhmjhx" event={"ID":"8a995fba-0e88-49e1-a7b1-e29698304ae2","Type":"ContainerDied","Data":"9b5461076ef78acf5ca0b6e9ee927b1504ba3c036cdac725375938ec4fd38cea"} Jul 10 00:35:59.693234 ip-10-0-130-146 kubenswrapper[2570]: I0710 00:35:59.693197 2570 generic.go:356] "Generic (PLEG): container finished" podID="8a995fba-0e88-49e1-a7b1-e29698304ae2" containerID="25d2990e623e0e6c64311665531be704ec2dbae86d8dab86d6caee1206fa50f9" exitCode=0 Jul 10 00:35:59.693579 ip-10-0-130-146 kubenswrapper[2570]: I0710 00:35:59.693238 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afhmjhx" event={"ID":"8a995fba-0e88-49e1-a7b1-e29698304ae2","Type":"ContainerDied","Data":"25d2990e623e0e6c64311665531be704ec2dbae86d8dab86d6caee1206fa50f9"} Jul 10 00:36:00.824082 ip-10-0-130-146 kubenswrapper[2570]: I0710 00:36:00.824035 2570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afhmjhx" Jul 10 00:36:00.918179 ip-10-0-130-146 kubenswrapper[2570]: I0710 00:36:00.918145 2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-kqsdg\" (UniqueName: \"kubernetes.io/projected/8a995fba-0e88-49e1-a7b1-e29698304ae2-kube-api-access-kqsdg\") pod \"8a995fba-0e88-49e1-a7b1-e29698304ae2\" (UID: \"8a995fba-0e88-49e1-a7b1-e29698304ae2\") " Jul 10 00:36:00.918368 ip-10-0-130-146 kubenswrapper[2570]: I0710 00:36:00.918212 2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/8a995fba-0e88-49e1-a7b1-e29698304ae2-bundle\") pod \"8a995fba-0e88-49e1-a7b1-e29698304ae2\" (UID: \"8a995fba-0e88-49e1-a7b1-e29698304ae2\") " Jul 10 00:36:00.918368 ip-10-0-130-146 kubenswrapper[2570]: I0710 00:36:00.918260 2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/8a995fba-0e88-49e1-a7b1-e29698304ae2-util\") pod \"8a995fba-0e88-49e1-a7b1-e29698304ae2\" (UID: \"8a995fba-0e88-49e1-a7b1-e29698304ae2\") " Jul 10 00:36:00.918916 ip-10-0-130-146 kubenswrapper[2570]: I0710 00:36:00.918892 2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8a995fba-0e88-49e1-a7b1-e29698304ae2-bundle" (OuterVolumeSpecName: "bundle") pod "8a995fba-0e88-49e1-a7b1-e29698304ae2" (UID: "8a995fba-0e88-49e1-a7b1-e29698304ae2"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 10 00:36:00.920387 ip-10-0-130-146 kubenswrapper[2570]: I0710 00:36:00.920365 2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8a995fba-0e88-49e1-a7b1-e29698304ae2-kube-api-access-kqsdg" (OuterVolumeSpecName: "kube-api-access-kqsdg") pod "8a995fba-0e88-49e1-a7b1-e29698304ae2" (UID: "8a995fba-0e88-49e1-a7b1-e29698304ae2"). InnerVolumeSpecName "kube-api-access-kqsdg". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 10 00:36:00.923644 ip-10-0-130-146 kubenswrapper[2570]: I0710 00:36:00.923623 2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8a995fba-0e88-49e1-a7b1-e29698304ae2-util" (OuterVolumeSpecName: "util") pod "8a995fba-0e88-49e1-a7b1-e29698304ae2" (UID: "8a995fba-0e88-49e1-a7b1-e29698304ae2"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 10 00:36:01.019166 ip-10-0-130-146 kubenswrapper[2570]: I0710 00:36:01.019094 2570 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/8a995fba-0e88-49e1-a7b1-e29698304ae2-bundle\") on node \"ip-10-0-130-146.ec2.internal\" DevicePath \"\"" Jul 10 00:36:01.019166 ip-10-0-130-146 kubenswrapper[2570]: I0710 00:36:01.019115 2570 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/8a995fba-0e88-49e1-a7b1-e29698304ae2-util\") on node \"ip-10-0-130-146.ec2.internal\" DevicePath \"\"" Jul 10 00:36:01.019166 ip-10-0-130-146 kubenswrapper[2570]: I0710 00:36:01.019124 2570 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kqsdg\" (UniqueName: \"kubernetes.io/projected/8a995fba-0e88-49e1-a7b1-e29698304ae2-kube-api-access-kqsdg\") on node \"ip-10-0-130-146.ec2.internal\" DevicePath \"\"" Jul 10 00:36:01.703869 ip-10-0-130-146 kubenswrapper[2570]: I0710 00:36:01.703839 2570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afhmjhx" Jul 10 00:36:01.703869 ip-10-0-130-146 kubenswrapper[2570]: I0710 00:36:01.703849 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afhmjhx" event={"ID":"8a995fba-0e88-49e1-a7b1-e29698304ae2","Type":"ContainerDied","Data":"c553038c1b1aa87a680a2f0de7531a778545be68540d4ef4ab00c0477cbad52f"} Jul 10 00:36:01.704091 ip-10-0-130-146 kubenswrapper[2570]: I0710 00:36:01.703890 2570 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="c553038c1b1aa87a680a2f0de7531a778545be68540d4ef4ab00c0477cbad52f" Jul 10 00:36:09.658474 ip-10-0-130-146 kubenswrapper[2570]: I0710 00:36:09.658442 2570 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-54ccd4f945-f4q9r"] Jul 10 00:36:09.658934 ip-10-0-130-146 kubenswrapper[2570]: I0710 00:36:09.658773 2570 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8a995fba-0e88-49e1-a7b1-e29698304ae2" containerName="extract" Jul 10 00:36:09.658934 ip-10-0-130-146 kubenswrapper[2570]: I0710 00:36:09.658785 2570 state_mem.go:107] "Deleted CPUSet assignment" podUID="8a995fba-0e88-49e1-a7b1-e29698304ae2" containerName="extract" Jul 10 00:36:09.658934 ip-10-0-130-146 kubenswrapper[2570]: I0710 00:36:09.658805 2570 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8a995fba-0e88-49e1-a7b1-e29698304ae2" containerName="pull" Jul 10 00:36:09.658934 ip-10-0-130-146 kubenswrapper[2570]: I0710 00:36:09.658810 2570 state_mem.go:107] "Deleted CPUSet assignment" podUID="8a995fba-0e88-49e1-a7b1-e29698304ae2" containerName="pull" Jul 10 00:36:09.658934 ip-10-0-130-146 kubenswrapper[2570]: I0710 00:36:09.658819 2570 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8a995fba-0e88-49e1-a7b1-e29698304ae2" containerName="util" Jul 10 00:36:09.658934 ip-10-0-130-146 kubenswrapper[2570]: I0710 00:36:09.658825 2570 state_mem.go:107] "Deleted CPUSet assignment" podUID="8a995fba-0e88-49e1-a7b1-e29698304ae2" containerName="util" Jul 10 00:36:09.658934 ip-10-0-130-146 kubenswrapper[2570]: I0710 00:36:09.658874 2570 memory_manager.go:368] "RemoveStaleState removing state" podUID="8a995fba-0e88-49e1-a7b1-e29698304ae2" containerName="extract" Jul 10 00:36:09.663129 ip-10-0-130-146 kubenswrapper[2570]: I0710 00:36:09.663106 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/opendatahub-operator-controller-manager-54ccd4f945-f4q9r" Jul 10 00:36:09.665745 ip-10-0-130-146 kubenswrapper[2570]: I0710 00:36:09.665718 2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-webhook-cert\"" Jul 10 00:36:09.665868 ip-10-0-130-146 kubenswrapper[2570]: I0710 00:36:09.665765 2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-manager-dockercfg-tlvtm\"" Jul 10 00:36:09.665921 ip-10-0-130-146 kubenswrapper[2570]: I0710 00:36:09.665872 2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"kube-root-ca.crt\"" Jul 10 00:36:09.666012 ip-10-0-130-146 kubenswrapper[2570]: I0710 00:36:09.665996 2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-manager-service-cert\"" Jul 10 00:36:09.666137 ip-10-0-130-146 kubenswrapper[2570]: I0710 00:36:09.666119 2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"openshift-service-ca.crt\"" Jul 10 00:36:09.680328 ip-10-0-130-146 kubenswrapper[2570]: I0710 00:36:09.680301 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-54ccd4f945-f4q9r"] Jul 10 00:36:09.697354 ip-10-0-130-146 kubenswrapper[2570]: I0710 00:36:09.697326 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/b62d531b-b70e-48c2-a443-8bb683e26da2-apiservice-cert\") pod \"opendatahub-operator-controller-manager-54ccd4f945-f4q9r\" (UID: \"b62d531b-b70e-48c2-a443-8bb683e26da2\") " pod="opendatahub/opendatahub-operator-controller-manager-54ccd4f945-f4q9r" Jul 10 00:36:09.697498 ip-10-0-130-146 kubenswrapper[2570]: I0710 00:36:09.697380 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/b62d531b-b70e-48c2-a443-8bb683e26da2-manifests\") pod \"opendatahub-operator-controller-manager-54ccd4f945-f4q9r\" (UID: \"b62d531b-b70e-48c2-a443-8bb683e26da2\") " pod="opendatahub/opendatahub-operator-controller-manager-54ccd4f945-f4q9r" Jul 10 00:36:09.697498 ip-10-0-130-146 kubenswrapper[2570]: I0710 00:36:09.697425 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/b62d531b-b70e-48c2-a443-8bb683e26da2-webhook-cert\") pod \"opendatahub-operator-controller-manager-54ccd4f945-f4q9r\" (UID: \"b62d531b-b70e-48c2-a443-8bb683e26da2\") " pod="opendatahub/opendatahub-operator-controller-manager-54ccd4f945-f4q9r" Jul 10 00:36:09.697585 ip-10-0-130-146 kubenswrapper[2570]: I0710 00:36:09.697492 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/b62d531b-b70e-48c2-a443-8bb683e26da2-tmp\") pod \"opendatahub-operator-controller-manager-54ccd4f945-f4q9r\" (UID: \"b62d531b-b70e-48c2-a443-8bb683e26da2\") " pod="opendatahub/opendatahub-operator-controller-manager-54ccd4f945-f4q9r" Jul 10 00:36:09.697585 ip-10-0-130-146 kubenswrapper[2570]: I0710 00:36:09.697566 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-26mtg\" (UniqueName: \"kubernetes.io/projected/b62d531b-b70e-48c2-a443-8bb683e26da2-kube-api-access-26mtg\") pod \"opendatahub-operator-controller-manager-54ccd4f945-f4q9r\" (UID: \"b62d531b-b70e-48c2-a443-8bb683e26da2\") " pod="opendatahub/opendatahub-operator-controller-manager-54ccd4f945-f4q9r" Jul 10 00:36:09.798834 ip-10-0-130-146 kubenswrapper[2570]: I0710 00:36:09.798796 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/b62d531b-b70e-48c2-a443-8bb683e26da2-apiservice-cert\") pod \"opendatahub-operator-controller-manager-54ccd4f945-f4q9r\" (UID: \"b62d531b-b70e-48c2-a443-8bb683e26da2\") " pod="opendatahub/opendatahub-operator-controller-manager-54ccd4f945-f4q9r" Jul 10 00:36:09.799038 ip-10-0-130-146 kubenswrapper[2570]: I0710 00:36:09.798863 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/b62d531b-b70e-48c2-a443-8bb683e26da2-manifests\") pod \"opendatahub-operator-controller-manager-54ccd4f945-f4q9r\" (UID: \"b62d531b-b70e-48c2-a443-8bb683e26da2\") " pod="opendatahub/opendatahub-operator-controller-manager-54ccd4f945-f4q9r" Jul 10 00:36:09.799038 ip-10-0-130-146 kubenswrapper[2570]: I0710 00:36:09.798892 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/b62d531b-b70e-48c2-a443-8bb683e26da2-webhook-cert\") pod \"opendatahub-operator-controller-manager-54ccd4f945-f4q9r\" (UID: \"b62d531b-b70e-48c2-a443-8bb683e26da2\") " pod="opendatahub/opendatahub-operator-controller-manager-54ccd4f945-f4q9r" Jul 10 00:36:09.799038 ip-10-0-130-146 kubenswrapper[2570]: I0710 00:36:09.798922 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/b62d531b-b70e-48c2-a443-8bb683e26da2-tmp\") pod \"opendatahub-operator-controller-manager-54ccd4f945-f4q9r\" (UID: \"b62d531b-b70e-48c2-a443-8bb683e26da2\") " pod="opendatahub/opendatahub-operator-controller-manager-54ccd4f945-f4q9r" Jul 10 00:36:09.799228 ip-10-0-130-146 kubenswrapper[2570]: I0710 00:36:09.798989 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-26mtg\" (UniqueName: \"kubernetes.io/projected/b62d531b-b70e-48c2-a443-8bb683e26da2-kube-api-access-26mtg\") pod \"opendatahub-operator-controller-manager-54ccd4f945-f4q9r\" (UID: \"b62d531b-b70e-48c2-a443-8bb683e26da2\") " pod="opendatahub/opendatahub-operator-controller-manager-54ccd4f945-f4q9r" Jul 10 00:36:09.799368 ip-10-0-130-146 kubenswrapper[2570]: I0710 00:36:09.799347 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/b62d531b-b70e-48c2-a443-8bb683e26da2-manifests\") pod \"opendatahub-operator-controller-manager-54ccd4f945-f4q9r\" (UID: \"b62d531b-b70e-48c2-a443-8bb683e26da2\") " pod="opendatahub/opendatahub-operator-controller-manager-54ccd4f945-f4q9r" Jul 10 00:36:09.801268 ip-10-0-130-146 kubenswrapper[2570]: I0710 00:36:09.801246 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/b62d531b-b70e-48c2-a443-8bb683e26da2-tmp\") pod \"opendatahub-operator-controller-manager-54ccd4f945-f4q9r\" (UID: \"b62d531b-b70e-48c2-a443-8bb683e26da2\") " pod="opendatahub/opendatahub-operator-controller-manager-54ccd4f945-f4q9r" Jul 10 00:36:09.801404 ip-10-0-130-146 kubenswrapper[2570]: I0710 00:36:09.801386 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/b62d531b-b70e-48c2-a443-8bb683e26da2-webhook-cert\") pod \"opendatahub-operator-controller-manager-54ccd4f945-f4q9r\" (UID: \"b62d531b-b70e-48c2-a443-8bb683e26da2\") " pod="opendatahub/opendatahub-operator-controller-manager-54ccd4f945-f4q9r" Jul 10 00:36:09.801456 ip-10-0-130-146 kubenswrapper[2570]: I0710 00:36:09.801434 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/b62d531b-b70e-48c2-a443-8bb683e26da2-apiservice-cert\") pod \"opendatahub-operator-controller-manager-54ccd4f945-f4q9r\" (UID: \"b62d531b-b70e-48c2-a443-8bb683e26da2\") " pod="opendatahub/opendatahub-operator-controller-manager-54ccd4f945-f4q9r" Jul 10 00:36:09.807513 ip-10-0-130-146 kubenswrapper[2570]: I0710 00:36:09.807487 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-26mtg\" (UniqueName: \"kubernetes.io/projected/b62d531b-b70e-48c2-a443-8bb683e26da2-kube-api-access-26mtg\") pod \"opendatahub-operator-controller-manager-54ccd4f945-f4q9r\" (UID: \"b62d531b-b70e-48c2-a443-8bb683e26da2\") " pod="opendatahub/opendatahub-operator-controller-manager-54ccd4f945-f4q9r" Jul 10 00:36:09.974021 ip-10-0-130-146 kubenswrapper[2570]: I0710 00:36:09.973931 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/opendatahub-operator-controller-manager-54ccd4f945-f4q9r" Jul 10 00:36:10.109233 ip-10-0-130-146 kubenswrapper[2570]: I0710 00:36:10.109206 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-54ccd4f945-f4q9r"] Jul 10 00:36:10.111629 ip-10-0-130-146 kubenswrapper[2570]: W0710 00:36:10.111603 2570 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb62d531b_b70e_48c2_a443_8bb683e26da2.slice/crio-eb0fed494e9694a7fbb24e52c32432e23d7c4e98735706051b11aef064e77b92 WatchSource:0}: Error finding container eb0fed494e9694a7fbb24e52c32432e23d7c4e98735706051b11aef064e77b92: Status 404 returned error can't find the container with id eb0fed494e9694a7fbb24e52c32432e23d7c4e98735706051b11aef064e77b92 Jul 10 00:36:10.745813 ip-10-0-130-146 kubenswrapper[2570]: I0710 00:36:10.745778 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-54ccd4f945-f4q9r" event={"ID":"b62d531b-b70e-48c2-a443-8bb683e26da2","Type":"ContainerStarted","Data":"eb0fed494e9694a7fbb24e52c32432e23d7c4e98735706051b11aef064e77b92"} Jul 10 00:36:13.763624 ip-10-0-130-146 kubenswrapper[2570]: I0710 00:36:13.763586 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-54ccd4f945-f4q9r" event={"ID":"b62d531b-b70e-48c2-a443-8bb683e26da2","Type":"ContainerStarted","Data":"5dd2a971533b1a6bb6d3f325ec067cfe1a7192d176ad57cdc1e977ea9cec6ea0"} Jul 10 00:36:14.769326 ip-10-0-130-146 kubenswrapper[2570]: I0710 00:36:14.769292 2570 generic.go:356] "Generic (PLEG): container finished" podID="b62d531b-b70e-48c2-a443-8bb683e26da2" containerID="5dd2a971533b1a6bb6d3f325ec067cfe1a7192d176ad57cdc1e977ea9cec6ea0" exitCode=0 Jul 10 00:36:14.769796 ip-10-0-130-146 kubenswrapper[2570]: I0710 00:36:14.769373 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-54ccd4f945-f4q9r" event={"ID":"b62d531b-b70e-48c2-a443-8bb683e26da2","Type":"ContainerDied","Data":"5dd2a971533b1a6bb6d3f325ec067cfe1a7192d176ad57cdc1e977ea9cec6ea0"} Jul 10 00:36:15.777541 ip-10-0-130-146 kubenswrapper[2570]: I0710 00:36:15.777503 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-54ccd4f945-f4q9r" event={"ID":"b62d531b-b70e-48c2-a443-8bb683e26da2","Type":"ContainerStarted","Data":"26b9d0320c6af9d239cce96f6b886d0b430eb371c8af1d25d7508fc1d90910ca"} Jul 10 00:36:15.778019 ip-10-0-130-146 kubenswrapper[2570]: I0710 00:36:15.777633 2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/opendatahub-operator-controller-manager-54ccd4f945-f4q9r" Jul 10 00:36:15.800120 ip-10-0-130-146 kubenswrapper[2570]: I0710 00:36:15.800070 2570 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/opendatahub-operator-controller-manager-54ccd4f945-f4q9r" podStartSLOduration=2.279348758 podStartE2EDuration="6.800032096s" totalImagesPullingTime="3.486299412s" totalInitContainerRuntime="1.034383926s" isStatefulPod=false podCreationTimestamp="2026-07-10 00:36:09 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 00:36:15.798003631 +0000 UTC m=+594.846382349" watchObservedRunningTime="2026-07-10 00:36:15.800032096 +0000 UTC m=+594.848410817" Jul 10 00:36:21.294175 ip-10-0-130-146 kubenswrapper[2570]: I0710 00:36:21.294144 2570 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7866497494-d6fcd_2b6cb4ca-79ff-42f0-b080-66af842f22d3/console-operator/2.log" Jul 10 00:36:21.295482 ip-10-0-130-146 kubenswrapper[2570]: I0710 00:36:21.295454 2570 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7866497494-d6fcd_2b6cb4ca-79ff-42f0-b080-66af842f22d3/console-operator/2.log" Jul 10 00:36:21.297637 ip-10-0-130-146 kubenswrapper[2570]: I0710 00:36:21.297614 2570 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-mbs58_c4ef019a-364a-4933-ab85-de86d86167f1/ovn-acl-logging/0.log" Jul 10 00:36:21.298807 ip-10-0-130-146 kubenswrapper[2570]: I0710 00:36:21.298788 2570 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-mbs58_c4ef019a-364a-4933-ab85-de86d86167f1/ovn-acl-logging/0.log" Jul 10 00:36:26.784905 ip-10-0-130-146 kubenswrapper[2570]: I0710 00:36:26.784873 2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/opendatahub-operator-controller-manager-54ccd4f945-f4q9r" Jul 10 00:36:27.678072 ip-10-0-130-146 kubenswrapper[2570]: I0710 00:36:27.678019 2570 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-lws-operator/lws-controller-manager-54495c74ff-9ls6p"] Jul 10 00:36:27.681803 ip-10-0-130-146 kubenswrapper[2570]: I0710 00:36:27.681725 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/lws-controller-manager-54495c74ff-9ls6p" Jul 10 00:36:27.686257 ip-10-0-130-146 kubenswrapper[2570]: I0710 00:36:27.686195 2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"lws-manager-config\"" Jul 10 00:36:27.686257 ip-10-0-130-146 kubenswrapper[2570]: I0710 00:36:27.686219 2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"lws-controller-manager-dockercfg-7tjhb\"" Jul 10 00:36:27.686486 ip-10-0-130-146 kubenswrapper[2570]: I0710 00:36:27.686288 2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"webhook-server-cert\"" Jul 10 00:36:27.686486 ip-10-0-130-146 kubenswrapper[2570]: I0710 00:36:27.686204 2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"metrics-server-cert\"" Jul 10 00:36:27.698443 ip-10-0-130-146 kubenswrapper[2570]: I0710 00:36:27.698415 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/lws-controller-manager-54495c74ff-9ls6p"] Jul 10 00:36:27.755214 ip-10-0-130-146 kubenswrapper[2570]: I0710 00:36:27.755187 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/fafa3ef8-223e-4842-9bb6-cc5f7743b430-cert\") pod \"lws-controller-manager-54495c74ff-9ls6p\" (UID: \"fafa3ef8-223e-4842-9bb6-cc5f7743b430\") " pod="openshift-lws-operator/lws-controller-manager-54495c74ff-9ls6p" Jul 10 00:36:27.755366 ip-10-0-130-146 kubenswrapper[2570]: I0710 00:36:27.755219 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xz2jt\" (UniqueName: \"kubernetes.io/projected/fafa3ef8-223e-4842-9bb6-cc5f7743b430-kube-api-access-xz2jt\") pod \"lws-controller-manager-54495c74ff-9ls6p\" (UID: \"fafa3ef8-223e-4842-9bb6-cc5f7743b430\") " pod="openshift-lws-operator/lws-controller-manager-54495c74ff-9ls6p" Jul 10 00:36:27.755366 ip-10-0-130-146 kubenswrapper[2570]: I0710 00:36:27.755321 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/fafa3ef8-223e-4842-9bb6-cc5f7743b430-manager-config\") pod \"lws-controller-manager-54495c74ff-9ls6p\" (UID: \"fafa3ef8-223e-4842-9bb6-cc5f7743b430\") " pod="openshift-lws-operator/lws-controller-manager-54495c74ff-9ls6p" Jul 10 00:36:27.755442 ip-10-0-130-146 kubenswrapper[2570]: I0710 00:36:27.755388 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/fafa3ef8-223e-4842-9bb6-cc5f7743b430-metrics-cert\") pod \"lws-controller-manager-54495c74ff-9ls6p\" (UID: \"fafa3ef8-223e-4842-9bb6-cc5f7743b430\") " pod="openshift-lws-operator/lws-controller-manager-54495c74ff-9ls6p" Jul 10 00:36:27.856561 ip-10-0-130-146 kubenswrapper[2570]: I0710 00:36:27.856532 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/fafa3ef8-223e-4842-9bb6-cc5f7743b430-manager-config\") pod \"lws-controller-manager-54495c74ff-9ls6p\" (UID: \"fafa3ef8-223e-4842-9bb6-cc5f7743b430\") " pod="openshift-lws-operator/lws-controller-manager-54495c74ff-9ls6p" Jul 10 00:36:27.856935 ip-10-0-130-146 kubenswrapper[2570]: I0710 00:36:27.856570 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/fafa3ef8-223e-4842-9bb6-cc5f7743b430-metrics-cert\") pod \"lws-controller-manager-54495c74ff-9ls6p\" (UID: \"fafa3ef8-223e-4842-9bb6-cc5f7743b430\") " pod="openshift-lws-operator/lws-controller-manager-54495c74ff-9ls6p" Jul 10 00:36:27.856935 ip-10-0-130-146 kubenswrapper[2570]: I0710 00:36:27.856598 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/fafa3ef8-223e-4842-9bb6-cc5f7743b430-cert\") pod \"lws-controller-manager-54495c74ff-9ls6p\" (UID: \"fafa3ef8-223e-4842-9bb6-cc5f7743b430\") " pod="openshift-lws-operator/lws-controller-manager-54495c74ff-9ls6p" Jul 10 00:36:27.856935 ip-10-0-130-146 kubenswrapper[2570]: I0710 00:36:27.856641 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xz2jt\" (UniqueName: \"kubernetes.io/projected/fafa3ef8-223e-4842-9bb6-cc5f7743b430-kube-api-access-xz2jt\") pod \"lws-controller-manager-54495c74ff-9ls6p\" (UID: \"fafa3ef8-223e-4842-9bb6-cc5f7743b430\") " pod="openshift-lws-operator/lws-controller-manager-54495c74ff-9ls6p" Jul 10 00:36:27.857304 ip-10-0-130-146 kubenswrapper[2570]: I0710 00:36:27.857274 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/fafa3ef8-223e-4842-9bb6-cc5f7743b430-manager-config\") pod \"lws-controller-manager-54495c74ff-9ls6p\" (UID: \"fafa3ef8-223e-4842-9bb6-cc5f7743b430\") " pod="openshift-lws-operator/lws-controller-manager-54495c74ff-9ls6p" Jul 10 00:36:27.859086 ip-10-0-130-146 kubenswrapper[2570]: I0710 00:36:27.859064 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/fafa3ef8-223e-4842-9bb6-cc5f7743b430-metrics-cert\") pod \"lws-controller-manager-54495c74ff-9ls6p\" (UID: \"fafa3ef8-223e-4842-9bb6-cc5f7743b430\") " pod="openshift-lws-operator/lws-controller-manager-54495c74ff-9ls6p" Jul 10 00:36:27.859193 ip-10-0-130-146 kubenswrapper[2570]: I0710 00:36:27.859097 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/fafa3ef8-223e-4842-9bb6-cc5f7743b430-cert\") pod \"lws-controller-manager-54495c74ff-9ls6p\" (UID: \"fafa3ef8-223e-4842-9bb6-cc5f7743b430\") " pod="openshift-lws-operator/lws-controller-manager-54495c74ff-9ls6p" Jul 10 00:36:27.864673 ip-10-0-130-146 kubenswrapper[2570]: I0710 00:36:27.864649 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xz2jt\" (UniqueName: \"kubernetes.io/projected/fafa3ef8-223e-4842-9bb6-cc5f7743b430-kube-api-access-xz2jt\") pod \"lws-controller-manager-54495c74ff-9ls6p\" (UID: \"fafa3ef8-223e-4842-9bb6-cc5f7743b430\") " pod="openshift-lws-operator/lws-controller-manager-54495c74ff-9ls6p" Jul 10 00:36:27.994086 ip-10-0-130-146 kubenswrapper[2570]: I0710 00:36:27.993965 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/lws-controller-manager-54495c74ff-9ls6p" Jul 10 00:36:28.131770 ip-10-0-130-146 kubenswrapper[2570]: I0710 00:36:28.131746 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/lws-controller-manager-54495c74ff-9ls6p"] Jul 10 00:36:28.133775 ip-10-0-130-146 kubenswrapper[2570]: W0710 00:36:28.133748 2570 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfafa3ef8_223e_4842_9bb6_cc5f7743b430.slice/crio-5b4d0f217ddc29c70e88c76f4a5178e9043400a2ea72f66c5b16d3b29ca1018c WatchSource:0}: Error finding container 5b4d0f217ddc29c70e88c76f4a5178e9043400a2ea72f66c5b16d3b29ca1018c: Status 404 returned error can't find the container with id 5b4d0f217ddc29c70e88c76f4a5178e9043400a2ea72f66c5b16d3b29ca1018c Jul 10 00:36:28.843311 ip-10-0-130-146 kubenswrapper[2570]: I0710 00:36:28.843269 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/lws-controller-manager-54495c74ff-9ls6p" event={"ID":"fafa3ef8-223e-4842-9bb6-cc5f7743b430","Type":"ContainerStarted","Data":"5b4d0f217ddc29c70e88c76f4a5178e9043400a2ea72f66c5b16d3b29ca1018c"} Jul 10 00:36:29.849860 ip-10-0-130-146 kubenswrapper[2570]: I0710 00:36:29.849817 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/lws-controller-manager-54495c74ff-9ls6p" event={"ID":"fafa3ef8-223e-4842-9bb6-cc5f7743b430","Type":"ContainerStarted","Data":"5fddcf035e4f0a456ee1c86865b5eb17d239ae46192b5c92709364dfbefa73de"} Jul 10 00:36:29.850330 ip-10-0-130-146 kubenswrapper[2570]: I0710 00:36:29.849962 2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-lws-operator/lws-controller-manager-54495c74ff-9ls6p" Jul 10 00:36:29.888704 ip-10-0-130-146 kubenswrapper[2570]: I0710 00:36:29.888648 2570 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-lws-operator/lws-controller-manager-54495c74ff-9ls6p" podStartSLOduration=1.338165943 podStartE2EDuration="2.888630412s" totalImagesPullingTime="1.550464469s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-10 00:36:27 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 00:36:29.884090306 +0000 UTC m=+608.932469054" watchObservedRunningTime="2026-07-10 00:36:29.888630412 +0000 UTC m=+608.937009133" Jul 10 00:36:30.096267 ip-10-0-130-146 kubenswrapper[2570]: I0710 00:36:30.096197 2570 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/kube-auth-proxy-7cb6b7dbc6-wwmdk"] Jul 10 00:36:30.099899 ip-10-0-130-146 kubenswrapper[2570]: I0710 00:36:30.099877 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/kube-auth-proxy-7cb6b7dbc6-wwmdk" Jul 10 00:36:30.103270 ip-10-0-130-146 kubenswrapper[2570]: I0710 00:36:30.103197 2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-tls\"" Jul 10 00:36:30.103270 ip-10-0-130-146 kubenswrapper[2570]: I0710 00:36:30.103209 2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"kube-root-ca.crt\"" Jul 10 00:36:30.103465 ip-10-0-130-146 kubenswrapper[2570]: I0710 00:36:30.103284 2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-creds\"" Jul 10 00:36:30.103465 ip-10-0-130-146 kubenswrapper[2570]: I0710 00:36:30.103381 2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-service-ca.crt\"" Jul 10 00:36:30.103660 ip-10-0-130-146 kubenswrapper[2570]: I0710 00:36:30.103639 2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-dockercfg-zjphx\"" Jul 10 00:36:30.111565 ip-10-0-130-146 kubenswrapper[2570]: I0710 00:36:30.111542 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/kube-auth-proxy-7cb6b7dbc6-wwmdk"] Jul 10 00:36:30.178120 ip-10-0-130-146 kubenswrapper[2570]: I0710 00:36:30.178096 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-d74vl\" (UniqueName: \"kubernetes.io/projected/49d9ee95-a4bb-4549-b5dc-92c63e33c475-kube-api-access-d74vl\") pod \"kube-auth-proxy-7cb6b7dbc6-wwmdk\" (UID: \"49d9ee95-a4bb-4549-b5dc-92c63e33c475\") " pod="openshift-ingress/kube-auth-proxy-7cb6b7dbc6-wwmdk" Jul 10 00:36:30.178264 ip-10-0-130-146 kubenswrapper[2570]: I0710 00:36:30.178128 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/49d9ee95-a4bb-4549-b5dc-92c63e33c475-tmp\") pod \"kube-auth-proxy-7cb6b7dbc6-wwmdk\" (UID: \"49d9ee95-a4bb-4549-b5dc-92c63e33c475\") " pod="openshift-ingress/kube-auth-proxy-7cb6b7dbc6-wwmdk" Jul 10 00:36:30.178264 ip-10-0-130-146 kubenswrapper[2570]: I0710 00:36:30.178151 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/49d9ee95-a4bb-4549-b5dc-92c63e33c475-tls-certs\") pod \"kube-auth-proxy-7cb6b7dbc6-wwmdk\" (UID: \"49d9ee95-a4bb-4549-b5dc-92c63e33c475\") " pod="openshift-ingress/kube-auth-proxy-7cb6b7dbc6-wwmdk" Jul 10 00:36:30.279007 ip-10-0-130-146 kubenswrapper[2570]: I0710 00:36:30.278962 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d74vl\" (UniqueName: \"kubernetes.io/projected/49d9ee95-a4bb-4549-b5dc-92c63e33c475-kube-api-access-d74vl\") pod \"kube-auth-proxy-7cb6b7dbc6-wwmdk\" (UID: \"49d9ee95-a4bb-4549-b5dc-92c63e33c475\") " pod="openshift-ingress/kube-auth-proxy-7cb6b7dbc6-wwmdk" Jul 10 00:36:30.279203 ip-10-0-130-146 kubenswrapper[2570]: I0710 00:36:30.279015 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/49d9ee95-a4bb-4549-b5dc-92c63e33c475-tmp\") pod \"kube-auth-proxy-7cb6b7dbc6-wwmdk\" (UID: \"49d9ee95-a4bb-4549-b5dc-92c63e33c475\") " pod="openshift-ingress/kube-auth-proxy-7cb6b7dbc6-wwmdk" Jul 10 00:36:30.279203 ip-10-0-130-146 kubenswrapper[2570]: I0710 00:36:30.279074 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/49d9ee95-a4bb-4549-b5dc-92c63e33c475-tls-certs\") pod \"kube-auth-proxy-7cb6b7dbc6-wwmdk\" (UID: \"49d9ee95-a4bb-4549-b5dc-92c63e33c475\") " pod="openshift-ingress/kube-auth-proxy-7cb6b7dbc6-wwmdk" Jul 10 00:36:30.282027 ip-10-0-130-146 kubenswrapper[2570]: I0710 00:36:30.281998 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/49d9ee95-a4bb-4549-b5dc-92c63e33c475-tmp\") pod \"kube-auth-proxy-7cb6b7dbc6-wwmdk\" (UID: \"49d9ee95-a4bb-4549-b5dc-92c63e33c475\") " pod="openshift-ingress/kube-auth-proxy-7cb6b7dbc6-wwmdk" Jul 10 00:36:30.282191 ip-10-0-130-146 kubenswrapper[2570]: I0710 00:36:30.282122 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/49d9ee95-a4bb-4549-b5dc-92c63e33c475-tls-certs\") pod \"kube-auth-proxy-7cb6b7dbc6-wwmdk\" (UID: \"49d9ee95-a4bb-4549-b5dc-92c63e33c475\") " pod="openshift-ingress/kube-auth-proxy-7cb6b7dbc6-wwmdk" Jul 10 00:36:30.297115 ip-10-0-130-146 kubenswrapper[2570]: I0710 00:36:30.297092 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-d74vl\" (UniqueName: \"kubernetes.io/projected/49d9ee95-a4bb-4549-b5dc-92c63e33c475-kube-api-access-d74vl\") pod \"kube-auth-proxy-7cb6b7dbc6-wwmdk\" (UID: \"49d9ee95-a4bb-4549-b5dc-92c63e33c475\") " pod="openshift-ingress/kube-auth-proxy-7cb6b7dbc6-wwmdk" Jul 10 00:36:30.412552 ip-10-0-130-146 kubenswrapper[2570]: I0710 00:36:30.412520 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/kube-auth-proxy-7cb6b7dbc6-wwmdk" Jul 10 00:36:30.591636 ip-10-0-130-146 kubenswrapper[2570]: I0710 00:36:30.591604 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/kube-auth-proxy-7cb6b7dbc6-wwmdk"] Jul 10 00:36:30.595614 ip-10-0-130-146 kubenswrapper[2570]: W0710 00:36:30.595578 2570 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod49d9ee95_a4bb_4549_b5dc_92c63e33c475.slice/crio-468d640c0f7fef7c2239e9b660ce3d47fefe1d3c173e74d08387a0edef132936 WatchSource:0}: Error finding container 468d640c0f7fef7c2239e9b660ce3d47fefe1d3c173e74d08387a0edef132936: Status 404 returned error can't find the container with id 468d640c0f7fef7c2239e9b660ce3d47fefe1d3c173e74d08387a0edef132936 Jul 10 00:36:30.597548 ip-10-0-130-146 kubenswrapper[2570]: I0710 00:36:30.597529 2570 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Jul 10 00:36:30.855730 ip-10-0-130-146 kubenswrapper[2570]: I0710 00:36:30.855634 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/kube-auth-proxy-7cb6b7dbc6-wwmdk" event={"ID":"49d9ee95-a4bb-4549-b5dc-92c63e33c475","Type":"ContainerStarted","Data":"468d640c0f7fef7c2239e9b660ce3d47fefe1d3c173e74d08387a0edef132936"} Jul 10 00:36:39.908015 ip-10-0-130-146 kubenswrapper[2570]: I0710 00:36:39.907959 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/kube-auth-proxy-7cb6b7dbc6-wwmdk" event={"ID":"49d9ee95-a4bb-4549-b5dc-92c63e33c475","Type":"ContainerStarted","Data":"3b408df4f7cf7abaced2f8067a83801c0279ff06295255c29a6f1e78a0854a98"} Jul 10 00:36:39.931145 ip-10-0-130-146 kubenswrapper[2570]: I0710 00:36:39.931093 2570 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/kube-auth-proxy-7cb6b7dbc6-wwmdk" podStartSLOduration=0.801543284 podStartE2EDuration="9.931076129s" totalImagesPullingTime="9.129532845s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-10 00:36:30 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 00:36:39.92897236 +0000 UTC m=+618.977351082" watchObservedRunningTime="2026-07-10 00:36:39.931076129 +0000 UTC m=+618.979454847" Jul 10 00:36:40.857808 ip-10-0-130-146 kubenswrapper[2570]: I0710 00:36:40.857776 2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-lws-operator/lws-controller-manager-54495c74ff-9ls6p" Jul 10 00:37:36.060888 ip-10-0-130-146 kubenswrapper[2570]: I0710 00:37:36.060856 2570 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-6b8698786d-qv9lq"] Jul 10 00:37:36.064504 ip-10-0-130-146 kubenswrapper[2570]: I0710 00:37:36.064476 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-6b8698786d-qv9lq" Jul 10 00:37:36.080720 ip-10-0-130-146 kubenswrapper[2570]: I0710 00:37:36.080697 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-6b8698786d-qv9lq"] Jul 10 00:37:36.168544 ip-10-0-130-146 kubenswrapper[2570]: I0710 00:37:36.168509 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/1ecefbf9-8ebe-4bc6-9ceb-385d5d5b0d06-trusted-ca-bundle\") pod \"console-6b8698786d-qv9lq\" (UID: \"1ecefbf9-8ebe-4bc6-9ceb-385d5d5b0d06\") " pod="openshift-console/console-6b8698786d-qv9lq" Jul 10 00:37:36.168763 ip-10-0-130-146 kubenswrapper[2570]: I0710 00:37:36.168583 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/1ecefbf9-8ebe-4bc6-9ceb-385d5d5b0d06-service-ca\") pod \"console-6b8698786d-qv9lq\" (UID: \"1ecefbf9-8ebe-4bc6-9ceb-385d5d5b0d06\") " pod="openshift-console/console-6b8698786d-qv9lq" Jul 10 00:37:36.168763 ip-10-0-130-146 kubenswrapper[2570]: I0710 00:37:36.168619 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/1ecefbf9-8ebe-4bc6-9ceb-385d5d5b0d06-console-config\") pod \"console-6b8698786d-qv9lq\" (UID: \"1ecefbf9-8ebe-4bc6-9ceb-385d5d5b0d06\") " pod="openshift-console/console-6b8698786d-qv9lq" Jul 10 00:37:36.168763 ip-10-0-130-146 kubenswrapper[2570]: I0710 00:37:36.168638 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/1ecefbf9-8ebe-4bc6-9ceb-385d5d5b0d06-console-oauth-config\") pod \"console-6b8698786d-qv9lq\" (UID: \"1ecefbf9-8ebe-4bc6-9ceb-385d5d5b0d06\") " pod="openshift-console/console-6b8698786d-qv9lq" Jul 10 00:37:36.168763 ip-10-0-130-146 kubenswrapper[2570]: I0710 00:37:36.168700 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/1ecefbf9-8ebe-4bc6-9ceb-385d5d5b0d06-console-serving-cert\") pod \"console-6b8698786d-qv9lq\" (UID: \"1ecefbf9-8ebe-4bc6-9ceb-385d5d5b0d06\") " pod="openshift-console/console-6b8698786d-qv9lq" Jul 10 00:37:36.168763 ip-10-0-130-146 kubenswrapper[2570]: I0710 00:37:36.168726 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/1ecefbf9-8ebe-4bc6-9ceb-385d5d5b0d06-oauth-serving-cert\") pod \"console-6b8698786d-qv9lq\" (UID: \"1ecefbf9-8ebe-4bc6-9ceb-385d5d5b0d06\") " pod="openshift-console/console-6b8698786d-qv9lq" Jul 10 00:37:36.168763 ip-10-0-130-146 kubenswrapper[2570]: I0710 00:37:36.168743 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-c7plz\" (UniqueName: \"kubernetes.io/projected/1ecefbf9-8ebe-4bc6-9ceb-385d5d5b0d06-kube-api-access-c7plz\") pod \"console-6b8698786d-qv9lq\" (UID: \"1ecefbf9-8ebe-4bc6-9ceb-385d5d5b0d06\") " pod="openshift-console/console-6b8698786d-qv9lq" Jul 10 00:37:36.269670 ip-10-0-130-146 kubenswrapper[2570]: I0710 00:37:36.269626 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/1ecefbf9-8ebe-4bc6-9ceb-385d5d5b0d06-service-ca\") pod \"console-6b8698786d-qv9lq\" (UID: \"1ecefbf9-8ebe-4bc6-9ceb-385d5d5b0d06\") " pod="openshift-console/console-6b8698786d-qv9lq" Jul 10 00:37:36.269845 ip-10-0-130-146 kubenswrapper[2570]: I0710 00:37:36.269685 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/1ecefbf9-8ebe-4bc6-9ceb-385d5d5b0d06-console-config\") pod \"console-6b8698786d-qv9lq\" (UID: \"1ecefbf9-8ebe-4bc6-9ceb-385d5d5b0d06\") " pod="openshift-console/console-6b8698786d-qv9lq" Jul 10 00:37:36.269845 ip-10-0-130-146 kubenswrapper[2570]: I0710 00:37:36.269714 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/1ecefbf9-8ebe-4bc6-9ceb-385d5d5b0d06-console-oauth-config\") pod \"console-6b8698786d-qv9lq\" (UID: \"1ecefbf9-8ebe-4bc6-9ceb-385d5d5b0d06\") " pod="openshift-console/console-6b8698786d-qv9lq" Jul 10 00:37:36.269845 ip-10-0-130-146 kubenswrapper[2570]: I0710 00:37:36.269758 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/1ecefbf9-8ebe-4bc6-9ceb-385d5d5b0d06-console-serving-cert\") pod \"console-6b8698786d-qv9lq\" (UID: \"1ecefbf9-8ebe-4bc6-9ceb-385d5d5b0d06\") " pod="openshift-console/console-6b8698786d-qv9lq" Jul 10 00:37:36.269845 ip-10-0-130-146 kubenswrapper[2570]: I0710 00:37:36.269798 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/1ecefbf9-8ebe-4bc6-9ceb-385d5d5b0d06-oauth-serving-cert\") pod \"console-6b8698786d-qv9lq\" (UID: \"1ecefbf9-8ebe-4bc6-9ceb-385d5d5b0d06\") " pod="openshift-console/console-6b8698786d-qv9lq" Jul 10 00:37:36.269845 ip-10-0-130-146 kubenswrapper[2570]: I0710 00:37:36.269818 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c7plz\" (UniqueName: \"kubernetes.io/projected/1ecefbf9-8ebe-4bc6-9ceb-385d5d5b0d06-kube-api-access-c7plz\") pod \"console-6b8698786d-qv9lq\" (UID: \"1ecefbf9-8ebe-4bc6-9ceb-385d5d5b0d06\") " pod="openshift-console/console-6b8698786d-qv9lq" Jul 10 00:37:36.270154 ip-10-0-130-146 kubenswrapper[2570]: I0710 00:37:36.269849 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/1ecefbf9-8ebe-4bc6-9ceb-385d5d5b0d06-trusted-ca-bundle\") pod \"console-6b8698786d-qv9lq\" (UID: \"1ecefbf9-8ebe-4bc6-9ceb-385d5d5b0d06\") " pod="openshift-console/console-6b8698786d-qv9lq" Jul 10 00:37:36.270487 ip-10-0-130-146 kubenswrapper[2570]: I0710 00:37:36.270457 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/1ecefbf9-8ebe-4bc6-9ceb-385d5d5b0d06-console-config\") pod \"console-6b8698786d-qv9lq\" (UID: \"1ecefbf9-8ebe-4bc6-9ceb-385d5d5b0d06\") " pod="openshift-console/console-6b8698786d-qv9lq" Jul 10 00:37:36.270601 ip-10-0-130-146 kubenswrapper[2570]: I0710 00:37:36.270458 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/1ecefbf9-8ebe-4bc6-9ceb-385d5d5b0d06-service-ca\") pod \"console-6b8698786d-qv9lq\" (UID: \"1ecefbf9-8ebe-4bc6-9ceb-385d5d5b0d06\") " pod="openshift-console/console-6b8698786d-qv9lq" Jul 10 00:37:36.270697 ip-10-0-130-146 kubenswrapper[2570]: I0710 00:37:36.270666 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/1ecefbf9-8ebe-4bc6-9ceb-385d5d5b0d06-oauth-serving-cert\") pod \"console-6b8698786d-qv9lq\" (UID: \"1ecefbf9-8ebe-4bc6-9ceb-385d5d5b0d06\") " pod="openshift-console/console-6b8698786d-qv9lq" Jul 10 00:37:36.270830 ip-10-0-130-146 kubenswrapper[2570]: I0710 00:37:36.270816 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/1ecefbf9-8ebe-4bc6-9ceb-385d5d5b0d06-trusted-ca-bundle\") pod \"console-6b8698786d-qv9lq\" (UID: \"1ecefbf9-8ebe-4bc6-9ceb-385d5d5b0d06\") " pod="openshift-console/console-6b8698786d-qv9lq" Jul 10 00:37:36.272251 ip-10-0-130-146 kubenswrapper[2570]: I0710 00:37:36.272232 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/1ecefbf9-8ebe-4bc6-9ceb-385d5d5b0d06-console-oauth-config\") pod \"console-6b8698786d-qv9lq\" (UID: \"1ecefbf9-8ebe-4bc6-9ceb-385d5d5b0d06\") " pod="openshift-console/console-6b8698786d-qv9lq" Jul 10 00:37:36.272343 ip-10-0-130-146 kubenswrapper[2570]: I0710 00:37:36.272323 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/1ecefbf9-8ebe-4bc6-9ceb-385d5d5b0d06-console-serving-cert\") pod \"console-6b8698786d-qv9lq\" (UID: \"1ecefbf9-8ebe-4bc6-9ceb-385d5d5b0d06\") " pod="openshift-console/console-6b8698786d-qv9lq" Jul 10 00:37:36.278224 ip-10-0-130-146 kubenswrapper[2570]: I0710 00:37:36.278205 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-c7plz\" (UniqueName: \"kubernetes.io/projected/1ecefbf9-8ebe-4bc6-9ceb-385d5d5b0d06-kube-api-access-c7plz\") pod \"console-6b8698786d-qv9lq\" (UID: \"1ecefbf9-8ebe-4bc6-9ceb-385d5d5b0d06\") " pod="openshift-console/console-6b8698786d-qv9lq" Jul 10 00:37:36.375390 ip-10-0-130-146 kubenswrapper[2570]: I0710 00:37:36.375363 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-6b8698786d-qv9lq" Jul 10 00:37:36.507673 ip-10-0-130-146 kubenswrapper[2570]: I0710 00:37:36.507652 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-6b8698786d-qv9lq"] Jul 10 00:37:36.509532 ip-10-0-130-146 kubenswrapper[2570]: W0710 00:37:36.509504 2570 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1ecefbf9_8ebe_4bc6_9ceb_385d5d5b0d06.slice/crio-a2a5c7927ac19a451ab3f5ded2becd8427a2a00da310991ca4f075fd729c73c5 WatchSource:0}: Error finding container a2a5c7927ac19a451ab3f5ded2becd8427a2a00da310991ca4f075fd729c73c5: Status 404 returned error can't find the container with id a2a5c7927ac19a451ab3f5ded2becd8427a2a00da310991ca4f075fd729c73c5 Jul 10 00:37:37.178870 ip-10-0-130-146 kubenswrapper[2570]: I0710 00:37:37.178837 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-6b8698786d-qv9lq" event={"ID":"1ecefbf9-8ebe-4bc6-9ceb-385d5d5b0d06","Type":"ContainerStarted","Data":"f19615b5b7d4dc3ba7decf2c0192109ce355c4dc16132339d135e7fb4476afc5"} Jul 10 00:37:37.179270 ip-10-0-130-146 kubenswrapper[2570]: I0710 00:37:37.178883 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-6b8698786d-qv9lq" event={"ID":"1ecefbf9-8ebe-4bc6-9ceb-385d5d5b0d06","Type":"ContainerStarted","Data":"a2a5c7927ac19a451ab3f5ded2becd8427a2a00da310991ca4f075fd729c73c5"} Jul 10 00:37:37.198028 ip-10-0-130-146 kubenswrapper[2570]: I0710 00:37:37.197970 2570 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-6b8698786d-qv9lq" podStartSLOduration=1.197955882 podStartE2EDuration="1.197955882s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-10 00:37:36 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 00:37:37.196188956 +0000 UTC m=+676.244567696" watchObservedRunningTime="2026-07-10 00:37:37.197955882 +0000 UTC m=+676.246334603" Jul 10 00:37:46.375864 ip-10-0-130-146 kubenswrapper[2570]: I0710 00:37:46.375833 2570 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-6b8698786d-qv9lq" Jul 10 00:37:46.376347 ip-10-0-130-146 kubenswrapper[2570]: I0710 00:37:46.375899 2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-6b8698786d-qv9lq" Jul 10 00:37:46.380346 ip-10-0-130-146 kubenswrapper[2570]: I0710 00:37:46.380323 2570 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-6b8698786d-qv9lq" Jul 10 00:37:46.839118 ip-10-0-130-146 kubenswrapper[2570]: I0710 00:37:46.836251 2570 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73rs9vd"] Jul 10 00:37:46.841528 ip-10-0-130-146 kubenswrapper[2570]: I0710 00:37:46.841507 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73rs9vd" Jul 10 00:37:46.844615 ip-10-0-130-146 kubenswrapper[2570]: I0710 00:37:46.844595 2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"default-dockercfg-kmd5v\"" Jul 10 00:37:46.844708 ip-10-0-130-146 kubenswrapper[2570]: I0710 00:37:46.844594 2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"openshift-service-ca.crt\"" Jul 10 00:37:46.845004 ip-10-0-130-146 kubenswrapper[2570]: I0710 00:37:46.844986 2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kube-root-ca.crt\"" Jul 10 00:37:46.855651 ip-10-0-130-146 kubenswrapper[2570]: I0710 00:37:46.855629 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73rs9vd"] Jul 10 00:37:46.959185 ip-10-0-130-146 kubenswrapper[2570]: I0710 00:37:46.959154 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/48dc2b69-a1c1-46b4-9f79-02abaf29e411-bundle\") pod \"5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73rs9vd\" (UID: \"48dc2b69-a1c1-46b4-9f79-02abaf29e411\") " pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73rs9vd" Jul 10 00:37:46.959355 ip-10-0-130-146 kubenswrapper[2570]: I0710 00:37:46.959216 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sk8gd\" (UniqueName: \"kubernetes.io/projected/48dc2b69-a1c1-46b4-9f79-02abaf29e411-kube-api-access-sk8gd\") pod \"5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73rs9vd\" (UID: \"48dc2b69-a1c1-46b4-9f79-02abaf29e411\") " pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73rs9vd" Jul 10 00:37:46.959355 ip-10-0-130-146 kubenswrapper[2570]: I0710 00:37:46.959300 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/48dc2b69-a1c1-46b4-9f79-02abaf29e411-util\") pod \"5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73rs9vd\" (UID: \"48dc2b69-a1c1-46b4-9f79-02abaf29e411\") " pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73rs9vd" Jul 10 00:37:47.060671 ip-10-0-130-146 kubenswrapper[2570]: I0710 00:37:47.060637 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sk8gd\" (UniqueName: \"kubernetes.io/projected/48dc2b69-a1c1-46b4-9f79-02abaf29e411-kube-api-access-sk8gd\") pod \"5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73rs9vd\" (UID: \"48dc2b69-a1c1-46b4-9f79-02abaf29e411\") " pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73rs9vd" Jul 10 00:37:47.060817 ip-10-0-130-146 kubenswrapper[2570]: I0710 00:37:47.060682 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/48dc2b69-a1c1-46b4-9f79-02abaf29e411-util\") pod \"5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73rs9vd\" (UID: \"48dc2b69-a1c1-46b4-9f79-02abaf29e411\") " pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73rs9vd" Jul 10 00:37:47.060817 ip-10-0-130-146 kubenswrapper[2570]: I0710 00:37:47.060722 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/48dc2b69-a1c1-46b4-9f79-02abaf29e411-bundle\") pod \"5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73rs9vd\" (UID: \"48dc2b69-a1c1-46b4-9f79-02abaf29e411\") " pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73rs9vd" Jul 10 00:37:47.061023 ip-10-0-130-146 kubenswrapper[2570]: I0710 00:37:47.061008 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/48dc2b69-a1c1-46b4-9f79-02abaf29e411-bundle\") pod \"5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73rs9vd\" (UID: \"48dc2b69-a1c1-46b4-9f79-02abaf29e411\") " pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73rs9vd" Jul 10 00:37:47.061091 ip-10-0-130-146 kubenswrapper[2570]: I0710 00:37:47.061076 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/48dc2b69-a1c1-46b4-9f79-02abaf29e411-util\") pod \"5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73rs9vd\" (UID: \"48dc2b69-a1c1-46b4-9f79-02abaf29e411\") " pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73rs9vd" Jul 10 00:37:47.068724 ip-10-0-130-146 kubenswrapper[2570]: I0710 00:37:47.068705 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-sk8gd\" (UniqueName: \"kubernetes.io/projected/48dc2b69-a1c1-46b4-9f79-02abaf29e411-kube-api-access-sk8gd\") pod \"5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73rs9vd\" (UID: \"48dc2b69-a1c1-46b4-9f79-02abaf29e411\") " pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73rs9vd" Jul 10 00:37:47.151661 ip-10-0-130-146 kubenswrapper[2570]: I0710 00:37:47.151631 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73rs9vd" Jul 10 00:37:47.231895 ip-10-0-130-146 kubenswrapper[2570]: I0710 00:37:47.231865 2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-6b8698786d-qv9lq" Jul 10 00:37:47.233969 ip-10-0-130-146 kubenswrapper[2570]: I0710 00:37:47.233946 2570 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1pfl48"] Jul 10 00:37:47.238481 ip-10-0-130-146 kubenswrapper[2570]: I0710 00:37:47.238373 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1pfl48" Jul 10 00:37:47.248545 ip-10-0-130-146 kubenswrapper[2570]: I0710 00:37:47.248294 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1pfl48"] Jul 10 00:37:47.287235 ip-10-0-130-146 kubenswrapper[2570]: I0710 00:37:47.287205 2570 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-6dc8d7758c-tl2pk"] Jul 10 00:37:47.291149 ip-10-0-130-146 kubenswrapper[2570]: I0710 00:37:47.290978 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73rs9vd"] Jul 10 00:37:47.296597 ip-10-0-130-146 kubenswrapper[2570]: W0710 00:37:47.295382 2570 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod48dc2b69_a1c1_46b4_9f79_02abaf29e411.slice/crio-9c6adf9e7cb034a60ccb9f6a4e6238ea151e0779ee871ebdc5efab3b48876530 WatchSource:0}: Error finding container 9c6adf9e7cb034a60ccb9f6a4e6238ea151e0779ee871ebdc5efab3b48876530: Status 404 returned error can't find the container with id 9c6adf9e7cb034a60ccb9f6a4e6238ea151e0779ee871ebdc5efab3b48876530 Jul 10 00:37:47.363994 ip-10-0-130-146 kubenswrapper[2570]: I0710 00:37:47.363968 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/126a17da-6d3b-4880-b310-d0cf7612be39-bundle\") pod \"9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1pfl48\" (UID: \"126a17da-6d3b-4880-b310-d0cf7612be39\") " pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1pfl48" Jul 10 00:37:47.364135 ip-10-0-130-146 kubenswrapper[2570]: I0710 00:37:47.364003 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hq75d\" (UniqueName: \"kubernetes.io/projected/126a17da-6d3b-4880-b310-d0cf7612be39-kube-api-access-hq75d\") pod \"9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1pfl48\" (UID: \"126a17da-6d3b-4880-b310-d0cf7612be39\") " pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1pfl48" Jul 10 00:37:47.364228 ip-10-0-130-146 kubenswrapper[2570]: I0710 00:37:47.364207 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/126a17da-6d3b-4880-b310-d0cf7612be39-util\") pod \"9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1pfl48\" (UID: \"126a17da-6d3b-4880-b310-d0cf7612be39\") " pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1pfl48" Jul 10 00:37:47.465689 ip-10-0-130-146 kubenswrapper[2570]: I0710 00:37:47.465647 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/126a17da-6d3b-4880-b310-d0cf7612be39-util\") pod \"9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1pfl48\" (UID: \"126a17da-6d3b-4880-b310-d0cf7612be39\") " pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1pfl48" Jul 10 00:37:47.466019 ip-10-0-130-146 kubenswrapper[2570]: I0710 00:37:47.465743 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/126a17da-6d3b-4880-b310-d0cf7612be39-bundle\") pod \"9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1pfl48\" (UID: \"126a17da-6d3b-4880-b310-d0cf7612be39\") " pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1pfl48" Jul 10 00:37:47.466019 ip-10-0-130-146 kubenswrapper[2570]: I0710 00:37:47.465773 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hq75d\" (UniqueName: \"kubernetes.io/projected/126a17da-6d3b-4880-b310-d0cf7612be39-kube-api-access-hq75d\") pod \"9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1pfl48\" (UID: \"126a17da-6d3b-4880-b310-d0cf7612be39\") " pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1pfl48" Jul 10 00:37:47.466019 ip-10-0-130-146 kubenswrapper[2570]: I0710 00:37:47.466009 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/126a17da-6d3b-4880-b310-d0cf7612be39-util\") pod \"9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1pfl48\" (UID: \"126a17da-6d3b-4880-b310-d0cf7612be39\") " pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1pfl48" Jul 10 00:37:47.466181 ip-10-0-130-146 kubenswrapper[2570]: I0710 00:37:47.466026 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/126a17da-6d3b-4880-b310-d0cf7612be39-bundle\") pod \"9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1pfl48\" (UID: \"126a17da-6d3b-4880-b310-d0cf7612be39\") " pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1pfl48" Jul 10 00:37:47.474402 ip-10-0-130-146 kubenswrapper[2570]: I0710 00:37:47.474381 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-hq75d\" (UniqueName: \"kubernetes.io/projected/126a17da-6d3b-4880-b310-d0cf7612be39-kube-api-access-hq75d\") pod \"9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1pfl48\" (UID: \"126a17da-6d3b-4880-b310-d0cf7612be39\") " pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1pfl48" Jul 10 00:37:47.551720 ip-10-0-130-146 kubenswrapper[2570]: I0710 00:37:47.551682 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1pfl48" Jul 10 00:37:47.638026 ip-10-0-130-146 kubenswrapper[2570]: I0710 00:37:47.638000 2570 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759k68sb"] Jul 10 00:37:47.641786 ip-10-0-130-146 kubenswrapper[2570]: I0710 00:37:47.641767 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759k68sb" Jul 10 00:37:47.650569 ip-10-0-130-146 kubenswrapper[2570]: I0710 00:37:47.650541 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759k68sb"] Jul 10 00:37:47.682278 ip-10-0-130-146 kubenswrapper[2570]: I0710 00:37:47.682254 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1pfl48"] Jul 10 00:37:47.683948 ip-10-0-130-146 kubenswrapper[2570]: W0710 00:37:47.683925 2570 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod126a17da_6d3b_4880_b310_d0cf7612be39.slice/crio-af0b921df4a94800182423268fd1b2a8da6a162bf14498551cf9785694c08ce3 WatchSource:0}: Error finding container af0b921df4a94800182423268fd1b2a8da6a162bf14498551cf9785694c08ce3: Status 404 returned error can't find the container with id af0b921df4a94800182423268fd1b2a8da6a162bf14498551cf9785694c08ce3 Jul 10 00:37:47.768461 ip-10-0-130-146 kubenswrapper[2570]: I0710 00:37:47.768427 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/a062b873-3bc8-4dc1-96e7-63353d46e0b5-bundle\") pod \"0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759k68sb\" (UID: \"a062b873-3bc8-4dc1-96e7-63353d46e0b5\") " pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759k68sb" Jul 10 00:37:47.768625 ip-10-0-130-146 kubenswrapper[2570]: I0710 00:37:47.768470 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/a062b873-3bc8-4dc1-96e7-63353d46e0b5-util\") pod \"0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759k68sb\" (UID: \"a062b873-3bc8-4dc1-96e7-63353d46e0b5\") " pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759k68sb" Jul 10 00:37:47.768625 ip-10-0-130-146 kubenswrapper[2570]: I0710 00:37:47.768564 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-c5k7g\" (UniqueName: \"kubernetes.io/projected/a062b873-3bc8-4dc1-96e7-63353d46e0b5-kube-api-access-c5k7g\") pod \"0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759k68sb\" (UID: \"a062b873-3bc8-4dc1-96e7-63353d46e0b5\") " pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759k68sb" Jul 10 00:37:47.869789 ip-10-0-130-146 kubenswrapper[2570]: I0710 00:37:47.869759 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/a062b873-3bc8-4dc1-96e7-63353d46e0b5-bundle\") pod \"0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759k68sb\" (UID: \"a062b873-3bc8-4dc1-96e7-63353d46e0b5\") " pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759k68sb" Jul 10 00:37:47.869916 ip-10-0-130-146 kubenswrapper[2570]: I0710 00:37:47.869793 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/a062b873-3bc8-4dc1-96e7-63353d46e0b5-util\") pod \"0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759k68sb\" (UID: \"a062b873-3bc8-4dc1-96e7-63353d46e0b5\") " pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759k68sb" Jul 10 00:37:47.869916 ip-10-0-130-146 kubenswrapper[2570]: I0710 00:37:47.869845 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c5k7g\" (UniqueName: \"kubernetes.io/projected/a062b873-3bc8-4dc1-96e7-63353d46e0b5-kube-api-access-c5k7g\") pod \"0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759k68sb\" (UID: \"a062b873-3bc8-4dc1-96e7-63353d46e0b5\") " pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759k68sb" Jul 10 00:37:47.870155 ip-10-0-130-146 kubenswrapper[2570]: I0710 00:37:47.870135 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/a062b873-3bc8-4dc1-96e7-63353d46e0b5-bundle\") pod \"0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759k68sb\" (UID: \"a062b873-3bc8-4dc1-96e7-63353d46e0b5\") " pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759k68sb" Jul 10 00:37:47.870222 ip-10-0-130-146 kubenswrapper[2570]: I0710 00:37:47.870205 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/a062b873-3bc8-4dc1-96e7-63353d46e0b5-util\") pod \"0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759k68sb\" (UID: \"a062b873-3bc8-4dc1-96e7-63353d46e0b5\") " pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759k68sb" Jul 10 00:37:47.878071 ip-10-0-130-146 kubenswrapper[2570]: I0710 00:37:47.878033 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-c5k7g\" (UniqueName: \"kubernetes.io/projected/a062b873-3bc8-4dc1-96e7-63353d46e0b5-kube-api-access-c5k7g\") pod \"0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759k68sb\" (UID: \"a062b873-3bc8-4dc1-96e7-63353d46e0b5\") " pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759k68sb" Jul 10 00:37:47.953505 ip-10-0-130-146 kubenswrapper[2570]: I0710 00:37:47.953442 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759k68sb" Jul 10 00:37:48.083930 ip-10-0-130-146 kubenswrapper[2570]: I0710 00:37:48.083912 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759k68sb"] Jul 10 00:37:48.086216 ip-10-0-130-146 kubenswrapper[2570]: W0710 00:37:48.086185 2570 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda062b873_3bc8_4dc1_96e7_63353d46e0b5.slice/crio-45c5c596560dd51b565e5b9836fadafbdd65d1ff6fe364fd253ad6da9d7349de WatchSource:0}: Error finding container 45c5c596560dd51b565e5b9836fadafbdd65d1ff6fe364fd253ad6da9d7349de: Status 404 returned error can't find the container with id 45c5c596560dd51b565e5b9836fadafbdd65d1ff6fe364fd253ad6da9d7349de Jul 10 00:37:48.232527 ip-10-0-130-146 kubenswrapper[2570]: I0710 00:37:48.232475 2570 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0hr2tc"] Jul 10 00:37:48.234676 ip-10-0-130-146 kubenswrapper[2570]: I0710 00:37:48.234653 2570 generic.go:356] "Generic (PLEG): container finished" podID="a062b873-3bc8-4dc1-96e7-63353d46e0b5" containerID="88010d83caf46848444b8f77ff47454881f1fcdd50acbb9929e9c0e5bf1764b2" exitCode=0 Jul 10 00:37:48.236128 ip-10-0-130-146 kubenswrapper[2570]: I0710 00:37:48.236106 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759k68sb" event={"ID":"a062b873-3bc8-4dc1-96e7-63353d46e0b5","Type":"ContainerDied","Data":"88010d83caf46848444b8f77ff47454881f1fcdd50acbb9929e9c0e5bf1764b2"} Jul 10 00:37:48.236216 ip-10-0-130-146 kubenswrapper[2570]: I0710 00:37:48.236139 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759k68sb" event={"ID":"a062b873-3bc8-4dc1-96e7-63353d46e0b5","Type":"ContainerStarted","Data":"45c5c596560dd51b565e5b9836fadafbdd65d1ff6fe364fd253ad6da9d7349de"} Jul 10 00:37:48.236271 ip-10-0-130-146 kubenswrapper[2570]: I0710 00:37:48.236225 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0hr2tc" Jul 10 00:37:48.236271 ip-10-0-130-146 kubenswrapper[2570]: I0710 00:37:48.236244 2570 generic.go:356] "Generic (PLEG): container finished" podID="48dc2b69-a1c1-46b4-9f79-02abaf29e411" containerID="7ad7c81cdcdeae59f05b5bce708c359fb6aac58d273a24776c07bd4b1f5315e4" exitCode=0 Jul 10 00:37:48.236349 ip-10-0-130-146 kubenswrapper[2570]: I0710 00:37:48.236271 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73rs9vd" event={"ID":"48dc2b69-a1c1-46b4-9f79-02abaf29e411","Type":"ContainerDied","Data":"7ad7c81cdcdeae59f05b5bce708c359fb6aac58d273a24776c07bd4b1f5315e4"} Jul 10 00:37:48.236349 ip-10-0-130-146 kubenswrapper[2570]: I0710 00:37:48.236308 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73rs9vd" event={"ID":"48dc2b69-a1c1-46b4-9f79-02abaf29e411","Type":"ContainerStarted","Data":"9c6adf9e7cb034a60ccb9f6a4e6238ea151e0779ee871ebdc5efab3b48876530"} Jul 10 00:37:48.237964 ip-10-0-130-146 kubenswrapper[2570]: I0710 00:37:48.237934 2570 generic.go:356] "Generic (PLEG): container finished" podID="126a17da-6d3b-4880-b310-d0cf7612be39" containerID="c16158652bf52763e21b8c38a95953ead90960038b5b29883747424ed4d529d5" exitCode=0 Jul 10 00:37:48.238079 ip-10-0-130-146 kubenswrapper[2570]: I0710 00:37:48.237999 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1pfl48" event={"ID":"126a17da-6d3b-4880-b310-d0cf7612be39","Type":"ContainerDied","Data":"c16158652bf52763e21b8c38a95953ead90960038b5b29883747424ed4d529d5"} Jul 10 00:37:48.238079 ip-10-0-130-146 kubenswrapper[2570]: I0710 00:37:48.238030 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1pfl48" event={"ID":"126a17da-6d3b-4880-b310-d0cf7612be39","Type":"ContainerStarted","Data":"af0b921df4a94800182423268fd1b2a8da6a162bf14498551cf9785694c08ce3"} Jul 10 00:37:48.247809 ip-10-0-130-146 kubenswrapper[2570]: I0710 00:37:48.247789 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0hr2tc"] Jul 10 00:37:48.374397 ip-10-0-130-146 kubenswrapper[2570]: I0710 00:37:48.374360 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/b407e13a-88fb-43a1-94fc-ab40c580d6c2-bundle\") pod \"19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0hr2tc\" (UID: \"b407e13a-88fb-43a1-94fc-ab40c580d6c2\") " pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0hr2tc" Jul 10 00:37:48.374558 ip-10-0-130-146 kubenswrapper[2570]: I0710 00:37:48.374481 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/b407e13a-88fb-43a1-94fc-ab40c580d6c2-util\") pod \"19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0hr2tc\" (UID: \"b407e13a-88fb-43a1-94fc-ab40c580d6c2\") " pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0hr2tc" Jul 10 00:37:48.374629 ip-10-0-130-146 kubenswrapper[2570]: I0710 00:37:48.374594 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jdnc5\" (UniqueName: \"kubernetes.io/projected/b407e13a-88fb-43a1-94fc-ab40c580d6c2-kube-api-access-jdnc5\") pod \"19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0hr2tc\" (UID: \"b407e13a-88fb-43a1-94fc-ab40c580d6c2\") " pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0hr2tc" Jul 10 00:37:48.476098 ip-10-0-130-146 kubenswrapper[2570]: I0710 00:37:48.476068 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/b407e13a-88fb-43a1-94fc-ab40c580d6c2-bundle\") pod \"19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0hr2tc\" (UID: \"b407e13a-88fb-43a1-94fc-ab40c580d6c2\") " pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0hr2tc" Jul 10 00:37:48.476456 ip-10-0-130-146 kubenswrapper[2570]: I0710 00:37:48.476107 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/b407e13a-88fb-43a1-94fc-ab40c580d6c2-util\") pod \"19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0hr2tc\" (UID: \"b407e13a-88fb-43a1-94fc-ab40c580d6c2\") " pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0hr2tc" Jul 10 00:37:48.476456 ip-10-0-130-146 kubenswrapper[2570]: I0710 00:37:48.476155 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jdnc5\" (UniqueName: \"kubernetes.io/projected/b407e13a-88fb-43a1-94fc-ab40c580d6c2-kube-api-access-jdnc5\") pod \"19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0hr2tc\" (UID: \"b407e13a-88fb-43a1-94fc-ab40c580d6c2\") " pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0hr2tc" Jul 10 00:37:48.476456 ip-10-0-130-146 kubenswrapper[2570]: I0710 00:37:48.476418 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/b407e13a-88fb-43a1-94fc-ab40c580d6c2-bundle\") pod \"19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0hr2tc\" (UID: \"b407e13a-88fb-43a1-94fc-ab40c580d6c2\") " pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0hr2tc" Jul 10 00:37:48.476456 ip-10-0-130-146 kubenswrapper[2570]: I0710 00:37:48.476449 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/b407e13a-88fb-43a1-94fc-ab40c580d6c2-util\") pod \"19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0hr2tc\" (UID: \"b407e13a-88fb-43a1-94fc-ab40c580d6c2\") " pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0hr2tc" Jul 10 00:37:48.485172 ip-10-0-130-146 kubenswrapper[2570]: I0710 00:37:48.485108 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jdnc5\" (UniqueName: \"kubernetes.io/projected/b407e13a-88fb-43a1-94fc-ab40c580d6c2-kube-api-access-jdnc5\") pod \"19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0hr2tc\" (UID: \"b407e13a-88fb-43a1-94fc-ab40c580d6c2\") " pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0hr2tc" Jul 10 00:37:48.554668 ip-10-0-130-146 kubenswrapper[2570]: I0710 00:37:48.554643 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0hr2tc" Jul 10 00:37:48.694200 ip-10-0-130-146 kubenswrapper[2570]: I0710 00:37:48.694174 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0hr2tc"] Jul 10 00:37:48.695521 ip-10-0-130-146 kubenswrapper[2570]: W0710 00:37:48.695496 2570 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb407e13a_88fb_43a1_94fc_ab40c580d6c2.slice/crio-bda9b6af3276bc8f70d93856fe2fa356294e910cd56ece6f92da4cab3e6c7294 WatchSource:0}: Error finding container bda9b6af3276bc8f70d93856fe2fa356294e910cd56ece6f92da4cab3e6c7294: Status 404 returned error can't find the container with id bda9b6af3276bc8f70d93856fe2fa356294e910cd56ece6f92da4cab3e6c7294 Jul 10 00:37:49.249131 ip-10-0-130-146 kubenswrapper[2570]: I0710 00:37:49.249093 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759k68sb" event={"ID":"a062b873-3bc8-4dc1-96e7-63353d46e0b5","Type":"ContainerStarted","Data":"10a78ede3ec2563024e9a48b71c9982321a6e447d4cf2e4ed476cbe9127fa111"} Jul 10 00:37:49.250614 ip-10-0-130-146 kubenswrapper[2570]: I0710 00:37:49.250587 2570 generic.go:356] "Generic (PLEG): container finished" podID="b407e13a-88fb-43a1-94fc-ab40c580d6c2" containerID="9984781d8dcdc76edf6994cc131324afd75d01da94fc4ce2a73dd13ba59aadf0" exitCode=0 Jul 10 00:37:49.250777 ip-10-0-130-146 kubenswrapper[2570]: I0710 00:37:49.250668 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0hr2tc" event={"ID":"b407e13a-88fb-43a1-94fc-ab40c580d6c2","Type":"ContainerDied","Data":"9984781d8dcdc76edf6994cc131324afd75d01da94fc4ce2a73dd13ba59aadf0"} Jul 10 00:37:49.250777 ip-10-0-130-146 kubenswrapper[2570]: I0710 00:37:49.250732 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0hr2tc" event={"ID":"b407e13a-88fb-43a1-94fc-ab40c580d6c2","Type":"ContainerStarted","Data":"bda9b6af3276bc8f70d93856fe2fa356294e910cd56ece6f92da4cab3e6c7294"} Jul 10 00:37:50.257730 ip-10-0-130-146 kubenswrapper[2570]: I0710 00:37:50.257638 2570 generic.go:356] "Generic (PLEG): container finished" podID="126a17da-6d3b-4880-b310-d0cf7612be39" containerID="722919a8547eb2f594a940a2833ef0271ffc8bbd2355f8d6904bd1d1d7de383f" exitCode=0 Jul 10 00:37:50.258140 ip-10-0-130-146 kubenswrapper[2570]: I0710 00:37:50.257726 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1pfl48" event={"ID":"126a17da-6d3b-4880-b310-d0cf7612be39","Type":"ContainerDied","Data":"722919a8547eb2f594a940a2833ef0271ffc8bbd2355f8d6904bd1d1d7de383f"} Jul 10 00:37:50.259710 ip-10-0-130-146 kubenswrapper[2570]: I0710 00:37:50.259683 2570 generic.go:356] "Generic (PLEG): container finished" podID="b407e13a-88fb-43a1-94fc-ab40c580d6c2" containerID="3810bca551970af003ce9b397a898e7bcb256e1a28a36434f8ea032012f6ad15" exitCode=0 Jul 10 00:37:50.259792 ip-10-0-130-146 kubenswrapper[2570]: I0710 00:37:50.259764 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0hr2tc" event={"ID":"b407e13a-88fb-43a1-94fc-ab40c580d6c2","Type":"ContainerDied","Data":"3810bca551970af003ce9b397a898e7bcb256e1a28a36434f8ea032012f6ad15"} Jul 10 00:37:50.261692 ip-10-0-130-146 kubenswrapper[2570]: I0710 00:37:50.261672 2570 generic.go:356] "Generic (PLEG): container finished" podID="a062b873-3bc8-4dc1-96e7-63353d46e0b5" containerID="10a78ede3ec2563024e9a48b71c9982321a6e447d4cf2e4ed476cbe9127fa111" exitCode=0 Jul 10 00:37:50.261791 ip-10-0-130-146 kubenswrapper[2570]: I0710 00:37:50.261692 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759k68sb" event={"ID":"a062b873-3bc8-4dc1-96e7-63353d46e0b5","Type":"ContainerDied","Data":"10a78ede3ec2563024e9a48b71c9982321a6e447d4cf2e4ed476cbe9127fa111"} Jul 10 00:37:50.263527 ip-10-0-130-146 kubenswrapper[2570]: I0710 00:37:50.263494 2570 generic.go:356] "Generic (PLEG): container finished" podID="48dc2b69-a1c1-46b4-9f79-02abaf29e411" containerID="c027b82c52ce6011112a2a1ed99ae21fb7ba2733e0c8489311b1b3e9c1d8ad3b" exitCode=0 Jul 10 00:37:50.263616 ip-10-0-130-146 kubenswrapper[2570]: I0710 00:37:50.263526 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73rs9vd" event={"ID":"48dc2b69-a1c1-46b4-9f79-02abaf29e411","Type":"ContainerDied","Data":"c027b82c52ce6011112a2a1ed99ae21fb7ba2733e0c8489311b1b3e9c1d8ad3b"} Jul 10 00:37:51.270837 ip-10-0-130-146 kubenswrapper[2570]: I0710 00:37:51.270801 2570 generic.go:356] "Generic (PLEG): container finished" podID="a062b873-3bc8-4dc1-96e7-63353d46e0b5" containerID="93b343b99df4feba050e21d7c504a1835ebd96eff6673047b59e9558002e41b4" exitCode=0 Jul 10 00:37:51.271291 ip-10-0-130-146 kubenswrapper[2570]: I0710 00:37:51.270882 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759k68sb" event={"ID":"a062b873-3bc8-4dc1-96e7-63353d46e0b5","Type":"ContainerDied","Data":"93b343b99df4feba050e21d7c504a1835ebd96eff6673047b59e9558002e41b4"} Jul 10 00:37:51.272832 ip-10-0-130-146 kubenswrapper[2570]: I0710 00:37:51.272810 2570 generic.go:356] "Generic (PLEG): container finished" podID="48dc2b69-a1c1-46b4-9f79-02abaf29e411" containerID="c41fc1e3a522c928d95267501fe925f4ecda133dee8df1ac0199771582ecc165" exitCode=0 Jul 10 00:37:51.272982 ip-10-0-130-146 kubenswrapper[2570]: I0710 00:37:51.272884 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73rs9vd" event={"ID":"48dc2b69-a1c1-46b4-9f79-02abaf29e411","Type":"ContainerDied","Data":"c41fc1e3a522c928d95267501fe925f4ecda133dee8df1ac0199771582ecc165"} Jul 10 00:37:51.274832 ip-10-0-130-146 kubenswrapper[2570]: I0710 00:37:51.274812 2570 generic.go:356] "Generic (PLEG): container finished" podID="126a17da-6d3b-4880-b310-d0cf7612be39" containerID="49afbba75828958851cabfd35138743f269cecc9f3c8317067bcb28970f50da6" exitCode=0 Jul 10 00:37:51.274919 ip-10-0-130-146 kubenswrapper[2570]: I0710 00:37:51.274880 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1pfl48" event={"ID":"126a17da-6d3b-4880-b310-d0cf7612be39","Type":"ContainerDied","Data":"49afbba75828958851cabfd35138743f269cecc9f3c8317067bcb28970f50da6"} Jul 10 00:37:51.276752 ip-10-0-130-146 kubenswrapper[2570]: I0710 00:37:51.276732 2570 generic.go:356] "Generic (PLEG): container finished" podID="b407e13a-88fb-43a1-94fc-ab40c580d6c2" containerID="236d3ac81663576e0dfe8a666adb4d8637bdc7ab5959a6f569336fe17e7e1a98" exitCode=0 Jul 10 00:37:51.276824 ip-10-0-130-146 kubenswrapper[2570]: I0710 00:37:51.276753 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0hr2tc" event={"ID":"b407e13a-88fb-43a1-94fc-ab40c580d6c2","Type":"ContainerDied","Data":"236d3ac81663576e0dfe8a666adb4d8637bdc7ab5959a6f569336fe17e7e1a98"} Jul 10 00:37:52.413595 ip-10-0-130-146 kubenswrapper[2570]: I0710 00:37:52.413563 2570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0hr2tc" Jul 10 00:37:52.468295 ip-10-0-130-146 kubenswrapper[2570]: I0710 00:37:52.468268 2570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759k68sb" Jul 10 00:37:52.492512 ip-10-0-130-146 kubenswrapper[2570]: I0710 00:37:52.492487 2570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1pfl48" Jul 10 00:37:52.496182 ip-10-0-130-146 kubenswrapper[2570]: I0710 00:37:52.496158 2570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73rs9vd" Jul 10 00:37:52.514449 ip-10-0-130-146 kubenswrapper[2570]: I0710 00:37:52.514421 2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/b407e13a-88fb-43a1-94fc-ab40c580d6c2-util\") pod \"b407e13a-88fb-43a1-94fc-ab40c580d6c2\" (UID: \"b407e13a-88fb-43a1-94fc-ab40c580d6c2\") " Jul 10 00:37:52.514588 ip-10-0-130-146 kubenswrapper[2570]: I0710 00:37:52.514538 2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-jdnc5\" (UniqueName: \"kubernetes.io/projected/b407e13a-88fb-43a1-94fc-ab40c580d6c2-kube-api-access-jdnc5\") pod \"b407e13a-88fb-43a1-94fc-ab40c580d6c2\" (UID: \"b407e13a-88fb-43a1-94fc-ab40c580d6c2\") " Jul 10 00:37:52.514588 ip-10-0-130-146 kubenswrapper[2570]: I0710 00:37:52.514581 2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/b407e13a-88fb-43a1-94fc-ab40c580d6c2-bundle\") pod \"b407e13a-88fb-43a1-94fc-ab40c580d6c2\" (UID: \"b407e13a-88fb-43a1-94fc-ab40c580d6c2\") " Jul 10 00:37:52.515435 ip-10-0-130-146 kubenswrapper[2570]: I0710 00:37:52.515395 2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b407e13a-88fb-43a1-94fc-ab40c580d6c2-bundle" (OuterVolumeSpecName: "bundle") pod "b407e13a-88fb-43a1-94fc-ab40c580d6c2" (UID: "b407e13a-88fb-43a1-94fc-ab40c580d6c2"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 10 00:37:52.516821 ip-10-0-130-146 kubenswrapper[2570]: I0710 00:37:52.516798 2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b407e13a-88fb-43a1-94fc-ab40c580d6c2-kube-api-access-jdnc5" (OuterVolumeSpecName: "kube-api-access-jdnc5") pod "b407e13a-88fb-43a1-94fc-ab40c580d6c2" (UID: "b407e13a-88fb-43a1-94fc-ab40c580d6c2"). InnerVolumeSpecName "kube-api-access-jdnc5". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 10 00:37:52.520854 ip-10-0-130-146 kubenswrapper[2570]: I0710 00:37:52.520834 2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b407e13a-88fb-43a1-94fc-ab40c580d6c2-util" (OuterVolumeSpecName: "util") pod "b407e13a-88fb-43a1-94fc-ab40c580d6c2" (UID: "b407e13a-88fb-43a1-94fc-ab40c580d6c2"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 10 00:37:52.616100 ip-10-0-130-146 kubenswrapper[2570]: I0710 00:37:52.616075 2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/48dc2b69-a1c1-46b4-9f79-02abaf29e411-bundle\") pod \"48dc2b69-a1c1-46b4-9f79-02abaf29e411\" (UID: \"48dc2b69-a1c1-46b4-9f79-02abaf29e411\") " Jul 10 00:37:52.616273 ip-10-0-130-146 kubenswrapper[2570]: I0710 00:37:52.616129 2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/a062b873-3bc8-4dc1-96e7-63353d46e0b5-util\") pod \"a062b873-3bc8-4dc1-96e7-63353d46e0b5\" (UID: \"a062b873-3bc8-4dc1-96e7-63353d46e0b5\") " Jul 10 00:37:52.616273 ip-10-0-130-146 kubenswrapper[2570]: I0710 00:37:52.616150 2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-sk8gd\" (UniqueName: \"kubernetes.io/projected/48dc2b69-a1c1-46b4-9f79-02abaf29e411-kube-api-access-sk8gd\") pod \"48dc2b69-a1c1-46b4-9f79-02abaf29e411\" (UID: \"48dc2b69-a1c1-46b4-9f79-02abaf29e411\") " Jul 10 00:37:52.616273 ip-10-0-130-146 kubenswrapper[2570]: I0710 00:37:52.616187 2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/48dc2b69-a1c1-46b4-9f79-02abaf29e411-util\") pod \"48dc2b69-a1c1-46b4-9f79-02abaf29e411\" (UID: \"48dc2b69-a1c1-46b4-9f79-02abaf29e411\") " Jul 10 00:37:52.616273 ip-10-0-130-146 kubenswrapper[2570]: I0710 00:37:52.616208 2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/126a17da-6d3b-4880-b310-d0cf7612be39-bundle\") pod \"126a17da-6d3b-4880-b310-d0cf7612be39\" (UID: \"126a17da-6d3b-4880-b310-d0cf7612be39\") " Jul 10 00:37:52.616494 ip-10-0-130-146 kubenswrapper[2570]: I0710 00:37:52.616402 2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/126a17da-6d3b-4880-b310-d0cf7612be39-util\") pod \"126a17da-6d3b-4880-b310-d0cf7612be39\" (UID: \"126a17da-6d3b-4880-b310-d0cf7612be39\") " Jul 10 00:37:52.616494 ip-10-0-130-146 kubenswrapper[2570]: I0710 00:37:52.616456 2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/a062b873-3bc8-4dc1-96e7-63353d46e0b5-bundle\") pod \"a062b873-3bc8-4dc1-96e7-63353d46e0b5\" (UID: \"a062b873-3bc8-4dc1-96e7-63353d46e0b5\") " Jul 10 00:37:52.616494 ip-10-0-130-146 kubenswrapper[2570]: I0710 00:37:52.616487 2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-c5k7g\" (UniqueName: \"kubernetes.io/projected/a062b873-3bc8-4dc1-96e7-63353d46e0b5-kube-api-access-c5k7g\") pod \"a062b873-3bc8-4dc1-96e7-63353d46e0b5\" (UID: \"a062b873-3bc8-4dc1-96e7-63353d46e0b5\") " Jul 10 00:37:52.616641 ip-10-0-130-146 kubenswrapper[2570]: I0710 00:37:52.616516 2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-hq75d\" (UniqueName: \"kubernetes.io/projected/126a17da-6d3b-4880-b310-d0cf7612be39-kube-api-access-hq75d\") pod \"126a17da-6d3b-4880-b310-d0cf7612be39\" (UID: \"126a17da-6d3b-4880-b310-d0cf7612be39\") " Jul 10 00:37:52.616987 ip-10-0-130-146 kubenswrapper[2570]: I0710 00:37:52.616820 2570 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jdnc5\" (UniqueName: \"kubernetes.io/projected/b407e13a-88fb-43a1-94fc-ab40c580d6c2-kube-api-access-jdnc5\") on node \"ip-10-0-130-146.ec2.internal\" DevicePath \"\"" Jul 10 00:37:52.616987 ip-10-0-130-146 kubenswrapper[2570]: I0710 00:37:52.616845 2570 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/b407e13a-88fb-43a1-94fc-ab40c580d6c2-bundle\") on node \"ip-10-0-130-146.ec2.internal\" DevicePath \"\"" Jul 10 00:37:52.616987 ip-10-0-130-146 kubenswrapper[2570]: I0710 00:37:52.616860 2570 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/b407e13a-88fb-43a1-94fc-ab40c580d6c2-util\") on node \"ip-10-0-130-146.ec2.internal\" DevicePath \"\"" Jul 10 00:37:52.616987 ip-10-0-130-146 kubenswrapper[2570]: I0710 00:37:52.616871 2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a062b873-3bc8-4dc1-96e7-63353d46e0b5-bundle" (OuterVolumeSpecName: "bundle") pod "a062b873-3bc8-4dc1-96e7-63353d46e0b5" (UID: "a062b873-3bc8-4dc1-96e7-63353d46e0b5"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 10 00:37:52.616987 ip-10-0-130-146 kubenswrapper[2570]: I0710 00:37:52.616929 2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/48dc2b69-a1c1-46b4-9f79-02abaf29e411-bundle" (OuterVolumeSpecName: "bundle") pod "48dc2b69-a1c1-46b4-9f79-02abaf29e411" (UID: "48dc2b69-a1c1-46b4-9f79-02abaf29e411"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 10 00:37:52.617369 ip-10-0-130-146 kubenswrapper[2570]: I0710 00:37:52.617231 2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/126a17da-6d3b-4880-b310-d0cf7612be39-bundle" (OuterVolumeSpecName: "bundle") pod "126a17da-6d3b-4880-b310-d0cf7612be39" (UID: "126a17da-6d3b-4880-b310-d0cf7612be39"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 10 00:37:52.618732 ip-10-0-130-146 kubenswrapper[2570]: I0710 00:37:52.618705 2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/48dc2b69-a1c1-46b4-9f79-02abaf29e411-kube-api-access-sk8gd" (OuterVolumeSpecName: "kube-api-access-sk8gd") pod "48dc2b69-a1c1-46b4-9f79-02abaf29e411" (UID: "48dc2b69-a1c1-46b4-9f79-02abaf29e411"). InnerVolumeSpecName "kube-api-access-sk8gd". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 10 00:37:52.618945 ip-10-0-130-146 kubenswrapper[2570]: I0710 00:37:52.618918 2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a062b873-3bc8-4dc1-96e7-63353d46e0b5-kube-api-access-c5k7g" (OuterVolumeSpecName: "kube-api-access-c5k7g") pod "a062b873-3bc8-4dc1-96e7-63353d46e0b5" (UID: "a062b873-3bc8-4dc1-96e7-63353d46e0b5"). InnerVolumeSpecName "kube-api-access-c5k7g". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 10 00:37:52.619270 ip-10-0-130-146 kubenswrapper[2570]: I0710 00:37:52.619251 2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/126a17da-6d3b-4880-b310-d0cf7612be39-kube-api-access-hq75d" (OuterVolumeSpecName: "kube-api-access-hq75d") pod "126a17da-6d3b-4880-b310-d0cf7612be39" (UID: "126a17da-6d3b-4880-b310-d0cf7612be39"). InnerVolumeSpecName "kube-api-access-hq75d". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 10 00:37:52.621765 ip-10-0-130-146 kubenswrapper[2570]: I0710 00:37:52.621739 2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/48dc2b69-a1c1-46b4-9f79-02abaf29e411-util" (OuterVolumeSpecName: "util") pod "48dc2b69-a1c1-46b4-9f79-02abaf29e411" (UID: "48dc2b69-a1c1-46b4-9f79-02abaf29e411"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 10 00:37:52.622555 ip-10-0-130-146 kubenswrapper[2570]: I0710 00:37:52.622535 2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a062b873-3bc8-4dc1-96e7-63353d46e0b5-util" (OuterVolumeSpecName: "util") pod "a062b873-3bc8-4dc1-96e7-63353d46e0b5" (UID: "a062b873-3bc8-4dc1-96e7-63353d46e0b5"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 10 00:37:52.623019 ip-10-0-130-146 kubenswrapper[2570]: I0710 00:37:52.622995 2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/126a17da-6d3b-4880-b310-d0cf7612be39-util" (OuterVolumeSpecName: "util") pod "126a17da-6d3b-4880-b310-d0cf7612be39" (UID: "126a17da-6d3b-4880-b310-d0cf7612be39"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 10 00:37:52.718089 ip-10-0-130-146 kubenswrapper[2570]: I0710 00:37:52.718030 2570 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/a062b873-3bc8-4dc1-96e7-63353d46e0b5-bundle\") on node \"ip-10-0-130-146.ec2.internal\" DevicePath \"\"" Jul 10 00:37:52.718089 ip-10-0-130-146 kubenswrapper[2570]: I0710 00:37:52.718085 2570 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-c5k7g\" (UniqueName: \"kubernetes.io/projected/a062b873-3bc8-4dc1-96e7-63353d46e0b5-kube-api-access-c5k7g\") on node \"ip-10-0-130-146.ec2.internal\" DevicePath \"\"" Jul 10 00:37:52.718089 ip-10-0-130-146 kubenswrapper[2570]: I0710 00:37:52.718097 2570 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hq75d\" (UniqueName: \"kubernetes.io/projected/126a17da-6d3b-4880-b310-d0cf7612be39-kube-api-access-hq75d\") on node \"ip-10-0-130-146.ec2.internal\" DevicePath \"\"" Jul 10 00:37:52.718313 ip-10-0-130-146 kubenswrapper[2570]: I0710 00:37:52.718107 2570 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/48dc2b69-a1c1-46b4-9f79-02abaf29e411-bundle\") on node \"ip-10-0-130-146.ec2.internal\" DevicePath \"\"" Jul 10 00:37:52.718313 ip-10-0-130-146 kubenswrapper[2570]: I0710 00:37:52.718115 2570 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/a062b873-3bc8-4dc1-96e7-63353d46e0b5-util\") on node \"ip-10-0-130-146.ec2.internal\" DevicePath \"\"" Jul 10 00:37:52.718313 ip-10-0-130-146 kubenswrapper[2570]: I0710 00:37:52.718123 2570 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sk8gd\" (UniqueName: \"kubernetes.io/projected/48dc2b69-a1c1-46b4-9f79-02abaf29e411-kube-api-access-sk8gd\") on node \"ip-10-0-130-146.ec2.internal\" DevicePath \"\"" Jul 10 00:37:52.718313 ip-10-0-130-146 kubenswrapper[2570]: I0710 00:37:52.718130 2570 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/48dc2b69-a1c1-46b4-9f79-02abaf29e411-util\") on node \"ip-10-0-130-146.ec2.internal\" DevicePath \"\"" Jul 10 00:37:52.718313 ip-10-0-130-146 kubenswrapper[2570]: I0710 00:37:52.718140 2570 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/126a17da-6d3b-4880-b310-d0cf7612be39-bundle\") on node \"ip-10-0-130-146.ec2.internal\" DevicePath \"\"" Jul 10 00:37:52.718313 ip-10-0-130-146 kubenswrapper[2570]: I0710 00:37:52.718148 2570 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/126a17da-6d3b-4880-b310-d0cf7612be39-util\") on node \"ip-10-0-130-146.ec2.internal\" DevicePath \"\"" Jul 10 00:37:53.288873 ip-10-0-130-146 kubenswrapper[2570]: I0710 00:37:53.288844 2570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73rs9vd" Jul 10 00:37:53.289092 ip-10-0-130-146 kubenswrapper[2570]: I0710 00:37:53.288872 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73rs9vd" event={"ID":"48dc2b69-a1c1-46b4-9f79-02abaf29e411","Type":"ContainerDied","Data":"9c6adf9e7cb034a60ccb9f6a4e6238ea151e0779ee871ebdc5efab3b48876530"} Jul 10 00:37:53.289092 ip-10-0-130-146 kubenswrapper[2570]: I0710 00:37:53.288930 2570 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="9c6adf9e7cb034a60ccb9f6a4e6238ea151e0779ee871ebdc5efab3b48876530" Jul 10 00:37:53.290811 ip-10-0-130-146 kubenswrapper[2570]: I0710 00:37:53.290789 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1pfl48" event={"ID":"126a17da-6d3b-4880-b310-d0cf7612be39","Type":"ContainerDied","Data":"af0b921df4a94800182423268fd1b2a8da6a162bf14498551cf9785694c08ce3"} Jul 10 00:37:53.290914 ip-10-0-130-146 kubenswrapper[2570]: I0710 00:37:53.290817 2570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1pfl48" Jul 10 00:37:53.290914 ip-10-0-130-146 kubenswrapper[2570]: I0710 00:37:53.290824 2570 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="af0b921df4a94800182423268fd1b2a8da6a162bf14498551cf9785694c08ce3" Jul 10 00:37:53.292465 ip-10-0-130-146 kubenswrapper[2570]: I0710 00:37:53.292449 2570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0hr2tc" Jul 10 00:37:53.292565 ip-10-0-130-146 kubenswrapper[2570]: I0710 00:37:53.292486 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0hr2tc" event={"ID":"b407e13a-88fb-43a1-94fc-ab40c580d6c2","Type":"ContainerDied","Data":"bda9b6af3276bc8f70d93856fe2fa356294e910cd56ece6f92da4cab3e6c7294"} Jul 10 00:37:53.292565 ip-10-0-130-146 kubenswrapper[2570]: I0710 00:37:53.292529 2570 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="bda9b6af3276bc8f70d93856fe2fa356294e910cd56ece6f92da4cab3e6c7294" Jul 10 00:37:53.294369 ip-10-0-130-146 kubenswrapper[2570]: I0710 00:37:53.294348 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759k68sb" event={"ID":"a062b873-3bc8-4dc1-96e7-63353d46e0b5","Type":"ContainerDied","Data":"45c5c596560dd51b565e5b9836fadafbdd65d1ff6fe364fd253ad6da9d7349de"} Jul 10 00:37:53.294452 ip-10-0-130-146 kubenswrapper[2570]: I0710 00:37:53.294377 2570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759k68sb" Jul 10 00:37:53.294452 ip-10-0-130-146 kubenswrapper[2570]: I0710 00:37:53.294386 2570 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="45c5c596560dd51b565e5b9836fadafbdd65d1ff6fe364fd253ad6da9d7349de" Jul 10 00:38:12.316881 ip-10-0-130-146 kubenswrapper[2570]: I0710 00:38:12.316815 2570 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-console/console-6dc8d7758c-tl2pk" podUID="2c2ca2d2-9dc3-45f7-a27a-e10da918748d" containerName="console" containerID="cri-o://78fdc68b523c2721ee456f9280f2d35e349a8e830ec30e102e715487f09a44ca" gracePeriod=15 Jul 10 00:38:12.576080 ip-10-0-130-146 kubenswrapper[2570]: I0710 00:38:12.576012 2570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-6dc8d7758c-tl2pk" Jul 10 00:38:12.599594 ip-10-0-130-146 kubenswrapper[2570]: I0710 00:38:12.599565 2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/2c2ca2d2-9dc3-45f7-a27a-e10da918748d-oauth-serving-cert\") pod \"2c2ca2d2-9dc3-45f7-a27a-e10da918748d\" (UID: \"2c2ca2d2-9dc3-45f7-a27a-e10da918748d\") " Jul 10 00:38:12.599761 ip-10-0-130-146 kubenswrapper[2570]: I0710 00:38:12.599605 2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/2c2ca2d2-9dc3-45f7-a27a-e10da918748d-service-ca\") pod \"2c2ca2d2-9dc3-45f7-a27a-e10da918748d\" (UID: \"2c2ca2d2-9dc3-45f7-a27a-e10da918748d\") " Jul 10 00:38:12.599761 ip-10-0-130-146 kubenswrapper[2570]: I0710 00:38:12.599647 2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-zxc2v\" (UniqueName: \"kubernetes.io/projected/2c2ca2d2-9dc3-45f7-a27a-e10da918748d-kube-api-access-zxc2v\") pod \"2c2ca2d2-9dc3-45f7-a27a-e10da918748d\" (UID: \"2c2ca2d2-9dc3-45f7-a27a-e10da918748d\") " Jul 10 00:38:12.599761 ip-10-0-130-146 kubenswrapper[2570]: I0710 00:38:12.599666 2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/2c2ca2d2-9dc3-45f7-a27a-e10da918748d-console-oauth-config\") pod \"2c2ca2d2-9dc3-45f7-a27a-e10da918748d\" (UID: \"2c2ca2d2-9dc3-45f7-a27a-e10da918748d\") " Jul 10 00:38:12.599761 ip-10-0-130-146 kubenswrapper[2570]: I0710 00:38:12.599686 2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/2c2ca2d2-9dc3-45f7-a27a-e10da918748d-console-serving-cert\") pod \"2c2ca2d2-9dc3-45f7-a27a-e10da918748d\" (UID: \"2c2ca2d2-9dc3-45f7-a27a-e10da918748d\") " Jul 10 00:38:12.599761 ip-10-0-130-146 kubenswrapper[2570]: I0710 00:38:12.599712 2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/2c2ca2d2-9dc3-45f7-a27a-e10da918748d-trusted-ca-bundle\") pod \"2c2ca2d2-9dc3-45f7-a27a-e10da918748d\" (UID: \"2c2ca2d2-9dc3-45f7-a27a-e10da918748d\") " Jul 10 00:38:12.599761 ip-10-0-130-146 kubenswrapper[2570]: I0710 00:38:12.599761 2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/2c2ca2d2-9dc3-45f7-a27a-e10da918748d-console-config\") pod \"2c2ca2d2-9dc3-45f7-a27a-e10da918748d\" (UID: \"2c2ca2d2-9dc3-45f7-a27a-e10da918748d\") " Jul 10 00:38:12.600087 ip-10-0-130-146 kubenswrapper[2570]: I0710 00:38:12.599966 2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2c2ca2d2-9dc3-45f7-a27a-e10da918748d-oauth-serving-cert" (OuterVolumeSpecName: "oauth-serving-cert") pod "2c2ca2d2-9dc3-45f7-a27a-e10da918748d" (UID: "2c2ca2d2-9dc3-45f7-a27a-e10da918748d"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 10 00:38:12.600087 ip-10-0-130-146 kubenswrapper[2570]: I0710 00:38:12.600038 2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2c2ca2d2-9dc3-45f7-a27a-e10da918748d-service-ca" (OuterVolumeSpecName: "service-ca") pod "2c2ca2d2-9dc3-45f7-a27a-e10da918748d" (UID: "2c2ca2d2-9dc3-45f7-a27a-e10da918748d"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 10 00:38:12.600356 ip-10-0-130-146 kubenswrapper[2570]: I0710 00:38:12.600334 2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2c2ca2d2-9dc3-45f7-a27a-e10da918748d-console-config" (OuterVolumeSpecName: "console-config") pod "2c2ca2d2-9dc3-45f7-a27a-e10da918748d" (UID: "2c2ca2d2-9dc3-45f7-a27a-e10da918748d"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 10 00:38:12.600436 ip-10-0-130-146 kubenswrapper[2570]: I0710 00:38:12.600416 2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2c2ca2d2-9dc3-45f7-a27a-e10da918748d-trusted-ca-bundle" (OuterVolumeSpecName: "trusted-ca-bundle") pod "2c2ca2d2-9dc3-45f7-a27a-e10da918748d" (UID: "2c2ca2d2-9dc3-45f7-a27a-e10da918748d"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 10 00:38:12.601917 ip-10-0-130-146 kubenswrapper[2570]: I0710 00:38:12.601888 2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2c2ca2d2-9dc3-45f7-a27a-e10da918748d-console-oauth-config" (OuterVolumeSpecName: "console-oauth-config") pod "2c2ca2d2-9dc3-45f7-a27a-e10da918748d" (UID: "2c2ca2d2-9dc3-45f7-a27a-e10da918748d"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 10 00:38:12.602025 ip-10-0-130-146 kubenswrapper[2570]: I0710 00:38:12.601938 2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2c2ca2d2-9dc3-45f7-a27a-e10da918748d-kube-api-access-zxc2v" (OuterVolumeSpecName: "kube-api-access-zxc2v") pod "2c2ca2d2-9dc3-45f7-a27a-e10da918748d" (UID: "2c2ca2d2-9dc3-45f7-a27a-e10da918748d"). InnerVolumeSpecName "kube-api-access-zxc2v". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 10 00:38:12.602118 ip-10-0-130-146 kubenswrapper[2570]: I0710 00:38:12.602092 2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2c2ca2d2-9dc3-45f7-a27a-e10da918748d-console-serving-cert" (OuterVolumeSpecName: "console-serving-cert") pod "2c2ca2d2-9dc3-45f7-a27a-e10da918748d" (UID: "2c2ca2d2-9dc3-45f7-a27a-e10da918748d"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 10 00:38:12.701424 ip-10-0-130-146 kubenswrapper[2570]: I0710 00:38:12.701386 2570 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/2c2ca2d2-9dc3-45f7-a27a-e10da918748d-service-ca\") on node \"ip-10-0-130-146.ec2.internal\" DevicePath \"\"" Jul 10 00:38:12.701424 ip-10-0-130-146 kubenswrapper[2570]: I0710 00:38:12.701419 2570 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zxc2v\" (UniqueName: \"kubernetes.io/projected/2c2ca2d2-9dc3-45f7-a27a-e10da918748d-kube-api-access-zxc2v\") on node \"ip-10-0-130-146.ec2.internal\" DevicePath \"\"" Jul 10 00:38:12.701424 ip-10-0-130-146 kubenswrapper[2570]: I0710 00:38:12.701429 2570 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/2c2ca2d2-9dc3-45f7-a27a-e10da918748d-console-oauth-config\") on node \"ip-10-0-130-146.ec2.internal\" DevicePath \"\"" Jul 10 00:38:12.701663 ip-10-0-130-146 kubenswrapper[2570]: I0710 00:38:12.701440 2570 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/2c2ca2d2-9dc3-45f7-a27a-e10da918748d-console-serving-cert\") on node \"ip-10-0-130-146.ec2.internal\" DevicePath \"\"" Jul 10 00:38:12.701663 ip-10-0-130-146 kubenswrapper[2570]: I0710 00:38:12.701449 2570 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/2c2ca2d2-9dc3-45f7-a27a-e10da918748d-trusted-ca-bundle\") on node \"ip-10-0-130-146.ec2.internal\" DevicePath \"\"" Jul 10 00:38:12.701663 ip-10-0-130-146 kubenswrapper[2570]: I0710 00:38:12.701457 2570 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/2c2ca2d2-9dc3-45f7-a27a-e10da918748d-console-config\") on node \"ip-10-0-130-146.ec2.internal\" DevicePath \"\"" Jul 10 00:38:12.701663 ip-10-0-130-146 kubenswrapper[2570]: I0710 00:38:12.701466 2570 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/2c2ca2d2-9dc3-45f7-a27a-e10da918748d-oauth-serving-cert\") on node \"ip-10-0-130-146.ec2.internal\" DevicePath \"\"" Jul 10 00:38:13.396376 ip-10-0-130-146 kubenswrapper[2570]: I0710 00:38:13.396343 2570 generic.go:356] "Generic (PLEG): container finished" podID="2c2ca2d2-9dc3-45f7-a27a-e10da918748d" containerID="78fdc68b523c2721ee456f9280f2d35e349a8e830ec30e102e715487f09a44ca" exitCode=0 Jul 10 00:38:13.396809 ip-10-0-130-146 kubenswrapper[2570]: I0710 00:38:13.396386 2570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-6dc8d7758c-tl2pk" Jul 10 00:38:13.396809 ip-10-0-130-146 kubenswrapper[2570]: I0710 00:38:13.396396 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-6dc8d7758c-tl2pk" event={"ID":"2c2ca2d2-9dc3-45f7-a27a-e10da918748d","Type":"ContainerDied","Data":"78fdc68b523c2721ee456f9280f2d35e349a8e830ec30e102e715487f09a44ca"} Jul 10 00:38:13.396809 ip-10-0-130-146 kubenswrapper[2570]: I0710 00:38:13.396434 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-6dc8d7758c-tl2pk" event={"ID":"2c2ca2d2-9dc3-45f7-a27a-e10da918748d","Type":"ContainerDied","Data":"53b6d02eb18fc06d3dc0fc1b716c7078174944ec3dd8b483277f268d46cf63f7"} Jul 10 00:38:13.396809 ip-10-0-130-146 kubenswrapper[2570]: I0710 00:38:13.396452 2570 scope.go:117] "RemoveContainer" containerID="78fdc68b523c2721ee456f9280f2d35e349a8e830ec30e102e715487f09a44ca" Jul 10 00:38:13.405616 ip-10-0-130-146 kubenswrapper[2570]: I0710 00:38:13.405598 2570 scope.go:117] "RemoveContainer" containerID="78fdc68b523c2721ee456f9280f2d35e349a8e830ec30e102e715487f09a44ca" Jul 10 00:38:13.405862 ip-10-0-130-146 kubenswrapper[2570]: E0710 00:38:13.405844 2570 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"78fdc68b523c2721ee456f9280f2d35e349a8e830ec30e102e715487f09a44ca\": container with ID starting with 78fdc68b523c2721ee456f9280f2d35e349a8e830ec30e102e715487f09a44ca not found: ID does not exist" containerID="78fdc68b523c2721ee456f9280f2d35e349a8e830ec30e102e715487f09a44ca" Jul 10 00:38:13.405904 ip-10-0-130-146 kubenswrapper[2570]: I0710 00:38:13.405888 2570 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"78fdc68b523c2721ee456f9280f2d35e349a8e830ec30e102e715487f09a44ca"} err="failed to get container status \"78fdc68b523c2721ee456f9280f2d35e349a8e830ec30e102e715487f09a44ca\": rpc error: code = NotFound desc = could not find container \"78fdc68b523c2721ee456f9280f2d35e349a8e830ec30e102e715487f09a44ca\": container with ID starting with 78fdc68b523c2721ee456f9280f2d35e349a8e830ec30e102e715487f09a44ca not found: ID does not exist" Jul 10 00:38:13.422985 ip-10-0-130-146 kubenswrapper[2570]: I0710 00:38:13.422958 2570 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-6dc8d7758c-tl2pk"] Jul 10 00:38:13.428147 ip-10-0-130-146 kubenswrapper[2570]: I0710 00:38:13.428119 2570 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-6dc8d7758c-tl2pk"] Jul 10 00:38:13.454550 ip-10-0-130-146 kubenswrapper[2570]: I0710 00:38:13.454524 2570 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="2c2ca2d2-9dc3-45f7-a27a-e10da918748d" path="/var/lib/kubelet/pods/2c2ca2d2-9dc3-45f7-a27a-e10da918748d/volumes" Jul 10 00:38:22.158747 ip-10-0-130-146 kubenswrapper[2570]: I0710 00:38:22.158711 2570 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-console-plugin-66b5774985-v9f84"] Jul 10 00:38:22.159298 ip-10-0-130-146 kubenswrapper[2570]: I0710 00:38:22.159272 2570 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="126a17da-6d3b-4880-b310-d0cf7612be39" containerName="pull" Jul 10 00:38:22.159298 ip-10-0-130-146 kubenswrapper[2570]: I0710 00:38:22.159295 2570 state_mem.go:107] "Deleted CPUSet assignment" podUID="126a17da-6d3b-4880-b310-d0cf7612be39" containerName="pull" Jul 10 00:38:22.159471 ip-10-0-130-146 kubenswrapper[2570]: I0710 00:38:22.159311 2570 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="48dc2b69-a1c1-46b4-9f79-02abaf29e411" containerName="pull" Jul 10 00:38:22.159471 ip-10-0-130-146 kubenswrapper[2570]: I0710 00:38:22.159320 2570 state_mem.go:107] "Deleted CPUSet assignment" podUID="48dc2b69-a1c1-46b4-9f79-02abaf29e411" containerName="pull" Jul 10 00:38:22.159471 ip-10-0-130-146 kubenswrapper[2570]: I0710 00:38:22.159335 2570 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a062b873-3bc8-4dc1-96e7-63353d46e0b5" containerName="pull" Jul 10 00:38:22.159471 ip-10-0-130-146 kubenswrapper[2570]: I0710 00:38:22.159344 2570 state_mem.go:107] "Deleted CPUSet assignment" podUID="a062b873-3bc8-4dc1-96e7-63353d46e0b5" containerName="pull" Jul 10 00:38:22.159471 ip-10-0-130-146 kubenswrapper[2570]: I0710 00:38:22.159356 2570 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b407e13a-88fb-43a1-94fc-ab40c580d6c2" containerName="extract" Jul 10 00:38:22.159471 ip-10-0-130-146 kubenswrapper[2570]: I0710 00:38:22.159363 2570 state_mem.go:107] "Deleted CPUSet assignment" podUID="b407e13a-88fb-43a1-94fc-ab40c580d6c2" containerName="extract" Jul 10 00:38:22.159471 ip-10-0-130-146 kubenswrapper[2570]: I0710 00:38:22.159369 2570 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="48dc2b69-a1c1-46b4-9f79-02abaf29e411" containerName="extract" Jul 10 00:38:22.159471 ip-10-0-130-146 kubenswrapper[2570]: I0710 00:38:22.159374 2570 state_mem.go:107] "Deleted CPUSet assignment" podUID="48dc2b69-a1c1-46b4-9f79-02abaf29e411" containerName="extract" Jul 10 00:38:22.159471 ip-10-0-130-146 kubenswrapper[2570]: I0710 00:38:22.159381 2570 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2c2ca2d2-9dc3-45f7-a27a-e10da918748d" containerName="console" Jul 10 00:38:22.159471 ip-10-0-130-146 kubenswrapper[2570]: I0710 00:38:22.159386 2570 state_mem.go:107] "Deleted CPUSet assignment" podUID="2c2ca2d2-9dc3-45f7-a27a-e10da918748d" containerName="console" Jul 10 00:38:22.159471 ip-10-0-130-146 kubenswrapper[2570]: I0710 00:38:22.159391 2570 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b407e13a-88fb-43a1-94fc-ab40c580d6c2" containerName="util" Jul 10 00:38:22.159471 ip-10-0-130-146 kubenswrapper[2570]: I0710 00:38:22.159396 2570 state_mem.go:107] "Deleted CPUSet assignment" podUID="b407e13a-88fb-43a1-94fc-ab40c580d6c2" containerName="util" Jul 10 00:38:22.159471 ip-10-0-130-146 kubenswrapper[2570]: I0710 00:38:22.159404 2570 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b407e13a-88fb-43a1-94fc-ab40c580d6c2" containerName="pull" Jul 10 00:38:22.159471 ip-10-0-130-146 kubenswrapper[2570]: I0710 00:38:22.159408 2570 state_mem.go:107] "Deleted CPUSet assignment" podUID="b407e13a-88fb-43a1-94fc-ab40c580d6c2" containerName="pull" Jul 10 00:38:22.159471 ip-10-0-130-146 kubenswrapper[2570]: I0710 00:38:22.159415 2570 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a062b873-3bc8-4dc1-96e7-63353d46e0b5" containerName="extract" Jul 10 00:38:22.159471 ip-10-0-130-146 kubenswrapper[2570]: I0710 00:38:22.159420 2570 state_mem.go:107] "Deleted CPUSet assignment" podUID="a062b873-3bc8-4dc1-96e7-63353d46e0b5" containerName="extract" Jul 10 00:38:22.159471 ip-10-0-130-146 kubenswrapper[2570]: I0710 00:38:22.159439 2570 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="48dc2b69-a1c1-46b4-9f79-02abaf29e411" containerName="util" Jul 10 00:38:22.159471 ip-10-0-130-146 kubenswrapper[2570]: I0710 00:38:22.159446 2570 state_mem.go:107] "Deleted CPUSet assignment" podUID="48dc2b69-a1c1-46b4-9f79-02abaf29e411" containerName="util" Jul 10 00:38:22.159471 ip-10-0-130-146 kubenswrapper[2570]: I0710 00:38:22.159451 2570 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="126a17da-6d3b-4880-b310-d0cf7612be39" containerName="extract" Jul 10 00:38:22.159471 ip-10-0-130-146 kubenswrapper[2570]: I0710 00:38:22.159456 2570 state_mem.go:107] "Deleted CPUSet assignment" podUID="126a17da-6d3b-4880-b310-d0cf7612be39" containerName="extract" Jul 10 00:38:22.159471 ip-10-0-130-146 kubenswrapper[2570]: I0710 00:38:22.159465 2570 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="126a17da-6d3b-4880-b310-d0cf7612be39" containerName="util" Jul 10 00:38:22.159471 ip-10-0-130-146 kubenswrapper[2570]: I0710 00:38:22.159470 2570 state_mem.go:107] "Deleted CPUSet assignment" podUID="126a17da-6d3b-4880-b310-d0cf7612be39" containerName="util" Jul 10 00:38:22.159471 ip-10-0-130-146 kubenswrapper[2570]: I0710 00:38:22.159477 2570 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a062b873-3bc8-4dc1-96e7-63353d46e0b5" containerName="util" Jul 10 00:38:22.159471 ip-10-0-130-146 kubenswrapper[2570]: I0710 00:38:22.159481 2570 state_mem.go:107] "Deleted CPUSet assignment" podUID="a062b873-3bc8-4dc1-96e7-63353d46e0b5" containerName="util" Jul 10 00:38:22.160269 ip-10-0-130-146 kubenswrapper[2570]: I0710 00:38:22.159554 2570 memory_manager.go:368] "RemoveStaleState removing state" podUID="a062b873-3bc8-4dc1-96e7-63353d46e0b5" containerName="extract" Jul 10 00:38:22.160269 ip-10-0-130-146 kubenswrapper[2570]: I0710 00:38:22.159563 2570 memory_manager.go:368] "RemoveStaleState removing state" podUID="126a17da-6d3b-4880-b310-d0cf7612be39" containerName="extract" Jul 10 00:38:22.160269 ip-10-0-130-146 kubenswrapper[2570]: I0710 00:38:22.159571 2570 memory_manager.go:368] "RemoveStaleState removing state" podUID="2c2ca2d2-9dc3-45f7-a27a-e10da918748d" containerName="console" Jul 10 00:38:22.160269 ip-10-0-130-146 kubenswrapper[2570]: I0710 00:38:22.159579 2570 memory_manager.go:368] "RemoveStaleState removing state" podUID="b407e13a-88fb-43a1-94fc-ab40c580d6c2" containerName="extract" Jul 10 00:38:22.160269 ip-10-0-130-146 kubenswrapper[2570]: I0710 00:38:22.159585 2570 memory_manager.go:368] "RemoveStaleState removing state" podUID="48dc2b69-a1c1-46b4-9f79-02abaf29e411" containerName="extract" Jul 10 00:38:22.164084 ip-10-0-130-146 kubenswrapper[2570]: I0710 00:38:22.164064 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-console-plugin-66b5774985-v9f84" Jul 10 00:38:22.166717 ip-10-0-130-146 kubenswrapper[2570]: I0710 00:38:22.166685 2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kube-root-ca.crt\"" Jul 10 00:38:22.166839 ip-10-0-130-146 kubenswrapper[2570]: I0710 00:38:22.166793 2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"default-dockercfg-kmd5v\"" Jul 10 00:38:22.166912 ip-10-0-130-146 kubenswrapper[2570]: I0710 00:38:22.166841 2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kuadrant-console-nginx-conf\"" Jul 10 00:38:22.166972 ip-10-0-130-146 kubenswrapper[2570]: I0710 00:38:22.166942 2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"openshift-service-ca.crt\"" Jul 10 00:38:22.167617 ip-10-0-130-146 kubenswrapper[2570]: I0710 00:38:22.167602 2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"plugin-serving-cert\"" Jul 10 00:38:22.172344 ip-10-0-130-146 kubenswrapper[2570]: I0710 00:38:22.172323 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-console-plugin-66b5774985-v9f84"] Jul 10 00:38:22.283856 ip-10-0-130-146 kubenswrapper[2570]: I0710 00:38:22.283827 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/84dc7f0b-4d03-461a-a462-1147a97ac1ad-nginx-conf\") pod \"kuadrant-console-plugin-66b5774985-v9f84\" (UID: \"84dc7f0b-4d03-461a-a462-1147a97ac1ad\") " pod="kuadrant-system/kuadrant-console-plugin-66b5774985-v9f84" Jul 10 00:38:22.283856 ip-10-0-130-146 kubenswrapper[2570]: I0710 00:38:22.283861 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zl6mx\" (UniqueName: \"kubernetes.io/projected/84dc7f0b-4d03-461a-a462-1147a97ac1ad-kube-api-access-zl6mx\") pod \"kuadrant-console-plugin-66b5774985-v9f84\" (UID: \"84dc7f0b-4d03-461a-a462-1147a97ac1ad\") " pod="kuadrant-system/kuadrant-console-plugin-66b5774985-v9f84" Jul 10 00:38:22.284067 ip-10-0-130-146 kubenswrapper[2570]: I0710 00:38:22.283912 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugin-serving-cert\" (UniqueName: \"kubernetes.io/secret/84dc7f0b-4d03-461a-a462-1147a97ac1ad-plugin-serving-cert\") pod \"kuadrant-console-plugin-66b5774985-v9f84\" (UID: \"84dc7f0b-4d03-461a-a462-1147a97ac1ad\") " pod="kuadrant-system/kuadrant-console-plugin-66b5774985-v9f84" Jul 10 00:38:22.385283 ip-10-0-130-146 kubenswrapper[2570]: I0710 00:38:22.385258 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/84dc7f0b-4d03-461a-a462-1147a97ac1ad-nginx-conf\") pod \"kuadrant-console-plugin-66b5774985-v9f84\" (UID: \"84dc7f0b-4d03-461a-a462-1147a97ac1ad\") " pod="kuadrant-system/kuadrant-console-plugin-66b5774985-v9f84" Jul 10 00:38:22.385433 ip-10-0-130-146 kubenswrapper[2570]: I0710 00:38:22.385289 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zl6mx\" (UniqueName: \"kubernetes.io/projected/84dc7f0b-4d03-461a-a462-1147a97ac1ad-kube-api-access-zl6mx\") pod \"kuadrant-console-plugin-66b5774985-v9f84\" (UID: \"84dc7f0b-4d03-461a-a462-1147a97ac1ad\") " pod="kuadrant-system/kuadrant-console-plugin-66b5774985-v9f84" Jul 10 00:38:22.385433 ip-10-0-130-146 kubenswrapper[2570]: I0710 00:38:22.385415 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugin-serving-cert\" (UniqueName: \"kubernetes.io/secret/84dc7f0b-4d03-461a-a462-1147a97ac1ad-plugin-serving-cert\") pod \"kuadrant-console-plugin-66b5774985-v9f84\" (UID: \"84dc7f0b-4d03-461a-a462-1147a97ac1ad\") " pod="kuadrant-system/kuadrant-console-plugin-66b5774985-v9f84" Jul 10 00:38:22.385533 ip-10-0-130-146 kubenswrapper[2570]: E0710 00:38:22.385520 2570 secret.go:189] Couldn't get secret kuadrant-system/plugin-serving-cert: secret "plugin-serving-cert" not found Jul 10 00:38:22.385598 ip-10-0-130-146 kubenswrapper[2570]: E0710 00:38:22.385590 2570 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/84dc7f0b-4d03-461a-a462-1147a97ac1ad-plugin-serving-cert podName:84dc7f0b-4d03-461a-a462-1147a97ac1ad nodeName:}" failed. No retries permitted until 2026-07-10 00:38:22.885574141 +0000 UTC m=+721.933952840 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "plugin-serving-cert" (UniqueName: "kubernetes.io/secret/84dc7f0b-4d03-461a-a462-1147a97ac1ad-plugin-serving-cert") pod "kuadrant-console-plugin-66b5774985-v9f84" (UID: "84dc7f0b-4d03-461a-a462-1147a97ac1ad") : secret "plugin-serving-cert" not found Jul 10 00:38:22.385949 ip-10-0-130-146 kubenswrapper[2570]: I0710 00:38:22.385929 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/84dc7f0b-4d03-461a-a462-1147a97ac1ad-nginx-conf\") pod \"kuadrant-console-plugin-66b5774985-v9f84\" (UID: \"84dc7f0b-4d03-461a-a462-1147a97ac1ad\") " pod="kuadrant-system/kuadrant-console-plugin-66b5774985-v9f84" Jul 10 00:38:22.394373 ip-10-0-130-146 kubenswrapper[2570]: I0710 00:38:22.394350 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-zl6mx\" (UniqueName: \"kubernetes.io/projected/84dc7f0b-4d03-461a-a462-1147a97ac1ad-kube-api-access-zl6mx\") pod \"kuadrant-console-plugin-66b5774985-v9f84\" (UID: \"84dc7f0b-4d03-461a-a462-1147a97ac1ad\") " pod="kuadrant-system/kuadrant-console-plugin-66b5774985-v9f84" Jul 10 00:38:22.891103 ip-10-0-130-146 kubenswrapper[2570]: I0710 00:38:22.891070 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugin-serving-cert\" (UniqueName: \"kubernetes.io/secret/84dc7f0b-4d03-461a-a462-1147a97ac1ad-plugin-serving-cert\") pod \"kuadrant-console-plugin-66b5774985-v9f84\" (UID: \"84dc7f0b-4d03-461a-a462-1147a97ac1ad\") " pod="kuadrant-system/kuadrant-console-plugin-66b5774985-v9f84" Jul 10 00:38:22.893388 ip-10-0-130-146 kubenswrapper[2570]: I0710 00:38:22.893364 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugin-serving-cert\" (UniqueName: \"kubernetes.io/secret/84dc7f0b-4d03-461a-a462-1147a97ac1ad-plugin-serving-cert\") pod \"kuadrant-console-plugin-66b5774985-v9f84\" (UID: \"84dc7f0b-4d03-461a-a462-1147a97ac1ad\") " pod="kuadrant-system/kuadrant-console-plugin-66b5774985-v9f84" Jul 10 00:38:23.075060 ip-10-0-130-146 kubenswrapper[2570]: I0710 00:38:23.075018 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-console-plugin-66b5774985-v9f84" Jul 10 00:38:23.208782 ip-10-0-130-146 kubenswrapper[2570]: I0710 00:38:23.208749 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-console-plugin-66b5774985-v9f84"] Jul 10 00:38:23.209841 ip-10-0-130-146 kubenswrapper[2570]: W0710 00:38:23.209817 2570 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod84dc7f0b_4d03_461a_a462_1147a97ac1ad.slice/crio-eae13196fb18b2475146053e554e6f5551f429da919b72597fc72d5359ee3c2f WatchSource:0}: Error finding container eae13196fb18b2475146053e554e6f5551f429da919b72597fc72d5359ee3c2f: Status 404 returned error can't find the container with id eae13196fb18b2475146053e554e6f5551f429da919b72597fc72d5359ee3c2f Jul 10 00:38:23.453747 ip-10-0-130-146 kubenswrapper[2570]: I0710 00:38:23.453666 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-console-plugin-66b5774985-v9f84" event={"ID":"84dc7f0b-4d03-461a-a462-1147a97ac1ad","Type":"ContainerStarted","Data":"eae13196fb18b2475146053e554e6f5551f429da919b72597fc72d5359ee3c2f"} Jul 10 00:38:47.602629 ip-10-0-130-146 kubenswrapper[2570]: I0710 00:38:47.602588 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-console-plugin-66b5774985-v9f84" event={"ID":"84dc7f0b-4d03-461a-a462-1147a97ac1ad","Type":"ContainerStarted","Data":"6ba91b1e4875f6d6ddba8a401188f0eda8d61fa902c8b57bf86d3e0428eefe19"} Jul 10 00:38:47.619635 ip-10-0-130-146 kubenswrapper[2570]: I0710 00:38:47.619583 2570 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-console-plugin-66b5774985-v9f84" podStartSLOduration=2.130401945 podStartE2EDuration="25.619568213s" totalImagesPullingTime="23.489166268s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-10 00:38:22 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 00:38:47.617994639 +0000 UTC m=+746.666373361" watchObservedRunningTime="2026-07-10 00:38:47.619568213 +0000 UTC m=+746.667946934" Jul 10 00:39:14.004738 ip-10-0-130-146 kubenswrapper[2570]: I0710 00:39:14.004709 2570 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-g92hs"] Jul 10 00:39:14.068794 ip-10-0-130-146 kubenswrapper[2570]: I0710 00:39:14.068761 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-g92hs"] Jul 10 00:39:14.068794 ip-10-0-130-146 kubenswrapper[2570]: I0710 00:39:14.068798 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-g92hs"] Jul 10 00:39:14.069042 ip-10-0-130-146 kubenswrapper[2570]: I0710 00:39:14.068893 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-54d6d988d5-g92hs" Jul 10 00:39:14.071308 ip-10-0-130-146 kubenswrapper[2570]: I0710 00:39:14.071280 2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"limitador-limits-config-limitador\"" Jul 10 00:39:14.147041 ip-10-0-130-146 kubenswrapper[2570]: I0710 00:39:14.147011 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/7ef64d51-9c51-48f6-b409-e157f1ec45a5-config-file\") pod \"limitador-limitador-54d6d988d5-g92hs\" (UID: \"7ef64d51-9c51-48f6-b409-e157f1ec45a5\") " pod="kuadrant-system/limitador-limitador-54d6d988d5-g92hs" Jul 10 00:39:14.147189 ip-10-0-130-146 kubenswrapper[2570]: I0710 00:39:14.147118 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lsgnz\" (UniqueName: \"kubernetes.io/projected/7ef64d51-9c51-48f6-b409-e157f1ec45a5-kube-api-access-lsgnz\") pod \"limitador-limitador-54d6d988d5-g92hs\" (UID: \"7ef64d51-9c51-48f6-b409-e157f1ec45a5\") " pod="kuadrant-system/limitador-limitador-54d6d988d5-g92hs" Jul 10 00:39:14.248000 ip-10-0-130-146 kubenswrapper[2570]: I0710 00:39:14.247962 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/7ef64d51-9c51-48f6-b409-e157f1ec45a5-config-file\") pod \"limitador-limitador-54d6d988d5-g92hs\" (UID: \"7ef64d51-9c51-48f6-b409-e157f1ec45a5\") " pod="kuadrant-system/limitador-limitador-54d6d988d5-g92hs" Jul 10 00:39:14.248176 ip-10-0-130-146 kubenswrapper[2570]: I0710 00:39:14.248069 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lsgnz\" (UniqueName: \"kubernetes.io/projected/7ef64d51-9c51-48f6-b409-e157f1ec45a5-kube-api-access-lsgnz\") pod \"limitador-limitador-54d6d988d5-g92hs\" (UID: \"7ef64d51-9c51-48f6-b409-e157f1ec45a5\") " pod="kuadrant-system/limitador-limitador-54d6d988d5-g92hs" Jul 10 00:39:14.248596 ip-10-0-130-146 kubenswrapper[2570]: I0710 00:39:14.248575 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/7ef64d51-9c51-48f6-b409-e157f1ec45a5-config-file\") pod \"limitador-limitador-54d6d988d5-g92hs\" (UID: \"7ef64d51-9c51-48f6-b409-e157f1ec45a5\") " pod="kuadrant-system/limitador-limitador-54d6d988d5-g92hs" Jul 10 00:39:14.256972 ip-10-0-130-146 kubenswrapper[2570]: I0710 00:39:14.256913 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-lsgnz\" (UniqueName: \"kubernetes.io/projected/7ef64d51-9c51-48f6-b409-e157f1ec45a5-kube-api-access-lsgnz\") pod \"limitador-limitador-54d6d988d5-g92hs\" (UID: \"7ef64d51-9c51-48f6-b409-e157f1ec45a5\") " pod="kuadrant-system/limitador-limitador-54d6d988d5-g92hs" Jul 10 00:39:14.379277 ip-10-0-130-146 kubenswrapper[2570]: I0710 00:39:14.379240 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-54d6d988d5-g92hs" Jul 10 00:39:14.556666 ip-10-0-130-146 kubenswrapper[2570]: I0710 00:39:14.556552 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-g92hs"] Jul 10 00:39:14.559717 ip-10-0-130-146 kubenswrapper[2570]: W0710 00:39:14.559678 2570 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7ef64d51_9c51_48f6_b409_e157f1ec45a5.slice/crio-f1b362f4498514e619aa4cad56f00cd9eb52d76769aada11b8a3146c321cc62b WatchSource:0}: Error finding container f1b362f4498514e619aa4cad56f00cd9eb52d76769aada11b8a3146c321cc62b: Status 404 returned error can't find the container with id f1b362f4498514e619aa4cad56f00cd9eb52d76769aada11b8a3146c321cc62b Jul 10 00:39:14.747598 ip-10-0-130-146 kubenswrapper[2570]: I0710 00:39:14.747556 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-54d6d988d5-g92hs" event={"ID":"7ef64d51-9c51-48f6-b409-e157f1ec45a5","Type":"ContainerStarted","Data":"f1b362f4498514e619aa4cad56f00cd9eb52d76769aada11b8a3146c321cc62b"} Jul 10 00:39:17.767771 ip-10-0-130-146 kubenswrapper[2570]: I0710 00:39:17.767687 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-54d6d988d5-g92hs" event={"ID":"7ef64d51-9c51-48f6-b409-e157f1ec45a5","Type":"ContainerStarted","Data":"088144dc7825694872d771dcefba4f3d40b63dd08039aae72bf242e884c5da1c"} Jul 10 00:39:17.768148 ip-10-0-130-146 kubenswrapper[2570]: I0710 00:39:17.767806 2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/limitador-limitador-54d6d988d5-g92hs" Jul 10 00:39:17.785476 ip-10-0-130-146 kubenswrapper[2570]: I0710 00:39:17.785430 2570 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/limitador-limitador-54d6d988d5-g92hs" podStartSLOduration=1.853567902 podStartE2EDuration="4.785415547s" totalImagesPullingTime="2.931847645s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-10 00:39:13 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 00:39:17.784329738 +0000 UTC m=+776.832708459" watchObservedRunningTime="2026-07-10 00:39:17.785415547 +0000 UTC m=+776.833794268" Jul 10 00:39:28.774541 ip-10-0-130-146 kubenswrapper[2570]: I0710 00:39:28.774512 2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/limitador-limitador-54d6d988d5-g92hs" Jul 10 00:40:34.402255 ip-10-0-130-146 kubenswrapper[2570]: I0710 00:40:34.402223 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-g92hs"] Jul 10 00:40:55.689610 ip-10-0-130-146 kubenswrapper[2570]: I0710 00:40:55.689564 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-g92hs"] Jul 10 00:41:02.087764 ip-10-0-130-146 kubenswrapper[2570]: I0710 00:41:02.087730 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-g92hs"] Jul 10 00:41:13.292888 ip-10-0-130-146 kubenswrapper[2570]: I0710 00:41:13.292809 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-g92hs"] Jul 10 00:41:20.786339 ip-10-0-130-146 kubenswrapper[2570]: I0710 00:41:20.786309 2570 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbrs2lp"] Jul 10 00:41:20.789712 ip-10-0-130-146 kubenswrapper[2570]: I0710 00:41:20.789694 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbrs2lp" Jul 10 00:41:20.792290 ip-10-0-130-146 kubenswrapper[2570]: I0710 00:41:20.792266 2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"kube-root-ca.crt\"" Jul 10 00:41:20.792406 ip-10-0-130-146 kubenswrapper[2570]: I0710 00:41:20.792353 2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"openshift-service-ca.crt\"" Jul 10 00:41:20.793442 ip-10-0-130-146 kubenswrapper[2570]: I0710 00:41:20.793420 2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-unab60ef4d3a239b5143b412cab04acac3-kserve-self-signed-certs\"" Jul 10 00:41:20.793555 ip-10-0-130-146 kubenswrapper[2570]: I0710 00:41:20.793444 2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"default-dockercfg-bqvgx\"" Jul 10 00:41:20.799864 ip-10-0-130-146 kubenswrapper[2570]: I0710 00:41:20.799841 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbrs2lp"] Jul 10 00:41:20.917661 ip-10-0-130-146 kubenswrapper[2570]: I0710 00:41:20.917631 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7debf947-0a80-4ee9-945d-b9ee2bfc98cb-dshm\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbrs2lp\" (UID: \"7debf947-0a80-4ee9-945d-b9ee2bfc98cb\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbrs2lp" Jul 10 00:41:20.917661 ip-10-0-130-146 kubenswrapper[2570]: I0710 00:41:20.917663 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7debf947-0a80-4ee9-945d-b9ee2bfc98cb-model-cache\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbrs2lp\" (UID: \"7debf947-0a80-4ee9-945d-b9ee2bfc98cb\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbrs2lp" Jul 10 00:41:20.918319 ip-10-0-130-146 kubenswrapper[2570]: I0710 00:41:20.917690 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7debf947-0a80-4ee9-945d-b9ee2bfc98cb-tmp-dir\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbrs2lp\" (UID: \"7debf947-0a80-4ee9-945d-b9ee2bfc98cb\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbrs2lp" Jul 10 00:41:20.918319 ip-10-0-130-146 kubenswrapper[2570]: I0710 00:41:20.917708 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7debf947-0a80-4ee9-945d-b9ee2bfc98cb-tls-certs\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbrs2lp\" (UID: \"7debf947-0a80-4ee9-945d-b9ee2bfc98cb\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbrs2lp" Jul 10 00:41:20.918319 ip-10-0-130-146 kubenswrapper[2570]: I0710 00:41:20.917767 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7debf947-0a80-4ee9-945d-b9ee2bfc98cb-home\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbrs2lp\" (UID: \"7debf947-0a80-4ee9-945d-b9ee2bfc98cb\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbrs2lp" Jul 10 00:41:20.918319 ip-10-0-130-146 kubenswrapper[2570]: I0710 00:41:20.917882 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xnrcx\" (UniqueName: \"kubernetes.io/projected/7debf947-0a80-4ee9-945d-b9ee2bfc98cb-kube-api-access-xnrcx\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbrs2lp\" (UID: \"7debf947-0a80-4ee9-945d-b9ee2bfc98cb\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbrs2lp" Jul 10 00:41:21.019098 ip-10-0-130-146 kubenswrapper[2570]: I0710 00:41:21.019031 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7debf947-0a80-4ee9-945d-b9ee2bfc98cb-dshm\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbrs2lp\" (UID: \"7debf947-0a80-4ee9-945d-b9ee2bfc98cb\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbrs2lp" Jul 10 00:41:21.019098 ip-10-0-130-146 kubenswrapper[2570]: I0710 00:41:21.019099 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7debf947-0a80-4ee9-945d-b9ee2bfc98cb-model-cache\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbrs2lp\" (UID: \"7debf947-0a80-4ee9-945d-b9ee2bfc98cb\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbrs2lp" Jul 10 00:41:21.019353 ip-10-0-130-146 kubenswrapper[2570]: I0710 00:41:21.019145 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7debf947-0a80-4ee9-945d-b9ee2bfc98cb-tmp-dir\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbrs2lp\" (UID: \"7debf947-0a80-4ee9-945d-b9ee2bfc98cb\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbrs2lp" Jul 10 00:41:21.019353 ip-10-0-130-146 kubenswrapper[2570]: I0710 00:41:21.019169 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7debf947-0a80-4ee9-945d-b9ee2bfc98cb-tls-certs\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbrs2lp\" (UID: \"7debf947-0a80-4ee9-945d-b9ee2bfc98cb\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbrs2lp" Jul 10 00:41:21.019353 ip-10-0-130-146 kubenswrapper[2570]: I0710 00:41:21.019186 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7debf947-0a80-4ee9-945d-b9ee2bfc98cb-home\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbrs2lp\" (UID: \"7debf947-0a80-4ee9-945d-b9ee2bfc98cb\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbrs2lp" Jul 10 00:41:21.019353 ip-10-0-130-146 kubenswrapper[2570]: I0710 00:41:21.019240 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xnrcx\" (UniqueName: \"kubernetes.io/projected/7debf947-0a80-4ee9-945d-b9ee2bfc98cb-kube-api-access-xnrcx\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbrs2lp\" (UID: \"7debf947-0a80-4ee9-945d-b9ee2bfc98cb\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbrs2lp" Jul 10 00:41:21.019602 ip-10-0-130-146 kubenswrapper[2570]: I0710 00:41:21.019575 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7debf947-0a80-4ee9-945d-b9ee2bfc98cb-model-cache\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbrs2lp\" (UID: \"7debf947-0a80-4ee9-945d-b9ee2bfc98cb\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbrs2lp" Jul 10 00:41:21.019662 ip-10-0-130-146 kubenswrapper[2570]: I0710 00:41:21.019602 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7debf947-0a80-4ee9-945d-b9ee2bfc98cb-tmp-dir\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbrs2lp\" (UID: \"7debf947-0a80-4ee9-945d-b9ee2bfc98cb\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbrs2lp" Jul 10 00:41:21.019698 ip-10-0-130-146 kubenswrapper[2570]: I0710 00:41:21.019662 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7debf947-0a80-4ee9-945d-b9ee2bfc98cb-home\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbrs2lp\" (UID: \"7debf947-0a80-4ee9-945d-b9ee2bfc98cb\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbrs2lp" Jul 10 00:41:21.021346 ip-10-0-130-146 kubenswrapper[2570]: I0710 00:41:21.021315 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7debf947-0a80-4ee9-945d-b9ee2bfc98cb-dshm\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbrs2lp\" (UID: \"7debf947-0a80-4ee9-945d-b9ee2bfc98cb\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbrs2lp" Jul 10 00:41:21.021562 ip-10-0-130-146 kubenswrapper[2570]: I0710 00:41:21.021544 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7debf947-0a80-4ee9-945d-b9ee2bfc98cb-tls-certs\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbrs2lp\" (UID: \"7debf947-0a80-4ee9-945d-b9ee2bfc98cb\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbrs2lp" Jul 10 00:41:21.027739 ip-10-0-130-146 kubenswrapper[2570]: I0710 00:41:21.027714 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xnrcx\" (UniqueName: \"kubernetes.io/projected/7debf947-0a80-4ee9-945d-b9ee2bfc98cb-kube-api-access-xnrcx\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbrs2lp\" (UID: \"7debf947-0a80-4ee9-945d-b9ee2bfc98cb\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbrs2lp" Jul 10 00:41:21.100425 ip-10-0-130-146 kubenswrapper[2570]: I0710 00:41:21.100336 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbrs2lp" Jul 10 00:41:21.236191 ip-10-0-130-146 kubenswrapper[2570]: I0710 00:41:21.236169 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbrs2lp"] Jul 10 00:41:21.237720 ip-10-0-130-146 kubenswrapper[2570]: W0710 00:41:21.237690 2570 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7debf947_0a80_4ee9_945d_b9ee2bfc98cb.slice/crio-86b98c2a568683d5a3d064f7c7d621d36a7a34bd4d549be85473244d0c22eec3 WatchSource:0}: Error finding container 86b98c2a568683d5a3d064f7c7d621d36a7a34bd4d549be85473244d0c22eec3: Status 404 returned error can't find the container with id 86b98c2a568683d5a3d064f7c7d621d36a7a34bd4d549be85473244d0c22eec3 Jul 10 00:41:21.330176 ip-10-0-130-146 kubenswrapper[2570]: I0710 00:41:21.330147 2570 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7866497494-d6fcd_2b6cb4ca-79ff-42f0-b080-66af842f22d3/console-operator/2.log" Jul 10 00:41:21.331218 ip-10-0-130-146 kubenswrapper[2570]: I0710 00:41:21.331198 2570 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7866497494-d6fcd_2b6cb4ca-79ff-42f0-b080-66af842f22d3/console-operator/2.log" Jul 10 00:41:21.333732 ip-10-0-130-146 kubenswrapper[2570]: I0710 00:41:21.333714 2570 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-mbs58_c4ef019a-364a-4933-ab85-de86d86167f1/ovn-acl-logging/0.log" Jul 10 00:41:21.334620 ip-10-0-130-146 kubenswrapper[2570]: I0710 00:41:21.334603 2570 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-mbs58_c4ef019a-364a-4933-ab85-de86d86167f1/ovn-acl-logging/0.log" Jul 10 00:41:21.406119 ip-10-0-130-146 kubenswrapper[2570]: I0710 00:41:21.406083 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbrs2lp" event={"ID":"7debf947-0a80-4ee9-945d-b9ee2bfc98cb","Type":"ContainerStarted","Data":"86b98c2a568683d5a3d064f7c7d621d36a7a34bd4d549be85473244d0c22eec3"} Jul 10 00:41:22.413272 ip-10-0-130-146 kubenswrapper[2570]: I0710 00:41:22.413236 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbrs2lp" event={"ID":"7debf947-0a80-4ee9-945d-b9ee2bfc98cb","Type":"ContainerStarted","Data":"66751a004e14016b25d7edbf37ea82e281f8526eeb4f1424a65f3ec532e5762a"} Jul 10 00:41:22.432156 ip-10-0-130-146 kubenswrapper[2570]: I0710 00:41:22.432108 2570 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbrs2lp" podStartSLOduration=1.415414396 podStartE2EDuration="2.432093105s" totalImagesPullingTime="1.016678709s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-10 00:41:20 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 00:41:22.430240661 +0000 UTC m=+901.478619381" watchObservedRunningTime="2026-07-10 00:41:22.432093105 +0000 UTC m=+901.480471826" Jul 10 00:41:22.490826 ip-10-0-130-146 kubenswrapper[2570]: I0710 00:41:22.490793 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-g92hs"] Jul 10 00:41:31.101445 ip-10-0-130-146 kubenswrapper[2570]: I0710 00:41:31.101357 2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbrs2lp" Jul 10 00:41:31.101445 ip-10-0-130-146 kubenswrapper[2570]: I0710 00:41:31.101413 2570 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbrs2lp" Jul 10 00:41:31.113738 ip-10-0-130-146 kubenswrapper[2570]: I0710 00:41:31.113709 2570 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbrs2lp" Jul 10 00:41:31.470205 ip-10-0-130-146 kubenswrapper[2570]: I0710 00:41:31.470179 2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbrs2lp" Jul 10 00:41:57.094911 ip-10-0-130-146 kubenswrapper[2570]: I0710 00:41:57.094877 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-g92hs"] Jul 10 00:42:01.987619 ip-10-0-130-146 kubenswrapper[2570]: I0710 00:42:01.987577 2570 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-2tjdx"] Jul 10 00:42:01.991224 ip-10-0-130-146 kubenswrapper[2570]: I0710 00:42:01.991201 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-2tjdx" Jul 10 00:42:01.994124 ip-10-0-130-146 kubenswrapper[2570]: I0710 00:42:01.994106 2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"premium-simulated-simulated-premium-kserve-self-signed-certs\"" Jul 10 00:42:01.999862 ip-10-0-130-146 kubenswrapper[2570]: I0710 00:42:01.999842 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-2tjdx"] Jul 10 00:42:02.072201 ip-10-0-130-146 kubenswrapper[2570]: I0710 00:42:02.072170 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e93dbf6e-62b9-4fae-b48a-299a00e3db81-dshm\") pod \"premium-simulated-simulated-premium-kserve-5fc6fb456f-2tjdx\" (UID: \"e93dbf6e-62b9-4fae-b48a-299a00e3db81\") " pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-2tjdx" Jul 10 00:42:02.072201 ip-10-0-130-146 kubenswrapper[2570]: I0710 00:42:02.072209 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e93dbf6e-62b9-4fae-b48a-299a00e3db81-tls-certs\") pod \"premium-simulated-simulated-premium-kserve-5fc6fb456f-2tjdx\" (UID: \"e93dbf6e-62b9-4fae-b48a-299a00e3db81\") " pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-2tjdx" Jul 10 00:42:02.072399 ip-10-0-130-146 kubenswrapper[2570]: I0710 00:42:02.072251 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e93dbf6e-62b9-4fae-b48a-299a00e3db81-home\") pod \"premium-simulated-simulated-premium-kserve-5fc6fb456f-2tjdx\" (UID: \"e93dbf6e-62b9-4fae-b48a-299a00e3db81\") " pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-2tjdx" Jul 10 00:42:02.072399 ip-10-0-130-146 kubenswrapper[2570]: I0710 00:42:02.072321 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e93dbf6e-62b9-4fae-b48a-299a00e3db81-tmp-dir\") pod \"premium-simulated-simulated-premium-kserve-5fc6fb456f-2tjdx\" (UID: \"e93dbf6e-62b9-4fae-b48a-299a00e3db81\") " pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-2tjdx" Jul 10 00:42:02.072399 ip-10-0-130-146 kubenswrapper[2570]: I0710 00:42:02.072363 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e93dbf6e-62b9-4fae-b48a-299a00e3db81-model-cache\") pod \"premium-simulated-simulated-premium-kserve-5fc6fb456f-2tjdx\" (UID: \"e93dbf6e-62b9-4fae-b48a-299a00e3db81\") " pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-2tjdx" Jul 10 00:42:02.072399 ip-10-0-130-146 kubenswrapper[2570]: I0710 00:42:02.072394 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hkss6\" (UniqueName: \"kubernetes.io/projected/e93dbf6e-62b9-4fae-b48a-299a00e3db81-kube-api-access-hkss6\") pod \"premium-simulated-simulated-premium-kserve-5fc6fb456f-2tjdx\" (UID: \"e93dbf6e-62b9-4fae-b48a-299a00e3db81\") " pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-2tjdx" Jul 10 00:42:02.172961 ip-10-0-130-146 kubenswrapper[2570]: I0710 00:42:02.172922 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e93dbf6e-62b9-4fae-b48a-299a00e3db81-dshm\") pod \"premium-simulated-simulated-premium-kserve-5fc6fb456f-2tjdx\" (UID: \"e93dbf6e-62b9-4fae-b48a-299a00e3db81\") " pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-2tjdx" Jul 10 00:42:02.173172 ip-10-0-130-146 kubenswrapper[2570]: I0710 00:42:02.172970 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e93dbf6e-62b9-4fae-b48a-299a00e3db81-tls-certs\") pod \"premium-simulated-simulated-premium-kserve-5fc6fb456f-2tjdx\" (UID: \"e93dbf6e-62b9-4fae-b48a-299a00e3db81\") " pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-2tjdx" Jul 10 00:42:02.173172 ip-10-0-130-146 kubenswrapper[2570]: I0710 00:42:02.173031 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e93dbf6e-62b9-4fae-b48a-299a00e3db81-home\") pod \"premium-simulated-simulated-premium-kserve-5fc6fb456f-2tjdx\" (UID: \"e93dbf6e-62b9-4fae-b48a-299a00e3db81\") " pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-2tjdx" Jul 10 00:42:02.173172 ip-10-0-130-146 kubenswrapper[2570]: I0710 00:42:02.173091 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e93dbf6e-62b9-4fae-b48a-299a00e3db81-tmp-dir\") pod \"premium-simulated-simulated-premium-kserve-5fc6fb456f-2tjdx\" (UID: \"e93dbf6e-62b9-4fae-b48a-299a00e3db81\") " pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-2tjdx" Jul 10 00:42:02.173172 ip-10-0-130-146 kubenswrapper[2570]: I0710 00:42:02.173136 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e93dbf6e-62b9-4fae-b48a-299a00e3db81-model-cache\") pod \"premium-simulated-simulated-premium-kserve-5fc6fb456f-2tjdx\" (UID: \"e93dbf6e-62b9-4fae-b48a-299a00e3db81\") " pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-2tjdx" Jul 10 00:42:02.173436 ip-10-0-130-146 kubenswrapper[2570]: I0710 00:42:02.173176 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hkss6\" (UniqueName: \"kubernetes.io/projected/e93dbf6e-62b9-4fae-b48a-299a00e3db81-kube-api-access-hkss6\") pod \"premium-simulated-simulated-premium-kserve-5fc6fb456f-2tjdx\" (UID: \"e93dbf6e-62b9-4fae-b48a-299a00e3db81\") " pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-2tjdx" Jul 10 00:42:02.173566 ip-10-0-130-146 kubenswrapper[2570]: I0710 00:42:02.173542 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e93dbf6e-62b9-4fae-b48a-299a00e3db81-home\") pod \"premium-simulated-simulated-premium-kserve-5fc6fb456f-2tjdx\" (UID: \"e93dbf6e-62b9-4fae-b48a-299a00e3db81\") " pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-2tjdx" Jul 10 00:42:02.173643 ip-10-0-130-146 kubenswrapper[2570]: I0710 00:42:02.173573 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e93dbf6e-62b9-4fae-b48a-299a00e3db81-tmp-dir\") pod \"premium-simulated-simulated-premium-kserve-5fc6fb456f-2tjdx\" (UID: \"e93dbf6e-62b9-4fae-b48a-299a00e3db81\") " pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-2tjdx" Jul 10 00:42:02.173643 ip-10-0-130-146 kubenswrapper[2570]: I0710 00:42:02.173616 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e93dbf6e-62b9-4fae-b48a-299a00e3db81-model-cache\") pod \"premium-simulated-simulated-premium-kserve-5fc6fb456f-2tjdx\" (UID: \"e93dbf6e-62b9-4fae-b48a-299a00e3db81\") " pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-2tjdx" Jul 10 00:42:02.175439 ip-10-0-130-146 kubenswrapper[2570]: I0710 00:42:02.175410 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e93dbf6e-62b9-4fae-b48a-299a00e3db81-dshm\") pod \"premium-simulated-simulated-premium-kserve-5fc6fb456f-2tjdx\" (UID: \"e93dbf6e-62b9-4fae-b48a-299a00e3db81\") " pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-2tjdx" Jul 10 00:42:02.175613 ip-10-0-130-146 kubenswrapper[2570]: I0710 00:42:02.175595 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e93dbf6e-62b9-4fae-b48a-299a00e3db81-tls-certs\") pod \"premium-simulated-simulated-premium-kserve-5fc6fb456f-2tjdx\" (UID: \"e93dbf6e-62b9-4fae-b48a-299a00e3db81\") " pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-2tjdx" Jul 10 00:42:02.181178 ip-10-0-130-146 kubenswrapper[2570]: I0710 00:42:02.181153 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-hkss6\" (UniqueName: \"kubernetes.io/projected/e93dbf6e-62b9-4fae-b48a-299a00e3db81-kube-api-access-hkss6\") pod \"premium-simulated-simulated-premium-kserve-5fc6fb456f-2tjdx\" (UID: \"e93dbf6e-62b9-4fae-b48a-299a00e3db81\") " pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-2tjdx" Jul 10 00:42:02.301378 ip-10-0-130-146 kubenswrapper[2570]: I0710 00:42:02.301297 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-2tjdx" Jul 10 00:42:02.432581 ip-10-0-130-146 kubenswrapper[2570]: I0710 00:42:02.432550 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-2tjdx"] Jul 10 00:42:02.434042 ip-10-0-130-146 kubenswrapper[2570]: W0710 00:42:02.434016 2570 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pode93dbf6e_62b9_4fae_b48a_299a00e3db81.slice/crio-b4e4e8c6104eb642432fe94e51b492fe1726e960c4a74724d9c4a3e995d0456c WatchSource:0}: Error finding container b4e4e8c6104eb642432fe94e51b492fe1726e960c4a74724d9c4a3e995d0456c: Status 404 returned error can't find the container with id b4e4e8c6104eb642432fe94e51b492fe1726e960c4a74724d9c4a3e995d0456c Jul 10 00:42:02.435273 ip-10-0-130-146 kubenswrapper[2570]: I0710 00:42:02.435258 2570 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Jul 10 00:42:02.616102 ip-10-0-130-146 kubenswrapper[2570]: I0710 00:42:02.616076 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-2tjdx" event={"ID":"e93dbf6e-62b9-4fae-b48a-299a00e3db81","Type":"ContainerStarted","Data":"b4e4e8c6104eb642432fe94e51b492fe1726e960c4a74724d9c4a3e995d0456c"} Jul 10 00:42:03.622373 ip-10-0-130-146 kubenswrapper[2570]: I0710 00:42:03.622332 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-2tjdx" event={"ID":"e93dbf6e-62b9-4fae-b48a-299a00e3db81","Type":"ContainerStarted","Data":"a9aa98d5541dd76f9ea1fda0a4653f7f37aeeacac3bfe49ba82621aa010f30ab"} Jul 10 00:42:03.642409 ip-10-0-130-146 kubenswrapper[2570]: I0710 00:42:03.642363 2570 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-2tjdx" podStartSLOduration=2.471996428 podStartE2EDuration="2.642347838s" totalImagesPullingTime="170.35141ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-10 00:42:01 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 00:42:03.640864035 +0000 UTC m=+942.689242757" watchObservedRunningTime="2026-07-10 00:42:03.642347838 +0000 UTC m=+942.690726558" Jul 10 00:42:05.198745 ip-10-0-130-146 kubenswrapper[2570]: I0710 00:42:05.198710 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-g92hs"] Jul 10 00:42:12.301936 ip-10-0-130-146 kubenswrapper[2570]: I0710 00:42:12.301891 2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-2tjdx" Jul 10 00:42:12.302353 ip-10-0-130-146 kubenswrapper[2570]: I0710 00:42:12.301973 2570 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-2tjdx" Jul 10 00:42:12.314529 ip-10-0-130-146 kubenswrapper[2570]: I0710 00:42:12.314509 2570 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-2tjdx" Jul 10 00:42:12.684732 ip-10-0-130-146 kubenswrapper[2570]: I0710 00:42:12.684706 2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-2tjdx" Jul 10 00:43:49.688196 ip-10-0-130-146 kubenswrapper[2570]: I0710 00:43:49.688164 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-g92hs"] Jul 10 00:43:58.969712 ip-10-0-130-146 kubenswrapper[2570]: I0710 00:43:58.969670 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-g92hs"] Jul 10 00:44:09.174838 ip-10-0-130-146 kubenswrapper[2570]: I0710 00:44:09.174804 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-g92hs"] Jul 10 00:44:17.880888 ip-10-0-130-146 kubenswrapper[2570]: I0710 00:44:17.880851 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-g92hs"] Jul 10 00:44:30.074534 ip-10-0-130-146 kubenswrapper[2570]: I0710 00:44:30.074496 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-g92hs"] Jul 10 00:45:01.480583 ip-10-0-130-146 kubenswrapper[2570]: I0710 00:45:01.480548 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-g92hs"] Jul 10 00:45:16.877902 ip-10-0-130-146 kubenswrapper[2570]: I0710 00:45:16.877867 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-g92hs"] Jul 10 00:45:51.376248 ip-10-0-130-146 kubenswrapper[2570]: I0710 00:45:51.376169 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-g92hs"] Jul 10 00:45:59.581943 ip-10-0-130-146 kubenswrapper[2570]: I0710 00:45:59.581909 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-g92hs"] Jul 10 00:46:09.982407 ip-10-0-130-146 kubenswrapper[2570]: I0710 00:46:09.982374 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-g92hs"] Jul 10 00:46:15.983994 ip-10-0-130-146 kubenswrapper[2570]: I0710 00:46:15.983957 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-g92hs"] Jul 10 00:46:21.365706 ip-10-0-130-146 kubenswrapper[2570]: I0710 00:46:21.365677 2570 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7866497494-d6fcd_2b6cb4ca-79ff-42f0-b080-66af842f22d3/console-operator/2.log" Jul 10 00:46:21.368261 ip-10-0-130-146 kubenswrapper[2570]: I0710 00:46:21.368237 2570 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7866497494-d6fcd_2b6cb4ca-79ff-42f0-b080-66af842f22d3/console-operator/2.log" Jul 10 00:46:21.369001 ip-10-0-130-146 kubenswrapper[2570]: I0710 00:46:21.368985 2570 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-mbs58_c4ef019a-364a-4933-ab85-de86d86167f1/ovn-acl-logging/0.log" Jul 10 00:46:21.371641 ip-10-0-130-146 kubenswrapper[2570]: I0710 00:46:21.371624 2570 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-mbs58_c4ef019a-364a-4933-ab85-de86d86167f1/ovn-acl-logging/0.log" Jul 10 00:46:26.685225 ip-10-0-130-146 kubenswrapper[2570]: I0710 00:46:26.685194 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-g92hs"] Jul 10 00:46:38.178905 ip-10-0-130-146 kubenswrapper[2570]: I0710 00:46:38.178867 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-g92hs"] Jul 10 00:46:49.479552 ip-10-0-130-146 kubenswrapper[2570]: I0710 00:46:49.479518 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-g92hs"] Jul 10 00:46:59.382755 ip-10-0-130-146 kubenswrapper[2570]: I0710 00:46:59.382721 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-g92hs"] Jul 10 00:47:21.680188 ip-10-0-130-146 kubenswrapper[2570]: I0710 00:47:21.680149 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-g92hs"] Jul 10 00:47:31.381127 ip-10-0-130-146 kubenswrapper[2570]: I0710 00:47:31.381093 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-g92hs"] Jul 10 00:47:54.780736 ip-10-0-130-146 kubenswrapper[2570]: I0710 00:47:54.780704 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-g92hs"] Jul 10 00:48:06.574844 ip-10-0-130-146 kubenswrapper[2570]: I0710 00:48:06.574813 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-g92hs"] Jul 10 00:48:22.984975 ip-10-0-130-146 kubenswrapper[2570]: I0710 00:48:22.984937 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-g92hs"] Jul 10 00:48:32.883273 ip-10-0-130-146 kubenswrapper[2570]: I0710 00:48:32.883237 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-g92hs"] Jul 10 00:48:43.779246 ip-10-0-130-146 kubenswrapper[2570]: I0710 00:48:43.779162 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-g92hs"] Jul 10 00:48:55.074923 ip-10-0-130-146 kubenswrapper[2570]: I0710 00:48:55.074886 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-g92hs"] Jul 10 00:50:49.980546 ip-10-0-130-146 kubenswrapper[2570]: I0710 00:50:49.980513 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-g92hs"] Jul 10 00:51:02.374723 ip-10-0-130-146 kubenswrapper[2570]: I0710 00:51:02.374686 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-g92hs"] Jul 10 00:51:11.788607 ip-10-0-130-146 kubenswrapper[2570]: I0710 00:51:11.788571 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-g92hs"] Jul 10 00:51:21.279349 ip-10-0-130-146 kubenswrapper[2570]: I0710 00:51:21.279314 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-g92hs"] Jul 10 00:51:21.401952 ip-10-0-130-146 kubenswrapper[2570]: I0710 00:51:21.401927 2570 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7866497494-d6fcd_2b6cb4ca-79ff-42f0-b080-66af842f22d3/console-operator/2.log" Jul 10 00:51:21.406124 ip-10-0-130-146 kubenswrapper[2570]: I0710 00:51:21.406102 2570 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-mbs58_c4ef019a-364a-4933-ab85-de86d86167f1/ovn-acl-logging/0.log" Jul 10 00:51:21.406458 ip-10-0-130-146 kubenswrapper[2570]: I0710 00:51:21.406441 2570 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7866497494-d6fcd_2b6cb4ca-79ff-42f0-b080-66af842f22d3/console-operator/2.log" Jul 10 00:51:21.410409 ip-10-0-130-146 kubenswrapper[2570]: I0710 00:51:21.410383 2570 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-mbs58_c4ef019a-364a-4933-ab85-de86d86167f1/ovn-acl-logging/0.log" Jul 10 00:51:39.183332 ip-10-0-130-146 kubenswrapper[2570]: I0710 00:51:39.183296 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-g92hs"] Jul 10 00:51:49.379237 ip-10-0-130-146 kubenswrapper[2570]: I0710 00:51:49.379147 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-g92hs"] Jul 10 00:52:01.078769 ip-10-0-130-146 kubenswrapper[2570]: I0710 00:52:01.078733 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-g92hs"] Jul 10 00:52:08.581853 ip-10-0-130-146 kubenswrapper[2570]: I0710 00:52:08.581816 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-g92hs"] Jul 10 00:52:19.979992 ip-10-0-130-146 kubenswrapper[2570]: I0710 00:52:19.979956 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-g92hs"] Jul 10 00:52:56.979194 ip-10-0-130-146 kubenswrapper[2570]: I0710 00:52:56.979163 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-g92hs"] Jul 10 00:53:06.083372 ip-10-0-130-146 kubenswrapper[2570]: I0710 00:53:06.083338 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-g92hs"] Jul 10 00:53:15.896795 ip-10-0-130-146 kubenswrapper[2570]: I0710 00:53:15.896765 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-g92hs"] Jul 10 00:53:27.080460 ip-10-0-130-146 kubenswrapper[2570]: I0710 00:53:27.080428 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-g92hs"] Jul 10 00:53:34.301123 ip-10-0-130-146 kubenswrapper[2570]: I0710 00:53:34.301093 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-g92hs"] Jul 10 00:53:46.993555 ip-10-0-130-146 kubenswrapper[2570]: I0710 00:53:46.993523 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-g92hs"] Jul 10 00:53:57.883902 ip-10-0-130-146 kubenswrapper[2570]: I0710 00:53:57.883867 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-g92hs"] Jul 10 00:54:07.200069 ip-10-0-130-146 kubenswrapper[2570]: I0710 00:54:07.200023 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-g92hs"] Jul 10 00:54:17.288833 ip-10-0-130-146 kubenswrapper[2570]: I0710 00:54:17.288795 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-g92hs"] Jul 10 00:54:26.585063 ip-10-0-130-146 kubenswrapper[2570]: I0710 00:54:26.585009 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-g92hs"] Jul 10 00:54:36.180285 ip-10-0-130-146 kubenswrapper[2570]: I0710 00:54:36.180248 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-g92hs"] Jul 10 00:54:49.082192 ip-10-0-130-146 kubenswrapper[2570]: I0710 00:54:49.082111 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-g92hs"] Jul 10 00:54:58.189266 ip-10-0-130-146 kubenswrapper[2570]: I0710 00:54:58.189229 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-g92hs"] Jul 10 00:55:07.989336 ip-10-0-130-146 kubenswrapper[2570]: I0710 00:55:07.989299 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-g92hs"] Jul 10 00:55:18.274652 ip-10-0-130-146 kubenswrapper[2570]: I0710 00:55:18.274614 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-g92hs"] Jul 10 00:55:30.387187 ip-10-0-130-146 kubenswrapper[2570]: I0710 00:55:30.387152 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-g92hs"] Jul 10 00:55:38.985312 ip-10-0-130-146 kubenswrapper[2570]: I0710 00:55:38.985277 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-g92hs"] Jul 10 00:55:51.984774 ip-10-0-130-146 kubenswrapper[2570]: I0710 00:55:51.984735 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-g92hs"] Jul 10 00:56:02.587192 ip-10-0-130-146 kubenswrapper[2570]: I0710 00:56:02.587160 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-g92hs"] Jul 10 00:56:15.584068 ip-10-0-130-146 kubenswrapper[2570]: I0710 00:56:15.584020 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-g92hs"] Jul 10 00:56:21.450716 ip-10-0-130-146 kubenswrapper[2570]: I0710 00:56:21.450685 2570 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7866497494-d6fcd_2b6cb4ca-79ff-42f0-b080-66af842f22d3/console-operator/2.log" Jul 10 00:56:21.454872 ip-10-0-130-146 kubenswrapper[2570]: I0710 00:56:21.454846 2570 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7866497494-d6fcd_2b6cb4ca-79ff-42f0-b080-66af842f22d3/console-operator/2.log" Jul 10 00:56:21.455251 ip-10-0-130-146 kubenswrapper[2570]: I0710 00:56:21.455227 2570 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-mbs58_c4ef019a-364a-4933-ab85-de86d86167f1/ovn-acl-logging/0.log" Jul 10 00:56:21.458494 ip-10-0-130-146 kubenswrapper[2570]: I0710 00:56:21.458473 2570 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-mbs58_c4ef019a-364a-4933-ab85-de86d86167f1/ovn-acl-logging/0.log" Jul 10 00:56:24.682430 ip-10-0-130-146 kubenswrapper[2570]: I0710 00:56:24.682397 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-g92hs"] Jul 10 00:56:44.683486 ip-10-0-130-146 kubenswrapper[2570]: I0710 00:56:44.683444 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-g92hs"] Jul 10 00:56:55.181548 ip-10-0-130-146 kubenswrapper[2570]: I0710 00:56:55.181513 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-g92hs"] Jul 10 00:57:07.173955 ip-10-0-130-146 kubenswrapper[2570]: I0710 00:57:07.173920 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-g92hs"] Jul 10 00:57:18.479594 ip-10-0-130-146 kubenswrapper[2570]: I0710 00:57:18.479562 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-g92hs"] Jul 10 00:57:26.180770 ip-10-0-130-146 kubenswrapper[2570]: I0710 00:57:26.180732 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-g92hs"] Jul 10 00:57:37.984912 ip-10-0-130-146 kubenswrapper[2570]: I0710 00:57:37.984874 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-g92hs"] Jul 10 00:58:56.583271 ip-10-0-130-146 kubenswrapper[2570]: I0710 00:58:56.583233 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-g92hs"] Jul 10 00:59:05.481112 ip-10-0-130-146 kubenswrapper[2570]: I0710 00:59:05.481081 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-g92hs"] Jul 10 00:59:18.983958 ip-10-0-130-146 kubenswrapper[2570]: I0710 00:59:18.983926 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-g92hs"] Jul 10 00:59:29.341969 ip-10-0-130-146 kubenswrapper[2570]: I0710 00:59:29.341931 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-g92hs"] Jul 10 00:59:58.178823 ip-10-0-130-146 kubenswrapper[2570]: I0710 00:59:58.178788 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-g92hs"] Jul 10 01:00:00.130093 ip-10-0-130-146 kubenswrapper[2570]: I0710 01:00:00.130044 2570 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29727420-hjhd8"] Jul 10 01:00:00.133588 ip-10-0-130-146 kubenswrapper[2570]: I0710 01:00:00.133572 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29727420-hjhd8" Jul 10 01:00:00.136163 ip-10-0-130-146 kubenswrapper[2570]: I0710 01:00:00.136144 2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-gsqbl\"" Jul 10 01:00:00.140076 ip-10-0-130-146 kubenswrapper[2570]: I0710 01:00:00.140041 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29727420-hjhd8"] Jul 10 01:00:00.145448 ip-10-0-130-146 kubenswrapper[2570]: I0710 01:00:00.145427 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29727420-hjhd8" Jul 10 01:00:00.275106 ip-10-0-130-146 kubenswrapper[2570]: I0710 01:00:00.275081 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29727420-hjhd8"] Jul 10 01:00:00.277167 ip-10-0-130-146 kubenswrapper[2570]: W0710 01:00:00.277144 2570 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podee00c80e_95c7_4212_aefa_94ddb355bc19.slice/crio-a27a35c96b55844600624b9ee05bac94b8012ecef7bc7997b8ed398f5604e9fa WatchSource:0}: Error finding container a27a35c96b55844600624b9ee05bac94b8012ecef7bc7997b8ed398f5604e9fa: Status 404 returned error can't find the container with id a27a35c96b55844600624b9ee05bac94b8012ecef7bc7997b8ed398f5604e9fa Jul 10 01:00:00.279017 ip-10-0-130-146 kubenswrapper[2570]: I0710 01:00:00.278997 2570 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Jul 10 01:00:00.331391 ip-10-0-130-146 kubenswrapper[2570]: I0710 01:00:00.331356 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29727420-hjhd8" event={"ID":"ee00c80e-95c7-4212-aefa-94ddb355bc19","Type":"ContainerStarted","Data":"a27a35c96b55844600624b9ee05bac94b8012ecef7bc7997b8ed398f5604e9fa"} Jul 10 01:00:05.362248 ip-10-0-130-146 kubenswrapper[2570]: I0710 01:00:05.362208 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29727420-hjhd8" event={"ID":"ee00c80e-95c7-4212-aefa-94ddb355bc19","Type":"ContainerStarted","Data":"5718b7ce673dc1879243d8c3d6b313986db05153741369c123a11ea6331960be"} Jul 10 01:00:05.377551 ip-10-0-130-146 kubenswrapper[2570]: I0710 01:00:05.377506 2570 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29727420-hjhd8" podStartSLOduration=1.054862262 podStartE2EDuration="5.377493612s" totalImagesPullingTime="4.32263135s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-10 01:00:00 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 01:00:05.376933976 +0000 UTC m=+2024.425312698" watchObservedRunningTime="2026-07-10 01:00:05.377493612 +0000 UTC m=+2024.425872333" Jul 10 01:00:12.479043 ip-10-0-130-146 kubenswrapper[2570]: I0710 01:00:12.479006 2570 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-mt-31899000/e2e-lifecycle-model-31899000-kserve-5448f45ff-28746"] Jul 10 01:00:12.482697 ip-10-0-130-146 kubenswrapper[2570]: I0710 01:00:12.482681 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-mt-31899000/e2e-lifecycle-model-31899000-kserve-5448f45ff-28746" Jul 10 01:00:12.487279 ip-10-0-130-146 kubenswrapper[2570]: I0710 01:00:12.487257 2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-mt-31899000\"/\"openshift-service-ca.crt\"" Jul 10 01:00:12.487279 ip-10-0-130-146 kubenswrapper[2570]: I0710 01:00:12.487268 2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-mt-31899000\"/\"e2e-lifecycle-model-31899000-kserve-self-signed-certs\"" Jul 10 01:00:12.487279 ip-10-0-130-146 kubenswrapper[2570]: I0710 01:00:12.487287 2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-mt-31899000\"/\"kube-root-ca.crt\"" Jul 10 01:00:12.487528 ip-10-0-130-146 kubenswrapper[2570]: I0710 01:00:12.487259 2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-mt-31899000\"/\"default-dockercfg-lptvs\"" Jul 10 01:00:12.494735 ip-10-0-130-146 kubenswrapper[2570]: I0710 01:00:12.494715 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-mt-31899000/e2e-lifecycle-model-31899000-kserve-5448f45ff-28746"] Jul 10 01:00:12.545875 ip-10-0-130-146 kubenswrapper[2570]: I0710 01:00:12.545855 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/98ac1d0b-e64f-4c26-9e5d-e4c2c251c2ca-home\") pod \"e2e-lifecycle-model-31899000-kserve-5448f45ff-28746\" (UID: \"98ac1d0b-e64f-4c26-9e5d-e4c2c251c2ca\") " pod="ai-tenant-e2e-mt-31899000/e2e-lifecycle-model-31899000-kserve-5448f45ff-28746" Jul 10 01:00:12.545999 ip-10-0-130-146 kubenswrapper[2570]: I0710 01:00:12.545893 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/98ac1d0b-e64f-4c26-9e5d-e4c2c251c2ca-model-cache\") pod \"e2e-lifecycle-model-31899000-kserve-5448f45ff-28746\" (UID: \"98ac1d0b-e64f-4c26-9e5d-e4c2c251c2ca\") " pod="ai-tenant-e2e-mt-31899000/e2e-lifecycle-model-31899000-kserve-5448f45ff-28746" Jul 10 01:00:12.545999 ip-10-0-130-146 kubenswrapper[2570]: I0710 01:00:12.545917 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/98ac1d0b-e64f-4c26-9e5d-e4c2c251c2ca-tls-certs\") pod \"e2e-lifecycle-model-31899000-kserve-5448f45ff-28746\" (UID: \"98ac1d0b-e64f-4c26-9e5d-e4c2c251c2ca\") " pod="ai-tenant-e2e-mt-31899000/e2e-lifecycle-model-31899000-kserve-5448f45ff-28746" Jul 10 01:00:12.546171 ip-10-0-130-146 kubenswrapper[2570]: I0710 01:00:12.546065 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/98ac1d0b-e64f-4c26-9e5d-e4c2c251c2ca-tmp-dir\") pod \"e2e-lifecycle-model-31899000-kserve-5448f45ff-28746\" (UID: \"98ac1d0b-e64f-4c26-9e5d-e4c2c251c2ca\") " pod="ai-tenant-e2e-mt-31899000/e2e-lifecycle-model-31899000-kserve-5448f45ff-28746" Jul 10 01:00:12.546171 ip-10-0-130-146 kubenswrapper[2570]: I0710 01:00:12.546116 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/98ac1d0b-e64f-4c26-9e5d-e4c2c251c2ca-dshm\") pod \"e2e-lifecycle-model-31899000-kserve-5448f45ff-28746\" (UID: \"98ac1d0b-e64f-4c26-9e5d-e4c2c251c2ca\") " pod="ai-tenant-e2e-mt-31899000/e2e-lifecycle-model-31899000-kserve-5448f45ff-28746" Jul 10 01:00:12.546256 ip-10-0-130-146 kubenswrapper[2570]: I0710 01:00:12.546196 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8s27f\" (UniqueName: \"kubernetes.io/projected/98ac1d0b-e64f-4c26-9e5d-e4c2c251c2ca-kube-api-access-8s27f\") pod \"e2e-lifecycle-model-31899000-kserve-5448f45ff-28746\" (UID: \"98ac1d0b-e64f-4c26-9e5d-e4c2c251c2ca\") " pod="ai-tenant-e2e-mt-31899000/e2e-lifecycle-model-31899000-kserve-5448f45ff-28746" Jul 10 01:00:12.647217 ip-10-0-130-146 kubenswrapper[2570]: I0710 01:00:12.647185 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8s27f\" (UniqueName: \"kubernetes.io/projected/98ac1d0b-e64f-4c26-9e5d-e4c2c251c2ca-kube-api-access-8s27f\") pod \"e2e-lifecycle-model-31899000-kserve-5448f45ff-28746\" (UID: \"98ac1d0b-e64f-4c26-9e5d-e4c2c251c2ca\") " pod="ai-tenant-e2e-mt-31899000/e2e-lifecycle-model-31899000-kserve-5448f45ff-28746" Jul 10 01:00:12.647361 ip-10-0-130-146 kubenswrapper[2570]: I0710 01:00:12.647236 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/98ac1d0b-e64f-4c26-9e5d-e4c2c251c2ca-home\") pod \"e2e-lifecycle-model-31899000-kserve-5448f45ff-28746\" (UID: \"98ac1d0b-e64f-4c26-9e5d-e4c2c251c2ca\") " pod="ai-tenant-e2e-mt-31899000/e2e-lifecycle-model-31899000-kserve-5448f45ff-28746" Jul 10 01:00:12.647361 ip-10-0-130-146 kubenswrapper[2570]: I0710 01:00:12.647267 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/98ac1d0b-e64f-4c26-9e5d-e4c2c251c2ca-model-cache\") pod \"e2e-lifecycle-model-31899000-kserve-5448f45ff-28746\" (UID: \"98ac1d0b-e64f-4c26-9e5d-e4c2c251c2ca\") " pod="ai-tenant-e2e-mt-31899000/e2e-lifecycle-model-31899000-kserve-5448f45ff-28746" Jul 10 01:00:12.647361 ip-10-0-130-146 kubenswrapper[2570]: I0710 01:00:12.647290 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/98ac1d0b-e64f-4c26-9e5d-e4c2c251c2ca-tls-certs\") pod \"e2e-lifecycle-model-31899000-kserve-5448f45ff-28746\" (UID: \"98ac1d0b-e64f-4c26-9e5d-e4c2c251c2ca\") " pod="ai-tenant-e2e-mt-31899000/e2e-lifecycle-model-31899000-kserve-5448f45ff-28746" Jul 10 01:00:12.647361 ip-10-0-130-146 kubenswrapper[2570]: I0710 01:00:12.647314 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/98ac1d0b-e64f-4c26-9e5d-e4c2c251c2ca-tmp-dir\") pod \"e2e-lifecycle-model-31899000-kserve-5448f45ff-28746\" (UID: \"98ac1d0b-e64f-4c26-9e5d-e4c2c251c2ca\") " pod="ai-tenant-e2e-mt-31899000/e2e-lifecycle-model-31899000-kserve-5448f45ff-28746" Jul 10 01:00:12.647361 ip-10-0-130-146 kubenswrapper[2570]: I0710 01:00:12.647347 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/98ac1d0b-e64f-4c26-9e5d-e4c2c251c2ca-dshm\") pod \"e2e-lifecycle-model-31899000-kserve-5448f45ff-28746\" (UID: \"98ac1d0b-e64f-4c26-9e5d-e4c2c251c2ca\") " pod="ai-tenant-e2e-mt-31899000/e2e-lifecycle-model-31899000-kserve-5448f45ff-28746" Jul 10 01:00:12.647722 ip-10-0-130-146 kubenswrapper[2570]: I0710 01:00:12.647701 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/98ac1d0b-e64f-4c26-9e5d-e4c2c251c2ca-home\") pod \"e2e-lifecycle-model-31899000-kserve-5448f45ff-28746\" (UID: \"98ac1d0b-e64f-4c26-9e5d-e4c2c251c2ca\") " pod="ai-tenant-e2e-mt-31899000/e2e-lifecycle-model-31899000-kserve-5448f45ff-28746" Jul 10 01:00:12.647781 ip-10-0-130-146 kubenswrapper[2570]: I0710 01:00:12.647703 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/98ac1d0b-e64f-4c26-9e5d-e4c2c251c2ca-model-cache\") pod \"e2e-lifecycle-model-31899000-kserve-5448f45ff-28746\" (UID: \"98ac1d0b-e64f-4c26-9e5d-e4c2c251c2ca\") " pod="ai-tenant-e2e-mt-31899000/e2e-lifecycle-model-31899000-kserve-5448f45ff-28746" Jul 10 01:00:12.647824 ip-10-0-130-146 kubenswrapper[2570]: I0710 01:00:12.647776 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/98ac1d0b-e64f-4c26-9e5d-e4c2c251c2ca-tmp-dir\") pod \"e2e-lifecycle-model-31899000-kserve-5448f45ff-28746\" (UID: \"98ac1d0b-e64f-4c26-9e5d-e4c2c251c2ca\") " pod="ai-tenant-e2e-mt-31899000/e2e-lifecycle-model-31899000-kserve-5448f45ff-28746" Jul 10 01:00:12.649606 ip-10-0-130-146 kubenswrapper[2570]: I0710 01:00:12.649589 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/98ac1d0b-e64f-4c26-9e5d-e4c2c251c2ca-dshm\") pod \"e2e-lifecycle-model-31899000-kserve-5448f45ff-28746\" (UID: \"98ac1d0b-e64f-4c26-9e5d-e4c2c251c2ca\") " pod="ai-tenant-e2e-mt-31899000/e2e-lifecycle-model-31899000-kserve-5448f45ff-28746" Jul 10 01:00:12.649865 ip-10-0-130-146 kubenswrapper[2570]: I0710 01:00:12.649850 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/98ac1d0b-e64f-4c26-9e5d-e4c2c251c2ca-tls-certs\") pod \"e2e-lifecycle-model-31899000-kserve-5448f45ff-28746\" (UID: \"98ac1d0b-e64f-4c26-9e5d-e4c2c251c2ca\") " pod="ai-tenant-e2e-mt-31899000/e2e-lifecycle-model-31899000-kserve-5448f45ff-28746" Jul 10 01:00:12.660409 ip-10-0-130-146 kubenswrapper[2570]: I0710 01:00:12.660384 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8s27f\" (UniqueName: \"kubernetes.io/projected/98ac1d0b-e64f-4c26-9e5d-e4c2c251c2ca-kube-api-access-8s27f\") pod \"e2e-lifecycle-model-31899000-kserve-5448f45ff-28746\" (UID: \"98ac1d0b-e64f-4c26-9e5d-e4c2c251c2ca\") " pod="ai-tenant-e2e-mt-31899000/e2e-lifecycle-model-31899000-kserve-5448f45ff-28746" Jul 10 01:00:12.791582 ip-10-0-130-146 kubenswrapper[2570]: I0710 01:00:12.791501 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-mt-31899000/e2e-lifecycle-model-31899000-kserve-5448f45ff-28746" Jul 10 01:00:12.921456 ip-10-0-130-146 kubenswrapper[2570]: I0710 01:00:12.921424 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-mt-31899000/e2e-lifecycle-model-31899000-kserve-5448f45ff-28746"] Jul 10 01:00:12.922215 ip-10-0-130-146 kubenswrapper[2570]: W0710 01:00:12.922192 2570 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod98ac1d0b_e64f_4c26_9e5d_e4c2c251c2ca.slice/crio-a666ff65c79f6128116575f7eb1c930aa10a79acc792796fb73a67f111c896fa WatchSource:0}: Error finding container a666ff65c79f6128116575f7eb1c930aa10a79acc792796fb73a67f111c896fa: Status 404 returned error can't find the container with id a666ff65c79f6128116575f7eb1c930aa10a79acc792796fb73a67f111c896fa Jul 10 01:00:13.406687 ip-10-0-130-146 kubenswrapper[2570]: I0710 01:00:13.406654 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-mt-31899000/e2e-lifecycle-model-31899000-kserve-5448f45ff-28746" event={"ID":"98ac1d0b-e64f-4c26-9e5d-e4c2c251c2ca","Type":"ContainerStarted","Data":"c72ae870a0512d2763088c0d8135c06d23cde0582bb41d489d1ee3583bd62585"} Jul 10 01:00:13.406869 ip-10-0-130-146 kubenswrapper[2570]: I0710 01:00:13.406697 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-mt-31899000/e2e-lifecycle-model-31899000-kserve-5448f45ff-28746" event={"ID":"98ac1d0b-e64f-4c26-9e5d-e4c2c251c2ca","Type":"ContainerStarted","Data":"a666ff65c79f6128116575f7eb1c930aa10a79acc792796fb73a67f111c896fa"} Jul 10 01:00:13.423955 ip-10-0-130-146 kubenswrapper[2570]: I0710 01:00:13.423909 2570 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-mt-31899000/e2e-lifecycle-model-31899000-kserve-5448f45ff-28746" podStartSLOduration=1.423893978 podStartE2EDuration="1.423893978s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-10 01:00:12 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 01:00:13.421597708 +0000 UTC m=+2032.469976441" watchObservedRunningTime="2026-07-10 01:00:13.423893978 +0000 UTC m=+2032.472272700" Jul 10 01:00:16.283274 ip-10-0-130-146 kubenswrapper[2570]: I0710 01:00:16.283239 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-g92hs"] Jul 10 01:00:22.792532 ip-10-0-130-146 kubenswrapper[2570]: I0710 01:00:22.792492 2570 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-mt-31899000/e2e-lifecycle-model-31899000-kserve-5448f45ff-28746" Jul 10 01:00:22.792532 ip-10-0-130-146 kubenswrapper[2570]: I0710 01:00:22.792540 2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-mt-31899000/e2e-lifecycle-model-31899000-kserve-5448f45ff-28746" Jul 10 01:00:22.804891 ip-10-0-130-146 kubenswrapper[2570]: I0710 01:00:22.804865 2570 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-mt-31899000/e2e-lifecycle-model-31899000-kserve-5448f45ff-28746" Jul 10 01:00:23.476496 ip-10-0-130-146 kubenswrapper[2570]: I0710 01:00:23.476467 2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-mt-31899000/e2e-lifecycle-model-31899000-kserve-5448f45ff-28746" Jul 10 01:00:32.385293 ip-10-0-130-146 kubenswrapper[2570]: I0710 01:00:32.385253 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-g92hs"] Jul 10 01:00:45.579212 ip-10-0-130-146 kubenswrapper[2570]: I0710 01:00:45.579117 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-g92hs"] Jul 10 01:00:45.826042 ip-10-0-130-146 kubenswrapper[2570]: I0710 01:00:45.826008 2570 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-mt-31899000/e2e-lifecycle-model-31899000-kserve-5448f45ff-28746"] Jul 10 01:01:00.010566 ip-10-0-130-146 kubenswrapper[2570]: I0710 01:01:00.010531 2570 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29727420-hjhd8"] Jul 10 01:01:00.011040 ip-10-0-130-146 kubenswrapper[2570]: I0710 01:01:00.010719 2570 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29727420-hjhd8" podUID="ee00c80e-95c7-4212-aefa-94ddb355bc19" containerName="cleanup" containerID="cri-o://5718b7ce673dc1879243d8c3d6b313986db05153741369c123a11ea6331960be" gracePeriod=30 Jul 10 01:01:00.847351 ip-10-0-130-146 kubenswrapper[2570]: I0710 01:01:00.847265 2570 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-mt-31899000/e2e-lifecycle-model-31899000-kserve-5448f45ff-28746" podUID="98ac1d0b-e64f-4c26-9e5d-e4c2c251c2ca" containerName="main" containerID="cri-o://c72ae870a0512d2763088c0d8135c06d23cde0582bb41d489d1ee3583bd62585" gracePeriod=45 Jul 10 01:01:01.095952 ip-10-0-130-146 kubenswrapper[2570]: I0710 01:01:01.095929 2570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-mt-31899000/e2e-lifecycle-model-31899000-kserve-5448f45ff-28746" Jul 10 01:01:01.177910 ip-10-0-130-146 kubenswrapper[2570]: I0710 01:01:01.177879 2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/98ac1d0b-e64f-4c26-9e5d-e4c2c251c2ca-model-cache\") pod \"98ac1d0b-e64f-4c26-9e5d-e4c2c251c2ca\" (UID: \"98ac1d0b-e64f-4c26-9e5d-e4c2c251c2ca\") " Jul 10 01:01:01.178100 ip-10-0-130-146 kubenswrapper[2570]: I0710 01:01:01.177933 2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/98ac1d0b-e64f-4c26-9e5d-e4c2c251c2ca-dshm\") pod \"98ac1d0b-e64f-4c26-9e5d-e4c2c251c2ca\" (UID: \"98ac1d0b-e64f-4c26-9e5d-e4c2c251c2ca\") " Jul 10 01:01:01.178100 ip-10-0-130-146 kubenswrapper[2570]: I0710 01:01:01.177968 2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-8s27f\" (UniqueName: \"kubernetes.io/projected/98ac1d0b-e64f-4c26-9e5d-e4c2c251c2ca-kube-api-access-8s27f\") pod \"98ac1d0b-e64f-4c26-9e5d-e4c2c251c2ca\" (UID: \"98ac1d0b-e64f-4c26-9e5d-e4c2c251c2ca\") " Jul 10 01:01:01.178100 ip-10-0-130-146 kubenswrapper[2570]: I0710 01:01:01.177989 2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/98ac1d0b-e64f-4c26-9e5d-e4c2c251c2ca-tls-certs\") pod \"98ac1d0b-e64f-4c26-9e5d-e4c2c251c2ca\" (UID: \"98ac1d0b-e64f-4c26-9e5d-e4c2c251c2ca\") " Jul 10 01:01:01.178100 ip-10-0-130-146 kubenswrapper[2570]: I0710 01:01:01.178003 2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/98ac1d0b-e64f-4c26-9e5d-e4c2c251c2ca-tmp-dir\") pod \"98ac1d0b-e64f-4c26-9e5d-e4c2c251c2ca\" (UID: \"98ac1d0b-e64f-4c26-9e5d-e4c2c251c2ca\") " Jul 10 01:01:01.178100 ip-10-0-130-146 kubenswrapper[2570]: I0710 01:01:01.178077 2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/98ac1d0b-e64f-4c26-9e5d-e4c2c251c2ca-home\") pod \"98ac1d0b-e64f-4c26-9e5d-e4c2c251c2ca\" (UID: \"98ac1d0b-e64f-4c26-9e5d-e4c2c251c2ca\") " Jul 10 01:01:01.178379 ip-10-0-130-146 kubenswrapper[2570]: I0710 01:01:01.178144 2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/98ac1d0b-e64f-4c26-9e5d-e4c2c251c2ca-model-cache" (OuterVolumeSpecName: "model-cache") pod "98ac1d0b-e64f-4c26-9e5d-e4c2c251c2ca" (UID: "98ac1d0b-e64f-4c26-9e5d-e4c2c251c2ca"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 10 01:01:01.178379 ip-10-0-130-146 kubenswrapper[2570]: I0710 01:01:01.178303 2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/98ac1d0b-e64f-4c26-9e5d-e4c2c251c2ca-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "98ac1d0b-e64f-4c26-9e5d-e4c2c251c2ca" (UID: "98ac1d0b-e64f-4c26-9e5d-e4c2c251c2ca"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 10 01:01:01.178379 ip-10-0-130-146 kubenswrapper[2570]: I0710 01:01:01.178330 2570 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/98ac1d0b-e64f-4c26-9e5d-e4c2c251c2ca-model-cache\") on node \"ip-10-0-130-146.ec2.internal\" DevicePath \"\"" Jul 10 01:01:01.178508 ip-10-0-130-146 kubenswrapper[2570]: I0710 01:01:01.178412 2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/98ac1d0b-e64f-4c26-9e5d-e4c2c251c2ca-home" (OuterVolumeSpecName: "home") pod "98ac1d0b-e64f-4c26-9e5d-e4c2c251c2ca" (UID: "98ac1d0b-e64f-4c26-9e5d-e4c2c251c2ca"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 10 01:01:01.180271 ip-10-0-130-146 kubenswrapper[2570]: I0710 01:01:01.180246 2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/98ac1d0b-e64f-4c26-9e5d-e4c2c251c2ca-dshm" (OuterVolumeSpecName: "dshm") pod "98ac1d0b-e64f-4c26-9e5d-e4c2c251c2ca" (UID: "98ac1d0b-e64f-4c26-9e5d-e4c2c251c2ca"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 10 01:01:01.180371 ip-10-0-130-146 kubenswrapper[2570]: I0710 01:01:01.180281 2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/98ac1d0b-e64f-4c26-9e5d-e4c2c251c2ca-kube-api-access-8s27f" (OuterVolumeSpecName: "kube-api-access-8s27f") pod "98ac1d0b-e64f-4c26-9e5d-e4c2c251c2ca" (UID: "98ac1d0b-e64f-4c26-9e5d-e4c2c251c2ca"). InnerVolumeSpecName "kube-api-access-8s27f". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 10 01:01:01.180371 ip-10-0-130-146 kubenswrapper[2570]: I0710 01:01:01.180302 2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/98ac1d0b-e64f-4c26-9e5d-e4c2c251c2ca-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "98ac1d0b-e64f-4c26-9e5d-e4c2c251c2ca" (UID: "98ac1d0b-e64f-4c26-9e5d-e4c2c251c2ca"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 10 01:01:01.279335 ip-10-0-130-146 kubenswrapper[2570]: I0710 01:01:01.279304 2570 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/98ac1d0b-e64f-4c26-9e5d-e4c2c251c2ca-dshm\") on node \"ip-10-0-130-146.ec2.internal\" DevicePath \"\"" Jul 10 01:01:01.279335 ip-10-0-130-146 kubenswrapper[2570]: I0710 01:01:01.279334 2570 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8s27f\" (UniqueName: \"kubernetes.io/projected/98ac1d0b-e64f-4c26-9e5d-e4c2c251c2ca-kube-api-access-8s27f\") on node \"ip-10-0-130-146.ec2.internal\" DevicePath \"\"" Jul 10 01:01:01.279545 ip-10-0-130-146 kubenswrapper[2570]: I0710 01:01:01.279346 2570 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/98ac1d0b-e64f-4c26-9e5d-e4c2c251c2ca-tls-certs\") on node \"ip-10-0-130-146.ec2.internal\" DevicePath \"\"" Jul 10 01:01:01.279545 ip-10-0-130-146 kubenswrapper[2570]: I0710 01:01:01.279355 2570 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/98ac1d0b-e64f-4c26-9e5d-e4c2c251c2ca-tmp-dir\") on node \"ip-10-0-130-146.ec2.internal\" DevicePath \"\"" Jul 10 01:01:01.279545 ip-10-0-130-146 kubenswrapper[2570]: I0710 01:01:01.279365 2570 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/98ac1d0b-e64f-4c26-9e5d-e4c2c251c2ca-home\") on node \"ip-10-0-130-146.ec2.internal\" DevicePath \"\"" Jul 10 01:01:01.668106 ip-10-0-130-146 kubenswrapper[2570]: I0710 01:01:01.668043 2570 generic.go:356] "Generic (PLEG): container finished" podID="98ac1d0b-e64f-4c26-9e5d-e4c2c251c2ca" containerID="c72ae870a0512d2763088c0d8135c06d23cde0582bb41d489d1ee3583bd62585" exitCode=0 Jul 10 01:01:01.668106 ip-10-0-130-146 kubenswrapper[2570]: I0710 01:01:01.668105 2570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-mt-31899000/e2e-lifecycle-model-31899000-kserve-5448f45ff-28746" Jul 10 01:01:01.668333 ip-10-0-130-146 kubenswrapper[2570]: I0710 01:01:01.668121 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-mt-31899000/e2e-lifecycle-model-31899000-kserve-5448f45ff-28746" event={"ID":"98ac1d0b-e64f-4c26-9e5d-e4c2c251c2ca","Type":"ContainerDied","Data":"c72ae870a0512d2763088c0d8135c06d23cde0582bb41d489d1ee3583bd62585"} Jul 10 01:01:01.668333 ip-10-0-130-146 kubenswrapper[2570]: I0710 01:01:01.668152 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-mt-31899000/e2e-lifecycle-model-31899000-kserve-5448f45ff-28746" event={"ID":"98ac1d0b-e64f-4c26-9e5d-e4c2c251c2ca","Type":"ContainerDied","Data":"a666ff65c79f6128116575f7eb1c930aa10a79acc792796fb73a67f111c896fa"} Jul 10 01:01:01.668333 ip-10-0-130-146 kubenswrapper[2570]: I0710 01:01:01.668168 2570 scope.go:117] "RemoveContainer" containerID="c72ae870a0512d2763088c0d8135c06d23cde0582bb41d489d1ee3583bd62585" Jul 10 01:01:01.677035 ip-10-0-130-146 kubenswrapper[2570]: I0710 01:01:01.677013 2570 scope.go:117] "RemoveContainer" containerID="c72ae870a0512d2763088c0d8135c06d23cde0582bb41d489d1ee3583bd62585" Jul 10 01:01:01.677296 ip-10-0-130-146 kubenswrapper[2570]: E0710 01:01:01.677279 2570 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c72ae870a0512d2763088c0d8135c06d23cde0582bb41d489d1ee3583bd62585\": container with ID starting with c72ae870a0512d2763088c0d8135c06d23cde0582bb41d489d1ee3583bd62585 not found: ID does not exist" containerID="c72ae870a0512d2763088c0d8135c06d23cde0582bb41d489d1ee3583bd62585" Jul 10 01:01:01.677352 ip-10-0-130-146 kubenswrapper[2570]: I0710 01:01:01.677306 2570 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c72ae870a0512d2763088c0d8135c06d23cde0582bb41d489d1ee3583bd62585"} err="failed to get container status \"c72ae870a0512d2763088c0d8135c06d23cde0582bb41d489d1ee3583bd62585\": rpc error: code = NotFound desc = could not find container \"c72ae870a0512d2763088c0d8135c06d23cde0582bb41d489d1ee3583bd62585\": container with ID starting with c72ae870a0512d2763088c0d8135c06d23cde0582bb41d489d1ee3583bd62585 not found: ID does not exist" Jul 10 01:01:01.685504 ip-10-0-130-146 kubenswrapper[2570]: I0710 01:01:01.685482 2570 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-mt-31899000/e2e-lifecycle-model-31899000-kserve-5448f45ff-28746"] Jul 10 01:01:01.690624 ip-10-0-130-146 kubenswrapper[2570]: I0710 01:01:01.690602 2570 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-mt-31899000/e2e-lifecycle-model-31899000-kserve-5448f45ff-28746"] Jul 10 01:01:03.453922 ip-10-0-130-146 kubenswrapper[2570]: I0710 01:01:03.453887 2570 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="98ac1d0b-e64f-4c26-9e5d-e4c2c251c2ca" path="/var/lib/kubelet/pods/98ac1d0b-e64f-4c26-9e5d-e4c2c251c2ca/volumes" Jul 10 01:01:21.489850 ip-10-0-130-146 kubenswrapper[2570]: I0710 01:01:21.489823 2570 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7866497494-d6fcd_2b6cb4ca-79ff-42f0-b080-66af842f22d3/console-operator/2.log" Jul 10 01:01:21.492866 ip-10-0-130-146 kubenswrapper[2570]: I0710 01:01:21.492845 2570 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7866497494-d6fcd_2b6cb4ca-79ff-42f0-b080-66af842f22d3/console-operator/2.log" Jul 10 01:01:21.493009 ip-10-0-130-146 kubenswrapper[2570]: I0710 01:01:21.492946 2570 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-mbs58_c4ef019a-364a-4933-ab85-de86d86167f1/ovn-acl-logging/0.log" Jul 10 01:01:21.496102 ip-10-0-130-146 kubenswrapper[2570]: I0710 01:01:21.496085 2570 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-mbs58_c4ef019a-364a-4933-ab85-de86d86167f1/ovn-acl-logging/0.log" Jul 10 01:01:24.987171 ip-10-0-130-146 kubenswrapper[2570]: I0710 01:01:24.987136 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-g92hs"] Jul 10 01:01:30.191076 ip-10-0-130-146 kubenswrapper[2570]: I0710 01:01:30.191027 2570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29727420-hjhd8" Jul 10 01:01:30.829708 ip-10-0-130-146 kubenswrapper[2570]: I0710 01:01:30.829675 2570 generic.go:356] "Generic (PLEG): container finished" podID="ee00c80e-95c7-4212-aefa-94ddb355bc19" containerID="5718b7ce673dc1879243d8c3d6b313986db05153741369c123a11ea6331960be" exitCode=137 Jul 10 01:01:30.829900 ip-10-0-130-146 kubenswrapper[2570]: I0710 01:01:30.829725 2570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29727420-hjhd8" Jul 10 01:01:30.829900 ip-10-0-130-146 kubenswrapper[2570]: I0710 01:01:30.829724 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29727420-hjhd8" event={"ID":"ee00c80e-95c7-4212-aefa-94ddb355bc19","Type":"ContainerDied","Data":"5718b7ce673dc1879243d8c3d6b313986db05153741369c123a11ea6331960be"} Jul 10 01:01:30.829900 ip-10-0-130-146 kubenswrapper[2570]: I0710 01:01:30.829846 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29727420-hjhd8" event={"ID":"ee00c80e-95c7-4212-aefa-94ddb355bc19","Type":"ContainerDied","Data":"a27a35c96b55844600624b9ee05bac94b8012ecef7bc7997b8ed398f5604e9fa"} Jul 10 01:01:30.829900 ip-10-0-130-146 kubenswrapper[2570]: I0710 01:01:30.829870 2570 scope.go:117] "RemoveContainer" containerID="5718b7ce673dc1879243d8c3d6b313986db05153741369c123a11ea6331960be" Jul 10 01:01:30.839211 ip-10-0-130-146 kubenswrapper[2570]: I0710 01:01:30.839196 2570 scope.go:117] "RemoveContainer" containerID="5718b7ce673dc1879243d8c3d6b313986db05153741369c123a11ea6331960be" Jul 10 01:01:30.839481 ip-10-0-130-146 kubenswrapper[2570]: E0710 01:01:30.839464 2570 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5718b7ce673dc1879243d8c3d6b313986db05153741369c123a11ea6331960be\": container with ID starting with 5718b7ce673dc1879243d8c3d6b313986db05153741369c123a11ea6331960be not found: ID does not exist" containerID="5718b7ce673dc1879243d8c3d6b313986db05153741369c123a11ea6331960be" Jul 10 01:01:30.839524 ip-10-0-130-146 kubenswrapper[2570]: I0710 01:01:30.839490 2570 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5718b7ce673dc1879243d8c3d6b313986db05153741369c123a11ea6331960be"} err="failed to get container status \"5718b7ce673dc1879243d8c3d6b313986db05153741369c123a11ea6331960be\": rpc error: code = NotFound desc = could not find container \"5718b7ce673dc1879243d8c3d6b313986db05153741369c123a11ea6331960be\": container with ID starting with 5718b7ce673dc1879243d8c3d6b313986db05153741369c123a11ea6331960be not found: ID does not exist" Jul 10 01:01:30.851634 ip-10-0-130-146 kubenswrapper[2570]: I0710 01:01:30.851611 2570 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29727420-hjhd8"] Jul 10 01:01:30.855027 ip-10-0-130-146 kubenswrapper[2570]: I0710 01:01:30.855006 2570 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29727420-hjhd8"] Jul 10 01:01:31.454088 ip-10-0-130-146 kubenswrapper[2570]: I0710 01:01:31.454032 2570 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="ee00c80e-95c7-4212-aefa-94ddb355bc19" path="/var/lib/kubelet/pods/ee00c80e-95c7-4212-aefa-94ddb355bc19/volumes" Jul 10 01:02:33.680348 ip-10-0-130-146 kubenswrapper[2570]: I0710 01:02:33.680312 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-g92hs"] Jul 10 01:02:45.976634 ip-10-0-130-146 kubenswrapper[2570]: I0710 01:02:45.976597 2570 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-inf-a-85dfb6/test-model-85dfb6-kserve-978dcf4dc-hn6sc"] Jul 10 01:02:45.977011 ip-10-0-130-146 kubenswrapper[2570]: I0710 01:02:45.976941 2570 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="98ac1d0b-e64f-4c26-9e5d-e4c2c251c2ca" containerName="main" Jul 10 01:02:45.977011 ip-10-0-130-146 kubenswrapper[2570]: I0710 01:02:45.976951 2570 state_mem.go:107] "Deleted CPUSet assignment" podUID="98ac1d0b-e64f-4c26-9e5d-e4c2c251c2ca" containerName="main" Jul 10 01:02:45.977011 ip-10-0-130-146 kubenswrapper[2570]: I0710 01:02:45.976974 2570 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ee00c80e-95c7-4212-aefa-94ddb355bc19" containerName="cleanup" Jul 10 01:02:45.977011 ip-10-0-130-146 kubenswrapper[2570]: I0710 01:02:45.976979 2570 state_mem.go:107] "Deleted CPUSet assignment" podUID="ee00c80e-95c7-4212-aefa-94ddb355bc19" containerName="cleanup" Jul 10 01:02:45.977171 ip-10-0-130-146 kubenswrapper[2570]: I0710 01:02:45.977036 2570 memory_manager.go:368] "RemoveStaleState removing state" podUID="ee00c80e-95c7-4212-aefa-94ddb355bc19" containerName="cleanup" Jul 10 01:02:45.977171 ip-10-0-130-146 kubenswrapper[2570]: I0710 01:02:45.977060 2570 memory_manager.go:368] "RemoveStaleState removing state" podUID="98ac1d0b-e64f-4c26-9e5d-e4c2c251c2ca" containerName="main" Jul 10 01:02:45.980110 ip-10-0-130-146 kubenswrapper[2570]: I0710 01:02:45.980093 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-85dfb6/test-model-85dfb6-kserve-978dcf4dc-hn6sc" Jul 10 01:02:45.982955 ip-10-0-130-146 kubenswrapper[2570]: I0710 01:02:45.982934 2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-a-85dfb6\"/\"kube-root-ca.crt\"" Jul 10 01:02:45.984011 ip-10-0-130-146 kubenswrapper[2570]: I0710 01:02:45.983989 2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-a-85dfb6\"/\"default-dockercfg-sfsw6\"" Jul 10 01:02:45.984155 ip-10-0-130-146 kubenswrapper[2570]: I0710 01:02:45.984008 2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-a-85dfb6\"/\"test-model-85dfb6-kserve-self-signed-certs\"" Jul 10 01:02:45.984155 ip-10-0-130-146 kubenswrapper[2570]: I0710 01:02:45.984067 2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-a-85dfb6\"/\"openshift-service-ca.crt\"" Jul 10 01:02:45.987836 ip-10-0-130-146 kubenswrapper[2570]: I0710 01:02:45.987812 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-a-85dfb6/test-model-85dfb6-kserve-978dcf4dc-hn6sc"] Jul 10 01:02:46.071734 ip-10-0-130-146 kubenswrapper[2570]: I0710 01:02:46.071698 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1b17c212-ada7-4589-97dc-9f898842f3ec-dshm\") pod \"test-model-85dfb6-kserve-978dcf4dc-hn6sc\" (UID: \"1b17c212-ada7-4589-97dc-9f898842f3ec\") " pod="ai-tenant-e2e-inf-a-85dfb6/test-model-85dfb6-kserve-978dcf4dc-hn6sc" Jul 10 01:02:46.071911 ip-10-0-130-146 kubenswrapper[2570]: I0710 01:02:46.071740 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1b17c212-ada7-4589-97dc-9f898842f3ec-tmp-dir\") pod \"test-model-85dfb6-kserve-978dcf4dc-hn6sc\" (UID: \"1b17c212-ada7-4589-97dc-9f898842f3ec\") " pod="ai-tenant-e2e-inf-a-85dfb6/test-model-85dfb6-kserve-978dcf4dc-hn6sc" Jul 10 01:02:46.071911 ip-10-0-130-146 kubenswrapper[2570]: I0710 01:02:46.071778 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1b17c212-ada7-4589-97dc-9f898842f3ec-model-cache\") pod \"test-model-85dfb6-kserve-978dcf4dc-hn6sc\" (UID: \"1b17c212-ada7-4589-97dc-9f898842f3ec\") " pod="ai-tenant-e2e-inf-a-85dfb6/test-model-85dfb6-kserve-978dcf4dc-hn6sc" Jul 10 01:02:46.071911 ip-10-0-130-146 kubenswrapper[2570]: I0710 01:02:46.071844 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1b17c212-ada7-4589-97dc-9f898842f3ec-home\") pod \"test-model-85dfb6-kserve-978dcf4dc-hn6sc\" (UID: \"1b17c212-ada7-4589-97dc-9f898842f3ec\") " pod="ai-tenant-e2e-inf-a-85dfb6/test-model-85dfb6-kserve-978dcf4dc-hn6sc" Jul 10 01:02:46.071911 ip-10-0-130-146 kubenswrapper[2570]: I0710 01:02:46.071869 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1b17c212-ada7-4589-97dc-9f898842f3ec-tls-certs\") pod \"test-model-85dfb6-kserve-978dcf4dc-hn6sc\" (UID: \"1b17c212-ada7-4589-97dc-9f898842f3ec\") " pod="ai-tenant-e2e-inf-a-85dfb6/test-model-85dfb6-kserve-978dcf4dc-hn6sc" Jul 10 01:02:46.072102 ip-10-0-130-146 kubenswrapper[2570]: I0710 01:02:46.071949 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ww2f6\" (UniqueName: \"kubernetes.io/projected/1b17c212-ada7-4589-97dc-9f898842f3ec-kube-api-access-ww2f6\") pod \"test-model-85dfb6-kserve-978dcf4dc-hn6sc\" (UID: \"1b17c212-ada7-4589-97dc-9f898842f3ec\") " pod="ai-tenant-e2e-inf-a-85dfb6/test-model-85dfb6-kserve-978dcf4dc-hn6sc" Jul 10 01:02:46.172932 ip-10-0-130-146 kubenswrapper[2570]: I0710 01:02:46.172885 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1b17c212-ada7-4589-97dc-9f898842f3ec-home\") pod \"test-model-85dfb6-kserve-978dcf4dc-hn6sc\" (UID: \"1b17c212-ada7-4589-97dc-9f898842f3ec\") " pod="ai-tenant-e2e-inf-a-85dfb6/test-model-85dfb6-kserve-978dcf4dc-hn6sc" Jul 10 01:02:46.172932 ip-10-0-130-146 kubenswrapper[2570]: I0710 01:02:46.172935 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1b17c212-ada7-4589-97dc-9f898842f3ec-tls-certs\") pod \"test-model-85dfb6-kserve-978dcf4dc-hn6sc\" (UID: \"1b17c212-ada7-4589-97dc-9f898842f3ec\") " pod="ai-tenant-e2e-inf-a-85dfb6/test-model-85dfb6-kserve-978dcf4dc-hn6sc" Jul 10 01:02:46.173238 ip-10-0-130-146 kubenswrapper[2570]: I0710 01:02:46.172993 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ww2f6\" (UniqueName: \"kubernetes.io/projected/1b17c212-ada7-4589-97dc-9f898842f3ec-kube-api-access-ww2f6\") pod \"test-model-85dfb6-kserve-978dcf4dc-hn6sc\" (UID: \"1b17c212-ada7-4589-97dc-9f898842f3ec\") " pod="ai-tenant-e2e-inf-a-85dfb6/test-model-85dfb6-kserve-978dcf4dc-hn6sc" Jul 10 01:02:46.173238 ip-10-0-130-146 kubenswrapper[2570]: I0710 01:02:46.173041 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1b17c212-ada7-4589-97dc-9f898842f3ec-dshm\") pod \"test-model-85dfb6-kserve-978dcf4dc-hn6sc\" (UID: \"1b17c212-ada7-4589-97dc-9f898842f3ec\") " pod="ai-tenant-e2e-inf-a-85dfb6/test-model-85dfb6-kserve-978dcf4dc-hn6sc" Jul 10 01:02:46.173238 ip-10-0-130-146 kubenswrapper[2570]: I0710 01:02:46.173100 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1b17c212-ada7-4589-97dc-9f898842f3ec-tmp-dir\") pod \"test-model-85dfb6-kserve-978dcf4dc-hn6sc\" (UID: \"1b17c212-ada7-4589-97dc-9f898842f3ec\") " pod="ai-tenant-e2e-inf-a-85dfb6/test-model-85dfb6-kserve-978dcf4dc-hn6sc" Jul 10 01:02:46.173238 ip-10-0-130-146 kubenswrapper[2570]: I0710 01:02:46.173128 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1b17c212-ada7-4589-97dc-9f898842f3ec-model-cache\") pod \"test-model-85dfb6-kserve-978dcf4dc-hn6sc\" (UID: \"1b17c212-ada7-4589-97dc-9f898842f3ec\") " pod="ai-tenant-e2e-inf-a-85dfb6/test-model-85dfb6-kserve-978dcf4dc-hn6sc" Jul 10 01:02:46.173435 ip-10-0-130-146 kubenswrapper[2570]: I0710 01:02:46.173345 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1b17c212-ada7-4589-97dc-9f898842f3ec-home\") pod \"test-model-85dfb6-kserve-978dcf4dc-hn6sc\" (UID: \"1b17c212-ada7-4589-97dc-9f898842f3ec\") " pod="ai-tenant-e2e-inf-a-85dfb6/test-model-85dfb6-kserve-978dcf4dc-hn6sc" Jul 10 01:02:46.173496 ip-10-0-130-146 kubenswrapper[2570]: I0710 01:02:46.173448 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1b17c212-ada7-4589-97dc-9f898842f3ec-tmp-dir\") pod \"test-model-85dfb6-kserve-978dcf4dc-hn6sc\" (UID: \"1b17c212-ada7-4589-97dc-9f898842f3ec\") " pod="ai-tenant-e2e-inf-a-85dfb6/test-model-85dfb6-kserve-978dcf4dc-hn6sc" Jul 10 01:02:46.173549 ip-10-0-130-146 kubenswrapper[2570]: I0710 01:02:46.173514 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1b17c212-ada7-4589-97dc-9f898842f3ec-model-cache\") pod \"test-model-85dfb6-kserve-978dcf4dc-hn6sc\" (UID: \"1b17c212-ada7-4589-97dc-9f898842f3ec\") " pod="ai-tenant-e2e-inf-a-85dfb6/test-model-85dfb6-kserve-978dcf4dc-hn6sc" Jul 10 01:02:46.175233 ip-10-0-130-146 kubenswrapper[2570]: I0710 01:02:46.175215 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1b17c212-ada7-4589-97dc-9f898842f3ec-dshm\") pod \"test-model-85dfb6-kserve-978dcf4dc-hn6sc\" (UID: \"1b17c212-ada7-4589-97dc-9f898842f3ec\") " pod="ai-tenant-e2e-inf-a-85dfb6/test-model-85dfb6-kserve-978dcf4dc-hn6sc" Jul 10 01:02:46.175497 ip-10-0-130-146 kubenswrapper[2570]: I0710 01:02:46.175481 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1b17c212-ada7-4589-97dc-9f898842f3ec-tls-certs\") pod \"test-model-85dfb6-kserve-978dcf4dc-hn6sc\" (UID: \"1b17c212-ada7-4589-97dc-9f898842f3ec\") " pod="ai-tenant-e2e-inf-a-85dfb6/test-model-85dfb6-kserve-978dcf4dc-hn6sc" Jul 10 01:02:46.181874 ip-10-0-130-146 kubenswrapper[2570]: I0710 01:02:46.181851 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ww2f6\" (UniqueName: \"kubernetes.io/projected/1b17c212-ada7-4589-97dc-9f898842f3ec-kube-api-access-ww2f6\") pod \"test-model-85dfb6-kserve-978dcf4dc-hn6sc\" (UID: \"1b17c212-ada7-4589-97dc-9f898842f3ec\") " pod="ai-tenant-e2e-inf-a-85dfb6/test-model-85dfb6-kserve-978dcf4dc-hn6sc" Jul 10 01:02:46.290908 ip-10-0-130-146 kubenswrapper[2570]: I0710 01:02:46.290834 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-85dfb6/test-model-85dfb6-kserve-978dcf4dc-hn6sc" Jul 10 01:02:46.424086 ip-10-0-130-146 kubenswrapper[2570]: I0710 01:02:46.424041 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-a-85dfb6/test-model-85dfb6-kserve-978dcf4dc-hn6sc"] Jul 10 01:02:46.424854 ip-10-0-130-146 kubenswrapper[2570]: W0710 01:02:46.424824 2570 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod1b17c212_ada7_4589_97dc_9f898842f3ec.slice/crio-7da633217f96c47be72143c348c82048dc56cde376a208fb2f7c37d64c3edd9d WatchSource:0}: Error finding container 7da633217f96c47be72143c348c82048dc56cde376a208fb2f7c37d64c3edd9d: Status 404 returned error can't find the container with id 7da633217f96c47be72143c348c82048dc56cde376a208fb2f7c37d64c3edd9d Jul 10 01:02:47.235140 ip-10-0-130-146 kubenswrapper[2570]: I0710 01:02:47.235105 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-85dfb6/test-model-85dfb6-kserve-978dcf4dc-hn6sc" event={"ID":"1b17c212-ada7-4589-97dc-9f898842f3ec","Type":"ContainerStarted","Data":"8f39cd6569297cb39e9dc25cab4b99187d2c85c7b897c236cb608b2ff01d1313"} Jul 10 01:02:47.235522 ip-10-0-130-146 kubenswrapper[2570]: I0710 01:02:47.235148 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-85dfb6/test-model-85dfb6-kserve-978dcf4dc-hn6sc" event={"ID":"1b17c212-ada7-4589-97dc-9f898842f3ec","Type":"ContainerStarted","Data":"7da633217f96c47be72143c348c82048dc56cde376a208fb2f7c37d64c3edd9d"} Jul 10 01:02:47.252467 ip-10-0-130-146 kubenswrapper[2570]: I0710 01:02:47.252415 2570 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-inf-a-85dfb6/test-model-85dfb6-kserve-978dcf4dc-hn6sc" podStartSLOduration=2.252400702 podStartE2EDuration="2.252400702s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-10 01:02:45 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 01:02:47.249960932 +0000 UTC m=+2186.298339653" watchObservedRunningTime="2026-07-10 01:02:47.252400702 +0000 UTC m=+2186.300779423" Jul 10 01:02:49.775979 ip-10-0-130-146 kubenswrapper[2570]: I0710 01:02:49.775948 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-g92hs"] Jul 10 01:02:56.291551 ip-10-0-130-146 kubenswrapper[2570]: I0710 01:02:56.291522 2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-inf-a-85dfb6/test-model-85dfb6-kserve-978dcf4dc-hn6sc" Jul 10 01:02:56.291985 ip-10-0-130-146 kubenswrapper[2570]: I0710 01:02:56.291567 2570 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-inf-a-85dfb6/test-model-85dfb6-kserve-978dcf4dc-hn6sc" Jul 10 01:02:56.303659 ip-10-0-130-146 kubenswrapper[2570]: I0710 01:02:56.303641 2570 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-inf-a-85dfb6/test-model-85dfb6-kserve-978dcf4dc-hn6sc" Jul 10 01:02:57.298276 ip-10-0-130-146 kubenswrapper[2570]: I0710 01:02:57.298226 2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-inf-a-85dfb6/test-model-85dfb6-kserve-978dcf4dc-hn6sc" Jul 10 01:03:05.276241 ip-10-0-130-146 kubenswrapper[2570]: I0710 01:03:05.276201 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-g92hs"] Jul 10 01:03:08.674765 ip-10-0-130-146 kubenswrapper[2570]: I0710 01:03:08.674735 2570 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-inf-b-8f8f47/test-model-8f8f47-kserve-5d5ff9b44c-bprmc"] Jul 10 01:03:08.678479 ip-10-0-130-146 kubenswrapper[2570]: I0710 01:03:08.678458 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-8f8f47/test-model-8f8f47-kserve-5d5ff9b44c-bprmc" Jul 10 01:03:08.682074 ip-10-0-130-146 kubenswrapper[2570]: I0710 01:03:08.682043 2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-b-8f8f47\"/\"default-dockercfg-n4rks\"" Jul 10 01:03:08.682074 ip-10-0-130-146 kubenswrapper[2570]: I0710 01:03:08.682067 2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-b-8f8f47\"/\"test-model-8f8f47-kserve-self-signed-certs\"" Jul 10 01:03:08.682222 ip-10-0-130-146 kubenswrapper[2570]: I0710 01:03:08.682190 2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-b-8f8f47\"/\"openshift-service-ca.crt\"" Jul 10 01:03:08.682262 ip-10-0-130-146 kubenswrapper[2570]: I0710 01:03:08.682229 2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-b-8f8f47\"/\"kube-root-ca.crt\"" Jul 10 01:03:08.688123 ip-10-0-130-146 kubenswrapper[2570]: I0710 01:03:08.688102 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-b-8f8f47/test-model-8f8f47-kserve-5d5ff9b44c-bprmc"] Jul 10 01:03:08.761315 ip-10-0-130-146 kubenswrapper[2570]: I0710 01:03:08.761282 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c4f9d91a-c62c-41bd-bbc2-fed40a121e92-home\") pod \"test-model-8f8f47-kserve-5d5ff9b44c-bprmc\" (UID: \"c4f9d91a-c62c-41bd-bbc2-fed40a121e92\") " pod="ai-tenant-e2e-inf-b-8f8f47/test-model-8f8f47-kserve-5d5ff9b44c-bprmc" Jul 10 01:03:08.761315 ip-10-0-130-146 kubenswrapper[2570]: I0710 01:03:08.761316 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xgm8r\" (UniqueName: \"kubernetes.io/projected/c4f9d91a-c62c-41bd-bbc2-fed40a121e92-kube-api-access-xgm8r\") pod \"test-model-8f8f47-kserve-5d5ff9b44c-bprmc\" (UID: \"c4f9d91a-c62c-41bd-bbc2-fed40a121e92\") " pod="ai-tenant-e2e-inf-b-8f8f47/test-model-8f8f47-kserve-5d5ff9b44c-bprmc" Jul 10 01:03:08.761498 ip-10-0-130-146 kubenswrapper[2570]: I0710 01:03:08.761392 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c4f9d91a-c62c-41bd-bbc2-fed40a121e92-dshm\") pod \"test-model-8f8f47-kserve-5d5ff9b44c-bprmc\" (UID: \"c4f9d91a-c62c-41bd-bbc2-fed40a121e92\") " pod="ai-tenant-e2e-inf-b-8f8f47/test-model-8f8f47-kserve-5d5ff9b44c-bprmc" Jul 10 01:03:08.761498 ip-10-0-130-146 kubenswrapper[2570]: I0710 01:03:08.761422 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c4f9d91a-c62c-41bd-bbc2-fed40a121e92-tls-certs\") pod \"test-model-8f8f47-kserve-5d5ff9b44c-bprmc\" (UID: \"c4f9d91a-c62c-41bd-bbc2-fed40a121e92\") " pod="ai-tenant-e2e-inf-b-8f8f47/test-model-8f8f47-kserve-5d5ff9b44c-bprmc" Jul 10 01:03:08.761565 ip-10-0-130-146 kubenswrapper[2570]: I0710 01:03:08.761499 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c4f9d91a-c62c-41bd-bbc2-fed40a121e92-model-cache\") pod \"test-model-8f8f47-kserve-5d5ff9b44c-bprmc\" (UID: \"c4f9d91a-c62c-41bd-bbc2-fed40a121e92\") " pod="ai-tenant-e2e-inf-b-8f8f47/test-model-8f8f47-kserve-5d5ff9b44c-bprmc" Jul 10 01:03:08.761565 ip-10-0-130-146 kubenswrapper[2570]: I0710 01:03:08.761515 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c4f9d91a-c62c-41bd-bbc2-fed40a121e92-tmp-dir\") pod \"test-model-8f8f47-kserve-5d5ff9b44c-bprmc\" (UID: \"c4f9d91a-c62c-41bd-bbc2-fed40a121e92\") " pod="ai-tenant-e2e-inf-b-8f8f47/test-model-8f8f47-kserve-5d5ff9b44c-bprmc" Jul 10 01:03:08.862063 ip-10-0-130-146 kubenswrapper[2570]: I0710 01:03:08.862015 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c4f9d91a-c62c-41bd-bbc2-fed40a121e92-model-cache\") pod \"test-model-8f8f47-kserve-5d5ff9b44c-bprmc\" (UID: \"c4f9d91a-c62c-41bd-bbc2-fed40a121e92\") " pod="ai-tenant-e2e-inf-b-8f8f47/test-model-8f8f47-kserve-5d5ff9b44c-bprmc" Jul 10 01:03:08.862242 ip-10-0-130-146 kubenswrapper[2570]: I0710 01:03:08.862088 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c4f9d91a-c62c-41bd-bbc2-fed40a121e92-tmp-dir\") pod \"test-model-8f8f47-kserve-5d5ff9b44c-bprmc\" (UID: \"c4f9d91a-c62c-41bd-bbc2-fed40a121e92\") " pod="ai-tenant-e2e-inf-b-8f8f47/test-model-8f8f47-kserve-5d5ff9b44c-bprmc" Jul 10 01:03:08.862242 ip-10-0-130-146 kubenswrapper[2570]: I0710 01:03:08.862119 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c4f9d91a-c62c-41bd-bbc2-fed40a121e92-home\") pod \"test-model-8f8f47-kserve-5d5ff9b44c-bprmc\" (UID: \"c4f9d91a-c62c-41bd-bbc2-fed40a121e92\") " pod="ai-tenant-e2e-inf-b-8f8f47/test-model-8f8f47-kserve-5d5ff9b44c-bprmc" Jul 10 01:03:08.862242 ip-10-0-130-146 kubenswrapper[2570]: I0710 01:03:08.862136 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xgm8r\" (UniqueName: \"kubernetes.io/projected/c4f9d91a-c62c-41bd-bbc2-fed40a121e92-kube-api-access-xgm8r\") pod \"test-model-8f8f47-kserve-5d5ff9b44c-bprmc\" (UID: \"c4f9d91a-c62c-41bd-bbc2-fed40a121e92\") " pod="ai-tenant-e2e-inf-b-8f8f47/test-model-8f8f47-kserve-5d5ff9b44c-bprmc" Jul 10 01:03:08.862242 ip-10-0-130-146 kubenswrapper[2570]: I0710 01:03:08.862180 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c4f9d91a-c62c-41bd-bbc2-fed40a121e92-dshm\") pod \"test-model-8f8f47-kserve-5d5ff9b44c-bprmc\" (UID: \"c4f9d91a-c62c-41bd-bbc2-fed40a121e92\") " pod="ai-tenant-e2e-inf-b-8f8f47/test-model-8f8f47-kserve-5d5ff9b44c-bprmc" Jul 10 01:03:08.862242 ip-10-0-130-146 kubenswrapper[2570]: I0710 01:03:08.862198 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c4f9d91a-c62c-41bd-bbc2-fed40a121e92-tls-certs\") pod \"test-model-8f8f47-kserve-5d5ff9b44c-bprmc\" (UID: \"c4f9d91a-c62c-41bd-bbc2-fed40a121e92\") " pod="ai-tenant-e2e-inf-b-8f8f47/test-model-8f8f47-kserve-5d5ff9b44c-bprmc" Jul 10 01:03:08.862524 ip-10-0-130-146 kubenswrapper[2570]: I0710 01:03:08.862444 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c4f9d91a-c62c-41bd-bbc2-fed40a121e92-model-cache\") pod \"test-model-8f8f47-kserve-5d5ff9b44c-bprmc\" (UID: \"c4f9d91a-c62c-41bd-bbc2-fed40a121e92\") " pod="ai-tenant-e2e-inf-b-8f8f47/test-model-8f8f47-kserve-5d5ff9b44c-bprmc" Jul 10 01:03:08.862524 ip-10-0-130-146 kubenswrapper[2570]: I0710 01:03:08.862490 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c4f9d91a-c62c-41bd-bbc2-fed40a121e92-tmp-dir\") pod \"test-model-8f8f47-kserve-5d5ff9b44c-bprmc\" (UID: \"c4f9d91a-c62c-41bd-bbc2-fed40a121e92\") " pod="ai-tenant-e2e-inf-b-8f8f47/test-model-8f8f47-kserve-5d5ff9b44c-bprmc" Jul 10 01:03:08.862613 ip-10-0-130-146 kubenswrapper[2570]: I0710 01:03:08.862535 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c4f9d91a-c62c-41bd-bbc2-fed40a121e92-home\") pod \"test-model-8f8f47-kserve-5d5ff9b44c-bprmc\" (UID: \"c4f9d91a-c62c-41bd-bbc2-fed40a121e92\") " pod="ai-tenant-e2e-inf-b-8f8f47/test-model-8f8f47-kserve-5d5ff9b44c-bprmc" Jul 10 01:03:08.864384 ip-10-0-130-146 kubenswrapper[2570]: I0710 01:03:08.864363 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c4f9d91a-c62c-41bd-bbc2-fed40a121e92-dshm\") pod \"test-model-8f8f47-kserve-5d5ff9b44c-bprmc\" (UID: \"c4f9d91a-c62c-41bd-bbc2-fed40a121e92\") " pod="ai-tenant-e2e-inf-b-8f8f47/test-model-8f8f47-kserve-5d5ff9b44c-bprmc" Jul 10 01:03:08.864655 ip-10-0-130-146 kubenswrapper[2570]: I0710 01:03:08.864637 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c4f9d91a-c62c-41bd-bbc2-fed40a121e92-tls-certs\") pod \"test-model-8f8f47-kserve-5d5ff9b44c-bprmc\" (UID: \"c4f9d91a-c62c-41bd-bbc2-fed40a121e92\") " pod="ai-tenant-e2e-inf-b-8f8f47/test-model-8f8f47-kserve-5d5ff9b44c-bprmc" Jul 10 01:03:08.869884 ip-10-0-130-146 kubenswrapper[2570]: I0710 01:03:08.869861 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xgm8r\" (UniqueName: \"kubernetes.io/projected/c4f9d91a-c62c-41bd-bbc2-fed40a121e92-kube-api-access-xgm8r\") pod \"test-model-8f8f47-kserve-5d5ff9b44c-bprmc\" (UID: \"c4f9d91a-c62c-41bd-bbc2-fed40a121e92\") " pod="ai-tenant-e2e-inf-b-8f8f47/test-model-8f8f47-kserve-5d5ff9b44c-bprmc" Jul 10 01:03:08.989774 ip-10-0-130-146 kubenswrapper[2570]: I0710 01:03:08.989684 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-8f8f47/test-model-8f8f47-kserve-5d5ff9b44c-bprmc" Jul 10 01:03:09.122729 ip-10-0-130-146 kubenswrapper[2570]: I0710 01:03:09.122703 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-b-8f8f47/test-model-8f8f47-kserve-5d5ff9b44c-bprmc"] Jul 10 01:03:09.124116 ip-10-0-130-146 kubenswrapper[2570]: W0710 01:03:09.124088 2570 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podc4f9d91a_c62c_41bd_bbc2_fed40a121e92.slice/crio-030c9d7194ba863a6ba3da971be43d4c847841d0baf7c6f03effec36fbd38198 WatchSource:0}: Error finding container 030c9d7194ba863a6ba3da971be43d4c847841d0baf7c6f03effec36fbd38198: Status 404 returned error can't find the container with id 030c9d7194ba863a6ba3da971be43d4c847841d0baf7c6f03effec36fbd38198 Jul 10 01:03:09.360324 ip-10-0-130-146 kubenswrapper[2570]: I0710 01:03:09.360221 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-8f8f47/test-model-8f8f47-kserve-5d5ff9b44c-bprmc" event={"ID":"c4f9d91a-c62c-41bd-bbc2-fed40a121e92","Type":"ContainerStarted","Data":"6cc878be8d7c8bade0618233ea0d9c4dc3a01d196771bc72cee20fa4fd7c4e20"} Jul 10 01:03:09.360324 ip-10-0-130-146 kubenswrapper[2570]: I0710 01:03:09.360270 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-8f8f47/test-model-8f8f47-kserve-5d5ff9b44c-bprmc" event={"ID":"c4f9d91a-c62c-41bd-bbc2-fed40a121e92","Type":"ContainerStarted","Data":"030c9d7194ba863a6ba3da971be43d4c847841d0baf7c6f03effec36fbd38198"} Jul 10 01:03:09.376810 ip-10-0-130-146 kubenswrapper[2570]: I0710 01:03:09.376761 2570 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-inf-b-8f8f47/test-model-8f8f47-kserve-5d5ff9b44c-bprmc" podStartSLOduration=1.3767453 podStartE2EDuration="1.3767453s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-10 01:03:08 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 01:03:09.375038002 +0000 UTC m=+2208.423416725" watchObservedRunningTime="2026-07-10 01:03:09.3767453 +0000 UTC m=+2208.425124021" Jul 10 01:03:18.990832 ip-10-0-130-146 kubenswrapper[2570]: I0710 01:03:18.990795 2570 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-inf-b-8f8f47/test-model-8f8f47-kserve-5d5ff9b44c-bprmc" Jul 10 01:03:18.991331 ip-10-0-130-146 kubenswrapper[2570]: I0710 01:03:18.990847 2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-inf-b-8f8f47/test-model-8f8f47-kserve-5d5ff9b44c-bprmc" Jul 10 01:03:19.003667 ip-10-0-130-146 kubenswrapper[2570]: I0710 01:03:19.003640 2570 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-inf-b-8f8f47/test-model-8f8f47-kserve-5d5ff9b44c-bprmc" Jul 10 01:03:19.424744 ip-10-0-130-146 kubenswrapper[2570]: I0710 01:03:19.424719 2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-inf-b-8f8f47/test-model-8f8f47-kserve-5d5ff9b44c-bprmc" Jul 10 01:03:22.084881 ip-10-0-130-146 kubenswrapper[2570]: I0710 01:03:22.084843 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-g92hs"] Jul 10 01:03:27.059951 ip-10-0-130-146 kubenswrapper[2570]: I0710 01:03:27.059915 2570 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-a-85dfb6/test-model-85dfb6-kserve-978dcf4dc-hn6sc"] Jul 10 01:03:27.966569 ip-10-0-130-146 kubenswrapper[2570]: I0710 01:03:27.966536 2570 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-b-8f8f47/test-model-8f8f47-kserve-5d5ff9b44c-bprmc"] Jul 10 01:03:39.280946 ip-10-0-130-146 kubenswrapper[2570]: I0710 01:03:39.280911 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-g92hs"] Jul 10 01:03:42.080330 ip-10-0-130-146 kubenswrapper[2570]: I0710 01:03:42.080279 2570 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-inf-a-85dfb6/test-model-85dfb6-kserve-978dcf4dc-hn6sc" podUID="1b17c212-ada7-4589-97dc-9f898842f3ec" containerName="main" containerID="cri-o://8f39cd6569297cb39e9dc25cab4b99187d2c85c7b897c236cb608b2ff01d1313" gracePeriod=45 Jul 10 01:03:42.326991 ip-10-0-130-146 kubenswrapper[2570]: I0710 01:03:42.326967 2570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-85dfb6/test-model-85dfb6-kserve-978dcf4dc-hn6sc" Jul 10 01:03:42.472960 ip-10-0-130-146 kubenswrapper[2570]: I0710 01:03:42.472925 2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-ww2f6\" (UniqueName: \"kubernetes.io/projected/1b17c212-ada7-4589-97dc-9f898842f3ec-kube-api-access-ww2f6\") pod \"1b17c212-ada7-4589-97dc-9f898842f3ec\" (UID: \"1b17c212-ada7-4589-97dc-9f898842f3ec\") " Jul 10 01:03:42.473159 ip-10-0-130-146 kubenswrapper[2570]: I0710 01:03:42.472994 2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1b17c212-ada7-4589-97dc-9f898842f3ec-tls-certs\") pod \"1b17c212-ada7-4589-97dc-9f898842f3ec\" (UID: \"1b17c212-ada7-4589-97dc-9f898842f3ec\") " Jul 10 01:03:42.473159 ip-10-0-130-146 kubenswrapper[2570]: I0710 01:03:42.473037 2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1b17c212-ada7-4589-97dc-9f898842f3ec-model-cache\") pod \"1b17c212-ada7-4589-97dc-9f898842f3ec\" (UID: \"1b17c212-ada7-4589-97dc-9f898842f3ec\") " Jul 10 01:03:42.473159 ip-10-0-130-146 kubenswrapper[2570]: I0710 01:03:42.473130 2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1b17c212-ada7-4589-97dc-9f898842f3ec-dshm\") pod \"1b17c212-ada7-4589-97dc-9f898842f3ec\" (UID: \"1b17c212-ada7-4589-97dc-9f898842f3ec\") " Jul 10 01:03:42.473312 ip-10-0-130-146 kubenswrapper[2570]: I0710 01:03:42.473177 2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1b17c212-ada7-4589-97dc-9f898842f3ec-home\") pod \"1b17c212-ada7-4589-97dc-9f898842f3ec\" (UID: \"1b17c212-ada7-4589-97dc-9f898842f3ec\") " Jul 10 01:03:42.473312 ip-10-0-130-146 kubenswrapper[2570]: I0710 01:03:42.473220 2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1b17c212-ada7-4589-97dc-9f898842f3ec-tmp-dir\") pod \"1b17c212-ada7-4589-97dc-9f898842f3ec\" (UID: \"1b17c212-ada7-4589-97dc-9f898842f3ec\") " Jul 10 01:03:42.473420 ip-10-0-130-146 kubenswrapper[2570]: I0710 01:03:42.473311 2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1b17c212-ada7-4589-97dc-9f898842f3ec-model-cache" (OuterVolumeSpecName: "model-cache") pod "1b17c212-ada7-4589-97dc-9f898842f3ec" (UID: "1b17c212-ada7-4589-97dc-9f898842f3ec"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 10 01:03:42.473504 ip-10-0-130-146 kubenswrapper[2570]: I0710 01:03:42.473470 2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1b17c212-ada7-4589-97dc-9f898842f3ec-home" (OuterVolumeSpecName: "home") pod "1b17c212-ada7-4589-97dc-9f898842f3ec" (UID: "1b17c212-ada7-4589-97dc-9f898842f3ec"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 10 01:03:42.473504 ip-10-0-130-146 kubenswrapper[2570]: I0710 01:03:42.473483 2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1b17c212-ada7-4589-97dc-9f898842f3ec-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "1b17c212-ada7-4589-97dc-9f898842f3ec" (UID: "1b17c212-ada7-4589-97dc-9f898842f3ec"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 10 01:03:42.473688 ip-10-0-130-146 kubenswrapper[2570]: I0710 01:03:42.473563 2570 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1b17c212-ada7-4589-97dc-9f898842f3ec-model-cache\") on node \"ip-10-0-130-146.ec2.internal\" DevicePath \"\"" Jul 10 01:03:42.475311 ip-10-0-130-146 kubenswrapper[2570]: I0710 01:03:42.475283 2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1b17c212-ada7-4589-97dc-9f898842f3ec-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "1b17c212-ada7-4589-97dc-9f898842f3ec" (UID: "1b17c212-ada7-4589-97dc-9f898842f3ec"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 10 01:03:42.475400 ip-10-0-130-146 kubenswrapper[2570]: I0710 01:03:42.475348 2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1b17c212-ada7-4589-97dc-9f898842f3ec-dshm" (OuterVolumeSpecName: "dshm") pod "1b17c212-ada7-4589-97dc-9f898842f3ec" (UID: "1b17c212-ada7-4589-97dc-9f898842f3ec"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 10 01:03:42.475400 ip-10-0-130-146 kubenswrapper[2570]: I0710 01:03:42.475365 2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1b17c212-ada7-4589-97dc-9f898842f3ec-kube-api-access-ww2f6" (OuterVolumeSpecName: "kube-api-access-ww2f6") pod "1b17c212-ada7-4589-97dc-9f898842f3ec" (UID: "1b17c212-ada7-4589-97dc-9f898842f3ec"). InnerVolumeSpecName "kube-api-access-ww2f6". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 10 01:03:42.535093 ip-10-0-130-146 kubenswrapper[2570]: I0710 01:03:42.535041 2570 generic.go:356] "Generic (PLEG): container finished" podID="1b17c212-ada7-4589-97dc-9f898842f3ec" containerID="8f39cd6569297cb39e9dc25cab4b99187d2c85c7b897c236cb608b2ff01d1313" exitCode=0 Jul 10 01:03:42.535234 ip-10-0-130-146 kubenswrapper[2570]: I0710 01:03:42.535103 2570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-85dfb6/test-model-85dfb6-kserve-978dcf4dc-hn6sc" Jul 10 01:03:42.535234 ip-10-0-130-146 kubenswrapper[2570]: I0710 01:03:42.535110 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-85dfb6/test-model-85dfb6-kserve-978dcf4dc-hn6sc" event={"ID":"1b17c212-ada7-4589-97dc-9f898842f3ec","Type":"ContainerDied","Data":"8f39cd6569297cb39e9dc25cab4b99187d2c85c7b897c236cb608b2ff01d1313"} Jul 10 01:03:42.535234 ip-10-0-130-146 kubenswrapper[2570]: I0710 01:03:42.535148 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-85dfb6/test-model-85dfb6-kserve-978dcf4dc-hn6sc" event={"ID":"1b17c212-ada7-4589-97dc-9f898842f3ec","Type":"ContainerDied","Data":"7da633217f96c47be72143c348c82048dc56cde376a208fb2f7c37d64c3edd9d"} Jul 10 01:03:42.535234 ip-10-0-130-146 kubenswrapper[2570]: I0710 01:03:42.535165 2570 scope.go:117] "RemoveContainer" containerID="8f39cd6569297cb39e9dc25cab4b99187d2c85c7b897c236cb608b2ff01d1313" Jul 10 01:03:42.544208 ip-10-0-130-146 kubenswrapper[2570]: I0710 01:03:42.544190 2570 scope.go:117] "RemoveContainer" containerID="8f39cd6569297cb39e9dc25cab4b99187d2c85c7b897c236cb608b2ff01d1313" Jul 10 01:03:42.544429 ip-10-0-130-146 kubenswrapper[2570]: E0710 01:03:42.544413 2570 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8f39cd6569297cb39e9dc25cab4b99187d2c85c7b897c236cb608b2ff01d1313\": container with ID starting with 8f39cd6569297cb39e9dc25cab4b99187d2c85c7b897c236cb608b2ff01d1313 not found: ID does not exist" containerID="8f39cd6569297cb39e9dc25cab4b99187d2c85c7b897c236cb608b2ff01d1313" Jul 10 01:03:42.544477 ip-10-0-130-146 kubenswrapper[2570]: I0710 01:03:42.544436 2570 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8f39cd6569297cb39e9dc25cab4b99187d2c85c7b897c236cb608b2ff01d1313"} err="failed to get container status \"8f39cd6569297cb39e9dc25cab4b99187d2c85c7b897c236cb608b2ff01d1313\": rpc error: code = NotFound desc = could not find container \"8f39cd6569297cb39e9dc25cab4b99187d2c85c7b897c236cb608b2ff01d1313\": container with ID starting with 8f39cd6569297cb39e9dc25cab4b99187d2c85c7b897c236cb608b2ff01d1313 not found: ID does not exist" Jul 10 01:03:42.556224 ip-10-0-130-146 kubenswrapper[2570]: I0710 01:03:42.556200 2570 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-a-85dfb6/test-model-85dfb6-kserve-978dcf4dc-hn6sc"] Jul 10 01:03:42.559844 ip-10-0-130-146 kubenswrapper[2570]: I0710 01:03:42.559825 2570 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-inf-a-85dfb6/test-model-85dfb6-kserve-978dcf4dc-hn6sc"] Jul 10 01:03:42.574438 ip-10-0-130-146 kubenswrapper[2570]: I0710 01:03:42.574421 2570 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1b17c212-ada7-4589-97dc-9f898842f3ec-dshm\") on node \"ip-10-0-130-146.ec2.internal\" DevicePath \"\"" Jul 10 01:03:42.574515 ip-10-0-130-146 kubenswrapper[2570]: I0710 01:03:42.574440 2570 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1b17c212-ada7-4589-97dc-9f898842f3ec-home\") on node \"ip-10-0-130-146.ec2.internal\" DevicePath \"\"" Jul 10 01:03:42.574515 ip-10-0-130-146 kubenswrapper[2570]: I0710 01:03:42.574449 2570 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1b17c212-ada7-4589-97dc-9f898842f3ec-tmp-dir\") on node \"ip-10-0-130-146.ec2.internal\" DevicePath \"\"" Jul 10 01:03:42.574515 ip-10-0-130-146 kubenswrapper[2570]: I0710 01:03:42.574457 2570 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ww2f6\" (UniqueName: \"kubernetes.io/projected/1b17c212-ada7-4589-97dc-9f898842f3ec-kube-api-access-ww2f6\") on node \"ip-10-0-130-146.ec2.internal\" DevicePath \"\"" Jul 10 01:03:42.574515 ip-10-0-130-146 kubenswrapper[2570]: I0710 01:03:42.574466 2570 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1b17c212-ada7-4589-97dc-9f898842f3ec-tls-certs\") on node \"ip-10-0-130-146.ec2.internal\" DevicePath \"\"" Jul 10 01:03:42.986666 ip-10-0-130-146 kubenswrapper[2570]: I0710 01:03:42.986623 2570 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-inf-b-8f8f47/test-model-8f8f47-kserve-5d5ff9b44c-bprmc" podUID="c4f9d91a-c62c-41bd-bbc2-fed40a121e92" containerName="main" containerID="cri-o://6cc878be8d7c8bade0618233ea0d9c4dc3a01d196771bc72cee20fa4fd7c4e20" gracePeriod=45 Jul 10 01:03:43.239443 ip-10-0-130-146 kubenswrapper[2570]: I0710 01:03:43.239389 2570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-8f8f47/test-model-8f8f47-kserve-5d5ff9b44c-bprmc" Jul 10 01:03:43.380935 ip-10-0-130-146 kubenswrapper[2570]: I0710 01:03:43.380902 2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c4f9d91a-c62c-41bd-bbc2-fed40a121e92-home\") pod \"c4f9d91a-c62c-41bd-bbc2-fed40a121e92\" (UID: \"c4f9d91a-c62c-41bd-bbc2-fed40a121e92\") " Jul 10 01:03:43.381159 ip-10-0-130-146 kubenswrapper[2570]: I0710 01:03:43.380941 2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-xgm8r\" (UniqueName: \"kubernetes.io/projected/c4f9d91a-c62c-41bd-bbc2-fed40a121e92-kube-api-access-xgm8r\") pod \"c4f9d91a-c62c-41bd-bbc2-fed40a121e92\" (UID: \"c4f9d91a-c62c-41bd-bbc2-fed40a121e92\") " Jul 10 01:03:43.381159 ip-10-0-130-146 kubenswrapper[2570]: I0710 01:03:43.381018 2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c4f9d91a-c62c-41bd-bbc2-fed40a121e92-dshm\") pod \"c4f9d91a-c62c-41bd-bbc2-fed40a121e92\" (UID: \"c4f9d91a-c62c-41bd-bbc2-fed40a121e92\") " Jul 10 01:03:43.381159 ip-10-0-130-146 kubenswrapper[2570]: I0710 01:03:43.381074 2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c4f9d91a-c62c-41bd-bbc2-fed40a121e92-tmp-dir\") pod \"c4f9d91a-c62c-41bd-bbc2-fed40a121e92\" (UID: \"c4f9d91a-c62c-41bd-bbc2-fed40a121e92\") " Jul 10 01:03:43.381159 ip-10-0-130-146 kubenswrapper[2570]: I0710 01:03:43.381125 2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c4f9d91a-c62c-41bd-bbc2-fed40a121e92-tls-certs\") pod \"c4f9d91a-c62c-41bd-bbc2-fed40a121e92\" (UID: \"c4f9d91a-c62c-41bd-bbc2-fed40a121e92\") " Jul 10 01:03:43.381390 ip-10-0-130-146 kubenswrapper[2570]: I0710 01:03:43.381172 2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c4f9d91a-c62c-41bd-bbc2-fed40a121e92-model-cache\") pod \"c4f9d91a-c62c-41bd-bbc2-fed40a121e92\" (UID: \"c4f9d91a-c62c-41bd-bbc2-fed40a121e92\") " Jul 10 01:03:43.381390 ip-10-0-130-146 kubenswrapper[2570]: I0710 01:03:43.381197 2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c4f9d91a-c62c-41bd-bbc2-fed40a121e92-home" (OuterVolumeSpecName: "home") pod "c4f9d91a-c62c-41bd-bbc2-fed40a121e92" (UID: "c4f9d91a-c62c-41bd-bbc2-fed40a121e92"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 10 01:03:43.381390 ip-10-0-130-146 kubenswrapper[2570]: I0710 01:03:43.381308 2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c4f9d91a-c62c-41bd-bbc2-fed40a121e92-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "c4f9d91a-c62c-41bd-bbc2-fed40a121e92" (UID: "c4f9d91a-c62c-41bd-bbc2-fed40a121e92"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 10 01:03:43.381541 ip-10-0-130-146 kubenswrapper[2570]: I0710 01:03:43.381461 2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c4f9d91a-c62c-41bd-bbc2-fed40a121e92-model-cache" (OuterVolumeSpecName: "model-cache") pod "c4f9d91a-c62c-41bd-bbc2-fed40a121e92" (UID: "c4f9d91a-c62c-41bd-bbc2-fed40a121e92"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 10 01:03:43.381541 ip-10-0-130-146 kubenswrapper[2570]: I0710 01:03:43.381477 2570 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c4f9d91a-c62c-41bd-bbc2-fed40a121e92-home\") on node \"ip-10-0-130-146.ec2.internal\" DevicePath \"\"" Jul 10 01:03:43.381541 ip-10-0-130-146 kubenswrapper[2570]: I0710 01:03:43.381495 2570 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c4f9d91a-c62c-41bd-bbc2-fed40a121e92-tmp-dir\") on node \"ip-10-0-130-146.ec2.internal\" DevicePath \"\"" Jul 10 01:03:43.383336 ip-10-0-130-146 kubenswrapper[2570]: I0710 01:03:43.383314 2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c4f9d91a-c62c-41bd-bbc2-fed40a121e92-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "c4f9d91a-c62c-41bd-bbc2-fed40a121e92" (UID: "c4f9d91a-c62c-41bd-bbc2-fed40a121e92"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 10 01:03:43.383446 ip-10-0-130-146 kubenswrapper[2570]: I0710 01:03:43.383346 2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c4f9d91a-c62c-41bd-bbc2-fed40a121e92-dshm" (OuterVolumeSpecName: "dshm") pod "c4f9d91a-c62c-41bd-bbc2-fed40a121e92" (UID: "c4f9d91a-c62c-41bd-bbc2-fed40a121e92"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 10 01:03:43.383446 ip-10-0-130-146 kubenswrapper[2570]: I0710 01:03:43.383361 2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c4f9d91a-c62c-41bd-bbc2-fed40a121e92-kube-api-access-xgm8r" (OuterVolumeSpecName: "kube-api-access-xgm8r") pod "c4f9d91a-c62c-41bd-bbc2-fed40a121e92" (UID: "c4f9d91a-c62c-41bd-bbc2-fed40a121e92"). InnerVolumeSpecName "kube-api-access-xgm8r". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 10 01:03:43.454164 ip-10-0-130-146 kubenswrapper[2570]: I0710 01:03:43.454132 2570 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="1b17c212-ada7-4589-97dc-9f898842f3ec" path="/var/lib/kubelet/pods/1b17c212-ada7-4589-97dc-9f898842f3ec/volumes" Jul 10 01:03:43.482331 ip-10-0-130-146 kubenswrapper[2570]: I0710 01:03:43.482305 2570 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c4f9d91a-c62c-41bd-bbc2-fed40a121e92-dshm\") on node \"ip-10-0-130-146.ec2.internal\" DevicePath \"\"" Jul 10 01:03:43.482331 ip-10-0-130-146 kubenswrapper[2570]: I0710 01:03:43.482326 2570 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c4f9d91a-c62c-41bd-bbc2-fed40a121e92-tls-certs\") on node \"ip-10-0-130-146.ec2.internal\" DevicePath \"\"" Jul 10 01:03:43.482474 ip-10-0-130-146 kubenswrapper[2570]: I0710 01:03:43.482337 2570 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c4f9d91a-c62c-41bd-bbc2-fed40a121e92-model-cache\") on node \"ip-10-0-130-146.ec2.internal\" DevicePath \"\"" Jul 10 01:03:43.482474 ip-10-0-130-146 kubenswrapper[2570]: I0710 01:03:43.482346 2570 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xgm8r\" (UniqueName: \"kubernetes.io/projected/c4f9d91a-c62c-41bd-bbc2-fed40a121e92-kube-api-access-xgm8r\") on node \"ip-10-0-130-146.ec2.internal\" DevicePath \"\"" Jul 10 01:03:43.541986 ip-10-0-130-146 kubenswrapper[2570]: I0710 01:03:43.541904 2570 generic.go:356] "Generic (PLEG): container finished" podID="c4f9d91a-c62c-41bd-bbc2-fed40a121e92" containerID="6cc878be8d7c8bade0618233ea0d9c4dc3a01d196771bc72cee20fa4fd7c4e20" exitCode=0 Jul 10 01:03:43.541986 ip-10-0-130-146 kubenswrapper[2570]: I0710 01:03:43.541952 2570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-8f8f47/test-model-8f8f47-kserve-5d5ff9b44c-bprmc" Jul 10 01:03:43.541986 ip-10-0-130-146 kubenswrapper[2570]: I0710 01:03:43.541977 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-8f8f47/test-model-8f8f47-kserve-5d5ff9b44c-bprmc" event={"ID":"c4f9d91a-c62c-41bd-bbc2-fed40a121e92","Type":"ContainerDied","Data":"6cc878be8d7c8bade0618233ea0d9c4dc3a01d196771bc72cee20fa4fd7c4e20"} Jul 10 01:03:43.542248 ip-10-0-130-146 kubenswrapper[2570]: I0710 01:03:43.542012 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-8f8f47/test-model-8f8f47-kserve-5d5ff9b44c-bprmc" event={"ID":"c4f9d91a-c62c-41bd-bbc2-fed40a121e92","Type":"ContainerDied","Data":"030c9d7194ba863a6ba3da971be43d4c847841d0baf7c6f03effec36fbd38198"} Jul 10 01:03:43.542248 ip-10-0-130-146 kubenswrapper[2570]: I0710 01:03:43.542029 2570 scope.go:117] "RemoveContainer" containerID="6cc878be8d7c8bade0618233ea0d9c4dc3a01d196771bc72cee20fa4fd7c4e20" Jul 10 01:03:43.553183 ip-10-0-130-146 kubenswrapper[2570]: I0710 01:03:43.553161 2570 scope.go:117] "RemoveContainer" containerID="6cc878be8d7c8bade0618233ea0d9c4dc3a01d196771bc72cee20fa4fd7c4e20" Jul 10 01:03:43.553563 ip-10-0-130-146 kubenswrapper[2570]: E0710 01:03:43.553545 2570 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6cc878be8d7c8bade0618233ea0d9c4dc3a01d196771bc72cee20fa4fd7c4e20\": container with ID starting with 6cc878be8d7c8bade0618233ea0d9c4dc3a01d196771bc72cee20fa4fd7c4e20 not found: ID does not exist" containerID="6cc878be8d7c8bade0618233ea0d9c4dc3a01d196771bc72cee20fa4fd7c4e20" Jul 10 01:03:43.553649 ip-10-0-130-146 kubenswrapper[2570]: I0710 01:03:43.553575 2570 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6cc878be8d7c8bade0618233ea0d9c4dc3a01d196771bc72cee20fa4fd7c4e20"} err="failed to get container status \"6cc878be8d7c8bade0618233ea0d9c4dc3a01d196771bc72cee20fa4fd7c4e20\": rpc error: code = NotFound desc = could not find container \"6cc878be8d7c8bade0618233ea0d9c4dc3a01d196771bc72cee20fa4fd7c4e20\": container with ID starting with 6cc878be8d7c8bade0618233ea0d9c4dc3a01d196771bc72cee20fa4fd7c4e20 not found: ID does not exist" Jul 10 01:03:43.558475 ip-10-0-130-146 kubenswrapper[2570]: I0710 01:03:43.558453 2570 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-b-8f8f47/test-model-8f8f47-kserve-5d5ff9b44c-bprmc"] Jul 10 01:03:43.560646 ip-10-0-130-146 kubenswrapper[2570]: I0710 01:03:43.560627 2570 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-inf-b-8f8f47/test-model-8f8f47-kserve-5d5ff9b44c-bprmc"] Jul 10 01:03:45.457246 ip-10-0-130-146 kubenswrapper[2570]: I0710 01:03:45.454972 2570 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="c4f9d91a-c62c-41bd-bbc2-fed40a121e92" path="/var/lib/kubelet/pods/c4f9d91a-c62c-41bd-bbc2-fed40a121e92/volumes" Jul 10 01:04:17.183865 ip-10-0-130-146 kubenswrapper[2570]: I0710 01:04:17.183832 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-g92hs"] Jul 10 01:04:28.576691 ip-10-0-130-146 kubenswrapper[2570]: I0710 01:04:28.576658 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-g92hs"] Jul 10 01:04:45.580696 ip-10-0-130-146 kubenswrapper[2570]: I0710 01:04:45.580661 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-g92hs"] Jul 10 01:05:00.386602 ip-10-0-130-146 kubenswrapper[2570]: I0710 01:05:00.386564 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-g92hs"] Jul 10 01:05:15.384140 ip-10-0-130-146 kubenswrapper[2570]: I0710 01:05:15.384097 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-g92hs"] Jul 10 01:05:26.908964 ip-10-0-130-146 kubenswrapper[2570]: I0710 01:05:26.908934 2570 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-a-48a750/auth-test-model-48a750-kserve-66ccc45475-v489h"] Jul 10 01:05:26.909418 ip-10-0-130-146 kubenswrapper[2570]: I0710 01:05:26.909336 2570 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1b17c212-ada7-4589-97dc-9f898842f3ec" containerName="main" Jul 10 01:05:26.909418 ip-10-0-130-146 kubenswrapper[2570]: I0710 01:05:26.909347 2570 state_mem.go:107] "Deleted CPUSet assignment" podUID="1b17c212-ada7-4589-97dc-9f898842f3ec" containerName="main" Jul 10 01:05:26.909418 ip-10-0-130-146 kubenswrapper[2570]: I0710 01:05:26.909374 2570 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c4f9d91a-c62c-41bd-bbc2-fed40a121e92" containerName="main" Jul 10 01:05:26.909418 ip-10-0-130-146 kubenswrapper[2570]: I0710 01:05:26.909379 2570 state_mem.go:107] "Deleted CPUSet assignment" podUID="c4f9d91a-c62c-41bd-bbc2-fed40a121e92" containerName="main" Jul 10 01:05:26.909555 ip-10-0-130-146 kubenswrapper[2570]: I0710 01:05:26.909436 2570 memory_manager.go:368] "RemoveStaleState removing state" podUID="1b17c212-ada7-4589-97dc-9f898842f3ec" containerName="main" Jul 10 01:05:26.909555 ip-10-0-130-146 kubenswrapper[2570]: I0710 01:05:26.909446 2570 memory_manager.go:368] "RemoveStaleState removing state" podUID="c4f9d91a-c62c-41bd-bbc2-fed40a121e92" containerName="main" Jul 10 01:05:26.912489 ip-10-0-130-146 kubenswrapper[2570]: I0710 01:05:26.912468 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-48a750/auth-test-model-48a750-kserve-66ccc45475-v489h" Jul 10 01:05:26.916133 ip-10-0-130-146 kubenswrapper[2570]: I0710 01:05:26.916113 2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-48a750\"/\"kube-root-ca.crt\"" Jul 10 01:05:26.916266 ip-10-0-130-146 kubenswrapper[2570]: I0710 01:05:26.916115 2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-48a750\"/\"openshift-service-ca.crt\"" Jul 10 01:05:26.916266 ip-10-0-130-146 kubenswrapper[2570]: I0710 01:05:26.916115 2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-48a750\"/\"default-dockercfg-sjrxd\"" Jul 10 01:05:26.916266 ip-10-0-130-146 kubenswrapper[2570]: I0710 01:05:26.916115 2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-48a750\"/\"auth-test-model-48a750-kserve-self-signed-certs\"" Jul 10 01:05:26.920770 ip-10-0-130-146 kubenswrapper[2570]: I0710 01:05:26.920751 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-48a750/auth-test-model-48a750-kserve-66ccc45475-v489h"] Jul 10 01:05:27.073304 ip-10-0-130-146 kubenswrapper[2570]: I0710 01:05:27.073270 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e7e455f3-aa01-4630-b72a-fedfce8c8b0c-dshm\") pod \"auth-test-model-48a750-kserve-66ccc45475-v489h\" (UID: \"e7e455f3-aa01-4630-b72a-fedfce8c8b0c\") " pod="ai-tenant-e2e-shared-a-48a750/auth-test-model-48a750-kserve-66ccc45475-v489h" Jul 10 01:05:27.073304 ip-10-0-130-146 kubenswrapper[2570]: I0710 01:05:27.073306 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e7e455f3-aa01-4630-b72a-fedfce8c8b0c-model-cache\") pod \"auth-test-model-48a750-kserve-66ccc45475-v489h\" (UID: \"e7e455f3-aa01-4630-b72a-fedfce8c8b0c\") " pod="ai-tenant-e2e-shared-a-48a750/auth-test-model-48a750-kserve-66ccc45475-v489h" Jul 10 01:05:27.073513 ip-10-0-130-146 kubenswrapper[2570]: I0710 01:05:27.073348 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ctmsw\" (UniqueName: \"kubernetes.io/projected/e7e455f3-aa01-4630-b72a-fedfce8c8b0c-kube-api-access-ctmsw\") pod \"auth-test-model-48a750-kserve-66ccc45475-v489h\" (UID: \"e7e455f3-aa01-4630-b72a-fedfce8c8b0c\") " pod="ai-tenant-e2e-shared-a-48a750/auth-test-model-48a750-kserve-66ccc45475-v489h" Jul 10 01:05:27.073513 ip-10-0-130-146 kubenswrapper[2570]: I0710 01:05:27.073417 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e7e455f3-aa01-4630-b72a-fedfce8c8b0c-tls-certs\") pod \"auth-test-model-48a750-kserve-66ccc45475-v489h\" (UID: \"e7e455f3-aa01-4630-b72a-fedfce8c8b0c\") " pod="ai-tenant-e2e-shared-a-48a750/auth-test-model-48a750-kserve-66ccc45475-v489h" Jul 10 01:05:27.073513 ip-10-0-130-146 kubenswrapper[2570]: I0710 01:05:27.073473 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e7e455f3-aa01-4630-b72a-fedfce8c8b0c-home\") pod \"auth-test-model-48a750-kserve-66ccc45475-v489h\" (UID: \"e7e455f3-aa01-4630-b72a-fedfce8c8b0c\") " pod="ai-tenant-e2e-shared-a-48a750/auth-test-model-48a750-kserve-66ccc45475-v489h" Jul 10 01:05:27.073616 ip-10-0-130-146 kubenswrapper[2570]: I0710 01:05:27.073527 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e7e455f3-aa01-4630-b72a-fedfce8c8b0c-tmp-dir\") pod \"auth-test-model-48a750-kserve-66ccc45475-v489h\" (UID: \"e7e455f3-aa01-4630-b72a-fedfce8c8b0c\") " pod="ai-tenant-e2e-shared-a-48a750/auth-test-model-48a750-kserve-66ccc45475-v489h" Jul 10 01:05:27.174280 ip-10-0-130-146 kubenswrapper[2570]: I0710 01:05:27.174202 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e7e455f3-aa01-4630-b72a-fedfce8c8b0c-tls-certs\") pod \"auth-test-model-48a750-kserve-66ccc45475-v489h\" (UID: \"e7e455f3-aa01-4630-b72a-fedfce8c8b0c\") " pod="ai-tenant-e2e-shared-a-48a750/auth-test-model-48a750-kserve-66ccc45475-v489h" Jul 10 01:05:27.174280 ip-10-0-130-146 kubenswrapper[2570]: I0710 01:05:27.174253 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e7e455f3-aa01-4630-b72a-fedfce8c8b0c-home\") pod \"auth-test-model-48a750-kserve-66ccc45475-v489h\" (UID: \"e7e455f3-aa01-4630-b72a-fedfce8c8b0c\") " pod="ai-tenant-e2e-shared-a-48a750/auth-test-model-48a750-kserve-66ccc45475-v489h" Jul 10 01:05:27.174459 ip-10-0-130-146 kubenswrapper[2570]: I0710 01:05:27.174372 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e7e455f3-aa01-4630-b72a-fedfce8c8b0c-tmp-dir\") pod \"auth-test-model-48a750-kserve-66ccc45475-v489h\" (UID: \"e7e455f3-aa01-4630-b72a-fedfce8c8b0c\") " pod="ai-tenant-e2e-shared-a-48a750/auth-test-model-48a750-kserve-66ccc45475-v489h" Jul 10 01:05:27.174459 ip-10-0-130-146 kubenswrapper[2570]: I0710 01:05:27.174426 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e7e455f3-aa01-4630-b72a-fedfce8c8b0c-dshm\") pod \"auth-test-model-48a750-kserve-66ccc45475-v489h\" (UID: \"e7e455f3-aa01-4630-b72a-fedfce8c8b0c\") " pod="ai-tenant-e2e-shared-a-48a750/auth-test-model-48a750-kserve-66ccc45475-v489h" Jul 10 01:05:27.174459 ip-10-0-130-146 kubenswrapper[2570]: I0710 01:05:27.174454 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e7e455f3-aa01-4630-b72a-fedfce8c8b0c-model-cache\") pod \"auth-test-model-48a750-kserve-66ccc45475-v489h\" (UID: \"e7e455f3-aa01-4630-b72a-fedfce8c8b0c\") " pod="ai-tenant-e2e-shared-a-48a750/auth-test-model-48a750-kserve-66ccc45475-v489h" Jul 10 01:05:27.174609 ip-10-0-130-146 kubenswrapper[2570]: I0710 01:05:27.174501 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ctmsw\" (UniqueName: \"kubernetes.io/projected/e7e455f3-aa01-4630-b72a-fedfce8c8b0c-kube-api-access-ctmsw\") pod \"auth-test-model-48a750-kserve-66ccc45475-v489h\" (UID: \"e7e455f3-aa01-4630-b72a-fedfce8c8b0c\") " pod="ai-tenant-e2e-shared-a-48a750/auth-test-model-48a750-kserve-66ccc45475-v489h" Jul 10 01:05:27.174609 ip-10-0-130-146 kubenswrapper[2570]: I0710 01:05:27.174558 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e7e455f3-aa01-4630-b72a-fedfce8c8b0c-home\") pod \"auth-test-model-48a750-kserve-66ccc45475-v489h\" (UID: \"e7e455f3-aa01-4630-b72a-fedfce8c8b0c\") " pod="ai-tenant-e2e-shared-a-48a750/auth-test-model-48a750-kserve-66ccc45475-v489h" Jul 10 01:05:27.174721 ip-10-0-130-146 kubenswrapper[2570]: I0710 01:05:27.174691 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e7e455f3-aa01-4630-b72a-fedfce8c8b0c-tmp-dir\") pod \"auth-test-model-48a750-kserve-66ccc45475-v489h\" (UID: \"e7e455f3-aa01-4630-b72a-fedfce8c8b0c\") " pod="ai-tenant-e2e-shared-a-48a750/auth-test-model-48a750-kserve-66ccc45475-v489h" Jul 10 01:05:27.174783 ip-10-0-130-146 kubenswrapper[2570]: I0710 01:05:27.174768 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e7e455f3-aa01-4630-b72a-fedfce8c8b0c-model-cache\") pod \"auth-test-model-48a750-kserve-66ccc45475-v489h\" (UID: \"e7e455f3-aa01-4630-b72a-fedfce8c8b0c\") " pod="ai-tenant-e2e-shared-a-48a750/auth-test-model-48a750-kserve-66ccc45475-v489h" Jul 10 01:05:27.176533 ip-10-0-130-146 kubenswrapper[2570]: I0710 01:05:27.176511 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e7e455f3-aa01-4630-b72a-fedfce8c8b0c-dshm\") pod \"auth-test-model-48a750-kserve-66ccc45475-v489h\" (UID: \"e7e455f3-aa01-4630-b72a-fedfce8c8b0c\") " pod="ai-tenant-e2e-shared-a-48a750/auth-test-model-48a750-kserve-66ccc45475-v489h" Jul 10 01:05:27.176712 ip-10-0-130-146 kubenswrapper[2570]: I0710 01:05:27.176693 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e7e455f3-aa01-4630-b72a-fedfce8c8b0c-tls-certs\") pod \"auth-test-model-48a750-kserve-66ccc45475-v489h\" (UID: \"e7e455f3-aa01-4630-b72a-fedfce8c8b0c\") " pod="ai-tenant-e2e-shared-a-48a750/auth-test-model-48a750-kserve-66ccc45475-v489h" Jul 10 01:05:27.183745 ip-10-0-130-146 kubenswrapper[2570]: I0710 01:05:27.183721 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ctmsw\" (UniqueName: \"kubernetes.io/projected/e7e455f3-aa01-4630-b72a-fedfce8c8b0c-kube-api-access-ctmsw\") pod \"auth-test-model-48a750-kserve-66ccc45475-v489h\" (UID: \"e7e455f3-aa01-4630-b72a-fedfce8c8b0c\") " pod="ai-tenant-e2e-shared-a-48a750/auth-test-model-48a750-kserve-66ccc45475-v489h" Jul 10 01:05:27.222664 ip-10-0-130-146 kubenswrapper[2570]: I0710 01:05:27.222641 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-48a750/auth-test-model-48a750-kserve-66ccc45475-v489h" Jul 10 01:05:27.355036 ip-10-0-130-146 kubenswrapper[2570]: I0710 01:05:27.355005 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-48a750/auth-test-model-48a750-kserve-66ccc45475-v489h"] Jul 10 01:05:27.356250 ip-10-0-130-146 kubenswrapper[2570]: W0710 01:05:27.356223 2570 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pode7e455f3_aa01_4630_b72a_fedfce8c8b0c.slice/crio-e2ef5819eee4d80a79d62ffeca7ee093e3c799763353bf758abb32bff359c805 WatchSource:0}: Error finding container e2ef5819eee4d80a79d62ffeca7ee093e3c799763353bf758abb32bff359c805: Status 404 returned error can't find the container with id e2ef5819eee4d80a79d62ffeca7ee093e3c799763353bf758abb32bff359c805 Jul 10 01:05:28.107172 ip-10-0-130-146 kubenswrapper[2570]: I0710 01:05:28.107142 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-48a750/auth-test-model-48a750-kserve-66ccc45475-v489h" event={"ID":"e7e455f3-aa01-4630-b72a-fedfce8c8b0c","Type":"ContainerStarted","Data":"6da943f94f43baf797aed197046b5586eac79d6c48fffb191ab454369b7bc695"} Jul 10 01:05:28.107554 ip-10-0-130-146 kubenswrapper[2570]: I0710 01:05:28.107186 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-48a750/auth-test-model-48a750-kserve-66ccc45475-v489h" event={"ID":"e7e455f3-aa01-4630-b72a-fedfce8c8b0c","Type":"ContainerStarted","Data":"e2ef5819eee4d80a79d62ffeca7ee093e3c799763353bf758abb32bff359c805"} Jul 10 01:05:28.124077 ip-10-0-130-146 kubenswrapper[2570]: I0710 01:05:28.124002 2570 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-a-48a750/auth-test-model-48a750-kserve-66ccc45475-v489h" podStartSLOduration=2.123984602 podStartE2EDuration="2.123984602s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-10 01:05:26 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 01:05:28.122887228 +0000 UTC m=+2347.171265953" watchObservedRunningTime="2026-07-10 01:05:28.123984602 +0000 UTC m=+2347.172363323" Jul 10 01:05:30.981101 ip-10-0-130-146 kubenswrapper[2570]: I0710 01:05:30.981063 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-g92hs"] Jul 10 01:05:37.223365 ip-10-0-130-146 kubenswrapper[2570]: I0710 01:05:37.223332 2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-a-48a750/auth-test-model-48a750-kserve-66ccc45475-v489h" Jul 10 01:05:37.223847 ip-10-0-130-146 kubenswrapper[2570]: I0710 01:05:37.223382 2570 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-a-48a750/auth-test-model-48a750-kserve-66ccc45475-v489h" Jul 10 01:05:37.242030 ip-10-0-130-146 kubenswrapper[2570]: I0710 01:05:37.242000 2570 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-a-48a750/auth-test-model-48a750-kserve-66ccc45475-v489h" Jul 10 01:05:38.171827 ip-10-0-130-146 kubenswrapper[2570]: I0710 01:05:38.171799 2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-a-48a750/auth-test-model-48a750-kserve-66ccc45475-v489h" Jul 10 01:05:52.986573 ip-10-0-130-146 kubenswrapper[2570]: I0710 01:05:52.986539 2570 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-b-aaea96/auth-test-model-aaea96-kserve-7fb8dc948c-d8b9c"] Jul 10 01:05:52.990160 ip-10-0-130-146 kubenswrapper[2570]: I0710 01:05:52.990143 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-aaea96/auth-test-model-aaea96-kserve-7fb8dc948c-d8b9c" Jul 10 01:05:52.993727 ip-10-0-130-146 kubenswrapper[2570]: I0710 01:05:52.993693 2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-aaea96\"/\"openshift-service-ca.crt\"" Jul 10 01:05:52.993727 ip-10-0-130-146 kubenswrapper[2570]: I0710 01:05:52.993718 2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-aaea96\"/\"kube-root-ca.crt\"" Jul 10 01:05:52.993958 ip-10-0-130-146 kubenswrapper[2570]: I0710 01:05:52.993699 2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-aaea96\"/\"default-dockercfg-dz7xs\"" Jul 10 01:05:52.993958 ip-10-0-130-146 kubenswrapper[2570]: I0710 01:05:52.993693 2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-aaea96\"/\"auth-test-model-aaea96-kserve-self-signed-certs\"" Jul 10 01:05:52.999392 ip-10-0-130-146 kubenswrapper[2570]: I0710 01:05:52.999370 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-aaea96/auth-test-model-aaea96-kserve-7fb8dc948c-d8b9c"] Jul 10 01:05:53.087450 ip-10-0-130-146 kubenswrapper[2570]: I0710 01:05:53.087416 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ca012a7b-c0de-4b94-a970-726459db491f-model-cache\") pod \"auth-test-model-aaea96-kserve-7fb8dc948c-d8b9c\" (UID: \"ca012a7b-c0de-4b94-a970-726459db491f\") " pod="ai-tenant-e2e-shared-b-aaea96/auth-test-model-aaea96-kserve-7fb8dc948c-d8b9c" Jul 10 01:05:53.087450 ip-10-0-130-146 kubenswrapper[2570]: I0710 01:05:53.087450 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ca012a7b-c0de-4b94-a970-726459db491f-tls-certs\") pod \"auth-test-model-aaea96-kserve-7fb8dc948c-d8b9c\" (UID: \"ca012a7b-c0de-4b94-a970-726459db491f\") " pod="ai-tenant-e2e-shared-b-aaea96/auth-test-model-aaea96-kserve-7fb8dc948c-d8b9c" Jul 10 01:05:53.087682 ip-10-0-130-146 kubenswrapper[2570]: I0710 01:05:53.087470 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2lszh\" (UniqueName: \"kubernetes.io/projected/ca012a7b-c0de-4b94-a970-726459db491f-kube-api-access-2lszh\") pod \"auth-test-model-aaea96-kserve-7fb8dc948c-d8b9c\" (UID: \"ca012a7b-c0de-4b94-a970-726459db491f\") " pod="ai-tenant-e2e-shared-b-aaea96/auth-test-model-aaea96-kserve-7fb8dc948c-d8b9c" Jul 10 01:05:53.087682 ip-10-0-130-146 kubenswrapper[2570]: I0710 01:05:53.087493 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ca012a7b-c0de-4b94-a970-726459db491f-home\") pod \"auth-test-model-aaea96-kserve-7fb8dc948c-d8b9c\" (UID: \"ca012a7b-c0de-4b94-a970-726459db491f\") " pod="ai-tenant-e2e-shared-b-aaea96/auth-test-model-aaea96-kserve-7fb8dc948c-d8b9c" Jul 10 01:05:53.087682 ip-10-0-130-146 kubenswrapper[2570]: I0710 01:05:53.087577 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ca012a7b-c0de-4b94-a970-726459db491f-dshm\") pod \"auth-test-model-aaea96-kserve-7fb8dc948c-d8b9c\" (UID: \"ca012a7b-c0de-4b94-a970-726459db491f\") " pod="ai-tenant-e2e-shared-b-aaea96/auth-test-model-aaea96-kserve-7fb8dc948c-d8b9c" Jul 10 01:05:53.087836 ip-10-0-130-146 kubenswrapper[2570]: I0710 01:05:53.087676 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ca012a7b-c0de-4b94-a970-726459db491f-tmp-dir\") pod \"auth-test-model-aaea96-kserve-7fb8dc948c-d8b9c\" (UID: \"ca012a7b-c0de-4b94-a970-726459db491f\") " pod="ai-tenant-e2e-shared-b-aaea96/auth-test-model-aaea96-kserve-7fb8dc948c-d8b9c" Jul 10 01:05:53.189037 ip-10-0-130-146 kubenswrapper[2570]: I0710 01:05:53.189008 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ca012a7b-c0de-4b94-a970-726459db491f-model-cache\") pod \"auth-test-model-aaea96-kserve-7fb8dc948c-d8b9c\" (UID: \"ca012a7b-c0de-4b94-a970-726459db491f\") " pod="ai-tenant-e2e-shared-b-aaea96/auth-test-model-aaea96-kserve-7fb8dc948c-d8b9c" Jul 10 01:05:53.189037 ip-10-0-130-146 kubenswrapper[2570]: I0710 01:05:53.189036 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ca012a7b-c0de-4b94-a970-726459db491f-tls-certs\") pod \"auth-test-model-aaea96-kserve-7fb8dc948c-d8b9c\" (UID: \"ca012a7b-c0de-4b94-a970-726459db491f\") " pod="ai-tenant-e2e-shared-b-aaea96/auth-test-model-aaea96-kserve-7fb8dc948c-d8b9c" Jul 10 01:05:53.189251 ip-10-0-130-146 kubenswrapper[2570]: I0710 01:05:53.189077 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2lszh\" (UniqueName: \"kubernetes.io/projected/ca012a7b-c0de-4b94-a970-726459db491f-kube-api-access-2lszh\") pod \"auth-test-model-aaea96-kserve-7fb8dc948c-d8b9c\" (UID: \"ca012a7b-c0de-4b94-a970-726459db491f\") " pod="ai-tenant-e2e-shared-b-aaea96/auth-test-model-aaea96-kserve-7fb8dc948c-d8b9c" Jul 10 01:05:53.189251 ip-10-0-130-146 kubenswrapper[2570]: I0710 01:05:53.189201 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ca012a7b-c0de-4b94-a970-726459db491f-home\") pod \"auth-test-model-aaea96-kserve-7fb8dc948c-d8b9c\" (UID: \"ca012a7b-c0de-4b94-a970-726459db491f\") " pod="ai-tenant-e2e-shared-b-aaea96/auth-test-model-aaea96-kserve-7fb8dc948c-d8b9c" Jul 10 01:05:53.189361 ip-10-0-130-146 kubenswrapper[2570]: I0710 01:05:53.189255 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ca012a7b-c0de-4b94-a970-726459db491f-dshm\") pod \"auth-test-model-aaea96-kserve-7fb8dc948c-d8b9c\" (UID: \"ca012a7b-c0de-4b94-a970-726459db491f\") " pod="ai-tenant-e2e-shared-b-aaea96/auth-test-model-aaea96-kserve-7fb8dc948c-d8b9c" Jul 10 01:05:53.189361 ip-10-0-130-146 kubenswrapper[2570]: I0710 01:05:53.189343 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ca012a7b-c0de-4b94-a970-726459db491f-tmp-dir\") pod \"auth-test-model-aaea96-kserve-7fb8dc948c-d8b9c\" (UID: \"ca012a7b-c0de-4b94-a970-726459db491f\") " pod="ai-tenant-e2e-shared-b-aaea96/auth-test-model-aaea96-kserve-7fb8dc948c-d8b9c" Jul 10 01:05:53.189470 ip-10-0-130-146 kubenswrapper[2570]: I0710 01:05:53.189434 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ca012a7b-c0de-4b94-a970-726459db491f-model-cache\") pod \"auth-test-model-aaea96-kserve-7fb8dc948c-d8b9c\" (UID: \"ca012a7b-c0de-4b94-a970-726459db491f\") " pod="ai-tenant-e2e-shared-b-aaea96/auth-test-model-aaea96-kserve-7fb8dc948c-d8b9c" Jul 10 01:05:53.189529 ip-10-0-130-146 kubenswrapper[2570]: I0710 01:05:53.189499 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ca012a7b-c0de-4b94-a970-726459db491f-home\") pod \"auth-test-model-aaea96-kserve-7fb8dc948c-d8b9c\" (UID: \"ca012a7b-c0de-4b94-a970-726459db491f\") " pod="ai-tenant-e2e-shared-b-aaea96/auth-test-model-aaea96-kserve-7fb8dc948c-d8b9c" Jul 10 01:05:53.189671 ip-10-0-130-146 kubenswrapper[2570]: I0710 01:05:53.189652 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ca012a7b-c0de-4b94-a970-726459db491f-tmp-dir\") pod \"auth-test-model-aaea96-kserve-7fb8dc948c-d8b9c\" (UID: \"ca012a7b-c0de-4b94-a970-726459db491f\") " pod="ai-tenant-e2e-shared-b-aaea96/auth-test-model-aaea96-kserve-7fb8dc948c-d8b9c" Jul 10 01:05:53.191293 ip-10-0-130-146 kubenswrapper[2570]: I0710 01:05:53.191272 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ca012a7b-c0de-4b94-a970-726459db491f-dshm\") pod \"auth-test-model-aaea96-kserve-7fb8dc948c-d8b9c\" (UID: \"ca012a7b-c0de-4b94-a970-726459db491f\") " pod="ai-tenant-e2e-shared-b-aaea96/auth-test-model-aaea96-kserve-7fb8dc948c-d8b9c" Jul 10 01:05:53.191564 ip-10-0-130-146 kubenswrapper[2570]: I0710 01:05:53.191544 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ca012a7b-c0de-4b94-a970-726459db491f-tls-certs\") pod \"auth-test-model-aaea96-kserve-7fb8dc948c-d8b9c\" (UID: \"ca012a7b-c0de-4b94-a970-726459db491f\") " pod="ai-tenant-e2e-shared-b-aaea96/auth-test-model-aaea96-kserve-7fb8dc948c-d8b9c" Jul 10 01:05:53.196589 ip-10-0-130-146 kubenswrapper[2570]: I0710 01:05:53.196568 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2lszh\" (UniqueName: \"kubernetes.io/projected/ca012a7b-c0de-4b94-a970-726459db491f-kube-api-access-2lszh\") pod \"auth-test-model-aaea96-kserve-7fb8dc948c-d8b9c\" (UID: \"ca012a7b-c0de-4b94-a970-726459db491f\") " pod="ai-tenant-e2e-shared-b-aaea96/auth-test-model-aaea96-kserve-7fb8dc948c-d8b9c" Jul 10 01:05:53.300438 ip-10-0-130-146 kubenswrapper[2570]: I0710 01:05:53.300380 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-aaea96/auth-test-model-aaea96-kserve-7fb8dc948c-d8b9c" Jul 10 01:05:53.432797 ip-10-0-130-146 kubenswrapper[2570]: I0710 01:05:53.432772 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-aaea96/auth-test-model-aaea96-kserve-7fb8dc948c-d8b9c"] Jul 10 01:05:53.434865 ip-10-0-130-146 kubenswrapper[2570]: W0710 01:05:53.434833 2570 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podca012a7b_c0de_4b94_a970_726459db491f.slice/crio-082449d59d2425aec95e8230e28d5e62576f801a029ad84b97885e757a82ef17 WatchSource:0}: Error finding container 082449d59d2425aec95e8230e28d5e62576f801a029ad84b97885e757a82ef17: Status 404 returned error can't find the container with id 082449d59d2425aec95e8230e28d5e62576f801a029ad84b97885e757a82ef17 Jul 10 01:05:54.247834 ip-10-0-130-146 kubenswrapper[2570]: I0710 01:05:54.247801 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-aaea96/auth-test-model-aaea96-kserve-7fb8dc948c-d8b9c" event={"ID":"ca012a7b-c0de-4b94-a970-726459db491f","Type":"ContainerStarted","Data":"d9483d2bd398b5533ed57837ef69a364d562ee1573325a755f864384291e72f3"} Jul 10 01:05:54.248237 ip-10-0-130-146 kubenswrapper[2570]: I0710 01:05:54.247843 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-aaea96/auth-test-model-aaea96-kserve-7fb8dc948c-d8b9c" event={"ID":"ca012a7b-c0de-4b94-a970-726459db491f","Type":"ContainerStarted","Data":"082449d59d2425aec95e8230e28d5e62576f801a029ad84b97885e757a82ef17"} Jul 10 01:05:54.266037 ip-10-0-130-146 kubenswrapper[2570]: I0710 01:05:54.265992 2570 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-b-aaea96/auth-test-model-aaea96-kserve-7fb8dc948c-d8b9c" podStartSLOduration=2.2659781 podStartE2EDuration="2.2659781s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-10 01:05:52 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 01:05:54.263743362 +0000 UTC m=+2373.312122106" watchObservedRunningTime="2026-07-10 01:05:54.2659781 +0000 UTC m=+2373.314356820" Jul 10 01:05:56.874909 ip-10-0-130-146 kubenswrapper[2570]: I0710 01:05:56.874878 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-g92hs"] Jul 10 01:06:03.300560 ip-10-0-130-146 kubenswrapper[2570]: I0710 01:06:03.300519 2570 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-b-aaea96/auth-test-model-aaea96-kserve-7fb8dc948c-d8b9c" Jul 10 01:06:03.300998 ip-10-0-130-146 kubenswrapper[2570]: I0710 01:06:03.300579 2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-b-aaea96/auth-test-model-aaea96-kserve-7fb8dc948c-d8b9c" Jul 10 01:06:03.312647 ip-10-0-130-146 kubenswrapper[2570]: I0710 01:06:03.312625 2570 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-b-aaea96/auth-test-model-aaea96-kserve-7fb8dc948c-d8b9c" Jul 10 01:06:04.318471 ip-10-0-130-146 kubenswrapper[2570]: I0710 01:06:04.318439 2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-b-aaea96/auth-test-model-aaea96-kserve-7fb8dc948c-d8b9c" Jul 10 01:06:11.285081 ip-10-0-130-146 kubenswrapper[2570]: I0710 01:06:11.285033 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-g92hs"] Jul 10 01:06:21.524522 ip-10-0-130-146 kubenswrapper[2570]: I0710 01:06:21.524490 2570 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7866497494-d6fcd_2b6cb4ca-79ff-42f0-b080-66af842f22d3/console-operator/2.log" Jul 10 01:06:21.527996 ip-10-0-130-146 kubenswrapper[2570]: I0710 01:06:21.527971 2570 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-mbs58_c4ef019a-364a-4933-ab85-de86d86167f1/ovn-acl-logging/0.log" Jul 10 01:06:21.529330 ip-10-0-130-146 kubenswrapper[2570]: I0710 01:06:21.529310 2570 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7866497494-d6fcd_2b6cb4ca-79ff-42f0-b080-66af842f22d3/console-operator/2.log" Jul 10 01:06:21.532579 ip-10-0-130-146 kubenswrapper[2570]: I0710 01:06:21.532561 2570 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-mbs58_c4ef019a-364a-4933-ab85-de86d86167f1/ovn-acl-logging/0.log" Jul 10 01:06:28.986648 ip-10-0-130-146 kubenswrapper[2570]: I0710 01:06:28.986611 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-g92hs"] Jul 10 01:06:43.180300 ip-10-0-130-146 kubenswrapper[2570]: I0710 01:06:43.180204 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-g92hs"] Jul 10 01:07:01.278702 ip-10-0-130-146 kubenswrapper[2570]: I0710 01:07:01.278664 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-g92hs"] Jul 10 01:07:15.694585 ip-10-0-130-146 kubenswrapper[2570]: I0710 01:07:15.694543 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-g92hs"] Jul 10 01:07:30.380142 ip-10-0-130-146 kubenswrapper[2570]: I0710 01:07:30.380110 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-g92hs"] Jul 10 01:07:45.085209 ip-10-0-130-146 kubenswrapper[2570]: I0710 01:07:45.085175 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-g92hs"] Jul 10 01:07:48.478762 ip-10-0-130-146 kubenswrapper[2570]: I0710 01:07:48.478731 2570 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-48a750/auth-test-model-48a750-kserve-66ccc45475-v489h"] Jul 10 01:07:49.490472 ip-10-0-130-146 kubenswrapper[2570]: I0710 01:07:49.490439 2570 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-aaea96/auth-test-model-aaea96-kserve-7fb8dc948c-d8b9c"] Jul 10 01:08:03.484368 ip-10-0-130-146 kubenswrapper[2570]: I0710 01:08:03.484338 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-g92hs"] Jul 10 01:08:03.499102 ip-10-0-130-146 kubenswrapper[2570]: I0710 01:08:03.499072 2570 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-shared-a-48a750/auth-test-model-48a750-kserve-66ccc45475-v489h" podUID="e7e455f3-aa01-4630-b72a-fedfce8c8b0c" containerName="main" containerID="cri-o://6da943f94f43baf797aed197046b5586eac79d6c48fffb191ab454369b7bc695" gracePeriod=45 Jul 10 01:08:03.747351 ip-10-0-130-146 kubenswrapper[2570]: I0710 01:08:03.747288 2570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-48a750/auth-test-model-48a750-kserve-66ccc45475-v489h" Jul 10 01:08:03.827861 ip-10-0-130-146 kubenswrapper[2570]: I0710 01:08:03.827834 2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-ctmsw\" (UniqueName: \"kubernetes.io/projected/e7e455f3-aa01-4630-b72a-fedfce8c8b0c-kube-api-access-ctmsw\") pod \"e7e455f3-aa01-4630-b72a-fedfce8c8b0c\" (UID: \"e7e455f3-aa01-4630-b72a-fedfce8c8b0c\") " Jul 10 01:08:03.828017 ip-10-0-130-146 kubenswrapper[2570]: I0710 01:08:03.827882 2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e7e455f3-aa01-4630-b72a-fedfce8c8b0c-home\") pod \"e7e455f3-aa01-4630-b72a-fedfce8c8b0c\" (UID: \"e7e455f3-aa01-4630-b72a-fedfce8c8b0c\") " Jul 10 01:08:03.828017 ip-10-0-130-146 kubenswrapper[2570]: I0710 01:08:03.827907 2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e7e455f3-aa01-4630-b72a-fedfce8c8b0c-tmp-dir\") pod \"e7e455f3-aa01-4630-b72a-fedfce8c8b0c\" (UID: \"e7e455f3-aa01-4630-b72a-fedfce8c8b0c\") " Jul 10 01:08:03.828017 ip-10-0-130-146 kubenswrapper[2570]: I0710 01:08:03.827931 2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e7e455f3-aa01-4630-b72a-fedfce8c8b0c-tls-certs\") pod \"e7e455f3-aa01-4630-b72a-fedfce8c8b0c\" (UID: \"e7e455f3-aa01-4630-b72a-fedfce8c8b0c\") " Jul 10 01:08:03.828017 ip-10-0-130-146 kubenswrapper[2570]: I0710 01:08:03.827968 2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e7e455f3-aa01-4630-b72a-fedfce8c8b0c-model-cache\") pod \"e7e455f3-aa01-4630-b72a-fedfce8c8b0c\" (UID: \"e7e455f3-aa01-4630-b72a-fedfce8c8b0c\") " Jul 10 01:08:03.828017 ip-10-0-130-146 kubenswrapper[2570]: I0710 01:08:03.828008 2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e7e455f3-aa01-4630-b72a-fedfce8c8b0c-dshm\") pod \"e7e455f3-aa01-4630-b72a-fedfce8c8b0c\" (UID: \"e7e455f3-aa01-4630-b72a-fedfce8c8b0c\") " Jul 10 01:08:03.828316 ip-10-0-130-146 kubenswrapper[2570]: I0710 01:08:03.828148 2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e7e455f3-aa01-4630-b72a-fedfce8c8b0c-home" (OuterVolumeSpecName: "home") pod "e7e455f3-aa01-4630-b72a-fedfce8c8b0c" (UID: "e7e455f3-aa01-4630-b72a-fedfce8c8b0c"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 10 01:08:03.828316 ip-10-0-130-146 kubenswrapper[2570]: I0710 01:08:03.828227 2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e7e455f3-aa01-4630-b72a-fedfce8c8b0c-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "e7e455f3-aa01-4630-b72a-fedfce8c8b0c" (UID: "e7e455f3-aa01-4630-b72a-fedfce8c8b0c"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 10 01:08:03.828316 ip-10-0-130-146 kubenswrapper[2570]: I0710 01:08:03.828260 2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e7e455f3-aa01-4630-b72a-fedfce8c8b0c-model-cache" (OuterVolumeSpecName: "model-cache") pod "e7e455f3-aa01-4630-b72a-fedfce8c8b0c" (UID: "e7e455f3-aa01-4630-b72a-fedfce8c8b0c"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 10 01:08:03.828433 ip-10-0-130-146 kubenswrapper[2570]: I0710 01:08:03.828336 2570 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e7e455f3-aa01-4630-b72a-fedfce8c8b0c-home\") on node \"ip-10-0-130-146.ec2.internal\" DevicePath \"\"" Jul 10 01:08:03.828433 ip-10-0-130-146 kubenswrapper[2570]: I0710 01:08:03.828351 2570 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e7e455f3-aa01-4630-b72a-fedfce8c8b0c-tmp-dir\") on node \"ip-10-0-130-146.ec2.internal\" DevicePath \"\"" Jul 10 01:08:03.828433 ip-10-0-130-146 kubenswrapper[2570]: I0710 01:08:03.828371 2570 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e7e455f3-aa01-4630-b72a-fedfce8c8b0c-model-cache\") on node \"ip-10-0-130-146.ec2.internal\" DevicePath \"\"" Jul 10 01:08:03.830104 ip-10-0-130-146 kubenswrapper[2570]: I0710 01:08:03.830081 2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e7e455f3-aa01-4630-b72a-fedfce8c8b0c-dshm" (OuterVolumeSpecName: "dshm") pod "e7e455f3-aa01-4630-b72a-fedfce8c8b0c" (UID: "e7e455f3-aa01-4630-b72a-fedfce8c8b0c"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 10 01:08:03.830208 ip-10-0-130-146 kubenswrapper[2570]: I0710 01:08:03.830131 2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e7e455f3-aa01-4630-b72a-fedfce8c8b0c-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "e7e455f3-aa01-4630-b72a-fedfce8c8b0c" (UID: "e7e455f3-aa01-4630-b72a-fedfce8c8b0c"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 10 01:08:03.830208 ip-10-0-130-146 kubenswrapper[2570]: I0710 01:08:03.830152 2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e7e455f3-aa01-4630-b72a-fedfce8c8b0c-kube-api-access-ctmsw" (OuterVolumeSpecName: "kube-api-access-ctmsw") pod "e7e455f3-aa01-4630-b72a-fedfce8c8b0c" (UID: "e7e455f3-aa01-4630-b72a-fedfce8c8b0c"). InnerVolumeSpecName "kube-api-access-ctmsw". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 10 01:08:03.928932 ip-10-0-130-146 kubenswrapper[2570]: I0710 01:08:03.928902 2570 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e7e455f3-aa01-4630-b72a-fedfce8c8b0c-dshm\") on node \"ip-10-0-130-146.ec2.internal\" DevicePath \"\"" Jul 10 01:08:03.928932 ip-10-0-130-146 kubenswrapper[2570]: I0710 01:08:03.928926 2570 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ctmsw\" (UniqueName: \"kubernetes.io/projected/e7e455f3-aa01-4630-b72a-fedfce8c8b0c-kube-api-access-ctmsw\") on node \"ip-10-0-130-146.ec2.internal\" DevicePath \"\"" Jul 10 01:08:03.928932 ip-10-0-130-146 kubenswrapper[2570]: I0710 01:08:03.928937 2570 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e7e455f3-aa01-4630-b72a-fedfce8c8b0c-tls-certs\") on node \"ip-10-0-130-146.ec2.internal\" DevicePath \"\"" Jul 10 01:08:03.957908 ip-10-0-130-146 kubenswrapper[2570]: I0710 01:08:03.957876 2570 generic.go:356] "Generic (PLEG): container finished" podID="e7e455f3-aa01-4630-b72a-fedfce8c8b0c" containerID="6da943f94f43baf797aed197046b5586eac79d6c48fffb191ab454369b7bc695" exitCode=0 Jul 10 01:08:03.958072 ip-10-0-130-146 kubenswrapper[2570]: I0710 01:08:03.957928 2570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-48a750/auth-test-model-48a750-kserve-66ccc45475-v489h" Jul 10 01:08:03.958072 ip-10-0-130-146 kubenswrapper[2570]: I0710 01:08:03.957967 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-48a750/auth-test-model-48a750-kserve-66ccc45475-v489h" event={"ID":"e7e455f3-aa01-4630-b72a-fedfce8c8b0c","Type":"ContainerDied","Data":"6da943f94f43baf797aed197046b5586eac79d6c48fffb191ab454369b7bc695"} Jul 10 01:08:03.958072 ip-10-0-130-146 kubenswrapper[2570]: I0710 01:08:03.958007 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-48a750/auth-test-model-48a750-kserve-66ccc45475-v489h" event={"ID":"e7e455f3-aa01-4630-b72a-fedfce8c8b0c","Type":"ContainerDied","Data":"e2ef5819eee4d80a79d62ffeca7ee093e3c799763353bf758abb32bff359c805"} Jul 10 01:08:03.958072 ip-10-0-130-146 kubenswrapper[2570]: I0710 01:08:03.958024 2570 scope.go:117] "RemoveContainer" containerID="6da943f94f43baf797aed197046b5586eac79d6c48fffb191ab454369b7bc695" Jul 10 01:08:03.967452 ip-10-0-130-146 kubenswrapper[2570]: I0710 01:08:03.967435 2570 scope.go:117] "RemoveContainer" containerID="6da943f94f43baf797aed197046b5586eac79d6c48fffb191ab454369b7bc695" Jul 10 01:08:03.967693 ip-10-0-130-146 kubenswrapper[2570]: E0710 01:08:03.967674 2570 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6da943f94f43baf797aed197046b5586eac79d6c48fffb191ab454369b7bc695\": container with ID starting with 6da943f94f43baf797aed197046b5586eac79d6c48fffb191ab454369b7bc695 not found: ID does not exist" containerID="6da943f94f43baf797aed197046b5586eac79d6c48fffb191ab454369b7bc695" Jul 10 01:08:03.967749 ip-10-0-130-146 kubenswrapper[2570]: I0710 01:08:03.967701 2570 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6da943f94f43baf797aed197046b5586eac79d6c48fffb191ab454369b7bc695"} err="failed to get container status \"6da943f94f43baf797aed197046b5586eac79d6c48fffb191ab454369b7bc695\": rpc error: code = NotFound desc = could not find container \"6da943f94f43baf797aed197046b5586eac79d6c48fffb191ab454369b7bc695\": container with ID starting with 6da943f94f43baf797aed197046b5586eac79d6c48fffb191ab454369b7bc695 not found: ID does not exist" Jul 10 01:08:03.981988 ip-10-0-130-146 kubenswrapper[2570]: I0710 01:08:03.981958 2570 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-48a750/auth-test-model-48a750-kserve-66ccc45475-v489h"] Jul 10 01:08:03.987648 ip-10-0-130-146 kubenswrapper[2570]: I0710 01:08:03.987624 2570 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-a-48a750/auth-test-model-48a750-kserve-66ccc45475-v489h"] Jul 10 01:08:04.512779 ip-10-0-130-146 kubenswrapper[2570]: I0710 01:08:04.512740 2570 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-shared-b-aaea96/auth-test-model-aaea96-kserve-7fb8dc948c-d8b9c" podUID="ca012a7b-c0de-4b94-a970-726459db491f" containerName="main" containerID="cri-o://d9483d2bd398b5533ed57837ef69a364d562ee1573325a755f864384291e72f3" gracePeriod=45 Jul 10 01:08:04.762517 ip-10-0-130-146 kubenswrapper[2570]: I0710 01:08:04.762487 2570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-aaea96/auth-test-model-aaea96-kserve-7fb8dc948c-d8b9c" Jul 10 01:08:04.836762 ip-10-0-130-146 kubenswrapper[2570]: I0710 01:08:04.836684 2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ca012a7b-c0de-4b94-a970-726459db491f-dshm\") pod \"ca012a7b-c0de-4b94-a970-726459db491f\" (UID: \"ca012a7b-c0de-4b94-a970-726459db491f\") " Jul 10 01:08:04.836762 ip-10-0-130-146 kubenswrapper[2570]: I0710 01:08:04.836727 2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ca012a7b-c0de-4b94-a970-726459db491f-model-cache\") pod \"ca012a7b-c0de-4b94-a970-726459db491f\" (UID: \"ca012a7b-c0de-4b94-a970-726459db491f\") " Jul 10 01:08:04.836762 ip-10-0-130-146 kubenswrapper[2570]: I0710 01:08:04.836764 2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ca012a7b-c0de-4b94-a970-726459db491f-tmp-dir\") pod \"ca012a7b-c0de-4b94-a970-726459db491f\" (UID: \"ca012a7b-c0de-4b94-a970-726459db491f\") " Jul 10 01:08:04.837022 ip-10-0-130-146 kubenswrapper[2570]: I0710 01:08:04.836798 2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ca012a7b-c0de-4b94-a970-726459db491f-tls-certs\") pod \"ca012a7b-c0de-4b94-a970-726459db491f\" (UID: \"ca012a7b-c0de-4b94-a970-726459db491f\") " Jul 10 01:08:04.837022 ip-10-0-130-146 kubenswrapper[2570]: I0710 01:08:04.836838 2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-2lszh\" (UniqueName: \"kubernetes.io/projected/ca012a7b-c0de-4b94-a970-726459db491f-kube-api-access-2lszh\") pod \"ca012a7b-c0de-4b94-a970-726459db491f\" (UID: \"ca012a7b-c0de-4b94-a970-726459db491f\") " Jul 10 01:08:04.837022 ip-10-0-130-146 kubenswrapper[2570]: I0710 01:08:04.836876 2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ca012a7b-c0de-4b94-a970-726459db491f-home\") pod \"ca012a7b-c0de-4b94-a970-726459db491f\" (UID: \"ca012a7b-c0de-4b94-a970-726459db491f\") " Jul 10 01:08:04.837022 ip-10-0-130-146 kubenswrapper[2570]: I0710 01:08:04.836998 2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ca012a7b-c0de-4b94-a970-726459db491f-model-cache" (OuterVolumeSpecName: "model-cache") pod "ca012a7b-c0de-4b94-a970-726459db491f" (UID: "ca012a7b-c0de-4b94-a970-726459db491f"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 10 01:08:04.837264 ip-10-0-130-146 kubenswrapper[2570]: I0710 01:08:04.837042 2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ca012a7b-c0de-4b94-a970-726459db491f-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "ca012a7b-c0de-4b94-a970-726459db491f" (UID: "ca012a7b-c0de-4b94-a970-726459db491f"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 10 01:08:04.837264 ip-10-0-130-146 kubenswrapper[2570]: I0710 01:08:04.837191 2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ca012a7b-c0de-4b94-a970-726459db491f-home" (OuterVolumeSpecName: "home") pod "ca012a7b-c0de-4b94-a970-726459db491f" (UID: "ca012a7b-c0de-4b94-a970-726459db491f"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 10 01:08:04.837264 ip-10-0-130-146 kubenswrapper[2570]: I0710 01:08:04.837206 2570 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ca012a7b-c0de-4b94-a970-726459db491f-model-cache\") on node \"ip-10-0-130-146.ec2.internal\" DevicePath \"\"" Jul 10 01:08:04.837264 ip-10-0-130-146 kubenswrapper[2570]: I0710 01:08:04.837222 2570 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ca012a7b-c0de-4b94-a970-726459db491f-tmp-dir\") on node \"ip-10-0-130-146.ec2.internal\" DevicePath \"\"" Jul 10 01:08:04.838946 ip-10-0-130-146 kubenswrapper[2570]: I0710 01:08:04.838924 2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ca012a7b-c0de-4b94-a970-726459db491f-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "ca012a7b-c0de-4b94-a970-726459db491f" (UID: "ca012a7b-c0de-4b94-a970-726459db491f"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 10 01:08:04.839043 ip-10-0-130-146 kubenswrapper[2570]: I0710 01:08:04.838972 2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ca012a7b-c0de-4b94-a970-726459db491f-dshm" (OuterVolumeSpecName: "dshm") pod "ca012a7b-c0de-4b94-a970-726459db491f" (UID: "ca012a7b-c0de-4b94-a970-726459db491f"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 10 01:08:04.839043 ip-10-0-130-146 kubenswrapper[2570]: I0710 01:08:04.839027 2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ca012a7b-c0de-4b94-a970-726459db491f-kube-api-access-2lszh" (OuterVolumeSpecName: "kube-api-access-2lszh") pod "ca012a7b-c0de-4b94-a970-726459db491f" (UID: "ca012a7b-c0de-4b94-a970-726459db491f"). InnerVolumeSpecName "kube-api-access-2lszh". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 10 01:08:04.937930 ip-10-0-130-146 kubenswrapper[2570]: I0710 01:08:04.937904 2570 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ca012a7b-c0de-4b94-a970-726459db491f-tls-certs\") on node \"ip-10-0-130-146.ec2.internal\" DevicePath \"\"" Jul 10 01:08:04.937930 ip-10-0-130-146 kubenswrapper[2570]: I0710 01:08:04.937929 2570 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2lszh\" (UniqueName: \"kubernetes.io/projected/ca012a7b-c0de-4b94-a970-726459db491f-kube-api-access-2lszh\") on node \"ip-10-0-130-146.ec2.internal\" DevicePath \"\"" Jul 10 01:08:04.938111 ip-10-0-130-146 kubenswrapper[2570]: I0710 01:08:04.937938 2570 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ca012a7b-c0de-4b94-a970-726459db491f-home\") on node \"ip-10-0-130-146.ec2.internal\" DevicePath \"\"" Jul 10 01:08:04.938111 ip-10-0-130-146 kubenswrapper[2570]: I0710 01:08:04.937946 2570 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ca012a7b-c0de-4b94-a970-726459db491f-dshm\") on node \"ip-10-0-130-146.ec2.internal\" DevicePath \"\"" Jul 10 01:08:04.965708 ip-10-0-130-146 kubenswrapper[2570]: I0710 01:08:04.965678 2570 generic.go:356] "Generic (PLEG): container finished" podID="ca012a7b-c0de-4b94-a970-726459db491f" containerID="d9483d2bd398b5533ed57837ef69a364d562ee1573325a755f864384291e72f3" exitCode=0 Jul 10 01:08:04.965905 ip-10-0-130-146 kubenswrapper[2570]: I0710 01:08:04.965728 2570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-aaea96/auth-test-model-aaea96-kserve-7fb8dc948c-d8b9c" Jul 10 01:08:04.965905 ip-10-0-130-146 kubenswrapper[2570]: I0710 01:08:04.965737 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-aaea96/auth-test-model-aaea96-kserve-7fb8dc948c-d8b9c" event={"ID":"ca012a7b-c0de-4b94-a970-726459db491f","Type":"ContainerDied","Data":"d9483d2bd398b5533ed57837ef69a364d562ee1573325a755f864384291e72f3"} Jul 10 01:08:04.965905 ip-10-0-130-146 kubenswrapper[2570]: I0710 01:08:04.965788 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-aaea96/auth-test-model-aaea96-kserve-7fb8dc948c-d8b9c" event={"ID":"ca012a7b-c0de-4b94-a970-726459db491f","Type":"ContainerDied","Data":"082449d59d2425aec95e8230e28d5e62576f801a029ad84b97885e757a82ef17"} Jul 10 01:08:04.965905 ip-10-0-130-146 kubenswrapper[2570]: I0710 01:08:04.965810 2570 scope.go:117] "RemoveContainer" containerID="d9483d2bd398b5533ed57837ef69a364d562ee1573325a755f864384291e72f3" Jul 10 01:08:04.974909 ip-10-0-130-146 kubenswrapper[2570]: I0710 01:08:04.974894 2570 scope.go:117] "RemoveContainer" containerID="d9483d2bd398b5533ed57837ef69a364d562ee1573325a755f864384291e72f3" Jul 10 01:08:04.975242 ip-10-0-130-146 kubenswrapper[2570]: E0710 01:08:04.975221 2570 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d9483d2bd398b5533ed57837ef69a364d562ee1573325a755f864384291e72f3\": container with ID starting with d9483d2bd398b5533ed57837ef69a364d562ee1573325a755f864384291e72f3 not found: ID does not exist" containerID="d9483d2bd398b5533ed57837ef69a364d562ee1573325a755f864384291e72f3" Jul 10 01:08:04.975333 ip-10-0-130-146 kubenswrapper[2570]: I0710 01:08:04.975250 2570 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d9483d2bd398b5533ed57837ef69a364d562ee1573325a755f864384291e72f3"} err="failed to get container status \"d9483d2bd398b5533ed57837ef69a364d562ee1573325a755f864384291e72f3\": rpc error: code = NotFound desc = could not find container \"d9483d2bd398b5533ed57837ef69a364d562ee1573325a755f864384291e72f3\": container with ID starting with d9483d2bd398b5533ed57837ef69a364d562ee1573325a755f864384291e72f3 not found: ID does not exist" Jul 10 01:08:04.986674 ip-10-0-130-146 kubenswrapper[2570]: I0710 01:08:04.986643 2570 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-aaea96/auth-test-model-aaea96-kserve-7fb8dc948c-d8b9c"] Jul 10 01:08:04.992273 ip-10-0-130-146 kubenswrapper[2570]: I0710 01:08:04.992254 2570 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-b-aaea96/auth-test-model-aaea96-kserve-7fb8dc948c-d8b9c"] Jul 10 01:08:05.453977 ip-10-0-130-146 kubenswrapper[2570]: I0710 01:08:05.453944 2570 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="ca012a7b-c0de-4b94-a970-726459db491f" path="/var/lib/kubelet/pods/ca012a7b-c0de-4b94-a970-726459db491f/volumes" Jul 10 01:08:05.454396 ip-10-0-130-146 kubenswrapper[2570]: I0710 01:08:05.454382 2570 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="e7e455f3-aa01-4630-b72a-fedfce8c8b0c" path="/var/lib/kubelet/pods/e7e455f3-aa01-4630-b72a-fedfce8c8b0c/volumes" Jul 10 01:08:38.585553 ip-10-0-130-146 kubenswrapper[2570]: I0710 01:08:38.585522 2570 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-b-aaea96/sub-test-model-aaea96-kserve-648db985d5-tccg9"] Jul 10 01:08:38.585939 ip-10-0-130-146 kubenswrapper[2570]: I0710 01:08:38.585874 2570 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e7e455f3-aa01-4630-b72a-fedfce8c8b0c" containerName="main" Jul 10 01:08:38.585939 ip-10-0-130-146 kubenswrapper[2570]: I0710 01:08:38.585885 2570 state_mem.go:107] "Deleted CPUSet assignment" podUID="e7e455f3-aa01-4630-b72a-fedfce8c8b0c" containerName="main" Jul 10 01:08:38.585939 ip-10-0-130-146 kubenswrapper[2570]: I0710 01:08:38.585898 2570 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ca012a7b-c0de-4b94-a970-726459db491f" containerName="main" Jul 10 01:08:38.585939 ip-10-0-130-146 kubenswrapper[2570]: I0710 01:08:38.585903 2570 state_mem.go:107] "Deleted CPUSet assignment" podUID="ca012a7b-c0de-4b94-a970-726459db491f" containerName="main" Jul 10 01:08:38.586097 ip-10-0-130-146 kubenswrapper[2570]: I0710 01:08:38.585968 2570 memory_manager.go:368] "RemoveStaleState removing state" podUID="ca012a7b-c0de-4b94-a970-726459db491f" containerName="main" Jul 10 01:08:38.586097 ip-10-0-130-146 kubenswrapper[2570]: I0710 01:08:38.585979 2570 memory_manager.go:368] "RemoveStaleState removing state" podUID="e7e455f3-aa01-4630-b72a-fedfce8c8b0c" containerName="main" Jul 10 01:08:38.588980 ip-10-0-130-146 kubenswrapper[2570]: I0710 01:08:38.588961 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-aaea96/sub-test-model-aaea96-kserve-648db985d5-tccg9" Jul 10 01:08:38.591515 ip-10-0-130-146 kubenswrapper[2570]: I0710 01:08:38.591498 2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-aaea96\"/\"default-dockercfg-dz7xs\"" Jul 10 01:08:38.592568 ip-10-0-130-146 kubenswrapper[2570]: I0710 01:08:38.592541 2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-aaea96\"/\"openshift-service-ca.crt\"" Jul 10 01:08:38.592672 ip-10-0-130-146 kubenswrapper[2570]: I0710 01:08:38.592654 2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-aaea96\"/\"kube-root-ca.crt\"" Jul 10 01:08:38.592736 ip-10-0-130-146 kubenswrapper[2570]: I0710 01:08:38.592721 2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-aaea96\"/\"sub-test-model-aaea96-kserve-self-signed-certs\"" Jul 10 01:08:38.599706 ip-10-0-130-146 kubenswrapper[2570]: I0710 01:08:38.599686 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-aaea96/sub-test-model-aaea96-kserve-648db985d5-tccg9"] Jul 10 01:08:38.732644 ip-10-0-130-146 kubenswrapper[2570]: I0710 01:08:38.732602 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/55246994-a20a-41be-b350-4bb216ab38dc-home\") pod \"sub-test-model-aaea96-kserve-648db985d5-tccg9\" (UID: \"55246994-a20a-41be-b350-4bb216ab38dc\") " pod="ai-tenant-e2e-shared-b-aaea96/sub-test-model-aaea96-kserve-648db985d5-tccg9" Jul 10 01:08:38.732644 ip-10-0-130-146 kubenswrapper[2570]: I0710 01:08:38.732645 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/55246994-a20a-41be-b350-4bb216ab38dc-dshm\") pod \"sub-test-model-aaea96-kserve-648db985d5-tccg9\" (UID: \"55246994-a20a-41be-b350-4bb216ab38dc\") " pod="ai-tenant-e2e-shared-b-aaea96/sub-test-model-aaea96-kserve-648db985d5-tccg9" Jul 10 01:08:38.732844 ip-10-0-130-146 kubenswrapper[2570]: I0710 01:08:38.732725 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/55246994-a20a-41be-b350-4bb216ab38dc-tls-certs\") pod \"sub-test-model-aaea96-kserve-648db985d5-tccg9\" (UID: \"55246994-a20a-41be-b350-4bb216ab38dc\") " pod="ai-tenant-e2e-shared-b-aaea96/sub-test-model-aaea96-kserve-648db985d5-tccg9" Jul 10 01:08:38.732844 ip-10-0-130-146 kubenswrapper[2570]: I0710 01:08:38.732801 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/55246994-a20a-41be-b350-4bb216ab38dc-tmp-dir\") pod \"sub-test-model-aaea96-kserve-648db985d5-tccg9\" (UID: \"55246994-a20a-41be-b350-4bb216ab38dc\") " pod="ai-tenant-e2e-shared-b-aaea96/sub-test-model-aaea96-kserve-648db985d5-tccg9" Jul 10 01:08:38.732916 ip-10-0-130-146 kubenswrapper[2570]: I0710 01:08:38.732842 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tk4kp\" (UniqueName: \"kubernetes.io/projected/55246994-a20a-41be-b350-4bb216ab38dc-kube-api-access-tk4kp\") pod \"sub-test-model-aaea96-kserve-648db985d5-tccg9\" (UID: \"55246994-a20a-41be-b350-4bb216ab38dc\") " pod="ai-tenant-e2e-shared-b-aaea96/sub-test-model-aaea96-kserve-648db985d5-tccg9" Jul 10 01:08:38.732916 ip-10-0-130-146 kubenswrapper[2570]: I0710 01:08:38.732887 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/55246994-a20a-41be-b350-4bb216ab38dc-model-cache\") pod \"sub-test-model-aaea96-kserve-648db985d5-tccg9\" (UID: \"55246994-a20a-41be-b350-4bb216ab38dc\") " pod="ai-tenant-e2e-shared-b-aaea96/sub-test-model-aaea96-kserve-648db985d5-tccg9" Jul 10 01:08:38.833324 ip-10-0-130-146 kubenswrapper[2570]: I0710 01:08:38.833291 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/55246994-a20a-41be-b350-4bb216ab38dc-tls-certs\") pod \"sub-test-model-aaea96-kserve-648db985d5-tccg9\" (UID: \"55246994-a20a-41be-b350-4bb216ab38dc\") " pod="ai-tenant-e2e-shared-b-aaea96/sub-test-model-aaea96-kserve-648db985d5-tccg9" Jul 10 01:08:38.833501 ip-10-0-130-146 kubenswrapper[2570]: I0710 01:08:38.833342 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/55246994-a20a-41be-b350-4bb216ab38dc-tmp-dir\") pod \"sub-test-model-aaea96-kserve-648db985d5-tccg9\" (UID: \"55246994-a20a-41be-b350-4bb216ab38dc\") " pod="ai-tenant-e2e-shared-b-aaea96/sub-test-model-aaea96-kserve-648db985d5-tccg9" Jul 10 01:08:38.833501 ip-10-0-130-146 kubenswrapper[2570]: I0710 01:08:38.833366 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tk4kp\" (UniqueName: \"kubernetes.io/projected/55246994-a20a-41be-b350-4bb216ab38dc-kube-api-access-tk4kp\") pod \"sub-test-model-aaea96-kserve-648db985d5-tccg9\" (UID: \"55246994-a20a-41be-b350-4bb216ab38dc\") " pod="ai-tenant-e2e-shared-b-aaea96/sub-test-model-aaea96-kserve-648db985d5-tccg9" Jul 10 01:08:38.833501 ip-10-0-130-146 kubenswrapper[2570]: I0710 01:08:38.833401 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/55246994-a20a-41be-b350-4bb216ab38dc-model-cache\") pod \"sub-test-model-aaea96-kserve-648db985d5-tccg9\" (UID: \"55246994-a20a-41be-b350-4bb216ab38dc\") " pod="ai-tenant-e2e-shared-b-aaea96/sub-test-model-aaea96-kserve-648db985d5-tccg9" Jul 10 01:08:38.833501 ip-10-0-130-146 kubenswrapper[2570]: I0710 01:08:38.833420 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/55246994-a20a-41be-b350-4bb216ab38dc-home\") pod \"sub-test-model-aaea96-kserve-648db985d5-tccg9\" (UID: \"55246994-a20a-41be-b350-4bb216ab38dc\") " pod="ai-tenant-e2e-shared-b-aaea96/sub-test-model-aaea96-kserve-648db985d5-tccg9" Jul 10 01:08:38.833501 ip-10-0-130-146 kubenswrapper[2570]: I0710 01:08:38.833434 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/55246994-a20a-41be-b350-4bb216ab38dc-dshm\") pod \"sub-test-model-aaea96-kserve-648db985d5-tccg9\" (UID: \"55246994-a20a-41be-b350-4bb216ab38dc\") " pod="ai-tenant-e2e-shared-b-aaea96/sub-test-model-aaea96-kserve-648db985d5-tccg9" Jul 10 01:08:38.833761 ip-10-0-130-146 kubenswrapper[2570]: I0710 01:08:38.833741 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/55246994-a20a-41be-b350-4bb216ab38dc-tmp-dir\") pod \"sub-test-model-aaea96-kserve-648db985d5-tccg9\" (UID: \"55246994-a20a-41be-b350-4bb216ab38dc\") " pod="ai-tenant-e2e-shared-b-aaea96/sub-test-model-aaea96-kserve-648db985d5-tccg9" Jul 10 01:08:38.833821 ip-10-0-130-146 kubenswrapper[2570]: I0710 01:08:38.833790 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/55246994-a20a-41be-b350-4bb216ab38dc-model-cache\") pod \"sub-test-model-aaea96-kserve-648db985d5-tccg9\" (UID: \"55246994-a20a-41be-b350-4bb216ab38dc\") " pod="ai-tenant-e2e-shared-b-aaea96/sub-test-model-aaea96-kserve-648db985d5-tccg9" Jul 10 01:08:38.833821 ip-10-0-130-146 kubenswrapper[2570]: I0710 01:08:38.833802 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/55246994-a20a-41be-b350-4bb216ab38dc-home\") pod \"sub-test-model-aaea96-kserve-648db985d5-tccg9\" (UID: \"55246994-a20a-41be-b350-4bb216ab38dc\") " pod="ai-tenant-e2e-shared-b-aaea96/sub-test-model-aaea96-kserve-648db985d5-tccg9" Jul 10 01:08:38.835730 ip-10-0-130-146 kubenswrapper[2570]: I0710 01:08:38.835678 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/55246994-a20a-41be-b350-4bb216ab38dc-dshm\") pod \"sub-test-model-aaea96-kserve-648db985d5-tccg9\" (UID: \"55246994-a20a-41be-b350-4bb216ab38dc\") " pod="ai-tenant-e2e-shared-b-aaea96/sub-test-model-aaea96-kserve-648db985d5-tccg9" Jul 10 01:08:38.835964 ip-10-0-130-146 kubenswrapper[2570]: I0710 01:08:38.835944 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/55246994-a20a-41be-b350-4bb216ab38dc-tls-certs\") pod \"sub-test-model-aaea96-kserve-648db985d5-tccg9\" (UID: \"55246994-a20a-41be-b350-4bb216ab38dc\") " pod="ai-tenant-e2e-shared-b-aaea96/sub-test-model-aaea96-kserve-648db985d5-tccg9" Jul 10 01:08:38.844619 ip-10-0-130-146 kubenswrapper[2570]: I0710 01:08:38.844599 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-tk4kp\" (UniqueName: \"kubernetes.io/projected/55246994-a20a-41be-b350-4bb216ab38dc-kube-api-access-tk4kp\") pod \"sub-test-model-aaea96-kserve-648db985d5-tccg9\" (UID: \"55246994-a20a-41be-b350-4bb216ab38dc\") " pod="ai-tenant-e2e-shared-b-aaea96/sub-test-model-aaea96-kserve-648db985d5-tccg9" Jul 10 01:08:38.899114 ip-10-0-130-146 kubenswrapper[2570]: I0710 01:08:38.899086 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-aaea96/sub-test-model-aaea96-kserve-648db985d5-tccg9" Jul 10 01:08:39.032173 ip-10-0-130-146 kubenswrapper[2570]: I0710 01:08:39.032145 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-aaea96/sub-test-model-aaea96-kserve-648db985d5-tccg9"] Jul 10 01:08:39.033691 ip-10-0-130-146 kubenswrapper[2570]: W0710 01:08:39.033655 2570 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod55246994_a20a_41be_b350_4bb216ab38dc.slice/crio-1a60df36616e846a45934002794fb24ac05e9f5f1b2a756595144c68aeeff60f WatchSource:0}: Error finding container 1a60df36616e846a45934002794fb24ac05e9f5f1b2a756595144c68aeeff60f: Status 404 returned error can't find the container with id 1a60df36616e846a45934002794fb24ac05e9f5f1b2a756595144c68aeeff60f Jul 10 01:08:39.151276 ip-10-0-130-146 kubenswrapper[2570]: I0710 01:08:39.151243 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-aaea96/sub-test-model-aaea96-kserve-648db985d5-tccg9" event={"ID":"55246994-a20a-41be-b350-4bb216ab38dc","Type":"ContainerStarted","Data":"7b68bbb7d6410707b7d3d633e141bfb94c50a5a68306a037d5a40efe96b776cd"} Jul 10 01:08:39.151437 ip-10-0-130-146 kubenswrapper[2570]: I0710 01:08:39.151282 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-aaea96/sub-test-model-aaea96-kserve-648db985d5-tccg9" event={"ID":"55246994-a20a-41be-b350-4bb216ab38dc","Type":"ContainerStarted","Data":"1a60df36616e846a45934002794fb24ac05e9f5f1b2a756595144c68aeeff60f"} Jul 10 01:08:39.171691 ip-10-0-130-146 kubenswrapper[2570]: I0710 01:08:39.171639 2570 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-b-aaea96/sub-test-model-aaea96-kserve-648db985d5-tccg9" podStartSLOduration=1.171624481 podStartE2EDuration="1.171624481s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-10 01:08:38 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 01:08:39.170506152 +0000 UTC m=+2538.218884873" watchObservedRunningTime="2026-07-10 01:08:39.171624481 +0000 UTC m=+2538.220003202" Jul 10 01:08:42.180790 ip-10-0-130-146 kubenswrapper[2570]: I0710 01:08:42.180759 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-g92hs"] Jul 10 01:08:48.899473 ip-10-0-130-146 kubenswrapper[2570]: I0710 01:08:48.899437 2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-b-aaea96/sub-test-model-aaea96-kserve-648db985d5-tccg9" Jul 10 01:08:48.899862 ip-10-0-130-146 kubenswrapper[2570]: I0710 01:08:48.899483 2570 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-b-aaea96/sub-test-model-aaea96-kserve-648db985d5-tccg9" Jul 10 01:08:48.911942 ip-10-0-130-146 kubenswrapper[2570]: I0710 01:08:48.911911 2570 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-b-aaea96/sub-test-model-aaea96-kserve-648db985d5-tccg9" Jul 10 01:08:49.216599 ip-10-0-130-146 kubenswrapper[2570]: I0710 01:08:49.216522 2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-b-aaea96/sub-test-model-aaea96-kserve-648db985d5-tccg9" Jul 10 01:08:58.479554 ip-10-0-130-146 kubenswrapper[2570]: I0710 01:08:58.479524 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-g92hs"] Jul 10 01:09:16.371474 ip-10-0-130-146 kubenswrapper[2570]: I0710 01:09:16.371441 2570 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-aaea96/sub-test-model-aaea96-kserve-648db985d5-tccg9"] Jul 10 01:09:19.886043 ip-10-0-130-146 kubenswrapper[2570]: I0710 01:09:19.886013 2570 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-a-48a750/rate-test-model-48a750-kserve-7dc6fff7c9-9gw6l"] Jul 10 01:09:19.889519 ip-10-0-130-146 kubenswrapper[2570]: I0710 01:09:19.889500 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-48a750/rate-test-model-48a750-kserve-7dc6fff7c9-9gw6l" Jul 10 01:09:19.892146 ip-10-0-130-146 kubenswrapper[2570]: I0710 01:09:19.892124 2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-48a750\"/\"kube-root-ca.crt\"" Jul 10 01:09:19.893389 ip-10-0-130-146 kubenswrapper[2570]: I0710 01:09:19.893372 2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-48a750\"/\"default-dockercfg-sjrxd\"" Jul 10 01:09:19.893524 ip-10-0-130-146 kubenswrapper[2570]: I0710 01:09:19.893398 2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-48a750\"/\"openshift-service-ca.crt\"" Jul 10 01:09:19.893524 ip-10-0-130-146 kubenswrapper[2570]: I0710 01:09:19.893447 2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-48a750\"/\"rate-test-model-48a750-kserve-self-signed-certs\"" Jul 10 01:09:19.901959 ip-10-0-130-146 kubenswrapper[2570]: I0710 01:09:19.900804 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-48a750/rate-test-model-48a750-kserve-7dc6fff7c9-9gw6l"] Jul 10 01:09:19.982841 ip-10-0-130-146 kubenswrapper[2570]: I0710 01:09:19.982807 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ba58ea7f-8427-4a47-8f16-099c76458c5d-home\") pod \"rate-test-model-48a750-kserve-7dc6fff7c9-9gw6l\" (UID: \"ba58ea7f-8427-4a47-8f16-099c76458c5d\") " pod="ai-tenant-e2e-shared-a-48a750/rate-test-model-48a750-kserve-7dc6fff7c9-9gw6l" Jul 10 01:09:19.982841 ip-10-0-130-146 kubenswrapper[2570]: I0710 01:09:19.982843 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ba58ea7f-8427-4a47-8f16-099c76458c5d-tmp-dir\") pod \"rate-test-model-48a750-kserve-7dc6fff7c9-9gw6l\" (UID: \"ba58ea7f-8427-4a47-8f16-099c76458c5d\") " pod="ai-tenant-e2e-shared-a-48a750/rate-test-model-48a750-kserve-7dc6fff7c9-9gw6l" Jul 10 01:09:19.983087 ip-10-0-130-146 kubenswrapper[2570]: I0710 01:09:19.982871 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ba58ea7f-8427-4a47-8f16-099c76458c5d-dshm\") pod \"rate-test-model-48a750-kserve-7dc6fff7c9-9gw6l\" (UID: \"ba58ea7f-8427-4a47-8f16-099c76458c5d\") " pod="ai-tenant-e2e-shared-a-48a750/rate-test-model-48a750-kserve-7dc6fff7c9-9gw6l" Jul 10 01:09:19.983087 ip-10-0-130-146 kubenswrapper[2570]: I0710 01:09:19.982933 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ba58ea7f-8427-4a47-8f16-099c76458c5d-tls-certs\") pod \"rate-test-model-48a750-kserve-7dc6fff7c9-9gw6l\" (UID: \"ba58ea7f-8427-4a47-8f16-099c76458c5d\") " pod="ai-tenant-e2e-shared-a-48a750/rate-test-model-48a750-kserve-7dc6fff7c9-9gw6l" Jul 10 01:09:19.983087 ip-10-0-130-146 kubenswrapper[2570]: I0710 01:09:19.982979 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kvndx\" (UniqueName: \"kubernetes.io/projected/ba58ea7f-8427-4a47-8f16-099c76458c5d-kube-api-access-kvndx\") pod \"rate-test-model-48a750-kserve-7dc6fff7c9-9gw6l\" (UID: \"ba58ea7f-8427-4a47-8f16-099c76458c5d\") " pod="ai-tenant-e2e-shared-a-48a750/rate-test-model-48a750-kserve-7dc6fff7c9-9gw6l" Jul 10 01:09:19.983087 ip-10-0-130-146 kubenswrapper[2570]: I0710 01:09:19.983032 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ba58ea7f-8427-4a47-8f16-099c76458c5d-model-cache\") pod \"rate-test-model-48a750-kserve-7dc6fff7c9-9gw6l\" (UID: \"ba58ea7f-8427-4a47-8f16-099c76458c5d\") " pod="ai-tenant-e2e-shared-a-48a750/rate-test-model-48a750-kserve-7dc6fff7c9-9gw6l" Jul 10 01:09:20.083810 ip-10-0-130-146 kubenswrapper[2570]: I0710 01:09:20.083773 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ba58ea7f-8427-4a47-8f16-099c76458c5d-home\") pod \"rate-test-model-48a750-kserve-7dc6fff7c9-9gw6l\" (UID: \"ba58ea7f-8427-4a47-8f16-099c76458c5d\") " pod="ai-tenant-e2e-shared-a-48a750/rate-test-model-48a750-kserve-7dc6fff7c9-9gw6l" Jul 10 01:09:20.083810 ip-10-0-130-146 kubenswrapper[2570]: I0710 01:09:20.083809 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ba58ea7f-8427-4a47-8f16-099c76458c5d-tmp-dir\") pod \"rate-test-model-48a750-kserve-7dc6fff7c9-9gw6l\" (UID: \"ba58ea7f-8427-4a47-8f16-099c76458c5d\") " pod="ai-tenant-e2e-shared-a-48a750/rate-test-model-48a750-kserve-7dc6fff7c9-9gw6l" Jul 10 01:09:20.084100 ip-10-0-130-146 kubenswrapper[2570]: I0710 01:09:20.083825 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ba58ea7f-8427-4a47-8f16-099c76458c5d-dshm\") pod \"rate-test-model-48a750-kserve-7dc6fff7c9-9gw6l\" (UID: \"ba58ea7f-8427-4a47-8f16-099c76458c5d\") " pod="ai-tenant-e2e-shared-a-48a750/rate-test-model-48a750-kserve-7dc6fff7c9-9gw6l" Jul 10 01:09:20.084100 ip-10-0-130-146 kubenswrapper[2570]: I0710 01:09:20.083913 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ba58ea7f-8427-4a47-8f16-099c76458c5d-tls-certs\") pod \"rate-test-model-48a750-kserve-7dc6fff7c9-9gw6l\" (UID: \"ba58ea7f-8427-4a47-8f16-099c76458c5d\") " pod="ai-tenant-e2e-shared-a-48a750/rate-test-model-48a750-kserve-7dc6fff7c9-9gw6l" Jul 10 01:09:20.084100 ip-10-0-130-146 kubenswrapper[2570]: I0710 01:09:20.083945 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kvndx\" (UniqueName: \"kubernetes.io/projected/ba58ea7f-8427-4a47-8f16-099c76458c5d-kube-api-access-kvndx\") pod \"rate-test-model-48a750-kserve-7dc6fff7c9-9gw6l\" (UID: \"ba58ea7f-8427-4a47-8f16-099c76458c5d\") " pod="ai-tenant-e2e-shared-a-48a750/rate-test-model-48a750-kserve-7dc6fff7c9-9gw6l" Jul 10 01:09:20.084100 ip-10-0-130-146 kubenswrapper[2570]: I0710 01:09:20.083999 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ba58ea7f-8427-4a47-8f16-099c76458c5d-model-cache\") pod \"rate-test-model-48a750-kserve-7dc6fff7c9-9gw6l\" (UID: \"ba58ea7f-8427-4a47-8f16-099c76458c5d\") " pod="ai-tenant-e2e-shared-a-48a750/rate-test-model-48a750-kserve-7dc6fff7c9-9gw6l" Jul 10 01:09:20.084324 ip-10-0-130-146 kubenswrapper[2570]: I0710 01:09:20.084268 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ba58ea7f-8427-4a47-8f16-099c76458c5d-home\") pod \"rate-test-model-48a750-kserve-7dc6fff7c9-9gw6l\" (UID: \"ba58ea7f-8427-4a47-8f16-099c76458c5d\") " pod="ai-tenant-e2e-shared-a-48a750/rate-test-model-48a750-kserve-7dc6fff7c9-9gw6l" Jul 10 01:09:20.084392 ip-10-0-130-146 kubenswrapper[2570]: I0710 01:09:20.084370 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ba58ea7f-8427-4a47-8f16-099c76458c5d-tmp-dir\") pod \"rate-test-model-48a750-kserve-7dc6fff7c9-9gw6l\" (UID: \"ba58ea7f-8427-4a47-8f16-099c76458c5d\") " pod="ai-tenant-e2e-shared-a-48a750/rate-test-model-48a750-kserve-7dc6fff7c9-9gw6l" Jul 10 01:09:20.084434 ip-10-0-130-146 kubenswrapper[2570]: I0710 01:09:20.084381 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ba58ea7f-8427-4a47-8f16-099c76458c5d-model-cache\") pod \"rate-test-model-48a750-kserve-7dc6fff7c9-9gw6l\" (UID: \"ba58ea7f-8427-4a47-8f16-099c76458c5d\") " pod="ai-tenant-e2e-shared-a-48a750/rate-test-model-48a750-kserve-7dc6fff7c9-9gw6l" Jul 10 01:09:20.086158 ip-10-0-130-146 kubenswrapper[2570]: I0710 01:09:20.086136 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ba58ea7f-8427-4a47-8f16-099c76458c5d-dshm\") pod \"rate-test-model-48a750-kserve-7dc6fff7c9-9gw6l\" (UID: \"ba58ea7f-8427-4a47-8f16-099c76458c5d\") " pod="ai-tenant-e2e-shared-a-48a750/rate-test-model-48a750-kserve-7dc6fff7c9-9gw6l" Jul 10 01:09:20.086444 ip-10-0-130-146 kubenswrapper[2570]: I0710 01:09:20.086423 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ba58ea7f-8427-4a47-8f16-099c76458c5d-tls-certs\") pod \"rate-test-model-48a750-kserve-7dc6fff7c9-9gw6l\" (UID: \"ba58ea7f-8427-4a47-8f16-099c76458c5d\") " pod="ai-tenant-e2e-shared-a-48a750/rate-test-model-48a750-kserve-7dc6fff7c9-9gw6l" Jul 10 01:09:20.092852 ip-10-0-130-146 kubenswrapper[2570]: I0710 01:09:20.092831 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-kvndx\" (UniqueName: \"kubernetes.io/projected/ba58ea7f-8427-4a47-8f16-099c76458c5d-kube-api-access-kvndx\") pod \"rate-test-model-48a750-kserve-7dc6fff7c9-9gw6l\" (UID: \"ba58ea7f-8427-4a47-8f16-099c76458c5d\") " pod="ai-tenant-e2e-shared-a-48a750/rate-test-model-48a750-kserve-7dc6fff7c9-9gw6l" Jul 10 01:09:20.205754 ip-10-0-130-146 kubenswrapper[2570]: I0710 01:09:20.205684 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-48a750/rate-test-model-48a750-kserve-7dc6fff7c9-9gw6l" Jul 10 01:09:20.335405 ip-10-0-130-146 kubenswrapper[2570]: I0710 01:09:20.335369 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-48a750/rate-test-model-48a750-kserve-7dc6fff7c9-9gw6l"] Jul 10 01:09:20.338272 ip-10-0-130-146 kubenswrapper[2570]: W0710 01:09:20.338242 2570 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podba58ea7f_8427_4a47_8f16_099c76458c5d.slice/crio-64000f4bf9cdc86bf4e67d82b753e92f9e2c1daf9cec586934e3b4b27e593e0b WatchSource:0}: Error finding container 64000f4bf9cdc86bf4e67d82b753e92f9e2c1daf9cec586934e3b4b27e593e0b: Status 404 returned error can't find the container with id 64000f4bf9cdc86bf4e67d82b753e92f9e2c1daf9cec586934e3b4b27e593e0b Jul 10 01:09:20.369792 ip-10-0-130-146 kubenswrapper[2570]: I0710 01:09:20.369765 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-48a750/rate-test-model-48a750-kserve-7dc6fff7c9-9gw6l" event={"ID":"ba58ea7f-8427-4a47-8f16-099c76458c5d","Type":"ContainerStarted","Data":"64000f4bf9cdc86bf4e67d82b753e92f9e2c1daf9cec586934e3b4b27e593e0b"} Jul 10 01:09:21.185234 ip-10-0-130-146 kubenswrapper[2570]: I0710 01:09:21.185204 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-g92hs"] Jul 10 01:09:21.377121 ip-10-0-130-146 kubenswrapper[2570]: I0710 01:09:21.377083 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-48a750/rate-test-model-48a750-kserve-7dc6fff7c9-9gw6l" event={"ID":"ba58ea7f-8427-4a47-8f16-099c76458c5d","Type":"ContainerStarted","Data":"b1a4ece366723a704d200189234447a63c1c58e33e3995db4c78ba2c8a07818d"} Jul 10 01:09:21.396278 ip-10-0-130-146 kubenswrapper[2570]: I0710 01:09:21.396232 2570 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-a-48a750/rate-test-model-48a750-kserve-7dc6fff7c9-9gw6l" podStartSLOduration=2.396216978 podStartE2EDuration="2.396216978s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-10 01:09:19 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 01:09:21.39312316 +0000 UTC m=+2580.441501880" watchObservedRunningTime="2026-07-10 01:09:21.396216978 +0000 UTC m=+2580.444595753" Jul 10 01:09:30.206155 ip-10-0-130-146 kubenswrapper[2570]: I0710 01:09:30.206108 2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-a-48a750/rate-test-model-48a750-kserve-7dc6fff7c9-9gw6l" Jul 10 01:09:30.206155 ip-10-0-130-146 kubenswrapper[2570]: I0710 01:09:30.206169 2570 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-a-48a750/rate-test-model-48a750-kserve-7dc6fff7c9-9gw6l" Jul 10 01:09:30.218699 ip-10-0-130-146 kubenswrapper[2570]: I0710 01:09:30.218666 2570 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-a-48a750/rate-test-model-48a750-kserve-7dc6fff7c9-9gw6l" Jul 10 01:09:30.438445 ip-10-0-130-146 kubenswrapper[2570]: I0710 01:09:30.438417 2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-a-48a750/rate-test-model-48a750-kserve-7dc6fff7c9-9gw6l" Jul 10 01:09:31.391553 ip-10-0-130-146 kubenswrapper[2570]: I0710 01:09:31.391495 2570 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-shared-b-aaea96/sub-test-model-aaea96-kserve-648db985d5-tccg9" podUID="55246994-a20a-41be-b350-4bb216ab38dc" containerName="main" containerID="cri-o://7b68bbb7d6410707b7d3d633e141bfb94c50a5a68306a037d5a40efe96b776cd" gracePeriod=45 Jul 10 01:09:31.644745 ip-10-0-130-146 kubenswrapper[2570]: I0710 01:09:31.644684 2570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-aaea96/sub-test-model-aaea96-kserve-648db985d5-tccg9" Jul 10 01:09:31.794764 ip-10-0-130-146 kubenswrapper[2570]: I0710 01:09:31.794737 2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/55246994-a20a-41be-b350-4bb216ab38dc-home\") pod \"55246994-a20a-41be-b350-4bb216ab38dc\" (UID: \"55246994-a20a-41be-b350-4bb216ab38dc\") " Jul 10 01:09:31.794764 ip-10-0-130-146 kubenswrapper[2570]: I0710 01:09:31.794769 2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-tk4kp\" (UniqueName: \"kubernetes.io/projected/55246994-a20a-41be-b350-4bb216ab38dc-kube-api-access-tk4kp\") pod \"55246994-a20a-41be-b350-4bb216ab38dc\" (UID: \"55246994-a20a-41be-b350-4bb216ab38dc\") " Jul 10 01:09:31.795035 ip-10-0-130-146 kubenswrapper[2570]: I0710 01:09:31.794814 2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/55246994-a20a-41be-b350-4bb216ab38dc-dshm\") pod \"55246994-a20a-41be-b350-4bb216ab38dc\" (UID: \"55246994-a20a-41be-b350-4bb216ab38dc\") " Jul 10 01:09:31.795035 ip-10-0-130-146 kubenswrapper[2570]: I0710 01:09:31.794844 2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/55246994-a20a-41be-b350-4bb216ab38dc-tls-certs\") pod \"55246994-a20a-41be-b350-4bb216ab38dc\" (UID: \"55246994-a20a-41be-b350-4bb216ab38dc\") " Jul 10 01:09:31.795035 ip-10-0-130-146 kubenswrapper[2570]: I0710 01:09:31.794870 2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/55246994-a20a-41be-b350-4bb216ab38dc-model-cache\") pod \"55246994-a20a-41be-b350-4bb216ab38dc\" (UID: \"55246994-a20a-41be-b350-4bb216ab38dc\") " Jul 10 01:09:31.795035 ip-10-0-130-146 kubenswrapper[2570]: I0710 01:09:31.794920 2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/55246994-a20a-41be-b350-4bb216ab38dc-tmp-dir\") pod \"55246994-a20a-41be-b350-4bb216ab38dc\" (UID: \"55246994-a20a-41be-b350-4bb216ab38dc\") " Jul 10 01:09:31.795035 ip-10-0-130-146 kubenswrapper[2570]: I0710 01:09:31.795012 2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/55246994-a20a-41be-b350-4bb216ab38dc-home" (OuterVolumeSpecName: "home") pod "55246994-a20a-41be-b350-4bb216ab38dc" (UID: "55246994-a20a-41be-b350-4bb216ab38dc"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 10 01:09:31.795351 ip-10-0-130-146 kubenswrapper[2570]: I0710 01:09:31.795228 2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/55246994-a20a-41be-b350-4bb216ab38dc-model-cache" (OuterVolumeSpecName: "model-cache") pod "55246994-a20a-41be-b350-4bb216ab38dc" (UID: "55246994-a20a-41be-b350-4bb216ab38dc"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 10 01:09:31.795351 ip-10-0-130-146 kubenswrapper[2570]: I0710 01:09:31.795242 2570 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/55246994-a20a-41be-b350-4bb216ab38dc-home\") on node \"ip-10-0-130-146.ec2.internal\" DevicePath \"\"" Jul 10 01:09:31.795351 ip-10-0-130-146 kubenswrapper[2570]: I0710 01:09:31.795266 2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/55246994-a20a-41be-b350-4bb216ab38dc-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "55246994-a20a-41be-b350-4bb216ab38dc" (UID: "55246994-a20a-41be-b350-4bb216ab38dc"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 10 01:09:31.796989 ip-10-0-130-146 kubenswrapper[2570]: I0710 01:09:31.796961 2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/55246994-a20a-41be-b350-4bb216ab38dc-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "55246994-a20a-41be-b350-4bb216ab38dc" (UID: "55246994-a20a-41be-b350-4bb216ab38dc"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 10 01:09:31.797141 ip-10-0-130-146 kubenswrapper[2570]: I0710 01:09:31.796990 2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/55246994-a20a-41be-b350-4bb216ab38dc-kube-api-access-tk4kp" (OuterVolumeSpecName: "kube-api-access-tk4kp") pod "55246994-a20a-41be-b350-4bb216ab38dc" (UID: "55246994-a20a-41be-b350-4bb216ab38dc"). InnerVolumeSpecName "kube-api-access-tk4kp". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 10 01:09:31.797141 ip-10-0-130-146 kubenswrapper[2570]: I0710 01:09:31.797074 2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/55246994-a20a-41be-b350-4bb216ab38dc-dshm" (OuterVolumeSpecName: "dshm") pod "55246994-a20a-41be-b350-4bb216ab38dc" (UID: "55246994-a20a-41be-b350-4bb216ab38dc"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 10 01:09:31.896731 ip-10-0-130-146 kubenswrapper[2570]: I0710 01:09:31.896648 2570 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/55246994-a20a-41be-b350-4bb216ab38dc-dshm\") on node \"ip-10-0-130-146.ec2.internal\" DevicePath \"\"" Jul 10 01:09:31.896731 ip-10-0-130-146 kubenswrapper[2570]: I0710 01:09:31.896677 2570 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/55246994-a20a-41be-b350-4bb216ab38dc-tls-certs\") on node \"ip-10-0-130-146.ec2.internal\" DevicePath \"\"" Jul 10 01:09:31.896731 ip-10-0-130-146 kubenswrapper[2570]: I0710 01:09:31.896687 2570 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/55246994-a20a-41be-b350-4bb216ab38dc-model-cache\") on node \"ip-10-0-130-146.ec2.internal\" DevicePath \"\"" Jul 10 01:09:31.896731 ip-10-0-130-146 kubenswrapper[2570]: I0710 01:09:31.896695 2570 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/55246994-a20a-41be-b350-4bb216ab38dc-tmp-dir\") on node \"ip-10-0-130-146.ec2.internal\" DevicePath \"\"" Jul 10 01:09:31.896731 ip-10-0-130-146 kubenswrapper[2570]: I0710 01:09:31.896704 2570 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tk4kp\" (UniqueName: \"kubernetes.io/projected/55246994-a20a-41be-b350-4bb216ab38dc-kube-api-access-tk4kp\") on node \"ip-10-0-130-146.ec2.internal\" DevicePath \"\"" Jul 10 01:09:32.438651 ip-10-0-130-146 kubenswrapper[2570]: I0710 01:09:32.438618 2570 generic.go:356] "Generic (PLEG): container finished" podID="55246994-a20a-41be-b350-4bb216ab38dc" containerID="7b68bbb7d6410707b7d3d633e141bfb94c50a5a68306a037d5a40efe96b776cd" exitCode=0 Jul 10 01:09:32.439126 ip-10-0-130-146 kubenswrapper[2570]: I0710 01:09:32.438660 2570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-aaea96/sub-test-model-aaea96-kserve-648db985d5-tccg9" Jul 10 01:09:32.439126 ip-10-0-130-146 kubenswrapper[2570]: I0710 01:09:32.438711 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-aaea96/sub-test-model-aaea96-kserve-648db985d5-tccg9" event={"ID":"55246994-a20a-41be-b350-4bb216ab38dc","Type":"ContainerDied","Data":"7b68bbb7d6410707b7d3d633e141bfb94c50a5a68306a037d5a40efe96b776cd"} Jul 10 01:09:32.439126 ip-10-0-130-146 kubenswrapper[2570]: I0710 01:09:32.438762 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-aaea96/sub-test-model-aaea96-kserve-648db985d5-tccg9" event={"ID":"55246994-a20a-41be-b350-4bb216ab38dc","Type":"ContainerDied","Data":"1a60df36616e846a45934002794fb24ac05e9f5f1b2a756595144c68aeeff60f"} Jul 10 01:09:32.439126 ip-10-0-130-146 kubenswrapper[2570]: I0710 01:09:32.438791 2570 scope.go:117] "RemoveContainer" containerID="7b68bbb7d6410707b7d3d633e141bfb94c50a5a68306a037d5a40efe96b776cd" Jul 10 01:09:32.448196 ip-10-0-130-146 kubenswrapper[2570]: I0710 01:09:32.448180 2570 scope.go:117] "RemoveContainer" containerID="7b68bbb7d6410707b7d3d633e141bfb94c50a5a68306a037d5a40efe96b776cd" Jul 10 01:09:32.448442 ip-10-0-130-146 kubenswrapper[2570]: E0710 01:09:32.448417 2570 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7b68bbb7d6410707b7d3d633e141bfb94c50a5a68306a037d5a40efe96b776cd\": container with ID starting with 7b68bbb7d6410707b7d3d633e141bfb94c50a5a68306a037d5a40efe96b776cd not found: ID does not exist" containerID="7b68bbb7d6410707b7d3d633e141bfb94c50a5a68306a037d5a40efe96b776cd" Jul 10 01:09:32.448548 ip-10-0-130-146 kubenswrapper[2570]: I0710 01:09:32.448447 2570 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7b68bbb7d6410707b7d3d633e141bfb94c50a5a68306a037d5a40efe96b776cd"} err="failed to get container status \"7b68bbb7d6410707b7d3d633e141bfb94c50a5a68306a037d5a40efe96b776cd\": rpc error: code = NotFound desc = could not find container \"7b68bbb7d6410707b7d3d633e141bfb94c50a5a68306a037d5a40efe96b776cd\": container with ID starting with 7b68bbb7d6410707b7d3d633e141bfb94c50a5a68306a037d5a40efe96b776cd not found: ID does not exist" Jul 10 01:09:32.459801 ip-10-0-130-146 kubenswrapper[2570]: I0710 01:09:32.459780 2570 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-aaea96/sub-test-model-aaea96-kserve-648db985d5-tccg9"] Jul 10 01:09:32.461895 ip-10-0-130-146 kubenswrapper[2570]: I0710 01:09:32.461876 2570 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-b-aaea96/sub-test-model-aaea96-kserve-648db985d5-tccg9"] Jul 10 01:09:33.453805 ip-10-0-130-146 kubenswrapper[2570]: I0710 01:09:33.453772 2570 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="55246994-a20a-41be-b350-4bb216ab38dc" path="/var/lib/kubelet/pods/55246994-a20a-41be-b350-4bb216ab38dc/volumes" Jul 10 01:09:38.086549 ip-10-0-130-146 kubenswrapper[2570]: I0710 01:09:38.086514 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-g92hs"] Jul 10 01:09:42.982657 ip-10-0-130-146 kubenswrapper[2570]: I0710 01:09:42.982582 2570 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-b-aaea96/rate-test-model-aaea96-kserve-59f4c77cdc-kb6c8"] Jul 10 01:09:42.983165 ip-10-0-130-146 kubenswrapper[2570]: I0710 01:09:42.983148 2570 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="55246994-a20a-41be-b350-4bb216ab38dc" containerName="main" Jul 10 01:09:42.983209 ip-10-0-130-146 kubenswrapper[2570]: I0710 01:09:42.983169 2570 state_mem.go:107] "Deleted CPUSet assignment" podUID="55246994-a20a-41be-b350-4bb216ab38dc" containerName="main" Jul 10 01:09:42.983290 ip-10-0-130-146 kubenswrapper[2570]: I0710 01:09:42.983280 2570 memory_manager.go:368] "RemoveStaleState removing state" podUID="55246994-a20a-41be-b350-4bb216ab38dc" containerName="main" Jul 10 01:09:42.987882 ip-10-0-130-146 kubenswrapper[2570]: I0710 01:09:42.987863 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-aaea96/rate-test-model-aaea96-kserve-59f4c77cdc-kb6c8" Jul 10 01:09:42.990754 ip-10-0-130-146 kubenswrapper[2570]: I0710 01:09:42.990730 2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-aaea96\"/\"openshift-service-ca.crt\"" Jul 10 01:09:42.991722 ip-10-0-130-146 kubenswrapper[2570]: I0710 01:09:42.991699 2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-aaea96\"/\"kube-root-ca.crt\"" Jul 10 01:09:42.991882 ip-10-0-130-146 kubenswrapper[2570]: I0710 01:09:42.991850 2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-aaea96\"/\"default-dockercfg-dz7xs\"" Jul 10 01:09:42.991954 ip-10-0-130-146 kubenswrapper[2570]: I0710 01:09:42.991927 2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-aaea96\"/\"rate-test-model-aaea96-kserve-self-signed-certs\"" Jul 10 01:09:42.992345 ip-10-0-130-146 kubenswrapper[2570]: I0710 01:09:42.992325 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-aaea96/rate-test-model-aaea96-kserve-59f4c77cdc-kb6c8"] Jul 10 01:09:43.087914 ip-10-0-130-146 kubenswrapper[2570]: I0710 01:09:43.087887 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2f77297e-4102-4012-89da-2d47ca86526b-dshm\") pod \"rate-test-model-aaea96-kserve-59f4c77cdc-kb6c8\" (UID: \"2f77297e-4102-4012-89da-2d47ca86526b\") " pod="ai-tenant-e2e-shared-b-aaea96/rate-test-model-aaea96-kserve-59f4c77cdc-kb6c8" Jul 10 01:09:43.088134 ip-10-0-130-146 kubenswrapper[2570]: I0710 01:09:43.087919 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2f77297e-4102-4012-89da-2d47ca86526b-tls-certs\") pod \"rate-test-model-aaea96-kserve-59f4c77cdc-kb6c8\" (UID: \"2f77297e-4102-4012-89da-2d47ca86526b\") " pod="ai-tenant-e2e-shared-b-aaea96/rate-test-model-aaea96-kserve-59f4c77cdc-kb6c8" Jul 10 01:09:43.088134 ip-10-0-130-146 kubenswrapper[2570]: I0710 01:09:43.087942 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2f77297e-4102-4012-89da-2d47ca86526b-model-cache\") pod \"rate-test-model-aaea96-kserve-59f4c77cdc-kb6c8\" (UID: \"2f77297e-4102-4012-89da-2d47ca86526b\") " pod="ai-tenant-e2e-shared-b-aaea96/rate-test-model-aaea96-kserve-59f4c77cdc-kb6c8" Jul 10 01:09:43.088134 ip-10-0-130-146 kubenswrapper[2570]: I0710 01:09:43.087966 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2f77297e-4102-4012-89da-2d47ca86526b-tmp-dir\") pod \"rate-test-model-aaea96-kserve-59f4c77cdc-kb6c8\" (UID: \"2f77297e-4102-4012-89da-2d47ca86526b\") " pod="ai-tenant-e2e-shared-b-aaea96/rate-test-model-aaea96-kserve-59f4c77cdc-kb6c8" Jul 10 01:09:43.088134 ip-10-0-130-146 kubenswrapper[2570]: I0710 01:09:43.088092 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2f77297e-4102-4012-89da-2d47ca86526b-home\") pod \"rate-test-model-aaea96-kserve-59f4c77cdc-kb6c8\" (UID: \"2f77297e-4102-4012-89da-2d47ca86526b\") " pod="ai-tenant-e2e-shared-b-aaea96/rate-test-model-aaea96-kserve-59f4c77cdc-kb6c8" Jul 10 01:09:43.088336 ip-10-0-130-146 kubenswrapper[2570]: I0710 01:09:43.088203 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6ktmb\" (UniqueName: \"kubernetes.io/projected/2f77297e-4102-4012-89da-2d47ca86526b-kube-api-access-6ktmb\") pod \"rate-test-model-aaea96-kserve-59f4c77cdc-kb6c8\" (UID: \"2f77297e-4102-4012-89da-2d47ca86526b\") " pod="ai-tenant-e2e-shared-b-aaea96/rate-test-model-aaea96-kserve-59f4c77cdc-kb6c8" Jul 10 01:09:43.188828 ip-10-0-130-146 kubenswrapper[2570]: I0710 01:09:43.188796 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2f77297e-4102-4012-89da-2d47ca86526b-tmp-dir\") pod \"rate-test-model-aaea96-kserve-59f4c77cdc-kb6c8\" (UID: \"2f77297e-4102-4012-89da-2d47ca86526b\") " pod="ai-tenant-e2e-shared-b-aaea96/rate-test-model-aaea96-kserve-59f4c77cdc-kb6c8" Jul 10 01:09:43.188946 ip-10-0-130-146 kubenswrapper[2570]: I0710 01:09:43.188840 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2f77297e-4102-4012-89da-2d47ca86526b-home\") pod \"rate-test-model-aaea96-kserve-59f4c77cdc-kb6c8\" (UID: \"2f77297e-4102-4012-89da-2d47ca86526b\") " pod="ai-tenant-e2e-shared-b-aaea96/rate-test-model-aaea96-kserve-59f4c77cdc-kb6c8" Jul 10 01:09:43.188946 ip-10-0-130-146 kubenswrapper[2570]: I0710 01:09:43.188894 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6ktmb\" (UniqueName: \"kubernetes.io/projected/2f77297e-4102-4012-89da-2d47ca86526b-kube-api-access-6ktmb\") pod \"rate-test-model-aaea96-kserve-59f4c77cdc-kb6c8\" (UID: \"2f77297e-4102-4012-89da-2d47ca86526b\") " pod="ai-tenant-e2e-shared-b-aaea96/rate-test-model-aaea96-kserve-59f4c77cdc-kb6c8" Jul 10 01:09:43.188946 ip-10-0-130-146 kubenswrapper[2570]: I0710 01:09:43.188921 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2f77297e-4102-4012-89da-2d47ca86526b-dshm\") pod \"rate-test-model-aaea96-kserve-59f4c77cdc-kb6c8\" (UID: \"2f77297e-4102-4012-89da-2d47ca86526b\") " pod="ai-tenant-e2e-shared-b-aaea96/rate-test-model-aaea96-kserve-59f4c77cdc-kb6c8" Jul 10 01:09:43.188946 ip-10-0-130-146 kubenswrapper[2570]: I0710 01:09:43.188938 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2f77297e-4102-4012-89da-2d47ca86526b-tls-certs\") pod \"rate-test-model-aaea96-kserve-59f4c77cdc-kb6c8\" (UID: \"2f77297e-4102-4012-89da-2d47ca86526b\") " pod="ai-tenant-e2e-shared-b-aaea96/rate-test-model-aaea96-kserve-59f4c77cdc-kb6c8" Jul 10 01:09:43.189192 ip-10-0-130-146 kubenswrapper[2570]: I0710 01:09:43.188955 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2f77297e-4102-4012-89da-2d47ca86526b-model-cache\") pod \"rate-test-model-aaea96-kserve-59f4c77cdc-kb6c8\" (UID: \"2f77297e-4102-4012-89da-2d47ca86526b\") " pod="ai-tenant-e2e-shared-b-aaea96/rate-test-model-aaea96-kserve-59f4c77cdc-kb6c8" Jul 10 01:09:43.189274 ip-10-0-130-146 kubenswrapper[2570]: I0710 01:09:43.189251 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2f77297e-4102-4012-89da-2d47ca86526b-tmp-dir\") pod \"rate-test-model-aaea96-kserve-59f4c77cdc-kb6c8\" (UID: \"2f77297e-4102-4012-89da-2d47ca86526b\") " pod="ai-tenant-e2e-shared-b-aaea96/rate-test-model-aaea96-kserve-59f4c77cdc-kb6c8" Jul 10 01:09:43.189334 ip-10-0-130-146 kubenswrapper[2570]: I0710 01:09:43.189291 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2f77297e-4102-4012-89da-2d47ca86526b-home\") pod \"rate-test-model-aaea96-kserve-59f4c77cdc-kb6c8\" (UID: \"2f77297e-4102-4012-89da-2d47ca86526b\") " pod="ai-tenant-e2e-shared-b-aaea96/rate-test-model-aaea96-kserve-59f4c77cdc-kb6c8" Jul 10 01:09:43.189334 ip-10-0-130-146 kubenswrapper[2570]: I0710 01:09:43.189306 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2f77297e-4102-4012-89da-2d47ca86526b-model-cache\") pod \"rate-test-model-aaea96-kserve-59f4c77cdc-kb6c8\" (UID: \"2f77297e-4102-4012-89da-2d47ca86526b\") " pod="ai-tenant-e2e-shared-b-aaea96/rate-test-model-aaea96-kserve-59f4c77cdc-kb6c8" Jul 10 01:09:43.191212 ip-10-0-130-146 kubenswrapper[2570]: I0710 01:09:43.191190 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2f77297e-4102-4012-89da-2d47ca86526b-dshm\") pod \"rate-test-model-aaea96-kserve-59f4c77cdc-kb6c8\" (UID: \"2f77297e-4102-4012-89da-2d47ca86526b\") " pod="ai-tenant-e2e-shared-b-aaea96/rate-test-model-aaea96-kserve-59f4c77cdc-kb6c8" Jul 10 01:09:43.191468 ip-10-0-130-146 kubenswrapper[2570]: I0710 01:09:43.191450 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2f77297e-4102-4012-89da-2d47ca86526b-tls-certs\") pod \"rate-test-model-aaea96-kserve-59f4c77cdc-kb6c8\" (UID: \"2f77297e-4102-4012-89da-2d47ca86526b\") " pod="ai-tenant-e2e-shared-b-aaea96/rate-test-model-aaea96-kserve-59f4c77cdc-kb6c8" Jul 10 01:09:43.195967 ip-10-0-130-146 kubenswrapper[2570]: I0710 01:09:43.195944 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6ktmb\" (UniqueName: \"kubernetes.io/projected/2f77297e-4102-4012-89da-2d47ca86526b-kube-api-access-6ktmb\") pod \"rate-test-model-aaea96-kserve-59f4c77cdc-kb6c8\" (UID: \"2f77297e-4102-4012-89da-2d47ca86526b\") " pod="ai-tenant-e2e-shared-b-aaea96/rate-test-model-aaea96-kserve-59f4c77cdc-kb6c8" Jul 10 01:09:43.297972 ip-10-0-130-146 kubenswrapper[2570]: I0710 01:09:43.297893 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-aaea96/rate-test-model-aaea96-kserve-59f4c77cdc-kb6c8" Jul 10 01:09:43.426131 ip-10-0-130-146 kubenswrapper[2570]: I0710 01:09:43.426108 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-aaea96/rate-test-model-aaea96-kserve-59f4c77cdc-kb6c8"] Jul 10 01:09:43.428211 ip-10-0-130-146 kubenswrapper[2570]: W0710 01:09:43.428183 2570 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod2f77297e_4102_4012_89da_2d47ca86526b.slice/crio-d156477538fe999bfb24951a97ac54ed1bcee3ff276b38ae84940f51c322d39e WatchSource:0}: Error finding container d156477538fe999bfb24951a97ac54ed1bcee3ff276b38ae84940f51c322d39e: Status 404 returned error can't find the container with id d156477538fe999bfb24951a97ac54ed1bcee3ff276b38ae84940f51c322d39e Jul 10 01:09:43.509120 ip-10-0-130-146 kubenswrapper[2570]: I0710 01:09:43.509087 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-aaea96/rate-test-model-aaea96-kserve-59f4c77cdc-kb6c8" event={"ID":"2f77297e-4102-4012-89da-2d47ca86526b","Type":"ContainerStarted","Data":"46ad053d0d507df9019e8896bdc68df2e7737e858e0d89be41f2399f19c87cfb"} Jul 10 01:09:43.509234 ip-10-0-130-146 kubenswrapper[2570]: I0710 01:09:43.509138 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-aaea96/rate-test-model-aaea96-kserve-59f4c77cdc-kb6c8" event={"ID":"2f77297e-4102-4012-89da-2d47ca86526b","Type":"ContainerStarted","Data":"d156477538fe999bfb24951a97ac54ed1bcee3ff276b38ae84940f51c322d39e"} Jul 10 01:09:43.525597 ip-10-0-130-146 kubenswrapper[2570]: I0710 01:09:43.525549 2570 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-b-aaea96/rate-test-model-aaea96-kserve-59f4c77cdc-kb6c8" podStartSLOduration=1.525534635 podStartE2EDuration="1.525534635s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-10 01:09:42 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 01:09:43.524002181 +0000 UTC m=+2602.572380928" watchObservedRunningTime="2026-07-10 01:09:43.525534635 +0000 UTC m=+2602.573913412" Jul 10 01:09:48.587362 ip-10-0-130-146 kubenswrapper[2570]: I0710 01:09:48.587331 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-g92hs"] Jul 10 01:09:53.298426 ip-10-0-130-146 kubenswrapper[2570]: I0710 01:09:53.298392 2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-b-aaea96/rate-test-model-aaea96-kserve-59f4c77cdc-kb6c8" Jul 10 01:09:53.298885 ip-10-0-130-146 kubenswrapper[2570]: I0710 01:09:53.298441 2570 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-b-aaea96/rate-test-model-aaea96-kserve-59f4c77cdc-kb6c8" Jul 10 01:09:53.310989 ip-10-0-130-146 kubenswrapper[2570]: I0710 01:09:53.310959 2570 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-b-aaea96/rate-test-model-aaea96-kserve-59f4c77cdc-kb6c8" Jul 10 01:09:53.569676 ip-10-0-130-146 kubenswrapper[2570]: I0710 01:09:53.569595 2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-b-aaea96/rate-test-model-aaea96-kserve-59f4c77cdc-kb6c8" Jul 10 01:10:03.590183 ip-10-0-130-146 kubenswrapper[2570]: I0710 01:10:03.590138 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-g92hs"] Jul 10 01:10:25.686412 ip-10-0-130-146 kubenswrapper[2570]: I0710 01:10:25.686371 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-g92hs"] Jul 10 01:10:29.882737 ip-10-0-130-146 kubenswrapper[2570]: I0710 01:10:29.882706 2570 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-48a750/rate-test-model-48a750-kserve-7dc6fff7c9-9gw6l"] Jul 10 01:10:30.864615 ip-10-0-130-146 kubenswrapper[2570]: I0710 01:10:30.864579 2570 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-aaea96/rate-test-model-aaea96-kserve-59f4c77cdc-kb6c8"] Jul 10 01:10:34.581858 ip-10-0-130-146 kubenswrapper[2570]: I0710 01:10:34.581815 2570 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/test-curl-7517-139919325498672"] Jul 10 01:10:34.585178 ip-10-0-130-146 kubenswrapper[2570]: I0710 01:10:34.585162 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/test-curl-7517-139919325498672" Jul 10 01:10:34.587480 ip-10-0-130-146 kubenswrapper[2570]: I0710 01:10:34.587460 2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"default-dockercfg-5ph6s\"" Jul 10 01:10:34.592185 ip-10-0-130-146 kubenswrapper[2570]: I0710 01:10:34.592162 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/test-curl-7517-139919325498672"] Jul 10 01:10:34.642428 ip-10-0-130-146 kubenswrapper[2570]: I0710 01:10:34.642399 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sdrpr\" (UniqueName: \"kubernetes.io/projected/ecab64c8-40e9-4b98-811b-fe8c15d2641c-kube-api-access-sdrpr\") pod \"test-curl-7517-139919325498672\" (UID: \"ecab64c8-40e9-4b98-811b-fe8c15d2641c\") " pod="openshift-ingress/test-curl-7517-139919325498672" Jul 10 01:10:34.743341 ip-10-0-130-146 kubenswrapper[2570]: I0710 01:10:34.743313 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sdrpr\" (UniqueName: \"kubernetes.io/projected/ecab64c8-40e9-4b98-811b-fe8c15d2641c-kube-api-access-sdrpr\") pod \"test-curl-7517-139919325498672\" (UID: \"ecab64c8-40e9-4b98-811b-fe8c15d2641c\") " pod="openshift-ingress/test-curl-7517-139919325498672" Jul 10 01:10:34.751344 ip-10-0-130-146 kubenswrapper[2570]: I0710 01:10:34.751321 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-sdrpr\" (UniqueName: \"kubernetes.io/projected/ecab64c8-40e9-4b98-811b-fe8c15d2641c-kube-api-access-sdrpr\") pod \"test-curl-7517-139919325498672\" (UID: \"ecab64c8-40e9-4b98-811b-fe8c15d2641c\") " pod="openshift-ingress/test-curl-7517-139919325498672" Jul 10 01:10:34.895819 ip-10-0-130-146 kubenswrapper[2570]: I0710 01:10:34.895798 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/test-curl-7517-139919325498672" Jul 10 01:10:35.026291 ip-10-0-130-146 kubenswrapper[2570]: I0710 01:10:35.026267 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/test-curl-7517-139919325498672"] Jul 10 01:10:35.027883 ip-10-0-130-146 kubenswrapper[2570]: W0710 01:10:35.027855 2570 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podecab64c8_40e9_4b98_811b_fe8c15d2641c.slice/crio-b18379a06ab83e2ca2b5739956a443cb4645d4512ac904969ebcfc2d4690dcc1 WatchSource:0}: Error finding container b18379a06ab83e2ca2b5739956a443cb4645d4512ac904969ebcfc2d4690dcc1: Status 404 returned error can't find the container with id b18379a06ab83e2ca2b5739956a443cb4645d4512ac904969ebcfc2d4690dcc1 Jul 10 01:10:35.029205 ip-10-0-130-146 kubenswrapper[2570]: I0710 01:10:35.029185 2570 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Jul 10 01:10:35.790486 ip-10-0-130-146 kubenswrapper[2570]: I0710 01:10:35.790451 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/test-curl-7517-139919325498672" event={"ID":"ecab64c8-40e9-4b98-811b-fe8c15d2641c","Type":"ContainerStarted","Data":"b18379a06ab83e2ca2b5739956a443cb4645d4512ac904969ebcfc2d4690dcc1"} Jul 10 01:10:36.796716 ip-10-0-130-146 kubenswrapper[2570]: I0710 01:10:36.796683 2570 generic.go:356] "Generic (PLEG): container finished" podID="ecab64c8-40e9-4b98-811b-fe8c15d2641c" containerID="955d55ec215df7a25067755e30dfbc655413d2737054564f0170299b41a23017" exitCode=0 Jul 10 01:10:36.797150 ip-10-0-130-146 kubenswrapper[2570]: I0710 01:10:36.796735 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/test-curl-7517-139919325498672" event={"ID":"ecab64c8-40e9-4b98-811b-fe8c15d2641c","Type":"ContainerDied","Data":"955d55ec215df7a25067755e30dfbc655413d2737054564f0170299b41a23017"} Jul 10 01:10:37.938981 ip-10-0-130-146 kubenswrapper[2570]: I0710 01:10:37.938956 2570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/test-curl-7517-139919325498672" Jul 10 01:10:37.973312 ip-10-0-130-146 kubenswrapper[2570]: I0710 01:10:37.973282 2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-sdrpr\" (UniqueName: \"kubernetes.io/projected/ecab64c8-40e9-4b98-811b-fe8c15d2641c-kube-api-access-sdrpr\") pod \"ecab64c8-40e9-4b98-811b-fe8c15d2641c\" (UID: \"ecab64c8-40e9-4b98-811b-fe8c15d2641c\") " Jul 10 01:10:37.973818 ip-10-0-130-146 kubenswrapper[2570]: I0710 01:10:37.973794 2570 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_test-curl-7517-139919325498672_ecab64c8-40e9-4b98-811b-fe8c15d2641c/test-curl-7517-139919325498672/0.log" Jul 10 01:10:37.975509 ip-10-0-130-146 kubenswrapper[2570]: I0710 01:10:37.975484 2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ecab64c8-40e9-4b98-811b-fe8c15d2641c-kube-api-access-sdrpr" (OuterVolumeSpecName: "kube-api-access-sdrpr") pod "ecab64c8-40e9-4b98-811b-fe8c15d2641c" (UID: "ecab64c8-40e9-4b98-811b-fe8c15d2641c"). InnerVolumeSpecName "kube-api-access-sdrpr". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 10 01:10:37.999798 ip-10-0-130-146 kubenswrapper[2570]: I0710 01:10:37.999770 2570 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/test-curl-7517-139919325498672"] Jul 10 01:10:38.003748 ip-10-0-130-146 kubenswrapper[2570]: I0710 01:10:38.003717 2570 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/test-curl-7517-139919325498672"] Jul 10 01:10:38.074480 ip-10-0-130-146 kubenswrapper[2570]: I0710 01:10:38.074417 2570 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sdrpr\" (UniqueName: \"kubernetes.io/projected/ecab64c8-40e9-4b98-811b-fe8c15d2641c-kube-api-access-sdrpr\") on node \"ip-10-0-130-146.ec2.internal\" DevicePath \"\"" Jul 10 01:10:38.170621 ip-10-0-130-146 kubenswrapper[2570]: I0710 01:10:38.170581 2570 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/test-curl-7517-139919324139216"] Jul 10 01:10:38.171041 ip-10-0-130-146 kubenswrapper[2570]: I0710 01:10:38.171023 2570 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ecab64c8-40e9-4b98-811b-fe8c15d2641c" containerName="test-curl-7517-139919325498672" Jul 10 01:10:38.171041 ip-10-0-130-146 kubenswrapper[2570]: I0710 01:10:38.171044 2570 state_mem.go:107] "Deleted CPUSet assignment" podUID="ecab64c8-40e9-4b98-811b-fe8c15d2641c" containerName="test-curl-7517-139919325498672" Jul 10 01:10:38.171206 ip-10-0-130-146 kubenswrapper[2570]: I0710 01:10:38.171191 2570 memory_manager.go:368] "RemoveStaleState removing state" podUID="ecab64c8-40e9-4b98-811b-fe8c15d2641c" containerName="test-curl-7517-139919325498672" Jul 10 01:10:38.174280 ip-10-0-130-146 kubenswrapper[2570]: I0710 01:10:38.174261 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/test-curl-7517-139919324139216" Jul 10 01:10:38.179604 ip-10-0-130-146 kubenswrapper[2570]: I0710 01:10:38.179576 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/test-curl-7517-139919324139216"] Jul 10 01:10:38.275381 ip-10-0-130-146 kubenswrapper[2570]: I0710 01:10:38.275350 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-d6dqh\" (UniqueName: \"kubernetes.io/projected/07334daf-efa9-41e2-b0ac-6f07a89492b4-kube-api-access-d6dqh\") pod \"test-curl-7517-139919324139216\" (UID: \"07334daf-efa9-41e2-b0ac-6f07a89492b4\") " pod="openshift-ingress/test-curl-7517-139919324139216" Jul 10 01:10:38.376667 ip-10-0-130-146 kubenswrapper[2570]: I0710 01:10:38.376641 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d6dqh\" (UniqueName: \"kubernetes.io/projected/07334daf-efa9-41e2-b0ac-6f07a89492b4-kube-api-access-d6dqh\") pod \"test-curl-7517-139919324139216\" (UID: \"07334daf-efa9-41e2-b0ac-6f07a89492b4\") " pod="openshift-ingress/test-curl-7517-139919324139216" Jul 10 01:10:38.384227 ip-10-0-130-146 kubenswrapper[2570]: I0710 01:10:38.384199 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-d6dqh\" (UniqueName: \"kubernetes.io/projected/07334daf-efa9-41e2-b0ac-6f07a89492b4-kube-api-access-d6dqh\") pod \"test-curl-7517-139919324139216\" (UID: \"07334daf-efa9-41e2-b0ac-6f07a89492b4\") " pod="openshift-ingress/test-curl-7517-139919324139216" Jul 10 01:10:38.483418 ip-10-0-130-146 kubenswrapper[2570]: I0710 01:10:38.483397 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/test-curl-7517-139919324139216" Jul 10 01:10:38.611689 ip-10-0-130-146 kubenswrapper[2570]: I0710 01:10:38.611658 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/test-curl-7517-139919324139216"] Jul 10 01:10:38.612594 ip-10-0-130-146 kubenswrapper[2570]: W0710 01:10:38.612559 2570 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod07334daf_efa9_41e2_b0ac_6f07a89492b4.slice/crio-2c07644a5f5c1ad18654e8cd939c5d6f13f85e82f4fcf1ac922ff22ae4ac8f3e WatchSource:0}: Error finding container 2c07644a5f5c1ad18654e8cd939c5d6f13f85e82f4fcf1ac922ff22ae4ac8f3e: Status 404 returned error can't find the container with id 2c07644a5f5c1ad18654e8cd939c5d6f13f85e82f4fcf1ac922ff22ae4ac8f3e Jul 10 01:10:38.808408 ip-10-0-130-146 kubenswrapper[2570]: I0710 01:10:38.808382 2570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/test-curl-7517-139919325498672" Jul 10 01:10:38.808576 ip-10-0-130-146 kubenswrapper[2570]: I0710 01:10:38.808426 2570 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="b18379a06ab83e2ca2b5739956a443cb4645d4512ac904969ebcfc2d4690dcc1" Jul 10 01:10:38.809624 ip-10-0-130-146 kubenswrapper[2570]: I0710 01:10:38.809597 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/test-curl-7517-139919324139216" event={"ID":"07334daf-efa9-41e2-b0ac-6f07a89492b4","Type":"ContainerStarted","Data":"2c07644a5f5c1ad18654e8cd939c5d6f13f85e82f4fcf1ac922ff22ae4ac8f3e"} Jul 10 01:10:38.821817 ip-10-0-130-146 kubenswrapper[2570]: E0710 01:10:38.821784 2570 status_manager.go:1018] "Failed to get status for pod" err="pods \"test-curl-7517-139919325498672\" is forbidden: User \"system:node:ip-10-0-130-146.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"openshift-ingress\": no relationship found between node 'ip-10-0-130-146.ec2.internal' and this object" podUID="ecab64c8-40e9-4b98-811b-fe8c15d2641c" pod="openshift-ingress/test-curl-7517-139919325498672" Jul 10 01:10:39.455247 ip-10-0-130-146 kubenswrapper[2570]: I0710 01:10:39.455213 2570 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="ecab64c8-40e9-4b98-811b-fe8c15d2641c" path="/var/lib/kubelet/pods/ecab64c8-40e9-4b98-811b-fe8c15d2641c/volumes" Jul 10 01:10:39.817447 ip-10-0-130-146 kubenswrapper[2570]: I0710 01:10:39.817356 2570 generic.go:356] "Generic (PLEG): container finished" podID="07334daf-efa9-41e2-b0ac-6f07a89492b4" containerID="ded0795504578580879d1fa169b33007169dd42b7c4cc2dc59b8b9613e077390" exitCode=0 Jul 10 01:10:39.817447 ip-10-0-130-146 kubenswrapper[2570]: I0710 01:10:39.817407 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/test-curl-7517-139919324139216" event={"ID":"07334daf-efa9-41e2-b0ac-6f07a89492b4","Type":"ContainerDied","Data":"ded0795504578580879d1fa169b33007169dd42b7c4cc2dc59b8b9613e077390"} Jul 10 01:10:40.959981 ip-10-0-130-146 kubenswrapper[2570]: I0710 01:10:40.959955 2570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/test-curl-7517-139919324139216" Jul 10 01:10:40.980368 ip-10-0-130-146 kubenswrapper[2570]: I0710 01:10:40.980346 2570 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_test-curl-7517-139919324139216_07334daf-efa9-41e2-b0ac-6f07a89492b4/test-curl-7517-139919324139216/0.log" Jul 10 01:10:40.999549 ip-10-0-130-146 kubenswrapper[2570]: I0710 01:10:40.999522 2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-d6dqh\" (UniqueName: \"kubernetes.io/projected/07334daf-efa9-41e2-b0ac-6f07a89492b4-kube-api-access-d6dqh\") pod \"07334daf-efa9-41e2-b0ac-6f07a89492b4\" (UID: \"07334daf-efa9-41e2-b0ac-6f07a89492b4\") " Jul 10 01:10:41.001571 ip-10-0-130-146 kubenswrapper[2570]: I0710 01:10:41.001542 2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/07334daf-efa9-41e2-b0ac-6f07a89492b4-kube-api-access-d6dqh" (OuterVolumeSpecName: "kube-api-access-d6dqh") pod "07334daf-efa9-41e2-b0ac-6f07a89492b4" (UID: "07334daf-efa9-41e2-b0ac-6f07a89492b4"). InnerVolumeSpecName "kube-api-access-d6dqh". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 10 01:10:41.004955 ip-10-0-130-146 kubenswrapper[2570]: I0710 01:10:41.004934 2570 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/test-curl-7517-139919324139216"] Jul 10 01:10:41.010908 ip-10-0-130-146 kubenswrapper[2570]: I0710 01:10:41.010887 2570 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/test-curl-7517-139919324139216"] Jul 10 01:10:41.100739 ip-10-0-130-146 kubenswrapper[2570]: I0710 01:10:41.100673 2570 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-d6dqh\" (UniqueName: \"kubernetes.io/projected/07334daf-efa9-41e2-b0ac-6f07a89492b4-kube-api-access-d6dqh\") on node \"ip-10-0-130-146.ec2.internal\" DevicePath \"\"" Jul 10 01:10:41.460352 ip-10-0-130-146 kubenswrapper[2570]: I0710 01:10:41.460315 2570 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="07334daf-efa9-41e2-b0ac-6f07a89492b4" path="/var/lib/kubelet/pods/07334daf-efa9-41e2-b0ac-6f07a89492b4/volumes" Jul 10 01:10:41.830260 ip-10-0-130-146 kubenswrapper[2570]: I0710 01:10:41.830177 2570 scope.go:117] "RemoveContainer" containerID="ded0795504578580879d1fa169b33007169dd42b7c4cc2dc59b8b9613e077390" Jul 10 01:10:41.830260 ip-10-0-130-146 kubenswrapper[2570]: I0710 01:10:41.830205 2570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/test-curl-7517-139919324139216" Jul 10 01:10:42.891603 ip-10-0-130-146 kubenswrapper[2570]: I0710 01:10:42.891572 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-g92hs"] Jul 10 01:10:43.646763 ip-10-0-130-146 kubenswrapper[2570]: I0710 01:10:43.646729 2570 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/test-curl-7517-139919326745344"] Jul 10 01:10:43.647154 ip-10-0-130-146 kubenswrapper[2570]: I0710 01:10:43.647139 2570 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="07334daf-efa9-41e2-b0ac-6f07a89492b4" containerName="test-curl-7517-139919324139216" Jul 10 01:10:43.647212 ip-10-0-130-146 kubenswrapper[2570]: I0710 01:10:43.647159 2570 state_mem.go:107] "Deleted CPUSet assignment" podUID="07334daf-efa9-41e2-b0ac-6f07a89492b4" containerName="test-curl-7517-139919324139216" Jul 10 01:10:43.647250 ip-10-0-130-146 kubenswrapper[2570]: I0710 01:10:43.647229 2570 memory_manager.go:368] "RemoveStaleState removing state" podUID="07334daf-efa9-41e2-b0ac-6f07a89492b4" containerName="test-curl-7517-139919324139216" Jul 10 01:10:43.651667 ip-10-0-130-146 kubenswrapper[2570]: I0710 01:10:43.651649 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/test-curl-7517-139919326745344" Jul 10 01:10:43.654182 ip-10-0-130-146 kubenswrapper[2570]: I0710 01:10:43.654154 2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"default-dockercfg-5ph6s\"" Jul 10 01:10:43.655232 ip-10-0-130-146 kubenswrapper[2570]: I0710 01:10:43.655212 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/test-curl-7517-139919326745344"] Jul 10 01:10:43.723358 ip-10-0-130-146 kubenswrapper[2570]: I0710 01:10:43.723331 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xk6mj\" (UniqueName: \"kubernetes.io/projected/9c35bd75-f4b0-4544-8fc9-07ae7d83fed7-kube-api-access-xk6mj\") pod \"test-curl-7517-139919326745344\" (UID: \"9c35bd75-f4b0-4544-8fc9-07ae7d83fed7\") " pod="openshift-ingress/test-curl-7517-139919326745344" Jul 10 01:10:43.824115 ip-10-0-130-146 kubenswrapper[2570]: I0710 01:10:43.824085 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xk6mj\" (UniqueName: \"kubernetes.io/projected/9c35bd75-f4b0-4544-8fc9-07ae7d83fed7-kube-api-access-xk6mj\") pod \"test-curl-7517-139919326745344\" (UID: \"9c35bd75-f4b0-4544-8fc9-07ae7d83fed7\") " pod="openshift-ingress/test-curl-7517-139919326745344" Jul 10 01:10:43.832452 ip-10-0-130-146 kubenswrapper[2570]: I0710 01:10:43.832420 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xk6mj\" (UniqueName: \"kubernetes.io/projected/9c35bd75-f4b0-4544-8fc9-07ae7d83fed7-kube-api-access-xk6mj\") pod \"test-curl-7517-139919326745344\" (UID: \"9c35bd75-f4b0-4544-8fc9-07ae7d83fed7\") " pod="openshift-ingress/test-curl-7517-139919326745344" Jul 10 01:10:43.961343 ip-10-0-130-146 kubenswrapper[2570]: I0710 01:10:43.961268 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/test-curl-7517-139919326745344" Jul 10 01:10:44.095286 ip-10-0-130-146 kubenswrapper[2570]: I0710 01:10:44.095256 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/test-curl-7517-139919326745344"] Jul 10 01:10:44.096628 ip-10-0-130-146 kubenswrapper[2570]: W0710 01:10:44.096599 2570 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod9c35bd75_f4b0_4544_8fc9_07ae7d83fed7.slice/crio-17b24c01ed15d5fde25e53787f5636fb67c1440c733c5612618077a8f2c5b64a WatchSource:0}: Error finding container 17b24c01ed15d5fde25e53787f5636fb67c1440c733c5612618077a8f2c5b64a: Status 404 returned error can't find the container with id 17b24c01ed15d5fde25e53787f5636fb67c1440c733c5612618077a8f2c5b64a Jul 10 01:10:44.849160 ip-10-0-130-146 kubenswrapper[2570]: I0710 01:10:44.849129 2570 generic.go:356] "Generic (PLEG): container finished" podID="9c35bd75-f4b0-4544-8fc9-07ae7d83fed7" containerID="fcd66da5c305a0c525af1959a7a5a365b275ba97a0023681d0511fbd5f2a8e97" exitCode=0 Jul 10 01:10:44.849327 ip-10-0-130-146 kubenswrapper[2570]: I0710 01:10:44.849220 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/test-curl-7517-139919326745344" event={"ID":"9c35bd75-f4b0-4544-8fc9-07ae7d83fed7","Type":"ContainerDied","Data":"fcd66da5c305a0c525af1959a7a5a365b275ba97a0023681d0511fbd5f2a8e97"} Jul 10 01:10:44.849327 ip-10-0-130-146 kubenswrapper[2570]: I0710 01:10:44.849268 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/test-curl-7517-139919326745344" event={"ID":"9c35bd75-f4b0-4544-8fc9-07ae7d83fed7","Type":"ContainerStarted","Data":"17b24c01ed15d5fde25e53787f5636fb67c1440c733c5612618077a8f2c5b64a"} Jul 10 01:10:44.902612 ip-10-0-130-146 kubenswrapper[2570]: I0710 01:10:44.902573 2570 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-shared-a-48a750/rate-test-model-48a750-kserve-7dc6fff7c9-9gw6l" podUID="ba58ea7f-8427-4a47-8f16-099c76458c5d" containerName="main" containerID="cri-o://b1a4ece366723a704d200189234447a63c1c58e33e3995db4c78ba2c8a07818d" gracePeriod=45 Jul 10 01:10:45.151995 ip-10-0-130-146 kubenswrapper[2570]: I0710 01:10:45.151973 2570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-48a750/rate-test-model-48a750-kserve-7dc6fff7c9-9gw6l" Jul 10 01:10:45.236793 ip-10-0-130-146 kubenswrapper[2570]: I0710 01:10:45.236757 2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ba58ea7f-8427-4a47-8f16-099c76458c5d-tls-certs\") pod \"ba58ea7f-8427-4a47-8f16-099c76458c5d\" (UID: \"ba58ea7f-8427-4a47-8f16-099c76458c5d\") " Jul 10 01:10:45.236970 ip-10-0-130-146 kubenswrapper[2570]: I0710 01:10:45.236862 2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ba58ea7f-8427-4a47-8f16-099c76458c5d-dshm\") pod \"ba58ea7f-8427-4a47-8f16-099c76458c5d\" (UID: \"ba58ea7f-8427-4a47-8f16-099c76458c5d\") " Jul 10 01:10:45.236970 ip-10-0-130-146 kubenswrapper[2570]: I0710 01:10:45.236888 2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ba58ea7f-8427-4a47-8f16-099c76458c5d-model-cache\") pod \"ba58ea7f-8427-4a47-8f16-099c76458c5d\" (UID: \"ba58ea7f-8427-4a47-8f16-099c76458c5d\") " Jul 10 01:10:45.236970 ip-10-0-130-146 kubenswrapper[2570]: I0710 01:10:45.236919 2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ba58ea7f-8427-4a47-8f16-099c76458c5d-home\") pod \"ba58ea7f-8427-4a47-8f16-099c76458c5d\" (UID: \"ba58ea7f-8427-4a47-8f16-099c76458c5d\") " Jul 10 01:10:45.236970 ip-10-0-130-146 kubenswrapper[2570]: I0710 01:10:45.236951 2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-kvndx\" (UniqueName: \"kubernetes.io/projected/ba58ea7f-8427-4a47-8f16-099c76458c5d-kube-api-access-kvndx\") pod \"ba58ea7f-8427-4a47-8f16-099c76458c5d\" (UID: \"ba58ea7f-8427-4a47-8f16-099c76458c5d\") " Jul 10 01:10:45.237208 ip-10-0-130-146 kubenswrapper[2570]: I0710 01:10:45.236988 2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ba58ea7f-8427-4a47-8f16-099c76458c5d-tmp-dir\") pod \"ba58ea7f-8427-4a47-8f16-099c76458c5d\" (UID: \"ba58ea7f-8427-4a47-8f16-099c76458c5d\") " Jul 10 01:10:45.237256 ip-10-0-130-146 kubenswrapper[2570]: I0710 01:10:45.237194 2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ba58ea7f-8427-4a47-8f16-099c76458c5d-model-cache" (OuterVolumeSpecName: "model-cache") pod "ba58ea7f-8427-4a47-8f16-099c76458c5d" (UID: "ba58ea7f-8427-4a47-8f16-099c76458c5d"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 10 01:10:45.237256 ip-10-0-130-146 kubenswrapper[2570]: I0710 01:10:45.237204 2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ba58ea7f-8427-4a47-8f16-099c76458c5d-home" (OuterVolumeSpecName: "home") pod "ba58ea7f-8427-4a47-8f16-099c76458c5d" (UID: "ba58ea7f-8427-4a47-8f16-099c76458c5d"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 10 01:10:45.237364 ip-10-0-130-146 kubenswrapper[2570]: I0710 01:10:45.237306 2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ba58ea7f-8427-4a47-8f16-099c76458c5d-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "ba58ea7f-8427-4a47-8f16-099c76458c5d" (UID: "ba58ea7f-8427-4a47-8f16-099c76458c5d"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 10 01:10:45.237364 ip-10-0-130-146 kubenswrapper[2570]: I0710 01:10:45.237349 2570 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ba58ea7f-8427-4a47-8f16-099c76458c5d-model-cache\") on node \"ip-10-0-130-146.ec2.internal\" DevicePath \"\"" Jul 10 01:10:45.237364 ip-10-0-130-146 kubenswrapper[2570]: I0710 01:10:45.237362 2570 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ba58ea7f-8427-4a47-8f16-099c76458c5d-home\") on node \"ip-10-0-130-146.ec2.internal\" DevicePath \"\"" Jul 10 01:10:45.238955 ip-10-0-130-146 kubenswrapper[2570]: I0710 01:10:45.238927 2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ba58ea7f-8427-4a47-8f16-099c76458c5d-dshm" (OuterVolumeSpecName: "dshm") pod "ba58ea7f-8427-4a47-8f16-099c76458c5d" (UID: "ba58ea7f-8427-4a47-8f16-099c76458c5d"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 10 01:10:45.238955 ip-10-0-130-146 kubenswrapper[2570]: I0710 01:10:45.238931 2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ba58ea7f-8427-4a47-8f16-099c76458c5d-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "ba58ea7f-8427-4a47-8f16-099c76458c5d" (UID: "ba58ea7f-8427-4a47-8f16-099c76458c5d"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 10 01:10:45.239116 ip-10-0-130-146 kubenswrapper[2570]: I0710 01:10:45.239005 2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ba58ea7f-8427-4a47-8f16-099c76458c5d-kube-api-access-kvndx" (OuterVolumeSpecName: "kube-api-access-kvndx") pod "ba58ea7f-8427-4a47-8f16-099c76458c5d" (UID: "ba58ea7f-8427-4a47-8f16-099c76458c5d"). InnerVolumeSpecName "kube-api-access-kvndx". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 10 01:10:45.338605 ip-10-0-130-146 kubenswrapper[2570]: I0710 01:10:45.338577 2570 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kvndx\" (UniqueName: \"kubernetes.io/projected/ba58ea7f-8427-4a47-8f16-099c76458c5d-kube-api-access-kvndx\") on node \"ip-10-0-130-146.ec2.internal\" DevicePath \"\"" Jul 10 01:10:45.338605 ip-10-0-130-146 kubenswrapper[2570]: I0710 01:10:45.338602 2570 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ba58ea7f-8427-4a47-8f16-099c76458c5d-tmp-dir\") on node \"ip-10-0-130-146.ec2.internal\" DevicePath \"\"" Jul 10 01:10:45.338605 ip-10-0-130-146 kubenswrapper[2570]: I0710 01:10:45.338611 2570 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ba58ea7f-8427-4a47-8f16-099c76458c5d-tls-certs\") on node \"ip-10-0-130-146.ec2.internal\" DevicePath \"\"" Jul 10 01:10:45.338825 ip-10-0-130-146 kubenswrapper[2570]: I0710 01:10:45.338620 2570 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ba58ea7f-8427-4a47-8f16-099c76458c5d-dshm\") on node \"ip-10-0-130-146.ec2.internal\" DevicePath \"\"" Jul 10 01:10:45.855602 ip-10-0-130-146 kubenswrapper[2570]: I0710 01:10:45.855526 2570 generic.go:356] "Generic (PLEG): container finished" podID="ba58ea7f-8427-4a47-8f16-099c76458c5d" containerID="b1a4ece366723a704d200189234447a63c1c58e33e3995db4c78ba2c8a07818d" exitCode=0 Jul 10 01:10:45.855602 ip-10-0-130-146 kubenswrapper[2570]: I0710 01:10:45.855567 2570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-48a750/rate-test-model-48a750-kserve-7dc6fff7c9-9gw6l" Jul 10 01:10:45.855775 ip-10-0-130-146 kubenswrapper[2570]: I0710 01:10:45.855606 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-48a750/rate-test-model-48a750-kserve-7dc6fff7c9-9gw6l" event={"ID":"ba58ea7f-8427-4a47-8f16-099c76458c5d","Type":"ContainerDied","Data":"b1a4ece366723a704d200189234447a63c1c58e33e3995db4c78ba2c8a07818d"} Jul 10 01:10:45.855775 ip-10-0-130-146 kubenswrapper[2570]: I0710 01:10:45.855644 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-48a750/rate-test-model-48a750-kserve-7dc6fff7c9-9gw6l" event={"ID":"ba58ea7f-8427-4a47-8f16-099c76458c5d","Type":"ContainerDied","Data":"64000f4bf9cdc86bf4e67d82b753e92f9e2c1daf9cec586934e3b4b27e593e0b"} Jul 10 01:10:45.855775 ip-10-0-130-146 kubenswrapper[2570]: I0710 01:10:45.855662 2570 scope.go:117] "RemoveContainer" containerID="b1a4ece366723a704d200189234447a63c1c58e33e3995db4c78ba2c8a07818d" Jul 10 01:10:45.864626 ip-10-0-130-146 kubenswrapper[2570]: I0710 01:10:45.864598 2570 scope.go:117] "RemoveContainer" containerID="b1a4ece366723a704d200189234447a63c1c58e33e3995db4c78ba2c8a07818d" Jul 10 01:10:45.864869 ip-10-0-130-146 kubenswrapper[2570]: E0710 01:10:45.864851 2570 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b1a4ece366723a704d200189234447a63c1c58e33e3995db4c78ba2c8a07818d\": container with ID starting with b1a4ece366723a704d200189234447a63c1c58e33e3995db4c78ba2c8a07818d not found: ID does not exist" containerID="b1a4ece366723a704d200189234447a63c1c58e33e3995db4c78ba2c8a07818d" Jul 10 01:10:45.864931 ip-10-0-130-146 kubenswrapper[2570]: I0710 01:10:45.864878 2570 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b1a4ece366723a704d200189234447a63c1c58e33e3995db4c78ba2c8a07818d"} err="failed to get container status \"b1a4ece366723a704d200189234447a63c1c58e33e3995db4c78ba2c8a07818d\": rpc error: code = NotFound desc = could not find container \"b1a4ece366723a704d200189234447a63c1c58e33e3995db4c78ba2c8a07818d\": container with ID starting with b1a4ece366723a704d200189234447a63c1c58e33e3995db4c78ba2c8a07818d not found: ID does not exist" Jul 10 01:10:45.879228 ip-10-0-130-146 kubenswrapper[2570]: I0710 01:10:45.879196 2570 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-48a750/rate-test-model-48a750-kserve-7dc6fff7c9-9gw6l"] Jul 10 01:10:45.885084 ip-10-0-130-146 kubenswrapper[2570]: I0710 01:10:45.885043 2570 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-a-48a750/rate-test-model-48a750-kserve-7dc6fff7c9-9gw6l"] Jul 10 01:10:45.888721 ip-10-0-130-146 kubenswrapper[2570]: I0710 01:10:45.888679 2570 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-shared-b-aaea96/rate-test-model-aaea96-kserve-59f4c77cdc-kb6c8" podUID="2f77297e-4102-4012-89da-2d47ca86526b" containerName="main" containerID="cri-o://46ad053d0d507df9019e8896bdc68df2e7737e858e0d89be41f2399f19c87cfb" gracePeriod=45 Jul 10 01:10:46.013661 ip-10-0-130-146 kubenswrapper[2570]: I0710 01:10:46.013635 2570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/test-curl-7517-139919326745344" Jul 10 01:10:46.037779 ip-10-0-130-146 kubenswrapper[2570]: I0710 01:10:46.037755 2570 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_test-curl-7517-139919326745344_9c35bd75-f4b0-4544-8fc9-07ae7d83fed7/test-curl-7517-139919326745344/0.log" Jul 10 01:10:46.063938 ip-10-0-130-146 kubenswrapper[2570]: I0710 01:10:46.063911 2570 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/test-curl-7517-139919326745344"] Jul 10 01:10:46.066029 ip-10-0-130-146 kubenswrapper[2570]: I0710 01:10:46.066007 2570 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/test-curl-7517-139919326745344"] Jul 10 01:10:46.132726 ip-10-0-130-146 kubenswrapper[2570]: I0710 01:10:46.132704 2570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-aaea96/rate-test-model-aaea96-kserve-59f4c77cdc-kb6c8" Jul 10 01:10:46.145869 ip-10-0-130-146 kubenswrapper[2570]: I0710 01:10:46.145847 2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-xk6mj\" (UniqueName: \"kubernetes.io/projected/9c35bd75-f4b0-4544-8fc9-07ae7d83fed7-kube-api-access-xk6mj\") pod \"9c35bd75-f4b0-4544-8fc9-07ae7d83fed7\" (UID: \"9c35bd75-f4b0-4544-8fc9-07ae7d83fed7\") " Jul 10 01:10:46.147727 ip-10-0-130-146 kubenswrapper[2570]: I0710 01:10:46.147705 2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9c35bd75-f4b0-4544-8fc9-07ae7d83fed7-kube-api-access-xk6mj" (OuterVolumeSpecName: "kube-api-access-xk6mj") pod "9c35bd75-f4b0-4544-8fc9-07ae7d83fed7" (UID: "9c35bd75-f4b0-4544-8fc9-07ae7d83fed7"). InnerVolumeSpecName "kube-api-access-xk6mj". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 10 01:10:46.192599 ip-10-0-130-146 kubenswrapper[2570]: I0710 01:10:46.192564 2570 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/test-curl-7517-139919326745344"] Jul 10 01:10:46.192982 ip-10-0-130-146 kubenswrapper[2570]: I0710 01:10:46.192937 2570 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ba58ea7f-8427-4a47-8f16-099c76458c5d" containerName="main" Jul 10 01:10:46.192982 ip-10-0-130-146 kubenswrapper[2570]: I0710 01:10:46.192948 2570 state_mem.go:107] "Deleted CPUSet assignment" podUID="ba58ea7f-8427-4a47-8f16-099c76458c5d" containerName="main" Jul 10 01:10:46.192982 ip-10-0-130-146 kubenswrapper[2570]: I0710 01:10:46.192967 2570 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2f77297e-4102-4012-89da-2d47ca86526b" containerName="main" Jul 10 01:10:46.192982 ip-10-0-130-146 kubenswrapper[2570]: I0710 01:10:46.192972 2570 state_mem.go:107] "Deleted CPUSet assignment" podUID="2f77297e-4102-4012-89da-2d47ca86526b" containerName="main" Jul 10 01:10:46.193152 ip-10-0-130-146 kubenswrapper[2570]: I0710 01:10:46.192988 2570 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9c35bd75-f4b0-4544-8fc9-07ae7d83fed7" containerName="test-curl-7517-139919326745344" Jul 10 01:10:46.193152 ip-10-0-130-146 kubenswrapper[2570]: I0710 01:10:46.192995 2570 state_mem.go:107] "Deleted CPUSet assignment" podUID="9c35bd75-f4b0-4544-8fc9-07ae7d83fed7" containerName="test-curl-7517-139919326745344" Jul 10 01:10:46.193152 ip-10-0-130-146 kubenswrapper[2570]: I0710 01:10:46.193060 2570 memory_manager.go:368] "RemoveStaleState removing state" podUID="2f77297e-4102-4012-89da-2d47ca86526b" containerName="main" Jul 10 01:10:46.193152 ip-10-0-130-146 kubenswrapper[2570]: I0710 01:10:46.193069 2570 memory_manager.go:368] "RemoveStaleState removing state" podUID="9c35bd75-f4b0-4544-8fc9-07ae7d83fed7" containerName="test-curl-7517-139919326745344" Jul 10 01:10:46.193152 ip-10-0-130-146 kubenswrapper[2570]: I0710 01:10:46.193079 2570 memory_manager.go:368] "RemoveStaleState removing state" podUID="ba58ea7f-8427-4a47-8f16-099c76458c5d" containerName="main" Jul 10 01:10:46.196159 ip-10-0-130-146 kubenswrapper[2570]: I0710 01:10:46.196138 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/test-curl-7517-139919326745344" Jul 10 01:10:46.201413 ip-10-0-130-146 kubenswrapper[2570]: I0710 01:10:46.201389 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/test-curl-7517-139919326745344"] Jul 10 01:10:46.246671 ip-10-0-130-146 kubenswrapper[2570]: I0710 01:10:46.246639 2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2f77297e-4102-4012-89da-2d47ca86526b-dshm\") pod \"2f77297e-4102-4012-89da-2d47ca86526b\" (UID: \"2f77297e-4102-4012-89da-2d47ca86526b\") " Jul 10 01:10:46.246671 ip-10-0-130-146 kubenswrapper[2570]: I0710 01:10:46.246677 2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2f77297e-4102-4012-89da-2d47ca86526b-tmp-dir\") pod \"2f77297e-4102-4012-89da-2d47ca86526b\" (UID: \"2f77297e-4102-4012-89da-2d47ca86526b\") " Jul 10 01:10:46.246933 ip-10-0-130-146 kubenswrapper[2570]: I0710 01:10:46.246747 2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-6ktmb\" (UniqueName: \"kubernetes.io/projected/2f77297e-4102-4012-89da-2d47ca86526b-kube-api-access-6ktmb\") pod \"2f77297e-4102-4012-89da-2d47ca86526b\" (UID: \"2f77297e-4102-4012-89da-2d47ca86526b\") " Jul 10 01:10:46.246933 ip-10-0-130-146 kubenswrapper[2570]: I0710 01:10:46.246792 2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2f77297e-4102-4012-89da-2d47ca86526b-model-cache\") pod \"2f77297e-4102-4012-89da-2d47ca86526b\" (UID: \"2f77297e-4102-4012-89da-2d47ca86526b\") " Jul 10 01:10:46.246933 ip-10-0-130-146 kubenswrapper[2570]: I0710 01:10:46.246815 2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2f77297e-4102-4012-89da-2d47ca86526b-tls-certs\") pod \"2f77297e-4102-4012-89da-2d47ca86526b\" (UID: \"2f77297e-4102-4012-89da-2d47ca86526b\") " Jul 10 01:10:46.246933 ip-10-0-130-146 kubenswrapper[2570]: I0710 01:10:46.246888 2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2f77297e-4102-4012-89da-2d47ca86526b-home\") pod \"2f77297e-4102-4012-89da-2d47ca86526b\" (UID: \"2f77297e-4102-4012-89da-2d47ca86526b\") " Jul 10 01:10:46.247182 ip-10-0-130-146 kubenswrapper[2570]: I0710 01:10:46.246967 2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2f77297e-4102-4012-89da-2d47ca86526b-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "2f77297e-4102-4012-89da-2d47ca86526b" (UID: "2f77297e-4102-4012-89da-2d47ca86526b"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 10 01:10:46.247182 ip-10-0-130-146 kubenswrapper[2570]: I0710 01:10:46.247005 2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2f77297e-4102-4012-89da-2d47ca86526b-model-cache" (OuterVolumeSpecName: "model-cache") pod "2f77297e-4102-4012-89da-2d47ca86526b" (UID: "2f77297e-4102-4012-89da-2d47ca86526b"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 10 01:10:46.247182 ip-10-0-130-146 kubenswrapper[2570]: I0710 01:10:46.247168 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5rssd\" (UniqueName: \"kubernetes.io/projected/a24448b4-b598-4d1b-a841-7ed31271a8f1-kube-api-access-5rssd\") pod \"test-curl-7517-139919326745344\" (UID: \"a24448b4-b598-4d1b-a841-7ed31271a8f1\") " pod="openshift-ingress/test-curl-7517-139919326745344" Jul 10 01:10:46.247331 ip-10-0-130-146 kubenswrapper[2570]: I0710 01:10:46.247206 2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2f77297e-4102-4012-89da-2d47ca86526b-home" (OuterVolumeSpecName: "home") pod "2f77297e-4102-4012-89da-2d47ca86526b" (UID: "2f77297e-4102-4012-89da-2d47ca86526b"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 10 01:10:46.247331 ip-10-0-130-146 kubenswrapper[2570]: I0710 01:10:46.247255 2570 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xk6mj\" (UniqueName: \"kubernetes.io/projected/9c35bd75-f4b0-4544-8fc9-07ae7d83fed7-kube-api-access-xk6mj\") on node \"ip-10-0-130-146.ec2.internal\" DevicePath \"\"" Jul 10 01:10:46.247331 ip-10-0-130-146 kubenswrapper[2570]: I0710 01:10:46.247279 2570 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2f77297e-4102-4012-89da-2d47ca86526b-tmp-dir\") on node \"ip-10-0-130-146.ec2.internal\" DevicePath \"\"" Jul 10 01:10:46.247331 ip-10-0-130-146 kubenswrapper[2570]: I0710 01:10:46.247315 2570 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2f77297e-4102-4012-89da-2d47ca86526b-model-cache\") on node \"ip-10-0-130-146.ec2.internal\" DevicePath \"\"" Jul 10 01:10:46.248957 ip-10-0-130-146 kubenswrapper[2570]: I0710 01:10:46.248935 2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2f77297e-4102-4012-89da-2d47ca86526b-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "2f77297e-4102-4012-89da-2d47ca86526b" (UID: "2f77297e-4102-4012-89da-2d47ca86526b"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 10 01:10:46.249041 ip-10-0-130-146 kubenswrapper[2570]: I0710 01:10:46.248937 2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2f77297e-4102-4012-89da-2d47ca86526b-dshm" (OuterVolumeSpecName: "dshm") pod "2f77297e-4102-4012-89da-2d47ca86526b" (UID: "2f77297e-4102-4012-89da-2d47ca86526b"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 10 01:10:46.249041 ip-10-0-130-146 kubenswrapper[2570]: I0710 01:10:46.248968 2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2f77297e-4102-4012-89da-2d47ca86526b-kube-api-access-6ktmb" (OuterVolumeSpecName: "kube-api-access-6ktmb") pod "2f77297e-4102-4012-89da-2d47ca86526b" (UID: "2f77297e-4102-4012-89da-2d47ca86526b"). InnerVolumeSpecName "kube-api-access-6ktmb". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 10 01:10:46.348417 ip-10-0-130-146 kubenswrapper[2570]: I0710 01:10:46.348384 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5rssd\" (UniqueName: \"kubernetes.io/projected/a24448b4-b598-4d1b-a841-7ed31271a8f1-kube-api-access-5rssd\") pod \"test-curl-7517-139919326745344\" (UID: \"a24448b4-b598-4d1b-a841-7ed31271a8f1\") " pod="openshift-ingress/test-curl-7517-139919326745344" Jul 10 01:10:46.348557 ip-10-0-130-146 kubenswrapper[2570]: I0710 01:10:46.348436 2570 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2f77297e-4102-4012-89da-2d47ca86526b-home\") on node \"ip-10-0-130-146.ec2.internal\" DevicePath \"\"" Jul 10 01:10:46.348557 ip-10-0-130-146 kubenswrapper[2570]: I0710 01:10:46.348447 2570 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2f77297e-4102-4012-89da-2d47ca86526b-dshm\") on node \"ip-10-0-130-146.ec2.internal\" DevicePath \"\"" Jul 10 01:10:46.348557 ip-10-0-130-146 kubenswrapper[2570]: I0710 01:10:46.348456 2570 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6ktmb\" (UniqueName: \"kubernetes.io/projected/2f77297e-4102-4012-89da-2d47ca86526b-kube-api-access-6ktmb\") on node \"ip-10-0-130-146.ec2.internal\" DevicePath \"\"" Jul 10 01:10:46.348557 ip-10-0-130-146 kubenswrapper[2570]: I0710 01:10:46.348465 2570 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2f77297e-4102-4012-89da-2d47ca86526b-tls-certs\") on node \"ip-10-0-130-146.ec2.internal\" DevicePath \"\"" Jul 10 01:10:46.356397 ip-10-0-130-146 kubenswrapper[2570]: I0710 01:10:46.356374 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5rssd\" (UniqueName: \"kubernetes.io/projected/a24448b4-b598-4d1b-a841-7ed31271a8f1-kube-api-access-5rssd\") pod \"test-curl-7517-139919326745344\" (UID: \"a24448b4-b598-4d1b-a841-7ed31271a8f1\") " pod="openshift-ingress/test-curl-7517-139919326745344" Jul 10 01:10:46.506644 ip-10-0-130-146 kubenswrapper[2570]: I0710 01:10:46.506559 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/test-curl-7517-139919326745344" Jul 10 01:10:46.635988 ip-10-0-130-146 kubenswrapper[2570]: I0710 01:10:46.635959 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/test-curl-7517-139919326745344"] Jul 10 01:10:46.637684 ip-10-0-130-146 kubenswrapper[2570]: W0710 01:10:46.637656 2570 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-poda24448b4_b598_4d1b_a841_7ed31271a8f1.slice/crio-a9468a8d1f5197337727aa8a3a496512c03b82229056fd1f339b886e06f60981 WatchSource:0}: Error finding container a9468a8d1f5197337727aa8a3a496512c03b82229056fd1f339b886e06f60981: Status 404 returned error can't find the container with id a9468a8d1f5197337727aa8a3a496512c03b82229056fd1f339b886e06f60981 Jul 10 01:10:46.862963 ip-10-0-130-146 kubenswrapper[2570]: I0710 01:10:46.862930 2570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/test-curl-7517-139919326745344" Jul 10 01:10:46.863123 ip-10-0-130-146 kubenswrapper[2570]: I0710 01:10:46.862933 2570 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="17b24c01ed15d5fde25e53787f5636fb67c1440c733c5612618077a8f2c5b64a" Jul 10 01:10:46.864039 ip-10-0-130-146 kubenswrapper[2570]: I0710 01:10:46.864009 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/test-curl-7517-139919326745344" event={"ID":"a24448b4-b598-4d1b-a841-7ed31271a8f1","Type":"ContainerStarted","Data":"a9468a8d1f5197337727aa8a3a496512c03b82229056fd1f339b886e06f60981"} Jul 10 01:10:46.865248 ip-10-0-130-146 kubenswrapper[2570]: I0710 01:10:46.865225 2570 generic.go:356] "Generic (PLEG): container finished" podID="2f77297e-4102-4012-89da-2d47ca86526b" containerID="46ad053d0d507df9019e8896bdc68df2e7737e858e0d89be41f2399f19c87cfb" exitCode=0 Jul 10 01:10:46.865344 ip-10-0-130-146 kubenswrapper[2570]: I0710 01:10:46.865259 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-aaea96/rate-test-model-aaea96-kserve-59f4c77cdc-kb6c8" event={"ID":"2f77297e-4102-4012-89da-2d47ca86526b","Type":"ContainerDied","Data":"46ad053d0d507df9019e8896bdc68df2e7737e858e0d89be41f2399f19c87cfb"} Jul 10 01:10:46.865344 ip-10-0-130-146 kubenswrapper[2570]: I0710 01:10:46.865285 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-aaea96/rate-test-model-aaea96-kserve-59f4c77cdc-kb6c8" event={"ID":"2f77297e-4102-4012-89da-2d47ca86526b","Type":"ContainerDied","Data":"d156477538fe999bfb24951a97ac54ed1bcee3ff276b38ae84940f51c322d39e"} Jul 10 01:10:46.865344 ip-10-0-130-146 kubenswrapper[2570]: I0710 01:10:46.865262 2570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-aaea96/rate-test-model-aaea96-kserve-59f4c77cdc-kb6c8" Jul 10 01:10:46.865344 ip-10-0-130-146 kubenswrapper[2570]: I0710 01:10:46.865306 2570 scope.go:117] "RemoveContainer" containerID="46ad053d0d507df9019e8896bdc68df2e7737e858e0d89be41f2399f19c87cfb" Jul 10 01:10:46.874580 ip-10-0-130-146 kubenswrapper[2570]: I0710 01:10:46.874565 2570 scope.go:117] "RemoveContainer" containerID="46ad053d0d507df9019e8896bdc68df2e7737e858e0d89be41f2399f19c87cfb" Jul 10 01:10:46.874824 ip-10-0-130-146 kubenswrapper[2570]: E0710 01:10:46.874809 2570 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"46ad053d0d507df9019e8896bdc68df2e7737e858e0d89be41f2399f19c87cfb\": container with ID starting with 46ad053d0d507df9019e8896bdc68df2e7737e858e0d89be41f2399f19c87cfb not found: ID does not exist" containerID="46ad053d0d507df9019e8896bdc68df2e7737e858e0d89be41f2399f19c87cfb" Jul 10 01:10:46.874878 ip-10-0-130-146 kubenswrapper[2570]: I0710 01:10:46.874831 2570 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"46ad053d0d507df9019e8896bdc68df2e7737e858e0d89be41f2399f19c87cfb"} err="failed to get container status \"46ad053d0d507df9019e8896bdc68df2e7737e858e0d89be41f2399f19c87cfb\": rpc error: code = NotFound desc = could not find container \"46ad053d0d507df9019e8896bdc68df2e7737e858e0d89be41f2399f19c87cfb\": container with ID starting with 46ad053d0d507df9019e8896bdc68df2e7737e858e0d89be41f2399f19c87cfb not found: ID does not exist" Jul 10 01:10:46.875471 ip-10-0-130-146 kubenswrapper[2570]: I0710 01:10:46.875449 2570 status_manager.go:1027] "Pod was deleted and then recreated, skipping status update" pod="openshift-ingress/test-curl-7517-139919326745344" oldPodUID="9c35bd75-f4b0-4544-8fc9-07ae7d83fed7" podUID="a24448b4-b598-4d1b-a841-7ed31271a8f1" Jul 10 01:10:46.890434 ip-10-0-130-146 kubenswrapper[2570]: I0710 01:10:46.890409 2570 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-aaea96/rate-test-model-aaea96-kserve-59f4c77cdc-kb6c8"] Jul 10 01:10:46.892648 ip-10-0-130-146 kubenswrapper[2570]: I0710 01:10:46.892625 2570 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-b-aaea96/rate-test-model-aaea96-kserve-59f4c77cdc-kb6c8"] Jul 10 01:10:47.454268 ip-10-0-130-146 kubenswrapper[2570]: I0710 01:10:47.454240 2570 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="2f77297e-4102-4012-89da-2d47ca86526b" path="/var/lib/kubelet/pods/2f77297e-4102-4012-89da-2d47ca86526b/volumes" Jul 10 01:10:47.454655 ip-10-0-130-146 kubenswrapper[2570]: I0710 01:10:47.454561 2570 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="9c35bd75-f4b0-4544-8fc9-07ae7d83fed7" path="/var/lib/kubelet/pods/9c35bd75-f4b0-4544-8fc9-07ae7d83fed7/volumes" Jul 10 01:10:47.454919 ip-10-0-130-146 kubenswrapper[2570]: I0710 01:10:47.454906 2570 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="ba58ea7f-8427-4a47-8f16-099c76458c5d" path="/var/lib/kubelet/pods/ba58ea7f-8427-4a47-8f16-099c76458c5d/volumes" Jul 10 01:10:47.872230 ip-10-0-130-146 kubenswrapper[2570]: I0710 01:10:47.872197 2570 generic.go:356] "Generic (PLEG): container finished" podID="a24448b4-b598-4d1b-a841-7ed31271a8f1" containerID="b85ce6e9430bf3525df0adea65919da4b0211c1c64105c9f7a72608326e0e684" exitCode=0 Jul 10 01:10:47.872428 ip-10-0-130-146 kubenswrapper[2570]: I0710 01:10:47.872281 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/test-curl-7517-139919326745344" event={"ID":"a24448b4-b598-4d1b-a841-7ed31271a8f1","Type":"ContainerDied","Data":"b85ce6e9430bf3525df0adea65919da4b0211c1c64105c9f7a72608326e0e684"} Jul 10 01:10:49.036988 ip-10-0-130-146 kubenswrapper[2570]: I0710 01:10:49.036965 2570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/test-curl-7517-139919326745344" Jul 10 01:10:49.058074 ip-10-0-130-146 kubenswrapper[2570]: I0710 01:10:49.058033 2570 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_test-curl-7517-139919326745344_a24448b4-b598-4d1b-a841-7ed31271a8f1/test-curl-7517-139919326745344/0.log" Jul 10 01:10:49.082254 ip-10-0-130-146 kubenswrapper[2570]: I0710 01:10:49.082232 2570 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/test-curl-7517-139919326745344"] Jul 10 01:10:49.084262 ip-10-0-130-146 kubenswrapper[2570]: I0710 01:10:49.084242 2570 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/test-curl-7517-139919326745344"] Jul 10 01:10:49.171962 ip-10-0-130-146 kubenswrapper[2570]: I0710 01:10:49.171909 2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-5rssd\" (UniqueName: \"kubernetes.io/projected/a24448b4-b598-4d1b-a841-7ed31271a8f1-kube-api-access-5rssd\") pod \"a24448b4-b598-4d1b-a841-7ed31271a8f1\" (UID: \"a24448b4-b598-4d1b-a841-7ed31271a8f1\") " Jul 10 01:10:49.173920 ip-10-0-130-146 kubenswrapper[2570]: I0710 01:10:49.173898 2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a24448b4-b598-4d1b-a841-7ed31271a8f1-kube-api-access-5rssd" (OuterVolumeSpecName: "kube-api-access-5rssd") pod "a24448b4-b598-4d1b-a841-7ed31271a8f1" (UID: "a24448b4-b598-4d1b-a841-7ed31271a8f1"). InnerVolumeSpecName "kube-api-access-5rssd". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 10 01:10:49.273353 ip-10-0-130-146 kubenswrapper[2570]: I0710 01:10:49.273324 2570 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5rssd\" (UniqueName: \"kubernetes.io/projected/a24448b4-b598-4d1b-a841-7ed31271a8f1-kube-api-access-5rssd\") on node \"ip-10-0-130-146.ec2.internal\" DevicePath \"\"" Jul 10 01:10:49.454638 ip-10-0-130-146 kubenswrapper[2570]: I0710 01:10:49.454563 2570 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="a24448b4-b598-4d1b-a841-7ed31271a8f1" path="/var/lib/kubelet/pods/a24448b4-b598-4d1b-a841-7ed31271a8f1/volumes" Jul 10 01:10:49.887657 ip-10-0-130-146 kubenswrapper[2570]: I0710 01:10:49.887628 2570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/test-curl-7517-139919326745344" Jul 10 01:10:49.887657 ip-10-0-130-146 kubenswrapper[2570]: I0710 01:10:49.887651 2570 scope.go:117] "RemoveContainer" containerID="b85ce6e9430bf3525df0adea65919da4b0211c1c64105c9f7a72608326e0e684" Jul 10 01:10:56.380061 ip-10-0-130-146 kubenswrapper[2570]: I0710 01:10:56.380026 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-g92hs"] Jul 10 01:11:05.879135 ip-10-0-130-146 kubenswrapper[2570]: I0710 01:11:05.879100 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-g92hs"] Jul 10 01:11:09.285576 ip-10-0-130-146 kubenswrapper[2570]: I0710 01:11:09.285531 2570 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-54ccd4f945-f4q9r_b62d531b-b70e-48c2-a443-8bb683e26da2/manager/0.log" Jul 10 01:11:09.292613 ip-10-0-130-146 kubenswrapper[2570]: I0710 01:11:09.292593 2570 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-54ccd4f945-f4q9r_b62d531b-b70e-48c2-a443-8bb683e26da2/copy-manifests/0.log" Jul 10 01:11:10.636969 ip-10-0-130-146 kubenswrapper[2570]: I0710 01:11:10.636937 2570 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759k68sb_a062b873-3bc8-4dc1-96e7-63353d46e0b5/util/0.log" Jul 10 01:11:10.643450 ip-10-0-130-146 kubenswrapper[2570]: I0710 01:11:10.643426 2570 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759k68sb_a062b873-3bc8-4dc1-96e7-63353d46e0b5/pull/0.log" Jul 10 01:11:10.649469 ip-10-0-130-146 kubenswrapper[2570]: I0710 01:11:10.649447 2570 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759k68sb_a062b873-3bc8-4dc1-96e7-63353d46e0b5/extract/0.log" Jul 10 01:11:10.767660 ip-10-0-130-146 kubenswrapper[2570]: I0710 01:11:10.767633 2570 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0hr2tc_b407e13a-88fb-43a1-94fc-ab40c580d6c2/util/0.log" Jul 10 01:11:10.774210 ip-10-0-130-146 kubenswrapper[2570]: I0710 01:11:10.774186 2570 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0hr2tc_b407e13a-88fb-43a1-94fc-ab40c580d6c2/pull/0.log" Jul 10 01:11:10.780382 ip-10-0-130-146 kubenswrapper[2570]: I0710 01:11:10.780362 2570 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0hr2tc_b407e13a-88fb-43a1-94fc-ab40c580d6c2/extract/0.log" Jul 10 01:11:10.891955 ip-10-0-130-146 kubenswrapper[2570]: I0710 01:11:10.891873 2570 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73rs9vd_48dc2b69-a1c1-46b4-9f79-02abaf29e411/util/0.log" Jul 10 01:11:10.898395 ip-10-0-130-146 kubenswrapper[2570]: I0710 01:11:10.898369 2570 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73rs9vd_48dc2b69-a1c1-46b4-9f79-02abaf29e411/pull/0.log" Jul 10 01:11:10.904633 ip-10-0-130-146 kubenswrapper[2570]: I0710 01:11:10.904613 2570 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73rs9vd_48dc2b69-a1c1-46b4-9f79-02abaf29e411/extract/0.log" Jul 10 01:11:11.015801 ip-10-0-130-146 kubenswrapper[2570]: I0710 01:11:11.015771 2570 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1pfl48_126a17da-6d3b-4880-b310-d0cf7612be39/util/0.log" Jul 10 01:11:11.022378 ip-10-0-130-146 kubenswrapper[2570]: I0710 01:11:11.022354 2570 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1pfl48_126a17da-6d3b-4880-b310-d0cf7612be39/pull/0.log" Jul 10 01:11:11.028597 ip-10-0-130-146 kubenswrapper[2570]: I0710 01:11:11.028574 2570 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1pfl48_126a17da-6d3b-4880-b310-d0cf7612be39/extract/0.log" Jul 10 01:11:11.498917 ip-10-0-130-146 kubenswrapper[2570]: I0710 01:11:11.498887 2570 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-console-plugin-66b5774985-v9f84_84dc7f0b-4d03-461a-a462-1147a97ac1ad/kuadrant-console-plugin/0.log" Jul 10 01:11:11.878518 ip-10-0-130-146 kubenswrapper[2570]: I0710 01:11:11.878485 2570 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-limitador-54d6d988d5-g92hs_7ef64d51-9c51-48f6-b409-e157f1ec45a5/limitador/0.log" Jul 10 01:11:12.693518 ip-10-0-130-146 kubenswrapper[2570]: I0710 01:11:12.693489 2570 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-7cb6b7dbc6-wwmdk_49d9ee95-a4bb-4549-b5dc-92c63e33c475/kube-auth-proxy/0.log" Jul 10 01:11:13.861866 ip-10-0-130-146 kubenswrapper[2570]: I0710 01:11:13.861838 2570 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbrs2lp_7debf947-0a80-4ee9-945d-b9ee2bfc98cb/main/0.log" Jul 10 01:11:14.089529 ip-10-0-130-146 kubenswrapper[2570]: I0710 01:11:14.089494 2570 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_premium-simulated-simulated-premium-kserve-5fc6fb456f-2tjdx_e93dbf6e-62b9-4fae-b48a-299a00e3db81/main/0.log" Jul 10 01:11:18.238190 ip-10-0-130-146 kubenswrapper[2570]: I0710 01:11:18.238157 2570 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-must-gather-z5pqd/must-gather-t52zq"] Jul 10 01:11:18.238564 ip-10-0-130-146 kubenswrapper[2570]: I0710 01:11:18.238517 2570 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a24448b4-b598-4d1b-a841-7ed31271a8f1" containerName="test-curl-7517-139919326745344" Jul 10 01:11:18.238564 ip-10-0-130-146 kubenswrapper[2570]: I0710 01:11:18.238529 2570 state_mem.go:107] "Deleted CPUSet assignment" podUID="a24448b4-b598-4d1b-a841-7ed31271a8f1" containerName="test-curl-7517-139919326745344" Jul 10 01:11:18.238695 ip-10-0-130-146 kubenswrapper[2570]: I0710 01:11:18.238595 2570 memory_manager.go:368] "RemoveStaleState removing state" podUID="a24448b4-b598-4d1b-a841-7ed31271a8f1" containerName="test-curl-7517-139919326745344" Jul 10 01:11:18.243885 ip-10-0-130-146 kubenswrapper[2570]: I0710 01:11:18.243862 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-z5pqd/must-gather-t52zq" Jul 10 01:11:18.246370 ip-10-0-130-146 kubenswrapper[2570]: I0710 01:11:18.246347 2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-z5pqd\"/\"openshift-service-ca.crt\"" Jul 10 01:11:18.247399 ip-10-0-130-146 kubenswrapper[2570]: I0710 01:11:18.247377 2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-z5pqd\"/\"default-dockercfg-dp87v\"" Jul 10 01:11:18.247514 ip-10-0-130-146 kubenswrapper[2570]: I0710 01:11:18.247419 2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-z5pqd\"/\"kube-root-ca.crt\"" Jul 10 01:11:18.251846 ip-10-0-130-146 kubenswrapper[2570]: I0710 01:11:18.251827 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-z5pqd/must-gather-t52zq"] Jul 10 01:11:18.325115 ip-10-0-130-146 kubenswrapper[2570]: I0710 01:11:18.325077 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/921aec8a-7838-47b4-9e61-6b7f43f4cf54-must-gather-output\") pod \"must-gather-t52zq\" (UID: \"921aec8a-7838-47b4-9e61-6b7f43f4cf54\") " pod="openshift-must-gather-z5pqd/must-gather-t52zq" Jul 10 01:11:18.325294 ip-10-0-130-146 kubenswrapper[2570]: I0710 01:11:18.325163 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lkbzj\" (UniqueName: \"kubernetes.io/projected/921aec8a-7838-47b4-9e61-6b7f43f4cf54-kube-api-access-lkbzj\") pod \"must-gather-t52zq\" (UID: \"921aec8a-7838-47b4-9e61-6b7f43f4cf54\") " pod="openshift-must-gather-z5pqd/must-gather-t52zq" Jul 10 01:11:18.426213 ip-10-0-130-146 kubenswrapper[2570]: I0710 01:11:18.426175 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lkbzj\" (UniqueName: \"kubernetes.io/projected/921aec8a-7838-47b4-9e61-6b7f43f4cf54-kube-api-access-lkbzj\") pod \"must-gather-t52zq\" (UID: \"921aec8a-7838-47b4-9e61-6b7f43f4cf54\") " pod="openshift-must-gather-z5pqd/must-gather-t52zq" Jul 10 01:11:18.426389 ip-10-0-130-146 kubenswrapper[2570]: I0710 01:11:18.426231 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/921aec8a-7838-47b4-9e61-6b7f43f4cf54-must-gather-output\") pod \"must-gather-t52zq\" (UID: \"921aec8a-7838-47b4-9e61-6b7f43f4cf54\") " pod="openshift-must-gather-z5pqd/must-gather-t52zq" Jul 10 01:11:18.426559 ip-10-0-130-146 kubenswrapper[2570]: I0710 01:11:18.426542 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/921aec8a-7838-47b4-9e61-6b7f43f4cf54-must-gather-output\") pod \"must-gather-t52zq\" (UID: \"921aec8a-7838-47b4-9e61-6b7f43f4cf54\") " pod="openshift-must-gather-z5pqd/must-gather-t52zq" Jul 10 01:11:18.434648 ip-10-0-130-146 kubenswrapper[2570]: I0710 01:11:18.434615 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-lkbzj\" (UniqueName: \"kubernetes.io/projected/921aec8a-7838-47b4-9e61-6b7f43f4cf54-kube-api-access-lkbzj\") pod \"must-gather-t52zq\" (UID: \"921aec8a-7838-47b4-9e61-6b7f43f4cf54\") " pod="openshift-must-gather-z5pqd/must-gather-t52zq" Jul 10 01:11:18.555148 ip-10-0-130-146 kubenswrapper[2570]: I0710 01:11:18.555080 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-z5pqd/must-gather-t52zq" Jul 10 01:11:18.724252 ip-10-0-130-146 kubenswrapper[2570]: I0710 01:11:18.724222 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-z5pqd/must-gather-t52zq"] Jul 10 01:11:18.725551 ip-10-0-130-146 kubenswrapper[2570]: W0710 01:11:18.725525 2570 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod921aec8a_7838_47b4_9e61_6b7f43f4cf54.slice/crio-f26fc5a7d9cec4f2dd9abed84256d8c1fde13d67cb85f7351233026aa8064d26 WatchSource:0}: Error finding container f26fc5a7d9cec4f2dd9abed84256d8c1fde13d67cb85f7351233026aa8064d26: Status 404 returned error can't find the container with id f26fc5a7d9cec4f2dd9abed84256d8c1fde13d67cb85f7351233026aa8064d26 Jul 10 01:11:19.049765 ip-10-0-130-146 kubenswrapper[2570]: I0710 01:11:19.049730 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-z5pqd/must-gather-t52zq" event={"ID":"921aec8a-7838-47b4-9e61-6b7f43f4cf54","Type":"ContainerStarted","Data":"f26fc5a7d9cec4f2dd9abed84256d8c1fde13d67cb85f7351233026aa8064d26"} Jul 10 01:11:20.059792 ip-10-0-130-146 kubenswrapper[2570]: I0710 01:11:20.059753 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-z5pqd/must-gather-t52zq" event={"ID":"921aec8a-7838-47b4-9e61-6b7f43f4cf54","Type":"ContainerStarted","Data":"57a349107785834c1f0f6f48f724746a3f83c4306e3498adb23fe3e50abbf8fc"} Jul 10 01:11:20.060216 ip-10-0-130-146 kubenswrapper[2570]: I0710 01:11:20.059804 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-z5pqd/must-gather-t52zq" event={"ID":"921aec8a-7838-47b4-9e61-6b7f43f4cf54","Type":"ContainerStarted","Data":"cc6b9df398760824900501c79ab16287379933ee61f52f7b29e83d77639b467f"} Jul 10 01:11:20.075666 ip-10-0-130-146 kubenswrapper[2570]: I0710 01:11:20.075602 2570 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-must-gather-z5pqd/must-gather-t52zq" podStartSLOduration=1.373844263 podStartE2EDuration="2.075581012s" totalImagesPullingTime="701.736749ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-10 01:11:18 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 01:11:20.073921107 +0000 UTC m=+2699.122299830" watchObservedRunningTime="2026-07-10 01:11:20.075581012 +0000 UTC m=+2699.123959734" Jul 10 01:11:20.910735 ip-10-0-130-146 kubenswrapper[2570]: I0710 01:11:20.910703 2570 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_global-pull-secret-syncer-6bqvb_5667efb4-6019-4bde-827a-0f1e68096b93/global-pull-secret-syncer/0.log" Jul 10 01:11:21.079943 ip-10-0-130-146 kubenswrapper[2570]: I0710 01:11:21.079909 2570 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_konnectivity-agent-995ch_5e419a12-f1a1-4b3d-8c72-7c6af574948f/konnectivity-agent/0.log" Jul 10 01:11:21.142329 ip-10-0-130-146 kubenswrapper[2570]: I0710 01:11:21.142286 2570 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kube-apiserver-proxy-ip-10-0-130-146.ec2.internal_277746f2e3b301c16ca7c3a5af84dd5d/haproxy/0.log" Jul 10 01:11:21.577903 ip-10-0-130-146 kubenswrapper[2570]: I0710 01:11:21.577873 2570 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7866497494-d6fcd_2b6cb4ca-79ff-42f0-b080-66af842f22d3/console-operator/2.log" Jul 10 01:11:21.581918 ip-10-0-130-146 kubenswrapper[2570]: I0710 01:11:21.581896 2570 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-mbs58_c4ef019a-364a-4933-ab85-de86d86167f1/ovn-acl-logging/0.log" Jul 10 01:11:21.582080 ip-10-0-130-146 kubenswrapper[2570]: I0710 01:11:21.581977 2570 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7866497494-d6fcd_2b6cb4ca-79ff-42f0-b080-66af842f22d3/console-operator/2.log" Jul 10 01:11:21.585909 ip-10-0-130-146 kubenswrapper[2570]: I0710 01:11:21.585891 2570 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-mbs58_c4ef019a-364a-4933-ab85-de86d86167f1/ovn-acl-logging/0.log" Jul 10 01:11:25.486382 ip-10-0-130-146 kubenswrapper[2570]: I0710 01:11:25.485858 2570 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759k68sb_a062b873-3bc8-4dc1-96e7-63353d46e0b5/extract/0.log" Jul 10 01:11:25.506482 ip-10-0-130-146 kubenswrapper[2570]: I0710 01:11:25.506454 2570 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759k68sb_a062b873-3bc8-4dc1-96e7-63353d46e0b5/util/0.log" Jul 10 01:11:25.531612 ip-10-0-130-146 kubenswrapper[2570]: I0710 01:11:25.531581 2570 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759k68sb_a062b873-3bc8-4dc1-96e7-63353d46e0b5/pull/0.log" Jul 10 01:11:25.564080 ip-10-0-130-146 kubenswrapper[2570]: I0710 01:11:25.564032 2570 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0hr2tc_b407e13a-88fb-43a1-94fc-ab40c580d6c2/extract/0.log" Jul 10 01:11:25.589606 ip-10-0-130-146 kubenswrapper[2570]: I0710 01:11:25.589572 2570 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0hr2tc_b407e13a-88fb-43a1-94fc-ab40c580d6c2/util/0.log" Jul 10 01:11:25.618095 ip-10-0-130-146 kubenswrapper[2570]: I0710 01:11:25.618039 2570 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0hr2tc_b407e13a-88fb-43a1-94fc-ab40c580d6c2/pull/0.log" Jul 10 01:11:25.650335 ip-10-0-130-146 kubenswrapper[2570]: I0710 01:11:25.650306 2570 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73rs9vd_48dc2b69-a1c1-46b4-9f79-02abaf29e411/extract/0.log" Jul 10 01:11:25.674963 ip-10-0-130-146 kubenswrapper[2570]: I0710 01:11:25.674933 2570 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73rs9vd_48dc2b69-a1c1-46b4-9f79-02abaf29e411/util/0.log" Jul 10 01:11:25.699236 ip-10-0-130-146 kubenswrapper[2570]: I0710 01:11:25.699187 2570 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73rs9vd_48dc2b69-a1c1-46b4-9f79-02abaf29e411/pull/0.log" Jul 10 01:11:25.726997 ip-10-0-130-146 kubenswrapper[2570]: I0710 01:11:25.726965 2570 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1pfl48_126a17da-6d3b-4880-b310-d0cf7612be39/extract/0.log" Jul 10 01:11:25.748170 ip-10-0-130-146 kubenswrapper[2570]: I0710 01:11:25.748081 2570 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1pfl48_126a17da-6d3b-4880-b310-d0cf7612be39/util/0.log" Jul 10 01:11:25.771129 ip-10-0-130-146 kubenswrapper[2570]: I0710 01:11:25.771098 2570 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1pfl48_126a17da-6d3b-4880-b310-d0cf7612be39/pull/0.log" Jul 10 01:11:26.142263 ip-10-0-130-146 kubenswrapper[2570]: I0710 01:11:26.142234 2570 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-console-plugin-66b5774985-v9f84_84dc7f0b-4d03-461a-a462-1147a97ac1ad/kuadrant-console-plugin/0.log" Jul 10 01:11:26.360600 ip-10-0-130-146 kubenswrapper[2570]: I0710 01:11:26.360574 2570 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-limitador-54d6d988d5-g92hs_7ef64d51-9c51-48f6-b409-e157f1ec45a5/limitador/0.log" Jul 10 01:11:28.186043 ip-10-0-130-146 kubenswrapper[2570]: I0710 01:11:28.186008 2570 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_kube-state-metrics-785778ff74-xxgzv_8d7a11c5-d953-4dd7-8d43-dd1e0e4582ff/kube-state-metrics/0.log" Jul 10 01:11:28.206112 ip-10-0-130-146 kubenswrapper[2570]: I0710 01:11:28.206084 2570 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_kube-state-metrics-785778ff74-xxgzv_8d7a11c5-d953-4dd7-8d43-dd1e0e4582ff/kube-rbac-proxy-main/0.log" Jul 10 01:11:28.227991 ip-10-0-130-146 kubenswrapper[2570]: I0710 01:11:28.227960 2570 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_kube-state-metrics-785778ff74-xxgzv_8d7a11c5-d953-4dd7-8d43-dd1e0e4582ff/kube-rbac-proxy-self/0.log" Jul 10 01:11:28.428038 ip-10-0-130-146 kubenswrapper[2570]: I0710 01:11:28.428003 2570 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-pfjbn_d2f81651-c77b-4ce3-b388-5753557860e3/node-exporter/0.log" Jul 10 01:11:28.458524 ip-10-0-130-146 kubenswrapper[2570]: I0710 01:11:28.458442 2570 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-pfjbn_d2f81651-c77b-4ce3-b388-5753557860e3/kube-rbac-proxy/0.log" Jul 10 01:11:28.481905 ip-10-0-130-146 kubenswrapper[2570]: I0710 01:11:28.481881 2570 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-pfjbn_d2f81651-c77b-4ce3-b388-5753557860e3/init-textfile/0.log" Jul 10 01:11:28.589459 ip-10-0-130-146 kubenswrapper[2570]: I0710 01:11:28.589432 2570 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_openshift-state-metrics-84bd68b896-wh4d2_2ffcf64d-9a08-46c6-9302-c20df48e965b/kube-rbac-proxy-main/0.log" Jul 10 01:11:28.610987 ip-10-0-130-146 kubenswrapper[2570]: I0710 01:11:28.610955 2570 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_openshift-state-metrics-84bd68b896-wh4d2_2ffcf64d-9a08-46c6-9302-c20df48e965b/kube-rbac-proxy-self/0.log" Jul 10 01:11:28.634501 ip-10-0-130-146 kubenswrapper[2570]: I0710 01:11:28.634470 2570 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_openshift-state-metrics-84bd68b896-wh4d2_2ffcf64d-9a08-46c6-9302-c20df48e965b/openshift-state-metrics/0.log" Jul 10 01:11:28.861707 ip-10-0-130-146 kubenswrapper[2570]: I0710 01:11:28.861625 2570 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-operator-64785b6675-499ct_15c29606-9069-42e0-8db8-2dae9df7f5a2/prometheus-operator/0.log" Jul 10 01:11:28.884652 ip-10-0-130-146 kubenswrapper[2570]: I0710 01:11:28.884599 2570 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-operator-64785b6675-499ct_15c29606-9069-42e0-8db8-2dae9df7f5a2/kube-rbac-proxy/0.log" Jul 10 01:11:28.910022 ip-10-0-130-146 kubenswrapper[2570]: I0710 01:11:28.909992 2570 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-operator-admission-webhook-77d98bc786-7br7w_be57dd6d-f83f-443f-9211-dc40cea396cc/prometheus-operator-admission-webhook/0.log" Jul 10 01:11:28.939321 ip-10-0-130-146 kubenswrapper[2570]: I0710 01:11:28.939290 2570 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_telemeter-client-68b5f47d8-swcn5_a60c4e5b-3332-4a4b-8d32-c94b0f823695/telemeter-client/0.log" Jul 10 01:11:28.963925 ip-10-0-130-146 kubenswrapper[2570]: I0710 01:11:28.963892 2570 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_telemeter-client-68b5f47d8-swcn5_a60c4e5b-3332-4a4b-8d32-c94b0f823695/reload/0.log" Jul 10 01:11:28.993462 ip-10-0-130-146 kubenswrapper[2570]: I0710 01:11:28.993431 2570 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_telemeter-client-68b5f47d8-swcn5_a60c4e5b-3332-4a4b-8d32-c94b0f823695/kube-rbac-proxy/0.log" Jul 10 01:11:30.428185 ip-10-0-130-146 kubenswrapper[2570]: I0710 01:11:30.428146 2570 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-network-console_networking-console-plugin-6cf856758c-45x56_d4b4eb0f-539c-4193-907e-cbff010aae7b/networking-console-plugin/0.log" Jul 10 01:11:31.039481 ip-10-0-130-146 kubenswrapper[2570]: I0710 01:11:31.039448 2570 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7866497494-d6fcd_2b6cb4ca-79ff-42f0-b080-66af842f22d3/console-operator/2.log" Jul 10 01:11:31.045409 ip-10-0-130-146 kubenswrapper[2570]: I0710 01:11:31.045379 2570 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7866497494-d6fcd_2b6cb4ca-79ff-42f0-b080-66af842f22d3/console-operator/3.log" Jul 10 01:11:31.295379 ip-10-0-130-146 kubenswrapper[2570]: I0710 01:11:31.295288 2570 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-must-gather-z5pqd/perf-node-gather-daemonset-667r8"] Jul 10 01:11:31.303144 ip-10-0-130-146 kubenswrapper[2570]: I0710 01:11:31.303111 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-z5pqd/perf-node-gather-daemonset-667r8" Jul 10 01:11:31.308552 ip-10-0-130-146 kubenswrapper[2570]: I0710 01:11:31.308523 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-z5pqd/perf-node-gather-daemonset-667r8"] Jul 10 01:11:31.372467 ip-10-0-130-146 kubenswrapper[2570]: I0710 01:11:31.372424 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/bf2a04f6-5ed8-4f01-8551-b8844300423e-sys\") pod \"perf-node-gather-daemonset-667r8\" (UID: \"bf2a04f6-5ed8-4f01-8551-b8844300423e\") " pod="openshift-must-gather-z5pqd/perf-node-gather-daemonset-667r8" Jul 10 01:11:31.372670 ip-10-0-130-146 kubenswrapper[2570]: I0710 01:11:31.372483 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8slzh\" (UniqueName: \"kubernetes.io/projected/bf2a04f6-5ed8-4f01-8551-b8844300423e-kube-api-access-8slzh\") pod \"perf-node-gather-daemonset-667r8\" (UID: \"bf2a04f6-5ed8-4f01-8551-b8844300423e\") " pod="openshift-must-gather-z5pqd/perf-node-gather-daemonset-667r8" Jul 10 01:11:31.372670 ip-10-0-130-146 kubenswrapper[2570]: I0710 01:11:31.372502 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/bf2a04f6-5ed8-4f01-8551-b8844300423e-podres\") pod \"perf-node-gather-daemonset-667r8\" (UID: \"bf2a04f6-5ed8-4f01-8551-b8844300423e\") " pod="openshift-must-gather-z5pqd/perf-node-gather-daemonset-667r8" Jul 10 01:11:31.372670 ip-10-0-130-146 kubenswrapper[2570]: I0710 01:11:31.372524 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/bf2a04f6-5ed8-4f01-8551-b8844300423e-proc\") pod \"perf-node-gather-daemonset-667r8\" (UID: \"bf2a04f6-5ed8-4f01-8551-b8844300423e\") " pod="openshift-must-gather-z5pqd/perf-node-gather-daemonset-667r8" Jul 10 01:11:31.372670 ip-10-0-130-146 kubenswrapper[2570]: I0710 01:11:31.372548 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/bf2a04f6-5ed8-4f01-8551-b8844300423e-lib-modules\") pod \"perf-node-gather-daemonset-667r8\" (UID: \"bf2a04f6-5ed8-4f01-8551-b8844300423e\") " pod="openshift-must-gather-z5pqd/perf-node-gather-daemonset-667r8" Jul 10 01:11:31.473519 ip-10-0-130-146 kubenswrapper[2570]: I0710 01:11:31.473484 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8slzh\" (UniqueName: \"kubernetes.io/projected/bf2a04f6-5ed8-4f01-8551-b8844300423e-kube-api-access-8slzh\") pod \"perf-node-gather-daemonset-667r8\" (UID: \"bf2a04f6-5ed8-4f01-8551-b8844300423e\") " pod="openshift-must-gather-z5pqd/perf-node-gather-daemonset-667r8" Jul 10 01:11:31.474020 ip-10-0-130-146 kubenswrapper[2570]: I0710 01:11:31.473529 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/bf2a04f6-5ed8-4f01-8551-b8844300423e-podres\") pod \"perf-node-gather-daemonset-667r8\" (UID: \"bf2a04f6-5ed8-4f01-8551-b8844300423e\") " pod="openshift-must-gather-z5pqd/perf-node-gather-daemonset-667r8" Jul 10 01:11:31.474020 ip-10-0-130-146 kubenswrapper[2570]: I0710 01:11:31.473566 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/bf2a04f6-5ed8-4f01-8551-b8844300423e-proc\") pod \"perf-node-gather-daemonset-667r8\" (UID: \"bf2a04f6-5ed8-4f01-8551-b8844300423e\") " pod="openshift-must-gather-z5pqd/perf-node-gather-daemonset-667r8" Jul 10 01:11:31.474020 ip-10-0-130-146 kubenswrapper[2570]: I0710 01:11:31.473604 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/bf2a04f6-5ed8-4f01-8551-b8844300423e-lib-modules\") pod \"perf-node-gather-daemonset-667r8\" (UID: \"bf2a04f6-5ed8-4f01-8551-b8844300423e\") " pod="openshift-must-gather-z5pqd/perf-node-gather-daemonset-667r8" Jul 10 01:11:31.474020 ip-10-0-130-146 kubenswrapper[2570]: I0710 01:11:31.473686 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/bf2a04f6-5ed8-4f01-8551-b8844300423e-sys\") pod \"perf-node-gather-daemonset-667r8\" (UID: \"bf2a04f6-5ed8-4f01-8551-b8844300423e\") " pod="openshift-must-gather-z5pqd/perf-node-gather-daemonset-667r8" Jul 10 01:11:31.474020 ip-10-0-130-146 kubenswrapper[2570]: I0710 01:11:31.473779 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/bf2a04f6-5ed8-4f01-8551-b8844300423e-sys\") pod \"perf-node-gather-daemonset-667r8\" (UID: \"bf2a04f6-5ed8-4f01-8551-b8844300423e\") " pod="openshift-must-gather-z5pqd/perf-node-gather-daemonset-667r8" Jul 10 01:11:31.474372 ip-10-0-130-146 kubenswrapper[2570]: I0710 01:11:31.474177 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/bf2a04f6-5ed8-4f01-8551-b8844300423e-podres\") pod \"perf-node-gather-daemonset-667r8\" (UID: \"bf2a04f6-5ed8-4f01-8551-b8844300423e\") " pod="openshift-must-gather-z5pqd/perf-node-gather-daemonset-667r8" Jul 10 01:11:31.474372 ip-10-0-130-146 kubenswrapper[2570]: I0710 01:11:31.474231 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/bf2a04f6-5ed8-4f01-8551-b8844300423e-proc\") pod \"perf-node-gather-daemonset-667r8\" (UID: \"bf2a04f6-5ed8-4f01-8551-b8844300423e\") " pod="openshift-must-gather-z5pqd/perf-node-gather-daemonset-667r8" Jul 10 01:11:31.474372 ip-10-0-130-146 kubenswrapper[2570]: I0710 01:11:31.474311 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/bf2a04f6-5ed8-4f01-8551-b8844300423e-lib-modules\") pod \"perf-node-gather-daemonset-667r8\" (UID: \"bf2a04f6-5ed8-4f01-8551-b8844300423e\") " pod="openshift-must-gather-z5pqd/perf-node-gather-daemonset-667r8" Jul 10 01:11:31.481814 ip-10-0-130-146 kubenswrapper[2570]: I0710 01:11:31.481787 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8slzh\" (UniqueName: \"kubernetes.io/projected/bf2a04f6-5ed8-4f01-8551-b8844300423e-kube-api-access-8slzh\") pod \"perf-node-gather-daemonset-667r8\" (UID: \"bf2a04f6-5ed8-4f01-8551-b8844300423e\") " pod="openshift-must-gather-z5pqd/perf-node-gather-daemonset-667r8" Jul 10 01:11:31.610700 ip-10-0-130-146 kubenswrapper[2570]: I0710 01:11:31.610618 2570 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console_console-6b8698786d-qv9lq_1ecefbf9-8ebe-4bc6-9ceb-385d5d5b0d06/console/0.log" Jul 10 01:11:31.617890 ip-10-0-130-146 kubenswrapper[2570]: I0710 01:11:31.617865 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-z5pqd/perf-node-gather-daemonset-667r8" Jul 10 01:11:31.651892 ip-10-0-130-146 kubenswrapper[2570]: I0710 01:11:31.651865 2570 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console_downloads-68c9cbb497-4prsj_e05c8204-bfbd-4bce-8cda-9a56ef492fcb/download-server/0.log" Jul 10 01:11:31.801685 ip-10-0-130-146 kubenswrapper[2570]: I0710 01:11:31.801647 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-z5pqd/perf-node-gather-daemonset-667r8"] Jul 10 01:11:31.804656 ip-10-0-130-146 kubenswrapper[2570]: W0710 01:11:31.804628 2570 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-podbf2a04f6_5ed8_4f01_8551_b8844300423e.slice/crio-ca81cbfba0de11629432453ffe6007f9e1ba5b0cb71bc3c6f7d9af5764de73c3 WatchSource:0}: Error finding container ca81cbfba0de11629432453ffe6007f9e1ba5b0cb71bc3c6f7d9af5764de73c3: Status 404 returned error can't find the container with id ca81cbfba0de11629432453ffe6007f9e1ba5b0cb71bc3c6f7d9af5764de73c3 Jul 10 01:11:32.173537 ip-10-0-130-146 kubenswrapper[2570]: I0710 01:11:32.173497 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-z5pqd/perf-node-gather-daemonset-667r8" event={"ID":"bf2a04f6-5ed8-4f01-8551-b8844300423e","Type":"ContainerStarted","Data":"66992c84bc0114ac4a7afc227fdb8e6e662b347205258b48bb80eaf8d219e8f4"} Jul 10 01:11:32.173718 ip-10-0-130-146 kubenswrapper[2570]: I0710 01:11:32.173564 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-z5pqd/perf-node-gather-daemonset-667r8" event={"ID":"bf2a04f6-5ed8-4f01-8551-b8844300423e","Type":"ContainerStarted","Data":"ca81cbfba0de11629432453ffe6007f9e1ba5b0cb71bc3c6f7d9af5764de73c3"} Jul 10 01:11:32.173718 ip-10-0-130-146 kubenswrapper[2570]: I0710 01:11:32.173597 2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-must-gather-z5pqd/perf-node-gather-daemonset-667r8" Jul 10 01:11:32.190941 ip-10-0-130-146 kubenswrapper[2570]: I0710 01:11:32.190885 2570 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-must-gather-z5pqd/perf-node-gather-daemonset-667r8" podStartSLOduration=1.1908662030000001 podStartE2EDuration="1.190866203s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-10 01:11:31 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 01:11:32.187292758 +0000 UTC m=+2711.235671504" watchObservedRunningTime="2026-07-10 01:11:32.190866203 +0000 UTC m=+2711.239244923" Jul 10 01:11:33.305541 ip-10-0-130-146 kubenswrapper[2570]: I0710 01:11:33.305508 2570 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-fcndn_e2252bd4-69bf-4c02-a315-d9cbcf4ed91e/dns/0.log" Jul 10 01:11:33.328074 ip-10-0-130-146 kubenswrapper[2570]: I0710 01:11:33.328036 2570 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-fcndn_e2252bd4-69bf-4c02-a315-d9cbcf4ed91e/kube-rbac-proxy/0.log" Jul 10 01:11:33.422140 ip-10-0-130-146 kubenswrapper[2570]: I0710 01:11:33.422110 2570 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-7tn6x_006ba44e-264c-42cd-9891-ad8d769386c8/dns-node-resolver/0.log" Jul 10 01:11:34.013291 ip-10-0-130-146 kubenswrapper[2570]: I0710 01:11:34.013249 2570 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-t526b_52179d08-5861-4a23-9c8d-653c13bed339/node-ca/0.log" Jul 10 01:11:35.221353 ip-10-0-130-146 kubenswrapper[2570]: I0710 01:11:35.221279 2570 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-7cb6b7dbc6-wwmdk_49d9ee95-a4bb-4549-b5dc-92c63e33c475/kube-auth-proxy/0.log" Jul 10 01:11:36.083951 ip-10-0-130-146 kubenswrapper[2570]: I0710 01:11:36.083918 2570 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-ck6gk_3ec525d5-c820-42fe-a411-99faa6ef373d/serve-healthcheck-canary/0.log" Jul 10 01:11:36.599766 ip-10-0-130-146 kubenswrapper[2570]: I0710 01:11:36.599734 2570 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-operator-68d9dd94cf-7rq8g_f612c430-f495-4773-9ca6-82f7417e3a0c/insights-operator/0.log" Jul 10 01:11:36.602341 ip-10-0-130-146 kubenswrapper[2570]: I0710 01:11:36.602317 2570 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-operator-68d9dd94cf-7rq8g_f612c430-f495-4773-9ca6-82f7417e3a0c/insights-operator/1.log" Jul 10 01:11:36.783284 ip-10-0-130-146 kubenswrapper[2570]: I0710 01:11:36.783247 2570 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-sp47j_95631388-25a0-4948-a49d-2950902d0998/kube-rbac-proxy/0.log" Jul 10 01:11:36.804241 ip-10-0-130-146 kubenswrapper[2570]: I0710 01:11:36.804213 2570 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-sp47j_95631388-25a0-4948-a49d-2950902d0998/exporter/0.log" Jul 10 01:11:36.826917 ip-10-0-130-146 kubenswrapper[2570]: I0710 01:11:36.826888 2570 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-sp47j_95631388-25a0-4948-a49d-2950902d0998/extractor/0.log" Jul 10 01:11:38.193091 ip-10-0-130-146 kubenswrapper[2570]: I0710 01:11:38.193063 2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-must-gather-z5pqd/perf-node-gather-daemonset-667r8" Jul 10 01:11:39.374952 ip-10-0-130-146 kubenswrapper[2570]: I0710 01:11:39.374921 2570 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-54ccd4f945-f4q9r_b62d531b-b70e-48c2-a443-8bb683e26da2/manager/0.log" Jul 10 01:11:39.395718 ip-10-0-130-146 kubenswrapper[2570]: I0710 01:11:39.395692 2570 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-54ccd4f945-f4q9r_b62d531b-b70e-48c2-a443-8bb683e26da2/copy-manifests/0.log" Jul 10 01:11:40.867590 ip-10-0-130-146 kubenswrapper[2570]: I0710 01:11:40.867539 2570 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-lws-operator_lws-controller-manager-54495c74ff-9ls6p_fafa3ef8-223e-4842-9bb6-cc5f7743b430/manager/0.log" Jul 10 01:11:40.893560 ip-10-0-130-146 kubenswrapper[2570]: I0710 01:11:40.893528 2570 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-lws-operator_openshift-lws-operator-fd8ccff4c-4d5fm_31700c7c-d83c-4d84-a9be-07bfecdf60f6/openshift-lws-operator/0.log" Jul 10 01:11:41.854894 ip-10-0-130-146 kubenswrapper[2570]: I0710 01:11:41.854855 2570 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-7cb6b7dbc6-wwmdk_49d9ee95-a4bb-4549-b5dc-92c63e33c475/kube-auth-proxy/0.log" Jul 10 01:11:43.415549 ip-10-0-130-146 kubenswrapper[2570]: I0710 01:11:43.415478 2570 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-7cb6b7dbc6-wwmdk_49d9ee95-a4bb-4549-b5dc-92c63e33c475/kube-auth-proxy/0.log" Jul 10 01:11:46.253879 ip-10-0-130-146 kubenswrapper[2570]: I0710 01:11:46.253844 2570 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator-operator_kube-storage-version-migrator-operator-799cb7f958-p44ms_6f87cbd8-9d72-4708-a230-2818fa5a836d/kube-storage-version-migrator-operator/1.log" Jul 10 01:11:46.255363 ip-10-0-130-146 kubenswrapper[2570]: I0710 01:11:46.255341 2570 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator-operator_kube-storage-version-migrator-operator-799cb7f958-p44ms_6f87cbd8-9d72-4708-a230-2818fa5a836d/kube-storage-version-migrator-operator/0.log" Jul 10 01:11:47.349170 ip-10-0-130-146 kubenswrapper[2570]: I0710 01:11:47.349141 2570 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-2pdkh_f726a6ab-3c92-403e-ad1b-bdaf7e37c5e3/kube-multus/0.log" Jul 10 01:11:47.377926 ip-10-0-130-146 kubenswrapper[2570]: I0710 01:11:47.377903 2570 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-7qfbg_78f9f66c-5c2f-4cee-bf34-39961a78dbcc/kube-multus-additional-cni-plugins/0.log" Jul 10 01:11:47.402592 ip-10-0-130-146 kubenswrapper[2570]: I0710 01:11:47.402567 2570 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-7qfbg_78f9f66c-5c2f-4cee-bf34-39961a78dbcc/egress-router-binary-copy/0.log" Jul 10 01:11:47.426524 ip-10-0-130-146 kubenswrapper[2570]: I0710 01:11:47.426488 2570 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-7qfbg_78f9f66c-5c2f-4cee-bf34-39961a78dbcc/cni-plugins/0.log" Jul 10 01:11:47.448424 ip-10-0-130-146 kubenswrapper[2570]: I0710 01:11:47.448396 2570 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-7qfbg_78f9f66c-5c2f-4cee-bf34-39961a78dbcc/bond-cni-plugin/0.log" Jul 10 01:11:47.468795 ip-10-0-130-146 kubenswrapper[2570]: I0710 01:11:47.468775 2570 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-7qfbg_78f9f66c-5c2f-4cee-bf34-39961a78dbcc/routeoverride-cni/0.log" Jul 10 01:11:47.490299 ip-10-0-130-146 kubenswrapper[2570]: I0710 01:11:47.490277 2570 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-7qfbg_78f9f66c-5c2f-4cee-bf34-39961a78dbcc/whereabouts-cni-bincopy/0.log" Jul 10 01:11:47.511015 ip-10-0-130-146 kubenswrapper[2570]: I0710 01:11:47.510995 2570 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-7qfbg_78f9f66c-5c2f-4cee-bf34-39961a78dbcc/whereabouts-cni/0.log" Jul 10 01:11:47.989179 ip-10-0-130-146 kubenswrapper[2570]: I0710 01:11:47.989152 2570 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_network-metrics-daemon-d8t56_b7aab806-00a2-491e-a997-ed3174bd547a/network-metrics-daemon/0.log" Jul 10 01:11:48.007920 ip-10-0-130-146 kubenswrapper[2570]: I0710 01:11:48.007892 2570 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_network-metrics-daemon-d8t56_b7aab806-00a2-491e-a997-ed3174bd547a/kube-rbac-proxy/0.log"