Jul 06 13:48:40.359592 ip-10-0-136-25 systemd[1]: kubelet.service: Failed to load environment files: No such file or directory Jul 06 13:48:40.359604 ip-10-0-136-25 systemd[1]: kubelet.service: Failed to run 'start-pre' task: No such file or directory Jul 06 13:48:40.359612 ip-10-0-136-25 systemd[1]: kubelet.service: Failed with result 'resources'. Jul 06 13:48:40.360008 ip-10-0-136-25 systemd[1]: Failed to start Kubernetes Kubelet. Jul 06 13:48:50.581132 ip-10-0-136-25 systemd[1]: kubelet.service: Failed to schedule restart job: Unit crio.service not found. Jul 06 13:48:50.581159 ip-10-0-136-25 systemd[1]: kubelet.service: Failed with result 'resources'. -- Boot 3f7012032eb8488f8be5e7d93553c040 -- Jul 06 13:50:22.655415 ip-10-0-136-25 systemd[1]: Starting Kubernetes Kubelet... Jul 06 13:50:23.056122 ip-10-0-136-25 kubenswrapper[2569]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Jul 06 13:50:23.056122 ip-10-0-136-25 kubenswrapper[2569]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version. Jul 06 13:50:23.056122 ip-10-0-136-25 kubenswrapper[2569]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Jul 06 13:50:23.056122 ip-10-0-136-25 kubenswrapper[2569]: Flag --pod-infra-container-image has been deprecated, will be removed in 1.35. Image garbage collector will get sandbox image information from CRI. Jul 06 13:50:23.056986 ip-10-0-136-25 kubenswrapper[2569]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Jul 06 13:50:23.057154 ip-10-0-136-25 kubenswrapper[2569]: I0706 13:50:23.057025 2569 server.go:213] "--pod-infra-container-image will not be pruned by the image garbage collector in kubelet and should also be set in the remote runtime" Jul 06 13:50:23.061533 ip-10-0-136-25 kubenswrapper[2569]: W0706 13:50:23.061516 2569 feature_gate.go:328] unrecognized feature gate: Example Jul 06 13:50:23.061533 ip-10-0-136-25 kubenswrapper[2569]: W0706 13:50:23.061532 2569 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 06 13:50:23.061533 ip-10-0-136-25 kubenswrapper[2569]: W0706 13:50:23.061535 2569 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 06 13:50:23.061656 ip-10-0-136-25 kubenswrapper[2569]: W0706 13:50:23.061539 2569 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 06 13:50:23.061656 ip-10-0-136-25 kubenswrapper[2569]: W0706 13:50:23.061544 2569 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 06 13:50:23.061656 ip-10-0-136-25 kubenswrapper[2569]: W0706 13:50:23.061548 2569 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 06 13:50:23.061656 ip-10-0-136-25 kubenswrapper[2569]: W0706 13:50:23.061551 2569 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 06 13:50:23.061656 ip-10-0-136-25 kubenswrapper[2569]: W0706 13:50:23.061554 2569 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 06 13:50:23.061656 ip-10-0-136-25 kubenswrapper[2569]: W0706 13:50:23.061557 2569 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 06 13:50:23.061656 ip-10-0-136-25 kubenswrapper[2569]: W0706 13:50:23.061560 2569 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 06 13:50:23.061656 ip-10-0-136-25 kubenswrapper[2569]: W0706 13:50:23.061563 2569 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 06 13:50:23.061656 ip-10-0-136-25 kubenswrapper[2569]: W0706 13:50:23.061566 2569 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 06 13:50:23.061656 ip-10-0-136-25 kubenswrapper[2569]: W0706 13:50:23.061586 2569 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 06 13:50:23.061656 ip-10-0-136-25 kubenswrapper[2569]: W0706 13:50:23.061589 2569 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 06 13:50:23.061656 ip-10-0-136-25 kubenswrapper[2569]: W0706 13:50:23.061592 2569 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 06 13:50:23.061656 ip-10-0-136-25 kubenswrapper[2569]: W0706 13:50:23.061595 2569 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 06 13:50:23.061656 ip-10-0-136-25 kubenswrapper[2569]: W0706 13:50:23.061598 2569 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 06 13:50:23.061656 ip-10-0-136-25 kubenswrapper[2569]: W0706 13:50:23.061601 2569 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 06 13:50:23.061656 ip-10-0-136-25 kubenswrapper[2569]: W0706 13:50:23.061603 2569 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 06 13:50:23.061656 ip-10-0-136-25 kubenswrapper[2569]: W0706 13:50:23.061606 2569 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 06 13:50:23.061656 ip-10-0-136-25 kubenswrapper[2569]: W0706 13:50:23.061608 2569 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 06 13:50:23.061656 ip-10-0-136-25 kubenswrapper[2569]: W0706 13:50:23.061612 2569 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 06 13:50:23.062347 ip-10-0-136-25 kubenswrapper[2569]: W0706 13:50:23.061615 2569 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 06 13:50:23.062347 ip-10-0-136-25 kubenswrapper[2569]: W0706 13:50:23.061617 2569 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 06 13:50:23.062347 ip-10-0-136-25 kubenswrapper[2569]: W0706 13:50:23.061619 2569 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 06 13:50:23.062347 ip-10-0-136-25 kubenswrapper[2569]: W0706 13:50:23.061622 2569 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 06 13:50:23.062347 ip-10-0-136-25 kubenswrapper[2569]: W0706 13:50:23.061625 2569 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 06 13:50:23.062347 ip-10-0-136-25 kubenswrapper[2569]: W0706 13:50:23.061627 2569 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 06 13:50:23.062347 ip-10-0-136-25 kubenswrapper[2569]: W0706 13:50:23.061630 2569 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 06 13:50:23.062347 ip-10-0-136-25 kubenswrapper[2569]: W0706 13:50:23.061632 2569 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 06 13:50:23.062347 ip-10-0-136-25 kubenswrapper[2569]: W0706 13:50:23.061635 2569 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 06 13:50:23.062347 ip-10-0-136-25 kubenswrapper[2569]: W0706 13:50:23.061637 2569 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 06 13:50:23.062347 ip-10-0-136-25 kubenswrapper[2569]: W0706 13:50:23.061640 2569 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 06 13:50:23.062347 ip-10-0-136-25 kubenswrapper[2569]: W0706 13:50:23.061642 2569 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 06 13:50:23.062347 ip-10-0-136-25 kubenswrapper[2569]: W0706 13:50:23.061645 2569 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 06 13:50:23.062347 ip-10-0-136-25 kubenswrapper[2569]: W0706 13:50:23.061647 2569 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 06 13:50:23.062347 ip-10-0-136-25 kubenswrapper[2569]: W0706 13:50:23.061650 2569 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 06 13:50:23.062347 ip-10-0-136-25 kubenswrapper[2569]: W0706 13:50:23.061653 2569 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 06 13:50:23.062347 ip-10-0-136-25 kubenswrapper[2569]: W0706 13:50:23.061656 2569 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 06 13:50:23.062347 ip-10-0-136-25 kubenswrapper[2569]: W0706 13:50:23.061659 2569 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 06 13:50:23.062347 ip-10-0-136-25 kubenswrapper[2569]: W0706 13:50:23.061662 2569 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 06 13:50:23.062347 ip-10-0-136-25 kubenswrapper[2569]: W0706 13:50:23.061665 2569 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 06 13:50:23.063100 ip-10-0-136-25 kubenswrapper[2569]: W0706 13:50:23.061668 2569 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 06 13:50:23.063100 ip-10-0-136-25 kubenswrapper[2569]: W0706 13:50:23.061670 2569 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 06 13:50:23.063100 ip-10-0-136-25 kubenswrapper[2569]: W0706 13:50:23.061673 2569 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 06 13:50:23.063100 ip-10-0-136-25 kubenswrapper[2569]: W0706 13:50:23.061677 2569 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 06 13:50:23.063100 ip-10-0-136-25 kubenswrapper[2569]: W0706 13:50:23.061679 2569 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 06 13:50:23.063100 ip-10-0-136-25 kubenswrapper[2569]: W0706 13:50:23.061682 2569 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 06 13:50:23.063100 ip-10-0-136-25 kubenswrapper[2569]: W0706 13:50:23.061685 2569 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 06 13:50:23.063100 ip-10-0-136-25 kubenswrapper[2569]: W0706 13:50:23.061688 2569 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 06 13:50:23.063100 ip-10-0-136-25 kubenswrapper[2569]: W0706 13:50:23.061691 2569 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 06 13:50:23.063100 ip-10-0-136-25 kubenswrapper[2569]: W0706 13:50:23.061693 2569 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 06 13:50:23.063100 ip-10-0-136-25 kubenswrapper[2569]: W0706 13:50:23.061696 2569 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 06 13:50:23.063100 ip-10-0-136-25 kubenswrapper[2569]: W0706 13:50:23.061698 2569 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 06 13:50:23.063100 ip-10-0-136-25 kubenswrapper[2569]: W0706 13:50:23.061701 2569 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 06 13:50:23.063100 ip-10-0-136-25 kubenswrapper[2569]: W0706 13:50:23.061703 2569 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 06 13:50:23.063100 ip-10-0-136-25 kubenswrapper[2569]: W0706 13:50:23.061706 2569 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 06 13:50:23.063100 ip-10-0-136-25 kubenswrapper[2569]: W0706 13:50:23.061708 2569 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 06 13:50:23.063100 ip-10-0-136-25 kubenswrapper[2569]: W0706 13:50:23.061711 2569 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 06 13:50:23.063100 ip-10-0-136-25 kubenswrapper[2569]: W0706 13:50:23.061713 2569 feature_gate.go:328] unrecognized feature gate: Example2 Jul 06 13:50:23.063100 ip-10-0-136-25 kubenswrapper[2569]: W0706 13:50:23.061716 2569 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 06 13:50:23.063100 ip-10-0-136-25 kubenswrapper[2569]: W0706 13:50:23.061718 2569 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 06 13:50:23.063812 ip-10-0-136-25 kubenswrapper[2569]: W0706 13:50:23.061721 2569 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 06 13:50:23.063812 ip-10-0-136-25 kubenswrapper[2569]: W0706 13:50:23.061723 2569 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 06 13:50:23.063812 ip-10-0-136-25 kubenswrapper[2569]: W0706 13:50:23.061726 2569 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 06 13:50:23.063812 ip-10-0-136-25 kubenswrapper[2569]: W0706 13:50:23.061728 2569 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 06 13:50:23.063812 ip-10-0-136-25 kubenswrapper[2569]: W0706 13:50:23.061732 2569 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 06 13:50:23.063812 ip-10-0-136-25 kubenswrapper[2569]: W0706 13:50:23.061734 2569 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 06 13:50:23.063812 ip-10-0-136-25 kubenswrapper[2569]: W0706 13:50:23.061737 2569 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 06 13:50:23.063812 ip-10-0-136-25 kubenswrapper[2569]: W0706 13:50:23.061740 2569 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 06 13:50:23.063812 ip-10-0-136-25 kubenswrapper[2569]: W0706 13:50:23.061743 2569 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 06 13:50:23.063812 ip-10-0-136-25 kubenswrapper[2569]: W0706 13:50:23.061746 2569 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 06 13:50:23.063812 ip-10-0-136-25 kubenswrapper[2569]: W0706 13:50:23.061748 2569 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 06 13:50:23.063812 ip-10-0-136-25 kubenswrapper[2569]: W0706 13:50:23.061751 2569 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 06 13:50:23.063812 ip-10-0-136-25 kubenswrapper[2569]: W0706 13:50:23.061753 2569 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 06 13:50:23.063812 ip-10-0-136-25 kubenswrapper[2569]: W0706 13:50:23.061758 2569 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 06 13:50:23.063812 ip-10-0-136-25 kubenswrapper[2569]: W0706 13:50:23.061761 2569 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 06 13:50:23.063812 ip-10-0-136-25 kubenswrapper[2569]: W0706 13:50:23.061764 2569 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 06 13:50:23.063812 ip-10-0-136-25 kubenswrapper[2569]: W0706 13:50:23.061767 2569 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 06 13:50:23.063812 ip-10-0-136-25 kubenswrapper[2569]: W0706 13:50:23.061769 2569 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 06 13:50:23.063812 ip-10-0-136-25 kubenswrapper[2569]: W0706 13:50:23.061772 2569 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 06 13:50:23.064494 ip-10-0-136-25 kubenswrapper[2569]: W0706 13:50:23.061774 2569 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 06 13:50:23.064494 ip-10-0-136-25 kubenswrapper[2569]: W0706 13:50:23.061777 2569 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 06 13:50:23.064494 ip-10-0-136-25 kubenswrapper[2569]: W0706 13:50:23.061779 2569 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 06 13:50:23.064494 ip-10-0-136-25 kubenswrapper[2569]: W0706 13:50:23.061782 2569 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 06 13:50:23.064494 ip-10-0-136-25 kubenswrapper[2569]: W0706 13:50:23.061784 2569 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 06 13:50:23.064494 ip-10-0-136-25 kubenswrapper[2569]: W0706 13:50:23.061786 2569 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 06 13:50:23.064494 ip-10-0-136-25 kubenswrapper[2569]: W0706 13:50:23.061789 2569 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 06 13:50:23.064494 ip-10-0-136-25 kubenswrapper[2569]: W0706 13:50:23.061804 2569 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 06 13:50:23.064494 ip-10-0-136-25 kubenswrapper[2569]: W0706 13:50:23.061808 2569 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 06 13:50:23.064494 ip-10-0-136-25 kubenswrapper[2569]: W0706 13:50:23.061811 2569 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 06 13:50:23.064494 ip-10-0-136-25 kubenswrapper[2569]: W0706 13:50:23.061814 2569 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 06 13:50:23.064494 ip-10-0-136-25 kubenswrapper[2569]: W0706 13:50:23.061816 2569 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 06 13:50:23.064494 ip-10-0-136-25 kubenswrapper[2569]: W0706 13:50:23.061819 2569 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 06 13:50:23.064494 ip-10-0-136-25 kubenswrapper[2569]: W0706 13:50:23.061822 2569 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 06 13:50:23.064494 ip-10-0-136-25 kubenswrapper[2569]: W0706 13:50:23.061825 2569 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 06 13:50:23.064494 ip-10-0-136-25 kubenswrapper[2569]: W0706 13:50:23.061828 2569 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 06 13:50:23.064494 ip-10-0-136-25 kubenswrapper[2569]: W0706 13:50:23.061831 2569 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 06 13:50:23.064494 ip-10-0-136-25 kubenswrapper[2569]: W0706 13:50:23.061834 2569 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 06 13:50:23.064494 ip-10-0-136-25 kubenswrapper[2569]: W0706 13:50:23.062563 2569 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 06 13:50:23.065230 ip-10-0-136-25 kubenswrapper[2569]: W0706 13:50:23.062570 2569 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 06 13:50:23.065230 ip-10-0-136-25 kubenswrapper[2569]: W0706 13:50:23.062573 2569 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 06 13:50:23.065230 ip-10-0-136-25 kubenswrapper[2569]: W0706 13:50:23.062577 2569 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 06 13:50:23.065230 ip-10-0-136-25 kubenswrapper[2569]: W0706 13:50:23.062580 2569 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 06 13:50:23.065230 ip-10-0-136-25 kubenswrapper[2569]: W0706 13:50:23.062583 2569 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 06 13:50:23.065230 ip-10-0-136-25 kubenswrapper[2569]: W0706 13:50:23.062585 2569 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 06 13:50:23.065230 ip-10-0-136-25 kubenswrapper[2569]: W0706 13:50:23.062588 2569 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 06 13:50:23.065230 ip-10-0-136-25 kubenswrapper[2569]: W0706 13:50:23.062591 2569 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 06 13:50:23.065230 ip-10-0-136-25 kubenswrapper[2569]: W0706 13:50:23.062593 2569 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 06 13:50:23.065230 ip-10-0-136-25 kubenswrapper[2569]: W0706 13:50:23.062596 2569 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 06 13:50:23.065230 ip-10-0-136-25 kubenswrapper[2569]: W0706 13:50:23.062599 2569 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 06 13:50:23.065230 ip-10-0-136-25 kubenswrapper[2569]: W0706 13:50:23.062602 2569 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 06 13:50:23.065230 ip-10-0-136-25 kubenswrapper[2569]: W0706 13:50:23.062605 2569 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 06 13:50:23.065230 ip-10-0-136-25 kubenswrapper[2569]: W0706 13:50:23.062608 2569 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 06 13:50:23.065230 ip-10-0-136-25 kubenswrapper[2569]: W0706 13:50:23.062610 2569 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 06 13:50:23.065230 ip-10-0-136-25 kubenswrapper[2569]: W0706 13:50:23.062613 2569 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 06 13:50:23.065230 ip-10-0-136-25 kubenswrapper[2569]: W0706 13:50:23.062616 2569 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 06 13:50:23.065230 ip-10-0-136-25 kubenswrapper[2569]: W0706 13:50:23.062619 2569 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 06 13:50:23.065230 ip-10-0-136-25 kubenswrapper[2569]: W0706 13:50:23.062621 2569 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 06 13:50:23.066098 ip-10-0-136-25 kubenswrapper[2569]: W0706 13:50:23.062624 2569 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 06 13:50:23.066098 ip-10-0-136-25 kubenswrapper[2569]: W0706 13:50:23.062626 2569 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 06 13:50:23.066098 ip-10-0-136-25 kubenswrapper[2569]: W0706 13:50:23.062629 2569 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 06 13:50:23.066098 ip-10-0-136-25 kubenswrapper[2569]: W0706 13:50:23.062633 2569 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 06 13:50:23.066098 ip-10-0-136-25 kubenswrapper[2569]: W0706 13:50:23.062637 2569 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 06 13:50:23.066098 ip-10-0-136-25 kubenswrapper[2569]: W0706 13:50:23.062639 2569 feature_gate.go:328] unrecognized feature gate: Example2 Jul 06 13:50:23.066098 ip-10-0-136-25 kubenswrapper[2569]: W0706 13:50:23.062642 2569 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 06 13:50:23.066098 ip-10-0-136-25 kubenswrapper[2569]: W0706 13:50:23.062645 2569 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 06 13:50:23.066098 ip-10-0-136-25 kubenswrapper[2569]: W0706 13:50:23.062648 2569 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 06 13:50:23.066098 ip-10-0-136-25 kubenswrapper[2569]: W0706 13:50:23.062650 2569 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 06 13:50:23.066098 ip-10-0-136-25 kubenswrapper[2569]: W0706 13:50:23.062653 2569 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 06 13:50:23.066098 ip-10-0-136-25 kubenswrapper[2569]: W0706 13:50:23.062656 2569 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 06 13:50:23.066098 ip-10-0-136-25 kubenswrapper[2569]: W0706 13:50:23.062658 2569 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 06 13:50:23.066098 ip-10-0-136-25 kubenswrapper[2569]: W0706 13:50:23.062661 2569 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 06 13:50:23.066098 ip-10-0-136-25 kubenswrapper[2569]: W0706 13:50:23.062663 2569 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 06 13:50:23.066098 ip-10-0-136-25 kubenswrapper[2569]: W0706 13:50:23.062666 2569 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 06 13:50:23.066098 ip-10-0-136-25 kubenswrapper[2569]: W0706 13:50:23.062668 2569 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 06 13:50:23.066098 ip-10-0-136-25 kubenswrapper[2569]: W0706 13:50:23.062671 2569 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 06 13:50:23.066098 ip-10-0-136-25 kubenswrapper[2569]: W0706 13:50:23.062673 2569 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 06 13:50:23.066790 ip-10-0-136-25 kubenswrapper[2569]: W0706 13:50:23.062675 2569 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 06 13:50:23.066790 ip-10-0-136-25 kubenswrapper[2569]: W0706 13:50:23.062678 2569 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 06 13:50:23.066790 ip-10-0-136-25 kubenswrapper[2569]: W0706 13:50:23.062680 2569 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 06 13:50:23.066790 ip-10-0-136-25 kubenswrapper[2569]: W0706 13:50:23.062683 2569 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 06 13:50:23.066790 ip-10-0-136-25 kubenswrapper[2569]: W0706 13:50:23.062685 2569 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 06 13:50:23.066790 ip-10-0-136-25 kubenswrapper[2569]: W0706 13:50:23.062688 2569 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 06 13:50:23.066790 ip-10-0-136-25 kubenswrapper[2569]: W0706 13:50:23.062691 2569 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 06 13:50:23.066790 ip-10-0-136-25 kubenswrapper[2569]: W0706 13:50:23.062694 2569 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 06 13:50:23.066790 ip-10-0-136-25 kubenswrapper[2569]: W0706 13:50:23.062696 2569 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 06 13:50:23.066790 ip-10-0-136-25 kubenswrapper[2569]: W0706 13:50:23.062699 2569 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 06 13:50:23.066790 ip-10-0-136-25 kubenswrapper[2569]: W0706 13:50:23.062702 2569 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 06 13:50:23.066790 ip-10-0-136-25 kubenswrapper[2569]: W0706 13:50:23.062704 2569 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 06 13:50:23.066790 ip-10-0-136-25 kubenswrapper[2569]: W0706 13:50:23.062707 2569 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 06 13:50:23.066790 ip-10-0-136-25 kubenswrapper[2569]: W0706 13:50:23.062709 2569 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 06 13:50:23.066790 ip-10-0-136-25 kubenswrapper[2569]: W0706 13:50:23.062712 2569 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 06 13:50:23.066790 ip-10-0-136-25 kubenswrapper[2569]: W0706 13:50:23.062715 2569 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 06 13:50:23.066790 ip-10-0-136-25 kubenswrapper[2569]: W0706 13:50:23.062717 2569 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 06 13:50:23.066790 ip-10-0-136-25 kubenswrapper[2569]: W0706 13:50:23.062720 2569 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 06 13:50:23.066790 ip-10-0-136-25 kubenswrapper[2569]: W0706 13:50:23.062723 2569 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 06 13:50:23.066790 ip-10-0-136-25 kubenswrapper[2569]: W0706 13:50:23.062725 2569 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 06 13:50:23.067550 ip-10-0-136-25 kubenswrapper[2569]: W0706 13:50:23.062728 2569 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 06 13:50:23.067550 ip-10-0-136-25 kubenswrapper[2569]: W0706 13:50:23.062730 2569 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 06 13:50:23.067550 ip-10-0-136-25 kubenswrapper[2569]: W0706 13:50:23.062733 2569 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 06 13:50:23.067550 ip-10-0-136-25 kubenswrapper[2569]: W0706 13:50:23.062736 2569 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 06 13:50:23.067550 ip-10-0-136-25 kubenswrapper[2569]: W0706 13:50:23.062739 2569 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 06 13:50:23.067550 ip-10-0-136-25 kubenswrapper[2569]: W0706 13:50:23.062742 2569 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 06 13:50:23.067550 ip-10-0-136-25 kubenswrapper[2569]: W0706 13:50:23.062744 2569 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 06 13:50:23.067550 ip-10-0-136-25 kubenswrapper[2569]: W0706 13:50:23.062748 2569 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 06 13:50:23.067550 ip-10-0-136-25 kubenswrapper[2569]: W0706 13:50:23.062752 2569 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 06 13:50:23.067550 ip-10-0-136-25 kubenswrapper[2569]: W0706 13:50:23.062755 2569 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 06 13:50:23.067550 ip-10-0-136-25 kubenswrapper[2569]: W0706 13:50:23.062758 2569 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 06 13:50:23.067550 ip-10-0-136-25 kubenswrapper[2569]: W0706 13:50:23.062760 2569 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 06 13:50:23.067550 ip-10-0-136-25 kubenswrapper[2569]: W0706 13:50:23.062763 2569 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 06 13:50:23.067550 ip-10-0-136-25 kubenswrapper[2569]: W0706 13:50:23.062766 2569 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 06 13:50:23.067550 ip-10-0-136-25 kubenswrapper[2569]: W0706 13:50:23.062768 2569 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 06 13:50:23.067550 ip-10-0-136-25 kubenswrapper[2569]: W0706 13:50:23.062772 2569 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 06 13:50:23.067550 ip-10-0-136-25 kubenswrapper[2569]: W0706 13:50:23.062775 2569 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 06 13:50:23.067550 ip-10-0-136-25 kubenswrapper[2569]: W0706 13:50:23.062778 2569 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 06 13:50:23.067550 ip-10-0-136-25 kubenswrapper[2569]: W0706 13:50:23.062781 2569 feature_gate.go:328] unrecognized feature gate: Example Jul 06 13:50:23.068258 ip-10-0-136-25 kubenswrapper[2569]: W0706 13:50:23.062784 2569 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 06 13:50:23.068258 ip-10-0-136-25 kubenswrapper[2569]: W0706 13:50:23.062787 2569 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 06 13:50:23.068258 ip-10-0-136-25 kubenswrapper[2569]: W0706 13:50:23.062790 2569 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 06 13:50:23.068258 ip-10-0-136-25 kubenswrapper[2569]: W0706 13:50:23.062793 2569 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 06 13:50:23.068258 ip-10-0-136-25 kubenswrapper[2569]: W0706 13:50:23.062795 2569 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 06 13:50:23.068258 ip-10-0-136-25 kubenswrapper[2569]: W0706 13:50:23.062798 2569 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 06 13:50:23.068258 ip-10-0-136-25 kubenswrapper[2569]: W0706 13:50:23.062801 2569 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 06 13:50:23.068258 ip-10-0-136-25 kubenswrapper[2569]: W0706 13:50:23.062804 2569 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 06 13:50:23.068258 ip-10-0-136-25 kubenswrapper[2569]: W0706 13:50:23.062806 2569 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 06 13:50:23.068258 ip-10-0-136-25 kubenswrapper[2569]: W0706 13:50:23.062809 2569 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 06 13:50:23.068258 ip-10-0-136-25 kubenswrapper[2569]: W0706 13:50:23.062811 2569 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 06 13:50:23.068258 ip-10-0-136-25 kubenswrapper[2569]: W0706 13:50:23.062814 2569 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 06 13:50:23.068258 ip-10-0-136-25 kubenswrapper[2569]: W0706 13:50:23.062817 2569 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 06 13:50:23.068258 ip-10-0-136-25 kubenswrapper[2569]: W0706 13:50:23.062820 2569 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 06 13:50:23.068258 ip-10-0-136-25 kubenswrapper[2569]: W0706 13:50:23.062822 2569 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 06 13:50:23.068258 ip-10-0-136-25 kubenswrapper[2569]: W0706 13:50:23.062825 2569 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 06 13:50:23.068258 ip-10-0-136-25 kubenswrapper[2569]: W0706 13:50:23.062827 2569 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 06 13:50:23.068258 ip-10-0-136-25 kubenswrapper[2569]: W0706 13:50:23.062830 2569 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 06 13:50:23.068258 ip-10-0-136-25 kubenswrapper[2569]: W0706 13:50:23.062832 2569 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 06 13:50:23.068258 ip-10-0-136-25 kubenswrapper[2569]: W0706 13:50:23.062835 2569 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 06 13:50:23.068968 ip-10-0-136-25 kubenswrapper[2569]: W0706 13:50:23.062837 2569 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 06 13:50:23.068968 ip-10-0-136-25 kubenswrapper[2569]: W0706 13:50:23.063244 2569 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 06 13:50:23.068968 ip-10-0-136-25 kubenswrapper[2569]: W0706 13:50:23.063250 2569 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 06 13:50:23.068968 ip-10-0-136-25 kubenswrapper[2569]: W0706 13:50:23.063252 2569 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 06 13:50:23.068968 ip-10-0-136-25 kubenswrapper[2569]: W0706 13:50:23.063255 2569 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 06 13:50:23.068968 ip-10-0-136-25 kubenswrapper[2569]: W0706 13:50:23.063258 2569 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 06 13:50:23.068968 ip-10-0-136-25 kubenswrapper[2569]: W0706 13:50:23.063260 2569 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 06 13:50:23.068968 ip-10-0-136-25 kubenswrapper[2569]: W0706 13:50:23.063263 2569 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 06 13:50:23.068968 ip-10-0-136-25 kubenswrapper[2569]: W0706 13:50:23.063266 2569 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 06 13:50:23.068968 ip-10-0-136-25 kubenswrapper[2569]: W0706 13:50:23.063269 2569 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 06 13:50:23.068968 ip-10-0-136-25 kubenswrapper[2569]: W0706 13:50:23.063271 2569 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 06 13:50:23.068968 ip-10-0-136-25 kubenswrapper[2569]: W0706 13:50:23.063274 2569 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 06 13:50:23.068968 ip-10-0-136-25 kubenswrapper[2569]: W0706 13:50:23.063277 2569 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 06 13:50:23.068968 ip-10-0-136-25 kubenswrapper[2569]: W0706 13:50:23.063284 2569 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 06 13:50:23.068968 ip-10-0-136-25 kubenswrapper[2569]: W0706 13:50:23.063287 2569 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 06 13:50:23.068968 ip-10-0-136-25 kubenswrapper[2569]: W0706 13:50:23.063290 2569 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 06 13:50:23.068968 ip-10-0-136-25 kubenswrapper[2569]: W0706 13:50:23.063292 2569 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 06 13:50:23.068968 ip-10-0-136-25 kubenswrapper[2569]: W0706 13:50:23.063295 2569 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 06 13:50:23.068968 ip-10-0-136-25 kubenswrapper[2569]: W0706 13:50:23.063298 2569 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 06 13:50:23.068968 ip-10-0-136-25 kubenswrapper[2569]: W0706 13:50:23.063300 2569 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 06 13:50:23.069696 ip-10-0-136-25 kubenswrapper[2569]: W0706 13:50:23.063303 2569 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 06 13:50:23.069696 ip-10-0-136-25 kubenswrapper[2569]: W0706 13:50:23.063305 2569 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 06 13:50:23.069696 ip-10-0-136-25 kubenswrapper[2569]: W0706 13:50:23.063308 2569 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 06 13:50:23.069696 ip-10-0-136-25 kubenswrapper[2569]: W0706 13:50:23.063311 2569 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 06 13:50:23.069696 ip-10-0-136-25 kubenswrapper[2569]: W0706 13:50:23.063314 2569 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 06 13:50:23.069696 ip-10-0-136-25 kubenswrapper[2569]: W0706 13:50:23.063316 2569 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 06 13:50:23.069696 ip-10-0-136-25 kubenswrapper[2569]: W0706 13:50:23.063319 2569 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 06 13:50:23.069696 ip-10-0-136-25 kubenswrapper[2569]: W0706 13:50:23.063323 2569 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 06 13:50:23.069696 ip-10-0-136-25 kubenswrapper[2569]: W0706 13:50:23.063326 2569 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 06 13:50:23.069696 ip-10-0-136-25 kubenswrapper[2569]: W0706 13:50:23.063329 2569 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 06 13:50:23.069696 ip-10-0-136-25 kubenswrapper[2569]: W0706 13:50:23.063332 2569 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 06 13:50:23.069696 ip-10-0-136-25 kubenswrapper[2569]: W0706 13:50:23.063335 2569 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 06 13:50:23.069696 ip-10-0-136-25 kubenswrapper[2569]: W0706 13:50:23.063338 2569 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 06 13:50:23.069696 ip-10-0-136-25 kubenswrapper[2569]: W0706 13:50:23.063340 2569 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 06 13:50:23.069696 ip-10-0-136-25 kubenswrapper[2569]: W0706 13:50:23.063343 2569 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 06 13:50:23.069696 ip-10-0-136-25 kubenswrapper[2569]: W0706 13:50:23.063345 2569 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 06 13:50:23.069696 ip-10-0-136-25 kubenswrapper[2569]: W0706 13:50:23.063348 2569 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 06 13:50:23.069696 ip-10-0-136-25 kubenswrapper[2569]: W0706 13:50:23.063350 2569 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 06 13:50:23.069696 ip-10-0-136-25 kubenswrapper[2569]: W0706 13:50:23.063353 2569 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 06 13:50:23.070385 ip-10-0-136-25 kubenswrapper[2569]: W0706 13:50:23.063355 2569 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 06 13:50:23.070385 ip-10-0-136-25 kubenswrapper[2569]: W0706 13:50:23.063357 2569 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 06 13:50:23.070385 ip-10-0-136-25 kubenswrapper[2569]: W0706 13:50:23.063360 2569 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 06 13:50:23.070385 ip-10-0-136-25 kubenswrapper[2569]: W0706 13:50:23.063362 2569 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 06 13:50:23.070385 ip-10-0-136-25 kubenswrapper[2569]: W0706 13:50:23.063365 2569 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 06 13:50:23.070385 ip-10-0-136-25 kubenswrapper[2569]: W0706 13:50:23.063367 2569 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 06 13:50:23.070385 ip-10-0-136-25 kubenswrapper[2569]: W0706 13:50:23.063372 2569 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 06 13:50:23.070385 ip-10-0-136-25 kubenswrapper[2569]: W0706 13:50:23.063374 2569 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 06 13:50:23.070385 ip-10-0-136-25 kubenswrapper[2569]: W0706 13:50:23.063377 2569 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 06 13:50:23.070385 ip-10-0-136-25 kubenswrapper[2569]: W0706 13:50:23.063379 2569 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 06 13:50:23.070385 ip-10-0-136-25 kubenswrapper[2569]: W0706 13:50:23.063382 2569 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 06 13:50:23.070385 ip-10-0-136-25 kubenswrapper[2569]: W0706 13:50:23.063384 2569 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 06 13:50:23.070385 ip-10-0-136-25 kubenswrapper[2569]: W0706 13:50:23.063386 2569 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 06 13:50:23.070385 ip-10-0-136-25 kubenswrapper[2569]: W0706 13:50:23.063390 2569 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 06 13:50:23.070385 ip-10-0-136-25 kubenswrapper[2569]: W0706 13:50:23.063392 2569 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 06 13:50:23.070385 ip-10-0-136-25 kubenswrapper[2569]: W0706 13:50:23.063396 2569 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 06 13:50:23.070385 ip-10-0-136-25 kubenswrapper[2569]: W0706 13:50:23.063399 2569 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 06 13:50:23.070385 ip-10-0-136-25 kubenswrapper[2569]: W0706 13:50:23.063401 2569 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 06 13:50:23.070385 ip-10-0-136-25 kubenswrapper[2569]: W0706 13:50:23.063404 2569 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 06 13:50:23.070385 ip-10-0-136-25 kubenswrapper[2569]: W0706 13:50:23.063406 2569 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 06 13:50:23.071091 ip-10-0-136-25 kubenswrapper[2569]: W0706 13:50:23.063409 2569 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 06 13:50:23.071091 ip-10-0-136-25 kubenswrapper[2569]: W0706 13:50:23.063411 2569 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 06 13:50:23.071091 ip-10-0-136-25 kubenswrapper[2569]: W0706 13:50:23.063414 2569 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 06 13:50:23.071091 ip-10-0-136-25 kubenswrapper[2569]: W0706 13:50:23.063416 2569 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 06 13:50:23.071091 ip-10-0-136-25 kubenswrapper[2569]: W0706 13:50:23.063419 2569 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 06 13:50:23.071091 ip-10-0-136-25 kubenswrapper[2569]: W0706 13:50:23.063422 2569 feature_gate.go:328] unrecognized feature gate: Example Jul 06 13:50:23.071091 ip-10-0-136-25 kubenswrapper[2569]: W0706 13:50:23.063426 2569 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 06 13:50:23.071091 ip-10-0-136-25 kubenswrapper[2569]: W0706 13:50:23.063429 2569 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 06 13:50:23.071091 ip-10-0-136-25 kubenswrapper[2569]: W0706 13:50:23.063432 2569 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 06 13:50:23.071091 ip-10-0-136-25 kubenswrapper[2569]: W0706 13:50:23.063435 2569 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 06 13:50:23.071091 ip-10-0-136-25 kubenswrapper[2569]: W0706 13:50:23.063438 2569 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 06 13:50:23.071091 ip-10-0-136-25 kubenswrapper[2569]: W0706 13:50:23.063441 2569 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 06 13:50:23.071091 ip-10-0-136-25 kubenswrapper[2569]: W0706 13:50:23.063443 2569 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 06 13:50:23.071091 ip-10-0-136-25 kubenswrapper[2569]: W0706 13:50:23.063446 2569 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 06 13:50:23.071091 ip-10-0-136-25 kubenswrapper[2569]: W0706 13:50:23.063448 2569 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 06 13:50:23.071091 ip-10-0-136-25 kubenswrapper[2569]: W0706 13:50:23.063451 2569 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 06 13:50:23.071091 ip-10-0-136-25 kubenswrapper[2569]: W0706 13:50:23.063454 2569 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 06 13:50:23.071091 ip-10-0-136-25 kubenswrapper[2569]: W0706 13:50:23.063457 2569 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 06 13:50:23.071091 ip-10-0-136-25 kubenswrapper[2569]: W0706 13:50:23.063461 2569 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 06 13:50:23.071770 ip-10-0-136-25 kubenswrapper[2569]: W0706 13:50:23.063465 2569 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 06 13:50:23.071770 ip-10-0-136-25 kubenswrapper[2569]: W0706 13:50:23.063468 2569 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 06 13:50:23.071770 ip-10-0-136-25 kubenswrapper[2569]: W0706 13:50:23.063471 2569 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 06 13:50:23.071770 ip-10-0-136-25 kubenswrapper[2569]: W0706 13:50:23.063473 2569 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 06 13:50:23.071770 ip-10-0-136-25 kubenswrapper[2569]: W0706 13:50:23.063476 2569 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 06 13:50:23.071770 ip-10-0-136-25 kubenswrapper[2569]: W0706 13:50:23.063478 2569 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 06 13:50:23.071770 ip-10-0-136-25 kubenswrapper[2569]: W0706 13:50:23.063481 2569 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 06 13:50:23.071770 ip-10-0-136-25 kubenswrapper[2569]: W0706 13:50:23.063483 2569 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 06 13:50:23.071770 ip-10-0-136-25 kubenswrapper[2569]: W0706 13:50:23.063487 2569 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 06 13:50:23.071770 ip-10-0-136-25 kubenswrapper[2569]: W0706 13:50:23.063490 2569 feature_gate.go:328] unrecognized feature gate: Example2 Jul 06 13:50:23.071770 ip-10-0-136-25 kubenswrapper[2569]: W0706 13:50:23.063492 2569 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 06 13:50:23.071770 ip-10-0-136-25 kubenswrapper[2569]: W0706 13:50:23.063495 2569 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 06 13:50:23.071770 ip-10-0-136-25 kubenswrapper[2569]: W0706 13:50:23.063497 2569 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 06 13:50:23.071770 ip-10-0-136-25 kubenswrapper[2569]: W0706 13:50:23.063500 2569 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 06 13:50:23.071770 ip-10-0-136-25 kubenswrapper[2569]: W0706 13:50:23.063502 2569 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 06 13:50:23.071770 ip-10-0-136-25 kubenswrapper[2569]: W0706 13:50:23.063505 2569 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 06 13:50:23.071770 ip-10-0-136-25 kubenswrapper[2569]: W0706 13:50:23.063507 2569 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 06 13:50:23.071770 ip-10-0-136-25 kubenswrapper[2569]: W0706 13:50:23.063510 2569 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 06 13:50:23.071770 ip-10-0-136-25 kubenswrapper[2569]: W0706 13:50:23.063512 2569 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 06 13:50:23.071770 ip-10-0-136-25 kubenswrapper[2569]: W0706 13:50:23.063515 2569 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 06 13:50:23.072491 ip-10-0-136-25 kubenswrapper[2569]: W0706 13:50:23.063517 2569 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 06 13:50:23.072491 ip-10-0-136-25 kubenswrapper[2569]: W0706 13:50:23.063519 2569 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 06 13:50:23.072491 ip-10-0-136-25 kubenswrapper[2569]: I0706 13:50:23.063590 2569 flags.go:64] FLAG: --address="0.0.0.0" Jul 06 13:50:23.072491 ip-10-0-136-25 kubenswrapper[2569]: I0706 13:50:23.063602 2569 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]" Jul 06 13:50:23.072491 ip-10-0-136-25 kubenswrapper[2569]: I0706 13:50:23.063608 2569 flags.go:64] FLAG: --anonymous-auth="true" Jul 06 13:50:23.072491 ip-10-0-136-25 kubenswrapper[2569]: I0706 13:50:23.063612 2569 flags.go:64] FLAG: --application-metrics-count-limit="100" Jul 06 13:50:23.072491 ip-10-0-136-25 kubenswrapper[2569]: I0706 13:50:23.063617 2569 flags.go:64] FLAG: --authentication-token-webhook="false" Jul 06 13:50:23.072491 ip-10-0-136-25 kubenswrapper[2569]: I0706 13:50:23.063620 2569 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s" Jul 06 13:50:23.072491 ip-10-0-136-25 kubenswrapper[2569]: I0706 13:50:23.063624 2569 flags.go:64] FLAG: --authorization-mode="AlwaysAllow" Jul 06 13:50:23.072491 ip-10-0-136-25 kubenswrapper[2569]: I0706 13:50:23.063628 2569 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s" Jul 06 13:50:23.072491 ip-10-0-136-25 kubenswrapper[2569]: I0706 13:50:23.063632 2569 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s" Jul 06 13:50:23.072491 ip-10-0-136-25 kubenswrapper[2569]: I0706 13:50:23.063636 2569 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id" Jul 06 13:50:23.072491 ip-10-0-136-25 kubenswrapper[2569]: I0706 13:50:23.063640 2569 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig" Jul 06 13:50:23.072491 ip-10-0-136-25 kubenswrapper[2569]: I0706 13:50:23.063644 2569 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki" Jul 06 13:50:23.072491 ip-10-0-136-25 kubenswrapper[2569]: I0706 13:50:23.063647 2569 flags.go:64] FLAG: --cgroup-driver="cgroupfs" Jul 06 13:50:23.072491 ip-10-0-136-25 kubenswrapper[2569]: I0706 13:50:23.063650 2569 flags.go:64] FLAG: --cgroup-root="" Jul 06 13:50:23.072491 ip-10-0-136-25 kubenswrapper[2569]: I0706 13:50:23.063653 2569 flags.go:64] FLAG: --cgroups-per-qos="true" Jul 06 13:50:23.072491 ip-10-0-136-25 kubenswrapper[2569]: I0706 13:50:23.063656 2569 flags.go:64] FLAG: --client-ca-file="" Jul 06 13:50:23.072491 ip-10-0-136-25 kubenswrapper[2569]: I0706 13:50:23.063659 2569 flags.go:64] FLAG: --cloud-provider="external" Jul 06 13:50:23.072491 ip-10-0-136-25 kubenswrapper[2569]: I0706 13:50:23.063662 2569 flags.go:64] FLAG: --cluster-dns="[]" Jul 06 13:50:23.072491 ip-10-0-136-25 kubenswrapper[2569]: I0706 13:50:23.063668 2569 flags.go:64] FLAG: --cluster-domain="" Jul 06 13:50:23.072491 ip-10-0-136-25 kubenswrapper[2569]: I0706 13:50:23.063671 2569 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf" Jul 06 13:50:23.072491 ip-10-0-136-25 kubenswrapper[2569]: I0706 13:50:23.063674 2569 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d" Jul 06 13:50:23.072491 ip-10-0-136-25 kubenswrapper[2569]: I0706 13:50:23.063677 2569 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json" Jul 06 13:50:23.073396 ip-10-0-136-25 kubenswrapper[2569]: I0706 13:50:23.063680 2569 flags.go:64] FLAG: --container-log-max-files="5" Jul 06 13:50:23.073396 ip-10-0-136-25 kubenswrapper[2569]: I0706 13:50:23.063684 2569 flags.go:64] FLAG: --container-log-max-size="10Mi" Jul 06 13:50:23.073396 ip-10-0-136-25 kubenswrapper[2569]: I0706 13:50:23.063687 2569 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock" Jul 06 13:50:23.073396 ip-10-0-136-25 kubenswrapper[2569]: I0706 13:50:23.063690 2569 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock" Jul 06 13:50:23.073396 ip-10-0-136-25 kubenswrapper[2569]: I0706 13:50:23.063693 2569 flags.go:64] FLAG: --containerd-namespace="k8s.io" Jul 06 13:50:23.073396 ip-10-0-136-25 kubenswrapper[2569]: I0706 13:50:23.063696 2569 flags.go:64] FLAG: --contention-profiling="false" Jul 06 13:50:23.073396 ip-10-0-136-25 kubenswrapper[2569]: I0706 13:50:23.063700 2569 flags.go:64] FLAG: --cpu-cfs-quota="true" Jul 06 13:50:23.073396 ip-10-0-136-25 kubenswrapper[2569]: I0706 13:50:23.063703 2569 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms" Jul 06 13:50:23.073396 ip-10-0-136-25 kubenswrapper[2569]: I0706 13:50:23.063706 2569 flags.go:64] FLAG: --cpu-manager-policy="none" Jul 06 13:50:23.073396 ip-10-0-136-25 kubenswrapper[2569]: I0706 13:50:23.063709 2569 flags.go:64] FLAG: --cpu-manager-policy-options="" Jul 06 13:50:23.073396 ip-10-0-136-25 kubenswrapper[2569]: I0706 13:50:23.063713 2569 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s" Jul 06 13:50:23.073396 ip-10-0-136-25 kubenswrapper[2569]: I0706 13:50:23.063717 2569 flags.go:64] FLAG: --enable-controller-attach-detach="true" Jul 06 13:50:23.073396 ip-10-0-136-25 kubenswrapper[2569]: I0706 13:50:23.063719 2569 flags.go:64] FLAG: --enable-debugging-handlers="true" Jul 06 13:50:23.073396 ip-10-0-136-25 kubenswrapper[2569]: I0706 13:50:23.063722 2569 flags.go:64] FLAG: --enable-load-reader="false" Jul 06 13:50:23.073396 ip-10-0-136-25 kubenswrapper[2569]: I0706 13:50:23.063725 2569 flags.go:64] FLAG: --enable-server="true" Jul 06 13:50:23.073396 ip-10-0-136-25 kubenswrapper[2569]: I0706 13:50:23.063728 2569 flags.go:64] FLAG: --enforce-node-allocatable="[pods]" Jul 06 13:50:23.073396 ip-10-0-136-25 kubenswrapper[2569]: I0706 13:50:23.063732 2569 flags.go:64] FLAG: --event-burst="100" Jul 06 13:50:23.073396 ip-10-0-136-25 kubenswrapper[2569]: I0706 13:50:23.063735 2569 flags.go:64] FLAG: --event-qps="50" Jul 06 13:50:23.073396 ip-10-0-136-25 kubenswrapper[2569]: I0706 13:50:23.063738 2569 flags.go:64] FLAG: --event-storage-age-limit="default=0" Jul 06 13:50:23.073396 ip-10-0-136-25 kubenswrapper[2569]: I0706 13:50:23.063743 2569 flags.go:64] FLAG: --event-storage-event-limit="default=0" Jul 06 13:50:23.073396 ip-10-0-136-25 kubenswrapper[2569]: I0706 13:50:23.063746 2569 flags.go:64] FLAG: --eviction-hard="" Jul 06 13:50:23.073396 ip-10-0-136-25 kubenswrapper[2569]: I0706 13:50:23.063750 2569 flags.go:64] FLAG: --eviction-max-pod-grace-period="0" Jul 06 13:50:23.073396 ip-10-0-136-25 kubenswrapper[2569]: I0706 13:50:23.063753 2569 flags.go:64] FLAG: --eviction-minimum-reclaim="" Jul 06 13:50:23.073396 ip-10-0-136-25 kubenswrapper[2569]: I0706 13:50:23.063756 2569 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s" Jul 06 13:50:23.073396 ip-10-0-136-25 kubenswrapper[2569]: I0706 13:50:23.063759 2569 flags.go:64] FLAG: --eviction-soft="" Jul 06 13:50:23.074299 ip-10-0-136-25 kubenswrapper[2569]: I0706 13:50:23.063762 2569 flags.go:64] FLAG: --eviction-soft-grace-period="" Jul 06 13:50:23.074299 ip-10-0-136-25 kubenswrapper[2569]: I0706 13:50:23.063765 2569 flags.go:64] FLAG: --exit-on-lock-contention="false" Jul 06 13:50:23.074299 ip-10-0-136-25 kubenswrapper[2569]: I0706 13:50:23.063768 2569 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false" Jul 06 13:50:23.074299 ip-10-0-136-25 kubenswrapper[2569]: I0706 13:50:23.063772 2569 flags.go:64] FLAG: --experimental-mounter-path="" Jul 06 13:50:23.074299 ip-10-0-136-25 kubenswrapper[2569]: I0706 13:50:23.063775 2569 flags.go:64] FLAG: --fail-cgroupv1="false" Jul 06 13:50:23.074299 ip-10-0-136-25 kubenswrapper[2569]: I0706 13:50:23.063778 2569 flags.go:64] FLAG: --fail-swap-on="true" Jul 06 13:50:23.074299 ip-10-0-136-25 kubenswrapper[2569]: I0706 13:50:23.063781 2569 flags.go:64] FLAG: --feature-gates="" Jul 06 13:50:23.074299 ip-10-0-136-25 kubenswrapper[2569]: I0706 13:50:23.063785 2569 flags.go:64] FLAG: --file-check-frequency="20s" Jul 06 13:50:23.074299 ip-10-0-136-25 kubenswrapper[2569]: I0706 13:50:23.063788 2569 flags.go:64] FLAG: --global-housekeeping-interval="1m0s" Jul 06 13:50:23.074299 ip-10-0-136-25 kubenswrapper[2569]: I0706 13:50:23.063791 2569 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge" Jul 06 13:50:23.074299 ip-10-0-136-25 kubenswrapper[2569]: I0706 13:50:23.063794 2569 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1" Jul 06 13:50:23.074299 ip-10-0-136-25 kubenswrapper[2569]: I0706 13:50:23.063797 2569 flags.go:64] FLAG: --healthz-port="10248" Jul 06 13:50:23.074299 ip-10-0-136-25 kubenswrapper[2569]: I0706 13:50:23.063800 2569 flags.go:64] FLAG: --help="false" Jul 06 13:50:23.074299 ip-10-0-136-25 kubenswrapper[2569]: I0706 13:50:23.063803 2569 flags.go:64] FLAG: --hostname-override="ip-10-0-136-25.ec2.internal" Jul 06 13:50:23.074299 ip-10-0-136-25 kubenswrapper[2569]: I0706 13:50:23.063806 2569 flags.go:64] FLAG: --housekeeping-interval="10s" Jul 06 13:50:23.074299 ip-10-0-136-25 kubenswrapper[2569]: I0706 13:50:23.063809 2569 flags.go:64] FLAG: --http-check-frequency="20s" Jul 06 13:50:23.074299 ip-10-0-136-25 kubenswrapper[2569]: I0706 13:50:23.063812 2569 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins" Jul 06 13:50:23.074299 ip-10-0-136-25 kubenswrapper[2569]: I0706 13:50:23.063815 2569 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml" Jul 06 13:50:23.074299 ip-10-0-136-25 kubenswrapper[2569]: I0706 13:50:23.063819 2569 flags.go:64] FLAG: --image-gc-high-threshold="85" Jul 06 13:50:23.074299 ip-10-0-136-25 kubenswrapper[2569]: I0706 13:50:23.063822 2569 flags.go:64] FLAG: --image-gc-low-threshold="80" Jul 06 13:50:23.074299 ip-10-0-136-25 kubenswrapper[2569]: I0706 13:50:23.063825 2569 flags.go:64] FLAG: --image-service-endpoint="" Jul 06 13:50:23.074299 ip-10-0-136-25 kubenswrapper[2569]: I0706 13:50:23.063828 2569 flags.go:64] FLAG: --kernel-memcg-notification="false" Jul 06 13:50:23.074299 ip-10-0-136-25 kubenswrapper[2569]: I0706 13:50:23.063831 2569 flags.go:64] FLAG: --kube-api-burst="100" Jul 06 13:50:23.075197 ip-10-0-136-25 kubenswrapper[2569]: I0706 13:50:23.063834 2569 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf" Jul 06 13:50:23.075197 ip-10-0-136-25 kubenswrapper[2569]: I0706 13:50:23.063837 2569 flags.go:64] FLAG: --kube-api-qps="50" Jul 06 13:50:23.075197 ip-10-0-136-25 kubenswrapper[2569]: I0706 13:50:23.063839 2569 flags.go:64] FLAG: --kube-reserved="" Jul 06 13:50:23.075197 ip-10-0-136-25 kubenswrapper[2569]: I0706 13:50:23.063844 2569 flags.go:64] FLAG: --kube-reserved-cgroup="" Jul 06 13:50:23.075197 ip-10-0-136-25 kubenswrapper[2569]: I0706 13:50:23.063846 2569 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig" Jul 06 13:50:23.075197 ip-10-0-136-25 kubenswrapper[2569]: I0706 13:50:23.063849 2569 flags.go:64] FLAG: --kubelet-cgroups="" Jul 06 13:50:23.075197 ip-10-0-136-25 kubenswrapper[2569]: I0706 13:50:23.063852 2569 flags.go:64] FLAG: --local-storage-capacity-isolation="true" Jul 06 13:50:23.075197 ip-10-0-136-25 kubenswrapper[2569]: I0706 13:50:23.063856 2569 flags.go:64] FLAG: --lock-file="" Jul 06 13:50:23.075197 ip-10-0-136-25 kubenswrapper[2569]: I0706 13:50:23.063859 2569 flags.go:64] FLAG: --log-cadvisor-usage="false" Jul 06 13:50:23.075197 ip-10-0-136-25 kubenswrapper[2569]: I0706 13:50:23.063862 2569 flags.go:64] FLAG: --log-flush-frequency="5s" Jul 06 13:50:23.075197 ip-10-0-136-25 kubenswrapper[2569]: I0706 13:50:23.063864 2569 flags.go:64] FLAG: --log-json-info-buffer-size="0" Jul 06 13:50:23.075197 ip-10-0-136-25 kubenswrapper[2569]: I0706 13:50:23.063870 2569 flags.go:64] FLAG: --log-json-split-stream="false" Jul 06 13:50:23.075197 ip-10-0-136-25 kubenswrapper[2569]: I0706 13:50:23.063874 2569 flags.go:64] FLAG: --log-text-info-buffer-size="0" Jul 06 13:50:23.075197 ip-10-0-136-25 kubenswrapper[2569]: I0706 13:50:23.063877 2569 flags.go:64] FLAG: --log-text-split-stream="false" Jul 06 13:50:23.075197 ip-10-0-136-25 kubenswrapper[2569]: I0706 13:50:23.063880 2569 flags.go:64] FLAG: --logging-format="text" Jul 06 13:50:23.075197 ip-10-0-136-25 kubenswrapper[2569]: I0706 13:50:23.063883 2569 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id" Jul 06 13:50:23.075197 ip-10-0-136-25 kubenswrapper[2569]: I0706 13:50:23.063886 2569 flags.go:64] FLAG: --make-iptables-util-chains="true" Jul 06 13:50:23.075197 ip-10-0-136-25 kubenswrapper[2569]: I0706 13:50:23.063889 2569 flags.go:64] FLAG: --manifest-url="" Jul 06 13:50:23.075197 ip-10-0-136-25 kubenswrapper[2569]: I0706 13:50:23.063891 2569 flags.go:64] FLAG: --manifest-url-header="" Jul 06 13:50:23.075197 ip-10-0-136-25 kubenswrapper[2569]: I0706 13:50:23.063896 2569 flags.go:64] FLAG: --max-housekeeping-interval="15s" Jul 06 13:50:23.075197 ip-10-0-136-25 kubenswrapper[2569]: I0706 13:50:23.063899 2569 flags.go:64] FLAG: --max-open-files="1000000" Jul 06 13:50:23.075197 ip-10-0-136-25 kubenswrapper[2569]: I0706 13:50:23.063903 2569 flags.go:64] FLAG: --max-pods="110" Jul 06 13:50:23.075197 ip-10-0-136-25 kubenswrapper[2569]: I0706 13:50:23.063905 2569 flags.go:64] FLAG: --maximum-dead-containers="-1" Jul 06 13:50:23.075197 ip-10-0-136-25 kubenswrapper[2569]: I0706 13:50:23.063908 2569 flags.go:64] FLAG: --maximum-dead-containers-per-container="1" Jul 06 13:50:23.075197 ip-10-0-136-25 kubenswrapper[2569]: I0706 13:50:23.063911 2569 flags.go:64] FLAG: --memory-manager-policy="None" Jul 06 13:50:23.076094 ip-10-0-136-25 kubenswrapper[2569]: I0706 13:50:23.063914 2569 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s" Jul 06 13:50:23.076094 ip-10-0-136-25 kubenswrapper[2569]: I0706 13:50:23.063917 2569 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s" Jul 06 13:50:23.076094 ip-10-0-136-25 kubenswrapper[2569]: I0706 13:50:23.063920 2569 flags.go:64] FLAG: --node-ip="0.0.0.0" Jul 06 13:50:23.076094 ip-10-0-136-25 kubenswrapper[2569]: I0706 13:50:23.063923 2569 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel" Jul 06 13:50:23.076094 ip-10-0-136-25 kubenswrapper[2569]: I0706 13:50:23.063929 2569 flags.go:64] FLAG: --node-status-max-images="50" Jul 06 13:50:23.076094 ip-10-0-136-25 kubenswrapper[2569]: I0706 13:50:23.063932 2569 flags.go:64] FLAG: --node-status-update-frequency="10s" Jul 06 13:50:23.076094 ip-10-0-136-25 kubenswrapper[2569]: I0706 13:50:23.063935 2569 flags.go:64] FLAG: --oom-score-adj="-999" Jul 06 13:50:23.076094 ip-10-0-136-25 kubenswrapper[2569]: I0706 13:50:23.063938 2569 flags.go:64] FLAG: --pod-cidr="" Jul 06 13:50:23.076094 ip-10-0-136-25 kubenswrapper[2569]: I0706 13:50:23.063941 2569 flags.go:64] FLAG: --pod-infra-container-image="quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:ce93f6b4197b9c62a26a3131f8057ffe9c9379c5079a4f62e5919b75151394e5" Jul 06 13:50:23.076094 ip-10-0-136-25 kubenswrapper[2569]: I0706 13:50:23.063947 2569 flags.go:64] FLAG: --pod-manifest-path="" Jul 06 13:50:23.076094 ip-10-0-136-25 kubenswrapper[2569]: I0706 13:50:23.063950 2569 flags.go:64] FLAG: --pod-max-pids="-1" Jul 06 13:50:23.076094 ip-10-0-136-25 kubenswrapper[2569]: I0706 13:50:23.063954 2569 flags.go:64] FLAG: --pods-per-core="0" Jul 06 13:50:23.076094 ip-10-0-136-25 kubenswrapper[2569]: I0706 13:50:23.063957 2569 flags.go:64] FLAG: --port="10250" Jul 06 13:50:23.076094 ip-10-0-136-25 kubenswrapper[2569]: I0706 13:50:23.063960 2569 flags.go:64] FLAG: --protect-kernel-defaults="false" Jul 06 13:50:23.076094 ip-10-0-136-25 kubenswrapper[2569]: I0706 13:50:23.063962 2569 flags.go:64] FLAG: --provider-id="aws:///us-east-1a/i-03ae7cd981128bfc7" Jul 06 13:50:23.076094 ip-10-0-136-25 kubenswrapper[2569]: I0706 13:50:23.063966 2569 flags.go:64] FLAG: --qos-reserved="" Jul 06 13:50:23.076094 ip-10-0-136-25 kubenswrapper[2569]: I0706 13:50:23.063969 2569 flags.go:64] FLAG: --read-only-port="10255" Jul 06 13:50:23.076094 ip-10-0-136-25 kubenswrapper[2569]: I0706 13:50:23.063972 2569 flags.go:64] FLAG: --register-node="true" Jul 06 13:50:23.076094 ip-10-0-136-25 kubenswrapper[2569]: I0706 13:50:23.063975 2569 flags.go:64] FLAG: --register-with-taints="" Jul 06 13:50:23.076094 ip-10-0-136-25 kubenswrapper[2569]: I0706 13:50:23.063980 2569 flags.go:64] FLAG: --registry-burst="10" Jul 06 13:50:23.076094 ip-10-0-136-25 kubenswrapper[2569]: I0706 13:50:23.063983 2569 flags.go:64] FLAG: --registry-qps="5" Jul 06 13:50:23.076094 ip-10-0-136-25 kubenswrapper[2569]: I0706 13:50:23.063986 2569 flags.go:64] FLAG: --reserved-cpus="" Jul 06 13:50:23.076094 ip-10-0-136-25 kubenswrapper[2569]: I0706 13:50:23.063989 2569 flags.go:64] FLAG: --reserved-memory="" Jul 06 13:50:23.076094 ip-10-0-136-25 kubenswrapper[2569]: I0706 13:50:23.063993 2569 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf" Jul 06 13:50:23.076094 ip-10-0-136-25 kubenswrapper[2569]: I0706 13:50:23.063996 2569 flags.go:64] FLAG: --root-dir="/var/lib/kubelet" Jul 06 13:50:23.077010 ip-10-0-136-25 kubenswrapper[2569]: I0706 13:50:23.063999 2569 flags.go:64] FLAG: --rotate-certificates="false" Jul 06 13:50:23.077010 ip-10-0-136-25 kubenswrapper[2569]: I0706 13:50:23.064002 2569 flags.go:64] FLAG: --rotate-server-certificates="false" Jul 06 13:50:23.077010 ip-10-0-136-25 kubenswrapper[2569]: I0706 13:50:23.064005 2569 flags.go:64] FLAG: --runonce="false" Jul 06 13:50:23.077010 ip-10-0-136-25 kubenswrapper[2569]: I0706 13:50:23.064008 2569 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service" Jul 06 13:50:23.077010 ip-10-0-136-25 kubenswrapper[2569]: I0706 13:50:23.064011 2569 flags.go:64] FLAG: --runtime-request-timeout="2m0s" Jul 06 13:50:23.077010 ip-10-0-136-25 kubenswrapper[2569]: I0706 13:50:23.064014 2569 flags.go:64] FLAG: --seccomp-default="false" Jul 06 13:50:23.077010 ip-10-0-136-25 kubenswrapper[2569]: I0706 13:50:23.064016 2569 flags.go:64] FLAG: --serialize-image-pulls="true" Jul 06 13:50:23.077010 ip-10-0-136-25 kubenswrapper[2569]: I0706 13:50:23.064019 2569 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s" Jul 06 13:50:23.077010 ip-10-0-136-25 kubenswrapper[2569]: I0706 13:50:23.064023 2569 flags.go:64] FLAG: --storage-driver-db="cadvisor" Jul 06 13:50:23.077010 ip-10-0-136-25 kubenswrapper[2569]: I0706 13:50:23.064026 2569 flags.go:64] FLAG: --storage-driver-host="localhost:8086" Jul 06 13:50:23.077010 ip-10-0-136-25 kubenswrapper[2569]: I0706 13:50:23.064029 2569 flags.go:64] FLAG: --storage-driver-password="root" Jul 06 13:50:23.077010 ip-10-0-136-25 kubenswrapper[2569]: I0706 13:50:23.064032 2569 flags.go:64] FLAG: --storage-driver-secure="false" Jul 06 13:50:23.077010 ip-10-0-136-25 kubenswrapper[2569]: I0706 13:50:23.064035 2569 flags.go:64] FLAG: --storage-driver-table="stats" Jul 06 13:50:23.077010 ip-10-0-136-25 kubenswrapper[2569]: I0706 13:50:23.064038 2569 flags.go:64] FLAG: --storage-driver-user="root" Jul 06 13:50:23.077010 ip-10-0-136-25 kubenswrapper[2569]: I0706 13:50:23.064040 2569 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s" Jul 06 13:50:23.077010 ip-10-0-136-25 kubenswrapper[2569]: I0706 13:50:23.064043 2569 flags.go:64] FLAG: --sync-frequency="1m0s" Jul 06 13:50:23.077010 ip-10-0-136-25 kubenswrapper[2569]: I0706 13:50:23.064046 2569 flags.go:64] FLAG: --system-cgroups="" Jul 06 13:50:23.077010 ip-10-0-136-25 kubenswrapper[2569]: I0706 13:50:23.064049 2569 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi" Jul 06 13:50:23.077010 ip-10-0-136-25 kubenswrapper[2569]: I0706 13:50:23.064056 2569 flags.go:64] FLAG: --system-reserved-cgroup="" Jul 06 13:50:23.077010 ip-10-0-136-25 kubenswrapper[2569]: I0706 13:50:23.064058 2569 flags.go:64] FLAG: --tls-cert-file="" Jul 06 13:50:23.077010 ip-10-0-136-25 kubenswrapper[2569]: I0706 13:50:23.064074 2569 flags.go:64] FLAG: --tls-cipher-suites="[]" Jul 06 13:50:23.077010 ip-10-0-136-25 kubenswrapper[2569]: I0706 13:50:23.064080 2569 flags.go:64] FLAG: --tls-min-version="" Jul 06 13:50:23.077010 ip-10-0-136-25 kubenswrapper[2569]: I0706 13:50:23.064083 2569 flags.go:64] FLAG: --tls-private-key-file="" Jul 06 13:50:23.077010 ip-10-0-136-25 kubenswrapper[2569]: I0706 13:50:23.064087 2569 flags.go:64] FLAG: --topology-manager-policy="none" Jul 06 13:50:23.077010 ip-10-0-136-25 kubenswrapper[2569]: I0706 13:50:23.064089 2569 flags.go:64] FLAG: --topology-manager-policy-options="" Jul 06 13:50:23.077963 ip-10-0-136-25 kubenswrapper[2569]: I0706 13:50:23.064093 2569 flags.go:64] FLAG: --topology-manager-scope="container" Jul 06 13:50:23.077963 ip-10-0-136-25 kubenswrapper[2569]: I0706 13:50:23.064096 2569 flags.go:64] FLAG: --v="2" Jul 06 13:50:23.077963 ip-10-0-136-25 kubenswrapper[2569]: I0706 13:50:23.064100 2569 flags.go:64] FLAG: --version="false" Jul 06 13:50:23.077963 ip-10-0-136-25 kubenswrapper[2569]: I0706 13:50:23.064104 2569 flags.go:64] FLAG: --vmodule="" Jul 06 13:50:23.077963 ip-10-0-136-25 kubenswrapper[2569]: I0706 13:50:23.064108 2569 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec" Jul 06 13:50:23.077963 ip-10-0-136-25 kubenswrapper[2569]: I0706 13:50:23.064111 2569 flags.go:64] FLAG: --volume-stats-agg-period="1m0s" Jul 06 13:50:23.077963 ip-10-0-136-25 kubenswrapper[2569]: W0706 13:50:23.064223 2569 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 06 13:50:23.077963 ip-10-0-136-25 kubenswrapper[2569]: W0706 13:50:23.064227 2569 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 06 13:50:23.077963 ip-10-0-136-25 kubenswrapper[2569]: W0706 13:50:23.064230 2569 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 06 13:50:23.077963 ip-10-0-136-25 kubenswrapper[2569]: W0706 13:50:23.064234 2569 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 06 13:50:23.077963 ip-10-0-136-25 kubenswrapper[2569]: W0706 13:50:23.064237 2569 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 06 13:50:23.077963 ip-10-0-136-25 kubenswrapper[2569]: W0706 13:50:23.064241 2569 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 06 13:50:23.077963 ip-10-0-136-25 kubenswrapper[2569]: W0706 13:50:23.064244 2569 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 06 13:50:23.077963 ip-10-0-136-25 kubenswrapper[2569]: W0706 13:50:23.064246 2569 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 06 13:50:23.077963 ip-10-0-136-25 kubenswrapper[2569]: W0706 13:50:23.064249 2569 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 06 13:50:23.077963 ip-10-0-136-25 kubenswrapper[2569]: W0706 13:50:23.064251 2569 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 06 13:50:23.077963 ip-10-0-136-25 kubenswrapper[2569]: W0706 13:50:23.064254 2569 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 06 13:50:23.077963 ip-10-0-136-25 kubenswrapper[2569]: W0706 13:50:23.064257 2569 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 06 13:50:23.077963 ip-10-0-136-25 kubenswrapper[2569]: W0706 13:50:23.064260 2569 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 06 13:50:23.077963 ip-10-0-136-25 kubenswrapper[2569]: W0706 13:50:23.064263 2569 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 06 13:50:23.077963 ip-10-0-136-25 kubenswrapper[2569]: W0706 13:50:23.064265 2569 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 06 13:50:23.078819 ip-10-0-136-25 kubenswrapper[2569]: W0706 13:50:23.064267 2569 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 06 13:50:23.078819 ip-10-0-136-25 kubenswrapper[2569]: W0706 13:50:23.064270 2569 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 06 13:50:23.078819 ip-10-0-136-25 kubenswrapper[2569]: W0706 13:50:23.064273 2569 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 06 13:50:23.078819 ip-10-0-136-25 kubenswrapper[2569]: W0706 13:50:23.064275 2569 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 06 13:50:23.078819 ip-10-0-136-25 kubenswrapper[2569]: W0706 13:50:23.064279 2569 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 06 13:50:23.078819 ip-10-0-136-25 kubenswrapper[2569]: W0706 13:50:23.064282 2569 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 06 13:50:23.078819 ip-10-0-136-25 kubenswrapper[2569]: W0706 13:50:23.064284 2569 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 06 13:50:23.078819 ip-10-0-136-25 kubenswrapper[2569]: W0706 13:50:23.064287 2569 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 06 13:50:23.078819 ip-10-0-136-25 kubenswrapper[2569]: W0706 13:50:23.064291 2569 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 06 13:50:23.078819 ip-10-0-136-25 kubenswrapper[2569]: W0706 13:50:23.064294 2569 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 06 13:50:23.078819 ip-10-0-136-25 kubenswrapper[2569]: W0706 13:50:23.064298 2569 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 06 13:50:23.078819 ip-10-0-136-25 kubenswrapper[2569]: W0706 13:50:23.064300 2569 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 06 13:50:23.078819 ip-10-0-136-25 kubenswrapper[2569]: W0706 13:50:23.064305 2569 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 06 13:50:23.078819 ip-10-0-136-25 kubenswrapper[2569]: W0706 13:50:23.064307 2569 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 06 13:50:23.078819 ip-10-0-136-25 kubenswrapper[2569]: W0706 13:50:23.064310 2569 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 06 13:50:23.078819 ip-10-0-136-25 kubenswrapper[2569]: W0706 13:50:23.064312 2569 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 06 13:50:23.078819 ip-10-0-136-25 kubenswrapper[2569]: W0706 13:50:23.064315 2569 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 06 13:50:23.078819 ip-10-0-136-25 kubenswrapper[2569]: W0706 13:50:23.064317 2569 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 06 13:50:23.078819 ip-10-0-136-25 kubenswrapper[2569]: W0706 13:50:23.064320 2569 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 06 13:50:23.078819 ip-10-0-136-25 kubenswrapper[2569]: W0706 13:50:23.064322 2569 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 06 13:50:23.079584 ip-10-0-136-25 kubenswrapper[2569]: W0706 13:50:23.064325 2569 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 06 13:50:23.079584 ip-10-0-136-25 kubenswrapper[2569]: W0706 13:50:23.064327 2569 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 06 13:50:23.079584 ip-10-0-136-25 kubenswrapper[2569]: W0706 13:50:23.064330 2569 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 06 13:50:23.079584 ip-10-0-136-25 kubenswrapper[2569]: W0706 13:50:23.064333 2569 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 06 13:50:23.079584 ip-10-0-136-25 kubenswrapper[2569]: W0706 13:50:23.064335 2569 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 06 13:50:23.079584 ip-10-0-136-25 kubenswrapper[2569]: W0706 13:50:23.064337 2569 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 06 13:50:23.079584 ip-10-0-136-25 kubenswrapper[2569]: W0706 13:50:23.064340 2569 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 06 13:50:23.079584 ip-10-0-136-25 kubenswrapper[2569]: W0706 13:50:23.064343 2569 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 06 13:50:23.079584 ip-10-0-136-25 kubenswrapper[2569]: W0706 13:50:23.064346 2569 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 06 13:50:23.079584 ip-10-0-136-25 kubenswrapper[2569]: W0706 13:50:23.064348 2569 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 06 13:50:23.079584 ip-10-0-136-25 kubenswrapper[2569]: W0706 13:50:23.064351 2569 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 06 13:50:23.079584 ip-10-0-136-25 kubenswrapper[2569]: W0706 13:50:23.064353 2569 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 06 13:50:23.079584 ip-10-0-136-25 kubenswrapper[2569]: W0706 13:50:23.064356 2569 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 06 13:50:23.079584 ip-10-0-136-25 kubenswrapper[2569]: W0706 13:50:23.064358 2569 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 06 13:50:23.079584 ip-10-0-136-25 kubenswrapper[2569]: W0706 13:50:23.064361 2569 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 06 13:50:23.079584 ip-10-0-136-25 kubenswrapper[2569]: W0706 13:50:23.064365 2569 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 06 13:50:23.079584 ip-10-0-136-25 kubenswrapper[2569]: W0706 13:50:23.064370 2569 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 06 13:50:23.079584 ip-10-0-136-25 kubenswrapper[2569]: W0706 13:50:23.064373 2569 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 06 13:50:23.079584 ip-10-0-136-25 kubenswrapper[2569]: W0706 13:50:23.064376 2569 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 06 13:50:23.079584 ip-10-0-136-25 kubenswrapper[2569]: W0706 13:50:23.064378 2569 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 06 13:50:23.080302 ip-10-0-136-25 kubenswrapper[2569]: W0706 13:50:23.064381 2569 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 06 13:50:23.080302 ip-10-0-136-25 kubenswrapper[2569]: W0706 13:50:23.064384 2569 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 06 13:50:23.080302 ip-10-0-136-25 kubenswrapper[2569]: W0706 13:50:23.064387 2569 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 06 13:50:23.080302 ip-10-0-136-25 kubenswrapper[2569]: W0706 13:50:23.064390 2569 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 06 13:50:23.080302 ip-10-0-136-25 kubenswrapper[2569]: W0706 13:50:23.064394 2569 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 06 13:50:23.080302 ip-10-0-136-25 kubenswrapper[2569]: W0706 13:50:23.064397 2569 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 06 13:50:23.080302 ip-10-0-136-25 kubenswrapper[2569]: W0706 13:50:23.064399 2569 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 06 13:50:23.080302 ip-10-0-136-25 kubenswrapper[2569]: W0706 13:50:23.064402 2569 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 06 13:50:23.080302 ip-10-0-136-25 kubenswrapper[2569]: W0706 13:50:23.064405 2569 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 06 13:50:23.080302 ip-10-0-136-25 kubenswrapper[2569]: W0706 13:50:23.064407 2569 feature_gate.go:328] unrecognized feature gate: Example2 Jul 06 13:50:23.080302 ip-10-0-136-25 kubenswrapper[2569]: W0706 13:50:23.064410 2569 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 06 13:50:23.080302 ip-10-0-136-25 kubenswrapper[2569]: W0706 13:50:23.064413 2569 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 06 13:50:23.080302 ip-10-0-136-25 kubenswrapper[2569]: W0706 13:50:23.064415 2569 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 06 13:50:23.080302 ip-10-0-136-25 kubenswrapper[2569]: W0706 13:50:23.064418 2569 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 06 13:50:23.080302 ip-10-0-136-25 kubenswrapper[2569]: W0706 13:50:23.064420 2569 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 06 13:50:23.080302 ip-10-0-136-25 kubenswrapper[2569]: W0706 13:50:23.064423 2569 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 06 13:50:23.080302 ip-10-0-136-25 kubenswrapper[2569]: W0706 13:50:23.064425 2569 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 06 13:50:23.080302 ip-10-0-136-25 kubenswrapper[2569]: W0706 13:50:23.064428 2569 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 06 13:50:23.080302 ip-10-0-136-25 kubenswrapper[2569]: W0706 13:50:23.064430 2569 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 06 13:50:23.080302 ip-10-0-136-25 kubenswrapper[2569]: W0706 13:50:23.064433 2569 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 06 13:50:23.081021 ip-10-0-136-25 kubenswrapper[2569]: W0706 13:50:23.064436 2569 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 06 13:50:23.081021 ip-10-0-136-25 kubenswrapper[2569]: W0706 13:50:23.064438 2569 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 06 13:50:23.081021 ip-10-0-136-25 kubenswrapper[2569]: W0706 13:50:23.064441 2569 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 06 13:50:23.081021 ip-10-0-136-25 kubenswrapper[2569]: W0706 13:50:23.064443 2569 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 06 13:50:23.081021 ip-10-0-136-25 kubenswrapper[2569]: W0706 13:50:23.064446 2569 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 06 13:50:23.081021 ip-10-0-136-25 kubenswrapper[2569]: W0706 13:50:23.064448 2569 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 06 13:50:23.081021 ip-10-0-136-25 kubenswrapper[2569]: W0706 13:50:23.064450 2569 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 06 13:50:23.081021 ip-10-0-136-25 kubenswrapper[2569]: W0706 13:50:23.064453 2569 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 06 13:50:23.081021 ip-10-0-136-25 kubenswrapper[2569]: W0706 13:50:23.064457 2569 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 06 13:50:23.081021 ip-10-0-136-25 kubenswrapper[2569]: W0706 13:50:23.064459 2569 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 06 13:50:23.081021 ip-10-0-136-25 kubenswrapper[2569]: W0706 13:50:23.064462 2569 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 06 13:50:23.081021 ip-10-0-136-25 kubenswrapper[2569]: W0706 13:50:23.064464 2569 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 06 13:50:23.081021 ip-10-0-136-25 kubenswrapper[2569]: W0706 13:50:23.064467 2569 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 06 13:50:23.081021 ip-10-0-136-25 kubenswrapper[2569]: W0706 13:50:23.064469 2569 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 06 13:50:23.081021 ip-10-0-136-25 kubenswrapper[2569]: W0706 13:50:23.064472 2569 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 06 13:50:23.081021 ip-10-0-136-25 kubenswrapper[2569]: W0706 13:50:23.064475 2569 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 06 13:50:23.081021 ip-10-0-136-25 kubenswrapper[2569]: W0706 13:50:23.064479 2569 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 06 13:50:23.081021 ip-10-0-136-25 kubenswrapper[2569]: W0706 13:50:23.064482 2569 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 06 13:50:23.081021 ip-10-0-136-25 kubenswrapper[2569]: W0706 13:50:23.064484 2569 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 06 13:50:23.081834 ip-10-0-136-25 kubenswrapper[2569]: W0706 13:50:23.064487 2569 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 06 13:50:23.081834 ip-10-0-136-25 kubenswrapper[2569]: W0706 13:50:23.064490 2569 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 06 13:50:23.081834 ip-10-0-136-25 kubenswrapper[2569]: W0706 13:50:23.064492 2569 feature_gate.go:328] unrecognized feature gate: Example Jul 06 13:50:23.081834 ip-10-0-136-25 kubenswrapper[2569]: W0706 13:50:23.064495 2569 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 06 13:50:23.081834 ip-10-0-136-25 kubenswrapper[2569]: W0706 13:50:23.064497 2569 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 06 13:50:23.081834 ip-10-0-136-25 kubenswrapper[2569]: I0706 13:50:23.065126 2569 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]} Jul 06 13:50:23.081834 ip-10-0-136-25 kubenswrapper[2569]: I0706 13:50:23.073376 2569 server.go:529] "Kubelet version" kubeletVersion="v1.34.8" Jul 06 13:50:23.081834 ip-10-0-136-25 kubenswrapper[2569]: I0706 13:50:23.073393 2569 server.go:531] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK="" Jul 06 13:50:23.081834 ip-10-0-136-25 kubenswrapper[2569]: I0706 13:50:23.073413 2569 watchdog_linux.go:95] "Systemd watchdog is not enabled" Jul 06 13:50:23.081834 ip-10-0-136-25 kubenswrapper[2569]: I0706 13:50:23.073421 2569 watchdog_linux.go:137] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started." Jul 06 13:50:23.081834 ip-10-0-136-25 kubenswrapper[2569]: W0706 13:50:23.073477 2569 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 06 13:50:23.081834 ip-10-0-136-25 kubenswrapper[2569]: W0706 13:50:23.073482 2569 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 06 13:50:23.081834 ip-10-0-136-25 kubenswrapper[2569]: W0706 13:50:23.073486 2569 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 06 13:50:23.081834 ip-10-0-136-25 kubenswrapper[2569]: W0706 13:50:23.073489 2569 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 06 13:50:23.081834 ip-10-0-136-25 kubenswrapper[2569]: W0706 13:50:23.073492 2569 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 06 13:50:23.082422 ip-10-0-136-25 kubenswrapper[2569]: W0706 13:50:23.073495 2569 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 06 13:50:23.082422 ip-10-0-136-25 kubenswrapper[2569]: W0706 13:50:23.073499 2569 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 06 13:50:23.082422 ip-10-0-136-25 kubenswrapper[2569]: W0706 13:50:23.073503 2569 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 06 13:50:23.082422 ip-10-0-136-25 kubenswrapper[2569]: W0706 13:50:23.073506 2569 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 06 13:50:23.082422 ip-10-0-136-25 kubenswrapper[2569]: W0706 13:50:23.073509 2569 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 06 13:50:23.082422 ip-10-0-136-25 kubenswrapper[2569]: W0706 13:50:23.073512 2569 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 06 13:50:23.082422 ip-10-0-136-25 kubenswrapper[2569]: W0706 13:50:23.073515 2569 feature_gate.go:328] unrecognized feature gate: Example Jul 06 13:50:23.082422 ip-10-0-136-25 kubenswrapper[2569]: W0706 13:50:23.073518 2569 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 06 13:50:23.082422 ip-10-0-136-25 kubenswrapper[2569]: W0706 13:50:23.073521 2569 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 06 13:50:23.082422 ip-10-0-136-25 kubenswrapper[2569]: W0706 13:50:23.073523 2569 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 06 13:50:23.082422 ip-10-0-136-25 kubenswrapper[2569]: W0706 13:50:23.073526 2569 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 06 13:50:23.082422 ip-10-0-136-25 kubenswrapper[2569]: W0706 13:50:23.073529 2569 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 06 13:50:23.082422 ip-10-0-136-25 kubenswrapper[2569]: W0706 13:50:23.073532 2569 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 06 13:50:23.082422 ip-10-0-136-25 kubenswrapper[2569]: W0706 13:50:23.073534 2569 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 06 13:50:23.082422 ip-10-0-136-25 kubenswrapper[2569]: W0706 13:50:23.073537 2569 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 06 13:50:23.082422 ip-10-0-136-25 kubenswrapper[2569]: W0706 13:50:23.073539 2569 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 06 13:50:23.082422 ip-10-0-136-25 kubenswrapper[2569]: W0706 13:50:23.073542 2569 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 06 13:50:23.082422 ip-10-0-136-25 kubenswrapper[2569]: W0706 13:50:23.073545 2569 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 06 13:50:23.082422 ip-10-0-136-25 kubenswrapper[2569]: W0706 13:50:23.073548 2569 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 06 13:50:23.083129 ip-10-0-136-25 kubenswrapper[2569]: W0706 13:50:23.073551 2569 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 06 13:50:23.083129 ip-10-0-136-25 kubenswrapper[2569]: W0706 13:50:23.073553 2569 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 06 13:50:23.083129 ip-10-0-136-25 kubenswrapper[2569]: W0706 13:50:23.073556 2569 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 06 13:50:23.083129 ip-10-0-136-25 kubenswrapper[2569]: W0706 13:50:23.073558 2569 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 06 13:50:23.083129 ip-10-0-136-25 kubenswrapper[2569]: W0706 13:50:23.073561 2569 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 06 13:50:23.083129 ip-10-0-136-25 kubenswrapper[2569]: W0706 13:50:23.073564 2569 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 06 13:50:23.083129 ip-10-0-136-25 kubenswrapper[2569]: W0706 13:50:23.073567 2569 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 06 13:50:23.083129 ip-10-0-136-25 kubenswrapper[2569]: W0706 13:50:23.073571 2569 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 06 13:50:23.083129 ip-10-0-136-25 kubenswrapper[2569]: W0706 13:50:23.073573 2569 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 06 13:50:23.083129 ip-10-0-136-25 kubenswrapper[2569]: W0706 13:50:23.073576 2569 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 06 13:50:23.083129 ip-10-0-136-25 kubenswrapper[2569]: W0706 13:50:23.073579 2569 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 06 13:50:23.083129 ip-10-0-136-25 kubenswrapper[2569]: W0706 13:50:23.073581 2569 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 06 13:50:23.083129 ip-10-0-136-25 kubenswrapper[2569]: W0706 13:50:23.073584 2569 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 06 13:50:23.083129 ip-10-0-136-25 kubenswrapper[2569]: W0706 13:50:23.073586 2569 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 06 13:50:23.083129 ip-10-0-136-25 kubenswrapper[2569]: W0706 13:50:23.073589 2569 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 06 13:50:23.083129 ip-10-0-136-25 kubenswrapper[2569]: W0706 13:50:23.073591 2569 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 06 13:50:23.083129 ip-10-0-136-25 kubenswrapper[2569]: W0706 13:50:23.073595 2569 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 06 13:50:23.083129 ip-10-0-136-25 kubenswrapper[2569]: W0706 13:50:23.073597 2569 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 06 13:50:23.083129 ip-10-0-136-25 kubenswrapper[2569]: W0706 13:50:23.073600 2569 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 06 13:50:23.083129 ip-10-0-136-25 kubenswrapper[2569]: W0706 13:50:23.073602 2569 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 06 13:50:23.083879 ip-10-0-136-25 kubenswrapper[2569]: W0706 13:50:23.073605 2569 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 06 13:50:23.083879 ip-10-0-136-25 kubenswrapper[2569]: W0706 13:50:23.073608 2569 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 06 13:50:23.083879 ip-10-0-136-25 kubenswrapper[2569]: W0706 13:50:23.073611 2569 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 06 13:50:23.083879 ip-10-0-136-25 kubenswrapper[2569]: W0706 13:50:23.073613 2569 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 06 13:50:23.083879 ip-10-0-136-25 kubenswrapper[2569]: W0706 13:50:23.073616 2569 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 06 13:50:23.083879 ip-10-0-136-25 kubenswrapper[2569]: W0706 13:50:23.073619 2569 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 06 13:50:23.083879 ip-10-0-136-25 kubenswrapper[2569]: W0706 13:50:23.073622 2569 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 06 13:50:23.083879 ip-10-0-136-25 kubenswrapper[2569]: W0706 13:50:23.073625 2569 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 06 13:50:23.083879 ip-10-0-136-25 kubenswrapper[2569]: W0706 13:50:23.073627 2569 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 06 13:50:23.083879 ip-10-0-136-25 kubenswrapper[2569]: W0706 13:50:23.073631 2569 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 06 13:50:23.083879 ip-10-0-136-25 kubenswrapper[2569]: W0706 13:50:23.073635 2569 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 06 13:50:23.083879 ip-10-0-136-25 kubenswrapper[2569]: W0706 13:50:23.073638 2569 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 06 13:50:23.083879 ip-10-0-136-25 kubenswrapper[2569]: W0706 13:50:23.073640 2569 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 06 13:50:23.083879 ip-10-0-136-25 kubenswrapper[2569]: W0706 13:50:23.073643 2569 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 06 13:50:23.083879 ip-10-0-136-25 kubenswrapper[2569]: W0706 13:50:23.073646 2569 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 06 13:50:23.083879 ip-10-0-136-25 kubenswrapper[2569]: W0706 13:50:23.073649 2569 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 06 13:50:23.083879 ip-10-0-136-25 kubenswrapper[2569]: W0706 13:50:23.073653 2569 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 06 13:50:23.083879 ip-10-0-136-25 kubenswrapper[2569]: W0706 13:50:23.073656 2569 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 06 13:50:23.083879 ip-10-0-136-25 kubenswrapper[2569]: W0706 13:50:23.073658 2569 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 06 13:50:23.083879 ip-10-0-136-25 kubenswrapper[2569]: W0706 13:50:23.073661 2569 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 06 13:50:23.084606 ip-10-0-136-25 kubenswrapper[2569]: W0706 13:50:23.073664 2569 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 06 13:50:23.084606 ip-10-0-136-25 kubenswrapper[2569]: W0706 13:50:23.073667 2569 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 06 13:50:23.084606 ip-10-0-136-25 kubenswrapper[2569]: W0706 13:50:23.073670 2569 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 06 13:50:23.084606 ip-10-0-136-25 kubenswrapper[2569]: W0706 13:50:23.073672 2569 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 06 13:50:23.084606 ip-10-0-136-25 kubenswrapper[2569]: W0706 13:50:23.073675 2569 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 06 13:50:23.084606 ip-10-0-136-25 kubenswrapper[2569]: W0706 13:50:23.073677 2569 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 06 13:50:23.084606 ip-10-0-136-25 kubenswrapper[2569]: W0706 13:50:23.073680 2569 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 06 13:50:23.084606 ip-10-0-136-25 kubenswrapper[2569]: W0706 13:50:23.073683 2569 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 06 13:50:23.084606 ip-10-0-136-25 kubenswrapper[2569]: W0706 13:50:23.073685 2569 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 06 13:50:23.084606 ip-10-0-136-25 kubenswrapper[2569]: W0706 13:50:23.073689 2569 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 06 13:50:23.084606 ip-10-0-136-25 kubenswrapper[2569]: W0706 13:50:23.073692 2569 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 06 13:50:23.084606 ip-10-0-136-25 kubenswrapper[2569]: W0706 13:50:23.073695 2569 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 06 13:50:23.084606 ip-10-0-136-25 kubenswrapper[2569]: W0706 13:50:23.073698 2569 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 06 13:50:23.084606 ip-10-0-136-25 kubenswrapper[2569]: W0706 13:50:23.073701 2569 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 06 13:50:23.084606 ip-10-0-136-25 kubenswrapper[2569]: W0706 13:50:23.073703 2569 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 06 13:50:23.084606 ip-10-0-136-25 kubenswrapper[2569]: W0706 13:50:23.073707 2569 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 06 13:50:23.084606 ip-10-0-136-25 kubenswrapper[2569]: W0706 13:50:23.073709 2569 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 06 13:50:23.084606 ip-10-0-136-25 kubenswrapper[2569]: W0706 13:50:23.073712 2569 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 06 13:50:23.084606 ip-10-0-136-25 kubenswrapper[2569]: W0706 13:50:23.073715 2569 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 06 13:50:23.084606 ip-10-0-136-25 kubenswrapper[2569]: W0706 13:50:23.073717 2569 feature_gate.go:328] unrecognized feature gate: Example2 Jul 06 13:50:23.085343 ip-10-0-136-25 kubenswrapper[2569]: W0706 13:50:23.073720 2569 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 06 13:50:23.085343 ip-10-0-136-25 kubenswrapper[2569]: W0706 13:50:23.073722 2569 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 06 13:50:23.085343 ip-10-0-136-25 kubenswrapper[2569]: W0706 13:50:23.073725 2569 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 06 13:50:23.085343 ip-10-0-136-25 kubenswrapper[2569]: W0706 13:50:23.073728 2569 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 06 13:50:23.085343 ip-10-0-136-25 kubenswrapper[2569]: W0706 13:50:23.073730 2569 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 06 13:50:23.085343 ip-10-0-136-25 kubenswrapper[2569]: W0706 13:50:23.073733 2569 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 06 13:50:23.085343 ip-10-0-136-25 kubenswrapper[2569]: W0706 13:50:23.073735 2569 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 06 13:50:23.085343 ip-10-0-136-25 kubenswrapper[2569]: W0706 13:50:23.073738 2569 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 06 13:50:23.085343 ip-10-0-136-25 kubenswrapper[2569]: W0706 13:50:23.073741 2569 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 06 13:50:23.085343 ip-10-0-136-25 kubenswrapper[2569]: W0706 13:50:23.073744 2569 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 06 13:50:23.085343 ip-10-0-136-25 kubenswrapper[2569]: W0706 13:50:23.073746 2569 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 06 13:50:23.085343 ip-10-0-136-25 kubenswrapper[2569]: W0706 13:50:23.073749 2569 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 06 13:50:23.085343 ip-10-0-136-25 kubenswrapper[2569]: W0706 13:50:23.073751 2569 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 06 13:50:23.085343 ip-10-0-136-25 kubenswrapper[2569]: W0706 13:50:23.073754 2569 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 06 13:50:23.085343 ip-10-0-136-25 kubenswrapper[2569]: W0706 13:50:23.073757 2569 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 06 13:50:23.085898 ip-10-0-136-25 kubenswrapper[2569]: I0706 13:50:23.073762 2569 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]} Jul 06 13:50:23.085898 ip-10-0-136-25 kubenswrapper[2569]: W0706 13:50:23.073894 2569 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 06 13:50:23.085898 ip-10-0-136-25 kubenswrapper[2569]: W0706 13:50:23.073900 2569 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 06 13:50:23.085898 ip-10-0-136-25 kubenswrapper[2569]: W0706 13:50:23.073903 2569 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 06 13:50:23.085898 ip-10-0-136-25 kubenswrapper[2569]: W0706 13:50:23.073906 2569 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 06 13:50:23.085898 ip-10-0-136-25 kubenswrapper[2569]: W0706 13:50:23.073908 2569 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 06 13:50:23.085898 ip-10-0-136-25 kubenswrapper[2569]: W0706 13:50:23.073911 2569 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 06 13:50:23.085898 ip-10-0-136-25 kubenswrapper[2569]: W0706 13:50:23.073914 2569 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 06 13:50:23.085898 ip-10-0-136-25 kubenswrapper[2569]: W0706 13:50:23.073917 2569 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 06 13:50:23.085898 ip-10-0-136-25 kubenswrapper[2569]: W0706 13:50:23.073920 2569 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 06 13:50:23.085898 ip-10-0-136-25 kubenswrapper[2569]: W0706 13:50:23.073922 2569 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 06 13:50:23.085898 ip-10-0-136-25 kubenswrapper[2569]: W0706 13:50:23.073925 2569 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 06 13:50:23.085898 ip-10-0-136-25 kubenswrapper[2569]: W0706 13:50:23.073928 2569 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 06 13:50:23.085898 ip-10-0-136-25 kubenswrapper[2569]: W0706 13:50:23.073931 2569 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 06 13:50:23.085898 ip-10-0-136-25 kubenswrapper[2569]: W0706 13:50:23.073934 2569 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 06 13:50:23.086538 ip-10-0-136-25 kubenswrapper[2569]: W0706 13:50:23.073936 2569 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 06 13:50:23.086538 ip-10-0-136-25 kubenswrapper[2569]: W0706 13:50:23.073939 2569 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 06 13:50:23.086538 ip-10-0-136-25 kubenswrapper[2569]: W0706 13:50:23.073942 2569 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 06 13:50:23.086538 ip-10-0-136-25 kubenswrapper[2569]: W0706 13:50:23.073944 2569 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 06 13:50:23.086538 ip-10-0-136-25 kubenswrapper[2569]: W0706 13:50:23.073947 2569 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 06 13:50:23.086538 ip-10-0-136-25 kubenswrapper[2569]: W0706 13:50:23.073950 2569 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 06 13:50:23.086538 ip-10-0-136-25 kubenswrapper[2569]: W0706 13:50:23.073952 2569 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 06 13:50:23.086538 ip-10-0-136-25 kubenswrapper[2569]: W0706 13:50:23.073955 2569 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 06 13:50:23.086538 ip-10-0-136-25 kubenswrapper[2569]: W0706 13:50:23.073957 2569 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 06 13:50:23.086538 ip-10-0-136-25 kubenswrapper[2569]: W0706 13:50:23.073960 2569 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 06 13:50:23.086538 ip-10-0-136-25 kubenswrapper[2569]: W0706 13:50:23.073963 2569 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 06 13:50:23.086538 ip-10-0-136-25 kubenswrapper[2569]: W0706 13:50:23.073965 2569 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 06 13:50:23.086538 ip-10-0-136-25 kubenswrapper[2569]: W0706 13:50:23.073968 2569 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 06 13:50:23.086538 ip-10-0-136-25 kubenswrapper[2569]: W0706 13:50:23.073970 2569 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 06 13:50:23.086538 ip-10-0-136-25 kubenswrapper[2569]: W0706 13:50:23.073973 2569 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 06 13:50:23.086538 ip-10-0-136-25 kubenswrapper[2569]: W0706 13:50:23.073975 2569 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 06 13:50:23.086538 ip-10-0-136-25 kubenswrapper[2569]: W0706 13:50:23.073978 2569 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 06 13:50:23.086538 ip-10-0-136-25 kubenswrapper[2569]: W0706 13:50:23.073980 2569 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 06 13:50:23.086538 ip-10-0-136-25 kubenswrapper[2569]: W0706 13:50:23.073983 2569 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 06 13:50:23.087246 ip-10-0-136-25 kubenswrapper[2569]: W0706 13:50:23.073985 2569 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 06 13:50:23.087246 ip-10-0-136-25 kubenswrapper[2569]: W0706 13:50:23.073988 2569 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 06 13:50:23.087246 ip-10-0-136-25 kubenswrapper[2569]: W0706 13:50:23.073990 2569 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 06 13:50:23.087246 ip-10-0-136-25 kubenswrapper[2569]: W0706 13:50:23.073993 2569 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 06 13:50:23.087246 ip-10-0-136-25 kubenswrapper[2569]: W0706 13:50:23.073995 2569 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 06 13:50:23.087246 ip-10-0-136-25 kubenswrapper[2569]: W0706 13:50:23.073998 2569 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 06 13:50:23.087246 ip-10-0-136-25 kubenswrapper[2569]: W0706 13:50:23.074000 2569 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 06 13:50:23.087246 ip-10-0-136-25 kubenswrapper[2569]: W0706 13:50:23.074003 2569 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 06 13:50:23.087246 ip-10-0-136-25 kubenswrapper[2569]: W0706 13:50:23.074006 2569 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 06 13:50:23.087246 ip-10-0-136-25 kubenswrapper[2569]: W0706 13:50:23.074009 2569 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 06 13:50:23.087246 ip-10-0-136-25 kubenswrapper[2569]: W0706 13:50:23.074012 2569 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 06 13:50:23.087246 ip-10-0-136-25 kubenswrapper[2569]: W0706 13:50:23.074016 2569 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 06 13:50:23.087246 ip-10-0-136-25 kubenswrapper[2569]: W0706 13:50:23.074020 2569 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 06 13:50:23.087246 ip-10-0-136-25 kubenswrapper[2569]: W0706 13:50:23.074024 2569 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 06 13:50:23.087246 ip-10-0-136-25 kubenswrapper[2569]: W0706 13:50:23.074026 2569 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 06 13:50:23.087246 ip-10-0-136-25 kubenswrapper[2569]: W0706 13:50:23.074029 2569 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 06 13:50:23.087246 ip-10-0-136-25 kubenswrapper[2569]: W0706 13:50:23.074032 2569 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 06 13:50:23.087246 ip-10-0-136-25 kubenswrapper[2569]: W0706 13:50:23.074035 2569 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 06 13:50:23.087246 ip-10-0-136-25 kubenswrapper[2569]: W0706 13:50:23.074038 2569 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 06 13:50:23.087931 ip-10-0-136-25 kubenswrapper[2569]: W0706 13:50:23.074041 2569 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 06 13:50:23.087931 ip-10-0-136-25 kubenswrapper[2569]: W0706 13:50:23.074044 2569 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 06 13:50:23.087931 ip-10-0-136-25 kubenswrapper[2569]: W0706 13:50:23.074046 2569 feature_gate.go:328] unrecognized feature gate: Example Jul 06 13:50:23.087931 ip-10-0-136-25 kubenswrapper[2569]: W0706 13:50:23.074049 2569 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 06 13:50:23.087931 ip-10-0-136-25 kubenswrapper[2569]: W0706 13:50:23.074052 2569 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 06 13:50:23.087931 ip-10-0-136-25 kubenswrapper[2569]: W0706 13:50:23.074055 2569 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 06 13:50:23.087931 ip-10-0-136-25 kubenswrapper[2569]: W0706 13:50:23.074057 2569 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 06 13:50:23.087931 ip-10-0-136-25 kubenswrapper[2569]: W0706 13:50:23.074076 2569 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 06 13:50:23.087931 ip-10-0-136-25 kubenswrapper[2569]: W0706 13:50:23.074080 2569 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 06 13:50:23.087931 ip-10-0-136-25 kubenswrapper[2569]: W0706 13:50:23.074082 2569 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 06 13:50:23.087931 ip-10-0-136-25 kubenswrapper[2569]: W0706 13:50:23.074085 2569 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 06 13:50:23.087931 ip-10-0-136-25 kubenswrapper[2569]: W0706 13:50:23.074088 2569 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 06 13:50:23.087931 ip-10-0-136-25 kubenswrapper[2569]: W0706 13:50:23.074091 2569 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 06 13:50:23.087931 ip-10-0-136-25 kubenswrapper[2569]: W0706 13:50:23.074093 2569 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 06 13:50:23.087931 ip-10-0-136-25 kubenswrapper[2569]: W0706 13:50:23.074096 2569 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 06 13:50:23.087931 ip-10-0-136-25 kubenswrapper[2569]: W0706 13:50:23.074098 2569 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 06 13:50:23.087931 ip-10-0-136-25 kubenswrapper[2569]: W0706 13:50:23.074101 2569 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 06 13:50:23.087931 ip-10-0-136-25 kubenswrapper[2569]: W0706 13:50:23.074104 2569 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 06 13:50:23.087931 ip-10-0-136-25 kubenswrapper[2569]: W0706 13:50:23.074106 2569 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 06 13:50:23.087931 ip-10-0-136-25 kubenswrapper[2569]: W0706 13:50:23.074109 2569 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 06 13:50:23.088649 ip-10-0-136-25 kubenswrapper[2569]: W0706 13:50:23.074111 2569 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 06 13:50:23.088649 ip-10-0-136-25 kubenswrapper[2569]: W0706 13:50:23.074114 2569 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 06 13:50:23.088649 ip-10-0-136-25 kubenswrapper[2569]: W0706 13:50:23.074117 2569 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 06 13:50:23.088649 ip-10-0-136-25 kubenswrapper[2569]: W0706 13:50:23.074120 2569 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 06 13:50:23.088649 ip-10-0-136-25 kubenswrapper[2569]: W0706 13:50:23.074122 2569 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 06 13:50:23.088649 ip-10-0-136-25 kubenswrapper[2569]: W0706 13:50:23.074125 2569 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 06 13:50:23.088649 ip-10-0-136-25 kubenswrapper[2569]: W0706 13:50:23.074131 2569 feature_gate.go:328] unrecognized feature gate: Example2 Jul 06 13:50:23.088649 ip-10-0-136-25 kubenswrapper[2569]: W0706 13:50:23.074135 2569 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 06 13:50:23.088649 ip-10-0-136-25 kubenswrapper[2569]: W0706 13:50:23.074138 2569 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 06 13:50:23.088649 ip-10-0-136-25 kubenswrapper[2569]: W0706 13:50:23.074141 2569 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 06 13:50:23.088649 ip-10-0-136-25 kubenswrapper[2569]: W0706 13:50:23.074144 2569 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 06 13:50:23.088649 ip-10-0-136-25 kubenswrapper[2569]: W0706 13:50:23.074146 2569 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 06 13:50:23.088649 ip-10-0-136-25 kubenswrapper[2569]: W0706 13:50:23.074149 2569 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 06 13:50:23.088649 ip-10-0-136-25 kubenswrapper[2569]: W0706 13:50:23.074151 2569 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 06 13:50:23.088649 ip-10-0-136-25 kubenswrapper[2569]: W0706 13:50:23.074155 2569 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 06 13:50:23.088649 ip-10-0-136-25 kubenswrapper[2569]: W0706 13:50:23.074158 2569 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 06 13:50:23.088649 ip-10-0-136-25 kubenswrapper[2569]: W0706 13:50:23.074161 2569 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 06 13:50:23.088649 ip-10-0-136-25 kubenswrapper[2569]: W0706 13:50:23.074164 2569 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 06 13:50:23.088649 ip-10-0-136-25 kubenswrapper[2569]: W0706 13:50:23.074167 2569 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 06 13:50:23.089436 ip-10-0-136-25 kubenswrapper[2569]: W0706 13:50:23.074170 2569 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 06 13:50:23.089436 ip-10-0-136-25 kubenswrapper[2569]: W0706 13:50:23.074173 2569 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 06 13:50:23.089436 ip-10-0-136-25 kubenswrapper[2569]: W0706 13:50:23.074175 2569 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 06 13:50:23.089436 ip-10-0-136-25 kubenswrapper[2569]: W0706 13:50:23.074178 2569 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 06 13:50:23.089436 ip-10-0-136-25 kubenswrapper[2569]: W0706 13:50:23.074180 2569 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 06 13:50:23.089436 ip-10-0-136-25 kubenswrapper[2569]: W0706 13:50:23.074183 2569 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 06 13:50:23.089436 ip-10-0-136-25 kubenswrapper[2569]: W0706 13:50:23.074186 2569 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 06 13:50:23.089436 ip-10-0-136-25 kubenswrapper[2569]: W0706 13:50:23.074188 2569 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 06 13:50:23.089436 ip-10-0-136-25 kubenswrapper[2569]: I0706 13:50:23.074193 2569 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]} Jul 06 13:50:23.089436 ip-10-0-136-25 kubenswrapper[2569]: I0706 13:50:23.074313 2569 server.go:962] "Client rotation is on, will bootstrap in background" Jul 06 13:50:23.089436 ip-10-0-136-25 kubenswrapper[2569]: I0706 13:50:23.078686 2569 bootstrap.go:101] "Use the bootstrap credentials to request a cert, and set kubeconfig to point to the certificate dir" Jul 06 13:50:23.089436 ip-10-0-136-25 kubenswrapper[2569]: I0706 13:50:23.080624 2569 server.go:1019] "Starting client certificate rotation" Jul 06 13:50:23.089436 ip-10-0-136-25 kubenswrapper[2569]: I0706 13:50:23.080738 2569 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet" Jul 06 13:50:23.089436 ip-10-0-136-25 kubenswrapper[2569]: I0706 13:50:23.080768 2569 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kube-apiserver-client-kubelet" Jul 06 13:50:23.089944 ip-10-0-136-25 kubenswrapper[2569]: I0706 13:50:23.083204 2569 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Jul 06 13:50:23.089944 ip-10-0-136-25 kubenswrapper[2569]: I0706 13:50:23.084585 2569 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Jul 06 13:50:23.099956 ip-10-0-136-25 kubenswrapper[2569]: I0706 13:50:23.099937 2569 log.go:25] "Validated CRI v1 runtime API" Jul 06 13:50:23.102207 ip-10-0-136-25 kubenswrapper[2569]: I0706 13:50:23.102194 2569 log.go:25] "Validated CRI v1 image API" Jul 06 13:50:23.104127 ip-10-0-136-25 kubenswrapper[2569]: I0706 13:50:23.104110 2569 server.go:1429] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd" Jul 06 13:50:23.107236 ip-10-0-136-25 kubenswrapper[2569]: I0706 13:50:23.107208 2569 fs.go:135] Filesystem UUIDs: map[387b3143-997f-497d-82e7-bb149a49d274:/dev/nvme0n1p3 7B77-95E7:/dev/nvme0n1p2 856ea776-e5d4-4bd7-8d31-86a2b2c3cdda:/dev/nvme0n1p4] Jul 06 13:50:23.107319 ip-10-0-136-25 kubenswrapper[2569]: I0706 13:50:23.107236 2569 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:22 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:24 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:31 fsType:tmpfs blockSize:0} composefs_0-33:{mountpoint:/ major:0 minor:33 fsType:overlay blockSize:0}] Jul 06 13:50:23.107391 ip-10-0-136-25 kubenswrapper[2569]: I0706 13:50:23.107375 2569 reflector.go:439] "Caches populated" logger="kubernetes.io/kube-apiserver-client-kubelet" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" Jul 06 13:50:23.113538 ip-10-0-136-25 kubenswrapper[2569]: I0706 13:50:23.113402 2569 manager.go:216] Machine: {Timestamp:2026-07-06 13:50:23.111917352 +0000 UTC m=+0.345626901 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3104848 MemoryCapacity:33164500992 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec2ecbb3ac12a1c6cf91922d1b432107 SystemUUID:ec2ecbb3-ac12-a1c6-cf91-922d1b432107 BootID:3f701203-2eb8-488f-8be5-e7d93553c040 Filesystems:[{Device:/dev/shm DeviceMajor:0 DeviceMinor:22 Capacity:16582250496 Type:vfs Inodes:4048401 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:24 Capacity:6632902656 Type:vfs Inodes:819200 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:composefs_0-33 DeviceMajor:0 DeviceMinor:33 Capacity:6107136 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:31 Capacity:16582250496 Type:vfs Inodes:1048576 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:br-ex MacAddress:02:4e:51:c2:e9:f9 Speed:0 Mtu:9001} {Name:ens5 MacAddress:02:4e:51:c2:e9:f9 Speed:0 Mtu:9001} {Name:ovs-system MacAddress:4e:d1:f0:42:43:47 Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:33164500992 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:37486592 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None} Jul 06 13:50:23.113611 ip-10-0-136-25 kubenswrapper[2569]: I0706 13:50:23.113538 2569 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available. Jul 06 13:50:23.113654 ip-10-0-136-25 kubenswrapper[2569]: I0706 13:50:23.113641 2569 manager.go:232] Version: {KernelVersion:5.14.0-570.125.1.el9_6.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.6.20260630-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:} Jul 06 13:50:23.117037 ip-10-0-136-25 kubenswrapper[2569]: I0706 13:50:23.117014 2569 container_manager_linux.go:273] "Container manager verified user specified cgroup-root exists" cgroupRoot=[] Jul 06 13:50:23.117741 ip-10-0-136-25 kubenswrapper[2569]: I0706 13:50:23.117137 2569 container_manager_linux.go:278] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-136-25.ec2.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2} Jul 06 13:50:23.118312 ip-10-0-136-25 kubenswrapper[2569]: I0706 13:50:23.118295 2569 topology_manager.go:138] "Creating topology manager with none policy" Jul 06 13:50:23.118402 ip-10-0-136-25 kubenswrapper[2569]: I0706 13:50:23.118394 2569 container_manager_linux.go:309] "Creating device plugin manager" Jul 06 13:50:23.118490 ip-10-0-136-25 kubenswrapper[2569]: I0706 13:50:23.118480 2569 manager.go:141] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock" Jul 06 13:50:23.119193 ip-10-0-136-25 kubenswrapper[2569]: I0706 13:50:23.119168 2569 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock" Jul 06 13:50:23.122199 ip-10-0-136-25 kubenswrapper[2569]: I0706 13:50:23.122184 2569 container_manager_linux.go:318] "Creating Dynamic Resource Allocation (DRA) manager" Jul 06 13:50:23.124864 ip-10-0-136-25 kubenswrapper[2569]: I0706 13:50:23.124850 2569 state_mem.go:36] "Initialized new in-memory state store" Jul 06 13:50:23.125005 ip-10-0-136-25 kubenswrapper[2569]: I0706 13:50:23.124994 2569 server.go:1240] "Using root directory" path="/var/lib/kubelet" Jul 06 13:50:23.125084 ip-10-0-136-25 kubenswrapper[2569]: I0706 13:50:23.125032 2569 kubelet.go:486] "Attempting to sync node with API server" Jul 06 13:50:23.125084 ip-10-0-136-25 kubenswrapper[2569]: I0706 13:50:23.125047 2569 kubelet.go:387] "Adding static pod path" path="/etc/kubernetes/manifests" Jul 06 13:50:23.125084 ip-10-0-136-25 kubenswrapper[2569]: I0706 13:50:23.125076 2569 file.go:69] "Watching path" path="/etc/kubernetes/manifests" Jul 06 13:50:23.125202 ip-10-0-136-25 kubenswrapper[2569]: I0706 13:50:23.125091 2569 kubelet.go:398] "Adding apiserver pod source" Jul 06 13:50:23.125202 ip-10-0-136-25 kubenswrapper[2569]: I0706 13:50:23.125104 2569 apiserver.go:42] "Waiting for node sync before watching apiserver pods" Jul 06 13:50:23.125578 ip-10-0-136-25 kubenswrapper[2569]: I0706 13:50:23.125564 2569 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Jul 06 13:50:23.125655 ip-10-0-136-25 kubenswrapper[2569]: I0706 13:50:23.125588 2569 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Jul 06 13:50:23.127735 ip-10-0-136-25 kubenswrapper[2569]: I0706 13:50:23.127717 2569 kuberuntime_manager.go:301] "Container runtime initialized" containerRuntime="cri-o" version="1.34.8-3.rhaos4.21.git91b4b0b.el9" apiVersion="v1" Jul 06 13:50:23.133181 ip-10-0-136-25 kubenswrapper[2569]: I0706 13:50:23.133163 2569 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-4kvx9" Jul 06 13:50:23.134835 ip-10-0-136-25 kubenswrapper[2569]: I0706 13:50:23.134820 2569 kubelet.go:960] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled" Jul 06 13:50:23.134912 ip-10-0-136-25 kubenswrapper[2569]: I0706 13:50:23.134850 2569 kubelet.go:984] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled" Jul 06 13:50:23.134976 ip-10-0-136-25 kubenswrapper[2569]: E0706 13:50:23.134953 2569 reflector.go:205] "Failed to watch" err="failed to list *v1.Service: services is forbidden: User \"system:anonymous\" cannot list resource \"services\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Service" Jul 06 13:50:23.135011 ip-10-0-136-25 kubenswrapper[2569]: E0706 13:50:23.134953 2569 reflector.go:205] "Failed to watch" err="failed to list *v1.Node: nodes \"ip-10-0-136-25.ec2.internal\" is forbidden: User \"system:anonymous\" cannot list resource \"nodes\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Node" Jul 06 13:50:23.136027 ip-10-0-136-25 kubenswrapper[2569]: I0706 13:50:23.136013 2569 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume" Jul 06 13:50:23.136106 ip-10-0-136-25 kubenswrapper[2569]: I0706 13:50:23.136033 2569 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir" Jul 06 13:50:23.136106 ip-10-0-136-25 kubenswrapper[2569]: I0706 13:50:23.136043 2569 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo" Jul 06 13:50:23.136106 ip-10-0-136-25 kubenswrapper[2569]: I0706 13:50:23.136051 2569 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path" Jul 06 13:50:23.136106 ip-10-0-136-25 kubenswrapper[2569]: I0706 13:50:23.136085 2569 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs" Jul 06 13:50:23.136106 ip-10-0-136-25 kubenswrapper[2569]: I0706 13:50:23.136095 2569 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret" Jul 06 13:50:23.136106 ip-10-0-136-25 kubenswrapper[2569]: I0706 13:50:23.136108 2569 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi" Jul 06 13:50:23.136305 ip-10-0-136-25 kubenswrapper[2569]: I0706 13:50:23.136116 2569 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api" Jul 06 13:50:23.136305 ip-10-0-136-25 kubenswrapper[2569]: I0706 13:50:23.136125 2569 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc" Jul 06 13:50:23.136305 ip-10-0-136-25 kubenswrapper[2569]: I0706 13:50:23.136135 2569 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap" Jul 06 13:50:23.136305 ip-10-0-136-25 kubenswrapper[2569]: I0706 13:50:23.136154 2569 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected" Jul 06 13:50:23.136305 ip-10-0-136-25 kubenswrapper[2569]: I0706 13:50:23.136162 2569 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume" Jul 06 13:50:23.136688 ip-10-0-136-25 kubenswrapper[2569]: I0706 13:50:23.136678 2569 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi" Jul 06 13:50:23.136739 ip-10-0-136-25 kubenswrapper[2569]: I0706 13:50:23.136690 2569 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image" Jul 06 13:50:23.139053 ip-10-0-136-25 kubenswrapper[2569]: I0706 13:50:23.139030 2569 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-136-25.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope Jul 06 13:50:23.139749 ip-10-0-136-25 kubenswrapper[2569]: I0706 13:50:23.139734 2569 server.go:1268] "Started kubelet" Jul 06 13:50:23.139938 ip-10-0-136-25 kubenswrapper[2569]: I0706 13:50:23.139845 2569 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10 Jul 06 13:50:23.140041 ip-10-0-136-25 kubenswrapper[2569]: I0706 13:50:23.140024 2569 server_v1.go:49] "podresources" method="list" useActivePods=true Jul 06 13:50:23.140402 ip-10-0-136-25 kubenswrapper[2569]: I0706 13:50:23.140373 2569 server.go:180] "Starting to listen" address="0.0.0.0" port=10250 Jul 06 13:50:23.140849 ip-10-0-136-25 systemd[1]: Started Kubernetes Kubelet. Jul 06 13:50:23.141217 ip-10-0-136-25 kubenswrapper[2569]: I0706 13:50:23.141195 2569 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-4kvx9" Jul 06 13:50:23.142256 ip-10-0-136-25 kubenswrapper[2569]: I0706 13:50:23.142241 2569 server.go:310] "Adding debug handlers to kubelet server" Jul 06 13:50:23.144443 ip-10-0-136-25 kubenswrapper[2569]: I0706 13:50:23.144427 2569 server.go:249] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock" Jul 06 13:50:23.146846 ip-10-0-136-25 kubenswrapper[2569]: E0706 13:50:23.146831 2569 kubelet.go:1638] "Image garbage collection failed once. Stats initialization may not have completed yet" err="failed to get imageFs info: unable to find data in memory cache" Jul 06 13:50:23.151301 ip-10-0-136-25 kubenswrapper[2569]: I0706 13:50:23.151284 2569 fs_resource_analyzer.go:67] "Starting FS ResourceAnalyzer" Jul 06 13:50:23.151301 ip-10-0-136-25 kubenswrapper[2569]: I0706 13:50:23.151300 2569 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving" Jul 06 13:50:23.151785 ip-10-0-136-25 kubenswrapper[2569]: E0706 13:50:23.151519 2569 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-136-25.ec2.internal\" not found" Jul 06 13:50:23.151890 ip-10-0-136-25 kubenswrapper[2569]: I0706 13:50:23.151876 2569 desired_state_of_world_populator.go:146] "Desired state populator starts to run" Jul 06 13:50:23.151950 ip-10-0-136-25 kubenswrapper[2569]: I0706 13:50:23.151877 2569 volume_manager.go:311] "The desired_state_of_world populator starts" Jul 06 13:50:23.151950 ip-10-0-136-25 kubenswrapper[2569]: I0706 13:50:23.151918 2569 volume_manager.go:313] "Starting Kubelet Volume Manager" Jul 06 13:50:23.152110 ip-10-0-136-25 kubenswrapper[2569]: I0706 13:50:23.152094 2569 reconstruct.go:97] "Volume reconstruction finished" Jul 06 13:50:23.152110 ip-10-0-136-25 kubenswrapper[2569]: I0706 13:50:23.152106 2569 reconciler.go:29] "Reconciler: start to sync state" Jul 06 13:50:23.152993 ip-10-0-136-25 kubenswrapper[2569]: I0706 13:50:23.152960 2569 factory.go:153] Registering CRI-O factory Jul 06 13:50:23.152993 ip-10-0-136-25 kubenswrapper[2569]: I0706 13:50:23.152992 2569 factory.go:223] Registration of the crio container factory successfully Jul 06 13:50:23.153145 ip-10-0-136-25 kubenswrapper[2569]: I0706 13:50:23.153048 2569 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory Jul 06 13:50:23.153145 ip-10-0-136-25 kubenswrapper[2569]: I0706 13:50:23.153089 2569 factory.go:55] Registering systemd factory Jul 06 13:50:23.153145 ip-10-0-136-25 kubenswrapper[2569]: I0706 13:50:23.153097 2569 factory.go:223] Registration of the systemd container factory successfully Jul 06 13:50:23.153145 ip-10-0-136-25 kubenswrapper[2569]: I0706 13:50:23.153119 2569 factory.go:103] Registering Raw factory Jul 06 13:50:23.153145 ip-10-0-136-25 kubenswrapper[2569]: I0706 13:50:23.153133 2569 manager.go:1196] Started watching for new ooms in manager Jul 06 13:50:23.153832 ip-10-0-136-25 kubenswrapper[2569]: I0706 13:50:23.153813 2569 manager.go:349] Starting recovery of all containers Jul 06 13:50:23.156576 ip-10-0-136-25 kubenswrapper[2569]: I0706 13:50:23.156553 2569 reflector.go:439] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:160" Jul 06 13:50:23.160346 ip-10-0-136-25 kubenswrapper[2569]: E0706 13:50:23.160319 2569 nodelease.go:49] "Failed to get node when trying to set owner ref to the node lease" err="nodes \"ip-10-0-136-25.ec2.internal\" not found" node="ip-10-0-136-25.ec2.internal" Jul 06 13:50:23.165283 ip-10-0-136-25 kubenswrapper[2569]: I0706 13:50:23.165269 2569 manager.go:354] Recovery completed Jul 06 13:50:23.169584 ip-10-0-136-25 kubenswrapper[2569]: I0706 13:50:23.169571 2569 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 06 13:50:23.171342 ip-10-0-136-25 kubenswrapper[2569]: I0706 13:50:23.171329 2569 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-136-25.ec2.internal" event="NodeHasSufficientMemory" Jul 06 13:50:23.171394 ip-10-0-136-25 kubenswrapper[2569]: I0706 13:50:23.171356 2569 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-136-25.ec2.internal" event="NodeHasNoDiskPressure" Jul 06 13:50:23.171394 ip-10-0-136-25 kubenswrapper[2569]: I0706 13:50:23.171365 2569 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-136-25.ec2.internal" event="NodeHasSufficientPID" Jul 06 13:50:23.171871 ip-10-0-136-25 kubenswrapper[2569]: I0706 13:50:23.171860 2569 cpu_manager.go:222] "Starting CPU manager" policy="none" Jul 06 13:50:23.171871 ip-10-0-136-25 kubenswrapper[2569]: I0706 13:50:23.171868 2569 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 06 13:50:23.171871 ip-10-0-136-25 kubenswrapper[2569]: I0706 13:50:23.171871 2569 cpu_manager.go:223] "Reconciling" reconcilePeriod="10s" Jul 06 13:50:23.172016 ip-10-0-136-25 kubenswrapper[2569]: I0706 13:50:23.171887 2569 state_mem.go:36] "Initialized new in-memory state store" Jul 06 13:50:23.172524 ip-10-0-136-25 kubenswrapper[2569]: I0706 13:50:23.172510 2569 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-136-25.ec2.internal" event="NodeHasSufficientMemory" Jul 06 13:50:23.172590 ip-10-0-136-25 kubenswrapper[2569]: I0706 13:50:23.172533 2569 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-136-25.ec2.internal" event="NodeHasNoDiskPressure" Jul 06 13:50:23.172590 ip-10-0-136-25 kubenswrapper[2569]: I0706 13:50:23.172542 2569 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-136-25.ec2.internal" event="NodeHasSufficientPID" Jul 06 13:50:23.173056 ip-10-0-136-25 kubenswrapper[2569]: I0706 13:50:23.173045 2569 policy_none.go:49] "None policy: Start" Jul 06 13:50:23.173112 ip-10-0-136-25 kubenswrapper[2569]: I0706 13:50:23.173076 2569 memory_manager.go:187] "Starting memorymanager" policy="None" Jul 06 13:50:23.173112 ip-10-0-136-25 kubenswrapper[2569]: I0706 13:50:23.173088 2569 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint" Jul 06 13:50:23.174095 ip-10-0-136-25 kubenswrapper[2569]: I0706 13:50:23.174084 2569 policy_none.go:47] "Start" Jul 06 13:50:23.210113 ip-10-0-136-25 kubenswrapper[2569]: I0706 13:50:23.209914 2569 manager.go:337] "Starting Device Plugin manager" Jul 06 13:50:23.222773 ip-10-0-136-25 kubenswrapper[2569]: E0706 13:50:23.210140 2569 manager.go:513] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint" Jul 06 13:50:23.222773 ip-10-0-136-25 kubenswrapper[2569]: I0706 13:50:23.210155 2569 server.go:87] "Starting device plugin registration server" Jul 06 13:50:23.222773 ip-10-0-136-25 kubenswrapper[2569]: I0706 13:50:23.210423 2569 eviction_manager.go:189] "Eviction manager: starting control loop" Jul 06 13:50:23.222773 ip-10-0-136-25 kubenswrapper[2569]: I0706 13:50:23.210437 2569 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s" Jul 06 13:50:23.222773 ip-10-0-136-25 kubenswrapper[2569]: I0706 13:50:23.210531 2569 plugin_watcher.go:51] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry" Jul 06 13:50:23.222773 ip-10-0-136-25 kubenswrapper[2569]: I0706 13:50:23.210611 2569 plugin_manager.go:116] "The desired_state_of_world populator (plugin watcher) starts" Jul 06 13:50:23.222773 ip-10-0-136-25 kubenswrapper[2569]: I0706 13:50:23.210619 2569 plugin_manager.go:118] "Starting Kubelet Plugin Manager" Jul 06 13:50:23.222773 ip-10-0-136-25 kubenswrapper[2569]: E0706 13:50:23.211240 2569 eviction_manager.go:267] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\"" Jul 06 13:50:23.222773 ip-10-0-136-25 kubenswrapper[2569]: E0706 13:50:23.211281 2569 eviction_manager.go:292] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-136-25.ec2.internal\" not found" Jul 06 13:50:23.289189 ip-10-0-136-25 kubenswrapper[2569]: I0706 13:50:23.289149 2569 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4" Jul 06 13:50:23.290428 ip-10-0-136-25 kubenswrapper[2569]: I0706 13:50:23.290409 2569 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6" Jul 06 13:50:23.290521 ip-10-0-136-25 kubenswrapper[2569]: I0706 13:50:23.290455 2569 status_manager.go:244] "Starting to sync pod status with apiserver" Jul 06 13:50:23.290521 ip-10-0-136-25 kubenswrapper[2569]: I0706 13:50:23.290485 2569 kubelet.go:2448] "Starting kubelet main sync loop" Jul 06 13:50:23.290610 ip-10-0-136-25 kubenswrapper[2569]: E0706 13:50:23.290528 2569 kubelet.go:2472] "Skipping pod synchronization" err="PLEG is not healthy: pleg has yet to be successful" Jul 06 13:50:23.294345 ip-10-0-136-25 kubenswrapper[2569]: I0706 13:50:23.294328 2569 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160" Jul 06 13:50:23.310924 ip-10-0-136-25 kubenswrapper[2569]: I0706 13:50:23.310875 2569 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 06 13:50:23.311761 ip-10-0-136-25 kubenswrapper[2569]: I0706 13:50:23.311746 2569 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-136-25.ec2.internal" event="NodeHasSufficientMemory" Jul 06 13:50:23.311848 ip-10-0-136-25 kubenswrapper[2569]: I0706 13:50:23.311773 2569 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-136-25.ec2.internal" event="NodeHasNoDiskPressure" Jul 06 13:50:23.311848 ip-10-0-136-25 kubenswrapper[2569]: I0706 13:50:23.311784 2569 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-136-25.ec2.internal" event="NodeHasSufficientPID" Jul 06 13:50:23.311848 ip-10-0-136-25 kubenswrapper[2569]: I0706 13:50:23.311808 2569 kubelet_node_status.go:78] "Attempting to register node" node="ip-10-0-136-25.ec2.internal" Jul 06 13:50:23.322550 ip-10-0-136-25 kubenswrapper[2569]: I0706 13:50:23.322536 2569 kubelet_node_status.go:81] "Successfully registered node" node="ip-10-0-136-25.ec2.internal" Jul 06 13:50:23.322614 ip-10-0-136-25 kubenswrapper[2569]: E0706 13:50:23.322558 2569 kubelet_node_status.go:536] "Error updating node status, will retry" err="error getting node \"ip-10-0-136-25.ec2.internal\": node \"ip-10-0-136-25.ec2.internal\" not found" Jul 06 13:50:23.334535 ip-10-0-136-25 kubenswrapper[2569]: E0706 13:50:23.334512 2569 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-136-25.ec2.internal\" not found" Jul 06 13:50:23.390672 ip-10-0-136-25 kubenswrapper[2569]: I0706 13:50:23.390628 2569 kubelet.go:2535] "SyncLoop ADD" source="file" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-136-25.ec2.internal","kube-system/kube-apiserver-proxy-ip-10-0-136-25.ec2.internal"] Jul 06 13:50:23.390809 ip-10-0-136-25 kubenswrapper[2569]: I0706 13:50:23.390728 2569 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 06 13:50:23.391735 ip-10-0-136-25 kubenswrapper[2569]: I0706 13:50:23.391719 2569 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-136-25.ec2.internal" event="NodeHasSufficientMemory" Jul 06 13:50:23.391819 ip-10-0-136-25 kubenswrapper[2569]: I0706 13:50:23.391753 2569 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-136-25.ec2.internal" event="NodeHasNoDiskPressure" Jul 06 13:50:23.391819 ip-10-0-136-25 kubenswrapper[2569]: I0706 13:50:23.391767 2569 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-136-25.ec2.internal" event="NodeHasSufficientPID" Jul 06 13:50:23.393159 ip-10-0-136-25 kubenswrapper[2569]: I0706 13:50:23.393145 2569 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 06 13:50:23.393272 ip-10-0-136-25 kubenswrapper[2569]: I0706 13:50:23.393247 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-136-25.ec2.internal" Jul 06 13:50:23.393272 ip-10-0-136-25 kubenswrapper[2569]: I0706 13:50:23.393273 2569 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 06 13:50:23.393862 ip-10-0-136-25 kubenswrapper[2569]: I0706 13:50:23.393846 2569 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-136-25.ec2.internal" event="NodeHasSufficientMemory" Jul 06 13:50:23.393932 ip-10-0-136-25 kubenswrapper[2569]: I0706 13:50:23.393881 2569 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-136-25.ec2.internal" event="NodeHasNoDiskPressure" Jul 06 13:50:23.393932 ip-10-0-136-25 kubenswrapper[2569]: I0706 13:50:23.393890 2569 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-136-25.ec2.internal" event="NodeHasSufficientPID" Jul 06 13:50:23.393932 ip-10-0-136-25 kubenswrapper[2569]: I0706 13:50:23.393846 2569 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-136-25.ec2.internal" event="NodeHasSufficientMemory" Jul 06 13:50:23.394019 ip-10-0-136-25 kubenswrapper[2569]: I0706 13:50:23.393940 2569 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-136-25.ec2.internal" event="NodeHasNoDiskPressure" Jul 06 13:50:23.394019 ip-10-0-136-25 kubenswrapper[2569]: I0706 13:50:23.393953 2569 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-136-25.ec2.internal" event="NodeHasSufficientPID" Jul 06 13:50:23.395155 ip-10-0-136-25 kubenswrapper[2569]: I0706 13:50:23.395136 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-136-25.ec2.internal" Jul 06 13:50:23.395233 ip-10-0-136-25 kubenswrapper[2569]: I0706 13:50:23.395163 2569 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 06 13:50:23.395878 ip-10-0-136-25 kubenswrapper[2569]: I0706 13:50:23.395861 2569 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-136-25.ec2.internal" event="NodeHasSufficientMemory" Jul 06 13:50:23.395941 ip-10-0-136-25 kubenswrapper[2569]: I0706 13:50:23.395886 2569 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-136-25.ec2.internal" event="NodeHasNoDiskPressure" Jul 06 13:50:23.395941 ip-10-0-136-25 kubenswrapper[2569]: I0706 13:50:23.395896 2569 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-136-25.ec2.internal" event="NodeHasSufficientPID" Jul 06 13:50:23.420753 ip-10-0-136-25 kubenswrapper[2569]: E0706 13:50:23.420731 2569 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-136-25.ec2.internal\" not found" node="ip-10-0-136-25.ec2.internal" Jul 06 13:50:23.424992 ip-10-0-136-25 kubenswrapper[2569]: E0706 13:50:23.424976 2569 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-136-25.ec2.internal\" not found" node="ip-10-0-136-25.ec2.internal" Jul 06 13:50:23.434678 ip-10-0-136-25 kubenswrapper[2569]: E0706 13:50:23.434659 2569 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-136-25.ec2.internal\" not found" Jul 06 13:50:23.454244 ip-10-0-136-25 kubenswrapper[2569]: I0706 13:50:23.454222 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/2a2f7f2a1d78cf5fb862a37fcc75004a-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-136-25.ec2.internal\" (UID: \"2a2f7f2a1d78cf5fb862a37fcc75004a\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-136-25.ec2.internal" Jul 06 13:50:23.454309 ip-10-0-136-25 kubenswrapper[2569]: I0706 13:50:23.454249 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/2a2f7f2a1d78cf5fb862a37fcc75004a-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-136-25.ec2.internal\" (UID: \"2a2f7f2a1d78cf5fb862a37fcc75004a\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-136-25.ec2.internal" Jul 06 13:50:23.454309 ip-10-0-136-25 kubenswrapper[2569]: I0706 13:50:23.454267 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/ba668ff1926990fccf3f73e0625e305c-config\") pod \"kube-apiserver-proxy-ip-10-0-136-25.ec2.internal\" (UID: \"ba668ff1926990fccf3f73e0625e305c\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-136-25.ec2.internal" Jul 06 13:50:23.535177 ip-10-0-136-25 kubenswrapper[2569]: E0706 13:50:23.535147 2569 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-136-25.ec2.internal\" not found" Jul 06 13:50:23.554510 ip-10-0-136-25 kubenswrapper[2569]: I0706 13:50:23.554482 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/2a2f7f2a1d78cf5fb862a37fcc75004a-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-136-25.ec2.internal\" (UID: \"2a2f7f2a1d78cf5fb862a37fcc75004a\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-136-25.ec2.internal" Jul 06 13:50:23.554601 ip-10-0-136-25 kubenswrapper[2569]: I0706 13:50:23.554511 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/2a2f7f2a1d78cf5fb862a37fcc75004a-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-136-25.ec2.internal\" (UID: \"2a2f7f2a1d78cf5fb862a37fcc75004a\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-136-25.ec2.internal" Jul 06 13:50:23.554601 ip-10-0-136-25 kubenswrapper[2569]: I0706 13:50:23.554530 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/ba668ff1926990fccf3f73e0625e305c-config\") pod \"kube-apiserver-proxy-ip-10-0-136-25.ec2.internal\" (UID: \"ba668ff1926990fccf3f73e0625e305c\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-136-25.ec2.internal" Jul 06 13:50:23.554601 ip-10-0-136-25 kubenswrapper[2569]: I0706 13:50:23.554581 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/2a2f7f2a1d78cf5fb862a37fcc75004a-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-136-25.ec2.internal\" (UID: \"2a2f7f2a1d78cf5fb862a37fcc75004a\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-136-25.ec2.internal" Jul 06 13:50:23.554697 ip-10-0-136-25 kubenswrapper[2569]: I0706 13:50:23.554645 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/ba668ff1926990fccf3f73e0625e305c-config\") pod \"kube-apiserver-proxy-ip-10-0-136-25.ec2.internal\" (UID: \"ba668ff1926990fccf3f73e0625e305c\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-136-25.ec2.internal" Jul 06 13:50:23.554697 ip-10-0-136-25 kubenswrapper[2569]: I0706 13:50:23.554672 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/2a2f7f2a1d78cf5fb862a37fcc75004a-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-136-25.ec2.internal\" (UID: \"2a2f7f2a1d78cf5fb862a37fcc75004a\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-136-25.ec2.internal" Jul 06 13:50:25.279100 ip-10-0-136-25 kubenswrapper[2569]: E0706 13:50:25.279069 2569 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-136-25.ec2.internal\" not found" Jul 06 13:50:25.367587 ip-10-0-136-25 kubenswrapper[2569]: I0706 13:50:25.367535 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-136-25.ec2.internal" Jul 06 13:50:25.372129 ip-10-0-136-25 kubenswrapper[2569]: I0706 13:50:25.371745 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-136-25.ec2.internal" Jul 06 13:50:25.379485 ip-10-0-136-25 kubenswrapper[2569]: E0706 13:50:25.379456 2569 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-136-25.ec2.internal\" not found" Jul 06 13:50:25.480006 ip-10-0-136-25 kubenswrapper[2569]: E0706 13:50:25.479967 2569 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-136-25.ec2.internal\" not found" Jul 06 13:50:25.580549 ip-10-0-136-25 kubenswrapper[2569]: E0706 13:50:25.580448 2569 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-136-25.ec2.internal\" not found" Jul 06 13:50:25.680981 ip-10-0-136-25 kubenswrapper[2569]: E0706 13:50:25.680949 2569 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-136-25.ec2.internal\" not found" Jul 06 13:50:25.700544 ip-10-0-136-25 kubenswrapper[2569]: I0706 13:50:25.700521 2569 reflector.go:439] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:160" Jul 06 13:50:25.723160 ip-10-0-136-25 kubenswrapper[2569]: I0706 13:50:25.723123 2569 transport.go:147] "Certificate rotation detected, shutting down client connections to start using new credentials" Jul 06 13:50:25.723316 ip-10-0-136-25 kubenswrapper[2569]: I0706 13:50:25.723295 2569 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Service" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received" Jul 06 13:50:25.723375 ip-10-0-136-25 kubenswrapper[2569]: I0706 13:50:25.723306 2569 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.RuntimeClass" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received" Jul 06 13:50:25.723375 ip-10-0-136-25 kubenswrapper[2569]: I0706 13:50:25.723297 2569 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.CSIDriver" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received" Jul 06 13:50:25.781618 ip-10-0-136-25 kubenswrapper[2569]: E0706 13:50:25.781591 2569 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-136-25.ec2.internal\" not found" Jul 06 13:50:25.792309 ip-10-0-136-25 kubenswrapper[2569]: I0706 13:50:25.792261 2569 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-07-05 13:45:24 +0000 UTC" deadline="2028-04-10 20:32:40.240884222 +0000 UTC" Jul 06 13:50:25.792309 ip-10-0-136-25 kubenswrapper[2569]: I0706 13:50:25.792297 2569 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="15462h42m14.448591429s" Jul 06 13:50:25.882616 ip-10-0-136-25 kubenswrapper[2569]: E0706 13:50:25.882576 2569 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-136-25.ec2.internal\" not found" Jul 06 13:50:25.890699 ip-10-0-136-25 kubenswrapper[2569]: W0706 13:50:25.890663 2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podba668ff1926990fccf3f73e0625e305c.slice/crio-17f9f5021cc4c8ac61666438c06cc197f59797116b4bb2ddae1b987684fe642b WatchSource:0}: Error finding container 17f9f5021cc4c8ac61666438c06cc197f59797116b4bb2ddae1b987684fe642b: Status 404 returned error can't find the container with id 17f9f5021cc4c8ac61666438c06cc197f59797116b4bb2ddae1b987684fe642b Jul 06 13:50:25.890975 ip-10-0-136-25 kubenswrapper[2569]: W0706 13:50:25.890954 2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2a2f7f2a1d78cf5fb862a37fcc75004a.slice/crio-64cf32c96cf3f0ffe18cb482024f7f914895901fc361d90121fc62b8048b831c WatchSource:0}: Error finding container 64cf32c96cf3f0ffe18cb482024f7f914895901fc361d90121fc62b8048b831c: Status 404 returned error can't find the container with id 64cf32c96cf3f0ffe18cb482024f7f914895901fc361d90121fc62b8048b831c Jul 06 13:50:25.895114 ip-10-0-136-25 kubenswrapper[2569]: I0706 13:50:25.895100 2569 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Jul 06 13:50:25.937319 ip-10-0-136-25 kubenswrapper[2569]: I0706 13:50:25.937263 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-136-25.ec2.internal" event={"ID":"2a2f7f2a1d78cf5fb862a37fcc75004a","Type":"ContainerStarted","Data":"64cf32c96cf3f0ffe18cb482024f7f914895901fc361d90121fc62b8048b831c"} Jul 06 13:50:25.938388 ip-10-0-136-25 kubenswrapper[2569]: I0706 13:50:25.938367 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-136-25.ec2.internal" event={"ID":"ba668ff1926990fccf3f73e0625e305c","Type":"ContainerStarted","Data":"17f9f5021cc4c8ac61666438c06cc197f59797116b4bb2ddae1b987684fe642b"} Jul 06 13:50:25.971241 ip-10-0-136-25 kubenswrapper[2569]: I0706 13:50:25.971054 2569 reflector.go:439] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:160" Jul 06 13:50:25.995419 ip-10-0-136-25 kubenswrapper[2569]: I0706 13:50:25.995393 2569 kubelet.go:3248] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-136-25.ec2.internal" Jul 06 13:50:26.003497 ip-10-0-136-25 kubenswrapper[2569]: I0706 13:50:26.003478 2569 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Jul 06 13:50:26.004480 ip-10-0-136-25 kubenswrapper[2569]: I0706 13:50:26.004465 2569 kubelet.go:3248] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-136-25.ec2.internal" Jul 06 13:50:26.027015 ip-10-0-136-25 kubenswrapper[2569]: I0706 13:50:26.026991 2569 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Jul 06 13:50:26.723282 ip-10-0-136-25 kubenswrapper[2569]: I0706 13:50:26.723243 2569 reflector.go:439] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:160" Jul 06 13:50:26.730921 ip-10-0-136-25 kubenswrapper[2569]: I0706 13:50:26.730887 2569 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160" Jul 06 13:50:26.770123 ip-10-0-136-25 kubenswrapper[2569]: I0706 13:50:26.770089 2569 apiserver.go:52] "Watching apiserver" Jul 06 13:50:26.776223 ip-10-0-136-25 kubenswrapper[2569]: I0706 13:50:26.776175 2569 reflector.go:439] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66" Jul 06 13:50:26.776658 ip-10-0-136-25 kubenswrapper[2569]: I0706 13:50:26.776637 2569 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ovn-kubernetes/ovnkube-node-vcxnc","kube-system/global-pull-secret-syncer-cg8ph","kube-system/konnectivity-agent-vqmcq","openshift-cluster-node-tuning-operator/tuned-2x29s","openshift-multus/multus-additional-cni-plugins-brmhg","openshift-multus/multus-dcdgc","openshift-network-diagnostics/network-check-target-gbq5d","openshift-network-operator/iptables-alerter-wxxjp","kube-system/kube-apiserver-proxy-ip-10-0-136-25.ec2.internal","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-2dnkc","openshift-dns/node-resolver-54w6r","openshift-image-registry/node-ca-c9sqv","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-136-25.ec2.internal","openshift-multus/network-metrics-daemon-hqt9l"] Jul 06 13:50:26.779359 ip-10-0-136-25 kubenswrapper[2569]: I0706 13:50:26.779335 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-vcxnc" Jul 06 13:50:26.781573 ip-10-0-136-25 kubenswrapper[2569]: I0706 13:50:26.781508 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-c9sqv" Jul 06 13:50:26.782150 ip-10-0-136-25 kubenswrapper[2569]: I0706 13:50:26.781961 2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\"" Jul 06 13:50:26.782150 ip-10-0-136-25 kubenswrapper[2569]: I0706 13:50:26.782003 2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\"" Jul 06 13:50:26.782150 ip-10-0-136-25 kubenswrapper[2569]: I0706 13:50:26.782018 2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\"" Jul 06 13:50:26.782150 ip-10-0-136-25 kubenswrapper[2569]: I0706 13:50:26.782073 2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\"" Jul 06 13:50:26.782484 ip-10-0-136-25 kubenswrapper[2569]: I0706 13:50:26.782333 2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\"" Jul 06 13:50:26.782541 ip-10-0-136-25 kubenswrapper[2569]: I0706 13:50:26.782499 2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\"" Jul 06 13:50:26.782725 ip-10-0-136-25 kubenswrapper[2569]: I0706 13:50:26.782679 2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-8f48c\"" Jul 06 13:50:26.783631 ip-10-0-136-25 kubenswrapper[2569]: I0706 13:50:26.783443 2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\"" Jul 06 13:50:26.783631 ip-10-0-136-25 kubenswrapper[2569]: I0706 13:50:26.783509 2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\"" Jul 06 13:50:26.784069 ip-10-0-136-25 kubenswrapper[2569]: I0706 13:50:26.783900 2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\"" Jul 06 13:50:26.784286 ip-10-0-136-25 kubenswrapper[2569]: I0706 13:50:26.784262 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-vqmcq" Jul 06 13:50:26.784667 ip-10-0-136-25 kubenswrapper[2569]: I0706 13:50:26.784647 2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-pl4rx\"" Jul 06 13:50:26.785999 ip-10-0-136-25 kubenswrapper[2569]: I0706 13:50:26.785980 2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\"" Jul 06 13:50:26.786703 ip-10-0-136-25 kubenswrapper[2569]: I0706 13:50:26.786413 2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"default-dockercfg-8gfj7\"" Jul 06 13:50:26.786703 ip-10-0-136-25 kubenswrapper[2569]: I0706 13:50:26.786460 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-gbq5d" Jul 06 13:50:26.786703 ip-10-0-136-25 kubenswrapper[2569]: I0706 13:50:26.786504 2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\"" Jul 06 13:50:26.786703 ip-10-0-136-25 kubenswrapper[2569]: E0706 13:50:26.786551 2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-gbq5d" podUID="d2d86b02-1e00-4267-b9c4-dd365b15342e" Jul 06 13:50:26.788808 ip-10-0-136-25 kubenswrapper[2569]: I0706 13:50:26.788771 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-brmhg" Jul 06 13:50:26.791200 ip-10-0-136-25 kubenswrapper[2569]: I0706 13:50:26.791180 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-dcdgc" Jul 06 13:50:26.792329 ip-10-0-136-25 kubenswrapper[2569]: I0706 13:50:26.792288 2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\"" Jul 06 13:50:26.793240 ip-10-0-136-25 kubenswrapper[2569]: I0706 13:50:26.793191 2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\"" Jul 06 13:50:26.793240 ip-10-0-136-25 kubenswrapper[2569]: I0706 13:50:26.793223 2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\"" Jul 06 13:50:26.793437 ip-10-0-136-25 kubenswrapper[2569]: I0706 13:50:26.793421 2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\"" Jul 06 13:50:26.794288 ip-10-0-136-25 kubenswrapper[2569]: I0706 13:50:26.793707 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-2x29s" Jul 06 13:50:26.794288 ip-10-0-136-25 kubenswrapper[2569]: I0706 13:50:26.794044 2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\"" Jul 06 13:50:26.794288 ip-10-0-136-25 kubenswrapper[2569]: I0706 13:50:26.794278 2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"default-dockercfg-vbtx2\"" Jul 06 13:50:26.798064 ip-10-0-136-25 kubenswrapper[2569]: I0706 13:50:26.795493 2569 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kubelet-serving" Jul 06 13:50:26.798064 ip-10-0-136-25 kubenswrapper[2569]: I0706 13:50:26.796842 2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\"" Jul 06 13:50:26.798064 ip-10-0-136-25 kubenswrapper[2569]: I0706 13:50:26.796911 2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-g67wq\"" Jul 06 13:50:26.798064 ip-10-0-136-25 kubenswrapper[2569]: I0706 13:50:26.797506 2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-sldrn\"" Jul 06 13:50:26.798064 ip-10-0-136-25 kubenswrapper[2569]: I0706 13:50:26.797819 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-wxxjp" Jul 06 13:50:26.798064 ip-10-0-136-25 kubenswrapper[2569]: I0706 13:50:26.798040 2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\"" Jul 06 13:50:26.798408 ip-10-0-136-25 kubenswrapper[2569]: I0706 13:50:26.798108 2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\"" Jul 06 13:50:26.800199 ip-10-0-136-25 kubenswrapper[2569]: I0706 13:50:26.800100 2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\"" Jul 06 13:50:26.800316 ip-10-0-136-25 kubenswrapper[2569]: I0706 13:50:26.800236 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-2dnkc" Jul 06 13:50:26.801509 ip-10-0-136-25 kubenswrapper[2569]: I0706 13:50:26.801483 2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\"" Jul 06 13:50:26.801691 ip-10-0-136-25 kubenswrapper[2569]: I0706 13:50:26.801524 2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-5wrgx\"" Jul 06 13:50:26.801802 ip-10-0-136-25 kubenswrapper[2569]: I0706 13:50:26.801762 2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\"" Jul 06 13:50:26.803181 ip-10-0-136-25 kubenswrapper[2569]: I0706 13:50:26.803162 2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-d922l\"" Jul 06 13:50:26.803509 ip-10-0-136-25 kubenswrapper[2569]: I0706 13:50:26.803492 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-54w6r" Jul 06 13:50:26.804128 ip-10-0-136-25 kubenswrapper[2569]: I0706 13:50:26.804106 2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\"" Jul 06 13:50:26.804128 ip-10-0-136-25 kubenswrapper[2569]: I0706 13:50:26.804125 2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\"" Jul 06 13:50:26.804303 ip-10-0-136-25 kubenswrapper[2569]: I0706 13:50:26.804201 2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\"" Jul 06 13:50:26.805699 ip-10-0-136-25 kubenswrapper[2569]: I0706 13:50:26.805683 2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\"" Jul 06 13:50:26.805834 ip-10-0-136-25 kubenswrapper[2569]: I0706 13:50:26.805725 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/4ae58dad-494b-4508-b5a5-129f13818d53-run-ovn\") pod \"ovnkube-node-vcxnc\" (UID: \"4ae58dad-494b-4508-b5a5-129f13818d53\") " pod="openshift-ovn-kubernetes/ovnkube-node-vcxnc" Jul 06 13:50:26.805834 ip-10-0-136-25 kubenswrapper[2569]: I0706 13:50:26.805758 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/4ae58dad-494b-4508-b5a5-129f13818d53-ovnkube-script-lib\") pod \"ovnkube-node-vcxnc\" (UID: \"4ae58dad-494b-4508-b5a5-129f13818d53\") " pod="openshift-ovn-kubernetes/ovnkube-node-vcxnc" Jul 06 13:50:26.805834 ip-10-0-136-25 kubenswrapper[2569]: I0706 13:50:26.805783 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/edbb90e8-546b-4edd-a223-5ad4d579adaa-cnibin\") pod \"multus-additional-cni-plugins-brmhg\" (UID: \"edbb90e8-546b-4edd-a223-5ad4d579adaa\") " pod="openshift-multus/multus-additional-cni-plugins-brmhg" Jul 06 13:50:26.805834 ip-10-0-136-25 kubenswrapper[2569]: I0706 13:50:26.805807 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/edbb90e8-546b-4edd-a223-5ad4d579adaa-cni-binary-copy\") pod \"multus-additional-cni-plugins-brmhg\" (UID: \"edbb90e8-546b-4edd-a223-5ad4d579adaa\") " pod="openshift-multus/multus-additional-cni-plugins-brmhg" Jul 06 13:50:26.805834 ip-10-0-136-25 kubenswrapper[2569]: I0706 13:50:26.805831 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/b7576bc8-a480-44ac-b68c-36f9abe93b7d-run\") pod \"tuned-2x29s\" (UID: \"b7576bc8-a480-44ac-b68c-36f9abe93b7d\") " pod="openshift-cluster-node-tuning-operator/tuned-2x29s" Jul 06 13:50:26.806165 ip-10-0-136-25 kubenswrapper[2569]: I0706 13:50:26.805851 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/b7576bc8-a480-44ac-b68c-36f9abe93b7d-sys\") pod \"tuned-2x29s\" (UID: \"b7576bc8-a480-44ac-b68c-36f9abe93b7d\") " pod="openshift-cluster-node-tuning-operator/tuned-2x29s" Jul 06 13:50:26.806165 ip-10-0-136-25 kubenswrapper[2569]: I0706 13:50:26.805919 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/5bbb0d3f-2cd8-4496-b676-6045376a5b13-iptables-alerter-script\") pod \"iptables-alerter-wxxjp\" (UID: \"5bbb0d3f-2cd8-4496-b676-6045376a5b13\") " pod="openshift-network-operator/iptables-alerter-wxxjp" Jul 06 13:50:26.806165 ip-10-0-136-25 kubenswrapper[2569]: I0706 13:50:26.805954 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/5bbb0d3f-2cd8-4496-b676-6045376a5b13-host-slash\") pod \"iptables-alerter-wxxjp\" (UID: \"5bbb0d3f-2cd8-4496-b676-6045376a5b13\") " pod="openshift-network-operator/iptables-alerter-wxxjp" Jul 06 13:50:26.806165 ip-10-0-136-25 kubenswrapper[2569]: I0706 13:50:26.805982 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/4ae58dad-494b-4508-b5a5-129f13818d53-run-openvswitch\") pod \"ovnkube-node-vcxnc\" (UID: \"4ae58dad-494b-4508-b5a5-129f13818d53\") " pod="openshift-ovn-kubernetes/ovnkube-node-vcxnc" Jul 06 13:50:26.806165 ip-10-0-136-25 kubenswrapper[2569]: I0706 13:50:26.806007 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/07d7de38-90f7-4810-b5d6-3a710a3af5a9-host-var-lib-cni-multus\") pod \"multus-dcdgc\" (UID: \"07d7de38-90f7-4810-b5d6-3a710a3af5a9\") " pod="openshift-multus/multus-dcdgc" Jul 06 13:50:26.806165 ip-10-0-136-25 kubenswrapper[2569]: I0706 13:50:26.806031 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/07d7de38-90f7-4810-b5d6-3a710a3af5a9-hostroot\") pod \"multus-dcdgc\" (UID: \"07d7de38-90f7-4810-b5d6-3a710a3af5a9\") " pod="openshift-multus/multus-dcdgc" Jul 06 13:50:26.806165 ip-10-0-136-25 kubenswrapper[2569]: I0706 13:50:26.806054 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/4ae58dad-494b-4508-b5a5-129f13818d53-host-slash\") pod \"ovnkube-node-vcxnc\" (UID: \"4ae58dad-494b-4508-b5a5-129f13818d53\") " pod="openshift-ovn-kubernetes/ovnkube-node-vcxnc" Jul 06 13:50:26.806165 ip-10-0-136-25 kubenswrapper[2569]: I0706 13:50:26.806076 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/4ae58dad-494b-4508-b5a5-129f13818d53-run-systemd\") pod \"ovnkube-node-vcxnc\" (UID: \"4ae58dad-494b-4508-b5a5-129f13818d53\") " pod="openshift-ovn-kubernetes/ovnkube-node-vcxnc" Jul 06 13:50:26.806165 ip-10-0-136-25 kubenswrapper[2569]: I0706 13:50:26.806141 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/b7576bc8-a480-44ac-b68c-36f9abe93b7d-etc-tuned\") pod \"tuned-2x29s\" (UID: \"b7576bc8-a480-44ac-b68c-36f9abe93b7d\") " pod="openshift-cluster-node-tuning-operator/tuned-2x29s" Jul 06 13:50:26.806165 ip-10-0-136-25 kubenswrapper[2569]: I0706 13:50:26.806164 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/4ae58dad-494b-4508-b5a5-129f13818d53-var-lib-openvswitch\") pod \"ovnkube-node-vcxnc\" (UID: \"4ae58dad-494b-4508-b5a5-129f13818d53\") " pod="openshift-ovn-kubernetes/ovnkube-node-vcxnc" Jul 06 13:50:26.806793 ip-10-0-136-25 kubenswrapper[2569]: I0706 13:50:26.806186 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/edbb90e8-546b-4edd-a223-5ad4d579adaa-os-release\") pod \"multus-additional-cni-plugins-brmhg\" (UID: \"edbb90e8-546b-4edd-a223-5ad4d579adaa\") " pod="openshift-multus/multus-additional-cni-plugins-brmhg" Jul 06 13:50:26.806793 ip-10-0-136-25 kubenswrapper[2569]: I0706 13:50:26.806188 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-cg8ph" Jul 06 13:50:26.806793 ip-10-0-136-25 kubenswrapper[2569]: I0706 13:50:26.806233 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/b7576bc8-a480-44ac-b68c-36f9abe93b7d-etc-sysctl-d\") pod \"tuned-2x29s\" (UID: \"b7576bc8-a480-44ac-b68c-36f9abe93b7d\") " pod="openshift-cluster-node-tuning-operator/tuned-2x29s" Jul 06 13:50:26.806793 ip-10-0-136-25 kubenswrapper[2569]: I0706 13:50:26.806274 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/07d7de38-90f7-4810-b5d6-3a710a3af5a9-cnibin\") pod \"multus-dcdgc\" (UID: \"07d7de38-90f7-4810-b5d6-3a710a3af5a9\") " pod="openshift-multus/multus-dcdgc" Jul 06 13:50:26.806793 ip-10-0-136-25 kubenswrapper[2569]: E0706 13:50:26.806287 2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-cg8ph" podUID="576fd8e3-f3a0-48d2-a0d6-faa782a25a63" Jul 06 13:50:26.806793 ip-10-0-136-25 kubenswrapper[2569]: I0706 13:50:26.806303 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/07d7de38-90f7-4810-b5d6-3a710a3af5a9-os-release\") pod \"multus-dcdgc\" (UID: \"07d7de38-90f7-4810-b5d6-3a710a3af5a9\") " pod="openshift-multus/multus-dcdgc" Jul 06 13:50:26.806793 ip-10-0-136-25 kubenswrapper[2569]: I0706 13:50:26.806378 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/cb1807ec-0568-4ab3-a9ca-e453709b820e-agent-certs\") pod \"konnectivity-agent-vqmcq\" (UID: \"cb1807ec-0568-4ab3-a9ca-e453709b820e\") " pod="kube-system/konnectivity-agent-vqmcq" Jul 06 13:50:26.806793 ip-10-0-136-25 kubenswrapper[2569]: I0706 13:50:26.806406 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/cb1807ec-0568-4ab3-a9ca-e453709b820e-konnectivity-ca\") pod \"konnectivity-agent-vqmcq\" (UID: \"cb1807ec-0568-4ab3-a9ca-e453709b820e\") " pod="kube-system/konnectivity-agent-vqmcq" Jul 06 13:50:26.806793 ip-10-0-136-25 kubenswrapper[2569]: I0706 13:50:26.806431 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/b7576bc8-a480-44ac-b68c-36f9abe93b7d-etc-sysconfig\") pod \"tuned-2x29s\" (UID: \"b7576bc8-a480-44ac-b68c-36f9abe93b7d\") " pod="openshift-cluster-node-tuning-operator/tuned-2x29s" Jul 06 13:50:26.806793 ip-10-0-136-25 kubenswrapper[2569]: I0706 13:50:26.806454 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/b7576bc8-a480-44ac-b68c-36f9abe93b7d-etc-systemd\") pod \"tuned-2x29s\" (UID: \"b7576bc8-a480-44ac-b68c-36f9abe93b7d\") " pod="openshift-cluster-node-tuning-operator/tuned-2x29s" Jul 06 13:50:26.806793 ip-10-0-136-25 kubenswrapper[2569]: I0706 13:50:26.806477 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/07d7de38-90f7-4810-b5d6-3a710a3af5a9-system-cni-dir\") pod \"multus-dcdgc\" (UID: \"07d7de38-90f7-4810-b5d6-3a710a3af5a9\") " pod="openshift-multus/multus-dcdgc" Jul 06 13:50:26.806793 ip-10-0-136-25 kubenswrapper[2569]: I0706 13:50:26.806499 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/4ae58dad-494b-4508-b5a5-129f13818d53-host-kubelet\") pod \"ovnkube-node-vcxnc\" (UID: \"4ae58dad-494b-4508-b5a5-129f13818d53\") " pod="openshift-ovn-kubernetes/ovnkube-node-vcxnc" Jul 06 13:50:26.806793 ip-10-0-136-25 kubenswrapper[2569]: I0706 13:50:26.806522 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/4ae58dad-494b-4508-b5a5-129f13818d53-env-overrides\") pod \"ovnkube-node-vcxnc\" (UID: \"4ae58dad-494b-4508-b5a5-129f13818d53\") " pod="openshift-ovn-kubernetes/ovnkube-node-vcxnc" Jul 06 13:50:26.806793 ip-10-0-136-25 kubenswrapper[2569]: I0706 13:50:26.806608 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/b7576bc8-a480-44ac-b68c-36f9abe93b7d-etc-modprobe-d\") pod \"tuned-2x29s\" (UID: \"b7576bc8-a480-44ac-b68c-36f9abe93b7d\") " pod="openshift-cluster-node-tuning-operator/tuned-2x29s" Jul 06 13:50:26.806793 ip-10-0-136-25 kubenswrapper[2569]: I0706 13:50:26.806641 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/4ae58dad-494b-4508-b5a5-129f13818d53-host-cni-bin\") pod \"ovnkube-node-vcxnc\" (UID: \"4ae58dad-494b-4508-b5a5-129f13818d53\") " pod="openshift-ovn-kubernetes/ovnkube-node-vcxnc" Jul 06 13:50:26.806793 ip-10-0-136-25 kubenswrapper[2569]: I0706 13:50:26.806671 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/b7576bc8-a480-44ac-b68c-36f9abe93b7d-etc-kubernetes\") pod \"tuned-2x29s\" (UID: \"b7576bc8-a480-44ac-b68c-36f9abe93b7d\") " pod="openshift-cluster-node-tuning-operator/tuned-2x29s" Jul 06 13:50:26.806793 ip-10-0-136-25 kubenswrapper[2569]: I0706 13:50:26.806697 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4kcgx\" (UniqueName: \"kubernetes.io/projected/d2d86b02-1e00-4267-b9c4-dd365b15342e-kube-api-access-4kcgx\") pod \"network-check-target-gbq5d\" (UID: \"d2d86b02-1e00-4267-b9c4-dd365b15342e\") " pod="openshift-network-diagnostics/network-check-target-gbq5d" Jul 06 13:50:26.807681 ip-10-0-136-25 kubenswrapper[2569]: I0706 13:50:26.806719 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/07d7de38-90f7-4810-b5d6-3a710a3af5a9-multus-cni-dir\") pod \"multus-dcdgc\" (UID: \"07d7de38-90f7-4810-b5d6-3a710a3af5a9\") " pod="openshift-multus/multus-dcdgc" Jul 06 13:50:26.807681 ip-10-0-136-25 kubenswrapper[2569]: I0706 13:50:26.806741 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/07d7de38-90f7-4810-b5d6-3a710a3af5a9-cni-binary-copy\") pod \"multus-dcdgc\" (UID: \"07d7de38-90f7-4810-b5d6-3a710a3af5a9\") " pod="openshift-multus/multus-dcdgc" Jul 06 13:50:26.807681 ip-10-0-136-25 kubenswrapper[2569]: I0706 13:50:26.806779 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/07d7de38-90f7-4810-b5d6-3a710a3af5a9-host-var-lib-cni-bin\") pod \"multus-dcdgc\" (UID: \"07d7de38-90f7-4810-b5d6-3a710a3af5a9\") " pod="openshift-multus/multus-dcdgc" Jul 06 13:50:26.807681 ip-10-0-136-25 kubenswrapper[2569]: I0706 13:50:26.806802 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/b7576bc8-a480-44ac-b68c-36f9abe93b7d-var-lib-kubelet\") pod \"tuned-2x29s\" (UID: \"b7576bc8-a480-44ac-b68c-36f9abe93b7d\") " pod="openshift-cluster-node-tuning-operator/tuned-2x29s" Jul 06 13:50:26.807681 ip-10-0-136-25 kubenswrapper[2569]: I0706 13:50:26.806823 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/b7576bc8-a480-44ac-b68c-36f9abe93b7d-host\") pod \"tuned-2x29s\" (UID: \"b7576bc8-a480-44ac-b68c-36f9abe93b7d\") " pod="openshift-cluster-node-tuning-operator/tuned-2x29s" Jul 06 13:50:26.807681 ip-10-0-136-25 kubenswrapper[2569]: I0706 13:50:26.806846 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/07d7de38-90f7-4810-b5d6-3a710a3af5a9-host-var-lib-kubelet\") pod \"multus-dcdgc\" (UID: \"07d7de38-90f7-4810-b5d6-3a710a3af5a9\") " pod="openshift-multus/multus-dcdgc" Jul 06 13:50:26.807681 ip-10-0-136-25 kubenswrapper[2569]: I0706 13:50:26.806920 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/b7576bc8-a480-44ac-b68c-36f9abe93b7d-tmp\") pod \"tuned-2x29s\" (UID: \"b7576bc8-a480-44ac-b68c-36f9abe93b7d\") " pod="openshift-cluster-node-tuning-operator/tuned-2x29s" Jul 06 13:50:26.807681 ip-10-0-136-25 kubenswrapper[2569]: I0706 13:50:26.806944 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/07d7de38-90f7-4810-b5d6-3a710a3af5a9-multus-socket-dir-parent\") pod \"multus-dcdgc\" (UID: \"07d7de38-90f7-4810-b5d6-3a710a3af5a9\") " pod="openshift-multus/multus-dcdgc" Jul 06 13:50:26.807681 ip-10-0-136-25 kubenswrapper[2569]: I0706 13:50:26.806966 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/4ae58dad-494b-4508-b5a5-129f13818d53-host-run-netns\") pod \"ovnkube-node-vcxnc\" (UID: \"4ae58dad-494b-4508-b5a5-129f13818d53\") " pod="openshift-ovn-kubernetes/ovnkube-node-vcxnc" Jul 06 13:50:26.807681 ip-10-0-136-25 kubenswrapper[2569]: I0706 13:50:26.806990 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/4ae58dad-494b-4508-b5a5-129f13818d53-host-cni-netd\") pod \"ovnkube-node-vcxnc\" (UID: \"4ae58dad-494b-4508-b5a5-129f13818d53\") " pod="openshift-ovn-kubernetes/ovnkube-node-vcxnc" Jul 06 13:50:26.807681 ip-10-0-136-25 kubenswrapper[2569]: I0706 13:50:26.807011 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4xkx2\" (UniqueName: \"kubernetes.io/projected/028cbff5-11b2-411a-90ed-c0bd3f563830-kube-api-access-4xkx2\") pod \"node-ca-c9sqv\" (UID: \"028cbff5-11b2-411a-90ed-c0bd3f563830\") " pod="openshift-image-registry/node-ca-c9sqv" Jul 06 13:50:26.807681 ip-10-0-136-25 kubenswrapper[2569]: I0706 13:50:26.807038 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/4ae58dad-494b-4508-b5a5-129f13818d53-node-log\") pod \"ovnkube-node-vcxnc\" (UID: \"4ae58dad-494b-4508-b5a5-129f13818d53\") " pod="openshift-ovn-kubernetes/ovnkube-node-vcxnc" Jul 06 13:50:26.807681 ip-10-0-136-25 kubenswrapper[2569]: I0706 13:50:26.807064 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/4ae58dad-494b-4508-b5a5-129f13818d53-host-run-ovn-kubernetes\") pod \"ovnkube-node-vcxnc\" (UID: \"4ae58dad-494b-4508-b5a5-129f13818d53\") " pod="openshift-ovn-kubernetes/ovnkube-node-vcxnc" Jul 06 13:50:26.807681 ip-10-0-136-25 kubenswrapper[2569]: I0706 13:50:26.807151 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/4ae58dad-494b-4508-b5a5-129f13818d53-ovnkube-config\") pod \"ovnkube-node-vcxnc\" (UID: \"4ae58dad-494b-4508-b5a5-129f13818d53\") " pod="openshift-ovn-kubernetes/ovnkube-node-vcxnc" Jul 06 13:50:26.807681 ip-10-0-136-25 kubenswrapper[2569]: I0706 13:50:26.807183 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/edbb90e8-546b-4edd-a223-5ad4d579adaa-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-brmhg\" (UID: \"edbb90e8-546b-4edd-a223-5ad4d579adaa\") " pod="openshift-multus/multus-additional-cni-plugins-brmhg" Jul 06 13:50:26.807681 ip-10-0-136-25 kubenswrapper[2569]: I0706 13:50:26.807238 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/edbb90e8-546b-4edd-a223-5ad4d579adaa-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-brmhg\" (UID: \"edbb90e8-546b-4edd-a223-5ad4d579adaa\") " pod="openshift-multus/multus-additional-cni-plugins-brmhg" Jul 06 13:50:26.807681 ip-10-0-136-25 kubenswrapper[2569]: I0706 13:50:26.807284 2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-ljs9p\"" Jul 06 13:50:26.807681 ip-10-0-136-25 kubenswrapper[2569]: I0706 13:50:26.807415 2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\"" Jul 06 13:50:26.808588 ip-10-0-136-25 kubenswrapper[2569]: I0706 13:50:26.807272 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/07d7de38-90f7-4810-b5d6-3a710a3af5a9-multus-conf-dir\") pod \"multus-dcdgc\" (UID: \"07d7de38-90f7-4810-b5d6-3a710a3af5a9\") " pod="openshift-multus/multus-dcdgc" Jul 06 13:50:26.808588 ip-10-0-136-25 kubenswrapper[2569]: I0706 13:50:26.807480 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/07d7de38-90f7-4810-b5d6-3a710a3af5a9-multus-daemon-config\") pod \"multus-dcdgc\" (UID: \"07d7de38-90f7-4810-b5d6-3a710a3af5a9\") " pod="openshift-multus/multus-dcdgc" Jul 06 13:50:26.808588 ip-10-0-136-25 kubenswrapper[2569]: I0706 13:50:26.807510 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fxb8c\" (UniqueName: \"kubernetes.io/projected/4ae58dad-494b-4508-b5a5-129f13818d53-kube-api-access-fxb8c\") pod \"ovnkube-node-vcxnc\" (UID: \"4ae58dad-494b-4508-b5a5-129f13818d53\") " pod="openshift-ovn-kubernetes/ovnkube-node-vcxnc" Jul 06 13:50:26.808588 ip-10-0-136-25 kubenswrapper[2569]: I0706 13:50:26.807540 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7nnml\" (UniqueName: \"kubernetes.io/projected/b7576bc8-a480-44ac-b68c-36f9abe93b7d-kube-api-access-7nnml\") pod \"tuned-2x29s\" (UID: \"b7576bc8-a480-44ac-b68c-36f9abe93b7d\") " pod="openshift-cluster-node-tuning-operator/tuned-2x29s" Jul 06 13:50:26.808588 ip-10-0-136-25 kubenswrapper[2569]: I0706 13:50:26.807577 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/07d7de38-90f7-4810-b5d6-3a710a3af5a9-host-run-multus-certs\") pod \"multus-dcdgc\" (UID: \"07d7de38-90f7-4810-b5d6-3a710a3af5a9\") " pod="openshift-multus/multus-dcdgc" Jul 06 13:50:26.808588 ip-10-0-136-25 kubenswrapper[2569]: I0706 13:50:26.807622 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/4ae58dad-494b-4508-b5a5-129f13818d53-etc-openvswitch\") pod \"ovnkube-node-vcxnc\" (UID: \"4ae58dad-494b-4508-b5a5-129f13818d53\") " pod="openshift-ovn-kubernetes/ovnkube-node-vcxnc" Jul 06 13:50:26.808588 ip-10-0-136-25 kubenswrapper[2569]: I0706 13:50:26.807651 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/b7576bc8-a480-44ac-b68c-36f9abe93b7d-lib-modules\") pod \"tuned-2x29s\" (UID: \"b7576bc8-a480-44ac-b68c-36f9abe93b7d\") " pod="openshift-cluster-node-tuning-operator/tuned-2x29s" Jul 06 13:50:26.808588 ip-10-0-136-25 kubenswrapper[2569]: I0706 13:50:26.807673 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/028cbff5-11b2-411a-90ed-c0bd3f563830-serviceca\") pod \"node-ca-c9sqv\" (UID: \"028cbff5-11b2-411a-90ed-c0bd3f563830\") " pod="openshift-image-registry/node-ca-c9sqv" Jul 06 13:50:26.808588 ip-10-0-136-25 kubenswrapper[2569]: I0706 13:50:26.807696 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mwkhd\" (UniqueName: \"kubernetes.io/projected/07d7de38-90f7-4810-b5d6-3a710a3af5a9-kube-api-access-mwkhd\") pod \"multus-dcdgc\" (UID: \"07d7de38-90f7-4810-b5d6-3a710a3af5a9\") " pod="openshift-multus/multus-dcdgc" Jul 06 13:50:26.808588 ip-10-0-136-25 kubenswrapper[2569]: I0706 13:50:26.807828 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/4ae58dad-494b-4508-b5a5-129f13818d53-systemd-units\") pod \"ovnkube-node-vcxnc\" (UID: \"4ae58dad-494b-4508-b5a5-129f13818d53\") " pod="openshift-ovn-kubernetes/ovnkube-node-vcxnc" Jul 06 13:50:26.808588 ip-10-0-136-25 kubenswrapper[2569]: I0706 13:50:26.807863 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/b7576bc8-a480-44ac-b68c-36f9abe93b7d-etc-sysctl-conf\") pod \"tuned-2x29s\" (UID: \"b7576bc8-a480-44ac-b68c-36f9abe93b7d\") " pod="openshift-cluster-node-tuning-operator/tuned-2x29s" Jul 06 13:50:26.808588 ip-10-0-136-25 kubenswrapper[2569]: I0706 13:50:26.807885 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/07d7de38-90f7-4810-b5d6-3a710a3af5a9-host-run-netns\") pod \"multus-dcdgc\" (UID: \"07d7de38-90f7-4810-b5d6-3a710a3af5a9\") " pod="openshift-multus/multus-dcdgc" Jul 06 13:50:26.808588 ip-10-0-136-25 kubenswrapper[2569]: I0706 13:50:26.807906 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/07d7de38-90f7-4810-b5d6-3a710a3af5a9-etc-kubernetes\") pod \"multus-dcdgc\" (UID: \"07d7de38-90f7-4810-b5d6-3a710a3af5a9\") " pod="openshift-multus/multus-dcdgc" Jul 06 13:50:26.808588 ip-10-0-136-25 kubenswrapper[2569]: I0706 13:50:26.807928 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/028cbff5-11b2-411a-90ed-c0bd3f563830-host\") pod \"node-ca-c9sqv\" (UID: \"028cbff5-11b2-411a-90ed-c0bd3f563830\") " pod="openshift-image-registry/node-ca-c9sqv" Jul 06 13:50:26.808588 ip-10-0-136-25 kubenswrapper[2569]: I0706 13:50:26.807956 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/07d7de38-90f7-4810-b5d6-3a710a3af5a9-host-run-k8s-cni-cncf-io\") pod \"multus-dcdgc\" (UID: \"07d7de38-90f7-4810-b5d6-3a710a3af5a9\") " pod="openshift-multus/multus-dcdgc" Jul 06 13:50:26.808588 ip-10-0-136-25 kubenswrapper[2569]: I0706 13:50:26.808176 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xd6qh\" (UniqueName: \"kubernetes.io/projected/5bbb0d3f-2cd8-4496-b676-6045376a5b13-kube-api-access-xd6qh\") pod \"iptables-alerter-wxxjp\" (UID: \"5bbb0d3f-2cd8-4496-b676-6045376a5b13\") " pod="openshift-network-operator/iptables-alerter-wxxjp" Jul 06 13:50:26.808588 ip-10-0-136-25 kubenswrapper[2569]: I0706 13:50:26.808202 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/4ae58dad-494b-4508-b5a5-129f13818d53-log-socket\") pod \"ovnkube-node-vcxnc\" (UID: \"4ae58dad-494b-4508-b5a5-129f13818d53\") " pod="openshift-ovn-kubernetes/ovnkube-node-vcxnc" Jul 06 13:50:26.809281 ip-10-0-136-25 kubenswrapper[2569]: I0706 13:50:26.808243 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/4ae58dad-494b-4508-b5a5-129f13818d53-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-vcxnc\" (UID: \"4ae58dad-494b-4508-b5a5-129f13818d53\") " pod="openshift-ovn-kubernetes/ovnkube-node-vcxnc" Jul 06 13:50:26.809281 ip-10-0-136-25 kubenswrapper[2569]: I0706 13:50:26.808329 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/4ae58dad-494b-4508-b5a5-129f13818d53-ovn-node-metrics-cert\") pod \"ovnkube-node-vcxnc\" (UID: \"4ae58dad-494b-4508-b5a5-129f13818d53\") " pod="openshift-ovn-kubernetes/ovnkube-node-vcxnc" Jul 06 13:50:26.809281 ip-10-0-136-25 kubenswrapper[2569]: I0706 13:50:26.808352 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/edbb90e8-546b-4edd-a223-5ad4d579adaa-system-cni-dir\") pod \"multus-additional-cni-plugins-brmhg\" (UID: \"edbb90e8-546b-4edd-a223-5ad4d579adaa\") " pod="openshift-multus/multus-additional-cni-plugins-brmhg" Jul 06 13:50:26.809281 ip-10-0-136-25 kubenswrapper[2569]: I0706 13:50:26.808375 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/edbb90e8-546b-4edd-a223-5ad4d579adaa-tuning-conf-dir\") pod \"multus-additional-cni-plugins-brmhg\" (UID: \"edbb90e8-546b-4edd-a223-5ad4d579adaa\") " pod="openshift-multus/multus-additional-cni-plugins-brmhg" Jul 06 13:50:26.809281 ip-10-0-136-25 kubenswrapper[2569]: I0706 13:50:26.808399 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5gxvp\" (UniqueName: \"kubernetes.io/projected/edbb90e8-546b-4edd-a223-5ad4d579adaa-kube-api-access-5gxvp\") pod \"multus-additional-cni-plugins-brmhg\" (UID: \"edbb90e8-546b-4edd-a223-5ad4d579adaa\") " pod="openshift-multus/multus-additional-cni-plugins-brmhg" Jul 06 13:50:26.809495 ip-10-0-136-25 kubenswrapper[2569]: I0706 13:50:26.809448 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-hqt9l" Jul 06 13:50:26.809543 ip-10-0-136-25 kubenswrapper[2569]: E0706 13:50:26.809507 2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-hqt9l" podUID="2c4b4cf7-31a7-4cde-8212-9d78cc184e6c" Jul 06 13:50:26.809622 ip-10-0-136-25 kubenswrapper[2569]: I0706 13:50:26.809602 2569 reflector.go:439] "Caches populated" logger="kubernetes.io/kubelet-serving" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" Jul 06 13:50:26.836487 ip-10-0-136-25 kubenswrapper[2569]: I0706 13:50:26.836448 2569 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kubelet-serving" csr="csr-jcmv4" Jul 06 13:50:26.846263 ip-10-0-136-25 kubenswrapper[2569]: I0706 13:50:26.846238 2569 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kubelet-serving" csr="csr-jcmv4" Jul 06 13:50:26.896380 ip-10-0-136-25 kubenswrapper[2569]: I0706 13:50:26.896345 2569 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world" Jul 06 13:50:26.908717 ip-10-0-136-25 kubenswrapper[2569]: I0706 13:50:26.908690 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/03d48054-9789-444b-82c7-ac9e3a504c9c-etc-selinux\") pod \"aws-ebs-csi-driver-node-2dnkc\" (UID: \"03d48054-9789-444b-82c7-ac9e3a504c9c\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-2dnkc" Jul 06 13:50:26.908717 ip-10-0-136-25 kubenswrapper[2569]: I0706 13:50:26.908720 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-62mdb\" (UniqueName: \"kubernetes.io/projected/03d48054-9789-444b-82c7-ac9e3a504c9c-kube-api-access-62mdb\") pod \"aws-ebs-csi-driver-node-2dnkc\" (UID: \"03d48054-9789-444b-82c7-ac9e3a504c9c\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-2dnkc" Jul 06 13:50:26.908954 ip-10-0-136-25 kubenswrapper[2569]: I0706 13:50:26.908736 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2c4b4cf7-31a7-4cde-8212-9d78cc184e6c-metrics-certs\") pod \"network-metrics-daemon-hqt9l\" (UID: \"2c4b4cf7-31a7-4cde-8212-9d78cc184e6c\") " pod="openshift-multus/network-metrics-daemon-hqt9l" Jul 06 13:50:26.908954 ip-10-0-136-25 kubenswrapper[2569]: I0706 13:50:26.908787 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/4ae58dad-494b-4508-b5a5-129f13818d53-etc-openvswitch\") pod \"ovnkube-node-vcxnc\" (UID: \"4ae58dad-494b-4508-b5a5-129f13818d53\") " pod="openshift-ovn-kubernetes/ovnkube-node-vcxnc" Jul 06 13:50:26.908954 ip-10-0-136-25 kubenswrapper[2569]: I0706 13:50:26.908825 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/b7576bc8-a480-44ac-b68c-36f9abe93b7d-lib-modules\") pod \"tuned-2x29s\" (UID: \"b7576bc8-a480-44ac-b68c-36f9abe93b7d\") " pod="openshift-cluster-node-tuning-operator/tuned-2x29s" Jul 06 13:50:26.908954 ip-10-0-136-25 kubenswrapper[2569]: I0706 13:50:26.908850 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/028cbff5-11b2-411a-90ed-c0bd3f563830-serviceca\") pod \"node-ca-c9sqv\" (UID: \"028cbff5-11b2-411a-90ed-c0bd3f563830\") " pod="openshift-image-registry/node-ca-c9sqv" Jul 06 13:50:26.908954 ip-10-0-136-25 kubenswrapper[2569]: I0706 13:50:26.908868 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mwkhd\" (UniqueName: \"kubernetes.io/projected/07d7de38-90f7-4810-b5d6-3a710a3af5a9-kube-api-access-mwkhd\") pod \"multus-dcdgc\" (UID: \"07d7de38-90f7-4810-b5d6-3a710a3af5a9\") " pod="openshift-multus/multus-dcdgc" Jul 06 13:50:26.908954 ip-10-0-136-25 kubenswrapper[2569]: I0706 13:50:26.908886 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/4ae58dad-494b-4508-b5a5-129f13818d53-systemd-units\") pod \"ovnkube-node-vcxnc\" (UID: \"4ae58dad-494b-4508-b5a5-129f13818d53\") " pod="openshift-ovn-kubernetes/ovnkube-node-vcxnc" Jul 06 13:50:26.908954 ip-10-0-136-25 kubenswrapper[2569]: I0706 13:50:26.908900 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/b7576bc8-a480-44ac-b68c-36f9abe93b7d-etc-sysctl-conf\") pod \"tuned-2x29s\" (UID: \"b7576bc8-a480-44ac-b68c-36f9abe93b7d\") " pod="openshift-cluster-node-tuning-operator/tuned-2x29s" Jul 06 13:50:26.908954 ip-10-0-136-25 kubenswrapper[2569]: I0706 13:50:26.908916 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/07d7de38-90f7-4810-b5d6-3a710a3af5a9-host-run-netns\") pod \"multus-dcdgc\" (UID: \"07d7de38-90f7-4810-b5d6-3a710a3af5a9\") " pod="openshift-multus/multus-dcdgc" Jul 06 13:50:26.908954 ip-10-0-136-25 kubenswrapper[2569]: I0706 13:50:26.908864 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/4ae58dad-494b-4508-b5a5-129f13818d53-etc-openvswitch\") pod \"ovnkube-node-vcxnc\" (UID: \"4ae58dad-494b-4508-b5a5-129f13818d53\") " pod="openshift-ovn-kubernetes/ovnkube-node-vcxnc" Jul 06 13:50:26.909387 ip-10-0-136-25 kubenswrapper[2569]: I0706 13:50:26.908931 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/07d7de38-90f7-4810-b5d6-3a710a3af5a9-etc-kubernetes\") pod \"multus-dcdgc\" (UID: \"07d7de38-90f7-4810-b5d6-3a710a3af5a9\") " pod="openshift-multus/multus-dcdgc" Jul 06 13:50:26.909387 ip-10-0-136-25 kubenswrapper[2569]: I0706 13:50:26.908983 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/07d7de38-90f7-4810-b5d6-3a710a3af5a9-etc-kubernetes\") pod \"multus-dcdgc\" (UID: \"07d7de38-90f7-4810-b5d6-3a710a3af5a9\") " pod="openshift-multus/multus-dcdgc" Jul 06 13:50:26.909387 ip-10-0-136-25 kubenswrapper[2569]: I0706 13:50:26.908998 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/03d48054-9789-444b-82c7-ac9e3a504c9c-kubelet-dir\") pod \"aws-ebs-csi-driver-node-2dnkc\" (UID: \"03d48054-9789-444b-82c7-ac9e3a504c9c\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-2dnkc" Jul 06 13:50:26.909387 ip-10-0-136-25 kubenswrapper[2569]: I0706 13:50:26.909005 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/b7576bc8-a480-44ac-b68c-36f9abe93b7d-lib-modules\") pod \"tuned-2x29s\" (UID: \"b7576bc8-a480-44ac-b68c-36f9abe93b7d\") " pod="openshift-cluster-node-tuning-operator/tuned-2x29s" Jul 06 13:50:26.909387 ip-10-0-136-25 kubenswrapper[2569]: I0706 13:50:26.909033 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/07d7de38-90f7-4810-b5d6-3a710a3af5a9-host-run-netns\") pod \"multus-dcdgc\" (UID: \"07d7de38-90f7-4810-b5d6-3a710a3af5a9\") " pod="openshift-multus/multus-dcdgc" Jul 06 13:50:26.909387 ip-10-0-136-25 kubenswrapper[2569]: I0706 13:50:26.909050 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/576fd8e3-f3a0-48d2-a0d6-faa782a25a63-kubelet-config\") pod \"global-pull-secret-syncer-cg8ph\" (UID: \"576fd8e3-f3a0-48d2-a0d6-faa782a25a63\") " pod="kube-system/global-pull-secret-syncer-cg8ph" Jul 06 13:50:26.909387 ip-10-0-136-25 kubenswrapper[2569]: I0706 13:50:26.909068 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/4ae58dad-494b-4508-b5a5-129f13818d53-systemd-units\") pod \"ovnkube-node-vcxnc\" (UID: \"4ae58dad-494b-4508-b5a5-129f13818d53\") " pod="openshift-ovn-kubernetes/ovnkube-node-vcxnc" Jul 06 13:50:26.909387 ip-10-0-136-25 kubenswrapper[2569]: I0706 13:50:26.909080 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/028cbff5-11b2-411a-90ed-c0bd3f563830-host\") pod \"node-ca-c9sqv\" (UID: \"028cbff5-11b2-411a-90ed-c0bd3f563830\") " pod="openshift-image-registry/node-ca-c9sqv" Jul 06 13:50:26.909387 ip-10-0-136-25 kubenswrapper[2569]: I0706 13:50:26.909106 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/07d7de38-90f7-4810-b5d6-3a710a3af5a9-host-run-k8s-cni-cncf-io\") pod \"multus-dcdgc\" (UID: \"07d7de38-90f7-4810-b5d6-3a710a3af5a9\") " pod="openshift-multus/multus-dcdgc" Jul 06 13:50:26.909387 ip-10-0-136-25 kubenswrapper[2569]: I0706 13:50:26.909104 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/b7576bc8-a480-44ac-b68c-36f9abe93b7d-etc-sysctl-conf\") pod \"tuned-2x29s\" (UID: \"b7576bc8-a480-44ac-b68c-36f9abe93b7d\") " pod="openshift-cluster-node-tuning-operator/tuned-2x29s" Jul 06 13:50:26.909387 ip-10-0-136-25 kubenswrapper[2569]: I0706 13:50:26.909130 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qvkxw\" (UniqueName: \"kubernetes.io/projected/2c4b4cf7-31a7-4cde-8212-9d78cc184e6c-kube-api-access-qvkxw\") pod \"network-metrics-daemon-hqt9l\" (UID: \"2c4b4cf7-31a7-4cde-8212-9d78cc184e6c\") " pod="openshift-multus/network-metrics-daemon-hqt9l" Jul 06 13:50:26.909387 ip-10-0-136-25 kubenswrapper[2569]: I0706 13:50:26.909175 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/028cbff5-11b2-411a-90ed-c0bd3f563830-host\") pod \"node-ca-c9sqv\" (UID: \"028cbff5-11b2-411a-90ed-c0bd3f563830\") " pod="openshift-image-registry/node-ca-c9sqv" Jul 06 13:50:26.909387 ip-10-0-136-25 kubenswrapper[2569]: I0706 13:50:26.909224 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xd6qh\" (UniqueName: \"kubernetes.io/projected/5bbb0d3f-2cd8-4496-b676-6045376a5b13-kube-api-access-xd6qh\") pod \"iptables-alerter-wxxjp\" (UID: \"5bbb0d3f-2cd8-4496-b676-6045376a5b13\") " pod="openshift-network-operator/iptables-alerter-wxxjp" Jul 06 13:50:26.909387 ip-10-0-136-25 kubenswrapper[2569]: I0706 13:50:26.909253 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/4ae58dad-494b-4508-b5a5-129f13818d53-log-socket\") pod \"ovnkube-node-vcxnc\" (UID: \"4ae58dad-494b-4508-b5a5-129f13818d53\") " pod="openshift-ovn-kubernetes/ovnkube-node-vcxnc" Jul 06 13:50:26.909387 ip-10-0-136-25 kubenswrapper[2569]: I0706 13:50:26.909256 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/07d7de38-90f7-4810-b5d6-3a710a3af5a9-host-run-k8s-cni-cncf-io\") pod \"multus-dcdgc\" (UID: \"07d7de38-90f7-4810-b5d6-3a710a3af5a9\") " pod="openshift-multus/multus-dcdgc" Jul 06 13:50:26.909387 ip-10-0-136-25 kubenswrapper[2569]: I0706 13:50:26.909280 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/4ae58dad-494b-4508-b5a5-129f13818d53-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-vcxnc\" (UID: \"4ae58dad-494b-4508-b5a5-129f13818d53\") " pod="openshift-ovn-kubernetes/ovnkube-node-vcxnc" Jul 06 13:50:26.909387 ip-10-0-136-25 kubenswrapper[2569]: I0706 13:50:26.909307 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/4ae58dad-494b-4508-b5a5-129f13818d53-ovn-node-metrics-cert\") pod \"ovnkube-node-vcxnc\" (UID: \"4ae58dad-494b-4508-b5a5-129f13818d53\") " pod="openshift-ovn-kubernetes/ovnkube-node-vcxnc" Jul 06 13:50:26.910130 ip-10-0-136-25 kubenswrapper[2569]: I0706 13:50:26.909332 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/edbb90e8-546b-4edd-a223-5ad4d579adaa-system-cni-dir\") pod \"multus-additional-cni-plugins-brmhg\" (UID: \"edbb90e8-546b-4edd-a223-5ad4d579adaa\") " pod="openshift-multus/multus-additional-cni-plugins-brmhg" Jul 06 13:50:26.910130 ip-10-0-136-25 kubenswrapper[2569]: I0706 13:50:26.909344 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/4ae58dad-494b-4508-b5a5-129f13818d53-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-vcxnc\" (UID: \"4ae58dad-494b-4508-b5a5-129f13818d53\") " pod="openshift-ovn-kubernetes/ovnkube-node-vcxnc" Jul 06 13:50:26.910130 ip-10-0-136-25 kubenswrapper[2569]: I0706 13:50:26.909368 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/edbb90e8-546b-4edd-a223-5ad4d579adaa-tuning-conf-dir\") pod \"multus-additional-cni-plugins-brmhg\" (UID: \"edbb90e8-546b-4edd-a223-5ad4d579adaa\") " pod="openshift-multus/multus-additional-cni-plugins-brmhg" Jul 06 13:50:26.910130 ip-10-0-136-25 kubenswrapper[2569]: I0706 13:50:26.909377 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/028cbff5-11b2-411a-90ed-c0bd3f563830-serviceca\") pod \"node-ca-c9sqv\" (UID: \"028cbff5-11b2-411a-90ed-c0bd3f563830\") " pod="openshift-image-registry/node-ca-c9sqv" Jul 06 13:50:26.910130 ip-10-0-136-25 kubenswrapper[2569]: I0706 13:50:26.909396 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/4ae58dad-494b-4508-b5a5-129f13818d53-log-socket\") pod \"ovnkube-node-vcxnc\" (UID: \"4ae58dad-494b-4508-b5a5-129f13818d53\") " pod="openshift-ovn-kubernetes/ovnkube-node-vcxnc" Jul 06 13:50:26.910130 ip-10-0-136-25 kubenswrapper[2569]: I0706 13:50:26.909438 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/edbb90e8-546b-4edd-a223-5ad4d579adaa-system-cni-dir\") pod \"multus-additional-cni-plugins-brmhg\" (UID: \"edbb90e8-546b-4edd-a223-5ad4d579adaa\") " pod="openshift-multus/multus-additional-cni-plugins-brmhg" Jul 06 13:50:26.910130 ip-10-0-136-25 kubenswrapper[2569]: I0706 13:50:26.909439 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5gxvp\" (UniqueName: \"kubernetes.io/projected/edbb90e8-546b-4edd-a223-5ad4d579adaa-kube-api-access-5gxvp\") pod \"multus-additional-cni-plugins-brmhg\" (UID: \"edbb90e8-546b-4edd-a223-5ad4d579adaa\") " pod="openshift-multus/multus-additional-cni-plugins-brmhg" Jul 06 13:50:26.910130 ip-10-0-136-25 kubenswrapper[2569]: I0706 13:50:26.909486 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/03d48054-9789-444b-82c7-ac9e3a504c9c-socket-dir\") pod \"aws-ebs-csi-driver-node-2dnkc\" (UID: \"03d48054-9789-444b-82c7-ac9e3a504c9c\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-2dnkc" Jul 06 13:50:26.910130 ip-10-0-136-25 kubenswrapper[2569]: I0706 13:50:26.909521 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/4ae58dad-494b-4508-b5a5-129f13818d53-run-ovn\") pod \"ovnkube-node-vcxnc\" (UID: \"4ae58dad-494b-4508-b5a5-129f13818d53\") " pod="openshift-ovn-kubernetes/ovnkube-node-vcxnc" Jul 06 13:50:26.910130 ip-10-0-136-25 kubenswrapper[2569]: I0706 13:50:26.909562 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/4ae58dad-494b-4508-b5a5-129f13818d53-run-ovn\") pod \"ovnkube-node-vcxnc\" (UID: \"4ae58dad-494b-4508-b5a5-129f13818d53\") " pod="openshift-ovn-kubernetes/ovnkube-node-vcxnc" Jul 06 13:50:26.910130 ip-10-0-136-25 kubenswrapper[2569]: I0706 13:50:26.909591 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/edbb90e8-546b-4edd-a223-5ad4d579adaa-tuning-conf-dir\") pod \"multus-additional-cni-plugins-brmhg\" (UID: \"edbb90e8-546b-4edd-a223-5ad4d579adaa\") " pod="openshift-multus/multus-additional-cni-plugins-brmhg" Jul 06 13:50:26.910130 ip-10-0-136-25 kubenswrapper[2569]: I0706 13:50:26.909593 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/4ae58dad-494b-4508-b5a5-129f13818d53-ovnkube-script-lib\") pod \"ovnkube-node-vcxnc\" (UID: \"4ae58dad-494b-4508-b5a5-129f13818d53\") " pod="openshift-ovn-kubernetes/ovnkube-node-vcxnc" Jul 06 13:50:26.910130 ip-10-0-136-25 kubenswrapper[2569]: I0706 13:50:26.909621 2569 swap_util.go:74] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory" Jul 06 13:50:26.910130 ip-10-0-136-25 kubenswrapper[2569]: I0706 13:50:26.909724 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/edbb90e8-546b-4edd-a223-5ad4d579adaa-cnibin\") pod \"multus-additional-cni-plugins-brmhg\" (UID: \"edbb90e8-546b-4edd-a223-5ad4d579adaa\") " pod="openshift-multus/multus-additional-cni-plugins-brmhg" Jul 06 13:50:26.910130 ip-10-0-136-25 kubenswrapper[2569]: I0706 13:50:26.909759 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/edbb90e8-546b-4edd-a223-5ad4d579adaa-cni-binary-copy\") pod \"multus-additional-cni-plugins-brmhg\" (UID: \"edbb90e8-546b-4edd-a223-5ad4d579adaa\") " pod="openshift-multus/multus-additional-cni-plugins-brmhg" Jul 06 13:50:26.910130 ip-10-0-136-25 kubenswrapper[2569]: I0706 13:50:26.909780 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/b7576bc8-a480-44ac-b68c-36f9abe93b7d-run\") pod \"tuned-2x29s\" (UID: \"b7576bc8-a480-44ac-b68c-36f9abe93b7d\") " pod="openshift-cluster-node-tuning-operator/tuned-2x29s" Jul 06 13:50:26.910130 ip-10-0-136-25 kubenswrapper[2569]: I0706 13:50:26.909798 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/b7576bc8-a480-44ac-b68c-36f9abe93b7d-sys\") pod \"tuned-2x29s\" (UID: \"b7576bc8-a480-44ac-b68c-36f9abe93b7d\") " pod="openshift-cluster-node-tuning-operator/tuned-2x29s" Jul 06 13:50:26.910897 ip-10-0-136-25 kubenswrapper[2569]: I0706 13:50:26.909816 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/5bbb0d3f-2cd8-4496-b676-6045376a5b13-iptables-alerter-script\") pod \"iptables-alerter-wxxjp\" (UID: \"5bbb0d3f-2cd8-4496-b676-6045376a5b13\") " pod="openshift-network-operator/iptables-alerter-wxxjp" Jul 06 13:50:26.910897 ip-10-0-136-25 kubenswrapper[2569]: I0706 13:50:26.909836 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/5bbb0d3f-2cd8-4496-b676-6045376a5b13-host-slash\") pod \"iptables-alerter-wxxjp\" (UID: \"5bbb0d3f-2cd8-4496-b676-6045376a5b13\") " pod="openshift-network-operator/iptables-alerter-wxxjp" Jul 06 13:50:26.910897 ip-10-0-136-25 kubenswrapper[2569]: I0706 13:50:26.909858 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/4ae58dad-494b-4508-b5a5-129f13818d53-run-openvswitch\") pod \"ovnkube-node-vcxnc\" (UID: \"4ae58dad-494b-4508-b5a5-129f13818d53\") " pod="openshift-ovn-kubernetes/ovnkube-node-vcxnc" Jul 06 13:50:26.910897 ip-10-0-136-25 kubenswrapper[2569]: I0706 13:50:26.909882 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/07d7de38-90f7-4810-b5d6-3a710a3af5a9-host-var-lib-cni-multus\") pod \"multus-dcdgc\" (UID: \"07d7de38-90f7-4810-b5d6-3a710a3af5a9\") " pod="openshift-multus/multus-dcdgc" Jul 06 13:50:26.910897 ip-10-0-136-25 kubenswrapper[2569]: I0706 13:50:26.909907 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/07d7de38-90f7-4810-b5d6-3a710a3af5a9-hostroot\") pod \"multus-dcdgc\" (UID: \"07d7de38-90f7-4810-b5d6-3a710a3af5a9\") " pod="openshift-multus/multus-dcdgc" Jul 06 13:50:26.910897 ip-10-0-136-25 kubenswrapper[2569]: I0706 13:50:26.909929 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/03d48054-9789-444b-82c7-ac9e3a504c9c-sys-fs\") pod \"aws-ebs-csi-driver-node-2dnkc\" (UID: \"03d48054-9789-444b-82c7-ac9e3a504c9c\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-2dnkc" Jul 06 13:50:26.910897 ip-10-0-136-25 kubenswrapper[2569]: I0706 13:50:26.909952 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/4ae58dad-494b-4508-b5a5-129f13818d53-host-slash\") pod \"ovnkube-node-vcxnc\" (UID: \"4ae58dad-494b-4508-b5a5-129f13818d53\") " pod="openshift-ovn-kubernetes/ovnkube-node-vcxnc" Jul 06 13:50:26.910897 ip-10-0-136-25 kubenswrapper[2569]: I0706 13:50:26.909979 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/4ae58dad-494b-4508-b5a5-129f13818d53-run-systemd\") pod \"ovnkube-node-vcxnc\" (UID: \"4ae58dad-494b-4508-b5a5-129f13818d53\") " pod="openshift-ovn-kubernetes/ovnkube-node-vcxnc" Jul 06 13:50:26.910897 ip-10-0-136-25 kubenswrapper[2569]: I0706 13:50:26.910005 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/b7576bc8-a480-44ac-b68c-36f9abe93b7d-etc-tuned\") pod \"tuned-2x29s\" (UID: \"b7576bc8-a480-44ac-b68c-36f9abe93b7d\") " pod="openshift-cluster-node-tuning-operator/tuned-2x29s" Jul 06 13:50:26.910897 ip-10-0-136-25 kubenswrapper[2569]: I0706 13:50:26.910018 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/b7576bc8-a480-44ac-b68c-36f9abe93b7d-run\") pod \"tuned-2x29s\" (UID: \"b7576bc8-a480-44ac-b68c-36f9abe93b7d\") " pod="openshift-cluster-node-tuning-operator/tuned-2x29s" Jul 06 13:50:26.910897 ip-10-0-136-25 kubenswrapper[2569]: I0706 13:50:26.910030 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/03d48054-9789-444b-82c7-ac9e3a504c9c-registration-dir\") pod \"aws-ebs-csi-driver-node-2dnkc\" (UID: \"03d48054-9789-444b-82c7-ac9e3a504c9c\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-2dnkc" Jul 06 13:50:26.910897 ip-10-0-136-25 kubenswrapper[2569]: I0706 13:50:26.910055 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/576fd8e3-f3a0-48d2-a0d6-faa782a25a63-dbus\") pod \"global-pull-secret-syncer-cg8ph\" (UID: \"576fd8e3-f3a0-48d2-a0d6-faa782a25a63\") " pod="kube-system/global-pull-secret-syncer-cg8ph" Jul 06 13:50:26.910897 ip-10-0-136-25 kubenswrapper[2569]: I0706 13:50:26.910078 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/ef1daf8b-7b12-4c6a-9fbd-2b7e9187fd8f-hosts-file\") pod \"node-resolver-54w6r\" (UID: \"ef1daf8b-7b12-4c6a-9fbd-2b7e9187fd8f\") " pod="openshift-dns/node-resolver-54w6r" Jul 06 13:50:26.910897 ip-10-0-136-25 kubenswrapper[2569]: I0706 13:50:26.910092 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/b7576bc8-a480-44ac-b68c-36f9abe93b7d-sys\") pod \"tuned-2x29s\" (UID: \"b7576bc8-a480-44ac-b68c-36f9abe93b7d\") " pod="openshift-cluster-node-tuning-operator/tuned-2x29s" Jul 06 13:50:26.910897 ip-10-0-136-25 kubenswrapper[2569]: I0706 13:50:26.910104 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/4ae58dad-494b-4508-b5a5-129f13818d53-var-lib-openvswitch\") pod \"ovnkube-node-vcxnc\" (UID: \"4ae58dad-494b-4508-b5a5-129f13818d53\") " pod="openshift-ovn-kubernetes/ovnkube-node-vcxnc" Jul 06 13:50:26.910897 ip-10-0-136-25 kubenswrapper[2569]: I0706 13:50:26.910130 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/edbb90e8-546b-4edd-a223-5ad4d579adaa-os-release\") pod \"multus-additional-cni-plugins-brmhg\" (UID: \"edbb90e8-546b-4edd-a223-5ad4d579adaa\") " pod="openshift-multus/multus-additional-cni-plugins-brmhg" Jul 06 13:50:26.910897 ip-10-0-136-25 kubenswrapper[2569]: I0706 13:50:26.910152 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/4ae58dad-494b-4508-b5a5-129f13818d53-ovnkube-script-lib\") pod \"ovnkube-node-vcxnc\" (UID: \"4ae58dad-494b-4508-b5a5-129f13818d53\") " pod="openshift-ovn-kubernetes/ovnkube-node-vcxnc" Jul 06 13:50:26.911746 ip-10-0-136-25 kubenswrapper[2569]: I0706 13:50:26.910155 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/b7576bc8-a480-44ac-b68c-36f9abe93b7d-etc-sysctl-d\") pod \"tuned-2x29s\" (UID: \"b7576bc8-a480-44ac-b68c-36f9abe93b7d\") " pod="openshift-cluster-node-tuning-operator/tuned-2x29s" Jul 06 13:50:26.911746 ip-10-0-136-25 kubenswrapper[2569]: I0706 13:50:26.910194 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/07d7de38-90f7-4810-b5d6-3a710a3af5a9-cnibin\") pod \"multus-dcdgc\" (UID: \"07d7de38-90f7-4810-b5d6-3a710a3af5a9\") " pod="openshift-multus/multus-dcdgc" Jul 06 13:50:26.911746 ip-10-0-136-25 kubenswrapper[2569]: I0706 13:50:26.910240 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/07d7de38-90f7-4810-b5d6-3a710a3af5a9-os-release\") pod \"multus-dcdgc\" (UID: \"07d7de38-90f7-4810-b5d6-3a710a3af5a9\") " pod="openshift-multus/multus-dcdgc" Jul 06 13:50:26.911746 ip-10-0-136-25 kubenswrapper[2569]: I0706 13:50:26.910278 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/cb1807ec-0568-4ab3-a9ca-e453709b820e-agent-certs\") pod \"konnectivity-agent-vqmcq\" (UID: \"cb1807ec-0568-4ab3-a9ca-e453709b820e\") " pod="kube-system/konnectivity-agent-vqmcq" Jul 06 13:50:26.911746 ip-10-0-136-25 kubenswrapper[2569]: I0706 13:50:26.910290 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/07d7de38-90f7-4810-b5d6-3a710a3af5a9-os-release\") pod \"multus-dcdgc\" (UID: \"07d7de38-90f7-4810-b5d6-3a710a3af5a9\") " pod="openshift-multus/multus-dcdgc" Jul 06 13:50:26.911746 ip-10-0-136-25 kubenswrapper[2569]: I0706 13:50:26.910296 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/b7576bc8-a480-44ac-b68c-36f9abe93b7d-etc-sysctl-d\") pod \"tuned-2x29s\" (UID: \"b7576bc8-a480-44ac-b68c-36f9abe93b7d\") " pod="openshift-cluster-node-tuning-operator/tuned-2x29s" Jul 06 13:50:26.911746 ip-10-0-136-25 kubenswrapper[2569]: I0706 13:50:26.909979 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/5bbb0d3f-2cd8-4496-b676-6045376a5b13-host-slash\") pod \"iptables-alerter-wxxjp\" (UID: \"5bbb0d3f-2cd8-4496-b676-6045376a5b13\") " pod="openshift-network-operator/iptables-alerter-wxxjp" Jul 06 13:50:26.911746 ip-10-0-136-25 kubenswrapper[2569]: I0706 13:50:26.910336 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/edbb90e8-546b-4edd-a223-5ad4d579adaa-cni-binary-copy\") pod \"multus-additional-cni-plugins-brmhg\" (UID: \"edbb90e8-546b-4edd-a223-5ad4d579adaa\") " pod="openshift-multus/multus-additional-cni-plugins-brmhg" Jul 06 13:50:26.911746 ip-10-0-136-25 kubenswrapper[2569]: I0706 13:50:26.910344 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/cb1807ec-0568-4ab3-a9ca-e453709b820e-konnectivity-ca\") pod \"konnectivity-agent-vqmcq\" (UID: \"cb1807ec-0568-4ab3-a9ca-e453709b820e\") " pod="kube-system/konnectivity-agent-vqmcq" Jul 06 13:50:26.911746 ip-10-0-136-25 kubenswrapper[2569]: I0706 13:50:26.910351 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/07d7de38-90f7-4810-b5d6-3a710a3af5a9-host-var-lib-cni-multus\") pod \"multus-dcdgc\" (UID: \"07d7de38-90f7-4810-b5d6-3a710a3af5a9\") " pod="openshift-multus/multus-dcdgc" Jul 06 13:50:26.911746 ip-10-0-136-25 kubenswrapper[2569]: I0706 13:50:26.910056 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/edbb90e8-546b-4edd-a223-5ad4d579adaa-cnibin\") pod \"multus-additional-cni-plugins-brmhg\" (UID: \"edbb90e8-546b-4edd-a223-5ad4d579adaa\") " pod="openshift-multus/multus-additional-cni-plugins-brmhg" Jul 06 13:50:26.911746 ip-10-0-136-25 kubenswrapper[2569]: I0706 13:50:26.910351 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/07d7de38-90f7-4810-b5d6-3a710a3af5a9-cnibin\") pod \"multus-dcdgc\" (UID: \"07d7de38-90f7-4810-b5d6-3a710a3af5a9\") " pod="openshift-multus/multus-dcdgc" Jul 06 13:50:26.911746 ip-10-0-136-25 kubenswrapper[2569]: I0706 13:50:26.910373 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/b7576bc8-a480-44ac-b68c-36f9abe93b7d-etc-sysconfig\") pod \"tuned-2x29s\" (UID: \"b7576bc8-a480-44ac-b68c-36f9abe93b7d\") " pod="openshift-cluster-node-tuning-operator/tuned-2x29s" Jul 06 13:50:26.911746 ip-10-0-136-25 kubenswrapper[2569]: I0706 13:50:26.910408 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/b7576bc8-a480-44ac-b68c-36f9abe93b7d-etc-systemd\") pod \"tuned-2x29s\" (UID: \"b7576bc8-a480-44ac-b68c-36f9abe93b7d\") " pod="openshift-cluster-node-tuning-operator/tuned-2x29s" Jul 06 13:50:26.911746 ip-10-0-136-25 kubenswrapper[2569]: I0706 13:50:26.910420 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/4ae58dad-494b-4508-b5a5-129f13818d53-run-openvswitch\") pod \"ovnkube-node-vcxnc\" (UID: \"4ae58dad-494b-4508-b5a5-129f13818d53\") " pod="openshift-ovn-kubernetes/ovnkube-node-vcxnc" Jul 06 13:50:26.911746 ip-10-0-136-25 kubenswrapper[2569]: I0706 13:50:26.910427 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/07d7de38-90f7-4810-b5d6-3a710a3af5a9-hostroot\") pod \"multus-dcdgc\" (UID: \"07d7de38-90f7-4810-b5d6-3a710a3af5a9\") " pod="openshift-multus/multus-dcdgc" Jul 06 13:50:26.911746 ip-10-0-136-25 kubenswrapper[2569]: I0706 13:50:26.910465 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/4ae58dad-494b-4508-b5a5-129f13818d53-var-lib-openvswitch\") pod \"ovnkube-node-vcxnc\" (UID: \"4ae58dad-494b-4508-b5a5-129f13818d53\") " pod="openshift-ovn-kubernetes/ovnkube-node-vcxnc" Jul 06 13:50:26.911746 ip-10-0-136-25 kubenswrapper[2569]: I0706 13:50:26.910476 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/4ae58dad-494b-4508-b5a5-129f13818d53-host-slash\") pod \"ovnkube-node-vcxnc\" (UID: \"4ae58dad-494b-4508-b5a5-129f13818d53\") " pod="openshift-ovn-kubernetes/ovnkube-node-vcxnc" Jul 06 13:50:26.912601 ip-10-0-136-25 kubenswrapper[2569]: I0706 13:50:26.910481 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/edbb90e8-546b-4edd-a223-5ad4d579adaa-os-release\") pod \"multus-additional-cni-plugins-brmhg\" (UID: \"edbb90e8-546b-4edd-a223-5ad4d579adaa\") " pod="openshift-multus/multus-additional-cni-plugins-brmhg" Jul 06 13:50:26.912601 ip-10-0-136-25 kubenswrapper[2569]: I0706 13:50:26.910517 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/b7576bc8-a480-44ac-b68c-36f9abe93b7d-etc-sysconfig\") pod \"tuned-2x29s\" (UID: \"b7576bc8-a480-44ac-b68c-36f9abe93b7d\") " pod="openshift-cluster-node-tuning-operator/tuned-2x29s" Jul 06 13:50:26.912601 ip-10-0-136-25 kubenswrapper[2569]: I0706 13:50:26.910525 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/07d7de38-90f7-4810-b5d6-3a710a3af5a9-system-cni-dir\") pod \"multus-dcdgc\" (UID: \"07d7de38-90f7-4810-b5d6-3a710a3af5a9\") " pod="openshift-multus/multus-dcdgc" Jul 06 13:50:26.912601 ip-10-0-136-25 kubenswrapper[2569]: I0706 13:50:26.910549 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/4ae58dad-494b-4508-b5a5-129f13818d53-run-systemd\") pod \"ovnkube-node-vcxnc\" (UID: \"4ae58dad-494b-4508-b5a5-129f13818d53\") " pod="openshift-ovn-kubernetes/ovnkube-node-vcxnc" Jul 06 13:50:26.912601 ip-10-0-136-25 kubenswrapper[2569]: I0706 13:50:26.910556 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gf7rl\" (UniqueName: \"kubernetes.io/projected/ef1daf8b-7b12-4c6a-9fbd-2b7e9187fd8f-kube-api-access-gf7rl\") pod \"node-resolver-54w6r\" (UID: \"ef1daf8b-7b12-4c6a-9fbd-2b7e9187fd8f\") " pod="openshift-dns/node-resolver-54w6r" Jul 06 13:50:26.912601 ip-10-0-136-25 kubenswrapper[2569]: I0706 13:50:26.910583 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/4ae58dad-494b-4508-b5a5-129f13818d53-host-kubelet\") pod \"ovnkube-node-vcxnc\" (UID: \"4ae58dad-494b-4508-b5a5-129f13818d53\") " pod="openshift-ovn-kubernetes/ovnkube-node-vcxnc" Jul 06 13:50:26.912601 ip-10-0-136-25 kubenswrapper[2569]: I0706 13:50:26.910607 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/4ae58dad-494b-4508-b5a5-129f13818d53-env-overrides\") pod \"ovnkube-node-vcxnc\" (UID: \"4ae58dad-494b-4508-b5a5-129f13818d53\") " pod="openshift-ovn-kubernetes/ovnkube-node-vcxnc" Jul 06 13:50:26.912601 ip-10-0-136-25 kubenswrapper[2569]: I0706 13:50:26.910618 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/5bbb0d3f-2cd8-4496-b676-6045376a5b13-iptables-alerter-script\") pod \"iptables-alerter-wxxjp\" (UID: \"5bbb0d3f-2cd8-4496-b676-6045376a5b13\") " pod="openshift-network-operator/iptables-alerter-wxxjp" Jul 06 13:50:26.912601 ip-10-0-136-25 kubenswrapper[2569]: I0706 13:50:26.910622 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/4ae58dad-494b-4508-b5a5-129f13818d53-host-kubelet\") pod \"ovnkube-node-vcxnc\" (UID: \"4ae58dad-494b-4508-b5a5-129f13818d53\") " pod="openshift-ovn-kubernetes/ovnkube-node-vcxnc" Jul 06 13:50:26.912601 ip-10-0-136-25 kubenswrapper[2569]: I0706 13:50:26.910630 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/b7576bc8-a480-44ac-b68c-36f9abe93b7d-etc-modprobe-d\") pod \"tuned-2x29s\" (UID: \"b7576bc8-a480-44ac-b68c-36f9abe93b7d\") " pod="openshift-cluster-node-tuning-operator/tuned-2x29s" Jul 06 13:50:26.912601 ip-10-0-136-25 kubenswrapper[2569]: I0706 13:50:26.910585 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/07d7de38-90f7-4810-b5d6-3a710a3af5a9-system-cni-dir\") pod \"multus-dcdgc\" (UID: \"07d7de38-90f7-4810-b5d6-3a710a3af5a9\") " pod="openshift-multus/multus-dcdgc" Jul 06 13:50:26.912601 ip-10-0-136-25 kubenswrapper[2569]: I0706 13:50:26.910671 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/4ae58dad-494b-4508-b5a5-129f13818d53-host-cni-bin\") pod \"ovnkube-node-vcxnc\" (UID: \"4ae58dad-494b-4508-b5a5-129f13818d53\") " pod="openshift-ovn-kubernetes/ovnkube-node-vcxnc" Jul 06 13:50:26.912601 ip-10-0-136-25 kubenswrapper[2569]: I0706 13:50:26.910696 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/b7576bc8-a480-44ac-b68c-36f9abe93b7d-etc-kubernetes\") pod \"tuned-2x29s\" (UID: \"b7576bc8-a480-44ac-b68c-36f9abe93b7d\") " pod="openshift-cluster-node-tuning-operator/tuned-2x29s" Jul 06 13:50:26.912601 ip-10-0-136-25 kubenswrapper[2569]: I0706 13:50:26.910703 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/b7576bc8-a480-44ac-b68c-36f9abe93b7d-etc-modprobe-d\") pod \"tuned-2x29s\" (UID: \"b7576bc8-a480-44ac-b68c-36f9abe93b7d\") " pod="openshift-cluster-node-tuning-operator/tuned-2x29s" Jul 06 13:50:26.912601 ip-10-0-136-25 kubenswrapper[2569]: I0706 13:50:26.910719 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4kcgx\" (UniqueName: \"kubernetes.io/projected/d2d86b02-1e00-4267-b9c4-dd365b15342e-kube-api-access-4kcgx\") pod \"network-check-target-gbq5d\" (UID: \"d2d86b02-1e00-4267-b9c4-dd365b15342e\") " pod="openshift-network-diagnostics/network-check-target-gbq5d" Jul 06 13:50:26.912601 ip-10-0-136-25 kubenswrapper[2569]: I0706 13:50:26.910741 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/4ae58dad-494b-4508-b5a5-129f13818d53-host-cni-bin\") pod \"ovnkube-node-vcxnc\" (UID: \"4ae58dad-494b-4508-b5a5-129f13818d53\") " pod="openshift-ovn-kubernetes/ovnkube-node-vcxnc" Jul 06 13:50:26.912601 ip-10-0-136-25 kubenswrapper[2569]: I0706 13:50:26.910744 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/07d7de38-90f7-4810-b5d6-3a710a3af5a9-multus-cni-dir\") pod \"multus-dcdgc\" (UID: \"07d7de38-90f7-4810-b5d6-3a710a3af5a9\") " pod="openshift-multus/multus-dcdgc" Jul 06 13:50:26.913467 ip-10-0-136-25 kubenswrapper[2569]: I0706 13:50:26.910792 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/b7576bc8-a480-44ac-b68c-36f9abe93b7d-etc-kubernetes\") pod \"tuned-2x29s\" (UID: \"b7576bc8-a480-44ac-b68c-36f9abe93b7d\") " pod="openshift-cluster-node-tuning-operator/tuned-2x29s" Jul 06 13:50:26.913467 ip-10-0-136-25 kubenswrapper[2569]: I0706 13:50:26.910820 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/07d7de38-90f7-4810-b5d6-3a710a3af5a9-cni-binary-copy\") pod \"multus-dcdgc\" (UID: \"07d7de38-90f7-4810-b5d6-3a710a3af5a9\") " pod="openshift-multus/multus-dcdgc" Jul 06 13:50:26.913467 ip-10-0-136-25 kubenswrapper[2569]: I0706 13:50:26.910844 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/07d7de38-90f7-4810-b5d6-3a710a3af5a9-host-var-lib-cni-bin\") pod \"multus-dcdgc\" (UID: \"07d7de38-90f7-4810-b5d6-3a710a3af5a9\") " pod="openshift-multus/multus-dcdgc" Jul 06 13:50:26.913467 ip-10-0-136-25 kubenswrapper[2569]: I0706 13:50:26.910863 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/07d7de38-90f7-4810-b5d6-3a710a3af5a9-multus-cni-dir\") pod \"multus-dcdgc\" (UID: \"07d7de38-90f7-4810-b5d6-3a710a3af5a9\") " pod="openshift-multus/multus-dcdgc" Jul 06 13:50:26.913467 ip-10-0-136-25 kubenswrapper[2569]: I0706 13:50:26.910866 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/b7576bc8-a480-44ac-b68c-36f9abe93b7d-var-lib-kubelet\") pod \"tuned-2x29s\" (UID: \"b7576bc8-a480-44ac-b68c-36f9abe93b7d\") " pod="openshift-cluster-node-tuning-operator/tuned-2x29s" Jul 06 13:50:26.913467 ip-10-0-136-25 kubenswrapper[2569]: I0706 13:50:26.910888 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/b7576bc8-a480-44ac-b68c-36f9abe93b7d-host\") pod \"tuned-2x29s\" (UID: \"b7576bc8-a480-44ac-b68c-36f9abe93b7d\") " pod="openshift-cluster-node-tuning-operator/tuned-2x29s" Jul 06 13:50:26.913467 ip-10-0-136-25 kubenswrapper[2569]: I0706 13:50:26.910910 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/cb1807ec-0568-4ab3-a9ca-e453709b820e-konnectivity-ca\") pod \"konnectivity-agent-vqmcq\" (UID: \"cb1807ec-0568-4ab3-a9ca-e453709b820e\") " pod="kube-system/konnectivity-agent-vqmcq" Jul 06 13:50:26.913467 ip-10-0-136-25 kubenswrapper[2569]: I0706 13:50:26.910945 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/07d7de38-90f7-4810-b5d6-3a710a3af5a9-host-var-lib-kubelet\") pod \"multus-dcdgc\" (UID: \"07d7de38-90f7-4810-b5d6-3a710a3af5a9\") " pod="openshift-multus/multus-dcdgc" Jul 06 13:50:26.913467 ip-10-0-136-25 kubenswrapper[2569]: I0706 13:50:26.910965 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/b7576bc8-a480-44ac-b68c-36f9abe93b7d-etc-systemd\") pod \"tuned-2x29s\" (UID: \"b7576bc8-a480-44ac-b68c-36f9abe93b7d\") " pod="openshift-cluster-node-tuning-operator/tuned-2x29s" Jul 06 13:50:26.913467 ip-10-0-136-25 kubenswrapper[2569]: I0706 13:50:26.910968 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/4ae58dad-494b-4508-b5a5-129f13818d53-env-overrides\") pod \"ovnkube-node-vcxnc\" (UID: \"4ae58dad-494b-4508-b5a5-129f13818d53\") " pod="openshift-ovn-kubernetes/ovnkube-node-vcxnc" Jul 06 13:50:26.913467 ip-10-0-136-25 kubenswrapper[2569]: I0706 13:50:26.910967 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/b7576bc8-a480-44ac-b68c-36f9abe93b7d-tmp\") pod \"tuned-2x29s\" (UID: \"b7576bc8-a480-44ac-b68c-36f9abe93b7d\") " pod="openshift-cluster-node-tuning-operator/tuned-2x29s" Jul 06 13:50:26.913467 ip-10-0-136-25 kubenswrapper[2569]: I0706 13:50:26.911019 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/07d7de38-90f7-4810-b5d6-3a710a3af5a9-host-var-lib-cni-bin\") pod \"multus-dcdgc\" (UID: \"07d7de38-90f7-4810-b5d6-3a710a3af5a9\") " pod="openshift-multus/multus-dcdgc" Jul 06 13:50:26.913467 ip-10-0-136-25 kubenswrapper[2569]: I0706 13:50:26.911043 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/07d7de38-90f7-4810-b5d6-3a710a3af5a9-multus-socket-dir-parent\") pod \"multus-dcdgc\" (UID: \"07d7de38-90f7-4810-b5d6-3a710a3af5a9\") " pod="openshift-multus/multus-dcdgc" Jul 06 13:50:26.913467 ip-10-0-136-25 kubenswrapper[2569]: I0706 13:50:26.911066 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ef1daf8b-7b12-4c6a-9fbd-2b7e9187fd8f-tmp-dir\") pod \"node-resolver-54w6r\" (UID: \"ef1daf8b-7b12-4c6a-9fbd-2b7e9187fd8f\") " pod="openshift-dns/node-resolver-54w6r" Jul 06 13:50:26.913467 ip-10-0-136-25 kubenswrapper[2569]: I0706 13:50:26.911090 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/4ae58dad-494b-4508-b5a5-129f13818d53-host-run-netns\") pod \"ovnkube-node-vcxnc\" (UID: \"4ae58dad-494b-4508-b5a5-129f13818d53\") " pod="openshift-ovn-kubernetes/ovnkube-node-vcxnc" Jul 06 13:50:26.913467 ip-10-0-136-25 kubenswrapper[2569]: I0706 13:50:26.911113 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/4ae58dad-494b-4508-b5a5-129f13818d53-host-cni-netd\") pod \"ovnkube-node-vcxnc\" (UID: \"4ae58dad-494b-4508-b5a5-129f13818d53\") " pod="openshift-ovn-kubernetes/ovnkube-node-vcxnc" Jul 06 13:50:26.913467 ip-10-0-136-25 kubenswrapper[2569]: I0706 13:50:26.911131 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/07d7de38-90f7-4810-b5d6-3a710a3af5a9-multus-socket-dir-parent\") pod \"multus-dcdgc\" (UID: \"07d7de38-90f7-4810-b5d6-3a710a3af5a9\") " pod="openshift-multus/multus-dcdgc" Jul 06 13:50:26.913467 ip-10-0-136-25 kubenswrapper[2569]: I0706 13:50:26.911138 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4xkx2\" (UniqueName: \"kubernetes.io/projected/028cbff5-11b2-411a-90ed-c0bd3f563830-kube-api-access-4xkx2\") pod \"node-ca-c9sqv\" (UID: \"028cbff5-11b2-411a-90ed-c0bd3f563830\") " pod="openshift-image-registry/node-ca-c9sqv" Jul 06 13:50:26.914317 ip-10-0-136-25 kubenswrapper[2569]: I0706 13:50:26.911171 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/03d48054-9789-444b-82c7-ac9e3a504c9c-device-dir\") pod \"aws-ebs-csi-driver-node-2dnkc\" (UID: \"03d48054-9789-444b-82c7-ac9e3a504c9c\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-2dnkc" Jul 06 13:50:26.914317 ip-10-0-136-25 kubenswrapper[2569]: I0706 13:50:26.911199 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/576fd8e3-f3a0-48d2-a0d6-faa782a25a63-original-pull-secret\") pod \"global-pull-secret-syncer-cg8ph\" (UID: \"576fd8e3-f3a0-48d2-a0d6-faa782a25a63\") " pod="kube-system/global-pull-secret-syncer-cg8ph" Jul 06 13:50:26.914317 ip-10-0-136-25 kubenswrapper[2569]: I0706 13:50:26.911251 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/4ae58dad-494b-4508-b5a5-129f13818d53-node-log\") pod \"ovnkube-node-vcxnc\" (UID: \"4ae58dad-494b-4508-b5a5-129f13818d53\") " pod="openshift-ovn-kubernetes/ovnkube-node-vcxnc" Jul 06 13:50:26.914317 ip-10-0-136-25 kubenswrapper[2569]: I0706 13:50:26.911290 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/4ae58dad-494b-4508-b5a5-129f13818d53-host-run-ovn-kubernetes\") pod \"ovnkube-node-vcxnc\" (UID: \"4ae58dad-494b-4508-b5a5-129f13818d53\") " pod="openshift-ovn-kubernetes/ovnkube-node-vcxnc" Jul 06 13:50:26.914317 ip-10-0-136-25 kubenswrapper[2569]: I0706 13:50:26.911319 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/4ae58dad-494b-4508-b5a5-129f13818d53-ovnkube-config\") pod \"ovnkube-node-vcxnc\" (UID: \"4ae58dad-494b-4508-b5a5-129f13818d53\") " pod="openshift-ovn-kubernetes/ovnkube-node-vcxnc" Jul 06 13:50:26.914317 ip-10-0-136-25 kubenswrapper[2569]: I0706 13:50:26.911344 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/edbb90e8-546b-4edd-a223-5ad4d579adaa-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-brmhg\" (UID: \"edbb90e8-546b-4edd-a223-5ad4d579adaa\") " pod="openshift-multus/multus-additional-cni-plugins-brmhg" Jul 06 13:50:26.914317 ip-10-0-136-25 kubenswrapper[2569]: I0706 13:50:26.911370 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/edbb90e8-546b-4edd-a223-5ad4d579adaa-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-brmhg\" (UID: \"edbb90e8-546b-4edd-a223-5ad4d579adaa\") " pod="openshift-multus/multus-additional-cni-plugins-brmhg" Jul 06 13:50:26.914317 ip-10-0-136-25 kubenswrapper[2569]: I0706 13:50:26.911398 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/07d7de38-90f7-4810-b5d6-3a710a3af5a9-multus-conf-dir\") pod \"multus-dcdgc\" (UID: \"07d7de38-90f7-4810-b5d6-3a710a3af5a9\") " pod="openshift-multus/multus-dcdgc" Jul 06 13:50:26.914317 ip-10-0-136-25 kubenswrapper[2569]: I0706 13:50:26.911404 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/4ae58dad-494b-4508-b5a5-129f13818d53-host-run-netns\") pod \"ovnkube-node-vcxnc\" (UID: \"4ae58dad-494b-4508-b5a5-129f13818d53\") " pod="openshift-ovn-kubernetes/ovnkube-node-vcxnc" Jul 06 13:50:26.914317 ip-10-0-136-25 kubenswrapper[2569]: I0706 13:50:26.911422 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/07d7de38-90f7-4810-b5d6-3a710a3af5a9-multus-daemon-config\") pod \"multus-dcdgc\" (UID: \"07d7de38-90f7-4810-b5d6-3a710a3af5a9\") " pod="openshift-multus/multus-dcdgc" Jul 06 13:50:26.914317 ip-10-0-136-25 kubenswrapper[2569]: I0706 13:50:26.911451 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fxb8c\" (UniqueName: \"kubernetes.io/projected/4ae58dad-494b-4508-b5a5-129f13818d53-kube-api-access-fxb8c\") pod \"ovnkube-node-vcxnc\" (UID: \"4ae58dad-494b-4508-b5a5-129f13818d53\") " pod="openshift-ovn-kubernetes/ovnkube-node-vcxnc" Jul 06 13:50:26.914317 ip-10-0-136-25 kubenswrapper[2569]: I0706 13:50:26.911475 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/07d7de38-90f7-4810-b5d6-3a710a3af5a9-cni-binary-copy\") pod \"multus-dcdgc\" (UID: \"07d7de38-90f7-4810-b5d6-3a710a3af5a9\") " pod="openshift-multus/multus-dcdgc" Jul 06 13:50:26.914317 ip-10-0-136-25 kubenswrapper[2569]: I0706 13:50:26.911504 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/4ae58dad-494b-4508-b5a5-129f13818d53-node-log\") pod \"ovnkube-node-vcxnc\" (UID: \"4ae58dad-494b-4508-b5a5-129f13818d53\") " pod="openshift-ovn-kubernetes/ovnkube-node-vcxnc" Jul 06 13:50:26.914317 ip-10-0-136-25 kubenswrapper[2569]: I0706 13:50:26.911548 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/4ae58dad-494b-4508-b5a5-129f13818d53-host-run-ovn-kubernetes\") pod \"ovnkube-node-vcxnc\" (UID: \"4ae58dad-494b-4508-b5a5-129f13818d53\") " pod="openshift-ovn-kubernetes/ovnkube-node-vcxnc" Jul 06 13:50:26.914317 ip-10-0-136-25 kubenswrapper[2569]: I0706 13:50:26.911589 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/07d7de38-90f7-4810-b5d6-3a710a3af5a9-multus-conf-dir\") pod \"multus-dcdgc\" (UID: \"07d7de38-90f7-4810-b5d6-3a710a3af5a9\") " pod="openshift-multus/multus-dcdgc" Jul 06 13:50:26.914317 ip-10-0-136-25 kubenswrapper[2569]: I0706 13:50:26.911477 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7nnml\" (UniqueName: \"kubernetes.io/projected/b7576bc8-a480-44ac-b68c-36f9abe93b7d-kube-api-access-7nnml\") pod \"tuned-2x29s\" (UID: \"b7576bc8-a480-44ac-b68c-36f9abe93b7d\") " pod="openshift-cluster-node-tuning-operator/tuned-2x29s" Jul 06 13:50:26.914317 ip-10-0-136-25 kubenswrapper[2569]: I0706 13:50:26.911626 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/07d7de38-90f7-4810-b5d6-3a710a3af5a9-host-run-multus-certs\") pod \"multus-dcdgc\" (UID: \"07d7de38-90f7-4810-b5d6-3a710a3af5a9\") " pod="openshift-multus/multus-dcdgc" Jul 06 13:50:26.915089 ip-10-0-136-25 kubenswrapper[2569]: I0706 13:50:26.911694 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/07d7de38-90f7-4810-b5d6-3a710a3af5a9-host-run-multus-certs\") pod \"multus-dcdgc\" (UID: \"07d7de38-90f7-4810-b5d6-3a710a3af5a9\") " pod="openshift-multus/multus-dcdgc" Jul 06 13:50:26.915089 ip-10-0-136-25 kubenswrapper[2569]: I0706 13:50:26.911730 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/07d7de38-90f7-4810-b5d6-3a710a3af5a9-host-var-lib-kubelet\") pod \"multus-dcdgc\" (UID: \"07d7de38-90f7-4810-b5d6-3a710a3af5a9\") " pod="openshift-multus/multus-dcdgc" Jul 06 13:50:26.915089 ip-10-0-136-25 kubenswrapper[2569]: I0706 13:50:26.911774 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/b7576bc8-a480-44ac-b68c-36f9abe93b7d-host\") pod \"tuned-2x29s\" (UID: \"b7576bc8-a480-44ac-b68c-36f9abe93b7d\") " pod="openshift-cluster-node-tuning-operator/tuned-2x29s" Jul 06 13:50:26.915089 ip-10-0-136-25 kubenswrapper[2569]: I0706 13:50:26.911907 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/edbb90e8-546b-4edd-a223-5ad4d579adaa-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-brmhg\" (UID: \"edbb90e8-546b-4edd-a223-5ad4d579adaa\") " pod="openshift-multus/multus-additional-cni-plugins-brmhg" Jul 06 13:50:26.915089 ip-10-0-136-25 kubenswrapper[2569]: I0706 13:50:26.911449 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/4ae58dad-494b-4508-b5a5-129f13818d53-host-cni-netd\") pod \"ovnkube-node-vcxnc\" (UID: \"4ae58dad-494b-4508-b5a5-129f13818d53\") " pod="openshift-ovn-kubernetes/ovnkube-node-vcxnc" Jul 06 13:50:26.915089 ip-10-0-136-25 kubenswrapper[2569]: I0706 13:50:26.912058 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/edbb90e8-546b-4edd-a223-5ad4d579adaa-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-brmhg\" (UID: \"edbb90e8-546b-4edd-a223-5ad4d579adaa\") " pod="openshift-multus/multus-additional-cni-plugins-brmhg" Jul 06 13:50:26.915089 ip-10-0-136-25 kubenswrapper[2569]: I0706 13:50:26.912118 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/b7576bc8-a480-44ac-b68c-36f9abe93b7d-var-lib-kubelet\") pod \"tuned-2x29s\" (UID: \"b7576bc8-a480-44ac-b68c-36f9abe93b7d\") " pod="openshift-cluster-node-tuning-operator/tuned-2x29s" Jul 06 13:50:26.915089 ip-10-0-136-25 kubenswrapper[2569]: I0706 13:50:26.912404 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/4ae58dad-494b-4508-b5a5-129f13818d53-ovnkube-config\") pod \"ovnkube-node-vcxnc\" (UID: \"4ae58dad-494b-4508-b5a5-129f13818d53\") " pod="openshift-ovn-kubernetes/ovnkube-node-vcxnc" Jul 06 13:50:26.915089 ip-10-0-136-25 kubenswrapper[2569]: I0706 13:50:26.912446 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/07d7de38-90f7-4810-b5d6-3a710a3af5a9-multus-daemon-config\") pod \"multus-dcdgc\" (UID: \"07d7de38-90f7-4810-b5d6-3a710a3af5a9\") " pod="openshift-multus/multus-dcdgc" Jul 06 13:50:26.915089 ip-10-0-136-25 kubenswrapper[2569]: I0706 13:50:26.914185 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/b7576bc8-a480-44ac-b68c-36f9abe93b7d-tmp\") pod \"tuned-2x29s\" (UID: \"b7576bc8-a480-44ac-b68c-36f9abe93b7d\") " pod="openshift-cluster-node-tuning-operator/tuned-2x29s" Jul 06 13:50:26.915089 ip-10-0-136-25 kubenswrapper[2569]: I0706 13:50:26.914187 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/b7576bc8-a480-44ac-b68c-36f9abe93b7d-etc-tuned\") pod \"tuned-2x29s\" (UID: \"b7576bc8-a480-44ac-b68c-36f9abe93b7d\") " pod="openshift-cluster-node-tuning-operator/tuned-2x29s" Jul 06 13:50:26.915089 ip-10-0-136-25 kubenswrapper[2569]: I0706 13:50:26.914295 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/4ae58dad-494b-4508-b5a5-129f13818d53-ovn-node-metrics-cert\") pod \"ovnkube-node-vcxnc\" (UID: \"4ae58dad-494b-4508-b5a5-129f13818d53\") " pod="openshift-ovn-kubernetes/ovnkube-node-vcxnc" Jul 06 13:50:26.915089 ip-10-0-136-25 kubenswrapper[2569]: I0706 13:50:26.914513 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/cb1807ec-0568-4ab3-a9ca-e453709b820e-agent-certs\") pod \"konnectivity-agent-vqmcq\" (UID: \"cb1807ec-0568-4ab3-a9ca-e453709b820e\") " pod="kube-system/konnectivity-agent-vqmcq" Jul 06 13:50:26.922737 ip-10-0-136-25 kubenswrapper[2569]: I0706 13:50:26.922715 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4xkx2\" (UniqueName: \"kubernetes.io/projected/028cbff5-11b2-411a-90ed-c0bd3f563830-kube-api-access-4xkx2\") pod \"node-ca-c9sqv\" (UID: \"028cbff5-11b2-411a-90ed-c0bd3f563830\") " pod="openshift-image-registry/node-ca-c9sqv" Jul 06 13:50:26.922950 ip-10-0-136-25 kubenswrapper[2569]: I0706 13:50:26.922771 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mwkhd\" (UniqueName: \"kubernetes.io/projected/07d7de38-90f7-4810-b5d6-3a710a3af5a9-kube-api-access-mwkhd\") pod \"multus-dcdgc\" (UID: \"07d7de38-90f7-4810-b5d6-3a710a3af5a9\") " pod="openshift-multus/multus-dcdgc" Jul 06 13:50:26.926103 ip-10-0-136-25 kubenswrapper[2569]: E0706 13:50:26.926076 2569 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 06 13:50:26.926103 ip-10-0-136-25 kubenswrapper[2569]: E0706 13:50:26.926105 2569 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 06 13:50:26.926265 ip-10-0-136-25 kubenswrapper[2569]: E0706 13:50:26.926118 2569 projected.go:196] Error preparing data for projected volume kube-api-access-4kcgx for pod openshift-network-diagnostics/network-check-target-gbq5d: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 06 13:50:26.926265 ip-10-0-136-25 kubenswrapper[2569]: E0706 13:50:26.926203 2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/d2d86b02-1e00-4267-b9c4-dd365b15342e-kube-api-access-4kcgx podName:d2d86b02-1e00-4267-b9c4-dd365b15342e nodeName:}" failed. No retries permitted until 2026-07-06 13:50:27.426181445 +0000 UTC m=+3.016743217 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-api-access-4kcgx" (UniqueName: "kubernetes.io/projected/d2d86b02-1e00-4267-b9c4-dd365b15342e-kube-api-access-4kcgx") pod "network-check-target-gbq5d" (UID: "d2d86b02-1e00-4267-b9c4-dd365b15342e") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 06 13:50:26.927703 ip-10-0-136-25 kubenswrapper[2569]: I0706 13:50:26.927675 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xd6qh\" (UniqueName: \"kubernetes.io/projected/5bbb0d3f-2cd8-4496-b676-6045376a5b13-kube-api-access-xd6qh\") pod \"iptables-alerter-wxxjp\" (UID: \"5bbb0d3f-2cd8-4496-b676-6045376a5b13\") " pod="openshift-network-operator/iptables-alerter-wxxjp" Jul 06 13:50:26.927806 ip-10-0-136-25 kubenswrapper[2569]: I0706 13:50:26.927730 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fxb8c\" (UniqueName: \"kubernetes.io/projected/4ae58dad-494b-4508-b5a5-129f13818d53-kube-api-access-fxb8c\") pod \"ovnkube-node-vcxnc\" (UID: \"4ae58dad-494b-4508-b5a5-129f13818d53\") " pod="openshift-ovn-kubernetes/ovnkube-node-vcxnc" Jul 06 13:50:26.928114 ip-10-0-136-25 kubenswrapper[2569]: I0706 13:50:26.928097 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7nnml\" (UniqueName: \"kubernetes.io/projected/b7576bc8-a480-44ac-b68c-36f9abe93b7d-kube-api-access-7nnml\") pod \"tuned-2x29s\" (UID: \"b7576bc8-a480-44ac-b68c-36f9abe93b7d\") " pod="openshift-cluster-node-tuning-operator/tuned-2x29s" Jul 06 13:50:26.928638 ip-10-0-136-25 kubenswrapper[2569]: I0706 13:50:26.928616 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5gxvp\" (UniqueName: \"kubernetes.io/projected/edbb90e8-546b-4edd-a223-5ad4d579adaa-kube-api-access-5gxvp\") pod \"multus-additional-cni-plugins-brmhg\" (UID: \"edbb90e8-546b-4edd-a223-5ad4d579adaa\") " pod="openshift-multus/multus-additional-cni-plugins-brmhg" Jul 06 13:50:27.011984 ip-10-0-136-25 kubenswrapper[2569]: I0706 13:50:27.011898 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/03d48054-9789-444b-82c7-ac9e3a504c9c-device-dir\") pod \"aws-ebs-csi-driver-node-2dnkc\" (UID: \"03d48054-9789-444b-82c7-ac9e3a504c9c\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-2dnkc" Jul 06 13:50:27.011984 ip-10-0-136-25 kubenswrapper[2569]: I0706 13:50:27.011934 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/576fd8e3-f3a0-48d2-a0d6-faa782a25a63-original-pull-secret\") pod \"global-pull-secret-syncer-cg8ph\" (UID: \"576fd8e3-f3a0-48d2-a0d6-faa782a25a63\") " pod="kube-system/global-pull-secret-syncer-cg8ph" Jul 06 13:50:27.011984 ip-10-0-136-25 kubenswrapper[2569]: I0706 13:50:27.011963 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/03d48054-9789-444b-82c7-ac9e3a504c9c-etc-selinux\") pod \"aws-ebs-csi-driver-node-2dnkc\" (UID: \"03d48054-9789-444b-82c7-ac9e3a504c9c\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-2dnkc" Jul 06 13:50:27.012293 ip-10-0-136-25 kubenswrapper[2569]: I0706 13:50:27.011986 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-62mdb\" (UniqueName: \"kubernetes.io/projected/03d48054-9789-444b-82c7-ac9e3a504c9c-kube-api-access-62mdb\") pod \"aws-ebs-csi-driver-node-2dnkc\" (UID: \"03d48054-9789-444b-82c7-ac9e3a504c9c\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-2dnkc" Jul 06 13:50:27.012293 ip-10-0-136-25 kubenswrapper[2569]: I0706 13:50:27.012010 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2c4b4cf7-31a7-4cde-8212-9d78cc184e6c-metrics-certs\") pod \"network-metrics-daemon-hqt9l\" (UID: \"2c4b4cf7-31a7-4cde-8212-9d78cc184e6c\") " pod="openshift-multus/network-metrics-daemon-hqt9l" Jul 06 13:50:27.012293 ip-10-0-136-25 kubenswrapper[2569]: I0706 13:50:27.012029 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/03d48054-9789-444b-82c7-ac9e3a504c9c-device-dir\") pod \"aws-ebs-csi-driver-node-2dnkc\" (UID: \"03d48054-9789-444b-82c7-ac9e3a504c9c\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-2dnkc" Jul 06 13:50:27.012293 ip-10-0-136-25 kubenswrapper[2569]: E0706 13:50:27.012056 2569 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 06 13:50:27.012293 ip-10-0-136-25 kubenswrapper[2569]: I0706 13:50:27.012090 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/03d48054-9789-444b-82c7-ac9e3a504c9c-kubelet-dir\") pod \"aws-ebs-csi-driver-node-2dnkc\" (UID: \"03d48054-9789-444b-82c7-ac9e3a504c9c\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-2dnkc" Jul 06 13:50:27.012293 ip-10-0-136-25 kubenswrapper[2569]: E0706 13:50:27.012130 2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/576fd8e3-f3a0-48d2-a0d6-faa782a25a63-original-pull-secret podName:576fd8e3-f3a0-48d2-a0d6-faa782a25a63 nodeName:}" failed. No retries permitted until 2026-07-06 13:50:27.512110335 +0000 UTC m=+3.102672106 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/576fd8e3-f3a0-48d2-a0d6-faa782a25a63-original-pull-secret") pod "global-pull-secret-syncer-cg8ph" (UID: "576fd8e3-f3a0-48d2-a0d6-faa782a25a63") : object "kube-system"/"original-pull-secret" not registered Jul 06 13:50:27.012293 ip-10-0-136-25 kubenswrapper[2569]: I0706 13:50:27.012037 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/03d48054-9789-444b-82c7-ac9e3a504c9c-kubelet-dir\") pod \"aws-ebs-csi-driver-node-2dnkc\" (UID: \"03d48054-9789-444b-82c7-ac9e3a504c9c\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-2dnkc" Jul 06 13:50:27.012293 ip-10-0-136-25 kubenswrapper[2569]: E0706 13:50:27.012170 2569 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 06 13:50:27.012293 ip-10-0-136-25 kubenswrapper[2569]: I0706 13:50:27.012188 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/576fd8e3-f3a0-48d2-a0d6-faa782a25a63-kubelet-config\") pod \"global-pull-secret-syncer-cg8ph\" (UID: \"576fd8e3-f3a0-48d2-a0d6-faa782a25a63\") " pod="kube-system/global-pull-secret-syncer-cg8ph" Jul 06 13:50:27.012293 ip-10-0-136-25 kubenswrapper[2569]: E0706 13:50:27.012234 2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2c4b4cf7-31a7-4cde-8212-9d78cc184e6c-metrics-certs podName:2c4b4cf7-31a7-4cde-8212-9d78cc184e6c nodeName:}" failed. No retries permitted until 2026-07-06 13:50:27.512201036 +0000 UTC m=+3.102762796 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/2c4b4cf7-31a7-4cde-8212-9d78cc184e6c-metrics-certs") pod "network-metrics-daemon-hqt9l" (UID: "2c4b4cf7-31a7-4cde-8212-9d78cc184e6c") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 06 13:50:27.012293 ip-10-0-136-25 kubenswrapper[2569]: I0706 13:50:27.012258 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/576fd8e3-f3a0-48d2-a0d6-faa782a25a63-kubelet-config\") pod \"global-pull-secret-syncer-cg8ph\" (UID: \"576fd8e3-f3a0-48d2-a0d6-faa782a25a63\") " pod="kube-system/global-pull-secret-syncer-cg8ph" Jul 06 13:50:27.012293 ip-10-0-136-25 kubenswrapper[2569]: I0706 13:50:27.012269 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/03d48054-9789-444b-82c7-ac9e3a504c9c-etc-selinux\") pod \"aws-ebs-csi-driver-node-2dnkc\" (UID: \"03d48054-9789-444b-82c7-ac9e3a504c9c\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-2dnkc" Jul 06 13:50:27.012293 ip-10-0-136-25 kubenswrapper[2569]: I0706 13:50:27.012261 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qvkxw\" (UniqueName: \"kubernetes.io/projected/2c4b4cf7-31a7-4cde-8212-9d78cc184e6c-kube-api-access-qvkxw\") pod \"network-metrics-daemon-hqt9l\" (UID: \"2c4b4cf7-31a7-4cde-8212-9d78cc184e6c\") " pod="openshift-multus/network-metrics-daemon-hqt9l" Jul 06 13:50:27.012888 ip-10-0-136-25 kubenswrapper[2569]: I0706 13:50:27.012331 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/03d48054-9789-444b-82c7-ac9e3a504c9c-socket-dir\") pod \"aws-ebs-csi-driver-node-2dnkc\" (UID: \"03d48054-9789-444b-82c7-ac9e3a504c9c\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-2dnkc" Jul 06 13:50:27.012888 ip-10-0-136-25 kubenswrapper[2569]: I0706 13:50:27.012364 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/03d48054-9789-444b-82c7-ac9e3a504c9c-sys-fs\") pod \"aws-ebs-csi-driver-node-2dnkc\" (UID: \"03d48054-9789-444b-82c7-ac9e3a504c9c\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-2dnkc" Jul 06 13:50:27.012888 ip-10-0-136-25 kubenswrapper[2569]: I0706 13:50:27.012393 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/03d48054-9789-444b-82c7-ac9e3a504c9c-registration-dir\") pod \"aws-ebs-csi-driver-node-2dnkc\" (UID: \"03d48054-9789-444b-82c7-ac9e3a504c9c\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-2dnkc" Jul 06 13:50:27.012888 ip-10-0-136-25 kubenswrapper[2569]: I0706 13:50:27.012415 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/576fd8e3-f3a0-48d2-a0d6-faa782a25a63-dbus\") pod \"global-pull-secret-syncer-cg8ph\" (UID: \"576fd8e3-f3a0-48d2-a0d6-faa782a25a63\") " pod="kube-system/global-pull-secret-syncer-cg8ph" Jul 06 13:50:27.012888 ip-10-0-136-25 kubenswrapper[2569]: I0706 13:50:27.012443 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/ef1daf8b-7b12-4c6a-9fbd-2b7e9187fd8f-hosts-file\") pod \"node-resolver-54w6r\" (UID: \"ef1daf8b-7b12-4c6a-9fbd-2b7e9187fd8f\") " pod="openshift-dns/node-resolver-54w6r" Jul 06 13:50:27.012888 ip-10-0-136-25 kubenswrapper[2569]: I0706 13:50:27.012465 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/03d48054-9789-444b-82c7-ac9e3a504c9c-sys-fs\") pod \"aws-ebs-csi-driver-node-2dnkc\" (UID: \"03d48054-9789-444b-82c7-ac9e3a504c9c\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-2dnkc" Jul 06 13:50:27.012888 ip-10-0-136-25 kubenswrapper[2569]: I0706 13:50:27.012477 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gf7rl\" (UniqueName: \"kubernetes.io/projected/ef1daf8b-7b12-4c6a-9fbd-2b7e9187fd8f-kube-api-access-gf7rl\") pod \"node-resolver-54w6r\" (UID: \"ef1daf8b-7b12-4c6a-9fbd-2b7e9187fd8f\") " pod="openshift-dns/node-resolver-54w6r" Jul 06 13:50:27.012888 ip-10-0-136-25 kubenswrapper[2569]: I0706 13:50:27.012486 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/03d48054-9789-444b-82c7-ac9e3a504c9c-socket-dir\") pod \"aws-ebs-csi-driver-node-2dnkc\" (UID: \"03d48054-9789-444b-82c7-ac9e3a504c9c\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-2dnkc" Jul 06 13:50:27.012888 ip-10-0-136-25 kubenswrapper[2569]: I0706 13:50:27.012488 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/03d48054-9789-444b-82c7-ac9e3a504c9c-registration-dir\") pod \"aws-ebs-csi-driver-node-2dnkc\" (UID: \"03d48054-9789-444b-82c7-ac9e3a504c9c\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-2dnkc" Jul 06 13:50:27.012888 ip-10-0-136-25 kubenswrapper[2569]: I0706 13:50:27.012509 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/ef1daf8b-7b12-4c6a-9fbd-2b7e9187fd8f-hosts-file\") pod \"node-resolver-54w6r\" (UID: \"ef1daf8b-7b12-4c6a-9fbd-2b7e9187fd8f\") " pod="openshift-dns/node-resolver-54w6r" Jul 06 13:50:27.012888 ip-10-0-136-25 kubenswrapper[2569]: I0706 13:50:27.012542 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ef1daf8b-7b12-4c6a-9fbd-2b7e9187fd8f-tmp-dir\") pod \"node-resolver-54w6r\" (UID: \"ef1daf8b-7b12-4c6a-9fbd-2b7e9187fd8f\") " pod="openshift-dns/node-resolver-54w6r" Jul 06 13:50:27.012888 ip-10-0-136-25 kubenswrapper[2569]: I0706 13:50:27.012586 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/576fd8e3-f3a0-48d2-a0d6-faa782a25a63-dbus\") pod \"global-pull-secret-syncer-cg8ph\" (UID: \"576fd8e3-f3a0-48d2-a0d6-faa782a25a63\") " pod="kube-system/global-pull-secret-syncer-cg8ph" Jul 06 13:50:27.012888 ip-10-0-136-25 kubenswrapper[2569]: I0706 13:50:27.012862 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ef1daf8b-7b12-4c6a-9fbd-2b7e9187fd8f-tmp-dir\") pod \"node-resolver-54w6r\" (UID: \"ef1daf8b-7b12-4c6a-9fbd-2b7e9187fd8f\") " pod="openshift-dns/node-resolver-54w6r" Jul 06 13:50:27.025091 ip-10-0-136-25 kubenswrapper[2569]: I0706 13:50:27.025067 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qvkxw\" (UniqueName: \"kubernetes.io/projected/2c4b4cf7-31a7-4cde-8212-9d78cc184e6c-kube-api-access-qvkxw\") pod \"network-metrics-daemon-hqt9l\" (UID: \"2c4b4cf7-31a7-4cde-8212-9d78cc184e6c\") " pod="openshift-multus/network-metrics-daemon-hqt9l" Jul 06 13:50:27.025628 ip-10-0-136-25 kubenswrapper[2569]: I0706 13:50:27.025605 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gf7rl\" (UniqueName: \"kubernetes.io/projected/ef1daf8b-7b12-4c6a-9fbd-2b7e9187fd8f-kube-api-access-gf7rl\") pod \"node-resolver-54w6r\" (UID: \"ef1daf8b-7b12-4c6a-9fbd-2b7e9187fd8f\") " pod="openshift-dns/node-resolver-54w6r" Jul 06 13:50:27.025718 ip-10-0-136-25 kubenswrapper[2569]: I0706 13:50:27.025663 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-62mdb\" (UniqueName: \"kubernetes.io/projected/03d48054-9789-444b-82c7-ac9e3a504c9c-kube-api-access-62mdb\") pod \"aws-ebs-csi-driver-node-2dnkc\" (UID: \"03d48054-9789-444b-82c7-ac9e3a504c9c\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-2dnkc" Jul 06 13:50:27.091805 ip-10-0-136-25 kubenswrapper[2569]: I0706 13:50:27.091771 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-vcxnc" Jul 06 13:50:27.099555 ip-10-0-136-25 kubenswrapper[2569]: I0706 13:50:27.099535 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-c9sqv" Jul 06 13:50:27.110143 ip-10-0-136-25 kubenswrapper[2569]: I0706 13:50:27.110121 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-vqmcq" Jul 06 13:50:27.115819 ip-10-0-136-25 kubenswrapper[2569]: I0706 13:50:27.115800 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-brmhg" Jul 06 13:50:27.123517 ip-10-0-136-25 kubenswrapper[2569]: I0706 13:50:27.123493 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-dcdgc" Jul 06 13:50:27.131194 ip-10-0-136-25 kubenswrapper[2569]: I0706 13:50:27.131175 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-2x29s" Jul 06 13:50:27.139672 ip-10-0-136-25 kubenswrapper[2569]: I0706 13:50:27.139653 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-wxxjp" Jul 06 13:50:27.147355 ip-10-0-136-25 kubenswrapper[2569]: I0706 13:50:27.147333 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-2dnkc" Jul 06 13:50:27.153105 ip-10-0-136-25 kubenswrapper[2569]: I0706 13:50:27.153086 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-54w6r" Jul 06 13:50:27.516131 ip-10-0-136-25 kubenswrapper[2569]: I0706 13:50:27.516108 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4kcgx\" (UniqueName: \"kubernetes.io/projected/d2d86b02-1e00-4267-b9c4-dd365b15342e-kube-api-access-4kcgx\") pod \"network-check-target-gbq5d\" (UID: \"d2d86b02-1e00-4267-b9c4-dd365b15342e\") " pod="openshift-network-diagnostics/network-check-target-gbq5d" Jul 06 13:50:27.516349 ip-10-0-136-25 kubenswrapper[2569]: I0706 13:50:27.516142 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/576fd8e3-f3a0-48d2-a0d6-faa782a25a63-original-pull-secret\") pod \"global-pull-secret-syncer-cg8ph\" (UID: \"576fd8e3-f3a0-48d2-a0d6-faa782a25a63\") " pod="kube-system/global-pull-secret-syncer-cg8ph" Jul 06 13:50:27.516349 ip-10-0-136-25 kubenswrapper[2569]: I0706 13:50:27.516168 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2c4b4cf7-31a7-4cde-8212-9d78cc184e6c-metrics-certs\") pod \"network-metrics-daemon-hqt9l\" (UID: \"2c4b4cf7-31a7-4cde-8212-9d78cc184e6c\") " pod="openshift-multus/network-metrics-daemon-hqt9l" Jul 06 13:50:27.516349 ip-10-0-136-25 kubenswrapper[2569]: E0706 13:50:27.516272 2569 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 06 13:50:27.516349 ip-10-0-136-25 kubenswrapper[2569]: E0706 13:50:27.516279 2569 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 06 13:50:27.516349 ip-10-0-136-25 kubenswrapper[2569]: E0706 13:50:27.516274 2569 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 06 13:50:27.516349 ip-10-0-136-25 kubenswrapper[2569]: E0706 13:50:27.516339 2569 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 06 13:50:27.516349 ip-10-0-136-25 kubenswrapper[2569]: E0706 13:50:27.516352 2569 projected.go:196] Error preparing data for projected volume kube-api-access-4kcgx for pod openshift-network-diagnostics/network-check-target-gbq5d: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 06 13:50:27.516604 ip-10-0-136-25 kubenswrapper[2569]: E0706 13:50:27.516322 2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2c4b4cf7-31a7-4cde-8212-9d78cc184e6c-metrics-certs podName:2c4b4cf7-31a7-4cde-8212-9d78cc184e6c nodeName:}" failed. No retries permitted until 2026-07-06 13:50:28.516305548 +0000 UTC m=+4.106867303 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/2c4b4cf7-31a7-4cde-8212-9d78cc184e6c-metrics-certs") pod "network-metrics-daemon-hqt9l" (UID: "2c4b4cf7-31a7-4cde-8212-9d78cc184e6c") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 06 13:50:27.516604 ip-10-0-136-25 kubenswrapper[2569]: E0706 13:50:27.516408 2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/576fd8e3-f3a0-48d2-a0d6-faa782a25a63-original-pull-secret podName:576fd8e3-f3a0-48d2-a0d6-faa782a25a63 nodeName:}" failed. No retries permitted until 2026-07-06 13:50:28.51638925 +0000 UTC m=+4.106951005 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/576fd8e3-f3a0-48d2-a0d6-faa782a25a63-original-pull-secret") pod "global-pull-secret-syncer-cg8ph" (UID: "576fd8e3-f3a0-48d2-a0d6-faa782a25a63") : object "kube-system"/"original-pull-secret" not registered Jul 06 13:50:27.516604 ip-10-0-136-25 kubenswrapper[2569]: E0706 13:50:27.516432 2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/d2d86b02-1e00-4267-b9c4-dd365b15342e-kube-api-access-4kcgx podName:d2d86b02-1e00-4267-b9c4-dd365b15342e nodeName:}" failed. No retries permitted until 2026-07-06 13:50:28.516421423 +0000 UTC m=+4.106983182 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-api-access-4kcgx" (UniqueName: "kubernetes.io/projected/d2d86b02-1e00-4267-b9c4-dd365b15342e-kube-api-access-4kcgx") pod "network-check-target-gbq5d" (UID: "d2d86b02-1e00-4267-b9c4-dd365b15342e") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 06 13:50:27.837585 ip-10-0-136-25 kubenswrapper[2569]: W0706 13:50:27.837561 2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcb1807ec_0568_4ab3_a9ca_e453709b820e.slice/crio-15ab953ea816302f5ef6b35714f6fbe5334f04ac196a1bfbc43f18565affd93f WatchSource:0}: Error finding container 15ab953ea816302f5ef6b35714f6fbe5334f04ac196a1bfbc43f18565affd93f: Status 404 returned error can't find the container with id 15ab953ea816302f5ef6b35714f6fbe5334f04ac196a1bfbc43f18565affd93f Jul 06 13:50:27.838448 ip-10-0-136-25 kubenswrapper[2569]: W0706 13:50:27.838426 2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb7576bc8_a480_44ac_b68c_36f9abe93b7d.slice/crio-f4b1796a650083675425f47aee85fe824203832991d2e15a5a4368db035b91a3 WatchSource:0}: Error finding container f4b1796a650083675425f47aee85fe824203832991d2e15a5a4368db035b91a3: Status 404 returned error can't find the container with id f4b1796a650083675425f47aee85fe824203832991d2e15a5a4368db035b91a3 Jul 06 13:50:27.839320 ip-10-0-136-25 kubenswrapper[2569]: W0706 13:50:27.839286 2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5bbb0d3f_2cd8_4496_b676_6045376a5b13.slice/crio-496f84e7d8e792eaf84693db590722ba3bb67a20e69f24b07c0d669a3ed01248 WatchSource:0}: Error finding container 496f84e7d8e792eaf84693db590722ba3bb67a20e69f24b07c0d669a3ed01248: Status 404 returned error can't find the container with id 496f84e7d8e792eaf84693db590722ba3bb67a20e69f24b07c0d669a3ed01248 Jul 06 13:50:27.840168 ip-10-0-136-25 kubenswrapper[2569]: W0706 13:50:27.840147 2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podedbb90e8_546b_4edd_a223_5ad4d579adaa.slice/crio-6e20f4ec82ca985b486e0674c58e2ea6869946c908096abceb4c04a286b1140a WatchSource:0}: Error finding container 6e20f4ec82ca985b486e0674c58e2ea6869946c908096abceb4c04a286b1140a: Status 404 returned error can't find the container with id 6e20f4ec82ca985b486e0674c58e2ea6869946c908096abceb4c04a286b1140a Jul 06 13:50:27.849710 ip-10-0-136-25 kubenswrapper[2569]: I0706 13:50:27.849661 2569 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-07-05 13:45:26 +0000 UTC" deadline="2028-03-10 01:36:30.97618924 +0000 UTC" Jul 06 13:50:27.849813 ip-10-0-136-25 kubenswrapper[2569]: I0706 13:50:27.849712 2569 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="14699h46m3.12648128s" Jul 06 13:50:27.851075 ip-10-0-136-25 kubenswrapper[2569]: W0706 13:50:27.851040 2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4ae58dad_494b_4508_b5a5_129f13818d53.slice/crio-639403d27c12ccf5171a11467287190837b50e7df45a71ce481160ff6cf38f50 WatchSource:0}: Error finding container 639403d27c12ccf5171a11467287190837b50e7df45a71ce481160ff6cf38f50: Status 404 returned error can't find the container with id 639403d27c12ccf5171a11467287190837b50e7df45a71ce481160ff6cf38f50 Jul 06 13:50:27.942709 ip-10-0-136-25 kubenswrapper[2569]: I0706 13:50:27.942678 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-136-25.ec2.internal" event={"ID":"ba668ff1926990fccf3f73e0625e305c","Type":"ContainerStarted","Data":"87e39e2c1106a69b48ba3328fa0a933e940d55229ed0e6c3248da6d839b5d1ef"} Jul 06 13:50:27.943878 ip-10-0-136-25 kubenswrapper[2569]: I0706 13:50:27.943846 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-vcxnc" event={"ID":"4ae58dad-494b-4508-b5a5-129f13818d53","Type":"ContainerStarted","Data":"639403d27c12ccf5171a11467287190837b50e7df45a71ce481160ff6cf38f50"} Jul 06 13:50:27.944983 ip-10-0-136-25 kubenswrapper[2569]: I0706 13:50:27.944959 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-54w6r" event={"ID":"ef1daf8b-7b12-4c6a-9fbd-2b7e9187fd8f","Type":"ContainerStarted","Data":"70b03527bc50aede9b559c4a8161b18ebfe5f5366eb04a75e028d002b7346041"} Jul 06 13:50:27.946096 ip-10-0-136-25 kubenswrapper[2569]: I0706 13:50:27.946074 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-c9sqv" event={"ID":"028cbff5-11b2-411a-90ed-c0bd3f563830","Type":"ContainerStarted","Data":"b7387efa61e31a6c835e7871629e1df1044158d4a9dbf5f4658e841173c6beb7"} Jul 06 13:50:27.947156 ip-10-0-136-25 kubenswrapper[2569]: I0706 13:50:27.947137 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-dcdgc" event={"ID":"07d7de38-90f7-4810-b5d6-3a710a3af5a9","Type":"ContainerStarted","Data":"46468f7dab8609f564f1a18bfb2555c5befaa8282d7813c2fb1fd54bf4b17ffe"} Jul 06 13:50:27.948348 ip-10-0-136-25 kubenswrapper[2569]: I0706 13:50:27.948277 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-wxxjp" event={"ID":"5bbb0d3f-2cd8-4496-b676-6045376a5b13","Type":"ContainerStarted","Data":"496f84e7d8e792eaf84693db590722ba3bb67a20e69f24b07c0d669a3ed01248"} Jul 06 13:50:27.951004 ip-10-0-136-25 kubenswrapper[2569]: I0706 13:50:27.950986 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-2dnkc" event={"ID":"03d48054-9789-444b-82c7-ac9e3a504c9c","Type":"ContainerStarted","Data":"af7e8e8e618c206c0a971fe161d6cff4128678d23e5bd93af7bd8ec050d11419"} Jul 06 13:50:27.952330 ip-10-0-136-25 kubenswrapper[2569]: I0706 13:50:27.952309 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-brmhg" event={"ID":"edbb90e8-546b-4edd-a223-5ad4d579adaa","Type":"ContainerStarted","Data":"6e20f4ec82ca985b486e0674c58e2ea6869946c908096abceb4c04a286b1140a"} Jul 06 13:50:27.953436 ip-10-0-136-25 kubenswrapper[2569]: I0706 13:50:27.953407 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-2x29s" event={"ID":"b7576bc8-a480-44ac-b68c-36f9abe93b7d","Type":"ContainerStarted","Data":"f4b1796a650083675425f47aee85fe824203832991d2e15a5a4368db035b91a3"} Jul 06 13:50:27.954383 ip-10-0-136-25 kubenswrapper[2569]: I0706 13:50:27.954366 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-vqmcq" event={"ID":"cb1807ec-0568-4ab3-a9ca-e453709b820e","Type":"ContainerStarted","Data":"15ab953ea816302f5ef6b35714f6fbe5334f04ac196a1bfbc43f18565affd93f"} Jul 06 13:50:27.956414 ip-10-0-136-25 kubenswrapper[2569]: I0706 13:50:27.956375 2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/kube-apiserver-proxy-ip-10-0-136-25.ec2.internal" podStartSLOduration=1.956362263 podStartE2EDuration="1.956362263s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-06 13:50:26 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 13:50:27.955841993 +0000 UTC m=+3.546403770" watchObservedRunningTime="2026-07-06 13:50:27.956362263 +0000 UTC m=+3.546924040" Jul 06 13:50:28.522493 ip-10-0-136-25 kubenswrapper[2569]: I0706 13:50:28.522405 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2c4b4cf7-31a7-4cde-8212-9d78cc184e6c-metrics-certs\") pod \"network-metrics-daemon-hqt9l\" (UID: \"2c4b4cf7-31a7-4cde-8212-9d78cc184e6c\") " pod="openshift-multus/network-metrics-daemon-hqt9l" Jul 06 13:50:28.522649 ip-10-0-136-25 kubenswrapper[2569]: I0706 13:50:28.522500 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4kcgx\" (UniqueName: \"kubernetes.io/projected/d2d86b02-1e00-4267-b9c4-dd365b15342e-kube-api-access-4kcgx\") pod \"network-check-target-gbq5d\" (UID: \"d2d86b02-1e00-4267-b9c4-dd365b15342e\") " pod="openshift-network-diagnostics/network-check-target-gbq5d" Jul 06 13:50:28.522649 ip-10-0-136-25 kubenswrapper[2569]: I0706 13:50:28.522540 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/576fd8e3-f3a0-48d2-a0d6-faa782a25a63-original-pull-secret\") pod \"global-pull-secret-syncer-cg8ph\" (UID: \"576fd8e3-f3a0-48d2-a0d6-faa782a25a63\") " pod="kube-system/global-pull-secret-syncer-cg8ph" Jul 06 13:50:28.522762 ip-10-0-136-25 kubenswrapper[2569]: E0706 13:50:28.522665 2569 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 06 13:50:28.522762 ip-10-0-136-25 kubenswrapper[2569]: E0706 13:50:28.522729 2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/576fd8e3-f3a0-48d2-a0d6-faa782a25a63-original-pull-secret podName:576fd8e3-f3a0-48d2-a0d6-faa782a25a63 nodeName:}" failed. No retries permitted until 2026-07-06 13:50:30.522709923 +0000 UTC m=+6.113271696 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/576fd8e3-f3a0-48d2-a0d6-faa782a25a63-original-pull-secret") pod "global-pull-secret-syncer-cg8ph" (UID: "576fd8e3-f3a0-48d2-a0d6-faa782a25a63") : object "kube-system"/"original-pull-secret" not registered Jul 06 13:50:28.523535 ip-10-0-136-25 kubenswrapper[2569]: E0706 13:50:28.523113 2569 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 06 13:50:28.523535 ip-10-0-136-25 kubenswrapper[2569]: E0706 13:50:28.523167 2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2c4b4cf7-31a7-4cde-8212-9d78cc184e6c-metrics-certs podName:2c4b4cf7-31a7-4cde-8212-9d78cc184e6c nodeName:}" failed. No retries permitted until 2026-07-06 13:50:30.523151429 +0000 UTC m=+6.113713187 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/2c4b4cf7-31a7-4cde-8212-9d78cc184e6c-metrics-certs") pod "network-metrics-daemon-hqt9l" (UID: "2c4b4cf7-31a7-4cde-8212-9d78cc184e6c") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 06 13:50:28.523535 ip-10-0-136-25 kubenswrapper[2569]: E0706 13:50:28.523265 2569 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 06 13:50:28.523535 ip-10-0-136-25 kubenswrapper[2569]: E0706 13:50:28.523280 2569 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 06 13:50:28.523535 ip-10-0-136-25 kubenswrapper[2569]: E0706 13:50:28.523292 2569 projected.go:196] Error preparing data for projected volume kube-api-access-4kcgx for pod openshift-network-diagnostics/network-check-target-gbq5d: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 06 13:50:28.523535 ip-10-0-136-25 kubenswrapper[2569]: E0706 13:50:28.523331 2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/d2d86b02-1e00-4267-b9c4-dd365b15342e-kube-api-access-4kcgx podName:d2d86b02-1e00-4267-b9c4-dd365b15342e nodeName:}" failed. No retries permitted until 2026-07-06 13:50:30.523318007 +0000 UTC m=+6.113879767 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-api-access-4kcgx" (UniqueName: "kubernetes.io/projected/d2d86b02-1e00-4267-b9c4-dd365b15342e-kube-api-access-4kcgx") pod "network-check-target-gbq5d" (UID: "d2d86b02-1e00-4267-b9c4-dd365b15342e") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 06 13:50:28.646110 ip-10-0-136-25 kubenswrapper[2569]: I0706 13:50:28.646080 2569 reflector.go:439] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:160" Jul 06 13:50:28.850680 ip-10-0-136-25 kubenswrapper[2569]: I0706 13:50:28.850594 2569 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-07-05 13:45:26 +0000 UTC" deadline="2027-12-11 08:42:27.795841762 +0000 UTC" Jul 06 13:50:28.850680 ip-10-0-136-25 kubenswrapper[2569]: I0706 13:50:28.850628 2569 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="12546h51m58.945218133s" Jul 06 13:50:28.934699 ip-10-0-136-25 kubenswrapper[2569]: I0706 13:50:28.934667 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-cg8ph" Jul 06 13:50:28.934885 ip-10-0-136-25 kubenswrapper[2569]: E0706 13:50:28.934805 2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-cg8ph" podUID="576fd8e3-f3a0-48d2-a0d6-faa782a25a63" Jul 06 13:50:28.935238 ip-10-0-136-25 kubenswrapper[2569]: I0706 13:50:28.935192 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-gbq5d" Jul 06 13:50:28.935348 ip-10-0-136-25 kubenswrapper[2569]: E0706 13:50:28.935326 2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-gbq5d" podUID="d2d86b02-1e00-4267-b9c4-dd365b15342e" Jul 06 13:50:28.937655 ip-10-0-136-25 kubenswrapper[2569]: I0706 13:50:28.937632 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-hqt9l" Jul 06 13:50:28.937765 ip-10-0-136-25 kubenswrapper[2569]: E0706 13:50:28.937741 2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-hqt9l" podUID="2c4b4cf7-31a7-4cde-8212-9d78cc184e6c" Jul 06 13:50:28.969826 ip-10-0-136-25 kubenswrapper[2569]: I0706 13:50:28.969790 2569 generic.go:356] "Generic (PLEG): container finished" podID="2a2f7f2a1d78cf5fb862a37fcc75004a" containerID="fd207332c5b2271a3135fb1b78488b9d952ddc7094fb3d6d1f6d021d59969ec4" exitCode=0 Jul 06 13:50:28.969989 ip-10-0-136-25 kubenswrapper[2569]: I0706 13:50:28.969951 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-136-25.ec2.internal" event={"ID":"2a2f7f2a1d78cf5fb862a37fcc75004a","Type":"ContainerDied","Data":"fd207332c5b2271a3135fb1b78488b9d952ddc7094fb3d6d1f6d021d59969ec4"} Jul 06 13:50:29.992284 ip-10-0-136-25 kubenswrapper[2569]: I0706 13:50:29.991449 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-136-25.ec2.internal" event={"ID":"2a2f7f2a1d78cf5fb862a37fcc75004a","Type":"ContainerStarted","Data":"a1196d47ee63233014ce39c3a4423e356b6161bf8dce7fae16e71ce439467494"} Jul 06 13:50:30.540390 ip-10-0-136-25 kubenswrapper[2569]: I0706 13:50:30.540303 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/576fd8e3-f3a0-48d2-a0d6-faa782a25a63-original-pull-secret\") pod \"global-pull-secret-syncer-cg8ph\" (UID: \"576fd8e3-f3a0-48d2-a0d6-faa782a25a63\") " pod="kube-system/global-pull-secret-syncer-cg8ph" Jul 06 13:50:30.540390 ip-10-0-136-25 kubenswrapper[2569]: I0706 13:50:30.540356 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2c4b4cf7-31a7-4cde-8212-9d78cc184e6c-metrics-certs\") pod \"network-metrics-daemon-hqt9l\" (UID: \"2c4b4cf7-31a7-4cde-8212-9d78cc184e6c\") " pod="openshift-multus/network-metrics-daemon-hqt9l" Jul 06 13:50:30.540642 ip-10-0-136-25 kubenswrapper[2569]: I0706 13:50:30.540415 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4kcgx\" (UniqueName: \"kubernetes.io/projected/d2d86b02-1e00-4267-b9c4-dd365b15342e-kube-api-access-4kcgx\") pod \"network-check-target-gbq5d\" (UID: \"d2d86b02-1e00-4267-b9c4-dd365b15342e\") " pod="openshift-network-diagnostics/network-check-target-gbq5d" Jul 06 13:50:30.540642 ip-10-0-136-25 kubenswrapper[2569]: E0706 13:50:30.540553 2569 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 06 13:50:30.540642 ip-10-0-136-25 kubenswrapper[2569]: E0706 13:50:30.540573 2569 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 06 13:50:30.540642 ip-10-0-136-25 kubenswrapper[2569]: E0706 13:50:30.540586 2569 projected.go:196] Error preparing data for projected volume kube-api-access-4kcgx for pod openshift-network-diagnostics/network-check-target-gbq5d: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 06 13:50:30.540642 ip-10-0-136-25 kubenswrapper[2569]: E0706 13:50:30.540641 2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/d2d86b02-1e00-4267-b9c4-dd365b15342e-kube-api-access-4kcgx podName:d2d86b02-1e00-4267-b9c4-dd365b15342e nodeName:}" failed. No retries permitted until 2026-07-06 13:50:34.540622185 +0000 UTC m=+10.131183944 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-api-access-4kcgx" (UniqueName: "kubernetes.io/projected/d2d86b02-1e00-4267-b9c4-dd365b15342e-kube-api-access-4kcgx") pod "network-check-target-gbq5d" (UID: "d2d86b02-1e00-4267-b9c4-dd365b15342e") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 06 13:50:30.541081 ip-10-0-136-25 kubenswrapper[2569]: E0706 13:50:30.541061 2569 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 06 13:50:30.541139 ip-10-0-136-25 kubenswrapper[2569]: E0706 13:50:30.541115 2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/576fd8e3-f3a0-48d2-a0d6-faa782a25a63-original-pull-secret podName:576fd8e3-f3a0-48d2-a0d6-faa782a25a63 nodeName:}" failed. No retries permitted until 2026-07-06 13:50:34.541100359 +0000 UTC m=+10.131662118 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/576fd8e3-f3a0-48d2-a0d6-faa782a25a63-original-pull-secret") pod "global-pull-secret-syncer-cg8ph" (UID: "576fd8e3-f3a0-48d2-a0d6-faa782a25a63") : object "kube-system"/"original-pull-secret" not registered Jul 06 13:50:30.541251 ip-10-0-136-25 kubenswrapper[2569]: E0706 13:50:30.541171 2569 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 06 13:50:30.541251 ip-10-0-136-25 kubenswrapper[2569]: E0706 13:50:30.541202 2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2c4b4cf7-31a7-4cde-8212-9d78cc184e6c-metrics-certs podName:2c4b4cf7-31a7-4cde-8212-9d78cc184e6c nodeName:}" failed. No retries permitted until 2026-07-06 13:50:34.541191214 +0000 UTC m=+10.131752978 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/2c4b4cf7-31a7-4cde-8212-9d78cc184e6c-metrics-certs") pod "network-metrics-daemon-hqt9l" (UID: "2c4b4cf7-31a7-4cde-8212-9d78cc184e6c") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 06 13:50:30.934912 ip-10-0-136-25 kubenswrapper[2569]: I0706 13:50:30.934383 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-hqt9l" Jul 06 13:50:30.934912 ip-10-0-136-25 kubenswrapper[2569]: E0706 13:50:30.934526 2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-hqt9l" podUID="2c4b4cf7-31a7-4cde-8212-9d78cc184e6c" Jul 06 13:50:30.934912 ip-10-0-136-25 kubenswrapper[2569]: I0706 13:50:30.934383 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-gbq5d" Jul 06 13:50:30.934912 ip-10-0-136-25 kubenswrapper[2569]: E0706 13:50:30.934649 2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-gbq5d" podUID="d2d86b02-1e00-4267-b9c4-dd365b15342e" Jul 06 13:50:30.934912 ip-10-0-136-25 kubenswrapper[2569]: I0706 13:50:30.934712 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-cg8ph" Jul 06 13:50:30.934912 ip-10-0-136-25 kubenswrapper[2569]: E0706 13:50:30.934786 2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-cg8ph" podUID="576fd8e3-f3a0-48d2-a0d6-faa782a25a63" Jul 06 13:50:32.934071 ip-10-0-136-25 kubenswrapper[2569]: I0706 13:50:32.934026 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-cg8ph" Jul 06 13:50:32.934661 ip-10-0-136-25 kubenswrapper[2569]: I0706 13:50:32.934031 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-gbq5d" Jul 06 13:50:32.934661 ip-10-0-136-25 kubenswrapper[2569]: I0706 13:50:32.934164 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-hqt9l" Jul 06 13:50:32.934661 ip-10-0-136-25 kubenswrapper[2569]: E0706 13:50:32.934159 2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-cg8ph" podUID="576fd8e3-f3a0-48d2-a0d6-faa782a25a63" Jul 06 13:50:32.934661 ip-10-0-136-25 kubenswrapper[2569]: E0706 13:50:32.934275 2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-gbq5d" podUID="d2d86b02-1e00-4267-b9c4-dd365b15342e" Jul 06 13:50:32.934661 ip-10-0-136-25 kubenswrapper[2569]: E0706 13:50:32.934373 2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-hqt9l" podUID="2c4b4cf7-31a7-4cde-8212-9d78cc184e6c" Jul 06 13:50:34.573687 ip-10-0-136-25 kubenswrapper[2569]: I0706 13:50:34.573643 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4kcgx\" (UniqueName: \"kubernetes.io/projected/d2d86b02-1e00-4267-b9c4-dd365b15342e-kube-api-access-4kcgx\") pod \"network-check-target-gbq5d\" (UID: \"d2d86b02-1e00-4267-b9c4-dd365b15342e\") " pod="openshift-network-diagnostics/network-check-target-gbq5d" Jul 06 13:50:34.574376 ip-10-0-136-25 kubenswrapper[2569]: I0706 13:50:34.573712 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/576fd8e3-f3a0-48d2-a0d6-faa782a25a63-original-pull-secret\") pod \"global-pull-secret-syncer-cg8ph\" (UID: \"576fd8e3-f3a0-48d2-a0d6-faa782a25a63\") " pod="kube-system/global-pull-secret-syncer-cg8ph" Jul 06 13:50:34.574376 ip-10-0-136-25 kubenswrapper[2569]: I0706 13:50:34.573748 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2c4b4cf7-31a7-4cde-8212-9d78cc184e6c-metrics-certs\") pod \"network-metrics-daemon-hqt9l\" (UID: \"2c4b4cf7-31a7-4cde-8212-9d78cc184e6c\") " pod="openshift-multus/network-metrics-daemon-hqt9l" Jul 06 13:50:34.574376 ip-10-0-136-25 kubenswrapper[2569]: E0706 13:50:34.573833 2569 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 06 13:50:34.574376 ip-10-0-136-25 kubenswrapper[2569]: E0706 13:50:34.573860 2569 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 06 13:50:34.574376 ip-10-0-136-25 kubenswrapper[2569]: E0706 13:50:34.573876 2569 projected.go:196] Error preparing data for projected volume kube-api-access-4kcgx for pod openshift-network-diagnostics/network-check-target-gbq5d: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 06 13:50:34.574376 ip-10-0-136-25 kubenswrapper[2569]: E0706 13:50:34.573873 2569 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 06 13:50:34.574376 ip-10-0-136-25 kubenswrapper[2569]: E0706 13:50:34.573882 2569 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 06 13:50:34.574376 ip-10-0-136-25 kubenswrapper[2569]: E0706 13:50:34.573929 2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/d2d86b02-1e00-4267-b9c4-dd365b15342e-kube-api-access-4kcgx podName:d2d86b02-1e00-4267-b9c4-dd365b15342e nodeName:}" failed. No retries permitted until 2026-07-06 13:50:42.573910466 +0000 UTC m=+18.164472225 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-api-access-4kcgx" (UniqueName: "kubernetes.io/projected/d2d86b02-1e00-4267-b9c4-dd365b15342e-kube-api-access-4kcgx") pod "network-check-target-gbq5d" (UID: "d2d86b02-1e00-4267-b9c4-dd365b15342e") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 06 13:50:34.574376 ip-10-0-136-25 kubenswrapper[2569]: E0706 13:50:34.573948 2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/576fd8e3-f3a0-48d2-a0d6-faa782a25a63-original-pull-secret podName:576fd8e3-f3a0-48d2-a0d6-faa782a25a63 nodeName:}" failed. No retries permitted until 2026-07-06 13:50:42.573939186 +0000 UTC m=+18.164500944 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/576fd8e3-f3a0-48d2-a0d6-faa782a25a63-original-pull-secret") pod "global-pull-secret-syncer-cg8ph" (UID: "576fd8e3-f3a0-48d2-a0d6-faa782a25a63") : object "kube-system"/"original-pull-secret" not registered Jul 06 13:50:34.574376 ip-10-0-136-25 kubenswrapper[2569]: E0706 13:50:34.573964 2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2c4b4cf7-31a7-4cde-8212-9d78cc184e6c-metrics-certs podName:2c4b4cf7-31a7-4cde-8212-9d78cc184e6c nodeName:}" failed. No retries permitted until 2026-07-06 13:50:42.573955017 +0000 UTC m=+18.164516776 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/2c4b4cf7-31a7-4cde-8212-9d78cc184e6c-metrics-certs") pod "network-metrics-daemon-hqt9l" (UID: "2c4b4cf7-31a7-4cde-8212-9d78cc184e6c") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 06 13:50:34.935526 ip-10-0-136-25 kubenswrapper[2569]: I0706 13:50:34.934929 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-cg8ph" Jul 06 13:50:34.935526 ip-10-0-136-25 kubenswrapper[2569]: E0706 13:50:34.935045 2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-cg8ph" podUID="576fd8e3-f3a0-48d2-a0d6-faa782a25a63" Jul 06 13:50:34.935526 ip-10-0-136-25 kubenswrapper[2569]: I0706 13:50:34.935433 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-hqt9l" Jul 06 13:50:34.935799 ip-10-0-136-25 kubenswrapper[2569]: E0706 13:50:34.935531 2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-hqt9l" podUID="2c4b4cf7-31a7-4cde-8212-9d78cc184e6c" Jul 06 13:50:34.935799 ip-10-0-136-25 kubenswrapper[2569]: I0706 13:50:34.935580 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-gbq5d" Jul 06 13:50:34.935799 ip-10-0-136-25 kubenswrapper[2569]: E0706 13:50:34.935638 2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-gbq5d" podUID="d2d86b02-1e00-4267-b9c4-dd365b15342e" Jul 06 13:50:36.934402 ip-10-0-136-25 kubenswrapper[2569]: I0706 13:50:36.934369 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-cg8ph" Jul 06 13:50:36.934907 ip-10-0-136-25 kubenswrapper[2569]: E0706 13:50:36.934507 2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-cg8ph" podUID="576fd8e3-f3a0-48d2-a0d6-faa782a25a63" Jul 06 13:50:36.934907 ip-10-0-136-25 kubenswrapper[2569]: I0706 13:50:36.934895 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-gbq5d" Jul 06 13:50:36.935034 ip-10-0-136-25 kubenswrapper[2569]: E0706 13:50:36.934999 2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-gbq5d" podUID="d2d86b02-1e00-4267-b9c4-dd365b15342e" Jul 06 13:50:36.935096 ip-10-0-136-25 kubenswrapper[2569]: I0706 13:50:36.935074 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-hqt9l" Jul 06 13:50:36.935242 ip-10-0-136-25 kubenswrapper[2569]: E0706 13:50:36.935188 2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-hqt9l" podUID="2c4b4cf7-31a7-4cde-8212-9d78cc184e6c" Jul 06 13:50:38.934816 ip-10-0-136-25 kubenswrapper[2569]: I0706 13:50:38.934776 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-gbq5d" Jul 06 13:50:38.935296 ip-10-0-136-25 kubenswrapper[2569]: I0706 13:50:38.934832 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-hqt9l" Jul 06 13:50:38.935296 ip-10-0-136-25 kubenswrapper[2569]: I0706 13:50:38.934776 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-cg8ph" Jul 06 13:50:38.935296 ip-10-0-136-25 kubenswrapper[2569]: E0706 13:50:38.934909 2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-gbq5d" podUID="d2d86b02-1e00-4267-b9c4-dd365b15342e" Jul 06 13:50:38.935296 ip-10-0-136-25 kubenswrapper[2569]: E0706 13:50:38.934991 2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-cg8ph" podUID="576fd8e3-f3a0-48d2-a0d6-faa782a25a63" Jul 06 13:50:38.935296 ip-10-0-136-25 kubenswrapper[2569]: E0706 13:50:38.935082 2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-hqt9l" podUID="2c4b4cf7-31a7-4cde-8212-9d78cc184e6c" Jul 06 13:50:40.934308 ip-10-0-136-25 kubenswrapper[2569]: I0706 13:50:40.934280 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-hqt9l" Jul 06 13:50:40.934762 ip-10-0-136-25 kubenswrapper[2569]: I0706 13:50:40.934280 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-gbq5d" Jul 06 13:50:40.934762 ip-10-0-136-25 kubenswrapper[2569]: E0706 13:50:40.934392 2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-hqt9l" podUID="2c4b4cf7-31a7-4cde-8212-9d78cc184e6c" Jul 06 13:50:40.934762 ip-10-0-136-25 kubenswrapper[2569]: I0706 13:50:40.934421 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-cg8ph" Jul 06 13:50:40.934762 ip-10-0-136-25 kubenswrapper[2569]: E0706 13:50:40.934508 2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-gbq5d" podUID="d2d86b02-1e00-4267-b9c4-dd365b15342e" Jul 06 13:50:40.934762 ip-10-0-136-25 kubenswrapper[2569]: E0706 13:50:40.934576 2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-cg8ph" podUID="576fd8e3-f3a0-48d2-a0d6-faa782a25a63" Jul 06 13:50:42.634541 ip-10-0-136-25 kubenswrapper[2569]: I0706 13:50:42.634500 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4kcgx\" (UniqueName: \"kubernetes.io/projected/d2d86b02-1e00-4267-b9c4-dd365b15342e-kube-api-access-4kcgx\") pod \"network-check-target-gbq5d\" (UID: \"d2d86b02-1e00-4267-b9c4-dd365b15342e\") " pod="openshift-network-diagnostics/network-check-target-gbq5d" Jul 06 13:50:42.634541 ip-10-0-136-25 kubenswrapper[2569]: I0706 13:50:42.634545 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/576fd8e3-f3a0-48d2-a0d6-faa782a25a63-original-pull-secret\") pod \"global-pull-secret-syncer-cg8ph\" (UID: \"576fd8e3-f3a0-48d2-a0d6-faa782a25a63\") " pod="kube-system/global-pull-secret-syncer-cg8ph" Jul 06 13:50:42.635061 ip-10-0-136-25 kubenswrapper[2569]: I0706 13:50:42.634567 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2c4b4cf7-31a7-4cde-8212-9d78cc184e6c-metrics-certs\") pod \"network-metrics-daemon-hqt9l\" (UID: \"2c4b4cf7-31a7-4cde-8212-9d78cc184e6c\") " pod="openshift-multus/network-metrics-daemon-hqt9l" Jul 06 13:50:42.635061 ip-10-0-136-25 kubenswrapper[2569]: E0706 13:50:42.634673 2569 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 06 13:50:42.635061 ip-10-0-136-25 kubenswrapper[2569]: E0706 13:50:42.634703 2569 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 06 13:50:42.635061 ip-10-0-136-25 kubenswrapper[2569]: E0706 13:50:42.634709 2569 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 06 13:50:42.635061 ip-10-0-136-25 kubenswrapper[2569]: E0706 13:50:42.634790 2569 projected.go:196] Error preparing data for projected volume kube-api-access-4kcgx for pod openshift-network-diagnostics/network-check-target-gbq5d: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 06 13:50:42.635061 ip-10-0-136-25 kubenswrapper[2569]: E0706 13:50:42.634678 2569 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 06 13:50:42.635061 ip-10-0-136-25 kubenswrapper[2569]: E0706 13:50:42.634774 2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/576fd8e3-f3a0-48d2-a0d6-faa782a25a63-original-pull-secret podName:576fd8e3-f3a0-48d2-a0d6-faa782a25a63 nodeName:}" failed. No retries permitted until 2026-07-06 13:50:58.634753466 +0000 UTC m=+34.225315244 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/576fd8e3-f3a0-48d2-a0d6-faa782a25a63-original-pull-secret") pod "global-pull-secret-syncer-cg8ph" (UID: "576fd8e3-f3a0-48d2-a0d6-faa782a25a63") : object "kube-system"/"original-pull-secret" not registered Jul 06 13:50:42.635061 ip-10-0-136-25 kubenswrapper[2569]: E0706 13:50:42.634857 2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/d2d86b02-1e00-4267-b9c4-dd365b15342e-kube-api-access-4kcgx podName:d2d86b02-1e00-4267-b9c4-dd365b15342e nodeName:}" failed. No retries permitted until 2026-07-06 13:50:58.63483962 +0000 UTC m=+34.225401389 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "kube-api-access-4kcgx" (UniqueName: "kubernetes.io/projected/d2d86b02-1e00-4267-b9c4-dd365b15342e-kube-api-access-4kcgx") pod "network-check-target-gbq5d" (UID: "d2d86b02-1e00-4267-b9c4-dd365b15342e") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 06 13:50:42.635061 ip-10-0-136-25 kubenswrapper[2569]: E0706 13:50:42.634872 2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2c4b4cf7-31a7-4cde-8212-9d78cc184e6c-metrics-certs podName:2c4b4cf7-31a7-4cde-8212-9d78cc184e6c nodeName:}" failed. No retries permitted until 2026-07-06 13:50:58.634864983 +0000 UTC m=+34.225426738 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/2c4b4cf7-31a7-4cde-8212-9d78cc184e6c-metrics-certs") pod "network-metrics-daemon-hqt9l" (UID: "2c4b4cf7-31a7-4cde-8212-9d78cc184e6c") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 06 13:50:42.934784 ip-10-0-136-25 kubenswrapper[2569]: I0706 13:50:42.934691 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-cg8ph" Jul 06 13:50:42.934784 ip-10-0-136-25 kubenswrapper[2569]: I0706 13:50:42.934721 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-gbq5d" Jul 06 13:50:42.935006 ip-10-0-136-25 kubenswrapper[2569]: I0706 13:50:42.934689 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-hqt9l" Jul 06 13:50:42.935006 ip-10-0-136-25 kubenswrapper[2569]: E0706 13:50:42.934821 2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-cg8ph" podUID="576fd8e3-f3a0-48d2-a0d6-faa782a25a63" Jul 06 13:50:42.935006 ip-10-0-136-25 kubenswrapper[2569]: E0706 13:50:42.934875 2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-gbq5d" podUID="d2d86b02-1e00-4267-b9c4-dd365b15342e" Jul 06 13:50:42.935006 ip-10-0-136-25 kubenswrapper[2569]: E0706 13:50:42.934997 2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-hqt9l" podUID="2c4b4cf7-31a7-4cde-8212-9d78cc184e6c" Jul 06 13:50:44.937109 ip-10-0-136-25 kubenswrapper[2569]: I0706 13:50:44.937080 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-cg8ph" Jul 06 13:50:44.937109 ip-10-0-136-25 kubenswrapper[2569]: I0706 13:50:44.937117 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-hqt9l" Jul 06 13:50:44.937675 ip-10-0-136-25 kubenswrapper[2569]: I0706 13:50:44.937078 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-gbq5d" Jul 06 13:50:44.937675 ip-10-0-136-25 kubenswrapper[2569]: E0706 13:50:44.937199 2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-cg8ph" podUID="576fd8e3-f3a0-48d2-a0d6-faa782a25a63" Jul 06 13:50:44.937675 ip-10-0-136-25 kubenswrapper[2569]: E0706 13:50:44.937306 2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-hqt9l" podUID="2c4b4cf7-31a7-4cde-8212-9d78cc184e6c" Jul 06 13:50:44.937675 ip-10-0-136-25 kubenswrapper[2569]: E0706 13:50:44.937400 2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-gbq5d" podUID="d2d86b02-1e00-4267-b9c4-dd365b15342e" Jul 06 13:50:46.934318 ip-10-0-136-25 kubenswrapper[2569]: I0706 13:50:46.933872 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-cg8ph" Jul 06 13:50:46.934966 ip-10-0-136-25 kubenswrapper[2569]: I0706 13:50:46.933872 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-gbq5d" Jul 06 13:50:46.934966 ip-10-0-136-25 kubenswrapper[2569]: E0706 13:50:46.934412 2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-cg8ph" podUID="576fd8e3-f3a0-48d2-a0d6-faa782a25a63" Jul 06 13:50:46.934966 ip-10-0-136-25 kubenswrapper[2569]: I0706 13:50:46.933872 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-hqt9l" Jul 06 13:50:46.934966 ip-10-0-136-25 kubenswrapper[2569]: E0706 13:50:46.934482 2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-gbq5d" podUID="d2d86b02-1e00-4267-b9c4-dd365b15342e" Jul 06 13:50:46.934966 ip-10-0-136-25 kubenswrapper[2569]: E0706 13:50:46.934609 2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-hqt9l" podUID="2c4b4cf7-31a7-4cde-8212-9d78cc184e6c" Jul 06 13:50:47.030877 ip-10-0-136-25 kubenswrapper[2569]: I0706 13:50:47.030846 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-2x29s" event={"ID":"b7576bc8-a480-44ac-b68c-36f9abe93b7d","Type":"ContainerStarted","Data":"ac2bb6f5cacccbed4fc6ee0b10183a8c02e59425fdd642917c2ecb6319a3ff5d"} Jul 06 13:50:47.032178 ip-10-0-136-25 kubenswrapper[2569]: I0706 13:50:47.032155 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-vqmcq" event={"ID":"cb1807ec-0568-4ab3-a9ca-e453709b820e","Type":"ContainerStarted","Data":"eafd4699aaba42af26fb5fdd41319a7c70a53ecacfce9f0ae7732e3e95fec44f"} Jul 06 13:50:47.034162 ip-10-0-136-25 kubenswrapper[2569]: I0706 13:50:47.034145 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-vcxnc" event={"ID":"4ae58dad-494b-4508-b5a5-129f13818d53","Type":"ContainerStarted","Data":"6ade16e76ac58a2da2eeb9a37cd17010ff81bfab38caf2f27e17715ddd1d15fa"} Jul 06 13:50:47.034264 ip-10-0-136-25 kubenswrapper[2569]: I0706 13:50:47.034227 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-vcxnc" event={"ID":"4ae58dad-494b-4508-b5a5-129f13818d53","Type":"ContainerStarted","Data":"f70663c9618abf359cee1fed4401c5095bb4880ecb3472619cb52d1f37dae553"} Jul 06 13:50:47.034327 ip-10-0-136-25 kubenswrapper[2569]: I0706 13:50:47.034298 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-vcxnc" event={"ID":"4ae58dad-494b-4508-b5a5-129f13818d53","Type":"ContainerStarted","Data":"f1d794084ca833e2cff74be22732324adb397a642e27213e91670f1600cf9d19"} Jul 06 13:50:47.034390 ip-10-0-136-25 kubenswrapper[2569]: I0706 13:50:47.034327 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-vcxnc" event={"ID":"4ae58dad-494b-4508-b5a5-129f13818d53","Type":"ContainerStarted","Data":"7e4f47ace63b0cc18a9044cd9d483158a0d6911cb4b926aac89c52e7b4f707b9"} Jul 06 13:50:47.035481 ip-10-0-136-25 kubenswrapper[2569]: I0706 13:50:47.035450 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-54w6r" event={"ID":"ef1daf8b-7b12-4c6a-9fbd-2b7e9187fd8f","Type":"ContainerStarted","Data":"53dd34963b8a575ee6fca59ccf991842421811e09d62ac0e65b1ad32aa4980c9"} Jul 06 13:50:47.036728 ip-10-0-136-25 kubenswrapper[2569]: I0706 13:50:47.036703 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-c9sqv" event={"ID":"028cbff5-11b2-411a-90ed-c0bd3f563830","Type":"ContainerStarted","Data":"e1801a9d0ca8f6f63f346bd846cc500d165604c2db29c651875b409938cb8e27"} Jul 06 13:50:47.037960 ip-10-0-136-25 kubenswrapper[2569]: I0706 13:50:47.037929 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-dcdgc" event={"ID":"07d7de38-90f7-4810-b5d6-3a710a3af5a9","Type":"ContainerStarted","Data":"1d1b371ed906449cd67eb95f811caf876e64e88c1c96d23d96911bef846f02c6"} Jul 06 13:50:47.039251 ip-10-0-136-25 kubenswrapper[2569]: I0706 13:50:47.039233 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-2dnkc" event={"ID":"03d48054-9789-444b-82c7-ac9e3a504c9c","Type":"ContainerStarted","Data":"02ce7dada80616bfa93a2adb516abe93ad0a85744fce6ed656adde6bca9b97ca"} Jul 06 13:50:47.040486 ip-10-0-136-25 kubenswrapper[2569]: I0706 13:50:47.040469 2569 generic.go:356] "Generic (PLEG): container finished" podID="edbb90e8-546b-4edd-a223-5ad4d579adaa" containerID="7d182288c72121ae22ee0437ce1aca03cb424048b090b2fef86e54b9987bdd41" exitCode=0 Jul 06 13:50:47.040560 ip-10-0-136-25 kubenswrapper[2569]: I0706 13:50:47.040493 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-brmhg" event={"ID":"edbb90e8-546b-4edd-a223-5ad4d579adaa","Type":"ContainerDied","Data":"7d182288c72121ae22ee0437ce1aca03cb424048b090b2fef86e54b9987bdd41"} Jul 06 13:50:47.057973 ip-10-0-136-25 kubenswrapper[2569]: I0706 13:50:47.057928 2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-136-25.ec2.internal" podStartSLOduration=21.057917888 podStartE2EDuration="21.057917888s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-06 13:50:26 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 13:50:30.008346752 +0000 UTC m=+5.598908530" watchObservedRunningTime="2026-07-06 13:50:47.057917888 +0000 UTC m=+22.648479664" Jul 06 13:50:47.058266 ip-10-0-136-25 kubenswrapper[2569]: I0706 13:50:47.058240 2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-node-tuning-operator/tuned-2x29s" podStartSLOduration=3.458780656 podStartE2EDuration="22.058233275s" totalImagesPullingTime="18.599452619s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-06 13:50:25 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 13:50:47.057897073 +0000 UTC m=+22.648458850" watchObservedRunningTime="2026-07-06 13:50:47.058233275 +0000 UTC m=+22.648795048" Jul 06 13:50:47.109785 ip-10-0-136-25 kubenswrapper[2569]: I0706 13:50:47.109732 2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/konnectivity-agent-vqmcq" podStartSLOduration=12.563292723 podStartE2EDuration="23.109717289s" totalImagesPullingTime="10.546424566s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-06 13:50:24 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 13:50:47.080235903 +0000 UTC m=+22.670797686" watchObservedRunningTime="2026-07-06 13:50:47.109717289 +0000 UTC m=+22.700279067" Jul 06 13:50:47.130149 ip-10-0-136-25 kubenswrapper[2569]: I0706 13:50:47.130109 2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-dcdgc" podStartSLOduration=3.476677253 podStartE2EDuration="22.130094681s" totalImagesPullingTime="18.653417428s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-06 13:50:25 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 13:50:47.110151111 +0000 UTC m=+22.700712889" watchObservedRunningTime="2026-07-06 13:50:47.130094681 +0000 UTC m=+22.720656480" Jul 06 13:50:47.130414 ip-10-0-136-25 kubenswrapper[2569]: I0706 13:50:47.130382 2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/node-resolver-54w6r" podStartSLOduration=3.544919754 podStartE2EDuration="22.130373703s" totalImagesPullingTime="18.585453949s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-06 13:50:25 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 13:50:47.12999002 +0000 UTC m=+22.720551822" watchObservedRunningTime="2026-07-06 13:50:47.130373703 +0000 UTC m=+22.720935480" Jul 06 13:50:47.156049 ip-10-0-136-25 kubenswrapper[2569]: I0706 13:50:47.156002 2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/node-ca-c9sqv" podStartSLOduration=4.568469293 podStartE2EDuration="23.155986908s" totalImagesPullingTime="18.587517615s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-06 13:50:24 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 13:50:47.155433229 +0000 UTC m=+22.745995007" watchObservedRunningTime="2026-07-06 13:50:47.155986908 +0000 UTC m=+22.746548684" Jul 06 13:50:47.943054 ip-10-0-136-25 kubenswrapper[2569]: I0706 13:50:47.943024 2569 plugin_watcher.go:194] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock" Jul 06 13:50:48.044972 ip-10-0-136-25 kubenswrapper[2569]: I0706 13:50:48.044921 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-wxxjp" event={"ID":"5bbb0d3f-2cd8-4496-b676-6045376a5b13","Type":"ContainerStarted","Data":"c64c56084a0b52a2ba7694986bc8390a6ce17de5c90c1de659894c08777217e3"} Jul 06 13:50:48.046900 ip-10-0-136-25 kubenswrapper[2569]: I0706 13:50:48.046877 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-2dnkc" event={"ID":"03d48054-9789-444b-82c7-ac9e3a504c9c","Type":"ContainerStarted","Data":"489bb30c5a3377bf65cbef3ea7ef491a41adcc6b1735fe68ead596bef0f42f6d"} Jul 06 13:50:48.050390 ip-10-0-136-25 kubenswrapper[2569]: I0706 13:50:48.050362 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-vcxnc" event={"ID":"4ae58dad-494b-4508-b5a5-129f13818d53","Type":"ContainerStarted","Data":"614118f1f0f775180359591f4f5aa59383a580a26eaaf43be8523de6f279ba9e"} Jul 06 13:50:48.050508 ip-10-0-136-25 kubenswrapper[2569]: I0706 13:50:48.050433 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-vcxnc" event={"ID":"4ae58dad-494b-4508-b5a5-129f13818d53","Type":"ContainerStarted","Data":"d91a13e1462530cd8664abbe8a25172f70f24c007f1a3112f395f532daa4d342"} Jul 06 13:50:48.060568 ip-10-0-136-25 kubenswrapper[2569]: I0706 13:50:48.060480 2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-operator/iptables-alerter-wxxjp" podStartSLOduration=4.46473208 podStartE2EDuration="23.060462411s" totalImagesPullingTime="18.595730331s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-06 13:50:25 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 13:50:48.059038896 +0000 UTC m=+23.649600692" watchObservedRunningTime="2026-07-06 13:50:48.060462411 +0000 UTC m=+23.651024188" Jul 06 13:50:48.865783 ip-10-0-136-25 kubenswrapper[2569]: I0706 13:50:48.865679 2569 reconciler.go:161] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-07-06T13:50:47.943047327Z","UUID":"723e8283-03bb-4048-bbcb-b1d41d2a250d","Handler":null,"Name":"","Endpoint":""} Jul 06 13:50:48.867439 ip-10-0-136-25 kubenswrapper[2569]: I0706 13:50:48.867415 2569 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0 Jul 06 13:50:48.867551 ip-10-0-136-25 kubenswrapper[2569]: I0706 13:50:48.867453 2569 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock Jul 06 13:50:48.934469 ip-10-0-136-25 kubenswrapper[2569]: I0706 13:50:48.934424 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-gbq5d" Jul 06 13:50:48.934655 ip-10-0-136-25 kubenswrapper[2569]: I0706 13:50:48.934424 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-hqt9l" Jul 06 13:50:48.934655 ip-10-0-136-25 kubenswrapper[2569]: E0706 13:50:48.934550 2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-gbq5d" podUID="d2d86b02-1e00-4267-b9c4-dd365b15342e" Jul 06 13:50:48.934655 ip-10-0-136-25 kubenswrapper[2569]: I0706 13:50:48.934424 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-cg8ph" Jul 06 13:50:48.934816 ip-10-0-136-25 kubenswrapper[2569]: E0706 13:50:48.934660 2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-hqt9l" podUID="2c4b4cf7-31a7-4cde-8212-9d78cc184e6c" Jul 06 13:50:48.934816 ip-10-0-136-25 kubenswrapper[2569]: E0706 13:50:48.934748 2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-cg8ph" podUID="576fd8e3-f3a0-48d2-a0d6-faa782a25a63" Jul 06 13:50:48.972397 ip-10-0-136-25 kubenswrapper[2569]: I0706 13:50:48.972373 2569 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kube-system/konnectivity-agent-vqmcq" Jul 06 13:50:48.973005 ip-10-0-136-25 kubenswrapper[2569]: I0706 13:50:48.972986 2569 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kube-system/konnectivity-agent-vqmcq" Jul 06 13:50:49.053569 ip-10-0-136-25 kubenswrapper[2569]: I0706 13:50:49.053464 2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kube-system/konnectivity-agent-vqmcq" Jul 06 13:50:49.053912 ip-10-0-136-25 kubenswrapper[2569]: I0706 13:50:49.053889 2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-vqmcq" Jul 06 13:50:50.057969 ip-10-0-136-25 kubenswrapper[2569]: I0706 13:50:50.057736 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-2dnkc" event={"ID":"03d48054-9789-444b-82c7-ac9e3a504c9c","Type":"ContainerStarted","Data":"c1c7d971f2538842e42af56fc31ad748e46280f7373492db9f3b2c3f31ad3e10"} Jul 06 13:50:50.061597 ip-10-0-136-25 kubenswrapper[2569]: I0706 13:50:50.061570 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-vcxnc" event={"ID":"4ae58dad-494b-4508-b5a5-129f13818d53","Type":"ContainerStarted","Data":"1006a6a3888d1dc9eb0f9e20ec307189b2ef349a02f65543b9a761920ce9d8b3"} Jul 06 13:50:50.079347 ip-10-0-136-25 kubenswrapper[2569]: I0706 13:50:50.079300 2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-2dnkc" podStartSLOduration=4.302918145 podStartE2EDuration="25.079280017s" totalImagesPullingTime="20.776361872s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-06 13:50:25 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 13:50:50.079116731 +0000 UTC m=+25.669678532" watchObservedRunningTime="2026-07-06 13:50:50.079280017 +0000 UTC m=+25.669841797" Jul 06 13:50:50.936813 ip-10-0-136-25 kubenswrapper[2569]: I0706 13:50:50.936791 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-gbq5d" Jul 06 13:50:50.936988 ip-10-0-136-25 kubenswrapper[2569]: E0706 13:50:50.936895 2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-gbq5d" podUID="d2d86b02-1e00-4267-b9c4-dd365b15342e" Jul 06 13:50:50.937039 ip-10-0-136-25 kubenswrapper[2569]: I0706 13:50:50.936978 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-cg8ph" Jul 06 13:50:50.937137 ip-10-0-136-25 kubenswrapper[2569]: E0706 13:50:50.937113 2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-cg8ph" podUID="576fd8e3-f3a0-48d2-a0d6-faa782a25a63" Jul 06 13:50:50.937320 ip-10-0-136-25 kubenswrapper[2569]: I0706 13:50:50.937306 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-hqt9l" Jul 06 13:50:50.937389 ip-10-0-136-25 kubenswrapper[2569]: E0706 13:50:50.937376 2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-hqt9l" podUID="2c4b4cf7-31a7-4cde-8212-9d78cc184e6c" Jul 06 13:50:52.067379 ip-10-0-136-25 kubenswrapper[2569]: I0706 13:50:52.067135 2569 generic.go:356] "Generic (PLEG): container finished" podID="edbb90e8-546b-4edd-a223-5ad4d579adaa" containerID="fed505552ab726d7698fdc7ce8f5b7b5ffcf0ddd2013b42f309e67998028f436" exitCode=0 Jul 06 13:50:52.068254 ip-10-0-136-25 kubenswrapper[2569]: I0706 13:50:52.067229 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-brmhg" event={"ID":"edbb90e8-546b-4edd-a223-5ad4d579adaa","Type":"ContainerDied","Data":"fed505552ab726d7698fdc7ce8f5b7b5ffcf0ddd2013b42f309e67998028f436"} Jul 06 13:50:52.071280 ip-10-0-136-25 kubenswrapper[2569]: I0706 13:50:52.071250 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-vcxnc" event={"ID":"4ae58dad-494b-4508-b5a5-129f13818d53","Type":"ContainerStarted","Data":"5f81f9891491f7ba1ab9c6b9ef74abfb0731f11cddd5e95164458a9d250dba78"} Jul 06 13:50:52.071518 ip-10-0-136-25 kubenswrapper[2569]: I0706 13:50:52.071494 2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-vcxnc" Jul 06 13:50:52.071647 ip-10-0-136-25 kubenswrapper[2569]: I0706 13:50:52.071582 2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-vcxnc" Jul 06 13:50:52.087173 ip-10-0-136-25 kubenswrapper[2569]: I0706 13:50:52.087148 2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-vcxnc" Jul 06 13:50:52.087307 ip-10-0-136-25 kubenswrapper[2569]: I0706 13:50:52.087283 2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-vcxnc" Jul 06 13:50:52.113815 ip-10-0-136-25 kubenswrapper[2569]: I0706 13:50:52.113758 2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ovn-kubernetes/ovnkube-node-vcxnc" podStartSLOduration=9.468977268 podStartE2EDuration="28.113743501s" totalImagesPullingTime="18.644766233s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-06 13:50:24 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 13:50:52.11283695 +0000 UTC m=+27.703398726" watchObservedRunningTime="2026-07-06 13:50:52.113743501 +0000 UTC m=+27.704305277" Jul 06 13:50:52.763522 ip-10-0-136-25 kubenswrapper[2569]: I0706 13:50:52.763320 2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-vcxnc" Jul 06 13:50:52.934046 ip-10-0-136-25 kubenswrapper[2569]: I0706 13:50:52.934020 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-gbq5d" Jul 06 13:50:52.934266 ip-10-0-136-25 kubenswrapper[2569]: I0706 13:50:52.934020 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-hqt9l" Jul 06 13:50:52.934266 ip-10-0-136-25 kubenswrapper[2569]: E0706 13:50:52.934142 2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-gbq5d" podUID="d2d86b02-1e00-4267-b9c4-dd365b15342e" Jul 06 13:50:52.934266 ip-10-0-136-25 kubenswrapper[2569]: I0706 13:50:52.934020 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-cg8ph" Jul 06 13:50:52.934266 ip-10-0-136-25 kubenswrapper[2569]: E0706 13:50:52.934251 2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-hqt9l" podUID="2c4b4cf7-31a7-4cde-8212-9d78cc184e6c" Jul 06 13:50:52.934469 ip-10-0-136-25 kubenswrapper[2569]: E0706 13:50:52.934311 2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-cg8ph" podUID="576fd8e3-f3a0-48d2-a0d6-faa782a25a63" Jul 06 13:50:53.478169 ip-10-0-136-25 kubenswrapper[2569]: I0706 13:50:53.477945 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-cg8ph"] Jul 06 13:50:53.478670 ip-10-0-136-25 kubenswrapper[2569]: I0706 13:50:53.478318 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-cg8ph" Jul 06 13:50:53.478670 ip-10-0-136-25 kubenswrapper[2569]: E0706 13:50:53.478409 2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-cg8ph" podUID="576fd8e3-f3a0-48d2-a0d6-faa782a25a63" Jul 06 13:50:53.478670 ip-10-0-136-25 kubenswrapper[2569]: I0706 13:50:53.478532 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-hqt9l"] Jul 06 13:50:53.478670 ip-10-0-136-25 kubenswrapper[2569]: I0706 13:50:53.478626 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-hqt9l" Jul 06 13:50:53.478803 ip-10-0-136-25 kubenswrapper[2569]: E0706 13:50:53.478701 2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-hqt9l" podUID="2c4b4cf7-31a7-4cde-8212-9d78cc184e6c" Jul 06 13:50:53.490626 ip-10-0-136-25 kubenswrapper[2569]: I0706 13:50:53.490602 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-gbq5d"] Jul 06 13:50:53.490778 ip-10-0-136-25 kubenswrapper[2569]: I0706 13:50:53.490705 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-gbq5d" Jul 06 13:50:53.490845 ip-10-0-136-25 kubenswrapper[2569]: E0706 13:50:53.490803 2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-gbq5d" podUID="d2d86b02-1e00-4267-b9c4-dd365b15342e" Jul 06 13:50:54.076885 ip-10-0-136-25 kubenswrapper[2569]: I0706 13:50:54.076848 2569 generic.go:356] "Generic (PLEG): container finished" podID="edbb90e8-546b-4edd-a223-5ad4d579adaa" containerID="661f8a6100dcaa882a878e99bc7456b2b8821d1bb13484b638cf1d291b9f29f6" exitCode=0 Jul 06 13:50:54.077127 ip-10-0-136-25 kubenswrapper[2569]: I0706 13:50:54.076934 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-brmhg" event={"ID":"edbb90e8-546b-4edd-a223-5ad4d579adaa","Type":"ContainerDied","Data":"661f8a6100dcaa882a878e99bc7456b2b8821d1bb13484b638cf1d291b9f29f6"} Jul 06 13:50:54.935538 ip-10-0-136-25 kubenswrapper[2569]: I0706 13:50:54.935504 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-hqt9l" Jul 06 13:50:54.936111 ip-10-0-136-25 kubenswrapper[2569]: E0706 13:50:54.935615 2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-hqt9l" podUID="2c4b4cf7-31a7-4cde-8212-9d78cc184e6c" Jul 06 13:50:54.936111 ip-10-0-136-25 kubenswrapper[2569]: I0706 13:50:54.935685 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-gbq5d" Jul 06 13:50:54.936111 ip-10-0-136-25 kubenswrapper[2569]: I0706 13:50:54.935767 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-cg8ph" Jul 06 13:50:54.936111 ip-10-0-136-25 kubenswrapper[2569]: E0706 13:50:54.935793 2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-gbq5d" podUID="d2d86b02-1e00-4267-b9c4-dd365b15342e" Jul 06 13:50:54.936111 ip-10-0-136-25 kubenswrapper[2569]: E0706 13:50:54.935872 2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-cg8ph" podUID="576fd8e3-f3a0-48d2-a0d6-faa782a25a63" Jul 06 13:50:56.084995 ip-10-0-136-25 kubenswrapper[2569]: I0706 13:50:56.084961 2569 generic.go:356] "Generic (PLEG): container finished" podID="edbb90e8-546b-4edd-a223-5ad4d579adaa" containerID="f217fb222d37d4993c37c34d57d834d9670b9da3d4ebaaab1cee891820785cd2" exitCode=0 Jul 06 13:50:56.085527 ip-10-0-136-25 kubenswrapper[2569]: I0706 13:50:56.085019 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-brmhg" event={"ID":"edbb90e8-546b-4edd-a223-5ad4d579adaa","Type":"ContainerDied","Data":"f217fb222d37d4993c37c34d57d834d9670b9da3d4ebaaab1cee891820785cd2"} Jul 06 13:50:56.934803 ip-10-0-136-25 kubenswrapper[2569]: I0706 13:50:56.934771 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-gbq5d" Jul 06 13:50:56.934996 ip-10-0-136-25 kubenswrapper[2569]: I0706 13:50:56.934842 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-hqt9l" Jul 06 13:50:56.934996 ip-10-0-136-25 kubenswrapper[2569]: I0706 13:50:56.934845 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-cg8ph" Jul 06 13:50:56.934996 ip-10-0-136-25 kubenswrapper[2569]: E0706 13:50:56.934947 2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-hqt9l" podUID="2c4b4cf7-31a7-4cde-8212-9d78cc184e6c" Jul 06 13:50:56.935163 ip-10-0-136-25 kubenswrapper[2569]: E0706 13:50:56.935036 2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-gbq5d" podUID="d2d86b02-1e00-4267-b9c4-dd365b15342e" Jul 06 13:50:56.935163 ip-10-0-136-25 kubenswrapper[2569]: E0706 13:50:56.935125 2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-cg8ph" podUID="576fd8e3-f3a0-48d2-a0d6-faa782a25a63" Jul 06 13:50:58.648120 ip-10-0-136-25 kubenswrapper[2569]: I0706 13:50:58.647899 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4kcgx\" (UniqueName: \"kubernetes.io/projected/d2d86b02-1e00-4267-b9c4-dd365b15342e-kube-api-access-4kcgx\") pod \"network-check-target-gbq5d\" (UID: \"d2d86b02-1e00-4267-b9c4-dd365b15342e\") " pod="openshift-network-diagnostics/network-check-target-gbq5d" Jul 06 13:50:58.648695 ip-10-0-136-25 kubenswrapper[2569]: I0706 13:50:58.648151 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/576fd8e3-f3a0-48d2-a0d6-faa782a25a63-original-pull-secret\") pod \"global-pull-secret-syncer-cg8ph\" (UID: \"576fd8e3-f3a0-48d2-a0d6-faa782a25a63\") " pod="kube-system/global-pull-secret-syncer-cg8ph" Jul 06 13:50:58.648695 ip-10-0-136-25 kubenswrapper[2569]: E0706 13:50:58.648049 2569 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 06 13:50:58.648695 ip-10-0-136-25 kubenswrapper[2569]: E0706 13:50:58.648248 2569 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 06 13:50:58.648695 ip-10-0-136-25 kubenswrapper[2569]: E0706 13:50:58.648263 2569 projected.go:196] Error preparing data for projected volume kube-api-access-4kcgx for pod openshift-network-diagnostics/network-check-target-gbq5d: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 06 13:50:58.648695 ip-10-0-136-25 kubenswrapper[2569]: E0706 13:50:58.648265 2569 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 06 13:50:58.648695 ip-10-0-136-25 kubenswrapper[2569]: I0706 13:50:58.648192 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2c4b4cf7-31a7-4cde-8212-9d78cc184e6c-metrics-certs\") pod \"network-metrics-daemon-hqt9l\" (UID: \"2c4b4cf7-31a7-4cde-8212-9d78cc184e6c\") " pod="openshift-multus/network-metrics-daemon-hqt9l" Jul 06 13:50:58.648695 ip-10-0-136-25 kubenswrapper[2569]: E0706 13:50:58.648291 2569 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 06 13:50:58.648695 ip-10-0-136-25 kubenswrapper[2569]: E0706 13:50:58.648317 2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/d2d86b02-1e00-4267-b9c4-dd365b15342e-kube-api-access-4kcgx podName:d2d86b02-1e00-4267-b9c4-dd365b15342e nodeName:}" failed. No retries permitted until 2026-07-06 13:51:30.648298793 +0000 UTC m=+66.238860549 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "kube-api-access-4kcgx" (UniqueName: "kubernetes.io/projected/d2d86b02-1e00-4267-b9c4-dd365b15342e-kube-api-access-4kcgx") pod "network-check-target-gbq5d" (UID: "d2d86b02-1e00-4267-b9c4-dd365b15342e") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 06 13:50:58.648695 ip-10-0-136-25 kubenswrapper[2569]: E0706 13:50:58.648354 2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2c4b4cf7-31a7-4cde-8212-9d78cc184e6c-metrics-certs podName:2c4b4cf7-31a7-4cde-8212-9d78cc184e6c nodeName:}" failed. No retries permitted until 2026-07-06 13:51:30.648342777 +0000 UTC m=+66.238904533 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/2c4b4cf7-31a7-4cde-8212-9d78cc184e6c-metrics-certs") pod "network-metrics-daemon-hqt9l" (UID: "2c4b4cf7-31a7-4cde-8212-9d78cc184e6c") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 06 13:50:58.648695 ip-10-0-136-25 kubenswrapper[2569]: E0706 13:50:58.648387 2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/576fd8e3-f3a0-48d2-a0d6-faa782a25a63-original-pull-secret podName:576fd8e3-f3a0-48d2-a0d6-faa782a25a63 nodeName:}" failed. No retries permitted until 2026-07-06 13:51:30.648367178 +0000 UTC m=+66.238928942 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/576fd8e3-f3a0-48d2-a0d6-faa782a25a63-original-pull-secret") pod "global-pull-secret-syncer-cg8ph" (UID: "576fd8e3-f3a0-48d2-a0d6-faa782a25a63") : object "kube-system"/"original-pull-secret" not registered Jul 06 13:50:58.934944 ip-10-0-136-25 kubenswrapper[2569]: I0706 13:50:58.934831 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-hqt9l" Jul 06 13:50:58.934944 ip-10-0-136-25 kubenswrapper[2569]: I0706 13:50:58.934874 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-gbq5d" Jul 06 13:50:58.934944 ip-10-0-136-25 kubenswrapper[2569]: I0706 13:50:58.934832 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-cg8ph" Jul 06 13:50:58.935203 ip-10-0-136-25 kubenswrapper[2569]: E0706 13:50:58.934974 2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-hqt9l" podUID="2c4b4cf7-31a7-4cde-8212-9d78cc184e6c" Jul 06 13:50:58.935203 ip-10-0-136-25 kubenswrapper[2569]: E0706 13:50:58.935123 2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-cg8ph" podUID="576fd8e3-f3a0-48d2-a0d6-faa782a25a63" Jul 06 13:50:58.935331 ip-10-0-136-25 kubenswrapper[2569]: E0706 13:50:58.935234 2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-gbq5d" podUID="d2d86b02-1e00-4267-b9c4-dd365b15342e" Jul 06 13:51:00.241091 ip-10-0-136-25 kubenswrapper[2569]: I0706 13:51:00.241061 2569 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-136-25.ec2.internal" event="NodeReady" Jul 06 13:51:00.241565 ip-10-0-136-25 kubenswrapper[2569]: I0706 13:51:00.241193 2569 kubelet_node_status.go:489] "Fast updating node status as it just became ready" Jul 06 13:51:00.296551 ip-10-0-136-25 kubenswrapper[2569]: I0706 13:51:00.296516 2569 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-dns/dns-default-kbv9m"] Jul 06 13:51:00.312330 ip-10-0-136-25 kubenswrapper[2569]: I0706 13:51:00.312302 2569 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress-canary/ingress-canary-4wkrh"] Jul 06 13:51:00.312481 ip-10-0-136-25 kubenswrapper[2569]: I0706 13:51:00.312419 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-kbv9m" Jul 06 13:51:00.315057 ip-10-0-136-25 kubenswrapper[2569]: I0706 13:51:00.314565 2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\"" Jul 06 13:51:00.315057 ip-10-0-136-25 kubenswrapper[2569]: I0706 13:51:00.314882 2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-szq2s\"" Jul 06 13:51:00.315281 ip-10-0-136-25 kubenswrapper[2569]: I0706 13:51:00.315228 2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\"" Jul 06 13:51:00.339543 ip-10-0-136-25 kubenswrapper[2569]: I0706 13:51:00.339522 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-kbv9m"] Jul 06 13:51:00.339672 ip-10-0-136-25 kubenswrapper[2569]: I0706 13:51:00.339556 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-4wkrh"] Jul 06 13:51:00.339672 ip-10-0-136-25 kubenswrapper[2569]: I0706 13:51:00.339632 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-4wkrh" Jul 06 13:51:00.341743 ip-10-0-136-25 kubenswrapper[2569]: I0706 13:51:00.341724 2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\"" Jul 06 13:51:00.341871 ip-10-0-136-25 kubenswrapper[2569]: I0706 13:51:00.341729 2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\"" Jul 06 13:51:00.341871 ip-10-0-136-25 kubenswrapper[2569]: I0706 13:51:00.341814 2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"default-dockercfg-fsr9j\"" Jul 06 13:51:00.342044 ip-10-0-136-25 kubenswrapper[2569]: I0706 13:51:00.342032 2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\"" Jul 06 13:51:00.360339 ip-10-0-136-25 kubenswrapper[2569]: I0706 13:51:00.360321 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/420fd5c7-d449-4869-960a-4b14a873c402-config-volume\") pod \"dns-default-kbv9m\" (UID: \"420fd5c7-d449-4869-960a-4b14a873c402\") " pod="openshift-dns/dns-default-kbv9m" Jul 06 13:51:00.360451 ip-10-0-136-25 kubenswrapper[2569]: I0706 13:51:00.360349 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/420fd5c7-d449-4869-960a-4b14a873c402-metrics-tls\") pod \"dns-default-kbv9m\" (UID: \"420fd5c7-d449-4869-960a-4b14a873c402\") " pod="openshift-dns/dns-default-kbv9m" Jul 06 13:51:00.360451 ip-10-0-136-25 kubenswrapper[2569]: I0706 13:51:00.360383 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/420fd5c7-d449-4869-960a-4b14a873c402-tmp-dir\") pod \"dns-default-kbv9m\" (UID: \"420fd5c7-d449-4869-960a-4b14a873c402\") " pod="openshift-dns/dns-default-kbv9m" Jul 06 13:51:00.360522 ip-10-0-136-25 kubenswrapper[2569]: I0706 13:51:00.360481 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vxx7q\" (UniqueName: \"kubernetes.io/projected/420fd5c7-d449-4869-960a-4b14a873c402-kube-api-access-vxx7q\") pod \"dns-default-kbv9m\" (UID: \"420fd5c7-d449-4869-960a-4b14a873c402\") " pod="openshift-dns/dns-default-kbv9m" Jul 06 13:51:00.461496 ip-10-0-136-25 kubenswrapper[2569]: I0706 13:51:00.461472 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/420fd5c7-d449-4869-960a-4b14a873c402-config-volume\") pod \"dns-default-kbv9m\" (UID: \"420fd5c7-d449-4869-960a-4b14a873c402\") " pod="openshift-dns/dns-default-kbv9m" Jul 06 13:51:00.461611 ip-10-0-136-25 kubenswrapper[2569]: I0706 13:51:00.461499 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/420fd5c7-d449-4869-960a-4b14a873c402-metrics-tls\") pod \"dns-default-kbv9m\" (UID: \"420fd5c7-d449-4869-960a-4b14a873c402\") " pod="openshift-dns/dns-default-kbv9m" Jul 06 13:51:00.461611 ip-10-0-136-25 kubenswrapper[2569]: I0706 13:51:00.461522 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/420fd5c7-d449-4869-960a-4b14a873c402-tmp-dir\") pod \"dns-default-kbv9m\" (UID: \"420fd5c7-d449-4869-960a-4b14a873c402\") " pod="openshift-dns/dns-default-kbv9m" Jul 06 13:51:00.461611 ip-10-0-136-25 kubenswrapper[2569]: I0706 13:51:00.461541 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4fxzh\" (UniqueName: \"kubernetes.io/projected/e5a5cf60-4e99-42b0-b705-96449966ca70-kube-api-access-4fxzh\") pod \"ingress-canary-4wkrh\" (UID: \"e5a5cf60-4e99-42b0-b705-96449966ca70\") " pod="openshift-ingress-canary/ingress-canary-4wkrh" Jul 06 13:51:00.461611 ip-10-0-136-25 kubenswrapper[2569]: I0706 13:51:00.461573 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/e5a5cf60-4e99-42b0-b705-96449966ca70-cert\") pod \"ingress-canary-4wkrh\" (UID: \"e5a5cf60-4e99-42b0-b705-96449966ca70\") " pod="openshift-ingress-canary/ingress-canary-4wkrh" Jul 06 13:51:00.461812 ip-10-0-136-25 kubenswrapper[2569]: I0706 13:51:00.461618 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vxx7q\" (UniqueName: \"kubernetes.io/projected/420fd5c7-d449-4869-960a-4b14a873c402-kube-api-access-vxx7q\") pod \"dns-default-kbv9m\" (UID: \"420fd5c7-d449-4869-960a-4b14a873c402\") " pod="openshift-dns/dns-default-kbv9m" Jul 06 13:51:00.461812 ip-10-0-136-25 kubenswrapper[2569]: E0706 13:51:00.461645 2569 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jul 06 13:51:00.461812 ip-10-0-136-25 kubenswrapper[2569]: E0706 13:51:00.461702 2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/420fd5c7-d449-4869-960a-4b14a873c402-metrics-tls podName:420fd5c7-d449-4869-960a-4b14a873c402 nodeName:}" failed. No retries permitted until 2026-07-06 13:51:00.961681723 +0000 UTC m=+36.552243479 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/420fd5c7-d449-4869-960a-4b14a873c402-metrics-tls") pod "dns-default-kbv9m" (UID: "420fd5c7-d449-4869-960a-4b14a873c402") : secret "dns-default-metrics-tls" not found Jul 06 13:51:00.461968 ip-10-0-136-25 kubenswrapper[2569]: I0706 13:51:00.461810 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/420fd5c7-d449-4869-960a-4b14a873c402-tmp-dir\") pod \"dns-default-kbv9m\" (UID: \"420fd5c7-d449-4869-960a-4b14a873c402\") " pod="openshift-dns/dns-default-kbv9m" Jul 06 13:51:00.462096 ip-10-0-136-25 kubenswrapper[2569]: I0706 13:51:00.462075 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/420fd5c7-d449-4869-960a-4b14a873c402-config-volume\") pod \"dns-default-kbv9m\" (UID: \"420fd5c7-d449-4869-960a-4b14a873c402\") " pod="openshift-dns/dns-default-kbv9m" Jul 06 13:51:00.472489 ip-10-0-136-25 kubenswrapper[2569]: I0706 13:51:00.472412 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vxx7q\" (UniqueName: \"kubernetes.io/projected/420fd5c7-d449-4869-960a-4b14a873c402-kube-api-access-vxx7q\") pod \"dns-default-kbv9m\" (UID: \"420fd5c7-d449-4869-960a-4b14a873c402\") " pod="openshift-dns/dns-default-kbv9m" Jul 06 13:51:00.562843 ip-10-0-136-25 kubenswrapper[2569]: I0706 13:51:00.562823 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4fxzh\" (UniqueName: \"kubernetes.io/projected/e5a5cf60-4e99-42b0-b705-96449966ca70-kube-api-access-4fxzh\") pod \"ingress-canary-4wkrh\" (UID: \"e5a5cf60-4e99-42b0-b705-96449966ca70\") " pod="openshift-ingress-canary/ingress-canary-4wkrh" Jul 06 13:51:00.562933 ip-10-0-136-25 kubenswrapper[2569]: I0706 13:51:00.562860 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/e5a5cf60-4e99-42b0-b705-96449966ca70-cert\") pod \"ingress-canary-4wkrh\" (UID: \"e5a5cf60-4e99-42b0-b705-96449966ca70\") " pod="openshift-ingress-canary/ingress-canary-4wkrh" Jul 06 13:51:00.565483 ip-10-0-136-25 kubenswrapper[2569]: I0706 13:51:00.565462 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/e5a5cf60-4e99-42b0-b705-96449966ca70-cert\") pod \"ingress-canary-4wkrh\" (UID: \"e5a5cf60-4e99-42b0-b705-96449966ca70\") " pod="openshift-ingress-canary/ingress-canary-4wkrh" Jul 06 13:51:00.574631 ip-10-0-136-25 kubenswrapper[2569]: I0706 13:51:00.574611 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4fxzh\" (UniqueName: \"kubernetes.io/projected/e5a5cf60-4e99-42b0-b705-96449966ca70-kube-api-access-4fxzh\") pod \"ingress-canary-4wkrh\" (UID: \"e5a5cf60-4e99-42b0-b705-96449966ca70\") " pod="openshift-ingress-canary/ingress-canary-4wkrh" Jul 06 13:51:00.647449 ip-10-0-136-25 kubenswrapper[2569]: I0706 13:51:00.647428 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-4wkrh" Jul 06 13:51:00.777654 ip-10-0-136-25 kubenswrapper[2569]: I0706 13:51:00.777582 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-4wkrh"] Jul 06 13:51:00.781431 ip-10-0-136-25 kubenswrapper[2569]: W0706 13:51:00.781397 2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode5a5cf60_4e99_42b0_b705_96449966ca70.slice/crio-1655377f141ae4a6d9156719cdd55603e3832a4b894172e45a91eaa4f738c53b WatchSource:0}: Error finding container 1655377f141ae4a6d9156719cdd55603e3832a4b894172e45a91eaa4f738c53b: Status 404 returned error can't find the container with id 1655377f141ae4a6d9156719cdd55603e3832a4b894172e45a91eaa4f738c53b Jul 06 13:51:00.934095 ip-10-0-136-25 kubenswrapper[2569]: I0706 13:51:00.934060 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-hqt9l" Jul 06 13:51:00.934295 ip-10-0-136-25 kubenswrapper[2569]: I0706 13:51:00.934060 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-gbq5d" Jul 06 13:51:00.934295 ip-10-0-136-25 kubenswrapper[2569]: I0706 13:51:00.934189 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-cg8ph" Jul 06 13:51:00.937012 ip-10-0-136-25 kubenswrapper[2569]: I0706 13:51:00.936993 2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\"" Jul 06 13:51:00.937131 ip-10-0-136-25 kubenswrapper[2569]: I0706 13:51:00.937094 2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-hr9ft\"" Jul 06 13:51:00.937559 ip-10-0-136-25 kubenswrapper[2569]: I0706 13:51:00.937540 2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\"" Jul 06 13:51:00.937670 ip-10-0-136-25 kubenswrapper[2569]: I0706 13:51:00.937544 2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\"" Jul 06 13:51:00.937670 ip-10-0-136-25 kubenswrapper[2569]: I0706 13:51:00.937565 2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-469kc\"" Jul 06 13:51:00.937670 ip-10-0-136-25 kubenswrapper[2569]: I0706 13:51:00.937588 2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\"" Jul 06 13:51:00.966243 ip-10-0-136-25 kubenswrapper[2569]: I0706 13:51:00.966204 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/420fd5c7-d449-4869-960a-4b14a873c402-metrics-tls\") pod \"dns-default-kbv9m\" (UID: \"420fd5c7-d449-4869-960a-4b14a873c402\") " pod="openshift-dns/dns-default-kbv9m" Jul 06 13:51:00.966340 ip-10-0-136-25 kubenswrapper[2569]: E0706 13:51:00.966317 2569 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jul 06 13:51:00.966401 ip-10-0-136-25 kubenswrapper[2569]: E0706 13:51:00.966388 2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/420fd5c7-d449-4869-960a-4b14a873c402-metrics-tls podName:420fd5c7-d449-4869-960a-4b14a873c402 nodeName:}" failed. No retries permitted until 2026-07-06 13:51:01.96636721 +0000 UTC m=+37.556928978 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/420fd5c7-d449-4869-960a-4b14a873c402-metrics-tls") pod "dns-default-kbv9m" (UID: "420fd5c7-d449-4869-960a-4b14a873c402") : secret "dns-default-metrics-tls" not found Jul 06 13:51:01.100655 ip-10-0-136-25 kubenswrapper[2569]: I0706 13:51:01.100572 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-4wkrh" event={"ID":"e5a5cf60-4e99-42b0-b705-96449966ca70","Type":"ContainerStarted","Data":"1655377f141ae4a6d9156719cdd55603e3832a4b894172e45a91eaa4f738c53b"} Jul 06 13:51:01.103228 ip-10-0-136-25 kubenswrapper[2569]: I0706 13:51:01.103194 2569 generic.go:356] "Generic (PLEG): container finished" podID="edbb90e8-546b-4edd-a223-5ad4d579adaa" containerID="e8f057508afec341b46e8aac284daa45588644f68853939285998a114f733616" exitCode=0 Jul 06 13:51:01.103344 ip-10-0-136-25 kubenswrapper[2569]: I0706 13:51:01.103234 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-brmhg" event={"ID":"edbb90e8-546b-4edd-a223-5ad4d579adaa","Type":"ContainerDied","Data":"e8f057508afec341b46e8aac284daa45588644f68853939285998a114f733616"} Jul 06 13:51:01.975349 ip-10-0-136-25 kubenswrapper[2569]: I0706 13:51:01.975103 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/420fd5c7-d449-4869-960a-4b14a873c402-metrics-tls\") pod \"dns-default-kbv9m\" (UID: \"420fd5c7-d449-4869-960a-4b14a873c402\") " pod="openshift-dns/dns-default-kbv9m" Jul 06 13:51:01.975785 ip-10-0-136-25 kubenswrapper[2569]: E0706 13:51:01.975285 2569 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jul 06 13:51:01.975785 ip-10-0-136-25 kubenswrapper[2569]: E0706 13:51:01.975430 2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/420fd5c7-d449-4869-960a-4b14a873c402-metrics-tls podName:420fd5c7-d449-4869-960a-4b14a873c402 nodeName:}" failed. No retries permitted until 2026-07-06 13:51:03.975412995 +0000 UTC m=+39.565974750 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/420fd5c7-d449-4869-960a-4b14a873c402-metrics-tls") pod "dns-default-kbv9m" (UID: "420fd5c7-d449-4869-960a-4b14a873c402") : secret "dns-default-metrics-tls" not found Jul 06 13:51:02.109568 ip-10-0-136-25 kubenswrapper[2569]: I0706 13:51:02.109529 2569 generic.go:356] "Generic (PLEG): container finished" podID="edbb90e8-546b-4edd-a223-5ad4d579adaa" containerID="5c8903db81fe702d09d59ff067b37451b8af3f1aeb1ad1f9fd7a4da0e82d5e3f" exitCode=0 Jul 06 13:51:02.109568 ip-10-0-136-25 kubenswrapper[2569]: I0706 13:51:02.109570 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-brmhg" event={"ID":"edbb90e8-546b-4edd-a223-5ad4d579adaa","Type":"ContainerDied","Data":"5c8903db81fe702d09d59ff067b37451b8af3f1aeb1ad1f9fd7a4da0e82d5e3f"} Jul 06 13:51:03.116425 ip-10-0-136-25 kubenswrapper[2569]: I0706 13:51:03.116395 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-brmhg" event={"ID":"edbb90e8-546b-4edd-a223-5ad4d579adaa","Type":"ContainerStarted","Data":"b5a30fa456253bd7190d5b4963a0b5f48e86213fcd6ae407466cd895e247bc0a"} Jul 06 13:51:03.144602 ip-10-0-136-25 kubenswrapper[2569]: I0706 13:51:03.144436 2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-additional-cni-plugins-brmhg" podStartSLOduration=8.849288375 podStartE2EDuration="38.14441619s" totalImagesPullingTime="29.295127815s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-06 13:50:25 +0000 UTC" imagePullSessionsCount=5 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 13:51:03.144120556 +0000 UTC m=+38.734682356" watchObservedRunningTime="2026-07-06 13:51:03.14441619 +0000 UTC m=+38.734977966" Jul 06 13:51:03.995545 ip-10-0-136-25 kubenswrapper[2569]: I0706 13:51:03.995512 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/420fd5c7-d449-4869-960a-4b14a873c402-metrics-tls\") pod \"dns-default-kbv9m\" (UID: \"420fd5c7-d449-4869-960a-4b14a873c402\") " pod="openshift-dns/dns-default-kbv9m" Jul 06 13:51:03.995695 ip-10-0-136-25 kubenswrapper[2569]: E0706 13:51:03.995630 2569 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jul 06 13:51:03.995695 ip-10-0-136-25 kubenswrapper[2569]: E0706 13:51:03.995682 2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/420fd5c7-d449-4869-960a-4b14a873c402-metrics-tls podName:420fd5c7-d449-4869-960a-4b14a873c402 nodeName:}" failed. No retries permitted until 2026-07-06 13:51:07.995668601 +0000 UTC m=+43.586230356 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/420fd5c7-d449-4869-960a-4b14a873c402-metrics-tls") pod "dns-default-kbv9m" (UID: "420fd5c7-d449-4869-960a-4b14a873c402") : secret "dns-default-metrics-tls" not found Jul 06 13:51:04.120413 ip-10-0-136-25 kubenswrapper[2569]: I0706 13:51:04.120376 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-4wkrh" event={"ID":"e5a5cf60-4e99-42b0-b705-96449966ca70","Type":"ContainerStarted","Data":"1cca0dbcdfed81198a3e28ca54e585fbdd7c295548f4830dccb801263f455118"} Jul 06 13:51:04.142462 ip-10-0-136-25 kubenswrapper[2569]: I0706 13:51:04.141112 2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress-canary/ingress-canary-4wkrh" podStartSLOduration=1.579665369 podStartE2EDuration="4.141092848s" totalImagesPullingTime="2.561427479s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-06 13:51:00 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 13:51:04.138961253 +0000 UTC m=+39.729523048" watchObservedRunningTime="2026-07-06 13:51:04.141092848 +0000 UTC m=+39.731654628" Jul 06 13:51:08.023363 ip-10-0-136-25 kubenswrapper[2569]: I0706 13:51:08.023324 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/420fd5c7-d449-4869-960a-4b14a873c402-metrics-tls\") pod \"dns-default-kbv9m\" (UID: \"420fd5c7-d449-4869-960a-4b14a873c402\") " pod="openshift-dns/dns-default-kbv9m" Jul 06 13:51:08.023764 ip-10-0-136-25 kubenswrapper[2569]: E0706 13:51:08.023478 2569 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jul 06 13:51:08.023764 ip-10-0-136-25 kubenswrapper[2569]: E0706 13:51:08.023553 2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/420fd5c7-d449-4869-960a-4b14a873c402-metrics-tls podName:420fd5c7-d449-4869-960a-4b14a873c402 nodeName:}" failed. No retries permitted until 2026-07-06 13:51:16.023536453 +0000 UTC m=+51.614098208 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/420fd5c7-d449-4869-960a-4b14a873c402-metrics-tls") pod "dns-default-kbv9m" (UID: "420fd5c7-d449-4869-960a-4b14a873c402") : secret "dns-default-metrics-tls" not found Jul 06 13:51:16.083820 ip-10-0-136-25 kubenswrapper[2569]: I0706 13:51:16.083775 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/420fd5c7-d449-4869-960a-4b14a873c402-metrics-tls\") pod \"dns-default-kbv9m\" (UID: \"420fd5c7-d449-4869-960a-4b14a873c402\") " pod="openshift-dns/dns-default-kbv9m" Jul 06 13:51:16.084447 ip-10-0-136-25 kubenswrapper[2569]: E0706 13:51:16.083951 2569 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jul 06 13:51:16.084447 ip-10-0-136-25 kubenswrapper[2569]: E0706 13:51:16.084049 2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/420fd5c7-d449-4869-960a-4b14a873c402-metrics-tls podName:420fd5c7-d449-4869-960a-4b14a873c402 nodeName:}" failed. No retries permitted until 2026-07-06 13:51:32.084027232 +0000 UTC m=+67.674589006 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/420fd5c7-d449-4869-960a-4b14a873c402-metrics-tls") pod "dns-default-kbv9m" (UID: "420fd5c7-d449-4869-960a-4b14a873c402") : secret "dns-default-metrics-tls" not found Jul 06 13:51:24.087121 ip-10-0-136-25 kubenswrapper[2569]: I0706 13:51:24.087094 2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-vcxnc" Jul 06 13:51:30.686433 ip-10-0-136-25 kubenswrapper[2569]: I0706 13:51:30.686395 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4kcgx\" (UniqueName: \"kubernetes.io/projected/d2d86b02-1e00-4267-b9c4-dd365b15342e-kube-api-access-4kcgx\") pod \"network-check-target-gbq5d\" (UID: \"d2d86b02-1e00-4267-b9c4-dd365b15342e\") " pod="openshift-network-diagnostics/network-check-target-gbq5d" Jul 06 13:51:30.686433 ip-10-0-136-25 kubenswrapper[2569]: I0706 13:51:30.686440 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/576fd8e3-f3a0-48d2-a0d6-faa782a25a63-original-pull-secret\") pod \"global-pull-secret-syncer-cg8ph\" (UID: \"576fd8e3-f3a0-48d2-a0d6-faa782a25a63\") " pod="kube-system/global-pull-secret-syncer-cg8ph" Jul 06 13:51:30.687050 ip-10-0-136-25 kubenswrapper[2569]: I0706 13:51:30.686465 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2c4b4cf7-31a7-4cde-8212-9d78cc184e6c-metrics-certs\") pod \"network-metrics-daemon-hqt9l\" (UID: \"2c4b4cf7-31a7-4cde-8212-9d78cc184e6c\") " pod="openshift-multus/network-metrics-daemon-hqt9l" Jul 06 13:51:30.688665 ip-10-0-136-25 kubenswrapper[2569]: I0706 13:51:30.688647 2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\"" Jul 06 13:51:30.688942 ip-10-0-136-25 kubenswrapper[2569]: I0706 13:51:30.688915 2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\"" Jul 06 13:51:30.688994 ip-10-0-136-25 kubenswrapper[2569]: I0706 13:51:30.688916 2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\"" Jul 06 13:51:30.697631 ip-10-0-136-25 kubenswrapper[2569]: E0706 13:51:30.697600 2569 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found Jul 06 13:51:30.697766 ip-10-0-136-25 kubenswrapper[2569]: E0706 13:51:30.697698 2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2c4b4cf7-31a7-4cde-8212-9d78cc184e6c-metrics-certs podName:2c4b4cf7-31a7-4cde-8212-9d78cc184e6c nodeName:}" failed. No retries permitted until 2026-07-06 13:52:34.697674563 +0000 UTC m=+130.288236332 (durationBeforeRetry 1m4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/2c4b4cf7-31a7-4cde-8212-9d78cc184e6c-metrics-certs") pod "network-metrics-daemon-hqt9l" (UID: "2c4b4cf7-31a7-4cde-8212-9d78cc184e6c") : secret "metrics-daemon-secret" not found Jul 06 13:51:30.699369 ip-10-0-136-25 kubenswrapper[2569]: I0706 13:51:30.699354 2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\"" Jul 06 13:51:30.701009 ip-10-0-136-25 kubenswrapper[2569]: I0706 13:51:30.700987 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/576fd8e3-f3a0-48d2-a0d6-faa782a25a63-original-pull-secret\") pod \"global-pull-secret-syncer-cg8ph\" (UID: \"576fd8e3-f3a0-48d2-a0d6-faa782a25a63\") " pod="kube-system/global-pull-secret-syncer-cg8ph" Jul 06 13:51:30.726664 ip-10-0-136-25 kubenswrapper[2569]: I0706 13:51:30.726560 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4kcgx\" (UniqueName: \"kubernetes.io/projected/d2d86b02-1e00-4267-b9c4-dd365b15342e-kube-api-access-4kcgx\") pod \"network-check-target-gbq5d\" (UID: \"d2d86b02-1e00-4267-b9c4-dd365b15342e\") " pod="openshift-network-diagnostics/network-check-target-gbq5d" Jul 06 13:51:30.952501 ip-10-0-136-25 kubenswrapper[2569]: I0706 13:51:30.952415 2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-hr9ft\"" Jul 06 13:51:30.954973 ip-10-0-136-25 kubenswrapper[2569]: I0706 13:51:30.954955 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-cg8ph" Jul 06 13:51:30.960675 ip-10-0-136-25 kubenswrapper[2569]: I0706 13:51:30.960654 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-gbq5d" Jul 06 13:51:31.091573 ip-10-0-136-25 kubenswrapper[2569]: I0706 13:51:31.091545 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-cg8ph"] Jul 06 13:51:31.094995 ip-10-0-136-25 kubenswrapper[2569]: W0706 13:51:31.094966 2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod576fd8e3_f3a0_48d2_a0d6_faa782a25a63.slice/crio-340760bac8f83cca05e29f720a3f234a2f0946e0e95284720f8c787654143eb1 WatchSource:0}: Error finding container 340760bac8f83cca05e29f720a3f234a2f0946e0e95284720f8c787654143eb1: Status 404 returned error can't find the container with id 340760bac8f83cca05e29f720a3f234a2f0946e0e95284720f8c787654143eb1 Jul 06 13:51:31.107364 ip-10-0-136-25 kubenswrapper[2569]: I0706 13:51:31.107335 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-gbq5d"] Jul 06 13:51:31.110652 ip-10-0-136-25 kubenswrapper[2569]: W0706 13:51:31.110626 2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd2d86b02_1e00_4267_b9c4_dd365b15342e.slice/crio-e665500ce4966fdfdaf47c9d460acc030b3c9aebe6e9e89d8af6433405060001 WatchSource:0}: Error finding container e665500ce4966fdfdaf47c9d460acc030b3c9aebe6e9e89d8af6433405060001: Status 404 returned error can't find the container with id e665500ce4966fdfdaf47c9d460acc030b3c9aebe6e9e89d8af6433405060001 Jul 06 13:51:31.185219 ip-10-0-136-25 kubenswrapper[2569]: I0706 13:51:31.185174 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-gbq5d" event={"ID":"d2d86b02-1e00-4267-b9c4-dd365b15342e","Type":"ContainerStarted","Data":"e665500ce4966fdfdaf47c9d460acc030b3c9aebe6e9e89d8af6433405060001"} Jul 06 13:51:31.186271 ip-10-0-136-25 kubenswrapper[2569]: I0706 13:51:31.186245 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-cg8ph" event={"ID":"576fd8e3-f3a0-48d2-a0d6-faa782a25a63","Type":"ContainerStarted","Data":"340760bac8f83cca05e29f720a3f234a2f0946e0e95284720f8c787654143eb1"} Jul 06 13:51:32.096727 ip-10-0-136-25 kubenswrapper[2569]: I0706 13:51:32.096683 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/420fd5c7-d449-4869-960a-4b14a873c402-metrics-tls\") pod \"dns-default-kbv9m\" (UID: \"420fd5c7-d449-4869-960a-4b14a873c402\") " pod="openshift-dns/dns-default-kbv9m" Jul 06 13:51:32.097193 ip-10-0-136-25 kubenswrapper[2569]: E0706 13:51:32.096892 2569 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jul 06 13:51:32.097193 ip-10-0-136-25 kubenswrapper[2569]: E0706 13:51:32.096983 2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/420fd5c7-d449-4869-960a-4b14a873c402-metrics-tls podName:420fd5c7-d449-4869-960a-4b14a873c402 nodeName:}" failed. No retries permitted until 2026-07-06 13:52:04.096959901 +0000 UTC m=+99.687521657 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/420fd5c7-d449-4869-960a-4b14a873c402-metrics-tls") pod "dns-default-kbv9m" (UID: "420fd5c7-d449-4869-960a-4b14a873c402") : secret "dns-default-metrics-tls" not found Jul 06 13:51:36.202478 ip-10-0-136-25 kubenswrapper[2569]: I0706 13:51:36.202390 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-gbq5d" event={"ID":"d2d86b02-1e00-4267-b9c4-dd365b15342e","Type":"ContainerStarted","Data":"de8ff8ad935b6db4b21293b077f80fbb06fb7b75f4140aa1fc50e840c7d7a709"} Jul 06 13:51:36.202887 ip-10-0-136-25 kubenswrapper[2569]: I0706 13:51:36.202527 2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-gbq5d" Jul 06 13:51:36.203907 ip-10-0-136-25 kubenswrapper[2569]: I0706 13:51:36.203887 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-cg8ph" event={"ID":"576fd8e3-f3a0-48d2-a0d6-faa782a25a63","Type":"ContainerStarted","Data":"f687c51a3f5203658d2db34ff796ad23b700c1d9326aab2a8db71ff8ea5fc3af"} Jul 06 13:51:36.239427 ip-10-0-136-25 kubenswrapper[2569]: I0706 13:51:36.239375 2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/global-pull-secret-syncer-cg8ph" podStartSLOduration=66.505809561 podStartE2EDuration="1m11.239359989s" totalImagesPullingTime="4.733550428s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-06 13:50:25 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 13:51:36.239077169 +0000 UTC m=+71.829638947" watchObservedRunningTime="2026-07-06 13:51:36.239359989 +0000 UTC m=+71.829921744" Jul 06 13:51:36.239605 ip-10-0-136-25 kubenswrapper[2569]: I0706 13:51:36.239510 2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-target-gbq5d" podStartSLOduration=66.532987604 podStartE2EDuration="1m11.239505032s" totalImagesPullingTime="4.706517428s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-06 13:50:25 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 13:51:36.221340697 +0000 UTC m=+71.811902474" watchObservedRunningTime="2026-07-06 13:51:36.239505032 +0000 UTC m=+71.830066830" Jul 06 13:52:04.110024 ip-10-0-136-25 kubenswrapper[2569]: I0706 13:52:04.109988 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/420fd5c7-d449-4869-960a-4b14a873c402-metrics-tls\") pod \"dns-default-kbv9m\" (UID: \"420fd5c7-d449-4869-960a-4b14a873c402\") " pod="openshift-dns/dns-default-kbv9m" Jul 06 13:52:04.110490 ip-10-0-136-25 kubenswrapper[2569]: E0706 13:52:04.110096 2569 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jul 06 13:52:04.110490 ip-10-0-136-25 kubenswrapper[2569]: E0706 13:52:04.110158 2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/420fd5c7-d449-4869-960a-4b14a873c402-metrics-tls podName:420fd5c7-d449-4869-960a-4b14a873c402 nodeName:}" failed. No retries permitted until 2026-07-06 13:53:08.110143794 +0000 UTC m=+163.700705548 (durationBeforeRetry 1m4s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/420fd5c7-d449-4869-960a-4b14a873c402-metrics-tls") pod "dns-default-kbv9m" (UID: "420fd5c7-d449-4869-960a-4b14a873c402") : secret "dns-default-metrics-tls" not found Jul 06 13:52:07.208475 ip-10-0-136-25 kubenswrapper[2569]: I0706 13:52:07.208442 2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-network-diagnostics/network-check-target-gbq5d" Jul 06 13:52:08.560072 ip-10-0-136-25 kubenswrapper[2569]: I0706 13:52:08.560037 2569 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-846bf668f7-52nxm"] Jul 06 13:52:08.588515 ip-10-0-136-25 kubenswrapper[2569]: I0706 13:52:08.588483 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-846bf668f7-52nxm"] Jul 06 13:52:08.588671 ip-10-0-136-25 kubenswrapper[2569]: I0706 13:52:08.588585 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-846bf668f7-52nxm" Jul 06 13:52:08.590932 ip-10-0-136-25 kubenswrapper[2569]: I0706 13:52:08.590908 2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"openshift-service-ca.crt\"" Jul 06 13:52:08.591075 ip-10-0-136-25 kubenswrapper[2569]: I0706 13:52:08.590934 2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"config\"" Jul 06 13:52:08.591075 ip-10-0-136-25 kubenswrapper[2569]: I0706 13:52:08.590941 2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"serving-cert\"" Jul 06 13:52:08.591547 ip-10-0-136-25 kubenswrapper[2569]: I0706 13:52:08.591525 2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"kube-storage-version-migrator-operator-dockercfg-7vvjp\"" Jul 06 13:52:08.591911 ip-10-0-136-25 kubenswrapper[2569]: I0706 13:52:08.591885 2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"kube-root-ca.crt\"" Jul 06 13:52:08.637835 ip-10-0-136-25 kubenswrapper[2569]: I0706 13:52:08.637792 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/2e5b9096-4169-413a-b200-49f75eb977c0-config\") pod \"kube-storage-version-migrator-operator-846bf668f7-52nxm\" (UID: \"2e5b9096-4169-413a-b200-49f75eb977c0\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-846bf668f7-52nxm" Jul 06 13:52:08.638000 ip-10-0-136-25 kubenswrapper[2569]: I0706 13:52:08.637855 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-h69zx\" (UniqueName: \"kubernetes.io/projected/2e5b9096-4169-413a-b200-49f75eb977c0-kube-api-access-h69zx\") pod \"kube-storage-version-migrator-operator-846bf668f7-52nxm\" (UID: \"2e5b9096-4169-413a-b200-49f75eb977c0\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-846bf668f7-52nxm" Jul 06 13:52:08.638000 ip-10-0-136-25 kubenswrapper[2569]: I0706 13:52:08.637958 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/2e5b9096-4169-413a-b200-49f75eb977c0-serving-cert\") pod \"kube-storage-version-migrator-operator-846bf668f7-52nxm\" (UID: \"2e5b9096-4169-413a-b200-49f75eb977c0\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-846bf668f7-52nxm" Jul 06 13:52:08.657037 ip-10-0-136-25 kubenswrapper[2569]: I0706 13:52:08.656993 2569 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-operator-5c847dcfc4-tdvhm"] Jul 06 13:52:08.672089 ip-10-0-136-25 kubenswrapper[2569]: I0706 13:52:08.672057 2569 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console-operator/console-operator-68b757fb67-z2h95"] Jul 06 13:52:08.672264 ip-10-0-136-25 kubenswrapper[2569]: I0706 13:52:08.672191 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-operator-5c847dcfc4-tdvhm" Jul 06 13:52:08.674231 ip-10-0-136-25 kubenswrapper[2569]: I0706 13:52:08.674188 2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"service-ca-bundle\"" Jul 06 13:52:08.674365 ip-10-0-136-25 kubenswrapper[2569]: I0706 13:52:08.674327 2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"operator-dockercfg-vnkrd\"" Jul 06 13:52:08.674365 ip-10-0-136-25 kubenswrapper[2569]: I0706 13:52:08.674353 2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\"" Jul 06 13:52:08.674626 ip-10-0-136-25 kubenswrapper[2569]: I0706 13:52:08.674606 2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\"" Jul 06 13:52:08.674700 ip-10-0-136-25 kubenswrapper[2569]: I0706 13:52:08.674671 2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"openshift-insights-serving-cert\"" Jul 06 13:52:08.679638 ip-10-0-136-25 kubenswrapper[2569]: I0706 13:52:08.679605 2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"trusted-ca-bundle\"" Jul 06 13:52:08.686193 ip-10-0-136-25 kubenswrapper[2569]: I0706 13:52:08.686170 2569 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/router-default-966c76789-kpw4q"] Jul 06 13:52:08.686335 ip-10-0-136-25 kubenswrapper[2569]: I0706 13:52:08.686294 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console-operator/console-operator-68b757fb67-z2h95" Jul 06 13:52:08.688092 ip-10-0-136-25 kubenswrapper[2569]: I0706 13:52:08.688069 2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console-operator\"/\"serving-cert\"" Jul 06 13:52:08.688351 ip-10-0-136-25 kubenswrapper[2569]: I0706 13:52:08.688328 2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"console-operator-config\"" Jul 06 13:52:08.688438 ip-10-0-136-25 kubenswrapper[2569]: I0706 13:52:08.688363 2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"kube-root-ca.crt\"" Jul 06 13:52:08.688438 ip-10-0-136-25 kubenswrapper[2569]: I0706 13:52:08.688332 2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"openshift-service-ca.crt\"" Jul 06 13:52:08.688438 ip-10-0-136-25 kubenswrapper[2569]: I0706 13:52:08.688337 2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console-operator\"/\"console-operator-dockercfg-5n7dw\"" Jul 06 13:52:08.693455 ip-10-0-136-25 kubenswrapper[2569]: I0706 13:52:08.693405 2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"trusted-ca\"" Jul 06 13:52:08.708340 ip-10-0-136-25 kubenswrapper[2569]: I0706 13:52:08.708317 2569 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/cluster-monitoring-operator-64786d9cbb-zpkk5"] Jul 06 13:52:08.708450 ip-10-0-136-25 kubenswrapper[2569]: I0706 13:52:08.708435 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/router-default-966c76789-kpw4q" Jul 06 13:52:08.710681 ip-10-0-136-25 kubenswrapper[2569]: I0706 13:52:08.710662 2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-dockercfg-brpsb\"" Jul 06 13:52:08.710787 ip-10-0-136-25 kubenswrapper[2569]: I0706 13:52:08.710694 2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"kube-root-ca.crt\"" Jul 06 13:52:08.710787 ip-10-0-136-25 kubenswrapper[2569]: I0706 13:52:08.710713 2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"default-ingress-cert\"" Jul 06 13:52:08.710946 ip-10-0-136-25 kubenswrapper[2569]: I0706 13:52:08.710923 2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"service-ca-bundle\"" Jul 06 13:52:08.710995 ip-10-0-136-25 kubenswrapper[2569]: I0706 13:52:08.710951 2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-metrics-certs-default\"" Jul 06 13:52:08.710995 ip-10-0-136-25 kubenswrapper[2569]: I0706 13:52:08.710986 2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-service-ca.crt\"" Jul 06 13:52:08.711071 ip-10-0-136-25 kubenswrapper[2569]: I0706 13:52:08.710958 2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-stats-default\"" Jul 06 13:52:08.732253 ip-10-0-136-25 kubenswrapper[2569]: I0706 13:52:08.732230 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-operator-5c847dcfc4-tdvhm"] Jul 06 13:52:08.732354 ip-10-0-136-25 kubenswrapper[2569]: I0706 13:52:08.732259 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/cluster-monitoring-operator-64786d9cbb-zpkk5"] Jul 06 13:52:08.732354 ip-10-0-136-25 kubenswrapper[2569]: I0706 13:52:08.732279 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/router-default-966c76789-kpw4q"] Jul 06 13:52:08.732354 ip-10-0-136-25 kubenswrapper[2569]: I0706 13:52:08.732293 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console-operator/console-operator-68b757fb67-z2h95"] Jul 06 13:52:08.732354 ip-10-0-136-25 kubenswrapper[2569]: I0706 13:52:08.732325 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/cluster-monitoring-operator-64786d9cbb-zpkk5" Jul 06 13:52:08.734467 ip-10-0-136-25 kubenswrapper[2569]: I0706 13:52:08.734448 2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\"" Jul 06 13:52:08.734580 ip-10-0-136-25 kubenswrapper[2569]: I0706 13:52:08.734450 2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"cluster-monitoring-operator-dockercfg-rtm87\"" Jul 06 13:52:08.734580 ip-10-0-136-25 kubenswrapper[2569]: I0706 13:52:08.734535 2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"telemetry-config\"" Jul 06 13:52:08.735020 ip-10-0-136-25 kubenswrapper[2569]: I0706 13:52:08.735005 2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\"" Jul 06 13:52:08.735066 ip-10-0-136-25 kubenswrapper[2569]: I0706 13:52:08.735034 2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"cluster-monitoring-operator-tls\"" Jul 06 13:52:08.738303 ip-10-0-136-25 kubenswrapper[2569]: I0706 13:52:08.738284 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/9746832c-8d02-489c-98f4-afb86bdc46b8-trusted-ca-bundle\") pod \"insights-operator-5c847dcfc4-tdvhm\" (UID: \"9746832c-8d02-489c-98f4-afb86bdc46b8\") " pod="openshift-insights/insights-operator-5c847dcfc4-tdvhm" Jul 06 13:52:08.738384 ip-10-0-136-25 kubenswrapper[2569]: I0706 13:52:08.738312 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/dfffcdca-94d6-4732-815f-bcb533f538e9-config\") pod \"console-operator-68b757fb67-z2h95\" (UID: \"dfffcdca-94d6-4732-815f-bcb533f538e9\") " pod="openshift-console-operator/console-operator-68b757fb67-z2h95" Jul 06 13:52:08.738384 ip-10-0-136-25 kubenswrapper[2569]: I0706 13:52:08.738328 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nmbt6\" (UniqueName: \"kubernetes.io/projected/1cb95852-b3c5-4b1a-80bf-9d3fa465084a-kube-api-access-nmbt6\") pod \"router-default-966c76789-kpw4q\" (UID: \"1cb95852-b3c5-4b1a-80bf-9d3fa465084a\") " pod="openshift-ingress/router-default-966c76789-kpw4q" Jul 06 13:52:08.738384 ip-10-0-136-25 kubenswrapper[2569]: I0706 13:52:08.738360 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/2e5b9096-4169-413a-b200-49f75eb977c0-serving-cert\") pod \"kube-storage-version-migrator-operator-846bf668f7-52nxm\" (UID: \"2e5b9096-4169-413a-b200-49f75eb977c0\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-846bf668f7-52nxm" Jul 06 13:52:08.738529 ip-10-0-136-25 kubenswrapper[2569]: I0706 13:52:08.738389 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/9746832c-8d02-489c-98f4-afb86bdc46b8-serving-cert\") pod \"insights-operator-5c847dcfc4-tdvhm\" (UID: \"9746832c-8d02-489c-98f4-afb86bdc46b8\") " pod="openshift-insights/insights-operator-5c847dcfc4-tdvhm" Jul 06 13:52:08.738529 ip-10-0-136-25 kubenswrapper[2569]: I0706 13:52:08.738412 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cgww9\" (UniqueName: \"kubernetes.io/projected/9746832c-8d02-489c-98f4-afb86bdc46b8-kube-api-access-cgww9\") pod \"insights-operator-5c847dcfc4-tdvhm\" (UID: \"9746832c-8d02-489c-98f4-afb86bdc46b8\") " pod="openshift-insights/insights-operator-5c847dcfc4-tdvhm" Jul 06 13:52:08.738529 ip-10-0-136-25 kubenswrapper[2569]: I0706 13:52:08.738437 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/dfffcdca-94d6-4732-815f-bcb533f538e9-trusted-ca\") pod \"console-operator-68b757fb67-z2h95\" (UID: \"dfffcdca-94d6-4732-815f-bcb533f538e9\") " pod="openshift-console-operator/console-operator-68b757fb67-z2h95" Jul 06 13:52:08.738529 ip-10-0-136-25 kubenswrapper[2569]: I0706 13:52:08.738460 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rhpj9\" (UniqueName: \"kubernetes.io/projected/dfffcdca-94d6-4732-815f-bcb533f538e9-kube-api-access-rhpj9\") pod \"console-operator-68b757fb67-z2h95\" (UID: \"dfffcdca-94d6-4732-815f-bcb533f538e9\") " pod="openshift-console-operator/console-operator-68b757fb67-z2h95" Jul 06 13:52:08.738529 ip-10-0-136-25 kubenswrapper[2569]: I0706 13:52:08.738483 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1cb95852-b3c5-4b1a-80bf-9d3fa465084a-metrics-certs\") pod \"router-default-966c76789-kpw4q\" (UID: \"1cb95852-b3c5-4b1a-80bf-9d3fa465084a\") " pod="openshift-ingress/router-default-966c76789-kpw4q" Jul 06 13:52:08.738846 ip-10-0-136-25 kubenswrapper[2569]: I0706 13:52:08.738529 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/2e5b9096-4169-413a-b200-49f75eb977c0-config\") pod \"kube-storage-version-migrator-operator-846bf668f7-52nxm\" (UID: \"2e5b9096-4169-413a-b200-49f75eb977c0\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-846bf668f7-52nxm" Jul 06 13:52:08.738846 ip-10-0-136-25 kubenswrapper[2569]: I0706 13:52:08.738558 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h69zx\" (UniqueName: \"kubernetes.io/projected/2e5b9096-4169-413a-b200-49f75eb977c0-kube-api-access-h69zx\") pod \"kube-storage-version-migrator-operator-846bf668f7-52nxm\" (UID: \"2e5b9096-4169-413a-b200-49f75eb977c0\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-846bf668f7-52nxm" Jul 06 13:52:08.738846 ip-10-0-136-25 kubenswrapper[2569]: I0706 13:52:08.738584 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/1cb95852-b3c5-4b1a-80bf-9d3fa465084a-service-ca-bundle\") pod \"router-default-966c76789-kpw4q\" (UID: \"1cb95852-b3c5-4b1a-80bf-9d3fa465084a\") " pod="openshift-ingress/router-default-966c76789-kpw4q" Jul 06 13:52:08.738846 ip-10-0-136-25 kubenswrapper[2569]: I0706 13:52:08.738630 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/9746832c-8d02-489c-98f4-afb86bdc46b8-tmp\") pod \"insights-operator-5c847dcfc4-tdvhm\" (UID: \"9746832c-8d02-489c-98f4-afb86bdc46b8\") " pod="openshift-insights/insights-operator-5c847dcfc4-tdvhm" Jul 06 13:52:08.738846 ip-10-0-136-25 kubenswrapper[2569]: I0706 13:52:08.738654 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/9746832c-8d02-489c-98f4-afb86bdc46b8-service-ca-bundle\") pod \"insights-operator-5c847dcfc4-tdvhm\" (UID: \"9746832c-8d02-489c-98f4-afb86bdc46b8\") " pod="openshift-insights/insights-operator-5c847dcfc4-tdvhm" Jul 06 13:52:08.738846 ip-10-0-136-25 kubenswrapper[2569]: I0706 13:52:08.738683 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"stats-auth\" (UniqueName: \"kubernetes.io/secret/1cb95852-b3c5-4b1a-80bf-9d3fa465084a-stats-auth\") pod \"router-default-966c76789-kpw4q\" (UID: \"1cb95852-b3c5-4b1a-80bf-9d3fa465084a\") " pod="openshift-ingress/router-default-966c76789-kpw4q" Jul 06 13:52:08.738846 ip-10-0-136-25 kubenswrapper[2569]: I0706 13:52:08.738712 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"snapshots\" (UniqueName: \"kubernetes.io/empty-dir/9746832c-8d02-489c-98f4-afb86bdc46b8-snapshots\") pod \"insights-operator-5c847dcfc4-tdvhm\" (UID: \"9746832c-8d02-489c-98f4-afb86bdc46b8\") " pod="openshift-insights/insights-operator-5c847dcfc4-tdvhm" Jul 06 13:52:08.738846 ip-10-0-136-25 kubenswrapper[2569]: I0706 13:52:08.738736 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/dfffcdca-94d6-4732-815f-bcb533f538e9-serving-cert\") pod \"console-operator-68b757fb67-z2h95\" (UID: \"dfffcdca-94d6-4732-815f-bcb533f538e9\") " pod="openshift-console-operator/console-operator-68b757fb67-z2h95" Jul 06 13:52:08.738846 ip-10-0-136-25 kubenswrapper[2569]: I0706 13:52:08.738778 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"default-certificate\" (UniqueName: \"kubernetes.io/secret/1cb95852-b3c5-4b1a-80bf-9d3fa465084a-default-certificate\") pod \"router-default-966c76789-kpw4q\" (UID: \"1cb95852-b3c5-4b1a-80bf-9d3fa465084a\") " pod="openshift-ingress/router-default-966c76789-kpw4q" Jul 06 13:52:08.739457 ip-10-0-136-25 kubenswrapper[2569]: I0706 13:52:08.739440 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/2e5b9096-4169-413a-b200-49f75eb977c0-config\") pod \"kube-storage-version-migrator-operator-846bf668f7-52nxm\" (UID: \"2e5b9096-4169-413a-b200-49f75eb977c0\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-846bf668f7-52nxm" Jul 06 13:52:08.741165 ip-10-0-136-25 kubenswrapper[2569]: I0706 13:52:08.741148 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/2e5b9096-4169-413a-b200-49f75eb977c0-serving-cert\") pod \"kube-storage-version-migrator-operator-846bf668f7-52nxm\" (UID: \"2e5b9096-4169-413a-b200-49f75eb977c0\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-846bf668f7-52nxm" Jul 06 13:52:08.746569 ip-10-0-136-25 kubenswrapper[2569]: I0706 13:52:08.746547 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-h69zx\" (UniqueName: \"kubernetes.io/projected/2e5b9096-4169-413a-b200-49f75eb977c0-kube-api-access-h69zx\") pod \"kube-storage-version-migrator-operator-846bf668f7-52nxm\" (UID: \"2e5b9096-4169-413a-b200-49f75eb977c0\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-846bf668f7-52nxm" Jul 06 13:52:08.839842 ip-10-0-136-25 kubenswrapper[2569]: I0706 13:52:08.839749 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/a52c7279-3809-4f22-9001-afb7325cfe8d-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-64786d9cbb-zpkk5\" (UID: \"a52c7279-3809-4f22-9001-afb7325cfe8d\") " pod="openshift-monitoring/cluster-monitoring-operator-64786d9cbb-zpkk5" Jul 06 13:52:08.839842 ip-10-0-136-25 kubenswrapper[2569]: I0706 13:52:08.839798 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/1cb95852-b3c5-4b1a-80bf-9d3fa465084a-service-ca-bundle\") pod \"router-default-966c76789-kpw4q\" (UID: \"1cb95852-b3c5-4b1a-80bf-9d3fa465084a\") " pod="openshift-ingress/router-default-966c76789-kpw4q" Jul 06 13:52:08.839842 ip-10-0-136-25 kubenswrapper[2569]: I0706 13:52:08.839819 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zbh5p\" (UniqueName: \"kubernetes.io/projected/a52c7279-3809-4f22-9001-afb7325cfe8d-kube-api-access-zbh5p\") pod \"cluster-monitoring-operator-64786d9cbb-zpkk5\" (UID: \"a52c7279-3809-4f22-9001-afb7325cfe8d\") " pod="openshift-monitoring/cluster-monitoring-operator-64786d9cbb-zpkk5" Jul 06 13:52:08.840116 ip-10-0-136-25 kubenswrapper[2569]: I0706 13:52:08.839851 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/9746832c-8d02-489c-98f4-afb86bdc46b8-service-ca-bundle\") pod \"insights-operator-5c847dcfc4-tdvhm\" (UID: \"9746832c-8d02-489c-98f4-afb86bdc46b8\") " pod="openshift-insights/insights-operator-5c847dcfc4-tdvhm" Jul 06 13:52:08.840116 ip-10-0-136-25 kubenswrapper[2569]: I0706 13:52:08.839876 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"snapshots\" (UniqueName: \"kubernetes.io/empty-dir/9746832c-8d02-489c-98f4-afb86bdc46b8-snapshots\") pod \"insights-operator-5c847dcfc4-tdvhm\" (UID: \"9746832c-8d02-489c-98f4-afb86bdc46b8\") " pod="openshift-insights/insights-operator-5c847dcfc4-tdvhm" Jul 06 13:52:08.840116 ip-10-0-136-25 kubenswrapper[2569]: I0706 13:52:08.839896 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"default-certificate\" (UniqueName: \"kubernetes.io/secret/1cb95852-b3c5-4b1a-80bf-9d3fa465084a-default-certificate\") pod \"router-default-966c76789-kpw4q\" (UID: \"1cb95852-b3c5-4b1a-80bf-9d3fa465084a\") " pod="openshift-ingress/router-default-966c76789-kpw4q" Jul 06 13:52:08.840116 ip-10-0-136-25 kubenswrapper[2569]: I0706 13:52:08.839928 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cgww9\" (UniqueName: \"kubernetes.io/projected/9746832c-8d02-489c-98f4-afb86bdc46b8-kube-api-access-cgww9\") pod \"insights-operator-5c847dcfc4-tdvhm\" (UID: \"9746832c-8d02-489c-98f4-afb86bdc46b8\") " pod="openshift-insights/insights-operator-5c847dcfc4-tdvhm" Jul 06 13:52:08.840116 ip-10-0-136-25 kubenswrapper[2569]: E0706 13:52:08.839990 2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/1cb95852-b3c5-4b1a-80bf-9d3fa465084a-service-ca-bundle podName:1cb95852-b3c5-4b1a-80bf-9d3fa465084a nodeName:}" failed. No retries permitted until 2026-07-06 13:52:09.339963001 +0000 UTC m=+104.930524760 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/1cb95852-b3c5-4b1a-80bf-9d3fa465084a-service-ca-bundle") pod "router-default-966c76789-kpw4q" (UID: "1cb95852-b3c5-4b1a-80bf-9d3fa465084a") : configmap references non-existent config key: service-ca.crt Jul 06 13:52:08.840116 ip-10-0-136-25 kubenswrapper[2569]: I0706 13:52:08.840048 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1cb95852-b3c5-4b1a-80bf-9d3fa465084a-metrics-certs\") pod \"router-default-966c76789-kpw4q\" (UID: \"1cb95852-b3c5-4b1a-80bf-9d3fa465084a\") " pod="openshift-ingress/router-default-966c76789-kpw4q" Jul 06 13:52:08.840116 ip-10-0-136-25 kubenswrapper[2569]: I0706 13:52:08.840081 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rhpj9\" (UniqueName: \"kubernetes.io/projected/dfffcdca-94d6-4732-815f-bcb533f538e9-kube-api-access-rhpj9\") pod \"console-operator-68b757fb67-z2h95\" (UID: \"dfffcdca-94d6-4732-815f-bcb533f538e9\") " pod="openshift-console-operator/console-operator-68b757fb67-z2h95" Jul 06 13:52:08.840476 ip-10-0-136-25 kubenswrapper[2569]: I0706 13:52:08.840132 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/9746832c-8d02-489c-98f4-afb86bdc46b8-tmp\") pod \"insights-operator-5c847dcfc4-tdvhm\" (UID: \"9746832c-8d02-489c-98f4-afb86bdc46b8\") " pod="openshift-insights/insights-operator-5c847dcfc4-tdvhm" Jul 06 13:52:08.840476 ip-10-0-136-25 kubenswrapper[2569]: E0706 13:52:08.840150 2569 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found Jul 06 13:52:08.840476 ip-10-0-136-25 kubenswrapper[2569]: I0706 13:52:08.840158 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"stats-auth\" (UniqueName: \"kubernetes.io/secret/1cb95852-b3c5-4b1a-80bf-9d3fa465084a-stats-auth\") pod \"router-default-966c76789-kpw4q\" (UID: \"1cb95852-b3c5-4b1a-80bf-9d3fa465084a\") " pod="openshift-ingress/router-default-966c76789-kpw4q" Jul 06 13:52:08.840476 ip-10-0-136-25 kubenswrapper[2569]: E0706 13:52:08.840206 2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1cb95852-b3c5-4b1a-80bf-9d3fa465084a-metrics-certs podName:1cb95852-b3c5-4b1a-80bf-9d3fa465084a nodeName:}" failed. No retries permitted until 2026-07-06 13:52:09.340190131 +0000 UTC m=+104.930751888 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/1cb95852-b3c5-4b1a-80bf-9d3fa465084a-metrics-certs") pod "router-default-966c76789-kpw4q" (UID: "1cb95852-b3c5-4b1a-80bf-9d3fa465084a") : secret "router-metrics-certs-default" not found Jul 06 13:52:08.840476 ip-10-0-136-25 kubenswrapper[2569]: I0706 13:52:08.840246 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/9746832c-8d02-489c-98f4-afb86bdc46b8-trusted-ca-bundle\") pod \"insights-operator-5c847dcfc4-tdvhm\" (UID: \"9746832c-8d02-489c-98f4-afb86bdc46b8\") " pod="openshift-insights/insights-operator-5c847dcfc4-tdvhm" Jul 06 13:52:08.840476 ip-10-0-136-25 kubenswrapper[2569]: I0706 13:52:08.840274 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nmbt6\" (UniqueName: \"kubernetes.io/projected/1cb95852-b3c5-4b1a-80bf-9d3fa465084a-kube-api-access-nmbt6\") pod \"router-default-966c76789-kpw4q\" (UID: \"1cb95852-b3c5-4b1a-80bf-9d3fa465084a\") " pod="openshift-ingress/router-default-966c76789-kpw4q" Jul 06 13:52:08.841008 ip-10-0-136-25 kubenswrapper[2569]: I0706 13:52:08.840975 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"snapshots\" (UniqueName: \"kubernetes.io/empty-dir/9746832c-8d02-489c-98f4-afb86bdc46b8-snapshots\") pod \"insights-operator-5c847dcfc4-tdvhm\" (UID: \"9746832c-8d02-489c-98f4-afb86bdc46b8\") " pod="openshift-insights/insights-operator-5c847dcfc4-tdvhm" Jul 06 13:52:08.841131 ip-10-0-136-25 kubenswrapper[2569]: I0706 13:52:08.841074 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/9746832c-8d02-489c-98f4-afb86bdc46b8-tmp\") pod \"insights-operator-5c847dcfc4-tdvhm\" (UID: \"9746832c-8d02-489c-98f4-afb86bdc46b8\") " pod="openshift-insights/insights-operator-5c847dcfc4-tdvhm" Jul 06 13:52:08.841131 ip-10-0-136-25 kubenswrapper[2569]: I0706 13:52:08.841097 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/9746832c-8d02-489c-98f4-afb86bdc46b8-service-ca-bundle\") pod \"insights-operator-5c847dcfc4-tdvhm\" (UID: \"9746832c-8d02-489c-98f4-afb86bdc46b8\") " pod="openshift-insights/insights-operator-5c847dcfc4-tdvhm" Jul 06 13:52:08.841264 ip-10-0-136-25 kubenswrapper[2569]: I0706 13:52:08.841169 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/dfffcdca-94d6-4732-815f-bcb533f538e9-config\") pod \"console-operator-68b757fb67-z2h95\" (UID: \"dfffcdca-94d6-4732-815f-bcb533f538e9\") " pod="openshift-console-operator/console-operator-68b757fb67-z2h95" Jul 06 13:52:08.841498 ip-10-0-136-25 kubenswrapper[2569]: I0706 13:52:08.841235 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"telemetry-config\" (UniqueName: \"kubernetes.io/configmap/a52c7279-3809-4f22-9001-afb7325cfe8d-telemetry-config\") pod \"cluster-monitoring-operator-64786d9cbb-zpkk5\" (UID: \"a52c7279-3809-4f22-9001-afb7325cfe8d\") " pod="openshift-monitoring/cluster-monitoring-operator-64786d9cbb-zpkk5" Jul 06 13:52:08.841592 ip-10-0-136-25 kubenswrapper[2569]: I0706 13:52:08.841534 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/dfffcdca-94d6-4732-815f-bcb533f538e9-trusted-ca\") pod \"console-operator-68b757fb67-z2h95\" (UID: \"dfffcdca-94d6-4732-815f-bcb533f538e9\") " pod="openshift-console-operator/console-operator-68b757fb67-z2h95" Jul 06 13:52:08.841592 ip-10-0-136-25 kubenswrapper[2569]: I0706 13:52:08.841564 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/9746832c-8d02-489c-98f4-afb86bdc46b8-serving-cert\") pod \"insights-operator-5c847dcfc4-tdvhm\" (UID: \"9746832c-8d02-489c-98f4-afb86bdc46b8\") " pod="openshift-insights/insights-operator-5c847dcfc4-tdvhm" Jul 06 13:52:08.841695 ip-10-0-136-25 kubenswrapper[2569]: I0706 13:52:08.841659 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/9746832c-8d02-489c-98f4-afb86bdc46b8-trusted-ca-bundle\") pod \"insights-operator-5c847dcfc4-tdvhm\" (UID: \"9746832c-8d02-489c-98f4-afb86bdc46b8\") " pod="openshift-insights/insights-operator-5c847dcfc4-tdvhm" Jul 06 13:52:08.841695 ip-10-0-136-25 kubenswrapper[2569]: I0706 13:52:08.841679 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/dfffcdca-94d6-4732-815f-bcb533f538e9-serving-cert\") pod \"console-operator-68b757fb67-z2h95\" (UID: \"dfffcdca-94d6-4732-815f-bcb533f538e9\") " pod="openshift-console-operator/console-operator-68b757fb67-z2h95" Jul 06 13:52:08.842356 ip-10-0-136-25 kubenswrapper[2569]: I0706 13:52:08.842334 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/dfffcdca-94d6-4732-815f-bcb533f538e9-config\") pod \"console-operator-68b757fb67-z2h95\" (UID: \"dfffcdca-94d6-4732-815f-bcb533f538e9\") " pod="openshift-console-operator/console-operator-68b757fb67-z2h95" Jul 06 13:52:08.843819 ip-10-0-136-25 kubenswrapper[2569]: I0706 13:52:08.843791 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"default-certificate\" (UniqueName: \"kubernetes.io/secret/1cb95852-b3c5-4b1a-80bf-9d3fa465084a-default-certificate\") pod \"router-default-966c76789-kpw4q\" (UID: \"1cb95852-b3c5-4b1a-80bf-9d3fa465084a\") " pod="openshift-ingress/router-default-966c76789-kpw4q" Jul 06 13:52:08.844344 ip-10-0-136-25 kubenswrapper[2569]: I0706 13:52:08.844316 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/9746832c-8d02-489c-98f4-afb86bdc46b8-serving-cert\") pod \"insights-operator-5c847dcfc4-tdvhm\" (UID: \"9746832c-8d02-489c-98f4-afb86bdc46b8\") " pod="openshift-insights/insights-operator-5c847dcfc4-tdvhm" Jul 06 13:52:08.844629 ip-10-0-136-25 kubenswrapper[2569]: I0706 13:52:08.844597 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"stats-auth\" (UniqueName: \"kubernetes.io/secret/1cb95852-b3c5-4b1a-80bf-9d3fa465084a-stats-auth\") pod \"router-default-966c76789-kpw4q\" (UID: \"1cb95852-b3c5-4b1a-80bf-9d3fa465084a\") " pod="openshift-ingress/router-default-966c76789-kpw4q" Jul 06 13:52:08.844785 ip-10-0-136-25 kubenswrapper[2569]: I0706 13:52:08.844762 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/dfffcdca-94d6-4732-815f-bcb533f538e9-serving-cert\") pod \"console-operator-68b757fb67-z2h95\" (UID: \"dfffcdca-94d6-4732-815f-bcb533f538e9\") " pod="openshift-console-operator/console-operator-68b757fb67-z2h95" Jul 06 13:52:08.847766 ip-10-0-136-25 kubenswrapper[2569]: I0706 13:52:08.847725 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/dfffcdca-94d6-4732-815f-bcb533f538e9-trusted-ca\") pod \"console-operator-68b757fb67-z2h95\" (UID: \"dfffcdca-94d6-4732-815f-bcb533f538e9\") " pod="openshift-console-operator/console-operator-68b757fb67-z2h95" Jul 06 13:52:08.851633 ip-10-0-136-25 kubenswrapper[2569]: I0706 13:52:08.851605 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rhpj9\" (UniqueName: \"kubernetes.io/projected/dfffcdca-94d6-4732-815f-bcb533f538e9-kube-api-access-rhpj9\") pod \"console-operator-68b757fb67-z2h95\" (UID: \"dfffcdca-94d6-4732-815f-bcb533f538e9\") " pod="openshift-console-operator/console-operator-68b757fb67-z2h95" Jul 06 13:52:08.851724 ip-10-0-136-25 kubenswrapper[2569]: I0706 13:52:08.851708 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nmbt6\" (UniqueName: \"kubernetes.io/projected/1cb95852-b3c5-4b1a-80bf-9d3fa465084a-kube-api-access-nmbt6\") pod \"router-default-966c76789-kpw4q\" (UID: \"1cb95852-b3c5-4b1a-80bf-9d3fa465084a\") " pod="openshift-ingress/router-default-966c76789-kpw4q" Jul 06 13:52:08.851771 ip-10-0-136-25 kubenswrapper[2569]: I0706 13:52:08.851721 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cgww9\" (UniqueName: \"kubernetes.io/projected/9746832c-8d02-489c-98f4-afb86bdc46b8-kube-api-access-cgww9\") pod \"insights-operator-5c847dcfc4-tdvhm\" (UID: \"9746832c-8d02-489c-98f4-afb86bdc46b8\") " pod="openshift-insights/insights-operator-5c847dcfc4-tdvhm" Jul 06 13:52:08.899439 ip-10-0-136-25 kubenswrapper[2569]: I0706 13:52:08.899404 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-846bf668f7-52nxm" Jul 06 13:52:08.942435 ip-10-0-136-25 kubenswrapper[2569]: I0706 13:52:08.942413 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zbh5p\" (UniqueName: \"kubernetes.io/projected/a52c7279-3809-4f22-9001-afb7325cfe8d-kube-api-access-zbh5p\") pod \"cluster-monitoring-operator-64786d9cbb-zpkk5\" (UID: \"a52c7279-3809-4f22-9001-afb7325cfe8d\") " pod="openshift-monitoring/cluster-monitoring-operator-64786d9cbb-zpkk5" Jul 06 13:52:08.942586 ip-10-0-136-25 kubenswrapper[2569]: I0706 13:52:08.942574 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"telemetry-config\" (UniqueName: \"kubernetes.io/configmap/a52c7279-3809-4f22-9001-afb7325cfe8d-telemetry-config\") pod \"cluster-monitoring-operator-64786d9cbb-zpkk5\" (UID: \"a52c7279-3809-4f22-9001-afb7325cfe8d\") " pod="openshift-monitoring/cluster-monitoring-operator-64786d9cbb-zpkk5" Jul 06 13:52:08.942627 ip-10-0-136-25 kubenswrapper[2569]: I0706 13:52:08.942603 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/a52c7279-3809-4f22-9001-afb7325cfe8d-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-64786d9cbb-zpkk5\" (UID: \"a52c7279-3809-4f22-9001-afb7325cfe8d\") " pod="openshift-monitoring/cluster-monitoring-operator-64786d9cbb-zpkk5" Jul 06 13:52:08.942693 ip-10-0-136-25 kubenswrapper[2569]: E0706 13:52:08.942681 2569 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found Jul 06 13:52:08.942742 ip-10-0-136-25 kubenswrapper[2569]: E0706 13:52:08.942733 2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a52c7279-3809-4f22-9001-afb7325cfe8d-cluster-monitoring-operator-tls podName:a52c7279-3809-4f22-9001-afb7325cfe8d nodeName:}" failed. No retries permitted until 2026-07-06 13:52:09.442719253 +0000 UTC m=+105.033281008 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/a52c7279-3809-4f22-9001-afb7325cfe8d-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-64786d9cbb-zpkk5" (UID: "a52c7279-3809-4f22-9001-afb7325cfe8d") : secret "cluster-monitoring-operator-tls" not found Jul 06 13:52:08.943278 ip-10-0-136-25 kubenswrapper[2569]: I0706 13:52:08.943260 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"telemetry-config\" (UniqueName: \"kubernetes.io/configmap/a52c7279-3809-4f22-9001-afb7325cfe8d-telemetry-config\") pod \"cluster-monitoring-operator-64786d9cbb-zpkk5\" (UID: \"a52c7279-3809-4f22-9001-afb7325cfe8d\") " pod="openshift-monitoring/cluster-monitoring-operator-64786d9cbb-zpkk5" Jul 06 13:52:08.950718 ip-10-0-136-25 kubenswrapper[2569]: I0706 13:52:08.950694 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zbh5p\" (UniqueName: \"kubernetes.io/projected/a52c7279-3809-4f22-9001-afb7325cfe8d-kube-api-access-zbh5p\") pod \"cluster-monitoring-operator-64786d9cbb-zpkk5\" (UID: \"a52c7279-3809-4f22-9001-afb7325cfe8d\") " pod="openshift-monitoring/cluster-monitoring-operator-64786d9cbb-zpkk5" Jul 06 13:52:08.983009 ip-10-0-136-25 kubenswrapper[2569]: I0706 13:52:08.982776 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-operator-5c847dcfc4-tdvhm" Jul 06 13:52:08.995350 ip-10-0-136-25 kubenswrapper[2569]: I0706 13:52:08.995187 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console-operator/console-operator-68b757fb67-z2h95" Jul 06 13:52:09.067875 ip-10-0-136-25 kubenswrapper[2569]: I0706 13:52:09.067804 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-846bf668f7-52nxm"] Jul 06 13:52:09.072667 ip-10-0-136-25 kubenswrapper[2569]: W0706 13:52:09.072635 2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2e5b9096_4169_413a_b200_49f75eb977c0.slice/crio-51a56c8896cece0be31733a977e05f2662aef06b9bd82c626e8e239d9ac94e18 WatchSource:0}: Error finding container 51a56c8896cece0be31733a977e05f2662aef06b9bd82c626e8e239d9ac94e18: Status 404 returned error can't find the container with id 51a56c8896cece0be31733a977e05f2662aef06b9bd82c626e8e239d9ac94e18 Jul 06 13:52:09.144479 ip-10-0-136-25 kubenswrapper[2569]: I0706 13:52:09.144448 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-operator-5c847dcfc4-tdvhm"] Jul 06 13:52:09.147929 ip-10-0-136-25 kubenswrapper[2569]: W0706 13:52:09.147900 2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9746832c_8d02_489c_98f4_afb86bdc46b8.slice/crio-2238b3b7d4c0818d4e5b476a4d23797bfa99bdd96d146c24f888c8db15785d74 WatchSource:0}: Error finding container 2238b3b7d4c0818d4e5b476a4d23797bfa99bdd96d146c24f888c8db15785d74: Status 404 returned error can't find the container with id 2238b3b7d4c0818d4e5b476a4d23797bfa99bdd96d146c24f888c8db15785d74 Jul 06 13:52:09.158327 ip-10-0-136-25 kubenswrapper[2569]: I0706 13:52:09.158305 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console-operator/console-operator-68b757fb67-z2h95"] Jul 06 13:52:09.161189 ip-10-0-136-25 kubenswrapper[2569]: W0706 13:52:09.161165 2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddfffcdca_94d6_4732_815f_bcb533f538e9.slice/crio-2643251f5f7ea566faa0d8684a461c2c51b3f15ac882e9f751691e74f550af6e WatchSource:0}: Error finding container 2643251f5f7ea566faa0d8684a461c2c51b3f15ac882e9f751691e74f550af6e: Status 404 returned error can't find the container with id 2643251f5f7ea566faa0d8684a461c2c51b3f15ac882e9f751691e74f550af6e Jul 06 13:52:09.289650 ip-10-0-136-25 kubenswrapper[2569]: I0706 13:52:09.289610 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-5c847dcfc4-tdvhm" event={"ID":"9746832c-8d02-489c-98f4-afb86bdc46b8","Type":"ContainerStarted","Data":"2238b3b7d4c0818d4e5b476a4d23797bfa99bdd96d146c24f888c8db15785d74"} Jul 06 13:52:09.290758 ip-10-0-136-25 kubenswrapper[2569]: I0706 13:52:09.290725 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-846bf668f7-52nxm" event={"ID":"2e5b9096-4169-413a-b200-49f75eb977c0","Type":"ContainerStarted","Data":"51a56c8896cece0be31733a977e05f2662aef06b9bd82c626e8e239d9ac94e18"} Jul 06 13:52:09.291627 ip-10-0-136-25 kubenswrapper[2569]: I0706 13:52:09.291602 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-68b757fb67-z2h95" event={"ID":"dfffcdca-94d6-4732-815f-bcb533f538e9","Type":"ContainerStarted","Data":"2643251f5f7ea566faa0d8684a461c2c51b3f15ac882e9f751691e74f550af6e"} Jul 06 13:52:09.345814 ip-10-0-136-25 kubenswrapper[2569]: I0706 13:52:09.345724 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/1cb95852-b3c5-4b1a-80bf-9d3fa465084a-service-ca-bundle\") pod \"router-default-966c76789-kpw4q\" (UID: \"1cb95852-b3c5-4b1a-80bf-9d3fa465084a\") " pod="openshift-ingress/router-default-966c76789-kpw4q" Jul 06 13:52:09.345814 ip-10-0-136-25 kubenswrapper[2569]: I0706 13:52:09.345777 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1cb95852-b3c5-4b1a-80bf-9d3fa465084a-metrics-certs\") pod \"router-default-966c76789-kpw4q\" (UID: \"1cb95852-b3c5-4b1a-80bf-9d3fa465084a\") " pod="openshift-ingress/router-default-966c76789-kpw4q" Jul 06 13:52:09.345994 ip-10-0-136-25 kubenswrapper[2569]: E0706 13:52:09.345877 2569 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found Jul 06 13:52:09.345994 ip-10-0-136-25 kubenswrapper[2569]: E0706 13:52:09.345903 2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/1cb95852-b3c5-4b1a-80bf-9d3fa465084a-service-ca-bundle podName:1cb95852-b3c5-4b1a-80bf-9d3fa465084a nodeName:}" failed. No retries permitted until 2026-07-06 13:52:10.345885825 +0000 UTC m=+105.936447581 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/1cb95852-b3c5-4b1a-80bf-9d3fa465084a-service-ca-bundle") pod "router-default-966c76789-kpw4q" (UID: "1cb95852-b3c5-4b1a-80bf-9d3fa465084a") : configmap references non-existent config key: service-ca.crt Jul 06 13:52:09.345994 ip-10-0-136-25 kubenswrapper[2569]: E0706 13:52:09.345926 2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1cb95852-b3c5-4b1a-80bf-9d3fa465084a-metrics-certs podName:1cb95852-b3c5-4b1a-80bf-9d3fa465084a nodeName:}" failed. No retries permitted until 2026-07-06 13:52:10.345919502 +0000 UTC m=+105.936481257 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/1cb95852-b3c5-4b1a-80bf-9d3fa465084a-metrics-certs") pod "router-default-966c76789-kpw4q" (UID: "1cb95852-b3c5-4b1a-80bf-9d3fa465084a") : secret "router-metrics-certs-default" not found Jul 06 13:52:09.446941 ip-10-0-136-25 kubenswrapper[2569]: I0706 13:52:09.446894 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/a52c7279-3809-4f22-9001-afb7325cfe8d-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-64786d9cbb-zpkk5\" (UID: \"a52c7279-3809-4f22-9001-afb7325cfe8d\") " pod="openshift-monitoring/cluster-monitoring-operator-64786d9cbb-zpkk5" Jul 06 13:52:09.447118 ip-10-0-136-25 kubenswrapper[2569]: E0706 13:52:09.447043 2569 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found Jul 06 13:52:09.447118 ip-10-0-136-25 kubenswrapper[2569]: E0706 13:52:09.447112 2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a52c7279-3809-4f22-9001-afb7325cfe8d-cluster-monitoring-operator-tls podName:a52c7279-3809-4f22-9001-afb7325cfe8d nodeName:}" failed. No retries permitted until 2026-07-06 13:52:10.447095287 +0000 UTC m=+106.037657041 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/a52c7279-3809-4f22-9001-afb7325cfe8d-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-64786d9cbb-zpkk5" (UID: "a52c7279-3809-4f22-9001-afb7325cfe8d") : secret "cluster-monitoring-operator-tls" not found Jul 06 13:52:10.355539 ip-10-0-136-25 kubenswrapper[2569]: I0706 13:52:10.355500 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/1cb95852-b3c5-4b1a-80bf-9d3fa465084a-service-ca-bundle\") pod \"router-default-966c76789-kpw4q\" (UID: \"1cb95852-b3c5-4b1a-80bf-9d3fa465084a\") " pod="openshift-ingress/router-default-966c76789-kpw4q" Jul 06 13:52:10.355962 ip-10-0-136-25 kubenswrapper[2569]: I0706 13:52:10.355563 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1cb95852-b3c5-4b1a-80bf-9d3fa465084a-metrics-certs\") pod \"router-default-966c76789-kpw4q\" (UID: \"1cb95852-b3c5-4b1a-80bf-9d3fa465084a\") " pod="openshift-ingress/router-default-966c76789-kpw4q" Jul 06 13:52:10.355962 ip-10-0-136-25 kubenswrapper[2569]: E0706 13:52:10.355700 2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/1cb95852-b3c5-4b1a-80bf-9d3fa465084a-service-ca-bundle podName:1cb95852-b3c5-4b1a-80bf-9d3fa465084a nodeName:}" failed. No retries permitted until 2026-07-06 13:52:12.355678743 +0000 UTC m=+107.946240519 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/1cb95852-b3c5-4b1a-80bf-9d3fa465084a-service-ca-bundle") pod "router-default-966c76789-kpw4q" (UID: "1cb95852-b3c5-4b1a-80bf-9d3fa465084a") : configmap references non-existent config key: service-ca.crt Jul 06 13:52:10.355962 ip-10-0-136-25 kubenswrapper[2569]: E0706 13:52:10.355703 2569 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found Jul 06 13:52:10.355962 ip-10-0-136-25 kubenswrapper[2569]: E0706 13:52:10.355754 2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1cb95852-b3c5-4b1a-80bf-9d3fa465084a-metrics-certs podName:1cb95852-b3c5-4b1a-80bf-9d3fa465084a nodeName:}" failed. No retries permitted until 2026-07-06 13:52:12.355745883 +0000 UTC m=+107.946307637 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/1cb95852-b3c5-4b1a-80bf-9d3fa465084a-metrics-certs") pod "router-default-966c76789-kpw4q" (UID: "1cb95852-b3c5-4b1a-80bf-9d3fa465084a") : secret "router-metrics-certs-default" not found Jul 06 13:52:10.456361 ip-10-0-136-25 kubenswrapper[2569]: I0706 13:52:10.456320 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/a52c7279-3809-4f22-9001-afb7325cfe8d-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-64786d9cbb-zpkk5\" (UID: \"a52c7279-3809-4f22-9001-afb7325cfe8d\") " pod="openshift-monitoring/cluster-monitoring-operator-64786d9cbb-zpkk5" Jul 06 13:52:10.456552 ip-10-0-136-25 kubenswrapper[2569]: E0706 13:52:10.456495 2569 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found Jul 06 13:52:10.456620 ip-10-0-136-25 kubenswrapper[2569]: E0706 13:52:10.456576 2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a52c7279-3809-4f22-9001-afb7325cfe8d-cluster-monitoring-operator-tls podName:a52c7279-3809-4f22-9001-afb7325cfe8d nodeName:}" failed. No retries permitted until 2026-07-06 13:52:12.456553258 +0000 UTC m=+108.047115017 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/a52c7279-3809-4f22-9001-afb7325cfe8d-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-64786d9cbb-zpkk5" (UID: "a52c7279-3809-4f22-9001-afb7325cfe8d") : secret "cluster-monitoring-operator-tls" not found Jul 06 13:52:12.372626 ip-10-0-136-25 kubenswrapper[2569]: I0706 13:52:12.372596 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1cb95852-b3c5-4b1a-80bf-9d3fa465084a-metrics-certs\") pod \"router-default-966c76789-kpw4q\" (UID: \"1cb95852-b3c5-4b1a-80bf-9d3fa465084a\") " pod="openshift-ingress/router-default-966c76789-kpw4q" Jul 06 13:52:12.372975 ip-10-0-136-25 kubenswrapper[2569]: I0706 13:52:12.372694 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/1cb95852-b3c5-4b1a-80bf-9d3fa465084a-service-ca-bundle\") pod \"router-default-966c76789-kpw4q\" (UID: \"1cb95852-b3c5-4b1a-80bf-9d3fa465084a\") " pod="openshift-ingress/router-default-966c76789-kpw4q" Jul 06 13:52:12.372975 ip-10-0-136-25 kubenswrapper[2569]: E0706 13:52:12.372745 2569 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found Jul 06 13:52:12.372975 ip-10-0-136-25 kubenswrapper[2569]: E0706 13:52:12.372817 2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1cb95852-b3c5-4b1a-80bf-9d3fa465084a-metrics-certs podName:1cb95852-b3c5-4b1a-80bf-9d3fa465084a nodeName:}" failed. No retries permitted until 2026-07-06 13:52:16.37280168 +0000 UTC m=+111.963363436 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/1cb95852-b3c5-4b1a-80bf-9d3fa465084a-metrics-certs") pod "router-default-966c76789-kpw4q" (UID: "1cb95852-b3c5-4b1a-80bf-9d3fa465084a") : secret "router-metrics-certs-default" not found Jul 06 13:52:12.372975 ip-10-0-136-25 kubenswrapper[2569]: E0706 13:52:12.372832 2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/1cb95852-b3c5-4b1a-80bf-9d3fa465084a-service-ca-bundle podName:1cb95852-b3c5-4b1a-80bf-9d3fa465084a nodeName:}" failed. No retries permitted until 2026-07-06 13:52:16.372826168 +0000 UTC m=+111.963387922 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/1cb95852-b3c5-4b1a-80bf-9d3fa465084a-service-ca-bundle") pod "router-default-966c76789-kpw4q" (UID: "1cb95852-b3c5-4b1a-80bf-9d3fa465084a") : configmap references non-existent config key: service-ca.crt Jul 06 13:52:12.474678 ip-10-0-136-25 kubenswrapper[2569]: I0706 13:52:12.473149 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/a52c7279-3809-4f22-9001-afb7325cfe8d-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-64786d9cbb-zpkk5\" (UID: \"a52c7279-3809-4f22-9001-afb7325cfe8d\") " pod="openshift-monitoring/cluster-monitoring-operator-64786d9cbb-zpkk5" Jul 06 13:52:12.474678 ip-10-0-136-25 kubenswrapper[2569]: E0706 13:52:12.473422 2569 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found Jul 06 13:52:12.474678 ip-10-0-136-25 kubenswrapper[2569]: E0706 13:52:12.473493 2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a52c7279-3809-4f22-9001-afb7325cfe8d-cluster-monitoring-operator-tls podName:a52c7279-3809-4f22-9001-afb7325cfe8d nodeName:}" failed. No retries permitted until 2026-07-06 13:52:16.473474886 +0000 UTC m=+112.064036644 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/a52c7279-3809-4f22-9001-afb7325cfe8d-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-64786d9cbb-zpkk5" (UID: "a52c7279-3809-4f22-9001-afb7325cfe8d") : secret "cluster-monitoring-operator-tls" not found Jul 06 13:52:13.185434 ip-10-0-136-25 kubenswrapper[2569]: I0706 13:52:13.185400 2569 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-runtime-extractor-kc97f"] Jul 06 13:52:13.187777 ip-10-0-136-25 kubenswrapper[2569]: I0706 13:52:13.187754 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-kc97f" Jul 06 13:52:13.189998 ip-10-0-136-25 kubenswrapper[2569]: I0706 13:52:13.189973 2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\"" Jul 06 13:52:13.190151 ip-10-0-136-25 kubenswrapper[2569]: I0706 13:52:13.189973 2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\"" Jul 06 13:52:13.190259 ip-10-0-136-25 kubenswrapper[2569]: I0706 13:52:13.190235 2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-cwmvs\"" Jul 06 13:52:13.203067 ip-10-0-136-25 kubenswrapper[2569]: I0706 13:52:13.203041 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-kc97f"] Jul 06 13:52:13.281287 ip-10-0-136-25 kubenswrapper[2569]: I0706 13:52:13.281245 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-k2p2r\" (UniqueName: \"kubernetes.io/projected/3c279543-7a20-4d29-9de1-6cd09f14f692-kube-api-access-k2p2r\") pod \"insights-runtime-extractor-kc97f\" (UID: \"3c279543-7a20-4d29-9de1-6cd09f14f692\") " pod="openshift-insights/insights-runtime-extractor-kc97f" Jul 06 13:52:13.281465 ip-10-0-136-25 kubenswrapper[2569]: I0706 13:52:13.281335 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/3c279543-7a20-4d29-9de1-6cd09f14f692-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-kc97f\" (UID: \"3c279543-7a20-4d29-9de1-6cd09f14f692\") " pod="openshift-insights/insights-runtime-extractor-kc97f" Jul 06 13:52:13.281465 ip-10-0-136-25 kubenswrapper[2569]: I0706 13:52:13.281365 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/3c279543-7a20-4d29-9de1-6cd09f14f692-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-kc97f\" (UID: \"3c279543-7a20-4d29-9de1-6cd09f14f692\") " pod="openshift-insights/insights-runtime-extractor-kc97f" Jul 06 13:52:13.281573 ip-10-0-136-25 kubenswrapper[2569]: I0706 13:52:13.281539 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/3c279543-7a20-4d29-9de1-6cd09f14f692-data-volume\") pod \"insights-runtime-extractor-kc97f\" (UID: \"3c279543-7a20-4d29-9de1-6cd09f14f692\") " pod="openshift-insights/insights-runtime-extractor-kc97f" Jul 06 13:52:13.281625 ip-10-0-136-25 kubenswrapper[2569]: I0706 13:52:13.281573 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/3c279543-7a20-4d29-9de1-6cd09f14f692-crio-socket\") pod \"insights-runtime-extractor-kc97f\" (UID: \"3c279543-7a20-4d29-9de1-6cd09f14f692\") " pod="openshift-insights/insights-runtime-extractor-kc97f" Jul 06 13:52:13.306138 ip-10-0-136-25 kubenswrapper[2569]: I0706 13:52:13.306107 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-68b757fb67-z2h95_dfffcdca-94d6-4732-815f-bcb533f538e9/console-operator/0.log" Jul 06 13:52:13.306322 ip-10-0-136-25 kubenswrapper[2569]: I0706 13:52:13.306156 2569 generic.go:356] "Generic (PLEG): container finished" podID="dfffcdca-94d6-4732-815f-bcb533f538e9" containerID="39c69f0f5fed55d5deb5dffa1f1bbf90b024effecd83ba6dfa76c6e2b1160d6d" exitCode=255 Jul 06 13:52:13.306322 ip-10-0-136-25 kubenswrapper[2569]: I0706 13:52:13.306254 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-68b757fb67-z2h95" event={"ID":"dfffcdca-94d6-4732-815f-bcb533f538e9","Type":"ContainerDied","Data":"39c69f0f5fed55d5deb5dffa1f1bbf90b024effecd83ba6dfa76c6e2b1160d6d"} Jul 06 13:52:13.306475 ip-10-0-136-25 kubenswrapper[2569]: I0706 13:52:13.306460 2569 scope.go:117] "RemoveContainer" containerID="39c69f0f5fed55d5deb5dffa1f1bbf90b024effecd83ba6dfa76c6e2b1160d6d" Jul 06 13:52:13.307827 ip-10-0-136-25 kubenswrapper[2569]: I0706 13:52:13.307793 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-5c847dcfc4-tdvhm" event={"ID":"9746832c-8d02-489c-98f4-afb86bdc46b8","Type":"ContainerStarted","Data":"353a2ebd48aa742b9410dbe567df0daea75ddbb2db6ed241b68b69d517bc6e9a"} Jul 06 13:52:13.309450 ip-10-0-136-25 kubenswrapper[2569]: I0706 13:52:13.309425 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-846bf668f7-52nxm" event={"ID":"2e5b9096-4169-413a-b200-49f75eb977c0","Type":"ContainerStarted","Data":"07fbed486b752e25861952905d86825294ef004a5ec3eb6fbe794f86761c0bb6"} Jul 06 13:52:13.340642 ip-10-0-136-25 kubenswrapper[2569]: I0706 13:52:13.340591 2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-846bf668f7-52nxm" podStartSLOduration=2.065798063 podStartE2EDuration="5.340574515s" totalImagesPullingTime="3.274776452s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-06 13:52:08 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 13:52:13.338431412 +0000 UTC m=+108.928993179" watchObservedRunningTime="2026-07-06 13:52:13.340574515 +0000 UTC m=+108.931136270" Jul 06 13:52:13.369546 ip-10-0-136-25 kubenswrapper[2569]: I0706 13:52:13.369496 2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-insights/insights-operator-5c847dcfc4-tdvhm" podStartSLOduration=2.175143599 podStartE2EDuration="5.36947847s" totalImagesPullingTime="3.194334871s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-06 13:52:08 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 13:52:13.368594999 +0000 UTC m=+108.959156789" watchObservedRunningTime="2026-07-06 13:52:13.36947847 +0000 UTC m=+108.960040247" Jul 06 13:52:13.382374 ip-10-0-136-25 kubenswrapper[2569]: I0706 13:52:13.382338 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/3c279543-7a20-4d29-9de1-6cd09f14f692-data-volume\") pod \"insights-runtime-extractor-kc97f\" (UID: \"3c279543-7a20-4d29-9de1-6cd09f14f692\") " pod="openshift-insights/insights-runtime-extractor-kc97f" Jul 06 13:52:13.382830 ip-10-0-136-25 kubenswrapper[2569]: I0706 13:52:13.382439 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/3c279543-7a20-4d29-9de1-6cd09f14f692-crio-socket\") pod \"insights-runtime-extractor-kc97f\" (UID: \"3c279543-7a20-4d29-9de1-6cd09f14f692\") " pod="openshift-insights/insights-runtime-extractor-kc97f" Jul 06 13:52:13.382830 ip-10-0-136-25 kubenswrapper[2569]: I0706 13:52:13.382536 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/3c279543-7a20-4d29-9de1-6cd09f14f692-crio-socket\") pod \"insights-runtime-extractor-kc97f\" (UID: \"3c279543-7a20-4d29-9de1-6cd09f14f692\") " pod="openshift-insights/insights-runtime-extractor-kc97f" Jul 06 13:52:13.382830 ip-10-0-136-25 kubenswrapper[2569]: I0706 13:52:13.382587 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k2p2r\" (UniqueName: \"kubernetes.io/projected/3c279543-7a20-4d29-9de1-6cd09f14f692-kube-api-access-k2p2r\") pod \"insights-runtime-extractor-kc97f\" (UID: \"3c279543-7a20-4d29-9de1-6cd09f14f692\") " pod="openshift-insights/insights-runtime-extractor-kc97f" Jul 06 13:52:13.382830 ip-10-0-136-25 kubenswrapper[2569]: I0706 13:52:13.382700 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/3c279543-7a20-4d29-9de1-6cd09f14f692-data-volume\") pod \"insights-runtime-extractor-kc97f\" (UID: \"3c279543-7a20-4d29-9de1-6cd09f14f692\") " pod="openshift-insights/insights-runtime-extractor-kc97f" Jul 06 13:52:13.382830 ip-10-0-136-25 kubenswrapper[2569]: I0706 13:52:13.382702 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/3c279543-7a20-4d29-9de1-6cd09f14f692-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-kc97f\" (UID: \"3c279543-7a20-4d29-9de1-6cd09f14f692\") " pod="openshift-insights/insights-runtime-extractor-kc97f" Jul 06 13:52:13.382830 ip-10-0-136-25 kubenswrapper[2569]: I0706 13:52:13.382737 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/3c279543-7a20-4d29-9de1-6cd09f14f692-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-kc97f\" (UID: \"3c279543-7a20-4d29-9de1-6cd09f14f692\") " pod="openshift-insights/insights-runtime-extractor-kc97f" Jul 06 13:52:13.383546 ip-10-0-136-25 kubenswrapper[2569]: E0706 13:52:13.383287 2569 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Jul 06 13:52:13.383546 ip-10-0-136-25 kubenswrapper[2569]: E0706 13:52:13.383346 2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/3c279543-7a20-4d29-9de1-6cd09f14f692-kube-rbac-proxy-cm podName:3c279543-7a20-4d29-9de1-6cd09f14f692 nodeName:}" failed. No retries permitted until 2026-07-06 13:52:13.883326347 +0000 UTC m=+109.473888111 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/3c279543-7a20-4d29-9de1-6cd09f14f692-kube-rbac-proxy-cm") pod "insights-runtime-extractor-kc97f" (UID: "3c279543-7a20-4d29-9de1-6cd09f14f692") : configmap "kube-rbac-proxy" not found Jul 06 13:52:13.383546 ip-10-0-136-25 kubenswrapper[2569]: E0706 13:52:13.383444 2569 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Jul 06 13:52:13.383546 ip-10-0-136-25 kubenswrapper[2569]: E0706 13:52:13.383519 2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3c279543-7a20-4d29-9de1-6cd09f14f692-insights-runtime-extractor-tls podName:3c279543-7a20-4d29-9de1-6cd09f14f692 nodeName:}" failed. No retries permitted until 2026-07-06 13:52:13.883497645 +0000 UTC m=+109.474059405 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/3c279543-7a20-4d29-9de1-6cd09f14f692-insights-runtime-extractor-tls") pod "insights-runtime-extractor-kc97f" (UID: "3c279543-7a20-4d29-9de1-6cd09f14f692") : secret "insights-runtime-extractor-tls" not found Jul 06 13:52:13.394906 ip-10-0-136-25 kubenswrapper[2569]: I0706 13:52:13.394871 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-k2p2r\" (UniqueName: \"kubernetes.io/projected/3c279543-7a20-4d29-9de1-6cd09f14f692-kube-api-access-k2p2r\") pod \"insights-runtime-extractor-kc97f\" (UID: \"3c279543-7a20-4d29-9de1-6cd09f14f692\") " pod="openshift-insights/insights-runtime-extractor-kc97f" Jul 06 13:52:13.886599 ip-10-0-136-25 kubenswrapper[2569]: I0706 13:52:13.886561 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/3c279543-7a20-4d29-9de1-6cd09f14f692-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-kc97f\" (UID: \"3c279543-7a20-4d29-9de1-6cd09f14f692\") " pod="openshift-insights/insights-runtime-extractor-kc97f" Jul 06 13:52:13.886599 ip-10-0-136-25 kubenswrapper[2569]: I0706 13:52:13.886600 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/3c279543-7a20-4d29-9de1-6cd09f14f692-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-kc97f\" (UID: \"3c279543-7a20-4d29-9de1-6cd09f14f692\") " pod="openshift-insights/insights-runtime-extractor-kc97f" Jul 06 13:52:13.886896 ip-10-0-136-25 kubenswrapper[2569]: E0706 13:52:13.886726 2569 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Jul 06 13:52:13.886896 ip-10-0-136-25 kubenswrapper[2569]: E0706 13:52:13.886745 2569 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Jul 06 13:52:13.886896 ip-10-0-136-25 kubenswrapper[2569]: E0706 13:52:13.886782 2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/3c279543-7a20-4d29-9de1-6cd09f14f692-kube-rbac-proxy-cm podName:3c279543-7a20-4d29-9de1-6cd09f14f692 nodeName:}" failed. No retries permitted until 2026-07-06 13:52:14.886767722 +0000 UTC m=+110.477329476 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/3c279543-7a20-4d29-9de1-6cd09f14f692-kube-rbac-proxy-cm") pod "insights-runtime-extractor-kc97f" (UID: "3c279543-7a20-4d29-9de1-6cd09f14f692") : configmap "kube-rbac-proxy" not found Jul 06 13:52:13.886896 ip-10-0-136-25 kubenswrapper[2569]: E0706 13:52:13.886820 2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3c279543-7a20-4d29-9de1-6cd09f14f692-insights-runtime-extractor-tls podName:3c279543-7a20-4d29-9de1-6cd09f14f692 nodeName:}" failed. No retries permitted until 2026-07-06 13:52:14.886804291 +0000 UTC m=+110.477366050 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/3c279543-7a20-4d29-9de1-6cd09f14f692-insights-runtime-extractor-tls") pod "insights-runtime-extractor-kc97f" (UID: "3c279543-7a20-4d29-9de1-6cd09f14f692") : secret "insights-runtime-extractor-tls" not found Jul 06 13:52:14.314079 ip-10-0-136-25 kubenswrapper[2569]: I0706 13:52:14.314046 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-68b757fb67-z2h95_dfffcdca-94d6-4732-815f-bcb533f538e9/console-operator/1.log" Jul 06 13:52:14.314539 ip-10-0-136-25 kubenswrapper[2569]: I0706 13:52:14.314520 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-68b757fb67-z2h95_dfffcdca-94d6-4732-815f-bcb533f538e9/console-operator/0.log" Jul 06 13:52:14.314668 ip-10-0-136-25 kubenswrapper[2569]: I0706 13:52:14.314562 2569 generic.go:356] "Generic (PLEG): container finished" podID="dfffcdca-94d6-4732-815f-bcb533f538e9" containerID="881ec37329e894d693c9d8a5839d9eb26285bbe019fdcfcc41dd6fd179641d82" exitCode=255 Jul 06 13:52:14.314735 ip-10-0-136-25 kubenswrapper[2569]: I0706 13:52:14.314665 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-68b757fb67-z2h95" event={"ID":"dfffcdca-94d6-4732-815f-bcb533f538e9","Type":"ContainerDied","Data":"881ec37329e894d693c9d8a5839d9eb26285bbe019fdcfcc41dd6fd179641d82"} Jul 06 13:52:14.314735 ip-10-0-136-25 kubenswrapper[2569]: I0706 13:52:14.314720 2569 scope.go:117] "RemoveContainer" containerID="39c69f0f5fed55d5deb5dffa1f1bbf90b024effecd83ba6dfa76c6e2b1160d6d" Jul 06 13:52:14.314904 ip-10-0-136-25 kubenswrapper[2569]: I0706 13:52:14.314887 2569 scope.go:117] "RemoveContainer" containerID="881ec37329e894d693c9d8a5839d9eb26285bbe019fdcfcc41dd6fd179641d82" Jul 06 13:52:14.315104 ip-10-0-136-25 kubenswrapper[2569]: E0706 13:52:14.315071 2569 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 10s restarting failed container=console-operator pod=console-operator-68b757fb67-z2h95_openshift-console-operator(dfffcdca-94d6-4732-815f-bcb533f538e9)\"" pod="openshift-console-operator/console-operator-68b757fb67-z2h95" podUID="dfffcdca-94d6-4732-815f-bcb533f538e9" Jul 06 13:52:14.895619 ip-10-0-136-25 kubenswrapper[2569]: I0706 13:52:14.895583 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/3c279543-7a20-4d29-9de1-6cd09f14f692-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-kc97f\" (UID: \"3c279543-7a20-4d29-9de1-6cd09f14f692\") " pod="openshift-insights/insights-runtime-extractor-kc97f" Jul 06 13:52:14.895619 ip-10-0-136-25 kubenswrapper[2569]: I0706 13:52:14.895619 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/3c279543-7a20-4d29-9de1-6cd09f14f692-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-kc97f\" (UID: \"3c279543-7a20-4d29-9de1-6cd09f14f692\") " pod="openshift-insights/insights-runtime-extractor-kc97f" Jul 06 13:52:14.896091 ip-10-0-136-25 kubenswrapper[2569]: E0706 13:52:14.895696 2569 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Jul 06 13:52:14.896091 ip-10-0-136-25 kubenswrapper[2569]: E0706 13:52:14.895753 2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/3c279543-7a20-4d29-9de1-6cd09f14f692-kube-rbac-proxy-cm podName:3c279543-7a20-4d29-9de1-6cd09f14f692 nodeName:}" failed. No retries permitted until 2026-07-06 13:52:16.89573662 +0000 UTC m=+112.486298377 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/3c279543-7a20-4d29-9de1-6cd09f14f692-kube-rbac-proxy-cm") pod "insights-runtime-extractor-kc97f" (UID: "3c279543-7a20-4d29-9de1-6cd09f14f692") : configmap "kube-rbac-proxy" not found Jul 06 13:52:14.896091 ip-10-0-136-25 kubenswrapper[2569]: E0706 13:52:14.895748 2569 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Jul 06 13:52:14.896091 ip-10-0-136-25 kubenswrapper[2569]: E0706 13:52:14.895817 2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3c279543-7a20-4d29-9de1-6cd09f14f692-insights-runtime-extractor-tls podName:3c279543-7a20-4d29-9de1-6cd09f14f692 nodeName:}" failed. No retries permitted until 2026-07-06 13:52:16.895799894 +0000 UTC m=+112.486361666 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/3c279543-7a20-4d29-9de1-6cd09f14f692-insights-runtime-extractor-tls") pod "insights-runtime-extractor-kc97f" (UID: "3c279543-7a20-4d29-9de1-6cd09f14f692") : secret "insights-runtime-extractor-tls" not found Jul 06 13:52:14.957730 ip-10-0-136-25 kubenswrapper[2569]: I0706 13:52:14.957703 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-54w6r_ef1daf8b-7b12-4c6a-9fbd-2b7e9187fd8f/dns-node-resolver/0.log" Jul 06 13:52:15.319037 ip-10-0-136-25 kubenswrapper[2569]: I0706 13:52:15.319007 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-68b757fb67-z2h95_dfffcdca-94d6-4732-815f-bcb533f538e9/console-operator/1.log" Jul 06 13:52:15.319367 ip-10-0-136-25 kubenswrapper[2569]: I0706 13:52:15.319352 2569 scope.go:117] "RemoveContainer" containerID="881ec37329e894d693c9d8a5839d9eb26285bbe019fdcfcc41dd6fd179641d82" Jul 06 13:52:15.319518 ip-10-0-136-25 kubenswrapper[2569]: E0706 13:52:15.319499 2569 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 10s restarting failed container=console-operator pod=console-operator-68b757fb67-z2h95_openshift-console-operator(dfffcdca-94d6-4732-815f-bcb533f538e9)\"" pod="openshift-console-operator/console-operator-68b757fb67-z2h95" podUID="dfffcdca-94d6-4732-815f-bcb533f538e9" Jul 06 13:52:15.759145 ip-10-0-136-25 kubenswrapper[2569]: I0706 13:52:15.759115 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-c9sqv_028cbff5-11b2-411a-90ed-c0bd3f563830/node-ca/0.log" Jul 06 13:52:16.408445 ip-10-0-136-25 kubenswrapper[2569]: I0706 13:52:16.408407 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1cb95852-b3c5-4b1a-80bf-9d3fa465084a-metrics-certs\") pod \"router-default-966c76789-kpw4q\" (UID: \"1cb95852-b3c5-4b1a-80bf-9d3fa465084a\") " pod="openshift-ingress/router-default-966c76789-kpw4q" Jul 06 13:52:16.408804 ip-10-0-136-25 kubenswrapper[2569]: I0706 13:52:16.408503 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/1cb95852-b3c5-4b1a-80bf-9d3fa465084a-service-ca-bundle\") pod \"router-default-966c76789-kpw4q\" (UID: \"1cb95852-b3c5-4b1a-80bf-9d3fa465084a\") " pod="openshift-ingress/router-default-966c76789-kpw4q" Jul 06 13:52:16.408804 ip-10-0-136-25 kubenswrapper[2569]: E0706 13:52:16.408559 2569 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found Jul 06 13:52:16.408804 ip-10-0-136-25 kubenswrapper[2569]: E0706 13:52:16.408621 2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1cb95852-b3c5-4b1a-80bf-9d3fa465084a-metrics-certs podName:1cb95852-b3c5-4b1a-80bf-9d3fa465084a nodeName:}" failed. No retries permitted until 2026-07-06 13:52:24.408606076 +0000 UTC m=+119.999167836 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/1cb95852-b3c5-4b1a-80bf-9d3fa465084a-metrics-certs") pod "router-default-966c76789-kpw4q" (UID: "1cb95852-b3c5-4b1a-80bf-9d3fa465084a") : secret "router-metrics-certs-default" not found Jul 06 13:52:16.408804 ip-10-0-136-25 kubenswrapper[2569]: E0706 13:52:16.408638 2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/1cb95852-b3c5-4b1a-80bf-9d3fa465084a-service-ca-bundle podName:1cb95852-b3c5-4b1a-80bf-9d3fa465084a nodeName:}" failed. No retries permitted until 2026-07-06 13:52:24.408631453 +0000 UTC m=+119.999193211 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/1cb95852-b3c5-4b1a-80bf-9d3fa465084a-service-ca-bundle") pod "router-default-966c76789-kpw4q" (UID: "1cb95852-b3c5-4b1a-80bf-9d3fa465084a") : configmap references non-existent config key: service-ca.crt Jul 06 13:52:16.509116 ip-10-0-136-25 kubenswrapper[2569]: I0706 13:52:16.509077 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/a52c7279-3809-4f22-9001-afb7325cfe8d-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-64786d9cbb-zpkk5\" (UID: \"a52c7279-3809-4f22-9001-afb7325cfe8d\") " pod="openshift-monitoring/cluster-monitoring-operator-64786d9cbb-zpkk5" Jul 06 13:52:16.509313 ip-10-0-136-25 kubenswrapper[2569]: E0706 13:52:16.509262 2569 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found Jul 06 13:52:16.509358 ip-10-0-136-25 kubenswrapper[2569]: E0706 13:52:16.509332 2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a52c7279-3809-4f22-9001-afb7325cfe8d-cluster-monitoring-operator-tls podName:a52c7279-3809-4f22-9001-afb7325cfe8d nodeName:}" failed. No retries permitted until 2026-07-06 13:52:24.509315075 +0000 UTC m=+120.099876831 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/a52c7279-3809-4f22-9001-afb7325cfe8d-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-64786d9cbb-zpkk5" (UID: "a52c7279-3809-4f22-9001-afb7325cfe8d") : secret "cluster-monitoring-operator-tls" not found Jul 06 13:52:16.557434 ip-10-0-136-25 kubenswrapper[2569]: I0706 13:52:16.557401 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-4wkrh_e5a5cf60-4e99-42b0-b705-96449966ca70/serve-healthcheck-canary/0.log" Jul 06 13:52:16.912672 ip-10-0-136-25 kubenswrapper[2569]: I0706 13:52:16.912637 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/3c279543-7a20-4d29-9de1-6cd09f14f692-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-kc97f\" (UID: \"3c279543-7a20-4d29-9de1-6cd09f14f692\") " pod="openshift-insights/insights-runtime-extractor-kc97f" Jul 06 13:52:16.912672 ip-10-0-136-25 kubenswrapper[2569]: I0706 13:52:16.912674 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/3c279543-7a20-4d29-9de1-6cd09f14f692-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-kc97f\" (UID: \"3c279543-7a20-4d29-9de1-6cd09f14f692\") " pod="openshift-insights/insights-runtime-extractor-kc97f" Jul 06 13:52:16.912888 ip-10-0-136-25 kubenswrapper[2569]: E0706 13:52:16.912793 2569 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Jul 06 13:52:16.912888 ip-10-0-136-25 kubenswrapper[2569]: E0706 13:52:16.912807 2569 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Jul 06 13:52:16.912888 ip-10-0-136-25 kubenswrapper[2569]: E0706 13:52:16.912860 2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3c279543-7a20-4d29-9de1-6cd09f14f692-insights-runtime-extractor-tls podName:3c279543-7a20-4d29-9de1-6cd09f14f692 nodeName:}" failed. No retries permitted until 2026-07-06 13:52:20.912844701 +0000 UTC m=+116.503406461 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/3c279543-7a20-4d29-9de1-6cd09f14f692-insights-runtime-extractor-tls") pod "insights-runtime-extractor-kc97f" (UID: "3c279543-7a20-4d29-9de1-6cd09f14f692") : secret "insights-runtime-extractor-tls" not found Jul 06 13:52:16.912888 ip-10-0-136-25 kubenswrapper[2569]: E0706 13:52:16.912877 2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/3c279543-7a20-4d29-9de1-6cd09f14f692-kube-rbac-proxy-cm podName:3c279543-7a20-4d29-9de1-6cd09f14f692 nodeName:}" failed. No retries permitted until 2026-07-06 13:52:20.912869714 +0000 UTC m=+116.503431473 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/3c279543-7a20-4d29-9de1-6cd09f14f692-kube-rbac-proxy-cm") pod "insights-runtime-extractor-kc97f" (UID: "3c279543-7a20-4d29-9de1-6cd09f14f692") : configmap "kube-rbac-proxy" not found Jul 06 13:52:18.995889 ip-10-0-136-25 kubenswrapper[2569]: I0706 13:52:18.995844 2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console-operator/console-operator-68b757fb67-z2h95" Jul 06 13:52:18.996317 ip-10-0-136-25 kubenswrapper[2569]: I0706 13:52:18.995917 2569 kubelet.go:2654] "SyncLoop (probe)" probe="liveness" status="unhealthy" pod="openshift-console-operator/console-operator-68b757fb67-z2h95" Jul 06 13:52:18.996317 ip-10-0-136-25 kubenswrapper[2569]: I0706 13:52:18.996271 2569 scope.go:117] "RemoveContainer" containerID="881ec37329e894d693c9d8a5839d9eb26285bbe019fdcfcc41dd6fd179641d82" Jul 06 13:52:18.996438 ip-10-0-136-25 kubenswrapper[2569]: E0706 13:52:18.996419 2569 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 10s restarting failed container=console-operator pod=console-operator-68b757fb67-z2h95_openshift-console-operator(dfffcdca-94d6-4732-815f-bcb533f538e9)\"" pod="openshift-console-operator/console-operator-68b757fb67-z2h95" podUID="dfffcdca-94d6-4732-815f-bcb533f538e9" Jul 06 13:52:20.945915 ip-10-0-136-25 kubenswrapper[2569]: I0706 13:52:20.945881 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/3c279543-7a20-4d29-9de1-6cd09f14f692-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-kc97f\" (UID: \"3c279543-7a20-4d29-9de1-6cd09f14f692\") " pod="openshift-insights/insights-runtime-extractor-kc97f" Jul 06 13:52:20.945915 ip-10-0-136-25 kubenswrapper[2569]: I0706 13:52:20.945915 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/3c279543-7a20-4d29-9de1-6cd09f14f692-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-kc97f\" (UID: \"3c279543-7a20-4d29-9de1-6cd09f14f692\") " pod="openshift-insights/insights-runtime-extractor-kc97f" Jul 06 13:52:20.946389 ip-10-0-136-25 kubenswrapper[2569]: E0706 13:52:20.945996 2569 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Jul 06 13:52:20.946389 ip-10-0-136-25 kubenswrapper[2569]: E0706 13:52:20.946043 2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/3c279543-7a20-4d29-9de1-6cd09f14f692-kube-rbac-proxy-cm podName:3c279543-7a20-4d29-9de1-6cd09f14f692 nodeName:}" failed. No retries permitted until 2026-07-06 13:52:28.946029855 +0000 UTC m=+124.536591611 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/3c279543-7a20-4d29-9de1-6cd09f14f692-kube-rbac-proxy-cm") pod "insights-runtime-extractor-kc97f" (UID: "3c279543-7a20-4d29-9de1-6cd09f14f692") : configmap "kube-rbac-proxy" not found Jul 06 13:52:20.946389 ip-10-0-136-25 kubenswrapper[2569]: E0706 13:52:20.946094 2569 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Jul 06 13:52:20.946389 ip-10-0-136-25 kubenswrapper[2569]: E0706 13:52:20.946146 2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3c279543-7a20-4d29-9de1-6cd09f14f692-insights-runtime-extractor-tls podName:3c279543-7a20-4d29-9de1-6cd09f14f692 nodeName:}" failed. No retries permitted until 2026-07-06 13:52:28.946134543 +0000 UTC m=+124.536696298 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/3c279543-7a20-4d29-9de1-6cd09f14f692-insights-runtime-extractor-tls") pod "insights-runtime-extractor-kc97f" (UID: "3c279543-7a20-4d29-9de1-6cd09f14f692") : secret "insights-runtime-extractor-tls" not found Jul 06 13:52:24.476470 ip-10-0-136-25 kubenswrapper[2569]: I0706 13:52:24.476429 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1cb95852-b3c5-4b1a-80bf-9d3fa465084a-metrics-certs\") pod \"router-default-966c76789-kpw4q\" (UID: \"1cb95852-b3c5-4b1a-80bf-9d3fa465084a\") " pod="openshift-ingress/router-default-966c76789-kpw4q" Jul 06 13:52:24.476896 ip-10-0-136-25 kubenswrapper[2569]: I0706 13:52:24.476557 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/1cb95852-b3c5-4b1a-80bf-9d3fa465084a-service-ca-bundle\") pod \"router-default-966c76789-kpw4q\" (UID: \"1cb95852-b3c5-4b1a-80bf-9d3fa465084a\") " pod="openshift-ingress/router-default-966c76789-kpw4q" Jul 06 13:52:24.476896 ip-10-0-136-25 kubenswrapper[2569]: E0706 13:52:24.476720 2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/1cb95852-b3c5-4b1a-80bf-9d3fa465084a-service-ca-bundle podName:1cb95852-b3c5-4b1a-80bf-9d3fa465084a nodeName:}" failed. No retries permitted until 2026-07-06 13:52:40.476703427 +0000 UTC m=+136.067265185 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/1cb95852-b3c5-4b1a-80bf-9d3fa465084a-service-ca-bundle") pod "router-default-966c76789-kpw4q" (UID: "1cb95852-b3c5-4b1a-80bf-9d3fa465084a") : configmap references non-existent config key: service-ca.crt Jul 06 13:52:24.478789 ip-10-0-136-25 kubenswrapper[2569]: I0706 13:52:24.478768 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1cb95852-b3c5-4b1a-80bf-9d3fa465084a-metrics-certs\") pod \"router-default-966c76789-kpw4q\" (UID: \"1cb95852-b3c5-4b1a-80bf-9d3fa465084a\") " pod="openshift-ingress/router-default-966c76789-kpw4q" Jul 06 13:52:24.577181 ip-10-0-136-25 kubenswrapper[2569]: I0706 13:52:24.577137 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/a52c7279-3809-4f22-9001-afb7325cfe8d-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-64786d9cbb-zpkk5\" (UID: \"a52c7279-3809-4f22-9001-afb7325cfe8d\") " pod="openshift-monitoring/cluster-monitoring-operator-64786d9cbb-zpkk5" Jul 06 13:52:24.577379 ip-10-0-136-25 kubenswrapper[2569]: E0706 13:52:24.577351 2569 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found Jul 06 13:52:24.577434 ip-10-0-136-25 kubenswrapper[2569]: E0706 13:52:24.577423 2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a52c7279-3809-4f22-9001-afb7325cfe8d-cluster-monitoring-operator-tls podName:a52c7279-3809-4f22-9001-afb7325cfe8d nodeName:}" failed. No retries permitted until 2026-07-06 13:52:40.577407056 +0000 UTC m=+136.167968811 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/a52c7279-3809-4f22-9001-afb7325cfe8d-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-64786d9cbb-zpkk5" (UID: "a52c7279-3809-4f22-9001-afb7325cfe8d") : secret "cluster-monitoring-operator-tls" not found Jul 06 13:52:29.018232 ip-10-0-136-25 kubenswrapper[2569]: I0706 13:52:29.018176 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/3c279543-7a20-4d29-9de1-6cd09f14f692-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-kc97f\" (UID: \"3c279543-7a20-4d29-9de1-6cd09f14f692\") " pod="openshift-insights/insights-runtime-extractor-kc97f" Jul 06 13:52:29.018679 ip-10-0-136-25 kubenswrapper[2569]: E0706 13:52:29.018335 2569 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Jul 06 13:52:29.018679 ip-10-0-136-25 kubenswrapper[2569]: I0706 13:52:29.018342 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/3c279543-7a20-4d29-9de1-6cd09f14f692-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-kc97f\" (UID: \"3c279543-7a20-4d29-9de1-6cd09f14f692\") " pod="openshift-insights/insights-runtime-extractor-kc97f" Jul 06 13:52:29.018679 ip-10-0-136-25 kubenswrapper[2569]: E0706 13:52:29.018399 2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3c279543-7a20-4d29-9de1-6cd09f14f692-insights-runtime-extractor-tls podName:3c279543-7a20-4d29-9de1-6cd09f14f692 nodeName:}" failed. No retries permitted until 2026-07-06 13:52:45.018378851 +0000 UTC m=+140.608940610 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/3c279543-7a20-4d29-9de1-6cd09f14f692-insights-runtime-extractor-tls") pod "insights-runtime-extractor-kc97f" (UID: "3c279543-7a20-4d29-9de1-6cd09f14f692") : secret "insights-runtime-extractor-tls" not found Jul 06 13:52:29.018864 ip-10-0-136-25 kubenswrapper[2569]: I0706 13:52:29.018795 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/3c279543-7a20-4d29-9de1-6cd09f14f692-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-kc97f\" (UID: \"3c279543-7a20-4d29-9de1-6cd09f14f692\") " pod="openshift-insights/insights-runtime-extractor-kc97f" Jul 06 13:52:31.935076 ip-10-0-136-25 kubenswrapper[2569]: I0706 13:52:31.935045 2569 scope.go:117] "RemoveContainer" containerID="881ec37329e894d693c9d8a5839d9eb26285bbe019fdcfcc41dd6fd179641d82" Jul 06 13:52:32.372679 ip-10-0-136-25 kubenswrapper[2569]: I0706 13:52:32.372648 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-68b757fb67-z2h95_dfffcdca-94d6-4732-815f-bcb533f538e9/console-operator/2.log" Jul 06 13:52:32.373052 ip-10-0-136-25 kubenswrapper[2569]: I0706 13:52:32.373037 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-68b757fb67-z2h95_dfffcdca-94d6-4732-815f-bcb533f538e9/console-operator/1.log" Jul 06 13:52:32.373109 ip-10-0-136-25 kubenswrapper[2569]: I0706 13:52:32.373068 2569 generic.go:356] "Generic (PLEG): container finished" podID="dfffcdca-94d6-4732-815f-bcb533f538e9" containerID="44529ca19da436ed637824808a9ac86d648c96ea3e9fb7da354016dd4df57d81" exitCode=255 Jul 06 13:52:32.373146 ip-10-0-136-25 kubenswrapper[2569]: I0706 13:52:32.373129 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-68b757fb67-z2h95" event={"ID":"dfffcdca-94d6-4732-815f-bcb533f538e9","Type":"ContainerDied","Data":"44529ca19da436ed637824808a9ac86d648c96ea3e9fb7da354016dd4df57d81"} Jul 06 13:52:32.373185 ip-10-0-136-25 kubenswrapper[2569]: I0706 13:52:32.373172 2569 scope.go:117] "RemoveContainer" containerID="881ec37329e894d693c9d8a5839d9eb26285bbe019fdcfcc41dd6fd179641d82" Jul 06 13:52:32.373492 ip-10-0-136-25 kubenswrapper[2569]: I0706 13:52:32.373471 2569 scope.go:117] "RemoveContainer" containerID="44529ca19da436ed637824808a9ac86d648c96ea3e9fb7da354016dd4df57d81" Jul 06 13:52:32.373673 ip-10-0-136-25 kubenswrapper[2569]: E0706 13:52:32.373653 2569 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 20s restarting failed container=console-operator pod=console-operator-68b757fb67-z2h95_openshift-console-operator(dfffcdca-94d6-4732-815f-bcb533f538e9)\"" pod="openshift-console-operator/console-operator-68b757fb67-z2h95" podUID="dfffcdca-94d6-4732-815f-bcb533f538e9" Jul 06 13:52:33.379896 ip-10-0-136-25 kubenswrapper[2569]: I0706 13:52:33.379860 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-68b757fb67-z2h95_dfffcdca-94d6-4732-815f-bcb533f538e9/console-operator/2.log" Jul 06 13:52:34.765251 ip-10-0-136-25 kubenswrapper[2569]: I0706 13:52:34.765183 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2c4b4cf7-31a7-4cde-8212-9d78cc184e6c-metrics-certs\") pod \"network-metrics-daemon-hqt9l\" (UID: \"2c4b4cf7-31a7-4cde-8212-9d78cc184e6c\") " pod="openshift-multus/network-metrics-daemon-hqt9l" Jul 06 13:52:34.767576 ip-10-0-136-25 kubenswrapper[2569]: I0706 13:52:34.767552 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2c4b4cf7-31a7-4cde-8212-9d78cc184e6c-metrics-certs\") pod \"network-metrics-daemon-hqt9l\" (UID: \"2c4b4cf7-31a7-4cde-8212-9d78cc184e6c\") " pod="openshift-multus/network-metrics-daemon-hqt9l" Jul 06 13:52:34.847508 ip-10-0-136-25 kubenswrapper[2569]: I0706 13:52:34.847472 2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-469kc\"" Jul 06 13:52:34.856447 ip-10-0-136-25 kubenswrapper[2569]: I0706 13:52:34.856423 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-hqt9l" Jul 06 13:52:34.979466 ip-10-0-136-25 kubenswrapper[2569]: I0706 13:52:34.979431 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-hqt9l"] Jul 06 13:52:34.983372 ip-10-0-136-25 kubenswrapper[2569]: W0706 13:52:34.983349 2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2c4b4cf7_31a7_4cde_8212_9d78cc184e6c.slice/crio-1deb53c164e4a193116d73696ffe2dc5d89f5739da26e6c2627c123f4636667c WatchSource:0}: Error finding container 1deb53c164e4a193116d73696ffe2dc5d89f5739da26e6c2627c123f4636667c: Status 404 returned error can't find the container with id 1deb53c164e4a193116d73696ffe2dc5d89f5739da26e6c2627c123f4636667c Jul 06 13:52:35.386772 ip-10-0-136-25 kubenswrapper[2569]: I0706 13:52:35.386738 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-hqt9l" event={"ID":"2c4b4cf7-31a7-4cde-8212-9d78cc184e6c","Type":"ContainerStarted","Data":"1deb53c164e4a193116d73696ffe2dc5d89f5739da26e6c2627c123f4636667c"} Jul 06 13:52:36.392510 ip-10-0-136-25 kubenswrapper[2569]: I0706 13:52:36.392473 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-hqt9l" event={"ID":"2c4b4cf7-31a7-4cde-8212-9d78cc184e6c","Type":"ContainerStarted","Data":"a1dc9eea80984770fc3e20f18c79e8d3b500918ba5f148407b70fd507561bdc2"} Jul 06 13:52:36.392989 ip-10-0-136-25 kubenswrapper[2569]: I0706 13:52:36.392528 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-hqt9l" event={"ID":"2c4b4cf7-31a7-4cde-8212-9d78cc184e6c","Type":"ContainerStarted","Data":"a9772c46d2060c0970d7eb44210ae7e378441d47a5c01c0104c96eac902e762c"} Jul 06 13:52:36.410038 ip-10-0-136-25 kubenswrapper[2569]: I0706 13:52:36.409991 2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/network-metrics-daemon-hqt9l" podStartSLOduration=130.52007577 podStartE2EDuration="2m11.409974595s" totalImagesPullingTime="889.898825ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-06 13:50:25 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 13:52:36.409578673 +0000 UTC m=+132.000140451" watchObservedRunningTime="2026-07-06 13:52:36.409974595 +0000 UTC m=+132.000536371" Jul 06 13:52:38.996120 ip-10-0-136-25 kubenswrapper[2569]: I0706 13:52:38.996080 2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console-operator/console-operator-68b757fb67-z2h95" Jul 06 13:52:38.996545 ip-10-0-136-25 kubenswrapper[2569]: I0706 13:52:38.996142 2569 kubelet.go:2654] "SyncLoop (probe)" probe="liveness" status="unhealthy" pod="openshift-console-operator/console-operator-68b757fb67-z2h95" Jul 06 13:52:38.996545 ip-10-0-136-25 kubenswrapper[2569]: I0706 13:52:38.996454 2569 scope.go:117] "RemoveContainer" containerID="44529ca19da436ed637824808a9ac86d648c96ea3e9fb7da354016dd4df57d81" Jul 06 13:52:38.996631 ip-10-0-136-25 kubenswrapper[2569]: E0706 13:52:38.996608 2569 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 20s restarting failed container=console-operator pod=console-operator-68b757fb67-z2h95_openshift-console-operator(dfffcdca-94d6-4732-815f-bcb533f538e9)\"" pod="openshift-console-operator/console-operator-68b757fb67-z2h95" podUID="dfffcdca-94d6-4732-815f-bcb533f538e9" Jul 06 13:52:40.517066 ip-10-0-136-25 kubenswrapper[2569]: I0706 13:52:40.517004 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/1cb95852-b3c5-4b1a-80bf-9d3fa465084a-service-ca-bundle\") pod \"router-default-966c76789-kpw4q\" (UID: \"1cb95852-b3c5-4b1a-80bf-9d3fa465084a\") " pod="openshift-ingress/router-default-966c76789-kpw4q" Jul 06 13:52:40.517656 ip-10-0-136-25 kubenswrapper[2569]: I0706 13:52:40.517633 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/1cb95852-b3c5-4b1a-80bf-9d3fa465084a-service-ca-bundle\") pod \"router-default-966c76789-kpw4q\" (UID: \"1cb95852-b3c5-4b1a-80bf-9d3fa465084a\") " pod="openshift-ingress/router-default-966c76789-kpw4q" Jul 06 13:52:40.617595 ip-10-0-136-25 kubenswrapper[2569]: I0706 13:52:40.617561 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/a52c7279-3809-4f22-9001-afb7325cfe8d-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-64786d9cbb-zpkk5\" (UID: \"a52c7279-3809-4f22-9001-afb7325cfe8d\") " pod="openshift-monitoring/cluster-monitoring-operator-64786d9cbb-zpkk5" Jul 06 13:52:40.619882 ip-10-0-136-25 kubenswrapper[2569]: I0706 13:52:40.619860 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/a52c7279-3809-4f22-9001-afb7325cfe8d-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-64786d9cbb-zpkk5\" (UID: \"a52c7279-3809-4f22-9001-afb7325cfe8d\") " pod="openshift-monitoring/cluster-monitoring-operator-64786d9cbb-zpkk5" Jul 06 13:52:40.818995 ip-10-0-136-25 kubenswrapper[2569]: I0706 13:52:40.818910 2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-dockercfg-brpsb\"" Jul 06 13:52:40.827019 ip-10-0-136-25 kubenswrapper[2569]: I0706 13:52:40.826985 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/router-default-966c76789-kpw4q" Jul 06 13:52:40.844505 ip-10-0-136-25 kubenswrapper[2569]: I0706 13:52:40.844479 2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"cluster-monitoring-operator-dockercfg-rtm87\"" Jul 06 13:52:40.853136 ip-10-0-136-25 kubenswrapper[2569]: I0706 13:52:40.853104 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/cluster-monitoring-operator-64786d9cbb-zpkk5" Jul 06 13:52:40.962437 ip-10-0-136-25 kubenswrapper[2569]: I0706 13:52:40.962408 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/router-default-966c76789-kpw4q"] Jul 06 13:52:40.965664 ip-10-0-136-25 kubenswrapper[2569]: W0706 13:52:40.965637 2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1cb95852_b3c5_4b1a_80bf_9d3fa465084a.slice/crio-b80c2225c3c5def8e4966cba4fddc783ee32ebbf4fcfe2ea7320c8b29364905c WatchSource:0}: Error finding container b80c2225c3c5def8e4966cba4fddc783ee32ebbf4fcfe2ea7320c8b29364905c: Status 404 returned error can't find the container with id b80c2225c3c5def8e4966cba4fddc783ee32ebbf4fcfe2ea7320c8b29364905c Jul 06 13:52:40.988976 ip-10-0-136-25 kubenswrapper[2569]: I0706 13:52:40.988950 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/cluster-monitoring-operator-64786d9cbb-zpkk5"] Jul 06 13:52:40.994197 ip-10-0-136-25 kubenswrapper[2569]: W0706 13:52:40.994173 2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda52c7279_3809_4f22_9001_afb7325cfe8d.slice/crio-5d37548b7a1eca5a047e36718f2150a41d8aed12ff931c4fb010eb06526b9c60 WatchSource:0}: Error finding container 5d37548b7a1eca5a047e36718f2150a41d8aed12ff931c4fb010eb06526b9c60: Status 404 returned error can't find the container with id 5d37548b7a1eca5a047e36718f2150a41d8aed12ff931c4fb010eb06526b9c60 Jul 06 13:52:41.407130 ip-10-0-136-25 kubenswrapper[2569]: I0706 13:52:41.407089 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/cluster-monitoring-operator-64786d9cbb-zpkk5" event={"ID":"a52c7279-3809-4f22-9001-afb7325cfe8d","Type":"ContainerStarted","Data":"5d37548b7a1eca5a047e36718f2150a41d8aed12ff931c4fb010eb06526b9c60"} Jul 06 13:52:41.408481 ip-10-0-136-25 kubenswrapper[2569]: I0706 13:52:41.408460 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/router-default-966c76789-kpw4q" event={"ID":"1cb95852-b3c5-4b1a-80bf-9d3fa465084a","Type":"ContainerStarted","Data":"67888c1d34c01a80b78c92bd45a07e99d07c146b012a128fcc4ce7b6dec2e827"} Jul 06 13:52:41.408579 ip-10-0-136-25 kubenswrapper[2569]: I0706 13:52:41.408500 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/router-default-966c76789-kpw4q" event={"ID":"1cb95852-b3c5-4b1a-80bf-9d3fa465084a","Type":"ContainerStarted","Data":"b80c2225c3c5def8e4966cba4fddc783ee32ebbf4fcfe2ea7320c8b29364905c"} Jul 06 13:52:41.428190 ip-10-0-136-25 kubenswrapper[2569]: I0706 13:52:41.428110 2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/router-default-966c76789-kpw4q" podStartSLOduration=33.428096073 podStartE2EDuration="33.428096073s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-06 13:52:08 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 13:52:41.427564184 +0000 UTC m=+137.018125962" watchObservedRunningTime="2026-07-06 13:52:41.428096073 +0000 UTC m=+137.018657849" Jul 06 13:52:41.827474 ip-10-0-136-25 kubenswrapper[2569]: I0706 13:52:41.827441 2569 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/router-default-966c76789-kpw4q" Jul 06 13:52:41.830315 ip-10-0-136-25 kubenswrapper[2569]: I0706 13:52:41.830290 2569 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/router-default-966c76789-kpw4q" Jul 06 13:52:42.412773 ip-10-0-136-25 kubenswrapper[2569]: I0706 13:52:42.412738 2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/router-default-966c76789-kpw4q" Jul 06 13:52:42.414153 ip-10-0-136-25 kubenswrapper[2569]: I0706 13:52:42.414125 2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/router-default-966c76789-kpw4q" Jul 06 13:52:42.647528 ip-10-0-136-25 kubenswrapper[2569]: I0706 13:52:42.647477 2569 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-network-console/networking-console-plugin-6bffdf9778-99974"] Jul 06 13:52:42.651651 ip-10-0-136-25 kubenswrapper[2569]: I0706 13:52:42.651626 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-console/networking-console-plugin-6bffdf9778-99974" Jul 06 13:52:42.653820 ip-10-0-136-25 kubenswrapper[2569]: I0706 13:52:42.653801 2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-console\"/\"default-dockercfg-d2f2x\"" Jul 06 13:52:42.653946 ip-10-0-136-25 kubenswrapper[2569]: I0706 13:52:42.653929 2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-console\"/\"networking-console-plugin-cert\"" Jul 06 13:52:42.654002 ip-10-0-136-25 kubenswrapper[2569]: I0706 13:52:42.653941 2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-console\"/\"networking-console-plugin\"" Jul 06 13:52:42.658048 ip-10-0-136-25 kubenswrapper[2569]: I0706 13:52:42.658025 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-console/networking-console-plugin-6bffdf9778-99974"] Jul 06 13:52:42.786387 ip-10-0-136-25 kubenswrapper[2569]: I0706 13:52:42.786356 2569 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-image-registry/image-registry-7c57596984-r6bgq"] Jul 06 13:52:42.790777 ip-10-0-136-25 kubenswrapper[2569]: I0706 13:52:42.790761 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-7c57596984-r6bgq" Jul 06 13:52:42.792806 ip-10-0-136-25 kubenswrapper[2569]: I0706 13:52:42.792783 2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"installation-pull-secrets\"" Jul 06 13:52:42.793002 ip-10-0-136-25 kubenswrapper[2569]: I0706 13:52:42.792838 2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-tls\"" Jul 06 13:52:42.793002 ip-10-0-136-25 kubenswrapper[2569]: I0706 13:52:42.792857 2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-private-configuration\"" Jul 06 13:52:42.793125 ip-10-0-136-25 kubenswrapper[2569]: I0706 13:52:42.793111 2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"registry-dockercfg-m9r9h\"" Jul 06 13:52:42.798626 ip-10-0-136-25 kubenswrapper[2569]: I0706 13:52:42.798607 2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"trusted-ca\"" Jul 06 13:52:42.804627 ip-10-0-136-25 kubenswrapper[2569]: I0706 13:52:42.804604 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-7c57596984-r6bgq"] Jul 06 13:52:42.837154 ip-10-0-136-25 kubenswrapper[2569]: I0706 13:52:42.837133 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/c198bd10-43bf-493b-9aa0-8f3dad5f8b59-nginx-conf\") pod \"networking-console-plugin-6bffdf9778-99974\" (UID: \"c198bd10-43bf-493b-9aa0-8f3dad5f8b59\") " pod="openshift-network-console/networking-console-plugin-6bffdf9778-99974" Jul 06 13:52:42.837438 ip-10-0-136-25 kubenswrapper[2569]: I0706 13:52:42.837176 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/c198bd10-43bf-493b-9aa0-8f3dad5f8b59-networking-console-plugin-cert\") pod \"networking-console-plugin-6bffdf9778-99974\" (UID: \"c198bd10-43bf-493b-9aa0-8f3dad5f8b59\") " pod="openshift-network-console/networking-console-plugin-6bffdf9778-99974" Jul 06 13:52:42.937580 ip-10-0-136-25 kubenswrapper[2569]: I0706 13:52:42.937550 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/83120e7b-a163-4061-a1c9-4b6d79ac47c5-installation-pull-secrets\") pod \"image-registry-7c57596984-r6bgq\" (UID: \"83120e7b-a163-4061-a1c9-4b6d79ac47c5\") " pod="openshift-image-registry/image-registry-7c57596984-r6bgq" Jul 06 13:52:42.937762 ip-10-0-136-25 kubenswrapper[2569]: I0706 13:52:42.937592 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/83120e7b-a163-4061-a1c9-4b6d79ac47c5-trusted-ca\") pod \"image-registry-7c57596984-r6bgq\" (UID: \"83120e7b-a163-4061-a1c9-4b6d79ac47c5\") " pod="openshift-image-registry/image-registry-7c57596984-r6bgq" Jul 06 13:52:42.937762 ip-10-0-136-25 kubenswrapper[2569]: I0706 13:52:42.937650 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/c198bd10-43bf-493b-9aa0-8f3dad5f8b59-networking-console-plugin-cert\") pod \"networking-console-plugin-6bffdf9778-99974\" (UID: \"c198bd10-43bf-493b-9aa0-8f3dad5f8b59\") " pod="openshift-network-console/networking-console-plugin-6bffdf9778-99974" Jul 06 13:52:42.937762 ip-10-0-136-25 kubenswrapper[2569]: I0706 13:52:42.937706 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/83120e7b-a163-4061-a1c9-4b6d79ac47c5-ca-trust-extracted\") pod \"image-registry-7c57596984-r6bgq\" (UID: \"83120e7b-a163-4061-a1c9-4b6d79ac47c5\") " pod="openshift-image-registry/image-registry-7c57596984-r6bgq" Jul 06 13:52:42.937762 ip-10-0-136-25 kubenswrapper[2569]: I0706 13:52:42.937738 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/83120e7b-a163-4061-a1c9-4b6d79ac47c5-bound-sa-token\") pod \"image-registry-7c57596984-r6bgq\" (UID: \"83120e7b-a163-4061-a1c9-4b6d79ac47c5\") " pod="openshift-image-registry/image-registry-7c57596984-r6bgq" Jul 06 13:52:42.937762 ip-10-0-136-25 kubenswrapper[2569]: I0706 13:52:42.937761 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/83120e7b-a163-4061-a1c9-4b6d79ac47c5-registry-certificates\") pod \"image-registry-7c57596984-r6bgq\" (UID: \"83120e7b-a163-4061-a1c9-4b6d79ac47c5\") " pod="openshift-image-registry/image-registry-7c57596984-r6bgq" Jul 06 13:52:42.937982 ip-10-0-136-25 kubenswrapper[2569]: I0706 13:52:42.937817 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/83120e7b-a163-4061-a1c9-4b6d79ac47c5-image-registry-private-configuration\") pod \"image-registry-7c57596984-r6bgq\" (UID: \"83120e7b-a163-4061-a1c9-4b6d79ac47c5\") " pod="openshift-image-registry/image-registry-7c57596984-r6bgq" Jul 06 13:52:42.937982 ip-10-0-136-25 kubenswrapper[2569]: I0706 13:52:42.937863 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/83120e7b-a163-4061-a1c9-4b6d79ac47c5-registry-tls\") pod \"image-registry-7c57596984-r6bgq\" (UID: \"83120e7b-a163-4061-a1c9-4b6d79ac47c5\") " pod="openshift-image-registry/image-registry-7c57596984-r6bgq" Jul 06 13:52:42.937982 ip-10-0-136-25 kubenswrapper[2569]: I0706 13:52:42.937924 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/c198bd10-43bf-493b-9aa0-8f3dad5f8b59-nginx-conf\") pod \"networking-console-plugin-6bffdf9778-99974\" (UID: \"c198bd10-43bf-493b-9aa0-8f3dad5f8b59\") " pod="openshift-network-console/networking-console-plugin-6bffdf9778-99974" Jul 06 13:52:42.937982 ip-10-0-136-25 kubenswrapper[2569]: I0706 13:52:42.937969 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vvsrf\" (UniqueName: \"kubernetes.io/projected/83120e7b-a163-4061-a1c9-4b6d79ac47c5-kube-api-access-vvsrf\") pod \"image-registry-7c57596984-r6bgq\" (UID: \"83120e7b-a163-4061-a1c9-4b6d79ac47c5\") " pod="openshift-image-registry/image-registry-7c57596984-r6bgq" Jul 06 13:52:42.938513 ip-10-0-136-25 kubenswrapper[2569]: I0706 13:52:42.938498 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/c198bd10-43bf-493b-9aa0-8f3dad5f8b59-nginx-conf\") pod \"networking-console-plugin-6bffdf9778-99974\" (UID: \"c198bd10-43bf-493b-9aa0-8f3dad5f8b59\") " pod="openshift-network-console/networking-console-plugin-6bffdf9778-99974" Jul 06 13:52:42.940038 ip-10-0-136-25 kubenswrapper[2569]: I0706 13:52:42.940021 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/c198bd10-43bf-493b-9aa0-8f3dad5f8b59-networking-console-plugin-cert\") pod \"networking-console-plugin-6bffdf9778-99974\" (UID: \"c198bd10-43bf-493b-9aa0-8f3dad5f8b59\") " pod="openshift-network-console/networking-console-plugin-6bffdf9778-99974" Jul 06 13:52:42.962109 ip-10-0-136-25 kubenswrapper[2569]: I0706 13:52:42.962071 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-console/networking-console-plugin-6bffdf9778-99974" Jul 06 13:52:43.038760 ip-10-0-136-25 kubenswrapper[2569]: I0706 13:52:43.038729 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/83120e7b-a163-4061-a1c9-4b6d79ac47c5-bound-sa-token\") pod \"image-registry-7c57596984-r6bgq\" (UID: \"83120e7b-a163-4061-a1c9-4b6d79ac47c5\") " pod="openshift-image-registry/image-registry-7c57596984-r6bgq" Jul 06 13:52:43.038760 ip-10-0-136-25 kubenswrapper[2569]: I0706 13:52:43.038769 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/83120e7b-a163-4061-a1c9-4b6d79ac47c5-registry-certificates\") pod \"image-registry-7c57596984-r6bgq\" (UID: \"83120e7b-a163-4061-a1c9-4b6d79ac47c5\") " pod="openshift-image-registry/image-registry-7c57596984-r6bgq" Jul 06 13:52:43.038982 ip-10-0-136-25 kubenswrapper[2569]: I0706 13:52:43.038795 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/83120e7b-a163-4061-a1c9-4b6d79ac47c5-image-registry-private-configuration\") pod \"image-registry-7c57596984-r6bgq\" (UID: \"83120e7b-a163-4061-a1c9-4b6d79ac47c5\") " pod="openshift-image-registry/image-registry-7c57596984-r6bgq" Jul 06 13:52:43.038982 ip-10-0-136-25 kubenswrapper[2569]: I0706 13:52:43.038832 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/83120e7b-a163-4061-a1c9-4b6d79ac47c5-registry-tls\") pod \"image-registry-7c57596984-r6bgq\" (UID: \"83120e7b-a163-4061-a1c9-4b6d79ac47c5\") " pod="openshift-image-registry/image-registry-7c57596984-r6bgq" Jul 06 13:52:43.038982 ip-10-0-136-25 kubenswrapper[2569]: I0706 13:52:43.038885 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vvsrf\" (UniqueName: \"kubernetes.io/projected/83120e7b-a163-4061-a1c9-4b6d79ac47c5-kube-api-access-vvsrf\") pod \"image-registry-7c57596984-r6bgq\" (UID: \"83120e7b-a163-4061-a1c9-4b6d79ac47c5\") " pod="openshift-image-registry/image-registry-7c57596984-r6bgq" Jul 06 13:52:43.038982 ip-10-0-136-25 kubenswrapper[2569]: I0706 13:52:43.038918 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/83120e7b-a163-4061-a1c9-4b6d79ac47c5-installation-pull-secrets\") pod \"image-registry-7c57596984-r6bgq\" (UID: \"83120e7b-a163-4061-a1c9-4b6d79ac47c5\") " pod="openshift-image-registry/image-registry-7c57596984-r6bgq" Jul 06 13:52:43.038982 ip-10-0-136-25 kubenswrapper[2569]: I0706 13:52:43.038949 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/83120e7b-a163-4061-a1c9-4b6d79ac47c5-trusted-ca\") pod \"image-registry-7c57596984-r6bgq\" (UID: \"83120e7b-a163-4061-a1c9-4b6d79ac47c5\") " pod="openshift-image-registry/image-registry-7c57596984-r6bgq" Jul 06 13:52:43.039254 ip-10-0-136-25 kubenswrapper[2569]: I0706 13:52:43.039013 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/83120e7b-a163-4061-a1c9-4b6d79ac47c5-ca-trust-extracted\") pod \"image-registry-7c57596984-r6bgq\" (UID: \"83120e7b-a163-4061-a1c9-4b6d79ac47c5\") " pod="openshift-image-registry/image-registry-7c57596984-r6bgq" Jul 06 13:52:43.039535 ip-10-0-136-25 kubenswrapper[2569]: I0706 13:52:43.039510 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/83120e7b-a163-4061-a1c9-4b6d79ac47c5-ca-trust-extracted\") pod \"image-registry-7c57596984-r6bgq\" (UID: \"83120e7b-a163-4061-a1c9-4b6d79ac47c5\") " pod="openshift-image-registry/image-registry-7c57596984-r6bgq" Jul 06 13:52:43.039664 ip-10-0-136-25 kubenswrapper[2569]: I0706 13:52:43.039642 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/83120e7b-a163-4061-a1c9-4b6d79ac47c5-registry-certificates\") pod \"image-registry-7c57596984-r6bgq\" (UID: \"83120e7b-a163-4061-a1c9-4b6d79ac47c5\") " pod="openshift-image-registry/image-registry-7c57596984-r6bgq" Jul 06 13:52:43.040446 ip-10-0-136-25 kubenswrapper[2569]: I0706 13:52:43.040419 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/83120e7b-a163-4061-a1c9-4b6d79ac47c5-trusted-ca\") pod \"image-registry-7c57596984-r6bgq\" (UID: \"83120e7b-a163-4061-a1c9-4b6d79ac47c5\") " pod="openshift-image-registry/image-registry-7c57596984-r6bgq" Jul 06 13:52:43.041813 ip-10-0-136-25 kubenswrapper[2569]: I0706 13:52:43.041789 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/83120e7b-a163-4061-a1c9-4b6d79ac47c5-image-registry-private-configuration\") pod \"image-registry-7c57596984-r6bgq\" (UID: \"83120e7b-a163-4061-a1c9-4b6d79ac47c5\") " pod="openshift-image-registry/image-registry-7c57596984-r6bgq" Jul 06 13:52:43.041905 ip-10-0-136-25 kubenswrapper[2569]: I0706 13:52:43.041838 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/83120e7b-a163-4061-a1c9-4b6d79ac47c5-registry-tls\") pod \"image-registry-7c57596984-r6bgq\" (UID: \"83120e7b-a163-4061-a1c9-4b6d79ac47c5\") " pod="openshift-image-registry/image-registry-7c57596984-r6bgq" Jul 06 13:52:43.042153 ip-10-0-136-25 kubenswrapper[2569]: I0706 13:52:43.042135 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/83120e7b-a163-4061-a1c9-4b6d79ac47c5-installation-pull-secrets\") pod \"image-registry-7c57596984-r6bgq\" (UID: \"83120e7b-a163-4061-a1c9-4b6d79ac47c5\") " pod="openshift-image-registry/image-registry-7c57596984-r6bgq" Jul 06 13:52:43.049469 ip-10-0-136-25 kubenswrapper[2569]: I0706 13:52:43.049287 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/83120e7b-a163-4061-a1c9-4b6d79ac47c5-bound-sa-token\") pod \"image-registry-7c57596984-r6bgq\" (UID: \"83120e7b-a163-4061-a1c9-4b6d79ac47c5\") " pod="openshift-image-registry/image-registry-7c57596984-r6bgq" Jul 06 13:52:43.050724 ip-10-0-136-25 kubenswrapper[2569]: I0706 13:52:43.049686 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vvsrf\" (UniqueName: \"kubernetes.io/projected/83120e7b-a163-4061-a1c9-4b6d79ac47c5-kube-api-access-vvsrf\") pod \"image-registry-7c57596984-r6bgq\" (UID: \"83120e7b-a163-4061-a1c9-4b6d79ac47c5\") " pod="openshift-image-registry/image-registry-7c57596984-r6bgq" Jul 06 13:52:43.099701 ip-10-0-136-25 kubenswrapper[2569]: I0706 13:52:43.099659 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-7c57596984-r6bgq" Jul 06 13:52:43.124543 ip-10-0-136-25 kubenswrapper[2569]: I0706 13:52:43.124510 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-console/networking-console-plugin-6bffdf9778-99974"] Jul 06 13:52:43.127799 ip-10-0-136-25 kubenswrapper[2569]: W0706 13:52:43.127767 2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc198bd10_43bf_493b_9aa0_8f3dad5f8b59.slice/crio-0949f69728bc5ba3b44431ce9137aade696dc3fcc1489652065d4c3bb42917dd WatchSource:0}: Error finding container 0949f69728bc5ba3b44431ce9137aade696dc3fcc1489652065d4c3bb42917dd: Status 404 returned error can't find the container with id 0949f69728bc5ba3b44431ce9137aade696dc3fcc1489652065d4c3bb42917dd Jul 06 13:52:43.265008 ip-10-0-136-25 kubenswrapper[2569]: I0706 13:52:43.264945 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-7c57596984-r6bgq"] Jul 06 13:52:43.268764 ip-10-0-136-25 kubenswrapper[2569]: W0706 13:52:43.268740 2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod83120e7b_a163_4061_a1c9_4b6d79ac47c5.slice/crio-9653d6946b7f957214edd74a7eea619c2ba78ffb4ac32ed206da6c9298c748a2 WatchSource:0}: Error finding container 9653d6946b7f957214edd74a7eea619c2ba78ffb4ac32ed206da6c9298c748a2: Status 404 returned error can't find the container with id 9653d6946b7f957214edd74a7eea619c2ba78ffb4ac32ed206da6c9298c748a2 Jul 06 13:52:43.417255 ip-10-0-136-25 kubenswrapper[2569]: I0706 13:52:43.417194 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/cluster-monitoring-operator-64786d9cbb-zpkk5" event={"ID":"a52c7279-3809-4f22-9001-afb7325cfe8d","Type":"ContainerStarted","Data":"29b01ef29576ab0c75543fc5cf98e6fadd510526e864b1f031fe2d031abe782d"} Jul 06 13:52:43.418777 ip-10-0-136-25 kubenswrapper[2569]: I0706 13:52:43.418752 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-7c57596984-r6bgq" event={"ID":"83120e7b-a163-4061-a1c9-4b6d79ac47c5","Type":"ContainerStarted","Data":"196dd21ed25914522faad13568ffa16ca72fa6767202f537ebb627bfdcd7cdea"} Jul 06 13:52:43.418925 ip-10-0-136-25 kubenswrapper[2569]: I0706 13:52:43.418795 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-7c57596984-r6bgq" event={"ID":"83120e7b-a163-4061-a1c9-4b6d79ac47c5","Type":"ContainerStarted","Data":"9653d6946b7f957214edd74a7eea619c2ba78ffb4ac32ed206da6c9298c748a2"} Jul 06 13:52:43.418925 ip-10-0-136-25 kubenswrapper[2569]: I0706 13:52:43.418835 2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-image-registry/image-registry-7c57596984-r6bgq" Jul 06 13:52:43.419943 ip-10-0-136-25 kubenswrapper[2569]: I0706 13:52:43.419918 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-console/networking-console-plugin-6bffdf9778-99974" event={"ID":"c198bd10-43bf-493b-9aa0-8f3dad5f8b59","Type":"ContainerStarted","Data":"0949f69728bc5ba3b44431ce9137aade696dc3fcc1489652065d4c3bb42917dd"} Jul 06 13:52:43.433010 ip-10-0-136-25 kubenswrapper[2569]: I0706 13:52:43.432959 2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/cluster-monitoring-operator-64786d9cbb-zpkk5" podStartSLOduration=33.601567664 podStartE2EDuration="35.432941988s" totalImagesPullingTime="1.831374324s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-06 13:52:08 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 13:52:43.431973317 +0000 UTC m=+139.022535094" watchObservedRunningTime="2026-07-06 13:52:43.432941988 +0000 UTC m=+139.023503765" Jul 06 13:52:43.452405 ip-10-0-136-25 kubenswrapper[2569]: I0706 13:52:43.452358 2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/image-registry-7c57596984-r6bgq" podStartSLOduration=1.452342464 podStartE2EDuration="1.452342464s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-06 13:52:42 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 13:52:43.451123326 +0000 UTC m=+139.041685104" watchObservedRunningTime="2026-07-06 13:52:43.452342464 +0000 UTC m=+139.042904224" Jul 06 13:52:45.055645 ip-10-0-136-25 kubenswrapper[2569]: I0706 13:52:45.055607 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/3c279543-7a20-4d29-9de1-6cd09f14f692-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-kc97f\" (UID: \"3c279543-7a20-4d29-9de1-6cd09f14f692\") " pod="openshift-insights/insights-runtime-extractor-kc97f" Jul 06 13:52:45.057902 ip-10-0-136-25 kubenswrapper[2569]: I0706 13:52:45.057882 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/3c279543-7a20-4d29-9de1-6cd09f14f692-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-kc97f\" (UID: \"3c279543-7a20-4d29-9de1-6cd09f14f692\") " pod="openshift-insights/insights-runtime-extractor-kc97f" Jul 06 13:52:45.299008 ip-10-0-136-25 kubenswrapper[2569]: I0706 13:52:45.298975 2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-cwmvs\"" Jul 06 13:52:45.307710 ip-10-0-136-25 kubenswrapper[2569]: I0706 13:52:45.307657 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-kc97f" Jul 06 13:52:45.431001 ip-10-0-136-25 kubenswrapper[2569]: I0706 13:52:45.430957 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-console/networking-console-plugin-6bffdf9778-99974" event={"ID":"c198bd10-43bf-493b-9aa0-8f3dad5f8b59","Type":"ContainerStarted","Data":"e3f047f1f797c92fbf9bd7fdcf9ea96d648dba72386ce44c78f20f78e6f6b7fb"} Jul 06 13:52:45.434769 ip-10-0-136-25 kubenswrapper[2569]: I0706 13:52:45.434739 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-kc97f"] Jul 06 13:52:45.438338 ip-10-0-136-25 kubenswrapper[2569]: W0706 13:52:45.438311 2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3c279543_7a20_4d29_9de1_6cd09f14f692.slice/crio-11a50cf4c44c6e0d5d3799015baade213b7a976087e296f09f12b2c933ea5ce8 WatchSource:0}: Error finding container 11a50cf4c44c6e0d5d3799015baade213b7a976087e296f09f12b2c933ea5ce8: Status 404 returned error can't find the container with id 11a50cf4c44c6e0d5d3799015baade213b7a976087e296f09f12b2c933ea5ce8 Jul 06 13:52:45.452309 ip-10-0-136-25 kubenswrapper[2569]: I0706 13:52:45.452261 2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-console/networking-console-plugin-6bffdf9778-99974" podStartSLOduration=1.9566178330000001 podStartE2EDuration="3.452246643s" totalImagesPullingTime="1.49562881s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-06 13:52:42 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 13:52:45.451609239 +0000 UTC m=+141.042171065" watchObservedRunningTime="2026-07-06 13:52:45.452246643 +0000 UTC m=+141.042808419" Jul 06 13:52:46.436621 ip-10-0-136-25 kubenswrapper[2569]: I0706 13:52:46.436562 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-kc97f" event={"ID":"3c279543-7a20-4d29-9de1-6cd09f14f692","Type":"ContainerStarted","Data":"11a50cf4c44c6e0d5d3799015baade213b7a976087e296f09f12b2c933ea5ce8"} Jul 06 13:52:49.448595 ip-10-0-136-25 kubenswrapper[2569]: I0706 13:52:49.448513 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-kc97f" event={"ID":"3c279543-7a20-4d29-9de1-6cd09f14f692","Type":"ContainerStarted","Data":"b1c00795e130e5040b8a0aef317b9f6286832cd67439078596ac1d444513734f"} Jul 06 13:52:50.454984 ip-10-0-136-25 kubenswrapper[2569]: I0706 13:52:50.454946 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-kc97f" event={"ID":"3c279543-7a20-4d29-9de1-6cd09f14f692","Type":"ContainerStarted","Data":"0e70a51590cdc3f7198033274ab28fc41aa0b5039d8b4c92771e7b3a321acbea"} Jul 06 13:52:52.051583 ip-10-0-136-25 kubenswrapper[2569]: I0706 13:52:52.051554 2569 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/openshift-state-metrics-54cf96dfcf-n4mw9"] Jul 06 13:52:52.065070 ip-10-0-136-25 kubenswrapper[2569]: I0706 13:52:52.065029 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/openshift-state-metrics-54cf96dfcf-n4mw9" Jul 06 13:52:52.068965 ip-10-0-136-25 kubenswrapper[2569]: I0706 13:52:52.068924 2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-kube-rbac-proxy-config\"" Jul 06 13:52:52.069621 ip-10-0-136-25 kubenswrapper[2569]: I0706 13:52:52.069441 2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\"" Jul 06 13:52:52.069621 ip-10-0-136-25 kubenswrapper[2569]: I0706 13:52:52.069450 2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-dockercfg-pppkn\"" Jul 06 13:52:52.070025 ip-10-0-136-25 kubenswrapper[2569]: I0706 13:52:52.069992 2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-tls\"" Jul 06 13:52:52.072465 ip-10-0-136-25 kubenswrapper[2569]: I0706 13:52:52.072384 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/openshift-state-metrics-54cf96dfcf-n4mw9"] Jul 06 13:52:52.111568 ip-10-0-136-25 kubenswrapper[2569]: I0706 13:52:52.111521 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/dc28734e-3746-4350-9fc7-4bcc1b9e842d-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-54cf96dfcf-n4mw9\" (UID: \"dc28734e-3746-4350-9fc7-4bcc1b9e842d\") " pod="openshift-monitoring/openshift-state-metrics-54cf96dfcf-n4mw9" Jul 06 13:52:52.111757 ip-10-0-136-25 kubenswrapper[2569]: I0706 13:52:52.111592 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/dc28734e-3746-4350-9fc7-4bcc1b9e842d-metrics-client-ca\") pod \"openshift-state-metrics-54cf96dfcf-n4mw9\" (UID: \"dc28734e-3746-4350-9fc7-4bcc1b9e842d\") " pod="openshift-monitoring/openshift-state-metrics-54cf96dfcf-n4mw9" Jul 06 13:52:52.111757 ip-10-0-136-25 kubenswrapper[2569]: I0706 13:52:52.111630 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-h58l9\" (UniqueName: \"kubernetes.io/projected/dc28734e-3746-4350-9fc7-4bcc1b9e842d-kube-api-access-h58l9\") pod \"openshift-state-metrics-54cf96dfcf-n4mw9\" (UID: \"dc28734e-3746-4350-9fc7-4bcc1b9e842d\") " pod="openshift-monitoring/openshift-state-metrics-54cf96dfcf-n4mw9" Jul 06 13:52:52.111757 ip-10-0-136-25 kubenswrapper[2569]: I0706 13:52:52.111660 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/dc28734e-3746-4350-9fc7-4bcc1b9e842d-openshift-state-metrics-tls\") pod \"openshift-state-metrics-54cf96dfcf-n4mw9\" (UID: \"dc28734e-3746-4350-9fc7-4bcc1b9e842d\") " pod="openshift-monitoring/openshift-state-metrics-54cf96dfcf-n4mw9" Jul 06 13:52:52.128800 ip-10-0-136-25 kubenswrapper[2569]: I0706 13:52:52.128761 2569 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/node-exporter-7fj7f"] Jul 06 13:52:52.135201 ip-10-0-136-25 kubenswrapper[2569]: I0706 13:52:52.135170 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-7fj7f" Jul 06 13:52:52.138102 ip-10-0-136-25 kubenswrapper[2569]: I0706 13:52:52.138060 2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\"" Jul 06 13:52:52.138278 ip-10-0-136-25 kubenswrapper[2569]: I0706 13:52:52.138154 2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\"" Jul 06 13:52:52.138278 ip-10-0-136-25 kubenswrapper[2569]: I0706 13:52:52.138183 2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\"" Jul 06 13:52:52.138278 ip-10-0-136-25 kubenswrapper[2569]: I0706 13:52:52.138178 2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-qhstm\"" Jul 06 13:52:52.212494 ip-10-0-136-25 kubenswrapper[2569]: I0706 13:52:52.212443 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/f906ef93-d6ed-491e-b7ad-3622d7398b8e-metrics-client-ca\") pod \"node-exporter-7fj7f\" (UID: \"f906ef93-d6ed-491e-b7ad-3622d7398b8e\") " pod="openshift-monitoring/node-exporter-7fj7f" Jul 06 13:52:52.212681 ip-10-0-136-25 kubenswrapper[2569]: I0706 13:52:52.212505 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/f906ef93-d6ed-491e-b7ad-3622d7398b8e-node-exporter-tls\") pod \"node-exporter-7fj7f\" (UID: \"f906ef93-d6ed-491e-b7ad-3622d7398b8e\") " pod="openshift-monitoring/node-exporter-7fj7f" Jul 06 13:52:52.212681 ip-10-0-136-25 kubenswrapper[2569]: I0706 13:52:52.212535 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/f906ef93-d6ed-491e-b7ad-3622d7398b8e-sys\") pod \"node-exporter-7fj7f\" (UID: \"f906ef93-d6ed-491e-b7ad-3622d7398b8e\") " pod="openshift-monitoring/node-exporter-7fj7f" Jul 06 13:52:52.212681 ip-10-0-136-25 kubenswrapper[2569]: I0706 13:52:52.212578 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/dc28734e-3746-4350-9fc7-4bcc1b9e842d-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-54cf96dfcf-n4mw9\" (UID: \"dc28734e-3746-4350-9fc7-4bcc1b9e842d\") " pod="openshift-monitoring/openshift-state-metrics-54cf96dfcf-n4mw9" Jul 06 13:52:52.212681 ip-10-0-136-25 kubenswrapper[2569]: I0706 13:52:52.212603 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/f906ef93-d6ed-491e-b7ad-3622d7398b8e-root\") pod \"node-exporter-7fj7f\" (UID: \"f906ef93-d6ed-491e-b7ad-3622d7398b8e\") " pod="openshift-monitoring/node-exporter-7fj7f" Jul 06 13:52:52.212681 ip-10-0-136-25 kubenswrapper[2569]: I0706 13:52:52.212623 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/f906ef93-d6ed-491e-b7ad-3622d7398b8e-node-exporter-wtmp\") pod \"node-exporter-7fj7f\" (UID: \"f906ef93-d6ed-491e-b7ad-3622d7398b8e\") " pod="openshift-monitoring/node-exporter-7fj7f" Jul 06 13:52:52.212681 ip-10-0-136-25 kubenswrapper[2569]: I0706 13:52:52.212650 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/f906ef93-d6ed-491e-b7ad-3622d7398b8e-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-7fj7f\" (UID: \"f906ef93-d6ed-491e-b7ad-3622d7398b8e\") " pod="openshift-monitoring/node-exporter-7fj7f" Jul 06 13:52:52.212681 ip-10-0-136-25 kubenswrapper[2569]: I0706 13:52:52.212676 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/f906ef93-d6ed-491e-b7ad-3622d7398b8e-node-exporter-accelerators-collector-config\") pod \"node-exporter-7fj7f\" (UID: \"f906ef93-d6ed-491e-b7ad-3622d7398b8e\") " pod="openshift-monitoring/node-exporter-7fj7f" Jul 06 13:52:52.213069 ip-10-0-136-25 kubenswrapper[2569]: I0706 13:52:52.212704 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-p8kjn\" (UniqueName: \"kubernetes.io/projected/f906ef93-d6ed-491e-b7ad-3622d7398b8e-kube-api-access-p8kjn\") pod \"node-exporter-7fj7f\" (UID: \"f906ef93-d6ed-491e-b7ad-3622d7398b8e\") " pod="openshift-monitoring/node-exporter-7fj7f" Jul 06 13:52:52.213069 ip-10-0-136-25 kubenswrapper[2569]: I0706 13:52:52.212733 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/dc28734e-3746-4350-9fc7-4bcc1b9e842d-metrics-client-ca\") pod \"openshift-state-metrics-54cf96dfcf-n4mw9\" (UID: \"dc28734e-3746-4350-9fc7-4bcc1b9e842d\") " pod="openshift-monitoring/openshift-state-metrics-54cf96dfcf-n4mw9" Jul 06 13:52:52.213069 ip-10-0-136-25 kubenswrapper[2569]: I0706 13:52:52.212756 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h58l9\" (UniqueName: \"kubernetes.io/projected/dc28734e-3746-4350-9fc7-4bcc1b9e842d-kube-api-access-h58l9\") pod \"openshift-state-metrics-54cf96dfcf-n4mw9\" (UID: \"dc28734e-3746-4350-9fc7-4bcc1b9e842d\") " pod="openshift-monitoring/openshift-state-metrics-54cf96dfcf-n4mw9" Jul 06 13:52:52.213069 ip-10-0-136-25 kubenswrapper[2569]: I0706 13:52:52.212811 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/dc28734e-3746-4350-9fc7-4bcc1b9e842d-openshift-state-metrics-tls\") pod \"openshift-state-metrics-54cf96dfcf-n4mw9\" (UID: \"dc28734e-3746-4350-9fc7-4bcc1b9e842d\") " pod="openshift-monitoring/openshift-state-metrics-54cf96dfcf-n4mw9" Jul 06 13:52:52.213069 ip-10-0-136-25 kubenswrapper[2569]: I0706 13:52:52.212885 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/f906ef93-d6ed-491e-b7ad-3622d7398b8e-node-exporter-textfile\") pod \"node-exporter-7fj7f\" (UID: \"f906ef93-d6ed-491e-b7ad-3622d7398b8e\") " pod="openshift-monitoring/node-exporter-7fj7f" Jul 06 13:52:52.213948 ip-10-0-136-25 kubenswrapper[2569]: I0706 13:52:52.213921 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/dc28734e-3746-4350-9fc7-4bcc1b9e842d-metrics-client-ca\") pod \"openshift-state-metrics-54cf96dfcf-n4mw9\" (UID: \"dc28734e-3746-4350-9fc7-4bcc1b9e842d\") " pod="openshift-monitoring/openshift-state-metrics-54cf96dfcf-n4mw9" Jul 06 13:52:52.215438 ip-10-0-136-25 kubenswrapper[2569]: I0706 13:52:52.215416 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/dc28734e-3746-4350-9fc7-4bcc1b9e842d-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-54cf96dfcf-n4mw9\" (UID: \"dc28734e-3746-4350-9fc7-4bcc1b9e842d\") " pod="openshift-monitoring/openshift-state-metrics-54cf96dfcf-n4mw9" Jul 06 13:52:52.215636 ip-10-0-136-25 kubenswrapper[2569]: I0706 13:52:52.215610 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/dc28734e-3746-4350-9fc7-4bcc1b9e842d-openshift-state-metrics-tls\") pod \"openshift-state-metrics-54cf96dfcf-n4mw9\" (UID: \"dc28734e-3746-4350-9fc7-4bcc1b9e842d\") " pod="openshift-monitoring/openshift-state-metrics-54cf96dfcf-n4mw9" Jul 06 13:52:52.223374 ip-10-0-136-25 kubenswrapper[2569]: I0706 13:52:52.223348 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-h58l9\" (UniqueName: \"kubernetes.io/projected/dc28734e-3746-4350-9fc7-4bcc1b9e842d-kube-api-access-h58l9\") pod \"openshift-state-metrics-54cf96dfcf-n4mw9\" (UID: \"dc28734e-3746-4350-9fc7-4bcc1b9e842d\") " pod="openshift-monitoring/openshift-state-metrics-54cf96dfcf-n4mw9" Jul 06 13:52:52.313657 ip-10-0-136-25 kubenswrapper[2569]: I0706 13:52:52.313567 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/f906ef93-d6ed-491e-b7ad-3622d7398b8e-node-exporter-textfile\") pod \"node-exporter-7fj7f\" (UID: \"f906ef93-d6ed-491e-b7ad-3622d7398b8e\") " pod="openshift-monitoring/node-exporter-7fj7f" Jul 06 13:52:52.313657 ip-10-0-136-25 kubenswrapper[2569]: I0706 13:52:52.313622 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/f906ef93-d6ed-491e-b7ad-3622d7398b8e-metrics-client-ca\") pod \"node-exporter-7fj7f\" (UID: \"f906ef93-d6ed-491e-b7ad-3622d7398b8e\") " pod="openshift-monitoring/node-exporter-7fj7f" Jul 06 13:52:52.313657 ip-10-0-136-25 kubenswrapper[2569]: I0706 13:52:52.313648 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/f906ef93-d6ed-491e-b7ad-3622d7398b8e-node-exporter-tls\") pod \"node-exporter-7fj7f\" (UID: \"f906ef93-d6ed-491e-b7ad-3622d7398b8e\") " pod="openshift-monitoring/node-exporter-7fj7f" Jul 06 13:52:52.313944 ip-10-0-136-25 kubenswrapper[2569]: I0706 13:52:52.313677 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/f906ef93-d6ed-491e-b7ad-3622d7398b8e-sys\") pod \"node-exporter-7fj7f\" (UID: \"f906ef93-d6ed-491e-b7ad-3622d7398b8e\") " pod="openshift-monitoring/node-exporter-7fj7f" Jul 06 13:52:52.313944 ip-10-0-136-25 kubenswrapper[2569]: I0706 13:52:52.313714 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/f906ef93-d6ed-491e-b7ad-3622d7398b8e-root\") pod \"node-exporter-7fj7f\" (UID: \"f906ef93-d6ed-491e-b7ad-3622d7398b8e\") " pod="openshift-monitoring/node-exporter-7fj7f" Jul 06 13:52:52.313944 ip-10-0-136-25 kubenswrapper[2569]: I0706 13:52:52.313737 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/f906ef93-d6ed-491e-b7ad-3622d7398b8e-node-exporter-wtmp\") pod \"node-exporter-7fj7f\" (UID: \"f906ef93-d6ed-491e-b7ad-3622d7398b8e\") " pod="openshift-monitoring/node-exporter-7fj7f" Jul 06 13:52:52.313944 ip-10-0-136-25 kubenswrapper[2569]: I0706 13:52:52.313760 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/f906ef93-d6ed-491e-b7ad-3622d7398b8e-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-7fj7f\" (UID: \"f906ef93-d6ed-491e-b7ad-3622d7398b8e\") " pod="openshift-monitoring/node-exporter-7fj7f" Jul 06 13:52:52.313944 ip-10-0-136-25 kubenswrapper[2569]: I0706 13:52:52.313786 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/f906ef93-d6ed-491e-b7ad-3622d7398b8e-node-exporter-accelerators-collector-config\") pod \"node-exporter-7fj7f\" (UID: \"f906ef93-d6ed-491e-b7ad-3622d7398b8e\") " pod="openshift-monitoring/node-exporter-7fj7f" Jul 06 13:52:52.313944 ip-10-0-136-25 kubenswrapper[2569]: I0706 13:52:52.313814 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/f906ef93-d6ed-491e-b7ad-3622d7398b8e-sys\") pod \"node-exporter-7fj7f\" (UID: \"f906ef93-d6ed-491e-b7ad-3622d7398b8e\") " pod="openshift-monitoring/node-exporter-7fj7f" Jul 06 13:52:52.313944 ip-10-0-136-25 kubenswrapper[2569]: I0706 13:52:52.313869 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"root\" (UniqueName: \"kubernetes.io/host-path/f906ef93-d6ed-491e-b7ad-3622d7398b8e-root\") pod \"node-exporter-7fj7f\" (UID: \"f906ef93-d6ed-491e-b7ad-3622d7398b8e\") " pod="openshift-monitoring/node-exporter-7fj7f" Jul 06 13:52:52.314238 ip-10-0-136-25 kubenswrapper[2569]: I0706 13:52:52.313996 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/f906ef93-d6ed-491e-b7ad-3622d7398b8e-node-exporter-wtmp\") pod \"node-exporter-7fj7f\" (UID: \"f906ef93-d6ed-491e-b7ad-3622d7398b8e\") " pod="openshift-monitoring/node-exporter-7fj7f" Jul 06 13:52:52.314238 ip-10-0-136-25 kubenswrapper[2569]: I0706 13:52:52.314000 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/f906ef93-d6ed-491e-b7ad-3622d7398b8e-node-exporter-textfile\") pod \"node-exporter-7fj7f\" (UID: \"f906ef93-d6ed-491e-b7ad-3622d7398b8e\") " pod="openshift-monitoring/node-exporter-7fj7f" Jul 06 13:52:52.314238 ip-10-0-136-25 kubenswrapper[2569]: I0706 13:52:52.313817 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p8kjn\" (UniqueName: \"kubernetes.io/projected/f906ef93-d6ed-491e-b7ad-3622d7398b8e-kube-api-access-p8kjn\") pod \"node-exporter-7fj7f\" (UID: \"f906ef93-d6ed-491e-b7ad-3622d7398b8e\") " pod="openshift-monitoring/node-exporter-7fj7f" Jul 06 13:52:52.314397 ip-10-0-136-25 kubenswrapper[2569]: I0706 13:52:52.314366 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/f906ef93-d6ed-491e-b7ad-3622d7398b8e-metrics-client-ca\") pod \"node-exporter-7fj7f\" (UID: \"f906ef93-d6ed-491e-b7ad-3622d7398b8e\") " pod="openshift-monitoring/node-exporter-7fj7f" Jul 06 13:52:52.314601 ip-10-0-136-25 kubenswrapper[2569]: I0706 13:52:52.314576 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/f906ef93-d6ed-491e-b7ad-3622d7398b8e-node-exporter-accelerators-collector-config\") pod \"node-exporter-7fj7f\" (UID: \"f906ef93-d6ed-491e-b7ad-3622d7398b8e\") " pod="openshift-monitoring/node-exporter-7fj7f" Jul 06 13:52:52.316328 ip-10-0-136-25 kubenswrapper[2569]: I0706 13:52:52.316302 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/f906ef93-d6ed-491e-b7ad-3622d7398b8e-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-7fj7f\" (UID: \"f906ef93-d6ed-491e-b7ad-3622d7398b8e\") " pod="openshift-monitoring/node-exporter-7fj7f" Jul 06 13:52:52.316673 ip-10-0-136-25 kubenswrapper[2569]: I0706 13:52:52.316653 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/f906ef93-d6ed-491e-b7ad-3622d7398b8e-node-exporter-tls\") pod \"node-exporter-7fj7f\" (UID: \"f906ef93-d6ed-491e-b7ad-3622d7398b8e\") " pod="openshift-monitoring/node-exporter-7fj7f" Jul 06 13:52:52.322603 ip-10-0-136-25 kubenswrapper[2569]: I0706 13:52:52.322568 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-p8kjn\" (UniqueName: \"kubernetes.io/projected/f906ef93-d6ed-491e-b7ad-3622d7398b8e-kube-api-access-p8kjn\") pod \"node-exporter-7fj7f\" (UID: \"f906ef93-d6ed-491e-b7ad-3622d7398b8e\") " pod="openshift-monitoring/node-exporter-7fj7f" Jul 06 13:52:52.377795 ip-10-0-136-25 kubenswrapper[2569]: I0706 13:52:52.377753 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/openshift-state-metrics-54cf96dfcf-n4mw9" Jul 06 13:52:52.446416 ip-10-0-136-25 kubenswrapper[2569]: I0706 13:52:52.446166 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-7fj7f" Jul 06 13:52:52.459122 ip-10-0-136-25 kubenswrapper[2569]: W0706 13:52:52.459070 2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf906ef93_d6ed_491e_b7ad_3622d7398b8e.slice/crio-58e81cdfb50c2dcd0631617933b070957799849de8d9fad758576f238889aa43 WatchSource:0}: Error finding container 58e81cdfb50c2dcd0631617933b070957799849de8d9fad758576f238889aa43: Status 404 returned error can't find the container with id 58e81cdfb50c2dcd0631617933b070957799849de8d9fad758576f238889aa43 Jul 06 13:52:52.527847 ip-10-0-136-25 kubenswrapper[2569]: I0706 13:52:52.527816 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/openshift-state-metrics-54cf96dfcf-n4mw9"] Jul 06 13:52:52.530450 ip-10-0-136-25 kubenswrapper[2569]: W0706 13:52:52.530425 2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddc28734e_3746_4350_9fc7_4bcc1b9e842d.slice/crio-c5c14d0a69eadcaa74c690050d6c2c114e148aae6ed62f831bf02872731540b4 WatchSource:0}: Error finding container c5c14d0a69eadcaa74c690050d6c2c114e148aae6ed62f831bf02872731540b4: Status 404 returned error can't find the container with id c5c14d0a69eadcaa74c690050d6c2c114e148aae6ed62f831bf02872731540b4 Jul 06 13:52:53.043301 ip-10-0-136-25 kubenswrapper[2569]: I0706 13:52:53.043263 2569 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Jul 06 13:52:53.047444 ip-10-0-136-25 kubenswrapper[2569]: I0706 13:52:53.047374 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0" Jul 06 13:52:53.049926 ip-10-0-136-25 kubenswrapper[2569]: I0706 13:52:53.049897 2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-metric\"" Jul 06 13:52:53.050074 ip-10-0-136-25 kubenswrapper[2569]: I0706 13:52:53.049898 2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-generated\"" Jul 06 13:52:53.050074 ip-10-0-136-25 kubenswrapper[2569]: I0706 13:52:53.049902 2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-cluster-tls-config\"" Jul 06 13:52:53.050343 ip-10-0-136-25 kubenswrapper[2569]: I0706 13:52:53.050324 2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy\"" Jul 06 13:52:53.050425 ip-10-0-136-25 kubenswrapper[2569]: I0706 13:52:53.050357 2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls-assets-0\"" Jul 06 13:52:53.050425 ip-10-0-136-25 kubenswrapper[2569]: I0706 13:52:53.050365 2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-web-config\"" Jul 06 13:52:53.050425 ip-10-0-136-25 kubenswrapper[2569]: I0706 13:52:53.050401 2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls\"" Jul 06 13:52:53.050575 ip-10-0-136-25 kubenswrapper[2569]: I0706 13:52:53.050357 2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"alertmanager-trusted-ca-bundle\"" Jul 06 13:52:53.051101 ip-10-0-136-25 kubenswrapper[2569]: I0706 13:52:53.050873 2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-dockercfg-lfcrt\"" Jul 06 13:52:53.051101 ip-10-0-136-25 kubenswrapper[2569]: I0706 13:52:53.050933 2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-web\"" Jul 06 13:52:53.063596 ip-10-0-136-25 kubenswrapper[2569]: I0706 13:52:53.063569 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Jul 06 13:52:53.119226 ip-10-0-136-25 kubenswrapper[2569]: I0706 13:52:53.119166 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/0985dbd8-3efa-4c39-9f96-8a906a0fe55c-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"0985dbd8-3efa-4c39-9f96-8a906a0fe55c\") " pod="openshift-monitoring/alertmanager-main-0" Jul 06 13:52:53.119405 ip-10-0-136-25 kubenswrapper[2569]: I0706 13:52:53.119321 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/0985dbd8-3efa-4c39-9f96-8a906a0fe55c-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"0985dbd8-3efa-4c39-9f96-8a906a0fe55c\") " pod="openshift-monitoring/alertmanager-main-0" Jul 06 13:52:53.119405 ip-10-0-136-25 kubenswrapper[2569]: I0706 13:52:53.119368 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/0985dbd8-3efa-4c39-9f96-8a906a0fe55c-config-volume\") pod \"alertmanager-main-0\" (UID: \"0985dbd8-3efa-4c39-9f96-8a906a0fe55c\") " pod="openshift-monitoring/alertmanager-main-0" Jul 06 13:52:53.119525 ip-10-0-136-25 kubenswrapper[2569]: I0706 13:52:53.119421 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/0985dbd8-3efa-4c39-9f96-8a906a0fe55c-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"0985dbd8-3efa-4c39-9f96-8a906a0fe55c\") " pod="openshift-monitoring/alertmanager-main-0" Jul 06 13:52:53.119525 ip-10-0-136-25 kubenswrapper[2569]: I0706 13:52:53.119449 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/0985dbd8-3efa-4c39-9f96-8a906a0fe55c-web-config\") pod \"alertmanager-main-0\" (UID: \"0985dbd8-3efa-4c39-9f96-8a906a0fe55c\") " pod="openshift-monitoring/alertmanager-main-0" Jul 06 13:52:53.119525 ip-10-0-136-25 kubenswrapper[2569]: I0706 13:52:53.119477 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/0985dbd8-3efa-4c39-9f96-8a906a0fe55c-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"0985dbd8-3efa-4c39-9f96-8a906a0fe55c\") " pod="openshift-monitoring/alertmanager-main-0" Jul 06 13:52:53.119525 ip-10-0-136-25 kubenswrapper[2569]: I0706 13:52:53.119501 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-t6j84\" (UniqueName: \"kubernetes.io/projected/0985dbd8-3efa-4c39-9f96-8a906a0fe55c-kube-api-access-t6j84\") pod \"alertmanager-main-0\" (UID: \"0985dbd8-3efa-4c39-9f96-8a906a0fe55c\") " pod="openshift-monitoring/alertmanager-main-0" Jul 06 13:52:53.119682 ip-10-0-136-25 kubenswrapper[2569]: I0706 13:52:53.119530 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/0985dbd8-3efa-4c39-9f96-8a906a0fe55c-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"0985dbd8-3efa-4c39-9f96-8a906a0fe55c\") " pod="openshift-monitoring/alertmanager-main-0" Jul 06 13:52:53.119682 ip-10-0-136-25 kubenswrapper[2569]: I0706 13:52:53.119612 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/0985dbd8-3efa-4c39-9f96-8a906a0fe55c-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"0985dbd8-3efa-4c39-9f96-8a906a0fe55c\") " pod="openshift-monitoring/alertmanager-main-0" Jul 06 13:52:53.119682 ip-10-0-136-25 kubenswrapper[2569]: I0706 13:52:53.119638 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/0985dbd8-3efa-4c39-9f96-8a906a0fe55c-config-out\") pod \"alertmanager-main-0\" (UID: \"0985dbd8-3efa-4c39-9f96-8a906a0fe55c\") " pod="openshift-monitoring/alertmanager-main-0" Jul 06 13:52:53.119682 ip-10-0-136-25 kubenswrapper[2569]: I0706 13:52:53.119663 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/0985dbd8-3efa-4c39-9f96-8a906a0fe55c-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"0985dbd8-3efa-4c39-9f96-8a906a0fe55c\") " pod="openshift-monitoring/alertmanager-main-0" Jul 06 13:52:53.119830 ip-10-0-136-25 kubenswrapper[2569]: I0706 13:52:53.119695 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/0985dbd8-3efa-4c39-9f96-8a906a0fe55c-tls-assets\") pod \"alertmanager-main-0\" (UID: \"0985dbd8-3efa-4c39-9f96-8a906a0fe55c\") " pod="openshift-monitoring/alertmanager-main-0" Jul 06 13:52:53.119830 ip-10-0-136-25 kubenswrapper[2569]: I0706 13:52:53.119718 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/0985dbd8-3efa-4c39-9f96-8a906a0fe55c-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"0985dbd8-3efa-4c39-9f96-8a906a0fe55c\") " pod="openshift-monitoring/alertmanager-main-0" Jul 06 13:52:53.220673 ip-10-0-136-25 kubenswrapper[2569]: I0706 13:52:53.220627 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/0985dbd8-3efa-4c39-9f96-8a906a0fe55c-tls-assets\") pod \"alertmanager-main-0\" (UID: \"0985dbd8-3efa-4c39-9f96-8a906a0fe55c\") " pod="openshift-monitoring/alertmanager-main-0" Jul 06 13:52:53.220673 ip-10-0-136-25 kubenswrapper[2569]: I0706 13:52:53.220679 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/0985dbd8-3efa-4c39-9f96-8a906a0fe55c-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"0985dbd8-3efa-4c39-9f96-8a906a0fe55c\") " pod="openshift-monitoring/alertmanager-main-0" Jul 06 13:52:53.220908 ip-10-0-136-25 kubenswrapper[2569]: I0706 13:52:53.220847 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/0985dbd8-3efa-4c39-9f96-8a906a0fe55c-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"0985dbd8-3efa-4c39-9f96-8a906a0fe55c\") " pod="openshift-monitoring/alertmanager-main-0" Jul 06 13:52:53.220968 ip-10-0-136-25 kubenswrapper[2569]: I0706 13:52:53.220936 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/0985dbd8-3efa-4c39-9f96-8a906a0fe55c-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"0985dbd8-3efa-4c39-9f96-8a906a0fe55c\") " pod="openshift-monitoring/alertmanager-main-0" Jul 06 13:52:53.221036 ip-10-0-136-25 kubenswrapper[2569]: I0706 13:52:53.220977 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/0985dbd8-3efa-4c39-9f96-8a906a0fe55c-config-volume\") pod \"alertmanager-main-0\" (UID: \"0985dbd8-3efa-4c39-9f96-8a906a0fe55c\") " pod="openshift-monitoring/alertmanager-main-0" Jul 06 13:52:53.221115 ip-10-0-136-25 kubenswrapper[2569]: I0706 13:52:53.221095 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/0985dbd8-3efa-4c39-9f96-8a906a0fe55c-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"0985dbd8-3efa-4c39-9f96-8a906a0fe55c\") " pod="openshift-monitoring/alertmanager-main-0" Jul 06 13:52:53.221170 ip-10-0-136-25 kubenswrapper[2569]: I0706 13:52:53.221129 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/0985dbd8-3efa-4c39-9f96-8a906a0fe55c-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"0985dbd8-3efa-4c39-9f96-8a906a0fe55c\") " pod="openshift-monitoring/alertmanager-main-0" Jul 06 13:52:53.221245 ip-10-0-136-25 kubenswrapper[2569]: I0706 13:52:53.221165 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/0985dbd8-3efa-4c39-9f96-8a906a0fe55c-web-config\") pod \"alertmanager-main-0\" (UID: \"0985dbd8-3efa-4c39-9f96-8a906a0fe55c\") " pod="openshift-monitoring/alertmanager-main-0" Jul 06 13:52:53.221245 ip-10-0-136-25 kubenswrapper[2569]: I0706 13:52:53.221195 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/0985dbd8-3efa-4c39-9f96-8a906a0fe55c-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"0985dbd8-3efa-4c39-9f96-8a906a0fe55c\") " pod="openshift-monitoring/alertmanager-main-0" Jul 06 13:52:53.221245 ip-10-0-136-25 kubenswrapper[2569]: I0706 13:52:53.221240 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t6j84\" (UniqueName: \"kubernetes.io/projected/0985dbd8-3efa-4c39-9f96-8a906a0fe55c-kube-api-access-t6j84\") pod \"alertmanager-main-0\" (UID: \"0985dbd8-3efa-4c39-9f96-8a906a0fe55c\") " pod="openshift-monitoring/alertmanager-main-0" Jul 06 13:52:53.221404 ip-10-0-136-25 kubenswrapper[2569]: I0706 13:52:53.221273 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/0985dbd8-3efa-4c39-9f96-8a906a0fe55c-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"0985dbd8-3efa-4c39-9f96-8a906a0fe55c\") " pod="openshift-monitoring/alertmanager-main-0" Jul 06 13:52:53.221404 ip-10-0-136-25 kubenswrapper[2569]: I0706 13:52:53.221332 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/0985dbd8-3efa-4c39-9f96-8a906a0fe55c-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"0985dbd8-3efa-4c39-9f96-8a906a0fe55c\") " pod="openshift-monitoring/alertmanager-main-0" Jul 06 13:52:53.221404 ip-10-0-136-25 kubenswrapper[2569]: I0706 13:52:53.221362 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/0985dbd8-3efa-4c39-9f96-8a906a0fe55c-config-out\") pod \"alertmanager-main-0\" (UID: \"0985dbd8-3efa-4c39-9f96-8a906a0fe55c\") " pod="openshift-monitoring/alertmanager-main-0" Jul 06 13:52:53.222381 ip-10-0-136-25 kubenswrapper[2569]: E0706 13:52:53.221713 2569 secret.go:189] Couldn't get secret openshift-monitoring/alertmanager-main-tls: secret "alertmanager-main-tls" not found Jul 06 13:52:53.222381 ip-10-0-136-25 kubenswrapper[2569]: I0706 13:52:53.221786 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/0985dbd8-3efa-4c39-9f96-8a906a0fe55c-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"0985dbd8-3efa-4c39-9f96-8a906a0fe55c\") " pod="openshift-monitoring/alertmanager-main-0" Jul 06 13:52:53.222381 ip-10-0-136-25 kubenswrapper[2569]: E0706 13:52:53.221793 2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0985dbd8-3efa-4c39-9f96-8a906a0fe55c-secret-alertmanager-main-tls podName:0985dbd8-3efa-4c39-9f96-8a906a0fe55c nodeName:}" failed. No retries permitted until 2026-07-06 13:52:53.721771433 +0000 UTC m=+149.312333216 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "secret-alertmanager-main-tls" (UniqueName: "kubernetes.io/secret/0985dbd8-3efa-4c39-9f96-8a906a0fe55c-secret-alertmanager-main-tls") pod "alertmanager-main-0" (UID: "0985dbd8-3efa-4c39-9f96-8a906a0fe55c") : secret "alertmanager-main-tls" not found Jul 06 13:52:53.222381 ip-10-0-136-25 kubenswrapper[2569]: I0706 13:52:53.221841 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/0985dbd8-3efa-4c39-9f96-8a906a0fe55c-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"0985dbd8-3efa-4c39-9f96-8a906a0fe55c\") " pod="openshift-monitoring/alertmanager-main-0" Jul 06 13:52:53.222381 ip-10-0-136-25 kubenswrapper[2569]: E0706 13:52:53.222065 2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/0985dbd8-3efa-4c39-9f96-8a906a0fe55c-alertmanager-trusted-ca-bundle podName:0985dbd8-3efa-4c39-9f96-8a906a0fe55c nodeName:}" failed. No retries permitted until 2026-07-06 13:52:53.722052025 +0000 UTC m=+149.312613786 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "alertmanager-trusted-ca-bundle" (UniqueName: "kubernetes.io/configmap/0985dbd8-3efa-4c39-9f96-8a906a0fe55c-alertmanager-trusted-ca-bundle") pod "alertmanager-main-0" (UID: "0985dbd8-3efa-4c39-9f96-8a906a0fe55c") : configmap references non-existent config key: ca-bundle.crt Jul 06 13:52:53.225377 ip-10-0-136-25 kubenswrapper[2569]: I0706 13:52:53.225349 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/0985dbd8-3efa-4c39-9f96-8a906a0fe55c-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"0985dbd8-3efa-4c39-9f96-8a906a0fe55c\") " pod="openshift-monitoring/alertmanager-main-0" Jul 06 13:52:53.225662 ip-10-0-136-25 kubenswrapper[2569]: I0706 13:52:53.225623 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/0985dbd8-3efa-4c39-9f96-8a906a0fe55c-config-out\") pod \"alertmanager-main-0\" (UID: \"0985dbd8-3efa-4c39-9f96-8a906a0fe55c\") " pod="openshift-monitoring/alertmanager-main-0" Jul 06 13:52:53.225766 ip-10-0-136-25 kubenswrapper[2569]: I0706 13:52:53.225701 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/0985dbd8-3efa-4c39-9f96-8a906a0fe55c-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"0985dbd8-3efa-4c39-9f96-8a906a0fe55c\") " pod="openshift-monitoring/alertmanager-main-0" Jul 06 13:52:53.226569 ip-10-0-136-25 kubenswrapper[2569]: I0706 13:52:53.226548 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/0985dbd8-3efa-4c39-9f96-8a906a0fe55c-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"0985dbd8-3efa-4c39-9f96-8a906a0fe55c\") " pod="openshift-monitoring/alertmanager-main-0" Jul 06 13:52:53.226735 ip-10-0-136-25 kubenswrapper[2569]: I0706 13:52:53.226711 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/0985dbd8-3efa-4c39-9f96-8a906a0fe55c-web-config\") pod \"alertmanager-main-0\" (UID: \"0985dbd8-3efa-4c39-9f96-8a906a0fe55c\") " pod="openshift-monitoring/alertmanager-main-0" Jul 06 13:52:53.226846 ip-10-0-136-25 kubenswrapper[2569]: I0706 13:52:53.226818 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/0985dbd8-3efa-4c39-9f96-8a906a0fe55c-config-volume\") pod \"alertmanager-main-0\" (UID: \"0985dbd8-3efa-4c39-9f96-8a906a0fe55c\") " pod="openshift-monitoring/alertmanager-main-0" Jul 06 13:52:53.227264 ip-10-0-136-25 kubenswrapper[2569]: I0706 13:52:53.227244 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/0985dbd8-3efa-4c39-9f96-8a906a0fe55c-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"0985dbd8-3efa-4c39-9f96-8a906a0fe55c\") " pod="openshift-monitoring/alertmanager-main-0" Jul 06 13:52:53.228070 ip-10-0-136-25 kubenswrapper[2569]: I0706 13:52:53.228048 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/0985dbd8-3efa-4c39-9f96-8a906a0fe55c-tls-assets\") pod \"alertmanager-main-0\" (UID: \"0985dbd8-3efa-4c39-9f96-8a906a0fe55c\") " pod="openshift-monitoring/alertmanager-main-0" Jul 06 13:52:53.230272 ip-10-0-136-25 kubenswrapper[2569]: I0706 13:52:53.230251 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-t6j84\" (UniqueName: \"kubernetes.io/projected/0985dbd8-3efa-4c39-9f96-8a906a0fe55c-kube-api-access-t6j84\") pod \"alertmanager-main-0\" (UID: \"0985dbd8-3efa-4c39-9f96-8a906a0fe55c\") " pod="openshift-monitoring/alertmanager-main-0" Jul 06 13:52:53.467624 ip-10-0-136-25 kubenswrapper[2569]: I0706 13:52:53.467524 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-7fj7f" event={"ID":"f906ef93-d6ed-491e-b7ad-3622d7398b8e","Type":"ContainerStarted","Data":"58e81cdfb50c2dcd0631617933b070957799849de8d9fad758576f238889aa43"} Jul 06 13:52:53.470098 ip-10-0-136-25 kubenswrapper[2569]: I0706 13:52:53.470069 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-kc97f" event={"ID":"3c279543-7a20-4d29-9de1-6cd09f14f692","Type":"ContainerStarted","Data":"1e99f0425309896c5b4eb1ac6824dacec73c699be3b763fae171fc7ed89a5501"} Jul 06 13:52:53.472039 ip-10-0-136-25 kubenswrapper[2569]: I0706 13:52:53.472012 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-54cf96dfcf-n4mw9" event={"ID":"dc28734e-3746-4350-9fc7-4bcc1b9e842d","Type":"ContainerStarted","Data":"acf1b375b805783e4a73c8975d402f996035232712b1e2a6d14cf1ba660309e6"} Jul 06 13:52:53.472132 ip-10-0-136-25 kubenswrapper[2569]: I0706 13:52:53.472068 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-54cf96dfcf-n4mw9" event={"ID":"dc28734e-3746-4350-9fc7-4bcc1b9e842d","Type":"ContainerStarted","Data":"4ad52848bbf5b76e204bdfe62b06525fadbeb20951e61f079602b737b8a9fba6"} Jul 06 13:52:53.472132 ip-10-0-136-25 kubenswrapper[2569]: I0706 13:52:53.472094 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-54cf96dfcf-n4mw9" event={"ID":"dc28734e-3746-4350-9fc7-4bcc1b9e842d","Type":"ContainerStarted","Data":"c5c14d0a69eadcaa74c690050d6c2c114e148aae6ed62f831bf02872731540b4"} Jul 06 13:52:53.488063 ip-10-0-136-25 kubenswrapper[2569]: I0706 13:52:53.488010 2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-insights/insights-runtime-extractor-kc97f" podStartSLOduration=33.599157323 podStartE2EDuration="40.487991982s" totalImagesPullingTime="6.888834659s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-06 13:52:13 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 13:52:53.487778794 +0000 UTC m=+149.078340571" watchObservedRunningTime="2026-07-06 13:52:53.487991982 +0000 UTC m=+149.078553760" Jul 06 13:52:53.726962 ip-10-0-136-25 kubenswrapper[2569]: I0706 13:52:53.726922 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/0985dbd8-3efa-4c39-9f96-8a906a0fe55c-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"0985dbd8-3efa-4c39-9f96-8a906a0fe55c\") " pod="openshift-monitoring/alertmanager-main-0" Jul 06 13:52:53.727151 ip-10-0-136-25 kubenswrapper[2569]: I0706 13:52:53.726983 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/0985dbd8-3efa-4c39-9f96-8a906a0fe55c-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"0985dbd8-3efa-4c39-9f96-8a906a0fe55c\") " pod="openshift-monitoring/alertmanager-main-0" Jul 06 13:52:53.727151 ip-10-0-136-25 kubenswrapper[2569]: E0706 13:52:53.727053 2569 secret.go:189] Couldn't get secret openshift-monitoring/alertmanager-main-tls: secret "alertmanager-main-tls" not found Jul 06 13:52:53.727151 ip-10-0-136-25 kubenswrapper[2569]: E0706 13:52:53.727118 2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0985dbd8-3efa-4c39-9f96-8a906a0fe55c-secret-alertmanager-main-tls podName:0985dbd8-3efa-4c39-9f96-8a906a0fe55c nodeName:}" failed. No retries permitted until 2026-07-06 13:52:54.727098766 +0000 UTC m=+150.317660538 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "secret-alertmanager-main-tls" (UniqueName: "kubernetes.io/secret/0985dbd8-3efa-4c39-9f96-8a906a0fe55c-secret-alertmanager-main-tls") pod "alertmanager-main-0" (UID: "0985dbd8-3efa-4c39-9f96-8a906a0fe55c") : secret "alertmanager-main-tls" not found Jul 06 13:52:53.727964 ip-10-0-136-25 kubenswrapper[2569]: I0706 13:52:53.727944 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/0985dbd8-3efa-4c39-9f96-8a906a0fe55c-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"0985dbd8-3efa-4c39-9f96-8a906a0fe55c\") " pod="openshift-monitoring/alertmanager-main-0" Jul 06 13:52:53.934950 ip-10-0-136-25 kubenswrapper[2569]: I0706 13:52:53.934917 2569 scope.go:117] "RemoveContainer" containerID="44529ca19da436ed637824808a9ac86d648c96ea3e9fb7da354016dd4df57d81" Jul 06 13:52:54.479640 ip-10-0-136-25 kubenswrapper[2569]: I0706 13:52:54.479614 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-68b757fb67-z2h95_dfffcdca-94d6-4732-815f-bcb533f538e9/console-operator/2.log" Jul 06 13:52:54.480090 ip-10-0-136-25 kubenswrapper[2569]: I0706 13:52:54.479708 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-68b757fb67-z2h95" event={"ID":"dfffcdca-94d6-4732-815f-bcb533f538e9","Type":"ContainerStarted","Data":"6c099daccc03e025aabf7c61f3f06aacda0486c3203d3fdf2f982af13275ae58"} Jul 06 13:52:54.480090 ip-10-0-136-25 kubenswrapper[2569]: I0706 13:52:54.480000 2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console-operator/console-operator-68b757fb67-z2h95" Jul 06 13:52:54.482020 ip-10-0-136-25 kubenswrapper[2569]: I0706 13:52:54.481993 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-54cf96dfcf-n4mw9" event={"ID":"dc28734e-3746-4350-9fc7-4bcc1b9e842d","Type":"ContainerStarted","Data":"8bd3c71384ae8179d657bd7483bae64adac1e1326d03f6ac5b7a0e8abf174baf"} Jul 06 13:52:54.483456 ip-10-0-136-25 kubenswrapper[2569]: I0706 13:52:54.483433 2569 generic.go:356] "Generic (PLEG): container finished" podID="f906ef93-d6ed-491e-b7ad-3622d7398b8e" containerID="f0258e5e9f08efe727cc53ba756eccf5d168287454690f3f6792153742265342" exitCode=0 Jul 06 13:52:54.483584 ip-10-0-136-25 kubenswrapper[2569]: I0706 13:52:54.483520 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-7fj7f" event={"ID":"f906ef93-d6ed-491e-b7ad-3622d7398b8e","Type":"ContainerDied","Data":"f0258e5e9f08efe727cc53ba756eccf5d168287454690f3f6792153742265342"} Jul 06 13:52:54.512761 ip-10-0-136-25 kubenswrapper[2569]: I0706 13:52:54.512718 2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console-operator/console-operator-68b757fb67-z2h95" podStartSLOduration=43.32782328 podStartE2EDuration="46.512702742s" totalImagesPullingTime="3.184879462s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-06 13:52:08 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 13:52:54.512623559 +0000 UTC m=+150.103185336" watchObservedRunningTime="2026-07-06 13:52:54.512702742 +0000 UTC m=+150.103264544" Jul 06 13:52:54.562965 ip-10-0-136-25 kubenswrapper[2569]: I0706 13:52:54.562922 2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/openshift-state-metrics-54cf96dfcf-n4mw9" podStartSLOduration=1.107275009 podStartE2EDuration="2.562904638s" totalImagesPullingTime="1.455629629s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-06 13:52:52 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 13:52:54.561586878 +0000 UTC m=+150.152148682" watchObservedRunningTime="2026-07-06 13:52:54.562904638 +0000 UTC m=+150.153466414" Jul 06 13:52:54.612164 ip-10-0-136-25 kubenswrapper[2569]: I0706 13:52:54.612136 2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console-operator/console-operator-68b757fb67-z2h95" Jul 06 13:52:54.737051 ip-10-0-136-25 kubenswrapper[2569]: I0706 13:52:54.736969 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/0985dbd8-3efa-4c39-9f96-8a906a0fe55c-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"0985dbd8-3efa-4c39-9f96-8a906a0fe55c\") " pod="openshift-monitoring/alertmanager-main-0" Jul 06 13:52:54.739382 ip-10-0-136-25 kubenswrapper[2569]: I0706 13:52:54.739361 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/0985dbd8-3efa-4c39-9f96-8a906a0fe55c-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"0985dbd8-3efa-4c39-9f96-8a906a0fe55c\") " pod="openshift-monitoring/alertmanager-main-0" Jul 06 13:52:54.819273 ip-10-0-136-25 kubenswrapper[2569]: I0706 13:52:54.819237 2569 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/downloads-685bfb5f64-nwr8k"] Jul 06 13:52:54.822549 ip-10-0-136-25 kubenswrapper[2569]: I0706 13:52:54.822527 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/downloads-685bfb5f64-nwr8k" Jul 06 13:52:54.825409 ip-10-0-136-25 kubenswrapper[2569]: I0706 13:52:54.825374 2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"kube-root-ca.crt\"" Jul 06 13:52:54.825594 ip-10-0-136-25 kubenswrapper[2569]: I0706 13:52:54.825423 2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"openshift-service-ca.crt\"" Jul 06 13:52:54.825594 ip-10-0-136-25 kubenswrapper[2569]: I0706 13:52:54.825422 2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"default-dockercfg-6nz2m\"" Jul 06 13:52:54.834384 ip-10-0-136-25 kubenswrapper[2569]: I0706 13:52:54.834358 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/downloads-685bfb5f64-nwr8k"] Jul 06 13:52:54.837488 ip-10-0-136-25 kubenswrapper[2569]: I0706 13:52:54.837467 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gdmj2\" (UniqueName: \"kubernetes.io/projected/e95b78c0-73d7-4e1d-aeab-d59969a3a319-kube-api-access-gdmj2\") pod \"downloads-685bfb5f64-nwr8k\" (UID: \"e95b78c0-73d7-4e1d-aeab-d59969a3a319\") " pod="openshift-console/downloads-685bfb5f64-nwr8k" Jul 06 13:52:54.858780 ip-10-0-136-25 kubenswrapper[2569]: I0706 13:52:54.858748 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0" Jul 06 13:52:54.938588 ip-10-0-136-25 kubenswrapper[2569]: I0706 13:52:54.938560 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gdmj2\" (UniqueName: \"kubernetes.io/projected/e95b78c0-73d7-4e1d-aeab-d59969a3a319-kube-api-access-gdmj2\") pod \"downloads-685bfb5f64-nwr8k\" (UID: \"e95b78c0-73d7-4e1d-aeab-d59969a3a319\") " pod="openshift-console/downloads-685bfb5f64-nwr8k" Jul 06 13:52:54.947625 ip-10-0-136-25 kubenswrapper[2569]: I0706 13:52:54.947592 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gdmj2\" (UniqueName: \"kubernetes.io/projected/e95b78c0-73d7-4e1d-aeab-d59969a3a319-kube-api-access-gdmj2\") pod \"downloads-685bfb5f64-nwr8k\" (UID: \"e95b78c0-73d7-4e1d-aeab-d59969a3a319\") " pod="openshift-console/downloads-685bfb5f64-nwr8k" Jul 06 13:52:55.008306 ip-10-0-136-25 kubenswrapper[2569]: I0706 13:52:55.008279 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Jul 06 13:52:55.010663 ip-10-0-136-25 kubenswrapper[2569]: W0706 13:52:55.010638 2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0985dbd8_3efa_4c39_9f96_8a906a0fe55c.slice/crio-81a917be48241ca46ef483310d97897a0e417ab6271062b85b76cf77fbebc3fc WatchSource:0}: Error finding container 81a917be48241ca46ef483310d97897a0e417ab6271062b85b76cf77fbebc3fc: Status 404 returned error can't find the container with id 81a917be48241ca46ef483310d97897a0e417ab6271062b85b76cf77fbebc3fc Jul 06 13:52:55.038062 ip-10-0-136-25 kubenswrapper[2569]: I0706 13:52:55.038026 2569 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/thanos-querier-7d96487c86-rnclw"] Jul 06 13:52:55.043284 ip-10-0-136-25 kubenswrapper[2569]: I0706 13:52:55.043249 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/thanos-querier-7d96487c86-rnclw" Jul 06 13:52:55.045676 ip-10-0-136-25 kubenswrapper[2569]: I0706 13:52:55.045649 2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-rules\"" Jul 06 13:52:55.045815 ip-10-0-136-25 kubenswrapper[2569]: I0706 13:52:55.045694 2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-metrics\"" Jul 06 13:52:55.045815 ip-10-0-136-25 kubenswrapper[2569]: I0706 13:52:55.045703 2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy\"" Jul 06 13:52:55.045815 ip-10-0-136-25 kubenswrapper[2569]: I0706 13:52:55.045793 2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-grpc-tls-5flm3c57si5l\"" Jul 06 13:52:55.046279 ip-10-0-136-25 kubenswrapper[2569]: I0706 13:52:55.046262 2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-dockercfg-xsl5r\"" Jul 06 13:52:55.046381 ip-10-0-136-25 kubenswrapper[2569]: I0706 13:52:55.046365 2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-web\"" Jul 06 13:52:55.046437 ip-10-0-136-25 kubenswrapper[2569]: I0706 13:52:55.046396 2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-tls\"" Jul 06 13:52:55.053827 ip-10-0-136-25 kubenswrapper[2569]: I0706 13:52:55.053799 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/thanos-querier-7d96487c86-rnclw"] Jul 06 13:52:55.136597 ip-10-0-136-25 kubenswrapper[2569]: I0706 13:52:55.132264 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/downloads-685bfb5f64-nwr8k" Jul 06 13:52:55.140119 ip-10-0-136-25 kubenswrapper[2569]: I0706 13:52:55.140090 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/35142749-f6cf-4c8c-b41b-324c97c06c4c-secret-thanos-querier-kube-rbac-proxy-web\") pod \"thanos-querier-7d96487c86-rnclw\" (UID: \"35142749-f6cf-4c8c-b41b-324c97c06c4c\") " pod="openshift-monitoring/thanos-querier-7d96487c86-rnclw" Jul 06 13:52:55.140119 ip-10-0-136-25 kubenswrapper[2569]: I0706 13:52:55.140128 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-tls\" (UniqueName: \"kubernetes.io/secret/35142749-f6cf-4c8c-b41b-324c97c06c4c-secret-thanos-querier-tls\") pod \"thanos-querier-7d96487c86-rnclw\" (UID: \"35142749-f6cf-4c8c-b41b-324c97c06c4c\") " pod="openshift-monitoring/thanos-querier-7d96487c86-rnclw" Jul 06 13:52:55.140351 ip-10-0-136-25 kubenswrapper[2569]: I0706 13:52:55.140154 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-rules\" (UniqueName: \"kubernetes.io/secret/35142749-f6cf-4c8c-b41b-324c97c06c4c-secret-thanos-querier-kube-rbac-proxy-rules\") pod \"thanos-querier-7d96487c86-rnclw\" (UID: \"35142749-f6cf-4c8c-b41b-324c97c06c4c\") " pod="openshift-monitoring/thanos-querier-7d96487c86-rnclw" Jul 06 13:52:55.140351 ip-10-0-136-25 kubenswrapper[2569]: I0706 13:52:55.140180 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/35142749-f6cf-4c8c-b41b-324c97c06c4c-secret-thanos-querier-kube-rbac-proxy\") pod \"thanos-querier-7d96487c86-rnclw\" (UID: \"35142749-f6cf-4c8c-b41b-324c97c06c4c\") " pod="openshift-monitoring/thanos-querier-7d96487c86-rnclw" Jul 06 13:52:55.140351 ip-10-0-136-25 kubenswrapper[2569]: I0706 13:52:55.140274 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/35142749-f6cf-4c8c-b41b-324c97c06c4c-metrics-client-ca\") pod \"thanos-querier-7d96487c86-rnclw\" (UID: \"35142749-f6cf-4c8c-b41b-324c97c06c4c\") " pod="openshift-monitoring/thanos-querier-7d96487c86-rnclw" Jul 06 13:52:55.140351 ip-10-0-136-25 kubenswrapper[2569]: I0706 13:52:55.140315 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/35142749-f6cf-4c8c-b41b-324c97c06c4c-secret-grpc-tls\") pod \"thanos-querier-7d96487c86-rnclw\" (UID: \"35142749-f6cf-4c8c-b41b-324c97c06c4c\") " pod="openshift-monitoring/thanos-querier-7d96487c86-rnclw" Jul 06 13:52:55.140547 ip-10-0-136-25 kubenswrapper[2569]: I0706 13:52:55.140364 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-metrics\" (UniqueName: \"kubernetes.io/secret/35142749-f6cf-4c8c-b41b-324c97c06c4c-secret-thanos-querier-kube-rbac-proxy-metrics\") pod \"thanos-querier-7d96487c86-rnclw\" (UID: \"35142749-f6cf-4c8c-b41b-324c97c06c4c\") " pod="openshift-monitoring/thanos-querier-7d96487c86-rnclw" Jul 06 13:52:55.140547 ip-10-0-136-25 kubenswrapper[2569]: I0706 13:52:55.140420 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mbrbq\" (UniqueName: \"kubernetes.io/projected/35142749-f6cf-4c8c-b41b-324c97c06c4c-kube-api-access-mbrbq\") pod \"thanos-querier-7d96487c86-rnclw\" (UID: \"35142749-f6cf-4c8c-b41b-324c97c06c4c\") " pod="openshift-monitoring/thanos-querier-7d96487c86-rnclw" Jul 06 13:52:55.241264 ip-10-0-136-25 kubenswrapper[2569]: I0706 13:52:55.241225 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/35142749-f6cf-4c8c-b41b-324c97c06c4c-metrics-client-ca\") pod \"thanos-querier-7d96487c86-rnclw\" (UID: \"35142749-f6cf-4c8c-b41b-324c97c06c4c\") " pod="openshift-monitoring/thanos-querier-7d96487c86-rnclw" Jul 06 13:52:55.241428 ip-10-0-136-25 kubenswrapper[2569]: I0706 13:52:55.241290 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/35142749-f6cf-4c8c-b41b-324c97c06c4c-secret-grpc-tls\") pod \"thanos-querier-7d96487c86-rnclw\" (UID: \"35142749-f6cf-4c8c-b41b-324c97c06c4c\") " pod="openshift-monitoring/thanos-querier-7d96487c86-rnclw" Jul 06 13:52:55.241428 ip-10-0-136-25 kubenswrapper[2569]: I0706 13:52:55.241345 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-metrics\" (UniqueName: \"kubernetes.io/secret/35142749-f6cf-4c8c-b41b-324c97c06c4c-secret-thanos-querier-kube-rbac-proxy-metrics\") pod \"thanos-querier-7d96487c86-rnclw\" (UID: \"35142749-f6cf-4c8c-b41b-324c97c06c4c\") " pod="openshift-monitoring/thanos-querier-7d96487c86-rnclw" Jul 06 13:52:55.241428 ip-10-0-136-25 kubenswrapper[2569]: I0706 13:52:55.241401 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mbrbq\" (UniqueName: \"kubernetes.io/projected/35142749-f6cf-4c8c-b41b-324c97c06c4c-kube-api-access-mbrbq\") pod \"thanos-querier-7d96487c86-rnclw\" (UID: \"35142749-f6cf-4c8c-b41b-324c97c06c4c\") " pod="openshift-monitoring/thanos-querier-7d96487c86-rnclw" Jul 06 13:52:55.241591 ip-10-0-136-25 kubenswrapper[2569]: I0706 13:52:55.241449 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/35142749-f6cf-4c8c-b41b-324c97c06c4c-secret-thanos-querier-kube-rbac-proxy-web\") pod \"thanos-querier-7d96487c86-rnclw\" (UID: \"35142749-f6cf-4c8c-b41b-324c97c06c4c\") " pod="openshift-monitoring/thanos-querier-7d96487c86-rnclw" Jul 06 13:52:55.241591 ip-10-0-136-25 kubenswrapper[2569]: I0706 13:52:55.241478 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-tls\" (UniqueName: \"kubernetes.io/secret/35142749-f6cf-4c8c-b41b-324c97c06c4c-secret-thanos-querier-tls\") pod \"thanos-querier-7d96487c86-rnclw\" (UID: \"35142749-f6cf-4c8c-b41b-324c97c06c4c\") " pod="openshift-monitoring/thanos-querier-7d96487c86-rnclw" Jul 06 13:52:55.241591 ip-10-0-136-25 kubenswrapper[2569]: I0706 13:52:55.241515 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-rules\" (UniqueName: \"kubernetes.io/secret/35142749-f6cf-4c8c-b41b-324c97c06c4c-secret-thanos-querier-kube-rbac-proxy-rules\") pod \"thanos-querier-7d96487c86-rnclw\" (UID: \"35142749-f6cf-4c8c-b41b-324c97c06c4c\") " pod="openshift-monitoring/thanos-querier-7d96487c86-rnclw" Jul 06 13:52:55.241731 ip-10-0-136-25 kubenswrapper[2569]: I0706 13:52:55.241695 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/35142749-f6cf-4c8c-b41b-324c97c06c4c-secret-thanos-querier-kube-rbac-proxy\") pod \"thanos-querier-7d96487c86-rnclw\" (UID: \"35142749-f6cf-4c8c-b41b-324c97c06c4c\") " pod="openshift-monitoring/thanos-querier-7d96487c86-rnclw" Jul 06 13:52:55.242241 ip-10-0-136-25 kubenswrapper[2569]: I0706 13:52:55.242192 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/35142749-f6cf-4c8c-b41b-324c97c06c4c-metrics-client-ca\") pod \"thanos-querier-7d96487c86-rnclw\" (UID: \"35142749-f6cf-4c8c-b41b-324c97c06c4c\") " pod="openshift-monitoring/thanos-querier-7d96487c86-rnclw" Jul 06 13:52:55.244292 ip-10-0-136-25 kubenswrapper[2569]: I0706 13:52:55.244266 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-tls\" (UniqueName: \"kubernetes.io/secret/35142749-f6cf-4c8c-b41b-324c97c06c4c-secret-thanos-querier-tls\") pod \"thanos-querier-7d96487c86-rnclw\" (UID: \"35142749-f6cf-4c8c-b41b-324c97c06c4c\") " pod="openshift-monitoring/thanos-querier-7d96487c86-rnclw" Jul 06 13:52:55.244509 ip-10-0-136-25 kubenswrapper[2569]: I0706 13:52:55.244486 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-metrics\" (UniqueName: \"kubernetes.io/secret/35142749-f6cf-4c8c-b41b-324c97c06c4c-secret-thanos-querier-kube-rbac-proxy-metrics\") pod \"thanos-querier-7d96487c86-rnclw\" (UID: \"35142749-f6cf-4c8c-b41b-324c97c06c4c\") " pod="openshift-monitoring/thanos-querier-7d96487c86-rnclw" Jul 06 13:52:55.244717 ip-10-0-136-25 kubenswrapper[2569]: I0706 13:52:55.244696 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/35142749-f6cf-4c8c-b41b-324c97c06c4c-secret-thanos-querier-kube-rbac-proxy-web\") pod \"thanos-querier-7d96487c86-rnclw\" (UID: \"35142749-f6cf-4c8c-b41b-324c97c06c4c\") " pod="openshift-monitoring/thanos-querier-7d96487c86-rnclw" Jul 06 13:52:55.244840 ip-10-0-136-25 kubenswrapper[2569]: I0706 13:52:55.244824 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/35142749-f6cf-4c8c-b41b-324c97c06c4c-secret-grpc-tls\") pod \"thanos-querier-7d96487c86-rnclw\" (UID: \"35142749-f6cf-4c8c-b41b-324c97c06c4c\") " pod="openshift-monitoring/thanos-querier-7d96487c86-rnclw" Jul 06 13:52:55.244895 ip-10-0-136-25 kubenswrapper[2569]: I0706 13:52:55.244875 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-rules\" (UniqueName: \"kubernetes.io/secret/35142749-f6cf-4c8c-b41b-324c97c06c4c-secret-thanos-querier-kube-rbac-proxy-rules\") pod \"thanos-querier-7d96487c86-rnclw\" (UID: \"35142749-f6cf-4c8c-b41b-324c97c06c4c\") " pod="openshift-monitoring/thanos-querier-7d96487c86-rnclw" Jul 06 13:52:55.244933 ip-10-0-136-25 kubenswrapper[2569]: I0706 13:52:55.244877 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/35142749-f6cf-4c8c-b41b-324c97c06c4c-secret-thanos-querier-kube-rbac-proxy\") pod \"thanos-querier-7d96487c86-rnclw\" (UID: \"35142749-f6cf-4c8c-b41b-324c97c06c4c\") " pod="openshift-monitoring/thanos-querier-7d96487c86-rnclw" Jul 06 13:52:55.249165 ip-10-0-136-25 kubenswrapper[2569]: I0706 13:52:55.249146 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mbrbq\" (UniqueName: \"kubernetes.io/projected/35142749-f6cf-4c8c-b41b-324c97c06c4c-kube-api-access-mbrbq\") pod \"thanos-querier-7d96487c86-rnclw\" (UID: \"35142749-f6cf-4c8c-b41b-324c97c06c4c\") " pod="openshift-monitoring/thanos-querier-7d96487c86-rnclw" Jul 06 13:52:55.261203 ip-10-0-136-25 kubenswrapper[2569]: I0706 13:52:55.261143 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/downloads-685bfb5f64-nwr8k"] Jul 06 13:52:55.264025 ip-10-0-136-25 kubenswrapper[2569]: W0706 13:52:55.264000 2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode95b78c0_73d7_4e1d_aeab_d59969a3a319.slice/crio-1c4227ade6ac029a3005d54bae16511eea20b76ea0629153596251d76ad5cc0f WatchSource:0}: Error finding container 1c4227ade6ac029a3005d54bae16511eea20b76ea0629153596251d76ad5cc0f: Status 404 returned error can't find the container with id 1c4227ade6ac029a3005d54bae16511eea20b76ea0629153596251d76ad5cc0f Jul 06 13:52:55.352835 ip-10-0-136-25 kubenswrapper[2569]: I0706 13:52:55.352805 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/thanos-querier-7d96487c86-rnclw" Jul 06 13:52:55.488031 ip-10-0-136-25 kubenswrapper[2569]: I0706 13:52:55.487996 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/downloads-685bfb5f64-nwr8k" event={"ID":"e95b78c0-73d7-4e1d-aeab-d59969a3a319","Type":"ContainerStarted","Data":"1c4227ade6ac029a3005d54bae16511eea20b76ea0629153596251d76ad5cc0f"} Jul 06 13:52:55.489094 ip-10-0-136-25 kubenswrapper[2569]: I0706 13:52:55.489071 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"0985dbd8-3efa-4c39-9f96-8a906a0fe55c","Type":"ContainerStarted","Data":"81a917be48241ca46ef483310d97897a0e417ab6271062b85b76cf77fbebc3fc"} Jul 06 13:52:55.491424 ip-10-0-136-25 kubenswrapper[2569]: I0706 13:52:55.491402 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-7fj7f" event={"ID":"f906ef93-d6ed-491e-b7ad-3622d7398b8e","Type":"ContainerStarted","Data":"4b6600af79c1b34ad3c6e7630c30ad08834d44ecd8ccb84392fdee5b0b08f4a9"} Jul 06 13:52:55.491514 ip-10-0-136-25 kubenswrapper[2569]: I0706 13:52:55.491445 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-7fj7f" event={"ID":"f906ef93-d6ed-491e-b7ad-3622d7398b8e","Type":"ContainerStarted","Data":"8271d7ec4c411437bde0246f24b697aef2092697e55dc1bdff114c0f61fec047"} Jul 06 13:52:55.492845 ip-10-0-136-25 kubenswrapper[2569]: I0706 13:52:55.492823 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/thanos-querier-7d96487c86-rnclw"] Jul 06 13:52:55.495846 ip-10-0-136-25 kubenswrapper[2569]: W0706 13:52:55.495826 2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod35142749_f6cf_4c8c_b41b_324c97c06c4c.slice/crio-04f4f0ec3db5a18b0468b16ff2f7c1df2f00eef68baf289bf5adcd4f067e7d31 WatchSource:0}: Error finding container 04f4f0ec3db5a18b0468b16ff2f7c1df2f00eef68baf289bf5adcd4f067e7d31: Status 404 returned error can't find the container with id 04f4f0ec3db5a18b0468b16ff2f7c1df2f00eef68baf289bf5adcd4f067e7d31 Jul 06 13:52:55.513979 ip-10-0-136-25 kubenswrapper[2569]: I0706 13:52:55.513891 2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/node-exporter-7fj7f" podStartSLOduration=2.375932783 podStartE2EDuration="3.513876581s" totalImagesPullingTime="1.069516896s" totalInitContainerRuntime="68.426902ms" isStatefulPod=true podCreationTimestamp="2026-07-06 13:52:52 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 13:52:55.511462771 +0000 UTC m=+151.102024558" watchObservedRunningTime="2026-07-06 13:52:55.513876581 +0000 UTC m=+151.104438358" Jul 06 13:52:56.497246 ip-10-0-136-25 kubenswrapper[2569]: I0706 13:52:56.497186 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-7d96487c86-rnclw" event={"ID":"35142749-f6cf-4c8c-b41b-324c97c06c4c","Type":"ContainerStarted","Data":"04f4f0ec3db5a18b0468b16ff2f7c1df2f00eef68baf289bf5adcd4f067e7d31"} Jul 06 13:52:57.503004 ip-10-0-136-25 kubenswrapper[2569]: I0706 13:52:57.502965 2569 generic.go:356] "Generic (PLEG): container finished" podID="0985dbd8-3efa-4c39-9f96-8a906a0fe55c" containerID="1232f2f32291fd7e7850626a6711cca4dbe7a6318a4bf635ab6d25c44f5574f5" exitCode=0 Jul 06 13:52:57.503514 ip-10-0-136-25 kubenswrapper[2569]: I0706 13:52:57.503054 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"0985dbd8-3efa-4c39-9f96-8a906a0fe55c","Type":"ContainerDied","Data":"1232f2f32291fd7e7850626a6711cca4dbe7a6318a4bf635ab6d25c44f5574f5"} Jul 06 13:52:58.269664 ip-10-0-136-25 kubenswrapper[2569]: I0706 13:52:58.269621 2569 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Jul 06 13:52:58.274135 ip-10-0-136-25 kubenswrapper[2569]: I0706 13:52:58.274109 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0" Jul 06 13:52:58.276312 ip-10-0-136-25 kubenswrapper[2569]: I0706 13:52:58.276286 2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-dockercfg-hk5tf\"" Jul 06 13:52:58.278838 ip-10-0-136-25 kubenswrapper[2569]: I0706 13:52:58.278812 2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"serving-certs-ca-bundle\"" Jul 06 13:52:58.278948 ip-10-0-136-25 kubenswrapper[2569]: I0706 13:52:58.278886 2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-kube-rbac-proxy-web\"" Jul 06 13:52:58.279290 ip-10-0-136-25 kubenswrapper[2569]: I0706 13:52:58.279015 2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-rbac-proxy\"" Jul 06 13:52:58.279290 ip-10-0-136-25 kubenswrapper[2569]: I0706 13:52:58.279105 2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s\"" Jul 06 13:52:58.279290 ip-10-0-136-25 kubenswrapper[2569]: I0706 13:52:58.279123 2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kubelet-serving-ca-bundle\"" Jul 06 13:52:58.279290 ip-10-0-136-25 kubenswrapper[2569]: I0706 13:52:58.279152 2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls\"" Jul 06 13:52:58.279290 ip-10-0-136-25 kubenswrapper[2569]: I0706 13:52:58.279164 2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-prometheus-http-client-file\"" Jul 06 13:52:58.279290 ip-10-0-136-25 kubenswrapper[2569]: I0706 13:52:58.279165 2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-web-config\"" Jul 06 13:52:58.279853 ip-10-0-136-25 kubenswrapper[2569]: I0706 13:52:58.279335 2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-trusted-ca-bundle\"" Jul 06 13:52:58.279853 ip-10-0-136-25 kubenswrapper[2569]: I0706 13:52:58.279378 2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-client-certs\"" Jul 06 13:52:58.279853 ip-10-0-136-25 kubenswrapper[2569]: I0706 13:52:58.279387 2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-grpc-tls-30phi2tl4q3fv\"" Jul 06 13:52:58.279853 ip-10-0-136-25 kubenswrapper[2569]: I0706 13:52:58.279398 2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls-assets-0\"" Jul 06 13:52:58.279853 ip-10-0-136-25 kubenswrapper[2569]: I0706 13:52:58.279336 2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-sidecar-tls\"" Jul 06 13:52:58.279853 ip-10-0-136-25 kubenswrapper[2569]: I0706 13:52:58.279378 2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-2\"" Jul 06 13:52:58.279853 ip-10-0-136-25 kubenswrapper[2569]: I0706 13:52:58.279649 2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-1\"" Jul 06 13:52:58.281359 ip-10-0-136-25 kubenswrapper[2569]: I0706 13:52:58.281338 2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-0\"" Jul 06 13:52:58.294508 ip-10-0-136-25 kubenswrapper[2569]: I0706 13:52:58.294483 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Jul 06 13:52:58.373638 ip-10-0-136-25 kubenswrapper[2569]: I0706 13:52:58.373606 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/b958bf61-a426-4f88-8a79-318280126542-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"b958bf61-a426-4f88-8a79-318280126542\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 06 13:52:58.373638 ip-10-0-136-25 kubenswrapper[2569]: I0706 13:52:58.373642 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b958bf61-a426-4f88-8a79-318280126542-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"b958bf61-a426-4f88-8a79-318280126542\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 06 13:52:58.373877 ip-10-0-136-25 kubenswrapper[2569]: I0706 13:52:58.373669 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xrqsm\" (UniqueName: \"kubernetes.io/projected/b958bf61-a426-4f88-8a79-318280126542-kube-api-access-xrqsm\") pod \"prometheus-k8s-0\" (UID: \"b958bf61-a426-4f88-8a79-318280126542\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 06 13:52:58.373877 ip-10-0-136-25 kubenswrapper[2569]: I0706 13:52:58.373705 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/b958bf61-a426-4f88-8a79-318280126542-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"b958bf61-a426-4f88-8a79-318280126542\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 06 13:52:58.373877 ip-10-0-136-25 kubenswrapper[2569]: I0706 13:52:58.373733 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/b958bf61-a426-4f88-8a79-318280126542-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"b958bf61-a426-4f88-8a79-318280126542\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 06 13:52:58.373877 ip-10-0-136-25 kubenswrapper[2569]: I0706 13:52:58.373763 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/b958bf61-a426-4f88-8a79-318280126542-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"b958bf61-a426-4f88-8a79-318280126542\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 06 13:52:58.373877 ip-10-0-136-25 kubenswrapper[2569]: I0706 13:52:58.373820 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/b958bf61-a426-4f88-8a79-318280126542-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"b958bf61-a426-4f88-8a79-318280126542\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 06 13:52:58.373877 ip-10-0-136-25 kubenswrapper[2569]: I0706 13:52:58.373873 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/b958bf61-a426-4f88-8a79-318280126542-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"b958bf61-a426-4f88-8a79-318280126542\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 06 13:52:58.374157 ip-10-0-136-25 kubenswrapper[2569]: I0706 13:52:58.373899 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/b958bf61-a426-4f88-8a79-318280126542-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"b958bf61-a426-4f88-8a79-318280126542\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 06 13:52:58.374157 ip-10-0-136-25 kubenswrapper[2569]: I0706 13:52:58.373917 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/b958bf61-a426-4f88-8a79-318280126542-config-out\") pod \"prometheus-k8s-0\" (UID: \"b958bf61-a426-4f88-8a79-318280126542\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 06 13:52:58.374157 ip-10-0-136-25 kubenswrapper[2569]: I0706 13:52:58.373932 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b958bf61-a426-4f88-8a79-318280126542-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"b958bf61-a426-4f88-8a79-318280126542\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 06 13:52:58.374157 ip-10-0-136-25 kubenswrapper[2569]: I0706 13:52:58.373950 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/b958bf61-a426-4f88-8a79-318280126542-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"b958bf61-a426-4f88-8a79-318280126542\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 06 13:52:58.374157 ip-10-0-136-25 kubenswrapper[2569]: I0706 13:52:58.374005 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b958bf61-a426-4f88-8a79-318280126542-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"b958bf61-a426-4f88-8a79-318280126542\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 06 13:52:58.374157 ip-10-0-136-25 kubenswrapper[2569]: I0706 13:52:58.374040 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/b958bf61-a426-4f88-8a79-318280126542-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"b958bf61-a426-4f88-8a79-318280126542\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 06 13:52:58.374157 ip-10-0-136-25 kubenswrapper[2569]: I0706 13:52:58.374076 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/b958bf61-a426-4f88-8a79-318280126542-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"b958bf61-a426-4f88-8a79-318280126542\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 06 13:52:58.374157 ip-10-0-136-25 kubenswrapper[2569]: I0706 13:52:58.374121 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/b958bf61-a426-4f88-8a79-318280126542-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"b958bf61-a426-4f88-8a79-318280126542\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 06 13:52:58.374638 ip-10-0-136-25 kubenswrapper[2569]: I0706 13:52:58.374169 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/b958bf61-a426-4f88-8a79-318280126542-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"b958bf61-a426-4f88-8a79-318280126542\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 06 13:52:58.374638 ip-10-0-136-25 kubenswrapper[2569]: I0706 13:52:58.374199 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/b958bf61-a426-4f88-8a79-318280126542-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"b958bf61-a426-4f88-8a79-318280126542\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 06 13:52:58.374638 ip-10-0-136-25 kubenswrapper[2569]: I0706 13:52:58.374246 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/b958bf61-a426-4f88-8a79-318280126542-web-config\") pod \"prometheus-k8s-0\" (UID: \"b958bf61-a426-4f88-8a79-318280126542\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 06 13:52:58.374638 ip-10-0-136-25 kubenswrapper[2569]: I0706 13:52:58.374283 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/b958bf61-a426-4f88-8a79-318280126542-config\") pod \"prometheus-k8s-0\" (UID: \"b958bf61-a426-4f88-8a79-318280126542\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 06 13:52:58.475682 ip-10-0-136-25 kubenswrapper[2569]: I0706 13:52:58.475165 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/b958bf61-a426-4f88-8a79-318280126542-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"b958bf61-a426-4f88-8a79-318280126542\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 06 13:52:58.475682 ip-10-0-136-25 kubenswrapper[2569]: I0706 13:52:58.475262 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/b958bf61-a426-4f88-8a79-318280126542-config-out\") pod \"prometheus-k8s-0\" (UID: \"b958bf61-a426-4f88-8a79-318280126542\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 06 13:52:58.475682 ip-10-0-136-25 kubenswrapper[2569]: I0706 13:52:58.475294 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b958bf61-a426-4f88-8a79-318280126542-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"b958bf61-a426-4f88-8a79-318280126542\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 06 13:52:58.475682 ip-10-0-136-25 kubenswrapper[2569]: I0706 13:52:58.475316 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/b958bf61-a426-4f88-8a79-318280126542-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"b958bf61-a426-4f88-8a79-318280126542\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 06 13:52:58.475682 ip-10-0-136-25 kubenswrapper[2569]: I0706 13:52:58.475600 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b958bf61-a426-4f88-8a79-318280126542-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"b958bf61-a426-4f88-8a79-318280126542\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 06 13:52:58.475682 ip-10-0-136-25 kubenswrapper[2569]: I0706 13:52:58.475638 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/b958bf61-a426-4f88-8a79-318280126542-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"b958bf61-a426-4f88-8a79-318280126542\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 06 13:52:58.475682 ip-10-0-136-25 kubenswrapper[2569]: I0706 13:52:58.475680 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/b958bf61-a426-4f88-8a79-318280126542-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"b958bf61-a426-4f88-8a79-318280126542\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 06 13:52:58.476145 ip-10-0-136-25 kubenswrapper[2569]: I0706 13:52:58.475728 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/b958bf61-a426-4f88-8a79-318280126542-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"b958bf61-a426-4f88-8a79-318280126542\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 06 13:52:58.476145 ip-10-0-136-25 kubenswrapper[2569]: I0706 13:52:58.475794 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/b958bf61-a426-4f88-8a79-318280126542-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"b958bf61-a426-4f88-8a79-318280126542\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 06 13:52:58.476145 ip-10-0-136-25 kubenswrapper[2569]: I0706 13:52:58.475825 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/b958bf61-a426-4f88-8a79-318280126542-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"b958bf61-a426-4f88-8a79-318280126542\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 06 13:52:58.476145 ip-10-0-136-25 kubenswrapper[2569]: I0706 13:52:58.475853 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/b958bf61-a426-4f88-8a79-318280126542-web-config\") pod \"prometheus-k8s-0\" (UID: \"b958bf61-a426-4f88-8a79-318280126542\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 06 13:52:58.476145 ip-10-0-136-25 kubenswrapper[2569]: I0706 13:52:58.475890 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/b958bf61-a426-4f88-8a79-318280126542-config\") pod \"prometheus-k8s-0\" (UID: \"b958bf61-a426-4f88-8a79-318280126542\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 06 13:52:58.476145 ip-10-0-136-25 kubenswrapper[2569]: I0706 13:52:58.475919 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/b958bf61-a426-4f88-8a79-318280126542-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"b958bf61-a426-4f88-8a79-318280126542\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 06 13:52:58.476145 ip-10-0-136-25 kubenswrapper[2569]: I0706 13:52:58.475940 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b958bf61-a426-4f88-8a79-318280126542-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"b958bf61-a426-4f88-8a79-318280126542\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 06 13:52:58.476145 ip-10-0-136-25 kubenswrapper[2569]: I0706 13:52:58.475969 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xrqsm\" (UniqueName: \"kubernetes.io/projected/b958bf61-a426-4f88-8a79-318280126542-kube-api-access-xrqsm\") pod \"prometheus-k8s-0\" (UID: \"b958bf61-a426-4f88-8a79-318280126542\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 06 13:52:58.476145 ip-10-0-136-25 kubenswrapper[2569]: I0706 13:52:58.476008 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/b958bf61-a426-4f88-8a79-318280126542-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"b958bf61-a426-4f88-8a79-318280126542\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 06 13:52:58.476145 ip-10-0-136-25 kubenswrapper[2569]: I0706 13:52:58.476036 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/b958bf61-a426-4f88-8a79-318280126542-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"b958bf61-a426-4f88-8a79-318280126542\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 06 13:52:58.476145 ip-10-0-136-25 kubenswrapper[2569]: I0706 13:52:58.476071 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/b958bf61-a426-4f88-8a79-318280126542-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"b958bf61-a426-4f88-8a79-318280126542\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 06 13:52:58.476145 ip-10-0-136-25 kubenswrapper[2569]: I0706 13:52:58.476079 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b958bf61-a426-4f88-8a79-318280126542-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"b958bf61-a426-4f88-8a79-318280126542\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 06 13:52:58.476145 ip-10-0-136-25 kubenswrapper[2569]: I0706 13:52:58.476111 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/b958bf61-a426-4f88-8a79-318280126542-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"b958bf61-a426-4f88-8a79-318280126542\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 06 13:52:58.476800 ip-10-0-136-25 kubenswrapper[2569]: I0706 13:52:58.476157 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/b958bf61-a426-4f88-8a79-318280126542-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"b958bf61-a426-4f88-8a79-318280126542\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 06 13:52:58.476800 ip-10-0-136-25 kubenswrapper[2569]: I0706 13:52:58.476380 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/b958bf61-a426-4f88-8a79-318280126542-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"b958bf61-a426-4f88-8a79-318280126542\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 06 13:52:58.476910 ip-10-0-136-25 kubenswrapper[2569]: I0706 13:52:58.476880 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/b958bf61-a426-4f88-8a79-318280126542-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"b958bf61-a426-4f88-8a79-318280126542\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 06 13:52:58.476910 ip-10-0-136-25 kubenswrapper[2569]: I0706 13:52:58.476887 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b958bf61-a426-4f88-8a79-318280126542-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"b958bf61-a426-4f88-8a79-318280126542\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 06 13:52:58.479894 ip-10-0-136-25 kubenswrapper[2569]: I0706 13:52:58.477308 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/b958bf61-a426-4f88-8a79-318280126542-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"b958bf61-a426-4f88-8a79-318280126542\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 06 13:52:58.479894 ip-10-0-136-25 kubenswrapper[2569]: I0706 13:52:58.477709 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b958bf61-a426-4f88-8a79-318280126542-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"b958bf61-a426-4f88-8a79-318280126542\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 06 13:52:58.479894 ip-10-0-136-25 kubenswrapper[2569]: I0706 13:52:58.478340 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/b958bf61-a426-4f88-8a79-318280126542-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"b958bf61-a426-4f88-8a79-318280126542\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 06 13:52:58.480415 ip-10-0-136-25 kubenswrapper[2569]: I0706 13:52:58.480397 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/b958bf61-a426-4f88-8a79-318280126542-config-out\") pod \"prometheus-k8s-0\" (UID: \"b958bf61-a426-4f88-8a79-318280126542\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 06 13:52:58.480569 ip-10-0-136-25 kubenswrapper[2569]: I0706 13:52:58.480532 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/b958bf61-a426-4f88-8a79-318280126542-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"b958bf61-a426-4f88-8a79-318280126542\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 06 13:52:58.484184 ip-10-0-136-25 kubenswrapper[2569]: I0706 13:52:58.484158 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/secret/b958bf61-a426-4f88-8a79-318280126542-config\") pod \"prometheus-k8s-0\" (UID: \"b958bf61-a426-4f88-8a79-318280126542\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 06 13:52:58.484299 ip-10-0-136-25 kubenswrapper[2569]: I0706 13:52:58.484187 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/b958bf61-a426-4f88-8a79-318280126542-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"b958bf61-a426-4f88-8a79-318280126542\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 06 13:52:58.484359 ip-10-0-136-25 kubenswrapper[2569]: I0706 13:52:58.484302 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/b958bf61-a426-4f88-8a79-318280126542-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"b958bf61-a426-4f88-8a79-318280126542\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 06 13:52:58.484694 ip-10-0-136-25 kubenswrapper[2569]: I0706 13:52:58.484654 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/b958bf61-a426-4f88-8a79-318280126542-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"b958bf61-a426-4f88-8a79-318280126542\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 06 13:52:58.484694 ip-10-0-136-25 kubenswrapper[2569]: I0706 13:52:58.484666 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/b958bf61-a426-4f88-8a79-318280126542-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"b958bf61-a426-4f88-8a79-318280126542\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 06 13:52:58.484853 ip-10-0-136-25 kubenswrapper[2569]: I0706 13:52:58.484757 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/b958bf61-a426-4f88-8a79-318280126542-web-config\") pod \"prometheus-k8s-0\" (UID: \"b958bf61-a426-4f88-8a79-318280126542\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 06 13:52:58.484853 ip-10-0-136-25 kubenswrapper[2569]: I0706 13:52:58.484841 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/b958bf61-a426-4f88-8a79-318280126542-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"b958bf61-a426-4f88-8a79-318280126542\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 06 13:52:58.484970 ip-10-0-136-25 kubenswrapper[2569]: I0706 13:52:58.484946 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/b958bf61-a426-4f88-8a79-318280126542-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"b958bf61-a426-4f88-8a79-318280126542\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 06 13:52:58.485091 ip-10-0-136-25 kubenswrapper[2569]: I0706 13:52:58.485072 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/b958bf61-a426-4f88-8a79-318280126542-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"b958bf61-a426-4f88-8a79-318280126542\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 06 13:52:58.485168 ip-10-0-136-25 kubenswrapper[2569]: I0706 13:52:58.485141 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/b958bf61-a426-4f88-8a79-318280126542-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"b958bf61-a426-4f88-8a79-318280126542\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 06 13:52:58.487350 ip-10-0-136-25 kubenswrapper[2569]: I0706 13:52:58.487326 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xrqsm\" (UniqueName: \"kubernetes.io/projected/b958bf61-a426-4f88-8a79-318280126542-kube-api-access-xrqsm\") pod \"prometheus-k8s-0\" (UID: \"b958bf61-a426-4f88-8a79-318280126542\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 06 13:52:58.509019 ip-10-0-136-25 kubenswrapper[2569]: I0706 13:52:58.508985 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-7d96487c86-rnclw" event={"ID":"35142749-f6cf-4c8c-b41b-324c97c06c4c","Type":"ContainerStarted","Data":"fb44985a21f6d605f7217217c1d3c0151b957c3e2473df471a896ce3f15e01a5"} Jul 06 13:52:58.509383 ip-10-0-136-25 kubenswrapper[2569]: I0706 13:52:58.509049 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-7d96487c86-rnclw" event={"ID":"35142749-f6cf-4c8c-b41b-324c97c06c4c","Type":"ContainerStarted","Data":"3a412d3c67a18b94324963ee9e18bde77ac1c86e5b7331fc295f7339a13693f5"} Jul 06 13:52:58.509383 ip-10-0-136-25 kubenswrapper[2569]: I0706 13:52:58.509075 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-7d96487c86-rnclw" event={"ID":"35142749-f6cf-4c8c-b41b-324c97c06c4c","Type":"ContainerStarted","Data":"b5b7819dc3430ecaa89b86596808970f27317edc501558739835b95459505aa1"} Jul 06 13:52:58.586951 ip-10-0-136-25 kubenswrapper[2569]: I0706 13:52:58.586863 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0" Jul 06 13:52:58.766829 ip-10-0-136-25 kubenswrapper[2569]: I0706 13:52:58.766791 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Jul 06 13:52:59.107147 ip-10-0-136-25 kubenswrapper[2569]: W0706 13:52:59.107097 2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb958bf61_a426_4f88_8a79_318280126542.slice/crio-11c7cc8493d5e89934787340d940c37aec05191e3a3d8d6a9b5d047e66a7ee91 WatchSource:0}: Error finding container 11c7cc8493d5e89934787340d940c37aec05191e3a3d8d6a9b5d047e66a7ee91: Status 404 returned error can't find the container with id 11c7cc8493d5e89934787340d940c37aec05191e3a3d8d6a9b5d047e66a7ee91 Jul 06 13:52:59.522385 ip-10-0-136-25 kubenswrapper[2569]: I0706 13:52:59.522348 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"0985dbd8-3efa-4c39-9f96-8a906a0fe55c","Type":"ContainerStarted","Data":"dc9aa7d37944bae166e453225ac53efd1a92314272d4abc9fe100d9216a50058"} Jul 06 13:52:59.522731 ip-10-0-136-25 kubenswrapper[2569]: I0706 13:52:59.522444 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"0985dbd8-3efa-4c39-9f96-8a906a0fe55c","Type":"ContainerStarted","Data":"0a51dd57c0208069e0257971ec59e56dd9299c83ba554ae2b8785441d2013533"} Jul 06 13:52:59.524740 ip-10-0-136-25 kubenswrapper[2569]: I0706 13:52:59.524462 2569 generic.go:356] "Generic (PLEG): container finished" podID="b958bf61-a426-4f88-8a79-318280126542" containerID="e1ab4427d197ff423c8e4185f16aa7d19d0c84ae571ef8c29a30b53b29671542" exitCode=0 Jul 06 13:52:59.524740 ip-10-0-136-25 kubenswrapper[2569]: I0706 13:52:59.524549 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"b958bf61-a426-4f88-8a79-318280126542","Type":"ContainerDied","Data":"e1ab4427d197ff423c8e4185f16aa7d19d0c84ae571ef8c29a30b53b29671542"} Jul 06 13:52:59.524740 ip-10-0-136-25 kubenswrapper[2569]: I0706 13:52:59.524615 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"b958bf61-a426-4f88-8a79-318280126542","Type":"ContainerStarted","Data":"11c7cc8493d5e89934787340d940c37aec05191e3a3d8d6a9b5d047e66a7ee91"} Jul 06 13:52:59.530607 ip-10-0-136-25 kubenswrapper[2569]: I0706 13:52:59.530579 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-7d96487c86-rnclw" event={"ID":"35142749-f6cf-4c8c-b41b-324c97c06c4c","Type":"ContainerStarted","Data":"cee43e89d452052b256e20dda59cdb0d5865ebb19d7bfbe87d98c5e45d071b58"} Jul 06 13:52:59.530688 ip-10-0-136-25 kubenswrapper[2569]: I0706 13:52:59.530658 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-7d96487c86-rnclw" event={"ID":"35142749-f6cf-4c8c-b41b-324c97c06c4c","Type":"ContainerStarted","Data":"594c0ad94927745f7be36e2cff82503d74822c1d496979b3278fa175847dc885"} Jul 06 13:53:00.540484 ip-10-0-136-25 kubenswrapper[2569]: I0706 13:53:00.540445 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"0985dbd8-3efa-4c39-9f96-8a906a0fe55c","Type":"ContainerStarted","Data":"4a37c09790087b541384d3db50e6f627eef67a92a4a36d0f806b6aee79a67009"} Jul 06 13:53:00.541034 ip-10-0-136-25 kubenswrapper[2569]: I0706 13:53:00.540565 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"0985dbd8-3efa-4c39-9f96-8a906a0fe55c","Type":"ContainerStarted","Data":"ef599c88133e1860afd7aceb2a0a749ed394c4616cfd12865745e480fdd526c3"} Jul 06 13:53:00.541034 ip-10-0-136-25 kubenswrapper[2569]: I0706 13:53:00.540612 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"0985dbd8-3efa-4c39-9f96-8a906a0fe55c","Type":"ContainerStarted","Data":"47adb58300b64c8bb0f580f0a0bdaaeed65cb5137e7441b39bcb0700a5e49d79"} Jul 06 13:53:00.541034 ip-10-0-136-25 kubenswrapper[2569]: I0706 13:53:00.540655 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"0985dbd8-3efa-4c39-9f96-8a906a0fe55c","Type":"ContainerStarted","Data":"4363150d5d322909780052194b5b561621f3be9e8a0d062be3b290281cc6e622"} Jul 06 13:53:00.544358 ip-10-0-136-25 kubenswrapper[2569]: I0706 13:53:00.544261 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-7d96487c86-rnclw" event={"ID":"35142749-f6cf-4c8c-b41b-324c97c06c4c","Type":"ContainerStarted","Data":"5ba51005408735f094617056cff110de5525ed246c16d3946436b4a6087211ee"} Jul 06 13:53:00.544500 ip-10-0-136-25 kubenswrapper[2569]: I0706 13:53:00.544430 2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/thanos-querier-7d96487c86-rnclw" Jul 06 13:53:00.569325 ip-10-0-136-25 kubenswrapper[2569]: I0706 13:53:00.569248 2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/alertmanager-main-0" podStartSLOduration=4.08818619 podStartE2EDuration="7.569190372s" totalImagesPullingTime="3.459695798s" totalInitContainerRuntime="21.308384ms" isStatefulPod=false podCreationTimestamp="2026-07-06 13:52:53 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 13:53:00.566544945 +0000 UTC m=+156.157106730" watchObservedRunningTime="2026-07-06 13:53:00.569190372 +0000 UTC m=+156.159752149" Jul 06 13:53:00.592062 ip-10-0-136-25 kubenswrapper[2569]: I0706 13:53:00.591767 2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/thanos-querier-7d96487c86-rnclw" podStartSLOduration=1.985486444 podStartE2EDuration="5.591741976s" totalImagesPullingTime="3.606255532s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-06 13:52:55 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 13:53:00.589452258 +0000 UTC m=+156.180014071" watchObservedRunningTime="2026-07-06 13:53:00.591741976 +0000 UTC m=+156.182303755" Jul 06 13:53:03.106577 ip-10-0-136-25 kubenswrapper[2569]: I0706 13:53:03.106524 2569 patch_prober.go:28] interesting pod/image-registry-7c57596984-r6bgq container/registry namespace/openshift-image-registry: Liveness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body={"errors":[{"code":"UNAVAILABLE","message":"service unavailable","detail":"health check failed: please see /debug/health"}]} Jul 06 13:53:03.107058 ip-10-0-136-25 kubenswrapper[2569]: I0706 13:53:03.106594 2569 prober.go:120] "Probe failed" probeType="Liveness" pod="openshift-image-registry/image-registry-7c57596984-r6bgq" podUID="83120e7b-a163-4061-a1c9-4b6d79ac47c5" containerName="registry" probeResult="failure" output="HTTP probe failed with statuscode: 503" Jul 06 13:53:03.323349 ip-10-0-136-25 kubenswrapper[2569]: E0706 13:53:03.323258 2569 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[metrics-tls], unattached volumes=[], failed to process volumes=[]: context deadline exceeded" pod="openshift-dns/dns-default-kbv9m" podUID="420fd5c7-d449-4869-960a-4b14a873c402" Jul 06 13:53:03.562223 ip-10-0-136-25 kubenswrapper[2569]: I0706 13:53:03.562142 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"b958bf61-a426-4f88-8a79-318280126542","Type":"ContainerStarted","Data":"38b5c18dc4173d78f48dd6c0b92c26ef4d1994462dbbbee24d92d5c5bf4b8b22"} Jul 06 13:53:03.562223 ip-10-0-136-25 kubenswrapper[2569]: I0706 13:53:03.562169 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-kbv9m" Jul 06 13:53:04.429068 ip-10-0-136-25 kubenswrapper[2569]: I0706 13:53:04.429026 2569 patch_prober.go:28] interesting pod/image-registry-7c57596984-r6bgq container/registry namespace/openshift-image-registry: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body={"errors":[{"code":"UNAVAILABLE","message":"service unavailable","detail":"health check failed: please see /debug/health"}]} Jul 06 13:53:04.429542 ip-10-0-136-25 kubenswrapper[2569]: I0706 13:53:04.429094 2569 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-image-registry/image-registry-7c57596984-r6bgq" podUID="83120e7b-a163-4061-a1c9-4b6d79ac47c5" containerName="registry" probeResult="failure" output="HTTP probe failed with statuscode: 503" Jul 06 13:53:04.570820 ip-10-0-136-25 kubenswrapper[2569]: I0706 13:53:04.570774 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"b958bf61-a426-4f88-8a79-318280126542","Type":"ContainerStarted","Data":"68acdea924854dcf3c90ecd92b451cb0b97fe47934007aeb7b804819d89a6e4b"} Jul 06 13:53:04.570988 ip-10-0-136-25 kubenswrapper[2569]: I0706 13:53:04.570862 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"b958bf61-a426-4f88-8a79-318280126542","Type":"ContainerStarted","Data":"4a911e6732e5fc7953e35b1935bce0e0785fa4e43e1559a3afa6d5af3a45edb3"} Jul 06 13:53:04.570988 ip-10-0-136-25 kubenswrapper[2569]: I0706 13:53:04.570907 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"b958bf61-a426-4f88-8a79-318280126542","Type":"ContainerStarted","Data":"cb9c1b6faf60392243d48d6313acb72e7dd5a407840405896b4bc85a5518fbd1"} Jul 06 13:53:04.570988 ip-10-0-136-25 kubenswrapper[2569]: I0706 13:53:04.570953 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"b958bf61-a426-4f88-8a79-318280126542","Type":"ContainerStarted","Data":"a95b56e8d5a128e827f4fdb56f61e5c0a13e51f3e63e5a4803f17c118e8f855a"} Jul 06 13:53:04.571146 ip-10-0-136-25 kubenswrapper[2569]: I0706 13:53:04.570993 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"b958bf61-a426-4f88-8a79-318280126542","Type":"ContainerStarted","Data":"9a6205213a3ffd386945d449a6c5658c4a864acbba41e8c357584dc086c95d4d"} Jul 06 13:53:04.605936 ip-10-0-136-25 kubenswrapper[2569]: I0706 13:53:04.605882 2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/prometheus-k8s-0" podStartSLOduration=2.738802299 podStartE2EDuration="6.605866018s" totalImagesPullingTime="3.841422696s" totalInitContainerRuntime="25.641023ms" isStatefulPod=false podCreationTimestamp="2026-07-06 13:52:58 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 13:53:04.603494076 +0000 UTC m=+160.194055845" watchObservedRunningTime="2026-07-06 13:53:04.605866018 +0000 UTC m=+160.196427800" Jul 06 13:53:06.556028 ip-10-0-136-25 kubenswrapper[2569]: I0706 13:53:06.556000 2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/thanos-querier-7d96487c86-rnclw" Jul 06 13:53:08.183726 ip-10-0-136-25 kubenswrapper[2569]: I0706 13:53:08.183687 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/420fd5c7-d449-4869-960a-4b14a873c402-metrics-tls\") pod \"dns-default-kbv9m\" (UID: \"420fd5c7-d449-4869-960a-4b14a873c402\") " pod="openshift-dns/dns-default-kbv9m" Jul 06 13:53:08.186953 ip-10-0-136-25 kubenswrapper[2569]: I0706 13:53:08.186925 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/420fd5c7-d449-4869-960a-4b14a873c402-metrics-tls\") pod \"dns-default-kbv9m\" (UID: \"420fd5c7-d449-4869-960a-4b14a873c402\") " pod="openshift-dns/dns-default-kbv9m" Jul 06 13:53:08.365314 ip-10-0-136-25 kubenswrapper[2569]: I0706 13:53:08.365280 2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-szq2s\"" Jul 06 13:53:08.373198 ip-10-0-136-25 kubenswrapper[2569]: I0706 13:53:08.373166 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-kbv9m" Jul 06 13:53:08.587432 ip-10-0-136-25 kubenswrapper[2569]: I0706 13:53:08.587403 2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/prometheus-k8s-0" Jul 06 13:53:13.105547 ip-10-0-136-25 kubenswrapper[2569]: I0706 13:53:13.105501 2569 patch_prober.go:28] interesting pod/image-registry-7c57596984-r6bgq container/registry namespace/openshift-image-registry: Liveness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body={"errors":[{"code":"UNAVAILABLE","message":"service unavailable","detail":"health check failed: please see /debug/health"}]} Jul 06 13:53:13.106001 ip-10-0-136-25 kubenswrapper[2569]: I0706 13:53:13.105563 2569 prober.go:120] "Probe failed" probeType="Liveness" pod="openshift-image-registry/image-registry-7c57596984-r6bgq" podUID="83120e7b-a163-4061-a1c9-4b6d79ac47c5" containerName="registry" probeResult="failure" output="HTTP probe failed with statuscode: 503" Jul 06 13:53:14.431671 ip-10-0-136-25 kubenswrapper[2569]: I0706 13:53:14.431611 2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-image-registry/image-registry-7c57596984-r6bgq" Jul 06 13:53:15.039771 ip-10-0-136-25 kubenswrapper[2569]: I0706 13:53:15.039733 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-kbv9m"] Jul 06 13:53:15.628576 ip-10-0-136-25 kubenswrapper[2569]: I0706 13:53:15.628529 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-kbv9m" event={"ID":"420fd5c7-d449-4869-960a-4b14a873c402","Type":"ContainerStarted","Data":"6d5291c9eb5ae14ba19d2f9f9aeea66545f88a550ea9fafe5e23e0ee2a78e939"} Jul 06 13:53:15.630431 ip-10-0-136-25 kubenswrapper[2569]: I0706 13:53:15.630401 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/downloads-685bfb5f64-nwr8k" event={"ID":"e95b78c0-73d7-4e1d-aeab-d59969a3a319","Type":"ContainerStarted","Data":"2506399db1b0e8343c290b28a90831cab1579fe344a0c47c0f6c7f9f0750d1ad"} Jul 06 13:53:15.630643 ip-10-0-136-25 kubenswrapper[2569]: I0706 13:53:15.630621 2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/downloads-685bfb5f64-nwr8k" Jul 06 13:53:15.647789 ip-10-0-136-25 kubenswrapper[2569]: I0706 13:53:15.647755 2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/downloads-685bfb5f64-nwr8k" Jul 06 13:53:15.655398 ip-10-0-136-25 kubenswrapper[2569]: I0706 13:53:15.655333 2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/downloads-685bfb5f64-nwr8k" podStartSLOduration=1.650369928 podStartE2EDuration="21.655313771s" totalImagesPullingTime="20.004943843s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-06 13:52:54 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 13:53:15.65377384 +0000 UTC m=+171.244335630" watchObservedRunningTime="2026-07-06 13:53:15.655313771 +0000 UTC m=+171.245875549" Jul 06 13:53:18.648557 ip-10-0-136-25 kubenswrapper[2569]: I0706 13:53:18.648518 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-kbv9m" event={"ID":"420fd5c7-d449-4869-960a-4b14a873c402","Type":"ContainerStarted","Data":"954e470460cad26dc43355e0eea54b4d22e9aef61c94e78b9ad76ebc9c29a4e4"} Jul 06 13:53:18.649098 ip-10-0-136-25 kubenswrapper[2569]: I0706 13:53:18.648587 2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-dns/dns-default-kbv9m" Jul 06 13:53:18.649098 ip-10-0-136-25 kubenswrapper[2569]: I0706 13:53:18.648611 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-kbv9m" event={"ID":"420fd5c7-d449-4869-960a-4b14a873c402","Type":"ContainerStarted","Data":"1ca83deb1d62b1042c07470460f8f9848f419ecc34840593b286f4f58e4becd5"} Jul 06 13:53:18.678328 ip-10-0-136-25 kubenswrapper[2569]: I0706 13:53:18.678269 2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/dns-default-kbv9m" podStartSLOduration=136.276007191 podStartE2EDuration="2m18.678243656s" totalImagesPullingTime="2.402236465s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-06 13:51:00 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 13:53:18.677585911 +0000 UTC m=+174.268147728" watchObservedRunningTime="2026-07-06 13:53:18.678243656 +0000 UTC m=+174.268805435" Jul 06 13:53:23.675104 ip-10-0-136-25 kubenswrapper[2569]: I0706 13:53:23.675068 2569 generic.go:356] "Generic (PLEG): container finished" podID="9746832c-8d02-489c-98f4-afb86bdc46b8" containerID="353a2ebd48aa742b9410dbe567df0daea75ddbb2db6ed241b68b69d517bc6e9a" exitCode=0 Jul 06 13:53:23.675645 ip-10-0-136-25 kubenswrapper[2569]: I0706 13:53:23.675147 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-5c847dcfc4-tdvhm" event={"ID":"9746832c-8d02-489c-98f4-afb86bdc46b8","Type":"ContainerDied","Data":"353a2ebd48aa742b9410dbe567df0daea75ddbb2db6ed241b68b69d517bc6e9a"} Jul 06 13:53:23.675645 ip-10-0-136-25 kubenswrapper[2569]: I0706 13:53:23.675583 2569 scope.go:117] "RemoveContainer" containerID="353a2ebd48aa742b9410dbe567df0daea75ddbb2db6ed241b68b69d517bc6e9a" Jul 06 13:53:24.680765 ip-10-0-136-25 kubenswrapper[2569]: I0706 13:53:24.680732 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-5c847dcfc4-tdvhm" event={"ID":"9746832c-8d02-489c-98f4-afb86bdc46b8","Type":"ContainerStarted","Data":"45392272989f04f882d97707dbcf34d5bf543e4264dd4271d46b847021e61ecc"} Jul 06 13:53:28.656593 ip-10-0-136-25 kubenswrapper[2569]: I0706 13:53:28.656560 2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-dns/dns-default-kbv9m" Jul 06 13:53:33.719035 ip-10-0-136-25 kubenswrapper[2569]: I0706 13:53:33.718957 2569 generic.go:356] "Generic (PLEG): container finished" podID="2e5b9096-4169-413a-b200-49f75eb977c0" containerID="07fbed486b752e25861952905d86825294ef004a5ec3eb6fbe794f86761c0bb6" exitCode=0 Jul 06 13:53:33.719035 ip-10-0-136-25 kubenswrapper[2569]: I0706 13:53:33.719017 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-846bf668f7-52nxm" event={"ID":"2e5b9096-4169-413a-b200-49f75eb977c0","Type":"ContainerDied","Data":"07fbed486b752e25861952905d86825294ef004a5ec3eb6fbe794f86761c0bb6"} Jul 06 13:53:33.719648 ip-10-0-136-25 kubenswrapper[2569]: I0706 13:53:33.719421 2569 scope.go:117] "RemoveContainer" containerID="07fbed486b752e25861952905d86825294ef004a5ec3eb6fbe794f86761c0bb6" Jul 06 13:53:34.725851 ip-10-0-136-25 kubenswrapper[2569]: I0706 13:53:34.725818 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-846bf668f7-52nxm" event={"ID":"2e5b9096-4169-413a-b200-49f75eb977c0","Type":"ContainerStarted","Data":"2a98d7f3591010df21e3aa96a5879a62ee9922eae1205f2374ca4284a3cf20d4"} Jul 06 13:53:58.588332 ip-10-0-136-25 kubenswrapper[2569]: I0706 13:53:58.588285 2569 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-monitoring/prometheus-k8s-0" Jul 06 13:53:58.604393 ip-10-0-136-25 kubenswrapper[2569]: I0706 13:53:58.604364 2569 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-monitoring/prometheus-k8s-0" Jul 06 13:53:58.837324 ip-10-0-136-25 kubenswrapper[2569]: I0706 13:53:58.837296 2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/prometheus-k8s-0" Jul 06 13:54:12.325446 ip-10-0-136-25 kubenswrapper[2569]: I0706 13:54:12.325409 2569 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Jul 06 13:54:12.325966 ip-10-0-136-25 kubenswrapper[2569]: I0706 13:54:12.325689 2569 kuberuntime_container.go:872] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="0985dbd8-3efa-4c39-9f96-8a906a0fe55c" containerName="alertmanager" containerID="cri-o://0a51dd57c0208069e0257971ec59e56dd9299c83ba554ae2b8785441d2013533" gracePeriod=120 Jul 06 13:54:12.325966 ip-10-0-136-25 kubenswrapper[2569]: I0706 13:54:12.325771 2569 kuberuntime_container.go:872] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="0985dbd8-3efa-4c39-9f96-8a906a0fe55c" containerName="kube-rbac-proxy-metric" containerID="cri-o://ef599c88133e1860afd7aceb2a0a749ed394c4616cfd12865745e480fdd526c3" gracePeriod=120 Jul 06 13:54:12.325966 ip-10-0-136-25 kubenswrapper[2569]: I0706 13:54:12.325802 2569 kuberuntime_container.go:872] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="0985dbd8-3efa-4c39-9f96-8a906a0fe55c" containerName="kube-rbac-proxy" containerID="cri-o://47adb58300b64c8bb0f580f0a0bdaaeed65cb5137e7441b39bcb0700a5e49d79" gracePeriod=120 Jul 06 13:54:12.325966 ip-10-0-136-25 kubenswrapper[2569]: I0706 13:54:12.325809 2569 kuberuntime_container.go:872] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="0985dbd8-3efa-4c39-9f96-8a906a0fe55c" containerName="prom-label-proxy" containerID="cri-o://4a37c09790087b541384d3db50e6f627eef67a92a4a36d0f806b6aee79a67009" gracePeriod=120 Jul 06 13:54:12.325966 ip-10-0-136-25 kubenswrapper[2569]: I0706 13:54:12.325782 2569 kuberuntime_container.go:872] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="0985dbd8-3efa-4c39-9f96-8a906a0fe55c" containerName="kube-rbac-proxy-web" containerID="cri-o://4363150d5d322909780052194b5b561621f3be9e8a0d062be3b290281cc6e622" gracePeriod=120 Jul 06 13:54:12.325966 ip-10-0-136-25 kubenswrapper[2569]: I0706 13:54:12.325809 2569 kuberuntime_container.go:872] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="0985dbd8-3efa-4c39-9f96-8a906a0fe55c" containerName="config-reloader" containerID="cri-o://dc9aa7d37944bae166e453225ac53efd1a92314272d4abc9fe100d9216a50058" gracePeriod=120 Jul 06 13:54:12.881729 ip-10-0-136-25 kubenswrapper[2569]: I0706 13:54:12.881690 2569 generic.go:356] "Generic (PLEG): container finished" podID="0985dbd8-3efa-4c39-9f96-8a906a0fe55c" containerID="4a37c09790087b541384d3db50e6f627eef67a92a4a36d0f806b6aee79a67009" exitCode=0 Jul 06 13:54:12.881729 ip-10-0-136-25 kubenswrapper[2569]: I0706 13:54:12.881716 2569 generic.go:356] "Generic (PLEG): container finished" podID="0985dbd8-3efa-4c39-9f96-8a906a0fe55c" containerID="47adb58300b64c8bb0f580f0a0bdaaeed65cb5137e7441b39bcb0700a5e49d79" exitCode=0 Jul 06 13:54:12.881729 ip-10-0-136-25 kubenswrapper[2569]: I0706 13:54:12.881723 2569 generic.go:356] "Generic (PLEG): container finished" podID="0985dbd8-3efa-4c39-9f96-8a906a0fe55c" containerID="dc9aa7d37944bae166e453225ac53efd1a92314272d4abc9fe100d9216a50058" exitCode=0 Jul 06 13:54:12.881729 ip-10-0-136-25 kubenswrapper[2569]: I0706 13:54:12.881732 2569 generic.go:356] "Generic (PLEG): container finished" podID="0985dbd8-3efa-4c39-9f96-8a906a0fe55c" containerID="0a51dd57c0208069e0257971ec59e56dd9299c83ba554ae2b8785441d2013533" exitCode=0 Jul 06 13:54:12.881988 ip-10-0-136-25 kubenswrapper[2569]: I0706 13:54:12.881761 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"0985dbd8-3efa-4c39-9f96-8a906a0fe55c","Type":"ContainerDied","Data":"4a37c09790087b541384d3db50e6f627eef67a92a4a36d0f806b6aee79a67009"} Jul 06 13:54:12.881988 ip-10-0-136-25 kubenswrapper[2569]: I0706 13:54:12.881823 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"0985dbd8-3efa-4c39-9f96-8a906a0fe55c","Type":"ContainerDied","Data":"47adb58300b64c8bb0f580f0a0bdaaeed65cb5137e7441b39bcb0700a5e49d79"} Jul 06 13:54:12.881988 ip-10-0-136-25 kubenswrapper[2569]: I0706 13:54:12.881849 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"0985dbd8-3efa-4c39-9f96-8a906a0fe55c","Type":"ContainerDied","Data":"dc9aa7d37944bae166e453225ac53efd1a92314272d4abc9fe100d9216a50058"} Jul 06 13:54:12.881988 ip-10-0-136-25 kubenswrapper[2569]: I0706 13:54:12.881873 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"0985dbd8-3efa-4c39-9f96-8a906a0fe55c","Type":"ContainerDied","Data":"0a51dd57c0208069e0257971ec59e56dd9299c83ba554ae2b8785441d2013533"} Jul 06 13:54:13.677673 ip-10-0-136-25 kubenswrapper[2569]: I0706 13:54:13.677650 2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0" Jul 06 13:54:13.790329 ip-10-0-136-25 kubenswrapper[2569]: I0706 13:54:13.790288 2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/0985dbd8-3efa-4c39-9f96-8a906a0fe55c-secret-alertmanager-kube-rbac-proxy-web\") pod \"0985dbd8-3efa-4c39-9f96-8a906a0fe55c\" (UID: \"0985dbd8-3efa-4c39-9f96-8a906a0fe55c\") " Jul 06 13:54:13.790518 ip-10-0-136-25 kubenswrapper[2569]: I0706 13:54:13.790339 2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/0985dbd8-3efa-4c39-9f96-8a906a0fe55c-secret-alertmanager-kube-rbac-proxy\") pod \"0985dbd8-3efa-4c39-9f96-8a906a0fe55c\" (UID: \"0985dbd8-3efa-4c39-9f96-8a906a0fe55c\") " Jul 06 13:54:13.790518 ip-10-0-136-25 kubenswrapper[2569]: I0706 13:54:13.790374 2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/0985dbd8-3efa-4c39-9f96-8a906a0fe55c-secret-alertmanager-kube-rbac-proxy-metric\") pod \"0985dbd8-3efa-4c39-9f96-8a906a0fe55c\" (UID: \"0985dbd8-3efa-4c39-9f96-8a906a0fe55c\") " Jul 06 13:54:13.790518 ip-10-0-136-25 kubenswrapper[2569]: I0706 13:54:13.790503 2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/0985dbd8-3efa-4c39-9f96-8a906a0fe55c-config-volume\") pod \"0985dbd8-3efa-4c39-9f96-8a906a0fe55c\" (UID: \"0985dbd8-3efa-4c39-9f96-8a906a0fe55c\") " Jul 06 13:54:13.790682 ip-10-0-136-25 kubenswrapper[2569]: I0706 13:54:13.790558 2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/0985dbd8-3efa-4c39-9f96-8a906a0fe55c-config-out\") pod \"0985dbd8-3efa-4c39-9f96-8a906a0fe55c\" (UID: \"0985dbd8-3efa-4c39-9f96-8a906a0fe55c\") " Jul 06 13:54:13.790682 ip-10-0-136-25 kubenswrapper[2569]: I0706 13:54:13.790596 2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/0985dbd8-3efa-4c39-9f96-8a906a0fe55c-alertmanager-trusted-ca-bundle\") pod \"0985dbd8-3efa-4c39-9f96-8a906a0fe55c\" (UID: \"0985dbd8-3efa-4c39-9f96-8a906a0fe55c\") " Jul 06 13:54:13.790682 ip-10-0-136-25 kubenswrapper[2569]: I0706 13:54:13.790626 2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/0985dbd8-3efa-4c39-9f96-8a906a0fe55c-alertmanager-main-db\") pod \"0985dbd8-3efa-4c39-9f96-8a906a0fe55c\" (UID: \"0985dbd8-3efa-4c39-9f96-8a906a0fe55c\") " Jul 06 13:54:13.790682 ip-10-0-136-25 kubenswrapper[2569]: I0706 13:54:13.790667 2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/0985dbd8-3efa-4c39-9f96-8a906a0fe55c-web-config\") pod \"0985dbd8-3efa-4c39-9f96-8a906a0fe55c\" (UID: \"0985dbd8-3efa-4c39-9f96-8a906a0fe55c\") " Jul 06 13:54:13.790886 ip-10-0-136-25 kubenswrapper[2569]: I0706 13:54:13.790698 2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-t6j84\" (UniqueName: \"kubernetes.io/projected/0985dbd8-3efa-4c39-9f96-8a906a0fe55c-kube-api-access-t6j84\") pod \"0985dbd8-3efa-4c39-9f96-8a906a0fe55c\" (UID: \"0985dbd8-3efa-4c39-9f96-8a906a0fe55c\") " Jul 06 13:54:13.790886 ip-10-0-136-25 kubenswrapper[2569]: I0706 13:54:13.790729 2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/0985dbd8-3efa-4c39-9f96-8a906a0fe55c-metrics-client-ca\") pod \"0985dbd8-3efa-4c39-9f96-8a906a0fe55c\" (UID: \"0985dbd8-3efa-4c39-9f96-8a906a0fe55c\") " Jul 06 13:54:13.790886 ip-10-0-136-25 kubenswrapper[2569]: I0706 13:54:13.790764 2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/0985dbd8-3efa-4c39-9f96-8a906a0fe55c-tls-assets\") pod \"0985dbd8-3efa-4c39-9f96-8a906a0fe55c\" (UID: \"0985dbd8-3efa-4c39-9f96-8a906a0fe55c\") " Jul 06 13:54:13.790886 ip-10-0-136-25 kubenswrapper[2569]: I0706 13:54:13.790831 2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/0985dbd8-3efa-4c39-9f96-8a906a0fe55c-secret-alertmanager-main-tls\") pod \"0985dbd8-3efa-4c39-9f96-8a906a0fe55c\" (UID: \"0985dbd8-3efa-4c39-9f96-8a906a0fe55c\") " Jul 06 13:54:13.790886 ip-10-0-136-25 kubenswrapper[2569]: I0706 13:54:13.790863 2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/0985dbd8-3efa-4c39-9f96-8a906a0fe55c-cluster-tls-config\") pod \"0985dbd8-3efa-4c39-9f96-8a906a0fe55c\" (UID: \"0985dbd8-3efa-4c39-9f96-8a906a0fe55c\") " Jul 06 13:54:13.791788 ip-10-0-136-25 kubenswrapper[2569]: I0706 13:54:13.791422 2569 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0985dbd8-3efa-4c39-9f96-8a906a0fe55c-alertmanager-main-db" (OuterVolumeSpecName: "alertmanager-main-db") pod "0985dbd8-3efa-4c39-9f96-8a906a0fe55c" (UID: "0985dbd8-3efa-4c39-9f96-8a906a0fe55c"). InnerVolumeSpecName "alertmanager-main-db". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 06 13:54:13.791788 ip-10-0-136-25 kubenswrapper[2569]: I0706 13:54:13.791534 2569 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0985dbd8-3efa-4c39-9f96-8a906a0fe55c-alertmanager-trusted-ca-bundle" (OuterVolumeSpecName: "alertmanager-trusted-ca-bundle") pod "0985dbd8-3efa-4c39-9f96-8a906a0fe55c" (UID: "0985dbd8-3efa-4c39-9f96-8a906a0fe55c"). InnerVolumeSpecName "alertmanager-trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 06 13:54:13.791788 ip-10-0-136-25 kubenswrapper[2569]: I0706 13:54:13.791547 2569 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0985dbd8-3efa-4c39-9f96-8a906a0fe55c-metrics-client-ca" (OuterVolumeSpecName: "metrics-client-ca") pod "0985dbd8-3efa-4c39-9f96-8a906a0fe55c" (UID: "0985dbd8-3efa-4c39-9f96-8a906a0fe55c"). InnerVolumeSpecName "metrics-client-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 06 13:54:13.793325 ip-10-0-136-25 kubenswrapper[2569]: I0706 13:54:13.793279 2569 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0985dbd8-3efa-4c39-9f96-8a906a0fe55c-secret-alertmanager-kube-rbac-proxy-web" (OuterVolumeSpecName: "secret-alertmanager-kube-rbac-proxy-web") pod "0985dbd8-3efa-4c39-9f96-8a906a0fe55c" (UID: "0985dbd8-3efa-4c39-9f96-8a906a0fe55c"). InnerVolumeSpecName "secret-alertmanager-kube-rbac-proxy-web". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 06 13:54:13.793666 ip-10-0-136-25 kubenswrapper[2569]: I0706 13:54:13.793625 2569 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0985dbd8-3efa-4c39-9f96-8a906a0fe55c-kube-api-access-t6j84" (OuterVolumeSpecName: "kube-api-access-t6j84") pod "0985dbd8-3efa-4c39-9f96-8a906a0fe55c" (UID: "0985dbd8-3efa-4c39-9f96-8a906a0fe55c"). InnerVolumeSpecName "kube-api-access-t6j84". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 06 13:54:13.793769 ip-10-0-136-25 kubenswrapper[2569]: I0706 13:54:13.793740 2569 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0985dbd8-3efa-4c39-9f96-8a906a0fe55c-config-volume" (OuterVolumeSpecName: "config-volume") pod "0985dbd8-3efa-4c39-9f96-8a906a0fe55c" (UID: "0985dbd8-3efa-4c39-9f96-8a906a0fe55c"). InnerVolumeSpecName "config-volume". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 06 13:54:13.794101 ip-10-0-136-25 kubenswrapper[2569]: I0706 13:54:13.794080 2569 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0985dbd8-3efa-4c39-9f96-8a906a0fe55c-secret-alertmanager-kube-rbac-proxy" (OuterVolumeSpecName: "secret-alertmanager-kube-rbac-proxy") pod "0985dbd8-3efa-4c39-9f96-8a906a0fe55c" (UID: "0985dbd8-3efa-4c39-9f96-8a906a0fe55c"). InnerVolumeSpecName "secret-alertmanager-kube-rbac-proxy". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 06 13:54:13.794721 ip-10-0-136-25 kubenswrapper[2569]: I0706 13:54:13.794689 2569 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0985dbd8-3efa-4c39-9f96-8a906a0fe55c-secret-alertmanager-kube-rbac-proxy-metric" (OuterVolumeSpecName: "secret-alertmanager-kube-rbac-proxy-metric") pod "0985dbd8-3efa-4c39-9f96-8a906a0fe55c" (UID: "0985dbd8-3efa-4c39-9f96-8a906a0fe55c"). InnerVolumeSpecName "secret-alertmanager-kube-rbac-proxy-metric". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 06 13:54:13.794811 ip-10-0-136-25 kubenswrapper[2569]: I0706 13:54:13.794772 2569 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0985dbd8-3efa-4c39-9f96-8a906a0fe55c-secret-alertmanager-main-tls" (OuterVolumeSpecName: "secret-alertmanager-main-tls") pod "0985dbd8-3efa-4c39-9f96-8a906a0fe55c" (UID: "0985dbd8-3efa-4c39-9f96-8a906a0fe55c"). InnerVolumeSpecName "secret-alertmanager-main-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 06 13:54:13.794862 ip-10-0-136-25 kubenswrapper[2569]: I0706 13:54:13.794804 2569 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0985dbd8-3efa-4c39-9f96-8a906a0fe55c-config-out" (OuterVolumeSpecName: "config-out") pod "0985dbd8-3efa-4c39-9f96-8a906a0fe55c" (UID: "0985dbd8-3efa-4c39-9f96-8a906a0fe55c"). InnerVolumeSpecName "config-out". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 06 13:54:13.795130 ip-10-0-136-25 kubenswrapper[2569]: I0706 13:54:13.795109 2569 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0985dbd8-3efa-4c39-9f96-8a906a0fe55c-tls-assets" (OuterVolumeSpecName: "tls-assets") pod "0985dbd8-3efa-4c39-9f96-8a906a0fe55c" (UID: "0985dbd8-3efa-4c39-9f96-8a906a0fe55c"). InnerVolumeSpecName "tls-assets". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 06 13:54:13.798023 ip-10-0-136-25 kubenswrapper[2569]: I0706 13:54:13.797994 2569 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0985dbd8-3efa-4c39-9f96-8a906a0fe55c-cluster-tls-config" (OuterVolumeSpecName: "cluster-tls-config") pod "0985dbd8-3efa-4c39-9f96-8a906a0fe55c" (UID: "0985dbd8-3efa-4c39-9f96-8a906a0fe55c"). InnerVolumeSpecName "cluster-tls-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 06 13:54:13.805022 ip-10-0-136-25 kubenswrapper[2569]: I0706 13:54:13.804997 2569 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0985dbd8-3efa-4c39-9f96-8a906a0fe55c-web-config" (OuterVolumeSpecName: "web-config") pod "0985dbd8-3efa-4c39-9f96-8a906a0fe55c" (UID: "0985dbd8-3efa-4c39-9f96-8a906a0fe55c"). InnerVolumeSpecName "web-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 06 13:54:13.889206 ip-10-0-136-25 kubenswrapper[2569]: I0706 13:54:13.889174 2569 generic.go:356] "Generic (PLEG): container finished" podID="0985dbd8-3efa-4c39-9f96-8a906a0fe55c" containerID="ef599c88133e1860afd7aceb2a0a749ed394c4616cfd12865745e480fdd526c3" exitCode=0 Jul 06 13:54:13.889206 ip-10-0-136-25 kubenswrapper[2569]: I0706 13:54:13.889199 2569 generic.go:356] "Generic (PLEG): container finished" podID="0985dbd8-3efa-4c39-9f96-8a906a0fe55c" containerID="4363150d5d322909780052194b5b561621f3be9e8a0d062be3b290281cc6e622" exitCode=0 Jul 06 13:54:13.889417 ip-10-0-136-25 kubenswrapper[2569]: I0706 13:54:13.889249 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"0985dbd8-3efa-4c39-9f96-8a906a0fe55c","Type":"ContainerDied","Data":"ef599c88133e1860afd7aceb2a0a749ed394c4616cfd12865745e480fdd526c3"} Jul 06 13:54:13.889417 ip-10-0-136-25 kubenswrapper[2569]: I0706 13:54:13.889290 2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0" Jul 06 13:54:13.889417 ip-10-0-136-25 kubenswrapper[2569]: I0706 13:54:13.889348 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"0985dbd8-3efa-4c39-9f96-8a906a0fe55c","Type":"ContainerDied","Data":"4363150d5d322909780052194b5b561621f3be9e8a0d062be3b290281cc6e622"} Jul 06 13:54:13.889417 ip-10-0-136-25 kubenswrapper[2569]: I0706 13:54:13.889383 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"0985dbd8-3efa-4c39-9f96-8a906a0fe55c","Type":"ContainerDied","Data":"81a917be48241ca46ef483310d97897a0e417ab6271062b85b76cf77fbebc3fc"} Jul 06 13:54:13.889417 ip-10-0-136-25 kubenswrapper[2569]: I0706 13:54:13.889413 2569 scope.go:117] "RemoveContainer" containerID="4a37c09790087b541384d3db50e6f627eef67a92a4a36d0f806b6aee79a67009" Jul 06 13:54:13.891719 ip-10-0-136-25 kubenswrapper[2569]: I0706 13:54:13.891697 2569 reconciler_common.go:299] "Volume detached for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/0985dbd8-3efa-4c39-9f96-8a906a0fe55c-secret-alertmanager-main-tls\") on node \"ip-10-0-136-25.ec2.internal\" DevicePath \"\"" Jul 06 13:54:13.891719 ip-10-0-136-25 kubenswrapper[2569]: I0706 13:54:13.891719 2569 reconciler_common.go:299] "Volume detached for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/0985dbd8-3efa-4c39-9f96-8a906a0fe55c-cluster-tls-config\") on node \"ip-10-0-136-25.ec2.internal\" DevicePath \"\"" Jul 06 13:54:13.891901 ip-10-0-136-25 kubenswrapper[2569]: I0706 13:54:13.891729 2569 reconciler_common.go:299] "Volume detached for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/0985dbd8-3efa-4c39-9f96-8a906a0fe55c-secret-alertmanager-kube-rbac-proxy-web\") on node \"ip-10-0-136-25.ec2.internal\" DevicePath \"\"" Jul 06 13:54:13.891901 ip-10-0-136-25 kubenswrapper[2569]: I0706 13:54:13.891739 2569 reconciler_common.go:299] "Volume detached for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/0985dbd8-3efa-4c39-9f96-8a906a0fe55c-secret-alertmanager-kube-rbac-proxy\") on node \"ip-10-0-136-25.ec2.internal\" DevicePath \"\"" Jul 06 13:54:13.891901 ip-10-0-136-25 kubenswrapper[2569]: I0706 13:54:13.891748 2569 reconciler_common.go:299] "Volume detached for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/0985dbd8-3efa-4c39-9f96-8a906a0fe55c-secret-alertmanager-kube-rbac-proxy-metric\") on node \"ip-10-0-136-25.ec2.internal\" DevicePath \"\"" Jul 06 13:54:13.891901 ip-10-0-136-25 kubenswrapper[2569]: I0706 13:54:13.891757 2569 reconciler_common.go:299] "Volume detached for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/0985dbd8-3efa-4c39-9f96-8a906a0fe55c-config-volume\") on node \"ip-10-0-136-25.ec2.internal\" DevicePath \"\"" Jul 06 13:54:13.891901 ip-10-0-136-25 kubenswrapper[2569]: I0706 13:54:13.891766 2569 reconciler_common.go:299] "Volume detached for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/0985dbd8-3efa-4c39-9f96-8a906a0fe55c-config-out\") on node \"ip-10-0-136-25.ec2.internal\" DevicePath \"\"" Jul 06 13:54:13.891901 ip-10-0-136-25 kubenswrapper[2569]: I0706 13:54:13.891778 2569 reconciler_common.go:299] "Volume detached for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/0985dbd8-3efa-4c39-9f96-8a906a0fe55c-alertmanager-trusted-ca-bundle\") on node \"ip-10-0-136-25.ec2.internal\" DevicePath \"\"" Jul 06 13:54:13.891901 ip-10-0-136-25 kubenswrapper[2569]: I0706 13:54:13.891788 2569 reconciler_common.go:299] "Volume detached for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/0985dbd8-3efa-4c39-9f96-8a906a0fe55c-alertmanager-main-db\") on node \"ip-10-0-136-25.ec2.internal\" DevicePath \"\"" Jul 06 13:54:13.891901 ip-10-0-136-25 kubenswrapper[2569]: I0706 13:54:13.891798 2569 reconciler_common.go:299] "Volume detached for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/0985dbd8-3efa-4c39-9f96-8a906a0fe55c-web-config\") on node \"ip-10-0-136-25.ec2.internal\" DevicePath \"\"" Jul 06 13:54:13.891901 ip-10-0-136-25 kubenswrapper[2569]: I0706 13:54:13.891806 2569 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-t6j84\" (UniqueName: \"kubernetes.io/projected/0985dbd8-3efa-4c39-9f96-8a906a0fe55c-kube-api-access-t6j84\") on node \"ip-10-0-136-25.ec2.internal\" DevicePath \"\"" Jul 06 13:54:13.891901 ip-10-0-136-25 kubenswrapper[2569]: I0706 13:54:13.891814 2569 reconciler_common.go:299] "Volume detached for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/0985dbd8-3efa-4c39-9f96-8a906a0fe55c-metrics-client-ca\") on node \"ip-10-0-136-25.ec2.internal\" DevicePath \"\"" Jul 06 13:54:13.891901 ip-10-0-136-25 kubenswrapper[2569]: I0706 13:54:13.891821 2569 reconciler_common.go:299] "Volume detached for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/0985dbd8-3efa-4c39-9f96-8a906a0fe55c-tls-assets\") on node \"ip-10-0-136-25.ec2.internal\" DevicePath \"\"" Jul 06 13:54:13.898685 ip-10-0-136-25 kubenswrapper[2569]: I0706 13:54:13.898658 2569 scope.go:117] "RemoveContainer" containerID="ef599c88133e1860afd7aceb2a0a749ed394c4616cfd12865745e480fdd526c3" Jul 06 13:54:13.905910 ip-10-0-136-25 kubenswrapper[2569]: I0706 13:54:13.905892 2569 scope.go:117] "RemoveContainer" containerID="47adb58300b64c8bb0f580f0a0bdaaeed65cb5137e7441b39bcb0700a5e49d79" Jul 06 13:54:13.913281 ip-10-0-136-25 kubenswrapper[2569]: I0706 13:54:13.913265 2569 scope.go:117] "RemoveContainer" containerID="4363150d5d322909780052194b5b561621f3be9e8a0d062be3b290281cc6e622" Jul 06 13:54:13.914914 ip-10-0-136-25 kubenswrapper[2569]: I0706 13:54:13.914891 2569 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Jul 06 13:54:13.920006 ip-10-0-136-25 kubenswrapper[2569]: I0706 13:54:13.919986 2569 scope.go:117] "RemoveContainer" containerID="dc9aa7d37944bae166e453225ac53efd1a92314272d4abc9fe100d9216a50058" Jul 06 13:54:13.922720 ip-10-0-136-25 kubenswrapper[2569]: I0706 13:54:13.922700 2569 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Jul 06 13:54:13.927224 ip-10-0-136-25 kubenswrapper[2569]: I0706 13:54:13.927192 2569 scope.go:117] "RemoveContainer" containerID="0a51dd57c0208069e0257971ec59e56dd9299c83ba554ae2b8785441d2013533" Jul 06 13:54:13.933981 ip-10-0-136-25 kubenswrapper[2569]: I0706 13:54:13.933963 2569 scope.go:117] "RemoveContainer" containerID="1232f2f32291fd7e7850626a6711cca4dbe7a6318a4bf635ab6d25c44f5574f5" Jul 06 13:54:13.940613 ip-10-0-136-25 kubenswrapper[2569]: I0706 13:54:13.940598 2569 scope.go:117] "RemoveContainer" containerID="4a37c09790087b541384d3db50e6f627eef67a92a4a36d0f806b6aee79a67009" Jul 06 13:54:13.940891 ip-10-0-136-25 kubenswrapper[2569]: E0706 13:54:13.940867 2569 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4a37c09790087b541384d3db50e6f627eef67a92a4a36d0f806b6aee79a67009\": container with ID starting with 4a37c09790087b541384d3db50e6f627eef67a92a4a36d0f806b6aee79a67009 not found: ID does not exist" containerID="4a37c09790087b541384d3db50e6f627eef67a92a4a36d0f806b6aee79a67009" Jul 06 13:54:13.940936 ip-10-0-136-25 kubenswrapper[2569]: I0706 13:54:13.940900 2569 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4a37c09790087b541384d3db50e6f627eef67a92a4a36d0f806b6aee79a67009"} err="failed to get container status \"4a37c09790087b541384d3db50e6f627eef67a92a4a36d0f806b6aee79a67009\": rpc error: code = NotFound desc = could not find container \"4a37c09790087b541384d3db50e6f627eef67a92a4a36d0f806b6aee79a67009\": container with ID starting with 4a37c09790087b541384d3db50e6f627eef67a92a4a36d0f806b6aee79a67009 not found: ID does not exist" Jul 06 13:54:13.940936 ip-10-0-136-25 kubenswrapper[2569]: I0706 13:54:13.940932 2569 scope.go:117] "RemoveContainer" containerID="ef599c88133e1860afd7aceb2a0a749ed394c4616cfd12865745e480fdd526c3" Jul 06 13:54:13.941144 ip-10-0-136-25 kubenswrapper[2569]: E0706 13:54:13.941123 2569 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ef599c88133e1860afd7aceb2a0a749ed394c4616cfd12865745e480fdd526c3\": container with ID starting with ef599c88133e1860afd7aceb2a0a749ed394c4616cfd12865745e480fdd526c3 not found: ID does not exist" containerID="ef599c88133e1860afd7aceb2a0a749ed394c4616cfd12865745e480fdd526c3" Jul 06 13:54:13.941193 ip-10-0-136-25 kubenswrapper[2569]: I0706 13:54:13.941153 2569 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ef599c88133e1860afd7aceb2a0a749ed394c4616cfd12865745e480fdd526c3"} err="failed to get container status \"ef599c88133e1860afd7aceb2a0a749ed394c4616cfd12865745e480fdd526c3\": rpc error: code = NotFound desc = could not find container \"ef599c88133e1860afd7aceb2a0a749ed394c4616cfd12865745e480fdd526c3\": container with ID starting with ef599c88133e1860afd7aceb2a0a749ed394c4616cfd12865745e480fdd526c3 not found: ID does not exist" Jul 06 13:54:13.941193 ip-10-0-136-25 kubenswrapper[2569]: I0706 13:54:13.941172 2569 scope.go:117] "RemoveContainer" containerID="47adb58300b64c8bb0f580f0a0bdaaeed65cb5137e7441b39bcb0700a5e49d79" Jul 06 13:54:13.941446 ip-10-0-136-25 kubenswrapper[2569]: E0706 13:54:13.941428 2569 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"47adb58300b64c8bb0f580f0a0bdaaeed65cb5137e7441b39bcb0700a5e49d79\": container with ID starting with 47adb58300b64c8bb0f580f0a0bdaaeed65cb5137e7441b39bcb0700a5e49d79 not found: ID does not exist" containerID="47adb58300b64c8bb0f580f0a0bdaaeed65cb5137e7441b39bcb0700a5e49d79" Jul 06 13:54:13.941506 ip-10-0-136-25 kubenswrapper[2569]: I0706 13:54:13.941453 2569 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"47adb58300b64c8bb0f580f0a0bdaaeed65cb5137e7441b39bcb0700a5e49d79"} err="failed to get container status \"47adb58300b64c8bb0f580f0a0bdaaeed65cb5137e7441b39bcb0700a5e49d79\": rpc error: code = NotFound desc = could not find container \"47adb58300b64c8bb0f580f0a0bdaaeed65cb5137e7441b39bcb0700a5e49d79\": container with ID starting with 47adb58300b64c8bb0f580f0a0bdaaeed65cb5137e7441b39bcb0700a5e49d79 not found: ID does not exist" Jul 06 13:54:13.941506 ip-10-0-136-25 kubenswrapper[2569]: I0706 13:54:13.941468 2569 scope.go:117] "RemoveContainer" containerID="4363150d5d322909780052194b5b561621f3be9e8a0d062be3b290281cc6e622" Jul 06 13:54:13.941773 ip-10-0-136-25 kubenswrapper[2569]: E0706 13:54:13.941748 2569 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4363150d5d322909780052194b5b561621f3be9e8a0d062be3b290281cc6e622\": container with ID starting with 4363150d5d322909780052194b5b561621f3be9e8a0d062be3b290281cc6e622 not found: ID does not exist" containerID="4363150d5d322909780052194b5b561621f3be9e8a0d062be3b290281cc6e622" Jul 06 13:54:13.941854 ip-10-0-136-25 kubenswrapper[2569]: I0706 13:54:13.941777 2569 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4363150d5d322909780052194b5b561621f3be9e8a0d062be3b290281cc6e622"} err="failed to get container status \"4363150d5d322909780052194b5b561621f3be9e8a0d062be3b290281cc6e622\": rpc error: code = NotFound desc = could not find container \"4363150d5d322909780052194b5b561621f3be9e8a0d062be3b290281cc6e622\": container with ID starting with 4363150d5d322909780052194b5b561621f3be9e8a0d062be3b290281cc6e622 not found: ID does not exist" Jul 06 13:54:13.941854 ip-10-0-136-25 kubenswrapper[2569]: I0706 13:54:13.941795 2569 scope.go:117] "RemoveContainer" containerID="dc9aa7d37944bae166e453225ac53efd1a92314272d4abc9fe100d9216a50058" Jul 06 13:54:13.942121 ip-10-0-136-25 kubenswrapper[2569]: E0706 13:54:13.942094 2569 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"dc9aa7d37944bae166e453225ac53efd1a92314272d4abc9fe100d9216a50058\": container with ID starting with dc9aa7d37944bae166e453225ac53efd1a92314272d4abc9fe100d9216a50058 not found: ID does not exist" containerID="dc9aa7d37944bae166e453225ac53efd1a92314272d4abc9fe100d9216a50058" Jul 06 13:54:13.942163 ip-10-0-136-25 kubenswrapper[2569]: I0706 13:54:13.942125 2569 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"dc9aa7d37944bae166e453225ac53efd1a92314272d4abc9fe100d9216a50058"} err="failed to get container status \"dc9aa7d37944bae166e453225ac53efd1a92314272d4abc9fe100d9216a50058\": rpc error: code = NotFound desc = could not find container \"dc9aa7d37944bae166e453225ac53efd1a92314272d4abc9fe100d9216a50058\": container with ID starting with dc9aa7d37944bae166e453225ac53efd1a92314272d4abc9fe100d9216a50058 not found: ID does not exist" Jul 06 13:54:13.942163 ip-10-0-136-25 kubenswrapper[2569]: I0706 13:54:13.942139 2569 scope.go:117] "RemoveContainer" containerID="0a51dd57c0208069e0257971ec59e56dd9299c83ba554ae2b8785441d2013533" Jul 06 13:54:13.942406 ip-10-0-136-25 kubenswrapper[2569]: E0706 13:54:13.942389 2569 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0a51dd57c0208069e0257971ec59e56dd9299c83ba554ae2b8785441d2013533\": container with ID starting with 0a51dd57c0208069e0257971ec59e56dd9299c83ba554ae2b8785441d2013533 not found: ID does not exist" containerID="0a51dd57c0208069e0257971ec59e56dd9299c83ba554ae2b8785441d2013533" Jul 06 13:54:13.942459 ip-10-0-136-25 kubenswrapper[2569]: I0706 13:54:13.942410 2569 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0a51dd57c0208069e0257971ec59e56dd9299c83ba554ae2b8785441d2013533"} err="failed to get container status \"0a51dd57c0208069e0257971ec59e56dd9299c83ba554ae2b8785441d2013533\": rpc error: code = NotFound desc = could not find container \"0a51dd57c0208069e0257971ec59e56dd9299c83ba554ae2b8785441d2013533\": container with ID starting with 0a51dd57c0208069e0257971ec59e56dd9299c83ba554ae2b8785441d2013533 not found: ID does not exist" Jul 06 13:54:13.942459 ip-10-0-136-25 kubenswrapper[2569]: I0706 13:54:13.942424 2569 scope.go:117] "RemoveContainer" containerID="1232f2f32291fd7e7850626a6711cca4dbe7a6318a4bf635ab6d25c44f5574f5" Jul 06 13:54:13.942635 ip-10-0-136-25 kubenswrapper[2569]: E0706 13:54:13.942617 2569 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1232f2f32291fd7e7850626a6711cca4dbe7a6318a4bf635ab6d25c44f5574f5\": container with ID starting with 1232f2f32291fd7e7850626a6711cca4dbe7a6318a4bf635ab6d25c44f5574f5 not found: ID does not exist" containerID="1232f2f32291fd7e7850626a6711cca4dbe7a6318a4bf635ab6d25c44f5574f5" Jul 06 13:54:13.942673 ip-10-0-136-25 kubenswrapper[2569]: I0706 13:54:13.942638 2569 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1232f2f32291fd7e7850626a6711cca4dbe7a6318a4bf635ab6d25c44f5574f5"} err="failed to get container status \"1232f2f32291fd7e7850626a6711cca4dbe7a6318a4bf635ab6d25c44f5574f5\": rpc error: code = NotFound desc = could not find container \"1232f2f32291fd7e7850626a6711cca4dbe7a6318a4bf635ab6d25c44f5574f5\": container with ID starting with 1232f2f32291fd7e7850626a6711cca4dbe7a6318a4bf635ab6d25c44f5574f5 not found: ID does not exist" Jul 06 13:54:13.942673 ip-10-0-136-25 kubenswrapper[2569]: I0706 13:54:13.942653 2569 scope.go:117] "RemoveContainer" containerID="4a37c09790087b541384d3db50e6f627eef67a92a4a36d0f806b6aee79a67009" Jul 06 13:54:13.942855 ip-10-0-136-25 kubenswrapper[2569]: I0706 13:54:13.942839 2569 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4a37c09790087b541384d3db50e6f627eef67a92a4a36d0f806b6aee79a67009"} err="failed to get container status \"4a37c09790087b541384d3db50e6f627eef67a92a4a36d0f806b6aee79a67009\": rpc error: code = NotFound desc = could not find container \"4a37c09790087b541384d3db50e6f627eef67a92a4a36d0f806b6aee79a67009\": container with ID starting with 4a37c09790087b541384d3db50e6f627eef67a92a4a36d0f806b6aee79a67009 not found: ID does not exist" Jul 06 13:54:13.942895 ip-10-0-136-25 kubenswrapper[2569]: I0706 13:54:13.942856 2569 scope.go:117] "RemoveContainer" containerID="ef599c88133e1860afd7aceb2a0a749ed394c4616cfd12865745e480fdd526c3" Jul 06 13:54:13.943065 ip-10-0-136-25 kubenswrapper[2569]: I0706 13:54:13.943047 2569 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ef599c88133e1860afd7aceb2a0a749ed394c4616cfd12865745e480fdd526c3"} err="failed to get container status \"ef599c88133e1860afd7aceb2a0a749ed394c4616cfd12865745e480fdd526c3\": rpc error: code = NotFound desc = could not find container \"ef599c88133e1860afd7aceb2a0a749ed394c4616cfd12865745e480fdd526c3\": container with ID starting with ef599c88133e1860afd7aceb2a0a749ed394c4616cfd12865745e480fdd526c3 not found: ID does not exist" Jul 06 13:54:13.943115 ip-10-0-136-25 kubenswrapper[2569]: I0706 13:54:13.943066 2569 scope.go:117] "RemoveContainer" containerID="47adb58300b64c8bb0f580f0a0bdaaeed65cb5137e7441b39bcb0700a5e49d79" Jul 06 13:54:13.943309 ip-10-0-136-25 kubenswrapper[2569]: I0706 13:54:13.943294 2569 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"47adb58300b64c8bb0f580f0a0bdaaeed65cb5137e7441b39bcb0700a5e49d79"} err="failed to get container status \"47adb58300b64c8bb0f580f0a0bdaaeed65cb5137e7441b39bcb0700a5e49d79\": rpc error: code = NotFound desc = could not find container \"47adb58300b64c8bb0f580f0a0bdaaeed65cb5137e7441b39bcb0700a5e49d79\": container with ID starting with 47adb58300b64c8bb0f580f0a0bdaaeed65cb5137e7441b39bcb0700a5e49d79 not found: ID does not exist" Jul 06 13:54:13.943361 ip-10-0-136-25 kubenswrapper[2569]: I0706 13:54:13.943310 2569 scope.go:117] "RemoveContainer" containerID="4363150d5d322909780052194b5b561621f3be9e8a0d062be3b290281cc6e622" Jul 06 13:54:13.943514 ip-10-0-136-25 kubenswrapper[2569]: I0706 13:54:13.943499 2569 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4363150d5d322909780052194b5b561621f3be9e8a0d062be3b290281cc6e622"} err="failed to get container status \"4363150d5d322909780052194b5b561621f3be9e8a0d062be3b290281cc6e622\": rpc error: code = NotFound desc = could not find container \"4363150d5d322909780052194b5b561621f3be9e8a0d062be3b290281cc6e622\": container with ID starting with 4363150d5d322909780052194b5b561621f3be9e8a0d062be3b290281cc6e622 not found: ID does not exist" Jul 06 13:54:13.943514 ip-10-0-136-25 kubenswrapper[2569]: I0706 13:54:13.943513 2569 scope.go:117] "RemoveContainer" containerID="dc9aa7d37944bae166e453225ac53efd1a92314272d4abc9fe100d9216a50058" Jul 06 13:54:13.943706 ip-10-0-136-25 kubenswrapper[2569]: I0706 13:54:13.943691 2569 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"dc9aa7d37944bae166e453225ac53efd1a92314272d4abc9fe100d9216a50058"} err="failed to get container status \"dc9aa7d37944bae166e453225ac53efd1a92314272d4abc9fe100d9216a50058\": rpc error: code = NotFound desc = could not find container \"dc9aa7d37944bae166e453225ac53efd1a92314272d4abc9fe100d9216a50058\": container with ID starting with dc9aa7d37944bae166e453225ac53efd1a92314272d4abc9fe100d9216a50058 not found: ID does not exist" Jul 06 13:54:13.943747 ip-10-0-136-25 kubenswrapper[2569]: I0706 13:54:13.943706 2569 scope.go:117] "RemoveContainer" containerID="0a51dd57c0208069e0257971ec59e56dd9299c83ba554ae2b8785441d2013533" Jul 06 13:54:13.943928 ip-10-0-136-25 kubenswrapper[2569]: I0706 13:54:13.943911 2569 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0a51dd57c0208069e0257971ec59e56dd9299c83ba554ae2b8785441d2013533"} err="failed to get container status \"0a51dd57c0208069e0257971ec59e56dd9299c83ba554ae2b8785441d2013533\": rpc error: code = NotFound desc = could not find container \"0a51dd57c0208069e0257971ec59e56dd9299c83ba554ae2b8785441d2013533\": container with ID starting with 0a51dd57c0208069e0257971ec59e56dd9299c83ba554ae2b8785441d2013533 not found: ID does not exist" Jul 06 13:54:13.943974 ip-10-0-136-25 kubenswrapper[2569]: I0706 13:54:13.943930 2569 scope.go:117] "RemoveContainer" containerID="1232f2f32291fd7e7850626a6711cca4dbe7a6318a4bf635ab6d25c44f5574f5" Jul 06 13:54:13.944130 ip-10-0-136-25 kubenswrapper[2569]: I0706 13:54:13.944113 2569 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1232f2f32291fd7e7850626a6711cca4dbe7a6318a4bf635ab6d25c44f5574f5"} err="failed to get container status \"1232f2f32291fd7e7850626a6711cca4dbe7a6318a4bf635ab6d25c44f5574f5\": rpc error: code = NotFound desc = could not find container \"1232f2f32291fd7e7850626a6711cca4dbe7a6318a4bf635ab6d25c44f5574f5\": container with ID starting with 1232f2f32291fd7e7850626a6711cca4dbe7a6318a4bf635ab6d25c44f5574f5 not found: ID does not exist" Jul 06 13:54:13.951714 ip-10-0-136-25 kubenswrapper[2569]: I0706 13:54:13.951693 2569 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Jul 06 13:54:13.952009 ip-10-0-136-25 kubenswrapper[2569]: I0706 13:54:13.951996 2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0985dbd8-3efa-4c39-9f96-8a906a0fe55c" containerName="kube-rbac-proxy" Jul 06 13:54:13.952050 ip-10-0-136-25 kubenswrapper[2569]: I0706 13:54:13.952011 2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="0985dbd8-3efa-4c39-9f96-8a906a0fe55c" containerName="kube-rbac-proxy" Jul 06 13:54:13.952050 ip-10-0-136-25 kubenswrapper[2569]: I0706 13:54:13.952022 2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0985dbd8-3efa-4c39-9f96-8a906a0fe55c" containerName="alertmanager" Jul 06 13:54:13.952050 ip-10-0-136-25 kubenswrapper[2569]: I0706 13:54:13.952027 2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="0985dbd8-3efa-4c39-9f96-8a906a0fe55c" containerName="alertmanager" Jul 06 13:54:13.952050 ip-10-0-136-25 kubenswrapper[2569]: I0706 13:54:13.952040 2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0985dbd8-3efa-4c39-9f96-8a906a0fe55c" containerName="prom-label-proxy" Jul 06 13:54:13.952050 ip-10-0-136-25 kubenswrapper[2569]: I0706 13:54:13.952046 2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="0985dbd8-3efa-4c39-9f96-8a906a0fe55c" containerName="prom-label-proxy" Jul 06 13:54:13.952205 ip-10-0-136-25 kubenswrapper[2569]: I0706 13:54:13.952053 2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0985dbd8-3efa-4c39-9f96-8a906a0fe55c" containerName="config-reloader" Jul 06 13:54:13.952205 ip-10-0-136-25 kubenswrapper[2569]: I0706 13:54:13.952058 2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="0985dbd8-3efa-4c39-9f96-8a906a0fe55c" containerName="config-reloader" Jul 06 13:54:13.952205 ip-10-0-136-25 kubenswrapper[2569]: I0706 13:54:13.952066 2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0985dbd8-3efa-4c39-9f96-8a906a0fe55c" containerName="kube-rbac-proxy-web" Jul 06 13:54:13.952205 ip-10-0-136-25 kubenswrapper[2569]: I0706 13:54:13.952070 2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="0985dbd8-3efa-4c39-9f96-8a906a0fe55c" containerName="kube-rbac-proxy-web" Jul 06 13:54:13.952205 ip-10-0-136-25 kubenswrapper[2569]: I0706 13:54:13.952077 2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0985dbd8-3efa-4c39-9f96-8a906a0fe55c" containerName="init-config-reloader" Jul 06 13:54:13.952205 ip-10-0-136-25 kubenswrapper[2569]: I0706 13:54:13.952082 2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="0985dbd8-3efa-4c39-9f96-8a906a0fe55c" containerName="init-config-reloader" Jul 06 13:54:13.952205 ip-10-0-136-25 kubenswrapper[2569]: I0706 13:54:13.952088 2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0985dbd8-3efa-4c39-9f96-8a906a0fe55c" containerName="kube-rbac-proxy-metric" Jul 06 13:54:13.952205 ip-10-0-136-25 kubenswrapper[2569]: I0706 13:54:13.952093 2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="0985dbd8-3efa-4c39-9f96-8a906a0fe55c" containerName="kube-rbac-proxy-metric" Jul 06 13:54:13.952205 ip-10-0-136-25 kubenswrapper[2569]: I0706 13:54:13.952142 2569 memory_manager.go:368] "RemoveStaleState removing state" podUID="0985dbd8-3efa-4c39-9f96-8a906a0fe55c" containerName="alertmanager" Jul 06 13:54:13.952205 ip-10-0-136-25 kubenswrapper[2569]: I0706 13:54:13.952151 2569 memory_manager.go:368] "RemoveStaleState removing state" podUID="0985dbd8-3efa-4c39-9f96-8a906a0fe55c" containerName="kube-rbac-proxy" Jul 06 13:54:13.952205 ip-10-0-136-25 kubenswrapper[2569]: I0706 13:54:13.952156 2569 memory_manager.go:368] "RemoveStaleState removing state" podUID="0985dbd8-3efa-4c39-9f96-8a906a0fe55c" containerName="kube-rbac-proxy-web" Jul 06 13:54:13.952205 ip-10-0-136-25 kubenswrapper[2569]: I0706 13:54:13.952162 2569 memory_manager.go:368] "RemoveStaleState removing state" podUID="0985dbd8-3efa-4c39-9f96-8a906a0fe55c" containerName="config-reloader" Jul 06 13:54:13.952205 ip-10-0-136-25 kubenswrapper[2569]: I0706 13:54:13.952169 2569 memory_manager.go:368] "RemoveStaleState removing state" podUID="0985dbd8-3efa-4c39-9f96-8a906a0fe55c" containerName="kube-rbac-proxy-metric" Jul 06 13:54:13.952205 ip-10-0-136-25 kubenswrapper[2569]: I0706 13:54:13.952176 2569 memory_manager.go:368] "RemoveStaleState removing state" podUID="0985dbd8-3efa-4c39-9f96-8a906a0fe55c" containerName="prom-label-proxy" Jul 06 13:54:13.955693 ip-10-0-136-25 kubenswrapper[2569]: I0706 13:54:13.955678 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0" Jul 06 13:54:13.958074 ip-10-0-136-25 kubenswrapper[2569]: I0706 13:54:13.958057 2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-metric\"" Jul 06 13:54:13.958169 ip-10-0-136-25 kubenswrapper[2569]: I0706 13:54:13.958076 2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-cluster-tls-config\"" Jul 06 13:54:13.958416 ip-10-0-136-25 kubenswrapper[2569]: I0706 13:54:13.958401 2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls\"" Jul 06 13:54:13.958479 ip-10-0-136-25 kubenswrapper[2569]: I0706 13:54:13.958418 2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-generated\"" Jul 06 13:54:13.958541 ip-10-0-136-25 kubenswrapper[2569]: I0706 13:54:13.958520 2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy\"" Jul 06 13:54:13.958588 ip-10-0-136-25 kubenswrapper[2569]: I0706 13:54:13.958560 2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-dockercfg-lfcrt\"" Jul 06 13:54:13.958638 ip-10-0-136-25 kubenswrapper[2569]: I0706 13:54:13.958608 2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-web\"" Jul 06 13:54:13.958690 ip-10-0-136-25 kubenswrapper[2569]: I0706 13:54:13.958636 2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-web-config\"" Jul 06 13:54:13.958690 ip-10-0-136-25 kubenswrapper[2569]: I0706 13:54:13.958647 2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls-assets-0\"" Jul 06 13:54:13.964083 ip-10-0-136-25 kubenswrapper[2569]: I0706 13:54:13.964063 2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"alertmanager-trusted-ca-bundle\"" Jul 06 13:54:13.968916 ip-10-0-136-25 kubenswrapper[2569]: I0706 13:54:13.968895 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Jul 06 13:54:14.092953 ip-10-0-136-25 kubenswrapper[2569]: I0706 13:54:14.092918 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/a9b01e76-1c06-484b-bf3a-6db72c95c536-config-out\") pod \"alertmanager-main-0\" (UID: \"a9b01e76-1c06-484b-bf3a-6db72c95c536\") " pod="openshift-monitoring/alertmanager-main-0" Jul 06 13:54:14.092953 ip-10-0-136-25 kubenswrapper[2569]: I0706 13:54:14.092950 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/a9b01e76-1c06-484b-bf3a-6db72c95c536-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"a9b01e76-1c06-484b-bf3a-6db72c95c536\") " pod="openshift-monitoring/alertmanager-main-0" Jul 06 13:54:14.093165 ip-10-0-136-25 kubenswrapper[2569]: I0706 13:54:14.092987 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/a9b01e76-1c06-484b-bf3a-6db72c95c536-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"a9b01e76-1c06-484b-bf3a-6db72c95c536\") " pod="openshift-monitoring/alertmanager-main-0" Jul 06 13:54:14.093165 ip-10-0-136-25 kubenswrapper[2569]: I0706 13:54:14.093008 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/a9b01e76-1c06-484b-bf3a-6db72c95c536-tls-assets\") pod \"alertmanager-main-0\" (UID: \"a9b01e76-1c06-484b-bf3a-6db72c95c536\") " pod="openshift-monitoring/alertmanager-main-0" Jul 06 13:54:14.093165 ip-10-0-136-25 kubenswrapper[2569]: I0706 13:54:14.093036 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/a9b01e76-1c06-484b-bf3a-6db72c95c536-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"a9b01e76-1c06-484b-bf3a-6db72c95c536\") " pod="openshift-monitoring/alertmanager-main-0" Jul 06 13:54:14.093165 ip-10-0-136-25 kubenswrapper[2569]: I0706 13:54:14.093055 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/a9b01e76-1c06-484b-bf3a-6db72c95c536-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"a9b01e76-1c06-484b-bf3a-6db72c95c536\") " pod="openshift-monitoring/alertmanager-main-0" Jul 06 13:54:14.093165 ip-10-0-136-25 kubenswrapper[2569]: I0706 13:54:14.093096 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/a9b01e76-1c06-484b-bf3a-6db72c95c536-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"a9b01e76-1c06-484b-bf3a-6db72c95c536\") " pod="openshift-monitoring/alertmanager-main-0" Jul 06 13:54:14.093165 ip-10-0-136-25 kubenswrapper[2569]: I0706 13:54:14.093146 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/a9b01e76-1c06-484b-bf3a-6db72c95c536-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"a9b01e76-1c06-484b-bf3a-6db72c95c536\") " pod="openshift-monitoring/alertmanager-main-0" Jul 06 13:54:14.093367 ip-10-0-136-25 kubenswrapper[2569]: I0706 13:54:14.093171 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/a9b01e76-1c06-484b-bf3a-6db72c95c536-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"a9b01e76-1c06-484b-bf3a-6db72c95c536\") " pod="openshift-monitoring/alertmanager-main-0" Jul 06 13:54:14.093367 ip-10-0-136-25 kubenswrapper[2569]: I0706 13:54:14.093258 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/a9b01e76-1c06-484b-bf3a-6db72c95c536-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"a9b01e76-1c06-484b-bf3a-6db72c95c536\") " pod="openshift-monitoring/alertmanager-main-0" Jul 06 13:54:14.093367 ip-10-0-136-25 kubenswrapper[2569]: I0706 13:54:14.093291 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-shst4\" (UniqueName: \"kubernetes.io/projected/a9b01e76-1c06-484b-bf3a-6db72c95c536-kube-api-access-shst4\") pod \"alertmanager-main-0\" (UID: \"a9b01e76-1c06-484b-bf3a-6db72c95c536\") " pod="openshift-monitoring/alertmanager-main-0" Jul 06 13:54:14.093367 ip-10-0-136-25 kubenswrapper[2569]: I0706 13:54:14.093324 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/a9b01e76-1c06-484b-bf3a-6db72c95c536-web-config\") pod \"alertmanager-main-0\" (UID: \"a9b01e76-1c06-484b-bf3a-6db72c95c536\") " pod="openshift-monitoring/alertmanager-main-0" Jul 06 13:54:14.093367 ip-10-0-136-25 kubenswrapper[2569]: I0706 13:54:14.093352 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/a9b01e76-1c06-484b-bf3a-6db72c95c536-config-volume\") pod \"alertmanager-main-0\" (UID: \"a9b01e76-1c06-484b-bf3a-6db72c95c536\") " pod="openshift-monitoring/alertmanager-main-0" Jul 06 13:54:14.194449 ip-10-0-136-25 kubenswrapper[2569]: I0706 13:54:14.194363 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/a9b01e76-1c06-484b-bf3a-6db72c95c536-web-config\") pod \"alertmanager-main-0\" (UID: \"a9b01e76-1c06-484b-bf3a-6db72c95c536\") " pod="openshift-monitoring/alertmanager-main-0" Jul 06 13:54:14.194449 ip-10-0-136-25 kubenswrapper[2569]: I0706 13:54:14.194403 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/a9b01e76-1c06-484b-bf3a-6db72c95c536-config-volume\") pod \"alertmanager-main-0\" (UID: \"a9b01e76-1c06-484b-bf3a-6db72c95c536\") " pod="openshift-monitoring/alertmanager-main-0" Jul 06 13:54:14.194449 ip-10-0-136-25 kubenswrapper[2569]: I0706 13:54:14.194434 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/a9b01e76-1c06-484b-bf3a-6db72c95c536-config-out\") pod \"alertmanager-main-0\" (UID: \"a9b01e76-1c06-484b-bf3a-6db72c95c536\") " pod="openshift-monitoring/alertmanager-main-0" Jul 06 13:54:14.194449 ip-10-0-136-25 kubenswrapper[2569]: I0706 13:54:14.194450 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/a9b01e76-1c06-484b-bf3a-6db72c95c536-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"a9b01e76-1c06-484b-bf3a-6db72c95c536\") " pod="openshift-monitoring/alertmanager-main-0" Jul 06 13:54:14.194758 ip-10-0-136-25 kubenswrapper[2569]: I0706 13:54:14.194491 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/a9b01e76-1c06-484b-bf3a-6db72c95c536-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"a9b01e76-1c06-484b-bf3a-6db72c95c536\") " pod="openshift-monitoring/alertmanager-main-0" Jul 06 13:54:14.194758 ip-10-0-136-25 kubenswrapper[2569]: I0706 13:54:14.194508 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/a9b01e76-1c06-484b-bf3a-6db72c95c536-tls-assets\") pod \"alertmanager-main-0\" (UID: \"a9b01e76-1c06-484b-bf3a-6db72c95c536\") " pod="openshift-monitoring/alertmanager-main-0" Jul 06 13:54:14.194758 ip-10-0-136-25 kubenswrapper[2569]: I0706 13:54:14.194530 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/a9b01e76-1c06-484b-bf3a-6db72c95c536-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"a9b01e76-1c06-484b-bf3a-6db72c95c536\") " pod="openshift-monitoring/alertmanager-main-0" Jul 06 13:54:14.194758 ip-10-0-136-25 kubenswrapper[2569]: I0706 13:54:14.194562 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/a9b01e76-1c06-484b-bf3a-6db72c95c536-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"a9b01e76-1c06-484b-bf3a-6db72c95c536\") " pod="openshift-monitoring/alertmanager-main-0" Jul 06 13:54:14.194758 ip-10-0-136-25 kubenswrapper[2569]: I0706 13:54:14.194586 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/a9b01e76-1c06-484b-bf3a-6db72c95c536-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"a9b01e76-1c06-484b-bf3a-6db72c95c536\") " pod="openshift-monitoring/alertmanager-main-0" Jul 06 13:54:14.194758 ip-10-0-136-25 kubenswrapper[2569]: I0706 13:54:14.194611 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/a9b01e76-1c06-484b-bf3a-6db72c95c536-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"a9b01e76-1c06-484b-bf3a-6db72c95c536\") " pod="openshift-monitoring/alertmanager-main-0" Jul 06 13:54:14.194758 ip-10-0-136-25 kubenswrapper[2569]: I0706 13:54:14.194637 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/a9b01e76-1c06-484b-bf3a-6db72c95c536-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"a9b01e76-1c06-484b-bf3a-6db72c95c536\") " pod="openshift-monitoring/alertmanager-main-0" Jul 06 13:54:14.194758 ip-10-0-136-25 kubenswrapper[2569]: I0706 13:54:14.194670 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/a9b01e76-1c06-484b-bf3a-6db72c95c536-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"a9b01e76-1c06-484b-bf3a-6db72c95c536\") " pod="openshift-monitoring/alertmanager-main-0" Jul 06 13:54:14.194758 ip-10-0-136-25 kubenswrapper[2569]: I0706 13:54:14.194706 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-shst4\" (UniqueName: \"kubernetes.io/projected/a9b01e76-1c06-484b-bf3a-6db72c95c536-kube-api-access-shst4\") pod \"alertmanager-main-0\" (UID: \"a9b01e76-1c06-484b-bf3a-6db72c95c536\") " pod="openshift-monitoring/alertmanager-main-0" Jul 06 13:54:14.195379 ip-10-0-136-25 kubenswrapper[2569]: I0706 13:54:14.195349 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/a9b01e76-1c06-484b-bf3a-6db72c95c536-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"a9b01e76-1c06-484b-bf3a-6db72c95c536\") " pod="openshift-monitoring/alertmanager-main-0" Jul 06 13:54:14.196291 ip-10-0-136-25 kubenswrapper[2569]: I0706 13:54:14.196264 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/a9b01e76-1c06-484b-bf3a-6db72c95c536-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"a9b01e76-1c06-484b-bf3a-6db72c95c536\") " pod="openshift-monitoring/alertmanager-main-0" Jul 06 13:54:14.196556 ip-10-0-136-25 kubenswrapper[2569]: I0706 13:54:14.196531 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/a9b01e76-1c06-484b-bf3a-6db72c95c536-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"a9b01e76-1c06-484b-bf3a-6db72c95c536\") " pod="openshift-monitoring/alertmanager-main-0" Jul 06 13:54:14.197486 ip-10-0-136-25 kubenswrapper[2569]: I0706 13:54:14.197461 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/a9b01e76-1c06-484b-bf3a-6db72c95c536-config-out\") pod \"alertmanager-main-0\" (UID: \"a9b01e76-1c06-484b-bf3a-6db72c95c536\") " pod="openshift-monitoring/alertmanager-main-0" Jul 06 13:54:14.197574 ip-10-0-136-25 kubenswrapper[2569]: I0706 13:54:14.197542 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/a9b01e76-1c06-484b-bf3a-6db72c95c536-web-config\") pod \"alertmanager-main-0\" (UID: \"a9b01e76-1c06-484b-bf3a-6db72c95c536\") " pod="openshift-monitoring/alertmanager-main-0" Jul 06 13:54:14.197647 ip-10-0-136-25 kubenswrapper[2569]: I0706 13:54:14.197605 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/a9b01e76-1c06-484b-bf3a-6db72c95c536-config-volume\") pod \"alertmanager-main-0\" (UID: \"a9b01e76-1c06-484b-bf3a-6db72c95c536\") " pod="openshift-monitoring/alertmanager-main-0" Jul 06 13:54:14.197774 ip-10-0-136-25 kubenswrapper[2569]: I0706 13:54:14.197749 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/a9b01e76-1c06-484b-bf3a-6db72c95c536-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"a9b01e76-1c06-484b-bf3a-6db72c95c536\") " pod="openshift-monitoring/alertmanager-main-0" Jul 06 13:54:14.197844 ip-10-0-136-25 kubenswrapper[2569]: I0706 13:54:14.197755 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/a9b01e76-1c06-484b-bf3a-6db72c95c536-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"a9b01e76-1c06-484b-bf3a-6db72c95c536\") " pod="openshift-monitoring/alertmanager-main-0" Jul 06 13:54:14.197953 ip-10-0-136-25 kubenswrapper[2569]: I0706 13:54:14.197928 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/a9b01e76-1c06-484b-bf3a-6db72c95c536-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"a9b01e76-1c06-484b-bf3a-6db72c95c536\") " pod="openshift-monitoring/alertmanager-main-0" Jul 06 13:54:14.198049 ip-10-0-136-25 kubenswrapper[2569]: I0706 13:54:14.197974 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/a9b01e76-1c06-484b-bf3a-6db72c95c536-tls-assets\") pod \"alertmanager-main-0\" (UID: \"a9b01e76-1c06-484b-bf3a-6db72c95c536\") " pod="openshift-monitoring/alertmanager-main-0" Jul 06 13:54:14.198719 ip-10-0-136-25 kubenswrapper[2569]: I0706 13:54:14.198702 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/a9b01e76-1c06-484b-bf3a-6db72c95c536-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"a9b01e76-1c06-484b-bf3a-6db72c95c536\") " pod="openshift-monitoring/alertmanager-main-0" Jul 06 13:54:14.199421 ip-10-0-136-25 kubenswrapper[2569]: I0706 13:54:14.199404 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/a9b01e76-1c06-484b-bf3a-6db72c95c536-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"a9b01e76-1c06-484b-bf3a-6db72c95c536\") " pod="openshift-monitoring/alertmanager-main-0" Jul 06 13:54:14.202617 ip-10-0-136-25 kubenswrapper[2569]: I0706 13:54:14.202597 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-shst4\" (UniqueName: \"kubernetes.io/projected/a9b01e76-1c06-484b-bf3a-6db72c95c536-kube-api-access-shst4\") pod \"alertmanager-main-0\" (UID: \"a9b01e76-1c06-484b-bf3a-6db72c95c536\") " pod="openshift-monitoring/alertmanager-main-0" Jul 06 13:54:14.265308 ip-10-0-136-25 kubenswrapper[2569]: I0706 13:54:14.265277 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0" Jul 06 13:54:14.427727 ip-10-0-136-25 kubenswrapper[2569]: I0706 13:54:14.427699 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Jul 06 13:54:14.430335 ip-10-0-136-25 kubenswrapper[2569]: W0706 13:54:14.430309 2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda9b01e76_1c06_484b_bf3a_6db72c95c536.slice/crio-b0ae22003e88fe699e468a66559cf28b45d2d47475bc87386ff0b13792979283 WatchSource:0}: Error finding container b0ae22003e88fe699e468a66559cf28b45d2d47475bc87386ff0b13792979283: Status 404 returned error can't find the container with id b0ae22003e88fe699e468a66559cf28b45d2d47475bc87386ff0b13792979283 Jul 06 13:54:14.895166 ip-10-0-136-25 kubenswrapper[2569]: I0706 13:54:14.895133 2569 generic.go:356] "Generic (PLEG): container finished" podID="a9b01e76-1c06-484b-bf3a-6db72c95c536" containerID="4b40a15f0f0831130004dcce748189a2338c6bfd0b37e61fd6add0a6b6bc86cf" exitCode=0 Jul 06 13:54:14.895568 ip-10-0-136-25 kubenswrapper[2569]: I0706 13:54:14.895185 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"a9b01e76-1c06-484b-bf3a-6db72c95c536","Type":"ContainerDied","Data":"4b40a15f0f0831130004dcce748189a2338c6bfd0b37e61fd6add0a6b6bc86cf"} Jul 06 13:54:14.895568 ip-10-0-136-25 kubenswrapper[2569]: I0706 13:54:14.895254 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"a9b01e76-1c06-484b-bf3a-6db72c95c536","Type":"ContainerStarted","Data":"b0ae22003e88fe699e468a66559cf28b45d2d47475bc87386ff0b13792979283"} Jul 06 13:54:14.939532 ip-10-0-136-25 kubenswrapper[2569]: I0706 13:54:14.939499 2569 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="0985dbd8-3efa-4c39-9f96-8a906a0fe55c" path="/var/lib/kubelet/pods/0985dbd8-3efa-4c39-9f96-8a906a0fe55c/volumes" Jul 06 13:54:15.906773 ip-10-0-136-25 kubenswrapper[2569]: I0706 13:54:15.906734 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"a9b01e76-1c06-484b-bf3a-6db72c95c536","Type":"ContainerStarted","Data":"caa10f20f8110f6cffba2ad6e75068d803b69928c2baa004d1523a54ca199466"} Jul 06 13:54:15.907268 ip-10-0-136-25 kubenswrapper[2569]: I0706 13:54:15.906830 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"a9b01e76-1c06-484b-bf3a-6db72c95c536","Type":"ContainerStarted","Data":"051aa36e7367fbd5d52c10b9cbfd405572130ce179549cf7fa55f6c2805dd930"} Jul 06 13:54:15.907268 ip-10-0-136-25 kubenswrapper[2569]: I0706 13:54:15.906856 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"a9b01e76-1c06-484b-bf3a-6db72c95c536","Type":"ContainerStarted","Data":"595f7b7bc1583811c4e59ee6085a4c013dccacdc7ce00bc130413fc646971cc8"} Jul 06 13:54:15.907268 ip-10-0-136-25 kubenswrapper[2569]: I0706 13:54:15.906879 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"a9b01e76-1c06-484b-bf3a-6db72c95c536","Type":"ContainerStarted","Data":"1c364131427692278dc34a4f6f483c22d530d2b01da3a1760b5edf0cc0658292"} Jul 06 13:54:15.907268 ip-10-0-136-25 kubenswrapper[2569]: I0706 13:54:15.906902 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"a9b01e76-1c06-484b-bf3a-6db72c95c536","Type":"ContainerStarted","Data":"26309c90b925c12321d696dd4d798788496c4bf31767ecf00dec3f01bd3d8944"} Jul 06 13:54:15.907268 ip-10-0-136-25 kubenswrapper[2569]: I0706 13:54:15.906923 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"a9b01e76-1c06-484b-bf3a-6db72c95c536","Type":"ContainerStarted","Data":"b0b42c0aa80c7d875ec83d191d402efc51b2c9383b8aa373a53d04844e09a056"} Jul 06 13:54:15.933282 ip-10-0-136-25 kubenswrapper[2569]: I0706 13:54:15.933196 2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/alertmanager-main-0" podStartSLOduration=2.9170568919999997 podStartE2EDuration="2.93317913s" totalImagesPullingTime="0s" totalInitContainerRuntime="16.122238ms" isStatefulPod=false podCreationTimestamp="2026-07-06 13:54:13 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 13:54:15.932001313 +0000 UTC m=+231.522563120" watchObservedRunningTime="2026-07-06 13:54:15.93317913 +0000 UTC m=+231.523740953" Jul 06 13:54:16.633345 ip-10-0-136-25 kubenswrapper[2569]: I0706 13:54:16.633303 2569 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Jul 06 13:54:16.633699 ip-10-0-136-25 kubenswrapper[2569]: I0706 13:54:16.633644 2569 kuberuntime_container.go:872] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="b958bf61-a426-4f88-8a79-318280126542" containerName="prometheus" containerID="cri-o://38b5c18dc4173d78f48dd6c0b92c26ef4d1994462dbbbee24d92d5c5bf4b8b22" gracePeriod=600 Jul 06 13:54:16.633825 ip-10-0-136-25 kubenswrapper[2569]: I0706 13:54:16.633695 2569 kuberuntime_container.go:872] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="b958bf61-a426-4f88-8a79-318280126542" containerName="kube-rbac-proxy-thanos" containerID="cri-o://4a911e6732e5fc7953e35b1935bce0e0785fa4e43e1559a3afa6d5af3a45edb3" gracePeriod=600 Jul 06 13:54:16.633825 ip-10-0-136-25 kubenswrapper[2569]: I0706 13:54:16.633717 2569 kuberuntime_container.go:872] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="b958bf61-a426-4f88-8a79-318280126542" containerName="thanos-sidecar" containerID="cri-o://9a6205213a3ffd386945d449a6c5658c4a864acbba41e8c357584dc086c95d4d" gracePeriod=600 Jul 06 13:54:16.633825 ip-10-0-136-25 kubenswrapper[2569]: I0706 13:54:16.633734 2569 kuberuntime_container.go:872] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="b958bf61-a426-4f88-8a79-318280126542" containerName="config-reloader" containerID="cri-o://68acdea924854dcf3c90ecd92b451cb0b97fe47934007aeb7b804819d89a6e4b" gracePeriod=600 Jul 06 13:54:16.633825 ip-10-0-136-25 kubenswrapper[2569]: I0706 13:54:16.633648 2569 kuberuntime_container.go:872] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="b958bf61-a426-4f88-8a79-318280126542" containerName="kube-rbac-proxy" containerID="cri-o://cb9c1b6faf60392243d48d6313acb72e7dd5a407840405896b4bc85a5518fbd1" gracePeriod=600 Jul 06 13:54:16.633825 ip-10-0-136-25 kubenswrapper[2569]: I0706 13:54:16.633704 2569 kuberuntime_container.go:872] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="b958bf61-a426-4f88-8a79-318280126542" containerName="kube-rbac-proxy-web" containerID="cri-o://a95b56e8d5a128e827f4fdb56f61e5c0a13e51f3e63e5a4803f17c118e8f855a" gracePeriod=600 Jul 06 13:54:16.869396 ip-10-0-136-25 kubenswrapper[2569]: I0706 13:54:16.869360 2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0" Jul 06 13:54:16.914647 ip-10-0-136-25 kubenswrapper[2569]: I0706 13:54:16.914572 2569 generic.go:356] "Generic (PLEG): container finished" podID="b958bf61-a426-4f88-8a79-318280126542" containerID="4a911e6732e5fc7953e35b1935bce0e0785fa4e43e1559a3afa6d5af3a45edb3" exitCode=0 Jul 06 13:54:16.914647 ip-10-0-136-25 kubenswrapper[2569]: I0706 13:54:16.914601 2569 generic.go:356] "Generic (PLEG): container finished" podID="b958bf61-a426-4f88-8a79-318280126542" containerID="cb9c1b6faf60392243d48d6313acb72e7dd5a407840405896b4bc85a5518fbd1" exitCode=0 Jul 06 13:54:16.914647 ip-10-0-136-25 kubenswrapper[2569]: I0706 13:54:16.914609 2569 generic.go:356] "Generic (PLEG): container finished" podID="b958bf61-a426-4f88-8a79-318280126542" containerID="a95b56e8d5a128e827f4fdb56f61e5c0a13e51f3e63e5a4803f17c118e8f855a" exitCode=0 Jul 06 13:54:16.914647 ip-10-0-136-25 kubenswrapper[2569]: I0706 13:54:16.914616 2569 generic.go:356] "Generic (PLEG): container finished" podID="b958bf61-a426-4f88-8a79-318280126542" containerID="9a6205213a3ffd386945d449a6c5658c4a864acbba41e8c357584dc086c95d4d" exitCode=0 Jul 06 13:54:16.914647 ip-10-0-136-25 kubenswrapper[2569]: I0706 13:54:16.914625 2569 generic.go:356] "Generic (PLEG): container finished" podID="b958bf61-a426-4f88-8a79-318280126542" containerID="68acdea924854dcf3c90ecd92b451cb0b97fe47934007aeb7b804819d89a6e4b" exitCode=0 Jul 06 13:54:16.914647 ip-10-0-136-25 kubenswrapper[2569]: I0706 13:54:16.914633 2569 generic.go:356] "Generic (PLEG): container finished" podID="b958bf61-a426-4f88-8a79-318280126542" containerID="38b5c18dc4173d78f48dd6c0b92c26ef4d1994462dbbbee24d92d5c5bf4b8b22" exitCode=0 Jul 06 13:54:16.915265 ip-10-0-136-25 kubenswrapper[2569]: I0706 13:54:16.914650 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"b958bf61-a426-4f88-8a79-318280126542","Type":"ContainerDied","Data":"4a911e6732e5fc7953e35b1935bce0e0785fa4e43e1559a3afa6d5af3a45edb3"} Jul 06 13:54:16.915265 ip-10-0-136-25 kubenswrapper[2569]: I0706 13:54:16.914670 2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0" Jul 06 13:54:16.915265 ip-10-0-136-25 kubenswrapper[2569]: I0706 13:54:16.914722 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"b958bf61-a426-4f88-8a79-318280126542","Type":"ContainerDied","Data":"cb9c1b6faf60392243d48d6313acb72e7dd5a407840405896b4bc85a5518fbd1"} Jul 06 13:54:16.915265 ip-10-0-136-25 kubenswrapper[2569]: I0706 13:54:16.914764 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"b958bf61-a426-4f88-8a79-318280126542","Type":"ContainerDied","Data":"a95b56e8d5a128e827f4fdb56f61e5c0a13e51f3e63e5a4803f17c118e8f855a"} Jul 06 13:54:16.915265 ip-10-0-136-25 kubenswrapper[2569]: I0706 13:54:16.914795 2569 scope.go:117] "RemoveContainer" containerID="4a911e6732e5fc7953e35b1935bce0e0785fa4e43e1559a3afa6d5af3a45edb3" Jul 06 13:54:16.915265 ip-10-0-136-25 kubenswrapper[2569]: I0706 13:54:16.914799 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"b958bf61-a426-4f88-8a79-318280126542","Type":"ContainerDied","Data":"9a6205213a3ffd386945d449a6c5658c4a864acbba41e8c357584dc086c95d4d"} Jul 06 13:54:16.915265 ip-10-0-136-25 kubenswrapper[2569]: I0706 13:54:16.914907 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"b958bf61-a426-4f88-8a79-318280126542","Type":"ContainerDied","Data":"68acdea924854dcf3c90ecd92b451cb0b97fe47934007aeb7b804819d89a6e4b"} Jul 06 13:54:16.915265 ip-10-0-136-25 kubenswrapper[2569]: I0706 13:54:16.914931 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"b958bf61-a426-4f88-8a79-318280126542","Type":"ContainerDied","Data":"38b5c18dc4173d78f48dd6c0b92c26ef4d1994462dbbbee24d92d5c5bf4b8b22"} Jul 06 13:54:16.915265 ip-10-0-136-25 kubenswrapper[2569]: I0706 13:54:16.914961 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"b958bf61-a426-4f88-8a79-318280126542","Type":"ContainerDied","Data":"11c7cc8493d5e89934787340d940c37aec05191e3a3d8d6a9b5d047e66a7ee91"} Jul 06 13:54:16.918955 ip-10-0-136-25 kubenswrapper[2569]: I0706 13:54:16.918932 2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/b958bf61-a426-4f88-8a79-318280126542-secret-kube-rbac-proxy\") pod \"b958bf61-a426-4f88-8a79-318280126542\" (UID: \"b958bf61-a426-4f88-8a79-318280126542\") " Jul 06 13:54:16.919092 ip-10-0-136-25 kubenswrapper[2569]: I0706 13:54:16.918965 2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/b958bf61-a426-4f88-8a79-318280126542-prometheus-k8s-rulefiles-2\") pod \"b958bf61-a426-4f88-8a79-318280126542\" (UID: \"b958bf61-a426-4f88-8a79-318280126542\") " Jul 06 13:54:16.919092 ip-10-0-136-25 kubenswrapper[2569]: I0706 13:54:16.918988 2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/b958bf61-a426-4f88-8a79-318280126542-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"b958bf61-a426-4f88-8a79-318280126542\" (UID: \"b958bf61-a426-4f88-8a79-318280126542\") " Jul 06 13:54:16.919092 ip-10-0-136-25 kubenswrapper[2569]: I0706 13:54:16.919013 2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/b958bf61-a426-4f88-8a79-318280126542-tls-assets\") pod \"b958bf61-a426-4f88-8a79-318280126542\" (UID: \"b958bf61-a426-4f88-8a79-318280126542\") " Jul 06 13:54:16.919092 ip-10-0-136-25 kubenswrapper[2569]: I0706 13:54:16.919054 2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b958bf61-a426-4f88-8a79-318280126542-configmap-serving-certs-ca-bundle\") pod \"b958bf61-a426-4f88-8a79-318280126542\" (UID: \"b958bf61-a426-4f88-8a79-318280126542\") " Jul 06 13:54:16.919092 ip-10-0-136-25 kubenswrapper[2569]: I0706 13:54:16.919082 2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/b958bf61-a426-4f88-8a79-318280126542-prometheus-k8s-db\") pod \"b958bf61-a426-4f88-8a79-318280126542\" (UID: \"b958bf61-a426-4f88-8a79-318280126542\") " Jul 06 13:54:16.919468 ip-10-0-136-25 kubenswrapper[2569]: I0706 13:54:16.919110 2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/b958bf61-a426-4f88-8a79-318280126542-config\") pod \"b958bf61-a426-4f88-8a79-318280126542\" (UID: \"b958bf61-a426-4f88-8a79-318280126542\") " Jul 06 13:54:16.920778 ip-10-0-136-25 kubenswrapper[2569]: I0706 13:54:16.920586 2569 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b958bf61-a426-4f88-8a79-318280126542-configmap-serving-certs-ca-bundle" (OuterVolumeSpecName: "configmap-serving-certs-ca-bundle") pod "b958bf61-a426-4f88-8a79-318280126542" (UID: "b958bf61-a426-4f88-8a79-318280126542"). InnerVolumeSpecName "configmap-serving-certs-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 06 13:54:16.920778 ip-10-0-136-25 kubenswrapper[2569]: I0706 13:54:16.920620 2569 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b958bf61-a426-4f88-8a79-318280126542-prometheus-k8s-rulefiles-2" (OuterVolumeSpecName: "prometheus-k8s-rulefiles-2") pod "b958bf61-a426-4f88-8a79-318280126542" (UID: "b958bf61-a426-4f88-8a79-318280126542"). InnerVolumeSpecName "prometheus-k8s-rulefiles-2". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 06 13:54:16.921467 ip-10-0-136-25 kubenswrapper[2569]: I0706 13:54:16.921441 2569 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b958bf61-a426-4f88-8a79-318280126542-prometheus-k8s-db" (OuterVolumeSpecName: "prometheus-k8s-db") pod "b958bf61-a426-4f88-8a79-318280126542" (UID: "b958bf61-a426-4f88-8a79-318280126542"). InnerVolumeSpecName "prometheus-k8s-db". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 06 13:54:16.921976 ip-10-0-136-25 kubenswrapper[2569]: I0706 13:54:16.921939 2569 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b958bf61-a426-4f88-8a79-318280126542-tls-assets" (OuterVolumeSpecName: "tls-assets") pod "b958bf61-a426-4f88-8a79-318280126542" (UID: "b958bf61-a426-4f88-8a79-318280126542"). InnerVolumeSpecName "tls-assets". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 06 13:54:16.922087 ip-10-0-136-25 kubenswrapper[2569]: I0706 13:54:16.922067 2569 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b958bf61-a426-4f88-8a79-318280126542-secret-prometheus-k8s-thanos-sidecar-tls" (OuterVolumeSpecName: "secret-prometheus-k8s-thanos-sidecar-tls") pod "b958bf61-a426-4f88-8a79-318280126542" (UID: "b958bf61-a426-4f88-8a79-318280126542"). InnerVolumeSpecName "secret-prometheus-k8s-thanos-sidecar-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 06 13:54:16.922566 ip-10-0-136-25 kubenswrapper[2569]: I0706 13:54:16.922542 2569 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b958bf61-a426-4f88-8a79-318280126542-config" (OuterVolumeSpecName: "config") pod "b958bf61-a426-4f88-8a79-318280126542" (UID: "b958bf61-a426-4f88-8a79-318280126542"). InnerVolumeSpecName "config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 06 13:54:16.922654 ip-10-0-136-25 kubenswrapper[2569]: I0706 13:54:16.922581 2569 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b958bf61-a426-4f88-8a79-318280126542-secret-kube-rbac-proxy" (OuterVolumeSpecName: "secret-kube-rbac-proxy") pod "b958bf61-a426-4f88-8a79-318280126542" (UID: "b958bf61-a426-4f88-8a79-318280126542"). InnerVolumeSpecName "secret-kube-rbac-proxy". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 06 13:54:16.924620 ip-10-0-136-25 kubenswrapper[2569]: I0706 13:54:16.924601 2569 scope.go:117] "RemoveContainer" containerID="cb9c1b6faf60392243d48d6313acb72e7dd5a407840405896b4bc85a5518fbd1" Jul 06 13:54:16.932954 ip-10-0-136-25 kubenswrapper[2569]: I0706 13:54:16.932937 2569 scope.go:117] "RemoveContainer" containerID="a95b56e8d5a128e827f4fdb56f61e5c0a13e51f3e63e5a4803f17c118e8f855a" Jul 06 13:54:16.941840 ip-10-0-136-25 kubenswrapper[2569]: I0706 13:54:16.941823 2569 scope.go:117] "RemoveContainer" containerID="9a6205213a3ffd386945d449a6c5658c4a864acbba41e8c357584dc086c95d4d" Jul 06 13:54:16.948829 ip-10-0-136-25 kubenswrapper[2569]: I0706 13:54:16.948791 2569 scope.go:117] "RemoveContainer" containerID="68acdea924854dcf3c90ecd92b451cb0b97fe47934007aeb7b804819d89a6e4b" Jul 06 13:54:16.955529 ip-10-0-136-25 kubenswrapper[2569]: I0706 13:54:16.955509 2569 scope.go:117] "RemoveContainer" containerID="38b5c18dc4173d78f48dd6c0b92c26ef4d1994462dbbbee24d92d5c5bf4b8b22" Jul 06 13:54:16.962611 ip-10-0-136-25 kubenswrapper[2569]: I0706 13:54:16.962594 2569 scope.go:117] "RemoveContainer" containerID="e1ab4427d197ff423c8e4185f16aa7d19d0c84ae571ef8c29a30b53b29671542" Jul 06 13:54:16.969142 ip-10-0-136-25 kubenswrapper[2569]: I0706 13:54:16.969126 2569 scope.go:117] "RemoveContainer" containerID="4a911e6732e5fc7953e35b1935bce0e0785fa4e43e1559a3afa6d5af3a45edb3" Jul 06 13:54:16.969412 ip-10-0-136-25 kubenswrapper[2569]: E0706 13:54:16.969389 2569 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4a911e6732e5fc7953e35b1935bce0e0785fa4e43e1559a3afa6d5af3a45edb3\": container with ID starting with 4a911e6732e5fc7953e35b1935bce0e0785fa4e43e1559a3afa6d5af3a45edb3 not found: ID does not exist" containerID="4a911e6732e5fc7953e35b1935bce0e0785fa4e43e1559a3afa6d5af3a45edb3" Jul 06 13:54:16.969494 ip-10-0-136-25 kubenswrapper[2569]: I0706 13:54:16.969418 2569 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4a911e6732e5fc7953e35b1935bce0e0785fa4e43e1559a3afa6d5af3a45edb3"} err="failed to get container status \"4a911e6732e5fc7953e35b1935bce0e0785fa4e43e1559a3afa6d5af3a45edb3\": rpc error: code = NotFound desc = could not find container \"4a911e6732e5fc7953e35b1935bce0e0785fa4e43e1559a3afa6d5af3a45edb3\": container with ID starting with 4a911e6732e5fc7953e35b1935bce0e0785fa4e43e1559a3afa6d5af3a45edb3 not found: ID does not exist" Jul 06 13:54:16.969494 ip-10-0-136-25 kubenswrapper[2569]: I0706 13:54:16.969444 2569 scope.go:117] "RemoveContainer" containerID="cb9c1b6faf60392243d48d6313acb72e7dd5a407840405896b4bc85a5518fbd1" Jul 06 13:54:16.969678 ip-10-0-136-25 kubenswrapper[2569]: E0706 13:54:16.969661 2569 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"cb9c1b6faf60392243d48d6313acb72e7dd5a407840405896b4bc85a5518fbd1\": container with ID starting with cb9c1b6faf60392243d48d6313acb72e7dd5a407840405896b4bc85a5518fbd1 not found: ID does not exist" containerID="cb9c1b6faf60392243d48d6313acb72e7dd5a407840405896b4bc85a5518fbd1" Jul 06 13:54:16.969788 ip-10-0-136-25 kubenswrapper[2569]: I0706 13:54:16.969688 2569 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cb9c1b6faf60392243d48d6313acb72e7dd5a407840405896b4bc85a5518fbd1"} err="failed to get container status \"cb9c1b6faf60392243d48d6313acb72e7dd5a407840405896b4bc85a5518fbd1\": rpc error: code = NotFound desc = could not find container \"cb9c1b6faf60392243d48d6313acb72e7dd5a407840405896b4bc85a5518fbd1\": container with ID starting with cb9c1b6faf60392243d48d6313acb72e7dd5a407840405896b4bc85a5518fbd1 not found: ID does not exist" Jul 06 13:54:16.969788 ip-10-0-136-25 kubenswrapper[2569]: I0706 13:54:16.969709 2569 scope.go:117] "RemoveContainer" containerID="a95b56e8d5a128e827f4fdb56f61e5c0a13e51f3e63e5a4803f17c118e8f855a" Jul 06 13:54:16.969939 ip-10-0-136-25 kubenswrapper[2569]: E0706 13:54:16.969924 2569 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a95b56e8d5a128e827f4fdb56f61e5c0a13e51f3e63e5a4803f17c118e8f855a\": container with ID starting with a95b56e8d5a128e827f4fdb56f61e5c0a13e51f3e63e5a4803f17c118e8f855a not found: ID does not exist" containerID="a95b56e8d5a128e827f4fdb56f61e5c0a13e51f3e63e5a4803f17c118e8f855a" Jul 06 13:54:16.969973 ip-10-0-136-25 kubenswrapper[2569]: I0706 13:54:16.969944 2569 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a95b56e8d5a128e827f4fdb56f61e5c0a13e51f3e63e5a4803f17c118e8f855a"} err="failed to get container status \"a95b56e8d5a128e827f4fdb56f61e5c0a13e51f3e63e5a4803f17c118e8f855a\": rpc error: code = NotFound desc = could not find container \"a95b56e8d5a128e827f4fdb56f61e5c0a13e51f3e63e5a4803f17c118e8f855a\": container with ID starting with a95b56e8d5a128e827f4fdb56f61e5c0a13e51f3e63e5a4803f17c118e8f855a not found: ID does not exist" Jul 06 13:54:16.969973 ip-10-0-136-25 kubenswrapper[2569]: I0706 13:54:16.969957 2569 scope.go:117] "RemoveContainer" containerID="9a6205213a3ffd386945d449a6c5658c4a864acbba41e8c357584dc086c95d4d" Jul 06 13:54:16.970166 ip-10-0-136-25 kubenswrapper[2569]: E0706 13:54:16.970148 2569 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9a6205213a3ffd386945d449a6c5658c4a864acbba41e8c357584dc086c95d4d\": container with ID starting with 9a6205213a3ffd386945d449a6c5658c4a864acbba41e8c357584dc086c95d4d not found: ID does not exist" containerID="9a6205213a3ffd386945d449a6c5658c4a864acbba41e8c357584dc086c95d4d" Jul 06 13:54:16.970200 ip-10-0-136-25 kubenswrapper[2569]: I0706 13:54:16.970171 2569 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9a6205213a3ffd386945d449a6c5658c4a864acbba41e8c357584dc086c95d4d"} err="failed to get container status \"9a6205213a3ffd386945d449a6c5658c4a864acbba41e8c357584dc086c95d4d\": rpc error: code = NotFound desc = could not find container \"9a6205213a3ffd386945d449a6c5658c4a864acbba41e8c357584dc086c95d4d\": container with ID starting with 9a6205213a3ffd386945d449a6c5658c4a864acbba41e8c357584dc086c95d4d not found: ID does not exist" Jul 06 13:54:16.970200 ip-10-0-136-25 kubenswrapper[2569]: I0706 13:54:16.970182 2569 scope.go:117] "RemoveContainer" containerID="68acdea924854dcf3c90ecd92b451cb0b97fe47934007aeb7b804819d89a6e4b" Jul 06 13:54:16.970453 ip-10-0-136-25 kubenswrapper[2569]: E0706 13:54:16.970435 2569 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"68acdea924854dcf3c90ecd92b451cb0b97fe47934007aeb7b804819d89a6e4b\": container with ID starting with 68acdea924854dcf3c90ecd92b451cb0b97fe47934007aeb7b804819d89a6e4b not found: ID does not exist" containerID="68acdea924854dcf3c90ecd92b451cb0b97fe47934007aeb7b804819d89a6e4b" Jul 06 13:54:16.970496 ip-10-0-136-25 kubenswrapper[2569]: I0706 13:54:16.970460 2569 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"68acdea924854dcf3c90ecd92b451cb0b97fe47934007aeb7b804819d89a6e4b"} err="failed to get container status \"68acdea924854dcf3c90ecd92b451cb0b97fe47934007aeb7b804819d89a6e4b\": rpc error: code = NotFound desc = could not find container \"68acdea924854dcf3c90ecd92b451cb0b97fe47934007aeb7b804819d89a6e4b\": container with ID starting with 68acdea924854dcf3c90ecd92b451cb0b97fe47934007aeb7b804819d89a6e4b not found: ID does not exist" Jul 06 13:54:16.970496 ip-10-0-136-25 kubenswrapper[2569]: I0706 13:54:16.970478 2569 scope.go:117] "RemoveContainer" containerID="38b5c18dc4173d78f48dd6c0b92c26ef4d1994462dbbbee24d92d5c5bf4b8b22" Jul 06 13:54:16.970680 ip-10-0-136-25 kubenswrapper[2569]: E0706 13:54:16.970665 2569 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"38b5c18dc4173d78f48dd6c0b92c26ef4d1994462dbbbee24d92d5c5bf4b8b22\": container with ID starting with 38b5c18dc4173d78f48dd6c0b92c26ef4d1994462dbbbee24d92d5c5bf4b8b22 not found: ID does not exist" containerID="38b5c18dc4173d78f48dd6c0b92c26ef4d1994462dbbbee24d92d5c5bf4b8b22" Jul 06 13:54:16.970715 ip-10-0-136-25 kubenswrapper[2569]: I0706 13:54:16.970682 2569 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"38b5c18dc4173d78f48dd6c0b92c26ef4d1994462dbbbee24d92d5c5bf4b8b22"} err="failed to get container status \"38b5c18dc4173d78f48dd6c0b92c26ef4d1994462dbbbee24d92d5c5bf4b8b22\": rpc error: code = NotFound desc = could not find container \"38b5c18dc4173d78f48dd6c0b92c26ef4d1994462dbbbee24d92d5c5bf4b8b22\": container with ID starting with 38b5c18dc4173d78f48dd6c0b92c26ef4d1994462dbbbee24d92d5c5bf4b8b22 not found: ID does not exist" Jul 06 13:54:16.970715 ip-10-0-136-25 kubenswrapper[2569]: I0706 13:54:16.970694 2569 scope.go:117] "RemoveContainer" containerID="e1ab4427d197ff423c8e4185f16aa7d19d0c84ae571ef8c29a30b53b29671542" Jul 06 13:54:16.970861 ip-10-0-136-25 kubenswrapper[2569]: E0706 13:54:16.970848 2569 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e1ab4427d197ff423c8e4185f16aa7d19d0c84ae571ef8c29a30b53b29671542\": container with ID starting with e1ab4427d197ff423c8e4185f16aa7d19d0c84ae571ef8c29a30b53b29671542 not found: ID does not exist" containerID="e1ab4427d197ff423c8e4185f16aa7d19d0c84ae571ef8c29a30b53b29671542" Jul 06 13:54:16.970908 ip-10-0-136-25 kubenswrapper[2569]: I0706 13:54:16.970863 2569 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e1ab4427d197ff423c8e4185f16aa7d19d0c84ae571ef8c29a30b53b29671542"} err="failed to get container status \"e1ab4427d197ff423c8e4185f16aa7d19d0c84ae571ef8c29a30b53b29671542\": rpc error: code = NotFound desc = could not find container \"e1ab4427d197ff423c8e4185f16aa7d19d0c84ae571ef8c29a30b53b29671542\": container with ID starting with e1ab4427d197ff423c8e4185f16aa7d19d0c84ae571ef8c29a30b53b29671542 not found: ID does not exist" Jul 06 13:54:16.970908 ip-10-0-136-25 kubenswrapper[2569]: I0706 13:54:16.970873 2569 scope.go:117] "RemoveContainer" containerID="4a911e6732e5fc7953e35b1935bce0e0785fa4e43e1559a3afa6d5af3a45edb3" Jul 06 13:54:16.971069 ip-10-0-136-25 kubenswrapper[2569]: I0706 13:54:16.971049 2569 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4a911e6732e5fc7953e35b1935bce0e0785fa4e43e1559a3afa6d5af3a45edb3"} err="failed to get container status \"4a911e6732e5fc7953e35b1935bce0e0785fa4e43e1559a3afa6d5af3a45edb3\": rpc error: code = NotFound desc = could not find container \"4a911e6732e5fc7953e35b1935bce0e0785fa4e43e1559a3afa6d5af3a45edb3\": container with ID starting with 4a911e6732e5fc7953e35b1935bce0e0785fa4e43e1559a3afa6d5af3a45edb3 not found: ID does not exist" Jul 06 13:54:16.971107 ip-10-0-136-25 kubenswrapper[2569]: I0706 13:54:16.971070 2569 scope.go:117] "RemoveContainer" containerID="cb9c1b6faf60392243d48d6313acb72e7dd5a407840405896b4bc85a5518fbd1" Jul 06 13:54:16.971291 ip-10-0-136-25 kubenswrapper[2569]: I0706 13:54:16.971273 2569 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cb9c1b6faf60392243d48d6313acb72e7dd5a407840405896b4bc85a5518fbd1"} err="failed to get container status \"cb9c1b6faf60392243d48d6313acb72e7dd5a407840405896b4bc85a5518fbd1\": rpc error: code = NotFound desc = could not find container \"cb9c1b6faf60392243d48d6313acb72e7dd5a407840405896b4bc85a5518fbd1\": container with ID starting with cb9c1b6faf60392243d48d6313acb72e7dd5a407840405896b4bc85a5518fbd1 not found: ID does not exist" Jul 06 13:54:16.971366 ip-10-0-136-25 kubenswrapper[2569]: I0706 13:54:16.971293 2569 scope.go:117] "RemoveContainer" containerID="a95b56e8d5a128e827f4fdb56f61e5c0a13e51f3e63e5a4803f17c118e8f855a" Jul 06 13:54:16.971531 ip-10-0-136-25 kubenswrapper[2569]: I0706 13:54:16.971512 2569 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a95b56e8d5a128e827f4fdb56f61e5c0a13e51f3e63e5a4803f17c118e8f855a"} err="failed to get container status \"a95b56e8d5a128e827f4fdb56f61e5c0a13e51f3e63e5a4803f17c118e8f855a\": rpc error: code = NotFound desc = could not find container \"a95b56e8d5a128e827f4fdb56f61e5c0a13e51f3e63e5a4803f17c118e8f855a\": container with ID starting with a95b56e8d5a128e827f4fdb56f61e5c0a13e51f3e63e5a4803f17c118e8f855a not found: ID does not exist" Jul 06 13:54:16.971566 ip-10-0-136-25 kubenswrapper[2569]: I0706 13:54:16.971531 2569 scope.go:117] "RemoveContainer" containerID="9a6205213a3ffd386945d449a6c5658c4a864acbba41e8c357584dc086c95d4d" Jul 06 13:54:16.971736 ip-10-0-136-25 kubenswrapper[2569]: I0706 13:54:16.971721 2569 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9a6205213a3ffd386945d449a6c5658c4a864acbba41e8c357584dc086c95d4d"} err="failed to get container status \"9a6205213a3ffd386945d449a6c5658c4a864acbba41e8c357584dc086c95d4d\": rpc error: code = NotFound desc = could not find container \"9a6205213a3ffd386945d449a6c5658c4a864acbba41e8c357584dc086c95d4d\": container with ID starting with 9a6205213a3ffd386945d449a6c5658c4a864acbba41e8c357584dc086c95d4d not found: ID does not exist" Jul 06 13:54:16.971821 ip-10-0-136-25 kubenswrapper[2569]: I0706 13:54:16.971736 2569 scope.go:117] "RemoveContainer" containerID="68acdea924854dcf3c90ecd92b451cb0b97fe47934007aeb7b804819d89a6e4b" Jul 06 13:54:16.971910 ip-10-0-136-25 kubenswrapper[2569]: I0706 13:54:16.971894 2569 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"68acdea924854dcf3c90ecd92b451cb0b97fe47934007aeb7b804819d89a6e4b"} err="failed to get container status \"68acdea924854dcf3c90ecd92b451cb0b97fe47934007aeb7b804819d89a6e4b\": rpc error: code = NotFound desc = could not find container \"68acdea924854dcf3c90ecd92b451cb0b97fe47934007aeb7b804819d89a6e4b\": container with ID starting with 68acdea924854dcf3c90ecd92b451cb0b97fe47934007aeb7b804819d89a6e4b not found: ID does not exist" Jul 06 13:54:16.971945 ip-10-0-136-25 kubenswrapper[2569]: I0706 13:54:16.971910 2569 scope.go:117] "RemoveContainer" containerID="38b5c18dc4173d78f48dd6c0b92c26ef4d1994462dbbbee24d92d5c5bf4b8b22" Jul 06 13:54:16.972093 ip-10-0-136-25 kubenswrapper[2569]: I0706 13:54:16.972080 2569 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"38b5c18dc4173d78f48dd6c0b92c26ef4d1994462dbbbee24d92d5c5bf4b8b22"} err="failed to get container status \"38b5c18dc4173d78f48dd6c0b92c26ef4d1994462dbbbee24d92d5c5bf4b8b22\": rpc error: code = NotFound desc = could not find container \"38b5c18dc4173d78f48dd6c0b92c26ef4d1994462dbbbee24d92d5c5bf4b8b22\": container with ID starting with 38b5c18dc4173d78f48dd6c0b92c26ef4d1994462dbbbee24d92d5c5bf4b8b22 not found: ID does not exist" Jul 06 13:54:16.972135 ip-10-0-136-25 kubenswrapper[2569]: I0706 13:54:16.972094 2569 scope.go:117] "RemoveContainer" containerID="e1ab4427d197ff423c8e4185f16aa7d19d0c84ae571ef8c29a30b53b29671542" Jul 06 13:54:16.972333 ip-10-0-136-25 kubenswrapper[2569]: I0706 13:54:16.972314 2569 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e1ab4427d197ff423c8e4185f16aa7d19d0c84ae571ef8c29a30b53b29671542"} err="failed to get container status \"e1ab4427d197ff423c8e4185f16aa7d19d0c84ae571ef8c29a30b53b29671542\": rpc error: code = NotFound desc = could not find container \"e1ab4427d197ff423c8e4185f16aa7d19d0c84ae571ef8c29a30b53b29671542\": container with ID starting with e1ab4427d197ff423c8e4185f16aa7d19d0c84ae571ef8c29a30b53b29671542 not found: ID does not exist" Jul 06 13:54:16.972393 ip-10-0-136-25 kubenswrapper[2569]: I0706 13:54:16.972333 2569 scope.go:117] "RemoveContainer" containerID="4a911e6732e5fc7953e35b1935bce0e0785fa4e43e1559a3afa6d5af3a45edb3" Jul 06 13:54:16.972540 ip-10-0-136-25 kubenswrapper[2569]: I0706 13:54:16.972523 2569 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4a911e6732e5fc7953e35b1935bce0e0785fa4e43e1559a3afa6d5af3a45edb3"} err="failed to get container status \"4a911e6732e5fc7953e35b1935bce0e0785fa4e43e1559a3afa6d5af3a45edb3\": rpc error: code = NotFound desc = could not find container \"4a911e6732e5fc7953e35b1935bce0e0785fa4e43e1559a3afa6d5af3a45edb3\": container with ID starting with 4a911e6732e5fc7953e35b1935bce0e0785fa4e43e1559a3afa6d5af3a45edb3 not found: ID does not exist" Jul 06 13:54:16.972578 ip-10-0-136-25 kubenswrapper[2569]: I0706 13:54:16.972540 2569 scope.go:117] "RemoveContainer" containerID="cb9c1b6faf60392243d48d6313acb72e7dd5a407840405896b4bc85a5518fbd1" Jul 06 13:54:16.972774 ip-10-0-136-25 kubenswrapper[2569]: I0706 13:54:16.972759 2569 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cb9c1b6faf60392243d48d6313acb72e7dd5a407840405896b4bc85a5518fbd1"} err="failed to get container status \"cb9c1b6faf60392243d48d6313acb72e7dd5a407840405896b4bc85a5518fbd1\": rpc error: code = NotFound desc = could not find container \"cb9c1b6faf60392243d48d6313acb72e7dd5a407840405896b4bc85a5518fbd1\": container with ID starting with cb9c1b6faf60392243d48d6313acb72e7dd5a407840405896b4bc85a5518fbd1 not found: ID does not exist" Jul 06 13:54:16.972826 ip-10-0-136-25 kubenswrapper[2569]: I0706 13:54:16.972774 2569 scope.go:117] "RemoveContainer" containerID="a95b56e8d5a128e827f4fdb56f61e5c0a13e51f3e63e5a4803f17c118e8f855a" Jul 06 13:54:16.972984 ip-10-0-136-25 kubenswrapper[2569]: I0706 13:54:16.972968 2569 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a95b56e8d5a128e827f4fdb56f61e5c0a13e51f3e63e5a4803f17c118e8f855a"} err="failed to get container status \"a95b56e8d5a128e827f4fdb56f61e5c0a13e51f3e63e5a4803f17c118e8f855a\": rpc error: code = NotFound desc = could not find container \"a95b56e8d5a128e827f4fdb56f61e5c0a13e51f3e63e5a4803f17c118e8f855a\": container with ID starting with a95b56e8d5a128e827f4fdb56f61e5c0a13e51f3e63e5a4803f17c118e8f855a not found: ID does not exist" Jul 06 13:54:16.973028 ip-10-0-136-25 kubenswrapper[2569]: I0706 13:54:16.972985 2569 scope.go:117] "RemoveContainer" containerID="9a6205213a3ffd386945d449a6c5658c4a864acbba41e8c357584dc086c95d4d" Jul 06 13:54:16.973163 ip-10-0-136-25 kubenswrapper[2569]: I0706 13:54:16.973147 2569 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9a6205213a3ffd386945d449a6c5658c4a864acbba41e8c357584dc086c95d4d"} err="failed to get container status \"9a6205213a3ffd386945d449a6c5658c4a864acbba41e8c357584dc086c95d4d\": rpc error: code = NotFound desc = could not find container \"9a6205213a3ffd386945d449a6c5658c4a864acbba41e8c357584dc086c95d4d\": container with ID starting with 9a6205213a3ffd386945d449a6c5658c4a864acbba41e8c357584dc086c95d4d not found: ID does not exist" Jul 06 13:54:16.973200 ip-10-0-136-25 kubenswrapper[2569]: I0706 13:54:16.973164 2569 scope.go:117] "RemoveContainer" containerID="68acdea924854dcf3c90ecd92b451cb0b97fe47934007aeb7b804819d89a6e4b" Jul 06 13:54:16.973369 ip-10-0-136-25 kubenswrapper[2569]: I0706 13:54:16.973347 2569 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"68acdea924854dcf3c90ecd92b451cb0b97fe47934007aeb7b804819d89a6e4b"} err="failed to get container status \"68acdea924854dcf3c90ecd92b451cb0b97fe47934007aeb7b804819d89a6e4b\": rpc error: code = NotFound desc = could not find container \"68acdea924854dcf3c90ecd92b451cb0b97fe47934007aeb7b804819d89a6e4b\": container with ID starting with 68acdea924854dcf3c90ecd92b451cb0b97fe47934007aeb7b804819d89a6e4b not found: ID does not exist" Jul 06 13:54:16.973447 ip-10-0-136-25 kubenswrapper[2569]: I0706 13:54:16.973372 2569 scope.go:117] "RemoveContainer" containerID="38b5c18dc4173d78f48dd6c0b92c26ef4d1994462dbbbee24d92d5c5bf4b8b22" Jul 06 13:54:16.973613 ip-10-0-136-25 kubenswrapper[2569]: I0706 13:54:16.973596 2569 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"38b5c18dc4173d78f48dd6c0b92c26ef4d1994462dbbbee24d92d5c5bf4b8b22"} err="failed to get container status \"38b5c18dc4173d78f48dd6c0b92c26ef4d1994462dbbbee24d92d5c5bf4b8b22\": rpc error: code = NotFound desc = could not find container \"38b5c18dc4173d78f48dd6c0b92c26ef4d1994462dbbbee24d92d5c5bf4b8b22\": container with ID starting with 38b5c18dc4173d78f48dd6c0b92c26ef4d1994462dbbbee24d92d5c5bf4b8b22 not found: ID does not exist" Jul 06 13:54:16.973660 ip-10-0-136-25 kubenswrapper[2569]: I0706 13:54:16.973614 2569 scope.go:117] "RemoveContainer" containerID="e1ab4427d197ff423c8e4185f16aa7d19d0c84ae571ef8c29a30b53b29671542" Jul 06 13:54:16.973825 ip-10-0-136-25 kubenswrapper[2569]: I0706 13:54:16.973806 2569 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e1ab4427d197ff423c8e4185f16aa7d19d0c84ae571ef8c29a30b53b29671542"} err="failed to get container status \"e1ab4427d197ff423c8e4185f16aa7d19d0c84ae571ef8c29a30b53b29671542\": rpc error: code = NotFound desc = could not find container \"e1ab4427d197ff423c8e4185f16aa7d19d0c84ae571ef8c29a30b53b29671542\": container with ID starting with e1ab4427d197ff423c8e4185f16aa7d19d0c84ae571ef8c29a30b53b29671542 not found: ID does not exist" Jul 06 13:54:16.973863 ip-10-0-136-25 kubenswrapper[2569]: I0706 13:54:16.973826 2569 scope.go:117] "RemoveContainer" containerID="4a911e6732e5fc7953e35b1935bce0e0785fa4e43e1559a3afa6d5af3a45edb3" Jul 06 13:54:16.974033 ip-10-0-136-25 kubenswrapper[2569]: I0706 13:54:16.974016 2569 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4a911e6732e5fc7953e35b1935bce0e0785fa4e43e1559a3afa6d5af3a45edb3"} err="failed to get container status \"4a911e6732e5fc7953e35b1935bce0e0785fa4e43e1559a3afa6d5af3a45edb3\": rpc error: code = NotFound desc = could not find container \"4a911e6732e5fc7953e35b1935bce0e0785fa4e43e1559a3afa6d5af3a45edb3\": container with ID starting with 4a911e6732e5fc7953e35b1935bce0e0785fa4e43e1559a3afa6d5af3a45edb3 not found: ID does not exist" Jul 06 13:54:16.974094 ip-10-0-136-25 kubenswrapper[2569]: I0706 13:54:16.974035 2569 scope.go:117] "RemoveContainer" containerID="cb9c1b6faf60392243d48d6313acb72e7dd5a407840405896b4bc85a5518fbd1" Jul 06 13:54:16.974259 ip-10-0-136-25 kubenswrapper[2569]: I0706 13:54:16.974241 2569 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cb9c1b6faf60392243d48d6313acb72e7dd5a407840405896b4bc85a5518fbd1"} err="failed to get container status \"cb9c1b6faf60392243d48d6313acb72e7dd5a407840405896b4bc85a5518fbd1\": rpc error: code = NotFound desc = could not find container \"cb9c1b6faf60392243d48d6313acb72e7dd5a407840405896b4bc85a5518fbd1\": container with ID starting with cb9c1b6faf60392243d48d6313acb72e7dd5a407840405896b4bc85a5518fbd1 not found: ID does not exist" Jul 06 13:54:16.974313 ip-10-0-136-25 kubenswrapper[2569]: I0706 13:54:16.974259 2569 scope.go:117] "RemoveContainer" containerID="a95b56e8d5a128e827f4fdb56f61e5c0a13e51f3e63e5a4803f17c118e8f855a" Jul 06 13:54:16.974464 ip-10-0-136-25 kubenswrapper[2569]: I0706 13:54:16.974445 2569 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a95b56e8d5a128e827f4fdb56f61e5c0a13e51f3e63e5a4803f17c118e8f855a"} err="failed to get container status \"a95b56e8d5a128e827f4fdb56f61e5c0a13e51f3e63e5a4803f17c118e8f855a\": rpc error: code = NotFound desc = could not find container \"a95b56e8d5a128e827f4fdb56f61e5c0a13e51f3e63e5a4803f17c118e8f855a\": container with ID starting with a95b56e8d5a128e827f4fdb56f61e5c0a13e51f3e63e5a4803f17c118e8f855a not found: ID does not exist" Jul 06 13:54:16.974506 ip-10-0-136-25 kubenswrapper[2569]: I0706 13:54:16.974466 2569 scope.go:117] "RemoveContainer" containerID="9a6205213a3ffd386945d449a6c5658c4a864acbba41e8c357584dc086c95d4d" Jul 06 13:54:16.974681 ip-10-0-136-25 kubenswrapper[2569]: I0706 13:54:16.974665 2569 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9a6205213a3ffd386945d449a6c5658c4a864acbba41e8c357584dc086c95d4d"} err="failed to get container status \"9a6205213a3ffd386945d449a6c5658c4a864acbba41e8c357584dc086c95d4d\": rpc error: code = NotFound desc = could not find container \"9a6205213a3ffd386945d449a6c5658c4a864acbba41e8c357584dc086c95d4d\": container with ID starting with 9a6205213a3ffd386945d449a6c5658c4a864acbba41e8c357584dc086c95d4d not found: ID does not exist" Jul 06 13:54:16.974734 ip-10-0-136-25 kubenswrapper[2569]: I0706 13:54:16.974681 2569 scope.go:117] "RemoveContainer" containerID="68acdea924854dcf3c90ecd92b451cb0b97fe47934007aeb7b804819d89a6e4b" Jul 06 13:54:16.974876 ip-10-0-136-25 kubenswrapper[2569]: I0706 13:54:16.974858 2569 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"68acdea924854dcf3c90ecd92b451cb0b97fe47934007aeb7b804819d89a6e4b"} err="failed to get container status \"68acdea924854dcf3c90ecd92b451cb0b97fe47934007aeb7b804819d89a6e4b\": rpc error: code = NotFound desc = could not find container \"68acdea924854dcf3c90ecd92b451cb0b97fe47934007aeb7b804819d89a6e4b\": container with ID starting with 68acdea924854dcf3c90ecd92b451cb0b97fe47934007aeb7b804819d89a6e4b not found: ID does not exist" Jul 06 13:54:16.974913 ip-10-0-136-25 kubenswrapper[2569]: I0706 13:54:16.974877 2569 scope.go:117] "RemoveContainer" containerID="38b5c18dc4173d78f48dd6c0b92c26ef4d1994462dbbbee24d92d5c5bf4b8b22" Jul 06 13:54:16.975067 ip-10-0-136-25 kubenswrapper[2569]: I0706 13:54:16.975051 2569 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"38b5c18dc4173d78f48dd6c0b92c26ef4d1994462dbbbee24d92d5c5bf4b8b22"} err="failed to get container status \"38b5c18dc4173d78f48dd6c0b92c26ef4d1994462dbbbee24d92d5c5bf4b8b22\": rpc error: code = NotFound desc = could not find container \"38b5c18dc4173d78f48dd6c0b92c26ef4d1994462dbbbee24d92d5c5bf4b8b22\": container with ID starting with 38b5c18dc4173d78f48dd6c0b92c26ef4d1994462dbbbee24d92d5c5bf4b8b22 not found: ID does not exist" Jul 06 13:54:16.975104 ip-10-0-136-25 kubenswrapper[2569]: I0706 13:54:16.975067 2569 scope.go:117] "RemoveContainer" containerID="e1ab4427d197ff423c8e4185f16aa7d19d0c84ae571ef8c29a30b53b29671542" Jul 06 13:54:16.975296 ip-10-0-136-25 kubenswrapper[2569]: I0706 13:54:16.975277 2569 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e1ab4427d197ff423c8e4185f16aa7d19d0c84ae571ef8c29a30b53b29671542"} err="failed to get container status \"e1ab4427d197ff423c8e4185f16aa7d19d0c84ae571ef8c29a30b53b29671542\": rpc error: code = NotFound desc = could not find container \"e1ab4427d197ff423c8e4185f16aa7d19d0c84ae571ef8c29a30b53b29671542\": container with ID starting with e1ab4427d197ff423c8e4185f16aa7d19d0c84ae571ef8c29a30b53b29671542 not found: ID does not exist" Jul 06 13:54:16.975346 ip-10-0-136-25 kubenswrapper[2569]: I0706 13:54:16.975297 2569 scope.go:117] "RemoveContainer" containerID="4a911e6732e5fc7953e35b1935bce0e0785fa4e43e1559a3afa6d5af3a45edb3" Jul 06 13:54:16.975491 ip-10-0-136-25 kubenswrapper[2569]: I0706 13:54:16.975475 2569 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4a911e6732e5fc7953e35b1935bce0e0785fa4e43e1559a3afa6d5af3a45edb3"} err="failed to get container status \"4a911e6732e5fc7953e35b1935bce0e0785fa4e43e1559a3afa6d5af3a45edb3\": rpc error: code = NotFound desc = could not find container \"4a911e6732e5fc7953e35b1935bce0e0785fa4e43e1559a3afa6d5af3a45edb3\": container with ID starting with 4a911e6732e5fc7953e35b1935bce0e0785fa4e43e1559a3afa6d5af3a45edb3 not found: ID does not exist" Jul 06 13:54:16.975534 ip-10-0-136-25 kubenswrapper[2569]: I0706 13:54:16.975492 2569 scope.go:117] "RemoveContainer" containerID="cb9c1b6faf60392243d48d6313acb72e7dd5a407840405896b4bc85a5518fbd1" Jul 06 13:54:16.975684 ip-10-0-136-25 kubenswrapper[2569]: I0706 13:54:16.975667 2569 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cb9c1b6faf60392243d48d6313acb72e7dd5a407840405896b4bc85a5518fbd1"} err="failed to get container status \"cb9c1b6faf60392243d48d6313acb72e7dd5a407840405896b4bc85a5518fbd1\": rpc error: code = NotFound desc = could not find container \"cb9c1b6faf60392243d48d6313acb72e7dd5a407840405896b4bc85a5518fbd1\": container with ID starting with cb9c1b6faf60392243d48d6313acb72e7dd5a407840405896b4bc85a5518fbd1 not found: ID does not exist" Jul 06 13:54:16.975731 ip-10-0-136-25 kubenswrapper[2569]: I0706 13:54:16.975683 2569 scope.go:117] "RemoveContainer" containerID="a95b56e8d5a128e827f4fdb56f61e5c0a13e51f3e63e5a4803f17c118e8f855a" Jul 06 13:54:16.975867 ip-10-0-136-25 kubenswrapper[2569]: I0706 13:54:16.975851 2569 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a95b56e8d5a128e827f4fdb56f61e5c0a13e51f3e63e5a4803f17c118e8f855a"} err="failed to get container status \"a95b56e8d5a128e827f4fdb56f61e5c0a13e51f3e63e5a4803f17c118e8f855a\": rpc error: code = NotFound desc = could not find container \"a95b56e8d5a128e827f4fdb56f61e5c0a13e51f3e63e5a4803f17c118e8f855a\": container with ID starting with a95b56e8d5a128e827f4fdb56f61e5c0a13e51f3e63e5a4803f17c118e8f855a not found: ID does not exist" Jul 06 13:54:16.975942 ip-10-0-136-25 kubenswrapper[2569]: I0706 13:54:16.975868 2569 scope.go:117] "RemoveContainer" containerID="9a6205213a3ffd386945d449a6c5658c4a864acbba41e8c357584dc086c95d4d" Jul 06 13:54:16.976070 ip-10-0-136-25 kubenswrapper[2569]: I0706 13:54:16.976056 2569 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9a6205213a3ffd386945d449a6c5658c4a864acbba41e8c357584dc086c95d4d"} err="failed to get container status \"9a6205213a3ffd386945d449a6c5658c4a864acbba41e8c357584dc086c95d4d\": rpc error: code = NotFound desc = could not find container \"9a6205213a3ffd386945d449a6c5658c4a864acbba41e8c357584dc086c95d4d\": container with ID starting with 9a6205213a3ffd386945d449a6c5658c4a864acbba41e8c357584dc086c95d4d not found: ID does not exist" Jul 06 13:54:16.976110 ip-10-0-136-25 kubenswrapper[2569]: I0706 13:54:16.976070 2569 scope.go:117] "RemoveContainer" containerID="68acdea924854dcf3c90ecd92b451cb0b97fe47934007aeb7b804819d89a6e4b" Jul 06 13:54:16.976284 ip-10-0-136-25 kubenswrapper[2569]: I0706 13:54:16.976268 2569 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"68acdea924854dcf3c90ecd92b451cb0b97fe47934007aeb7b804819d89a6e4b"} err="failed to get container status \"68acdea924854dcf3c90ecd92b451cb0b97fe47934007aeb7b804819d89a6e4b\": rpc error: code = NotFound desc = could not find container \"68acdea924854dcf3c90ecd92b451cb0b97fe47934007aeb7b804819d89a6e4b\": container with ID starting with 68acdea924854dcf3c90ecd92b451cb0b97fe47934007aeb7b804819d89a6e4b not found: ID does not exist" Jul 06 13:54:16.976332 ip-10-0-136-25 kubenswrapper[2569]: I0706 13:54:16.976284 2569 scope.go:117] "RemoveContainer" containerID="38b5c18dc4173d78f48dd6c0b92c26ef4d1994462dbbbee24d92d5c5bf4b8b22" Jul 06 13:54:16.976475 ip-10-0-136-25 kubenswrapper[2569]: I0706 13:54:16.976460 2569 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"38b5c18dc4173d78f48dd6c0b92c26ef4d1994462dbbbee24d92d5c5bf4b8b22"} err="failed to get container status \"38b5c18dc4173d78f48dd6c0b92c26ef4d1994462dbbbee24d92d5c5bf4b8b22\": rpc error: code = NotFound desc = could not find container \"38b5c18dc4173d78f48dd6c0b92c26ef4d1994462dbbbee24d92d5c5bf4b8b22\": container with ID starting with 38b5c18dc4173d78f48dd6c0b92c26ef4d1994462dbbbee24d92d5c5bf4b8b22 not found: ID does not exist" Jul 06 13:54:16.976515 ip-10-0-136-25 kubenswrapper[2569]: I0706 13:54:16.976476 2569 scope.go:117] "RemoveContainer" containerID="e1ab4427d197ff423c8e4185f16aa7d19d0c84ae571ef8c29a30b53b29671542" Jul 06 13:54:16.976683 ip-10-0-136-25 kubenswrapper[2569]: I0706 13:54:16.976667 2569 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e1ab4427d197ff423c8e4185f16aa7d19d0c84ae571ef8c29a30b53b29671542"} err="failed to get container status \"e1ab4427d197ff423c8e4185f16aa7d19d0c84ae571ef8c29a30b53b29671542\": rpc error: code = NotFound desc = could not find container \"e1ab4427d197ff423c8e4185f16aa7d19d0c84ae571ef8c29a30b53b29671542\": container with ID starting with e1ab4427d197ff423c8e4185f16aa7d19d0c84ae571ef8c29a30b53b29671542 not found: ID does not exist" Jul 06 13:54:16.976725 ip-10-0-136-25 kubenswrapper[2569]: I0706 13:54:16.976683 2569 scope.go:117] "RemoveContainer" containerID="4a911e6732e5fc7953e35b1935bce0e0785fa4e43e1559a3afa6d5af3a45edb3" Jul 06 13:54:16.976896 ip-10-0-136-25 kubenswrapper[2569]: I0706 13:54:16.976881 2569 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4a911e6732e5fc7953e35b1935bce0e0785fa4e43e1559a3afa6d5af3a45edb3"} err="failed to get container status \"4a911e6732e5fc7953e35b1935bce0e0785fa4e43e1559a3afa6d5af3a45edb3\": rpc error: code = NotFound desc = could not find container \"4a911e6732e5fc7953e35b1935bce0e0785fa4e43e1559a3afa6d5af3a45edb3\": container with ID starting with 4a911e6732e5fc7953e35b1935bce0e0785fa4e43e1559a3afa6d5af3a45edb3 not found: ID does not exist" Jul 06 13:54:16.976940 ip-10-0-136-25 kubenswrapper[2569]: I0706 13:54:16.976897 2569 scope.go:117] "RemoveContainer" containerID="cb9c1b6faf60392243d48d6313acb72e7dd5a407840405896b4bc85a5518fbd1" Jul 06 13:54:16.977125 ip-10-0-136-25 kubenswrapper[2569]: I0706 13:54:16.977105 2569 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cb9c1b6faf60392243d48d6313acb72e7dd5a407840405896b4bc85a5518fbd1"} err="failed to get container status \"cb9c1b6faf60392243d48d6313acb72e7dd5a407840405896b4bc85a5518fbd1\": rpc error: code = NotFound desc = could not find container \"cb9c1b6faf60392243d48d6313acb72e7dd5a407840405896b4bc85a5518fbd1\": container with ID starting with cb9c1b6faf60392243d48d6313acb72e7dd5a407840405896b4bc85a5518fbd1 not found: ID does not exist" Jul 06 13:54:16.977125 ip-10-0-136-25 kubenswrapper[2569]: I0706 13:54:16.977125 2569 scope.go:117] "RemoveContainer" containerID="a95b56e8d5a128e827f4fdb56f61e5c0a13e51f3e63e5a4803f17c118e8f855a" Jul 06 13:54:16.977356 ip-10-0-136-25 kubenswrapper[2569]: I0706 13:54:16.977340 2569 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a95b56e8d5a128e827f4fdb56f61e5c0a13e51f3e63e5a4803f17c118e8f855a"} err="failed to get container status \"a95b56e8d5a128e827f4fdb56f61e5c0a13e51f3e63e5a4803f17c118e8f855a\": rpc error: code = NotFound desc = could not find container \"a95b56e8d5a128e827f4fdb56f61e5c0a13e51f3e63e5a4803f17c118e8f855a\": container with ID starting with a95b56e8d5a128e827f4fdb56f61e5c0a13e51f3e63e5a4803f17c118e8f855a not found: ID does not exist" Jul 06 13:54:16.977403 ip-10-0-136-25 kubenswrapper[2569]: I0706 13:54:16.977357 2569 scope.go:117] "RemoveContainer" containerID="9a6205213a3ffd386945d449a6c5658c4a864acbba41e8c357584dc086c95d4d" Jul 06 13:54:16.977524 ip-10-0-136-25 kubenswrapper[2569]: I0706 13:54:16.977510 2569 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9a6205213a3ffd386945d449a6c5658c4a864acbba41e8c357584dc086c95d4d"} err="failed to get container status \"9a6205213a3ffd386945d449a6c5658c4a864acbba41e8c357584dc086c95d4d\": rpc error: code = NotFound desc = could not find container \"9a6205213a3ffd386945d449a6c5658c4a864acbba41e8c357584dc086c95d4d\": container with ID starting with 9a6205213a3ffd386945d449a6c5658c4a864acbba41e8c357584dc086c95d4d not found: ID does not exist" Jul 06 13:54:16.977561 ip-10-0-136-25 kubenswrapper[2569]: I0706 13:54:16.977524 2569 scope.go:117] "RemoveContainer" containerID="68acdea924854dcf3c90ecd92b451cb0b97fe47934007aeb7b804819d89a6e4b" Jul 06 13:54:16.977719 ip-10-0-136-25 kubenswrapper[2569]: I0706 13:54:16.977701 2569 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"68acdea924854dcf3c90ecd92b451cb0b97fe47934007aeb7b804819d89a6e4b"} err="failed to get container status \"68acdea924854dcf3c90ecd92b451cb0b97fe47934007aeb7b804819d89a6e4b\": rpc error: code = NotFound desc = could not find container \"68acdea924854dcf3c90ecd92b451cb0b97fe47934007aeb7b804819d89a6e4b\": container with ID starting with 68acdea924854dcf3c90ecd92b451cb0b97fe47934007aeb7b804819d89a6e4b not found: ID does not exist" Jul 06 13:54:16.977763 ip-10-0-136-25 kubenswrapper[2569]: I0706 13:54:16.977720 2569 scope.go:117] "RemoveContainer" containerID="38b5c18dc4173d78f48dd6c0b92c26ef4d1994462dbbbee24d92d5c5bf4b8b22" Jul 06 13:54:16.977894 ip-10-0-136-25 kubenswrapper[2569]: I0706 13:54:16.977879 2569 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"38b5c18dc4173d78f48dd6c0b92c26ef4d1994462dbbbee24d92d5c5bf4b8b22"} err="failed to get container status \"38b5c18dc4173d78f48dd6c0b92c26ef4d1994462dbbbee24d92d5c5bf4b8b22\": rpc error: code = NotFound desc = could not find container \"38b5c18dc4173d78f48dd6c0b92c26ef4d1994462dbbbee24d92d5c5bf4b8b22\": container with ID starting with 38b5c18dc4173d78f48dd6c0b92c26ef4d1994462dbbbee24d92d5c5bf4b8b22 not found: ID does not exist" Jul 06 13:54:16.977930 ip-10-0-136-25 kubenswrapper[2569]: I0706 13:54:16.977894 2569 scope.go:117] "RemoveContainer" containerID="e1ab4427d197ff423c8e4185f16aa7d19d0c84ae571ef8c29a30b53b29671542" Jul 06 13:54:16.978063 ip-10-0-136-25 kubenswrapper[2569]: I0706 13:54:16.978046 2569 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e1ab4427d197ff423c8e4185f16aa7d19d0c84ae571ef8c29a30b53b29671542"} err="failed to get container status \"e1ab4427d197ff423c8e4185f16aa7d19d0c84ae571ef8c29a30b53b29671542\": rpc error: code = NotFound desc = could not find container \"e1ab4427d197ff423c8e4185f16aa7d19d0c84ae571ef8c29a30b53b29671542\": container with ID starting with e1ab4427d197ff423c8e4185f16aa7d19d0c84ae571ef8c29a30b53b29671542 not found: ID does not exist" Jul 06 13:54:17.019787 ip-10-0-136-25 kubenswrapper[2569]: I0706 13:54:17.019766 2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/b958bf61-a426-4f88-8a79-318280126542-thanos-prometheus-http-client-file\") pod \"b958bf61-a426-4f88-8a79-318280126542\" (UID: \"b958bf61-a426-4f88-8a79-318280126542\") " Jul 06 13:54:17.019856 ip-10-0-136-25 kubenswrapper[2569]: I0706 13:54:17.019835 2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/b958bf61-a426-4f88-8a79-318280126542-web-config\") pod \"b958bf61-a426-4f88-8a79-318280126542\" (UID: \"b958bf61-a426-4f88-8a79-318280126542\") " Jul 06 13:54:17.019890 ip-10-0-136-25 kubenswrapper[2569]: I0706 13:54:17.019863 2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b958bf61-a426-4f88-8a79-318280126542-configmap-kubelet-serving-ca-bundle\") pod \"b958bf61-a426-4f88-8a79-318280126542\" (UID: \"b958bf61-a426-4f88-8a79-318280126542\") " Jul 06 13:54:17.019932 ip-10-0-136-25 kubenswrapper[2569]: I0706 13:54:17.019895 2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/b958bf61-a426-4f88-8a79-318280126542-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"b958bf61-a426-4f88-8a79-318280126542\" (UID: \"b958bf61-a426-4f88-8a79-318280126542\") " Jul 06 13:54:17.019932 ip-10-0-136-25 kubenswrapper[2569]: I0706 13:54:17.019922 2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/b958bf61-a426-4f88-8a79-318280126542-prometheus-k8s-rulefiles-1\") pod \"b958bf61-a426-4f88-8a79-318280126542\" (UID: \"b958bf61-a426-4f88-8a79-318280126542\") " Jul 06 13:54:17.020064 ip-10-0-136-25 kubenswrapper[2569]: I0706 13:54:17.020047 2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b958bf61-a426-4f88-8a79-318280126542-prometheus-trusted-ca-bundle\") pod \"b958bf61-a426-4f88-8a79-318280126542\" (UID: \"b958bf61-a426-4f88-8a79-318280126542\") " Jul 06 13:54:17.020141 ip-10-0-136-25 kubenswrapper[2569]: I0706 13:54:17.020082 2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-xrqsm\" (UniqueName: \"kubernetes.io/projected/b958bf61-a426-4f88-8a79-318280126542-kube-api-access-xrqsm\") pod \"b958bf61-a426-4f88-8a79-318280126542\" (UID: \"b958bf61-a426-4f88-8a79-318280126542\") " Jul 06 13:54:17.020141 ip-10-0-136-25 kubenswrapper[2569]: I0706 13:54:17.020108 2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/b958bf61-a426-4f88-8a79-318280126542-prometheus-k8s-rulefiles-0\") pod \"b958bf61-a426-4f88-8a79-318280126542\" (UID: \"b958bf61-a426-4f88-8a79-318280126542\") " Jul 06 13:54:17.020141 ip-10-0-136-25 kubenswrapper[2569]: I0706 13:54:17.020133 2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/b958bf61-a426-4f88-8a79-318280126542-config-out\") pod \"b958bf61-a426-4f88-8a79-318280126542\" (UID: \"b958bf61-a426-4f88-8a79-318280126542\") " Jul 06 13:54:17.020315 ip-10-0-136-25 kubenswrapper[2569]: I0706 13:54:17.020166 2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/b958bf61-a426-4f88-8a79-318280126542-configmap-metrics-client-ca\") pod \"b958bf61-a426-4f88-8a79-318280126542\" (UID: \"b958bf61-a426-4f88-8a79-318280126542\") " Jul 06 13:54:17.020315 ip-10-0-136-25 kubenswrapper[2569]: I0706 13:54:17.020229 2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/b958bf61-a426-4f88-8a79-318280126542-secret-prometheus-k8s-tls\") pod \"b958bf61-a426-4f88-8a79-318280126542\" (UID: \"b958bf61-a426-4f88-8a79-318280126542\") " Jul 06 13:54:17.020315 ip-10-0-136-25 kubenswrapper[2569]: I0706 13:54:17.020262 2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/b958bf61-a426-4f88-8a79-318280126542-secret-grpc-tls\") pod \"b958bf61-a426-4f88-8a79-318280126542\" (UID: \"b958bf61-a426-4f88-8a79-318280126542\") " Jul 06 13:54:17.020315 ip-10-0-136-25 kubenswrapper[2569]: I0706 13:54:17.020298 2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/b958bf61-a426-4f88-8a79-318280126542-secret-metrics-client-certs\") pod \"b958bf61-a426-4f88-8a79-318280126542\" (UID: \"b958bf61-a426-4f88-8a79-318280126542\") " Jul 06 13:54:17.020604 ip-10-0-136-25 kubenswrapper[2569]: I0706 13:54:17.020581 2569 reconciler_common.go:299] "Volume detached for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/b958bf61-a426-4f88-8a79-318280126542-secret-kube-rbac-proxy\") on node \"ip-10-0-136-25.ec2.internal\" DevicePath \"\"" Jul 06 13:54:17.020700 ip-10-0-136-25 kubenswrapper[2569]: I0706 13:54:17.020606 2569 reconciler_common.go:299] "Volume detached for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/b958bf61-a426-4f88-8a79-318280126542-prometheus-k8s-rulefiles-2\") on node \"ip-10-0-136-25.ec2.internal\" DevicePath \"\"" Jul 06 13:54:17.020700 ip-10-0-136-25 kubenswrapper[2569]: I0706 13:54:17.020621 2569 reconciler_common.go:299] "Volume detached for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/b958bf61-a426-4f88-8a79-318280126542-secret-prometheus-k8s-thanos-sidecar-tls\") on node \"ip-10-0-136-25.ec2.internal\" DevicePath \"\"" Jul 06 13:54:17.020700 ip-10-0-136-25 kubenswrapper[2569]: I0706 13:54:17.020681 2569 reconciler_common.go:299] "Volume detached for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/b958bf61-a426-4f88-8a79-318280126542-tls-assets\") on node \"ip-10-0-136-25.ec2.internal\" DevicePath \"\"" Jul 06 13:54:17.020700 ip-10-0-136-25 kubenswrapper[2569]: I0706 13:54:17.020698 2569 reconciler_common.go:299] "Volume detached for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b958bf61-a426-4f88-8a79-318280126542-configmap-serving-certs-ca-bundle\") on node \"ip-10-0-136-25.ec2.internal\" DevicePath \"\"" Jul 06 13:54:17.020987 ip-10-0-136-25 kubenswrapper[2569]: I0706 13:54:17.020712 2569 reconciler_common.go:299] "Volume detached for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/b958bf61-a426-4f88-8a79-318280126542-prometheus-k8s-db\") on node \"ip-10-0-136-25.ec2.internal\" DevicePath \"\"" Jul 06 13:54:17.020987 ip-10-0-136-25 kubenswrapper[2569]: I0706 13:54:17.020258 2569 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b958bf61-a426-4f88-8a79-318280126542-prometheus-k8s-rulefiles-1" (OuterVolumeSpecName: "prometheus-k8s-rulefiles-1") pod "b958bf61-a426-4f88-8a79-318280126542" (UID: "b958bf61-a426-4f88-8a79-318280126542"). InnerVolumeSpecName "prometheus-k8s-rulefiles-1". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 06 13:54:17.020987 ip-10-0-136-25 kubenswrapper[2569]: I0706 13:54:17.020727 2569 reconciler_common.go:299] "Volume detached for volume \"config\" (UniqueName: \"kubernetes.io/secret/b958bf61-a426-4f88-8a79-318280126542-config\") on node \"ip-10-0-136-25.ec2.internal\" DevicePath \"\"" Jul 06 13:54:17.020987 ip-10-0-136-25 kubenswrapper[2569]: I0706 13:54:17.020375 2569 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b958bf61-a426-4f88-8a79-318280126542-configmap-kubelet-serving-ca-bundle" (OuterVolumeSpecName: "configmap-kubelet-serving-ca-bundle") pod "b958bf61-a426-4f88-8a79-318280126542" (UID: "b958bf61-a426-4f88-8a79-318280126542"). InnerVolumeSpecName "configmap-kubelet-serving-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 06 13:54:17.020987 ip-10-0-136-25 kubenswrapper[2569]: I0706 13:54:17.020461 2569 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b958bf61-a426-4f88-8a79-318280126542-prometheus-trusted-ca-bundle" (OuterVolumeSpecName: "prometheus-trusted-ca-bundle") pod "b958bf61-a426-4f88-8a79-318280126542" (UID: "b958bf61-a426-4f88-8a79-318280126542"). InnerVolumeSpecName "prometheus-trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 06 13:54:17.021327 ip-10-0-136-25 kubenswrapper[2569]: I0706 13:54:17.021194 2569 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b958bf61-a426-4f88-8a79-318280126542-configmap-metrics-client-ca" (OuterVolumeSpecName: "configmap-metrics-client-ca") pod "b958bf61-a426-4f88-8a79-318280126542" (UID: "b958bf61-a426-4f88-8a79-318280126542"). InnerVolumeSpecName "configmap-metrics-client-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 06 13:54:17.022489 ip-10-0-136-25 kubenswrapper[2569]: I0706 13:54:17.022461 2569 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b958bf61-a426-4f88-8a79-318280126542-secret-prometheus-k8s-kube-rbac-proxy-web" (OuterVolumeSpecName: "secret-prometheus-k8s-kube-rbac-proxy-web") pod "b958bf61-a426-4f88-8a79-318280126542" (UID: "b958bf61-a426-4f88-8a79-318280126542"). InnerVolumeSpecName "secret-prometheus-k8s-kube-rbac-proxy-web". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 06 13:54:17.022583 ip-10-0-136-25 kubenswrapper[2569]: I0706 13:54:17.022503 2569 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b958bf61-a426-4f88-8a79-318280126542-thanos-prometheus-http-client-file" (OuterVolumeSpecName: "thanos-prometheus-http-client-file") pod "b958bf61-a426-4f88-8a79-318280126542" (UID: "b958bf61-a426-4f88-8a79-318280126542"). InnerVolumeSpecName "thanos-prometheus-http-client-file". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 06 13:54:17.022733 ip-10-0-136-25 kubenswrapper[2569]: I0706 13:54:17.022707 2569 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b958bf61-a426-4f88-8a79-318280126542-config-out" (OuterVolumeSpecName: "config-out") pod "b958bf61-a426-4f88-8a79-318280126542" (UID: "b958bf61-a426-4f88-8a79-318280126542"). InnerVolumeSpecName "config-out". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 06 13:54:17.022821 ip-10-0-136-25 kubenswrapper[2569]: I0706 13:54:17.022711 2569 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b958bf61-a426-4f88-8a79-318280126542-secret-prometheus-k8s-tls" (OuterVolumeSpecName: "secret-prometheus-k8s-tls") pod "b958bf61-a426-4f88-8a79-318280126542" (UID: "b958bf61-a426-4f88-8a79-318280126542"). InnerVolumeSpecName "secret-prometheus-k8s-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 06 13:54:17.022922 ip-10-0-136-25 kubenswrapper[2569]: I0706 13:54:17.022899 2569 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b958bf61-a426-4f88-8a79-318280126542-prometheus-k8s-rulefiles-0" (OuterVolumeSpecName: "prometheus-k8s-rulefiles-0") pod "b958bf61-a426-4f88-8a79-318280126542" (UID: "b958bf61-a426-4f88-8a79-318280126542"). InnerVolumeSpecName "prometheus-k8s-rulefiles-0". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 06 13:54:17.023416 ip-10-0-136-25 kubenswrapper[2569]: I0706 13:54:17.023397 2569 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b958bf61-a426-4f88-8a79-318280126542-kube-api-access-xrqsm" (OuterVolumeSpecName: "kube-api-access-xrqsm") pod "b958bf61-a426-4f88-8a79-318280126542" (UID: "b958bf61-a426-4f88-8a79-318280126542"). InnerVolumeSpecName "kube-api-access-xrqsm". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 06 13:54:17.023612 ip-10-0-136-25 kubenswrapper[2569]: I0706 13:54:17.023594 2569 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b958bf61-a426-4f88-8a79-318280126542-secret-metrics-client-certs" (OuterVolumeSpecName: "secret-metrics-client-certs") pod "b958bf61-a426-4f88-8a79-318280126542" (UID: "b958bf61-a426-4f88-8a79-318280126542"). InnerVolumeSpecName "secret-metrics-client-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 06 13:54:17.023792 ip-10-0-136-25 kubenswrapper[2569]: I0706 13:54:17.023776 2569 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b958bf61-a426-4f88-8a79-318280126542-secret-grpc-tls" (OuterVolumeSpecName: "secret-grpc-tls") pod "b958bf61-a426-4f88-8a79-318280126542" (UID: "b958bf61-a426-4f88-8a79-318280126542"). InnerVolumeSpecName "secret-grpc-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 06 13:54:17.032945 ip-10-0-136-25 kubenswrapper[2569]: I0706 13:54:17.032928 2569 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b958bf61-a426-4f88-8a79-318280126542-web-config" (OuterVolumeSpecName: "web-config") pod "b958bf61-a426-4f88-8a79-318280126542" (UID: "b958bf61-a426-4f88-8a79-318280126542"). InnerVolumeSpecName "web-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 06 13:54:17.121472 ip-10-0-136-25 kubenswrapper[2569]: I0706 13:54:17.121439 2569 reconciler_common.go:299] "Volume detached for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/b958bf61-a426-4f88-8a79-318280126542-web-config\") on node \"ip-10-0-136-25.ec2.internal\" DevicePath \"\"" Jul 06 13:54:17.121472 ip-10-0-136-25 kubenswrapper[2569]: I0706 13:54:17.121469 2569 reconciler_common.go:299] "Volume detached for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b958bf61-a426-4f88-8a79-318280126542-configmap-kubelet-serving-ca-bundle\") on node \"ip-10-0-136-25.ec2.internal\" DevicePath \"\"" Jul 06 13:54:17.121649 ip-10-0-136-25 kubenswrapper[2569]: I0706 13:54:17.121480 2569 reconciler_common.go:299] "Volume detached for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/b958bf61-a426-4f88-8a79-318280126542-secret-prometheus-k8s-kube-rbac-proxy-web\") on node \"ip-10-0-136-25.ec2.internal\" DevicePath \"\"" Jul 06 13:54:17.121649 ip-10-0-136-25 kubenswrapper[2569]: I0706 13:54:17.121491 2569 reconciler_common.go:299] "Volume detached for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/b958bf61-a426-4f88-8a79-318280126542-prometheus-k8s-rulefiles-1\") on node \"ip-10-0-136-25.ec2.internal\" DevicePath \"\"" Jul 06 13:54:17.121649 ip-10-0-136-25 kubenswrapper[2569]: I0706 13:54:17.121499 2569 reconciler_common.go:299] "Volume detached for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b958bf61-a426-4f88-8a79-318280126542-prometheus-trusted-ca-bundle\") on node \"ip-10-0-136-25.ec2.internal\" DevicePath \"\"" Jul 06 13:54:17.121649 ip-10-0-136-25 kubenswrapper[2569]: I0706 13:54:17.121510 2569 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xrqsm\" (UniqueName: \"kubernetes.io/projected/b958bf61-a426-4f88-8a79-318280126542-kube-api-access-xrqsm\") on node \"ip-10-0-136-25.ec2.internal\" DevicePath \"\"" Jul 06 13:54:17.121649 ip-10-0-136-25 kubenswrapper[2569]: I0706 13:54:17.121519 2569 reconciler_common.go:299] "Volume detached for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/b958bf61-a426-4f88-8a79-318280126542-prometheus-k8s-rulefiles-0\") on node \"ip-10-0-136-25.ec2.internal\" DevicePath \"\"" Jul 06 13:54:17.121649 ip-10-0-136-25 kubenswrapper[2569]: I0706 13:54:17.121528 2569 reconciler_common.go:299] "Volume detached for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/b958bf61-a426-4f88-8a79-318280126542-config-out\") on node \"ip-10-0-136-25.ec2.internal\" DevicePath \"\"" Jul 06 13:54:17.121649 ip-10-0-136-25 kubenswrapper[2569]: I0706 13:54:17.121536 2569 reconciler_common.go:299] "Volume detached for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/b958bf61-a426-4f88-8a79-318280126542-configmap-metrics-client-ca\") on node \"ip-10-0-136-25.ec2.internal\" DevicePath \"\"" Jul 06 13:54:17.121649 ip-10-0-136-25 kubenswrapper[2569]: I0706 13:54:17.121544 2569 reconciler_common.go:299] "Volume detached for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/b958bf61-a426-4f88-8a79-318280126542-secret-prometheus-k8s-tls\") on node \"ip-10-0-136-25.ec2.internal\" DevicePath \"\"" Jul 06 13:54:17.121649 ip-10-0-136-25 kubenswrapper[2569]: I0706 13:54:17.121554 2569 reconciler_common.go:299] "Volume detached for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/b958bf61-a426-4f88-8a79-318280126542-secret-grpc-tls\") on node \"ip-10-0-136-25.ec2.internal\" DevicePath \"\"" Jul 06 13:54:17.121649 ip-10-0-136-25 kubenswrapper[2569]: I0706 13:54:17.121562 2569 reconciler_common.go:299] "Volume detached for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/b958bf61-a426-4f88-8a79-318280126542-secret-metrics-client-certs\") on node \"ip-10-0-136-25.ec2.internal\" DevicePath \"\"" Jul 06 13:54:17.121649 ip-10-0-136-25 kubenswrapper[2569]: I0706 13:54:17.121570 2569 reconciler_common.go:299] "Volume detached for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/b958bf61-a426-4f88-8a79-318280126542-thanos-prometheus-http-client-file\") on node \"ip-10-0-136-25.ec2.internal\" DevicePath \"\"" Jul 06 13:54:17.238873 ip-10-0-136-25 kubenswrapper[2569]: I0706 13:54:17.238846 2569 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Jul 06 13:54:17.242552 ip-10-0-136-25 kubenswrapper[2569]: I0706 13:54:17.242486 2569 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Jul 06 13:54:17.269047 ip-10-0-136-25 kubenswrapper[2569]: I0706 13:54:17.269021 2569 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Jul 06 13:54:17.269368 ip-10-0-136-25 kubenswrapper[2569]: I0706 13:54:17.269355 2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b958bf61-a426-4f88-8a79-318280126542" containerName="init-config-reloader" Jul 06 13:54:17.269418 ip-10-0-136-25 kubenswrapper[2569]: I0706 13:54:17.269370 2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="b958bf61-a426-4f88-8a79-318280126542" containerName="init-config-reloader" Jul 06 13:54:17.269418 ip-10-0-136-25 kubenswrapper[2569]: I0706 13:54:17.269380 2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b958bf61-a426-4f88-8a79-318280126542" containerName="kube-rbac-proxy-thanos" Jul 06 13:54:17.269418 ip-10-0-136-25 kubenswrapper[2569]: I0706 13:54:17.269388 2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="b958bf61-a426-4f88-8a79-318280126542" containerName="kube-rbac-proxy-thanos" Jul 06 13:54:17.269418 ip-10-0-136-25 kubenswrapper[2569]: I0706 13:54:17.269408 2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b958bf61-a426-4f88-8a79-318280126542" containerName="prometheus" Jul 06 13:54:17.269418 ip-10-0-136-25 kubenswrapper[2569]: I0706 13:54:17.269414 2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="b958bf61-a426-4f88-8a79-318280126542" containerName="prometheus" Jul 06 13:54:17.269564 ip-10-0-136-25 kubenswrapper[2569]: I0706 13:54:17.269426 2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b958bf61-a426-4f88-8a79-318280126542" containerName="kube-rbac-proxy-web" Jul 06 13:54:17.269564 ip-10-0-136-25 kubenswrapper[2569]: I0706 13:54:17.269432 2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="b958bf61-a426-4f88-8a79-318280126542" containerName="kube-rbac-proxy-web" Jul 06 13:54:17.269564 ip-10-0-136-25 kubenswrapper[2569]: I0706 13:54:17.269439 2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b958bf61-a426-4f88-8a79-318280126542" containerName="config-reloader" Jul 06 13:54:17.269564 ip-10-0-136-25 kubenswrapper[2569]: I0706 13:54:17.269444 2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="b958bf61-a426-4f88-8a79-318280126542" containerName="config-reloader" Jul 06 13:54:17.269564 ip-10-0-136-25 kubenswrapper[2569]: I0706 13:54:17.269450 2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b958bf61-a426-4f88-8a79-318280126542" containerName="thanos-sidecar" Jul 06 13:54:17.269564 ip-10-0-136-25 kubenswrapper[2569]: I0706 13:54:17.269455 2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="b958bf61-a426-4f88-8a79-318280126542" containerName="thanos-sidecar" Jul 06 13:54:17.269564 ip-10-0-136-25 kubenswrapper[2569]: I0706 13:54:17.269464 2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b958bf61-a426-4f88-8a79-318280126542" containerName="kube-rbac-proxy" Jul 06 13:54:17.269564 ip-10-0-136-25 kubenswrapper[2569]: I0706 13:54:17.269473 2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="b958bf61-a426-4f88-8a79-318280126542" containerName="kube-rbac-proxy" Jul 06 13:54:17.269564 ip-10-0-136-25 kubenswrapper[2569]: I0706 13:54:17.269524 2569 memory_manager.go:368] "RemoveStaleState removing state" podUID="b958bf61-a426-4f88-8a79-318280126542" containerName="kube-rbac-proxy" Jul 06 13:54:17.269564 ip-10-0-136-25 kubenswrapper[2569]: I0706 13:54:17.269535 2569 memory_manager.go:368] "RemoveStaleState removing state" podUID="b958bf61-a426-4f88-8a79-318280126542" containerName="thanos-sidecar" Jul 06 13:54:17.269564 ip-10-0-136-25 kubenswrapper[2569]: I0706 13:54:17.269542 2569 memory_manager.go:368] "RemoveStaleState removing state" podUID="b958bf61-a426-4f88-8a79-318280126542" containerName="kube-rbac-proxy-web" Jul 06 13:54:17.269564 ip-10-0-136-25 kubenswrapper[2569]: I0706 13:54:17.269552 2569 memory_manager.go:368] "RemoveStaleState removing state" podUID="b958bf61-a426-4f88-8a79-318280126542" containerName="prometheus" Jul 06 13:54:17.269564 ip-10-0-136-25 kubenswrapper[2569]: I0706 13:54:17.269561 2569 memory_manager.go:368] "RemoveStaleState removing state" podUID="b958bf61-a426-4f88-8a79-318280126542" containerName="kube-rbac-proxy-thanos" Jul 06 13:54:17.269564 ip-10-0-136-25 kubenswrapper[2569]: I0706 13:54:17.269569 2569 memory_manager.go:368] "RemoveStaleState removing state" podUID="b958bf61-a426-4f88-8a79-318280126542" containerName="config-reloader" Jul 06 13:54:17.273382 ip-10-0-136-25 kubenswrapper[2569]: I0706 13:54:17.273367 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0" Jul 06 13:54:17.275486 ip-10-0-136-25 kubenswrapper[2569]: I0706 13:54:17.275450 2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-prometheus-http-client-file\"" Jul 06 13:54:17.275486 ip-10-0-136-25 kubenswrapper[2569]: I0706 13:54:17.275454 2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-kube-rbac-proxy-web\"" Jul 06 13:54:17.275763 ip-10-0-136-25 kubenswrapper[2569]: I0706 13:54:17.275742 2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-client-certs\"" Jul 06 13:54:17.275861 ip-10-0-136-25 kubenswrapper[2569]: I0706 13:54:17.275799 2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kubelet-serving-ca-bundle\"" Jul 06 13:54:17.275915 ip-10-0-136-25 kubenswrapper[2569]: I0706 13:54:17.275855 2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-web-config\"" Jul 06 13:54:17.276100 ip-10-0-136-25 kubenswrapper[2569]: I0706 13:54:17.276081 2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-sidecar-tls\"" Jul 06 13:54:17.276198 ip-10-0-136-25 kubenswrapper[2569]: I0706 13:54:17.276184 2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls-assets-0\"" Jul 06 13:54:17.276867 ip-10-0-136-25 kubenswrapper[2569]: I0706 13:54:17.276850 2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-dockercfg-hk5tf\"" Jul 06 13:54:17.276967 ip-10-0-136-25 kubenswrapper[2569]: I0706 13:54:17.276909 2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-rbac-proxy\"" Jul 06 13:54:17.276967 ip-10-0-136-25 kubenswrapper[2569]: I0706 13:54:17.276925 2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-1\"" Jul 06 13:54:17.276967 ip-10-0-136-25 kubenswrapper[2569]: I0706 13:54:17.276927 2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"serving-certs-ca-bundle\"" Jul 06 13:54:17.277137 ip-10-0-136-25 kubenswrapper[2569]: I0706 13:54:17.277128 2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-grpc-tls-30phi2tl4q3fv\"" Jul 06 13:54:17.277309 ip-10-0-136-25 kubenswrapper[2569]: I0706 13:54:17.277285 2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls\"" Jul 06 13:54:17.277357 ip-10-0-136-25 kubenswrapper[2569]: I0706 13:54:17.277324 2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-2\"" Jul 06 13:54:17.278095 ip-10-0-136-25 kubenswrapper[2569]: I0706 13:54:17.278081 2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s\"" Jul 06 13:54:17.279772 ip-10-0-136-25 kubenswrapper[2569]: I0706 13:54:17.279753 2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-0\"" Jul 06 13:54:17.282728 ip-10-0-136-25 kubenswrapper[2569]: I0706 13:54:17.282709 2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-trusted-ca-bundle\"" Jul 06 13:54:17.288205 ip-10-0-136-25 kubenswrapper[2569]: I0706 13:54:17.288184 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Jul 06 13:54:17.322391 ip-10-0-136-25 kubenswrapper[2569]: I0706 13:54:17.322359 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/1c64c06c-4a9b-4767-bdd8-cc5fb5bed504-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"1c64c06c-4a9b-4767-bdd8-cc5fb5bed504\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 06 13:54:17.322391 ip-10-0-136-25 kubenswrapper[2569]: I0706 13:54:17.322390 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/1c64c06c-4a9b-4767-bdd8-cc5fb5bed504-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"1c64c06c-4a9b-4767-bdd8-cc5fb5bed504\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 06 13:54:17.322583 ip-10-0-136-25 kubenswrapper[2569]: I0706 13:54:17.322409 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/1c64c06c-4a9b-4767-bdd8-cc5fb5bed504-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"1c64c06c-4a9b-4767-bdd8-cc5fb5bed504\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 06 13:54:17.322583 ip-10-0-136-25 kubenswrapper[2569]: I0706 13:54:17.322432 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-l8kkj\" (UniqueName: \"kubernetes.io/projected/1c64c06c-4a9b-4767-bdd8-cc5fb5bed504-kube-api-access-l8kkj\") pod \"prometheus-k8s-0\" (UID: \"1c64c06c-4a9b-4767-bdd8-cc5fb5bed504\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 06 13:54:17.322583 ip-10-0-136-25 kubenswrapper[2569]: I0706 13:54:17.322497 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/1c64c06c-4a9b-4767-bdd8-cc5fb5bed504-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"1c64c06c-4a9b-4767-bdd8-cc5fb5bed504\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 06 13:54:17.322583 ip-10-0-136-25 kubenswrapper[2569]: I0706 13:54:17.322526 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/1c64c06c-4a9b-4767-bdd8-cc5fb5bed504-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"1c64c06c-4a9b-4767-bdd8-cc5fb5bed504\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 06 13:54:17.322733 ip-10-0-136-25 kubenswrapper[2569]: I0706 13:54:17.322594 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/1c64c06c-4a9b-4767-bdd8-cc5fb5bed504-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"1c64c06c-4a9b-4767-bdd8-cc5fb5bed504\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 06 13:54:17.322733 ip-10-0-136-25 kubenswrapper[2569]: I0706 13:54:17.322652 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/1c64c06c-4a9b-4767-bdd8-cc5fb5bed504-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"1c64c06c-4a9b-4767-bdd8-cc5fb5bed504\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 06 13:54:17.322733 ip-10-0-136-25 kubenswrapper[2569]: I0706 13:54:17.322672 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/1c64c06c-4a9b-4767-bdd8-cc5fb5bed504-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"1c64c06c-4a9b-4767-bdd8-cc5fb5bed504\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 06 13:54:17.322733 ip-10-0-136-25 kubenswrapper[2569]: I0706 13:54:17.322693 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/1c64c06c-4a9b-4767-bdd8-cc5fb5bed504-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"1c64c06c-4a9b-4767-bdd8-cc5fb5bed504\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 06 13:54:17.322733 ip-10-0-136-25 kubenswrapper[2569]: I0706 13:54:17.322713 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/1c64c06c-4a9b-4767-bdd8-cc5fb5bed504-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"1c64c06c-4a9b-4767-bdd8-cc5fb5bed504\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 06 13:54:17.322733 ip-10-0-136-25 kubenswrapper[2569]: I0706 13:54:17.322729 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/1c64c06c-4a9b-4767-bdd8-cc5fb5bed504-config-out\") pod \"prometheus-k8s-0\" (UID: \"1c64c06c-4a9b-4767-bdd8-cc5fb5bed504\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 06 13:54:17.323001 ip-10-0-136-25 kubenswrapper[2569]: I0706 13:54:17.322744 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/1c64c06c-4a9b-4767-bdd8-cc5fb5bed504-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"1c64c06c-4a9b-4767-bdd8-cc5fb5bed504\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 06 13:54:17.323001 ip-10-0-136-25 kubenswrapper[2569]: I0706 13:54:17.322769 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/1c64c06c-4a9b-4767-bdd8-cc5fb5bed504-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"1c64c06c-4a9b-4767-bdd8-cc5fb5bed504\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 06 13:54:17.323001 ip-10-0-136-25 kubenswrapper[2569]: I0706 13:54:17.322793 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/1c64c06c-4a9b-4767-bdd8-cc5fb5bed504-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"1c64c06c-4a9b-4767-bdd8-cc5fb5bed504\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 06 13:54:17.323001 ip-10-0-136-25 kubenswrapper[2569]: I0706 13:54:17.322852 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/1c64c06c-4a9b-4767-bdd8-cc5fb5bed504-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"1c64c06c-4a9b-4767-bdd8-cc5fb5bed504\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 06 13:54:17.323001 ip-10-0-136-25 kubenswrapper[2569]: I0706 13:54:17.322890 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/1c64c06c-4a9b-4767-bdd8-cc5fb5bed504-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"1c64c06c-4a9b-4767-bdd8-cc5fb5bed504\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 06 13:54:17.323001 ip-10-0-136-25 kubenswrapper[2569]: I0706 13:54:17.322944 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/1c64c06c-4a9b-4767-bdd8-cc5fb5bed504-config\") pod \"prometheus-k8s-0\" (UID: \"1c64c06c-4a9b-4767-bdd8-cc5fb5bed504\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 06 13:54:17.328028 ip-10-0-136-25 kubenswrapper[2569]: I0706 13:54:17.322997 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/1c64c06c-4a9b-4767-bdd8-cc5fb5bed504-web-config\") pod \"prometheus-k8s-0\" (UID: \"1c64c06c-4a9b-4767-bdd8-cc5fb5bed504\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 06 13:54:17.328028 ip-10-0-136-25 kubenswrapper[2569]: I0706 13:54:17.323649 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/1c64c06c-4a9b-4767-bdd8-cc5fb5bed504-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"1c64c06c-4a9b-4767-bdd8-cc5fb5bed504\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 06 13:54:17.424515 ip-10-0-136-25 kubenswrapper[2569]: I0706 13:54:17.424474 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/1c64c06c-4a9b-4767-bdd8-cc5fb5bed504-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"1c64c06c-4a9b-4767-bdd8-cc5fb5bed504\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 06 13:54:17.424515 ip-10-0-136-25 kubenswrapper[2569]: I0706 13:54:17.424516 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/1c64c06c-4a9b-4767-bdd8-cc5fb5bed504-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"1c64c06c-4a9b-4767-bdd8-cc5fb5bed504\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 06 13:54:17.424767 ip-10-0-136-25 kubenswrapper[2569]: I0706 13:54:17.424538 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/1c64c06c-4a9b-4767-bdd8-cc5fb5bed504-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"1c64c06c-4a9b-4767-bdd8-cc5fb5bed504\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 06 13:54:17.424767 ip-10-0-136-25 kubenswrapper[2569]: I0706 13:54:17.424559 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/1c64c06c-4a9b-4767-bdd8-cc5fb5bed504-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"1c64c06c-4a9b-4767-bdd8-cc5fb5bed504\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 06 13:54:17.424767 ip-10-0-136-25 kubenswrapper[2569]: I0706 13:54:17.424680 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/1c64c06c-4a9b-4767-bdd8-cc5fb5bed504-config-out\") pod \"prometheus-k8s-0\" (UID: \"1c64c06c-4a9b-4767-bdd8-cc5fb5bed504\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 06 13:54:17.424767 ip-10-0-136-25 kubenswrapper[2569]: I0706 13:54:17.424722 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/1c64c06c-4a9b-4767-bdd8-cc5fb5bed504-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"1c64c06c-4a9b-4767-bdd8-cc5fb5bed504\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 06 13:54:17.424767 ip-10-0-136-25 kubenswrapper[2569]: I0706 13:54:17.424768 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/1c64c06c-4a9b-4767-bdd8-cc5fb5bed504-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"1c64c06c-4a9b-4767-bdd8-cc5fb5bed504\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 06 13:54:17.425026 ip-10-0-136-25 kubenswrapper[2569]: I0706 13:54:17.424813 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/1c64c06c-4a9b-4767-bdd8-cc5fb5bed504-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"1c64c06c-4a9b-4767-bdd8-cc5fb5bed504\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 06 13:54:17.425026 ip-10-0-136-25 kubenswrapper[2569]: I0706 13:54:17.424853 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/1c64c06c-4a9b-4767-bdd8-cc5fb5bed504-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"1c64c06c-4a9b-4767-bdd8-cc5fb5bed504\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 06 13:54:17.425026 ip-10-0-136-25 kubenswrapper[2569]: I0706 13:54:17.424895 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/1c64c06c-4a9b-4767-bdd8-cc5fb5bed504-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"1c64c06c-4a9b-4767-bdd8-cc5fb5bed504\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 06 13:54:17.425026 ip-10-0-136-25 kubenswrapper[2569]: I0706 13:54:17.424935 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/1c64c06c-4a9b-4767-bdd8-cc5fb5bed504-config\") pod \"prometheus-k8s-0\" (UID: \"1c64c06c-4a9b-4767-bdd8-cc5fb5bed504\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 06 13:54:17.425026 ip-10-0-136-25 kubenswrapper[2569]: I0706 13:54:17.424972 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/1c64c06c-4a9b-4767-bdd8-cc5fb5bed504-web-config\") pod \"prometheus-k8s-0\" (UID: \"1c64c06c-4a9b-4767-bdd8-cc5fb5bed504\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 06 13:54:17.425026 ip-10-0-136-25 kubenswrapper[2569]: I0706 13:54:17.425016 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/1c64c06c-4a9b-4767-bdd8-cc5fb5bed504-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"1c64c06c-4a9b-4767-bdd8-cc5fb5bed504\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 06 13:54:17.425343 ip-10-0-136-25 kubenswrapper[2569]: I0706 13:54:17.425055 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/1c64c06c-4a9b-4767-bdd8-cc5fb5bed504-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"1c64c06c-4a9b-4767-bdd8-cc5fb5bed504\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 06 13:54:17.425343 ip-10-0-136-25 kubenswrapper[2569]: I0706 13:54:17.425081 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/1c64c06c-4a9b-4767-bdd8-cc5fb5bed504-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"1c64c06c-4a9b-4767-bdd8-cc5fb5bed504\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 06 13:54:17.425343 ip-10-0-136-25 kubenswrapper[2569]: I0706 13:54:17.425113 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/1c64c06c-4a9b-4767-bdd8-cc5fb5bed504-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"1c64c06c-4a9b-4767-bdd8-cc5fb5bed504\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 06 13:54:17.425343 ip-10-0-136-25 kubenswrapper[2569]: I0706 13:54:17.425149 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l8kkj\" (UniqueName: \"kubernetes.io/projected/1c64c06c-4a9b-4767-bdd8-cc5fb5bed504-kube-api-access-l8kkj\") pod \"prometheus-k8s-0\" (UID: \"1c64c06c-4a9b-4767-bdd8-cc5fb5bed504\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 06 13:54:17.425343 ip-10-0-136-25 kubenswrapper[2569]: I0706 13:54:17.425196 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/1c64c06c-4a9b-4767-bdd8-cc5fb5bed504-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"1c64c06c-4a9b-4767-bdd8-cc5fb5bed504\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 06 13:54:17.425343 ip-10-0-136-25 kubenswrapper[2569]: I0706 13:54:17.425252 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/1c64c06c-4a9b-4767-bdd8-cc5fb5bed504-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"1c64c06c-4a9b-4767-bdd8-cc5fb5bed504\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 06 13:54:17.425343 ip-10-0-136-25 kubenswrapper[2569]: I0706 13:54:17.425284 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/1c64c06c-4a9b-4767-bdd8-cc5fb5bed504-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"1c64c06c-4a9b-4767-bdd8-cc5fb5bed504\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 06 13:54:17.425677 ip-10-0-136-25 kubenswrapper[2569]: I0706 13:54:17.425346 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/1c64c06c-4a9b-4767-bdd8-cc5fb5bed504-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"1c64c06c-4a9b-4767-bdd8-cc5fb5bed504\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 06 13:54:17.425677 ip-10-0-136-25 kubenswrapper[2569]: I0706 13:54:17.425462 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/1c64c06c-4a9b-4767-bdd8-cc5fb5bed504-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"1c64c06c-4a9b-4767-bdd8-cc5fb5bed504\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 06 13:54:17.425896 ip-10-0-136-25 kubenswrapper[2569]: I0706 13:54:17.425870 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/1c64c06c-4a9b-4767-bdd8-cc5fb5bed504-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"1c64c06c-4a9b-4767-bdd8-cc5fb5bed504\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 06 13:54:17.426190 ip-10-0-136-25 kubenswrapper[2569]: I0706 13:54:17.426166 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/1c64c06c-4a9b-4767-bdd8-cc5fb5bed504-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"1c64c06c-4a9b-4767-bdd8-cc5fb5bed504\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 06 13:54:17.426998 ip-10-0-136-25 kubenswrapper[2569]: I0706 13:54:17.426970 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/1c64c06c-4a9b-4767-bdd8-cc5fb5bed504-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"1c64c06c-4a9b-4767-bdd8-cc5fb5bed504\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 06 13:54:17.428189 ip-10-0-136-25 kubenswrapper[2569]: I0706 13:54:17.427647 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/1c64c06c-4a9b-4767-bdd8-cc5fb5bed504-config-out\") pod \"prometheus-k8s-0\" (UID: \"1c64c06c-4a9b-4767-bdd8-cc5fb5bed504\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 06 13:54:17.428189 ip-10-0-136-25 kubenswrapper[2569]: I0706 13:54:17.427743 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/1c64c06c-4a9b-4767-bdd8-cc5fb5bed504-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"1c64c06c-4a9b-4767-bdd8-cc5fb5bed504\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 06 13:54:17.428189 ip-10-0-136-25 kubenswrapper[2569]: I0706 13:54:17.427816 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/1c64c06c-4a9b-4767-bdd8-cc5fb5bed504-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"1c64c06c-4a9b-4767-bdd8-cc5fb5bed504\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 06 13:54:17.428189 ip-10-0-136-25 kubenswrapper[2569]: I0706 13:54:17.427869 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/1c64c06c-4a9b-4767-bdd8-cc5fb5bed504-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"1c64c06c-4a9b-4767-bdd8-cc5fb5bed504\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 06 13:54:17.428189 ip-10-0-136-25 kubenswrapper[2569]: I0706 13:54:17.428012 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/1c64c06c-4a9b-4767-bdd8-cc5fb5bed504-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"1c64c06c-4a9b-4767-bdd8-cc5fb5bed504\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 06 13:54:17.428189 ip-10-0-136-25 kubenswrapper[2569]: I0706 13:54:17.428087 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/1c64c06c-4a9b-4767-bdd8-cc5fb5bed504-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"1c64c06c-4a9b-4767-bdd8-cc5fb5bed504\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 06 13:54:17.428757 ip-10-0-136-25 kubenswrapper[2569]: I0706 13:54:17.428477 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/1c64c06c-4a9b-4767-bdd8-cc5fb5bed504-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"1c64c06c-4a9b-4767-bdd8-cc5fb5bed504\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 06 13:54:17.428757 ip-10-0-136-25 kubenswrapper[2569]: I0706 13:54:17.428514 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/1c64c06c-4a9b-4767-bdd8-cc5fb5bed504-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"1c64c06c-4a9b-4767-bdd8-cc5fb5bed504\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 06 13:54:17.429032 ip-10-0-136-25 kubenswrapper[2569]: I0706 13:54:17.429004 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/1c64c06c-4a9b-4767-bdd8-cc5fb5bed504-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"1c64c06c-4a9b-4767-bdd8-cc5fb5bed504\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 06 13:54:17.429636 ip-10-0-136-25 kubenswrapper[2569]: I0706 13:54:17.429613 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/secret/1c64c06c-4a9b-4767-bdd8-cc5fb5bed504-config\") pod \"prometheus-k8s-0\" (UID: \"1c64c06c-4a9b-4767-bdd8-cc5fb5bed504\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 06 13:54:17.430027 ip-10-0-136-25 kubenswrapper[2569]: I0706 13:54:17.430006 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/1c64c06c-4a9b-4767-bdd8-cc5fb5bed504-web-config\") pod \"prometheus-k8s-0\" (UID: \"1c64c06c-4a9b-4767-bdd8-cc5fb5bed504\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 06 13:54:17.430223 ip-10-0-136-25 kubenswrapper[2569]: I0706 13:54:17.430189 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/1c64c06c-4a9b-4767-bdd8-cc5fb5bed504-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"1c64c06c-4a9b-4767-bdd8-cc5fb5bed504\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 06 13:54:17.430474 ip-10-0-136-25 kubenswrapper[2569]: I0706 13:54:17.430459 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/1c64c06c-4a9b-4767-bdd8-cc5fb5bed504-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"1c64c06c-4a9b-4767-bdd8-cc5fb5bed504\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 06 13:54:17.430831 ip-10-0-136-25 kubenswrapper[2569]: I0706 13:54:17.430815 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/1c64c06c-4a9b-4767-bdd8-cc5fb5bed504-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"1c64c06c-4a9b-4767-bdd8-cc5fb5bed504\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 06 13:54:17.436744 ip-10-0-136-25 kubenswrapper[2569]: I0706 13:54:17.436724 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-l8kkj\" (UniqueName: \"kubernetes.io/projected/1c64c06c-4a9b-4767-bdd8-cc5fb5bed504-kube-api-access-l8kkj\") pod \"prometheus-k8s-0\" (UID: \"1c64c06c-4a9b-4767-bdd8-cc5fb5bed504\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 06 13:54:17.583641 ip-10-0-136-25 kubenswrapper[2569]: I0706 13:54:17.583544 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0" Jul 06 13:54:17.750539 ip-10-0-136-25 kubenswrapper[2569]: I0706 13:54:17.750509 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Jul 06 13:54:17.753837 ip-10-0-136-25 kubenswrapper[2569]: W0706 13:54:17.753802 2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1c64c06c_4a9b_4767_bdd8_cc5fb5bed504.slice/crio-2cfdf2fc35da47903f565b300bc468b2d4ade908365199e8dcb05827d88b978e WatchSource:0}: Error finding container 2cfdf2fc35da47903f565b300bc468b2d4ade908365199e8dcb05827d88b978e: Status 404 returned error can't find the container with id 2cfdf2fc35da47903f565b300bc468b2d4ade908365199e8dcb05827d88b978e Jul 06 13:54:17.920433 ip-10-0-136-25 kubenswrapper[2569]: I0706 13:54:17.920396 2569 generic.go:356] "Generic (PLEG): container finished" podID="1c64c06c-4a9b-4767-bdd8-cc5fb5bed504" containerID="f15e1426f68de884066fd9f762985778077887adee76aa9f13c07eebb6464950" exitCode=0 Jul 06 13:54:17.920875 ip-10-0-136-25 kubenswrapper[2569]: I0706 13:54:17.920480 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"1c64c06c-4a9b-4767-bdd8-cc5fb5bed504","Type":"ContainerDied","Data":"f15e1426f68de884066fd9f762985778077887adee76aa9f13c07eebb6464950"} Jul 06 13:54:17.920875 ip-10-0-136-25 kubenswrapper[2569]: I0706 13:54:17.920553 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"1c64c06c-4a9b-4767-bdd8-cc5fb5bed504","Type":"ContainerStarted","Data":"2cfdf2fc35da47903f565b300bc468b2d4ade908365199e8dcb05827d88b978e"} Jul 06 13:54:18.929873 ip-10-0-136-25 kubenswrapper[2569]: I0706 13:54:18.929777 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"1c64c06c-4a9b-4767-bdd8-cc5fb5bed504","Type":"ContainerStarted","Data":"2d4d02056e5b3079cb2881d8eed1aa2ce7dca6a89d0c36aa161861a8fe9d5879"} Jul 06 13:54:18.929873 ip-10-0-136-25 kubenswrapper[2569]: I0706 13:54:18.929840 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"1c64c06c-4a9b-4767-bdd8-cc5fb5bed504","Type":"ContainerStarted","Data":"b5610009100e295da2e0eeabcf3d88b73a55e0b730e91826a860f3883045bd09"} Jul 06 13:54:18.929873 ip-10-0-136-25 kubenswrapper[2569]: I0706 13:54:18.929868 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"1c64c06c-4a9b-4767-bdd8-cc5fb5bed504","Type":"ContainerStarted","Data":"9d6b105391d16b478e972361ab3933329266e1164afb1426b41377182ad060ff"} Jul 06 13:54:18.930457 ip-10-0-136-25 kubenswrapper[2569]: I0706 13:54:18.929893 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"1c64c06c-4a9b-4767-bdd8-cc5fb5bed504","Type":"ContainerStarted","Data":"889078b5df45b62b220f2972f0b2335dc03ef6828741f037d4cc5d449b8e5927"} Jul 06 13:54:18.930457 ip-10-0-136-25 kubenswrapper[2569]: I0706 13:54:18.929915 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"1c64c06c-4a9b-4767-bdd8-cc5fb5bed504","Type":"ContainerStarted","Data":"b1894ca047861870cec564e9f6152340a0a57fd9af7e843efd387b9ab6f6d69a"} Jul 06 13:54:18.930457 ip-10-0-136-25 kubenswrapper[2569]: I0706 13:54:18.929938 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"1c64c06c-4a9b-4767-bdd8-cc5fb5bed504","Type":"ContainerStarted","Data":"a397b355a363c1f7ea06cd1fcae68122d9d732fbb733a36cb39779edd1b59206"} Jul 06 13:54:18.939311 ip-10-0-136-25 kubenswrapper[2569]: I0706 13:54:18.939285 2569 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="b958bf61-a426-4f88-8a79-318280126542" path="/var/lib/kubelet/pods/b958bf61-a426-4f88-8a79-318280126542/volumes" Jul 06 13:54:18.964009 ip-10-0-136-25 kubenswrapper[2569]: I0706 13:54:18.963967 2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/prometheus-k8s-0" podStartSLOduration=1.946527028 podStartE2EDuration="1.963936103s" totalImagesPullingTime="0s" totalInitContainerRuntime="17.409075ms" isStatefulPod=false podCreationTimestamp="2026-07-06 13:54:17 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 13:54:18.962248951 +0000 UTC m=+234.552810729" watchObservedRunningTime="2026-07-06 13:54:18.963936103 +0000 UTC m=+234.554497883" Jul 06 13:54:22.584141 ip-10-0-136-25 kubenswrapper[2569]: I0706 13:54:22.584101 2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/prometheus-k8s-0" Jul 06 13:55:17.584576 ip-10-0-136-25 kubenswrapper[2569]: I0706 13:55:17.584524 2569 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-monitoring/prometheus-k8s-0" Jul 06 13:55:17.600150 ip-10-0-136-25 kubenswrapper[2569]: I0706 13:55:17.600126 2569 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-monitoring/prometheus-k8s-0" Jul 06 13:55:18.181765 ip-10-0-136-25 kubenswrapper[2569]: I0706 13:55:18.181740 2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/prometheus-k8s-0" Jul 06 13:55:24.806813 ip-10-0-136-25 kubenswrapper[2569]: I0706 13:55:24.806786 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-68b757fb67-z2h95_dfffcdca-94d6-4732-815f-bcb533f538e9/console-operator/2.log" Jul 06 13:55:24.812316 ip-10-0-136-25 kubenswrapper[2569]: I0706 13:55:24.812286 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-68b757fb67-z2h95_dfffcdca-94d6-4732-815f-bcb533f538e9/console-operator/2.log" Jul 06 13:55:24.818425 ip-10-0-136-25 kubenswrapper[2569]: I0706 13:55:24.818393 2569 kubelet.go:1648] "Image garbage collection succeeded" Jul 06 13:57:37.818694 ip-10-0-136-25 kubenswrapper[2569]: I0706 13:57:37.818660 2569 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-lws-operator/lws-controller-manager-6c95bc9d56-t4784"] Jul 06 13:57:37.821032 ip-10-0-136-25 kubenswrapper[2569]: I0706 13:57:37.821010 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/lws-controller-manager-6c95bc9d56-t4784" Jul 06 13:57:37.823069 ip-10-0-136-25 kubenswrapper[2569]: I0706 13:57:37.823049 2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"lws-controller-manager-dockercfg-5pfjr\"" Jul 06 13:57:37.823615 ip-10-0-136-25 kubenswrapper[2569]: I0706 13:57:37.823536 2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"metrics-server-cert\"" Jul 06 13:57:37.823770 ip-10-0-136-25 kubenswrapper[2569]: I0706 13:57:37.823622 2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"webhook-server-cert\"" Jul 06 13:57:37.823770 ip-10-0-136-25 kubenswrapper[2569]: I0706 13:57:37.823635 2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"lws-manager-config\"" Jul 06 13:57:37.823883 ip-10-0-136-25 kubenswrapper[2569]: I0706 13:57:37.823865 2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"kube-root-ca.crt\"" Jul 06 13:57:37.824017 ip-10-0-136-25 kubenswrapper[2569]: I0706 13:57:37.823997 2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"openshift-service-ca.crt\"" Jul 06 13:57:37.842717 ip-10-0-136-25 kubenswrapper[2569]: I0706 13:57:37.842692 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/lws-controller-manager-6c95bc9d56-t4784"] Jul 06 13:57:37.957362 ip-10-0-136-25 kubenswrapper[2569]: I0706 13:57:37.957332 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/f9633d41-40e9-4d38-8264-926db6a773c3-manager-config\") pod \"lws-controller-manager-6c95bc9d56-t4784\" (UID: \"f9633d41-40e9-4d38-8264-926db6a773c3\") " pod="openshift-lws-operator/lws-controller-manager-6c95bc9d56-t4784" Jul 06 13:57:37.957362 ip-10-0-136-25 kubenswrapper[2569]: I0706 13:57:37.957363 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/f9633d41-40e9-4d38-8264-926db6a773c3-cert\") pod \"lws-controller-manager-6c95bc9d56-t4784\" (UID: \"f9633d41-40e9-4d38-8264-926db6a773c3\") " pod="openshift-lws-operator/lws-controller-manager-6c95bc9d56-t4784" Jul 06 13:57:37.957581 ip-10-0-136-25 kubenswrapper[2569]: I0706 13:57:37.957397 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/f9633d41-40e9-4d38-8264-926db6a773c3-metrics-cert\") pod \"lws-controller-manager-6c95bc9d56-t4784\" (UID: \"f9633d41-40e9-4d38-8264-926db6a773c3\") " pod="openshift-lws-operator/lws-controller-manager-6c95bc9d56-t4784" Jul 06 13:57:37.957581 ip-10-0-136-25 kubenswrapper[2569]: I0706 13:57:37.957474 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7kmnn\" (UniqueName: \"kubernetes.io/projected/f9633d41-40e9-4d38-8264-926db6a773c3-kube-api-access-7kmnn\") pod \"lws-controller-manager-6c95bc9d56-t4784\" (UID: \"f9633d41-40e9-4d38-8264-926db6a773c3\") " pod="openshift-lws-operator/lws-controller-manager-6c95bc9d56-t4784" Jul 06 13:57:38.058588 ip-10-0-136-25 kubenswrapper[2569]: I0706 13:57:38.058550 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/f9633d41-40e9-4d38-8264-926db6a773c3-metrics-cert\") pod \"lws-controller-manager-6c95bc9d56-t4784\" (UID: \"f9633d41-40e9-4d38-8264-926db6a773c3\") " pod="openshift-lws-operator/lws-controller-manager-6c95bc9d56-t4784" Jul 06 13:57:38.058744 ip-10-0-136-25 kubenswrapper[2569]: I0706 13:57:38.058602 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7kmnn\" (UniqueName: \"kubernetes.io/projected/f9633d41-40e9-4d38-8264-926db6a773c3-kube-api-access-7kmnn\") pod \"lws-controller-manager-6c95bc9d56-t4784\" (UID: \"f9633d41-40e9-4d38-8264-926db6a773c3\") " pod="openshift-lws-operator/lws-controller-manager-6c95bc9d56-t4784" Jul 06 13:57:38.058744 ip-10-0-136-25 kubenswrapper[2569]: I0706 13:57:38.058674 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/f9633d41-40e9-4d38-8264-926db6a773c3-manager-config\") pod \"lws-controller-manager-6c95bc9d56-t4784\" (UID: \"f9633d41-40e9-4d38-8264-926db6a773c3\") " pod="openshift-lws-operator/lws-controller-manager-6c95bc9d56-t4784" Jul 06 13:57:38.058744 ip-10-0-136-25 kubenswrapper[2569]: I0706 13:57:38.058703 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/f9633d41-40e9-4d38-8264-926db6a773c3-cert\") pod \"lws-controller-manager-6c95bc9d56-t4784\" (UID: \"f9633d41-40e9-4d38-8264-926db6a773c3\") " pod="openshift-lws-operator/lws-controller-manager-6c95bc9d56-t4784" Jul 06 13:57:38.059406 ip-10-0-136-25 kubenswrapper[2569]: I0706 13:57:38.059386 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/f9633d41-40e9-4d38-8264-926db6a773c3-manager-config\") pod \"lws-controller-manager-6c95bc9d56-t4784\" (UID: \"f9633d41-40e9-4d38-8264-926db6a773c3\") " pod="openshift-lws-operator/lws-controller-manager-6c95bc9d56-t4784" Jul 06 13:57:38.061144 ip-10-0-136-25 kubenswrapper[2569]: I0706 13:57:38.061121 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/f9633d41-40e9-4d38-8264-926db6a773c3-cert\") pod \"lws-controller-manager-6c95bc9d56-t4784\" (UID: \"f9633d41-40e9-4d38-8264-926db6a773c3\") " pod="openshift-lws-operator/lws-controller-manager-6c95bc9d56-t4784" Jul 06 13:57:38.061279 ip-10-0-136-25 kubenswrapper[2569]: I0706 13:57:38.061252 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/f9633d41-40e9-4d38-8264-926db6a773c3-metrics-cert\") pod \"lws-controller-manager-6c95bc9d56-t4784\" (UID: \"f9633d41-40e9-4d38-8264-926db6a773c3\") " pod="openshift-lws-operator/lws-controller-manager-6c95bc9d56-t4784" Jul 06 13:57:38.072576 ip-10-0-136-25 kubenswrapper[2569]: I0706 13:57:38.072527 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7kmnn\" (UniqueName: \"kubernetes.io/projected/f9633d41-40e9-4d38-8264-926db6a773c3-kube-api-access-7kmnn\") pod \"lws-controller-manager-6c95bc9d56-t4784\" (UID: \"f9633d41-40e9-4d38-8264-926db6a773c3\") " pod="openshift-lws-operator/lws-controller-manager-6c95bc9d56-t4784" Jul 06 13:57:38.130515 ip-10-0-136-25 kubenswrapper[2569]: I0706 13:57:38.130492 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/lws-controller-manager-6c95bc9d56-t4784" Jul 06 13:57:38.257456 ip-10-0-136-25 kubenswrapper[2569]: I0706 13:57:38.257417 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/lws-controller-manager-6c95bc9d56-t4784"] Jul 06 13:57:38.260537 ip-10-0-136-25 kubenswrapper[2569]: W0706 13:57:38.260515 2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf9633d41_40e9_4d38_8264_926db6a773c3.slice/crio-0479aa1f6545e4100706ee7605f59697a34e5101dde8045598b1962b373df171 WatchSource:0}: Error finding container 0479aa1f6545e4100706ee7605f59697a34e5101dde8045598b1962b373df171: Status 404 returned error can't find the container with id 0479aa1f6545e4100706ee7605f59697a34e5101dde8045598b1962b373df171 Jul 06 13:57:38.262353 ip-10-0-136-25 kubenswrapper[2569]: I0706 13:57:38.262336 2569 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Jul 06 13:57:38.736879 ip-10-0-136-25 kubenswrapper[2569]: I0706 13:57:38.736841 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/lws-controller-manager-6c95bc9d56-t4784" event={"ID":"f9633d41-40e9-4d38-8264-926db6a773c3","Type":"ContainerStarted","Data":"0479aa1f6545e4100706ee7605f59697a34e5101dde8045598b1962b373df171"} Jul 06 13:57:40.559765 ip-10-0-136-25 kubenswrapper[2569]: I0706 13:57:40.559736 2569 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-5f8f457c8b-lvnlj"] Jul 06 13:57:40.562591 ip-10-0-136-25 kubenswrapper[2569]: I0706 13:57:40.562568 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/opendatahub-operator-controller-manager-5f8f457c8b-lvnlj" Jul 06 13:57:40.565007 ip-10-0-136-25 kubenswrapper[2569]: I0706 13:57:40.564975 2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"openshift-service-ca.crt\"" Jul 06 13:57:40.565007 ip-10-0-136-25 kubenswrapper[2569]: I0706 13:57:40.564991 2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-webhook-cert\"" Jul 06 13:57:40.565184 ip-10-0-136-25 kubenswrapper[2569]: I0706 13:57:40.565099 2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-manager-service-cert\"" Jul 06 13:57:40.565184 ip-10-0-136-25 kubenswrapper[2569]: I0706 13:57:40.565139 2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-manager-dockercfg-hd8wp\"" Jul 06 13:57:40.565368 ip-10-0-136-25 kubenswrapper[2569]: I0706 13:57:40.565355 2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"kube-root-ca.crt\"" Jul 06 13:57:40.578542 ip-10-0-136-25 kubenswrapper[2569]: I0706 13:57:40.578459 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-5f8f457c8b-lvnlj"] Jul 06 13:57:40.683513 ip-10-0-136-25 kubenswrapper[2569]: I0706 13:57:40.683481 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/487646d3-ef03-4bcf-acf3-7f7051ff672f-webhook-cert\") pod \"opendatahub-operator-controller-manager-5f8f457c8b-lvnlj\" (UID: \"487646d3-ef03-4bcf-acf3-7f7051ff672f\") " pod="opendatahub/opendatahub-operator-controller-manager-5f8f457c8b-lvnlj" Jul 06 13:57:40.683711 ip-10-0-136-25 kubenswrapper[2569]: I0706 13:57:40.683527 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/487646d3-ef03-4bcf-acf3-7f7051ff672f-apiservice-cert\") pod \"opendatahub-operator-controller-manager-5f8f457c8b-lvnlj\" (UID: \"487646d3-ef03-4bcf-acf3-7f7051ff672f\") " pod="opendatahub/opendatahub-operator-controller-manager-5f8f457c8b-lvnlj" Jul 06 13:57:40.683711 ip-10-0-136-25 kubenswrapper[2569]: I0706 13:57:40.683557 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/487646d3-ef03-4bcf-acf3-7f7051ff672f-manifests\") pod \"opendatahub-operator-controller-manager-5f8f457c8b-lvnlj\" (UID: \"487646d3-ef03-4bcf-acf3-7f7051ff672f\") " pod="opendatahub/opendatahub-operator-controller-manager-5f8f457c8b-lvnlj" Jul 06 13:57:40.683711 ip-10-0-136-25 kubenswrapper[2569]: I0706 13:57:40.683660 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qnpf8\" (UniqueName: \"kubernetes.io/projected/487646d3-ef03-4bcf-acf3-7f7051ff672f-kube-api-access-qnpf8\") pod \"opendatahub-operator-controller-manager-5f8f457c8b-lvnlj\" (UID: \"487646d3-ef03-4bcf-acf3-7f7051ff672f\") " pod="opendatahub/opendatahub-operator-controller-manager-5f8f457c8b-lvnlj" Jul 06 13:57:40.683854 ip-10-0-136-25 kubenswrapper[2569]: I0706 13:57:40.683710 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/487646d3-ef03-4bcf-acf3-7f7051ff672f-tmp\") pod \"opendatahub-operator-controller-manager-5f8f457c8b-lvnlj\" (UID: \"487646d3-ef03-4bcf-acf3-7f7051ff672f\") " pod="opendatahub/opendatahub-operator-controller-manager-5f8f457c8b-lvnlj" Jul 06 13:57:40.784852 ip-10-0-136-25 kubenswrapper[2569]: I0706 13:57:40.784817 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qnpf8\" (UniqueName: \"kubernetes.io/projected/487646d3-ef03-4bcf-acf3-7f7051ff672f-kube-api-access-qnpf8\") pod \"opendatahub-operator-controller-manager-5f8f457c8b-lvnlj\" (UID: \"487646d3-ef03-4bcf-acf3-7f7051ff672f\") " pod="opendatahub/opendatahub-operator-controller-manager-5f8f457c8b-lvnlj" Jul 06 13:57:40.785069 ip-10-0-136-25 kubenswrapper[2569]: I0706 13:57:40.784878 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/487646d3-ef03-4bcf-acf3-7f7051ff672f-tmp\") pod \"opendatahub-operator-controller-manager-5f8f457c8b-lvnlj\" (UID: \"487646d3-ef03-4bcf-acf3-7f7051ff672f\") " pod="opendatahub/opendatahub-operator-controller-manager-5f8f457c8b-lvnlj" Jul 06 13:57:40.785069 ip-10-0-136-25 kubenswrapper[2569]: I0706 13:57:40.784965 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/487646d3-ef03-4bcf-acf3-7f7051ff672f-webhook-cert\") pod \"opendatahub-operator-controller-manager-5f8f457c8b-lvnlj\" (UID: \"487646d3-ef03-4bcf-acf3-7f7051ff672f\") " pod="opendatahub/opendatahub-operator-controller-manager-5f8f457c8b-lvnlj" Jul 06 13:57:40.785069 ip-10-0-136-25 kubenswrapper[2569]: I0706 13:57:40.784993 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/487646d3-ef03-4bcf-acf3-7f7051ff672f-apiservice-cert\") pod \"opendatahub-operator-controller-manager-5f8f457c8b-lvnlj\" (UID: \"487646d3-ef03-4bcf-acf3-7f7051ff672f\") " pod="opendatahub/opendatahub-operator-controller-manager-5f8f457c8b-lvnlj" Jul 06 13:57:40.785069 ip-10-0-136-25 kubenswrapper[2569]: I0706 13:57:40.785021 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/487646d3-ef03-4bcf-acf3-7f7051ff672f-manifests\") pod \"opendatahub-operator-controller-manager-5f8f457c8b-lvnlj\" (UID: \"487646d3-ef03-4bcf-acf3-7f7051ff672f\") " pod="opendatahub/opendatahub-operator-controller-manager-5f8f457c8b-lvnlj" Jul 06 13:57:40.785454 ip-10-0-136-25 kubenswrapper[2569]: I0706 13:57:40.785427 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/487646d3-ef03-4bcf-acf3-7f7051ff672f-manifests\") pod \"opendatahub-operator-controller-manager-5f8f457c8b-lvnlj\" (UID: \"487646d3-ef03-4bcf-acf3-7f7051ff672f\") " pod="opendatahub/opendatahub-operator-controller-manager-5f8f457c8b-lvnlj" Jul 06 13:57:40.787702 ip-10-0-136-25 kubenswrapper[2569]: I0706 13:57:40.787637 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/487646d3-ef03-4bcf-acf3-7f7051ff672f-tmp\") pod \"opendatahub-operator-controller-manager-5f8f457c8b-lvnlj\" (UID: \"487646d3-ef03-4bcf-acf3-7f7051ff672f\") " pod="opendatahub/opendatahub-operator-controller-manager-5f8f457c8b-lvnlj" Jul 06 13:57:40.787860 ip-10-0-136-25 kubenswrapper[2569]: I0706 13:57:40.787840 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/487646d3-ef03-4bcf-acf3-7f7051ff672f-webhook-cert\") pod \"opendatahub-operator-controller-manager-5f8f457c8b-lvnlj\" (UID: \"487646d3-ef03-4bcf-acf3-7f7051ff672f\") " pod="opendatahub/opendatahub-operator-controller-manager-5f8f457c8b-lvnlj" Jul 06 13:57:40.787951 ip-10-0-136-25 kubenswrapper[2569]: I0706 13:57:40.787931 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/487646d3-ef03-4bcf-acf3-7f7051ff672f-apiservice-cert\") pod \"opendatahub-operator-controller-manager-5f8f457c8b-lvnlj\" (UID: \"487646d3-ef03-4bcf-acf3-7f7051ff672f\") " pod="opendatahub/opendatahub-operator-controller-manager-5f8f457c8b-lvnlj" Jul 06 13:57:40.792734 ip-10-0-136-25 kubenswrapper[2569]: I0706 13:57:40.792709 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qnpf8\" (UniqueName: \"kubernetes.io/projected/487646d3-ef03-4bcf-acf3-7f7051ff672f-kube-api-access-qnpf8\") pod \"opendatahub-operator-controller-manager-5f8f457c8b-lvnlj\" (UID: \"487646d3-ef03-4bcf-acf3-7f7051ff672f\") " pod="opendatahub/opendatahub-operator-controller-manager-5f8f457c8b-lvnlj" Jul 06 13:57:40.875018 ip-10-0-136-25 kubenswrapper[2569]: I0706 13:57:40.874947 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/opendatahub-operator-controller-manager-5f8f457c8b-lvnlj" Jul 06 13:57:41.027206 ip-10-0-136-25 kubenswrapper[2569]: I0706 13:57:41.027175 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-5f8f457c8b-lvnlj"] Jul 06 13:57:41.030631 ip-10-0-136-25 kubenswrapper[2569]: W0706 13:57:41.030592 2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod487646d3_ef03_4bcf_acf3_7f7051ff672f.slice/crio-679c74a996c6ff64a7bc81a98bad0f4e563ac4e8fe8cf73df93ec6394bce89cd WatchSource:0}: Error finding container 679c74a996c6ff64a7bc81a98bad0f4e563ac4e8fe8cf73df93ec6394bce89cd: Status 404 returned error can't find the container with id 679c74a996c6ff64a7bc81a98bad0f4e563ac4e8fe8cf73df93ec6394bce89cd Jul 06 13:57:41.754247 ip-10-0-136-25 kubenswrapper[2569]: I0706 13:57:41.754190 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-5f8f457c8b-lvnlj" event={"ID":"487646d3-ef03-4bcf-acf3-7f7051ff672f","Type":"ContainerStarted","Data":"679c74a996c6ff64a7bc81a98bad0f4e563ac4e8fe8cf73df93ec6394bce89cd"} Jul 06 13:57:41.755990 ip-10-0-136-25 kubenswrapper[2569]: I0706 13:57:41.755958 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/lws-controller-manager-6c95bc9d56-t4784" event={"ID":"f9633d41-40e9-4d38-8264-926db6a773c3","Type":"ContainerStarted","Data":"07e46fe397277c743e52d8abcdc57bfb84f5fded55e0ad47c154046fc9db58de"} Jul 06 13:57:41.756194 ip-10-0-136-25 kubenswrapper[2569]: I0706 13:57:41.756176 2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-lws-operator/lws-controller-manager-6c95bc9d56-t4784" Jul 06 13:57:41.772346 ip-10-0-136-25 kubenswrapper[2569]: I0706 13:57:41.772295 2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-lws-operator/lws-controller-manager-6c95bc9d56-t4784" podStartSLOduration=2.14461449 podStartE2EDuration="4.772278871s" totalImagesPullingTime="2.627664381s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-06 13:57:37 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 13:57:41.771868255 +0000 UTC m=+437.362430044" watchObservedRunningTime="2026-07-06 13:57:41.772278871 +0000 UTC m=+437.362840651" Jul 06 13:57:44.774584 ip-10-0-136-25 kubenswrapper[2569]: I0706 13:57:44.774542 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-5f8f457c8b-lvnlj" event={"ID":"487646d3-ef03-4bcf-acf3-7f7051ff672f","Type":"ContainerStarted","Data":"8e465d29ed65c9bc0cf1d925a6714eb110457eb5b759c228a8538cae57eed42a"} Jul 06 13:57:45.780584 ip-10-0-136-25 kubenswrapper[2569]: I0706 13:57:45.780548 2569 generic.go:356] "Generic (PLEG): container finished" podID="487646d3-ef03-4bcf-acf3-7f7051ff672f" containerID="8e465d29ed65c9bc0cf1d925a6714eb110457eb5b759c228a8538cae57eed42a" exitCode=0 Jul 06 13:57:45.781066 ip-10-0-136-25 kubenswrapper[2569]: I0706 13:57:45.780608 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-5f8f457c8b-lvnlj" event={"ID":"487646d3-ef03-4bcf-acf3-7f7051ff672f","Type":"ContainerDied","Data":"8e465d29ed65c9bc0cf1d925a6714eb110457eb5b759c228a8538cae57eed42a"} Jul 06 13:57:46.787044 ip-10-0-136-25 kubenswrapper[2569]: I0706 13:57:46.786958 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-5f8f457c8b-lvnlj" event={"ID":"487646d3-ef03-4bcf-acf3-7f7051ff672f","Type":"ContainerStarted","Data":"25ce60b2cc7ca29b8f8604e9ac56a63e0c540cb93a9ef30173198cc6d1c4aa84"} Jul 06 13:57:46.787443 ip-10-0-136-25 kubenswrapper[2569]: I0706 13:57:46.787094 2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/opendatahub-operator-controller-manager-5f8f457c8b-lvnlj" Jul 06 13:57:46.810774 ip-10-0-136-25 kubenswrapper[2569]: I0706 13:57:46.810716 2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/opendatahub-operator-controller-manager-5f8f457c8b-lvnlj" podStartSLOduration=2.44838978 podStartE2EDuration="6.810695843s" totalImagesPullingTime="3.269329075s" totalInitContainerRuntime="1.092976988s" isStatefulPod=false podCreationTimestamp="2026-07-06 13:57:40 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 13:57:46.808535678 +0000 UTC m=+442.399097455" watchObservedRunningTime="2026-07-06 13:57:46.810695843 +0000 UTC m=+442.401257620" Jul 06 13:57:52.765444 ip-10-0-136-25 kubenswrapper[2569]: I0706 13:57:52.765415 2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-lws-operator/lws-controller-manager-6c95bc9d56-t4784" Jul 06 13:57:57.794051 ip-10-0-136-25 kubenswrapper[2569]: I0706 13:57:57.794019 2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/opendatahub-operator-controller-manager-5f8f457c8b-lvnlj" Jul 06 13:58:10.541576 ip-10-0-136-25 kubenswrapper[2569]: I0706 13:58:10.541539 2569 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/kube-auth-proxy-7c6c964b95-sfw6w"] Jul 06 13:58:10.551943 ip-10-0-136-25 kubenswrapper[2569]: I0706 13:58:10.551923 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/kube-auth-proxy-7c6c964b95-sfw6w" Jul 06 13:58:10.554600 ip-10-0-136-25 kubenswrapper[2569]: I0706 13:58:10.554476 2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-creds\"" Jul 06 13:58:10.555272 ip-10-0-136-25 kubenswrapper[2569]: I0706 13:58:10.555249 2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-dockercfg-2nh95\"" Jul 06 13:58:10.555515 ip-10-0-136-25 kubenswrapper[2569]: I0706 13:58:10.555496 2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-tls\"" Jul 06 13:58:10.557160 ip-10-0-136-25 kubenswrapper[2569]: I0706 13:58:10.557138 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/kube-auth-proxy-7c6c964b95-sfw6w"] Jul 06 13:58:10.648082 ip-10-0-136-25 kubenswrapper[2569]: I0706 13:58:10.648048 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/1970cda6-a1e1-4aed-af85-44bf5bfd5dad-tmp\") pod \"kube-auth-proxy-7c6c964b95-sfw6w\" (UID: \"1970cda6-a1e1-4aed-af85-44bf5bfd5dad\") " pod="openshift-ingress/kube-auth-proxy-7c6c964b95-sfw6w" Jul 06 13:58:10.648317 ip-10-0-136-25 kubenswrapper[2569]: I0706 13:58:10.648107 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1970cda6-a1e1-4aed-af85-44bf5bfd5dad-tls-certs\") pod \"kube-auth-proxy-7c6c964b95-sfw6w\" (UID: \"1970cda6-a1e1-4aed-af85-44bf5bfd5dad\") " pod="openshift-ingress/kube-auth-proxy-7c6c964b95-sfw6w" Jul 06 13:58:10.648317 ip-10-0-136-25 kubenswrapper[2569]: I0706 13:58:10.648183 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2rmz4\" (UniqueName: \"kubernetes.io/projected/1970cda6-a1e1-4aed-af85-44bf5bfd5dad-kube-api-access-2rmz4\") pod \"kube-auth-proxy-7c6c964b95-sfw6w\" (UID: \"1970cda6-a1e1-4aed-af85-44bf5bfd5dad\") " pod="openshift-ingress/kube-auth-proxy-7c6c964b95-sfw6w" Jul 06 13:58:10.748768 ip-10-0-136-25 kubenswrapper[2569]: I0706 13:58:10.748734 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/1970cda6-a1e1-4aed-af85-44bf5bfd5dad-tmp\") pod \"kube-auth-proxy-7c6c964b95-sfw6w\" (UID: \"1970cda6-a1e1-4aed-af85-44bf5bfd5dad\") " pod="openshift-ingress/kube-auth-proxy-7c6c964b95-sfw6w" Jul 06 13:58:10.748963 ip-10-0-136-25 kubenswrapper[2569]: I0706 13:58:10.748778 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1970cda6-a1e1-4aed-af85-44bf5bfd5dad-tls-certs\") pod \"kube-auth-proxy-7c6c964b95-sfw6w\" (UID: \"1970cda6-a1e1-4aed-af85-44bf5bfd5dad\") " pod="openshift-ingress/kube-auth-proxy-7c6c964b95-sfw6w" Jul 06 13:58:10.748963 ip-10-0-136-25 kubenswrapper[2569]: I0706 13:58:10.748825 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2rmz4\" (UniqueName: \"kubernetes.io/projected/1970cda6-a1e1-4aed-af85-44bf5bfd5dad-kube-api-access-2rmz4\") pod \"kube-auth-proxy-7c6c964b95-sfw6w\" (UID: \"1970cda6-a1e1-4aed-af85-44bf5bfd5dad\") " pod="openshift-ingress/kube-auth-proxy-7c6c964b95-sfw6w" Jul 06 13:58:10.748963 ip-10-0-136-25 kubenswrapper[2569]: E0706 13:58:10.748954 2569 secret.go:189] Couldn't get secret openshift-ingress/kube-auth-proxy-tls: secret "kube-auth-proxy-tls" not found Jul 06 13:58:10.749133 ip-10-0-136-25 kubenswrapper[2569]: E0706 13:58:10.749015 2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1970cda6-a1e1-4aed-af85-44bf5bfd5dad-tls-certs podName:1970cda6-a1e1-4aed-af85-44bf5bfd5dad nodeName:}" failed. No retries permitted until 2026-07-06 13:58:11.248999602 +0000 UTC m=+466.839561357 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "tls-certs" (UniqueName: "kubernetes.io/secret/1970cda6-a1e1-4aed-af85-44bf5bfd5dad-tls-certs") pod "kube-auth-proxy-7c6c964b95-sfw6w" (UID: "1970cda6-a1e1-4aed-af85-44bf5bfd5dad") : secret "kube-auth-proxy-tls" not found Jul 06 13:58:10.751063 ip-10-0-136-25 kubenswrapper[2569]: I0706 13:58:10.751039 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/1970cda6-a1e1-4aed-af85-44bf5bfd5dad-tmp\") pod \"kube-auth-proxy-7c6c964b95-sfw6w\" (UID: \"1970cda6-a1e1-4aed-af85-44bf5bfd5dad\") " pod="openshift-ingress/kube-auth-proxy-7c6c964b95-sfw6w" Jul 06 13:58:10.756684 ip-10-0-136-25 kubenswrapper[2569]: I0706 13:58:10.756660 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2rmz4\" (UniqueName: \"kubernetes.io/projected/1970cda6-a1e1-4aed-af85-44bf5bfd5dad-kube-api-access-2rmz4\") pod \"kube-auth-proxy-7c6c964b95-sfw6w\" (UID: \"1970cda6-a1e1-4aed-af85-44bf5bfd5dad\") " pod="openshift-ingress/kube-auth-proxy-7c6c964b95-sfw6w" Jul 06 13:58:11.253147 ip-10-0-136-25 kubenswrapper[2569]: I0706 13:58:11.253110 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1970cda6-a1e1-4aed-af85-44bf5bfd5dad-tls-certs\") pod \"kube-auth-proxy-7c6c964b95-sfw6w\" (UID: \"1970cda6-a1e1-4aed-af85-44bf5bfd5dad\") " pod="openshift-ingress/kube-auth-proxy-7c6c964b95-sfw6w" Jul 06 13:58:11.255505 ip-10-0-136-25 kubenswrapper[2569]: I0706 13:58:11.255485 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1970cda6-a1e1-4aed-af85-44bf5bfd5dad-tls-certs\") pod \"kube-auth-proxy-7c6c964b95-sfw6w\" (UID: \"1970cda6-a1e1-4aed-af85-44bf5bfd5dad\") " pod="openshift-ingress/kube-auth-proxy-7c6c964b95-sfw6w" Jul 06 13:58:11.462291 ip-10-0-136-25 kubenswrapper[2569]: I0706 13:58:11.462237 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/kube-auth-proxy-7c6c964b95-sfw6w" Jul 06 13:58:11.592958 ip-10-0-136-25 kubenswrapper[2569]: I0706 13:58:11.592928 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/kube-auth-proxy-7c6c964b95-sfw6w"] Jul 06 13:58:11.595656 ip-10-0-136-25 kubenswrapper[2569]: W0706 13:58:11.595632 2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1970cda6_a1e1_4aed_af85_44bf5bfd5dad.slice/crio-b32d6c3772dbe4ccd427c1ac84b186ca4c2625da578d9882151a3d59988a3f24 WatchSource:0}: Error finding container b32d6c3772dbe4ccd427c1ac84b186ca4c2625da578d9882151a3d59988a3f24: Status 404 returned error can't find the container with id b32d6c3772dbe4ccd427c1ac84b186ca4c2625da578d9882151a3d59988a3f24 Jul 06 13:58:11.899641 ip-10-0-136-25 kubenswrapper[2569]: I0706 13:58:11.899551 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/kube-auth-proxy-7c6c964b95-sfw6w" event={"ID":"1970cda6-a1e1-4aed-af85-44bf5bfd5dad","Type":"ContainerStarted","Data":"b32d6c3772dbe4ccd427c1ac84b186ca4c2625da578d9882151a3d59988a3f24"} Jul 06 13:58:14.914851 ip-10-0-136-25 kubenswrapper[2569]: I0706 13:58:14.914810 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/kube-auth-proxy-7c6c964b95-sfw6w" event={"ID":"1970cda6-a1e1-4aed-af85-44bf5bfd5dad","Type":"ContainerStarted","Data":"f224129d22613624adb95335888e2a9cf3cd053638e6b2bb806b21a8aa0c5f1e"} Jul 06 13:58:14.931379 ip-10-0-136-25 kubenswrapper[2569]: I0706 13:58:14.931284 2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/kube-auth-proxy-7c6c964b95-sfw6w" podStartSLOduration=1.851732357 podStartE2EDuration="4.931269199s" totalImagesPullingTime="3.079536842s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-06 13:58:10 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 13:58:14.930657184 +0000 UTC m=+470.521218962" watchObservedRunningTime="2026-07-06 13:58:14.931269199 +0000 UTC m=+470.521830975" Jul 06 13:59:49.307285 ip-10-0-136-25 kubenswrapper[2569]: I0706 13:59:49.307251 2569 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-console-plugin-66b5774985-gbw6w"] Jul 06 13:59:49.310704 ip-10-0-136-25 kubenswrapper[2569]: I0706 13:59:49.310687 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-console-plugin-66b5774985-gbw6w" Jul 06 13:59:49.312947 ip-10-0-136-25 kubenswrapper[2569]: I0706 13:59:49.312911 2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"openshift-service-ca.crt\"" Jul 06 13:59:49.313092 ip-10-0-136-25 kubenswrapper[2569]: I0706 13:59:49.312957 2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"plugin-serving-cert\"" Jul 06 13:59:49.313092 ip-10-0-136-25 kubenswrapper[2569]: I0706 13:59:49.312995 2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kuadrant-console-nginx-conf\"" Jul 06 13:59:49.313092 ip-10-0-136-25 kubenswrapper[2569]: I0706 13:59:49.313009 2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kube-root-ca.crt\"" Jul 06 13:59:49.313349 ip-10-0-136-25 kubenswrapper[2569]: I0706 13:59:49.313337 2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"default-dockercfg-9rxxc\"" Jul 06 13:59:49.321175 ip-10-0-136-25 kubenswrapper[2569]: I0706 13:59:49.321152 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-console-plugin-66b5774985-gbw6w"] Jul 06 13:59:49.419112 ip-10-0-136-25 kubenswrapper[2569]: I0706 13:59:49.419074 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vwpcs\" (UniqueName: \"kubernetes.io/projected/2e801db1-650f-4f52-8ce4-37ce2381ec24-kube-api-access-vwpcs\") pod \"kuadrant-console-plugin-66b5774985-gbw6w\" (UID: \"2e801db1-650f-4f52-8ce4-37ce2381ec24\") " pod="kuadrant-system/kuadrant-console-plugin-66b5774985-gbw6w" Jul 06 13:59:49.419306 ip-10-0-136-25 kubenswrapper[2569]: I0706 13:59:49.419131 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/2e801db1-650f-4f52-8ce4-37ce2381ec24-nginx-conf\") pod \"kuadrant-console-plugin-66b5774985-gbw6w\" (UID: \"2e801db1-650f-4f52-8ce4-37ce2381ec24\") " pod="kuadrant-system/kuadrant-console-plugin-66b5774985-gbw6w" Jul 06 13:59:49.419306 ip-10-0-136-25 kubenswrapper[2569]: I0706 13:59:49.419270 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugin-serving-cert\" (UniqueName: \"kubernetes.io/secret/2e801db1-650f-4f52-8ce4-37ce2381ec24-plugin-serving-cert\") pod \"kuadrant-console-plugin-66b5774985-gbw6w\" (UID: \"2e801db1-650f-4f52-8ce4-37ce2381ec24\") " pod="kuadrant-system/kuadrant-console-plugin-66b5774985-gbw6w" Jul 06 13:59:49.520004 ip-10-0-136-25 kubenswrapper[2569]: I0706 13:59:49.519972 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugin-serving-cert\" (UniqueName: \"kubernetes.io/secret/2e801db1-650f-4f52-8ce4-37ce2381ec24-plugin-serving-cert\") pod \"kuadrant-console-plugin-66b5774985-gbw6w\" (UID: \"2e801db1-650f-4f52-8ce4-37ce2381ec24\") " pod="kuadrant-system/kuadrant-console-plugin-66b5774985-gbw6w" Jul 06 13:59:49.520171 ip-10-0-136-25 kubenswrapper[2569]: I0706 13:59:49.520017 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vwpcs\" (UniqueName: \"kubernetes.io/projected/2e801db1-650f-4f52-8ce4-37ce2381ec24-kube-api-access-vwpcs\") pod \"kuadrant-console-plugin-66b5774985-gbw6w\" (UID: \"2e801db1-650f-4f52-8ce4-37ce2381ec24\") " pod="kuadrant-system/kuadrant-console-plugin-66b5774985-gbw6w" Jul 06 13:59:49.520171 ip-10-0-136-25 kubenswrapper[2569]: I0706 13:59:49.520067 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/2e801db1-650f-4f52-8ce4-37ce2381ec24-nginx-conf\") pod \"kuadrant-console-plugin-66b5774985-gbw6w\" (UID: \"2e801db1-650f-4f52-8ce4-37ce2381ec24\") " pod="kuadrant-system/kuadrant-console-plugin-66b5774985-gbw6w" Jul 06 13:59:49.520784 ip-10-0-136-25 kubenswrapper[2569]: I0706 13:59:49.520753 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/2e801db1-650f-4f52-8ce4-37ce2381ec24-nginx-conf\") pod \"kuadrant-console-plugin-66b5774985-gbw6w\" (UID: \"2e801db1-650f-4f52-8ce4-37ce2381ec24\") " pod="kuadrant-system/kuadrant-console-plugin-66b5774985-gbw6w" Jul 06 13:59:49.522524 ip-10-0-136-25 kubenswrapper[2569]: I0706 13:59:49.522500 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugin-serving-cert\" (UniqueName: \"kubernetes.io/secret/2e801db1-650f-4f52-8ce4-37ce2381ec24-plugin-serving-cert\") pod \"kuadrant-console-plugin-66b5774985-gbw6w\" (UID: \"2e801db1-650f-4f52-8ce4-37ce2381ec24\") " pod="kuadrant-system/kuadrant-console-plugin-66b5774985-gbw6w" Jul 06 13:59:49.527655 ip-10-0-136-25 kubenswrapper[2569]: I0706 13:59:49.527634 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vwpcs\" (UniqueName: \"kubernetes.io/projected/2e801db1-650f-4f52-8ce4-37ce2381ec24-kube-api-access-vwpcs\") pod \"kuadrant-console-plugin-66b5774985-gbw6w\" (UID: \"2e801db1-650f-4f52-8ce4-37ce2381ec24\") " pod="kuadrant-system/kuadrant-console-plugin-66b5774985-gbw6w" Jul 06 13:59:49.634428 ip-10-0-136-25 kubenswrapper[2569]: I0706 13:59:49.634343 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-console-plugin-66b5774985-gbw6w" Jul 06 13:59:49.768011 ip-10-0-136-25 kubenswrapper[2569]: I0706 13:59:49.767981 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-console-plugin-66b5774985-gbw6w"] Jul 06 13:59:49.770989 ip-10-0-136-25 kubenswrapper[2569]: W0706 13:59:49.770962 2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod2e801db1_650f_4f52_8ce4_37ce2381ec24.slice/crio-cd12ae742aa22085a4092257cf7ef7537b00235ccce5eb1bd76f3c856188c610 WatchSource:0}: Error finding container cd12ae742aa22085a4092257cf7ef7537b00235ccce5eb1bd76f3c856188c610: Status 404 returned error can't find the container with id cd12ae742aa22085a4092257cf7ef7537b00235ccce5eb1bd76f3c856188c610 Jul 06 13:59:50.339266 ip-10-0-136-25 kubenswrapper[2569]: I0706 13:59:50.339206 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-console-plugin-66b5774985-gbw6w" event={"ID":"2e801db1-650f-4f52-8ce4-37ce2381ec24","Type":"ContainerStarted","Data":"cd12ae742aa22085a4092257cf7ef7537b00235ccce5eb1bd76f3c856188c610"} Jul 06 14:00:13.467914 ip-10-0-136-25 kubenswrapper[2569]: I0706 14:00:13.467873 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-console-plugin-66b5774985-gbw6w" event={"ID":"2e801db1-650f-4f52-8ce4-37ce2381ec24","Type":"ContainerStarted","Data":"228c46593b5395203a85c3b50c544b6f0d24f7e27e228e14f8e3cbb7994927ac"} Jul 06 14:00:13.488393 ip-10-0-136-25 kubenswrapper[2569]: I0706 14:00:13.488333 2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-console-plugin-66b5774985-gbw6w" podStartSLOduration=1.8035812949999999 podStartE2EDuration="24.488318282s" totalImagesPullingTime="22.684736987s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-06 13:59:49 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 14:00:13.486274113 +0000 UTC m=+589.076835903" watchObservedRunningTime="2026-07-06 14:00:13.488318282 +0000 UTC m=+589.078880137" Jul 06 14:00:24.839288 ip-10-0-136-25 kubenswrapper[2569]: I0706 14:00:24.839262 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-68b757fb67-z2h95_dfffcdca-94d6-4732-815f-bcb533f538e9/console-operator/2.log" Jul 06 14:00:24.841281 ip-10-0-136-25 kubenswrapper[2569]: I0706 14:00:24.841262 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-68b757fb67-z2h95_dfffcdca-94d6-4732-815f-bcb533f538e9/console-operator/2.log" Jul 06 14:00:41.659929 ip-10-0-136-25 kubenswrapper[2569]: I0706 14:00:41.659892 2569 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-g9qp6"] Jul 06 14:00:41.784161 ip-10-0-136-25 kubenswrapper[2569]: I0706 14:00:41.784126 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-g9qp6"] Jul 06 14:00:41.784161 ip-10-0-136-25 kubenswrapper[2569]: I0706 14:00:41.784168 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-g9qp6"] Jul 06 14:00:41.784374 ip-10-0-136-25 kubenswrapper[2569]: I0706 14:00:41.784259 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-54d6d988d5-g9qp6" Jul 06 14:00:41.786498 ip-10-0-136-25 kubenswrapper[2569]: I0706 14:00:41.786480 2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"limitador-limits-config-limitador\"" Jul 06 14:00:41.911767 ip-10-0-136-25 kubenswrapper[2569]: I0706 14:00:41.911676 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dn8tq\" (UniqueName: \"kubernetes.io/projected/c296255d-ef13-4904-8569-02e8dc6d3da0-kube-api-access-dn8tq\") pod \"limitador-limitador-54d6d988d5-g9qp6\" (UID: \"c296255d-ef13-4904-8569-02e8dc6d3da0\") " pod="kuadrant-system/limitador-limitador-54d6d988d5-g9qp6" Jul 06 14:00:41.911767 ip-10-0-136-25 kubenswrapper[2569]: I0706 14:00:41.911719 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/c296255d-ef13-4904-8569-02e8dc6d3da0-config-file\") pod \"limitador-limitador-54d6d988d5-g9qp6\" (UID: \"c296255d-ef13-4904-8569-02e8dc6d3da0\") " pod="kuadrant-system/limitador-limitador-54d6d988d5-g9qp6" Jul 06 14:00:42.013138 ip-10-0-136-25 kubenswrapper[2569]: I0706 14:00:42.013097 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dn8tq\" (UniqueName: \"kubernetes.io/projected/c296255d-ef13-4904-8569-02e8dc6d3da0-kube-api-access-dn8tq\") pod \"limitador-limitador-54d6d988d5-g9qp6\" (UID: \"c296255d-ef13-4904-8569-02e8dc6d3da0\") " pod="kuadrant-system/limitador-limitador-54d6d988d5-g9qp6" Jul 06 14:00:42.013138 ip-10-0-136-25 kubenswrapper[2569]: I0706 14:00:42.013138 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/c296255d-ef13-4904-8569-02e8dc6d3da0-config-file\") pod \"limitador-limitador-54d6d988d5-g9qp6\" (UID: \"c296255d-ef13-4904-8569-02e8dc6d3da0\") " pod="kuadrant-system/limitador-limitador-54d6d988d5-g9qp6" Jul 06 14:00:42.013946 ip-10-0-136-25 kubenswrapper[2569]: I0706 14:00:42.013909 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/c296255d-ef13-4904-8569-02e8dc6d3da0-config-file\") pod \"limitador-limitador-54d6d988d5-g9qp6\" (UID: \"c296255d-ef13-4904-8569-02e8dc6d3da0\") " pod="kuadrant-system/limitador-limitador-54d6d988d5-g9qp6" Jul 06 14:00:42.021055 ip-10-0-136-25 kubenswrapper[2569]: I0706 14:00:42.021035 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dn8tq\" (UniqueName: \"kubernetes.io/projected/c296255d-ef13-4904-8569-02e8dc6d3da0-kube-api-access-dn8tq\") pod \"limitador-limitador-54d6d988d5-g9qp6\" (UID: \"c296255d-ef13-4904-8569-02e8dc6d3da0\") " pod="kuadrant-system/limitador-limitador-54d6d988d5-g9qp6" Jul 06 14:00:42.095074 ip-10-0-136-25 kubenswrapper[2569]: I0706 14:00:42.095040 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-54d6d988d5-g9qp6" Jul 06 14:00:42.245611 ip-10-0-136-25 kubenswrapper[2569]: I0706 14:00:42.245560 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-g9qp6"] Jul 06 14:00:42.247972 ip-10-0-136-25 kubenswrapper[2569]: W0706 14:00:42.247945 2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc296255d_ef13_4904_8569_02e8dc6d3da0.slice/crio-323262b3f3f3d60d32826b04b0245dccabb62b5f6b74c453d178a4260d705468 WatchSource:0}: Error finding container 323262b3f3f3d60d32826b04b0245dccabb62b5f6b74c453d178a4260d705468: Status 404 returned error can't find the container with id 323262b3f3f3d60d32826b04b0245dccabb62b5f6b74c453d178a4260d705468 Jul 06 14:00:42.609746 ip-10-0-136-25 kubenswrapper[2569]: I0706 14:00:42.609709 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-54d6d988d5-g9qp6" event={"ID":"c296255d-ef13-4904-8569-02e8dc6d3da0","Type":"ContainerStarted","Data":"323262b3f3f3d60d32826b04b0245dccabb62b5f6b74c453d178a4260d705468"} Jul 06 14:00:45.624775 ip-10-0-136-25 kubenswrapper[2569]: I0706 14:00:45.624733 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-54d6d988d5-g9qp6" event={"ID":"c296255d-ef13-4904-8569-02e8dc6d3da0","Type":"ContainerStarted","Data":"7605dd57bc39b9ebd1c7ef1a1f6ac61c52f15012947013113e727534d71d2b27"} Jul 06 14:00:45.625197 ip-10-0-136-25 kubenswrapper[2569]: I0706 14:00:45.624907 2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/limitador-limitador-54d6d988d5-g9qp6" Jul 06 14:00:45.641987 ip-10-0-136-25 kubenswrapper[2569]: I0706 14:00:45.641937 2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/limitador-limitador-54d6d988d5-g9qp6" podStartSLOduration=1.8650617710000001 podStartE2EDuration="4.641923926s" totalImagesPullingTime="2.776862155s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-06 14:00:41 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 14:00:45.640952898 +0000 UTC m=+621.231514688" watchObservedRunningTime="2026-07-06 14:00:45.641923926 +0000 UTC m=+621.232485703" Jul 06 14:00:56.630474 ip-10-0-136-25 kubenswrapper[2569]: I0706 14:00:56.630445 2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/limitador-limitador-54d6d988d5-g9qp6" Jul 06 14:01:13.960489 ip-10-0-136-25 kubenswrapper[2569]: I0706 14:01:13.960450 2569 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-84fcb57f94-mql2k"] Jul 06 14:01:13.963902 ip-10-0-136-25 kubenswrapper[2569]: I0706 14:01:13.963884 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-84fcb57f94-mql2k" Jul 06 14:01:13.966375 ip-10-0-136-25 kubenswrapper[2569]: I0706 14:01:13.966354 2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-authorino-dockercfg-c5vnx\"" Jul 06 14:01:13.966375 ip-10-0-136-25 kubenswrapper[2569]: I0706 14:01:13.966374 2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-server-cert\"" Jul 06 14:01:13.971013 ip-10-0-136-25 kubenswrapper[2569]: I0706 14:01:13.970986 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-84fcb57f94-mql2k"] Jul 06 14:01:14.008799 ip-10-0-136-25 kubenswrapper[2569]: I0706 14:01:14.008768 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/3b62a439-91b3-416e-a8a7-206ee18c6a40-tls-cert\") pod \"authorino-84fcb57f94-mql2k\" (UID: \"3b62a439-91b3-416e-a8a7-206ee18c6a40\") " pod="kuadrant-system/authorino-84fcb57f94-mql2k" Jul 06 14:01:14.008961 ip-10-0-136-25 kubenswrapper[2569]: I0706 14:01:14.008826 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nzp8s\" (UniqueName: \"kubernetes.io/projected/3b62a439-91b3-416e-a8a7-206ee18c6a40-kube-api-access-nzp8s\") pod \"authorino-84fcb57f94-mql2k\" (UID: \"3b62a439-91b3-416e-a8a7-206ee18c6a40\") " pod="kuadrant-system/authorino-84fcb57f94-mql2k" Jul 06 14:01:14.110142 ip-10-0-136-25 kubenswrapper[2569]: I0706 14:01:14.110100 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/3b62a439-91b3-416e-a8a7-206ee18c6a40-tls-cert\") pod \"authorino-84fcb57f94-mql2k\" (UID: \"3b62a439-91b3-416e-a8a7-206ee18c6a40\") " pod="kuadrant-system/authorino-84fcb57f94-mql2k" Jul 06 14:01:14.110358 ip-10-0-136-25 kubenswrapper[2569]: I0706 14:01:14.110172 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nzp8s\" (UniqueName: \"kubernetes.io/projected/3b62a439-91b3-416e-a8a7-206ee18c6a40-kube-api-access-nzp8s\") pod \"authorino-84fcb57f94-mql2k\" (UID: \"3b62a439-91b3-416e-a8a7-206ee18c6a40\") " pod="kuadrant-system/authorino-84fcb57f94-mql2k" Jul 06 14:01:14.112691 ip-10-0-136-25 kubenswrapper[2569]: I0706 14:01:14.112670 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/3b62a439-91b3-416e-a8a7-206ee18c6a40-tls-cert\") pod \"authorino-84fcb57f94-mql2k\" (UID: \"3b62a439-91b3-416e-a8a7-206ee18c6a40\") " pod="kuadrant-system/authorino-84fcb57f94-mql2k" Jul 06 14:01:14.118653 ip-10-0-136-25 kubenswrapper[2569]: I0706 14:01:14.118635 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nzp8s\" (UniqueName: \"kubernetes.io/projected/3b62a439-91b3-416e-a8a7-206ee18c6a40-kube-api-access-nzp8s\") pod \"authorino-84fcb57f94-mql2k\" (UID: \"3b62a439-91b3-416e-a8a7-206ee18c6a40\") " pod="kuadrant-system/authorino-84fcb57f94-mql2k" Jul 06 14:01:14.273587 ip-10-0-136-25 kubenswrapper[2569]: I0706 14:01:14.273553 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-84fcb57f94-mql2k" Jul 06 14:01:14.416594 ip-10-0-136-25 kubenswrapper[2569]: I0706 14:01:14.416560 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-84fcb57f94-mql2k"] Jul 06 14:01:14.420046 ip-10-0-136-25 kubenswrapper[2569]: W0706 14:01:14.420015 2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod3b62a439_91b3_416e_a8a7_206ee18c6a40.slice/crio-eff87b6c924b3a32c4ec3071735e624b9b5f79e790db49e5092ec6abc6d991aa WatchSource:0}: Error finding container eff87b6c924b3a32c4ec3071735e624b9b5f79e790db49e5092ec6abc6d991aa: Status 404 returned error can't find the container with id eff87b6c924b3a32c4ec3071735e624b9b5f79e790db49e5092ec6abc6d991aa Jul 06 14:01:14.761259 ip-10-0-136-25 kubenswrapper[2569]: I0706 14:01:14.761221 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-84fcb57f94-mql2k" event={"ID":"3b62a439-91b3-416e-a8a7-206ee18c6a40","Type":"ContainerStarted","Data":"eff87b6c924b3a32c4ec3071735e624b9b5f79e790db49e5092ec6abc6d991aa"} Jul 06 14:01:18.784409 ip-10-0-136-25 kubenswrapper[2569]: I0706 14:01:18.784370 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-84fcb57f94-mql2k" event={"ID":"3b62a439-91b3-416e-a8a7-206ee18c6a40","Type":"ContainerStarted","Data":"d025b0e6da6aec74546811d81530f72c0c557755a646ef975c1adfefcafcec98"} Jul 06 14:01:18.801274 ip-10-0-136-25 kubenswrapper[2569]: I0706 14:01:18.801220 2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/authorino-84fcb57f94-mql2k" podStartSLOduration=2.3552244399999998 podStartE2EDuration="5.801193578s" totalImagesPullingTime="3.445969138s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-06 14:01:13 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 14:01:18.799403078 +0000 UTC m=+654.389964868" watchObservedRunningTime="2026-07-06 14:01:18.801193578 +0000 UTC m=+654.391755355" Jul 06 14:02:01.138198 ip-10-0-136-25 kubenswrapper[2569]: I0706 14:02:01.138161 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-g9qp6"] Jul 06 14:02:17.382557 ip-10-0-136-25 kubenswrapper[2569]: I0706 14:02:17.382466 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-g9qp6"] Jul 06 14:02:29.839622 ip-10-0-136-25 kubenswrapper[2569]: I0706 14:02:29.839579 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-g9qp6"] Jul 06 14:02:46.240394 ip-10-0-136-25 kubenswrapper[2569]: I0706 14:02:46.240357 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-g9qp6"] Jul 06 14:02:53.484757 ip-10-0-136-25 kubenswrapper[2569]: I0706 14:02:53.484719 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-g9qp6"] Jul 06 14:03:10.143626 ip-10-0-136-25 kubenswrapper[2569]: I0706 14:03:10.143590 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-g9qp6"] Jul 06 14:03:27.142365 ip-10-0-136-25 kubenswrapper[2569]: I0706 14:03:27.142326 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-g9qp6"] Jul 06 14:04:23.396239 ip-10-0-136-25 kubenswrapper[2569]: I0706 14:04:23.396194 2569 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-f549ff76c-wqjl4"] Jul 06 14:04:23.399491 ip-10-0-136-25 kubenswrapper[2569]: I0706 14:04:23.399474 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-f549ff76c-wqjl4" Jul 06 14:04:23.407804 ip-10-0-136-25 kubenswrapper[2569]: I0706 14:04:23.407781 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-f549ff76c-wqjl4"] Jul 06 14:04:23.566231 ip-10-0-136-25 kubenswrapper[2569]: I0706 14:04:23.566170 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/cf359e55-20b7-43b2-baa5-848ae0a9e07d-tls-cert\") pod \"authorino-f549ff76c-wqjl4\" (UID: \"cf359e55-20b7-43b2-baa5-848ae0a9e07d\") " pod="kuadrant-system/authorino-f549ff76c-wqjl4" Jul 06 14:04:23.566413 ip-10-0-136-25 kubenswrapper[2569]: I0706 14:04:23.566302 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cmc4w\" (UniqueName: \"kubernetes.io/projected/cf359e55-20b7-43b2-baa5-848ae0a9e07d-kube-api-access-cmc4w\") pod \"authorino-f549ff76c-wqjl4\" (UID: \"cf359e55-20b7-43b2-baa5-848ae0a9e07d\") " pod="kuadrant-system/authorino-f549ff76c-wqjl4" Jul 06 14:04:23.667721 ip-10-0-136-25 kubenswrapper[2569]: I0706 14:04:23.667621 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cmc4w\" (UniqueName: \"kubernetes.io/projected/cf359e55-20b7-43b2-baa5-848ae0a9e07d-kube-api-access-cmc4w\") pod \"authorino-f549ff76c-wqjl4\" (UID: \"cf359e55-20b7-43b2-baa5-848ae0a9e07d\") " pod="kuadrant-system/authorino-f549ff76c-wqjl4" Jul 06 14:04:23.667721 ip-10-0-136-25 kubenswrapper[2569]: I0706 14:04:23.667702 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/cf359e55-20b7-43b2-baa5-848ae0a9e07d-tls-cert\") pod \"authorino-f549ff76c-wqjl4\" (UID: \"cf359e55-20b7-43b2-baa5-848ae0a9e07d\") " pod="kuadrant-system/authorino-f549ff76c-wqjl4" Jul 06 14:04:23.670100 ip-10-0-136-25 kubenswrapper[2569]: I0706 14:04:23.670074 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/cf359e55-20b7-43b2-baa5-848ae0a9e07d-tls-cert\") pod \"authorino-f549ff76c-wqjl4\" (UID: \"cf359e55-20b7-43b2-baa5-848ae0a9e07d\") " pod="kuadrant-system/authorino-f549ff76c-wqjl4" Jul 06 14:04:23.675366 ip-10-0-136-25 kubenswrapper[2569]: I0706 14:04:23.675343 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cmc4w\" (UniqueName: \"kubernetes.io/projected/cf359e55-20b7-43b2-baa5-848ae0a9e07d-kube-api-access-cmc4w\") pod \"authorino-f549ff76c-wqjl4\" (UID: \"cf359e55-20b7-43b2-baa5-848ae0a9e07d\") " pod="kuadrant-system/authorino-f549ff76c-wqjl4" Jul 06 14:04:23.709234 ip-10-0-136-25 kubenswrapper[2569]: I0706 14:04:23.709199 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-f549ff76c-wqjl4" Jul 06 14:04:23.839774 ip-10-0-136-25 kubenswrapper[2569]: I0706 14:04:23.839736 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-f549ff76c-wqjl4"] Jul 06 14:04:23.841929 ip-10-0-136-25 kubenswrapper[2569]: W0706 14:04:23.841901 2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podcf359e55_20b7_43b2_baa5_848ae0a9e07d.slice/crio-ac5a3c9da23c6ee50bdf0c7d037ca97dd25a1ce60fd56ee304db2dc675aa2007 WatchSource:0}: Error finding container ac5a3c9da23c6ee50bdf0c7d037ca97dd25a1ce60fd56ee304db2dc675aa2007: Status 404 returned error can't find the container with id ac5a3c9da23c6ee50bdf0c7d037ca97dd25a1ce60fd56ee304db2dc675aa2007 Jul 06 14:04:23.843440 ip-10-0-136-25 kubenswrapper[2569]: I0706 14:04:23.843420 2569 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Jul 06 14:04:24.636662 ip-10-0-136-25 kubenswrapper[2569]: I0706 14:04:24.636567 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-f549ff76c-wqjl4" event={"ID":"cf359e55-20b7-43b2-baa5-848ae0a9e07d","Type":"ContainerStarted","Data":"293b966b56cb820361f3e28e98d8be67ea292ae8a4406596120ef303a6babd10"} Jul 06 14:04:24.636662 ip-10-0-136-25 kubenswrapper[2569]: I0706 14:04:24.636618 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-f549ff76c-wqjl4" event={"ID":"cf359e55-20b7-43b2-baa5-848ae0a9e07d","Type":"ContainerStarted","Data":"ac5a3c9da23c6ee50bdf0c7d037ca97dd25a1ce60fd56ee304db2dc675aa2007"} Jul 06 14:04:24.656977 ip-10-0-136-25 kubenswrapper[2569]: I0706 14:04:24.656913 2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/authorino-f549ff76c-wqjl4" podStartSLOduration=1.162921892 podStartE2EDuration="1.656890174s" totalImagesPullingTime="493.968282ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-06 14:04:23 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 14:04:24.654819858 +0000 UTC m=+840.245381639" watchObservedRunningTime="2026-07-06 14:04:24.656890174 +0000 UTC m=+840.247451948" Jul 06 14:04:24.689422 ip-10-0-136-25 kubenswrapper[2569]: I0706 14:04:24.689382 2569 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-84fcb57f94-mql2k"] Jul 06 14:04:24.689610 ip-10-0-136-25 kubenswrapper[2569]: I0706 14:04:24.689586 2569 kuberuntime_container.go:872] "Killing container with a grace period" pod="kuadrant-system/authorino-84fcb57f94-mql2k" podUID="3b62a439-91b3-416e-a8a7-206ee18c6a40" containerName="authorino" containerID="cri-o://d025b0e6da6aec74546811d81530f72c0c557755a646ef975c1adfefcafcec98" gracePeriod=30 Jul 06 14:04:24.955063 ip-10-0-136-25 kubenswrapper[2569]: I0706 14:04:24.955038 2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-84fcb57f94-mql2k" Jul 06 14:04:25.080057 ip-10-0-136-25 kubenswrapper[2569]: I0706 14:04:25.080027 2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-nzp8s\" (UniqueName: \"kubernetes.io/projected/3b62a439-91b3-416e-a8a7-206ee18c6a40-kube-api-access-nzp8s\") pod \"3b62a439-91b3-416e-a8a7-206ee18c6a40\" (UID: \"3b62a439-91b3-416e-a8a7-206ee18c6a40\") " Jul 06 14:04:25.080410 ip-10-0-136-25 kubenswrapper[2569]: I0706 14:04:25.080388 2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/3b62a439-91b3-416e-a8a7-206ee18c6a40-tls-cert\") pod \"3b62a439-91b3-416e-a8a7-206ee18c6a40\" (UID: \"3b62a439-91b3-416e-a8a7-206ee18c6a40\") " Jul 06 14:04:25.087442 ip-10-0-136-25 kubenswrapper[2569]: I0706 14:04:25.087324 2569 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3b62a439-91b3-416e-a8a7-206ee18c6a40-kube-api-access-nzp8s" (OuterVolumeSpecName: "kube-api-access-nzp8s") pod "3b62a439-91b3-416e-a8a7-206ee18c6a40" (UID: "3b62a439-91b3-416e-a8a7-206ee18c6a40"). InnerVolumeSpecName "kube-api-access-nzp8s". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 06 14:04:25.094425 ip-10-0-136-25 kubenswrapper[2569]: I0706 14:04:25.094395 2569 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3b62a439-91b3-416e-a8a7-206ee18c6a40-tls-cert" (OuterVolumeSpecName: "tls-cert") pod "3b62a439-91b3-416e-a8a7-206ee18c6a40" (UID: "3b62a439-91b3-416e-a8a7-206ee18c6a40"). InnerVolumeSpecName "tls-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 06 14:04:25.183842 ip-10-0-136-25 kubenswrapper[2569]: I0706 14:04:25.183760 2569 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nzp8s\" (UniqueName: \"kubernetes.io/projected/3b62a439-91b3-416e-a8a7-206ee18c6a40-kube-api-access-nzp8s\") on node \"ip-10-0-136-25.ec2.internal\" DevicePath \"\"" Jul 06 14:04:25.183842 ip-10-0-136-25 kubenswrapper[2569]: I0706 14:04:25.183787 2569 reconciler_common.go:299] "Volume detached for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/3b62a439-91b3-416e-a8a7-206ee18c6a40-tls-cert\") on node \"ip-10-0-136-25.ec2.internal\" DevicePath \"\"" Jul 06 14:04:25.642719 ip-10-0-136-25 kubenswrapper[2569]: I0706 14:04:25.642677 2569 generic.go:356] "Generic (PLEG): container finished" podID="3b62a439-91b3-416e-a8a7-206ee18c6a40" containerID="d025b0e6da6aec74546811d81530f72c0c557755a646ef975c1adfefcafcec98" exitCode=0 Jul 06 14:04:25.642719 ip-10-0-136-25 kubenswrapper[2569]: I0706 14:04:25.642724 2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-84fcb57f94-mql2k" Jul 06 14:04:25.643291 ip-10-0-136-25 kubenswrapper[2569]: I0706 14:04:25.642759 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-84fcb57f94-mql2k" event={"ID":"3b62a439-91b3-416e-a8a7-206ee18c6a40","Type":"ContainerDied","Data":"d025b0e6da6aec74546811d81530f72c0c557755a646ef975c1adfefcafcec98"} Jul 06 14:04:25.643291 ip-10-0-136-25 kubenswrapper[2569]: I0706 14:04:25.642799 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-84fcb57f94-mql2k" event={"ID":"3b62a439-91b3-416e-a8a7-206ee18c6a40","Type":"ContainerDied","Data":"eff87b6c924b3a32c4ec3071735e624b9b5f79e790db49e5092ec6abc6d991aa"} Jul 06 14:04:25.643291 ip-10-0-136-25 kubenswrapper[2569]: I0706 14:04:25.642816 2569 scope.go:117] "RemoveContainer" containerID="d025b0e6da6aec74546811d81530f72c0c557755a646ef975c1adfefcafcec98" Jul 06 14:04:25.652143 ip-10-0-136-25 kubenswrapper[2569]: I0706 14:04:25.652125 2569 scope.go:117] "RemoveContainer" containerID="d025b0e6da6aec74546811d81530f72c0c557755a646ef975c1adfefcafcec98" Jul 06 14:04:25.652430 ip-10-0-136-25 kubenswrapper[2569]: E0706 14:04:25.652406 2569 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d025b0e6da6aec74546811d81530f72c0c557755a646ef975c1adfefcafcec98\": container with ID starting with d025b0e6da6aec74546811d81530f72c0c557755a646ef975c1adfefcafcec98 not found: ID does not exist" containerID="d025b0e6da6aec74546811d81530f72c0c557755a646ef975c1adfefcafcec98" Jul 06 14:04:25.652510 ip-10-0-136-25 kubenswrapper[2569]: I0706 14:04:25.652437 2569 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d025b0e6da6aec74546811d81530f72c0c557755a646ef975c1adfefcafcec98"} err="failed to get container status \"d025b0e6da6aec74546811d81530f72c0c557755a646ef975c1adfefcafcec98\": rpc error: code = NotFound desc = could not find container \"d025b0e6da6aec74546811d81530f72c0c557755a646ef975c1adfefcafcec98\": container with ID starting with d025b0e6da6aec74546811d81530f72c0c557755a646ef975c1adfefcafcec98 not found: ID does not exist" Jul 06 14:04:25.695247 ip-10-0-136-25 kubenswrapper[2569]: I0706 14:04:25.695197 2569 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-84fcb57f94-mql2k"] Jul 06 14:04:25.711280 ip-10-0-136-25 kubenswrapper[2569]: I0706 14:04:25.711245 2569 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-84fcb57f94-mql2k"] Jul 06 14:04:26.943826 ip-10-0-136-25 kubenswrapper[2569]: I0706 14:04:26.943460 2569 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="3b62a439-91b3-416e-a8a7-206ee18c6a40" path="/var/lib/kubelet/pods/3b62a439-91b3-416e-a8a7-206ee18c6a40/volumes" Jul 06 14:05:13.244434 ip-10-0-136-25 kubenswrapper[2569]: I0706 14:05:13.244397 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-g9qp6"] Jul 06 14:05:22.940749 ip-10-0-136-25 kubenswrapper[2569]: I0706 14:05:22.940669 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-g9qp6"] Jul 06 14:05:24.870382 ip-10-0-136-25 kubenswrapper[2569]: I0706 14:05:24.870351 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-68b757fb67-z2h95_dfffcdca-94d6-4732-815f-bcb533f538e9/console-operator/2.log" Jul 06 14:05:24.871033 ip-10-0-136-25 kubenswrapper[2569]: I0706 14:05:24.871017 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-68b757fb67-z2h95_dfffcdca-94d6-4732-815f-bcb533f538e9/console-operator/2.log" Jul 06 14:05:33.934034 ip-10-0-136-25 kubenswrapper[2569]: I0706 14:05:33.934001 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-g9qp6"] Jul 06 14:05:43.144991 ip-10-0-136-25 kubenswrapper[2569]: I0706 14:05:43.144957 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-g9qp6"] Jul 06 14:05:53.854531 ip-10-0-136-25 kubenswrapper[2569]: I0706 14:05:53.854490 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-g9qp6"] Jul 06 14:06:21.008177 ip-10-0-136-25 kubenswrapper[2569]: I0706 14:06:21.008121 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-g9qp6"] Jul 06 14:06:31.860500 ip-10-0-136-25 kubenswrapper[2569]: I0706 14:06:31.860464 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-g9qp6"] Jul 06 14:07:08.138911 ip-10-0-136-25 kubenswrapper[2569]: I0706 14:07:08.138822 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-g9qp6"] Jul 06 14:07:17.050239 ip-10-0-136-25 kubenswrapper[2569]: I0706 14:07:17.050190 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-g9qp6"] Jul 06 14:07:25.139550 ip-10-0-136-25 kubenswrapper[2569]: I0706 14:07:25.139510 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-g9qp6"] Jul 06 14:07:35.448293 ip-10-0-136-25 kubenswrapper[2569]: I0706 14:07:35.448258 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-g9qp6"] Jul 06 14:07:45.894103 ip-10-0-136-25 kubenswrapper[2569]: I0706 14:07:45.894066 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-g9qp6"] Jul 06 14:07:56.640271 ip-10-0-136-25 kubenswrapper[2569]: I0706 14:07:56.640235 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-g9qp6"] Jul 06 14:08:08.332059 ip-10-0-136-25 kubenswrapper[2569]: I0706 14:08:08.332021 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-g9qp6"] Jul 06 14:08:17.738321 ip-10-0-136-25 kubenswrapper[2569]: I0706 14:08:17.738239 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-g9qp6"] Jul 06 14:08:41.144295 ip-10-0-136-25 kubenswrapper[2569]: I0706 14:08:41.144253 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-g9qp6"] Jul 06 14:08:49.444829 ip-10-0-136-25 kubenswrapper[2569]: I0706 14:08:49.444790 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-g9qp6"] Jul 06 14:09:12.940084 ip-10-0-136-25 kubenswrapper[2569]: I0706 14:09:12.940054 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-g9qp6"] Jul 06 14:09:24.445738 ip-10-0-136-25 kubenswrapper[2569]: I0706 14:09:24.445700 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-g9qp6"] Jul 06 14:09:38.843524 ip-10-0-136-25 kubenswrapper[2569]: I0706 14:09:38.843488 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-g9qp6"] Jul 06 14:09:49.346097 ip-10-0-136-25 kubenswrapper[2569]: I0706 14:09:49.346060 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-g9qp6"] Jul 06 14:09:58.753682 ip-10-0-136-25 kubenswrapper[2569]: I0706 14:09:58.753641 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-g9qp6"] Jul 06 14:10:07.539623 ip-10-0-136-25 kubenswrapper[2569]: I0706 14:10:07.539581 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-g9qp6"] Jul 06 14:10:16.844753 ip-10-0-136-25 kubenswrapper[2569]: I0706 14:10:16.844714 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-g9qp6"] Jul 06 14:10:24.908005 ip-10-0-136-25 kubenswrapper[2569]: I0706 14:10:24.907965 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-68b757fb67-z2h95_dfffcdca-94d6-4732-815f-bcb533f538e9/console-operator/2.log" Jul 06 14:10:24.911239 ip-10-0-136-25 kubenswrapper[2569]: I0706 14:10:24.911191 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-68b757fb67-z2h95_dfffcdca-94d6-4732-815f-bcb533f538e9/console-operator/2.log" Jul 06 14:12:07.645779 ip-10-0-136-25 kubenswrapper[2569]: I0706 14:12:07.645744 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-g9qp6"] Jul 06 14:12:18.246978 ip-10-0-136-25 kubenswrapper[2569]: I0706 14:12:18.246941 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-g9qp6"] Jul 06 14:12:27.945746 ip-10-0-136-25 kubenswrapper[2569]: I0706 14:12:27.945709 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-g9qp6"] Jul 06 14:12:39.539593 ip-10-0-136-25 kubenswrapper[2569]: I0706 14:12:39.539532 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-g9qp6"] Jul 06 14:12:51.437594 ip-10-0-136-25 kubenswrapper[2569]: I0706 14:12:51.437504 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-g9qp6"] Jul 06 14:12:59.675394 ip-10-0-136-25 kubenswrapper[2569]: I0706 14:12:59.675349 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-g9qp6"] Jul 06 14:13:07.331434 ip-10-0-136-25 kubenswrapper[2569]: I0706 14:13:07.331395 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-g9qp6"] Jul 06 14:13:15.842400 ip-10-0-136-25 kubenswrapper[2569]: I0706 14:13:15.842363 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-g9qp6"] Jul 06 14:13:25.237414 ip-10-0-136-25 kubenswrapper[2569]: I0706 14:13:25.237375 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-g9qp6"] Jul 06 14:13:35.540584 ip-10-0-136-25 kubenswrapper[2569]: I0706 14:13:35.540546 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-g9qp6"] Jul 06 14:13:47.440636 ip-10-0-136-25 kubenswrapper[2569]: I0706 14:13:47.440602 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-g9qp6"] Jul 06 14:13:54.336839 ip-10-0-136-25 kubenswrapper[2569]: I0706 14:13:54.336798 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-g9qp6"] Jul 06 14:14:15.837807 ip-10-0-136-25 kubenswrapper[2569]: I0706 14:14:15.837764 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-g9qp6"] Jul 06 14:14:22.736484 ip-10-0-136-25 kubenswrapper[2569]: I0706 14:14:22.736400 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-g9qp6"] Jul 06 14:14:31.635711 ip-10-0-136-25 kubenswrapper[2569]: I0706 14:14:31.635668 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-g9qp6"] Jul 06 14:14:42.637599 ip-10-0-136-25 kubenswrapper[2569]: I0706 14:14:42.637559 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-g9qp6"] Jul 06 14:15:00.449053 ip-10-0-136-25 kubenswrapper[2569]: I0706 14:15:00.449011 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-g9qp6"] Jul 06 14:15:08.843251 ip-10-0-136-25 kubenswrapper[2569]: I0706 14:15:08.843190 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-g9qp6"] Jul 06 14:15:18.945339 ip-10-0-136-25 kubenswrapper[2569]: I0706 14:15:18.945310 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-g9qp6"] Jul 06 14:15:24.946613 ip-10-0-136-25 kubenswrapper[2569]: I0706 14:15:24.946581 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-68b757fb67-z2h95_dfffcdca-94d6-4732-815f-bcb533f538e9/console-operator/2.log" Jul 06 14:15:24.951363 ip-10-0-136-25 kubenswrapper[2569]: I0706 14:15:24.951341 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-68b757fb67-z2h95_dfffcdca-94d6-4732-815f-bcb533f538e9/console-operator/2.log" Jul 06 14:15:28.448718 ip-10-0-136-25 kubenswrapper[2569]: I0706 14:15:28.448675 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-g9qp6"] Jul 06 14:15:36.842312 ip-10-0-136-25 kubenswrapper[2569]: I0706 14:15:36.842275 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-g9qp6"] Jul 06 14:15:45.448570 ip-10-0-136-25 kubenswrapper[2569]: I0706 14:15:45.448529 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-g9qp6"] Jul 06 14:15:51.077630 ip-10-0-136-25 kubenswrapper[2569]: I0706 14:15:51.077518 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-g9qp6"] Jul 06 14:15:59.849949 ip-10-0-136-25 kubenswrapper[2569]: I0706 14:15:59.849905 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-g9qp6"] Jul 06 14:16:08.644136 ip-10-0-136-25 kubenswrapper[2569]: I0706 14:16:08.644096 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-g9qp6"] Jul 06 14:16:17.649495 ip-10-0-136-25 kubenswrapper[2569]: I0706 14:16:17.649448 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-g9qp6"] Jul 06 14:16:28.485950 ip-10-0-136-25 kubenswrapper[2569]: I0706 14:16:28.485919 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-g9qp6"] Jul 06 14:16:39.536525 ip-10-0-136-25 kubenswrapper[2569]: I0706 14:16:39.536486 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-g9qp6"] Jul 06 14:16:48.143819 ip-10-0-136-25 kubenswrapper[2569]: I0706 14:16:48.143780 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-g9qp6"] Jul 06 14:17:01.244930 ip-10-0-136-25 kubenswrapper[2569]: I0706 14:17:01.244887 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-g9qp6"] Jul 06 14:17:13.544177 ip-10-0-136-25 kubenswrapper[2569]: I0706 14:17:13.544124 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-g9qp6"] Jul 06 14:17:26.035181 ip-10-0-136-25 kubenswrapper[2569]: I0706 14:17:26.035089 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-g9qp6"] Jul 06 14:17:45.082888 ip-10-0-136-25 kubenswrapper[2569]: I0706 14:17:45.082847 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-g9qp6"] Jul 06 14:17:55.760029 ip-10-0-136-25 kubenswrapper[2569]: I0706 14:17:55.759994 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-g9qp6"] Jul 06 14:18:06.239894 ip-10-0-136-25 kubenswrapper[2569]: I0706 14:18:06.239855 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-g9qp6"] Jul 06 14:18:15.539621 ip-10-0-136-25 kubenswrapper[2569]: I0706 14:18:15.539590 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-g9qp6"] Jul 06 14:18:23.338948 ip-10-0-136-25 kubenswrapper[2569]: I0706 14:18:23.338908 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-g9qp6"] Jul 06 14:19:47.040674 ip-10-0-136-25 kubenswrapper[2569]: I0706 14:19:47.040634 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-g9qp6"] Jul 06 14:19:54.343425 ip-10-0-136-25 kubenswrapper[2569]: I0706 14:19:54.343388 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-g9qp6"] Jul 06 14:20:01.600305 ip-10-0-136-25 kubenswrapper[2569]: I0706 14:20:01.600263 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-g9qp6"] Jul 06 14:20:09.250608 ip-10-0-136-25 kubenswrapper[2569]: I0706 14:20:09.250569 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-g9qp6"] Jul 06 14:20:21.239383 ip-10-0-136-25 kubenswrapper[2569]: I0706 14:20:21.239301 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-g9qp6"] Jul 06 14:20:24.976058 ip-10-0-136-25 kubenswrapper[2569]: I0706 14:20:24.976029 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-68b757fb67-z2h95_dfffcdca-94d6-4732-815f-bcb533f538e9/console-operator/2.log" Jul 06 14:20:24.981076 ip-10-0-136-25 kubenswrapper[2569]: I0706 14:20:24.981054 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-68b757fb67-z2h95_dfffcdca-94d6-4732-815f-bcb533f538e9/console-operator/2.log" Jul 06 14:20:53.246691 ip-10-0-136-25 kubenswrapper[2569]: I0706 14:20:53.246652 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-g9qp6"] Jul 06 14:21:06.648439 ip-10-0-136-25 kubenswrapper[2569]: I0706 14:21:06.648402 2569 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-mt-2b745686/e2e-lifecycle-model-2b745686-kserve-7ffdf54555-gh4bs"] Jul 06 14:21:06.648882 ip-10-0-136-25 kubenswrapper[2569]: I0706 14:21:06.648799 2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3b62a439-91b3-416e-a8a7-206ee18c6a40" containerName="authorino" Jul 06 14:21:06.648882 ip-10-0-136-25 kubenswrapper[2569]: I0706 14:21:06.648810 2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="3b62a439-91b3-416e-a8a7-206ee18c6a40" containerName="authorino" Jul 06 14:21:06.648882 ip-10-0-136-25 kubenswrapper[2569]: I0706 14:21:06.648866 2569 memory_manager.go:368] "RemoveStaleState removing state" podUID="3b62a439-91b3-416e-a8a7-206ee18c6a40" containerName="authorino" Jul 06 14:21:06.652076 ip-10-0-136-25 kubenswrapper[2569]: I0706 14:21:06.652058 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-mt-2b745686/e2e-lifecycle-model-2b745686-kserve-7ffdf54555-gh4bs" Jul 06 14:21:06.654799 ip-10-0-136-25 kubenswrapper[2569]: I0706 14:21:06.654773 2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-mt-2b745686\"/\"default-dockercfg-7jzvq\"" Jul 06 14:21:06.654927 ip-10-0-136-25 kubenswrapper[2569]: I0706 14:21:06.654773 2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-mt-2b745686\"/\"kube-root-ca.crt\"" Jul 06 14:21:06.654927 ip-10-0-136-25 kubenswrapper[2569]: I0706 14:21:06.654779 2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-mt-2b745686\"/\"e2e-lifecycle-model-2b745686-kserve-self-signed-certs\"" Jul 06 14:21:06.654927 ip-10-0-136-25 kubenswrapper[2569]: I0706 14:21:06.654785 2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-mt-2b745686\"/\"openshift-service-ca.crt\"" Jul 06 14:21:06.662082 ip-10-0-136-25 kubenswrapper[2569]: I0706 14:21:06.662061 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-mt-2b745686/e2e-lifecycle-model-2b745686-kserve-7ffdf54555-gh4bs"] Jul 06 14:21:06.746099 ip-10-0-136-25 kubenswrapper[2569]: I0706 14:21:06.746061 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/39ce5a2a-6220-4b39-b075-e4ca9d8d1071-tmp-dir\") pod \"e2e-lifecycle-model-2b745686-kserve-7ffdf54555-gh4bs\" (UID: \"39ce5a2a-6220-4b39-b075-e4ca9d8d1071\") " pod="ai-tenant-e2e-mt-2b745686/e2e-lifecycle-model-2b745686-kserve-7ffdf54555-gh4bs" Jul 06 14:21:06.746337 ip-10-0-136-25 kubenswrapper[2569]: I0706 14:21:06.746123 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/39ce5a2a-6220-4b39-b075-e4ca9d8d1071-home\") pod \"e2e-lifecycle-model-2b745686-kserve-7ffdf54555-gh4bs\" (UID: \"39ce5a2a-6220-4b39-b075-e4ca9d8d1071\") " pod="ai-tenant-e2e-mt-2b745686/e2e-lifecycle-model-2b745686-kserve-7ffdf54555-gh4bs" Jul 06 14:21:06.746337 ip-10-0-136-25 kubenswrapper[2569]: I0706 14:21:06.746162 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/39ce5a2a-6220-4b39-b075-e4ca9d8d1071-dshm\") pod \"e2e-lifecycle-model-2b745686-kserve-7ffdf54555-gh4bs\" (UID: \"39ce5a2a-6220-4b39-b075-e4ca9d8d1071\") " pod="ai-tenant-e2e-mt-2b745686/e2e-lifecycle-model-2b745686-kserve-7ffdf54555-gh4bs" Jul 06 14:21:06.746337 ip-10-0-136-25 kubenswrapper[2569]: I0706 14:21:06.746185 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/39ce5a2a-6220-4b39-b075-e4ca9d8d1071-model-cache\") pod \"e2e-lifecycle-model-2b745686-kserve-7ffdf54555-gh4bs\" (UID: \"39ce5a2a-6220-4b39-b075-e4ca9d8d1071\") " pod="ai-tenant-e2e-mt-2b745686/e2e-lifecycle-model-2b745686-kserve-7ffdf54555-gh4bs" Jul 06 14:21:06.746337 ip-10-0-136-25 kubenswrapper[2569]: I0706 14:21:06.746201 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-p9xlj\" (UniqueName: \"kubernetes.io/projected/39ce5a2a-6220-4b39-b075-e4ca9d8d1071-kube-api-access-p9xlj\") pod \"e2e-lifecycle-model-2b745686-kserve-7ffdf54555-gh4bs\" (UID: \"39ce5a2a-6220-4b39-b075-e4ca9d8d1071\") " pod="ai-tenant-e2e-mt-2b745686/e2e-lifecycle-model-2b745686-kserve-7ffdf54555-gh4bs" Jul 06 14:21:06.746337 ip-10-0-136-25 kubenswrapper[2569]: I0706 14:21:06.746300 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/39ce5a2a-6220-4b39-b075-e4ca9d8d1071-tls-certs\") pod \"e2e-lifecycle-model-2b745686-kserve-7ffdf54555-gh4bs\" (UID: \"39ce5a2a-6220-4b39-b075-e4ca9d8d1071\") " pod="ai-tenant-e2e-mt-2b745686/e2e-lifecycle-model-2b745686-kserve-7ffdf54555-gh4bs" Jul 06 14:21:06.847036 ip-10-0-136-25 kubenswrapper[2569]: I0706 14:21:06.846992 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/39ce5a2a-6220-4b39-b075-e4ca9d8d1071-dshm\") pod \"e2e-lifecycle-model-2b745686-kserve-7ffdf54555-gh4bs\" (UID: \"39ce5a2a-6220-4b39-b075-e4ca9d8d1071\") " pod="ai-tenant-e2e-mt-2b745686/e2e-lifecycle-model-2b745686-kserve-7ffdf54555-gh4bs" Jul 06 14:21:06.847257 ip-10-0-136-25 kubenswrapper[2569]: I0706 14:21:06.847047 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/39ce5a2a-6220-4b39-b075-e4ca9d8d1071-model-cache\") pod \"e2e-lifecycle-model-2b745686-kserve-7ffdf54555-gh4bs\" (UID: \"39ce5a2a-6220-4b39-b075-e4ca9d8d1071\") " pod="ai-tenant-e2e-mt-2b745686/e2e-lifecycle-model-2b745686-kserve-7ffdf54555-gh4bs" Jul 06 14:21:06.847257 ip-10-0-136-25 kubenswrapper[2569]: I0706 14:21:06.847072 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p9xlj\" (UniqueName: \"kubernetes.io/projected/39ce5a2a-6220-4b39-b075-e4ca9d8d1071-kube-api-access-p9xlj\") pod \"e2e-lifecycle-model-2b745686-kserve-7ffdf54555-gh4bs\" (UID: \"39ce5a2a-6220-4b39-b075-e4ca9d8d1071\") " pod="ai-tenant-e2e-mt-2b745686/e2e-lifecycle-model-2b745686-kserve-7ffdf54555-gh4bs" Jul 06 14:21:06.847257 ip-10-0-136-25 kubenswrapper[2569]: I0706 14:21:06.847121 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/39ce5a2a-6220-4b39-b075-e4ca9d8d1071-tls-certs\") pod \"e2e-lifecycle-model-2b745686-kserve-7ffdf54555-gh4bs\" (UID: \"39ce5a2a-6220-4b39-b075-e4ca9d8d1071\") " pod="ai-tenant-e2e-mt-2b745686/e2e-lifecycle-model-2b745686-kserve-7ffdf54555-gh4bs" Jul 06 14:21:06.847257 ip-10-0-136-25 kubenswrapper[2569]: I0706 14:21:06.847170 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/39ce5a2a-6220-4b39-b075-e4ca9d8d1071-tmp-dir\") pod \"e2e-lifecycle-model-2b745686-kserve-7ffdf54555-gh4bs\" (UID: \"39ce5a2a-6220-4b39-b075-e4ca9d8d1071\") " pod="ai-tenant-e2e-mt-2b745686/e2e-lifecycle-model-2b745686-kserve-7ffdf54555-gh4bs" Jul 06 14:21:06.847494 ip-10-0-136-25 kubenswrapper[2569]: I0706 14:21:06.847256 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/39ce5a2a-6220-4b39-b075-e4ca9d8d1071-home\") pod \"e2e-lifecycle-model-2b745686-kserve-7ffdf54555-gh4bs\" (UID: \"39ce5a2a-6220-4b39-b075-e4ca9d8d1071\") " pod="ai-tenant-e2e-mt-2b745686/e2e-lifecycle-model-2b745686-kserve-7ffdf54555-gh4bs" Jul 06 14:21:06.847552 ip-10-0-136-25 kubenswrapper[2569]: I0706 14:21:06.847523 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/39ce5a2a-6220-4b39-b075-e4ca9d8d1071-model-cache\") pod \"e2e-lifecycle-model-2b745686-kserve-7ffdf54555-gh4bs\" (UID: \"39ce5a2a-6220-4b39-b075-e4ca9d8d1071\") " pod="ai-tenant-e2e-mt-2b745686/e2e-lifecycle-model-2b745686-kserve-7ffdf54555-gh4bs" Jul 06 14:21:06.847618 ip-10-0-136-25 kubenswrapper[2569]: I0706 14:21:06.847593 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/39ce5a2a-6220-4b39-b075-e4ca9d8d1071-home\") pod \"e2e-lifecycle-model-2b745686-kserve-7ffdf54555-gh4bs\" (UID: \"39ce5a2a-6220-4b39-b075-e4ca9d8d1071\") " pod="ai-tenant-e2e-mt-2b745686/e2e-lifecycle-model-2b745686-kserve-7ffdf54555-gh4bs" Jul 06 14:21:06.847662 ip-10-0-136-25 kubenswrapper[2569]: I0706 14:21:06.847615 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/39ce5a2a-6220-4b39-b075-e4ca9d8d1071-tmp-dir\") pod \"e2e-lifecycle-model-2b745686-kserve-7ffdf54555-gh4bs\" (UID: \"39ce5a2a-6220-4b39-b075-e4ca9d8d1071\") " pod="ai-tenant-e2e-mt-2b745686/e2e-lifecycle-model-2b745686-kserve-7ffdf54555-gh4bs" Jul 06 14:21:06.849470 ip-10-0-136-25 kubenswrapper[2569]: I0706 14:21:06.849451 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/39ce5a2a-6220-4b39-b075-e4ca9d8d1071-dshm\") pod \"e2e-lifecycle-model-2b745686-kserve-7ffdf54555-gh4bs\" (UID: \"39ce5a2a-6220-4b39-b075-e4ca9d8d1071\") " pod="ai-tenant-e2e-mt-2b745686/e2e-lifecycle-model-2b745686-kserve-7ffdf54555-gh4bs" Jul 06 14:21:06.849637 ip-10-0-136-25 kubenswrapper[2569]: I0706 14:21:06.849622 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/39ce5a2a-6220-4b39-b075-e4ca9d8d1071-tls-certs\") pod \"e2e-lifecycle-model-2b745686-kserve-7ffdf54555-gh4bs\" (UID: \"39ce5a2a-6220-4b39-b075-e4ca9d8d1071\") " pod="ai-tenant-e2e-mt-2b745686/e2e-lifecycle-model-2b745686-kserve-7ffdf54555-gh4bs" Jul 06 14:21:06.854809 ip-10-0-136-25 kubenswrapper[2569]: I0706 14:21:06.854787 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-p9xlj\" (UniqueName: \"kubernetes.io/projected/39ce5a2a-6220-4b39-b075-e4ca9d8d1071-kube-api-access-p9xlj\") pod \"e2e-lifecycle-model-2b745686-kserve-7ffdf54555-gh4bs\" (UID: \"39ce5a2a-6220-4b39-b075-e4ca9d8d1071\") " pod="ai-tenant-e2e-mt-2b745686/e2e-lifecycle-model-2b745686-kserve-7ffdf54555-gh4bs" Jul 06 14:21:06.964415 ip-10-0-136-25 kubenswrapper[2569]: I0706 14:21:06.964318 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-mt-2b745686/e2e-lifecycle-model-2b745686-kserve-7ffdf54555-gh4bs" Jul 06 14:21:07.098070 ip-10-0-136-25 kubenswrapper[2569]: I0706 14:21:07.098043 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-mt-2b745686/e2e-lifecycle-model-2b745686-kserve-7ffdf54555-gh4bs"] Jul 06 14:21:07.100326 ip-10-0-136-25 kubenswrapper[2569]: W0706 14:21:07.100299 2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod39ce5a2a_6220_4b39_b075_e4ca9d8d1071.slice/crio-6d7ea9e85efb717144eac48fef5cfe82c7456134ce1bf7849b031d3baa622769 WatchSource:0}: Error finding container 6d7ea9e85efb717144eac48fef5cfe82c7456134ce1bf7849b031d3baa622769: Status 404 returned error can't find the container with id 6d7ea9e85efb717144eac48fef5cfe82c7456134ce1bf7849b031d3baa622769 Jul 06 14:21:07.102137 ip-10-0-136-25 kubenswrapper[2569]: I0706 14:21:07.102121 2569 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Jul 06 14:21:07.316459 ip-10-0-136-25 kubenswrapper[2569]: I0706 14:21:07.316416 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-mt-2b745686/e2e-lifecycle-model-2b745686-kserve-7ffdf54555-gh4bs" event={"ID":"39ce5a2a-6220-4b39-b075-e4ca9d8d1071","Type":"ContainerStarted","Data":"6d7ea9e85efb717144eac48fef5cfe82c7456134ce1bf7849b031d3baa622769"} Jul 06 14:21:08.323563 ip-10-0-136-25 kubenswrapper[2569]: I0706 14:21:08.323520 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-mt-2b745686/e2e-lifecycle-model-2b745686-kserve-7ffdf54555-gh4bs" event={"ID":"39ce5a2a-6220-4b39-b075-e4ca9d8d1071","Type":"ContainerStarted","Data":"420688d5077e63cb381725cf038f7e764516c2a6d735967dc3cbd8373561a594"} Jul 06 14:21:08.342226 ip-10-0-136-25 kubenswrapper[2569]: I0706 14:21:08.342153 2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-mt-2b745686/e2e-lifecycle-model-2b745686-kserve-7ffdf54555-gh4bs" podStartSLOduration=1.217594895 podStartE2EDuration="2.342138393s" totalImagesPullingTime="1.124543498s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-06 14:21:06 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 14:21:08.340202979 +0000 UTC m=+1843.930764781" watchObservedRunningTime="2026-07-06 14:21:08.342138393 +0000 UTC m=+1843.932700170" Jul 06 14:21:09.835628 ip-10-0-136-25 kubenswrapper[2569]: I0706 14:21:09.835584 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-g9qp6"] Jul 06 14:21:16.965043 ip-10-0-136-25 kubenswrapper[2569]: I0706 14:21:16.964997 2569 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-mt-2b745686/e2e-lifecycle-model-2b745686-kserve-7ffdf54555-gh4bs" Jul 06 14:21:16.965043 ip-10-0-136-25 kubenswrapper[2569]: I0706 14:21:16.965055 2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-mt-2b745686/e2e-lifecycle-model-2b745686-kserve-7ffdf54555-gh4bs" Jul 06 14:21:16.977508 ip-10-0-136-25 kubenswrapper[2569]: I0706 14:21:16.977477 2569 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-mt-2b745686/e2e-lifecycle-model-2b745686-kserve-7ffdf54555-gh4bs" Jul 06 14:21:17.376413 ip-10-0-136-25 kubenswrapper[2569]: I0706 14:21:17.376381 2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-mt-2b745686/e2e-lifecycle-model-2b745686-kserve-7ffdf54555-gh4bs" Jul 06 14:21:29.834731 ip-10-0-136-25 kubenswrapper[2569]: I0706 14:21:29.834682 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-g9qp6"] Jul 06 14:21:41.430639 ip-10-0-136-25 kubenswrapper[2569]: I0706 14:21:41.430599 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-g9qp6"] Jul 06 14:21:41.556432 ip-10-0-136-25 kubenswrapper[2569]: I0706 14:21:41.556393 2569 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-mt-2b745686/e2e-lifecycle-model-2b745686-kserve-7ffdf54555-gh4bs"] Jul 06 14:21:56.576874 ip-10-0-136-25 kubenswrapper[2569]: I0706 14:21:56.576785 2569 kuberuntime_container.go:872] "Killing container with a grace period" pod="ai-tenant-e2e-mt-2b745686/e2e-lifecycle-model-2b745686-kserve-7ffdf54555-gh4bs" podUID="39ce5a2a-6220-4b39-b075-e4ca9d8d1071" containerName="main" containerID="cri-o://420688d5077e63cb381725cf038f7e764516c2a6d735967dc3cbd8373561a594" gracePeriod=45 Jul 06 14:21:56.833497 ip-10-0-136-25 kubenswrapper[2569]: I0706 14:21:56.833432 2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-mt-2b745686/e2e-lifecycle-model-2b745686-kserve-7ffdf54555-gh4bs" Jul 06 14:21:56.911460 ip-10-0-136-25 kubenswrapper[2569]: I0706 14:21:56.911417 2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/39ce5a2a-6220-4b39-b075-e4ca9d8d1071-tmp-dir\") pod \"39ce5a2a-6220-4b39-b075-e4ca9d8d1071\" (UID: \"39ce5a2a-6220-4b39-b075-e4ca9d8d1071\") " Jul 06 14:21:56.911654 ip-10-0-136-25 kubenswrapper[2569]: I0706 14:21:56.911476 2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/39ce5a2a-6220-4b39-b075-e4ca9d8d1071-tls-certs\") pod \"39ce5a2a-6220-4b39-b075-e4ca9d8d1071\" (UID: \"39ce5a2a-6220-4b39-b075-e4ca9d8d1071\") " Jul 06 14:21:56.911654 ip-10-0-136-25 kubenswrapper[2569]: I0706 14:21:56.911503 2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/39ce5a2a-6220-4b39-b075-e4ca9d8d1071-home\") pod \"39ce5a2a-6220-4b39-b075-e4ca9d8d1071\" (UID: \"39ce5a2a-6220-4b39-b075-e4ca9d8d1071\") " Jul 06 14:21:56.911654 ip-10-0-136-25 kubenswrapper[2569]: I0706 14:21:56.911525 2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/39ce5a2a-6220-4b39-b075-e4ca9d8d1071-dshm\") pod \"39ce5a2a-6220-4b39-b075-e4ca9d8d1071\" (UID: \"39ce5a2a-6220-4b39-b075-e4ca9d8d1071\") " Jul 06 14:21:56.911654 ip-10-0-136-25 kubenswrapper[2569]: I0706 14:21:56.911552 2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-p9xlj\" (UniqueName: \"kubernetes.io/projected/39ce5a2a-6220-4b39-b075-e4ca9d8d1071-kube-api-access-p9xlj\") pod \"39ce5a2a-6220-4b39-b075-e4ca9d8d1071\" (UID: \"39ce5a2a-6220-4b39-b075-e4ca9d8d1071\") " Jul 06 14:21:56.911654 ip-10-0-136-25 kubenswrapper[2569]: I0706 14:21:56.911617 2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/39ce5a2a-6220-4b39-b075-e4ca9d8d1071-model-cache\") pod \"39ce5a2a-6220-4b39-b075-e4ca9d8d1071\" (UID: \"39ce5a2a-6220-4b39-b075-e4ca9d8d1071\") " Jul 06 14:21:56.911910 ip-10-0-136-25 kubenswrapper[2569]: I0706 14:21:56.911777 2569 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/39ce5a2a-6220-4b39-b075-e4ca9d8d1071-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "39ce5a2a-6220-4b39-b075-e4ca9d8d1071" (UID: "39ce5a2a-6220-4b39-b075-e4ca9d8d1071"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 06 14:21:56.911910 ip-10-0-136-25 kubenswrapper[2569]: I0706 14:21:56.911813 2569 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/39ce5a2a-6220-4b39-b075-e4ca9d8d1071-home" (OuterVolumeSpecName: "home") pod "39ce5a2a-6220-4b39-b075-e4ca9d8d1071" (UID: "39ce5a2a-6220-4b39-b075-e4ca9d8d1071"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 06 14:21:56.912028 ip-10-0-136-25 kubenswrapper[2569]: I0706 14:21:56.911949 2569 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/39ce5a2a-6220-4b39-b075-e4ca9d8d1071-model-cache" (OuterVolumeSpecName: "model-cache") pod "39ce5a2a-6220-4b39-b075-e4ca9d8d1071" (UID: "39ce5a2a-6220-4b39-b075-e4ca9d8d1071"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 06 14:21:56.912028 ip-10-0-136-25 kubenswrapper[2569]: I0706 14:21:56.911966 2569 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/39ce5a2a-6220-4b39-b075-e4ca9d8d1071-tmp-dir\") on node \"ip-10-0-136-25.ec2.internal\" DevicePath \"\"" Jul 06 14:21:56.912028 ip-10-0-136-25 kubenswrapper[2569]: I0706 14:21:56.911991 2569 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/39ce5a2a-6220-4b39-b075-e4ca9d8d1071-home\") on node \"ip-10-0-136-25.ec2.internal\" DevicePath \"\"" Jul 06 14:21:56.913851 ip-10-0-136-25 kubenswrapper[2569]: I0706 14:21:56.913826 2569 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/39ce5a2a-6220-4b39-b075-e4ca9d8d1071-dshm" (OuterVolumeSpecName: "dshm") pod "39ce5a2a-6220-4b39-b075-e4ca9d8d1071" (UID: "39ce5a2a-6220-4b39-b075-e4ca9d8d1071"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 06 14:21:56.913851 ip-10-0-136-25 kubenswrapper[2569]: I0706 14:21:56.913842 2569 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/39ce5a2a-6220-4b39-b075-e4ca9d8d1071-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "39ce5a2a-6220-4b39-b075-e4ca9d8d1071" (UID: "39ce5a2a-6220-4b39-b075-e4ca9d8d1071"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 06 14:21:56.913995 ip-10-0-136-25 kubenswrapper[2569]: I0706 14:21:56.913844 2569 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/39ce5a2a-6220-4b39-b075-e4ca9d8d1071-kube-api-access-p9xlj" (OuterVolumeSpecName: "kube-api-access-p9xlj") pod "39ce5a2a-6220-4b39-b075-e4ca9d8d1071" (UID: "39ce5a2a-6220-4b39-b075-e4ca9d8d1071"). InnerVolumeSpecName "kube-api-access-p9xlj". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 06 14:21:57.012852 ip-10-0-136-25 kubenswrapper[2569]: I0706 14:21:57.012816 2569 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/39ce5a2a-6220-4b39-b075-e4ca9d8d1071-dshm\") on node \"ip-10-0-136-25.ec2.internal\" DevicePath \"\"" Jul 06 14:21:57.012852 ip-10-0-136-25 kubenswrapper[2569]: I0706 14:21:57.012854 2569 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-p9xlj\" (UniqueName: \"kubernetes.io/projected/39ce5a2a-6220-4b39-b075-e4ca9d8d1071-kube-api-access-p9xlj\") on node \"ip-10-0-136-25.ec2.internal\" DevicePath \"\"" Jul 06 14:21:57.012852 ip-10-0-136-25 kubenswrapper[2569]: I0706 14:21:57.012869 2569 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/39ce5a2a-6220-4b39-b075-e4ca9d8d1071-model-cache\") on node \"ip-10-0-136-25.ec2.internal\" DevicePath \"\"" Jul 06 14:21:57.013164 ip-10-0-136-25 kubenswrapper[2569]: I0706 14:21:57.012881 2569 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/39ce5a2a-6220-4b39-b075-e4ca9d8d1071-tls-certs\") on node \"ip-10-0-136-25.ec2.internal\" DevicePath \"\"" Jul 06 14:21:57.556086 ip-10-0-136-25 kubenswrapper[2569]: I0706 14:21:57.556050 2569 generic.go:356] "Generic (PLEG): container finished" podID="39ce5a2a-6220-4b39-b075-e4ca9d8d1071" containerID="420688d5077e63cb381725cf038f7e764516c2a6d735967dc3cbd8373561a594" exitCode=0 Jul 06 14:21:57.556307 ip-10-0-136-25 kubenswrapper[2569]: I0706 14:21:57.556097 2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-mt-2b745686/e2e-lifecycle-model-2b745686-kserve-7ffdf54555-gh4bs" Jul 06 14:21:57.556307 ip-10-0-136-25 kubenswrapper[2569]: I0706 14:21:57.556129 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-mt-2b745686/e2e-lifecycle-model-2b745686-kserve-7ffdf54555-gh4bs" event={"ID":"39ce5a2a-6220-4b39-b075-e4ca9d8d1071","Type":"ContainerDied","Data":"420688d5077e63cb381725cf038f7e764516c2a6d735967dc3cbd8373561a594"} Jul 06 14:21:57.556307 ip-10-0-136-25 kubenswrapper[2569]: I0706 14:21:57.556174 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-mt-2b745686/e2e-lifecycle-model-2b745686-kserve-7ffdf54555-gh4bs" event={"ID":"39ce5a2a-6220-4b39-b075-e4ca9d8d1071","Type":"ContainerDied","Data":"6d7ea9e85efb717144eac48fef5cfe82c7456134ce1bf7849b031d3baa622769"} Jul 06 14:21:57.556307 ip-10-0-136-25 kubenswrapper[2569]: I0706 14:21:57.556194 2569 scope.go:117] "RemoveContainer" containerID="420688d5077e63cb381725cf038f7e764516c2a6d735967dc3cbd8373561a594" Jul 06 14:21:57.564835 ip-10-0-136-25 kubenswrapper[2569]: I0706 14:21:57.564818 2569 scope.go:117] "RemoveContainer" containerID="420688d5077e63cb381725cf038f7e764516c2a6d735967dc3cbd8373561a594" Jul 06 14:21:57.565101 ip-10-0-136-25 kubenswrapper[2569]: E0706 14:21:57.565083 2569 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"420688d5077e63cb381725cf038f7e764516c2a6d735967dc3cbd8373561a594\": container with ID starting with 420688d5077e63cb381725cf038f7e764516c2a6d735967dc3cbd8373561a594 not found: ID does not exist" containerID="420688d5077e63cb381725cf038f7e764516c2a6d735967dc3cbd8373561a594" Jul 06 14:21:57.565154 ip-10-0-136-25 kubenswrapper[2569]: I0706 14:21:57.565112 2569 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"420688d5077e63cb381725cf038f7e764516c2a6d735967dc3cbd8373561a594"} err="failed to get container status \"420688d5077e63cb381725cf038f7e764516c2a6d735967dc3cbd8373561a594\": rpc error: code = NotFound desc = could not find container \"420688d5077e63cb381725cf038f7e764516c2a6d735967dc3cbd8373561a594\": container with ID starting with 420688d5077e63cb381725cf038f7e764516c2a6d735967dc3cbd8373561a594 not found: ID does not exist" Jul 06 14:21:57.571604 ip-10-0-136-25 kubenswrapper[2569]: I0706 14:21:57.571578 2569 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-mt-2b745686/e2e-lifecycle-model-2b745686-kserve-7ffdf54555-gh4bs"] Jul 06 14:21:57.577012 ip-10-0-136-25 kubenswrapper[2569]: I0706 14:21:57.576989 2569 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-mt-2b745686/e2e-lifecycle-model-2b745686-kserve-7ffdf54555-gh4bs"] Jul 06 14:21:58.940617 ip-10-0-136-25 kubenswrapper[2569]: I0706 14:21:58.940583 2569 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="39ce5a2a-6220-4b39-b075-e4ca9d8d1071" path="/var/lib/kubelet/pods/39ce5a2a-6220-4b39-b075-e4ca9d8d1071/volumes" Jul 06 14:22:21.144800 ip-10-0-136-25 kubenswrapper[2569]: I0706 14:22:21.144763 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-g9qp6"] Jul 06 14:22:35.442193 ip-10-0-136-25 kubenswrapper[2569]: I0706 14:22:35.442159 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-g9qp6"] Jul 06 14:22:46.529876 ip-10-0-136-25 kubenswrapper[2569]: I0706 14:22:46.529832 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-g9qp6"] Jul 06 14:23:36.835648 ip-10-0-136-25 kubenswrapper[2569]: I0706 14:23:36.835553 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-g9qp6"] Jul 06 14:23:45.432470 ip-10-0-136-25 kubenswrapper[2569]: I0706 14:23:45.432435 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-g9qp6"] Jul 06 14:23:59.034528 ip-10-0-136-25 kubenswrapper[2569]: I0706 14:23:59.034489 2569 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-inf-a-2b44a1/test-model-2b44a1-kserve-7677d7f497-8t58d"] Jul 06 14:23:59.034905 ip-10-0-136-25 kubenswrapper[2569]: I0706 14:23:59.034859 2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="39ce5a2a-6220-4b39-b075-e4ca9d8d1071" containerName="main" Jul 06 14:23:59.034905 ip-10-0-136-25 kubenswrapper[2569]: I0706 14:23:59.034872 2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="39ce5a2a-6220-4b39-b075-e4ca9d8d1071" containerName="main" Jul 06 14:23:59.034976 ip-10-0-136-25 kubenswrapper[2569]: I0706 14:23:59.034949 2569 memory_manager.go:368] "RemoveStaleState removing state" podUID="39ce5a2a-6220-4b39-b075-e4ca9d8d1071" containerName="main" Jul 06 14:23:59.037859 ip-10-0-136-25 kubenswrapper[2569]: I0706 14:23:59.037843 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-2b44a1/test-model-2b44a1-kserve-7677d7f497-8t58d" Jul 06 14:23:59.040479 ip-10-0-136-25 kubenswrapper[2569]: I0706 14:23:59.040454 2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-a-2b44a1\"/\"openshift-service-ca.crt\"" Jul 06 14:23:59.041248 ip-10-0-136-25 kubenswrapper[2569]: I0706 14:23:59.041229 2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-a-2b44a1\"/\"kube-root-ca.crt\"" Jul 06 14:23:59.041382 ip-10-0-136-25 kubenswrapper[2569]: I0706 14:23:59.041229 2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-a-2b44a1\"/\"test-model-2b44a1-kserve-self-signed-certs\"" Jul 06 14:23:59.041382 ip-10-0-136-25 kubenswrapper[2569]: I0706 14:23:59.041360 2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-a-2b44a1\"/\"default-dockercfg-bw2w6\"" Jul 06 14:23:59.045838 ip-10-0-136-25 kubenswrapper[2569]: I0706 14:23:59.045816 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-a-2b44a1/test-model-2b44a1-kserve-7677d7f497-8t58d"] Jul 06 14:23:59.112060 ip-10-0-136-25 kubenswrapper[2569]: I0706 14:23:59.112024 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f084a079-fcb5-4320-b1fb-67f1d6e34bba-home\") pod \"test-model-2b44a1-kserve-7677d7f497-8t58d\" (UID: \"f084a079-fcb5-4320-b1fb-67f1d6e34bba\") " pod="ai-tenant-e2e-inf-a-2b44a1/test-model-2b44a1-kserve-7677d7f497-8t58d" Jul 06 14:23:59.112278 ip-10-0-136-25 kubenswrapper[2569]: I0706 14:23:59.112083 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f084a079-fcb5-4320-b1fb-67f1d6e34bba-model-cache\") pod \"test-model-2b44a1-kserve-7677d7f497-8t58d\" (UID: \"f084a079-fcb5-4320-b1fb-67f1d6e34bba\") " pod="ai-tenant-e2e-inf-a-2b44a1/test-model-2b44a1-kserve-7677d7f497-8t58d" Jul 06 14:23:59.112278 ip-10-0-136-25 kubenswrapper[2569]: I0706 14:23:59.112152 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f084a079-fcb5-4320-b1fb-67f1d6e34bba-tls-certs\") pod \"test-model-2b44a1-kserve-7677d7f497-8t58d\" (UID: \"f084a079-fcb5-4320-b1fb-67f1d6e34bba\") " pod="ai-tenant-e2e-inf-a-2b44a1/test-model-2b44a1-kserve-7677d7f497-8t58d" Jul 06 14:23:59.112278 ip-10-0-136-25 kubenswrapper[2569]: I0706 14:23:59.112185 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f084a079-fcb5-4320-b1fb-67f1d6e34bba-tmp-dir\") pod \"test-model-2b44a1-kserve-7677d7f497-8t58d\" (UID: \"f084a079-fcb5-4320-b1fb-67f1d6e34bba\") " pod="ai-tenant-e2e-inf-a-2b44a1/test-model-2b44a1-kserve-7677d7f497-8t58d" Jul 06 14:23:59.112278 ip-10-0-136-25 kubenswrapper[2569]: I0706 14:23:59.112206 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gs2wt\" (UniqueName: \"kubernetes.io/projected/f084a079-fcb5-4320-b1fb-67f1d6e34bba-kube-api-access-gs2wt\") pod \"test-model-2b44a1-kserve-7677d7f497-8t58d\" (UID: \"f084a079-fcb5-4320-b1fb-67f1d6e34bba\") " pod="ai-tenant-e2e-inf-a-2b44a1/test-model-2b44a1-kserve-7677d7f497-8t58d" Jul 06 14:23:59.112422 ip-10-0-136-25 kubenswrapper[2569]: I0706 14:23:59.112310 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f084a079-fcb5-4320-b1fb-67f1d6e34bba-dshm\") pod \"test-model-2b44a1-kserve-7677d7f497-8t58d\" (UID: \"f084a079-fcb5-4320-b1fb-67f1d6e34bba\") " pod="ai-tenant-e2e-inf-a-2b44a1/test-model-2b44a1-kserve-7677d7f497-8t58d" Jul 06 14:23:59.213595 ip-10-0-136-25 kubenswrapper[2569]: I0706 14:23:59.213548 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f084a079-fcb5-4320-b1fb-67f1d6e34bba-dshm\") pod \"test-model-2b44a1-kserve-7677d7f497-8t58d\" (UID: \"f084a079-fcb5-4320-b1fb-67f1d6e34bba\") " pod="ai-tenant-e2e-inf-a-2b44a1/test-model-2b44a1-kserve-7677d7f497-8t58d" Jul 06 14:23:59.213781 ip-10-0-136-25 kubenswrapper[2569]: I0706 14:23:59.213685 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f084a079-fcb5-4320-b1fb-67f1d6e34bba-home\") pod \"test-model-2b44a1-kserve-7677d7f497-8t58d\" (UID: \"f084a079-fcb5-4320-b1fb-67f1d6e34bba\") " pod="ai-tenant-e2e-inf-a-2b44a1/test-model-2b44a1-kserve-7677d7f497-8t58d" Jul 06 14:23:59.213781 ip-10-0-136-25 kubenswrapper[2569]: I0706 14:23:59.213722 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f084a079-fcb5-4320-b1fb-67f1d6e34bba-model-cache\") pod \"test-model-2b44a1-kserve-7677d7f497-8t58d\" (UID: \"f084a079-fcb5-4320-b1fb-67f1d6e34bba\") " pod="ai-tenant-e2e-inf-a-2b44a1/test-model-2b44a1-kserve-7677d7f497-8t58d" Jul 06 14:23:59.213781 ip-10-0-136-25 kubenswrapper[2569]: I0706 14:23:59.213747 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f084a079-fcb5-4320-b1fb-67f1d6e34bba-tls-certs\") pod \"test-model-2b44a1-kserve-7677d7f497-8t58d\" (UID: \"f084a079-fcb5-4320-b1fb-67f1d6e34bba\") " pod="ai-tenant-e2e-inf-a-2b44a1/test-model-2b44a1-kserve-7677d7f497-8t58d" Jul 06 14:23:59.213945 ip-10-0-136-25 kubenswrapper[2569]: I0706 14:23:59.213873 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f084a079-fcb5-4320-b1fb-67f1d6e34bba-tmp-dir\") pod \"test-model-2b44a1-kserve-7677d7f497-8t58d\" (UID: \"f084a079-fcb5-4320-b1fb-67f1d6e34bba\") " pod="ai-tenant-e2e-inf-a-2b44a1/test-model-2b44a1-kserve-7677d7f497-8t58d" Jul 06 14:23:59.213945 ip-10-0-136-25 kubenswrapper[2569]: I0706 14:23:59.213902 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gs2wt\" (UniqueName: \"kubernetes.io/projected/f084a079-fcb5-4320-b1fb-67f1d6e34bba-kube-api-access-gs2wt\") pod \"test-model-2b44a1-kserve-7677d7f497-8t58d\" (UID: \"f084a079-fcb5-4320-b1fb-67f1d6e34bba\") " pod="ai-tenant-e2e-inf-a-2b44a1/test-model-2b44a1-kserve-7677d7f497-8t58d" Jul 06 14:23:59.214198 ip-10-0-136-25 kubenswrapper[2569]: I0706 14:23:59.214171 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f084a079-fcb5-4320-b1fb-67f1d6e34bba-home\") pod \"test-model-2b44a1-kserve-7677d7f497-8t58d\" (UID: \"f084a079-fcb5-4320-b1fb-67f1d6e34bba\") " pod="ai-tenant-e2e-inf-a-2b44a1/test-model-2b44a1-kserve-7677d7f497-8t58d" Jul 06 14:23:59.214356 ip-10-0-136-25 kubenswrapper[2569]: I0706 14:23:59.214204 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f084a079-fcb5-4320-b1fb-67f1d6e34bba-tmp-dir\") pod \"test-model-2b44a1-kserve-7677d7f497-8t58d\" (UID: \"f084a079-fcb5-4320-b1fb-67f1d6e34bba\") " pod="ai-tenant-e2e-inf-a-2b44a1/test-model-2b44a1-kserve-7677d7f497-8t58d" Jul 06 14:23:59.214356 ip-10-0-136-25 kubenswrapper[2569]: I0706 14:23:59.214305 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f084a079-fcb5-4320-b1fb-67f1d6e34bba-model-cache\") pod \"test-model-2b44a1-kserve-7677d7f497-8t58d\" (UID: \"f084a079-fcb5-4320-b1fb-67f1d6e34bba\") " pod="ai-tenant-e2e-inf-a-2b44a1/test-model-2b44a1-kserve-7677d7f497-8t58d" Jul 06 14:23:59.216003 ip-10-0-136-25 kubenswrapper[2569]: I0706 14:23:59.215976 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f084a079-fcb5-4320-b1fb-67f1d6e34bba-dshm\") pod \"test-model-2b44a1-kserve-7677d7f497-8t58d\" (UID: \"f084a079-fcb5-4320-b1fb-67f1d6e34bba\") " pod="ai-tenant-e2e-inf-a-2b44a1/test-model-2b44a1-kserve-7677d7f497-8t58d" Jul 06 14:23:59.216123 ip-10-0-136-25 kubenswrapper[2569]: I0706 14:23:59.216112 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f084a079-fcb5-4320-b1fb-67f1d6e34bba-tls-certs\") pod \"test-model-2b44a1-kserve-7677d7f497-8t58d\" (UID: \"f084a079-fcb5-4320-b1fb-67f1d6e34bba\") " pod="ai-tenant-e2e-inf-a-2b44a1/test-model-2b44a1-kserve-7677d7f497-8t58d" Jul 06 14:23:59.222732 ip-10-0-136-25 kubenswrapper[2569]: I0706 14:23:59.222707 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gs2wt\" (UniqueName: \"kubernetes.io/projected/f084a079-fcb5-4320-b1fb-67f1d6e34bba-kube-api-access-gs2wt\") pod \"test-model-2b44a1-kserve-7677d7f497-8t58d\" (UID: \"f084a079-fcb5-4320-b1fb-67f1d6e34bba\") " pod="ai-tenant-e2e-inf-a-2b44a1/test-model-2b44a1-kserve-7677d7f497-8t58d" Jul 06 14:23:59.348011 ip-10-0-136-25 kubenswrapper[2569]: I0706 14:23:59.347907 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-2b44a1/test-model-2b44a1-kserve-7677d7f497-8t58d" Jul 06 14:23:59.488690 ip-10-0-136-25 kubenswrapper[2569]: I0706 14:23:59.488658 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-a-2b44a1/test-model-2b44a1-kserve-7677d7f497-8t58d"] Jul 06 14:23:59.491644 ip-10-0-136-25 kubenswrapper[2569]: W0706 14:23:59.491613 2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podf084a079_fcb5_4320_b1fb_67f1d6e34bba.slice/crio-9c590608d777c42d5f9897a8bfcaf40db062918f8995b1b4950a120a214b59d1 WatchSource:0}: Error finding container 9c590608d777c42d5f9897a8bfcaf40db062918f8995b1b4950a120a214b59d1: Status 404 returned error can't find the container with id 9c590608d777c42d5f9897a8bfcaf40db062918f8995b1b4950a120a214b59d1 Jul 06 14:24:00.130138 ip-10-0-136-25 kubenswrapper[2569]: I0706 14:24:00.130098 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-2b44a1/test-model-2b44a1-kserve-7677d7f497-8t58d" event={"ID":"f084a079-fcb5-4320-b1fb-67f1d6e34bba","Type":"ContainerStarted","Data":"b1dcb0f0d65edb62483342e4bd582951656869c746583796f47631d4bb939d2f"} Jul 06 14:24:00.130557 ip-10-0-136-25 kubenswrapper[2569]: I0706 14:24:00.130149 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-2b44a1/test-model-2b44a1-kserve-7677d7f497-8t58d" event={"ID":"f084a079-fcb5-4320-b1fb-67f1d6e34bba","Type":"ContainerStarted","Data":"9c590608d777c42d5f9897a8bfcaf40db062918f8995b1b4950a120a214b59d1"} Jul 06 14:24:00.147112 ip-10-0-136-25 kubenswrapper[2569]: I0706 14:24:00.147062 2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-inf-a-2b44a1/test-model-2b44a1-kserve-7677d7f497-8t58d" podStartSLOduration=1.147043795 podStartE2EDuration="1.147043795s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-06 14:23:59 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 14:24:00.146497118 +0000 UTC m=+2015.737058912" watchObservedRunningTime="2026-07-06 14:24:00.147043795 +0000 UTC m=+2015.737605572" Jul 06 14:24:02.944350 ip-10-0-136-25 kubenswrapper[2569]: I0706 14:24:02.944318 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-g9qp6"] Jul 06 14:24:09.348051 ip-10-0-136-25 kubenswrapper[2569]: I0706 14:24:09.348011 2569 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-inf-a-2b44a1/test-model-2b44a1-kserve-7677d7f497-8t58d" Jul 06 14:24:09.348051 ip-10-0-136-25 kubenswrapper[2569]: I0706 14:24:09.348058 2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-inf-a-2b44a1/test-model-2b44a1-kserve-7677d7f497-8t58d" Jul 06 14:24:09.360566 ip-10-0-136-25 kubenswrapper[2569]: I0706 14:24:09.360541 2569 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-inf-a-2b44a1/test-model-2b44a1-kserve-7677d7f497-8t58d" Jul 06 14:24:10.195747 ip-10-0-136-25 kubenswrapper[2569]: I0706 14:24:10.195713 2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-inf-a-2b44a1/test-model-2b44a1-kserve-7677d7f497-8t58d" Jul 06 14:24:23.038806 ip-10-0-136-25 kubenswrapper[2569]: I0706 14:24:23.038769 2569 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-inf-b-198d8c/test-model-198d8c-kserve-85bd467bf9-pdvgh"] Jul 06 14:24:23.042063 ip-10-0-136-25 kubenswrapper[2569]: I0706 14:24:23.042035 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-198d8c/test-model-198d8c-kserve-85bd467bf9-pdvgh" Jul 06 14:24:23.044845 ip-10-0-136-25 kubenswrapper[2569]: I0706 14:24:23.044820 2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-b-198d8c\"/\"kube-root-ca.crt\"" Jul 06 14:24:23.044845 ip-10-0-136-25 kubenswrapper[2569]: I0706 14:24:23.044838 2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-b-198d8c\"/\"openshift-service-ca.crt\"" Jul 06 14:24:23.045039 ip-10-0-136-25 kubenswrapper[2569]: I0706 14:24:23.044820 2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-b-198d8c\"/\"default-dockercfg-7kvfq\"" Jul 06 14:24:23.045039 ip-10-0-136-25 kubenswrapper[2569]: I0706 14:24:23.044838 2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-b-198d8c\"/\"test-model-198d8c-kserve-self-signed-certs\"" Jul 06 14:24:23.052104 ip-10-0-136-25 kubenswrapper[2569]: I0706 14:24:23.052075 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-b-198d8c/test-model-198d8c-kserve-85bd467bf9-pdvgh"] Jul 06 14:24:23.130218 ip-10-0-136-25 kubenswrapper[2569]: I0706 14:24:23.130183 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/fbdc2f37-2483-42f1-8392-b24e042f346b-tls-certs\") pod \"test-model-198d8c-kserve-85bd467bf9-pdvgh\" (UID: \"fbdc2f37-2483-42f1-8392-b24e042f346b\") " pod="ai-tenant-e2e-inf-b-198d8c/test-model-198d8c-kserve-85bd467bf9-pdvgh" Jul 06 14:24:23.130396 ip-10-0-136-25 kubenswrapper[2569]: I0706 14:24:23.130307 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/fbdc2f37-2483-42f1-8392-b24e042f346b-model-cache\") pod \"test-model-198d8c-kserve-85bd467bf9-pdvgh\" (UID: \"fbdc2f37-2483-42f1-8392-b24e042f346b\") " pod="ai-tenant-e2e-inf-b-198d8c/test-model-198d8c-kserve-85bd467bf9-pdvgh" Jul 06 14:24:23.130396 ip-10-0-136-25 kubenswrapper[2569]: I0706 14:24:23.130354 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/fbdc2f37-2483-42f1-8392-b24e042f346b-dshm\") pod \"test-model-198d8c-kserve-85bd467bf9-pdvgh\" (UID: \"fbdc2f37-2483-42f1-8392-b24e042f346b\") " pod="ai-tenant-e2e-inf-b-198d8c/test-model-198d8c-kserve-85bd467bf9-pdvgh" Jul 06 14:24:23.130396 ip-10-0-136-25 kubenswrapper[2569]: I0706 14:24:23.130382 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/fbdc2f37-2483-42f1-8392-b24e042f346b-home\") pod \"test-model-198d8c-kserve-85bd467bf9-pdvgh\" (UID: \"fbdc2f37-2483-42f1-8392-b24e042f346b\") " pod="ai-tenant-e2e-inf-b-198d8c/test-model-198d8c-kserve-85bd467bf9-pdvgh" Jul 06 14:24:23.130579 ip-10-0-136-25 kubenswrapper[2569]: I0706 14:24:23.130403 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wjm9p\" (UniqueName: \"kubernetes.io/projected/fbdc2f37-2483-42f1-8392-b24e042f346b-kube-api-access-wjm9p\") pod \"test-model-198d8c-kserve-85bd467bf9-pdvgh\" (UID: \"fbdc2f37-2483-42f1-8392-b24e042f346b\") " pod="ai-tenant-e2e-inf-b-198d8c/test-model-198d8c-kserve-85bd467bf9-pdvgh" Jul 06 14:24:23.130579 ip-10-0-136-25 kubenswrapper[2569]: I0706 14:24:23.130420 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/fbdc2f37-2483-42f1-8392-b24e042f346b-tmp-dir\") pod \"test-model-198d8c-kserve-85bd467bf9-pdvgh\" (UID: \"fbdc2f37-2483-42f1-8392-b24e042f346b\") " pod="ai-tenant-e2e-inf-b-198d8c/test-model-198d8c-kserve-85bd467bf9-pdvgh" Jul 06 14:24:23.231528 ip-10-0-136-25 kubenswrapper[2569]: I0706 14:24:23.231487 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/fbdc2f37-2483-42f1-8392-b24e042f346b-dshm\") pod \"test-model-198d8c-kserve-85bd467bf9-pdvgh\" (UID: \"fbdc2f37-2483-42f1-8392-b24e042f346b\") " pod="ai-tenant-e2e-inf-b-198d8c/test-model-198d8c-kserve-85bd467bf9-pdvgh" Jul 06 14:24:23.231528 ip-10-0-136-25 kubenswrapper[2569]: I0706 14:24:23.231534 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/fbdc2f37-2483-42f1-8392-b24e042f346b-home\") pod \"test-model-198d8c-kserve-85bd467bf9-pdvgh\" (UID: \"fbdc2f37-2483-42f1-8392-b24e042f346b\") " pod="ai-tenant-e2e-inf-b-198d8c/test-model-198d8c-kserve-85bd467bf9-pdvgh" Jul 06 14:24:23.231806 ip-10-0-136-25 kubenswrapper[2569]: I0706 14:24:23.231558 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wjm9p\" (UniqueName: \"kubernetes.io/projected/fbdc2f37-2483-42f1-8392-b24e042f346b-kube-api-access-wjm9p\") pod \"test-model-198d8c-kserve-85bd467bf9-pdvgh\" (UID: \"fbdc2f37-2483-42f1-8392-b24e042f346b\") " pod="ai-tenant-e2e-inf-b-198d8c/test-model-198d8c-kserve-85bd467bf9-pdvgh" Jul 06 14:24:23.231806 ip-10-0-136-25 kubenswrapper[2569]: I0706 14:24:23.231591 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/fbdc2f37-2483-42f1-8392-b24e042f346b-tmp-dir\") pod \"test-model-198d8c-kserve-85bd467bf9-pdvgh\" (UID: \"fbdc2f37-2483-42f1-8392-b24e042f346b\") " pod="ai-tenant-e2e-inf-b-198d8c/test-model-198d8c-kserve-85bd467bf9-pdvgh" Jul 06 14:24:23.231806 ip-10-0-136-25 kubenswrapper[2569]: I0706 14:24:23.231620 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/fbdc2f37-2483-42f1-8392-b24e042f346b-tls-certs\") pod \"test-model-198d8c-kserve-85bd467bf9-pdvgh\" (UID: \"fbdc2f37-2483-42f1-8392-b24e042f346b\") " pod="ai-tenant-e2e-inf-b-198d8c/test-model-198d8c-kserve-85bd467bf9-pdvgh" Jul 06 14:24:23.231806 ip-10-0-136-25 kubenswrapper[2569]: I0706 14:24:23.231684 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/fbdc2f37-2483-42f1-8392-b24e042f346b-model-cache\") pod \"test-model-198d8c-kserve-85bd467bf9-pdvgh\" (UID: \"fbdc2f37-2483-42f1-8392-b24e042f346b\") " pod="ai-tenant-e2e-inf-b-198d8c/test-model-198d8c-kserve-85bd467bf9-pdvgh" Jul 06 14:24:23.232061 ip-10-0-136-25 kubenswrapper[2569]: I0706 14:24:23.231932 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/fbdc2f37-2483-42f1-8392-b24e042f346b-home\") pod \"test-model-198d8c-kserve-85bd467bf9-pdvgh\" (UID: \"fbdc2f37-2483-42f1-8392-b24e042f346b\") " pod="ai-tenant-e2e-inf-b-198d8c/test-model-198d8c-kserve-85bd467bf9-pdvgh" Jul 06 14:24:23.232061 ip-10-0-136-25 kubenswrapper[2569]: I0706 14:24:23.232017 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/fbdc2f37-2483-42f1-8392-b24e042f346b-tmp-dir\") pod \"test-model-198d8c-kserve-85bd467bf9-pdvgh\" (UID: \"fbdc2f37-2483-42f1-8392-b24e042f346b\") " pod="ai-tenant-e2e-inf-b-198d8c/test-model-198d8c-kserve-85bd467bf9-pdvgh" Jul 06 14:24:23.232132 ip-10-0-136-25 kubenswrapper[2569]: I0706 14:24:23.232111 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/fbdc2f37-2483-42f1-8392-b24e042f346b-model-cache\") pod \"test-model-198d8c-kserve-85bd467bf9-pdvgh\" (UID: \"fbdc2f37-2483-42f1-8392-b24e042f346b\") " pod="ai-tenant-e2e-inf-b-198d8c/test-model-198d8c-kserve-85bd467bf9-pdvgh" Jul 06 14:24:23.233958 ip-10-0-136-25 kubenswrapper[2569]: I0706 14:24:23.233935 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/fbdc2f37-2483-42f1-8392-b24e042f346b-dshm\") pod \"test-model-198d8c-kserve-85bd467bf9-pdvgh\" (UID: \"fbdc2f37-2483-42f1-8392-b24e042f346b\") " pod="ai-tenant-e2e-inf-b-198d8c/test-model-198d8c-kserve-85bd467bf9-pdvgh" Jul 06 14:24:23.234201 ip-10-0-136-25 kubenswrapper[2569]: I0706 14:24:23.234184 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/fbdc2f37-2483-42f1-8392-b24e042f346b-tls-certs\") pod \"test-model-198d8c-kserve-85bd467bf9-pdvgh\" (UID: \"fbdc2f37-2483-42f1-8392-b24e042f346b\") " pod="ai-tenant-e2e-inf-b-198d8c/test-model-198d8c-kserve-85bd467bf9-pdvgh" Jul 06 14:24:23.239868 ip-10-0-136-25 kubenswrapper[2569]: I0706 14:24:23.239840 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wjm9p\" (UniqueName: \"kubernetes.io/projected/fbdc2f37-2483-42f1-8392-b24e042f346b-kube-api-access-wjm9p\") pod \"test-model-198d8c-kserve-85bd467bf9-pdvgh\" (UID: \"fbdc2f37-2483-42f1-8392-b24e042f346b\") " pod="ai-tenant-e2e-inf-b-198d8c/test-model-198d8c-kserve-85bd467bf9-pdvgh" Jul 06 14:24:23.352943 ip-10-0-136-25 kubenswrapper[2569]: I0706 14:24:23.352839 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-198d8c/test-model-198d8c-kserve-85bd467bf9-pdvgh" Jul 06 14:24:23.489734 ip-10-0-136-25 kubenswrapper[2569]: I0706 14:24:23.489708 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-b-198d8c/test-model-198d8c-kserve-85bd467bf9-pdvgh"] Jul 06 14:24:23.491939 ip-10-0-136-25 kubenswrapper[2569]: W0706 14:24:23.491911 2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podfbdc2f37_2483_42f1_8392_b24e042f346b.slice/crio-65f0cd1476898f030e23cc4efa8b4d329a1b3298e16d6a24b6fa7332412dbdbd WatchSource:0}: Error finding container 65f0cd1476898f030e23cc4efa8b4d329a1b3298e16d6a24b6fa7332412dbdbd: Status 404 returned error can't find the container with id 65f0cd1476898f030e23cc4efa8b4d329a1b3298e16d6a24b6fa7332412dbdbd Jul 06 14:24:24.254103 ip-10-0-136-25 kubenswrapper[2569]: I0706 14:24:24.254060 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-198d8c/test-model-198d8c-kserve-85bd467bf9-pdvgh" event={"ID":"fbdc2f37-2483-42f1-8392-b24e042f346b","Type":"ContainerStarted","Data":"0fda2b463870cb6dd079d40022d216dcca5c4cff3fb261e70ccbef1509380a38"} Jul 06 14:24:24.254103 ip-10-0-136-25 kubenswrapper[2569]: I0706 14:24:24.254115 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-198d8c/test-model-198d8c-kserve-85bd467bf9-pdvgh" event={"ID":"fbdc2f37-2483-42f1-8392-b24e042f346b","Type":"ContainerStarted","Data":"65f0cd1476898f030e23cc4efa8b4d329a1b3298e16d6a24b6fa7332412dbdbd"} Jul 06 14:24:24.271296 ip-10-0-136-25 kubenswrapper[2569]: I0706 14:24:24.271240 2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-inf-b-198d8c/test-model-198d8c-kserve-85bd467bf9-pdvgh" podStartSLOduration=1.271194971 podStartE2EDuration="1.271194971s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-06 14:24:23 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 14:24:24.270039124 +0000 UTC m=+2039.860600931" watchObservedRunningTime="2026-07-06 14:24:24.271194971 +0000 UTC m=+2039.861756747" Jul 06 14:24:26.944589 ip-10-0-136-25 kubenswrapper[2569]: I0706 14:24:26.944549 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-g9qp6"] Jul 06 14:24:33.358096 ip-10-0-136-25 kubenswrapper[2569]: I0706 14:24:33.358051 2569 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-inf-b-198d8c/test-model-198d8c-kserve-85bd467bf9-pdvgh" Jul 06 14:24:33.358625 ip-10-0-136-25 kubenswrapper[2569]: I0706 14:24:33.358196 2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-inf-b-198d8c/test-model-198d8c-kserve-85bd467bf9-pdvgh" Jul 06 14:24:33.375882 ip-10-0-136-25 kubenswrapper[2569]: I0706 14:24:33.375821 2569 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-inf-b-198d8c/test-model-198d8c-kserve-85bd467bf9-pdvgh" Jul 06 14:24:34.311317 ip-10-0-136-25 kubenswrapper[2569]: I0706 14:24:34.311289 2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-inf-b-198d8c/test-model-198d8c-kserve-85bd467bf9-pdvgh" Jul 06 14:24:39.140262 ip-10-0-136-25 kubenswrapper[2569]: I0706 14:24:39.140221 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-g9qp6"] Jul 06 14:24:42.632739 ip-10-0-136-25 kubenswrapper[2569]: I0706 14:24:42.632700 2569 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-a-2b44a1/test-model-2b44a1-kserve-7677d7f497-8t58d"] Jul 06 14:24:43.548300 ip-10-0-136-25 kubenswrapper[2569]: I0706 14:24:43.548261 2569 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-b-198d8c/test-model-198d8c-kserve-85bd467bf9-pdvgh"] Jul 06 14:24:55.537425 ip-10-0-136-25 kubenswrapper[2569]: I0706 14:24:55.537331 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-g9qp6"] Jul 06 14:24:57.653368 ip-10-0-136-25 kubenswrapper[2569]: I0706 14:24:57.653293 2569 kuberuntime_container.go:872] "Killing container with a grace period" pod="ai-tenant-e2e-inf-a-2b44a1/test-model-2b44a1-kserve-7677d7f497-8t58d" podUID="f084a079-fcb5-4320-b1fb-67f1d6e34bba" containerName="main" containerID="cri-o://b1dcb0f0d65edb62483342e4bd582951656869c746583796f47631d4bb939d2f" gracePeriod=45 Jul 06 14:24:57.913092 ip-10-0-136-25 kubenswrapper[2569]: I0706 14:24:57.913029 2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-2b44a1/test-model-2b44a1-kserve-7677d7f497-8t58d" Jul 06 14:24:58.063910 ip-10-0-136-25 kubenswrapper[2569]: I0706 14:24:58.063870 2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f084a079-fcb5-4320-b1fb-67f1d6e34bba-tmp-dir\") pod \"f084a079-fcb5-4320-b1fb-67f1d6e34bba\" (UID: \"f084a079-fcb5-4320-b1fb-67f1d6e34bba\") " Jul 06 14:24:58.064238 ip-10-0-136-25 kubenswrapper[2569]: I0706 14:24:58.063956 2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f084a079-fcb5-4320-b1fb-67f1d6e34bba-home\") pod \"f084a079-fcb5-4320-b1fb-67f1d6e34bba\" (UID: \"f084a079-fcb5-4320-b1fb-67f1d6e34bba\") " Jul 06 14:24:58.064238 ip-10-0-136-25 kubenswrapper[2569]: I0706 14:24:58.063984 2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f084a079-fcb5-4320-b1fb-67f1d6e34bba-model-cache\") pod \"f084a079-fcb5-4320-b1fb-67f1d6e34bba\" (UID: \"f084a079-fcb5-4320-b1fb-67f1d6e34bba\") " Jul 06 14:24:58.064238 ip-10-0-136-25 kubenswrapper[2569]: I0706 14:24:58.064003 2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f084a079-fcb5-4320-b1fb-67f1d6e34bba-tls-certs\") pod \"f084a079-fcb5-4320-b1fb-67f1d6e34bba\" (UID: \"f084a079-fcb5-4320-b1fb-67f1d6e34bba\") " Jul 06 14:24:58.064238 ip-10-0-136-25 kubenswrapper[2569]: I0706 14:24:58.064029 2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f084a079-fcb5-4320-b1fb-67f1d6e34bba-dshm\") pod \"f084a079-fcb5-4320-b1fb-67f1d6e34bba\" (UID: \"f084a079-fcb5-4320-b1fb-67f1d6e34bba\") " Jul 06 14:24:58.064238 ip-10-0-136-25 kubenswrapper[2569]: I0706 14:24:58.064059 2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-gs2wt\" (UniqueName: \"kubernetes.io/projected/f084a079-fcb5-4320-b1fb-67f1d6e34bba-kube-api-access-gs2wt\") pod \"f084a079-fcb5-4320-b1fb-67f1d6e34bba\" (UID: \"f084a079-fcb5-4320-b1fb-67f1d6e34bba\") " Jul 06 14:24:58.064238 ip-10-0-136-25 kubenswrapper[2569]: I0706 14:24:58.064164 2569 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f084a079-fcb5-4320-b1fb-67f1d6e34bba-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "f084a079-fcb5-4320-b1fb-67f1d6e34bba" (UID: "f084a079-fcb5-4320-b1fb-67f1d6e34bba"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 06 14:24:58.064566 ip-10-0-136-25 kubenswrapper[2569]: I0706 14:24:58.064304 2569 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f084a079-fcb5-4320-b1fb-67f1d6e34bba-model-cache" (OuterVolumeSpecName: "model-cache") pod "f084a079-fcb5-4320-b1fb-67f1d6e34bba" (UID: "f084a079-fcb5-4320-b1fb-67f1d6e34bba"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 06 14:24:58.064566 ip-10-0-136-25 kubenswrapper[2569]: I0706 14:24:58.064330 2569 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f084a079-fcb5-4320-b1fb-67f1d6e34bba-home" (OuterVolumeSpecName: "home") pod "f084a079-fcb5-4320-b1fb-67f1d6e34bba" (UID: "f084a079-fcb5-4320-b1fb-67f1d6e34bba"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 06 14:24:58.064566 ip-10-0-136-25 kubenswrapper[2569]: I0706 14:24:58.064527 2569 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f084a079-fcb5-4320-b1fb-67f1d6e34bba-home\") on node \"ip-10-0-136-25.ec2.internal\" DevicePath \"\"" Jul 06 14:24:58.064566 ip-10-0-136-25 kubenswrapper[2569]: I0706 14:24:58.064541 2569 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f084a079-fcb5-4320-b1fb-67f1d6e34bba-model-cache\") on node \"ip-10-0-136-25.ec2.internal\" DevicePath \"\"" Jul 06 14:24:58.064566 ip-10-0-136-25 kubenswrapper[2569]: I0706 14:24:58.064552 2569 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f084a079-fcb5-4320-b1fb-67f1d6e34bba-tmp-dir\") on node \"ip-10-0-136-25.ec2.internal\" DevicePath \"\"" Jul 06 14:24:58.066373 ip-10-0-136-25 kubenswrapper[2569]: I0706 14:24:58.066343 2569 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f084a079-fcb5-4320-b1fb-67f1d6e34bba-dshm" (OuterVolumeSpecName: "dshm") pod "f084a079-fcb5-4320-b1fb-67f1d6e34bba" (UID: "f084a079-fcb5-4320-b1fb-67f1d6e34bba"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 06 14:24:58.066470 ip-10-0-136-25 kubenswrapper[2569]: I0706 14:24:58.066398 2569 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f084a079-fcb5-4320-b1fb-67f1d6e34bba-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "f084a079-fcb5-4320-b1fb-67f1d6e34bba" (UID: "f084a079-fcb5-4320-b1fb-67f1d6e34bba"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 06 14:24:58.066518 ip-10-0-136-25 kubenswrapper[2569]: I0706 14:24:58.066500 2569 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f084a079-fcb5-4320-b1fb-67f1d6e34bba-kube-api-access-gs2wt" (OuterVolumeSpecName: "kube-api-access-gs2wt") pod "f084a079-fcb5-4320-b1fb-67f1d6e34bba" (UID: "f084a079-fcb5-4320-b1fb-67f1d6e34bba"). InnerVolumeSpecName "kube-api-access-gs2wt". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 06 14:24:58.165268 ip-10-0-136-25 kubenswrapper[2569]: I0706 14:24:58.165151 2569 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gs2wt\" (UniqueName: \"kubernetes.io/projected/f084a079-fcb5-4320-b1fb-67f1d6e34bba-kube-api-access-gs2wt\") on node \"ip-10-0-136-25.ec2.internal\" DevicePath \"\"" Jul 06 14:24:58.165268 ip-10-0-136-25 kubenswrapper[2569]: I0706 14:24:58.165187 2569 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f084a079-fcb5-4320-b1fb-67f1d6e34bba-tls-certs\") on node \"ip-10-0-136-25.ec2.internal\" DevicePath \"\"" Jul 06 14:24:58.165268 ip-10-0-136-25 kubenswrapper[2569]: I0706 14:24:58.165203 2569 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f084a079-fcb5-4320-b1fb-67f1d6e34bba-dshm\") on node \"ip-10-0-136-25.ec2.internal\" DevicePath \"\"" Jul 06 14:24:58.419679 ip-10-0-136-25 kubenswrapper[2569]: I0706 14:24:58.419584 2569 generic.go:356] "Generic (PLEG): container finished" podID="f084a079-fcb5-4320-b1fb-67f1d6e34bba" containerID="b1dcb0f0d65edb62483342e4bd582951656869c746583796f47631d4bb939d2f" exitCode=0 Jul 06 14:24:58.419679 ip-10-0-136-25 kubenswrapper[2569]: I0706 14:24:58.419622 2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-2b44a1/test-model-2b44a1-kserve-7677d7f497-8t58d" Jul 06 14:24:58.419872 ip-10-0-136-25 kubenswrapper[2569]: I0706 14:24:58.419671 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-2b44a1/test-model-2b44a1-kserve-7677d7f497-8t58d" event={"ID":"f084a079-fcb5-4320-b1fb-67f1d6e34bba","Type":"ContainerDied","Data":"b1dcb0f0d65edb62483342e4bd582951656869c746583796f47631d4bb939d2f"} Jul 06 14:24:58.419872 ip-10-0-136-25 kubenswrapper[2569]: I0706 14:24:58.419717 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-2b44a1/test-model-2b44a1-kserve-7677d7f497-8t58d" event={"ID":"f084a079-fcb5-4320-b1fb-67f1d6e34bba","Type":"ContainerDied","Data":"9c590608d777c42d5f9897a8bfcaf40db062918f8995b1b4950a120a214b59d1"} Jul 06 14:24:58.419872 ip-10-0-136-25 kubenswrapper[2569]: I0706 14:24:58.419735 2569 scope.go:117] "RemoveContainer" containerID="b1dcb0f0d65edb62483342e4bd582951656869c746583796f47631d4bb939d2f" Jul 06 14:24:58.428660 ip-10-0-136-25 kubenswrapper[2569]: I0706 14:24:58.428641 2569 scope.go:117] "RemoveContainer" containerID="b1dcb0f0d65edb62483342e4bd582951656869c746583796f47631d4bb939d2f" Jul 06 14:24:58.428964 ip-10-0-136-25 kubenswrapper[2569]: E0706 14:24:58.428940 2569 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b1dcb0f0d65edb62483342e4bd582951656869c746583796f47631d4bb939d2f\": container with ID starting with b1dcb0f0d65edb62483342e4bd582951656869c746583796f47631d4bb939d2f not found: ID does not exist" containerID="b1dcb0f0d65edb62483342e4bd582951656869c746583796f47631d4bb939d2f" Jul 06 14:24:58.429038 ip-10-0-136-25 kubenswrapper[2569]: I0706 14:24:58.428991 2569 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b1dcb0f0d65edb62483342e4bd582951656869c746583796f47631d4bb939d2f"} err="failed to get container status \"b1dcb0f0d65edb62483342e4bd582951656869c746583796f47631d4bb939d2f\": rpc error: code = NotFound desc = could not find container \"b1dcb0f0d65edb62483342e4bd582951656869c746583796f47631d4bb939d2f\": container with ID starting with b1dcb0f0d65edb62483342e4bd582951656869c746583796f47631d4bb939d2f not found: ID does not exist" Jul 06 14:24:58.438906 ip-10-0-136-25 kubenswrapper[2569]: I0706 14:24:58.438880 2569 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-a-2b44a1/test-model-2b44a1-kserve-7677d7f497-8t58d"] Jul 06 14:24:58.442654 ip-10-0-136-25 kubenswrapper[2569]: I0706 14:24:58.442630 2569 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-inf-a-2b44a1/test-model-2b44a1-kserve-7677d7f497-8t58d"] Jul 06 14:24:58.568877 ip-10-0-136-25 kubenswrapper[2569]: I0706 14:24:58.568814 2569 kuberuntime_container.go:872] "Killing container with a grace period" pod="ai-tenant-e2e-inf-b-198d8c/test-model-198d8c-kserve-85bd467bf9-pdvgh" podUID="fbdc2f37-2483-42f1-8392-b24e042f346b" containerName="main" containerID="cri-o://0fda2b463870cb6dd079d40022d216dcca5c4cff3fb261e70ccbef1509380a38" gracePeriod=45 Jul 06 14:24:58.826588 ip-10-0-136-25 kubenswrapper[2569]: I0706 14:24:58.826560 2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-198d8c/test-model-198d8c-kserve-85bd467bf9-pdvgh" Jul 06 14:24:58.870387 ip-10-0-136-25 kubenswrapper[2569]: I0706 14:24:58.870352 2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/fbdc2f37-2483-42f1-8392-b24e042f346b-model-cache\") pod \"fbdc2f37-2483-42f1-8392-b24e042f346b\" (UID: \"fbdc2f37-2483-42f1-8392-b24e042f346b\") " Jul 06 14:24:58.870594 ip-10-0-136-25 kubenswrapper[2569]: I0706 14:24:58.870398 2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/fbdc2f37-2483-42f1-8392-b24e042f346b-dshm\") pod \"fbdc2f37-2483-42f1-8392-b24e042f346b\" (UID: \"fbdc2f37-2483-42f1-8392-b24e042f346b\") " Jul 06 14:24:58.870594 ip-10-0-136-25 kubenswrapper[2569]: I0706 14:24:58.870439 2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-wjm9p\" (UniqueName: \"kubernetes.io/projected/fbdc2f37-2483-42f1-8392-b24e042f346b-kube-api-access-wjm9p\") pod \"fbdc2f37-2483-42f1-8392-b24e042f346b\" (UID: \"fbdc2f37-2483-42f1-8392-b24e042f346b\") " Jul 06 14:24:58.870594 ip-10-0-136-25 kubenswrapper[2569]: I0706 14:24:58.870478 2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/fbdc2f37-2483-42f1-8392-b24e042f346b-tls-certs\") pod \"fbdc2f37-2483-42f1-8392-b24e042f346b\" (UID: \"fbdc2f37-2483-42f1-8392-b24e042f346b\") " Jul 06 14:24:58.870594 ip-10-0-136-25 kubenswrapper[2569]: I0706 14:24:58.870556 2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/fbdc2f37-2483-42f1-8392-b24e042f346b-tmp-dir\") pod \"fbdc2f37-2483-42f1-8392-b24e042f346b\" (UID: \"fbdc2f37-2483-42f1-8392-b24e042f346b\") " Jul 06 14:24:58.870832 ip-10-0-136-25 kubenswrapper[2569]: I0706 14:24:58.870661 2569 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fbdc2f37-2483-42f1-8392-b24e042f346b-model-cache" (OuterVolumeSpecName: "model-cache") pod "fbdc2f37-2483-42f1-8392-b24e042f346b" (UID: "fbdc2f37-2483-42f1-8392-b24e042f346b"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 06 14:24:58.870902 ip-10-0-136-25 kubenswrapper[2569]: I0706 14:24:58.870871 2569 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/fbdc2f37-2483-42f1-8392-b24e042f346b-model-cache\") on node \"ip-10-0-136-25.ec2.internal\" DevicePath \"\"" Jul 06 14:24:58.870902 ip-10-0-136-25 kubenswrapper[2569]: I0706 14:24:58.870862 2569 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fbdc2f37-2483-42f1-8392-b24e042f346b-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "fbdc2f37-2483-42f1-8392-b24e042f346b" (UID: "fbdc2f37-2483-42f1-8392-b24e042f346b"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 06 14:24:58.872811 ip-10-0-136-25 kubenswrapper[2569]: I0706 14:24:58.872783 2569 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fbdc2f37-2483-42f1-8392-b24e042f346b-dshm" (OuterVolumeSpecName: "dshm") pod "fbdc2f37-2483-42f1-8392-b24e042f346b" (UID: "fbdc2f37-2483-42f1-8392-b24e042f346b"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 06 14:24:58.872922 ip-10-0-136-25 kubenswrapper[2569]: I0706 14:24:58.872814 2569 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/fbdc2f37-2483-42f1-8392-b24e042f346b-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "fbdc2f37-2483-42f1-8392-b24e042f346b" (UID: "fbdc2f37-2483-42f1-8392-b24e042f346b"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 06 14:24:58.872922 ip-10-0-136-25 kubenswrapper[2569]: I0706 14:24:58.872899 2569 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fbdc2f37-2483-42f1-8392-b24e042f346b-kube-api-access-wjm9p" (OuterVolumeSpecName: "kube-api-access-wjm9p") pod "fbdc2f37-2483-42f1-8392-b24e042f346b" (UID: "fbdc2f37-2483-42f1-8392-b24e042f346b"). InnerVolumeSpecName "kube-api-access-wjm9p". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 06 14:24:58.939929 ip-10-0-136-25 kubenswrapper[2569]: I0706 14:24:58.939823 2569 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="f084a079-fcb5-4320-b1fb-67f1d6e34bba" path="/var/lib/kubelet/pods/f084a079-fcb5-4320-b1fb-67f1d6e34bba/volumes" Jul 06 14:24:58.971864 ip-10-0-136-25 kubenswrapper[2569]: I0706 14:24:58.971831 2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/fbdc2f37-2483-42f1-8392-b24e042f346b-home\") pod \"fbdc2f37-2483-42f1-8392-b24e042f346b\" (UID: \"fbdc2f37-2483-42f1-8392-b24e042f346b\") " Jul 06 14:24:58.972111 ip-10-0-136-25 kubenswrapper[2569]: I0706 14:24:58.972088 2569 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fbdc2f37-2483-42f1-8392-b24e042f346b-home" (OuterVolumeSpecName: "home") pod "fbdc2f37-2483-42f1-8392-b24e042f346b" (UID: "fbdc2f37-2483-42f1-8392-b24e042f346b"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 06 14:24:58.972190 ip-10-0-136-25 kubenswrapper[2569]: I0706 14:24:58.972109 2569 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/fbdc2f37-2483-42f1-8392-b24e042f346b-tmp-dir\") on node \"ip-10-0-136-25.ec2.internal\" DevicePath \"\"" Jul 06 14:24:58.972190 ip-10-0-136-25 kubenswrapper[2569]: I0706 14:24:58.972130 2569 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/fbdc2f37-2483-42f1-8392-b24e042f346b-dshm\") on node \"ip-10-0-136-25.ec2.internal\" DevicePath \"\"" Jul 06 14:24:58.972190 ip-10-0-136-25 kubenswrapper[2569]: I0706 14:24:58.972142 2569 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wjm9p\" (UniqueName: \"kubernetes.io/projected/fbdc2f37-2483-42f1-8392-b24e042f346b-kube-api-access-wjm9p\") on node \"ip-10-0-136-25.ec2.internal\" DevicePath \"\"" Jul 06 14:24:58.972190 ip-10-0-136-25 kubenswrapper[2569]: I0706 14:24:58.972158 2569 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/fbdc2f37-2483-42f1-8392-b24e042f346b-tls-certs\") on node \"ip-10-0-136-25.ec2.internal\" DevicePath \"\"" Jul 06 14:24:59.073580 ip-10-0-136-25 kubenswrapper[2569]: I0706 14:24:59.073537 2569 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/fbdc2f37-2483-42f1-8392-b24e042f346b-home\") on node \"ip-10-0-136-25.ec2.internal\" DevicePath \"\"" Jul 06 14:24:59.427524 ip-10-0-136-25 kubenswrapper[2569]: I0706 14:24:59.427492 2569 generic.go:356] "Generic (PLEG): container finished" podID="fbdc2f37-2483-42f1-8392-b24e042f346b" containerID="0fda2b463870cb6dd079d40022d216dcca5c4cff3fb261e70ccbef1509380a38" exitCode=0 Jul 06 14:24:59.427722 ip-10-0-136-25 kubenswrapper[2569]: I0706 14:24:59.427553 2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-198d8c/test-model-198d8c-kserve-85bd467bf9-pdvgh" Jul 06 14:24:59.427722 ip-10-0-136-25 kubenswrapper[2569]: I0706 14:24:59.427583 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-198d8c/test-model-198d8c-kserve-85bd467bf9-pdvgh" event={"ID":"fbdc2f37-2483-42f1-8392-b24e042f346b","Type":"ContainerDied","Data":"0fda2b463870cb6dd079d40022d216dcca5c4cff3fb261e70ccbef1509380a38"} Jul 06 14:24:59.427722 ip-10-0-136-25 kubenswrapper[2569]: I0706 14:24:59.427632 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-198d8c/test-model-198d8c-kserve-85bd467bf9-pdvgh" event={"ID":"fbdc2f37-2483-42f1-8392-b24e042f346b","Type":"ContainerDied","Data":"65f0cd1476898f030e23cc4efa8b4d329a1b3298e16d6a24b6fa7332412dbdbd"} Jul 06 14:24:59.427722 ip-10-0-136-25 kubenswrapper[2569]: I0706 14:24:59.427657 2569 scope.go:117] "RemoveContainer" containerID="0fda2b463870cb6dd079d40022d216dcca5c4cff3fb261e70ccbef1509380a38" Jul 06 14:24:59.436866 ip-10-0-136-25 kubenswrapper[2569]: I0706 14:24:59.436848 2569 scope.go:117] "RemoveContainer" containerID="0fda2b463870cb6dd079d40022d216dcca5c4cff3fb261e70ccbef1509380a38" Jul 06 14:24:59.437192 ip-10-0-136-25 kubenswrapper[2569]: E0706 14:24:59.437173 2569 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0fda2b463870cb6dd079d40022d216dcca5c4cff3fb261e70ccbef1509380a38\": container with ID starting with 0fda2b463870cb6dd079d40022d216dcca5c4cff3fb261e70ccbef1509380a38 not found: ID does not exist" containerID="0fda2b463870cb6dd079d40022d216dcca5c4cff3fb261e70ccbef1509380a38" Jul 06 14:24:59.437376 ip-10-0-136-25 kubenswrapper[2569]: I0706 14:24:59.437201 2569 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0fda2b463870cb6dd079d40022d216dcca5c4cff3fb261e70ccbef1509380a38"} err="failed to get container status \"0fda2b463870cb6dd079d40022d216dcca5c4cff3fb261e70ccbef1509380a38\": rpc error: code = NotFound desc = could not find container \"0fda2b463870cb6dd079d40022d216dcca5c4cff3fb261e70ccbef1509380a38\": container with ID starting with 0fda2b463870cb6dd079d40022d216dcca5c4cff3fb261e70ccbef1509380a38 not found: ID does not exist" Jul 06 14:24:59.447300 ip-10-0-136-25 kubenswrapper[2569]: I0706 14:24:59.447273 2569 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-b-198d8c/test-model-198d8c-kserve-85bd467bf9-pdvgh"] Jul 06 14:24:59.453078 ip-10-0-136-25 kubenswrapper[2569]: I0706 14:24:59.453056 2569 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-inf-b-198d8c/test-model-198d8c-kserve-85bd467bf9-pdvgh"] Jul 06 14:25:00.939477 ip-10-0-136-25 kubenswrapper[2569]: I0706 14:25:00.939447 2569 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="fbdc2f37-2483-42f1-8392-b24e042f346b" path="/var/lib/kubelet/pods/fbdc2f37-2483-42f1-8392-b24e042f346b/volumes" Jul 06 14:25:25.005238 ip-10-0-136-25 kubenswrapper[2569]: I0706 14:25:25.005184 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-68b757fb67-z2h95_dfffcdca-94d6-4732-815f-bcb533f538e9/console-operator/2.log" Jul 06 14:25:25.010405 ip-10-0-136-25 kubenswrapper[2569]: I0706 14:25:25.010378 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-68b757fb67-z2h95_dfffcdca-94d6-4732-815f-bcb533f538e9/console-operator/2.log" Jul 06 14:25:29.737621 ip-10-0-136-25 kubenswrapper[2569]: I0706 14:25:29.737575 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-g9qp6"] Jul 06 14:25:42.441132 ip-10-0-136-25 kubenswrapper[2569]: I0706 14:25:42.441096 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-g9qp6"] Jul 06 14:25:54.841443 ip-10-0-136-25 kubenswrapper[2569]: I0706 14:25:54.841409 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-g9qp6"] Jul 06 14:26:04.547283 ip-10-0-136-25 kubenswrapper[2569]: I0706 14:26:04.547237 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-g9qp6"] Jul 06 14:26:17.130425 ip-10-0-136-25 kubenswrapper[2569]: I0706 14:26:17.130341 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-g9qp6"] Jul 06 14:26:25.538379 ip-10-0-136-25 kubenswrapper[2569]: I0706 14:26:25.538342 2569 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/maas-api-e2e-shared-b-eb526a-6cdc9f8d77-kpfm9"] Jul 06 14:26:25.538758 ip-10-0-136-25 kubenswrapper[2569]: I0706 14:26:25.538716 2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="fbdc2f37-2483-42f1-8392-b24e042f346b" containerName="main" Jul 06 14:26:25.538758 ip-10-0-136-25 kubenswrapper[2569]: I0706 14:26:25.538727 2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="fbdc2f37-2483-42f1-8392-b24e042f346b" containerName="main" Jul 06 14:26:25.538758 ip-10-0-136-25 kubenswrapper[2569]: I0706 14:26:25.538737 2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f084a079-fcb5-4320-b1fb-67f1d6e34bba" containerName="main" Jul 06 14:26:25.538758 ip-10-0-136-25 kubenswrapper[2569]: I0706 14:26:25.538746 2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="f084a079-fcb5-4320-b1fb-67f1d6e34bba" containerName="main" Jul 06 14:26:25.538894 ip-10-0-136-25 kubenswrapper[2569]: I0706 14:26:25.538817 2569 memory_manager.go:368] "RemoveStaleState removing state" podUID="f084a079-fcb5-4320-b1fb-67f1d6e34bba" containerName="main" Jul 06 14:26:25.538894 ip-10-0-136-25 kubenswrapper[2569]: I0706 14:26:25.538828 2569 memory_manager.go:368] "RemoveStaleState removing state" podUID="fbdc2f37-2483-42f1-8392-b24e042f346b" containerName="main" Jul 06 14:26:25.541934 ip-10-0-136-25 kubenswrapper[2569]: I0706 14:26:25.541916 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-api-e2e-shared-b-eb526a-6cdc9f8d77-kpfm9" Jul 06 14:26:25.543872 ip-10-0-136-25 kubenswrapper[2569]: I0706 14:26:25.543843 2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-api-e2e-shared-b-eb526a-serving-cert\"" Jul 06 14:26:25.544021 ip-10-0-136-25 kubenswrapper[2569]: I0706 14:26:25.543948 2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-api-dockercfg-dln66\"" Jul 06 14:26:25.551420 ip-10-0-136-25 kubenswrapper[2569]: I0706 14:26:25.551395 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-api-e2e-shared-b-eb526a-6cdc9f8d77-kpfm9"] Jul 06 14:26:25.659264 ip-10-0-136-25 kubenswrapper[2569]: I0706 14:26:25.659227 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-277zr\" (UniqueName: \"kubernetes.io/projected/afcdff00-c419-4f06-b9d4-70d980dc4372-kube-api-access-277zr\") pod \"maas-api-e2e-shared-b-eb526a-6cdc9f8d77-kpfm9\" (UID: \"afcdff00-c419-4f06-b9d4-70d980dc4372\") " pod="opendatahub/maas-api-e2e-shared-b-eb526a-6cdc9f8d77-kpfm9" Jul 06 14:26:25.659454 ip-10-0-136-25 kubenswrapper[2569]: I0706 14:26:25.659315 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/afcdff00-c419-4f06-b9d4-70d980dc4372-maas-api-tls\") pod \"maas-api-e2e-shared-b-eb526a-6cdc9f8d77-kpfm9\" (UID: \"afcdff00-c419-4f06-b9d4-70d980dc4372\") " pod="opendatahub/maas-api-e2e-shared-b-eb526a-6cdc9f8d77-kpfm9" Jul 06 14:26:25.659454 ip-10-0-136-25 kubenswrapper[2569]: I0706 14:26:25.659368 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/afcdff00-c419-4f06-b9d4-70d980dc4372-openshift-service-ca\") pod \"maas-api-e2e-shared-b-eb526a-6cdc9f8d77-kpfm9\" (UID: \"afcdff00-c419-4f06-b9d4-70d980dc4372\") " pod="opendatahub/maas-api-e2e-shared-b-eb526a-6cdc9f8d77-kpfm9" Jul 06 14:26:25.760086 ip-10-0-136-25 kubenswrapper[2569]: I0706 14:26:25.760042 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/afcdff00-c419-4f06-b9d4-70d980dc4372-maas-api-tls\") pod \"maas-api-e2e-shared-b-eb526a-6cdc9f8d77-kpfm9\" (UID: \"afcdff00-c419-4f06-b9d4-70d980dc4372\") " pod="opendatahub/maas-api-e2e-shared-b-eb526a-6cdc9f8d77-kpfm9" Jul 06 14:26:25.760328 ip-10-0-136-25 kubenswrapper[2569]: I0706 14:26:25.760103 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/afcdff00-c419-4f06-b9d4-70d980dc4372-openshift-service-ca\") pod \"maas-api-e2e-shared-b-eb526a-6cdc9f8d77-kpfm9\" (UID: \"afcdff00-c419-4f06-b9d4-70d980dc4372\") " pod="opendatahub/maas-api-e2e-shared-b-eb526a-6cdc9f8d77-kpfm9" Jul 06 14:26:25.760328 ip-10-0-136-25 kubenswrapper[2569]: I0706 14:26:25.760138 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-277zr\" (UniqueName: \"kubernetes.io/projected/afcdff00-c419-4f06-b9d4-70d980dc4372-kube-api-access-277zr\") pod \"maas-api-e2e-shared-b-eb526a-6cdc9f8d77-kpfm9\" (UID: \"afcdff00-c419-4f06-b9d4-70d980dc4372\") " pod="opendatahub/maas-api-e2e-shared-b-eb526a-6cdc9f8d77-kpfm9" Jul 06 14:26:25.760855 ip-10-0-136-25 kubenswrapper[2569]: I0706 14:26:25.760833 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/afcdff00-c419-4f06-b9d4-70d980dc4372-openshift-service-ca\") pod \"maas-api-e2e-shared-b-eb526a-6cdc9f8d77-kpfm9\" (UID: \"afcdff00-c419-4f06-b9d4-70d980dc4372\") " pod="opendatahub/maas-api-e2e-shared-b-eb526a-6cdc9f8d77-kpfm9" Jul 06 14:26:25.762558 ip-10-0-136-25 kubenswrapper[2569]: I0706 14:26:25.762541 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/afcdff00-c419-4f06-b9d4-70d980dc4372-maas-api-tls\") pod \"maas-api-e2e-shared-b-eb526a-6cdc9f8d77-kpfm9\" (UID: \"afcdff00-c419-4f06-b9d4-70d980dc4372\") " pod="opendatahub/maas-api-e2e-shared-b-eb526a-6cdc9f8d77-kpfm9" Jul 06 14:26:25.767691 ip-10-0-136-25 kubenswrapper[2569]: I0706 14:26:25.767668 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-277zr\" (UniqueName: \"kubernetes.io/projected/afcdff00-c419-4f06-b9d4-70d980dc4372-kube-api-access-277zr\") pod \"maas-api-e2e-shared-b-eb526a-6cdc9f8d77-kpfm9\" (UID: \"afcdff00-c419-4f06-b9d4-70d980dc4372\") " pod="opendatahub/maas-api-e2e-shared-b-eb526a-6cdc9f8d77-kpfm9" Jul 06 14:26:25.853102 ip-10-0-136-25 kubenswrapper[2569]: I0706 14:26:25.853004 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-api-e2e-shared-b-eb526a-6cdc9f8d77-kpfm9" Jul 06 14:26:26.191472 ip-10-0-136-25 kubenswrapper[2569]: I0706 14:26:26.191446 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-api-e2e-shared-b-eb526a-6cdc9f8d77-kpfm9"] Jul 06 14:26:26.194130 ip-10-0-136-25 kubenswrapper[2569]: W0706 14:26:26.194099 2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podafcdff00_c419_4f06_b9d4_70d980dc4372.slice/crio-98c8d17221241d7f20086ff1c058f1401ab70c9913e0b0264402d4ff6c1db9b2 WatchSource:0}: Error finding container 98c8d17221241d7f20086ff1c058f1401ab70c9913e0b0264402d4ff6c1db9b2: Status 404 returned error can't find the container with id 98c8d17221241d7f20086ff1c058f1401ab70c9913e0b0264402d4ff6c1db9b2 Jul 06 14:26:26.195889 ip-10-0-136-25 kubenswrapper[2569]: I0706 14:26:26.195872 2569 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Jul 06 14:26:26.834832 ip-10-0-136-25 kubenswrapper[2569]: I0706 14:26:26.834791 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-api-e2e-shared-b-eb526a-6cdc9f8d77-kpfm9" event={"ID":"afcdff00-c419-4f06-b9d4-70d980dc4372","Type":"ContainerStarted","Data":"98c8d17221241d7f20086ff1c058f1401ab70c9913e0b0264402d4ff6c1db9b2"} Jul 06 14:26:28.387234 ip-10-0-136-25 kubenswrapper[2569]: I0706 14:26:28.387187 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-g9qp6"] Jul 06 14:26:28.845666 ip-10-0-136-25 kubenswrapper[2569]: I0706 14:26:28.845624 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-api-e2e-shared-b-eb526a-6cdc9f8d77-kpfm9" event={"ID":"afcdff00-c419-4f06-b9d4-70d980dc4372","Type":"ContainerStarted","Data":"273ec04de628e24fae2e86906e16cde9f7710b72af14b29b366f59e40875a70e"} Jul 06 14:26:28.845902 ip-10-0-136-25 kubenswrapper[2569]: I0706 14:26:28.845884 2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-api-e2e-shared-b-eb526a-6cdc9f8d77-kpfm9" Jul 06 14:26:28.862596 ip-10-0-136-25 kubenswrapper[2569]: I0706 14:26:28.862545 2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/maas-api-e2e-shared-b-eb526a-6cdc9f8d77-kpfm9" podStartSLOduration=1.712688977 podStartE2EDuration="3.862528981s" totalImagesPullingTime="2.149840004s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-06 14:26:25 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 14:26:28.861449312 +0000 UTC m=+2164.452011091" watchObservedRunningTime="2026-07-06 14:26:28.862528981 +0000 UTC m=+2164.453090758" Jul 06 14:26:34.857153 ip-10-0-136-25 kubenswrapper[2569]: I0706 14:26:34.857117 2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/maas-api-e2e-shared-b-eb526a-6cdc9f8d77-kpfm9" Jul 06 14:26:37.246950 ip-10-0-136-25 kubenswrapper[2569]: I0706 14:26:37.246913 2569 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-a-2c86e0/auth-test-model-2c86e0-kserve-84899654f5-kd6k2"] Jul 06 14:26:37.250415 ip-10-0-136-25 kubenswrapper[2569]: I0706 14:26:37.250397 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-2c86e0/auth-test-model-2c86e0-kserve-84899654f5-kd6k2" Jul 06 14:26:37.252388 ip-10-0-136-25 kubenswrapper[2569]: I0706 14:26:37.252361 2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-2c86e0\"/\"openshift-service-ca.crt\"" Jul 06 14:26:37.252388 ip-10-0-136-25 kubenswrapper[2569]: I0706 14:26:37.252383 2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-2c86e0\"/\"kube-root-ca.crt\"" Jul 06 14:26:37.253044 ip-10-0-136-25 kubenswrapper[2569]: I0706 14:26:37.253022 2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-2c86e0\"/\"default-dockercfg-prtvh\"" Jul 06 14:26:37.253153 ip-10-0-136-25 kubenswrapper[2569]: I0706 14:26:37.253064 2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-2c86e0\"/\"auth-test-model-2c86e0-kserve-self-signed-certs\"" Jul 06 14:26:37.260092 ip-10-0-136-25 kubenswrapper[2569]: I0706 14:26:37.260070 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-2c86e0/auth-test-model-2c86e0-kserve-84899654f5-kd6k2"] Jul 06 14:26:37.358278 ip-10-0-136-25 kubenswrapper[2569]: I0706 14:26:37.358204 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gbqxl\" (UniqueName: \"kubernetes.io/projected/967aea57-432f-47ab-a21e-6e1065a5be16-kube-api-access-gbqxl\") pod \"auth-test-model-2c86e0-kserve-84899654f5-kd6k2\" (UID: \"967aea57-432f-47ab-a21e-6e1065a5be16\") " pod="ai-tenant-e2e-shared-a-2c86e0/auth-test-model-2c86e0-kserve-84899654f5-kd6k2" Jul 06 14:26:37.358477 ip-10-0-136-25 kubenswrapper[2569]: I0706 14:26:37.358332 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/967aea57-432f-47ab-a21e-6e1065a5be16-tmp-dir\") pod \"auth-test-model-2c86e0-kserve-84899654f5-kd6k2\" (UID: \"967aea57-432f-47ab-a21e-6e1065a5be16\") " pod="ai-tenant-e2e-shared-a-2c86e0/auth-test-model-2c86e0-kserve-84899654f5-kd6k2" Jul 06 14:26:37.358477 ip-10-0-136-25 kubenswrapper[2569]: I0706 14:26:37.358400 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/967aea57-432f-47ab-a21e-6e1065a5be16-dshm\") pod \"auth-test-model-2c86e0-kserve-84899654f5-kd6k2\" (UID: \"967aea57-432f-47ab-a21e-6e1065a5be16\") " pod="ai-tenant-e2e-shared-a-2c86e0/auth-test-model-2c86e0-kserve-84899654f5-kd6k2" Jul 06 14:26:37.358477 ip-10-0-136-25 kubenswrapper[2569]: I0706 14:26:37.358422 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/967aea57-432f-47ab-a21e-6e1065a5be16-model-cache\") pod \"auth-test-model-2c86e0-kserve-84899654f5-kd6k2\" (UID: \"967aea57-432f-47ab-a21e-6e1065a5be16\") " pod="ai-tenant-e2e-shared-a-2c86e0/auth-test-model-2c86e0-kserve-84899654f5-kd6k2" Jul 06 14:26:37.358477 ip-10-0-136-25 kubenswrapper[2569]: I0706 14:26:37.358443 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/967aea57-432f-47ab-a21e-6e1065a5be16-tls-certs\") pod \"auth-test-model-2c86e0-kserve-84899654f5-kd6k2\" (UID: \"967aea57-432f-47ab-a21e-6e1065a5be16\") " pod="ai-tenant-e2e-shared-a-2c86e0/auth-test-model-2c86e0-kserve-84899654f5-kd6k2" Jul 06 14:26:37.358638 ip-10-0-136-25 kubenswrapper[2569]: I0706 14:26:37.358461 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/967aea57-432f-47ab-a21e-6e1065a5be16-home\") pod \"auth-test-model-2c86e0-kserve-84899654f5-kd6k2\" (UID: \"967aea57-432f-47ab-a21e-6e1065a5be16\") " pod="ai-tenant-e2e-shared-a-2c86e0/auth-test-model-2c86e0-kserve-84899654f5-kd6k2" Jul 06 14:26:37.459569 ip-10-0-136-25 kubenswrapper[2569]: I0706 14:26:37.459536 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/967aea57-432f-47ab-a21e-6e1065a5be16-dshm\") pod \"auth-test-model-2c86e0-kserve-84899654f5-kd6k2\" (UID: \"967aea57-432f-47ab-a21e-6e1065a5be16\") " pod="ai-tenant-e2e-shared-a-2c86e0/auth-test-model-2c86e0-kserve-84899654f5-kd6k2" Jul 06 14:26:37.459569 ip-10-0-136-25 kubenswrapper[2569]: I0706 14:26:37.459571 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/967aea57-432f-47ab-a21e-6e1065a5be16-model-cache\") pod \"auth-test-model-2c86e0-kserve-84899654f5-kd6k2\" (UID: \"967aea57-432f-47ab-a21e-6e1065a5be16\") " pod="ai-tenant-e2e-shared-a-2c86e0/auth-test-model-2c86e0-kserve-84899654f5-kd6k2" Jul 06 14:26:37.459815 ip-10-0-136-25 kubenswrapper[2569]: I0706 14:26:37.459595 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/967aea57-432f-47ab-a21e-6e1065a5be16-tls-certs\") pod \"auth-test-model-2c86e0-kserve-84899654f5-kd6k2\" (UID: \"967aea57-432f-47ab-a21e-6e1065a5be16\") " pod="ai-tenant-e2e-shared-a-2c86e0/auth-test-model-2c86e0-kserve-84899654f5-kd6k2" Jul 06 14:26:37.459815 ip-10-0-136-25 kubenswrapper[2569]: I0706 14:26:37.459616 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/967aea57-432f-47ab-a21e-6e1065a5be16-home\") pod \"auth-test-model-2c86e0-kserve-84899654f5-kd6k2\" (UID: \"967aea57-432f-47ab-a21e-6e1065a5be16\") " pod="ai-tenant-e2e-shared-a-2c86e0/auth-test-model-2c86e0-kserve-84899654f5-kd6k2" Jul 06 14:26:37.459815 ip-10-0-136-25 kubenswrapper[2569]: I0706 14:26:37.459652 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gbqxl\" (UniqueName: \"kubernetes.io/projected/967aea57-432f-47ab-a21e-6e1065a5be16-kube-api-access-gbqxl\") pod \"auth-test-model-2c86e0-kserve-84899654f5-kd6k2\" (UID: \"967aea57-432f-47ab-a21e-6e1065a5be16\") " pod="ai-tenant-e2e-shared-a-2c86e0/auth-test-model-2c86e0-kserve-84899654f5-kd6k2" Jul 06 14:26:37.459815 ip-10-0-136-25 kubenswrapper[2569]: I0706 14:26:37.459690 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/967aea57-432f-47ab-a21e-6e1065a5be16-tmp-dir\") pod \"auth-test-model-2c86e0-kserve-84899654f5-kd6k2\" (UID: \"967aea57-432f-47ab-a21e-6e1065a5be16\") " pod="ai-tenant-e2e-shared-a-2c86e0/auth-test-model-2c86e0-kserve-84899654f5-kd6k2" Jul 06 14:26:37.460009 ip-10-0-136-25 kubenswrapper[2569]: I0706 14:26:37.459985 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/967aea57-432f-47ab-a21e-6e1065a5be16-model-cache\") pod \"auth-test-model-2c86e0-kserve-84899654f5-kd6k2\" (UID: \"967aea57-432f-47ab-a21e-6e1065a5be16\") " pod="ai-tenant-e2e-shared-a-2c86e0/auth-test-model-2c86e0-kserve-84899654f5-kd6k2" Jul 06 14:26:37.460067 ip-10-0-136-25 kubenswrapper[2569]: I0706 14:26:37.460033 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/967aea57-432f-47ab-a21e-6e1065a5be16-home\") pod \"auth-test-model-2c86e0-kserve-84899654f5-kd6k2\" (UID: \"967aea57-432f-47ab-a21e-6e1065a5be16\") " pod="ai-tenant-e2e-shared-a-2c86e0/auth-test-model-2c86e0-kserve-84899654f5-kd6k2" Jul 06 14:26:37.460139 ip-10-0-136-25 kubenswrapper[2569]: I0706 14:26:37.460121 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/967aea57-432f-47ab-a21e-6e1065a5be16-tmp-dir\") pod \"auth-test-model-2c86e0-kserve-84899654f5-kd6k2\" (UID: \"967aea57-432f-47ab-a21e-6e1065a5be16\") " pod="ai-tenant-e2e-shared-a-2c86e0/auth-test-model-2c86e0-kserve-84899654f5-kd6k2" Jul 06 14:26:37.461980 ip-10-0-136-25 kubenswrapper[2569]: I0706 14:26:37.461964 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/967aea57-432f-47ab-a21e-6e1065a5be16-dshm\") pod \"auth-test-model-2c86e0-kserve-84899654f5-kd6k2\" (UID: \"967aea57-432f-47ab-a21e-6e1065a5be16\") " pod="ai-tenant-e2e-shared-a-2c86e0/auth-test-model-2c86e0-kserve-84899654f5-kd6k2" Jul 06 14:26:37.462297 ip-10-0-136-25 kubenswrapper[2569]: I0706 14:26:37.462276 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/967aea57-432f-47ab-a21e-6e1065a5be16-tls-certs\") pod \"auth-test-model-2c86e0-kserve-84899654f5-kd6k2\" (UID: \"967aea57-432f-47ab-a21e-6e1065a5be16\") " pod="ai-tenant-e2e-shared-a-2c86e0/auth-test-model-2c86e0-kserve-84899654f5-kd6k2" Jul 06 14:26:37.467128 ip-10-0-136-25 kubenswrapper[2569]: I0706 14:26:37.467107 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gbqxl\" (UniqueName: \"kubernetes.io/projected/967aea57-432f-47ab-a21e-6e1065a5be16-kube-api-access-gbqxl\") pod \"auth-test-model-2c86e0-kserve-84899654f5-kd6k2\" (UID: \"967aea57-432f-47ab-a21e-6e1065a5be16\") " pod="ai-tenant-e2e-shared-a-2c86e0/auth-test-model-2c86e0-kserve-84899654f5-kd6k2" Jul 06 14:26:37.560249 ip-10-0-136-25 kubenswrapper[2569]: I0706 14:26:37.560130 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-2c86e0/auth-test-model-2c86e0-kserve-84899654f5-kd6k2" Jul 06 14:26:37.710360 ip-10-0-136-25 kubenswrapper[2569]: I0706 14:26:37.710334 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-2c86e0/auth-test-model-2c86e0-kserve-84899654f5-kd6k2"] Jul 06 14:26:37.712883 ip-10-0-136-25 kubenswrapper[2569]: W0706 14:26:37.712854 2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod967aea57_432f_47ab_a21e_6e1065a5be16.slice/crio-51e2b12d9f1e5e54b391508a1eef1269c4dc3f13b9b011da4673c19303274f7e WatchSource:0}: Error finding container 51e2b12d9f1e5e54b391508a1eef1269c4dc3f13b9b011da4673c19303274f7e: Status 404 returned error can't find the container with id 51e2b12d9f1e5e54b391508a1eef1269c4dc3f13b9b011da4673c19303274f7e Jul 06 14:26:37.892978 ip-10-0-136-25 kubenswrapper[2569]: I0706 14:26:37.892879 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-2c86e0/auth-test-model-2c86e0-kserve-84899654f5-kd6k2" event={"ID":"967aea57-432f-47ab-a21e-6e1065a5be16","Type":"ContainerStarted","Data":"35ee3fdcb1a8aa117e0192350c5f553cceed2d2bed6b776bc0e1db8087a49a8f"} Jul 06 14:26:37.892978 ip-10-0-136-25 kubenswrapper[2569]: I0706 14:26:37.892918 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-2c86e0/auth-test-model-2c86e0-kserve-84899654f5-kd6k2" event={"ID":"967aea57-432f-47ab-a21e-6e1065a5be16","Type":"ContainerStarted","Data":"51e2b12d9f1e5e54b391508a1eef1269c4dc3f13b9b011da4673c19303274f7e"} Jul 06 14:26:37.909625 ip-10-0-136-25 kubenswrapper[2569]: I0706 14:26:37.909574 2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-a-2c86e0/auth-test-model-2c86e0-kserve-84899654f5-kd6k2" podStartSLOduration=0.909559061 podStartE2EDuration="909.559061ms" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-06 14:26:37 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 14:26:37.908279287 +0000 UTC m=+2173.498841062" watchObservedRunningTime="2026-07-06 14:26:37.909559061 +0000 UTC m=+2173.500120838" Jul 06 14:26:39.443817 ip-10-0-136-25 kubenswrapper[2569]: I0706 14:26:39.443765 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-g9qp6"] Jul 06 14:26:47.560960 ip-10-0-136-25 kubenswrapper[2569]: I0706 14:26:47.560902 2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-a-2c86e0/auth-test-model-2c86e0-kserve-84899654f5-kd6k2" Jul 06 14:26:47.560960 ip-10-0-136-25 kubenswrapper[2569]: I0706 14:26:47.560980 2569 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-a-2c86e0/auth-test-model-2c86e0-kserve-84899654f5-kd6k2" Jul 06 14:26:47.573531 ip-10-0-136-25 kubenswrapper[2569]: I0706 14:26:47.573501 2569 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-a-2c86e0/auth-test-model-2c86e0-kserve-84899654f5-kd6k2" Jul 06 14:26:47.955429 ip-10-0-136-25 kubenswrapper[2569]: I0706 14:26:47.955342 2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-a-2c86e0/auth-test-model-2c86e0-kserve-84899654f5-kd6k2" Jul 06 14:27:14.648194 ip-10-0-136-25 kubenswrapper[2569]: I0706 14:27:14.648158 2569 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-b-eb526a/auth-test-model-eb526a-kserve-5d5886757f-nx7tw"] Jul 06 14:27:14.651981 ip-10-0-136-25 kubenswrapper[2569]: I0706 14:27:14.651960 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-eb526a/auth-test-model-eb526a-kserve-5d5886757f-nx7tw" Jul 06 14:27:14.654356 ip-10-0-136-25 kubenswrapper[2569]: I0706 14:27:14.654335 2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-eb526a\"/\"kube-root-ca.crt\"" Jul 06 14:27:14.655132 ip-10-0-136-25 kubenswrapper[2569]: I0706 14:27:14.655111 2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-eb526a\"/\"default-dockercfg-jnhmf\"" Jul 06 14:27:14.655283 ip-10-0-136-25 kubenswrapper[2569]: I0706 14:27:14.655149 2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-eb526a\"/\"openshift-service-ca.crt\"" Jul 06 14:27:14.655283 ip-10-0-136-25 kubenswrapper[2569]: I0706 14:27:14.655154 2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-eb526a\"/\"auth-test-model-eb526a-kserve-self-signed-certs\"" Jul 06 14:27:14.661807 ip-10-0-136-25 kubenswrapper[2569]: I0706 14:27:14.661784 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-eb526a/auth-test-model-eb526a-kserve-5d5886757f-nx7tw"] Jul 06 14:27:14.801551 ip-10-0-136-25 kubenswrapper[2569]: I0706 14:27:14.801516 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0c5514a5-e417-4b9e-ad2d-d409189280a6-tmp-dir\") pod \"auth-test-model-eb526a-kserve-5d5886757f-nx7tw\" (UID: \"0c5514a5-e417-4b9e-ad2d-d409189280a6\") " pod="ai-tenant-e2e-shared-b-eb526a/auth-test-model-eb526a-kserve-5d5886757f-nx7tw" Jul 06 14:27:14.801745 ip-10-0-136-25 kubenswrapper[2569]: I0706 14:27:14.801565 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0c5514a5-e417-4b9e-ad2d-d409189280a6-home\") pod \"auth-test-model-eb526a-kserve-5d5886757f-nx7tw\" (UID: \"0c5514a5-e417-4b9e-ad2d-d409189280a6\") " pod="ai-tenant-e2e-shared-b-eb526a/auth-test-model-eb526a-kserve-5d5886757f-nx7tw" Jul 06 14:27:14.801745 ip-10-0-136-25 kubenswrapper[2569]: I0706 14:27:14.801588 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7lbqw\" (UniqueName: \"kubernetes.io/projected/0c5514a5-e417-4b9e-ad2d-d409189280a6-kube-api-access-7lbqw\") pod \"auth-test-model-eb526a-kserve-5d5886757f-nx7tw\" (UID: \"0c5514a5-e417-4b9e-ad2d-d409189280a6\") " pod="ai-tenant-e2e-shared-b-eb526a/auth-test-model-eb526a-kserve-5d5886757f-nx7tw" Jul 06 14:27:14.801745 ip-10-0-136-25 kubenswrapper[2569]: I0706 14:27:14.801663 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0c5514a5-e417-4b9e-ad2d-d409189280a6-tls-certs\") pod \"auth-test-model-eb526a-kserve-5d5886757f-nx7tw\" (UID: \"0c5514a5-e417-4b9e-ad2d-d409189280a6\") " pod="ai-tenant-e2e-shared-b-eb526a/auth-test-model-eb526a-kserve-5d5886757f-nx7tw" Jul 06 14:27:14.801921 ip-10-0-136-25 kubenswrapper[2569]: I0706 14:27:14.801755 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0c5514a5-e417-4b9e-ad2d-d409189280a6-dshm\") pod \"auth-test-model-eb526a-kserve-5d5886757f-nx7tw\" (UID: \"0c5514a5-e417-4b9e-ad2d-d409189280a6\") " pod="ai-tenant-e2e-shared-b-eb526a/auth-test-model-eb526a-kserve-5d5886757f-nx7tw" Jul 06 14:27:14.801921 ip-10-0-136-25 kubenswrapper[2569]: I0706 14:27:14.801775 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0c5514a5-e417-4b9e-ad2d-d409189280a6-model-cache\") pod \"auth-test-model-eb526a-kserve-5d5886757f-nx7tw\" (UID: \"0c5514a5-e417-4b9e-ad2d-d409189280a6\") " pod="ai-tenant-e2e-shared-b-eb526a/auth-test-model-eb526a-kserve-5d5886757f-nx7tw" Jul 06 14:27:14.902494 ip-10-0-136-25 kubenswrapper[2569]: I0706 14:27:14.902390 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0c5514a5-e417-4b9e-ad2d-d409189280a6-dshm\") pod \"auth-test-model-eb526a-kserve-5d5886757f-nx7tw\" (UID: \"0c5514a5-e417-4b9e-ad2d-d409189280a6\") " pod="ai-tenant-e2e-shared-b-eb526a/auth-test-model-eb526a-kserve-5d5886757f-nx7tw" Jul 06 14:27:14.902494 ip-10-0-136-25 kubenswrapper[2569]: I0706 14:27:14.902431 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0c5514a5-e417-4b9e-ad2d-d409189280a6-model-cache\") pod \"auth-test-model-eb526a-kserve-5d5886757f-nx7tw\" (UID: \"0c5514a5-e417-4b9e-ad2d-d409189280a6\") " pod="ai-tenant-e2e-shared-b-eb526a/auth-test-model-eb526a-kserve-5d5886757f-nx7tw" Jul 06 14:27:14.902494 ip-10-0-136-25 kubenswrapper[2569]: I0706 14:27:14.902471 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0c5514a5-e417-4b9e-ad2d-d409189280a6-tmp-dir\") pod \"auth-test-model-eb526a-kserve-5d5886757f-nx7tw\" (UID: \"0c5514a5-e417-4b9e-ad2d-d409189280a6\") " pod="ai-tenant-e2e-shared-b-eb526a/auth-test-model-eb526a-kserve-5d5886757f-nx7tw" Jul 06 14:27:14.902783 ip-10-0-136-25 kubenswrapper[2569]: I0706 14:27:14.902501 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0c5514a5-e417-4b9e-ad2d-d409189280a6-home\") pod \"auth-test-model-eb526a-kserve-5d5886757f-nx7tw\" (UID: \"0c5514a5-e417-4b9e-ad2d-d409189280a6\") " pod="ai-tenant-e2e-shared-b-eb526a/auth-test-model-eb526a-kserve-5d5886757f-nx7tw" Jul 06 14:27:14.902783 ip-10-0-136-25 kubenswrapper[2569]: I0706 14:27:14.902520 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7lbqw\" (UniqueName: \"kubernetes.io/projected/0c5514a5-e417-4b9e-ad2d-d409189280a6-kube-api-access-7lbqw\") pod \"auth-test-model-eb526a-kserve-5d5886757f-nx7tw\" (UID: \"0c5514a5-e417-4b9e-ad2d-d409189280a6\") " pod="ai-tenant-e2e-shared-b-eb526a/auth-test-model-eb526a-kserve-5d5886757f-nx7tw" Jul 06 14:27:14.902783 ip-10-0-136-25 kubenswrapper[2569]: I0706 14:27:14.902545 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0c5514a5-e417-4b9e-ad2d-d409189280a6-tls-certs\") pod \"auth-test-model-eb526a-kserve-5d5886757f-nx7tw\" (UID: \"0c5514a5-e417-4b9e-ad2d-d409189280a6\") " pod="ai-tenant-e2e-shared-b-eb526a/auth-test-model-eb526a-kserve-5d5886757f-nx7tw" Jul 06 14:27:14.902951 ip-10-0-136-25 kubenswrapper[2569]: I0706 14:27:14.902912 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0c5514a5-e417-4b9e-ad2d-d409189280a6-model-cache\") pod \"auth-test-model-eb526a-kserve-5d5886757f-nx7tw\" (UID: \"0c5514a5-e417-4b9e-ad2d-d409189280a6\") " pod="ai-tenant-e2e-shared-b-eb526a/auth-test-model-eb526a-kserve-5d5886757f-nx7tw" Jul 06 14:27:14.903007 ip-10-0-136-25 kubenswrapper[2569]: I0706 14:27:14.902948 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0c5514a5-e417-4b9e-ad2d-d409189280a6-home\") pod \"auth-test-model-eb526a-kserve-5d5886757f-nx7tw\" (UID: \"0c5514a5-e417-4b9e-ad2d-d409189280a6\") " pod="ai-tenant-e2e-shared-b-eb526a/auth-test-model-eb526a-kserve-5d5886757f-nx7tw" Jul 06 14:27:14.903007 ip-10-0-136-25 kubenswrapper[2569]: I0706 14:27:14.902993 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0c5514a5-e417-4b9e-ad2d-d409189280a6-tmp-dir\") pod \"auth-test-model-eb526a-kserve-5d5886757f-nx7tw\" (UID: \"0c5514a5-e417-4b9e-ad2d-d409189280a6\") " pod="ai-tenant-e2e-shared-b-eb526a/auth-test-model-eb526a-kserve-5d5886757f-nx7tw" Jul 06 14:27:14.904797 ip-10-0-136-25 kubenswrapper[2569]: I0706 14:27:14.904770 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0c5514a5-e417-4b9e-ad2d-d409189280a6-dshm\") pod \"auth-test-model-eb526a-kserve-5d5886757f-nx7tw\" (UID: \"0c5514a5-e417-4b9e-ad2d-d409189280a6\") " pod="ai-tenant-e2e-shared-b-eb526a/auth-test-model-eb526a-kserve-5d5886757f-nx7tw" Jul 06 14:27:14.905052 ip-10-0-136-25 kubenswrapper[2569]: I0706 14:27:14.905035 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0c5514a5-e417-4b9e-ad2d-d409189280a6-tls-certs\") pod \"auth-test-model-eb526a-kserve-5d5886757f-nx7tw\" (UID: \"0c5514a5-e417-4b9e-ad2d-d409189280a6\") " pod="ai-tenant-e2e-shared-b-eb526a/auth-test-model-eb526a-kserve-5d5886757f-nx7tw" Jul 06 14:27:14.909508 ip-10-0-136-25 kubenswrapper[2569]: I0706 14:27:14.909471 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7lbqw\" (UniqueName: \"kubernetes.io/projected/0c5514a5-e417-4b9e-ad2d-d409189280a6-kube-api-access-7lbqw\") pod \"auth-test-model-eb526a-kserve-5d5886757f-nx7tw\" (UID: \"0c5514a5-e417-4b9e-ad2d-d409189280a6\") " pod="ai-tenant-e2e-shared-b-eb526a/auth-test-model-eb526a-kserve-5d5886757f-nx7tw" Jul 06 14:27:14.962176 ip-10-0-136-25 kubenswrapper[2569]: I0706 14:27:14.962130 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-eb526a/auth-test-model-eb526a-kserve-5d5886757f-nx7tw" Jul 06 14:27:15.096877 ip-10-0-136-25 kubenswrapper[2569]: I0706 14:27:15.096849 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-eb526a/auth-test-model-eb526a-kserve-5d5886757f-nx7tw"] Jul 06 14:27:15.102063 ip-10-0-136-25 kubenswrapper[2569]: W0706 14:27:15.102023 2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod0c5514a5_e417_4b9e_ad2d_d409189280a6.slice/crio-dfe3379f327b4111e3edb4608a65a794d428086d6f99d02650693c1382a72d46 WatchSource:0}: Error finding container dfe3379f327b4111e3edb4608a65a794d428086d6f99d02650693c1382a72d46: Status 404 returned error can't find the container with id dfe3379f327b4111e3edb4608a65a794d428086d6f99d02650693c1382a72d46 Jul 06 14:27:16.076795 ip-10-0-136-25 kubenswrapper[2569]: I0706 14:27:16.076759 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-eb526a/auth-test-model-eb526a-kserve-5d5886757f-nx7tw" event={"ID":"0c5514a5-e417-4b9e-ad2d-d409189280a6","Type":"ContainerStarted","Data":"4da486ae0538cfe5e3a4d5021b6f3b1a8178f6d787aa01bb78fe367f2f6f5dc7"} Jul 06 14:27:16.076795 ip-10-0-136-25 kubenswrapper[2569]: I0706 14:27:16.076799 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-eb526a/auth-test-model-eb526a-kserve-5d5886757f-nx7tw" event={"ID":"0c5514a5-e417-4b9e-ad2d-d409189280a6","Type":"ContainerStarted","Data":"dfe3379f327b4111e3edb4608a65a794d428086d6f99d02650693c1382a72d46"} Jul 06 14:27:16.093853 ip-10-0-136-25 kubenswrapper[2569]: I0706 14:27:16.093799 2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-b-eb526a/auth-test-model-eb526a-kserve-5d5886757f-nx7tw" podStartSLOduration=2.09378485 podStartE2EDuration="2.09378485s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-06 14:27:14 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 14:27:16.092392126 +0000 UTC m=+2211.682953902" watchObservedRunningTime="2026-07-06 14:27:16.09378485 +0000 UTC m=+2211.684346640" Jul 06 14:27:18.136356 ip-10-0-136-25 kubenswrapper[2569]: I0706 14:27:18.136322 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-g9qp6"] Jul 06 14:27:24.962442 ip-10-0-136-25 kubenswrapper[2569]: I0706 14:27:24.962402 2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-b-eb526a/auth-test-model-eb526a-kserve-5d5886757f-nx7tw" Jul 06 14:27:24.962442 ip-10-0-136-25 kubenswrapper[2569]: I0706 14:27:24.962452 2569 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-b-eb526a/auth-test-model-eb526a-kserve-5d5886757f-nx7tw" Jul 06 14:27:24.975372 ip-10-0-136-25 kubenswrapper[2569]: I0706 14:27:24.975347 2569 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-b-eb526a/auth-test-model-eb526a-kserve-5d5886757f-nx7tw" Jul 06 14:27:25.130474 ip-10-0-136-25 kubenswrapper[2569]: I0706 14:27:25.130446 2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-b-eb526a/auth-test-model-eb526a-kserve-5d5886757f-nx7tw" Jul 06 14:27:43.440295 ip-10-0-136-25 kubenswrapper[2569]: I0706 14:27:43.440253 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-g9qp6"] Jul 06 14:27:59.138471 ip-10-0-136-25 kubenswrapper[2569]: I0706 14:27:59.138386 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-g9qp6"] Jul 06 14:28:14.042836 ip-10-0-136-25 kubenswrapper[2569]: I0706 14:28:14.042794 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-g9qp6"] Jul 06 14:28:30.741905 ip-10-0-136-25 kubenswrapper[2569]: I0706 14:28:30.741856 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-g9qp6"] Jul 06 14:28:48.132236 ip-10-0-136-25 kubenswrapper[2569]: I0706 14:28:48.132188 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-g9qp6"] Jul 06 14:29:05.230277 ip-10-0-136-25 kubenswrapper[2569]: I0706 14:29:05.230233 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-g9qp6"] Jul 06 14:29:22.140172 ip-10-0-136-25 kubenswrapper[2569]: I0706 14:29:22.140087 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-g9qp6"] Jul 06 14:29:26.497042 ip-10-0-136-25 kubenswrapper[2569]: I0706 14:29:26.497003 2569 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-2c86e0/auth-test-model-2c86e0-kserve-84899654f5-kd6k2"] Jul 06 14:29:27.322552 ip-10-0-136-25 kubenswrapper[2569]: I0706 14:29:27.322518 2569 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-eb526a/auth-test-model-eb526a-kserve-5d5886757f-nx7tw"] Jul 06 14:29:33.339679 ip-10-0-136-25 kubenswrapper[2569]: I0706 14:29:33.339645 2569 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-a-2c86e0/sub-test-model-2c86e0-kserve-86f676b867-58bj2"] Jul 06 14:29:33.342834 ip-10-0-136-25 kubenswrapper[2569]: I0706 14:29:33.342817 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-2c86e0/sub-test-model-2c86e0-kserve-86f676b867-58bj2" Jul 06 14:29:33.344905 ip-10-0-136-25 kubenswrapper[2569]: I0706 14:29:33.344886 2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-2c86e0\"/\"sub-test-model-2c86e0-kserve-self-signed-certs\"" Jul 06 14:29:33.349853 ip-10-0-136-25 kubenswrapper[2569]: I0706 14:29:33.349822 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-2c86e0/sub-test-model-2c86e0-kserve-86f676b867-58bj2"] Jul 06 14:29:33.405389 ip-10-0-136-25 kubenswrapper[2569]: I0706 14:29:33.405353 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f03ec316-06ae-470e-ad36-337dd335f037-tls-certs\") pod \"sub-test-model-2c86e0-kserve-86f676b867-58bj2\" (UID: \"f03ec316-06ae-470e-ad36-337dd335f037\") " pod="ai-tenant-e2e-shared-a-2c86e0/sub-test-model-2c86e0-kserve-86f676b867-58bj2" Jul 06 14:29:33.405573 ip-10-0-136-25 kubenswrapper[2569]: I0706 14:29:33.405427 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f03ec316-06ae-470e-ad36-337dd335f037-tmp-dir\") pod \"sub-test-model-2c86e0-kserve-86f676b867-58bj2\" (UID: \"f03ec316-06ae-470e-ad36-337dd335f037\") " pod="ai-tenant-e2e-shared-a-2c86e0/sub-test-model-2c86e0-kserve-86f676b867-58bj2" Jul 06 14:29:33.405573 ip-10-0-136-25 kubenswrapper[2569]: I0706 14:29:33.405467 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f03ec316-06ae-470e-ad36-337dd335f037-model-cache\") pod \"sub-test-model-2c86e0-kserve-86f676b867-58bj2\" (UID: \"f03ec316-06ae-470e-ad36-337dd335f037\") " pod="ai-tenant-e2e-shared-a-2c86e0/sub-test-model-2c86e0-kserve-86f676b867-58bj2" Jul 06 14:29:33.405573 ip-10-0-136-25 kubenswrapper[2569]: I0706 14:29:33.405483 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f03ec316-06ae-470e-ad36-337dd335f037-home\") pod \"sub-test-model-2c86e0-kserve-86f676b867-58bj2\" (UID: \"f03ec316-06ae-470e-ad36-337dd335f037\") " pod="ai-tenant-e2e-shared-a-2c86e0/sub-test-model-2c86e0-kserve-86f676b867-58bj2" Jul 06 14:29:33.405706 ip-10-0-136-25 kubenswrapper[2569]: I0706 14:29:33.405584 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jxtf9\" (UniqueName: \"kubernetes.io/projected/f03ec316-06ae-470e-ad36-337dd335f037-kube-api-access-jxtf9\") pod \"sub-test-model-2c86e0-kserve-86f676b867-58bj2\" (UID: \"f03ec316-06ae-470e-ad36-337dd335f037\") " pod="ai-tenant-e2e-shared-a-2c86e0/sub-test-model-2c86e0-kserve-86f676b867-58bj2" Jul 06 14:29:33.405706 ip-10-0-136-25 kubenswrapper[2569]: I0706 14:29:33.405630 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f03ec316-06ae-470e-ad36-337dd335f037-dshm\") pod \"sub-test-model-2c86e0-kserve-86f676b867-58bj2\" (UID: \"f03ec316-06ae-470e-ad36-337dd335f037\") " pod="ai-tenant-e2e-shared-a-2c86e0/sub-test-model-2c86e0-kserve-86f676b867-58bj2" Jul 06 14:29:33.506903 ip-10-0-136-25 kubenswrapper[2569]: I0706 14:29:33.506857 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f03ec316-06ae-470e-ad36-337dd335f037-model-cache\") pod \"sub-test-model-2c86e0-kserve-86f676b867-58bj2\" (UID: \"f03ec316-06ae-470e-ad36-337dd335f037\") " pod="ai-tenant-e2e-shared-a-2c86e0/sub-test-model-2c86e0-kserve-86f676b867-58bj2" Jul 06 14:29:33.506903 ip-10-0-136-25 kubenswrapper[2569]: I0706 14:29:33.506909 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f03ec316-06ae-470e-ad36-337dd335f037-home\") pod \"sub-test-model-2c86e0-kserve-86f676b867-58bj2\" (UID: \"f03ec316-06ae-470e-ad36-337dd335f037\") " pod="ai-tenant-e2e-shared-a-2c86e0/sub-test-model-2c86e0-kserve-86f676b867-58bj2" Jul 06 14:29:33.507166 ip-10-0-136-25 kubenswrapper[2569]: I0706 14:29:33.506944 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jxtf9\" (UniqueName: \"kubernetes.io/projected/f03ec316-06ae-470e-ad36-337dd335f037-kube-api-access-jxtf9\") pod \"sub-test-model-2c86e0-kserve-86f676b867-58bj2\" (UID: \"f03ec316-06ae-470e-ad36-337dd335f037\") " pod="ai-tenant-e2e-shared-a-2c86e0/sub-test-model-2c86e0-kserve-86f676b867-58bj2" Jul 06 14:29:33.507166 ip-10-0-136-25 kubenswrapper[2569]: I0706 14:29:33.506983 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f03ec316-06ae-470e-ad36-337dd335f037-dshm\") pod \"sub-test-model-2c86e0-kserve-86f676b867-58bj2\" (UID: \"f03ec316-06ae-470e-ad36-337dd335f037\") " pod="ai-tenant-e2e-shared-a-2c86e0/sub-test-model-2c86e0-kserve-86f676b867-58bj2" Jul 06 14:29:33.507166 ip-10-0-136-25 kubenswrapper[2569]: I0706 14:29:33.507059 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f03ec316-06ae-470e-ad36-337dd335f037-tls-certs\") pod \"sub-test-model-2c86e0-kserve-86f676b867-58bj2\" (UID: \"f03ec316-06ae-470e-ad36-337dd335f037\") " pod="ai-tenant-e2e-shared-a-2c86e0/sub-test-model-2c86e0-kserve-86f676b867-58bj2" Jul 06 14:29:33.507166 ip-10-0-136-25 kubenswrapper[2569]: I0706 14:29:33.507142 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f03ec316-06ae-470e-ad36-337dd335f037-tmp-dir\") pod \"sub-test-model-2c86e0-kserve-86f676b867-58bj2\" (UID: \"f03ec316-06ae-470e-ad36-337dd335f037\") " pod="ai-tenant-e2e-shared-a-2c86e0/sub-test-model-2c86e0-kserve-86f676b867-58bj2" Jul 06 14:29:33.507401 ip-10-0-136-25 kubenswrapper[2569]: I0706 14:29:33.507364 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f03ec316-06ae-470e-ad36-337dd335f037-model-cache\") pod \"sub-test-model-2c86e0-kserve-86f676b867-58bj2\" (UID: \"f03ec316-06ae-470e-ad36-337dd335f037\") " pod="ai-tenant-e2e-shared-a-2c86e0/sub-test-model-2c86e0-kserve-86f676b867-58bj2" Jul 06 14:29:33.507518 ip-10-0-136-25 kubenswrapper[2569]: I0706 14:29:33.507409 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f03ec316-06ae-470e-ad36-337dd335f037-home\") pod \"sub-test-model-2c86e0-kserve-86f676b867-58bj2\" (UID: \"f03ec316-06ae-470e-ad36-337dd335f037\") " pod="ai-tenant-e2e-shared-a-2c86e0/sub-test-model-2c86e0-kserve-86f676b867-58bj2" Jul 06 14:29:33.507587 ip-10-0-136-25 kubenswrapper[2569]: I0706 14:29:33.507568 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f03ec316-06ae-470e-ad36-337dd335f037-tmp-dir\") pod \"sub-test-model-2c86e0-kserve-86f676b867-58bj2\" (UID: \"f03ec316-06ae-470e-ad36-337dd335f037\") " pod="ai-tenant-e2e-shared-a-2c86e0/sub-test-model-2c86e0-kserve-86f676b867-58bj2" Jul 06 14:29:33.509427 ip-10-0-136-25 kubenswrapper[2569]: I0706 14:29:33.509404 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f03ec316-06ae-470e-ad36-337dd335f037-dshm\") pod \"sub-test-model-2c86e0-kserve-86f676b867-58bj2\" (UID: \"f03ec316-06ae-470e-ad36-337dd335f037\") " pod="ai-tenant-e2e-shared-a-2c86e0/sub-test-model-2c86e0-kserve-86f676b867-58bj2" Jul 06 14:29:33.509568 ip-10-0-136-25 kubenswrapper[2569]: I0706 14:29:33.509549 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f03ec316-06ae-470e-ad36-337dd335f037-tls-certs\") pod \"sub-test-model-2c86e0-kserve-86f676b867-58bj2\" (UID: \"f03ec316-06ae-470e-ad36-337dd335f037\") " pod="ai-tenant-e2e-shared-a-2c86e0/sub-test-model-2c86e0-kserve-86f676b867-58bj2" Jul 06 14:29:33.514143 ip-10-0-136-25 kubenswrapper[2569]: I0706 14:29:33.514123 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jxtf9\" (UniqueName: \"kubernetes.io/projected/f03ec316-06ae-470e-ad36-337dd335f037-kube-api-access-jxtf9\") pod \"sub-test-model-2c86e0-kserve-86f676b867-58bj2\" (UID: \"f03ec316-06ae-470e-ad36-337dd335f037\") " pod="ai-tenant-e2e-shared-a-2c86e0/sub-test-model-2c86e0-kserve-86f676b867-58bj2" Jul 06 14:29:33.652663 ip-10-0-136-25 kubenswrapper[2569]: I0706 14:29:33.652573 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-2c86e0/sub-test-model-2c86e0-kserve-86f676b867-58bj2" Jul 06 14:29:33.786171 ip-10-0-136-25 kubenswrapper[2569]: I0706 14:29:33.786142 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-2c86e0/sub-test-model-2c86e0-kserve-86f676b867-58bj2"] Jul 06 14:29:33.787907 ip-10-0-136-25 kubenswrapper[2569]: W0706 14:29:33.787880 2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podf03ec316_06ae_470e_ad36_337dd335f037.slice/crio-49a5377eefe4a80ac6739dd27b27a65dc294edc1431625601de336a5b896e026 WatchSource:0}: Error finding container 49a5377eefe4a80ac6739dd27b27a65dc294edc1431625601de336a5b896e026: Status 404 returned error can't find the container with id 49a5377eefe4a80ac6739dd27b27a65dc294edc1431625601de336a5b896e026 Jul 06 14:29:34.735491 ip-10-0-136-25 kubenswrapper[2569]: I0706 14:29:34.735452 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-2c86e0/sub-test-model-2c86e0-kserve-86f676b867-58bj2" event={"ID":"f03ec316-06ae-470e-ad36-337dd335f037","Type":"ContainerStarted","Data":"76b42082ec511d29de6f9b9d08f727d3ecbf9a5e03eb8ebadada25af17900055"} Jul 06 14:29:34.735888 ip-10-0-136-25 kubenswrapper[2569]: I0706 14:29:34.735502 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-2c86e0/sub-test-model-2c86e0-kserve-86f676b867-58bj2" event={"ID":"f03ec316-06ae-470e-ad36-337dd335f037","Type":"ContainerStarted","Data":"49a5377eefe4a80ac6739dd27b27a65dc294edc1431625601de336a5b896e026"} Jul 06 14:29:34.752555 ip-10-0-136-25 kubenswrapper[2569]: I0706 14:29:34.752505 2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-a-2c86e0/sub-test-model-2c86e0-kserve-86f676b867-58bj2" podStartSLOduration=1.752488888 podStartE2EDuration="1.752488888s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-06 14:29:33 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 14:29:34.751237031 +0000 UTC m=+2350.341798829" watchObservedRunningTime="2026-07-06 14:29:34.752488888 +0000 UTC m=+2350.343050664" Jul 06 14:29:40.848079 ip-10-0-136-25 kubenswrapper[2569]: I0706 14:29:40.848040 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-g9qp6"] Jul 06 14:29:41.517179 ip-10-0-136-25 kubenswrapper[2569]: I0706 14:29:41.517135 2569 kuberuntime_container.go:872] "Killing container with a grace period" pod="ai-tenant-e2e-shared-a-2c86e0/auth-test-model-2c86e0-kserve-84899654f5-kd6k2" podUID="967aea57-432f-47ab-a21e-6e1065a5be16" containerName="main" containerID="cri-o://35ee3fdcb1a8aa117e0192350c5f553cceed2d2bed6b776bc0e1db8087a49a8f" gracePeriod=45 Jul 06 14:29:41.769057 ip-10-0-136-25 kubenswrapper[2569]: I0706 14:29:41.768978 2569 generic.go:356] "Generic (PLEG): container finished" podID="967aea57-432f-47ab-a21e-6e1065a5be16" containerID="35ee3fdcb1a8aa117e0192350c5f553cceed2d2bed6b776bc0e1db8087a49a8f" exitCode=0 Jul 06 14:29:41.769057 ip-10-0-136-25 kubenswrapper[2569]: I0706 14:29:41.769046 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-2c86e0/auth-test-model-2c86e0-kserve-84899654f5-kd6k2" event={"ID":"967aea57-432f-47ab-a21e-6e1065a5be16","Type":"ContainerDied","Data":"35ee3fdcb1a8aa117e0192350c5f553cceed2d2bed6b776bc0e1db8087a49a8f"} Jul 06 14:29:41.769204 ip-10-0-136-25 kubenswrapper[2569]: I0706 14:29:41.769087 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-2c86e0/auth-test-model-2c86e0-kserve-84899654f5-kd6k2" event={"ID":"967aea57-432f-47ab-a21e-6e1065a5be16","Type":"ContainerDied","Data":"51e2b12d9f1e5e54b391508a1eef1269c4dc3f13b9b011da4673c19303274f7e"} Jul 06 14:29:41.769204 ip-10-0-136-25 kubenswrapper[2569]: I0706 14:29:41.769100 2569 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="51e2b12d9f1e5e54b391508a1eef1269c4dc3f13b9b011da4673c19303274f7e" Jul 06 14:29:41.776292 ip-10-0-136-25 kubenswrapper[2569]: I0706 14:29:41.776274 2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-2c86e0/auth-test-model-2c86e0-kserve-84899654f5-kd6k2" Jul 06 14:29:41.886589 ip-10-0-136-25 kubenswrapper[2569]: I0706 14:29:41.886550 2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/967aea57-432f-47ab-a21e-6e1065a5be16-home\") pod \"967aea57-432f-47ab-a21e-6e1065a5be16\" (UID: \"967aea57-432f-47ab-a21e-6e1065a5be16\") " Jul 06 14:29:41.887049 ip-10-0-136-25 kubenswrapper[2569]: I0706 14:29:41.886626 2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/967aea57-432f-47ab-a21e-6e1065a5be16-tls-certs\") pod \"967aea57-432f-47ab-a21e-6e1065a5be16\" (UID: \"967aea57-432f-47ab-a21e-6e1065a5be16\") " Jul 06 14:29:41.887049 ip-10-0-136-25 kubenswrapper[2569]: I0706 14:29:41.886660 2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/967aea57-432f-47ab-a21e-6e1065a5be16-tmp-dir\") pod \"967aea57-432f-47ab-a21e-6e1065a5be16\" (UID: \"967aea57-432f-47ab-a21e-6e1065a5be16\") " Jul 06 14:29:41.887049 ip-10-0-136-25 kubenswrapper[2569]: I0706 14:29:41.886684 2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/967aea57-432f-47ab-a21e-6e1065a5be16-model-cache\") pod \"967aea57-432f-47ab-a21e-6e1065a5be16\" (UID: \"967aea57-432f-47ab-a21e-6e1065a5be16\") " Jul 06 14:29:41.887049 ip-10-0-136-25 kubenswrapper[2569]: I0706 14:29:41.886713 2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-gbqxl\" (UniqueName: \"kubernetes.io/projected/967aea57-432f-47ab-a21e-6e1065a5be16-kube-api-access-gbqxl\") pod \"967aea57-432f-47ab-a21e-6e1065a5be16\" (UID: \"967aea57-432f-47ab-a21e-6e1065a5be16\") " Jul 06 14:29:41.887049 ip-10-0-136-25 kubenswrapper[2569]: I0706 14:29:41.886737 2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/967aea57-432f-47ab-a21e-6e1065a5be16-dshm\") pod \"967aea57-432f-47ab-a21e-6e1065a5be16\" (UID: \"967aea57-432f-47ab-a21e-6e1065a5be16\") " Jul 06 14:29:41.887049 ip-10-0-136-25 kubenswrapper[2569]: I0706 14:29:41.886885 2569 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/967aea57-432f-47ab-a21e-6e1065a5be16-home" (OuterVolumeSpecName: "home") pod "967aea57-432f-47ab-a21e-6e1065a5be16" (UID: "967aea57-432f-47ab-a21e-6e1065a5be16"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 06 14:29:41.887049 ip-10-0-136-25 kubenswrapper[2569]: I0706 14:29:41.886984 2569 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/967aea57-432f-47ab-a21e-6e1065a5be16-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "967aea57-432f-47ab-a21e-6e1065a5be16" (UID: "967aea57-432f-47ab-a21e-6e1065a5be16"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 06 14:29:41.887049 ip-10-0-136-25 kubenswrapper[2569]: I0706 14:29:41.886999 2569 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/967aea57-432f-47ab-a21e-6e1065a5be16-model-cache" (OuterVolumeSpecName: "model-cache") pod "967aea57-432f-47ab-a21e-6e1065a5be16" (UID: "967aea57-432f-47ab-a21e-6e1065a5be16"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 06 14:29:41.887394 ip-10-0-136-25 kubenswrapper[2569]: I0706 14:29:41.887084 2569 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/967aea57-432f-47ab-a21e-6e1065a5be16-home\") on node \"ip-10-0-136-25.ec2.internal\" DevicePath \"\"" Jul 06 14:29:41.887394 ip-10-0-136-25 kubenswrapper[2569]: I0706 14:29:41.887097 2569 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/967aea57-432f-47ab-a21e-6e1065a5be16-tmp-dir\") on node \"ip-10-0-136-25.ec2.internal\" DevicePath \"\"" Jul 06 14:29:41.887394 ip-10-0-136-25 kubenswrapper[2569]: I0706 14:29:41.887106 2569 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/967aea57-432f-47ab-a21e-6e1065a5be16-model-cache\") on node \"ip-10-0-136-25.ec2.internal\" DevicePath \"\"" Jul 06 14:29:41.888944 ip-10-0-136-25 kubenswrapper[2569]: I0706 14:29:41.888924 2569 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/967aea57-432f-47ab-a21e-6e1065a5be16-dshm" (OuterVolumeSpecName: "dshm") pod "967aea57-432f-47ab-a21e-6e1065a5be16" (UID: "967aea57-432f-47ab-a21e-6e1065a5be16"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 06 14:29:41.889020 ip-10-0-136-25 kubenswrapper[2569]: I0706 14:29:41.888999 2569 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/967aea57-432f-47ab-a21e-6e1065a5be16-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "967aea57-432f-47ab-a21e-6e1065a5be16" (UID: "967aea57-432f-47ab-a21e-6e1065a5be16"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 06 14:29:41.889111 ip-10-0-136-25 kubenswrapper[2569]: I0706 14:29:41.889089 2569 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/967aea57-432f-47ab-a21e-6e1065a5be16-kube-api-access-gbqxl" (OuterVolumeSpecName: "kube-api-access-gbqxl") pod "967aea57-432f-47ab-a21e-6e1065a5be16" (UID: "967aea57-432f-47ab-a21e-6e1065a5be16"). InnerVolumeSpecName "kube-api-access-gbqxl". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 06 14:29:41.988176 ip-10-0-136-25 kubenswrapper[2569]: I0706 14:29:41.988117 2569 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gbqxl\" (UniqueName: \"kubernetes.io/projected/967aea57-432f-47ab-a21e-6e1065a5be16-kube-api-access-gbqxl\") on node \"ip-10-0-136-25.ec2.internal\" DevicePath \"\"" Jul 06 14:29:41.988176 ip-10-0-136-25 kubenswrapper[2569]: I0706 14:29:41.988167 2569 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/967aea57-432f-47ab-a21e-6e1065a5be16-dshm\") on node \"ip-10-0-136-25.ec2.internal\" DevicePath \"\"" Jul 06 14:29:41.988176 ip-10-0-136-25 kubenswrapper[2569]: I0706 14:29:41.988177 2569 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/967aea57-432f-47ab-a21e-6e1065a5be16-tls-certs\") on node \"ip-10-0-136-25.ec2.internal\" DevicePath \"\"" Jul 06 14:29:42.344524 ip-10-0-136-25 kubenswrapper[2569]: I0706 14:29:42.344456 2569 kuberuntime_container.go:872] "Killing container with a grace period" pod="ai-tenant-e2e-shared-b-eb526a/auth-test-model-eb526a-kserve-5d5886757f-nx7tw" podUID="0c5514a5-e417-4b9e-ad2d-d409189280a6" containerName="main" containerID="cri-o://4da486ae0538cfe5e3a4d5021b6f3b1a8178f6d787aa01bb78fe367f2f6f5dc7" gracePeriod=45 Jul 06 14:29:42.596451 ip-10-0-136-25 kubenswrapper[2569]: I0706 14:29:42.596388 2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-eb526a/auth-test-model-eb526a-kserve-5d5886757f-nx7tw" Jul 06 14:29:42.694011 ip-10-0-136-25 kubenswrapper[2569]: I0706 14:29:42.693981 2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0c5514a5-e417-4b9e-ad2d-d409189280a6-dshm\") pod \"0c5514a5-e417-4b9e-ad2d-d409189280a6\" (UID: \"0c5514a5-e417-4b9e-ad2d-d409189280a6\") " Jul 06 14:29:42.694202 ip-10-0-136-25 kubenswrapper[2569]: I0706 14:29:42.694039 2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0c5514a5-e417-4b9e-ad2d-d409189280a6-tls-certs\") pod \"0c5514a5-e417-4b9e-ad2d-d409189280a6\" (UID: \"0c5514a5-e417-4b9e-ad2d-d409189280a6\") " Jul 06 14:29:42.694202 ip-10-0-136-25 kubenswrapper[2569]: I0706 14:29:42.694060 2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0c5514a5-e417-4b9e-ad2d-d409189280a6-tmp-dir\") pod \"0c5514a5-e417-4b9e-ad2d-d409189280a6\" (UID: \"0c5514a5-e417-4b9e-ad2d-d409189280a6\") " Jul 06 14:29:42.694202 ip-10-0-136-25 kubenswrapper[2569]: I0706 14:29:42.694084 2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-7lbqw\" (UniqueName: \"kubernetes.io/projected/0c5514a5-e417-4b9e-ad2d-d409189280a6-kube-api-access-7lbqw\") pod \"0c5514a5-e417-4b9e-ad2d-d409189280a6\" (UID: \"0c5514a5-e417-4b9e-ad2d-d409189280a6\") " Jul 06 14:29:42.694202 ip-10-0-136-25 kubenswrapper[2569]: I0706 14:29:42.694104 2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0c5514a5-e417-4b9e-ad2d-d409189280a6-home\") pod \"0c5514a5-e417-4b9e-ad2d-d409189280a6\" (UID: \"0c5514a5-e417-4b9e-ad2d-d409189280a6\") " Jul 06 14:29:42.694202 ip-10-0-136-25 kubenswrapper[2569]: I0706 14:29:42.694153 2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0c5514a5-e417-4b9e-ad2d-d409189280a6-model-cache\") pod \"0c5514a5-e417-4b9e-ad2d-d409189280a6\" (UID: \"0c5514a5-e417-4b9e-ad2d-d409189280a6\") " Jul 06 14:29:42.694534 ip-10-0-136-25 kubenswrapper[2569]: I0706 14:29:42.694402 2569 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0c5514a5-e417-4b9e-ad2d-d409189280a6-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "0c5514a5-e417-4b9e-ad2d-d409189280a6" (UID: "0c5514a5-e417-4b9e-ad2d-d409189280a6"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 06 14:29:42.694534 ip-10-0-136-25 kubenswrapper[2569]: I0706 14:29:42.694416 2569 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0c5514a5-e417-4b9e-ad2d-d409189280a6-home" (OuterVolumeSpecName: "home") pod "0c5514a5-e417-4b9e-ad2d-d409189280a6" (UID: "0c5514a5-e417-4b9e-ad2d-d409189280a6"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 06 14:29:42.694534 ip-10-0-136-25 kubenswrapper[2569]: I0706 14:29:42.694489 2569 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0c5514a5-e417-4b9e-ad2d-d409189280a6-model-cache" (OuterVolumeSpecName: "model-cache") pod "0c5514a5-e417-4b9e-ad2d-d409189280a6" (UID: "0c5514a5-e417-4b9e-ad2d-d409189280a6"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 06 14:29:42.694676 ip-10-0-136-25 kubenswrapper[2569]: I0706 14:29:42.694561 2569 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0c5514a5-e417-4b9e-ad2d-d409189280a6-tmp-dir\") on node \"ip-10-0-136-25.ec2.internal\" DevicePath \"\"" Jul 06 14:29:42.694676 ip-10-0-136-25 kubenswrapper[2569]: I0706 14:29:42.694578 2569 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0c5514a5-e417-4b9e-ad2d-d409189280a6-home\") on node \"ip-10-0-136-25.ec2.internal\" DevicePath \"\"" Jul 06 14:29:42.696422 ip-10-0-136-25 kubenswrapper[2569]: I0706 14:29:42.696402 2569 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0c5514a5-e417-4b9e-ad2d-d409189280a6-dshm" (OuterVolumeSpecName: "dshm") pod "0c5514a5-e417-4b9e-ad2d-d409189280a6" (UID: "0c5514a5-e417-4b9e-ad2d-d409189280a6"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 06 14:29:42.696529 ip-10-0-136-25 kubenswrapper[2569]: I0706 14:29:42.696444 2569 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0c5514a5-e417-4b9e-ad2d-d409189280a6-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "0c5514a5-e417-4b9e-ad2d-d409189280a6" (UID: "0c5514a5-e417-4b9e-ad2d-d409189280a6"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 06 14:29:42.696529 ip-10-0-136-25 kubenswrapper[2569]: I0706 14:29:42.696476 2569 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0c5514a5-e417-4b9e-ad2d-d409189280a6-kube-api-access-7lbqw" (OuterVolumeSpecName: "kube-api-access-7lbqw") pod "0c5514a5-e417-4b9e-ad2d-d409189280a6" (UID: "0c5514a5-e417-4b9e-ad2d-d409189280a6"). InnerVolumeSpecName "kube-api-access-7lbqw". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 06 14:29:42.777377 ip-10-0-136-25 kubenswrapper[2569]: I0706 14:29:42.777338 2569 generic.go:356] "Generic (PLEG): container finished" podID="0c5514a5-e417-4b9e-ad2d-d409189280a6" containerID="4da486ae0538cfe5e3a4d5021b6f3b1a8178f6d787aa01bb78fe367f2f6f5dc7" exitCode=0 Jul 06 14:29:42.777547 ip-10-0-136-25 kubenswrapper[2569]: I0706 14:29:42.777383 2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-eb526a/auth-test-model-eb526a-kserve-5d5886757f-nx7tw" Jul 06 14:29:42.777547 ip-10-0-136-25 kubenswrapper[2569]: I0706 14:29:42.777405 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-eb526a/auth-test-model-eb526a-kserve-5d5886757f-nx7tw" event={"ID":"0c5514a5-e417-4b9e-ad2d-d409189280a6","Type":"ContainerDied","Data":"4da486ae0538cfe5e3a4d5021b6f3b1a8178f6d787aa01bb78fe367f2f6f5dc7"} Jul 06 14:29:42.777547 ip-10-0-136-25 kubenswrapper[2569]: I0706 14:29:42.777443 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-eb526a/auth-test-model-eb526a-kserve-5d5886757f-nx7tw" event={"ID":"0c5514a5-e417-4b9e-ad2d-d409189280a6","Type":"ContainerDied","Data":"dfe3379f327b4111e3edb4608a65a794d428086d6f99d02650693c1382a72d46"} Jul 06 14:29:42.777547 ip-10-0-136-25 kubenswrapper[2569]: I0706 14:29:42.777460 2569 scope.go:117] "RemoveContainer" containerID="4da486ae0538cfe5e3a4d5021b6f3b1a8178f6d787aa01bb78fe367f2f6f5dc7" Jul 06 14:29:42.777770 ip-10-0-136-25 kubenswrapper[2569]: I0706 14:29:42.777618 2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-2c86e0/auth-test-model-2c86e0-kserve-84899654f5-kd6k2" Jul 06 14:29:42.787966 ip-10-0-136-25 kubenswrapper[2569]: I0706 14:29:42.787949 2569 scope.go:117] "RemoveContainer" containerID="4da486ae0538cfe5e3a4d5021b6f3b1a8178f6d787aa01bb78fe367f2f6f5dc7" Jul 06 14:29:42.788374 ip-10-0-136-25 kubenswrapper[2569]: E0706 14:29:42.788351 2569 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4da486ae0538cfe5e3a4d5021b6f3b1a8178f6d787aa01bb78fe367f2f6f5dc7\": container with ID starting with 4da486ae0538cfe5e3a4d5021b6f3b1a8178f6d787aa01bb78fe367f2f6f5dc7 not found: ID does not exist" containerID="4da486ae0538cfe5e3a4d5021b6f3b1a8178f6d787aa01bb78fe367f2f6f5dc7" Jul 06 14:29:42.788473 ip-10-0-136-25 kubenswrapper[2569]: I0706 14:29:42.788380 2569 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4da486ae0538cfe5e3a4d5021b6f3b1a8178f6d787aa01bb78fe367f2f6f5dc7"} err="failed to get container status \"4da486ae0538cfe5e3a4d5021b6f3b1a8178f6d787aa01bb78fe367f2f6f5dc7\": rpc error: code = NotFound desc = could not find container \"4da486ae0538cfe5e3a4d5021b6f3b1a8178f6d787aa01bb78fe367f2f6f5dc7\": container with ID starting with 4da486ae0538cfe5e3a4d5021b6f3b1a8178f6d787aa01bb78fe367f2f6f5dc7 not found: ID does not exist" Jul 06 14:29:42.795386 ip-10-0-136-25 kubenswrapper[2569]: I0706 14:29:42.795353 2569 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0c5514a5-e417-4b9e-ad2d-d409189280a6-tls-certs\") on node \"ip-10-0-136-25.ec2.internal\" DevicePath \"\"" Jul 06 14:29:42.795386 ip-10-0-136-25 kubenswrapper[2569]: I0706 14:29:42.795383 2569 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7lbqw\" (UniqueName: \"kubernetes.io/projected/0c5514a5-e417-4b9e-ad2d-d409189280a6-kube-api-access-7lbqw\") on node \"ip-10-0-136-25.ec2.internal\" DevicePath \"\"" Jul 06 14:29:42.795386 ip-10-0-136-25 kubenswrapper[2569]: I0706 14:29:42.795395 2569 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0c5514a5-e417-4b9e-ad2d-d409189280a6-model-cache\") on node \"ip-10-0-136-25.ec2.internal\" DevicePath \"\"" Jul 06 14:29:42.795536 ip-10-0-136-25 kubenswrapper[2569]: I0706 14:29:42.795403 2569 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0c5514a5-e417-4b9e-ad2d-d409189280a6-dshm\") on node \"ip-10-0-136-25.ec2.internal\" DevicePath \"\"" Jul 06 14:29:42.798292 ip-10-0-136-25 kubenswrapper[2569]: I0706 14:29:42.798271 2569 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-eb526a/auth-test-model-eb526a-kserve-5d5886757f-nx7tw"] Jul 06 14:29:42.801540 ip-10-0-136-25 kubenswrapper[2569]: I0706 14:29:42.801503 2569 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-b-eb526a/auth-test-model-eb526a-kserve-5d5886757f-nx7tw"] Jul 06 14:29:42.810861 ip-10-0-136-25 kubenswrapper[2569]: I0706 14:29:42.810842 2569 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-2c86e0/auth-test-model-2c86e0-kserve-84899654f5-kd6k2"] Jul 06 14:29:42.813950 ip-10-0-136-25 kubenswrapper[2569]: I0706 14:29:42.813927 2569 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-a-2c86e0/auth-test-model-2c86e0-kserve-84899654f5-kd6k2"] Jul 06 14:29:42.940732 ip-10-0-136-25 kubenswrapper[2569]: I0706 14:29:42.940642 2569 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="0c5514a5-e417-4b9e-ad2d-d409189280a6" path="/var/lib/kubelet/pods/0c5514a5-e417-4b9e-ad2d-d409189280a6/volumes" Jul 06 14:29:42.941281 ip-10-0-136-25 kubenswrapper[2569]: I0706 14:29:42.941262 2569 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="967aea57-432f-47ab-a21e-6e1065a5be16" path="/var/lib/kubelet/pods/967aea57-432f-47ab-a21e-6e1065a5be16/volumes" Jul 06 14:29:43.652880 ip-10-0-136-25 kubenswrapper[2569]: I0706 14:29:43.652846 2569 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-a-2c86e0/sub-test-model-2c86e0-kserve-86f676b867-58bj2" Jul 06 14:29:43.653109 ip-10-0-136-25 kubenswrapper[2569]: I0706 14:29:43.652907 2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-a-2c86e0/sub-test-model-2c86e0-kserve-86f676b867-58bj2" Jul 06 14:29:43.665057 ip-10-0-136-25 kubenswrapper[2569]: I0706 14:29:43.665032 2569 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-a-2c86e0/sub-test-model-2c86e0-kserve-86f676b867-58bj2" Jul 06 14:29:43.795700 ip-10-0-136-25 kubenswrapper[2569]: I0706 14:29:43.795673 2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-a-2c86e0/sub-test-model-2c86e0-kserve-86f676b867-58bj2" Jul 06 14:29:57.341805 ip-10-0-136-25 kubenswrapper[2569]: I0706 14:29:57.341767 2569 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-b-eb526a/sub-test-model-eb526a-kserve-6b975b487c-fmv6t"] Jul 06 14:29:57.342331 ip-10-0-136-25 kubenswrapper[2569]: I0706 14:29:57.342123 2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0c5514a5-e417-4b9e-ad2d-d409189280a6" containerName="main" Jul 06 14:29:57.342331 ip-10-0-136-25 kubenswrapper[2569]: I0706 14:29:57.342133 2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="0c5514a5-e417-4b9e-ad2d-d409189280a6" containerName="main" Jul 06 14:29:57.342331 ip-10-0-136-25 kubenswrapper[2569]: I0706 14:29:57.342152 2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="967aea57-432f-47ab-a21e-6e1065a5be16" containerName="main" Jul 06 14:29:57.342331 ip-10-0-136-25 kubenswrapper[2569]: I0706 14:29:57.342158 2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="967aea57-432f-47ab-a21e-6e1065a5be16" containerName="main" Jul 06 14:29:57.342331 ip-10-0-136-25 kubenswrapper[2569]: I0706 14:29:57.342239 2569 memory_manager.go:368] "RemoveStaleState removing state" podUID="0c5514a5-e417-4b9e-ad2d-d409189280a6" containerName="main" Jul 06 14:29:57.342331 ip-10-0-136-25 kubenswrapper[2569]: I0706 14:29:57.342248 2569 memory_manager.go:368] "RemoveStaleState removing state" podUID="967aea57-432f-47ab-a21e-6e1065a5be16" containerName="main" Jul 06 14:29:57.346471 ip-10-0-136-25 kubenswrapper[2569]: I0706 14:29:57.346446 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-eb526a/sub-test-model-eb526a-kserve-6b975b487c-fmv6t" Jul 06 14:29:57.349108 ip-10-0-136-25 kubenswrapper[2569]: I0706 14:29:57.349086 2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-eb526a\"/\"sub-test-model-eb526a-kserve-self-signed-certs\"" Jul 06 14:29:57.349388 ip-10-0-136-25 kubenswrapper[2569]: I0706 14:29:57.349365 2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-eb526a\"/\"default-dockercfg-jnhmf\"" Jul 06 14:29:57.349488 ip-10-0-136-25 kubenswrapper[2569]: I0706 14:29:57.349404 2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-eb526a\"/\"kube-root-ca.crt\"" Jul 06 14:29:57.349488 ip-10-0-136-25 kubenswrapper[2569]: I0706 14:29:57.349447 2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-eb526a\"/\"openshift-service-ca.crt\"" Jul 06 14:29:57.352540 ip-10-0-136-25 kubenswrapper[2569]: I0706 14:29:57.352519 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-eb526a/sub-test-model-eb526a-kserve-6b975b487c-fmv6t"] Jul 06 14:29:57.434843 ip-10-0-136-25 kubenswrapper[2569]: I0706 14:29:57.434807 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7c74954a-aa61-48aa-8964-f0ec9c480401-dshm\") pod \"sub-test-model-eb526a-kserve-6b975b487c-fmv6t\" (UID: \"7c74954a-aa61-48aa-8964-f0ec9c480401\") " pod="ai-tenant-e2e-shared-b-eb526a/sub-test-model-eb526a-kserve-6b975b487c-fmv6t" Jul 06 14:29:57.435049 ip-10-0-136-25 kubenswrapper[2569]: I0706 14:29:57.434866 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7c74954a-aa61-48aa-8964-f0ec9c480401-home\") pod \"sub-test-model-eb526a-kserve-6b975b487c-fmv6t\" (UID: \"7c74954a-aa61-48aa-8964-f0ec9c480401\") " pod="ai-tenant-e2e-shared-b-eb526a/sub-test-model-eb526a-kserve-6b975b487c-fmv6t" Jul 06 14:29:57.435049 ip-10-0-136-25 kubenswrapper[2569]: I0706 14:29:57.434914 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7c74954a-aa61-48aa-8964-f0ec9c480401-tmp-dir\") pod \"sub-test-model-eb526a-kserve-6b975b487c-fmv6t\" (UID: \"7c74954a-aa61-48aa-8964-f0ec9c480401\") " pod="ai-tenant-e2e-shared-b-eb526a/sub-test-model-eb526a-kserve-6b975b487c-fmv6t" Jul 06 14:29:57.435049 ip-10-0-136-25 kubenswrapper[2569]: I0706 14:29:57.434979 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tb6bq\" (UniqueName: \"kubernetes.io/projected/7c74954a-aa61-48aa-8964-f0ec9c480401-kube-api-access-tb6bq\") pod \"sub-test-model-eb526a-kserve-6b975b487c-fmv6t\" (UID: \"7c74954a-aa61-48aa-8964-f0ec9c480401\") " pod="ai-tenant-e2e-shared-b-eb526a/sub-test-model-eb526a-kserve-6b975b487c-fmv6t" Jul 06 14:29:57.435049 ip-10-0-136-25 kubenswrapper[2569]: I0706 14:29:57.435037 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7c74954a-aa61-48aa-8964-f0ec9c480401-model-cache\") pod \"sub-test-model-eb526a-kserve-6b975b487c-fmv6t\" (UID: \"7c74954a-aa61-48aa-8964-f0ec9c480401\") " pod="ai-tenant-e2e-shared-b-eb526a/sub-test-model-eb526a-kserve-6b975b487c-fmv6t" Jul 06 14:29:57.435247 ip-10-0-136-25 kubenswrapper[2569]: I0706 14:29:57.435116 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7c74954a-aa61-48aa-8964-f0ec9c480401-tls-certs\") pod \"sub-test-model-eb526a-kserve-6b975b487c-fmv6t\" (UID: \"7c74954a-aa61-48aa-8964-f0ec9c480401\") " pod="ai-tenant-e2e-shared-b-eb526a/sub-test-model-eb526a-kserve-6b975b487c-fmv6t" Jul 06 14:29:57.536159 ip-10-0-136-25 kubenswrapper[2569]: I0706 14:29:57.536119 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7c74954a-aa61-48aa-8964-f0ec9c480401-model-cache\") pod \"sub-test-model-eb526a-kserve-6b975b487c-fmv6t\" (UID: \"7c74954a-aa61-48aa-8964-f0ec9c480401\") " pod="ai-tenant-e2e-shared-b-eb526a/sub-test-model-eb526a-kserve-6b975b487c-fmv6t" Jul 06 14:29:57.536367 ip-10-0-136-25 kubenswrapper[2569]: I0706 14:29:57.536170 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7c74954a-aa61-48aa-8964-f0ec9c480401-tls-certs\") pod \"sub-test-model-eb526a-kserve-6b975b487c-fmv6t\" (UID: \"7c74954a-aa61-48aa-8964-f0ec9c480401\") " pod="ai-tenant-e2e-shared-b-eb526a/sub-test-model-eb526a-kserve-6b975b487c-fmv6t" Jul 06 14:29:57.536367 ip-10-0-136-25 kubenswrapper[2569]: I0706 14:29:57.536295 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7c74954a-aa61-48aa-8964-f0ec9c480401-dshm\") pod \"sub-test-model-eb526a-kserve-6b975b487c-fmv6t\" (UID: \"7c74954a-aa61-48aa-8964-f0ec9c480401\") " pod="ai-tenant-e2e-shared-b-eb526a/sub-test-model-eb526a-kserve-6b975b487c-fmv6t" Jul 06 14:29:57.536457 ip-10-0-136-25 kubenswrapper[2569]: I0706 14:29:57.536395 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7c74954a-aa61-48aa-8964-f0ec9c480401-home\") pod \"sub-test-model-eb526a-kserve-6b975b487c-fmv6t\" (UID: \"7c74954a-aa61-48aa-8964-f0ec9c480401\") " pod="ai-tenant-e2e-shared-b-eb526a/sub-test-model-eb526a-kserve-6b975b487c-fmv6t" Jul 06 14:29:57.536457 ip-10-0-136-25 kubenswrapper[2569]: I0706 14:29:57.536421 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7c74954a-aa61-48aa-8964-f0ec9c480401-tmp-dir\") pod \"sub-test-model-eb526a-kserve-6b975b487c-fmv6t\" (UID: \"7c74954a-aa61-48aa-8964-f0ec9c480401\") " pod="ai-tenant-e2e-shared-b-eb526a/sub-test-model-eb526a-kserve-6b975b487c-fmv6t" Jul 06 14:29:57.536565 ip-10-0-136-25 kubenswrapper[2569]: I0706 14:29:57.536471 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tb6bq\" (UniqueName: \"kubernetes.io/projected/7c74954a-aa61-48aa-8964-f0ec9c480401-kube-api-access-tb6bq\") pod \"sub-test-model-eb526a-kserve-6b975b487c-fmv6t\" (UID: \"7c74954a-aa61-48aa-8964-f0ec9c480401\") " pod="ai-tenant-e2e-shared-b-eb526a/sub-test-model-eb526a-kserve-6b975b487c-fmv6t" Jul 06 14:29:57.536647 ip-10-0-136-25 kubenswrapper[2569]: I0706 14:29:57.536625 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7c74954a-aa61-48aa-8964-f0ec9c480401-model-cache\") pod \"sub-test-model-eb526a-kserve-6b975b487c-fmv6t\" (UID: \"7c74954a-aa61-48aa-8964-f0ec9c480401\") " pod="ai-tenant-e2e-shared-b-eb526a/sub-test-model-eb526a-kserve-6b975b487c-fmv6t" Jul 06 14:29:57.536754 ip-10-0-136-25 kubenswrapper[2569]: I0706 14:29:57.536735 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7c74954a-aa61-48aa-8964-f0ec9c480401-home\") pod \"sub-test-model-eb526a-kserve-6b975b487c-fmv6t\" (UID: \"7c74954a-aa61-48aa-8964-f0ec9c480401\") " pod="ai-tenant-e2e-shared-b-eb526a/sub-test-model-eb526a-kserve-6b975b487c-fmv6t" Jul 06 14:29:57.536818 ip-10-0-136-25 kubenswrapper[2569]: I0706 14:29:57.536789 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7c74954a-aa61-48aa-8964-f0ec9c480401-tmp-dir\") pod \"sub-test-model-eb526a-kserve-6b975b487c-fmv6t\" (UID: \"7c74954a-aa61-48aa-8964-f0ec9c480401\") " pod="ai-tenant-e2e-shared-b-eb526a/sub-test-model-eb526a-kserve-6b975b487c-fmv6t" Jul 06 14:29:57.538568 ip-10-0-136-25 kubenswrapper[2569]: I0706 14:29:57.538544 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7c74954a-aa61-48aa-8964-f0ec9c480401-dshm\") pod \"sub-test-model-eb526a-kserve-6b975b487c-fmv6t\" (UID: \"7c74954a-aa61-48aa-8964-f0ec9c480401\") " pod="ai-tenant-e2e-shared-b-eb526a/sub-test-model-eb526a-kserve-6b975b487c-fmv6t" Jul 06 14:29:57.538681 ip-10-0-136-25 kubenswrapper[2569]: I0706 14:29:57.538670 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7c74954a-aa61-48aa-8964-f0ec9c480401-tls-certs\") pod \"sub-test-model-eb526a-kserve-6b975b487c-fmv6t\" (UID: \"7c74954a-aa61-48aa-8964-f0ec9c480401\") " pod="ai-tenant-e2e-shared-b-eb526a/sub-test-model-eb526a-kserve-6b975b487c-fmv6t" Jul 06 14:29:57.543863 ip-10-0-136-25 kubenswrapper[2569]: I0706 14:29:57.543844 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tb6bq\" (UniqueName: \"kubernetes.io/projected/7c74954a-aa61-48aa-8964-f0ec9c480401-kube-api-access-tb6bq\") pod \"sub-test-model-eb526a-kserve-6b975b487c-fmv6t\" (UID: \"7c74954a-aa61-48aa-8964-f0ec9c480401\") " pod="ai-tenant-e2e-shared-b-eb526a/sub-test-model-eb526a-kserve-6b975b487c-fmv6t" Jul 06 14:29:57.657724 ip-10-0-136-25 kubenswrapper[2569]: I0706 14:29:57.657629 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-eb526a/sub-test-model-eb526a-kserve-6b975b487c-fmv6t" Jul 06 14:29:57.789587 ip-10-0-136-25 kubenswrapper[2569]: I0706 14:29:57.789548 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-eb526a/sub-test-model-eb526a-kserve-6b975b487c-fmv6t"] Jul 06 14:29:57.792160 ip-10-0-136-25 kubenswrapper[2569]: W0706 14:29:57.792130 2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod7c74954a_aa61_48aa_8964_f0ec9c480401.slice/crio-d9ba84547ca9bdf21c1c22fb35b3a3901bcb1d7d9e38e6bd2134059555e8fc8c WatchSource:0}: Error finding container d9ba84547ca9bdf21c1c22fb35b3a3901bcb1d7d9e38e6bd2134059555e8fc8c: Status 404 returned error can't find the container with id d9ba84547ca9bdf21c1c22fb35b3a3901bcb1d7d9e38e6bd2134059555e8fc8c Jul 06 14:29:57.850909 ip-10-0-136-25 kubenswrapper[2569]: I0706 14:29:57.850881 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-eb526a/sub-test-model-eb526a-kserve-6b975b487c-fmv6t" event={"ID":"7c74954a-aa61-48aa-8964-f0ec9c480401","Type":"ContainerStarted","Data":"d9ba84547ca9bdf21c1c22fb35b3a3901bcb1d7d9e38e6bd2134059555e8fc8c"} Jul 06 14:29:58.857454 ip-10-0-136-25 kubenswrapper[2569]: I0706 14:29:58.857413 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-eb526a/sub-test-model-eb526a-kserve-6b975b487c-fmv6t" event={"ID":"7c74954a-aa61-48aa-8964-f0ec9c480401","Type":"ContainerStarted","Data":"ac2aa7261f0a26e81eae78d34706eeddb881de2c0f03a939b88583de1730ccc3"} Jul 06 14:29:58.875282 ip-10-0-136-25 kubenswrapper[2569]: I0706 14:29:58.875204 2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-b-eb526a/sub-test-model-eb526a-kserve-6b975b487c-fmv6t" podStartSLOduration=1.8751849969999999 podStartE2EDuration="1.875184997s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-06 14:29:57 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 14:29:58.874541107 +0000 UTC m=+2374.465102916" watchObservedRunningTime="2026-07-06 14:29:58.875184997 +0000 UTC m=+2374.465746774" Jul 06 14:30:01.139137 ip-10-0-136-25 kubenswrapper[2569]: I0706 14:30:01.139096 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-g9qp6"] Jul 06 14:30:07.658120 ip-10-0-136-25 kubenswrapper[2569]: I0706 14:30:07.658083 2569 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-b-eb526a/sub-test-model-eb526a-kserve-6b975b487c-fmv6t" Jul 06 14:30:07.658120 ip-10-0-136-25 kubenswrapper[2569]: I0706 14:30:07.658130 2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-b-eb526a/sub-test-model-eb526a-kserve-6b975b487c-fmv6t" Jul 06 14:30:07.671056 ip-10-0-136-25 kubenswrapper[2569]: I0706 14:30:07.671029 2569 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-b-eb526a/sub-test-model-eb526a-kserve-6b975b487c-fmv6t" Jul 06 14:30:07.914123 ip-10-0-136-25 kubenswrapper[2569]: I0706 14:30:07.914039 2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-b-eb526a/sub-test-model-eb526a-kserve-6b975b487c-fmv6t" Jul 06 14:30:20.435980 ip-10-0-136-25 kubenswrapper[2569]: I0706 14:30:20.435941 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-g9qp6"] Jul 06 14:30:25.036165 ip-10-0-136-25 kubenswrapper[2569]: I0706 14:30:25.036136 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-68b757fb67-z2h95_dfffcdca-94d6-4732-815f-bcb533f538e9/console-operator/2.log" Jul 06 14:30:25.043034 ip-10-0-136-25 kubenswrapper[2569]: I0706 14:30:25.043008 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-68b757fb67-z2h95_dfffcdca-94d6-4732-815f-bcb533f538e9/console-operator/2.log" Jul 06 14:30:38.833614 ip-10-0-136-25 kubenswrapper[2569]: I0706 14:30:38.833573 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-g9qp6"] Jul 06 14:30:44.021528 ip-10-0-136-25 kubenswrapper[2569]: I0706 14:30:44.021488 2569 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-2c86e0/sub-test-model-2c86e0-kserve-86f676b867-58bj2"] Jul 06 14:30:44.756367 ip-10-0-136-25 kubenswrapper[2569]: I0706 14:30:44.756333 2569 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-eb526a/sub-test-model-eb526a-kserve-6b975b487c-fmv6t"] Jul 06 14:30:57.032348 ip-10-0-136-25 kubenswrapper[2569]: I0706 14:30:57.032261 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-g9qp6"] Jul 06 14:30:57.643867 ip-10-0-136-25 kubenswrapper[2569]: I0706 14:30:57.643834 2569 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-a-2c86e0/rate-test-model-2c86e0-kserve-7595955ff5-klsjd"] Jul 06 14:30:57.647227 ip-10-0-136-25 kubenswrapper[2569]: I0706 14:30:57.647178 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-2c86e0/rate-test-model-2c86e0-kserve-7595955ff5-klsjd" Jul 06 14:30:57.649582 ip-10-0-136-25 kubenswrapper[2569]: I0706 14:30:57.649557 2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-2c86e0\"/\"rate-test-model-2c86e0-kserve-self-signed-certs\"" Jul 06 14:30:57.655832 ip-10-0-136-25 kubenswrapper[2569]: I0706 14:30:57.655797 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-2c86e0/rate-test-model-2c86e0-kserve-7595955ff5-klsjd"] Jul 06 14:30:57.724432 ip-10-0-136-25 kubenswrapper[2569]: I0706 14:30:57.724391 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/cd6e9013-7033-4a64-b035-63140d84cbf7-tls-certs\") pod \"rate-test-model-2c86e0-kserve-7595955ff5-klsjd\" (UID: \"cd6e9013-7033-4a64-b035-63140d84cbf7\") " pod="ai-tenant-e2e-shared-a-2c86e0/rate-test-model-2c86e0-kserve-7595955ff5-klsjd" Jul 06 14:30:57.724432 ip-10-0-136-25 kubenswrapper[2569]: I0706 14:30:57.724434 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/cd6e9013-7033-4a64-b035-63140d84cbf7-home\") pod \"rate-test-model-2c86e0-kserve-7595955ff5-klsjd\" (UID: \"cd6e9013-7033-4a64-b035-63140d84cbf7\") " pod="ai-tenant-e2e-shared-a-2c86e0/rate-test-model-2c86e0-kserve-7595955ff5-klsjd" Jul 06 14:30:57.724668 ip-10-0-136-25 kubenswrapper[2569]: I0706 14:30:57.724477 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/cd6e9013-7033-4a64-b035-63140d84cbf7-dshm\") pod \"rate-test-model-2c86e0-kserve-7595955ff5-klsjd\" (UID: \"cd6e9013-7033-4a64-b035-63140d84cbf7\") " pod="ai-tenant-e2e-shared-a-2c86e0/rate-test-model-2c86e0-kserve-7595955ff5-klsjd" Jul 06 14:30:57.724668 ip-10-0-136-25 kubenswrapper[2569]: I0706 14:30:57.724563 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/cd6e9013-7033-4a64-b035-63140d84cbf7-tmp-dir\") pod \"rate-test-model-2c86e0-kserve-7595955ff5-klsjd\" (UID: \"cd6e9013-7033-4a64-b035-63140d84cbf7\") " pod="ai-tenant-e2e-shared-a-2c86e0/rate-test-model-2c86e0-kserve-7595955ff5-klsjd" Jul 06 14:30:57.724668 ip-10-0-136-25 kubenswrapper[2569]: I0706 14:30:57.724621 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/cd6e9013-7033-4a64-b035-63140d84cbf7-model-cache\") pod \"rate-test-model-2c86e0-kserve-7595955ff5-klsjd\" (UID: \"cd6e9013-7033-4a64-b035-63140d84cbf7\") " pod="ai-tenant-e2e-shared-a-2c86e0/rate-test-model-2c86e0-kserve-7595955ff5-klsjd" Jul 06 14:30:57.724668 ip-10-0-136-25 kubenswrapper[2569]: I0706 14:30:57.724659 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-m2bt2\" (UniqueName: \"kubernetes.io/projected/cd6e9013-7033-4a64-b035-63140d84cbf7-kube-api-access-m2bt2\") pod \"rate-test-model-2c86e0-kserve-7595955ff5-klsjd\" (UID: \"cd6e9013-7033-4a64-b035-63140d84cbf7\") " pod="ai-tenant-e2e-shared-a-2c86e0/rate-test-model-2c86e0-kserve-7595955ff5-klsjd" Jul 06 14:30:57.825747 ip-10-0-136-25 kubenswrapper[2569]: I0706 14:30:57.825701 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/cd6e9013-7033-4a64-b035-63140d84cbf7-tmp-dir\") pod \"rate-test-model-2c86e0-kserve-7595955ff5-klsjd\" (UID: \"cd6e9013-7033-4a64-b035-63140d84cbf7\") " pod="ai-tenant-e2e-shared-a-2c86e0/rate-test-model-2c86e0-kserve-7595955ff5-klsjd" Jul 06 14:30:57.825928 ip-10-0-136-25 kubenswrapper[2569]: I0706 14:30:57.825772 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/cd6e9013-7033-4a64-b035-63140d84cbf7-model-cache\") pod \"rate-test-model-2c86e0-kserve-7595955ff5-klsjd\" (UID: \"cd6e9013-7033-4a64-b035-63140d84cbf7\") " pod="ai-tenant-e2e-shared-a-2c86e0/rate-test-model-2c86e0-kserve-7595955ff5-klsjd" Jul 06 14:30:57.825928 ip-10-0-136-25 kubenswrapper[2569]: I0706 14:30:57.825811 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m2bt2\" (UniqueName: \"kubernetes.io/projected/cd6e9013-7033-4a64-b035-63140d84cbf7-kube-api-access-m2bt2\") pod \"rate-test-model-2c86e0-kserve-7595955ff5-klsjd\" (UID: \"cd6e9013-7033-4a64-b035-63140d84cbf7\") " pod="ai-tenant-e2e-shared-a-2c86e0/rate-test-model-2c86e0-kserve-7595955ff5-klsjd" Jul 06 14:30:57.825928 ip-10-0-136-25 kubenswrapper[2569]: I0706 14:30:57.825851 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/cd6e9013-7033-4a64-b035-63140d84cbf7-tls-certs\") pod \"rate-test-model-2c86e0-kserve-7595955ff5-klsjd\" (UID: \"cd6e9013-7033-4a64-b035-63140d84cbf7\") " pod="ai-tenant-e2e-shared-a-2c86e0/rate-test-model-2c86e0-kserve-7595955ff5-klsjd" Jul 06 14:30:57.825928 ip-10-0-136-25 kubenswrapper[2569]: I0706 14:30:57.825917 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/cd6e9013-7033-4a64-b035-63140d84cbf7-home\") pod \"rate-test-model-2c86e0-kserve-7595955ff5-klsjd\" (UID: \"cd6e9013-7033-4a64-b035-63140d84cbf7\") " pod="ai-tenant-e2e-shared-a-2c86e0/rate-test-model-2c86e0-kserve-7595955ff5-klsjd" Jul 06 14:30:57.826082 ip-10-0-136-25 kubenswrapper[2569]: I0706 14:30:57.825985 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/cd6e9013-7033-4a64-b035-63140d84cbf7-dshm\") pod \"rate-test-model-2c86e0-kserve-7595955ff5-klsjd\" (UID: \"cd6e9013-7033-4a64-b035-63140d84cbf7\") " pod="ai-tenant-e2e-shared-a-2c86e0/rate-test-model-2c86e0-kserve-7595955ff5-klsjd" Jul 06 14:30:57.826205 ip-10-0-136-25 kubenswrapper[2569]: I0706 14:30:57.826186 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/cd6e9013-7033-4a64-b035-63140d84cbf7-tmp-dir\") pod \"rate-test-model-2c86e0-kserve-7595955ff5-klsjd\" (UID: \"cd6e9013-7033-4a64-b035-63140d84cbf7\") " pod="ai-tenant-e2e-shared-a-2c86e0/rate-test-model-2c86e0-kserve-7595955ff5-klsjd" Jul 06 14:30:57.826314 ip-10-0-136-25 kubenswrapper[2569]: I0706 14:30:57.826251 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/cd6e9013-7033-4a64-b035-63140d84cbf7-model-cache\") pod \"rate-test-model-2c86e0-kserve-7595955ff5-klsjd\" (UID: \"cd6e9013-7033-4a64-b035-63140d84cbf7\") " pod="ai-tenant-e2e-shared-a-2c86e0/rate-test-model-2c86e0-kserve-7595955ff5-klsjd" Jul 06 14:30:57.826368 ip-10-0-136-25 kubenswrapper[2569]: I0706 14:30:57.826312 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/cd6e9013-7033-4a64-b035-63140d84cbf7-home\") pod \"rate-test-model-2c86e0-kserve-7595955ff5-klsjd\" (UID: \"cd6e9013-7033-4a64-b035-63140d84cbf7\") " pod="ai-tenant-e2e-shared-a-2c86e0/rate-test-model-2c86e0-kserve-7595955ff5-klsjd" Jul 06 14:30:57.828186 ip-10-0-136-25 kubenswrapper[2569]: I0706 14:30:57.828157 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/cd6e9013-7033-4a64-b035-63140d84cbf7-dshm\") pod \"rate-test-model-2c86e0-kserve-7595955ff5-klsjd\" (UID: \"cd6e9013-7033-4a64-b035-63140d84cbf7\") " pod="ai-tenant-e2e-shared-a-2c86e0/rate-test-model-2c86e0-kserve-7595955ff5-klsjd" Jul 06 14:30:57.828347 ip-10-0-136-25 kubenswrapper[2569]: I0706 14:30:57.828326 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/cd6e9013-7033-4a64-b035-63140d84cbf7-tls-certs\") pod \"rate-test-model-2c86e0-kserve-7595955ff5-klsjd\" (UID: \"cd6e9013-7033-4a64-b035-63140d84cbf7\") " pod="ai-tenant-e2e-shared-a-2c86e0/rate-test-model-2c86e0-kserve-7595955ff5-klsjd" Jul 06 14:30:57.834504 ip-10-0-136-25 kubenswrapper[2569]: I0706 14:30:57.834487 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-m2bt2\" (UniqueName: \"kubernetes.io/projected/cd6e9013-7033-4a64-b035-63140d84cbf7-kube-api-access-m2bt2\") pod \"rate-test-model-2c86e0-kserve-7595955ff5-klsjd\" (UID: \"cd6e9013-7033-4a64-b035-63140d84cbf7\") " pod="ai-tenant-e2e-shared-a-2c86e0/rate-test-model-2c86e0-kserve-7595955ff5-klsjd" Jul 06 14:30:57.958120 ip-10-0-136-25 kubenswrapper[2569]: I0706 14:30:57.958024 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-2c86e0/rate-test-model-2c86e0-kserve-7595955ff5-klsjd" Jul 06 14:30:58.094328 ip-10-0-136-25 kubenswrapper[2569]: I0706 14:30:58.094301 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-2c86e0/rate-test-model-2c86e0-kserve-7595955ff5-klsjd"] Jul 06 14:30:58.096696 ip-10-0-136-25 kubenswrapper[2569]: W0706 14:30:58.096664 2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podcd6e9013_7033_4a64_b035_63140d84cbf7.slice/crio-b0adc793c8c6bc9d9d363db48de015c238154dd642d087d7e09866151a9cf565 WatchSource:0}: Error finding container b0adc793c8c6bc9d9d363db48de015c238154dd642d087d7e09866151a9cf565: Status 404 returned error can't find the container with id b0adc793c8c6bc9d9d363db48de015c238154dd642d087d7e09866151a9cf565 Jul 06 14:30:58.147331 ip-10-0-136-25 kubenswrapper[2569]: I0706 14:30:58.147301 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-2c86e0/rate-test-model-2c86e0-kserve-7595955ff5-klsjd" event={"ID":"cd6e9013-7033-4a64-b035-63140d84cbf7","Type":"ContainerStarted","Data":"b0adc793c8c6bc9d9d363db48de015c238154dd642d087d7e09866151a9cf565"} Jul 06 14:30:59.043226 ip-10-0-136-25 kubenswrapper[2569]: I0706 14:30:59.043172 2569 kuberuntime_container.go:872] "Killing container with a grace period" pod="ai-tenant-e2e-shared-a-2c86e0/sub-test-model-2c86e0-kserve-86f676b867-58bj2" podUID="f03ec316-06ae-470e-ad36-337dd335f037" containerName="main" containerID="cri-o://76b42082ec511d29de6f9b9d08f727d3ecbf9a5e03eb8ebadada25af17900055" gracePeriod=45 Jul 06 14:30:59.153525 ip-10-0-136-25 kubenswrapper[2569]: I0706 14:30:59.153487 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-2c86e0/rate-test-model-2c86e0-kserve-7595955ff5-klsjd" event={"ID":"cd6e9013-7033-4a64-b035-63140d84cbf7","Type":"ContainerStarted","Data":"ef41db9c42a49db5790fb4c980671d6688760ccbf86a9b7f21b9972457e1720b"} Jul 06 14:30:59.170749 ip-10-0-136-25 kubenswrapper[2569]: I0706 14:30:59.170696 2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-a-2c86e0/rate-test-model-2c86e0-kserve-7595955ff5-klsjd" podStartSLOduration=2.170679573 podStartE2EDuration="2.170679573s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-06 14:30:57 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 14:30:59.169699183 +0000 UTC m=+2434.760260981" watchObservedRunningTime="2026-07-06 14:30:59.170679573 +0000 UTC m=+2434.761241349" Jul 06 14:30:59.292281 ip-10-0-136-25 kubenswrapper[2569]: I0706 14:30:59.292254 2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-2c86e0/sub-test-model-2c86e0-kserve-86f676b867-58bj2" Jul 06 14:30:59.441569 ip-10-0-136-25 kubenswrapper[2569]: I0706 14:30:59.441474 2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f03ec316-06ae-470e-ad36-337dd335f037-model-cache\") pod \"f03ec316-06ae-470e-ad36-337dd335f037\" (UID: \"f03ec316-06ae-470e-ad36-337dd335f037\") " Jul 06 14:30:59.441569 ip-10-0-136-25 kubenswrapper[2569]: I0706 14:30:59.441558 2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f03ec316-06ae-470e-ad36-337dd335f037-home\") pod \"f03ec316-06ae-470e-ad36-337dd335f037\" (UID: \"f03ec316-06ae-470e-ad36-337dd335f037\") " Jul 06 14:30:59.441764 ip-10-0-136-25 kubenswrapper[2569]: I0706 14:30:59.441593 2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f03ec316-06ae-470e-ad36-337dd335f037-tmp-dir\") pod \"f03ec316-06ae-470e-ad36-337dd335f037\" (UID: \"f03ec316-06ae-470e-ad36-337dd335f037\") " Jul 06 14:30:59.441764 ip-10-0-136-25 kubenswrapper[2569]: I0706 14:30:59.441618 2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f03ec316-06ae-470e-ad36-337dd335f037-dshm\") pod \"f03ec316-06ae-470e-ad36-337dd335f037\" (UID: \"f03ec316-06ae-470e-ad36-337dd335f037\") " Jul 06 14:30:59.441764 ip-10-0-136-25 kubenswrapper[2569]: I0706 14:30:59.441677 2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-jxtf9\" (UniqueName: \"kubernetes.io/projected/f03ec316-06ae-470e-ad36-337dd335f037-kube-api-access-jxtf9\") pod \"f03ec316-06ae-470e-ad36-337dd335f037\" (UID: \"f03ec316-06ae-470e-ad36-337dd335f037\") " Jul 06 14:30:59.441764 ip-10-0-136-25 kubenswrapper[2569]: I0706 14:30:59.441705 2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f03ec316-06ae-470e-ad36-337dd335f037-tls-certs\") pod \"f03ec316-06ae-470e-ad36-337dd335f037\" (UID: \"f03ec316-06ae-470e-ad36-337dd335f037\") " Jul 06 14:30:59.441962 ip-10-0-136-25 kubenswrapper[2569]: I0706 14:30:59.441810 2569 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f03ec316-06ae-470e-ad36-337dd335f037-model-cache" (OuterVolumeSpecName: "model-cache") pod "f03ec316-06ae-470e-ad36-337dd335f037" (UID: "f03ec316-06ae-470e-ad36-337dd335f037"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 06 14:30:59.441962 ip-10-0-136-25 kubenswrapper[2569]: I0706 14:30:59.441856 2569 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f03ec316-06ae-470e-ad36-337dd335f037-home" (OuterVolumeSpecName: "home") pod "f03ec316-06ae-470e-ad36-337dd335f037" (UID: "f03ec316-06ae-470e-ad36-337dd335f037"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 06 14:30:59.441962 ip-10-0-136-25 kubenswrapper[2569]: I0706 14:30:59.441904 2569 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f03ec316-06ae-470e-ad36-337dd335f037-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "f03ec316-06ae-470e-ad36-337dd335f037" (UID: "f03ec316-06ae-470e-ad36-337dd335f037"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 06 14:30:59.442129 ip-10-0-136-25 kubenswrapper[2569]: I0706 14:30:59.442071 2569 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f03ec316-06ae-470e-ad36-337dd335f037-home\") on node \"ip-10-0-136-25.ec2.internal\" DevicePath \"\"" Jul 06 14:30:59.442129 ip-10-0-136-25 kubenswrapper[2569]: I0706 14:30:59.442086 2569 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f03ec316-06ae-470e-ad36-337dd335f037-tmp-dir\") on node \"ip-10-0-136-25.ec2.internal\" DevicePath \"\"" Jul 06 14:30:59.442129 ip-10-0-136-25 kubenswrapper[2569]: I0706 14:30:59.442095 2569 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f03ec316-06ae-470e-ad36-337dd335f037-model-cache\") on node \"ip-10-0-136-25.ec2.internal\" DevicePath \"\"" Jul 06 14:30:59.443817 ip-10-0-136-25 kubenswrapper[2569]: I0706 14:30:59.443789 2569 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f03ec316-06ae-470e-ad36-337dd335f037-kube-api-access-jxtf9" (OuterVolumeSpecName: "kube-api-access-jxtf9") pod "f03ec316-06ae-470e-ad36-337dd335f037" (UID: "f03ec316-06ae-470e-ad36-337dd335f037"). InnerVolumeSpecName "kube-api-access-jxtf9". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 06 14:30:59.443902 ip-10-0-136-25 kubenswrapper[2569]: I0706 14:30:59.443849 2569 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f03ec316-06ae-470e-ad36-337dd335f037-dshm" (OuterVolumeSpecName: "dshm") pod "f03ec316-06ae-470e-ad36-337dd335f037" (UID: "f03ec316-06ae-470e-ad36-337dd335f037"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 06 14:30:59.443942 ip-10-0-136-25 kubenswrapper[2569]: I0706 14:30:59.443908 2569 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f03ec316-06ae-470e-ad36-337dd335f037-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "f03ec316-06ae-470e-ad36-337dd335f037" (UID: "f03ec316-06ae-470e-ad36-337dd335f037"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 06 14:30:59.543374 ip-10-0-136-25 kubenswrapper[2569]: I0706 14:30:59.543330 2569 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f03ec316-06ae-470e-ad36-337dd335f037-dshm\") on node \"ip-10-0-136-25.ec2.internal\" DevicePath \"\"" Jul 06 14:30:59.543374 ip-10-0-136-25 kubenswrapper[2569]: I0706 14:30:59.543364 2569 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jxtf9\" (UniqueName: \"kubernetes.io/projected/f03ec316-06ae-470e-ad36-337dd335f037-kube-api-access-jxtf9\") on node \"ip-10-0-136-25.ec2.internal\" DevicePath \"\"" Jul 06 14:30:59.543374 ip-10-0-136-25 kubenswrapper[2569]: I0706 14:30:59.543379 2569 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f03ec316-06ae-470e-ad36-337dd335f037-tls-certs\") on node \"ip-10-0-136-25.ec2.internal\" DevicePath \"\"" Jul 06 14:30:59.777860 ip-10-0-136-25 kubenswrapper[2569]: I0706 14:30:59.777796 2569 kuberuntime_container.go:872] "Killing container with a grace period" pod="ai-tenant-e2e-shared-b-eb526a/sub-test-model-eb526a-kserve-6b975b487c-fmv6t" podUID="7c74954a-aa61-48aa-8964-f0ec9c480401" containerName="main" containerID="cri-o://ac2aa7261f0a26e81eae78d34706eeddb881de2c0f03a939b88583de1730ccc3" gracePeriod=45 Jul 06 14:31:00.028854 ip-10-0-136-25 kubenswrapper[2569]: I0706 14:31:00.028787 2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-eb526a/sub-test-model-eb526a-kserve-6b975b487c-fmv6t" Jul 06 14:31:00.148613 ip-10-0-136-25 kubenswrapper[2569]: I0706 14:31:00.148575 2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-tb6bq\" (UniqueName: \"kubernetes.io/projected/7c74954a-aa61-48aa-8964-f0ec9c480401-kube-api-access-tb6bq\") pod \"7c74954a-aa61-48aa-8964-f0ec9c480401\" (UID: \"7c74954a-aa61-48aa-8964-f0ec9c480401\") " Jul 06 14:31:00.148792 ip-10-0-136-25 kubenswrapper[2569]: I0706 14:31:00.148645 2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7c74954a-aa61-48aa-8964-f0ec9c480401-dshm\") pod \"7c74954a-aa61-48aa-8964-f0ec9c480401\" (UID: \"7c74954a-aa61-48aa-8964-f0ec9c480401\") " Jul 06 14:31:00.148792 ip-10-0-136-25 kubenswrapper[2569]: I0706 14:31:00.148676 2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7c74954a-aa61-48aa-8964-f0ec9c480401-tmp-dir\") pod \"7c74954a-aa61-48aa-8964-f0ec9c480401\" (UID: \"7c74954a-aa61-48aa-8964-f0ec9c480401\") " Jul 06 14:31:00.148792 ip-10-0-136-25 kubenswrapper[2569]: I0706 14:31:00.148708 2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7c74954a-aa61-48aa-8964-f0ec9c480401-model-cache\") pod \"7c74954a-aa61-48aa-8964-f0ec9c480401\" (UID: \"7c74954a-aa61-48aa-8964-f0ec9c480401\") " Jul 06 14:31:00.148792 ip-10-0-136-25 kubenswrapper[2569]: I0706 14:31:00.148726 2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7c74954a-aa61-48aa-8964-f0ec9c480401-tls-certs\") pod \"7c74954a-aa61-48aa-8964-f0ec9c480401\" (UID: \"7c74954a-aa61-48aa-8964-f0ec9c480401\") " Jul 06 14:31:00.148792 ip-10-0-136-25 kubenswrapper[2569]: I0706 14:31:00.148758 2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7c74954a-aa61-48aa-8964-f0ec9c480401-home\") pod \"7c74954a-aa61-48aa-8964-f0ec9c480401\" (UID: \"7c74954a-aa61-48aa-8964-f0ec9c480401\") " Jul 06 14:31:00.149089 ip-10-0-136-25 kubenswrapper[2569]: I0706 14:31:00.148998 2569 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7c74954a-aa61-48aa-8964-f0ec9c480401-model-cache" (OuterVolumeSpecName: "model-cache") pod "7c74954a-aa61-48aa-8964-f0ec9c480401" (UID: "7c74954a-aa61-48aa-8964-f0ec9c480401"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 06 14:31:00.149089 ip-10-0-136-25 kubenswrapper[2569]: I0706 14:31:00.149030 2569 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7c74954a-aa61-48aa-8964-f0ec9c480401-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "7c74954a-aa61-48aa-8964-f0ec9c480401" (UID: "7c74954a-aa61-48aa-8964-f0ec9c480401"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 06 14:31:00.149181 ip-10-0-136-25 kubenswrapper[2569]: I0706 14:31:00.149079 2569 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7c74954a-aa61-48aa-8964-f0ec9c480401-home" (OuterVolumeSpecName: "home") pod "7c74954a-aa61-48aa-8964-f0ec9c480401" (UID: "7c74954a-aa61-48aa-8964-f0ec9c480401"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 06 14:31:00.149181 ip-10-0-136-25 kubenswrapper[2569]: I0706 14:31:00.149158 2569 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7c74954a-aa61-48aa-8964-f0ec9c480401-tmp-dir\") on node \"ip-10-0-136-25.ec2.internal\" DevicePath \"\"" Jul 06 14:31:00.149181 ip-10-0-136-25 kubenswrapper[2569]: I0706 14:31:00.149176 2569 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7c74954a-aa61-48aa-8964-f0ec9c480401-model-cache\") on node \"ip-10-0-136-25.ec2.internal\" DevicePath \"\"" Jul 06 14:31:00.150917 ip-10-0-136-25 kubenswrapper[2569]: I0706 14:31:00.150891 2569 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7c74954a-aa61-48aa-8964-f0ec9c480401-dshm" (OuterVolumeSpecName: "dshm") pod "7c74954a-aa61-48aa-8964-f0ec9c480401" (UID: "7c74954a-aa61-48aa-8964-f0ec9c480401"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 06 14:31:00.151035 ip-10-0-136-25 kubenswrapper[2569]: I0706 14:31:00.150966 2569 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7c74954a-aa61-48aa-8964-f0ec9c480401-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "7c74954a-aa61-48aa-8964-f0ec9c480401" (UID: "7c74954a-aa61-48aa-8964-f0ec9c480401"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 06 14:31:00.151035 ip-10-0-136-25 kubenswrapper[2569]: I0706 14:31:00.151001 2569 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7c74954a-aa61-48aa-8964-f0ec9c480401-kube-api-access-tb6bq" (OuterVolumeSpecName: "kube-api-access-tb6bq") pod "7c74954a-aa61-48aa-8964-f0ec9c480401" (UID: "7c74954a-aa61-48aa-8964-f0ec9c480401"). InnerVolumeSpecName "kube-api-access-tb6bq". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 06 14:31:00.160963 ip-10-0-136-25 kubenswrapper[2569]: I0706 14:31:00.160937 2569 generic.go:356] "Generic (PLEG): container finished" podID="7c74954a-aa61-48aa-8964-f0ec9c480401" containerID="ac2aa7261f0a26e81eae78d34706eeddb881de2c0f03a939b88583de1730ccc3" exitCode=0 Jul 06 14:31:00.161400 ip-10-0-136-25 kubenswrapper[2569]: I0706 14:31:00.160982 2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-eb526a/sub-test-model-eb526a-kserve-6b975b487c-fmv6t" Jul 06 14:31:00.161400 ip-10-0-136-25 kubenswrapper[2569]: I0706 14:31:00.161024 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-eb526a/sub-test-model-eb526a-kserve-6b975b487c-fmv6t" event={"ID":"7c74954a-aa61-48aa-8964-f0ec9c480401","Type":"ContainerDied","Data":"ac2aa7261f0a26e81eae78d34706eeddb881de2c0f03a939b88583de1730ccc3"} Jul 06 14:31:00.161400 ip-10-0-136-25 kubenswrapper[2569]: I0706 14:31:00.161075 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-eb526a/sub-test-model-eb526a-kserve-6b975b487c-fmv6t" event={"ID":"7c74954a-aa61-48aa-8964-f0ec9c480401","Type":"ContainerDied","Data":"d9ba84547ca9bdf21c1c22fb35b3a3901bcb1d7d9e38e6bd2134059555e8fc8c"} Jul 06 14:31:00.161400 ip-10-0-136-25 kubenswrapper[2569]: I0706 14:31:00.161102 2569 scope.go:117] "RemoveContainer" containerID="ac2aa7261f0a26e81eae78d34706eeddb881de2c0f03a939b88583de1730ccc3" Jul 06 14:31:00.162682 ip-10-0-136-25 kubenswrapper[2569]: I0706 14:31:00.162656 2569 generic.go:356] "Generic (PLEG): container finished" podID="f03ec316-06ae-470e-ad36-337dd335f037" containerID="76b42082ec511d29de6f9b9d08f727d3ecbf9a5e03eb8ebadada25af17900055" exitCode=0 Jul 06 14:31:00.162773 ip-10-0-136-25 kubenswrapper[2569]: I0706 14:31:00.162703 2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-2c86e0/sub-test-model-2c86e0-kserve-86f676b867-58bj2" Jul 06 14:31:00.162773 ip-10-0-136-25 kubenswrapper[2569]: I0706 14:31:00.162748 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-2c86e0/sub-test-model-2c86e0-kserve-86f676b867-58bj2" event={"ID":"f03ec316-06ae-470e-ad36-337dd335f037","Type":"ContainerDied","Data":"76b42082ec511d29de6f9b9d08f727d3ecbf9a5e03eb8ebadada25af17900055"} Jul 06 14:31:00.162887 ip-10-0-136-25 kubenswrapper[2569]: I0706 14:31:00.162782 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-2c86e0/sub-test-model-2c86e0-kserve-86f676b867-58bj2" event={"ID":"f03ec316-06ae-470e-ad36-337dd335f037","Type":"ContainerDied","Data":"49a5377eefe4a80ac6739dd27b27a65dc294edc1431625601de336a5b896e026"} Jul 06 14:31:00.174701 ip-10-0-136-25 kubenswrapper[2569]: I0706 14:31:00.174684 2569 scope.go:117] "RemoveContainer" containerID="ac2aa7261f0a26e81eae78d34706eeddb881de2c0f03a939b88583de1730ccc3" Jul 06 14:31:00.175008 ip-10-0-136-25 kubenswrapper[2569]: E0706 14:31:00.174987 2569 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ac2aa7261f0a26e81eae78d34706eeddb881de2c0f03a939b88583de1730ccc3\": container with ID starting with ac2aa7261f0a26e81eae78d34706eeddb881de2c0f03a939b88583de1730ccc3 not found: ID does not exist" containerID="ac2aa7261f0a26e81eae78d34706eeddb881de2c0f03a939b88583de1730ccc3" Jul 06 14:31:00.175062 ip-10-0-136-25 kubenswrapper[2569]: I0706 14:31:00.175018 2569 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ac2aa7261f0a26e81eae78d34706eeddb881de2c0f03a939b88583de1730ccc3"} err="failed to get container status \"ac2aa7261f0a26e81eae78d34706eeddb881de2c0f03a939b88583de1730ccc3\": rpc error: code = NotFound desc = could not find container \"ac2aa7261f0a26e81eae78d34706eeddb881de2c0f03a939b88583de1730ccc3\": container with ID starting with ac2aa7261f0a26e81eae78d34706eeddb881de2c0f03a939b88583de1730ccc3 not found: ID does not exist" Jul 06 14:31:00.175062 ip-10-0-136-25 kubenswrapper[2569]: I0706 14:31:00.175034 2569 scope.go:117] "RemoveContainer" containerID="76b42082ec511d29de6f9b9d08f727d3ecbf9a5e03eb8ebadada25af17900055" Jul 06 14:31:00.182843 ip-10-0-136-25 kubenswrapper[2569]: I0706 14:31:00.182820 2569 scope.go:117] "RemoveContainer" containerID="76b42082ec511d29de6f9b9d08f727d3ecbf9a5e03eb8ebadada25af17900055" Jul 06 14:31:00.183328 ip-10-0-136-25 kubenswrapper[2569]: E0706 14:31:00.183271 2569 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"76b42082ec511d29de6f9b9d08f727d3ecbf9a5e03eb8ebadada25af17900055\": container with ID starting with 76b42082ec511d29de6f9b9d08f727d3ecbf9a5e03eb8ebadada25af17900055 not found: ID does not exist" containerID="76b42082ec511d29de6f9b9d08f727d3ecbf9a5e03eb8ebadada25af17900055" Jul 06 14:31:00.183328 ip-10-0-136-25 kubenswrapper[2569]: I0706 14:31:00.183308 2569 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"76b42082ec511d29de6f9b9d08f727d3ecbf9a5e03eb8ebadada25af17900055"} err="failed to get container status \"76b42082ec511d29de6f9b9d08f727d3ecbf9a5e03eb8ebadada25af17900055\": rpc error: code = NotFound desc = could not find container \"76b42082ec511d29de6f9b9d08f727d3ecbf9a5e03eb8ebadada25af17900055\": container with ID starting with 76b42082ec511d29de6f9b9d08f727d3ecbf9a5e03eb8ebadada25af17900055 not found: ID does not exist" Jul 06 14:31:00.185629 ip-10-0-136-25 kubenswrapper[2569]: I0706 14:31:00.185601 2569 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-eb526a/sub-test-model-eb526a-kserve-6b975b487c-fmv6t"] Jul 06 14:31:00.187247 ip-10-0-136-25 kubenswrapper[2569]: I0706 14:31:00.187227 2569 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-b-eb526a/sub-test-model-eb526a-kserve-6b975b487c-fmv6t"] Jul 06 14:31:00.195811 ip-10-0-136-25 kubenswrapper[2569]: I0706 14:31:00.195782 2569 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-2c86e0/sub-test-model-2c86e0-kserve-86f676b867-58bj2"] Jul 06 14:31:00.198116 ip-10-0-136-25 kubenswrapper[2569]: I0706 14:31:00.198093 2569 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-a-2c86e0/sub-test-model-2c86e0-kserve-86f676b867-58bj2"] Jul 06 14:31:00.249940 ip-10-0-136-25 kubenswrapper[2569]: I0706 14:31:00.249905 2569 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tb6bq\" (UniqueName: \"kubernetes.io/projected/7c74954a-aa61-48aa-8964-f0ec9c480401-kube-api-access-tb6bq\") on node \"ip-10-0-136-25.ec2.internal\" DevicePath \"\"" Jul 06 14:31:00.249940 ip-10-0-136-25 kubenswrapper[2569]: I0706 14:31:00.249935 2569 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7c74954a-aa61-48aa-8964-f0ec9c480401-dshm\") on node \"ip-10-0-136-25.ec2.internal\" DevicePath \"\"" Jul 06 14:31:00.249940 ip-10-0-136-25 kubenswrapper[2569]: I0706 14:31:00.249945 2569 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7c74954a-aa61-48aa-8964-f0ec9c480401-tls-certs\") on node \"ip-10-0-136-25.ec2.internal\" DevicePath \"\"" Jul 06 14:31:00.250138 ip-10-0-136-25 kubenswrapper[2569]: I0706 14:31:00.249952 2569 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7c74954a-aa61-48aa-8964-f0ec9c480401-home\") on node \"ip-10-0-136-25.ec2.internal\" DevicePath \"\"" Jul 06 14:31:00.940221 ip-10-0-136-25 kubenswrapper[2569]: I0706 14:31:00.940186 2569 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="7c74954a-aa61-48aa-8964-f0ec9c480401" path="/var/lib/kubelet/pods/7c74954a-aa61-48aa-8964-f0ec9c480401/volumes" Jul 06 14:31:00.940622 ip-10-0-136-25 kubenswrapper[2569]: I0706 14:31:00.940609 2569 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="f03ec316-06ae-470e-ad36-337dd335f037" path="/var/lib/kubelet/pods/f03ec316-06ae-470e-ad36-337dd335f037/volumes" Jul 06 14:31:07.958324 ip-10-0-136-25 kubenswrapper[2569]: I0706 14:31:07.958279 2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-a-2c86e0/rate-test-model-2c86e0-kserve-7595955ff5-klsjd" Jul 06 14:31:07.958324 ip-10-0-136-25 kubenswrapper[2569]: I0706 14:31:07.958334 2569 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-a-2c86e0/rate-test-model-2c86e0-kserve-7595955ff5-klsjd" Jul 06 14:31:07.971263 ip-10-0-136-25 kubenswrapper[2569]: I0706 14:31:07.971235 2569 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-a-2c86e0/rate-test-model-2c86e0-kserve-7595955ff5-klsjd" Jul 06 14:31:08.221780 ip-10-0-136-25 kubenswrapper[2569]: I0706 14:31:08.221680 2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-a-2c86e0/rate-test-model-2c86e0-kserve-7595955ff5-klsjd" Jul 06 14:31:08.632935 ip-10-0-136-25 kubenswrapper[2569]: I0706 14:31:08.632901 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-g9qp6"] Jul 06 14:31:25.051182 ip-10-0-136-25 kubenswrapper[2569]: I0706 14:31:25.051150 2569 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-b-eb526a/rate-test-model-eb526a-kserve-7b96f4886c-s7g2n"] Jul 06 14:31:25.051563 ip-10-0-136-25 kubenswrapper[2569]: I0706 14:31:25.051555 2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7c74954a-aa61-48aa-8964-f0ec9c480401" containerName="main" Jul 06 14:31:25.051606 ip-10-0-136-25 kubenswrapper[2569]: I0706 14:31:25.051568 2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="7c74954a-aa61-48aa-8964-f0ec9c480401" containerName="main" Jul 06 14:31:25.051606 ip-10-0-136-25 kubenswrapper[2569]: I0706 14:31:25.051585 2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f03ec316-06ae-470e-ad36-337dd335f037" containerName="main" Jul 06 14:31:25.051673 ip-10-0-136-25 kubenswrapper[2569]: I0706 14:31:25.051605 2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="f03ec316-06ae-470e-ad36-337dd335f037" containerName="main" Jul 06 14:31:25.051707 ip-10-0-136-25 kubenswrapper[2569]: I0706 14:31:25.051676 2569 memory_manager.go:368] "RemoveStaleState removing state" podUID="f03ec316-06ae-470e-ad36-337dd335f037" containerName="main" Jul 06 14:31:25.051707 ip-10-0-136-25 kubenswrapper[2569]: I0706 14:31:25.051686 2569 memory_manager.go:368] "RemoveStaleState removing state" podUID="7c74954a-aa61-48aa-8964-f0ec9c480401" containerName="main" Jul 06 14:31:25.054872 ip-10-0-136-25 kubenswrapper[2569]: I0706 14:31:25.054856 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-eb526a/rate-test-model-eb526a-kserve-7b96f4886c-s7g2n" Jul 06 14:31:25.057569 ip-10-0-136-25 kubenswrapper[2569]: I0706 14:31:25.057542 2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-eb526a\"/\"kube-root-ca.crt\"" Jul 06 14:31:25.057729 ip-10-0-136-25 kubenswrapper[2569]: I0706 14:31:25.057575 2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-eb526a\"/\"default-dockercfg-jnhmf\"" Jul 06 14:31:25.057729 ip-10-0-136-25 kubenswrapper[2569]: I0706 14:31:25.057606 2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-eb526a\"/\"openshift-service-ca.crt\"" Jul 06 14:31:25.057729 ip-10-0-136-25 kubenswrapper[2569]: I0706 14:31:25.057584 2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-eb526a\"/\"rate-test-model-eb526a-kserve-self-signed-certs\"" Jul 06 14:31:25.063096 ip-10-0-136-25 kubenswrapper[2569]: I0706 14:31:25.063059 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-eb526a/rate-test-model-eb526a-kserve-7b96f4886c-s7g2n"] Jul 06 14:31:25.083743 ip-10-0-136-25 kubenswrapper[2569]: I0706 14:31:25.083708 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e96748da-0b81-43ba-aa7a-0e5668d1cc9f-home\") pod \"rate-test-model-eb526a-kserve-7b96f4886c-s7g2n\" (UID: \"e96748da-0b81-43ba-aa7a-0e5668d1cc9f\") " pod="ai-tenant-e2e-shared-b-eb526a/rate-test-model-eb526a-kserve-7b96f4886c-s7g2n" Jul 06 14:31:25.083743 ip-10-0-136-25 kubenswrapper[2569]: I0706 14:31:25.083759 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e96748da-0b81-43ba-aa7a-0e5668d1cc9f-tmp-dir\") pod \"rate-test-model-eb526a-kserve-7b96f4886c-s7g2n\" (UID: \"e96748da-0b81-43ba-aa7a-0e5668d1cc9f\") " pod="ai-tenant-e2e-shared-b-eb526a/rate-test-model-eb526a-kserve-7b96f4886c-s7g2n" Jul 06 14:31:25.084017 ip-10-0-136-25 kubenswrapper[2569]: I0706 14:31:25.083903 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e96748da-0b81-43ba-aa7a-0e5668d1cc9f-tls-certs\") pod \"rate-test-model-eb526a-kserve-7b96f4886c-s7g2n\" (UID: \"e96748da-0b81-43ba-aa7a-0e5668d1cc9f\") " pod="ai-tenant-e2e-shared-b-eb526a/rate-test-model-eb526a-kserve-7b96f4886c-s7g2n" Jul 06 14:31:25.084017 ip-10-0-136-25 kubenswrapper[2569]: I0706 14:31:25.083951 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-j2vh2\" (UniqueName: \"kubernetes.io/projected/e96748da-0b81-43ba-aa7a-0e5668d1cc9f-kube-api-access-j2vh2\") pod \"rate-test-model-eb526a-kserve-7b96f4886c-s7g2n\" (UID: \"e96748da-0b81-43ba-aa7a-0e5668d1cc9f\") " pod="ai-tenant-e2e-shared-b-eb526a/rate-test-model-eb526a-kserve-7b96f4886c-s7g2n" Jul 06 14:31:25.084017 ip-10-0-136-25 kubenswrapper[2569]: I0706 14:31:25.083989 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e96748da-0b81-43ba-aa7a-0e5668d1cc9f-dshm\") pod \"rate-test-model-eb526a-kserve-7b96f4886c-s7g2n\" (UID: \"e96748da-0b81-43ba-aa7a-0e5668d1cc9f\") " pod="ai-tenant-e2e-shared-b-eb526a/rate-test-model-eb526a-kserve-7b96f4886c-s7g2n" Jul 06 14:31:25.084201 ip-10-0-136-25 kubenswrapper[2569]: I0706 14:31:25.084030 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e96748da-0b81-43ba-aa7a-0e5668d1cc9f-model-cache\") pod \"rate-test-model-eb526a-kserve-7b96f4886c-s7g2n\" (UID: \"e96748da-0b81-43ba-aa7a-0e5668d1cc9f\") " pod="ai-tenant-e2e-shared-b-eb526a/rate-test-model-eb526a-kserve-7b96f4886c-s7g2n" Jul 06 14:31:25.185429 ip-10-0-136-25 kubenswrapper[2569]: I0706 14:31:25.185390 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e96748da-0b81-43ba-aa7a-0e5668d1cc9f-tls-certs\") pod \"rate-test-model-eb526a-kserve-7b96f4886c-s7g2n\" (UID: \"e96748da-0b81-43ba-aa7a-0e5668d1cc9f\") " pod="ai-tenant-e2e-shared-b-eb526a/rate-test-model-eb526a-kserve-7b96f4886c-s7g2n" Jul 06 14:31:25.185617 ip-10-0-136-25 kubenswrapper[2569]: I0706 14:31:25.185448 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j2vh2\" (UniqueName: \"kubernetes.io/projected/e96748da-0b81-43ba-aa7a-0e5668d1cc9f-kube-api-access-j2vh2\") pod \"rate-test-model-eb526a-kserve-7b96f4886c-s7g2n\" (UID: \"e96748da-0b81-43ba-aa7a-0e5668d1cc9f\") " pod="ai-tenant-e2e-shared-b-eb526a/rate-test-model-eb526a-kserve-7b96f4886c-s7g2n" Jul 06 14:31:25.185617 ip-10-0-136-25 kubenswrapper[2569]: I0706 14:31:25.185480 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e96748da-0b81-43ba-aa7a-0e5668d1cc9f-dshm\") pod \"rate-test-model-eb526a-kserve-7b96f4886c-s7g2n\" (UID: \"e96748da-0b81-43ba-aa7a-0e5668d1cc9f\") " pod="ai-tenant-e2e-shared-b-eb526a/rate-test-model-eb526a-kserve-7b96f4886c-s7g2n" Jul 06 14:31:25.185617 ip-10-0-136-25 kubenswrapper[2569]: I0706 14:31:25.185514 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e96748da-0b81-43ba-aa7a-0e5668d1cc9f-model-cache\") pod \"rate-test-model-eb526a-kserve-7b96f4886c-s7g2n\" (UID: \"e96748da-0b81-43ba-aa7a-0e5668d1cc9f\") " pod="ai-tenant-e2e-shared-b-eb526a/rate-test-model-eb526a-kserve-7b96f4886c-s7g2n" Jul 06 14:31:25.185617 ip-10-0-136-25 kubenswrapper[2569]: I0706 14:31:25.185544 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e96748da-0b81-43ba-aa7a-0e5668d1cc9f-home\") pod \"rate-test-model-eb526a-kserve-7b96f4886c-s7g2n\" (UID: \"e96748da-0b81-43ba-aa7a-0e5668d1cc9f\") " pod="ai-tenant-e2e-shared-b-eb526a/rate-test-model-eb526a-kserve-7b96f4886c-s7g2n" Jul 06 14:31:25.185617 ip-10-0-136-25 kubenswrapper[2569]: I0706 14:31:25.185592 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e96748da-0b81-43ba-aa7a-0e5668d1cc9f-tmp-dir\") pod \"rate-test-model-eb526a-kserve-7b96f4886c-s7g2n\" (UID: \"e96748da-0b81-43ba-aa7a-0e5668d1cc9f\") " pod="ai-tenant-e2e-shared-b-eb526a/rate-test-model-eb526a-kserve-7b96f4886c-s7g2n" Jul 06 14:31:25.186014 ip-10-0-136-25 kubenswrapper[2569]: I0706 14:31:25.185991 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e96748da-0b81-43ba-aa7a-0e5668d1cc9f-model-cache\") pod \"rate-test-model-eb526a-kserve-7b96f4886c-s7g2n\" (UID: \"e96748da-0b81-43ba-aa7a-0e5668d1cc9f\") " pod="ai-tenant-e2e-shared-b-eb526a/rate-test-model-eb526a-kserve-7b96f4886c-s7g2n" Jul 06 14:31:25.186086 ip-10-0-136-25 kubenswrapper[2569]: I0706 14:31:25.186009 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e96748da-0b81-43ba-aa7a-0e5668d1cc9f-home\") pod \"rate-test-model-eb526a-kserve-7b96f4886c-s7g2n\" (UID: \"e96748da-0b81-43ba-aa7a-0e5668d1cc9f\") " pod="ai-tenant-e2e-shared-b-eb526a/rate-test-model-eb526a-kserve-7b96f4886c-s7g2n" Jul 06 14:31:25.186127 ip-10-0-136-25 kubenswrapper[2569]: I0706 14:31:25.186079 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e96748da-0b81-43ba-aa7a-0e5668d1cc9f-tmp-dir\") pod \"rate-test-model-eb526a-kserve-7b96f4886c-s7g2n\" (UID: \"e96748da-0b81-43ba-aa7a-0e5668d1cc9f\") " pod="ai-tenant-e2e-shared-b-eb526a/rate-test-model-eb526a-kserve-7b96f4886c-s7g2n" Jul 06 14:31:25.187913 ip-10-0-136-25 kubenswrapper[2569]: I0706 14:31:25.187888 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e96748da-0b81-43ba-aa7a-0e5668d1cc9f-dshm\") pod \"rate-test-model-eb526a-kserve-7b96f4886c-s7g2n\" (UID: \"e96748da-0b81-43ba-aa7a-0e5668d1cc9f\") " pod="ai-tenant-e2e-shared-b-eb526a/rate-test-model-eb526a-kserve-7b96f4886c-s7g2n" Jul 06 14:31:25.188188 ip-10-0-136-25 kubenswrapper[2569]: I0706 14:31:25.188169 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e96748da-0b81-43ba-aa7a-0e5668d1cc9f-tls-certs\") pod \"rate-test-model-eb526a-kserve-7b96f4886c-s7g2n\" (UID: \"e96748da-0b81-43ba-aa7a-0e5668d1cc9f\") " pod="ai-tenant-e2e-shared-b-eb526a/rate-test-model-eb526a-kserve-7b96f4886c-s7g2n" Jul 06 14:31:25.193173 ip-10-0-136-25 kubenswrapper[2569]: I0706 14:31:25.193147 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-j2vh2\" (UniqueName: \"kubernetes.io/projected/e96748da-0b81-43ba-aa7a-0e5668d1cc9f-kube-api-access-j2vh2\") pod \"rate-test-model-eb526a-kserve-7b96f4886c-s7g2n\" (UID: \"e96748da-0b81-43ba-aa7a-0e5668d1cc9f\") " pod="ai-tenant-e2e-shared-b-eb526a/rate-test-model-eb526a-kserve-7b96f4886c-s7g2n" Jul 06 14:31:25.365647 ip-10-0-136-25 kubenswrapper[2569]: I0706 14:31:25.365554 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-eb526a/rate-test-model-eb526a-kserve-7b96f4886c-s7g2n" Jul 06 14:31:25.500864 ip-10-0-136-25 kubenswrapper[2569]: I0706 14:31:25.500828 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-eb526a/rate-test-model-eb526a-kserve-7b96f4886c-s7g2n"] Jul 06 14:31:25.504406 ip-10-0-136-25 kubenswrapper[2569]: W0706 14:31:25.504376 2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pode96748da_0b81_43ba_aa7a_0e5668d1cc9f.slice/crio-c49aadd29589e2ea99126cdf0146c5ed533e1d39c62495a22994d9d4c4c82886 WatchSource:0}: Error finding container c49aadd29589e2ea99126cdf0146c5ed533e1d39c62495a22994d9d4c4c82886: Status 404 returned error can't find the container with id c49aadd29589e2ea99126cdf0146c5ed533e1d39c62495a22994d9d4c4c82886 Jul 06 14:31:26.296172 ip-10-0-136-25 kubenswrapper[2569]: I0706 14:31:26.296129 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-eb526a/rate-test-model-eb526a-kserve-7b96f4886c-s7g2n" event={"ID":"e96748da-0b81-43ba-aa7a-0e5668d1cc9f","Type":"ContainerStarted","Data":"93647c7a06f783988cd2eb2f128499bee5ba7b63fd98f281bb1ca48448ad61c9"} Jul 06 14:31:26.296172 ip-10-0-136-25 kubenswrapper[2569]: I0706 14:31:26.296175 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-eb526a/rate-test-model-eb526a-kserve-7b96f4886c-s7g2n" event={"ID":"e96748da-0b81-43ba-aa7a-0e5668d1cc9f","Type":"ContainerStarted","Data":"c49aadd29589e2ea99126cdf0146c5ed533e1d39c62495a22994d9d4c4c82886"} Jul 06 14:31:26.311580 ip-10-0-136-25 kubenswrapper[2569]: I0706 14:31:26.311530 2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-b-eb526a/rate-test-model-eb526a-kserve-7b96f4886c-s7g2n" podStartSLOduration=1.31151283 podStartE2EDuration="1.31151283s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-06 14:31:25 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 14:31:26.310545926 +0000 UTC m=+2461.901107723" watchObservedRunningTime="2026-07-06 14:31:26.31151283 +0000 UTC m=+2461.902074606" Jul 06 14:31:28.742606 ip-10-0-136-25 kubenswrapper[2569]: I0706 14:31:28.742566 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-g9qp6"] Jul 06 14:31:35.366519 ip-10-0-136-25 kubenswrapper[2569]: I0706 14:31:35.366477 2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-b-eb526a/rate-test-model-eb526a-kserve-7b96f4886c-s7g2n" Jul 06 14:31:35.366956 ip-10-0-136-25 kubenswrapper[2569]: I0706 14:31:35.366762 2569 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-b-eb526a/rate-test-model-eb526a-kserve-7b96f4886c-s7g2n" Jul 06 14:31:35.379166 ip-10-0-136-25 kubenswrapper[2569]: I0706 14:31:35.379139 2569 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-b-eb526a/rate-test-model-eb526a-kserve-7b96f4886c-s7g2n" Jul 06 14:31:36.355840 ip-10-0-136-25 kubenswrapper[2569]: I0706 14:31:36.355803 2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-b-eb526a/rate-test-model-eb526a-kserve-7b96f4886c-s7g2n" Jul 06 14:31:42.941764 ip-10-0-136-25 kubenswrapper[2569]: I0706 14:31:42.941730 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-g9qp6"] Jul 06 14:31:59.734502 ip-10-0-136-25 kubenswrapper[2569]: I0706 14:31:59.734469 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-g9qp6"] Jul 06 14:32:07.814920 ip-10-0-136-25 kubenswrapper[2569]: I0706 14:32:07.814883 2569 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-2c86e0/rate-test-model-2c86e0-kserve-7595955ff5-klsjd"] Jul 06 14:32:08.634089 ip-10-0-136-25 kubenswrapper[2569]: I0706 14:32:08.634052 2569 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-eb526a/rate-test-model-eb526a-kserve-7b96f4886c-s7g2n"] Jul 06 14:32:13.834228 ip-10-0-136-25 kubenswrapper[2569]: I0706 14:32:13.834170 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-g9qp6"] Jul 06 14:32:22.835569 ip-10-0-136-25 kubenswrapper[2569]: I0706 14:32:22.835445 2569 kuberuntime_container.go:872] "Killing container with a grace period" pod="ai-tenant-e2e-shared-a-2c86e0/rate-test-model-2c86e0-kserve-7595955ff5-klsjd" podUID="cd6e9013-7033-4a64-b035-63140d84cbf7" containerName="main" containerID="cri-o://ef41db9c42a49db5790fb4c980671d6688760ccbf86a9b7f21b9972457e1720b" gracePeriod=45 Jul 06 14:32:23.092102 ip-10-0-136-25 kubenswrapper[2569]: I0706 14:32:23.092037 2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-2c86e0/rate-test-model-2c86e0-kserve-7595955ff5-klsjd" Jul 06 14:32:23.133984 ip-10-0-136-25 kubenswrapper[2569]: I0706 14:32:23.133949 2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/cd6e9013-7033-4a64-b035-63140d84cbf7-model-cache\") pod \"cd6e9013-7033-4a64-b035-63140d84cbf7\" (UID: \"cd6e9013-7033-4a64-b035-63140d84cbf7\") " Jul 06 14:32:23.134163 ip-10-0-136-25 kubenswrapper[2569]: I0706 14:32:23.134007 2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/cd6e9013-7033-4a64-b035-63140d84cbf7-home\") pod \"cd6e9013-7033-4a64-b035-63140d84cbf7\" (UID: \"cd6e9013-7033-4a64-b035-63140d84cbf7\") " Jul 06 14:32:23.134163 ip-10-0-136-25 kubenswrapper[2569]: I0706 14:32:23.134023 2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/cd6e9013-7033-4a64-b035-63140d84cbf7-dshm\") pod \"cd6e9013-7033-4a64-b035-63140d84cbf7\" (UID: \"cd6e9013-7033-4a64-b035-63140d84cbf7\") " Jul 06 14:32:23.134163 ip-10-0-136-25 kubenswrapper[2569]: I0706 14:32:23.134053 2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/cd6e9013-7033-4a64-b035-63140d84cbf7-tls-certs\") pod \"cd6e9013-7033-4a64-b035-63140d84cbf7\" (UID: \"cd6e9013-7033-4a64-b035-63140d84cbf7\") " Jul 06 14:32:23.134163 ip-10-0-136-25 kubenswrapper[2569]: I0706 14:32:23.134076 2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-m2bt2\" (UniqueName: \"kubernetes.io/projected/cd6e9013-7033-4a64-b035-63140d84cbf7-kube-api-access-m2bt2\") pod \"cd6e9013-7033-4a64-b035-63140d84cbf7\" (UID: \"cd6e9013-7033-4a64-b035-63140d84cbf7\") " Jul 06 14:32:23.134163 ip-10-0-136-25 kubenswrapper[2569]: I0706 14:32:23.134111 2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/cd6e9013-7033-4a64-b035-63140d84cbf7-tmp-dir\") pod \"cd6e9013-7033-4a64-b035-63140d84cbf7\" (UID: \"cd6e9013-7033-4a64-b035-63140d84cbf7\") " Jul 06 14:32:23.134450 ip-10-0-136-25 kubenswrapper[2569]: I0706 14:32:23.134329 2569 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cd6e9013-7033-4a64-b035-63140d84cbf7-model-cache" (OuterVolumeSpecName: "model-cache") pod "cd6e9013-7033-4a64-b035-63140d84cbf7" (UID: "cd6e9013-7033-4a64-b035-63140d84cbf7"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 06 14:32:23.134450 ip-10-0-136-25 kubenswrapper[2569]: I0706 14:32:23.134353 2569 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cd6e9013-7033-4a64-b035-63140d84cbf7-home" (OuterVolumeSpecName: "home") pod "cd6e9013-7033-4a64-b035-63140d84cbf7" (UID: "cd6e9013-7033-4a64-b035-63140d84cbf7"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 06 14:32:23.134556 ip-10-0-136-25 kubenswrapper[2569]: I0706 14:32:23.134500 2569 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cd6e9013-7033-4a64-b035-63140d84cbf7-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "cd6e9013-7033-4a64-b035-63140d84cbf7" (UID: "cd6e9013-7033-4a64-b035-63140d84cbf7"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 06 14:32:23.134603 ip-10-0-136-25 kubenswrapper[2569]: I0706 14:32:23.134587 2569 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/cd6e9013-7033-4a64-b035-63140d84cbf7-model-cache\") on node \"ip-10-0-136-25.ec2.internal\" DevicePath \"\"" Jul 06 14:32:23.134645 ip-10-0-136-25 kubenswrapper[2569]: I0706 14:32:23.134607 2569 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/cd6e9013-7033-4a64-b035-63140d84cbf7-home\") on node \"ip-10-0-136-25.ec2.internal\" DevicePath \"\"" Jul 06 14:32:23.136345 ip-10-0-136-25 kubenswrapper[2569]: I0706 14:32:23.136313 2569 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/cd6e9013-7033-4a64-b035-63140d84cbf7-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "cd6e9013-7033-4a64-b035-63140d84cbf7" (UID: "cd6e9013-7033-4a64-b035-63140d84cbf7"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 06 14:32:23.136462 ip-10-0-136-25 kubenswrapper[2569]: I0706 14:32:23.136361 2569 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cd6e9013-7033-4a64-b035-63140d84cbf7-dshm" (OuterVolumeSpecName: "dshm") pod "cd6e9013-7033-4a64-b035-63140d84cbf7" (UID: "cd6e9013-7033-4a64-b035-63140d84cbf7"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 06 14:32:23.136462 ip-10-0-136-25 kubenswrapper[2569]: I0706 14:32:23.136387 2569 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cd6e9013-7033-4a64-b035-63140d84cbf7-kube-api-access-m2bt2" (OuterVolumeSpecName: "kube-api-access-m2bt2") pod "cd6e9013-7033-4a64-b035-63140d84cbf7" (UID: "cd6e9013-7033-4a64-b035-63140d84cbf7"). InnerVolumeSpecName "kube-api-access-m2bt2". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 06 14:32:23.235931 ip-10-0-136-25 kubenswrapper[2569]: I0706 14:32:23.235882 2569 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/cd6e9013-7033-4a64-b035-63140d84cbf7-dshm\") on node \"ip-10-0-136-25.ec2.internal\" DevicePath \"\"" Jul 06 14:32:23.235931 ip-10-0-136-25 kubenswrapper[2569]: I0706 14:32:23.235922 2569 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/cd6e9013-7033-4a64-b035-63140d84cbf7-tls-certs\") on node \"ip-10-0-136-25.ec2.internal\" DevicePath \"\"" Jul 06 14:32:23.235931 ip-10-0-136-25 kubenswrapper[2569]: I0706 14:32:23.235934 2569 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-m2bt2\" (UniqueName: \"kubernetes.io/projected/cd6e9013-7033-4a64-b035-63140d84cbf7-kube-api-access-m2bt2\") on node \"ip-10-0-136-25.ec2.internal\" DevicePath \"\"" Jul 06 14:32:23.235931 ip-10-0-136-25 kubenswrapper[2569]: I0706 14:32:23.235943 2569 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/cd6e9013-7033-4a64-b035-63140d84cbf7-tmp-dir\") on node \"ip-10-0-136-25.ec2.internal\" DevicePath \"\"" Jul 06 14:32:23.568282 ip-10-0-136-25 kubenswrapper[2569]: I0706 14:32:23.568245 2569 generic.go:356] "Generic (PLEG): container finished" podID="cd6e9013-7033-4a64-b035-63140d84cbf7" containerID="ef41db9c42a49db5790fb4c980671d6688760ccbf86a9b7f21b9972457e1720b" exitCode=0 Jul 06 14:32:23.568498 ip-10-0-136-25 kubenswrapper[2569]: I0706 14:32:23.568315 2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-2c86e0/rate-test-model-2c86e0-kserve-7595955ff5-klsjd" Jul 06 14:32:23.568498 ip-10-0-136-25 kubenswrapper[2569]: I0706 14:32:23.568343 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-2c86e0/rate-test-model-2c86e0-kserve-7595955ff5-klsjd" event={"ID":"cd6e9013-7033-4a64-b035-63140d84cbf7","Type":"ContainerDied","Data":"ef41db9c42a49db5790fb4c980671d6688760ccbf86a9b7f21b9972457e1720b"} Jul 06 14:32:23.568498 ip-10-0-136-25 kubenswrapper[2569]: I0706 14:32:23.568395 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-2c86e0/rate-test-model-2c86e0-kserve-7595955ff5-klsjd" event={"ID":"cd6e9013-7033-4a64-b035-63140d84cbf7","Type":"ContainerDied","Data":"b0adc793c8c6bc9d9d363db48de015c238154dd642d087d7e09866151a9cf565"} Jul 06 14:32:23.568498 ip-10-0-136-25 kubenswrapper[2569]: I0706 14:32:23.568420 2569 scope.go:117] "RemoveContainer" containerID="ef41db9c42a49db5790fb4c980671d6688760ccbf86a9b7f21b9972457e1720b" Jul 06 14:32:23.581125 ip-10-0-136-25 kubenswrapper[2569]: I0706 14:32:23.581104 2569 scope.go:117] "RemoveContainer" containerID="ef41db9c42a49db5790fb4c980671d6688760ccbf86a9b7f21b9972457e1720b" Jul 06 14:32:23.581429 ip-10-0-136-25 kubenswrapper[2569]: E0706 14:32:23.581407 2569 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ef41db9c42a49db5790fb4c980671d6688760ccbf86a9b7f21b9972457e1720b\": container with ID starting with ef41db9c42a49db5790fb4c980671d6688760ccbf86a9b7f21b9972457e1720b not found: ID does not exist" containerID="ef41db9c42a49db5790fb4c980671d6688760ccbf86a9b7f21b9972457e1720b" Jul 06 14:32:23.581515 ip-10-0-136-25 kubenswrapper[2569]: I0706 14:32:23.581434 2569 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ef41db9c42a49db5790fb4c980671d6688760ccbf86a9b7f21b9972457e1720b"} err="failed to get container status \"ef41db9c42a49db5790fb4c980671d6688760ccbf86a9b7f21b9972457e1720b\": rpc error: code = NotFound desc = could not find container \"ef41db9c42a49db5790fb4c980671d6688760ccbf86a9b7f21b9972457e1720b\": container with ID starting with ef41db9c42a49db5790fb4c980671d6688760ccbf86a9b7f21b9972457e1720b not found: ID does not exist" Jul 06 14:32:23.590783 ip-10-0-136-25 kubenswrapper[2569]: I0706 14:32:23.590754 2569 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-2c86e0/rate-test-model-2c86e0-kserve-7595955ff5-klsjd"] Jul 06 14:32:23.594136 ip-10-0-136-25 kubenswrapper[2569]: I0706 14:32:23.594114 2569 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-a-2c86e0/rate-test-model-2c86e0-kserve-7595955ff5-klsjd"] Jul 06 14:32:23.654522 ip-10-0-136-25 kubenswrapper[2569]: I0706 14:32:23.654476 2569 kuberuntime_container.go:872] "Killing container with a grace period" pod="ai-tenant-e2e-shared-b-eb526a/rate-test-model-eb526a-kserve-7b96f4886c-s7g2n" podUID="e96748da-0b81-43ba-aa7a-0e5668d1cc9f" containerName="main" containerID="cri-o://93647c7a06f783988cd2eb2f128499bee5ba7b63fd98f281bb1ca48448ad61c9" gracePeriod=45 Jul 06 14:32:23.905093 ip-10-0-136-25 kubenswrapper[2569]: I0706 14:32:23.905066 2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-eb526a/rate-test-model-eb526a-kserve-7b96f4886c-s7g2n" Jul 06 14:32:24.043521 ip-10-0-136-25 kubenswrapper[2569]: I0706 14:32:24.043478 2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e96748da-0b81-43ba-aa7a-0e5668d1cc9f-dshm\") pod \"e96748da-0b81-43ba-aa7a-0e5668d1cc9f\" (UID: \"e96748da-0b81-43ba-aa7a-0e5668d1cc9f\") " Jul 06 14:32:24.043685 ip-10-0-136-25 kubenswrapper[2569]: I0706 14:32:24.043569 2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e96748da-0b81-43ba-aa7a-0e5668d1cc9f-tmp-dir\") pod \"e96748da-0b81-43ba-aa7a-0e5668d1cc9f\" (UID: \"e96748da-0b81-43ba-aa7a-0e5668d1cc9f\") " Jul 06 14:32:24.043685 ip-10-0-136-25 kubenswrapper[2569]: I0706 14:32:24.043610 2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e96748da-0b81-43ba-aa7a-0e5668d1cc9f-home\") pod \"e96748da-0b81-43ba-aa7a-0e5668d1cc9f\" (UID: \"e96748da-0b81-43ba-aa7a-0e5668d1cc9f\") " Jul 06 14:32:24.043685 ip-10-0-136-25 kubenswrapper[2569]: I0706 14:32:24.043643 2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-j2vh2\" (UniqueName: \"kubernetes.io/projected/e96748da-0b81-43ba-aa7a-0e5668d1cc9f-kube-api-access-j2vh2\") pod \"e96748da-0b81-43ba-aa7a-0e5668d1cc9f\" (UID: \"e96748da-0b81-43ba-aa7a-0e5668d1cc9f\") " Jul 06 14:32:24.043685 ip-10-0-136-25 kubenswrapper[2569]: I0706 14:32:24.043676 2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e96748da-0b81-43ba-aa7a-0e5668d1cc9f-model-cache\") pod \"e96748da-0b81-43ba-aa7a-0e5668d1cc9f\" (UID: \"e96748da-0b81-43ba-aa7a-0e5668d1cc9f\") " Jul 06 14:32:24.043917 ip-10-0-136-25 kubenswrapper[2569]: I0706 14:32:24.043700 2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e96748da-0b81-43ba-aa7a-0e5668d1cc9f-tls-certs\") pod \"e96748da-0b81-43ba-aa7a-0e5668d1cc9f\" (UID: \"e96748da-0b81-43ba-aa7a-0e5668d1cc9f\") " Jul 06 14:32:24.043917 ip-10-0-136-25 kubenswrapper[2569]: I0706 14:32:24.043862 2569 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e96748da-0b81-43ba-aa7a-0e5668d1cc9f-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "e96748da-0b81-43ba-aa7a-0e5668d1cc9f" (UID: "e96748da-0b81-43ba-aa7a-0e5668d1cc9f"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 06 14:32:24.043917 ip-10-0-136-25 kubenswrapper[2569]: I0706 14:32:24.043880 2569 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e96748da-0b81-43ba-aa7a-0e5668d1cc9f-home" (OuterVolumeSpecName: "home") pod "e96748da-0b81-43ba-aa7a-0e5668d1cc9f" (UID: "e96748da-0b81-43ba-aa7a-0e5668d1cc9f"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 06 14:32:24.044075 ip-10-0-136-25 kubenswrapper[2569]: I0706 14:32:24.043981 2569 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e96748da-0b81-43ba-aa7a-0e5668d1cc9f-model-cache" (OuterVolumeSpecName: "model-cache") pod "e96748da-0b81-43ba-aa7a-0e5668d1cc9f" (UID: "e96748da-0b81-43ba-aa7a-0e5668d1cc9f"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 06 14:32:24.044124 ip-10-0-136-25 kubenswrapper[2569]: I0706 14:32:24.044091 2569 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e96748da-0b81-43ba-aa7a-0e5668d1cc9f-tmp-dir\") on node \"ip-10-0-136-25.ec2.internal\" DevicePath \"\"" Jul 06 14:32:24.044124 ip-10-0-136-25 kubenswrapper[2569]: I0706 14:32:24.044112 2569 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e96748da-0b81-43ba-aa7a-0e5668d1cc9f-home\") on node \"ip-10-0-136-25.ec2.internal\" DevicePath \"\"" Jul 06 14:32:24.044189 ip-10-0-136-25 kubenswrapper[2569]: I0706 14:32:24.044129 2569 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e96748da-0b81-43ba-aa7a-0e5668d1cc9f-model-cache\") on node \"ip-10-0-136-25.ec2.internal\" DevicePath \"\"" Jul 06 14:32:24.045828 ip-10-0-136-25 kubenswrapper[2569]: I0706 14:32:24.045802 2569 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e96748da-0b81-43ba-aa7a-0e5668d1cc9f-dshm" (OuterVolumeSpecName: "dshm") pod "e96748da-0b81-43ba-aa7a-0e5668d1cc9f" (UID: "e96748da-0b81-43ba-aa7a-0e5668d1cc9f"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 06 14:32:24.045932 ip-10-0-136-25 kubenswrapper[2569]: I0706 14:32:24.045849 2569 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e96748da-0b81-43ba-aa7a-0e5668d1cc9f-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "e96748da-0b81-43ba-aa7a-0e5668d1cc9f" (UID: "e96748da-0b81-43ba-aa7a-0e5668d1cc9f"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 06 14:32:24.045932 ip-10-0-136-25 kubenswrapper[2569]: I0706 14:32:24.045864 2569 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e96748da-0b81-43ba-aa7a-0e5668d1cc9f-kube-api-access-j2vh2" (OuterVolumeSpecName: "kube-api-access-j2vh2") pod "e96748da-0b81-43ba-aa7a-0e5668d1cc9f" (UID: "e96748da-0b81-43ba-aa7a-0e5668d1cc9f"). InnerVolumeSpecName "kube-api-access-j2vh2". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 06 14:32:24.145713 ip-10-0-136-25 kubenswrapper[2569]: I0706 14:32:24.145621 2569 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e96748da-0b81-43ba-aa7a-0e5668d1cc9f-dshm\") on node \"ip-10-0-136-25.ec2.internal\" DevicePath \"\"" Jul 06 14:32:24.145713 ip-10-0-136-25 kubenswrapper[2569]: I0706 14:32:24.145655 2569 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-j2vh2\" (UniqueName: \"kubernetes.io/projected/e96748da-0b81-43ba-aa7a-0e5668d1cc9f-kube-api-access-j2vh2\") on node \"ip-10-0-136-25.ec2.internal\" DevicePath \"\"" Jul 06 14:32:24.145713 ip-10-0-136-25 kubenswrapper[2569]: I0706 14:32:24.145666 2569 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e96748da-0b81-43ba-aa7a-0e5668d1cc9f-tls-certs\") on node \"ip-10-0-136-25.ec2.internal\" DevicePath \"\"" Jul 06 14:32:24.574661 ip-10-0-136-25 kubenswrapper[2569]: I0706 14:32:24.574629 2569 generic.go:356] "Generic (PLEG): container finished" podID="e96748da-0b81-43ba-aa7a-0e5668d1cc9f" containerID="93647c7a06f783988cd2eb2f128499bee5ba7b63fd98f281bb1ca48448ad61c9" exitCode=0 Jul 06 14:32:24.574857 ip-10-0-136-25 kubenswrapper[2569]: I0706 14:32:24.574673 2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-eb526a/rate-test-model-eb526a-kserve-7b96f4886c-s7g2n" Jul 06 14:32:24.574857 ip-10-0-136-25 kubenswrapper[2569]: I0706 14:32:24.574728 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-eb526a/rate-test-model-eb526a-kserve-7b96f4886c-s7g2n" event={"ID":"e96748da-0b81-43ba-aa7a-0e5668d1cc9f","Type":"ContainerDied","Data":"93647c7a06f783988cd2eb2f128499bee5ba7b63fd98f281bb1ca48448ad61c9"} Jul 06 14:32:24.574857 ip-10-0-136-25 kubenswrapper[2569]: I0706 14:32:24.574769 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-eb526a/rate-test-model-eb526a-kserve-7b96f4886c-s7g2n" event={"ID":"e96748da-0b81-43ba-aa7a-0e5668d1cc9f","Type":"ContainerDied","Data":"c49aadd29589e2ea99126cdf0146c5ed533e1d39c62495a22994d9d4c4c82886"} Jul 06 14:32:24.574857 ip-10-0-136-25 kubenswrapper[2569]: I0706 14:32:24.574793 2569 scope.go:117] "RemoveContainer" containerID="93647c7a06f783988cd2eb2f128499bee5ba7b63fd98f281bb1ca48448ad61c9" Jul 06 14:32:24.584340 ip-10-0-136-25 kubenswrapper[2569]: I0706 14:32:24.584319 2569 scope.go:117] "RemoveContainer" containerID="93647c7a06f783988cd2eb2f128499bee5ba7b63fd98f281bb1ca48448ad61c9" Jul 06 14:32:24.584606 ip-10-0-136-25 kubenswrapper[2569]: E0706 14:32:24.584581 2569 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"93647c7a06f783988cd2eb2f128499bee5ba7b63fd98f281bb1ca48448ad61c9\": container with ID starting with 93647c7a06f783988cd2eb2f128499bee5ba7b63fd98f281bb1ca48448ad61c9 not found: ID does not exist" containerID="93647c7a06f783988cd2eb2f128499bee5ba7b63fd98f281bb1ca48448ad61c9" Jul 06 14:32:24.584674 ip-10-0-136-25 kubenswrapper[2569]: I0706 14:32:24.584613 2569 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"93647c7a06f783988cd2eb2f128499bee5ba7b63fd98f281bb1ca48448ad61c9"} err="failed to get container status \"93647c7a06f783988cd2eb2f128499bee5ba7b63fd98f281bb1ca48448ad61c9\": rpc error: code = NotFound desc = could not find container \"93647c7a06f783988cd2eb2f128499bee5ba7b63fd98f281bb1ca48448ad61c9\": container with ID starting with 93647c7a06f783988cd2eb2f128499bee5ba7b63fd98f281bb1ca48448ad61c9 not found: ID does not exist" Jul 06 14:32:24.594180 ip-10-0-136-25 kubenswrapper[2569]: I0706 14:32:24.594149 2569 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-eb526a/rate-test-model-eb526a-kserve-7b96f4886c-s7g2n"] Jul 06 14:32:24.598814 ip-10-0-136-25 kubenswrapper[2569]: I0706 14:32:24.598790 2569 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-b-eb526a/rate-test-model-eb526a-kserve-7b96f4886c-s7g2n"] Jul 06 14:32:24.942169 ip-10-0-136-25 kubenswrapper[2569]: I0706 14:32:24.942095 2569 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="cd6e9013-7033-4a64-b035-63140d84cbf7" path="/var/lib/kubelet/pods/cd6e9013-7033-4a64-b035-63140d84cbf7/volumes" Jul 06 14:32:24.942534 ip-10-0-136-25 kubenswrapper[2569]: I0706 14:32:24.942520 2569 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="e96748da-0b81-43ba-aa7a-0e5668d1cc9f" path="/var/lib/kubelet/pods/e96748da-0b81-43ba-aa7a-0e5668d1cc9f/volumes" Jul 06 14:32:50.153967 ip-10-0-136-25 kubenswrapper[2569]: I0706 14:32:50.153930 2569 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-api-e2e-shared-b-eb526a-6cdc9f8d77-kpfm9"] Jul 06 14:32:50.154737 ip-10-0-136-25 kubenswrapper[2569]: I0706 14:32:50.154170 2569 kuberuntime_container.go:872] "Killing container with a grace period" pod="opendatahub/maas-api-e2e-shared-b-eb526a-6cdc9f8d77-kpfm9" podUID="afcdff00-c419-4f06-b9d4-70d980dc4372" containerName="maas-api" containerID="cri-o://273ec04de628e24fae2e86906e16cde9f7710b72af14b29b366f59e40875a70e" gracePeriod=30 Jul 06 14:32:50.405462 ip-10-0-136-25 kubenswrapper[2569]: I0706 14:32:50.405391 2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-api-e2e-shared-b-eb526a-6cdc9f8d77-kpfm9" Jul 06 14:32:50.495448 ip-10-0-136-25 kubenswrapper[2569]: I0706 14:32:50.495419 2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-277zr\" (UniqueName: \"kubernetes.io/projected/afcdff00-c419-4f06-b9d4-70d980dc4372-kube-api-access-277zr\") pod \"afcdff00-c419-4f06-b9d4-70d980dc4372\" (UID: \"afcdff00-c419-4f06-b9d4-70d980dc4372\") " Jul 06 14:32:50.495629 ip-10-0-136-25 kubenswrapper[2569]: I0706 14:32:50.495455 2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/afcdff00-c419-4f06-b9d4-70d980dc4372-maas-api-tls\") pod \"afcdff00-c419-4f06-b9d4-70d980dc4372\" (UID: \"afcdff00-c419-4f06-b9d4-70d980dc4372\") " Jul 06 14:32:50.495629 ip-10-0-136-25 kubenswrapper[2569]: I0706 14:32:50.495514 2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/afcdff00-c419-4f06-b9d4-70d980dc4372-openshift-service-ca\") pod \"afcdff00-c419-4f06-b9d4-70d980dc4372\" (UID: \"afcdff00-c419-4f06-b9d4-70d980dc4372\") " Jul 06 14:32:50.495949 ip-10-0-136-25 kubenswrapper[2569]: I0706 14:32:50.495919 2569 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/afcdff00-c419-4f06-b9d4-70d980dc4372-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "afcdff00-c419-4f06-b9d4-70d980dc4372" (UID: "afcdff00-c419-4f06-b9d4-70d980dc4372"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 06 14:32:50.497629 ip-10-0-136-25 kubenswrapper[2569]: I0706 14:32:50.497602 2569 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/afcdff00-c419-4f06-b9d4-70d980dc4372-kube-api-access-277zr" (OuterVolumeSpecName: "kube-api-access-277zr") pod "afcdff00-c419-4f06-b9d4-70d980dc4372" (UID: "afcdff00-c419-4f06-b9d4-70d980dc4372"). InnerVolumeSpecName "kube-api-access-277zr". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 06 14:32:50.497717 ip-10-0-136-25 kubenswrapper[2569]: I0706 14:32:50.497681 2569 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/afcdff00-c419-4f06-b9d4-70d980dc4372-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "afcdff00-c419-4f06-b9d4-70d980dc4372" (UID: "afcdff00-c419-4f06-b9d4-70d980dc4372"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 06 14:32:50.597053 ip-10-0-136-25 kubenswrapper[2569]: I0706 14:32:50.597015 2569 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-277zr\" (UniqueName: \"kubernetes.io/projected/afcdff00-c419-4f06-b9d4-70d980dc4372-kube-api-access-277zr\") on node \"ip-10-0-136-25.ec2.internal\" DevicePath \"\"" Jul 06 14:32:50.597053 ip-10-0-136-25 kubenswrapper[2569]: I0706 14:32:50.597046 2569 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/afcdff00-c419-4f06-b9d4-70d980dc4372-maas-api-tls\") on node \"ip-10-0-136-25.ec2.internal\" DevicePath \"\"" Jul 06 14:32:50.597053 ip-10-0-136-25 kubenswrapper[2569]: I0706 14:32:50.597055 2569 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/afcdff00-c419-4f06-b9d4-70d980dc4372-openshift-service-ca\") on node \"ip-10-0-136-25.ec2.internal\" DevicePath \"\"" Jul 06 14:32:50.697886 ip-10-0-136-25 kubenswrapper[2569]: I0706 14:32:50.697799 2569 generic.go:356] "Generic (PLEG): container finished" podID="afcdff00-c419-4f06-b9d4-70d980dc4372" containerID="273ec04de628e24fae2e86906e16cde9f7710b72af14b29b366f59e40875a70e" exitCode=0 Jul 06 14:32:50.697886 ip-10-0-136-25 kubenswrapper[2569]: I0706 14:32:50.697842 2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-api-e2e-shared-b-eb526a-6cdc9f8d77-kpfm9" Jul 06 14:32:50.698118 ip-10-0-136-25 kubenswrapper[2569]: I0706 14:32:50.697843 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-api-e2e-shared-b-eb526a-6cdc9f8d77-kpfm9" event={"ID":"afcdff00-c419-4f06-b9d4-70d980dc4372","Type":"ContainerDied","Data":"273ec04de628e24fae2e86906e16cde9f7710b72af14b29b366f59e40875a70e"} Jul 06 14:32:50.698118 ip-10-0-136-25 kubenswrapper[2569]: I0706 14:32:50.697975 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-api-e2e-shared-b-eb526a-6cdc9f8d77-kpfm9" event={"ID":"afcdff00-c419-4f06-b9d4-70d980dc4372","Type":"ContainerDied","Data":"98c8d17221241d7f20086ff1c058f1401ab70c9913e0b0264402d4ff6c1db9b2"} Jul 06 14:32:50.698118 ip-10-0-136-25 kubenswrapper[2569]: I0706 14:32:50.698012 2569 scope.go:117] "RemoveContainer" containerID="273ec04de628e24fae2e86906e16cde9f7710b72af14b29b366f59e40875a70e" Jul 06 14:32:50.706959 ip-10-0-136-25 kubenswrapper[2569]: I0706 14:32:50.706942 2569 scope.go:117] "RemoveContainer" containerID="273ec04de628e24fae2e86906e16cde9f7710b72af14b29b366f59e40875a70e" Jul 06 14:32:50.707232 ip-10-0-136-25 kubenswrapper[2569]: E0706 14:32:50.707193 2569 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"273ec04de628e24fae2e86906e16cde9f7710b72af14b29b366f59e40875a70e\": container with ID starting with 273ec04de628e24fae2e86906e16cde9f7710b72af14b29b366f59e40875a70e not found: ID does not exist" containerID="273ec04de628e24fae2e86906e16cde9f7710b72af14b29b366f59e40875a70e" Jul 06 14:32:50.707338 ip-10-0-136-25 kubenswrapper[2569]: I0706 14:32:50.707320 2569 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"273ec04de628e24fae2e86906e16cde9f7710b72af14b29b366f59e40875a70e"} err="failed to get container status \"273ec04de628e24fae2e86906e16cde9f7710b72af14b29b366f59e40875a70e\": rpc error: code = NotFound desc = could not find container \"273ec04de628e24fae2e86906e16cde9f7710b72af14b29b366f59e40875a70e\": container with ID starting with 273ec04de628e24fae2e86906e16cde9f7710b72af14b29b366f59e40875a70e not found: ID does not exist" Jul 06 14:32:50.717224 ip-10-0-136-25 kubenswrapper[2569]: I0706 14:32:50.717188 2569 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-api-e2e-shared-b-eb526a-6cdc9f8d77-kpfm9"] Jul 06 14:32:50.720458 ip-10-0-136-25 kubenswrapper[2569]: I0706 14:32:50.720437 2569 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/maas-api-e2e-shared-b-eb526a-6cdc9f8d77-kpfm9"] Jul 06 14:32:50.940801 ip-10-0-136-25 kubenswrapper[2569]: I0706 14:32:50.940770 2569 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="afcdff00-c419-4f06-b9d4-70d980dc4372" path="/var/lib/kubelet/pods/afcdff00-c419-4f06-b9d4-70d980dc4372/volumes" Jul 06 14:32:54.036015 ip-10-0-136-25 kubenswrapper[2569]: I0706 14:32:54.035980 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-g9qp6"] Jul 06 14:32:57.219479 ip-10-0-136-25 kubenswrapper[2569]: I0706 14:32:57.219447 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-f549ff76c-wqjl4_cf359e55-20b7-43b2-baa5-848ae0a9e07d/authorino/0.log" Jul 06 14:33:02.761231 ip-10-0-136-25 kubenswrapper[2569]: I0706 14:33:02.761180 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-5f8f457c8b-lvnlj_487646d3-ef03-4bcf-acf3-7f7051ff672f/copy-manifests/0.log" Jul 06 14:33:02.767850 ip-10-0-136-25 kubenswrapper[2569]: I0706 14:33:02.767820 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-5f8f457c8b-lvnlj_487646d3-ef03-4bcf-acf3-7f7051ff672f/manager/0.log" Jul 06 14:33:04.275285 ip-10-0-136-25 kubenswrapper[2569]: I0706 14:33:04.275254 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-f549ff76c-wqjl4_cf359e55-20b7-43b2-baa5-848ae0a9e07d/authorino/0.log" Jul 06 14:33:04.630776 ip-10-0-136-25 kubenswrapper[2569]: I0706 14:33:04.630693 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-console-plugin-66b5774985-gbw6w_2e801db1-650f-4f52-8ce4-37ce2381ec24/kuadrant-console-plugin/0.log" Jul 06 14:33:04.974180 ip-10-0-136-25 kubenswrapper[2569]: I0706 14:33:04.974153 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-limitador-54d6d988d5-g9qp6_c296255d-ef13-4904-8569-02e8dc6d3da0/limitador/0.log" Jul 06 14:33:05.679628 ip-10-0-136-25 kubenswrapper[2569]: I0706 14:33:05.679600 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-7c6c964b95-sfw6w_1970cda6-a1e1-4aed-af85-44bf5bfd5dad/kube-auth-proxy/0.log" Jul 06 14:33:06.227732 ip-10-0-136-25 kubenswrapper[2569]: I0706 14:33:06.227685 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_router-default-966c76789-kpw4q_1cb95852-b3c5-4b1a-80bf-9d3fa465084a/router/0.log" Jul 06 14:33:11.191487 ip-10-0-136-25 kubenswrapper[2569]: I0706 14:33:11.191456 2569 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-must-gather-55pjj/must-gather-cddlc"] Jul 06 14:33:11.191884 ip-10-0-136-25 kubenswrapper[2569]: I0706 14:33:11.191803 2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="afcdff00-c419-4f06-b9d4-70d980dc4372" containerName="maas-api" Jul 06 14:33:11.191884 ip-10-0-136-25 kubenswrapper[2569]: I0706 14:33:11.191813 2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="afcdff00-c419-4f06-b9d4-70d980dc4372" containerName="maas-api" Jul 06 14:33:11.191884 ip-10-0-136-25 kubenswrapper[2569]: I0706 14:33:11.191829 2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="cd6e9013-7033-4a64-b035-63140d84cbf7" containerName="main" Jul 06 14:33:11.191884 ip-10-0-136-25 kubenswrapper[2569]: I0706 14:33:11.191834 2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="cd6e9013-7033-4a64-b035-63140d84cbf7" containerName="main" Jul 06 14:33:11.191884 ip-10-0-136-25 kubenswrapper[2569]: I0706 14:33:11.191841 2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e96748da-0b81-43ba-aa7a-0e5668d1cc9f" containerName="main" Jul 06 14:33:11.191884 ip-10-0-136-25 kubenswrapper[2569]: I0706 14:33:11.191847 2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="e96748da-0b81-43ba-aa7a-0e5668d1cc9f" containerName="main" Jul 06 14:33:11.192085 ip-10-0-136-25 kubenswrapper[2569]: I0706 14:33:11.191908 2569 memory_manager.go:368] "RemoveStaleState removing state" podUID="e96748da-0b81-43ba-aa7a-0e5668d1cc9f" containerName="main" Jul 06 14:33:11.192085 ip-10-0-136-25 kubenswrapper[2569]: I0706 14:33:11.191915 2569 memory_manager.go:368] "RemoveStaleState removing state" podUID="afcdff00-c419-4f06-b9d4-70d980dc4372" containerName="maas-api" Jul 06 14:33:11.192085 ip-10-0-136-25 kubenswrapper[2569]: I0706 14:33:11.191922 2569 memory_manager.go:368] "RemoveStaleState removing state" podUID="cd6e9013-7033-4a64-b035-63140d84cbf7" containerName="main" Jul 06 14:33:11.194759 ip-10-0-136-25 kubenswrapper[2569]: I0706 14:33:11.194740 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-55pjj/must-gather-cddlc" Jul 06 14:33:11.197447 ip-10-0-136-25 kubenswrapper[2569]: I0706 14:33:11.197427 2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-55pjj\"/\"kube-root-ca.crt\"" Jul 06 14:33:11.197537 ip-10-0-136-25 kubenswrapper[2569]: I0706 14:33:11.197453 2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-55pjj\"/\"openshift-service-ca.crt\"" Jul 06 14:33:11.197537 ip-10-0-136-25 kubenswrapper[2569]: I0706 14:33:11.197428 2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-55pjj\"/\"default-dockercfg-gmtfx\"" Jul 06 14:33:11.201943 ip-10-0-136-25 kubenswrapper[2569]: I0706 14:33:11.201888 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-55pjj/must-gather-cddlc"] Jul 06 14:33:11.281756 ip-10-0-136-25 kubenswrapper[2569]: I0706 14:33:11.281718 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mrp6b\" (UniqueName: \"kubernetes.io/projected/9e2a27c0-67dd-4a85-9695-d0d067afb64b-kube-api-access-mrp6b\") pod \"must-gather-cddlc\" (UID: \"9e2a27c0-67dd-4a85-9695-d0d067afb64b\") " pod="openshift-must-gather-55pjj/must-gather-cddlc" Jul 06 14:33:11.281905 ip-10-0-136-25 kubenswrapper[2569]: I0706 14:33:11.281864 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/9e2a27c0-67dd-4a85-9695-d0d067afb64b-must-gather-output\") pod \"must-gather-cddlc\" (UID: \"9e2a27c0-67dd-4a85-9695-d0d067afb64b\") " pod="openshift-must-gather-55pjj/must-gather-cddlc" Jul 06 14:33:11.382930 ip-10-0-136-25 kubenswrapper[2569]: I0706 14:33:11.382887 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/9e2a27c0-67dd-4a85-9695-d0d067afb64b-must-gather-output\") pod \"must-gather-cddlc\" (UID: \"9e2a27c0-67dd-4a85-9695-d0d067afb64b\") " pod="openshift-must-gather-55pjj/must-gather-cddlc" Jul 06 14:33:11.383110 ip-10-0-136-25 kubenswrapper[2569]: I0706 14:33:11.382952 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mrp6b\" (UniqueName: \"kubernetes.io/projected/9e2a27c0-67dd-4a85-9695-d0d067afb64b-kube-api-access-mrp6b\") pod \"must-gather-cddlc\" (UID: \"9e2a27c0-67dd-4a85-9695-d0d067afb64b\") " pod="openshift-must-gather-55pjj/must-gather-cddlc" Jul 06 14:33:11.383275 ip-10-0-136-25 kubenswrapper[2569]: I0706 14:33:11.383254 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/9e2a27c0-67dd-4a85-9695-d0d067afb64b-must-gather-output\") pod \"must-gather-cddlc\" (UID: \"9e2a27c0-67dd-4a85-9695-d0d067afb64b\") " pod="openshift-must-gather-55pjj/must-gather-cddlc" Jul 06 14:33:11.390218 ip-10-0-136-25 kubenswrapper[2569]: I0706 14:33:11.390192 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mrp6b\" (UniqueName: \"kubernetes.io/projected/9e2a27c0-67dd-4a85-9695-d0d067afb64b-kube-api-access-mrp6b\") pod \"must-gather-cddlc\" (UID: \"9e2a27c0-67dd-4a85-9695-d0d067afb64b\") " pod="openshift-must-gather-55pjj/must-gather-cddlc" Jul 06 14:33:11.504033 ip-10-0-136-25 kubenswrapper[2569]: I0706 14:33:11.504007 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-55pjj/must-gather-cddlc" Jul 06 14:33:11.642915 ip-10-0-136-25 kubenswrapper[2569]: I0706 14:33:11.642889 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-55pjj/must-gather-cddlc"] Jul 06 14:33:11.647255 ip-10-0-136-25 kubenswrapper[2569]: W0706 14:33:11.647204 2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod9e2a27c0_67dd_4a85_9695_d0d067afb64b.slice/crio-4b5bec38702a24e6f9e9b3d26010817a78eff89960d284d881118cf7ee3b16b5 WatchSource:0}: Error finding container 4b5bec38702a24e6f9e9b3d26010817a78eff89960d284d881118cf7ee3b16b5: Status 404 returned error can't find the container with id 4b5bec38702a24e6f9e9b3d26010817a78eff89960d284d881118cf7ee3b16b5 Jul 06 14:33:11.649190 ip-10-0-136-25 kubenswrapper[2569]: I0706 14:33:11.649173 2569 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Jul 06 14:33:11.797640 ip-10-0-136-25 kubenswrapper[2569]: I0706 14:33:11.797549 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-55pjj/must-gather-cddlc" event={"ID":"9e2a27c0-67dd-4a85-9695-d0d067afb64b","Type":"ContainerStarted","Data":"4b5bec38702a24e6f9e9b3d26010817a78eff89960d284d881118cf7ee3b16b5"} Jul 06 14:33:13.811918 ip-10-0-136-25 kubenswrapper[2569]: I0706 14:33:13.811884 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-55pjj/must-gather-cddlc" event={"ID":"9e2a27c0-67dd-4a85-9695-d0d067afb64b","Type":"ContainerStarted","Data":"8fe0fe827079fd3ed8c6ff9bacf161e65b2a91824537ea85933903b0c301be39"} Jul 06 14:33:13.812326 ip-10-0-136-25 kubenswrapper[2569]: I0706 14:33:13.811931 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-55pjj/must-gather-cddlc" event={"ID":"9e2a27c0-67dd-4a85-9695-d0d067afb64b","Type":"ContainerStarted","Data":"6ad41688da4faf4df6ad68fe4535770b2bde5e7a5731a1cf15df3d6c63eaeb3c"} Jul 06 14:33:13.826225 ip-10-0-136-25 kubenswrapper[2569]: I0706 14:33:13.826151 2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-must-gather-55pjj/must-gather-cddlc" podStartSLOduration=1.60241517 podStartE2EDuration="2.826129079s" totalImagesPullingTime="1.223713909s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-06 14:33:11 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 14:33:13.825725472 +0000 UTC m=+2569.416287272" watchObservedRunningTime="2026-07-06 14:33:13.826129079 +0000 UTC m=+2569.416690859" Jul 06 14:33:14.529087 ip-10-0-136-25 kubenswrapper[2569]: I0706 14:33:14.529041 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_global-pull-secret-syncer-cg8ph_576fd8e3-f3a0-48d2-a0d6-faa782a25a63/global-pull-secret-syncer/0.log" Jul 06 14:33:14.650431 ip-10-0-136-25 kubenswrapper[2569]: I0706 14:33:14.650388 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_konnectivity-agent-vqmcq_cb1807ec-0568-4ab3-a9ca-e453709b820e/konnectivity-agent/0.log" Jul 06 14:33:14.754386 ip-10-0-136-25 kubenswrapper[2569]: I0706 14:33:14.754353 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kube-apiserver-proxy-ip-10-0-136-25.ec2.internal_ba668ff1926990fccf3f73e0625e305c/haproxy/0.log" Jul 06 14:33:20.565540 ip-10-0-136-25 kubenswrapper[2569]: I0706 14:33:20.565454 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-f549ff76c-wqjl4_cf359e55-20b7-43b2-baa5-848ae0a9e07d/authorino/0.log" Jul 06 14:33:20.667380 ip-10-0-136-25 kubenswrapper[2569]: I0706 14:33:20.667350 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-console-plugin-66b5774985-gbw6w_2e801db1-650f-4f52-8ce4-37ce2381ec24/kuadrant-console-plugin/0.log" Jul 06 14:33:20.864626 ip-10-0-136-25 kubenswrapper[2569]: I0706 14:33:20.864543 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-limitador-54d6d988d5-g9qp6_c296255d-ef13-4904-8569-02e8dc6d3da0/limitador/0.log" Jul 06 14:33:22.616799 ip-10-0-136-25 kubenswrapper[2569]: I0706 14:33:22.616714 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_a9b01e76-1c06-484b-bf3a-6db72c95c536/alertmanager/0.log" Jul 06 14:33:22.641011 ip-10-0-136-25 kubenswrapper[2569]: I0706 14:33:22.640945 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_a9b01e76-1c06-484b-bf3a-6db72c95c536/config-reloader/0.log" Jul 06 14:33:22.667647 ip-10-0-136-25 kubenswrapper[2569]: I0706 14:33:22.667598 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_a9b01e76-1c06-484b-bf3a-6db72c95c536/kube-rbac-proxy-web/0.log" Jul 06 14:33:22.691765 ip-10-0-136-25 kubenswrapper[2569]: I0706 14:33:22.691672 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_a9b01e76-1c06-484b-bf3a-6db72c95c536/kube-rbac-proxy/0.log" Jul 06 14:33:22.714066 ip-10-0-136-25 kubenswrapper[2569]: I0706 14:33:22.714032 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_a9b01e76-1c06-484b-bf3a-6db72c95c536/kube-rbac-proxy-metric/0.log" Jul 06 14:33:22.741671 ip-10-0-136-25 kubenswrapper[2569]: I0706 14:33:22.741644 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_a9b01e76-1c06-484b-bf3a-6db72c95c536/prom-label-proxy/0.log" Jul 06 14:33:22.765462 ip-10-0-136-25 kubenswrapper[2569]: I0706 14:33:22.765432 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_a9b01e76-1c06-484b-bf3a-6db72c95c536/init-config-reloader/0.log" Jul 06 14:33:22.814045 ip-10-0-136-25 kubenswrapper[2569]: I0706 14:33:22.814006 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_cluster-monitoring-operator-64786d9cbb-zpkk5_a52c7279-3809-4f22-9001-afb7325cfe8d/cluster-monitoring-operator/0.log" Jul 06 14:33:22.987011 ip-10-0-136-25 kubenswrapper[2569]: I0706 14:33:22.986932 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-7fj7f_f906ef93-d6ed-491e-b7ad-3622d7398b8e/node-exporter/0.log" Jul 06 14:33:23.011944 ip-10-0-136-25 kubenswrapper[2569]: I0706 14:33:23.011912 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-7fj7f_f906ef93-d6ed-491e-b7ad-3622d7398b8e/kube-rbac-proxy/0.log" Jul 06 14:33:23.034691 ip-10-0-136-25 kubenswrapper[2569]: I0706 14:33:23.034654 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-7fj7f_f906ef93-d6ed-491e-b7ad-3622d7398b8e/init-textfile/0.log" Jul 06 14:33:23.226311 ip-10-0-136-25 kubenswrapper[2569]: I0706 14:33:23.226280 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_openshift-state-metrics-54cf96dfcf-n4mw9_dc28734e-3746-4350-9fc7-4bcc1b9e842d/kube-rbac-proxy-main/0.log" Jul 06 14:33:23.248411 ip-10-0-136-25 kubenswrapper[2569]: I0706 14:33:23.248380 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_openshift-state-metrics-54cf96dfcf-n4mw9_dc28734e-3746-4350-9fc7-4bcc1b9e842d/kube-rbac-proxy-self/0.log" Jul 06 14:33:23.270878 ip-10-0-136-25 kubenswrapper[2569]: I0706 14:33:23.270847 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_openshift-state-metrics-54cf96dfcf-n4mw9_dc28734e-3746-4350-9fc7-4bcc1b9e842d/openshift-state-metrics/0.log" Jul 06 14:33:23.310780 ip-10-0-136-25 kubenswrapper[2569]: I0706 14:33:23.310747 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_1c64c06c-4a9b-4767-bdd8-cc5fb5bed504/prometheus/0.log" Jul 06 14:33:23.338547 ip-10-0-136-25 kubenswrapper[2569]: I0706 14:33:23.338518 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_1c64c06c-4a9b-4767-bdd8-cc5fb5bed504/config-reloader/0.log" Jul 06 14:33:23.360984 ip-10-0-136-25 kubenswrapper[2569]: I0706 14:33:23.360939 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_1c64c06c-4a9b-4767-bdd8-cc5fb5bed504/thanos-sidecar/0.log" Jul 06 14:33:23.383071 ip-10-0-136-25 kubenswrapper[2569]: I0706 14:33:23.383039 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_1c64c06c-4a9b-4767-bdd8-cc5fb5bed504/kube-rbac-proxy-web/0.log" Jul 06 14:33:23.407876 ip-10-0-136-25 kubenswrapper[2569]: I0706 14:33:23.407840 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_1c64c06c-4a9b-4767-bdd8-cc5fb5bed504/kube-rbac-proxy/0.log" Jul 06 14:33:23.432766 ip-10-0-136-25 kubenswrapper[2569]: I0706 14:33:23.432739 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_1c64c06c-4a9b-4767-bdd8-cc5fb5bed504/kube-rbac-proxy-thanos/0.log" Jul 06 14:33:23.458810 ip-10-0-136-25 kubenswrapper[2569]: I0706 14:33:23.458779 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_1c64c06c-4a9b-4767-bdd8-cc5fb5bed504/init-config-reloader/0.log" Jul 06 14:33:23.644120 ip-10-0-136-25 kubenswrapper[2569]: I0706 14:33:23.644042 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-7d96487c86-rnclw_35142749-f6cf-4c8c-b41b-324c97c06c4c/thanos-query/0.log" Jul 06 14:33:23.666120 ip-10-0-136-25 kubenswrapper[2569]: I0706 14:33:23.666095 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-7d96487c86-rnclw_35142749-f6cf-4c8c-b41b-324c97c06c4c/kube-rbac-proxy-web/0.log" Jul 06 14:33:23.688482 ip-10-0-136-25 kubenswrapper[2569]: I0706 14:33:23.688448 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-7d96487c86-rnclw_35142749-f6cf-4c8c-b41b-324c97c06c4c/kube-rbac-proxy/0.log" Jul 06 14:33:23.710823 ip-10-0-136-25 kubenswrapper[2569]: I0706 14:33:23.710789 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-7d96487c86-rnclw_35142749-f6cf-4c8c-b41b-324c97c06c4c/prom-label-proxy/0.log" Jul 06 14:33:23.736752 ip-10-0-136-25 kubenswrapper[2569]: I0706 14:33:23.736727 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-7d96487c86-rnclw_35142749-f6cf-4c8c-b41b-324c97c06c4c/kube-rbac-proxy-rules/0.log" Jul 06 14:33:23.759306 ip-10-0-136-25 kubenswrapper[2569]: I0706 14:33:23.759274 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-7d96487c86-rnclw_35142749-f6cf-4c8c-b41b-324c97c06c4c/kube-rbac-proxy-metrics/0.log" Jul 06 14:33:24.772013 ip-10-0-136-25 kubenswrapper[2569]: I0706 14:33:24.771981 2569 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-must-gather-55pjj/perf-node-gather-daemonset-l4kdc"] Jul 06 14:33:24.780832 ip-10-0-136-25 kubenswrapper[2569]: I0706 14:33:24.780800 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-55pjj/perf-node-gather-daemonset-l4kdc" Jul 06 14:33:24.783574 ip-10-0-136-25 kubenswrapper[2569]: I0706 14:33:24.783548 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-55pjj/perf-node-gather-daemonset-l4kdc"] Jul 06 14:33:24.940177 ip-10-0-136-25 kubenswrapper[2569]: I0706 14:33:24.940141 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/8b20fa8f-441e-4f6a-92dd-a9d05e3660d0-proc\") pod \"perf-node-gather-daemonset-l4kdc\" (UID: \"8b20fa8f-441e-4f6a-92dd-a9d05e3660d0\") " pod="openshift-must-gather-55pjj/perf-node-gather-daemonset-l4kdc" Jul 06 14:33:24.940381 ip-10-0-136-25 kubenswrapper[2569]: I0706 14:33:24.940194 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/8b20fa8f-441e-4f6a-92dd-a9d05e3660d0-podres\") pod \"perf-node-gather-daemonset-l4kdc\" (UID: \"8b20fa8f-441e-4f6a-92dd-a9d05e3660d0\") " pod="openshift-must-gather-55pjj/perf-node-gather-daemonset-l4kdc" Jul 06 14:33:24.940381 ip-10-0-136-25 kubenswrapper[2569]: I0706 14:33:24.940286 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/8b20fa8f-441e-4f6a-92dd-a9d05e3660d0-lib-modules\") pod \"perf-node-gather-daemonset-l4kdc\" (UID: \"8b20fa8f-441e-4f6a-92dd-a9d05e3660d0\") " pod="openshift-must-gather-55pjj/perf-node-gather-daemonset-l4kdc" Jul 06 14:33:24.940381 ip-10-0-136-25 kubenswrapper[2569]: I0706 14:33:24.940316 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/8b20fa8f-441e-4f6a-92dd-a9d05e3660d0-sys\") pod \"perf-node-gather-daemonset-l4kdc\" (UID: \"8b20fa8f-441e-4f6a-92dd-a9d05e3660d0\") " pod="openshift-must-gather-55pjj/perf-node-gather-daemonset-l4kdc" Jul 06 14:33:24.940381 ip-10-0-136-25 kubenswrapper[2569]: I0706 14:33:24.940357 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rb4v4\" (UniqueName: \"kubernetes.io/projected/8b20fa8f-441e-4f6a-92dd-a9d05e3660d0-kube-api-access-rb4v4\") pod \"perf-node-gather-daemonset-l4kdc\" (UID: \"8b20fa8f-441e-4f6a-92dd-a9d05e3660d0\") " pod="openshift-must-gather-55pjj/perf-node-gather-daemonset-l4kdc" Jul 06 14:33:24.942895 ip-10-0-136-25 kubenswrapper[2569]: I0706 14:33:24.942867 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-network-console_networking-console-plugin-6bffdf9778-99974_c198bd10-43bf-493b-9aa0-8f3dad5f8b59/networking-console-plugin/0.log" Jul 06 14:33:25.041595 ip-10-0-136-25 kubenswrapper[2569]: I0706 14:33:25.041492 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/8b20fa8f-441e-4f6a-92dd-a9d05e3660d0-lib-modules\") pod \"perf-node-gather-daemonset-l4kdc\" (UID: \"8b20fa8f-441e-4f6a-92dd-a9d05e3660d0\") " pod="openshift-must-gather-55pjj/perf-node-gather-daemonset-l4kdc" Jul 06 14:33:25.041595 ip-10-0-136-25 kubenswrapper[2569]: I0706 14:33:25.041551 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/8b20fa8f-441e-4f6a-92dd-a9d05e3660d0-sys\") pod \"perf-node-gather-daemonset-l4kdc\" (UID: \"8b20fa8f-441e-4f6a-92dd-a9d05e3660d0\") " pod="openshift-must-gather-55pjj/perf-node-gather-daemonset-l4kdc" Jul 06 14:33:25.041595 ip-10-0-136-25 kubenswrapper[2569]: I0706 14:33:25.041592 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rb4v4\" (UniqueName: \"kubernetes.io/projected/8b20fa8f-441e-4f6a-92dd-a9d05e3660d0-kube-api-access-rb4v4\") pod \"perf-node-gather-daemonset-l4kdc\" (UID: \"8b20fa8f-441e-4f6a-92dd-a9d05e3660d0\") " pod="openshift-must-gather-55pjj/perf-node-gather-daemonset-l4kdc" Jul 06 14:33:25.041888 ip-10-0-136-25 kubenswrapper[2569]: I0706 14:33:25.041684 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/8b20fa8f-441e-4f6a-92dd-a9d05e3660d0-proc\") pod \"perf-node-gather-daemonset-l4kdc\" (UID: \"8b20fa8f-441e-4f6a-92dd-a9d05e3660d0\") " pod="openshift-must-gather-55pjj/perf-node-gather-daemonset-l4kdc" Jul 06 14:33:25.041888 ip-10-0-136-25 kubenswrapper[2569]: I0706 14:33:25.041732 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/8b20fa8f-441e-4f6a-92dd-a9d05e3660d0-podres\") pod \"perf-node-gather-daemonset-l4kdc\" (UID: \"8b20fa8f-441e-4f6a-92dd-a9d05e3660d0\") " pod="openshift-must-gather-55pjj/perf-node-gather-daemonset-l4kdc" Jul 06 14:33:25.042005 ip-10-0-136-25 kubenswrapper[2569]: I0706 14:33:25.041894 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/8b20fa8f-441e-4f6a-92dd-a9d05e3660d0-podres\") pod \"perf-node-gather-daemonset-l4kdc\" (UID: \"8b20fa8f-441e-4f6a-92dd-a9d05e3660d0\") " pod="openshift-must-gather-55pjj/perf-node-gather-daemonset-l4kdc" Jul 06 14:33:25.042005 ip-10-0-136-25 kubenswrapper[2569]: I0706 14:33:25.041988 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/8b20fa8f-441e-4f6a-92dd-a9d05e3660d0-lib-modules\") pod \"perf-node-gather-daemonset-l4kdc\" (UID: \"8b20fa8f-441e-4f6a-92dd-a9d05e3660d0\") " pod="openshift-must-gather-55pjj/perf-node-gather-daemonset-l4kdc" Jul 06 14:33:25.042114 ip-10-0-136-25 kubenswrapper[2569]: I0706 14:33:25.042032 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/8b20fa8f-441e-4f6a-92dd-a9d05e3660d0-sys\") pod \"perf-node-gather-daemonset-l4kdc\" (UID: \"8b20fa8f-441e-4f6a-92dd-a9d05e3660d0\") " pod="openshift-must-gather-55pjj/perf-node-gather-daemonset-l4kdc" Jul 06 14:33:25.042895 ip-10-0-136-25 kubenswrapper[2569]: I0706 14:33:25.042868 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/8b20fa8f-441e-4f6a-92dd-a9d05e3660d0-proc\") pod \"perf-node-gather-daemonset-l4kdc\" (UID: \"8b20fa8f-441e-4f6a-92dd-a9d05e3660d0\") " pod="openshift-must-gather-55pjj/perf-node-gather-daemonset-l4kdc" Jul 06 14:33:25.051922 ip-10-0-136-25 kubenswrapper[2569]: I0706 14:33:25.051885 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rb4v4\" (UniqueName: \"kubernetes.io/projected/8b20fa8f-441e-4f6a-92dd-a9d05e3660d0-kube-api-access-rb4v4\") pod \"perf-node-gather-daemonset-l4kdc\" (UID: \"8b20fa8f-441e-4f6a-92dd-a9d05e3660d0\") " pod="openshift-must-gather-55pjj/perf-node-gather-daemonset-l4kdc" Jul 06 14:33:25.094157 ip-10-0-136-25 kubenswrapper[2569]: I0706 14:33:25.094120 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-55pjj/perf-node-gather-daemonset-l4kdc" Jul 06 14:33:25.157727 ip-10-0-136-25 kubenswrapper[2569]: I0706 14:33:25.157698 2569 scope.go:117] "RemoveContainer" containerID="35ee3fdcb1a8aa117e0192350c5f553cceed2d2bed6b776bc0e1db8087a49a8f" Jul 06 14:33:25.283346 ip-10-0-136-25 kubenswrapper[2569]: I0706 14:33:25.283319 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-55pjj/perf-node-gather-daemonset-l4kdc"] Jul 06 14:33:25.286304 ip-10-0-136-25 kubenswrapper[2569]: W0706 14:33:25.286262 2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-pod8b20fa8f_441e_4f6a_92dd_a9d05e3660d0.slice/crio-a498b5eafe02cb7861bf674f21fd8a6504a81658d37abd2e8e095206a65119fd WatchSource:0}: Error finding container a498b5eafe02cb7861bf674f21fd8a6504a81658d37abd2e8e095206a65119fd: Status 404 returned error can't find the container with id a498b5eafe02cb7861bf674f21fd8a6504a81658d37abd2e8e095206a65119fd Jul 06 14:33:25.537439 ip-10-0-136-25 kubenswrapper[2569]: I0706 14:33:25.537374 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-68b757fb67-z2h95_dfffcdca-94d6-4732-815f-bcb533f538e9/console-operator/2.log" Jul 06 14:33:25.543115 ip-10-0-136-25 kubenswrapper[2569]: I0706 14:33:25.543085 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-68b757fb67-z2h95_dfffcdca-94d6-4732-815f-bcb533f538e9/console-operator/3.log" Jul 06 14:33:25.897160 ip-10-0-136-25 kubenswrapper[2569]: I0706 14:33:25.897117 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-55pjj/perf-node-gather-daemonset-l4kdc" event={"ID":"8b20fa8f-441e-4f6a-92dd-a9d05e3660d0","Type":"ContainerStarted","Data":"106a7f5508047a72bcddd0215a0da7c86b9dcfe2e88466e8aa40733bda4f8f9c"} Jul 06 14:33:25.897674 ip-10-0-136-25 kubenswrapper[2569]: I0706 14:33:25.897204 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-55pjj/perf-node-gather-daemonset-l4kdc" event={"ID":"8b20fa8f-441e-4f6a-92dd-a9d05e3660d0","Type":"ContainerStarted","Data":"a498b5eafe02cb7861bf674f21fd8a6504a81658d37abd2e8e095206a65119fd"} Jul 06 14:33:25.898110 ip-10-0-136-25 kubenswrapper[2569]: I0706 14:33:25.897999 2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-must-gather-55pjj/perf-node-gather-daemonset-l4kdc" Jul 06 14:33:25.913447 ip-10-0-136-25 kubenswrapper[2569]: I0706 14:33:25.913382 2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-must-gather-55pjj/perf-node-gather-daemonset-l4kdc" podStartSLOduration=1.913358184 podStartE2EDuration="1.913358184s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-06 14:33:24 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 14:33:25.911910551 +0000 UTC m=+2581.502472361" watchObservedRunningTime="2026-07-06 14:33:25.913358184 +0000 UTC m=+2581.503919963" Jul 06 14:33:26.046940 ip-10-0-136-25 kubenswrapper[2569]: I0706 14:33:26.046902 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console_downloads-685bfb5f64-nwr8k_e95b78c0-73d7-4e1d-aeab-d59969a3a319/download-server/0.log" Jul 06 14:33:27.678549 ip-10-0-136-25 kubenswrapper[2569]: I0706 14:33:27.678522 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-kbv9m_420fd5c7-d449-4869-960a-4b14a873c402/dns/0.log" Jul 06 14:33:27.700082 ip-10-0-136-25 kubenswrapper[2569]: I0706 14:33:27.700058 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-kbv9m_420fd5c7-d449-4869-960a-4b14a873c402/kube-rbac-proxy/0.log" Jul 06 14:33:27.771354 ip-10-0-136-25 kubenswrapper[2569]: I0706 14:33:27.771326 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-54w6r_ef1daf8b-7b12-4c6a-9fbd-2b7e9187fd8f/dns-node-resolver/0.log" Jul 06 14:33:28.356010 ip-10-0-136-25 kubenswrapper[2569]: I0706 14:33:28.355977 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_image-registry-7c57596984-r6bgq_83120e7b-a163-4061-a1c9-4b6d79ac47c5/registry/0.log" Jul 06 14:33:28.378022 ip-10-0-136-25 kubenswrapper[2569]: I0706 14:33:28.377993 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-c9sqv_028cbff5-11b2-411a-90ed-c0bd3f563830/node-ca/0.log" Jul 06 14:33:29.451158 ip-10-0-136-25 kubenswrapper[2569]: I0706 14:33:29.451128 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-7c6c964b95-sfw6w_1970cda6-a1e1-4aed-af85-44bf5bfd5dad/kube-auth-proxy/0.log" Jul 06 14:33:29.630944 ip-10-0-136-25 kubenswrapper[2569]: I0706 14:33:29.630906 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_router-default-966c76789-kpw4q_1cb95852-b3c5-4b1a-80bf-9d3fa465084a/router/0.log" Jul 06 14:33:30.290792 ip-10-0-136-25 kubenswrapper[2569]: I0706 14:33:30.290760 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-4wkrh_e5a5cf60-4e99-42b0-b705-96449966ca70/serve-healthcheck-canary/0.log" Jul 06 14:33:30.816039 ip-10-0-136-25 kubenswrapper[2569]: I0706 14:33:30.816002 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-operator-5c847dcfc4-tdvhm_9746832c-8d02-489c-98f4-afb86bdc46b8/insights-operator/0.log" Jul 06 14:33:30.818342 ip-10-0-136-25 kubenswrapper[2569]: I0706 14:33:30.818320 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-operator-5c847dcfc4-tdvhm_9746832c-8d02-489c-98f4-afb86bdc46b8/insights-operator/1.log" Jul 06 14:33:30.973182 ip-10-0-136-25 kubenswrapper[2569]: I0706 14:33:30.973155 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-kc97f_3c279543-7a20-4d29-9de1-6cd09f14f692/kube-rbac-proxy/0.log" Jul 06 14:33:30.994703 ip-10-0-136-25 kubenswrapper[2569]: I0706 14:33:30.994673 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-kc97f_3c279543-7a20-4d29-9de1-6cd09f14f692/exporter/0.log" Jul 06 14:33:31.017413 ip-10-0-136-25 kubenswrapper[2569]: I0706 14:33:31.017382 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-kc97f_3c279543-7a20-4d29-9de1-6cd09f14f692/extractor/0.log" Jul 06 14:33:31.913954 ip-10-0-136-25 kubenswrapper[2569]: I0706 14:33:31.913922 2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-must-gather-55pjj/perf-node-gather-daemonset-l4kdc" Jul 06 14:33:33.571608 ip-10-0-136-25 kubenswrapper[2569]: I0706 14:33:33.571573 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-5f8f457c8b-lvnlj_487646d3-ef03-4bcf-acf3-7f7051ff672f/manager/0.log" Jul 06 14:33:33.593369 ip-10-0-136-25 kubenswrapper[2569]: I0706 14:33:33.593329 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-5f8f457c8b-lvnlj_487646d3-ef03-4bcf-acf3-7f7051ff672f/copy-manifests/0.log" Jul 06 14:33:35.244543 ip-10-0-136-25 kubenswrapper[2569]: I0706 14:33:35.244467 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-lws-operator_lws-controller-manager-6c95bc9d56-t4784_f9633d41-40e9-4d38-8264-926db6a773c3/manager/0.log" Jul 06 14:33:35.641593 ip-10-0-136-25 kubenswrapper[2569]: I0706 14:33:35.641562 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-7c6c964b95-sfw6w_1970cda6-a1e1-4aed-af85-44bf5bfd5dad/kube-auth-proxy/0.log" Jul 06 14:33:35.780542 ip-10-0-136-25 kubenswrapper[2569]: I0706 14:33:35.780510 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_router-default-966c76789-kpw4q_1cb95852-b3c5-4b1a-80bf-9d3fa465084a/router/0.log" Jul 06 14:33:37.106312 ip-10-0-136-25 kubenswrapper[2569]: I0706 14:33:37.106270 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-7c6c964b95-sfw6w_1970cda6-a1e1-4aed-af85-44bf5bfd5dad/kube-auth-proxy/0.log" Jul 06 14:33:37.268582 ip-10-0-136-25 kubenswrapper[2569]: I0706 14:33:37.268547 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_router-default-966c76789-kpw4q_1cb95852-b3c5-4b1a-80bf-9d3fa465084a/router/0.log" Jul 06 14:33:40.605566 ip-10-0-136-25 kubenswrapper[2569]: I0706 14:33:40.605530 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator-operator_kube-storage-version-migrator-operator-846bf668f7-52nxm_2e5b9096-4169-413a-b200-49f75eb977c0/kube-storage-version-migrator-operator/1.log" Jul 06 14:33:40.606545 ip-10-0-136-25 kubenswrapper[2569]: I0706 14:33:40.606514 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator-operator_kube-storage-version-migrator-operator-846bf668f7-52nxm_2e5b9096-4169-413a-b200-49f75eb977c0/kube-storage-version-migrator-operator/0.log" Jul 06 14:33:41.579710 ip-10-0-136-25 kubenswrapper[2569]: I0706 14:33:41.579683 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-brmhg_edbb90e8-546b-4edd-a223-5ad4d579adaa/kube-multus-additional-cni-plugins/0.log" Jul 06 14:33:41.601493 ip-10-0-136-25 kubenswrapper[2569]: I0706 14:33:41.601471 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-brmhg_edbb90e8-546b-4edd-a223-5ad4d579adaa/egress-router-binary-copy/0.log" Jul 06 14:33:41.623506 ip-10-0-136-25 kubenswrapper[2569]: I0706 14:33:41.623481 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-brmhg_edbb90e8-546b-4edd-a223-5ad4d579adaa/cni-plugins/0.log" Jul 06 14:33:41.645394 ip-10-0-136-25 kubenswrapper[2569]: I0706 14:33:41.645367 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-brmhg_edbb90e8-546b-4edd-a223-5ad4d579adaa/bond-cni-plugin/0.log" Jul 06 14:33:41.665666 ip-10-0-136-25 kubenswrapper[2569]: I0706 14:33:41.665639 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-brmhg_edbb90e8-546b-4edd-a223-5ad4d579adaa/routeoverride-cni/0.log" Jul 06 14:33:41.689703 ip-10-0-136-25 kubenswrapper[2569]: I0706 14:33:41.689678 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-brmhg_edbb90e8-546b-4edd-a223-5ad4d579adaa/whereabouts-cni-bincopy/0.log" Jul 06 14:33:41.711648 ip-10-0-136-25 kubenswrapper[2569]: I0706 14:33:41.711622 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-brmhg_edbb90e8-546b-4edd-a223-5ad4d579adaa/whereabouts-cni/0.log" Jul 06 14:33:42.062800 ip-10-0-136-25 kubenswrapper[2569]: I0706 14:33:42.062770 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-dcdgc_07d7de38-90f7-4810-b5d6-3a710a3af5a9/kube-multus/0.log"