Jul 07 10:25:31.442843 ip-10-0-135-34 systemd[1]: kubelet.service: Failed to load environment files: No such file or directory Jul 07 10:25:31.442855 ip-10-0-135-34 systemd[1]: kubelet.service: Failed to run 'start-pre' task: No such file or directory Jul 07 10:25:31.442862 ip-10-0-135-34 systemd[1]: kubelet.service: Failed with result 'resources'. Jul 07 10:25:31.443078 ip-10-0-135-34 systemd[1]: Failed to start Kubernetes Kubelet. Jul 07 10:25:41.528257 ip-10-0-135-34 systemd[1]: kubelet.service: Failed to schedule restart job: Unit openvswitch.service not found. Jul 07 10:25:41.528269 ip-10-0-135-34 systemd[1]: kubelet.service: Failed with result 'resources'. -- Boot d14720fe859b4f0187f91c2f9f0860be -- Jul 07 10:27:11.195066 ip-10-0-135-34 systemd[1]: Starting Kubernetes Kubelet... Jul 07 10:27:11.642753 ip-10-0-135-34 kubenswrapper[2578]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Jul 07 10:27:11.642753 ip-10-0-135-34 kubenswrapper[2578]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version. Jul 07 10:27:11.642753 ip-10-0-135-34 kubenswrapper[2578]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Jul 07 10:27:11.644347 ip-10-0-135-34 kubenswrapper[2578]: Flag --pod-infra-container-image has been deprecated, will be removed in 1.35. Image garbage collector will get sandbox image information from CRI. Jul 07 10:27:11.644347 ip-10-0-135-34 kubenswrapper[2578]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Jul 07 10:27:11.645361 ip-10-0-135-34 kubenswrapper[2578]: I0707 10:27:11.645275 2578 server.go:213] "--pod-infra-container-image will not be pruned by the image garbage collector in kubelet and should also be set in the remote runtime" Jul 07 10:27:11.650575 ip-10-0-135-34 kubenswrapper[2578]: W0707 10:27:11.650555 2578 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 07 10:27:11.650575 ip-10-0-135-34 kubenswrapper[2578]: W0707 10:27:11.650576 2578 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 07 10:27:11.650676 ip-10-0-135-34 kubenswrapper[2578]: W0707 10:27:11.650579 2578 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 07 10:27:11.650676 ip-10-0-135-34 kubenswrapper[2578]: W0707 10:27:11.650583 2578 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 07 10:27:11.650676 ip-10-0-135-34 kubenswrapper[2578]: W0707 10:27:11.650586 2578 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 07 10:27:11.650676 ip-10-0-135-34 kubenswrapper[2578]: W0707 10:27:11.650589 2578 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 07 10:27:11.650676 ip-10-0-135-34 kubenswrapper[2578]: W0707 10:27:11.650591 2578 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 07 10:27:11.650676 ip-10-0-135-34 kubenswrapper[2578]: W0707 10:27:11.650595 2578 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 07 10:27:11.650676 ip-10-0-135-34 kubenswrapper[2578]: W0707 10:27:11.650597 2578 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 07 10:27:11.650676 ip-10-0-135-34 kubenswrapper[2578]: W0707 10:27:11.650600 2578 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 07 10:27:11.650676 ip-10-0-135-34 kubenswrapper[2578]: W0707 10:27:11.650603 2578 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 07 10:27:11.650676 ip-10-0-135-34 kubenswrapper[2578]: W0707 10:27:11.650605 2578 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 07 10:27:11.650676 ip-10-0-135-34 kubenswrapper[2578]: W0707 10:27:11.650608 2578 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 07 10:27:11.650676 ip-10-0-135-34 kubenswrapper[2578]: W0707 10:27:11.650617 2578 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 07 10:27:11.650676 ip-10-0-135-34 kubenswrapper[2578]: W0707 10:27:11.650620 2578 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 07 10:27:11.650676 ip-10-0-135-34 kubenswrapper[2578]: W0707 10:27:11.650623 2578 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 07 10:27:11.650676 ip-10-0-135-34 kubenswrapper[2578]: W0707 10:27:11.650625 2578 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 07 10:27:11.650676 ip-10-0-135-34 kubenswrapper[2578]: W0707 10:27:11.650628 2578 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 07 10:27:11.650676 ip-10-0-135-34 kubenswrapper[2578]: W0707 10:27:11.650631 2578 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 07 10:27:11.650676 ip-10-0-135-34 kubenswrapper[2578]: W0707 10:27:11.650633 2578 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 07 10:27:11.650676 ip-10-0-135-34 kubenswrapper[2578]: W0707 10:27:11.650636 2578 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 07 10:27:11.650676 ip-10-0-135-34 kubenswrapper[2578]: W0707 10:27:11.650639 2578 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 07 10:27:11.651174 ip-10-0-135-34 kubenswrapper[2578]: W0707 10:27:11.650641 2578 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 07 10:27:11.651174 ip-10-0-135-34 kubenswrapper[2578]: W0707 10:27:11.650644 2578 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 07 10:27:11.651174 ip-10-0-135-34 kubenswrapper[2578]: W0707 10:27:11.650647 2578 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 07 10:27:11.651174 ip-10-0-135-34 kubenswrapper[2578]: W0707 10:27:11.650650 2578 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 07 10:27:11.651174 ip-10-0-135-34 kubenswrapper[2578]: W0707 10:27:11.650653 2578 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 07 10:27:11.651174 ip-10-0-135-34 kubenswrapper[2578]: W0707 10:27:11.650655 2578 feature_gate.go:328] unrecognized feature gate: Example Jul 07 10:27:11.651174 ip-10-0-135-34 kubenswrapper[2578]: W0707 10:27:11.650657 2578 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 07 10:27:11.651174 ip-10-0-135-34 kubenswrapper[2578]: W0707 10:27:11.650660 2578 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 07 10:27:11.651174 ip-10-0-135-34 kubenswrapper[2578]: W0707 10:27:11.650662 2578 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 07 10:27:11.651174 ip-10-0-135-34 kubenswrapper[2578]: W0707 10:27:11.650665 2578 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 07 10:27:11.651174 ip-10-0-135-34 kubenswrapper[2578]: W0707 10:27:11.650667 2578 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 07 10:27:11.651174 ip-10-0-135-34 kubenswrapper[2578]: W0707 10:27:11.650670 2578 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 07 10:27:11.651174 ip-10-0-135-34 kubenswrapper[2578]: W0707 10:27:11.650673 2578 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 07 10:27:11.651174 ip-10-0-135-34 kubenswrapper[2578]: W0707 10:27:11.650676 2578 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 07 10:27:11.651174 ip-10-0-135-34 kubenswrapper[2578]: W0707 10:27:11.650679 2578 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 07 10:27:11.651174 ip-10-0-135-34 kubenswrapper[2578]: W0707 10:27:11.650682 2578 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 07 10:27:11.651174 ip-10-0-135-34 kubenswrapper[2578]: W0707 10:27:11.650684 2578 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 07 10:27:11.651174 ip-10-0-135-34 kubenswrapper[2578]: W0707 10:27:11.650689 2578 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 07 10:27:11.651174 ip-10-0-135-34 kubenswrapper[2578]: W0707 10:27:11.650693 2578 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 07 10:27:11.651631 ip-10-0-135-34 kubenswrapper[2578]: W0707 10:27:11.650696 2578 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 07 10:27:11.651631 ip-10-0-135-34 kubenswrapper[2578]: W0707 10:27:11.650699 2578 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 07 10:27:11.651631 ip-10-0-135-34 kubenswrapper[2578]: W0707 10:27:11.650702 2578 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 07 10:27:11.651631 ip-10-0-135-34 kubenswrapper[2578]: W0707 10:27:11.650705 2578 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 07 10:27:11.651631 ip-10-0-135-34 kubenswrapper[2578]: W0707 10:27:11.650707 2578 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 07 10:27:11.651631 ip-10-0-135-34 kubenswrapper[2578]: W0707 10:27:11.650710 2578 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 07 10:27:11.651631 ip-10-0-135-34 kubenswrapper[2578]: W0707 10:27:11.650713 2578 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 07 10:27:11.651631 ip-10-0-135-34 kubenswrapper[2578]: W0707 10:27:11.650716 2578 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 07 10:27:11.651631 ip-10-0-135-34 kubenswrapper[2578]: W0707 10:27:11.650718 2578 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 07 10:27:11.651631 ip-10-0-135-34 kubenswrapper[2578]: W0707 10:27:11.650721 2578 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 07 10:27:11.651631 ip-10-0-135-34 kubenswrapper[2578]: W0707 10:27:11.650724 2578 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 07 10:27:11.651631 ip-10-0-135-34 kubenswrapper[2578]: W0707 10:27:11.650726 2578 feature_gate.go:328] unrecognized feature gate: Example2 Jul 07 10:27:11.651631 ip-10-0-135-34 kubenswrapper[2578]: W0707 10:27:11.650729 2578 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 07 10:27:11.651631 ip-10-0-135-34 kubenswrapper[2578]: W0707 10:27:11.650731 2578 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 07 10:27:11.651631 ip-10-0-135-34 kubenswrapper[2578]: W0707 10:27:11.650734 2578 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 07 10:27:11.651631 ip-10-0-135-34 kubenswrapper[2578]: W0707 10:27:11.650737 2578 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 07 10:27:11.651631 ip-10-0-135-34 kubenswrapper[2578]: W0707 10:27:11.650739 2578 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 07 10:27:11.651631 ip-10-0-135-34 kubenswrapper[2578]: W0707 10:27:11.650742 2578 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 07 10:27:11.651631 ip-10-0-135-34 kubenswrapper[2578]: W0707 10:27:11.650745 2578 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 07 10:27:11.651631 ip-10-0-135-34 kubenswrapper[2578]: W0707 10:27:11.650747 2578 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 07 10:27:11.652172 ip-10-0-135-34 kubenswrapper[2578]: W0707 10:27:11.650750 2578 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 07 10:27:11.652172 ip-10-0-135-34 kubenswrapper[2578]: W0707 10:27:11.650752 2578 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 07 10:27:11.652172 ip-10-0-135-34 kubenswrapper[2578]: W0707 10:27:11.650754 2578 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 07 10:27:11.652172 ip-10-0-135-34 kubenswrapper[2578]: W0707 10:27:11.650757 2578 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 07 10:27:11.652172 ip-10-0-135-34 kubenswrapper[2578]: W0707 10:27:11.650759 2578 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 07 10:27:11.652172 ip-10-0-135-34 kubenswrapper[2578]: W0707 10:27:11.650781 2578 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 07 10:27:11.652172 ip-10-0-135-34 kubenswrapper[2578]: W0707 10:27:11.650785 2578 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 07 10:27:11.652172 ip-10-0-135-34 kubenswrapper[2578]: W0707 10:27:11.650787 2578 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 07 10:27:11.652172 ip-10-0-135-34 kubenswrapper[2578]: W0707 10:27:11.650792 2578 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 07 10:27:11.652172 ip-10-0-135-34 kubenswrapper[2578]: W0707 10:27:11.650795 2578 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 07 10:27:11.652172 ip-10-0-135-34 kubenswrapper[2578]: W0707 10:27:11.650799 2578 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 07 10:27:11.652172 ip-10-0-135-34 kubenswrapper[2578]: W0707 10:27:11.650801 2578 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 07 10:27:11.652172 ip-10-0-135-34 kubenswrapper[2578]: W0707 10:27:11.650807 2578 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 07 10:27:11.652172 ip-10-0-135-34 kubenswrapper[2578]: W0707 10:27:11.650810 2578 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 07 10:27:11.652172 ip-10-0-135-34 kubenswrapper[2578]: W0707 10:27:11.650813 2578 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 07 10:27:11.652172 ip-10-0-135-34 kubenswrapper[2578]: W0707 10:27:11.650816 2578 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 07 10:27:11.652172 ip-10-0-135-34 kubenswrapper[2578]: W0707 10:27:11.650818 2578 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 07 10:27:11.652172 ip-10-0-135-34 kubenswrapper[2578]: W0707 10:27:11.650821 2578 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 07 10:27:11.652172 ip-10-0-135-34 kubenswrapper[2578]: W0707 10:27:11.650823 2578 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 07 10:27:11.652627 ip-10-0-135-34 kubenswrapper[2578]: W0707 10:27:11.650826 2578 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 07 10:27:11.652627 ip-10-0-135-34 kubenswrapper[2578]: W0707 10:27:11.650828 2578 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 07 10:27:11.652627 ip-10-0-135-34 kubenswrapper[2578]: W0707 10:27:11.650831 2578 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 07 10:27:11.652627 ip-10-0-135-34 kubenswrapper[2578]: W0707 10:27:11.650834 2578 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 07 10:27:11.652627 ip-10-0-135-34 kubenswrapper[2578]: W0707 10:27:11.650836 2578 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 07 10:27:11.652627 ip-10-0-135-34 kubenswrapper[2578]: W0707 10:27:11.650839 2578 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 07 10:27:11.652627 ip-10-0-135-34 kubenswrapper[2578]: W0707 10:27:11.650841 2578 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 07 10:27:11.652627 ip-10-0-135-34 kubenswrapper[2578]: W0707 10:27:11.650844 2578 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 07 10:27:11.652627 ip-10-0-135-34 kubenswrapper[2578]: W0707 10:27:11.650846 2578 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 07 10:27:11.652627 ip-10-0-135-34 kubenswrapper[2578]: W0707 10:27:11.650850 2578 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 07 10:27:11.652627 ip-10-0-135-34 kubenswrapper[2578]: W0707 10:27:11.650852 2578 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 07 10:27:11.652627 ip-10-0-135-34 kubenswrapper[2578]: W0707 10:27:11.650855 2578 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 07 10:27:11.652627 ip-10-0-135-34 kubenswrapper[2578]: W0707 10:27:11.650857 2578 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 07 10:27:11.652627 ip-10-0-135-34 kubenswrapper[2578]: W0707 10:27:11.650860 2578 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 07 10:27:11.652627 ip-10-0-135-34 kubenswrapper[2578]: W0707 10:27:11.650862 2578 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 07 10:27:11.652627 ip-10-0-135-34 kubenswrapper[2578]: W0707 10:27:11.650865 2578 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 07 10:27:11.652627 ip-10-0-135-34 kubenswrapper[2578]: W0707 10:27:11.650868 2578 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 07 10:27:11.652627 ip-10-0-135-34 kubenswrapper[2578]: W0707 10:27:11.650871 2578 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 07 10:27:11.652627 ip-10-0-135-34 kubenswrapper[2578]: W0707 10:27:11.650874 2578 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 07 10:27:11.652627 ip-10-0-135-34 kubenswrapper[2578]: W0707 10:27:11.651605 2578 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 07 10:27:11.653119 ip-10-0-135-34 kubenswrapper[2578]: W0707 10:27:11.651612 2578 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 07 10:27:11.653119 ip-10-0-135-34 kubenswrapper[2578]: W0707 10:27:11.651615 2578 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 07 10:27:11.653119 ip-10-0-135-34 kubenswrapper[2578]: W0707 10:27:11.651619 2578 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 07 10:27:11.653119 ip-10-0-135-34 kubenswrapper[2578]: W0707 10:27:11.651621 2578 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 07 10:27:11.653119 ip-10-0-135-34 kubenswrapper[2578]: W0707 10:27:11.651624 2578 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 07 10:27:11.653119 ip-10-0-135-34 kubenswrapper[2578]: W0707 10:27:11.651627 2578 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 07 10:27:11.653119 ip-10-0-135-34 kubenswrapper[2578]: W0707 10:27:11.651629 2578 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 07 10:27:11.653119 ip-10-0-135-34 kubenswrapper[2578]: W0707 10:27:11.651632 2578 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 07 10:27:11.653119 ip-10-0-135-34 kubenswrapper[2578]: W0707 10:27:11.651634 2578 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 07 10:27:11.653119 ip-10-0-135-34 kubenswrapper[2578]: W0707 10:27:11.651637 2578 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 07 10:27:11.653119 ip-10-0-135-34 kubenswrapper[2578]: W0707 10:27:11.651640 2578 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 07 10:27:11.653119 ip-10-0-135-34 kubenswrapper[2578]: W0707 10:27:11.651642 2578 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 07 10:27:11.653119 ip-10-0-135-34 kubenswrapper[2578]: W0707 10:27:11.651645 2578 feature_gate.go:328] unrecognized feature gate: Example2 Jul 07 10:27:11.653119 ip-10-0-135-34 kubenswrapper[2578]: W0707 10:27:11.651648 2578 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 07 10:27:11.653119 ip-10-0-135-34 kubenswrapper[2578]: W0707 10:27:11.651650 2578 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 07 10:27:11.653119 ip-10-0-135-34 kubenswrapper[2578]: W0707 10:27:11.651653 2578 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 07 10:27:11.653119 ip-10-0-135-34 kubenswrapper[2578]: W0707 10:27:11.651655 2578 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 07 10:27:11.653119 ip-10-0-135-34 kubenswrapper[2578]: W0707 10:27:11.651658 2578 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 07 10:27:11.653119 ip-10-0-135-34 kubenswrapper[2578]: W0707 10:27:11.651661 2578 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 07 10:27:11.653119 ip-10-0-135-34 kubenswrapper[2578]: W0707 10:27:11.651663 2578 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 07 10:27:11.653607 ip-10-0-135-34 kubenswrapper[2578]: W0707 10:27:11.651666 2578 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 07 10:27:11.653607 ip-10-0-135-34 kubenswrapper[2578]: W0707 10:27:11.651668 2578 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 07 10:27:11.653607 ip-10-0-135-34 kubenswrapper[2578]: W0707 10:27:11.651671 2578 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 07 10:27:11.653607 ip-10-0-135-34 kubenswrapper[2578]: W0707 10:27:11.651673 2578 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 07 10:27:11.653607 ip-10-0-135-34 kubenswrapper[2578]: W0707 10:27:11.651676 2578 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 07 10:27:11.653607 ip-10-0-135-34 kubenswrapper[2578]: W0707 10:27:11.651678 2578 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 07 10:27:11.653607 ip-10-0-135-34 kubenswrapper[2578]: W0707 10:27:11.651681 2578 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 07 10:27:11.653607 ip-10-0-135-34 kubenswrapper[2578]: W0707 10:27:11.651684 2578 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 07 10:27:11.653607 ip-10-0-135-34 kubenswrapper[2578]: W0707 10:27:11.651686 2578 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 07 10:27:11.653607 ip-10-0-135-34 kubenswrapper[2578]: W0707 10:27:11.651689 2578 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 07 10:27:11.653607 ip-10-0-135-34 kubenswrapper[2578]: W0707 10:27:11.651691 2578 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 07 10:27:11.653607 ip-10-0-135-34 kubenswrapper[2578]: W0707 10:27:11.651694 2578 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 07 10:27:11.653607 ip-10-0-135-34 kubenswrapper[2578]: W0707 10:27:11.651702 2578 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 07 10:27:11.653607 ip-10-0-135-34 kubenswrapper[2578]: W0707 10:27:11.651704 2578 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 07 10:27:11.653607 ip-10-0-135-34 kubenswrapper[2578]: W0707 10:27:11.651707 2578 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 07 10:27:11.653607 ip-10-0-135-34 kubenswrapper[2578]: W0707 10:27:11.651709 2578 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 07 10:27:11.653607 ip-10-0-135-34 kubenswrapper[2578]: W0707 10:27:11.651712 2578 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 07 10:27:11.653607 ip-10-0-135-34 kubenswrapper[2578]: W0707 10:27:11.651714 2578 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 07 10:27:11.653607 ip-10-0-135-34 kubenswrapper[2578]: W0707 10:27:11.651717 2578 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 07 10:27:11.653607 ip-10-0-135-34 kubenswrapper[2578]: W0707 10:27:11.651719 2578 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 07 10:27:11.654110 ip-10-0-135-34 kubenswrapper[2578]: W0707 10:27:11.651723 2578 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 07 10:27:11.654110 ip-10-0-135-34 kubenswrapper[2578]: W0707 10:27:11.651727 2578 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 07 10:27:11.654110 ip-10-0-135-34 kubenswrapper[2578]: W0707 10:27:11.651730 2578 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 07 10:27:11.654110 ip-10-0-135-34 kubenswrapper[2578]: W0707 10:27:11.651732 2578 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 07 10:27:11.654110 ip-10-0-135-34 kubenswrapper[2578]: W0707 10:27:11.651736 2578 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 07 10:27:11.654110 ip-10-0-135-34 kubenswrapper[2578]: W0707 10:27:11.651739 2578 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 07 10:27:11.654110 ip-10-0-135-34 kubenswrapper[2578]: W0707 10:27:11.651741 2578 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 07 10:27:11.654110 ip-10-0-135-34 kubenswrapper[2578]: W0707 10:27:11.651744 2578 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 07 10:27:11.654110 ip-10-0-135-34 kubenswrapper[2578]: W0707 10:27:11.651746 2578 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 07 10:27:11.654110 ip-10-0-135-34 kubenswrapper[2578]: W0707 10:27:11.651750 2578 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 07 10:27:11.654110 ip-10-0-135-34 kubenswrapper[2578]: W0707 10:27:11.651753 2578 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 07 10:27:11.654110 ip-10-0-135-34 kubenswrapper[2578]: W0707 10:27:11.651755 2578 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 07 10:27:11.654110 ip-10-0-135-34 kubenswrapper[2578]: W0707 10:27:11.651758 2578 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 07 10:27:11.654110 ip-10-0-135-34 kubenswrapper[2578]: W0707 10:27:11.651761 2578 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 07 10:27:11.654110 ip-10-0-135-34 kubenswrapper[2578]: W0707 10:27:11.651763 2578 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 07 10:27:11.654110 ip-10-0-135-34 kubenswrapper[2578]: W0707 10:27:11.651781 2578 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 07 10:27:11.654110 ip-10-0-135-34 kubenswrapper[2578]: W0707 10:27:11.651784 2578 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 07 10:27:11.654110 ip-10-0-135-34 kubenswrapper[2578]: W0707 10:27:11.651786 2578 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 07 10:27:11.654110 ip-10-0-135-34 kubenswrapper[2578]: W0707 10:27:11.651789 2578 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 07 10:27:11.654581 ip-10-0-135-34 kubenswrapper[2578]: W0707 10:27:11.651791 2578 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 07 10:27:11.654581 ip-10-0-135-34 kubenswrapper[2578]: W0707 10:27:11.651793 2578 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 07 10:27:11.654581 ip-10-0-135-34 kubenswrapper[2578]: W0707 10:27:11.651796 2578 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 07 10:27:11.654581 ip-10-0-135-34 kubenswrapper[2578]: W0707 10:27:11.651801 2578 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 07 10:27:11.654581 ip-10-0-135-34 kubenswrapper[2578]: W0707 10:27:11.651804 2578 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 07 10:27:11.654581 ip-10-0-135-34 kubenswrapper[2578]: W0707 10:27:11.651807 2578 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 07 10:27:11.654581 ip-10-0-135-34 kubenswrapper[2578]: W0707 10:27:11.651810 2578 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 07 10:27:11.654581 ip-10-0-135-34 kubenswrapper[2578]: W0707 10:27:11.651813 2578 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 07 10:27:11.654581 ip-10-0-135-34 kubenswrapper[2578]: W0707 10:27:11.651815 2578 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 07 10:27:11.654581 ip-10-0-135-34 kubenswrapper[2578]: W0707 10:27:11.651818 2578 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 07 10:27:11.654581 ip-10-0-135-34 kubenswrapper[2578]: W0707 10:27:11.651821 2578 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 07 10:27:11.654581 ip-10-0-135-34 kubenswrapper[2578]: W0707 10:27:11.651823 2578 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 07 10:27:11.654581 ip-10-0-135-34 kubenswrapper[2578]: W0707 10:27:11.651826 2578 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 07 10:27:11.654581 ip-10-0-135-34 kubenswrapper[2578]: W0707 10:27:11.651829 2578 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 07 10:27:11.654581 ip-10-0-135-34 kubenswrapper[2578]: W0707 10:27:11.651831 2578 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 07 10:27:11.654581 ip-10-0-135-34 kubenswrapper[2578]: W0707 10:27:11.651833 2578 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 07 10:27:11.654581 ip-10-0-135-34 kubenswrapper[2578]: W0707 10:27:11.651836 2578 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 07 10:27:11.654581 ip-10-0-135-34 kubenswrapper[2578]: W0707 10:27:11.651838 2578 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 07 10:27:11.654581 ip-10-0-135-34 kubenswrapper[2578]: W0707 10:27:11.651840 2578 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 07 10:27:11.655074 ip-10-0-135-34 kubenswrapper[2578]: W0707 10:27:11.651842 2578 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 07 10:27:11.655074 ip-10-0-135-34 kubenswrapper[2578]: W0707 10:27:11.651845 2578 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 07 10:27:11.655074 ip-10-0-135-34 kubenswrapper[2578]: W0707 10:27:11.651847 2578 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 07 10:27:11.655074 ip-10-0-135-34 kubenswrapper[2578]: W0707 10:27:11.651850 2578 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 07 10:27:11.655074 ip-10-0-135-34 kubenswrapper[2578]: W0707 10:27:11.651852 2578 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 07 10:27:11.655074 ip-10-0-135-34 kubenswrapper[2578]: W0707 10:27:11.651855 2578 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 07 10:27:11.655074 ip-10-0-135-34 kubenswrapper[2578]: W0707 10:27:11.651857 2578 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 07 10:27:11.655074 ip-10-0-135-34 kubenswrapper[2578]: W0707 10:27:11.651860 2578 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 07 10:27:11.655074 ip-10-0-135-34 kubenswrapper[2578]: W0707 10:27:11.651863 2578 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 07 10:27:11.655074 ip-10-0-135-34 kubenswrapper[2578]: W0707 10:27:11.651865 2578 feature_gate.go:328] unrecognized feature gate: Example Jul 07 10:27:11.655074 ip-10-0-135-34 kubenswrapper[2578]: W0707 10:27:11.651867 2578 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 07 10:27:11.655074 ip-10-0-135-34 kubenswrapper[2578]: W0707 10:27:11.651870 2578 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 07 10:27:11.655074 ip-10-0-135-34 kubenswrapper[2578]: W0707 10:27:11.651872 2578 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 07 10:27:11.655074 ip-10-0-135-34 kubenswrapper[2578]: W0707 10:27:11.651875 2578 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 07 10:27:11.655074 ip-10-0-135-34 kubenswrapper[2578]: W0707 10:27:11.651877 2578 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 07 10:27:11.655074 ip-10-0-135-34 kubenswrapper[2578]: W0707 10:27:11.651879 2578 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 07 10:27:11.655074 ip-10-0-135-34 kubenswrapper[2578]: W0707 10:27:11.651882 2578 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 07 10:27:11.655074 ip-10-0-135-34 kubenswrapper[2578]: W0707 10:27:11.651884 2578 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 07 10:27:11.655074 ip-10-0-135-34 kubenswrapper[2578]: W0707 10:27:11.651886 2578 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 07 10:27:11.655074 ip-10-0-135-34 kubenswrapper[2578]: W0707 10:27:11.651889 2578 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 07 10:27:11.655574 ip-10-0-135-34 kubenswrapper[2578]: W0707 10:27:11.652300 2578 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 07 10:27:11.655574 ip-10-0-135-34 kubenswrapper[2578]: W0707 10:27:11.652305 2578 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 07 10:27:11.655574 ip-10-0-135-34 kubenswrapper[2578]: W0707 10:27:11.652311 2578 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 07 10:27:11.655574 ip-10-0-135-34 kubenswrapper[2578]: W0707 10:27:11.652313 2578 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 07 10:27:11.655574 ip-10-0-135-34 kubenswrapper[2578]: W0707 10:27:11.652316 2578 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 07 10:27:11.655574 ip-10-0-135-34 kubenswrapper[2578]: W0707 10:27:11.652318 2578 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 07 10:27:11.655574 ip-10-0-135-34 kubenswrapper[2578]: W0707 10:27:11.652321 2578 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 07 10:27:11.655574 ip-10-0-135-34 kubenswrapper[2578]: W0707 10:27:11.652324 2578 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 07 10:27:11.655574 ip-10-0-135-34 kubenswrapper[2578]: W0707 10:27:11.652326 2578 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 07 10:27:11.655574 ip-10-0-135-34 kubenswrapper[2578]: W0707 10:27:11.652328 2578 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 07 10:27:11.655574 ip-10-0-135-34 kubenswrapper[2578]: W0707 10:27:11.652331 2578 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 07 10:27:11.655574 ip-10-0-135-34 kubenswrapper[2578]: W0707 10:27:11.652335 2578 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 07 10:27:11.655574 ip-10-0-135-34 kubenswrapper[2578]: W0707 10:27:11.652339 2578 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 07 10:27:11.655574 ip-10-0-135-34 kubenswrapper[2578]: W0707 10:27:11.652342 2578 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 07 10:27:11.655574 ip-10-0-135-34 kubenswrapper[2578]: W0707 10:27:11.652344 2578 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 07 10:27:11.655574 ip-10-0-135-34 kubenswrapper[2578]: W0707 10:27:11.652347 2578 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 07 10:27:11.655574 ip-10-0-135-34 kubenswrapper[2578]: W0707 10:27:11.652349 2578 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 07 10:27:11.655574 ip-10-0-135-34 kubenswrapper[2578]: W0707 10:27:11.652352 2578 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 07 10:27:11.655574 ip-10-0-135-34 kubenswrapper[2578]: W0707 10:27:11.652354 2578 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 07 10:27:11.656068 ip-10-0-135-34 kubenswrapper[2578]: W0707 10:27:11.652357 2578 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 07 10:27:11.656068 ip-10-0-135-34 kubenswrapper[2578]: W0707 10:27:11.652360 2578 feature_gate.go:328] unrecognized feature gate: Example Jul 07 10:27:11.656068 ip-10-0-135-34 kubenswrapper[2578]: W0707 10:27:11.652363 2578 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 07 10:27:11.656068 ip-10-0-135-34 kubenswrapper[2578]: W0707 10:27:11.652365 2578 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 07 10:27:11.656068 ip-10-0-135-34 kubenswrapper[2578]: W0707 10:27:11.652368 2578 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 07 10:27:11.656068 ip-10-0-135-34 kubenswrapper[2578]: W0707 10:27:11.652370 2578 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 07 10:27:11.656068 ip-10-0-135-34 kubenswrapper[2578]: W0707 10:27:11.652373 2578 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 07 10:27:11.656068 ip-10-0-135-34 kubenswrapper[2578]: W0707 10:27:11.652375 2578 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 07 10:27:11.656068 ip-10-0-135-34 kubenswrapper[2578]: W0707 10:27:11.652378 2578 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 07 10:27:11.656068 ip-10-0-135-34 kubenswrapper[2578]: W0707 10:27:11.652382 2578 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 07 10:27:11.656068 ip-10-0-135-34 kubenswrapper[2578]: W0707 10:27:11.652385 2578 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 07 10:27:11.656068 ip-10-0-135-34 kubenswrapper[2578]: W0707 10:27:11.652387 2578 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 07 10:27:11.656068 ip-10-0-135-34 kubenswrapper[2578]: W0707 10:27:11.652390 2578 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 07 10:27:11.656068 ip-10-0-135-34 kubenswrapper[2578]: W0707 10:27:11.652392 2578 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 07 10:27:11.656068 ip-10-0-135-34 kubenswrapper[2578]: W0707 10:27:11.652395 2578 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 07 10:27:11.656068 ip-10-0-135-34 kubenswrapper[2578]: W0707 10:27:11.652399 2578 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 07 10:27:11.656068 ip-10-0-135-34 kubenswrapper[2578]: W0707 10:27:11.652401 2578 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 07 10:27:11.656068 ip-10-0-135-34 kubenswrapper[2578]: W0707 10:27:11.652404 2578 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 07 10:27:11.656068 ip-10-0-135-34 kubenswrapper[2578]: W0707 10:27:11.652408 2578 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 07 10:27:11.656517 ip-10-0-135-34 kubenswrapper[2578]: W0707 10:27:11.652411 2578 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 07 10:27:11.656517 ip-10-0-135-34 kubenswrapper[2578]: W0707 10:27:11.652414 2578 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 07 10:27:11.656517 ip-10-0-135-34 kubenswrapper[2578]: W0707 10:27:11.652416 2578 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 07 10:27:11.656517 ip-10-0-135-34 kubenswrapper[2578]: W0707 10:27:11.652419 2578 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 07 10:27:11.656517 ip-10-0-135-34 kubenswrapper[2578]: W0707 10:27:11.652422 2578 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 07 10:27:11.656517 ip-10-0-135-34 kubenswrapper[2578]: W0707 10:27:11.652424 2578 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 07 10:27:11.656517 ip-10-0-135-34 kubenswrapper[2578]: W0707 10:27:11.652427 2578 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 07 10:27:11.656517 ip-10-0-135-34 kubenswrapper[2578]: W0707 10:27:11.652429 2578 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 07 10:27:11.656517 ip-10-0-135-34 kubenswrapper[2578]: W0707 10:27:11.652432 2578 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 07 10:27:11.656517 ip-10-0-135-34 kubenswrapper[2578]: W0707 10:27:11.652434 2578 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 07 10:27:11.656517 ip-10-0-135-34 kubenswrapper[2578]: W0707 10:27:11.652437 2578 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 07 10:27:11.656517 ip-10-0-135-34 kubenswrapper[2578]: W0707 10:27:11.652439 2578 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 07 10:27:11.656517 ip-10-0-135-34 kubenswrapper[2578]: W0707 10:27:11.652441 2578 feature_gate.go:328] unrecognized feature gate: Example2 Jul 07 10:27:11.656517 ip-10-0-135-34 kubenswrapper[2578]: W0707 10:27:11.652444 2578 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 07 10:27:11.656517 ip-10-0-135-34 kubenswrapper[2578]: W0707 10:27:11.652446 2578 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 07 10:27:11.656517 ip-10-0-135-34 kubenswrapper[2578]: W0707 10:27:11.652448 2578 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 07 10:27:11.656517 ip-10-0-135-34 kubenswrapper[2578]: W0707 10:27:11.652451 2578 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 07 10:27:11.656517 ip-10-0-135-34 kubenswrapper[2578]: W0707 10:27:11.652454 2578 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 07 10:27:11.656517 ip-10-0-135-34 kubenswrapper[2578]: W0707 10:27:11.652456 2578 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 07 10:27:11.656517 ip-10-0-135-34 kubenswrapper[2578]: W0707 10:27:11.652459 2578 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 07 10:27:11.657023 ip-10-0-135-34 kubenswrapper[2578]: W0707 10:27:11.652461 2578 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 07 10:27:11.657023 ip-10-0-135-34 kubenswrapper[2578]: W0707 10:27:11.652464 2578 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 07 10:27:11.657023 ip-10-0-135-34 kubenswrapper[2578]: W0707 10:27:11.652466 2578 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 07 10:27:11.657023 ip-10-0-135-34 kubenswrapper[2578]: W0707 10:27:11.652469 2578 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 07 10:27:11.657023 ip-10-0-135-34 kubenswrapper[2578]: W0707 10:27:11.652471 2578 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 07 10:27:11.657023 ip-10-0-135-34 kubenswrapper[2578]: W0707 10:27:11.652473 2578 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 07 10:27:11.657023 ip-10-0-135-34 kubenswrapper[2578]: W0707 10:27:11.652476 2578 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 07 10:27:11.657023 ip-10-0-135-34 kubenswrapper[2578]: W0707 10:27:11.652479 2578 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 07 10:27:11.657023 ip-10-0-135-34 kubenswrapper[2578]: W0707 10:27:11.652483 2578 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 07 10:27:11.657023 ip-10-0-135-34 kubenswrapper[2578]: W0707 10:27:11.652485 2578 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 07 10:27:11.657023 ip-10-0-135-34 kubenswrapper[2578]: W0707 10:27:11.652488 2578 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 07 10:27:11.657023 ip-10-0-135-34 kubenswrapper[2578]: W0707 10:27:11.652490 2578 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 07 10:27:11.657023 ip-10-0-135-34 kubenswrapper[2578]: W0707 10:27:11.652492 2578 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 07 10:27:11.657023 ip-10-0-135-34 kubenswrapper[2578]: W0707 10:27:11.652495 2578 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 07 10:27:11.657023 ip-10-0-135-34 kubenswrapper[2578]: W0707 10:27:11.652497 2578 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 07 10:27:11.657023 ip-10-0-135-34 kubenswrapper[2578]: W0707 10:27:11.652501 2578 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 07 10:27:11.657023 ip-10-0-135-34 kubenswrapper[2578]: W0707 10:27:11.652503 2578 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 07 10:27:11.657023 ip-10-0-135-34 kubenswrapper[2578]: W0707 10:27:11.652506 2578 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 07 10:27:11.657023 ip-10-0-135-34 kubenswrapper[2578]: W0707 10:27:11.652508 2578 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 07 10:27:11.657023 ip-10-0-135-34 kubenswrapper[2578]: W0707 10:27:11.652511 2578 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 07 10:27:11.657556 ip-10-0-135-34 kubenswrapper[2578]: W0707 10:27:11.652517 2578 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 07 10:27:11.657556 ip-10-0-135-34 kubenswrapper[2578]: W0707 10:27:11.652519 2578 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 07 10:27:11.657556 ip-10-0-135-34 kubenswrapper[2578]: W0707 10:27:11.652522 2578 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 07 10:27:11.657556 ip-10-0-135-34 kubenswrapper[2578]: W0707 10:27:11.652524 2578 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 07 10:27:11.657556 ip-10-0-135-34 kubenswrapper[2578]: W0707 10:27:11.652527 2578 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 07 10:27:11.657556 ip-10-0-135-34 kubenswrapper[2578]: W0707 10:27:11.652529 2578 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 07 10:27:11.657556 ip-10-0-135-34 kubenswrapper[2578]: W0707 10:27:11.652531 2578 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 07 10:27:11.657556 ip-10-0-135-34 kubenswrapper[2578]: W0707 10:27:11.652534 2578 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 07 10:27:11.657556 ip-10-0-135-34 kubenswrapper[2578]: W0707 10:27:11.652536 2578 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 07 10:27:11.657556 ip-10-0-135-34 kubenswrapper[2578]: W0707 10:27:11.652539 2578 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 07 10:27:11.657556 ip-10-0-135-34 kubenswrapper[2578]: W0707 10:27:11.652542 2578 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 07 10:27:11.657556 ip-10-0-135-34 kubenswrapper[2578]: W0707 10:27:11.652545 2578 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 07 10:27:11.657556 ip-10-0-135-34 kubenswrapper[2578]: W0707 10:27:11.652547 2578 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 07 10:27:11.657556 ip-10-0-135-34 kubenswrapper[2578]: W0707 10:27:11.652549 2578 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 07 10:27:11.657556 ip-10-0-135-34 kubenswrapper[2578]: W0707 10:27:11.652552 2578 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 07 10:27:11.657556 ip-10-0-135-34 kubenswrapper[2578]: W0707 10:27:11.652554 2578 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 07 10:27:11.657556 ip-10-0-135-34 kubenswrapper[2578]: W0707 10:27:11.652557 2578 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 07 10:27:11.657556 ip-10-0-135-34 kubenswrapper[2578]: W0707 10:27:11.652559 2578 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 07 10:27:11.657556 ip-10-0-135-34 kubenswrapper[2578]: W0707 10:27:11.652561 2578 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 07 10:27:11.657556 ip-10-0-135-34 kubenswrapper[2578]: W0707 10:27:11.652564 2578 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 07 10:27:11.658043 ip-10-0-135-34 kubenswrapper[2578]: W0707 10:27:11.652568 2578 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 07 10:27:11.658043 ip-10-0-135-34 kubenswrapper[2578]: I0707 10:27:11.652640 2578 flags.go:64] FLAG: --address="0.0.0.0" Jul 07 10:27:11.658043 ip-10-0-135-34 kubenswrapper[2578]: I0707 10:27:11.652647 2578 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]" Jul 07 10:27:11.658043 ip-10-0-135-34 kubenswrapper[2578]: I0707 10:27:11.652654 2578 flags.go:64] FLAG: --anonymous-auth="true" Jul 07 10:27:11.658043 ip-10-0-135-34 kubenswrapper[2578]: I0707 10:27:11.652658 2578 flags.go:64] FLAG: --application-metrics-count-limit="100" Jul 07 10:27:11.658043 ip-10-0-135-34 kubenswrapper[2578]: I0707 10:27:11.652663 2578 flags.go:64] FLAG: --authentication-token-webhook="false" Jul 07 10:27:11.658043 ip-10-0-135-34 kubenswrapper[2578]: I0707 10:27:11.652666 2578 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s" Jul 07 10:27:11.658043 ip-10-0-135-34 kubenswrapper[2578]: I0707 10:27:11.652671 2578 flags.go:64] FLAG: --authorization-mode="AlwaysAllow" Jul 07 10:27:11.658043 ip-10-0-135-34 kubenswrapper[2578]: I0707 10:27:11.652678 2578 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s" Jul 07 10:27:11.658043 ip-10-0-135-34 kubenswrapper[2578]: I0707 10:27:11.652681 2578 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s" Jul 07 10:27:11.658043 ip-10-0-135-34 kubenswrapper[2578]: I0707 10:27:11.652684 2578 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id" Jul 07 10:27:11.658043 ip-10-0-135-34 kubenswrapper[2578]: I0707 10:27:11.652688 2578 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig" Jul 07 10:27:11.658043 ip-10-0-135-34 kubenswrapper[2578]: I0707 10:27:11.652691 2578 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki" Jul 07 10:27:11.658043 ip-10-0-135-34 kubenswrapper[2578]: I0707 10:27:11.652694 2578 flags.go:64] FLAG: --cgroup-driver="cgroupfs" Jul 07 10:27:11.658043 ip-10-0-135-34 kubenswrapper[2578]: I0707 10:27:11.652697 2578 flags.go:64] FLAG: --cgroup-root="" Jul 07 10:27:11.658043 ip-10-0-135-34 kubenswrapper[2578]: I0707 10:27:11.652700 2578 flags.go:64] FLAG: --cgroups-per-qos="true" Jul 07 10:27:11.658043 ip-10-0-135-34 kubenswrapper[2578]: I0707 10:27:11.652703 2578 flags.go:64] FLAG: --client-ca-file="" Jul 07 10:27:11.658043 ip-10-0-135-34 kubenswrapper[2578]: I0707 10:27:11.652705 2578 flags.go:64] FLAG: --cloud-provider="external" Jul 07 10:27:11.658043 ip-10-0-135-34 kubenswrapper[2578]: I0707 10:27:11.652708 2578 flags.go:64] FLAG: --cluster-dns="[]" Jul 07 10:27:11.658043 ip-10-0-135-34 kubenswrapper[2578]: I0707 10:27:11.652713 2578 flags.go:64] FLAG: --cluster-domain="" Jul 07 10:27:11.658043 ip-10-0-135-34 kubenswrapper[2578]: I0707 10:27:11.652716 2578 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf" Jul 07 10:27:11.658043 ip-10-0-135-34 kubenswrapper[2578]: I0707 10:27:11.652719 2578 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d" Jul 07 10:27:11.658043 ip-10-0-135-34 kubenswrapper[2578]: I0707 10:27:11.652722 2578 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json" Jul 07 10:27:11.658043 ip-10-0-135-34 kubenswrapper[2578]: I0707 10:27:11.652725 2578 flags.go:64] FLAG: --container-log-max-files="5" Jul 07 10:27:11.658639 ip-10-0-135-34 kubenswrapper[2578]: I0707 10:27:11.652729 2578 flags.go:64] FLAG: --container-log-max-size="10Mi" Jul 07 10:27:11.658639 ip-10-0-135-34 kubenswrapper[2578]: I0707 10:27:11.652732 2578 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock" Jul 07 10:27:11.658639 ip-10-0-135-34 kubenswrapper[2578]: I0707 10:27:11.652735 2578 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock" Jul 07 10:27:11.658639 ip-10-0-135-34 kubenswrapper[2578]: I0707 10:27:11.652738 2578 flags.go:64] FLAG: --containerd-namespace="k8s.io" Jul 07 10:27:11.658639 ip-10-0-135-34 kubenswrapper[2578]: I0707 10:27:11.652741 2578 flags.go:64] FLAG: --contention-profiling="false" Jul 07 10:27:11.658639 ip-10-0-135-34 kubenswrapper[2578]: I0707 10:27:11.652744 2578 flags.go:64] FLAG: --cpu-cfs-quota="true" Jul 07 10:27:11.658639 ip-10-0-135-34 kubenswrapper[2578]: I0707 10:27:11.652747 2578 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms" Jul 07 10:27:11.658639 ip-10-0-135-34 kubenswrapper[2578]: I0707 10:27:11.652751 2578 flags.go:64] FLAG: --cpu-manager-policy="none" Jul 07 10:27:11.658639 ip-10-0-135-34 kubenswrapper[2578]: I0707 10:27:11.652754 2578 flags.go:64] FLAG: --cpu-manager-policy-options="" Jul 07 10:27:11.658639 ip-10-0-135-34 kubenswrapper[2578]: I0707 10:27:11.652760 2578 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s" Jul 07 10:27:11.658639 ip-10-0-135-34 kubenswrapper[2578]: I0707 10:27:11.652763 2578 flags.go:64] FLAG: --enable-controller-attach-detach="true" Jul 07 10:27:11.658639 ip-10-0-135-34 kubenswrapper[2578]: I0707 10:27:11.652780 2578 flags.go:64] FLAG: --enable-debugging-handlers="true" Jul 07 10:27:11.658639 ip-10-0-135-34 kubenswrapper[2578]: I0707 10:27:11.652783 2578 flags.go:64] FLAG: --enable-load-reader="false" Jul 07 10:27:11.658639 ip-10-0-135-34 kubenswrapper[2578]: I0707 10:27:11.652787 2578 flags.go:64] FLAG: --enable-server="true" Jul 07 10:27:11.658639 ip-10-0-135-34 kubenswrapper[2578]: I0707 10:27:11.652789 2578 flags.go:64] FLAG: --enforce-node-allocatable="[pods]" Jul 07 10:27:11.658639 ip-10-0-135-34 kubenswrapper[2578]: I0707 10:27:11.652794 2578 flags.go:64] FLAG: --event-burst="100" Jul 07 10:27:11.658639 ip-10-0-135-34 kubenswrapper[2578]: I0707 10:27:11.652796 2578 flags.go:64] FLAG: --event-qps="50" Jul 07 10:27:11.658639 ip-10-0-135-34 kubenswrapper[2578]: I0707 10:27:11.652799 2578 flags.go:64] FLAG: --event-storage-age-limit="default=0" Jul 07 10:27:11.658639 ip-10-0-135-34 kubenswrapper[2578]: I0707 10:27:11.652802 2578 flags.go:64] FLAG: --event-storage-event-limit="default=0" Jul 07 10:27:11.658639 ip-10-0-135-34 kubenswrapper[2578]: I0707 10:27:11.652805 2578 flags.go:64] FLAG: --eviction-hard="" Jul 07 10:27:11.658639 ip-10-0-135-34 kubenswrapper[2578]: I0707 10:27:11.652809 2578 flags.go:64] FLAG: --eviction-max-pod-grace-period="0" Jul 07 10:27:11.658639 ip-10-0-135-34 kubenswrapper[2578]: I0707 10:27:11.652811 2578 flags.go:64] FLAG: --eviction-minimum-reclaim="" Jul 07 10:27:11.658639 ip-10-0-135-34 kubenswrapper[2578]: I0707 10:27:11.652814 2578 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s" Jul 07 10:27:11.658639 ip-10-0-135-34 kubenswrapper[2578]: I0707 10:27:11.652817 2578 flags.go:64] FLAG: --eviction-soft="" Jul 07 10:27:11.658639 ip-10-0-135-34 kubenswrapper[2578]: I0707 10:27:11.652820 2578 flags.go:64] FLAG: --eviction-soft-grace-period="" Jul 07 10:27:11.659308 ip-10-0-135-34 kubenswrapper[2578]: I0707 10:27:11.652823 2578 flags.go:64] FLAG: --exit-on-lock-contention="false" Jul 07 10:27:11.659308 ip-10-0-135-34 kubenswrapper[2578]: I0707 10:27:11.652826 2578 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false" Jul 07 10:27:11.659308 ip-10-0-135-34 kubenswrapper[2578]: I0707 10:27:11.652829 2578 flags.go:64] FLAG: --experimental-mounter-path="" Jul 07 10:27:11.659308 ip-10-0-135-34 kubenswrapper[2578]: I0707 10:27:11.652832 2578 flags.go:64] FLAG: --fail-cgroupv1="false" Jul 07 10:27:11.659308 ip-10-0-135-34 kubenswrapper[2578]: I0707 10:27:11.652835 2578 flags.go:64] FLAG: --fail-swap-on="true" Jul 07 10:27:11.659308 ip-10-0-135-34 kubenswrapper[2578]: I0707 10:27:11.652837 2578 flags.go:64] FLAG: --feature-gates="" Jul 07 10:27:11.659308 ip-10-0-135-34 kubenswrapper[2578]: I0707 10:27:11.652841 2578 flags.go:64] FLAG: --file-check-frequency="20s" Jul 07 10:27:11.659308 ip-10-0-135-34 kubenswrapper[2578]: I0707 10:27:11.652844 2578 flags.go:64] FLAG: --global-housekeeping-interval="1m0s" Jul 07 10:27:11.659308 ip-10-0-135-34 kubenswrapper[2578]: I0707 10:27:11.652847 2578 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge" Jul 07 10:27:11.659308 ip-10-0-135-34 kubenswrapper[2578]: I0707 10:27:11.652849 2578 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1" Jul 07 10:27:11.659308 ip-10-0-135-34 kubenswrapper[2578]: I0707 10:27:11.652852 2578 flags.go:64] FLAG: --healthz-port="10248" Jul 07 10:27:11.659308 ip-10-0-135-34 kubenswrapper[2578]: I0707 10:27:11.652855 2578 flags.go:64] FLAG: --help="false" Jul 07 10:27:11.659308 ip-10-0-135-34 kubenswrapper[2578]: I0707 10:27:11.652858 2578 flags.go:64] FLAG: --hostname-override="ip-10-0-135-34.ec2.internal" Jul 07 10:27:11.659308 ip-10-0-135-34 kubenswrapper[2578]: I0707 10:27:11.652861 2578 flags.go:64] FLAG: --housekeeping-interval="10s" Jul 07 10:27:11.659308 ip-10-0-135-34 kubenswrapper[2578]: I0707 10:27:11.652864 2578 flags.go:64] FLAG: --http-check-frequency="20s" Jul 07 10:27:11.659308 ip-10-0-135-34 kubenswrapper[2578]: I0707 10:27:11.652867 2578 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins" Jul 07 10:27:11.659308 ip-10-0-135-34 kubenswrapper[2578]: I0707 10:27:11.652871 2578 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml" Jul 07 10:27:11.659308 ip-10-0-135-34 kubenswrapper[2578]: I0707 10:27:11.652874 2578 flags.go:64] FLAG: --image-gc-high-threshold="85" Jul 07 10:27:11.659308 ip-10-0-135-34 kubenswrapper[2578]: I0707 10:27:11.652877 2578 flags.go:64] FLAG: --image-gc-low-threshold="80" Jul 07 10:27:11.659308 ip-10-0-135-34 kubenswrapper[2578]: I0707 10:27:11.652880 2578 flags.go:64] FLAG: --image-service-endpoint="" Jul 07 10:27:11.659308 ip-10-0-135-34 kubenswrapper[2578]: I0707 10:27:11.652882 2578 flags.go:64] FLAG: --kernel-memcg-notification="false" Jul 07 10:27:11.659308 ip-10-0-135-34 kubenswrapper[2578]: I0707 10:27:11.652885 2578 flags.go:64] FLAG: --kube-api-burst="100" Jul 07 10:27:11.659308 ip-10-0-135-34 kubenswrapper[2578]: I0707 10:27:11.652888 2578 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf" Jul 07 10:27:11.659916 ip-10-0-135-34 kubenswrapper[2578]: I0707 10:27:11.652891 2578 flags.go:64] FLAG: --kube-api-qps="50" Jul 07 10:27:11.659916 ip-10-0-135-34 kubenswrapper[2578]: I0707 10:27:11.652893 2578 flags.go:64] FLAG: --kube-reserved="" Jul 07 10:27:11.659916 ip-10-0-135-34 kubenswrapper[2578]: I0707 10:27:11.652896 2578 flags.go:64] FLAG: --kube-reserved-cgroup="" Jul 07 10:27:11.659916 ip-10-0-135-34 kubenswrapper[2578]: I0707 10:27:11.652898 2578 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig" Jul 07 10:27:11.659916 ip-10-0-135-34 kubenswrapper[2578]: I0707 10:27:11.652901 2578 flags.go:64] FLAG: --kubelet-cgroups="" Jul 07 10:27:11.659916 ip-10-0-135-34 kubenswrapper[2578]: I0707 10:27:11.652904 2578 flags.go:64] FLAG: --local-storage-capacity-isolation="true" Jul 07 10:27:11.659916 ip-10-0-135-34 kubenswrapper[2578]: I0707 10:27:11.652907 2578 flags.go:64] FLAG: --lock-file="" Jul 07 10:27:11.659916 ip-10-0-135-34 kubenswrapper[2578]: I0707 10:27:11.652909 2578 flags.go:64] FLAG: --log-cadvisor-usage="false" Jul 07 10:27:11.659916 ip-10-0-135-34 kubenswrapper[2578]: I0707 10:27:11.652912 2578 flags.go:64] FLAG: --log-flush-frequency="5s" Jul 07 10:27:11.659916 ip-10-0-135-34 kubenswrapper[2578]: I0707 10:27:11.652915 2578 flags.go:64] FLAG: --log-json-info-buffer-size="0" Jul 07 10:27:11.659916 ip-10-0-135-34 kubenswrapper[2578]: I0707 10:27:11.652920 2578 flags.go:64] FLAG: --log-json-split-stream="false" Jul 07 10:27:11.659916 ip-10-0-135-34 kubenswrapper[2578]: I0707 10:27:11.652923 2578 flags.go:64] FLAG: --log-text-info-buffer-size="0" Jul 07 10:27:11.659916 ip-10-0-135-34 kubenswrapper[2578]: I0707 10:27:11.652925 2578 flags.go:64] FLAG: --log-text-split-stream="false" Jul 07 10:27:11.659916 ip-10-0-135-34 kubenswrapper[2578]: I0707 10:27:11.652928 2578 flags.go:64] FLAG: --logging-format="text" Jul 07 10:27:11.659916 ip-10-0-135-34 kubenswrapper[2578]: I0707 10:27:11.652931 2578 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id" Jul 07 10:27:11.659916 ip-10-0-135-34 kubenswrapper[2578]: I0707 10:27:11.652934 2578 flags.go:64] FLAG: --make-iptables-util-chains="true" Jul 07 10:27:11.659916 ip-10-0-135-34 kubenswrapper[2578]: I0707 10:27:11.652937 2578 flags.go:64] FLAG: --manifest-url="" Jul 07 10:27:11.659916 ip-10-0-135-34 kubenswrapper[2578]: I0707 10:27:11.652940 2578 flags.go:64] FLAG: --manifest-url-header="" Jul 07 10:27:11.659916 ip-10-0-135-34 kubenswrapper[2578]: I0707 10:27:11.652944 2578 flags.go:64] FLAG: --max-housekeeping-interval="15s" Jul 07 10:27:11.659916 ip-10-0-135-34 kubenswrapper[2578]: I0707 10:27:11.652947 2578 flags.go:64] FLAG: --max-open-files="1000000" Jul 07 10:27:11.659916 ip-10-0-135-34 kubenswrapper[2578]: I0707 10:27:11.652951 2578 flags.go:64] FLAG: --max-pods="110" Jul 07 10:27:11.659916 ip-10-0-135-34 kubenswrapper[2578]: I0707 10:27:11.652954 2578 flags.go:64] FLAG: --maximum-dead-containers="-1" Jul 07 10:27:11.659916 ip-10-0-135-34 kubenswrapper[2578]: I0707 10:27:11.652956 2578 flags.go:64] FLAG: --maximum-dead-containers-per-container="1" Jul 07 10:27:11.659916 ip-10-0-135-34 kubenswrapper[2578]: I0707 10:27:11.652959 2578 flags.go:64] FLAG: --memory-manager-policy="None" Jul 07 10:27:11.659916 ip-10-0-135-34 kubenswrapper[2578]: I0707 10:27:11.652962 2578 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s" Jul 07 10:27:11.660518 ip-10-0-135-34 kubenswrapper[2578]: I0707 10:27:11.652965 2578 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s" Jul 07 10:27:11.660518 ip-10-0-135-34 kubenswrapper[2578]: I0707 10:27:11.652968 2578 flags.go:64] FLAG: --node-ip="0.0.0.0" Jul 07 10:27:11.660518 ip-10-0-135-34 kubenswrapper[2578]: I0707 10:27:11.652971 2578 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel" Jul 07 10:27:11.660518 ip-10-0-135-34 kubenswrapper[2578]: I0707 10:27:11.652978 2578 flags.go:64] FLAG: --node-status-max-images="50" Jul 07 10:27:11.660518 ip-10-0-135-34 kubenswrapper[2578]: I0707 10:27:11.652981 2578 flags.go:64] FLAG: --node-status-update-frequency="10s" Jul 07 10:27:11.660518 ip-10-0-135-34 kubenswrapper[2578]: I0707 10:27:11.652984 2578 flags.go:64] FLAG: --oom-score-adj="-999" Jul 07 10:27:11.660518 ip-10-0-135-34 kubenswrapper[2578]: I0707 10:27:11.652987 2578 flags.go:64] FLAG: --pod-cidr="" Jul 07 10:27:11.660518 ip-10-0-135-34 kubenswrapper[2578]: I0707 10:27:11.652990 2578 flags.go:64] FLAG: --pod-infra-container-image="quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:ce93f6b4197b9c62a26a3131f8057ffe9c9379c5079a4f62e5919b75151394e5" Jul 07 10:27:11.660518 ip-10-0-135-34 kubenswrapper[2578]: I0707 10:27:11.652995 2578 flags.go:64] FLAG: --pod-manifest-path="" Jul 07 10:27:11.660518 ip-10-0-135-34 kubenswrapper[2578]: I0707 10:27:11.652998 2578 flags.go:64] FLAG: --pod-max-pids="-1" Jul 07 10:27:11.660518 ip-10-0-135-34 kubenswrapper[2578]: I0707 10:27:11.653001 2578 flags.go:64] FLAG: --pods-per-core="0" Jul 07 10:27:11.660518 ip-10-0-135-34 kubenswrapper[2578]: I0707 10:27:11.653003 2578 flags.go:64] FLAG: --port="10250" Jul 07 10:27:11.660518 ip-10-0-135-34 kubenswrapper[2578]: I0707 10:27:11.653006 2578 flags.go:64] FLAG: --protect-kernel-defaults="false" Jul 07 10:27:11.660518 ip-10-0-135-34 kubenswrapper[2578]: I0707 10:27:11.653009 2578 flags.go:64] FLAG: --provider-id="aws:///us-east-1a/i-05ccfbfb5362fb050" Jul 07 10:27:11.660518 ip-10-0-135-34 kubenswrapper[2578]: I0707 10:27:11.653012 2578 flags.go:64] FLAG: --qos-reserved="" Jul 07 10:27:11.660518 ip-10-0-135-34 kubenswrapper[2578]: I0707 10:27:11.653015 2578 flags.go:64] FLAG: --read-only-port="10255" Jul 07 10:27:11.660518 ip-10-0-135-34 kubenswrapper[2578]: I0707 10:27:11.653018 2578 flags.go:64] FLAG: --register-node="true" Jul 07 10:27:11.660518 ip-10-0-135-34 kubenswrapper[2578]: I0707 10:27:11.653021 2578 flags.go:64] FLAG: --register-with-taints="" Jul 07 10:27:11.660518 ip-10-0-135-34 kubenswrapper[2578]: I0707 10:27:11.653024 2578 flags.go:64] FLAG: --registry-burst="10" Jul 07 10:27:11.660518 ip-10-0-135-34 kubenswrapper[2578]: I0707 10:27:11.653027 2578 flags.go:64] FLAG: --registry-qps="5" Jul 07 10:27:11.660518 ip-10-0-135-34 kubenswrapper[2578]: I0707 10:27:11.653030 2578 flags.go:64] FLAG: --reserved-cpus="" Jul 07 10:27:11.660518 ip-10-0-135-34 kubenswrapper[2578]: I0707 10:27:11.653033 2578 flags.go:64] FLAG: --reserved-memory="" Jul 07 10:27:11.660518 ip-10-0-135-34 kubenswrapper[2578]: I0707 10:27:11.653036 2578 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf" Jul 07 10:27:11.660518 ip-10-0-135-34 kubenswrapper[2578]: I0707 10:27:11.653039 2578 flags.go:64] FLAG: --root-dir="/var/lib/kubelet" Jul 07 10:27:11.660518 ip-10-0-135-34 kubenswrapper[2578]: I0707 10:27:11.653042 2578 flags.go:64] FLAG: --rotate-certificates="false" Jul 07 10:27:11.661533 ip-10-0-135-34 kubenswrapper[2578]: I0707 10:27:11.653044 2578 flags.go:64] FLAG: --rotate-server-certificates="false" Jul 07 10:27:11.661533 ip-10-0-135-34 kubenswrapper[2578]: I0707 10:27:11.653047 2578 flags.go:64] FLAG: --runonce="false" Jul 07 10:27:11.661533 ip-10-0-135-34 kubenswrapper[2578]: I0707 10:27:11.653049 2578 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service" Jul 07 10:27:11.661533 ip-10-0-135-34 kubenswrapper[2578]: I0707 10:27:11.653052 2578 flags.go:64] FLAG: --runtime-request-timeout="2m0s" Jul 07 10:27:11.661533 ip-10-0-135-34 kubenswrapper[2578]: I0707 10:27:11.653058 2578 flags.go:64] FLAG: --seccomp-default="false" Jul 07 10:27:11.661533 ip-10-0-135-34 kubenswrapper[2578]: I0707 10:27:11.653061 2578 flags.go:64] FLAG: --serialize-image-pulls="true" Jul 07 10:27:11.661533 ip-10-0-135-34 kubenswrapper[2578]: I0707 10:27:11.653064 2578 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s" Jul 07 10:27:11.661533 ip-10-0-135-34 kubenswrapper[2578]: I0707 10:27:11.653067 2578 flags.go:64] FLAG: --storage-driver-db="cadvisor" Jul 07 10:27:11.661533 ip-10-0-135-34 kubenswrapper[2578]: I0707 10:27:11.653070 2578 flags.go:64] FLAG: --storage-driver-host="localhost:8086" Jul 07 10:27:11.661533 ip-10-0-135-34 kubenswrapper[2578]: I0707 10:27:11.653073 2578 flags.go:64] FLAG: --storage-driver-password="root" Jul 07 10:27:11.661533 ip-10-0-135-34 kubenswrapper[2578]: I0707 10:27:11.653076 2578 flags.go:64] FLAG: --storage-driver-secure="false" Jul 07 10:27:11.661533 ip-10-0-135-34 kubenswrapper[2578]: I0707 10:27:11.653079 2578 flags.go:64] FLAG: --storage-driver-table="stats" Jul 07 10:27:11.661533 ip-10-0-135-34 kubenswrapper[2578]: I0707 10:27:11.653082 2578 flags.go:64] FLAG: --storage-driver-user="root" Jul 07 10:27:11.661533 ip-10-0-135-34 kubenswrapper[2578]: I0707 10:27:11.653085 2578 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s" Jul 07 10:27:11.661533 ip-10-0-135-34 kubenswrapper[2578]: I0707 10:27:11.653088 2578 flags.go:64] FLAG: --sync-frequency="1m0s" Jul 07 10:27:11.661533 ip-10-0-135-34 kubenswrapper[2578]: I0707 10:27:11.653090 2578 flags.go:64] FLAG: --system-cgroups="" Jul 07 10:27:11.661533 ip-10-0-135-34 kubenswrapper[2578]: I0707 10:27:11.653093 2578 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi" Jul 07 10:27:11.661533 ip-10-0-135-34 kubenswrapper[2578]: I0707 10:27:11.653098 2578 flags.go:64] FLAG: --system-reserved-cgroup="" Jul 07 10:27:11.661533 ip-10-0-135-34 kubenswrapper[2578]: I0707 10:27:11.653101 2578 flags.go:64] FLAG: --tls-cert-file="" Jul 07 10:27:11.661533 ip-10-0-135-34 kubenswrapper[2578]: I0707 10:27:11.653104 2578 flags.go:64] FLAG: --tls-cipher-suites="[]" Jul 07 10:27:11.661533 ip-10-0-135-34 kubenswrapper[2578]: I0707 10:27:11.653108 2578 flags.go:64] FLAG: --tls-min-version="" Jul 07 10:27:11.661533 ip-10-0-135-34 kubenswrapper[2578]: I0707 10:27:11.653111 2578 flags.go:64] FLAG: --tls-private-key-file="" Jul 07 10:27:11.661533 ip-10-0-135-34 kubenswrapper[2578]: I0707 10:27:11.653114 2578 flags.go:64] FLAG: --topology-manager-policy="none" Jul 07 10:27:11.661533 ip-10-0-135-34 kubenswrapper[2578]: I0707 10:27:11.653116 2578 flags.go:64] FLAG: --topology-manager-policy-options="" Jul 07 10:27:11.661533 ip-10-0-135-34 kubenswrapper[2578]: I0707 10:27:11.653119 2578 flags.go:64] FLAG: --topology-manager-scope="container" Jul 07 10:27:11.662189 ip-10-0-135-34 kubenswrapper[2578]: I0707 10:27:11.653122 2578 flags.go:64] FLAG: --v="2" Jul 07 10:27:11.662189 ip-10-0-135-34 kubenswrapper[2578]: I0707 10:27:11.653126 2578 flags.go:64] FLAG: --version="false" Jul 07 10:27:11.662189 ip-10-0-135-34 kubenswrapper[2578]: I0707 10:27:11.653130 2578 flags.go:64] FLAG: --vmodule="" Jul 07 10:27:11.662189 ip-10-0-135-34 kubenswrapper[2578]: I0707 10:27:11.653134 2578 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec" Jul 07 10:27:11.662189 ip-10-0-135-34 kubenswrapper[2578]: I0707 10:27:11.653137 2578 flags.go:64] FLAG: --volume-stats-agg-period="1m0s" Jul 07 10:27:11.662189 ip-10-0-135-34 kubenswrapper[2578]: W0707 10:27:11.653256 2578 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 07 10:27:11.662189 ip-10-0-135-34 kubenswrapper[2578]: W0707 10:27:11.653260 2578 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 07 10:27:11.662189 ip-10-0-135-34 kubenswrapper[2578]: W0707 10:27:11.653263 2578 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 07 10:27:11.662189 ip-10-0-135-34 kubenswrapper[2578]: W0707 10:27:11.653266 2578 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 07 10:27:11.662189 ip-10-0-135-34 kubenswrapper[2578]: W0707 10:27:11.653269 2578 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 07 10:27:11.662189 ip-10-0-135-34 kubenswrapper[2578]: W0707 10:27:11.653272 2578 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 07 10:27:11.662189 ip-10-0-135-34 kubenswrapper[2578]: W0707 10:27:11.653280 2578 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 07 10:27:11.662189 ip-10-0-135-34 kubenswrapper[2578]: W0707 10:27:11.653283 2578 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 07 10:27:11.662189 ip-10-0-135-34 kubenswrapper[2578]: W0707 10:27:11.653286 2578 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 07 10:27:11.662189 ip-10-0-135-34 kubenswrapper[2578]: W0707 10:27:11.653288 2578 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 07 10:27:11.662189 ip-10-0-135-34 kubenswrapper[2578]: W0707 10:27:11.653292 2578 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 07 10:27:11.662189 ip-10-0-135-34 kubenswrapper[2578]: W0707 10:27:11.653296 2578 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 07 10:27:11.662189 ip-10-0-135-34 kubenswrapper[2578]: W0707 10:27:11.653299 2578 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 07 10:27:11.662189 ip-10-0-135-34 kubenswrapper[2578]: W0707 10:27:11.653302 2578 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 07 10:27:11.662189 ip-10-0-135-34 kubenswrapper[2578]: W0707 10:27:11.653305 2578 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 07 10:27:11.662189 ip-10-0-135-34 kubenswrapper[2578]: W0707 10:27:11.653308 2578 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 07 10:27:11.662750 ip-10-0-135-34 kubenswrapper[2578]: W0707 10:27:11.653311 2578 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 07 10:27:11.662750 ip-10-0-135-34 kubenswrapper[2578]: W0707 10:27:11.653313 2578 feature_gate.go:328] unrecognized feature gate: Example2 Jul 07 10:27:11.662750 ip-10-0-135-34 kubenswrapper[2578]: W0707 10:27:11.653316 2578 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 07 10:27:11.662750 ip-10-0-135-34 kubenswrapper[2578]: W0707 10:27:11.653319 2578 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 07 10:27:11.662750 ip-10-0-135-34 kubenswrapper[2578]: W0707 10:27:11.653323 2578 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 07 10:27:11.662750 ip-10-0-135-34 kubenswrapper[2578]: W0707 10:27:11.653326 2578 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 07 10:27:11.662750 ip-10-0-135-34 kubenswrapper[2578]: W0707 10:27:11.653328 2578 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 07 10:27:11.662750 ip-10-0-135-34 kubenswrapper[2578]: W0707 10:27:11.653331 2578 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 07 10:27:11.662750 ip-10-0-135-34 kubenswrapper[2578]: W0707 10:27:11.653334 2578 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 07 10:27:11.662750 ip-10-0-135-34 kubenswrapper[2578]: W0707 10:27:11.653336 2578 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 07 10:27:11.662750 ip-10-0-135-34 kubenswrapper[2578]: W0707 10:27:11.653338 2578 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 07 10:27:11.662750 ip-10-0-135-34 kubenswrapper[2578]: W0707 10:27:11.653341 2578 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 07 10:27:11.662750 ip-10-0-135-34 kubenswrapper[2578]: W0707 10:27:11.653343 2578 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 07 10:27:11.662750 ip-10-0-135-34 kubenswrapper[2578]: W0707 10:27:11.653347 2578 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 07 10:27:11.662750 ip-10-0-135-34 kubenswrapper[2578]: W0707 10:27:11.653350 2578 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 07 10:27:11.662750 ip-10-0-135-34 kubenswrapper[2578]: W0707 10:27:11.653353 2578 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 07 10:27:11.662750 ip-10-0-135-34 kubenswrapper[2578]: W0707 10:27:11.653356 2578 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 07 10:27:11.662750 ip-10-0-135-34 kubenswrapper[2578]: W0707 10:27:11.653359 2578 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 07 10:27:11.662750 ip-10-0-135-34 kubenswrapper[2578]: W0707 10:27:11.653362 2578 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 07 10:27:11.663249 ip-10-0-135-34 kubenswrapper[2578]: W0707 10:27:11.653364 2578 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 07 10:27:11.663249 ip-10-0-135-34 kubenswrapper[2578]: W0707 10:27:11.653367 2578 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 07 10:27:11.663249 ip-10-0-135-34 kubenswrapper[2578]: W0707 10:27:11.653369 2578 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 07 10:27:11.663249 ip-10-0-135-34 kubenswrapper[2578]: W0707 10:27:11.653374 2578 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 07 10:27:11.663249 ip-10-0-135-34 kubenswrapper[2578]: W0707 10:27:11.653377 2578 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 07 10:27:11.663249 ip-10-0-135-34 kubenswrapper[2578]: W0707 10:27:11.653380 2578 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 07 10:27:11.663249 ip-10-0-135-34 kubenswrapper[2578]: W0707 10:27:11.653382 2578 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 07 10:27:11.663249 ip-10-0-135-34 kubenswrapper[2578]: W0707 10:27:11.653385 2578 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 07 10:27:11.663249 ip-10-0-135-34 kubenswrapper[2578]: W0707 10:27:11.653387 2578 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 07 10:27:11.663249 ip-10-0-135-34 kubenswrapper[2578]: W0707 10:27:11.653390 2578 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 07 10:27:11.663249 ip-10-0-135-34 kubenswrapper[2578]: W0707 10:27:11.653392 2578 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 07 10:27:11.663249 ip-10-0-135-34 kubenswrapper[2578]: W0707 10:27:11.653396 2578 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 07 10:27:11.663249 ip-10-0-135-34 kubenswrapper[2578]: W0707 10:27:11.653399 2578 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 07 10:27:11.663249 ip-10-0-135-34 kubenswrapper[2578]: W0707 10:27:11.653401 2578 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 07 10:27:11.663249 ip-10-0-135-34 kubenswrapper[2578]: W0707 10:27:11.653404 2578 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 07 10:27:11.663249 ip-10-0-135-34 kubenswrapper[2578]: W0707 10:27:11.653406 2578 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 07 10:27:11.663249 ip-10-0-135-34 kubenswrapper[2578]: W0707 10:27:11.653408 2578 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 07 10:27:11.663249 ip-10-0-135-34 kubenswrapper[2578]: W0707 10:27:11.653411 2578 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 07 10:27:11.663249 ip-10-0-135-34 kubenswrapper[2578]: W0707 10:27:11.653413 2578 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 07 10:27:11.663249 ip-10-0-135-34 kubenswrapper[2578]: W0707 10:27:11.653416 2578 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 07 10:27:11.663737 ip-10-0-135-34 kubenswrapper[2578]: W0707 10:27:11.653418 2578 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 07 10:27:11.663737 ip-10-0-135-34 kubenswrapper[2578]: W0707 10:27:11.653421 2578 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 07 10:27:11.663737 ip-10-0-135-34 kubenswrapper[2578]: W0707 10:27:11.653423 2578 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 07 10:27:11.663737 ip-10-0-135-34 kubenswrapper[2578]: W0707 10:27:11.653425 2578 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 07 10:27:11.663737 ip-10-0-135-34 kubenswrapper[2578]: W0707 10:27:11.653428 2578 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 07 10:27:11.663737 ip-10-0-135-34 kubenswrapper[2578]: W0707 10:27:11.653430 2578 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 07 10:27:11.663737 ip-10-0-135-34 kubenswrapper[2578]: W0707 10:27:11.653432 2578 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 07 10:27:11.663737 ip-10-0-135-34 kubenswrapper[2578]: W0707 10:27:11.653435 2578 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 07 10:27:11.663737 ip-10-0-135-34 kubenswrapper[2578]: W0707 10:27:11.653438 2578 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 07 10:27:11.663737 ip-10-0-135-34 kubenswrapper[2578]: W0707 10:27:11.653440 2578 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 07 10:27:11.663737 ip-10-0-135-34 kubenswrapper[2578]: W0707 10:27:11.653443 2578 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 07 10:27:11.663737 ip-10-0-135-34 kubenswrapper[2578]: W0707 10:27:11.653445 2578 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 07 10:27:11.663737 ip-10-0-135-34 kubenswrapper[2578]: W0707 10:27:11.653447 2578 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 07 10:27:11.663737 ip-10-0-135-34 kubenswrapper[2578]: W0707 10:27:11.653450 2578 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 07 10:27:11.663737 ip-10-0-135-34 kubenswrapper[2578]: W0707 10:27:11.653452 2578 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 07 10:27:11.663737 ip-10-0-135-34 kubenswrapper[2578]: W0707 10:27:11.653457 2578 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 07 10:27:11.663737 ip-10-0-135-34 kubenswrapper[2578]: W0707 10:27:11.653459 2578 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 07 10:27:11.663737 ip-10-0-135-34 kubenswrapper[2578]: W0707 10:27:11.653462 2578 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 07 10:27:11.663737 ip-10-0-135-34 kubenswrapper[2578]: W0707 10:27:11.653464 2578 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 07 10:27:11.663737 ip-10-0-135-34 kubenswrapper[2578]: W0707 10:27:11.653467 2578 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 07 10:27:11.664253 ip-10-0-135-34 kubenswrapper[2578]: W0707 10:27:11.653469 2578 feature_gate.go:328] unrecognized feature gate: Example Jul 07 10:27:11.664253 ip-10-0-135-34 kubenswrapper[2578]: W0707 10:27:11.653472 2578 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 07 10:27:11.664253 ip-10-0-135-34 kubenswrapper[2578]: W0707 10:27:11.653474 2578 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 07 10:27:11.664253 ip-10-0-135-34 kubenswrapper[2578]: W0707 10:27:11.653476 2578 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 07 10:27:11.664253 ip-10-0-135-34 kubenswrapper[2578]: W0707 10:27:11.653479 2578 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 07 10:27:11.664253 ip-10-0-135-34 kubenswrapper[2578]: W0707 10:27:11.653482 2578 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 07 10:27:11.664253 ip-10-0-135-34 kubenswrapper[2578]: W0707 10:27:11.653485 2578 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 07 10:27:11.664253 ip-10-0-135-34 kubenswrapper[2578]: W0707 10:27:11.653488 2578 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 07 10:27:11.664253 ip-10-0-135-34 kubenswrapper[2578]: W0707 10:27:11.653490 2578 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 07 10:27:11.664253 ip-10-0-135-34 kubenswrapper[2578]: W0707 10:27:11.653493 2578 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 07 10:27:11.664253 ip-10-0-135-34 kubenswrapper[2578]: W0707 10:27:11.653495 2578 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 07 10:27:11.664253 ip-10-0-135-34 kubenswrapper[2578]: W0707 10:27:11.653498 2578 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 07 10:27:11.664253 ip-10-0-135-34 kubenswrapper[2578]: W0707 10:27:11.653500 2578 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 07 10:27:11.664253 ip-10-0-135-34 kubenswrapper[2578]: W0707 10:27:11.653502 2578 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 07 10:27:11.664253 ip-10-0-135-34 kubenswrapper[2578]: W0707 10:27:11.653505 2578 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 07 10:27:11.664253 ip-10-0-135-34 kubenswrapper[2578]: W0707 10:27:11.653507 2578 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 07 10:27:11.664253 ip-10-0-135-34 kubenswrapper[2578]: W0707 10:27:11.653510 2578 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 07 10:27:11.664253 ip-10-0-135-34 kubenswrapper[2578]: W0707 10:27:11.653512 2578 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 07 10:27:11.664253 ip-10-0-135-34 kubenswrapper[2578]: W0707 10:27:11.653514 2578 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 07 10:27:11.664253 ip-10-0-135-34 kubenswrapper[2578]: W0707 10:27:11.653517 2578 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 07 10:27:11.664741 ip-10-0-135-34 kubenswrapper[2578]: W0707 10:27:11.653519 2578 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 07 10:27:11.664741 ip-10-0-135-34 kubenswrapper[2578]: W0707 10:27:11.653522 2578 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 07 10:27:11.664741 ip-10-0-135-34 kubenswrapper[2578]: W0707 10:27:11.653524 2578 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 07 10:27:11.664741 ip-10-0-135-34 kubenswrapper[2578]: W0707 10:27:11.653526 2578 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 07 10:27:11.664741 ip-10-0-135-34 kubenswrapper[2578]: I0707 10:27:11.653532 2578 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]} Jul 07 10:27:11.664741 ip-10-0-135-34 kubenswrapper[2578]: I0707 10:27:11.662528 2578 server.go:529] "Kubelet version" kubeletVersion="v1.34.8" Jul 07 10:27:11.664741 ip-10-0-135-34 kubenswrapper[2578]: I0707 10:27:11.662546 2578 server.go:531] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK="" Jul 07 10:27:11.664741 ip-10-0-135-34 kubenswrapper[2578]: I0707 10:27:11.662566 2578 watchdog_linux.go:95] "Systemd watchdog is not enabled" Jul 07 10:27:11.664741 ip-10-0-135-34 kubenswrapper[2578]: I0707 10:27:11.662571 2578 watchdog_linux.go:137] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started." Jul 07 10:27:11.664741 ip-10-0-135-34 kubenswrapper[2578]: W0707 10:27:11.662632 2578 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 07 10:27:11.664741 ip-10-0-135-34 kubenswrapper[2578]: W0707 10:27:11.662637 2578 feature_gate.go:328] unrecognized feature gate: Example Jul 07 10:27:11.664741 ip-10-0-135-34 kubenswrapper[2578]: W0707 10:27:11.662641 2578 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 07 10:27:11.664741 ip-10-0-135-34 kubenswrapper[2578]: W0707 10:27:11.662643 2578 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 07 10:27:11.664741 ip-10-0-135-34 kubenswrapper[2578]: W0707 10:27:11.662646 2578 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 07 10:27:11.664741 ip-10-0-135-34 kubenswrapper[2578]: W0707 10:27:11.662649 2578 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 07 10:27:11.665200 ip-10-0-135-34 kubenswrapper[2578]: W0707 10:27:11.662651 2578 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 07 10:27:11.665200 ip-10-0-135-34 kubenswrapper[2578]: W0707 10:27:11.662654 2578 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 07 10:27:11.665200 ip-10-0-135-34 kubenswrapper[2578]: W0707 10:27:11.662657 2578 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 07 10:27:11.665200 ip-10-0-135-34 kubenswrapper[2578]: W0707 10:27:11.662659 2578 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 07 10:27:11.665200 ip-10-0-135-34 kubenswrapper[2578]: W0707 10:27:11.662662 2578 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 07 10:27:11.665200 ip-10-0-135-34 kubenswrapper[2578]: W0707 10:27:11.662664 2578 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 07 10:27:11.665200 ip-10-0-135-34 kubenswrapper[2578]: W0707 10:27:11.662667 2578 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 07 10:27:11.665200 ip-10-0-135-34 kubenswrapper[2578]: W0707 10:27:11.662670 2578 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 07 10:27:11.665200 ip-10-0-135-34 kubenswrapper[2578]: W0707 10:27:11.662672 2578 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 07 10:27:11.665200 ip-10-0-135-34 kubenswrapper[2578]: W0707 10:27:11.662674 2578 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 07 10:27:11.665200 ip-10-0-135-34 kubenswrapper[2578]: W0707 10:27:11.662677 2578 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 07 10:27:11.665200 ip-10-0-135-34 kubenswrapper[2578]: W0707 10:27:11.662681 2578 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 07 10:27:11.665200 ip-10-0-135-34 kubenswrapper[2578]: W0707 10:27:11.662685 2578 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 07 10:27:11.665200 ip-10-0-135-34 kubenswrapper[2578]: W0707 10:27:11.662688 2578 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 07 10:27:11.665200 ip-10-0-135-34 kubenswrapper[2578]: W0707 10:27:11.662690 2578 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 07 10:27:11.665200 ip-10-0-135-34 kubenswrapper[2578]: W0707 10:27:11.662693 2578 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 07 10:27:11.665200 ip-10-0-135-34 kubenswrapper[2578]: W0707 10:27:11.662696 2578 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 07 10:27:11.665200 ip-10-0-135-34 kubenswrapper[2578]: W0707 10:27:11.662699 2578 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 07 10:27:11.665200 ip-10-0-135-34 kubenswrapper[2578]: W0707 10:27:11.662702 2578 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 07 10:27:11.665200 ip-10-0-135-34 kubenswrapper[2578]: W0707 10:27:11.662705 2578 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 07 10:27:11.665680 ip-10-0-135-34 kubenswrapper[2578]: W0707 10:27:11.662707 2578 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 07 10:27:11.665680 ip-10-0-135-34 kubenswrapper[2578]: W0707 10:27:11.662711 2578 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 07 10:27:11.665680 ip-10-0-135-34 kubenswrapper[2578]: W0707 10:27:11.662713 2578 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 07 10:27:11.665680 ip-10-0-135-34 kubenswrapper[2578]: W0707 10:27:11.662717 2578 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 07 10:27:11.665680 ip-10-0-135-34 kubenswrapper[2578]: W0707 10:27:11.662720 2578 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 07 10:27:11.665680 ip-10-0-135-34 kubenswrapper[2578]: W0707 10:27:11.662722 2578 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 07 10:27:11.665680 ip-10-0-135-34 kubenswrapper[2578]: W0707 10:27:11.662724 2578 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 07 10:27:11.665680 ip-10-0-135-34 kubenswrapper[2578]: W0707 10:27:11.662727 2578 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 07 10:27:11.665680 ip-10-0-135-34 kubenswrapper[2578]: W0707 10:27:11.662730 2578 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 07 10:27:11.665680 ip-10-0-135-34 kubenswrapper[2578]: W0707 10:27:11.662732 2578 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 07 10:27:11.665680 ip-10-0-135-34 kubenswrapper[2578]: W0707 10:27:11.662735 2578 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 07 10:27:11.665680 ip-10-0-135-34 kubenswrapper[2578]: W0707 10:27:11.662737 2578 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 07 10:27:11.665680 ip-10-0-135-34 kubenswrapper[2578]: W0707 10:27:11.662740 2578 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 07 10:27:11.665680 ip-10-0-135-34 kubenswrapper[2578]: W0707 10:27:11.662742 2578 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 07 10:27:11.665680 ip-10-0-135-34 kubenswrapper[2578]: W0707 10:27:11.662745 2578 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 07 10:27:11.665680 ip-10-0-135-34 kubenswrapper[2578]: W0707 10:27:11.662749 2578 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 07 10:27:11.665680 ip-10-0-135-34 kubenswrapper[2578]: W0707 10:27:11.662752 2578 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 07 10:27:11.665680 ip-10-0-135-34 kubenswrapper[2578]: W0707 10:27:11.662756 2578 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 07 10:27:11.665680 ip-10-0-135-34 kubenswrapper[2578]: W0707 10:27:11.662759 2578 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 07 10:27:11.666165 ip-10-0-135-34 kubenswrapper[2578]: W0707 10:27:11.662762 2578 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 07 10:27:11.666165 ip-10-0-135-34 kubenswrapper[2578]: W0707 10:27:11.662765 2578 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 07 10:27:11.666165 ip-10-0-135-34 kubenswrapper[2578]: W0707 10:27:11.662786 2578 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 07 10:27:11.666165 ip-10-0-135-34 kubenswrapper[2578]: W0707 10:27:11.662789 2578 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 07 10:27:11.666165 ip-10-0-135-34 kubenswrapper[2578]: W0707 10:27:11.662792 2578 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 07 10:27:11.666165 ip-10-0-135-34 kubenswrapper[2578]: W0707 10:27:11.662794 2578 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 07 10:27:11.666165 ip-10-0-135-34 kubenswrapper[2578]: W0707 10:27:11.662797 2578 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 07 10:27:11.666165 ip-10-0-135-34 kubenswrapper[2578]: W0707 10:27:11.662800 2578 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 07 10:27:11.666165 ip-10-0-135-34 kubenswrapper[2578]: W0707 10:27:11.662802 2578 feature_gate.go:328] unrecognized feature gate: Example2 Jul 07 10:27:11.666165 ip-10-0-135-34 kubenswrapper[2578]: W0707 10:27:11.662805 2578 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 07 10:27:11.666165 ip-10-0-135-34 kubenswrapper[2578]: W0707 10:27:11.662807 2578 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 07 10:27:11.666165 ip-10-0-135-34 kubenswrapper[2578]: W0707 10:27:11.662810 2578 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 07 10:27:11.666165 ip-10-0-135-34 kubenswrapper[2578]: W0707 10:27:11.662812 2578 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 07 10:27:11.666165 ip-10-0-135-34 kubenswrapper[2578]: W0707 10:27:11.662814 2578 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 07 10:27:11.666165 ip-10-0-135-34 kubenswrapper[2578]: W0707 10:27:11.662817 2578 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 07 10:27:11.666165 ip-10-0-135-34 kubenswrapper[2578]: W0707 10:27:11.662821 2578 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 07 10:27:11.666165 ip-10-0-135-34 kubenswrapper[2578]: W0707 10:27:11.662826 2578 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 07 10:27:11.666165 ip-10-0-135-34 kubenswrapper[2578]: W0707 10:27:11.662829 2578 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 07 10:27:11.666165 ip-10-0-135-34 kubenswrapper[2578]: W0707 10:27:11.662832 2578 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 07 10:27:11.666619 ip-10-0-135-34 kubenswrapper[2578]: W0707 10:27:11.662835 2578 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 07 10:27:11.666619 ip-10-0-135-34 kubenswrapper[2578]: W0707 10:27:11.662837 2578 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 07 10:27:11.666619 ip-10-0-135-34 kubenswrapper[2578]: W0707 10:27:11.662840 2578 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 07 10:27:11.666619 ip-10-0-135-34 kubenswrapper[2578]: W0707 10:27:11.662842 2578 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 07 10:27:11.666619 ip-10-0-135-34 kubenswrapper[2578]: W0707 10:27:11.662845 2578 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 07 10:27:11.666619 ip-10-0-135-34 kubenswrapper[2578]: W0707 10:27:11.662847 2578 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 07 10:27:11.666619 ip-10-0-135-34 kubenswrapper[2578]: W0707 10:27:11.662850 2578 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 07 10:27:11.666619 ip-10-0-135-34 kubenswrapper[2578]: W0707 10:27:11.662852 2578 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 07 10:27:11.666619 ip-10-0-135-34 kubenswrapper[2578]: W0707 10:27:11.662855 2578 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 07 10:27:11.666619 ip-10-0-135-34 kubenswrapper[2578]: W0707 10:27:11.662857 2578 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 07 10:27:11.666619 ip-10-0-135-34 kubenswrapper[2578]: W0707 10:27:11.662860 2578 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 07 10:27:11.666619 ip-10-0-135-34 kubenswrapper[2578]: W0707 10:27:11.662862 2578 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 07 10:27:11.666619 ip-10-0-135-34 kubenswrapper[2578]: W0707 10:27:11.662865 2578 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 07 10:27:11.666619 ip-10-0-135-34 kubenswrapper[2578]: W0707 10:27:11.662867 2578 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 07 10:27:11.666619 ip-10-0-135-34 kubenswrapper[2578]: W0707 10:27:11.662870 2578 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 07 10:27:11.666619 ip-10-0-135-34 kubenswrapper[2578]: W0707 10:27:11.662873 2578 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 07 10:27:11.666619 ip-10-0-135-34 kubenswrapper[2578]: W0707 10:27:11.662876 2578 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 07 10:27:11.666619 ip-10-0-135-34 kubenswrapper[2578]: W0707 10:27:11.662878 2578 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 07 10:27:11.666619 ip-10-0-135-34 kubenswrapper[2578]: W0707 10:27:11.662881 2578 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 07 10:27:11.666619 ip-10-0-135-34 kubenswrapper[2578]: W0707 10:27:11.662884 2578 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 07 10:27:11.667126 ip-10-0-135-34 kubenswrapper[2578]: W0707 10:27:11.662886 2578 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 07 10:27:11.667126 ip-10-0-135-34 kubenswrapper[2578]: W0707 10:27:11.662888 2578 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 07 10:27:11.667126 ip-10-0-135-34 kubenswrapper[2578]: W0707 10:27:11.662891 2578 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 07 10:27:11.667126 ip-10-0-135-34 kubenswrapper[2578]: W0707 10:27:11.662893 2578 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 07 10:27:11.667126 ip-10-0-135-34 kubenswrapper[2578]: W0707 10:27:11.662896 2578 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 07 10:27:11.667126 ip-10-0-135-34 kubenswrapper[2578]: W0707 10:27:11.662898 2578 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 07 10:27:11.667126 ip-10-0-135-34 kubenswrapper[2578]: W0707 10:27:11.662900 2578 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 07 10:27:11.667126 ip-10-0-135-34 kubenswrapper[2578]: W0707 10:27:11.662903 2578 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 07 10:27:11.667126 ip-10-0-135-34 kubenswrapper[2578]: W0707 10:27:11.662906 2578 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 07 10:27:11.667126 ip-10-0-135-34 kubenswrapper[2578]: W0707 10:27:11.662909 2578 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 07 10:27:11.667126 ip-10-0-135-34 kubenswrapper[2578]: W0707 10:27:11.662912 2578 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 07 10:27:11.667126 ip-10-0-135-34 kubenswrapper[2578]: W0707 10:27:11.662915 2578 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 07 10:27:11.667126 ip-10-0-135-34 kubenswrapper[2578]: W0707 10:27:11.662917 2578 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 07 10:27:11.667126 ip-10-0-135-34 kubenswrapper[2578]: W0707 10:27:11.662920 2578 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 07 10:27:11.667126 ip-10-0-135-34 kubenswrapper[2578]: W0707 10:27:11.662923 2578 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 07 10:27:11.667500 ip-10-0-135-34 kubenswrapper[2578]: I0707 10:27:11.662928 2578 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]} Jul 07 10:27:11.667500 ip-10-0-135-34 kubenswrapper[2578]: W0707 10:27:11.663048 2578 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 07 10:27:11.667500 ip-10-0-135-34 kubenswrapper[2578]: W0707 10:27:11.663054 2578 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 07 10:27:11.667500 ip-10-0-135-34 kubenswrapper[2578]: W0707 10:27:11.663057 2578 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 07 10:27:11.667500 ip-10-0-135-34 kubenswrapper[2578]: W0707 10:27:11.663060 2578 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 07 10:27:11.667500 ip-10-0-135-34 kubenswrapper[2578]: W0707 10:27:11.663063 2578 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 07 10:27:11.667500 ip-10-0-135-34 kubenswrapper[2578]: W0707 10:27:11.663066 2578 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 07 10:27:11.667500 ip-10-0-135-34 kubenswrapper[2578]: W0707 10:27:11.663068 2578 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 07 10:27:11.667500 ip-10-0-135-34 kubenswrapper[2578]: W0707 10:27:11.663071 2578 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 07 10:27:11.667500 ip-10-0-135-34 kubenswrapper[2578]: W0707 10:27:11.663074 2578 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 07 10:27:11.667500 ip-10-0-135-34 kubenswrapper[2578]: W0707 10:27:11.663077 2578 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 07 10:27:11.667500 ip-10-0-135-34 kubenswrapper[2578]: W0707 10:27:11.663079 2578 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 07 10:27:11.667500 ip-10-0-135-34 kubenswrapper[2578]: W0707 10:27:11.663082 2578 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 07 10:27:11.667500 ip-10-0-135-34 kubenswrapper[2578]: W0707 10:27:11.663085 2578 feature_gate.go:328] unrecognized feature gate: Example2 Jul 07 10:27:11.667500 ip-10-0-135-34 kubenswrapper[2578]: W0707 10:27:11.663088 2578 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 07 10:27:11.667895 ip-10-0-135-34 kubenswrapper[2578]: W0707 10:27:11.663090 2578 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 07 10:27:11.667895 ip-10-0-135-34 kubenswrapper[2578]: W0707 10:27:11.663093 2578 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 07 10:27:11.667895 ip-10-0-135-34 kubenswrapper[2578]: W0707 10:27:11.663095 2578 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 07 10:27:11.667895 ip-10-0-135-34 kubenswrapper[2578]: W0707 10:27:11.663098 2578 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 07 10:27:11.667895 ip-10-0-135-34 kubenswrapper[2578]: W0707 10:27:11.663100 2578 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 07 10:27:11.667895 ip-10-0-135-34 kubenswrapper[2578]: W0707 10:27:11.663102 2578 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 07 10:27:11.667895 ip-10-0-135-34 kubenswrapper[2578]: W0707 10:27:11.663105 2578 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 07 10:27:11.667895 ip-10-0-135-34 kubenswrapper[2578]: W0707 10:27:11.663107 2578 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 07 10:27:11.667895 ip-10-0-135-34 kubenswrapper[2578]: W0707 10:27:11.663110 2578 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 07 10:27:11.667895 ip-10-0-135-34 kubenswrapper[2578]: W0707 10:27:11.663112 2578 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 07 10:27:11.667895 ip-10-0-135-34 kubenswrapper[2578]: W0707 10:27:11.663115 2578 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 07 10:27:11.667895 ip-10-0-135-34 kubenswrapper[2578]: W0707 10:27:11.663117 2578 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 07 10:27:11.667895 ip-10-0-135-34 kubenswrapper[2578]: W0707 10:27:11.663120 2578 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 07 10:27:11.667895 ip-10-0-135-34 kubenswrapper[2578]: W0707 10:27:11.663123 2578 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 07 10:27:11.667895 ip-10-0-135-34 kubenswrapper[2578]: W0707 10:27:11.663126 2578 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 07 10:27:11.667895 ip-10-0-135-34 kubenswrapper[2578]: W0707 10:27:11.663131 2578 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 07 10:27:11.667895 ip-10-0-135-34 kubenswrapper[2578]: W0707 10:27:11.663134 2578 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 07 10:27:11.667895 ip-10-0-135-34 kubenswrapper[2578]: W0707 10:27:11.663138 2578 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 07 10:27:11.667895 ip-10-0-135-34 kubenswrapper[2578]: W0707 10:27:11.663141 2578 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 07 10:27:11.668354 ip-10-0-135-34 kubenswrapper[2578]: W0707 10:27:11.663144 2578 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 07 10:27:11.668354 ip-10-0-135-34 kubenswrapper[2578]: W0707 10:27:11.663146 2578 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 07 10:27:11.668354 ip-10-0-135-34 kubenswrapper[2578]: W0707 10:27:11.663149 2578 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 07 10:27:11.668354 ip-10-0-135-34 kubenswrapper[2578]: W0707 10:27:11.663151 2578 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 07 10:27:11.668354 ip-10-0-135-34 kubenswrapper[2578]: W0707 10:27:11.663154 2578 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 07 10:27:11.668354 ip-10-0-135-34 kubenswrapper[2578]: W0707 10:27:11.663157 2578 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 07 10:27:11.668354 ip-10-0-135-34 kubenswrapper[2578]: W0707 10:27:11.663160 2578 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 07 10:27:11.668354 ip-10-0-135-34 kubenswrapper[2578]: W0707 10:27:11.663162 2578 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 07 10:27:11.668354 ip-10-0-135-34 kubenswrapper[2578]: W0707 10:27:11.663165 2578 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 07 10:27:11.668354 ip-10-0-135-34 kubenswrapper[2578]: W0707 10:27:11.663168 2578 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 07 10:27:11.668354 ip-10-0-135-34 kubenswrapper[2578]: W0707 10:27:11.663170 2578 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 07 10:27:11.668354 ip-10-0-135-34 kubenswrapper[2578]: W0707 10:27:11.663173 2578 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 07 10:27:11.668354 ip-10-0-135-34 kubenswrapper[2578]: W0707 10:27:11.663176 2578 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 07 10:27:11.668354 ip-10-0-135-34 kubenswrapper[2578]: W0707 10:27:11.663178 2578 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 07 10:27:11.668354 ip-10-0-135-34 kubenswrapper[2578]: W0707 10:27:11.663181 2578 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 07 10:27:11.668354 ip-10-0-135-34 kubenswrapper[2578]: W0707 10:27:11.663183 2578 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 07 10:27:11.668354 ip-10-0-135-34 kubenswrapper[2578]: W0707 10:27:11.663186 2578 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 07 10:27:11.668354 ip-10-0-135-34 kubenswrapper[2578]: W0707 10:27:11.663188 2578 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 07 10:27:11.668354 ip-10-0-135-34 kubenswrapper[2578]: W0707 10:27:11.663191 2578 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 07 10:27:11.668354 ip-10-0-135-34 kubenswrapper[2578]: W0707 10:27:11.663193 2578 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 07 10:27:11.668848 ip-10-0-135-34 kubenswrapper[2578]: W0707 10:27:11.663195 2578 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 07 10:27:11.668848 ip-10-0-135-34 kubenswrapper[2578]: W0707 10:27:11.663197 2578 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 07 10:27:11.668848 ip-10-0-135-34 kubenswrapper[2578]: W0707 10:27:11.663200 2578 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 07 10:27:11.668848 ip-10-0-135-34 kubenswrapper[2578]: W0707 10:27:11.663202 2578 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 07 10:27:11.668848 ip-10-0-135-34 kubenswrapper[2578]: W0707 10:27:11.663205 2578 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 07 10:27:11.668848 ip-10-0-135-34 kubenswrapper[2578]: W0707 10:27:11.663207 2578 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 07 10:27:11.668848 ip-10-0-135-34 kubenswrapper[2578]: W0707 10:27:11.663210 2578 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 07 10:27:11.668848 ip-10-0-135-34 kubenswrapper[2578]: W0707 10:27:11.663213 2578 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 07 10:27:11.668848 ip-10-0-135-34 kubenswrapper[2578]: W0707 10:27:11.663215 2578 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 07 10:27:11.668848 ip-10-0-135-34 kubenswrapper[2578]: W0707 10:27:11.663217 2578 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 07 10:27:11.668848 ip-10-0-135-34 kubenswrapper[2578]: W0707 10:27:11.663220 2578 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 07 10:27:11.668848 ip-10-0-135-34 kubenswrapper[2578]: W0707 10:27:11.663223 2578 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 07 10:27:11.668848 ip-10-0-135-34 kubenswrapper[2578]: W0707 10:27:11.663225 2578 feature_gate.go:328] unrecognized feature gate: Example Jul 07 10:27:11.668848 ip-10-0-135-34 kubenswrapper[2578]: W0707 10:27:11.663227 2578 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 07 10:27:11.668848 ip-10-0-135-34 kubenswrapper[2578]: W0707 10:27:11.663231 2578 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 07 10:27:11.668848 ip-10-0-135-34 kubenswrapper[2578]: W0707 10:27:11.663234 2578 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 07 10:27:11.668848 ip-10-0-135-34 kubenswrapper[2578]: W0707 10:27:11.663237 2578 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 07 10:27:11.668848 ip-10-0-135-34 kubenswrapper[2578]: W0707 10:27:11.663239 2578 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 07 10:27:11.668848 ip-10-0-135-34 kubenswrapper[2578]: W0707 10:27:11.663241 2578 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 07 10:27:11.668848 ip-10-0-135-34 kubenswrapper[2578]: W0707 10:27:11.663244 2578 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 07 10:27:11.669322 ip-10-0-135-34 kubenswrapper[2578]: W0707 10:27:11.663246 2578 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 07 10:27:11.669322 ip-10-0-135-34 kubenswrapper[2578]: W0707 10:27:11.663249 2578 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 07 10:27:11.669322 ip-10-0-135-34 kubenswrapper[2578]: W0707 10:27:11.663251 2578 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 07 10:27:11.669322 ip-10-0-135-34 kubenswrapper[2578]: W0707 10:27:11.663254 2578 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 07 10:27:11.669322 ip-10-0-135-34 kubenswrapper[2578]: W0707 10:27:11.663256 2578 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 07 10:27:11.669322 ip-10-0-135-34 kubenswrapper[2578]: W0707 10:27:11.663259 2578 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 07 10:27:11.669322 ip-10-0-135-34 kubenswrapper[2578]: W0707 10:27:11.663262 2578 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 07 10:27:11.669322 ip-10-0-135-34 kubenswrapper[2578]: W0707 10:27:11.663264 2578 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 07 10:27:11.669322 ip-10-0-135-34 kubenswrapper[2578]: W0707 10:27:11.663267 2578 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 07 10:27:11.669322 ip-10-0-135-34 kubenswrapper[2578]: W0707 10:27:11.663269 2578 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 07 10:27:11.669322 ip-10-0-135-34 kubenswrapper[2578]: W0707 10:27:11.663272 2578 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 07 10:27:11.669322 ip-10-0-135-34 kubenswrapper[2578]: W0707 10:27:11.663274 2578 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 07 10:27:11.669322 ip-10-0-135-34 kubenswrapper[2578]: W0707 10:27:11.663276 2578 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 07 10:27:11.669322 ip-10-0-135-34 kubenswrapper[2578]: W0707 10:27:11.663279 2578 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 07 10:27:11.669322 ip-10-0-135-34 kubenswrapper[2578]: W0707 10:27:11.663281 2578 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 07 10:27:11.669322 ip-10-0-135-34 kubenswrapper[2578]: W0707 10:27:11.663283 2578 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 07 10:27:11.669322 ip-10-0-135-34 kubenswrapper[2578]: W0707 10:27:11.663286 2578 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 07 10:27:11.669322 ip-10-0-135-34 kubenswrapper[2578]: W0707 10:27:11.663289 2578 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 07 10:27:11.669322 ip-10-0-135-34 kubenswrapper[2578]: W0707 10:27:11.663291 2578 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 07 10:27:11.669322 ip-10-0-135-34 kubenswrapper[2578]: W0707 10:27:11.663294 2578 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 07 10:27:11.669825 ip-10-0-135-34 kubenswrapper[2578]: W0707 10:27:11.663296 2578 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 07 10:27:11.669825 ip-10-0-135-34 kubenswrapper[2578]: W0707 10:27:11.663298 2578 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 07 10:27:11.669825 ip-10-0-135-34 kubenswrapper[2578]: W0707 10:27:11.663301 2578 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 07 10:27:11.669825 ip-10-0-135-34 kubenswrapper[2578]: W0707 10:27:11.663303 2578 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 07 10:27:11.669825 ip-10-0-135-34 kubenswrapper[2578]: W0707 10:27:11.663306 2578 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 07 10:27:11.669825 ip-10-0-135-34 kubenswrapper[2578]: W0707 10:27:11.663308 2578 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 07 10:27:11.669825 ip-10-0-135-34 kubenswrapper[2578]: I0707 10:27:11.663313 2578 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]} Jul 07 10:27:11.669825 ip-10-0-135-34 kubenswrapper[2578]: I0707 10:27:11.663439 2578 server.go:962] "Client rotation is on, will bootstrap in background" Jul 07 10:27:11.669825 ip-10-0-135-34 kubenswrapper[2578]: I0707 10:27:11.667576 2578 bootstrap.go:101] "Use the bootstrap credentials to request a cert, and set kubeconfig to point to the certificate dir" Jul 07 10:27:11.669825 ip-10-0-135-34 kubenswrapper[2578]: I0707 10:27:11.669127 2578 server.go:1019] "Starting client certificate rotation" Jul 07 10:27:11.672834 ip-10-0-135-34 kubenswrapper[2578]: I0707 10:27:11.672819 2578 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Jul 07 10:27:11.673852 ip-10-0-135-34 kubenswrapper[2578]: I0707 10:27:11.673831 2578 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet" Jul 07 10:27:11.673906 ip-10-0-135-34 kubenswrapper[2578]: I0707 10:27:11.673879 2578 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kube-apiserver-client-kubelet" Jul 07 10:27:11.677642 ip-10-0-135-34 kubenswrapper[2578]: I0707 10:27:11.677621 2578 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Jul 07 10:27:11.688281 ip-10-0-135-34 kubenswrapper[2578]: I0707 10:27:11.688259 2578 log.go:25] "Validated CRI v1 runtime API" Jul 07 10:27:11.690566 ip-10-0-135-34 kubenswrapper[2578]: I0707 10:27:11.690550 2578 log.go:25] "Validated CRI v1 image API" Jul 07 10:27:11.693695 ip-10-0-135-34 kubenswrapper[2578]: I0707 10:27:11.693677 2578 server.go:1429] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd" Jul 07 10:27:11.698839 ip-10-0-135-34 kubenswrapper[2578]: I0707 10:27:11.698816 2578 fs.go:135] Filesystem UUIDs: map[68daa57f-630f-45c9-ac52-5a1c67ea3592:/dev/nvme0n1p4 7B77-95E7:/dev/nvme0n1p2 f19687ef-7938-400d-a0d2-743991b46b89:/dev/nvme0n1p3] Jul 07 10:27:11.698913 ip-10-0-135-34 kubenswrapper[2578]: I0707 10:27:11.698836 2578 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:22 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:24 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:31 fsType:tmpfs blockSize:0} composefs_0-33:{mountpoint:/ major:0 minor:33 fsType:overlay blockSize:0}] Jul 07 10:27:11.704996 ip-10-0-135-34 kubenswrapper[2578]: I0707 10:27:11.704864 2578 manager.go:216] Machine: {Timestamp:2026-07-07 10:27:11.702605364 +0000 UTC m=+0.374564899 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3102183 MemoryCapacity:32812179456 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec28a24ff5393f63e5d0e83938523915 SystemUUID:ec28a24f-f539-3f63-e5d0-e83938523915 BootID:d14720fe-859b-4f01-87f9-1c2f9f0860be Filesystems:[{Device:/dev/shm DeviceMajor:0 DeviceMinor:22 Capacity:16406089728 Type:vfs Inodes:4005393 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:24 Capacity:6562439168 Type:vfs Inodes:819200 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:composefs_0-33 DeviceMajor:0 DeviceMinor:33 Capacity:6107136 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:31 Capacity:16406089728 Type:vfs Inodes:1048576 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:br-ex MacAddress:02:59:59:ac:9b:7d Speed:0 Mtu:9001} {Name:ens5 MacAddress:02:59:59:ac:9b:7d Speed:0 Mtu:9001} {Name:ovs-system MacAddress:de:bd:83:81:bb:52 Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:32812179456 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:34603008 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None} Jul 07 10:27:11.704996 ip-10-0-135-34 kubenswrapper[2578]: I0707 10:27:11.704993 2578 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available. Jul 07 10:27:11.705112 ip-10-0-135-34 kubenswrapper[2578]: I0707 10:27:11.705086 2578 manager.go:232] Version: {KernelVersion:5.14.0-570.125.1.el9_6.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.6.20260630-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:} Jul 07 10:27:11.707498 ip-10-0-135-34 kubenswrapper[2578]: I0707 10:27:11.707473 2578 container_manager_linux.go:273] "Container manager verified user specified cgroup-root exists" cgroupRoot=[] Jul 07 10:27:11.707637 ip-10-0-135-34 kubenswrapper[2578]: I0707 10:27:11.707505 2578 container_manager_linux.go:278] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-135-34.ec2.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2} Jul 07 10:27:11.708197 ip-10-0-135-34 kubenswrapper[2578]: I0707 10:27:11.708187 2578 topology_manager.go:138] "Creating topology manager with none policy" Jul 07 10:27:11.708237 ip-10-0-135-34 kubenswrapper[2578]: I0707 10:27:11.708198 2578 container_manager_linux.go:309] "Creating device plugin manager" Jul 07 10:27:11.708237 ip-10-0-135-34 kubenswrapper[2578]: I0707 10:27:11.708206 2578 manager.go:141] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock" Jul 07 10:27:11.708922 ip-10-0-135-34 kubenswrapper[2578]: I0707 10:27:11.708910 2578 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock" Jul 07 10:27:11.709088 ip-10-0-135-34 kubenswrapper[2578]: I0707 10:27:11.709073 2578 reflector.go:439] "Caches populated" logger="kubernetes.io/kube-apiserver-client-kubelet" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" Jul 07 10:27:11.710281 ip-10-0-135-34 kubenswrapper[2578]: I0707 10:27:11.710269 2578 container_manager_linux.go:318] "Creating Dynamic Resource Allocation (DRA) manager" Jul 07 10:27:11.713714 ip-10-0-135-34 kubenswrapper[2578]: I0707 10:27:11.713701 2578 state_mem.go:36] "Initialized new in-memory state store" Jul 07 10:27:11.714043 ip-10-0-135-34 kubenswrapper[2578]: I0707 10:27:11.714032 2578 server.go:1240] "Using root directory" path="/var/lib/kubelet" Jul 07 10:27:11.714074 ip-10-0-135-34 kubenswrapper[2578]: I0707 10:27:11.714067 2578 kubelet.go:486] "Attempting to sync node with API server" Jul 07 10:27:11.714108 ip-10-0-135-34 kubenswrapper[2578]: I0707 10:27:11.714078 2578 kubelet.go:387] "Adding static pod path" path="/etc/kubernetes/manifests" Jul 07 10:27:11.714108 ip-10-0-135-34 kubenswrapper[2578]: I0707 10:27:11.714090 2578 file.go:69] "Watching path" path="/etc/kubernetes/manifests" Jul 07 10:27:11.714108 ip-10-0-135-34 kubenswrapper[2578]: I0707 10:27:11.714100 2578 kubelet.go:398] "Adding apiserver pod source" Jul 07 10:27:11.714186 ip-10-0-135-34 kubenswrapper[2578]: I0707 10:27:11.714110 2578 apiserver.go:42] "Waiting for node sync before watching apiserver pods" Jul 07 10:27:11.714539 ip-10-0-135-34 kubenswrapper[2578]: I0707 10:27:11.714524 2578 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Jul 07 10:27:11.714614 ip-10-0-135-34 kubenswrapper[2578]: I0707 10:27:11.714542 2578 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Jul 07 10:27:11.719035 ip-10-0-135-34 kubenswrapper[2578]: I0707 10:27:11.719015 2578 kuberuntime_manager.go:301] "Container runtime initialized" containerRuntime="cri-o" version="1.34.8-3.rhaos4.21.git91b4b0b.el9" apiVersion="v1" Jul 07 10:27:11.721740 ip-10-0-135-34 kubenswrapper[2578]: I0707 10:27:11.721726 2578 kubelet.go:960] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled" Jul 07 10:27:11.721809 ip-10-0-135-34 kubenswrapper[2578]: I0707 10:27:11.721757 2578 kubelet.go:984] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled" Jul 07 10:27:11.723138 ip-10-0-135-34 kubenswrapper[2578]: I0707 10:27:11.723125 2578 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume" Jul 07 10:27:11.723188 ip-10-0-135-34 kubenswrapper[2578]: I0707 10:27:11.723140 2578 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir" Jul 07 10:27:11.723188 ip-10-0-135-34 kubenswrapper[2578]: I0707 10:27:11.723147 2578 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo" Jul 07 10:27:11.723188 ip-10-0-135-34 kubenswrapper[2578]: I0707 10:27:11.723153 2578 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path" Jul 07 10:27:11.723188 ip-10-0-135-34 kubenswrapper[2578]: I0707 10:27:11.723162 2578 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs" Jul 07 10:27:11.723188 ip-10-0-135-34 kubenswrapper[2578]: I0707 10:27:11.723167 2578 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret" Jul 07 10:27:11.723188 ip-10-0-135-34 kubenswrapper[2578]: I0707 10:27:11.723174 2578 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi" Jul 07 10:27:11.723188 ip-10-0-135-34 kubenswrapper[2578]: I0707 10:27:11.723179 2578 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api" Jul 07 10:27:11.723188 ip-10-0-135-34 kubenswrapper[2578]: I0707 10:27:11.723185 2578 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc" Jul 07 10:27:11.723188 ip-10-0-135-34 kubenswrapper[2578]: I0707 10:27:11.723191 2578 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap" Jul 07 10:27:11.723549 ip-10-0-135-34 kubenswrapper[2578]: I0707 10:27:11.723200 2578 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected" Jul 07 10:27:11.723549 ip-10-0-135-34 kubenswrapper[2578]: I0707 10:27:11.723205 2578 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume" Jul 07 10:27:11.723945 ip-10-0-135-34 kubenswrapper[2578]: I0707 10:27:11.723932 2578 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi" Jul 07 10:27:11.723945 ip-10-0-135-34 kubenswrapper[2578]: I0707 10:27:11.723944 2578 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image" Jul 07 10:27:11.725602 ip-10-0-135-34 kubenswrapper[2578]: E0707 10:27:11.725573 2578 reflector.go:205] "Failed to watch" err="failed to list *v1.Node: nodes \"ip-10-0-135-34.ec2.internal\" is forbidden: User \"system:anonymous\" cannot list resource \"nodes\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Node" Jul 07 10:27:11.725643 ip-10-0-135-34 kubenswrapper[2578]: E0707 10:27:11.725612 2578 reflector.go:205] "Failed to watch" err="failed to list *v1.Service: services is forbidden: User \"system:anonymous\" cannot list resource \"services\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Service" Jul 07 10:27:11.725704 ip-10-0-135-34 kubenswrapper[2578]: I0707 10:27:11.725679 2578 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-135-34.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope Jul 07 10:27:11.727523 ip-10-0-135-34 kubenswrapper[2578]: I0707 10:27:11.727510 2578 server.go:1268] "Started kubelet" Jul 07 10:27:11.727662 ip-10-0-135-34 kubenswrapper[2578]: I0707 10:27:11.727607 2578 server.go:180] "Starting to listen" address="0.0.0.0" port=10250 Jul 07 10:27:11.727662 ip-10-0-135-34 kubenswrapper[2578]: I0707 10:27:11.727618 2578 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10 Jul 07 10:27:11.727812 ip-10-0-135-34 kubenswrapper[2578]: I0707 10:27:11.727702 2578 server_v1.go:49] "podresources" method="list" useActivePods=true Jul 07 10:27:11.728433 ip-10-0-135-34 systemd[1]: Started Kubernetes Kubelet. Jul 07 10:27:11.729093 ip-10-0-135-34 kubenswrapper[2578]: I0707 10:27:11.729070 2578 server.go:249] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock" Jul 07 10:27:11.729830 ip-10-0-135-34 kubenswrapper[2578]: I0707 10:27:11.729816 2578 server.go:310] "Adding debug handlers to kubelet server" Jul 07 10:27:11.734181 ip-10-0-135-34 kubenswrapper[2578]: E0707 10:27:11.734159 2578 kubelet.go:1638] "Image garbage collection failed once. Stats initialization may not have completed yet" err="failed to get imageFs info: unable to find data in memory cache" Jul 07 10:27:11.736940 ip-10-0-135-34 kubenswrapper[2578]: I0707 10:27:11.736924 2578 fs_resource_analyzer.go:67] "Starting FS ResourceAnalyzer" Jul 07 10:27:11.737017 ip-10-0-135-34 kubenswrapper[2578]: I0707 10:27:11.736939 2578 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving" Jul 07 10:27:11.737072 ip-10-0-135-34 kubenswrapper[2578]: E0707 10:27:11.737024 2578 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-135-34.ec2.internal\" not found" Jul 07 10:27:11.737606 ip-10-0-135-34 kubenswrapper[2578]: I0707 10:27:11.737585 2578 desired_state_of_world_populator.go:146] "Desired state populator starts to run" Jul 07 10:27:11.737868 ip-10-0-135-34 kubenswrapper[2578]: I0707 10:27:11.737852 2578 factory.go:55] Registering systemd factory Jul 07 10:27:11.737868 ip-10-0-135-34 kubenswrapper[2578]: I0707 10:27:11.737870 2578 factory.go:223] Registration of the systemd container factory successfully Jul 07 10:27:11.737998 ip-10-0-135-34 kubenswrapper[2578]: I0707 10:27:11.737979 2578 volume_manager.go:311] "The desired_state_of_world populator starts" Jul 07 10:27:11.737998 ip-10-0-135-34 kubenswrapper[2578]: I0707 10:27:11.737989 2578 volume_manager.go:313] "Starting Kubelet Volume Manager" Jul 07 10:27:11.738103 ip-10-0-135-34 kubenswrapper[2578]: I0707 10:27:11.738090 2578 factory.go:153] Registering CRI-O factory Jul 07 10:27:11.738137 ip-10-0-135-34 kubenswrapper[2578]: I0707 10:27:11.738106 2578 factory.go:223] Registration of the crio container factory successfully Jul 07 10:27:11.738137 ip-10-0-135-34 kubenswrapper[2578]: I0707 10:27:11.738111 2578 reconstruct.go:97] "Volume reconstruction finished" Jul 07 10:27:11.738137 ip-10-0-135-34 kubenswrapper[2578]: I0707 10:27:11.738117 2578 reconciler.go:29] "Reconciler: start to sync state" Jul 07 10:27:11.738239 ip-10-0-135-34 kubenswrapper[2578]: I0707 10:27:11.738159 2578 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory Jul 07 10:27:11.738239 ip-10-0-135-34 kubenswrapper[2578]: I0707 10:27:11.738185 2578 factory.go:103] Registering Raw factory Jul 07 10:27:11.738239 ip-10-0-135-34 kubenswrapper[2578]: I0707 10:27:11.738203 2578 manager.go:1196] Started watching for new ooms in manager Jul 07 10:27:11.738604 ip-10-0-135-34 kubenswrapper[2578]: I0707 10:27:11.738592 2578 manager.go:349] Starting recovery of all containers Jul 07 10:27:11.743225 ip-10-0-135-34 kubenswrapper[2578]: E0707 10:27:11.743200 2578 reflector.go:205] "Failed to watch" err="failed to list *v1.CSIDriver: csidrivers.storage.k8s.io is forbidden: User \"system:anonymous\" cannot list resource \"csidrivers\" in API group \"storage.k8s.io\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.CSIDriver" Jul 07 10:27:11.743349 ip-10-0-135-34 kubenswrapper[2578]: E0707 10:27:11.743308 2578 controller.go:145] "Failed to ensure lease exists, will retry" err="leases.coordination.k8s.io \"ip-10-0-135-34.ec2.internal\" is forbidden: User \"system:anonymous\" cannot get resource \"leases\" in API group \"coordination.k8s.io\" in the namespace \"kube-node-lease\"" interval="200ms" Jul 07 10:27:11.746665 ip-10-0-135-34 kubenswrapper[2578]: E0707 10:27:11.743215 2578 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-135-34.ec2.internal.18bffb11e46ea9b8 default 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-135-34.ec2.internal,UID:ip-10-0-135-34.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:Starting,Message:Starting kubelet.,Source:EventSource{Component:kubelet,Host:ip-10-0-135-34.ec2.internal,},FirstTimestamp:2026-07-07 10:27:11.727479224 +0000 UTC m=+0.399438762,LastTimestamp:2026-07-07 10:27:11.727479224 +0000 UTC m=+0.399438762,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-135-34.ec2.internal,}" Jul 07 10:27:11.752381 ip-10-0-135-34 kubenswrapper[2578]: I0707 10:27:11.752360 2578 manager.go:354] Recovery completed Jul 07 10:27:11.756996 ip-10-0-135-34 kubenswrapper[2578]: I0707 10:27:11.756979 2578 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-skh5d" Jul 07 10:27:11.757653 ip-10-0-135-34 kubenswrapper[2578]: I0707 10:27:11.757641 2578 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 07 10:27:11.759743 ip-10-0-135-34 kubenswrapper[2578]: I0707 10:27:11.759728 2578 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-34.ec2.internal" event="NodeHasSufficientMemory" Jul 07 10:27:11.759858 ip-10-0-135-34 kubenswrapper[2578]: I0707 10:27:11.759758 2578 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-34.ec2.internal" event="NodeHasNoDiskPressure" Jul 07 10:27:11.759858 ip-10-0-135-34 kubenswrapper[2578]: I0707 10:27:11.759789 2578 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-34.ec2.internal" event="NodeHasSufficientPID" Jul 07 10:27:11.760272 ip-10-0-135-34 kubenswrapper[2578]: I0707 10:27:11.760259 2578 cpu_manager.go:222] "Starting CPU manager" policy="none" Jul 07 10:27:11.760272 ip-10-0-135-34 kubenswrapper[2578]: I0707 10:27:11.760271 2578 cpu_manager.go:223] "Reconciling" reconcilePeriod="10s" Jul 07 10:27:11.760355 ip-10-0-135-34 kubenswrapper[2578]: I0707 10:27:11.760286 2578 state_mem.go:36] "Initialized new in-memory state store" Jul 07 10:27:11.760355 ip-10-0-135-34 kubenswrapper[2578]: I0707 10:27:11.760320 2578 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 07 10:27:11.761021 ip-10-0-135-34 kubenswrapper[2578]: I0707 10:27:11.761009 2578 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-34.ec2.internal" event="NodeHasSufficientMemory" Jul 07 10:27:11.761069 ip-10-0-135-34 kubenswrapper[2578]: I0707 10:27:11.761034 2578 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-34.ec2.internal" event="NodeHasNoDiskPressure" Jul 07 10:27:11.761069 ip-10-0-135-34 kubenswrapper[2578]: I0707 10:27:11.761044 2578 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-34.ec2.internal" event="NodeHasSufficientPID" Jul 07 10:27:11.761497 ip-10-0-135-34 kubenswrapper[2578]: I0707 10:27:11.761484 2578 policy_none.go:49] "None policy: Start" Jul 07 10:27:11.761529 ip-10-0-135-34 kubenswrapper[2578]: I0707 10:27:11.761511 2578 memory_manager.go:187] "Starting memorymanager" policy="None" Jul 07 10:27:11.761558 ip-10-0-135-34 kubenswrapper[2578]: I0707 10:27:11.761527 2578 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint" Jul 07 10:27:11.761705 ip-10-0-135-34 kubenswrapper[2578]: E0707 10:27:11.761618 2578 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-135-34.ec2.internal.18bffb11e65afed2 default 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-135-34.ec2.internal,UID:ip-10-0-135-34.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeHasSufficientMemory,Message:Node ip-10-0-135-34.ec2.internal status is now: NodeHasSufficientMemory,Source:EventSource{Component:kubelet,Host:ip-10-0-135-34.ec2.internal,},FirstTimestamp:2026-07-07 10:27:11.759744722 +0000 UTC m=+0.431704260,LastTimestamp:2026-07-07 10:27:11.759744722 +0000 UTC m=+0.431704260,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-135-34.ec2.internal,}" Jul 07 10:27:11.762620 ip-10-0-135-34 kubenswrapper[2578]: I0707 10:27:11.762607 2578 policy_none.go:47] "Start" Jul 07 10:27:11.764799 ip-10-0-135-34 kubenswrapper[2578]: I0707 10:27:11.764781 2578 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-skh5d" Jul 07 10:27:11.792436 ip-10-0-135-34 kubenswrapper[2578]: I0707 10:27:11.792418 2578 manager.go:337] "Starting Device Plugin manager" Jul 07 10:27:11.792563 ip-10-0-135-34 kubenswrapper[2578]: E0707 10:27:11.792460 2578 manager.go:513] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint" Jul 07 10:27:11.792563 ip-10-0-135-34 kubenswrapper[2578]: I0707 10:27:11.792472 2578 server.go:87] "Starting device plugin registration server" Jul 07 10:27:11.792744 ip-10-0-135-34 kubenswrapper[2578]: I0707 10:27:11.792731 2578 eviction_manager.go:189] "Eviction manager: starting control loop" Jul 07 10:27:11.792847 ip-10-0-135-34 kubenswrapper[2578]: I0707 10:27:11.792748 2578 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s" Jul 07 10:27:11.792847 ip-10-0-135-34 kubenswrapper[2578]: I0707 10:27:11.792822 2578 plugin_watcher.go:51] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry" Jul 07 10:27:11.792939 ip-10-0-135-34 kubenswrapper[2578]: I0707 10:27:11.792923 2578 plugin_manager.go:116] "The desired_state_of_world populator (plugin watcher) starts" Jul 07 10:27:11.792939 ip-10-0-135-34 kubenswrapper[2578]: I0707 10:27:11.792935 2578 plugin_manager.go:118] "Starting Kubelet Plugin Manager" Jul 07 10:27:11.793570 ip-10-0-135-34 kubenswrapper[2578]: E0707 10:27:11.793548 2578 eviction_manager.go:267] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\"" Jul 07 10:27:11.793680 ip-10-0-135-34 kubenswrapper[2578]: E0707 10:27:11.793601 2578 eviction_manager.go:292] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-135-34.ec2.internal\" not found" Jul 07 10:27:11.813600 ip-10-0-135-34 kubenswrapper[2578]: I0707 10:27:11.813558 2578 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4" Jul 07 10:27:11.814965 ip-10-0-135-34 kubenswrapper[2578]: I0707 10:27:11.814945 2578 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6" Jul 07 10:27:11.815050 ip-10-0-135-34 kubenswrapper[2578]: I0707 10:27:11.814993 2578 status_manager.go:244] "Starting to sync pod status with apiserver" Jul 07 10:27:11.815050 ip-10-0-135-34 kubenswrapper[2578]: I0707 10:27:11.815030 2578 kubelet.go:2448] "Starting kubelet main sync loop" Jul 07 10:27:11.815112 ip-10-0-135-34 kubenswrapper[2578]: E0707 10:27:11.815071 2578 kubelet.go:2472] "Skipping pod synchronization" err="PLEG is not healthy: pleg has yet to be successful" Jul 07 10:27:11.817535 ip-10-0-135-34 kubenswrapper[2578]: I0707 10:27:11.817513 2578 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160" Jul 07 10:27:11.893101 ip-10-0-135-34 kubenswrapper[2578]: I0707 10:27:11.892986 2578 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 07 10:27:11.894241 ip-10-0-135-34 kubenswrapper[2578]: I0707 10:27:11.894225 2578 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-34.ec2.internal" event="NodeHasSufficientMemory" Jul 07 10:27:11.894299 ip-10-0-135-34 kubenswrapper[2578]: I0707 10:27:11.894255 2578 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-34.ec2.internal" event="NodeHasNoDiskPressure" Jul 07 10:27:11.894299 ip-10-0-135-34 kubenswrapper[2578]: I0707 10:27:11.894280 2578 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-34.ec2.internal" event="NodeHasSufficientPID" Jul 07 10:27:11.894363 ip-10-0-135-34 kubenswrapper[2578]: I0707 10:27:11.894302 2578 kubelet_node_status.go:78] "Attempting to register node" node="ip-10-0-135-34.ec2.internal" Jul 07 10:27:11.904516 ip-10-0-135-34 kubenswrapper[2578]: I0707 10:27:11.904485 2578 kubelet_node_status.go:81] "Successfully registered node" node="ip-10-0-135-34.ec2.internal" Jul 07 10:27:11.904516 ip-10-0-135-34 kubenswrapper[2578]: E0707 10:27:11.904518 2578 kubelet_node_status.go:536] "Error updating node status, will retry" err="error getting node \"ip-10-0-135-34.ec2.internal\": node \"ip-10-0-135-34.ec2.internal\" not found" Jul 07 10:27:11.916078 ip-10-0-135-34 kubenswrapper[2578]: I0707 10:27:11.916049 2578 kubelet.go:2535] "SyncLoop ADD" source="file" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-34.ec2.internal","kube-system/kube-apiserver-proxy-ip-10-0-135-34.ec2.internal"] Jul 07 10:27:11.916198 ip-10-0-135-34 kubenswrapper[2578]: I0707 10:27:11.916113 2578 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 07 10:27:11.917019 ip-10-0-135-34 kubenswrapper[2578]: I0707 10:27:11.916996 2578 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-34.ec2.internal" event="NodeHasSufficientMemory" Jul 07 10:27:11.917156 ip-10-0-135-34 kubenswrapper[2578]: I0707 10:27:11.917034 2578 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-34.ec2.internal" event="NodeHasNoDiskPressure" Jul 07 10:27:11.917156 ip-10-0-135-34 kubenswrapper[2578]: I0707 10:27:11.917048 2578 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-34.ec2.internal" event="NodeHasSufficientPID" Jul 07 10:27:11.918495 ip-10-0-135-34 kubenswrapper[2578]: I0707 10:27:11.918480 2578 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 07 10:27:11.918596 ip-10-0-135-34 kubenswrapper[2578]: I0707 10:27:11.918573 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-34.ec2.internal" Jul 07 10:27:11.918641 ip-10-0-135-34 kubenswrapper[2578]: I0707 10:27:11.918613 2578 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 07 10:27:11.919233 ip-10-0-135-34 kubenswrapper[2578]: I0707 10:27:11.919217 2578 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-34.ec2.internal" event="NodeHasSufficientMemory" Jul 07 10:27:11.919311 ip-10-0-135-34 kubenswrapper[2578]: I0707 10:27:11.919244 2578 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-34.ec2.internal" event="NodeHasNoDiskPressure" Jul 07 10:27:11.919311 ip-10-0-135-34 kubenswrapper[2578]: I0707 10:27:11.919257 2578 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-34.ec2.internal" event="NodeHasSufficientPID" Jul 07 10:27:11.919311 ip-10-0-135-34 kubenswrapper[2578]: I0707 10:27:11.919266 2578 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-34.ec2.internal" event="NodeHasSufficientMemory" Jul 07 10:27:11.919311 ip-10-0-135-34 kubenswrapper[2578]: I0707 10:27:11.919286 2578 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-34.ec2.internal" event="NodeHasNoDiskPressure" Jul 07 10:27:11.919311 ip-10-0-135-34 kubenswrapper[2578]: I0707 10:27:11.919298 2578 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-34.ec2.internal" event="NodeHasSufficientPID" Jul 07 10:27:11.920579 ip-10-0-135-34 kubenswrapper[2578]: I0707 10:27:11.920564 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-135-34.ec2.internal" Jul 07 10:27:11.920644 ip-10-0-135-34 kubenswrapper[2578]: I0707 10:27:11.920593 2578 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 07 10:27:11.921273 ip-10-0-135-34 kubenswrapper[2578]: I0707 10:27:11.921257 2578 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-34.ec2.internal" event="NodeHasSufficientMemory" Jul 07 10:27:11.921353 ip-10-0-135-34 kubenswrapper[2578]: I0707 10:27:11.921289 2578 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-34.ec2.internal" event="NodeHasNoDiskPressure" Jul 07 10:27:11.921353 ip-10-0-135-34 kubenswrapper[2578]: I0707 10:27:11.921302 2578 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-34.ec2.internal" event="NodeHasSufficientPID" Jul 07 10:27:11.937289 ip-10-0-135-34 kubenswrapper[2578]: E0707 10:27:11.937270 2578 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-135-34.ec2.internal\" not found" Jul 07 10:27:11.938400 ip-10-0-135-34 kubenswrapper[2578]: E0707 10:27:11.938384 2578 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-135-34.ec2.internal\" not found" node="ip-10-0-135-34.ec2.internal" Jul 07 10:27:11.939146 ip-10-0-135-34 kubenswrapper[2578]: I0707 10:27:11.939128 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/fc94ccac0285e755088555819ea0238a-config\") pod \"kube-apiserver-proxy-ip-10-0-135-34.ec2.internal\" (UID: \"fc94ccac0285e755088555819ea0238a\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-135-34.ec2.internal" Jul 07 10:27:11.939194 ip-10-0-135-34 kubenswrapper[2578]: I0707 10:27:11.939154 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/f3763173479b9a6f050d77b1ce5123b0-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-135-34.ec2.internal\" (UID: \"f3763173479b9a6f050d77b1ce5123b0\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-34.ec2.internal" Jul 07 10:27:11.939194 ip-10-0-135-34 kubenswrapper[2578]: I0707 10:27:11.939175 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/f3763173479b9a6f050d77b1ce5123b0-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-135-34.ec2.internal\" (UID: \"f3763173479b9a6f050d77b1ce5123b0\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-34.ec2.internal" Jul 07 10:27:11.942802 ip-10-0-135-34 kubenswrapper[2578]: E0707 10:27:11.942785 2578 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-135-34.ec2.internal\" not found" node="ip-10-0-135-34.ec2.internal" Jul 07 10:27:12.037495 ip-10-0-135-34 kubenswrapper[2578]: E0707 10:27:12.037462 2578 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-135-34.ec2.internal\" not found" Jul 07 10:27:12.039676 ip-10-0-135-34 kubenswrapper[2578]: I0707 10:27:12.039658 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/fc94ccac0285e755088555819ea0238a-config\") pod \"kube-apiserver-proxy-ip-10-0-135-34.ec2.internal\" (UID: \"fc94ccac0285e755088555819ea0238a\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-135-34.ec2.internal" Jul 07 10:27:12.039741 ip-10-0-135-34 kubenswrapper[2578]: I0707 10:27:12.039684 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/f3763173479b9a6f050d77b1ce5123b0-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-135-34.ec2.internal\" (UID: \"f3763173479b9a6f050d77b1ce5123b0\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-34.ec2.internal" Jul 07 10:27:12.039741 ip-10-0-135-34 kubenswrapper[2578]: I0707 10:27:12.039701 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/f3763173479b9a6f050d77b1ce5123b0-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-135-34.ec2.internal\" (UID: \"f3763173479b9a6f050d77b1ce5123b0\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-34.ec2.internal" Jul 07 10:27:12.039741 ip-10-0-135-34 kubenswrapper[2578]: I0707 10:27:12.039728 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/f3763173479b9a6f050d77b1ce5123b0-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-135-34.ec2.internal\" (UID: \"f3763173479b9a6f050d77b1ce5123b0\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-34.ec2.internal" Jul 07 10:27:12.039887 ip-10-0-135-34 kubenswrapper[2578]: I0707 10:27:12.039749 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/fc94ccac0285e755088555819ea0238a-config\") pod \"kube-apiserver-proxy-ip-10-0-135-34.ec2.internal\" (UID: \"fc94ccac0285e755088555819ea0238a\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-135-34.ec2.internal" Jul 07 10:27:12.039887 ip-10-0-135-34 kubenswrapper[2578]: I0707 10:27:12.039810 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/f3763173479b9a6f050d77b1ce5123b0-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-135-34.ec2.internal\" (UID: \"f3763173479b9a6f050d77b1ce5123b0\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-34.ec2.internal" Jul 07 10:27:12.137877 ip-10-0-135-34 kubenswrapper[2578]: E0707 10:27:12.137843 2578 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-135-34.ec2.internal\" not found" Jul 07 10:27:12.238688 ip-10-0-135-34 kubenswrapper[2578]: E0707 10:27:12.238617 2578 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-135-34.ec2.internal\" not found" Jul 07 10:27:12.241867 ip-10-0-135-34 kubenswrapper[2578]: I0707 10:27:12.241852 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-34.ec2.internal" Jul 07 10:27:12.246917 ip-10-0-135-34 kubenswrapper[2578]: I0707 10:27:12.246897 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-135-34.ec2.internal" Jul 07 10:27:12.339088 ip-10-0-135-34 kubenswrapper[2578]: E0707 10:27:12.339044 2578 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-135-34.ec2.internal\" not found" Jul 07 10:27:12.439559 ip-10-0-135-34 kubenswrapper[2578]: E0707 10:27:12.439530 2578 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-135-34.ec2.internal\" not found" Jul 07 10:27:12.540130 ip-10-0-135-34 kubenswrapper[2578]: E0707 10:27:12.540048 2578 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-135-34.ec2.internal\" not found" Jul 07 10:27:12.640647 ip-10-0-135-34 kubenswrapper[2578]: E0707 10:27:12.640609 2578 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-135-34.ec2.internal\" not found" Jul 07 10:27:12.674932 ip-10-0-135-34 kubenswrapper[2578]: I0707 10:27:12.674910 2578 transport.go:147] "Certificate rotation detected, shutting down client connections to start using new credentials" Jul 07 10:27:12.675465 ip-10-0-135-34 kubenswrapper[2578]: I0707 10:27:12.675098 2578 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.RuntimeClass" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received" Jul 07 10:27:12.736143 ip-10-0-135-34 kubenswrapper[2578]: I0707 10:27:12.736112 2578 reflector.go:439] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:160" Jul 07 10:27:12.737099 ip-10-0-135-34 kubenswrapper[2578]: I0707 10:27:12.737083 2578 kubelet.go:3248] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-34.ec2.internal" Jul 07 10:27:12.737099 ip-10-0-135-34 kubenswrapper[2578]: I0707 10:27:12.737092 2578 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kubelet-serving" Jul 07 10:27:12.750923 ip-10-0-135-34 kubenswrapper[2578]: I0707 10:27:12.750898 2578 reflector.go:439] "Caches populated" logger="kubernetes.io/kubelet-serving" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" Jul 07 10:27:12.755281 ip-10-0-135-34 kubenswrapper[2578]: I0707 10:27:12.755263 2578 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Jul 07 10:27:12.757289 ip-10-0-135-34 kubenswrapper[2578]: I0707 10:27:12.757272 2578 kubelet.go:3248] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-135-34.ec2.internal" Jul 07 10:27:12.766293 ip-10-0-135-34 kubenswrapper[2578]: I0707 10:27:12.766269 2578 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Jul 07 10:27:12.769416 ip-10-0-135-34 kubenswrapper[2578]: I0707 10:27:12.769387 2578 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-07-06 10:22:12 +0000 UTC" deadline="2028-03-25 20:23:41.118407094 +0000 UTC" Jul 07 10:27:12.769483 ip-10-0-135-34 kubenswrapper[2578]: I0707 10:27:12.769417 2578 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="15057h56m28.34899432s" Jul 07 10:27:12.773388 ip-10-0-135-34 kubenswrapper[2578]: I0707 10:27:12.773370 2578 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kubelet-serving" csr="csr-cd5jd" Jul 07 10:27:12.781709 ip-10-0-135-34 kubenswrapper[2578]: I0707 10:27:12.781681 2578 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kubelet-serving" csr="csr-cd5jd" Jul 07 10:27:12.811930 ip-10-0-135-34 kubenswrapper[2578]: I0707 10:27:12.811865 2578 reflector.go:439] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:160" Jul 07 10:27:12.951891 ip-10-0-135-34 kubenswrapper[2578]: W0707 10:27:12.951855 2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfc94ccac0285e755088555819ea0238a.slice/crio-29d1ad5421f2c8b2b7e953a90b966ee477a39c5ad9cb4d6d12d6e453d749a3dd WatchSource:0}: Error finding container 29d1ad5421f2c8b2b7e953a90b966ee477a39c5ad9cb4d6d12d6e453d749a3dd: Status 404 returned error can't find the container with id 29d1ad5421f2c8b2b7e953a90b966ee477a39c5ad9cb4d6d12d6e453d749a3dd Jul 07 10:27:12.952147 ip-10-0-135-34 kubenswrapper[2578]: W0707 10:27:12.952129 2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf3763173479b9a6f050d77b1ce5123b0.slice/crio-69e706223c12e8313c7a1b94782b258020b3087a246437cd45554d769776b38a WatchSource:0}: Error finding container 69e706223c12e8313c7a1b94782b258020b3087a246437cd45554d769776b38a: Status 404 returned error can't find the container with id 69e706223c12e8313c7a1b94782b258020b3087a246437cd45554d769776b38a Jul 07 10:27:12.958622 ip-10-0-135-34 kubenswrapper[2578]: I0707 10:27:12.958605 2578 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Jul 07 10:27:13.264537 ip-10-0-135-34 kubenswrapper[2578]: I0707 10:27:13.264501 2578 reflector.go:439] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:160" Jul 07 10:27:13.716222 ip-10-0-135-34 kubenswrapper[2578]: I0707 10:27:13.716131 2578 apiserver.go:52] "Watching apiserver" Jul 07 10:27:13.724905 ip-10-0-135-34 kubenswrapper[2578]: I0707 10:27:13.724882 2578 reflector.go:439] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66" Jul 07 10:27:13.726103 ip-10-0-135-34 kubenswrapper[2578]: I0707 10:27:13.726078 2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kube-system/global-pull-secret-syncer-7lv79","kube-system/konnectivity-agent-t82fl","openshift-dns/node-resolver-rdrnq","openshift-multus/multus-hwz82","openshift-network-diagnostics/network-check-target-26sg7","openshift-network-operator/iptables-alerter-lr7dk","openshift-ovn-kubernetes/ovnkube-node-qhhw7","kube-system/kube-apiserver-proxy-ip-10-0-135-34.ec2.internal","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-dll6x","openshift-cluster-node-tuning-operator/tuned-lj6lw","openshift-image-registry/node-ca-pb2p8","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-34.ec2.internal","openshift-multus/multus-additional-cni-plugins-l8d7k","openshift-multus/network-metrics-daemon-p2ks2"] Jul 07 10:27:13.727823 ip-10-0-135-34 kubenswrapper[2578]: I0707 10:27:13.727809 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-lj6lw" Jul 07 10:27:13.729008 ip-10-0-135-34 kubenswrapper[2578]: I0707 10:27:13.728986 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-l8d7k" Jul 07 10:27:13.730107 ip-10-0-135-34 kubenswrapper[2578]: I0707 10:27:13.730091 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-rdrnq" Jul 07 10:27:13.730587 ip-10-0-135-34 kubenswrapper[2578]: I0707 10:27:13.730571 2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\"" Jul 07 10:27:13.730587 ip-10-0-135-34 kubenswrapper[2578]: I0707 10:27:13.730580 2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\"" Jul 07 10:27:13.730726 ip-10-0-135-34 kubenswrapper[2578]: I0707 10:27:13.730587 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-rkq9g\"" Jul 07 10:27:13.731935 ip-10-0-135-34 kubenswrapper[2578]: I0707 10:27:13.731908 2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\"" Jul 07 10:27:13.732373 ip-10-0-135-34 kubenswrapper[2578]: I0707 10:27:13.732352 2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\"" Jul 07 10:27:13.732482 ip-10-0-135-34 kubenswrapper[2578]: I0707 10:27:13.732371 2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\"" Jul 07 10:27:13.732482 ip-10-0-135-34 kubenswrapper[2578]: I0707 10:27:13.732466 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-hwz82" Jul 07 10:27:13.732809 ip-10-0-135-34 kubenswrapper[2578]: I0707 10:27:13.732789 2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\"" Jul 07 10:27:13.732947 ip-10-0-135-34 kubenswrapper[2578]: I0707 10:27:13.732930 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-vb56t\"" Jul 07 10:27:13.733019 ip-10-0-135-34 kubenswrapper[2578]: I0707 10:27:13.733008 2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\"" Jul 07 10:27:13.733079 ip-10-0-135-34 kubenswrapper[2578]: I0707 10:27:13.733030 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-cq8kd\"" Jul 07 10:27:13.733210 ip-10-0-135-34 kubenswrapper[2578]: I0707 10:27:13.733195 2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\"" Jul 07 10:27:13.733277 ip-10-0-135-34 kubenswrapper[2578]: I0707 10:27:13.733241 2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\"" Jul 07 10:27:13.733905 ip-10-0-135-34 kubenswrapper[2578]: I0707 10:27:13.733885 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-26sg7" Jul 07 10:27:13.734014 ip-10-0-135-34 kubenswrapper[2578]: E0707 10:27:13.733967 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-26sg7" podUID="51fe14a4-6b46-4f88-be7d-b611b45961ff" Jul 07 10:27:13.735150 ip-10-0-135-34 kubenswrapper[2578]: I0707 10:27:13.735132 2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\"" Jul 07 10:27:13.735387 ip-10-0-135-34 kubenswrapper[2578]: I0707 10:27:13.735371 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"default-dockercfg-27pvk\"" Jul 07 10:27:13.735999 ip-10-0-135-34 kubenswrapper[2578]: I0707 10:27:13.735980 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-lr7dk" Jul 07 10:27:13.736871 ip-10-0-135-34 kubenswrapper[2578]: I0707 10:27:13.736851 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-dll6x" Jul 07 10:27:13.738222 ip-10-0-135-34 kubenswrapper[2578]: I0707 10:27:13.738202 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-p2ks2" Jul 07 10:27:13.738309 ip-10-0-135-34 kubenswrapper[2578]: E0707 10:27:13.738288 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-p2ks2" podUID="931a7d75-9ee8-4758-9f23-3521f3db5eae" Jul 07 10:27:13.738546 ip-10-0-135-34 kubenswrapper[2578]: I0707 10:27:13.738530 2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\"" Jul 07 10:27:13.740239 ip-10-0-135-34 kubenswrapper[2578]: I0707 10:27:13.739263 2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\"" Jul 07 10:27:13.740239 ip-10-0-135-34 kubenswrapper[2578]: I0707 10:27:13.739283 2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\"" Jul 07 10:27:13.740239 ip-10-0-135-34 kubenswrapper[2578]: I0707 10:27:13.739399 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-8ckfm\"" Jul 07 10:27:13.741444 ip-10-0-135-34 kubenswrapper[2578]: I0707 10:27:13.741198 2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\"" Jul 07 10:27:13.741444 ip-10-0-135-34 kubenswrapper[2578]: I0707 10:27:13.741229 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-jd76f\"" Jul 07 10:27:13.741444 ip-10-0-135-34 kubenswrapper[2578]: I0707 10:27:13.741207 2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\"" Jul 07 10:27:13.741444 ip-10-0-135-34 kubenswrapper[2578]: I0707 10:27:13.741286 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-t82fl" Jul 07 10:27:13.741444 ip-10-0-135-34 kubenswrapper[2578]: I0707 10:27:13.741209 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\"" Jul 07 10:27:13.742774 ip-10-0-135-34 kubenswrapper[2578]: I0707 10:27:13.741473 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-7lv79" Jul 07 10:27:13.742774 ip-10-0-135-34 kubenswrapper[2578]: E0707 10:27:13.741540 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-7lv79" podUID="f85fac08-720e-4ffd-80af-19c140c36804" Jul 07 10:27:13.743660 ip-10-0-135-34 kubenswrapper[2578]: I0707 10:27:13.743639 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-pb2p8" Jul 07 10:27:13.744632 ip-10-0-135-34 kubenswrapper[2578]: I0707 10:27:13.744534 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"default-dockercfg-jjvj8\"" Jul 07 10:27:13.745012 ip-10-0-135-34 kubenswrapper[2578]: I0707 10:27:13.744941 2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\"" Jul 07 10:27:13.745266 ip-10-0-135-34 kubenswrapper[2578]: I0707 10:27:13.745248 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-qhhw7" Jul 07 10:27:13.745353 ip-10-0-135-34 kubenswrapper[2578]: I0707 10:27:13.745335 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\"" Jul 07 10:27:13.746213 ip-10-0-135-34 kubenswrapper[2578]: I0707 10:27:13.746186 2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\"" Jul 07 10:27:13.746833 ip-10-0-135-34 kubenswrapper[2578]: I0707 10:27:13.746790 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-hbjnp\"" Jul 07 10:27:13.746949 ip-10-0-135-34 kubenswrapper[2578]: I0707 10:27:13.746927 2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\"" Jul 07 10:27:13.747209 ip-10-0-135-34 kubenswrapper[2578]: I0707 10:27:13.747051 2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\"" Jul 07 10:27:13.747867 ip-10-0-135-34 kubenswrapper[2578]: I0707 10:27:13.747830 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-6xbbt\"" Jul 07 10:27:13.748199 ip-10-0-135-34 kubenswrapper[2578]: I0707 10:27:13.748182 2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\"" Jul 07 10:27:13.748324 ip-10-0-135-34 kubenswrapper[2578]: I0707 10:27:13.748187 2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\"" Jul 07 10:27:13.748438 ip-10-0-135-34 kubenswrapper[2578]: I0707 10:27:13.748406 2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\"" Jul 07 10:27:13.748900 ip-10-0-135-34 kubenswrapper[2578]: I0707 10:27:13.748727 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\"" Jul 07 10:27:13.748900 ip-10-0-135-34 kubenswrapper[2578]: I0707 10:27:13.748748 2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\"" Jul 07 10:27:13.749062 ip-10-0-135-34 kubenswrapper[2578]: I0707 10:27:13.748998 2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\"" Jul 07 10:27:13.749260 ip-10-0-135-34 kubenswrapper[2578]: I0707 10:27:13.749236 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/a415d240-ff8a-4b08-aeb6-b4018195c46b-os-release\") pod \"multus-additional-cni-plugins-l8d7k\" (UID: \"a415d240-ff8a-4b08-aeb6-b4018195c46b\") " pod="openshift-multus/multus-additional-cni-plugins-l8d7k" Jul 07 10:27:13.749345 ip-10-0-135-34 kubenswrapper[2578]: I0707 10:27:13.749274 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/2410fb3f-6b21-40b9-b23b-ffdbbdb132a7-host-run-k8s-cni-cncf-io\") pod \"multus-hwz82\" (UID: \"2410fb3f-6b21-40b9-b23b-ffdbbdb132a7\") " pod="openshift-multus/multus-hwz82" Jul 07 10:27:13.749345 ip-10-0-135-34 kubenswrapper[2578]: I0707 10:27:13.749301 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-29rjr\" (UniqueName: \"kubernetes.io/projected/609c4941-f311-408a-82ec-1d6b1a54572d-kube-api-access-29rjr\") pod \"iptables-alerter-lr7dk\" (UID: \"609c4941-f311-408a-82ec-1d6b1a54572d\") " pod="openshift-network-operator/iptables-alerter-lr7dk" Jul 07 10:27:13.749467 ip-10-0-135-34 kubenswrapper[2578]: I0707 10:27:13.749342 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/1e1b149f-fe01-49d2-8387-4cf7aea820e7-etc-modprobe-d\") pod \"tuned-lj6lw\" (UID: \"1e1b149f-fe01-49d2-8387-4cf7aea820e7\") " pod="openshift-cluster-node-tuning-operator/tuned-lj6lw" Jul 07 10:27:13.749467 ip-10-0-135-34 kubenswrapper[2578]: I0707 10:27:13.749367 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/1e1b149f-fe01-49d2-8387-4cf7aea820e7-host\") pod \"tuned-lj6lw\" (UID: \"1e1b149f-fe01-49d2-8387-4cf7aea820e7\") " pod="openshift-cluster-node-tuning-operator/tuned-lj6lw" Jul 07 10:27:13.749467 ip-10-0-135-34 kubenswrapper[2578]: I0707 10:27:13.749402 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/b1617485-94f6-4eba-95e9-866fc2400643-sys-fs\") pod \"aws-ebs-csi-driver-node-dll6x\" (UID: \"b1617485-94f6-4eba-95e9-866fc2400643\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-dll6x" Jul 07 10:27:13.749467 ip-10-0-135-34 kubenswrapper[2578]: I0707 10:27:13.749429 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vw57m\" (UniqueName: \"kubernetes.io/projected/b1617485-94f6-4eba-95e9-866fc2400643-kube-api-access-vw57m\") pod \"aws-ebs-csi-driver-node-dll6x\" (UID: \"b1617485-94f6-4eba-95e9-866fc2400643\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-dll6x" Jul 07 10:27:13.749467 ip-10-0-135-34 kubenswrapper[2578]: I0707 10:27:13.749455 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/1e1b149f-fe01-49d2-8387-4cf7aea820e7-var-lib-kubelet\") pod \"tuned-lj6lw\" (UID: \"1e1b149f-fe01-49d2-8387-4cf7aea820e7\") " pod="openshift-cluster-node-tuning-operator/tuned-lj6lw" Jul 07 10:27:13.749716 ip-10-0-135-34 kubenswrapper[2578]: I0707 10:27:13.749479 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/1e1b149f-fe01-49d2-8387-4cf7aea820e7-etc-tuned\") pod \"tuned-lj6lw\" (UID: \"1e1b149f-fe01-49d2-8387-4cf7aea820e7\") " pod="openshift-cluster-node-tuning-operator/tuned-lj6lw" Jul 07 10:27:13.749716 ip-10-0-135-34 kubenswrapper[2578]: I0707 10:27:13.749610 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/1e1b149f-fe01-49d2-8387-4cf7aea820e7-etc-sysconfig\") pod \"tuned-lj6lw\" (UID: \"1e1b149f-fe01-49d2-8387-4cf7aea820e7\") " pod="openshift-cluster-node-tuning-operator/tuned-lj6lw" Jul 07 10:27:13.749716 ip-10-0-135-34 kubenswrapper[2578]: I0707 10:27:13.749645 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/1e1b149f-fe01-49d2-8387-4cf7aea820e7-etc-sysctl-conf\") pod \"tuned-lj6lw\" (UID: \"1e1b149f-fe01-49d2-8387-4cf7aea820e7\") " pod="openshift-cluster-node-tuning-operator/tuned-lj6lw" Jul 07 10:27:13.749716 ip-10-0-135-34 kubenswrapper[2578]: I0707 10:27:13.749674 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9vbqq\" (UniqueName: \"kubernetes.io/projected/51fe14a4-6b46-4f88-be7d-b611b45961ff-kube-api-access-9vbqq\") pod \"network-check-target-26sg7\" (UID: \"51fe14a4-6b46-4f88-be7d-b611b45961ff\") " pod="openshift-network-diagnostics/network-check-target-26sg7" Jul 07 10:27:13.749929 ip-10-0-135-34 kubenswrapper[2578]: I0707 10:27:13.749719 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/2410fb3f-6b21-40b9-b23b-ffdbbdb132a7-hostroot\") pod \"multus-hwz82\" (UID: \"2410fb3f-6b21-40b9-b23b-ffdbbdb132a7\") " pod="openshift-multus/multus-hwz82" Jul 07 10:27:13.749929 ip-10-0-135-34 kubenswrapper[2578]: I0707 10:27:13.749746 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/2410fb3f-6b21-40b9-b23b-ffdbbdb132a7-host-run-multus-certs\") pod \"multus-hwz82\" (UID: \"2410fb3f-6b21-40b9-b23b-ffdbbdb132a7\") " pod="openshift-multus/multus-hwz82" Jul 07 10:27:13.749929 ip-10-0-135-34 kubenswrapper[2578]: I0707 10:27:13.749763 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/1e1b149f-fe01-49d2-8387-4cf7aea820e7-run\") pod \"tuned-lj6lw\" (UID: \"1e1b149f-fe01-49d2-8387-4cf7aea820e7\") " pod="openshift-cluster-node-tuning-operator/tuned-lj6lw" Jul 07 10:27:13.749929 ip-10-0-135-34 kubenswrapper[2578]: I0707 10:27:13.749799 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/b1617485-94f6-4eba-95e9-866fc2400643-registration-dir\") pod \"aws-ebs-csi-driver-node-dll6x\" (UID: \"b1617485-94f6-4eba-95e9-866fc2400643\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-dll6x" Jul 07 10:27:13.749929 ip-10-0-135-34 kubenswrapper[2578]: I0707 10:27:13.749825 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/2410fb3f-6b21-40b9-b23b-ffdbbdb132a7-multus-cni-dir\") pod \"multus-hwz82\" (UID: \"2410fb3f-6b21-40b9-b23b-ffdbbdb132a7\") " pod="openshift-multus/multus-hwz82" Jul 07 10:27:13.749929 ip-10-0-135-34 kubenswrapper[2578]: I0707 10:27:13.749848 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/2410fb3f-6b21-40b9-b23b-ffdbbdb132a7-multus-daemon-config\") pod \"multus-hwz82\" (UID: \"2410fb3f-6b21-40b9-b23b-ffdbbdb132a7\") " pod="openshift-multus/multus-hwz82" Jul 07 10:27:13.749929 ip-10-0-135-34 kubenswrapper[2578]: I0707 10:27:13.749874 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/1e1b149f-fe01-49d2-8387-4cf7aea820e7-etc-kubernetes\") pod \"tuned-lj6lw\" (UID: \"1e1b149f-fe01-49d2-8387-4cf7aea820e7\") " pod="openshift-cluster-node-tuning-operator/tuned-lj6lw" Jul 07 10:27:13.749929 ip-10-0-135-34 kubenswrapper[2578]: I0707 10:27:13.749900 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/2410fb3f-6b21-40b9-b23b-ffdbbdb132a7-cni-binary-copy\") pod \"multus-hwz82\" (UID: \"2410fb3f-6b21-40b9-b23b-ffdbbdb132a7\") " pod="openshift-multus/multus-hwz82" Jul 07 10:27:13.750310 ip-10-0-135-34 kubenswrapper[2578]: I0707 10:27:13.749946 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/a415d240-ff8a-4b08-aeb6-b4018195c46b-cni-binary-copy\") pod \"multus-additional-cni-plugins-l8d7k\" (UID: \"a415d240-ff8a-4b08-aeb6-b4018195c46b\") " pod="openshift-multus/multus-additional-cni-plugins-l8d7k" Jul 07 10:27:13.750310 ip-10-0-135-34 kubenswrapper[2578]: I0707 10:27:13.749981 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/609c4941-f311-408a-82ec-1d6b1a54572d-host-slash\") pod \"iptables-alerter-lr7dk\" (UID: \"609c4941-f311-408a-82ec-1d6b1a54572d\") " pod="openshift-network-operator/iptables-alerter-lr7dk" Jul 07 10:27:13.750310 ip-10-0-135-34 kubenswrapper[2578]: I0707 10:27:13.750017 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/1e1b149f-fe01-49d2-8387-4cf7aea820e7-etc-systemd\") pod \"tuned-lj6lw\" (UID: \"1e1b149f-fe01-49d2-8387-4cf7aea820e7\") " pod="openshift-cluster-node-tuning-operator/tuned-lj6lw" Jul 07 10:27:13.750310 ip-10-0-135-34 kubenswrapper[2578]: I0707 10:27:13.750056 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/b1617485-94f6-4eba-95e9-866fc2400643-device-dir\") pod \"aws-ebs-csi-driver-node-dll6x\" (UID: \"b1617485-94f6-4eba-95e9-866fc2400643\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-dll6x" Jul 07 10:27:13.750310 ip-10-0-135-34 kubenswrapper[2578]: I0707 10:27:13.750083 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/a415d240-ff8a-4b08-aeb6-b4018195c46b-tuning-conf-dir\") pod \"multus-additional-cni-plugins-l8d7k\" (UID: \"a415d240-ff8a-4b08-aeb6-b4018195c46b\") " pod="openshift-multus/multus-additional-cni-plugins-l8d7k" Jul 07 10:27:13.750310 ip-10-0-135-34 kubenswrapper[2578]: I0707 10:27:13.750107 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/2410fb3f-6b21-40b9-b23b-ffdbbdb132a7-host-var-lib-cni-bin\") pod \"multus-hwz82\" (UID: \"2410fb3f-6b21-40b9-b23b-ffdbbdb132a7\") " pod="openshift-multus/multus-hwz82" Jul 07 10:27:13.750310 ip-10-0-135-34 kubenswrapper[2578]: I0707 10:27:13.750134 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/2410fb3f-6b21-40b9-b23b-ffdbbdb132a7-host-var-lib-kubelet\") pod \"multus-hwz82\" (UID: \"2410fb3f-6b21-40b9-b23b-ffdbbdb132a7\") " pod="openshift-multus/multus-hwz82" Jul 07 10:27:13.750310 ip-10-0-135-34 kubenswrapper[2578]: I0707 10:27:13.750164 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/931a7d75-9ee8-4758-9f23-3521f3db5eae-metrics-certs\") pod \"network-metrics-daemon-p2ks2\" (UID: \"931a7d75-9ee8-4758-9f23-3521f3db5eae\") " pod="openshift-multus/network-metrics-daemon-p2ks2" Jul 07 10:27:13.750310 ip-10-0-135-34 kubenswrapper[2578]: I0707 10:27:13.750182 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/1e1b149f-fe01-49d2-8387-4cf7aea820e7-etc-sysctl-d\") pod \"tuned-lj6lw\" (UID: \"1e1b149f-fe01-49d2-8387-4cf7aea820e7\") " pod="openshift-cluster-node-tuning-operator/tuned-lj6lw" Jul 07 10:27:13.750310 ip-10-0-135-34 kubenswrapper[2578]: I0707 10:27:13.750196 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wbblv\" (UniqueName: \"kubernetes.io/projected/dfa64e14-3729-4f28-870f-eeab6cec9944-kube-api-access-wbblv\") pod \"node-resolver-rdrnq\" (UID: \"dfa64e14-3729-4f28-870f-eeab6cec9944\") " pod="openshift-dns/node-resolver-rdrnq" Jul 07 10:27:13.750310 ip-10-0-135-34 kubenswrapper[2578]: I0707 10:27:13.750212 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/b1617485-94f6-4eba-95e9-866fc2400643-etc-selinux\") pod \"aws-ebs-csi-driver-node-dll6x\" (UID: \"b1617485-94f6-4eba-95e9-866fc2400643\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-dll6x" Jul 07 10:27:13.750310 ip-10-0-135-34 kubenswrapper[2578]: I0707 10:27:13.750242 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/1e1b149f-fe01-49d2-8387-4cf7aea820e7-sys\") pod \"tuned-lj6lw\" (UID: \"1e1b149f-fe01-49d2-8387-4cf7aea820e7\") " pod="openshift-cluster-node-tuning-operator/tuned-lj6lw" Jul 07 10:27:13.750310 ip-10-0-135-34 kubenswrapper[2578]: I0707 10:27:13.750267 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/1e1b149f-fe01-49d2-8387-4cf7aea820e7-lib-modules\") pod \"tuned-lj6lw\" (UID: \"1e1b149f-fe01-49d2-8387-4cf7aea820e7\") " pod="openshift-cluster-node-tuning-operator/tuned-lj6lw" Jul 07 10:27:13.750310 ip-10-0-135-34 kubenswrapper[2578]: I0707 10:27:13.750289 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/b1617485-94f6-4eba-95e9-866fc2400643-socket-dir\") pod \"aws-ebs-csi-driver-node-dll6x\" (UID: \"b1617485-94f6-4eba-95e9-866fc2400643\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-dll6x" Jul 07 10:27:13.750752 ip-10-0-135-34 kubenswrapper[2578]: I0707 10:27:13.750326 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/2410fb3f-6b21-40b9-b23b-ffdbbdb132a7-cnibin\") pod \"multus-hwz82\" (UID: \"2410fb3f-6b21-40b9-b23b-ffdbbdb132a7\") " pod="openshift-multus/multus-hwz82" Jul 07 10:27:13.750752 ip-10-0-135-34 kubenswrapper[2578]: I0707 10:27:13.750354 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/2410fb3f-6b21-40b9-b23b-ffdbbdb132a7-multus-socket-dir-parent\") pod \"multus-hwz82\" (UID: \"2410fb3f-6b21-40b9-b23b-ffdbbdb132a7\") " pod="openshift-multus/multus-hwz82" Jul 07 10:27:13.750752 ip-10-0-135-34 kubenswrapper[2578]: I0707 10:27:13.750393 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/dfa64e14-3729-4f28-870f-eeab6cec9944-hosts-file\") pod \"node-resolver-rdrnq\" (UID: \"dfa64e14-3729-4f28-870f-eeab6cec9944\") " pod="openshift-dns/node-resolver-rdrnq" Jul 07 10:27:13.750752 ip-10-0-135-34 kubenswrapper[2578]: I0707 10:27:13.750442 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/dfa64e14-3729-4f28-870f-eeab6cec9944-tmp-dir\") pod \"node-resolver-rdrnq\" (UID: \"dfa64e14-3729-4f28-870f-eeab6cec9944\") " pod="openshift-dns/node-resolver-rdrnq" Jul 07 10:27:13.750752 ip-10-0-135-34 kubenswrapper[2578]: I0707 10:27:13.750472 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9n5dz\" (UniqueName: \"kubernetes.io/projected/a415d240-ff8a-4b08-aeb6-b4018195c46b-kube-api-access-9n5dz\") pod \"multus-additional-cni-plugins-l8d7k\" (UID: \"a415d240-ff8a-4b08-aeb6-b4018195c46b\") " pod="openshift-multus/multus-additional-cni-plugins-l8d7k" Jul 07 10:27:13.750752 ip-10-0-135-34 kubenswrapper[2578]: I0707 10:27:13.750501 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ncrl8\" (UniqueName: \"kubernetes.io/projected/931a7d75-9ee8-4758-9f23-3521f3db5eae-kube-api-access-ncrl8\") pod \"network-metrics-daemon-p2ks2\" (UID: \"931a7d75-9ee8-4758-9f23-3521f3db5eae\") " pod="openshift-multus/network-metrics-daemon-p2ks2" Jul 07 10:27:13.750752 ip-10-0-135-34 kubenswrapper[2578]: I0707 10:27:13.750524 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/1e1b149f-fe01-49d2-8387-4cf7aea820e7-tmp\") pod \"tuned-lj6lw\" (UID: \"1e1b149f-fe01-49d2-8387-4cf7aea820e7\") " pod="openshift-cluster-node-tuning-operator/tuned-lj6lw" Jul 07 10:27:13.750752 ip-10-0-135-34 kubenswrapper[2578]: I0707 10:27:13.750578 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/b1617485-94f6-4eba-95e9-866fc2400643-kubelet-dir\") pod \"aws-ebs-csi-driver-node-dll6x\" (UID: \"b1617485-94f6-4eba-95e9-866fc2400643\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-dll6x" Jul 07 10:27:13.750752 ip-10-0-135-34 kubenswrapper[2578]: I0707 10:27:13.750606 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/a415d240-ff8a-4b08-aeb6-b4018195c46b-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-l8d7k\" (UID: \"a415d240-ff8a-4b08-aeb6-b4018195c46b\") " pod="openshift-multus/multus-additional-cni-plugins-l8d7k" Jul 07 10:27:13.750752 ip-10-0-135-34 kubenswrapper[2578]: I0707 10:27:13.750622 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/2410fb3f-6b21-40b9-b23b-ffdbbdb132a7-host-var-lib-cni-multus\") pod \"multus-hwz82\" (UID: \"2410fb3f-6b21-40b9-b23b-ffdbbdb132a7\") " pod="openshift-multus/multus-hwz82" Jul 07 10:27:13.750752 ip-10-0-135-34 kubenswrapper[2578]: I0707 10:27:13.750637 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-f8c4t\" (UniqueName: \"kubernetes.io/projected/2410fb3f-6b21-40b9-b23b-ffdbbdb132a7-kube-api-access-f8c4t\") pod \"multus-hwz82\" (UID: \"2410fb3f-6b21-40b9-b23b-ffdbbdb132a7\") " pod="openshift-multus/multus-hwz82" Jul 07 10:27:13.750752 ip-10-0-135-34 kubenswrapper[2578]: I0707 10:27:13.750655 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/a415d240-ff8a-4b08-aeb6-b4018195c46b-system-cni-dir\") pod \"multus-additional-cni-plugins-l8d7k\" (UID: \"a415d240-ff8a-4b08-aeb6-b4018195c46b\") " pod="openshift-multus/multus-additional-cni-plugins-l8d7k" Jul 07 10:27:13.750752 ip-10-0-135-34 kubenswrapper[2578]: I0707 10:27:13.750682 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/2410fb3f-6b21-40b9-b23b-ffdbbdb132a7-multus-conf-dir\") pod \"multus-hwz82\" (UID: \"2410fb3f-6b21-40b9-b23b-ffdbbdb132a7\") " pod="openshift-multus/multus-hwz82" Jul 07 10:27:13.750752 ip-10-0-135-34 kubenswrapper[2578]: I0707 10:27:13.750701 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/2410fb3f-6b21-40b9-b23b-ffdbbdb132a7-etc-kubernetes\") pod \"multus-hwz82\" (UID: \"2410fb3f-6b21-40b9-b23b-ffdbbdb132a7\") " pod="openshift-multus/multus-hwz82" Jul 07 10:27:13.750752 ip-10-0-135-34 kubenswrapper[2578]: I0707 10:27:13.750735 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bw8ks\" (UniqueName: \"kubernetes.io/projected/1e1b149f-fe01-49d2-8387-4cf7aea820e7-kube-api-access-bw8ks\") pod \"tuned-lj6lw\" (UID: \"1e1b149f-fe01-49d2-8387-4cf7aea820e7\") " pod="openshift-cluster-node-tuning-operator/tuned-lj6lw" Jul 07 10:27:13.751368 ip-10-0-135-34 kubenswrapper[2578]: I0707 10:27:13.750775 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/a415d240-ff8a-4b08-aeb6-b4018195c46b-cnibin\") pod \"multus-additional-cni-plugins-l8d7k\" (UID: \"a415d240-ff8a-4b08-aeb6-b4018195c46b\") " pod="openshift-multus/multus-additional-cni-plugins-l8d7k" Jul 07 10:27:13.751368 ip-10-0-135-34 kubenswrapper[2578]: I0707 10:27:13.750797 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/609c4941-f311-408a-82ec-1d6b1a54572d-iptables-alerter-script\") pod \"iptables-alerter-lr7dk\" (UID: \"609c4941-f311-408a-82ec-1d6b1a54572d\") " pod="openshift-network-operator/iptables-alerter-lr7dk" Jul 07 10:27:13.751368 ip-10-0-135-34 kubenswrapper[2578]: I0707 10:27:13.750813 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/a415d240-ff8a-4b08-aeb6-b4018195c46b-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-l8d7k\" (UID: \"a415d240-ff8a-4b08-aeb6-b4018195c46b\") " pod="openshift-multus/multus-additional-cni-plugins-l8d7k" Jul 07 10:27:13.751368 ip-10-0-135-34 kubenswrapper[2578]: I0707 10:27:13.750829 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/2410fb3f-6b21-40b9-b23b-ffdbbdb132a7-system-cni-dir\") pod \"multus-hwz82\" (UID: \"2410fb3f-6b21-40b9-b23b-ffdbbdb132a7\") " pod="openshift-multus/multus-hwz82" Jul 07 10:27:13.751368 ip-10-0-135-34 kubenswrapper[2578]: I0707 10:27:13.750844 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/2410fb3f-6b21-40b9-b23b-ffdbbdb132a7-os-release\") pod \"multus-hwz82\" (UID: \"2410fb3f-6b21-40b9-b23b-ffdbbdb132a7\") " pod="openshift-multus/multus-hwz82" Jul 07 10:27:13.751368 ip-10-0-135-34 kubenswrapper[2578]: I0707 10:27:13.750875 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/2410fb3f-6b21-40b9-b23b-ffdbbdb132a7-host-run-netns\") pod \"multus-hwz82\" (UID: \"2410fb3f-6b21-40b9-b23b-ffdbbdb132a7\") " pod="openshift-multus/multus-hwz82" Jul 07 10:27:13.783566 ip-10-0-135-34 kubenswrapper[2578]: I0707 10:27:13.783515 2578 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-07-06 10:22:13 +0000 UTC" deadline="2027-12-24 01:16:18.640970048 +0000 UTC" Jul 07 10:27:13.783566 ip-10-0-135-34 kubenswrapper[2578]: I0707 10:27:13.783561 2578 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="12830h49m4.857411929s" Jul 07 10:27:13.821386 ip-10-0-135-34 kubenswrapper[2578]: I0707 10:27:13.821312 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-135-34.ec2.internal" event={"ID":"fc94ccac0285e755088555819ea0238a","Type":"ContainerStarted","Data":"29d1ad5421f2c8b2b7e953a90b966ee477a39c5ad9cb4d6d12d6e453d749a3dd"} Jul 07 10:27:13.823228 ip-10-0-135-34 kubenswrapper[2578]: I0707 10:27:13.823194 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-34.ec2.internal" event={"ID":"f3763173479b9a6f050d77b1ce5123b0","Type":"ContainerStarted","Data":"69e706223c12e8313c7a1b94782b258020b3087a246437cd45554d769776b38a"} Jul 07 10:27:13.838833 ip-10-0-135-34 kubenswrapper[2578]: I0707 10:27:13.838801 2578 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world" Jul 07 10:27:13.851691 ip-10-0-135-34 kubenswrapper[2578]: I0707 10:27:13.851640 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/a415d240-ff8a-4b08-aeb6-b4018195c46b-os-release\") pod \"multus-additional-cni-plugins-l8d7k\" (UID: \"a415d240-ff8a-4b08-aeb6-b4018195c46b\") " pod="openshift-multus/multus-additional-cni-plugins-l8d7k" Jul 07 10:27:13.851691 ip-10-0-135-34 kubenswrapper[2578]: I0707 10:27:13.851691 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/2410fb3f-6b21-40b9-b23b-ffdbbdb132a7-host-run-k8s-cni-cncf-io\") pod \"multus-hwz82\" (UID: \"2410fb3f-6b21-40b9-b23b-ffdbbdb132a7\") " pod="openshift-multus/multus-hwz82" Jul 07 10:27:13.851955 ip-10-0-135-34 kubenswrapper[2578]: I0707 10:27:13.851719 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-29rjr\" (UniqueName: \"kubernetes.io/projected/609c4941-f311-408a-82ec-1d6b1a54572d-kube-api-access-29rjr\") pod \"iptables-alerter-lr7dk\" (UID: \"609c4941-f311-408a-82ec-1d6b1a54572d\") " pod="openshift-network-operator/iptables-alerter-lr7dk" Jul 07 10:27:13.851955 ip-10-0-135-34 kubenswrapper[2578]: I0707 10:27:13.851745 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/1e1b149f-fe01-49d2-8387-4cf7aea820e7-etc-modprobe-d\") pod \"tuned-lj6lw\" (UID: \"1e1b149f-fe01-49d2-8387-4cf7aea820e7\") " pod="openshift-cluster-node-tuning-operator/tuned-lj6lw" Jul 07 10:27:13.851955 ip-10-0-135-34 kubenswrapper[2578]: I0707 10:27:13.851767 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/1e1b149f-fe01-49d2-8387-4cf7aea820e7-host\") pod \"tuned-lj6lw\" (UID: \"1e1b149f-fe01-49d2-8387-4cf7aea820e7\") " pod="openshift-cluster-node-tuning-operator/tuned-lj6lw" Jul 07 10:27:13.851955 ip-10-0-135-34 kubenswrapper[2578]: I0707 10:27:13.851829 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/b1617485-94f6-4eba-95e9-866fc2400643-sys-fs\") pod \"aws-ebs-csi-driver-node-dll6x\" (UID: \"b1617485-94f6-4eba-95e9-866fc2400643\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-dll6x" Jul 07 10:27:13.851955 ip-10-0-135-34 kubenswrapper[2578]: I0707 10:27:13.851858 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vw57m\" (UniqueName: \"kubernetes.io/projected/b1617485-94f6-4eba-95e9-866fc2400643-kube-api-access-vw57m\") pod \"aws-ebs-csi-driver-node-dll6x\" (UID: \"b1617485-94f6-4eba-95e9-866fc2400643\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-dll6x" Jul 07 10:27:13.851955 ip-10-0-135-34 kubenswrapper[2578]: I0707 10:27:13.851884 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/025a30f8-f923-4bc8-adf5-7ef97ee3bfd3-serviceca\") pod \"node-ca-pb2p8\" (UID: \"025a30f8-f923-4bc8-adf5-7ef97ee3bfd3\") " pod="openshift-image-registry/node-ca-pb2p8" Jul 07 10:27:13.851955 ip-10-0-135-34 kubenswrapper[2578]: I0707 10:27:13.851900 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/1e1b149f-fe01-49d2-8387-4cf7aea820e7-var-lib-kubelet\") pod \"tuned-lj6lw\" (UID: \"1e1b149f-fe01-49d2-8387-4cf7aea820e7\") " pod="openshift-cluster-node-tuning-operator/tuned-lj6lw" Jul 07 10:27:13.851955 ip-10-0-135-34 kubenswrapper[2578]: I0707 10:27:13.851922 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/1e1b149f-fe01-49d2-8387-4cf7aea820e7-etc-tuned\") pod \"tuned-lj6lw\" (UID: \"1e1b149f-fe01-49d2-8387-4cf7aea820e7\") " pod="openshift-cluster-node-tuning-operator/tuned-lj6lw" Jul 07 10:27:13.851955 ip-10-0-135-34 kubenswrapper[2578]: I0707 10:27:13.851945 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/aca7528d-ddbe-4282-979f-1b496011f0e1-var-lib-openvswitch\") pod \"ovnkube-node-qhhw7\" (UID: \"aca7528d-ddbe-4282-979f-1b496011f0e1\") " pod="openshift-ovn-kubernetes/ovnkube-node-qhhw7" Jul 07 10:27:13.852322 ip-10-0-135-34 kubenswrapper[2578]: I0707 10:27:13.851962 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/aca7528d-ddbe-4282-979f-1b496011f0e1-run-openvswitch\") pod \"ovnkube-node-qhhw7\" (UID: \"aca7528d-ddbe-4282-979f-1b496011f0e1\") " pod="openshift-ovn-kubernetes/ovnkube-node-qhhw7" Jul 07 10:27:13.852322 ip-10-0-135-34 kubenswrapper[2578]: I0707 10:27:13.851979 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/aca7528d-ddbe-4282-979f-1b496011f0e1-ovnkube-script-lib\") pod \"ovnkube-node-qhhw7\" (UID: \"aca7528d-ddbe-4282-979f-1b496011f0e1\") " pod="openshift-ovn-kubernetes/ovnkube-node-qhhw7" Jul 07 10:27:13.852322 ip-10-0-135-34 kubenswrapper[2578]: I0707 10:27:13.851995 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-n4rff\" (UniqueName: \"kubernetes.io/projected/aca7528d-ddbe-4282-979f-1b496011f0e1-kube-api-access-n4rff\") pod \"ovnkube-node-qhhw7\" (UID: \"aca7528d-ddbe-4282-979f-1b496011f0e1\") " pod="openshift-ovn-kubernetes/ovnkube-node-qhhw7" Jul 07 10:27:13.852322 ip-10-0-135-34 kubenswrapper[2578]: I0707 10:27:13.852010 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/1e1b149f-fe01-49d2-8387-4cf7aea820e7-etc-sysconfig\") pod \"tuned-lj6lw\" (UID: \"1e1b149f-fe01-49d2-8387-4cf7aea820e7\") " pod="openshift-cluster-node-tuning-operator/tuned-lj6lw" Jul 07 10:27:13.852322 ip-10-0-135-34 kubenswrapper[2578]: I0707 10:27:13.852027 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/1e1b149f-fe01-49d2-8387-4cf7aea820e7-etc-sysctl-conf\") pod \"tuned-lj6lw\" (UID: \"1e1b149f-fe01-49d2-8387-4cf7aea820e7\") " pod="openshift-cluster-node-tuning-operator/tuned-lj6lw" Jul 07 10:27:13.852322 ip-10-0-135-34 kubenswrapper[2578]: I0707 10:27:13.852054 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9vbqq\" (UniqueName: \"kubernetes.io/projected/51fe14a4-6b46-4f88-be7d-b611b45961ff-kube-api-access-9vbqq\") pod \"network-check-target-26sg7\" (UID: \"51fe14a4-6b46-4f88-be7d-b611b45961ff\") " pod="openshift-network-diagnostics/network-check-target-26sg7" Jul 07 10:27:13.852322 ip-10-0-135-34 kubenswrapper[2578]: I0707 10:27:13.852074 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/11079224-e507-45d1-9fbb-da5043002c51-konnectivity-ca\") pod \"konnectivity-agent-t82fl\" (UID: \"11079224-e507-45d1-9fbb-da5043002c51\") " pod="kube-system/konnectivity-agent-t82fl" Jul 07 10:27:13.852322 ip-10-0-135-34 kubenswrapper[2578]: I0707 10:27:13.852101 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/aca7528d-ddbe-4282-979f-1b496011f0e1-run-ovn\") pod \"ovnkube-node-qhhw7\" (UID: \"aca7528d-ddbe-4282-979f-1b496011f0e1\") " pod="openshift-ovn-kubernetes/ovnkube-node-qhhw7" Jul 07 10:27:13.852322 ip-10-0-135-34 kubenswrapper[2578]: I0707 10:27:13.852123 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/aca7528d-ddbe-4282-979f-1b496011f0e1-host-run-ovn-kubernetes\") pod \"ovnkube-node-qhhw7\" (UID: \"aca7528d-ddbe-4282-979f-1b496011f0e1\") " pod="openshift-ovn-kubernetes/ovnkube-node-qhhw7" Jul 07 10:27:13.852322 ip-10-0-135-34 kubenswrapper[2578]: I0707 10:27:13.852146 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/aca7528d-ddbe-4282-979f-1b496011f0e1-host-cni-bin\") pod \"ovnkube-node-qhhw7\" (UID: \"aca7528d-ddbe-4282-979f-1b496011f0e1\") " pod="openshift-ovn-kubernetes/ovnkube-node-qhhw7" Jul 07 10:27:13.852322 ip-10-0-135-34 kubenswrapper[2578]: I0707 10:27:13.852162 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/aca7528d-ddbe-4282-979f-1b496011f0e1-ovn-node-metrics-cert\") pod \"ovnkube-node-qhhw7\" (UID: \"aca7528d-ddbe-4282-979f-1b496011f0e1\") " pod="openshift-ovn-kubernetes/ovnkube-node-qhhw7" Jul 07 10:27:13.852322 ip-10-0-135-34 kubenswrapper[2578]: I0707 10:27:13.852179 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/2410fb3f-6b21-40b9-b23b-ffdbbdb132a7-hostroot\") pod \"multus-hwz82\" (UID: \"2410fb3f-6b21-40b9-b23b-ffdbbdb132a7\") " pod="openshift-multus/multus-hwz82" Jul 07 10:27:13.852322 ip-10-0-135-34 kubenswrapper[2578]: I0707 10:27:13.852198 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/2410fb3f-6b21-40b9-b23b-ffdbbdb132a7-host-run-multus-certs\") pod \"multus-hwz82\" (UID: \"2410fb3f-6b21-40b9-b23b-ffdbbdb132a7\") " pod="openshift-multus/multus-hwz82" Jul 07 10:27:13.852322 ip-10-0-135-34 kubenswrapper[2578]: I0707 10:27:13.852214 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/1e1b149f-fe01-49d2-8387-4cf7aea820e7-run\") pod \"tuned-lj6lw\" (UID: \"1e1b149f-fe01-49d2-8387-4cf7aea820e7\") " pod="openshift-cluster-node-tuning-operator/tuned-lj6lw" Jul 07 10:27:13.852322 ip-10-0-135-34 kubenswrapper[2578]: I0707 10:27:13.852233 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/b1617485-94f6-4eba-95e9-866fc2400643-registration-dir\") pod \"aws-ebs-csi-driver-node-dll6x\" (UID: \"b1617485-94f6-4eba-95e9-866fc2400643\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-dll6x" Jul 07 10:27:13.852322 ip-10-0-135-34 kubenswrapper[2578]: I0707 10:27:13.852257 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/aca7528d-ddbe-4282-979f-1b496011f0e1-host-cni-netd\") pod \"ovnkube-node-qhhw7\" (UID: \"aca7528d-ddbe-4282-979f-1b496011f0e1\") " pod="openshift-ovn-kubernetes/ovnkube-node-qhhw7" Jul 07 10:27:13.852322 ip-10-0-135-34 kubenswrapper[2578]: I0707 10:27:13.852282 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/aca7528d-ddbe-4282-979f-1b496011f0e1-ovnkube-config\") pod \"ovnkube-node-qhhw7\" (UID: \"aca7528d-ddbe-4282-979f-1b496011f0e1\") " pod="openshift-ovn-kubernetes/ovnkube-node-qhhw7" Jul 07 10:27:13.852941 ip-10-0-135-34 kubenswrapper[2578]: I0707 10:27:13.852306 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/aca7528d-ddbe-4282-979f-1b496011f0e1-env-overrides\") pod \"ovnkube-node-qhhw7\" (UID: \"aca7528d-ddbe-4282-979f-1b496011f0e1\") " pod="openshift-ovn-kubernetes/ovnkube-node-qhhw7" Jul 07 10:27:13.852941 ip-10-0-135-34 kubenswrapper[2578]: I0707 10:27:13.852332 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/2410fb3f-6b21-40b9-b23b-ffdbbdb132a7-multus-cni-dir\") pod \"multus-hwz82\" (UID: \"2410fb3f-6b21-40b9-b23b-ffdbbdb132a7\") " pod="openshift-multus/multus-hwz82" Jul 07 10:27:13.852941 ip-10-0-135-34 kubenswrapper[2578]: I0707 10:27:13.852360 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/2410fb3f-6b21-40b9-b23b-ffdbbdb132a7-multus-daemon-config\") pod \"multus-hwz82\" (UID: \"2410fb3f-6b21-40b9-b23b-ffdbbdb132a7\") " pod="openshift-multus/multus-hwz82" Jul 07 10:27:13.852941 ip-10-0-135-34 kubenswrapper[2578]: I0707 10:27:13.852384 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/1e1b149f-fe01-49d2-8387-4cf7aea820e7-etc-kubernetes\") pod \"tuned-lj6lw\" (UID: \"1e1b149f-fe01-49d2-8387-4cf7aea820e7\") " pod="openshift-cluster-node-tuning-operator/tuned-lj6lw" Jul 07 10:27:13.852941 ip-10-0-135-34 kubenswrapper[2578]: I0707 10:27:13.852410 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/2410fb3f-6b21-40b9-b23b-ffdbbdb132a7-cni-binary-copy\") pod \"multus-hwz82\" (UID: \"2410fb3f-6b21-40b9-b23b-ffdbbdb132a7\") " pod="openshift-multus/multus-hwz82" Jul 07 10:27:13.852941 ip-10-0-135-34 kubenswrapper[2578]: I0707 10:27:13.852437 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/f85fac08-720e-4ffd-80af-19c140c36804-original-pull-secret\") pod \"global-pull-secret-syncer-7lv79\" (UID: \"f85fac08-720e-4ffd-80af-19c140c36804\") " pod="kube-system/global-pull-secret-syncer-7lv79" Jul 07 10:27:13.852941 ip-10-0-135-34 kubenswrapper[2578]: I0707 10:27:13.852466 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/aca7528d-ddbe-4282-979f-1b496011f0e1-host-run-netns\") pod \"ovnkube-node-qhhw7\" (UID: \"aca7528d-ddbe-4282-979f-1b496011f0e1\") " pod="openshift-ovn-kubernetes/ovnkube-node-qhhw7" Jul 07 10:27:13.852941 ip-10-0-135-34 kubenswrapper[2578]: I0707 10:27:13.852498 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/a415d240-ff8a-4b08-aeb6-b4018195c46b-cni-binary-copy\") pod \"multus-additional-cni-plugins-l8d7k\" (UID: \"a415d240-ff8a-4b08-aeb6-b4018195c46b\") " pod="openshift-multus/multus-additional-cni-plugins-l8d7k" Jul 07 10:27:13.852941 ip-10-0-135-34 kubenswrapper[2578]: I0707 10:27:13.852527 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/609c4941-f311-408a-82ec-1d6b1a54572d-host-slash\") pod \"iptables-alerter-lr7dk\" (UID: \"609c4941-f311-408a-82ec-1d6b1a54572d\") " pod="openshift-network-operator/iptables-alerter-lr7dk" Jul 07 10:27:13.852941 ip-10-0-135-34 kubenswrapper[2578]: I0707 10:27:13.852553 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/aca7528d-ddbe-4282-979f-1b496011f0e1-host-kubelet\") pod \"ovnkube-node-qhhw7\" (UID: \"aca7528d-ddbe-4282-979f-1b496011f0e1\") " pod="openshift-ovn-kubernetes/ovnkube-node-qhhw7" Jul 07 10:27:13.852941 ip-10-0-135-34 kubenswrapper[2578]: I0707 10:27:13.852570 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/1e1b149f-fe01-49d2-8387-4cf7aea820e7-etc-systemd\") pod \"tuned-lj6lw\" (UID: \"1e1b149f-fe01-49d2-8387-4cf7aea820e7\") " pod="openshift-cluster-node-tuning-operator/tuned-lj6lw" Jul 07 10:27:13.852941 ip-10-0-135-34 kubenswrapper[2578]: I0707 10:27:13.852588 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/b1617485-94f6-4eba-95e9-866fc2400643-device-dir\") pod \"aws-ebs-csi-driver-node-dll6x\" (UID: \"b1617485-94f6-4eba-95e9-866fc2400643\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-dll6x" Jul 07 10:27:13.852941 ip-10-0-135-34 kubenswrapper[2578]: I0707 10:27:13.852603 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/aca7528d-ddbe-4282-979f-1b496011f0e1-run-systemd\") pod \"ovnkube-node-qhhw7\" (UID: \"aca7528d-ddbe-4282-979f-1b496011f0e1\") " pod="openshift-ovn-kubernetes/ovnkube-node-qhhw7" Jul 07 10:27:13.852941 ip-10-0-135-34 kubenswrapper[2578]: I0707 10:27:13.852619 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/a415d240-ff8a-4b08-aeb6-b4018195c46b-tuning-conf-dir\") pod \"multus-additional-cni-plugins-l8d7k\" (UID: \"a415d240-ff8a-4b08-aeb6-b4018195c46b\") " pod="openshift-multus/multus-additional-cni-plugins-l8d7k" Jul 07 10:27:13.852941 ip-10-0-135-34 kubenswrapper[2578]: I0707 10:27:13.852635 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/2410fb3f-6b21-40b9-b23b-ffdbbdb132a7-host-var-lib-cni-bin\") pod \"multus-hwz82\" (UID: \"2410fb3f-6b21-40b9-b23b-ffdbbdb132a7\") " pod="openshift-multus/multus-hwz82" Jul 07 10:27:13.852941 ip-10-0-135-34 kubenswrapper[2578]: I0707 10:27:13.852652 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/2410fb3f-6b21-40b9-b23b-ffdbbdb132a7-host-var-lib-kubelet\") pod \"multus-hwz82\" (UID: \"2410fb3f-6b21-40b9-b23b-ffdbbdb132a7\") " pod="openshift-multus/multus-hwz82" Jul 07 10:27:13.852941 ip-10-0-135-34 kubenswrapper[2578]: I0707 10:27:13.852669 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/931a7d75-9ee8-4758-9f23-3521f3db5eae-metrics-certs\") pod \"network-metrics-daemon-p2ks2\" (UID: \"931a7d75-9ee8-4758-9f23-3521f3db5eae\") " pod="openshift-multus/network-metrics-daemon-p2ks2" Jul 07 10:27:13.853727 ip-10-0-135-34 kubenswrapper[2578]: I0707 10:27:13.852695 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/1e1b149f-fe01-49d2-8387-4cf7aea820e7-etc-sysctl-d\") pod \"tuned-lj6lw\" (UID: \"1e1b149f-fe01-49d2-8387-4cf7aea820e7\") " pod="openshift-cluster-node-tuning-operator/tuned-lj6lw" Jul 07 10:27:13.853727 ip-10-0-135-34 kubenswrapper[2578]: I0707 10:27:13.852714 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wbblv\" (UniqueName: \"kubernetes.io/projected/dfa64e14-3729-4f28-870f-eeab6cec9944-kube-api-access-wbblv\") pod \"node-resolver-rdrnq\" (UID: \"dfa64e14-3729-4f28-870f-eeab6cec9944\") " pod="openshift-dns/node-resolver-rdrnq" Jul 07 10:27:13.853727 ip-10-0-135-34 kubenswrapper[2578]: I0707 10:27:13.852732 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/b1617485-94f6-4eba-95e9-866fc2400643-etc-selinux\") pod \"aws-ebs-csi-driver-node-dll6x\" (UID: \"b1617485-94f6-4eba-95e9-866fc2400643\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-dll6x" Jul 07 10:27:13.853727 ip-10-0-135-34 kubenswrapper[2578]: I0707 10:27:13.852747 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/aca7528d-ddbe-4282-979f-1b496011f0e1-log-socket\") pod \"ovnkube-node-qhhw7\" (UID: \"aca7528d-ddbe-4282-979f-1b496011f0e1\") " pod="openshift-ovn-kubernetes/ovnkube-node-qhhw7" Jul 07 10:27:13.853727 ip-10-0-135-34 kubenswrapper[2578]: I0707 10:27:13.852767 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/1e1b149f-fe01-49d2-8387-4cf7aea820e7-sys\") pod \"tuned-lj6lw\" (UID: \"1e1b149f-fe01-49d2-8387-4cf7aea820e7\") " pod="openshift-cluster-node-tuning-operator/tuned-lj6lw" Jul 07 10:27:13.853727 ip-10-0-135-34 kubenswrapper[2578]: I0707 10:27:13.852794 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/1e1b149f-fe01-49d2-8387-4cf7aea820e7-lib-modules\") pod \"tuned-lj6lw\" (UID: \"1e1b149f-fe01-49d2-8387-4cf7aea820e7\") " pod="openshift-cluster-node-tuning-operator/tuned-lj6lw" Jul 07 10:27:13.853727 ip-10-0-135-34 kubenswrapper[2578]: I0707 10:27:13.852810 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/b1617485-94f6-4eba-95e9-866fc2400643-socket-dir\") pod \"aws-ebs-csi-driver-node-dll6x\" (UID: \"b1617485-94f6-4eba-95e9-866fc2400643\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-dll6x" Jul 07 10:27:13.853727 ip-10-0-135-34 kubenswrapper[2578]: I0707 10:27:13.852827 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/2410fb3f-6b21-40b9-b23b-ffdbbdb132a7-cnibin\") pod \"multus-hwz82\" (UID: \"2410fb3f-6b21-40b9-b23b-ffdbbdb132a7\") " pod="openshift-multus/multus-hwz82" Jul 07 10:27:13.853727 ip-10-0-135-34 kubenswrapper[2578]: I0707 10:27:13.852868 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/2410fb3f-6b21-40b9-b23b-ffdbbdb132a7-multus-socket-dir-parent\") pod \"multus-hwz82\" (UID: \"2410fb3f-6b21-40b9-b23b-ffdbbdb132a7\") " pod="openshift-multus/multus-hwz82" Jul 07 10:27:13.853727 ip-10-0-135-34 kubenswrapper[2578]: I0707 10:27:13.852888 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/dfa64e14-3729-4f28-870f-eeab6cec9944-hosts-file\") pod \"node-resolver-rdrnq\" (UID: \"dfa64e14-3729-4f28-870f-eeab6cec9944\") " pod="openshift-dns/node-resolver-rdrnq" Jul 07 10:27:13.853727 ip-10-0-135-34 kubenswrapper[2578]: I0707 10:27:13.852910 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/dfa64e14-3729-4f28-870f-eeab6cec9944-tmp-dir\") pod \"node-resolver-rdrnq\" (UID: \"dfa64e14-3729-4f28-870f-eeab6cec9944\") " pod="openshift-dns/node-resolver-rdrnq" Jul 07 10:27:13.853727 ip-10-0-135-34 kubenswrapper[2578]: I0707 10:27:13.852928 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gx8bz\" (UniqueName: \"kubernetes.io/projected/025a30f8-f923-4bc8-adf5-7ef97ee3bfd3-kube-api-access-gx8bz\") pod \"node-ca-pb2p8\" (UID: \"025a30f8-f923-4bc8-adf5-7ef97ee3bfd3\") " pod="openshift-image-registry/node-ca-pb2p8" Jul 07 10:27:13.853727 ip-10-0-135-34 kubenswrapper[2578]: I0707 10:27:13.852951 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/aca7528d-ddbe-4282-979f-1b496011f0e1-node-log\") pod \"ovnkube-node-qhhw7\" (UID: \"aca7528d-ddbe-4282-979f-1b496011f0e1\") " pod="openshift-ovn-kubernetes/ovnkube-node-qhhw7" Jul 07 10:27:13.853727 ip-10-0-135-34 kubenswrapper[2578]: I0707 10:27:13.852976 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9n5dz\" (UniqueName: \"kubernetes.io/projected/a415d240-ff8a-4b08-aeb6-b4018195c46b-kube-api-access-9n5dz\") pod \"multus-additional-cni-plugins-l8d7k\" (UID: \"a415d240-ff8a-4b08-aeb6-b4018195c46b\") " pod="openshift-multus/multus-additional-cni-plugins-l8d7k" Jul 07 10:27:13.853727 ip-10-0-135-34 kubenswrapper[2578]: I0707 10:27:13.852995 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ncrl8\" (UniqueName: \"kubernetes.io/projected/931a7d75-9ee8-4758-9f23-3521f3db5eae-kube-api-access-ncrl8\") pod \"network-metrics-daemon-p2ks2\" (UID: \"931a7d75-9ee8-4758-9f23-3521f3db5eae\") " pod="openshift-multus/network-metrics-daemon-p2ks2" Jul 07 10:27:13.853727 ip-10-0-135-34 kubenswrapper[2578]: I0707 10:27:13.853012 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/1e1b149f-fe01-49d2-8387-4cf7aea820e7-tmp\") pod \"tuned-lj6lw\" (UID: \"1e1b149f-fe01-49d2-8387-4cf7aea820e7\") " pod="openshift-cluster-node-tuning-operator/tuned-lj6lw" Jul 07 10:27:13.853727 ip-10-0-135-34 kubenswrapper[2578]: I0707 10:27:13.853030 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/b1617485-94f6-4eba-95e9-866fc2400643-kubelet-dir\") pod \"aws-ebs-csi-driver-node-dll6x\" (UID: \"b1617485-94f6-4eba-95e9-866fc2400643\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-dll6x" Jul 07 10:27:13.854456 ip-10-0-135-34 kubenswrapper[2578]: I0707 10:27:13.853057 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/025a30f8-f923-4bc8-adf5-7ef97ee3bfd3-host\") pod \"node-ca-pb2p8\" (UID: \"025a30f8-f923-4bc8-adf5-7ef97ee3bfd3\") " pod="openshift-image-registry/node-ca-pb2p8" Jul 07 10:27:13.854456 ip-10-0-135-34 kubenswrapper[2578]: I0707 10:27:13.853076 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/f85fac08-720e-4ffd-80af-19c140c36804-kubelet-config\") pod \"global-pull-secret-syncer-7lv79\" (UID: \"f85fac08-720e-4ffd-80af-19c140c36804\") " pod="kube-system/global-pull-secret-syncer-7lv79" Jul 07 10:27:13.854456 ip-10-0-135-34 kubenswrapper[2578]: I0707 10:27:13.853100 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/a415d240-ff8a-4b08-aeb6-b4018195c46b-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-l8d7k\" (UID: \"a415d240-ff8a-4b08-aeb6-b4018195c46b\") " pod="openshift-multus/multus-additional-cni-plugins-l8d7k" Jul 07 10:27:13.854456 ip-10-0-135-34 kubenswrapper[2578]: I0707 10:27:13.853123 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/2410fb3f-6b21-40b9-b23b-ffdbbdb132a7-host-var-lib-cni-multus\") pod \"multus-hwz82\" (UID: \"2410fb3f-6b21-40b9-b23b-ffdbbdb132a7\") " pod="openshift-multus/multus-hwz82" Jul 07 10:27:13.854456 ip-10-0-135-34 kubenswrapper[2578]: I0707 10:27:13.853147 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f8c4t\" (UniqueName: \"kubernetes.io/projected/2410fb3f-6b21-40b9-b23b-ffdbbdb132a7-kube-api-access-f8c4t\") pod \"multus-hwz82\" (UID: \"2410fb3f-6b21-40b9-b23b-ffdbbdb132a7\") " pod="openshift-multus/multus-hwz82" Jul 07 10:27:13.854456 ip-10-0-135-34 kubenswrapper[2578]: I0707 10:27:13.853164 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/aca7528d-ddbe-4282-979f-1b496011f0e1-host-slash\") pod \"ovnkube-node-qhhw7\" (UID: \"aca7528d-ddbe-4282-979f-1b496011f0e1\") " pod="openshift-ovn-kubernetes/ovnkube-node-qhhw7" Jul 07 10:27:13.854456 ip-10-0-135-34 kubenswrapper[2578]: I0707 10:27:13.853185 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/a415d240-ff8a-4b08-aeb6-b4018195c46b-system-cni-dir\") pod \"multus-additional-cni-plugins-l8d7k\" (UID: \"a415d240-ff8a-4b08-aeb6-b4018195c46b\") " pod="openshift-multus/multus-additional-cni-plugins-l8d7k" Jul 07 10:27:13.854456 ip-10-0-135-34 kubenswrapper[2578]: I0707 10:27:13.853203 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/2410fb3f-6b21-40b9-b23b-ffdbbdb132a7-multus-conf-dir\") pod \"multus-hwz82\" (UID: \"2410fb3f-6b21-40b9-b23b-ffdbbdb132a7\") " pod="openshift-multus/multus-hwz82" Jul 07 10:27:13.854456 ip-10-0-135-34 kubenswrapper[2578]: I0707 10:27:13.853227 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/2410fb3f-6b21-40b9-b23b-ffdbbdb132a7-etc-kubernetes\") pod \"multus-hwz82\" (UID: \"2410fb3f-6b21-40b9-b23b-ffdbbdb132a7\") " pod="openshift-multus/multus-hwz82" Jul 07 10:27:13.854456 ip-10-0-135-34 kubenswrapper[2578]: I0707 10:27:13.853247 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bw8ks\" (UniqueName: \"kubernetes.io/projected/1e1b149f-fe01-49d2-8387-4cf7aea820e7-kube-api-access-bw8ks\") pod \"tuned-lj6lw\" (UID: \"1e1b149f-fe01-49d2-8387-4cf7aea820e7\") " pod="openshift-cluster-node-tuning-operator/tuned-lj6lw" Jul 07 10:27:13.854456 ip-10-0-135-34 kubenswrapper[2578]: I0707 10:27:13.853264 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/f85fac08-720e-4ffd-80af-19c140c36804-dbus\") pod \"global-pull-secret-syncer-7lv79\" (UID: \"f85fac08-720e-4ffd-80af-19c140c36804\") " pod="kube-system/global-pull-secret-syncer-7lv79" Jul 07 10:27:13.854456 ip-10-0-135-34 kubenswrapper[2578]: I0707 10:27:13.853283 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/a415d240-ff8a-4b08-aeb6-b4018195c46b-cnibin\") pod \"multus-additional-cni-plugins-l8d7k\" (UID: \"a415d240-ff8a-4b08-aeb6-b4018195c46b\") " pod="openshift-multus/multus-additional-cni-plugins-l8d7k" Jul 07 10:27:13.854456 ip-10-0-135-34 kubenswrapper[2578]: I0707 10:27:13.853304 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/609c4941-f311-408a-82ec-1d6b1a54572d-iptables-alerter-script\") pod \"iptables-alerter-lr7dk\" (UID: \"609c4941-f311-408a-82ec-1d6b1a54572d\") " pod="openshift-network-operator/iptables-alerter-lr7dk" Jul 07 10:27:13.854456 ip-10-0-135-34 kubenswrapper[2578]: I0707 10:27:13.853326 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/11079224-e507-45d1-9fbb-da5043002c51-agent-certs\") pod \"konnectivity-agent-t82fl\" (UID: \"11079224-e507-45d1-9fbb-da5043002c51\") " pod="kube-system/konnectivity-agent-t82fl" Jul 07 10:27:13.854456 ip-10-0-135-34 kubenswrapper[2578]: I0707 10:27:13.853343 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/aca7528d-ddbe-4282-979f-1b496011f0e1-systemd-units\") pod \"ovnkube-node-qhhw7\" (UID: \"aca7528d-ddbe-4282-979f-1b496011f0e1\") " pod="openshift-ovn-kubernetes/ovnkube-node-qhhw7" Jul 07 10:27:13.854456 ip-10-0-135-34 kubenswrapper[2578]: I0707 10:27:13.853361 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/aca7528d-ddbe-4282-979f-1b496011f0e1-etc-openvswitch\") pod \"ovnkube-node-qhhw7\" (UID: \"aca7528d-ddbe-4282-979f-1b496011f0e1\") " pod="openshift-ovn-kubernetes/ovnkube-node-qhhw7" Jul 07 10:27:13.854456 ip-10-0-135-34 kubenswrapper[2578]: I0707 10:27:13.853393 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/a415d240-ff8a-4b08-aeb6-b4018195c46b-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-l8d7k\" (UID: \"a415d240-ff8a-4b08-aeb6-b4018195c46b\") " pod="openshift-multus/multus-additional-cni-plugins-l8d7k" Jul 07 10:27:13.855062 ip-10-0-135-34 kubenswrapper[2578]: I0707 10:27:13.853418 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/2410fb3f-6b21-40b9-b23b-ffdbbdb132a7-system-cni-dir\") pod \"multus-hwz82\" (UID: \"2410fb3f-6b21-40b9-b23b-ffdbbdb132a7\") " pod="openshift-multus/multus-hwz82" Jul 07 10:27:13.855062 ip-10-0-135-34 kubenswrapper[2578]: I0707 10:27:13.853439 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/2410fb3f-6b21-40b9-b23b-ffdbbdb132a7-os-release\") pod \"multus-hwz82\" (UID: \"2410fb3f-6b21-40b9-b23b-ffdbbdb132a7\") " pod="openshift-multus/multus-hwz82" Jul 07 10:27:13.855062 ip-10-0-135-34 kubenswrapper[2578]: I0707 10:27:13.853459 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/2410fb3f-6b21-40b9-b23b-ffdbbdb132a7-host-run-netns\") pod \"multus-hwz82\" (UID: \"2410fb3f-6b21-40b9-b23b-ffdbbdb132a7\") " pod="openshift-multus/multus-hwz82" Jul 07 10:27:13.855062 ip-10-0-135-34 kubenswrapper[2578]: I0707 10:27:13.853479 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/aca7528d-ddbe-4282-979f-1b496011f0e1-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-qhhw7\" (UID: \"aca7528d-ddbe-4282-979f-1b496011f0e1\") " pod="openshift-ovn-kubernetes/ovnkube-node-qhhw7" Jul 07 10:27:13.855062 ip-10-0-135-34 kubenswrapper[2578]: I0707 10:27:13.853612 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/a415d240-ff8a-4b08-aeb6-b4018195c46b-os-release\") pod \"multus-additional-cni-plugins-l8d7k\" (UID: \"a415d240-ff8a-4b08-aeb6-b4018195c46b\") " pod="openshift-multus/multus-additional-cni-plugins-l8d7k" Jul 07 10:27:13.855062 ip-10-0-135-34 kubenswrapper[2578]: I0707 10:27:13.853644 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/2410fb3f-6b21-40b9-b23b-ffdbbdb132a7-host-run-k8s-cni-cncf-io\") pod \"multus-hwz82\" (UID: \"2410fb3f-6b21-40b9-b23b-ffdbbdb132a7\") " pod="openshift-multus/multus-hwz82" Jul 07 10:27:13.855062 ip-10-0-135-34 kubenswrapper[2578]: I0707 10:27:13.854028 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/1e1b149f-fe01-49d2-8387-4cf7aea820e7-etc-modprobe-d\") pod \"tuned-lj6lw\" (UID: \"1e1b149f-fe01-49d2-8387-4cf7aea820e7\") " pod="openshift-cluster-node-tuning-operator/tuned-lj6lw" Jul 07 10:27:13.855062 ip-10-0-135-34 kubenswrapper[2578]: I0707 10:27:13.854086 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/1e1b149f-fe01-49d2-8387-4cf7aea820e7-host\") pod \"tuned-lj6lw\" (UID: \"1e1b149f-fe01-49d2-8387-4cf7aea820e7\") " pod="openshift-cluster-node-tuning-operator/tuned-lj6lw" Jul 07 10:27:13.855062 ip-10-0-135-34 kubenswrapper[2578]: I0707 10:27:13.854126 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/b1617485-94f6-4eba-95e9-866fc2400643-sys-fs\") pod \"aws-ebs-csi-driver-node-dll6x\" (UID: \"b1617485-94f6-4eba-95e9-866fc2400643\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-dll6x" Jul 07 10:27:13.855062 ip-10-0-135-34 kubenswrapper[2578]: I0707 10:27:13.854337 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/1e1b149f-fe01-49d2-8387-4cf7aea820e7-var-lib-kubelet\") pod \"tuned-lj6lw\" (UID: \"1e1b149f-fe01-49d2-8387-4cf7aea820e7\") " pod="openshift-cluster-node-tuning-operator/tuned-lj6lw" Jul 07 10:27:13.855062 ip-10-0-135-34 kubenswrapper[2578]: I0707 10:27:13.854609 2578 swap_util.go:74] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory" Jul 07 10:27:13.855062 ip-10-0-135-34 kubenswrapper[2578]: I0707 10:27:13.854679 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/1e1b149f-fe01-49d2-8387-4cf7aea820e7-etc-sysctl-d\") pod \"tuned-lj6lw\" (UID: \"1e1b149f-fe01-49d2-8387-4cf7aea820e7\") " pod="openshift-cluster-node-tuning-operator/tuned-lj6lw" Jul 07 10:27:13.855062 ip-10-0-135-34 kubenswrapper[2578]: I0707 10:27:13.854751 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/1e1b149f-fe01-49d2-8387-4cf7aea820e7-etc-sysconfig\") pod \"tuned-lj6lw\" (UID: \"1e1b149f-fe01-49d2-8387-4cf7aea820e7\") " pod="openshift-cluster-node-tuning-operator/tuned-lj6lw" Jul 07 10:27:13.855062 ip-10-0-135-34 kubenswrapper[2578]: I0707 10:27:13.854888 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/1e1b149f-fe01-49d2-8387-4cf7aea820e7-etc-sysctl-conf\") pod \"tuned-lj6lw\" (UID: \"1e1b149f-fe01-49d2-8387-4cf7aea820e7\") " pod="openshift-cluster-node-tuning-operator/tuned-lj6lw" Jul 07 10:27:13.855695 ip-10-0-135-34 kubenswrapper[2578]: I0707 10:27:13.855162 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/2410fb3f-6b21-40b9-b23b-ffdbbdb132a7-hostroot\") pod \"multus-hwz82\" (UID: \"2410fb3f-6b21-40b9-b23b-ffdbbdb132a7\") " pod="openshift-multus/multus-hwz82" Jul 07 10:27:13.855695 ip-10-0-135-34 kubenswrapper[2578]: I0707 10:27:13.855204 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/2410fb3f-6b21-40b9-b23b-ffdbbdb132a7-host-run-multus-certs\") pod \"multus-hwz82\" (UID: \"2410fb3f-6b21-40b9-b23b-ffdbbdb132a7\") " pod="openshift-multus/multus-hwz82" Jul 07 10:27:13.855695 ip-10-0-135-34 kubenswrapper[2578]: I0707 10:27:13.855243 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/1e1b149f-fe01-49d2-8387-4cf7aea820e7-run\") pod \"tuned-lj6lw\" (UID: \"1e1b149f-fe01-49d2-8387-4cf7aea820e7\") " pod="openshift-cluster-node-tuning-operator/tuned-lj6lw" Jul 07 10:27:13.855695 ip-10-0-135-34 kubenswrapper[2578]: I0707 10:27:13.855286 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/b1617485-94f6-4eba-95e9-866fc2400643-registration-dir\") pod \"aws-ebs-csi-driver-node-dll6x\" (UID: \"b1617485-94f6-4eba-95e9-866fc2400643\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-dll6x" Jul 07 10:27:13.855695 ip-10-0-135-34 kubenswrapper[2578]: I0707 10:27:13.855371 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/b1617485-94f6-4eba-95e9-866fc2400643-etc-selinux\") pod \"aws-ebs-csi-driver-node-dll6x\" (UID: \"b1617485-94f6-4eba-95e9-866fc2400643\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-dll6x" Jul 07 10:27:13.855695 ip-10-0-135-34 kubenswrapper[2578]: I0707 10:27:13.855460 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/1e1b149f-fe01-49d2-8387-4cf7aea820e7-sys\") pod \"tuned-lj6lw\" (UID: \"1e1b149f-fe01-49d2-8387-4cf7aea820e7\") " pod="openshift-cluster-node-tuning-operator/tuned-lj6lw" Jul 07 10:27:13.855695 ip-10-0-135-34 kubenswrapper[2578]: I0707 10:27:13.855492 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/2410fb3f-6b21-40b9-b23b-ffdbbdb132a7-multus-cni-dir\") pod \"multus-hwz82\" (UID: \"2410fb3f-6b21-40b9-b23b-ffdbbdb132a7\") " pod="openshift-multus/multus-hwz82" Jul 07 10:27:13.855695 ip-10-0-135-34 kubenswrapper[2578]: I0707 10:27:13.855551 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/1e1b149f-fe01-49d2-8387-4cf7aea820e7-lib-modules\") pod \"tuned-lj6lw\" (UID: \"1e1b149f-fe01-49d2-8387-4cf7aea820e7\") " pod="openshift-cluster-node-tuning-operator/tuned-lj6lw" Jul 07 10:27:13.855695 ip-10-0-135-34 kubenswrapper[2578]: I0707 10:27:13.855678 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/b1617485-94f6-4eba-95e9-866fc2400643-socket-dir\") pod \"aws-ebs-csi-driver-node-dll6x\" (UID: \"b1617485-94f6-4eba-95e9-866fc2400643\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-dll6x" Jul 07 10:27:13.856171 ip-10-0-135-34 kubenswrapper[2578]: I0707 10:27:13.855754 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/2410fb3f-6b21-40b9-b23b-ffdbbdb132a7-cnibin\") pod \"multus-hwz82\" (UID: \"2410fb3f-6b21-40b9-b23b-ffdbbdb132a7\") " pod="openshift-multus/multus-hwz82" Jul 07 10:27:13.856171 ip-10-0-135-34 kubenswrapper[2578]: I0707 10:27:13.855831 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/2410fb3f-6b21-40b9-b23b-ffdbbdb132a7-multus-socket-dir-parent\") pod \"multus-hwz82\" (UID: \"2410fb3f-6b21-40b9-b23b-ffdbbdb132a7\") " pod="openshift-multus/multus-hwz82" Jul 07 10:27:13.856171 ip-10-0-135-34 kubenswrapper[2578]: I0707 10:27:13.855890 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/dfa64e14-3729-4f28-870f-eeab6cec9944-hosts-file\") pod \"node-resolver-rdrnq\" (UID: \"dfa64e14-3729-4f28-870f-eeab6cec9944\") " pod="openshift-dns/node-resolver-rdrnq" Jul 07 10:27:13.856171 ip-10-0-135-34 kubenswrapper[2578]: I0707 10:27:13.856121 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/2410fb3f-6b21-40b9-b23b-ffdbbdb132a7-multus-daemon-config\") pod \"multus-hwz82\" (UID: \"2410fb3f-6b21-40b9-b23b-ffdbbdb132a7\") " pod="openshift-multus/multus-hwz82" Jul 07 10:27:13.856436 ip-10-0-135-34 kubenswrapper[2578]: I0707 10:27:13.856185 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/dfa64e14-3729-4f28-870f-eeab6cec9944-tmp-dir\") pod \"node-resolver-rdrnq\" (UID: \"dfa64e14-3729-4f28-870f-eeab6cec9944\") " pod="openshift-dns/node-resolver-rdrnq" Jul 07 10:27:13.856436 ip-10-0-135-34 kubenswrapper[2578]: I0707 10:27:13.856199 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/1e1b149f-fe01-49d2-8387-4cf7aea820e7-etc-kubernetes\") pod \"tuned-lj6lw\" (UID: \"1e1b149f-fe01-49d2-8387-4cf7aea820e7\") " pod="openshift-cluster-node-tuning-operator/tuned-lj6lw" Jul 07 10:27:13.857826 ip-10-0-135-34 kubenswrapper[2578]: I0707 10:27:13.856605 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/2410fb3f-6b21-40b9-b23b-ffdbbdb132a7-etc-kubernetes\") pod \"multus-hwz82\" (UID: \"2410fb3f-6b21-40b9-b23b-ffdbbdb132a7\") " pod="openshift-multus/multus-hwz82" Jul 07 10:27:13.857826 ip-10-0-135-34 kubenswrapper[2578]: I0707 10:27:13.856631 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/2410fb3f-6b21-40b9-b23b-ffdbbdb132a7-cni-binary-copy\") pod \"multus-hwz82\" (UID: \"2410fb3f-6b21-40b9-b23b-ffdbbdb132a7\") " pod="openshift-multus/multus-hwz82" Jul 07 10:27:13.857826 ip-10-0-135-34 kubenswrapper[2578]: I0707 10:27:13.856723 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/2410fb3f-6b21-40b9-b23b-ffdbbdb132a7-multus-conf-dir\") pod \"multus-hwz82\" (UID: \"2410fb3f-6b21-40b9-b23b-ffdbbdb132a7\") " pod="openshift-multus/multus-hwz82" Jul 07 10:27:13.857826 ip-10-0-135-34 kubenswrapper[2578]: I0707 10:27:13.857330 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/a415d240-ff8a-4b08-aeb6-b4018195c46b-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-l8d7k\" (UID: \"a415d240-ff8a-4b08-aeb6-b4018195c46b\") " pod="openshift-multus/multus-additional-cni-plugins-l8d7k" Jul 07 10:27:13.857826 ip-10-0-135-34 kubenswrapper[2578]: I0707 10:27:13.857376 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/2410fb3f-6b21-40b9-b23b-ffdbbdb132a7-host-var-lib-cni-bin\") pod \"multus-hwz82\" (UID: \"2410fb3f-6b21-40b9-b23b-ffdbbdb132a7\") " pod="openshift-multus/multus-hwz82" Jul 07 10:27:13.857826 ip-10-0-135-34 kubenswrapper[2578]: I0707 10:27:13.857402 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/1e1b149f-fe01-49d2-8387-4cf7aea820e7-etc-systemd\") pod \"tuned-lj6lw\" (UID: \"1e1b149f-fe01-49d2-8387-4cf7aea820e7\") " pod="openshift-cluster-node-tuning-operator/tuned-lj6lw" Jul 07 10:27:13.857826 ip-10-0-135-34 kubenswrapper[2578]: I0707 10:27:13.857412 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/2410fb3f-6b21-40b9-b23b-ffdbbdb132a7-os-release\") pod \"multus-hwz82\" (UID: \"2410fb3f-6b21-40b9-b23b-ffdbbdb132a7\") " pod="openshift-multus/multus-hwz82" Jul 07 10:27:13.857826 ip-10-0-135-34 kubenswrapper[2578]: I0707 10:27:13.857421 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/609c4941-f311-408a-82ec-1d6b1a54572d-host-slash\") pod \"iptables-alerter-lr7dk\" (UID: \"609c4941-f311-408a-82ec-1d6b1a54572d\") " pod="openshift-network-operator/iptables-alerter-lr7dk" Jul 07 10:27:13.857826 ip-10-0-135-34 kubenswrapper[2578]: I0707 10:27:13.857434 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/a415d240-ff8a-4b08-aeb6-b4018195c46b-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-l8d7k\" (UID: \"a415d240-ff8a-4b08-aeb6-b4018195c46b\") " pod="openshift-multus/multus-additional-cni-plugins-l8d7k" Jul 07 10:27:13.857826 ip-10-0-135-34 kubenswrapper[2578]: I0707 10:27:13.857438 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/2410fb3f-6b21-40b9-b23b-ffdbbdb132a7-host-var-lib-kubelet\") pod \"multus-hwz82\" (UID: \"2410fb3f-6b21-40b9-b23b-ffdbbdb132a7\") " pod="openshift-multus/multus-hwz82" Jul 07 10:27:13.857826 ip-10-0-135-34 kubenswrapper[2578]: I0707 10:27:13.857454 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/b1617485-94f6-4eba-95e9-866fc2400643-device-dir\") pod \"aws-ebs-csi-driver-node-dll6x\" (UID: \"b1617485-94f6-4eba-95e9-866fc2400643\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-dll6x" Jul 07 10:27:13.857826 ip-10-0-135-34 kubenswrapper[2578]: I0707 10:27:13.857460 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/2410fb3f-6b21-40b9-b23b-ffdbbdb132a7-host-run-netns\") pod \"multus-hwz82\" (UID: \"2410fb3f-6b21-40b9-b23b-ffdbbdb132a7\") " pod="openshift-multus/multus-hwz82" Jul 07 10:27:13.857826 ip-10-0-135-34 kubenswrapper[2578]: I0707 10:27:13.857467 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/b1617485-94f6-4eba-95e9-866fc2400643-kubelet-dir\") pod \"aws-ebs-csi-driver-node-dll6x\" (UID: \"b1617485-94f6-4eba-95e9-866fc2400643\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-dll6x" Jul 07 10:27:13.857826 ip-10-0-135-34 kubenswrapper[2578]: I0707 10:27:13.857501 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/2410fb3f-6b21-40b9-b23b-ffdbbdb132a7-host-var-lib-cni-multus\") pod \"multus-hwz82\" (UID: \"2410fb3f-6b21-40b9-b23b-ffdbbdb132a7\") " pod="openshift-multus/multus-hwz82" Jul 07 10:27:13.857826 ip-10-0-135-34 kubenswrapper[2578]: I0707 10:27:13.857614 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/a415d240-ff8a-4b08-aeb6-b4018195c46b-tuning-conf-dir\") pod \"multus-additional-cni-plugins-l8d7k\" (UID: \"a415d240-ff8a-4b08-aeb6-b4018195c46b\") " pod="openshift-multus/multus-additional-cni-plugins-l8d7k" Jul 07 10:27:13.857826 ip-10-0-135-34 kubenswrapper[2578]: E0707 10:27:13.857651 2578 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 07 10:27:13.857826 ip-10-0-135-34 kubenswrapper[2578]: I0707 10:27:13.857688 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/a415d240-ff8a-4b08-aeb6-b4018195c46b-cnibin\") pod \"multus-additional-cni-plugins-l8d7k\" (UID: \"a415d240-ff8a-4b08-aeb6-b4018195c46b\") " pod="openshift-multus/multus-additional-cni-plugins-l8d7k" Jul 07 10:27:13.857826 ip-10-0-135-34 kubenswrapper[2578]: I0707 10:27:13.857655 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/a415d240-ff8a-4b08-aeb6-b4018195c46b-system-cni-dir\") pod \"multus-additional-cni-plugins-l8d7k\" (UID: \"a415d240-ff8a-4b08-aeb6-b4018195c46b\") " pod="openshift-multus/multus-additional-cni-plugins-l8d7k" Jul 07 10:27:13.858823 ip-10-0-135-34 kubenswrapper[2578]: E0707 10:27:13.857737 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/931a7d75-9ee8-4758-9f23-3521f3db5eae-metrics-certs podName:931a7d75-9ee8-4758-9f23-3521f3db5eae nodeName:}" failed. No retries permitted until 2026-07-07 10:27:14.357705692 +0000 UTC m=+3.029665239 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/931a7d75-9ee8-4758-9f23-3521f3db5eae-metrics-certs") pod "network-metrics-daemon-p2ks2" (UID: "931a7d75-9ee8-4758-9f23-3521f3db5eae") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 07 10:27:13.858823 ip-10-0-135-34 kubenswrapper[2578]: I0707 10:27:13.857809 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/2410fb3f-6b21-40b9-b23b-ffdbbdb132a7-system-cni-dir\") pod \"multus-hwz82\" (UID: \"2410fb3f-6b21-40b9-b23b-ffdbbdb132a7\") " pod="openshift-multus/multus-hwz82" Jul 07 10:27:13.858823 ip-10-0-135-34 kubenswrapper[2578]: I0707 10:27:13.858055 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/609c4941-f311-408a-82ec-1d6b1a54572d-iptables-alerter-script\") pod \"iptables-alerter-lr7dk\" (UID: \"609c4941-f311-408a-82ec-1d6b1a54572d\") " pod="openshift-network-operator/iptables-alerter-lr7dk" Jul 07 10:27:13.858823 ip-10-0-135-34 kubenswrapper[2578]: I0707 10:27:13.858121 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/a415d240-ff8a-4b08-aeb6-b4018195c46b-cni-binary-copy\") pod \"multus-additional-cni-plugins-l8d7k\" (UID: \"a415d240-ff8a-4b08-aeb6-b4018195c46b\") " pod="openshift-multus/multus-additional-cni-plugins-l8d7k" Jul 07 10:27:13.858823 ip-10-0-135-34 kubenswrapper[2578]: I0707 10:27:13.858399 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/1e1b149f-fe01-49d2-8387-4cf7aea820e7-etc-tuned\") pod \"tuned-lj6lw\" (UID: \"1e1b149f-fe01-49d2-8387-4cf7aea820e7\") " pod="openshift-cluster-node-tuning-operator/tuned-lj6lw" Jul 07 10:27:13.860492 ip-10-0-135-34 kubenswrapper[2578]: I0707 10:27:13.860471 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/1e1b149f-fe01-49d2-8387-4cf7aea820e7-tmp\") pod \"tuned-lj6lw\" (UID: \"1e1b149f-fe01-49d2-8387-4cf7aea820e7\") " pod="openshift-cluster-node-tuning-operator/tuned-lj6lw" Jul 07 10:27:13.866235 ip-10-0-135-34 kubenswrapper[2578]: E0707 10:27:13.866136 2578 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 07 10:27:13.866235 ip-10-0-135-34 kubenswrapper[2578]: E0707 10:27:13.866238 2578 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 07 10:27:13.866434 ip-10-0-135-34 kubenswrapper[2578]: E0707 10:27:13.866249 2578 projected.go:196] Error preparing data for projected volume kube-api-access-9vbqq for pod openshift-network-diagnostics/network-check-target-26sg7: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 07 10:27:13.866434 ip-10-0-135-34 kubenswrapper[2578]: E0707 10:27:13.866321 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/51fe14a4-6b46-4f88-be7d-b611b45961ff-kube-api-access-9vbqq podName:51fe14a4-6b46-4f88-be7d-b611b45961ff nodeName:}" failed. No retries permitted until 2026-07-07 10:27:14.366305818 +0000 UTC m=+3.038265342 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-api-access-9vbqq" (UniqueName: "kubernetes.io/projected/51fe14a4-6b46-4f88-be7d-b611b45961ff-kube-api-access-9vbqq") pod "network-check-target-26sg7" (UID: "51fe14a4-6b46-4f88-be7d-b611b45961ff") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 07 10:27:13.871410 ip-10-0-135-34 kubenswrapper[2578]: I0707 10:27:13.871385 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bw8ks\" (UniqueName: \"kubernetes.io/projected/1e1b149f-fe01-49d2-8387-4cf7aea820e7-kube-api-access-bw8ks\") pod \"tuned-lj6lw\" (UID: \"1e1b149f-fe01-49d2-8387-4cf7aea820e7\") " pod="openshift-cluster-node-tuning-operator/tuned-lj6lw" Jul 07 10:27:13.874711 ip-10-0-135-34 kubenswrapper[2578]: I0707 10:27:13.874681 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-29rjr\" (UniqueName: \"kubernetes.io/projected/609c4941-f311-408a-82ec-1d6b1a54572d-kube-api-access-29rjr\") pod \"iptables-alerter-lr7dk\" (UID: \"609c4941-f311-408a-82ec-1d6b1a54572d\") " pod="openshift-network-operator/iptables-alerter-lr7dk" Jul 07 10:27:13.878274 ip-10-0-135-34 kubenswrapper[2578]: I0707 10:27:13.877634 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9n5dz\" (UniqueName: \"kubernetes.io/projected/a415d240-ff8a-4b08-aeb6-b4018195c46b-kube-api-access-9n5dz\") pod \"multus-additional-cni-plugins-l8d7k\" (UID: \"a415d240-ff8a-4b08-aeb6-b4018195c46b\") " pod="openshift-multus/multus-additional-cni-plugins-l8d7k" Jul 07 10:27:13.878274 ip-10-0-135-34 kubenswrapper[2578]: I0707 10:27:13.878118 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wbblv\" (UniqueName: \"kubernetes.io/projected/dfa64e14-3729-4f28-870f-eeab6cec9944-kube-api-access-wbblv\") pod \"node-resolver-rdrnq\" (UID: \"dfa64e14-3729-4f28-870f-eeab6cec9944\") " pod="openshift-dns/node-resolver-rdrnq" Jul 07 10:27:13.878388 ip-10-0-135-34 kubenswrapper[2578]: I0707 10:27:13.878274 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ncrl8\" (UniqueName: \"kubernetes.io/projected/931a7d75-9ee8-4758-9f23-3521f3db5eae-kube-api-access-ncrl8\") pod \"network-metrics-daemon-p2ks2\" (UID: \"931a7d75-9ee8-4758-9f23-3521f3db5eae\") " pod="openshift-multus/network-metrics-daemon-p2ks2" Jul 07 10:27:13.878596 ip-10-0-135-34 kubenswrapper[2578]: I0707 10:27:13.878574 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vw57m\" (UniqueName: \"kubernetes.io/projected/b1617485-94f6-4eba-95e9-866fc2400643-kube-api-access-vw57m\") pod \"aws-ebs-csi-driver-node-dll6x\" (UID: \"b1617485-94f6-4eba-95e9-866fc2400643\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-dll6x" Jul 07 10:27:13.880353 ip-10-0-135-34 kubenswrapper[2578]: I0707 10:27:13.880328 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-f8c4t\" (UniqueName: \"kubernetes.io/projected/2410fb3f-6b21-40b9-b23b-ffdbbdb132a7-kube-api-access-f8c4t\") pod \"multus-hwz82\" (UID: \"2410fb3f-6b21-40b9-b23b-ffdbbdb132a7\") " pod="openshift-multus/multus-hwz82" Jul 07 10:27:13.943294 ip-10-0-135-34 kubenswrapper[2578]: I0707 10:27:13.943228 2578 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160" Jul 07 10:27:13.956390 ip-10-0-135-34 kubenswrapper[2578]: I0707 10:27:13.954653 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/f85fac08-720e-4ffd-80af-19c140c36804-dbus\") pod \"global-pull-secret-syncer-7lv79\" (UID: \"f85fac08-720e-4ffd-80af-19c140c36804\") " pod="kube-system/global-pull-secret-syncer-7lv79" Jul 07 10:27:13.956390 ip-10-0-135-34 kubenswrapper[2578]: I0707 10:27:13.954769 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/11079224-e507-45d1-9fbb-da5043002c51-agent-certs\") pod \"konnectivity-agent-t82fl\" (UID: \"11079224-e507-45d1-9fbb-da5043002c51\") " pod="kube-system/konnectivity-agent-t82fl" Jul 07 10:27:13.956390 ip-10-0-135-34 kubenswrapper[2578]: I0707 10:27:13.954817 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/aca7528d-ddbe-4282-979f-1b496011f0e1-systemd-units\") pod \"ovnkube-node-qhhw7\" (UID: \"aca7528d-ddbe-4282-979f-1b496011f0e1\") " pod="openshift-ovn-kubernetes/ovnkube-node-qhhw7" Jul 07 10:27:13.956390 ip-10-0-135-34 kubenswrapper[2578]: I0707 10:27:13.954884 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/aca7528d-ddbe-4282-979f-1b496011f0e1-etc-openvswitch\") pod \"ovnkube-node-qhhw7\" (UID: \"aca7528d-ddbe-4282-979f-1b496011f0e1\") " pod="openshift-ovn-kubernetes/ovnkube-node-qhhw7" Jul 07 10:27:13.956390 ip-10-0-135-34 kubenswrapper[2578]: I0707 10:27:13.954917 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/aca7528d-ddbe-4282-979f-1b496011f0e1-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-qhhw7\" (UID: \"aca7528d-ddbe-4282-979f-1b496011f0e1\") " pod="openshift-ovn-kubernetes/ovnkube-node-qhhw7" Jul 07 10:27:13.956390 ip-10-0-135-34 kubenswrapper[2578]: I0707 10:27:13.954941 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/025a30f8-f923-4bc8-adf5-7ef97ee3bfd3-serviceca\") pod \"node-ca-pb2p8\" (UID: \"025a30f8-f923-4bc8-adf5-7ef97ee3bfd3\") " pod="openshift-image-registry/node-ca-pb2p8" Jul 07 10:27:13.956390 ip-10-0-135-34 kubenswrapper[2578]: I0707 10:27:13.955042 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/aca7528d-ddbe-4282-979f-1b496011f0e1-var-lib-openvswitch\") pod \"ovnkube-node-qhhw7\" (UID: \"aca7528d-ddbe-4282-979f-1b496011f0e1\") " pod="openshift-ovn-kubernetes/ovnkube-node-qhhw7" Jul 07 10:27:13.956390 ip-10-0-135-34 kubenswrapper[2578]: I0707 10:27:13.955060 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/aca7528d-ddbe-4282-979f-1b496011f0e1-run-openvswitch\") pod \"ovnkube-node-qhhw7\" (UID: \"aca7528d-ddbe-4282-979f-1b496011f0e1\") " pod="openshift-ovn-kubernetes/ovnkube-node-qhhw7" Jul 07 10:27:13.956390 ip-10-0-135-34 kubenswrapper[2578]: I0707 10:27:13.955080 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/aca7528d-ddbe-4282-979f-1b496011f0e1-ovnkube-script-lib\") pod \"ovnkube-node-qhhw7\" (UID: \"aca7528d-ddbe-4282-979f-1b496011f0e1\") " pod="openshift-ovn-kubernetes/ovnkube-node-qhhw7" Jul 07 10:27:13.956390 ip-10-0-135-34 kubenswrapper[2578]: I0707 10:27:13.955106 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n4rff\" (UniqueName: \"kubernetes.io/projected/aca7528d-ddbe-4282-979f-1b496011f0e1-kube-api-access-n4rff\") pod \"ovnkube-node-qhhw7\" (UID: \"aca7528d-ddbe-4282-979f-1b496011f0e1\") " pod="openshift-ovn-kubernetes/ovnkube-node-qhhw7" Jul 07 10:27:13.956390 ip-10-0-135-34 kubenswrapper[2578]: I0707 10:27:13.955161 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/11079224-e507-45d1-9fbb-da5043002c51-konnectivity-ca\") pod \"konnectivity-agent-t82fl\" (UID: \"11079224-e507-45d1-9fbb-da5043002c51\") " pod="kube-system/konnectivity-agent-t82fl" Jul 07 10:27:13.956390 ip-10-0-135-34 kubenswrapper[2578]: I0707 10:27:13.955178 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/aca7528d-ddbe-4282-979f-1b496011f0e1-run-ovn\") pod \"ovnkube-node-qhhw7\" (UID: \"aca7528d-ddbe-4282-979f-1b496011f0e1\") " pod="openshift-ovn-kubernetes/ovnkube-node-qhhw7" Jul 07 10:27:13.956390 ip-10-0-135-34 kubenswrapper[2578]: I0707 10:27:13.955195 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/aca7528d-ddbe-4282-979f-1b496011f0e1-host-run-ovn-kubernetes\") pod \"ovnkube-node-qhhw7\" (UID: \"aca7528d-ddbe-4282-979f-1b496011f0e1\") " pod="openshift-ovn-kubernetes/ovnkube-node-qhhw7" Jul 07 10:27:13.956390 ip-10-0-135-34 kubenswrapper[2578]: I0707 10:27:13.955220 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/aca7528d-ddbe-4282-979f-1b496011f0e1-host-cni-bin\") pod \"ovnkube-node-qhhw7\" (UID: \"aca7528d-ddbe-4282-979f-1b496011f0e1\") " pod="openshift-ovn-kubernetes/ovnkube-node-qhhw7" Jul 07 10:27:13.956390 ip-10-0-135-34 kubenswrapper[2578]: I0707 10:27:13.955244 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/aca7528d-ddbe-4282-979f-1b496011f0e1-ovn-node-metrics-cert\") pod \"ovnkube-node-qhhw7\" (UID: \"aca7528d-ddbe-4282-979f-1b496011f0e1\") " pod="openshift-ovn-kubernetes/ovnkube-node-qhhw7" Jul 07 10:27:13.956390 ip-10-0-135-34 kubenswrapper[2578]: I0707 10:27:13.955267 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/aca7528d-ddbe-4282-979f-1b496011f0e1-host-cni-netd\") pod \"ovnkube-node-qhhw7\" (UID: \"aca7528d-ddbe-4282-979f-1b496011f0e1\") " pod="openshift-ovn-kubernetes/ovnkube-node-qhhw7" Jul 07 10:27:13.956390 ip-10-0-135-34 kubenswrapper[2578]: I0707 10:27:13.955282 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/aca7528d-ddbe-4282-979f-1b496011f0e1-ovnkube-config\") pod \"ovnkube-node-qhhw7\" (UID: \"aca7528d-ddbe-4282-979f-1b496011f0e1\") " pod="openshift-ovn-kubernetes/ovnkube-node-qhhw7" Jul 07 10:27:13.957294 ip-10-0-135-34 kubenswrapper[2578]: I0707 10:27:13.955302 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/aca7528d-ddbe-4282-979f-1b496011f0e1-run-openvswitch\") pod \"ovnkube-node-qhhw7\" (UID: \"aca7528d-ddbe-4282-979f-1b496011f0e1\") " pod="openshift-ovn-kubernetes/ovnkube-node-qhhw7" Jul 07 10:27:13.957294 ip-10-0-135-34 kubenswrapper[2578]: I0707 10:27:13.954879 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/f85fac08-720e-4ffd-80af-19c140c36804-dbus\") pod \"global-pull-secret-syncer-7lv79\" (UID: \"f85fac08-720e-4ffd-80af-19c140c36804\") " pod="kube-system/global-pull-secret-syncer-7lv79" Jul 07 10:27:13.957294 ip-10-0-135-34 kubenswrapper[2578]: I0707 10:27:13.955310 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/aca7528d-ddbe-4282-979f-1b496011f0e1-env-overrides\") pod \"ovnkube-node-qhhw7\" (UID: \"aca7528d-ddbe-4282-979f-1b496011f0e1\") " pod="openshift-ovn-kubernetes/ovnkube-node-qhhw7" Jul 07 10:27:13.957294 ip-10-0-135-34 kubenswrapper[2578]: I0707 10:27:13.955377 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/aca7528d-ddbe-4282-979f-1b496011f0e1-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-qhhw7\" (UID: \"aca7528d-ddbe-4282-979f-1b496011f0e1\") " pod="openshift-ovn-kubernetes/ovnkube-node-qhhw7" Jul 07 10:27:13.957294 ip-10-0-135-34 kubenswrapper[2578]: I0707 10:27:13.955387 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/aca7528d-ddbe-4282-979f-1b496011f0e1-systemd-units\") pod \"ovnkube-node-qhhw7\" (UID: \"aca7528d-ddbe-4282-979f-1b496011f0e1\") " pod="openshift-ovn-kubernetes/ovnkube-node-qhhw7" Jul 07 10:27:13.957294 ip-10-0-135-34 kubenswrapper[2578]: I0707 10:27:13.955391 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/f85fac08-720e-4ffd-80af-19c140c36804-original-pull-secret\") pod \"global-pull-secret-syncer-7lv79\" (UID: \"f85fac08-720e-4ffd-80af-19c140c36804\") " pod="kube-system/global-pull-secret-syncer-7lv79" Jul 07 10:27:13.957294 ip-10-0-135-34 kubenswrapper[2578]: I0707 10:27:13.955428 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/aca7528d-ddbe-4282-979f-1b496011f0e1-host-run-ovn-kubernetes\") pod \"ovnkube-node-qhhw7\" (UID: \"aca7528d-ddbe-4282-979f-1b496011f0e1\") " pod="openshift-ovn-kubernetes/ovnkube-node-qhhw7" Jul 07 10:27:13.957294 ip-10-0-135-34 kubenswrapper[2578]: I0707 10:27:13.955436 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/aca7528d-ddbe-4282-979f-1b496011f0e1-host-run-netns\") pod \"ovnkube-node-qhhw7\" (UID: \"aca7528d-ddbe-4282-979f-1b496011f0e1\") " pod="openshift-ovn-kubernetes/ovnkube-node-qhhw7" Jul 07 10:27:13.957294 ip-10-0-135-34 kubenswrapper[2578]: I0707 10:27:13.955466 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/aca7528d-ddbe-4282-979f-1b496011f0e1-host-run-netns\") pod \"ovnkube-node-qhhw7\" (UID: \"aca7528d-ddbe-4282-979f-1b496011f0e1\") " pod="openshift-ovn-kubernetes/ovnkube-node-qhhw7" Jul 07 10:27:13.957294 ip-10-0-135-34 kubenswrapper[2578]: E0707 10:27:13.955470 2578 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 07 10:27:13.957294 ip-10-0-135-34 kubenswrapper[2578]: I0707 10:27:13.955511 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/aca7528d-ddbe-4282-979f-1b496011f0e1-etc-openvswitch\") pod \"ovnkube-node-qhhw7\" (UID: \"aca7528d-ddbe-4282-979f-1b496011f0e1\") " pod="openshift-ovn-kubernetes/ovnkube-node-qhhw7" Jul 07 10:27:13.957294 ip-10-0-135-34 kubenswrapper[2578]: E0707 10:27:13.955539 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f85fac08-720e-4ffd-80af-19c140c36804-original-pull-secret podName:f85fac08-720e-4ffd-80af-19c140c36804 nodeName:}" failed. No retries permitted until 2026-07-07 10:27:14.455520269 +0000 UTC m=+3.127479808 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/f85fac08-720e-4ffd-80af-19c140c36804-original-pull-secret") pod "global-pull-secret-syncer-7lv79" (UID: "f85fac08-720e-4ffd-80af-19c140c36804") : object "kube-system"/"original-pull-secret" not registered Jul 07 10:27:13.957294 ip-10-0-135-34 kubenswrapper[2578]: I0707 10:27:13.955561 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/aca7528d-ddbe-4282-979f-1b496011f0e1-host-kubelet\") pod \"ovnkube-node-qhhw7\" (UID: \"aca7528d-ddbe-4282-979f-1b496011f0e1\") " pod="openshift-ovn-kubernetes/ovnkube-node-qhhw7" Jul 07 10:27:13.957294 ip-10-0-135-34 kubenswrapper[2578]: I0707 10:27:13.955591 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/aca7528d-ddbe-4282-979f-1b496011f0e1-run-systemd\") pod \"ovnkube-node-qhhw7\" (UID: \"aca7528d-ddbe-4282-979f-1b496011f0e1\") " pod="openshift-ovn-kubernetes/ovnkube-node-qhhw7" Jul 07 10:27:13.957294 ip-10-0-135-34 kubenswrapper[2578]: I0707 10:27:13.955635 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/aca7528d-ddbe-4282-979f-1b496011f0e1-log-socket\") pod \"ovnkube-node-qhhw7\" (UID: \"aca7528d-ddbe-4282-979f-1b496011f0e1\") " pod="openshift-ovn-kubernetes/ovnkube-node-qhhw7" Jul 07 10:27:13.957294 ip-10-0-135-34 kubenswrapper[2578]: I0707 10:27:13.955683 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gx8bz\" (UniqueName: \"kubernetes.io/projected/025a30f8-f923-4bc8-adf5-7ef97ee3bfd3-kube-api-access-gx8bz\") pod \"node-ca-pb2p8\" (UID: \"025a30f8-f923-4bc8-adf5-7ef97ee3bfd3\") " pod="openshift-image-registry/node-ca-pb2p8" Jul 07 10:27:13.957294 ip-10-0-135-34 kubenswrapper[2578]: I0707 10:27:13.955755 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/aca7528d-ddbe-4282-979f-1b496011f0e1-node-log\") pod \"ovnkube-node-qhhw7\" (UID: \"aca7528d-ddbe-4282-979f-1b496011f0e1\") " pod="openshift-ovn-kubernetes/ovnkube-node-qhhw7" Jul 07 10:27:13.958147 ip-10-0-135-34 kubenswrapper[2578]: I0707 10:27:13.955778 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/aca7528d-ddbe-4282-979f-1b496011f0e1-env-overrides\") pod \"ovnkube-node-qhhw7\" (UID: \"aca7528d-ddbe-4282-979f-1b496011f0e1\") " pod="openshift-ovn-kubernetes/ovnkube-node-qhhw7" Jul 07 10:27:13.958147 ip-10-0-135-34 kubenswrapper[2578]: I0707 10:27:13.955788 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/025a30f8-f923-4bc8-adf5-7ef97ee3bfd3-host\") pod \"node-ca-pb2p8\" (UID: \"025a30f8-f923-4bc8-adf5-7ef97ee3bfd3\") " pod="openshift-image-registry/node-ca-pb2p8" Jul 07 10:27:13.958147 ip-10-0-135-34 kubenswrapper[2578]: I0707 10:27:13.955824 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/f85fac08-720e-4ffd-80af-19c140c36804-kubelet-config\") pod \"global-pull-secret-syncer-7lv79\" (UID: \"f85fac08-720e-4ffd-80af-19c140c36804\") " pod="kube-system/global-pull-secret-syncer-7lv79" Jul 07 10:27:13.958147 ip-10-0-135-34 kubenswrapper[2578]: I0707 10:27:13.955854 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/025a30f8-f923-4bc8-adf5-7ef97ee3bfd3-host\") pod \"node-ca-pb2p8\" (UID: \"025a30f8-f923-4bc8-adf5-7ef97ee3bfd3\") " pod="openshift-image-registry/node-ca-pb2p8" Jul 07 10:27:13.958147 ip-10-0-135-34 kubenswrapper[2578]: I0707 10:27:13.955887 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/025a30f8-f923-4bc8-adf5-7ef97ee3bfd3-serviceca\") pod \"node-ca-pb2p8\" (UID: \"025a30f8-f923-4bc8-adf5-7ef97ee3bfd3\") " pod="openshift-image-registry/node-ca-pb2p8" Jul 07 10:27:13.958147 ip-10-0-135-34 kubenswrapper[2578]: I0707 10:27:13.955880 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/aca7528d-ddbe-4282-979f-1b496011f0e1-host-slash\") pod \"ovnkube-node-qhhw7\" (UID: \"aca7528d-ddbe-4282-979f-1b496011f0e1\") " pod="openshift-ovn-kubernetes/ovnkube-node-qhhw7" Jul 07 10:27:13.958147 ip-10-0-135-34 kubenswrapper[2578]: I0707 10:27:13.955980 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/aca7528d-ddbe-4282-979f-1b496011f0e1-var-lib-openvswitch\") pod \"ovnkube-node-qhhw7\" (UID: \"aca7528d-ddbe-4282-979f-1b496011f0e1\") " pod="openshift-ovn-kubernetes/ovnkube-node-qhhw7" Jul 07 10:27:13.958147 ip-10-0-135-34 kubenswrapper[2578]: I0707 10:27:13.955977 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/aca7528d-ddbe-4282-979f-1b496011f0e1-host-slash\") pod \"ovnkube-node-qhhw7\" (UID: \"aca7528d-ddbe-4282-979f-1b496011f0e1\") " pod="openshift-ovn-kubernetes/ovnkube-node-qhhw7" Jul 07 10:27:13.958147 ip-10-0-135-34 kubenswrapper[2578]: I0707 10:27:13.956026 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/aca7528d-ddbe-4282-979f-1b496011f0e1-host-cni-netd\") pod \"ovnkube-node-qhhw7\" (UID: \"aca7528d-ddbe-4282-979f-1b496011f0e1\") " pod="openshift-ovn-kubernetes/ovnkube-node-qhhw7" Jul 07 10:27:13.958147 ip-10-0-135-34 kubenswrapper[2578]: I0707 10:27:13.956058 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/aca7528d-ddbe-4282-979f-1b496011f0e1-host-kubelet\") pod \"ovnkube-node-qhhw7\" (UID: \"aca7528d-ddbe-4282-979f-1b496011f0e1\") " pod="openshift-ovn-kubernetes/ovnkube-node-qhhw7" Jul 07 10:27:13.958147 ip-10-0-135-34 kubenswrapper[2578]: I0707 10:27:13.956092 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/aca7528d-ddbe-4282-979f-1b496011f0e1-ovnkube-script-lib\") pod \"ovnkube-node-qhhw7\" (UID: \"aca7528d-ddbe-4282-979f-1b496011f0e1\") " pod="openshift-ovn-kubernetes/ovnkube-node-qhhw7" Jul 07 10:27:13.958147 ip-10-0-135-34 kubenswrapper[2578]: I0707 10:27:13.956068 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/aca7528d-ddbe-4282-979f-1b496011f0e1-host-cni-bin\") pod \"ovnkube-node-qhhw7\" (UID: \"aca7528d-ddbe-4282-979f-1b496011f0e1\") " pod="openshift-ovn-kubernetes/ovnkube-node-qhhw7" Jul 07 10:27:13.958147 ip-10-0-135-34 kubenswrapper[2578]: I0707 10:27:13.956148 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/aca7528d-ddbe-4282-979f-1b496011f0e1-log-socket\") pod \"ovnkube-node-qhhw7\" (UID: \"aca7528d-ddbe-4282-979f-1b496011f0e1\") " pod="openshift-ovn-kubernetes/ovnkube-node-qhhw7" Jul 07 10:27:13.958147 ip-10-0-135-34 kubenswrapper[2578]: I0707 10:27:13.956152 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/aca7528d-ddbe-4282-979f-1b496011f0e1-node-log\") pod \"ovnkube-node-qhhw7\" (UID: \"aca7528d-ddbe-4282-979f-1b496011f0e1\") " pod="openshift-ovn-kubernetes/ovnkube-node-qhhw7" Jul 07 10:27:13.958147 ip-10-0-135-34 kubenswrapper[2578]: I0707 10:27:13.956158 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/f85fac08-720e-4ffd-80af-19c140c36804-kubelet-config\") pod \"global-pull-secret-syncer-7lv79\" (UID: \"f85fac08-720e-4ffd-80af-19c140c36804\") " pod="kube-system/global-pull-secret-syncer-7lv79" Jul 07 10:27:13.958147 ip-10-0-135-34 kubenswrapper[2578]: I0707 10:27:13.956106 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/aca7528d-ddbe-4282-979f-1b496011f0e1-run-systemd\") pod \"ovnkube-node-qhhw7\" (UID: \"aca7528d-ddbe-4282-979f-1b496011f0e1\") " pod="openshift-ovn-kubernetes/ovnkube-node-qhhw7" Jul 07 10:27:13.958147 ip-10-0-135-34 kubenswrapper[2578]: I0707 10:27:13.956164 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/11079224-e507-45d1-9fbb-da5043002c51-konnectivity-ca\") pod \"konnectivity-agent-t82fl\" (UID: \"11079224-e507-45d1-9fbb-da5043002c51\") " pod="kube-system/konnectivity-agent-t82fl" Jul 07 10:27:13.958147 ip-10-0-135-34 kubenswrapper[2578]: I0707 10:27:13.956197 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/aca7528d-ddbe-4282-979f-1b496011f0e1-run-ovn\") pod \"ovnkube-node-qhhw7\" (UID: \"aca7528d-ddbe-4282-979f-1b496011f0e1\") " pod="openshift-ovn-kubernetes/ovnkube-node-qhhw7" Jul 07 10:27:13.959014 ip-10-0-135-34 kubenswrapper[2578]: I0707 10:27:13.956521 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/aca7528d-ddbe-4282-979f-1b496011f0e1-ovnkube-config\") pod \"ovnkube-node-qhhw7\" (UID: \"aca7528d-ddbe-4282-979f-1b496011f0e1\") " pod="openshift-ovn-kubernetes/ovnkube-node-qhhw7" Jul 07 10:27:13.959484 ip-10-0-135-34 kubenswrapper[2578]: I0707 10:27:13.959365 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/11079224-e507-45d1-9fbb-da5043002c51-agent-certs\") pod \"konnectivity-agent-t82fl\" (UID: \"11079224-e507-45d1-9fbb-da5043002c51\") " pod="kube-system/konnectivity-agent-t82fl" Jul 07 10:27:13.962359 ip-10-0-135-34 kubenswrapper[2578]: I0707 10:27:13.962325 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/aca7528d-ddbe-4282-979f-1b496011f0e1-ovn-node-metrics-cert\") pod \"ovnkube-node-qhhw7\" (UID: \"aca7528d-ddbe-4282-979f-1b496011f0e1\") " pod="openshift-ovn-kubernetes/ovnkube-node-qhhw7" Jul 07 10:27:13.971864 ip-10-0-135-34 kubenswrapper[2578]: I0707 10:27:13.971788 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-n4rff\" (UniqueName: \"kubernetes.io/projected/aca7528d-ddbe-4282-979f-1b496011f0e1-kube-api-access-n4rff\") pod \"ovnkube-node-qhhw7\" (UID: \"aca7528d-ddbe-4282-979f-1b496011f0e1\") " pod="openshift-ovn-kubernetes/ovnkube-node-qhhw7" Jul 07 10:27:13.976284 ip-10-0-135-34 kubenswrapper[2578]: I0707 10:27:13.976256 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gx8bz\" (UniqueName: \"kubernetes.io/projected/025a30f8-f923-4bc8-adf5-7ef97ee3bfd3-kube-api-access-gx8bz\") pod \"node-ca-pb2p8\" (UID: \"025a30f8-f923-4bc8-adf5-7ef97ee3bfd3\") " pod="openshift-image-registry/node-ca-pb2p8" Jul 07 10:27:14.042031 ip-10-0-135-34 kubenswrapper[2578]: I0707 10:27:14.041996 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-lj6lw" Jul 07 10:27:14.051385 ip-10-0-135-34 kubenswrapper[2578]: I0707 10:27:14.051351 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-l8d7k" Jul 07 10:27:14.059680 ip-10-0-135-34 kubenswrapper[2578]: I0707 10:27:14.059652 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-rdrnq" Jul 07 10:27:14.065226 ip-10-0-135-34 kubenswrapper[2578]: I0707 10:27:14.065053 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-hwz82" Jul 07 10:27:14.074054 ip-10-0-135-34 kubenswrapper[2578]: I0707 10:27:14.074015 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-lr7dk" Jul 07 10:27:14.080640 ip-10-0-135-34 kubenswrapper[2578]: I0707 10:27:14.080428 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-dll6x" Jul 07 10:27:14.088523 ip-10-0-135-34 kubenswrapper[2578]: I0707 10:27:14.088447 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-t82fl" Jul 07 10:27:14.097889 ip-10-0-135-34 kubenswrapper[2578]: I0707 10:27:14.097663 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-pb2p8" Jul 07 10:27:14.105427 ip-10-0-135-34 kubenswrapper[2578]: I0707 10:27:14.105247 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-qhhw7" Jul 07 10:27:14.359424 ip-10-0-135-34 kubenswrapper[2578]: I0707 10:27:14.359321 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/931a7d75-9ee8-4758-9f23-3521f3db5eae-metrics-certs\") pod \"network-metrics-daemon-p2ks2\" (UID: \"931a7d75-9ee8-4758-9f23-3521f3db5eae\") " pod="openshift-multus/network-metrics-daemon-p2ks2" Jul 07 10:27:14.359604 ip-10-0-135-34 kubenswrapper[2578]: E0707 10:27:14.359516 2578 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 07 10:27:14.359604 ip-10-0-135-34 kubenswrapper[2578]: E0707 10:27:14.359603 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/931a7d75-9ee8-4758-9f23-3521f3db5eae-metrics-certs podName:931a7d75-9ee8-4758-9f23-3521f3db5eae nodeName:}" failed. No retries permitted until 2026-07-07 10:27:15.359580826 +0000 UTC m=+4.031540366 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/931a7d75-9ee8-4758-9f23-3521f3db5eae-metrics-certs") pod "network-metrics-daemon-p2ks2" (UID: "931a7d75-9ee8-4758-9f23-3521f3db5eae") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 07 10:27:14.459731 ip-10-0-135-34 kubenswrapper[2578]: I0707 10:27:14.459687 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/f85fac08-720e-4ffd-80af-19c140c36804-original-pull-secret\") pod \"global-pull-secret-syncer-7lv79\" (UID: \"f85fac08-720e-4ffd-80af-19c140c36804\") " pod="kube-system/global-pull-secret-syncer-7lv79" Jul 07 10:27:14.459929 ip-10-0-135-34 kubenswrapper[2578]: I0707 10:27:14.459819 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9vbqq\" (UniqueName: \"kubernetes.io/projected/51fe14a4-6b46-4f88-be7d-b611b45961ff-kube-api-access-9vbqq\") pod \"network-check-target-26sg7\" (UID: \"51fe14a4-6b46-4f88-be7d-b611b45961ff\") " pod="openshift-network-diagnostics/network-check-target-26sg7" Jul 07 10:27:14.459929 ip-10-0-135-34 kubenswrapper[2578]: E0707 10:27:14.459887 2578 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 07 10:27:14.460057 ip-10-0-135-34 kubenswrapper[2578]: E0707 10:27:14.459938 2578 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 07 10:27:14.460057 ip-10-0-135-34 kubenswrapper[2578]: E0707 10:27:14.459954 2578 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 07 10:27:14.460057 ip-10-0-135-34 kubenswrapper[2578]: E0707 10:27:14.459966 2578 projected.go:196] Error preparing data for projected volume kube-api-access-9vbqq for pod openshift-network-diagnostics/network-check-target-26sg7: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 07 10:27:14.460057 ip-10-0-135-34 kubenswrapper[2578]: E0707 10:27:14.459967 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f85fac08-720e-4ffd-80af-19c140c36804-original-pull-secret podName:f85fac08-720e-4ffd-80af-19c140c36804 nodeName:}" failed. No retries permitted until 2026-07-07 10:27:15.459945655 +0000 UTC m=+4.131905197 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/f85fac08-720e-4ffd-80af-19c140c36804-original-pull-secret") pod "global-pull-secret-syncer-7lv79" (UID: "f85fac08-720e-4ffd-80af-19c140c36804") : object "kube-system"/"original-pull-secret" not registered Jul 07 10:27:14.460057 ip-10-0-135-34 kubenswrapper[2578]: E0707 10:27:14.460022 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/51fe14a4-6b46-4f88-be7d-b611b45961ff-kube-api-access-9vbqq podName:51fe14a4-6b46-4f88-be7d-b611b45961ff nodeName:}" failed. No retries permitted until 2026-07-07 10:27:15.460005957 +0000 UTC m=+4.131965505 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-api-access-9vbqq" (UniqueName: "kubernetes.io/projected/51fe14a4-6b46-4f88-be7d-b611b45961ff-kube-api-access-9vbqq") pod "network-check-target-26sg7" (UID: "51fe14a4-6b46-4f88-be7d-b611b45961ff") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 07 10:27:14.784878 ip-10-0-135-34 kubenswrapper[2578]: I0707 10:27:14.784761 2578 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-07-06 10:22:13 +0000 UTC" deadline="2028-01-22 05:15:41.966519848 +0000 UTC" Jul 07 10:27:14.784878 ip-10-0-135-34 kubenswrapper[2578]: I0707 10:27:14.784833 2578 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="13530h48m27.181691495s" Jul 07 10:27:15.152370 ip-10-0-135-34 kubenswrapper[2578]: W0707 10:27:15.152119 2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2410fb3f_6b21_40b9_b23b_ffdbbdb132a7.slice/crio-8f0730a620214c634b5fff1e1c57680250c48df3f5f18ac76c793c93be41d736 WatchSource:0}: Error finding container 8f0730a620214c634b5fff1e1c57680250c48df3f5f18ac76c793c93be41d736: Status 404 returned error can't find the container with id 8f0730a620214c634b5fff1e1c57680250c48df3f5f18ac76c793c93be41d736 Jul 07 10:27:15.155605 ip-10-0-135-34 kubenswrapper[2578]: W0707 10:27:15.154983 2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod11079224_e507_45d1_9fbb_da5043002c51.slice/crio-e53360c1286bec3859fb76d0504486f748b9d7919d75bbc604199275c81a063c WatchSource:0}: Error finding container e53360c1286bec3859fb76d0504486f748b9d7919d75bbc604199275c81a063c: Status 404 returned error can't find the container with id e53360c1286bec3859fb76d0504486f748b9d7919d75bbc604199275c81a063c Jul 07 10:27:15.161355 ip-10-0-135-34 kubenswrapper[2578]: W0707 10:27:15.161329 2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod025a30f8_f923_4bc8_adf5_7ef97ee3bfd3.slice/crio-235784e8c32152acfc24869d1b238303403d0d4a8e19a6ca8212fdf4fe1bbe61 WatchSource:0}: Error finding container 235784e8c32152acfc24869d1b238303403d0d4a8e19a6ca8212fdf4fe1bbe61: Status 404 returned error can't find the container with id 235784e8c32152acfc24869d1b238303403d0d4a8e19a6ca8212fdf4fe1bbe61 Jul 07 10:27:15.162770 ip-10-0-135-34 kubenswrapper[2578]: W0707 10:27:15.162740 2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podaca7528d_ddbe_4282_979f_1b496011f0e1.slice/crio-a1473a3e2779d6a8ec34c4b709e53acfa4b20d09c70def752ec8fd6fb1d1d3d9 WatchSource:0}: Error finding container a1473a3e2779d6a8ec34c4b709e53acfa4b20d09c70def752ec8fd6fb1d1d3d9: Status 404 returned error can't find the container with id a1473a3e2779d6a8ec34c4b709e53acfa4b20d09c70def752ec8fd6fb1d1d3d9 Jul 07 10:27:15.164755 ip-10-0-135-34 kubenswrapper[2578]: W0707 10:27:15.163884 2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod609c4941_f311_408a_82ec_1d6b1a54572d.slice/crio-d09a5070bc278953498ea0a814f65b48087e1efea2ad90a7d6ab2c9b8c7b892f WatchSource:0}: Error finding container d09a5070bc278953498ea0a814f65b48087e1efea2ad90a7d6ab2c9b8c7b892f: Status 404 returned error can't find the container with id d09a5070bc278953498ea0a814f65b48087e1efea2ad90a7d6ab2c9b8c7b892f Jul 07 10:27:15.166520 ip-10-0-135-34 kubenswrapper[2578]: W0707 10:27:15.165014 2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1e1b149f_fe01_49d2_8387_4cf7aea820e7.slice/crio-38ea63547cdcabcdf273779567c1fcc821f83539018f91c61418336f24f98ce4 WatchSource:0}: Error finding container 38ea63547cdcabcdf273779567c1fcc821f83539018f91c61418336f24f98ce4: Status 404 returned error can't find the container with id 38ea63547cdcabcdf273779567c1fcc821f83539018f91c61418336f24f98ce4 Jul 07 10:27:15.166520 ip-10-0-135-34 kubenswrapper[2578]: W0707 10:27:15.166166 2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda415d240_ff8a_4b08_aeb6_b4018195c46b.slice/crio-775c35c97036f6c0c289cb3a1cc8ba53b66d2b86147d297a60e3f39ed059fb85 WatchSource:0}: Error finding container 775c35c97036f6c0c289cb3a1cc8ba53b66d2b86147d297a60e3f39ed059fb85: Status 404 returned error can't find the container with id 775c35c97036f6c0c289cb3a1cc8ba53b66d2b86147d297a60e3f39ed059fb85 Jul 07 10:27:15.366301 ip-10-0-135-34 kubenswrapper[2578]: I0707 10:27:15.366267 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/931a7d75-9ee8-4758-9f23-3521f3db5eae-metrics-certs\") pod \"network-metrics-daemon-p2ks2\" (UID: \"931a7d75-9ee8-4758-9f23-3521f3db5eae\") " pod="openshift-multus/network-metrics-daemon-p2ks2" Jul 07 10:27:15.366463 ip-10-0-135-34 kubenswrapper[2578]: E0707 10:27:15.366429 2578 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 07 10:27:15.366514 ip-10-0-135-34 kubenswrapper[2578]: E0707 10:27:15.366503 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/931a7d75-9ee8-4758-9f23-3521f3db5eae-metrics-certs podName:931a7d75-9ee8-4758-9f23-3521f3db5eae nodeName:}" failed. No retries permitted until 2026-07-07 10:27:17.366487281 +0000 UTC m=+6.038446806 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/931a7d75-9ee8-4758-9f23-3521f3db5eae-metrics-certs") pod "network-metrics-daemon-p2ks2" (UID: "931a7d75-9ee8-4758-9f23-3521f3db5eae") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 07 10:27:15.466819 ip-10-0-135-34 kubenswrapper[2578]: I0707 10:27:15.466658 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9vbqq\" (UniqueName: \"kubernetes.io/projected/51fe14a4-6b46-4f88-be7d-b611b45961ff-kube-api-access-9vbqq\") pod \"network-check-target-26sg7\" (UID: \"51fe14a4-6b46-4f88-be7d-b611b45961ff\") " pod="openshift-network-diagnostics/network-check-target-26sg7" Jul 07 10:27:15.466819 ip-10-0-135-34 kubenswrapper[2578]: I0707 10:27:15.466700 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/f85fac08-720e-4ffd-80af-19c140c36804-original-pull-secret\") pod \"global-pull-secret-syncer-7lv79\" (UID: \"f85fac08-720e-4ffd-80af-19c140c36804\") " pod="kube-system/global-pull-secret-syncer-7lv79" Jul 07 10:27:15.466923 ip-10-0-135-34 kubenswrapper[2578]: E0707 10:27:15.466872 2578 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 07 10:27:15.466923 ip-10-0-135-34 kubenswrapper[2578]: E0707 10:27:15.466875 2578 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 07 10:27:15.466923 ip-10-0-135-34 kubenswrapper[2578]: E0707 10:27:15.466898 2578 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 07 10:27:15.466923 ip-10-0-135-34 kubenswrapper[2578]: E0707 10:27:15.466913 2578 projected.go:196] Error preparing data for projected volume kube-api-access-9vbqq for pod openshift-network-diagnostics/network-check-target-26sg7: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 07 10:27:15.467070 ip-10-0-135-34 kubenswrapper[2578]: E0707 10:27:15.466939 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f85fac08-720e-4ffd-80af-19c140c36804-original-pull-secret podName:f85fac08-720e-4ffd-80af-19c140c36804 nodeName:}" failed. No retries permitted until 2026-07-07 10:27:17.466920116 +0000 UTC m=+6.138879664 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/f85fac08-720e-4ffd-80af-19c140c36804-original-pull-secret") pod "global-pull-secret-syncer-7lv79" (UID: "f85fac08-720e-4ffd-80af-19c140c36804") : object "kube-system"/"original-pull-secret" not registered Jul 07 10:27:15.467070 ip-10-0-135-34 kubenswrapper[2578]: E0707 10:27:15.466957 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/51fe14a4-6b46-4f88-be7d-b611b45961ff-kube-api-access-9vbqq podName:51fe14a4-6b46-4f88-be7d-b611b45961ff nodeName:}" failed. No retries permitted until 2026-07-07 10:27:17.466945721 +0000 UTC m=+6.138905263 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-api-access-9vbqq" (UniqueName: "kubernetes.io/projected/51fe14a4-6b46-4f88-be7d-b611b45961ff-kube-api-access-9vbqq") pod "network-check-target-26sg7" (UID: "51fe14a4-6b46-4f88-be7d-b611b45961ff") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 07 10:27:15.817141 ip-10-0-135-34 kubenswrapper[2578]: I0707 10:27:15.816754 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-7lv79" Jul 07 10:27:15.817141 ip-10-0-135-34 kubenswrapper[2578]: I0707 10:27:15.816789 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-p2ks2" Jul 07 10:27:15.817141 ip-10-0-135-34 kubenswrapper[2578]: I0707 10:27:15.816754 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-26sg7" Jul 07 10:27:15.817141 ip-10-0-135-34 kubenswrapper[2578]: E0707 10:27:15.816918 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-p2ks2" podUID="931a7d75-9ee8-4758-9f23-3521f3db5eae" Jul 07 10:27:15.817141 ip-10-0-135-34 kubenswrapper[2578]: E0707 10:27:15.816993 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-26sg7" podUID="51fe14a4-6b46-4f88-be7d-b611b45961ff" Jul 07 10:27:15.817141 ip-10-0-135-34 kubenswrapper[2578]: E0707 10:27:15.817083 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-7lv79" podUID="f85fac08-720e-4ffd-80af-19c140c36804" Jul 07 10:27:15.829792 ip-10-0-135-34 kubenswrapper[2578]: I0707 10:27:15.829734 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-pb2p8" event={"ID":"025a30f8-f923-4bc8-adf5-7ef97ee3bfd3","Type":"ContainerStarted","Data":"235784e8c32152acfc24869d1b238303403d0d4a8e19a6ca8212fdf4fe1bbe61"} Jul 07 10:27:15.831337 ip-10-0-135-34 kubenswrapper[2578]: I0707 10:27:15.831295 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-t82fl" event={"ID":"11079224-e507-45d1-9fbb-da5043002c51","Type":"ContainerStarted","Data":"e53360c1286bec3859fb76d0504486f748b9d7919d75bbc604199275c81a063c"} Jul 07 10:27:15.833319 ip-10-0-135-34 kubenswrapper[2578]: I0707 10:27:15.833291 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-hwz82" event={"ID":"2410fb3f-6b21-40b9-b23b-ffdbbdb132a7","Type":"ContainerStarted","Data":"8f0730a620214c634b5fff1e1c57680250c48df3f5f18ac76c793c93be41d736"} Jul 07 10:27:15.836538 ip-10-0-135-34 kubenswrapper[2578]: I0707 10:27:15.836512 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-135-34.ec2.internal" event={"ID":"fc94ccac0285e755088555819ea0238a","Type":"ContainerStarted","Data":"abc0aa475c374dd67dad1b3e513d7ee9a0916293cc0f4f29f129919edad53013"} Jul 07 10:27:15.840116 ip-10-0-135-34 kubenswrapper[2578]: I0707 10:27:15.840089 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-rdrnq" event={"ID":"dfa64e14-3729-4f28-870f-eeab6cec9944","Type":"ContainerStarted","Data":"729a48dc0c07099b41b73dd81fa3735e1161635373b7675b1594bc30b75a6e4b"} Jul 07 10:27:15.842596 ip-10-0-135-34 kubenswrapper[2578]: I0707 10:27:15.842568 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-l8d7k" event={"ID":"a415d240-ff8a-4b08-aeb6-b4018195c46b","Type":"ContainerStarted","Data":"775c35c97036f6c0c289cb3a1cc8ba53b66d2b86147d297a60e3f39ed059fb85"} Jul 07 10:27:15.844675 ip-10-0-135-34 kubenswrapper[2578]: I0707 10:27:15.844649 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-qhhw7" event={"ID":"aca7528d-ddbe-4282-979f-1b496011f0e1","Type":"ContainerStarted","Data":"a1473a3e2779d6a8ec34c4b709e53acfa4b20d09c70def752ec8fd6fb1d1d3d9"} Jul 07 10:27:15.847328 ip-10-0-135-34 kubenswrapper[2578]: I0707 10:27:15.847297 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-lr7dk" event={"ID":"609c4941-f311-408a-82ec-1d6b1a54572d","Type":"ContainerStarted","Data":"d09a5070bc278953498ea0a814f65b48087e1efea2ad90a7d6ab2c9b8c7b892f"} Jul 07 10:27:15.854856 ip-10-0-135-34 kubenswrapper[2578]: I0707 10:27:15.854304 2578 generic.go:356] "Generic (PLEG): container finished" podID="f3763173479b9a6f050d77b1ce5123b0" containerID="e2c3e43b3f9f1b4f135e010f740a5b6d6d342c14370e9703e181cf31993dceb2" exitCode=0 Jul 07 10:27:15.854856 ip-10-0-135-34 kubenswrapper[2578]: I0707 10:27:15.854390 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-34.ec2.internal" event={"ID":"f3763173479b9a6f050d77b1ce5123b0","Type":"ContainerDied","Data":"e2c3e43b3f9f1b4f135e010f740a5b6d6d342c14370e9703e181cf31993dceb2"} Jul 07 10:27:15.856911 ip-10-0-135-34 kubenswrapper[2578]: I0707 10:27:15.856880 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-dll6x" event={"ID":"b1617485-94f6-4eba-95e9-866fc2400643","Type":"ContainerStarted","Data":"81685dabd38f609d54a129fe3447922ea7eb41f745b2f1ffdf9465f13e95d93f"} Jul 07 10:27:15.859071 ip-10-0-135-34 kubenswrapper[2578]: I0707 10:27:15.859044 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-lj6lw" event={"ID":"1e1b149f-fe01-49d2-8387-4cf7aea820e7","Type":"ContainerStarted","Data":"38ea63547cdcabcdf273779567c1fcc821f83539018f91c61418336f24f98ce4"} Jul 07 10:27:15.871719 ip-10-0-135-34 kubenswrapper[2578]: I0707 10:27:15.871676 2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/kube-apiserver-proxy-ip-10-0-135-34.ec2.internal" podStartSLOduration=2.871658089 podStartE2EDuration="2.871658089s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-07 10:27:13 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 10:27:15.853252757 +0000 UTC m=+4.525212316" watchObservedRunningTime="2026-07-07 10:27:15.871658089 +0000 UTC m=+4.543617636" Jul 07 10:27:16.886931 ip-10-0-135-34 kubenswrapper[2578]: I0707 10:27:16.886835 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-34.ec2.internal" event={"ID":"f3763173479b9a6f050d77b1ce5123b0","Type":"ContainerStarted","Data":"41d5a90d0975d635da045ef7409ca9af5a3ac94e4ba4c962faa27848fb73aecd"} Jul 07 10:27:16.907602 ip-10-0-135-34 kubenswrapper[2578]: I0707 10:27:16.906575 2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-34.ec2.internal" podStartSLOduration=3.90655279 podStartE2EDuration="3.90655279s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-07 10:27:13 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 10:27:16.906426245 +0000 UTC m=+5.578385793" watchObservedRunningTime="2026-07-07 10:27:16.90655279 +0000 UTC m=+5.578512339" Jul 07 10:27:17.381175 ip-10-0-135-34 kubenswrapper[2578]: I0707 10:27:17.381001 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/931a7d75-9ee8-4758-9f23-3521f3db5eae-metrics-certs\") pod \"network-metrics-daemon-p2ks2\" (UID: \"931a7d75-9ee8-4758-9f23-3521f3db5eae\") " pod="openshift-multus/network-metrics-daemon-p2ks2" Jul 07 10:27:17.381359 ip-10-0-135-34 kubenswrapper[2578]: E0707 10:27:17.381202 2578 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 07 10:27:17.381359 ip-10-0-135-34 kubenswrapper[2578]: E0707 10:27:17.381284 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/931a7d75-9ee8-4758-9f23-3521f3db5eae-metrics-certs podName:931a7d75-9ee8-4758-9f23-3521f3db5eae nodeName:}" failed. No retries permitted until 2026-07-07 10:27:21.381262537 +0000 UTC m=+10.053222064 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/931a7d75-9ee8-4758-9f23-3521f3db5eae-metrics-certs") pod "network-metrics-daemon-p2ks2" (UID: "931a7d75-9ee8-4758-9f23-3521f3db5eae") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 07 10:27:17.481986 ip-10-0-135-34 kubenswrapper[2578]: I0707 10:27:17.481936 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9vbqq\" (UniqueName: \"kubernetes.io/projected/51fe14a4-6b46-4f88-be7d-b611b45961ff-kube-api-access-9vbqq\") pod \"network-check-target-26sg7\" (UID: \"51fe14a4-6b46-4f88-be7d-b611b45961ff\") " pod="openshift-network-diagnostics/network-check-target-26sg7" Jul 07 10:27:17.482189 ip-10-0-135-34 kubenswrapper[2578]: I0707 10:27:17.482003 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/f85fac08-720e-4ffd-80af-19c140c36804-original-pull-secret\") pod \"global-pull-secret-syncer-7lv79\" (UID: \"f85fac08-720e-4ffd-80af-19c140c36804\") " pod="kube-system/global-pull-secret-syncer-7lv79" Jul 07 10:27:17.482189 ip-10-0-135-34 kubenswrapper[2578]: E0707 10:27:17.482161 2578 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 07 10:27:17.482307 ip-10-0-135-34 kubenswrapper[2578]: E0707 10:27:17.482234 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f85fac08-720e-4ffd-80af-19c140c36804-original-pull-secret podName:f85fac08-720e-4ffd-80af-19c140c36804 nodeName:}" failed. No retries permitted until 2026-07-07 10:27:21.48221167 +0000 UTC m=+10.154171196 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/f85fac08-720e-4ffd-80af-19c140c36804-original-pull-secret") pod "global-pull-secret-syncer-7lv79" (UID: "f85fac08-720e-4ffd-80af-19c140c36804") : object "kube-system"/"original-pull-secret" not registered Jul 07 10:27:17.482693 ip-10-0-135-34 kubenswrapper[2578]: E0707 10:27:17.482667 2578 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 07 10:27:17.482693 ip-10-0-135-34 kubenswrapper[2578]: E0707 10:27:17.482691 2578 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 07 10:27:17.482826 ip-10-0-135-34 kubenswrapper[2578]: E0707 10:27:17.482704 2578 projected.go:196] Error preparing data for projected volume kube-api-access-9vbqq for pod openshift-network-diagnostics/network-check-target-26sg7: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 07 10:27:17.482826 ip-10-0-135-34 kubenswrapper[2578]: E0707 10:27:17.482749 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/51fe14a4-6b46-4f88-be7d-b611b45961ff-kube-api-access-9vbqq podName:51fe14a4-6b46-4f88-be7d-b611b45961ff nodeName:}" failed. No retries permitted until 2026-07-07 10:27:21.482733821 +0000 UTC m=+10.154693352 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-api-access-9vbqq" (UniqueName: "kubernetes.io/projected/51fe14a4-6b46-4f88-be7d-b611b45961ff-kube-api-access-9vbqq") pod "network-check-target-26sg7" (UID: "51fe14a4-6b46-4f88-be7d-b611b45961ff") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 07 10:27:17.818669 ip-10-0-135-34 kubenswrapper[2578]: I0707 10:27:17.818587 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-p2ks2" Jul 07 10:27:17.818832 ip-10-0-135-34 kubenswrapper[2578]: E0707 10:27:17.818737 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-p2ks2" podUID="931a7d75-9ee8-4758-9f23-3521f3db5eae" Jul 07 10:27:17.819195 ip-10-0-135-34 kubenswrapper[2578]: I0707 10:27:17.819168 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-7lv79" Jul 07 10:27:17.819306 ip-10-0-135-34 kubenswrapper[2578]: E0707 10:27:17.819282 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-7lv79" podUID="f85fac08-720e-4ffd-80af-19c140c36804" Jul 07 10:27:17.819368 ip-10-0-135-34 kubenswrapper[2578]: I0707 10:27:17.819340 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-26sg7" Jul 07 10:27:17.819439 ip-10-0-135-34 kubenswrapper[2578]: E0707 10:27:17.819412 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-26sg7" podUID="51fe14a4-6b46-4f88-be7d-b611b45961ff" Jul 07 10:27:19.818448 ip-10-0-135-34 kubenswrapper[2578]: I0707 10:27:19.818410 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-26sg7" Jul 07 10:27:19.818958 ip-10-0-135-34 kubenswrapper[2578]: E0707 10:27:19.818543 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-26sg7" podUID="51fe14a4-6b46-4f88-be7d-b611b45961ff" Jul 07 10:27:19.819034 ip-10-0-135-34 kubenswrapper[2578]: I0707 10:27:19.819014 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-7lv79" Jul 07 10:27:19.819079 ip-10-0-135-34 kubenswrapper[2578]: I0707 10:27:19.819050 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-p2ks2" Jul 07 10:27:19.819309 ip-10-0-135-34 kubenswrapper[2578]: E0707 10:27:19.819155 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-7lv79" podUID="f85fac08-720e-4ffd-80af-19c140c36804" Jul 07 10:27:19.819309 ip-10-0-135-34 kubenswrapper[2578]: E0707 10:27:19.819275 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-p2ks2" podUID="931a7d75-9ee8-4758-9f23-3521f3db5eae" Jul 07 10:27:21.423564 ip-10-0-135-34 kubenswrapper[2578]: I0707 10:27:21.422912 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/931a7d75-9ee8-4758-9f23-3521f3db5eae-metrics-certs\") pod \"network-metrics-daemon-p2ks2\" (UID: \"931a7d75-9ee8-4758-9f23-3521f3db5eae\") " pod="openshift-multus/network-metrics-daemon-p2ks2" Jul 07 10:27:21.423564 ip-10-0-135-34 kubenswrapper[2578]: E0707 10:27:21.423132 2578 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 07 10:27:21.423564 ip-10-0-135-34 kubenswrapper[2578]: E0707 10:27:21.423201 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/931a7d75-9ee8-4758-9f23-3521f3db5eae-metrics-certs podName:931a7d75-9ee8-4758-9f23-3521f3db5eae nodeName:}" failed. No retries permitted until 2026-07-07 10:27:29.423180393 +0000 UTC m=+18.095139923 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/931a7d75-9ee8-4758-9f23-3521f3db5eae-metrics-certs") pod "network-metrics-daemon-p2ks2" (UID: "931a7d75-9ee8-4758-9f23-3521f3db5eae") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 07 10:27:21.524137 ip-10-0-135-34 kubenswrapper[2578]: I0707 10:27:21.524096 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9vbqq\" (UniqueName: \"kubernetes.io/projected/51fe14a4-6b46-4f88-be7d-b611b45961ff-kube-api-access-9vbqq\") pod \"network-check-target-26sg7\" (UID: \"51fe14a4-6b46-4f88-be7d-b611b45961ff\") " pod="openshift-network-diagnostics/network-check-target-26sg7" Jul 07 10:27:21.524331 ip-10-0-135-34 kubenswrapper[2578]: I0707 10:27:21.524160 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/f85fac08-720e-4ffd-80af-19c140c36804-original-pull-secret\") pod \"global-pull-secret-syncer-7lv79\" (UID: \"f85fac08-720e-4ffd-80af-19c140c36804\") " pod="kube-system/global-pull-secret-syncer-7lv79" Jul 07 10:27:21.524400 ip-10-0-135-34 kubenswrapper[2578]: E0707 10:27:21.524321 2578 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 07 10:27:21.524400 ip-10-0-135-34 kubenswrapper[2578]: E0707 10:27:21.524353 2578 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 07 10:27:21.524400 ip-10-0-135-34 kubenswrapper[2578]: E0707 10:27:21.524366 2578 projected.go:196] Error preparing data for projected volume kube-api-access-9vbqq for pod openshift-network-diagnostics/network-check-target-26sg7: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 07 10:27:21.524560 ip-10-0-135-34 kubenswrapper[2578]: E0707 10:27:21.524417 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/51fe14a4-6b46-4f88-be7d-b611b45961ff-kube-api-access-9vbqq podName:51fe14a4-6b46-4f88-be7d-b611b45961ff nodeName:}" failed. No retries permitted until 2026-07-07 10:27:29.524402634 +0000 UTC m=+18.196362158 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-api-access-9vbqq" (UniqueName: "kubernetes.io/projected/51fe14a4-6b46-4f88-be7d-b611b45961ff-kube-api-access-9vbqq") pod "network-check-target-26sg7" (UID: "51fe14a4-6b46-4f88-be7d-b611b45961ff") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 07 10:27:21.524560 ip-10-0-135-34 kubenswrapper[2578]: E0707 10:27:21.524490 2578 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 07 10:27:21.524560 ip-10-0-135-34 kubenswrapper[2578]: E0707 10:27:21.524526 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f85fac08-720e-4ffd-80af-19c140c36804-original-pull-secret podName:f85fac08-720e-4ffd-80af-19c140c36804 nodeName:}" failed. No retries permitted until 2026-07-07 10:27:29.524513359 +0000 UTC m=+18.196472908 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/f85fac08-720e-4ffd-80af-19c140c36804-original-pull-secret") pod "global-pull-secret-syncer-7lv79" (UID: "f85fac08-720e-4ffd-80af-19c140c36804") : object "kube-system"/"original-pull-secret" not registered Jul 07 10:27:21.819382 ip-10-0-135-34 kubenswrapper[2578]: I0707 10:27:21.819241 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-7lv79" Jul 07 10:27:21.819555 ip-10-0-135-34 kubenswrapper[2578]: E0707 10:27:21.819373 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-7lv79" podUID="f85fac08-720e-4ffd-80af-19c140c36804" Jul 07 10:27:21.819788 ip-10-0-135-34 kubenswrapper[2578]: I0707 10:27:21.819746 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-26sg7" Jul 07 10:27:21.819896 ip-10-0-135-34 kubenswrapper[2578]: E0707 10:27:21.819871 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-26sg7" podUID="51fe14a4-6b46-4f88-be7d-b611b45961ff" Jul 07 10:27:21.819966 ip-10-0-135-34 kubenswrapper[2578]: I0707 10:27:21.819923 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-p2ks2" Jul 07 10:27:21.820027 ip-10-0-135-34 kubenswrapper[2578]: E0707 10:27:21.820007 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-p2ks2" podUID="931a7d75-9ee8-4758-9f23-3521f3db5eae" Jul 07 10:27:23.816953 ip-10-0-135-34 kubenswrapper[2578]: I0707 10:27:23.816911 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-p2ks2" Jul 07 10:27:23.817349 ip-10-0-135-34 kubenswrapper[2578]: I0707 10:27:23.816908 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-26sg7" Jul 07 10:27:23.817349 ip-10-0-135-34 kubenswrapper[2578]: E0707 10:27:23.817169 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-26sg7" podUID="51fe14a4-6b46-4f88-be7d-b611b45961ff" Jul 07 10:27:23.817349 ip-10-0-135-34 kubenswrapper[2578]: E0707 10:27:23.817058 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-p2ks2" podUID="931a7d75-9ee8-4758-9f23-3521f3db5eae" Jul 07 10:27:23.817349 ip-10-0-135-34 kubenswrapper[2578]: I0707 10:27:23.817237 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-7lv79" Jul 07 10:27:23.817349 ip-10-0-135-34 kubenswrapper[2578]: E0707 10:27:23.817309 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-7lv79" podUID="f85fac08-720e-4ffd-80af-19c140c36804" Jul 07 10:27:25.815710 ip-10-0-135-34 kubenswrapper[2578]: I0707 10:27:25.815675 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-26sg7" Jul 07 10:27:25.816142 ip-10-0-135-34 kubenswrapper[2578]: I0707 10:27:25.815675 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-p2ks2" Jul 07 10:27:25.816142 ip-10-0-135-34 kubenswrapper[2578]: E0707 10:27:25.815829 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-26sg7" podUID="51fe14a4-6b46-4f88-be7d-b611b45961ff" Jul 07 10:27:25.816142 ip-10-0-135-34 kubenswrapper[2578]: I0707 10:27:25.815682 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-7lv79" Jul 07 10:27:25.816142 ip-10-0-135-34 kubenswrapper[2578]: E0707 10:27:25.815943 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-p2ks2" podUID="931a7d75-9ee8-4758-9f23-3521f3db5eae" Jul 07 10:27:25.816142 ip-10-0-135-34 kubenswrapper[2578]: E0707 10:27:25.816015 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-7lv79" podUID="f85fac08-720e-4ffd-80af-19c140c36804" Jul 07 10:27:27.815911 ip-10-0-135-34 kubenswrapper[2578]: I0707 10:27:27.815871 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-26sg7" Jul 07 10:27:27.816345 ip-10-0-135-34 kubenswrapper[2578]: I0707 10:27:27.815870 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-p2ks2" Jul 07 10:27:27.816345 ip-10-0-135-34 kubenswrapper[2578]: I0707 10:27:27.816011 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-7lv79" Jul 07 10:27:27.816345 ip-10-0-135-34 kubenswrapper[2578]: E0707 10:27:27.816008 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-26sg7" podUID="51fe14a4-6b46-4f88-be7d-b611b45961ff" Jul 07 10:27:27.816345 ip-10-0-135-34 kubenswrapper[2578]: E0707 10:27:27.816099 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-p2ks2" podUID="931a7d75-9ee8-4758-9f23-3521f3db5eae" Jul 07 10:27:27.816345 ip-10-0-135-34 kubenswrapper[2578]: E0707 10:27:27.816192 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-7lv79" podUID="f85fac08-720e-4ffd-80af-19c140c36804" Jul 07 10:27:29.486865 ip-10-0-135-34 kubenswrapper[2578]: I0707 10:27:29.486830 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/931a7d75-9ee8-4758-9f23-3521f3db5eae-metrics-certs\") pod \"network-metrics-daemon-p2ks2\" (UID: \"931a7d75-9ee8-4758-9f23-3521f3db5eae\") " pod="openshift-multus/network-metrics-daemon-p2ks2" Jul 07 10:27:29.487314 ip-10-0-135-34 kubenswrapper[2578]: E0707 10:27:29.487019 2578 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 07 10:27:29.487314 ip-10-0-135-34 kubenswrapper[2578]: E0707 10:27:29.487100 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/931a7d75-9ee8-4758-9f23-3521f3db5eae-metrics-certs podName:931a7d75-9ee8-4758-9f23-3521f3db5eae nodeName:}" failed. No retries permitted until 2026-07-07 10:27:45.48707867 +0000 UTC m=+34.159038209 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/931a7d75-9ee8-4758-9f23-3521f3db5eae-metrics-certs") pod "network-metrics-daemon-p2ks2" (UID: "931a7d75-9ee8-4758-9f23-3521f3db5eae") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 07 10:27:29.587169 ip-10-0-135-34 kubenswrapper[2578]: I0707 10:27:29.587131 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9vbqq\" (UniqueName: \"kubernetes.io/projected/51fe14a4-6b46-4f88-be7d-b611b45961ff-kube-api-access-9vbqq\") pod \"network-check-target-26sg7\" (UID: \"51fe14a4-6b46-4f88-be7d-b611b45961ff\") " pod="openshift-network-diagnostics/network-check-target-26sg7" Jul 07 10:27:29.587352 ip-10-0-135-34 kubenswrapper[2578]: I0707 10:27:29.587192 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/f85fac08-720e-4ffd-80af-19c140c36804-original-pull-secret\") pod \"global-pull-secret-syncer-7lv79\" (UID: \"f85fac08-720e-4ffd-80af-19c140c36804\") " pod="kube-system/global-pull-secret-syncer-7lv79" Jul 07 10:27:29.587352 ip-10-0-135-34 kubenswrapper[2578]: E0707 10:27:29.587334 2578 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 07 10:27:29.587428 ip-10-0-135-34 kubenswrapper[2578]: E0707 10:27:29.587363 2578 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 07 10:27:29.587428 ip-10-0-135-34 kubenswrapper[2578]: E0707 10:27:29.587375 2578 projected.go:196] Error preparing data for projected volume kube-api-access-9vbqq for pod openshift-network-diagnostics/network-check-target-26sg7: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 07 10:27:29.587428 ip-10-0-135-34 kubenswrapper[2578]: E0707 10:27:29.587422 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/51fe14a4-6b46-4f88-be7d-b611b45961ff-kube-api-access-9vbqq podName:51fe14a4-6b46-4f88-be7d-b611b45961ff nodeName:}" failed. No retries permitted until 2026-07-07 10:27:45.587408778 +0000 UTC m=+34.259368314 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "kube-api-access-9vbqq" (UniqueName: "kubernetes.io/projected/51fe14a4-6b46-4f88-be7d-b611b45961ff-kube-api-access-9vbqq") pod "network-check-target-26sg7" (UID: "51fe14a4-6b46-4f88-be7d-b611b45961ff") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 07 10:27:29.587575 ip-10-0-135-34 kubenswrapper[2578]: E0707 10:27:29.587334 2578 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 07 10:27:29.587575 ip-10-0-135-34 kubenswrapper[2578]: E0707 10:27:29.587481 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f85fac08-720e-4ffd-80af-19c140c36804-original-pull-secret podName:f85fac08-720e-4ffd-80af-19c140c36804 nodeName:}" failed. No retries permitted until 2026-07-07 10:27:45.587466998 +0000 UTC m=+34.259426520 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/f85fac08-720e-4ffd-80af-19c140c36804-original-pull-secret") pod "global-pull-secret-syncer-7lv79" (UID: "f85fac08-720e-4ffd-80af-19c140c36804") : object "kube-system"/"original-pull-secret" not registered Jul 07 10:27:29.816291 ip-10-0-135-34 kubenswrapper[2578]: I0707 10:27:29.816204 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-p2ks2" Jul 07 10:27:29.816429 ip-10-0-135-34 kubenswrapper[2578]: E0707 10:27:29.816335 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-p2ks2" podUID="931a7d75-9ee8-4758-9f23-3521f3db5eae" Jul 07 10:27:29.816429 ip-10-0-135-34 kubenswrapper[2578]: I0707 10:27:29.816385 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-7lv79" Jul 07 10:27:29.816495 ip-10-0-135-34 kubenswrapper[2578]: E0707 10:27:29.816463 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-7lv79" podUID="f85fac08-720e-4ffd-80af-19c140c36804" Jul 07 10:27:29.816495 ip-10-0-135-34 kubenswrapper[2578]: I0707 10:27:29.816204 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-26sg7" Jul 07 10:27:29.816552 ip-10-0-135-34 kubenswrapper[2578]: E0707 10:27:29.816518 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-26sg7" podUID="51fe14a4-6b46-4f88-be7d-b611b45961ff" Jul 07 10:27:31.817923 ip-10-0-135-34 kubenswrapper[2578]: I0707 10:27:31.817887 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-26sg7" Jul 07 10:27:31.818362 ip-10-0-135-34 kubenswrapper[2578]: I0707 10:27:31.817992 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-7lv79" Jul 07 10:27:31.818362 ip-10-0-135-34 kubenswrapper[2578]: E0707 10:27:31.818008 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-26sg7" podUID="51fe14a4-6b46-4f88-be7d-b611b45961ff" Jul 07 10:27:31.818362 ip-10-0-135-34 kubenswrapper[2578]: E0707 10:27:31.818079 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-7lv79" podUID="f85fac08-720e-4ffd-80af-19c140c36804" Jul 07 10:27:31.818362 ip-10-0-135-34 kubenswrapper[2578]: I0707 10:27:31.818114 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-p2ks2" Jul 07 10:27:31.818362 ip-10-0-135-34 kubenswrapper[2578]: E0707 10:27:31.818214 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-p2ks2" podUID="931a7d75-9ee8-4758-9f23-3521f3db5eae" Jul 07 10:27:33.815532 ip-10-0-135-34 kubenswrapper[2578]: I0707 10:27:33.815491 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-26sg7" Jul 07 10:27:33.815532 ip-10-0-135-34 kubenswrapper[2578]: I0707 10:27:33.815526 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-7lv79" Jul 07 10:27:33.816157 ip-10-0-135-34 kubenswrapper[2578]: I0707 10:27:33.815496 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-p2ks2" Jul 07 10:27:33.816157 ip-10-0-135-34 kubenswrapper[2578]: E0707 10:27:33.815631 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-26sg7" podUID="51fe14a4-6b46-4f88-be7d-b611b45961ff" Jul 07 10:27:33.816157 ip-10-0-135-34 kubenswrapper[2578]: E0707 10:27:33.815734 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-p2ks2" podUID="931a7d75-9ee8-4758-9f23-3521f3db5eae" Jul 07 10:27:33.816157 ip-10-0-135-34 kubenswrapper[2578]: E0707 10:27:33.815846 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-7lv79" podUID="f85fac08-720e-4ffd-80af-19c140c36804" Jul 07 10:27:35.820063 ip-10-0-135-34 kubenswrapper[2578]: I0707 10:27:35.819635 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-7lv79" Jul 07 10:27:35.820842 ip-10-0-135-34 kubenswrapper[2578]: I0707 10:27:35.819686 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-p2ks2" Jul 07 10:27:35.820842 ip-10-0-135-34 kubenswrapper[2578]: E0707 10:27:35.820168 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-7lv79" podUID="f85fac08-720e-4ffd-80af-19c140c36804" Jul 07 10:27:35.820842 ip-10-0-135-34 kubenswrapper[2578]: I0707 10:27:35.819686 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-26sg7" Jul 07 10:27:35.820842 ip-10-0-135-34 kubenswrapper[2578]: E0707 10:27:35.820241 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-p2ks2" podUID="931a7d75-9ee8-4758-9f23-3521f3db5eae" Jul 07 10:27:35.820842 ip-10-0-135-34 kubenswrapper[2578]: E0707 10:27:35.820291 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-26sg7" podUID="51fe14a4-6b46-4f88-be7d-b611b45961ff" Jul 07 10:27:35.938547 ip-10-0-135-34 kubenswrapper[2578]: I0707 10:27:35.938523 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-qhhw7_aca7528d-ddbe-4282-979f-1b496011f0e1/ovn-acl-logging/0.log" Jul 07 10:27:35.938901 ip-10-0-135-34 kubenswrapper[2578]: I0707 10:27:35.938879 2578 generic.go:356] "Generic (PLEG): container finished" podID="aca7528d-ddbe-4282-979f-1b496011f0e1" containerID="0fbad1d286819e02338d30b50fe5aa043f3dd107591b44c2a35f6d0a31e79311" exitCode=1 Jul 07 10:27:35.938998 ip-10-0-135-34 kubenswrapper[2578]: I0707 10:27:35.938943 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-qhhw7" event={"ID":"aca7528d-ddbe-4282-979f-1b496011f0e1","Type":"ContainerStarted","Data":"964b0fe36d91e56cf96e59e73a4e768567508c5009f0a3d57f57ef3ef7e2332c"} Jul 07 10:27:35.939063 ip-10-0-135-34 kubenswrapper[2578]: I0707 10:27:35.939036 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-qhhw7" event={"ID":"aca7528d-ddbe-4282-979f-1b496011f0e1","Type":"ContainerStarted","Data":"70775cefe9dd99a6d579ab55272a2ffef2a9de109a4d9c0e9bbfd3745e721807"} Jul 07 10:27:35.939123 ip-10-0-135-34 kubenswrapper[2578]: I0707 10:27:35.939088 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-qhhw7" event={"ID":"aca7528d-ddbe-4282-979f-1b496011f0e1","Type":"ContainerStarted","Data":"df5f99349bcb80e9012ea5069a37410620cf0ac176ef13e23497326d36c16aa2"} Jul 07 10:27:35.939176 ip-10-0-135-34 kubenswrapper[2578]: I0707 10:27:35.939139 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-qhhw7" event={"ID":"aca7528d-ddbe-4282-979f-1b496011f0e1","Type":"ContainerStarted","Data":"c795285be51f02e9b894e695225d0f41d58ad2a04fdf416f41493bb4620c3aa4"} Jul 07 10:27:35.939229 ip-10-0-135-34 kubenswrapper[2578]: I0707 10:27:35.939188 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-qhhw7" event={"ID":"aca7528d-ddbe-4282-979f-1b496011f0e1","Type":"ContainerDied","Data":"0fbad1d286819e02338d30b50fe5aa043f3dd107591b44c2a35f6d0a31e79311"} Jul 07 10:27:35.939282 ip-10-0-135-34 kubenswrapper[2578]: I0707 10:27:35.939239 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-qhhw7" event={"ID":"aca7528d-ddbe-4282-979f-1b496011f0e1","Type":"ContainerStarted","Data":"8cbbe805af5286b4121842f31e6f2939bd360f7bfb237ee5cc9e5c057027569a"} Jul 07 10:27:35.940378 ip-10-0-135-34 kubenswrapper[2578]: I0707 10:27:35.940357 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-dll6x" event={"ID":"b1617485-94f6-4eba-95e9-866fc2400643","Type":"ContainerStarted","Data":"0370f55196a8b602a8ba3350c3e69faf28304f4ec7868a400e92800292f9c6c9"} Jul 07 10:27:35.941747 ip-10-0-135-34 kubenswrapper[2578]: I0707 10:27:35.941723 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-lj6lw" event={"ID":"1e1b149f-fe01-49d2-8387-4cf7aea820e7","Type":"ContainerStarted","Data":"917ad7158238639d50f430f94fdaeda0f31cd9596e7582659f88137cd392e5e2"} Jul 07 10:27:35.943131 ip-10-0-135-34 kubenswrapper[2578]: I0707 10:27:35.943105 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-pb2p8" event={"ID":"025a30f8-f923-4bc8-adf5-7ef97ee3bfd3","Type":"ContainerStarted","Data":"ba57f34b90c962028cf4b0b0eed8da7d0693f928b294eb56c82aab894ddd215b"} Jul 07 10:27:35.944640 ip-10-0-135-34 kubenswrapper[2578]: I0707 10:27:35.944610 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-t82fl" event={"ID":"11079224-e507-45d1-9fbb-da5043002c51","Type":"ContainerStarted","Data":"732d01cc0aedc286dbba0c068daf1adf84c4a860aea8e0f0781815cc12bf3899"} Jul 07 10:27:35.948376 ip-10-0-135-34 kubenswrapper[2578]: I0707 10:27:35.946949 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-hwz82" event={"ID":"2410fb3f-6b21-40b9-b23b-ffdbbdb132a7","Type":"ContainerStarted","Data":"bdb3794ce904902f08469dad70f60a1140ff732a676a4de5ca5bf0964aee28f4"} Jul 07 10:27:35.948888 ip-10-0-135-34 kubenswrapper[2578]: I0707 10:27:35.948732 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-rdrnq" event={"ID":"dfa64e14-3729-4f28-870f-eeab6cec9944","Type":"ContainerStarted","Data":"cf88bd57da0e8dc87e3662a97af8673c7bf79ead34f1c0ef7c846ba6f063aac6"} Jul 07 10:27:35.950634 ip-10-0-135-34 kubenswrapper[2578]: I0707 10:27:35.950604 2578 generic.go:356] "Generic (PLEG): container finished" podID="a415d240-ff8a-4b08-aeb6-b4018195c46b" containerID="d3e332de1485eadf07e83ec79f45f3dcd5a64a4f6aa450c7880fda480d667b46" exitCode=0 Jul 07 10:27:35.950739 ip-10-0-135-34 kubenswrapper[2578]: I0707 10:27:35.950636 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-l8d7k" event={"ID":"a415d240-ff8a-4b08-aeb6-b4018195c46b","Type":"ContainerDied","Data":"d3e332de1485eadf07e83ec79f45f3dcd5a64a4f6aa450c7880fda480d667b46"} Jul 07 10:27:35.964397 ip-10-0-135-34 kubenswrapper[2578]: I0707 10:27:35.964345 2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-node-tuning-operator/tuned-lj6lw" podStartSLOduration=4.106285233 podStartE2EDuration="23.964329584s" totalImagesPullingTime="19.858044351s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-07 10:27:12 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 10:27:35.964140325 +0000 UTC m=+24.636099871" watchObservedRunningTime="2026-07-07 10:27:35.964329584 +0000 UTC m=+24.636289136" Jul 07 10:27:35.986483 ip-10-0-135-34 kubenswrapper[2578]: I0707 10:27:35.986426 2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/konnectivity-agent-t82fl" podStartSLOduration=12.746911796 podStartE2EDuration="23.986405228s" totalImagesPullingTime="11.239493432s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-07 10:27:12 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 10:27:35.986221589 +0000 UTC m=+24.658181133" watchObservedRunningTime="2026-07-07 10:27:35.986405228 +0000 UTC m=+24.658364775" Jul 07 10:27:36.006967 ip-10-0-135-34 kubenswrapper[2578]: I0707 10:27:36.006896 2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/node-ca-pb2p8" podStartSLOduration=4.167030758 podStartE2EDuration="24.006878742s" totalImagesPullingTime="19.839847984s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-07 10:27:12 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 10:27:36.00621253 +0000 UTC m=+24.678172074" watchObservedRunningTime="2026-07-07 10:27:36.006878742 +0000 UTC m=+24.678838285" Jul 07 10:27:36.013643 ip-10-0-135-34 kubenswrapper[2578]: I0707 10:27:36.013611 2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kube-system/konnectivity-agent-t82fl" Jul 07 10:27:36.014228 ip-10-0-135-34 kubenswrapper[2578]: I0707 10:27:36.014203 2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kube-system/konnectivity-agent-t82fl" Jul 07 10:27:36.066709 ip-10-0-135-34 kubenswrapper[2578]: I0707 10:27:36.066654 2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/node-resolver-rdrnq" podStartSLOduration=4.443237247 podStartE2EDuration="24.066632339s" totalImagesPullingTime="19.623395092s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-07 10:27:12 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 10:27:36.046167215 +0000 UTC m=+24.718126776" watchObservedRunningTime="2026-07-07 10:27:36.066632339 +0000 UTC m=+24.738591888" Jul 07 10:27:36.066869 ip-10-0-135-34 kubenswrapper[2578]: I0707 10:27:36.066760 2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-hwz82" podStartSLOduration=4.183712939 podStartE2EDuration="24.066753078s" totalImagesPullingTime="19.883040139s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-07 10:27:12 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 10:27:36.06649206 +0000 UTC m=+24.738451656" watchObservedRunningTime="2026-07-07 10:27:36.066753078 +0000 UTC m=+24.738712622" Jul 07 10:27:36.216866 ip-10-0-135-34 kubenswrapper[2578]: I0707 10:27:36.216841 2578 plugin_watcher.go:194] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock" Jul 07 10:27:36.819345 ip-10-0-135-34 kubenswrapper[2578]: I0707 10:27:36.819188 2578 reconciler.go:161] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-07-07T10:27:36.216860042Z","UUID":"8acd3126-acfb-4b53-9614-43658af39e00","Handler":null,"Name":"","Endpoint":""} Jul 07 10:27:36.821069 ip-10-0-135-34 kubenswrapper[2578]: I0707 10:27:36.821044 2578 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0 Jul 07 10:27:36.821462 ip-10-0-135-34 kubenswrapper[2578]: I0707 10:27:36.821093 2578 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock Jul 07 10:27:36.955737 ip-10-0-135-34 kubenswrapper[2578]: I0707 10:27:36.955691 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-lr7dk" event={"ID":"609c4941-f311-408a-82ec-1d6b1a54572d","Type":"ContainerStarted","Data":"d0b7e6ec1ec79efcce7033ea54fc9d2d92162ecda5dbab4e8ed807d78c0a7729"} Jul 07 10:27:36.958508 ip-10-0-135-34 kubenswrapper[2578]: I0707 10:27:36.958472 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-dll6x" event={"ID":"b1617485-94f6-4eba-95e9-866fc2400643","Type":"ContainerStarted","Data":"b85c964403be9bf6d2217a19ffc324c4450aadb05d90bf91fcc156be364754bf"} Jul 07 10:27:36.971560 ip-10-0-135-34 kubenswrapper[2578]: I0707 10:27:36.971502 2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-operator/iptables-alerter-lr7dk" podStartSLOduration=5.112628379 podStartE2EDuration="24.971480071s" totalImagesPullingTime="19.858851692s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-07 10:27:12 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 10:27:36.971318252 +0000 UTC m=+25.643277797" watchObservedRunningTime="2026-07-07 10:27:36.971480071 +0000 UTC m=+25.643439618" Jul 07 10:27:37.820293 ip-10-0-135-34 kubenswrapper[2578]: I0707 10:27:37.820108 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-26sg7" Jul 07 10:27:37.820465 ip-10-0-135-34 kubenswrapper[2578]: I0707 10:27:37.820177 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-7lv79" Jul 07 10:27:37.820465 ip-10-0-135-34 kubenswrapper[2578]: E0707 10:27:37.820394 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-26sg7" podUID="51fe14a4-6b46-4f88-be7d-b611b45961ff" Jul 07 10:27:37.820465 ip-10-0-135-34 kubenswrapper[2578]: I0707 10:27:37.820177 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-p2ks2" Jul 07 10:27:37.820628 ip-10-0-135-34 kubenswrapper[2578]: E0707 10:27:37.820476 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-7lv79" podUID="f85fac08-720e-4ffd-80af-19c140c36804" Jul 07 10:27:37.820628 ip-10-0-135-34 kubenswrapper[2578]: E0707 10:27:37.820534 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-p2ks2" podUID="931a7d75-9ee8-4758-9f23-3521f3db5eae" Jul 07 10:27:37.964389 ip-10-0-135-34 kubenswrapper[2578]: I0707 10:27:37.964289 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-qhhw7_aca7528d-ddbe-4282-979f-1b496011f0e1/ovn-acl-logging/0.log" Jul 07 10:27:37.965171 ip-10-0-135-34 kubenswrapper[2578]: I0707 10:27:37.964812 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-qhhw7" event={"ID":"aca7528d-ddbe-4282-979f-1b496011f0e1","Type":"ContainerStarted","Data":"48ce13139300268d05033bac45b46cd0de5cb970122149b12fc614948e5208df"} Jul 07 10:27:37.967700 ip-10-0-135-34 kubenswrapper[2578]: I0707 10:27:37.967663 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-dll6x" event={"ID":"b1617485-94f6-4eba-95e9-866fc2400643","Type":"ContainerStarted","Data":"2a44438a49ab4096c2e7ba4380fe92c9c2514d4b8f213e63f2c924ec398f9006"} Jul 07 10:27:37.967885 ip-10-0-135-34 kubenswrapper[2578]: I0707 10:27:37.967765 2578 prober_manager.go:312] "Failed to trigger a manual run" probe="Readiness" Jul 07 10:27:37.985238 ip-10-0-135-34 kubenswrapper[2578]: I0707 10:27:37.985181 2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-dll6x" podStartSLOduration=4.838985231 podStartE2EDuration="25.985164684s" totalImagesPullingTime="21.146179453s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-07 10:27:12 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 10:27:37.984932774 +0000 UTC m=+26.656892315" watchObservedRunningTime="2026-07-07 10:27:37.985164684 +0000 UTC m=+26.657124228" Jul 07 10:27:39.820361 ip-10-0-135-34 kubenswrapper[2578]: I0707 10:27:39.819964 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-7lv79" Jul 07 10:27:39.820361 ip-10-0-135-34 kubenswrapper[2578]: I0707 10:27:39.820075 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-26sg7" Jul 07 10:27:39.820361 ip-10-0-135-34 kubenswrapper[2578]: E0707 10:27:39.820076 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-7lv79" podUID="f85fac08-720e-4ffd-80af-19c140c36804" Jul 07 10:27:39.820361 ip-10-0-135-34 kubenswrapper[2578]: I0707 10:27:39.820083 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-p2ks2" Jul 07 10:27:39.820361 ip-10-0-135-34 kubenswrapper[2578]: E0707 10:27:39.820173 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-26sg7" podUID="51fe14a4-6b46-4f88-be7d-b611b45961ff" Jul 07 10:27:39.820361 ip-10-0-135-34 kubenswrapper[2578]: E0707 10:27:39.820253 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-p2ks2" podUID="931a7d75-9ee8-4758-9f23-3521f3db5eae" Jul 07 10:27:40.050046 ip-10-0-135-34 kubenswrapper[2578]: I0707 10:27:40.049965 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kube-system/konnectivity-agent-t82fl" Jul 07 10:27:40.050197 ip-10-0-135-34 kubenswrapper[2578]: I0707 10:27:40.050096 2578 prober_manager.go:312] "Failed to trigger a manual run" probe="Readiness" Jul 07 10:27:40.050544 ip-10-0-135-34 kubenswrapper[2578]: I0707 10:27:40.050519 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-t82fl" Jul 07 10:27:40.976614 ip-10-0-135-34 kubenswrapper[2578]: I0707 10:27:40.976420 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-l8d7k" event={"ID":"a415d240-ff8a-4b08-aeb6-b4018195c46b","Type":"ContainerStarted","Data":"08937cd5f0863fa9e08a0fbe7b2b58a52250cef1bf18a434d0c22a01a8700f33"} Jul 07 10:27:40.979894 ip-10-0-135-34 kubenswrapper[2578]: I0707 10:27:40.979871 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-qhhw7_aca7528d-ddbe-4282-979f-1b496011f0e1/ovn-acl-logging/0.log" Jul 07 10:27:40.980322 ip-10-0-135-34 kubenswrapper[2578]: I0707 10:27:40.980294 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-qhhw7" event={"ID":"aca7528d-ddbe-4282-979f-1b496011f0e1","Type":"ContainerStarted","Data":"9c2eede626f671b8c0dd092f02427b05d8d47d40b580213613624067847dc42c"} Jul 07 10:27:40.980833 ip-10-0-135-34 kubenswrapper[2578]: I0707 10:27:40.980816 2578 scope.go:117] "RemoveContainer" containerID="0fbad1d286819e02338d30b50fe5aa043f3dd107591b44c2a35f6d0a31e79311" Jul 07 10:27:41.817544 ip-10-0-135-34 kubenswrapper[2578]: I0707 10:27:41.817462 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-26sg7" Jul 07 10:27:41.817717 ip-10-0-135-34 kubenswrapper[2578]: I0707 10:27:41.817548 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-p2ks2" Jul 07 10:27:41.817717 ip-10-0-135-34 kubenswrapper[2578]: I0707 10:27:41.817588 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-7lv79" Jul 07 10:27:41.817717 ip-10-0-135-34 kubenswrapper[2578]: E0707 10:27:41.817593 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-26sg7" podUID="51fe14a4-6b46-4f88-be7d-b611b45961ff" Jul 07 10:27:41.817717 ip-10-0-135-34 kubenswrapper[2578]: E0707 10:27:41.817659 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-7lv79" podUID="f85fac08-720e-4ffd-80af-19c140c36804" Jul 07 10:27:41.817937 ip-10-0-135-34 kubenswrapper[2578]: E0707 10:27:41.817755 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-p2ks2" podUID="931a7d75-9ee8-4758-9f23-3521f3db5eae" Jul 07 10:27:41.984440 ip-10-0-135-34 kubenswrapper[2578]: I0707 10:27:41.984404 2578 generic.go:356] "Generic (PLEG): container finished" podID="a415d240-ff8a-4b08-aeb6-b4018195c46b" containerID="08937cd5f0863fa9e08a0fbe7b2b58a52250cef1bf18a434d0c22a01a8700f33" exitCode=0 Jul 07 10:27:41.984991 ip-10-0-135-34 kubenswrapper[2578]: I0707 10:27:41.984499 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-l8d7k" event={"ID":"a415d240-ff8a-4b08-aeb6-b4018195c46b","Type":"ContainerDied","Data":"08937cd5f0863fa9e08a0fbe7b2b58a52250cef1bf18a434d0c22a01a8700f33"} Jul 07 10:27:41.988196 ip-10-0-135-34 kubenswrapper[2578]: I0707 10:27:41.988176 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-qhhw7_aca7528d-ddbe-4282-979f-1b496011f0e1/ovn-acl-logging/0.log" Jul 07 10:27:41.988574 ip-10-0-135-34 kubenswrapper[2578]: I0707 10:27:41.988553 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-qhhw7" event={"ID":"aca7528d-ddbe-4282-979f-1b496011f0e1","Type":"ContainerStarted","Data":"44681a22be9be9c09eecf9102b52f9946ef2ae14a5aed8e2f0425e93f3b50d3b"} Jul 07 10:27:41.988831 ip-10-0-135-34 kubenswrapper[2578]: I0707 10:27:41.988801 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-qhhw7" Jul 07 10:27:41.988934 ip-10-0-135-34 kubenswrapper[2578]: I0707 10:27:41.988869 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-qhhw7" Jul 07 10:27:41.988934 ip-10-0-135-34 kubenswrapper[2578]: I0707 10:27:41.988925 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-qhhw7" Jul 07 10:27:42.004848 ip-10-0-135-34 kubenswrapper[2578]: I0707 10:27:42.004822 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-qhhw7" Jul 07 10:27:42.004989 ip-10-0-135-34 kubenswrapper[2578]: I0707 10:27:42.004937 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-qhhw7" Jul 07 10:27:42.055085 ip-10-0-135-34 kubenswrapper[2578]: I0707 10:27:42.055036 2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ovn-kubernetes/ovnkube-node-qhhw7" podStartSLOduration=9.141030765 podStartE2EDuration="29.055018632s" totalImagesPullingTime="19.913987867s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-07 10:27:13 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 10:27:42.053260098 +0000 UTC m=+30.725219645" watchObservedRunningTime="2026-07-07 10:27:42.055018632 +0000 UTC m=+30.726978176" Jul 07 10:27:42.687328 ip-10-0-135-34 kubenswrapper[2578]: E0707 10:27:42.687285 2578 log.go:32] "PullImage from image service failed" err="rpc error: code = Unknown desc = unable to pull image or OCI artifact: pull image err: copying system image from manifest list: parsing image configuration: fetching blob: received unexpected HTTP status: 503 Service Unavailable; artifact err: provided artifact is a container image" image="quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:73de210988add57998b2362d226fd6093b3b80c6ae31e2987402f4fecab193b8" Jul 07 10:27:42.687492 ip-10-0-135-34 kubenswrapper[2578]: E0707 10:27:42.687393 2578 kuberuntime_manager.go:1459] "Unhandled Error" err="init container bond-cni-plugin start failed in pod multus-additional-cni-plugins-l8d7k_openshift-multus(a415d240-ff8a-4b08-aeb6-b4018195c46b): ErrImagePull: unable to pull image or OCI artifact: pull image err: copying system image from manifest list: parsing image configuration: fetching blob: received unexpected HTTP status: 503 Service Unavailable; artifact err: provided artifact is a container image" logger="UnhandledError" Jul 07 10:27:42.687492 ip-10-0-135-34 kubenswrapper[2578]: E0707 10:27:42.687425 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"bond-cni-plugin\" with ErrImagePull: \"unable to pull image or OCI artifact: pull image err: copying system image from manifest list: parsing image configuration: fetching blob: received unexpected HTTP status: 503 Service Unavailable; artifact err: provided artifact is a container image\"" pod="openshift-multus/multus-additional-cni-plugins-l8d7k" podUID="a415d240-ff8a-4b08-aeb6-b4018195c46b" Jul 07 10:27:42.992851 ip-10-0-135-34 kubenswrapper[2578]: E0707 10:27:42.992639 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"bond-cni-plugin\" with ImagePullBackOff: \"Back-off pulling image \\\"quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:73de210988add57998b2362d226fd6093b3b80c6ae31e2987402f4fecab193b8\\\": ErrImagePull: unable to pull image or OCI artifact: pull image err: copying system image from manifest list: parsing image configuration: fetching blob: received unexpected HTTP status: 503 Service Unavailable; artifact err: provided artifact is a container image\"" pod="openshift-multus/multus-additional-cni-plugins-l8d7k" podUID="a415d240-ff8a-4b08-aeb6-b4018195c46b" Jul 07 10:27:43.060306 ip-10-0-135-34 kubenswrapper[2578]: I0707 10:27:43.060269 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-26sg7"] Jul 07 10:27:43.060478 ip-10-0-135-34 kubenswrapper[2578]: I0707 10:27:43.060403 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-26sg7" Jul 07 10:27:43.060535 ip-10-0-135-34 kubenswrapper[2578]: E0707 10:27:43.060491 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-26sg7" podUID="51fe14a4-6b46-4f88-be7d-b611b45961ff" Jul 07 10:27:43.060912 ip-10-0-135-34 kubenswrapper[2578]: I0707 10:27:43.060890 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-7lv79"] Jul 07 10:27:43.061003 ip-10-0-135-34 kubenswrapper[2578]: I0707 10:27:43.060991 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-7lv79" Jul 07 10:27:43.061106 ip-10-0-135-34 kubenswrapper[2578]: E0707 10:27:43.061087 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-7lv79" podUID="f85fac08-720e-4ffd-80af-19c140c36804" Jul 07 10:27:43.073084 ip-10-0-135-34 kubenswrapper[2578]: I0707 10:27:43.073059 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-p2ks2"] Jul 07 10:27:43.073198 ip-10-0-135-34 kubenswrapper[2578]: I0707 10:27:43.073168 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-p2ks2" Jul 07 10:27:43.073307 ip-10-0-135-34 kubenswrapper[2578]: E0707 10:27:43.073282 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-p2ks2" podUID="931a7d75-9ee8-4758-9f23-3521f3db5eae" Jul 07 10:27:44.815731 ip-10-0-135-34 kubenswrapper[2578]: I0707 10:27:44.815641 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-7lv79" Jul 07 10:27:44.816130 ip-10-0-135-34 kubenswrapper[2578]: I0707 10:27:44.815646 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-26sg7" Jul 07 10:27:44.816130 ip-10-0-135-34 kubenswrapper[2578]: I0707 10:27:44.815760 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-p2ks2" Jul 07 10:27:44.816130 ip-10-0-135-34 kubenswrapper[2578]: E0707 10:27:44.815758 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-7lv79" podUID="f85fac08-720e-4ffd-80af-19c140c36804" Jul 07 10:27:44.816130 ip-10-0-135-34 kubenswrapper[2578]: E0707 10:27:44.815846 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-26sg7" podUID="51fe14a4-6b46-4f88-be7d-b611b45961ff" Jul 07 10:27:44.816130 ip-10-0-135-34 kubenswrapper[2578]: E0707 10:27:44.815931 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-p2ks2" podUID="931a7d75-9ee8-4758-9f23-3521f3db5eae" Jul 07 10:27:45.501515 ip-10-0-135-34 kubenswrapper[2578]: I0707 10:27:45.501477 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/931a7d75-9ee8-4758-9f23-3521f3db5eae-metrics-certs\") pod \"network-metrics-daemon-p2ks2\" (UID: \"931a7d75-9ee8-4758-9f23-3521f3db5eae\") " pod="openshift-multus/network-metrics-daemon-p2ks2" Jul 07 10:27:45.501697 ip-10-0-135-34 kubenswrapper[2578]: E0707 10:27:45.501675 2578 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 07 10:27:45.501755 ip-10-0-135-34 kubenswrapper[2578]: E0707 10:27:45.501744 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/931a7d75-9ee8-4758-9f23-3521f3db5eae-metrics-certs podName:931a7d75-9ee8-4758-9f23-3521f3db5eae nodeName:}" failed. No retries permitted until 2026-07-07 10:28:17.501729076 +0000 UTC m=+66.173688601 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/931a7d75-9ee8-4758-9f23-3521f3db5eae-metrics-certs") pod "network-metrics-daemon-p2ks2" (UID: "931a7d75-9ee8-4758-9f23-3521f3db5eae") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 07 10:27:45.601827 ip-10-0-135-34 kubenswrapper[2578]: I0707 10:27:45.601755 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9vbqq\" (UniqueName: \"kubernetes.io/projected/51fe14a4-6b46-4f88-be7d-b611b45961ff-kube-api-access-9vbqq\") pod \"network-check-target-26sg7\" (UID: \"51fe14a4-6b46-4f88-be7d-b611b45961ff\") " pod="openshift-network-diagnostics/network-check-target-26sg7" Jul 07 10:27:45.601827 ip-10-0-135-34 kubenswrapper[2578]: I0707 10:27:45.601832 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/f85fac08-720e-4ffd-80af-19c140c36804-original-pull-secret\") pod \"global-pull-secret-syncer-7lv79\" (UID: \"f85fac08-720e-4ffd-80af-19c140c36804\") " pod="kube-system/global-pull-secret-syncer-7lv79" Jul 07 10:27:45.602042 ip-10-0-135-34 kubenswrapper[2578]: E0707 10:27:45.601931 2578 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 07 10:27:45.602042 ip-10-0-135-34 kubenswrapper[2578]: E0707 10:27:45.601937 2578 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 07 10:27:45.602042 ip-10-0-135-34 kubenswrapper[2578]: E0707 10:27:45.601952 2578 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 07 10:27:45.602042 ip-10-0-135-34 kubenswrapper[2578]: E0707 10:27:45.601962 2578 projected.go:196] Error preparing data for projected volume kube-api-access-9vbqq for pod openshift-network-diagnostics/network-check-target-26sg7: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 07 10:27:45.602042 ip-10-0-135-34 kubenswrapper[2578]: E0707 10:27:45.601994 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f85fac08-720e-4ffd-80af-19c140c36804-original-pull-secret podName:f85fac08-720e-4ffd-80af-19c140c36804 nodeName:}" failed. No retries permitted until 2026-07-07 10:28:17.601982624 +0000 UTC m=+66.273942163 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/f85fac08-720e-4ffd-80af-19c140c36804-original-pull-secret") pod "global-pull-secret-syncer-7lv79" (UID: "f85fac08-720e-4ffd-80af-19c140c36804") : object "kube-system"/"original-pull-secret" not registered Jul 07 10:27:45.602042 ip-10-0-135-34 kubenswrapper[2578]: E0707 10:27:45.602007 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/51fe14a4-6b46-4f88-be7d-b611b45961ff-kube-api-access-9vbqq podName:51fe14a4-6b46-4f88-be7d-b611b45961ff nodeName:}" failed. No retries permitted until 2026-07-07 10:28:17.602001417 +0000 UTC m=+66.273960939 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "kube-api-access-9vbqq" (UniqueName: "kubernetes.io/projected/51fe14a4-6b46-4f88-be7d-b611b45961ff-kube-api-access-9vbqq") pod "network-check-target-26sg7" (UID: "51fe14a4-6b46-4f88-be7d-b611b45961ff") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 07 10:27:46.815662 ip-10-0-135-34 kubenswrapper[2578]: I0707 10:27:46.815614 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-7lv79" Jul 07 10:27:46.815662 ip-10-0-135-34 kubenswrapper[2578]: I0707 10:27:46.815656 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-26sg7" Jul 07 10:27:46.816219 ip-10-0-135-34 kubenswrapper[2578]: I0707 10:27:46.815706 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-p2ks2" Jul 07 10:27:46.816219 ip-10-0-135-34 kubenswrapper[2578]: E0707 10:27:46.815719 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-7lv79" podUID="f85fac08-720e-4ffd-80af-19c140c36804" Jul 07 10:27:46.816219 ip-10-0-135-34 kubenswrapper[2578]: E0707 10:27:46.815833 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-p2ks2" podUID="931a7d75-9ee8-4758-9f23-3521f3db5eae" Jul 07 10:27:46.816219 ip-10-0-135-34 kubenswrapper[2578]: E0707 10:27:46.815911 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-26sg7" podUID="51fe14a4-6b46-4f88-be7d-b611b45961ff" Jul 07 10:27:48.157330 ip-10-0-135-34 kubenswrapper[2578]: I0707 10:27:48.157303 2578 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-34.ec2.internal" event="NodeReady" Jul 07 10:27:48.157704 ip-10-0-135-34 kubenswrapper[2578]: I0707 10:27:48.157423 2578 kubelet_node_status.go:489] "Fast updating node status as it just became ready" Jul 07 10:27:48.197793 ip-10-0-135-34 kubenswrapper[2578]: I0707 10:27:48.197742 2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-image-registry/image-registry-cb7847d49-4nszw"] Jul 07 10:27:48.218613 ip-10-0-135-34 kubenswrapper[2578]: I0707 10:27:48.218581 2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-dns/dns-default-j2ks5"] Jul 07 10:27:48.218824 ip-10-0-135-34 kubenswrapper[2578]: I0707 10:27:48.218700 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-cb7847d49-4nszw" Jul 07 10:27:48.221263 ip-10-0-135-34 kubenswrapper[2578]: I0707 10:27:48.221238 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-tls\"" Jul 07 10:27:48.221451 ip-10-0-135-34 kubenswrapper[2578]: I0707 10:27:48.221429 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"registry-dockercfg-tvxzc\"" Jul 07 10:27:48.221528 ip-10-0-135-34 kubenswrapper[2578]: I0707 10:27:48.221334 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-private-configuration\"" Jul 07 10:27:48.222614 ip-10-0-135-34 kubenswrapper[2578]: I0707 10:27:48.222594 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"installation-pull-secrets\"" Jul 07 10:27:48.232078 ip-10-0-135-34 kubenswrapper[2578]: I0707 10:27:48.232054 2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"trusted-ca\"" Jul 07 10:27:48.236700 ip-10-0-135-34 kubenswrapper[2578]: I0707 10:27:48.236675 2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress-canary/ingress-canary-s9zln"] Jul 07 10:27:48.236833 ip-10-0-135-34 kubenswrapper[2578]: I0707 10:27:48.236817 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-j2ks5" Jul 07 10:27:48.239556 ip-10-0-135-34 kubenswrapper[2578]: I0707 10:27:48.239532 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\"" Jul 07 10:27:48.239686 ip-10-0-135-34 kubenswrapper[2578]: I0707 10:27:48.239557 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-wq5jm\"" Jul 07 10:27:48.239686 ip-10-0-135-34 kubenswrapper[2578]: I0707 10:27:48.239557 2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\"" Jul 07 10:27:48.258735 ip-10-0-135-34 kubenswrapper[2578]: I0707 10:27:48.258705 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-cb7847d49-4nszw"] Jul 07 10:27:48.258902 ip-10-0-135-34 kubenswrapper[2578]: I0707 10:27:48.258747 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-s9zln"] Jul 07 10:27:48.258902 ip-10-0-135-34 kubenswrapper[2578]: I0707 10:27:48.258764 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-j2ks5"] Jul 07 10:27:48.258902 ip-10-0-135-34 kubenswrapper[2578]: I0707 10:27:48.258845 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-s9zln" Jul 07 10:27:48.261404 ip-10-0-135-34 kubenswrapper[2578]: I0707 10:27:48.261380 2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\"" Jul 07 10:27:48.261564 ip-10-0-135-34 kubenswrapper[2578]: I0707 10:27:48.261388 2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\"" Jul 07 10:27:48.261640 ip-10-0-135-34 kubenswrapper[2578]: I0707 10:27:48.261566 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"default-dockercfg-mdcws\"" Jul 07 10:27:48.261698 ip-10-0-135-34 kubenswrapper[2578]: I0707 10:27:48.261641 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\"" Jul 07 10:27:48.321089 ip-10-0-135-34 kubenswrapper[2578]: I0707 10:27:48.321047 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/a1048018-11a0-4d5a-96ab-86508dac8366-config-volume\") pod \"dns-default-j2ks5\" (UID: \"a1048018-11a0-4d5a-96ab-86508dac8366\") " pod="openshift-dns/dns-default-j2ks5" Jul 07 10:27:48.321250 ip-10-0-135-34 kubenswrapper[2578]: I0707 10:27:48.321092 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9tgml\" (UniqueName: \"kubernetes.io/projected/c4552170-c8e7-43e3-9472-a066d5379772-kube-api-access-9tgml\") pod \"image-registry-cb7847d49-4nszw\" (UID: \"c4552170-c8e7-43e3-9472-a066d5379772\") " pod="openshift-image-registry/image-registry-cb7847d49-4nszw" Jul 07 10:27:48.321250 ip-10-0-135-34 kubenswrapper[2578]: I0707 10:27:48.321131 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-68v5k\" (UniqueName: \"kubernetes.io/projected/a1048018-11a0-4d5a-96ab-86508dac8366-kube-api-access-68v5k\") pod \"dns-default-j2ks5\" (UID: \"a1048018-11a0-4d5a-96ab-86508dac8366\") " pod="openshift-dns/dns-default-j2ks5" Jul 07 10:27:48.321250 ip-10-0-135-34 kubenswrapper[2578]: I0707 10:27:48.321209 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/c4552170-c8e7-43e3-9472-a066d5379772-trusted-ca\") pod \"image-registry-cb7847d49-4nszw\" (UID: \"c4552170-c8e7-43e3-9472-a066d5379772\") " pod="openshift-image-registry/image-registry-cb7847d49-4nszw" Jul 07 10:27:48.321250 ip-10-0-135-34 kubenswrapper[2578]: I0707 10:27:48.321229 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/c4552170-c8e7-43e3-9472-a066d5379772-installation-pull-secrets\") pod \"image-registry-cb7847d49-4nszw\" (UID: \"c4552170-c8e7-43e3-9472-a066d5379772\") " pod="openshift-image-registry/image-registry-cb7847d49-4nszw" Jul 07 10:27:48.321441 ip-10-0-135-34 kubenswrapper[2578]: I0707 10:27:48.321253 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/c4552170-c8e7-43e3-9472-a066d5379772-image-registry-private-configuration\") pod \"image-registry-cb7847d49-4nszw\" (UID: \"c4552170-c8e7-43e3-9472-a066d5379772\") " pod="openshift-image-registry/image-registry-cb7847d49-4nszw" Jul 07 10:27:48.321441 ip-10-0-135-34 kubenswrapper[2578]: I0707 10:27:48.321278 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/c4552170-c8e7-43e3-9472-a066d5379772-registry-tls\") pod \"image-registry-cb7847d49-4nszw\" (UID: \"c4552170-c8e7-43e3-9472-a066d5379772\") " pod="openshift-image-registry/image-registry-cb7847d49-4nszw" Jul 07 10:27:48.321441 ip-10-0-135-34 kubenswrapper[2578]: I0707 10:27:48.321379 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a1048018-11a0-4d5a-96ab-86508dac8366-tmp-dir\") pod \"dns-default-j2ks5\" (UID: \"a1048018-11a0-4d5a-96ab-86508dac8366\") " pod="openshift-dns/dns-default-j2ks5" Jul 07 10:27:48.321441 ip-10-0-135-34 kubenswrapper[2578]: I0707 10:27:48.321429 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/c4552170-c8e7-43e3-9472-a066d5379772-ca-trust-extracted\") pod \"image-registry-cb7847d49-4nszw\" (UID: \"c4552170-c8e7-43e3-9472-a066d5379772\") " pod="openshift-image-registry/image-registry-cb7847d49-4nszw" Jul 07 10:27:48.321601 ip-10-0-135-34 kubenswrapper[2578]: I0707 10:27:48.321462 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/c4552170-c8e7-43e3-9472-a066d5379772-registry-certificates\") pod \"image-registry-cb7847d49-4nszw\" (UID: \"c4552170-c8e7-43e3-9472-a066d5379772\") " pod="openshift-image-registry/image-registry-cb7847d49-4nszw" Jul 07 10:27:48.321601 ip-10-0-135-34 kubenswrapper[2578]: I0707 10:27:48.321492 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/c4552170-c8e7-43e3-9472-a066d5379772-bound-sa-token\") pod \"image-registry-cb7847d49-4nszw\" (UID: \"c4552170-c8e7-43e3-9472-a066d5379772\") " pod="openshift-image-registry/image-registry-cb7847d49-4nszw" Jul 07 10:27:48.321601 ip-10-0-135-34 kubenswrapper[2578]: I0707 10:27:48.321520 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/a1048018-11a0-4d5a-96ab-86508dac8366-metrics-tls\") pod \"dns-default-j2ks5\" (UID: \"a1048018-11a0-4d5a-96ab-86508dac8366\") " pod="openshift-dns/dns-default-j2ks5" Jul 07 10:27:48.422057 ip-10-0-135-34 kubenswrapper[2578]: I0707 10:27:48.421964 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a1048018-11a0-4d5a-96ab-86508dac8366-tmp-dir\") pod \"dns-default-j2ks5\" (UID: \"a1048018-11a0-4d5a-96ab-86508dac8366\") " pod="openshift-dns/dns-default-j2ks5" Jul 07 10:27:48.422057 ip-10-0-135-34 kubenswrapper[2578]: I0707 10:27:48.422011 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/c4552170-c8e7-43e3-9472-a066d5379772-ca-trust-extracted\") pod \"image-registry-cb7847d49-4nszw\" (UID: \"c4552170-c8e7-43e3-9472-a066d5379772\") " pod="openshift-image-registry/image-registry-cb7847d49-4nszw" Jul 07 10:27:48.422057 ip-10-0-135-34 kubenswrapper[2578]: I0707 10:27:48.422034 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/c4552170-c8e7-43e3-9472-a066d5379772-registry-certificates\") pod \"image-registry-cb7847d49-4nszw\" (UID: \"c4552170-c8e7-43e3-9472-a066d5379772\") " pod="openshift-image-registry/image-registry-cb7847d49-4nszw" Jul 07 10:27:48.422305 ip-10-0-135-34 kubenswrapper[2578]: I0707 10:27:48.422062 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/c4552170-c8e7-43e3-9472-a066d5379772-bound-sa-token\") pod \"image-registry-cb7847d49-4nszw\" (UID: \"c4552170-c8e7-43e3-9472-a066d5379772\") " pod="openshift-image-registry/image-registry-cb7847d49-4nszw" Jul 07 10:27:48.422305 ip-10-0-135-34 kubenswrapper[2578]: I0707 10:27:48.422087 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/a1048018-11a0-4d5a-96ab-86508dac8366-metrics-tls\") pod \"dns-default-j2ks5\" (UID: \"a1048018-11a0-4d5a-96ab-86508dac8366\") " pod="openshift-dns/dns-default-j2ks5" Jul 07 10:27:48.422305 ip-10-0-135-34 kubenswrapper[2578]: I0707 10:27:48.422114 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-flqn9\" (UniqueName: \"kubernetes.io/projected/eae6c59c-de76-4a6e-8f2b-1de5dbeedf78-kube-api-access-flqn9\") pod \"ingress-canary-s9zln\" (UID: \"eae6c59c-de76-4a6e-8f2b-1de5dbeedf78\") " pod="openshift-ingress-canary/ingress-canary-s9zln" Jul 07 10:27:48.422305 ip-10-0-135-34 kubenswrapper[2578]: I0707 10:27:48.422182 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/eae6c59c-de76-4a6e-8f2b-1de5dbeedf78-cert\") pod \"ingress-canary-s9zln\" (UID: \"eae6c59c-de76-4a6e-8f2b-1de5dbeedf78\") " pod="openshift-ingress-canary/ingress-canary-s9zln" Jul 07 10:27:48.422305 ip-10-0-135-34 kubenswrapper[2578]: I0707 10:27:48.422247 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/a1048018-11a0-4d5a-96ab-86508dac8366-config-volume\") pod \"dns-default-j2ks5\" (UID: \"a1048018-11a0-4d5a-96ab-86508dac8366\") " pod="openshift-dns/dns-default-j2ks5" Jul 07 10:27:48.422305 ip-10-0-135-34 kubenswrapper[2578]: I0707 10:27:48.422279 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9tgml\" (UniqueName: \"kubernetes.io/projected/c4552170-c8e7-43e3-9472-a066d5379772-kube-api-access-9tgml\") pod \"image-registry-cb7847d49-4nszw\" (UID: \"c4552170-c8e7-43e3-9472-a066d5379772\") " pod="openshift-image-registry/image-registry-cb7847d49-4nszw" Jul 07 10:27:48.422305 ip-10-0-135-34 kubenswrapper[2578]: E0707 10:27:48.422248 2578 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jul 07 10:27:48.422622 ip-10-0-135-34 kubenswrapper[2578]: I0707 10:27:48.422321 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-68v5k\" (UniqueName: \"kubernetes.io/projected/a1048018-11a0-4d5a-96ab-86508dac8366-kube-api-access-68v5k\") pod \"dns-default-j2ks5\" (UID: \"a1048018-11a0-4d5a-96ab-86508dac8366\") " pod="openshift-dns/dns-default-j2ks5" Jul 07 10:27:48.422622 ip-10-0-135-34 kubenswrapper[2578]: E0707 10:27:48.422370 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a1048018-11a0-4d5a-96ab-86508dac8366-metrics-tls podName:a1048018-11a0-4d5a-96ab-86508dac8366 nodeName:}" failed. No retries permitted until 2026-07-07 10:27:48.922348631 +0000 UTC m=+37.594308160 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/a1048018-11a0-4d5a-96ab-86508dac8366-metrics-tls") pod "dns-default-j2ks5" (UID: "a1048018-11a0-4d5a-96ab-86508dac8366") : secret "dns-default-metrics-tls" not found Jul 07 10:27:48.422622 ip-10-0-135-34 kubenswrapper[2578]: I0707 10:27:48.422408 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a1048018-11a0-4d5a-96ab-86508dac8366-tmp-dir\") pod \"dns-default-j2ks5\" (UID: \"a1048018-11a0-4d5a-96ab-86508dac8366\") " pod="openshift-dns/dns-default-j2ks5" Jul 07 10:27:48.422622 ip-10-0-135-34 kubenswrapper[2578]: I0707 10:27:48.422421 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/c4552170-c8e7-43e3-9472-a066d5379772-trusted-ca\") pod \"image-registry-cb7847d49-4nszw\" (UID: \"c4552170-c8e7-43e3-9472-a066d5379772\") " pod="openshift-image-registry/image-registry-cb7847d49-4nszw" Jul 07 10:27:48.422622 ip-10-0-135-34 kubenswrapper[2578]: I0707 10:27:48.422445 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/c4552170-c8e7-43e3-9472-a066d5379772-installation-pull-secrets\") pod \"image-registry-cb7847d49-4nszw\" (UID: \"c4552170-c8e7-43e3-9472-a066d5379772\") " pod="openshift-image-registry/image-registry-cb7847d49-4nszw" Jul 07 10:27:48.422622 ip-10-0-135-34 kubenswrapper[2578]: I0707 10:27:48.422457 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/c4552170-c8e7-43e3-9472-a066d5379772-ca-trust-extracted\") pod \"image-registry-cb7847d49-4nszw\" (UID: \"c4552170-c8e7-43e3-9472-a066d5379772\") " pod="openshift-image-registry/image-registry-cb7847d49-4nszw" Jul 07 10:27:48.422622 ip-10-0-135-34 kubenswrapper[2578]: I0707 10:27:48.422476 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/c4552170-c8e7-43e3-9472-a066d5379772-image-registry-private-configuration\") pod \"image-registry-cb7847d49-4nszw\" (UID: \"c4552170-c8e7-43e3-9472-a066d5379772\") " pod="openshift-image-registry/image-registry-cb7847d49-4nszw" Jul 07 10:27:48.422622 ip-10-0-135-34 kubenswrapper[2578]: I0707 10:27:48.422503 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/c4552170-c8e7-43e3-9472-a066d5379772-registry-tls\") pod \"image-registry-cb7847d49-4nszw\" (UID: \"c4552170-c8e7-43e3-9472-a066d5379772\") " pod="openshift-image-registry/image-registry-cb7847d49-4nszw" Jul 07 10:27:48.423047 ip-10-0-135-34 kubenswrapper[2578]: E0707 10:27:48.422628 2578 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Jul 07 10:27:48.423047 ip-10-0-135-34 kubenswrapper[2578]: E0707 10:27:48.422642 2578 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-cb7847d49-4nszw: secret "image-registry-tls" not found Jul 07 10:27:48.423047 ip-10-0-135-34 kubenswrapper[2578]: E0707 10:27:48.422696 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/c4552170-c8e7-43e3-9472-a066d5379772-registry-tls podName:c4552170-c8e7-43e3-9472-a066d5379772 nodeName:}" failed. No retries permitted until 2026-07-07 10:27:48.922682377 +0000 UTC m=+37.594641913 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/c4552170-c8e7-43e3-9472-a066d5379772-registry-tls") pod "image-registry-cb7847d49-4nszw" (UID: "c4552170-c8e7-43e3-9472-a066d5379772") : secret "image-registry-tls" not found Jul 07 10:27:48.423047 ip-10-0-135-34 kubenswrapper[2578]: I0707 10:27:48.422743 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/c4552170-c8e7-43e3-9472-a066d5379772-registry-certificates\") pod \"image-registry-cb7847d49-4nszw\" (UID: \"c4552170-c8e7-43e3-9472-a066d5379772\") " pod="openshift-image-registry/image-registry-cb7847d49-4nszw" Jul 07 10:27:48.423047 ip-10-0-135-34 kubenswrapper[2578]: I0707 10:27:48.422809 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/a1048018-11a0-4d5a-96ab-86508dac8366-config-volume\") pod \"dns-default-j2ks5\" (UID: \"a1048018-11a0-4d5a-96ab-86508dac8366\") " pod="openshift-dns/dns-default-j2ks5" Jul 07 10:27:48.423389 ip-10-0-135-34 kubenswrapper[2578]: I0707 10:27:48.423369 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/c4552170-c8e7-43e3-9472-a066d5379772-trusted-ca\") pod \"image-registry-cb7847d49-4nszw\" (UID: \"c4552170-c8e7-43e3-9472-a066d5379772\") " pod="openshift-image-registry/image-registry-cb7847d49-4nszw" Jul 07 10:27:48.426523 ip-10-0-135-34 kubenswrapper[2578]: I0707 10:27:48.426498 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/c4552170-c8e7-43e3-9472-a066d5379772-installation-pull-secrets\") pod \"image-registry-cb7847d49-4nszw\" (UID: \"c4552170-c8e7-43e3-9472-a066d5379772\") " pod="openshift-image-registry/image-registry-cb7847d49-4nszw" Jul 07 10:27:48.426598 ip-10-0-135-34 kubenswrapper[2578]: I0707 10:27:48.426503 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/c4552170-c8e7-43e3-9472-a066d5379772-image-registry-private-configuration\") pod \"image-registry-cb7847d49-4nszw\" (UID: \"c4552170-c8e7-43e3-9472-a066d5379772\") " pod="openshift-image-registry/image-registry-cb7847d49-4nszw" Jul 07 10:27:48.433345 ip-10-0-135-34 kubenswrapper[2578]: I0707 10:27:48.433315 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/c4552170-c8e7-43e3-9472-a066d5379772-bound-sa-token\") pod \"image-registry-cb7847d49-4nszw\" (UID: \"c4552170-c8e7-43e3-9472-a066d5379772\") " pod="openshift-image-registry/image-registry-cb7847d49-4nszw" Jul 07 10:27:48.433345 ip-10-0-135-34 kubenswrapper[2578]: I0707 10:27:48.433343 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-68v5k\" (UniqueName: \"kubernetes.io/projected/a1048018-11a0-4d5a-96ab-86508dac8366-kube-api-access-68v5k\") pod \"dns-default-j2ks5\" (UID: \"a1048018-11a0-4d5a-96ab-86508dac8366\") " pod="openshift-dns/dns-default-j2ks5" Jul 07 10:27:48.433599 ip-10-0-135-34 kubenswrapper[2578]: I0707 10:27:48.433575 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9tgml\" (UniqueName: \"kubernetes.io/projected/c4552170-c8e7-43e3-9472-a066d5379772-kube-api-access-9tgml\") pod \"image-registry-cb7847d49-4nszw\" (UID: \"c4552170-c8e7-43e3-9472-a066d5379772\") " pod="openshift-image-registry/image-registry-cb7847d49-4nszw" Jul 07 10:27:48.523897 ip-10-0-135-34 kubenswrapper[2578]: I0707 10:27:48.523857 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-flqn9\" (UniqueName: \"kubernetes.io/projected/eae6c59c-de76-4a6e-8f2b-1de5dbeedf78-kube-api-access-flqn9\") pod \"ingress-canary-s9zln\" (UID: \"eae6c59c-de76-4a6e-8f2b-1de5dbeedf78\") " pod="openshift-ingress-canary/ingress-canary-s9zln" Jul 07 10:27:48.523897 ip-10-0-135-34 kubenswrapper[2578]: I0707 10:27:48.523900 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/eae6c59c-de76-4a6e-8f2b-1de5dbeedf78-cert\") pod \"ingress-canary-s9zln\" (UID: \"eae6c59c-de76-4a6e-8f2b-1de5dbeedf78\") " pod="openshift-ingress-canary/ingress-canary-s9zln" Jul 07 10:27:48.526456 ip-10-0-135-34 kubenswrapper[2578]: I0707 10:27:48.526435 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/eae6c59c-de76-4a6e-8f2b-1de5dbeedf78-cert\") pod \"ingress-canary-s9zln\" (UID: \"eae6c59c-de76-4a6e-8f2b-1de5dbeedf78\") " pod="openshift-ingress-canary/ingress-canary-s9zln" Jul 07 10:27:48.532823 ip-10-0-135-34 kubenswrapper[2578]: I0707 10:27:48.532803 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-flqn9\" (UniqueName: \"kubernetes.io/projected/eae6c59c-de76-4a6e-8f2b-1de5dbeedf78-kube-api-access-flqn9\") pod \"ingress-canary-s9zln\" (UID: \"eae6c59c-de76-4a6e-8f2b-1de5dbeedf78\") " pod="openshift-ingress-canary/ingress-canary-s9zln" Jul 07 10:27:48.568429 ip-10-0-135-34 kubenswrapper[2578]: I0707 10:27:48.568392 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-s9zln" Jul 07 10:27:48.757267 ip-10-0-135-34 kubenswrapper[2578]: I0707 10:27:48.757235 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-s9zln"] Jul 07 10:27:48.761504 ip-10-0-135-34 kubenswrapper[2578]: W0707 10:27:48.761480 2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podeae6c59c_de76_4a6e_8f2b_1de5dbeedf78.slice/crio-678011e8c063f01d83cf7c3bef5a7be80366fcd0fa4f968df0baf7c8b4f60088 WatchSource:0}: Error finding container 678011e8c063f01d83cf7c3bef5a7be80366fcd0fa4f968df0baf7c8b4f60088: Status 404 returned error can't find the container with id 678011e8c063f01d83cf7c3bef5a7be80366fcd0fa4f968df0baf7c8b4f60088 Jul 07 10:27:48.815575 ip-10-0-135-34 kubenswrapper[2578]: I0707 10:27:48.815528 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-p2ks2" Jul 07 10:27:48.815752 ip-10-0-135-34 kubenswrapper[2578]: I0707 10:27:48.815544 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-7lv79" Jul 07 10:27:48.815752 ip-10-0-135-34 kubenswrapper[2578]: I0707 10:27:48.815663 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-26sg7" Jul 07 10:27:48.818531 ip-10-0-135-34 kubenswrapper[2578]: I0707 10:27:48.818509 2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\"" Jul 07 10:27:48.818656 ip-10-0-135-34 kubenswrapper[2578]: I0707 10:27:48.818626 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-68x57\"" Jul 07 10:27:48.818656 ip-10-0-135-34 kubenswrapper[2578]: I0707 10:27:48.818638 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\"" Jul 07 10:27:48.818656 ip-10-0-135-34 kubenswrapper[2578]: I0707 10:27:48.818650 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-qdqsn\"" Jul 07 10:27:48.818934 ip-10-0-135-34 kubenswrapper[2578]: I0707 10:27:48.818916 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\"" Jul 07 10:27:48.819087 ip-10-0-135-34 kubenswrapper[2578]: I0707 10:27:48.819071 2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\"" Jul 07 10:27:48.927970 ip-10-0-135-34 kubenswrapper[2578]: I0707 10:27:48.927880 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/c4552170-c8e7-43e3-9472-a066d5379772-registry-tls\") pod \"image-registry-cb7847d49-4nszw\" (UID: \"c4552170-c8e7-43e3-9472-a066d5379772\") " pod="openshift-image-registry/image-registry-cb7847d49-4nszw" Jul 07 10:27:48.927970 ip-10-0-135-34 kubenswrapper[2578]: I0707 10:27:48.927956 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/a1048018-11a0-4d5a-96ab-86508dac8366-metrics-tls\") pod \"dns-default-j2ks5\" (UID: \"a1048018-11a0-4d5a-96ab-86508dac8366\") " pod="openshift-dns/dns-default-j2ks5" Jul 07 10:27:48.928152 ip-10-0-135-34 kubenswrapper[2578]: E0707 10:27:48.928051 2578 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Jul 07 10:27:48.928152 ip-10-0-135-34 kubenswrapper[2578]: E0707 10:27:48.928070 2578 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-cb7847d49-4nszw: secret "image-registry-tls" not found Jul 07 10:27:48.928152 ip-10-0-135-34 kubenswrapper[2578]: E0707 10:27:48.928068 2578 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jul 07 10:27:48.928152 ip-10-0-135-34 kubenswrapper[2578]: E0707 10:27:48.928134 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a1048018-11a0-4d5a-96ab-86508dac8366-metrics-tls podName:a1048018-11a0-4d5a-96ab-86508dac8366 nodeName:}" failed. No retries permitted until 2026-07-07 10:27:49.928117779 +0000 UTC m=+38.600077301 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/a1048018-11a0-4d5a-96ab-86508dac8366-metrics-tls") pod "dns-default-j2ks5" (UID: "a1048018-11a0-4d5a-96ab-86508dac8366") : secret "dns-default-metrics-tls" not found Jul 07 10:27:48.928152 ip-10-0-135-34 kubenswrapper[2578]: E0707 10:27:48.928147 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/c4552170-c8e7-43e3-9472-a066d5379772-registry-tls podName:c4552170-c8e7-43e3-9472-a066d5379772 nodeName:}" failed. No retries permitted until 2026-07-07 10:27:49.928140973 +0000 UTC m=+38.600100495 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/c4552170-c8e7-43e3-9472-a066d5379772-registry-tls") pod "image-registry-cb7847d49-4nszw" (UID: "c4552170-c8e7-43e3-9472-a066d5379772") : secret "image-registry-tls" not found Jul 07 10:27:49.005829 ip-10-0-135-34 kubenswrapper[2578]: I0707 10:27:49.005785 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-s9zln" event={"ID":"eae6c59c-de76-4a6e-8f2b-1de5dbeedf78","Type":"ContainerStarted","Data":"678011e8c063f01d83cf7c3bef5a7be80366fcd0fa4f968df0baf7c8b4f60088"} Jul 07 10:27:49.936946 ip-10-0-135-34 kubenswrapper[2578]: I0707 10:27:49.936909 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/c4552170-c8e7-43e3-9472-a066d5379772-registry-tls\") pod \"image-registry-cb7847d49-4nszw\" (UID: \"c4552170-c8e7-43e3-9472-a066d5379772\") " pod="openshift-image-registry/image-registry-cb7847d49-4nszw" Jul 07 10:27:49.937703 ip-10-0-135-34 kubenswrapper[2578]: I0707 10:27:49.936994 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/a1048018-11a0-4d5a-96ab-86508dac8366-metrics-tls\") pod \"dns-default-j2ks5\" (UID: \"a1048018-11a0-4d5a-96ab-86508dac8366\") " pod="openshift-dns/dns-default-j2ks5" Jul 07 10:27:49.937703 ip-10-0-135-34 kubenswrapper[2578]: E0707 10:27:49.937082 2578 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Jul 07 10:27:49.937703 ip-10-0-135-34 kubenswrapper[2578]: E0707 10:27:49.937102 2578 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-cb7847d49-4nszw: secret "image-registry-tls" not found Jul 07 10:27:49.937703 ip-10-0-135-34 kubenswrapper[2578]: E0707 10:27:49.937134 2578 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jul 07 10:27:49.937703 ip-10-0-135-34 kubenswrapper[2578]: E0707 10:27:49.937159 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/c4552170-c8e7-43e3-9472-a066d5379772-registry-tls podName:c4552170-c8e7-43e3-9472-a066d5379772 nodeName:}" failed. No retries permitted until 2026-07-07 10:27:51.937140762 +0000 UTC m=+40.609100310 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/c4552170-c8e7-43e3-9472-a066d5379772-registry-tls") pod "image-registry-cb7847d49-4nszw" (UID: "c4552170-c8e7-43e3-9472-a066d5379772") : secret "image-registry-tls" not found Jul 07 10:27:49.937703 ip-10-0-135-34 kubenswrapper[2578]: E0707 10:27:49.937183 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a1048018-11a0-4d5a-96ab-86508dac8366-metrics-tls podName:a1048018-11a0-4d5a-96ab-86508dac8366 nodeName:}" failed. No retries permitted until 2026-07-07 10:27:51.937167288 +0000 UTC m=+40.609126811 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/a1048018-11a0-4d5a-96ab-86508dac8366-metrics-tls") pod "dns-default-j2ks5" (UID: "a1048018-11a0-4d5a-96ab-86508dac8366") : secret "dns-default-metrics-tls" not found Jul 07 10:27:51.951450 ip-10-0-135-34 kubenswrapper[2578]: I0707 10:27:51.951294 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/c4552170-c8e7-43e3-9472-a066d5379772-registry-tls\") pod \"image-registry-cb7847d49-4nszw\" (UID: \"c4552170-c8e7-43e3-9472-a066d5379772\") " pod="openshift-image-registry/image-registry-cb7847d49-4nszw" Jul 07 10:27:51.951814 ip-10-0-135-34 kubenswrapper[2578]: I0707 10:27:51.951490 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/a1048018-11a0-4d5a-96ab-86508dac8366-metrics-tls\") pod \"dns-default-j2ks5\" (UID: \"a1048018-11a0-4d5a-96ab-86508dac8366\") " pod="openshift-dns/dns-default-j2ks5" Jul 07 10:27:51.951814 ip-10-0-135-34 kubenswrapper[2578]: E0707 10:27:51.951437 2578 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Jul 07 10:27:51.951814 ip-10-0-135-34 kubenswrapper[2578]: E0707 10:27:51.951530 2578 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-cb7847d49-4nszw: secret "image-registry-tls" not found Jul 07 10:27:51.951814 ip-10-0-135-34 kubenswrapper[2578]: E0707 10:27:51.951598 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/c4552170-c8e7-43e3-9472-a066d5379772-registry-tls podName:c4552170-c8e7-43e3-9472-a066d5379772 nodeName:}" failed. No retries permitted until 2026-07-07 10:27:55.951575852 +0000 UTC m=+44.623535392 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/c4552170-c8e7-43e3-9472-a066d5379772-registry-tls") pod "image-registry-cb7847d49-4nszw" (UID: "c4552170-c8e7-43e3-9472-a066d5379772") : secret "image-registry-tls" not found Jul 07 10:27:51.951814 ip-10-0-135-34 kubenswrapper[2578]: E0707 10:27:51.951614 2578 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jul 07 10:27:51.951814 ip-10-0-135-34 kubenswrapper[2578]: E0707 10:27:51.951659 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a1048018-11a0-4d5a-96ab-86508dac8366-metrics-tls podName:a1048018-11a0-4d5a-96ab-86508dac8366 nodeName:}" failed. No retries permitted until 2026-07-07 10:27:55.951645732 +0000 UTC m=+44.623605275 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/a1048018-11a0-4d5a-96ab-86508dac8366-metrics-tls") pod "dns-default-j2ks5" (UID: "a1048018-11a0-4d5a-96ab-86508dac8366") : secret "dns-default-metrics-tls" not found Jul 07 10:27:52.014153 ip-10-0-135-34 kubenswrapper[2578]: I0707 10:27:52.014115 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-s9zln" event={"ID":"eae6c59c-de76-4a6e-8f2b-1de5dbeedf78","Type":"ContainerStarted","Data":"2e29ea842265cbda47aad7426f09cd5d25c8b94c7453b6c2fd4c1bf37e642705"} Jul 07 10:27:52.037714 ip-10-0-135-34 kubenswrapper[2578]: I0707 10:27:52.037637 2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress-canary/ingress-canary-s9zln" podStartSLOduration=0.892389152 podStartE2EDuration="4.037618755s" totalImagesPullingTime="3.145229603s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-07 10:27:48 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 10:27:52.036838695 +0000 UTC m=+40.708798281" watchObservedRunningTime="2026-07-07 10:27:52.037618755 +0000 UTC m=+40.709578300" Jul 07 10:27:55.979422 ip-10-0-135-34 kubenswrapper[2578]: I0707 10:27:55.979382 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/c4552170-c8e7-43e3-9472-a066d5379772-registry-tls\") pod \"image-registry-cb7847d49-4nszw\" (UID: \"c4552170-c8e7-43e3-9472-a066d5379772\") " pod="openshift-image-registry/image-registry-cb7847d49-4nszw" Jul 07 10:27:55.979927 ip-10-0-135-34 kubenswrapper[2578]: I0707 10:27:55.979481 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/a1048018-11a0-4d5a-96ab-86508dac8366-metrics-tls\") pod \"dns-default-j2ks5\" (UID: \"a1048018-11a0-4d5a-96ab-86508dac8366\") " pod="openshift-dns/dns-default-j2ks5" Jul 07 10:27:55.979927 ip-10-0-135-34 kubenswrapper[2578]: E0707 10:27:55.979515 2578 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Jul 07 10:27:55.979927 ip-10-0-135-34 kubenswrapper[2578]: E0707 10:27:55.979541 2578 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-cb7847d49-4nszw: secret "image-registry-tls" not found Jul 07 10:27:55.979927 ip-10-0-135-34 kubenswrapper[2578]: E0707 10:27:55.979610 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/c4552170-c8e7-43e3-9472-a066d5379772-registry-tls podName:c4552170-c8e7-43e3-9472-a066d5379772 nodeName:}" failed. No retries permitted until 2026-07-07 10:28:03.979587408 +0000 UTC m=+52.651546933 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/c4552170-c8e7-43e3-9472-a066d5379772-registry-tls") pod "image-registry-cb7847d49-4nszw" (UID: "c4552170-c8e7-43e3-9472-a066d5379772") : secret "image-registry-tls" not found Jul 07 10:27:55.979927 ip-10-0-135-34 kubenswrapper[2578]: E0707 10:27:55.979664 2578 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jul 07 10:27:55.979927 ip-10-0-135-34 kubenswrapper[2578]: E0707 10:27:55.979731 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a1048018-11a0-4d5a-96ab-86508dac8366-metrics-tls podName:a1048018-11a0-4d5a-96ab-86508dac8366 nodeName:}" failed. No retries permitted until 2026-07-07 10:28:03.97971269 +0000 UTC m=+52.651672225 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/a1048018-11a0-4d5a-96ab-86508dac8366-metrics-tls") pod "dns-default-j2ks5" (UID: "a1048018-11a0-4d5a-96ab-86508dac8366") : secret "dns-default-metrics-tls" not found Jul 07 10:27:58.030137 ip-10-0-135-34 kubenswrapper[2578]: I0707 10:27:58.030098 2578 generic.go:356] "Generic (PLEG): container finished" podID="a415d240-ff8a-4b08-aeb6-b4018195c46b" containerID="525680d09e9cf5895813ed354353e2833128aa2dc571a5f5bd2d42aad77e777c" exitCode=0 Jul 07 10:27:58.030602 ip-10-0-135-34 kubenswrapper[2578]: I0707 10:27:58.030165 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-l8d7k" event={"ID":"a415d240-ff8a-4b08-aeb6-b4018195c46b","Type":"ContainerDied","Data":"525680d09e9cf5895813ed354353e2833128aa2dc571a5f5bd2d42aad77e777c"} Jul 07 10:28:00.036962 ip-10-0-135-34 kubenswrapper[2578]: I0707 10:28:00.036930 2578 generic.go:356] "Generic (PLEG): container finished" podID="a415d240-ff8a-4b08-aeb6-b4018195c46b" containerID="be7ff5f406c2d5cd5872c816cb792a1c1caf9db3f71ef06ef0834de52e4184a4" exitCode=0 Jul 07 10:28:00.037335 ip-10-0-135-34 kubenswrapper[2578]: I0707 10:28:00.036986 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-l8d7k" event={"ID":"a415d240-ff8a-4b08-aeb6-b4018195c46b","Type":"ContainerDied","Data":"be7ff5f406c2d5cd5872c816cb792a1c1caf9db3f71ef06ef0834de52e4184a4"} Jul 07 10:28:04.046099 ip-10-0-135-34 kubenswrapper[2578]: I0707 10:28:04.046057 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/a1048018-11a0-4d5a-96ab-86508dac8366-metrics-tls\") pod \"dns-default-j2ks5\" (UID: \"a1048018-11a0-4d5a-96ab-86508dac8366\") " pod="openshift-dns/dns-default-j2ks5" Jul 07 10:28:04.046557 ip-10-0-135-34 kubenswrapper[2578]: I0707 10:28:04.046170 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/c4552170-c8e7-43e3-9472-a066d5379772-registry-tls\") pod \"image-registry-cb7847d49-4nszw\" (UID: \"c4552170-c8e7-43e3-9472-a066d5379772\") " pod="openshift-image-registry/image-registry-cb7847d49-4nszw" Jul 07 10:28:04.046557 ip-10-0-135-34 kubenswrapper[2578]: E0707 10:28:04.046214 2578 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jul 07 10:28:04.046557 ip-10-0-135-34 kubenswrapper[2578]: E0707 10:28:04.046278 2578 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Jul 07 10:28:04.046557 ip-10-0-135-34 kubenswrapper[2578]: E0707 10:28:04.046292 2578 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-cb7847d49-4nszw: secret "image-registry-tls" not found Jul 07 10:28:04.046557 ip-10-0-135-34 kubenswrapper[2578]: E0707 10:28:04.046283 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a1048018-11a0-4d5a-96ab-86508dac8366-metrics-tls podName:a1048018-11a0-4d5a-96ab-86508dac8366 nodeName:}" failed. No retries permitted until 2026-07-07 10:28:20.046264662 +0000 UTC m=+68.718224202 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/a1048018-11a0-4d5a-96ab-86508dac8366-metrics-tls") pod "dns-default-j2ks5" (UID: "a1048018-11a0-4d5a-96ab-86508dac8366") : secret "dns-default-metrics-tls" not found Jul 07 10:28:04.046557 ip-10-0-135-34 kubenswrapper[2578]: E0707 10:28:04.046344 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/c4552170-c8e7-43e3-9472-a066d5379772-registry-tls podName:c4552170-c8e7-43e3-9472-a066d5379772 nodeName:}" failed. No retries permitted until 2026-07-07 10:28:20.046334656 +0000 UTC m=+68.718294177 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/c4552170-c8e7-43e3-9472-a066d5379772-registry-tls") pod "image-registry-cb7847d49-4nszw" (UID: "c4552170-c8e7-43e3-9472-a066d5379772") : secret "image-registry-tls" not found Jul 07 10:28:05.079581 ip-10-0-135-34 kubenswrapper[2578]: I0707 10:28:05.079547 2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-network-diagnostics/network-check-source-989f8b8dc-2fvvk"] Jul 07 10:28:05.081500 ip-10-0-135-34 kubenswrapper[2578]: I0707 10:28:05.081482 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-source-989f8b8dc-2fvvk" Jul 07 10:28:05.083867 ip-10-0-135-34 kubenswrapper[2578]: I0707 10:28:05.083841 2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-network-console/networking-console-plugin-6bffdf9778-lmcz7"] Jul 07 10:28:05.085574 ip-10-0-135-34 kubenswrapper[2578]: I0707 10:28:05.085548 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-console/networking-console-plugin-6bffdf9778-lmcz7" Jul 07 10:28:05.085643 ip-10-0-135-34 kubenswrapper[2578]: I0707 10:28:05.085595 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"network-diagnostics-dockercfg-rfvvq\"" Jul 07 10:28:05.088132 ip-10-0-135-34 kubenswrapper[2578]: I0707 10:28:05.088111 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-console\"/\"default-dockercfg-676kr\"" Jul 07 10:28:05.088241 ip-10-0-135-34 kubenswrapper[2578]: I0707 10:28:05.088131 2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-console\"/\"networking-console-plugin\"" Jul 07 10:28:05.088293 ip-10-0-135-34 kubenswrapper[2578]: I0707 10:28:05.088239 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-console\"/\"networking-console-plugin-cert\"" Jul 07 10:28:05.096815 ip-10-0-135-34 kubenswrapper[2578]: I0707 10:28:05.096784 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-source-989f8b8dc-2fvvk"] Jul 07 10:28:05.099033 ip-10-0-135-34 kubenswrapper[2578]: I0707 10:28:05.099006 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-console/networking-console-plugin-6bffdf9778-lmcz7"] Jul 07 10:28:05.182623 ip-10-0-135-34 kubenswrapper[2578]: I0707 10:28:05.182593 2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-cluster-samples-operator/cluster-samples-operator-56fcf86bf-lc4lc"] Jul 07 10:28:05.184383 ip-10-0-135-34 kubenswrapper[2578]: I0707 10:28:05.184366 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-samples-operator/cluster-samples-operator-56fcf86bf-lc4lc" Jul 07 10:28:05.187007 ip-10-0-135-34 kubenswrapper[2578]: I0707 10:28:05.186984 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-samples-operator\"/\"samples-operator-tls\"" Jul 07 10:28:05.187007 ip-10-0-135-34 kubenswrapper[2578]: I0707 10:28:05.187000 2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-samples-operator\"/\"openshift-service-ca.crt\"" Jul 07 10:28:05.187214 ip-10-0-135-34 kubenswrapper[2578]: I0707 10:28:05.187075 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-samples-operator\"/\"cluster-samples-operator-dockercfg-cw69h\"" Jul 07 10:28:05.187214 ip-10-0-135-34 kubenswrapper[2578]: I0707 10:28:05.187103 2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-samples-operator\"/\"kube-root-ca.crt\"" Jul 07 10:28:05.197039 ip-10-0-135-34 kubenswrapper[2578]: I0707 10:28:05.197011 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-samples-operator/cluster-samples-operator-56fcf86bf-lc4lc"] Jul 07 10:28:05.255820 ip-10-0-135-34 kubenswrapper[2578]: I0707 10:28:05.255707 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/327fdc7e-f662-4adf-a589-5a22e48daa18-networking-console-plugin-cert\") pod \"networking-console-plugin-6bffdf9778-lmcz7\" (UID: \"327fdc7e-f662-4adf-a589-5a22e48daa18\") " pod="openshift-network-console/networking-console-plugin-6bffdf9778-lmcz7" Jul 07 10:28:05.255820 ip-10-0-135-34 kubenswrapper[2578]: I0707 10:28:05.255788 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/327fdc7e-f662-4adf-a589-5a22e48daa18-nginx-conf\") pod \"networking-console-plugin-6bffdf9778-lmcz7\" (UID: \"327fdc7e-f662-4adf-a589-5a22e48daa18\") " pod="openshift-network-console/networking-console-plugin-6bffdf9778-lmcz7" Jul 07 10:28:05.255999 ip-10-0-135-34 kubenswrapper[2578]: I0707 10:28:05.255862 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2kzss\" (UniqueName: \"kubernetes.io/projected/54cc4603-03ac-473f-bf42-a89647cd378d-kube-api-access-2kzss\") pod \"network-check-source-989f8b8dc-2fvvk\" (UID: \"54cc4603-03ac-473f-bf42-a89647cd378d\") " pod="openshift-network-diagnostics/network-check-source-989f8b8dc-2fvvk" Jul 07 10:28:05.356836 ip-10-0-135-34 kubenswrapper[2578]: I0707 10:28:05.356795 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/327fdc7e-f662-4adf-a589-5a22e48daa18-networking-console-plugin-cert\") pod \"networking-console-plugin-6bffdf9778-lmcz7\" (UID: \"327fdc7e-f662-4adf-a589-5a22e48daa18\") " pod="openshift-network-console/networking-console-plugin-6bffdf9778-lmcz7" Jul 07 10:28:05.357030 ip-10-0-135-34 kubenswrapper[2578]: I0707 10:28:05.356863 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/327fdc7e-f662-4adf-a589-5a22e48daa18-nginx-conf\") pod \"networking-console-plugin-6bffdf9778-lmcz7\" (UID: \"327fdc7e-f662-4adf-a589-5a22e48daa18\") " pod="openshift-network-console/networking-console-plugin-6bffdf9778-lmcz7" Jul 07 10:28:05.357030 ip-10-0-135-34 kubenswrapper[2578]: I0707 10:28:05.356894 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/2ee234eb-c9b5-48d9-972a-ed53412ca83e-samples-operator-tls\") pod \"cluster-samples-operator-56fcf86bf-lc4lc\" (UID: \"2ee234eb-c9b5-48d9-972a-ed53412ca83e\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-56fcf86bf-lc4lc" Jul 07 10:28:05.357030 ip-10-0-135-34 kubenswrapper[2578]: I0707 10:28:05.356937 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2kzss\" (UniqueName: \"kubernetes.io/projected/54cc4603-03ac-473f-bf42-a89647cd378d-kube-api-access-2kzss\") pod \"network-check-source-989f8b8dc-2fvvk\" (UID: \"54cc4603-03ac-473f-bf42-a89647cd378d\") " pod="openshift-network-diagnostics/network-check-source-989f8b8dc-2fvvk" Jul 07 10:28:05.357030 ip-10-0-135-34 kubenswrapper[2578]: E0707 10:28:05.356951 2578 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found Jul 07 10:28:05.357212 ip-10-0-135-34 kubenswrapper[2578]: E0707 10:28:05.357040 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/327fdc7e-f662-4adf-a589-5a22e48daa18-networking-console-plugin-cert podName:327fdc7e-f662-4adf-a589-5a22e48daa18 nodeName:}" failed. No retries permitted until 2026-07-07 10:28:05.857015729 +0000 UTC m=+54.528975255 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/327fdc7e-f662-4adf-a589-5a22e48daa18-networking-console-plugin-cert") pod "networking-console-plugin-6bffdf9778-lmcz7" (UID: "327fdc7e-f662-4adf-a589-5a22e48daa18") : secret "networking-console-plugin-cert" not found Jul 07 10:28:05.357212 ip-10-0-135-34 kubenswrapper[2578]: I0707 10:28:05.357089 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7v5n4\" (UniqueName: \"kubernetes.io/projected/2ee234eb-c9b5-48d9-972a-ed53412ca83e-kube-api-access-7v5n4\") pod \"cluster-samples-operator-56fcf86bf-lc4lc\" (UID: \"2ee234eb-c9b5-48d9-972a-ed53412ca83e\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-56fcf86bf-lc4lc" Jul 07 10:28:05.357485 ip-10-0-135-34 kubenswrapper[2578]: I0707 10:28:05.357464 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/327fdc7e-f662-4adf-a589-5a22e48daa18-nginx-conf\") pod \"networking-console-plugin-6bffdf9778-lmcz7\" (UID: \"327fdc7e-f662-4adf-a589-5a22e48daa18\") " pod="openshift-network-console/networking-console-plugin-6bffdf9778-lmcz7" Jul 07 10:28:05.367376 ip-10-0-135-34 kubenswrapper[2578]: I0707 10:28:05.366985 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2kzss\" (UniqueName: \"kubernetes.io/projected/54cc4603-03ac-473f-bf42-a89647cd378d-kube-api-access-2kzss\") pod \"network-check-source-989f8b8dc-2fvvk\" (UID: \"54cc4603-03ac-473f-bf42-a89647cd378d\") " pod="openshift-network-diagnostics/network-check-source-989f8b8dc-2fvvk" Jul 07 10:28:05.392226 ip-10-0-135-34 kubenswrapper[2578]: I0707 10:28:05.392175 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-source-989f8b8dc-2fvvk" Jul 07 10:28:05.459114 ip-10-0-135-34 kubenswrapper[2578]: I0707 10:28:05.458256 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/2ee234eb-c9b5-48d9-972a-ed53412ca83e-samples-operator-tls\") pod \"cluster-samples-operator-56fcf86bf-lc4lc\" (UID: \"2ee234eb-c9b5-48d9-972a-ed53412ca83e\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-56fcf86bf-lc4lc" Jul 07 10:28:05.459114 ip-10-0-135-34 kubenswrapper[2578]: I0707 10:28:05.458345 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7v5n4\" (UniqueName: \"kubernetes.io/projected/2ee234eb-c9b5-48d9-972a-ed53412ca83e-kube-api-access-7v5n4\") pod \"cluster-samples-operator-56fcf86bf-lc4lc\" (UID: \"2ee234eb-c9b5-48d9-972a-ed53412ca83e\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-56fcf86bf-lc4lc" Jul 07 10:28:05.459114 ip-10-0-135-34 kubenswrapper[2578]: E0707 10:28:05.458854 2578 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found Jul 07 10:28:05.459114 ip-10-0-135-34 kubenswrapper[2578]: E0707 10:28:05.458916 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2ee234eb-c9b5-48d9-972a-ed53412ca83e-samples-operator-tls podName:2ee234eb-c9b5-48d9-972a-ed53412ca83e nodeName:}" failed. No retries permitted until 2026-07-07 10:28:05.958896797 +0000 UTC m=+54.630856326 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/2ee234eb-c9b5-48d9-972a-ed53412ca83e-samples-operator-tls") pod "cluster-samples-operator-56fcf86bf-lc4lc" (UID: "2ee234eb-c9b5-48d9-972a-ed53412ca83e") : secret "samples-operator-tls" not found Jul 07 10:28:05.476901 ip-10-0-135-34 kubenswrapper[2578]: I0707 10:28:05.476868 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7v5n4\" (UniqueName: \"kubernetes.io/projected/2ee234eb-c9b5-48d9-972a-ed53412ca83e-kube-api-access-7v5n4\") pod \"cluster-samples-operator-56fcf86bf-lc4lc\" (UID: \"2ee234eb-c9b5-48d9-972a-ed53412ca83e\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-56fcf86bf-lc4lc" Jul 07 10:28:05.542306 ip-10-0-135-34 kubenswrapper[2578]: I0707 10:28:05.542232 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-source-989f8b8dc-2fvvk"] Jul 07 10:28:05.545255 ip-10-0-135-34 kubenswrapper[2578]: W0707 10:28:05.545222 2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod54cc4603_03ac_473f_bf42_a89647cd378d.slice/crio-9fe25edbaedd79a228b931e91ad40f679adfcc09572165f143b07913ee712d16 WatchSource:0}: Error finding container 9fe25edbaedd79a228b931e91ad40f679adfcc09572165f143b07913ee712d16: Status 404 returned error can't find the container with id 9fe25edbaedd79a228b931e91ad40f679adfcc09572165f143b07913ee712d16 Jul 07 10:28:05.862304 ip-10-0-135-34 kubenswrapper[2578]: I0707 10:28:05.862277 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/327fdc7e-f662-4adf-a589-5a22e48daa18-networking-console-plugin-cert\") pod \"networking-console-plugin-6bffdf9778-lmcz7\" (UID: \"327fdc7e-f662-4adf-a589-5a22e48daa18\") " pod="openshift-network-console/networking-console-plugin-6bffdf9778-lmcz7" Jul 07 10:28:05.862469 ip-10-0-135-34 kubenswrapper[2578]: E0707 10:28:05.862452 2578 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found Jul 07 10:28:05.862529 ip-10-0-135-34 kubenswrapper[2578]: E0707 10:28:05.862517 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/327fdc7e-f662-4adf-a589-5a22e48daa18-networking-console-plugin-cert podName:327fdc7e-f662-4adf-a589-5a22e48daa18 nodeName:}" failed. No retries permitted until 2026-07-07 10:28:06.862497499 +0000 UTC m=+55.534457041 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/327fdc7e-f662-4adf-a589-5a22e48daa18-networking-console-plugin-cert") pod "networking-console-plugin-6bffdf9778-lmcz7" (UID: "327fdc7e-f662-4adf-a589-5a22e48daa18") : secret "networking-console-plugin-cert" not found Jul 07 10:28:05.962922 ip-10-0-135-34 kubenswrapper[2578]: I0707 10:28:05.962869 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/2ee234eb-c9b5-48d9-972a-ed53412ca83e-samples-operator-tls\") pod \"cluster-samples-operator-56fcf86bf-lc4lc\" (UID: \"2ee234eb-c9b5-48d9-972a-ed53412ca83e\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-56fcf86bf-lc4lc" Jul 07 10:28:05.963100 ip-10-0-135-34 kubenswrapper[2578]: E0707 10:28:05.963033 2578 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found Jul 07 10:28:05.963139 ip-10-0-135-34 kubenswrapper[2578]: E0707 10:28:05.963103 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2ee234eb-c9b5-48d9-972a-ed53412ca83e-samples-operator-tls podName:2ee234eb-c9b5-48d9-972a-ed53412ca83e nodeName:}" failed. No retries permitted until 2026-07-07 10:28:06.963087671 +0000 UTC m=+55.635047193 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/2ee234eb-c9b5-48d9-972a-ed53412ca83e-samples-operator-tls") pod "cluster-samples-operator-56fcf86bf-lc4lc" (UID: "2ee234eb-c9b5-48d9-972a-ed53412ca83e") : secret "samples-operator-tls" not found Jul 07 10:28:06.055828 ip-10-0-135-34 kubenswrapper[2578]: I0707 10:28:06.055790 2578 generic.go:356] "Generic (PLEG): container finished" podID="a415d240-ff8a-4b08-aeb6-b4018195c46b" containerID="655d5e4916b6bdc8b8ca3d05e5759736c82913ea8ca4242d818efc66b31f9a5c" exitCode=0 Jul 07 10:28:06.056007 ip-10-0-135-34 kubenswrapper[2578]: I0707 10:28:06.055805 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-l8d7k" event={"ID":"a415d240-ff8a-4b08-aeb6-b4018195c46b","Type":"ContainerDied","Data":"655d5e4916b6bdc8b8ca3d05e5759736c82913ea8ca4242d818efc66b31f9a5c"} Jul 07 10:28:06.057095 ip-10-0-135-34 kubenswrapper[2578]: I0707 10:28:06.057071 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-source-989f8b8dc-2fvvk" event={"ID":"54cc4603-03ac-473f-bf42-a89647cd378d","Type":"ContainerStarted","Data":"9fe25edbaedd79a228b931e91ad40f679adfcc09572165f143b07913ee712d16"} Jul 07 10:28:06.868667 ip-10-0-135-34 kubenswrapper[2578]: I0707 10:28:06.868628 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/327fdc7e-f662-4adf-a589-5a22e48daa18-networking-console-plugin-cert\") pod \"networking-console-plugin-6bffdf9778-lmcz7\" (UID: \"327fdc7e-f662-4adf-a589-5a22e48daa18\") " pod="openshift-network-console/networking-console-plugin-6bffdf9778-lmcz7" Jul 07 10:28:06.869149 ip-10-0-135-34 kubenswrapper[2578]: E0707 10:28:06.868821 2578 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found Jul 07 10:28:06.869149 ip-10-0-135-34 kubenswrapper[2578]: E0707 10:28:06.868907 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/327fdc7e-f662-4adf-a589-5a22e48daa18-networking-console-plugin-cert podName:327fdc7e-f662-4adf-a589-5a22e48daa18 nodeName:}" failed. No retries permitted until 2026-07-07 10:28:08.86888455 +0000 UTC m=+57.540844085 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/327fdc7e-f662-4adf-a589-5a22e48daa18-networking-console-plugin-cert") pod "networking-console-plugin-6bffdf9778-lmcz7" (UID: "327fdc7e-f662-4adf-a589-5a22e48daa18") : secret "networking-console-plugin-cert" not found Jul 07 10:28:06.969564 ip-10-0-135-34 kubenswrapper[2578]: I0707 10:28:06.969520 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/2ee234eb-c9b5-48d9-972a-ed53412ca83e-samples-operator-tls\") pod \"cluster-samples-operator-56fcf86bf-lc4lc\" (UID: \"2ee234eb-c9b5-48d9-972a-ed53412ca83e\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-56fcf86bf-lc4lc" Jul 07 10:28:06.969750 ip-10-0-135-34 kubenswrapper[2578]: E0707 10:28:06.969693 2578 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found Jul 07 10:28:06.969854 ip-10-0-135-34 kubenswrapper[2578]: E0707 10:28:06.969795 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2ee234eb-c9b5-48d9-972a-ed53412ca83e-samples-operator-tls podName:2ee234eb-c9b5-48d9-972a-ed53412ca83e nodeName:}" failed. No retries permitted until 2026-07-07 10:28:08.969756262 +0000 UTC m=+57.641715785 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/2ee234eb-c9b5-48d9-972a-ed53412ca83e-samples-operator-tls") pod "cluster-samples-operator-56fcf86bf-lc4lc" (UID: "2ee234eb-c9b5-48d9-972a-ed53412ca83e") : secret "samples-operator-tls" not found Jul 07 10:28:07.063476 ip-10-0-135-34 kubenswrapper[2578]: I0707 10:28:07.063444 2578 generic.go:356] "Generic (PLEG): container finished" podID="a415d240-ff8a-4b08-aeb6-b4018195c46b" containerID="2038ab1a73a3e9c33f118e36e4e01f3fe6053f5e9f3b06dc151ffb7691958d96" exitCode=0 Jul 07 10:28:07.063630 ip-10-0-135-34 kubenswrapper[2578]: I0707 10:28:07.063511 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-l8d7k" event={"ID":"a415d240-ff8a-4b08-aeb6-b4018195c46b","Type":"ContainerDied","Data":"2038ab1a73a3e9c33f118e36e4e01f3fe6053f5e9f3b06dc151ffb7691958d96"} Jul 07 10:28:08.071087 ip-10-0-135-34 kubenswrapper[2578]: I0707 10:28:08.071039 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-l8d7k" event={"ID":"a415d240-ff8a-4b08-aeb6-b4018195c46b","Type":"ContainerStarted","Data":"d3d6798228eba304df14ee838e1b33d73ece6e1467fa9ce6c7001c49a869079f"} Jul 07 10:28:08.095270 ip-10-0-135-34 kubenswrapper[2578]: I0707 10:28:08.095218 2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-additional-cni-plugins-l8d7k" podStartSLOduration=8.411209217 podStartE2EDuration="56.095195146s" totalImagesPullingTime="47.683985929s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-07 10:27:12 +0000 UTC" imagePullSessionsCount=5 imagePullSessionsStartsCount=1 observedRunningTime="2026-07-07 10:28:08.093149534 +0000 UTC m=+56.765109079" watchObservedRunningTime="2026-07-07 10:28:08.095195146 +0000 UTC m=+56.767154690" Jul 07 10:28:08.882543 ip-10-0-135-34 kubenswrapper[2578]: I0707 10:28:08.882443 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/327fdc7e-f662-4adf-a589-5a22e48daa18-networking-console-plugin-cert\") pod \"networking-console-plugin-6bffdf9778-lmcz7\" (UID: \"327fdc7e-f662-4adf-a589-5a22e48daa18\") " pod="openshift-network-console/networking-console-plugin-6bffdf9778-lmcz7" Jul 07 10:28:08.882752 ip-10-0-135-34 kubenswrapper[2578]: E0707 10:28:08.882582 2578 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found Jul 07 10:28:08.882752 ip-10-0-135-34 kubenswrapper[2578]: E0707 10:28:08.882646 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/327fdc7e-f662-4adf-a589-5a22e48daa18-networking-console-plugin-cert podName:327fdc7e-f662-4adf-a589-5a22e48daa18 nodeName:}" failed. No retries permitted until 2026-07-07 10:28:12.882630661 +0000 UTC m=+61.554590182 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/327fdc7e-f662-4adf-a589-5a22e48daa18-networking-console-plugin-cert") pod "networking-console-plugin-6bffdf9778-lmcz7" (UID: "327fdc7e-f662-4adf-a589-5a22e48daa18") : secret "networking-console-plugin-cert" not found Jul 07 10:28:08.983112 ip-10-0-135-34 kubenswrapper[2578]: I0707 10:28:08.983071 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/2ee234eb-c9b5-48d9-972a-ed53412ca83e-samples-operator-tls\") pod \"cluster-samples-operator-56fcf86bf-lc4lc\" (UID: \"2ee234eb-c9b5-48d9-972a-ed53412ca83e\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-56fcf86bf-lc4lc" Jul 07 10:28:08.983299 ip-10-0-135-34 kubenswrapper[2578]: E0707 10:28:08.983219 2578 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found Jul 07 10:28:08.983299 ip-10-0-135-34 kubenswrapper[2578]: E0707 10:28:08.983283 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2ee234eb-c9b5-48d9-972a-ed53412ca83e-samples-operator-tls podName:2ee234eb-c9b5-48d9-972a-ed53412ca83e nodeName:}" failed. No retries permitted until 2026-07-07 10:28:12.983267794 +0000 UTC m=+61.655227315 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/2ee234eb-c9b5-48d9-972a-ed53412ca83e-samples-operator-tls") pod "cluster-samples-operator-56fcf86bf-lc4lc" (UID: "2ee234eb-c9b5-48d9-972a-ed53412ca83e") : secret "samples-operator-tls" not found Jul 07 10:28:09.075233 ip-10-0-135-34 kubenswrapper[2578]: I0707 10:28:09.075193 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-source-989f8b8dc-2fvvk" event={"ID":"54cc4603-03ac-473f-bf42-a89647cd378d","Type":"ContainerStarted","Data":"abbca0ab4918eaec7e1580c0f38f44b3b6f1ca59730bdcacff1f985c79dd2392"} Jul 07 10:28:09.092190 ip-10-0-135-34 kubenswrapper[2578]: I0707 10:28:09.092135 2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-source-989f8b8dc-2fvvk" podStartSLOduration=1.014762251 podStartE2EDuration="4.092118611s" totalImagesPullingTime="3.07735636s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-07 10:28:05 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 10:28:09.091445562 +0000 UTC m=+57.763405108" watchObservedRunningTime="2026-07-07 10:28:09.092118611 +0000 UTC m=+57.764078401" Jul 07 10:28:10.458457 ip-10-0-135-34 kubenswrapper[2578]: I0707 10:28:10.458420 2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-runtime-extractor-t79dx"] Jul 07 10:28:10.484959 ip-10-0-135-34 kubenswrapper[2578]: I0707 10:28:10.484922 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-t79dx" Jul 07 10:28:10.489394 ip-10-0-135-34 kubenswrapper[2578]: I0707 10:28:10.489369 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\"" Jul 07 10:28:10.491898 ip-10-0-135-34 kubenswrapper[2578]: I0707 10:28:10.491876 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-t79dx"] Jul 07 10:28:10.494002 ip-10-0-135-34 kubenswrapper[2578]: I0707 10:28:10.493980 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/0a3e5a60-3150-4633-8ff1-4c8ebca70a41-crio-socket\") pod \"insights-runtime-extractor-t79dx\" (UID: \"0a3e5a60-3150-4633-8ff1-4c8ebca70a41\") " pod="openshift-insights/insights-runtime-extractor-t79dx" Jul 07 10:28:10.494089 ip-10-0-135-34 kubenswrapper[2578]: I0707 10:28:10.494021 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/0a3e5a60-3150-4633-8ff1-4c8ebca70a41-data-volume\") pod \"insights-runtime-extractor-t79dx\" (UID: \"0a3e5a60-3150-4633-8ff1-4c8ebca70a41\") " pod="openshift-insights/insights-runtime-extractor-t79dx" Jul 07 10:28:10.494134 ip-10-0-135-34 kubenswrapper[2578]: I0707 10:28:10.494080 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/0a3e5a60-3150-4633-8ff1-4c8ebca70a41-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-t79dx\" (UID: \"0a3e5a60-3150-4633-8ff1-4c8ebca70a41\") " pod="openshift-insights/insights-runtime-extractor-t79dx" Jul 07 10:28:10.494134 ip-10-0-135-34 kubenswrapper[2578]: I0707 10:28:10.494127 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/0a3e5a60-3150-4633-8ff1-4c8ebca70a41-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-t79dx\" (UID: \"0a3e5a60-3150-4633-8ff1-4c8ebca70a41\") " pod="openshift-insights/insights-runtime-extractor-t79dx" Jul 07 10:28:10.494223 ip-10-0-135-34 kubenswrapper[2578]: I0707 10:28:10.494145 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nwspc\" (UniqueName: \"kubernetes.io/projected/0a3e5a60-3150-4633-8ff1-4c8ebca70a41-kube-api-access-nwspc\") pod \"insights-runtime-extractor-t79dx\" (UID: \"0a3e5a60-3150-4633-8ff1-4c8ebca70a41\") " pod="openshift-insights/insights-runtime-extractor-t79dx" Jul 07 10:28:10.494389 ip-10-0-135-34 kubenswrapper[2578]: I0707 10:28:10.494372 2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\"" Jul 07 10:28:10.495435 ip-10-0-135-34 kubenswrapper[2578]: I0707 10:28:10.495399 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-wt2hz\"" Jul 07 10:28:10.495610 ip-10-0-135-34 kubenswrapper[2578]: I0707 10:28:10.495597 2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\"" Jul 07 10:28:10.495659 ip-10-0-135-34 kubenswrapper[2578]: I0707 10:28:10.495622 2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\"" Jul 07 10:28:10.595488 ip-10-0-135-34 kubenswrapper[2578]: I0707 10:28:10.595446 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/0a3e5a60-3150-4633-8ff1-4c8ebca70a41-crio-socket\") pod \"insights-runtime-extractor-t79dx\" (UID: \"0a3e5a60-3150-4633-8ff1-4c8ebca70a41\") " pod="openshift-insights/insights-runtime-extractor-t79dx" Jul 07 10:28:10.595488 ip-10-0-135-34 kubenswrapper[2578]: I0707 10:28:10.595494 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/0a3e5a60-3150-4633-8ff1-4c8ebca70a41-data-volume\") pod \"insights-runtime-extractor-t79dx\" (UID: \"0a3e5a60-3150-4633-8ff1-4c8ebca70a41\") " pod="openshift-insights/insights-runtime-extractor-t79dx" Jul 07 10:28:10.595730 ip-10-0-135-34 kubenswrapper[2578]: I0707 10:28:10.595623 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/0a3e5a60-3150-4633-8ff1-4c8ebca70a41-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-t79dx\" (UID: \"0a3e5a60-3150-4633-8ff1-4c8ebca70a41\") " pod="openshift-insights/insights-runtime-extractor-t79dx" Jul 07 10:28:10.595730 ip-10-0-135-34 kubenswrapper[2578]: I0707 10:28:10.595653 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/0a3e5a60-3150-4633-8ff1-4c8ebca70a41-crio-socket\") pod \"insights-runtime-extractor-t79dx\" (UID: \"0a3e5a60-3150-4633-8ff1-4c8ebca70a41\") " pod="openshift-insights/insights-runtime-extractor-t79dx" Jul 07 10:28:10.595730 ip-10-0-135-34 kubenswrapper[2578]: I0707 10:28:10.595694 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/0a3e5a60-3150-4633-8ff1-4c8ebca70a41-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-t79dx\" (UID: \"0a3e5a60-3150-4633-8ff1-4c8ebca70a41\") " pod="openshift-insights/insights-runtime-extractor-t79dx" Jul 07 10:28:10.595730 ip-10-0-135-34 kubenswrapper[2578]: E0707 10:28:10.595719 2578 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Jul 07 10:28:10.595730 ip-10-0-135-34 kubenswrapper[2578]: I0707 10:28:10.595727 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nwspc\" (UniqueName: \"kubernetes.io/projected/0a3e5a60-3150-4633-8ff1-4c8ebca70a41-kube-api-access-nwspc\") pod \"insights-runtime-extractor-t79dx\" (UID: \"0a3e5a60-3150-4633-8ff1-4c8ebca70a41\") " pod="openshift-insights/insights-runtime-extractor-t79dx" Jul 07 10:28:10.596002 ip-10-0-135-34 kubenswrapper[2578]: I0707 10:28:10.595760 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/0a3e5a60-3150-4633-8ff1-4c8ebca70a41-data-volume\") pod \"insights-runtime-extractor-t79dx\" (UID: \"0a3e5a60-3150-4633-8ff1-4c8ebca70a41\") " pod="openshift-insights/insights-runtime-extractor-t79dx" Jul 07 10:28:10.596002 ip-10-0-135-34 kubenswrapper[2578]: E0707 10:28:10.595801 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/0a3e5a60-3150-4633-8ff1-4c8ebca70a41-kube-rbac-proxy-cm podName:0a3e5a60-3150-4633-8ff1-4c8ebca70a41 nodeName:}" failed. No retries permitted until 2026-07-07 10:28:11.095785621 +0000 UTC m=+59.767745156 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/0a3e5a60-3150-4633-8ff1-4c8ebca70a41-kube-rbac-proxy-cm") pod "insights-runtime-extractor-t79dx" (UID: "0a3e5a60-3150-4633-8ff1-4c8ebca70a41") : configmap "kube-rbac-proxy" not found Jul 07 10:28:10.596002 ip-10-0-135-34 kubenswrapper[2578]: E0707 10:28:10.595851 2578 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Jul 07 10:28:10.596002 ip-10-0-135-34 kubenswrapper[2578]: E0707 10:28:10.595907 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0a3e5a60-3150-4633-8ff1-4c8ebca70a41-insights-runtime-extractor-tls podName:0a3e5a60-3150-4633-8ff1-4c8ebca70a41 nodeName:}" failed. No retries permitted until 2026-07-07 10:28:11.095888739 +0000 UTC m=+59.767848274 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/0a3e5a60-3150-4633-8ff1-4c8ebca70a41-insights-runtime-extractor-tls") pod "insights-runtime-extractor-t79dx" (UID: "0a3e5a60-3150-4633-8ff1-4c8ebca70a41") : secret "insights-runtime-extractor-tls" not found Jul 07 10:28:10.605901 ip-10-0-135-34 kubenswrapper[2578]: I0707 10:28:10.605871 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nwspc\" (UniqueName: \"kubernetes.io/projected/0a3e5a60-3150-4633-8ff1-4c8ebca70a41-kube-api-access-nwspc\") pod \"insights-runtime-extractor-t79dx\" (UID: \"0a3e5a60-3150-4633-8ff1-4c8ebca70a41\") " pod="openshift-insights/insights-runtime-extractor-t79dx" Jul 07 10:28:11.099861 ip-10-0-135-34 kubenswrapper[2578]: I0707 10:28:11.099821 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/0a3e5a60-3150-4633-8ff1-4c8ebca70a41-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-t79dx\" (UID: \"0a3e5a60-3150-4633-8ff1-4c8ebca70a41\") " pod="openshift-insights/insights-runtime-extractor-t79dx" Jul 07 10:28:11.099861 ip-10-0-135-34 kubenswrapper[2578]: I0707 10:28:11.099870 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/0a3e5a60-3150-4633-8ff1-4c8ebca70a41-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-t79dx\" (UID: \"0a3e5a60-3150-4633-8ff1-4c8ebca70a41\") " pod="openshift-insights/insights-runtime-extractor-t79dx" Jul 07 10:28:11.100104 ip-10-0-135-34 kubenswrapper[2578]: E0707 10:28:11.099950 2578 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Jul 07 10:28:11.100104 ip-10-0-135-34 kubenswrapper[2578]: E0707 10:28:11.099959 2578 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Jul 07 10:28:11.100104 ip-10-0-135-34 kubenswrapper[2578]: E0707 10:28:11.100016 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0a3e5a60-3150-4633-8ff1-4c8ebca70a41-insights-runtime-extractor-tls podName:0a3e5a60-3150-4633-8ff1-4c8ebca70a41 nodeName:}" failed. No retries permitted until 2026-07-07 10:28:12.100003094 +0000 UTC m=+60.771962616 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/0a3e5a60-3150-4633-8ff1-4c8ebca70a41-insights-runtime-extractor-tls") pod "insights-runtime-extractor-t79dx" (UID: "0a3e5a60-3150-4633-8ff1-4c8ebca70a41") : secret "insights-runtime-extractor-tls" not found Jul 07 10:28:11.100104 ip-10-0-135-34 kubenswrapper[2578]: E0707 10:28:11.100085 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/0a3e5a60-3150-4633-8ff1-4c8ebca70a41-kube-rbac-proxy-cm podName:0a3e5a60-3150-4633-8ff1-4c8ebca70a41 nodeName:}" failed. No retries permitted until 2026-07-07 10:28:12.100065271 +0000 UTC m=+60.772024796 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/0a3e5a60-3150-4633-8ff1-4c8ebca70a41-kube-rbac-proxy-cm") pod "insights-runtime-extractor-t79dx" (UID: "0a3e5a60-3150-4633-8ff1-4c8ebca70a41") : configmap "kube-rbac-proxy" not found Jul 07 10:28:12.108232 ip-10-0-135-34 kubenswrapper[2578]: I0707 10:28:12.108189 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/0a3e5a60-3150-4633-8ff1-4c8ebca70a41-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-t79dx\" (UID: \"0a3e5a60-3150-4633-8ff1-4c8ebca70a41\") " pod="openshift-insights/insights-runtime-extractor-t79dx" Jul 07 10:28:12.108632 ip-10-0-135-34 kubenswrapper[2578]: I0707 10:28:12.108242 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/0a3e5a60-3150-4633-8ff1-4c8ebca70a41-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-t79dx\" (UID: \"0a3e5a60-3150-4633-8ff1-4c8ebca70a41\") " pod="openshift-insights/insights-runtime-extractor-t79dx" Jul 07 10:28:12.108632 ip-10-0-135-34 kubenswrapper[2578]: E0707 10:28:12.108318 2578 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Jul 07 10:28:12.108632 ip-10-0-135-34 kubenswrapper[2578]: E0707 10:28:12.108375 2578 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Jul 07 10:28:12.108632 ip-10-0-135-34 kubenswrapper[2578]: E0707 10:28:12.108388 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/0a3e5a60-3150-4633-8ff1-4c8ebca70a41-kube-rbac-proxy-cm podName:0a3e5a60-3150-4633-8ff1-4c8ebca70a41 nodeName:}" failed. No retries permitted until 2026-07-07 10:28:14.108369076 +0000 UTC m=+62.780328598 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/0a3e5a60-3150-4633-8ff1-4c8ebca70a41-kube-rbac-proxy-cm") pod "insights-runtime-extractor-t79dx" (UID: "0a3e5a60-3150-4633-8ff1-4c8ebca70a41") : configmap "kube-rbac-proxy" not found Jul 07 10:28:12.108632 ip-10-0-135-34 kubenswrapper[2578]: E0707 10:28:12.108411 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0a3e5a60-3150-4633-8ff1-4c8ebca70a41-insights-runtime-extractor-tls podName:0a3e5a60-3150-4633-8ff1-4c8ebca70a41 nodeName:}" failed. No retries permitted until 2026-07-07 10:28:14.108400232 +0000 UTC m=+62.780359753 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/0a3e5a60-3150-4633-8ff1-4c8ebca70a41-insights-runtime-extractor-tls") pod "insights-runtime-extractor-t79dx" (UID: "0a3e5a60-3150-4633-8ff1-4c8ebca70a41") : secret "insights-runtime-extractor-tls" not found Jul 07 10:28:12.361953 ip-10-0-135-34 kubenswrapper[2578]: I0707 10:28:12.361883 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-rdrnq_dfa64e14-3729-4f28-870f-eeab6cec9944/dns-node-resolver/0.log" Jul 07 10:28:12.914791 ip-10-0-135-34 kubenswrapper[2578]: I0707 10:28:12.914738 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/327fdc7e-f662-4adf-a589-5a22e48daa18-networking-console-plugin-cert\") pod \"networking-console-plugin-6bffdf9778-lmcz7\" (UID: \"327fdc7e-f662-4adf-a589-5a22e48daa18\") " pod="openshift-network-console/networking-console-plugin-6bffdf9778-lmcz7" Jul 07 10:28:12.914962 ip-10-0-135-34 kubenswrapper[2578]: E0707 10:28:12.914896 2578 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found Jul 07 10:28:12.914962 ip-10-0-135-34 kubenswrapper[2578]: E0707 10:28:12.914960 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/327fdc7e-f662-4adf-a589-5a22e48daa18-networking-console-plugin-cert podName:327fdc7e-f662-4adf-a589-5a22e48daa18 nodeName:}" failed. No retries permitted until 2026-07-07 10:28:20.914943959 +0000 UTC m=+69.586903500 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/327fdc7e-f662-4adf-a589-5a22e48daa18-networking-console-plugin-cert") pod "networking-console-plugin-6bffdf9778-lmcz7" (UID: "327fdc7e-f662-4adf-a589-5a22e48daa18") : secret "networking-console-plugin-cert" not found Jul 07 10:28:12.920630 ip-10-0-135-34 kubenswrapper[2578]: I0707 10:28:12.920604 2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-service-ca/service-ca-758f599c5-s2tl7"] Jul 07 10:28:12.923199 ip-10-0-135-34 kubenswrapper[2578]: I0707 10:28:12.923183 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca/service-ca-758f599c5-s2tl7" Jul 07 10:28:12.926067 ip-10-0-135-34 kubenswrapper[2578]: I0707 10:28:12.926031 2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"signing-cabundle\"" Jul 07 10:28:12.926067 ip-10-0-135-34 kubenswrapper[2578]: I0707 10:28:12.926061 2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"openshift-service-ca.crt\"" Jul 07 10:28:12.926235 ip-10-0-135-34 kubenswrapper[2578]: I0707 10:28:12.926067 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca\"/\"signing-key\"" Jul 07 10:28:12.927228 ip-10-0-135-34 kubenswrapper[2578]: I0707 10:28:12.927210 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca\"/\"service-ca-dockercfg-4kv7t\"" Jul 07 10:28:12.927336 ip-10-0-135-34 kubenswrapper[2578]: I0707 10:28:12.927245 2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"kube-root-ca.crt\"" Jul 07 10:28:12.935727 ip-10-0-135-34 kubenswrapper[2578]: I0707 10:28:12.935705 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca/service-ca-758f599c5-s2tl7"] Jul 07 10:28:13.015512 ip-10-0-135-34 kubenswrapper[2578]: I0707 10:28:13.015478 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/c7cf9ddc-58aa-4084-855a-5144e7f8c060-signing-key\") pod \"service-ca-758f599c5-s2tl7\" (UID: \"c7cf9ddc-58aa-4084-855a-5144e7f8c060\") " pod="openshift-service-ca/service-ca-758f599c5-s2tl7" Jul 07 10:28:13.015512 ip-10-0-135-34 kubenswrapper[2578]: I0707 10:28:13.015516 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pmwfn\" (UniqueName: \"kubernetes.io/projected/c7cf9ddc-58aa-4084-855a-5144e7f8c060-kube-api-access-pmwfn\") pod \"service-ca-758f599c5-s2tl7\" (UID: \"c7cf9ddc-58aa-4084-855a-5144e7f8c060\") " pod="openshift-service-ca/service-ca-758f599c5-s2tl7" Jul 07 10:28:13.015745 ip-10-0-135-34 kubenswrapper[2578]: I0707 10:28:13.015587 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/2ee234eb-c9b5-48d9-972a-ed53412ca83e-samples-operator-tls\") pod \"cluster-samples-operator-56fcf86bf-lc4lc\" (UID: \"2ee234eb-c9b5-48d9-972a-ed53412ca83e\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-56fcf86bf-lc4lc" Jul 07 10:28:13.015745 ip-10-0-135-34 kubenswrapper[2578]: E0707 10:28:13.015640 2578 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found Jul 07 10:28:13.015745 ip-10-0-135-34 kubenswrapper[2578]: I0707 10:28:13.015650 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/c7cf9ddc-58aa-4084-855a-5144e7f8c060-signing-cabundle\") pod \"service-ca-758f599c5-s2tl7\" (UID: \"c7cf9ddc-58aa-4084-855a-5144e7f8c060\") " pod="openshift-service-ca/service-ca-758f599c5-s2tl7" Jul 07 10:28:13.015745 ip-10-0-135-34 kubenswrapper[2578]: E0707 10:28:13.015696 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2ee234eb-c9b5-48d9-972a-ed53412ca83e-samples-operator-tls podName:2ee234eb-c9b5-48d9-972a-ed53412ca83e nodeName:}" failed. No retries permitted until 2026-07-07 10:28:21.015681199 +0000 UTC m=+69.687640721 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/2ee234eb-c9b5-48d9-972a-ed53412ca83e-samples-operator-tls") pod "cluster-samples-operator-56fcf86bf-lc4lc" (UID: "2ee234eb-c9b5-48d9-972a-ed53412ca83e") : secret "samples-operator-tls" not found Jul 07 10:28:13.116654 ip-10-0-135-34 kubenswrapper[2578]: I0707 10:28:13.116612 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/c7cf9ddc-58aa-4084-855a-5144e7f8c060-signing-key\") pod \"service-ca-758f599c5-s2tl7\" (UID: \"c7cf9ddc-58aa-4084-855a-5144e7f8c060\") " pod="openshift-service-ca/service-ca-758f599c5-s2tl7" Jul 07 10:28:13.116654 ip-10-0-135-34 kubenswrapper[2578]: I0707 10:28:13.116654 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pmwfn\" (UniqueName: \"kubernetes.io/projected/c7cf9ddc-58aa-4084-855a-5144e7f8c060-kube-api-access-pmwfn\") pod \"service-ca-758f599c5-s2tl7\" (UID: \"c7cf9ddc-58aa-4084-855a-5144e7f8c060\") " pod="openshift-service-ca/service-ca-758f599c5-s2tl7" Jul 07 10:28:13.117108 ip-10-0-135-34 kubenswrapper[2578]: I0707 10:28:13.116862 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/c7cf9ddc-58aa-4084-855a-5144e7f8c060-signing-cabundle\") pod \"service-ca-758f599c5-s2tl7\" (UID: \"c7cf9ddc-58aa-4084-855a-5144e7f8c060\") " pod="openshift-service-ca/service-ca-758f599c5-s2tl7" Jul 07 10:28:13.117513 ip-10-0-135-34 kubenswrapper[2578]: I0707 10:28:13.117495 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/c7cf9ddc-58aa-4084-855a-5144e7f8c060-signing-cabundle\") pod \"service-ca-758f599c5-s2tl7\" (UID: \"c7cf9ddc-58aa-4084-855a-5144e7f8c060\") " pod="openshift-service-ca/service-ca-758f599c5-s2tl7" Jul 07 10:28:13.119268 ip-10-0-135-34 kubenswrapper[2578]: I0707 10:28:13.119245 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/c7cf9ddc-58aa-4084-855a-5144e7f8c060-signing-key\") pod \"service-ca-758f599c5-s2tl7\" (UID: \"c7cf9ddc-58aa-4084-855a-5144e7f8c060\") " pod="openshift-service-ca/service-ca-758f599c5-s2tl7" Jul 07 10:28:13.125400 ip-10-0-135-34 kubenswrapper[2578]: I0707 10:28:13.125377 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pmwfn\" (UniqueName: \"kubernetes.io/projected/c7cf9ddc-58aa-4084-855a-5144e7f8c060-kube-api-access-pmwfn\") pod \"service-ca-758f599c5-s2tl7\" (UID: \"c7cf9ddc-58aa-4084-855a-5144e7f8c060\") " pod="openshift-service-ca/service-ca-758f599c5-s2tl7" Jul 07 10:28:13.231407 ip-10-0-135-34 kubenswrapper[2578]: I0707 10:28:13.231321 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca/service-ca-758f599c5-s2tl7" Jul 07 10:28:13.358118 ip-10-0-135-34 kubenswrapper[2578]: I0707 10:28:13.358086 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca/service-ca-758f599c5-s2tl7"] Jul 07 10:28:13.361168 ip-10-0-135-34 kubenswrapper[2578]: W0707 10:28:13.361129 2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc7cf9ddc_58aa_4084_855a_5144e7f8c060.slice/crio-a204c67885e3dd701f43ec9baf29c8a37a6ffad9181b49720456601893a564c6 WatchSource:0}: Error finding container a204c67885e3dd701f43ec9baf29c8a37a6ffad9181b49720456601893a564c6: Status 404 returned error can't find the container with id a204c67885e3dd701f43ec9baf29c8a37a6ffad9181b49720456601893a564c6 Jul 07 10:28:13.361653 ip-10-0-135-34 kubenswrapper[2578]: I0707 10:28:13.361636 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-pb2p8_025a30f8-f923-4bc8-adf5-7ef97ee3bfd3/node-ca/0.log" Jul 07 10:28:14.004554 ip-10-0-135-34 kubenswrapper[2578]: I0707 10:28:14.004524 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-qhhw7" Jul 07 10:28:14.088783 ip-10-0-135-34 kubenswrapper[2578]: I0707 10:28:14.088730 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca/service-ca-758f599c5-s2tl7" event={"ID":"c7cf9ddc-58aa-4084-855a-5144e7f8c060","Type":"ContainerStarted","Data":"a204c67885e3dd701f43ec9baf29c8a37a6ffad9181b49720456601893a564c6"} Jul 07 10:28:14.126642 ip-10-0-135-34 kubenswrapper[2578]: I0707 10:28:14.126606 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/0a3e5a60-3150-4633-8ff1-4c8ebca70a41-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-t79dx\" (UID: \"0a3e5a60-3150-4633-8ff1-4c8ebca70a41\") " pod="openshift-insights/insights-runtime-extractor-t79dx" Jul 07 10:28:14.127103 ip-10-0-135-34 kubenswrapper[2578]: I0707 10:28:14.126704 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/0a3e5a60-3150-4633-8ff1-4c8ebca70a41-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-t79dx\" (UID: \"0a3e5a60-3150-4633-8ff1-4c8ebca70a41\") " pod="openshift-insights/insights-runtime-extractor-t79dx" Jul 07 10:28:14.127103 ip-10-0-135-34 kubenswrapper[2578]: E0707 10:28:14.126809 2578 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Jul 07 10:28:14.127103 ip-10-0-135-34 kubenswrapper[2578]: E0707 10:28:14.126823 2578 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Jul 07 10:28:14.127103 ip-10-0-135-34 kubenswrapper[2578]: E0707 10:28:14.126886 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0a3e5a60-3150-4633-8ff1-4c8ebca70a41-insights-runtime-extractor-tls podName:0a3e5a60-3150-4633-8ff1-4c8ebca70a41 nodeName:}" failed. No retries permitted until 2026-07-07 10:28:18.126865473 +0000 UTC m=+66.798824998 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/0a3e5a60-3150-4633-8ff1-4c8ebca70a41-insights-runtime-extractor-tls") pod "insights-runtime-extractor-t79dx" (UID: "0a3e5a60-3150-4633-8ff1-4c8ebca70a41") : secret "insights-runtime-extractor-tls" not found Jul 07 10:28:14.127103 ip-10-0-135-34 kubenswrapper[2578]: E0707 10:28:14.126910 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/0a3e5a60-3150-4633-8ff1-4c8ebca70a41-kube-rbac-proxy-cm podName:0a3e5a60-3150-4633-8ff1-4c8ebca70a41 nodeName:}" failed. No retries permitted until 2026-07-07 10:28:18.126898635 +0000 UTC m=+66.798858163 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/0a3e5a60-3150-4633-8ff1-4c8ebca70a41-kube-rbac-proxy-cm") pod "insights-runtime-extractor-t79dx" (UID: "0a3e5a60-3150-4633-8ff1-4c8ebca70a41") : configmap "kube-rbac-proxy" not found Jul 07 10:28:14.365137 ip-10-0-135-34 kubenswrapper[2578]: I0707 10:28:14.365041 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-s9zln_eae6c59c-de76-4a6e-8f2b-1de5dbeedf78/serve-healthcheck-canary/0.log" Jul 07 10:28:16.095135 ip-10-0-135-34 kubenswrapper[2578]: I0707 10:28:16.095096 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca/service-ca-758f599c5-s2tl7" event={"ID":"c7cf9ddc-58aa-4084-855a-5144e7f8c060","Type":"ContainerStarted","Data":"82541ebdcb218b9dfda67eaf5d0e9730b1d68fc96d21e5ae798597c7ddfb2004"} Jul 07 10:28:16.124817 ip-10-0-135-34 kubenswrapper[2578]: I0707 10:28:16.124738 2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-service-ca/service-ca-758f599c5-s2tl7" podStartSLOduration=2.262500985 podStartE2EDuration="4.124715102s" totalImagesPullingTime="1.862214117s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-07 10:28:12 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 10:28:16.124063165 +0000 UTC m=+64.796022728" watchObservedRunningTime="2026-07-07 10:28:16.124715102 +0000 UTC m=+64.796674648" Jul 07 10:28:17.555901 ip-10-0-135-34 kubenswrapper[2578]: I0707 10:28:17.555857 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/931a7d75-9ee8-4758-9f23-3521f3db5eae-metrics-certs\") pod \"network-metrics-daemon-p2ks2\" (UID: \"931a7d75-9ee8-4758-9f23-3521f3db5eae\") " pod="openshift-multus/network-metrics-daemon-p2ks2" Jul 07 10:28:17.558621 ip-10-0-135-34 kubenswrapper[2578]: I0707 10:28:17.558603 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\"" Jul 07 10:28:17.567059 ip-10-0-135-34 kubenswrapper[2578]: E0707 10:28:17.567029 2578 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found Jul 07 10:28:17.567149 ip-10-0-135-34 kubenswrapper[2578]: E0707 10:28:17.567120 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/931a7d75-9ee8-4758-9f23-3521f3db5eae-metrics-certs podName:931a7d75-9ee8-4758-9f23-3521f3db5eae nodeName:}" failed. No retries permitted until 2026-07-07 10:29:21.567104809 +0000 UTC m=+130.239064345 (durationBeforeRetry 1m4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/931a7d75-9ee8-4758-9f23-3521f3db5eae-metrics-certs") pod "network-metrics-daemon-p2ks2" (UID: "931a7d75-9ee8-4758-9f23-3521f3db5eae") : secret "metrics-daemon-secret" not found Jul 07 10:28:17.656418 ip-10-0-135-34 kubenswrapper[2578]: I0707 10:28:17.656378 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/f85fac08-720e-4ffd-80af-19c140c36804-original-pull-secret\") pod \"global-pull-secret-syncer-7lv79\" (UID: \"f85fac08-720e-4ffd-80af-19c140c36804\") " pod="kube-system/global-pull-secret-syncer-7lv79" Jul 07 10:28:17.656617 ip-10-0-135-34 kubenswrapper[2578]: I0707 10:28:17.656521 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9vbqq\" (UniqueName: \"kubernetes.io/projected/51fe14a4-6b46-4f88-be7d-b611b45961ff-kube-api-access-9vbqq\") pod \"network-check-target-26sg7\" (UID: \"51fe14a4-6b46-4f88-be7d-b611b45961ff\") " pod="openshift-network-diagnostics/network-check-target-26sg7" Jul 07 10:28:17.659120 ip-10-0-135-34 kubenswrapper[2578]: I0707 10:28:17.659096 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9vbqq\" (UniqueName: \"kubernetes.io/projected/51fe14a4-6b46-4f88-be7d-b611b45961ff-kube-api-access-9vbqq\") pod \"network-check-target-26sg7\" (UID: \"51fe14a4-6b46-4f88-be7d-b611b45961ff\") " pod="openshift-network-diagnostics/network-check-target-26sg7" Jul 07 10:28:17.659464 ip-10-0-135-34 kubenswrapper[2578]: I0707 10:28:17.659449 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\"" Jul 07 10:28:17.669273 ip-10-0-135-34 kubenswrapper[2578]: I0707 10:28:17.669249 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/f85fac08-720e-4ffd-80af-19c140c36804-original-pull-secret\") pod \"global-pull-secret-syncer-7lv79\" (UID: \"f85fac08-720e-4ffd-80af-19c140c36804\") " pod="kube-system/global-pull-secret-syncer-7lv79" Jul 07 10:28:17.931751 ip-10-0-135-34 kubenswrapper[2578]: I0707 10:28:17.931698 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-7lv79" Jul 07 10:28:17.938282 ip-10-0-135-34 kubenswrapper[2578]: I0707 10:28:17.938264 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-qdqsn\"" Jul 07 10:28:17.945996 ip-10-0-135-34 kubenswrapper[2578]: I0707 10:28:17.945971 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-26sg7" Jul 07 10:28:18.071446 ip-10-0-135-34 kubenswrapper[2578]: I0707 10:28:18.071414 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-7lv79"] Jul 07 10:28:18.074277 ip-10-0-135-34 kubenswrapper[2578]: W0707 10:28:18.074250 2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf85fac08_720e_4ffd_80af_19c140c36804.slice/crio-a724b120c9177af99e11ef2c737e1781fed491b12320851233298d44f9d6f60c WatchSource:0}: Error finding container a724b120c9177af99e11ef2c737e1781fed491b12320851233298d44f9d6f60c: Status 404 returned error can't find the container with id a724b120c9177af99e11ef2c737e1781fed491b12320851233298d44f9d6f60c Jul 07 10:28:18.090140 ip-10-0-135-34 kubenswrapper[2578]: I0707 10:28:18.090113 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-26sg7"] Jul 07 10:28:18.094467 ip-10-0-135-34 kubenswrapper[2578]: W0707 10:28:18.094439 2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod51fe14a4_6b46_4f88_be7d_b611b45961ff.slice/crio-5cb83ed55afa7b8bb213e1d0ca7af690bf169161541cb2a1e9d724f30f9fe95d WatchSource:0}: Error finding container 5cb83ed55afa7b8bb213e1d0ca7af690bf169161541cb2a1e9d724f30f9fe95d: Status 404 returned error can't find the container with id 5cb83ed55afa7b8bb213e1d0ca7af690bf169161541cb2a1e9d724f30f9fe95d Jul 07 10:28:18.100881 ip-10-0-135-34 kubenswrapper[2578]: I0707 10:28:18.100846 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-26sg7" event={"ID":"51fe14a4-6b46-4f88-be7d-b611b45961ff","Type":"ContainerStarted","Data":"5cb83ed55afa7b8bb213e1d0ca7af690bf169161541cb2a1e9d724f30f9fe95d"} Jul 07 10:28:18.101871 ip-10-0-135-34 kubenswrapper[2578]: I0707 10:28:18.101852 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-7lv79" event={"ID":"f85fac08-720e-4ffd-80af-19c140c36804","Type":"ContainerStarted","Data":"a724b120c9177af99e11ef2c737e1781fed491b12320851233298d44f9d6f60c"} Jul 07 10:28:18.161380 ip-10-0-135-34 kubenswrapper[2578]: I0707 10:28:18.161352 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/0a3e5a60-3150-4633-8ff1-4c8ebca70a41-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-t79dx\" (UID: \"0a3e5a60-3150-4633-8ff1-4c8ebca70a41\") " pod="openshift-insights/insights-runtime-extractor-t79dx" Jul 07 10:28:18.161487 ip-10-0-135-34 kubenswrapper[2578]: I0707 10:28:18.161421 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/0a3e5a60-3150-4633-8ff1-4c8ebca70a41-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-t79dx\" (UID: \"0a3e5a60-3150-4633-8ff1-4c8ebca70a41\") " pod="openshift-insights/insights-runtime-extractor-t79dx" Jul 07 10:28:18.161532 ip-10-0-135-34 kubenswrapper[2578]: E0707 10:28:18.161494 2578 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Jul 07 10:28:18.161532 ip-10-0-135-34 kubenswrapper[2578]: E0707 10:28:18.161512 2578 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Jul 07 10:28:18.161594 ip-10-0-135-34 kubenswrapper[2578]: E0707 10:28:18.161553 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0a3e5a60-3150-4633-8ff1-4c8ebca70a41-insights-runtime-extractor-tls podName:0a3e5a60-3150-4633-8ff1-4c8ebca70a41 nodeName:}" failed. No retries permitted until 2026-07-07 10:28:26.161539311 +0000 UTC m=+74.833498832 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/0a3e5a60-3150-4633-8ff1-4c8ebca70a41-insights-runtime-extractor-tls") pod "insights-runtime-extractor-t79dx" (UID: "0a3e5a60-3150-4633-8ff1-4c8ebca70a41") : secret "insights-runtime-extractor-tls" not found Jul 07 10:28:18.161594 ip-10-0-135-34 kubenswrapper[2578]: E0707 10:28:18.161570 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/0a3e5a60-3150-4633-8ff1-4c8ebca70a41-kube-rbac-proxy-cm podName:0a3e5a60-3150-4633-8ff1-4c8ebca70a41 nodeName:}" failed. No retries permitted until 2026-07-07 10:28:26.161563473 +0000 UTC m=+74.833522994 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/0a3e5a60-3150-4633-8ff1-4c8ebca70a41-kube-rbac-proxy-cm") pod "insights-runtime-extractor-t79dx" (UID: "0a3e5a60-3150-4633-8ff1-4c8ebca70a41") : configmap "kube-rbac-proxy" not found Jul 07 10:28:19.105850 ip-10-0-135-34 kubenswrapper[2578]: I0707 10:28:19.105813 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-26sg7" event={"ID":"51fe14a4-6b46-4f88-be7d-b611b45961ff","Type":"ContainerStarted","Data":"2bce01dc8b1aa6b2a05a7ee8792e660dfcad42246410b7fa5f6bb62cbe1299ba"} Jul 07 10:28:19.106445 ip-10-0-135-34 kubenswrapper[2578]: I0707 10:28:19.105964 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-26sg7" Jul 07 10:28:19.138616 ip-10-0-135-34 kubenswrapper[2578]: I0707 10:28:19.138553 2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-target-26sg7" podStartSLOduration=67.138530458 podStartE2EDuration="1m7.138530458s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-07 10:27:12 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 10:28:19.13764083 +0000 UTC m=+67.809600374" watchObservedRunningTime="2026-07-07 10:28:19.138530458 +0000 UTC m=+67.810490003" Jul 07 10:28:20.080546 ip-10-0-135-34 kubenswrapper[2578]: I0707 10:28:20.080504 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/a1048018-11a0-4d5a-96ab-86508dac8366-metrics-tls\") pod \"dns-default-j2ks5\" (UID: \"a1048018-11a0-4d5a-96ab-86508dac8366\") " pod="openshift-dns/dns-default-j2ks5" Jul 07 10:28:20.080763 ip-10-0-135-34 kubenswrapper[2578]: I0707 10:28:20.080579 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/c4552170-c8e7-43e3-9472-a066d5379772-registry-tls\") pod \"image-registry-cb7847d49-4nszw\" (UID: \"c4552170-c8e7-43e3-9472-a066d5379772\") " pod="openshift-image-registry/image-registry-cb7847d49-4nszw" Jul 07 10:28:20.080763 ip-10-0-135-34 kubenswrapper[2578]: E0707 10:28:20.080679 2578 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jul 07 10:28:20.080763 ip-10-0-135-34 kubenswrapper[2578]: E0707 10:28:20.080756 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a1048018-11a0-4d5a-96ab-86508dac8366-metrics-tls podName:a1048018-11a0-4d5a-96ab-86508dac8366 nodeName:}" failed. No retries permitted until 2026-07-07 10:28:52.080734507 +0000 UTC m=+100.752694042 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/a1048018-11a0-4d5a-96ab-86508dac8366-metrics-tls") pod "dns-default-j2ks5" (UID: "a1048018-11a0-4d5a-96ab-86508dac8366") : secret "dns-default-metrics-tls" not found Jul 07 10:28:20.083408 ip-10-0-135-34 kubenswrapper[2578]: I0707 10:28:20.083383 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/c4552170-c8e7-43e3-9472-a066d5379772-registry-tls\") pod \"image-registry-cb7847d49-4nszw\" (UID: \"c4552170-c8e7-43e3-9472-a066d5379772\") " pod="openshift-image-registry/image-registry-cb7847d49-4nszw" Jul 07 10:28:20.332573 ip-10-0-135-34 kubenswrapper[2578]: I0707 10:28:20.332495 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"registry-dockercfg-tvxzc\"" Jul 07 10:28:20.340203 ip-10-0-135-34 kubenswrapper[2578]: I0707 10:28:20.340161 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-cb7847d49-4nszw" Jul 07 10:28:20.500045 ip-10-0-135-34 kubenswrapper[2578]: I0707 10:28:20.499966 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-cb7847d49-4nszw"] Jul 07 10:28:20.503918 ip-10-0-135-34 kubenswrapper[2578]: W0707 10:28:20.503885 2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc4552170_c8e7_43e3_9472_a066d5379772.slice/crio-95d24df0a2dce2b51eae9a8077395857eb821f52e2274003b9737aff8caf2224 WatchSource:0}: Error finding container 95d24df0a2dce2b51eae9a8077395857eb821f52e2274003b9737aff8caf2224: Status 404 returned error can't find the container with id 95d24df0a2dce2b51eae9a8077395857eb821f52e2274003b9737aff8caf2224 Jul 07 10:28:20.989195 ip-10-0-135-34 kubenswrapper[2578]: I0707 10:28:20.989150 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/327fdc7e-f662-4adf-a589-5a22e48daa18-networking-console-plugin-cert\") pod \"networking-console-plugin-6bffdf9778-lmcz7\" (UID: \"327fdc7e-f662-4adf-a589-5a22e48daa18\") " pod="openshift-network-console/networking-console-plugin-6bffdf9778-lmcz7" Jul 07 10:28:20.989384 ip-10-0-135-34 kubenswrapper[2578]: E0707 10:28:20.989326 2578 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found Jul 07 10:28:20.989441 ip-10-0-135-34 kubenswrapper[2578]: E0707 10:28:20.989422 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/327fdc7e-f662-4adf-a589-5a22e48daa18-networking-console-plugin-cert podName:327fdc7e-f662-4adf-a589-5a22e48daa18 nodeName:}" failed. No retries permitted until 2026-07-07 10:28:36.989396824 +0000 UTC m=+85.661356369 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/327fdc7e-f662-4adf-a589-5a22e48daa18-networking-console-plugin-cert") pod "networking-console-plugin-6bffdf9778-lmcz7" (UID: "327fdc7e-f662-4adf-a589-5a22e48daa18") : secret "networking-console-plugin-cert" not found Jul 07 10:28:21.089956 ip-10-0-135-34 kubenswrapper[2578]: I0707 10:28:21.089911 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/2ee234eb-c9b5-48d9-972a-ed53412ca83e-samples-operator-tls\") pod \"cluster-samples-operator-56fcf86bf-lc4lc\" (UID: \"2ee234eb-c9b5-48d9-972a-ed53412ca83e\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-56fcf86bf-lc4lc" Jul 07 10:28:21.093016 ip-10-0-135-34 kubenswrapper[2578]: I0707 10:28:21.092964 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/2ee234eb-c9b5-48d9-972a-ed53412ca83e-samples-operator-tls\") pod \"cluster-samples-operator-56fcf86bf-lc4lc\" (UID: \"2ee234eb-c9b5-48d9-972a-ed53412ca83e\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-56fcf86bf-lc4lc" Jul 07 10:28:21.106483 ip-10-0-135-34 kubenswrapper[2578]: I0707 10:28:21.106455 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-samples-operator\"/\"cluster-samples-operator-dockercfg-cw69h\"" Jul 07 10:28:21.113743 ip-10-0-135-34 kubenswrapper[2578]: I0707 10:28:21.113708 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-cb7847d49-4nszw" event={"ID":"c4552170-c8e7-43e3-9472-a066d5379772","Type":"ContainerStarted","Data":"b055f09fd0b50ea99f0dfed3b12956f71b9216d7b6ab504375e0d391b44325a1"} Jul 07 10:28:21.113913 ip-10-0-135-34 kubenswrapper[2578]: I0707 10:28:21.113788 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-cb7847d49-4nszw" event={"ID":"c4552170-c8e7-43e3-9472-a066d5379772","Type":"ContainerStarted","Data":"95d24df0a2dce2b51eae9a8077395857eb821f52e2274003b9737aff8caf2224"} Jul 07 10:28:21.114033 ip-10-0-135-34 kubenswrapper[2578]: I0707 10:28:21.114009 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-samples-operator/cluster-samples-operator-56fcf86bf-lc4lc" Jul 07 10:28:21.114523 ip-10-0-135-34 kubenswrapper[2578]: I0707 10:28:21.114507 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-image-registry/image-registry-cb7847d49-4nszw" Jul 07 10:28:21.139160 ip-10-0-135-34 kubenswrapper[2578]: I0707 10:28:21.139080 2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/image-registry-cb7847d49-4nszw" podStartSLOduration=69.139063753 podStartE2EDuration="1m9.139063753s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-07 10:27:12 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 10:28:21.138013968 +0000 UTC m=+69.809973513" watchObservedRunningTime="2026-07-07 10:28:21.139063753 +0000 UTC m=+69.811023300" Jul 07 10:28:22.541481 ip-10-0-135-34 kubenswrapper[2578]: I0707 10:28:22.541450 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-samples-operator/cluster-samples-operator-56fcf86bf-lc4lc"] Jul 07 10:28:23.121311 ip-10-0-135-34 kubenswrapper[2578]: I0707 10:28:23.121272 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-samples-operator/cluster-samples-operator-56fcf86bf-lc4lc" event={"ID":"2ee234eb-c9b5-48d9-972a-ed53412ca83e","Type":"ContainerStarted","Data":"36ac9d9a16d262163f35436397d7adc044688430621db9fd9b3d548d7a5cc1aa"} Jul 07 10:28:23.122789 ip-10-0-135-34 kubenswrapper[2578]: I0707 10:28:23.122748 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-7lv79" event={"ID":"f85fac08-720e-4ffd-80af-19c140c36804","Type":"ContainerStarted","Data":"3d6abe1bf31ab8aff6ff06765866a48d93e724ce2d928cf4f7c468e9bd2da012"} Jul 07 10:28:23.143584 ip-10-0-135-34 kubenswrapper[2578]: I0707 10:28:23.143529 2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/global-pull-secret-syncer-7lv79" podStartSLOduration=66.788118353 podStartE2EDuration="1m11.143513094s" totalImagesPullingTime="4.355394741s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-07 10:27:12 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 10:28:23.143064758 +0000 UTC m=+71.815024298" watchObservedRunningTime="2026-07-07 10:28:23.143513094 +0000 UTC m=+71.815472628" Jul 07 10:28:25.131604 ip-10-0-135-34 kubenswrapper[2578]: I0707 10:28:25.131559 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-samples-operator/cluster-samples-operator-56fcf86bf-lc4lc" event={"ID":"2ee234eb-c9b5-48d9-972a-ed53412ca83e","Type":"ContainerStarted","Data":"84822b5cff88d551f56da1324098249aebd2fe157a46d948dc84a5366a06a61e"} Jul 07 10:28:25.132033 ip-10-0-135-34 kubenswrapper[2578]: I0707 10:28:25.131622 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-samples-operator/cluster-samples-operator-56fcf86bf-lc4lc" event={"ID":"2ee234eb-c9b5-48d9-972a-ed53412ca83e","Type":"ContainerStarted","Data":"e19cd341b973fe287503890c4eab107e267f983716c72231dfc6ec98e2a6dda0"} Jul 07 10:28:25.150729 ip-10-0-135-34 kubenswrapper[2578]: I0707 10:28:25.150670 2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-samples-operator/cluster-samples-operator-56fcf86bf-lc4lc" podStartSLOduration=18.179841785 podStartE2EDuration="20.15065326s" totalImagesPullingTime="1.970811475s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-07 10:28:05 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 10:28:25.149987833 +0000 UTC m=+73.821947379" watchObservedRunningTime="2026-07-07 10:28:25.15065326 +0000 UTC m=+73.822612782" Jul 07 10:28:26.237005 ip-10-0-135-34 kubenswrapper[2578]: I0707 10:28:26.236943 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/0a3e5a60-3150-4633-8ff1-4c8ebca70a41-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-t79dx\" (UID: \"0a3e5a60-3150-4633-8ff1-4c8ebca70a41\") " pod="openshift-insights/insights-runtime-extractor-t79dx" Jul 07 10:28:26.237442 ip-10-0-135-34 kubenswrapper[2578]: I0707 10:28:26.237148 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/0a3e5a60-3150-4633-8ff1-4c8ebca70a41-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-t79dx\" (UID: \"0a3e5a60-3150-4633-8ff1-4c8ebca70a41\") " pod="openshift-insights/insights-runtime-extractor-t79dx" Jul 07 10:28:26.237442 ip-10-0-135-34 kubenswrapper[2578]: E0707 10:28:26.237292 2578 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Jul 07 10:28:26.237442 ip-10-0-135-34 kubenswrapper[2578]: E0707 10:28:26.237355 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0a3e5a60-3150-4633-8ff1-4c8ebca70a41-insights-runtime-extractor-tls podName:0a3e5a60-3150-4633-8ff1-4c8ebca70a41 nodeName:}" failed. No retries permitted until 2026-07-07 10:28:42.237340457 +0000 UTC m=+90.909299993 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/0a3e5a60-3150-4633-8ff1-4c8ebca70a41-insights-runtime-extractor-tls") pod "insights-runtime-extractor-t79dx" (UID: "0a3e5a60-3150-4633-8ff1-4c8ebca70a41") : secret "insights-runtime-extractor-tls" not found Jul 07 10:28:26.237559 ip-10-0-135-34 kubenswrapper[2578]: I0707 10:28:26.237496 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/0a3e5a60-3150-4633-8ff1-4c8ebca70a41-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-t79dx\" (UID: \"0a3e5a60-3150-4633-8ff1-4c8ebca70a41\") " pod="openshift-insights/insights-runtime-extractor-t79dx" Jul 07 10:28:37.027309 ip-10-0-135-34 kubenswrapper[2578]: I0707 10:28:37.027259 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/327fdc7e-f662-4adf-a589-5a22e48daa18-networking-console-plugin-cert\") pod \"networking-console-plugin-6bffdf9778-lmcz7\" (UID: \"327fdc7e-f662-4adf-a589-5a22e48daa18\") " pod="openshift-network-console/networking-console-plugin-6bffdf9778-lmcz7" Jul 07 10:28:37.029868 ip-10-0-135-34 kubenswrapper[2578]: I0707 10:28:37.029846 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/327fdc7e-f662-4adf-a589-5a22e48daa18-networking-console-plugin-cert\") pod \"networking-console-plugin-6bffdf9778-lmcz7\" (UID: \"327fdc7e-f662-4adf-a589-5a22e48daa18\") " pod="openshift-network-console/networking-console-plugin-6bffdf9778-lmcz7" Jul 07 10:28:37.202034 ip-10-0-135-34 kubenswrapper[2578]: I0707 10:28:37.202003 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-console\"/\"default-dockercfg-676kr\"" Jul 07 10:28:37.209601 ip-10-0-135-34 kubenswrapper[2578]: I0707 10:28:37.209575 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-console/networking-console-plugin-6bffdf9778-lmcz7" Jul 07 10:28:37.356454 ip-10-0-135-34 kubenswrapper[2578]: I0707 10:28:37.356415 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-console/networking-console-plugin-6bffdf9778-lmcz7"] Jul 07 10:28:37.359687 ip-10-0-135-34 kubenswrapper[2578]: W0707 10:28:37.359660 2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod327fdc7e_f662_4adf_a589_5a22e48daa18.slice/crio-7547a5e028fe42df4ef6c25fca1342935468f264f24ef7ac0dfa058bff8d90f4 WatchSource:0}: Error finding container 7547a5e028fe42df4ef6c25fca1342935468f264f24ef7ac0dfa058bff8d90f4: Status 404 returned error can't find the container with id 7547a5e028fe42df4ef6c25fca1342935468f264f24ef7ac0dfa058bff8d90f4 Jul 07 10:28:38.172089 ip-10-0-135-34 kubenswrapper[2578]: I0707 10:28:38.172039 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-console/networking-console-plugin-6bffdf9778-lmcz7" event={"ID":"327fdc7e-f662-4adf-a589-5a22e48daa18","Type":"ContainerStarted","Data":"7547a5e028fe42df4ef6c25fca1342935468f264f24ef7ac0dfa058bff8d90f4"} Jul 07 10:28:39.176505 ip-10-0-135-34 kubenswrapper[2578]: I0707 10:28:39.176467 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-console/networking-console-plugin-6bffdf9778-lmcz7" event={"ID":"327fdc7e-f662-4adf-a589-5a22e48daa18","Type":"ContainerStarted","Data":"ce73467499a1068108c478d6f3ed3ff04565c55b424bba66776a3c4d3bbe5eb2"} Jul 07 10:28:40.345582 ip-10-0-135-34 kubenswrapper[2578]: I0707 10:28:40.345544 2578 patch_prober.go:28] interesting pod/image-registry-cb7847d49-4nszw container/registry namespace/openshift-image-registry: Liveness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body={"errors":[{"code":"UNAVAILABLE","message":"service unavailable","detail":"health check failed: please see /debug/health"}]} Jul 07 10:28:40.345971 ip-10-0-135-34 kubenswrapper[2578]: I0707 10:28:40.345596 2578 prober.go:120] "Probe failed" probeType="Liveness" pod="openshift-image-registry/image-registry-cb7847d49-4nszw" podUID="c4552170-c8e7-43e3-9472-a066d5379772" containerName="registry" probeResult="failure" output="HTTP probe failed with statuscode: 503" Jul 07 10:28:42.271842 ip-10-0-135-34 kubenswrapper[2578]: I0707 10:28:42.271801 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/0a3e5a60-3150-4633-8ff1-4c8ebca70a41-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-t79dx\" (UID: \"0a3e5a60-3150-4633-8ff1-4c8ebca70a41\") " pod="openshift-insights/insights-runtime-extractor-t79dx" Jul 07 10:28:42.274397 ip-10-0-135-34 kubenswrapper[2578]: I0707 10:28:42.274367 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/0a3e5a60-3150-4633-8ff1-4c8ebca70a41-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-t79dx\" (UID: \"0a3e5a60-3150-4633-8ff1-4c8ebca70a41\") " pod="openshift-insights/insights-runtime-extractor-t79dx" Jul 07 10:28:42.297437 ip-10-0-135-34 kubenswrapper[2578]: I0707 10:28:42.297406 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-wt2hz\"" Jul 07 10:28:42.305117 ip-10-0-135-34 kubenswrapper[2578]: I0707 10:28:42.305090 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-t79dx" Jul 07 10:28:42.445618 ip-10-0-135-34 kubenswrapper[2578]: I0707 10:28:42.445538 2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-console/networking-console-plugin-6bffdf9778-lmcz7" podStartSLOduration=36.26075862 podStartE2EDuration="37.445515872s" totalImagesPullingTime="1.184757252s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-07 10:28:05 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 10:28:39.202543249 +0000 UTC m=+87.874502809" watchObservedRunningTime="2026-07-07 10:28:42.445515872 +0000 UTC m=+91.117475417" Jul 07 10:28:42.446682 ip-10-0-135-34 kubenswrapper[2578]: I0707 10:28:42.446641 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-t79dx"] Jul 07 10:28:42.452425 ip-10-0-135-34 kubenswrapper[2578]: W0707 10:28:42.452396 2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0a3e5a60_3150_4633_8ff1_4c8ebca70a41.slice/crio-79b72bb43cf85c2a7a4304bb490cf1810273f6d8f8748d24fb36004947215dab WatchSource:0}: Error finding container 79b72bb43cf85c2a7a4304bb490cf1810273f6d8f8748d24fb36004947215dab: Status 404 returned error can't find the container with id 79b72bb43cf85c2a7a4304bb490cf1810273f6d8f8748d24fb36004947215dab Jul 07 10:28:43.129706 ip-10-0-135-34 kubenswrapper[2578]: I0707 10:28:43.129668 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-image-registry/image-registry-cb7847d49-4nszw" Jul 07 10:28:43.191222 ip-10-0-135-34 kubenswrapper[2578]: I0707 10:28:43.191160 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-t79dx" event={"ID":"0a3e5a60-3150-4633-8ff1-4c8ebca70a41","Type":"ContainerStarted","Data":"79b72bb43cf85c2a7a4304bb490cf1810273f6d8f8748d24fb36004947215dab"} Jul 07 10:28:43.283905 ip-10-0-135-34 kubenswrapper[2578]: I0707 10:28:43.283872 2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-operator-567f7b8b7c-snzgt"] Jul 07 10:28:43.286335 ip-10-0-135-34 kubenswrapper[2578]: I0707 10:28:43.286313 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-567f7b8b7c-snzgt" Jul 07 10:28:43.289538 ip-10-0-135-34 kubenswrapper[2578]: I0707 10:28:43.289513 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-dockercfg-lfjlm\"" Jul 07 10:28:43.289983 ip-10-0-135-34 kubenswrapper[2578]: I0707 10:28:43.289831 2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\"" Jul 07 10:28:43.290609 ip-10-0-135-34 kubenswrapper[2578]: I0707 10:28:43.290589 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-tls\"" Jul 07 10:28:43.292189 ip-10-0-135-34 kubenswrapper[2578]: I0707 10:28:43.290845 2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\"" Jul 07 10:28:43.292189 ip-10-0-135-34 kubenswrapper[2578]: I0707 10:28:43.290898 2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\"" Jul 07 10:28:43.292189 ip-10-0-135-34 kubenswrapper[2578]: I0707 10:28:43.291084 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-kube-rbac-proxy-config\"" Jul 07 10:28:43.308360 ip-10-0-135-34 kubenswrapper[2578]: I0707 10:28:43.308327 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-567f7b8b7c-snzgt"] Jul 07 10:28:43.384401 ip-10-0-135-34 kubenswrapper[2578]: I0707 10:28:43.384087 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ks92h\" (UniqueName: \"kubernetes.io/projected/07c77276-527a-4178-ab38-919317f14156-kube-api-access-ks92h\") pod \"prometheus-operator-567f7b8b7c-snzgt\" (UID: \"07c77276-527a-4178-ab38-919317f14156\") " pod="openshift-monitoring/prometheus-operator-567f7b8b7c-snzgt" Jul 07 10:28:43.384401 ip-10-0-135-34 kubenswrapper[2578]: I0707 10:28:43.384178 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-operator-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/07c77276-527a-4178-ab38-919317f14156-prometheus-operator-kube-rbac-proxy-config\") pod \"prometheus-operator-567f7b8b7c-snzgt\" (UID: \"07c77276-527a-4178-ab38-919317f14156\") " pod="openshift-monitoring/prometheus-operator-567f7b8b7c-snzgt" Jul 07 10:28:43.384401 ip-10-0-135-34 kubenswrapper[2578]: I0707 10:28:43.384213 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/07c77276-527a-4178-ab38-919317f14156-prometheus-operator-tls\") pod \"prometheus-operator-567f7b8b7c-snzgt\" (UID: \"07c77276-527a-4178-ab38-919317f14156\") " pod="openshift-monitoring/prometheus-operator-567f7b8b7c-snzgt" Jul 07 10:28:43.384401 ip-10-0-135-34 kubenswrapper[2578]: I0707 10:28:43.384258 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/07c77276-527a-4178-ab38-919317f14156-metrics-client-ca\") pod \"prometheus-operator-567f7b8b7c-snzgt\" (UID: \"07c77276-527a-4178-ab38-919317f14156\") " pod="openshift-monitoring/prometheus-operator-567f7b8b7c-snzgt" Jul 07 10:28:43.485710 ip-10-0-135-34 kubenswrapper[2578]: I0707 10:28:43.485671 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-operator-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/07c77276-527a-4178-ab38-919317f14156-prometheus-operator-kube-rbac-proxy-config\") pod \"prometheus-operator-567f7b8b7c-snzgt\" (UID: \"07c77276-527a-4178-ab38-919317f14156\") " pod="openshift-monitoring/prometheus-operator-567f7b8b7c-snzgt" Jul 07 10:28:43.485909 ip-10-0-135-34 kubenswrapper[2578]: I0707 10:28:43.485716 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/07c77276-527a-4178-ab38-919317f14156-prometheus-operator-tls\") pod \"prometheus-operator-567f7b8b7c-snzgt\" (UID: \"07c77276-527a-4178-ab38-919317f14156\") " pod="openshift-monitoring/prometheus-operator-567f7b8b7c-snzgt" Jul 07 10:28:43.485909 ip-10-0-135-34 kubenswrapper[2578]: I0707 10:28:43.485764 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/07c77276-527a-4178-ab38-919317f14156-metrics-client-ca\") pod \"prometheus-operator-567f7b8b7c-snzgt\" (UID: \"07c77276-527a-4178-ab38-919317f14156\") " pod="openshift-monitoring/prometheus-operator-567f7b8b7c-snzgt" Jul 07 10:28:43.485909 ip-10-0-135-34 kubenswrapper[2578]: E0707 10:28:43.485882 2578 secret.go:189] Couldn't get secret openshift-monitoring/prometheus-operator-tls: secret "prometheus-operator-tls" not found Jul 07 10:28:43.486063 ip-10-0-135-34 kubenswrapper[2578]: E0707 10:28:43.485958 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/07c77276-527a-4178-ab38-919317f14156-prometheus-operator-tls podName:07c77276-527a-4178-ab38-919317f14156 nodeName:}" failed. No retries permitted until 2026-07-07 10:28:43.985936864 +0000 UTC m=+92.657896403 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "prometheus-operator-tls" (UniqueName: "kubernetes.io/secret/07c77276-527a-4178-ab38-919317f14156-prometheus-operator-tls") pod "prometheus-operator-567f7b8b7c-snzgt" (UID: "07c77276-527a-4178-ab38-919317f14156") : secret "prometheus-operator-tls" not found Jul 07 10:28:43.486251 ip-10-0-135-34 kubenswrapper[2578]: I0707 10:28:43.486233 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ks92h\" (UniqueName: \"kubernetes.io/projected/07c77276-527a-4178-ab38-919317f14156-kube-api-access-ks92h\") pod \"prometheus-operator-567f7b8b7c-snzgt\" (UID: \"07c77276-527a-4178-ab38-919317f14156\") " pod="openshift-monitoring/prometheus-operator-567f7b8b7c-snzgt" Jul 07 10:28:43.486541 ip-10-0-135-34 kubenswrapper[2578]: I0707 10:28:43.486497 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/07c77276-527a-4178-ab38-919317f14156-metrics-client-ca\") pod \"prometheus-operator-567f7b8b7c-snzgt\" (UID: \"07c77276-527a-4178-ab38-919317f14156\") " pod="openshift-monitoring/prometheus-operator-567f7b8b7c-snzgt" Jul 07 10:28:43.488965 ip-10-0-135-34 kubenswrapper[2578]: I0707 10:28:43.488939 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-operator-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/07c77276-527a-4178-ab38-919317f14156-prometheus-operator-kube-rbac-proxy-config\") pod \"prometheus-operator-567f7b8b7c-snzgt\" (UID: \"07c77276-527a-4178-ab38-919317f14156\") " pod="openshift-monitoring/prometheus-operator-567f7b8b7c-snzgt" Jul 07 10:28:43.508640 ip-10-0-135-34 kubenswrapper[2578]: I0707 10:28:43.508608 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ks92h\" (UniqueName: \"kubernetes.io/projected/07c77276-527a-4178-ab38-919317f14156-kube-api-access-ks92h\") pod \"prometheus-operator-567f7b8b7c-snzgt\" (UID: \"07c77276-527a-4178-ab38-919317f14156\") " pod="openshift-monitoring/prometheus-operator-567f7b8b7c-snzgt" Jul 07 10:28:43.896481 ip-10-0-135-34 kubenswrapper[2578]: I0707 10:28:43.896447 2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-fc54c99d9-nh7qj"] Jul 07 10:28:43.898764 ip-10-0-135-34 kubenswrapper[2578]: I0707 10:28:43.898737 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-fc54c99d9-nh7qj" Jul 07 10:28:43.905699 ip-10-0-135-34 kubenswrapper[2578]: I0707 10:28:43.905665 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-oauth-config\"" Jul 07 10:28:43.908440 ip-10-0-135-34 kubenswrapper[2578]: I0707 10:28:43.908419 2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"oauth-serving-cert\"" Jul 07 10:28:43.909445 ip-10-0-135-34 kubenswrapper[2578]: I0707 10:28:43.909415 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-serving-cert\"" Jul 07 10:28:43.909445 ip-10-0-135-34 kubenswrapper[2578]: I0707 10:28:43.909439 2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"service-ca\"" Jul 07 10:28:43.909700 ip-10-0-135-34 kubenswrapper[2578]: I0707 10:28:43.909671 2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"console-config\"" Jul 07 10:28:43.909791 ip-10-0-135-34 kubenswrapper[2578]: I0707 10:28:43.909711 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-dockercfg-d4kbr\"" Jul 07 10:28:43.909791 ip-10-0-135-34 kubenswrapper[2578]: I0707 10:28:43.909714 2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"kube-root-ca.crt\"" Jul 07 10:28:43.909911 ip-10-0-135-34 kubenswrapper[2578]: I0707 10:28:43.909672 2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"openshift-service-ca.crt\"" Jul 07 10:28:43.914915 ip-10-0-135-34 kubenswrapper[2578]: I0707 10:28:43.914891 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-fc54c99d9-nh7qj"] Jul 07 10:28:43.992176 ip-10-0-135-34 kubenswrapper[2578]: I0707 10:28:43.992144 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/f8dcb050-7bdf-40c1-b47b-1019b93b1f9d-console-oauth-config\") pod \"console-fc54c99d9-nh7qj\" (UID: \"f8dcb050-7bdf-40c1-b47b-1019b93b1f9d\") " pod="openshift-console/console-fc54c99d9-nh7qj" Jul 07 10:28:43.992176 ip-10-0-135-34 kubenswrapper[2578]: I0707 10:28:43.992182 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5j64c\" (UniqueName: \"kubernetes.io/projected/f8dcb050-7bdf-40c1-b47b-1019b93b1f9d-kube-api-access-5j64c\") pod \"console-fc54c99d9-nh7qj\" (UID: \"f8dcb050-7bdf-40c1-b47b-1019b93b1f9d\") " pod="openshift-console/console-fc54c99d9-nh7qj" Jul 07 10:28:43.992410 ip-10-0-135-34 kubenswrapper[2578]: I0707 10:28:43.992210 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/f8dcb050-7bdf-40c1-b47b-1019b93b1f9d-service-ca\") pod \"console-fc54c99d9-nh7qj\" (UID: \"f8dcb050-7bdf-40c1-b47b-1019b93b1f9d\") " pod="openshift-console/console-fc54c99d9-nh7qj" Jul 07 10:28:43.992410 ip-10-0-135-34 kubenswrapper[2578]: I0707 10:28:43.992307 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/f8dcb050-7bdf-40c1-b47b-1019b93b1f9d-console-serving-cert\") pod \"console-fc54c99d9-nh7qj\" (UID: \"f8dcb050-7bdf-40c1-b47b-1019b93b1f9d\") " pod="openshift-console/console-fc54c99d9-nh7qj" Jul 07 10:28:43.992410 ip-10-0-135-34 kubenswrapper[2578]: I0707 10:28:43.992349 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/07c77276-527a-4178-ab38-919317f14156-prometheus-operator-tls\") pod \"prometheus-operator-567f7b8b7c-snzgt\" (UID: \"07c77276-527a-4178-ab38-919317f14156\") " pod="openshift-monitoring/prometheus-operator-567f7b8b7c-snzgt" Jul 07 10:28:43.992532 ip-10-0-135-34 kubenswrapper[2578]: I0707 10:28:43.992416 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/f8dcb050-7bdf-40c1-b47b-1019b93b1f9d-console-config\") pod \"console-fc54c99d9-nh7qj\" (UID: \"f8dcb050-7bdf-40c1-b47b-1019b93b1f9d\") " pod="openshift-console/console-fc54c99d9-nh7qj" Jul 07 10:28:43.992532 ip-10-0-135-34 kubenswrapper[2578]: I0707 10:28:43.992446 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/f8dcb050-7bdf-40c1-b47b-1019b93b1f9d-oauth-serving-cert\") pod \"console-fc54c99d9-nh7qj\" (UID: \"f8dcb050-7bdf-40c1-b47b-1019b93b1f9d\") " pod="openshift-console/console-fc54c99d9-nh7qj" Jul 07 10:28:43.995250 ip-10-0-135-34 kubenswrapper[2578]: I0707 10:28:43.995213 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/07c77276-527a-4178-ab38-919317f14156-prometheus-operator-tls\") pod \"prometheus-operator-567f7b8b7c-snzgt\" (UID: \"07c77276-527a-4178-ab38-919317f14156\") " pod="openshift-monitoring/prometheus-operator-567f7b8b7c-snzgt" Jul 07 10:28:44.093161 ip-10-0-135-34 kubenswrapper[2578]: I0707 10:28:44.093119 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/f8dcb050-7bdf-40c1-b47b-1019b93b1f9d-console-oauth-config\") pod \"console-fc54c99d9-nh7qj\" (UID: \"f8dcb050-7bdf-40c1-b47b-1019b93b1f9d\") " pod="openshift-console/console-fc54c99d9-nh7qj" Jul 07 10:28:44.093161 ip-10-0-135-34 kubenswrapper[2578]: I0707 10:28:44.093164 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5j64c\" (UniqueName: \"kubernetes.io/projected/f8dcb050-7bdf-40c1-b47b-1019b93b1f9d-kube-api-access-5j64c\") pod \"console-fc54c99d9-nh7qj\" (UID: \"f8dcb050-7bdf-40c1-b47b-1019b93b1f9d\") " pod="openshift-console/console-fc54c99d9-nh7qj" Jul 07 10:28:44.093418 ip-10-0-135-34 kubenswrapper[2578]: I0707 10:28:44.093186 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/f8dcb050-7bdf-40c1-b47b-1019b93b1f9d-service-ca\") pod \"console-fc54c99d9-nh7qj\" (UID: \"f8dcb050-7bdf-40c1-b47b-1019b93b1f9d\") " pod="openshift-console/console-fc54c99d9-nh7qj" Jul 07 10:28:44.093418 ip-10-0-135-34 kubenswrapper[2578]: I0707 10:28:44.093261 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/f8dcb050-7bdf-40c1-b47b-1019b93b1f9d-console-serving-cert\") pod \"console-fc54c99d9-nh7qj\" (UID: \"f8dcb050-7bdf-40c1-b47b-1019b93b1f9d\") " pod="openshift-console/console-fc54c99d9-nh7qj" Jul 07 10:28:44.093418 ip-10-0-135-34 kubenswrapper[2578]: I0707 10:28:44.093325 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/f8dcb050-7bdf-40c1-b47b-1019b93b1f9d-console-config\") pod \"console-fc54c99d9-nh7qj\" (UID: \"f8dcb050-7bdf-40c1-b47b-1019b93b1f9d\") " pod="openshift-console/console-fc54c99d9-nh7qj" Jul 07 10:28:44.093418 ip-10-0-135-34 kubenswrapper[2578]: I0707 10:28:44.093343 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/f8dcb050-7bdf-40c1-b47b-1019b93b1f9d-oauth-serving-cert\") pod \"console-fc54c99d9-nh7qj\" (UID: \"f8dcb050-7bdf-40c1-b47b-1019b93b1f9d\") " pod="openshift-console/console-fc54c99d9-nh7qj" Jul 07 10:28:44.094096 ip-10-0-135-34 kubenswrapper[2578]: I0707 10:28:44.094032 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/f8dcb050-7bdf-40c1-b47b-1019b93b1f9d-console-config\") pod \"console-fc54c99d9-nh7qj\" (UID: \"f8dcb050-7bdf-40c1-b47b-1019b93b1f9d\") " pod="openshift-console/console-fc54c99d9-nh7qj" Jul 07 10:28:44.094096 ip-10-0-135-34 kubenswrapper[2578]: I0707 10:28:44.094043 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/f8dcb050-7bdf-40c1-b47b-1019b93b1f9d-oauth-serving-cert\") pod \"console-fc54c99d9-nh7qj\" (UID: \"f8dcb050-7bdf-40c1-b47b-1019b93b1f9d\") " pod="openshift-console/console-fc54c99d9-nh7qj" Jul 07 10:28:44.094269 ip-10-0-135-34 kubenswrapper[2578]: I0707 10:28:44.094133 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/f8dcb050-7bdf-40c1-b47b-1019b93b1f9d-service-ca\") pod \"console-fc54c99d9-nh7qj\" (UID: \"f8dcb050-7bdf-40c1-b47b-1019b93b1f9d\") " pod="openshift-console/console-fc54c99d9-nh7qj" Jul 07 10:28:44.096032 ip-10-0-135-34 kubenswrapper[2578]: I0707 10:28:44.096002 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/f8dcb050-7bdf-40c1-b47b-1019b93b1f9d-console-oauth-config\") pod \"console-fc54c99d9-nh7qj\" (UID: \"f8dcb050-7bdf-40c1-b47b-1019b93b1f9d\") " pod="openshift-console/console-fc54c99d9-nh7qj" Jul 07 10:28:44.096184 ip-10-0-135-34 kubenswrapper[2578]: I0707 10:28:44.096161 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/f8dcb050-7bdf-40c1-b47b-1019b93b1f9d-console-serving-cert\") pod \"console-fc54c99d9-nh7qj\" (UID: \"f8dcb050-7bdf-40c1-b47b-1019b93b1f9d\") " pod="openshift-console/console-fc54c99d9-nh7qj" Jul 07 10:28:44.104227 ip-10-0-135-34 kubenswrapper[2578]: I0707 10:28:44.104197 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5j64c\" (UniqueName: \"kubernetes.io/projected/f8dcb050-7bdf-40c1-b47b-1019b93b1f9d-kube-api-access-5j64c\") pod \"console-fc54c99d9-nh7qj\" (UID: \"f8dcb050-7bdf-40c1-b47b-1019b93b1f9d\") " pod="openshift-console/console-fc54c99d9-nh7qj" Jul 07 10:28:44.201015 ip-10-0-135-34 kubenswrapper[2578]: I0707 10:28:44.200937 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-567f7b8b7c-snzgt" Jul 07 10:28:44.210359 ip-10-0-135-34 kubenswrapper[2578]: I0707 10:28:44.210334 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-fc54c99d9-nh7qj" Jul 07 10:28:46.287488 ip-10-0-135-34 kubenswrapper[2578]: I0707 10:28:46.287370 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-fc54c99d9-nh7qj"] Jul 07 10:28:46.289880 ip-10-0-135-34 kubenswrapper[2578]: W0707 10:28:46.289856 2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf8dcb050_7bdf_40c1_b47b_1019b93b1f9d.slice/crio-b60bb3d0c8dbb20ae6d49171f9c2bcaac6c40a249d3899b4976887464614143f WatchSource:0}: Error finding container b60bb3d0c8dbb20ae6d49171f9c2bcaac6c40a249d3899b4976887464614143f: Status 404 returned error can't find the container with id b60bb3d0c8dbb20ae6d49171f9c2bcaac6c40a249d3899b4976887464614143f Jul 07 10:28:46.295272 ip-10-0-135-34 kubenswrapper[2578]: I0707 10:28:46.295245 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-567f7b8b7c-snzgt"] Jul 07 10:28:46.298606 ip-10-0-135-34 kubenswrapper[2578]: W0707 10:28:46.298582 2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod07c77276_527a_4178_ab38_919317f14156.slice/crio-c11bd64ed711d766fb03230d564f147a4ef50de52117861c97795cab891b97c1 WatchSource:0}: Error finding container c11bd64ed711d766fb03230d564f147a4ef50de52117861c97795cab891b97c1: Status 404 returned error can't find the container with id c11bd64ed711d766fb03230d564f147a4ef50de52117861c97795cab891b97c1 Jul 07 10:28:47.208032 ip-10-0-135-34 kubenswrapper[2578]: I0707 10:28:47.207920 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-fc54c99d9-nh7qj" event={"ID":"f8dcb050-7bdf-40c1-b47b-1019b93b1f9d","Type":"ContainerStarted","Data":"b60bb3d0c8dbb20ae6d49171f9c2bcaac6c40a249d3899b4976887464614143f"} Jul 07 10:28:47.209946 ip-10-0-135-34 kubenswrapper[2578]: I0707 10:28:47.209875 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-567f7b8b7c-snzgt" event={"ID":"07c77276-527a-4178-ab38-919317f14156","Type":"ContainerStarted","Data":"c11bd64ed711d766fb03230d564f147a4ef50de52117861c97795cab891b97c1"} Jul 07 10:28:47.211966 ip-10-0-135-34 kubenswrapper[2578]: I0707 10:28:47.211933 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-t79dx" event={"ID":"0a3e5a60-3150-4633-8ff1-4c8ebca70a41","Type":"ContainerStarted","Data":"52fdadf3e6a6e269a1ef2cd6e465a6e11a9145c9373a364e4cb84135279dea52"} Jul 07 10:28:47.212089 ip-10-0-135-34 kubenswrapper[2578]: I0707 10:28:47.211994 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-t79dx" event={"ID":"0a3e5a60-3150-4633-8ff1-4c8ebca70a41","Type":"ContainerStarted","Data":"0d6aa3a6fc7ea0e0ad687e82b8335fa88d82294b14cc2263fc1a11642c387fef"} Jul 07 10:28:48.218645 ip-10-0-135-34 kubenswrapper[2578]: I0707 10:28:48.218595 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-567f7b8b7c-snzgt" event={"ID":"07c77276-527a-4178-ab38-919317f14156","Type":"ContainerStarted","Data":"eb0d263ad0c51ae2e06cde485e2e9bd229fe04c34d55cd043302f59c8f5a0cb8"} Jul 07 10:28:48.219125 ip-10-0-135-34 kubenswrapper[2578]: I0707 10:28:48.218671 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-567f7b8b7c-snzgt" event={"ID":"07c77276-527a-4178-ab38-919317f14156","Type":"ContainerStarted","Data":"d8ebad1c237c204aecb0aafe94b34ce751ffd0633a6d82d11ee35ca44896c945"} Jul 07 10:28:48.248181 ip-10-0-135-34 kubenswrapper[2578]: I0707 10:28:48.247820 2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/prometheus-operator-567f7b8b7c-snzgt" podStartSLOduration=3.82061024 podStartE2EDuration="5.247797368s" totalImagesPullingTime="1.427187128s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-07 10:28:43 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 10:28:48.246412011 +0000 UTC m=+96.918371557" watchObservedRunningTime="2026-07-07 10:28:48.247797368 +0000 UTC m=+96.919756911" Jul 07 10:28:50.111539 ip-10-0-135-34 kubenswrapper[2578]: I0707 10:28:50.111506 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-network-diagnostics/network-check-target-26sg7" Jul 07 10:28:50.778904 ip-10-0-135-34 kubenswrapper[2578]: I0707 10:28:50.777403 2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/openshift-state-metrics-54cf96dfcf-ffft9"] Jul 07 10:28:50.781310 ip-10-0-135-34 kubenswrapper[2578]: I0707 10:28:50.781224 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/openshift-state-metrics-54cf96dfcf-ffft9" Jul 07 10:28:50.788102 ip-10-0-135-34 kubenswrapper[2578]: I0707 10:28:50.787917 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-kube-rbac-proxy-config\"" Jul 07 10:28:50.788102 ip-10-0-135-34 kubenswrapper[2578]: I0707 10:28:50.787940 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-dockercfg-vsvdn\"" Jul 07 10:28:50.790170 ip-10-0-135-34 kubenswrapper[2578]: I0707 10:28:50.788689 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-tls\"" Jul 07 10:28:50.815604 ip-10-0-135-34 kubenswrapper[2578]: I0707 10:28:50.815552 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/openshift-state-metrics-54cf96dfcf-ffft9"] Jul 07 10:28:50.845429 ip-10-0-135-34 kubenswrapper[2578]: I0707 10:28:50.845391 2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/node-exporter-bkr4f"] Jul 07 10:28:50.849096 ip-10-0-135-34 kubenswrapper[2578]: I0707 10:28:50.849063 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-bkr4f" Jul 07 10:28:50.852627 ip-10-0-135-34 kubenswrapper[2578]: I0707 10:28:50.852595 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-lfkb5\"" Jul 07 10:28:50.852828 ip-10-0-135-34 kubenswrapper[2578]: I0707 10:28:50.852595 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\"" Jul 07 10:28:50.853490 ip-10-0-135-34 kubenswrapper[2578]: I0707 10:28:50.853470 2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\"" Jul 07 10:28:50.853576 ip-10-0-135-34 kubenswrapper[2578]: I0707 10:28:50.853501 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\"" Jul 07 10:28:50.958729 ip-10-0-135-34 kubenswrapper[2578]: I0707 10:28:50.958628 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/12d81bd7-9a8f-4221-ae12-72ff1ad8716b-metrics-client-ca\") pod \"node-exporter-bkr4f\" (UID: \"12d81bd7-9a8f-4221-ae12-72ff1ad8716b\") " pod="openshift-monitoring/node-exporter-bkr4f" Jul 07 10:28:50.958729 ip-10-0-135-34 kubenswrapper[2578]: I0707 10:28:50.958678 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/12d81bd7-9a8f-4221-ae12-72ff1ad8716b-root\") pod \"node-exporter-bkr4f\" (UID: \"12d81bd7-9a8f-4221-ae12-72ff1ad8716b\") " pod="openshift-monitoring/node-exporter-bkr4f" Jul 07 10:28:50.958729 ip-10-0-135-34 kubenswrapper[2578]: I0707 10:28:50.958703 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dqtls\" (UniqueName: \"kubernetes.io/projected/12d81bd7-9a8f-4221-ae12-72ff1ad8716b-kube-api-access-dqtls\") pod \"node-exporter-bkr4f\" (UID: \"12d81bd7-9a8f-4221-ae12-72ff1ad8716b\") " pod="openshift-monitoring/node-exporter-bkr4f" Jul 07 10:28:50.958729 ip-10-0-135-34 kubenswrapper[2578]: I0707 10:28:50.958721 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/12d81bd7-9a8f-4221-ae12-72ff1ad8716b-node-exporter-wtmp\") pod \"node-exporter-bkr4f\" (UID: \"12d81bd7-9a8f-4221-ae12-72ff1ad8716b\") " pod="openshift-monitoring/node-exporter-bkr4f" Jul 07 10:28:50.959097 ip-10-0-135-34 kubenswrapper[2578]: I0707 10:28:50.958741 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/12d81bd7-9a8f-4221-ae12-72ff1ad8716b-node-exporter-accelerators-collector-config\") pod \"node-exporter-bkr4f\" (UID: \"12d81bd7-9a8f-4221-ae12-72ff1ad8716b\") " pod="openshift-monitoring/node-exporter-bkr4f" Jul 07 10:28:50.959097 ip-10-0-135-34 kubenswrapper[2578]: I0707 10:28:50.958763 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xprr2\" (UniqueName: \"kubernetes.io/projected/f41994cd-3ce0-43a3-bbd9-a853835da0bf-kube-api-access-xprr2\") pod \"openshift-state-metrics-54cf96dfcf-ffft9\" (UID: \"f41994cd-3ce0-43a3-bbd9-a853835da0bf\") " pod="openshift-monitoring/openshift-state-metrics-54cf96dfcf-ffft9" Jul 07 10:28:50.959097 ip-10-0-135-34 kubenswrapper[2578]: I0707 10:28:50.958857 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/f41994cd-3ce0-43a3-bbd9-a853835da0bf-openshift-state-metrics-tls\") pod \"openshift-state-metrics-54cf96dfcf-ffft9\" (UID: \"f41994cd-3ce0-43a3-bbd9-a853835da0bf\") " pod="openshift-monitoring/openshift-state-metrics-54cf96dfcf-ffft9" Jul 07 10:28:50.959097 ip-10-0-135-34 kubenswrapper[2578]: I0707 10:28:50.958896 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/12d81bd7-9a8f-4221-ae12-72ff1ad8716b-node-exporter-tls\") pod \"node-exporter-bkr4f\" (UID: \"12d81bd7-9a8f-4221-ae12-72ff1ad8716b\") " pod="openshift-monitoring/node-exporter-bkr4f" Jul 07 10:28:50.959097 ip-10-0-135-34 kubenswrapper[2578]: I0707 10:28:50.958914 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/12d81bd7-9a8f-4221-ae12-72ff1ad8716b-sys\") pod \"node-exporter-bkr4f\" (UID: \"12d81bd7-9a8f-4221-ae12-72ff1ad8716b\") " pod="openshift-monitoring/node-exporter-bkr4f" Jul 07 10:28:50.959097 ip-10-0-135-34 kubenswrapper[2578]: I0707 10:28:50.958933 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/12d81bd7-9a8f-4221-ae12-72ff1ad8716b-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-bkr4f\" (UID: \"12d81bd7-9a8f-4221-ae12-72ff1ad8716b\") " pod="openshift-monitoring/node-exporter-bkr4f" Jul 07 10:28:50.959097 ip-10-0-135-34 kubenswrapper[2578]: I0707 10:28:50.958987 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/f41994cd-3ce0-43a3-bbd9-a853835da0bf-metrics-client-ca\") pod \"openshift-state-metrics-54cf96dfcf-ffft9\" (UID: \"f41994cd-3ce0-43a3-bbd9-a853835da0bf\") " pod="openshift-monitoring/openshift-state-metrics-54cf96dfcf-ffft9" Jul 07 10:28:50.959097 ip-10-0-135-34 kubenswrapper[2578]: I0707 10:28:50.959002 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/12d81bd7-9a8f-4221-ae12-72ff1ad8716b-node-exporter-textfile\") pod \"node-exporter-bkr4f\" (UID: \"12d81bd7-9a8f-4221-ae12-72ff1ad8716b\") " pod="openshift-monitoring/node-exporter-bkr4f" Jul 07 10:28:50.959097 ip-10-0-135-34 kubenswrapper[2578]: I0707 10:28:50.959025 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/f41994cd-3ce0-43a3-bbd9-a853835da0bf-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-54cf96dfcf-ffft9\" (UID: \"f41994cd-3ce0-43a3-bbd9-a853835da0bf\") " pod="openshift-monitoring/openshift-state-metrics-54cf96dfcf-ffft9" Jul 07 10:28:51.059908 ip-10-0-135-34 kubenswrapper[2578]: I0707 10:28:51.059873 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/f41994cd-3ce0-43a3-bbd9-a853835da0bf-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-54cf96dfcf-ffft9\" (UID: \"f41994cd-3ce0-43a3-bbd9-a853835da0bf\") " pod="openshift-monitoring/openshift-state-metrics-54cf96dfcf-ffft9" Jul 07 10:28:51.060090 ip-10-0-135-34 kubenswrapper[2578]: I0707 10:28:51.059921 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/12d81bd7-9a8f-4221-ae12-72ff1ad8716b-metrics-client-ca\") pod \"node-exporter-bkr4f\" (UID: \"12d81bd7-9a8f-4221-ae12-72ff1ad8716b\") " pod="openshift-monitoring/node-exporter-bkr4f" Jul 07 10:28:51.060090 ip-10-0-135-34 kubenswrapper[2578]: I0707 10:28:51.059953 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/12d81bd7-9a8f-4221-ae12-72ff1ad8716b-root\") pod \"node-exporter-bkr4f\" (UID: \"12d81bd7-9a8f-4221-ae12-72ff1ad8716b\") " pod="openshift-monitoring/node-exporter-bkr4f" Jul 07 10:28:51.060090 ip-10-0-135-34 kubenswrapper[2578]: I0707 10:28:51.059977 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dqtls\" (UniqueName: \"kubernetes.io/projected/12d81bd7-9a8f-4221-ae12-72ff1ad8716b-kube-api-access-dqtls\") pod \"node-exporter-bkr4f\" (UID: \"12d81bd7-9a8f-4221-ae12-72ff1ad8716b\") " pod="openshift-monitoring/node-exporter-bkr4f" Jul 07 10:28:51.060090 ip-10-0-135-34 kubenswrapper[2578]: I0707 10:28:51.060034 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"root\" (UniqueName: \"kubernetes.io/host-path/12d81bd7-9a8f-4221-ae12-72ff1ad8716b-root\") pod \"node-exporter-bkr4f\" (UID: \"12d81bd7-9a8f-4221-ae12-72ff1ad8716b\") " pod="openshift-monitoring/node-exporter-bkr4f" Jul 07 10:28:51.060090 ip-10-0-135-34 kubenswrapper[2578]: I0707 10:28:51.060076 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/12d81bd7-9a8f-4221-ae12-72ff1ad8716b-node-exporter-wtmp\") pod \"node-exporter-bkr4f\" (UID: \"12d81bd7-9a8f-4221-ae12-72ff1ad8716b\") " pod="openshift-monitoring/node-exporter-bkr4f" Jul 07 10:28:51.060809 ip-10-0-135-34 kubenswrapper[2578]: I0707 10:28:51.060103 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/12d81bd7-9a8f-4221-ae12-72ff1ad8716b-node-exporter-accelerators-collector-config\") pod \"node-exporter-bkr4f\" (UID: \"12d81bd7-9a8f-4221-ae12-72ff1ad8716b\") " pod="openshift-monitoring/node-exporter-bkr4f" Jul 07 10:28:51.060809 ip-10-0-135-34 kubenswrapper[2578]: I0707 10:28:51.060133 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xprr2\" (UniqueName: \"kubernetes.io/projected/f41994cd-3ce0-43a3-bbd9-a853835da0bf-kube-api-access-xprr2\") pod \"openshift-state-metrics-54cf96dfcf-ffft9\" (UID: \"f41994cd-3ce0-43a3-bbd9-a853835da0bf\") " pod="openshift-monitoring/openshift-state-metrics-54cf96dfcf-ffft9" Jul 07 10:28:51.060809 ip-10-0-135-34 kubenswrapper[2578]: I0707 10:28:51.060247 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/f41994cd-3ce0-43a3-bbd9-a853835da0bf-openshift-state-metrics-tls\") pod \"openshift-state-metrics-54cf96dfcf-ffft9\" (UID: \"f41994cd-3ce0-43a3-bbd9-a853835da0bf\") " pod="openshift-monitoring/openshift-state-metrics-54cf96dfcf-ffft9" Jul 07 10:28:51.060809 ip-10-0-135-34 kubenswrapper[2578]: I0707 10:28:51.060279 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/12d81bd7-9a8f-4221-ae12-72ff1ad8716b-node-exporter-tls\") pod \"node-exporter-bkr4f\" (UID: \"12d81bd7-9a8f-4221-ae12-72ff1ad8716b\") " pod="openshift-monitoring/node-exporter-bkr4f" Jul 07 10:28:51.060809 ip-10-0-135-34 kubenswrapper[2578]: I0707 10:28:51.060265 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/12d81bd7-9a8f-4221-ae12-72ff1ad8716b-node-exporter-wtmp\") pod \"node-exporter-bkr4f\" (UID: \"12d81bd7-9a8f-4221-ae12-72ff1ad8716b\") " pod="openshift-monitoring/node-exporter-bkr4f" Jul 07 10:28:51.060809 ip-10-0-135-34 kubenswrapper[2578]: E0707 10:28:51.060382 2578 secret.go:189] Couldn't get secret openshift-monitoring/node-exporter-tls: secret "node-exporter-tls" not found Jul 07 10:28:51.060809 ip-10-0-135-34 kubenswrapper[2578]: E0707 10:28:51.060438 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/12d81bd7-9a8f-4221-ae12-72ff1ad8716b-node-exporter-tls podName:12d81bd7-9a8f-4221-ae12-72ff1ad8716b nodeName:}" failed. No retries permitted until 2026-07-07 10:28:51.560419766 +0000 UTC m=+100.232379302 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "node-exporter-tls" (UniqueName: "kubernetes.io/secret/12d81bd7-9a8f-4221-ae12-72ff1ad8716b-node-exporter-tls") pod "node-exporter-bkr4f" (UID: "12d81bd7-9a8f-4221-ae12-72ff1ad8716b") : secret "node-exporter-tls" not found Jul 07 10:28:51.060809 ip-10-0-135-34 kubenswrapper[2578]: I0707 10:28:51.060553 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/12d81bd7-9a8f-4221-ae12-72ff1ad8716b-sys\") pod \"node-exporter-bkr4f\" (UID: \"12d81bd7-9a8f-4221-ae12-72ff1ad8716b\") " pod="openshift-monitoring/node-exporter-bkr4f" Jul 07 10:28:51.060809 ip-10-0-135-34 kubenswrapper[2578]: I0707 10:28:51.060615 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/12d81bd7-9a8f-4221-ae12-72ff1ad8716b-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-bkr4f\" (UID: \"12d81bd7-9a8f-4221-ae12-72ff1ad8716b\") " pod="openshift-monitoring/node-exporter-bkr4f" Jul 07 10:28:51.060809 ip-10-0-135-34 kubenswrapper[2578]: I0707 10:28:51.060677 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/12d81bd7-9a8f-4221-ae12-72ff1ad8716b-sys\") pod \"node-exporter-bkr4f\" (UID: \"12d81bd7-9a8f-4221-ae12-72ff1ad8716b\") " pod="openshift-monitoring/node-exporter-bkr4f" Jul 07 10:28:51.060809 ip-10-0-135-34 kubenswrapper[2578]: I0707 10:28:51.060683 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/f41994cd-3ce0-43a3-bbd9-a853835da0bf-metrics-client-ca\") pod \"openshift-state-metrics-54cf96dfcf-ffft9\" (UID: \"f41994cd-3ce0-43a3-bbd9-a853835da0bf\") " pod="openshift-monitoring/openshift-state-metrics-54cf96dfcf-ffft9" Jul 07 10:28:51.060809 ip-10-0-135-34 kubenswrapper[2578]: I0707 10:28:51.060719 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/12d81bd7-9a8f-4221-ae12-72ff1ad8716b-node-exporter-textfile\") pod \"node-exporter-bkr4f\" (UID: \"12d81bd7-9a8f-4221-ae12-72ff1ad8716b\") " pod="openshift-monitoring/node-exporter-bkr4f" Jul 07 10:28:51.060809 ip-10-0-135-34 kubenswrapper[2578]: I0707 10:28:51.060753 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/12d81bd7-9a8f-4221-ae12-72ff1ad8716b-node-exporter-accelerators-collector-config\") pod \"node-exporter-bkr4f\" (UID: \"12d81bd7-9a8f-4221-ae12-72ff1ad8716b\") " pod="openshift-monitoring/node-exporter-bkr4f" Jul 07 10:28:51.061460 ip-10-0-135-34 kubenswrapper[2578]: I0707 10:28:51.060823 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/12d81bd7-9a8f-4221-ae12-72ff1ad8716b-metrics-client-ca\") pod \"node-exporter-bkr4f\" (UID: \"12d81bd7-9a8f-4221-ae12-72ff1ad8716b\") " pod="openshift-monitoring/node-exporter-bkr4f" Jul 07 10:28:51.061460 ip-10-0-135-34 kubenswrapper[2578]: I0707 10:28:51.061039 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/12d81bd7-9a8f-4221-ae12-72ff1ad8716b-node-exporter-textfile\") pod \"node-exporter-bkr4f\" (UID: \"12d81bd7-9a8f-4221-ae12-72ff1ad8716b\") " pod="openshift-monitoring/node-exporter-bkr4f" Jul 07 10:28:51.061460 ip-10-0-135-34 kubenswrapper[2578]: I0707 10:28:51.061364 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/f41994cd-3ce0-43a3-bbd9-a853835da0bf-metrics-client-ca\") pod \"openshift-state-metrics-54cf96dfcf-ffft9\" (UID: \"f41994cd-3ce0-43a3-bbd9-a853835da0bf\") " pod="openshift-monitoring/openshift-state-metrics-54cf96dfcf-ffft9" Jul 07 10:28:51.062970 ip-10-0-135-34 kubenswrapper[2578]: I0707 10:28:51.062941 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/f41994cd-3ce0-43a3-bbd9-a853835da0bf-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-54cf96dfcf-ffft9\" (UID: \"f41994cd-3ce0-43a3-bbd9-a853835da0bf\") " pod="openshift-monitoring/openshift-state-metrics-54cf96dfcf-ffft9" Jul 07 10:28:51.063069 ip-10-0-135-34 kubenswrapper[2578]: I0707 10:28:51.063051 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/12d81bd7-9a8f-4221-ae12-72ff1ad8716b-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-bkr4f\" (UID: \"12d81bd7-9a8f-4221-ae12-72ff1ad8716b\") " pod="openshift-monitoring/node-exporter-bkr4f" Jul 07 10:28:51.063300 ip-10-0-135-34 kubenswrapper[2578]: I0707 10:28:51.063278 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/f41994cd-3ce0-43a3-bbd9-a853835da0bf-openshift-state-metrics-tls\") pod \"openshift-state-metrics-54cf96dfcf-ffft9\" (UID: \"f41994cd-3ce0-43a3-bbd9-a853835da0bf\") " pod="openshift-monitoring/openshift-state-metrics-54cf96dfcf-ffft9" Jul 07 10:28:51.090529 ip-10-0-135-34 kubenswrapper[2578]: I0707 10:28:51.090492 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dqtls\" (UniqueName: \"kubernetes.io/projected/12d81bd7-9a8f-4221-ae12-72ff1ad8716b-kube-api-access-dqtls\") pod \"node-exporter-bkr4f\" (UID: \"12d81bd7-9a8f-4221-ae12-72ff1ad8716b\") " pod="openshift-monitoring/node-exporter-bkr4f" Jul 07 10:28:51.091269 ip-10-0-135-34 kubenswrapper[2578]: I0707 10:28:51.091250 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xprr2\" (UniqueName: \"kubernetes.io/projected/f41994cd-3ce0-43a3-bbd9-a853835da0bf-kube-api-access-xprr2\") pod \"openshift-state-metrics-54cf96dfcf-ffft9\" (UID: \"f41994cd-3ce0-43a3-bbd9-a853835da0bf\") " pod="openshift-monitoring/openshift-state-metrics-54cf96dfcf-ffft9" Jul 07 10:28:51.094062 ip-10-0-135-34 kubenswrapper[2578]: I0707 10:28:51.094044 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/openshift-state-metrics-54cf96dfcf-ffft9" Jul 07 10:28:51.233578 ip-10-0-135-34 kubenswrapper[2578]: I0707 10:28:51.233470 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-t79dx" event={"ID":"0a3e5a60-3150-4633-8ff1-4c8ebca70a41","Type":"ContainerStarted","Data":"abe84fabc316544ba2237c9aec89789b7d1d0aafd255550b0bc2c969e928d45b"} Jul 07 10:28:51.235002 ip-10-0-135-34 kubenswrapper[2578]: I0707 10:28:51.234978 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-fc54c99d9-nh7qj" event={"ID":"f8dcb050-7bdf-40c1-b47b-1019b93b1f9d","Type":"ContainerStarted","Data":"5191c185fcc084c87808b8edba7d91d52a55b99f59c636756d12199822faa9b1"} Jul 07 10:28:51.258713 ip-10-0-135-34 kubenswrapper[2578]: I0707 10:28:51.258679 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/openshift-state-metrics-54cf96dfcf-ffft9"] Jul 07 10:28:51.263665 ip-10-0-135-34 kubenswrapper[2578]: W0707 10:28:51.263633 2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf41994cd_3ce0_43a3_bbd9_a853835da0bf.slice/crio-6be45a6c089bcfb62f6dc390e911d1591a1eab89a931f497fcd297a5edb0d410 WatchSource:0}: Error finding container 6be45a6c089bcfb62f6dc390e911d1591a1eab89a931f497fcd297a5edb0d410: Status 404 returned error can't find the container with id 6be45a6c089bcfb62f6dc390e911d1591a1eab89a931f497fcd297a5edb0d410 Jul 07 10:28:51.276370 ip-10-0-135-34 kubenswrapper[2578]: I0707 10:28:51.276311 2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-insights/insights-runtime-extractor-t79dx" podStartSLOduration=33.227153225 podStartE2EDuration="41.276288944s" totalImagesPullingTime="8.049135719s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-07 10:28:10 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 10:28:51.275116676 +0000 UTC m=+99.947076221" watchObservedRunningTime="2026-07-07 10:28:51.276288944 +0000 UTC m=+99.948248489" Jul 07 10:28:51.304931 ip-10-0-135-34 kubenswrapper[2578]: I0707 10:28:51.304885 2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-fc54c99d9-nh7qj" podStartSLOduration=3.944802571 podStartE2EDuration="8.304867455s" totalImagesPullingTime="4.360064884s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-07 10:28:43 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 10:28:51.30451096 +0000 UTC m=+99.976470522" watchObservedRunningTime="2026-07-07 10:28:51.304867455 +0000 UTC m=+99.976827046" Jul 07 10:28:51.565247 ip-10-0-135-34 kubenswrapper[2578]: I0707 10:28:51.565156 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/12d81bd7-9a8f-4221-ae12-72ff1ad8716b-node-exporter-tls\") pod \"node-exporter-bkr4f\" (UID: \"12d81bd7-9a8f-4221-ae12-72ff1ad8716b\") " pod="openshift-monitoring/node-exporter-bkr4f" Jul 07 10:28:51.567903 ip-10-0-135-34 kubenswrapper[2578]: I0707 10:28:51.567874 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/12d81bd7-9a8f-4221-ae12-72ff1ad8716b-node-exporter-tls\") pod \"node-exporter-bkr4f\" (UID: \"12d81bd7-9a8f-4221-ae12-72ff1ad8716b\") " pod="openshift-monitoring/node-exporter-bkr4f" Jul 07 10:28:51.760225 ip-10-0-135-34 kubenswrapper[2578]: I0707 10:28:51.760188 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-bkr4f" Jul 07 10:28:51.773464 ip-10-0-135-34 kubenswrapper[2578]: W0707 10:28:51.773427 2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod12d81bd7_9a8f_4221_ae12_72ff1ad8716b.slice/crio-493054c94991c7259327f01348f8a69792a1e1aba0a210b4208564c0269b28cb WatchSource:0}: Error finding container 493054c94991c7259327f01348f8a69792a1e1aba0a210b4208564c0269b28cb: Status 404 returned error can't find the container with id 493054c94991c7259327f01348f8a69792a1e1aba0a210b4208564c0269b28cb Jul 07 10:28:52.126661 ip-10-0-135-34 kubenswrapper[2578]: I0707 10:28:52.126628 2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Jul 07 10:28:52.138735 ip-10-0-135-34 kubenswrapper[2578]: I0707 10:28:52.138704 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0" Jul 07 10:28:52.172823 ip-10-0-135-34 kubenswrapper[2578]: I0707 10:28:52.170024 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-metric\"" Jul 07 10:28:52.172823 ip-10-0-135-34 kubenswrapper[2578]: I0707 10:28:52.170241 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/a1048018-11a0-4d5a-96ab-86508dac8366-metrics-tls\") pod \"dns-default-j2ks5\" (UID: \"a1048018-11a0-4d5a-96ab-86508dac8366\") " pod="openshift-dns/dns-default-j2ks5" Jul 07 10:28:52.172823 ip-10-0-135-34 kubenswrapper[2578]: I0707 10:28:52.170974 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-web\"" Jul 07 10:28:52.177375 ip-10-0-135-34 kubenswrapper[2578]: I0707 10:28:52.176061 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-generated\"" Jul 07 10:28:52.177375 ip-10-0-135-34 kubenswrapper[2578]: I0707 10:28:52.176163 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/a1048018-11a0-4d5a-96ab-86508dac8366-metrics-tls\") pod \"dns-default-j2ks5\" (UID: \"a1048018-11a0-4d5a-96ab-86508dac8366\") " pod="openshift-dns/dns-default-j2ks5" Jul 07 10:28:52.177375 ip-10-0-135-34 kubenswrapper[2578]: I0707 10:28:52.176320 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-web-config\"" Jul 07 10:28:52.177375 ip-10-0-135-34 kubenswrapper[2578]: I0707 10:28:52.176356 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy\"" Jul 07 10:28:52.177375 ip-10-0-135-34 kubenswrapper[2578]: I0707 10:28:52.176582 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls-assets-0\"" Jul 07 10:28:52.177375 ip-10-0-135-34 kubenswrapper[2578]: I0707 10:28:52.176604 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls\"" Jul 07 10:28:52.177375 ip-10-0-135-34 kubenswrapper[2578]: I0707 10:28:52.176807 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-cluster-tls-config\"" Jul 07 10:28:52.177375 ip-10-0-135-34 kubenswrapper[2578]: I0707 10:28:52.176988 2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"alertmanager-trusted-ca-bundle\"" Jul 07 10:28:52.177375 ip-10-0-135-34 kubenswrapper[2578]: I0707 10:28:52.177001 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-dockercfg-vb6pv\"" Jul 07 10:28:52.201792 ip-10-0-135-34 kubenswrapper[2578]: I0707 10:28:52.201749 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Jul 07 10:28:52.240413 ip-10-0-135-34 kubenswrapper[2578]: I0707 10:28:52.240323 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-bkr4f" event={"ID":"12d81bd7-9a8f-4221-ae12-72ff1ad8716b","Type":"ContainerStarted","Data":"493054c94991c7259327f01348f8a69792a1e1aba0a210b4208564c0269b28cb"} Jul 07 10:28:52.242841 ip-10-0-135-34 kubenswrapper[2578]: I0707 10:28:52.242678 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-54cf96dfcf-ffft9" event={"ID":"f41994cd-3ce0-43a3-bbd9-a853835da0bf","Type":"ContainerStarted","Data":"755d49ee403b1310c7c24491d0b625e1f597f75de09edeced22aa0f26d1f1a81"} Jul 07 10:28:52.242841 ip-10-0-135-34 kubenswrapper[2578]: I0707 10:28:52.242741 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-54cf96dfcf-ffft9" event={"ID":"f41994cd-3ce0-43a3-bbd9-a853835da0bf","Type":"ContainerStarted","Data":"b975f86653e8d48961e6a8d1b9971366db6b1c9b153cc396c0bf9d5480941a53"} Jul 07 10:28:52.242841 ip-10-0-135-34 kubenswrapper[2578]: I0707 10:28:52.242802 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-54cf96dfcf-ffft9" event={"ID":"f41994cd-3ce0-43a3-bbd9-a853835da0bf","Type":"ContainerStarted","Data":"6be45a6c089bcfb62f6dc390e911d1591a1eab89a931f497fcd297a5edb0d410"} Jul 07 10:28:52.271663 ip-10-0-135-34 kubenswrapper[2578]: I0707 10:28:52.271628 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/229bf452-da72-420d-a150-9a9047364092-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"229bf452-da72-420d-a150-9a9047364092\") " pod="openshift-monitoring/alertmanager-main-0" Jul 07 10:28:52.271840 ip-10-0-135-34 kubenswrapper[2578]: I0707 10:28:52.271673 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/229bf452-da72-420d-a150-9a9047364092-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"229bf452-da72-420d-a150-9a9047364092\") " pod="openshift-monitoring/alertmanager-main-0" Jul 07 10:28:52.271840 ip-10-0-135-34 kubenswrapper[2578]: I0707 10:28:52.271702 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/229bf452-da72-420d-a150-9a9047364092-web-config\") pod \"alertmanager-main-0\" (UID: \"229bf452-da72-420d-a150-9a9047364092\") " pod="openshift-monitoring/alertmanager-main-0" Jul 07 10:28:52.271840 ip-10-0-135-34 kubenswrapper[2578]: I0707 10:28:52.271720 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/229bf452-da72-420d-a150-9a9047364092-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"229bf452-da72-420d-a150-9a9047364092\") " pod="openshift-monitoring/alertmanager-main-0" Jul 07 10:28:52.271840 ip-10-0-135-34 kubenswrapper[2578]: I0707 10:28:52.271786 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/229bf452-da72-420d-a150-9a9047364092-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"229bf452-da72-420d-a150-9a9047364092\") " pod="openshift-monitoring/alertmanager-main-0" Jul 07 10:28:52.271840 ip-10-0-135-34 kubenswrapper[2578]: I0707 10:28:52.271830 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/229bf452-da72-420d-a150-9a9047364092-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"229bf452-da72-420d-a150-9a9047364092\") " pod="openshift-monitoring/alertmanager-main-0" Jul 07 10:28:52.272079 ip-10-0-135-34 kubenswrapper[2578]: I0707 10:28:52.271878 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/229bf452-da72-420d-a150-9a9047364092-tls-assets\") pod \"alertmanager-main-0\" (UID: \"229bf452-da72-420d-a150-9a9047364092\") " pod="openshift-monitoring/alertmanager-main-0" Jul 07 10:28:52.272079 ip-10-0-135-34 kubenswrapper[2578]: I0707 10:28:52.271940 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/229bf452-da72-420d-a150-9a9047364092-config-volume\") pod \"alertmanager-main-0\" (UID: \"229bf452-da72-420d-a150-9a9047364092\") " pod="openshift-monitoring/alertmanager-main-0" Jul 07 10:28:52.272079 ip-10-0-135-34 kubenswrapper[2578]: I0707 10:28:52.271971 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/229bf452-da72-420d-a150-9a9047364092-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"229bf452-da72-420d-a150-9a9047364092\") " pod="openshift-monitoring/alertmanager-main-0" Jul 07 10:28:52.272079 ip-10-0-135-34 kubenswrapper[2578]: I0707 10:28:52.272005 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/229bf452-da72-420d-a150-9a9047364092-config-out\") pod \"alertmanager-main-0\" (UID: \"229bf452-da72-420d-a150-9a9047364092\") " pod="openshift-monitoring/alertmanager-main-0" Jul 07 10:28:52.272079 ip-10-0-135-34 kubenswrapper[2578]: I0707 10:28:52.272030 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/229bf452-da72-420d-a150-9a9047364092-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"229bf452-da72-420d-a150-9a9047364092\") " pod="openshift-monitoring/alertmanager-main-0" Jul 07 10:28:52.272079 ip-10-0-135-34 kubenswrapper[2578]: I0707 10:28:52.272061 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/229bf452-da72-420d-a150-9a9047364092-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"229bf452-da72-420d-a150-9a9047364092\") " pod="openshift-monitoring/alertmanager-main-0" Jul 07 10:28:52.272079 ip-10-0-135-34 kubenswrapper[2578]: I0707 10:28:52.272078 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-v7pxm\" (UniqueName: \"kubernetes.io/projected/229bf452-da72-420d-a150-9a9047364092-kube-api-access-v7pxm\") pod \"alertmanager-main-0\" (UID: \"229bf452-da72-420d-a150-9a9047364092\") " pod="openshift-monitoring/alertmanager-main-0" Jul 07 10:28:52.373143 ip-10-0-135-34 kubenswrapper[2578]: I0707 10:28:52.373086 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/229bf452-da72-420d-a150-9a9047364092-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"229bf452-da72-420d-a150-9a9047364092\") " pod="openshift-monitoring/alertmanager-main-0" Jul 07 10:28:52.373143 ip-10-0-135-34 kubenswrapper[2578]: I0707 10:28:52.373143 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/229bf452-da72-420d-a150-9a9047364092-web-config\") pod \"alertmanager-main-0\" (UID: \"229bf452-da72-420d-a150-9a9047364092\") " pod="openshift-monitoring/alertmanager-main-0" Jul 07 10:28:52.373419 ip-10-0-135-34 kubenswrapper[2578]: I0707 10:28:52.373385 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/229bf452-da72-420d-a150-9a9047364092-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"229bf452-da72-420d-a150-9a9047364092\") " pod="openshift-monitoring/alertmanager-main-0" Jul 07 10:28:52.373503 ip-10-0-135-34 kubenswrapper[2578]: I0707 10:28:52.373443 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/229bf452-da72-420d-a150-9a9047364092-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"229bf452-da72-420d-a150-9a9047364092\") " pod="openshift-monitoring/alertmanager-main-0" Jul 07 10:28:52.373503 ip-10-0-135-34 kubenswrapper[2578]: I0707 10:28:52.373477 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/229bf452-da72-420d-a150-9a9047364092-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"229bf452-da72-420d-a150-9a9047364092\") " pod="openshift-monitoring/alertmanager-main-0" Jul 07 10:28:52.373610 ip-10-0-135-34 kubenswrapper[2578]: I0707 10:28:52.373585 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/229bf452-da72-420d-a150-9a9047364092-tls-assets\") pod \"alertmanager-main-0\" (UID: \"229bf452-da72-420d-a150-9a9047364092\") " pod="openshift-monitoring/alertmanager-main-0" Jul 07 10:28:52.373746 ip-10-0-135-34 kubenswrapper[2578]: I0707 10:28:52.373724 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/229bf452-da72-420d-a150-9a9047364092-config-volume\") pod \"alertmanager-main-0\" (UID: \"229bf452-da72-420d-a150-9a9047364092\") " pod="openshift-monitoring/alertmanager-main-0" Jul 07 10:28:52.373843 ip-10-0-135-34 kubenswrapper[2578]: I0707 10:28:52.373764 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/229bf452-da72-420d-a150-9a9047364092-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"229bf452-da72-420d-a150-9a9047364092\") " pod="openshift-monitoring/alertmanager-main-0" Jul 07 10:28:52.373843 ip-10-0-135-34 kubenswrapper[2578]: I0707 10:28:52.373824 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/229bf452-da72-420d-a150-9a9047364092-config-out\") pod \"alertmanager-main-0\" (UID: \"229bf452-da72-420d-a150-9a9047364092\") " pod="openshift-monitoring/alertmanager-main-0" Jul 07 10:28:52.373959 ip-10-0-135-34 kubenswrapper[2578]: I0707 10:28:52.373864 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/229bf452-da72-420d-a150-9a9047364092-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"229bf452-da72-420d-a150-9a9047364092\") " pod="openshift-monitoring/alertmanager-main-0" Jul 07 10:28:52.373959 ip-10-0-135-34 kubenswrapper[2578]: I0707 10:28:52.373896 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/229bf452-da72-420d-a150-9a9047364092-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"229bf452-da72-420d-a150-9a9047364092\") " pod="openshift-monitoring/alertmanager-main-0" Jul 07 10:28:52.374456 ip-10-0-135-34 kubenswrapper[2578]: I0707 10:28:52.374427 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v7pxm\" (UniqueName: \"kubernetes.io/projected/229bf452-da72-420d-a150-9a9047364092-kube-api-access-v7pxm\") pod \"alertmanager-main-0\" (UID: \"229bf452-da72-420d-a150-9a9047364092\") " pod="openshift-monitoring/alertmanager-main-0" Jul 07 10:28:52.374548 ip-10-0-135-34 kubenswrapper[2578]: I0707 10:28:52.374513 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/229bf452-da72-420d-a150-9a9047364092-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"229bf452-da72-420d-a150-9a9047364092\") " pod="openshift-monitoring/alertmanager-main-0" Jul 07 10:28:52.374548 ip-10-0-135-34 kubenswrapper[2578]: I0707 10:28:52.374534 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/229bf452-da72-420d-a150-9a9047364092-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"229bf452-da72-420d-a150-9a9047364092\") " pod="openshift-monitoring/alertmanager-main-0" Jul 07 10:28:52.375829 ip-10-0-135-34 kubenswrapper[2578]: I0707 10:28:52.375108 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/229bf452-da72-420d-a150-9a9047364092-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"229bf452-da72-420d-a150-9a9047364092\") " pod="openshift-monitoring/alertmanager-main-0" Jul 07 10:28:52.375829 ip-10-0-135-34 kubenswrapper[2578]: I0707 10:28:52.375226 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/229bf452-da72-420d-a150-9a9047364092-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"229bf452-da72-420d-a150-9a9047364092\") " pod="openshift-monitoring/alertmanager-main-0" Jul 07 10:28:52.377478 ip-10-0-135-34 kubenswrapper[2578]: I0707 10:28:52.377414 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/229bf452-da72-420d-a150-9a9047364092-config-volume\") pod \"alertmanager-main-0\" (UID: \"229bf452-da72-420d-a150-9a9047364092\") " pod="openshift-monitoring/alertmanager-main-0" Jul 07 10:28:52.378280 ip-10-0-135-34 kubenswrapper[2578]: I0707 10:28:52.378106 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/229bf452-da72-420d-a150-9a9047364092-web-config\") pod \"alertmanager-main-0\" (UID: \"229bf452-da72-420d-a150-9a9047364092\") " pod="openshift-monitoring/alertmanager-main-0" Jul 07 10:28:52.379382 ip-10-0-135-34 kubenswrapper[2578]: I0707 10:28:52.379320 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/229bf452-da72-420d-a150-9a9047364092-config-out\") pod \"alertmanager-main-0\" (UID: \"229bf452-da72-420d-a150-9a9047364092\") " pod="openshift-monitoring/alertmanager-main-0" Jul 07 10:28:52.380807 ip-10-0-135-34 kubenswrapper[2578]: I0707 10:28:52.379478 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/229bf452-da72-420d-a150-9a9047364092-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"229bf452-da72-420d-a150-9a9047364092\") " pod="openshift-monitoring/alertmanager-main-0" Jul 07 10:28:52.380807 ip-10-0-135-34 kubenswrapper[2578]: I0707 10:28:52.379660 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/229bf452-da72-420d-a150-9a9047364092-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"229bf452-da72-420d-a150-9a9047364092\") " pod="openshift-monitoring/alertmanager-main-0" Jul 07 10:28:52.380807 ip-10-0-135-34 kubenswrapper[2578]: I0707 10:28:52.380450 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/229bf452-da72-420d-a150-9a9047364092-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"229bf452-da72-420d-a150-9a9047364092\") " pod="openshift-monitoring/alertmanager-main-0" Jul 07 10:28:52.381381 ip-10-0-135-34 kubenswrapper[2578]: I0707 10:28:52.381160 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/229bf452-da72-420d-a150-9a9047364092-tls-assets\") pod \"alertmanager-main-0\" (UID: \"229bf452-da72-420d-a150-9a9047364092\") " pod="openshift-monitoring/alertmanager-main-0" Jul 07 10:28:52.381969 ip-10-0-135-34 kubenswrapper[2578]: I0707 10:28:52.381949 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/229bf452-da72-420d-a150-9a9047364092-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"229bf452-da72-420d-a150-9a9047364092\") " pod="openshift-monitoring/alertmanager-main-0" Jul 07 10:28:52.382632 ip-10-0-135-34 kubenswrapper[2578]: I0707 10:28:52.382612 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/229bf452-da72-420d-a150-9a9047364092-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"229bf452-da72-420d-a150-9a9047364092\") " pod="openshift-monitoring/alertmanager-main-0" Jul 07 10:28:52.392895 ip-10-0-135-34 kubenswrapper[2578]: I0707 10:28:52.392864 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-v7pxm\" (UniqueName: \"kubernetes.io/projected/229bf452-da72-420d-a150-9a9047364092-kube-api-access-v7pxm\") pod \"alertmanager-main-0\" (UID: \"229bf452-da72-420d-a150-9a9047364092\") " pod="openshift-monitoring/alertmanager-main-0" Jul 07 10:28:52.450900 ip-10-0-135-34 kubenswrapper[2578]: I0707 10:28:52.450868 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0" Jul 07 10:28:52.451098 ip-10-0-135-34 kubenswrapper[2578]: I0707 10:28:52.451002 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-wq5jm\"" Jul 07 10:28:52.456343 ip-10-0-135-34 kubenswrapper[2578]: I0707 10:28:52.456310 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-j2ks5" Jul 07 10:28:52.689238 ip-10-0-135-34 kubenswrapper[2578]: I0707 10:28:52.689108 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-j2ks5"] Jul 07 10:28:52.727193 ip-10-0-135-34 kubenswrapper[2578]: I0707 10:28:52.727161 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Jul 07 10:28:52.864392 ip-10-0-135-34 kubenswrapper[2578]: W0707 10:28:52.864346 2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda1048018_11a0_4d5a_96ab_86508dac8366.slice/crio-710b405b91afa5949708cc7cb69d94839d41f9e0a95a31637cb90798c3b81837 WatchSource:0}: Error finding container 710b405b91afa5949708cc7cb69d94839d41f9e0a95a31637cb90798c3b81837: Status 404 returned error can't find the container with id 710b405b91afa5949708cc7cb69d94839d41f9e0a95a31637cb90798c3b81837 Jul 07 10:28:52.865261 ip-10-0-135-34 kubenswrapper[2578]: W0707 10:28:52.865110 2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod229bf452_da72_420d_a150_9a9047364092.slice/crio-bbbea4088da42a2ec0a723142a76d94663f1f2794d543c0c8ed1b5ec901c87e8 WatchSource:0}: Error finding container bbbea4088da42a2ec0a723142a76d94663f1f2794d543c0c8ed1b5ec901c87e8: Status 404 returned error can't find the container with id bbbea4088da42a2ec0a723142a76d94663f1f2794d543c0c8ed1b5ec901c87e8 Jul 07 10:28:53.248446 ip-10-0-135-34 kubenswrapper[2578]: I0707 10:28:53.248340 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-54cf96dfcf-ffft9" event={"ID":"f41994cd-3ce0-43a3-bbd9-a853835da0bf","Type":"ContainerStarted","Data":"8afff425b21eae410f65bc907d0e54485afebabf7af79e5325c8e89a0221d7fe"} Jul 07 10:28:53.249667 ip-10-0-135-34 kubenswrapper[2578]: I0707 10:28:53.249640 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-j2ks5" event={"ID":"a1048018-11a0-4d5a-96ab-86508dac8366","Type":"ContainerStarted","Data":"710b405b91afa5949708cc7cb69d94839d41f9e0a95a31637cb90798c3b81837"} Jul 07 10:28:53.250840 ip-10-0-135-34 kubenswrapper[2578]: I0707 10:28:53.250815 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"229bf452-da72-420d-a150-9a9047364092","Type":"ContainerStarted","Data":"bbbea4088da42a2ec0a723142a76d94663f1f2794d543c0c8ed1b5ec901c87e8"} Jul 07 10:28:53.253003 ip-10-0-135-34 kubenswrapper[2578]: I0707 10:28:53.252978 2578 generic.go:356] "Generic (PLEG): container finished" podID="12d81bd7-9a8f-4221-ae12-72ff1ad8716b" containerID="0aa4a9ed7e62fb5b5d6104690e77516b64d4524f40f89fb07f2130dff17228c4" exitCode=0 Jul 07 10:28:53.253078 ip-10-0-135-34 kubenswrapper[2578]: I0707 10:28:53.253012 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-bkr4f" event={"ID":"12d81bd7-9a8f-4221-ae12-72ff1ad8716b","Type":"ContainerDied","Data":"0aa4a9ed7e62fb5b5d6104690e77516b64d4524f40f89fb07f2130dff17228c4"} Jul 07 10:28:53.290899 ip-10-0-135-34 kubenswrapper[2578]: I0707 10:28:53.290849 2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/openshift-state-metrics-54cf96dfcf-ffft9" podStartSLOduration=2.147229804 podStartE2EDuration="3.29083247s" totalImagesPullingTime="1.143602666s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-07 10:28:50 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 10:28:53.289683379 +0000 UTC m=+101.961642920" watchObservedRunningTime="2026-07-07 10:28:53.29083247 +0000 UTC m=+101.962792014" Jul 07 10:28:54.210979 ip-10-0-135-34 kubenswrapper[2578]: I0707 10:28:54.210937 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-fc54c99d9-nh7qj" Jul 07 10:28:54.211158 ip-10-0-135-34 kubenswrapper[2578]: I0707 10:28:54.211009 2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-fc54c99d9-nh7qj" Jul 07 10:28:54.216702 ip-10-0-135-34 kubenswrapper[2578]: I0707 10:28:54.216671 2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-fc54c99d9-nh7qj" Jul 07 10:28:54.264406 ip-10-0-135-34 kubenswrapper[2578]: I0707 10:28:54.264348 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-bkr4f" event={"ID":"12d81bd7-9a8f-4221-ae12-72ff1ad8716b","Type":"ContainerStarted","Data":"455a51968f1d12b3ce52b22b4bbe109532d8c96f5ecc32d420d1a53f69a6e668"} Jul 07 10:28:54.264844 ip-10-0-135-34 kubenswrapper[2578]: I0707 10:28:54.264443 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-bkr4f" event={"ID":"12d81bd7-9a8f-4221-ae12-72ff1ad8716b","Type":"ContainerStarted","Data":"6fe5613206f827230e4ee8630d0a5258d77022392cd539f6e8480e2851bcbb8f"} Jul 07 10:28:54.270210 ip-10-0-135-34 kubenswrapper[2578]: I0707 10:28:54.270181 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-fc54c99d9-nh7qj" Jul 07 10:28:54.384674 ip-10-0-135-34 kubenswrapper[2578]: I0707 10:28:54.384604 2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/node-exporter-bkr4f" podStartSLOduration=3.177243635 podStartE2EDuration="4.384580824s" totalImagesPullingTime="1.134378555s" totalInitContainerRuntime="72.958634ms" isStatefulPod=true podCreationTimestamp="2026-07-07 10:28:50 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 10:28:54.311166951 +0000 UTC m=+102.983126515" watchObservedRunningTime="2026-07-07 10:28:54.384580824 +0000 UTC m=+103.056540368" Jul 07 10:28:56.274427 ip-10-0-135-34 kubenswrapper[2578]: I0707 10:28:56.274390 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-j2ks5" event={"ID":"a1048018-11a0-4d5a-96ab-86508dac8366","Type":"ContainerStarted","Data":"d248b1a4f5f9e1173ed6b1fc1a62f1bc3fa29c8234517fb81c63b0009f2680f6"} Jul 07 10:28:56.274855 ip-10-0-135-34 kubenswrapper[2578]: I0707 10:28:56.274547 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-dns/dns-default-j2ks5" Jul 07 10:28:56.274855 ip-10-0-135-34 kubenswrapper[2578]: I0707 10:28:56.274596 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-j2ks5" event={"ID":"a1048018-11a0-4d5a-96ab-86508dac8366","Type":"ContainerStarted","Data":"99a41a20c2c32c327534b7d38aace33ca6496d7ed91b8023d8abbab7f7798ead"} Jul 07 10:28:56.276120 ip-10-0-135-34 kubenswrapper[2578]: I0707 10:28:56.276098 2578 generic.go:356] "Generic (PLEG): container finished" podID="229bf452-da72-420d-a150-9a9047364092" containerID="03c99232f21674d142de0d4e2dbf5b4db736507cb33e48af080e97d8374d5a4a" exitCode=0 Jul 07 10:28:56.276204 ip-10-0-135-34 kubenswrapper[2578]: I0707 10:28:56.276184 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"229bf452-da72-420d-a150-9a9047364092","Type":"ContainerDied","Data":"03c99232f21674d142de0d4e2dbf5b4db736507cb33e48af080e97d8374d5a4a"} Jul 07 10:28:56.306658 ip-10-0-135-34 kubenswrapper[2578]: I0707 10:28:56.306608 2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/dns-default-j2ks5" podStartSLOduration=65.894372581 podStartE2EDuration="1m8.306591562s" totalImagesPullingTime="2.412218981s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-07 10:27:48 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 10:28:56.306068965 +0000 UTC m=+104.978028509" watchObservedRunningTime="2026-07-07 10:28:56.306591562 +0000 UTC m=+104.978551482" Jul 07 10:28:56.537196 ip-10-0-135-34 kubenswrapper[2578]: I0707 10:28:56.537089 2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-cb7847d49-4nszw"] Jul 07 10:28:58.289146 ip-10-0-135-34 kubenswrapper[2578]: I0707 10:28:58.289112 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"229bf452-da72-420d-a150-9a9047364092","Type":"ContainerStarted","Data":"acf54c7d774968536e7d8f960577a780c8fdd5d703c6e4b2a20c099f48361853"} Jul 07 10:28:59.297185 ip-10-0-135-34 kubenswrapper[2578]: I0707 10:28:59.297149 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"229bf452-da72-420d-a150-9a9047364092","Type":"ContainerStarted","Data":"61bf8d8145f073651698e6fe61d6b71c2610f1cea2a1bafc05b63747358cbd64"} Jul 07 10:28:59.297647 ip-10-0-135-34 kubenswrapper[2578]: I0707 10:28:59.297213 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"229bf452-da72-420d-a150-9a9047364092","Type":"ContainerStarted","Data":"dcd01319e5000de9e2a766332f63932db5c37b469368f48696dae3efdc0b5705"} Jul 07 10:28:59.297647 ip-10-0-135-34 kubenswrapper[2578]: I0707 10:28:59.297266 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"229bf452-da72-420d-a150-9a9047364092","Type":"ContainerStarted","Data":"9cb5f333071c79a41f014f36b65bc8321c179495a470064c61c72bddeada4b5f"} Jul 07 10:28:59.297647 ip-10-0-135-34 kubenswrapper[2578]: I0707 10:28:59.297291 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"229bf452-da72-420d-a150-9a9047364092","Type":"ContainerStarted","Data":"6533c9269624e19743317012be74f60871604e781029330cce2a5b0837ec76d5"} Jul 07 10:28:59.297647 ip-10-0-135-34 kubenswrapper[2578]: I0707 10:28:59.297313 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"229bf452-da72-420d-a150-9a9047364092","Type":"ContainerStarted","Data":"fa5103f6ae6a1fe8dcef27a7f7d2c9f46a40d470fee33c13bf42f81fc10b6aaf"} Jul 07 10:28:59.336203 ip-10-0-135-34 kubenswrapper[2578]: I0707 10:28:59.336146 2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/alertmanager-main-0" podStartSLOduration=2.323973456 podStartE2EDuration="7.336126739s" totalImagesPullingTime="4.994361451s" totalInitContainerRuntime="17.791832ms" isStatefulPod=false podCreationTimestamp="2026-07-07 10:28:52 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 10:28:59.334931707 +0000 UTC m=+108.006891265" watchObservedRunningTime="2026-07-07 10:28:59.336126739 +0000 UTC m=+108.008086280" Jul 07 10:29:06.285101 ip-10-0-135-34 kubenswrapper[2578]: I0707 10:29:06.285068 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-dns/dns-default-j2ks5" Jul 07 10:29:07.411312 ip-10-0-135-34 kubenswrapper[2578]: I0707 10:29:07.411277 2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-fc54c99d9-nh7qj"] Jul 07 10:29:21.557498 ip-10-0-135-34 kubenswrapper[2578]: I0707 10:29:21.557447 2578 kuberuntime_container.go:872] "Killing container with a grace period" pod="openshift-image-registry/image-registry-cb7847d49-4nszw" podUID="c4552170-c8e7-43e3-9472-a066d5379772" containerName="registry" containerID="cri-o://b055f09fd0b50ea99f0dfed3b12956f71b9216d7b6ab504375e0d391b44325a1" gracePeriod=30 Jul 07 10:29:21.629660 ip-10-0-135-34 kubenswrapper[2578]: I0707 10:29:21.629617 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/931a7d75-9ee8-4758-9f23-3521f3db5eae-metrics-certs\") pod \"network-metrics-daemon-p2ks2\" (UID: \"931a7d75-9ee8-4758-9f23-3521f3db5eae\") " pod="openshift-multus/network-metrics-daemon-p2ks2" Jul 07 10:29:21.632217 ip-10-0-135-34 kubenswrapper[2578]: I0707 10:29:21.632188 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/931a7d75-9ee8-4758-9f23-3521f3db5eae-metrics-certs\") pod \"network-metrics-daemon-p2ks2\" (UID: \"931a7d75-9ee8-4758-9f23-3521f3db5eae\") " pod="openshift-multus/network-metrics-daemon-p2ks2" Jul 07 10:29:21.828750 ip-10-0-135-34 kubenswrapper[2578]: I0707 10:29:21.828672 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-68x57\"" Jul 07 10:29:21.835572 ip-10-0-135-34 kubenswrapper[2578]: I0707 10:29:21.835551 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-p2ks2" Jul 07 10:29:21.998492 ip-10-0-135-34 kubenswrapper[2578]: I0707 10:29:21.998461 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-p2ks2"] Jul 07 10:29:22.001220 ip-10-0-135-34 kubenswrapper[2578]: W0707 10:29:22.001132 2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod931a7d75_9ee8_4758_9f23_3521f3db5eae.slice/crio-f4252df45606bc7711aa6d4feeb3c0442f1b1035e70ce068db05e71309dd397b WatchSource:0}: Error finding container f4252df45606bc7711aa6d4feeb3c0442f1b1035e70ce068db05e71309dd397b: Status 404 returned error can't find the container with id f4252df45606bc7711aa6d4feeb3c0442f1b1035e70ce068db05e71309dd397b Jul 07 10:29:22.302468 ip-10-0-135-34 kubenswrapper[2578]: I0707 10:29:22.302434 2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-cb7847d49-4nszw" Jul 07 10:29:22.389979 ip-10-0-135-34 kubenswrapper[2578]: I0707 10:29:22.389939 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-p2ks2" event={"ID":"931a7d75-9ee8-4758-9f23-3521f3db5eae","Type":"ContainerStarted","Data":"f4252df45606bc7711aa6d4feeb3c0442f1b1035e70ce068db05e71309dd397b"} Jul 07 10:29:22.391077 ip-10-0-135-34 kubenswrapper[2578]: I0707 10:29:22.391043 2578 generic.go:356] "Generic (PLEG): container finished" podID="c4552170-c8e7-43e3-9472-a066d5379772" containerID="b055f09fd0b50ea99f0dfed3b12956f71b9216d7b6ab504375e0d391b44325a1" exitCode=0 Jul 07 10:29:22.391077 ip-10-0-135-34 kubenswrapper[2578]: I0707 10:29:22.391097 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-cb7847d49-4nszw" event={"ID":"c4552170-c8e7-43e3-9472-a066d5379772","Type":"ContainerDied","Data":"b055f09fd0b50ea99f0dfed3b12956f71b9216d7b6ab504375e0d391b44325a1"} Jul 07 10:29:22.391077 ip-10-0-135-34 kubenswrapper[2578]: I0707 10:29:22.391105 2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-cb7847d49-4nszw" Jul 07 10:29:22.391312 ip-10-0-135-34 kubenswrapper[2578]: I0707 10:29:22.391139 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-cb7847d49-4nszw" event={"ID":"c4552170-c8e7-43e3-9472-a066d5379772","Type":"ContainerDied","Data":"95d24df0a2dce2b51eae9a8077395857eb821f52e2274003b9737aff8caf2224"} Jul 07 10:29:22.391312 ip-10-0-135-34 kubenswrapper[2578]: I0707 10:29:22.391164 2578 scope.go:117] "RemoveContainer" containerID="b055f09fd0b50ea99f0dfed3b12956f71b9216d7b6ab504375e0d391b44325a1" Jul 07 10:29:22.399096 ip-10-0-135-34 kubenswrapper[2578]: I0707 10:29:22.399080 2578 scope.go:117] "RemoveContainer" containerID="b055f09fd0b50ea99f0dfed3b12956f71b9216d7b6ab504375e0d391b44325a1" Jul 07 10:29:22.399363 ip-10-0-135-34 kubenswrapper[2578]: E0707 10:29:22.399344 2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b055f09fd0b50ea99f0dfed3b12956f71b9216d7b6ab504375e0d391b44325a1\": container with ID starting with b055f09fd0b50ea99f0dfed3b12956f71b9216d7b6ab504375e0d391b44325a1 not found: ID does not exist" containerID="b055f09fd0b50ea99f0dfed3b12956f71b9216d7b6ab504375e0d391b44325a1" Jul 07 10:29:22.399405 ip-10-0-135-34 kubenswrapper[2578]: I0707 10:29:22.399372 2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b055f09fd0b50ea99f0dfed3b12956f71b9216d7b6ab504375e0d391b44325a1"} err="failed to get container status \"b055f09fd0b50ea99f0dfed3b12956f71b9216d7b6ab504375e0d391b44325a1\": rpc error: code = NotFound desc = could not find container \"b055f09fd0b50ea99f0dfed3b12956f71b9216d7b6ab504375e0d391b44325a1\": container with ID starting with b055f09fd0b50ea99f0dfed3b12956f71b9216d7b6ab504375e0d391b44325a1 not found: ID does not exist" Jul 07 10:29:22.435905 ip-10-0-135-34 kubenswrapper[2578]: I0707 10:29:22.435824 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/c4552170-c8e7-43e3-9472-a066d5379772-trusted-ca\") pod \"c4552170-c8e7-43e3-9472-a066d5379772\" (UID: \"c4552170-c8e7-43e3-9472-a066d5379772\") " Jul 07 10:29:22.435905 ip-10-0-135-34 kubenswrapper[2578]: I0707 10:29:22.435858 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/c4552170-c8e7-43e3-9472-a066d5379772-bound-sa-token\") pod \"c4552170-c8e7-43e3-9472-a066d5379772\" (UID: \"c4552170-c8e7-43e3-9472-a066d5379772\") " Jul 07 10:29:22.435905 ip-10-0-135-34 kubenswrapper[2578]: I0707 10:29:22.435906 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/c4552170-c8e7-43e3-9472-a066d5379772-registry-certificates\") pod \"c4552170-c8e7-43e3-9472-a066d5379772\" (UID: \"c4552170-c8e7-43e3-9472-a066d5379772\") " Jul 07 10:29:22.436163 ip-10-0-135-34 kubenswrapper[2578]: I0707 10:29:22.435926 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-9tgml\" (UniqueName: \"kubernetes.io/projected/c4552170-c8e7-43e3-9472-a066d5379772-kube-api-access-9tgml\") pod \"c4552170-c8e7-43e3-9472-a066d5379772\" (UID: \"c4552170-c8e7-43e3-9472-a066d5379772\") " Jul 07 10:29:22.436163 ip-10-0-135-34 kubenswrapper[2578]: I0707 10:29:22.435951 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/c4552170-c8e7-43e3-9472-a066d5379772-image-registry-private-configuration\") pod \"c4552170-c8e7-43e3-9472-a066d5379772\" (UID: \"c4552170-c8e7-43e3-9472-a066d5379772\") " Jul 07 10:29:22.436163 ip-10-0-135-34 kubenswrapper[2578]: I0707 10:29:22.435973 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/c4552170-c8e7-43e3-9472-a066d5379772-ca-trust-extracted\") pod \"c4552170-c8e7-43e3-9472-a066d5379772\" (UID: \"c4552170-c8e7-43e3-9472-a066d5379772\") " Jul 07 10:29:22.436163 ip-10-0-135-34 kubenswrapper[2578]: I0707 10:29:22.435990 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/c4552170-c8e7-43e3-9472-a066d5379772-registry-tls\") pod \"c4552170-c8e7-43e3-9472-a066d5379772\" (UID: \"c4552170-c8e7-43e3-9472-a066d5379772\") " Jul 07 10:29:22.436163 ip-10-0-135-34 kubenswrapper[2578]: I0707 10:29:22.436007 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/c4552170-c8e7-43e3-9472-a066d5379772-installation-pull-secrets\") pod \"c4552170-c8e7-43e3-9472-a066d5379772\" (UID: \"c4552170-c8e7-43e3-9472-a066d5379772\") " Jul 07 10:29:22.436397 ip-10-0-135-34 kubenswrapper[2578]: I0707 10:29:22.436308 2578 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c4552170-c8e7-43e3-9472-a066d5379772-registry-certificates" (OuterVolumeSpecName: "registry-certificates") pod "c4552170-c8e7-43e3-9472-a066d5379772" (UID: "c4552170-c8e7-43e3-9472-a066d5379772"). InnerVolumeSpecName "registry-certificates". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 07 10:29:22.436397 ip-10-0-135-34 kubenswrapper[2578]: I0707 10:29:22.436389 2578 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c4552170-c8e7-43e3-9472-a066d5379772-trusted-ca" (OuterVolumeSpecName: "trusted-ca") pod "c4552170-c8e7-43e3-9472-a066d5379772" (UID: "c4552170-c8e7-43e3-9472-a066d5379772"). InnerVolumeSpecName "trusted-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 07 10:29:22.438612 ip-10-0-135-34 kubenswrapper[2578]: I0707 10:29:22.438580 2578 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c4552170-c8e7-43e3-9472-a066d5379772-installation-pull-secrets" (OuterVolumeSpecName: "installation-pull-secrets") pod "c4552170-c8e7-43e3-9472-a066d5379772" (UID: "c4552170-c8e7-43e3-9472-a066d5379772"). InnerVolumeSpecName "installation-pull-secrets". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 07 10:29:22.439041 ip-10-0-135-34 kubenswrapper[2578]: I0707 10:29:22.439011 2578 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c4552170-c8e7-43e3-9472-a066d5379772-bound-sa-token" (OuterVolumeSpecName: "bound-sa-token") pod "c4552170-c8e7-43e3-9472-a066d5379772" (UID: "c4552170-c8e7-43e3-9472-a066d5379772"). InnerVolumeSpecName "bound-sa-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 07 10:29:22.439159 ip-10-0-135-34 kubenswrapper[2578]: I0707 10:29:22.439030 2578 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c4552170-c8e7-43e3-9472-a066d5379772-image-registry-private-configuration" (OuterVolumeSpecName: "image-registry-private-configuration") pod "c4552170-c8e7-43e3-9472-a066d5379772" (UID: "c4552170-c8e7-43e3-9472-a066d5379772"). InnerVolumeSpecName "image-registry-private-configuration". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 07 10:29:22.439159 ip-10-0-135-34 kubenswrapper[2578]: I0707 10:29:22.439095 2578 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c4552170-c8e7-43e3-9472-a066d5379772-kube-api-access-9tgml" (OuterVolumeSpecName: "kube-api-access-9tgml") pod "c4552170-c8e7-43e3-9472-a066d5379772" (UID: "c4552170-c8e7-43e3-9472-a066d5379772"). InnerVolumeSpecName "kube-api-access-9tgml". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 07 10:29:22.439159 ip-10-0-135-34 kubenswrapper[2578]: I0707 10:29:22.439127 2578 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c4552170-c8e7-43e3-9472-a066d5379772-registry-tls" (OuterVolumeSpecName: "registry-tls") pod "c4552170-c8e7-43e3-9472-a066d5379772" (UID: "c4552170-c8e7-43e3-9472-a066d5379772"). InnerVolumeSpecName "registry-tls". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 07 10:29:22.443889 ip-10-0-135-34 kubenswrapper[2578]: I0707 10:29:22.443862 2578 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c4552170-c8e7-43e3-9472-a066d5379772-ca-trust-extracted" (OuterVolumeSpecName: "ca-trust-extracted") pod "c4552170-c8e7-43e3-9472-a066d5379772" (UID: "c4552170-c8e7-43e3-9472-a066d5379772"). InnerVolumeSpecName "ca-trust-extracted". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 07 10:29:22.537219 ip-10-0-135-34 kubenswrapper[2578]: I0707 10:29:22.537187 2578 reconciler_common.go:299] "Volume detached for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/c4552170-c8e7-43e3-9472-a066d5379772-registry-certificates\") on node \"ip-10-0-135-34.ec2.internal\" DevicePath \"\"" Jul 07 10:29:22.537219 ip-10-0-135-34 kubenswrapper[2578]: I0707 10:29:22.537214 2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9tgml\" (UniqueName: \"kubernetes.io/projected/c4552170-c8e7-43e3-9472-a066d5379772-kube-api-access-9tgml\") on node \"ip-10-0-135-34.ec2.internal\" DevicePath \"\"" Jul 07 10:29:22.537219 ip-10-0-135-34 kubenswrapper[2578]: I0707 10:29:22.537224 2578 reconciler_common.go:299] "Volume detached for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/c4552170-c8e7-43e3-9472-a066d5379772-image-registry-private-configuration\") on node \"ip-10-0-135-34.ec2.internal\" DevicePath \"\"" Jul 07 10:29:22.537416 ip-10-0-135-34 kubenswrapper[2578]: I0707 10:29:22.537233 2578 reconciler_common.go:299] "Volume detached for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/c4552170-c8e7-43e3-9472-a066d5379772-ca-trust-extracted\") on node \"ip-10-0-135-34.ec2.internal\" DevicePath \"\"" Jul 07 10:29:22.537416 ip-10-0-135-34 kubenswrapper[2578]: I0707 10:29:22.537242 2578 reconciler_common.go:299] "Volume detached for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/c4552170-c8e7-43e3-9472-a066d5379772-registry-tls\") on node \"ip-10-0-135-34.ec2.internal\" DevicePath \"\"" Jul 07 10:29:22.537416 ip-10-0-135-34 kubenswrapper[2578]: I0707 10:29:22.537251 2578 reconciler_common.go:299] "Volume detached for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/c4552170-c8e7-43e3-9472-a066d5379772-installation-pull-secrets\") on node \"ip-10-0-135-34.ec2.internal\" DevicePath \"\"" Jul 07 10:29:22.537416 ip-10-0-135-34 kubenswrapper[2578]: I0707 10:29:22.537259 2578 reconciler_common.go:299] "Volume detached for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/c4552170-c8e7-43e3-9472-a066d5379772-trusted-ca\") on node \"ip-10-0-135-34.ec2.internal\" DevicePath \"\"" Jul 07 10:29:22.537416 ip-10-0-135-34 kubenswrapper[2578]: I0707 10:29:22.537267 2578 reconciler_common.go:299] "Volume detached for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/c4552170-c8e7-43e3-9472-a066d5379772-bound-sa-token\") on node \"ip-10-0-135-34.ec2.internal\" DevicePath \"\"" Jul 07 10:29:22.722479 ip-10-0-135-34 kubenswrapper[2578]: I0707 10:29:22.722443 2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-cb7847d49-4nszw"] Jul 07 10:29:22.729321 ip-10-0-135-34 kubenswrapper[2578]: I0707 10:29:22.729295 2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-image-registry/image-registry-cb7847d49-4nszw"] Jul 07 10:29:23.820608 ip-10-0-135-34 kubenswrapper[2578]: I0707 10:29:23.820512 2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="c4552170-c8e7-43e3-9472-a066d5379772" path="/var/lib/kubelet/pods/c4552170-c8e7-43e3-9472-a066d5379772/volumes" Jul 07 10:29:24.401218 ip-10-0-135-34 kubenswrapper[2578]: I0707 10:29:24.401186 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-p2ks2" event={"ID":"931a7d75-9ee8-4758-9f23-3521f3db5eae","Type":"ContainerStarted","Data":"6cbe90c96e68f874e59ceb48faeb93431ab972bf5c9da05131d3692ab5ecc7f6"} Jul 07 10:29:24.401534 ip-10-0-135-34 kubenswrapper[2578]: I0707 10:29:24.401237 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-p2ks2" event={"ID":"931a7d75-9ee8-4758-9f23-3521f3db5eae","Type":"ContainerStarted","Data":"8dc909aeb335d523f7c493c9b01ea5b7948c08b401a83b97948b8d806d0a5ccf"} Jul 07 10:29:24.435096 ip-10-0-135-34 kubenswrapper[2578]: I0707 10:29:24.435048 2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/network-metrics-daemon-p2ks2" podStartSLOduration=131.03611396 podStartE2EDuration="2m12.43503339s" totalImagesPullingTime="1.39891943s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-07 10:27:12 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 10:29:24.434460635 +0000 UTC m=+133.106420179" watchObservedRunningTime="2026-07-07 10:29:24.43503339 +0000 UTC m=+133.106992934" Jul 07 10:29:32.432218 ip-10-0-135-34 kubenswrapper[2578]: I0707 10:29:32.432143 2578 kuberuntime_container.go:872] "Killing container with a grace period" pod="openshift-console/console-fc54c99d9-nh7qj" podUID="f8dcb050-7bdf-40c1-b47b-1019b93b1f9d" containerName="console" containerID="cri-o://5191c185fcc084c87808b8edba7d91d52a55b99f59c636756d12199822faa9b1" gracePeriod=15 Jul 07 10:29:32.674842 ip-10-0-135-34 kubenswrapper[2578]: I0707 10:29:32.674815 2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-fc54c99d9-nh7qj" Jul 07 10:29:32.823661 ip-10-0-135-34 kubenswrapper[2578]: I0707 10:29:32.823567 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/f8dcb050-7bdf-40c1-b47b-1019b93b1f9d-console-serving-cert\") pod \"f8dcb050-7bdf-40c1-b47b-1019b93b1f9d\" (UID: \"f8dcb050-7bdf-40c1-b47b-1019b93b1f9d\") " Jul 07 10:29:32.823661 ip-10-0-135-34 kubenswrapper[2578]: I0707 10:29:32.823608 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/f8dcb050-7bdf-40c1-b47b-1019b93b1f9d-service-ca\") pod \"f8dcb050-7bdf-40c1-b47b-1019b93b1f9d\" (UID: \"f8dcb050-7bdf-40c1-b47b-1019b93b1f9d\") " Jul 07 10:29:32.823661 ip-10-0-135-34 kubenswrapper[2578]: I0707 10:29:32.823624 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/f8dcb050-7bdf-40c1-b47b-1019b93b1f9d-console-config\") pod \"f8dcb050-7bdf-40c1-b47b-1019b93b1f9d\" (UID: \"f8dcb050-7bdf-40c1-b47b-1019b93b1f9d\") " Jul 07 10:29:32.823661 ip-10-0-135-34 kubenswrapper[2578]: I0707 10:29:32.823658 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-5j64c\" (UniqueName: \"kubernetes.io/projected/f8dcb050-7bdf-40c1-b47b-1019b93b1f9d-kube-api-access-5j64c\") pod \"f8dcb050-7bdf-40c1-b47b-1019b93b1f9d\" (UID: \"f8dcb050-7bdf-40c1-b47b-1019b93b1f9d\") " Jul 07 10:29:32.824031 ip-10-0-135-34 kubenswrapper[2578]: I0707 10:29:32.823801 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/f8dcb050-7bdf-40c1-b47b-1019b93b1f9d-oauth-serving-cert\") pod \"f8dcb050-7bdf-40c1-b47b-1019b93b1f9d\" (UID: \"f8dcb050-7bdf-40c1-b47b-1019b93b1f9d\") " Jul 07 10:29:32.824031 ip-10-0-135-34 kubenswrapper[2578]: I0707 10:29:32.823866 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/f8dcb050-7bdf-40c1-b47b-1019b93b1f9d-console-oauth-config\") pod \"f8dcb050-7bdf-40c1-b47b-1019b93b1f9d\" (UID: \"f8dcb050-7bdf-40c1-b47b-1019b93b1f9d\") " Jul 07 10:29:32.824133 ip-10-0-135-34 kubenswrapper[2578]: I0707 10:29:32.824077 2578 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f8dcb050-7bdf-40c1-b47b-1019b93b1f9d-console-config" (OuterVolumeSpecName: "console-config") pod "f8dcb050-7bdf-40c1-b47b-1019b93b1f9d" (UID: "f8dcb050-7bdf-40c1-b47b-1019b93b1f9d"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 07 10:29:32.824133 ip-10-0-135-34 kubenswrapper[2578]: I0707 10:29:32.824101 2578 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f8dcb050-7bdf-40c1-b47b-1019b93b1f9d-service-ca" (OuterVolumeSpecName: "service-ca") pod "f8dcb050-7bdf-40c1-b47b-1019b93b1f9d" (UID: "f8dcb050-7bdf-40c1-b47b-1019b93b1f9d"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 07 10:29:32.824205 ip-10-0-135-34 kubenswrapper[2578]: I0707 10:29:32.824143 2578 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f8dcb050-7bdf-40c1-b47b-1019b93b1f9d-oauth-serving-cert" (OuterVolumeSpecName: "oauth-serving-cert") pod "f8dcb050-7bdf-40c1-b47b-1019b93b1f9d" (UID: "f8dcb050-7bdf-40c1-b47b-1019b93b1f9d"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 07 10:29:32.826116 ip-10-0-135-34 kubenswrapper[2578]: I0707 10:29:32.826088 2578 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f8dcb050-7bdf-40c1-b47b-1019b93b1f9d-console-serving-cert" (OuterVolumeSpecName: "console-serving-cert") pod "f8dcb050-7bdf-40c1-b47b-1019b93b1f9d" (UID: "f8dcb050-7bdf-40c1-b47b-1019b93b1f9d"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 07 10:29:32.826116 ip-10-0-135-34 kubenswrapper[2578]: I0707 10:29:32.826098 2578 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f8dcb050-7bdf-40c1-b47b-1019b93b1f9d-kube-api-access-5j64c" (OuterVolumeSpecName: "kube-api-access-5j64c") pod "f8dcb050-7bdf-40c1-b47b-1019b93b1f9d" (UID: "f8dcb050-7bdf-40c1-b47b-1019b93b1f9d"). InnerVolumeSpecName "kube-api-access-5j64c". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 07 10:29:32.826249 ip-10-0-135-34 kubenswrapper[2578]: I0707 10:29:32.826146 2578 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f8dcb050-7bdf-40c1-b47b-1019b93b1f9d-console-oauth-config" (OuterVolumeSpecName: "console-oauth-config") pod "f8dcb050-7bdf-40c1-b47b-1019b93b1f9d" (UID: "f8dcb050-7bdf-40c1-b47b-1019b93b1f9d"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 07 10:29:32.924427 ip-10-0-135-34 kubenswrapper[2578]: I0707 10:29:32.924391 2578 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/f8dcb050-7bdf-40c1-b47b-1019b93b1f9d-console-oauth-config\") on node \"ip-10-0-135-34.ec2.internal\" DevicePath \"\"" Jul 07 10:29:32.924427 ip-10-0-135-34 kubenswrapper[2578]: I0707 10:29:32.924419 2578 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/f8dcb050-7bdf-40c1-b47b-1019b93b1f9d-console-serving-cert\") on node \"ip-10-0-135-34.ec2.internal\" DevicePath \"\"" Jul 07 10:29:32.924427 ip-10-0-135-34 kubenswrapper[2578]: I0707 10:29:32.924429 2578 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/f8dcb050-7bdf-40c1-b47b-1019b93b1f9d-service-ca\") on node \"ip-10-0-135-34.ec2.internal\" DevicePath \"\"" Jul 07 10:29:32.924427 ip-10-0-135-34 kubenswrapper[2578]: I0707 10:29:32.924437 2578 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/f8dcb050-7bdf-40c1-b47b-1019b93b1f9d-console-config\") on node \"ip-10-0-135-34.ec2.internal\" DevicePath \"\"" Jul 07 10:29:32.924670 ip-10-0-135-34 kubenswrapper[2578]: I0707 10:29:32.924446 2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5j64c\" (UniqueName: \"kubernetes.io/projected/f8dcb050-7bdf-40c1-b47b-1019b93b1f9d-kube-api-access-5j64c\") on node \"ip-10-0-135-34.ec2.internal\" DevicePath \"\"" Jul 07 10:29:32.924670 ip-10-0-135-34 kubenswrapper[2578]: I0707 10:29:32.924456 2578 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/f8dcb050-7bdf-40c1-b47b-1019b93b1f9d-oauth-serving-cert\") on node \"ip-10-0-135-34.ec2.internal\" DevicePath \"\"" Jul 07 10:29:33.434040 ip-10-0-135-34 kubenswrapper[2578]: I0707 10:29:33.434007 2578 generic.go:356] "Generic (PLEG): container finished" podID="f8dcb050-7bdf-40c1-b47b-1019b93b1f9d" containerID="5191c185fcc084c87808b8edba7d91d52a55b99f59c636756d12199822faa9b1" exitCode=0 Jul 07 10:29:33.434471 ip-10-0-135-34 kubenswrapper[2578]: I0707 10:29:33.434052 2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-fc54c99d9-nh7qj" Jul 07 10:29:33.434471 ip-10-0-135-34 kubenswrapper[2578]: I0707 10:29:33.434094 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-fc54c99d9-nh7qj" event={"ID":"f8dcb050-7bdf-40c1-b47b-1019b93b1f9d","Type":"ContainerDied","Data":"5191c185fcc084c87808b8edba7d91d52a55b99f59c636756d12199822faa9b1"} Jul 07 10:29:33.434471 ip-10-0-135-34 kubenswrapper[2578]: I0707 10:29:33.434145 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-fc54c99d9-nh7qj" event={"ID":"f8dcb050-7bdf-40c1-b47b-1019b93b1f9d","Type":"ContainerDied","Data":"b60bb3d0c8dbb20ae6d49171f9c2bcaac6c40a249d3899b4976887464614143f"} Jul 07 10:29:33.434471 ip-10-0-135-34 kubenswrapper[2578]: I0707 10:29:33.434167 2578 scope.go:117] "RemoveContainer" containerID="5191c185fcc084c87808b8edba7d91d52a55b99f59c636756d12199822faa9b1" Jul 07 10:29:33.443718 ip-10-0-135-34 kubenswrapper[2578]: I0707 10:29:33.443699 2578 scope.go:117] "RemoveContainer" containerID="5191c185fcc084c87808b8edba7d91d52a55b99f59c636756d12199822faa9b1" Jul 07 10:29:33.444035 ip-10-0-135-34 kubenswrapper[2578]: E0707 10:29:33.444016 2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5191c185fcc084c87808b8edba7d91d52a55b99f59c636756d12199822faa9b1\": container with ID starting with 5191c185fcc084c87808b8edba7d91d52a55b99f59c636756d12199822faa9b1 not found: ID does not exist" containerID="5191c185fcc084c87808b8edba7d91d52a55b99f59c636756d12199822faa9b1" Jul 07 10:29:33.444115 ip-10-0-135-34 kubenswrapper[2578]: I0707 10:29:33.444044 2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5191c185fcc084c87808b8edba7d91d52a55b99f59c636756d12199822faa9b1"} err="failed to get container status \"5191c185fcc084c87808b8edba7d91d52a55b99f59c636756d12199822faa9b1\": rpc error: code = NotFound desc = could not find container \"5191c185fcc084c87808b8edba7d91d52a55b99f59c636756d12199822faa9b1\": container with ID starting with 5191c185fcc084c87808b8edba7d91d52a55b99f59c636756d12199822faa9b1 not found: ID does not exist" Jul 07 10:29:33.459137 ip-10-0-135-34 kubenswrapper[2578]: I0707 10:29:33.459106 2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-fc54c99d9-nh7qj"] Jul 07 10:29:33.466577 ip-10-0-135-34 kubenswrapper[2578]: I0707 10:29:33.466549 2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-fc54c99d9-nh7qj"] Jul 07 10:29:33.820263 ip-10-0-135-34 kubenswrapper[2578]: I0707 10:29:33.820186 2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="f8dcb050-7bdf-40c1-b47b-1019b93b1f9d" path="/var/lib/kubelet/pods/f8dcb050-7bdf-40c1-b47b-1019b93b1f9d/volumes" Jul 07 10:30:11.991690 ip-10-0-135-34 kubenswrapper[2578]: I0707 10:30:11.990377 2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Jul 07 10:30:11.991690 ip-10-0-135-34 kubenswrapper[2578]: I0707 10:30:11.990705 2578 kuberuntime_container.go:872] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="229bf452-da72-420d-a150-9a9047364092" containerName="alertmanager" containerID="cri-o://acf54c7d774968536e7d8f960577a780c8fdd5d703c6e4b2a20c099f48361853" gracePeriod=120 Jul 07 10:30:11.991690 ip-10-0-135-34 kubenswrapper[2578]: I0707 10:30:11.991042 2578 kuberuntime_container.go:872] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="229bf452-da72-420d-a150-9a9047364092" containerName="prom-label-proxy" containerID="cri-o://61bf8d8145f073651698e6fe61d6b71c2610f1cea2a1bafc05b63747358cbd64" gracePeriod=120 Jul 07 10:30:11.991690 ip-10-0-135-34 kubenswrapper[2578]: I0707 10:30:11.991142 2578 kuberuntime_container.go:872] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="229bf452-da72-420d-a150-9a9047364092" containerName="kube-rbac-proxy-metric" containerID="cri-o://dcd01319e5000de9e2a766332f63932db5c37b469368f48696dae3efdc0b5705" gracePeriod=120 Jul 07 10:30:11.991690 ip-10-0-135-34 kubenswrapper[2578]: I0707 10:30:11.991211 2578 kuberuntime_container.go:872] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="229bf452-da72-420d-a150-9a9047364092" containerName="kube-rbac-proxy" containerID="cri-o://9cb5f333071c79a41f014f36b65bc8321c179495a470064c61c72bddeada4b5f" gracePeriod=120 Jul 07 10:30:11.991690 ip-10-0-135-34 kubenswrapper[2578]: I0707 10:30:11.991280 2578 kuberuntime_container.go:872] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="229bf452-da72-420d-a150-9a9047364092" containerName="kube-rbac-proxy-web" containerID="cri-o://6533c9269624e19743317012be74f60871604e781029330cce2a5b0837ec76d5" gracePeriod=120 Jul 07 10:30:11.991690 ip-10-0-135-34 kubenswrapper[2578]: I0707 10:30:11.991378 2578 kuberuntime_container.go:872] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="229bf452-da72-420d-a150-9a9047364092" containerName="config-reloader" containerID="cri-o://fa5103f6ae6a1fe8dcef27a7f7d2c9f46a40d470fee33c13bf42f81fc10b6aaf" gracePeriod=120 Jul 07 10:30:12.584923 ip-10-0-135-34 kubenswrapper[2578]: I0707 10:30:12.584889 2578 generic.go:356] "Generic (PLEG): container finished" podID="229bf452-da72-420d-a150-9a9047364092" containerID="61bf8d8145f073651698e6fe61d6b71c2610f1cea2a1bafc05b63747358cbd64" exitCode=0 Jul 07 10:30:12.584923 ip-10-0-135-34 kubenswrapper[2578]: I0707 10:30:12.584917 2578 generic.go:356] "Generic (PLEG): container finished" podID="229bf452-da72-420d-a150-9a9047364092" containerID="9cb5f333071c79a41f014f36b65bc8321c179495a470064c61c72bddeada4b5f" exitCode=0 Jul 07 10:30:12.584923 ip-10-0-135-34 kubenswrapper[2578]: I0707 10:30:12.584923 2578 generic.go:356] "Generic (PLEG): container finished" podID="229bf452-da72-420d-a150-9a9047364092" containerID="fa5103f6ae6a1fe8dcef27a7f7d2c9f46a40d470fee33c13bf42f81fc10b6aaf" exitCode=0 Jul 07 10:30:12.584923 ip-10-0-135-34 kubenswrapper[2578]: I0707 10:30:12.584931 2578 generic.go:356] "Generic (PLEG): container finished" podID="229bf452-da72-420d-a150-9a9047364092" containerID="acf54c7d774968536e7d8f960577a780c8fdd5d703c6e4b2a20c099f48361853" exitCode=0 Jul 07 10:30:12.585182 ip-10-0-135-34 kubenswrapper[2578]: I0707 10:30:12.584964 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"229bf452-da72-420d-a150-9a9047364092","Type":"ContainerDied","Data":"61bf8d8145f073651698e6fe61d6b71c2610f1cea2a1bafc05b63747358cbd64"} Jul 07 10:30:12.585182 ip-10-0-135-34 kubenswrapper[2578]: I0707 10:30:12.585051 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"229bf452-da72-420d-a150-9a9047364092","Type":"ContainerDied","Data":"9cb5f333071c79a41f014f36b65bc8321c179495a470064c61c72bddeada4b5f"} Jul 07 10:30:12.585182 ip-10-0-135-34 kubenswrapper[2578]: I0707 10:30:12.585078 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"229bf452-da72-420d-a150-9a9047364092","Type":"ContainerDied","Data":"fa5103f6ae6a1fe8dcef27a7f7d2c9f46a40d470fee33c13bf42f81fc10b6aaf"} Jul 07 10:30:12.585182 ip-10-0-135-34 kubenswrapper[2578]: I0707 10:30:12.585102 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"229bf452-da72-420d-a150-9a9047364092","Type":"ContainerDied","Data":"acf54c7d774968536e7d8f960577a780c8fdd5d703c6e4b2a20c099f48361853"} Jul 07 10:30:13.244091 ip-10-0-135-34 kubenswrapper[2578]: I0707 10:30:13.244055 2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0" Jul 07 10:30:13.326079 ip-10-0-135-34 kubenswrapper[2578]: I0707 10:30:13.325994 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/229bf452-da72-420d-a150-9a9047364092-cluster-tls-config\") pod \"229bf452-da72-420d-a150-9a9047364092\" (UID: \"229bf452-da72-420d-a150-9a9047364092\") " Jul 07 10:30:13.326079 ip-10-0-135-34 kubenswrapper[2578]: I0707 10:30:13.326034 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/229bf452-da72-420d-a150-9a9047364092-secret-alertmanager-kube-rbac-proxy-web\") pod \"229bf452-da72-420d-a150-9a9047364092\" (UID: \"229bf452-da72-420d-a150-9a9047364092\") " Jul 07 10:30:13.326079 ip-10-0-135-34 kubenswrapper[2578]: I0707 10:30:13.326056 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-v7pxm\" (UniqueName: \"kubernetes.io/projected/229bf452-da72-420d-a150-9a9047364092-kube-api-access-v7pxm\") pod \"229bf452-da72-420d-a150-9a9047364092\" (UID: \"229bf452-da72-420d-a150-9a9047364092\") " Jul 07 10:30:13.326079 ip-10-0-135-34 kubenswrapper[2578]: I0707 10:30:13.326077 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/229bf452-da72-420d-a150-9a9047364092-secret-alertmanager-kube-rbac-proxy\") pod \"229bf452-da72-420d-a150-9a9047364092\" (UID: \"229bf452-da72-420d-a150-9a9047364092\") " Jul 07 10:30:13.326387 ip-10-0-135-34 kubenswrapper[2578]: I0707 10:30:13.326110 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/229bf452-da72-420d-a150-9a9047364092-tls-assets\") pod \"229bf452-da72-420d-a150-9a9047364092\" (UID: \"229bf452-da72-420d-a150-9a9047364092\") " Jul 07 10:30:13.326387 ip-10-0-135-34 kubenswrapper[2578]: I0707 10:30:13.326137 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/229bf452-da72-420d-a150-9a9047364092-web-config\") pod \"229bf452-da72-420d-a150-9a9047364092\" (UID: \"229bf452-da72-420d-a150-9a9047364092\") " Jul 07 10:30:13.326387 ip-10-0-135-34 kubenswrapper[2578]: I0707 10:30:13.326170 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/229bf452-da72-420d-a150-9a9047364092-config-volume\") pod \"229bf452-da72-420d-a150-9a9047364092\" (UID: \"229bf452-da72-420d-a150-9a9047364092\") " Jul 07 10:30:13.326387 ip-10-0-135-34 kubenswrapper[2578]: I0707 10:30:13.326210 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/229bf452-da72-420d-a150-9a9047364092-secret-alertmanager-main-tls\") pod \"229bf452-da72-420d-a150-9a9047364092\" (UID: \"229bf452-da72-420d-a150-9a9047364092\") " Jul 07 10:30:13.328983 ip-10-0-135-34 kubenswrapper[2578]: I0707 10:30:13.328942 2578 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/229bf452-da72-420d-a150-9a9047364092-secret-alertmanager-kube-rbac-proxy-web" (OuterVolumeSpecName: "secret-alertmanager-kube-rbac-proxy-web") pod "229bf452-da72-420d-a150-9a9047364092" (UID: "229bf452-da72-420d-a150-9a9047364092"). InnerVolumeSpecName "secret-alertmanager-kube-rbac-proxy-web". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 07 10:30:13.329245 ip-10-0-135-34 kubenswrapper[2578]: I0707 10:30:13.329199 2578 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/229bf452-da72-420d-a150-9a9047364092-kube-api-access-v7pxm" (OuterVolumeSpecName: "kube-api-access-v7pxm") pod "229bf452-da72-420d-a150-9a9047364092" (UID: "229bf452-da72-420d-a150-9a9047364092"). InnerVolumeSpecName "kube-api-access-v7pxm". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 07 10:30:13.329374 ip-10-0-135-34 kubenswrapper[2578]: I0707 10:30:13.329266 2578 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/229bf452-da72-420d-a150-9a9047364092-secret-alertmanager-main-tls" (OuterVolumeSpecName: "secret-alertmanager-main-tls") pod "229bf452-da72-420d-a150-9a9047364092" (UID: "229bf452-da72-420d-a150-9a9047364092"). InnerVolumeSpecName "secret-alertmanager-main-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 07 10:30:13.329738 ip-10-0-135-34 kubenswrapper[2578]: I0707 10:30:13.329707 2578 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/229bf452-da72-420d-a150-9a9047364092-secret-alertmanager-kube-rbac-proxy" (OuterVolumeSpecName: "secret-alertmanager-kube-rbac-proxy") pod "229bf452-da72-420d-a150-9a9047364092" (UID: "229bf452-da72-420d-a150-9a9047364092"). InnerVolumeSpecName "secret-alertmanager-kube-rbac-proxy". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 07 10:30:13.330334 ip-10-0-135-34 kubenswrapper[2578]: I0707 10:30:13.330293 2578 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/229bf452-da72-420d-a150-9a9047364092-tls-assets" (OuterVolumeSpecName: "tls-assets") pod "229bf452-da72-420d-a150-9a9047364092" (UID: "229bf452-da72-420d-a150-9a9047364092"). InnerVolumeSpecName "tls-assets". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 07 10:30:13.330424 ip-10-0-135-34 kubenswrapper[2578]: I0707 10:30:13.330359 2578 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/229bf452-da72-420d-a150-9a9047364092-config-volume" (OuterVolumeSpecName: "config-volume") pod "229bf452-da72-420d-a150-9a9047364092" (UID: "229bf452-da72-420d-a150-9a9047364092"). InnerVolumeSpecName "config-volume". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 07 10:30:13.333716 ip-10-0-135-34 kubenswrapper[2578]: I0707 10:30:13.333698 2578 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/229bf452-da72-420d-a150-9a9047364092-cluster-tls-config" (OuterVolumeSpecName: "cluster-tls-config") pod "229bf452-da72-420d-a150-9a9047364092" (UID: "229bf452-da72-420d-a150-9a9047364092"). InnerVolumeSpecName "cluster-tls-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 07 10:30:13.339738 ip-10-0-135-34 kubenswrapper[2578]: I0707 10:30:13.339713 2578 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/229bf452-da72-420d-a150-9a9047364092-web-config" (OuterVolumeSpecName: "web-config") pod "229bf452-da72-420d-a150-9a9047364092" (UID: "229bf452-da72-420d-a150-9a9047364092"). InnerVolumeSpecName "web-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 07 10:30:13.426974 ip-10-0-135-34 kubenswrapper[2578]: I0707 10:30:13.426932 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/229bf452-da72-420d-a150-9a9047364092-config-out\") pod \"229bf452-da72-420d-a150-9a9047364092\" (UID: \"229bf452-da72-420d-a150-9a9047364092\") " Jul 07 10:30:13.427156 ip-10-0-135-34 kubenswrapper[2578]: I0707 10:30:13.427019 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/229bf452-da72-420d-a150-9a9047364092-metrics-client-ca\") pod \"229bf452-da72-420d-a150-9a9047364092\" (UID: \"229bf452-da72-420d-a150-9a9047364092\") " Jul 07 10:30:13.427156 ip-10-0-135-34 kubenswrapper[2578]: I0707 10:30:13.427039 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/229bf452-da72-420d-a150-9a9047364092-secret-alertmanager-kube-rbac-proxy-metric\") pod \"229bf452-da72-420d-a150-9a9047364092\" (UID: \"229bf452-da72-420d-a150-9a9047364092\") " Jul 07 10:30:13.427156 ip-10-0-135-34 kubenswrapper[2578]: I0707 10:30:13.427061 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/229bf452-da72-420d-a150-9a9047364092-alertmanager-main-db\") pod \"229bf452-da72-420d-a150-9a9047364092\" (UID: \"229bf452-da72-420d-a150-9a9047364092\") " Jul 07 10:30:13.427156 ip-10-0-135-34 kubenswrapper[2578]: I0707 10:30:13.427079 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/229bf452-da72-420d-a150-9a9047364092-alertmanager-trusted-ca-bundle\") pod \"229bf452-da72-420d-a150-9a9047364092\" (UID: \"229bf452-da72-420d-a150-9a9047364092\") " Jul 07 10:30:13.427352 ip-10-0-135-34 kubenswrapper[2578]: I0707 10:30:13.427187 2578 reconciler_common.go:299] "Volume detached for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/229bf452-da72-420d-a150-9a9047364092-secret-alertmanager-main-tls\") on node \"ip-10-0-135-34.ec2.internal\" DevicePath \"\"" Jul 07 10:30:13.427352 ip-10-0-135-34 kubenswrapper[2578]: I0707 10:30:13.427202 2578 reconciler_common.go:299] "Volume detached for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/229bf452-da72-420d-a150-9a9047364092-cluster-tls-config\") on node \"ip-10-0-135-34.ec2.internal\" DevicePath \"\"" Jul 07 10:30:13.427352 ip-10-0-135-34 kubenswrapper[2578]: I0707 10:30:13.427217 2578 reconciler_common.go:299] "Volume detached for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/229bf452-da72-420d-a150-9a9047364092-secret-alertmanager-kube-rbac-proxy-web\") on node \"ip-10-0-135-34.ec2.internal\" DevicePath \"\"" Jul 07 10:30:13.427352 ip-10-0-135-34 kubenswrapper[2578]: I0707 10:30:13.427232 2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-v7pxm\" (UniqueName: \"kubernetes.io/projected/229bf452-da72-420d-a150-9a9047364092-kube-api-access-v7pxm\") on node \"ip-10-0-135-34.ec2.internal\" DevicePath \"\"" Jul 07 10:30:13.427352 ip-10-0-135-34 kubenswrapper[2578]: I0707 10:30:13.427249 2578 reconciler_common.go:299] "Volume detached for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/229bf452-da72-420d-a150-9a9047364092-secret-alertmanager-kube-rbac-proxy\") on node \"ip-10-0-135-34.ec2.internal\" DevicePath \"\"" Jul 07 10:30:13.427352 ip-10-0-135-34 kubenswrapper[2578]: I0707 10:30:13.427263 2578 reconciler_common.go:299] "Volume detached for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/229bf452-da72-420d-a150-9a9047364092-tls-assets\") on node \"ip-10-0-135-34.ec2.internal\" DevicePath \"\"" Jul 07 10:30:13.427352 ip-10-0-135-34 kubenswrapper[2578]: I0707 10:30:13.427276 2578 reconciler_common.go:299] "Volume detached for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/229bf452-da72-420d-a150-9a9047364092-web-config\") on node \"ip-10-0-135-34.ec2.internal\" DevicePath \"\"" Jul 07 10:30:13.427352 ip-10-0-135-34 kubenswrapper[2578]: I0707 10:30:13.427288 2578 reconciler_common.go:299] "Volume detached for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/229bf452-da72-420d-a150-9a9047364092-config-volume\") on node \"ip-10-0-135-34.ec2.internal\" DevicePath \"\"" Jul 07 10:30:13.427622 ip-10-0-135-34 kubenswrapper[2578]: I0707 10:30:13.427397 2578 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/229bf452-da72-420d-a150-9a9047364092-alertmanager-main-db" (OuterVolumeSpecName: "alertmanager-main-db") pod "229bf452-da72-420d-a150-9a9047364092" (UID: "229bf452-da72-420d-a150-9a9047364092"). InnerVolumeSpecName "alertmanager-main-db". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 07 10:30:13.427622 ip-10-0-135-34 kubenswrapper[2578]: I0707 10:30:13.427439 2578 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/229bf452-da72-420d-a150-9a9047364092-metrics-client-ca" (OuterVolumeSpecName: "metrics-client-ca") pod "229bf452-da72-420d-a150-9a9047364092" (UID: "229bf452-da72-420d-a150-9a9047364092"). InnerVolumeSpecName "metrics-client-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 07 10:30:13.427622 ip-10-0-135-34 kubenswrapper[2578]: I0707 10:30:13.427501 2578 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/229bf452-da72-420d-a150-9a9047364092-alertmanager-trusted-ca-bundle" (OuterVolumeSpecName: "alertmanager-trusted-ca-bundle") pod "229bf452-da72-420d-a150-9a9047364092" (UID: "229bf452-da72-420d-a150-9a9047364092"). InnerVolumeSpecName "alertmanager-trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 07 10:30:13.429140 ip-10-0-135-34 kubenswrapper[2578]: I0707 10:30:13.429117 2578 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/229bf452-da72-420d-a150-9a9047364092-config-out" (OuterVolumeSpecName: "config-out") pod "229bf452-da72-420d-a150-9a9047364092" (UID: "229bf452-da72-420d-a150-9a9047364092"). InnerVolumeSpecName "config-out". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 07 10:30:13.429246 ip-10-0-135-34 kubenswrapper[2578]: I0707 10:30:13.429227 2578 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/229bf452-da72-420d-a150-9a9047364092-secret-alertmanager-kube-rbac-proxy-metric" (OuterVolumeSpecName: "secret-alertmanager-kube-rbac-proxy-metric") pod "229bf452-da72-420d-a150-9a9047364092" (UID: "229bf452-da72-420d-a150-9a9047364092"). InnerVolumeSpecName "secret-alertmanager-kube-rbac-proxy-metric". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 07 10:30:13.528466 ip-10-0-135-34 kubenswrapper[2578]: I0707 10:30:13.528418 2578 reconciler_common.go:299] "Volume detached for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/229bf452-da72-420d-a150-9a9047364092-metrics-client-ca\") on node \"ip-10-0-135-34.ec2.internal\" DevicePath \"\"" Jul 07 10:30:13.528466 ip-10-0-135-34 kubenswrapper[2578]: I0707 10:30:13.528448 2578 reconciler_common.go:299] "Volume detached for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/229bf452-da72-420d-a150-9a9047364092-secret-alertmanager-kube-rbac-proxy-metric\") on node \"ip-10-0-135-34.ec2.internal\" DevicePath \"\"" Jul 07 10:30:13.528466 ip-10-0-135-34 kubenswrapper[2578]: I0707 10:30:13.528458 2578 reconciler_common.go:299] "Volume detached for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/229bf452-da72-420d-a150-9a9047364092-alertmanager-main-db\") on node \"ip-10-0-135-34.ec2.internal\" DevicePath \"\"" Jul 07 10:30:13.528466 ip-10-0-135-34 kubenswrapper[2578]: I0707 10:30:13.528467 2578 reconciler_common.go:299] "Volume detached for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/229bf452-da72-420d-a150-9a9047364092-alertmanager-trusted-ca-bundle\") on node \"ip-10-0-135-34.ec2.internal\" DevicePath \"\"" Jul 07 10:30:13.528466 ip-10-0-135-34 kubenswrapper[2578]: I0707 10:30:13.528476 2578 reconciler_common.go:299] "Volume detached for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/229bf452-da72-420d-a150-9a9047364092-config-out\") on node \"ip-10-0-135-34.ec2.internal\" DevicePath \"\"" Jul 07 10:30:13.591936 ip-10-0-135-34 kubenswrapper[2578]: I0707 10:30:13.591855 2578 generic.go:356] "Generic (PLEG): container finished" podID="229bf452-da72-420d-a150-9a9047364092" containerID="dcd01319e5000de9e2a766332f63932db5c37b469368f48696dae3efdc0b5705" exitCode=0 Jul 07 10:30:13.591936 ip-10-0-135-34 kubenswrapper[2578]: I0707 10:30:13.591879 2578 generic.go:356] "Generic (PLEG): container finished" podID="229bf452-da72-420d-a150-9a9047364092" containerID="6533c9269624e19743317012be74f60871604e781029330cce2a5b0837ec76d5" exitCode=0 Jul 07 10:30:13.591936 ip-10-0-135-34 kubenswrapper[2578]: I0707 10:30:13.591894 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"229bf452-da72-420d-a150-9a9047364092","Type":"ContainerDied","Data":"dcd01319e5000de9e2a766332f63932db5c37b469368f48696dae3efdc0b5705"} Jul 07 10:30:13.591936 ip-10-0-135-34 kubenswrapper[2578]: I0707 10:30:13.591928 2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0" Jul 07 10:30:13.592167 ip-10-0-135-34 kubenswrapper[2578]: I0707 10:30:13.591964 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"229bf452-da72-420d-a150-9a9047364092","Type":"ContainerDied","Data":"6533c9269624e19743317012be74f60871604e781029330cce2a5b0837ec76d5"} Jul 07 10:30:13.592167 ip-10-0-135-34 kubenswrapper[2578]: I0707 10:30:13.591995 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"229bf452-da72-420d-a150-9a9047364092","Type":"ContainerDied","Data":"bbbea4088da42a2ec0a723142a76d94663f1f2794d543c0c8ed1b5ec901c87e8"} Jul 07 10:30:13.592167 ip-10-0-135-34 kubenswrapper[2578]: I0707 10:30:13.592033 2578 scope.go:117] "RemoveContainer" containerID="61bf8d8145f073651698e6fe61d6b71c2610f1cea2a1bafc05b63747358cbd64" Jul 07 10:30:13.600022 ip-10-0-135-34 kubenswrapper[2578]: I0707 10:30:13.600005 2578 scope.go:117] "RemoveContainer" containerID="dcd01319e5000de9e2a766332f63932db5c37b469368f48696dae3efdc0b5705" Jul 07 10:30:13.607184 ip-10-0-135-34 kubenswrapper[2578]: I0707 10:30:13.607160 2578 scope.go:117] "RemoveContainer" containerID="9cb5f333071c79a41f014f36b65bc8321c179495a470064c61c72bddeada4b5f" Jul 07 10:30:13.614534 ip-10-0-135-34 kubenswrapper[2578]: I0707 10:30:13.614510 2578 scope.go:117] "RemoveContainer" containerID="6533c9269624e19743317012be74f60871604e781029330cce2a5b0837ec76d5" Jul 07 10:30:13.616606 ip-10-0-135-34 kubenswrapper[2578]: I0707 10:30:13.616588 2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Jul 07 10:30:13.621936 ip-10-0-135-34 kubenswrapper[2578]: I0707 10:30:13.621919 2578 scope.go:117] "RemoveContainer" containerID="fa5103f6ae6a1fe8dcef27a7f7d2c9f46a40d470fee33c13bf42f81fc10b6aaf" Jul 07 10:30:13.624661 ip-10-0-135-34 kubenswrapper[2578]: I0707 10:30:13.624640 2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Jul 07 10:30:13.629493 ip-10-0-135-34 kubenswrapper[2578]: I0707 10:30:13.629475 2578 scope.go:117] "RemoveContainer" containerID="acf54c7d774968536e7d8f960577a780c8fdd5d703c6e4b2a20c099f48361853" Jul 07 10:30:13.637012 ip-10-0-135-34 kubenswrapper[2578]: I0707 10:30:13.636994 2578 scope.go:117] "RemoveContainer" containerID="03c99232f21674d142de0d4e2dbf5b4db736507cb33e48af080e97d8374d5a4a" Jul 07 10:30:13.644197 ip-10-0-135-34 kubenswrapper[2578]: I0707 10:30:13.644176 2578 scope.go:117] "RemoveContainer" containerID="61bf8d8145f073651698e6fe61d6b71c2610f1cea2a1bafc05b63747358cbd64" Jul 07 10:30:13.644463 ip-10-0-135-34 kubenswrapper[2578]: E0707 10:30:13.644444 2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"61bf8d8145f073651698e6fe61d6b71c2610f1cea2a1bafc05b63747358cbd64\": container with ID starting with 61bf8d8145f073651698e6fe61d6b71c2610f1cea2a1bafc05b63747358cbd64 not found: ID does not exist" containerID="61bf8d8145f073651698e6fe61d6b71c2610f1cea2a1bafc05b63747358cbd64" Jul 07 10:30:13.644532 ip-10-0-135-34 kubenswrapper[2578]: I0707 10:30:13.644476 2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"61bf8d8145f073651698e6fe61d6b71c2610f1cea2a1bafc05b63747358cbd64"} err="failed to get container status \"61bf8d8145f073651698e6fe61d6b71c2610f1cea2a1bafc05b63747358cbd64\": rpc error: code = NotFound desc = could not find container \"61bf8d8145f073651698e6fe61d6b71c2610f1cea2a1bafc05b63747358cbd64\": container with ID starting with 61bf8d8145f073651698e6fe61d6b71c2610f1cea2a1bafc05b63747358cbd64 not found: ID does not exist" Jul 07 10:30:13.644532 ip-10-0-135-34 kubenswrapper[2578]: I0707 10:30:13.644504 2578 scope.go:117] "RemoveContainer" containerID="dcd01319e5000de9e2a766332f63932db5c37b469368f48696dae3efdc0b5705" Jul 07 10:30:13.644744 ip-10-0-135-34 kubenswrapper[2578]: E0707 10:30:13.644728 2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"dcd01319e5000de9e2a766332f63932db5c37b469368f48696dae3efdc0b5705\": container with ID starting with dcd01319e5000de9e2a766332f63932db5c37b469368f48696dae3efdc0b5705 not found: ID does not exist" containerID="dcd01319e5000de9e2a766332f63932db5c37b469368f48696dae3efdc0b5705" Jul 07 10:30:13.644831 ip-10-0-135-34 kubenswrapper[2578]: I0707 10:30:13.644754 2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"dcd01319e5000de9e2a766332f63932db5c37b469368f48696dae3efdc0b5705"} err="failed to get container status \"dcd01319e5000de9e2a766332f63932db5c37b469368f48696dae3efdc0b5705\": rpc error: code = NotFound desc = could not find container \"dcd01319e5000de9e2a766332f63932db5c37b469368f48696dae3efdc0b5705\": container with ID starting with dcd01319e5000de9e2a766332f63932db5c37b469368f48696dae3efdc0b5705 not found: ID does not exist" Jul 07 10:30:13.644831 ip-10-0-135-34 kubenswrapper[2578]: I0707 10:30:13.644793 2578 scope.go:117] "RemoveContainer" containerID="9cb5f333071c79a41f014f36b65bc8321c179495a470064c61c72bddeada4b5f" Jul 07 10:30:13.645015 ip-10-0-135-34 kubenswrapper[2578]: E0707 10:30:13.645000 2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9cb5f333071c79a41f014f36b65bc8321c179495a470064c61c72bddeada4b5f\": container with ID starting with 9cb5f333071c79a41f014f36b65bc8321c179495a470064c61c72bddeada4b5f not found: ID does not exist" containerID="9cb5f333071c79a41f014f36b65bc8321c179495a470064c61c72bddeada4b5f" Jul 07 10:30:13.645055 ip-10-0-135-34 kubenswrapper[2578]: I0707 10:30:13.645020 2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9cb5f333071c79a41f014f36b65bc8321c179495a470064c61c72bddeada4b5f"} err="failed to get container status \"9cb5f333071c79a41f014f36b65bc8321c179495a470064c61c72bddeada4b5f\": rpc error: code = NotFound desc = could not find container \"9cb5f333071c79a41f014f36b65bc8321c179495a470064c61c72bddeada4b5f\": container with ID starting with 9cb5f333071c79a41f014f36b65bc8321c179495a470064c61c72bddeada4b5f not found: ID does not exist" Jul 07 10:30:13.645055 ip-10-0-135-34 kubenswrapper[2578]: I0707 10:30:13.645036 2578 scope.go:117] "RemoveContainer" containerID="6533c9269624e19743317012be74f60871604e781029330cce2a5b0837ec76d5" Jul 07 10:30:13.645260 ip-10-0-135-34 kubenswrapper[2578]: E0707 10:30:13.645242 2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6533c9269624e19743317012be74f60871604e781029330cce2a5b0837ec76d5\": container with ID starting with 6533c9269624e19743317012be74f60871604e781029330cce2a5b0837ec76d5 not found: ID does not exist" containerID="6533c9269624e19743317012be74f60871604e781029330cce2a5b0837ec76d5" Jul 07 10:30:13.645301 ip-10-0-135-34 kubenswrapper[2578]: I0707 10:30:13.645267 2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6533c9269624e19743317012be74f60871604e781029330cce2a5b0837ec76d5"} err="failed to get container status \"6533c9269624e19743317012be74f60871604e781029330cce2a5b0837ec76d5\": rpc error: code = NotFound desc = could not find container \"6533c9269624e19743317012be74f60871604e781029330cce2a5b0837ec76d5\": container with ID starting with 6533c9269624e19743317012be74f60871604e781029330cce2a5b0837ec76d5 not found: ID does not exist" Jul 07 10:30:13.645301 ip-10-0-135-34 kubenswrapper[2578]: I0707 10:30:13.645287 2578 scope.go:117] "RemoveContainer" containerID="fa5103f6ae6a1fe8dcef27a7f7d2c9f46a40d470fee33c13bf42f81fc10b6aaf" Jul 07 10:30:13.645494 ip-10-0-135-34 kubenswrapper[2578]: E0707 10:30:13.645476 2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fa5103f6ae6a1fe8dcef27a7f7d2c9f46a40d470fee33c13bf42f81fc10b6aaf\": container with ID starting with fa5103f6ae6a1fe8dcef27a7f7d2c9f46a40d470fee33c13bf42f81fc10b6aaf not found: ID does not exist" containerID="fa5103f6ae6a1fe8dcef27a7f7d2c9f46a40d470fee33c13bf42f81fc10b6aaf" Jul 07 10:30:13.645563 ip-10-0-135-34 kubenswrapper[2578]: I0707 10:30:13.645501 2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fa5103f6ae6a1fe8dcef27a7f7d2c9f46a40d470fee33c13bf42f81fc10b6aaf"} err="failed to get container status \"fa5103f6ae6a1fe8dcef27a7f7d2c9f46a40d470fee33c13bf42f81fc10b6aaf\": rpc error: code = NotFound desc = could not find container \"fa5103f6ae6a1fe8dcef27a7f7d2c9f46a40d470fee33c13bf42f81fc10b6aaf\": container with ID starting with fa5103f6ae6a1fe8dcef27a7f7d2c9f46a40d470fee33c13bf42f81fc10b6aaf not found: ID does not exist" Jul 07 10:30:13.645563 ip-10-0-135-34 kubenswrapper[2578]: I0707 10:30:13.645522 2578 scope.go:117] "RemoveContainer" containerID="acf54c7d774968536e7d8f960577a780c8fdd5d703c6e4b2a20c099f48361853" Jul 07 10:30:13.645740 ip-10-0-135-34 kubenswrapper[2578]: E0707 10:30:13.645723 2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"acf54c7d774968536e7d8f960577a780c8fdd5d703c6e4b2a20c099f48361853\": container with ID starting with acf54c7d774968536e7d8f960577a780c8fdd5d703c6e4b2a20c099f48361853 not found: ID does not exist" containerID="acf54c7d774968536e7d8f960577a780c8fdd5d703c6e4b2a20c099f48361853" Jul 07 10:30:13.645812 ip-10-0-135-34 kubenswrapper[2578]: I0707 10:30:13.645744 2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"acf54c7d774968536e7d8f960577a780c8fdd5d703c6e4b2a20c099f48361853"} err="failed to get container status \"acf54c7d774968536e7d8f960577a780c8fdd5d703c6e4b2a20c099f48361853\": rpc error: code = NotFound desc = could not find container \"acf54c7d774968536e7d8f960577a780c8fdd5d703c6e4b2a20c099f48361853\": container with ID starting with acf54c7d774968536e7d8f960577a780c8fdd5d703c6e4b2a20c099f48361853 not found: ID does not exist" Jul 07 10:30:13.645812 ip-10-0-135-34 kubenswrapper[2578]: I0707 10:30:13.645756 2578 scope.go:117] "RemoveContainer" containerID="03c99232f21674d142de0d4e2dbf5b4db736507cb33e48af080e97d8374d5a4a" Jul 07 10:30:13.645999 ip-10-0-135-34 kubenswrapper[2578]: E0707 10:30:13.645982 2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"03c99232f21674d142de0d4e2dbf5b4db736507cb33e48af080e97d8374d5a4a\": container with ID starting with 03c99232f21674d142de0d4e2dbf5b4db736507cb33e48af080e97d8374d5a4a not found: ID does not exist" containerID="03c99232f21674d142de0d4e2dbf5b4db736507cb33e48af080e97d8374d5a4a" Jul 07 10:30:13.646053 ip-10-0-135-34 kubenswrapper[2578]: I0707 10:30:13.646002 2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"03c99232f21674d142de0d4e2dbf5b4db736507cb33e48af080e97d8374d5a4a"} err="failed to get container status \"03c99232f21674d142de0d4e2dbf5b4db736507cb33e48af080e97d8374d5a4a\": rpc error: code = NotFound desc = could not find container \"03c99232f21674d142de0d4e2dbf5b4db736507cb33e48af080e97d8374d5a4a\": container with ID starting with 03c99232f21674d142de0d4e2dbf5b4db736507cb33e48af080e97d8374d5a4a not found: ID does not exist" Jul 07 10:30:13.646053 ip-10-0-135-34 kubenswrapper[2578]: I0707 10:30:13.646015 2578 scope.go:117] "RemoveContainer" containerID="61bf8d8145f073651698e6fe61d6b71c2610f1cea2a1bafc05b63747358cbd64" Jul 07 10:30:13.646221 ip-10-0-135-34 kubenswrapper[2578]: I0707 10:30:13.646204 2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"61bf8d8145f073651698e6fe61d6b71c2610f1cea2a1bafc05b63747358cbd64"} err="failed to get container status \"61bf8d8145f073651698e6fe61d6b71c2610f1cea2a1bafc05b63747358cbd64\": rpc error: code = NotFound desc = could not find container \"61bf8d8145f073651698e6fe61d6b71c2610f1cea2a1bafc05b63747358cbd64\": container with ID starting with 61bf8d8145f073651698e6fe61d6b71c2610f1cea2a1bafc05b63747358cbd64 not found: ID does not exist" Jul 07 10:30:13.646287 ip-10-0-135-34 kubenswrapper[2578]: I0707 10:30:13.646225 2578 scope.go:117] "RemoveContainer" containerID="dcd01319e5000de9e2a766332f63932db5c37b469368f48696dae3efdc0b5705" Jul 07 10:30:13.646479 ip-10-0-135-34 kubenswrapper[2578]: I0707 10:30:13.646463 2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"dcd01319e5000de9e2a766332f63932db5c37b469368f48696dae3efdc0b5705"} err="failed to get container status \"dcd01319e5000de9e2a766332f63932db5c37b469368f48696dae3efdc0b5705\": rpc error: code = NotFound desc = could not find container \"dcd01319e5000de9e2a766332f63932db5c37b469368f48696dae3efdc0b5705\": container with ID starting with dcd01319e5000de9e2a766332f63932db5c37b469368f48696dae3efdc0b5705 not found: ID does not exist" Jul 07 10:30:13.646520 ip-10-0-135-34 kubenswrapper[2578]: I0707 10:30:13.646480 2578 scope.go:117] "RemoveContainer" containerID="9cb5f333071c79a41f014f36b65bc8321c179495a470064c61c72bddeada4b5f" Jul 07 10:30:13.646673 ip-10-0-135-34 kubenswrapper[2578]: I0707 10:30:13.646656 2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9cb5f333071c79a41f014f36b65bc8321c179495a470064c61c72bddeada4b5f"} err="failed to get container status \"9cb5f333071c79a41f014f36b65bc8321c179495a470064c61c72bddeada4b5f\": rpc error: code = NotFound desc = could not find container \"9cb5f333071c79a41f014f36b65bc8321c179495a470064c61c72bddeada4b5f\": container with ID starting with 9cb5f333071c79a41f014f36b65bc8321c179495a470064c61c72bddeada4b5f not found: ID does not exist" Jul 07 10:30:13.646719 ip-10-0-135-34 kubenswrapper[2578]: I0707 10:30:13.646674 2578 scope.go:117] "RemoveContainer" containerID="6533c9269624e19743317012be74f60871604e781029330cce2a5b0837ec76d5" Jul 07 10:30:13.646896 ip-10-0-135-34 kubenswrapper[2578]: I0707 10:30:13.646866 2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6533c9269624e19743317012be74f60871604e781029330cce2a5b0837ec76d5"} err="failed to get container status \"6533c9269624e19743317012be74f60871604e781029330cce2a5b0837ec76d5\": rpc error: code = NotFound desc = could not find container \"6533c9269624e19743317012be74f60871604e781029330cce2a5b0837ec76d5\": container with ID starting with 6533c9269624e19743317012be74f60871604e781029330cce2a5b0837ec76d5 not found: ID does not exist" Jul 07 10:30:13.646896 ip-10-0-135-34 kubenswrapper[2578]: I0707 10:30:13.646887 2578 scope.go:117] "RemoveContainer" containerID="fa5103f6ae6a1fe8dcef27a7f7d2c9f46a40d470fee33c13bf42f81fc10b6aaf" Jul 07 10:30:13.647080 ip-10-0-135-34 kubenswrapper[2578]: I0707 10:30:13.647065 2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fa5103f6ae6a1fe8dcef27a7f7d2c9f46a40d470fee33c13bf42f81fc10b6aaf"} err="failed to get container status \"fa5103f6ae6a1fe8dcef27a7f7d2c9f46a40d470fee33c13bf42f81fc10b6aaf\": rpc error: code = NotFound desc = could not find container \"fa5103f6ae6a1fe8dcef27a7f7d2c9f46a40d470fee33c13bf42f81fc10b6aaf\": container with ID starting with fa5103f6ae6a1fe8dcef27a7f7d2c9f46a40d470fee33c13bf42f81fc10b6aaf not found: ID does not exist" Jul 07 10:30:13.647080 ip-10-0-135-34 kubenswrapper[2578]: I0707 10:30:13.647079 2578 scope.go:117] "RemoveContainer" containerID="acf54c7d774968536e7d8f960577a780c8fdd5d703c6e4b2a20c099f48361853" Jul 07 10:30:13.647316 ip-10-0-135-34 kubenswrapper[2578]: I0707 10:30:13.647296 2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"acf54c7d774968536e7d8f960577a780c8fdd5d703c6e4b2a20c099f48361853"} err="failed to get container status \"acf54c7d774968536e7d8f960577a780c8fdd5d703c6e4b2a20c099f48361853\": rpc error: code = NotFound desc = could not find container \"acf54c7d774968536e7d8f960577a780c8fdd5d703c6e4b2a20c099f48361853\": container with ID starting with acf54c7d774968536e7d8f960577a780c8fdd5d703c6e4b2a20c099f48361853 not found: ID does not exist" Jul 07 10:30:13.647400 ip-10-0-135-34 kubenswrapper[2578]: I0707 10:30:13.647317 2578 scope.go:117] "RemoveContainer" containerID="03c99232f21674d142de0d4e2dbf5b4db736507cb33e48af080e97d8374d5a4a" Jul 07 10:30:13.647546 ip-10-0-135-34 kubenswrapper[2578]: I0707 10:30:13.647528 2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"03c99232f21674d142de0d4e2dbf5b4db736507cb33e48af080e97d8374d5a4a"} err="failed to get container status \"03c99232f21674d142de0d4e2dbf5b4db736507cb33e48af080e97d8374d5a4a\": rpc error: code = NotFound desc = could not find container \"03c99232f21674d142de0d4e2dbf5b4db736507cb33e48af080e97d8374d5a4a\": container with ID starting with 03c99232f21674d142de0d4e2dbf5b4db736507cb33e48af080e97d8374d5a4a not found: ID does not exist" Jul 07 10:30:13.671257 ip-10-0-135-34 kubenswrapper[2578]: I0707 10:30:13.671222 2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Jul 07 10:30:13.671515 ip-10-0-135-34 kubenswrapper[2578]: I0707 10:30:13.671500 2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="229bf452-da72-420d-a150-9a9047364092" containerName="prom-label-proxy" Jul 07 10:30:13.671515 ip-10-0-135-34 kubenswrapper[2578]: I0707 10:30:13.671516 2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="229bf452-da72-420d-a150-9a9047364092" containerName="prom-label-proxy" Jul 07 10:30:13.671602 ip-10-0-135-34 kubenswrapper[2578]: I0707 10:30:13.671526 2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="229bf452-da72-420d-a150-9a9047364092" containerName="kube-rbac-proxy-metric" Jul 07 10:30:13.671602 ip-10-0-135-34 kubenswrapper[2578]: I0707 10:30:13.671532 2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="229bf452-da72-420d-a150-9a9047364092" containerName="kube-rbac-proxy-metric" Jul 07 10:30:13.671602 ip-10-0-135-34 kubenswrapper[2578]: I0707 10:30:13.671540 2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="229bf452-da72-420d-a150-9a9047364092" containerName="kube-rbac-proxy-web" Jul 07 10:30:13.671602 ip-10-0-135-34 kubenswrapper[2578]: I0707 10:30:13.671545 2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="229bf452-da72-420d-a150-9a9047364092" containerName="kube-rbac-proxy-web" Jul 07 10:30:13.671602 ip-10-0-135-34 kubenswrapper[2578]: I0707 10:30:13.671553 2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="229bf452-da72-420d-a150-9a9047364092" containerName="init-config-reloader" Jul 07 10:30:13.671602 ip-10-0-135-34 kubenswrapper[2578]: I0707 10:30:13.671559 2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="229bf452-da72-420d-a150-9a9047364092" containerName="init-config-reloader" Jul 07 10:30:13.671602 ip-10-0-135-34 kubenswrapper[2578]: I0707 10:30:13.671566 2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="229bf452-da72-420d-a150-9a9047364092" containerName="config-reloader" Jul 07 10:30:13.671602 ip-10-0-135-34 kubenswrapper[2578]: I0707 10:30:13.671572 2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="229bf452-da72-420d-a150-9a9047364092" containerName="config-reloader" Jul 07 10:30:13.671602 ip-10-0-135-34 kubenswrapper[2578]: I0707 10:30:13.671578 2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="229bf452-da72-420d-a150-9a9047364092" containerName="kube-rbac-proxy" Jul 07 10:30:13.671602 ip-10-0-135-34 kubenswrapper[2578]: I0707 10:30:13.671583 2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="229bf452-da72-420d-a150-9a9047364092" containerName="kube-rbac-proxy" Jul 07 10:30:13.671602 ip-10-0-135-34 kubenswrapper[2578]: I0707 10:30:13.671590 2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f8dcb050-7bdf-40c1-b47b-1019b93b1f9d" containerName="console" Jul 07 10:30:13.671602 ip-10-0-135-34 kubenswrapper[2578]: I0707 10:30:13.671594 2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="f8dcb050-7bdf-40c1-b47b-1019b93b1f9d" containerName="console" Jul 07 10:30:13.671602 ip-10-0-135-34 kubenswrapper[2578]: I0707 10:30:13.671601 2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="229bf452-da72-420d-a150-9a9047364092" containerName="alertmanager" Jul 07 10:30:13.671602 ip-10-0-135-34 kubenswrapper[2578]: I0707 10:30:13.671606 2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="229bf452-da72-420d-a150-9a9047364092" containerName="alertmanager" Jul 07 10:30:13.671996 ip-10-0-135-34 kubenswrapper[2578]: I0707 10:30:13.671611 2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c4552170-c8e7-43e3-9472-a066d5379772" containerName="registry" Jul 07 10:30:13.671996 ip-10-0-135-34 kubenswrapper[2578]: I0707 10:30:13.671616 2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="c4552170-c8e7-43e3-9472-a066d5379772" containerName="registry" Jul 07 10:30:13.671996 ip-10-0-135-34 kubenswrapper[2578]: I0707 10:30:13.671656 2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="229bf452-da72-420d-a150-9a9047364092" containerName="kube-rbac-proxy-metric" Jul 07 10:30:13.671996 ip-10-0-135-34 kubenswrapper[2578]: I0707 10:30:13.671662 2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="229bf452-da72-420d-a150-9a9047364092" containerName="alertmanager" Jul 07 10:30:13.671996 ip-10-0-135-34 kubenswrapper[2578]: I0707 10:30:13.671670 2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="229bf452-da72-420d-a150-9a9047364092" containerName="kube-rbac-proxy" Jul 07 10:30:13.671996 ip-10-0-135-34 kubenswrapper[2578]: I0707 10:30:13.671675 2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="229bf452-da72-420d-a150-9a9047364092" containerName="config-reloader" Jul 07 10:30:13.671996 ip-10-0-135-34 kubenswrapper[2578]: I0707 10:30:13.671681 2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="c4552170-c8e7-43e3-9472-a066d5379772" containerName="registry" Jul 07 10:30:13.671996 ip-10-0-135-34 kubenswrapper[2578]: I0707 10:30:13.671686 2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="f8dcb050-7bdf-40c1-b47b-1019b93b1f9d" containerName="console" Jul 07 10:30:13.671996 ip-10-0-135-34 kubenswrapper[2578]: I0707 10:30:13.671691 2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="229bf452-da72-420d-a150-9a9047364092" containerName="kube-rbac-proxy-web" Jul 07 10:30:13.671996 ip-10-0-135-34 kubenswrapper[2578]: I0707 10:30:13.671697 2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="229bf452-da72-420d-a150-9a9047364092" containerName="prom-label-proxy" Jul 07 10:30:13.678788 ip-10-0-135-34 kubenswrapper[2578]: I0707 10:30:13.678743 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0" Jul 07 10:30:13.681710 ip-10-0-135-34 kubenswrapper[2578]: I0707 10:30:13.681684 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-web-config\"" Jul 07 10:30:13.681710 ip-10-0-135-34 kubenswrapper[2578]: I0707 10:30:13.681702 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls-assets-0\"" Jul 07 10:30:13.681932 ip-10-0-135-34 kubenswrapper[2578]: I0707 10:30:13.681717 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-dockercfg-vb6pv\"" Jul 07 10:30:13.681932 ip-10-0-135-34 kubenswrapper[2578]: I0707 10:30:13.681687 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-metric\"" Jul 07 10:30:13.681932 ip-10-0-135-34 kubenswrapper[2578]: I0707 10:30:13.681680 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-cluster-tls-config\"" Jul 07 10:30:13.681932 ip-10-0-135-34 kubenswrapper[2578]: I0707 10:30:13.681833 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-web\"" Jul 07 10:30:13.682120 ip-10-0-135-34 kubenswrapper[2578]: I0707 10:30:13.682084 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-generated\"" Jul 07 10:30:13.682341 ip-10-0-135-34 kubenswrapper[2578]: I0707 10:30:13.682328 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy\"" Jul 07 10:30:13.682600 ip-10-0-135-34 kubenswrapper[2578]: I0707 10:30:13.682586 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls\"" Jul 07 10:30:13.689841 ip-10-0-135-34 kubenswrapper[2578]: I0707 10:30:13.689824 2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"alertmanager-trusted-ca-bundle\"" Jul 07 10:30:13.695335 ip-10-0-135-34 kubenswrapper[2578]: I0707 10:30:13.695313 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Jul 07 10:30:13.730547 ip-10-0-135-34 kubenswrapper[2578]: I0707 10:30:13.730515 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/c398d9f1-8285-4a95-8104-9144a247063d-config-out\") pod \"alertmanager-main-0\" (UID: \"c398d9f1-8285-4a95-8104-9144a247063d\") " pod="openshift-monitoring/alertmanager-main-0" Jul 07 10:30:13.730725 ip-10-0-135-34 kubenswrapper[2578]: I0707 10:30:13.730553 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/c398d9f1-8285-4a95-8104-9144a247063d-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"c398d9f1-8285-4a95-8104-9144a247063d\") " pod="openshift-monitoring/alertmanager-main-0" Jul 07 10:30:13.730725 ip-10-0-135-34 kubenswrapper[2578]: I0707 10:30:13.730581 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/c398d9f1-8285-4a95-8104-9144a247063d-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"c398d9f1-8285-4a95-8104-9144a247063d\") " pod="openshift-monitoring/alertmanager-main-0" Jul 07 10:30:13.730725 ip-10-0-135-34 kubenswrapper[2578]: I0707 10:30:13.730598 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/c398d9f1-8285-4a95-8104-9144a247063d-config-volume\") pod \"alertmanager-main-0\" (UID: \"c398d9f1-8285-4a95-8104-9144a247063d\") " pod="openshift-monitoring/alertmanager-main-0" Jul 07 10:30:13.730725 ip-10-0-135-34 kubenswrapper[2578]: I0707 10:30:13.730668 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/c398d9f1-8285-4a95-8104-9144a247063d-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"c398d9f1-8285-4a95-8104-9144a247063d\") " pod="openshift-monitoring/alertmanager-main-0" Jul 07 10:30:13.730725 ip-10-0-135-34 kubenswrapper[2578]: I0707 10:30:13.730703 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/c398d9f1-8285-4a95-8104-9144a247063d-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"c398d9f1-8285-4a95-8104-9144a247063d\") " pod="openshift-monitoring/alertmanager-main-0" Jul 07 10:30:13.730919 ip-10-0-135-34 kubenswrapper[2578]: I0707 10:30:13.730731 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c398d9f1-8285-4a95-8104-9144a247063d-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"c398d9f1-8285-4a95-8104-9144a247063d\") " pod="openshift-monitoring/alertmanager-main-0" Jul 07 10:30:13.730919 ip-10-0-135-34 kubenswrapper[2578]: I0707 10:30:13.730823 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/c398d9f1-8285-4a95-8104-9144a247063d-tls-assets\") pod \"alertmanager-main-0\" (UID: \"c398d9f1-8285-4a95-8104-9144a247063d\") " pod="openshift-monitoring/alertmanager-main-0" Jul 07 10:30:13.730919 ip-10-0-135-34 kubenswrapper[2578]: I0707 10:30:13.730843 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/c398d9f1-8285-4a95-8104-9144a247063d-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"c398d9f1-8285-4a95-8104-9144a247063d\") " pod="openshift-monitoring/alertmanager-main-0" Jul 07 10:30:13.730919 ip-10-0-135-34 kubenswrapper[2578]: I0707 10:30:13.730862 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/c398d9f1-8285-4a95-8104-9144a247063d-web-config\") pod \"alertmanager-main-0\" (UID: \"c398d9f1-8285-4a95-8104-9144a247063d\") " pod="openshift-monitoring/alertmanager-main-0" Jul 07 10:30:13.730919 ip-10-0-135-34 kubenswrapper[2578]: I0707 10:30:13.730883 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/c398d9f1-8285-4a95-8104-9144a247063d-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"c398d9f1-8285-4a95-8104-9144a247063d\") " pod="openshift-monitoring/alertmanager-main-0" Jul 07 10:30:13.730919 ip-10-0-135-34 kubenswrapper[2578]: I0707 10:30:13.730899 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/c398d9f1-8285-4a95-8104-9144a247063d-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"c398d9f1-8285-4a95-8104-9144a247063d\") " pod="openshift-monitoring/alertmanager-main-0" Jul 07 10:30:13.731087 ip-10-0-135-34 kubenswrapper[2578]: I0707 10:30:13.730919 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zdjrz\" (UniqueName: \"kubernetes.io/projected/c398d9f1-8285-4a95-8104-9144a247063d-kube-api-access-zdjrz\") pod \"alertmanager-main-0\" (UID: \"c398d9f1-8285-4a95-8104-9144a247063d\") " pod="openshift-monitoring/alertmanager-main-0" Jul 07 10:30:13.820511 ip-10-0-135-34 kubenswrapper[2578]: I0707 10:30:13.820481 2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="229bf452-da72-420d-a150-9a9047364092" path="/var/lib/kubelet/pods/229bf452-da72-420d-a150-9a9047364092/volumes" Jul 07 10:30:13.832015 ip-10-0-135-34 kubenswrapper[2578]: I0707 10:30:13.831990 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c398d9f1-8285-4a95-8104-9144a247063d-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"c398d9f1-8285-4a95-8104-9144a247063d\") " pod="openshift-monitoring/alertmanager-main-0" Jul 07 10:30:13.832122 ip-10-0-135-34 kubenswrapper[2578]: I0707 10:30:13.832041 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/c398d9f1-8285-4a95-8104-9144a247063d-tls-assets\") pod \"alertmanager-main-0\" (UID: \"c398d9f1-8285-4a95-8104-9144a247063d\") " pod="openshift-monitoring/alertmanager-main-0" Jul 07 10:30:13.832122 ip-10-0-135-34 kubenswrapper[2578]: I0707 10:30:13.832074 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/c398d9f1-8285-4a95-8104-9144a247063d-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"c398d9f1-8285-4a95-8104-9144a247063d\") " pod="openshift-monitoring/alertmanager-main-0" Jul 07 10:30:13.832295 ip-10-0-135-34 kubenswrapper[2578]: I0707 10:30:13.832229 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/c398d9f1-8285-4a95-8104-9144a247063d-web-config\") pod \"alertmanager-main-0\" (UID: \"c398d9f1-8285-4a95-8104-9144a247063d\") " pod="openshift-monitoring/alertmanager-main-0" Jul 07 10:30:13.832412 ip-10-0-135-34 kubenswrapper[2578]: I0707 10:30:13.832390 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/c398d9f1-8285-4a95-8104-9144a247063d-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"c398d9f1-8285-4a95-8104-9144a247063d\") " pod="openshift-monitoring/alertmanager-main-0" Jul 07 10:30:13.832463 ip-10-0-135-34 kubenswrapper[2578]: I0707 10:30:13.832430 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/c398d9f1-8285-4a95-8104-9144a247063d-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"c398d9f1-8285-4a95-8104-9144a247063d\") " pod="openshift-monitoring/alertmanager-main-0" Jul 07 10:30:13.832512 ip-10-0-135-34 kubenswrapper[2578]: I0707 10:30:13.832464 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zdjrz\" (UniqueName: \"kubernetes.io/projected/c398d9f1-8285-4a95-8104-9144a247063d-kube-api-access-zdjrz\") pod \"alertmanager-main-0\" (UID: \"c398d9f1-8285-4a95-8104-9144a247063d\") " pod="openshift-monitoring/alertmanager-main-0" Jul 07 10:30:13.832512 ip-10-0-135-34 kubenswrapper[2578]: I0707 10:30:13.832503 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/c398d9f1-8285-4a95-8104-9144a247063d-config-out\") pod \"alertmanager-main-0\" (UID: \"c398d9f1-8285-4a95-8104-9144a247063d\") " pod="openshift-monitoring/alertmanager-main-0" Jul 07 10:30:13.832611 ip-10-0-135-34 kubenswrapper[2578]: I0707 10:30:13.832536 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/c398d9f1-8285-4a95-8104-9144a247063d-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"c398d9f1-8285-4a95-8104-9144a247063d\") " pod="openshift-monitoring/alertmanager-main-0" Jul 07 10:30:13.832611 ip-10-0-135-34 kubenswrapper[2578]: I0707 10:30:13.832577 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/c398d9f1-8285-4a95-8104-9144a247063d-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"c398d9f1-8285-4a95-8104-9144a247063d\") " pod="openshift-monitoring/alertmanager-main-0" Jul 07 10:30:13.832611 ip-10-0-135-34 kubenswrapper[2578]: I0707 10:30:13.832603 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/c398d9f1-8285-4a95-8104-9144a247063d-config-volume\") pod \"alertmanager-main-0\" (UID: \"c398d9f1-8285-4a95-8104-9144a247063d\") " pod="openshift-monitoring/alertmanager-main-0" Jul 07 10:30:13.832743 ip-10-0-135-34 kubenswrapper[2578]: I0707 10:30:13.832637 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/c398d9f1-8285-4a95-8104-9144a247063d-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"c398d9f1-8285-4a95-8104-9144a247063d\") " pod="openshift-monitoring/alertmanager-main-0" Jul 07 10:30:13.832743 ip-10-0-135-34 kubenswrapper[2578]: I0707 10:30:13.832668 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/c398d9f1-8285-4a95-8104-9144a247063d-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"c398d9f1-8285-4a95-8104-9144a247063d\") " pod="openshift-monitoring/alertmanager-main-0" Jul 07 10:30:13.834345 ip-10-0-135-34 kubenswrapper[2578]: I0707 10:30:13.833410 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c398d9f1-8285-4a95-8104-9144a247063d-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"c398d9f1-8285-4a95-8104-9144a247063d\") " pod="openshift-monitoring/alertmanager-main-0" Jul 07 10:30:13.834345 ip-10-0-135-34 kubenswrapper[2578]: I0707 10:30:13.834061 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/c398d9f1-8285-4a95-8104-9144a247063d-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"c398d9f1-8285-4a95-8104-9144a247063d\") " pod="openshift-monitoring/alertmanager-main-0" Jul 07 10:30:13.834345 ip-10-0-135-34 kubenswrapper[2578]: I0707 10:30:13.834299 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/c398d9f1-8285-4a95-8104-9144a247063d-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"c398d9f1-8285-4a95-8104-9144a247063d\") " pod="openshift-monitoring/alertmanager-main-0" Jul 07 10:30:13.835711 ip-10-0-135-34 kubenswrapper[2578]: I0707 10:30:13.835251 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/c398d9f1-8285-4a95-8104-9144a247063d-tls-assets\") pod \"alertmanager-main-0\" (UID: \"c398d9f1-8285-4a95-8104-9144a247063d\") " pod="openshift-monitoring/alertmanager-main-0" Jul 07 10:30:13.835711 ip-10-0-135-34 kubenswrapper[2578]: I0707 10:30:13.835311 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/c398d9f1-8285-4a95-8104-9144a247063d-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"c398d9f1-8285-4a95-8104-9144a247063d\") " pod="openshift-monitoring/alertmanager-main-0" Jul 07 10:30:13.835711 ip-10-0-135-34 kubenswrapper[2578]: I0707 10:30:13.835628 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/c398d9f1-8285-4a95-8104-9144a247063d-web-config\") pod \"alertmanager-main-0\" (UID: \"c398d9f1-8285-4a95-8104-9144a247063d\") " pod="openshift-monitoring/alertmanager-main-0" Jul 07 10:30:13.835711 ip-10-0-135-34 kubenswrapper[2578]: I0707 10:30:13.835678 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/c398d9f1-8285-4a95-8104-9144a247063d-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"c398d9f1-8285-4a95-8104-9144a247063d\") " pod="openshift-monitoring/alertmanager-main-0" Jul 07 10:30:13.835711 ip-10-0-135-34 kubenswrapper[2578]: I0707 10:30:13.835712 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/c398d9f1-8285-4a95-8104-9144a247063d-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"c398d9f1-8285-4a95-8104-9144a247063d\") " pod="openshift-monitoring/alertmanager-main-0" Jul 07 10:30:13.836053 ip-10-0-135-34 kubenswrapper[2578]: I0707 10:30:13.835758 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/c398d9f1-8285-4a95-8104-9144a247063d-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"c398d9f1-8285-4a95-8104-9144a247063d\") " pod="openshift-monitoring/alertmanager-main-0" Jul 07 10:30:13.836637 ip-10-0-135-34 kubenswrapper[2578]: I0707 10:30:13.836617 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/c398d9f1-8285-4a95-8104-9144a247063d-config-volume\") pod \"alertmanager-main-0\" (UID: \"c398d9f1-8285-4a95-8104-9144a247063d\") " pod="openshift-monitoring/alertmanager-main-0" Jul 07 10:30:13.836953 ip-10-0-135-34 kubenswrapper[2578]: I0707 10:30:13.836933 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/c398d9f1-8285-4a95-8104-9144a247063d-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"c398d9f1-8285-4a95-8104-9144a247063d\") " pod="openshift-monitoring/alertmanager-main-0" Jul 07 10:30:13.837174 ip-10-0-135-34 kubenswrapper[2578]: I0707 10:30:13.837158 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/c398d9f1-8285-4a95-8104-9144a247063d-config-out\") pod \"alertmanager-main-0\" (UID: \"c398d9f1-8285-4a95-8104-9144a247063d\") " pod="openshift-monitoring/alertmanager-main-0" Jul 07 10:30:13.842183 ip-10-0-135-34 kubenswrapper[2578]: I0707 10:30:13.842139 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zdjrz\" (UniqueName: \"kubernetes.io/projected/c398d9f1-8285-4a95-8104-9144a247063d-kube-api-access-zdjrz\") pod \"alertmanager-main-0\" (UID: \"c398d9f1-8285-4a95-8104-9144a247063d\") " pod="openshift-monitoring/alertmanager-main-0" Jul 07 10:30:13.988917 ip-10-0-135-34 kubenswrapper[2578]: I0707 10:30:13.988877 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0" Jul 07 10:30:14.136671 ip-10-0-135-34 kubenswrapper[2578]: I0707 10:30:14.136629 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Jul 07 10:30:14.141224 ip-10-0-135-34 kubenswrapper[2578]: W0707 10:30:14.141198 2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc398d9f1_8285_4a95_8104_9144a247063d.slice/crio-502a801233e7185239763dcf16adbc4684a8206e7083e6e2525209cafd9f3d33 WatchSource:0}: Error finding container 502a801233e7185239763dcf16adbc4684a8206e7083e6e2525209cafd9f3d33: Status 404 returned error can't find the container with id 502a801233e7185239763dcf16adbc4684a8206e7083e6e2525209cafd9f3d33 Jul 07 10:30:14.598242 ip-10-0-135-34 kubenswrapper[2578]: I0707 10:30:14.598160 2578 generic.go:356] "Generic (PLEG): container finished" podID="c398d9f1-8285-4a95-8104-9144a247063d" containerID="ebc91769cc36ceabfce3cba8b4887acdb453e5b0e3f3d10344176ab1a67ce0e5" exitCode=0 Jul 07 10:30:14.598242 ip-10-0-135-34 kubenswrapper[2578]: I0707 10:30:14.598214 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"c398d9f1-8285-4a95-8104-9144a247063d","Type":"ContainerDied","Data":"ebc91769cc36ceabfce3cba8b4887acdb453e5b0e3f3d10344176ab1a67ce0e5"} Jul 07 10:30:14.598619 ip-10-0-135-34 kubenswrapper[2578]: I0707 10:30:14.598292 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"c398d9f1-8285-4a95-8104-9144a247063d","Type":"ContainerStarted","Data":"502a801233e7185239763dcf16adbc4684a8206e7083e6e2525209cafd9f3d33"} Jul 07 10:30:15.604648 ip-10-0-135-34 kubenswrapper[2578]: I0707 10:30:15.604610 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"c398d9f1-8285-4a95-8104-9144a247063d","Type":"ContainerStarted","Data":"2c2ed96000e377d4da0b1868a6d45096166272a863648e07e7f6312b7cb6ab4b"} Jul 07 10:30:15.605018 ip-10-0-135-34 kubenswrapper[2578]: I0707 10:30:15.604677 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"c398d9f1-8285-4a95-8104-9144a247063d","Type":"ContainerStarted","Data":"0032ce09a5c9678378f2f3f7066be596ea115646d992495b9136cba4c13d1153"} Jul 07 10:30:15.605018 ip-10-0-135-34 kubenswrapper[2578]: I0707 10:30:15.604703 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"c398d9f1-8285-4a95-8104-9144a247063d","Type":"ContainerStarted","Data":"33ac0077b95a44ed1ae6ddf617ef0adb32983e9fb304d5ccec4b491f1b469013"} Jul 07 10:30:15.605018 ip-10-0-135-34 kubenswrapper[2578]: I0707 10:30:15.604729 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"c398d9f1-8285-4a95-8104-9144a247063d","Type":"ContainerStarted","Data":"bdd716a6a089e318189fcea11d7066f1202d90ad83356aae9d9ea2dc3dcb87c0"} Jul 07 10:30:15.605018 ip-10-0-135-34 kubenswrapper[2578]: I0707 10:30:15.604757 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"c398d9f1-8285-4a95-8104-9144a247063d","Type":"ContainerStarted","Data":"55688782e73c194e28f73e1bb57047492ebbd51ceee85132b2481ff8c678bd6e"} Jul 07 10:30:15.605018 ip-10-0-135-34 kubenswrapper[2578]: I0707 10:30:15.604816 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"c398d9f1-8285-4a95-8104-9144a247063d","Type":"ContainerStarted","Data":"715cdd52ee1e260ba1e8856705735b4740bb9ab8093d1faada4ac35eca050d74"} Jul 07 10:30:15.639081 ip-10-0-135-34 kubenswrapper[2578]: I0707 10:30:15.639026 2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/alertmanager-main-0" podStartSLOduration=2.626336567 podStartE2EDuration="2.63900925s" totalImagesPullingTime="0s" totalInitContainerRuntime="12.672683ms" isStatefulPod=false podCreationTimestamp="2026-07-07 10:30:13 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 10:30:15.638595313 +0000 UTC m=+184.310554856" watchObservedRunningTime="2026-07-07 10:30:15.63900925 +0000 UTC m=+184.310968823" Jul 07 10:30:16.052683 ip-10-0-135-34 kubenswrapper[2578]: I0707 10:30:16.052597 2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/telemeter-client-57d74984bd-r76np"] Jul 07 10:30:16.056493 ip-10-0-135-34 kubenswrapper[2578]: I0707 10:30:16.056462 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/telemeter-client-57d74984bd-r76np" Jul 07 10:30:16.059682 ip-10-0-135-34 kubenswrapper[2578]: I0707 10:30:16.059660 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client\"" Jul 07 10:30:16.059880 ip-10-0-135-34 kubenswrapper[2578]: I0707 10:30:16.059863 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"federate-client-certs\"" Jul 07 10:30:16.060039 ip-10-0-135-34 kubenswrapper[2578]: I0707 10:30:16.060019 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client-tls\"" Jul 07 10:30:16.060172 ip-10-0-135-34 kubenswrapper[2578]: I0707 10:30:16.060153 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client-dockercfg-pk59m\"" Jul 07 10:30:16.060421 ip-10-0-135-34 kubenswrapper[2578]: I0707 10:30:16.060404 2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"telemeter-client-serving-certs-ca-bundle\"" Jul 07 10:30:16.060483 ip-10-0-135-34 kubenswrapper[2578]: I0707 10:30:16.060468 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client-kube-rbac-proxy-config\"" Jul 07 10:30:16.067886 ip-10-0-135-34 kubenswrapper[2578]: I0707 10:30:16.067863 2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"telemeter-trusted-ca-bundle-8i12ta5c71j38\"" Jul 07 10:30:16.072897 ip-10-0-135-34 kubenswrapper[2578]: I0707 10:30:16.072873 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/telemeter-client-57d74984bd-r76np"] Jul 07 10:30:16.150089 ip-10-0-135-34 kubenswrapper[2578]: I0707 10:30:16.150045 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"telemeter-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b1a95b49-97ce-4b2a-8500-f27f64dcfba7-telemeter-trusted-ca-bundle\") pod \"telemeter-client-57d74984bd-r76np\" (UID: \"b1a95b49-97ce-4b2a-8500-f27f64dcfba7\") " pod="openshift-monitoring/telemeter-client-57d74984bd-r76np" Jul 07 10:30:16.150277 ip-10-0-135-34 kubenswrapper[2578]: I0707 10:30:16.150105 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-telemeter-client-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/b1a95b49-97ce-4b2a-8500-f27f64dcfba7-secret-telemeter-client-kube-rbac-proxy-config\") pod \"telemeter-client-57d74984bd-r76np\" (UID: \"b1a95b49-97ce-4b2a-8500-f27f64dcfba7\") " pod="openshift-monitoring/telemeter-client-57d74984bd-r76np" Jul 07 10:30:16.150277 ip-10-0-135-34 kubenswrapper[2578]: I0707 10:30:16.150132 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"telemeter-client-tls\" (UniqueName: \"kubernetes.io/secret/b1a95b49-97ce-4b2a-8500-f27f64dcfba7-telemeter-client-tls\") pod \"telemeter-client-57d74984bd-r76np\" (UID: \"b1a95b49-97ce-4b2a-8500-f27f64dcfba7\") " pod="openshift-monitoring/telemeter-client-57d74984bd-r76np" Jul 07 10:30:16.150277 ip-10-0-135-34 kubenswrapper[2578]: I0707 10:30:16.150161 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-telemeter-client\" (UniqueName: \"kubernetes.io/secret/b1a95b49-97ce-4b2a-8500-f27f64dcfba7-secret-telemeter-client\") pod \"telemeter-client-57d74984bd-r76np\" (UID: \"b1a95b49-97ce-4b2a-8500-f27f64dcfba7\") " pod="openshift-monitoring/telemeter-client-57d74984bd-r76np" Jul 07 10:30:16.150277 ip-10-0-135-34 kubenswrapper[2578]: I0707 10:30:16.150202 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/b1a95b49-97ce-4b2a-8500-f27f64dcfba7-metrics-client-ca\") pod \"telemeter-client-57d74984bd-r76np\" (UID: \"b1a95b49-97ce-4b2a-8500-f27f64dcfba7\") " pod="openshift-monitoring/telemeter-client-57d74984bd-r76np" Jul 07 10:30:16.150277 ip-10-0-135-34 kubenswrapper[2578]: I0707 10:30:16.150233 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b1a95b49-97ce-4b2a-8500-f27f64dcfba7-serving-certs-ca-bundle\") pod \"telemeter-client-57d74984bd-r76np\" (UID: \"b1a95b49-97ce-4b2a-8500-f27f64dcfba7\") " pod="openshift-monitoring/telemeter-client-57d74984bd-r76np" Jul 07 10:30:16.150277 ip-10-0-135-34 kubenswrapper[2578]: I0707 10:30:16.150266 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-scbcz\" (UniqueName: \"kubernetes.io/projected/b1a95b49-97ce-4b2a-8500-f27f64dcfba7-kube-api-access-scbcz\") pod \"telemeter-client-57d74984bd-r76np\" (UID: \"b1a95b49-97ce-4b2a-8500-f27f64dcfba7\") " pod="openshift-monitoring/telemeter-client-57d74984bd-r76np" Jul 07 10:30:16.150535 ip-10-0-135-34 kubenswrapper[2578]: I0707 10:30:16.150296 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"federate-client-tls\" (UniqueName: \"kubernetes.io/secret/b1a95b49-97ce-4b2a-8500-f27f64dcfba7-federate-client-tls\") pod \"telemeter-client-57d74984bd-r76np\" (UID: \"b1a95b49-97ce-4b2a-8500-f27f64dcfba7\") " pod="openshift-monitoring/telemeter-client-57d74984bd-r76np" Jul 07 10:30:16.251920 ip-10-0-135-34 kubenswrapper[2578]: I0707 10:30:16.251874 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"federate-client-tls\" (UniqueName: \"kubernetes.io/secret/b1a95b49-97ce-4b2a-8500-f27f64dcfba7-federate-client-tls\") pod \"telemeter-client-57d74984bd-r76np\" (UID: \"b1a95b49-97ce-4b2a-8500-f27f64dcfba7\") " pod="openshift-monitoring/telemeter-client-57d74984bd-r76np" Jul 07 10:30:16.252069 ip-10-0-135-34 kubenswrapper[2578]: I0707 10:30:16.251935 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"telemeter-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b1a95b49-97ce-4b2a-8500-f27f64dcfba7-telemeter-trusted-ca-bundle\") pod \"telemeter-client-57d74984bd-r76np\" (UID: \"b1a95b49-97ce-4b2a-8500-f27f64dcfba7\") " pod="openshift-monitoring/telemeter-client-57d74984bd-r76np" Jul 07 10:30:16.252069 ip-10-0-135-34 kubenswrapper[2578]: I0707 10:30:16.251966 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-telemeter-client-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/b1a95b49-97ce-4b2a-8500-f27f64dcfba7-secret-telemeter-client-kube-rbac-proxy-config\") pod \"telemeter-client-57d74984bd-r76np\" (UID: \"b1a95b49-97ce-4b2a-8500-f27f64dcfba7\") " pod="openshift-monitoring/telemeter-client-57d74984bd-r76np" Jul 07 10:30:16.252069 ip-10-0-135-34 kubenswrapper[2578]: I0707 10:30:16.251984 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"telemeter-client-tls\" (UniqueName: \"kubernetes.io/secret/b1a95b49-97ce-4b2a-8500-f27f64dcfba7-telemeter-client-tls\") pod \"telemeter-client-57d74984bd-r76np\" (UID: \"b1a95b49-97ce-4b2a-8500-f27f64dcfba7\") " pod="openshift-monitoring/telemeter-client-57d74984bd-r76np" Jul 07 10:30:16.252069 ip-10-0-135-34 kubenswrapper[2578]: I0707 10:30:16.252004 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-telemeter-client\" (UniqueName: \"kubernetes.io/secret/b1a95b49-97ce-4b2a-8500-f27f64dcfba7-secret-telemeter-client\") pod \"telemeter-client-57d74984bd-r76np\" (UID: \"b1a95b49-97ce-4b2a-8500-f27f64dcfba7\") " pod="openshift-monitoring/telemeter-client-57d74984bd-r76np" Jul 07 10:30:16.252069 ip-10-0-135-34 kubenswrapper[2578]: I0707 10:30:16.252031 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/b1a95b49-97ce-4b2a-8500-f27f64dcfba7-metrics-client-ca\") pod \"telemeter-client-57d74984bd-r76np\" (UID: \"b1a95b49-97ce-4b2a-8500-f27f64dcfba7\") " pod="openshift-monitoring/telemeter-client-57d74984bd-r76np" Jul 07 10:30:16.252069 ip-10-0-135-34 kubenswrapper[2578]: I0707 10:30:16.252063 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b1a95b49-97ce-4b2a-8500-f27f64dcfba7-serving-certs-ca-bundle\") pod \"telemeter-client-57d74984bd-r76np\" (UID: \"b1a95b49-97ce-4b2a-8500-f27f64dcfba7\") " pod="openshift-monitoring/telemeter-client-57d74984bd-r76np" Jul 07 10:30:16.252424 ip-10-0-135-34 kubenswrapper[2578]: I0707 10:30:16.252097 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-scbcz\" (UniqueName: \"kubernetes.io/projected/b1a95b49-97ce-4b2a-8500-f27f64dcfba7-kube-api-access-scbcz\") pod \"telemeter-client-57d74984bd-r76np\" (UID: \"b1a95b49-97ce-4b2a-8500-f27f64dcfba7\") " pod="openshift-monitoring/telemeter-client-57d74984bd-r76np" Jul 07 10:30:16.253082 ip-10-0-135-34 kubenswrapper[2578]: I0707 10:30:16.253050 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b1a95b49-97ce-4b2a-8500-f27f64dcfba7-serving-certs-ca-bundle\") pod \"telemeter-client-57d74984bd-r76np\" (UID: \"b1a95b49-97ce-4b2a-8500-f27f64dcfba7\") " pod="openshift-monitoring/telemeter-client-57d74984bd-r76np" Jul 07 10:30:16.253215 ip-10-0-135-34 kubenswrapper[2578]: I0707 10:30:16.253050 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/b1a95b49-97ce-4b2a-8500-f27f64dcfba7-metrics-client-ca\") pod \"telemeter-client-57d74984bd-r76np\" (UID: \"b1a95b49-97ce-4b2a-8500-f27f64dcfba7\") " pod="openshift-monitoring/telemeter-client-57d74984bd-r76np" Jul 07 10:30:16.253215 ip-10-0-135-34 kubenswrapper[2578]: I0707 10:30:16.253135 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"telemeter-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b1a95b49-97ce-4b2a-8500-f27f64dcfba7-telemeter-trusted-ca-bundle\") pod \"telemeter-client-57d74984bd-r76np\" (UID: \"b1a95b49-97ce-4b2a-8500-f27f64dcfba7\") " pod="openshift-monitoring/telemeter-client-57d74984bd-r76np" Jul 07 10:30:16.254642 ip-10-0-135-34 kubenswrapper[2578]: I0707 10:30:16.254613 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"federate-client-tls\" (UniqueName: \"kubernetes.io/secret/b1a95b49-97ce-4b2a-8500-f27f64dcfba7-federate-client-tls\") pod \"telemeter-client-57d74984bd-r76np\" (UID: \"b1a95b49-97ce-4b2a-8500-f27f64dcfba7\") " pod="openshift-monitoring/telemeter-client-57d74984bd-r76np" Jul 07 10:30:16.254818 ip-10-0-135-34 kubenswrapper[2578]: I0707 10:30:16.254794 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-telemeter-client-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/b1a95b49-97ce-4b2a-8500-f27f64dcfba7-secret-telemeter-client-kube-rbac-proxy-config\") pod \"telemeter-client-57d74984bd-r76np\" (UID: \"b1a95b49-97ce-4b2a-8500-f27f64dcfba7\") " pod="openshift-monitoring/telemeter-client-57d74984bd-r76np" Jul 07 10:30:16.254892 ip-10-0-135-34 kubenswrapper[2578]: I0707 10:30:16.254860 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"telemeter-client-tls\" (UniqueName: \"kubernetes.io/secret/b1a95b49-97ce-4b2a-8500-f27f64dcfba7-telemeter-client-tls\") pod \"telemeter-client-57d74984bd-r76np\" (UID: \"b1a95b49-97ce-4b2a-8500-f27f64dcfba7\") " pod="openshift-monitoring/telemeter-client-57d74984bd-r76np" Jul 07 10:30:16.254962 ip-10-0-135-34 kubenswrapper[2578]: I0707 10:30:16.254941 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-telemeter-client\" (UniqueName: \"kubernetes.io/secret/b1a95b49-97ce-4b2a-8500-f27f64dcfba7-secret-telemeter-client\") pod \"telemeter-client-57d74984bd-r76np\" (UID: \"b1a95b49-97ce-4b2a-8500-f27f64dcfba7\") " pod="openshift-monitoring/telemeter-client-57d74984bd-r76np" Jul 07 10:30:16.262626 ip-10-0-135-34 kubenswrapper[2578]: I0707 10:30:16.262602 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-scbcz\" (UniqueName: \"kubernetes.io/projected/b1a95b49-97ce-4b2a-8500-f27f64dcfba7-kube-api-access-scbcz\") pod \"telemeter-client-57d74984bd-r76np\" (UID: \"b1a95b49-97ce-4b2a-8500-f27f64dcfba7\") " pod="openshift-monitoring/telemeter-client-57d74984bd-r76np" Jul 07 10:30:16.366071 ip-10-0-135-34 kubenswrapper[2578]: I0707 10:30:16.366031 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/telemeter-client-57d74984bd-r76np" Jul 07 10:30:16.554712 ip-10-0-135-34 kubenswrapper[2578]: I0707 10:30:16.554686 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/telemeter-client-57d74984bd-r76np"] Jul 07 10:30:16.557090 ip-10-0-135-34 kubenswrapper[2578]: W0707 10:30:16.557066 2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb1a95b49_97ce_4b2a_8500_f27f64dcfba7.slice/crio-9ccdc5e71c3a42d6783950d844320ca1e390957d91c82712ee59686b64bcda45 WatchSource:0}: Error finding container 9ccdc5e71c3a42d6783950d844320ca1e390957d91c82712ee59686b64bcda45: Status 404 returned error can't find the container with id 9ccdc5e71c3a42d6783950d844320ca1e390957d91c82712ee59686b64bcda45 Jul 07 10:30:16.615541 ip-10-0-135-34 kubenswrapper[2578]: I0707 10:30:16.615454 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-57d74984bd-r76np" event={"ID":"b1a95b49-97ce-4b2a-8500-f27f64dcfba7","Type":"ContainerStarted","Data":"9ccdc5e71c3a42d6783950d844320ca1e390957d91c82712ee59686b64bcda45"} Jul 07 10:30:18.624860 ip-10-0-135-34 kubenswrapper[2578]: I0707 10:30:18.624818 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-57d74984bd-r76np" event={"ID":"b1a95b49-97ce-4b2a-8500-f27f64dcfba7","Type":"ContainerStarted","Data":"0d9dc192d0b3bcaef213813da81b3729d0a212e05e95dc71795698535f01a639"} Jul 07 10:30:18.625345 ip-10-0-135-34 kubenswrapper[2578]: I0707 10:30:18.624879 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-57d74984bd-r76np" event={"ID":"b1a95b49-97ce-4b2a-8500-f27f64dcfba7","Type":"ContainerStarted","Data":"01d7e53df53048d39343da3f1231cf2bf32b5b6166cbe141cc45a0af55a8c937"} Jul 07 10:30:18.625345 ip-10-0-135-34 kubenswrapper[2578]: I0707 10:30:18.624909 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-57d74984bd-r76np" event={"ID":"b1a95b49-97ce-4b2a-8500-f27f64dcfba7","Type":"ContainerStarted","Data":"1b2b981318b5920a046d7d038bacabb722ffaeadbe2a3fdad89b06621b8811bf"} Jul 07 10:30:18.660618 ip-10-0-135-34 kubenswrapper[2578]: I0707 10:30:18.660561 2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/telemeter-client-57d74984bd-r76np" podStartSLOduration=0.806656628 podStartE2EDuration="2.660539799s" totalImagesPullingTime="1.853883171s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-07 10:30:16 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 10:30:18.659905689 +0000 UTC m=+187.331865233" watchObservedRunningTime="2026-07-07 10:30:18.660539799 +0000 UTC m=+187.332499368" Jul 07 10:30:19.472971 ip-10-0-135-34 kubenswrapper[2578]: I0707 10:30:19.472930 2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-58f756fcd7-rcqbc"] Jul 07 10:30:19.477733 ip-10-0-135-34 kubenswrapper[2578]: I0707 10:30:19.477702 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-58f756fcd7-rcqbc" Jul 07 10:30:19.483525 ip-10-0-135-34 kubenswrapper[2578]: I0707 10:30:19.483500 2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"console-config\"" Jul 07 10:30:19.483667 ip-10-0-135-34 kubenswrapper[2578]: I0707 10:30:19.483500 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-oauth-config\"" Jul 07 10:30:19.483667 ip-10-0-135-34 kubenswrapper[2578]: I0707 10:30:19.483549 2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"kube-root-ca.crt\"" Jul 07 10:30:19.483958 ip-10-0-135-34 kubenswrapper[2578]: I0707 10:30:19.483938 2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"service-ca\"" Jul 07 10:30:19.484051 ip-10-0-135-34 kubenswrapper[2578]: I0707 10:30:19.483939 2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"openshift-service-ca.crt\"" Jul 07 10:30:19.484906 ip-10-0-135-34 kubenswrapper[2578]: I0707 10:30:19.484875 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-serving-cert\"" Jul 07 10:30:19.485025 ip-10-0-135-34 kubenswrapper[2578]: I0707 10:30:19.485008 2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"oauth-serving-cert\"" Jul 07 10:30:19.485233 ip-10-0-135-34 kubenswrapper[2578]: I0707 10:30:19.485215 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-dockercfg-d4kbr\"" Jul 07 10:30:19.488964 ip-10-0-135-34 kubenswrapper[2578]: I0707 10:30:19.488941 2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"trusted-ca-bundle\"" Jul 07 10:30:19.490839 ip-10-0-135-34 kubenswrapper[2578]: I0707 10:30:19.490808 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-58f756fcd7-rcqbc"] Jul 07 10:30:19.579472 ip-10-0-135-34 kubenswrapper[2578]: I0707 10:30:19.579432 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vfstj\" (UniqueName: \"kubernetes.io/projected/09a1d515-7b44-4cd8-a7d0-7683031291ed-kube-api-access-vfstj\") pod \"console-58f756fcd7-rcqbc\" (UID: \"09a1d515-7b44-4cd8-a7d0-7683031291ed\") " pod="openshift-console/console-58f756fcd7-rcqbc" Jul 07 10:30:19.579802 ip-10-0-135-34 kubenswrapper[2578]: I0707 10:30:19.579752 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/09a1d515-7b44-4cd8-a7d0-7683031291ed-console-serving-cert\") pod \"console-58f756fcd7-rcqbc\" (UID: \"09a1d515-7b44-4cd8-a7d0-7683031291ed\") " pod="openshift-console/console-58f756fcd7-rcqbc" Jul 07 10:30:19.580010 ip-10-0-135-34 kubenswrapper[2578]: I0707 10:30:19.579992 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/09a1d515-7b44-4cd8-a7d0-7683031291ed-console-config\") pod \"console-58f756fcd7-rcqbc\" (UID: \"09a1d515-7b44-4cd8-a7d0-7683031291ed\") " pod="openshift-console/console-58f756fcd7-rcqbc" Jul 07 10:30:19.580141 ip-10-0-135-34 kubenswrapper[2578]: I0707 10:30:19.580128 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/09a1d515-7b44-4cd8-a7d0-7683031291ed-console-oauth-config\") pod \"console-58f756fcd7-rcqbc\" (UID: \"09a1d515-7b44-4cd8-a7d0-7683031291ed\") " pod="openshift-console/console-58f756fcd7-rcqbc" Jul 07 10:30:19.580233 ip-10-0-135-34 kubenswrapper[2578]: I0707 10:30:19.580221 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/09a1d515-7b44-4cd8-a7d0-7683031291ed-service-ca\") pod \"console-58f756fcd7-rcqbc\" (UID: \"09a1d515-7b44-4cd8-a7d0-7683031291ed\") " pod="openshift-console/console-58f756fcd7-rcqbc" Jul 07 10:30:19.580331 ip-10-0-135-34 kubenswrapper[2578]: I0707 10:30:19.580319 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/09a1d515-7b44-4cd8-a7d0-7683031291ed-oauth-serving-cert\") pod \"console-58f756fcd7-rcqbc\" (UID: \"09a1d515-7b44-4cd8-a7d0-7683031291ed\") " pod="openshift-console/console-58f756fcd7-rcqbc" Jul 07 10:30:19.580424 ip-10-0-135-34 kubenswrapper[2578]: I0707 10:30:19.580413 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/09a1d515-7b44-4cd8-a7d0-7683031291ed-trusted-ca-bundle\") pod \"console-58f756fcd7-rcqbc\" (UID: \"09a1d515-7b44-4cd8-a7d0-7683031291ed\") " pod="openshift-console/console-58f756fcd7-rcqbc" Jul 07 10:30:19.681694 ip-10-0-135-34 kubenswrapper[2578]: I0707 10:30:19.681658 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vfstj\" (UniqueName: \"kubernetes.io/projected/09a1d515-7b44-4cd8-a7d0-7683031291ed-kube-api-access-vfstj\") pod \"console-58f756fcd7-rcqbc\" (UID: \"09a1d515-7b44-4cd8-a7d0-7683031291ed\") " pod="openshift-console/console-58f756fcd7-rcqbc" Jul 07 10:30:19.682175 ip-10-0-135-34 kubenswrapper[2578]: I0707 10:30:19.681725 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/09a1d515-7b44-4cd8-a7d0-7683031291ed-console-serving-cert\") pod \"console-58f756fcd7-rcqbc\" (UID: \"09a1d515-7b44-4cd8-a7d0-7683031291ed\") " pod="openshift-console/console-58f756fcd7-rcqbc" Jul 07 10:30:19.682175 ip-10-0-135-34 kubenswrapper[2578]: I0707 10:30:19.681824 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/09a1d515-7b44-4cd8-a7d0-7683031291ed-console-config\") pod \"console-58f756fcd7-rcqbc\" (UID: \"09a1d515-7b44-4cd8-a7d0-7683031291ed\") " pod="openshift-console/console-58f756fcd7-rcqbc" Jul 07 10:30:19.682175 ip-10-0-135-34 kubenswrapper[2578]: I0707 10:30:19.681936 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/09a1d515-7b44-4cd8-a7d0-7683031291ed-console-oauth-config\") pod \"console-58f756fcd7-rcqbc\" (UID: \"09a1d515-7b44-4cd8-a7d0-7683031291ed\") " pod="openshift-console/console-58f756fcd7-rcqbc" Jul 07 10:30:19.682175 ip-10-0-135-34 kubenswrapper[2578]: I0707 10:30:19.681962 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/09a1d515-7b44-4cd8-a7d0-7683031291ed-service-ca\") pod \"console-58f756fcd7-rcqbc\" (UID: \"09a1d515-7b44-4cd8-a7d0-7683031291ed\") " pod="openshift-console/console-58f756fcd7-rcqbc" Jul 07 10:30:19.682175 ip-10-0-135-34 kubenswrapper[2578]: I0707 10:30:19.681990 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/09a1d515-7b44-4cd8-a7d0-7683031291ed-oauth-serving-cert\") pod \"console-58f756fcd7-rcqbc\" (UID: \"09a1d515-7b44-4cd8-a7d0-7683031291ed\") " pod="openshift-console/console-58f756fcd7-rcqbc" Jul 07 10:30:19.682175 ip-10-0-135-34 kubenswrapper[2578]: I0707 10:30:19.682022 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/09a1d515-7b44-4cd8-a7d0-7683031291ed-trusted-ca-bundle\") pod \"console-58f756fcd7-rcqbc\" (UID: \"09a1d515-7b44-4cd8-a7d0-7683031291ed\") " pod="openshift-console/console-58f756fcd7-rcqbc" Jul 07 10:30:19.682671 ip-10-0-135-34 kubenswrapper[2578]: I0707 10:30:19.682643 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/09a1d515-7b44-4cd8-a7d0-7683031291ed-console-config\") pod \"console-58f756fcd7-rcqbc\" (UID: \"09a1d515-7b44-4cd8-a7d0-7683031291ed\") " pod="openshift-console/console-58f756fcd7-rcqbc" Jul 07 10:30:19.682756 ip-10-0-135-34 kubenswrapper[2578]: I0707 10:30:19.682681 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/09a1d515-7b44-4cd8-a7d0-7683031291ed-service-ca\") pod \"console-58f756fcd7-rcqbc\" (UID: \"09a1d515-7b44-4cd8-a7d0-7683031291ed\") " pod="openshift-console/console-58f756fcd7-rcqbc" Jul 07 10:30:19.682957 ip-10-0-135-34 kubenswrapper[2578]: I0707 10:30:19.682934 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/09a1d515-7b44-4cd8-a7d0-7683031291ed-oauth-serving-cert\") pod \"console-58f756fcd7-rcqbc\" (UID: \"09a1d515-7b44-4cd8-a7d0-7683031291ed\") " pod="openshift-console/console-58f756fcd7-rcqbc" Jul 07 10:30:19.683064 ip-10-0-135-34 kubenswrapper[2578]: I0707 10:30:19.683049 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/09a1d515-7b44-4cd8-a7d0-7683031291ed-trusted-ca-bundle\") pod \"console-58f756fcd7-rcqbc\" (UID: \"09a1d515-7b44-4cd8-a7d0-7683031291ed\") " pod="openshift-console/console-58f756fcd7-rcqbc" Jul 07 10:30:19.684926 ip-10-0-135-34 kubenswrapper[2578]: I0707 10:30:19.684902 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/09a1d515-7b44-4cd8-a7d0-7683031291ed-console-serving-cert\") pod \"console-58f756fcd7-rcqbc\" (UID: \"09a1d515-7b44-4cd8-a7d0-7683031291ed\") " pod="openshift-console/console-58f756fcd7-rcqbc" Jul 07 10:30:19.685021 ip-10-0-135-34 kubenswrapper[2578]: I0707 10:30:19.684966 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/09a1d515-7b44-4cd8-a7d0-7683031291ed-console-oauth-config\") pod \"console-58f756fcd7-rcqbc\" (UID: \"09a1d515-7b44-4cd8-a7d0-7683031291ed\") " pod="openshift-console/console-58f756fcd7-rcqbc" Jul 07 10:30:19.693417 ip-10-0-135-34 kubenswrapper[2578]: I0707 10:30:19.693388 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vfstj\" (UniqueName: \"kubernetes.io/projected/09a1d515-7b44-4cd8-a7d0-7683031291ed-kube-api-access-vfstj\") pod \"console-58f756fcd7-rcqbc\" (UID: \"09a1d515-7b44-4cd8-a7d0-7683031291ed\") " pod="openshift-console/console-58f756fcd7-rcqbc" Jul 07 10:30:19.787174 ip-10-0-135-34 kubenswrapper[2578]: I0707 10:30:19.787065 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-58f756fcd7-rcqbc" Jul 07 10:30:19.931049 ip-10-0-135-34 kubenswrapper[2578]: I0707 10:30:19.931018 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-58f756fcd7-rcqbc"] Jul 07 10:30:19.934825 ip-10-0-135-34 kubenswrapper[2578]: W0707 10:30:19.934795 2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod09a1d515_7b44_4cd8_a7d0_7683031291ed.slice/crio-70483a243110075ea1d9803d27b58df636e318387273288d47830c3731ce6395 WatchSource:0}: Error finding container 70483a243110075ea1d9803d27b58df636e318387273288d47830c3731ce6395: Status 404 returned error can't find the container with id 70483a243110075ea1d9803d27b58df636e318387273288d47830c3731ce6395 Jul 07 10:30:20.633630 ip-10-0-135-34 kubenswrapper[2578]: I0707 10:30:20.633594 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-58f756fcd7-rcqbc" event={"ID":"09a1d515-7b44-4cd8-a7d0-7683031291ed","Type":"ContainerStarted","Data":"29632f0c75d88767f31810d1395613f4d6f9f8acb594c463d634fdab1d819064"} Jul 07 10:30:20.633871 ip-10-0-135-34 kubenswrapper[2578]: I0707 10:30:20.633641 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-58f756fcd7-rcqbc" event={"ID":"09a1d515-7b44-4cd8-a7d0-7683031291ed","Type":"ContainerStarted","Data":"70483a243110075ea1d9803d27b58df636e318387273288d47830c3731ce6395"} Jul 07 10:30:20.661086 ip-10-0-135-34 kubenswrapper[2578]: I0707 10:30:20.661035 2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-58f756fcd7-rcqbc" podStartSLOduration=1.6610195559999998 podStartE2EDuration="1.661019556s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-07 10:30:19 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 10:30:20.659011151 +0000 UTC m=+189.330970696" watchObservedRunningTime="2026-07-07 10:30:20.661019556 +0000 UTC m=+189.332979100" Jul 07 10:30:29.787839 ip-10-0-135-34 kubenswrapper[2578]: I0707 10:30:29.787803 2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-58f756fcd7-rcqbc" Jul 07 10:30:29.788315 ip-10-0-135-34 kubenswrapper[2578]: I0707 10:30:29.787867 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-58f756fcd7-rcqbc" Jul 07 10:30:29.792382 ip-10-0-135-34 kubenswrapper[2578]: I0707 10:30:29.792357 2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-58f756fcd7-rcqbc" Jul 07 10:30:30.673858 ip-10-0-135-34 kubenswrapper[2578]: I0707 10:30:30.673829 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-58f756fcd7-rcqbc" Jul 07 10:31:40.221511 ip-10-0-135-34 kubenswrapper[2578]: I0707 10:31:40.221475 2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-58f756fcd7-rcqbc"] Jul 07 10:32:05.242188 ip-10-0-135-34 kubenswrapper[2578]: I0707 10:32:05.242117 2578 kuberuntime_container.go:872] "Killing container with a grace period" pod="openshift-console/console-58f756fcd7-rcqbc" podUID="09a1d515-7b44-4cd8-a7d0-7683031291ed" containerName="console" containerID="cri-o://29632f0c75d88767f31810d1395613f4d6f9f8acb594c463d634fdab1d819064" gracePeriod=15 Jul 07 10:32:05.488681 ip-10-0-135-34 kubenswrapper[2578]: I0707 10:32:05.488658 2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-58f756fcd7-rcqbc" Jul 07 10:32:05.621893 ip-10-0-135-34 kubenswrapper[2578]: I0707 10:32:05.621860 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/09a1d515-7b44-4cd8-a7d0-7683031291ed-trusted-ca-bundle\") pod \"09a1d515-7b44-4cd8-a7d0-7683031291ed\" (UID: \"09a1d515-7b44-4cd8-a7d0-7683031291ed\") " Jul 07 10:32:05.622085 ip-10-0-135-34 kubenswrapper[2578]: I0707 10:32:05.621905 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/09a1d515-7b44-4cd8-a7d0-7683031291ed-service-ca\") pod \"09a1d515-7b44-4cd8-a7d0-7683031291ed\" (UID: \"09a1d515-7b44-4cd8-a7d0-7683031291ed\") " Jul 07 10:32:05.622085 ip-10-0-135-34 kubenswrapper[2578]: I0707 10:32:05.621944 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/09a1d515-7b44-4cd8-a7d0-7683031291ed-oauth-serving-cert\") pod \"09a1d515-7b44-4cd8-a7d0-7683031291ed\" (UID: \"09a1d515-7b44-4cd8-a7d0-7683031291ed\") " Jul 07 10:32:05.622085 ip-10-0-135-34 kubenswrapper[2578]: I0707 10:32:05.622059 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/09a1d515-7b44-4cd8-a7d0-7683031291ed-console-config\") pod \"09a1d515-7b44-4cd8-a7d0-7683031291ed\" (UID: \"09a1d515-7b44-4cd8-a7d0-7683031291ed\") " Jul 07 10:32:05.622268 ip-10-0-135-34 kubenswrapper[2578]: I0707 10:32:05.622104 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/09a1d515-7b44-4cd8-a7d0-7683031291ed-console-serving-cert\") pod \"09a1d515-7b44-4cd8-a7d0-7683031291ed\" (UID: \"09a1d515-7b44-4cd8-a7d0-7683031291ed\") " Jul 07 10:32:05.622268 ip-10-0-135-34 kubenswrapper[2578]: I0707 10:32:05.622147 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/09a1d515-7b44-4cd8-a7d0-7683031291ed-console-oauth-config\") pod \"09a1d515-7b44-4cd8-a7d0-7683031291ed\" (UID: \"09a1d515-7b44-4cd8-a7d0-7683031291ed\") " Jul 07 10:32:05.622268 ip-10-0-135-34 kubenswrapper[2578]: I0707 10:32:05.622175 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-vfstj\" (UniqueName: \"kubernetes.io/projected/09a1d515-7b44-4cd8-a7d0-7683031291ed-kube-api-access-vfstj\") pod \"09a1d515-7b44-4cd8-a7d0-7683031291ed\" (UID: \"09a1d515-7b44-4cd8-a7d0-7683031291ed\") " Jul 07 10:32:05.622419 ip-10-0-135-34 kubenswrapper[2578]: I0707 10:32:05.622380 2578 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/09a1d515-7b44-4cd8-a7d0-7683031291ed-trusted-ca-bundle" (OuterVolumeSpecName: "trusted-ca-bundle") pod "09a1d515-7b44-4cd8-a7d0-7683031291ed" (UID: "09a1d515-7b44-4cd8-a7d0-7683031291ed"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 07 10:32:05.622419 ip-10-0-135-34 kubenswrapper[2578]: I0707 10:32:05.622406 2578 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/09a1d515-7b44-4cd8-a7d0-7683031291ed-service-ca" (OuterVolumeSpecName: "service-ca") pod "09a1d515-7b44-4cd8-a7d0-7683031291ed" (UID: "09a1d515-7b44-4cd8-a7d0-7683031291ed"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 07 10:32:05.622522 ip-10-0-135-34 kubenswrapper[2578]: I0707 10:32:05.622503 2578 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/09a1d515-7b44-4cd8-a7d0-7683031291ed-oauth-serving-cert" (OuterVolumeSpecName: "oauth-serving-cert") pod "09a1d515-7b44-4cd8-a7d0-7683031291ed" (UID: "09a1d515-7b44-4cd8-a7d0-7683031291ed"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 07 10:32:05.622566 ip-10-0-135-34 kubenswrapper[2578]: I0707 10:32:05.622513 2578 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/09a1d515-7b44-4cd8-a7d0-7683031291ed-console-config" (OuterVolumeSpecName: "console-config") pod "09a1d515-7b44-4cd8-a7d0-7683031291ed" (UID: "09a1d515-7b44-4cd8-a7d0-7683031291ed"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 07 10:32:05.624474 ip-10-0-135-34 kubenswrapper[2578]: I0707 10:32:05.624442 2578 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/09a1d515-7b44-4cd8-a7d0-7683031291ed-console-oauth-config" (OuterVolumeSpecName: "console-oauth-config") pod "09a1d515-7b44-4cd8-a7d0-7683031291ed" (UID: "09a1d515-7b44-4cd8-a7d0-7683031291ed"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 07 10:32:05.624474 ip-10-0-135-34 kubenswrapper[2578]: I0707 10:32:05.624451 2578 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/09a1d515-7b44-4cd8-a7d0-7683031291ed-kube-api-access-vfstj" (OuterVolumeSpecName: "kube-api-access-vfstj") pod "09a1d515-7b44-4cd8-a7d0-7683031291ed" (UID: "09a1d515-7b44-4cd8-a7d0-7683031291ed"). InnerVolumeSpecName "kube-api-access-vfstj". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 07 10:32:05.624474 ip-10-0-135-34 kubenswrapper[2578]: I0707 10:32:05.624462 2578 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/09a1d515-7b44-4cd8-a7d0-7683031291ed-console-serving-cert" (OuterVolumeSpecName: "console-serving-cert") pod "09a1d515-7b44-4cd8-a7d0-7683031291ed" (UID: "09a1d515-7b44-4cd8-a7d0-7683031291ed"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 07 10:32:05.723130 ip-10-0-135-34 kubenswrapper[2578]: I0707 10:32:05.723089 2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vfstj\" (UniqueName: \"kubernetes.io/projected/09a1d515-7b44-4cd8-a7d0-7683031291ed-kube-api-access-vfstj\") on node \"ip-10-0-135-34.ec2.internal\" DevicePath \"\"" Jul 07 10:32:05.723130 ip-10-0-135-34 kubenswrapper[2578]: I0707 10:32:05.723121 2578 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/09a1d515-7b44-4cd8-a7d0-7683031291ed-trusted-ca-bundle\") on node \"ip-10-0-135-34.ec2.internal\" DevicePath \"\"" Jul 07 10:32:05.723130 ip-10-0-135-34 kubenswrapper[2578]: I0707 10:32:05.723133 2578 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/09a1d515-7b44-4cd8-a7d0-7683031291ed-service-ca\") on node \"ip-10-0-135-34.ec2.internal\" DevicePath \"\"" Jul 07 10:32:05.723361 ip-10-0-135-34 kubenswrapper[2578]: I0707 10:32:05.723145 2578 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/09a1d515-7b44-4cd8-a7d0-7683031291ed-oauth-serving-cert\") on node \"ip-10-0-135-34.ec2.internal\" DevicePath \"\"" Jul 07 10:32:05.723361 ip-10-0-135-34 kubenswrapper[2578]: I0707 10:32:05.723157 2578 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/09a1d515-7b44-4cd8-a7d0-7683031291ed-console-config\") on node \"ip-10-0-135-34.ec2.internal\" DevicePath \"\"" Jul 07 10:32:05.723361 ip-10-0-135-34 kubenswrapper[2578]: I0707 10:32:05.723168 2578 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/09a1d515-7b44-4cd8-a7d0-7683031291ed-console-serving-cert\") on node \"ip-10-0-135-34.ec2.internal\" DevicePath \"\"" Jul 07 10:32:05.723361 ip-10-0-135-34 kubenswrapper[2578]: I0707 10:32:05.723180 2578 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/09a1d515-7b44-4cd8-a7d0-7683031291ed-console-oauth-config\") on node \"ip-10-0-135-34.ec2.internal\" DevicePath \"\"" Jul 07 10:32:06.023632 ip-10-0-135-34 kubenswrapper[2578]: I0707 10:32:06.023543 2578 generic.go:356] "Generic (PLEG): container finished" podID="09a1d515-7b44-4cd8-a7d0-7683031291ed" containerID="29632f0c75d88767f31810d1395613f4d6f9f8acb594c463d634fdab1d819064" exitCode=0 Jul 07 10:32:06.023632 ip-10-0-135-34 kubenswrapper[2578]: I0707 10:32:06.023595 2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-58f756fcd7-rcqbc" Jul 07 10:32:06.023632 ip-10-0-135-34 kubenswrapper[2578]: I0707 10:32:06.023601 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-58f756fcd7-rcqbc" event={"ID":"09a1d515-7b44-4cd8-a7d0-7683031291ed","Type":"ContainerDied","Data":"29632f0c75d88767f31810d1395613f4d6f9f8acb594c463d634fdab1d819064"} Jul 07 10:32:06.023863 ip-10-0-135-34 kubenswrapper[2578]: I0707 10:32:06.023646 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-58f756fcd7-rcqbc" event={"ID":"09a1d515-7b44-4cd8-a7d0-7683031291ed","Type":"ContainerDied","Data":"70483a243110075ea1d9803d27b58df636e318387273288d47830c3731ce6395"} Jul 07 10:32:06.023863 ip-10-0-135-34 kubenswrapper[2578]: I0707 10:32:06.023665 2578 scope.go:117] "RemoveContainer" containerID="29632f0c75d88767f31810d1395613f4d6f9f8acb594c463d634fdab1d819064" Jul 07 10:32:06.031887 ip-10-0-135-34 kubenswrapper[2578]: I0707 10:32:06.031864 2578 scope.go:117] "RemoveContainer" containerID="29632f0c75d88767f31810d1395613f4d6f9f8acb594c463d634fdab1d819064" Jul 07 10:32:06.032152 ip-10-0-135-34 kubenswrapper[2578]: E0707 10:32:06.032131 2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"29632f0c75d88767f31810d1395613f4d6f9f8acb594c463d634fdab1d819064\": container with ID starting with 29632f0c75d88767f31810d1395613f4d6f9f8acb594c463d634fdab1d819064 not found: ID does not exist" containerID="29632f0c75d88767f31810d1395613f4d6f9f8acb594c463d634fdab1d819064" Jul 07 10:32:06.032220 ip-10-0-135-34 kubenswrapper[2578]: I0707 10:32:06.032159 2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"29632f0c75d88767f31810d1395613f4d6f9f8acb594c463d634fdab1d819064"} err="failed to get container status \"29632f0c75d88767f31810d1395613f4d6f9f8acb594c463d634fdab1d819064\": rpc error: code = NotFound desc = could not find container \"29632f0c75d88767f31810d1395613f4d6f9f8acb594c463d634fdab1d819064\": container with ID starting with 29632f0c75d88767f31810d1395613f4d6f9f8acb594c463d634fdab1d819064 not found: ID does not exist" Jul 07 10:32:06.043861 ip-10-0-135-34 kubenswrapper[2578]: I0707 10:32:06.043831 2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-58f756fcd7-rcqbc"] Jul 07 10:32:06.049787 ip-10-0-135-34 kubenswrapper[2578]: I0707 10:32:06.049745 2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-58f756fcd7-rcqbc"] Jul 07 10:32:07.820712 ip-10-0-135-34 kubenswrapper[2578]: I0707 10:32:07.820679 2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="09a1d515-7b44-4cd8-a7d0-7683031291ed" path="/var/lib/kubelet/pods/09a1d515-7b44-4cd8-a7d0-7683031291ed/volumes" Jul 07 10:32:11.754399 ip-10-0-135-34 kubenswrapper[2578]: I0707 10:32:11.754373 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-qhhw7_aca7528d-ddbe-4282-979f-1b496011f0e1/ovn-acl-logging/0.log" Jul 07 10:32:11.757282 ip-10-0-135-34 kubenswrapper[2578]: I0707 10:32:11.757258 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-qhhw7_aca7528d-ddbe-4282-979f-1b496011f0e1/ovn-acl-logging/0.log" Jul 07 10:32:11.760875 ip-10-0-135-34 kubenswrapper[2578]: I0707 10:32:11.760839 2578 kubelet.go:1648] "Image garbage collection succeeded" Jul 07 10:32:16.938940 ip-10-0-135-34 kubenswrapper[2578]: I0707 10:32:16.938904 2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540jk84n"] Jul 07 10:32:16.941192 ip-10-0-135-34 kubenswrapper[2578]: I0707 10:32:16.939202 2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="09a1d515-7b44-4cd8-a7d0-7683031291ed" containerName="console" Jul 07 10:32:16.941192 ip-10-0-135-34 kubenswrapper[2578]: I0707 10:32:16.939214 2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="09a1d515-7b44-4cd8-a7d0-7683031291ed" containerName="console" Jul 07 10:32:16.941192 ip-10-0-135-34 kubenswrapper[2578]: I0707 10:32:16.939264 2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="09a1d515-7b44-4cd8-a7d0-7683031291ed" containerName="console" Jul 07 10:32:16.942056 ip-10-0-135-34 kubenswrapper[2578]: I0707 10:32:16.942040 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540jk84n" Jul 07 10:32:16.944992 ip-10-0-135-34 kubenswrapper[2578]: I0707 10:32:16.944971 2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"openshift-service-ca.crt\"" Jul 07 10:32:16.945551 ip-10-0-135-34 kubenswrapper[2578]: I0707 10:32:16.945533 2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"kube-root-ca.crt\"" Jul 07 10:32:16.946425 ip-10-0-135-34 kubenswrapper[2578]: I0707 10:32:16.946408 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"default-dockercfg-dnl5z\"" Jul 07 10:32:16.955054 ip-10-0-135-34 kubenswrapper[2578]: I0707 10:32:16.955026 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540jk84n"] Jul 07 10:32:17.106306 ip-10-0-135-34 kubenswrapper[2578]: I0707 10:32:17.106272 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/67fae9ba-ad51-43f9-a1cf-feb1e858ea85-bundle\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540jk84n\" (UID: \"67fae9ba-ad51-43f9-a1cf-feb1e858ea85\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540jk84n" Jul 07 10:32:17.106497 ip-10-0-135-34 kubenswrapper[2578]: I0707 10:32:17.106328 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/67fae9ba-ad51-43f9-a1cf-feb1e858ea85-util\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540jk84n\" (UID: \"67fae9ba-ad51-43f9-a1cf-feb1e858ea85\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540jk84n" Jul 07 10:32:17.106497 ip-10-0-135-34 kubenswrapper[2578]: I0707 10:32:17.106354 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lz52g\" (UniqueName: \"kubernetes.io/projected/67fae9ba-ad51-43f9-a1cf-feb1e858ea85-kube-api-access-lz52g\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540jk84n\" (UID: \"67fae9ba-ad51-43f9-a1cf-feb1e858ea85\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540jk84n" Jul 07 10:32:17.206722 ip-10-0-135-34 kubenswrapper[2578]: I0707 10:32:17.206640 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/67fae9ba-ad51-43f9-a1cf-feb1e858ea85-bundle\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540jk84n\" (UID: \"67fae9ba-ad51-43f9-a1cf-feb1e858ea85\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540jk84n" Jul 07 10:32:17.206722 ip-10-0-135-34 kubenswrapper[2578]: I0707 10:32:17.206698 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/67fae9ba-ad51-43f9-a1cf-feb1e858ea85-util\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540jk84n\" (UID: \"67fae9ba-ad51-43f9-a1cf-feb1e858ea85\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540jk84n" Jul 07 10:32:17.206879 ip-10-0-135-34 kubenswrapper[2578]: I0707 10:32:17.206821 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lz52g\" (UniqueName: \"kubernetes.io/projected/67fae9ba-ad51-43f9-a1cf-feb1e858ea85-kube-api-access-lz52g\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540jk84n\" (UID: \"67fae9ba-ad51-43f9-a1cf-feb1e858ea85\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540jk84n" Jul 07 10:32:17.207048 ip-10-0-135-34 kubenswrapper[2578]: I0707 10:32:17.207034 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/67fae9ba-ad51-43f9-a1cf-feb1e858ea85-util\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540jk84n\" (UID: \"67fae9ba-ad51-43f9-a1cf-feb1e858ea85\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540jk84n" Jul 07 10:32:17.207092 ip-10-0-135-34 kubenswrapper[2578]: I0707 10:32:17.207076 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/67fae9ba-ad51-43f9-a1cf-feb1e858ea85-bundle\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540jk84n\" (UID: \"67fae9ba-ad51-43f9-a1cf-feb1e858ea85\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540jk84n" Jul 07 10:32:17.219758 ip-10-0-135-34 kubenswrapper[2578]: I0707 10:32:17.219720 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lz52g\" (UniqueName: \"kubernetes.io/projected/67fae9ba-ad51-43f9-a1cf-feb1e858ea85-kube-api-access-lz52g\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540jk84n\" (UID: \"67fae9ba-ad51-43f9-a1cf-feb1e858ea85\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540jk84n" Jul 07 10:32:17.251380 ip-10-0-135-34 kubenswrapper[2578]: I0707 10:32:17.251342 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540jk84n" Jul 07 10:32:17.389888 ip-10-0-135-34 kubenswrapper[2578]: I0707 10:32:17.389856 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540jk84n"] Jul 07 10:32:17.393384 ip-10-0-135-34 kubenswrapper[2578]: W0707 10:32:17.393354 2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod67fae9ba_ad51_43f9_a1cf_feb1e858ea85.slice/crio-262869b5ab72cbc7c50d1af519f584f6768b0d0a1c5e8d1cb95f14f22146b515 WatchSource:0}: Error finding container 262869b5ab72cbc7c50d1af519f584f6768b0d0a1c5e8d1cb95f14f22146b515: Status 404 returned error can't find the container with id 262869b5ab72cbc7c50d1af519f584f6768b0d0a1c5e8d1cb95f14f22146b515 Jul 07 10:32:17.395455 ip-10-0-135-34 kubenswrapper[2578]: I0707 10:32:17.395434 2578 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Jul 07 10:32:18.066023 ip-10-0-135-34 kubenswrapper[2578]: I0707 10:32:18.065982 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540jk84n" event={"ID":"67fae9ba-ad51-43f9-a1cf-feb1e858ea85","Type":"ContainerStarted","Data":"262869b5ab72cbc7c50d1af519f584f6768b0d0a1c5e8d1cb95f14f22146b515"} Jul 07 10:32:25.096496 ip-10-0-135-34 kubenswrapper[2578]: I0707 10:32:25.096459 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540jk84n" event={"ID":"67fae9ba-ad51-43f9-a1cf-feb1e858ea85","Type":"ContainerStarted","Data":"dd0822d5a92baa86868e00b6349cb0b3acd156202b17c381fee92c67d0819d0a"} Jul 07 10:32:26.100940 ip-10-0-135-34 kubenswrapper[2578]: I0707 10:32:26.100903 2578 generic.go:356] "Generic (PLEG): container finished" podID="67fae9ba-ad51-43f9-a1cf-feb1e858ea85" containerID="dd0822d5a92baa86868e00b6349cb0b3acd156202b17c381fee92c67d0819d0a" exitCode=0 Jul 07 10:32:26.101290 ip-10-0-135-34 kubenswrapper[2578]: I0707 10:32:26.100985 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540jk84n" event={"ID":"67fae9ba-ad51-43f9-a1cf-feb1e858ea85","Type":"ContainerDied","Data":"dd0822d5a92baa86868e00b6349cb0b3acd156202b17c381fee92c67d0819d0a"} Jul 07 10:32:29.114607 ip-10-0-135-34 kubenswrapper[2578]: I0707 10:32:29.114573 2578 generic.go:356] "Generic (PLEG): container finished" podID="67fae9ba-ad51-43f9-a1cf-feb1e858ea85" containerID="a1144a35d7f28fa25c44c732f3cdcb38cba572be04867bb5f55485ea4c7e1c42" exitCode=0 Jul 07 10:32:29.115069 ip-10-0-135-34 kubenswrapper[2578]: I0707 10:32:29.114642 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540jk84n" event={"ID":"67fae9ba-ad51-43f9-a1cf-feb1e858ea85","Type":"ContainerDied","Data":"a1144a35d7f28fa25c44c732f3cdcb38cba572be04867bb5f55485ea4c7e1c42"} Jul 07 10:32:36.148283 ip-10-0-135-34 kubenswrapper[2578]: I0707 10:32:36.148249 2578 generic.go:356] "Generic (PLEG): container finished" podID="67fae9ba-ad51-43f9-a1cf-feb1e858ea85" containerID="f6bba49db5fc1771576e7c95cf66f048a4121f2c7434f5f85466fc73d1515398" exitCode=0 Jul 07 10:32:36.148646 ip-10-0-135-34 kubenswrapper[2578]: I0707 10:32:36.148291 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540jk84n" event={"ID":"67fae9ba-ad51-43f9-a1cf-feb1e858ea85","Type":"ContainerDied","Data":"f6bba49db5fc1771576e7c95cf66f048a4121f2c7434f5f85466fc73d1515398"} Jul 07 10:32:37.282058 ip-10-0-135-34 kubenswrapper[2578]: I0707 10:32:37.282024 2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540jk84n" Jul 07 10:32:37.372348 ip-10-0-135-34 kubenswrapper[2578]: I0707 10:32:37.372311 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-lz52g\" (UniqueName: \"kubernetes.io/projected/67fae9ba-ad51-43f9-a1cf-feb1e858ea85-kube-api-access-lz52g\") pod \"67fae9ba-ad51-43f9-a1cf-feb1e858ea85\" (UID: \"67fae9ba-ad51-43f9-a1cf-feb1e858ea85\") " Jul 07 10:32:37.372348 ip-10-0-135-34 kubenswrapper[2578]: I0707 10:32:37.372357 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/67fae9ba-ad51-43f9-a1cf-feb1e858ea85-util\") pod \"67fae9ba-ad51-43f9-a1cf-feb1e858ea85\" (UID: \"67fae9ba-ad51-43f9-a1cf-feb1e858ea85\") " Jul 07 10:32:37.372546 ip-10-0-135-34 kubenswrapper[2578]: I0707 10:32:37.372401 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/67fae9ba-ad51-43f9-a1cf-feb1e858ea85-bundle\") pod \"67fae9ba-ad51-43f9-a1cf-feb1e858ea85\" (UID: \"67fae9ba-ad51-43f9-a1cf-feb1e858ea85\") " Jul 07 10:32:37.373072 ip-10-0-135-34 kubenswrapper[2578]: I0707 10:32:37.373043 2578 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/67fae9ba-ad51-43f9-a1cf-feb1e858ea85-bundle" (OuterVolumeSpecName: "bundle") pod "67fae9ba-ad51-43f9-a1cf-feb1e858ea85" (UID: "67fae9ba-ad51-43f9-a1cf-feb1e858ea85"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 07 10:32:37.374734 ip-10-0-135-34 kubenswrapper[2578]: I0707 10:32:37.374706 2578 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/67fae9ba-ad51-43f9-a1cf-feb1e858ea85-kube-api-access-lz52g" (OuterVolumeSpecName: "kube-api-access-lz52g") pod "67fae9ba-ad51-43f9-a1cf-feb1e858ea85" (UID: "67fae9ba-ad51-43f9-a1cf-feb1e858ea85"). InnerVolumeSpecName "kube-api-access-lz52g". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 07 10:32:37.377591 ip-10-0-135-34 kubenswrapper[2578]: I0707 10:32:37.377570 2578 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/67fae9ba-ad51-43f9-a1cf-feb1e858ea85-util" (OuterVolumeSpecName: "util") pod "67fae9ba-ad51-43f9-a1cf-feb1e858ea85" (UID: "67fae9ba-ad51-43f9-a1cf-feb1e858ea85"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 07 10:32:37.473726 ip-10-0-135-34 kubenswrapper[2578]: I0707 10:32:37.473630 2578 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/67fae9ba-ad51-43f9-a1cf-feb1e858ea85-util\") on node \"ip-10-0-135-34.ec2.internal\" DevicePath \"\"" Jul 07 10:32:37.473726 ip-10-0-135-34 kubenswrapper[2578]: I0707 10:32:37.473662 2578 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/67fae9ba-ad51-43f9-a1cf-feb1e858ea85-bundle\") on node \"ip-10-0-135-34.ec2.internal\" DevicePath \"\"" Jul 07 10:32:37.473726 ip-10-0-135-34 kubenswrapper[2578]: I0707 10:32:37.473675 2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lz52g\" (UniqueName: \"kubernetes.io/projected/67fae9ba-ad51-43f9-a1cf-feb1e858ea85-kube-api-access-lz52g\") on node \"ip-10-0-135-34.ec2.internal\" DevicePath \"\"" Jul 07 10:32:38.157182 ip-10-0-135-34 kubenswrapper[2578]: I0707 10:32:38.157151 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540jk84n" event={"ID":"67fae9ba-ad51-43f9-a1cf-feb1e858ea85","Type":"ContainerDied","Data":"262869b5ab72cbc7c50d1af519f584f6768b0d0a1c5e8d1cb95f14f22146b515"} Jul 07 10:32:38.157182 ip-10-0-135-34 kubenswrapper[2578]: I0707 10:32:38.157181 2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540jk84n" Jul 07 10:32:38.157375 ip-10-0-135-34 kubenswrapper[2578]: I0707 10:32:38.157198 2578 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="262869b5ab72cbc7c50d1af519f584f6768b0d0a1c5e8d1cb95f14f22146b515" Jul 07 10:32:40.124362 ip-10-0-135-34 kubenswrapper[2578]: I0707 10:32:40.124324 2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-859f485f68-m2mxt"] Jul 07 10:32:40.124832 ip-10-0-135-34 kubenswrapper[2578]: I0707 10:32:40.124621 2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="67fae9ba-ad51-43f9-a1cf-feb1e858ea85" containerName="util" Jul 07 10:32:40.124832 ip-10-0-135-34 kubenswrapper[2578]: I0707 10:32:40.124631 2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="67fae9ba-ad51-43f9-a1cf-feb1e858ea85" containerName="util" Jul 07 10:32:40.124832 ip-10-0-135-34 kubenswrapper[2578]: I0707 10:32:40.124643 2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="67fae9ba-ad51-43f9-a1cf-feb1e858ea85" containerName="extract" Jul 07 10:32:40.124832 ip-10-0-135-34 kubenswrapper[2578]: I0707 10:32:40.124648 2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="67fae9ba-ad51-43f9-a1cf-feb1e858ea85" containerName="extract" Jul 07 10:32:40.124832 ip-10-0-135-34 kubenswrapper[2578]: I0707 10:32:40.124662 2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="67fae9ba-ad51-43f9-a1cf-feb1e858ea85" containerName="pull" Jul 07 10:32:40.124832 ip-10-0-135-34 kubenswrapper[2578]: I0707 10:32:40.124667 2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="67fae9ba-ad51-43f9-a1cf-feb1e858ea85" containerName="pull" Jul 07 10:32:40.124832 ip-10-0-135-34 kubenswrapper[2578]: I0707 10:32:40.124710 2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="67fae9ba-ad51-43f9-a1cf-feb1e858ea85" containerName="extract" Jul 07 10:32:40.156757 ip-10-0-135-34 kubenswrapper[2578]: I0707 10:32:40.156719 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-859f485f68-m2mxt"] Jul 07 10:32:40.156939 ip-10-0-135-34 kubenswrapper[2578]: I0707 10:32:40.156835 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-859f485f68-m2mxt" Jul 07 10:32:40.160997 ip-10-0-135-34 kubenswrapper[2578]: I0707 10:32:40.160957 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"managed-serviceaccount-dockercfg-ldll4\"" Jul 07 10:32:40.161202 ip-10-0-135-34 kubenswrapper[2578]: I0707 10:32:40.161015 2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"open-cluster-management-agent-addon\"/\"openshift-service-ca.crt\"" Jul 07 10:32:40.161202 ip-10-0-135-34 kubenswrapper[2578]: I0707 10:32:40.160966 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"open-cluster-management-image-pull-credentials\"" Jul 07 10:32:40.161202 ip-10-0-135-34 kubenswrapper[2578]: I0707 10:32:40.160966 2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"open-cluster-management-agent-addon\"/\"kube-root-ca.crt\"" Jul 07 10:32:40.161202 ip-10-0-135-34 kubenswrapper[2578]: I0707 10:32:40.161057 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"managed-serviceaccount-hub-kubeconfig\"" Jul 07 10:32:40.193900 ip-10-0-135-34 kubenswrapper[2578]: I0707 10:32:40.193866 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/e2ade962-cdf2-4348-b14c-8c0ec87dfc1f-hub-kubeconfig\") pod \"managed-serviceaccount-addon-agent-859f485f68-m2mxt\" (UID: \"e2ade962-cdf2-4348-b14c-8c0ec87dfc1f\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-859f485f68-m2mxt" Jul 07 10:32:40.194069 ip-10-0-135-34 kubenswrapper[2578]: I0707 10:32:40.193905 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zcwcf\" (UniqueName: \"kubernetes.io/projected/e2ade962-cdf2-4348-b14c-8c0ec87dfc1f-kube-api-access-zcwcf\") pod \"managed-serviceaccount-addon-agent-859f485f68-m2mxt\" (UID: \"e2ade962-cdf2-4348-b14c-8c0ec87dfc1f\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-859f485f68-m2mxt" Jul 07 10:32:40.210725 ip-10-0-135-34 kubenswrapper[2578]: I0707 10:32:40.210686 2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["open-cluster-management-agent-addon/klusterlet-addon-workmgr-8db7848f-4g968"] Jul 07 10:32:40.213049 ip-10-0-135-34 kubenswrapper[2578]: I0707 10:32:40.213033 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-8db7848f-4g968" Jul 07 10:32:40.216373 ip-10-0-135-34 kubenswrapper[2578]: I0707 10:32:40.216351 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"work-manager-hub-kubeconfig\"" Jul 07 10:32:40.226514 ip-10-0-135-34 kubenswrapper[2578]: I0707 10:32:40.226490 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/klusterlet-addon-workmgr-8db7848f-4g968"] Jul 07 10:32:40.272905 ip-10-0-135-34 kubenswrapper[2578]: I0707 10:32:40.272875 2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["open-cluster-management-agent-addon/cluster-proxy-proxy-agent-86bdf6857b-mzclv"] Jul 07 10:32:40.275112 ip-10-0-135-34 kubenswrapper[2578]: I0707 10:32:40.275095 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-86bdf6857b-mzclv" Jul 07 10:32:40.277897 ip-10-0-135-34 kubenswrapper[2578]: I0707 10:32:40.277872 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"cluster-proxy-open-cluster-management.io-proxy-agent-signer-client-cert\"" Jul 07 10:32:40.278184 ip-10-0-135-34 kubenswrapper[2578]: I0707 10:32:40.278166 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"cluster-proxy-ca\"" Jul 07 10:32:40.278274 ip-10-0-135-34 kubenswrapper[2578]: I0707 10:32:40.278257 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"cluster-proxy-hub-kubeconfig\"" Jul 07 10:32:40.278584 ip-10-0-135-34 kubenswrapper[2578]: I0707 10:32:40.278566 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"cluster-proxy-service-proxy-server-certificates\"" Jul 07 10:32:40.290394 ip-10-0-135-34 kubenswrapper[2578]: I0707 10:32:40.290370 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/cluster-proxy-proxy-agent-86bdf6857b-mzclv"] Jul 07 10:32:40.295111 ip-10-0-135-34 kubenswrapper[2578]: I0707 10:32:40.295087 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zcwcf\" (UniqueName: \"kubernetes.io/projected/e2ade962-cdf2-4348-b14c-8c0ec87dfc1f-kube-api-access-zcwcf\") pod \"managed-serviceaccount-addon-agent-859f485f68-m2mxt\" (UID: \"e2ade962-cdf2-4348-b14c-8c0ec87dfc1f\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-859f485f68-m2mxt" Jul 07 10:32:40.295241 ip-10-0-135-34 kubenswrapper[2578]: I0707 10:32:40.295125 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/98011804-d069-49a8-8f61-3af59050c935-tmp\") pod \"klusterlet-addon-workmgr-8db7848f-4g968\" (UID: \"98011804-d069-49a8-8f61-3af59050c935\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-8db7848f-4g968" Jul 07 10:32:40.295241 ip-10-0-135-34 kubenswrapper[2578]: I0707 10:32:40.295153 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-szq8v\" (UniqueName: \"kubernetes.io/projected/98011804-d069-49a8-8f61-3af59050c935-kube-api-access-szq8v\") pod \"klusterlet-addon-workmgr-8db7848f-4g968\" (UID: \"98011804-d069-49a8-8f61-3af59050c935\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-8db7848f-4g968" Jul 07 10:32:40.295241 ip-10-0-135-34 kubenswrapper[2578]: I0707 10:32:40.295195 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"klusterlet-config\" (UniqueName: \"kubernetes.io/secret/98011804-d069-49a8-8f61-3af59050c935-klusterlet-config\") pod \"klusterlet-addon-workmgr-8db7848f-4g968\" (UID: \"98011804-d069-49a8-8f61-3af59050c935\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-8db7848f-4g968" Jul 07 10:32:40.295418 ip-10-0-135-34 kubenswrapper[2578]: I0707 10:32:40.295253 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/e2ade962-cdf2-4348-b14c-8c0ec87dfc1f-hub-kubeconfig\") pod \"managed-serviceaccount-addon-agent-859f485f68-m2mxt\" (UID: \"e2ade962-cdf2-4348-b14c-8c0ec87dfc1f\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-859f485f68-m2mxt" Jul 07 10:32:40.297798 ip-10-0-135-34 kubenswrapper[2578]: I0707 10:32:40.297760 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/e2ade962-cdf2-4348-b14c-8c0ec87dfc1f-hub-kubeconfig\") pod \"managed-serviceaccount-addon-agent-859f485f68-m2mxt\" (UID: \"e2ade962-cdf2-4348-b14c-8c0ec87dfc1f\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-859f485f68-m2mxt" Jul 07 10:32:40.308806 ip-10-0-135-34 kubenswrapper[2578]: I0707 10:32:40.308753 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zcwcf\" (UniqueName: \"kubernetes.io/projected/e2ade962-cdf2-4348-b14c-8c0ec87dfc1f-kube-api-access-zcwcf\") pod \"managed-serviceaccount-addon-agent-859f485f68-m2mxt\" (UID: \"e2ade962-cdf2-4348-b14c-8c0ec87dfc1f\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-859f485f68-m2mxt" Jul 07 10:32:40.396247 ip-10-0-135-34 kubenswrapper[2578]: I0707 10:32:40.396145 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"klusterlet-config\" (UniqueName: \"kubernetes.io/secret/98011804-d069-49a8-8f61-3af59050c935-klusterlet-config\") pod \"klusterlet-addon-workmgr-8db7848f-4g968\" (UID: \"98011804-d069-49a8-8f61-3af59050c935\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-8db7848f-4g968" Jul 07 10:32:40.396247 ip-10-0-135-34 kubenswrapper[2578]: I0707 10:32:40.396201 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"additional-service-ca\" (UniqueName: \"kubernetes.io/configmap/1af1a7bb-2e46-4203-a06c-76ec4beac9ef-additional-service-ca\") pod \"cluster-proxy-proxy-agent-86bdf6857b-mzclv\" (UID: \"1af1a7bb-2e46-4203-a06c-76ec4beac9ef\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-86bdf6857b-mzclv" Jul 07 10:32:40.396247 ip-10-0-135-34 kubenswrapper[2578]: I0707 10:32:40.396229 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca\" (UniqueName: \"kubernetes.io/secret/1af1a7bb-2e46-4203-a06c-76ec4beac9ef-ca\") pod \"cluster-proxy-proxy-agent-86bdf6857b-mzclv\" (UID: \"1af1a7bb-2e46-4203-a06c-76ec4beac9ef\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-86bdf6857b-mzclv" Jul 07 10:32:40.396247 ip-10-0-135-34 kubenswrapper[2578]: I0707 10:32:40.396247 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/98011804-d069-49a8-8f61-3af59050c935-tmp\") pod \"klusterlet-addon-workmgr-8db7848f-4g968\" (UID: \"98011804-d069-49a8-8f61-3af59050c935\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-8db7848f-4g968" Jul 07 10:32:40.396582 ip-10-0-135-34 kubenswrapper[2578]: I0707 10:32:40.396268 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/1af1a7bb-2e46-4203-a06c-76ec4beac9ef-hub-kubeconfig\") pod \"cluster-proxy-proxy-agent-86bdf6857b-mzclv\" (UID: \"1af1a7bb-2e46-4203-a06c-76ec4beac9ef\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-86bdf6857b-mzclv" Jul 07 10:32:40.396582 ip-10-0-135-34 kubenswrapper[2578]: I0707 10:32:40.396292 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-szq8v\" (UniqueName: \"kubernetes.io/projected/98011804-d069-49a8-8f61-3af59050c935-kube-api-access-szq8v\") pod \"klusterlet-addon-workmgr-8db7848f-4g968\" (UID: \"98011804-d069-49a8-8f61-3af59050c935\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-8db7848f-4g968" Jul 07 10:32:40.396582 ip-10-0-135-34 kubenswrapper[2578]: I0707 10:32:40.396372 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hub\" (UniqueName: \"kubernetes.io/secret/1af1a7bb-2e46-4203-a06c-76ec4beac9ef-hub\") pod \"cluster-proxy-proxy-agent-86bdf6857b-mzclv\" (UID: \"1af1a7bb-2e46-4203-a06c-76ec4beac9ef\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-86bdf6857b-mzclv" Jul 07 10:32:40.396582 ip-10-0-135-34 kubenswrapper[2578]: I0707 10:32:40.396416 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-proxy-server-cert\" (UniqueName: \"kubernetes.io/secret/1af1a7bb-2e46-4203-a06c-76ec4beac9ef-service-proxy-server-cert\") pod \"cluster-proxy-proxy-agent-86bdf6857b-mzclv\" (UID: \"1af1a7bb-2e46-4203-a06c-76ec4beac9ef\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-86bdf6857b-mzclv" Jul 07 10:32:40.396582 ip-10-0-135-34 kubenswrapper[2578]: I0707 10:32:40.396444 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qnpkr\" (UniqueName: \"kubernetes.io/projected/1af1a7bb-2e46-4203-a06c-76ec4beac9ef-kube-api-access-qnpkr\") pod \"cluster-proxy-proxy-agent-86bdf6857b-mzclv\" (UID: \"1af1a7bb-2e46-4203-a06c-76ec4beac9ef\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-86bdf6857b-mzclv" Jul 07 10:32:40.396762 ip-10-0-135-34 kubenswrapper[2578]: I0707 10:32:40.396715 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/98011804-d069-49a8-8f61-3af59050c935-tmp\") pod \"klusterlet-addon-workmgr-8db7848f-4g968\" (UID: \"98011804-d069-49a8-8f61-3af59050c935\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-8db7848f-4g968" Jul 07 10:32:40.398978 ip-10-0-135-34 kubenswrapper[2578]: I0707 10:32:40.398958 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"klusterlet-config\" (UniqueName: \"kubernetes.io/secret/98011804-d069-49a8-8f61-3af59050c935-klusterlet-config\") pod \"klusterlet-addon-workmgr-8db7848f-4g968\" (UID: \"98011804-d069-49a8-8f61-3af59050c935\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-8db7848f-4g968" Jul 07 10:32:40.408326 ip-10-0-135-34 kubenswrapper[2578]: I0707 10:32:40.408294 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-szq8v\" (UniqueName: \"kubernetes.io/projected/98011804-d069-49a8-8f61-3af59050c935-kube-api-access-szq8v\") pod \"klusterlet-addon-workmgr-8db7848f-4g968\" (UID: \"98011804-d069-49a8-8f61-3af59050c935\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-8db7848f-4g968" Jul 07 10:32:40.483039 ip-10-0-135-34 kubenswrapper[2578]: I0707 10:32:40.482999 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-859f485f68-m2mxt" Jul 07 10:32:40.497261 ip-10-0-135-34 kubenswrapper[2578]: I0707 10:32:40.497231 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hub\" (UniqueName: \"kubernetes.io/secret/1af1a7bb-2e46-4203-a06c-76ec4beac9ef-hub\") pod \"cluster-proxy-proxy-agent-86bdf6857b-mzclv\" (UID: \"1af1a7bb-2e46-4203-a06c-76ec4beac9ef\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-86bdf6857b-mzclv" Jul 07 10:32:40.497410 ip-10-0-135-34 kubenswrapper[2578]: I0707 10:32:40.497273 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-proxy-server-cert\" (UniqueName: \"kubernetes.io/secret/1af1a7bb-2e46-4203-a06c-76ec4beac9ef-service-proxy-server-cert\") pod \"cluster-proxy-proxy-agent-86bdf6857b-mzclv\" (UID: \"1af1a7bb-2e46-4203-a06c-76ec4beac9ef\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-86bdf6857b-mzclv" Jul 07 10:32:40.497410 ip-10-0-135-34 kubenswrapper[2578]: I0707 10:32:40.497296 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qnpkr\" (UniqueName: \"kubernetes.io/projected/1af1a7bb-2e46-4203-a06c-76ec4beac9ef-kube-api-access-qnpkr\") pod \"cluster-proxy-proxy-agent-86bdf6857b-mzclv\" (UID: \"1af1a7bb-2e46-4203-a06c-76ec4beac9ef\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-86bdf6857b-mzclv" Jul 07 10:32:40.497410 ip-10-0-135-34 kubenswrapper[2578]: I0707 10:32:40.497343 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"additional-service-ca\" (UniqueName: \"kubernetes.io/configmap/1af1a7bb-2e46-4203-a06c-76ec4beac9ef-additional-service-ca\") pod \"cluster-proxy-proxy-agent-86bdf6857b-mzclv\" (UID: \"1af1a7bb-2e46-4203-a06c-76ec4beac9ef\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-86bdf6857b-mzclv" Jul 07 10:32:40.497410 ip-10-0-135-34 kubenswrapper[2578]: I0707 10:32:40.497373 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca\" (UniqueName: \"kubernetes.io/secret/1af1a7bb-2e46-4203-a06c-76ec4beac9ef-ca\") pod \"cluster-proxy-proxy-agent-86bdf6857b-mzclv\" (UID: \"1af1a7bb-2e46-4203-a06c-76ec4beac9ef\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-86bdf6857b-mzclv" Jul 07 10:32:40.497410 ip-10-0-135-34 kubenswrapper[2578]: I0707 10:32:40.497396 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/1af1a7bb-2e46-4203-a06c-76ec4beac9ef-hub-kubeconfig\") pod \"cluster-proxy-proxy-agent-86bdf6857b-mzclv\" (UID: \"1af1a7bb-2e46-4203-a06c-76ec4beac9ef\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-86bdf6857b-mzclv" Jul 07 10:32:40.498375 ip-10-0-135-34 kubenswrapper[2578]: I0707 10:32:40.498334 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"additional-service-ca\" (UniqueName: \"kubernetes.io/configmap/1af1a7bb-2e46-4203-a06c-76ec4beac9ef-additional-service-ca\") pod \"cluster-proxy-proxy-agent-86bdf6857b-mzclv\" (UID: \"1af1a7bb-2e46-4203-a06c-76ec4beac9ef\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-86bdf6857b-mzclv" Jul 07 10:32:40.499955 ip-10-0-135-34 kubenswrapper[2578]: I0707 10:32:40.499934 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ca\" (UniqueName: \"kubernetes.io/secret/1af1a7bb-2e46-4203-a06c-76ec4beac9ef-ca\") pod \"cluster-proxy-proxy-agent-86bdf6857b-mzclv\" (UID: \"1af1a7bb-2e46-4203-a06c-76ec4beac9ef\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-86bdf6857b-mzclv" Jul 07 10:32:40.499955 ip-10-0-135-34 kubenswrapper[2578]: I0707 10:32:40.499946 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hub\" (UniqueName: \"kubernetes.io/secret/1af1a7bb-2e46-4203-a06c-76ec4beac9ef-hub\") pod \"cluster-proxy-proxy-agent-86bdf6857b-mzclv\" (UID: \"1af1a7bb-2e46-4203-a06c-76ec4beac9ef\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-86bdf6857b-mzclv" Jul 07 10:32:40.500082 ip-10-0-135-34 kubenswrapper[2578]: I0707 10:32:40.499973 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-proxy-server-cert\" (UniqueName: \"kubernetes.io/secret/1af1a7bb-2e46-4203-a06c-76ec4beac9ef-service-proxy-server-cert\") pod \"cluster-proxy-proxy-agent-86bdf6857b-mzclv\" (UID: \"1af1a7bb-2e46-4203-a06c-76ec4beac9ef\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-86bdf6857b-mzclv" Jul 07 10:32:40.500243 ip-10-0-135-34 kubenswrapper[2578]: I0707 10:32:40.500226 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/1af1a7bb-2e46-4203-a06c-76ec4beac9ef-hub-kubeconfig\") pod \"cluster-proxy-proxy-agent-86bdf6857b-mzclv\" (UID: \"1af1a7bb-2e46-4203-a06c-76ec4beac9ef\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-86bdf6857b-mzclv" Jul 07 10:32:40.507682 ip-10-0-135-34 kubenswrapper[2578]: I0707 10:32:40.507661 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qnpkr\" (UniqueName: \"kubernetes.io/projected/1af1a7bb-2e46-4203-a06c-76ec4beac9ef-kube-api-access-qnpkr\") pod \"cluster-proxy-proxy-agent-86bdf6857b-mzclv\" (UID: \"1af1a7bb-2e46-4203-a06c-76ec4beac9ef\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-86bdf6857b-mzclv" Jul 07 10:32:40.522883 ip-10-0-135-34 kubenswrapper[2578]: I0707 10:32:40.522843 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-8db7848f-4g968" Jul 07 10:32:40.584664 ip-10-0-135-34 kubenswrapper[2578]: I0707 10:32:40.584323 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-86bdf6857b-mzclv" Jul 07 10:32:40.628887 ip-10-0-135-34 kubenswrapper[2578]: I0707 10:32:40.628807 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-859f485f68-m2mxt"] Jul 07 10:32:40.635117 ip-10-0-135-34 kubenswrapper[2578]: W0707 10:32:40.635008 2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pode2ade962_cdf2_4348_b14c_8c0ec87dfc1f.slice/crio-bdb59df9976790aaa06a09e36a44e0df60bee10ab492640a050f393b665342a7 WatchSource:0}: Error finding container bdb59df9976790aaa06a09e36a44e0df60bee10ab492640a050f393b665342a7: Status 404 returned error can't find the container with id bdb59df9976790aaa06a09e36a44e0df60bee10ab492640a050f393b665342a7 Jul 07 10:32:40.675640 ip-10-0-135-34 kubenswrapper[2578]: I0707 10:32:40.675608 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/klusterlet-addon-workmgr-8db7848f-4g968"] Jul 07 10:32:40.681396 ip-10-0-135-34 kubenswrapper[2578]: W0707 10:32:40.681367 2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod98011804_d069_49a8_8f61_3af59050c935.slice/crio-ed5f5717516486215e69360f4e6c69c2bca137af930b1258182a2dfea42241dc WatchSource:0}: Error finding container ed5f5717516486215e69360f4e6c69c2bca137af930b1258182a2dfea42241dc: Status 404 returned error can't find the container with id ed5f5717516486215e69360f4e6c69c2bca137af930b1258182a2dfea42241dc Jul 07 10:32:40.730667 ip-10-0-135-34 kubenswrapper[2578]: I0707 10:32:40.730634 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/cluster-proxy-proxy-agent-86bdf6857b-mzclv"] Jul 07 10:32:40.733336 ip-10-0-135-34 kubenswrapper[2578]: W0707 10:32:40.733307 2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod1af1a7bb_2e46_4203_a06c_76ec4beac9ef.slice/crio-8a6f70cc78be5adf72270599c4bda26fa0482a4d59007261bc1541455aade7b6 WatchSource:0}: Error finding container 8a6f70cc78be5adf72270599c4bda26fa0482a4d59007261bc1541455aade7b6: Status 404 returned error can't find the container with id 8a6f70cc78be5adf72270599c4bda26fa0482a4d59007261bc1541455aade7b6 Jul 07 10:32:41.169555 ip-10-0-135-34 kubenswrapper[2578]: I0707 10:32:41.169515 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-8db7848f-4g968" event={"ID":"98011804-d069-49a8-8f61-3af59050c935","Type":"ContainerStarted","Data":"ed5f5717516486215e69360f4e6c69c2bca137af930b1258182a2dfea42241dc"} Jul 07 10:32:41.170652 ip-10-0-135-34 kubenswrapper[2578]: I0707 10:32:41.170630 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-859f485f68-m2mxt" event={"ID":"e2ade962-cdf2-4348-b14c-8c0ec87dfc1f","Type":"ContainerStarted","Data":"bdb59df9976790aaa06a09e36a44e0df60bee10ab492640a050f393b665342a7"} Jul 07 10:32:41.171621 ip-10-0-135-34 kubenswrapper[2578]: I0707 10:32:41.171598 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-86bdf6857b-mzclv" event={"ID":"1af1a7bb-2e46-4203-a06c-76ec4beac9ef","Type":"ContainerStarted","Data":"8a6f70cc78be5adf72270599c4bda26fa0482a4d59007261bc1541455aade7b6"} Jul 07 10:32:47.204053 ip-10-0-135-34 kubenswrapper[2578]: I0707 10:32:47.204014 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-859f485f68-m2mxt" event={"ID":"e2ade962-cdf2-4348-b14c-8c0ec87dfc1f","Type":"ContainerStarted","Data":"57993f8a78435b655def2414210095c0807a40bbfed8fb7fb43d245322416b38"} Jul 07 10:32:47.206041 ip-10-0-135-34 kubenswrapper[2578]: I0707 10:32:47.206002 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-86bdf6857b-mzclv" event={"ID":"1af1a7bb-2e46-4203-a06c-76ec4beac9ef","Type":"ContainerStarted","Data":"3f4bfa5bb8fc0a153726147b25d2b0b959aab5e05067ef4401dbd46546a34690"} Jul 07 10:32:47.206187 ip-10-0-135-34 kubenswrapper[2578]: I0707 10:32:47.206047 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-86bdf6857b-mzclv" event={"ID":"1af1a7bb-2e46-4203-a06c-76ec4beac9ef","Type":"ContainerStarted","Data":"77c5288d328d91d965122df153f8b89b69a01516c460d6dac02a7eea7f1019bb"} Jul 07 10:32:47.206187 ip-10-0-135-34 kubenswrapper[2578]: I0707 10:32:47.206063 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-86bdf6857b-mzclv" event={"ID":"1af1a7bb-2e46-4203-a06c-76ec4beac9ef","Type":"ContainerStarted","Data":"14f65d94d2683eba012028c86078c3a9aac91d85298887c2445a3d17ebce23ca"} Jul 07 10:32:47.207310 ip-10-0-135-34 kubenswrapper[2578]: I0707 10:32:47.207283 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-8db7848f-4g968" event={"ID":"98011804-d069-49a8-8f61-3af59050c935","Type":"ContainerStarted","Data":"c75508db8fc1d71a7fe61322bf49127c4a3e6349efc0b6aab72045148710732b"} Jul 07 10:32:47.207481 ip-10-0-135-34 kubenswrapper[2578]: I0707 10:32:47.207466 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-8db7848f-4g968" Jul 07 10:32:47.209256 ip-10-0-135-34 kubenswrapper[2578]: I0707 10:32:47.209236 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-8db7848f-4g968" Jul 07 10:32:47.227092 ip-10-0-135-34 kubenswrapper[2578]: I0707 10:32:47.227036 2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-859f485f68-m2mxt" podStartSLOduration=1.563390965 podStartE2EDuration="7.227019102s" totalImagesPullingTime="5.663628137s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-07 10:32:40 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 10:32:47.226679241 +0000 UTC m=+335.898638787" watchObservedRunningTime="2026-07-07 10:32:47.227019102 +0000 UTC m=+335.898978687" Jul 07 10:32:47.290937 ip-10-0-135-34 kubenswrapper[2578]: I0707 10:32:47.290876 2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-86bdf6857b-mzclv" podStartSLOduration=1.7030786409999998 podStartE2EDuration="7.290854156s" totalImagesPullingTime="5.587775515s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-07 10:32:40 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 10:32:47.290027781 +0000 UTC m=+335.961987340" watchObservedRunningTime="2026-07-07 10:32:47.290854156 +0000 UTC m=+335.962813704" Jul 07 10:32:47.291159 ip-10-0-135-34 kubenswrapper[2578]: I0707 10:32:47.291139 2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-8db7848f-4g968" podStartSLOduration=1.649992865 podStartE2EDuration="7.291132388s" totalImagesPullingTime="5.641139523s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-07 10:32:40 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 10:32:47.256282659 +0000 UTC m=+335.928242232" watchObservedRunningTime="2026-07-07 10:32:47.291132388 +0000 UTC m=+335.963091932" Jul 07 10:32:49.886886 ip-10-0-135-34 kubenswrapper[2578]: I0707 10:32:49.886846 2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fnss6b"] Jul 07 10:32:49.889437 ip-10-0-135-34 kubenswrapper[2578]: I0707 10:32:49.889420 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fnss6b" Jul 07 10:32:49.892459 ip-10-0-135-34 kubenswrapper[2578]: I0707 10:32:49.892433 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"default-dockercfg-dnl5z\"" Jul 07 10:32:49.892459 ip-10-0-135-34 kubenswrapper[2578]: I0707 10:32:49.892456 2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"openshift-service-ca.crt\"" Jul 07 10:32:49.892637 ip-10-0-135-34 kubenswrapper[2578]: I0707 10:32:49.892474 2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"kube-root-ca.crt\"" Jul 07 10:32:49.910285 ip-10-0-135-34 kubenswrapper[2578]: I0707 10:32:49.910258 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fnss6b"] Jul 07 10:32:49.979638 ip-10-0-135-34 kubenswrapper[2578]: I0707 10:32:49.979595 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5w9hs\" (UniqueName: \"kubernetes.io/projected/c0f4da97-17fc-4c5a-bde0-ec06857263d4-kube-api-access-5w9hs\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fnss6b\" (UID: \"c0f4da97-17fc-4c5a-bde0-ec06857263d4\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fnss6b" Jul 07 10:32:49.979870 ip-10-0-135-34 kubenswrapper[2578]: I0707 10:32:49.979653 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/c0f4da97-17fc-4c5a-bde0-ec06857263d4-bundle\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fnss6b\" (UID: \"c0f4da97-17fc-4c5a-bde0-ec06857263d4\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fnss6b" Jul 07 10:32:49.979870 ip-10-0-135-34 kubenswrapper[2578]: I0707 10:32:49.979687 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/c0f4da97-17fc-4c5a-bde0-ec06857263d4-util\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fnss6b\" (UID: \"c0f4da97-17fc-4c5a-bde0-ec06857263d4\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fnss6b" Jul 07 10:32:50.080304 ip-10-0-135-34 kubenswrapper[2578]: I0707 10:32:50.080266 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/c0f4da97-17fc-4c5a-bde0-ec06857263d4-bundle\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fnss6b\" (UID: \"c0f4da97-17fc-4c5a-bde0-ec06857263d4\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fnss6b" Jul 07 10:32:50.080304 ip-10-0-135-34 kubenswrapper[2578]: I0707 10:32:50.080309 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/c0f4da97-17fc-4c5a-bde0-ec06857263d4-util\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fnss6b\" (UID: \"c0f4da97-17fc-4c5a-bde0-ec06857263d4\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fnss6b" Jul 07 10:32:50.080486 ip-10-0-135-34 kubenswrapper[2578]: I0707 10:32:50.080370 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5w9hs\" (UniqueName: \"kubernetes.io/projected/c0f4da97-17fc-4c5a-bde0-ec06857263d4-kube-api-access-5w9hs\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fnss6b\" (UID: \"c0f4da97-17fc-4c5a-bde0-ec06857263d4\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fnss6b" Jul 07 10:32:50.080660 ip-10-0-135-34 kubenswrapper[2578]: I0707 10:32:50.080639 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/c0f4da97-17fc-4c5a-bde0-ec06857263d4-bundle\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fnss6b\" (UID: \"c0f4da97-17fc-4c5a-bde0-ec06857263d4\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fnss6b" Jul 07 10:32:50.080736 ip-10-0-135-34 kubenswrapper[2578]: I0707 10:32:50.080696 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/c0f4da97-17fc-4c5a-bde0-ec06857263d4-util\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fnss6b\" (UID: \"c0f4da97-17fc-4c5a-bde0-ec06857263d4\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fnss6b" Jul 07 10:32:50.090451 ip-10-0-135-34 kubenswrapper[2578]: I0707 10:32:50.090417 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5w9hs\" (UniqueName: \"kubernetes.io/projected/c0f4da97-17fc-4c5a-bde0-ec06857263d4-kube-api-access-5w9hs\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fnss6b\" (UID: \"c0f4da97-17fc-4c5a-bde0-ec06857263d4\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fnss6b" Jul 07 10:32:50.198346 ip-10-0-135-34 kubenswrapper[2578]: I0707 10:32:50.198241 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fnss6b" Jul 07 10:32:50.341147 ip-10-0-135-34 kubenswrapper[2578]: I0707 10:32:50.341111 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fnss6b"] Jul 07 10:32:50.343981 ip-10-0-135-34 kubenswrapper[2578]: W0707 10:32:50.343951 2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc0f4da97_17fc_4c5a_bde0_ec06857263d4.slice/crio-0a1832131006a1ac4c5275eb283a49eed2773a53a212a63871010a57f38ce447 WatchSource:0}: Error finding container 0a1832131006a1ac4c5275eb283a49eed2773a53a212a63871010a57f38ce447: Status 404 returned error can't find the container with id 0a1832131006a1ac4c5275eb283a49eed2773a53a212a63871010a57f38ce447 Jul 07 10:32:51.228644 ip-10-0-135-34 kubenswrapper[2578]: I0707 10:32:51.228613 2578 generic.go:356] "Generic (PLEG): container finished" podID="c0f4da97-17fc-4c5a-bde0-ec06857263d4" containerID="1f02cd306864aa61636554edcd600c7c86a26d1209e36d7b99eb0a0af69b25bb" exitCode=0 Jul 07 10:32:51.229052 ip-10-0-135-34 kubenswrapper[2578]: I0707 10:32:51.228667 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fnss6b" event={"ID":"c0f4da97-17fc-4c5a-bde0-ec06857263d4","Type":"ContainerDied","Data":"1f02cd306864aa61636554edcd600c7c86a26d1209e36d7b99eb0a0af69b25bb"} Jul 07 10:32:51.229052 ip-10-0-135-34 kubenswrapper[2578]: I0707 10:32:51.228704 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fnss6b" event={"ID":"c0f4da97-17fc-4c5a-bde0-ec06857263d4","Type":"ContainerStarted","Data":"0a1832131006a1ac4c5275eb283a49eed2773a53a212a63871010a57f38ce447"} Jul 07 10:32:53.239610 ip-10-0-135-34 kubenswrapper[2578]: I0707 10:32:53.239568 2578 generic.go:356] "Generic (PLEG): container finished" podID="c0f4da97-17fc-4c5a-bde0-ec06857263d4" containerID="00722cab559295151573d9f6f2f142cf8b1271e2d1143024067820ce798b46e5" exitCode=0 Jul 07 10:32:53.240035 ip-10-0-135-34 kubenswrapper[2578]: I0707 10:32:53.239661 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fnss6b" event={"ID":"c0f4da97-17fc-4c5a-bde0-ec06857263d4","Type":"ContainerDied","Data":"00722cab559295151573d9f6f2f142cf8b1271e2d1143024067820ce798b46e5"} Jul 07 10:32:54.251316 ip-10-0-135-34 kubenswrapper[2578]: I0707 10:32:54.251273 2578 generic.go:356] "Generic (PLEG): container finished" podID="c0f4da97-17fc-4c5a-bde0-ec06857263d4" containerID="d3ec38ce95c1ba15a521e9adec9ecdc64fe7d2239027e797df4c600755cb5e7d" exitCode=0 Jul 07 10:32:54.251680 ip-10-0-135-34 kubenswrapper[2578]: I0707 10:32:54.251333 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fnss6b" event={"ID":"c0f4da97-17fc-4c5a-bde0-ec06857263d4","Type":"ContainerDied","Data":"d3ec38ce95c1ba15a521e9adec9ecdc64fe7d2239027e797df4c600755cb5e7d"} Jul 07 10:32:55.380128 ip-10-0-135-34 kubenswrapper[2578]: I0707 10:32:55.380103 2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fnss6b" Jul 07 10:32:55.529241 ip-10-0-135-34 kubenswrapper[2578]: I0707 10:32:55.529147 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/c0f4da97-17fc-4c5a-bde0-ec06857263d4-bundle\") pod \"c0f4da97-17fc-4c5a-bde0-ec06857263d4\" (UID: \"c0f4da97-17fc-4c5a-bde0-ec06857263d4\") " Jul 07 10:32:55.529241 ip-10-0-135-34 kubenswrapper[2578]: I0707 10:32:55.529196 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-5w9hs\" (UniqueName: \"kubernetes.io/projected/c0f4da97-17fc-4c5a-bde0-ec06857263d4-kube-api-access-5w9hs\") pod \"c0f4da97-17fc-4c5a-bde0-ec06857263d4\" (UID: \"c0f4da97-17fc-4c5a-bde0-ec06857263d4\") " Jul 07 10:32:55.529241 ip-10-0-135-34 kubenswrapper[2578]: I0707 10:32:55.529218 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/c0f4da97-17fc-4c5a-bde0-ec06857263d4-util\") pod \"c0f4da97-17fc-4c5a-bde0-ec06857263d4\" (UID: \"c0f4da97-17fc-4c5a-bde0-ec06857263d4\") " Jul 07 10:32:55.529541 ip-10-0-135-34 kubenswrapper[2578]: I0707 10:32:55.529518 2578 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c0f4da97-17fc-4c5a-bde0-ec06857263d4-bundle" (OuterVolumeSpecName: "bundle") pod "c0f4da97-17fc-4c5a-bde0-ec06857263d4" (UID: "c0f4da97-17fc-4c5a-bde0-ec06857263d4"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 07 10:32:55.531438 ip-10-0-135-34 kubenswrapper[2578]: I0707 10:32:55.531408 2578 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c0f4da97-17fc-4c5a-bde0-ec06857263d4-kube-api-access-5w9hs" (OuterVolumeSpecName: "kube-api-access-5w9hs") pod "c0f4da97-17fc-4c5a-bde0-ec06857263d4" (UID: "c0f4da97-17fc-4c5a-bde0-ec06857263d4"). InnerVolumeSpecName "kube-api-access-5w9hs". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 07 10:32:55.535117 ip-10-0-135-34 kubenswrapper[2578]: I0707 10:32:55.535091 2578 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c0f4da97-17fc-4c5a-bde0-ec06857263d4-util" (OuterVolumeSpecName: "util") pod "c0f4da97-17fc-4c5a-bde0-ec06857263d4" (UID: "c0f4da97-17fc-4c5a-bde0-ec06857263d4"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 07 10:32:55.630000 ip-10-0-135-34 kubenswrapper[2578]: I0707 10:32:55.629965 2578 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/c0f4da97-17fc-4c5a-bde0-ec06857263d4-bundle\") on node \"ip-10-0-135-34.ec2.internal\" DevicePath \"\"" Jul 07 10:32:55.630000 ip-10-0-135-34 kubenswrapper[2578]: I0707 10:32:55.629998 2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5w9hs\" (UniqueName: \"kubernetes.io/projected/c0f4da97-17fc-4c5a-bde0-ec06857263d4-kube-api-access-5w9hs\") on node \"ip-10-0-135-34.ec2.internal\" DevicePath \"\"" Jul 07 10:32:55.630000 ip-10-0-135-34 kubenswrapper[2578]: I0707 10:32:55.630010 2578 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/c0f4da97-17fc-4c5a-bde0-ec06857263d4-util\") on node \"ip-10-0-135-34.ec2.internal\" DevicePath \"\"" Jul 07 10:32:56.261547 ip-10-0-135-34 kubenswrapper[2578]: I0707 10:32:56.261514 2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fnss6b" Jul 07 10:32:56.261711 ip-10-0-135-34 kubenswrapper[2578]: I0707 10:32:56.261522 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fnss6b" event={"ID":"c0f4da97-17fc-4c5a-bde0-ec06857263d4","Type":"ContainerDied","Data":"0a1832131006a1ac4c5275eb283a49eed2773a53a212a63871010a57f38ce447"} Jul 07 10:32:56.261711 ip-10-0-135-34 kubenswrapper[2578]: I0707 10:32:56.261632 2578 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="0a1832131006a1ac4c5275eb283a49eed2773a53a212a63871010a57f38ce447" Jul 07 10:33:06.270686 ip-10-0-135-34 kubenswrapper[2578]: I0707 10:33:06.270602 2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afrbf6z"] Jul 07 10:33:06.271082 ip-10-0-135-34 kubenswrapper[2578]: I0707 10:33:06.270926 2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c0f4da97-17fc-4c5a-bde0-ec06857263d4" containerName="util" Jul 07 10:33:06.271082 ip-10-0-135-34 kubenswrapper[2578]: I0707 10:33:06.270939 2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="c0f4da97-17fc-4c5a-bde0-ec06857263d4" containerName="util" Jul 07 10:33:06.271082 ip-10-0-135-34 kubenswrapper[2578]: I0707 10:33:06.270954 2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c0f4da97-17fc-4c5a-bde0-ec06857263d4" containerName="pull" Jul 07 10:33:06.271082 ip-10-0-135-34 kubenswrapper[2578]: I0707 10:33:06.270960 2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="c0f4da97-17fc-4c5a-bde0-ec06857263d4" containerName="pull" Jul 07 10:33:06.271082 ip-10-0-135-34 kubenswrapper[2578]: I0707 10:33:06.270971 2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c0f4da97-17fc-4c5a-bde0-ec06857263d4" containerName="extract" Jul 07 10:33:06.271082 ip-10-0-135-34 kubenswrapper[2578]: I0707 10:33:06.270976 2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="c0f4da97-17fc-4c5a-bde0-ec06857263d4" containerName="extract" Jul 07 10:33:06.271082 ip-10-0-135-34 kubenswrapper[2578]: I0707 10:33:06.271031 2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="c0f4da97-17fc-4c5a-bde0-ec06857263d4" containerName="extract" Jul 07 10:33:06.274665 ip-10-0-135-34 kubenswrapper[2578]: I0707 10:33:06.274648 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afrbf6z" Jul 07 10:33:06.277489 ip-10-0-135-34 kubenswrapper[2578]: I0707 10:33:06.277465 2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"openshift-service-ca.crt\"" Jul 07 10:33:06.278435 ip-10-0-135-34 kubenswrapper[2578]: I0707 10:33:06.278414 2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"kube-root-ca.crt\"" Jul 07 10:33:06.278517 ip-10-0-135-34 kubenswrapper[2578]: I0707 10:33:06.278414 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"default-dockercfg-dnl5z\"" Jul 07 10:33:06.283350 ip-10-0-135-34 kubenswrapper[2578]: I0707 10:33:06.283320 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afrbf6z"] Jul 07 10:33:06.416452 ip-10-0-135-34 kubenswrapper[2578]: I0707 10:33:06.416417 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/fb918268-2c2b-4ea9-ba53-676b42b1cec7-util\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afrbf6z\" (UID: \"fb918268-2c2b-4ea9-ba53-676b42b1cec7\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afrbf6z" Jul 07 10:33:06.416452 ip-10-0-135-34 kubenswrapper[2578]: I0707 10:33:06.416458 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sw8q4\" (UniqueName: \"kubernetes.io/projected/fb918268-2c2b-4ea9-ba53-676b42b1cec7-kube-api-access-sw8q4\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afrbf6z\" (UID: \"fb918268-2c2b-4ea9-ba53-676b42b1cec7\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afrbf6z" Jul 07 10:33:06.416640 ip-10-0-135-34 kubenswrapper[2578]: I0707 10:33:06.416560 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/fb918268-2c2b-4ea9-ba53-676b42b1cec7-bundle\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afrbf6z\" (UID: \"fb918268-2c2b-4ea9-ba53-676b42b1cec7\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afrbf6z" Jul 07 10:33:06.518029 ip-10-0-135-34 kubenswrapper[2578]: I0707 10:33:06.517989 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/fb918268-2c2b-4ea9-ba53-676b42b1cec7-bundle\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afrbf6z\" (UID: \"fb918268-2c2b-4ea9-ba53-676b42b1cec7\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afrbf6z" Jul 07 10:33:06.518219 ip-10-0-135-34 kubenswrapper[2578]: I0707 10:33:06.518050 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/fb918268-2c2b-4ea9-ba53-676b42b1cec7-util\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afrbf6z\" (UID: \"fb918268-2c2b-4ea9-ba53-676b42b1cec7\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afrbf6z" Jul 07 10:33:06.518219 ip-10-0-135-34 kubenswrapper[2578]: I0707 10:33:06.518073 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sw8q4\" (UniqueName: \"kubernetes.io/projected/fb918268-2c2b-4ea9-ba53-676b42b1cec7-kube-api-access-sw8q4\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afrbf6z\" (UID: \"fb918268-2c2b-4ea9-ba53-676b42b1cec7\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afrbf6z" Jul 07 10:33:06.518359 ip-10-0-135-34 kubenswrapper[2578]: I0707 10:33:06.518333 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/fb918268-2c2b-4ea9-ba53-676b42b1cec7-bundle\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afrbf6z\" (UID: \"fb918268-2c2b-4ea9-ba53-676b42b1cec7\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afrbf6z" Jul 07 10:33:06.518414 ip-10-0-135-34 kubenswrapper[2578]: I0707 10:33:06.518371 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/fb918268-2c2b-4ea9-ba53-676b42b1cec7-util\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afrbf6z\" (UID: \"fb918268-2c2b-4ea9-ba53-676b42b1cec7\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afrbf6z" Jul 07 10:33:06.528324 ip-10-0-135-34 kubenswrapper[2578]: I0707 10:33:06.528265 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-sw8q4\" (UniqueName: \"kubernetes.io/projected/fb918268-2c2b-4ea9-ba53-676b42b1cec7-kube-api-access-sw8q4\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afrbf6z\" (UID: \"fb918268-2c2b-4ea9-ba53-676b42b1cec7\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afrbf6z" Jul 07 10:33:06.584311 ip-10-0-135-34 kubenswrapper[2578]: I0707 10:33:06.584279 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afrbf6z" Jul 07 10:33:06.739719 ip-10-0-135-34 kubenswrapper[2578]: I0707 10:33:06.739696 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afrbf6z"] Jul 07 10:33:06.741791 ip-10-0-135-34 kubenswrapper[2578]: W0707 10:33:06.741742 2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfb918268_2c2b_4ea9_ba53_676b42b1cec7.slice/crio-800c60337e021f234c0073cacf096521258ea17487a5fd23ba4650c7e251a796 WatchSource:0}: Error finding container 800c60337e021f234c0073cacf096521258ea17487a5fd23ba4650c7e251a796: Status 404 returned error can't find the container with id 800c60337e021f234c0073cacf096521258ea17487a5fd23ba4650c7e251a796 Jul 07 10:33:07.304993 ip-10-0-135-34 kubenswrapper[2578]: I0707 10:33:07.304959 2578 generic.go:356] "Generic (PLEG): container finished" podID="fb918268-2c2b-4ea9-ba53-676b42b1cec7" containerID="1cb36a3c21a6b3444b446d82b4cd43c91124097eb6cf1199c4221f76eae433b3" exitCode=0 Jul 07 10:33:07.305354 ip-10-0-135-34 kubenswrapper[2578]: I0707 10:33:07.305012 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afrbf6z" event={"ID":"fb918268-2c2b-4ea9-ba53-676b42b1cec7","Type":"ContainerDied","Data":"1cb36a3c21a6b3444b446d82b4cd43c91124097eb6cf1199c4221f76eae433b3"} Jul 07 10:33:07.305354 ip-10-0-135-34 kubenswrapper[2578]: I0707 10:33:07.305048 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afrbf6z" event={"ID":"fb918268-2c2b-4ea9-ba53-676b42b1cec7","Type":"ContainerStarted","Data":"800c60337e021f234c0073cacf096521258ea17487a5fd23ba4650c7e251a796"} Jul 07 10:33:09.314472 ip-10-0-135-34 kubenswrapper[2578]: I0707 10:33:09.314438 2578 generic.go:356] "Generic (PLEG): container finished" podID="fb918268-2c2b-4ea9-ba53-676b42b1cec7" containerID="57dfdab7ac088ef8ae6614d66e687113bde81993eab1740967b264ea85932e47" exitCode=0 Jul 07 10:33:09.314870 ip-10-0-135-34 kubenswrapper[2578]: I0707 10:33:09.314533 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afrbf6z" event={"ID":"fb918268-2c2b-4ea9-ba53-676b42b1cec7","Type":"ContainerDied","Data":"57dfdab7ac088ef8ae6614d66e687113bde81993eab1740967b264ea85932e47"} Jul 07 10:33:10.322333 ip-10-0-135-34 kubenswrapper[2578]: I0707 10:33:10.322294 2578 generic.go:356] "Generic (PLEG): container finished" podID="fb918268-2c2b-4ea9-ba53-676b42b1cec7" containerID="29dc7f3f8a3f5cafa47f59359e21930f8841156426bf4fbb1236fb5b2cb7b6e5" exitCode=0 Jul 07 10:33:10.322734 ip-10-0-135-34 kubenswrapper[2578]: I0707 10:33:10.322378 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afrbf6z" event={"ID":"fb918268-2c2b-4ea9-ba53-676b42b1cec7","Type":"ContainerDied","Data":"29dc7f3f8a3f5cafa47f59359e21930f8841156426bf4fbb1236fb5b2cb7b6e5"} Jul 07 10:33:11.453013 ip-10-0-135-34 kubenswrapper[2578]: I0707 10:33:11.452985 2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afrbf6z" Jul 07 10:33:11.556092 ip-10-0-135-34 kubenswrapper[2578]: I0707 10:33:11.556052 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/fb918268-2c2b-4ea9-ba53-676b42b1cec7-bundle\") pod \"fb918268-2c2b-4ea9-ba53-676b42b1cec7\" (UID: \"fb918268-2c2b-4ea9-ba53-676b42b1cec7\") " Jul 07 10:33:11.556280 ip-10-0-135-34 kubenswrapper[2578]: I0707 10:33:11.556114 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-sw8q4\" (UniqueName: \"kubernetes.io/projected/fb918268-2c2b-4ea9-ba53-676b42b1cec7-kube-api-access-sw8q4\") pod \"fb918268-2c2b-4ea9-ba53-676b42b1cec7\" (UID: \"fb918268-2c2b-4ea9-ba53-676b42b1cec7\") " Jul 07 10:33:11.556280 ip-10-0-135-34 kubenswrapper[2578]: I0707 10:33:11.556148 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/fb918268-2c2b-4ea9-ba53-676b42b1cec7-util\") pod \"fb918268-2c2b-4ea9-ba53-676b42b1cec7\" (UID: \"fb918268-2c2b-4ea9-ba53-676b42b1cec7\") " Jul 07 10:33:11.556876 ip-10-0-135-34 kubenswrapper[2578]: I0707 10:33:11.556836 2578 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fb918268-2c2b-4ea9-ba53-676b42b1cec7-bundle" (OuterVolumeSpecName: "bundle") pod "fb918268-2c2b-4ea9-ba53-676b42b1cec7" (UID: "fb918268-2c2b-4ea9-ba53-676b42b1cec7"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 07 10:33:11.558395 ip-10-0-135-34 kubenswrapper[2578]: I0707 10:33:11.558365 2578 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fb918268-2c2b-4ea9-ba53-676b42b1cec7-kube-api-access-sw8q4" (OuterVolumeSpecName: "kube-api-access-sw8q4") pod "fb918268-2c2b-4ea9-ba53-676b42b1cec7" (UID: "fb918268-2c2b-4ea9-ba53-676b42b1cec7"). InnerVolumeSpecName "kube-api-access-sw8q4". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 07 10:33:11.561935 ip-10-0-135-34 kubenswrapper[2578]: I0707 10:33:11.561914 2578 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fb918268-2c2b-4ea9-ba53-676b42b1cec7-util" (OuterVolumeSpecName: "util") pod "fb918268-2c2b-4ea9-ba53-676b42b1cec7" (UID: "fb918268-2c2b-4ea9-ba53-676b42b1cec7"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 07 10:33:11.657395 ip-10-0-135-34 kubenswrapper[2578]: I0707 10:33:11.657353 2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sw8q4\" (UniqueName: \"kubernetes.io/projected/fb918268-2c2b-4ea9-ba53-676b42b1cec7-kube-api-access-sw8q4\") on node \"ip-10-0-135-34.ec2.internal\" DevicePath \"\"" Jul 07 10:33:11.657395 ip-10-0-135-34 kubenswrapper[2578]: I0707 10:33:11.657387 2578 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/fb918268-2c2b-4ea9-ba53-676b42b1cec7-util\") on node \"ip-10-0-135-34.ec2.internal\" DevicePath \"\"" Jul 07 10:33:11.657395 ip-10-0-135-34 kubenswrapper[2578]: I0707 10:33:11.657396 2578 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/fb918268-2c2b-4ea9-ba53-676b42b1cec7-bundle\") on node \"ip-10-0-135-34.ec2.internal\" DevicePath \"\"" Jul 07 10:33:12.332405 ip-10-0-135-34 kubenswrapper[2578]: I0707 10:33:12.332379 2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afrbf6z" Jul 07 10:33:12.332405 ip-10-0-135-34 kubenswrapper[2578]: I0707 10:33:12.332395 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afrbf6z" event={"ID":"fb918268-2c2b-4ea9-ba53-676b42b1cec7","Type":"ContainerDied","Data":"800c60337e021f234c0073cacf096521258ea17487a5fd23ba4650c7e251a796"} Jul 07 10:33:12.332618 ip-10-0-135-34 kubenswrapper[2578]: I0707 10:33:12.332447 2578 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="800c60337e021f234c0073cacf096521258ea17487a5fd23ba4650c7e251a796" Jul 07 10:33:22.364171 ip-10-0-135-34 kubenswrapper[2578]: I0707 10:33:22.364136 2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-586fcf4d5b-8hlhv"] Jul 07 10:33:22.364635 ip-10-0-135-34 kubenswrapper[2578]: I0707 10:33:22.364556 2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="fb918268-2c2b-4ea9-ba53-676b42b1cec7" containerName="util" Jul 07 10:33:22.364635 ip-10-0-135-34 kubenswrapper[2578]: I0707 10:33:22.364572 2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="fb918268-2c2b-4ea9-ba53-676b42b1cec7" containerName="util" Jul 07 10:33:22.364635 ip-10-0-135-34 kubenswrapper[2578]: I0707 10:33:22.364585 2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="fb918268-2c2b-4ea9-ba53-676b42b1cec7" containerName="pull" Jul 07 10:33:22.364635 ip-10-0-135-34 kubenswrapper[2578]: I0707 10:33:22.364593 2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="fb918268-2c2b-4ea9-ba53-676b42b1cec7" containerName="pull" Jul 07 10:33:22.364635 ip-10-0-135-34 kubenswrapper[2578]: I0707 10:33:22.364611 2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="fb918268-2c2b-4ea9-ba53-676b42b1cec7" containerName="extract" Jul 07 10:33:22.364635 ip-10-0-135-34 kubenswrapper[2578]: I0707 10:33:22.364619 2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="fb918268-2c2b-4ea9-ba53-676b42b1cec7" containerName="extract" Jul 07 10:33:22.365005 ip-10-0-135-34 kubenswrapper[2578]: I0707 10:33:22.364706 2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="fb918268-2c2b-4ea9-ba53-676b42b1cec7" containerName="extract" Jul 07 10:33:22.369439 ip-10-0-135-34 kubenswrapper[2578]: I0707 10:33:22.369415 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/opendatahub-operator-controller-manager-586fcf4d5b-8hlhv" Jul 07 10:33:22.372608 ip-10-0-135-34 kubenswrapper[2578]: I0707 10:33:22.372587 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-manager-service-cert\"" Jul 07 10:33:22.373646 ip-10-0-135-34 kubenswrapper[2578]: I0707 10:33:22.373627 2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"openshift-service-ca.crt\"" Jul 07 10:33:22.373866 ip-10-0-135-34 kubenswrapper[2578]: I0707 10:33:22.373851 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-webhook-cert\"" Jul 07 10:33:22.373954 ip-10-0-135-34 kubenswrapper[2578]: I0707 10:33:22.373893 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-manager-dockercfg-9dx5k\"" Jul 07 10:33:22.374127 ip-10-0-135-34 kubenswrapper[2578]: I0707 10:33:22.374113 2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"kube-root-ca.crt\"" Jul 07 10:33:22.403117 ip-10-0-135-34 kubenswrapper[2578]: I0707 10:33:22.403081 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-586fcf4d5b-8hlhv"] Jul 07 10:33:22.553378 ip-10-0-135-34 kubenswrapper[2578]: I0707 10:33:22.553342 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/30a48a8d-7e8f-49eb-9dc5-c4d114f35dc1-manifests\") pod \"opendatahub-operator-controller-manager-586fcf4d5b-8hlhv\" (UID: \"30a48a8d-7e8f-49eb-9dc5-c4d114f35dc1\") " pod="opendatahub/opendatahub-operator-controller-manager-586fcf4d5b-8hlhv" Jul 07 10:33:22.553554 ip-10-0-135-34 kubenswrapper[2578]: I0707 10:33:22.553390 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/30a48a8d-7e8f-49eb-9dc5-c4d114f35dc1-webhook-cert\") pod \"opendatahub-operator-controller-manager-586fcf4d5b-8hlhv\" (UID: \"30a48a8d-7e8f-49eb-9dc5-c4d114f35dc1\") " pod="opendatahub/opendatahub-operator-controller-manager-586fcf4d5b-8hlhv" Jul 07 10:33:22.553554 ip-10-0-135-34 kubenswrapper[2578]: I0707 10:33:22.553415 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9kpcp\" (UniqueName: \"kubernetes.io/projected/30a48a8d-7e8f-49eb-9dc5-c4d114f35dc1-kube-api-access-9kpcp\") pod \"opendatahub-operator-controller-manager-586fcf4d5b-8hlhv\" (UID: \"30a48a8d-7e8f-49eb-9dc5-c4d114f35dc1\") " pod="opendatahub/opendatahub-operator-controller-manager-586fcf4d5b-8hlhv" Jul 07 10:33:22.553554 ip-10-0-135-34 kubenswrapper[2578]: I0707 10:33:22.553478 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/30a48a8d-7e8f-49eb-9dc5-c4d114f35dc1-tmp\") pod \"opendatahub-operator-controller-manager-586fcf4d5b-8hlhv\" (UID: \"30a48a8d-7e8f-49eb-9dc5-c4d114f35dc1\") " pod="opendatahub/opendatahub-operator-controller-manager-586fcf4d5b-8hlhv" Jul 07 10:33:22.553554 ip-10-0-135-34 kubenswrapper[2578]: I0707 10:33:22.553516 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/30a48a8d-7e8f-49eb-9dc5-c4d114f35dc1-apiservice-cert\") pod \"opendatahub-operator-controller-manager-586fcf4d5b-8hlhv\" (UID: \"30a48a8d-7e8f-49eb-9dc5-c4d114f35dc1\") " pod="opendatahub/opendatahub-operator-controller-manager-586fcf4d5b-8hlhv" Jul 07 10:33:22.654437 ip-10-0-135-34 kubenswrapper[2578]: I0707 10:33:22.654344 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/30a48a8d-7e8f-49eb-9dc5-c4d114f35dc1-manifests\") pod \"opendatahub-operator-controller-manager-586fcf4d5b-8hlhv\" (UID: \"30a48a8d-7e8f-49eb-9dc5-c4d114f35dc1\") " pod="opendatahub/opendatahub-operator-controller-manager-586fcf4d5b-8hlhv" Jul 07 10:33:22.654437 ip-10-0-135-34 kubenswrapper[2578]: I0707 10:33:22.654395 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/30a48a8d-7e8f-49eb-9dc5-c4d114f35dc1-webhook-cert\") pod \"opendatahub-operator-controller-manager-586fcf4d5b-8hlhv\" (UID: \"30a48a8d-7e8f-49eb-9dc5-c4d114f35dc1\") " pod="opendatahub/opendatahub-operator-controller-manager-586fcf4d5b-8hlhv" Jul 07 10:33:22.654437 ip-10-0-135-34 kubenswrapper[2578]: I0707 10:33:22.654420 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9kpcp\" (UniqueName: \"kubernetes.io/projected/30a48a8d-7e8f-49eb-9dc5-c4d114f35dc1-kube-api-access-9kpcp\") pod \"opendatahub-operator-controller-manager-586fcf4d5b-8hlhv\" (UID: \"30a48a8d-7e8f-49eb-9dc5-c4d114f35dc1\") " pod="opendatahub/opendatahub-operator-controller-manager-586fcf4d5b-8hlhv" Jul 07 10:33:22.654713 ip-10-0-135-34 kubenswrapper[2578]: I0707 10:33:22.654446 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/30a48a8d-7e8f-49eb-9dc5-c4d114f35dc1-tmp\") pod \"opendatahub-operator-controller-manager-586fcf4d5b-8hlhv\" (UID: \"30a48a8d-7e8f-49eb-9dc5-c4d114f35dc1\") " pod="opendatahub/opendatahub-operator-controller-manager-586fcf4d5b-8hlhv" Jul 07 10:33:22.654713 ip-10-0-135-34 kubenswrapper[2578]: I0707 10:33:22.654465 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/30a48a8d-7e8f-49eb-9dc5-c4d114f35dc1-apiservice-cert\") pod \"opendatahub-operator-controller-manager-586fcf4d5b-8hlhv\" (UID: \"30a48a8d-7e8f-49eb-9dc5-c4d114f35dc1\") " pod="opendatahub/opendatahub-operator-controller-manager-586fcf4d5b-8hlhv" Jul 07 10:33:22.654880 ip-10-0-135-34 kubenswrapper[2578]: I0707 10:33:22.654727 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/30a48a8d-7e8f-49eb-9dc5-c4d114f35dc1-manifests\") pod \"opendatahub-operator-controller-manager-586fcf4d5b-8hlhv\" (UID: \"30a48a8d-7e8f-49eb-9dc5-c4d114f35dc1\") " pod="opendatahub/opendatahub-operator-controller-manager-586fcf4d5b-8hlhv" Jul 07 10:33:22.656993 ip-10-0-135-34 kubenswrapper[2578]: I0707 10:33:22.656967 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/30a48a8d-7e8f-49eb-9dc5-c4d114f35dc1-tmp\") pod \"opendatahub-operator-controller-manager-586fcf4d5b-8hlhv\" (UID: \"30a48a8d-7e8f-49eb-9dc5-c4d114f35dc1\") " pod="opendatahub/opendatahub-operator-controller-manager-586fcf4d5b-8hlhv" Jul 07 10:33:22.657099 ip-10-0-135-34 kubenswrapper[2578]: I0707 10:33:22.657020 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/30a48a8d-7e8f-49eb-9dc5-c4d114f35dc1-apiservice-cert\") pod \"opendatahub-operator-controller-manager-586fcf4d5b-8hlhv\" (UID: \"30a48a8d-7e8f-49eb-9dc5-c4d114f35dc1\") " pod="opendatahub/opendatahub-operator-controller-manager-586fcf4d5b-8hlhv" Jul 07 10:33:22.657099 ip-10-0-135-34 kubenswrapper[2578]: I0707 10:33:22.657086 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/30a48a8d-7e8f-49eb-9dc5-c4d114f35dc1-webhook-cert\") pod \"opendatahub-operator-controller-manager-586fcf4d5b-8hlhv\" (UID: \"30a48a8d-7e8f-49eb-9dc5-c4d114f35dc1\") " pod="opendatahub/opendatahub-operator-controller-manager-586fcf4d5b-8hlhv" Jul 07 10:33:22.665193 ip-10-0-135-34 kubenswrapper[2578]: I0707 10:33:22.665167 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9kpcp\" (UniqueName: \"kubernetes.io/projected/30a48a8d-7e8f-49eb-9dc5-c4d114f35dc1-kube-api-access-9kpcp\") pod \"opendatahub-operator-controller-manager-586fcf4d5b-8hlhv\" (UID: \"30a48a8d-7e8f-49eb-9dc5-c4d114f35dc1\") " pod="opendatahub/opendatahub-operator-controller-manager-586fcf4d5b-8hlhv" Jul 07 10:33:22.680990 ip-10-0-135-34 kubenswrapper[2578]: I0707 10:33:22.680958 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/opendatahub-operator-controller-manager-586fcf4d5b-8hlhv" Jul 07 10:33:22.858657 ip-10-0-135-34 kubenswrapper[2578]: I0707 10:33:22.858633 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-586fcf4d5b-8hlhv"] Jul 07 10:33:22.860536 ip-10-0-135-34 kubenswrapper[2578]: W0707 10:33:22.860509 2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod30a48a8d_7e8f_49eb_9dc5_c4d114f35dc1.slice/crio-9a9694db0595e403ae14fa4346eaf7ee2ae33d7b69d272b8711d1b802e064820 WatchSource:0}: Error finding container 9a9694db0595e403ae14fa4346eaf7ee2ae33d7b69d272b8711d1b802e064820: Status 404 returned error can't find the container with id 9a9694db0595e403ae14fa4346eaf7ee2ae33d7b69d272b8711d1b802e064820 Jul 07 10:33:23.377746 ip-10-0-135-34 kubenswrapper[2578]: I0707 10:33:23.377707 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-586fcf4d5b-8hlhv" event={"ID":"30a48a8d-7e8f-49eb-9dc5-c4d114f35dc1","Type":"ContainerStarted","Data":"9a9694db0595e403ae14fa4346eaf7ee2ae33d7b69d272b8711d1b802e064820"} Jul 07 10:33:26.395932 ip-10-0-135-34 kubenswrapper[2578]: I0707 10:33:26.395892 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-586fcf4d5b-8hlhv" event={"ID":"30a48a8d-7e8f-49eb-9dc5-c4d114f35dc1","Type":"ContainerStarted","Data":"2def4b4cd1f19dca4cf8df7624a49a381097372b05c9f5c0a05406856fa85b85"} Jul 07 10:33:27.401285 ip-10-0-135-34 kubenswrapper[2578]: I0707 10:33:27.401252 2578 generic.go:356] "Generic (PLEG): container finished" podID="30a48a8d-7e8f-49eb-9dc5-c4d114f35dc1" containerID="2def4b4cd1f19dca4cf8df7624a49a381097372b05c9f5c0a05406856fa85b85" exitCode=0 Jul 07 10:33:27.401761 ip-10-0-135-34 kubenswrapper[2578]: I0707 10:33:27.401297 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-586fcf4d5b-8hlhv" event={"ID":"30a48a8d-7e8f-49eb-9dc5-c4d114f35dc1","Type":"ContainerDied","Data":"2def4b4cd1f19dca4cf8df7624a49a381097372b05c9f5c0a05406856fa85b85"} Jul 07 10:33:28.407430 ip-10-0-135-34 kubenswrapper[2578]: I0707 10:33:28.407395 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-586fcf4d5b-8hlhv" event={"ID":"30a48a8d-7e8f-49eb-9dc5-c4d114f35dc1","Type":"ContainerStarted","Data":"555ae67e0ed47bd287f29fad1afa5aab11b4d1e9a87862606ed477b50fbdfbd7"} Jul 07 10:33:28.407827 ip-10-0-135-34 kubenswrapper[2578]: I0707 10:33:28.407519 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/opendatahub-operator-controller-manager-586fcf4d5b-8hlhv" Jul 07 10:33:28.431498 ip-10-0-135-34 kubenswrapper[2578]: I0707 10:33:28.431448 2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/opendatahub-operator-controller-manager-586fcf4d5b-8hlhv" podStartSLOduration=2.016445299 podStartE2EDuration="6.431429783s" totalImagesPullingTime="3.383389639s" totalInitContainerRuntime="1.031594845s" isStatefulPod=false podCreationTimestamp="2026-07-07 10:33:22 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 10:33:28.429785317 +0000 UTC m=+377.101744871" watchObservedRunningTime="2026-07-07 10:33:28.431429783 +0000 UTC m=+377.103389326" Jul 07 10:33:39.414077 ip-10-0-135-34 kubenswrapper[2578]: I0707 10:33:39.414049 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/opendatahub-operator-controller-manager-586fcf4d5b-8hlhv" Jul 07 10:33:42.195004 ip-10-0-135-34 kubenswrapper[2578]: I0707 10:33:42.194966 2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/kube-auth-proxy-5c48467d48-dhpwl"] Jul 07 10:33:42.198347 ip-10-0-135-34 kubenswrapper[2578]: I0707 10:33:42.198329 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/kube-auth-proxy-5c48467d48-dhpwl" Jul 07 10:33:42.201756 ip-10-0-135-34 kubenswrapper[2578]: I0707 10:33:42.201734 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-creds\"" Jul 07 10:33:42.201890 ip-10-0-135-34 kubenswrapper[2578]: I0707 10:33:42.201848 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-dockercfg-tdvmf\"" Jul 07 10:33:42.202635 ip-10-0-135-34 kubenswrapper[2578]: I0707 10:33:42.202618 2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-service-ca.crt\"" Jul 07 10:33:42.204110 ip-10-0-135-34 kubenswrapper[2578]: I0707 10:33:42.204092 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-tls\"" Jul 07 10:33:42.205007 ip-10-0-135-34 kubenswrapper[2578]: I0707 10:33:42.204989 2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"kube-root-ca.crt\"" Jul 07 10:33:42.236557 ip-10-0-135-34 kubenswrapper[2578]: I0707 10:33:42.236524 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/kube-auth-proxy-5c48467d48-dhpwl"] Jul 07 10:33:42.307662 ip-10-0-135-34 kubenswrapper[2578]: I0707 10:33:42.307632 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1437df6f-5e77-4b9b-8ca1-877cc48b0d84-tls-certs\") pod \"kube-auth-proxy-5c48467d48-dhpwl\" (UID: \"1437df6f-5e77-4b9b-8ca1-877cc48b0d84\") " pod="openshift-ingress/kube-auth-proxy-5c48467d48-dhpwl" Jul 07 10:33:42.307881 ip-10-0-135-34 kubenswrapper[2578]: I0707 10:33:42.307678 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hhpx2\" (UniqueName: \"kubernetes.io/projected/1437df6f-5e77-4b9b-8ca1-877cc48b0d84-kube-api-access-hhpx2\") pod \"kube-auth-proxy-5c48467d48-dhpwl\" (UID: \"1437df6f-5e77-4b9b-8ca1-877cc48b0d84\") " pod="openshift-ingress/kube-auth-proxy-5c48467d48-dhpwl" Jul 07 10:33:42.307881 ip-10-0-135-34 kubenswrapper[2578]: I0707 10:33:42.307706 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/1437df6f-5e77-4b9b-8ca1-877cc48b0d84-tmp\") pod \"kube-auth-proxy-5c48467d48-dhpwl\" (UID: \"1437df6f-5e77-4b9b-8ca1-877cc48b0d84\") " pod="openshift-ingress/kube-auth-proxy-5c48467d48-dhpwl" Jul 07 10:33:42.409096 ip-10-0-135-34 kubenswrapper[2578]: I0707 10:33:42.409056 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1437df6f-5e77-4b9b-8ca1-877cc48b0d84-tls-certs\") pod \"kube-auth-proxy-5c48467d48-dhpwl\" (UID: \"1437df6f-5e77-4b9b-8ca1-877cc48b0d84\") " pod="openshift-ingress/kube-auth-proxy-5c48467d48-dhpwl" Jul 07 10:33:42.409268 ip-10-0-135-34 kubenswrapper[2578]: I0707 10:33:42.409113 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hhpx2\" (UniqueName: \"kubernetes.io/projected/1437df6f-5e77-4b9b-8ca1-877cc48b0d84-kube-api-access-hhpx2\") pod \"kube-auth-proxy-5c48467d48-dhpwl\" (UID: \"1437df6f-5e77-4b9b-8ca1-877cc48b0d84\") " pod="openshift-ingress/kube-auth-proxy-5c48467d48-dhpwl" Jul 07 10:33:42.409268 ip-10-0-135-34 kubenswrapper[2578]: I0707 10:33:42.409144 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/1437df6f-5e77-4b9b-8ca1-877cc48b0d84-tmp\") pod \"kube-auth-proxy-5c48467d48-dhpwl\" (UID: \"1437df6f-5e77-4b9b-8ca1-877cc48b0d84\") " pod="openshift-ingress/kube-auth-proxy-5c48467d48-dhpwl" Jul 07 10:33:42.411486 ip-10-0-135-34 kubenswrapper[2578]: I0707 10:33:42.411457 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/1437df6f-5e77-4b9b-8ca1-877cc48b0d84-tmp\") pod \"kube-auth-proxy-5c48467d48-dhpwl\" (UID: \"1437df6f-5e77-4b9b-8ca1-877cc48b0d84\") " pod="openshift-ingress/kube-auth-proxy-5c48467d48-dhpwl" Jul 07 10:33:42.411821 ip-10-0-135-34 kubenswrapper[2578]: I0707 10:33:42.411799 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1437df6f-5e77-4b9b-8ca1-877cc48b0d84-tls-certs\") pod \"kube-auth-proxy-5c48467d48-dhpwl\" (UID: \"1437df6f-5e77-4b9b-8ca1-877cc48b0d84\") " pod="openshift-ingress/kube-auth-proxy-5c48467d48-dhpwl" Jul 07 10:33:42.449288 ip-10-0-135-34 kubenswrapper[2578]: I0707 10:33:42.449202 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hhpx2\" (UniqueName: \"kubernetes.io/projected/1437df6f-5e77-4b9b-8ca1-877cc48b0d84-kube-api-access-hhpx2\") pod \"kube-auth-proxy-5c48467d48-dhpwl\" (UID: \"1437df6f-5e77-4b9b-8ca1-877cc48b0d84\") " pod="openshift-ingress/kube-auth-proxy-5c48467d48-dhpwl" Jul 07 10:33:42.507290 ip-10-0-135-34 kubenswrapper[2578]: I0707 10:33:42.507249 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/kube-auth-proxy-5c48467d48-dhpwl" Jul 07 10:33:42.686351 ip-10-0-135-34 kubenswrapper[2578]: I0707 10:33:42.686280 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/kube-auth-proxy-5c48467d48-dhpwl"] Jul 07 10:33:43.474318 ip-10-0-135-34 kubenswrapper[2578]: I0707 10:33:43.474274 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/kube-auth-proxy-5c48467d48-dhpwl" event={"ID":"1437df6f-5e77-4b9b-8ca1-877cc48b0d84","Type":"ContainerStarted","Data":"3643c83b5dd21580c77311d34432b7c30283e2a203db8a252462bbaf6d4ee0cf"} Jul 07 10:33:46.490665 ip-10-0-135-34 kubenswrapper[2578]: I0707 10:33:46.490625 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/kube-auth-proxy-5c48467d48-dhpwl" event={"ID":"1437df6f-5e77-4b9b-8ca1-877cc48b0d84","Type":"ContainerStarted","Data":"d2806d4e4c46b8ddcf2e99621e2494c8b13ed9a6fd0c01241fdef74e5fea8bd1"} Jul 07 10:33:46.511698 ip-10-0-135-34 kubenswrapper[2578]: I0707 10:33:46.511650 2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/kube-auth-proxy-5c48467d48-dhpwl" podStartSLOduration=1.790423611 podStartE2EDuration="4.511633768s" totalImagesPullingTime="2.721210157s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-07 10:33:42 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 10:33:46.510062065 +0000 UTC m=+395.182021610" watchObservedRunningTime="2026-07-07 10:33:46.511633768 +0000 UTC m=+395.183593312" Jul 07 10:33:49.850673 ip-10-0-135-34 kubenswrapper[2578]: I0707 10:33:49.850638 2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-lws-operator/lws-controller-manager-59649ddfff-4mzhd"] Jul 07 10:33:49.856481 ip-10-0-135-34 kubenswrapper[2578]: I0707 10:33:49.856446 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/lws-controller-manager-59649ddfff-4mzhd" Jul 07 10:33:49.859939 ip-10-0-135-34 kubenswrapper[2578]: I0707 10:33:49.859904 2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"lws-manager-config\"" Jul 07 10:33:49.860913 ip-10-0-135-34 kubenswrapper[2578]: I0707 10:33:49.860895 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"metrics-server-cert\"" Jul 07 10:33:49.861025 ip-10-0-135-34 kubenswrapper[2578]: I0707 10:33:49.860935 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"lws-controller-manager-dockercfg-lgdmg\"" Jul 07 10:33:49.861169 ip-10-0-135-34 kubenswrapper[2578]: I0707 10:33:49.861152 2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"openshift-service-ca.crt\"" Jul 07 10:33:49.861233 ip-10-0-135-34 kubenswrapper[2578]: I0707 10:33:49.861191 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"webhook-server-cert\"" Jul 07 10:33:49.861331 ip-10-0-135-34 kubenswrapper[2578]: I0707 10:33:49.861317 2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"kube-root-ca.crt\"" Jul 07 10:33:49.873546 ip-10-0-135-34 kubenswrapper[2578]: I0707 10:33:49.873521 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/482b392b-7cc6-4e18-aa67-306d6a473bfe-cert\") pod \"lws-controller-manager-59649ddfff-4mzhd\" (UID: \"482b392b-7cc6-4e18-aa67-306d6a473bfe\") " pod="openshift-lws-operator/lws-controller-manager-59649ddfff-4mzhd" Jul 07 10:33:49.873672 ip-10-0-135-34 kubenswrapper[2578]: I0707 10:33:49.873593 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/482b392b-7cc6-4e18-aa67-306d6a473bfe-manager-config\") pod \"lws-controller-manager-59649ddfff-4mzhd\" (UID: \"482b392b-7cc6-4e18-aa67-306d6a473bfe\") " pod="openshift-lws-operator/lws-controller-manager-59649ddfff-4mzhd" Jul 07 10:33:49.873672 ip-10-0-135-34 kubenswrapper[2578]: I0707 10:33:49.873623 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/482b392b-7cc6-4e18-aa67-306d6a473bfe-metrics-cert\") pod \"lws-controller-manager-59649ddfff-4mzhd\" (UID: \"482b392b-7cc6-4e18-aa67-306d6a473bfe\") " pod="openshift-lws-operator/lws-controller-manager-59649ddfff-4mzhd" Jul 07 10:33:49.873672 ip-10-0-135-34 kubenswrapper[2578]: I0707 10:33:49.873661 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dlqbr\" (UniqueName: \"kubernetes.io/projected/482b392b-7cc6-4e18-aa67-306d6a473bfe-kube-api-access-dlqbr\") pod \"lws-controller-manager-59649ddfff-4mzhd\" (UID: \"482b392b-7cc6-4e18-aa67-306d6a473bfe\") " pod="openshift-lws-operator/lws-controller-manager-59649ddfff-4mzhd" Jul 07 10:33:49.882733 ip-10-0-135-34 kubenswrapper[2578]: I0707 10:33:49.882698 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/lws-controller-manager-59649ddfff-4mzhd"] Jul 07 10:33:49.975026 ip-10-0-135-34 kubenswrapper[2578]: I0707 10:33:49.974979 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/482b392b-7cc6-4e18-aa67-306d6a473bfe-cert\") pod \"lws-controller-manager-59649ddfff-4mzhd\" (UID: \"482b392b-7cc6-4e18-aa67-306d6a473bfe\") " pod="openshift-lws-operator/lws-controller-manager-59649ddfff-4mzhd" Jul 07 10:33:49.975236 ip-10-0-135-34 kubenswrapper[2578]: I0707 10:33:49.975078 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/482b392b-7cc6-4e18-aa67-306d6a473bfe-manager-config\") pod \"lws-controller-manager-59649ddfff-4mzhd\" (UID: \"482b392b-7cc6-4e18-aa67-306d6a473bfe\") " pod="openshift-lws-operator/lws-controller-manager-59649ddfff-4mzhd" Jul 07 10:33:49.975236 ip-10-0-135-34 kubenswrapper[2578]: I0707 10:33:49.975102 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/482b392b-7cc6-4e18-aa67-306d6a473bfe-metrics-cert\") pod \"lws-controller-manager-59649ddfff-4mzhd\" (UID: \"482b392b-7cc6-4e18-aa67-306d6a473bfe\") " pod="openshift-lws-operator/lws-controller-manager-59649ddfff-4mzhd" Jul 07 10:33:49.975236 ip-10-0-135-34 kubenswrapper[2578]: I0707 10:33:49.975141 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dlqbr\" (UniqueName: \"kubernetes.io/projected/482b392b-7cc6-4e18-aa67-306d6a473bfe-kube-api-access-dlqbr\") pod \"lws-controller-manager-59649ddfff-4mzhd\" (UID: \"482b392b-7cc6-4e18-aa67-306d6a473bfe\") " pod="openshift-lws-operator/lws-controller-manager-59649ddfff-4mzhd" Jul 07 10:33:49.975842 ip-10-0-135-34 kubenswrapper[2578]: I0707 10:33:49.975821 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/482b392b-7cc6-4e18-aa67-306d6a473bfe-manager-config\") pod \"lws-controller-manager-59649ddfff-4mzhd\" (UID: \"482b392b-7cc6-4e18-aa67-306d6a473bfe\") " pod="openshift-lws-operator/lws-controller-manager-59649ddfff-4mzhd" Jul 07 10:33:49.977713 ip-10-0-135-34 kubenswrapper[2578]: I0707 10:33:49.977691 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/482b392b-7cc6-4e18-aa67-306d6a473bfe-metrics-cert\") pod \"lws-controller-manager-59649ddfff-4mzhd\" (UID: \"482b392b-7cc6-4e18-aa67-306d6a473bfe\") " pod="openshift-lws-operator/lws-controller-manager-59649ddfff-4mzhd" Jul 07 10:33:49.977799 ip-10-0-135-34 kubenswrapper[2578]: I0707 10:33:49.977785 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/482b392b-7cc6-4e18-aa67-306d6a473bfe-cert\") pod \"lws-controller-manager-59649ddfff-4mzhd\" (UID: \"482b392b-7cc6-4e18-aa67-306d6a473bfe\") " pod="openshift-lws-operator/lws-controller-manager-59649ddfff-4mzhd" Jul 07 10:33:49.997564 ip-10-0-135-34 kubenswrapper[2578]: I0707 10:33:49.997531 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dlqbr\" (UniqueName: \"kubernetes.io/projected/482b392b-7cc6-4e18-aa67-306d6a473bfe-kube-api-access-dlqbr\") pod \"lws-controller-manager-59649ddfff-4mzhd\" (UID: \"482b392b-7cc6-4e18-aa67-306d6a473bfe\") " pod="openshift-lws-operator/lws-controller-manager-59649ddfff-4mzhd" Jul 07 10:33:50.167611 ip-10-0-135-34 kubenswrapper[2578]: I0707 10:33:50.167523 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/lws-controller-manager-59649ddfff-4mzhd" Jul 07 10:33:50.341675 ip-10-0-135-34 kubenswrapper[2578]: I0707 10:33:50.341651 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/lws-controller-manager-59649ddfff-4mzhd"] Jul 07 10:33:50.343322 ip-10-0-135-34 kubenswrapper[2578]: W0707 10:33:50.343295 2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod482b392b_7cc6_4e18_aa67_306d6a473bfe.slice/crio-1ae08a3670f75ab69ec8f4dc1f53282571a8275cb2ebe5df34c1a928b708ec6c WatchSource:0}: Error finding container 1ae08a3670f75ab69ec8f4dc1f53282571a8275cb2ebe5df34c1a928b708ec6c: Status 404 returned error can't find the container with id 1ae08a3670f75ab69ec8f4dc1f53282571a8275cb2ebe5df34c1a928b708ec6c Jul 07 10:33:50.508935 ip-10-0-135-34 kubenswrapper[2578]: I0707 10:33:50.508826 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/lws-controller-manager-59649ddfff-4mzhd" event={"ID":"482b392b-7cc6-4e18-aa67-306d6a473bfe","Type":"ContainerStarted","Data":"1ae08a3670f75ab69ec8f4dc1f53282571a8275cb2ebe5df34c1a928b708ec6c"} Jul 07 10:34:02.569336 ip-10-0-135-34 kubenswrapper[2578]: I0707 10:34:02.569301 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/lws-controller-manager-59649ddfff-4mzhd" event={"ID":"482b392b-7cc6-4e18-aa67-306d6a473bfe","Type":"ContainerStarted","Data":"e5822a006815789a990baf1fa42e304c1028a793f1e6fcf24f03eed8eeff25be"} Jul 07 10:34:02.569763 ip-10-0-135-34 kubenswrapper[2578]: I0707 10:34:02.569389 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-lws-operator/lws-controller-manager-59649ddfff-4mzhd" Jul 07 10:34:02.593019 ip-10-0-135-34 kubenswrapper[2578]: I0707 10:34:02.592965 2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-lws-operator/lws-controller-manager-59649ddfff-4mzhd" podStartSLOduration=1.8847042109999999 podStartE2EDuration="13.592948579s" totalImagesPullingTime="11.708244368s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-07 10:33:49 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 10:34:02.590276615 +0000 UTC m=+411.262236158" watchObservedRunningTime="2026-07-07 10:34:02.592948579 +0000 UTC m=+411.264908122" Jul 07 10:34:13.576270 ip-10-0-135-34 kubenswrapper[2578]: I0707 10:34:13.576240 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-lws-operator/lws-controller-manager-59649ddfff-4mzhd" Jul 07 10:34:59.383544 ip-10-0-135-34 kubenswrapper[2578]: I0707 10:34:59.383505 2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed735pdxq"] Jul 07 10:34:59.387593 ip-10-0-135-34 kubenswrapper[2578]: I0707 10:34:59.387575 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed735pdxq" Jul 07 10:34:59.391066 ip-10-0-135-34 kubenswrapper[2578]: I0707 10:34:59.391040 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"default-dockercfg-cp7dp\"" Jul 07 10:34:59.392030 ip-10-0-135-34 kubenswrapper[2578]: I0707 10:34:59.392009 2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kube-root-ca.crt\"" Jul 07 10:34:59.392143 ip-10-0-135-34 kubenswrapper[2578]: I0707 10:34:59.392093 2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"openshift-service-ca.crt\"" Jul 07 10:34:59.408929 ip-10-0-135-34 kubenswrapper[2578]: I0707 10:34:59.408900 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed735pdxq"] Jul 07 10:34:59.476946 ip-10-0-135-34 kubenswrapper[2578]: I0707 10:34:59.476911 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/06b9a5a9-0884-43cf-ad51-7b8f0bcd255f-util\") pod \"5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed735pdxq\" (UID: \"06b9a5a9-0884-43cf-ad51-7b8f0bcd255f\") " pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed735pdxq" Jul 07 10:34:59.477122 ip-10-0-135-34 kubenswrapper[2578]: I0707 10:34:59.476957 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/06b9a5a9-0884-43cf-ad51-7b8f0bcd255f-bundle\") pod \"5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed735pdxq\" (UID: \"06b9a5a9-0884-43cf-ad51-7b8f0bcd255f\") " pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed735pdxq" Jul 07 10:34:59.477122 ip-10-0-135-34 kubenswrapper[2578]: I0707 10:34:59.477025 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4z9cs\" (UniqueName: \"kubernetes.io/projected/06b9a5a9-0884-43cf-ad51-7b8f0bcd255f-kube-api-access-4z9cs\") pod \"5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed735pdxq\" (UID: \"06b9a5a9-0884-43cf-ad51-7b8f0bcd255f\") " pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed735pdxq" Jul 07 10:34:59.578166 ip-10-0-135-34 kubenswrapper[2578]: I0707 10:34:59.578130 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/06b9a5a9-0884-43cf-ad51-7b8f0bcd255f-util\") pod \"5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed735pdxq\" (UID: \"06b9a5a9-0884-43cf-ad51-7b8f0bcd255f\") " pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed735pdxq" Jul 07 10:34:59.578298 ip-10-0-135-34 kubenswrapper[2578]: I0707 10:34:59.578176 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/06b9a5a9-0884-43cf-ad51-7b8f0bcd255f-bundle\") pod \"5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed735pdxq\" (UID: \"06b9a5a9-0884-43cf-ad51-7b8f0bcd255f\") " pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed735pdxq" Jul 07 10:34:59.578298 ip-10-0-135-34 kubenswrapper[2578]: I0707 10:34:59.578201 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4z9cs\" (UniqueName: \"kubernetes.io/projected/06b9a5a9-0884-43cf-ad51-7b8f0bcd255f-kube-api-access-4z9cs\") pod \"5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed735pdxq\" (UID: \"06b9a5a9-0884-43cf-ad51-7b8f0bcd255f\") " pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed735pdxq" Jul 07 10:34:59.578532 ip-10-0-135-34 kubenswrapper[2578]: I0707 10:34:59.578509 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/06b9a5a9-0884-43cf-ad51-7b8f0bcd255f-util\") pod \"5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed735pdxq\" (UID: \"06b9a5a9-0884-43cf-ad51-7b8f0bcd255f\") " pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed735pdxq" Jul 07 10:34:59.578592 ip-10-0-135-34 kubenswrapper[2578]: I0707 10:34:59.578573 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/06b9a5a9-0884-43cf-ad51-7b8f0bcd255f-bundle\") pod \"5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed735pdxq\" (UID: \"06b9a5a9-0884-43cf-ad51-7b8f0bcd255f\") " pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed735pdxq" Jul 07 10:34:59.589161 ip-10-0-135-34 kubenswrapper[2578]: I0707 10:34:59.589130 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4z9cs\" (UniqueName: \"kubernetes.io/projected/06b9a5a9-0884-43cf-ad51-7b8f0bcd255f-kube-api-access-4z9cs\") pod \"5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed735pdxq\" (UID: \"06b9a5a9-0884-43cf-ad51-7b8f0bcd255f\") " pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed735pdxq" Jul 07 10:34:59.696874 ip-10-0-135-34 kubenswrapper[2578]: I0707 10:34:59.696754 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed735pdxq" Jul 07 10:34:59.842225 ip-10-0-135-34 kubenswrapper[2578]: I0707 10:34:59.842193 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed735pdxq"] Jul 07 10:34:59.842518 ip-10-0-135-34 kubenswrapper[2578]: W0707 10:34:59.842495 2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod06b9a5a9_0884_43cf_ad51_7b8f0bcd255f.slice/crio-0382c3e610d817c71bd3d275785f67a589cf3c58e57c4670e2daae0d5cf208e5 WatchSource:0}: Error finding container 0382c3e610d817c71bd3d275785f67a589cf3c58e57c4670e2daae0d5cf208e5: Status 404 returned error can't find the container with id 0382c3e610d817c71bd3d275785f67a589cf3c58e57c4670e2daae0d5cf208e5 Jul 07 10:35:00.161629 ip-10-0-135-34 kubenswrapper[2578]: I0707 10:35:00.161595 2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0wrzpr"] Jul 07 10:35:00.165231 ip-10-0-135-34 kubenswrapper[2578]: I0707 10:35:00.165213 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0wrzpr" Jul 07 10:35:00.177195 ip-10-0-135-34 kubenswrapper[2578]: I0707 10:35:00.177168 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0wrzpr"] Jul 07 10:35:00.284107 ip-10-0-135-34 kubenswrapper[2578]: I0707 10:35:00.284027 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jvlmn\" (UniqueName: \"kubernetes.io/projected/3fac3d82-02a4-4ddf-af58-d7392178ec4b-kube-api-access-jvlmn\") pod \"19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0wrzpr\" (UID: \"3fac3d82-02a4-4ddf-af58-d7392178ec4b\") " pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0wrzpr" Jul 07 10:35:00.284107 ip-10-0-135-34 kubenswrapper[2578]: I0707 10:35:00.284067 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/3fac3d82-02a4-4ddf-af58-d7392178ec4b-bundle\") pod \"19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0wrzpr\" (UID: \"3fac3d82-02a4-4ddf-af58-d7392178ec4b\") " pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0wrzpr" Jul 07 10:35:00.284292 ip-10-0-135-34 kubenswrapper[2578]: I0707 10:35:00.284133 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/3fac3d82-02a4-4ddf-af58-d7392178ec4b-util\") pod \"19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0wrzpr\" (UID: \"3fac3d82-02a4-4ddf-af58-d7392178ec4b\") " pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0wrzpr" Jul 07 10:35:00.385578 ip-10-0-135-34 kubenswrapper[2578]: I0707 10:35:00.385542 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/3fac3d82-02a4-4ddf-af58-d7392178ec4b-util\") pod \"19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0wrzpr\" (UID: \"3fac3d82-02a4-4ddf-af58-d7392178ec4b\") " pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0wrzpr" Jul 07 10:35:00.385972 ip-10-0-135-34 kubenswrapper[2578]: I0707 10:35:00.385622 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jvlmn\" (UniqueName: \"kubernetes.io/projected/3fac3d82-02a4-4ddf-af58-d7392178ec4b-kube-api-access-jvlmn\") pod \"19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0wrzpr\" (UID: \"3fac3d82-02a4-4ddf-af58-d7392178ec4b\") " pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0wrzpr" Jul 07 10:35:00.385972 ip-10-0-135-34 kubenswrapper[2578]: I0707 10:35:00.385651 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/3fac3d82-02a4-4ddf-af58-d7392178ec4b-bundle\") pod \"19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0wrzpr\" (UID: \"3fac3d82-02a4-4ddf-af58-d7392178ec4b\") " pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0wrzpr" Jul 07 10:35:00.386063 ip-10-0-135-34 kubenswrapper[2578]: I0707 10:35:00.386040 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/3fac3d82-02a4-4ddf-af58-d7392178ec4b-util\") pod \"19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0wrzpr\" (UID: \"3fac3d82-02a4-4ddf-af58-d7392178ec4b\") " pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0wrzpr" Jul 07 10:35:00.386063 ip-10-0-135-34 kubenswrapper[2578]: I0707 10:35:00.386056 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/3fac3d82-02a4-4ddf-af58-d7392178ec4b-bundle\") pod \"19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0wrzpr\" (UID: \"3fac3d82-02a4-4ddf-af58-d7392178ec4b\") " pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0wrzpr" Jul 07 10:35:00.401817 ip-10-0-135-34 kubenswrapper[2578]: I0707 10:35:00.401788 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jvlmn\" (UniqueName: \"kubernetes.io/projected/3fac3d82-02a4-4ddf-af58-d7392178ec4b-kube-api-access-jvlmn\") pod \"19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0wrzpr\" (UID: \"3fac3d82-02a4-4ddf-af58-d7392178ec4b\") " pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0wrzpr" Jul 07 10:35:00.474504 ip-10-0-135-34 kubenswrapper[2578]: I0707 10:35:00.474471 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0wrzpr" Jul 07 10:35:00.625162 ip-10-0-135-34 kubenswrapper[2578]: I0707 10:35:00.625139 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0wrzpr"] Jul 07 10:35:00.627082 ip-10-0-135-34 kubenswrapper[2578]: W0707 10:35:00.627056 2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3fac3d82_02a4_4ddf_af58_d7392178ec4b.slice/crio-5043c99909e86e5e1903dfc51d7ab6771482d3dc5198ed420ed182d622b6f08a WatchSource:0}: Error finding container 5043c99909e86e5e1903dfc51d7ab6771482d3dc5198ed420ed182d622b6f08a: Status 404 returned error can't find the container with id 5043c99909e86e5e1903dfc51d7ab6771482d3dc5198ed420ed182d622b6f08a Jul 07 10:35:00.834670 ip-10-0-135-34 kubenswrapper[2578]: I0707 10:35:00.834572 2578 generic.go:356] "Generic (PLEG): container finished" podID="3fac3d82-02a4-4ddf-af58-d7392178ec4b" containerID="bb66eb2a38a47bb9504ebeae589927541d8a42c3c89216c17dbb0db2f2412026" exitCode=0 Jul 07 10:35:00.834670 ip-10-0-135-34 kubenswrapper[2578]: I0707 10:35:00.834649 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0wrzpr" event={"ID":"3fac3d82-02a4-4ddf-af58-d7392178ec4b","Type":"ContainerDied","Data":"bb66eb2a38a47bb9504ebeae589927541d8a42c3c89216c17dbb0db2f2412026"} Jul 07 10:35:00.834944 ip-10-0-135-34 kubenswrapper[2578]: I0707 10:35:00.834744 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0wrzpr" event={"ID":"3fac3d82-02a4-4ddf-af58-d7392178ec4b","Type":"ContainerStarted","Data":"5043c99909e86e5e1903dfc51d7ab6771482d3dc5198ed420ed182d622b6f08a"} Jul 07 10:35:00.835199 ip-10-0-135-34 kubenswrapper[2578]: I0707 10:35:00.835176 2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1zqfm2"] Jul 07 10:35:00.836524 ip-10-0-135-34 kubenswrapper[2578]: I0707 10:35:00.836503 2578 generic.go:356] "Generic (PLEG): container finished" podID="06b9a5a9-0884-43cf-ad51-7b8f0bcd255f" containerID="81e82c06dd51546716119dad6231ec19986bde67f9f3255fc79d6bb3f5a555aa" exitCode=0 Jul 07 10:35:00.838984 ip-10-0-135-34 kubenswrapper[2578]: I0707 10:35:00.838963 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed735pdxq" event={"ID":"06b9a5a9-0884-43cf-ad51-7b8f0bcd255f","Type":"ContainerDied","Data":"81e82c06dd51546716119dad6231ec19986bde67f9f3255fc79d6bb3f5a555aa"} Jul 07 10:35:00.839060 ip-10-0-135-34 kubenswrapper[2578]: I0707 10:35:00.838994 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed735pdxq" event={"ID":"06b9a5a9-0884-43cf-ad51-7b8f0bcd255f","Type":"ContainerStarted","Data":"0382c3e610d817c71bd3d275785f67a589cf3c58e57c4670e2daae0d5cf208e5"} Jul 07 10:35:00.839120 ip-10-0-135-34 kubenswrapper[2578]: I0707 10:35:00.839100 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1zqfm2" Jul 07 10:35:00.855277 ip-10-0-135-34 kubenswrapper[2578]: I0707 10:35:00.855246 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1zqfm2"] Jul 07 10:35:00.990460 ip-10-0-135-34 kubenswrapper[2578]: I0707 10:35:00.990417 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/bf1fee85-d50a-4a52-84cd-e7f81323dd61-bundle\") pod \"9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1zqfm2\" (UID: \"bf1fee85-d50a-4a52-84cd-e7f81323dd61\") " pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1zqfm2" Jul 07 10:35:00.990659 ip-10-0-135-34 kubenswrapper[2578]: I0707 10:35:00.990483 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tj4ql\" (UniqueName: \"kubernetes.io/projected/bf1fee85-d50a-4a52-84cd-e7f81323dd61-kube-api-access-tj4ql\") pod \"9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1zqfm2\" (UID: \"bf1fee85-d50a-4a52-84cd-e7f81323dd61\") " pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1zqfm2" Jul 07 10:35:00.990659 ip-10-0-135-34 kubenswrapper[2578]: I0707 10:35:00.990510 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/bf1fee85-d50a-4a52-84cd-e7f81323dd61-util\") pod \"9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1zqfm2\" (UID: \"bf1fee85-d50a-4a52-84cd-e7f81323dd61\") " pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1zqfm2" Jul 07 10:35:01.091680 ip-10-0-135-34 kubenswrapper[2578]: I0707 10:35:01.091583 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/bf1fee85-d50a-4a52-84cd-e7f81323dd61-bundle\") pod \"9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1zqfm2\" (UID: \"bf1fee85-d50a-4a52-84cd-e7f81323dd61\") " pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1zqfm2" Jul 07 10:35:01.091680 ip-10-0-135-34 kubenswrapper[2578]: I0707 10:35:01.091669 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tj4ql\" (UniqueName: \"kubernetes.io/projected/bf1fee85-d50a-4a52-84cd-e7f81323dd61-kube-api-access-tj4ql\") pod \"9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1zqfm2\" (UID: \"bf1fee85-d50a-4a52-84cd-e7f81323dd61\") " pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1zqfm2" Jul 07 10:35:01.091960 ip-10-0-135-34 kubenswrapper[2578]: I0707 10:35:01.091690 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/bf1fee85-d50a-4a52-84cd-e7f81323dd61-util\") pod \"9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1zqfm2\" (UID: \"bf1fee85-d50a-4a52-84cd-e7f81323dd61\") " pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1zqfm2" Jul 07 10:35:01.092033 ip-10-0-135-34 kubenswrapper[2578]: I0707 10:35:01.092011 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/bf1fee85-d50a-4a52-84cd-e7f81323dd61-bundle\") pod \"9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1zqfm2\" (UID: \"bf1fee85-d50a-4a52-84cd-e7f81323dd61\") " pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1zqfm2" Jul 07 10:35:01.092089 ip-10-0-135-34 kubenswrapper[2578]: I0707 10:35:01.092037 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/bf1fee85-d50a-4a52-84cd-e7f81323dd61-util\") pod \"9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1zqfm2\" (UID: \"bf1fee85-d50a-4a52-84cd-e7f81323dd61\") " pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1zqfm2" Jul 07 10:35:01.103296 ip-10-0-135-34 kubenswrapper[2578]: I0707 10:35:01.103266 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tj4ql\" (UniqueName: \"kubernetes.io/projected/bf1fee85-d50a-4a52-84cd-e7f81323dd61-kube-api-access-tj4ql\") pod \"9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1zqfm2\" (UID: \"bf1fee85-d50a-4a52-84cd-e7f81323dd61\") " pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1zqfm2" Jul 07 10:35:01.148329 ip-10-0-135-34 kubenswrapper[2578]: I0707 10:35:01.148290 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1zqfm2" Jul 07 10:35:01.298722 ip-10-0-135-34 kubenswrapper[2578]: I0707 10:35:01.298696 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1zqfm2"] Jul 07 10:35:01.300958 ip-10-0-135-34 kubenswrapper[2578]: W0707 10:35:01.300928 2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbf1fee85_d50a_4a52_84cd_e7f81323dd61.slice/crio-33ad30b52f1442ad823511aae82f736f365cf4d9d80185e130c8e046faab525f WatchSource:0}: Error finding container 33ad30b52f1442ad823511aae82f736f365cf4d9d80185e130c8e046faab525f: Status 404 returned error can't find the container with id 33ad30b52f1442ad823511aae82f736f365cf4d9d80185e130c8e046faab525f Jul 07 10:35:01.404751 ip-10-0-135-34 kubenswrapper[2578]: I0707 10:35:01.404725 2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759mgzz2"] Jul 07 10:35:01.408157 ip-10-0-135-34 kubenswrapper[2578]: I0707 10:35:01.408140 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759mgzz2" Jul 07 10:35:01.429058 ip-10-0-135-34 kubenswrapper[2578]: I0707 10:35:01.429028 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759mgzz2"] Jul 07 10:35:01.496349 ip-10-0-135-34 kubenswrapper[2578]: I0707 10:35:01.496295 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/26ed6eac-aef6-43e7-9f47-b739c096a031-bundle\") pod \"0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759mgzz2\" (UID: \"26ed6eac-aef6-43e7-9f47-b739c096a031\") " pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759mgzz2" Jul 07 10:35:01.496349 ip-10-0-135-34 kubenswrapper[2578]: I0707 10:35:01.496344 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fbrcd\" (UniqueName: \"kubernetes.io/projected/26ed6eac-aef6-43e7-9f47-b739c096a031-kube-api-access-fbrcd\") pod \"0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759mgzz2\" (UID: \"26ed6eac-aef6-43e7-9f47-b739c096a031\") " pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759mgzz2" Jul 07 10:35:01.496544 ip-10-0-135-34 kubenswrapper[2578]: I0707 10:35:01.496413 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/26ed6eac-aef6-43e7-9f47-b739c096a031-util\") pod \"0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759mgzz2\" (UID: \"26ed6eac-aef6-43e7-9f47-b739c096a031\") " pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759mgzz2" Jul 07 10:35:01.597919 ip-10-0-135-34 kubenswrapper[2578]: I0707 10:35:01.597848 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/26ed6eac-aef6-43e7-9f47-b739c096a031-bundle\") pod \"0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759mgzz2\" (UID: \"26ed6eac-aef6-43e7-9f47-b739c096a031\") " pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759mgzz2" Jul 07 10:35:01.597919 ip-10-0-135-34 kubenswrapper[2578]: I0707 10:35:01.597890 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fbrcd\" (UniqueName: \"kubernetes.io/projected/26ed6eac-aef6-43e7-9f47-b739c096a031-kube-api-access-fbrcd\") pod \"0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759mgzz2\" (UID: \"26ed6eac-aef6-43e7-9f47-b739c096a031\") " pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759mgzz2" Jul 07 10:35:01.598129 ip-10-0-135-34 kubenswrapper[2578]: I0707 10:35:01.597941 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/26ed6eac-aef6-43e7-9f47-b739c096a031-util\") pod \"0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759mgzz2\" (UID: \"26ed6eac-aef6-43e7-9f47-b739c096a031\") " pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759mgzz2" Jul 07 10:35:01.598278 ip-10-0-135-34 kubenswrapper[2578]: I0707 10:35:01.598254 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/26ed6eac-aef6-43e7-9f47-b739c096a031-bundle\") pod \"0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759mgzz2\" (UID: \"26ed6eac-aef6-43e7-9f47-b739c096a031\") " pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759mgzz2" Jul 07 10:35:01.598346 ip-10-0-135-34 kubenswrapper[2578]: I0707 10:35:01.598276 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/26ed6eac-aef6-43e7-9f47-b739c096a031-util\") pod \"0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759mgzz2\" (UID: \"26ed6eac-aef6-43e7-9f47-b739c096a031\") " pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759mgzz2" Jul 07 10:35:01.609143 ip-10-0-135-34 kubenswrapper[2578]: I0707 10:35:01.609121 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fbrcd\" (UniqueName: \"kubernetes.io/projected/26ed6eac-aef6-43e7-9f47-b739c096a031-kube-api-access-fbrcd\") pod \"0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759mgzz2\" (UID: \"26ed6eac-aef6-43e7-9f47-b739c096a031\") " pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759mgzz2" Jul 07 10:35:01.770005 ip-10-0-135-34 kubenswrapper[2578]: I0707 10:35:01.769970 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759mgzz2" Jul 07 10:35:01.844340 ip-10-0-135-34 kubenswrapper[2578]: I0707 10:35:01.844308 2578 generic.go:356] "Generic (PLEG): container finished" podID="06b9a5a9-0884-43cf-ad51-7b8f0bcd255f" containerID="a92454ff838699395009abb851c458b32db4a12898002a63a3ee02f824dab118" exitCode=0 Jul 07 10:35:01.844510 ip-10-0-135-34 kubenswrapper[2578]: I0707 10:35:01.844382 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed735pdxq" event={"ID":"06b9a5a9-0884-43cf-ad51-7b8f0bcd255f","Type":"ContainerDied","Data":"a92454ff838699395009abb851c458b32db4a12898002a63a3ee02f824dab118"} Jul 07 10:35:01.846217 ip-10-0-135-34 kubenswrapper[2578]: I0707 10:35:01.846190 2578 generic.go:356] "Generic (PLEG): container finished" podID="bf1fee85-d50a-4a52-84cd-e7f81323dd61" containerID="6700d357ccbc30d38aa7f224f9a1dee5200bc2c9f10455b0f61ffc480bbf323c" exitCode=0 Jul 07 10:35:01.846357 ip-10-0-135-34 kubenswrapper[2578]: I0707 10:35:01.846223 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1zqfm2" event={"ID":"bf1fee85-d50a-4a52-84cd-e7f81323dd61","Type":"ContainerDied","Data":"6700d357ccbc30d38aa7f224f9a1dee5200bc2c9f10455b0f61ffc480bbf323c"} Jul 07 10:35:01.846357 ip-10-0-135-34 kubenswrapper[2578]: I0707 10:35:01.846277 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1zqfm2" event={"ID":"bf1fee85-d50a-4a52-84cd-e7f81323dd61","Type":"ContainerStarted","Data":"33ad30b52f1442ad823511aae82f736f365cf4d9d80185e130c8e046faab525f"} Jul 07 10:35:01.848499 ip-10-0-135-34 kubenswrapper[2578]: I0707 10:35:01.848443 2578 generic.go:356] "Generic (PLEG): container finished" podID="3fac3d82-02a4-4ddf-af58-d7392178ec4b" containerID="8dac33c346f6f80cf90700b2e5911ce84b3a74d17d8856818f21bb0841e4a68a" exitCode=0 Jul 07 10:35:01.848499 ip-10-0-135-34 kubenswrapper[2578]: I0707 10:35:01.848474 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0wrzpr" event={"ID":"3fac3d82-02a4-4ddf-af58-d7392178ec4b","Type":"ContainerDied","Data":"8dac33c346f6f80cf90700b2e5911ce84b3a74d17d8856818f21bb0841e4a68a"} Jul 07 10:35:01.941862 ip-10-0-135-34 kubenswrapper[2578]: I0707 10:35:01.941828 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759mgzz2"] Jul 07 10:35:01.944939 ip-10-0-135-34 kubenswrapper[2578]: W0707 10:35:01.944906 2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod26ed6eac_aef6_43e7_9f47_b739c096a031.slice/crio-15c2ea31ac36a4a9a3d882eab94c8248d50fb3be417265bd5ffee993f6c41f94 WatchSource:0}: Error finding container 15c2ea31ac36a4a9a3d882eab94c8248d50fb3be417265bd5ffee993f6c41f94: Status 404 returned error can't find the container with id 15c2ea31ac36a4a9a3d882eab94c8248d50fb3be417265bd5ffee993f6c41f94 Jul 07 10:35:02.854541 ip-10-0-135-34 kubenswrapper[2578]: I0707 10:35:02.854509 2578 generic.go:356] "Generic (PLEG): container finished" podID="26ed6eac-aef6-43e7-9f47-b739c096a031" containerID="d870da6eeb1b10407e8e5701188625d76522afc99bd98c20722b457b83588c36" exitCode=0 Jul 07 10:35:02.855028 ip-10-0-135-34 kubenswrapper[2578]: I0707 10:35:02.854555 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759mgzz2" event={"ID":"26ed6eac-aef6-43e7-9f47-b739c096a031","Type":"ContainerDied","Data":"d870da6eeb1b10407e8e5701188625d76522afc99bd98c20722b457b83588c36"} Jul 07 10:35:02.855028 ip-10-0-135-34 kubenswrapper[2578]: I0707 10:35:02.854617 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759mgzz2" event={"ID":"26ed6eac-aef6-43e7-9f47-b739c096a031","Type":"ContainerStarted","Data":"15c2ea31ac36a4a9a3d882eab94c8248d50fb3be417265bd5ffee993f6c41f94"} Jul 07 10:35:02.856919 ip-10-0-135-34 kubenswrapper[2578]: I0707 10:35:02.856897 2578 generic.go:356] "Generic (PLEG): container finished" podID="3fac3d82-02a4-4ddf-af58-d7392178ec4b" containerID="7c4c0bf4a028eacc95876e83c00ecd362c89d103e8e8d1a3aa97e0f5d77e96b0" exitCode=0 Jul 07 10:35:02.857012 ip-10-0-135-34 kubenswrapper[2578]: I0707 10:35:02.856985 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0wrzpr" event={"ID":"3fac3d82-02a4-4ddf-af58-d7392178ec4b","Type":"ContainerDied","Data":"7c4c0bf4a028eacc95876e83c00ecd362c89d103e8e8d1a3aa97e0f5d77e96b0"} Jul 07 10:35:02.864225 ip-10-0-135-34 kubenswrapper[2578]: I0707 10:35:02.862666 2578 generic.go:356] "Generic (PLEG): container finished" podID="06b9a5a9-0884-43cf-ad51-7b8f0bcd255f" containerID="c024a1ef317ae4fff563676d514920a20e476c36390f7727372060259a05206e" exitCode=0 Jul 07 10:35:02.864225 ip-10-0-135-34 kubenswrapper[2578]: I0707 10:35:02.862706 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed735pdxq" event={"ID":"06b9a5a9-0884-43cf-ad51-7b8f0bcd255f","Type":"ContainerDied","Data":"c024a1ef317ae4fff563676d514920a20e476c36390f7727372060259a05206e"} Jul 07 10:35:03.872620 ip-10-0-135-34 kubenswrapper[2578]: I0707 10:35:03.872589 2578 generic.go:356] "Generic (PLEG): container finished" podID="26ed6eac-aef6-43e7-9f47-b739c096a031" containerID="c6656e24abad3fd0357a538a3893e2c9edd101a037b7604941ea5817eb91c52d" exitCode=0 Jul 07 10:35:03.873061 ip-10-0-135-34 kubenswrapper[2578]: I0707 10:35:03.872680 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759mgzz2" event={"ID":"26ed6eac-aef6-43e7-9f47-b739c096a031","Type":"ContainerDied","Data":"c6656e24abad3fd0357a538a3893e2c9edd101a037b7604941ea5817eb91c52d"} Jul 07 10:35:03.874805 ip-10-0-135-34 kubenswrapper[2578]: I0707 10:35:03.874765 2578 generic.go:356] "Generic (PLEG): container finished" podID="bf1fee85-d50a-4a52-84cd-e7f81323dd61" containerID="0209a38137f1167bc75da6a4b2b4b61287e9f960238e659d94b6d3a5b9596dcc" exitCode=0 Jul 07 10:35:03.874892 ip-10-0-135-34 kubenswrapper[2578]: I0707 10:35:03.874871 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1zqfm2" event={"ID":"bf1fee85-d50a-4a52-84cd-e7f81323dd61","Type":"ContainerDied","Data":"0209a38137f1167bc75da6a4b2b4b61287e9f960238e659d94b6d3a5b9596dcc"} Jul 07 10:35:04.019329 ip-10-0-135-34 kubenswrapper[2578]: I0707 10:35:04.019250 2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0wrzpr" Jul 07 10:35:04.120493 ip-10-0-135-34 kubenswrapper[2578]: I0707 10:35:04.120458 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/3fac3d82-02a4-4ddf-af58-d7392178ec4b-util\") pod \"3fac3d82-02a4-4ddf-af58-d7392178ec4b\" (UID: \"3fac3d82-02a4-4ddf-af58-d7392178ec4b\") " Jul 07 10:35:04.120628 ip-10-0-135-34 kubenswrapper[2578]: I0707 10:35:04.120528 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-jvlmn\" (UniqueName: \"kubernetes.io/projected/3fac3d82-02a4-4ddf-af58-d7392178ec4b-kube-api-access-jvlmn\") pod \"3fac3d82-02a4-4ddf-af58-d7392178ec4b\" (UID: \"3fac3d82-02a4-4ddf-af58-d7392178ec4b\") " Jul 07 10:35:04.120628 ip-10-0-135-34 kubenswrapper[2578]: I0707 10:35:04.120561 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/3fac3d82-02a4-4ddf-af58-d7392178ec4b-bundle\") pod \"3fac3d82-02a4-4ddf-af58-d7392178ec4b\" (UID: \"3fac3d82-02a4-4ddf-af58-d7392178ec4b\") " Jul 07 10:35:04.121103 ip-10-0-135-34 kubenswrapper[2578]: I0707 10:35:04.121083 2578 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3fac3d82-02a4-4ddf-af58-d7392178ec4b-bundle" (OuterVolumeSpecName: "bundle") pod "3fac3d82-02a4-4ddf-af58-d7392178ec4b" (UID: "3fac3d82-02a4-4ddf-af58-d7392178ec4b"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 07 10:35:04.122957 ip-10-0-135-34 kubenswrapper[2578]: I0707 10:35:04.122933 2578 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3fac3d82-02a4-4ddf-af58-d7392178ec4b-kube-api-access-jvlmn" (OuterVolumeSpecName: "kube-api-access-jvlmn") pod "3fac3d82-02a4-4ddf-af58-d7392178ec4b" (UID: "3fac3d82-02a4-4ddf-af58-d7392178ec4b"). InnerVolumeSpecName "kube-api-access-jvlmn". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 07 10:35:04.125814 ip-10-0-135-34 kubenswrapper[2578]: I0707 10:35:04.125788 2578 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3fac3d82-02a4-4ddf-af58-d7392178ec4b-util" (OuterVolumeSpecName: "util") pod "3fac3d82-02a4-4ddf-af58-d7392178ec4b" (UID: "3fac3d82-02a4-4ddf-af58-d7392178ec4b"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 07 10:35:04.130689 ip-10-0-135-34 kubenswrapper[2578]: I0707 10:35:04.130670 2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed735pdxq" Jul 07 10:35:04.221884 ip-10-0-135-34 kubenswrapper[2578]: I0707 10:35:04.221841 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/06b9a5a9-0884-43cf-ad51-7b8f0bcd255f-util\") pod \"06b9a5a9-0884-43cf-ad51-7b8f0bcd255f\" (UID: \"06b9a5a9-0884-43cf-ad51-7b8f0bcd255f\") " Jul 07 10:35:04.222072 ip-10-0-135-34 kubenswrapper[2578]: I0707 10:35:04.221921 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/06b9a5a9-0884-43cf-ad51-7b8f0bcd255f-bundle\") pod \"06b9a5a9-0884-43cf-ad51-7b8f0bcd255f\" (UID: \"06b9a5a9-0884-43cf-ad51-7b8f0bcd255f\") " Jul 07 10:35:04.222072 ip-10-0-135-34 kubenswrapper[2578]: I0707 10:35:04.221960 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-4z9cs\" (UniqueName: \"kubernetes.io/projected/06b9a5a9-0884-43cf-ad51-7b8f0bcd255f-kube-api-access-4z9cs\") pod \"06b9a5a9-0884-43cf-ad51-7b8f0bcd255f\" (UID: \"06b9a5a9-0884-43cf-ad51-7b8f0bcd255f\") " Jul 07 10:35:04.222276 ip-10-0-135-34 kubenswrapper[2578]: I0707 10:35:04.222260 2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jvlmn\" (UniqueName: \"kubernetes.io/projected/3fac3d82-02a4-4ddf-af58-d7392178ec4b-kube-api-access-jvlmn\") on node \"ip-10-0-135-34.ec2.internal\" DevicePath \"\"" Jul 07 10:35:04.222333 ip-10-0-135-34 kubenswrapper[2578]: I0707 10:35:04.222281 2578 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/3fac3d82-02a4-4ddf-af58-d7392178ec4b-bundle\") on node \"ip-10-0-135-34.ec2.internal\" DevicePath \"\"" Jul 07 10:35:04.222333 ip-10-0-135-34 kubenswrapper[2578]: I0707 10:35:04.222297 2578 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/3fac3d82-02a4-4ddf-af58-d7392178ec4b-util\") on node \"ip-10-0-135-34.ec2.internal\" DevicePath \"\"" Jul 07 10:35:04.222472 ip-10-0-135-34 kubenswrapper[2578]: I0707 10:35:04.222451 2578 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/06b9a5a9-0884-43cf-ad51-7b8f0bcd255f-bundle" (OuterVolumeSpecName: "bundle") pod "06b9a5a9-0884-43cf-ad51-7b8f0bcd255f" (UID: "06b9a5a9-0884-43cf-ad51-7b8f0bcd255f"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 07 10:35:04.224279 ip-10-0-135-34 kubenswrapper[2578]: I0707 10:35:04.224257 2578 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/06b9a5a9-0884-43cf-ad51-7b8f0bcd255f-kube-api-access-4z9cs" (OuterVolumeSpecName: "kube-api-access-4z9cs") pod "06b9a5a9-0884-43cf-ad51-7b8f0bcd255f" (UID: "06b9a5a9-0884-43cf-ad51-7b8f0bcd255f"). InnerVolumeSpecName "kube-api-access-4z9cs". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 07 10:35:04.227110 ip-10-0-135-34 kubenswrapper[2578]: I0707 10:35:04.227086 2578 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/06b9a5a9-0884-43cf-ad51-7b8f0bcd255f-util" (OuterVolumeSpecName: "util") pod "06b9a5a9-0884-43cf-ad51-7b8f0bcd255f" (UID: "06b9a5a9-0884-43cf-ad51-7b8f0bcd255f"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 07 10:35:04.323424 ip-10-0-135-34 kubenswrapper[2578]: I0707 10:35:04.323334 2578 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/06b9a5a9-0884-43cf-ad51-7b8f0bcd255f-util\") on node \"ip-10-0-135-34.ec2.internal\" DevicePath \"\"" Jul 07 10:35:04.323424 ip-10-0-135-34 kubenswrapper[2578]: I0707 10:35:04.323361 2578 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/06b9a5a9-0884-43cf-ad51-7b8f0bcd255f-bundle\") on node \"ip-10-0-135-34.ec2.internal\" DevicePath \"\"" Jul 07 10:35:04.323424 ip-10-0-135-34 kubenswrapper[2578]: I0707 10:35:04.323370 2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4z9cs\" (UniqueName: \"kubernetes.io/projected/06b9a5a9-0884-43cf-ad51-7b8f0bcd255f-kube-api-access-4z9cs\") on node \"ip-10-0-135-34.ec2.internal\" DevicePath \"\"" Jul 07 10:35:04.886823 ip-10-0-135-34 kubenswrapper[2578]: I0707 10:35:04.886759 2578 generic.go:356] "Generic (PLEG): container finished" podID="bf1fee85-d50a-4a52-84cd-e7f81323dd61" containerID="071a51a067b30c136b4167a529817dd221b6ce501f5bcd9a3395684a36e59996" exitCode=0 Jul 07 10:35:04.886823 ip-10-0-135-34 kubenswrapper[2578]: I0707 10:35:04.886805 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1zqfm2" event={"ID":"bf1fee85-d50a-4a52-84cd-e7f81323dd61","Type":"ContainerDied","Data":"071a51a067b30c136b4167a529817dd221b6ce501f5bcd9a3395684a36e59996"} Jul 07 10:35:04.888942 ip-10-0-135-34 kubenswrapper[2578]: I0707 10:35:04.888913 2578 generic.go:356] "Generic (PLEG): container finished" podID="26ed6eac-aef6-43e7-9f47-b739c096a031" containerID="e7ee09ca82c4af56ae4ceface60882c04b0407dd9d3da839ba0676a04bbb767d" exitCode=0 Jul 07 10:35:04.889059 ip-10-0-135-34 kubenswrapper[2578]: I0707 10:35:04.888990 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759mgzz2" event={"ID":"26ed6eac-aef6-43e7-9f47-b739c096a031","Type":"ContainerDied","Data":"e7ee09ca82c4af56ae4ceface60882c04b0407dd9d3da839ba0676a04bbb767d"} Jul 07 10:35:04.890941 ip-10-0-135-34 kubenswrapper[2578]: I0707 10:35:04.890920 2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0wrzpr" Jul 07 10:35:04.891057 ip-10-0-135-34 kubenswrapper[2578]: I0707 10:35:04.890923 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0wrzpr" event={"ID":"3fac3d82-02a4-4ddf-af58-d7392178ec4b","Type":"ContainerDied","Data":"5043c99909e86e5e1903dfc51d7ab6771482d3dc5198ed420ed182d622b6f08a"} Jul 07 10:35:04.891057 ip-10-0-135-34 kubenswrapper[2578]: I0707 10:35:04.891031 2578 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="5043c99909e86e5e1903dfc51d7ab6771482d3dc5198ed420ed182d622b6f08a" Jul 07 10:35:04.892804 ip-10-0-135-34 kubenswrapper[2578]: I0707 10:35:04.892783 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed735pdxq" event={"ID":"06b9a5a9-0884-43cf-ad51-7b8f0bcd255f","Type":"ContainerDied","Data":"0382c3e610d817c71bd3d275785f67a589cf3c58e57c4670e2daae0d5cf208e5"} Jul 07 10:35:04.892892 ip-10-0-135-34 kubenswrapper[2578]: I0707 10:35:04.892817 2578 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="0382c3e610d817c71bd3d275785f67a589cf3c58e57c4670e2daae0d5cf208e5" Jul 07 10:35:04.892892 ip-10-0-135-34 kubenswrapper[2578]: I0707 10:35:04.892838 2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed735pdxq" Jul 07 10:35:06.025697 ip-10-0-135-34 kubenswrapper[2578]: I0707 10:35:06.025665 2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759mgzz2" Jul 07 10:35:06.056398 ip-10-0-135-34 kubenswrapper[2578]: I0707 10:35:06.056372 2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1zqfm2" Jul 07 10:35:06.139507 ip-10-0-135-34 kubenswrapper[2578]: I0707 10:35:06.139467 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-tj4ql\" (UniqueName: \"kubernetes.io/projected/bf1fee85-d50a-4a52-84cd-e7f81323dd61-kube-api-access-tj4ql\") pod \"bf1fee85-d50a-4a52-84cd-e7f81323dd61\" (UID: \"bf1fee85-d50a-4a52-84cd-e7f81323dd61\") " Jul 07 10:35:06.139507 ip-10-0-135-34 kubenswrapper[2578]: I0707 10:35:06.139506 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-fbrcd\" (UniqueName: \"kubernetes.io/projected/26ed6eac-aef6-43e7-9f47-b739c096a031-kube-api-access-fbrcd\") pod \"26ed6eac-aef6-43e7-9f47-b739c096a031\" (UID: \"26ed6eac-aef6-43e7-9f47-b739c096a031\") " Jul 07 10:35:06.139717 ip-10-0-135-34 kubenswrapper[2578]: I0707 10:35:06.139548 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/26ed6eac-aef6-43e7-9f47-b739c096a031-bundle\") pod \"26ed6eac-aef6-43e7-9f47-b739c096a031\" (UID: \"26ed6eac-aef6-43e7-9f47-b739c096a031\") " Jul 07 10:35:06.139717 ip-10-0-135-34 kubenswrapper[2578]: I0707 10:35:06.139583 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/bf1fee85-d50a-4a52-84cd-e7f81323dd61-bundle\") pod \"bf1fee85-d50a-4a52-84cd-e7f81323dd61\" (UID: \"bf1fee85-d50a-4a52-84cd-e7f81323dd61\") " Jul 07 10:35:06.139717 ip-10-0-135-34 kubenswrapper[2578]: I0707 10:35:06.139601 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/26ed6eac-aef6-43e7-9f47-b739c096a031-util\") pod \"26ed6eac-aef6-43e7-9f47-b739c096a031\" (UID: \"26ed6eac-aef6-43e7-9f47-b739c096a031\") " Jul 07 10:35:06.139717 ip-10-0-135-34 kubenswrapper[2578]: I0707 10:35:06.139667 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/bf1fee85-d50a-4a52-84cd-e7f81323dd61-util\") pod \"bf1fee85-d50a-4a52-84cd-e7f81323dd61\" (UID: \"bf1fee85-d50a-4a52-84cd-e7f81323dd61\") " Jul 07 10:35:06.140182 ip-10-0-135-34 kubenswrapper[2578]: I0707 10:35:06.140148 2578 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/26ed6eac-aef6-43e7-9f47-b739c096a031-bundle" (OuterVolumeSpecName: "bundle") pod "26ed6eac-aef6-43e7-9f47-b739c096a031" (UID: "26ed6eac-aef6-43e7-9f47-b739c096a031"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 07 10:35:06.140310 ip-10-0-135-34 kubenswrapper[2578]: I0707 10:35:06.140209 2578 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bf1fee85-d50a-4a52-84cd-e7f81323dd61-bundle" (OuterVolumeSpecName: "bundle") pod "bf1fee85-d50a-4a52-84cd-e7f81323dd61" (UID: "bf1fee85-d50a-4a52-84cd-e7f81323dd61"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 07 10:35:06.141869 ip-10-0-135-34 kubenswrapper[2578]: I0707 10:35:06.141829 2578 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bf1fee85-d50a-4a52-84cd-e7f81323dd61-kube-api-access-tj4ql" (OuterVolumeSpecName: "kube-api-access-tj4ql") pod "bf1fee85-d50a-4a52-84cd-e7f81323dd61" (UID: "bf1fee85-d50a-4a52-84cd-e7f81323dd61"). InnerVolumeSpecName "kube-api-access-tj4ql". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 07 10:35:06.142148 ip-10-0-135-34 kubenswrapper[2578]: I0707 10:35:06.142132 2578 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/26ed6eac-aef6-43e7-9f47-b739c096a031-kube-api-access-fbrcd" (OuterVolumeSpecName: "kube-api-access-fbrcd") pod "26ed6eac-aef6-43e7-9f47-b739c096a031" (UID: "26ed6eac-aef6-43e7-9f47-b739c096a031"). InnerVolumeSpecName "kube-api-access-fbrcd". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 07 10:35:06.145614 ip-10-0-135-34 kubenswrapper[2578]: I0707 10:35:06.145591 2578 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/26ed6eac-aef6-43e7-9f47-b739c096a031-util" (OuterVolumeSpecName: "util") pod "26ed6eac-aef6-43e7-9f47-b739c096a031" (UID: "26ed6eac-aef6-43e7-9f47-b739c096a031"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 07 10:35:06.147742 ip-10-0-135-34 kubenswrapper[2578]: I0707 10:35:06.147660 2578 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bf1fee85-d50a-4a52-84cd-e7f81323dd61-util" (OuterVolumeSpecName: "util") pod "bf1fee85-d50a-4a52-84cd-e7f81323dd61" (UID: "bf1fee85-d50a-4a52-84cd-e7f81323dd61"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 07 10:35:06.240506 ip-10-0-135-34 kubenswrapper[2578]: I0707 10:35:06.240450 2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tj4ql\" (UniqueName: \"kubernetes.io/projected/bf1fee85-d50a-4a52-84cd-e7f81323dd61-kube-api-access-tj4ql\") on node \"ip-10-0-135-34.ec2.internal\" DevicePath \"\"" Jul 07 10:35:06.240506 ip-10-0-135-34 kubenswrapper[2578]: I0707 10:35:06.240500 2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fbrcd\" (UniqueName: \"kubernetes.io/projected/26ed6eac-aef6-43e7-9f47-b739c096a031-kube-api-access-fbrcd\") on node \"ip-10-0-135-34.ec2.internal\" DevicePath \"\"" Jul 07 10:35:06.240506 ip-10-0-135-34 kubenswrapper[2578]: I0707 10:35:06.240513 2578 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/26ed6eac-aef6-43e7-9f47-b739c096a031-bundle\") on node \"ip-10-0-135-34.ec2.internal\" DevicePath \"\"" Jul 07 10:35:06.240506 ip-10-0-135-34 kubenswrapper[2578]: I0707 10:35:06.240524 2578 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/bf1fee85-d50a-4a52-84cd-e7f81323dd61-bundle\") on node \"ip-10-0-135-34.ec2.internal\" DevicePath \"\"" Jul 07 10:35:06.240800 ip-10-0-135-34 kubenswrapper[2578]: I0707 10:35:06.240535 2578 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/26ed6eac-aef6-43e7-9f47-b739c096a031-util\") on node \"ip-10-0-135-34.ec2.internal\" DevicePath \"\"" Jul 07 10:35:06.240800 ip-10-0-135-34 kubenswrapper[2578]: I0707 10:35:06.240544 2578 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/bf1fee85-d50a-4a52-84cd-e7f81323dd61-util\") on node \"ip-10-0-135-34.ec2.internal\" DevicePath \"\"" Jul 07 10:35:06.902718 ip-10-0-135-34 kubenswrapper[2578]: I0707 10:35:06.902685 2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1zqfm2" Jul 07 10:35:06.902718 ip-10-0-135-34 kubenswrapper[2578]: I0707 10:35:06.902693 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1zqfm2" event={"ID":"bf1fee85-d50a-4a52-84cd-e7f81323dd61","Type":"ContainerDied","Data":"33ad30b52f1442ad823511aae82f736f365cf4d9d80185e130c8e046faab525f"} Jul 07 10:35:06.902975 ip-10-0-135-34 kubenswrapper[2578]: I0707 10:35:06.902742 2578 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="33ad30b52f1442ad823511aae82f736f365cf4d9d80185e130c8e046faab525f" Jul 07 10:35:06.904456 ip-10-0-135-34 kubenswrapper[2578]: I0707 10:35:06.904439 2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759mgzz2" Jul 07 10:35:06.904456 ip-10-0-135-34 kubenswrapper[2578]: I0707 10:35:06.904455 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759mgzz2" event={"ID":"26ed6eac-aef6-43e7-9f47-b739c096a031","Type":"ContainerDied","Data":"15c2ea31ac36a4a9a3d882eab94c8248d50fb3be417265bd5ffee993f6c41f94"} Jul 07 10:35:06.904595 ip-10-0-135-34 kubenswrapper[2578]: I0707 10:35:06.904484 2578 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="15c2ea31ac36a4a9a3d882eab94c8248d50fb3be417265bd5ffee993f6c41f94" Jul 07 10:36:24.987543 ip-10-0-135-34 kubenswrapper[2578]: I0707 10:36:24.987459 2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-lv7nd"] Jul 07 10:36:24.988020 ip-10-0-135-34 kubenswrapper[2578]: I0707 10:36:24.987780 2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3fac3d82-02a4-4ddf-af58-d7392178ec4b" containerName="util" Jul 07 10:36:24.988020 ip-10-0-135-34 kubenswrapper[2578]: I0707 10:36:24.987793 2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="3fac3d82-02a4-4ddf-af58-d7392178ec4b" containerName="util" Jul 07 10:36:24.988020 ip-10-0-135-34 kubenswrapper[2578]: I0707 10:36:24.987802 2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="bf1fee85-d50a-4a52-84cd-e7f81323dd61" containerName="util" Jul 07 10:36:24.988020 ip-10-0-135-34 kubenswrapper[2578]: I0707 10:36:24.987808 2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="bf1fee85-d50a-4a52-84cd-e7f81323dd61" containerName="util" Jul 07 10:36:24.988020 ip-10-0-135-34 kubenswrapper[2578]: I0707 10:36:24.987813 2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3fac3d82-02a4-4ddf-af58-d7392178ec4b" containerName="extract" Jul 07 10:36:24.988020 ip-10-0-135-34 kubenswrapper[2578]: I0707 10:36:24.987819 2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="3fac3d82-02a4-4ddf-af58-d7392178ec4b" containerName="extract" Jul 07 10:36:24.988020 ip-10-0-135-34 kubenswrapper[2578]: I0707 10:36:24.987831 2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="bf1fee85-d50a-4a52-84cd-e7f81323dd61" containerName="extract" Jul 07 10:36:24.988020 ip-10-0-135-34 kubenswrapper[2578]: I0707 10:36:24.987836 2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="bf1fee85-d50a-4a52-84cd-e7f81323dd61" containerName="extract" Jul 07 10:36:24.988020 ip-10-0-135-34 kubenswrapper[2578]: I0707 10:36:24.987841 2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="26ed6eac-aef6-43e7-9f47-b739c096a031" containerName="pull" Jul 07 10:36:24.988020 ip-10-0-135-34 kubenswrapper[2578]: I0707 10:36:24.987846 2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="26ed6eac-aef6-43e7-9f47-b739c096a031" containerName="pull" Jul 07 10:36:24.988020 ip-10-0-135-34 kubenswrapper[2578]: I0707 10:36:24.987852 2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3fac3d82-02a4-4ddf-af58-d7392178ec4b" containerName="pull" Jul 07 10:36:24.988020 ip-10-0-135-34 kubenswrapper[2578]: I0707 10:36:24.987856 2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="3fac3d82-02a4-4ddf-af58-d7392178ec4b" containerName="pull" Jul 07 10:36:24.988020 ip-10-0-135-34 kubenswrapper[2578]: I0707 10:36:24.987864 2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="bf1fee85-d50a-4a52-84cd-e7f81323dd61" containerName="pull" Jul 07 10:36:24.988020 ip-10-0-135-34 kubenswrapper[2578]: I0707 10:36:24.987869 2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="bf1fee85-d50a-4a52-84cd-e7f81323dd61" containerName="pull" Jul 07 10:36:24.988020 ip-10-0-135-34 kubenswrapper[2578]: I0707 10:36:24.987875 2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="06b9a5a9-0884-43cf-ad51-7b8f0bcd255f" containerName="pull" Jul 07 10:36:24.988020 ip-10-0-135-34 kubenswrapper[2578]: I0707 10:36:24.987880 2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="06b9a5a9-0884-43cf-ad51-7b8f0bcd255f" containerName="pull" Jul 07 10:36:24.988020 ip-10-0-135-34 kubenswrapper[2578]: I0707 10:36:24.987884 2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="26ed6eac-aef6-43e7-9f47-b739c096a031" containerName="util" Jul 07 10:36:24.988020 ip-10-0-135-34 kubenswrapper[2578]: I0707 10:36:24.987889 2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="26ed6eac-aef6-43e7-9f47-b739c096a031" containerName="util" Jul 07 10:36:24.988020 ip-10-0-135-34 kubenswrapper[2578]: I0707 10:36:24.987895 2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="06b9a5a9-0884-43cf-ad51-7b8f0bcd255f" containerName="extract" Jul 07 10:36:24.988020 ip-10-0-135-34 kubenswrapper[2578]: I0707 10:36:24.987901 2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="06b9a5a9-0884-43cf-ad51-7b8f0bcd255f" containerName="extract" Jul 07 10:36:24.988020 ip-10-0-135-34 kubenswrapper[2578]: I0707 10:36:24.987906 2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="06b9a5a9-0884-43cf-ad51-7b8f0bcd255f" containerName="util" Jul 07 10:36:24.988020 ip-10-0-135-34 kubenswrapper[2578]: I0707 10:36:24.987911 2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="06b9a5a9-0884-43cf-ad51-7b8f0bcd255f" containerName="util" Jul 07 10:36:24.988020 ip-10-0-135-34 kubenswrapper[2578]: I0707 10:36:24.987917 2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="26ed6eac-aef6-43e7-9f47-b739c096a031" containerName="extract" Jul 07 10:36:24.988020 ip-10-0-135-34 kubenswrapper[2578]: I0707 10:36:24.987922 2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="26ed6eac-aef6-43e7-9f47-b739c096a031" containerName="extract" Jul 07 10:36:24.988020 ip-10-0-135-34 kubenswrapper[2578]: I0707 10:36:24.987970 2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="06b9a5a9-0884-43cf-ad51-7b8f0bcd255f" containerName="extract" Jul 07 10:36:24.988020 ip-10-0-135-34 kubenswrapper[2578]: I0707 10:36:24.987976 2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="bf1fee85-d50a-4a52-84cd-e7f81323dd61" containerName="extract" Jul 07 10:36:24.988020 ip-10-0-135-34 kubenswrapper[2578]: I0707 10:36:24.987983 2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="3fac3d82-02a4-4ddf-af58-d7392178ec4b" containerName="extract" Jul 07 10:36:24.988020 ip-10-0-135-34 kubenswrapper[2578]: I0707 10:36:24.987988 2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="26ed6eac-aef6-43e7-9f47-b739c096a031" containerName="extract" Jul 07 10:36:24.990619 ip-10-0-135-34 kubenswrapper[2578]: I0707 10:36:24.990597 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-54d6d988d5-lv7nd" Jul 07 10:36:24.994577 ip-10-0-135-34 kubenswrapper[2578]: I0707 10:36:24.994225 2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"openshift-service-ca.crt\"" Jul 07 10:36:24.994577 ip-10-0-135-34 kubenswrapper[2578]: I0707 10:36:24.994230 2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kube-root-ca.crt\"" Jul 07 10:36:24.998920 ip-10-0-135-34 kubenswrapper[2578]: I0707 10:36:24.995366 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"default-dockercfg-cp7dp\"" Jul 07 10:36:24.998920 ip-10-0-135-34 kubenswrapper[2578]: I0707 10:36:24.995841 2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"limitador-limits-config-limitador\"" Jul 07 10:36:25.002025 ip-10-0-135-34 kubenswrapper[2578]: I0707 10:36:25.001996 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/e594c258-2852-4c44-b46d-0b48eb7828a7-config-file\") pod \"limitador-limitador-54d6d988d5-lv7nd\" (UID: \"e594c258-2852-4c44-b46d-0b48eb7828a7\") " pod="kuadrant-system/limitador-limitador-54d6d988d5-lv7nd" Jul 07 10:36:25.002247 ip-10-0-135-34 kubenswrapper[2578]: I0707 10:36:25.002227 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ggc7m\" (UniqueName: \"kubernetes.io/projected/e594c258-2852-4c44-b46d-0b48eb7828a7-kube-api-access-ggc7m\") pod \"limitador-limitador-54d6d988d5-lv7nd\" (UID: \"e594c258-2852-4c44-b46d-0b48eb7828a7\") " pod="kuadrant-system/limitador-limitador-54d6d988d5-lv7nd" Jul 07 10:36:25.004062 ip-10-0-135-34 kubenswrapper[2578]: I0707 10:36:25.004044 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-lv7nd"] Jul 07 10:36:25.033018 ip-10-0-135-34 kubenswrapper[2578]: I0707 10:36:25.032979 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-lv7nd"] Jul 07 10:36:25.103159 ip-10-0-135-34 kubenswrapper[2578]: I0707 10:36:25.103122 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/e594c258-2852-4c44-b46d-0b48eb7828a7-config-file\") pod \"limitador-limitador-54d6d988d5-lv7nd\" (UID: \"e594c258-2852-4c44-b46d-0b48eb7828a7\") " pod="kuadrant-system/limitador-limitador-54d6d988d5-lv7nd" Jul 07 10:36:25.103329 ip-10-0-135-34 kubenswrapper[2578]: I0707 10:36:25.103170 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ggc7m\" (UniqueName: \"kubernetes.io/projected/e594c258-2852-4c44-b46d-0b48eb7828a7-kube-api-access-ggc7m\") pod \"limitador-limitador-54d6d988d5-lv7nd\" (UID: \"e594c258-2852-4c44-b46d-0b48eb7828a7\") " pod="kuadrant-system/limitador-limitador-54d6d988d5-lv7nd" Jul 07 10:36:25.103765 ip-10-0-135-34 kubenswrapper[2578]: I0707 10:36:25.103742 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/e594c258-2852-4c44-b46d-0b48eb7828a7-config-file\") pod \"limitador-limitador-54d6d988d5-lv7nd\" (UID: \"e594c258-2852-4c44-b46d-0b48eb7828a7\") " pod="kuadrant-system/limitador-limitador-54d6d988d5-lv7nd" Jul 07 10:36:25.114975 ip-10-0-135-34 kubenswrapper[2578]: I0707 10:36:25.114937 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ggc7m\" (UniqueName: \"kubernetes.io/projected/e594c258-2852-4c44-b46d-0b48eb7828a7-kube-api-access-ggc7m\") pod \"limitador-limitador-54d6d988d5-lv7nd\" (UID: \"e594c258-2852-4c44-b46d-0b48eb7828a7\") " pod="kuadrant-system/limitador-limitador-54d6d988d5-lv7nd" Jul 07 10:36:25.304744 ip-10-0-135-34 kubenswrapper[2578]: I0707 10:36:25.304652 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-54d6d988d5-lv7nd" Jul 07 10:36:25.442920 ip-10-0-135-34 kubenswrapper[2578]: I0707 10:36:25.442884 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-lv7nd"] Jul 07 10:36:25.446397 ip-10-0-135-34 kubenswrapper[2578]: W0707 10:36:25.446373 2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode594c258_2852_4c44_b46d_0b48eb7828a7.slice/crio-186052969fc544037951aecda9436c153d60547b4a4ea277e7e5e0d1bb50768b WatchSource:0}: Error finding container 186052969fc544037951aecda9436c153d60547b4a4ea277e7e5e0d1bb50768b: Status 404 returned error can't find the container with id 186052969fc544037951aecda9436c153d60547b4a4ea277e7e5e0d1bb50768b Jul 07 10:36:25.808648 ip-10-0-135-34 kubenswrapper[2578]: I0707 10:36:25.808618 2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-75d556bd77-x2dwv"] Jul 07 10:36:25.813240 ip-10-0-135-34 kubenswrapper[2578]: I0707 10:36:25.813218 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-75d556bd77-x2dwv" Jul 07 10:36:25.816196 ip-10-0-135-34 kubenswrapper[2578]: I0707 10:36:25.816170 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-authorino-dockercfg-dts5m\"" Jul 07 10:36:25.822855 ip-10-0-135-34 kubenswrapper[2578]: I0707 10:36:25.822835 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-75d556bd77-x2dwv"] Jul 07 10:36:25.908590 ip-10-0-135-34 kubenswrapper[2578]: I0707 10:36:25.908556 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5hv85\" (UniqueName: \"kubernetes.io/projected/f30f4a86-8e56-47a3-a1c7-850e5462b4c3-kube-api-access-5hv85\") pod \"authorino-75d556bd77-x2dwv\" (UID: \"f30f4a86-8e56-47a3-a1c7-850e5462b4c3\") " pod="kuadrant-system/authorino-75d556bd77-x2dwv" Jul 07 10:36:26.009883 ip-10-0-135-34 kubenswrapper[2578]: I0707 10:36:26.009838 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5hv85\" (UniqueName: \"kubernetes.io/projected/f30f4a86-8e56-47a3-a1c7-850e5462b4c3-kube-api-access-5hv85\") pod \"authorino-75d556bd77-x2dwv\" (UID: \"f30f4a86-8e56-47a3-a1c7-850e5462b4c3\") " pod="kuadrant-system/authorino-75d556bd77-x2dwv" Jul 07 10:36:26.028509 ip-10-0-135-34 kubenswrapper[2578]: I0707 10:36:26.028467 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5hv85\" (UniqueName: \"kubernetes.io/projected/f30f4a86-8e56-47a3-a1c7-850e5462b4c3-kube-api-access-5hv85\") pod \"authorino-75d556bd77-x2dwv\" (UID: \"f30f4a86-8e56-47a3-a1c7-850e5462b4c3\") " pod="kuadrant-system/authorino-75d556bd77-x2dwv" Jul 07 10:36:26.124542 ip-10-0-135-34 kubenswrapper[2578]: I0707 10:36:26.124312 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-75d556bd77-x2dwv" Jul 07 10:36:26.281958 ip-10-0-135-34 kubenswrapper[2578]: I0707 10:36:26.281868 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-54d6d988d5-lv7nd" event={"ID":"e594c258-2852-4c44-b46d-0b48eb7828a7","Type":"ContainerStarted","Data":"186052969fc544037951aecda9436c153d60547b4a4ea277e7e5e0d1bb50768b"} Jul 07 10:36:26.314879 ip-10-0-135-34 kubenswrapper[2578]: I0707 10:36:26.314801 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-75d556bd77-x2dwv"] Jul 07 10:36:26.316458 ip-10-0-135-34 kubenswrapper[2578]: W0707 10:36:26.316417 2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podf30f4a86_8e56_47a3_a1c7_850e5462b4c3.slice/crio-668ed63a633cf4262cea7be958449c63a2f984953b8e378e29fdb406b57ba06f WatchSource:0}: Error finding container 668ed63a633cf4262cea7be958449c63a2f984953b8e378e29fdb406b57ba06f: Status 404 returned error can't find the container with id 668ed63a633cf4262cea7be958449c63a2f984953b8e378e29fdb406b57ba06f Jul 07 10:36:27.290571 ip-10-0-135-34 kubenswrapper[2578]: I0707 10:36:27.290534 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-75d556bd77-x2dwv" event={"ID":"f30f4a86-8e56-47a3-a1c7-850e5462b4c3","Type":"ContainerStarted","Data":"668ed63a633cf4262cea7be958449c63a2f984953b8e378e29fdb406b57ba06f"} Jul 07 10:36:28.297955 ip-10-0-135-34 kubenswrapper[2578]: I0707 10:36:28.297920 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-54d6d988d5-lv7nd" event={"ID":"e594c258-2852-4c44-b46d-0b48eb7828a7","Type":"ContainerStarted","Data":"ab179c98a4e2040ebcbee0ab27fa3d9e0e8e77b90d93114c66cef543af7f64b3"} Jul 07 10:36:28.298403 ip-10-0-135-34 kubenswrapper[2578]: I0707 10:36:28.298067 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/limitador-limitador-54d6d988d5-lv7nd" Jul 07 10:36:28.317643 ip-10-0-135-34 kubenswrapper[2578]: I0707 10:36:28.317525 2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/limitador-limitador-54d6d988d5-lv7nd" podStartSLOduration=1.714849492 podStartE2EDuration="4.317503449s" totalImagesPullingTime="2.602653957s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-07 10:36:24 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 10:36:28.316735286 +0000 UTC m=+556.988694832" watchObservedRunningTime="2026-07-07 10:36:28.317503449 +0000 UTC m=+556.989462994" Jul 07 10:36:30.310384 ip-10-0-135-34 kubenswrapper[2578]: I0707 10:36:30.310283 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-75d556bd77-x2dwv" event={"ID":"f30f4a86-8e56-47a3-a1c7-850e5462b4c3","Type":"ContainerStarted","Data":"e1bc0dfc50708433cb776be9b510b1196f8ba703517ef2d3153a01bda6f89cb8"} Jul 07 10:36:39.304731 ip-10-0-135-34 kubenswrapper[2578]: I0707 10:36:39.304694 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/limitador-limitador-54d6d988d5-lv7nd" Jul 07 10:36:39.325806 ip-10-0-135-34 kubenswrapper[2578]: I0707 10:36:39.325743 2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/authorino-75d556bd77-x2dwv" podStartSLOduration=10.69572047 podStartE2EDuration="14.325727203s" totalImagesPullingTime="3.630006733s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-07 10:36:25 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 10:36:30.337950576 +0000 UTC m=+559.009910133" watchObservedRunningTime="2026-07-07 10:36:39.325727203 +0000 UTC m=+567.997686746" Jul 07 10:36:53.315496 ip-10-0-135-34 kubenswrapper[2578]: I0707 10:36:53.315461 2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-5d4b67c446-s5wgx"] Jul 07 10:36:53.318836 ip-10-0-135-34 kubenswrapper[2578]: I0707 10:36:53.318817 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-5d4b67c446-s5wgx" Jul 07 10:36:53.335733 ip-10-0-135-34 kubenswrapper[2578]: I0707 10:36:53.335700 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-5d4b67c446-s5wgx"] Jul 07 10:36:53.466615 ip-10-0-135-34 kubenswrapper[2578]: I0707 10:36:53.466576 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zf8sx\" (UniqueName: \"kubernetes.io/projected/4fb69bb0-3b22-4902-865a-82780824d422-kube-api-access-zf8sx\") pod \"authorino-5d4b67c446-s5wgx\" (UID: \"4fb69bb0-3b22-4902-865a-82780824d422\") " pod="kuadrant-system/authorino-5d4b67c446-s5wgx" Jul 07 10:36:53.567230 ip-10-0-135-34 kubenswrapper[2578]: I0707 10:36:53.567139 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zf8sx\" (UniqueName: \"kubernetes.io/projected/4fb69bb0-3b22-4902-865a-82780824d422-kube-api-access-zf8sx\") pod \"authorino-5d4b67c446-s5wgx\" (UID: \"4fb69bb0-3b22-4902-865a-82780824d422\") " pod="kuadrant-system/authorino-5d4b67c446-s5wgx" Jul 07 10:36:53.580410 ip-10-0-135-34 kubenswrapper[2578]: I0707 10:36:53.580376 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zf8sx\" (UniqueName: \"kubernetes.io/projected/4fb69bb0-3b22-4902-865a-82780824d422-kube-api-access-zf8sx\") pod \"authorino-5d4b67c446-s5wgx\" (UID: \"4fb69bb0-3b22-4902-865a-82780824d422\") " pod="kuadrant-system/authorino-5d4b67c446-s5wgx" Jul 07 10:36:53.628170 ip-10-0-135-34 kubenswrapper[2578]: I0707 10:36:53.628137 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-5d4b67c446-s5wgx" Jul 07 10:36:53.635244 ip-10-0-135-34 kubenswrapper[2578]: I0707 10:36:53.635215 2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-5d4b67c446-s5wgx"] Jul 07 10:36:53.785521 ip-10-0-135-34 kubenswrapper[2578]: I0707 10:36:53.785503 2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-5d4b67c446-s5wgx"] Jul 07 10:36:53.788048 ip-10-0-135-34 kubenswrapper[2578]: W0707 10:36:53.788016 2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod4fb69bb0_3b22_4902_865a_82780824d422.slice/crio-2e154609fa318ee6300ff676224ae9a7fac4af050272f3275da6ed15c8ea9cd8 WatchSource:0}: Error finding container 2e154609fa318ee6300ff676224ae9a7fac4af050272f3275da6ed15c8ea9cd8: Status 404 returned error can't find the container with id 2e154609fa318ee6300ff676224ae9a7fac4af050272f3275da6ed15c8ea9cd8 Jul 07 10:36:54.423895 ip-10-0-135-34 kubenswrapper[2578]: I0707 10:36:54.423781 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-5d4b67c446-s5wgx" event={"ID":"4fb69bb0-3b22-4902-865a-82780824d422","Type":"ContainerStarted","Data":"a0c93b593b87841f7cecbb370cc7838c75a770f53ac3ca5764c8c799431cc546"} Jul 07 10:36:54.423895 ip-10-0-135-34 kubenswrapper[2578]: I0707 10:36:54.423839 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-5d4b67c446-s5wgx" event={"ID":"4fb69bb0-3b22-4902-865a-82780824d422","Type":"ContainerStarted","Data":"2e154609fa318ee6300ff676224ae9a7fac4af050272f3275da6ed15c8ea9cd8"} Jul 07 10:36:54.424366 ip-10-0-135-34 kubenswrapper[2578]: I0707 10:36:54.423898 2578 kuberuntime_container.go:872] "Killing container with a grace period" pod="kuadrant-system/authorino-5d4b67c446-s5wgx" podUID="4fb69bb0-3b22-4902-865a-82780824d422" containerName="authorino" containerID="cri-o://a0c93b593b87841f7cecbb370cc7838c75a770f53ac3ca5764c8c799431cc546" gracePeriod=30 Jul 07 10:36:54.474937 ip-10-0-135-34 kubenswrapper[2578]: I0707 10:36:54.474888 2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/authorino-5d4b67c446-s5wgx" podStartSLOduration=1.100052329 podStartE2EDuration="1.4748699s" totalImagesPullingTime="374.817571ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-07 10:36:53 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 10:36:54.474460923 +0000 UTC m=+583.146420467" watchObservedRunningTime="2026-07-07 10:36:54.4748699 +0000 UTC m=+583.146829443" Jul 07 10:36:54.677286 ip-10-0-135-34 kubenswrapper[2578]: I0707 10:36:54.677223 2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-5d4b67c446-s5wgx" Jul 07 10:36:54.765194 ip-10-0-135-34 kubenswrapper[2578]: I0707 10:36:54.765157 2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-75d556bd77-x2dwv"] Jul 07 10:36:54.765353 ip-10-0-135-34 kubenswrapper[2578]: I0707 10:36:54.765331 2578 kuberuntime_container.go:872] "Killing container with a grace period" pod="kuadrant-system/authorino-75d556bd77-x2dwv" podUID="f30f4a86-8e56-47a3-a1c7-850e5462b4c3" containerName="authorino" containerID="cri-o://e1bc0dfc50708433cb776be9b510b1196f8ba703517ef2d3153a01bda6f89cb8" gracePeriod=30 Jul 07 10:36:54.778431 ip-10-0-135-34 kubenswrapper[2578]: I0707 10:36:54.778147 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-zf8sx\" (UniqueName: \"kubernetes.io/projected/4fb69bb0-3b22-4902-865a-82780824d422-kube-api-access-zf8sx\") pod \"4fb69bb0-3b22-4902-865a-82780824d422\" (UID: \"4fb69bb0-3b22-4902-865a-82780824d422\") " Jul 07 10:36:54.780505 ip-10-0-135-34 kubenswrapper[2578]: I0707 10:36:54.780479 2578 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4fb69bb0-3b22-4902-865a-82780824d422-kube-api-access-zf8sx" (OuterVolumeSpecName: "kube-api-access-zf8sx") pod "4fb69bb0-3b22-4902-865a-82780824d422" (UID: "4fb69bb0-3b22-4902-865a-82780824d422"). InnerVolumeSpecName "kube-api-access-zf8sx". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 07 10:36:54.879663 ip-10-0-135-34 kubenswrapper[2578]: I0707 10:36:54.879632 2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zf8sx\" (UniqueName: \"kubernetes.io/projected/4fb69bb0-3b22-4902-865a-82780824d422-kube-api-access-zf8sx\") on node \"ip-10-0-135-34.ec2.internal\" DevicePath \"\"" Jul 07 10:36:55.000670 ip-10-0-135-34 kubenswrapper[2578]: I0707 10:36:55.000645 2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-75d556bd77-x2dwv" Jul 07 10:36:55.182627 ip-10-0-135-34 kubenswrapper[2578]: I0707 10:36:55.182535 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-5hv85\" (UniqueName: \"kubernetes.io/projected/f30f4a86-8e56-47a3-a1c7-850e5462b4c3-kube-api-access-5hv85\") pod \"f30f4a86-8e56-47a3-a1c7-850e5462b4c3\" (UID: \"f30f4a86-8e56-47a3-a1c7-850e5462b4c3\") " Jul 07 10:36:55.184746 ip-10-0-135-34 kubenswrapper[2578]: I0707 10:36:55.184723 2578 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f30f4a86-8e56-47a3-a1c7-850e5462b4c3-kube-api-access-5hv85" (OuterVolumeSpecName: "kube-api-access-5hv85") pod "f30f4a86-8e56-47a3-a1c7-850e5462b4c3" (UID: "f30f4a86-8e56-47a3-a1c7-850e5462b4c3"). InnerVolumeSpecName "kube-api-access-5hv85". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 07 10:36:55.283495 ip-10-0-135-34 kubenswrapper[2578]: I0707 10:36:55.283456 2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5hv85\" (UniqueName: \"kubernetes.io/projected/f30f4a86-8e56-47a3-a1c7-850e5462b4c3-kube-api-access-5hv85\") on node \"ip-10-0-135-34.ec2.internal\" DevicePath \"\"" Jul 07 10:36:55.430397 ip-10-0-135-34 kubenswrapper[2578]: I0707 10:36:55.430363 2578 generic.go:356] "Generic (PLEG): container finished" podID="4fb69bb0-3b22-4902-865a-82780824d422" containerID="a0c93b593b87841f7cecbb370cc7838c75a770f53ac3ca5764c8c799431cc546" exitCode=0 Jul 07 10:36:55.430846 ip-10-0-135-34 kubenswrapper[2578]: I0707 10:36:55.430408 2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-5d4b67c446-s5wgx" Jul 07 10:36:55.430846 ip-10-0-135-34 kubenswrapper[2578]: I0707 10:36:55.430452 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-5d4b67c446-s5wgx" event={"ID":"4fb69bb0-3b22-4902-865a-82780824d422","Type":"ContainerDied","Data":"a0c93b593b87841f7cecbb370cc7838c75a770f53ac3ca5764c8c799431cc546"} Jul 07 10:36:55.430846 ip-10-0-135-34 kubenswrapper[2578]: I0707 10:36:55.430492 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-5d4b67c446-s5wgx" event={"ID":"4fb69bb0-3b22-4902-865a-82780824d422","Type":"ContainerDied","Data":"2e154609fa318ee6300ff676224ae9a7fac4af050272f3275da6ed15c8ea9cd8"} Jul 07 10:36:55.430846 ip-10-0-135-34 kubenswrapper[2578]: I0707 10:36:55.430514 2578 scope.go:117] "RemoveContainer" containerID="a0c93b593b87841f7cecbb370cc7838c75a770f53ac3ca5764c8c799431cc546" Jul 07 10:36:55.432049 ip-10-0-135-34 kubenswrapper[2578]: I0707 10:36:55.432022 2578 generic.go:356] "Generic (PLEG): container finished" podID="f30f4a86-8e56-47a3-a1c7-850e5462b4c3" containerID="e1bc0dfc50708433cb776be9b510b1196f8ba703517ef2d3153a01bda6f89cb8" exitCode=0 Jul 07 10:36:55.432156 ip-10-0-135-34 kubenswrapper[2578]: I0707 10:36:55.432051 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-75d556bd77-x2dwv" event={"ID":"f30f4a86-8e56-47a3-a1c7-850e5462b4c3","Type":"ContainerDied","Data":"e1bc0dfc50708433cb776be9b510b1196f8ba703517ef2d3153a01bda6f89cb8"} Jul 07 10:36:55.432156 ip-10-0-135-34 kubenswrapper[2578]: I0707 10:36:55.432079 2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-75d556bd77-x2dwv" Jul 07 10:36:55.432156 ip-10-0-135-34 kubenswrapper[2578]: I0707 10:36:55.432086 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-75d556bd77-x2dwv" event={"ID":"f30f4a86-8e56-47a3-a1c7-850e5462b4c3","Type":"ContainerDied","Data":"668ed63a633cf4262cea7be958449c63a2f984953b8e378e29fdb406b57ba06f"} Jul 07 10:36:55.440148 ip-10-0-135-34 kubenswrapper[2578]: I0707 10:36:55.440126 2578 scope.go:117] "RemoveContainer" containerID="a0c93b593b87841f7cecbb370cc7838c75a770f53ac3ca5764c8c799431cc546" Jul 07 10:36:55.440505 ip-10-0-135-34 kubenswrapper[2578]: E0707 10:36:55.440481 2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a0c93b593b87841f7cecbb370cc7838c75a770f53ac3ca5764c8c799431cc546\": container with ID starting with a0c93b593b87841f7cecbb370cc7838c75a770f53ac3ca5764c8c799431cc546 not found: ID does not exist" containerID="a0c93b593b87841f7cecbb370cc7838c75a770f53ac3ca5764c8c799431cc546" Jul 07 10:36:55.440603 ip-10-0-135-34 kubenswrapper[2578]: I0707 10:36:55.440518 2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a0c93b593b87841f7cecbb370cc7838c75a770f53ac3ca5764c8c799431cc546"} err="failed to get container status \"a0c93b593b87841f7cecbb370cc7838c75a770f53ac3ca5764c8c799431cc546\": rpc error: code = NotFound desc = could not find container \"a0c93b593b87841f7cecbb370cc7838c75a770f53ac3ca5764c8c799431cc546\": container with ID starting with a0c93b593b87841f7cecbb370cc7838c75a770f53ac3ca5764c8c799431cc546 not found: ID does not exist" Jul 07 10:36:55.440603 ip-10-0-135-34 kubenswrapper[2578]: I0707 10:36:55.440537 2578 scope.go:117] "RemoveContainer" containerID="e1bc0dfc50708433cb776be9b510b1196f8ba703517ef2d3153a01bda6f89cb8" Jul 07 10:36:55.448990 ip-10-0-135-34 kubenswrapper[2578]: I0707 10:36:55.448974 2578 scope.go:117] "RemoveContainer" containerID="e1bc0dfc50708433cb776be9b510b1196f8ba703517ef2d3153a01bda6f89cb8" Jul 07 10:36:55.449253 ip-10-0-135-34 kubenswrapper[2578]: E0707 10:36:55.449234 2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e1bc0dfc50708433cb776be9b510b1196f8ba703517ef2d3153a01bda6f89cb8\": container with ID starting with e1bc0dfc50708433cb776be9b510b1196f8ba703517ef2d3153a01bda6f89cb8 not found: ID does not exist" containerID="e1bc0dfc50708433cb776be9b510b1196f8ba703517ef2d3153a01bda6f89cb8" Jul 07 10:36:55.449315 ip-10-0-135-34 kubenswrapper[2578]: I0707 10:36:55.449259 2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e1bc0dfc50708433cb776be9b510b1196f8ba703517ef2d3153a01bda6f89cb8"} err="failed to get container status \"e1bc0dfc50708433cb776be9b510b1196f8ba703517ef2d3153a01bda6f89cb8\": rpc error: code = NotFound desc = could not find container \"e1bc0dfc50708433cb776be9b510b1196f8ba703517ef2d3153a01bda6f89cb8\": container with ID starting with e1bc0dfc50708433cb776be9b510b1196f8ba703517ef2d3153a01bda6f89cb8 not found: ID does not exist" Jul 07 10:36:55.489335 ip-10-0-135-34 kubenswrapper[2578]: I0707 10:36:55.489292 2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-75d556bd77-x2dwv"] Jul 07 10:36:55.511682 ip-10-0-135-34 kubenswrapper[2578]: I0707 10:36:55.511649 2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-75d556bd77-x2dwv"] Jul 07 10:36:55.576881 ip-10-0-135-34 kubenswrapper[2578]: I0707 10:36:55.576840 2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-5d4b67c446-s5wgx"] Jul 07 10:36:55.600821 ip-10-0-135-34 kubenswrapper[2578]: I0707 10:36:55.600784 2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-5d4b67c446-s5wgx"] Jul 07 10:36:55.821659 ip-10-0-135-34 kubenswrapper[2578]: I0707 10:36:55.821570 2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="4fb69bb0-3b22-4902-865a-82780824d422" path="/var/lib/kubelet/pods/4fb69bb0-3b22-4902-865a-82780824d422/volumes" Jul 07 10:36:55.822011 ip-10-0-135-34 kubenswrapper[2578]: I0707 10:36:55.821992 2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="f30f4a86-8e56-47a3-a1c7-850e5462b4c3" path="/var/lib/kubelet/pods/f30f4a86-8e56-47a3-a1c7-850e5462b4c3/volumes" Jul 07 10:37:11.793381 ip-10-0-135-34 kubenswrapper[2578]: I0707 10:37:11.793347 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-qhhw7_aca7528d-ddbe-4282-979f-1b496011f0e1/ovn-acl-logging/0.log" Jul 07 10:37:11.793903 ip-10-0-135-34 kubenswrapper[2578]: I0707 10:37:11.793535 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-qhhw7_aca7528d-ddbe-4282-979f-1b496011f0e1/ovn-acl-logging/0.log" Jul 07 10:37:33.984230 ip-10-0-135-34 kubenswrapper[2578]: I0707 10:37:33.984131 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-lv7nd"] Jul 07 10:37:54.527527 ip-10-0-135-34 kubenswrapper[2578]: I0707 10:37:54.527489 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-lv7nd"] Jul 07 10:38:04.924783 ip-10-0-135-34 kubenswrapper[2578]: I0707 10:38:04.924729 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-lv7nd"] Jul 07 10:38:17.982356 ip-10-0-135-34 kubenswrapper[2578]: I0707 10:38:17.982319 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-lv7nd"] Jul 07 10:38:22.227543 ip-10-0-135-34 kubenswrapper[2578]: I0707 10:38:22.227496 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-lv7nd"] Jul 07 10:38:27.485655 ip-10-0-135-34 kubenswrapper[2578]: I0707 10:38:27.485614 2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/facebook-opt-125m-simulated-kserve-74b68447df-7l6qx"] Jul 07 10:38:27.486126 ip-10-0-135-34 kubenswrapper[2578]: I0707 10:38:27.485998 2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f30f4a86-8e56-47a3-a1c7-850e5462b4c3" containerName="authorino" Jul 07 10:38:27.486126 ip-10-0-135-34 kubenswrapper[2578]: I0707 10:38:27.486010 2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="f30f4a86-8e56-47a3-a1c7-850e5462b4c3" containerName="authorino" Jul 07 10:38:27.486126 ip-10-0-135-34 kubenswrapper[2578]: I0707 10:38:27.486027 2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4fb69bb0-3b22-4902-865a-82780824d422" containerName="authorino" Jul 07 10:38:27.486126 ip-10-0-135-34 kubenswrapper[2578]: I0707 10:38:27.486032 2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="4fb69bb0-3b22-4902-865a-82780824d422" containerName="authorino" Jul 07 10:38:27.486126 ip-10-0-135-34 kubenswrapper[2578]: I0707 10:38:27.486081 2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="4fb69bb0-3b22-4902-865a-82780824d422" containerName="authorino" Jul 07 10:38:27.486126 ip-10-0-135-34 kubenswrapper[2578]: I0707 10:38:27.486089 2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="f30f4a86-8e56-47a3-a1c7-850e5462b4c3" containerName="authorino" Jul 07 10:38:27.488076 ip-10-0-135-34 kubenswrapper[2578]: I0707 10:38:27.488059 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-7l6qx" Jul 07 10:38:27.493655 ip-10-0-135-34 kubenswrapper[2578]: I0707 10:38:27.493631 2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"kube-root-ca.crt\"" Jul 07 10:38:27.493655 ip-10-0-135-34 kubenswrapper[2578]: I0707 10:38:27.493644 2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"openshift-service-ca.crt\"" Jul 07 10:38:27.493896 ip-10-0-135-34 kubenswrapper[2578]: I0707 10:38:27.493741 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"facebook-opt-125m-simulated-kserve-self-signed-certs\"" Jul 07 10:38:27.493965 ip-10-0-135-34 kubenswrapper[2578]: I0707 10:38:27.493911 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"default-dockercfg-6qgk6\"" Jul 07 10:38:27.579476 ip-10-0-135-34 kubenswrapper[2578]: I0707 10:38:27.579435 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/facebook-opt-125m-simulated-kserve-74b68447df-7l6qx"] Jul 07 10:38:27.583476 ip-10-0-135-34 kubenswrapper[2578]: I0707 10:38:27.583447 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/19a3aba6-912b-4d5e-9d59-85fc40770a05-tls-certs\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-7l6qx\" (UID: \"19a3aba6-912b-4d5e-9d59-85fc40770a05\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-7l6qx" Jul 07 10:38:27.583648 ip-10-0-135-34 kubenswrapper[2578]: I0707 10:38:27.583498 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/19a3aba6-912b-4d5e-9d59-85fc40770a05-home\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-7l6qx\" (UID: \"19a3aba6-912b-4d5e-9d59-85fc40770a05\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-7l6qx" Jul 07 10:38:27.583648 ip-10-0-135-34 kubenswrapper[2578]: I0707 10:38:27.583564 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-85qvl\" (UniqueName: \"kubernetes.io/projected/19a3aba6-912b-4d5e-9d59-85fc40770a05-kube-api-access-85qvl\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-7l6qx\" (UID: \"19a3aba6-912b-4d5e-9d59-85fc40770a05\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-7l6qx" Jul 07 10:38:27.583648 ip-10-0-135-34 kubenswrapper[2578]: I0707 10:38:27.583628 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/19a3aba6-912b-4d5e-9d59-85fc40770a05-dshm\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-7l6qx\" (UID: \"19a3aba6-912b-4d5e-9d59-85fc40770a05\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-7l6qx" Jul 07 10:38:27.583828 ip-10-0-135-34 kubenswrapper[2578]: I0707 10:38:27.583664 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/19a3aba6-912b-4d5e-9d59-85fc40770a05-tmp-dir\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-7l6qx\" (UID: \"19a3aba6-912b-4d5e-9d59-85fc40770a05\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-7l6qx" Jul 07 10:38:27.583828 ip-10-0-135-34 kubenswrapper[2578]: I0707 10:38:27.583714 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/19a3aba6-912b-4d5e-9d59-85fc40770a05-model-cache\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-7l6qx\" (UID: \"19a3aba6-912b-4d5e-9d59-85fc40770a05\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-7l6qx" Jul 07 10:38:27.684589 ip-10-0-135-34 kubenswrapper[2578]: I0707 10:38:27.684549 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/19a3aba6-912b-4d5e-9d59-85fc40770a05-dshm\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-7l6qx\" (UID: \"19a3aba6-912b-4d5e-9d59-85fc40770a05\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-7l6qx" Jul 07 10:38:27.684589 ip-10-0-135-34 kubenswrapper[2578]: I0707 10:38:27.684597 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/19a3aba6-912b-4d5e-9d59-85fc40770a05-tmp-dir\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-7l6qx\" (UID: \"19a3aba6-912b-4d5e-9d59-85fc40770a05\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-7l6qx" Jul 07 10:38:27.684863 ip-10-0-135-34 kubenswrapper[2578]: I0707 10:38:27.684630 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/19a3aba6-912b-4d5e-9d59-85fc40770a05-model-cache\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-7l6qx\" (UID: \"19a3aba6-912b-4d5e-9d59-85fc40770a05\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-7l6qx" Jul 07 10:38:27.684863 ip-10-0-135-34 kubenswrapper[2578]: I0707 10:38:27.684659 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/19a3aba6-912b-4d5e-9d59-85fc40770a05-tls-certs\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-7l6qx\" (UID: \"19a3aba6-912b-4d5e-9d59-85fc40770a05\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-7l6qx" Jul 07 10:38:27.684863 ip-10-0-135-34 kubenswrapper[2578]: I0707 10:38:27.684683 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/19a3aba6-912b-4d5e-9d59-85fc40770a05-home\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-7l6qx\" (UID: \"19a3aba6-912b-4d5e-9d59-85fc40770a05\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-7l6qx" Jul 07 10:38:27.684863 ip-10-0-135-34 kubenswrapper[2578]: I0707 10:38:27.684708 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-85qvl\" (UniqueName: \"kubernetes.io/projected/19a3aba6-912b-4d5e-9d59-85fc40770a05-kube-api-access-85qvl\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-7l6qx\" (UID: \"19a3aba6-912b-4d5e-9d59-85fc40770a05\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-7l6qx" Jul 07 10:38:27.685073 ip-10-0-135-34 kubenswrapper[2578]: I0707 10:38:27.685018 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/19a3aba6-912b-4d5e-9d59-85fc40770a05-model-cache\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-7l6qx\" (UID: \"19a3aba6-912b-4d5e-9d59-85fc40770a05\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-7l6qx" Jul 07 10:38:27.685128 ip-10-0-135-34 kubenswrapper[2578]: I0707 10:38:27.685092 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/19a3aba6-912b-4d5e-9d59-85fc40770a05-tmp-dir\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-7l6qx\" (UID: \"19a3aba6-912b-4d5e-9d59-85fc40770a05\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-7l6qx" Jul 07 10:38:27.685194 ip-10-0-135-34 kubenswrapper[2578]: I0707 10:38:27.685172 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/19a3aba6-912b-4d5e-9d59-85fc40770a05-home\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-7l6qx\" (UID: \"19a3aba6-912b-4d5e-9d59-85fc40770a05\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-7l6qx" Jul 07 10:38:27.687217 ip-10-0-135-34 kubenswrapper[2578]: I0707 10:38:27.687186 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/19a3aba6-912b-4d5e-9d59-85fc40770a05-dshm\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-7l6qx\" (UID: \"19a3aba6-912b-4d5e-9d59-85fc40770a05\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-7l6qx" Jul 07 10:38:27.687486 ip-10-0-135-34 kubenswrapper[2578]: I0707 10:38:27.687465 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/19a3aba6-912b-4d5e-9d59-85fc40770a05-tls-certs\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-7l6qx\" (UID: \"19a3aba6-912b-4d5e-9d59-85fc40770a05\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-7l6qx" Jul 07 10:38:27.699953 ip-10-0-135-34 kubenswrapper[2578]: I0707 10:38:27.699925 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-85qvl\" (UniqueName: \"kubernetes.io/projected/19a3aba6-912b-4d5e-9d59-85fc40770a05-kube-api-access-85qvl\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-7l6qx\" (UID: \"19a3aba6-912b-4d5e-9d59-85fc40770a05\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-7l6qx" Jul 07 10:38:27.798319 ip-10-0-135-34 kubenswrapper[2578]: I0707 10:38:27.798223 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-7l6qx" Jul 07 10:38:27.939216 ip-10-0-135-34 kubenswrapper[2578]: I0707 10:38:27.939188 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/facebook-opt-125m-simulated-kserve-74b68447df-7l6qx"] Jul 07 10:38:27.941645 ip-10-0-135-34 kubenswrapper[2578]: W0707 10:38:27.941617 2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod19a3aba6_912b_4d5e_9d59_85fc40770a05.slice/crio-1caa3de92c5a0ec6cbddb8f6b6c81abc0999a61c8000ba6ad016a0326bde27a1 WatchSource:0}: Error finding container 1caa3de92c5a0ec6cbddb8f6b6c81abc0999a61c8000ba6ad016a0326bde27a1: Status 404 returned error can't find the container with id 1caa3de92c5a0ec6cbddb8f6b6c81abc0999a61c8000ba6ad016a0326bde27a1 Jul 07 10:38:27.942953 ip-10-0-135-34 kubenswrapper[2578]: I0707 10:38:27.942937 2578 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Jul 07 10:38:28.892189 ip-10-0-135-34 kubenswrapper[2578]: I0707 10:38:28.892142 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-7l6qx" event={"ID":"19a3aba6-912b-4d5e-9d59-85fc40770a05","Type":"ContainerStarted","Data":"1caa3de92c5a0ec6cbddb8f6b6c81abc0999a61c8000ba6ad016a0326bde27a1"} Jul 07 10:38:29.898737 ip-10-0-135-34 kubenswrapper[2578]: I0707 10:38:29.898697 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-7l6qx" event={"ID":"19a3aba6-912b-4d5e-9d59-85fc40770a05","Type":"ContainerStarted","Data":"d92182a092698a4db22a8067138d450d9208f2f82443bce344ac1aec78433727"} Jul 07 10:38:29.918906 ip-10-0-135-34 kubenswrapper[2578]: I0707 10:38:29.918852 2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-7l6qx" podStartSLOduration=1.8968256079999999 podStartE2EDuration="2.918835147s" totalImagesPullingTime="1.022009539s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-07 10:38:27 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 10:38:29.917169413 +0000 UTC m=+678.589128957" watchObservedRunningTime="2026-07-07 10:38:29.918835147 +0000 UTC m=+678.590794691" Jul 07 10:38:29.936221 ip-10-0-135-34 kubenswrapper[2578]: I0707 10:38:29.936187 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-lv7nd"] Jul 07 10:38:37.798921 ip-10-0-135-34 kubenswrapper[2578]: I0707 10:38:37.798878 2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-7l6qx" Jul 07 10:38:37.799403 ip-10-0-135-34 kubenswrapper[2578]: I0707 10:38:37.798948 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-7l6qx" Jul 07 10:38:37.811461 ip-10-0-135-34 kubenswrapper[2578]: I0707 10:38:37.811435 2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-7l6qx" Jul 07 10:38:37.951201 ip-10-0-135-34 kubenswrapper[2578]: I0707 10:38:37.951171 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-7l6qx" Jul 07 10:38:45.190182 ip-10-0-135-34 kubenswrapper[2578]: I0707 10:38:45.190149 2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-xw6hn"] Jul 07 10:38:45.193405 ip-10-0-135-34 kubenswrapper[2578]: I0707 10:38:45.193383 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-xw6hn" Jul 07 10:38:45.196623 ip-10-0-135-34 kubenswrapper[2578]: I0707 10:38:45.196603 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"premium-simulated-simulated-premium-kserve-self-signed-certs\"" Jul 07 10:38:45.205944 ip-10-0-135-34 kubenswrapper[2578]: I0707 10:38:45.205917 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-xw6hn"] Jul 07 10:38:45.237211 ip-10-0-135-34 kubenswrapper[2578]: I0707 10:38:45.237170 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/75c93d19-f81e-4106-bb4b-1432fdaf9402-dshm\") pod \"premium-simulated-simulated-premium-kserve-5fc6fb456f-xw6hn\" (UID: \"75c93d19-f81e-4106-bb4b-1432fdaf9402\") " pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-xw6hn" Jul 07 10:38:45.237389 ip-10-0-135-34 kubenswrapper[2578]: I0707 10:38:45.237222 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/75c93d19-f81e-4106-bb4b-1432fdaf9402-tls-certs\") pod \"premium-simulated-simulated-premium-kserve-5fc6fb456f-xw6hn\" (UID: \"75c93d19-f81e-4106-bb4b-1432fdaf9402\") " pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-xw6hn" Jul 07 10:38:45.237389 ip-10-0-135-34 kubenswrapper[2578]: I0707 10:38:45.237250 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/75c93d19-f81e-4106-bb4b-1432fdaf9402-home\") pod \"premium-simulated-simulated-premium-kserve-5fc6fb456f-xw6hn\" (UID: \"75c93d19-f81e-4106-bb4b-1432fdaf9402\") " pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-xw6hn" Jul 07 10:38:45.237389 ip-10-0-135-34 kubenswrapper[2578]: I0707 10:38:45.237290 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/75c93d19-f81e-4106-bb4b-1432fdaf9402-model-cache\") pod \"premium-simulated-simulated-premium-kserve-5fc6fb456f-xw6hn\" (UID: \"75c93d19-f81e-4106-bb4b-1432fdaf9402\") " pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-xw6hn" Jul 07 10:38:45.237389 ip-10-0-135-34 kubenswrapper[2578]: I0707 10:38:45.237334 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lz6lw\" (UniqueName: \"kubernetes.io/projected/75c93d19-f81e-4106-bb4b-1432fdaf9402-kube-api-access-lz6lw\") pod \"premium-simulated-simulated-premium-kserve-5fc6fb456f-xw6hn\" (UID: \"75c93d19-f81e-4106-bb4b-1432fdaf9402\") " pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-xw6hn" Jul 07 10:38:45.237389 ip-10-0-135-34 kubenswrapper[2578]: I0707 10:38:45.237365 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/75c93d19-f81e-4106-bb4b-1432fdaf9402-tmp-dir\") pod \"premium-simulated-simulated-premium-kserve-5fc6fb456f-xw6hn\" (UID: \"75c93d19-f81e-4106-bb4b-1432fdaf9402\") " pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-xw6hn" Jul 07 10:38:45.338105 ip-10-0-135-34 kubenswrapper[2578]: I0707 10:38:45.338062 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/75c93d19-f81e-4106-bb4b-1432fdaf9402-dshm\") pod \"premium-simulated-simulated-premium-kserve-5fc6fb456f-xw6hn\" (UID: \"75c93d19-f81e-4106-bb4b-1432fdaf9402\") " pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-xw6hn" Jul 07 10:38:45.338316 ip-10-0-135-34 kubenswrapper[2578]: I0707 10:38:45.338135 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/75c93d19-f81e-4106-bb4b-1432fdaf9402-tls-certs\") pod \"premium-simulated-simulated-premium-kserve-5fc6fb456f-xw6hn\" (UID: \"75c93d19-f81e-4106-bb4b-1432fdaf9402\") " pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-xw6hn" Jul 07 10:38:45.338316 ip-10-0-135-34 kubenswrapper[2578]: I0707 10:38:45.338157 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/75c93d19-f81e-4106-bb4b-1432fdaf9402-home\") pod \"premium-simulated-simulated-premium-kserve-5fc6fb456f-xw6hn\" (UID: \"75c93d19-f81e-4106-bb4b-1432fdaf9402\") " pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-xw6hn" Jul 07 10:38:45.338316 ip-10-0-135-34 kubenswrapper[2578]: I0707 10:38:45.338174 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/75c93d19-f81e-4106-bb4b-1432fdaf9402-model-cache\") pod \"premium-simulated-simulated-premium-kserve-5fc6fb456f-xw6hn\" (UID: \"75c93d19-f81e-4106-bb4b-1432fdaf9402\") " pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-xw6hn" Jul 07 10:38:45.338316 ip-10-0-135-34 kubenswrapper[2578]: I0707 10:38:45.338202 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lz6lw\" (UniqueName: \"kubernetes.io/projected/75c93d19-f81e-4106-bb4b-1432fdaf9402-kube-api-access-lz6lw\") pod \"premium-simulated-simulated-premium-kserve-5fc6fb456f-xw6hn\" (UID: \"75c93d19-f81e-4106-bb4b-1432fdaf9402\") " pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-xw6hn" Jul 07 10:38:45.338316 ip-10-0-135-34 kubenswrapper[2578]: I0707 10:38:45.338232 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/75c93d19-f81e-4106-bb4b-1432fdaf9402-tmp-dir\") pod \"premium-simulated-simulated-premium-kserve-5fc6fb456f-xw6hn\" (UID: \"75c93d19-f81e-4106-bb4b-1432fdaf9402\") " pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-xw6hn" Jul 07 10:38:45.338585 ip-10-0-135-34 kubenswrapper[2578]: I0707 10:38:45.338561 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/75c93d19-f81e-4106-bb4b-1432fdaf9402-home\") pod \"premium-simulated-simulated-premium-kserve-5fc6fb456f-xw6hn\" (UID: \"75c93d19-f81e-4106-bb4b-1432fdaf9402\") " pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-xw6hn" Jul 07 10:38:45.338633 ip-10-0-135-34 kubenswrapper[2578]: I0707 10:38:45.338582 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/75c93d19-f81e-4106-bb4b-1432fdaf9402-tmp-dir\") pod \"premium-simulated-simulated-premium-kserve-5fc6fb456f-xw6hn\" (UID: \"75c93d19-f81e-4106-bb4b-1432fdaf9402\") " pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-xw6hn" Jul 07 10:38:45.338673 ip-10-0-135-34 kubenswrapper[2578]: I0707 10:38:45.338653 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/75c93d19-f81e-4106-bb4b-1432fdaf9402-model-cache\") pod \"premium-simulated-simulated-premium-kserve-5fc6fb456f-xw6hn\" (UID: \"75c93d19-f81e-4106-bb4b-1432fdaf9402\") " pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-xw6hn" Jul 07 10:38:45.340599 ip-10-0-135-34 kubenswrapper[2578]: I0707 10:38:45.340580 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/75c93d19-f81e-4106-bb4b-1432fdaf9402-dshm\") pod \"premium-simulated-simulated-premium-kserve-5fc6fb456f-xw6hn\" (UID: \"75c93d19-f81e-4106-bb4b-1432fdaf9402\") " pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-xw6hn" Jul 07 10:38:45.340968 ip-10-0-135-34 kubenswrapper[2578]: I0707 10:38:45.340945 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/75c93d19-f81e-4106-bb4b-1432fdaf9402-tls-certs\") pod \"premium-simulated-simulated-premium-kserve-5fc6fb456f-xw6hn\" (UID: \"75c93d19-f81e-4106-bb4b-1432fdaf9402\") " pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-xw6hn" Jul 07 10:38:45.348264 ip-10-0-135-34 kubenswrapper[2578]: I0707 10:38:45.348239 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lz6lw\" (UniqueName: \"kubernetes.io/projected/75c93d19-f81e-4106-bb4b-1432fdaf9402-kube-api-access-lz6lw\") pod \"premium-simulated-simulated-premium-kserve-5fc6fb456f-xw6hn\" (UID: \"75c93d19-f81e-4106-bb4b-1432fdaf9402\") " pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-xw6hn" Jul 07 10:38:45.502358 ip-10-0-135-34 kubenswrapper[2578]: I0707 10:38:45.502274 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-xw6hn" Jul 07 10:38:45.643568 ip-10-0-135-34 kubenswrapper[2578]: I0707 10:38:45.643534 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-xw6hn"] Jul 07 10:38:45.647269 ip-10-0-135-34 kubenswrapper[2578]: W0707 10:38:45.647234 2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod75c93d19_f81e_4106_bb4b_1432fdaf9402.slice/crio-6a31d5bc81e3ea38df44ccc9bb4f4c7ea6ad4470a9cdfbaf10a778165971b1d5 WatchSource:0}: Error finding container 6a31d5bc81e3ea38df44ccc9bb4f4c7ea6ad4470a9cdfbaf10a778165971b1d5: Status 404 returned error can't find the container with id 6a31d5bc81e3ea38df44ccc9bb4f4c7ea6ad4470a9cdfbaf10a778165971b1d5 Jul 07 10:38:45.981154 ip-10-0-135-34 kubenswrapper[2578]: I0707 10:38:45.981111 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-xw6hn" event={"ID":"75c93d19-f81e-4106-bb4b-1432fdaf9402","Type":"ContainerStarted","Data":"89b9398e449fc0d8de1a268a431508b9307817176b2a2332447df4365f4e8015"} Jul 07 10:38:45.981154 ip-10-0-135-34 kubenswrapper[2578]: I0707 10:38:45.981160 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-xw6hn" event={"ID":"75c93d19-f81e-4106-bb4b-1432fdaf9402","Type":"ContainerStarted","Data":"6a31d5bc81e3ea38df44ccc9bb4f4c7ea6ad4470a9cdfbaf10a778165971b1d5"} Jul 07 10:38:46.003514 ip-10-0-135-34 kubenswrapper[2578]: I0707 10:38:46.003451 2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-xw6hn" podStartSLOduration=0.795800758 podStartE2EDuration="1.003433878s" totalImagesPullingTime="207.63312ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-07 10:38:45 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 10:38:46.001256374 +0000 UTC m=+694.673215945" watchObservedRunningTime="2026-07-07 10:38:46.003433878 +0000 UTC m=+694.675393422" Jul 07 10:38:47.426504 ip-10-0-135-34 kubenswrapper[2578]: I0707 10:38:47.426468 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-lv7nd"] Jul 07 10:38:55.502672 ip-10-0-135-34 kubenswrapper[2578]: I0707 10:38:55.502634 2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-xw6hn" Jul 07 10:38:55.502672 ip-10-0-135-34 kubenswrapper[2578]: I0707 10:38:55.502689 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-xw6hn" Jul 07 10:38:55.515501 ip-10-0-135-34 kubenswrapper[2578]: I0707 10:38:55.515472 2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-xw6hn" Jul 07 10:38:56.044140 ip-10-0-135-34 kubenswrapper[2578]: I0707 10:38:56.044100 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-xw6hn" Jul 07 10:40:03.554708 ip-10-0-135-34 kubenswrapper[2578]: I0707 10:40:03.554654 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-lv7nd"] Jul 07 10:40:13.526289 ip-10-0-135-34 kubenswrapper[2578]: I0707 10:40:13.526249 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-lv7nd"] Jul 07 10:40:23.838074 ip-10-0-135-34 kubenswrapper[2578]: I0707 10:40:23.838043 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-lv7nd"] Jul 07 10:40:33.520239 ip-10-0-135-34 kubenswrapper[2578]: I0707 10:40:33.520150 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-lv7nd"] Jul 07 10:40:44.327181 ip-10-0-135-34 kubenswrapper[2578]: I0707 10:40:44.327141 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-lv7nd"] Jul 07 10:41:12.330361 ip-10-0-135-34 kubenswrapper[2578]: I0707 10:41:12.330310 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-lv7nd"] Jul 07 10:41:23.726812 ip-10-0-135-34 kubenswrapper[2578]: I0707 10:41:23.726759 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-lv7nd"] Jul 07 10:41:58.040221 ip-10-0-135-34 kubenswrapper[2578]: I0707 10:41:58.040185 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-lv7nd"] Jul 07 10:42:05.441744 ip-10-0-135-34 kubenswrapper[2578]: I0707 10:42:05.441661 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-lv7nd"] Jul 07 10:42:11.828457 ip-10-0-135-34 kubenswrapper[2578]: I0707 10:42:11.828430 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-qhhw7_aca7528d-ddbe-4282-979f-1b496011f0e1/ovn-acl-logging/0.log" Jul 07 10:42:11.829431 ip-10-0-135-34 kubenswrapper[2578]: I0707 10:42:11.829412 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-qhhw7_aca7528d-ddbe-4282-979f-1b496011f0e1/ovn-acl-logging/0.log" Jul 07 10:42:15.872088 ip-10-0-135-34 kubenswrapper[2578]: I0707 10:42:15.872047 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-lv7nd"] Jul 07 10:42:24.058699 ip-10-0-135-34 kubenswrapper[2578]: I0707 10:42:24.058650 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-lv7nd"] Jul 07 10:42:33.667095 ip-10-0-135-34 kubenswrapper[2578]: I0707 10:42:33.667057 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-lv7nd"] Jul 07 10:42:45.347319 ip-10-0-135-34 kubenswrapper[2578]: I0707 10:42:45.347276 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-lv7nd"] Jul 07 10:42:56.450640 ip-10-0-135-34 kubenswrapper[2578]: I0707 10:42:56.450596 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-lv7nd"] Jul 07 10:43:05.732333 ip-10-0-135-34 kubenswrapper[2578]: I0707 10:43:05.732294 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-lv7nd"] Jul 07 10:43:28.825996 ip-10-0-135-34 kubenswrapper[2578]: I0707 10:43:28.825958 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-lv7nd"] Jul 07 10:43:38.535091 ip-10-0-135-34 kubenswrapper[2578]: I0707 10:43:38.535051 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-lv7nd"] Jul 07 10:44:02.831207 ip-10-0-135-34 kubenswrapper[2578]: I0707 10:44:02.831170 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-lv7nd"] Jul 07 10:44:12.729037 ip-10-0-135-34 kubenswrapper[2578]: I0707 10:44:12.728996 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-lv7nd"] Jul 07 10:44:27.548098 ip-10-0-135-34 kubenswrapper[2578]: I0707 10:44:27.548066 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-lv7nd"] Jul 07 10:44:36.336237 ip-10-0-135-34 kubenswrapper[2578]: I0707 10:44:36.336199 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-lv7nd"] Jul 07 10:44:47.323218 ip-10-0-135-34 kubenswrapper[2578]: I0707 10:44:47.323176 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-lv7nd"] Jul 07 10:44:56.537530 ip-10-0-135-34 kubenswrapper[2578]: I0707 10:44:56.537491 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-lv7nd"] Jul 07 10:45:05.745673 ip-10-0-135-34 kubenswrapper[2578]: I0707 10:45:05.745589 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-lv7nd"] Jul 07 10:46:53.726764 ip-10-0-135-34 kubenswrapper[2578]: I0707 10:46:53.726708 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-lv7nd"] Jul 07 10:47:04.829619 ip-10-0-135-34 kubenswrapper[2578]: I0707 10:47:04.829583 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-lv7nd"] Jul 07 10:47:11.860528 ip-10-0-135-34 kubenswrapper[2578]: I0707 10:47:11.860496 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-qhhw7_aca7528d-ddbe-4282-979f-1b496011f0e1/ovn-acl-logging/0.log" Jul 07 10:47:11.861385 ip-10-0-135-34 kubenswrapper[2578]: I0707 10:47:11.861362 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-qhhw7_aca7528d-ddbe-4282-979f-1b496011f0e1/ovn-acl-logging/0.log" Jul 07 10:47:15.655893 ip-10-0-135-34 kubenswrapper[2578]: I0707 10:47:15.655860 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-lv7nd"] Jul 07 10:47:24.939268 ip-10-0-135-34 kubenswrapper[2578]: I0707 10:47:24.939229 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-lv7nd"] Jul 07 10:47:42.040175 ip-10-0-135-34 kubenswrapper[2578]: I0707 10:47:42.040139 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-lv7nd"] Jul 07 10:47:54.826077 ip-10-0-135-34 kubenswrapper[2578]: I0707 10:47:54.826041 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-lv7nd"] Jul 07 10:48:03.829538 ip-10-0-135-34 kubenswrapper[2578]: I0707 10:48:03.829457 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-lv7nd"] Jul 07 10:48:12.727151 ip-10-0-135-34 kubenswrapper[2578]: I0707 10:48:12.727112 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-lv7nd"] Jul 07 10:48:23.427273 ip-10-0-135-34 kubenswrapper[2578]: I0707 10:48:23.427233 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-lv7nd"] Jul 07 10:48:34.621284 ip-10-0-135-34 kubenswrapper[2578]: I0707 10:48:34.621245 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-lv7nd"] Jul 07 10:48:41.929987 ip-10-0-135-34 kubenswrapper[2578]: I0707 10:48:41.929952 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-lv7nd"] Jul 07 10:49:00.729170 ip-10-0-135-34 kubenswrapper[2578]: I0707 10:49:00.729135 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-lv7nd"] Jul 07 10:49:09.934011 ip-10-0-135-34 kubenswrapper[2578]: I0707 10:49:09.933974 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-lv7nd"] Jul 07 10:49:19.627341 ip-10-0-135-34 kubenswrapper[2578]: I0707 10:49:19.627301 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-lv7nd"] Jul 07 10:49:31.623042 ip-10-0-135-34 kubenswrapper[2578]: I0707 10:49:31.623004 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-lv7nd"] Jul 07 10:49:40.731969 ip-10-0-135-34 kubenswrapper[2578]: I0707 10:49:40.731925 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-lv7nd"] Jul 07 10:49:48.656366 ip-10-0-135-34 kubenswrapper[2578]: I0707 10:49:48.656325 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-lv7nd"] Jul 07 10:49:57.736798 ip-10-0-135-34 kubenswrapper[2578]: I0707 10:49:57.736748 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-lv7nd"] Jul 07 10:50:07.030985 ip-10-0-135-34 kubenswrapper[2578]: I0707 10:50:07.030944 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-lv7nd"] Jul 07 10:50:14.830148 ip-10-0-135-34 kubenswrapper[2578]: I0707 10:50:14.830105 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-lv7nd"] Jul 07 10:50:25.037878 ip-10-0-135-34 kubenswrapper[2578]: I0707 10:50:25.037841 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-lv7nd"] Jul 07 10:50:33.927709 ip-10-0-135-34 kubenswrapper[2578]: I0707 10:50:33.927675 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-lv7nd"] Jul 07 10:50:46.529865 ip-10-0-135-34 kubenswrapper[2578]: I0707 10:50:46.529830 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-lv7nd"] Jul 07 10:50:56.533974 ip-10-0-135-34 kubenswrapper[2578]: I0707 10:50:56.533930 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-lv7nd"] Jul 07 10:51:08.185746 ip-10-0-135-34 kubenswrapper[2578]: I0707 10:51:08.185663 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-lv7nd"] Jul 07 10:51:17.821791 ip-10-0-135-34 kubenswrapper[2578]: I0707 10:51:17.821751 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-lv7nd"] Jul 07 10:51:37.522730 ip-10-0-135-34 kubenswrapper[2578]: I0707 10:51:37.522689 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-lv7nd"] Jul 07 10:51:49.637497 ip-10-0-135-34 kubenswrapper[2578]: I0707 10:51:49.637462 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-lv7nd"] Jul 07 10:51:59.604764 ip-10-0-135-34 kubenswrapper[2578]: I0707 10:51:59.604717 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-lv7nd"] Jul 07 10:52:11.222329 ip-10-0-135-34 kubenswrapper[2578]: I0707 10:52:11.222288 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-lv7nd"] Jul 07 10:52:11.899138 ip-10-0-135-34 kubenswrapper[2578]: I0707 10:52:11.899107 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-qhhw7_aca7528d-ddbe-4282-979f-1b496011f0e1/ovn-acl-logging/0.log" Jul 07 10:52:11.902447 ip-10-0-135-34 kubenswrapper[2578]: I0707 10:52:11.902424 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-qhhw7_aca7528d-ddbe-4282-979f-1b496011f0e1/ovn-acl-logging/0.log" Jul 07 10:52:23.748429 ip-10-0-135-34 kubenswrapper[2578]: I0707 10:52:23.748384 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-lv7nd"] Jul 07 10:52:32.333955 ip-10-0-135-34 kubenswrapper[2578]: I0707 10:52:32.333921 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-lv7nd"] Jul 07 10:52:41.836889 ip-10-0-135-34 kubenswrapper[2578]: I0707 10:52:41.836851 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-lv7nd"] Jul 07 10:52:57.790288 ip-10-0-135-34 kubenswrapper[2578]: I0707 10:52:57.790237 2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/maas-api-e2e-ait-e1f095cc-7f8f66b8cb-cjbtz"] Jul 07 10:52:57.795208 ip-10-0-135-34 kubenswrapper[2578]: I0707 10:52:57.795171 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-api-e2e-ait-e1f095cc-7f8f66b8cb-cjbtz" Jul 07 10:52:57.798696 ip-10-0-135-34 kubenswrapper[2578]: I0707 10:52:57.798663 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-api-e2e-ait-e1f095cc-serving-cert\"" Jul 07 10:52:57.798860 ip-10-0-135-34 kubenswrapper[2578]: I0707 10:52:57.798822 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-api-dockercfg-qlm6h\"" Jul 07 10:52:57.804127 ip-10-0-135-34 kubenswrapper[2578]: I0707 10:52:57.803648 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-api-e2e-ait-e1f095cc-7f8f66b8cb-cjbtz"] Jul 07 10:52:57.901123 ip-10-0-135-34 kubenswrapper[2578]: I0707 10:52:57.901087 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/eb8ac160-3e93-4194-b510-eaff23b4d514-openshift-service-ca\") pod \"maas-api-e2e-ait-e1f095cc-7f8f66b8cb-cjbtz\" (UID: \"eb8ac160-3e93-4194-b510-eaff23b4d514\") " pod="opendatahub/maas-api-e2e-ait-e1f095cc-7f8f66b8cb-cjbtz" Jul 07 10:52:57.901305 ip-10-0-135-34 kubenswrapper[2578]: I0707 10:52:57.901134 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/eb8ac160-3e93-4194-b510-eaff23b4d514-maas-api-tls\") pod \"maas-api-e2e-ait-e1f095cc-7f8f66b8cb-cjbtz\" (UID: \"eb8ac160-3e93-4194-b510-eaff23b4d514\") " pod="opendatahub/maas-api-e2e-ait-e1f095cc-7f8f66b8cb-cjbtz" Jul 07 10:52:57.901305 ip-10-0-135-34 kubenswrapper[2578]: I0707 10:52:57.901164 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tl5vb\" (UniqueName: \"kubernetes.io/projected/eb8ac160-3e93-4194-b510-eaff23b4d514-kube-api-access-tl5vb\") pod \"maas-api-e2e-ait-e1f095cc-7f8f66b8cb-cjbtz\" (UID: \"eb8ac160-3e93-4194-b510-eaff23b4d514\") " pod="opendatahub/maas-api-e2e-ait-e1f095cc-7f8f66b8cb-cjbtz" Jul 07 10:52:58.002074 ip-10-0-135-34 kubenswrapper[2578]: I0707 10:52:58.002037 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/eb8ac160-3e93-4194-b510-eaff23b4d514-openshift-service-ca\") pod \"maas-api-e2e-ait-e1f095cc-7f8f66b8cb-cjbtz\" (UID: \"eb8ac160-3e93-4194-b510-eaff23b4d514\") " pod="opendatahub/maas-api-e2e-ait-e1f095cc-7f8f66b8cb-cjbtz" Jul 07 10:52:58.002074 ip-10-0-135-34 kubenswrapper[2578]: I0707 10:52:58.002082 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/eb8ac160-3e93-4194-b510-eaff23b4d514-maas-api-tls\") pod \"maas-api-e2e-ait-e1f095cc-7f8f66b8cb-cjbtz\" (UID: \"eb8ac160-3e93-4194-b510-eaff23b4d514\") " pod="opendatahub/maas-api-e2e-ait-e1f095cc-7f8f66b8cb-cjbtz" Jul 07 10:52:58.002294 ip-10-0-135-34 kubenswrapper[2578]: I0707 10:52:58.002207 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tl5vb\" (UniqueName: \"kubernetes.io/projected/eb8ac160-3e93-4194-b510-eaff23b4d514-kube-api-access-tl5vb\") pod \"maas-api-e2e-ait-e1f095cc-7f8f66b8cb-cjbtz\" (UID: \"eb8ac160-3e93-4194-b510-eaff23b4d514\") " pod="opendatahub/maas-api-e2e-ait-e1f095cc-7f8f66b8cb-cjbtz" Jul 07 10:52:58.002836 ip-10-0-135-34 kubenswrapper[2578]: I0707 10:52:58.002763 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/eb8ac160-3e93-4194-b510-eaff23b4d514-openshift-service-ca\") pod \"maas-api-e2e-ait-e1f095cc-7f8f66b8cb-cjbtz\" (UID: \"eb8ac160-3e93-4194-b510-eaff23b4d514\") " pod="opendatahub/maas-api-e2e-ait-e1f095cc-7f8f66b8cb-cjbtz" Jul 07 10:52:58.004651 ip-10-0-135-34 kubenswrapper[2578]: I0707 10:52:58.004626 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/eb8ac160-3e93-4194-b510-eaff23b4d514-maas-api-tls\") pod \"maas-api-e2e-ait-e1f095cc-7f8f66b8cb-cjbtz\" (UID: \"eb8ac160-3e93-4194-b510-eaff23b4d514\") " pod="opendatahub/maas-api-e2e-ait-e1f095cc-7f8f66b8cb-cjbtz" Jul 07 10:52:58.014916 ip-10-0-135-34 kubenswrapper[2578]: I0707 10:52:58.014883 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tl5vb\" (UniqueName: \"kubernetes.io/projected/eb8ac160-3e93-4194-b510-eaff23b4d514-kube-api-access-tl5vb\") pod \"maas-api-e2e-ait-e1f095cc-7f8f66b8cb-cjbtz\" (UID: \"eb8ac160-3e93-4194-b510-eaff23b4d514\") " pod="opendatahub/maas-api-e2e-ait-e1f095cc-7f8f66b8cb-cjbtz" Jul 07 10:52:58.108964 ip-10-0-135-34 kubenswrapper[2578]: I0707 10:52:58.108931 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-api-e2e-ait-e1f095cc-7f8f66b8cb-cjbtz" Jul 07 10:52:58.254029 ip-10-0-135-34 kubenswrapper[2578]: I0707 10:52:58.254004 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-api-e2e-ait-e1f095cc-7f8f66b8cb-cjbtz"] Jul 07 10:52:58.256518 ip-10-0-135-34 kubenswrapper[2578]: W0707 10:52:58.256489 2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podeb8ac160_3e93_4194_b510_eaff23b4d514.slice/crio-9f1b142876ed262508d4a222a7291241321c178b2a2966b4083e31a6aaf72d00 WatchSource:0}: Error finding container 9f1b142876ed262508d4a222a7291241321c178b2a2966b4083e31a6aaf72d00: Status 404 returned error can't find the container with id 9f1b142876ed262508d4a222a7291241321c178b2a2966b4083e31a6aaf72d00 Jul 07 10:52:58.257856 ip-10-0-135-34 kubenswrapper[2578]: I0707 10:52:58.257834 2578 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Jul 07 10:52:59.192029 ip-10-0-135-34 kubenswrapper[2578]: I0707 10:52:59.191988 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-api-e2e-ait-e1f095cc-7f8f66b8cb-cjbtz" event={"ID":"eb8ac160-3e93-4194-b510-eaff23b4d514","Type":"ContainerStarted","Data":"9f1b142876ed262508d4a222a7291241321c178b2a2966b4083e31a6aaf72d00"} Jul 07 10:53:00.057180 ip-10-0-135-34 kubenswrapper[2578]: I0707 10:53:00.057139 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-lv7nd"] Jul 07 10:53:00.199834 ip-10-0-135-34 kubenswrapper[2578]: I0707 10:53:00.199797 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-api-e2e-ait-e1f095cc-7f8f66b8cb-cjbtz" event={"ID":"eb8ac160-3e93-4194-b510-eaff23b4d514","Type":"ContainerStarted","Data":"380990607c52146fd84e1beed630f1906921992c47dd311de39dd4d76dbad7a7"} Jul 07 10:53:00.200201 ip-10-0-135-34 kubenswrapper[2578]: I0707 10:53:00.199857 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-api-e2e-ait-e1f095cc-7f8f66b8cb-cjbtz" Jul 07 10:53:00.222417 ip-10-0-135-34 kubenswrapper[2578]: I0707 10:53:00.222368 2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/maas-api-e2e-ait-e1f095cc-7f8f66b8cb-cjbtz" podStartSLOduration=1.721347369 podStartE2EDuration="3.22235076s" totalImagesPullingTime="1.501003391s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-07 10:52:57 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 10:53:00.221186822 +0000 UTC m=+1548.893146368" watchObservedRunningTime="2026-07-07 10:53:00.22235076 +0000 UTC m=+1548.894310304" Jul 07 10:53:03.059782 ip-10-0-135-34 kubenswrapper[2578]: I0707 10:53:03.059732 2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-api-e2e-ait-e1f095cc-7f8f66b8cb-cjbtz"] Jul 07 10:53:03.060279 ip-10-0-135-34 kubenswrapper[2578]: I0707 10:53:03.060028 2578 kuberuntime_container.go:872] "Killing container with a grace period" pod="opendatahub/maas-api-e2e-ait-e1f095cc-7f8f66b8cb-cjbtz" podUID="eb8ac160-3e93-4194-b510-eaff23b4d514" containerName="maas-api" containerID="cri-o://380990607c52146fd84e1beed630f1906921992c47dd311de39dd4d76dbad7a7" gracePeriod=30 Jul 07 10:53:03.215873 ip-10-0-135-34 kubenswrapper[2578]: I0707 10:53:03.215840 2578 generic.go:356] "Generic (PLEG): container finished" podID="eb8ac160-3e93-4194-b510-eaff23b4d514" containerID="380990607c52146fd84e1beed630f1906921992c47dd311de39dd4d76dbad7a7" exitCode=0 Jul 07 10:53:03.216039 ip-10-0-135-34 kubenswrapper[2578]: I0707 10:53:03.215913 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-api-e2e-ait-e1f095cc-7f8f66b8cb-cjbtz" event={"ID":"eb8ac160-3e93-4194-b510-eaff23b4d514","Type":"ContainerDied","Data":"380990607c52146fd84e1beed630f1906921992c47dd311de39dd4d76dbad7a7"} Jul 07 10:53:03.300497 ip-10-0-135-34 kubenswrapper[2578]: I0707 10:53:03.300470 2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-api-e2e-ait-e1f095cc-7f8f66b8cb-cjbtz" Jul 07 10:53:03.352622 ip-10-0-135-34 kubenswrapper[2578]: I0707 10:53:03.352597 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-tl5vb\" (UniqueName: \"kubernetes.io/projected/eb8ac160-3e93-4194-b510-eaff23b4d514-kube-api-access-tl5vb\") pod \"eb8ac160-3e93-4194-b510-eaff23b4d514\" (UID: \"eb8ac160-3e93-4194-b510-eaff23b4d514\") " Jul 07 10:53:03.352814 ip-10-0-135-34 kubenswrapper[2578]: I0707 10:53:03.352651 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/eb8ac160-3e93-4194-b510-eaff23b4d514-maas-api-tls\") pod \"eb8ac160-3e93-4194-b510-eaff23b4d514\" (UID: \"eb8ac160-3e93-4194-b510-eaff23b4d514\") " Jul 07 10:53:03.352814 ip-10-0-135-34 kubenswrapper[2578]: I0707 10:53:03.352701 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/eb8ac160-3e93-4194-b510-eaff23b4d514-openshift-service-ca\") pod \"eb8ac160-3e93-4194-b510-eaff23b4d514\" (UID: \"eb8ac160-3e93-4194-b510-eaff23b4d514\") " Jul 07 10:53:03.353142 ip-10-0-135-34 kubenswrapper[2578]: I0707 10:53:03.353114 2578 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/eb8ac160-3e93-4194-b510-eaff23b4d514-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "eb8ac160-3e93-4194-b510-eaff23b4d514" (UID: "eb8ac160-3e93-4194-b510-eaff23b4d514"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 07 10:53:03.354990 ip-10-0-135-34 kubenswrapper[2578]: I0707 10:53:03.354958 2578 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/eb8ac160-3e93-4194-b510-eaff23b4d514-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "eb8ac160-3e93-4194-b510-eaff23b4d514" (UID: "eb8ac160-3e93-4194-b510-eaff23b4d514"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 07 10:53:03.354990 ip-10-0-135-34 kubenswrapper[2578]: I0707 10:53:03.354969 2578 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/eb8ac160-3e93-4194-b510-eaff23b4d514-kube-api-access-tl5vb" (OuterVolumeSpecName: "kube-api-access-tl5vb") pod "eb8ac160-3e93-4194-b510-eaff23b4d514" (UID: "eb8ac160-3e93-4194-b510-eaff23b4d514"). InnerVolumeSpecName "kube-api-access-tl5vb". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 07 10:53:03.454354 ip-10-0-135-34 kubenswrapper[2578]: I0707 10:53:03.454313 2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tl5vb\" (UniqueName: \"kubernetes.io/projected/eb8ac160-3e93-4194-b510-eaff23b4d514-kube-api-access-tl5vb\") on node \"ip-10-0-135-34.ec2.internal\" DevicePath \"\"" Jul 07 10:53:03.454354 ip-10-0-135-34 kubenswrapper[2578]: I0707 10:53:03.454358 2578 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/eb8ac160-3e93-4194-b510-eaff23b4d514-maas-api-tls\") on node \"ip-10-0-135-34.ec2.internal\" DevicePath \"\"" Jul 07 10:53:03.454575 ip-10-0-135-34 kubenswrapper[2578]: I0707 10:53:03.454373 2578 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/eb8ac160-3e93-4194-b510-eaff23b4d514-openshift-service-ca\") on node \"ip-10-0-135-34.ec2.internal\" DevicePath \"\"" Jul 07 10:53:04.222094 ip-10-0-135-34 kubenswrapper[2578]: I0707 10:53:04.222065 2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-api-e2e-ait-e1f095cc-7f8f66b8cb-cjbtz" Jul 07 10:53:04.222511 ip-10-0-135-34 kubenswrapper[2578]: I0707 10:53:04.222055 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-api-e2e-ait-e1f095cc-7f8f66b8cb-cjbtz" event={"ID":"eb8ac160-3e93-4194-b510-eaff23b4d514","Type":"ContainerDied","Data":"9f1b142876ed262508d4a222a7291241321c178b2a2966b4083e31a6aaf72d00"} Jul 07 10:53:04.222511 ip-10-0-135-34 kubenswrapper[2578]: I0707 10:53:04.222202 2578 scope.go:117] "RemoveContainer" containerID="380990607c52146fd84e1beed630f1906921992c47dd311de39dd4d76dbad7a7" Jul 07 10:53:04.242099 ip-10-0-135-34 kubenswrapper[2578]: I0707 10:53:04.242068 2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-api-e2e-ait-e1f095cc-7f8f66b8cb-cjbtz"] Jul 07 10:53:04.246075 ip-10-0-135-34 kubenswrapper[2578]: I0707 10:53:04.246048 2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/maas-api-e2e-ait-e1f095cc-7f8f66b8cb-cjbtz"] Jul 07 10:53:05.823059 ip-10-0-135-34 kubenswrapper[2578]: I0707 10:53:05.823026 2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="eb8ac160-3e93-4194-b510-eaff23b4d514" path="/var/lib/kubelet/pods/eb8ac160-3e93-4194-b510-eaff23b4d514/volumes" Jul 07 10:53:09.736465 ip-10-0-135-34 kubenswrapper[2578]: I0707 10:53:09.736429 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-lv7nd"] Jul 07 10:53:17.738694 ip-10-0-135-34 kubenswrapper[2578]: I0707 10:53:17.738650 2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/maas-api-e2e-derive-593c3056-5b4977b7bc-ww8b4"] Jul 07 10:53:17.739180 ip-10-0-135-34 kubenswrapper[2578]: I0707 10:53:17.739165 2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="eb8ac160-3e93-4194-b510-eaff23b4d514" containerName="maas-api" Jul 07 10:53:17.739232 ip-10-0-135-34 kubenswrapper[2578]: I0707 10:53:17.739183 2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="eb8ac160-3e93-4194-b510-eaff23b4d514" containerName="maas-api" Jul 07 10:53:17.739312 ip-10-0-135-34 kubenswrapper[2578]: I0707 10:53:17.739301 2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="eb8ac160-3e93-4194-b510-eaff23b4d514" containerName="maas-api" Jul 07 10:53:17.743845 ip-10-0-135-34 kubenswrapper[2578]: I0707 10:53:17.743825 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-api-e2e-derive-593c3056-5b4977b7bc-ww8b4" Jul 07 10:53:17.747854 ip-10-0-135-34 kubenswrapper[2578]: I0707 10:53:17.747826 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-api-e2e-derive-593c3056-serving-cert\"" Jul 07 10:53:17.748005 ip-10-0-135-34 kubenswrapper[2578]: I0707 10:53:17.747853 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-api-dockercfg-qlm6h\"" Jul 07 10:53:17.764257 ip-10-0-135-34 kubenswrapper[2578]: I0707 10:53:17.764225 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-api-e2e-derive-593c3056-5b4977b7bc-ww8b4"] Jul 07 10:53:17.882399 ip-10-0-135-34 kubenswrapper[2578]: I0707 10:53:17.882360 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/bd1baacd-8710-4dd5-b6c9-83e6a2e8aa48-openshift-service-ca\") pod \"maas-api-e2e-derive-593c3056-5b4977b7bc-ww8b4\" (UID: \"bd1baacd-8710-4dd5-b6c9-83e6a2e8aa48\") " pod="opendatahub/maas-api-e2e-derive-593c3056-5b4977b7bc-ww8b4" Jul 07 10:53:17.882591 ip-10-0-135-34 kubenswrapper[2578]: I0707 10:53:17.882432 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/bd1baacd-8710-4dd5-b6c9-83e6a2e8aa48-maas-api-tls\") pod \"maas-api-e2e-derive-593c3056-5b4977b7bc-ww8b4\" (UID: \"bd1baacd-8710-4dd5-b6c9-83e6a2e8aa48\") " pod="opendatahub/maas-api-e2e-derive-593c3056-5b4977b7bc-ww8b4" Jul 07 10:53:17.882591 ip-10-0-135-34 kubenswrapper[2578]: I0707 10:53:17.882463 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xv45z\" (UniqueName: \"kubernetes.io/projected/bd1baacd-8710-4dd5-b6c9-83e6a2e8aa48-kube-api-access-xv45z\") pod \"maas-api-e2e-derive-593c3056-5b4977b7bc-ww8b4\" (UID: \"bd1baacd-8710-4dd5-b6c9-83e6a2e8aa48\") " pod="opendatahub/maas-api-e2e-derive-593c3056-5b4977b7bc-ww8b4" Jul 07 10:53:17.983854 ip-10-0-135-34 kubenswrapper[2578]: I0707 10:53:17.983816 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/bd1baacd-8710-4dd5-b6c9-83e6a2e8aa48-openshift-service-ca\") pod \"maas-api-e2e-derive-593c3056-5b4977b7bc-ww8b4\" (UID: \"bd1baacd-8710-4dd5-b6c9-83e6a2e8aa48\") " pod="opendatahub/maas-api-e2e-derive-593c3056-5b4977b7bc-ww8b4" Jul 07 10:53:17.984037 ip-10-0-135-34 kubenswrapper[2578]: I0707 10:53:17.983882 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/bd1baacd-8710-4dd5-b6c9-83e6a2e8aa48-maas-api-tls\") pod \"maas-api-e2e-derive-593c3056-5b4977b7bc-ww8b4\" (UID: \"bd1baacd-8710-4dd5-b6c9-83e6a2e8aa48\") " pod="opendatahub/maas-api-e2e-derive-593c3056-5b4977b7bc-ww8b4" Jul 07 10:53:17.984037 ip-10-0-135-34 kubenswrapper[2578]: I0707 10:53:17.983914 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xv45z\" (UniqueName: \"kubernetes.io/projected/bd1baacd-8710-4dd5-b6c9-83e6a2e8aa48-kube-api-access-xv45z\") pod \"maas-api-e2e-derive-593c3056-5b4977b7bc-ww8b4\" (UID: \"bd1baacd-8710-4dd5-b6c9-83e6a2e8aa48\") " pod="opendatahub/maas-api-e2e-derive-593c3056-5b4977b7bc-ww8b4" Jul 07 10:53:17.984037 ip-10-0-135-34 kubenswrapper[2578]: E0707 10:53:17.984029 2578 secret.go:189] Couldn't get secret opendatahub/maas-api-e2e-derive-593c3056-serving-cert: secret "maas-api-e2e-derive-593c3056-serving-cert" not found Jul 07 10:53:17.984147 ip-10-0-135-34 kubenswrapper[2578]: E0707 10:53:17.984106 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/bd1baacd-8710-4dd5-b6c9-83e6a2e8aa48-maas-api-tls podName:bd1baacd-8710-4dd5-b6c9-83e6a2e8aa48 nodeName:}" failed. No retries permitted until 2026-07-07 10:53:18.484086191 +0000 UTC m=+1567.156045714 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/bd1baacd-8710-4dd5-b6c9-83e6a2e8aa48-maas-api-tls") pod "maas-api-e2e-derive-593c3056-5b4977b7bc-ww8b4" (UID: "bd1baacd-8710-4dd5-b6c9-83e6a2e8aa48") : secret "maas-api-e2e-derive-593c3056-serving-cert" not found Jul 07 10:53:17.984496 ip-10-0-135-34 kubenswrapper[2578]: I0707 10:53:17.984476 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/bd1baacd-8710-4dd5-b6c9-83e6a2e8aa48-openshift-service-ca\") pod \"maas-api-e2e-derive-593c3056-5b4977b7bc-ww8b4\" (UID: \"bd1baacd-8710-4dd5-b6c9-83e6a2e8aa48\") " pod="opendatahub/maas-api-e2e-derive-593c3056-5b4977b7bc-ww8b4" Jul 07 10:53:17.999733 ip-10-0-135-34 kubenswrapper[2578]: I0707 10:53:17.999653 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xv45z\" (UniqueName: \"kubernetes.io/projected/bd1baacd-8710-4dd5-b6c9-83e6a2e8aa48-kube-api-access-xv45z\") pod \"maas-api-e2e-derive-593c3056-5b4977b7bc-ww8b4\" (UID: \"bd1baacd-8710-4dd5-b6c9-83e6a2e8aa48\") " pod="opendatahub/maas-api-e2e-derive-593c3056-5b4977b7bc-ww8b4" Jul 07 10:53:18.488395 ip-10-0-135-34 kubenswrapper[2578]: I0707 10:53:18.488363 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/bd1baacd-8710-4dd5-b6c9-83e6a2e8aa48-maas-api-tls\") pod \"maas-api-e2e-derive-593c3056-5b4977b7bc-ww8b4\" (UID: \"bd1baacd-8710-4dd5-b6c9-83e6a2e8aa48\") " pod="opendatahub/maas-api-e2e-derive-593c3056-5b4977b7bc-ww8b4" Jul 07 10:53:18.490999 ip-10-0-135-34 kubenswrapper[2578]: I0707 10:53:18.490968 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/bd1baacd-8710-4dd5-b6c9-83e6a2e8aa48-maas-api-tls\") pod \"maas-api-e2e-derive-593c3056-5b4977b7bc-ww8b4\" (UID: \"bd1baacd-8710-4dd5-b6c9-83e6a2e8aa48\") " pod="opendatahub/maas-api-e2e-derive-593c3056-5b4977b7bc-ww8b4" Jul 07 10:53:18.657151 ip-10-0-135-34 kubenswrapper[2578]: I0707 10:53:18.657102 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-api-e2e-derive-593c3056-5b4977b7bc-ww8b4" Jul 07 10:53:18.819862 ip-10-0-135-34 kubenswrapper[2578]: I0707 10:53:18.819822 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-api-e2e-derive-593c3056-5b4977b7bc-ww8b4"] Jul 07 10:53:18.820756 ip-10-0-135-34 kubenswrapper[2578]: W0707 10:53:18.820731 2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbd1baacd_8710_4dd5_b6c9_83e6a2e8aa48.slice/crio-f17be7165883aeab336a7eb6c5c829b8d78be4859b1849d03f118bac6d826907 WatchSource:0}: Error finding container f17be7165883aeab336a7eb6c5c829b8d78be4859b1849d03f118bac6d826907: Status 404 returned error can't find the container with id f17be7165883aeab336a7eb6c5c829b8d78be4859b1849d03f118bac6d826907 Jul 07 10:53:19.299266 ip-10-0-135-34 kubenswrapper[2578]: I0707 10:53:19.299221 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-api-e2e-derive-593c3056-5b4977b7bc-ww8b4" event={"ID":"bd1baacd-8710-4dd5-b6c9-83e6a2e8aa48","Type":"ContainerStarted","Data":"d37dafa1103cdc126880b8676de8832b6881c12fda1f069f80a65fd1b48a136e"} Jul 07 10:53:19.299413 ip-10-0-135-34 kubenswrapper[2578]: I0707 10:53:19.299306 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-api-e2e-derive-593c3056-5b4977b7bc-ww8b4" event={"ID":"bd1baacd-8710-4dd5-b6c9-83e6a2e8aa48","Type":"ContainerStarted","Data":"f17be7165883aeab336a7eb6c5c829b8d78be4859b1849d03f118bac6d826907"} Jul 07 10:53:19.299413 ip-10-0-135-34 kubenswrapper[2578]: I0707 10:53:19.299328 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-api-e2e-derive-593c3056-5b4977b7bc-ww8b4" Jul 07 10:53:19.414202 ip-10-0-135-34 kubenswrapper[2578]: I0707 10:53:19.414090 2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/maas-api-e2e-derive-593c3056-5b4977b7bc-ww8b4" podStartSLOduration=2.074598124 podStartE2EDuration="2.414075486s" totalImagesPullingTime="339.477362ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-07 10:53:17 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 10:53:19.411041649 +0000 UTC m=+1568.083001194" watchObservedRunningTime="2026-07-07 10:53:19.414075486 +0000 UTC m=+1568.086035065" Jul 07 10:53:21.303291 ip-10-0-135-34 kubenswrapper[2578]: I0707 10:53:21.303244 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-lv7nd"] Jul 07 10:53:22.551875 ip-10-0-135-34 kubenswrapper[2578]: I0707 10:53:22.551838 2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-api-e2e-derive-593c3056-5b4977b7bc-ww8b4"] Jul 07 10:53:22.552296 ip-10-0-135-34 kubenswrapper[2578]: I0707 10:53:22.552028 2578 kuberuntime_container.go:872] "Killing container with a grace period" pod="opendatahub/maas-api-e2e-derive-593c3056-5b4977b7bc-ww8b4" podUID="bd1baacd-8710-4dd5-b6c9-83e6a2e8aa48" containerName="maas-api" containerID="cri-o://d37dafa1103cdc126880b8676de8832b6881c12fda1f069f80a65fd1b48a136e" gracePeriod=30 Jul 07 10:53:22.796413 ip-10-0-135-34 kubenswrapper[2578]: I0707 10:53:22.796387 2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-api-e2e-derive-593c3056-5b4977b7bc-ww8b4" Jul 07 10:53:22.930741 ip-10-0-135-34 kubenswrapper[2578]: I0707 10:53:22.930705 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/bd1baacd-8710-4dd5-b6c9-83e6a2e8aa48-openshift-service-ca\") pod \"bd1baacd-8710-4dd5-b6c9-83e6a2e8aa48\" (UID: \"bd1baacd-8710-4dd5-b6c9-83e6a2e8aa48\") " Jul 07 10:53:22.930741 ip-10-0-135-34 kubenswrapper[2578]: I0707 10:53:22.930743 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/bd1baacd-8710-4dd5-b6c9-83e6a2e8aa48-maas-api-tls\") pod \"bd1baacd-8710-4dd5-b6c9-83e6a2e8aa48\" (UID: \"bd1baacd-8710-4dd5-b6c9-83e6a2e8aa48\") " Jul 07 10:53:22.931033 ip-10-0-135-34 kubenswrapper[2578]: I0707 10:53:22.930821 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-xv45z\" (UniqueName: \"kubernetes.io/projected/bd1baacd-8710-4dd5-b6c9-83e6a2e8aa48-kube-api-access-xv45z\") pod \"bd1baacd-8710-4dd5-b6c9-83e6a2e8aa48\" (UID: \"bd1baacd-8710-4dd5-b6c9-83e6a2e8aa48\") " Jul 07 10:53:22.931191 ip-10-0-135-34 kubenswrapper[2578]: I0707 10:53:22.931165 2578 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bd1baacd-8710-4dd5-b6c9-83e6a2e8aa48-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "bd1baacd-8710-4dd5-b6c9-83e6a2e8aa48" (UID: "bd1baacd-8710-4dd5-b6c9-83e6a2e8aa48"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 07 10:53:22.933171 ip-10-0-135-34 kubenswrapper[2578]: I0707 10:53:22.933139 2578 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/bd1baacd-8710-4dd5-b6c9-83e6a2e8aa48-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "bd1baacd-8710-4dd5-b6c9-83e6a2e8aa48" (UID: "bd1baacd-8710-4dd5-b6c9-83e6a2e8aa48"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 07 10:53:22.933286 ip-10-0-135-34 kubenswrapper[2578]: I0707 10:53:22.933184 2578 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bd1baacd-8710-4dd5-b6c9-83e6a2e8aa48-kube-api-access-xv45z" (OuterVolumeSpecName: "kube-api-access-xv45z") pod "bd1baacd-8710-4dd5-b6c9-83e6a2e8aa48" (UID: "bd1baacd-8710-4dd5-b6c9-83e6a2e8aa48"). InnerVolumeSpecName "kube-api-access-xv45z". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 07 10:53:23.031477 ip-10-0-135-34 kubenswrapper[2578]: I0707 10:53:23.031442 2578 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/bd1baacd-8710-4dd5-b6c9-83e6a2e8aa48-openshift-service-ca\") on node \"ip-10-0-135-34.ec2.internal\" DevicePath \"\"" Jul 07 10:53:23.031477 ip-10-0-135-34 kubenswrapper[2578]: I0707 10:53:23.031470 2578 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/bd1baacd-8710-4dd5-b6c9-83e6a2e8aa48-maas-api-tls\") on node \"ip-10-0-135-34.ec2.internal\" DevicePath \"\"" Jul 07 10:53:23.031477 ip-10-0-135-34 kubenswrapper[2578]: I0707 10:53:23.031479 2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xv45z\" (UniqueName: \"kubernetes.io/projected/bd1baacd-8710-4dd5-b6c9-83e6a2e8aa48-kube-api-access-xv45z\") on node \"ip-10-0-135-34.ec2.internal\" DevicePath \"\"" Jul 07 10:53:23.318968 ip-10-0-135-34 kubenswrapper[2578]: I0707 10:53:23.318875 2578 generic.go:356] "Generic (PLEG): container finished" podID="bd1baacd-8710-4dd5-b6c9-83e6a2e8aa48" containerID="d37dafa1103cdc126880b8676de8832b6881c12fda1f069f80a65fd1b48a136e" exitCode=0 Jul 07 10:53:23.318968 ip-10-0-135-34 kubenswrapper[2578]: I0707 10:53:23.318916 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-api-e2e-derive-593c3056-5b4977b7bc-ww8b4" event={"ID":"bd1baacd-8710-4dd5-b6c9-83e6a2e8aa48","Type":"ContainerDied","Data":"d37dafa1103cdc126880b8676de8832b6881c12fda1f069f80a65fd1b48a136e"} Jul 07 10:53:23.318968 ip-10-0-135-34 kubenswrapper[2578]: I0707 10:53:23.318919 2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-api-e2e-derive-593c3056-5b4977b7bc-ww8b4" Jul 07 10:53:23.318968 ip-10-0-135-34 kubenswrapper[2578]: I0707 10:53:23.318955 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-api-e2e-derive-593c3056-5b4977b7bc-ww8b4" event={"ID":"bd1baacd-8710-4dd5-b6c9-83e6a2e8aa48","Type":"ContainerDied","Data":"f17be7165883aeab336a7eb6c5c829b8d78be4859b1849d03f118bac6d826907"} Jul 07 10:53:23.319219 ip-10-0-135-34 kubenswrapper[2578]: I0707 10:53:23.318977 2578 scope.go:117] "RemoveContainer" containerID="d37dafa1103cdc126880b8676de8832b6881c12fda1f069f80a65fd1b48a136e" Jul 07 10:53:23.328803 ip-10-0-135-34 kubenswrapper[2578]: I0707 10:53:23.328762 2578 scope.go:117] "RemoveContainer" containerID="d37dafa1103cdc126880b8676de8832b6881c12fda1f069f80a65fd1b48a136e" Jul 07 10:53:23.329088 ip-10-0-135-34 kubenswrapper[2578]: E0707 10:53:23.329068 2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d37dafa1103cdc126880b8676de8832b6881c12fda1f069f80a65fd1b48a136e\": container with ID starting with d37dafa1103cdc126880b8676de8832b6881c12fda1f069f80a65fd1b48a136e not found: ID does not exist" containerID="d37dafa1103cdc126880b8676de8832b6881c12fda1f069f80a65fd1b48a136e" Jul 07 10:53:23.329156 ip-10-0-135-34 kubenswrapper[2578]: I0707 10:53:23.329098 2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d37dafa1103cdc126880b8676de8832b6881c12fda1f069f80a65fd1b48a136e"} err="failed to get container status \"d37dafa1103cdc126880b8676de8832b6881c12fda1f069f80a65fd1b48a136e\": rpc error: code = NotFound desc = could not find container \"d37dafa1103cdc126880b8676de8832b6881c12fda1f069f80a65fd1b48a136e\": container with ID starting with d37dafa1103cdc126880b8676de8832b6881c12fda1f069f80a65fd1b48a136e not found: ID does not exist" Jul 07 10:53:23.356070 ip-10-0-135-34 kubenswrapper[2578]: I0707 10:53:23.356030 2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-api-e2e-derive-593c3056-5b4977b7bc-ww8b4"] Jul 07 10:53:23.364115 ip-10-0-135-34 kubenswrapper[2578]: I0707 10:53:23.364081 2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/maas-api-e2e-derive-593c3056-5b4977b7bc-ww8b4"] Jul 07 10:53:23.821804 ip-10-0-135-34 kubenswrapper[2578]: I0707 10:53:23.821734 2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="bd1baacd-8710-4dd5-b6c9-83e6a2e8aa48" path="/var/lib/kubelet/pods/bd1baacd-8710-4dd5-b6c9-83e6a2e8aa48/volumes" Jul 07 10:53:29.799321 ip-10-0-135-34 kubenswrapper[2578]: I0707 10:53:29.799284 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-lv7nd"] Jul 07 10:54:52.439864 ip-10-0-135-34 kubenswrapper[2578]: I0707 10:54:52.439825 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-lv7nd"] Jul 07 10:54:59.937209 ip-10-0-135-34 kubenswrapper[2578]: I0707 10:54:59.937171 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-lv7nd"] Jul 07 10:55:08.524633 ip-10-0-135-34 kubenswrapper[2578]: I0707 10:55:08.524596 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-lv7nd"] Jul 07 10:55:18.024118 ip-10-0-135-34 kubenswrapper[2578]: I0707 10:55:18.024073 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-lv7nd"] Jul 07 10:55:24.669418 ip-10-0-135-34 kubenswrapper[2578]: I0707 10:55:24.669383 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-lv7nd"] Jul 07 10:55:58.215066 ip-10-0-135-34 kubenswrapper[2578]: I0707 10:55:58.215029 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-lv7nd"] Jul 07 10:56:02.608383 ip-10-0-135-34 kubenswrapper[2578]: I0707 10:56:02.608351 2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-mt-17d0cfdd/e2e-lifecycle-model-17d0cfdd-kserve-756bf9f76c-4hpj7"] Jul 07 10:56:02.608744 ip-10-0-135-34 kubenswrapper[2578]: I0707 10:56:02.608708 2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="bd1baacd-8710-4dd5-b6c9-83e6a2e8aa48" containerName="maas-api" Jul 07 10:56:02.608744 ip-10-0-135-34 kubenswrapper[2578]: I0707 10:56:02.608719 2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="bd1baacd-8710-4dd5-b6c9-83e6a2e8aa48" containerName="maas-api" Jul 07 10:56:02.608847 ip-10-0-135-34 kubenswrapper[2578]: I0707 10:56:02.608796 2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="bd1baacd-8710-4dd5-b6c9-83e6a2e8aa48" containerName="maas-api" Jul 07 10:56:02.611684 ip-10-0-135-34 kubenswrapper[2578]: I0707 10:56:02.611664 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-mt-17d0cfdd/e2e-lifecycle-model-17d0cfdd-kserve-756bf9f76c-4hpj7" Jul 07 10:56:02.615743 ip-10-0-135-34 kubenswrapper[2578]: I0707 10:56:02.615718 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-mt-17d0cfdd\"/\"default-dockercfg-4k9rq\"" Jul 07 10:56:02.615900 ip-10-0-135-34 kubenswrapper[2578]: I0707 10:56:02.615746 2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-mt-17d0cfdd\"/\"kube-root-ca.crt\"" Jul 07 10:56:02.615900 ip-10-0-135-34 kubenswrapper[2578]: I0707 10:56:02.615789 2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-mt-17d0cfdd\"/\"openshift-service-ca.crt\"" Jul 07 10:56:02.615900 ip-10-0-135-34 kubenswrapper[2578]: I0707 10:56:02.615803 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-mt-17d0cfdd\"/\"e2e-lifecycle-model-17d0cfdd-kserve-self-signed-certs\"" Jul 07 10:56:02.622105 ip-10-0-135-34 kubenswrapper[2578]: I0707 10:56:02.622080 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-mt-17d0cfdd/e2e-lifecycle-model-17d0cfdd-kserve-756bf9f76c-4hpj7"] Jul 07 10:56:02.632302 ip-10-0-135-34 kubenswrapper[2578]: I0707 10:56:02.632274 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ncf9n\" (UniqueName: \"kubernetes.io/projected/a01b2bef-e3fe-4037-96fa-124a3e5c29f9-kube-api-access-ncf9n\") pod \"e2e-lifecycle-model-17d0cfdd-kserve-756bf9f76c-4hpj7\" (UID: \"a01b2bef-e3fe-4037-96fa-124a3e5c29f9\") " pod="ai-tenant-e2e-mt-17d0cfdd/e2e-lifecycle-model-17d0cfdd-kserve-756bf9f76c-4hpj7" Jul 07 10:56:02.632464 ip-10-0-135-34 kubenswrapper[2578]: I0707 10:56:02.632320 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a01b2bef-e3fe-4037-96fa-124a3e5c29f9-home\") pod \"e2e-lifecycle-model-17d0cfdd-kserve-756bf9f76c-4hpj7\" (UID: \"a01b2bef-e3fe-4037-96fa-124a3e5c29f9\") " pod="ai-tenant-e2e-mt-17d0cfdd/e2e-lifecycle-model-17d0cfdd-kserve-756bf9f76c-4hpj7" Jul 07 10:56:02.632464 ip-10-0-135-34 kubenswrapper[2578]: I0707 10:56:02.632380 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a01b2bef-e3fe-4037-96fa-124a3e5c29f9-tls-certs\") pod \"e2e-lifecycle-model-17d0cfdd-kserve-756bf9f76c-4hpj7\" (UID: \"a01b2bef-e3fe-4037-96fa-124a3e5c29f9\") " pod="ai-tenant-e2e-mt-17d0cfdd/e2e-lifecycle-model-17d0cfdd-kserve-756bf9f76c-4hpj7" Jul 07 10:56:02.632464 ip-10-0-135-34 kubenswrapper[2578]: I0707 10:56:02.632408 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a01b2bef-e3fe-4037-96fa-124a3e5c29f9-dshm\") pod \"e2e-lifecycle-model-17d0cfdd-kserve-756bf9f76c-4hpj7\" (UID: \"a01b2bef-e3fe-4037-96fa-124a3e5c29f9\") " pod="ai-tenant-e2e-mt-17d0cfdd/e2e-lifecycle-model-17d0cfdd-kserve-756bf9f76c-4hpj7" Jul 07 10:56:02.632464 ip-10-0-135-34 kubenswrapper[2578]: I0707 10:56:02.632436 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a01b2bef-e3fe-4037-96fa-124a3e5c29f9-model-cache\") pod \"e2e-lifecycle-model-17d0cfdd-kserve-756bf9f76c-4hpj7\" (UID: \"a01b2bef-e3fe-4037-96fa-124a3e5c29f9\") " pod="ai-tenant-e2e-mt-17d0cfdd/e2e-lifecycle-model-17d0cfdd-kserve-756bf9f76c-4hpj7" Jul 07 10:56:02.632595 ip-10-0-135-34 kubenswrapper[2578]: I0707 10:56:02.632513 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a01b2bef-e3fe-4037-96fa-124a3e5c29f9-tmp-dir\") pod \"e2e-lifecycle-model-17d0cfdd-kserve-756bf9f76c-4hpj7\" (UID: \"a01b2bef-e3fe-4037-96fa-124a3e5c29f9\") " pod="ai-tenant-e2e-mt-17d0cfdd/e2e-lifecycle-model-17d0cfdd-kserve-756bf9f76c-4hpj7" Jul 07 10:56:02.733243 ip-10-0-135-34 kubenswrapper[2578]: I0707 10:56:02.733199 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a01b2bef-e3fe-4037-96fa-124a3e5c29f9-tls-certs\") pod \"e2e-lifecycle-model-17d0cfdd-kserve-756bf9f76c-4hpj7\" (UID: \"a01b2bef-e3fe-4037-96fa-124a3e5c29f9\") " pod="ai-tenant-e2e-mt-17d0cfdd/e2e-lifecycle-model-17d0cfdd-kserve-756bf9f76c-4hpj7" Jul 07 10:56:02.733243 ip-10-0-135-34 kubenswrapper[2578]: I0707 10:56:02.733238 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a01b2bef-e3fe-4037-96fa-124a3e5c29f9-dshm\") pod \"e2e-lifecycle-model-17d0cfdd-kserve-756bf9f76c-4hpj7\" (UID: \"a01b2bef-e3fe-4037-96fa-124a3e5c29f9\") " pod="ai-tenant-e2e-mt-17d0cfdd/e2e-lifecycle-model-17d0cfdd-kserve-756bf9f76c-4hpj7" Jul 07 10:56:02.733520 ip-10-0-135-34 kubenswrapper[2578]: I0707 10:56:02.733262 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a01b2bef-e3fe-4037-96fa-124a3e5c29f9-model-cache\") pod \"e2e-lifecycle-model-17d0cfdd-kserve-756bf9f76c-4hpj7\" (UID: \"a01b2bef-e3fe-4037-96fa-124a3e5c29f9\") " pod="ai-tenant-e2e-mt-17d0cfdd/e2e-lifecycle-model-17d0cfdd-kserve-756bf9f76c-4hpj7" Jul 07 10:56:02.733520 ip-10-0-135-34 kubenswrapper[2578]: I0707 10:56:02.733324 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a01b2bef-e3fe-4037-96fa-124a3e5c29f9-tmp-dir\") pod \"e2e-lifecycle-model-17d0cfdd-kserve-756bf9f76c-4hpj7\" (UID: \"a01b2bef-e3fe-4037-96fa-124a3e5c29f9\") " pod="ai-tenant-e2e-mt-17d0cfdd/e2e-lifecycle-model-17d0cfdd-kserve-756bf9f76c-4hpj7" Jul 07 10:56:02.733520 ip-10-0-135-34 kubenswrapper[2578]: I0707 10:56:02.733350 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ncf9n\" (UniqueName: \"kubernetes.io/projected/a01b2bef-e3fe-4037-96fa-124a3e5c29f9-kube-api-access-ncf9n\") pod \"e2e-lifecycle-model-17d0cfdd-kserve-756bf9f76c-4hpj7\" (UID: \"a01b2bef-e3fe-4037-96fa-124a3e5c29f9\") " pod="ai-tenant-e2e-mt-17d0cfdd/e2e-lifecycle-model-17d0cfdd-kserve-756bf9f76c-4hpj7" Jul 07 10:56:02.733520 ip-10-0-135-34 kubenswrapper[2578]: I0707 10:56:02.733416 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a01b2bef-e3fe-4037-96fa-124a3e5c29f9-home\") pod \"e2e-lifecycle-model-17d0cfdd-kserve-756bf9f76c-4hpj7\" (UID: \"a01b2bef-e3fe-4037-96fa-124a3e5c29f9\") " pod="ai-tenant-e2e-mt-17d0cfdd/e2e-lifecycle-model-17d0cfdd-kserve-756bf9f76c-4hpj7" Jul 07 10:56:02.733822 ip-10-0-135-34 kubenswrapper[2578]: I0707 10:56:02.733752 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a01b2bef-e3fe-4037-96fa-124a3e5c29f9-model-cache\") pod \"e2e-lifecycle-model-17d0cfdd-kserve-756bf9f76c-4hpj7\" (UID: \"a01b2bef-e3fe-4037-96fa-124a3e5c29f9\") " pod="ai-tenant-e2e-mt-17d0cfdd/e2e-lifecycle-model-17d0cfdd-kserve-756bf9f76c-4hpj7" Jul 07 10:56:02.733822 ip-10-0-135-34 kubenswrapper[2578]: I0707 10:56:02.733803 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a01b2bef-e3fe-4037-96fa-124a3e5c29f9-home\") pod \"e2e-lifecycle-model-17d0cfdd-kserve-756bf9f76c-4hpj7\" (UID: \"a01b2bef-e3fe-4037-96fa-124a3e5c29f9\") " pod="ai-tenant-e2e-mt-17d0cfdd/e2e-lifecycle-model-17d0cfdd-kserve-756bf9f76c-4hpj7" Jul 07 10:56:02.733999 ip-10-0-135-34 kubenswrapper[2578]: I0707 10:56:02.733862 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a01b2bef-e3fe-4037-96fa-124a3e5c29f9-tmp-dir\") pod \"e2e-lifecycle-model-17d0cfdd-kserve-756bf9f76c-4hpj7\" (UID: \"a01b2bef-e3fe-4037-96fa-124a3e5c29f9\") " pod="ai-tenant-e2e-mt-17d0cfdd/e2e-lifecycle-model-17d0cfdd-kserve-756bf9f76c-4hpj7" Jul 07 10:56:02.735745 ip-10-0-135-34 kubenswrapper[2578]: I0707 10:56:02.735720 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a01b2bef-e3fe-4037-96fa-124a3e5c29f9-dshm\") pod \"e2e-lifecycle-model-17d0cfdd-kserve-756bf9f76c-4hpj7\" (UID: \"a01b2bef-e3fe-4037-96fa-124a3e5c29f9\") " pod="ai-tenant-e2e-mt-17d0cfdd/e2e-lifecycle-model-17d0cfdd-kserve-756bf9f76c-4hpj7" Jul 07 10:56:02.735924 ip-10-0-135-34 kubenswrapper[2578]: I0707 10:56:02.735905 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a01b2bef-e3fe-4037-96fa-124a3e5c29f9-tls-certs\") pod \"e2e-lifecycle-model-17d0cfdd-kserve-756bf9f76c-4hpj7\" (UID: \"a01b2bef-e3fe-4037-96fa-124a3e5c29f9\") " pod="ai-tenant-e2e-mt-17d0cfdd/e2e-lifecycle-model-17d0cfdd-kserve-756bf9f76c-4hpj7" Jul 07 10:56:02.743001 ip-10-0-135-34 kubenswrapper[2578]: I0707 10:56:02.742976 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ncf9n\" (UniqueName: \"kubernetes.io/projected/a01b2bef-e3fe-4037-96fa-124a3e5c29f9-kube-api-access-ncf9n\") pod \"e2e-lifecycle-model-17d0cfdd-kserve-756bf9f76c-4hpj7\" (UID: \"a01b2bef-e3fe-4037-96fa-124a3e5c29f9\") " pod="ai-tenant-e2e-mt-17d0cfdd/e2e-lifecycle-model-17d0cfdd-kserve-756bf9f76c-4hpj7" Jul 07 10:56:02.921175 ip-10-0-135-34 kubenswrapper[2578]: I0707 10:56:02.921079 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-mt-17d0cfdd/e2e-lifecycle-model-17d0cfdd-kserve-756bf9f76c-4hpj7" Jul 07 10:56:03.061386 ip-10-0-135-34 kubenswrapper[2578]: I0707 10:56:03.061359 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-mt-17d0cfdd/e2e-lifecycle-model-17d0cfdd-kserve-756bf9f76c-4hpj7"] Jul 07 10:56:03.063543 ip-10-0-135-34 kubenswrapper[2578]: W0707 10:56:03.063511 2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-poda01b2bef_e3fe_4037_96fa_124a3e5c29f9.slice/crio-3dc78042fa490f7735cb73c2990e471e0296eaa9463c6e7cad22a8ae9eb76e97 WatchSource:0}: Error finding container 3dc78042fa490f7735cb73c2990e471e0296eaa9463c6e7cad22a8ae9eb76e97: Status 404 returned error can't find the container with id 3dc78042fa490f7735cb73c2990e471e0296eaa9463c6e7cad22a8ae9eb76e97 Jul 07 10:56:03.120672 ip-10-0-135-34 kubenswrapper[2578]: I0707 10:56:03.120646 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-mt-17d0cfdd/e2e-lifecycle-model-17d0cfdd-kserve-756bf9f76c-4hpj7" event={"ID":"a01b2bef-e3fe-4037-96fa-124a3e5c29f9","Type":"ContainerStarted","Data":"3dc78042fa490f7735cb73c2990e471e0296eaa9463c6e7cad22a8ae9eb76e97"} Jul 07 10:56:04.127400 ip-10-0-135-34 kubenswrapper[2578]: I0707 10:56:04.127362 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-mt-17d0cfdd/e2e-lifecycle-model-17d0cfdd-kserve-756bf9f76c-4hpj7" event={"ID":"a01b2bef-e3fe-4037-96fa-124a3e5c29f9","Type":"ContainerStarted","Data":"d9d4fa15ff981e9f8cbfd7dff9513b8c94b747d636cf006baebdb6aa23463f73"} Jul 07 10:56:04.147209 ip-10-0-135-34 kubenswrapper[2578]: I0707 10:56:04.147157 2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-mt-17d0cfdd/e2e-lifecycle-model-17d0cfdd-kserve-756bf9f76c-4hpj7" podStartSLOduration=2.147142862 podStartE2EDuration="2.147142862s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-07 10:56:02 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 10:56:04.145822573 +0000 UTC m=+1732.817782116" watchObservedRunningTime="2026-07-07 10:56:04.147142862 +0000 UTC m=+1732.819102414" Jul 07 10:56:05.030229 ip-10-0-135-34 kubenswrapper[2578]: I0707 10:56:05.030184 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-lv7nd"] Jul 07 10:56:12.921350 ip-10-0-135-34 kubenswrapper[2578]: I0707 10:56:12.921302 2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-mt-17d0cfdd/e2e-lifecycle-model-17d0cfdd-kserve-756bf9f76c-4hpj7" Jul 07 10:56:12.921950 ip-10-0-135-34 kubenswrapper[2578]: I0707 10:56:12.921363 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-mt-17d0cfdd/e2e-lifecycle-model-17d0cfdd-kserve-756bf9f76c-4hpj7" Jul 07 10:56:12.933960 ip-10-0-135-34 kubenswrapper[2578]: I0707 10:56:12.933934 2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-mt-17d0cfdd/e2e-lifecycle-model-17d0cfdd-kserve-756bf9f76c-4hpj7" Jul 07 10:56:13.181963 ip-10-0-135-34 kubenswrapper[2578]: I0707 10:56:13.181876 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-mt-17d0cfdd/e2e-lifecycle-model-17d0cfdd-kserve-756bf9f76c-4hpj7" Jul 07 10:56:31.019599 ip-10-0-135-34 kubenswrapper[2578]: I0707 10:56:31.019563 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-lv7nd"] Jul 07 10:56:44.894616 ip-10-0-135-34 kubenswrapper[2578]: I0707 10:56:44.894575 2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-mt-17d0cfdd/e2e-lifecycle-model-17d0cfdd-kserve-756bf9f76c-4hpj7"] Jul 07 10:56:45.333265 ip-10-0-135-34 kubenswrapper[2578]: I0707 10:56:45.333174 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-lv7nd"] Jul 07 10:56:59.915836 ip-10-0-135-34 kubenswrapper[2578]: I0707 10:56:59.915761 2578 kuberuntime_container.go:872] "Killing container with a grace period" pod="ai-tenant-e2e-mt-17d0cfdd/e2e-lifecycle-model-17d0cfdd-kserve-756bf9f76c-4hpj7" podUID="a01b2bef-e3fe-4037-96fa-124a3e5c29f9" containerName="main" containerID="cri-o://d9d4fa15ff981e9f8cbfd7dff9513b8c94b747d636cf006baebdb6aa23463f73" gracePeriod=45 Jul 07 10:57:00.163335 ip-10-0-135-34 kubenswrapper[2578]: I0707 10:57:00.163305 2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-mt-17d0cfdd/e2e-lifecycle-model-17d0cfdd-kserve-756bf9f76c-4hpj7" Jul 07 10:57:00.319330 ip-10-0-135-34 kubenswrapper[2578]: I0707 10:57:00.319247 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a01b2bef-e3fe-4037-96fa-124a3e5c29f9-model-cache\") pod \"a01b2bef-e3fe-4037-96fa-124a3e5c29f9\" (UID: \"a01b2bef-e3fe-4037-96fa-124a3e5c29f9\") " Jul 07 10:57:00.319330 ip-10-0-135-34 kubenswrapper[2578]: I0707 10:57:00.319303 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a01b2bef-e3fe-4037-96fa-124a3e5c29f9-tmp-dir\") pod \"a01b2bef-e3fe-4037-96fa-124a3e5c29f9\" (UID: \"a01b2bef-e3fe-4037-96fa-124a3e5c29f9\") " Jul 07 10:57:00.319545 ip-10-0-135-34 kubenswrapper[2578]: I0707 10:57:00.319340 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a01b2bef-e3fe-4037-96fa-124a3e5c29f9-home\") pod \"a01b2bef-e3fe-4037-96fa-124a3e5c29f9\" (UID: \"a01b2bef-e3fe-4037-96fa-124a3e5c29f9\") " Jul 07 10:57:00.319545 ip-10-0-135-34 kubenswrapper[2578]: I0707 10:57:00.319384 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-ncf9n\" (UniqueName: \"kubernetes.io/projected/a01b2bef-e3fe-4037-96fa-124a3e5c29f9-kube-api-access-ncf9n\") pod \"a01b2bef-e3fe-4037-96fa-124a3e5c29f9\" (UID: \"a01b2bef-e3fe-4037-96fa-124a3e5c29f9\") " Jul 07 10:57:00.319545 ip-10-0-135-34 kubenswrapper[2578]: I0707 10:57:00.319425 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a01b2bef-e3fe-4037-96fa-124a3e5c29f9-dshm\") pod \"a01b2bef-e3fe-4037-96fa-124a3e5c29f9\" (UID: \"a01b2bef-e3fe-4037-96fa-124a3e5c29f9\") " Jul 07 10:57:00.319545 ip-10-0-135-34 kubenswrapper[2578]: I0707 10:57:00.319455 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a01b2bef-e3fe-4037-96fa-124a3e5c29f9-tls-certs\") pod \"a01b2bef-e3fe-4037-96fa-124a3e5c29f9\" (UID: \"a01b2bef-e3fe-4037-96fa-124a3e5c29f9\") " Jul 07 10:57:00.319810 ip-10-0-135-34 kubenswrapper[2578]: I0707 10:57:00.319534 2578 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a01b2bef-e3fe-4037-96fa-124a3e5c29f9-model-cache" (OuterVolumeSpecName: "model-cache") pod "a01b2bef-e3fe-4037-96fa-124a3e5c29f9" (UID: "a01b2bef-e3fe-4037-96fa-124a3e5c29f9"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 07 10:57:00.319810 ip-10-0-135-34 kubenswrapper[2578]: I0707 10:57:00.319608 2578 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a01b2bef-e3fe-4037-96fa-124a3e5c29f9-home" (OuterVolumeSpecName: "home") pod "a01b2bef-e3fe-4037-96fa-124a3e5c29f9" (UID: "a01b2bef-e3fe-4037-96fa-124a3e5c29f9"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 07 10:57:00.319810 ip-10-0-135-34 kubenswrapper[2578]: I0707 10:57:00.319626 2578 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a01b2bef-e3fe-4037-96fa-124a3e5c29f9-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "a01b2bef-e3fe-4037-96fa-124a3e5c29f9" (UID: "a01b2bef-e3fe-4037-96fa-124a3e5c29f9"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 07 10:57:00.319810 ip-10-0-135-34 kubenswrapper[2578]: I0707 10:57:00.319801 2578 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a01b2bef-e3fe-4037-96fa-124a3e5c29f9-home\") on node \"ip-10-0-135-34.ec2.internal\" DevicePath \"\"" Jul 07 10:57:00.320000 ip-10-0-135-34 kubenswrapper[2578]: I0707 10:57:00.319822 2578 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a01b2bef-e3fe-4037-96fa-124a3e5c29f9-model-cache\") on node \"ip-10-0-135-34.ec2.internal\" DevicePath \"\"" Jul 07 10:57:00.320000 ip-10-0-135-34 kubenswrapper[2578]: I0707 10:57:00.319836 2578 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a01b2bef-e3fe-4037-96fa-124a3e5c29f9-tmp-dir\") on node \"ip-10-0-135-34.ec2.internal\" DevicePath \"\"" Jul 07 10:57:00.321907 ip-10-0-135-34 kubenswrapper[2578]: I0707 10:57:00.321884 2578 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a01b2bef-e3fe-4037-96fa-124a3e5c29f9-dshm" (OuterVolumeSpecName: "dshm") pod "a01b2bef-e3fe-4037-96fa-124a3e5c29f9" (UID: "a01b2bef-e3fe-4037-96fa-124a3e5c29f9"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 07 10:57:00.321995 ip-10-0-135-34 kubenswrapper[2578]: I0707 10:57:00.321918 2578 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a01b2bef-e3fe-4037-96fa-124a3e5c29f9-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "a01b2bef-e3fe-4037-96fa-124a3e5c29f9" (UID: "a01b2bef-e3fe-4037-96fa-124a3e5c29f9"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 07 10:57:00.321995 ip-10-0-135-34 kubenswrapper[2578]: I0707 10:57:00.321924 2578 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a01b2bef-e3fe-4037-96fa-124a3e5c29f9-kube-api-access-ncf9n" (OuterVolumeSpecName: "kube-api-access-ncf9n") pod "a01b2bef-e3fe-4037-96fa-124a3e5c29f9" (UID: "a01b2bef-e3fe-4037-96fa-124a3e5c29f9"). InnerVolumeSpecName "kube-api-access-ncf9n". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 07 10:57:00.411362 ip-10-0-135-34 kubenswrapper[2578]: I0707 10:57:00.411329 2578 generic.go:356] "Generic (PLEG): container finished" podID="a01b2bef-e3fe-4037-96fa-124a3e5c29f9" containerID="d9d4fa15ff981e9f8cbfd7dff9513b8c94b747d636cf006baebdb6aa23463f73" exitCode=0 Jul 07 10:57:00.411569 ip-10-0-135-34 kubenswrapper[2578]: I0707 10:57:00.411381 2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-mt-17d0cfdd/e2e-lifecycle-model-17d0cfdd-kserve-756bf9f76c-4hpj7" Jul 07 10:57:00.411569 ip-10-0-135-34 kubenswrapper[2578]: I0707 10:57:00.411417 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-mt-17d0cfdd/e2e-lifecycle-model-17d0cfdd-kserve-756bf9f76c-4hpj7" event={"ID":"a01b2bef-e3fe-4037-96fa-124a3e5c29f9","Type":"ContainerDied","Data":"d9d4fa15ff981e9f8cbfd7dff9513b8c94b747d636cf006baebdb6aa23463f73"} Jul 07 10:57:00.411569 ip-10-0-135-34 kubenswrapper[2578]: I0707 10:57:00.411464 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-mt-17d0cfdd/e2e-lifecycle-model-17d0cfdd-kserve-756bf9f76c-4hpj7" event={"ID":"a01b2bef-e3fe-4037-96fa-124a3e5c29f9","Type":"ContainerDied","Data":"3dc78042fa490f7735cb73c2990e471e0296eaa9463c6e7cad22a8ae9eb76e97"} Jul 07 10:57:00.411569 ip-10-0-135-34 kubenswrapper[2578]: I0707 10:57:00.411481 2578 scope.go:117] "RemoveContainer" containerID="d9d4fa15ff981e9f8cbfd7dff9513b8c94b747d636cf006baebdb6aa23463f73" Jul 07 10:57:00.420726 ip-10-0-135-34 kubenswrapper[2578]: I0707 10:57:00.420706 2578 scope.go:117] "RemoveContainer" containerID="d9d4fa15ff981e9f8cbfd7dff9513b8c94b747d636cf006baebdb6aa23463f73" Jul 07 10:57:00.420882 ip-10-0-135-34 kubenswrapper[2578]: I0707 10:57:00.420864 2578 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a01b2bef-e3fe-4037-96fa-124a3e5c29f9-tls-certs\") on node \"ip-10-0-135-34.ec2.internal\" DevicePath \"\"" Jul 07 10:57:00.420937 ip-10-0-135-34 kubenswrapper[2578]: I0707 10:57:00.420886 2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ncf9n\" (UniqueName: \"kubernetes.io/projected/a01b2bef-e3fe-4037-96fa-124a3e5c29f9-kube-api-access-ncf9n\") on node \"ip-10-0-135-34.ec2.internal\" DevicePath \"\"" Jul 07 10:57:00.420937 ip-10-0-135-34 kubenswrapper[2578]: I0707 10:57:00.420896 2578 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a01b2bef-e3fe-4037-96fa-124a3e5c29f9-dshm\") on node \"ip-10-0-135-34.ec2.internal\" DevicePath \"\"" Jul 07 10:57:00.421032 ip-10-0-135-34 kubenswrapper[2578]: E0707 10:57:00.421020 2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d9d4fa15ff981e9f8cbfd7dff9513b8c94b747d636cf006baebdb6aa23463f73\": container with ID starting with d9d4fa15ff981e9f8cbfd7dff9513b8c94b747d636cf006baebdb6aa23463f73 not found: ID does not exist" containerID="d9d4fa15ff981e9f8cbfd7dff9513b8c94b747d636cf006baebdb6aa23463f73" Jul 07 10:57:00.421068 ip-10-0-135-34 kubenswrapper[2578]: I0707 10:57:00.421045 2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d9d4fa15ff981e9f8cbfd7dff9513b8c94b747d636cf006baebdb6aa23463f73"} err="failed to get container status \"d9d4fa15ff981e9f8cbfd7dff9513b8c94b747d636cf006baebdb6aa23463f73\": rpc error: code = NotFound desc = could not find container \"d9d4fa15ff981e9f8cbfd7dff9513b8c94b747d636cf006baebdb6aa23463f73\": container with ID starting with d9d4fa15ff981e9f8cbfd7dff9513b8c94b747d636cf006baebdb6aa23463f73 not found: ID does not exist" Jul 07 10:57:00.434003 ip-10-0-135-34 kubenswrapper[2578]: I0707 10:57:00.433970 2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-mt-17d0cfdd/e2e-lifecycle-model-17d0cfdd-kserve-756bf9f76c-4hpj7"] Jul 07 10:57:00.441618 ip-10-0-135-34 kubenswrapper[2578]: I0707 10:57:00.441592 2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-mt-17d0cfdd/e2e-lifecycle-model-17d0cfdd-kserve-756bf9f76c-4hpj7"] Jul 07 10:57:01.821687 ip-10-0-135-34 kubenswrapper[2578]: I0707 10:57:01.821655 2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="a01b2bef-e3fe-4037-96fa-124a3e5c29f9" path="/var/lib/kubelet/pods/a01b2bef-e3fe-4037-96fa-124a3e5c29f9/volumes" Jul 07 10:57:11.931336 ip-10-0-135-34 kubenswrapper[2578]: I0707 10:57:11.931305 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-qhhw7_aca7528d-ddbe-4282-979f-1b496011f0e1/ovn-acl-logging/0.log" Jul 07 10:57:11.936745 ip-10-0-135-34 kubenswrapper[2578]: I0707 10:57:11.936720 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-qhhw7_aca7528d-ddbe-4282-979f-1b496011f0e1/ovn-acl-logging/0.log" Jul 07 10:57:24.326982 ip-10-0-135-34 kubenswrapper[2578]: I0707 10:57:24.326948 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-lv7nd"] Jul 07 10:57:38.032374 ip-10-0-135-34 kubenswrapper[2578]: I0707 10:57:38.032334 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-lv7nd"] Jul 07 10:57:49.524152 ip-10-0-135-34 kubenswrapper[2578]: I0707 10:57:49.524111 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-lv7nd"] Jul 07 10:58:32.442890 ip-10-0-135-34 kubenswrapper[2578]: I0707 10:58:32.442852 2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/maas-api-e2e-inf-a-27bc88-6d5b4fb5c5-lnwlw"] Jul 07 10:58:32.443376 ip-10-0-135-34 kubenswrapper[2578]: I0707 10:58:32.443209 2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a01b2bef-e3fe-4037-96fa-124a3e5c29f9" containerName="main" Jul 07 10:58:32.443376 ip-10-0-135-34 kubenswrapper[2578]: I0707 10:58:32.443220 2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="a01b2bef-e3fe-4037-96fa-124a3e5c29f9" containerName="main" Jul 07 10:58:32.443376 ip-10-0-135-34 kubenswrapper[2578]: I0707 10:58:32.443279 2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="a01b2bef-e3fe-4037-96fa-124a3e5c29f9" containerName="main" Jul 07 10:58:32.446113 ip-10-0-135-34 kubenswrapper[2578]: I0707 10:58:32.446095 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-api-e2e-inf-a-27bc88-6d5b4fb5c5-lnwlw" Jul 07 10:58:32.450437 ip-10-0-135-34 kubenswrapper[2578]: I0707 10:58:32.450415 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-api-dockercfg-qlm6h\"" Jul 07 10:58:32.450781 ip-10-0-135-34 kubenswrapper[2578]: I0707 10:58:32.450750 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-api-e2e-inf-a-27bc88-serving-cert\"" Jul 07 10:58:32.459721 ip-10-0-135-34 kubenswrapper[2578]: I0707 10:58:32.459693 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-api-e2e-inf-a-27bc88-6d5b4fb5c5-lnwlw"] Jul 07 10:58:32.533480 ip-10-0-135-34 kubenswrapper[2578]: I0707 10:58:32.533440 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jcq5p\" (UniqueName: \"kubernetes.io/projected/adda5fbb-b371-4ae4-a414-a7c5f28f0fdc-kube-api-access-jcq5p\") pod \"maas-api-e2e-inf-a-27bc88-6d5b4fb5c5-lnwlw\" (UID: \"adda5fbb-b371-4ae4-a414-a7c5f28f0fdc\") " pod="opendatahub/maas-api-e2e-inf-a-27bc88-6d5b4fb5c5-lnwlw" Jul 07 10:58:32.533480 ip-10-0-135-34 kubenswrapper[2578]: I0707 10:58:32.533478 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/adda5fbb-b371-4ae4-a414-a7c5f28f0fdc-openshift-service-ca\") pod \"maas-api-e2e-inf-a-27bc88-6d5b4fb5c5-lnwlw\" (UID: \"adda5fbb-b371-4ae4-a414-a7c5f28f0fdc\") " pod="opendatahub/maas-api-e2e-inf-a-27bc88-6d5b4fb5c5-lnwlw" Jul 07 10:58:32.533739 ip-10-0-135-34 kubenswrapper[2578]: I0707 10:58:32.533586 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/adda5fbb-b371-4ae4-a414-a7c5f28f0fdc-maas-api-tls\") pod \"maas-api-e2e-inf-a-27bc88-6d5b4fb5c5-lnwlw\" (UID: \"adda5fbb-b371-4ae4-a414-a7c5f28f0fdc\") " pod="opendatahub/maas-api-e2e-inf-a-27bc88-6d5b4fb5c5-lnwlw" Jul 07 10:58:32.634398 ip-10-0-135-34 kubenswrapper[2578]: I0707 10:58:32.634356 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jcq5p\" (UniqueName: \"kubernetes.io/projected/adda5fbb-b371-4ae4-a414-a7c5f28f0fdc-kube-api-access-jcq5p\") pod \"maas-api-e2e-inf-a-27bc88-6d5b4fb5c5-lnwlw\" (UID: \"adda5fbb-b371-4ae4-a414-a7c5f28f0fdc\") " pod="opendatahub/maas-api-e2e-inf-a-27bc88-6d5b4fb5c5-lnwlw" Jul 07 10:58:32.634398 ip-10-0-135-34 kubenswrapper[2578]: I0707 10:58:32.634397 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/adda5fbb-b371-4ae4-a414-a7c5f28f0fdc-openshift-service-ca\") pod \"maas-api-e2e-inf-a-27bc88-6d5b4fb5c5-lnwlw\" (UID: \"adda5fbb-b371-4ae4-a414-a7c5f28f0fdc\") " pod="opendatahub/maas-api-e2e-inf-a-27bc88-6d5b4fb5c5-lnwlw" Jul 07 10:58:32.634651 ip-10-0-135-34 kubenswrapper[2578]: I0707 10:58:32.634444 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/adda5fbb-b371-4ae4-a414-a7c5f28f0fdc-maas-api-tls\") pod \"maas-api-e2e-inf-a-27bc88-6d5b4fb5c5-lnwlw\" (UID: \"adda5fbb-b371-4ae4-a414-a7c5f28f0fdc\") " pod="opendatahub/maas-api-e2e-inf-a-27bc88-6d5b4fb5c5-lnwlw" Jul 07 10:58:32.635212 ip-10-0-135-34 kubenswrapper[2578]: I0707 10:58:32.635184 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/adda5fbb-b371-4ae4-a414-a7c5f28f0fdc-openshift-service-ca\") pod \"maas-api-e2e-inf-a-27bc88-6d5b4fb5c5-lnwlw\" (UID: \"adda5fbb-b371-4ae4-a414-a7c5f28f0fdc\") " pod="opendatahub/maas-api-e2e-inf-a-27bc88-6d5b4fb5c5-lnwlw" Jul 07 10:58:32.637062 ip-10-0-135-34 kubenswrapper[2578]: I0707 10:58:32.637040 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/adda5fbb-b371-4ae4-a414-a7c5f28f0fdc-maas-api-tls\") pod \"maas-api-e2e-inf-a-27bc88-6d5b4fb5c5-lnwlw\" (UID: \"adda5fbb-b371-4ae4-a414-a7c5f28f0fdc\") " pod="opendatahub/maas-api-e2e-inf-a-27bc88-6d5b4fb5c5-lnwlw" Jul 07 10:58:32.645730 ip-10-0-135-34 kubenswrapper[2578]: I0707 10:58:32.645707 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jcq5p\" (UniqueName: \"kubernetes.io/projected/adda5fbb-b371-4ae4-a414-a7c5f28f0fdc-kube-api-access-jcq5p\") pod \"maas-api-e2e-inf-a-27bc88-6d5b4fb5c5-lnwlw\" (UID: \"adda5fbb-b371-4ae4-a414-a7c5f28f0fdc\") " pod="opendatahub/maas-api-e2e-inf-a-27bc88-6d5b4fb5c5-lnwlw" Jul 07 10:58:32.759563 ip-10-0-135-34 kubenswrapper[2578]: I0707 10:58:32.759415 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-api-e2e-inf-a-27bc88-6d5b4fb5c5-lnwlw" Jul 07 10:58:32.898339 ip-10-0-135-34 kubenswrapper[2578]: I0707 10:58:32.898314 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-api-e2e-inf-a-27bc88-6d5b4fb5c5-lnwlw"] Jul 07 10:58:32.901036 ip-10-0-135-34 kubenswrapper[2578]: W0707 10:58:32.900992 2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podadda5fbb_b371_4ae4_a414_a7c5f28f0fdc.slice/crio-ac545135a728b40e59372b17315f76481f9a0adc18e04f5d7c163f31d3e7fefe WatchSource:0}: Error finding container ac545135a728b40e59372b17315f76481f9a0adc18e04f5d7c163f31d3e7fefe: Status 404 returned error can't find the container with id ac545135a728b40e59372b17315f76481f9a0adc18e04f5d7c163f31d3e7fefe Jul 07 10:58:32.902289 ip-10-0-135-34 kubenswrapper[2578]: I0707 10:58:32.902272 2578 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Jul 07 10:58:33.882444 ip-10-0-135-34 kubenswrapper[2578]: I0707 10:58:33.882413 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-api-e2e-inf-a-27bc88-6d5b4fb5c5-lnwlw" event={"ID":"adda5fbb-b371-4ae4-a414-a7c5f28f0fdc","Type":"ContainerStarted","Data":"5626c69164c1ffd2e27b24c049bf8d1afe50b67b647ff820f077a6c44cb0c2eb"} Jul 07 10:58:33.882964 ip-10-0-135-34 kubenswrapper[2578]: I0707 10:58:33.882463 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-api-e2e-inf-a-27bc88-6d5b4fb5c5-lnwlw" event={"ID":"adda5fbb-b371-4ae4-a414-a7c5f28f0fdc","Type":"ContainerStarted","Data":"ac545135a728b40e59372b17315f76481f9a0adc18e04f5d7c163f31d3e7fefe"} Jul 07 10:58:33.882964 ip-10-0-135-34 kubenswrapper[2578]: I0707 10:58:33.882508 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-api-e2e-inf-a-27bc88-6d5b4fb5c5-lnwlw" Jul 07 10:58:33.900346 ip-10-0-135-34 kubenswrapper[2578]: I0707 10:58:33.900296 2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/maas-api-e2e-inf-a-27bc88-6d5b4fb5c5-lnwlw" podStartSLOduration=1.455052183 podStartE2EDuration="1.900276866s" totalImagesPullingTime="445.224683ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-07 10:58:32 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 10:58:33.89946876 +0000 UTC m=+1882.571428303" watchObservedRunningTime="2026-07-07 10:58:33.900276866 +0000 UTC m=+1882.572236388" Jul 07 10:58:39.825681 ip-10-0-135-34 kubenswrapper[2578]: I0707 10:58:39.825648 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-lv7nd"] Jul 07 10:58:39.893423 ip-10-0-135-34 kubenswrapper[2578]: I0707 10:58:39.893392 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/maas-api-e2e-inf-a-27bc88-6d5b4fb5c5-lnwlw" Jul 07 10:58:49.825225 ip-10-0-135-34 kubenswrapper[2578]: I0707 10:58:49.825190 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-lv7nd"] Jul 07 10:59:05.564858 ip-10-0-135-34 kubenswrapper[2578]: I0707 10:59:05.564820 2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-inf-a-27bc88/test-model-27bc88-kserve-796f4c7669-hwmvb"] Jul 07 10:59:05.568151 ip-10-0-135-34 kubenswrapper[2578]: I0707 10:59:05.568133 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-27bc88/test-model-27bc88-kserve-796f4c7669-hwmvb" Jul 07 10:59:05.572797 ip-10-0-135-34 kubenswrapper[2578]: I0707 10:59:05.572755 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-a-27bc88\"/\"default-dockercfg-v29wx\"" Jul 07 10:59:05.572797 ip-10-0-135-34 kubenswrapper[2578]: I0707 10:59:05.572755 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-a-27bc88\"/\"test-model-27bc88-kserve-self-signed-certs\"" Jul 07 10:59:05.572977 ip-10-0-135-34 kubenswrapper[2578]: I0707 10:59:05.572808 2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-a-27bc88\"/\"openshift-service-ca.crt\"" Jul 07 10:59:05.572977 ip-10-0-135-34 kubenswrapper[2578]: I0707 10:59:05.572811 2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-a-27bc88\"/\"kube-root-ca.crt\"" Jul 07 10:59:05.577887 ip-10-0-135-34 kubenswrapper[2578]: I0707 10:59:05.577858 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-a-27bc88/test-model-27bc88-kserve-796f4c7669-hwmvb"] Jul 07 10:59:05.731176 ip-10-0-135-34 kubenswrapper[2578]: I0707 10:59:05.731114 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/abf6f049-7a87-47e0-8ff6-bd933f589629-model-cache\") pod \"test-model-27bc88-kserve-796f4c7669-hwmvb\" (UID: \"abf6f049-7a87-47e0-8ff6-bd933f589629\") " pod="ai-tenant-e2e-inf-a-27bc88/test-model-27bc88-kserve-796f4c7669-hwmvb" Jul 07 10:59:05.731367 ip-10-0-135-34 kubenswrapper[2578]: I0707 10:59:05.731238 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-n2t7r\" (UniqueName: \"kubernetes.io/projected/abf6f049-7a87-47e0-8ff6-bd933f589629-kube-api-access-n2t7r\") pod \"test-model-27bc88-kserve-796f4c7669-hwmvb\" (UID: \"abf6f049-7a87-47e0-8ff6-bd933f589629\") " pod="ai-tenant-e2e-inf-a-27bc88/test-model-27bc88-kserve-796f4c7669-hwmvb" Jul 07 10:59:05.731367 ip-10-0-135-34 kubenswrapper[2578]: I0707 10:59:05.731292 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/abf6f049-7a87-47e0-8ff6-bd933f589629-tmp-dir\") pod \"test-model-27bc88-kserve-796f4c7669-hwmvb\" (UID: \"abf6f049-7a87-47e0-8ff6-bd933f589629\") " pod="ai-tenant-e2e-inf-a-27bc88/test-model-27bc88-kserve-796f4c7669-hwmvb" Jul 07 10:59:05.731367 ip-10-0-135-34 kubenswrapper[2578]: I0707 10:59:05.731315 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/abf6f049-7a87-47e0-8ff6-bd933f589629-tls-certs\") pod \"test-model-27bc88-kserve-796f4c7669-hwmvb\" (UID: \"abf6f049-7a87-47e0-8ff6-bd933f589629\") " pod="ai-tenant-e2e-inf-a-27bc88/test-model-27bc88-kserve-796f4c7669-hwmvb" Jul 07 10:59:05.731482 ip-10-0-135-34 kubenswrapper[2578]: I0707 10:59:05.731425 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/abf6f049-7a87-47e0-8ff6-bd933f589629-dshm\") pod \"test-model-27bc88-kserve-796f4c7669-hwmvb\" (UID: \"abf6f049-7a87-47e0-8ff6-bd933f589629\") " pod="ai-tenant-e2e-inf-a-27bc88/test-model-27bc88-kserve-796f4c7669-hwmvb" Jul 07 10:59:05.731482 ip-10-0-135-34 kubenswrapper[2578]: I0707 10:59:05.731461 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/abf6f049-7a87-47e0-8ff6-bd933f589629-home\") pod \"test-model-27bc88-kserve-796f4c7669-hwmvb\" (UID: \"abf6f049-7a87-47e0-8ff6-bd933f589629\") " pod="ai-tenant-e2e-inf-a-27bc88/test-model-27bc88-kserve-796f4c7669-hwmvb" Jul 07 10:59:05.832808 ip-10-0-135-34 kubenswrapper[2578]: I0707 10:59:05.832697 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n2t7r\" (UniqueName: \"kubernetes.io/projected/abf6f049-7a87-47e0-8ff6-bd933f589629-kube-api-access-n2t7r\") pod \"test-model-27bc88-kserve-796f4c7669-hwmvb\" (UID: \"abf6f049-7a87-47e0-8ff6-bd933f589629\") " pod="ai-tenant-e2e-inf-a-27bc88/test-model-27bc88-kserve-796f4c7669-hwmvb" Jul 07 10:59:05.832808 ip-10-0-135-34 kubenswrapper[2578]: I0707 10:59:05.832749 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/abf6f049-7a87-47e0-8ff6-bd933f589629-tmp-dir\") pod \"test-model-27bc88-kserve-796f4c7669-hwmvb\" (UID: \"abf6f049-7a87-47e0-8ff6-bd933f589629\") " pod="ai-tenant-e2e-inf-a-27bc88/test-model-27bc88-kserve-796f4c7669-hwmvb" Jul 07 10:59:05.833032 ip-10-0-135-34 kubenswrapper[2578]: I0707 10:59:05.832891 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/abf6f049-7a87-47e0-8ff6-bd933f589629-tls-certs\") pod \"test-model-27bc88-kserve-796f4c7669-hwmvb\" (UID: \"abf6f049-7a87-47e0-8ff6-bd933f589629\") " pod="ai-tenant-e2e-inf-a-27bc88/test-model-27bc88-kserve-796f4c7669-hwmvb" Jul 07 10:59:05.833032 ip-10-0-135-34 kubenswrapper[2578]: I0707 10:59:05.832980 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/abf6f049-7a87-47e0-8ff6-bd933f589629-dshm\") pod \"test-model-27bc88-kserve-796f4c7669-hwmvb\" (UID: \"abf6f049-7a87-47e0-8ff6-bd933f589629\") " pod="ai-tenant-e2e-inf-a-27bc88/test-model-27bc88-kserve-796f4c7669-hwmvb" Jul 07 10:59:05.833032 ip-10-0-135-34 kubenswrapper[2578]: I0707 10:59:05.833018 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/abf6f049-7a87-47e0-8ff6-bd933f589629-home\") pod \"test-model-27bc88-kserve-796f4c7669-hwmvb\" (UID: \"abf6f049-7a87-47e0-8ff6-bd933f589629\") " pod="ai-tenant-e2e-inf-a-27bc88/test-model-27bc88-kserve-796f4c7669-hwmvb" Jul 07 10:59:05.833199 ip-10-0-135-34 kubenswrapper[2578]: I0707 10:59:05.833054 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/abf6f049-7a87-47e0-8ff6-bd933f589629-model-cache\") pod \"test-model-27bc88-kserve-796f4c7669-hwmvb\" (UID: \"abf6f049-7a87-47e0-8ff6-bd933f589629\") " pod="ai-tenant-e2e-inf-a-27bc88/test-model-27bc88-kserve-796f4c7669-hwmvb" Jul 07 10:59:05.833199 ip-10-0-135-34 kubenswrapper[2578]: I0707 10:59:05.833107 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/abf6f049-7a87-47e0-8ff6-bd933f589629-tmp-dir\") pod \"test-model-27bc88-kserve-796f4c7669-hwmvb\" (UID: \"abf6f049-7a87-47e0-8ff6-bd933f589629\") " pod="ai-tenant-e2e-inf-a-27bc88/test-model-27bc88-kserve-796f4c7669-hwmvb" Jul 07 10:59:05.833408 ip-10-0-135-34 kubenswrapper[2578]: I0707 10:59:05.833383 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/abf6f049-7a87-47e0-8ff6-bd933f589629-home\") pod \"test-model-27bc88-kserve-796f4c7669-hwmvb\" (UID: \"abf6f049-7a87-47e0-8ff6-bd933f589629\") " pod="ai-tenant-e2e-inf-a-27bc88/test-model-27bc88-kserve-796f4c7669-hwmvb" Jul 07 10:59:05.833408 ip-10-0-135-34 kubenswrapper[2578]: I0707 10:59:05.833400 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/abf6f049-7a87-47e0-8ff6-bd933f589629-model-cache\") pod \"test-model-27bc88-kserve-796f4c7669-hwmvb\" (UID: \"abf6f049-7a87-47e0-8ff6-bd933f589629\") " pod="ai-tenant-e2e-inf-a-27bc88/test-model-27bc88-kserve-796f4c7669-hwmvb" Jul 07 10:59:05.835339 ip-10-0-135-34 kubenswrapper[2578]: I0707 10:59:05.835320 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/abf6f049-7a87-47e0-8ff6-bd933f589629-dshm\") pod \"test-model-27bc88-kserve-796f4c7669-hwmvb\" (UID: \"abf6f049-7a87-47e0-8ff6-bd933f589629\") " pod="ai-tenant-e2e-inf-a-27bc88/test-model-27bc88-kserve-796f4c7669-hwmvb" Jul 07 10:59:05.835684 ip-10-0-135-34 kubenswrapper[2578]: I0707 10:59:05.835663 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/abf6f049-7a87-47e0-8ff6-bd933f589629-tls-certs\") pod \"test-model-27bc88-kserve-796f4c7669-hwmvb\" (UID: \"abf6f049-7a87-47e0-8ff6-bd933f589629\") " pod="ai-tenant-e2e-inf-a-27bc88/test-model-27bc88-kserve-796f4c7669-hwmvb" Jul 07 10:59:05.842410 ip-10-0-135-34 kubenswrapper[2578]: I0707 10:59:05.842383 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-n2t7r\" (UniqueName: \"kubernetes.io/projected/abf6f049-7a87-47e0-8ff6-bd933f589629-kube-api-access-n2t7r\") pod \"test-model-27bc88-kserve-796f4c7669-hwmvb\" (UID: \"abf6f049-7a87-47e0-8ff6-bd933f589629\") " pod="ai-tenant-e2e-inf-a-27bc88/test-model-27bc88-kserve-796f4c7669-hwmvb" Jul 07 10:59:05.878313 ip-10-0-135-34 kubenswrapper[2578]: I0707 10:59:05.878279 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-27bc88/test-model-27bc88-kserve-796f4c7669-hwmvb" Jul 07 10:59:06.018176 ip-10-0-135-34 kubenswrapper[2578]: I0707 10:59:06.018143 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-a-27bc88/test-model-27bc88-kserve-796f4c7669-hwmvb"] Jul 07 10:59:06.019738 ip-10-0-135-34 kubenswrapper[2578]: W0707 10:59:06.019704 2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podabf6f049_7a87_47e0_8ff6_bd933f589629.slice/crio-241beb3250fbfc70cc9600e01a2665e7851b91fb4860ebd6ce2db45afc749f91 WatchSource:0}: Error finding container 241beb3250fbfc70cc9600e01a2665e7851b91fb4860ebd6ce2db45afc749f91: Status 404 returned error can't find the container with id 241beb3250fbfc70cc9600e01a2665e7851b91fb4860ebd6ce2db45afc749f91 Jul 07 10:59:06.043948 ip-10-0-135-34 kubenswrapper[2578]: I0707 10:59:06.043911 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-27bc88/test-model-27bc88-kserve-796f4c7669-hwmvb" event={"ID":"abf6f049-7a87-47e0-8ff6-bd933f589629","Type":"ContainerStarted","Data":"241beb3250fbfc70cc9600e01a2665e7851b91fb4860ebd6ce2db45afc749f91"} Jul 07 10:59:07.050172 ip-10-0-135-34 kubenswrapper[2578]: I0707 10:59:07.050133 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-27bc88/test-model-27bc88-kserve-796f4c7669-hwmvb" event={"ID":"abf6f049-7a87-47e0-8ff6-bd933f589629","Type":"ContainerStarted","Data":"c945e6481f28c3d4286763f6228a2895071a3e895bff7b64dc7fe92bcfb4c9c8"} Jul 07 10:59:07.068443 ip-10-0-135-34 kubenswrapper[2578]: I0707 10:59:07.068393 2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-inf-a-27bc88/test-model-27bc88-kserve-796f4c7669-hwmvb" podStartSLOduration=2.06837745 podStartE2EDuration="2.06837745s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-07 10:59:05 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 10:59:07.066302653 +0000 UTC m=+1915.738262199" watchObservedRunningTime="2026-07-07 10:59:07.06837745 +0000 UTC m=+1915.740336995" Jul 07 10:59:09.143915 ip-10-0-135-34 kubenswrapper[2578]: I0707 10:59:09.143882 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-lv7nd"] Jul 07 10:59:15.879272 ip-10-0-135-34 kubenswrapper[2578]: I0707 10:59:15.879232 2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-inf-a-27bc88/test-model-27bc88-kserve-796f4c7669-hwmvb" Jul 07 10:59:15.879691 ip-10-0-135-34 kubenswrapper[2578]: I0707 10:59:15.879295 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-inf-a-27bc88/test-model-27bc88-kserve-796f4c7669-hwmvb" Jul 07 10:59:15.891809 ip-10-0-135-34 kubenswrapper[2578]: I0707 10:59:15.891781 2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-inf-a-27bc88/test-model-27bc88-kserve-796f4c7669-hwmvb" Jul 07 10:59:16.106982 ip-10-0-135-34 kubenswrapper[2578]: I0707 10:59:16.106948 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-inf-a-27bc88/test-model-27bc88-kserve-796f4c7669-hwmvb" Jul 07 10:59:46.161634 ip-10-0-135-34 kubenswrapper[2578]: I0707 10:59:46.161595 2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-inf-b-845408/test-model-845408-kserve-5c7ccf7c6c-dmj8q"] Jul 07 10:59:46.165205 ip-10-0-135-34 kubenswrapper[2578]: I0707 10:59:46.165188 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-845408/test-model-845408-kserve-5c7ccf7c6c-dmj8q" Jul 07 10:59:46.169254 ip-10-0-135-34 kubenswrapper[2578]: I0707 10:59:46.169220 2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-b-845408\"/\"openshift-service-ca.crt\"" Jul 07 10:59:46.169396 ip-10-0-135-34 kubenswrapper[2578]: I0707 10:59:46.169263 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-b-845408\"/\"test-model-845408-kserve-self-signed-certs\"" Jul 07 10:59:46.169396 ip-10-0-135-34 kubenswrapper[2578]: I0707 10:59:46.169265 2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-b-845408\"/\"kube-root-ca.crt\"" Jul 07 10:59:46.169396 ip-10-0-135-34 kubenswrapper[2578]: I0707 10:59:46.169318 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-b-845408\"/\"default-dockercfg-rzjsz\"" Jul 07 10:59:46.175710 ip-10-0-135-34 kubenswrapper[2578]: I0707 10:59:46.175683 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-b-845408/test-model-845408-kserve-5c7ccf7c6c-dmj8q"] Jul 07 10:59:46.299875 ip-10-0-135-34 kubenswrapper[2578]: I0707 10:59:46.299841 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e039759a-4860-4740-942e-a9b7a18d96da-model-cache\") pod \"test-model-845408-kserve-5c7ccf7c6c-dmj8q\" (UID: \"e039759a-4860-4740-942e-a9b7a18d96da\") " pod="ai-tenant-e2e-inf-b-845408/test-model-845408-kserve-5c7ccf7c6c-dmj8q" Jul 07 10:59:46.300055 ip-10-0-135-34 kubenswrapper[2578]: I0707 10:59:46.299892 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zrfs4\" (UniqueName: \"kubernetes.io/projected/e039759a-4860-4740-942e-a9b7a18d96da-kube-api-access-zrfs4\") pod \"test-model-845408-kserve-5c7ccf7c6c-dmj8q\" (UID: \"e039759a-4860-4740-942e-a9b7a18d96da\") " pod="ai-tenant-e2e-inf-b-845408/test-model-845408-kserve-5c7ccf7c6c-dmj8q" Jul 07 10:59:46.300055 ip-10-0-135-34 kubenswrapper[2578]: I0707 10:59:46.299914 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e039759a-4860-4740-942e-a9b7a18d96da-home\") pod \"test-model-845408-kserve-5c7ccf7c6c-dmj8q\" (UID: \"e039759a-4860-4740-942e-a9b7a18d96da\") " pod="ai-tenant-e2e-inf-b-845408/test-model-845408-kserve-5c7ccf7c6c-dmj8q" Jul 07 10:59:46.300055 ip-10-0-135-34 kubenswrapper[2578]: I0707 10:59:46.300025 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e039759a-4860-4740-942e-a9b7a18d96da-dshm\") pod \"test-model-845408-kserve-5c7ccf7c6c-dmj8q\" (UID: \"e039759a-4860-4740-942e-a9b7a18d96da\") " pod="ai-tenant-e2e-inf-b-845408/test-model-845408-kserve-5c7ccf7c6c-dmj8q" Jul 07 10:59:46.300167 ip-10-0-135-34 kubenswrapper[2578]: I0707 10:59:46.300054 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e039759a-4860-4740-942e-a9b7a18d96da-tmp-dir\") pod \"test-model-845408-kserve-5c7ccf7c6c-dmj8q\" (UID: \"e039759a-4860-4740-942e-a9b7a18d96da\") " pod="ai-tenant-e2e-inf-b-845408/test-model-845408-kserve-5c7ccf7c6c-dmj8q" Jul 07 10:59:46.300167 ip-10-0-135-34 kubenswrapper[2578]: I0707 10:59:46.300074 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e039759a-4860-4740-942e-a9b7a18d96da-tls-certs\") pod \"test-model-845408-kserve-5c7ccf7c6c-dmj8q\" (UID: \"e039759a-4860-4740-942e-a9b7a18d96da\") " pod="ai-tenant-e2e-inf-b-845408/test-model-845408-kserve-5c7ccf7c6c-dmj8q" Jul 07 10:59:46.401243 ip-10-0-135-34 kubenswrapper[2578]: I0707 10:59:46.401201 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e039759a-4860-4740-942e-a9b7a18d96da-model-cache\") pod \"test-model-845408-kserve-5c7ccf7c6c-dmj8q\" (UID: \"e039759a-4860-4740-942e-a9b7a18d96da\") " pod="ai-tenant-e2e-inf-b-845408/test-model-845408-kserve-5c7ccf7c6c-dmj8q" Jul 07 10:59:46.401435 ip-10-0-135-34 kubenswrapper[2578]: I0707 10:59:46.401254 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zrfs4\" (UniqueName: \"kubernetes.io/projected/e039759a-4860-4740-942e-a9b7a18d96da-kube-api-access-zrfs4\") pod \"test-model-845408-kserve-5c7ccf7c6c-dmj8q\" (UID: \"e039759a-4860-4740-942e-a9b7a18d96da\") " pod="ai-tenant-e2e-inf-b-845408/test-model-845408-kserve-5c7ccf7c6c-dmj8q" Jul 07 10:59:46.401435 ip-10-0-135-34 kubenswrapper[2578]: I0707 10:59:46.401283 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e039759a-4860-4740-942e-a9b7a18d96da-home\") pod \"test-model-845408-kserve-5c7ccf7c6c-dmj8q\" (UID: \"e039759a-4860-4740-942e-a9b7a18d96da\") " pod="ai-tenant-e2e-inf-b-845408/test-model-845408-kserve-5c7ccf7c6c-dmj8q" Jul 07 10:59:46.401435 ip-10-0-135-34 kubenswrapper[2578]: I0707 10:59:46.401322 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e039759a-4860-4740-942e-a9b7a18d96da-dshm\") pod \"test-model-845408-kserve-5c7ccf7c6c-dmj8q\" (UID: \"e039759a-4860-4740-942e-a9b7a18d96da\") " pod="ai-tenant-e2e-inf-b-845408/test-model-845408-kserve-5c7ccf7c6c-dmj8q" Jul 07 10:59:46.401435 ip-10-0-135-34 kubenswrapper[2578]: I0707 10:59:46.401346 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e039759a-4860-4740-942e-a9b7a18d96da-tmp-dir\") pod \"test-model-845408-kserve-5c7ccf7c6c-dmj8q\" (UID: \"e039759a-4860-4740-942e-a9b7a18d96da\") " pod="ai-tenant-e2e-inf-b-845408/test-model-845408-kserve-5c7ccf7c6c-dmj8q" Jul 07 10:59:46.401435 ip-10-0-135-34 kubenswrapper[2578]: I0707 10:59:46.401371 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e039759a-4860-4740-942e-a9b7a18d96da-tls-certs\") pod \"test-model-845408-kserve-5c7ccf7c6c-dmj8q\" (UID: \"e039759a-4860-4740-942e-a9b7a18d96da\") " pod="ai-tenant-e2e-inf-b-845408/test-model-845408-kserve-5c7ccf7c6c-dmj8q" Jul 07 10:59:46.401698 ip-10-0-135-34 kubenswrapper[2578]: I0707 10:59:46.401643 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e039759a-4860-4740-942e-a9b7a18d96da-model-cache\") pod \"test-model-845408-kserve-5c7ccf7c6c-dmj8q\" (UID: \"e039759a-4860-4740-942e-a9b7a18d96da\") " pod="ai-tenant-e2e-inf-b-845408/test-model-845408-kserve-5c7ccf7c6c-dmj8q" Jul 07 10:59:46.401698 ip-10-0-135-34 kubenswrapper[2578]: I0707 10:59:46.401688 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e039759a-4860-4740-942e-a9b7a18d96da-home\") pod \"test-model-845408-kserve-5c7ccf7c6c-dmj8q\" (UID: \"e039759a-4860-4740-942e-a9b7a18d96da\") " pod="ai-tenant-e2e-inf-b-845408/test-model-845408-kserve-5c7ccf7c6c-dmj8q" Jul 07 10:59:46.401823 ip-10-0-135-34 kubenswrapper[2578]: I0707 10:59:46.401743 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e039759a-4860-4740-942e-a9b7a18d96da-tmp-dir\") pod \"test-model-845408-kserve-5c7ccf7c6c-dmj8q\" (UID: \"e039759a-4860-4740-942e-a9b7a18d96da\") " pod="ai-tenant-e2e-inf-b-845408/test-model-845408-kserve-5c7ccf7c6c-dmj8q" Jul 07 10:59:46.403762 ip-10-0-135-34 kubenswrapper[2578]: I0707 10:59:46.403737 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e039759a-4860-4740-942e-a9b7a18d96da-dshm\") pod \"test-model-845408-kserve-5c7ccf7c6c-dmj8q\" (UID: \"e039759a-4860-4740-942e-a9b7a18d96da\") " pod="ai-tenant-e2e-inf-b-845408/test-model-845408-kserve-5c7ccf7c6c-dmj8q" Jul 07 10:59:46.404090 ip-10-0-135-34 kubenswrapper[2578]: I0707 10:59:46.404073 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e039759a-4860-4740-942e-a9b7a18d96da-tls-certs\") pod \"test-model-845408-kserve-5c7ccf7c6c-dmj8q\" (UID: \"e039759a-4860-4740-942e-a9b7a18d96da\") " pod="ai-tenant-e2e-inf-b-845408/test-model-845408-kserve-5c7ccf7c6c-dmj8q" Jul 07 10:59:46.409595 ip-10-0-135-34 kubenswrapper[2578]: I0707 10:59:46.409570 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zrfs4\" (UniqueName: \"kubernetes.io/projected/e039759a-4860-4740-942e-a9b7a18d96da-kube-api-access-zrfs4\") pod \"test-model-845408-kserve-5c7ccf7c6c-dmj8q\" (UID: \"e039759a-4860-4740-942e-a9b7a18d96da\") " pod="ai-tenant-e2e-inf-b-845408/test-model-845408-kserve-5c7ccf7c6c-dmj8q" Jul 07 10:59:46.475463 ip-10-0-135-34 kubenswrapper[2578]: I0707 10:59:46.475370 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-845408/test-model-845408-kserve-5c7ccf7c6c-dmj8q" Jul 07 10:59:46.614284 ip-10-0-135-34 kubenswrapper[2578]: I0707 10:59:46.614257 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-b-845408/test-model-845408-kserve-5c7ccf7c6c-dmj8q"] Jul 07 10:59:46.615832 ip-10-0-135-34 kubenswrapper[2578]: W0707 10:59:46.615796 2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pode039759a_4860_4740_942e_a9b7a18d96da.slice/crio-f6dab02433561dc5b567796e95c00d3f224e8223a3d77af497cf03cd89de9507 WatchSource:0}: Error finding container f6dab02433561dc5b567796e95c00d3f224e8223a3d77af497cf03cd89de9507: Status 404 returned error can't find the container with id f6dab02433561dc5b567796e95c00d3f224e8223a3d77af497cf03cd89de9507 Jul 07 10:59:47.250953 ip-10-0-135-34 kubenswrapper[2578]: I0707 10:59:47.250917 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-845408/test-model-845408-kserve-5c7ccf7c6c-dmj8q" event={"ID":"e039759a-4860-4740-942e-a9b7a18d96da","Type":"ContainerStarted","Data":"c0587fdc9155b5a16c6280be684e87d7c9739788407e7fff0b52636840846d57"} Jul 07 10:59:47.251394 ip-10-0-135-34 kubenswrapper[2578]: I0707 10:59:47.250965 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-845408/test-model-845408-kserve-5c7ccf7c6c-dmj8q" event={"ID":"e039759a-4860-4740-942e-a9b7a18d96da","Type":"ContainerStarted","Data":"f6dab02433561dc5b567796e95c00d3f224e8223a3d77af497cf03cd89de9507"} Jul 07 10:59:47.269846 ip-10-0-135-34 kubenswrapper[2578]: I0707 10:59:47.269794 2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-inf-b-845408/test-model-845408-kserve-5c7ccf7c6c-dmj8q" podStartSLOduration=1.269779066 podStartE2EDuration="1.269779066s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-07 10:59:46 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 10:59:47.267318942 +0000 UTC m=+1955.939278486" watchObservedRunningTime="2026-07-07 10:59:47.269779066 +0000 UTC m=+1955.941738602" Jul 07 10:59:50.837608 ip-10-0-135-34 kubenswrapper[2578]: I0707 10:59:50.837559 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-lv7nd"] Jul 07 10:59:56.475861 ip-10-0-135-34 kubenswrapper[2578]: I0707 10:59:56.475829 2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-inf-b-845408/test-model-845408-kserve-5c7ccf7c6c-dmj8q" Jul 07 10:59:56.476234 ip-10-0-135-34 kubenswrapper[2578]: I0707 10:59:56.475871 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-inf-b-845408/test-model-845408-kserve-5c7ccf7c6c-dmj8q" Jul 07 10:59:56.488224 ip-10-0-135-34 kubenswrapper[2578]: I0707 10:59:56.488197 2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-inf-b-845408/test-model-845408-kserve-5c7ccf7c6c-dmj8q" Jul 07 10:59:57.319156 ip-10-0-135-34 kubenswrapper[2578]: I0707 10:59:57.319121 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-inf-b-845408/test-model-845408-kserve-5c7ccf7c6c-dmj8q" Jul 07 11:00:04.199743 ip-10-0-135-34 kubenswrapper[2578]: I0707 11:00:04.199649 2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-a-27bc88/test-model-27bc88-kserve-796f4c7669-hwmvb"] Jul 07 11:00:05.119567 ip-10-0-135-34 kubenswrapper[2578]: I0707 11:00:05.119536 2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-b-845408/test-model-845408-kserve-5c7ccf7c6c-dmj8q"] Jul 07 11:00:06.123037 ip-10-0-135-34 kubenswrapper[2578]: I0707 11:00:06.123002 2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-api-e2e-inf-a-27bc88-6d5b4fb5c5-lnwlw"] Jul 07 11:00:06.123449 ip-10-0-135-34 kubenswrapper[2578]: I0707 11:00:06.123188 2578 kuberuntime_container.go:872] "Killing container with a grace period" pod="opendatahub/maas-api-e2e-inf-a-27bc88-6d5b4fb5c5-lnwlw" podUID="adda5fbb-b371-4ae4-a414-a7c5f28f0fdc" containerName="maas-api" containerID="cri-o://5626c69164c1ffd2e27b24c049bf8d1afe50b67b647ff820f077a6c44cb0c2eb" gracePeriod=30 Jul 07 11:00:06.567718 ip-10-0-135-34 kubenswrapper[2578]: I0707 11:00:06.567689 2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-api-e2e-inf-a-27bc88-6d5b4fb5c5-lnwlw" Jul 07 11:00:06.677114 ip-10-0-135-34 kubenswrapper[2578]: I0707 11:00:06.677025 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/adda5fbb-b371-4ae4-a414-a7c5f28f0fdc-maas-api-tls\") pod \"adda5fbb-b371-4ae4-a414-a7c5f28f0fdc\" (UID: \"adda5fbb-b371-4ae4-a414-a7c5f28f0fdc\") " Jul 07 11:00:06.677114 ip-10-0-135-34 kubenswrapper[2578]: I0707 11:00:06.677104 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/adda5fbb-b371-4ae4-a414-a7c5f28f0fdc-openshift-service-ca\") pod \"adda5fbb-b371-4ae4-a414-a7c5f28f0fdc\" (UID: \"adda5fbb-b371-4ae4-a414-a7c5f28f0fdc\") " Jul 07 11:00:06.677310 ip-10-0-135-34 kubenswrapper[2578]: I0707 11:00:06.677177 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-jcq5p\" (UniqueName: \"kubernetes.io/projected/adda5fbb-b371-4ae4-a414-a7c5f28f0fdc-kube-api-access-jcq5p\") pod \"adda5fbb-b371-4ae4-a414-a7c5f28f0fdc\" (UID: \"adda5fbb-b371-4ae4-a414-a7c5f28f0fdc\") " Jul 07 11:00:06.677469 ip-10-0-135-34 kubenswrapper[2578]: I0707 11:00:06.677448 2578 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/adda5fbb-b371-4ae4-a414-a7c5f28f0fdc-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "adda5fbb-b371-4ae4-a414-a7c5f28f0fdc" (UID: "adda5fbb-b371-4ae4-a414-a7c5f28f0fdc"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 07 11:00:06.679402 ip-10-0-135-34 kubenswrapper[2578]: I0707 11:00:06.679370 2578 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/adda5fbb-b371-4ae4-a414-a7c5f28f0fdc-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "adda5fbb-b371-4ae4-a414-a7c5f28f0fdc" (UID: "adda5fbb-b371-4ae4-a414-a7c5f28f0fdc"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 07 11:00:06.679511 ip-10-0-135-34 kubenswrapper[2578]: I0707 11:00:06.679453 2578 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/adda5fbb-b371-4ae4-a414-a7c5f28f0fdc-kube-api-access-jcq5p" (OuterVolumeSpecName: "kube-api-access-jcq5p") pod "adda5fbb-b371-4ae4-a414-a7c5f28f0fdc" (UID: "adda5fbb-b371-4ae4-a414-a7c5f28f0fdc"). InnerVolumeSpecName "kube-api-access-jcq5p". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 07 11:00:06.777969 ip-10-0-135-34 kubenswrapper[2578]: I0707 11:00:06.777931 2578 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/adda5fbb-b371-4ae4-a414-a7c5f28f0fdc-maas-api-tls\") on node \"ip-10-0-135-34.ec2.internal\" DevicePath \"\"" Jul 07 11:00:06.777969 ip-10-0-135-34 kubenswrapper[2578]: I0707 11:00:06.777962 2578 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/adda5fbb-b371-4ae4-a414-a7c5f28f0fdc-openshift-service-ca\") on node \"ip-10-0-135-34.ec2.internal\" DevicePath \"\"" Jul 07 11:00:06.777969 ip-10-0-135-34 kubenswrapper[2578]: I0707 11:00:06.777974 2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jcq5p\" (UniqueName: \"kubernetes.io/projected/adda5fbb-b371-4ae4-a414-a7c5f28f0fdc-kube-api-access-jcq5p\") on node \"ip-10-0-135-34.ec2.internal\" DevicePath \"\"" Jul 07 11:00:07.359233 ip-10-0-135-34 kubenswrapper[2578]: I0707 11:00:07.359198 2578 generic.go:356] "Generic (PLEG): container finished" podID="adda5fbb-b371-4ae4-a414-a7c5f28f0fdc" containerID="5626c69164c1ffd2e27b24c049bf8d1afe50b67b647ff820f077a6c44cb0c2eb" exitCode=0 Jul 07 11:00:07.359729 ip-10-0-135-34 kubenswrapper[2578]: I0707 11:00:07.359259 2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-api-e2e-inf-a-27bc88-6d5b4fb5c5-lnwlw" Jul 07 11:00:07.359729 ip-10-0-135-34 kubenswrapper[2578]: I0707 11:00:07.359289 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-api-e2e-inf-a-27bc88-6d5b4fb5c5-lnwlw" event={"ID":"adda5fbb-b371-4ae4-a414-a7c5f28f0fdc","Type":"ContainerDied","Data":"5626c69164c1ffd2e27b24c049bf8d1afe50b67b647ff820f077a6c44cb0c2eb"} Jul 07 11:00:07.359729 ip-10-0-135-34 kubenswrapper[2578]: I0707 11:00:07.359383 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-api-e2e-inf-a-27bc88-6d5b4fb5c5-lnwlw" event={"ID":"adda5fbb-b371-4ae4-a414-a7c5f28f0fdc","Type":"ContainerDied","Data":"ac545135a728b40e59372b17315f76481f9a0adc18e04f5d7c163f31d3e7fefe"} Jul 07 11:00:07.359729 ip-10-0-135-34 kubenswrapper[2578]: I0707 11:00:07.359416 2578 scope.go:117] "RemoveContainer" containerID="5626c69164c1ffd2e27b24c049bf8d1afe50b67b647ff820f077a6c44cb0c2eb" Jul 07 11:00:07.370611 ip-10-0-135-34 kubenswrapper[2578]: I0707 11:00:07.370593 2578 scope.go:117] "RemoveContainer" containerID="5626c69164c1ffd2e27b24c049bf8d1afe50b67b647ff820f077a6c44cb0c2eb" Jul 07 11:00:07.370956 ip-10-0-135-34 kubenswrapper[2578]: E0707 11:00:07.370928 2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5626c69164c1ffd2e27b24c049bf8d1afe50b67b647ff820f077a6c44cb0c2eb\": container with ID starting with 5626c69164c1ffd2e27b24c049bf8d1afe50b67b647ff820f077a6c44cb0c2eb not found: ID does not exist" containerID="5626c69164c1ffd2e27b24c049bf8d1afe50b67b647ff820f077a6c44cb0c2eb" Jul 07 11:00:07.371047 ip-10-0-135-34 kubenswrapper[2578]: I0707 11:00:07.370960 2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5626c69164c1ffd2e27b24c049bf8d1afe50b67b647ff820f077a6c44cb0c2eb"} err="failed to get container status \"5626c69164c1ffd2e27b24c049bf8d1afe50b67b647ff820f077a6c44cb0c2eb\": rpc error: code = NotFound desc = could not find container \"5626c69164c1ffd2e27b24c049bf8d1afe50b67b647ff820f077a6c44cb0c2eb\": container with ID starting with 5626c69164c1ffd2e27b24c049bf8d1afe50b67b647ff820f077a6c44cb0c2eb not found: ID does not exist" Jul 07 11:00:07.386138 ip-10-0-135-34 kubenswrapper[2578]: I0707 11:00:07.386103 2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-api-e2e-inf-a-27bc88-6d5b4fb5c5-lnwlw"] Jul 07 11:00:07.393066 ip-10-0-135-34 kubenswrapper[2578]: I0707 11:00:07.393038 2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/maas-api-e2e-inf-a-27bc88-6d5b4fb5c5-lnwlw"] Jul 07 11:00:07.821948 ip-10-0-135-34 kubenswrapper[2578]: I0707 11:00:07.821873 2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="adda5fbb-b371-4ae4-a414-a7c5f28f0fdc" path="/var/lib/kubelet/pods/adda5fbb-b371-4ae4-a414-a7c5f28f0fdc/volumes" Jul 07 11:00:08.923707 ip-10-0-135-34 kubenswrapper[2578]: I0707 11:00:08.923669 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-lv7nd"] Jul 07 11:00:19.221152 ip-10-0-135-34 kubenswrapper[2578]: I0707 11:00:19.221082 2578 kuberuntime_container.go:872] "Killing container with a grace period" pod="ai-tenant-e2e-inf-a-27bc88/test-model-27bc88-kserve-796f4c7669-hwmvb" podUID="abf6f049-7a87-47e0-8ff6-bd933f589629" containerName="main" containerID="cri-o://c945e6481f28c3d4286763f6228a2895071a3e895bff7b64dc7fe92bcfb4c9c8" gracePeriod=45 Jul 07 11:00:19.425087 ip-10-0-135-34 kubenswrapper[2578]: I0707 11:00:19.425038 2578 generic.go:356] "Generic (PLEG): container finished" podID="abf6f049-7a87-47e0-8ff6-bd933f589629" containerID="c945e6481f28c3d4286763f6228a2895071a3e895bff7b64dc7fe92bcfb4c9c8" exitCode=0 Jul 07 11:00:19.425241 ip-10-0-135-34 kubenswrapper[2578]: I0707 11:00:19.425110 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-27bc88/test-model-27bc88-kserve-796f4c7669-hwmvb" event={"ID":"abf6f049-7a87-47e0-8ff6-bd933f589629","Type":"ContainerDied","Data":"c945e6481f28c3d4286763f6228a2895071a3e895bff7b64dc7fe92bcfb4c9c8"} Jul 07 11:00:19.470827 ip-10-0-135-34 kubenswrapper[2578]: I0707 11:00:19.470798 2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-27bc88/test-model-27bc88-kserve-796f4c7669-hwmvb" Jul 07 11:00:19.488833 ip-10-0-135-34 kubenswrapper[2578]: I0707 11:00:19.488733 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/abf6f049-7a87-47e0-8ff6-bd933f589629-tls-certs\") pod \"abf6f049-7a87-47e0-8ff6-bd933f589629\" (UID: \"abf6f049-7a87-47e0-8ff6-bd933f589629\") " Jul 07 11:00:19.488833 ip-10-0-135-34 kubenswrapper[2578]: I0707 11:00:19.488802 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/abf6f049-7a87-47e0-8ff6-bd933f589629-home\") pod \"abf6f049-7a87-47e0-8ff6-bd933f589629\" (UID: \"abf6f049-7a87-47e0-8ff6-bd933f589629\") " Jul 07 11:00:19.489061 ip-10-0-135-34 kubenswrapper[2578]: I0707 11:00:19.488840 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/abf6f049-7a87-47e0-8ff6-bd933f589629-dshm\") pod \"abf6f049-7a87-47e0-8ff6-bd933f589629\" (UID: \"abf6f049-7a87-47e0-8ff6-bd933f589629\") " Jul 07 11:00:19.489061 ip-10-0-135-34 kubenswrapper[2578]: I0707 11:00:19.488887 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/abf6f049-7a87-47e0-8ff6-bd933f589629-tmp-dir\") pod \"abf6f049-7a87-47e0-8ff6-bd933f589629\" (UID: \"abf6f049-7a87-47e0-8ff6-bd933f589629\") " Jul 07 11:00:19.489061 ip-10-0-135-34 kubenswrapper[2578]: I0707 11:00:19.488935 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-n2t7r\" (UniqueName: \"kubernetes.io/projected/abf6f049-7a87-47e0-8ff6-bd933f589629-kube-api-access-n2t7r\") pod \"abf6f049-7a87-47e0-8ff6-bd933f589629\" (UID: \"abf6f049-7a87-47e0-8ff6-bd933f589629\") " Jul 07 11:00:19.489061 ip-10-0-135-34 kubenswrapper[2578]: I0707 11:00:19.488982 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/abf6f049-7a87-47e0-8ff6-bd933f589629-model-cache\") pod \"abf6f049-7a87-47e0-8ff6-bd933f589629\" (UID: \"abf6f049-7a87-47e0-8ff6-bd933f589629\") " Jul 07 11:00:19.489279 ip-10-0-135-34 kubenswrapper[2578]: I0707 11:00:19.489113 2578 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/abf6f049-7a87-47e0-8ff6-bd933f589629-home" (OuterVolumeSpecName: "home") pod "abf6f049-7a87-47e0-8ff6-bd933f589629" (UID: "abf6f049-7a87-47e0-8ff6-bd933f589629"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 07 11:00:19.489279 ip-10-0-135-34 kubenswrapper[2578]: I0707 11:00:19.489214 2578 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/abf6f049-7a87-47e0-8ff6-bd933f589629-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "abf6f049-7a87-47e0-8ff6-bd933f589629" (UID: "abf6f049-7a87-47e0-8ff6-bd933f589629"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 07 11:00:19.489393 ip-10-0-135-34 kubenswrapper[2578]: I0707 11:00:19.489314 2578 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/abf6f049-7a87-47e0-8ff6-bd933f589629-model-cache" (OuterVolumeSpecName: "model-cache") pod "abf6f049-7a87-47e0-8ff6-bd933f589629" (UID: "abf6f049-7a87-47e0-8ff6-bd933f589629"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 07 11:00:19.489393 ip-10-0-135-34 kubenswrapper[2578]: I0707 11:00:19.489344 2578 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/abf6f049-7a87-47e0-8ff6-bd933f589629-tmp-dir\") on node \"ip-10-0-135-34.ec2.internal\" DevicePath \"\"" Jul 07 11:00:19.489393 ip-10-0-135-34 kubenswrapper[2578]: I0707 11:00:19.489366 2578 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/abf6f049-7a87-47e0-8ff6-bd933f589629-home\") on node \"ip-10-0-135-34.ec2.internal\" DevicePath \"\"" Jul 07 11:00:19.491683 ip-10-0-135-34 kubenswrapper[2578]: I0707 11:00:19.491614 2578 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/abf6f049-7a87-47e0-8ff6-bd933f589629-dshm" (OuterVolumeSpecName: "dshm") pod "abf6f049-7a87-47e0-8ff6-bd933f589629" (UID: "abf6f049-7a87-47e0-8ff6-bd933f589629"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 07 11:00:19.491864 ip-10-0-135-34 kubenswrapper[2578]: I0707 11:00:19.491842 2578 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/abf6f049-7a87-47e0-8ff6-bd933f589629-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "abf6f049-7a87-47e0-8ff6-bd933f589629" (UID: "abf6f049-7a87-47e0-8ff6-bd933f589629"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 07 11:00:19.491947 ip-10-0-135-34 kubenswrapper[2578]: I0707 11:00:19.491924 2578 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/abf6f049-7a87-47e0-8ff6-bd933f589629-kube-api-access-n2t7r" (OuterVolumeSpecName: "kube-api-access-n2t7r") pod "abf6f049-7a87-47e0-8ff6-bd933f589629" (UID: "abf6f049-7a87-47e0-8ff6-bd933f589629"). InnerVolumeSpecName "kube-api-access-n2t7r". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 07 11:00:19.590455 ip-10-0-135-34 kubenswrapper[2578]: I0707 11:00:19.590410 2578 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/abf6f049-7a87-47e0-8ff6-bd933f589629-model-cache\") on node \"ip-10-0-135-34.ec2.internal\" DevicePath \"\"" Jul 07 11:00:19.590455 ip-10-0-135-34 kubenswrapper[2578]: I0707 11:00:19.590446 2578 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/abf6f049-7a87-47e0-8ff6-bd933f589629-tls-certs\") on node \"ip-10-0-135-34.ec2.internal\" DevicePath \"\"" Jul 07 11:00:19.590455 ip-10-0-135-34 kubenswrapper[2578]: I0707 11:00:19.590454 2578 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/abf6f049-7a87-47e0-8ff6-bd933f589629-dshm\") on node \"ip-10-0-135-34.ec2.internal\" DevicePath \"\"" Jul 07 11:00:19.590455 ip-10-0-135-34 kubenswrapper[2578]: I0707 11:00:19.590462 2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-n2t7r\" (UniqueName: \"kubernetes.io/projected/abf6f049-7a87-47e0-8ff6-bd933f589629-kube-api-access-n2t7r\") on node \"ip-10-0-135-34.ec2.internal\" DevicePath \"\"" Jul 07 11:00:20.140945 ip-10-0-135-34 kubenswrapper[2578]: I0707 11:00:20.140885 2578 kuberuntime_container.go:872] "Killing container with a grace period" pod="ai-tenant-e2e-inf-b-845408/test-model-845408-kserve-5c7ccf7c6c-dmj8q" podUID="e039759a-4860-4740-942e-a9b7a18d96da" containerName="main" containerID="cri-o://c0587fdc9155b5a16c6280be684e87d7c9739788407e7fff0b52636840846d57" gracePeriod=45 Jul 07 11:00:20.405479 ip-10-0-135-34 kubenswrapper[2578]: I0707 11:00:20.405412 2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-845408/test-model-845408-kserve-5c7ccf7c6c-dmj8q" Jul 07 11:00:20.432471 ip-10-0-135-34 kubenswrapper[2578]: I0707 11:00:20.432444 2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-27bc88/test-model-27bc88-kserve-796f4c7669-hwmvb" Jul 07 11:00:20.432660 ip-10-0-135-34 kubenswrapper[2578]: I0707 11:00:20.432474 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-27bc88/test-model-27bc88-kserve-796f4c7669-hwmvb" event={"ID":"abf6f049-7a87-47e0-8ff6-bd933f589629","Type":"ContainerDied","Data":"241beb3250fbfc70cc9600e01a2665e7851b91fb4860ebd6ce2db45afc749f91"} Jul 07 11:00:20.432660 ip-10-0-135-34 kubenswrapper[2578]: I0707 11:00:20.432541 2578 scope.go:117] "RemoveContainer" containerID="c945e6481f28c3d4286763f6228a2895071a3e895bff7b64dc7fe92bcfb4c9c8" Jul 07 11:00:20.433905 ip-10-0-135-34 kubenswrapper[2578]: I0707 11:00:20.433877 2578 generic.go:356] "Generic (PLEG): container finished" podID="e039759a-4860-4740-942e-a9b7a18d96da" containerID="c0587fdc9155b5a16c6280be684e87d7c9739788407e7fff0b52636840846d57" exitCode=0 Jul 07 11:00:20.434030 ip-10-0-135-34 kubenswrapper[2578]: I0707 11:00:20.433927 2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-845408/test-model-845408-kserve-5c7ccf7c6c-dmj8q" Jul 07 11:00:20.434030 ip-10-0-135-34 kubenswrapper[2578]: I0707 11:00:20.433955 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-845408/test-model-845408-kserve-5c7ccf7c6c-dmj8q" event={"ID":"e039759a-4860-4740-942e-a9b7a18d96da","Type":"ContainerDied","Data":"c0587fdc9155b5a16c6280be684e87d7c9739788407e7fff0b52636840846d57"} Jul 07 11:00:20.434030 ip-10-0-135-34 kubenswrapper[2578]: I0707 11:00:20.433987 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-845408/test-model-845408-kserve-5c7ccf7c6c-dmj8q" event={"ID":"e039759a-4860-4740-942e-a9b7a18d96da","Type":"ContainerDied","Data":"f6dab02433561dc5b567796e95c00d3f224e8223a3d77af497cf03cd89de9507"} Jul 07 11:00:20.443443 ip-10-0-135-34 kubenswrapper[2578]: I0707 11:00:20.443425 2578 scope.go:117] "RemoveContainer" containerID="c0587fdc9155b5a16c6280be684e87d7c9739788407e7fff0b52636840846d57" Jul 07 11:00:20.453215 ip-10-0-135-34 kubenswrapper[2578]: I0707 11:00:20.453194 2578 scope.go:117] "RemoveContainer" containerID="c0587fdc9155b5a16c6280be684e87d7c9739788407e7fff0b52636840846d57" Jul 07 11:00:20.453288 ip-10-0-135-34 kubenswrapper[2578]: I0707 11:00:20.453247 2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-a-27bc88/test-model-27bc88-kserve-796f4c7669-hwmvb"] Jul 07 11:00:20.453520 ip-10-0-135-34 kubenswrapper[2578]: E0707 11:00:20.453502 2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c0587fdc9155b5a16c6280be684e87d7c9739788407e7fff0b52636840846d57\": container with ID starting with c0587fdc9155b5a16c6280be684e87d7c9739788407e7fff0b52636840846d57 not found: ID does not exist" containerID="c0587fdc9155b5a16c6280be684e87d7c9739788407e7fff0b52636840846d57" Jul 07 11:00:20.453567 ip-10-0-135-34 kubenswrapper[2578]: I0707 11:00:20.453529 2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c0587fdc9155b5a16c6280be684e87d7c9739788407e7fff0b52636840846d57"} err="failed to get container status \"c0587fdc9155b5a16c6280be684e87d7c9739788407e7fff0b52636840846d57\": rpc error: code = NotFound desc = could not find container \"c0587fdc9155b5a16c6280be684e87d7c9739788407e7fff0b52636840846d57\": container with ID starting with c0587fdc9155b5a16c6280be684e87d7c9739788407e7fff0b52636840846d57 not found: ID does not exist" Jul 07 11:00:20.456724 ip-10-0-135-34 kubenswrapper[2578]: I0707 11:00:20.456697 2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-inf-a-27bc88/test-model-27bc88-kserve-796f4c7669-hwmvb"] Jul 07 11:00:20.496752 ip-10-0-135-34 kubenswrapper[2578]: I0707 11:00:20.496716 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e039759a-4860-4740-942e-a9b7a18d96da-model-cache\") pod \"e039759a-4860-4740-942e-a9b7a18d96da\" (UID: \"e039759a-4860-4740-942e-a9b7a18d96da\") " Jul 07 11:00:20.496942 ip-10-0-135-34 kubenswrapper[2578]: I0707 11:00:20.496760 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-zrfs4\" (UniqueName: \"kubernetes.io/projected/e039759a-4860-4740-942e-a9b7a18d96da-kube-api-access-zrfs4\") pod \"e039759a-4860-4740-942e-a9b7a18d96da\" (UID: \"e039759a-4860-4740-942e-a9b7a18d96da\") " Jul 07 11:00:20.496942 ip-10-0-135-34 kubenswrapper[2578]: I0707 11:00:20.496807 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e039759a-4860-4740-942e-a9b7a18d96da-dshm\") pod \"e039759a-4860-4740-942e-a9b7a18d96da\" (UID: \"e039759a-4860-4740-942e-a9b7a18d96da\") " Jul 07 11:00:20.496942 ip-10-0-135-34 kubenswrapper[2578]: I0707 11:00:20.496892 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e039759a-4860-4740-942e-a9b7a18d96da-tmp-dir\") pod \"e039759a-4860-4740-942e-a9b7a18d96da\" (UID: \"e039759a-4860-4740-942e-a9b7a18d96da\") " Jul 07 11:00:20.497113 ip-10-0-135-34 kubenswrapper[2578]: I0707 11:00:20.496976 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e039759a-4860-4740-942e-a9b7a18d96da-home\") pod \"e039759a-4860-4740-942e-a9b7a18d96da\" (UID: \"e039759a-4860-4740-942e-a9b7a18d96da\") " Jul 07 11:00:20.497113 ip-10-0-135-34 kubenswrapper[2578]: I0707 11:00:20.497010 2578 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e039759a-4860-4740-942e-a9b7a18d96da-model-cache" (OuterVolumeSpecName: "model-cache") pod "e039759a-4860-4740-942e-a9b7a18d96da" (UID: "e039759a-4860-4740-942e-a9b7a18d96da"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 07 11:00:20.497113 ip-10-0-135-34 kubenswrapper[2578]: I0707 11:00:20.497043 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e039759a-4860-4740-942e-a9b7a18d96da-tls-certs\") pod \"e039759a-4860-4740-942e-a9b7a18d96da\" (UID: \"e039759a-4860-4740-942e-a9b7a18d96da\") " Jul 07 11:00:20.497113 ip-10-0-135-34 kubenswrapper[2578]: I0707 11:00:20.497099 2578 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e039759a-4860-4740-942e-a9b7a18d96da-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "e039759a-4860-4740-942e-a9b7a18d96da" (UID: "e039759a-4860-4740-942e-a9b7a18d96da"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 07 11:00:20.497302 ip-10-0-135-34 kubenswrapper[2578]: I0707 11:00:20.497243 2578 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e039759a-4860-4740-942e-a9b7a18d96da-home" (OuterVolumeSpecName: "home") pod "e039759a-4860-4740-942e-a9b7a18d96da" (UID: "e039759a-4860-4740-942e-a9b7a18d96da"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 07 11:00:20.497380 ip-10-0-135-34 kubenswrapper[2578]: I0707 11:00:20.497362 2578 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e039759a-4860-4740-942e-a9b7a18d96da-model-cache\") on node \"ip-10-0-135-34.ec2.internal\" DevicePath \"\"" Jul 07 11:00:20.497430 ip-10-0-135-34 kubenswrapper[2578]: I0707 11:00:20.497385 2578 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e039759a-4860-4740-942e-a9b7a18d96da-tmp-dir\") on node \"ip-10-0-135-34.ec2.internal\" DevicePath \"\"" Jul 07 11:00:20.497430 ip-10-0-135-34 kubenswrapper[2578]: I0707 11:00:20.497398 2578 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e039759a-4860-4740-942e-a9b7a18d96da-home\") on node \"ip-10-0-135-34.ec2.internal\" DevicePath \"\"" Jul 07 11:00:20.499232 ip-10-0-135-34 kubenswrapper[2578]: I0707 11:00:20.499204 2578 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e039759a-4860-4740-942e-a9b7a18d96da-dshm" (OuterVolumeSpecName: "dshm") pod "e039759a-4860-4740-942e-a9b7a18d96da" (UID: "e039759a-4860-4740-942e-a9b7a18d96da"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 07 11:00:20.499336 ip-10-0-135-34 kubenswrapper[2578]: I0707 11:00:20.499271 2578 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e039759a-4860-4740-942e-a9b7a18d96da-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "e039759a-4860-4740-942e-a9b7a18d96da" (UID: "e039759a-4860-4740-942e-a9b7a18d96da"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 07 11:00:20.499336 ip-10-0-135-34 kubenswrapper[2578]: I0707 11:00:20.499275 2578 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e039759a-4860-4740-942e-a9b7a18d96da-kube-api-access-zrfs4" (OuterVolumeSpecName: "kube-api-access-zrfs4") pod "e039759a-4860-4740-942e-a9b7a18d96da" (UID: "e039759a-4860-4740-942e-a9b7a18d96da"). InnerVolumeSpecName "kube-api-access-zrfs4". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 07 11:00:20.598499 ip-10-0-135-34 kubenswrapper[2578]: I0707 11:00:20.598466 2578 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e039759a-4860-4740-942e-a9b7a18d96da-tls-certs\") on node \"ip-10-0-135-34.ec2.internal\" DevicePath \"\"" Jul 07 11:00:20.598499 ip-10-0-135-34 kubenswrapper[2578]: I0707 11:00:20.598498 2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zrfs4\" (UniqueName: \"kubernetes.io/projected/e039759a-4860-4740-942e-a9b7a18d96da-kube-api-access-zrfs4\") on node \"ip-10-0-135-34.ec2.internal\" DevicePath \"\"" Jul 07 11:00:20.598499 ip-10-0-135-34 kubenswrapper[2578]: I0707 11:00:20.598508 2578 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e039759a-4860-4740-942e-a9b7a18d96da-dshm\") on node \"ip-10-0-135-34.ec2.internal\" DevicePath \"\"" Jul 07 11:00:20.757742 ip-10-0-135-34 kubenswrapper[2578]: I0707 11:00:20.757701 2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-b-845408/test-model-845408-kserve-5c7ccf7c6c-dmj8q"] Jul 07 11:00:20.760558 ip-10-0-135-34 kubenswrapper[2578]: I0707 11:00:20.760531 2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-inf-b-845408/test-model-845408-kserve-5c7ccf7c6c-dmj8q"] Jul 07 11:00:21.821442 ip-10-0-135-34 kubenswrapper[2578]: I0707 11:00:21.821408 2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="abf6f049-7a87-47e0-8ff6-bd933f589629" path="/var/lib/kubelet/pods/abf6f049-7a87-47e0-8ff6-bd933f589629/volumes" Jul 07 11:00:21.821833 ip-10-0-135-34 kubenswrapper[2578]: I0707 11:00:21.821785 2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="e039759a-4860-4740-942e-a9b7a18d96da" path="/var/lib/kubelet/pods/e039759a-4860-4740-942e-a9b7a18d96da/volumes" Jul 07 11:00:51.326710 ip-10-0-135-34 kubenswrapper[2578]: I0707 11:00:51.326663 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-lv7nd"] Jul 07 11:00:59.368516 ip-10-0-135-34 kubenswrapper[2578]: I0707 11:00:59.368480 2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/maas-api-e2e-api-a-5a74a9-78d59dcf89-5zqqc"] Jul 07 11:00:59.368909 ip-10-0-135-34 kubenswrapper[2578]: I0707 11:00:59.368852 2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="abf6f049-7a87-47e0-8ff6-bd933f589629" containerName="main" Jul 07 11:00:59.368909 ip-10-0-135-34 kubenswrapper[2578]: I0707 11:00:59.368867 2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="abf6f049-7a87-47e0-8ff6-bd933f589629" containerName="main" Jul 07 11:00:59.368909 ip-10-0-135-34 kubenswrapper[2578]: I0707 11:00:59.368890 2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="adda5fbb-b371-4ae4-a414-a7c5f28f0fdc" containerName="maas-api" Jul 07 11:00:59.368909 ip-10-0-135-34 kubenswrapper[2578]: I0707 11:00:59.368896 2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="adda5fbb-b371-4ae4-a414-a7c5f28f0fdc" containerName="maas-api" Jul 07 11:00:59.368909 ip-10-0-135-34 kubenswrapper[2578]: I0707 11:00:59.368904 2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e039759a-4860-4740-942e-a9b7a18d96da" containerName="main" Jul 07 11:00:59.368909 ip-10-0-135-34 kubenswrapper[2578]: I0707 11:00:59.368909 2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="e039759a-4860-4740-942e-a9b7a18d96da" containerName="main" Jul 07 11:00:59.369101 ip-10-0-135-34 kubenswrapper[2578]: I0707 11:00:59.368970 2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="abf6f049-7a87-47e0-8ff6-bd933f589629" containerName="main" Jul 07 11:00:59.369101 ip-10-0-135-34 kubenswrapper[2578]: I0707 11:00:59.368981 2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="adda5fbb-b371-4ae4-a414-a7c5f28f0fdc" containerName="maas-api" Jul 07 11:00:59.369101 ip-10-0-135-34 kubenswrapper[2578]: I0707 11:00:59.368990 2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="e039759a-4860-4740-942e-a9b7a18d96da" containerName="main" Jul 07 11:00:59.372379 ip-10-0-135-34 kubenswrapper[2578]: I0707 11:00:59.372356 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-api-e2e-api-a-5a74a9-78d59dcf89-5zqqc" Jul 07 11:00:59.375254 ip-10-0-135-34 kubenswrapper[2578]: I0707 11:00:59.375226 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-api-e2e-api-a-5a74a9-serving-cert\"" Jul 07 11:00:59.375378 ip-10-0-135-34 kubenswrapper[2578]: I0707 11:00:59.375303 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-api-dockercfg-qlm6h\"" Jul 07 11:00:59.383499 ip-10-0-135-34 kubenswrapper[2578]: I0707 11:00:59.383477 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-api-e2e-api-a-5a74a9-78d59dcf89-5zqqc"] Jul 07 11:00:59.539013 ip-10-0-135-34 kubenswrapper[2578]: I0707 11:00:59.538978 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/8de653d2-aebf-424b-aaec-dfeab0767a03-openshift-service-ca\") pod \"maas-api-e2e-api-a-5a74a9-78d59dcf89-5zqqc\" (UID: \"8de653d2-aebf-424b-aaec-dfeab0767a03\") " pod="opendatahub/maas-api-e2e-api-a-5a74a9-78d59dcf89-5zqqc" Jul 07 11:00:59.539013 ip-10-0-135-34 kubenswrapper[2578]: I0707 11:00:59.539016 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-v8hr7\" (UniqueName: \"kubernetes.io/projected/8de653d2-aebf-424b-aaec-dfeab0767a03-kube-api-access-v8hr7\") pod \"maas-api-e2e-api-a-5a74a9-78d59dcf89-5zqqc\" (UID: \"8de653d2-aebf-424b-aaec-dfeab0767a03\") " pod="opendatahub/maas-api-e2e-api-a-5a74a9-78d59dcf89-5zqqc" Jul 07 11:00:59.539236 ip-10-0-135-34 kubenswrapper[2578]: I0707 11:00:59.539093 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8de653d2-aebf-424b-aaec-dfeab0767a03-maas-api-tls\") pod \"maas-api-e2e-api-a-5a74a9-78d59dcf89-5zqqc\" (UID: \"8de653d2-aebf-424b-aaec-dfeab0767a03\") " pod="opendatahub/maas-api-e2e-api-a-5a74a9-78d59dcf89-5zqqc" Jul 07 11:00:59.640456 ip-10-0-135-34 kubenswrapper[2578]: I0707 11:00:59.640359 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/8de653d2-aebf-424b-aaec-dfeab0767a03-openshift-service-ca\") pod \"maas-api-e2e-api-a-5a74a9-78d59dcf89-5zqqc\" (UID: \"8de653d2-aebf-424b-aaec-dfeab0767a03\") " pod="opendatahub/maas-api-e2e-api-a-5a74a9-78d59dcf89-5zqqc" Jul 07 11:00:59.640456 ip-10-0-135-34 kubenswrapper[2578]: I0707 11:00:59.640401 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v8hr7\" (UniqueName: \"kubernetes.io/projected/8de653d2-aebf-424b-aaec-dfeab0767a03-kube-api-access-v8hr7\") pod \"maas-api-e2e-api-a-5a74a9-78d59dcf89-5zqqc\" (UID: \"8de653d2-aebf-424b-aaec-dfeab0767a03\") " pod="opendatahub/maas-api-e2e-api-a-5a74a9-78d59dcf89-5zqqc" Jul 07 11:00:59.640698 ip-10-0-135-34 kubenswrapper[2578]: I0707 11:00:59.640461 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8de653d2-aebf-424b-aaec-dfeab0767a03-maas-api-tls\") pod \"maas-api-e2e-api-a-5a74a9-78d59dcf89-5zqqc\" (UID: \"8de653d2-aebf-424b-aaec-dfeab0767a03\") " pod="opendatahub/maas-api-e2e-api-a-5a74a9-78d59dcf89-5zqqc" Jul 07 11:00:59.640698 ip-10-0-135-34 kubenswrapper[2578]: E0707 11:00:59.640593 2578 secret.go:189] Couldn't get secret opendatahub/maas-api-e2e-api-a-5a74a9-serving-cert: secret "maas-api-e2e-api-a-5a74a9-serving-cert" not found Jul 07 11:00:59.640698 ip-10-0-135-34 kubenswrapper[2578]: E0707 11:00:59.640672 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8de653d2-aebf-424b-aaec-dfeab0767a03-maas-api-tls podName:8de653d2-aebf-424b-aaec-dfeab0767a03 nodeName:}" failed. No retries permitted until 2026-07-07 11:01:00.140651004 +0000 UTC m=+2028.812610546 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/8de653d2-aebf-424b-aaec-dfeab0767a03-maas-api-tls") pod "maas-api-e2e-api-a-5a74a9-78d59dcf89-5zqqc" (UID: "8de653d2-aebf-424b-aaec-dfeab0767a03") : secret "maas-api-e2e-api-a-5a74a9-serving-cert" not found Jul 07 11:00:59.641115 ip-10-0-135-34 kubenswrapper[2578]: I0707 11:00:59.641095 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/8de653d2-aebf-424b-aaec-dfeab0767a03-openshift-service-ca\") pod \"maas-api-e2e-api-a-5a74a9-78d59dcf89-5zqqc\" (UID: \"8de653d2-aebf-424b-aaec-dfeab0767a03\") " pod="opendatahub/maas-api-e2e-api-a-5a74a9-78d59dcf89-5zqqc" Jul 07 11:00:59.650171 ip-10-0-135-34 kubenswrapper[2578]: I0707 11:00:59.650138 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-v8hr7\" (UniqueName: \"kubernetes.io/projected/8de653d2-aebf-424b-aaec-dfeab0767a03-kube-api-access-v8hr7\") pod \"maas-api-e2e-api-a-5a74a9-78d59dcf89-5zqqc\" (UID: \"8de653d2-aebf-424b-aaec-dfeab0767a03\") " pod="opendatahub/maas-api-e2e-api-a-5a74a9-78d59dcf89-5zqqc" Jul 07 11:01:00.145215 ip-10-0-135-34 kubenswrapper[2578]: I0707 11:01:00.145180 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8de653d2-aebf-424b-aaec-dfeab0767a03-maas-api-tls\") pod \"maas-api-e2e-api-a-5a74a9-78d59dcf89-5zqqc\" (UID: \"8de653d2-aebf-424b-aaec-dfeab0767a03\") " pod="opendatahub/maas-api-e2e-api-a-5a74a9-78d59dcf89-5zqqc" Jul 07 11:01:00.147807 ip-10-0-135-34 kubenswrapper[2578]: I0707 11:01:00.147766 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8de653d2-aebf-424b-aaec-dfeab0767a03-maas-api-tls\") pod \"maas-api-e2e-api-a-5a74a9-78d59dcf89-5zqqc\" (UID: \"8de653d2-aebf-424b-aaec-dfeab0767a03\") " pod="opendatahub/maas-api-e2e-api-a-5a74a9-78d59dcf89-5zqqc" Jul 07 11:01:00.286196 ip-10-0-135-34 kubenswrapper[2578]: I0707 11:01:00.286165 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-api-e2e-api-a-5a74a9-78d59dcf89-5zqqc" Jul 07 11:01:00.422431 ip-10-0-135-34 kubenswrapper[2578]: I0707 11:01:00.422403 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-api-e2e-api-a-5a74a9-78d59dcf89-5zqqc"] Jul 07 11:01:00.424599 ip-10-0-135-34 kubenswrapper[2578]: W0707 11:01:00.424573 2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8de653d2_aebf_424b_aaec_dfeab0767a03.slice/crio-a9cf1ade8a44bde20deb0754020c92626b7488a762e9880a54276d990d26f587 WatchSource:0}: Error finding container a9cf1ade8a44bde20deb0754020c92626b7488a762e9880a54276d990d26f587: Status 404 returned error can't find the container with id a9cf1ade8a44bde20deb0754020c92626b7488a762e9880a54276d990d26f587 Jul 07 11:01:00.630177 ip-10-0-135-34 kubenswrapper[2578]: I0707 11:01:00.630140 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-api-e2e-api-a-5a74a9-78d59dcf89-5zqqc" event={"ID":"8de653d2-aebf-424b-aaec-dfeab0767a03","Type":"ContainerStarted","Data":"a9cf1ade8a44bde20deb0754020c92626b7488a762e9880a54276d990d26f587"} Jul 07 11:01:01.636600 ip-10-0-135-34 kubenswrapper[2578]: I0707 11:01:01.636559 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-api-e2e-api-a-5a74a9-78d59dcf89-5zqqc" event={"ID":"8de653d2-aebf-424b-aaec-dfeab0767a03","Type":"ContainerStarted","Data":"55d438954e5a611179380f10e6c3b8d1181d9c84555f70edd3e2ab9534f818e3"} Jul 07 11:01:01.637061 ip-10-0-135-34 kubenswrapper[2578]: I0707 11:01:01.636712 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-api-e2e-api-a-5a74a9-78d59dcf89-5zqqc" Jul 07 11:01:01.657091 ip-10-0-135-34 kubenswrapper[2578]: I0707 11:01:01.657043 2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/maas-api-e2e-api-a-5a74a9-78d59dcf89-5zqqc" podStartSLOduration=1.63375505 podStartE2EDuration="2.657024208s" totalImagesPullingTime="1.023269158s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-07 11:00:59 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 11:01:01.65549509 +0000 UTC m=+2030.327454645" watchObservedRunningTime="2026-07-07 11:01:01.657024208 +0000 UTC m=+2030.328983752" Jul 07 11:01:04.526639 ip-10-0-135-34 kubenswrapper[2578]: I0707 11:01:04.526599 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-lv7nd"] Jul 07 11:01:07.647654 ip-10-0-135-34 kubenswrapper[2578]: I0707 11:01:07.647626 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/maas-api-e2e-api-a-5a74a9-78d59dcf89-5zqqc" Jul 07 11:01:17.221823 ip-10-0-135-34 kubenswrapper[2578]: I0707 11:01:17.221785 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-lv7nd"] Jul 07 11:01:20.654609 ip-10-0-135-34 kubenswrapper[2578]: I0707 11:01:20.654575 2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-api-e2e-api-a-5a74a9-78d59dcf89-5zqqc"] Jul 07 11:01:20.655053 ip-10-0-135-34 kubenswrapper[2578]: I0707 11:01:20.654795 2578 kuberuntime_container.go:872] "Killing container with a grace period" pod="opendatahub/maas-api-e2e-api-a-5a74a9-78d59dcf89-5zqqc" podUID="8de653d2-aebf-424b-aaec-dfeab0767a03" containerName="maas-api" containerID="cri-o://55d438954e5a611179380f10e6c3b8d1181d9c84555f70edd3e2ab9534f818e3" gracePeriod=30 Jul 07 11:01:20.910710 ip-10-0-135-34 kubenswrapper[2578]: I0707 11:01:20.910636 2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-api-e2e-api-a-5a74a9-78d59dcf89-5zqqc" Jul 07 11:01:21.021481 ip-10-0-135-34 kubenswrapper[2578]: I0707 11:01:21.021439 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/8de653d2-aebf-424b-aaec-dfeab0767a03-openshift-service-ca\") pod \"8de653d2-aebf-424b-aaec-dfeab0767a03\" (UID: \"8de653d2-aebf-424b-aaec-dfeab0767a03\") " Jul 07 11:01:21.021481 ip-10-0-135-34 kubenswrapper[2578]: I0707 11:01:21.021489 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8de653d2-aebf-424b-aaec-dfeab0767a03-maas-api-tls\") pod \"8de653d2-aebf-424b-aaec-dfeab0767a03\" (UID: \"8de653d2-aebf-424b-aaec-dfeab0767a03\") " Jul 07 11:01:21.021672 ip-10-0-135-34 kubenswrapper[2578]: I0707 11:01:21.021519 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-v8hr7\" (UniqueName: \"kubernetes.io/projected/8de653d2-aebf-424b-aaec-dfeab0767a03-kube-api-access-v8hr7\") pod \"8de653d2-aebf-424b-aaec-dfeab0767a03\" (UID: \"8de653d2-aebf-424b-aaec-dfeab0767a03\") " Jul 07 11:01:21.021871 ip-10-0-135-34 kubenswrapper[2578]: I0707 11:01:21.021846 2578 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8de653d2-aebf-424b-aaec-dfeab0767a03-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "8de653d2-aebf-424b-aaec-dfeab0767a03" (UID: "8de653d2-aebf-424b-aaec-dfeab0767a03"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 07 11:01:21.023805 ip-10-0-135-34 kubenswrapper[2578]: I0707 11:01:21.023753 2578 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8de653d2-aebf-424b-aaec-dfeab0767a03-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "8de653d2-aebf-424b-aaec-dfeab0767a03" (UID: "8de653d2-aebf-424b-aaec-dfeab0767a03"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 07 11:01:21.023805 ip-10-0-135-34 kubenswrapper[2578]: I0707 11:01:21.023794 2578 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8de653d2-aebf-424b-aaec-dfeab0767a03-kube-api-access-v8hr7" (OuterVolumeSpecName: "kube-api-access-v8hr7") pod "8de653d2-aebf-424b-aaec-dfeab0767a03" (UID: "8de653d2-aebf-424b-aaec-dfeab0767a03"). InnerVolumeSpecName "kube-api-access-v8hr7". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 07 11:01:21.122928 ip-10-0-135-34 kubenswrapper[2578]: I0707 11:01:21.122879 2578 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8de653d2-aebf-424b-aaec-dfeab0767a03-maas-api-tls\") on node \"ip-10-0-135-34.ec2.internal\" DevicePath \"\"" Jul 07 11:01:21.122928 ip-10-0-135-34 kubenswrapper[2578]: I0707 11:01:21.122921 2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-v8hr7\" (UniqueName: \"kubernetes.io/projected/8de653d2-aebf-424b-aaec-dfeab0767a03-kube-api-access-v8hr7\") on node \"ip-10-0-135-34.ec2.internal\" DevicePath \"\"" Jul 07 11:01:21.122928 ip-10-0-135-34 kubenswrapper[2578]: I0707 11:01:21.122935 2578 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/8de653d2-aebf-424b-aaec-dfeab0767a03-openshift-service-ca\") on node \"ip-10-0-135-34.ec2.internal\" DevicePath \"\"" Jul 07 11:01:21.736886 ip-10-0-135-34 kubenswrapper[2578]: I0707 11:01:21.736853 2578 generic.go:356] "Generic (PLEG): container finished" podID="8de653d2-aebf-424b-aaec-dfeab0767a03" containerID="55d438954e5a611179380f10e6c3b8d1181d9c84555f70edd3e2ab9534f818e3" exitCode=0 Jul 07 11:01:21.737303 ip-10-0-135-34 kubenswrapper[2578]: I0707 11:01:21.736901 2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-api-e2e-api-a-5a74a9-78d59dcf89-5zqqc" Jul 07 11:01:21.737303 ip-10-0-135-34 kubenswrapper[2578]: I0707 11:01:21.736900 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-api-e2e-api-a-5a74a9-78d59dcf89-5zqqc" event={"ID":"8de653d2-aebf-424b-aaec-dfeab0767a03","Type":"ContainerDied","Data":"55d438954e5a611179380f10e6c3b8d1181d9c84555f70edd3e2ab9534f818e3"} Jul 07 11:01:21.737303 ip-10-0-135-34 kubenswrapper[2578]: I0707 11:01:21.737031 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-api-e2e-api-a-5a74a9-78d59dcf89-5zqqc" event={"ID":"8de653d2-aebf-424b-aaec-dfeab0767a03","Type":"ContainerDied","Data":"a9cf1ade8a44bde20deb0754020c92626b7488a762e9880a54276d990d26f587"} Jul 07 11:01:21.737303 ip-10-0-135-34 kubenswrapper[2578]: I0707 11:01:21.737067 2578 scope.go:117] "RemoveContainer" containerID="55d438954e5a611179380f10e6c3b8d1181d9c84555f70edd3e2ab9534f818e3" Jul 07 11:01:21.746045 ip-10-0-135-34 kubenswrapper[2578]: I0707 11:01:21.746025 2578 scope.go:117] "RemoveContainer" containerID="55d438954e5a611179380f10e6c3b8d1181d9c84555f70edd3e2ab9534f818e3" Jul 07 11:01:21.746335 ip-10-0-135-34 kubenswrapper[2578]: E0707 11:01:21.746315 2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"55d438954e5a611179380f10e6c3b8d1181d9c84555f70edd3e2ab9534f818e3\": container with ID starting with 55d438954e5a611179380f10e6c3b8d1181d9c84555f70edd3e2ab9534f818e3 not found: ID does not exist" containerID="55d438954e5a611179380f10e6c3b8d1181d9c84555f70edd3e2ab9534f818e3" Jul 07 11:01:21.746389 ip-10-0-135-34 kubenswrapper[2578]: I0707 11:01:21.746344 2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"55d438954e5a611179380f10e6c3b8d1181d9c84555f70edd3e2ab9534f818e3"} err="failed to get container status \"55d438954e5a611179380f10e6c3b8d1181d9c84555f70edd3e2ab9534f818e3\": rpc error: code = NotFound desc = could not find container \"55d438954e5a611179380f10e6c3b8d1181d9c84555f70edd3e2ab9534f818e3\": container with ID starting with 55d438954e5a611179380f10e6c3b8d1181d9c84555f70edd3e2ab9534f818e3 not found: ID does not exist" Jul 07 11:01:21.758247 ip-10-0-135-34 kubenswrapper[2578]: I0707 11:01:21.758223 2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-api-e2e-api-a-5a74a9-78d59dcf89-5zqqc"] Jul 07 11:01:21.763267 ip-10-0-135-34 kubenswrapper[2578]: I0707 11:01:21.763244 2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/maas-api-e2e-api-a-5a74a9-78d59dcf89-5zqqc"] Jul 07 11:01:21.821598 ip-10-0-135-34 kubenswrapper[2578]: I0707 11:01:21.821566 2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="8de653d2-aebf-424b-aaec-dfeab0767a03" path="/var/lib/kubelet/pods/8de653d2-aebf-424b-aaec-dfeab0767a03/volumes" Jul 07 11:01:30.421025 ip-10-0-135-34 kubenswrapper[2578]: I0707 11:01:30.420988 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-lv7nd"] Jul 07 11:01:37.461789 ip-10-0-135-34 kubenswrapper[2578]: I0707 11:01:37.461734 2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/maas-api-e2e-shared-a-1273de-7f7fb4697-sdqlv"] Jul 07 11:01:37.462260 ip-10-0-135-34 kubenswrapper[2578]: I0707 11:01:37.462126 2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8de653d2-aebf-424b-aaec-dfeab0767a03" containerName="maas-api" Jul 07 11:01:37.462260 ip-10-0-135-34 kubenswrapper[2578]: I0707 11:01:37.462140 2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="8de653d2-aebf-424b-aaec-dfeab0767a03" containerName="maas-api" Jul 07 11:01:37.462260 ip-10-0-135-34 kubenswrapper[2578]: I0707 11:01:37.462213 2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="8de653d2-aebf-424b-aaec-dfeab0767a03" containerName="maas-api" Jul 07 11:01:37.466415 ip-10-0-135-34 kubenswrapper[2578]: I0707 11:01:37.466386 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-api-e2e-shared-a-1273de-7f7fb4697-sdqlv" Jul 07 11:01:37.469094 ip-10-0-135-34 kubenswrapper[2578]: I0707 11:01:37.469067 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-api-e2e-shared-a-1273de-serving-cert\"" Jul 07 11:01:37.469244 ip-10-0-135-34 kubenswrapper[2578]: I0707 11:01:37.469159 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-api-dockercfg-qlm6h\"" Jul 07 11:01:37.475484 ip-10-0-135-34 kubenswrapper[2578]: I0707 11:01:37.475459 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-api-e2e-shared-a-1273de-7f7fb4697-sdqlv"] Jul 07 11:01:37.568861 ip-10-0-135-34 kubenswrapper[2578]: I0707 11:01:37.568818 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kmcqp\" (UniqueName: \"kubernetes.io/projected/3a107748-57b4-4e96-8684-8bd1232af914-kube-api-access-kmcqp\") pod \"maas-api-e2e-shared-a-1273de-7f7fb4697-sdqlv\" (UID: \"3a107748-57b4-4e96-8684-8bd1232af914\") " pod="opendatahub/maas-api-e2e-shared-a-1273de-7f7fb4697-sdqlv" Jul 07 11:01:37.569075 ip-10-0-135-34 kubenswrapper[2578]: I0707 11:01:37.568940 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/3a107748-57b4-4e96-8684-8bd1232af914-openshift-service-ca\") pod \"maas-api-e2e-shared-a-1273de-7f7fb4697-sdqlv\" (UID: \"3a107748-57b4-4e96-8684-8bd1232af914\") " pod="opendatahub/maas-api-e2e-shared-a-1273de-7f7fb4697-sdqlv" Jul 07 11:01:37.569075 ip-10-0-135-34 kubenswrapper[2578]: I0707 11:01:37.568984 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3a107748-57b4-4e96-8684-8bd1232af914-maas-api-tls\") pod \"maas-api-e2e-shared-a-1273de-7f7fb4697-sdqlv\" (UID: \"3a107748-57b4-4e96-8684-8bd1232af914\") " pod="opendatahub/maas-api-e2e-shared-a-1273de-7f7fb4697-sdqlv" Jul 07 11:01:37.670093 ip-10-0-135-34 kubenswrapper[2578]: I0707 11:01:37.670056 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kmcqp\" (UniqueName: \"kubernetes.io/projected/3a107748-57b4-4e96-8684-8bd1232af914-kube-api-access-kmcqp\") pod \"maas-api-e2e-shared-a-1273de-7f7fb4697-sdqlv\" (UID: \"3a107748-57b4-4e96-8684-8bd1232af914\") " pod="opendatahub/maas-api-e2e-shared-a-1273de-7f7fb4697-sdqlv" Jul 07 11:01:37.670301 ip-10-0-135-34 kubenswrapper[2578]: I0707 11:01:37.670140 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/3a107748-57b4-4e96-8684-8bd1232af914-openshift-service-ca\") pod \"maas-api-e2e-shared-a-1273de-7f7fb4697-sdqlv\" (UID: \"3a107748-57b4-4e96-8684-8bd1232af914\") " pod="opendatahub/maas-api-e2e-shared-a-1273de-7f7fb4697-sdqlv" Jul 07 11:01:37.670301 ip-10-0-135-34 kubenswrapper[2578]: I0707 11:01:37.670182 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3a107748-57b4-4e96-8684-8bd1232af914-maas-api-tls\") pod \"maas-api-e2e-shared-a-1273de-7f7fb4697-sdqlv\" (UID: \"3a107748-57b4-4e96-8684-8bd1232af914\") " pod="opendatahub/maas-api-e2e-shared-a-1273de-7f7fb4697-sdqlv" Jul 07 11:01:37.670301 ip-10-0-135-34 kubenswrapper[2578]: E0707 11:01:37.670285 2578 secret.go:189] Couldn't get secret opendatahub/maas-api-e2e-shared-a-1273de-serving-cert: secret "maas-api-e2e-shared-a-1273de-serving-cert" not found Jul 07 11:01:37.670459 ip-10-0-135-34 kubenswrapper[2578]: E0707 11:01:37.670349 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3a107748-57b4-4e96-8684-8bd1232af914-maas-api-tls podName:3a107748-57b4-4e96-8684-8bd1232af914 nodeName:}" failed. No retries permitted until 2026-07-07 11:01:38.170330404 +0000 UTC m=+2066.842289929 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/3a107748-57b4-4e96-8684-8bd1232af914-maas-api-tls") pod "maas-api-e2e-shared-a-1273de-7f7fb4697-sdqlv" (UID: "3a107748-57b4-4e96-8684-8bd1232af914") : secret "maas-api-e2e-shared-a-1273de-serving-cert" not found Jul 07 11:01:37.670816 ip-10-0-135-34 kubenswrapper[2578]: I0707 11:01:37.670793 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/3a107748-57b4-4e96-8684-8bd1232af914-openshift-service-ca\") pod \"maas-api-e2e-shared-a-1273de-7f7fb4697-sdqlv\" (UID: \"3a107748-57b4-4e96-8684-8bd1232af914\") " pod="opendatahub/maas-api-e2e-shared-a-1273de-7f7fb4697-sdqlv" Jul 07 11:01:37.679320 ip-10-0-135-34 kubenswrapper[2578]: I0707 11:01:37.679297 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kmcqp\" (UniqueName: \"kubernetes.io/projected/3a107748-57b4-4e96-8684-8bd1232af914-kube-api-access-kmcqp\") pod \"maas-api-e2e-shared-a-1273de-7f7fb4697-sdqlv\" (UID: \"3a107748-57b4-4e96-8684-8bd1232af914\") " pod="opendatahub/maas-api-e2e-shared-a-1273de-7f7fb4697-sdqlv" Jul 07 11:01:38.174933 ip-10-0-135-34 kubenswrapper[2578]: I0707 11:01:38.174887 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3a107748-57b4-4e96-8684-8bd1232af914-maas-api-tls\") pod \"maas-api-e2e-shared-a-1273de-7f7fb4697-sdqlv\" (UID: \"3a107748-57b4-4e96-8684-8bd1232af914\") " pod="opendatahub/maas-api-e2e-shared-a-1273de-7f7fb4697-sdqlv" Jul 07 11:01:38.177447 ip-10-0-135-34 kubenswrapper[2578]: I0707 11:01:38.177418 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3a107748-57b4-4e96-8684-8bd1232af914-maas-api-tls\") pod \"maas-api-e2e-shared-a-1273de-7f7fb4697-sdqlv\" (UID: \"3a107748-57b4-4e96-8684-8bd1232af914\") " pod="opendatahub/maas-api-e2e-shared-a-1273de-7f7fb4697-sdqlv" Jul 07 11:01:38.378888 ip-10-0-135-34 kubenswrapper[2578]: I0707 11:01:38.378853 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-api-e2e-shared-a-1273de-7f7fb4697-sdqlv" Jul 07 11:01:38.517369 ip-10-0-135-34 kubenswrapper[2578]: I0707 11:01:38.517343 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-api-e2e-shared-a-1273de-7f7fb4697-sdqlv"] Jul 07 11:01:38.519683 ip-10-0-135-34 kubenswrapper[2578]: W0707 11:01:38.519656 2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3a107748_57b4_4e96_8684_8bd1232af914.slice/crio-3e602563b5647fd71fcf3ef46ae1c2fdacca78ffdcf9f03461d366c4c0a44766 WatchSource:0}: Error finding container 3e602563b5647fd71fcf3ef46ae1c2fdacca78ffdcf9f03461d366c4c0a44766: Status 404 returned error can't find the container with id 3e602563b5647fd71fcf3ef46ae1c2fdacca78ffdcf9f03461d366c4c0a44766 Jul 07 11:01:38.824476 ip-10-0-135-34 kubenswrapper[2578]: I0707 11:01:38.824392 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-api-e2e-shared-a-1273de-7f7fb4697-sdqlv" event={"ID":"3a107748-57b4-4e96-8684-8bd1232af914","Type":"ContainerStarted","Data":"3e602563b5647fd71fcf3ef46ae1c2fdacca78ffdcf9f03461d366c4c0a44766"} Jul 07 11:01:39.830821 ip-10-0-135-34 kubenswrapper[2578]: I0707 11:01:39.830785 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-api-e2e-shared-a-1273de-7f7fb4697-sdqlv" event={"ID":"3a107748-57b4-4e96-8684-8bd1232af914","Type":"ContainerStarted","Data":"6d95630a7f54775b731250fcc83679765cd2761735aafba99ae5be0c1e053fc3"} Jul 07 11:01:39.831245 ip-10-0-135-34 kubenswrapper[2578]: I0707 11:01:39.830872 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-api-e2e-shared-a-1273de-7f7fb4697-sdqlv" Jul 07 11:01:39.849498 ip-10-0-135-34 kubenswrapper[2578]: I0707 11:01:39.849442 2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/maas-api-e2e-shared-a-1273de-7f7fb4697-sdqlv" podStartSLOduration=2.4973204190000002 podStartE2EDuration="2.849425665s" totalImagesPullingTime="352.105246ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-07 11:01:37 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 11:01:39.848275902 +0000 UTC m=+2068.520235445" watchObservedRunningTime="2026-07-07 11:01:39.849425665 +0000 UTC m=+2068.521385187" Jul 07 11:01:45.120488 ip-10-0-135-34 kubenswrapper[2578]: I0707 11:01:45.120443 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-lv7nd"] Jul 07 11:01:45.841479 ip-10-0-135-34 kubenswrapper[2578]: I0707 11:01:45.841447 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/maas-api-e2e-shared-a-1273de-7f7fb4697-sdqlv" Jul 07 11:01:56.127610 ip-10-0-135-34 kubenswrapper[2578]: I0707 11:01:56.127568 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-lv7nd"] Jul 07 11:02:11.964817 ip-10-0-135-34 kubenswrapper[2578]: I0707 11:02:11.964763 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-qhhw7_aca7528d-ddbe-4282-979f-1b496011f0e1/ovn-acl-logging/0.log" Jul 07 11:02:11.970976 ip-10-0-135-34 kubenswrapper[2578]: I0707 11:02:11.970954 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-qhhw7_aca7528d-ddbe-4282-979f-1b496011f0e1/ovn-acl-logging/0.log" Jul 07 11:02:14.685095 ip-10-0-135-34 kubenswrapper[2578]: I0707 11:02:14.685062 2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-a-1273de/auth-test-model-1273de-kserve-84ff499d67-c4jmh"] Jul 07 11:02:14.687375 ip-10-0-135-34 kubenswrapper[2578]: I0707 11:02:14.687359 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-1273de/auth-test-model-1273de-kserve-84ff499d67-c4jmh" Jul 07 11:02:14.691339 ip-10-0-135-34 kubenswrapper[2578]: I0707 11:02:14.691319 2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-1273de\"/\"openshift-service-ca.crt\"" Jul 07 11:02:14.691457 ip-10-0-135-34 kubenswrapper[2578]: I0707 11:02:14.691349 2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-1273de\"/\"kube-root-ca.crt\"" Jul 07 11:02:14.691457 ip-10-0-135-34 kubenswrapper[2578]: I0707 11:02:14.691322 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-1273de\"/\"auth-test-model-1273de-kserve-self-signed-certs\"" Jul 07 11:02:14.691457 ip-10-0-135-34 kubenswrapper[2578]: I0707 11:02:14.691322 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-1273de\"/\"default-dockercfg-kbxpw\"" Jul 07 11:02:14.696952 ip-10-0-135-34 kubenswrapper[2578]: I0707 11:02:14.696932 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-1273de/auth-test-model-1273de-kserve-84ff499d67-c4jmh"] Jul 07 11:02:14.780695 ip-10-0-135-34 kubenswrapper[2578]: I0707 11:02:14.780659 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2z7kv\" (UniqueName: \"kubernetes.io/projected/124dff2e-e721-4aea-b5e7-4fcc5369e86a-kube-api-access-2z7kv\") pod \"auth-test-model-1273de-kserve-84ff499d67-c4jmh\" (UID: \"124dff2e-e721-4aea-b5e7-4fcc5369e86a\") " pod="ai-tenant-e2e-shared-a-1273de/auth-test-model-1273de-kserve-84ff499d67-c4jmh" Jul 07 11:02:14.780907 ip-10-0-135-34 kubenswrapper[2578]: I0707 11:02:14.780718 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/124dff2e-e721-4aea-b5e7-4fcc5369e86a-dshm\") pod \"auth-test-model-1273de-kserve-84ff499d67-c4jmh\" (UID: \"124dff2e-e721-4aea-b5e7-4fcc5369e86a\") " pod="ai-tenant-e2e-shared-a-1273de/auth-test-model-1273de-kserve-84ff499d67-c4jmh" Jul 07 11:02:14.780907 ip-10-0-135-34 kubenswrapper[2578]: I0707 11:02:14.780752 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/124dff2e-e721-4aea-b5e7-4fcc5369e86a-home\") pod \"auth-test-model-1273de-kserve-84ff499d67-c4jmh\" (UID: \"124dff2e-e721-4aea-b5e7-4fcc5369e86a\") " pod="ai-tenant-e2e-shared-a-1273de/auth-test-model-1273de-kserve-84ff499d67-c4jmh" Jul 07 11:02:14.780907 ip-10-0-135-34 kubenswrapper[2578]: I0707 11:02:14.780808 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/124dff2e-e721-4aea-b5e7-4fcc5369e86a-tmp-dir\") pod \"auth-test-model-1273de-kserve-84ff499d67-c4jmh\" (UID: \"124dff2e-e721-4aea-b5e7-4fcc5369e86a\") " pod="ai-tenant-e2e-shared-a-1273de/auth-test-model-1273de-kserve-84ff499d67-c4jmh" Jul 07 11:02:14.780907 ip-10-0-135-34 kubenswrapper[2578]: I0707 11:02:14.780833 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/124dff2e-e721-4aea-b5e7-4fcc5369e86a-model-cache\") pod \"auth-test-model-1273de-kserve-84ff499d67-c4jmh\" (UID: \"124dff2e-e721-4aea-b5e7-4fcc5369e86a\") " pod="ai-tenant-e2e-shared-a-1273de/auth-test-model-1273de-kserve-84ff499d67-c4jmh" Jul 07 11:02:14.780907 ip-10-0-135-34 kubenswrapper[2578]: I0707 11:02:14.780858 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/124dff2e-e721-4aea-b5e7-4fcc5369e86a-tls-certs\") pod \"auth-test-model-1273de-kserve-84ff499d67-c4jmh\" (UID: \"124dff2e-e721-4aea-b5e7-4fcc5369e86a\") " pod="ai-tenant-e2e-shared-a-1273de/auth-test-model-1273de-kserve-84ff499d67-c4jmh" Jul 07 11:02:14.881815 ip-10-0-135-34 kubenswrapper[2578]: I0707 11:02:14.881751 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2z7kv\" (UniqueName: \"kubernetes.io/projected/124dff2e-e721-4aea-b5e7-4fcc5369e86a-kube-api-access-2z7kv\") pod \"auth-test-model-1273de-kserve-84ff499d67-c4jmh\" (UID: \"124dff2e-e721-4aea-b5e7-4fcc5369e86a\") " pod="ai-tenant-e2e-shared-a-1273de/auth-test-model-1273de-kserve-84ff499d67-c4jmh" Jul 07 11:02:14.882009 ip-10-0-135-34 kubenswrapper[2578]: I0707 11:02:14.881834 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/124dff2e-e721-4aea-b5e7-4fcc5369e86a-dshm\") pod \"auth-test-model-1273de-kserve-84ff499d67-c4jmh\" (UID: \"124dff2e-e721-4aea-b5e7-4fcc5369e86a\") " pod="ai-tenant-e2e-shared-a-1273de/auth-test-model-1273de-kserve-84ff499d67-c4jmh" Jul 07 11:02:14.882009 ip-10-0-135-34 kubenswrapper[2578]: I0707 11:02:14.881869 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/124dff2e-e721-4aea-b5e7-4fcc5369e86a-home\") pod \"auth-test-model-1273de-kserve-84ff499d67-c4jmh\" (UID: \"124dff2e-e721-4aea-b5e7-4fcc5369e86a\") " pod="ai-tenant-e2e-shared-a-1273de/auth-test-model-1273de-kserve-84ff499d67-c4jmh" Jul 07 11:02:14.882009 ip-10-0-135-34 kubenswrapper[2578]: I0707 11:02:14.881897 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/124dff2e-e721-4aea-b5e7-4fcc5369e86a-tmp-dir\") pod \"auth-test-model-1273de-kserve-84ff499d67-c4jmh\" (UID: \"124dff2e-e721-4aea-b5e7-4fcc5369e86a\") " pod="ai-tenant-e2e-shared-a-1273de/auth-test-model-1273de-kserve-84ff499d67-c4jmh" Jul 07 11:02:14.882009 ip-10-0-135-34 kubenswrapper[2578]: I0707 11:02:14.881921 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/124dff2e-e721-4aea-b5e7-4fcc5369e86a-model-cache\") pod \"auth-test-model-1273de-kserve-84ff499d67-c4jmh\" (UID: \"124dff2e-e721-4aea-b5e7-4fcc5369e86a\") " pod="ai-tenant-e2e-shared-a-1273de/auth-test-model-1273de-kserve-84ff499d67-c4jmh" Jul 07 11:02:14.882229 ip-10-0-135-34 kubenswrapper[2578]: I0707 11:02:14.882010 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/124dff2e-e721-4aea-b5e7-4fcc5369e86a-tls-certs\") pod \"auth-test-model-1273de-kserve-84ff499d67-c4jmh\" (UID: \"124dff2e-e721-4aea-b5e7-4fcc5369e86a\") " pod="ai-tenant-e2e-shared-a-1273de/auth-test-model-1273de-kserve-84ff499d67-c4jmh" Jul 07 11:02:14.882390 ip-10-0-135-34 kubenswrapper[2578]: I0707 11:02:14.882356 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/124dff2e-e721-4aea-b5e7-4fcc5369e86a-tmp-dir\") pod \"auth-test-model-1273de-kserve-84ff499d67-c4jmh\" (UID: \"124dff2e-e721-4aea-b5e7-4fcc5369e86a\") " pod="ai-tenant-e2e-shared-a-1273de/auth-test-model-1273de-kserve-84ff499d67-c4jmh" Jul 07 11:02:14.882527 ip-10-0-135-34 kubenswrapper[2578]: I0707 11:02:14.882392 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/124dff2e-e721-4aea-b5e7-4fcc5369e86a-home\") pod \"auth-test-model-1273de-kserve-84ff499d67-c4jmh\" (UID: \"124dff2e-e721-4aea-b5e7-4fcc5369e86a\") " pod="ai-tenant-e2e-shared-a-1273de/auth-test-model-1273de-kserve-84ff499d67-c4jmh" Jul 07 11:02:14.882527 ip-10-0-135-34 kubenswrapper[2578]: I0707 11:02:14.882421 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/124dff2e-e721-4aea-b5e7-4fcc5369e86a-model-cache\") pod \"auth-test-model-1273de-kserve-84ff499d67-c4jmh\" (UID: \"124dff2e-e721-4aea-b5e7-4fcc5369e86a\") " pod="ai-tenant-e2e-shared-a-1273de/auth-test-model-1273de-kserve-84ff499d67-c4jmh" Jul 07 11:02:14.884324 ip-10-0-135-34 kubenswrapper[2578]: I0707 11:02:14.884305 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/124dff2e-e721-4aea-b5e7-4fcc5369e86a-dshm\") pod \"auth-test-model-1273de-kserve-84ff499d67-c4jmh\" (UID: \"124dff2e-e721-4aea-b5e7-4fcc5369e86a\") " pod="ai-tenant-e2e-shared-a-1273de/auth-test-model-1273de-kserve-84ff499d67-c4jmh" Jul 07 11:02:14.884706 ip-10-0-135-34 kubenswrapper[2578]: I0707 11:02:14.884685 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/124dff2e-e721-4aea-b5e7-4fcc5369e86a-tls-certs\") pod \"auth-test-model-1273de-kserve-84ff499d67-c4jmh\" (UID: \"124dff2e-e721-4aea-b5e7-4fcc5369e86a\") " pod="ai-tenant-e2e-shared-a-1273de/auth-test-model-1273de-kserve-84ff499d67-c4jmh" Jul 07 11:02:14.890570 ip-10-0-135-34 kubenswrapper[2578]: I0707 11:02:14.890546 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2z7kv\" (UniqueName: \"kubernetes.io/projected/124dff2e-e721-4aea-b5e7-4fcc5369e86a-kube-api-access-2z7kv\") pod \"auth-test-model-1273de-kserve-84ff499d67-c4jmh\" (UID: \"124dff2e-e721-4aea-b5e7-4fcc5369e86a\") " pod="ai-tenant-e2e-shared-a-1273de/auth-test-model-1273de-kserve-84ff499d67-c4jmh" Jul 07 11:02:14.997006 ip-10-0-135-34 kubenswrapper[2578]: I0707 11:02:14.996907 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-1273de/auth-test-model-1273de-kserve-84ff499d67-c4jmh" Jul 07 11:02:15.136239 ip-10-0-135-34 kubenswrapper[2578]: I0707 11:02:15.136209 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-1273de/auth-test-model-1273de-kserve-84ff499d67-c4jmh"] Jul 07 11:02:15.138360 ip-10-0-135-34 kubenswrapper[2578]: W0707 11:02:15.138338 2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod124dff2e_e721_4aea_b5e7_4fcc5369e86a.slice/crio-396fc7c7425ac2e96e0a85e46049478c0f7caaa4ebb6da8a6a0a0a2023c66e2d WatchSource:0}: Error finding container 396fc7c7425ac2e96e0a85e46049478c0f7caaa4ebb6da8a6a0a0a2023c66e2d: Status 404 returned error can't find the container with id 396fc7c7425ac2e96e0a85e46049478c0f7caaa4ebb6da8a6a0a0a2023c66e2d Jul 07 11:02:16.013608 ip-10-0-135-34 kubenswrapper[2578]: I0707 11:02:16.013574 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-1273de/auth-test-model-1273de-kserve-84ff499d67-c4jmh" event={"ID":"124dff2e-e721-4aea-b5e7-4fcc5369e86a","Type":"ContainerStarted","Data":"e5686ed5c01b76dea8d659b999bbf594d05d982ba7f44f9a1f68bbe3f593a54d"} Jul 07 11:02:16.013608 ip-10-0-135-34 kubenswrapper[2578]: I0707 11:02:16.013614 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-1273de/auth-test-model-1273de-kserve-84ff499d67-c4jmh" event={"ID":"124dff2e-e721-4aea-b5e7-4fcc5369e86a","Type":"ContainerStarted","Data":"396fc7c7425ac2e96e0a85e46049478c0f7caaa4ebb6da8a6a0a0a2023c66e2d"} Jul 07 11:02:16.035215 ip-10-0-135-34 kubenswrapper[2578]: I0707 11:02:16.035160 2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-a-1273de/auth-test-model-1273de-kserve-84ff499d67-c4jmh" podStartSLOduration=2.035144403 podStartE2EDuration="2.035144403s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-07 11:02:14 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 11:02:16.03329035 +0000 UTC m=+2104.705249896" watchObservedRunningTime="2026-07-07 11:02:16.035144403 +0000 UTC m=+2104.707103947" Jul 07 11:02:17.743465 ip-10-0-135-34 kubenswrapper[2578]: I0707 11:02:17.743423 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-lv7nd"] Jul 07 11:02:24.997697 ip-10-0-135-34 kubenswrapper[2578]: I0707 11:02:24.997667 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-a-1273de/auth-test-model-1273de-kserve-84ff499d67-c4jmh" Jul 07 11:02:24.998115 ip-10-0-135-34 kubenswrapper[2578]: I0707 11:02:24.997923 2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-a-1273de/auth-test-model-1273de-kserve-84ff499d67-c4jmh" Jul 07 11:02:25.010721 ip-10-0-135-34 kubenswrapper[2578]: I0707 11:02:25.010695 2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-a-1273de/auth-test-model-1273de-kserve-84ff499d67-c4jmh" Jul 07 11:02:25.068999 ip-10-0-135-34 kubenswrapper[2578]: I0707 11:02:25.068975 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-a-1273de/auth-test-model-1273de-kserve-84ff499d67-c4jmh" Jul 07 11:02:35.464760 ip-10-0-135-34 kubenswrapper[2578]: I0707 11:02:35.464725 2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-b-2a3633/auth-test-model-2a3633-kserve-58564d7979-58nsd"] Jul 07 11:02:35.469186 ip-10-0-135-34 kubenswrapper[2578]: I0707 11:02:35.469163 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-2a3633/auth-test-model-2a3633-kserve-58564d7979-58nsd" Jul 07 11:02:35.472406 ip-10-0-135-34 kubenswrapper[2578]: I0707 11:02:35.472375 2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-2a3633\"/\"openshift-service-ca.crt\"" Jul 07 11:02:35.473402 ip-10-0-135-34 kubenswrapper[2578]: I0707 11:02:35.473380 2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-2a3633\"/\"kube-root-ca.crt\"" Jul 07 11:02:35.473564 ip-10-0-135-34 kubenswrapper[2578]: I0707 11:02:35.473509 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-2a3633\"/\"default-dockercfg-nq96c\"" Jul 07 11:02:35.473665 ip-10-0-135-34 kubenswrapper[2578]: I0707 11:02:35.473572 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-2a3633\"/\"auth-test-model-2a3633-kserve-self-signed-certs\"" Jul 07 11:02:35.479220 ip-10-0-135-34 kubenswrapper[2578]: I0707 11:02:35.479196 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-2a3633/auth-test-model-2a3633-kserve-58564d7979-58nsd"] Jul 07 11:02:35.555104 ip-10-0-135-34 kubenswrapper[2578]: I0707 11:02:35.555051 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/66a90be4-108f-44d8-9b32-ec86003d6d35-model-cache\") pod \"auth-test-model-2a3633-kserve-58564d7979-58nsd\" (UID: \"66a90be4-108f-44d8-9b32-ec86003d6d35\") " pod="ai-tenant-e2e-shared-b-2a3633/auth-test-model-2a3633-kserve-58564d7979-58nsd" Jul 07 11:02:35.555305 ip-10-0-135-34 kubenswrapper[2578]: I0707 11:02:35.555143 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/66a90be4-108f-44d8-9b32-ec86003d6d35-tls-certs\") pod \"auth-test-model-2a3633-kserve-58564d7979-58nsd\" (UID: \"66a90be4-108f-44d8-9b32-ec86003d6d35\") " pod="ai-tenant-e2e-shared-b-2a3633/auth-test-model-2a3633-kserve-58564d7979-58nsd" Jul 07 11:02:35.555305 ip-10-0-135-34 kubenswrapper[2578]: I0707 11:02:35.555175 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/66a90be4-108f-44d8-9b32-ec86003d6d35-tmp-dir\") pod \"auth-test-model-2a3633-kserve-58564d7979-58nsd\" (UID: \"66a90be4-108f-44d8-9b32-ec86003d6d35\") " pod="ai-tenant-e2e-shared-b-2a3633/auth-test-model-2a3633-kserve-58564d7979-58nsd" Jul 07 11:02:35.555305 ip-10-0-135-34 kubenswrapper[2578]: I0707 11:02:35.555206 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ww9w2\" (UniqueName: \"kubernetes.io/projected/66a90be4-108f-44d8-9b32-ec86003d6d35-kube-api-access-ww9w2\") pod \"auth-test-model-2a3633-kserve-58564d7979-58nsd\" (UID: \"66a90be4-108f-44d8-9b32-ec86003d6d35\") " pod="ai-tenant-e2e-shared-b-2a3633/auth-test-model-2a3633-kserve-58564d7979-58nsd" Jul 07 11:02:35.555305 ip-10-0-135-34 kubenswrapper[2578]: I0707 11:02:35.555229 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/66a90be4-108f-44d8-9b32-ec86003d6d35-dshm\") pod \"auth-test-model-2a3633-kserve-58564d7979-58nsd\" (UID: \"66a90be4-108f-44d8-9b32-ec86003d6d35\") " pod="ai-tenant-e2e-shared-b-2a3633/auth-test-model-2a3633-kserve-58564d7979-58nsd" Jul 07 11:02:35.555305 ip-10-0-135-34 kubenswrapper[2578]: I0707 11:02:35.555247 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/66a90be4-108f-44d8-9b32-ec86003d6d35-home\") pod \"auth-test-model-2a3633-kserve-58564d7979-58nsd\" (UID: \"66a90be4-108f-44d8-9b32-ec86003d6d35\") " pod="ai-tenant-e2e-shared-b-2a3633/auth-test-model-2a3633-kserve-58564d7979-58nsd" Jul 07 11:02:35.655689 ip-10-0-135-34 kubenswrapper[2578]: I0707 11:02:35.655649 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/66a90be4-108f-44d8-9b32-ec86003d6d35-tls-certs\") pod \"auth-test-model-2a3633-kserve-58564d7979-58nsd\" (UID: \"66a90be4-108f-44d8-9b32-ec86003d6d35\") " pod="ai-tenant-e2e-shared-b-2a3633/auth-test-model-2a3633-kserve-58564d7979-58nsd" Jul 07 11:02:35.655895 ip-10-0-135-34 kubenswrapper[2578]: I0707 11:02:35.655694 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/66a90be4-108f-44d8-9b32-ec86003d6d35-tmp-dir\") pod \"auth-test-model-2a3633-kserve-58564d7979-58nsd\" (UID: \"66a90be4-108f-44d8-9b32-ec86003d6d35\") " pod="ai-tenant-e2e-shared-b-2a3633/auth-test-model-2a3633-kserve-58564d7979-58nsd" Jul 07 11:02:35.655895 ip-10-0-135-34 kubenswrapper[2578]: I0707 11:02:35.655737 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ww9w2\" (UniqueName: \"kubernetes.io/projected/66a90be4-108f-44d8-9b32-ec86003d6d35-kube-api-access-ww9w2\") pod \"auth-test-model-2a3633-kserve-58564d7979-58nsd\" (UID: \"66a90be4-108f-44d8-9b32-ec86003d6d35\") " pod="ai-tenant-e2e-shared-b-2a3633/auth-test-model-2a3633-kserve-58564d7979-58nsd" Jul 07 11:02:35.655895 ip-10-0-135-34 kubenswrapper[2578]: I0707 11:02:35.655764 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/66a90be4-108f-44d8-9b32-ec86003d6d35-dshm\") pod \"auth-test-model-2a3633-kserve-58564d7979-58nsd\" (UID: \"66a90be4-108f-44d8-9b32-ec86003d6d35\") " pod="ai-tenant-e2e-shared-b-2a3633/auth-test-model-2a3633-kserve-58564d7979-58nsd" Jul 07 11:02:35.656079 ip-10-0-135-34 kubenswrapper[2578]: I0707 11:02:35.655907 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/66a90be4-108f-44d8-9b32-ec86003d6d35-home\") pod \"auth-test-model-2a3633-kserve-58564d7979-58nsd\" (UID: \"66a90be4-108f-44d8-9b32-ec86003d6d35\") " pod="ai-tenant-e2e-shared-b-2a3633/auth-test-model-2a3633-kserve-58564d7979-58nsd" Jul 07 11:02:35.656079 ip-10-0-135-34 kubenswrapper[2578]: I0707 11:02:35.655974 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/66a90be4-108f-44d8-9b32-ec86003d6d35-model-cache\") pod \"auth-test-model-2a3633-kserve-58564d7979-58nsd\" (UID: \"66a90be4-108f-44d8-9b32-ec86003d6d35\") " pod="ai-tenant-e2e-shared-b-2a3633/auth-test-model-2a3633-kserve-58564d7979-58nsd" Jul 07 11:02:35.656197 ip-10-0-135-34 kubenswrapper[2578]: I0707 11:02:35.656133 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/66a90be4-108f-44d8-9b32-ec86003d6d35-tmp-dir\") pod \"auth-test-model-2a3633-kserve-58564d7979-58nsd\" (UID: \"66a90be4-108f-44d8-9b32-ec86003d6d35\") " pod="ai-tenant-e2e-shared-b-2a3633/auth-test-model-2a3633-kserve-58564d7979-58nsd" Jul 07 11:02:35.656239 ip-10-0-135-34 kubenswrapper[2578]: I0707 11:02:35.656213 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/66a90be4-108f-44d8-9b32-ec86003d6d35-home\") pod \"auth-test-model-2a3633-kserve-58564d7979-58nsd\" (UID: \"66a90be4-108f-44d8-9b32-ec86003d6d35\") " pod="ai-tenant-e2e-shared-b-2a3633/auth-test-model-2a3633-kserve-58564d7979-58nsd" Jul 07 11:02:35.656299 ip-10-0-135-34 kubenswrapper[2578]: I0707 11:02:35.656279 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/66a90be4-108f-44d8-9b32-ec86003d6d35-model-cache\") pod \"auth-test-model-2a3633-kserve-58564d7979-58nsd\" (UID: \"66a90be4-108f-44d8-9b32-ec86003d6d35\") " pod="ai-tenant-e2e-shared-b-2a3633/auth-test-model-2a3633-kserve-58564d7979-58nsd" Jul 07 11:02:35.658135 ip-10-0-135-34 kubenswrapper[2578]: I0707 11:02:35.658115 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/66a90be4-108f-44d8-9b32-ec86003d6d35-dshm\") pod \"auth-test-model-2a3633-kserve-58564d7979-58nsd\" (UID: \"66a90be4-108f-44d8-9b32-ec86003d6d35\") " pod="ai-tenant-e2e-shared-b-2a3633/auth-test-model-2a3633-kserve-58564d7979-58nsd" Jul 07 11:02:35.658312 ip-10-0-135-34 kubenswrapper[2578]: I0707 11:02:35.658294 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/66a90be4-108f-44d8-9b32-ec86003d6d35-tls-certs\") pod \"auth-test-model-2a3633-kserve-58564d7979-58nsd\" (UID: \"66a90be4-108f-44d8-9b32-ec86003d6d35\") " pod="ai-tenant-e2e-shared-b-2a3633/auth-test-model-2a3633-kserve-58564d7979-58nsd" Jul 07 11:02:35.665525 ip-10-0-135-34 kubenswrapper[2578]: I0707 11:02:35.665489 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ww9w2\" (UniqueName: \"kubernetes.io/projected/66a90be4-108f-44d8-9b32-ec86003d6d35-kube-api-access-ww9w2\") pod \"auth-test-model-2a3633-kserve-58564d7979-58nsd\" (UID: \"66a90be4-108f-44d8-9b32-ec86003d6d35\") " pod="ai-tenant-e2e-shared-b-2a3633/auth-test-model-2a3633-kserve-58564d7979-58nsd" Jul 07 11:02:35.780261 ip-10-0-135-34 kubenswrapper[2578]: I0707 11:02:35.780153 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-2a3633/auth-test-model-2a3633-kserve-58564d7979-58nsd" Jul 07 11:02:35.923575 ip-10-0-135-34 kubenswrapper[2578]: I0707 11:02:35.923421 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-2a3633/auth-test-model-2a3633-kserve-58564d7979-58nsd"] Jul 07 11:02:35.925819 ip-10-0-135-34 kubenswrapper[2578]: W0707 11:02:35.925787 2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod66a90be4_108f_44d8_9b32_ec86003d6d35.slice/crio-089700c1031066868a0f23d8bbf8691c212c124691283235bd5399fe0d207dc7 WatchSource:0}: Error finding container 089700c1031066868a0f23d8bbf8691c212c124691283235bd5399fe0d207dc7: Status 404 returned error can't find the container with id 089700c1031066868a0f23d8bbf8691c212c124691283235bd5399fe0d207dc7 Jul 07 11:02:36.112411 ip-10-0-135-34 kubenswrapper[2578]: I0707 11:02:36.112376 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-2a3633/auth-test-model-2a3633-kserve-58564d7979-58nsd" event={"ID":"66a90be4-108f-44d8-9b32-ec86003d6d35","Type":"ContainerStarted","Data":"3acb6777d61d4dcccddd76261e94b895728a6baab7d205e42a5b109ad681b178"} Jul 07 11:02:36.112585 ip-10-0-135-34 kubenswrapper[2578]: I0707 11:02:36.112416 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-2a3633/auth-test-model-2a3633-kserve-58564d7979-58nsd" event={"ID":"66a90be4-108f-44d8-9b32-ec86003d6d35","Type":"ContainerStarted","Data":"089700c1031066868a0f23d8bbf8691c212c124691283235bd5399fe0d207dc7"} Jul 07 11:02:36.133718 ip-10-0-135-34 kubenswrapper[2578]: I0707 11:02:36.133661 2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-b-2a3633/auth-test-model-2a3633-kserve-58564d7979-58nsd" podStartSLOduration=1.133645799 podStartE2EDuration="1.133645799s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-07 11:02:35 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 11:02:36.131508621 +0000 UTC m=+2124.803468165" watchObservedRunningTime="2026-07-07 11:02:36.133645799 +0000 UTC m=+2124.805605352" Jul 07 11:02:38.832172 ip-10-0-135-34 kubenswrapper[2578]: I0707 11:02:38.832141 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-lv7nd"] Jul 07 11:02:45.780752 ip-10-0-135-34 kubenswrapper[2578]: I0707 11:02:45.780708 2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-b-2a3633/auth-test-model-2a3633-kserve-58564d7979-58nsd" Jul 07 11:02:45.781188 ip-10-0-135-34 kubenswrapper[2578]: I0707 11:02:45.780800 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-b-2a3633/auth-test-model-2a3633-kserve-58564d7979-58nsd" Jul 07 11:02:45.793264 ip-10-0-135-34 kubenswrapper[2578]: I0707 11:02:45.793239 2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-b-2a3633/auth-test-model-2a3633-kserve-58564d7979-58nsd" Jul 07 11:02:46.175034 ip-10-0-135-34 kubenswrapper[2578]: I0707 11:02:46.175003 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-b-2a3633/auth-test-model-2a3633-kserve-58564d7979-58nsd" Jul 07 11:02:52.121837 ip-10-0-135-34 kubenswrapper[2578]: I0707 11:02:52.121800 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-lv7nd"] Jul 07 11:03:08.827417 ip-10-0-135-34 kubenswrapper[2578]: I0707 11:03:08.827337 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-lv7nd"] Jul 07 11:03:22.435825 ip-10-0-135-34 kubenswrapper[2578]: I0707 11:03:22.435793 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-lv7nd"] Jul 07 11:03:36.549396 ip-10-0-135-34 kubenswrapper[2578]: I0707 11:03:36.549350 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-lv7nd"] Jul 07 11:03:52.929542 ip-10-0-135-34 kubenswrapper[2578]: I0707 11:03:52.929503 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-lv7nd"] Jul 07 11:04:10.052287 ip-10-0-135-34 kubenswrapper[2578]: I0707 11:04:10.052247 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-lv7nd"] Jul 07 11:04:27.373453 ip-10-0-135-34 kubenswrapper[2578]: I0707 11:04:27.373396 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-lv7nd"] Jul 07 11:04:31.709110 ip-10-0-135-34 kubenswrapper[2578]: I0707 11:04:31.709074 2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-1273de/auth-test-model-1273de-kserve-84ff499d67-c4jmh"] Jul 07 11:04:32.599168 ip-10-0-135-34 kubenswrapper[2578]: I0707 11:04:32.599132 2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-2a3633/auth-test-model-2a3633-kserve-58564d7979-58nsd"] Jul 07 11:04:43.271844 ip-10-0-135-34 kubenswrapper[2578]: I0707 11:04:43.271800 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-lv7nd"] Jul 07 11:04:46.729451 ip-10-0-135-34 kubenswrapper[2578]: I0707 11:04:46.729408 2578 kuberuntime_container.go:872] "Killing container with a grace period" pod="ai-tenant-e2e-shared-a-1273de/auth-test-model-1273de-kserve-84ff499d67-c4jmh" podUID="124dff2e-e721-4aea-b5e7-4fcc5369e86a" containerName="main" containerID="cri-o://e5686ed5c01b76dea8d659b999bbf594d05d982ba7f44f9a1f68bbe3f593a54d" gracePeriod=45 Jul 07 11:04:46.982744 ip-10-0-135-34 kubenswrapper[2578]: I0707 11:04:46.982678 2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-1273de/auth-test-model-1273de-kserve-84ff499d67-c4jmh" Jul 07 11:04:47.132988 ip-10-0-135-34 kubenswrapper[2578]: I0707 11:04:47.132951 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/124dff2e-e721-4aea-b5e7-4fcc5369e86a-home\") pod \"124dff2e-e721-4aea-b5e7-4fcc5369e86a\" (UID: \"124dff2e-e721-4aea-b5e7-4fcc5369e86a\") " Jul 07 11:04:47.132988 ip-10-0-135-34 kubenswrapper[2578]: I0707 11:04:47.132993 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-2z7kv\" (UniqueName: \"kubernetes.io/projected/124dff2e-e721-4aea-b5e7-4fcc5369e86a-kube-api-access-2z7kv\") pod \"124dff2e-e721-4aea-b5e7-4fcc5369e86a\" (UID: \"124dff2e-e721-4aea-b5e7-4fcc5369e86a\") " Jul 07 11:04:47.133234 ip-10-0-135-34 kubenswrapper[2578]: I0707 11:04:47.133019 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/124dff2e-e721-4aea-b5e7-4fcc5369e86a-model-cache\") pod \"124dff2e-e721-4aea-b5e7-4fcc5369e86a\" (UID: \"124dff2e-e721-4aea-b5e7-4fcc5369e86a\") " Jul 07 11:04:47.133234 ip-10-0-135-34 kubenswrapper[2578]: I0707 11:04:47.133064 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/124dff2e-e721-4aea-b5e7-4fcc5369e86a-tmp-dir\") pod \"124dff2e-e721-4aea-b5e7-4fcc5369e86a\" (UID: \"124dff2e-e721-4aea-b5e7-4fcc5369e86a\") " Jul 07 11:04:47.133234 ip-10-0-135-34 kubenswrapper[2578]: I0707 11:04:47.133156 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/124dff2e-e721-4aea-b5e7-4fcc5369e86a-tls-certs\") pod \"124dff2e-e721-4aea-b5e7-4fcc5369e86a\" (UID: \"124dff2e-e721-4aea-b5e7-4fcc5369e86a\") " Jul 07 11:04:47.133234 ip-10-0-135-34 kubenswrapper[2578]: I0707 11:04:47.133192 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/124dff2e-e721-4aea-b5e7-4fcc5369e86a-dshm\") pod \"124dff2e-e721-4aea-b5e7-4fcc5369e86a\" (UID: \"124dff2e-e721-4aea-b5e7-4fcc5369e86a\") " Jul 07 11:04:47.133462 ip-10-0-135-34 kubenswrapper[2578]: I0707 11:04:47.133352 2578 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/124dff2e-e721-4aea-b5e7-4fcc5369e86a-home" (OuterVolumeSpecName: "home") pod "124dff2e-e721-4aea-b5e7-4fcc5369e86a" (UID: "124dff2e-e721-4aea-b5e7-4fcc5369e86a"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 07 11:04:47.133462 ip-10-0-135-34 kubenswrapper[2578]: I0707 11:04:47.133351 2578 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/124dff2e-e721-4aea-b5e7-4fcc5369e86a-model-cache" (OuterVolumeSpecName: "model-cache") pod "124dff2e-e721-4aea-b5e7-4fcc5369e86a" (UID: "124dff2e-e721-4aea-b5e7-4fcc5369e86a"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 07 11:04:47.133462 ip-10-0-135-34 kubenswrapper[2578]: I0707 11:04:47.133412 2578 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/124dff2e-e721-4aea-b5e7-4fcc5369e86a-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "124dff2e-e721-4aea-b5e7-4fcc5369e86a" (UID: "124dff2e-e721-4aea-b5e7-4fcc5369e86a"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 07 11:04:47.133621 ip-10-0-135-34 kubenswrapper[2578]: I0707 11:04:47.133554 2578 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/124dff2e-e721-4aea-b5e7-4fcc5369e86a-home\") on node \"ip-10-0-135-34.ec2.internal\" DevicePath \"\"" Jul 07 11:04:47.133621 ip-10-0-135-34 kubenswrapper[2578]: I0707 11:04:47.133570 2578 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/124dff2e-e721-4aea-b5e7-4fcc5369e86a-model-cache\") on node \"ip-10-0-135-34.ec2.internal\" DevicePath \"\"" Jul 07 11:04:47.133621 ip-10-0-135-34 kubenswrapper[2578]: I0707 11:04:47.133584 2578 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/124dff2e-e721-4aea-b5e7-4fcc5369e86a-tmp-dir\") on node \"ip-10-0-135-34.ec2.internal\" DevicePath \"\"" Jul 07 11:04:47.135630 ip-10-0-135-34 kubenswrapper[2578]: I0707 11:04:47.135604 2578 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/124dff2e-e721-4aea-b5e7-4fcc5369e86a-dshm" (OuterVolumeSpecName: "dshm") pod "124dff2e-e721-4aea-b5e7-4fcc5369e86a" (UID: "124dff2e-e721-4aea-b5e7-4fcc5369e86a"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 07 11:04:47.135717 ip-10-0-135-34 kubenswrapper[2578]: I0707 11:04:47.135660 2578 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/124dff2e-e721-4aea-b5e7-4fcc5369e86a-kube-api-access-2z7kv" (OuterVolumeSpecName: "kube-api-access-2z7kv") pod "124dff2e-e721-4aea-b5e7-4fcc5369e86a" (UID: "124dff2e-e721-4aea-b5e7-4fcc5369e86a"). InnerVolumeSpecName "kube-api-access-2z7kv". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 07 11:04:47.135717 ip-10-0-135-34 kubenswrapper[2578]: I0707 11:04:47.135688 2578 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/124dff2e-e721-4aea-b5e7-4fcc5369e86a-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "124dff2e-e721-4aea-b5e7-4fcc5369e86a" (UID: "124dff2e-e721-4aea-b5e7-4fcc5369e86a"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 07 11:04:47.234114 ip-10-0-135-34 kubenswrapper[2578]: I0707 11:04:47.234016 2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2z7kv\" (UniqueName: \"kubernetes.io/projected/124dff2e-e721-4aea-b5e7-4fcc5369e86a-kube-api-access-2z7kv\") on node \"ip-10-0-135-34.ec2.internal\" DevicePath \"\"" Jul 07 11:04:47.234114 ip-10-0-135-34 kubenswrapper[2578]: I0707 11:04:47.234058 2578 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/124dff2e-e721-4aea-b5e7-4fcc5369e86a-tls-certs\") on node \"ip-10-0-135-34.ec2.internal\" DevicePath \"\"" Jul 07 11:04:47.234114 ip-10-0-135-34 kubenswrapper[2578]: I0707 11:04:47.234069 2578 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/124dff2e-e721-4aea-b5e7-4fcc5369e86a-dshm\") on node \"ip-10-0-135-34.ec2.internal\" DevicePath \"\"" Jul 07 11:04:47.619734 ip-10-0-135-34 kubenswrapper[2578]: I0707 11:04:47.619661 2578 kuberuntime_container.go:872] "Killing container with a grace period" pod="ai-tenant-e2e-shared-b-2a3633/auth-test-model-2a3633-kserve-58564d7979-58nsd" podUID="66a90be4-108f-44d8-9b32-ec86003d6d35" containerName="main" containerID="cri-o://3acb6777d61d4dcccddd76261e94b895728a6baab7d205e42a5b109ad681b178" gracePeriod=45 Jul 07 11:04:47.790020 ip-10-0-135-34 kubenswrapper[2578]: I0707 11:04:47.789989 2578 generic.go:356] "Generic (PLEG): container finished" podID="66a90be4-108f-44d8-9b32-ec86003d6d35" containerID="3acb6777d61d4dcccddd76261e94b895728a6baab7d205e42a5b109ad681b178" exitCode=0 Jul 07 11:04:47.790409 ip-10-0-135-34 kubenswrapper[2578]: I0707 11:04:47.790058 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-2a3633/auth-test-model-2a3633-kserve-58564d7979-58nsd" event={"ID":"66a90be4-108f-44d8-9b32-ec86003d6d35","Type":"ContainerDied","Data":"3acb6777d61d4dcccddd76261e94b895728a6baab7d205e42a5b109ad681b178"} Jul 07 11:04:47.791509 ip-10-0-135-34 kubenswrapper[2578]: I0707 11:04:47.791485 2578 generic.go:356] "Generic (PLEG): container finished" podID="124dff2e-e721-4aea-b5e7-4fcc5369e86a" containerID="e5686ed5c01b76dea8d659b999bbf594d05d982ba7f44f9a1f68bbe3f593a54d" exitCode=0 Jul 07 11:04:47.791642 ip-10-0-135-34 kubenswrapper[2578]: I0707 11:04:47.791542 2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-1273de/auth-test-model-1273de-kserve-84ff499d67-c4jmh" Jul 07 11:04:47.791642 ip-10-0-135-34 kubenswrapper[2578]: I0707 11:04:47.791555 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-1273de/auth-test-model-1273de-kserve-84ff499d67-c4jmh" event={"ID":"124dff2e-e721-4aea-b5e7-4fcc5369e86a","Type":"ContainerDied","Data":"e5686ed5c01b76dea8d659b999bbf594d05d982ba7f44f9a1f68bbe3f593a54d"} Jul 07 11:04:47.791642 ip-10-0-135-34 kubenswrapper[2578]: I0707 11:04:47.791585 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-1273de/auth-test-model-1273de-kserve-84ff499d67-c4jmh" event={"ID":"124dff2e-e721-4aea-b5e7-4fcc5369e86a","Type":"ContainerDied","Data":"396fc7c7425ac2e96e0a85e46049478c0f7caaa4ebb6da8a6a0a0a2023c66e2d"} Jul 07 11:04:47.791642 ip-10-0-135-34 kubenswrapper[2578]: I0707 11:04:47.791603 2578 scope.go:117] "RemoveContainer" containerID="e5686ed5c01b76dea8d659b999bbf594d05d982ba7f44f9a1f68bbe3f593a54d" Jul 07 11:04:47.802746 ip-10-0-135-34 kubenswrapper[2578]: I0707 11:04:47.802360 2578 scope.go:117] "RemoveContainer" containerID="e5686ed5c01b76dea8d659b999bbf594d05d982ba7f44f9a1f68bbe3f593a54d" Jul 07 11:04:47.802746 ip-10-0-135-34 kubenswrapper[2578]: E0707 11:04:47.802697 2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e5686ed5c01b76dea8d659b999bbf594d05d982ba7f44f9a1f68bbe3f593a54d\": container with ID starting with e5686ed5c01b76dea8d659b999bbf594d05d982ba7f44f9a1f68bbe3f593a54d not found: ID does not exist" containerID="e5686ed5c01b76dea8d659b999bbf594d05d982ba7f44f9a1f68bbe3f593a54d" Jul 07 11:04:47.802746 ip-10-0-135-34 kubenswrapper[2578]: I0707 11:04:47.802728 2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e5686ed5c01b76dea8d659b999bbf594d05d982ba7f44f9a1f68bbe3f593a54d"} err="failed to get container status \"e5686ed5c01b76dea8d659b999bbf594d05d982ba7f44f9a1f68bbe3f593a54d\": rpc error: code = NotFound desc = could not find container \"e5686ed5c01b76dea8d659b999bbf594d05d982ba7f44f9a1f68bbe3f593a54d\": container with ID starting with e5686ed5c01b76dea8d659b999bbf594d05d982ba7f44f9a1f68bbe3f593a54d not found: ID does not exist" Jul 07 11:04:47.835826 ip-10-0-135-34 kubenswrapper[2578]: I0707 11:04:47.835794 2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-1273de/auth-test-model-1273de-kserve-84ff499d67-c4jmh"] Jul 07 11:04:47.857525 ip-10-0-135-34 kubenswrapper[2578]: I0707 11:04:47.857502 2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-a-1273de/auth-test-model-1273de-kserve-84ff499d67-c4jmh"] Jul 07 11:04:47.871821 ip-10-0-135-34 kubenswrapper[2578]: I0707 11:04:47.871730 2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-2a3633/auth-test-model-2a3633-kserve-58564d7979-58nsd" Jul 07 11:04:48.042363 ip-10-0-135-34 kubenswrapper[2578]: I0707 11:04:48.042330 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/66a90be4-108f-44d8-9b32-ec86003d6d35-dshm\") pod \"66a90be4-108f-44d8-9b32-ec86003d6d35\" (UID: \"66a90be4-108f-44d8-9b32-ec86003d6d35\") " Jul 07 11:04:48.042363 ip-10-0-135-34 kubenswrapper[2578]: I0707 11:04:48.042367 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-ww9w2\" (UniqueName: \"kubernetes.io/projected/66a90be4-108f-44d8-9b32-ec86003d6d35-kube-api-access-ww9w2\") pod \"66a90be4-108f-44d8-9b32-ec86003d6d35\" (UID: \"66a90be4-108f-44d8-9b32-ec86003d6d35\") " Jul 07 11:04:48.042613 ip-10-0-135-34 kubenswrapper[2578]: I0707 11:04:48.042410 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/66a90be4-108f-44d8-9b32-ec86003d6d35-tls-certs\") pod \"66a90be4-108f-44d8-9b32-ec86003d6d35\" (UID: \"66a90be4-108f-44d8-9b32-ec86003d6d35\") " Jul 07 11:04:48.042613 ip-10-0-135-34 kubenswrapper[2578]: I0707 11:04:48.042439 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/66a90be4-108f-44d8-9b32-ec86003d6d35-model-cache\") pod \"66a90be4-108f-44d8-9b32-ec86003d6d35\" (UID: \"66a90be4-108f-44d8-9b32-ec86003d6d35\") " Jul 07 11:04:48.042613 ip-10-0-135-34 kubenswrapper[2578]: I0707 11:04:48.042479 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/66a90be4-108f-44d8-9b32-ec86003d6d35-home\") pod \"66a90be4-108f-44d8-9b32-ec86003d6d35\" (UID: \"66a90be4-108f-44d8-9b32-ec86003d6d35\") " Jul 07 11:04:48.042613 ip-10-0-135-34 kubenswrapper[2578]: I0707 11:04:48.042519 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/66a90be4-108f-44d8-9b32-ec86003d6d35-tmp-dir\") pod \"66a90be4-108f-44d8-9b32-ec86003d6d35\" (UID: \"66a90be4-108f-44d8-9b32-ec86003d6d35\") " Jul 07 11:04:48.042876 ip-10-0-135-34 kubenswrapper[2578]: I0707 11:04:48.042803 2578 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/66a90be4-108f-44d8-9b32-ec86003d6d35-model-cache" (OuterVolumeSpecName: "model-cache") pod "66a90be4-108f-44d8-9b32-ec86003d6d35" (UID: "66a90be4-108f-44d8-9b32-ec86003d6d35"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 07 11:04:48.042876 ip-10-0-135-34 kubenswrapper[2578]: I0707 11:04:48.042827 2578 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/66a90be4-108f-44d8-9b32-ec86003d6d35-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "66a90be4-108f-44d8-9b32-ec86003d6d35" (UID: "66a90be4-108f-44d8-9b32-ec86003d6d35"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 07 11:04:48.042876 ip-10-0-135-34 kubenswrapper[2578]: I0707 11:04:48.042812 2578 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/66a90be4-108f-44d8-9b32-ec86003d6d35-home" (OuterVolumeSpecName: "home") pod "66a90be4-108f-44d8-9b32-ec86003d6d35" (UID: "66a90be4-108f-44d8-9b32-ec86003d6d35"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 07 11:04:48.043029 ip-10-0-135-34 kubenswrapper[2578]: I0707 11:04:48.042918 2578 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/66a90be4-108f-44d8-9b32-ec86003d6d35-tmp-dir\") on node \"ip-10-0-135-34.ec2.internal\" DevicePath \"\"" Jul 07 11:04:48.043029 ip-10-0-135-34 kubenswrapper[2578]: I0707 11:04:48.042931 2578 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/66a90be4-108f-44d8-9b32-ec86003d6d35-model-cache\") on node \"ip-10-0-135-34.ec2.internal\" DevicePath \"\"" Jul 07 11:04:48.043029 ip-10-0-135-34 kubenswrapper[2578]: I0707 11:04:48.042941 2578 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/66a90be4-108f-44d8-9b32-ec86003d6d35-home\") on node \"ip-10-0-135-34.ec2.internal\" DevicePath \"\"" Jul 07 11:04:48.044864 ip-10-0-135-34 kubenswrapper[2578]: I0707 11:04:48.044838 2578 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/66a90be4-108f-44d8-9b32-ec86003d6d35-dshm" (OuterVolumeSpecName: "dshm") pod "66a90be4-108f-44d8-9b32-ec86003d6d35" (UID: "66a90be4-108f-44d8-9b32-ec86003d6d35"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 07 11:04:48.044864 ip-10-0-135-34 kubenswrapper[2578]: I0707 11:04:48.044857 2578 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/66a90be4-108f-44d8-9b32-ec86003d6d35-kube-api-access-ww9w2" (OuterVolumeSpecName: "kube-api-access-ww9w2") pod "66a90be4-108f-44d8-9b32-ec86003d6d35" (UID: "66a90be4-108f-44d8-9b32-ec86003d6d35"). InnerVolumeSpecName "kube-api-access-ww9w2". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 07 11:04:48.045006 ip-10-0-135-34 kubenswrapper[2578]: I0707 11:04:48.044908 2578 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/66a90be4-108f-44d8-9b32-ec86003d6d35-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "66a90be4-108f-44d8-9b32-ec86003d6d35" (UID: "66a90be4-108f-44d8-9b32-ec86003d6d35"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 07 11:04:48.144010 ip-10-0-135-34 kubenswrapper[2578]: I0707 11:04:48.143921 2578 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/66a90be4-108f-44d8-9b32-ec86003d6d35-dshm\") on node \"ip-10-0-135-34.ec2.internal\" DevicePath \"\"" Jul 07 11:04:48.144010 ip-10-0-135-34 kubenswrapper[2578]: I0707 11:04:48.143949 2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ww9w2\" (UniqueName: \"kubernetes.io/projected/66a90be4-108f-44d8-9b32-ec86003d6d35-kube-api-access-ww9w2\") on node \"ip-10-0-135-34.ec2.internal\" DevicePath \"\"" Jul 07 11:04:48.144010 ip-10-0-135-34 kubenswrapper[2578]: I0707 11:04:48.143958 2578 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/66a90be4-108f-44d8-9b32-ec86003d6d35-tls-certs\") on node \"ip-10-0-135-34.ec2.internal\" DevicePath \"\"" Jul 07 11:04:48.797787 ip-10-0-135-34 kubenswrapper[2578]: I0707 11:04:48.797749 2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-2a3633/auth-test-model-2a3633-kserve-58564d7979-58nsd" Jul 07 11:04:48.798264 ip-10-0-135-34 kubenswrapper[2578]: I0707 11:04:48.797797 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-2a3633/auth-test-model-2a3633-kserve-58564d7979-58nsd" event={"ID":"66a90be4-108f-44d8-9b32-ec86003d6d35","Type":"ContainerDied","Data":"089700c1031066868a0f23d8bbf8691c212c124691283235bd5399fe0d207dc7"} Jul 07 11:04:48.798264 ip-10-0-135-34 kubenswrapper[2578]: I0707 11:04:48.797851 2578 scope.go:117] "RemoveContainer" containerID="3acb6777d61d4dcccddd76261e94b895728a6baab7d205e42a5b109ad681b178" Jul 07 11:04:48.830466 ip-10-0-135-34 kubenswrapper[2578]: I0707 11:04:48.830432 2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-2a3633/auth-test-model-2a3633-kserve-58564d7979-58nsd"] Jul 07 11:04:48.838480 ip-10-0-135-34 kubenswrapper[2578]: I0707 11:04:48.838452 2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-b-2a3633/auth-test-model-2a3633-kserve-58564d7979-58nsd"] Jul 07 11:04:49.821709 ip-10-0-135-34 kubenswrapper[2578]: I0707 11:04:49.821667 2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="124dff2e-e721-4aea-b5e7-4fcc5369e86a" path="/var/lib/kubelet/pods/124dff2e-e721-4aea-b5e7-4fcc5369e86a/volumes" Jul 07 11:04:49.822119 ip-10-0-135-34 kubenswrapper[2578]: I0707 11:04:49.822080 2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="66a90be4-108f-44d8-9b32-ec86003d6d35" path="/var/lib/kubelet/pods/66a90be4-108f-44d8-9b32-ec86003d6d35/volumes" Jul 07 11:04:51.987394 ip-10-0-135-34 kubenswrapper[2578]: I0707 11:04:51.987356 2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-a-1273de/sub-test-model-1273de-kserve-687cd89fbc-fd9tf"] Jul 07 11:04:51.987861 ip-10-0-135-34 kubenswrapper[2578]: I0707 11:04:51.987750 2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="124dff2e-e721-4aea-b5e7-4fcc5369e86a" containerName="main" Jul 07 11:04:51.987861 ip-10-0-135-34 kubenswrapper[2578]: I0707 11:04:51.987762 2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="124dff2e-e721-4aea-b5e7-4fcc5369e86a" containerName="main" Jul 07 11:04:51.987861 ip-10-0-135-34 kubenswrapper[2578]: I0707 11:04:51.987796 2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="66a90be4-108f-44d8-9b32-ec86003d6d35" containerName="main" Jul 07 11:04:51.987861 ip-10-0-135-34 kubenswrapper[2578]: I0707 11:04:51.987804 2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="66a90be4-108f-44d8-9b32-ec86003d6d35" containerName="main" Jul 07 11:04:51.988001 ip-10-0-135-34 kubenswrapper[2578]: I0707 11:04:51.987869 2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="124dff2e-e721-4aea-b5e7-4fcc5369e86a" containerName="main" Jul 07 11:04:51.988001 ip-10-0-135-34 kubenswrapper[2578]: I0707 11:04:51.987879 2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="66a90be4-108f-44d8-9b32-ec86003d6d35" containerName="main" Jul 07 11:04:51.992836 ip-10-0-135-34 kubenswrapper[2578]: I0707 11:04:51.992808 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-1273de/sub-test-model-1273de-kserve-687cd89fbc-fd9tf" Jul 07 11:04:51.995963 ip-10-0-135-34 kubenswrapper[2578]: I0707 11:04:51.995935 2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-1273de\"/\"openshift-service-ca.crt\"" Jul 07 11:04:51.996105 ip-10-0-135-34 kubenswrapper[2578]: I0707 11:04:51.995996 2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-1273de\"/\"kube-root-ca.crt\"" Jul 07 11:04:51.997070 ip-10-0-135-34 kubenswrapper[2578]: I0707 11:04:51.997048 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-1273de\"/\"default-dockercfg-kbxpw\"" Jul 07 11:04:51.997205 ip-10-0-135-34 kubenswrapper[2578]: I0707 11:04:51.997182 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-1273de\"/\"sub-test-model-1273de-kserve-self-signed-certs\"" Jul 07 11:04:52.024451 ip-10-0-135-34 kubenswrapper[2578]: I0707 11:04:52.024421 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-1273de/sub-test-model-1273de-kserve-687cd89fbc-fd9tf"] Jul 07 11:04:52.077614 ip-10-0-135-34 kubenswrapper[2578]: I0707 11:04:52.077574 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1a1fb38c-a193-47b3-a8de-66f6228475af-home\") pod \"sub-test-model-1273de-kserve-687cd89fbc-fd9tf\" (UID: \"1a1fb38c-a193-47b3-a8de-66f6228475af\") " pod="ai-tenant-e2e-shared-a-1273de/sub-test-model-1273de-kserve-687cd89fbc-fd9tf" Jul 07 11:04:52.077818 ip-10-0-135-34 kubenswrapper[2578]: I0707 11:04:52.077667 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1a1fb38c-a193-47b3-a8de-66f6228475af-tls-certs\") pod \"sub-test-model-1273de-kserve-687cd89fbc-fd9tf\" (UID: \"1a1fb38c-a193-47b3-a8de-66f6228475af\") " pod="ai-tenant-e2e-shared-a-1273de/sub-test-model-1273de-kserve-687cd89fbc-fd9tf" Jul 07 11:04:52.077818 ip-10-0-135-34 kubenswrapper[2578]: I0707 11:04:52.077712 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1a1fb38c-a193-47b3-a8de-66f6228475af-dshm\") pod \"sub-test-model-1273de-kserve-687cd89fbc-fd9tf\" (UID: \"1a1fb38c-a193-47b3-a8de-66f6228475af\") " pod="ai-tenant-e2e-shared-a-1273de/sub-test-model-1273de-kserve-687cd89fbc-fd9tf" Jul 07 11:04:52.077818 ip-10-0-135-34 kubenswrapper[2578]: I0707 11:04:52.077747 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1a1fb38c-a193-47b3-a8de-66f6228475af-model-cache\") pod \"sub-test-model-1273de-kserve-687cd89fbc-fd9tf\" (UID: \"1a1fb38c-a193-47b3-a8de-66f6228475af\") " pod="ai-tenant-e2e-shared-a-1273de/sub-test-model-1273de-kserve-687cd89fbc-fd9tf" Jul 07 11:04:52.077996 ip-10-0-135-34 kubenswrapper[2578]: I0707 11:04:52.077887 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1a1fb38c-a193-47b3-a8de-66f6228475af-tmp-dir\") pod \"sub-test-model-1273de-kserve-687cd89fbc-fd9tf\" (UID: \"1a1fb38c-a193-47b3-a8de-66f6228475af\") " pod="ai-tenant-e2e-shared-a-1273de/sub-test-model-1273de-kserve-687cd89fbc-fd9tf" Jul 07 11:04:52.077996 ip-10-0-135-34 kubenswrapper[2578]: I0707 11:04:52.077922 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-t7fw6\" (UniqueName: \"kubernetes.io/projected/1a1fb38c-a193-47b3-a8de-66f6228475af-kube-api-access-t7fw6\") pod \"sub-test-model-1273de-kserve-687cd89fbc-fd9tf\" (UID: \"1a1fb38c-a193-47b3-a8de-66f6228475af\") " pod="ai-tenant-e2e-shared-a-1273de/sub-test-model-1273de-kserve-687cd89fbc-fd9tf" Jul 07 11:04:52.179226 ip-10-0-135-34 kubenswrapper[2578]: I0707 11:04:52.179174 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1a1fb38c-a193-47b3-a8de-66f6228475af-home\") pod \"sub-test-model-1273de-kserve-687cd89fbc-fd9tf\" (UID: \"1a1fb38c-a193-47b3-a8de-66f6228475af\") " pod="ai-tenant-e2e-shared-a-1273de/sub-test-model-1273de-kserve-687cd89fbc-fd9tf" Jul 07 11:04:52.179407 ip-10-0-135-34 kubenswrapper[2578]: I0707 11:04:52.179233 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1a1fb38c-a193-47b3-a8de-66f6228475af-tls-certs\") pod \"sub-test-model-1273de-kserve-687cd89fbc-fd9tf\" (UID: \"1a1fb38c-a193-47b3-a8de-66f6228475af\") " pod="ai-tenant-e2e-shared-a-1273de/sub-test-model-1273de-kserve-687cd89fbc-fd9tf" Jul 07 11:04:52.179407 ip-10-0-135-34 kubenswrapper[2578]: I0707 11:04:52.179265 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1a1fb38c-a193-47b3-a8de-66f6228475af-dshm\") pod \"sub-test-model-1273de-kserve-687cd89fbc-fd9tf\" (UID: \"1a1fb38c-a193-47b3-a8de-66f6228475af\") " pod="ai-tenant-e2e-shared-a-1273de/sub-test-model-1273de-kserve-687cd89fbc-fd9tf" Jul 07 11:04:52.179407 ip-10-0-135-34 kubenswrapper[2578]: I0707 11:04:52.179319 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1a1fb38c-a193-47b3-a8de-66f6228475af-model-cache\") pod \"sub-test-model-1273de-kserve-687cd89fbc-fd9tf\" (UID: \"1a1fb38c-a193-47b3-a8de-66f6228475af\") " pod="ai-tenant-e2e-shared-a-1273de/sub-test-model-1273de-kserve-687cd89fbc-fd9tf" Jul 07 11:04:52.179578 ip-10-0-135-34 kubenswrapper[2578]: I0707 11:04:52.179415 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1a1fb38c-a193-47b3-a8de-66f6228475af-tmp-dir\") pod \"sub-test-model-1273de-kserve-687cd89fbc-fd9tf\" (UID: \"1a1fb38c-a193-47b3-a8de-66f6228475af\") " pod="ai-tenant-e2e-shared-a-1273de/sub-test-model-1273de-kserve-687cd89fbc-fd9tf" Jul 07 11:04:52.179578 ip-10-0-135-34 kubenswrapper[2578]: I0707 11:04:52.179439 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t7fw6\" (UniqueName: \"kubernetes.io/projected/1a1fb38c-a193-47b3-a8de-66f6228475af-kube-api-access-t7fw6\") pod \"sub-test-model-1273de-kserve-687cd89fbc-fd9tf\" (UID: \"1a1fb38c-a193-47b3-a8de-66f6228475af\") " pod="ai-tenant-e2e-shared-a-1273de/sub-test-model-1273de-kserve-687cd89fbc-fd9tf" Jul 07 11:04:52.179687 ip-10-0-135-34 kubenswrapper[2578]: I0707 11:04:52.179664 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1a1fb38c-a193-47b3-a8de-66f6228475af-home\") pod \"sub-test-model-1273de-kserve-687cd89fbc-fd9tf\" (UID: \"1a1fb38c-a193-47b3-a8de-66f6228475af\") " pod="ai-tenant-e2e-shared-a-1273de/sub-test-model-1273de-kserve-687cd89fbc-fd9tf" Jul 07 11:04:52.179826 ip-10-0-135-34 kubenswrapper[2578]: I0707 11:04:52.179765 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1a1fb38c-a193-47b3-a8de-66f6228475af-tmp-dir\") pod \"sub-test-model-1273de-kserve-687cd89fbc-fd9tf\" (UID: \"1a1fb38c-a193-47b3-a8de-66f6228475af\") " pod="ai-tenant-e2e-shared-a-1273de/sub-test-model-1273de-kserve-687cd89fbc-fd9tf" Jul 07 11:04:52.179893 ip-10-0-135-34 kubenswrapper[2578]: I0707 11:04:52.179877 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1a1fb38c-a193-47b3-a8de-66f6228475af-model-cache\") pod \"sub-test-model-1273de-kserve-687cd89fbc-fd9tf\" (UID: \"1a1fb38c-a193-47b3-a8de-66f6228475af\") " pod="ai-tenant-e2e-shared-a-1273de/sub-test-model-1273de-kserve-687cd89fbc-fd9tf" Jul 07 11:04:52.181735 ip-10-0-135-34 kubenswrapper[2578]: I0707 11:04:52.181716 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1a1fb38c-a193-47b3-a8de-66f6228475af-dshm\") pod \"sub-test-model-1273de-kserve-687cd89fbc-fd9tf\" (UID: \"1a1fb38c-a193-47b3-a8de-66f6228475af\") " pod="ai-tenant-e2e-shared-a-1273de/sub-test-model-1273de-kserve-687cd89fbc-fd9tf" Jul 07 11:04:52.181997 ip-10-0-135-34 kubenswrapper[2578]: I0707 11:04:52.181970 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1a1fb38c-a193-47b3-a8de-66f6228475af-tls-certs\") pod \"sub-test-model-1273de-kserve-687cd89fbc-fd9tf\" (UID: \"1a1fb38c-a193-47b3-a8de-66f6228475af\") " pod="ai-tenant-e2e-shared-a-1273de/sub-test-model-1273de-kserve-687cd89fbc-fd9tf" Jul 07 11:04:52.197224 ip-10-0-135-34 kubenswrapper[2578]: I0707 11:04:52.197192 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-t7fw6\" (UniqueName: \"kubernetes.io/projected/1a1fb38c-a193-47b3-a8de-66f6228475af-kube-api-access-t7fw6\") pod \"sub-test-model-1273de-kserve-687cd89fbc-fd9tf\" (UID: \"1a1fb38c-a193-47b3-a8de-66f6228475af\") " pod="ai-tenant-e2e-shared-a-1273de/sub-test-model-1273de-kserve-687cd89fbc-fd9tf" Jul 07 11:04:52.303333 ip-10-0-135-34 kubenswrapper[2578]: I0707 11:04:52.303238 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-1273de/sub-test-model-1273de-kserve-687cd89fbc-fd9tf" Jul 07 11:04:52.470848 ip-10-0-135-34 kubenswrapper[2578]: I0707 11:04:52.470826 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-1273de/sub-test-model-1273de-kserve-687cd89fbc-fd9tf"] Jul 07 11:04:52.473266 ip-10-0-135-34 kubenswrapper[2578]: W0707 11:04:52.473241 2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod1a1fb38c_a193_47b3_a8de_66f6228475af.slice/crio-71ccfc40fd345e6fe21933ebcb4296b7b1f5fa006a71e2ee3c760f81aad1ca94 WatchSource:0}: Error finding container 71ccfc40fd345e6fe21933ebcb4296b7b1f5fa006a71e2ee3c760f81aad1ca94: Status 404 returned error can't find the container with id 71ccfc40fd345e6fe21933ebcb4296b7b1f5fa006a71e2ee3c760f81aad1ca94 Jul 07 11:04:52.822853 ip-10-0-135-34 kubenswrapper[2578]: I0707 11:04:52.822819 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-1273de/sub-test-model-1273de-kserve-687cd89fbc-fd9tf" event={"ID":"1a1fb38c-a193-47b3-a8de-66f6228475af","Type":"ContainerStarted","Data":"31405dabcc8e48e775365ca14a5f9333beffc5aa4b74e1f5add6205fef22e9fe"} Jul 07 11:04:52.823063 ip-10-0-135-34 kubenswrapper[2578]: I0707 11:04:52.822862 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-1273de/sub-test-model-1273de-kserve-687cd89fbc-fd9tf" event={"ID":"1a1fb38c-a193-47b3-a8de-66f6228475af","Type":"ContainerStarted","Data":"71ccfc40fd345e6fe21933ebcb4296b7b1f5fa006a71e2ee3c760f81aad1ca94"} Jul 07 11:04:52.867801 ip-10-0-135-34 kubenswrapper[2578]: I0707 11:04:52.867736 2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-a-1273de/sub-test-model-1273de-kserve-687cd89fbc-fd9tf" podStartSLOduration=1.867720227 podStartE2EDuration="1.867720227s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-07 11:04:51 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 11:04:52.866542814 +0000 UTC m=+2261.538502377" watchObservedRunningTime="2026-07-07 11:04:52.867720227 +0000 UTC m=+2261.539679771" Jul 07 11:04:55.417338 ip-10-0-135-34 kubenswrapper[2578]: I0707 11:04:55.417296 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-lv7nd"] Jul 07 11:05:02.304378 ip-10-0-135-34 kubenswrapper[2578]: I0707 11:05:02.304336 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-a-1273de/sub-test-model-1273de-kserve-687cd89fbc-fd9tf" Jul 07 11:05:02.304378 ip-10-0-135-34 kubenswrapper[2578]: I0707 11:05:02.304391 2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-a-1273de/sub-test-model-1273de-kserve-687cd89fbc-fd9tf" Jul 07 11:05:02.318439 ip-10-0-135-34 kubenswrapper[2578]: I0707 11:05:02.318404 2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-a-1273de/sub-test-model-1273de-kserve-687cd89fbc-fd9tf" Jul 07 11:05:02.887081 ip-10-0-135-34 kubenswrapper[2578]: I0707 11:05:02.887042 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-a-1273de/sub-test-model-1273de-kserve-687cd89fbc-fd9tf" Jul 07 11:05:14.603806 ip-10-0-135-34 kubenswrapper[2578]: I0707 11:05:14.603755 2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-b-2a3633/sub-test-model-2a3633-kserve-684c78dc-xm9dl"] Jul 07 11:05:14.607435 ip-10-0-135-34 kubenswrapper[2578]: I0707 11:05:14.607411 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-2a3633/sub-test-model-2a3633-kserve-684c78dc-xm9dl" Jul 07 11:05:14.611290 ip-10-0-135-34 kubenswrapper[2578]: I0707 11:05:14.611269 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-2a3633\"/\"default-dockercfg-nq96c\"" Jul 07 11:05:14.611931 ip-10-0-135-34 kubenswrapper[2578]: I0707 11:05:14.611911 2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-2a3633\"/\"openshift-service-ca.crt\"" Jul 07 11:05:14.612677 ip-10-0-135-34 kubenswrapper[2578]: I0707 11:05:14.612652 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-2a3633\"/\"sub-test-model-2a3633-kserve-self-signed-certs\"" Jul 07 11:05:14.612753 ip-10-0-135-34 kubenswrapper[2578]: I0707 11:05:14.612689 2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-2a3633\"/\"kube-root-ca.crt\"" Jul 07 11:05:14.631321 ip-10-0-135-34 kubenswrapper[2578]: I0707 11:05:14.631288 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-2a3633/sub-test-model-2a3633-kserve-684c78dc-xm9dl"] Jul 07 11:05:14.673884 ip-10-0-135-34 kubenswrapper[2578]: I0707 11:05:14.673847 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d8380902-3ad7-4421-9c0f-9bcedb5f848c-model-cache\") pod \"sub-test-model-2a3633-kserve-684c78dc-xm9dl\" (UID: \"d8380902-3ad7-4421-9c0f-9bcedb5f848c\") " pod="ai-tenant-e2e-shared-b-2a3633/sub-test-model-2a3633-kserve-684c78dc-xm9dl" Jul 07 11:05:14.674065 ip-10-0-135-34 kubenswrapper[2578]: I0707 11:05:14.673914 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d8380902-3ad7-4421-9c0f-9bcedb5f848c-tls-certs\") pod \"sub-test-model-2a3633-kserve-684c78dc-xm9dl\" (UID: \"d8380902-3ad7-4421-9c0f-9bcedb5f848c\") " pod="ai-tenant-e2e-shared-b-2a3633/sub-test-model-2a3633-kserve-684c78dc-xm9dl" Jul 07 11:05:14.674065 ip-10-0-135-34 kubenswrapper[2578]: I0707 11:05:14.673969 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d8380902-3ad7-4421-9c0f-9bcedb5f848c-tmp-dir\") pod \"sub-test-model-2a3633-kserve-684c78dc-xm9dl\" (UID: \"d8380902-3ad7-4421-9c0f-9bcedb5f848c\") " pod="ai-tenant-e2e-shared-b-2a3633/sub-test-model-2a3633-kserve-684c78dc-xm9dl" Jul 07 11:05:14.674065 ip-10-0-135-34 kubenswrapper[2578]: I0707 11:05:14.673985 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vzrtm\" (UniqueName: \"kubernetes.io/projected/d8380902-3ad7-4421-9c0f-9bcedb5f848c-kube-api-access-vzrtm\") pod \"sub-test-model-2a3633-kserve-684c78dc-xm9dl\" (UID: \"d8380902-3ad7-4421-9c0f-9bcedb5f848c\") " pod="ai-tenant-e2e-shared-b-2a3633/sub-test-model-2a3633-kserve-684c78dc-xm9dl" Jul 07 11:05:14.674065 ip-10-0-135-34 kubenswrapper[2578]: I0707 11:05:14.674021 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d8380902-3ad7-4421-9c0f-9bcedb5f848c-home\") pod \"sub-test-model-2a3633-kserve-684c78dc-xm9dl\" (UID: \"d8380902-3ad7-4421-9c0f-9bcedb5f848c\") " pod="ai-tenant-e2e-shared-b-2a3633/sub-test-model-2a3633-kserve-684c78dc-xm9dl" Jul 07 11:05:14.674065 ip-10-0-135-34 kubenswrapper[2578]: I0707 11:05:14.674046 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d8380902-3ad7-4421-9c0f-9bcedb5f848c-dshm\") pod \"sub-test-model-2a3633-kserve-684c78dc-xm9dl\" (UID: \"d8380902-3ad7-4421-9c0f-9bcedb5f848c\") " pod="ai-tenant-e2e-shared-b-2a3633/sub-test-model-2a3633-kserve-684c78dc-xm9dl" Jul 07 11:05:14.774725 ip-10-0-135-34 kubenswrapper[2578]: I0707 11:05:14.774679 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d8380902-3ad7-4421-9c0f-9bcedb5f848c-tls-certs\") pod \"sub-test-model-2a3633-kserve-684c78dc-xm9dl\" (UID: \"d8380902-3ad7-4421-9c0f-9bcedb5f848c\") " pod="ai-tenant-e2e-shared-b-2a3633/sub-test-model-2a3633-kserve-684c78dc-xm9dl" Jul 07 11:05:14.774960 ip-10-0-135-34 kubenswrapper[2578]: I0707 11:05:14.774742 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d8380902-3ad7-4421-9c0f-9bcedb5f848c-tmp-dir\") pod \"sub-test-model-2a3633-kserve-684c78dc-xm9dl\" (UID: \"d8380902-3ad7-4421-9c0f-9bcedb5f848c\") " pod="ai-tenant-e2e-shared-b-2a3633/sub-test-model-2a3633-kserve-684c78dc-xm9dl" Jul 07 11:05:14.774960 ip-10-0-135-34 kubenswrapper[2578]: I0707 11:05:14.774795 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vzrtm\" (UniqueName: \"kubernetes.io/projected/d8380902-3ad7-4421-9c0f-9bcedb5f848c-kube-api-access-vzrtm\") pod \"sub-test-model-2a3633-kserve-684c78dc-xm9dl\" (UID: \"d8380902-3ad7-4421-9c0f-9bcedb5f848c\") " pod="ai-tenant-e2e-shared-b-2a3633/sub-test-model-2a3633-kserve-684c78dc-xm9dl" Jul 07 11:05:14.774960 ip-10-0-135-34 kubenswrapper[2578]: I0707 11:05:14.774841 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d8380902-3ad7-4421-9c0f-9bcedb5f848c-home\") pod \"sub-test-model-2a3633-kserve-684c78dc-xm9dl\" (UID: \"d8380902-3ad7-4421-9c0f-9bcedb5f848c\") " pod="ai-tenant-e2e-shared-b-2a3633/sub-test-model-2a3633-kserve-684c78dc-xm9dl" Jul 07 11:05:14.774960 ip-10-0-135-34 kubenswrapper[2578]: I0707 11:05:14.774878 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d8380902-3ad7-4421-9c0f-9bcedb5f848c-dshm\") pod \"sub-test-model-2a3633-kserve-684c78dc-xm9dl\" (UID: \"d8380902-3ad7-4421-9c0f-9bcedb5f848c\") " pod="ai-tenant-e2e-shared-b-2a3633/sub-test-model-2a3633-kserve-684c78dc-xm9dl" Jul 07 11:05:14.775181 ip-10-0-135-34 kubenswrapper[2578]: I0707 11:05:14.775024 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d8380902-3ad7-4421-9c0f-9bcedb5f848c-model-cache\") pod \"sub-test-model-2a3633-kserve-684c78dc-xm9dl\" (UID: \"d8380902-3ad7-4421-9c0f-9bcedb5f848c\") " pod="ai-tenant-e2e-shared-b-2a3633/sub-test-model-2a3633-kserve-684c78dc-xm9dl" Jul 07 11:05:14.775259 ip-10-0-135-34 kubenswrapper[2578]: I0707 11:05:14.775240 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d8380902-3ad7-4421-9c0f-9bcedb5f848c-tmp-dir\") pod \"sub-test-model-2a3633-kserve-684c78dc-xm9dl\" (UID: \"d8380902-3ad7-4421-9c0f-9bcedb5f848c\") " pod="ai-tenant-e2e-shared-b-2a3633/sub-test-model-2a3633-kserve-684c78dc-xm9dl" Jul 07 11:05:14.775314 ip-10-0-135-34 kubenswrapper[2578]: I0707 11:05:14.775294 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d8380902-3ad7-4421-9c0f-9bcedb5f848c-home\") pod \"sub-test-model-2a3633-kserve-684c78dc-xm9dl\" (UID: \"d8380902-3ad7-4421-9c0f-9bcedb5f848c\") " pod="ai-tenant-e2e-shared-b-2a3633/sub-test-model-2a3633-kserve-684c78dc-xm9dl" Jul 07 11:05:14.775385 ip-10-0-135-34 kubenswrapper[2578]: I0707 11:05:14.775369 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d8380902-3ad7-4421-9c0f-9bcedb5f848c-model-cache\") pod \"sub-test-model-2a3633-kserve-684c78dc-xm9dl\" (UID: \"d8380902-3ad7-4421-9c0f-9bcedb5f848c\") " pod="ai-tenant-e2e-shared-b-2a3633/sub-test-model-2a3633-kserve-684c78dc-xm9dl" Jul 07 11:05:14.777210 ip-10-0-135-34 kubenswrapper[2578]: I0707 11:05:14.777191 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d8380902-3ad7-4421-9c0f-9bcedb5f848c-dshm\") pod \"sub-test-model-2a3633-kserve-684c78dc-xm9dl\" (UID: \"d8380902-3ad7-4421-9c0f-9bcedb5f848c\") " pod="ai-tenant-e2e-shared-b-2a3633/sub-test-model-2a3633-kserve-684c78dc-xm9dl" Jul 07 11:05:14.777493 ip-10-0-135-34 kubenswrapper[2578]: I0707 11:05:14.777472 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d8380902-3ad7-4421-9c0f-9bcedb5f848c-tls-certs\") pod \"sub-test-model-2a3633-kserve-684c78dc-xm9dl\" (UID: \"d8380902-3ad7-4421-9c0f-9bcedb5f848c\") " pod="ai-tenant-e2e-shared-b-2a3633/sub-test-model-2a3633-kserve-684c78dc-xm9dl" Jul 07 11:05:14.797755 ip-10-0-135-34 kubenswrapper[2578]: I0707 11:05:14.797724 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vzrtm\" (UniqueName: \"kubernetes.io/projected/d8380902-3ad7-4421-9c0f-9bcedb5f848c-kube-api-access-vzrtm\") pod \"sub-test-model-2a3633-kserve-684c78dc-xm9dl\" (UID: \"d8380902-3ad7-4421-9c0f-9bcedb5f848c\") " pod="ai-tenant-e2e-shared-b-2a3633/sub-test-model-2a3633-kserve-684c78dc-xm9dl" Jul 07 11:05:14.916501 ip-10-0-135-34 kubenswrapper[2578]: I0707 11:05:14.916413 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-2a3633/sub-test-model-2a3633-kserve-684c78dc-xm9dl" Jul 07 11:05:15.069223 ip-10-0-135-34 kubenswrapper[2578]: I0707 11:05:15.069195 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-2a3633/sub-test-model-2a3633-kserve-684c78dc-xm9dl"] Jul 07 11:05:15.070900 ip-10-0-135-34 kubenswrapper[2578]: W0707 11:05:15.070877 2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podd8380902_3ad7_4421_9c0f_9bcedb5f848c.slice/crio-78350307f00ba57c587900428008ef3ecf0457a2976a89e7f6050bf89aee06db WatchSource:0}: Error finding container 78350307f00ba57c587900428008ef3ecf0457a2976a89e7f6050bf89aee06db: Status 404 returned error can't find the container with id 78350307f00ba57c587900428008ef3ecf0457a2976a89e7f6050bf89aee06db Jul 07 11:05:15.941817 ip-10-0-135-34 kubenswrapper[2578]: I0707 11:05:15.941782 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-2a3633/sub-test-model-2a3633-kserve-684c78dc-xm9dl" event={"ID":"d8380902-3ad7-4421-9c0f-9bcedb5f848c","Type":"ContainerStarted","Data":"5c1936e15a5e2ad5346a266dd82d2988a1a5042e79fa4e2d15e483af2c5314c3"} Jul 07 11:05:15.942205 ip-10-0-135-34 kubenswrapper[2578]: I0707 11:05:15.941826 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-2a3633/sub-test-model-2a3633-kserve-684c78dc-xm9dl" event={"ID":"d8380902-3ad7-4421-9c0f-9bcedb5f848c","Type":"ContainerStarted","Data":"78350307f00ba57c587900428008ef3ecf0457a2976a89e7f6050bf89aee06db"} Jul 07 11:05:15.973414 ip-10-0-135-34 kubenswrapper[2578]: I0707 11:05:15.973364 2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-b-2a3633/sub-test-model-2a3633-kserve-684c78dc-xm9dl" podStartSLOduration=1.973348559 podStartE2EDuration="1.973348559s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-07 11:05:14 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 11:05:15.97312233 +0000 UTC m=+2284.645081874" watchObservedRunningTime="2026-07-07 11:05:15.973348559 +0000 UTC m=+2284.645308102" Jul 07 11:05:19.501704 ip-10-0-135-34 kubenswrapper[2578]: I0707 11:05:19.501658 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-lv7nd"] Jul 07 11:05:24.917155 ip-10-0-135-34 kubenswrapper[2578]: I0707 11:05:24.917117 2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-b-2a3633/sub-test-model-2a3633-kserve-684c78dc-xm9dl" Jul 07 11:05:24.917548 ip-10-0-135-34 kubenswrapper[2578]: I0707 11:05:24.917172 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-b-2a3633/sub-test-model-2a3633-kserve-684c78dc-xm9dl" Jul 07 11:05:24.929403 ip-10-0-135-34 kubenswrapper[2578]: I0707 11:05:24.929377 2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-b-2a3633/sub-test-model-2a3633-kserve-684c78dc-xm9dl" Jul 07 11:05:25.005853 ip-10-0-135-34 kubenswrapper[2578]: I0707 11:05:25.005815 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-b-2a3633/sub-test-model-2a3633-kserve-684c78dc-xm9dl" Jul 07 11:05:32.476240 ip-10-0-135-34 kubenswrapper[2578]: I0707 11:05:32.476198 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-lv7nd"] Jul 07 11:05:46.715978 ip-10-0-135-34 kubenswrapper[2578]: I0707 11:05:46.715940 2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-1273de/sub-test-model-1273de-kserve-687cd89fbc-fd9tf"] Jul 07 11:05:47.604700 ip-10-0-135-34 kubenswrapper[2578]: I0707 11:05:47.604660 2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-2a3633/sub-test-model-2a3633-kserve-684c78dc-xm9dl"] Jul 07 11:05:50.830806 ip-10-0-135-34 kubenswrapper[2578]: I0707 11:05:50.830754 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-lv7nd"] Jul 07 11:05:53.337488 ip-10-0-135-34 kubenswrapper[2578]: I0707 11:05:53.337454 2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-a-1273de/rate-test-model-1273de-kserve-57bcd69fbc-df5m8"] Jul 07 11:05:53.341046 ip-10-0-135-34 kubenswrapper[2578]: I0707 11:05:53.341027 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-1273de/rate-test-model-1273de-kserve-57bcd69fbc-df5m8" Jul 07 11:05:53.344382 ip-10-0-135-34 kubenswrapper[2578]: I0707 11:05:53.344354 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-1273de\"/\"rate-test-model-1273de-kserve-self-signed-certs\"" Jul 07 11:05:53.357485 ip-10-0-135-34 kubenswrapper[2578]: I0707 11:05:53.357460 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-1273de/rate-test-model-1273de-kserve-57bcd69fbc-df5m8"] Jul 07 11:05:53.426076 ip-10-0-135-34 kubenswrapper[2578]: I0707 11:05:53.426040 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3959f4ca-7868-4862-b576-b679546ac4ad-dshm\") pod \"rate-test-model-1273de-kserve-57bcd69fbc-df5m8\" (UID: \"3959f4ca-7868-4862-b576-b679546ac4ad\") " pod="ai-tenant-e2e-shared-a-1273de/rate-test-model-1273de-kserve-57bcd69fbc-df5m8" Jul 07 11:05:53.426273 ip-10-0-135-34 kubenswrapper[2578]: I0707 11:05:53.426099 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3959f4ca-7868-4862-b576-b679546ac4ad-tmp-dir\") pod \"rate-test-model-1273de-kserve-57bcd69fbc-df5m8\" (UID: \"3959f4ca-7868-4862-b576-b679546ac4ad\") " pod="ai-tenant-e2e-shared-a-1273de/rate-test-model-1273de-kserve-57bcd69fbc-df5m8" Jul 07 11:05:53.426273 ip-10-0-135-34 kubenswrapper[2578]: I0707 11:05:53.426180 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lsqb8\" (UniqueName: \"kubernetes.io/projected/3959f4ca-7868-4862-b576-b679546ac4ad-kube-api-access-lsqb8\") pod \"rate-test-model-1273de-kserve-57bcd69fbc-df5m8\" (UID: \"3959f4ca-7868-4862-b576-b679546ac4ad\") " pod="ai-tenant-e2e-shared-a-1273de/rate-test-model-1273de-kserve-57bcd69fbc-df5m8" Jul 07 11:05:53.426273 ip-10-0-135-34 kubenswrapper[2578]: I0707 11:05:53.426221 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3959f4ca-7868-4862-b576-b679546ac4ad-tls-certs\") pod \"rate-test-model-1273de-kserve-57bcd69fbc-df5m8\" (UID: \"3959f4ca-7868-4862-b576-b679546ac4ad\") " pod="ai-tenant-e2e-shared-a-1273de/rate-test-model-1273de-kserve-57bcd69fbc-df5m8" Jul 07 11:05:53.426273 ip-10-0-135-34 kubenswrapper[2578]: I0707 11:05:53.426255 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3959f4ca-7868-4862-b576-b679546ac4ad-model-cache\") pod \"rate-test-model-1273de-kserve-57bcd69fbc-df5m8\" (UID: \"3959f4ca-7868-4862-b576-b679546ac4ad\") " pod="ai-tenant-e2e-shared-a-1273de/rate-test-model-1273de-kserve-57bcd69fbc-df5m8" Jul 07 11:05:53.426412 ip-10-0-135-34 kubenswrapper[2578]: I0707 11:05:53.426288 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3959f4ca-7868-4862-b576-b679546ac4ad-home\") pod \"rate-test-model-1273de-kserve-57bcd69fbc-df5m8\" (UID: \"3959f4ca-7868-4862-b576-b679546ac4ad\") " pod="ai-tenant-e2e-shared-a-1273de/rate-test-model-1273de-kserve-57bcd69fbc-df5m8" Jul 07 11:05:53.527242 ip-10-0-135-34 kubenswrapper[2578]: I0707 11:05:53.527199 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3959f4ca-7868-4862-b576-b679546ac4ad-tmp-dir\") pod \"rate-test-model-1273de-kserve-57bcd69fbc-df5m8\" (UID: \"3959f4ca-7868-4862-b576-b679546ac4ad\") " pod="ai-tenant-e2e-shared-a-1273de/rate-test-model-1273de-kserve-57bcd69fbc-df5m8" Jul 07 11:05:53.527433 ip-10-0-135-34 kubenswrapper[2578]: I0707 11:05:53.527253 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lsqb8\" (UniqueName: \"kubernetes.io/projected/3959f4ca-7868-4862-b576-b679546ac4ad-kube-api-access-lsqb8\") pod \"rate-test-model-1273de-kserve-57bcd69fbc-df5m8\" (UID: \"3959f4ca-7868-4862-b576-b679546ac4ad\") " pod="ai-tenant-e2e-shared-a-1273de/rate-test-model-1273de-kserve-57bcd69fbc-df5m8" Jul 07 11:05:53.527433 ip-10-0-135-34 kubenswrapper[2578]: I0707 11:05:53.527275 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3959f4ca-7868-4862-b576-b679546ac4ad-tls-certs\") pod \"rate-test-model-1273de-kserve-57bcd69fbc-df5m8\" (UID: \"3959f4ca-7868-4862-b576-b679546ac4ad\") " pod="ai-tenant-e2e-shared-a-1273de/rate-test-model-1273de-kserve-57bcd69fbc-df5m8" Jul 07 11:05:53.527433 ip-10-0-135-34 kubenswrapper[2578]: I0707 11:05:53.527303 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3959f4ca-7868-4862-b576-b679546ac4ad-model-cache\") pod \"rate-test-model-1273de-kserve-57bcd69fbc-df5m8\" (UID: \"3959f4ca-7868-4862-b576-b679546ac4ad\") " pod="ai-tenant-e2e-shared-a-1273de/rate-test-model-1273de-kserve-57bcd69fbc-df5m8" Jul 07 11:05:53.527433 ip-10-0-135-34 kubenswrapper[2578]: I0707 11:05:53.527335 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3959f4ca-7868-4862-b576-b679546ac4ad-home\") pod \"rate-test-model-1273de-kserve-57bcd69fbc-df5m8\" (UID: \"3959f4ca-7868-4862-b576-b679546ac4ad\") " pod="ai-tenant-e2e-shared-a-1273de/rate-test-model-1273de-kserve-57bcd69fbc-df5m8" Jul 07 11:05:53.527433 ip-10-0-135-34 kubenswrapper[2578]: I0707 11:05:53.527362 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3959f4ca-7868-4862-b576-b679546ac4ad-dshm\") pod \"rate-test-model-1273de-kserve-57bcd69fbc-df5m8\" (UID: \"3959f4ca-7868-4862-b576-b679546ac4ad\") " pod="ai-tenant-e2e-shared-a-1273de/rate-test-model-1273de-kserve-57bcd69fbc-df5m8" Jul 07 11:05:53.527705 ip-10-0-135-34 kubenswrapper[2578]: I0707 11:05:53.527678 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3959f4ca-7868-4862-b576-b679546ac4ad-tmp-dir\") pod \"rate-test-model-1273de-kserve-57bcd69fbc-df5m8\" (UID: \"3959f4ca-7868-4862-b576-b679546ac4ad\") " pod="ai-tenant-e2e-shared-a-1273de/rate-test-model-1273de-kserve-57bcd69fbc-df5m8" Jul 07 11:05:53.527747 ip-10-0-135-34 kubenswrapper[2578]: I0707 11:05:53.527710 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3959f4ca-7868-4862-b576-b679546ac4ad-model-cache\") pod \"rate-test-model-1273de-kserve-57bcd69fbc-df5m8\" (UID: \"3959f4ca-7868-4862-b576-b679546ac4ad\") " pod="ai-tenant-e2e-shared-a-1273de/rate-test-model-1273de-kserve-57bcd69fbc-df5m8" Jul 07 11:05:53.527826 ip-10-0-135-34 kubenswrapper[2578]: I0707 11:05:53.527749 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3959f4ca-7868-4862-b576-b679546ac4ad-home\") pod \"rate-test-model-1273de-kserve-57bcd69fbc-df5m8\" (UID: \"3959f4ca-7868-4862-b576-b679546ac4ad\") " pod="ai-tenant-e2e-shared-a-1273de/rate-test-model-1273de-kserve-57bcd69fbc-df5m8" Jul 07 11:05:53.529881 ip-10-0-135-34 kubenswrapper[2578]: I0707 11:05:53.529859 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3959f4ca-7868-4862-b576-b679546ac4ad-dshm\") pod \"rate-test-model-1273de-kserve-57bcd69fbc-df5m8\" (UID: \"3959f4ca-7868-4862-b576-b679546ac4ad\") " pod="ai-tenant-e2e-shared-a-1273de/rate-test-model-1273de-kserve-57bcd69fbc-df5m8" Jul 07 11:05:53.530041 ip-10-0-135-34 kubenswrapper[2578]: I0707 11:05:53.530022 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3959f4ca-7868-4862-b576-b679546ac4ad-tls-certs\") pod \"rate-test-model-1273de-kserve-57bcd69fbc-df5m8\" (UID: \"3959f4ca-7868-4862-b576-b679546ac4ad\") " pod="ai-tenant-e2e-shared-a-1273de/rate-test-model-1273de-kserve-57bcd69fbc-df5m8" Jul 07 11:05:53.537837 ip-10-0-135-34 kubenswrapper[2578]: I0707 11:05:53.537814 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lsqb8\" (UniqueName: \"kubernetes.io/projected/3959f4ca-7868-4862-b576-b679546ac4ad-kube-api-access-lsqb8\") pod \"rate-test-model-1273de-kserve-57bcd69fbc-df5m8\" (UID: \"3959f4ca-7868-4862-b576-b679546ac4ad\") " pod="ai-tenant-e2e-shared-a-1273de/rate-test-model-1273de-kserve-57bcd69fbc-df5m8" Jul 07 11:05:53.650858 ip-10-0-135-34 kubenswrapper[2578]: I0707 11:05:53.650823 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-1273de/rate-test-model-1273de-kserve-57bcd69fbc-df5m8" Jul 07 11:05:53.787091 ip-10-0-135-34 kubenswrapper[2578]: I0707 11:05:53.787058 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-1273de/rate-test-model-1273de-kserve-57bcd69fbc-df5m8"] Jul 07 11:05:53.788618 ip-10-0-135-34 kubenswrapper[2578]: W0707 11:05:53.788593 2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod3959f4ca_7868_4862_b576_b679546ac4ad.slice/crio-c05fdae1f3791b9e041eff15e363a09c11a9d6eee4e09f7c8173cbb0d701434a WatchSource:0}: Error finding container c05fdae1f3791b9e041eff15e363a09c11a9d6eee4e09f7c8173cbb0d701434a: Status 404 returned error can't find the container with id c05fdae1f3791b9e041eff15e363a09c11a9d6eee4e09f7c8173cbb0d701434a Jul 07 11:05:54.132946 ip-10-0-135-34 kubenswrapper[2578]: I0707 11:05:54.132906 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-1273de/rate-test-model-1273de-kserve-57bcd69fbc-df5m8" event={"ID":"3959f4ca-7868-4862-b576-b679546ac4ad","Type":"ContainerStarted","Data":"08876ea07fd6fdd679884dcda41b983d6cc1ecc5e20a49c8abe7c4b341cfcda2"} Jul 07 11:05:54.133121 ip-10-0-135-34 kubenswrapper[2578]: I0707 11:05:54.132956 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-1273de/rate-test-model-1273de-kserve-57bcd69fbc-df5m8" event={"ID":"3959f4ca-7868-4862-b576-b679546ac4ad","Type":"ContainerStarted","Data":"c05fdae1f3791b9e041eff15e363a09c11a9d6eee4e09f7c8173cbb0d701434a"} Jul 07 11:05:54.158130 ip-10-0-135-34 kubenswrapper[2578]: I0707 11:05:54.158081 2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-a-1273de/rate-test-model-1273de-kserve-57bcd69fbc-df5m8" podStartSLOduration=1.158064892 podStartE2EDuration="1.158064892s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-07 11:05:53 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 11:05:54.156388735 +0000 UTC m=+2322.828348280" watchObservedRunningTime="2026-07-07 11:05:54.158064892 +0000 UTC m=+2322.830024435" Jul 07 11:06:01.736165 ip-10-0-135-34 kubenswrapper[2578]: I0707 11:06:01.736095 2578 kuberuntime_container.go:872] "Killing container with a grace period" pod="ai-tenant-e2e-shared-a-1273de/sub-test-model-1273de-kserve-687cd89fbc-fd9tf" podUID="1a1fb38c-a193-47b3-a8de-66f6228475af" containerName="main" containerID="cri-o://31405dabcc8e48e775365ca14a5f9333beffc5aa4b74e1f5add6205fef22e9fe" gracePeriod=45 Jul 07 11:06:01.985822 ip-10-0-135-34 kubenswrapper[2578]: I0707 11:06:01.985799 2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-1273de/sub-test-model-1273de-kserve-687cd89fbc-fd9tf" Jul 07 11:06:02.104763 ip-10-0-135-34 kubenswrapper[2578]: I0707 11:06:02.104735 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-t7fw6\" (UniqueName: \"kubernetes.io/projected/1a1fb38c-a193-47b3-a8de-66f6228475af-kube-api-access-t7fw6\") pod \"1a1fb38c-a193-47b3-a8de-66f6228475af\" (UID: \"1a1fb38c-a193-47b3-a8de-66f6228475af\") " Jul 07 11:06:02.104949 ip-10-0-135-34 kubenswrapper[2578]: I0707 11:06:02.104811 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1a1fb38c-a193-47b3-a8de-66f6228475af-tmp-dir\") pod \"1a1fb38c-a193-47b3-a8de-66f6228475af\" (UID: \"1a1fb38c-a193-47b3-a8de-66f6228475af\") " Jul 07 11:06:02.104949 ip-10-0-135-34 kubenswrapper[2578]: I0707 11:06:02.104856 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1a1fb38c-a193-47b3-a8de-66f6228475af-dshm\") pod \"1a1fb38c-a193-47b3-a8de-66f6228475af\" (UID: \"1a1fb38c-a193-47b3-a8de-66f6228475af\") " Jul 07 11:06:02.104949 ip-10-0-135-34 kubenswrapper[2578]: I0707 11:06:02.104899 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1a1fb38c-a193-47b3-a8de-66f6228475af-home\") pod \"1a1fb38c-a193-47b3-a8de-66f6228475af\" (UID: \"1a1fb38c-a193-47b3-a8de-66f6228475af\") " Jul 07 11:06:02.105199 ip-10-0-135-34 kubenswrapper[2578]: I0707 11:06:02.104947 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1a1fb38c-a193-47b3-a8de-66f6228475af-tls-certs\") pod \"1a1fb38c-a193-47b3-a8de-66f6228475af\" (UID: \"1a1fb38c-a193-47b3-a8de-66f6228475af\") " Jul 07 11:06:02.105199 ip-10-0-135-34 kubenswrapper[2578]: I0707 11:06:02.105000 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1a1fb38c-a193-47b3-a8de-66f6228475af-model-cache\") pod \"1a1fb38c-a193-47b3-a8de-66f6228475af\" (UID: \"1a1fb38c-a193-47b3-a8de-66f6228475af\") " Jul 07 11:06:02.105199 ip-10-0-135-34 kubenswrapper[2578]: I0707 11:06:02.105128 2578 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1a1fb38c-a193-47b3-a8de-66f6228475af-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "1a1fb38c-a193-47b3-a8de-66f6228475af" (UID: "1a1fb38c-a193-47b3-a8de-66f6228475af"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 07 11:06:02.105353 ip-10-0-135-34 kubenswrapper[2578]: I0707 11:06:02.105228 2578 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1a1fb38c-a193-47b3-a8de-66f6228475af-home" (OuterVolumeSpecName: "home") pod "1a1fb38c-a193-47b3-a8de-66f6228475af" (UID: "1a1fb38c-a193-47b3-a8de-66f6228475af"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 07 11:06:02.105353 ip-10-0-135-34 kubenswrapper[2578]: I0707 11:06:02.105262 2578 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1a1fb38c-a193-47b3-a8de-66f6228475af-tmp-dir\") on node \"ip-10-0-135-34.ec2.internal\" DevicePath \"\"" Jul 07 11:06:02.105437 ip-10-0-135-34 kubenswrapper[2578]: I0707 11:06:02.105388 2578 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1a1fb38c-a193-47b3-a8de-66f6228475af-model-cache" (OuterVolumeSpecName: "model-cache") pod "1a1fb38c-a193-47b3-a8de-66f6228475af" (UID: "1a1fb38c-a193-47b3-a8de-66f6228475af"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 07 11:06:02.107123 ip-10-0-135-34 kubenswrapper[2578]: I0707 11:06:02.107096 2578 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1a1fb38c-a193-47b3-a8de-66f6228475af-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "1a1fb38c-a193-47b3-a8de-66f6228475af" (UID: "1a1fb38c-a193-47b3-a8de-66f6228475af"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 07 11:06:02.107226 ip-10-0-135-34 kubenswrapper[2578]: I0707 11:06:02.107100 2578 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1a1fb38c-a193-47b3-a8de-66f6228475af-kube-api-access-t7fw6" (OuterVolumeSpecName: "kube-api-access-t7fw6") pod "1a1fb38c-a193-47b3-a8de-66f6228475af" (UID: "1a1fb38c-a193-47b3-a8de-66f6228475af"). InnerVolumeSpecName "kube-api-access-t7fw6". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 07 11:06:02.107226 ip-10-0-135-34 kubenswrapper[2578]: I0707 11:06:02.107136 2578 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1a1fb38c-a193-47b3-a8de-66f6228475af-dshm" (OuterVolumeSpecName: "dshm") pod "1a1fb38c-a193-47b3-a8de-66f6228475af" (UID: "1a1fb38c-a193-47b3-a8de-66f6228475af"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 07 11:06:02.182155 ip-10-0-135-34 kubenswrapper[2578]: I0707 11:06:02.182117 2578 generic.go:356] "Generic (PLEG): container finished" podID="1a1fb38c-a193-47b3-a8de-66f6228475af" containerID="31405dabcc8e48e775365ca14a5f9333beffc5aa4b74e1f5add6205fef22e9fe" exitCode=0 Jul 07 11:06:02.182155 ip-10-0-135-34 kubenswrapper[2578]: I0707 11:06:02.182158 2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-1273de/sub-test-model-1273de-kserve-687cd89fbc-fd9tf" Jul 07 11:06:02.182359 ip-10-0-135-34 kubenswrapper[2578]: I0707 11:06:02.182203 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-1273de/sub-test-model-1273de-kserve-687cd89fbc-fd9tf" event={"ID":"1a1fb38c-a193-47b3-a8de-66f6228475af","Type":"ContainerDied","Data":"31405dabcc8e48e775365ca14a5f9333beffc5aa4b74e1f5add6205fef22e9fe"} Jul 07 11:06:02.182359 ip-10-0-135-34 kubenswrapper[2578]: I0707 11:06:02.182243 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-1273de/sub-test-model-1273de-kserve-687cd89fbc-fd9tf" event={"ID":"1a1fb38c-a193-47b3-a8de-66f6228475af","Type":"ContainerDied","Data":"71ccfc40fd345e6fe21933ebcb4296b7b1f5fa006a71e2ee3c760f81aad1ca94"} Jul 07 11:06:02.182359 ip-10-0-135-34 kubenswrapper[2578]: I0707 11:06:02.182260 2578 scope.go:117] "RemoveContainer" containerID="31405dabcc8e48e775365ca14a5f9333beffc5aa4b74e1f5add6205fef22e9fe" Jul 07 11:06:02.191726 ip-10-0-135-34 kubenswrapper[2578]: I0707 11:06:02.191709 2578 scope.go:117] "RemoveContainer" containerID="31405dabcc8e48e775365ca14a5f9333beffc5aa4b74e1f5add6205fef22e9fe" Jul 07 11:06:02.192099 ip-10-0-135-34 kubenswrapper[2578]: E0707 11:06:02.192054 2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"31405dabcc8e48e775365ca14a5f9333beffc5aa4b74e1f5add6205fef22e9fe\": container with ID starting with 31405dabcc8e48e775365ca14a5f9333beffc5aa4b74e1f5add6205fef22e9fe not found: ID does not exist" containerID="31405dabcc8e48e775365ca14a5f9333beffc5aa4b74e1f5add6205fef22e9fe" Jul 07 11:06:02.192099 ip-10-0-135-34 kubenswrapper[2578]: I0707 11:06:02.192081 2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"31405dabcc8e48e775365ca14a5f9333beffc5aa4b74e1f5add6205fef22e9fe"} err="failed to get container status \"31405dabcc8e48e775365ca14a5f9333beffc5aa4b74e1f5add6205fef22e9fe\": rpc error: code = NotFound desc = could not find container \"31405dabcc8e48e775365ca14a5f9333beffc5aa4b74e1f5add6205fef22e9fe\": container with ID starting with 31405dabcc8e48e775365ca14a5f9333beffc5aa4b74e1f5add6205fef22e9fe not found: ID does not exist" Jul 07 11:06:02.206537 ip-10-0-135-34 kubenswrapper[2578]: I0707 11:06:02.206513 2578 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1a1fb38c-a193-47b3-a8de-66f6228475af-tls-certs\") on node \"ip-10-0-135-34.ec2.internal\" DevicePath \"\"" Jul 07 11:06:02.206537 ip-10-0-135-34 kubenswrapper[2578]: I0707 11:06:02.206535 2578 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1a1fb38c-a193-47b3-a8de-66f6228475af-model-cache\") on node \"ip-10-0-135-34.ec2.internal\" DevicePath \"\"" Jul 07 11:06:02.206658 ip-10-0-135-34 kubenswrapper[2578]: I0707 11:06:02.206545 2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-t7fw6\" (UniqueName: \"kubernetes.io/projected/1a1fb38c-a193-47b3-a8de-66f6228475af-kube-api-access-t7fw6\") on node \"ip-10-0-135-34.ec2.internal\" DevicePath \"\"" Jul 07 11:06:02.206658 ip-10-0-135-34 kubenswrapper[2578]: I0707 11:06:02.206554 2578 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1a1fb38c-a193-47b3-a8de-66f6228475af-dshm\") on node \"ip-10-0-135-34.ec2.internal\" DevicePath \"\"" Jul 07 11:06:02.206658 ip-10-0-135-34 kubenswrapper[2578]: I0707 11:06:02.206564 2578 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1a1fb38c-a193-47b3-a8de-66f6228475af-home\") on node \"ip-10-0-135-34.ec2.internal\" DevicePath \"\"" Jul 07 11:06:02.210113 ip-10-0-135-34 kubenswrapper[2578]: I0707 11:06:02.210088 2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-1273de/sub-test-model-1273de-kserve-687cd89fbc-fd9tf"] Jul 07 11:06:02.216010 ip-10-0-135-34 kubenswrapper[2578]: I0707 11:06:02.215987 2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-a-1273de/sub-test-model-1273de-kserve-687cd89fbc-fd9tf"] Jul 07 11:06:02.624921 ip-10-0-135-34 kubenswrapper[2578]: I0707 11:06:02.624887 2578 kuberuntime_container.go:872] "Killing container with a grace period" pod="ai-tenant-e2e-shared-b-2a3633/sub-test-model-2a3633-kserve-684c78dc-xm9dl" podUID="d8380902-3ad7-4421-9c0f-9bcedb5f848c" containerName="main" containerID="cri-o://5c1936e15a5e2ad5346a266dd82d2988a1a5042e79fa4e2d15e483af2c5314c3" gracePeriod=45 Jul 07 11:06:02.877240 ip-10-0-135-34 kubenswrapper[2578]: I0707 11:06:02.877176 2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-2a3633/sub-test-model-2a3633-kserve-684c78dc-xm9dl" Jul 07 11:06:03.013613 ip-10-0-135-34 kubenswrapper[2578]: I0707 11:06:03.013575 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-vzrtm\" (UniqueName: \"kubernetes.io/projected/d8380902-3ad7-4421-9c0f-9bcedb5f848c-kube-api-access-vzrtm\") pod \"d8380902-3ad7-4421-9c0f-9bcedb5f848c\" (UID: \"d8380902-3ad7-4421-9c0f-9bcedb5f848c\") " Jul 07 11:06:03.013810 ip-10-0-135-34 kubenswrapper[2578]: I0707 11:06:03.013635 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d8380902-3ad7-4421-9c0f-9bcedb5f848c-tls-certs\") pod \"d8380902-3ad7-4421-9c0f-9bcedb5f848c\" (UID: \"d8380902-3ad7-4421-9c0f-9bcedb5f848c\") " Jul 07 11:06:03.013810 ip-10-0-135-34 kubenswrapper[2578]: I0707 11:06:03.013675 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d8380902-3ad7-4421-9c0f-9bcedb5f848c-model-cache\") pod \"d8380902-3ad7-4421-9c0f-9bcedb5f848c\" (UID: \"d8380902-3ad7-4421-9c0f-9bcedb5f848c\") " Jul 07 11:06:03.013810 ip-10-0-135-34 kubenswrapper[2578]: I0707 11:06:03.013755 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d8380902-3ad7-4421-9c0f-9bcedb5f848c-home\") pod \"d8380902-3ad7-4421-9c0f-9bcedb5f848c\" (UID: \"d8380902-3ad7-4421-9c0f-9bcedb5f848c\") " Jul 07 11:06:03.014000 ip-10-0-135-34 kubenswrapper[2578]: I0707 11:06:03.013818 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d8380902-3ad7-4421-9c0f-9bcedb5f848c-dshm\") pod \"d8380902-3ad7-4421-9c0f-9bcedb5f848c\" (UID: \"d8380902-3ad7-4421-9c0f-9bcedb5f848c\") " Jul 07 11:06:03.014000 ip-10-0-135-34 kubenswrapper[2578]: I0707 11:06:03.013845 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d8380902-3ad7-4421-9c0f-9bcedb5f848c-tmp-dir\") pod \"d8380902-3ad7-4421-9c0f-9bcedb5f848c\" (UID: \"d8380902-3ad7-4421-9c0f-9bcedb5f848c\") " Jul 07 11:06:03.014098 ip-10-0-135-34 kubenswrapper[2578]: I0707 11:06:03.014018 2578 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d8380902-3ad7-4421-9c0f-9bcedb5f848c-model-cache" (OuterVolumeSpecName: "model-cache") pod "d8380902-3ad7-4421-9c0f-9bcedb5f848c" (UID: "d8380902-3ad7-4421-9c0f-9bcedb5f848c"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 07 11:06:03.014157 ip-10-0-135-34 kubenswrapper[2578]: I0707 11:06:03.014120 2578 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d8380902-3ad7-4421-9c0f-9bcedb5f848c-home" (OuterVolumeSpecName: "home") pod "d8380902-3ad7-4421-9c0f-9bcedb5f848c" (UID: "d8380902-3ad7-4421-9c0f-9bcedb5f848c"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 07 11:06:03.014250 ip-10-0-135-34 kubenswrapper[2578]: I0707 11:06:03.014229 2578 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d8380902-3ad7-4421-9c0f-9bcedb5f848c-model-cache\") on node \"ip-10-0-135-34.ec2.internal\" DevicePath \"\"" Jul 07 11:06:03.014250 ip-10-0-135-34 kubenswrapper[2578]: I0707 11:06:03.014253 2578 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d8380902-3ad7-4421-9c0f-9bcedb5f848c-home\") on node \"ip-10-0-135-34.ec2.internal\" DevicePath \"\"" Jul 07 11:06:03.014393 ip-10-0-135-34 kubenswrapper[2578]: I0707 11:06:03.014261 2578 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d8380902-3ad7-4421-9c0f-9bcedb5f848c-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "d8380902-3ad7-4421-9c0f-9bcedb5f848c" (UID: "d8380902-3ad7-4421-9c0f-9bcedb5f848c"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 07 11:06:03.015990 ip-10-0-135-34 kubenswrapper[2578]: I0707 11:06:03.015970 2578 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d8380902-3ad7-4421-9c0f-9bcedb5f848c-kube-api-access-vzrtm" (OuterVolumeSpecName: "kube-api-access-vzrtm") pod "d8380902-3ad7-4421-9c0f-9bcedb5f848c" (UID: "d8380902-3ad7-4421-9c0f-9bcedb5f848c"). InnerVolumeSpecName "kube-api-access-vzrtm". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 07 11:06:03.016458 ip-10-0-135-34 kubenswrapper[2578]: I0707 11:06:03.016438 2578 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d8380902-3ad7-4421-9c0f-9bcedb5f848c-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "d8380902-3ad7-4421-9c0f-9bcedb5f848c" (UID: "d8380902-3ad7-4421-9c0f-9bcedb5f848c"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 07 11:06:03.016532 ip-10-0-135-34 kubenswrapper[2578]: I0707 11:06:03.016457 2578 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d8380902-3ad7-4421-9c0f-9bcedb5f848c-dshm" (OuterVolumeSpecName: "dshm") pod "d8380902-3ad7-4421-9c0f-9bcedb5f848c" (UID: "d8380902-3ad7-4421-9c0f-9bcedb5f848c"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 07 11:06:03.115177 ip-10-0-135-34 kubenswrapper[2578]: I0707 11:06:03.115141 2578 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d8380902-3ad7-4421-9c0f-9bcedb5f848c-dshm\") on node \"ip-10-0-135-34.ec2.internal\" DevicePath \"\"" Jul 07 11:06:03.115177 ip-10-0-135-34 kubenswrapper[2578]: I0707 11:06:03.115173 2578 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d8380902-3ad7-4421-9c0f-9bcedb5f848c-tmp-dir\") on node \"ip-10-0-135-34.ec2.internal\" DevicePath \"\"" Jul 07 11:06:03.115177 ip-10-0-135-34 kubenswrapper[2578]: I0707 11:06:03.115183 2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vzrtm\" (UniqueName: \"kubernetes.io/projected/d8380902-3ad7-4421-9c0f-9bcedb5f848c-kube-api-access-vzrtm\") on node \"ip-10-0-135-34.ec2.internal\" DevicePath \"\"" Jul 07 11:06:03.115408 ip-10-0-135-34 kubenswrapper[2578]: I0707 11:06:03.115192 2578 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d8380902-3ad7-4421-9c0f-9bcedb5f848c-tls-certs\") on node \"ip-10-0-135-34.ec2.internal\" DevicePath \"\"" Jul 07 11:06:03.195133 ip-10-0-135-34 kubenswrapper[2578]: I0707 11:06:03.195046 2578 generic.go:356] "Generic (PLEG): container finished" podID="d8380902-3ad7-4421-9c0f-9bcedb5f848c" containerID="5c1936e15a5e2ad5346a266dd82d2988a1a5042e79fa4e2d15e483af2c5314c3" exitCode=0 Jul 07 11:06:03.195133 ip-10-0-135-34 kubenswrapper[2578]: I0707 11:06:03.195102 2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-2a3633/sub-test-model-2a3633-kserve-684c78dc-xm9dl" Jul 07 11:06:03.195133 ip-10-0-135-34 kubenswrapper[2578]: I0707 11:06:03.195114 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-2a3633/sub-test-model-2a3633-kserve-684c78dc-xm9dl" event={"ID":"d8380902-3ad7-4421-9c0f-9bcedb5f848c","Type":"ContainerDied","Data":"5c1936e15a5e2ad5346a266dd82d2988a1a5042e79fa4e2d15e483af2c5314c3"} Jul 07 11:06:03.195404 ip-10-0-135-34 kubenswrapper[2578]: I0707 11:06:03.195152 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-2a3633/sub-test-model-2a3633-kserve-684c78dc-xm9dl" event={"ID":"d8380902-3ad7-4421-9c0f-9bcedb5f848c","Type":"ContainerDied","Data":"78350307f00ba57c587900428008ef3ecf0457a2976a89e7f6050bf89aee06db"} Jul 07 11:06:03.195404 ip-10-0-135-34 kubenswrapper[2578]: I0707 11:06:03.195172 2578 scope.go:117] "RemoveContainer" containerID="5c1936e15a5e2ad5346a266dd82d2988a1a5042e79fa4e2d15e483af2c5314c3" Jul 07 11:06:03.204639 ip-10-0-135-34 kubenswrapper[2578]: I0707 11:06:03.204623 2578 scope.go:117] "RemoveContainer" containerID="5c1936e15a5e2ad5346a266dd82d2988a1a5042e79fa4e2d15e483af2c5314c3" Jul 07 11:06:03.204943 ip-10-0-135-34 kubenswrapper[2578]: E0707 11:06:03.204922 2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5c1936e15a5e2ad5346a266dd82d2988a1a5042e79fa4e2d15e483af2c5314c3\": container with ID starting with 5c1936e15a5e2ad5346a266dd82d2988a1a5042e79fa4e2d15e483af2c5314c3 not found: ID does not exist" containerID="5c1936e15a5e2ad5346a266dd82d2988a1a5042e79fa4e2d15e483af2c5314c3" Jul 07 11:06:03.205004 ip-10-0-135-34 kubenswrapper[2578]: I0707 11:06:03.204950 2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5c1936e15a5e2ad5346a266dd82d2988a1a5042e79fa4e2d15e483af2c5314c3"} err="failed to get container status \"5c1936e15a5e2ad5346a266dd82d2988a1a5042e79fa4e2d15e483af2c5314c3\": rpc error: code = NotFound desc = could not find container \"5c1936e15a5e2ad5346a266dd82d2988a1a5042e79fa4e2d15e483af2c5314c3\": container with ID starting with 5c1936e15a5e2ad5346a266dd82d2988a1a5042e79fa4e2d15e483af2c5314c3 not found: ID does not exist" Jul 07 11:06:03.219290 ip-10-0-135-34 kubenswrapper[2578]: I0707 11:06:03.219257 2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-2a3633/sub-test-model-2a3633-kserve-684c78dc-xm9dl"] Jul 07 11:06:03.228161 ip-10-0-135-34 kubenswrapper[2578]: I0707 11:06:03.228128 2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-b-2a3633/sub-test-model-2a3633-kserve-684c78dc-xm9dl"] Jul 07 11:06:03.651987 ip-10-0-135-34 kubenswrapper[2578]: I0707 11:06:03.651951 2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-a-1273de/rate-test-model-1273de-kserve-57bcd69fbc-df5m8" Jul 07 11:06:03.652172 ip-10-0-135-34 kubenswrapper[2578]: I0707 11:06:03.652000 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-a-1273de/rate-test-model-1273de-kserve-57bcd69fbc-df5m8" Jul 07 11:06:03.664620 ip-10-0-135-34 kubenswrapper[2578]: I0707 11:06:03.664587 2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-a-1273de/rate-test-model-1273de-kserve-57bcd69fbc-df5m8" Jul 07 11:06:03.821921 ip-10-0-135-34 kubenswrapper[2578]: I0707 11:06:03.821891 2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="1a1fb38c-a193-47b3-a8de-66f6228475af" path="/var/lib/kubelet/pods/1a1fb38c-a193-47b3-a8de-66f6228475af/volumes" Jul 07 11:06:03.822260 ip-10-0-135-34 kubenswrapper[2578]: I0707 11:06:03.822246 2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="d8380902-3ad7-4421-9c0f-9bcedb5f848c" path="/var/lib/kubelet/pods/d8380902-3ad7-4421-9c0f-9bcedb5f848c/volumes" Jul 07 11:06:04.213388 ip-10-0-135-34 kubenswrapper[2578]: I0707 11:06:04.213358 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-a-1273de/rate-test-model-1273de-kserve-57bcd69fbc-df5m8" Jul 07 11:06:05.838545 ip-10-0-135-34 kubenswrapper[2578]: I0707 11:06:05.838512 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-lv7nd"] Jul 07 11:06:19.883357 ip-10-0-135-34 kubenswrapper[2578]: I0707 11:06:19.883317 2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-b-2a3633/rate-test-model-2a3633-kserve-5cdd4c4f6c-wrhl9"] Jul 07 11:06:19.883847 ip-10-0-135-34 kubenswrapper[2578]: I0707 11:06:19.883690 2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1a1fb38c-a193-47b3-a8de-66f6228475af" containerName="main" Jul 07 11:06:19.883847 ip-10-0-135-34 kubenswrapper[2578]: I0707 11:06:19.883702 2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="1a1fb38c-a193-47b3-a8de-66f6228475af" containerName="main" Jul 07 11:06:19.883847 ip-10-0-135-34 kubenswrapper[2578]: I0707 11:06:19.883713 2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d8380902-3ad7-4421-9c0f-9bcedb5f848c" containerName="main" Jul 07 11:06:19.883847 ip-10-0-135-34 kubenswrapper[2578]: I0707 11:06:19.883719 2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="d8380902-3ad7-4421-9c0f-9bcedb5f848c" containerName="main" Jul 07 11:06:19.883847 ip-10-0-135-34 kubenswrapper[2578]: I0707 11:06:19.883795 2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="1a1fb38c-a193-47b3-a8de-66f6228475af" containerName="main" Jul 07 11:06:19.883847 ip-10-0-135-34 kubenswrapper[2578]: I0707 11:06:19.883812 2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="d8380902-3ad7-4421-9c0f-9bcedb5f848c" containerName="main" Jul 07 11:06:19.886510 ip-10-0-135-34 kubenswrapper[2578]: I0707 11:06:19.886494 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-2a3633/rate-test-model-2a3633-kserve-5cdd4c4f6c-wrhl9" Jul 07 11:06:19.889697 ip-10-0-135-34 kubenswrapper[2578]: I0707 11:06:19.889678 2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-2a3633\"/\"openshift-service-ca.crt\"" Jul 07 11:06:19.890168 ip-10-0-135-34 kubenswrapper[2578]: I0707 11:06:19.890154 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-2a3633\"/\"default-dockercfg-nq96c\"" Jul 07 11:06:19.891028 ip-10-0-135-34 kubenswrapper[2578]: I0707 11:06:19.891012 2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-2a3633\"/\"kube-root-ca.crt\"" Jul 07 11:06:19.891491 ip-10-0-135-34 kubenswrapper[2578]: I0707 11:06:19.891474 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-2a3633\"/\"rate-test-model-2a3633-kserve-self-signed-certs\"" Jul 07 11:06:19.898812 ip-10-0-135-34 kubenswrapper[2578]: I0707 11:06:19.898789 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-2a3633/rate-test-model-2a3633-kserve-5cdd4c4f6c-wrhl9"] Jul 07 11:06:20.072147 ip-10-0-135-34 kubenswrapper[2578]: I0707 11:06:20.072102 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7msq9\" (UniqueName: \"kubernetes.io/projected/188f6f12-3862-4c8e-baef-ed84674490cc-kube-api-access-7msq9\") pod \"rate-test-model-2a3633-kserve-5cdd4c4f6c-wrhl9\" (UID: \"188f6f12-3862-4c8e-baef-ed84674490cc\") " pod="ai-tenant-e2e-shared-b-2a3633/rate-test-model-2a3633-kserve-5cdd4c4f6c-wrhl9" Jul 07 11:06:20.072364 ip-10-0-135-34 kubenswrapper[2578]: I0707 11:06:20.072158 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/188f6f12-3862-4c8e-baef-ed84674490cc-tls-certs\") pod \"rate-test-model-2a3633-kserve-5cdd4c4f6c-wrhl9\" (UID: \"188f6f12-3862-4c8e-baef-ed84674490cc\") " pod="ai-tenant-e2e-shared-b-2a3633/rate-test-model-2a3633-kserve-5cdd4c4f6c-wrhl9" Jul 07 11:06:20.072364 ip-10-0-135-34 kubenswrapper[2578]: I0707 11:06:20.072297 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/188f6f12-3862-4c8e-baef-ed84674490cc-dshm\") pod \"rate-test-model-2a3633-kserve-5cdd4c4f6c-wrhl9\" (UID: \"188f6f12-3862-4c8e-baef-ed84674490cc\") " pod="ai-tenant-e2e-shared-b-2a3633/rate-test-model-2a3633-kserve-5cdd4c4f6c-wrhl9" Jul 07 11:06:20.072499 ip-10-0-135-34 kubenswrapper[2578]: I0707 11:06:20.072366 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/188f6f12-3862-4c8e-baef-ed84674490cc-home\") pod \"rate-test-model-2a3633-kserve-5cdd4c4f6c-wrhl9\" (UID: \"188f6f12-3862-4c8e-baef-ed84674490cc\") " pod="ai-tenant-e2e-shared-b-2a3633/rate-test-model-2a3633-kserve-5cdd4c4f6c-wrhl9" Jul 07 11:06:20.072499 ip-10-0-135-34 kubenswrapper[2578]: I0707 11:06:20.072394 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/188f6f12-3862-4c8e-baef-ed84674490cc-model-cache\") pod \"rate-test-model-2a3633-kserve-5cdd4c4f6c-wrhl9\" (UID: \"188f6f12-3862-4c8e-baef-ed84674490cc\") " pod="ai-tenant-e2e-shared-b-2a3633/rate-test-model-2a3633-kserve-5cdd4c4f6c-wrhl9" Jul 07 11:06:20.072499 ip-10-0-135-34 kubenswrapper[2578]: I0707 11:06:20.072418 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/188f6f12-3862-4c8e-baef-ed84674490cc-tmp-dir\") pod \"rate-test-model-2a3633-kserve-5cdd4c4f6c-wrhl9\" (UID: \"188f6f12-3862-4c8e-baef-ed84674490cc\") " pod="ai-tenant-e2e-shared-b-2a3633/rate-test-model-2a3633-kserve-5cdd4c4f6c-wrhl9" Jul 07 11:06:20.173633 ip-10-0-135-34 kubenswrapper[2578]: I0707 11:06:20.173532 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/188f6f12-3862-4c8e-baef-ed84674490cc-dshm\") pod \"rate-test-model-2a3633-kserve-5cdd4c4f6c-wrhl9\" (UID: \"188f6f12-3862-4c8e-baef-ed84674490cc\") " pod="ai-tenant-e2e-shared-b-2a3633/rate-test-model-2a3633-kserve-5cdd4c4f6c-wrhl9" Jul 07 11:06:20.173633 ip-10-0-135-34 kubenswrapper[2578]: I0707 11:06:20.173589 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/188f6f12-3862-4c8e-baef-ed84674490cc-home\") pod \"rate-test-model-2a3633-kserve-5cdd4c4f6c-wrhl9\" (UID: \"188f6f12-3862-4c8e-baef-ed84674490cc\") " pod="ai-tenant-e2e-shared-b-2a3633/rate-test-model-2a3633-kserve-5cdd4c4f6c-wrhl9" Jul 07 11:06:20.173912 ip-10-0-135-34 kubenswrapper[2578]: I0707 11:06:20.173735 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/188f6f12-3862-4c8e-baef-ed84674490cc-model-cache\") pod \"rate-test-model-2a3633-kserve-5cdd4c4f6c-wrhl9\" (UID: \"188f6f12-3862-4c8e-baef-ed84674490cc\") " pod="ai-tenant-e2e-shared-b-2a3633/rate-test-model-2a3633-kserve-5cdd4c4f6c-wrhl9" Jul 07 11:06:20.173912 ip-10-0-135-34 kubenswrapper[2578]: I0707 11:06:20.173765 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/188f6f12-3862-4c8e-baef-ed84674490cc-tmp-dir\") pod \"rate-test-model-2a3633-kserve-5cdd4c4f6c-wrhl9\" (UID: \"188f6f12-3862-4c8e-baef-ed84674490cc\") " pod="ai-tenant-e2e-shared-b-2a3633/rate-test-model-2a3633-kserve-5cdd4c4f6c-wrhl9" Jul 07 11:06:20.173912 ip-10-0-135-34 kubenswrapper[2578]: I0707 11:06:20.173829 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7msq9\" (UniqueName: \"kubernetes.io/projected/188f6f12-3862-4c8e-baef-ed84674490cc-kube-api-access-7msq9\") pod \"rate-test-model-2a3633-kserve-5cdd4c4f6c-wrhl9\" (UID: \"188f6f12-3862-4c8e-baef-ed84674490cc\") " pod="ai-tenant-e2e-shared-b-2a3633/rate-test-model-2a3633-kserve-5cdd4c4f6c-wrhl9" Jul 07 11:06:20.173912 ip-10-0-135-34 kubenswrapper[2578]: I0707 11:06:20.173876 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/188f6f12-3862-4c8e-baef-ed84674490cc-tls-certs\") pod \"rate-test-model-2a3633-kserve-5cdd4c4f6c-wrhl9\" (UID: \"188f6f12-3862-4c8e-baef-ed84674490cc\") " pod="ai-tenant-e2e-shared-b-2a3633/rate-test-model-2a3633-kserve-5cdd4c4f6c-wrhl9" Jul 07 11:06:20.174140 ip-10-0-135-34 kubenswrapper[2578]: I0707 11:06:20.174000 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/188f6f12-3862-4c8e-baef-ed84674490cc-home\") pod \"rate-test-model-2a3633-kserve-5cdd4c4f6c-wrhl9\" (UID: \"188f6f12-3862-4c8e-baef-ed84674490cc\") " pod="ai-tenant-e2e-shared-b-2a3633/rate-test-model-2a3633-kserve-5cdd4c4f6c-wrhl9" Jul 07 11:06:20.174140 ip-10-0-135-34 kubenswrapper[2578]: I0707 11:06:20.174089 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/188f6f12-3862-4c8e-baef-ed84674490cc-tmp-dir\") pod \"rate-test-model-2a3633-kserve-5cdd4c4f6c-wrhl9\" (UID: \"188f6f12-3862-4c8e-baef-ed84674490cc\") " pod="ai-tenant-e2e-shared-b-2a3633/rate-test-model-2a3633-kserve-5cdd4c4f6c-wrhl9" Jul 07 11:06:20.174238 ip-10-0-135-34 kubenswrapper[2578]: I0707 11:06:20.174139 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/188f6f12-3862-4c8e-baef-ed84674490cc-model-cache\") pod \"rate-test-model-2a3633-kserve-5cdd4c4f6c-wrhl9\" (UID: \"188f6f12-3862-4c8e-baef-ed84674490cc\") " pod="ai-tenant-e2e-shared-b-2a3633/rate-test-model-2a3633-kserve-5cdd4c4f6c-wrhl9" Jul 07 11:06:20.176075 ip-10-0-135-34 kubenswrapper[2578]: I0707 11:06:20.176049 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/188f6f12-3862-4c8e-baef-ed84674490cc-dshm\") pod \"rate-test-model-2a3633-kserve-5cdd4c4f6c-wrhl9\" (UID: \"188f6f12-3862-4c8e-baef-ed84674490cc\") " pod="ai-tenant-e2e-shared-b-2a3633/rate-test-model-2a3633-kserve-5cdd4c4f6c-wrhl9" Jul 07 11:06:20.176407 ip-10-0-135-34 kubenswrapper[2578]: I0707 11:06:20.176391 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/188f6f12-3862-4c8e-baef-ed84674490cc-tls-certs\") pod \"rate-test-model-2a3633-kserve-5cdd4c4f6c-wrhl9\" (UID: \"188f6f12-3862-4c8e-baef-ed84674490cc\") " pod="ai-tenant-e2e-shared-b-2a3633/rate-test-model-2a3633-kserve-5cdd4c4f6c-wrhl9" Jul 07 11:06:20.183306 ip-10-0-135-34 kubenswrapper[2578]: I0707 11:06:20.183281 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7msq9\" (UniqueName: \"kubernetes.io/projected/188f6f12-3862-4c8e-baef-ed84674490cc-kube-api-access-7msq9\") pod \"rate-test-model-2a3633-kserve-5cdd4c4f6c-wrhl9\" (UID: \"188f6f12-3862-4c8e-baef-ed84674490cc\") " pod="ai-tenant-e2e-shared-b-2a3633/rate-test-model-2a3633-kserve-5cdd4c4f6c-wrhl9" Jul 07 11:06:20.195933 ip-10-0-135-34 kubenswrapper[2578]: I0707 11:06:20.195896 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-2a3633/rate-test-model-2a3633-kserve-5cdd4c4f6c-wrhl9" Jul 07 11:06:20.332884 ip-10-0-135-34 kubenswrapper[2578]: I0707 11:06:20.332856 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-2a3633/rate-test-model-2a3633-kserve-5cdd4c4f6c-wrhl9"] Jul 07 11:06:20.334515 ip-10-0-135-34 kubenswrapper[2578]: W0707 11:06:20.334489 2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod188f6f12_3862_4c8e_baef_ed84674490cc.slice/crio-70512a2742fc99bfb686c7632f650c9f1d77d0bb6e73bc2f763b2e799d5530f2 WatchSource:0}: Error finding container 70512a2742fc99bfb686c7632f650c9f1d77d0bb6e73bc2f763b2e799d5530f2: Status 404 returned error can't find the container with id 70512a2742fc99bfb686c7632f650c9f1d77d0bb6e73bc2f763b2e799d5530f2 Jul 07 11:06:21.287709 ip-10-0-135-34 kubenswrapper[2578]: I0707 11:06:21.287666 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-2a3633/rate-test-model-2a3633-kserve-5cdd4c4f6c-wrhl9" event={"ID":"188f6f12-3862-4c8e-baef-ed84674490cc","Type":"ContainerStarted","Data":"d5b74e2004a5c6472d297c33c34caa741fafaba06d76a900e1982be6e406107e"} Jul 07 11:06:21.288128 ip-10-0-135-34 kubenswrapper[2578]: I0707 11:06:21.287721 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-2a3633/rate-test-model-2a3633-kserve-5cdd4c4f6c-wrhl9" event={"ID":"188f6f12-3862-4c8e-baef-ed84674490cc","Type":"ContainerStarted","Data":"70512a2742fc99bfb686c7632f650c9f1d77d0bb6e73bc2f763b2e799d5530f2"} Jul 07 11:06:21.308394 ip-10-0-135-34 kubenswrapper[2578]: I0707 11:06:21.308337 2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-b-2a3633/rate-test-model-2a3633-kserve-5cdd4c4f6c-wrhl9" podStartSLOduration=2.308321373 podStartE2EDuration="2.308321373s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-07 11:06:19 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 11:06:21.306501002 +0000 UTC m=+2349.978460546" watchObservedRunningTime="2026-07-07 11:06:21.308321373 +0000 UTC m=+2349.980281108" Jul 07 11:06:23.726406 ip-10-0-135-34 kubenswrapper[2578]: I0707 11:06:23.726368 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-lv7nd"] Jul 07 11:06:30.196758 ip-10-0-135-34 kubenswrapper[2578]: I0707 11:06:30.196702 2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-b-2a3633/rate-test-model-2a3633-kserve-5cdd4c4f6c-wrhl9" Jul 07 11:06:30.196758 ip-10-0-135-34 kubenswrapper[2578]: I0707 11:06:30.196762 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-b-2a3633/rate-test-model-2a3633-kserve-5cdd4c4f6c-wrhl9" Jul 07 11:06:30.209167 ip-10-0-135-34 kubenswrapper[2578]: I0707 11:06:30.209136 2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-b-2a3633/rate-test-model-2a3633-kserve-5cdd4c4f6c-wrhl9" Jul 07 11:06:30.341938 ip-10-0-135-34 kubenswrapper[2578]: I0707 11:06:30.341903 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-b-2a3633/rate-test-model-2a3633-kserve-5cdd4c4f6c-wrhl9" Jul 07 11:06:47.122433 ip-10-0-135-34 kubenswrapper[2578]: I0707 11:06:47.122395 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-lv7nd"] Jul 07 11:07:00.331839 ip-10-0-135-34 kubenswrapper[2578]: I0707 11:07:00.331799 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-lv7nd"] Jul 07 11:07:06.329382 ip-10-0-135-34 kubenswrapper[2578]: I0707 11:07:06.329344 2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-1273de/rate-test-model-1273de-kserve-57bcd69fbc-df5m8"] Jul 07 11:07:07.358728 ip-10-0-135-34 kubenswrapper[2578]: I0707 11:07:07.358695 2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-2a3633/rate-test-model-2a3633-kserve-5cdd4c4f6c-wrhl9"] Jul 07 11:07:09.151278 ip-10-0-135-34 kubenswrapper[2578]: I0707 11:07:09.151248 2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-api-e2e-shared-a-1273de-7f7fb4697-sdqlv"] Jul 07 11:07:09.151715 ip-10-0-135-34 kubenswrapper[2578]: I0707 11:07:09.151463 2578 kuberuntime_container.go:872] "Killing container with a grace period" pod="opendatahub/maas-api-e2e-shared-a-1273de-7f7fb4697-sdqlv" podUID="3a107748-57b4-4e96-8684-8bd1232af914" containerName="maas-api" containerID="cri-o://6d95630a7f54775b731250fcc83679765cd2761735aafba99ae5be0c1e053fc3" gracePeriod=30 Jul 07 11:07:09.400223 ip-10-0-135-34 kubenswrapper[2578]: I0707 11:07:09.400199 2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-api-e2e-shared-a-1273de-7f7fb4697-sdqlv" Jul 07 11:07:09.525186 ip-10-0-135-34 kubenswrapper[2578]: I0707 11:07:09.525099 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/3a107748-57b4-4e96-8684-8bd1232af914-openshift-service-ca\") pod \"3a107748-57b4-4e96-8684-8bd1232af914\" (UID: \"3a107748-57b4-4e96-8684-8bd1232af914\") " Jul 07 11:07:09.525186 ip-10-0-135-34 kubenswrapper[2578]: I0707 11:07:09.525185 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-kmcqp\" (UniqueName: \"kubernetes.io/projected/3a107748-57b4-4e96-8684-8bd1232af914-kube-api-access-kmcqp\") pod \"3a107748-57b4-4e96-8684-8bd1232af914\" (UID: \"3a107748-57b4-4e96-8684-8bd1232af914\") " Jul 07 11:07:09.525416 ip-10-0-135-34 kubenswrapper[2578]: I0707 11:07:09.525244 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3a107748-57b4-4e96-8684-8bd1232af914-maas-api-tls\") pod \"3a107748-57b4-4e96-8684-8bd1232af914\" (UID: \"3a107748-57b4-4e96-8684-8bd1232af914\") " Jul 07 11:07:09.525523 ip-10-0-135-34 kubenswrapper[2578]: I0707 11:07:09.525502 2578 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3a107748-57b4-4e96-8684-8bd1232af914-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "3a107748-57b4-4e96-8684-8bd1232af914" (UID: "3a107748-57b4-4e96-8684-8bd1232af914"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 07 11:07:09.527865 ip-10-0-135-34 kubenswrapper[2578]: I0707 11:07:09.527833 2578 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3a107748-57b4-4e96-8684-8bd1232af914-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "3a107748-57b4-4e96-8684-8bd1232af914" (UID: "3a107748-57b4-4e96-8684-8bd1232af914"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 07 11:07:09.531264 ip-10-0-135-34 kubenswrapper[2578]: I0707 11:07:09.531230 2578 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3a107748-57b4-4e96-8684-8bd1232af914-kube-api-access-kmcqp" (OuterVolumeSpecName: "kube-api-access-kmcqp") pod "3a107748-57b4-4e96-8684-8bd1232af914" (UID: "3a107748-57b4-4e96-8684-8bd1232af914"). InnerVolumeSpecName "kube-api-access-kmcqp". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 07 11:07:09.533043 ip-10-0-135-34 kubenswrapper[2578]: I0707 11:07:09.533014 2578 generic.go:356] "Generic (PLEG): container finished" podID="3a107748-57b4-4e96-8684-8bd1232af914" containerID="6d95630a7f54775b731250fcc83679765cd2761735aafba99ae5be0c1e053fc3" exitCode=0 Jul 07 11:07:09.533176 ip-10-0-135-34 kubenswrapper[2578]: I0707 11:07:09.533066 2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-api-e2e-shared-a-1273de-7f7fb4697-sdqlv" Jul 07 11:07:09.533176 ip-10-0-135-34 kubenswrapper[2578]: I0707 11:07:09.533095 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-api-e2e-shared-a-1273de-7f7fb4697-sdqlv" event={"ID":"3a107748-57b4-4e96-8684-8bd1232af914","Type":"ContainerDied","Data":"6d95630a7f54775b731250fcc83679765cd2761735aafba99ae5be0c1e053fc3"} Jul 07 11:07:09.533176 ip-10-0-135-34 kubenswrapper[2578]: I0707 11:07:09.533150 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-api-e2e-shared-a-1273de-7f7fb4697-sdqlv" event={"ID":"3a107748-57b4-4e96-8684-8bd1232af914","Type":"ContainerDied","Data":"3e602563b5647fd71fcf3ef46ae1c2fdacca78ffdcf9f03461d366c4c0a44766"} Jul 07 11:07:09.533176 ip-10-0-135-34 kubenswrapper[2578]: I0707 11:07:09.533174 2578 scope.go:117] "RemoveContainer" containerID="6d95630a7f54775b731250fcc83679765cd2761735aafba99ae5be0c1e053fc3" Jul 07 11:07:09.547344 ip-10-0-135-34 kubenswrapper[2578]: I0707 11:07:09.547324 2578 scope.go:117] "RemoveContainer" containerID="6d95630a7f54775b731250fcc83679765cd2761735aafba99ae5be0c1e053fc3" Jul 07 11:07:09.547663 ip-10-0-135-34 kubenswrapper[2578]: E0707 11:07:09.547641 2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6d95630a7f54775b731250fcc83679765cd2761735aafba99ae5be0c1e053fc3\": container with ID starting with 6d95630a7f54775b731250fcc83679765cd2761735aafba99ae5be0c1e053fc3 not found: ID does not exist" containerID="6d95630a7f54775b731250fcc83679765cd2761735aafba99ae5be0c1e053fc3" Jul 07 11:07:09.547734 ip-10-0-135-34 kubenswrapper[2578]: I0707 11:07:09.547672 2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6d95630a7f54775b731250fcc83679765cd2761735aafba99ae5be0c1e053fc3"} err="failed to get container status \"6d95630a7f54775b731250fcc83679765cd2761735aafba99ae5be0c1e053fc3\": rpc error: code = NotFound desc = could not find container \"6d95630a7f54775b731250fcc83679765cd2761735aafba99ae5be0c1e053fc3\": container with ID starting with 6d95630a7f54775b731250fcc83679765cd2761735aafba99ae5be0c1e053fc3 not found: ID does not exist" Jul 07 11:07:09.568216 ip-10-0-135-34 kubenswrapper[2578]: I0707 11:07:09.568181 2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-api-e2e-shared-a-1273de-7f7fb4697-sdqlv"] Jul 07 11:07:09.574196 ip-10-0-135-34 kubenswrapper[2578]: I0707 11:07:09.574161 2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/maas-api-e2e-shared-a-1273de-7f7fb4697-sdqlv"] Jul 07 11:07:09.626438 ip-10-0-135-34 kubenswrapper[2578]: I0707 11:07:09.626393 2578 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/3a107748-57b4-4e96-8684-8bd1232af914-openshift-service-ca\") on node \"ip-10-0-135-34.ec2.internal\" DevicePath \"\"" Jul 07 11:07:09.626438 ip-10-0-135-34 kubenswrapper[2578]: I0707 11:07:09.626430 2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kmcqp\" (UniqueName: \"kubernetes.io/projected/3a107748-57b4-4e96-8684-8bd1232af914-kube-api-access-kmcqp\") on node \"ip-10-0-135-34.ec2.internal\" DevicePath \"\"" Jul 07 11:07:09.626665 ip-10-0-135-34 kubenswrapper[2578]: I0707 11:07:09.626445 2578 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3a107748-57b4-4e96-8684-8bd1232af914-maas-api-tls\") on node \"ip-10-0-135-34.ec2.internal\" DevicePath \"\"" Jul 07 11:07:09.821641 ip-10-0-135-34 kubenswrapper[2578]: I0707 11:07:09.821563 2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="3a107748-57b4-4e96-8684-8bd1232af914" path="/var/lib/kubelet/pods/3a107748-57b4-4e96-8684-8bd1232af914/volumes" Jul 07 11:07:12.001889 ip-10-0-135-34 kubenswrapper[2578]: I0707 11:07:12.001855 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-qhhw7_aca7528d-ddbe-4282-979f-1b496011f0e1/ovn-acl-logging/0.log" Jul 07 11:07:12.010298 ip-10-0-135-34 kubenswrapper[2578]: I0707 11:07:12.010271 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-qhhw7_aca7528d-ddbe-4282-979f-1b496011f0e1/ovn-acl-logging/0.log" Jul 07 11:07:17.135854 ip-10-0-135-34 kubenswrapper[2578]: I0707 11:07:17.135814 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-lv7nd"] Jul 07 11:07:21.349702 ip-10-0-135-34 kubenswrapper[2578]: I0707 11:07:21.349640 2578 kuberuntime_container.go:872] "Killing container with a grace period" pod="ai-tenant-e2e-shared-a-1273de/rate-test-model-1273de-kserve-57bcd69fbc-df5m8" podUID="3959f4ca-7868-4862-b576-b679546ac4ad" containerName="main" containerID="cri-o://08876ea07fd6fdd679884dcda41b983d6cc1ecc5e20a49c8abe7c4b341cfcda2" gracePeriod=45 Jul 07 11:07:21.598853 ip-10-0-135-34 kubenswrapper[2578]: I0707 11:07:21.598822 2578 generic.go:356] "Generic (PLEG): container finished" podID="3959f4ca-7868-4862-b576-b679546ac4ad" containerID="08876ea07fd6fdd679884dcda41b983d6cc1ecc5e20a49c8abe7c4b341cfcda2" exitCode=0 Jul 07 11:07:21.599013 ip-10-0-135-34 kubenswrapper[2578]: I0707 11:07:21.598897 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-1273de/rate-test-model-1273de-kserve-57bcd69fbc-df5m8" event={"ID":"3959f4ca-7868-4862-b576-b679546ac4ad","Type":"ContainerDied","Data":"08876ea07fd6fdd679884dcda41b983d6cc1ecc5e20a49c8abe7c4b341cfcda2"} Jul 07 11:07:21.599013 ip-10-0-135-34 kubenswrapper[2578]: I0707 11:07:21.598938 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-1273de/rate-test-model-1273de-kserve-57bcd69fbc-df5m8" event={"ID":"3959f4ca-7868-4862-b576-b679546ac4ad","Type":"ContainerDied","Data":"c05fdae1f3791b9e041eff15e363a09c11a9d6eee4e09f7c8173cbb0d701434a"} Jul 07 11:07:21.599013 ip-10-0-135-34 kubenswrapper[2578]: I0707 11:07:21.598951 2578 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="c05fdae1f3791b9e041eff15e363a09c11a9d6eee4e09f7c8173cbb0d701434a" Jul 07 11:07:21.610661 ip-10-0-135-34 kubenswrapper[2578]: I0707 11:07:21.610638 2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-1273de/rate-test-model-1273de-kserve-57bcd69fbc-df5m8" Jul 07 11:07:21.630653 ip-10-0-135-34 kubenswrapper[2578]: I0707 11:07:21.630622 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-lsqb8\" (UniqueName: \"kubernetes.io/projected/3959f4ca-7868-4862-b576-b679546ac4ad-kube-api-access-lsqb8\") pod \"3959f4ca-7868-4862-b576-b679546ac4ad\" (UID: \"3959f4ca-7868-4862-b576-b679546ac4ad\") " Jul 07 11:07:21.630849 ip-10-0-135-34 kubenswrapper[2578]: I0707 11:07:21.630723 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3959f4ca-7868-4862-b576-b679546ac4ad-tmp-dir\") pod \"3959f4ca-7868-4862-b576-b679546ac4ad\" (UID: \"3959f4ca-7868-4862-b576-b679546ac4ad\") " Jul 07 11:07:21.630940 ip-10-0-135-34 kubenswrapper[2578]: I0707 11:07:21.630918 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3959f4ca-7868-4862-b576-b679546ac4ad-dshm\") pod \"3959f4ca-7868-4862-b576-b679546ac4ad\" (UID: \"3959f4ca-7868-4862-b576-b679546ac4ad\") " Jul 07 11:07:21.630997 ip-10-0-135-34 kubenswrapper[2578]: I0707 11:07:21.630974 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3959f4ca-7868-4862-b576-b679546ac4ad-tls-certs\") pod \"3959f4ca-7868-4862-b576-b679546ac4ad\" (UID: \"3959f4ca-7868-4862-b576-b679546ac4ad\") " Jul 07 11:07:21.631052 ip-10-0-135-34 kubenswrapper[2578]: I0707 11:07:21.630992 2578 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3959f4ca-7868-4862-b576-b679546ac4ad-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "3959f4ca-7868-4862-b576-b679546ac4ad" (UID: "3959f4ca-7868-4862-b576-b679546ac4ad"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 07 11:07:21.631052 ip-10-0-135-34 kubenswrapper[2578]: I0707 11:07:21.631001 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3959f4ca-7868-4862-b576-b679546ac4ad-home\") pod \"3959f4ca-7868-4862-b576-b679546ac4ad\" (UID: \"3959f4ca-7868-4862-b576-b679546ac4ad\") " Jul 07 11:07:21.631153 ip-10-0-135-34 kubenswrapper[2578]: I0707 11:07:21.631085 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3959f4ca-7868-4862-b576-b679546ac4ad-model-cache\") pod \"3959f4ca-7868-4862-b576-b679546ac4ad\" (UID: \"3959f4ca-7868-4862-b576-b679546ac4ad\") " Jul 07 11:07:21.631431 ip-10-0-135-34 kubenswrapper[2578]: I0707 11:07:21.631379 2578 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3959f4ca-7868-4862-b576-b679546ac4ad-tmp-dir\") on node \"ip-10-0-135-34.ec2.internal\" DevicePath \"\"" Jul 07 11:07:21.631431 ip-10-0-135-34 kubenswrapper[2578]: I0707 11:07:21.631370 2578 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3959f4ca-7868-4862-b576-b679546ac4ad-model-cache" (OuterVolumeSpecName: "model-cache") pod "3959f4ca-7868-4862-b576-b679546ac4ad" (UID: "3959f4ca-7868-4862-b576-b679546ac4ad"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 07 11:07:21.631653 ip-10-0-135-34 kubenswrapper[2578]: I0707 11:07:21.631634 2578 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3959f4ca-7868-4862-b576-b679546ac4ad-home" (OuterVolumeSpecName: "home") pod "3959f4ca-7868-4862-b576-b679546ac4ad" (UID: "3959f4ca-7868-4862-b576-b679546ac4ad"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 07 11:07:21.633954 ip-10-0-135-34 kubenswrapper[2578]: I0707 11:07:21.633926 2578 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3959f4ca-7868-4862-b576-b679546ac4ad-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "3959f4ca-7868-4862-b576-b679546ac4ad" (UID: "3959f4ca-7868-4862-b576-b679546ac4ad"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 07 11:07:21.634067 ip-10-0-135-34 kubenswrapper[2578]: I0707 11:07:21.633989 2578 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3959f4ca-7868-4862-b576-b679546ac4ad-dshm" (OuterVolumeSpecName: "dshm") pod "3959f4ca-7868-4862-b576-b679546ac4ad" (UID: "3959f4ca-7868-4862-b576-b679546ac4ad"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 07 11:07:21.635894 ip-10-0-135-34 kubenswrapper[2578]: I0707 11:07:21.635869 2578 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3959f4ca-7868-4862-b576-b679546ac4ad-kube-api-access-lsqb8" (OuterVolumeSpecName: "kube-api-access-lsqb8") pod "3959f4ca-7868-4862-b576-b679546ac4ad" (UID: "3959f4ca-7868-4862-b576-b679546ac4ad"). InnerVolumeSpecName "kube-api-access-lsqb8". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 07 11:07:21.733049 ip-10-0-135-34 kubenswrapper[2578]: I0707 11:07:21.733009 2578 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3959f4ca-7868-4862-b576-b679546ac4ad-dshm\") on node \"ip-10-0-135-34.ec2.internal\" DevicePath \"\"" Jul 07 11:07:21.733049 ip-10-0-135-34 kubenswrapper[2578]: I0707 11:07:21.733049 2578 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3959f4ca-7868-4862-b576-b679546ac4ad-tls-certs\") on node \"ip-10-0-135-34.ec2.internal\" DevicePath \"\"" Jul 07 11:07:21.733316 ip-10-0-135-34 kubenswrapper[2578]: I0707 11:07:21.733072 2578 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3959f4ca-7868-4862-b576-b679546ac4ad-home\") on node \"ip-10-0-135-34.ec2.internal\" DevicePath \"\"" Jul 07 11:07:21.733316 ip-10-0-135-34 kubenswrapper[2578]: I0707 11:07:21.733086 2578 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3959f4ca-7868-4862-b576-b679546ac4ad-model-cache\") on node \"ip-10-0-135-34.ec2.internal\" DevicePath \"\"" Jul 07 11:07:21.733316 ip-10-0-135-34 kubenswrapper[2578]: I0707 11:07:21.733100 2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lsqb8\" (UniqueName: \"kubernetes.io/projected/3959f4ca-7868-4862-b576-b679546ac4ad-kube-api-access-lsqb8\") on node \"ip-10-0-135-34.ec2.internal\" DevicePath \"\"" Jul 07 11:07:22.379448 ip-10-0-135-34 kubenswrapper[2578]: I0707 11:07:22.379380 2578 kuberuntime_container.go:872] "Killing container with a grace period" pod="ai-tenant-e2e-shared-b-2a3633/rate-test-model-2a3633-kserve-5cdd4c4f6c-wrhl9" podUID="188f6f12-3862-4c8e-baef-ed84674490cc" containerName="main" containerID="cri-o://d5b74e2004a5c6472d297c33c34caa741fafaba06d76a900e1982be6e406107e" gracePeriod=45 Jul 07 11:07:22.606564 ip-10-0-135-34 kubenswrapper[2578]: I0707 11:07:22.606530 2578 generic.go:356] "Generic (PLEG): container finished" podID="188f6f12-3862-4c8e-baef-ed84674490cc" containerID="d5b74e2004a5c6472d297c33c34caa741fafaba06d76a900e1982be6e406107e" exitCode=0 Jul 07 11:07:22.606787 ip-10-0-135-34 kubenswrapper[2578]: I0707 11:07:22.606603 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-2a3633/rate-test-model-2a3633-kserve-5cdd4c4f6c-wrhl9" event={"ID":"188f6f12-3862-4c8e-baef-ed84674490cc","Type":"ContainerDied","Data":"d5b74e2004a5c6472d297c33c34caa741fafaba06d76a900e1982be6e406107e"} Jul 07 11:07:22.606787 ip-10-0-135-34 kubenswrapper[2578]: I0707 11:07:22.606648 2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-1273de/rate-test-model-1273de-kserve-57bcd69fbc-df5m8" Jul 07 11:07:22.624669 ip-10-0-135-34 kubenswrapper[2578]: I0707 11:07:22.624637 2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-1273de/rate-test-model-1273de-kserve-57bcd69fbc-df5m8"] Jul 07 11:07:22.626691 ip-10-0-135-34 kubenswrapper[2578]: I0707 11:07:22.626662 2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-a-1273de/rate-test-model-1273de-kserve-57bcd69fbc-df5m8"] Jul 07 11:07:22.637546 ip-10-0-135-34 kubenswrapper[2578]: I0707 11:07:22.637484 2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-2a3633/rate-test-model-2a3633-kserve-5cdd4c4f6c-wrhl9" Jul 07 11:07:22.743209 ip-10-0-135-34 kubenswrapper[2578]: I0707 11:07:22.743169 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/188f6f12-3862-4c8e-baef-ed84674490cc-model-cache\") pod \"188f6f12-3862-4c8e-baef-ed84674490cc\" (UID: \"188f6f12-3862-4c8e-baef-ed84674490cc\") " Jul 07 11:07:22.743400 ip-10-0-135-34 kubenswrapper[2578]: I0707 11:07:22.743236 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/188f6f12-3862-4c8e-baef-ed84674490cc-home\") pod \"188f6f12-3862-4c8e-baef-ed84674490cc\" (UID: \"188f6f12-3862-4c8e-baef-ed84674490cc\") " Jul 07 11:07:22.743400 ip-10-0-135-34 kubenswrapper[2578]: I0707 11:07:22.743268 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/188f6f12-3862-4c8e-baef-ed84674490cc-tmp-dir\") pod \"188f6f12-3862-4c8e-baef-ed84674490cc\" (UID: \"188f6f12-3862-4c8e-baef-ed84674490cc\") " Jul 07 11:07:22.743400 ip-10-0-135-34 kubenswrapper[2578]: I0707 11:07:22.743291 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/188f6f12-3862-4c8e-baef-ed84674490cc-tls-certs\") pod \"188f6f12-3862-4c8e-baef-ed84674490cc\" (UID: \"188f6f12-3862-4c8e-baef-ed84674490cc\") " Jul 07 11:07:22.743400 ip-10-0-135-34 kubenswrapper[2578]: I0707 11:07:22.743326 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-7msq9\" (UniqueName: \"kubernetes.io/projected/188f6f12-3862-4c8e-baef-ed84674490cc-kube-api-access-7msq9\") pod \"188f6f12-3862-4c8e-baef-ed84674490cc\" (UID: \"188f6f12-3862-4c8e-baef-ed84674490cc\") " Jul 07 11:07:22.743400 ip-10-0-135-34 kubenswrapper[2578]: I0707 11:07:22.743367 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/188f6f12-3862-4c8e-baef-ed84674490cc-dshm\") pod \"188f6f12-3862-4c8e-baef-ed84674490cc\" (UID: \"188f6f12-3862-4c8e-baef-ed84674490cc\") " Jul 07 11:07:22.743705 ip-10-0-135-34 kubenswrapper[2578]: I0707 11:07:22.743494 2578 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/188f6f12-3862-4c8e-baef-ed84674490cc-model-cache" (OuterVolumeSpecName: "model-cache") pod "188f6f12-3862-4c8e-baef-ed84674490cc" (UID: "188f6f12-3862-4c8e-baef-ed84674490cc"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 07 11:07:22.743705 ip-10-0-135-34 kubenswrapper[2578]: I0707 11:07:22.743520 2578 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/188f6f12-3862-4c8e-baef-ed84674490cc-home" (OuterVolumeSpecName: "home") pod "188f6f12-3862-4c8e-baef-ed84674490cc" (UID: "188f6f12-3862-4c8e-baef-ed84674490cc"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 07 11:07:22.743705 ip-10-0-135-34 kubenswrapper[2578]: I0707 11:07:22.743537 2578 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/188f6f12-3862-4c8e-baef-ed84674490cc-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "188f6f12-3862-4c8e-baef-ed84674490cc" (UID: "188f6f12-3862-4c8e-baef-ed84674490cc"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 07 11:07:22.743705 ip-10-0-135-34 kubenswrapper[2578]: I0707 11:07:22.743637 2578 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/188f6f12-3862-4c8e-baef-ed84674490cc-model-cache\") on node \"ip-10-0-135-34.ec2.internal\" DevicePath \"\"" Jul 07 11:07:22.743705 ip-10-0-135-34 kubenswrapper[2578]: I0707 11:07:22.743654 2578 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/188f6f12-3862-4c8e-baef-ed84674490cc-home\") on node \"ip-10-0-135-34.ec2.internal\" DevicePath \"\"" Jul 07 11:07:22.743705 ip-10-0-135-34 kubenswrapper[2578]: I0707 11:07:22.743667 2578 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/188f6f12-3862-4c8e-baef-ed84674490cc-tmp-dir\") on node \"ip-10-0-135-34.ec2.internal\" DevicePath \"\"" Jul 07 11:07:22.745566 ip-10-0-135-34 kubenswrapper[2578]: I0707 11:07:22.745539 2578 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/188f6f12-3862-4c8e-baef-ed84674490cc-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "188f6f12-3862-4c8e-baef-ed84674490cc" (UID: "188f6f12-3862-4c8e-baef-ed84674490cc"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 07 11:07:22.745670 ip-10-0-135-34 kubenswrapper[2578]: I0707 11:07:22.745601 2578 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/188f6f12-3862-4c8e-baef-ed84674490cc-kube-api-access-7msq9" (OuterVolumeSpecName: "kube-api-access-7msq9") pod "188f6f12-3862-4c8e-baef-ed84674490cc" (UID: "188f6f12-3862-4c8e-baef-ed84674490cc"). InnerVolumeSpecName "kube-api-access-7msq9". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 07 11:07:22.745670 ip-10-0-135-34 kubenswrapper[2578]: I0707 11:07:22.745635 2578 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/188f6f12-3862-4c8e-baef-ed84674490cc-dshm" (OuterVolumeSpecName: "dshm") pod "188f6f12-3862-4c8e-baef-ed84674490cc" (UID: "188f6f12-3862-4c8e-baef-ed84674490cc"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 07 11:07:22.844377 ip-10-0-135-34 kubenswrapper[2578]: I0707 11:07:22.844336 2578 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/188f6f12-3862-4c8e-baef-ed84674490cc-tls-certs\") on node \"ip-10-0-135-34.ec2.internal\" DevicePath \"\"" Jul 07 11:07:22.844377 ip-10-0-135-34 kubenswrapper[2578]: I0707 11:07:22.844370 2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7msq9\" (UniqueName: \"kubernetes.io/projected/188f6f12-3862-4c8e-baef-ed84674490cc-kube-api-access-7msq9\") on node \"ip-10-0-135-34.ec2.internal\" DevicePath \"\"" Jul 07 11:07:22.844377 ip-10-0-135-34 kubenswrapper[2578]: I0707 11:07:22.844381 2578 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/188f6f12-3862-4c8e-baef-ed84674490cc-dshm\") on node \"ip-10-0-135-34.ec2.internal\" DevicePath \"\"" Jul 07 11:07:23.613309 ip-10-0-135-34 kubenswrapper[2578]: I0707 11:07:23.613273 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-2a3633/rate-test-model-2a3633-kserve-5cdd4c4f6c-wrhl9" event={"ID":"188f6f12-3862-4c8e-baef-ed84674490cc","Type":"ContainerDied","Data":"70512a2742fc99bfb686c7632f650c9f1d77d0bb6e73bc2f763b2e799d5530f2"} Jul 07 11:07:23.613309 ip-10-0-135-34 kubenswrapper[2578]: I0707 11:07:23.613305 2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-2a3633/rate-test-model-2a3633-kserve-5cdd4c4f6c-wrhl9" Jul 07 11:07:23.613764 ip-10-0-135-34 kubenswrapper[2578]: I0707 11:07:23.613335 2578 scope.go:117] "RemoveContainer" containerID="d5b74e2004a5c6472d297c33c34caa741fafaba06d76a900e1982be6e406107e" Jul 07 11:07:23.635158 ip-10-0-135-34 kubenswrapper[2578]: I0707 11:07:23.635111 2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-2a3633/rate-test-model-2a3633-kserve-5cdd4c4f6c-wrhl9"] Jul 07 11:07:23.640816 ip-10-0-135-34 kubenswrapper[2578]: I0707 11:07:23.640753 2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-b-2a3633/rate-test-model-2a3633-kserve-5cdd4c4f6c-wrhl9"] Jul 07 11:07:23.821233 ip-10-0-135-34 kubenswrapper[2578]: I0707 11:07:23.821201 2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="188f6f12-3862-4c8e-baef-ed84674490cc" path="/var/lib/kubelet/pods/188f6f12-3862-4c8e-baef-ed84674490cc/volumes" Jul 07 11:07:23.821605 ip-10-0-135-34 kubenswrapper[2578]: I0707 11:07:23.821592 2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="3959f4ca-7868-4862-b576-b679546ac4ad" path="/var/lib/kubelet/pods/3959f4ca-7868-4862-b576-b679546ac4ad/volumes" Jul 07 11:07:55.594005 ip-10-0-135-34 kubenswrapper[2578]: I0707 11:07:55.593968 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-lv7nd"] Jul 07 11:08:04.274816 ip-10-0-135-34 kubenswrapper[2578]: I0707 11:08:04.274763 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-586fcf4d5b-8hlhv_30a48a8d-7e8f-49eb-9dc5-c4d114f35dc1/copy-manifests/0.log" Jul 07 11:08:04.282136 ip-10-0-135-34 kubenswrapper[2578]: I0707 11:08:04.282113 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-586fcf4d5b-8hlhv_30a48a8d-7e8f-49eb-9dc5-c4d114f35dc1/manager/0.log" Jul 07 11:08:05.422260 ip-10-0-135-34 kubenswrapper[2578]: I0707 11:08:05.422224 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759mgzz2_26ed6eac-aef6-43e7-9f47-b739c096a031/util/0.log" Jul 07 11:08:05.428980 ip-10-0-135-34 kubenswrapper[2578]: I0707 11:08:05.428956 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759mgzz2_26ed6eac-aef6-43e7-9f47-b739c096a031/pull/0.log" Jul 07 11:08:05.435533 ip-10-0-135-34 kubenswrapper[2578]: I0707 11:08:05.435510 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759mgzz2_26ed6eac-aef6-43e7-9f47-b739c096a031/extract/0.log" Jul 07 11:08:05.548862 ip-10-0-135-34 kubenswrapper[2578]: I0707 11:08:05.548829 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0wrzpr_3fac3d82-02a4-4ddf-af58-d7392178ec4b/util/0.log" Jul 07 11:08:05.556318 ip-10-0-135-34 kubenswrapper[2578]: I0707 11:08:05.556278 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0wrzpr_3fac3d82-02a4-4ddf-af58-d7392178ec4b/pull/0.log" Jul 07 11:08:05.564479 ip-10-0-135-34 kubenswrapper[2578]: I0707 11:08:05.564433 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0wrzpr_3fac3d82-02a4-4ddf-af58-d7392178ec4b/extract/0.log" Jul 07 11:08:05.676231 ip-10-0-135-34 kubenswrapper[2578]: I0707 11:08:05.676138 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed735pdxq_06b9a5a9-0884-43cf-ad51-7b8f0bcd255f/util/0.log" Jul 07 11:08:05.683067 ip-10-0-135-34 kubenswrapper[2578]: I0707 11:08:05.683044 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed735pdxq_06b9a5a9-0884-43cf-ad51-7b8f0bcd255f/pull/0.log" Jul 07 11:08:05.692381 ip-10-0-135-34 kubenswrapper[2578]: I0707 11:08:05.692357 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed735pdxq_06b9a5a9-0884-43cf-ad51-7b8f0bcd255f/extract/0.log" Jul 07 11:08:05.805266 ip-10-0-135-34 kubenswrapper[2578]: I0707 11:08:05.805231 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1zqfm2_bf1fee85-d50a-4a52-84cd-e7f81323dd61/util/0.log" Jul 07 11:08:05.811671 ip-10-0-135-34 kubenswrapper[2578]: I0707 11:08:05.811655 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1zqfm2_bf1fee85-d50a-4a52-84cd-e7f81323dd61/pull/0.log" Jul 07 11:08:05.818086 ip-10-0-135-34 kubenswrapper[2578]: I0707 11:08:05.818041 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1zqfm2_bf1fee85-d50a-4a52-84cd-e7f81323dd61/extract/0.log" Jul 07 11:08:06.667185 ip-10-0-135-34 kubenswrapper[2578]: I0707 11:08:06.667157 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-limitador-54d6d988d5-lv7nd_e594c258-2852-4c44-b46d-0b48eb7828a7/limitador/0.log" Jul 07 11:08:07.576041 ip-10-0-135-34 kubenswrapper[2578]: I0707 11:08:07.576010 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-5c48467d48-dhpwl_1437df6f-5e77-4b9b-8ca1-877cc48b0d84/kube-auth-proxy/0.log" Jul 07 11:08:08.962876 ip-10-0-135-34 kubenswrapper[2578]: I0707 11:08:08.962849 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_facebook-opt-125m-simulated-kserve-74b68447df-7l6qx_19a3aba6-912b-4d5e-9d59-85fc40770a05/main/0.log" Jul 07 11:08:09.072898 ip-10-0-135-34 kubenswrapper[2578]: I0707 11:08:09.072868 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_premium-simulated-simulated-premium-kserve-5fc6fb456f-xw6hn_75c93d19-f81e-4106-bb4b-1432fdaf9402/main/0.log" Jul 07 11:08:13.225787 ip-10-0-135-34 kubenswrapper[2578]: I0707 11:08:13.225739 2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-must-gather-rgbf7/must-gather-wfmg9"] Jul 07 11:08:13.226180 ip-10-0-135-34 kubenswrapper[2578]: I0707 11:08:13.226084 2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3a107748-57b4-4e96-8684-8bd1232af914" containerName="maas-api" Jul 07 11:08:13.226180 ip-10-0-135-34 kubenswrapper[2578]: I0707 11:08:13.226095 2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="3a107748-57b4-4e96-8684-8bd1232af914" containerName="maas-api" Jul 07 11:08:13.226180 ip-10-0-135-34 kubenswrapper[2578]: I0707 11:08:13.226110 2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3959f4ca-7868-4862-b576-b679546ac4ad" containerName="main" Jul 07 11:08:13.226180 ip-10-0-135-34 kubenswrapper[2578]: I0707 11:08:13.226115 2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="3959f4ca-7868-4862-b576-b679546ac4ad" containerName="main" Jul 07 11:08:13.226180 ip-10-0-135-34 kubenswrapper[2578]: I0707 11:08:13.226123 2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="188f6f12-3862-4c8e-baef-ed84674490cc" containerName="main" Jul 07 11:08:13.226180 ip-10-0-135-34 kubenswrapper[2578]: I0707 11:08:13.226129 2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="188f6f12-3862-4c8e-baef-ed84674490cc" containerName="main" Jul 07 11:08:13.226397 ip-10-0-135-34 kubenswrapper[2578]: I0707 11:08:13.226197 2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="3a107748-57b4-4e96-8684-8bd1232af914" containerName="maas-api" Jul 07 11:08:13.226397 ip-10-0-135-34 kubenswrapper[2578]: I0707 11:08:13.226203 2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="3959f4ca-7868-4862-b576-b679546ac4ad" containerName="main" Jul 07 11:08:13.226397 ip-10-0-135-34 kubenswrapper[2578]: I0707 11:08:13.226209 2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="188f6f12-3862-4c8e-baef-ed84674490cc" containerName="main" Jul 07 11:08:13.230846 ip-10-0-135-34 kubenswrapper[2578]: I0707 11:08:13.230825 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-rgbf7/must-gather-wfmg9" Jul 07 11:08:13.233604 ip-10-0-135-34 kubenswrapper[2578]: I0707 11:08:13.233580 2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-rgbf7\"/\"kube-root-ca.crt\"" Jul 07 11:08:13.234728 ip-10-0-135-34 kubenswrapper[2578]: I0707 11:08:13.234710 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-rgbf7\"/\"default-dockercfg-9prl2\"" Jul 07 11:08:13.234848 ip-10-0-135-34 kubenswrapper[2578]: I0707 11:08:13.234730 2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-rgbf7\"/\"openshift-service-ca.crt\"" Jul 07 11:08:13.238420 ip-10-0-135-34 kubenswrapper[2578]: I0707 11:08:13.238397 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-rgbf7/must-gather-wfmg9"] Jul 07 11:08:13.286730 ip-10-0-135-34 kubenswrapper[2578]: I0707 11:08:13.286690 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nqmpd\" (UniqueName: \"kubernetes.io/projected/e9e4f3fa-193a-4471-96fe-dcc608369a8b-kube-api-access-nqmpd\") pod \"must-gather-wfmg9\" (UID: \"e9e4f3fa-193a-4471-96fe-dcc608369a8b\") " pod="openshift-must-gather-rgbf7/must-gather-wfmg9" Jul 07 11:08:13.286931 ip-10-0-135-34 kubenswrapper[2578]: I0707 11:08:13.286823 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/e9e4f3fa-193a-4471-96fe-dcc608369a8b-must-gather-output\") pod \"must-gather-wfmg9\" (UID: \"e9e4f3fa-193a-4471-96fe-dcc608369a8b\") " pod="openshift-must-gather-rgbf7/must-gather-wfmg9" Jul 07 11:08:13.387582 ip-10-0-135-34 kubenswrapper[2578]: I0707 11:08:13.387552 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nqmpd\" (UniqueName: \"kubernetes.io/projected/e9e4f3fa-193a-4471-96fe-dcc608369a8b-kube-api-access-nqmpd\") pod \"must-gather-wfmg9\" (UID: \"e9e4f3fa-193a-4471-96fe-dcc608369a8b\") " pod="openshift-must-gather-rgbf7/must-gather-wfmg9" Jul 07 11:08:13.387745 ip-10-0-135-34 kubenswrapper[2578]: I0707 11:08:13.387607 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/e9e4f3fa-193a-4471-96fe-dcc608369a8b-must-gather-output\") pod \"must-gather-wfmg9\" (UID: \"e9e4f3fa-193a-4471-96fe-dcc608369a8b\") " pod="openshift-must-gather-rgbf7/must-gather-wfmg9" Jul 07 11:08:13.387949 ip-10-0-135-34 kubenswrapper[2578]: I0707 11:08:13.387930 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/e9e4f3fa-193a-4471-96fe-dcc608369a8b-must-gather-output\") pod \"must-gather-wfmg9\" (UID: \"e9e4f3fa-193a-4471-96fe-dcc608369a8b\") " pod="openshift-must-gather-rgbf7/must-gather-wfmg9" Jul 07 11:08:13.397000 ip-10-0-135-34 kubenswrapper[2578]: I0707 11:08:13.396976 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nqmpd\" (UniqueName: \"kubernetes.io/projected/e9e4f3fa-193a-4471-96fe-dcc608369a8b-kube-api-access-nqmpd\") pod \"must-gather-wfmg9\" (UID: \"e9e4f3fa-193a-4471-96fe-dcc608369a8b\") " pod="openshift-must-gather-rgbf7/must-gather-wfmg9" Jul 07 11:08:13.540692 ip-10-0-135-34 kubenswrapper[2578]: I0707 11:08:13.540609 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-rgbf7/must-gather-wfmg9" Jul 07 11:08:13.699152 ip-10-0-135-34 kubenswrapper[2578]: I0707 11:08:13.699122 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-rgbf7/must-gather-wfmg9"] Jul 07 11:08:13.700842 ip-10-0-135-34 kubenswrapper[2578]: W0707 11:08:13.700816 2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pode9e4f3fa_193a_4471_96fe_dcc608369a8b.slice/crio-d9f8af25d93cd3ef3963dcafd4da6ba5098bbdcdb378c2be5f96938c72638c83 WatchSource:0}: Error finding container d9f8af25d93cd3ef3963dcafd4da6ba5098bbdcdb378c2be5f96938c72638c83: Status 404 returned error can't find the container with id d9f8af25d93cd3ef3963dcafd4da6ba5098bbdcdb378c2be5f96938c72638c83 Jul 07 11:08:13.703057 ip-10-0-135-34 kubenswrapper[2578]: I0707 11:08:13.703042 2578 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Jul 07 11:08:13.860395 ip-10-0-135-34 kubenswrapper[2578]: I0707 11:08:13.860362 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-rgbf7/must-gather-wfmg9" event={"ID":"e9e4f3fa-193a-4471-96fe-dcc608369a8b","Type":"ContainerStarted","Data":"d9f8af25d93cd3ef3963dcafd4da6ba5098bbdcdb378c2be5f96938c72638c83"} Jul 07 11:08:15.873981 ip-10-0-135-34 kubenswrapper[2578]: I0707 11:08:15.873944 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-rgbf7/must-gather-wfmg9" event={"ID":"e9e4f3fa-193a-4471-96fe-dcc608369a8b","Type":"ContainerStarted","Data":"a3e743e71283c2f9a6d8236bbc12fbcdc83324c51a742b0433c27b747ab38403"} Jul 07 11:08:15.874486 ip-10-0-135-34 kubenswrapper[2578]: I0707 11:08:15.874457 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-rgbf7/must-gather-wfmg9" event={"ID":"e9e4f3fa-193a-4471-96fe-dcc608369a8b","Type":"ContainerStarted","Data":"b64c4214ca77beb5f9e809921a2a38d4e6991da00f4f83e056a3c01fd877ea91"} Jul 07 11:08:15.891619 ip-10-0-135-34 kubenswrapper[2578]: I0707 11:08:15.891536 2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-must-gather-rgbf7/must-gather-wfmg9" podStartSLOduration=1.763718487 podStartE2EDuration="2.891521017s" totalImagesPullingTime="1.12780253s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-07 11:08:13 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 11:08:15.890827833 +0000 UTC m=+2464.562787404" watchObservedRunningTime="2026-07-07 11:08:15.891521017 +0000 UTC m=+2464.563480561" Jul 07 11:08:16.441704 ip-10-0-135-34 kubenswrapper[2578]: I0707 11:08:16.441669 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_global-pull-secret-syncer-7lv79_f85fac08-720e-4ffd-80af-19c140c36804/global-pull-secret-syncer/0.log" Jul 07 11:08:16.588994 ip-10-0-135-34 kubenswrapper[2578]: I0707 11:08:16.588921 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_konnectivity-agent-t82fl_11079224-e507-45d1-9fbb-da5043002c51/konnectivity-agent/0.log" Jul 07 11:08:16.647952 ip-10-0-135-34 kubenswrapper[2578]: I0707 11:08:16.647923 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kube-apiserver-proxy-ip-10-0-135-34.ec2.internal_fc94ccac0285e755088555819ea0238a/haproxy/0.log" Jul 07 11:08:20.576199 ip-10-0-135-34 kubenswrapper[2578]: I0707 11:08:20.576065 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759mgzz2_26ed6eac-aef6-43e7-9f47-b739c096a031/extract/0.log" Jul 07 11:08:20.603225 ip-10-0-135-34 kubenswrapper[2578]: I0707 11:08:20.603192 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759mgzz2_26ed6eac-aef6-43e7-9f47-b739c096a031/util/0.log" Jul 07 11:08:20.632138 ip-10-0-135-34 kubenswrapper[2578]: I0707 11:08:20.632101 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759mgzz2_26ed6eac-aef6-43e7-9f47-b739c096a031/pull/0.log" Jul 07 11:08:20.666098 ip-10-0-135-34 kubenswrapper[2578]: I0707 11:08:20.666068 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0wrzpr_3fac3d82-02a4-4ddf-af58-d7392178ec4b/extract/0.log" Jul 07 11:08:20.707627 ip-10-0-135-34 kubenswrapper[2578]: I0707 11:08:20.707597 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0wrzpr_3fac3d82-02a4-4ddf-af58-d7392178ec4b/util/0.log" Jul 07 11:08:20.742399 ip-10-0-135-34 kubenswrapper[2578]: I0707 11:08:20.742214 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0wrzpr_3fac3d82-02a4-4ddf-af58-d7392178ec4b/pull/0.log" Jul 07 11:08:20.784646 ip-10-0-135-34 kubenswrapper[2578]: I0707 11:08:20.784616 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed735pdxq_06b9a5a9-0884-43cf-ad51-7b8f0bcd255f/extract/0.log" Jul 07 11:08:20.817541 ip-10-0-135-34 kubenswrapper[2578]: I0707 11:08:20.817515 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed735pdxq_06b9a5a9-0884-43cf-ad51-7b8f0bcd255f/util/0.log" Jul 07 11:08:20.853828 ip-10-0-135-34 kubenswrapper[2578]: I0707 11:08:20.853800 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed735pdxq_06b9a5a9-0884-43cf-ad51-7b8f0bcd255f/pull/0.log" Jul 07 11:08:20.889590 ip-10-0-135-34 kubenswrapper[2578]: I0707 11:08:20.889422 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1zqfm2_bf1fee85-d50a-4a52-84cd-e7f81323dd61/extract/0.log" Jul 07 11:08:20.921130 ip-10-0-135-34 kubenswrapper[2578]: I0707 11:08:20.921099 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1zqfm2_bf1fee85-d50a-4a52-84cd-e7f81323dd61/util/0.log" Jul 07 11:08:20.947815 ip-10-0-135-34 kubenswrapper[2578]: I0707 11:08:20.947763 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1zqfm2_bf1fee85-d50a-4a52-84cd-e7f81323dd61/pull/0.log" Jul 07 11:08:21.805307 ip-10-0-135-34 kubenswrapper[2578]: I0707 11:08:21.805273 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-limitador-54d6d988d5-lv7nd_e594c258-2852-4c44-b46d-0b48eb7828a7/limitador/0.log" Jul 07 11:08:23.587583 ip-10-0-135-34 kubenswrapper[2578]: I0707 11:08:23.587523 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_c398d9f1-8285-4a95-8104-9144a247063d/alertmanager/0.log" Jul 07 11:08:23.628914 ip-10-0-135-34 kubenswrapper[2578]: I0707 11:08:23.628889 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_c398d9f1-8285-4a95-8104-9144a247063d/config-reloader/0.log" Jul 07 11:08:23.708787 ip-10-0-135-34 kubenswrapper[2578]: I0707 11:08:23.708661 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_c398d9f1-8285-4a95-8104-9144a247063d/kube-rbac-proxy-web/0.log" Jul 07 11:08:23.751560 ip-10-0-135-34 kubenswrapper[2578]: I0707 11:08:23.751525 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_c398d9f1-8285-4a95-8104-9144a247063d/kube-rbac-proxy/0.log" Jul 07 11:08:23.782366 ip-10-0-135-34 kubenswrapper[2578]: I0707 11:08:23.782333 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_c398d9f1-8285-4a95-8104-9144a247063d/kube-rbac-proxy-metric/0.log" Jul 07 11:08:23.813488 ip-10-0-135-34 kubenswrapper[2578]: I0707 11:08:23.813457 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_c398d9f1-8285-4a95-8104-9144a247063d/prom-label-proxy/0.log" Jul 07 11:08:23.842525 ip-10-0-135-34 kubenswrapper[2578]: I0707 11:08:23.842437 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_c398d9f1-8285-4a95-8104-9144a247063d/init-config-reloader/0.log" Jul 07 11:08:24.084824 ip-10-0-135-34 kubenswrapper[2578]: I0707 11:08:24.084791 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-bkr4f_12d81bd7-9a8f-4221-ae12-72ff1ad8716b/node-exporter/0.log" Jul 07 11:08:24.114718 ip-10-0-135-34 kubenswrapper[2578]: I0707 11:08:24.114675 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-bkr4f_12d81bd7-9a8f-4221-ae12-72ff1ad8716b/kube-rbac-proxy/0.log" Jul 07 11:08:24.139936 ip-10-0-135-34 kubenswrapper[2578]: I0707 11:08:24.139841 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-bkr4f_12d81bd7-9a8f-4221-ae12-72ff1ad8716b/init-textfile/0.log" Jul 07 11:08:24.326640 ip-10-0-135-34 kubenswrapper[2578]: I0707 11:08:24.326603 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_openshift-state-metrics-54cf96dfcf-ffft9_f41994cd-3ce0-43a3-bbd9-a853835da0bf/kube-rbac-proxy-main/0.log" Jul 07 11:08:24.354741 ip-10-0-135-34 kubenswrapper[2578]: I0707 11:08:24.354707 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_openshift-state-metrics-54cf96dfcf-ffft9_f41994cd-3ce0-43a3-bbd9-a853835da0bf/kube-rbac-proxy-self/0.log" Jul 07 11:08:24.377061 ip-10-0-135-34 kubenswrapper[2578]: I0707 11:08:24.376987 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_openshift-state-metrics-54cf96dfcf-ffft9_f41994cd-3ce0-43a3-bbd9-a853835da0bf/openshift-state-metrics/0.log" Jul 07 11:08:24.635131 ip-10-0-135-34 kubenswrapper[2578]: I0707 11:08:24.635021 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-operator-567f7b8b7c-snzgt_07c77276-527a-4178-ab38-919317f14156/prometheus-operator/0.log" Jul 07 11:08:24.673394 ip-10-0-135-34 kubenswrapper[2578]: I0707 11:08:24.673363 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-operator-567f7b8b7c-snzgt_07c77276-527a-4178-ab38-919317f14156/kube-rbac-proxy/0.log" Jul 07 11:08:24.743066 ip-10-0-135-34 kubenswrapper[2578]: I0707 11:08:24.743036 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_telemeter-client-57d74984bd-r76np_b1a95b49-97ce-4b2a-8500-f27f64dcfba7/telemeter-client/0.log" Jul 07 11:08:24.771998 ip-10-0-135-34 kubenswrapper[2578]: I0707 11:08:24.771967 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_telemeter-client-57d74984bd-r76np_b1a95b49-97ce-4b2a-8500-f27f64dcfba7/reload/0.log" Jul 07 11:08:24.798922 ip-10-0-135-34 kubenswrapper[2578]: I0707 11:08:24.798892 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_telemeter-client-57d74984bd-r76np_b1a95b49-97ce-4b2a-8500-f27f64dcfba7/kube-rbac-proxy/0.log" Jul 07 11:08:26.125059 ip-10-0-135-34 kubenswrapper[2578]: I0707 11:08:26.125010 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-network-console_networking-console-plugin-6bffdf9778-lmcz7_327fdc7e-f662-4adf-a589-5a22e48daa18/networking-console-plugin/0.log" Jul 07 11:08:26.363549 ip-10-0-135-34 kubenswrapper[2578]: I0707 11:08:26.363502 2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-must-gather-rgbf7/perf-node-gather-daemonset-5qz8w"] Jul 07 11:08:26.371170 ip-10-0-135-34 kubenswrapper[2578]: I0707 11:08:26.371134 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-rgbf7/perf-node-gather-daemonset-5qz8w" Jul 07 11:08:26.377320 ip-10-0-135-34 kubenswrapper[2578]: I0707 11:08:26.377245 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-rgbf7/perf-node-gather-daemonset-5qz8w"] Jul 07 11:08:26.430418 ip-10-0-135-34 kubenswrapper[2578]: I0707 11:08:26.430367 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/79a95e75-5d6a-4077-9103-8e5a35873ad3-lib-modules\") pod \"perf-node-gather-daemonset-5qz8w\" (UID: \"79a95e75-5d6a-4077-9103-8e5a35873ad3\") " pod="openshift-must-gather-rgbf7/perf-node-gather-daemonset-5qz8w" Jul 07 11:08:26.430606 ip-10-0-135-34 kubenswrapper[2578]: I0707 11:08:26.430452 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/79a95e75-5d6a-4077-9103-8e5a35873ad3-sys\") pod \"perf-node-gather-daemonset-5qz8w\" (UID: \"79a95e75-5d6a-4077-9103-8e5a35873ad3\") " pod="openshift-must-gather-rgbf7/perf-node-gather-daemonset-5qz8w" Jul 07 11:08:26.430606 ip-10-0-135-34 kubenswrapper[2578]: I0707 11:08:26.430496 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/79a95e75-5d6a-4077-9103-8e5a35873ad3-proc\") pod \"perf-node-gather-daemonset-5qz8w\" (UID: \"79a95e75-5d6a-4077-9103-8e5a35873ad3\") " pod="openshift-must-gather-rgbf7/perf-node-gather-daemonset-5qz8w" Jul 07 11:08:26.430606 ip-10-0-135-34 kubenswrapper[2578]: I0707 11:08:26.430565 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tvlf6\" (UniqueName: \"kubernetes.io/projected/79a95e75-5d6a-4077-9103-8e5a35873ad3-kube-api-access-tvlf6\") pod \"perf-node-gather-daemonset-5qz8w\" (UID: \"79a95e75-5d6a-4077-9103-8e5a35873ad3\") " pod="openshift-must-gather-rgbf7/perf-node-gather-daemonset-5qz8w" Jul 07 11:08:26.430606 ip-10-0-135-34 kubenswrapper[2578]: I0707 11:08:26.430604 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/79a95e75-5d6a-4077-9103-8e5a35873ad3-podres\") pod \"perf-node-gather-daemonset-5qz8w\" (UID: \"79a95e75-5d6a-4077-9103-8e5a35873ad3\") " pod="openshift-must-gather-rgbf7/perf-node-gather-daemonset-5qz8w" Jul 07 11:08:26.531976 ip-10-0-135-34 kubenswrapper[2578]: I0707 11:08:26.531943 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/79a95e75-5d6a-4077-9103-8e5a35873ad3-sys\") pod \"perf-node-gather-daemonset-5qz8w\" (UID: \"79a95e75-5d6a-4077-9103-8e5a35873ad3\") " pod="openshift-must-gather-rgbf7/perf-node-gather-daemonset-5qz8w" Jul 07 11:08:26.531976 ip-10-0-135-34 kubenswrapper[2578]: I0707 11:08:26.531990 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/79a95e75-5d6a-4077-9103-8e5a35873ad3-proc\") pod \"perf-node-gather-daemonset-5qz8w\" (UID: \"79a95e75-5d6a-4077-9103-8e5a35873ad3\") " pod="openshift-must-gather-rgbf7/perf-node-gather-daemonset-5qz8w" Jul 07 11:08:26.532227 ip-10-0-135-34 kubenswrapper[2578]: I0707 11:08:26.532041 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tvlf6\" (UniqueName: \"kubernetes.io/projected/79a95e75-5d6a-4077-9103-8e5a35873ad3-kube-api-access-tvlf6\") pod \"perf-node-gather-daemonset-5qz8w\" (UID: \"79a95e75-5d6a-4077-9103-8e5a35873ad3\") " pod="openshift-must-gather-rgbf7/perf-node-gather-daemonset-5qz8w" Jul 07 11:08:26.532227 ip-10-0-135-34 kubenswrapper[2578]: I0707 11:08:26.532069 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/79a95e75-5d6a-4077-9103-8e5a35873ad3-podres\") pod \"perf-node-gather-daemonset-5qz8w\" (UID: \"79a95e75-5d6a-4077-9103-8e5a35873ad3\") " pod="openshift-must-gather-rgbf7/perf-node-gather-daemonset-5qz8w" Jul 07 11:08:26.532227 ip-10-0-135-34 kubenswrapper[2578]: I0707 11:08:26.532075 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/79a95e75-5d6a-4077-9103-8e5a35873ad3-sys\") pod \"perf-node-gather-daemonset-5qz8w\" (UID: \"79a95e75-5d6a-4077-9103-8e5a35873ad3\") " pod="openshift-must-gather-rgbf7/perf-node-gather-daemonset-5qz8w" Jul 07 11:08:26.532227 ip-10-0-135-34 kubenswrapper[2578]: I0707 11:08:26.532104 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/79a95e75-5d6a-4077-9103-8e5a35873ad3-lib-modules\") pod \"perf-node-gather-daemonset-5qz8w\" (UID: \"79a95e75-5d6a-4077-9103-8e5a35873ad3\") " pod="openshift-must-gather-rgbf7/perf-node-gather-daemonset-5qz8w" Jul 07 11:08:26.532227 ip-10-0-135-34 kubenswrapper[2578]: I0707 11:08:26.532144 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/79a95e75-5d6a-4077-9103-8e5a35873ad3-proc\") pod \"perf-node-gather-daemonset-5qz8w\" (UID: \"79a95e75-5d6a-4077-9103-8e5a35873ad3\") " pod="openshift-must-gather-rgbf7/perf-node-gather-daemonset-5qz8w" Jul 07 11:08:26.532227 ip-10-0-135-34 kubenswrapper[2578]: I0707 11:08:26.532207 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/79a95e75-5d6a-4077-9103-8e5a35873ad3-lib-modules\") pod \"perf-node-gather-daemonset-5qz8w\" (UID: \"79a95e75-5d6a-4077-9103-8e5a35873ad3\") " pod="openshift-must-gather-rgbf7/perf-node-gather-daemonset-5qz8w" Jul 07 11:08:26.532444 ip-10-0-135-34 kubenswrapper[2578]: I0707 11:08:26.532229 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/79a95e75-5d6a-4077-9103-8e5a35873ad3-podres\") pod \"perf-node-gather-daemonset-5qz8w\" (UID: \"79a95e75-5d6a-4077-9103-8e5a35873ad3\") " pod="openshift-must-gather-rgbf7/perf-node-gather-daemonset-5qz8w" Jul 07 11:08:26.541299 ip-10-0-135-34 kubenswrapper[2578]: I0707 11:08:26.541265 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tvlf6\" (UniqueName: \"kubernetes.io/projected/79a95e75-5d6a-4077-9103-8e5a35873ad3-kube-api-access-tvlf6\") pod \"perf-node-gather-daemonset-5qz8w\" (UID: \"79a95e75-5d6a-4077-9103-8e5a35873ad3\") " pod="openshift-must-gather-rgbf7/perf-node-gather-daemonset-5qz8w" Jul 07 11:08:26.686900 ip-10-0-135-34 kubenswrapper[2578]: I0707 11:08:26.686799 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-rgbf7/perf-node-gather-daemonset-5qz8w" Jul 07 11:08:26.884554 ip-10-0-135-34 kubenswrapper[2578]: I0707 11:08:26.884521 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-rgbf7/perf-node-gather-daemonset-5qz8w"] Jul 07 11:08:26.886633 ip-10-0-135-34 kubenswrapper[2578]: W0707 11:08:26.886589 2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-pod79a95e75_5d6a_4077_9103_8e5a35873ad3.slice/crio-561feb0c91e6c0eafaae4efafa19dcd0d470ee750a23f0e665c2047ba5c05cfa WatchSource:0}: Error finding container 561feb0c91e6c0eafaae4efafa19dcd0d470ee750a23f0e665c2047ba5c05cfa: Status 404 returned error can't find the container with id 561feb0c91e6c0eafaae4efafa19dcd0d470ee750a23f0e665c2047ba5c05cfa Jul 07 11:08:26.960029 ip-10-0-135-34 kubenswrapper[2578]: I0707 11:08:26.959992 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-rgbf7/perf-node-gather-daemonset-5qz8w" event={"ID":"79a95e75-5d6a-4077-9103-8e5a35873ad3","Type":"ContainerStarted","Data":"561feb0c91e6c0eafaae4efafa19dcd0d470ee750a23f0e665c2047ba5c05cfa"} Jul 07 11:08:27.969408 ip-10-0-135-34 kubenswrapper[2578]: I0707 11:08:27.969362 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-rgbf7/perf-node-gather-daemonset-5qz8w" event={"ID":"79a95e75-5d6a-4077-9103-8e5a35873ad3","Type":"ContainerStarted","Data":"1fb200dcbc78915af0ce76a889ffaf1396e414f3f5b355b88bf1002c65eb4ad5"} Jul 07 11:08:27.969822 ip-10-0-135-34 kubenswrapper[2578]: I0707 11:08:27.969458 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-must-gather-rgbf7/perf-node-gather-daemonset-5qz8w" Jul 07 11:08:27.988984 ip-10-0-135-34 kubenswrapper[2578]: I0707 11:08:27.988913 2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-must-gather-rgbf7/perf-node-gather-daemonset-5qz8w" podStartSLOduration=1.988891805 podStartE2EDuration="1.988891805s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-07 11:08:26 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 11:08:27.985706513 +0000 UTC m=+2476.657666059" watchObservedRunningTime="2026-07-07 11:08:27.988891805 +0000 UTC m=+2476.660851350" Jul 07 11:08:29.445136 ip-10-0-135-34 kubenswrapper[2578]: I0707 11:08:29.445102 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-j2ks5_a1048018-11a0-4d5a-96ab-86508dac8366/dns/0.log" Jul 07 11:08:29.474603 ip-10-0-135-34 kubenswrapper[2578]: I0707 11:08:29.474572 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-j2ks5_a1048018-11a0-4d5a-96ab-86508dac8366/kube-rbac-proxy/0.log" Jul 07 11:08:29.535084 ip-10-0-135-34 kubenswrapper[2578]: I0707 11:08:29.535057 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-rdrnq_dfa64e14-3729-4f28-870f-eeab6cec9944/dns-node-resolver/0.log" Jul 07 11:08:30.207077 ip-10-0-135-34 kubenswrapper[2578]: I0707 11:08:30.207045 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-pb2p8_025a30f8-f923-4bc8-adf5-7ef97ee3bfd3/node-ca/0.log" Jul 07 11:08:31.393727 ip-10-0-135-34 kubenswrapper[2578]: I0707 11:08:31.393699 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-5c48467d48-dhpwl_1437df6f-5e77-4b9b-8ca1-877cc48b0d84/kube-auth-proxy/0.log" Jul 07 11:08:32.456362 ip-10-0-135-34 kubenswrapper[2578]: I0707 11:08:32.456330 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-s9zln_eae6c59c-de76-4a6e-8f2b-1de5dbeedf78/serve-healthcheck-canary/0.log" Jul 07 11:08:33.150326 ip-10-0-135-34 kubenswrapper[2578]: I0707 11:08:33.150294 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-t79dx_0a3e5a60-3150-4633-8ff1-4c8ebca70a41/kube-rbac-proxy/0.log" Jul 07 11:08:33.176696 ip-10-0-135-34 kubenswrapper[2578]: I0707 11:08:33.176671 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-t79dx_0a3e5a60-3150-4633-8ff1-4c8ebca70a41/exporter/0.log" Jul 07 11:08:33.203167 ip-10-0-135-34 kubenswrapper[2578]: I0707 11:08:33.203137 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-t79dx_0a3e5a60-3150-4633-8ff1-4c8ebca70a41/extractor/0.log" Jul 07 11:08:33.986440 ip-10-0-135-34 kubenswrapper[2578]: I0707 11:08:33.986411 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-must-gather-rgbf7/perf-node-gather-daemonset-5qz8w" Jul 07 11:08:36.901655 ip-10-0-135-34 kubenswrapper[2578]: I0707 11:08:36.901615 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-586fcf4d5b-8hlhv_30a48a8d-7e8f-49eb-9dc5-c4d114f35dc1/manager/0.log" Jul 07 11:08:36.923672 ip-10-0-135-34 kubenswrapper[2578]: I0707 11:08:36.923635 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-586fcf4d5b-8hlhv_30a48a8d-7e8f-49eb-9dc5-c4d114f35dc1/copy-manifests/0.log" Jul 07 11:08:37.340952 ip-10-0-135-34 kubenswrapper[2578]: I0707 11:08:37.340869 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-5c48467d48-dhpwl_1437df6f-5e77-4b9b-8ca1-877cc48b0d84/kube-auth-proxy/0.log" Jul 07 11:08:38.508304 ip-10-0-135-34 kubenswrapper[2578]: I0707 11:08:38.508276 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-lws-operator_lws-controller-manager-59649ddfff-4mzhd_482b392b-7cc6-4e18-aa67-306d6a473bfe/manager/0.log" Jul 07 11:08:38.861617 ip-10-0-135-34 kubenswrapper[2578]: I0707 11:08:38.861583 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-5c48467d48-dhpwl_1437df6f-5e77-4b9b-8ca1-877cc48b0d84/kube-auth-proxy/0.log"