Jul 30 00:15:37.477244 ip-10-0-142-6 systemd[1]: kubelet.service: Failed to load environment files: No such file or directory Jul 30 00:15:37.477254 ip-10-0-142-6 systemd[1]: kubelet.service: Failed to run 'start-pre' task: No such file or directory Jul 30 00:15:37.477261 ip-10-0-142-6 systemd[1]: kubelet.service: Failed with result 'resources'. Jul 30 00:15:37.477475 ip-10-0-142-6 systemd[1]: Failed to start Kubernetes Kubelet. Jul 30 00:15:47.693991 ip-10-0-142-6 systemd[1]: kubelet.service: Failed to schedule restart job: Unit crio.service not found. Jul 30 00:15:47.694007 ip-10-0-142-6 systemd[1]: kubelet.service: Failed with result 'resources'. -- Boot f75ae0d214304df49e5f7d064a1fe4ac -- Jul 30 00:17:47.336240 ip-10-0-142-6 systemd[1]: Starting Kubernetes Kubelet... Jul 30 00:17:47.722396 ip-10-0-142-6 kubenswrapper[2564]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Jul 30 00:17:47.722396 ip-10-0-142-6 kubenswrapper[2564]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version. Jul 30 00:17:47.722396 ip-10-0-142-6 kubenswrapper[2564]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Jul 30 00:17:47.722396 ip-10-0-142-6 kubenswrapper[2564]: Flag --pod-infra-container-image has been deprecated, will be removed in 1.35. Image garbage collector will get sandbox image information from CRI. Jul 30 00:17:47.722396 ip-10-0-142-6 kubenswrapper[2564]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Jul 30 00:17:47.723467 ip-10-0-142-6 kubenswrapper[2564]: I0730 00:17:47.723349 2564 server.go:213] "--pod-infra-container-image will not be pruned by the image garbage collector in kubelet and should also be set in the remote runtime" Jul 30 00:17:47.727671 ip-10-0-142-6 kubenswrapper[2564]: W0730 00:17:47.727653 2564 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 30 00:17:47.727671 ip-10-0-142-6 kubenswrapper[2564]: W0730 00:17:47.727671 2564 feature_gate.go:328] unrecognized feature gate: Example Jul 30 00:17:47.727760 ip-10-0-142-6 kubenswrapper[2564]: W0730 00:17:47.727675 2564 feature_gate.go:328] unrecognized feature gate: Example2 Jul 30 00:17:47.727760 ip-10-0-142-6 kubenswrapper[2564]: W0730 00:17:47.727679 2564 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 30 00:17:47.727760 ip-10-0-142-6 kubenswrapper[2564]: W0730 00:17:47.727682 2564 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 30 00:17:47.727760 ip-10-0-142-6 kubenswrapper[2564]: W0730 00:17:47.727685 2564 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 30 00:17:47.727760 ip-10-0-142-6 kubenswrapper[2564]: W0730 00:17:47.727688 2564 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 30 00:17:47.727760 ip-10-0-142-6 kubenswrapper[2564]: W0730 00:17:47.727691 2564 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 30 00:17:47.727760 ip-10-0-142-6 kubenswrapper[2564]: W0730 00:17:47.727694 2564 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 30 00:17:47.727760 ip-10-0-142-6 kubenswrapper[2564]: W0730 00:17:47.727697 2564 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 30 00:17:47.727760 ip-10-0-142-6 kubenswrapper[2564]: W0730 00:17:47.727700 2564 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 30 00:17:47.727760 ip-10-0-142-6 kubenswrapper[2564]: W0730 00:17:47.727702 2564 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 30 00:17:47.727760 ip-10-0-142-6 kubenswrapper[2564]: W0730 00:17:47.727705 2564 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 30 00:17:47.727760 ip-10-0-142-6 kubenswrapper[2564]: W0730 00:17:47.727708 2564 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 30 00:17:47.727760 ip-10-0-142-6 kubenswrapper[2564]: W0730 00:17:47.727711 2564 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 30 00:17:47.727760 ip-10-0-142-6 kubenswrapper[2564]: W0730 00:17:47.727714 2564 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 30 00:17:47.727760 ip-10-0-142-6 kubenswrapper[2564]: W0730 00:17:47.727718 2564 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 30 00:17:47.727760 ip-10-0-142-6 kubenswrapper[2564]: W0730 00:17:47.727720 2564 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 30 00:17:47.727760 ip-10-0-142-6 kubenswrapper[2564]: W0730 00:17:47.727723 2564 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 30 00:17:47.727760 ip-10-0-142-6 kubenswrapper[2564]: W0730 00:17:47.727725 2564 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 30 00:17:47.727760 ip-10-0-142-6 kubenswrapper[2564]: W0730 00:17:47.727728 2564 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 30 00:17:47.727760 ip-10-0-142-6 kubenswrapper[2564]: W0730 00:17:47.727730 2564 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 30 00:17:48.867570 ip-10-0-142-6 kubenswrapper[2564]: W0730 00:17:47.727733 2564 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 30 00:17:48.867570 ip-10-0-142-6 kubenswrapper[2564]: W0730 00:17:47.727736 2564 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 30 00:17:48.867570 ip-10-0-142-6 kubenswrapper[2564]: W0730 00:17:47.727738 2564 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 30 00:17:48.867570 ip-10-0-142-6 kubenswrapper[2564]: W0730 00:17:47.727741 2564 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 30 00:17:48.867570 ip-10-0-142-6 kubenswrapper[2564]: W0730 00:17:47.727743 2564 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 30 00:17:48.867570 ip-10-0-142-6 kubenswrapper[2564]: W0730 00:17:47.727746 2564 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 30 00:17:48.867570 ip-10-0-142-6 kubenswrapper[2564]: W0730 00:17:47.727749 2564 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 30 00:17:48.867570 ip-10-0-142-6 kubenswrapper[2564]: W0730 00:17:47.727751 2564 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 30 00:17:48.867570 ip-10-0-142-6 kubenswrapper[2564]: W0730 00:17:47.727754 2564 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 30 00:17:48.867570 ip-10-0-142-6 kubenswrapper[2564]: W0730 00:17:47.727756 2564 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 30 00:17:48.867570 ip-10-0-142-6 kubenswrapper[2564]: W0730 00:17:47.727759 2564 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 30 00:17:48.867570 ip-10-0-142-6 kubenswrapper[2564]: W0730 00:17:47.727764 2564 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 30 00:17:48.867570 ip-10-0-142-6 kubenswrapper[2564]: W0730 00:17:47.727768 2564 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 30 00:17:48.867570 ip-10-0-142-6 kubenswrapper[2564]: W0730 00:17:47.727773 2564 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 30 00:17:48.867570 ip-10-0-142-6 kubenswrapper[2564]: W0730 00:17:47.727776 2564 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 30 00:17:48.867570 ip-10-0-142-6 kubenswrapper[2564]: W0730 00:17:47.727780 2564 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 30 00:17:48.867570 ip-10-0-142-6 kubenswrapper[2564]: W0730 00:17:47.727784 2564 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 30 00:17:48.867570 ip-10-0-142-6 kubenswrapper[2564]: W0730 00:17:47.727787 2564 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 30 00:17:48.867570 ip-10-0-142-6 kubenswrapper[2564]: W0730 00:17:47.727791 2564 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 30 00:17:48.867570 ip-10-0-142-6 kubenswrapper[2564]: W0730 00:17:47.727795 2564 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 30 00:17:48.868859 ip-10-0-142-6 kubenswrapper[2564]: W0730 00:17:47.727798 2564 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 30 00:17:48.868859 ip-10-0-142-6 kubenswrapper[2564]: W0730 00:17:47.727802 2564 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 30 00:17:48.868859 ip-10-0-142-6 kubenswrapper[2564]: W0730 00:17:47.727806 2564 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 30 00:17:48.868859 ip-10-0-142-6 kubenswrapper[2564]: W0730 00:17:47.727809 2564 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 30 00:17:48.868859 ip-10-0-142-6 kubenswrapper[2564]: W0730 00:17:47.727813 2564 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 30 00:17:48.868859 ip-10-0-142-6 kubenswrapper[2564]: W0730 00:17:47.727816 2564 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 30 00:17:48.868859 ip-10-0-142-6 kubenswrapper[2564]: W0730 00:17:47.727820 2564 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 30 00:17:48.868859 ip-10-0-142-6 kubenswrapper[2564]: W0730 00:17:47.727823 2564 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 30 00:17:48.868859 ip-10-0-142-6 kubenswrapper[2564]: W0730 00:17:47.727827 2564 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 30 00:17:48.868859 ip-10-0-142-6 kubenswrapper[2564]: W0730 00:17:47.727830 2564 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 30 00:17:48.868859 ip-10-0-142-6 kubenswrapper[2564]: W0730 00:17:47.727834 2564 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 30 00:17:48.868859 ip-10-0-142-6 kubenswrapper[2564]: W0730 00:17:47.727838 2564 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 30 00:17:48.868859 ip-10-0-142-6 kubenswrapper[2564]: W0730 00:17:47.727841 2564 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 30 00:17:48.868859 ip-10-0-142-6 kubenswrapper[2564]: W0730 00:17:47.727848 2564 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 30 00:17:48.868859 ip-10-0-142-6 kubenswrapper[2564]: W0730 00:17:47.727854 2564 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 30 00:17:48.868859 ip-10-0-142-6 kubenswrapper[2564]: W0730 00:17:47.727858 2564 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 30 00:17:48.868859 ip-10-0-142-6 kubenswrapper[2564]: W0730 00:17:47.727861 2564 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 30 00:17:48.868859 ip-10-0-142-6 kubenswrapper[2564]: W0730 00:17:47.727865 2564 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 30 00:17:48.868859 ip-10-0-142-6 kubenswrapper[2564]: W0730 00:17:47.727869 2564 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 30 00:17:48.869415 ip-10-0-142-6 kubenswrapper[2564]: W0730 00:17:47.727873 2564 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 30 00:17:48.869415 ip-10-0-142-6 kubenswrapper[2564]: W0730 00:17:47.727876 2564 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 30 00:17:48.869415 ip-10-0-142-6 kubenswrapper[2564]: W0730 00:17:47.727880 2564 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 30 00:17:48.869415 ip-10-0-142-6 kubenswrapper[2564]: W0730 00:17:47.727883 2564 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 30 00:17:48.869415 ip-10-0-142-6 kubenswrapper[2564]: W0730 00:17:47.727886 2564 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 30 00:17:48.869415 ip-10-0-142-6 kubenswrapper[2564]: W0730 00:17:47.727890 2564 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 30 00:17:48.869415 ip-10-0-142-6 kubenswrapper[2564]: W0730 00:17:47.727894 2564 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 30 00:17:48.869415 ip-10-0-142-6 kubenswrapper[2564]: W0730 00:17:47.727898 2564 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 30 00:17:48.869415 ip-10-0-142-6 kubenswrapper[2564]: W0730 00:17:47.727901 2564 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 30 00:17:48.869415 ip-10-0-142-6 kubenswrapper[2564]: W0730 00:17:47.727923 2564 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 30 00:17:48.869415 ip-10-0-142-6 kubenswrapper[2564]: W0730 00:17:47.727929 2564 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 30 00:17:48.869415 ip-10-0-142-6 kubenswrapper[2564]: W0730 00:17:47.727933 2564 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 30 00:17:48.869415 ip-10-0-142-6 kubenswrapper[2564]: W0730 00:17:48.867164 2564 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 30 00:17:48.869415 ip-10-0-142-6 kubenswrapper[2564]: W0730 00:17:48.867219 2564 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 30 00:17:48.869415 ip-10-0-142-6 kubenswrapper[2564]: W0730 00:17:48.867227 2564 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 30 00:17:48.869415 ip-10-0-142-6 kubenswrapper[2564]: W0730 00:17:48.867231 2564 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 30 00:17:48.869415 ip-10-0-142-6 kubenswrapper[2564]: W0730 00:17:48.867234 2564 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 30 00:17:48.869415 ip-10-0-142-6 kubenswrapper[2564]: W0730 00:17:48.867237 2564 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 30 00:17:48.869415 ip-10-0-142-6 kubenswrapper[2564]: W0730 00:17:48.867240 2564 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 30 00:17:48.869903 ip-10-0-142-6 kubenswrapper[2564]: W0730 00:17:48.867243 2564 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 30 00:17:48.869903 ip-10-0-142-6 kubenswrapper[2564]: W0730 00:17:48.867246 2564 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 30 00:17:48.869903 ip-10-0-142-6 kubenswrapper[2564]: W0730 00:17:48.867250 2564 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 30 00:17:48.869903 ip-10-0-142-6 kubenswrapper[2564]: W0730 00:17:48.867253 2564 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 30 00:17:48.869903 ip-10-0-142-6 kubenswrapper[2564]: W0730 00:17:48.867256 2564 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 30 00:17:48.869903 ip-10-0-142-6 kubenswrapper[2564]: W0730 00:17:48.867259 2564 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 30 00:17:48.869903 ip-10-0-142-6 kubenswrapper[2564]: W0730 00:17:48.867266 2564 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 30 00:17:48.869903 ip-10-0-142-6 kubenswrapper[2564]: W0730 00:17:48.867270 2564 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 30 00:17:48.869903 ip-10-0-142-6 kubenswrapper[2564]: W0730 00:17:48.867274 2564 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 30 00:17:48.869903 ip-10-0-142-6 kubenswrapper[2564]: W0730 00:17:48.867428 2564 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 30 00:17:48.869903 ip-10-0-142-6 kubenswrapper[2564]: W0730 00:17:48.867454 2564 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 30 00:17:48.869903 ip-10-0-142-6 kubenswrapper[2564]: W0730 00:17:48.867463 2564 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 30 00:17:48.869903 ip-10-0-142-6 kubenswrapper[2564]: W0730 00:17:48.867469 2564 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 30 00:17:48.869903 ip-10-0-142-6 kubenswrapper[2564]: W0730 00:17:48.867476 2564 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 30 00:17:48.869903 ip-10-0-142-6 kubenswrapper[2564]: W0730 00:17:48.867482 2564 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 30 00:17:48.869903 ip-10-0-142-6 kubenswrapper[2564]: W0730 00:17:48.867487 2564 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 30 00:17:48.869903 ip-10-0-142-6 kubenswrapper[2564]: W0730 00:17:48.867493 2564 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 30 00:17:48.869903 ip-10-0-142-6 kubenswrapper[2564]: W0730 00:17:48.867498 2564 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 30 00:17:48.869903 ip-10-0-142-6 kubenswrapper[2564]: W0730 00:17:48.867503 2564 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 30 00:17:48.870374 ip-10-0-142-6 kubenswrapper[2564]: W0730 00:17:48.869423 2564 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 30 00:17:48.870374 ip-10-0-142-6 kubenswrapper[2564]: W0730 00:17:48.869434 2564 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 30 00:17:48.870374 ip-10-0-142-6 kubenswrapper[2564]: W0730 00:17:48.869438 2564 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 30 00:17:48.870374 ip-10-0-142-6 kubenswrapper[2564]: W0730 00:17:48.869442 2564 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 30 00:17:48.870374 ip-10-0-142-6 kubenswrapper[2564]: W0730 00:17:48.869445 2564 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 30 00:17:48.870374 ip-10-0-142-6 kubenswrapper[2564]: W0730 00:17:48.869448 2564 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 30 00:17:48.870374 ip-10-0-142-6 kubenswrapper[2564]: W0730 00:17:48.869451 2564 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 30 00:17:48.870374 ip-10-0-142-6 kubenswrapper[2564]: W0730 00:17:48.869454 2564 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 30 00:17:48.870374 ip-10-0-142-6 kubenswrapper[2564]: W0730 00:17:48.869456 2564 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 30 00:17:48.870374 ip-10-0-142-6 kubenswrapper[2564]: W0730 00:17:48.869459 2564 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 30 00:17:48.870374 ip-10-0-142-6 kubenswrapper[2564]: W0730 00:17:48.869461 2564 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 30 00:17:48.870374 ip-10-0-142-6 kubenswrapper[2564]: W0730 00:17:48.869464 2564 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 30 00:17:48.870374 ip-10-0-142-6 kubenswrapper[2564]: W0730 00:17:48.869467 2564 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 30 00:17:48.870374 ip-10-0-142-6 kubenswrapper[2564]: W0730 00:17:48.869469 2564 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 30 00:17:48.870374 ip-10-0-142-6 kubenswrapper[2564]: W0730 00:17:48.869472 2564 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 30 00:17:48.870374 ip-10-0-142-6 kubenswrapper[2564]: W0730 00:17:48.869475 2564 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 30 00:17:48.870374 ip-10-0-142-6 kubenswrapper[2564]: W0730 00:17:48.869478 2564 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 30 00:17:48.870374 ip-10-0-142-6 kubenswrapper[2564]: W0730 00:17:48.869486 2564 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 30 00:17:48.870374 ip-10-0-142-6 kubenswrapper[2564]: W0730 00:17:48.869489 2564 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 30 00:17:48.870374 ip-10-0-142-6 kubenswrapper[2564]: W0730 00:17:48.869491 2564 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 30 00:17:48.870868 ip-10-0-142-6 kubenswrapper[2564]: W0730 00:17:48.869494 2564 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 30 00:17:48.870868 ip-10-0-142-6 kubenswrapper[2564]: W0730 00:17:48.869497 2564 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 30 00:17:48.870868 ip-10-0-142-6 kubenswrapper[2564]: W0730 00:17:48.869499 2564 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 30 00:17:48.870868 ip-10-0-142-6 kubenswrapper[2564]: W0730 00:17:48.869503 2564 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 30 00:17:48.870868 ip-10-0-142-6 kubenswrapper[2564]: W0730 00:17:48.869506 2564 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 30 00:17:48.870868 ip-10-0-142-6 kubenswrapper[2564]: W0730 00:17:48.869508 2564 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 30 00:17:48.870868 ip-10-0-142-6 kubenswrapper[2564]: W0730 00:17:48.869511 2564 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 30 00:17:48.870868 ip-10-0-142-6 kubenswrapper[2564]: W0730 00:17:48.869513 2564 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 30 00:17:48.870868 ip-10-0-142-6 kubenswrapper[2564]: W0730 00:17:48.869516 2564 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 30 00:17:48.870868 ip-10-0-142-6 kubenswrapper[2564]: W0730 00:17:48.869519 2564 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 30 00:17:48.870868 ip-10-0-142-6 kubenswrapper[2564]: W0730 00:17:48.869521 2564 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 30 00:17:48.870868 ip-10-0-142-6 kubenswrapper[2564]: W0730 00:17:48.869523 2564 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 30 00:17:48.870868 ip-10-0-142-6 kubenswrapper[2564]: W0730 00:17:48.869527 2564 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 30 00:17:48.870868 ip-10-0-142-6 kubenswrapper[2564]: W0730 00:17:48.869530 2564 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 30 00:17:48.870868 ip-10-0-142-6 kubenswrapper[2564]: W0730 00:17:48.869533 2564 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 30 00:17:48.870868 ip-10-0-142-6 kubenswrapper[2564]: W0730 00:17:48.869535 2564 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 30 00:17:48.870868 ip-10-0-142-6 kubenswrapper[2564]: W0730 00:17:48.869538 2564 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 30 00:17:48.870868 ip-10-0-142-6 kubenswrapper[2564]: W0730 00:17:48.869540 2564 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 30 00:17:48.870868 ip-10-0-142-6 kubenswrapper[2564]: W0730 00:17:48.869543 2564 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 30 00:17:48.870868 ip-10-0-142-6 kubenswrapper[2564]: W0730 00:17:48.869545 2564 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 30 00:17:48.871480 ip-10-0-142-6 kubenswrapper[2564]: W0730 00:17:48.869548 2564 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 30 00:17:48.871480 ip-10-0-142-6 kubenswrapper[2564]: W0730 00:17:48.869550 2564 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 30 00:17:48.871480 ip-10-0-142-6 kubenswrapper[2564]: W0730 00:17:48.869553 2564 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 30 00:17:48.871480 ip-10-0-142-6 kubenswrapper[2564]: W0730 00:17:48.869555 2564 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 30 00:17:48.871480 ip-10-0-142-6 kubenswrapper[2564]: W0730 00:17:48.869558 2564 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 30 00:17:48.871480 ip-10-0-142-6 kubenswrapper[2564]: W0730 00:17:48.869561 2564 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 30 00:17:48.871480 ip-10-0-142-6 kubenswrapper[2564]: W0730 00:17:48.869563 2564 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 30 00:17:48.871480 ip-10-0-142-6 kubenswrapper[2564]: W0730 00:17:48.869566 2564 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 30 00:17:48.871480 ip-10-0-142-6 kubenswrapper[2564]: W0730 00:17:48.869569 2564 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 30 00:17:48.871480 ip-10-0-142-6 kubenswrapper[2564]: W0730 00:17:48.869572 2564 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 30 00:17:48.871480 ip-10-0-142-6 kubenswrapper[2564]: W0730 00:17:48.869575 2564 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 30 00:17:48.871480 ip-10-0-142-6 kubenswrapper[2564]: W0730 00:17:48.869577 2564 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 30 00:17:48.871480 ip-10-0-142-6 kubenswrapper[2564]: W0730 00:17:48.869580 2564 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 30 00:17:48.871480 ip-10-0-142-6 kubenswrapper[2564]: W0730 00:17:48.869582 2564 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 30 00:17:48.871480 ip-10-0-142-6 kubenswrapper[2564]: W0730 00:17:48.869585 2564 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 30 00:17:48.871480 ip-10-0-142-6 kubenswrapper[2564]: W0730 00:17:48.869587 2564 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 30 00:17:48.871480 ip-10-0-142-6 kubenswrapper[2564]: W0730 00:17:48.869590 2564 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 30 00:17:48.871480 ip-10-0-142-6 kubenswrapper[2564]: W0730 00:17:48.869592 2564 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 30 00:17:48.871480 ip-10-0-142-6 kubenswrapper[2564]: W0730 00:17:48.869594 2564 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 30 00:17:48.871940 ip-10-0-142-6 kubenswrapper[2564]: W0730 00:17:48.869597 2564 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 30 00:17:48.871940 ip-10-0-142-6 kubenswrapper[2564]: W0730 00:17:48.869600 2564 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 30 00:17:48.871940 ip-10-0-142-6 kubenswrapper[2564]: W0730 00:17:48.869602 2564 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 30 00:17:48.871940 ip-10-0-142-6 kubenswrapper[2564]: W0730 00:17:48.869604 2564 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 30 00:17:48.871940 ip-10-0-142-6 kubenswrapper[2564]: W0730 00:17:48.869607 2564 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 30 00:17:48.871940 ip-10-0-142-6 kubenswrapper[2564]: W0730 00:17:48.869610 2564 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 30 00:17:48.871940 ip-10-0-142-6 kubenswrapper[2564]: W0730 00:17:48.869613 2564 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 30 00:17:48.871940 ip-10-0-142-6 kubenswrapper[2564]: W0730 00:17:48.869616 2564 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 30 00:17:48.871940 ip-10-0-142-6 kubenswrapper[2564]: W0730 00:17:48.869619 2564 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 30 00:17:48.871940 ip-10-0-142-6 kubenswrapper[2564]: W0730 00:17:48.869621 2564 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 30 00:17:48.871940 ip-10-0-142-6 kubenswrapper[2564]: W0730 00:17:48.869624 2564 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 30 00:17:48.871940 ip-10-0-142-6 kubenswrapper[2564]: W0730 00:17:48.869626 2564 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 30 00:17:48.871940 ip-10-0-142-6 kubenswrapper[2564]: W0730 00:17:48.869629 2564 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 30 00:17:48.871940 ip-10-0-142-6 kubenswrapper[2564]: W0730 00:17:48.869631 2564 feature_gate.go:328] unrecognized feature gate: Example Jul 30 00:17:48.871940 ip-10-0-142-6 kubenswrapper[2564]: W0730 00:17:48.869634 2564 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 30 00:17:48.871940 ip-10-0-142-6 kubenswrapper[2564]: W0730 00:17:48.869636 2564 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 30 00:17:48.871940 ip-10-0-142-6 kubenswrapper[2564]: W0730 00:17:48.869639 2564 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 30 00:17:48.871940 ip-10-0-142-6 kubenswrapper[2564]: W0730 00:17:48.869641 2564 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 30 00:17:48.871940 ip-10-0-142-6 kubenswrapper[2564]: W0730 00:17:48.869644 2564 feature_gate.go:328] unrecognized feature gate: Example2 Jul 30 00:17:48.871940 ip-10-0-142-6 kubenswrapper[2564]: W0730 00:17:48.869646 2564 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 30 00:17:48.872449 ip-10-0-142-6 kubenswrapper[2564]: W0730 00:17:48.869651 2564 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 30 00:17:48.872449 ip-10-0-142-6 kubenswrapper[2564]: W0730 00:17:48.869655 2564 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 30 00:17:48.872449 ip-10-0-142-6 kubenswrapper[2564]: W0730 00:17:48.869658 2564 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 30 00:17:48.872449 ip-10-0-142-6 kubenswrapper[2564]: W0730 00:17:48.869661 2564 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 30 00:17:48.872449 ip-10-0-142-6 kubenswrapper[2564]: W0730 00:17:48.869664 2564 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 30 00:17:48.872449 ip-10-0-142-6 kubenswrapper[2564]: W0730 00:17:48.869666 2564 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 30 00:17:48.872449 ip-10-0-142-6 kubenswrapper[2564]: W0730 00:17:48.869669 2564 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 30 00:17:48.872449 ip-10-0-142-6 kubenswrapper[2564]: W0730 00:17:48.869671 2564 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 30 00:17:48.872449 ip-10-0-142-6 kubenswrapper[2564]: W0730 00:17:48.869674 2564 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 30 00:17:48.872449 ip-10-0-142-6 kubenswrapper[2564]: W0730 00:17:48.869676 2564 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 30 00:17:48.872449 ip-10-0-142-6 kubenswrapper[2564]: W0730 00:17:48.869679 2564 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 30 00:17:48.872449 ip-10-0-142-6 kubenswrapper[2564]: W0730 00:17:48.869682 2564 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 30 00:17:48.872449 ip-10-0-142-6 kubenswrapper[2564]: W0730 00:17:48.869684 2564 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 30 00:17:48.872449 ip-10-0-142-6 kubenswrapper[2564]: W0730 00:17:48.869686 2564 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 30 00:17:48.872449 ip-10-0-142-6 kubenswrapper[2564]: W0730 00:17:48.869690 2564 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 30 00:17:48.872449 ip-10-0-142-6 kubenswrapper[2564]: W0730 00:17:48.869694 2564 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 30 00:17:48.872449 ip-10-0-142-6 kubenswrapper[2564]: W0730 00:17:48.869697 2564 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 30 00:17:48.872449 ip-10-0-142-6 kubenswrapper[2564]: W0730 00:17:48.869701 2564 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 30 00:17:48.872449 ip-10-0-142-6 kubenswrapper[2564]: W0730 00:17:48.869703 2564 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 30 00:17:48.872955 ip-10-0-142-6 kubenswrapper[2564]: W0730 00:17:48.869708 2564 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 30 00:17:48.872955 ip-10-0-142-6 kubenswrapper[2564]: W0730 00:17:48.870137 2564 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 30 00:17:48.872955 ip-10-0-142-6 kubenswrapper[2564]: W0730 00:17:48.870143 2564 feature_gate.go:328] unrecognized feature gate: Example Jul 30 00:17:48.872955 ip-10-0-142-6 kubenswrapper[2564]: W0730 00:17:48.870146 2564 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 30 00:17:48.872955 ip-10-0-142-6 kubenswrapper[2564]: W0730 00:17:48.870149 2564 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 30 00:17:48.872955 ip-10-0-142-6 kubenswrapper[2564]: W0730 00:17:48.870152 2564 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 30 00:17:48.872955 ip-10-0-142-6 kubenswrapper[2564]: W0730 00:17:48.870155 2564 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 30 00:17:48.872955 ip-10-0-142-6 kubenswrapper[2564]: W0730 00:17:48.870158 2564 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 30 00:17:48.872955 ip-10-0-142-6 kubenswrapper[2564]: W0730 00:17:48.870160 2564 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 30 00:17:48.872955 ip-10-0-142-6 kubenswrapper[2564]: W0730 00:17:48.870163 2564 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 30 00:17:48.872955 ip-10-0-142-6 kubenswrapper[2564]: W0730 00:17:48.870165 2564 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 30 00:17:48.872955 ip-10-0-142-6 kubenswrapper[2564]: W0730 00:17:48.870168 2564 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 30 00:17:48.872955 ip-10-0-142-6 kubenswrapper[2564]: W0730 00:17:48.870170 2564 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 30 00:17:48.872955 ip-10-0-142-6 kubenswrapper[2564]: W0730 00:17:48.870174 2564 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 30 00:17:48.872955 ip-10-0-142-6 kubenswrapper[2564]: W0730 00:17:48.870178 2564 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 30 00:17:48.872955 ip-10-0-142-6 kubenswrapper[2564]: W0730 00:17:48.870181 2564 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 30 00:17:48.872955 ip-10-0-142-6 kubenswrapper[2564]: W0730 00:17:48.870183 2564 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 30 00:17:48.872955 ip-10-0-142-6 kubenswrapper[2564]: W0730 00:17:48.870186 2564 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 30 00:17:48.873425 ip-10-0-142-6 kubenswrapper[2564]: W0730 00:17:48.870189 2564 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 30 00:17:48.873425 ip-10-0-142-6 kubenswrapper[2564]: W0730 00:17:48.870192 2564 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 30 00:17:48.873425 ip-10-0-142-6 kubenswrapper[2564]: W0730 00:17:48.870195 2564 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 30 00:17:48.873425 ip-10-0-142-6 kubenswrapper[2564]: W0730 00:17:48.870198 2564 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 30 00:17:48.873425 ip-10-0-142-6 kubenswrapper[2564]: W0730 00:17:48.870200 2564 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 30 00:17:48.873425 ip-10-0-142-6 kubenswrapper[2564]: W0730 00:17:48.870203 2564 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 30 00:17:48.873425 ip-10-0-142-6 kubenswrapper[2564]: W0730 00:17:48.870207 2564 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 30 00:17:48.873425 ip-10-0-142-6 kubenswrapper[2564]: W0730 00:17:48.870209 2564 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 30 00:17:48.873425 ip-10-0-142-6 kubenswrapper[2564]: W0730 00:17:48.870212 2564 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 30 00:17:48.873425 ip-10-0-142-6 kubenswrapper[2564]: W0730 00:17:48.870215 2564 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 30 00:17:48.873425 ip-10-0-142-6 kubenswrapper[2564]: W0730 00:17:48.870218 2564 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 30 00:17:48.873425 ip-10-0-142-6 kubenswrapper[2564]: W0730 00:17:48.870221 2564 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 30 00:17:48.873425 ip-10-0-142-6 kubenswrapper[2564]: W0730 00:17:48.870225 2564 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 30 00:17:48.873425 ip-10-0-142-6 kubenswrapper[2564]: W0730 00:17:48.870228 2564 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 30 00:17:48.873425 ip-10-0-142-6 kubenswrapper[2564]: W0730 00:17:48.870230 2564 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 30 00:17:48.873425 ip-10-0-142-6 kubenswrapper[2564]: W0730 00:17:48.870234 2564 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 30 00:17:48.873425 ip-10-0-142-6 kubenswrapper[2564]: W0730 00:17:48.870237 2564 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 30 00:17:48.873425 ip-10-0-142-6 kubenswrapper[2564]: W0730 00:17:48.870240 2564 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 30 00:17:48.873425 ip-10-0-142-6 kubenswrapper[2564]: W0730 00:17:48.870243 2564 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 30 00:17:48.873425 ip-10-0-142-6 kubenswrapper[2564]: W0730 00:17:48.870245 2564 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 30 00:17:48.873909 ip-10-0-142-6 kubenswrapper[2564]: W0730 00:17:48.870248 2564 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 30 00:17:48.873909 ip-10-0-142-6 kubenswrapper[2564]: W0730 00:17:48.870250 2564 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 30 00:17:48.873909 ip-10-0-142-6 kubenswrapper[2564]: W0730 00:17:48.870253 2564 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 30 00:17:48.873909 ip-10-0-142-6 kubenswrapper[2564]: W0730 00:17:48.870255 2564 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 30 00:17:48.873909 ip-10-0-142-6 kubenswrapper[2564]: W0730 00:17:48.870264 2564 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 30 00:17:48.873909 ip-10-0-142-6 kubenswrapper[2564]: W0730 00:17:48.870267 2564 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 30 00:17:48.873909 ip-10-0-142-6 kubenswrapper[2564]: W0730 00:17:48.870269 2564 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 30 00:17:48.873909 ip-10-0-142-6 kubenswrapper[2564]: W0730 00:17:48.870272 2564 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 30 00:17:48.873909 ip-10-0-142-6 kubenswrapper[2564]: W0730 00:17:48.870275 2564 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 30 00:17:48.873909 ip-10-0-142-6 kubenswrapper[2564]: W0730 00:17:48.870277 2564 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 30 00:17:48.873909 ip-10-0-142-6 kubenswrapper[2564]: W0730 00:17:48.870279 2564 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 30 00:17:48.873909 ip-10-0-142-6 kubenswrapper[2564]: W0730 00:17:48.870282 2564 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 30 00:17:48.873909 ip-10-0-142-6 kubenswrapper[2564]: W0730 00:17:48.870284 2564 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 30 00:17:48.873909 ip-10-0-142-6 kubenswrapper[2564]: W0730 00:17:48.870286 2564 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 30 00:17:48.873909 ip-10-0-142-6 kubenswrapper[2564]: W0730 00:17:48.870289 2564 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 30 00:17:48.873909 ip-10-0-142-6 kubenswrapper[2564]: W0730 00:17:48.870291 2564 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 30 00:17:48.873909 ip-10-0-142-6 kubenswrapper[2564]: W0730 00:17:48.870294 2564 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 30 00:17:48.873909 ip-10-0-142-6 kubenswrapper[2564]: W0730 00:17:48.870296 2564 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 30 00:17:48.873909 ip-10-0-142-6 kubenswrapper[2564]: W0730 00:17:48.870299 2564 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 30 00:17:48.873909 ip-10-0-142-6 kubenswrapper[2564]: W0730 00:17:48.870302 2564 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 30 00:17:48.874444 ip-10-0-142-6 kubenswrapper[2564]: W0730 00:17:48.870305 2564 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 30 00:17:48.874444 ip-10-0-142-6 kubenswrapper[2564]: W0730 00:17:48.870308 2564 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 30 00:17:48.874444 ip-10-0-142-6 kubenswrapper[2564]: W0730 00:17:48.870310 2564 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 30 00:17:48.874444 ip-10-0-142-6 kubenswrapper[2564]: W0730 00:17:48.870314 2564 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 30 00:17:48.874444 ip-10-0-142-6 kubenswrapper[2564]: W0730 00:17:48.870317 2564 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 30 00:17:48.874444 ip-10-0-142-6 kubenswrapper[2564]: W0730 00:17:48.870319 2564 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 30 00:17:48.874444 ip-10-0-142-6 kubenswrapper[2564]: W0730 00:17:48.870322 2564 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 30 00:17:48.874444 ip-10-0-142-6 kubenswrapper[2564]: W0730 00:17:48.870324 2564 feature_gate.go:328] unrecognized feature gate: Example2 Jul 30 00:17:48.874444 ip-10-0-142-6 kubenswrapper[2564]: W0730 00:17:48.870327 2564 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 30 00:17:48.874444 ip-10-0-142-6 kubenswrapper[2564]: W0730 00:17:48.870329 2564 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 30 00:17:48.874444 ip-10-0-142-6 kubenswrapper[2564]: W0730 00:17:48.870332 2564 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 30 00:17:48.874444 ip-10-0-142-6 kubenswrapper[2564]: W0730 00:17:48.870334 2564 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 30 00:17:48.874444 ip-10-0-142-6 kubenswrapper[2564]: W0730 00:17:48.870337 2564 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 30 00:17:48.874444 ip-10-0-142-6 kubenswrapper[2564]: W0730 00:17:48.870340 2564 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 30 00:17:48.874444 ip-10-0-142-6 kubenswrapper[2564]: W0730 00:17:48.870342 2564 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 30 00:17:48.874444 ip-10-0-142-6 kubenswrapper[2564]: W0730 00:17:48.870344 2564 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 30 00:17:48.874444 ip-10-0-142-6 kubenswrapper[2564]: W0730 00:17:48.870348 2564 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 30 00:17:48.874444 ip-10-0-142-6 kubenswrapper[2564]: W0730 00:17:48.870351 2564 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 30 00:17:48.874444 ip-10-0-142-6 kubenswrapper[2564]: W0730 00:17:48.870353 2564 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 30 00:17:48.874444 ip-10-0-142-6 kubenswrapper[2564]: W0730 00:17:48.870355 2564 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 30 00:17:48.874943 ip-10-0-142-6 kubenswrapper[2564]: W0730 00:17:48.870358 2564 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 30 00:17:48.874943 ip-10-0-142-6 kubenswrapper[2564]: W0730 00:17:48.870361 2564 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 30 00:17:48.874943 ip-10-0-142-6 kubenswrapper[2564]: W0730 00:17:48.870363 2564 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 30 00:17:48.874943 ip-10-0-142-6 kubenswrapper[2564]: W0730 00:17:48.870365 2564 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 30 00:17:48.874943 ip-10-0-142-6 kubenswrapper[2564]: W0730 00:17:48.870368 2564 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 30 00:17:48.874943 ip-10-0-142-6 kubenswrapper[2564]: W0730 00:17:48.870370 2564 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 30 00:17:48.874943 ip-10-0-142-6 kubenswrapper[2564]: W0730 00:17:48.870373 2564 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 30 00:17:48.874943 ip-10-0-142-6 kubenswrapper[2564]: W0730 00:17:48.870376 2564 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 30 00:17:48.874943 ip-10-0-142-6 kubenswrapper[2564]: W0730 00:17:48.870379 2564 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 30 00:17:48.874943 ip-10-0-142-6 kubenswrapper[2564]: W0730 00:17:48.870381 2564 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 30 00:17:48.874943 ip-10-0-142-6 kubenswrapper[2564]: W0730 00:17:48.870385 2564 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 30 00:17:48.874943 ip-10-0-142-6 kubenswrapper[2564]: W0730 00:17:48.870387 2564 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 30 00:17:48.874943 ip-10-0-142-6 kubenswrapper[2564]: W0730 00:17:48.870390 2564 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 30 00:17:48.874943 ip-10-0-142-6 kubenswrapper[2564]: W0730 00:17:48.870394 2564 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 30 00:17:48.874943 ip-10-0-142-6 kubenswrapper[2564]: W0730 00:17:48.870397 2564 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 30 00:17:48.874943 ip-10-0-142-6 kubenswrapper[2564]: W0730 00:17:48.870400 2564 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 30 00:17:48.874943 ip-10-0-142-6 kubenswrapper[2564]: W0730 00:17:48.870403 2564 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 30 00:17:48.874943 ip-10-0-142-6 kubenswrapper[2564]: W0730 00:17:48.870406 2564 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 30 00:17:48.874943 ip-10-0-142-6 kubenswrapper[2564]: W0730 00:17:48.870409 2564 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 30 00:17:48.875419 ip-10-0-142-6 kubenswrapper[2564]: W0730 00:17:48.870412 2564 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 30 00:17:48.875419 ip-10-0-142-6 kubenswrapper[2564]: W0730 00:17:48.870415 2564 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 30 00:17:48.875419 ip-10-0-142-6 kubenswrapper[2564]: W0730 00:17:48.870417 2564 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 30 00:17:48.875419 ip-10-0-142-6 kubenswrapper[2564]: I0730 00:17:48.870492 2564 flags.go:64] FLAG: --address="0.0.0.0" Jul 30 00:17:48.875419 ip-10-0-142-6 kubenswrapper[2564]: I0730 00:17:48.870499 2564 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]" Jul 30 00:17:48.875419 ip-10-0-142-6 kubenswrapper[2564]: I0730 00:17:48.870506 2564 flags.go:64] FLAG: --anonymous-auth="true" Jul 30 00:17:48.875419 ip-10-0-142-6 kubenswrapper[2564]: I0730 00:17:48.870511 2564 flags.go:64] FLAG: --application-metrics-count-limit="100" Jul 30 00:17:48.875419 ip-10-0-142-6 kubenswrapper[2564]: I0730 00:17:48.870515 2564 flags.go:64] FLAG: --authentication-token-webhook="false" Jul 30 00:17:48.875419 ip-10-0-142-6 kubenswrapper[2564]: I0730 00:17:48.870518 2564 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s" Jul 30 00:17:48.875419 ip-10-0-142-6 kubenswrapper[2564]: I0730 00:17:48.870522 2564 flags.go:64] FLAG: --authorization-mode="AlwaysAllow" Jul 30 00:17:48.875419 ip-10-0-142-6 kubenswrapper[2564]: I0730 00:17:48.870527 2564 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s" Jul 30 00:17:48.875419 ip-10-0-142-6 kubenswrapper[2564]: I0730 00:17:48.870530 2564 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s" Jul 30 00:17:48.875419 ip-10-0-142-6 kubenswrapper[2564]: I0730 00:17:48.870533 2564 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id" Jul 30 00:17:48.875419 ip-10-0-142-6 kubenswrapper[2564]: I0730 00:17:48.870536 2564 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig" Jul 30 00:17:48.875419 ip-10-0-142-6 kubenswrapper[2564]: I0730 00:17:48.870540 2564 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki" Jul 30 00:17:48.875419 ip-10-0-142-6 kubenswrapper[2564]: I0730 00:17:48.870543 2564 flags.go:64] FLAG: --cgroup-driver="cgroupfs" Jul 30 00:17:48.875419 ip-10-0-142-6 kubenswrapper[2564]: I0730 00:17:48.870546 2564 flags.go:64] FLAG: --cgroup-root="" Jul 30 00:17:48.875419 ip-10-0-142-6 kubenswrapper[2564]: I0730 00:17:48.870549 2564 flags.go:64] FLAG: --cgroups-per-qos="true" Jul 30 00:17:48.875419 ip-10-0-142-6 kubenswrapper[2564]: I0730 00:17:48.870552 2564 flags.go:64] FLAG: --client-ca-file="" Jul 30 00:17:48.875419 ip-10-0-142-6 kubenswrapper[2564]: I0730 00:17:48.870555 2564 flags.go:64] FLAG: --cloud-provider="external" Jul 30 00:17:48.875419 ip-10-0-142-6 kubenswrapper[2564]: I0730 00:17:48.870558 2564 flags.go:64] FLAG: --cluster-dns="[]" Jul 30 00:17:48.875419 ip-10-0-142-6 kubenswrapper[2564]: I0730 00:17:48.870562 2564 flags.go:64] FLAG: --cluster-domain="" Jul 30 00:17:48.875419 ip-10-0-142-6 kubenswrapper[2564]: I0730 00:17:48.870565 2564 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf" Jul 30 00:17:48.875419 ip-10-0-142-6 kubenswrapper[2564]: I0730 00:17:48.870568 2564 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d" Jul 30 00:17:48.876025 ip-10-0-142-6 kubenswrapper[2564]: I0730 00:17:48.870571 2564 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json" Jul 30 00:17:48.876025 ip-10-0-142-6 kubenswrapper[2564]: I0730 00:17:48.870575 2564 flags.go:64] FLAG: --container-log-max-files="5" Jul 30 00:17:48.876025 ip-10-0-142-6 kubenswrapper[2564]: I0730 00:17:48.870579 2564 flags.go:64] FLAG: --container-log-max-size="10Mi" Jul 30 00:17:48.876025 ip-10-0-142-6 kubenswrapper[2564]: I0730 00:17:48.870582 2564 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock" Jul 30 00:17:48.876025 ip-10-0-142-6 kubenswrapper[2564]: I0730 00:17:48.870586 2564 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock" Jul 30 00:17:48.876025 ip-10-0-142-6 kubenswrapper[2564]: I0730 00:17:48.870589 2564 flags.go:64] FLAG: --containerd-namespace="k8s.io" Jul 30 00:17:48.876025 ip-10-0-142-6 kubenswrapper[2564]: I0730 00:17:48.870592 2564 flags.go:64] FLAG: --contention-profiling="false" Jul 30 00:17:48.876025 ip-10-0-142-6 kubenswrapper[2564]: I0730 00:17:48.870595 2564 flags.go:64] FLAG: --cpu-cfs-quota="true" Jul 30 00:17:48.876025 ip-10-0-142-6 kubenswrapper[2564]: I0730 00:17:48.870598 2564 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms" Jul 30 00:17:48.876025 ip-10-0-142-6 kubenswrapper[2564]: I0730 00:17:48.870601 2564 flags.go:64] FLAG: --cpu-manager-policy="none" Jul 30 00:17:48.876025 ip-10-0-142-6 kubenswrapper[2564]: I0730 00:17:48.870604 2564 flags.go:64] FLAG: --cpu-manager-policy-options="" Jul 30 00:17:48.876025 ip-10-0-142-6 kubenswrapper[2564]: I0730 00:17:48.870608 2564 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s" Jul 30 00:17:48.876025 ip-10-0-142-6 kubenswrapper[2564]: I0730 00:17:48.870611 2564 flags.go:64] FLAG: --enable-controller-attach-detach="true" Jul 30 00:17:48.876025 ip-10-0-142-6 kubenswrapper[2564]: I0730 00:17:48.870614 2564 flags.go:64] FLAG: --enable-debugging-handlers="true" Jul 30 00:17:48.876025 ip-10-0-142-6 kubenswrapper[2564]: I0730 00:17:48.870617 2564 flags.go:64] FLAG: --enable-load-reader="false" Jul 30 00:17:48.876025 ip-10-0-142-6 kubenswrapper[2564]: I0730 00:17:48.870619 2564 flags.go:64] FLAG: --enable-server="true" Jul 30 00:17:48.876025 ip-10-0-142-6 kubenswrapper[2564]: I0730 00:17:48.870622 2564 flags.go:64] FLAG: --enforce-node-allocatable="[pods]" Jul 30 00:17:48.876025 ip-10-0-142-6 kubenswrapper[2564]: I0730 00:17:48.870628 2564 flags.go:64] FLAG: --event-burst="100" Jul 30 00:17:48.876025 ip-10-0-142-6 kubenswrapper[2564]: I0730 00:17:48.870631 2564 flags.go:64] FLAG: --event-qps="50" Jul 30 00:17:48.876025 ip-10-0-142-6 kubenswrapper[2564]: I0730 00:17:48.870634 2564 flags.go:64] FLAG: --event-storage-age-limit="default=0" Jul 30 00:17:48.876025 ip-10-0-142-6 kubenswrapper[2564]: I0730 00:17:48.870637 2564 flags.go:64] FLAG: --event-storage-event-limit="default=0" Jul 30 00:17:48.876025 ip-10-0-142-6 kubenswrapper[2564]: I0730 00:17:48.870639 2564 flags.go:64] FLAG: --eviction-hard="" Jul 30 00:17:48.876025 ip-10-0-142-6 kubenswrapper[2564]: I0730 00:17:48.870643 2564 flags.go:64] FLAG: --eviction-max-pod-grace-period="0" Jul 30 00:17:48.876025 ip-10-0-142-6 kubenswrapper[2564]: I0730 00:17:48.870646 2564 flags.go:64] FLAG: --eviction-minimum-reclaim="" Jul 30 00:17:48.876621 ip-10-0-142-6 kubenswrapper[2564]: I0730 00:17:48.870649 2564 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s" Jul 30 00:17:48.876621 ip-10-0-142-6 kubenswrapper[2564]: I0730 00:17:48.870652 2564 flags.go:64] FLAG: --eviction-soft="" Jul 30 00:17:48.876621 ip-10-0-142-6 kubenswrapper[2564]: I0730 00:17:48.870655 2564 flags.go:64] FLAG: --eviction-soft-grace-period="" Jul 30 00:17:48.876621 ip-10-0-142-6 kubenswrapper[2564]: I0730 00:17:48.870658 2564 flags.go:64] FLAG: --exit-on-lock-contention="false" Jul 30 00:17:48.876621 ip-10-0-142-6 kubenswrapper[2564]: I0730 00:17:48.870662 2564 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false" Jul 30 00:17:48.876621 ip-10-0-142-6 kubenswrapper[2564]: I0730 00:17:48.870664 2564 flags.go:64] FLAG: --experimental-mounter-path="" Jul 30 00:17:48.876621 ip-10-0-142-6 kubenswrapper[2564]: I0730 00:17:48.870667 2564 flags.go:64] FLAG: --fail-cgroupv1="false" Jul 30 00:17:48.876621 ip-10-0-142-6 kubenswrapper[2564]: I0730 00:17:48.870670 2564 flags.go:64] FLAG: --fail-swap-on="true" Jul 30 00:17:48.876621 ip-10-0-142-6 kubenswrapper[2564]: I0730 00:17:48.870673 2564 flags.go:64] FLAG: --feature-gates="" Jul 30 00:17:48.876621 ip-10-0-142-6 kubenswrapper[2564]: I0730 00:17:48.870677 2564 flags.go:64] FLAG: --file-check-frequency="20s" Jul 30 00:17:48.876621 ip-10-0-142-6 kubenswrapper[2564]: I0730 00:17:48.870680 2564 flags.go:64] FLAG: --global-housekeeping-interval="1m0s" Jul 30 00:17:48.876621 ip-10-0-142-6 kubenswrapper[2564]: I0730 00:17:48.870683 2564 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge" Jul 30 00:17:48.876621 ip-10-0-142-6 kubenswrapper[2564]: I0730 00:17:48.870687 2564 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1" Jul 30 00:17:48.876621 ip-10-0-142-6 kubenswrapper[2564]: I0730 00:17:48.870690 2564 flags.go:64] FLAG: --healthz-port="10248" Jul 30 00:17:48.876621 ip-10-0-142-6 kubenswrapper[2564]: I0730 00:17:48.870693 2564 flags.go:64] FLAG: --help="false" Jul 30 00:17:48.876621 ip-10-0-142-6 kubenswrapper[2564]: I0730 00:17:48.870696 2564 flags.go:64] FLAG: --hostname-override="ip-10-0-142-6.ec2.internal" Jul 30 00:17:48.876621 ip-10-0-142-6 kubenswrapper[2564]: I0730 00:17:48.870699 2564 flags.go:64] FLAG: --housekeeping-interval="10s" Jul 30 00:17:48.876621 ip-10-0-142-6 kubenswrapper[2564]: I0730 00:17:48.870702 2564 flags.go:64] FLAG: --http-check-frequency="20s" Jul 30 00:17:48.876621 ip-10-0-142-6 kubenswrapper[2564]: I0730 00:17:48.870705 2564 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins" Jul 30 00:17:48.876621 ip-10-0-142-6 kubenswrapper[2564]: I0730 00:17:48.870708 2564 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml" Jul 30 00:17:48.876621 ip-10-0-142-6 kubenswrapper[2564]: I0730 00:17:48.870712 2564 flags.go:64] FLAG: --image-gc-high-threshold="85" Jul 30 00:17:48.876621 ip-10-0-142-6 kubenswrapper[2564]: I0730 00:17:48.870715 2564 flags.go:64] FLAG: --image-gc-low-threshold="80" Jul 30 00:17:48.876621 ip-10-0-142-6 kubenswrapper[2564]: I0730 00:17:48.870718 2564 flags.go:64] FLAG: --image-service-endpoint="" Jul 30 00:17:48.877279 ip-10-0-142-6 kubenswrapper[2564]: I0730 00:17:48.870721 2564 flags.go:64] FLAG: --kernel-memcg-notification="false" Jul 30 00:17:48.877279 ip-10-0-142-6 kubenswrapper[2564]: I0730 00:17:48.870723 2564 flags.go:64] FLAG: --kube-api-burst="100" Jul 30 00:17:48.877279 ip-10-0-142-6 kubenswrapper[2564]: I0730 00:17:48.870726 2564 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf" Jul 30 00:17:48.877279 ip-10-0-142-6 kubenswrapper[2564]: I0730 00:17:48.870730 2564 flags.go:64] FLAG: --kube-api-qps="50" Jul 30 00:17:48.877279 ip-10-0-142-6 kubenswrapper[2564]: I0730 00:17:48.870732 2564 flags.go:64] FLAG: --kube-reserved="" Jul 30 00:17:48.877279 ip-10-0-142-6 kubenswrapper[2564]: I0730 00:17:48.870735 2564 flags.go:64] FLAG: --kube-reserved-cgroup="" Jul 30 00:17:48.877279 ip-10-0-142-6 kubenswrapper[2564]: I0730 00:17:48.870738 2564 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig" Jul 30 00:17:48.877279 ip-10-0-142-6 kubenswrapper[2564]: I0730 00:17:48.870741 2564 flags.go:64] FLAG: --kubelet-cgroups="" Jul 30 00:17:48.877279 ip-10-0-142-6 kubenswrapper[2564]: I0730 00:17:48.870744 2564 flags.go:64] FLAG: --local-storage-capacity-isolation="true" Jul 30 00:17:48.877279 ip-10-0-142-6 kubenswrapper[2564]: I0730 00:17:48.870746 2564 flags.go:64] FLAG: --lock-file="" Jul 30 00:17:48.877279 ip-10-0-142-6 kubenswrapper[2564]: I0730 00:17:48.870749 2564 flags.go:64] FLAG: --log-cadvisor-usage="false" Jul 30 00:17:48.877279 ip-10-0-142-6 kubenswrapper[2564]: I0730 00:17:48.870752 2564 flags.go:64] FLAG: --log-flush-frequency="5s" Jul 30 00:17:48.877279 ip-10-0-142-6 kubenswrapper[2564]: I0730 00:17:48.870755 2564 flags.go:64] FLAG: --log-json-info-buffer-size="0" Jul 30 00:17:48.877279 ip-10-0-142-6 kubenswrapper[2564]: I0730 00:17:48.870763 2564 flags.go:64] FLAG: --log-json-split-stream="false" Jul 30 00:17:48.877279 ip-10-0-142-6 kubenswrapper[2564]: I0730 00:17:48.870765 2564 flags.go:64] FLAG: --log-text-info-buffer-size="0" Jul 30 00:17:48.877279 ip-10-0-142-6 kubenswrapper[2564]: I0730 00:17:48.870768 2564 flags.go:64] FLAG: --log-text-split-stream="false" Jul 30 00:17:48.877279 ip-10-0-142-6 kubenswrapper[2564]: I0730 00:17:48.870771 2564 flags.go:64] FLAG: --logging-format="text" Jul 30 00:17:48.877279 ip-10-0-142-6 kubenswrapper[2564]: I0730 00:17:48.870774 2564 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id" Jul 30 00:17:48.877279 ip-10-0-142-6 kubenswrapper[2564]: I0730 00:17:48.870778 2564 flags.go:64] FLAG: --make-iptables-util-chains="true" Jul 30 00:17:48.877279 ip-10-0-142-6 kubenswrapper[2564]: I0730 00:17:48.870780 2564 flags.go:64] FLAG: --manifest-url="" Jul 30 00:17:48.877279 ip-10-0-142-6 kubenswrapper[2564]: I0730 00:17:48.870783 2564 flags.go:64] FLAG: --manifest-url-header="" Jul 30 00:17:48.877279 ip-10-0-142-6 kubenswrapper[2564]: I0730 00:17:48.870789 2564 flags.go:64] FLAG: --max-housekeeping-interval="15s" Jul 30 00:17:48.877279 ip-10-0-142-6 kubenswrapper[2564]: I0730 00:17:48.870792 2564 flags.go:64] FLAG: --max-open-files="1000000" Jul 30 00:17:48.877279 ip-10-0-142-6 kubenswrapper[2564]: I0730 00:17:48.870796 2564 flags.go:64] FLAG: --max-pods="110" Jul 30 00:17:48.877279 ip-10-0-142-6 kubenswrapper[2564]: I0730 00:17:48.870799 2564 flags.go:64] FLAG: --maximum-dead-containers="-1" Jul 30 00:17:48.877894 ip-10-0-142-6 kubenswrapper[2564]: I0730 00:17:48.870802 2564 flags.go:64] FLAG: --maximum-dead-containers-per-container="1" Jul 30 00:17:48.877894 ip-10-0-142-6 kubenswrapper[2564]: I0730 00:17:48.870805 2564 flags.go:64] FLAG: --memory-manager-policy="None" Jul 30 00:17:48.877894 ip-10-0-142-6 kubenswrapper[2564]: I0730 00:17:48.870808 2564 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s" Jul 30 00:17:48.877894 ip-10-0-142-6 kubenswrapper[2564]: I0730 00:17:48.870811 2564 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s" Jul 30 00:17:48.877894 ip-10-0-142-6 kubenswrapper[2564]: I0730 00:17:48.870814 2564 flags.go:64] FLAG: --node-ip="0.0.0.0" Jul 30 00:17:48.877894 ip-10-0-142-6 kubenswrapper[2564]: I0730 00:17:48.870817 2564 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel" Jul 30 00:17:48.877894 ip-10-0-142-6 kubenswrapper[2564]: I0730 00:17:48.870824 2564 flags.go:64] FLAG: --node-status-max-images="50" Jul 30 00:17:48.877894 ip-10-0-142-6 kubenswrapper[2564]: I0730 00:17:48.870827 2564 flags.go:64] FLAG: --node-status-update-frequency="10s" Jul 30 00:17:48.877894 ip-10-0-142-6 kubenswrapper[2564]: I0730 00:17:48.870830 2564 flags.go:64] FLAG: --oom-score-adj="-999" Jul 30 00:17:48.877894 ip-10-0-142-6 kubenswrapper[2564]: I0730 00:17:48.870833 2564 flags.go:64] FLAG: --pod-cidr="" Jul 30 00:17:48.877894 ip-10-0-142-6 kubenswrapper[2564]: I0730 00:17:48.870836 2564 flags.go:64] FLAG: --pod-infra-container-image="quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:7505336a406394b8c1c77d14d890a9f628aa4cfff172312179cb9af8c28c30bb" Jul 30 00:17:48.877894 ip-10-0-142-6 kubenswrapper[2564]: I0730 00:17:48.870841 2564 flags.go:64] FLAG: --pod-manifest-path="" Jul 30 00:17:48.877894 ip-10-0-142-6 kubenswrapper[2564]: I0730 00:17:48.870844 2564 flags.go:64] FLAG: --pod-max-pids="-1" Jul 30 00:17:48.877894 ip-10-0-142-6 kubenswrapper[2564]: I0730 00:17:48.870847 2564 flags.go:64] FLAG: --pods-per-core="0" Jul 30 00:17:48.877894 ip-10-0-142-6 kubenswrapper[2564]: I0730 00:17:48.870850 2564 flags.go:64] FLAG: --port="10250" Jul 30 00:17:48.877894 ip-10-0-142-6 kubenswrapper[2564]: I0730 00:17:48.870853 2564 flags.go:64] FLAG: --protect-kernel-defaults="false" Jul 30 00:17:48.877894 ip-10-0-142-6 kubenswrapper[2564]: I0730 00:17:48.870856 2564 flags.go:64] FLAG: --provider-id="aws:///us-east-1a/i-04ab0dccbcbfa1607" Jul 30 00:17:48.877894 ip-10-0-142-6 kubenswrapper[2564]: I0730 00:17:48.870860 2564 flags.go:64] FLAG: --qos-reserved="" Jul 30 00:17:48.877894 ip-10-0-142-6 kubenswrapper[2564]: I0730 00:17:48.870862 2564 flags.go:64] FLAG: --read-only-port="10255" Jul 30 00:17:48.877894 ip-10-0-142-6 kubenswrapper[2564]: I0730 00:17:48.870865 2564 flags.go:64] FLAG: --register-node="true" Jul 30 00:17:48.877894 ip-10-0-142-6 kubenswrapper[2564]: I0730 00:17:48.870871 2564 flags.go:64] FLAG: --register-with-taints="" Jul 30 00:17:48.877894 ip-10-0-142-6 kubenswrapper[2564]: I0730 00:17:48.870874 2564 flags.go:64] FLAG: --registry-burst="10" Jul 30 00:17:48.877894 ip-10-0-142-6 kubenswrapper[2564]: I0730 00:17:48.870877 2564 flags.go:64] FLAG: --registry-qps="5" Jul 30 00:17:48.877894 ip-10-0-142-6 kubenswrapper[2564]: I0730 00:17:48.870880 2564 flags.go:64] FLAG: --reserved-cpus="" Jul 30 00:17:48.878484 ip-10-0-142-6 kubenswrapper[2564]: I0730 00:17:48.870883 2564 flags.go:64] FLAG: --reserved-memory="" Jul 30 00:17:48.878484 ip-10-0-142-6 kubenswrapper[2564]: I0730 00:17:48.870887 2564 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf" Jul 30 00:17:48.878484 ip-10-0-142-6 kubenswrapper[2564]: I0730 00:17:48.870890 2564 flags.go:64] FLAG: --root-dir="/var/lib/kubelet" Jul 30 00:17:48.878484 ip-10-0-142-6 kubenswrapper[2564]: I0730 00:17:48.870893 2564 flags.go:64] FLAG: --rotate-certificates="false" Jul 30 00:17:48.878484 ip-10-0-142-6 kubenswrapper[2564]: I0730 00:17:48.870897 2564 flags.go:64] FLAG: --rotate-server-certificates="false" Jul 30 00:17:48.878484 ip-10-0-142-6 kubenswrapper[2564]: I0730 00:17:48.870900 2564 flags.go:64] FLAG: --runonce="false" Jul 30 00:17:48.878484 ip-10-0-142-6 kubenswrapper[2564]: I0730 00:17:48.870903 2564 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service" Jul 30 00:17:48.878484 ip-10-0-142-6 kubenswrapper[2564]: I0730 00:17:48.870906 2564 flags.go:64] FLAG: --runtime-request-timeout="2m0s" Jul 30 00:17:48.878484 ip-10-0-142-6 kubenswrapper[2564]: I0730 00:17:48.870909 2564 flags.go:64] FLAG: --seccomp-default="false" Jul 30 00:17:48.878484 ip-10-0-142-6 kubenswrapper[2564]: I0730 00:17:48.870912 2564 flags.go:64] FLAG: --serialize-image-pulls="true" Jul 30 00:17:48.878484 ip-10-0-142-6 kubenswrapper[2564]: I0730 00:17:48.870915 2564 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s" Jul 30 00:17:48.878484 ip-10-0-142-6 kubenswrapper[2564]: I0730 00:17:48.870918 2564 flags.go:64] FLAG: --storage-driver-db="cadvisor" Jul 30 00:17:48.878484 ip-10-0-142-6 kubenswrapper[2564]: I0730 00:17:48.870921 2564 flags.go:64] FLAG: --storage-driver-host="localhost:8086" Jul 30 00:17:48.878484 ip-10-0-142-6 kubenswrapper[2564]: I0730 00:17:48.870924 2564 flags.go:64] FLAG: --storage-driver-password="root" Jul 30 00:17:48.878484 ip-10-0-142-6 kubenswrapper[2564]: I0730 00:17:48.870927 2564 flags.go:64] FLAG: --storage-driver-secure="false" Jul 30 00:17:48.878484 ip-10-0-142-6 kubenswrapper[2564]: I0730 00:17:48.870929 2564 flags.go:64] FLAG: --storage-driver-table="stats" Jul 30 00:17:48.878484 ip-10-0-142-6 kubenswrapper[2564]: I0730 00:17:48.870932 2564 flags.go:64] FLAG: --storage-driver-user="root" Jul 30 00:17:48.878484 ip-10-0-142-6 kubenswrapper[2564]: I0730 00:17:48.870935 2564 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s" Jul 30 00:17:48.878484 ip-10-0-142-6 kubenswrapper[2564]: I0730 00:17:48.870938 2564 flags.go:64] FLAG: --sync-frequency="1m0s" Jul 30 00:17:48.878484 ip-10-0-142-6 kubenswrapper[2564]: I0730 00:17:48.870941 2564 flags.go:64] FLAG: --system-cgroups="" Jul 30 00:17:48.878484 ip-10-0-142-6 kubenswrapper[2564]: I0730 00:17:48.870943 2564 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi" Jul 30 00:17:48.878484 ip-10-0-142-6 kubenswrapper[2564]: I0730 00:17:48.870949 2564 flags.go:64] FLAG: --system-reserved-cgroup="" Jul 30 00:17:48.878484 ip-10-0-142-6 kubenswrapper[2564]: I0730 00:17:48.870951 2564 flags.go:64] FLAG: --tls-cert-file="" Jul 30 00:17:48.878484 ip-10-0-142-6 kubenswrapper[2564]: I0730 00:17:48.870954 2564 flags.go:64] FLAG: --tls-cipher-suites="[]" Jul 30 00:17:48.878484 ip-10-0-142-6 kubenswrapper[2564]: I0730 00:17:48.870958 2564 flags.go:64] FLAG: --tls-min-version="" Jul 30 00:17:48.879093 ip-10-0-142-6 kubenswrapper[2564]: I0730 00:17:48.870961 2564 flags.go:64] FLAG: --tls-private-key-file="" Jul 30 00:17:48.879093 ip-10-0-142-6 kubenswrapper[2564]: I0730 00:17:48.870963 2564 flags.go:64] FLAG: --topology-manager-policy="none" Jul 30 00:17:48.879093 ip-10-0-142-6 kubenswrapper[2564]: I0730 00:17:48.870966 2564 flags.go:64] FLAG: --topology-manager-policy-options="" Jul 30 00:17:48.879093 ip-10-0-142-6 kubenswrapper[2564]: I0730 00:17:48.870971 2564 flags.go:64] FLAG: --topology-manager-scope="container" Jul 30 00:17:48.879093 ip-10-0-142-6 kubenswrapper[2564]: I0730 00:17:48.870974 2564 flags.go:64] FLAG: --v="2" Jul 30 00:17:48.879093 ip-10-0-142-6 kubenswrapper[2564]: I0730 00:17:48.870978 2564 flags.go:64] FLAG: --version="false" Jul 30 00:17:48.879093 ip-10-0-142-6 kubenswrapper[2564]: I0730 00:17:48.870981 2564 flags.go:64] FLAG: --vmodule="" Jul 30 00:17:48.879093 ip-10-0-142-6 kubenswrapper[2564]: I0730 00:17:48.870985 2564 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec" Jul 30 00:17:48.879093 ip-10-0-142-6 kubenswrapper[2564]: I0730 00:17:48.870988 2564 flags.go:64] FLAG: --volume-stats-agg-period="1m0s" Jul 30 00:17:48.879093 ip-10-0-142-6 kubenswrapper[2564]: W0730 00:17:48.871138 2564 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 30 00:17:48.879093 ip-10-0-142-6 kubenswrapper[2564]: W0730 00:17:48.871143 2564 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 30 00:17:48.879093 ip-10-0-142-6 kubenswrapper[2564]: W0730 00:17:48.871145 2564 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 30 00:17:48.879093 ip-10-0-142-6 kubenswrapper[2564]: W0730 00:17:48.871148 2564 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 30 00:17:48.879093 ip-10-0-142-6 kubenswrapper[2564]: W0730 00:17:48.871152 2564 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 30 00:17:48.879093 ip-10-0-142-6 kubenswrapper[2564]: W0730 00:17:48.871154 2564 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 30 00:17:48.879093 ip-10-0-142-6 kubenswrapper[2564]: W0730 00:17:48.871157 2564 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 30 00:17:48.879093 ip-10-0-142-6 kubenswrapper[2564]: W0730 00:17:48.871161 2564 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 30 00:17:48.879093 ip-10-0-142-6 kubenswrapper[2564]: W0730 00:17:48.871164 2564 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 30 00:17:48.879093 ip-10-0-142-6 kubenswrapper[2564]: W0730 00:17:48.871166 2564 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 30 00:17:48.879093 ip-10-0-142-6 kubenswrapper[2564]: W0730 00:17:48.871169 2564 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 30 00:17:48.879093 ip-10-0-142-6 kubenswrapper[2564]: W0730 00:17:48.871172 2564 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 30 00:17:48.879093 ip-10-0-142-6 kubenswrapper[2564]: W0730 00:17:48.871175 2564 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 30 00:17:48.879670 ip-10-0-142-6 kubenswrapper[2564]: W0730 00:17:48.871177 2564 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 30 00:17:48.879670 ip-10-0-142-6 kubenswrapper[2564]: W0730 00:17:48.871180 2564 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 30 00:17:48.879670 ip-10-0-142-6 kubenswrapper[2564]: W0730 00:17:48.871182 2564 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 30 00:17:48.879670 ip-10-0-142-6 kubenswrapper[2564]: W0730 00:17:48.871185 2564 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 30 00:17:48.879670 ip-10-0-142-6 kubenswrapper[2564]: W0730 00:17:48.871187 2564 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 30 00:17:48.879670 ip-10-0-142-6 kubenswrapper[2564]: W0730 00:17:48.871190 2564 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 30 00:17:48.879670 ip-10-0-142-6 kubenswrapper[2564]: W0730 00:17:48.871194 2564 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 30 00:17:48.879670 ip-10-0-142-6 kubenswrapper[2564]: W0730 00:17:48.871197 2564 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 30 00:17:48.879670 ip-10-0-142-6 kubenswrapper[2564]: W0730 00:17:48.871200 2564 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 30 00:17:48.879670 ip-10-0-142-6 kubenswrapper[2564]: W0730 00:17:48.871203 2564 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 30 00:17:48.879670 ip-10-0-142-6 kubenswrapper[2564]: W0730 00:17:48.871206 2564 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 30 00:17:48.879670 ip-10-0-142-6 kubenswrapper[2564]: W0730 00:17:48.871208 2564 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 30 00:17:48.879670 ip-10-0-142-6 kubenswrapper[2564]: W0730 00:17:48.871212 2564 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 30 00:17:48.879670 ip-10-0-142-6 kubenswrapper[2564]: W0730 00:17:48.871217 2564 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 30 00:17:48.879670 ip-10-0-142-6 kubenswrapper[2564]: W0730 00:17:48.871220 2564 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 30 00:17:48.879670 ip-10-0-142-6 kubenswrapper[2564]: W0730 00:17:48.871222 2564 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 30 00:17:48.879670 ip-10-0-142-6 kubenswrapper[2564]: W0730 00:17:48.871225 2564 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 30 00:17:48.879670 ip-10-0-142-6 kubenswrapper[2564]: W0730 00:17:48.871228 2564 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 30 00:17:48.879670 ip-10-0-142-6 kubenswrapper[2564]: W0730 00:17:48.871230 2564 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 30 00:17:48.880253 ip-10-0-142-6 kubenswrapper[2564]: W0730 00:17:48.871233 2564 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 30 00:17:48.880253 ip-10-0-142-6 kubenswrapper[2564]: W0730 00:17:48.871236 2564 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 30 00:17:48.880253 ip-10-0-142-6 kubenswrapper[2564]: W0730 00:17:48.871239 2564 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 30 00:17:48.880253 ip-10-0-142-6 kubenswrapper[2564]: W0730 00:17:48.871241 2564 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 30 00:17:48.880253 ip-10-0-142-6 kubenswrapper[2564]: W0730 00:17:48.871244 2564 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 30 00:17:48.880253 ip-10-0-142-6 kubenswrapper[2564]: W0730 00:17:48.871247 2564 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 30 00:17:48.880253 ip-10-0-142-6 kubenswrapper[2564]: W0730 00:17:48.871250 2564 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 30 00:17:48.880253 ip-10-0-142-6 kubenswrapper[2564]: W0730 00:17:48.871252 2564 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 30 00:17:48.880253 ip-10-0-142-6 kubenswrapper[2564]: W0730 00:17:48.871254 2564 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 30 00:17:48.880253 ip-10-0-142-6 kubenswrapper[2564]: W0730 00:17:48.871257 2564 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 30 00:17:48.880253 ip-10-0-142-6 kubenswrapper[2564]: W0730 00:17:48.871260 2564 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 30 00:17:48.880253 ip-10-0-142-6 kubenswrapper[2564]: W0730 00:17:48.871262 2564 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 30 00:17:48.880253 ip-10-0-142-6 kubenswrapper[2564]: W0730 00:17:48.871264 2564 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 30 00:17:48.880253 ip-10-0-142-6 kubenswrapper[2564]: W0730 00:17:48.871267 2564 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 30 00:17:48.880253 ip-10-0-142-6 kubenswrapper[2564]: W0730 00:17:48.871269 2564 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 30 00:17:48.880253 ip-10-0-142-6 kubenswrapper[2564]: W0730 00:17:48.871271 2564 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 30 00:17:48.880253 ip-10-0-142-6 kubenswrapper[2564]: W0730 00:17:48.871274 2564 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 30 00:17:48.880253 ip-10-0-142-6 kubenswrapper[2564]: W0730 00:17:48.871276 2564 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 30 00:17:48.880253 ip-10-0-142-6 kubenswrapper[2564]: W0730 00:17:48.871279 2564 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 30 00:17:48.880253 ip-10-0-142-6 kubenswrapper[2564]: W0730 00:17:48.871281 2564 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 30 00:17:48.880735 ip-10-0-142-6 kubenswrapper[2564]: W0730 00:17:48.871284 2564 feature_gate.go:328] unrecognized feature gate: Example Jul 30 00:17:48.880735 ip-10-0-142-6 kubenswrapper[2564]: W0730 00:17:48.871287 2564 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 30 00:17:48.880735 ip-10-0-142-6 kubenswrapper[2564]: W0730 00:17:48.871289 2564 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 30 00:17:48.880735 ip-10-0-142-6 kubenswrapper[2564]: W0730 00:17:48.871292 2564 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 30 00:17:48.880735 ip-10-0-142-6 kubenswrapper[2564]: W0730 00:17:48.871294 2564 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 30 00:17:48.880735 ip-10-0-142-6 kubenswrapper[2564]: W0730 00:17:48.871297 2564 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 30 00:17:48.880735 ip-10-0-142-6 kubenswrapper[2564]: W0730 00:17:48.871301 2564 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 30 00:17:48.880735 ip-10-0-142-6 kubenswrapper[2564]: W0730 00:17:48.871303 2564 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 30 00:17:48.880735 ip-10-0-142-6 kubenswrapper[2564]: W0730 00:17:48.871306 2564 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 30 00:17:48.880735 ip-10-0-142-6 kubenswrapper[2564]: W0730 00:17:48.871308 2564 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 30 00:17:48.880735 ip-10-0-142-6 kubenswrapper[2564]: W0730 00:17:48.871311 2564 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 30 00:17:48.880735 ip-10-0-142-6 kubenswrapper[2564]: W0730 00:17:48.871313 2564 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 30 00:17:48.880735 ip-10-0-142-6 kubenswrapper[2564]: W0730 00:17:48.871316 2564 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 30 00:17:48.880735 ip-10-0-142-6 kubenswrapper[2564]: W0730 00:17:48.871319 2564 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 30 00:17:48.880735 ip-10-0-142-6 kubenswrapper[2564]: W0730 00:17:48.871321 2564 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 30 00:17:48.880735 ip-10-0-142-6 kubenswrapper[2564]: W0730 00:17:48.871324 2564 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 30 00:17:48.880735 ip-10-0-142-6 kubenswrapper[2564]: W0730 00:17:48.871326 2564 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 30 00:17:48.880735 ip-10-0-142-6 kubenswrapper[2564]: W0730 00:17:48.871329 2564 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 30 00:17:48.880735 ip-10-0-142-6 kubenswrapper[2564]: W0730 00:17:48.871332 2564 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 30 00:17:48.881210 ip-10-0-142-6 kubenswrapper[2564]: W0730 00:17:48.871336 2564 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 30 00:17:48.881210 ip-10-0-142-6 kubenswrapper[2564]: W0730 00:17:48.871339 2564 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 30 00:17:48.881210 ip-10-0-142-6 kubenswrapper[2564]: W0730 00:17:48.871342 2564 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 30 00:17:48.881210 ip-10-0-142-6 kubenswrapper[2564]: W0730 00:17:48.871345 2564 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 30 00:17:48.881210 ip-10-0-142-6 kubenswrapper[2564]: W0730 00:17:48.871347 2564 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 30 00:17:48.881210 ip-10-0-142-6 kubenswrapper[2564]: W0730 00:17:48.871350 2564 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 30 00:17:48.881210 ip-10-0-142-6 kubenswrapper[2564]: W0730 00:17:48.871352 2564 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 30 00:17:48.881210 ip-10-0-142-6 kubenswrapper[2564]: W0730 00:17:48.871354 2564 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 30 00:17:48.881210 ip-10-0-142-6 kubenswrapper[2564]: W0730 00:17:48.871357 2564 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 30 00:17:48.881210 ip-10-0-142-6 kubenswrapper[2564]: W0730 00:17:48.871359 2564 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 30 00:17:48.881210 ip-10-0-142-6 kubenswrapper[2564]: W0730 00:17:48.871362 2564 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 30 00:17:48.881210 ip-10-0-142-6 kubenswrapper[2564]: W0730 00:17:48.871364 2564 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 30 00:17:48.881210 ip-10-0-142-6 kubenswrapper[2564]: W0730 00:17:48.871367 2564 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 30 00:17:48.881210 ip-10-0-142-6 kubenswrapper[2564]: W0730 00:17:48.871369 2564 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 30 00:17:48.881210 ip-10-0-142-6 kubenswrapper[2564]: W0730 00:17:48.871372 2564 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 30 00:17:48.881210 ip-10-0-142-6 kubenswrapper[2564]: W0730 00:17:48.871375 2564 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 30 00:17:48.881210 ip-10-0-142-6 kubenswrapper[2564]: W0730 00:17:48.871377 2564 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 30 00:17:48.881210 ip-10-0-142-6 kubenswrapper[2564]: W0730 00:17:48.871380 2564 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 30 00:17:48.881210 ip-10-0-142-6 kubenswrapper[2564]: W0730 00:17:48.871382 2564 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 30 00:17:48.881210 ip-10-0-142-6 kubenswrapper[2564]: W0730 00:17:48.871386 2564 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 30 00:17:48.881689 ip-10-0-142-6 kubenswrapper[2564]: W0730 00:17:48.871389 2564 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 30 00:17:48.881689 ip-10-0-142-6 kubenswrapper[2564]: W0730 00:17:48.871391 2564 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 30 00:17:48.881689 ip-10-0-142-6 kubenswrapper[2564]: W0730 00:17:48.871394 2564 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 30 00:17:48.881689 ip-10-0-142-6 kubenswrapper[2564]: W0730 00:17:48.871396 2564 feature_gate.go:328] unrecognized feature gate: Example2 Jul 30 00:17:48.881689 ip-10-0-142-6 kubenswrapper[2564]: W0730 00:17:48.871399 2564 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 30 00:17:48.881689 ip-10-0-142-6 kubenswrapper[2564]: W0730 00:17:48.871402 2564 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 30 00:17:48.881689 ip-10-0-142-6 kubenswrapper[2564]: W0730 00:17:48.871406 2564 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 30 00:17:48.881689 ip-10-0-142-6 kubenswrapper[2564]: W0730 00:17:48.871408 2564 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 30 00:17:48.881689 ip-10-0-142-6 kubenswrapper[2564]: I0730 00:17:48.871413 2564 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]} Jul 30 00:17:48.881689 ip-10-0-142-6 kubenswrapper[2564]: I0730 00:17:48.878877 2564 server.go:529] "Kubelet version" kubeletVersion="v1.34.9" Jul 30 00:17:48.881689 ip-10-0-142-6 kubenswrapper[2564]: I0730 00:17:48.878894 2564 server.go:531] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK="" Jul 30 00:17:48.881689 ip-10-0-142-6 kubenswrapper[2564]: I0730 00:17:48.878915 2564 watchdog_linux.go:95] "Systemd watchdog is not enabled" Jul 30 00:17:48.881689 ip-10-0-142-6 kubenswrapper[2564]: I0730 00:17:48.878920 2564 watchdog_linux.go:137] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started." Jul 30 00:17:48.881689 ip-10-0-142-6 kubenswrapper[2564]: W0730 00:17:48.878990 2564 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 30 00:17:48.881689 ip-10-0-142-6 kubenswrapper[2564]: W0730 00:17:48.878995 2564 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 30 00:17:48.882056 ip-10-0-142-6 kubenswrapper[2564]: W0730 00:17:48.878999 2564 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 30 00:17:48.882056 ip-10-0-142-6 kubenswrapper[2564]: W0730 00:17:48.879002 2564 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 30 00:17:48.882056 ip-10-0-142-6 kubenswrapper[2564]: W0730 00:17:48.879005 2564 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 30 00:17:48.882056 ip-10-0-142-6 kubenswrapper[2564]: W0730 00:17:48.879008 2564 feature_gate.go:328] unrecognized feature gate: Example2 Jul 30 00:17:48.882056 ip-10-0-142-6 kubenswrapper[2564]: W0730 00:17:48.879011 2564 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 30 00:17:48.882056 ip-10-0-142-6 kubenswrapper[2564]: W0730 00:17:48.879015 2564 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 30 00:17:48.882056 ip-10-0-142-6 kubenswrapper[2564]: W0730 00:17:48.879019 2564 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 30 00:17:48.882056 ip-10-0-142-6 kubenswrapper[2564]: W0730 00:17:48.879022 2564 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 30 00:17:48.882056 ip-10-0-142-6 kubenswrapper[2564]: W0730 00:17:48.879024 2564 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 30 00:17:48.882056 ip-10-0-142-6 kubenswrapper[2564]: W0730 00:17:48.879027 2564 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 30 00:17:48.882056 ip-10-0-142-6 kubenswrapper[2564]: W0730 00:17:48.879030 2564 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 30 00:17:48.882056 ip-10-0-142-6 kubenswrapper[2564]: W0730 00:17:48.879032 2564 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 30 00:17:48.882056 ip-10-0-142-6 kubenswrapper[2564]: W0730 00:17:48.879035 2564 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 30 00:17:48.882056 ip-10-0-142-6 kubenswrapper[2564]: W0730 00:17:48.879038 2564 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 30 00:17:48.882056 ip-10-0-142-6 kubenswrapper[2564]: W0730 00:17:48.879040 2564 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 30 00:17:48.882056 ip-10-0-142-6 kubenswrapper[2564]: W0730 00:17:48.879043 2564 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 30 00:17:48.882056 ip-10-0-142-6 kubenswrapper[2564]: W0730 00:17:48.879046 2564 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 30 00:17:48.882056 ip-10-0-142-6 kubenswrapper[2564]: W0730 00:17:48.879048 2564 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 30 00:17:48.882056 ip-10-0-142-6 kubenswrapper[2564]: W0730 00:17:48.879051 2564 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 30 00:17:48.882056 ip-10-0-142-6 kubenswrapper[2564]: W0730 00:17:48.879053 2564 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 30 00:17:48.882568 ip-10-0-142-6 kubenswrapper[2564]: W0730 00:17:48.879056 2564 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 30 00:17:48.882568 ip-10-0-142-6 kubenswrapper[2564]: W0730 00:17:48.879058 2564 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 30 00:17:48.882568 ip-10-0-142-6 kubenswrapper[2564]: W0730 00:17:48.879061 2564 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 30 00:17:48.882568 ip-10-0-142-6 kubenswrapper[2564]: W0730 00:17:48.879064 2564 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 30 00:17:48.882568 ip-10-0-142-6 kubenswrapper[2564]: W0730 00:17:48.879066 2564 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 30 00:17:48.882568 ip-10-0-142-6 kubenswrapper[2564]: W0730 00:17:48.879083 2564 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 30 00:17:48.882568 ip-10-0-142-6 kubenswrapper[2564]: W0730 00:17:48.879087 2564 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 30 00:17:48.882568 ip-10-0-142-6 kubenswrapper[2564]: W0730 00:17:48.879093 2564 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 30 00:17:48.882568 ip-10-0-142-6 kubenswrapper[2564]: W0730 00:17:48.879096 2564 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 30 00:17:48.882568 ip-10-0-142-6 kubenswrapper[2564]: W0730 00:17:48.879098 2564 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 30 00:17:48.882568 ip-10-0-142-6 kubenswrapper[2564]: W0730 00:17:48.879101 2564 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 30 00:17:48.882568 ip-10-0-142-6 kubenswrapper[2564]: W0730 00:17:48.879104 2564 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 30 00:17:48.882568 ip-10-0-142-6 kubenswrapper[2564]: W0730 00:17:48.879107 2564 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 30 00:17:48.882568 ip-10-0-142-6 kubenswrapper[2564]: W0730 00:17:48.879111 2564 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 30 00:17:48.882568 ip-10-0-142-6 kubenswrapper[2564]: W0730 00:17:48.879115 2564 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 30 00:17:48.882568 ip-10-0-142-6 kubenswrapper[2564]: W0730 00:17:48.879118 2564 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 30 00:17:48.882568 ip-10-0-142-6 kubenswrapper[2564]: W0730 00:17:48.879121 2564 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 30 00:17:48.882568 ip-10-0-142-6 kubenswrapper[2564]: W0730 00:17:48.879124 2564 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 30 00:17:48.882568 ip-10-0-142-6 kubenswrapper[2564]: W0730 00:17:48.879127 2564 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 30 00:17:48.883036 ip-10-0-142-6 kubenswrapper[2564]: W0730 00:17:48.879129 2564 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 30 00:17:48.883036 ip-10-0-142-6 kubenswrapper[2564]: W0730 00:17:48.879132 2564 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 30 00:17:48.883036 ip-10-0-142-6 kubenswrapper[2564]: W0730 00:17:48.879135 2564 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 30 00:17:48.883036 ip-10-0-142-6 kubenswrapper[2564]: W0730 00:17:48.879137 2564 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 30 00:17:48.883036 ip-10-0-142-6 kubenswrapper[2564]: W0730 00:17:48.879140 2564 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 30 00:17:48.883036 ip-10-0-142-6 kubenswrapper[2564]: W0730 00:17:48.879142 2564 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 30 00:17:48.883036 ip-10-0-142-6 kubenswrapper[2564]: W0730 00:17:48.879145 2564 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 30 00:17:48.883036 ip-10-0-142-6 kubenswrapper[2564]: W0730 00:17:48.879148 2564 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 30 00:17:48.883036 ip-10-0-142-6 kubenswrapper[2564]: W0730 00:17:48.879151 2564 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 30 00:17:48.883036 ip-10-0-142-6 kubenswrapper[2564]: W0730 00:17:48.879153 2564 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 30 00:17:48.883036 ip-10-0-142-6 kubenswrapper[2564]: W0730 00:17:48.879156 2564 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 30 00:17:48.883036 ip-10-0-142-6 kubenswrapper[2564]: W0730 00:17:48.879158 2564 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 30 00:17:48.883036 ip-10-0-142-6 kubenswrapper[2564]: W0730 00:17:48.879162 2564 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 30 00:17:48.883036 ip-10-0-142-6 kubenswrapper[2564]: W0730 00:17:48.879164 2564 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 30 00:17:48.883036 ip-10-0-142-6 kubenswrapper[2564]: W0730 00:17:48.879167 2564 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 30 00:17:48.883036 ip-10-0-142-6 kubenswrapper[2564]: W0730 00:17:48.879169 2564 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 30 00:17:48.883036 ip-10-0-142-6 kubenswrapper[2564]: W0730 00:17:48.879173 2564 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 30 00:17:48.883036 ip-10-0-142-6 kubenswrapper[2564]: W0730 00:17:48.879176 2564 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 30 00:17:48.883036 ip-10-0-142-6 kubenswrapper[2564]: W0730 00:17:48.879179 2564 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 30 00:17:48.883036 ip-10-0-142-6 kubenswrapper[2564]: W0730 00:17:48.879182 2564 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 30 00:17:48.883530 ip-10-0-142-6 kubenswrapper[2564]: W0730 00:17:48.879185 2564 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 30 00:17:48.883530 ip-10-0-142-6 kubenswrapper[2564]: W0730 00:17:48.879189 2564 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 30 00:17:48.883530 ip-10-0-142-6 kubenswrapper[2564]: W0730 00:17:48.879192 2564 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 30 00:17:48.883530 ip-10-0-142-6 kubenswrapper[2564]: W0730 00:17:48.879194 2564 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 30 00:17:48.883530 ip-10-0-142-6 kubenswrapper[2564]: W0730 00:17:48.879197 2564 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 30 00:17:48.883530 ip-10-0-142-6 kubenswrapper[2564]: W0730 00:17:48.879200 2564 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 30 00:17:48.883530 ip-10-0-142-6 kubenswrapper[2564]: W0730 00:17:48.879203 2564 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 30 00:17:48.883530 ip-10-0-142-6 kubenswrapper[2564]: W0730 00:17:48.879206 2564 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 30 00:17:48.883530 ip-10-0-142-6 kubenswrapper[2564]: W0730 00:17:48.879208 2564 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 30 00:17:48.883530 ip-10-0-142-6 kubenswrapper[2564]: W0730 00:17:48.879211 2564 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 30 00:17:48.883530 ip-10-0-142-6 kubenswrapper[2564]: W0730 00:17:48.879213 2564 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 30 00:17:48.883530 ip-10-0-142-6 kubenswrapper[2564]: W0730 00:17:48.879216 2564 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 30 00:17:48.883530 ip-10-0-142-6 kubenswrapper[2564]: W0730 00:17:48.879218 2564 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 30 00:17:48.883530 ip-10-0-142-6 kubenswrapper[2564]: W0730 00:17:48.879221 2564 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 30 00:17:48.883530 ip-10-0-142-6 kubenswrapper[2564]: W0730 00:17:48.879224 2564 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 30 00:17:48.883530 ip-10-0-142-6 kubenswrapper[2564]: W0730 00:17:48.879226 2564 feature_gate.go:328] unrecognized feature gate: Example Jul 30 00:17:48.883530 ip-10-0-142-6 kubenswrapper[2564]: W0730 00:17:48.879228 2564 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 30 00:17:48.883530 ip-10-0-142-6 kubenswrapper[2564]: W0730 00:17:48.879231 2564 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 30 00:17:48.883530 ip-10-0-142-6 kubenswrapper[2564]: W0730 00:17:48.879233 2564 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 30 00:17:48.883530 ip-10-0-142-6 kubenswrapper[2564]: W0730 00:17:48.879236 2564 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 30 00:17:48.884015 ip-10-0-142-6 kubenswrapper[2564]: W0730 00:17:48.879239 2564 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 30 00:17:48.884015 ip-10-0-142-6 kubenswrapper[2564]: W0730 00:17:48.879242 2564 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 30 00:17:48.884015 ip-10-0-142-6 kubenswrapper[2564]: W0730 00:17:48.879244 2564 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 30 00:17:48.884015 ip-10-0-142-6 kubenswrapper[2564]: W0730 00:17:48.879247 2564 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 30 00:17:48.884015 ip-10-0-142-6 kubenswrapper[2564]: W0730 00:17:48.879249 2564 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 30 00:17:48.884015 ip-10-0-142-6 kubenswrapper[2564]: W0730 00:17:48.879252 2564 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 30 00:17:48.884015 ip-10-0-142-6 kubenswrapper[2564]: W0730 00:17:48.879254 2564 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 30 00:17:48.884015 ip-10-0-142-6 kubenswrapper[2564]: W0730 00:17:48.879257 2564 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 30 00:17:48.884015 ip-10-0-142-6 kubenswrapper[2564]: W0730 00:17:48.879259 2564 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 30 00:17:48.884015 ip-10-0-142-6 kubenswrapper[2564]: W0730 00:17:48.879262 2564 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 30 00:17:48.884015 ip-10-0-142-6 kubenswrapper[2564]: W0730 00:17:48.879264 2564 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 30 00:17:48.884015 ip-10-0-142-6 kubenswrapper[2564]: W0730 00:17:48.879267 2564 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 30 00:17:48.884015 ip-10-0-142-6 kubenswrapper[2564]: W0730 00:17:48.879270 2564 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 30 00:17:48.884015 ip-10-0-142-6 kubenswrapper[2564]: W0730 00:17:48.879272 2564 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 30 00:17:48.884015 ip-10-0-142-6 kubenswrapper[2564]: W0730 00:17:48.879275 2564 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 30 00:17:48.884015 ip-10-0-142-6 kubenswrapper[2564]: W0730 00:17:48.879278 2564 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 30 00:17:48.884015 ip-10-0-142-6 kubenswrapper[2564]: W0730 00:17:48.879280 2564 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 30 00:17:48.884015 ip-10-0-142-6 kubenswrapper[2564]: W0730 00:17:48.879283 2564 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 30 00:17:48.884560 ip-10-0-142-6 kubenswrapper[2564]: I0730 00:17:48.879288 2564 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]} Jul 30 00:17:48.884560 ip-10-0-142-6 kubenswrapper[2564]: W0730 00:17:48.879420 2564 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 30 00:17:48.884560 ip-10-0-142-6 kubenswrapper[2564]: W0730 00:17:48.879424 2564 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 30 00:17:48.884560 ip-10-0-142-6 kubenswrapper[2564]: W0730 00:17:48.879428 2564 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 30 00:17:48.884560 ip-10-0-142-6 kubenswrapper[2564]: W0730 00:17:48.879430 2564 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 30 00:17:48.884560 ip-10-0-142-6 kubenswrapper[2564]: W0730 00:17:48.879432 2564 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 30 00:17:48.884560 ip-10-0-142-6 kubenswrapper[2564]: W0730 00:17:48.879435 2564 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 30 00:17:48.884560 ip-10-0-142-6 kubenswrapper[2564]: W0730 00:17:48.879438 2564 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 30 00:17:48.884560 ip-10-0-142-6 kubenswrapper[2564]: W0730 00:17:48.879440 2564 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 30 00:17:48.884560 ip-10-0-142-6 kubenswrapper[2564]: W0730 00:17:48.879443 2564 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 30 00:17:48.884560 ip-10-0-142-6 kubenswrapper[2564]: W0730 00:17:48.879445 2564 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 30 00:17:48.884560 ip-10-0-142-6 kubenswrapper[2564]: W0730 00:17:48.879448 2564 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 30 00:17:48.884560 ip-10-0-142-6 kubenswrapper[2564]: W0730 00:17:48.879450 2564 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 30 00:17:48.884560 ip-10-0-142-6 kubenswrapper[2564]: W0730 00:17:48.879453 2564 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 30 00:17:48.884560 ip-10-0-142-6 kubenswrapper[2564]: W0730 00:17:48.879455 2564 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 30 00:17:48.884934 ip-10-0-142-6 kubenswrapper[2564]: W0730 00:17:48.879458 2564 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 30 00:17:48.884934 ip-10-0-142-6 kubenswrapper[2564]: W0730 00:17:48.879460 2564 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 30 00:17:48.884934 ip-10-0-142-6 kubenswrapper[2564]: W0730 00:17:48.879463 2564 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 30 00:17:48.884934 ip-10-0-142-6 kubenswrapper[2564]: W0730 00:17:48.879465 2564 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 30 00:17:48.884934 ip-10-0-142-6 kubenswrapper[2564]: W0730 00:17:48.879468 2564 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 30 00:17:48.884934 ip-10-0-142-6 kubenswrapper[2564]: W0730 00:17:48.879470 2564 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 30 00:17:48.884934 ip-10-0-142-6 kubenswrapper[2564]: W0730 00:17:48.879473 2564 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 30 00:17:48.884934 ip-10-0-142-6 kubenswrapper[2564]: W0730 00:17:48.879475 2564 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 30 00:17:48.884934 ip-10-0-142-6 kubenswrapper[2564]: W0730 00:17:48.879478 2564 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 30 00:17:48.884934 ip-10-0-142-6 kubenswrapper[2564]: W0730 00:17:48.879480 2564 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 30 00:17:48.884934 ip-10-0-142-6 kubenswrapper[2564]: W0730 00:17:48.879483 2564 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 30 00:17:48.884934 ip-10-0-142-6 kubenswrapper[2564]: W0730 00:17:48.879487 2564 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 30 00:17:48.884934 ip-10-0-142-6 kubenswrapper[2564]: W0730 00:17:48.879489 2564 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 30 00:17:48.884934 ip-10-0-142-6 kubenswrapper[2564]: W0730 00:17:48.879492 2564 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 30 00:17:48.884934 ip-10-0-142-6 kubenswrapper[2564]: W0730 00:17:48.879495 2564 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 30 00:17:48.884934 ip-10-0-142-6 kubenswrapper[2564]: W0730 00:17:48.879498 2564 feature_gate.go:328] unrecognized feature gate: Example2 Jul 30 00:17:48.884934 ip-10-0-142-6 kubenswrapper[2564]: W0730 00:17:48.879500 2564 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 30 00:17:48.884934 ip-10-0-142-6 kubenswrapper[2564]: W0730 00:17:48.879503 2564 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 30 00:17:48.884934 ip-10-0-142-6 kubenswrapper[2564]: W0730 00:17:48.879505 2564 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 30 00:17:48.885407 ip-10-0-142-6 kubenswrapper[2564]: W0730 00:17:48.879508 2564 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 30 00:17:48.885407 ip-10-0-142-6 kubenswrapper[2564]: W0730 00:17:48.879510 2564 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 30 00:17:48.885407 ip-10-0-142-6 kubenswrapper[2564]: W0730 00:17:48.879513 2564 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 30 00:17:48.885407 ip-10-0-142-6 kubenswrapper[2564]: W0730 00:17:48.879515 2564 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 30 00:17:48.885407 ip-10-0-142-6 kubenswrapper[2564]: W0730 00:17:48.879518 2564 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 30 00:17:48.885407 ip-10-0-142-6 kubenswrapper[2564]: W0730 00:17:48.879520 2564 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 30 00:17:48.885407 ip-10-0-142-6 kubenswrapper[2564]: W0730 00:17:48.879523 2564 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 30 00:17:48.885407 ip-10-0-142-6 kubenswrapper[2564]: W0730 00:17:48.879525 2564 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 30 00:17:48.885407 ip-10-0-142-6 kubenswrapper[2564]: W0730 00:17:48.879529 2564 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 30 00:17:48.885407 ip-10-0-142-6 kubenswrapper[2564]: W0730 00:17:48.879532 2564 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 30 00:17:48.885407 ip-10-0-142-6 kubenswrapper[2564]: W0730 00:17:48.879535 2564 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 30 00:17:48.885407 ip-10-0-142-6 kubenswrapper[2564]: W0730 00:17:48.879537 2564 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 30 00:17:48.885407 ip-10-0-142-6 kubenswrapper[2564]: W0730 00:17:48.879540 2564 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 30 00:17:48.885407 ip-10-0-142-6 kubenswrapper[2564]: W0730 00:17:48.879543 2564 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 30 00:17:48.885407 ip-10-0-142-6 kubenswrapper[2564]: W0730 00:17:48.879545 2564 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 30 00:17:48.885407 ip-10-0-142-6 kubenswrapper[2564]: W0730 00:17:48.879548 2564 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 30 00:17:48.885407 ip-10-0-142-6 kubenswrapper[2564]: W0730 00:17:48.879550 2564 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 30 00:17:48.885407 ip-10-0-142-6 kubenswrapper[2564]: W0730 00:17:48.879553 2564 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 30 00:17:48.885407 ip-10-0-142-6 kubenswrapper[2564]: W0730 00:17:48.879555 2564 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 30 00:17:48.885407 ip-10-0-142-6 kubenswrapper[2564]: W0730 00:17:48.879558 2564 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 30 00:17:48.885954 ip-10-0-142-6 kubenswrapper[2564]: W0730 00:17:48.879560 2564 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 30 00:17:48.885954 ip-10-0-142-6 kubenswrapper[2564]: W0730 00:17:48.879564 2564 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 30 00:17:48.885954 ip-10-0-142-6 kubenswrapper[2564]: W0730 00:17:48.879567 2564 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 30 00:17:48.885954 ip-10-0-142-6 kubenswrapper[2564]: W0730 00:17:48.879570 2564 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 30 00:17:48.885954 ip-10-0-142-6 kubenswrapper[2564]: W0730 00:17:48.879574 2564 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 30 00:17:48.885954 ip-10-0-142-6 kubenswrapper[2564]: W0730 00:17:48.879576 2564 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 30 00:17:48.885954 ip-10-0-142-6 kubenswrapper[2564]: W0730 00:17:48.879579 2564 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 30 00:17:48.885954 ip-10-0-142-6 kubenswrapper[2564]: W0730 00:17:48.879581 2564 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 30 00:17:48.885954 ip-10-0-142-6 kubenswrapper[2564]: W0730 00:17:48.879584 2564 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 30 00:17:48.885954 ip-10-0-142-6 kubenswrapper[2564]: W0730 00:17:48.879587 2564 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 30 00:17:48.885954 ip-10-0-142-6 kubenswrapper[2564]: W0730 00:17:48.879589 2564 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 30 00:17:48.885954 ip-10-0-142-6 kubenswrapper[2564]: W0730 00:17:48.879592 2564 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 30 00:17:48.885954 ip-10-0-142-6 kubenswrapper[2564]: W0730 00:17:48.879594 2564 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 30 00:17:48.885954 ip-10-0-142-6 kubenswrapper[2564]: W0730 00:17:48.879597 2564 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 30 00:17:48.885954 ip-10-0-142-6 kubenswrapper[2564]: W0730 00:17:48.879600 2564 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 30 00:17:48.885954 ip-10-0-142-6 kubenswrapper[2564]: W0730 00:17:48.879603 2564 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 30 00:17:48.885954 ip-10-0-142-6 kubenswrapper[2564]: W0730 00:17:48.879605 2564 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 30 00:17:48.885954 ip-10-0-142-6 kubenswrapper[2564]: W0730 00:17:48.879607 2564 feature_gate.go:328] unrecognized feature gate: Example Jul 30 00:17:48.885954 ip-10-0-142-6 kubenswrapper[2564]: W0730 00:17:48.879610 2564 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 30 00:17:48.885954 ip-10-0-142-6 kubenswrapper[2564]: W0730 00:17:48.879612 2564 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 30 00:17:48.886460 ip-10-0-142-6 kubenswrapper[2564]: W0730 00:17:48.879615 2564 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 30 00:17:48.886460 ip-10-0-142-6 kubenswrapper[2564]: W0730 00:17:48.879617 2564 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 30 00:17:48.886460 ip-10-0-142-6 kubenswrapper[2564]: W0730 00:17:48.879620 2564 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 30 00:17:48.886460 ip-10-0-142-6 kubenswrapper[2564]: W0730 00:17:48.879622 2564 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 30 00:17:48.886460 ip-10-0-142-6 kubenswrapper[2564]: W0730 00:17:48.879625 2564 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 30 00:17:48.886460 ip-10-0-142-6 kubenswrapper[2564]: W0730 00:17:48.879628 2564 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 30 00:17:48.886460 ip-10-0-142-6 kubenswrapper[2564]: W0730 00:17:48.879630 2564 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 30 00:17:48.886460 ip-10-0-142-6 kubenswrapper[2564]: W0730 00:17:48.879633 2564 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 30 00:17:48.886460 ip-10-0-142-6 kubenswrapper[2564]: W0730 00:17:48.879636 2564 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 30 00:17:48.886460 ip-10-0-142-6 kubenswrapper[2564]: W0730 00:17:48.879639 2564 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 30 00:17:48.886460 ip-10-0-142-6 kubenswrapper[2564]: W0730 00:17:48.879642 2564 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 30 00:17:48.886460 ip-10-0-142-6 kubenswrapper[2564]: W0730 00:17:48.879645 2564 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 30 00:17:48.886460 ip-10-0-142-6 kubenswrapper[2564]: W0730 00:17:48.879647 2564 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 30 00:17:48.886460 ip-10-0-142-6 kubenswrapper[2564]: W0730 00:17:48.879650 2564 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 30 00:17:48.886460 ip-10-0-142-6 kubenswrapper[2564]: W0730 00:17:48.879653 2564 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 30 00:17:48.886460 ip-10-0-142-6 kubenswrapper[2564]: W0730 00:17:48.879655 2564 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 30 00:17:48.886460 ip-10-0-142-6 kubenswrapper[2564]: W0730 00:17:48.879658 2564 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 30 00:17:48.886460 ip-10-0-142-6 kubenswrapper[2564]: W0730 00:17:48.879661 2564 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 30 00:17:48.886460 ip-10-0-142-6 kubenswrapper[2564]: W0730 00:17:48.879663 2564 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 30 00:17:48.886460 ip-10-0-142-6 kubenswrapper[2564]: W0730 00:17:48.879665 2564 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 30 00:17:48.886956 ip-10-0-142-6 kubenswrapper[2564]: W0730 00:17:48.879668 2564 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 30 00:17:48.886956 ip-10-0-142-6 kubenswrapper[2564]: W0730 00:17:48.879671 2564 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 30 00:17:48.886956 ip-10-0-142-6 kubenswrapper[2564]: W0730 00:17:48.879673 2564 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 30 00:17:48.886956 ip-10-0-142-6 kubenswrapper[2564]: W0730 00:17:48.879676 2564 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 30 00:17:48.886956 ip-10-0-142-6 kubenswrapper[2564]: W0730 00:17:48.879679 2564 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 30 00:17:48.886956 ip-10-0-142-6 kubenswrapper[2564]: W0730 00:17:48.879682 2564 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 30 00:17:48.886956 ip-10-0-142-6 kubenswrapper[2564]: I0730 00:17:48.879687 2564 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]} Jul 30 00:17:48.886956 ip-10-0-142-6 kubenswrapper[2564]: I0730 00:17:48.879829 2564 server.go:962] "Client rotation is on, will bootstrap in background" Jul 30 00:17:48.886956 ip-10-0-142-6 kubenswrapper[2564]: I0730 00:17:48.881934 2564 bootstrap.go:101] "Use the bootstrap credentials to request a cert, and set kubeconfig to point to the certificate dir" Jul 30 00:17:48.886956 ip-10-0-142-6 kubenswrapper[2564]: I0730 00:17:48.883203 2564 server.go:1019] "Starting client certificate rotation" Jul 30 00:17:48.886956 ip-10-0-142-6 kubenswrapper[2564]: I0730 00:17:48.883298 2564 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet" Jul 30 00:17:48.886956 ip-10-0-142-6 kubenswrapper[2564]: I0730 00:17:48.883329 2564 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kube-apiserver-client-kubelet" Jul 30 00:17:48.886956 ip-10-0-142-6 kubenswrapper[2564]: I0730 00:17:48.885548 2564 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Jul 30 00:17:48.887410 ip-10-0-142-6 kubenswrapper[2564]: I0730 00:17:48.887391 2564 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Jul 30 00:17:48.896171 ip-10-0-142-6 kubenswrapper[2564]: I0730 00:17:48.896151 2564 log.go:25] "Validated CRI v1 runtime API" Jul 30 00:17:48.898104 ip-10-0-142-6 kubenswrapper[2564]: I0730 00:17:48.898090 2564 log.go:25] "Validated CRI v1 image API" Jul 30 00:17:48.902086 ip-10-0-142-6 kubenswrapper[2564]: I0730 00:17:48.902050 2564 server.go:1429] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd" Jul 30 00:17:48.905812 ip-10-0-142-6 kubenswrapper[2564]: I0730 00:17:48.905791 2564 fs.go:135] Filesystem UUIDs: map[388df5c9-d3b5-4d21-8cb5-da271ed11b45:/dev/nvme0n1p4 6444d407-3e8f-43db-ae64-d0c17ae527e0:/dev/nvme0n1p3 7B77-95E7:/dev/nvme0n1p2] Jul 30 00:17:48.905892 ip-10-0-142-6 kubenswrapper[2564]: I0730 00:17:48.905811 2564 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:22 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:24 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:31 fsType:tmpfs blockSize:0} composefs_0-33:{mountpoint:/ major:0 minor:33 fsType:overlay blockSize:0}] Jul 30 00:17:48.910342 ip-10-0-142-6 kubenswrapper[2564]: I0730 00:17:48.910230 2564 manager.go:216] Machine: {Timestamp:2026-07-30 00:17:48.909732127 +0000 UTC m=+0.324472051 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3100709 MemoryCapacity:33164488704 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec2551ac5dd14fbb139c60488f8f156a SystemUUID:ec2551ac-5dd1-4fbb-139c-60488f8f156a BootID:f75ae0d2-1430-4df4-9e5f-7d064a1fe4ac Filesystems:[{Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true} {Device:/dev/shm DeviceMajor:0 DeviceMinor:22 Capacity:16582242304 Type:vfs Inodes:4048399 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:24 Capacity:6632898560 Type:vfs Inodes:819200 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:composefs_0-33 DeviceMajor:0 DeviceMinor:33 Capacity:6111232 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:31 Capacity:16582246400 Type:vfs Inodes:1048576 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:br-ex MacAddress:02:c8:90:31:ae:fb Speed:0 Mtu:9001} {Name:ens5 MacAddress:02:c8:90:31:ae:fb Speed:0 Mtu:9001} {Name:ovs-system MacAddress:aa:8c:a0:ba:da:9b Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:33164488704 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:37486592 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None} Jul 30 00:17:48.910342 ip-10-0-142-6 kubenswrapper[2564]: I0730 00:17:48.910341 2564 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available. Jul 30 00:17:48.910449 ip-10-0-142-6 kubenswrapper[2564]: I0730 00:17:48.910417 2564 manager.go:232] Version: {KernelVersion:5.14.0-570.129.1.el9_6.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.6.20260717-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:} Jul 30 00:17:48.913861 ip-10-0-142-6 kubenswrapper[2564]: I0730 00:17:48.913836 2564 container_manager_linux.go:273] "Container manager verified user specified cgroup-root exists" cgroupRoot=[] Jul 30 00:17:48.913940 ip-10-0-142-6 kubenswrapper[2564]: I0730 00:17:48.913926 2564 reflector.go:439] "Caches populated" logger="kubernetes.io/kube-apiserver-client-kubelet" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" Jul 30 00:17:48.914005 ip-10-0-142-6 kubenswrapper[2564]: I0730 00:17:48.913866 2564 container_manager_linux.go:278] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-142-6.ec2.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2} Jul 30 00:17:48.914997 ip-10-0-142-6 kubenswrapper[2564]: I0730 00:17:48.914986 2564 topology_manager.go:138] "Creating topology manager with none policy" Jul 30 00:17:48.915029 ip-10-0-142-6 kubenswrapper[2564]: I0730 00:17:48.914999 2564 container_manager_linux.go:309] "Creating device plugin manager" Jul 30 00:17:48.915029 ip-10-0-142-6 kubenswrapper[2564]: I0730 00:17:48.915006 2564 manager.go:141] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock" Jul 30 00:17:48.915667 ip-10-0-142-6 kubenswrapper[2564]: I0730 00:17:48.915655 2564 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock" Jul 30 00:17:48.916925 ip-10-0-142-6 kubenswrapper[2564]: I0730 00:17:48.916914 2564 container_manager_linux.go:318] "Creating Dynamic Resource Allocation (DRA) manager" Jul 30 00:17:48.919525 ip-10-0-142-6 kubenswrapper[2564]: I0730 00:17:48.919512 2564 state_mem.go:36] "Initialized new in-memory state store" Jul 30 00:17:48.919645 ip-10-0-142-6 kubenswrapper[2564]: I0730 00:17:48.919636 2564 server.go:1240] "Using root directory" path="/var/lib/kubelet" Jul 30 00:17:48.919680 ip-10-0-142-6 kubenswrapper[2564]: I0730 00:17:48.919673 2564 kubelet.go:486] "Attempting to sync node with API server" Jul 30 00:17:48.919707 ip-10-0-142-6 kubenswrapper[2564]: I0730 00:17:48.919687 2564 kubelet.go:387] "Adding static pod path" path="/etc/kubernetes/manifests" Jul 30 00:17:48.919707 ip-10-0-142-6 kubenswrapper[2564]: I0730 00:17:48.919701 2564 file.go:69] "Watching path" path="/etc/kubernetes/manifests" Jul 30 00:17:48.919756 ip-10-0-142-6 kubenswrapper[2564]: I0730 00:17:48.919712 2564 kubelet.go:398] "Adding apiserver pod source" Jul 30 00:17:48.919756 ip-10-0-142-6 kubenswrapper[2564]: I0730 00:17:48.919720 2564 apiserver.go:42] "Waiting for node sync before watching apiserver pods" Jul 30 00:17:48.920152 ip-10-0-142-6 kubenswrapper[2564]: I0730 00:17:48.920138 2564 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Jul 30 00:17:48.920204 ip-10-0-142-6 kubenswrapper[2564]: I0730 00:17:48.920164 2564 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Jul 30 00:17:48.922122 ip-10-0-142-6 kubenswrapper[2564]: I0730 00:17:48.922105 2564 kuberuntime_manager.go:301] "Container runtime initialized" containerRuntime="cri-o" version="1.34.10-2.rhaos4.21.git83ee25b.el9" apiVersion="v1" Jul 30 00:17:48.923921 ip-10-0-142-6 kubenswrapper[2564]: I0730 00:17:48.923908 2564 kubelet.go:960] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled" Jul 30 00:17:48.923963 ip-10-0-142-6 kubenswrapper[2564]: I0730 00:17:48.923943 2564 kubelet.go:984] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled" Jul 30 00:17:48.925915 ip-10-0-142-6 kubenswrapper[2564]: I0730 00:17:48.925902 2564 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume" Jul 30 00:17:48.925957 ip-10-0-142-6 kubenswrapper[2564]: I0730 00:17:48.925924 2564 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir" Jul 30 00:17:48.925957 ip-10-0-142-6 kubenswrapper[2564]: I0730 00:17:48.925931 2564 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo" Jul 30 00:17:48.925957 ip-10-0-142-6 kubenswrapper[2564]: I0730 00:17:48.925937 2564 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path" Jul 30 00:17:48.925957 ip-10-0-142-6 kubenswrapper[2564]: I0730 00:17:48.925949 2564 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs" Jul 30 00:17:48.925957 ip-10-0-142-6 kubenswrapper[2564]: I0730 00:17:48.925954 2564 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret" Jul 30 00:17:48.926165 ip-10-0-142-6 kubenswrapper[2564]: I0730 00:17:48.925963 2564 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi" Jul 30 00:17:48.926165 ip-10-0-142-6 kubenswrapper[2564]: I0730 00:17:48.925968 2564 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api" Jul 30 00:17:48.926165 ip-10-0-142-6 kubenswrapper[2564]: I0730 00:17:48.925975 2564 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc" Jul 30 00:17:48.926165 ip-10-0-142-6 kubenswrapper[2564]: I0730 00:17:48.925980 2564 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap" Jul 30 00:17:48.926165 ip-10-0-142-6 kubenswrapper[2564]: I0730 00:17:48.925994 2564 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected" Jul 30 00:17:48.926165 ip-10-0-142-6 kubenswrapper[2564]: I0730 00:17:48.926000 2564 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume" Jul 30 00:17:48.926165 ip-10-0-142-6 kubenswrapper[2564]: I0730 00:17:48.926025 2564 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi" Jul 30 00:17:48.926165 ip-10-0-142-6 kubenswrapper[2564]: I0730 00:17:48.926033 2564 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image" Jul 30 00:17:48.930168 ip-10-0-142-6 kubenswrapper[2564]: I0730 00:17:48.930145 2564 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-142-6.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope Jul 30 00:17:48.930279 ip-10-0-142-6 kubenswrapper[2564]: E0730 00:17:48.930254 2564 reflector.go:205] "Failed to watch" err="failed to list *v1.Node: nodes \"ip-10-0-142-6.ec2.internal\" is forbidden: User \"system:anonymous\" cannot list resource \"nodes\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Node" Jul 30 00:17:48.930353 ip-10-0-142-6 kubenswrapper[2564]: E0730 00:17:48.930320 2564 reflector.go:205] "Failed to watch" err="failed to list *v1.Service: services is forbidden: User \"system:anonymous\" cannot list resource \"services\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Service" Jul 30 00:17:48.930701 ip-10-0-142-6 kubenswrapper[2564]: I0730 00:17:48.930688 2564 server.go:1268] "Started kubelet" Jul 30 00:17:48.930781 ip-10-0-142-6 kubenswrapper[2564]: I0730 00:17:48.930756 2564 server.go:180] "Starting to listen" address="0.0.0.0" port=10250 Jul 30 00:17:48.930781 ip-10-0-142-6 kubenswrapper[2564]: I0730 00:17:48.930756 2564 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10 Jul 30 00:17:48.930846 ip-10-0-142-6 kubenswrapper[2564]: I0730 00:17:48.930813 2564 server_v1.go:49] "podresources" method="list" useActivePods=true Jul 30 00:17:48.931651 ip-10-0-142-6 systemd[1]: Started Kubernetes Kubelet. Jul 30 00:17:48.931965 ip-10-0-142-6 kubenswrapper[2564]: I0730 00:17:48.931892 2564 server.go:249] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock" Jul 30 00:17:48.931965 ip-10-0-142-6 kubenswrapper[2564]: I0730 00:17:48.931927 2564 server.go:310] "Adding debug handlers to kubelet server" Jul 30 00:17:48.933814 ip-10-0-142-6 kubenswrapper[2564]: I0730 00:17:48.933787 2564 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-cjlnp" Jul 30 00:17:48.936690 ip-10-0-142-6 kubenswrapper[2564]: E0730 00:17:48.936671 2564 kubelet.go:1638] "Image garbage collection failed once. Stats initialization may not have completed yet" err="failed to get imageFs info: unable to find data in memory cache" Jul 30 00:17:48.936773 ip-10-0-142-6 kubenswrapper[2564]: E0730 00:17:48.935920 2564 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-142-6.ec2.internal.18c6e92a020361bc default 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-142-6.ec2.internal,UID:ip-10-0-142-6.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:Starting,Message:Starting kubelet.,Source:EventSource{Component:kubelet,Host:ip-10-0-142-6.ec2.internal,},FirstTimestamp:2026-07-30 00:17:48.930654652 +0000 UTC m=+0.345394576,LastTimestamp:2026-07-30 00:17:48.930654652 +0000 UTC m=+0.345394576,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-142-6.ec2.internal,}" Jul 30 00:17:48.938732 ip-10-0-142-6 kubenswrapper[2564]: I0730 00:17:48.938714 2564 fs_resource_analyzer.go:67] "Starting FS ResourceAnalyzer" Jul 30 00:17:48.938828 ip-10-0-142-6 kubenswrapper[2564]: I0730 00:17:48.938714 2564 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving" Jul 30 00:17:48.938828 ip-10-0-142-6 kubenswrapper[2564]: E0730 00:17:48.938820 2564 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-142-6.ec2.internal\" not found" Jul 30 00:17:48.939669 ip-10-0-142-6 kubenswrapper[2564]: I0730 00:17:48.939649 2564 factory.go:55] Registering systemd factory Jul 30 00:17:48.939669 ip-10-0-142-6 kubenswrapper[2564]: I0730 00:17:48.939671 2564 factory.go:223] Registration of the systemd container factory successfully Jul 30 00:17:48.939951 ip-10-0-142-6 kubenswrapper[2564]: I0730 00:17:48.939936 2564 factory.go:153] Registering CRI-O factory Jul 30 00:17:48.939951 ip-10-0-142-6 kubenswrapper[2564]: I0730 00:17:48.939953 2564 factory.go:223] Registration of the crio container factory successfully Jul 30 00:17:48.940095 ip-10-0-142-6 kubenswrapper[2564]: I0730 00:17:48.940012 2564 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory Jul 30 00:17:48.940095 ip-10-0-142-6 kubenswrapper[2564]: I0730 00:17:48.940038 2564 factory.go:103] Registering Raw factory Jul 30 00:17:48.940095 ip-10-0-142-6 kubenswrapper[2564]: I0730 00:17:48.940053 2564 manager.go:1196] Started watching for new ooms in manager Jul 30 00:17:48.940095 ip-10-0-142-6 kubenswrapper[2564]: I0730 00:17:48.940069 2564 volume_manager.go:311] "The desired_state_of_world populator starts" Jul 30 00:17:48.940296 ip-10-0-142-6 kubenswrapper[2564]: I0730 00:17:48.940100 2564 volume_manager.go:313] "Starting Kubelet Volume Manager" Jul 30 00:17:48.940296 ip-10-0-142-6 kubenswrapper[2564]: I0730 00:17:48.940085 2564 desired_state_of_world_populator.go:146] "Desired state populator starts to run" Jul 30 00:17:48.940296 ip-10-0-142-6 kubenswrapper[2564]: I0730 00:17:48.940215 2564 reconstruct.go:97] "Volume reconstruction finished" Jul 30 00:17:48.940296 ip-10-0-142-6 kubenswrapper[2564]: I0730 00:17:48.940223 2564 reconciler.go:29] "Reconciler: start to sync state" Jul 30 00:17:48.940537 ip-10-0-142-6 kubenswrapper[2564]: I0730 00:17:48.940525 2564 manager.go:349] Starting recovery of all containers Jul 30 00:17:48.941339 ip-10-0-142-6 kubenswrapper[2564]: I0730 00:17:48.941314 2564 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-cjlnp" Jul 30 00:17:48.947669 ip-10-0-142-6 kubenswrapper[2564]: E0730 00:17:48.946532 2564 controller.go:145] "Failed to ensure lease exists, will retry" err="leases.coordination.k8s.io \"ip-10-0-142-6.ec2.internal\" is forbidden: User \"system:anonymous\" cannot get resource \"leases\" in API group \"coordination.k8s.io\" in the namespace \"kube-node-lease\"" interval="200ms" Jul 30 00:17:48.947669 ip-10-0-142-6 kubenswrapper[2564]: E0730 00:17:48.946634 2564 reflector.go:205] "Failed to watch" err="failed to list *v1.CSIDriver: csidrivers.storage.k8s.io is forbidden: User \"system:anonymous\" cannot list resource \"csidrivers\" in API group \"storage.k8s.io\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.CSIDriver" Jul 30 00:17:48.949359 ip-10-0-142-6 kubenswrapper[2564]: I0730 00:17:48.949336 2564 manager.go:354] Recovery completed Jul 30 00:17:48.955280 ip-10-0-142-6 kubenswrapper[2564]: I0730 00:17:48.955268 2564 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 30 00:17:48.957437 ip-10-0-142-6 kubenswrapper[2564]: I0730 00:17:48.957418 2564 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-142-6.ec2.internal" event="NodeHasSufficientMemory" Jul 30 00:17:48.957505 ip-10-0-142-6 kubenswrapper[2564]: I0730 00:17:48.957453 2564 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-142-6.ec2.internal" event="NodeHasNoDiskPressure" Jul 30 00:17:48.957505 ip-10-0-142-6 kubenswrapper[2564]: I0730 00:17:48.957467 2564 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-142-6.ec2.internal" event="NodeHasSufficientPID" Jul 30 00:17:48.958046 ip-10-0-142-6 kubenswrapper[2564]: I0730 00:17:48.958030 2564 cpu_manager.go:222] "Starting CPU manager" policy="none" Jul 30 00:17:48.958046 ip-10-0-142-6 kubenswrapper[2564]: I0730 00:17:48.958045 2564 cpu_manager.go:223] "Reconciling" reconcilePeriod="10s" Jul 30 00:17:48.958161 ip-10-0-142-6 kubenswrapper[2564]: I0730 00:17:48.958061 2564 state_mem.go:36] "Initialized new in-memory state store" Jul 30 00:17:48.958161 ip-10-0-142-6 kubenswrapper[2564]: I0730 00:17:48.958032 2564 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 30 00:17:48.958861 ip-10-0-142-6 kubenswrapper[2564]: I0730 00:17:48.958847 2564 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-142-6.ec2.internal" event="NodeHasSufficientMemory" Jul 30 00:17:48.958908 ip-10-0-142-6 kubenswrapper[2564]: I0730 00:17:48.958879 2564 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-142-6.ec2.internal" event="NodeHasNoDiskPressure" Jul 30 00:17:48.958908 ip-10-0-142-6 kubenswrapper[2564]: I0730 00:17:48.958893 2564 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-142-6.ec2.internal" event="NodeHasSufficientPID" Jul 30 00:17:48.959807 ip-10-0-142-6 kubenswrapper[2564]: I0730 00:17:48.959796 2564 policy_none.go:49] "None policy: Start" Jul 30 00:17:48.959845 ip-10-0-142-6 kubenswrapper[2564]: I0730 00:17:48.959811 2564 memory_manager.go:187] "Starting memorymanager" policy="None" Jul 30 00:17:48.959845 ip-10-0-142-6 kubenswrapper[2564]: I0730 00:17:48.959821 2564 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint" Jul 30 00:17:48.960798 ip-10-0-142-6 kubenswrapper[2564]: I0730 00:17:48.960788 2564 policy_none.go:47] "Start" Jul 30 00:17:49.003984 ip-10-0-142-6 kubenswrapper[2564]: I0730 00:17:49.003814 2564 manager.go:337] "Starting Device Plugin manager" Jul 30 00:17:49.004166 ip-10-0-142-6 kubenswrapper[2564]: E0730 00:17:49.004047 2564 manager.go:513] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint" Jul 30 00:17:49.004166 ip-10-0-142-6 kubenswrapper[2564]: I0730 00:17:49.004062 2564 server.go:87] "Starting device plugin registration server" Jul 30 00:17:49.004417 ip-10-0-142-6 kubenswrapper[2564]: I0730 00:17:49.004402 2564 eviction_manager.go:189] "Eviction manager: starting control loop" Jul 30 00:17:49.004464 ip-10-0-142-6 kubenswrapper[2564]: I0730 00:17:49.004420 2564 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s" Jul 30 00:17:49.004524 ip-10-0-142-6 kubenswrapper[2564]: I0730 00:17:49.004513 2564 plugin_watcher.go:51] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry" Jul 30 00:17:49.004612 ip-10-0-142-6 kubenswrapper[2564]: I0730 00:17:49.004596 2564 plugin_manager.go:116] "The desired_state_of_world populator (plugin watcher) starts" Jul 30 00:17:49.004612 ip-10-0-142-6 kubenswrapper[2564]: I0730 00:17:49.004606 2564 plugin_manager.go:118] "Starting Kubelet Plugin Manager" Jul 30 00:17:49.005195 ip-10-0-142-6 kubenswrapper[2564]: E0730 00:17:49.005169 2564 eviction_manager.go:267] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\"" Jul 30 00:17:49.005291 ip-10-0-142-6 kubenswrapper[2564]: E0730 00:17:49.005228 2564 eviction_manager.go:292] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-142-6.ec2.internal\" not found" Jul 30 00:17:49.105323 ip-10-0-142-6 kubenswrapper[2564]: I0730 00:17:49.105279 2564 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 30 00:17:49.106355 ip-10-0-142-6 kubenswrapper[2564]: I0730 00:17:49.106267 2564 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-142-6.ec2.internal" event="NodeHasSufficientMemory" Jul 30 00:17:49.106355 ip-10-0-142-6 kubenswrapper[2564]: I0730 00:17:49.106305 2564 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-142-6.ec2.internal" event="NodeHasNoDiskPressure" Jul 30 00:17:49.106355 ip-10-0-142-6 kubenswrapper[2564]: I0730 00:17:49.106320 2564 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-142-6.ec2.internal" event="NodeHasSufficientPID" Jul 30 00:17:49.106355 ip-10-0-142-6 kubenswrapper[2564]: I0730 00:17:49.106354 2564 kubelet_node_status.go:78] "Attempting to register node" node="ip-10-0-142-6.ec2.internal" Jul 30 00:17:49.114178 ip-10-0-142-6 kubenswrapper[2564]: I0730 00:17:49.114130 2564 kubelet_node_status.go:81] "Successfully registered node" node="ip-10-0-142-6.ec2.internal" Jul 30 00:17:49.114178 ip-10-0-142-6 kubenswrapper[2564]: E0730 00:17:49.114154 2564 kubelet_node_status.go:536] "Error updating node status, will retry" err="error getting node \"ip-10-0-142-6.ec2.internal\": node \"ip-10-0-142-6.ec2.internal\" not found" Jul 30 00:17:49.126871 ip-10-0-142-6 kubenswrapper[2564]: E0730 00:17:49.126847 2564 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-142-6.ec2.internal\" not found" Jul 30 00:17:49.131998 ip-10-0-142-6 kubenswrapper[2564]: I0730 00:17:49.131971 2564 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4" Jul 30 00:17:49.133111 ip-10-0-142-6 kubenswrapper[2564]: I0730 00:17:49.133096 2564 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6" Jul 30 00:17:49.133234 ip-10-0-142-6 kubenswrapper[2564]: I0730 00:17:49.133128 2564 status_manager.go:244] "Starting to sync pod status with apiserver" Jul 30 00:17:49.133234 ip-10-0-142-6 kubenswrapper[2564]: I0730 00:17:49.133148 2564 kubelet.go:2448] "Starting kubelet main sync loop" Jul 30 00:17:49.133234 ip-10-0-142-6 kubenswrapper[2564]: E0730 00:17:49.133178 2564 kubelet.go:2472] "Skipping pod synchronization" err="PLEG is not healthy: pleg has yet to be successful" Jul 30 00:17:49.135210 ip-10-0-142-6 kubenswrapper[2564]: I0730 00:17:49.135195 2564 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160" Jul 30 00:17:49.227177 ip-10-0-142-6 kubenswrapper[2564]: E0730 00:17:49.227130 2564 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-142-6.ec2.internal\" not found" Jul 30 00:17:49.234309 ip-10-0-142-6 kubenswrapper[2564]: I0730 00:17:49.234275 2564 kubelet.go:2535] "SyncLoop ADD" source="file" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-142-6.ec2.internal","kube-system/kube-apiserver-proxy-ip-10-0-142-6.ec2.internal"] Jul 30 00:17:49.234444 ip-10-0-142-6 kubenswrapper[2564]: I0730 00:17:49.234367 2564 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 30 00:17:49.235883 ip-10-0-142-6 kubenswrapper[2564]: I0730 00:17:49.235868 2564 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-142-6.ec2.internal" event="NodeHasSufficientMemory" Jul 30 00:17:49.235978 ip-10-0-142-6 kubenswrapper[2564]: I0730 00:17:49.235895 2564 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-142-6.ec2.internal" event="NodeHasNoDiskPressure" Jul 30 00:17:49.235978 ip-10-0-142-6 kubenswrapper[2564]: I0730 00:17:49.235906 2564 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-142-6.ec2.internal" event="NodeHasSufficientPID" Jul 30 00:17:49.237185 ip-10-0-142-6 kubenswrapper[2564]: I0730 00:17:49.237170 2564 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 30 00:17:49.237296 ip-10-0-142-6 kubenswrapper[2564]: I0730 00:17:49.237280 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-142-6.ec2.internal" Jul 30 00:17:49.237331 ip-10-0-142-6 kubenswrapper[2564]: I0730 00:17:49.237314 2564 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 30 00:17:49.237876 ip-10-0-142-6 kubenswrapper[2564]: I0730 00:17:49.237860 2564 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-142-6.ec2.internal" event="NodeHasSufficientMemory" Jul 30 00:17:49.237945 ip-10-0-142-6 kubenswrapper[2564]: I0730 00:17:49.237881 2564 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-142-6.ec2.internal" event="NodeHasNoDiskPressure" Jul 30 00:17:49.237945 ip-10-0-142-6 kubenswrapper[2564]: I0730 00:17:49.237894 2564 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-142-6.ec2.internal" event="NodeHasSufficientPID" Jul 30 00:17:49.237945 ip-10-0-142-6 kubenswrapper[2564]: I0730 00:17:49.237918 2564 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-142-6.ec2.internal" event="NodeHasSufficientMemory" Jul 30 00:17:49.237945 ip-10-0-142-6 kubenswrapper[2564]: I0730 00:17:49.237940 2564 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-142-6.ec2.internal" event="NodeHasNoDiskPressure" Jul 30 00:17:49.238085 ip-10-0-142-6 kubenswrapper[2564]: I0730 00:17:49.237958 2564 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-142-6.ec2.internal" event="NodeHasSufficientPID" Jul 30 00:17:49.239161 ip-10-0-142-6 kubenswrapper[2564]: I0730 00:17:49.239148 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-142-6.ec2.internal" Jul 30 00:17:49.239204 ip-10-0-142-6 kubenswrapper[2564]: I0730 00:17:49.239171 2564 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 30 00:17:49.239842 ip-10-0-142-6 kubenswrapper[2564]: I0730 00:17:49.239826 2564 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-142-6.ec2.internal" event="NodeHasSufficientMemory" Jul 30 00:17:49.239904 ip-10-0-142-6 kubenswrapper[2564]: I0730 00:17:49.239856 2564 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-142-6.ec2.internal" event="NodeHasNoDiskPressure" Jul 30 00:17:49.239904 ip-10-0-142-6 kubenswrapper[2564]: I0730 00:17:49.239871 2564 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-142-6.ec2.internal" event="NodeHasSufficientPID" Jul 30 00:17:49.241926 ip-10-0-142-6 kubenswrapper[2564]: I0730 00:17:49.241909 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/7c269f27f5372a2f0fc63809270fef7a-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-142-6.ec2.internal\" (UID: \"7c269f27f5372a2f0fc63809270fef7a\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-142-6.ec2.internal" Jul 30 00:17:49.242001 ip-10-0-142-6 kubenswrapper[2564]: I0730 00:17:49.241935 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/7c269f27f5372a2f0fc63809270fef7a-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-142-6.ec2.internal\" (UID: \"7c269f27f5372a2f0fc63809270fef7a\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-142-6.ec2.internal" Jul 30 00:17:49.242001 ip-10-0-142-6 kubenswrapper[2564]: I0730 00:17:49.241952 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/514977950851753a7cc9d5e8c3264eee-config\") pod \"kube-apiserver-proxy-ip-10-0-142-6.ec2.internal\" (UID: \"514977950851753a7cc9d5e8c3264eee\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-142-6.ec2.internal" Jul 30 00:17:49.262824 ip-10-0-142-6 kubenswrapper[2564]: E0730 00:17:49.262804 2564 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-142-6.ec2.internal\" not found" node="ip-10-0-142-6.ec2.internal" Jul 30 00:17:49.266409 ip-10-0-142-6 kubenswrapper[2564]: E0730 00:17:49.266394 2564 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-142-6.ec2.internal\" not found" node="ip-10-0-142-6.ec2.internal" Jul 30 00:17:49.327572 ip-10-0-142-6 kubenswrapper[2564]: E0730 00:17:49.327539 2564 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-142-6.ec2.internal\" not found" Jul 30 00:17:49.342979 ip-10-0-142-6 kubenswrapper[2564]: I0730 00:17:49.342940 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/7c269f27f5372a2f0fc63809270fef7a-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-142-6.ec2.internal\" (UID: \"7c269f27f5372a2f0fc63809270fef7a\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-142-6.ec2.internal" Jul 30 00:17:49.342979 ip-10-0-142-6 kubenswrapper[2564]: I0730 00:17:49.342972 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/7c269f27f5372a2f0fc63809270fef7a-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-142-6.ec2.internal\" (UID: \"7c269f27f5372a2f0fc63809270fef7a\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-142-6.ec2.internal" Jul 30 00:17:49.343175 ip-10-0-142-6 kubenswrapper[2564]: I0730 00:17:49.342988 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/7c269f27f5372a2f0fc63809270fef7a-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-142-6.ec2.internal\" (UID: \"7c269f27f5372a2f0fc63809270fef7a\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-142-6.ec2.internal" Jul 30 00:17:49.343175 ip-10-0-142-6 kubenswrapper[2564]: I0730 00:17:49.343014 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/514977950851753a7cc9d5e8c3264eee-config\") pod \"kube-apiserver-proxy-ip-10-0-142-6.ec2.internal\" (UID: \"514977950851753a7cc9d5e8c3264eee\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-142-6.ec2.internal" Jul 30 00:17:49.343175 ip-10-0-142-6 kubenswrapper[2564]: I0730 00:17:49.342994 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/514977950851753a7cc9d5e8c3264eee-config\") pod \"kube-apiserver-proxy-ip-10-0-142-6.ec2.internal\" (UID: \"514977950851753a7cc9d5e8c3264eee\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-142-6.ec2.internal" Jul 30 00:17:49.343175 ip-10-0-142-6 kubenswrapper[2564]: I0730 00:17:49.343039 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/7c269f27f5372a2f0fc63809270fef7a-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-142-6.ec2.internal\" (UID: \"7c269f27f5372a2f0fc63809270fef7a\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-142-6.ec2.internal" Jul 30 00:17:49.428495 ip-10-0-142-6 kubenswrapper[2564]: E0730 00:17:49.428408 2564 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-142-6.ec2.internal\" not found" Jul 30 00:17:49.529011 ip-10-0-142-6 kubenswrapper[2564]: E0730 00:17:49.528970 2564 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-142-6.ec2.internal\" not found" Jul 30 00:17:49.564542 ip-10-0-142-6 kubenswrapper[2564]: I0730 00:17:49.564514 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-142-6.ec2.internal" Jul 30 00:17:49.570476 ip-10-0-142-6 kubenswrapper[2564]: I0730 00:17:49.570454 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-142-6.ec2.internal" Jul 30 00:17:49.629141 ip-10-0-142-6 kubenswrapper[2564]: E0730 00:17:49.629095 2564 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-142-6.ec2.internal\" not found" Jul 30 00:17:49.729671 ip-10-0-142-6 kubenswrapper[2564]: E0730 00:17:49.729613 2564 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-142-6.ec2.internal\" not found" Jul 30 00:17:49.830221 ip-10-0-142-6 kubenswrapper[2564]: E0730 00:17:49.830184 2564 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-142-6.ec2.internal\" not found" Jul 30 00:17:49.882765 ip-10-0-142-6 kubenswrapper[2564]: I0730 00:17:49.882716 2564 transport.go:147] "Certificate rotation detected, shutting down client connections to start using new credentials" Jul 30 00:17:49.883252 ip-10-0-142-6 kubenswrapper[2564]: I0730 00:17:49.882886 2564 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.RuntimeClass" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received" Jul 30 00:17:49.930494 ip-10-0-142-6 kubenswrapper[2564]: E0730 00:17:49.930460 2564 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-142-6.ec2.internal\" not found" Jul 30 00:17:49.945924 ip-10-0-142-6 kubenswrapper[2564]: I0730 00:17:49.945881 2564 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-07-29 00:12:48 +0000 UTC" deadline="2028-01-15 14:36:32.145083213 +0000 UTC" Jul 30 00:17:49.945924 ip-10-0-142-6 kubenswrapper[2564]: I0730 00:17:49.945914 2564 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="12830h18m42.199174763s" Jul 30 00:17:50.031192 ip-10-0-142-6 kubenswrapper[2564]: E0730 00:17:50.031118 2564 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-142-6.ec2.internal\" not found" Jul 30 00:17:50.131703 ip-10-0-142-6 kubenswrapper[2564]: E0730 00:17:50.131673 2564 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-142-6.ec2.internal\" not found" Jul 30 00:17:50.170886 ip-10-0-142-6 kubenswrapper[2564]: W0730 00:17:50.170855 2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod514977950851753a7cc9d5e8c3264eee.slice/crio-7df5f57222d51a768713c23f1ed86256a534a687ef821c574f0e52e3e12be38d WatchSource:0}: Error finding container 7df5f57222d51a768713c23f1ed86256a534a687ef821c574f0e52e3e12be38d: Status 404 returned error can't find the container with id 7df5f57222d51a768713c23f1ed86256a534a687ef821c574f0e52e3e12be38d Jul 30 00:17:50.171408 ip-10-0-142-6 kubenswrapper[2564]: W0730 00:17:50.171386 2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7c269f27f5372a2f0fc63809270fef7a.slice/crio-b561ea95d9260d35056b9fdf9036f568b2b57605155c615dfa6a9fd5e629c320 WatchSource:0}: Error finding container b561ea95d9260d35056b9fdf9036f568b2b57605155c615dfa6a9fd5e629c320: Status 404 returned error can't find the container with id b561ea95d9260d35056b9fdf9036f568b2b57605155c615dfa6a9fd5e629c320 Jul 30 00:17:50.174715 ip-10-0-142-6 kubenswrapper[2564]: I0730 00:17:50.174695 2564 reflector.go:439] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:160" Jul 30 00:17:50.175705 ip-10-0-142-6 kubenswrapper[2564]: I0730 00:17:50.175689 2564 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Jul 30 00:17:50.232423 ip-10-0-142-6 kubenswrapper[2564]: E0730 00:17:50.232392 2564 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-142-6.ec2.internal\" not found" Jul 30 00:17:50.233650 ip-10-0-142-6 kubenswrapper[2564]: I0730 00:17:50.233630 2564 reflector.go:439] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:160" Jul 30 00:17:50.239505 ip-10-0-142-6 kubenswrapper[2564]: I0730 00:17:50.239486 2564 kubelet.go:3248] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-142-6.ec2.internal" Jul 30 00:17:50.250934 ip-10-0-142-6 kubenswrapper[2564]: I0730 00:17:50.250912 2564 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Jul 30 00:17:50.252656 ip-10-0-142-6 kubenswrapper[2564]: I0730 00:17:50.252641 2564 kubelet.go:3248] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-142-6.ec2.internal" Jul 30 00:17:50.260186 ip-10-0-142-6 kubenswrapper[2564]: I0730 00:17:50.260169 2564 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Jul 30 00:17:50.301121 ip-10-0-142-6 kubenswrapper[2564]: I0730 00:17:50.301035 2564 reflector.go:439] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:160" Jul 30 00:17:50.921048 ip-10-0-142-6 kubenswrapper[2564]: I0730 00:17:50.921014 2564 apiserver.go:52] "Watching apiserver" Jul 30 00:17:50.932089 ip-10-0-142-6 kubenswrapper[2564]: I0730 00:17:50.932050 2564 reflector.go:439] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66" Jul 30 00:17:50.933096 ip-10-0-142-6 kubenswrapper[2564]: I0730 00:17:50.933046 2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-multus/multus-tphm4","openshift-multus/network-metrics-daemon-5skn2","openshift-network-diagnostics/network-check-target-j4ckt","kube-system/kube-apiserver-proxy-ip-10-0-142-6.ec2.internal","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6khjg","openshift-dns/node-resolver-6ssnm","openshift-image-registry/node-ca-jc8px","openshift-network-operator/iptables-alerter-k9l5m","openshift-ovn-kubernetes/ovnkube-node-w8grm","kube-system/global-pull-secret-syncer-d6phj","kube-system/konnectivity-agent-thhcs","openshift-cluster-node-tuning-operator/tuned-t69rb","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-142-6.ec2.internal","openshift-multus/multus-additional-cni-plugins-c6sdp"] Jul 30 00:17:50.934849 ip-10-0-142-6 kubenswrapper[2564]: I0730 00:17:50.934539 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-d6phj" Jul 30 00:17:50.934849 ip-10-0-142-6 kubenswrapper[2564]: E0730 00:17:50.934629 2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-d6phj" podUID="81450593-06b9-4f5a-81da-6e5fe67679f1" Jul 30 00:17:50.935850 ip-10-0-142-6 kubenswrapper[2564]: I0730 00:17:50.935830 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-thhcs" Jul 30 00:17:50.937059 ip-10-0-142-6 kubenswrapper[2564]: I0730 00:17:50.937038 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-j4ckt" Jul 30 00:17:50.937181 ip-10-0-142-6 kubenswrapper[2564]: E0730 00:17:50.937135 2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-j4ckt" podUID="a0bef212-99cc-47c3-b1da-316f26f776b9" Jul 30 00:17:50.938184 ip-10-0-142-6 kubenswrapper[2564]: I0730 00:17:50.938163 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-c6sdp" Jul 30 00:17:50.938941 ip-10-0-142-6 kubenswrapper[2564]: I0730 00:17:50.938912 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"default-dockercfg-4t5zd\"" Jul 30 00:17:50.939032 ip-10-0-142-6 kubenswrapper[2564]: I0730 00:17:50.939015 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\"" Jul 30 00:17:50.939120 ip-10-0-142-6 kubenswrapper[2564]: I0730 00:17:50.939103 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\"" Jul 30 00:17:50.939239 ip-10-0-142-6 kubenswrapper[2564]: I0730 00:17:50.939225 2564 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kubelet-serving" Jul 30 00:17:50.939505 ip-10-0-142-6 kubenswrapper[2564]: I0730 00:17:50.939484 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6khjg" Jul 30 00:17:50.940927 ip-10-0-142-6 kubenswrapper[2564]: I0730 00:17:50.940908 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-6ssnm" Jul 30 00:17:50.941052 ip-10-0-142-6 kubenswrapper[2564]: I0730 00:17:50.941036 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\"" Jul 30 00:17:50.941158 ip-10-0-142-6 kubenswrapper[2564]: I0730 00:17:50.941115 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\"" Jul 30 00:17:50.941158 ip-10-0-142-6 kubenswrapper[2564]: I0730 00:17:50.941041 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-rz2tc\"" Jul 30 00:17:50.941906 ip-10-0-142-6 kubenswrapper[2564]: I0730 00:17:50.941887 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\"" Jul 30 00:17:50.942260 ip-10-0-142-6 kubenswrapper[2564]: I0730 00:17:50.942242 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\"" Jul 30 00:17:50.942882 ip-10-0-142-6 kubenswrapper[2564]: I0730 00:17:50.942558 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-w8grm" Jul 30 00:17:50.943946 ip-10-0-142-6 kubenswrapper[2564]: I0730 00:17:50.943924 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\"" Jul 30 00:17:50.944230 ip-10-0-142-6 kubenswrapper[2564]: I0730 00:17:50.944213 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\"" Jul 30 00:17:50.944538 ip-10-0-142-6 kubenswrapper[2564]: I0730 00:17:50.944520 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\"" Jul 30 00:17:50.945147 ip-10-0-142-6 kubenswrapper[2564]: I0730 00:17:50.945130 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-k9l5m" Jul 30 00:17:50.945716 ip-10-0-142-6 kubenswrapper[2564]: I0730 00:17:50.945697 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-5skn2" Jul 30 00:17:50.945793 ip-10-0-142-6 kubenswrapper[2564]: E0730 00:17:50.945767 2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-5skn2" podUID="f82506d3-a3ca-415e-b6c0-4de2f2bf35f5" Jul 30 00:17:50.947068 ip-10-0-142-6 kubenswrapper[2564]: I0730 00:17:50.946535 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\"" Jul 30 00:17:50.947068 ip-10-0-142-6 kubenswrapper[2564]: I0730 00:17:50.946568 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-tphm4" Jul 30 00:17:50.947068 ip-10-0-142-6 kubenswrapper[2564]: I0730 00:17:50.946955 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-mnfll\"" Jul 30 00:17:50.947696 ip-10-0-142-6 kubenswrapper[2564]: I0730 00:17:50.947681 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\"" Jul 30 00:17:50.948010 ip-10-0-142-6 kubenswrapper[2564]: I0730 00:17:50.947993 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\"" Jul 30 00:17:50.956912 ip-10-0-142-6 kubenswrapper[2564]: I0730 00:17:50.956813 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/6a73d10b-1baa-42ac-a83a-2336014e3600-konnectivity-ca\") pod \"konnectivity-agent-thhcs\" (UID: \"6a73d10b-1baa-42ac-a83a-2336014e3600\") " pod="kube-system/konnectivity-agent-thhcs" Jul 30 00:17:50.957098 ip-10-0-142-6 kubenswrapper[2564]: I0730 00:17:50.956934 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/b1186f7d-220d-4210-a7f8-6f3f0cf554e8-os-release\") pod \"multus-additional-cni-plugins-c6sdp\" (UID: \"b1186f7d-220d-4210-a7f8-6f3f0cf554e8\") " pod="openshift-multus/multus-additional-cni-plugins-c6sdp" Jul 30 00:17:50.957098 ip-10-0-142-6 kubenswrapper[2564]: I0730 00:17:50.956965 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-jc8px" Jul 30 00:17:50.957098 ip-10-0-142-6 kubenswrapper[2564]: I0730 00:17:50.956969 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/b1186f7d-220d-4210-a7f8-6f3f0cf554e8-tuning-conf-dir\") pod \"multus-additional-cni-plugins-c6sdp\" (UID: \"b1186f7d-220d-4210-a7f8-6f3f0cf554e8\") " pod="openshift-multus/multus-additional-cni-plugins-c6sdp" Jul 30 00:17:50.957229 ip-10-0-142-6 kubenswrapper[2564]: I0730 00:17:50.957104 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gvd6g\" (UniqueName: \"kubernetes.io/projected/b1186f7d-220d-4210-a7f8-6f3f0cf554e8-kube-api-access-gvd6g\") pod \"multus-additional-cni-plugins-c6sdp\" (UID: \"b1186f7d-220d-4210-a7f8-6f3f0cf554e8\") " pod="openshift-multus/multus-additional-cni-plugins-c6sdp" Jul 30 00:17:50.957229 ip-10-0-142-6 kubenswrapper[2564]: I0730 00:17:50.957139 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9gm94\" (UniqueName: \"kubernetes.io/projected/99deae04-e4b4-4491-a31f-945927bd1e72-kube-api-access-9gm94\") pod \"node-resolver-6ssnm\" (UID: \"99deae04-e4b4-4491-a31f-945927bd1e72\") " pod="openshift-dns/node-resolver-6ssnm" Jul 30 00:17:50.957374 ip-10-0-142-6 kubenswrapper[2564]: I0730 00:17:50.957353 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/81450593-06b9-4f5a-81da-6e5fe67679f1-kubelet-config\") pod \"global-pull-secret-syncer-d6phj\" (UID: \"81450593-06b9-4f5a-81da-6e5fe67679f1\") " pod="kube-system/global-pull-secret-syncer-d6phj" Jul 30 00:17:50.957436 ip-10-0-142-6 kubenswrapper[2564]: I0730 00:17:50.957416 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/b1186f7d-220d-4210-a7f8-6f3f0cf554e8-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-c6sdp\" (UID: \"b1186f7d-220d-4210-a7f8-6f3f0cf554e8\") " pod="openshift-multus/multus-additional-cni-plugins-c6sdp" Jul 30 00:17:50.957471 ip-10-0-142-6 kubenswrapper[2564]: I0730 00:17:50.957461 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/3a12258d-f063-4ae8-8fec-9eb3b012df06-kubelet-dir\") pod \"aws-ebs-csi-driver-node-6khjg\" (UID: \"3a12258d-f063-4ae8-8fec-9eb3b012df06\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6khjg" Jul 30 00:17:50.958371 ip-10-0-142-6 kubenswrapper[2564]: I0730 00:17:50.957491 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/3a12258d-f063-4ae8-8fec-9eb3b012df06-registration-dir\") pod \"aws-ebs-csi-driver-node-6khjg\" (UID: \"3a12258d-f063-4ae8-8fec-9eb3b012df06\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6khjg" Jul 30 00:17:50.958371 ip-10-0-142-6 kubenswrapper[2564]: I0730 00:17:50.957615 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/3a12258d-f063-4ae8-8fec-9eb3b012df06-sys-fs\") pod \"aws-ebs-csi-driver-node-6khjg\" (UID: \"3a12258d-f063-4ae8-8fec-9eb3b012df06\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6khjg" Jul 30 00:17:50.958371 ip-10-0-142-6 kubenswrapper[2564]: I0730 00:17:50.957682 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/99deae04-e4b4-4491-a31f-945927bd1e72-hosts-file\") pod \"node-resolver-6ssnm\" (UID: \"99deae04-e4b4-4491-a31f-945927bd1e72\") " pod="openshift-dns/node-resolver-6ssnm" Jul 30 00:17:50.958371 ip-10-0-142-6 kubenswrapper[2564]: I0730 00:17:50.957735 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/99deae04-e4b4-4491-a31f-945927bd1e72-tmp-dir\") pod \"node-resolver-6ssnm\" (UID: \"99deae04-e4b4-4491-a31f-945927bd1e72\") " pod="openshift-dns/node-resolver-6ssnm" Jul 30 00:17:50.958371 ip-10-0-142-6 kubenswrapper[2564]: I0730 00:17:50.957768 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/b1186f7d-220d-4210-a7f8-6f3f0cf554e8-system-cni-dir\") pod \"multus-additional-cni-plugins-c6sdp\" (UID: \"b1186f7d-220d-4210-a7f8-6f3f0cf554e8\") " pod="openshift-multus/multus-additional-cni-plugins-c6sdp" Jul 30 00:17:50.958371 ip-10-0-142-6 kubenswrapper[2564]: I0730 00:17:50.957803 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/b1186f7d-220d-4210-a7f8-6f3f0cf554e8-cnibin\") pod \"multus-additional-cni-plugins-c6sdp\" (UID: \"b1186f7d-220d-4210-a7f8-6f3f0cf554e8\") " pod="openshift-multus/multus-additional-cni-plugins-c6sdp" Jul 30 00:17:50.958371 ip-10-0-142-6 kubenswrapper[2564]: I0730 00:17:50.957856 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/b1186f7d-220d-4210-a7f8-6f3f0cf554e8-cni-binary-copy\") pod \"multus-additional-cni-plugins-c6sdp\" (UID: \"b1186f7d-220d-4210-a7f8-6f3f0cf554e8\") " pod="openshift-multus/multus-additional-cni-plugins-c6sdp" Jul 30 00:17:50.958371 ip-10-0-142-6 kubenswrapper[2564]: I0730 00:17:50.957891 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/3a12258d-f063-4ae8-8fec-9eb3b012df06-device-dir\") pod \"aws-ebs-csi-driver-node-6khjg\" (UID: \"3a12258d-f063-4ae8-8fec-9eb3b012df06\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6khjg" Jul 30 00:17:50.958371 ip-10-0-142-6 kubenswrapper[2564]: I0730 00:17:50.957959 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/81450593-06b9-4f5a-81da-6e5fe67679f1-dbus\") pod \"global-pull-secret-syncer-d6phj\" (UID: \"81450593-06b9-4f5a-81da-6e5fe67679f1\") " pod="kube-system/global-pull-secret-syncer-d6phj" Jul 30 00:17:50.958371 ip-10-0-142-6 kubenswrapper[2564]: I0730 00:17:50.958007 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\"" Jul 30 00:17:50.958371 ip-10-0-142-6 kubenswrapper[2564]: I0730 00:17:50.958011 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/6a73d10b-1baa-42ac-a83a-2336014e3600-agent-certs\") pod \"konnectivity-agent-thhcs\" (UID: \"6a73d10b-1baa-42ac-a83a-2336014e3600\") " pod="kube-system/konnectivity-agent-thhcs" Jul 30 00:17:50.958371 ip-10-0-142-6 kubenswrapper[2564]: I0730 00:17:50.958145 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\"" Jul 30 00:17:50.958371 ip-10-0-142-6 kubenswrapper[2564]: I0730 00:17:50.958161 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\"" Jul 30 00:17:50.958371 ip-10-0-142-6 kubenswrapper[2564]: I0730 00:17:50.958206 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-4jzh5\"" Jul 30 00:17:50.958371 ip-10-0-142-6 kubenswrapper[2564]: I0730 00:17:50.958216 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\"" Jul 30 00:17:50.958371 ip-10-0-142-6 kubenswrapper[2564]: I0730 00:17:50.958233 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\"" Jul 30 00:17:50.958371 ip-10-0-142-6 kubenswrapper[2564]: I0730 00:17:50.958305 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-bdwfx\"" Jul 30 00:17:50.958371 ip-10-0-142-6 kubenswrapper[2564]: I0730 00:17:50.958155 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5v98n\" (UniqueName: \"kubernetes.io/projected/a0bef212-99cc-47c3-b1da-316f26f776b9-kube-api-access-5v98n\") pod \"network-check-target-j4ckt\" (UID: \"a0bef212-99cc-47c3-b1da-316f26f776b9\") " pod="openshift-network-diagnostics/network-check-target-j4ckt" Jul 30 00:17:50.959269 ip-10-0-142-6 kubenswrapper[2564]: I0730 00:17:50.958846 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/b1186f7d-220d-4210-a7f8-6f3f0cf554e8-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-c6sdp\" (UID: \"b1186f7d-220d-4210-a7f8-6f3f0cf554e8\") " pod="openshift-multus/multus-additional-cni-plugins-c6sdp" Jul 30 00:17:50.959269 ip-10-0-142-6 kubenswrapper[2564]: I0730 00:17:50.958880 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/3a12258d-f063-4ae8-8fec-9eb3b012df06-socket-dir\") pod \"aws-ebs-csi-driver-node-6khjg\" (UID: \"3a12258d-f063-4ae8-8fec-9eb3b012df06\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6khjg" Jul 30 00:17:50.959269 ip-10-0-142-6 kubenswrapper[2564]: I0730 00:17:50.958907 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/3a12258d-f063-4ae8-8fec-9eb3b012df06-etc-selinux\") pod \"aws-ebs-csi-driver-node-6khjg\" (UID: \"3a12258d-f063-4ae8-8fec-9eb3b012df06\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6khjg" Jul 30 00:17:50.959269 ip-10-0-142-6 kubenswrapper[2564]: I0730 00:17:50.958932 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ndwzs\" (UniqueName: \"kubernetes.io/projected/3a12258d-f063-4ae8-8fec-9eb3b012df06-kube-api-access-ndwzs\") pod \"aws-ebs-csi-driver-node-6khjg\" (UID: \"3a12258d-f063-4ae8-8fec-9eb3b012df06\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6khjg" Jul 30 00:17:50.959269 ip-10-0-142-6 kubenswrapper[2564]: I0730 00:17:50.958955 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/81450593-06b9-4f5a-81da-6e5fe67679f1-original-pull-secret\") pod \"global-pull-secret-syncer-d6phj\" (UID: \"81450593-06b9-4f5a-81da-6e5fe67679f1\") " pod="kube-system/global-pull-secret-syncer-d6phj" Jul 30 00:17:50.959269 ip-10-0-142-6 kubenswrapper[2564]: I0730 00:17:50.959176 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-t69rb" Jul 30 00:17:50.959526 ip-10-0-142-6 kubenswrapper[2564]: I0730 00:17:50.959345 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\"" Jul 30 00:17:50.959526 ip-10-0-142-6 kubenswrapper[2564]: I0730 00:17:50.959406 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-2zz5r\"" Jul 30 00:17:50.959585 ip-10-0-142-6 kubenswrapper[2564]: I0730 00:17:50.959569 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\"" Jul 30 00:17:50.959616 ip-10-0-142-6 kubenswrapper[2564]: I0730 00:17:50.959596 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"default-dockercfg-zf875\"" Jul 30 00:17:50.961020 ip-10-0-142-6 kubenswrapper[2564]: I0730 00:17:50.959665 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\"" Jul 30 00:17:50.961020 ip-10-0-142-6 kubenswrapper[2564]: I0730 00:17:50.959777 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\"" Jul 30 00:17:50.961020 ip-10-0-142-6 kubenswrapper[2564]: I0730 00:17:50.959780 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\"" Jul 30 00:17:50.961020 ip-10-0-142-6 kubenswrapper[2564]: I0730 00:17:50.959800 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\"" Jul 30 00:17:50.961020 ip-10-0-142-6 kubenswrapper[2564]: I0730 00:17:50.960607 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-kgd84\"" Jul 30 00:17:50.961020 ip-10-0-142-6 kubenswrapper[2564]: I0730 00:17:50.960630 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\"" Jul 30 00:17:50.971695 ip-10-0-142-6 kubenswrapper[2564]: I0730 00:17:50.971474 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\"" Jul 30 00:17:50.971695 ip-10-0-142-6 kubenswrapper[2564]: I0730 00:17:50.971505 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\"" Jul 30 00:17:50.971695 ip-10-0-142-6 kubenswrapper[2564]: I0730 00:17:50.971579 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-stk5q\"" Jul 30 00:17:50.973650 ip-10-0-142-6 kubenswrapper[2564]: I0730 00:17:50.973219 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\"" Jul 30 00:17:50.982134 ip-10-0-142-6 kubenswrapper[2564]: I0730 00:17:50.982111 2564 reflector.go:439] "Caches populated" logger="kubernetes.io/kubelet-serving" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" Jul 30 00:17:51.005082 ip-10-0-142-6 kubenswrapper[2564]: I0730 00:17:51.005033 2564 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kubelet-serving" csr="csr-gpksb" Jul 30 00:17:51.016801 ip-10-0-142-6 kubenswrapper[2564]: I0730 00:17:51.016713 2564 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kubelet-serving" csr="csr-gpksb" Jul 30 00:17:51.041702 ip-10-0-142-6 kubenswrapper[2564]: I0730 00:17:51.041674 2564 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world" Jul 30 00:17:51.059412 ip-10-0-142-6 kubenswrapper[2564]: I0730 00:17:51.059378 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/e7215f9f-4c75-4ab5-85c7-e10ab231e916-node-log\") pod \"ovnkube-node-w8grm\" (UID: \"e7215f9f-4c75-4ab5-85c7-e10ab231e916\") " pod="openshift-ovn-kubernetes/ovnkube-node-w8grm" Jul 30 00:17:51.059577 ip-10-0-142-6 kubenswrapper[2564]: I0730 00:17:51.059419 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/e7215f9f-4c75-4ab5-85c7-e10ab231e916-ovn-node-metrics-cert\") pod \"ovnkube-node-w8grm\" (UID: \"e7215f9f-4c75-4ab5-85c7-e10ab231e916\") " pod="openshift-ovn-kubernetes/ovnkube-node-w8grm" Jul 30 00:17:51.059577 ip-10-0-142-6 kubenswrapper[2564]: I0730 00:17:51.059450 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/d2866704-b7d7-4515-9b9d-4906b1508f68-system-cni-dir\") pod \"multus-tphm4\" (UID: \"d2866704-b7d7-4515-9b9d-4906b1508f68\") " pod="openshift-multus/multus-tphm4" Jul 30 00:17:51.059577 ip-10-0-142-6 kubenswrapper[2564]: I0730 00:17:51.059498 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/d2866704-b7d7-4515-9b9d-4906b1508f68-os-release\") pod \"multus-tphm4\" (UID: \"d2866704-b7d7-4515-9b9d-4906b1508f68\") " pod="openshift-multus/multus-tphm4" Jul 30 00:17:51.059745 ip-10-0-142-6 kubenswrapper[2564]: I0730 00:17:51.059571 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/d2866704-b7d7-4515-9b9d-4906b1508f68-host-var-lib-cni-multus\") pod \"multus-tphm4\" (UID: \"d2866704-b7d7-4515-9b9d-4906b1508f68\") " pod="openshift-multus/multus-tphm4" Jul 30 00:17:51.059745 ip-10-0-142-6 kubenswrapper[2564]: I0730 00:17:51.059610 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/d2866704-b7d7-4515-9b9d-4906b1508f68-multus-conf-dir\") pod \"multus-tphm4\" (UID: \"d2866704-b7d7-4515-9b9d-4906b1508f68\") " pod="openshift-multus/multus-tphm4" Jul 30 00:17:51.059745 ip-10-0-142-6 kubenswrapper[2564]: I0730 00:17:51.059643 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/7ff24a6e-78df-4c03-830a-453386386626-host\") pod \"node-ca-jc8px\" (UID: \"7ff24a6e-78df-4c03-830a-453386386626\") " pod="openshift-image-registry/node-ca-jc8px" Jul 30 00:17:51.059745 ip-10-0-142-6 kubenswrapper[2564]: I0730 00:17:51.059669 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/6a73d10b-1baa-42ac-a83a-2336014e3600-konnectivity-ca\") pod \"konnectivity-agent-thhcs\" (UID: \"6a73d10b-1baa-42ac-a83a-2336014e3600\") " pod="kube-system/konnectivity-agent-thhcs" Jul 30 00:17:51.059745 ip-10-0-142-6 kubenswrapper[2564]: I0730 00:17:51.059687 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/b1186f7d-220d-4210-a7f8-6f3f0cf554e8-os-release\") pod \"multus-additional-cni-plugins-c6sdp\" (UID: \"b1186f7d-220d-4210-a7f8-6f3f0cf554e8\") " pod="openshift-multus/multus-additional-cni-plugins-c6sdp" Jul 30 00:17:51.059745 ip-10-0-142-6 kubenswrapper[2564]: I0730 00:17:51.059714 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/853c653e-d075-4220-ae9c-86b76caa2229-etc-kubernetes\") pod \"tuned-t69rb\" (UID: \"853c653e-d075-4220-ae9c-86b76caa2229\") " pod="openshift-cluster-node-tuning-operator/tuned-t69rb" Jul 30 00:17:51.059745 ip-10-0-142-6 kubenswrapper[2564]: I0730 00:17:51.059736 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/e7215f9f-4c75-4ab5-85c7-e10ab231e916-host-run-ovn-kubernetes\") pod \"ovnkube-node-w8grm\" (UID: \"e7215f9f-4c75-4ab5-85c7-e10ab231e916\") " pod="openshift-ovn-kubernetes/ovnkube-node-w8grm" Jul 30 00:17:51.060105 ip-10-0-142-6 kubenswrapper[2564]: I0730 00:17:51.059759 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/d2866704-b7d7-4515-9b9d-4906b1508f68-multus-cni-dir\") pod \"multus-tphm4\" (UID: \"d2866704-b7d7-4515-9b9d-4906b1508f68\") " pod="openshift-multus/multus-tphm4" Jul 30 00:17:51.060105 ip-10-0-142-6 kubenswrapper[2564]: I0730 00:17:51.059801 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/d2866704-b7d7-4515-9b9d-4906b1508f68-host-run-k8s-cni-cncf-io\") pod \"multus-tphm4\" (UID: \"d2866704-b7d7-4515-9b9d-4906b1508f68\") " pod="openshift-multus/multus-tphm4" Jul 30 00:17:51.060105 ip-10-0-142-6 kubenswrapper[2564]: I0730 00:17:51.059842 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/b1186f7d-220d-4210-a7f8-6f3f0cf554e8-system-cni-dir\") pod \"multus-additional-cni-plugins-c6sdp\" (UID: \"b1186f7d-220d-4210-a7f8-6f3f0cf554e8\") " pod="openshift-multus/multus-additional-cni-plugins-c6sdp" Jul 30 00:17:51.060105 ip-10-0-142-6 kubenswrapper[2564]: I0730 00:17:51.059868 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/b1186f7d-220d-4210-a7f8-6f3f0cf554e8-os-release\") pod \"multus-additional-cni-plugins-c6sdp\" (UID: \"b1186f7d-220d-4210-a7f8-6f3f0cf554e8\") " pod="openshift-multus/multus-additional-cni-plugins-c6sdp" Jul 30 00:17:51.060105 ip-10-0-142-6 kubenswrapper[2564]: I0730 00:17:51.059900 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/b1186f7d-220d-4210-a7f8-6f3f0cf554e8-system-cni-dir\") pod \"multus-additional-cni-plugins-c6sdp\" (UID: \"b1186f7d-220d-4210-a7f8-6f3f0cf554e8\") " pod="openshift-multus/multus-additional-cni-plugins-c6sdp" Jul 30 00:17:51.060105 ip-10-0-142-6 kubenswrapper[2564]: I0730 00:17:51.059915 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/b1186f7d-220d-4210-a7f8-6f3f0cf554e8-cni-binary-copy\") pod \"multus-additional-cni-plugins-c6sdp\" (UID: \"b1186f7d-220d-4210-a7f8-6f3f0cf554e8\") " pod="openshift-multus/multus-additional-cni-plugins-c6sdp" Jul 30 00:17:51.060105 ip-10-0-142-6 kubenswrapper[2564]: I0730 00:17:51.059950 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/e7215f9f-4c75-4ab5-85c7-e10ab231e916-host-run-netns\") pod \"ovnkube-node-w8grm\" (UID: \"e7215f9f-4c75-4ab5-85c7-e10ab231e916\") " pod="openshift-ovn-kubernetes/ovnkube-node-w8grm" Jul 30 00:17:51.060105 ip-10-0-142-6 kubenswrapper[2564]: I0730 00:17:51.060100 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/853c653e-d075-4220-ae9c-86b76caa2229-etc-sysconfig\") pod \"tuned-t69rb\" (UID: \"853c653e-d075-4220-ae9c-86b76caa2229\") " pod="openshift-cluster-node-tuning-operator/tuned-t69rb" Jul 30 00:17:51.060563 ip-10-0-142-6 kubenswrapper[2564]: I0730 00:17:51.060140 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5v98n\" (UniqueName: \"kubernetes.io/projected/a0bef212-99cc-47c3-b1da-316f26f776b9-kube-api-access-5v98n\") pod \"network-check-target-j4ckt\" (UID: \"a0bef212-99cc-47c3-b1da-316f26f776b9\") " pod="openshift-network-diagnostics/network-check-target-j4ckt" Jul 30 00:17:51.060563 ip-10-0-142-6 kubenswrapper[2564]: I0730 00:17:51.060185 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/3a12258d-f063-4ae8-8fec-9eb3b012df06-etc-selinux\") pod \"aws-ebs-csi-driver-node-6khjg\" (UID: \"3a12258d-f063-4ae8-8fec-9eb3b012df06\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6khjg" Jul 30 00:17:51.060563 ip-10-0-142-6 kubenswrapper[2564]: I0730 00:17:51.060210 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/81450593-06b9-4f5a-81da-6e5fe67679f1-original-pull-secret\") pod \"global-pull-secret-syncer-d6phj\" (UID: \"81450593-06b9-4f5a-81da-6e5fe67679f1\") " pod="kube-system/global-pull-secret-syncer-d6phj" Jul 30 00:17:51.060563 ip-10-0-142-6 kubenswrapper[2564]: I0730 00:17:51.060330 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/3a12258d-f063-4ae8-8fec-9eb3b012df06-etc-selinux\") pod \"aws-ebs-csi-driver-node-6khjg\" (UID: \"3a12258d-f063-4ae8-8fec-9eb3b012df06\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6khjg" Jul 30 00:17:51.060563 ip-10-0-142-6 kubenswrapper[2564]: I0730 00:17:51.060330 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/6a73d10b-1baa-42ac-a83a-2336014e3600-konnectivity-ca\") pod \"konnectivity-agent-thhcs\" (UID: \"6a73d10b-1baa-42ac-a83a-2336014e3600\") " pod="kube-system/konnectivity-agent-thhcs" Jul 30 00:17:51.060563 ip-10-0-142-6 kubenswrapper[2564]: I0730 00:17:51.060464 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/e7215f9f-4c75-4ab5-85c7-e10ab231e916-host-slash\") pod \"ovnkube-node-w8grm\" (UID: \"e7215f9f-4c75-4ab5-85c7-e10ab231e916\") " pod="openshift-ovn-kubernetes/ovnkube-node-w8grm" Jul 30 00:17:51.060563 ip-10-0-142-6 kubenswrapper[2564]: I0730 00:17:51.060508 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/e7215f9f-4c75-4ab5-85c7-e10ab231e916-run-ovn\") pod \"ovnkube-node-w8grm\" (UID: \"e7215f9f-4c75-4ab5-85c7-e10ab231e916\") " pod="openshift-ovn-kubernetes/ovnkube-node-w8grm" Jul 30 00:17:51.060882 ip-10-0-142-6 kubenswrapper[2564]: I0730 00:17:51.060572 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/853c653e-d075-4220-ae9c-86b76caa2229-etc-sysctl-d\") pod \"tuned-t69rb\" (UID: \"853c653e-d075-4220-ae9c-86b76caa2229\") " pod="openshift-cluster-node-tuning-operator/tuned-t69rb" Jul 30 00:17:51.060882 ip-10-0-142-6 kubenswrapper[2564]: I0730 00:17:51.060622 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/d2866704-b7d7-4515-9b9d-4906b1508f68-multus-socket-dir-parent\") pod \"multus-tphm4\" (UID: \"d2866704-b7d7-4515-9b9d-4906b1508f68\") " pod="openshift-multus/multus-tphm4" Jul 30 00:17:51.060882 ip-10-0-142-6 kubenswrapper[2564]: I0730 00:17:51.060657 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/853c653e-d075-4220-ae9c-86b76caa2229-host\") pod \"tuned-t69rb\" (UID: \"853c653e-d075-4220-ae9c-86b76caa2229\") " pod="openshift-cluster-node-tuning-operator/tuned-t69rb" Jul 30 00:17:51.060882 ip-10-0-142-6 kubenswrapper[2564]: I0730 00:17:51.060683 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/853c653e-d075-4220-ae9c-86b76caa2229-etc-tuned\") pod \"tuned-t69rb\" (UID: \"853c653e-d075-4220-ae9c-86b76caa2229\") " pod="openshift-cluster-node-tuning-operator/tuned-t69rb" Jul 30 00:17:51.060882 ip-10-0-142-6 kubenswrapper[2564]: I0730 00:17:51.060717 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5drxq\" (UniqueName: \"kubernetes.io/projected/853c653e-d075-4220-ae9c-86b76caa2229-kube-api-access-5drxq\") pod \"tuned-t69rb\" (UID: \"853c653e-d075-4220-ae9c-86b76caa2229\") " pod="openshift-cluster-node-tuning-operator/tuned-t69rb" Jul 30 00:17:51.060882 ip-10-0-142-6 kubenswrapper[2564]: I0730 00:17:51.060752 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/3a12258d-f063-4ae8-8fec-9eb3b012df06-registration-dir\") pod \"aws-ebs-csi-driver-node-6khjg\" (UID: \"3a12258d-f063-4ae8-8fec-9eb3b012df06\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6khjg" Jul 30 00:17:51.060882 ip-10-0-142-6 kubenswrapper[2564]: I0730 00:17:51.060776 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/3a12258d-f063-4ae8-8fec-9eb3b012df06-sys-fs\") pod \"aws-ebs-csi-driver-node-6khjg\" (UID: \"3a12258d-f063-4ae8-8fec-9eb3b012df06\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6khjg" Jul 30 00:17:51.060882 ip-10-0-142-6 kubenswrapper[2564]: I0730 00:17:51.060804 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/e7215f9f-4c75-4ab5-85c7-e10ab231e916-run-openvswitch\") pod \"ovnkube-node-w8grm\" (UID: \"e7215f9f-4c75-4ab5-85c7-e10ab231e916\") " pod="openshift-ovn-kubernetes/ovnkube-node-w8grm" Jul 30 00:17:51.060882 ip-10-0-142-6 kubenswrapper[2564]: I0730 00:17:51.060869 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/3a12258d-f063-4ae8-8fec-9eb3b012df06-registration-dir\") pod \"aws-ebs-csi-driver-node-6khjg\" (UID: \"3a12258d-f063-4ae8-8fec-9eb3b012df06\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6khjg" Jul 30 00:17:51.060882 ip-10-0-142-6 kubenswrapper[2564]: I0730 00:17:51.060869 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/3a12258d-f063-4ae8-8fec-9eb3b012df06-sys-fs\") pod \"aws-ebs-csi-driver-node-6khjg\" (UID: \"3a12258d-f063-4ae8-8fec-9eb3b012df06\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6khjg" Jul 30 00:17:51.061451 ip-10-0-142-6 kubenswrapper[2564]: I0730 00:17:51.060887 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/b1186f7d-220d-4210-a7f8-6f3f0cf554e8-cni-binary-copy\") pod \"multus-additional-cni-plugins-c6sdp\" (UID: \"b1186f7d-220d-4210-a7f8-6f3f0cf554e8\") " pod="openshift-multus/multus-additional-cni-plugins-c6sdp" Jul 30 00:17:51.061451 ip-10-0-142-6 kubenswrapper[2564]: I0730 00:17:51.060890 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/d2866704-b7d7-4515-9b9d-4906b1508f68-host-var-lib-kubelet\") pod \"multus-tphm4\" (UID: \"d2866704-b7d7-4515-9b9d-4906b1508f68\") " pod="openshift-multus/multus-tphm4" Jul 30 00:17:51.061451 ip-10-0-142-6 kubenswrapper[2564]: I0730 00:17:51.060940 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/dcc57a98-213c-4ed7-8b59-c0dd84a39b36-host-slash\") pod \"iptables-alerter-k9l5m\" (UID: \"dcc57a98-213c-4ed7-8b59-c0dd84a39b36\") " pod="openshift-network-operator/iptables-alerter-k9l5m" Jul 30 00:17:51.061451 ip-10-0-142-6 kubenswrapper[2564]: I0730 00:17:51.060965 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5bftn\" (UniqueName: \"kubernetes.io/projected/dcc57a98-213c-4ed7-8b59-c0dd84a39b36-kube-api-access-5bftn\") pod \"iptables-alerter-k9l5m\" (UID: \"dcc57a98-213c-4ed7-8b59-c0dd84a39b36\") " pod="openshift-network-operator/iptables-alerter-k9l5m" Jul 30 00:17:51.061451 ip-10-0-142-6 kubenswrapper[2564]: I0730 00:17:51.060991 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/853c653e-d075-4220-ae9c-86b76caa2229-sys\") pod \"tuned-t69rb\" (UID: \"853c653e-d075-4220-ae9c-86b76caa2229\") " pod="openshift-cluster-node-tuning-operator/tuned-t69rb" Jul 30 00:17:51.061451 ip-10-0-142-6 kubenswrapper[2564]: I0730 00:17:51.061013 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/e7215f9f-4c75-4ab5-85c7-e10ab231e916-ovnkube-config\") pod \"ovnkube-node-w8grm\" (UID: \"e7215f9f-4c75-4ab5-85c7-e10ab231e916\") " pod="openshift-ovn-kubernetes/ovnkube-node-w8grm" Jul 30 00:17:51.061451 ip-10-0-142-6 kubenswrapper[2564]: E0730 00:17:51.061045 2564 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 30 00:17:51.061451 ip-10-0-142-6 kubenswrapper[2564]: I0730 00:17:51.061036 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/e7215f9f-4c75-4ab5-85c7-e10ab231e916-ovnkube-script-lib\") pod \"ovnkube-node-w8grm\" (UID: \"e7215f9f-4c75-4ab5-85c7-e10ab231e916\") " pod="openshift-ovn-kubernetes/ovnkube-node-w8grm" Jul 30 00:17:51.061451 ip-10-0-142-6 kubenswrapper[2564]: I0730 00:17:51.061097 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6v9nv\" (UniqueName: \"kubernetes.io/projected/f82506d3-a3ca-415e-b6c0-4de2f2bf35f5-kube-api-access-6v9nv\") pod \"network-metrics-daemon-5skn2\" (UID: \"f82506d3-a3ca-415e-b6c0-4de2f2bf35f5\") " pod="openshift-multus/network-metrics-daemon-5skn2" Jul 30 00:17:51.061451 ip-10-0-142-6 kubenswrapper[2564]: E0730 00:17:51.061140 2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/81450593-06b9-4f5a-81da-6e5fe67679f1-original-pull-secret podName:81450593-06b9-4f5a-81da-6e5fe67679f1 nodeName:}" failed. No retries permitted until 2026-07-30 00:17:51.561108486 +0000 UTC m=+2.975848403 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/81450593-06b9-4f5a-81da-6e5fe67679f1-original-pull-secret") pod "global-pull-secret-syncer-d6phj" (UID: "81450593-06b9-4f5a-81da-6e5fe67679f1") : object "kube-system"/"original-pull-secret" not registered Jul 30 00:17:51.061451 ip-10-0-142-6 kubenswrapper[2564]: I0730 00:17:51.061160 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/853c653e-d075-4220-ae9c-86b76caa2229-etc-modprobe-d\") pod \"tuned-t69rb\" (UID: \"853c653e-d075-4220-ae9c-86b76caa2229\") " pod="openshift-cluster-node-tuning-operator/tuned-t69rb" Jul 30 00:17:51.061451 ip-10-0-142-6 kubenswrapper[2564]: I0730 00:17:51.061184 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/853c653e-d075-4220-ae9c-86b76caa2229-run\") pod \"tuned-t69rb\" (UID: \"853c653e-d075-4220-ae9c-86b76caa2229\") " pod="openshift-cluster-node-tuning-operator/tuned-t69rb" Jul 30 00:17:51.061451 ip-10-0-142-6 kubenswrapper[2564]: I0730 00:17:51.061208 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/853c653e-d075-4220-ae9c-86b76caa2229-var-lib-kubelet\") pod \"tuned-t69rb\" (UID: \"853c653e-d075-4220-ae9c-86b76caa2229\") " pod="openshift-cluster-node-tuning-operator/tuned-t69rb" Jul 30 00:17:51.061451 ip-10-0-142-6 kubenswrapper[2564]: I0730 00:17:51.061234 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/6a73d10b-1baa-42ac-a83a-2336014e3600-agent-certs\") pod \"konnectivity-agent-thhcs\" (UID: \"6a73d10b-1baa-42ac-a83a-2336014e3600\") " pod="kube-system/konnectivity-agent-thhcs" Jul 30 00:17:51.061451 ip-10-0-142-6 kubenswrapper[2564]: I0730 00:17:51.061348 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/b1186f7d-220d-4210-a7f8-6f3f0cf554e8-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-c6sdp\" (UID: \"b1186f7d-220d-4210-a7f8-6f3f0cf554e8\") " pod="openshift-multus/multus-additional-cni-plugins-c6sdp" Jul 30 00:17:51.061451 ip-10-0-142-6 kubenswrapper[2564]: I0730 00:17:51.061390 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f82506d3-a3ca-415e-b6c0-4de2f2bf35f5-metrics-certs\") pod \"network-metrics-daemon-5skn2\" (UID: \"f82506d3-a3ca-415e-b6c0-4de2f2bf35f5\") " pod="openshift-multus/network-metrics-daemon-5skn2" Jul 30 00:17:51.062249 ip-10-0-142-6 kubenswrapper[2564]: I0730 00:17:51.061417 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/b1186f7d-220d-4210-a7f8-6f3f0cf554e8-tuning-conf-dir\") pod \"multus-additional-cni-plugins-c6sdp\" (UID: \"b1186f7d-220d-4210-a7f8-6f3f0cf554e8\") " pod="openshift-multus/multus-additional-cni-plugins-c6sdp" Jul 30 00:17:51.062249 ip-10-0-142-6 kubenswrapper[2564]: I0730 00:17:51.061472 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gvd6g\" (UniqueName: \"kubernetes.io/projected/b1186f7d-220d-4210-a7f8-6f3f0cf554e8-kube-api-access-gvd6g\") pod \"multus-additional-cni-plugins-c6sdp\" (UID: \"b1186f7d-220d-4210-a7f8-6f3f0cf554e8\") " pod="openshift-multus/multus-additional-cni-plugins-c6sdp" Jul 30 00:17:51.062249 ip-10-0-142-6 kubenswrapper[2564]: I0730 00:17:51.061501 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9gm94\" (UniqueName: \"kubernetes.io/projected/99deae04-e4b4-4491-a31f-945927bd1e72-kube-api-access-9gm94\") pod \"node-resolver-6ssnm\" (UID: \"99deae04-e4b4-4491-a31f-945927bd1e72\") " pod="openshift-dns/node-resolver-6ssnm" Jul 30 00:17:51.062249 ip-10-0-142-6 kubenswrapper[2564]: I0730 00:17:51.061539 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/81450593-06b9-4f5a-81da-6e5fe67679f1-kubelet-config\") pod \"global-pull-secret-syncer-d6phj\" (UID: \"81450593-06b9-4f5a-81da-6e5fe67679f1\") " pod="kube-system/global-pull-secret-syncer-d6phj" Jul 30 00:17:51.062249 ip-10-0-142-6 kubenswrapper[2564]: I0730 00:17:51.061583 2564 swap_util.go:74] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory" Jul 30 00:17:51.062249 ip-10-0-142-6 kubenswrapper[2564]: I0730 00:17:51.061595 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/d2866704-b7d7-4515-9b9d-4906b1508f68-etc-kubernetes\") pod \"multus-tphm4\" (UID: \"d2866704-b7d7-4515-9b9d-4906b1508f68\") " pod="openshift-multus/multus-tphm4" Jul 30 00:17:51.062249 ip-10-0-142-6 kubenswrapper[2564]: I0730 00:17:51.061622 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-t2mrd\" (UniqueName: \"kubernetes.io/projected/7ff24a6e-78df-4c03-830a-453386386626-kube-api-access-t2mrd\") pod \"node-ca-jc8px\" (UID: \"7ff24a6e-78df-4c03-830a-453386386626\") " pod="openshift-image-registry/node-ca-jc8px" Jul 30 00:17:51.062249 ip-10-0-142-6 kubenswrapper[2564]: I0730 00:17:51.061646 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/b1186f7d-220d-4210-a7f8-6f3f0cf554e8-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-c6sdp\" (UID: \"b1186f7d-220d-4210-a7f8-6f3f0cf554e8\") " pod="openshift-multus/multus-additional-cni-plugins-c6sdp" Jul 30 00:17:51.062249 ip-10-0-142-6 kubenswrapper[2564]: I0730 00:17:51.061667 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/99deae04-e4b4-4491-a31f-945927bd1e72-tmp-dir\") pod \"node-resolver-6ssnm\" (UID: \"99deae04-e4b4-4491-a31f-945927bd1e72\") " pod="openshift-dns/node-resolver-6ssnm" Jul 30 00:17:51.062249 ip-10-0-142-6 kubenswrapper[2564]: I0730 00:17:51.061685 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/d2866704-b7d7-4515-9b9d-4906b1508f68-cnibin\") pod \"multus-tphm4\" (UID: \"d2866704-b7d7-4515-9b9d-4906b1508f68\") " pod="openshift-multus/multus-tphm4" Jul 30 00:17:51.062249 ip-10-0-142-6 kubenswrapper[2564]: I0730 00:17:51.061915 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/d2866704-b7d7-4515-9b9d-4906b1508f68-hostroot\") pod \"multus-tphm4\" (UID: \"d2866704-b7d7-4515-9b9d-4906b1508f68\") " pod="openshift-multus/multus-tphm4" Jul 30 00:17:51.062249 ip-10-0-142-6 kubenswrapper[2564]: I0730 00:17:51.061949 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xvk2p\" (UniqueName: \"kubernetes.io/projected/d2866704-b7d7-4515-9b9d-4906b1508f68-kube-api-access-xvk2p\") pod \"multus-tphm4\" (UID: \"d2866704-b7d7-4515-9b9d-4906b1508f68\") " pod="openshift-multus/multus-tphm4" Jul 30 00:17:51.062249 ip-10-0-142-6 kubenswrapper[2564]: I0730 00:17:51.061976 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/853c653e-d075-4220-ae9c-86b76caa2229-etc-systemd\") pod \"tuned-t69rb\" (UID: \"853c653e-d075-4220-ae9c-86b76caa2229\") " pod="openshift-cluster-node-tuning-operator/tuned-t69rb" Jul 30 00:17:51.062249 ip-10-0-142-6 kubenswrapper[2564]: I0730 00:17:51.061963 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/99deae04-e4b4-4491-a31f-945927bd1e72-tmp-dir\") pod \"node-resolver-6ssnm\" (UID: \"99deae04-e4b4-4491-a31f-945927bd1e72\") " pod="openshift-dns/node-resolver-6ssnm" Jul 30 00:17:51.062249 ip-10-0-142-6 kubenswrapper[2564]: I0730 00:17:51.062214 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/81450593-06b9-4f5a-81da-6e5fe67679f1-kubelet-config\") pod \"global-pull-secret-syncer-d6phj\" (UID: \"81450593-06b9-4f5a-81da-6e5fe67679f1\") " pod="kube-system/global-pull-secret-syncer-d6phj" Jul 30 00:17:51.062249 ip-10-0-142-6 kubenswrapper[2564]: I0730 00:17:51.062202 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/e7215f9f-4c75-4ab5-85c7-e10ab231e916-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-w8grm\" (UID: \"e7215f9f-4c75-4ab5-85c7-e10ab231e916\") " pod="openshift-ovn-kubernetes/ovnkube-node-w8grm" Jul 30 00:17:51.062998 ip-10-0-142-6 kubenswrapper[2564]: I0730 00:17:51.062264 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/b1186f7d-220d-4210-a7f8-6f3f0cf554e8-cnibin\") pod \"multus-additional-cni-plugins-c6sdp\" (UID: \"b1186f7d-220d-4210-a7f8-6f3f0cf554e8\") " pod="openshift-multus/multus-additional-cni-plugins-c6sdp" Jul 30 00:17:51.062998 ip-10-0-142-6 kubenswrapper[2564]: I0730 00:17:51.062285 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/b1186f7d-220d-4210-a7f8-6f3f0cf554e8-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-c6sdp\" (UID: \"b1186f7d-220d-4210-a7f8-6f3f0cf554e8\") " pod="openshift-multus/multus-additional-cni-plugins-c6sdp" Jul 30 00:17:51.062998 ip-10-0-142-6 kubenswrapper[2564]: I0730 00:17:51.062304 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/3a12258d-f063-4ae8-8fec-9eb3b012df06-device-dir\") pod \"aws-ebs-csi-driver-node-6khjg\" (UID: \"3a12258d-f063-4ae8-8fec-9eb3b012df06\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6khjg" Jul 30 00:17:51.062998 ip-10-0-142-6 kubenswrapper[2564]: I0730 00:17:51.062335 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/3a12258d-f063-4ae8-8fec-9eb3b012df06-device-dir\") pod \"aws-ebs-csi-driver-node-6khjg\" (UID: \"3a12258d-f063-4ae8-8fec-9eb3b012df06\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6khjg" Jul 30 00:17:51.062998 ip-10-0-142-6 kubenswrapper[2564]: I0730 00:17:51.062347 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/b1186f7d-220d-4210-a7f8-6f3f0cf554e8-cnibin\") pod \"multus-additional-cni-plugins-c6sdp\" (UID: \"b1186f7d-220d-4210-a7f8-6f3f0cf554e8\") " pod="openshift-multus/multus-additional-cni-plugins-c6sdp" Jul 30 00:17:51.062998 ip-10-0-142-6 kubenswrapper[2564]: I0730 00:17:51.062361 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/b1186f7d-220d-4210-a7f8-6f3f0cf554e8-tuning-conf-dir\") pod \"multus-additional-cni-plugins-c6sdp\" (UID: \"b1186f7d-220d-4210-a7f8-6f3f0cf554e8\") " pod="openshift-multus/multus-additional-cni-plugins-c6sdp" Jul 30 00:17:51.062998 ip-10-0-142-6 kubenswrapper[2564]: I0730 00:17:51.062361 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/e7215f9f-4c75-4ab5-85c7-e10ab231e916-host-kubelet\") pod \"ovnkube-node-w8grm\" (UID: \"e7215f9f-4c75-4ab5-85c7-e10ab231e916\") " pod="openshift-ovn-kubernetes/ovnkube-node-w8grm" Jul 30 00:17:51.062998 ip-10-0-142-6 kubenswrapper[2564]: I0730 00:17:51.062397 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/e7215f9f-4c75-4ab5-85c7-e10ab231e916-systemd-units\") pod \"ovnkube-node-w8grm\" (UID: \"e7215f9f-4c75-4ab5-85c7-e10ab231e916\") " pod="openshift-ovn-kubernetes/ovnkube-node-w8grm" Jul 30 00:17:51.062998 ip-10-0-142-6 kubenswrapper[2564]: I0730 00:17:51.062410 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/b1186f7d-220d-4210-a7f8-6f3f0cf554e8-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-c6sdp\" (UID: \"b1186f7d-220d-4210-a7f8-6f3f0cf554e8\") " pod="openshift-multus/multus-additional-cni-plugins-c6sdp" Jul 30 00:17:51.062998 ip-10-0-142-6 kubenswrapper[2564]: I0730 00:17:51.062432 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/e7215f9f-4c75-4ab5-85c7-e10ab231e916-host-cni-netd\") pod \"ovnkube-node-w8grm\" (UID: \"e7215f9f-4c75-4ab5-85c7-e10ab231e916\") " pod="openshift-ovn-kubernetes/ovnkube-node-w8grm" Jul 30 00:17:51.062998 ip-10-0-142-6 kubenswrapper[2564]: I0730 00:17:51.062467 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/dcc57a98-213c-4ed7-8b59-c0dd84a39b36-iptables-alerter-script\") pod \"iptables-alerter-k9l5m\" (UID: \"dcc57a98-213c-4ed7-8b59-c0dd84a39b36\") " pod="openshift-network-operator/iptables-alerter-k9l5m" Jul 30 00:17:51.062998 ip-10-0-142-6 kubenswrapper[2564]: I0730 00:17:51.062503 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/3a12258d-f063-4ae8-8fec-9eb3b012df06-socket-dir\") pod \"aws-ebs-csi-driver-node-6khjg\" (UID: \"3a12258d-f063-4ae8-8fec-9eb3b012df06\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6khjg" Jul 30 00:17:51.062998 ip-10-0-142-6 kubenswrapper[2564]: I0730 00:17:51.062540 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ndwzs\" (UniqueName: \"kubernetes.io/projected/3a12258d-f063-4ae8-8fec-9eb3b012df06-kube-api-access-ndwzs\") pod \"aws-ebs-csi-driver-node-6khjg\" (UID: \"3a12258d-f063-4ae8-8fec-9eb3b012df06\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6khjg" Jul 30 00:17:51.062998 ip-10-0-142-6 kubenswrapper[2564]: I0730 00:17:51.062569 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/e7215f9f-4c75-4ab5-85c7-e10ab231e916-var-lib-openvswitch\") pod \"ovnkube-node-w8grm\" (UID: \"e7215f9f-4c75-4ab5-85c7-e10ab231e916\") " pod="openshift-ovn-kubernetes/ovnkube-node-w8grm" Jul 30 00:17:51.062998 ip-10-0-142-6 kubenswrapper[2564]: I0730 00:17:51.062589 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2zflx\" (UniqueName: \"kubernetes.io/projected/e7215f9f-4c75-4ab5-85c7-e10ab231e916-kube-api-access-2zflx\") pod \"ovnkube-node-w8grm\" (UID: \"e7215f9f-4c75-4ab5-85c7-e10ab231e916\") " pod="openshift-ovn-kubernetes/ovnkube-node-w8grm" Jul 30 00:17:51.062998 ip-10-0-142-6 kubenswrapper[2564]: I0730 00:17:51.062595 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/3a12258d-f063-4ae8-8fec-9eb3b012df06-socket-dir\") pod \"aws-ebs-csi-driver-node-6khjg\" (UID: \"3a12258d-f063-4ae8-8fec-9eb3b012df06\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6khjg" Jul 30 00:17:51.063791 ip-10-0-142-6 kubenswrapper[2564]: I0730 00:17:51.062679 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/853c653e-d075-4220-ae9c-86b76caa2229-tmp\") pod \"tuned-t69rb\" (UID: \"853c653e-d075-4220-ae9c-86b76caa2229\") " pod="openshift-cluster-node-tuning-operator/tuned-t69rb" Jul 30 00:17:51.063791 ip-10-0-142-6 kubenswrapper[2564]: I0730 00:17:51.062723 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/e7215f9f-4c75-4ab5-85c7-e10ab231e916-etc-openvswitch\") pod \"ovnkube-node-w8grm\" (UID: \"e7215f9f-4c75-4ab5-85c7-e10ab231e916\") " pod="openshift-ovn-kubernetes/ovnkube-node-w8grm" Jul 30 00:17:51.063791 ip-10-0-142-6 kubenswrapper[2564]: I0730 00:17:51.062761 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/e7215f9f-4c75-4ab5-85c7-e10ab231e916-env-overrides\") pod \"ovnkube-node-w8grm\" (UID: \"e7215f9f-4c75-4ab5-85c7-e10ab231e916\") " pod="openshift-ovn-kubernetes/ovnkube-node-w8grm" Jul 30 00:17:51.063791 ip-10-0-142-6 kubenswrapper[2564]: I0730 00:17:51.062784 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/d2866704-b7d7-4515-9b9d-4906b1508f68-host-run-netns\") pod \"multus-tphm4\" (UID: \"d2866704-b7d7-4515-9b9d-4906b1508f68\") " pod="openshift-multus/multus-tphm4" Jul 30 00:17:51.063791 ip-10-0-142-6 kubenswrapper[2564]: I0730 00:17:51.062807 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/d2866704-b7d7-4515-9b9d-4906b1508f68-host-var-lib-cni-bin\") pod \"multus-tphm4\" (UID: \"d2866704-b7d7-4515-9b9d-4906b1508f68\") " pod="openshift-multus/multus-tphm4" Jul 30 00:17:51.063791 ip-10-0-142-6 kubenswrapper[2564]: I0730 00:17:51.062835 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/d2866704-b7d7-4515-9b9d-4906b1508f68-host-run-multus-certs\") pod \"multus-tphm4\" (UID: \"d2866704-b7d7-4515-9b9d-4906b1508f68\") " pod="openshift-multus/multus-tphm4" Jul 30 00:17:51.063791 ip-10-0-142-6 kubenswrapper[2564]: I0730 00:17:51.062863 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/3a12258d-f063-4ae8-8fec-9eb3b012df06-kubelet-dir\") pod \"aws-ebs-csi-driver-node-6khjg\" (UID: \"3a12258d-f063-4ae8-8fec-9eb3b012df06\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6khjg" Jul 30 00:17:51.063791 ip-10-0-142-6 kubenswrapper[2564]: I0730 00:17:51.062888 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/99deae04-e4b4-4491-a31f-945927bd1e72-hosts-file\") pod \"node-resolver-6ssnm\" (UID: \"99deae04-e4b4-4491-a31f-945927bd1e72\") " pod="openshift-dns/node-resolver-6ssnm" Jul 30 00:17:51.063791 ip-10-0-142-6 kubenswrapper[2564]: I0730 00:17:51.062913 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/81450593-06b9-4f5a-81da-6e5fe67679f1-dbus\") pod \"global-pull-secret-syncer-d6phj\" (UID: \"81450593-06b9-4f5a-81da-6e5fe67679f1\") " pod="kube-system/global-pull-secret-syncer-d6phj" Jul 30 00:17:51.063791 ip-10-0-142-6 kubenswrapper[2564]: I0730 00:17:51.062923 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/3a12258d-f063-4ae8-8fec-9eb3b012df06-kubelet-dir\") pod \"aws-ebs-csi-driver-node-6khjg\" (UID: \"3a12258d-f063-4ae8-8fec-9eb3b012df06\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6khjg" Jul 30 00:17:51.063791 ip-10-0-142-6 kubenswrapper[2564]: I0730 00:17:51.062936 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/e7215f9f-4c75-4ab5-85c7-e10ab231e916-run-systemd\") pod \"ovnkube-node-w8grm\" (UID: \"e7215f9f-4c75-4ab5-85c7-e10ab231e916\") " pod="openshift-ovn-kubernetes/ovnkube-node-w8grm" Jul 30 00:17:51.063791 ip-10-0-142-6 kubenswrapper[2564]: I0730 00:17:51.062941 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/99deae04-e4b4-4491-a31f-945927bd1e72-hosts-file\") pod \"node-resolver-6ssnm\" (UID: \"99deae04-e4b4-4491-a31f-945927bd1e72\") " pod="openshift-dns/node-resolver-6ssnm" Jul 30 00:17:51.063791 ip-10-0-142-6 kubenswrapper[2564]: I0730 00:17:51.062964 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/e7215f9f-4c75-4ab5-85c7-e10ab231e916-host-cni-bin\") pod \"ovnkube-node-w8grm\" (UID: \"e7215f9f-4c75-4ab5-85c7-e10ab231e916\") " pod="openshift-ovn-kubernetes/ovnkube-node-w8grm" Jul 30 00:17:51.063791 ip-10-0-142-6 kubenswrapper[2564]: I0730 00:17:51.062986 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/d2866704-b7d7-4515-9b9d-4906b1508f68-cni-binary-copy\") pod \"multus-tphm4\" (UID: \"d2866704-b7d7-4515-9b9d-4906b1508f68\") " pod="openshift-multus/multus-tphm4" Jul 30 00:17:51.063791 ip-10-0-142-6 kubenswrapper[2564]: I0730 00:17:51.063008 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/d2866704-b7d7-4515-9b9d-4906b1508f68-multus-daemon-config\") pod \"multus-tphm4\" (UID: \"d2866704-b7d7-4515-9b9d-4906b1508f68\") " pod="openshift-multus/multus-tphm4" Jul 30 00:17:51.063791 ip-10-0-142-6 kubenswrapper[2564]: I0730 00:17:51.063031 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/81450593-06b9-4f5a-81da-6e5fe67679f1-dbus\") pod \"global-pull-secret-syncer-d6phj\" (UID: \"81450593-06b9-4f5a-81da-6e5fe67679f1\") " pod="kube-system/global-pull-secret-syncer-d6phj" Jul 30 00:17:51.063791 ip-10-0-142-6 kubenswrapper[2564]: I0730 00:17:51.063031 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/e7215f9f-4c75-4ab5-85c7-e10ab231e916-log-socket\") pod \"ovnkube-node-w8grm\" (UID: \"e7215f9f-4c75-4ab5-85c7-e10ab231e916\") " pod="openshift-ovn-kubernetes/ovnkube-node-w8grm" Jul 30 00:17:51.064622 ip-10-0-142-6 kubenswrapper[2564]: I0730 00:17:51.063089 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/7ff24a6e-78df-4c03-830a-453386386626-serviceca\") pod \"node-ca-jc8px\" (UID: \"7ff24a6e-78df-4c03-830a-453386386626\") " pod="openshift-image-registry/node-ca-jc8px" Jul 30 00:17:51.064622 ip-10-0-142-6 kubenswrapper[2564]: I0730 00:17:51.063119 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/853c653e-d075-4220-ae9c-86b76caa2229-etc-sysctl-conf\") pod \"tuned-t69rb\" (UID: \"853c653e-d075-4220-ae9c-86b76caa2229\") " pod="openshift-cluster-node-tuning-operator/tuned-t69rb" Jul 30 00:17:51.064622 ip-10-0-142-6 kubenswrapper[2564]: I0730 00:17:51.063143 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/853c653e-d075-4220-ae9c-86b76caa2229-lib-modules\") pod \"tuned-t69rb\" (UID: \"853c653e-d075-4220-ae9c-86b76caa2229\") " pod="openshift-cluster-node-tuning-operator/tuned-t69rb" Jul 30 00:17:51.066154 ip-10-0-142-6 kubenswrapper[2564]: I0730 00:17:51.066129 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/6a73d10b-1baa-42ac-a83a-2336014e3600-agent-certs\") pod \"konnectivity-agent-thhcs\" (UID: \"6a73d10b-1baa-42ac-a83a-2336014e3600\") " pod="kube-system/konnectivity-agent-thhcs" Jul 30 00:17:51.067723 ip-10-0-142-6 kubenswrapper[2564]: E0730 00:17:51.067600 2564 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 30 00:17:51.067723 ip-10-0-142-6 kubenswrapper[2564]: E0730 00:17:51.067626 2564 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 30 00:17:51.067723 ip-10-0-142-6 kubenswrapper[2564]: E0730 00:17:51.067639 2564 projected.go:196] Error preparing data for projected volume kube-api-access-5v98n for pod openshift-network-diagnostics/network-check-target-j4ckt: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 30 00:17:51.067723 ip-10-0-142-6 kubenswrapper[2564]: E0730 00:17:51.067696 2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/a0bef212-99cc-47c3-b1da-316f26f776b9-kube-api-access-5v98n podName:a0bef212-99cc-47c3-b1da-316f26f776b9 nodeName:}" failed. No retries permitted until 2026-07-30 00:17:51.567677468 +0000 UTC m=+2.982417381 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-api-access-5v98n" (UniqueName: "kubernetes.io/projected/a0bef212-99cc-47c3-b1da-316f26f776b9-kube-api-access-5v98n") pod "network-check-target-j4ckt" (UID: "a0bef212-99cc-47c3-b1da-316f26f776b9") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 30 00:17:51.073001 ip-10-0-142-6 kubenswrapper[2564]: I0730 00:17:51.072890 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ndwzs\" (UniqueName: \"kubernetes.io/projected/3a12258d-f063-4ae8-8fec-9eb3b012df06-kube-api-access-ndwzs\") pod \"aws-ebs-csi-driver-node-6khjg\" (UID: \"3a12258d-f063-4ae8-8fec-9eb3b012df06\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6khjg" Jul 30 00:17:51.075577 ip-10-0-142-6 kubenswrapper[2564]: I0730 00:17:51.075446 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-gvd6g\" (UniqueName: \"kubernetes.io/projected/b1186f7d-220d-4210-a7f8-6f3f0cf554e8-kube-api-access-gvd6g\") pod \"multus-additional-cni-plugins-c6sdp\" (UID: \"b1186f7d-220d-4210-a7f8-6f3f0cf554e8\") " pod="openshift-multus/multus-additional-cni-plugins-c6sdp" Jul 30 00:17:51.075577 ip-10-0-142-6 kubenswrapper[2564]: I0730 00:17:51.075530 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9gm94\" (UniqueName: \"kubernetes.io/projected/99deae04-e4b4-4491-a31f-945927bd1e72-kube-api-access-9gm94\") pod \"node-resolver-6ssnm\" (UID: \"99deae04-e4b4-4491-a31f-945927bd1e72\") " pod="openshift-dns/node-resolver-6ssnm" Jul 30 00:17:51.138783 ip-10-0-142-6 kubenswrapper[2564]: I0730 00:17:51.138723 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-142-6.ec2.internal" event={"ID":"514977950851753a7cc9d5e8c3264eee","Type":"ContainerStarted","Data":"7df5f57222d51a768713c23f1ed86256a534a687ef821c574f0e52e3e12be38d"} Jul 30 00:17:51.139997 ip-10-0-142-6 kubenswrapper[2564]: I0730 00:17:51.139971 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-142-6.ec2.internal" event={"ID":"7c269f27f5372a2f0fc63809270fef7a","Type":"ContainerStarted","Data":"b561ea95d9260d35056b9fdf9036f568b2b57605155c615dfa6a9fd5e629c320"} Jul 30 00:17:51.163694 ip-10-0-142-6 kubenswrapper[2564]: I0730 00:17:51.163657 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/e7215f9f-4c75-4ab5-85c7-e10ab231e916-host-run-ovn-kubernetes\") pod \"ovnkube-node-w8grm\" (UID: \"e7215f9f-4c75-4ab5-85c7-e10ab231e916\") " pod="openshift-ovn-kubernetes/ovnkube-node-w8grm" Jul 30 00:17:51.163694 ip-10-0-142-6 kubenswrapper[2564]: I0730 00:17:51.163695 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/d2866704-b7d7-4515-9b9d-4906b1508f68-multus-cni-dir\") pod \"multus-tphm4\" (UID: \"d2866704-b7d7-4515-9b9d-4906b1508f68\") " pod="openshift-multus/multus-tphm4" Jul 30 00:17:51.163937 ip-10-0-142-6 kubenswrapper[2564]: I0730 00:17:51.163721 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/d2866704-b7d7-4515-9b9d-4906b1508f68-host-run-k8s-cni-cncf-io\") pod \"multus-tphm4\" (UID: \"d2866704-b7d7-4515-9b9d-4906b1508f68\") " pod="openshift-multus/multus-tphm4" Jul 30 00:17:51.163937 ip-10-0-142-6 kubenswrapper[2564]: I0730 00:17:51.163742 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/e7215f9f-4c75-4ab5-85c7-e10ab231e916-host-run-netns\") pod \"ovnkube-node-w8grm\" (UID: \"e7215f9f-4c75-4ab5-85c7-e10ab231e916\") " pod="openshift-ovn-kubernetes/ovnkube-node-w8grm" Jul 30 00:17:51.163937 ip-10-0-142-6 kubenswrapper[2564]: I0730 00:17:51.163761 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/853c653e-d075-4220-ae9c-86b76caa2229-etc-sysconfig\") pod \"tuned-t69rb\" (UID: \"853c653e-d075-4220-ae9c-86b76caa2229\") " pod="openshift-cluster-node-tuning-operator/tuned-t69rb" Jul 30 00:17:51.163937 ip-10-0-142-6 kubenswrapper[2564]: I0730 00:17:51.163774 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/e7215f9f-4c75-4ab5-85c7-e10ab231e916-host-run-ovn-kubernetes\") pod \"ovnkube-node-w8grm\" (UID: \"e7215f9f-4c75-4ab5-85c7-e10ab231e916\") " pod="openshift-ovn-kubernetes/ovnkube-node-w8grm" Jul 30 00:17:51.163937 ip-10-0-142-6 kubenswrapper[2564]: I0730 00:17:51.163805 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/e7215f9f-4c75-4ab5-85c7-e10ab231e916-host-slash\") pod \"ovnkube-node-w8grm\" (UID: \"e7215f9f-4c75-4ab5-85c7-e10ab231e916\") " pod="openshift-ovn-kubernetes/ovnkube-node-w8grm" Jul 30 00:17:51.163937 ip-10-0-142-6 kubenswrapper[2564]: I0730 00:17:51.163820 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/d2866704-b7d7-4515-9b9d-4906b1508f68-host-run-k8s-cni-cncf-io\") pod \"multus-tphm4\" (UID: \"d2866704-b7d7-4515-9b9d-4906b1508f68\") " pod="openshift-multus/multus-tphm4" Jul 30 00:17:51.163937 ip-10-0-142-6 kubenswrapper[2564]: I0730 00:17:51.163847 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/e7215f9f-4c75-4ab5-85c7-e10ab231e916-host-slash\") pod \"ovnkube-node-w8grm\" (UID: \"e7215f9f-4c75-4ab5-85c7-e10ab231e916\") " pod="openshift-ovn-kubernetes/ovnkube-node-w8grm" Jul 30 00:17:51.163937 ip-10-0-142-6 kubenswrapper[2564]: I0730 00:17:51.163847 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/e7215f9f-4c75-4ab5-85c7-e10ab231e916-run-ovn\") pod \"ovnkube-node-w8grm\" (UID: \"e7215f9f-4c75-4ab5-85c7-e10ab231e916\") " pod="openshift-ovn-kubernetes/ovnkube-node-w8grm" Jul 30 00:17:51.163937 ip-10-0-142-6 kubenswrapper[2564]: I0730 00:17:51.163888 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/853c653e-d075-4220-ae9c-86b76caa2229-etc-sysctl-d\") pod \"tuned-t69rb\" (UID: \"853c653e-d075-4220-ae9c-86b76caa2229\") " pod="openshift-cluster-node-tuning-operator/tuned-t69rb" Jul 30 00:17:51.163937 ip-10-0-142-6 kubenswrapper[2564]: I0730 00:17:51.163902 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/d2866704-b7d7-4515-9b9d-4906b1508f68-multus-cni-dir\") pod \"multus-tphm4\" (UID: \"d2866704-b7d7-4515-9b9d-4906b1508f68\") " pod="openshift-multus/multus-tphm4" Jul 30 00:17:51.163937 ip-10-0-142-6 kubenswrapper[2564]: I0730 00:17:51.163910 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/d2866704-b7d7-4515-9b9d-4906b1508f68-multus-socket-dir-parent\") pod \"multus-tphm4\" (UID: \"d2866704-b7d7-4515-9b9d-4906b1508f68\") " pod="openshift-multus/multus-tphm4" Jul 30 00:17:51.163937 ip-10-0-142-6 kubenswrapper[2564]: I0730 00:17:51.163936 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/853c653e-d075-4220-ae9c-86b76caa2229-etc-sysconfig\") pod \"tuned-t69rb\" (UID: \"853c653e-d075-4220-ae9c-86b76caa2229\") " pod="openshift-cluster-node-tuning-operator/tuned-t69rb" Jul 30 00:17:51.164401 ip-10-0-142-6 kubenswrapper[2564]: I0730 00:17:51.163945 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/853c653e-d075-4220-ae9c-86b76caa2229-host\") pod \"tuned-t69rb\" (UID: \"853c653e-d075-4220-ae9c-86b76caa2229\") " pod="openshift-cluster-node-tuning-operator/tuned-t69rb" Jul 30 00:17:51.164401 ip-10-0-142-6 kubenswrapper[2564]: I0730 00:17:51.163953 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/d2866704-b7d7-4515-9b9d-4906b1508f68-multus-socket-dir-parent\") pod \"multus-tphm4\" (UID: \"d2866704-b7d7-4515-9b9d-4906b1508f68\") " pod="openshift-multus/multus-tphm4" Jul 30 00:17:51.164401 ip-10-0-142-6 kubenswrapper[2564]: I0730 00:17:51.163890 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/e7215f9f-4c75-4ab5-85c7-e10ab231e916-host-run-netns\") pod \"ovnkube-node-w8grm\" (UID: \"e7215f9f-4c75-4ab5-85c7-e10ab231e916\") " pod="openshift-ovn-kubernetes/ovnkube-node-w8grm" Jul 30 00:17:51.164401 ip-10-0-142-6 kubenswrapper[2564]: I0730 00:17:51.163970 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/853c653e-d075-4220-ae9c-86b76caa2229-etc-tuned\") pod \"tuned-t69rb\" (UID: \"853c653e-d075-4220-ae9c-86b76caa2229\") " pod="openshift-cluster-node-tuning-operator/tuned-t69rb" Jul 30 00:17:51.164401 ip-10-0-142-6 kubenswrapper[2564]: I0730 00:17:51.164022 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5drxq\" (UniqueName: \"kubernetes.io/projected/853c653e-d075-4220-ae9c-86b76caa2229-kube-api-access-5drxq\") pod \"tuned-t69rb\" (UID: \"853c653e-d075-4220-ae9c-86b76caa2229\") " pod="openshift-cluster-node-tuning-operator/tuned-t69rb" Jul 30 00:17:51.164401 ip-10-0-142-6 kubenswrapper[2564]: I0730 00:17:51.164057 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/853c653e-d075-4220-ae9c-86b76caa2229-host\") pod \"tuned-t69rb\" (UID: \"853c653e-d075-4220-ae9c-86b76caa2229\") " pod="openshift-cluster-node-tuning-operator/tuned-t69rb" Jul 30 00:17:51.164401 ip-10-0-142-6 kubenswrapper[2564]: I0730 00:17:51.164065 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/853c653e-d075-4220-ae9c-86b76caa2229-etc-sysctl-d\") pod \"tuned-t69rb\" (UID: \"853c653e-d075-4220-ae9c-86b76caa2229\") " pod="openshift-cluster-node-tuning-operator/tuned-t69rb" Jul 30 00:17:51.164401 ip-10-0-142-6 kubenswrapper[2564]: I0730 00:17:51.164063 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/e7215f9f-4c75-4ab5-85c7-e10ab231e916-run-openvswitch\") pod \"ovnkube-node-w8grm\" (UID: \"e7215f9f-4c75-4ab5-85c7-e10ab231e916\") " pod="openshift-ovn-kubernetes/ovnkube-node-w8grm" Jul 30 00:17:51.164401 ip-10-0-142-6 kubenswrapper[2564]: I0730 00:17:51.164121 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/d2866704-b7d7-4515-9b9d-4906b1508f68-host-var-lib-kubelet\") pod \"multus-tphm4\" (UID: \"d2866704-b7d7-4515-9b9d-4906b1508f68\") " pod="openshift-multus/multus-tphm4" Jul 30 00:17:51.164401 ip-10-0-142-6 kubenswrapper[2564]: I0730 00:17:51.164148 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/dcc57a98-213c-4ed7-8b59-c0dd84a39b36-host-slash\") pod \"iptables-alerter-k9l5m\" (UID: \"dcc57a98-213c-4ed7-8b59-c0dd84a39b36\") " pod="openshift-network-operator/iptables-alerter-k9l5m" Jul 30 00:17:51.164401 ip-10-0-142-6 kubenswrapper[2564]: I0730 00:17:51.164173 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5bftn\" (UniqueName: \"kubernetes.io/projected/dcc57a98-213c-4ed7-8b59-c0dd84a39b36-kube-api-access-5bftn\") pod \"iptables-alerter-k9l5m\" (UID: \"dcc57a98-213c-4ed7-8b59-c0dd84a39b36\") " pod="openshift-network-operator/iptables-alerter-k9l5m" Jul 30 00:17:51.164401 ip-10-0-142-6 kubenswrapper[2564]: I0730 00:17:51.164196 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/853c653e-d075-4220-ae9c-86b76caa2229-sys\") pod \"tuned-t69rb\" (UID: \"853c653e-d075-4220-ae9c-86b76caa2229\") " pod="openshift-cluster-node-tuning-operator/tuned-t69rb" Jul 30 00:17:51.164401 ip-10-0-142-6 kubenswrapper[2564]: I0730 00:17:51.164219 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/e7215f9f-4c75-4ab5-85c7-e10ab231e916-ovnkube-config\") pod \"ovnkube-node-w8grm\" (UID: \"e7215f9f-4c75-4ab5-85c7-e10ab231e916\") " pod="openshift-ovn-kubernetes/ovnkube-node-w8grm" Jul 30 00:17:51.164401 ip-10-0-142-6 kubenswrapper[2564]: I0730 00:17:51.164242 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/e7215f9f-4c75-4ab5-85c7-e10ab231e916-ovnkube-script-lib\") pod \"ovnkube-node-w8grm\" (UID: \"e7215f9f-4c75-4ab5-85c7-e10ab231e916\") " pod="openshift-ovn-kubernetes/ovnkube-node-w8grm" Jul 30 00:17:51.164401 ip-10-0-142-6 kubenswrapper[2564]: I0730 00:17:51.164266 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6v9nv\" (UniqueName: \"kubernetes.io/projected/f82506d3-a3ca-415e-b6c0-4de2f2bf35f5-kube-api-access-6v9nv\") pod \"network-metrics-daemon-5skn2\" (UID: \"f82506d3-a3ca-415e-b6c0-4de2f2bf35f5\") " pod="openshift-multus/network-metrics-daemon-5skn2" Jul 30 00:17:51.164401 ip-10-0-142-6 kubenswrapper[2564]: I0730 00:17:51.164290 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/853c653e-d075-4220-ae9c-86b76caa2229-etc-modprobe-d\") pod \"tuned-t69rb\" (UID: \"853c653e-d075-4220-ae9c-86b76caa2229\") " pod="openshift-cluster-node-tuning-operator/tuned-t69rb" Jul 30 00:17:51.164401 ip-10-0-142-6 kubenswrapper[2564]: I0730 00:17:51.164311 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/853c653e-d075-4220-ae9c-86b76caa2229-run\") pod \"tuned-t69rb\" (UID: \"853c653e-d075-4220-ae9c-86b76caa2229\") " pod="openshift-cluster-node-tuning-operator/tuned-t69rb" Jul 30 00:17:51.165174 ip-10-0-142-6 kubenswrapper[2564]: I0730 00:17:51.164334 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/853c653e-d075-4220-ae9c-86b76caa2229-var-lib-kubelet\") pod \"tuned-t69rb\" (UID: \"853c653e-d075-4220-ae9c-86b76caa2229\") " pod="openshift-cluster-node-tuning-operator/tuned-t69rb" Jul 30 00:17:51.165174 ip-10-0-142-6 kubenswrapper[2564]: I0730 00:17:51.164398 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/e7215f9f-4c75-4ab5-85c7-e10ab231e916-run-openvswitch\") pod \"ovnkube-node-w8grm\" (UID: \"e7215f9f-4c75-4ab5-85c7-e10ab231e916\") " pod="openshift-ovn-kubernetes/ovnkube-node-w8grm" Jul 30 00:17:51.165174 ip-10-0-142-6 kubenswrapper[2564]: I0730 00:17:51.164438 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/e7215f9f-4c75-4ab5-85c7-e10ab231e916-run-ovn\") pod \"ovnkube-node-w8grm\" (UID: \"e7215f9f-4c75-4ab5-85c7-e10ab231e916\") " pod="openshift-ovn-kubernetes/ovnkube-node-w8grm" Jul 30 00:17:51.165174 ip-10-0-142-6 kubenswrapper[2564]: I0730 00:17:51.164441 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/853c653e-d075-4220-ae9c-86b76caa2229-sys\") pod \"tuned-t69rb\" (UID: \"853c653e-d075-4220-ae9c-86b76caa2229\") " pod="openshift-cluster-node-tuning-operator/tuned-t69rb" Jul 30 00:17:51.165174 ip-10-0-142-6 kubenswrapper[2564]: I0730 00:17:51.164497 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/d2866704-b7d7-4515-9b9d-4906b1508f68-host-var-lib-kubelet\") pod \"multus-tphm4\" (UID: \"d2866704-b7d7-4515-9b9d-4906b1508f68\") " pod="openshift-multus/multus-tphm4" Jul 30 00:17:51.165174 ip-10-0-142-6 kubenswrapper[2564]: I0730 00:17:51.164503 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/853c653e-d075-4220-ae9c-86b76caa2229-var-lib-kubelet\") pod \"tuned-t69rb\" (UID: \"853c653e-d075-4220-ae9c-86b76caa2229\") " pod="openshift-cluster-node-tuning-operator/tuned-t69rb" Jul 30 00:17:51.165174 ip-10-0-142-6 kubenswrapper[2564]: I0730 00:17:51.164506 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/dcc57a98-213c-4ed7-8b59-c0dd84a39b36-host-slash\") pod \"iptables-alerter-k9l5m\" (UID: \"dcc57a98-213c-4ed7-8b59-c0dd84a39b36\") " pod="openshift-network-operator/iptables-alerter-k9l5m" Jul 30 00:17:51.165174 ip-10-0-142-6 kubenswrapper[2564]: I0730 00:17:51.164543 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f82506d3-a3ca-415e-b6c0-4de2f2bf35f5-metrics-certs\") pod \"network-metrics-daemon-5skn2\" (UID: \"f82506d3-a3ca-415e-b6c0-4de2f2bf35f5\") " pod="openshift-multus/network-metrics-daemon-5skn2" Jul 30 00:17:51.165174 ip-10-0-142-6 kubenswrapper[2564]: I0730 00:17:51.164554 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/853c653e-d075-4220-ae9c-86b76caa2229-run\") pod \"tuned-t69rb\" (UID: \"853c653e-d075-4220-ae9c-86b76caa2229\") " pod="openshift-cluster-node-tuning-operator/tuned-t69rb" Jul 30 00:17:51.165174 ip-10-0-142-6 kubenswrapper[2564]: I0730 00:17:51.164584 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/d2866704-b7d7-4515-9b9d-4906b1508f68-etc-kubernetes\") pod \"multus-tphm4\" (UID: \"d2866704-b7d7-4515-9b9d-4906b1508f68\") " pod="openshift-multus/multus-tphm4" Jul 30 00:17:51.165174 ip-10-0-142-6 kubenswrapper[2564]: I0730 00:17:51.164601 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/853c653e-d075-4220-ae9c-86b76caa2229-etc-modprobe-d\") pod \"tuned-t69rb\" (UID: \"853c653e-d075-4220-ae9c-86b76caa2229\") " pod="openshift-cluster-node-tuning-operator/tuned-t69rb" Jul 30 00:17:51.165174 ip-10-0-142-6 kubenswrapper[2564]: I0730 00:17:51.164611 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t2mrd\" (UniqueName: \"kubernetes.io/projected/7ff24a6e-78df-4c03-830a-453386386626-kube-api-access-t2mrd\") pod \"node-ca-jc8px\" (UID: \"7ff24a6e-78df-4c03-830a-453386386626\") " pod="openshift-image-registry/node-ca-jc8px" Jul 30 00:17:51.165174 ip-10-0-142-6 kubenswrapper[2564]: E0730 00:17:51.164621 2564 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 30 00:17:51.165174 ip-10-0-142-6 kubenswrapper[2564]: I0730 00:17:51.164638 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/d2866704-b7d7-4515-9b9d-4906b1508f68-cnibin\") pod \"multus-tphm4\" (UID: \"d2866704-b7d7-4515-9b9d-4906b1508f68\") " pod="openshift-multus/multus-tphm4" Jul 30 00:17:51.165174 ip-10-0-142-6 kubenswrapper[2564]: I0730 00:17:51.164641 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/d2866704-b7d7-4515-9b9d-4906b1508f68-etc-kubernetes\") pod \"multus-tphm4\" (UID: \"d2866704-b7d7-4515-9b9d-4906b1508f68\") " pod="openshift-multus/multus-tphm4" Jul 30 00:17:51.165174 ip-10-0-142-6 kubenswrapper[2564]: E0730 00:17:51.164676 2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f82506d3-a3ca-415e-b6c0-4de2f2bf35f5-metrics-certs podName:f82506d3-a3ca-415e-b6c0-4de2f2bf35f5 nodeName:}" failed. No retries permitted until 2026-07-30 00:17:51.664657348 +0000 UTC m=+3.079397278 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/f82506d3-a3ca-415e-b6c0-4de2f2bf35f5-metrics-certs") pod "network-metrics-daemon-5skn2" (UID: "f82506d3-a3ca-415e-b6c0-4de2f2bf35f5") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 30 00:17:51.165174 ip-10-0-142-6 kubenswrapper[2564]: I0730 00:17:51.164686 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/d2866704-b7d7-4515-9b9d-4906b1508f68-cnibin\") pod \"multus-tphm4\" (UID: \"d2866704-b7d7-4515-9b9d-4906b1508f68\") " pod="openshift-multus/multus-tphm4" Jul 30 00:17:51.165174 ip-10-0-142-6 kubenswrapper[2564]: I0730 00:17:51.164779 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/d2866704-b7d7-4515-9b9d-4906b1508f68-hostroot\") pod \"multus-tphm4\" (UID: \"d2866704-b7d7-4515-9b9d-4906b1508f68\") " pod="openshift-multus/multus-tphm4" Jul 30 00:17:51.166010 ip-10-0-142-6 kubenswrapper[2564]: I0730 00:17:51.164819 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xvk2p\" (UniqueName: \"kubernetes.io/projected/d2866704-b7d7-4515-9b9d-4906b1508f68-kube-api-access-xvk2p\") pod \"multus-tphm4\" (UID: \"d2866704-b7d7-4515-9b9d-4906b1508f68\") " pod="openshift-multus/multus-tphm4" Jul 30 00:17:51.166010 ip-10-0-142-6 kubenswrapper[2564]: I0730 00:17:51.164846 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/853c653e-d075-4220-ae9c-86b76caa2229-etc-systemd\") pod \"tuned-t69rb\" (UID: \"853c653e-d075-4220-ae9c-86b76caa2229\") " pod="openshift-cluster-node-tuning-operator/tuned-t69rb" Jul 30 00:17:51.166010 ip-10-0-142-6 kubenswrapper[2564]: I0730 00:17:51.164885 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/d2866704-b7d7-4515-9b9d-4906b1508f68-hostroot\") pod \"multus-tphm4\" (UID: \"d2866704-b7d7-4515-9b9d-4906b1508f68\") " pod="openshift-multus/multus-tphm4" Jul 30 00:17:51.166010 ip-10-0-142-6 kubenswrapper[2564]: I0730 00:17:51.164891 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/e7215f9f-4c75-4ab5-85c7-e10ab231e916-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-w8grm\" (UID: \"e7215f9f-4c75-4ab5-85c7-e10ab231e916\") " pod="openshift-ovn-kubernetes/ovnkube-node-w8grm" Jul 30 00:17:51.166010 ip-10-0-142-6 kubenswrapper[2564]: I0730 00:17:51.164935 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/e7215f9f-4c75-4ab5-85c7-e10ab231e916-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-w8grm\" (UID: \"e7215f9f-4c75-4ab5-85c7-e10ab231e916\") " pod="openshift-ovn-kubernetes/ovnkube-node-w8grm" Jul 30 00:17:51.166010 ip-10-0-142-6 kubenswrapper[2564]: I0730 00:17:51.164948 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/e7215f9f-4c75-4ab5-85c7-e10ab231e916-host-kubelet\") pod \"ovnkube-node-w8grm\" (UID: \"e7215f9f-4c75-4ab5-85c7-e10ab231e916\") " pod="openshift-ovn-kubernetes/ovnkube-node-w8grm" Jul 30 00:17:51.166010 ip-10-0-142-6 kubenswrapper[2564]: I0730 00:17:51.164964 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/e7215f9f-4c75-4ab5-85c7-e10ab231e916-ovnkube-script-lib\") pod \"ovnkube-node-w8grm\" (UID: \"e7215f9f-4c75-4ab5-85c7-e10ab231e916\") " pod="openshift-ovn-kubernetes/ovnkube-node-w8grm" Jul 30 00:17:51.166010 ip-10-0-142-6 kubenswrapper[2564]: I0730 00:17:51.164975 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/e7215f9f-4c75-4ab5-85c7-e10ab231e916-systemd-units\") pod \"ovnkube-node-w8grm\" (UID: \"e7215f9f-4c75-4ab5-85c7-e10ab231e916\") " pod="openshift-ovn-kubernetes/ovnkube-node-w8grm" Jul 30 00:17:51.166010 ip-10-0-142-6 kubenswrapper[2564]: I0730 00:17:51.164997 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/e7215f9f-4c75-4ab5-85c7-e10ab231e916-host-cni-netd\") pod \"ovnkube-node-w8grm\" (UID: \"e7215f9f-4c75-4ab5-85c7-e10ab231e916\") " pod="openshift-ovn-kubernetes/ovnkube-node-w8grm" Jul 30 00:17:51.166010 ip-10-0-142-6 kubenswrapper[2564]: I0730 00:17:51.165031 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/dcc57a98-213c-4ed7-8b59-c0dd84a39b36-iptables-alerter-script\") pod \"iptables-alerter-k9l5m\" (UID: \"dcc57a98-213c-4ed7-8b59-c0dd84a39b36\") " pod="openshift-network-operator/iptables-alerter-k9l5m" Jul 30 00:17:51.166010 ip-10-0-142-6 kubenswrapper[2564]: I0730 00:17:51.165036 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/e7215f9f-4c75-4ab5-85c7-e10ab231e916-host-kubelet\") pod \"ovnkube-node-w8grm\" (UID: \"e7215f9f-4c75-4ab5-85c7-e10ab231e916\") " pod="openshift-ovn-kubernetes/ovnkube-node-w8grm" Jul 30 00:17:51.166010 ip-10-0-142-6 kubenswrapper[2564]: I0730 00:17:51.165059 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/e7215f9f-4c75-4ab5-85c7-e10ab231e916-var-lib-openvswitch\") pod \"ovnkube-node-w8grm\" (UID: \"e7215f9f-4c75-4ab5-85c7-e10ab231e916\") " pod="openshift-ovn-kubernetes/ovnkube-node-w8grm" Jul 30 00:17:51.166010 ip-10-0-142-6 kubenswrapper[2564]: I0730 00:17:51.165095 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/e7215f9f-4c75-4ab5-85c7-e10ab231e916-host-cni-netd\") pod \"ovnkube-node-w8grm\" (UID: \"e7215f9f-4c75-4ab5-85c7-e10ab231e916\") " pod="openshift-ovn-kubernetes/ovnkube-node-w8grm" Jul 30 00:17:51.166010 ip-10-0-142-6 kubenswrapper[2564]: I0730 00:17:51.165099 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2zflx\" (UniqueName: \"kubernetes.io/projected/e7215f9f-4c75-4ab5-85c7-e10ab231e916-kube-api-access-2zflx\") pod \"ovnkube-node-w8grm\" (UID: \"e7215f9f-4c75-4ab5-85c7-e10ab231e916\") " pod="openshift-ovn-kubernetes/ovnkube-node-w8grm" Jul 30 00:17:51.166010 ip-10-0-142-6 kubenswrapper[2564]: I0730 00:17:51.165121 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/853c653e-d075-4220-ae9c-86b76caa2229-etc-systemd\") pod \"tuned-t69rb\" (UID: \"853c653e-d075-4220-ae9c-86b76caa2229\") " pod="openshift-cluster-node-tuning-operator/tuned-t69rb" Jul 30 00:17:51.166010 ip-10-0-142-6 kubenswrapper[2564]: I0730 00:17:51.165133 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/e7215f9f-4c75-4ab5-85c7-e10ab231e916-systemd-units\") pod \"ovnkube-node-w8grm\" (UID: \"e7215f9f-4c75-4ab5-85c7-e10ab231e916\") " pod="openshift-ovn-kubernetes/ovnkube-node-w8grm" Jul 30 00:17:51.166010 ip-10-0-142-6 kubenswrapper[2564]: I0730 00:17:51.165164 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/853c653e-d075-4220-ae9c-86b76caa2229-tmp\") pod \"tuned-t69rb\" (UID: \"853c653e-d075-4220-ae9c-86b76caa2229\") " pod="openshift-cluster-node-tuning-operator/tuned-t69rb" Jul 30 00:17:51.166703 ip-10-0-142-6 kubenswrapper[2564]: I0730 00:17:51.165189 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/e7215f9f-4c75-4ab5-85c7-e10ab231e916-etc-openvswitch\") pod \"ovnkube-node-w8grm\" (UID: \"e7215f9f-4c75-4ab5-85c7-e10ab231e916\") " pod="openshift-ovn-kubernetes/ovnkube-node-w8grm" Jul 30 00:17:51.166703 ip-10-0-142-6 kubenswrapper[2564]: I0730 00:17:51.165224 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/e7215f9f-4c75-4ab5-85c7-e10ab231e916-env-overrides\") pod \"ovnkube-node-w8grm\" (UID: \"e7215f9f-4c75-4ab5-85c7-e10ab231e916\") " pod="openshift-ovn-kubernetes/ovnkube-node-w8grm" Jul 30 00:17:51.166703 ip-10-0-142-6 kubenswrapper[2564]: I0730 00:17:51.165247 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/d2866704-b7d7-4515-9b9d-4906b1508f68-host-run-netns\") pod \"multus-tphm4\" (UID: \"d2866704-b7d7-4515-9b9d-4906b1508f68\") " pod="openshift-multus/multus-tphm4" Jul 30 00:17:51.166703 ip-10-0-142-6 kubenswrapper[2564]: I0730 00:17:51.165264 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/d2866704-b7d7-4515-9b9d-4906b1508f68-host-var-lib-cni-bin\") pod \"multus-tphm4\" (UID: \"d2866704-b7d7-4515-9b9d-4906b1508f68\") " pod="openshift-multus/multus-tphm4" Jul 30 00:17:51.166703 ip-10-0-142-6 kubenswrapper[2564]: I0730 00:17:51.165281 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/d2866704-b7d7-4515-9b9d-4906b1508f68-host-run-multus-certs\") pod \"multus-tphm4\" (UID: \"d2866704-b7d7-4515-9b9d-4906b1508f68\") " pod="openshift-multus/multus-tphm4" Jul 30 00:17:51.166703 ip-10-0-142-6 kubenswrapper[2564]: I0730 00:17:51.165274 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/e7215f9f-4c75-4ab5-85c7-e10ab231e916-ovnkube-config\") pod \"ovnkube-node-w8grm\" (UID: \"e7215f9f-4c75-4ab5-85c7-e10ab231e916\") " pod="openshift-ovn-kubernetes/ovnkube-node-w8grm" Jul 30 00:17:51.166703 ip-10-0-142-6 kubenswrapper[2564]: I0730 00:17:51.165299 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/e7215f9f-4c75-4ab5-85c7-e10ab231e916-run-systemd\") pod \"ovnkube-node-w8grm\" (UID: \"e7215f9f-4c75-4ab5-85c7-e10ab231e916\") " pod="openshift-ovn-kubernetes/ovnkube-node-w8grm" Jul 30 00:17:51.166703 ip-10-0-142-6 kubenswrapper[2564]: I0730 00:17:51.165316 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/e7215f9f-4c75-4ab5-85c7-e10ab231e916-host-cni-bin\") pod \"ovnkube-node-w8grm\" (UID: \"e7215f9f-4c75-4ab5-85c7-e10ab231e916\") " pod="openshift-ovn-kubernetes/ovnkube-node-w8grm" Jul 30 00:17:51.166703 ip-10-0-142-6 kubenswrapper[2564]: I0730 00:17:51.165330 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/d2866704-b7d7-4515-9b9d-4906b1508f68-cni-binary-copy\") pod \"multus-tphm4\" (UID: \"d2866704-b7d7-4515-9b9d-4906b1508f68\") " pod="openshift-multus/multus-tphm4" Jul 30 00:17:51.166703 ip-10-0-142-6 kubenswrapper[2564]: I0730 00:17:51.165343 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/e7215f9f-4c75-4ab5-85c7-e10ab231e916-var-lib-openvswitch\") pod \"ovnkube-node-w8grm\" (UID: \"e7215f9f-4c75-4ab5-85c7-e10ab231e916\") " pod="openshift-ovn-kubernetes/ovnkube-node-w8grm" Jul 30 00:17:51.166703 ip-10-0-142-6 kubenswrapper[2564]: I0730 00:17:51.165347 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/d2866704-b7d7-4515-9b9d-4906b1508f68-host-var-lib-cni-bin\") pod \"multus-tphm4\" (UID: \"d2866704-b7d7-4515-9b9d-4906b1508f68\") " pod="openshift-multus/multus-tphm4" Jul 30 00:17:51.166703 ip-10-0-142-6 kubenswrapper[2564]: I0730 00:17:51.165362 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/d2866704-b7d7-4515-9b9d-4906b1508f68-multus-daemon-config\") pod \"multus-tphm4\" (UID: \"d2866704-b7d7-4515-9b9d-4906b1508f68\") " pod="openshift-multus/multus-tphm4" Jul 30 00:17:51.166703 ip-10-0-142-6 kubenswrapper[2564]: I0730 00:17:51.165379 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/e7215f9f-4c75-4ab5-85c7-e10ab231e916-etc-openvswitch\") pod \"ovnkube-node-w8grm\" (UID: \"e7215f9f-4c75-4ab5-85c7-e10ab231e916\") " pod="openshift-ovn-kubernetes/ovnkube-node-w8grm" Jul 30 00:17:51.166703 ip-10-0-142-6 kubenswrapper[2564]: I0730 00:17:51.165380 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/e7215f9f-4c75-4ab5-85c7-e10ab231e916-log-socket\") pod \"ovnkube-node-w8grm\" (UID: \"e7215f9f-4c75-4ab5-85c7-e10ab231e916\") " pod="openshift-ovn-kubernetes/ovnkube-node-w8grm" Jul 30 00:17:51.166703 ip-10-0-142-6 kubenswrapper[2564]: I0730 00:17:51.165392 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/d2866704-b7d7-4515-9b9d-4906b1508f68-host-run-netns\") pod \"multus-tphm4\" (UID: \"d2866704-b7d7-4515-9b9d-4906b1508f68\") " pod="openshift-multus/multus-tphm4" Jul 30 00:17:51.166703 ip-10-0-142-6 kubenswrapper[2564]: I0730 00:17:51.165399 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/7ff24a6e-78df-4c03-830a-453386386626-serviceca\") pod \"node-ca-jc8px\" (UID: \"7ff24a6e-78df-4c03-830a-453386386626\") " pod="openshift-image-registry/node-ca-jc8px" Jul 30 00:17:51.166703 ip-10-0-142-6 kubenswrapper[2564]: I0730 00:17:51.165429 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/853c653e-d075-4220-ae9c-86b76caa2229-etc-sysctl-conf\") pod \"tuned-t69rb\" (UID: \"853c653e-d075-4220-ae9c-86b76caa2229\") " pod="openshift-cluster-node-tuning-operator/tuned-t69rb" Jul 30 00:17:51.166703 ip-10-0-142-6 kubenswrapper[2564]: I0730 00:17:51.165440 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/d2866704-b7d7-4515-9b9d-4906b1508f68-host-run-multus-certs\") pod \"multus-tphm4\" (UID: \"d2866704-b7d7-4515-9b9d-4906b1508f68\") " pod="openshift-multus/multus-tphm4" Jul 30 00:17:51.167280 ip-10-0-142-6 kubenswrapper[2564]: I0730 00:17:51.165460 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/853c653e-d075-4220-ae9c-86b76caa2229-lib-modules\") pod \"tuned-t69rb\" (UID: \"853c653e-d075-4220-ae9c-86b76caa2229\") " pod="openshift-cluster-node-tuning-operator/tuned-t69rb" Jul 30 00:17:51.167280 ip-10-0-142-6 kubenswrapper[2564]: I0730 00:17:51.165474 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/e7215f9f-4c75-4ab5-85c7-e10ab231e916-node-log\") pod \"ovnkube-node-w8grm\" (UID: \"e7215f9f-4c75-4ab5-85c7-e10ab231e916\") " pod="openshift-ovn-kubernetes/ovnkube-node-w8grm" Jul 30 00:17:51.167280 ip-10-0-142-6 kubenswrapper[2564]: I0730 00:17:51.165489 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/e7215f9f-4c75-4ab5-85c7-e10ab231e916-ovn-node-metrics-cert\") pod \"ovnkube-node-w8grm\" (UID: \"e7215f9f-4c75-4ab5-85c7-e10ab231e916\") " pod="openshift-ovn-kubernetes/ovnkube-node-w8grm" Jul 30 00:17:51.167280 ip-10-0-142-6 kubenswrapper[2564]: I0730 00:17:51.165504 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/d2866704-b7d7-4515-9b9d-4906b1508f68-system-cni-dir\") pod \"multus-tphm4\" (UID: \"d2866704-b7d7-4515-9b9d-4906b1508f68\") " pod="openshift-multus/multus-tphm4" Jul 30 00:17:51.167280 ip-10-0-142-6 kubenswrapper[2564]: I0730 00:17:51.165533 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/d2866704-b7d7-4515-9b9d-4906b1508f68-os-release\") pod \"multus-tphm4\" (UID: \"d2866704-b7d7-4515-9b9d-4906b1508f68\") " pod="openshift-multus/multus-tphm4" Jul 30 00:17:51.167280 ip-10-0-142-6 kubenswrapper[2564]: I0730 00:17:51.165546 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/853c653e-d075-4220-ae9c-86b76caa2229-etc-sysctl-conf\") pod \"tuned-t69rb\" (UID: \"853c653e-d075-4220-ae9c-86b76caa2229\") " pod="openshift-cluster-node-tuning-operator/tuned-t69rb" Jul 30 00:17:51.167280 ip-10-0-142-6 kubenswrapper[2564]: I0730 00:17:51.165557 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/d2866704-b7d7-4515-9b9d-4906b1508f68-host-var-lib-cni-multus\") pod \"multus-tphm4\" (UID: \"d2866704-b7d7-4515-9b9d-4906b1508f68\") " pod="openshift-multus/multus-tphm4" Jul 30 00:17:51.167280 ip-10-0-142-6 kubenswrapper[2564]: I0730 00:17:51.165589 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/d2866704-b7d7-4515-9b9d-4906b1508f68-multus-conf-dir\") pod \"multus-tphm4\" (UID: \"d2866704-b7d7-4515-9b9d-4906b1508f68\") " pod="openshift-multus/multus-tphm4" Jul 30 00:17:51.167280 ip-10-0-142-6 kubenswrapper[2564]: I0730 00:17:51.165621 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/e7215f9f-4c75-4ab5-85c7-e10ab231e916-log-socket\") pod \"ovnkube-node-w8grm\" (UID: \"e7215f9f-4c75-4ab5-85c7-e10ab231e916\") " pod="openshift-ovn-kubernetes/ovnkube-node-w8grm" Jul 30 00:17:51.167280 ip-10-0-142-6 kubenswrapper[2564]: I0730 00:17:51.165611 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/7ff24a6e-78df-4c03-830a-453386386626-host\") pod \"node-ca-jc8px\" (UID: \"7ff24a6e-78df-4c03-830a-453386386626\") " pod="openshift-image-registry/node-ca-jc8px" Jul 30 00:17:51.167280 ip-10-0-142-6 kubenswrapper[2564]: I0730 00:17:51.165666 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/853c653e-d075-4220-ae9c-86b76caa2229-etc-kubernetes\") pod \"tuned-t69rb\" (UID: \"853c653e-d075-4220-ae9c-86b76caa2229\") " pod="openshift-cluster-node-tuning-operator/tuned-t69rb" Jul 30 00:17:51.167280 ip-10-0-142-6 kubenswrapper[2564]: I0730 00:17:51.165692 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/dcc57a98-213c-4ed7-8b59-c0dd84a39b36-iptables-alerter-script\") pod \"iptables-alerter-k9l5m\" (UID: \"dcc57a98-213c-4ed7-8b59-c0dd84a39b36\") " pod="openshift-network-operator/iptables-alerter-k9l5m" Jul 30 00:17:51.167280 ip-10-0-142-6 kubenswrapper[2564]: I0730 00:17:51.165670 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/7ff24a6e-78df-4c03-830a-453386386626-host\") pod \"node-ca-jc8px\" (UID: \"7ff24a6e-78df-4c03-830a-453386386626\") " pod="openshift-image-registry/node-ca-jc8px" Jul 30 00:17:51.167280 ip-10-0-142-6 kubenswrapper[2564]: I0730 00:17:51.165734 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/e7215f9f-4c75-4ab5-85c7-e10ab231e916-env-overrides\") pod \"ovnkube-node-w8grm\" (UID: \"e7215f9f-4c75-4ab5-85c7-e10ab231e916\") " pod="openshift-ovn-kubernetes/ovnkube-node-w8grm" Jul 30 00:17:51.167280 ip-10-0-142-6 kubenswrapper[2564]: I0730 00:17:51.165753 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/853c653e-d075-4220-ae9c-86b76caa2229-lib-modules\") pod \"tuned-t69rb\" (UID: \"853c653e-d075-4220-ae9c-86b76caa2229\") " pod="openshift-cluster-node-tuning-operator/tuned-t69rb" Jul 30 00:17:51.167280 ip-10-0-142-6 kubenswrapper[2564]: I0730 00:17:51.165763 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/853c653e-d075-4220-ae9c-86b76caa2229-etc-kubernetes\") pod \"tuned-t69rb\" (UID: \"853c653e-d075-4220-ae9c-86b76caa2229\") " pod="openshift-cluster-node-tuning-operator/tuned-t69rb" Jul 30 00:17:51.167280 ip-10-0-142-6 kubenswrapper[2564]: I0730 00:17:51.165787 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/e7215f9f-4c75-4ab5-85c7-e10ab231e916-run-systemd\") pod \"ovnkube-node-w8grm\" (UID: \"e7215f9f-4c75-4ab5-85c7-e10ab231e916\") " pod="openshift-ovn-kubernetes/ovnkube-node-w8grm" Jul 30 00:17:51.167280 ip-10-0-142-6 kubenswrapper[2564]: I0730 00:17:51.165803 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/e7215f9f-4c75-4ab5-85c7-e10ab231e916-node-log\") pod \"ovnkube-node-w8grm\" (UID: \"e7215f9f-4c75-4ab5-85c7-e10ab231e916\") " pod="openshift-ovn-kubernetes/ovnkube-node-w8grm" Jul 30 00:17:51.167818 ip-10-0-142-6 kubenswrapper[2564]: I0730 00:17:51.165824 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/d2866704-b7d7-4515-9b9d-4906b1508f68-os-release\") pod \"multus-tphm4\" (UID: \"d2866704-b7d7-4515-9b9d-4906b1508f68\") " pod="openshift-multus/multus-tphm4" Jul 30 00:17:51.167818 ip-10-0-142-6 kubenswrapper[2564]: I0730 00:17:51.165832 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/e7215f9f-4c75-4ab5-85c7-e10ab231e916-host-cni-bin\") pod \"ovnkube-node-w8grm\" (UID: \"e7215f9f-4c75-4ab5-85c7-e10ab231e916\") " pod="openshift-ovn-kubernetes/ovnkube-node-w8grm" Jul 30 00:17:51.167818 ip-10-0-142-6 kubenswrapper[2564]: I0730 00:17:51.165843 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/d2866704-b7d7-4515-9b9d-4906b1508f68-host-var-lib-cni-multus\") pod \"multus-tphm4\" (UID: \"d2866704-b7d7-4515-9b9d-4906b1508f68\") " pod="openshift-multus/multus-tphm4" Jul 30 00:17:51.167818 ip-10-0-142-6 kubenswrapper[2564]: I0730 00:17:51.165880 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/d2866704-b7d7-4515-9b9d-4906b1508f68-multus-conf-dir\") pod \"multus-tphm4\" (UID: \"d2866704-b7d7-4515-9b9d-4906b1508f68\") " pod="openshift-multus/multus-tphm4" Jul 30 00:17:51.167818 ip-10-0-142-6 kubenswrapper[2564]: I0730 00:17:51.165882 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/d2866704-b7d7-4515-9b9d-4906b1508f68-system-cni-dir\") pod \"multus-tphm4\" (UID: \"d2866704-b7d7-4515-9b9d-4906b1508f68\") " pod="openshift-multus/multus-tphm4" Jul 30 00:17:51.167818 ip-10-0-142-6 kubenswrapper[2564]: I0730 00:17:51.165978 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/7ff24a6e-78df-4c03-830a-453386386626-serviceca\") pod \"node-ca-jc8px\" (UID: \"7ff24a6e-78df-4c03-830a-453386386626\") " pod="openshift-image-registry/node-ca-jc8px" Jul 30 00:17:51.167818 ip-10-0-142-6 kubenswrapper[2564]: I0730 00:17:51.166233 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/d2866704-b7d7-4515-9b9d-4906b1508f68-cni-binary-copy\") pod \"multus-tphm4\" (UID: \"d2866704-b7d7-4515-9b9d-4906b1508f68\") " pod="openshift-multus/multus-tphm4" Jul 30 00:17:51.167818 ip-10-0-142-6 kubenswrapper[2564]: I0730 00:17:51.166368 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/d2866704-b7d7-4515-9b9d-4906b1508f68-multus-daemon-config\") pod \"multus-tphm4\" (UID: \"d2866704-b7d7-4515-9b9d-4906b1508f68\") " pod="openshift-multus/multus-tphm4" Jul 30 00:17:51.167818 ip-10-0-142-6 kubenswrapper[2564]: I0730 00:17:51.167319 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/853c653e-d075-4220-ae9c-86b76caa2229-etc-tuned\") pod \"tuned-t69rb\" (UID: \"853c653e-d075-4220-ae9c-86b76caa2229\") " pod="openshift-cluster-node-tuning-operator/tuned-t69rb" Jul 30 00:17:51.168103 ip-10-0-142-6 kubenswrapper[2564]: I0730 00:17:51.167825 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/853c653e-d075-4220-ae9c-86b76caa2229-tmp\") pod \"tuned-t69rb\" (UID: \"853c653e-d075-4220-ae9c-86b76caa2229\") " pod="openshift-cluster-node-tuning-operator/tuned-t69rb" Jul 30 00:17:51.168299 ip-10-0-142-6 kubenswrapper[2564]: I0730 00:17:51.168274 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/e7215f9f-4c75-4ab5-85c7-e10ab231e916-ovn-node-metrics-cert\") pod \"ovnkube-node-w8grm\" (UID: \"e7215f9f-4c75-4ab5-85c7-e10ab231e916\") " pod="openshift-ovn-kubernetes/ovnkube-node-w8grm" Jul 30 00:17:51.174538 ip-10-0-142-6 kubenswrapper[2564]: I0730 00:17:51.174467 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5bftn\" (UniqueName: \"kubernetes.io/projected/dcc57a98-213c-4ed7-8b59-c0dd84a39b36-kube-api-access-5bftn\") pod \"iptables-alerter-k9l5m\" (UID: \"dcc57a98-213c-4ed7-8b59-c0dd84a39b36\") " pod="openshift-network-operator/iptables-alerter-k9l5m" Jul 30 00:17:51.174885 ip-10-0-142-6 kubenswrapper[2564]: I0730 00:17:51.174865 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5drxq\" (UniqueName: \"kubernetes.io/projected/853c653e-d075-4220-ae9c-86b76caa2229-kube-api-access-5drxq\") pod \"tuned-t69rb\" (UID: \"853c653e-d075-4220-ae9c-86b76caa2229\") " pod="openshift-cluster-node-tuning-operator/tuned-t69rb" Jul 30 00:17:51.175011 ip-10-0-142-6 kubenswrapper[2564]: I0730 00:17:51.174989 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6v9nv\" (UniqueName: \"kubernetes.io/projected/f82506d3-a3ca-415e-b6c0-4de2f2bf35f5-kube-api-access-6v9nv\") pod \"network-metrics-daemon-5skn2\" (UID: \"f82506d3-a3ca-415e-b6c0-4de2f2bf35f5\") " pod="openshift-multus/network-metrics-daemon-5skn2" Jul 30 00:17:51.175350 ip-10-0-142-6 kubenswrapper[2564]: I0730 00:17:51.175330 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xvk2p\" (UniqueName: \"kubernetes.io/projected/d2866704-b7d7-4515-9b9d-4906b1508f68-kube-api-access-xvk2p\") pod \"multus-tphm4\" (UID: \"d2866704-b7d7-4515-9b9d-4906b1508f68\") " pod="openshift-multus/multus-tphm4" Jul 30 00:17:51.176295 ip-10-0-142-6 kubenswrapper[2564]: I0730 00:17:51.176276 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2zflx\" (UniqueName: \"kubernetes.io/projected/e7215f9f-4c75-4ab5-85c7-e10ab231e916-kube-api-access-2zflx\") pod \"ovnkube-node-w8grm\" (UID: \"e7215f9f-4c75-4ab5-85c7-e10ab231e916\") " pod="openshift-ovn-kubernetes/ovnkube-node-w8grm" Jul 30 00:17:51.176965 ip-10-0-142-6 kubenswrapper[2564]: I0730 00:17:51.176948 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-t2mrd\" (UniqueName: \"kubernetes.io/projected/7ff24a6e-78df-4c03-830a-453386386626-kube-api-access-t2mrd\") pod \"node-ca-jc8px\" (UID: \"7ff24a6e-78df-4c03-830a-453386386626\") " pod="openshift-image-registry/node-ca-jc8px" Jul 30 00:17:51.258831 ip-10-0-142-6 kubenswrapper[2564]: I0730 00:17:51.258798 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-thhcs" Jul 30 00:17:51.266805 ip-10-0-142-6 kubenswrapper[2564]: I0730 00:17:51.266779 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-c6sdp" Jul 30 00:17:51.278553 ip-10-0-142-6 kubenswrapper[2564]: I0730 00:17:51.278522 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6khjg" Jul 30 00:17:51.286308 ip-10-0-142-6 kubenswrapper[2564]: I0730 00:17:51.286284 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-6ssnm" Jul 30 00:17:51.292186 ip-10-0-142-6 kubenswrapper[2564]: I0730 00:17:51.292162 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-w8grm" Jul 30 00:17:51.301963 ip-10-0-142-6 kubenswrapper[2564]: I0730 00:17:51.301928 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-k9l5m" Jul 30 00:17:51.311855 ip-10-0-142-6 kubenswrapper[2564]: I0730 00:17:51.311823 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-tphm4" Jul 30 00:17:51.322563 ip-10-0-142-6 kubenswrapper[2564]: I0730 00:17:51.322534 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-jc8px" Jul 30 00:17:51.328448 ip-10-0-142-6 kubenswrapper[2564]: I0730 00:17:51.328420 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-t69rb" Jul 30 00:17:51.376668 ip-10-0-142-6 kubenswrapper[2564]: I0730 00:17:51.376631 2564 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160" Jul 30 00:17:51.568641 ip-10-0-142-6 kubenswrapper[2564]: I0730 00:17:51.568550 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5v98n\" (UniqueName: \"kubernetes.io/projected/a0bef212-99cc-47c3-b1da-316f26f776b9-kube-api-access-5v98n\") pod \"network-check-target-j4ckt\" (UID: \"a0bef212-99cc-47c3-b1da-316f26f776b9\") " pod="openshift-network-diagnostics/network-check-target-j4ckt" Jul 30 00:17:51.568641 ip-10-0-142-6 kubenswrapper[2564]: I0730 00:17:51.568593 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/81450593-06b9-4f5a-81da-6e5fe67679f1-original-pull-secret\") pod \"global-pull-secret-syncer-d6phj\" (UID: \"81450593-06b9-4f5a-81da-6e5fe67679f1\") " pod="kube-system/global-pull-secret-syncer-d6phj" Jul 30 00:17:51.568857 ip-10-0-142-6 kubenswrapper[2564]: E0730 00:17:51.568725 2564 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 30 00:17:51.568857 ip-10-0-142-6 kubenswrapper[2564]: E0730 00:17:51.568788 2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/81450593-06b9-4f5a-81da-6e5fe67679f1-original-pull-secret podName:81450593-06b9-4f5a-81da-6e5fe67679f1 nodeName:}" failed. No retries permitted until 2026-07-30 00:17:52.568769425 +0000 UTC m=+3.983509340 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/81450593-06b9-4f5a-81da-6e5fe67679f1-original-pull-secret") pod "global-pull-secret-syncer-d6phj" (UID: "81450593-06b9-4f5a-81da-6e5fe67679f1") : object "kube-system"/"original-pull-secret" not registered Jul 30 00:17:51.568857 ip-10-0-142-6 kubenswrapper[2564]: E0730 00:17:51.568729 2564 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 30 00:17:51.568857 ip-10-0-142-6 kubenswrapper[2564]: E0730 00:17:51.568834 2564 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 30 00:17:51.568857 ip-10-0-142-6 kubenswrapper[2564]: E0730 00:17:51.568854 2564 projected.go:196] Error preparing data for projected volume kube-api-access-5v98n for pod openshift-network-diagnostics/network-check-target-j4ckt: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 30 00:17:51.569058 ip-10-0-142-6 kubenswrapper[2564]: E0730 00:17:51.568908 2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/a0bef212-99cc-47c3-b1da-316f26f776b9-kube-api-access-5v98n podName:a0bef212-99cc-47c3-b1da-316f26f776b9 nodeName:}" failed. No retries permitted until 2026-07-30 00:17:52.568892508 +0000 UTC m=+3.983632423 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-api-access-5v98n" (UniqueName: "kubernetes.io/projected/a0bef212-99cc-47c3-b1da-316f26f776b9-kube-api-access-5v98n") pod "network-check-target-j4ckt" (UID: "a0bef212-99cc-47c3-b1da-316f26f776b9") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 30 00:17:51.669829 ip-10-0-142-6 kubenswrapper[2564]: I0730 00:17:51.669790 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f82506d3-a3ca-415e-b6c0-4de2f2bf35f5-metrics-certs\") pod \"network-metrics-daemon-5skn2\" (UID: \"f82506d3-a3ca-415e-b6c0-4de2f2bf35f5\") " pod="openshift-multus/network-metrics-daemon-5skn2" Jul 30 00:17:51.670019 ip-10-0-142-6 kubenswrapper[2564]: E0730 00:17:51.669950 2564 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 30 00:17:51.670106 ip-10-0-142-6 kubenswrapper[2564]: E0730 00:17:51.670043 2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f82506d3-a3ca-415e-b6c0-4de2f2bf35f5-metrics-certs podName:f82506d3-a3ca-415e-b6c0-4de2f2bf35f5 nodeName:}" failed. No retries permitted until 2026-07-30 00:17:52.670021678 +0000 UTC m=+4.084761594 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/f82506d3-a3ca-415e-b6c0-4de2f2bf35f5-metrics-certs") pod "network-metrics-daemon-5skn2" (UID: "f82506d3-a3ca-415e-b6c0-4de2f2bf35f5") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 30 00:17:52.019486 ip-10-0-142-6 kubenswrapper[2564]: I0730 00:17:52.019376 2564 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-07-29 00:12:51 +0000 UTC" deadline="2028-02-12 21:22:34.34647289 +0000 UTC" Jul 30 00:17:52.019486 ip-10-0-142-6 kubenswrapper[2564]: I0730 00:17:52.019430 2564 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="13509h4m42.327046202s" Jul 30 00:17:52.134264 ip-10-0-142-6 kubenswrapper[2564]: I0730 00:17:52.134233 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-j4ckt" Jul 30 00:17:52.135156 ip-10-0-142-6 kubenswrapper[2564]: E0730 00:17:52.134367 2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-j4ckt" podUID="a0bef212-99cc-47c3-b1da-316f26f776b9" Jul 30 00:17:52.150959 ip-10-0-142-6 kubenswrapper[2564]: W0730 00:17:52.150871 2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode7215f9f_4c75_4ab5_85c7_e10ab231e916.slice/crio-9bdff5ad2635aed5ac1069822571c41eb554ef270a75ec0754eabc1e90f1a9c5 WatchSource:0}: Error finding container 9bdff5ad2635aed5ac1069822571c41eb554ef270a75ec0754eabc1e90f1a9c5: Status 404 returned error can't find the container with id 9bdff5ad2635aed5ac1069822571c41eb554ef270a75ec0754eabc1e90f1a9c5 Jul 30 00:17:52.154513 ip-10-0-142-6 kubenswrapper[2564]: W0730 00:17:52.154487 2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd2866704_b7d7_4515_9b9d_4906b1508f68.slice/crio-79974176dc40dc4b78cb0880c355335fb2a7a5f29cbd0b26849414929bc64206 WatchSource:0}: Error finding container 79974176dc40dc4b78cb0880c355335fb2a7a5f29cbd0b26849414929bc64206: Status 404 returned error can't find the container with id 79974176dc40dc4b78cb0880c355335fb2a7a5f29cbd0b26849414929bc64206 Jul 30 00:17:52.155196 ip-10-0-142-6 kubenswrapper[2564]: W0730 00:17:52.155122 2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3a12258d_f063_4ae8_8fec_9eb3b012df06.slice/crio-1629256386b8badd904f9a2981ac76fa5d87a3c90dfcf0f8cc5047e2237e496b WatchSource:0}: Error finding container 1629256386b8badd904f9a2981ac76fa5d87a3c90dfcf0f8cc5047e2237e496b: Status 404 returned error can't find the container with id 1629256386b8badd904f9a2981ac76fa5d87a3c90dfcf0f8cc5047e2237e496b Jul 30 00:17:52.156259 ip-10-0-142-6 kubenswrapper[2564]: W0730 00:17:52.155983 2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod99deae04_e4b4_4491_a31f_945927bd1e72.slice/crio-d28fe964f1b9afd6b1e1c3a37f0e5ca72f26f8c906f807b0078d87f5a19257b2 WatchSource:0}: Error finding container d28fe964f1b9afd6b1e1c3a37f0e5ca72f26f8c906f807b0078d87f5a19257b2: Status 404 returned error can't find the container with id d28fe964f1b9afd6b1e1c3a37f0e5ca72f26f8c906f807b0078d87f5a19257b2 Jul 30 00:17:52.157024 ip-10-0-142-6 kubenswrapper[2564]: W0730 00:17:52.156798 2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddcc57a98_213c_4ed7_8b59_c0dd84a39b36.slice/crio-b052727051352b70f1e22d6af294eec6af9bd1aee70d607532ab315386ecd0cc WatchSource:0}: Error finding container b052727051352b70f1e22d6af294eec6af9bd1aee70d607532ab315386ecd0cc: Status 404 returned error can't find the container with id b052727051352b70f1e22d6af294eec6af9bd1aee70d607532ab315386ecd0cc Jul 30 00:17:52.158660 ip-10-0-142-6 kubenswrapper[2564]: W0730 00:17:52.158639 2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7ff24a6e_78df_4c03_830a_453386386626.slice/crio-73e996f2de88eb855e2ddef9280726adbc9a0161ccde9dfa7f09570208b043cb WatchSource:0}: Error finding container 73e996f2de88eb855e2ddef9280726adbc9a0161ccde9dfa7f09570208b043cb: Status 404 returned error can't find the container with id 73e996f2de88eb855e2ddef9280726adbc9a0161ccde9dfa7f09570208b043cb Jul 30 00:17:52.160839 ip-10-0-142-6 kubenswrapper[2564]: W0730 00:17:52.160782 2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6a73d10b_1baa_42ac_a83a_2336014e3600.slice/crio-8fd50e2cffccb64623616eb04c9b63c4adc4d25f6a2d86d45fd2d2f0e3fc528d WatchSource:0}: Error finding container 8fd50e2cffccb64623616eb04c9b63c4adc4d25f6a2d86d45fd2d2f0e3fc528d: Status 404 returned error can't find the container with id 8fd50e2cffccb64623616eb04c9b63c4adc4d25f6a2d86d45fd2d2f0e3fc528d Jul 30 00:17:52.162003 ip-10-0-142-6 kubenswrapper[2564]: W0730 00:17:52.161946 2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb1186f7d_220d_4210_a7f8_6f3f0cf554e8.slice/crio-c58107f16c3e237b642f85d27f16ef45432d1c68967fb6cbe2e0a5dca2e7a879 WatchSource:0}: Error finding container c58107f16c3e237b642f85d27f16ef45432d1c68967fb6cbe2e0a5dca2e7a879: Status 404 returned error can't find the container with id c58107f16c3e237b642f85d27f16ef45432d1c68967fb6cbe2e0a5dca2e7a879 Jul 30 00:17:52.577447 ip-10-0-142-6 kubenswrapper[2564]: I0730 00:17:52.577185 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5v98n\" (UniqueName: \"kubernetes.io/projected/a0bef212-99cc-47c3-b1da-316f26f776b9-kube-api-access-5v98n\") pod \"network-check-target-j4ckt\" (UID: \"a0bef212-99cc-47c3-b1da-316f26f776b9\") " pod="openshift-network-diagnostics/network-check-target-j4ckt" Jul 30 00:17:52.577620 ip-10-0-142-6 kubenswrapper[2564]: E0730 00:17:52.577464 2564 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 30 00:17:52.577620 ip-10-0-142-6 kubenswrapper[2564]: I0730 00:17:52.577470 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/81450593-06b9-4f5a-81da-6e5fe67679f1-original-pull-secret\") pod \"global-pull-secret-syncer-d6phj\" (UID: \"81450593-06b9-4f5a-81da-6e5fe67679f1\") " pod="kube-system/global-pull-secret-syncer-d6phj" Jul 30 00:17:52.577620 ip-10-0-142-6 kubenswrapper[2564]: E0730 00:17:52.577489 2564 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 30 00:17:52.577620 ip-10-0-142-6 kubenswrapper[2564]: E0730 00:17:52.577503 2564 projected.go:196] Error preparing data for projected volume kube-api-access-5v98n for pod openshift-network-diagnostics/network-check-target-j4ckt: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 30 00:17:52.577620 ip-10-0-142-6 kubenswrapper[2564]: E0730 00:17:52.577559 2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/a0bef212-99cc-47c3-b1da-316f26f776b9-kube-api-access-5v98n podName:a0bef212-99cc-47c3-b1da-316f26f776b9 nodeName:}" failed. No retries permitted until 2026-07-30 00:17:54.577540526 +0000 UTC m=+5.992280438 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-api-access-5v98n" (UniqueName: "kubernetes.io/projected/a0bef212-99cc-47c3-b1da-316f26f776b9-kube-api-access-5v98n") pod "network-check-target-j4ckt" (UID: "a0bef212-99cc-47c3-b1da-316f26f776b9") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 30 00:17:52.577620 ip-10-0-142-6 kubenswrapper[2564]: E0730 00:17:52.577617 2564 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 30 00:17:52.577922 ip-10-0-142-6 kubenswrapper[2564]: E0730 00:17:52.577670 2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/81450593-06b9-4f5a-81da-6e5fe67679f1-original-pull-secret podName:81450593-06b9-4f5a-81da-6e5fe67679f1 nodeName:}" failed. No retries permitted until 2026-07-30 00:17:54.5776545 +0000 UTC m=+5.992394413 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/81450593-06b9-4f5a-81da-6e5fe67679f1-original-pull-secret") pod "global-pull-secret-syncer-d6phj" (UID: "81450593-06b9-4f5a-81da-6e5fe67679f1") : object "kube-system"/"original-pull-secret" not registered Jul 30 00:17:52.678887 ip-10-0-142-6 kubenswrapper[2564]: I0730 00:17:52.678852 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f82506d3-a3ca-415e-b6c0-4de2f2bf35f5-metrics-certs\") pod \"network-metrics-daemon-5skn2\" (UID: \"f82506d3-a3ca-415e-b6c0-4de2f2bf35f5\") " pod="openshift-multus/network-metrics-daemon-5skn2" Jul 30 00:17:52.679062 ip-10-0-142-6 kubenswrapper[2564]: E0730 00:17:52.678996 2564 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 30 00:17:52.679062 ip-10-0-142-6 kubenswrapper[2564]: E0730 00:17:52.679056 2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f82506d3-a3ca-415e-b6c0-4de2f2bf35f5-metrics-certs podName:f82506d3-a3ca-415e-b6c0-4de2f2bf35f5 nodeName:}" failed. No retries permitted until 2026-07-30 00:17:54.679038291 +0000 UTC m=+6.093778204 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/f82506d3-a3ca-415e-b6c0-4de2f2bf35f5-metrics-certs") pod "network-metrics-daemon-5skn2" (UID: "f82506d3-a3ca-415e-b6c0-4de2f2bf35f5") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 30 00:17:53.019952 ip-10-0-142-6 kubenswrapper[2564]: I0730 00:17:53.019831 2564 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-07-29 00:12:51 +0000 UTC" deadline="2028-02-29 23:57:32.660671301 +0000 UTC" Jul 30 00:17:53.019952 ip-10-0-142-6 kubenswrapper[2564]: I0730 00:17:53.019894 2564 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="13919h39m39.640781618s" Jul 30 00:17:53.133798 ip-10-0-142-6 kubenswrapper[2564]: I0730 00:17:53.133762 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-d6phj" Jul 30 00:17:53.133976 ip-10-0-142-6 kubenswrapper[2564]: I0730 00:17:53.133814 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-5skn2" Jul 30 00:17:53.133976 ip-10-0-142-6 kubenswrapper[2564]: E0730 00:17:53.133894 2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-d6phj" podUID="81450593-06b9-4f5a-81da-6e5fe67679f1" Jul 30 00:17:53.134107 ip-10-0-142-6 kubenswrapper[2564]: E0730 00:17:53.134027 2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-5skn2" podUID="f82506d3-a3ca-415e-b6c0-4de2f2bf35f5" Jul 30 00:17:53.149938 ip-10-0-142-6 kubenswrapper[2564]: I0730 00:17:53.149899 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-t69rb" event={"ID":"853c653e-d075-4220-ae9c-86b76caa2229","Type":"ContainerStarted","Data":"fabca95ada7f94aa12b6f8d5958d9187e7cb4d6897464f05b166be632fd25ffb"} Jul 30 00:17:53.153105 ip-10-0-142-6 kubenswrapper[2564]: I0730 00:17:53.153055 2564 generic.go:356] "Generic (PLEG): container finished" podID="7c269f27f5372a2f0fc63809270fef7a" containerID="1f1dcbb452402f28acbc8d0b95be047b4371365a7d47b695916ddab435bf34e2" exitCode=0 Jul 30 00:17:53.153250 ip-10-0-142-6 kubenswrapper[2564]: I0730 00:17:53.153152 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-142-6.ec2.internal" event={"ID":"7c269f27f5372a2f0fc63809270fef7a","Type":"ContainerDied","Data":"1f1dcbb452402f28acbc8d0b95be047b4371365a7d47b695916ddab435bf34e2"} Jul 30 00:17:53.158956 ip-10-0-142-6 kubenswrapper[2564]: I0730 00:17:53.158929 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-thhcs" event={"ID":"6a73d10b-1baa-42ac-a83a-2336014e3600","Type":"ContainerStarted","Data":"8fd50e2cffccb64623616eb04c9b63c4adc4d25f6a2d86d45fd2d2f0e3fc528d"} Jul 30 00:17:53.160779 ip-10-0-142-6 kubenswrapper[2564]: I0730 00:17:53.160753 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-jc8px" event={"ID":"7ff24a6e-78df-4c03-830a-453386386626","Type":"ContainerStarted","Data":"73e996f2de88eb855e2ddef9280726adbc9a0161ccde9dfa7f09570208b043cb"} Jul 30 00:17:53.174483 ip-10-0-142-6 kubenswrapper[2564]: I0730 00:17:53.174452 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-tphm4" event={"ID":"d2866704-b7d7-4515-9b9d-4906b1508f68","Type":"ContainerStarted","Data":"79974176dc40dc4b78cb0880c355335fb2a7a5f29cbd0b26849414929bc64206"} Jul 30 00:17:53.193238 ip-10-0-142-6 kubenswrapper[2564]: I0730 00:17:53.193200 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-w8grm" event={"ID":"e7215f9f-4c75-4ab5-85c7-e10ab231e916","Type":"ContainerStarted","Data":"9bdff5ad2635aed5ac1069822571c41eb554ef270a75ec0754eabc1e90f1a9c5"} Jul 30 00:17:53.200377 ip-10-0-142-6 kubenswrapper[2564]: I0730 00:17:53.200323 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-142-6.ec2.internal" event={"ID":"514977950851753a7cc9d5e8c3264eee","Type":"ContainerStarted","Data":"029f98ad5f12149ff77add223b5b515a0b8bd7060b63138244ee55c67fc1cd5b"} Jul 30 00:17:53.222865 ip-10-0-142-6 kubenswrapper[2564]: I0730 00:17:53.222830 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-c6sdp" event={"ID":"b1186f7d-220d-4210-a7f8-6f3f0cf554e8","Type":"ContainerStarted","Data":"c58107f16c3e237b642f85d27f16ef45432d1c68967fb6cbe2e0a5dca2e7a879"} Jul 30 00:17:53.230729 ip-10-0-142-6 kubenswrapper[2564]: I0730 00:17:53.230697 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-k9l5m" event={"ID":"dcc57a98-213c-4ed7-8b59-c0dd84a39b36","Type":"ContainerStarted","Data":"b052727051352b70f1e22d6af294eec6af9bd1aee70d607532ab315386ecd0cc"} Jul 30 00:17:53.235040 ip-10-0-142-6 kubenswrapper[2564]: I0730 00:17:53.235008 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-6ssnm" event={"ID":"99deae04-e4b4-4491-a31f-945927bd1e72","Type":"ContainerStarted","Data":"d28fe964f1b9afd6b1e1c3a37f0e5ca72f26f8c906f807b0078d87f5a19257b2"} Jul 30 00:17:53.239602 ip-10-0-142-6 kubenswrapper[2564]: I0730 00:17:53.239575 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6khjg" event={"ID":"3a12258d-f063-4ae8-8fec-9eb3b012df06","Type":"ContainerStarted","Data":"1629256386b8badd904f9a2981ac76fa5d87a3c90dfcf0f8cc5047e2237e496b"} Jul 30 00:17:54.134203 ip-10-0-142-6 kubenswrapper[2564]: I0730 00:17:54.134164 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-j4ckt" Jul 30 00:17:54.134669 ip-10-0-142-6 kubenswrapper[2564]: E0730 00:17:54.134294 2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-j4ckt" podUID="a0bef212-99cc-47c3-b1da-316f26f776b9" Jul 30 00:17:54.265388 ip-10-0-142-6 kubenswrapper[2564]: I0730 00:17:54.265352 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-142-6.ec2.internal" event={"ID":"7c269f27f5372a2f0fc63809270fef7a","Type":"ContainerStarted","Data":"b2bf62ff039586bf533ba5006cb82c225e356aa31f2cf685ef464c9d045803a5"} Jul 30 00:17:54.281861 ip-10-0-142-6 kubenswrapper[2564]: I0730 00:17:54.281791 2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/kube-apiserver-proxy-ip-10-0-142-6.ec2.internal" podStartSLOduration=4.281769555 podStartE2EDuration="4.281769555s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-30 00:17:50 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-30 00:17:53.215233326 +0000 UTC m=+4.629973260" watchObservedRunningTime="2026-07-30 00:17:54.281769555 +0000 UTC m=+5.696509489" Jul 30 00:17:54.594046 ip-10-0-142-6 kubenswrapper[2564]: I0730 00:17:54.593961 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5v98n\" (UniqueName: \"kubernetes.io/projected/a0bef212-99cc-47c3-b1da-316f26f776b9-kube-api-access-5v98n\") pod \"network-check-target-j4ckt\" (UID: \"a0bef212-99cc-47c3-b1da-316f26f776b9\") " pod="openshift-network-diagnostics/network-check-target-j4ckt" Jul 30 00:17:54.594046 ip-10-0-142-6 kubenswrapper[2564]: I0730 00:17:54.594011 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/81450593-06b9-4f5a-81da-6e5fe67679f1-original-pull-secret\") pod \"global-pull-secret-syncer-d6phj\" (UID: \"81450593-06b9-4f5a-81da-6e5fe67679f1\") " pod="kube-system/global-pull-secret-syncer-d6phj" Jul 30 00:17:54.594271 ip-10-0-142-6 kubenswrapper[2564]: E0730 00:17:54.594176 2564 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 30 00:17:54.594271 ip-10-0-142-6 kubenswrapper[2564]: E0730 00:17:54.594239 2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/81450593-06b9-4f5a-81da-6e5fe67679f1-original-pull-secret podName:81450593-06b9-4f5a-81da-6e5fe67679f1 nodeName:}" failed. No retries permitted until 2026-07-30 00:17:58.594221085 +0000 UTC m=+10.008961015 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/81450593-06b9-4f5a-81da-6e5fe67679f1-original-pull-secret") pod "global-pull-secret-syncer-d6phj" (UID: "81450593-06b9-4f5a-81da-6e5fe67679f1") : object "kube-system"/"original-pull-secret" not registered Jul 30 00:17:54.594685 ip-10-0-142-6 kubenswrapper[2564]: E0730 00:17:54.594667 2564 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 30 00:17:54.594799 ip-10-0-142-6 kubenswrapper[2564]: E0730 00:17:54.594690 2564 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 30 00:17:54.594799 ip-10-0-142-6 kubenswrapper[2564]: E0730 00:17:54.594702 2564 projected.go:196] Error preparing data for projected volume kube-api-access-5v98n for pod openshift-network-diagnostics/network-check-target-j4ckt: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 30 00:17:54.594799 ip-10-0-142-6 kubenswrapper[2564]: E0730 00:17:54.594747 2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/a0bef212-99cc-47c3-b1da-316f26f776b9-kube-api-access-5v98n podName:a0bef212-99cc-47c3-b1da-316f26f776b9 nodeName:}" failed. No retries permitted until 2026-07-30 00:17:58.594731932 +0000 UTC m=+10.009471863 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-api-access-5v98n" (UniqueName: "kubernetes.io/projected/a0bef212-99cc-47c3-b1da-316f26f776b9-kube-api-access-5v98n") pod "network-check-target-j4ckt" (UID: "a0bef212-99cc-47c3-b1da-316f26f776b9") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 30 00:17:54.694985 ip-10-0-142-6 kubenswrapper[2564]: I0730 00:17:54.694945 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f82506d3-a3ca-415e-b6c0-4de2f2bf35f5-metrics-certs\") pod \"network-metrics-daemon-5skn2\" (UID: \"f82506d3-a3ca-415e-b6c0-4de2f2bf35f5\") " pod="openshift-multus/network-metrics-daemon-5skn2" Jul 30 00:17:54.695182 ip-10-0-142-6 kubenswrapper[2564]: E0730 00:17:54.695130 2564 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 30 00:17:54.695270 ip-10-0-142-6 kubenswrapper[2564]: E0730 00:17:54.695200 2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f82506d3-a3ca-415e-b6c0-4de2f2bf35f5-metrics-certs podName:f82506d3-a3ca-415e-b6c0-4de2f2bf35f5 nodeName:}" failed. No retries permitted until 2026-07-30 00:17:58.695179517 +0000 UTC m=+10.109919431 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/f82506d3-a3ca-415e-b6c0-4de2f2bf35f5-metrics-certs") pod "network-metrics-daemon-5skn2" (UID: "f82506d3-a3ca-415e-b6c0-4de2f2bf35f5") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 30 00:17:55.134234 ip-10-0-142-6 kubenswrapper[2564]: I0730 00:17:55.134201 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-d6phj" Jul 30 00:17:55.134684 ip-10-0-142-6 kubenswrapper[2564]: E0730 00:17:55.134343 2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-d6phj" podUID="81450593-06b9-4f5a-81da-6e5fe67679f1" Jul 30 00:17:55.134684 ip-10-0-142-6 kubenswrapper[2564]: I0730 00:17:55.134200 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-5skn2" Jul 30 00:17:55.134804 ip-10-0-142-6 kubenswrapper[2564]: E0730 00:17:55.134786 2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-5skn2" podUID="f82506d3-a3ca-415e-b6c0-4de2f2bf35f5" Jul 30 00:17:56.134525 ip-10-0-142-6 kubenswrapper[2564]: I0730 00:17:56.134141 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-j4ckt" Jul 30 00:17:56.134525 ip-10-0-142-6 kubenswrapper[2564]: E0730 00:17:56.134284 2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-j4ckt" podUID="a0bef212-99cc-47c3-b1da-316f26f776b9" Jul 30 00:17:57.134583 ip-10-0-142-6 kubenswrapper[2564]: I0730 00:17:57.134106 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-5skn2" Jul 30 00:17:57.134583 ip-10-0-142-6 kubenswrapper[2564]: E0730 00:17:57.134241 2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-5skn2" podUID="f82506d3-a3ca-415e-b6c0-4de2f2bf35f5" Jul 30 00:17:57.135115 ip-10-0-142-6 kubenswrapper[2564]: I0730 00:17:57.134658 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-d6phj" Jul 30 00:17:57.135115 ip-10-0-142-6 kubenswrapper[2564]: E0730 00:17:57.134778 2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-d6phj" podUID="81450593-06b9-4f5a-81da-6e5fe67679f1" Jul 30 00:17:58.134736 ip-10-0-142-6 kubenswrapper[2564]: I0730 00:17:58.134226 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-j4ckt" Jul 30 00:17:58.134736 ip-10-0-142-6 kubenswrapper[2564]: E0730 00:17:58.134373 2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-j4ckt" podUID="a0bef212-99cc-47c3-b1da-316f26f776b9" Jul 30 00:17:58.628590 ip-10-0-142-6 kubenswrapper[2564]: I0730 00:17:58.628550 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5v98n\" (UniqueName: \"kubernetes.io/projected/a0bef212-99cc-47c3-b1da-316f26f776b9-kube-api-access-5v98n\") pod \"network-check-target-j4ckt\" (UID: \"a0bef212-99cc-47c3-b1da-316f26f776b9\") " pod="openshift-network-diagnostics/network-check-target-j4ckt" Jul 30 00:17:58.628590 ip-10-0-142-6 kubenswrapper[2564]: I0730 00:17:58.628607 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/81450593-06b9-4f5a-81da-6e5fe67679f1-original-pull-secret\") pod \"global-pull-secret-syncer-d6phj\" (UID: \"81450593-06b9-4f5a-81da-6e5fe67679f1\") " pod="kube-system/global-pull-secret-syncer-d6phj" Jul 30 00:17:58.628808 ip-10-0-142-6 kubenswrapper[2564]: E0730 00:17:58.628746 2564 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 30 00:17:58.628808 ip-10-0-142-6 kubenswrapper[2564]: E0730 00:17:58.628795 2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/81450593-06b9-4f5a-81da-6e5fe67679f1-original-pull-secret podName:81450593-06b9-4f5a-81da-6e5fe67679f1 nodeName:}" failed. No retries permitted until 2026-07-30 00:18:06.62878136 +0000 UTC m=+18.043521271 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/81450593-06b9-4f5a-81da-6e5fe67679f1-original-pull-secret") pod "global-pull-secret-syncer-d6phj" (UID: "81450593-06b9-4f5a-81da-6e5fe67679f1") : object "kube-system"/"original-pull-secret" not registered Jul 30 00:17:58.629179 ip-10-0-142-6 kubenswrapper[2564]: E0730 00:17:58.629162 2564 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 30 00:17:58.629269 ip-10-0-142-6 kubenswrapper[2564]: E0730 00:17:58.629184 2564 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 30 00:17:58.629269 ip-10-0-142-6 kubenswrapper[2564]: E0730 00:17:58.629195 2564 projected.go:196] Error preparing data for projected volume kube-api-access-5v98n for pod openshift-network-diagnostics/network-check-target-j4ckt: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 30 00:17:58.629269 ip-10-0-142-6 kubenswrapper[2564]: E0730 00:17:58.629230 2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/a0bef212-99cc-47c3-b1da-316f26f776b9-kube-api-access-5v98n podName:a0bef212-99cc-47c3-b1da-316f26f776b9 nodeName:}" failed. No retries permitted until 2026-07-30 00:18:06.629218996 +0000 UTC m=+18.043958907 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-api-access-5v98n" (UniqueName: "kubernetes.io/projected/a0bef212-99cc-47c3-b1da-316f26f776b9-kube-api-access-5v98n") pod "network-check-target-j4ckt" (UID: "a0bef212-99cc-47c3-b1da-316f26f776b9") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 30 00:17:58.729459 ip-10-0-142-6 kubenswrapper[2564]: I0730 00:17:58.729378 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f82506d3-a3ca-415e-b6c0-4de2f2bf35f5-metrics-certs\") pod \"network-metrics-daemon-5skn2\" (UID: \"f82506d3-a3ca-415e-b6c0-4de2f2bf35f5\") " pod="openshift-multus/network-metrics-daemon-5skn2" Jul 30 00:17:58.729607 ip-10-0-142-6 kubenswrapper[2564]: E0730 00:17:58.729533 2564 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 30 00:17:58.729647 ip-10-0-142-6 kubenswrapper[2564]: E0730 00:17:58.729609 2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f82506d3-a3ca-415e-b6c0-4de2f2bf35f5-metrics-certs podName:f82506d3-a3ca-415e-b6c0-4de2f2bf35f5 nodeName:}" failed. No retries permitted until 2026-07-30 00:18:06.729585982 +0000 UTC m=+18.144325908 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/f82506d3-a3ca-415e-b6c0-4de2f2bf35f5-metrics-certs") pod "network-metrics-daemon-5skn2" (UID: "f82506d3-a3ca-415e-b6c0-4de2f2bf35f5") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 30 00:17:59.135886 ip-10-0-142-6 kubenswrapper[2564]: I0730 00:17:59.135236 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-5skn2" Jul 30 00:17:59.135886 ip-10-0-142-6 kubenswrapper[2564]: E0730 00:17:59.135347 2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-5skn2" podUID="f82506d3-a3ca-415e-b6c0-4de2f2bf35f5" Jul 30 00:17:59.135886 ip-10-0-142-6 kubenswrapper[2564]: I0730 00:17:59.135744 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-d6phj" Jul 30 00:17:59.135886 ip-10-0-142-6 kubenswrapper[2564]: E0730 00:17:59.135837 2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-d6phj" podUID="81450593-06b9-4f5a-81da-6e5fe67679f1" Jul 30 00:18:00.134794 ip-10-0-142-6 kubenswrapper[2564]: I0730 00:18:00.134307 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-j4ckt" Jul 30 00:18:00.134794 ip-10-0-142-6 kubenswrapper[2564]: E0730 00:18:00.134440 2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-j4ckt" podUID="a0bef212-99cc-47c3-b1da-316f26f776b9" Jul 30 00:18:01.134639 ip-10-0-142-6 kubenswrapper[2564]: I0730 00:18:01.134177 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-5skn2" Jul 30 00:18:01.134639 ip-10-0-142-6 kubenswrapper[2564]: I0730 00:18:01.134194 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-d6phj" Jul 30 00:18:01.134639 ip-10-0-142-6 kubenswrapper[2564]: E0730 00:18:01.134306 2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-5skn2" podUID="f82506d3-a3ca-415e-b6c0-4de2f2bf35f5" Jul 30 00:18:01.134639 ip-10-0-142-6 kubenswrapper[2564]: E0730 00:18:01.134423 2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-d6phj" podUID="81450593-06b9-4f5a-81da-6e5fe67679f1" Jul 30 00:18:02.133350 ip-10-0-142-6 kubenswrapper[2564]: I0730 00:18:02.133308 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-j4ckt" Jul 30 00:18:02.133534 ip-10-0-142-6 kubenswrapper[2564]: E0730 00:18:02.133426 2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-j4ckt" podUID="a0bef212-99cc-47c3-b1da-316f26f776b9" Jul 30 00:18:03.133687 ip-10-0-142-6 kubenswrapper[2564]: I0730 00:18:03.133654 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-5skn2" Jul 30 00:18:03.134063 ip-10-0-142-6 kubenswrapper[2564]: E0730 00:18:03.133835 2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-5skn2" podUID="f82506d3-a3ca-415e-b6c0-4de2f2bf35f5" Jul 30 00:18:03.134063 ip-10-0-142-6 kubenswrapper[2564]: I0730 00:18:03.133986 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-d6phj" Jul 30 00:18:03.134173 ip-10-0-142-6 kubenswrapper[2564]: E0730 00:18:03.134121 2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-d6phj" podUID="81450593-06b9-4f5a-81da-6e5fe67679f1" Jul 30 00:18:04.134086 ip-10-0-142-6 kubenswrapper[2564]: I0730 00:18:04.134039 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-j4ckt" Jul 30 00:18:04.134524 ip-10-0-142-6 kubenswrapper[2564]: E0730 00:18:04.134164 2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-j4ckt" podUID="a0bef212-99cc-47c3-b1da-316f26f776b9" Jul 30 00:18:05.133711 ip-10-0-142-6 kubenswrapper[2564]: I0730 00:18:05.133673 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-d6phj" Jul 30 00:18:05.133900 ip-10-0-142-6 kubenswrapper[2564]: I0730 00:18:05.133673 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-5skn2" Jul 30 00:18:05.133900 ip-10-0-142-6 kubenswrapper[2564]: E0730 00:18:05.133818 2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-d6phj" podUID="81450593-06b9-4f5a-81da-6e5fe67679f1" Jul 30 00:18:05.133900 ip-10-0-142-6 kubenswrapper[2564]: E0730 00:18:05.133868 2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-5skn2" podUID="f82506d3-a3ca-415e-b6c0-4de2f2bf35f5" Jul 30 00:18:06.133447 ip-10-0-142-6 kubenswrapper[2564]: I0730 00:18:06.133410 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-j4ckt" Jul 30 00:18:06.133952 ip-10-0-142-6 kubenswrapper[2564]: E0730 00:18:06.133545 2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-j4ckt" podUID="a0bef212-99cc-47c3-b1da-316f26f776b9" Jul 30 00:18:06.691228 ip-10-0-142-6 kubenswrapper[2564]: I0730 00:18:06.691196 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5v98n\" (UniqueName: \"kubernetes.io/projected/a0bef212-99cc-47c3-b1da-316f26f776b9-kube-api-access-5v98n\") pod \"network-check-target-j4ckt\" (UID: \"a0bef212-99cc-47c3-b1da-316f26f776b9\") " pod="openshift-network-diagnostics/network-check-target-j4ckt" Jul 30 00:18:06.691228 ip-10-0-142-6 kubenswrapper[2564]: I0730 00:18:06.691233 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/81450593-06b9-4f5a-81da-6e5fe67679f1-original-pull-secret\") pod \"global-pull-secret-syncer-d6phj\" (UID: \"81450593-06b9-4f5a-81da-6e5fe67679f1\") " pod="kube-system/global-pull-secret-syncer-d6phj" Jul 30 00:18:06.691437 ip-10-0-142-6 kubenswrapper[2564]: E0730 00:18:06.691327 2564 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 30 00:18:06.691437 ip-10-0-142-6 kubenswrapper[2564]: E0730 00:18:06.691360 2564 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 30 00:18:06.691437 ip-10-0-142-6 kubenswrapper[2564]: E0730 00:18:06.691385 2564 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 30 00:18:06.691437 ip-10-0-142-6 kubenswrapper[2564]: E0730 00:18:06.691396 2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/81450593-06b9-4f5a-81da-6e5fe67679f1-original-pull-secret podName:81450593-06b9-4f5a-81da-6e5fe67679f1 nodeName:}" failed. No retries permitted until 2026-07-30 00:18:22.691375455 +0000 UTC m=+34.106115377 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/81450593-06b9-4f5a-81da-6e5fe67679f1-original-pull-secret") pod "global-pull-secret-syncer-d6phj" (UID: "81450593-06b9-4f5a-81da-6e5fe67679f1") : object "kube-system"/"original-pull-secret" not registered Jul 30 00:18:06.691437 ip-10-0-142-6 kubenswrapper[2564]: E0730 00:18:06.691398 2564 projected.go:196] Error preparing data for projected volume kube-api-access-5v98n for pod openshift-network-diagnostics/network-check-target-j4ckt: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 30 00:18:06.691437 ip-10-0-142-6 kubenswrapper[2564]: E0730 00:18:06.691433 2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/a0bef212-99cc-47c3-b1da-316f26f776b9-kube-api-access-5v98n podName:a0bef212-99cc-47c3-b1da-316f26f776b9 nodeName:}" failed. No retries permitted until 2026-07-30 00:18:22.691427542 +0000 UTC m=+34.106167453 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "kube-api-access-5v98n" (UniqueName: "kubernetes.io/projected/a0bef212-99cc-47c3-b1da-316f26f776b9-kube-api-access-5v98n") pod "network-check-target-j4ckt" (UID: "a0bef212-99cc-47c3-b1da-316f26f776b9") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 30 00:18:06.792033 ip-10-0-142-6 kubenswrapper[2564]: I0730 00:18:06.791956 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f82506d3-a3ca-415e-b6c0-4de2f2bf35f5-metrics-certs\") pod \"network-metrics-daemon-5skn2\" (UID: \"f82506d3-a3ca-415e-b6c0-4de2f2bf35f5\") " pod="openshift-multus/network-metrics-daemon-5skn2" Jul 30 00:18:06.792184 ip-10-0-142-6 kubenswrapper[2564]: E0730 00:18:06.792130 2564 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 30 00:18:06.792233 ip-10-0-142-6 kubenswrapper[2564]: E0730 00:18:06.792188 2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f82506d3-a3ca-415e-b6c0-4de2f2bf35f5-metrics-certs podName:f82506d3-a3ca-415e-b6c0-4de2f2bf35f5 nodeName:}" failed. No retries permitted until 2026-07-30 00:18:22.792173791 +0000 UTC m=+34.206913703 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/f82506d3-a3ca-415e-b6c0-4de2f2bf35f5-metrics-certs") pod "network-metrics-daemon-5skn2" (UID: "f82506d3-a3ca-415e-b6c0-4de2f2bf35f5") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 30 00:18:07.134039 ip-10-0-142-6 kubenswrapper[2564]: I0730 00:18:07.134009 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-5skn2" Jul 30 00:18:07.134471 ip-10-0-142-6 kubenswrapper[2564]: E0730 00:18:07.134160 2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-5skn2" podUID="f82506d3-a3ca-415e-b6c0-4de2f2bf35f5" Jul 30 00:18:07.134471 ip-10-0-142-6 kubenswrapper[2564]: I0730 00:18:07.134201 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-d6phj" Jul 30 00:18:07.134471 ip-10-0-142-6 kubenswrapper[2564]: E0730 00:18:07.134273 2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-d6phj" podUID="81450593-06b9-4f5a-81da-6e5fe67679f1" Jul 30 00:18:08.133594 ip-10-0-142-6 kubenswrapper[2564]: I0730 00:18:08.133557 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-j4ckt" Jul 30 00:18:08.133784 ip-10-0-142-6 kubenswrapper[2564]: E0730 00:18:08.133693 2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-j4ckt" podUID="a0bef212-99cc-47c3-b1da-316f26f776b9" Jul 30 00:18:09.134348 ip-10-0-142-6 kubenswrapper[2564]: I0730 00:18:09.134313 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-5skn2" Jul 30 00:18:09.134854 ip-10-0-142-6 kubenswrapper[2564]: E0730 00:18:09.134430 2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-5skn2" podUID="f82506d3-a3ca-415e-b6c0-4de2f2bf35f5" Jul 30 00:18:09.134854 ip-10-0-142-6 kubenswrapper[2564]: I0730 00:18:09.134516 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-d6phj" Jul 30 00:18:09.134854 ip-10-0-142-6 kubenswrapper[2564]: E0730 00:18:09.134629 2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-d6phj" podUID="81450593-06b9-4f5a-81da-6e5fe67679f1" Jul 30 00:18:10.134268 ip-10-0-142-6 kubenswrapper[2564]: I0730 00:18:10.134227 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-j4ckt" Jul 30 00:18:10.134427 ip-10-0-142-6 kubenswrapper[2564]: E0730 00:18:10.134358 2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-j4ckt" podUID="a0bef212-99cc-47c3-b1da-316f26f776b9" Jul 30 00:18:11.133735 ip-10-0-142-6 kubenswrapper[2564]: I0730 00:18:11.133410 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-5skn2" Jul 30 00:18:11.133929 ip-10-0-142-6 kubenswrapper[2564]: I0730 00:18:11.133432 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-d6phj" Jul 30 00:18:11.133929 ip-10-0-142-6 kubenswrapper[2564]: E0730 00:18:11.133846 2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-5skn2" podUID="f82506d3-a3ca-415e-b6c0-4de2f2bf35f5" Jul 30 00:18:11.133929 ip-10-0-142-6 kubenswrapper[2564]: E0730 00:18:11.133893 2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-d6phj" podUID="81450593-06b9-4f5a-81da-6e5fe67679f1" Jul 30 00:18:11.303373 ip-10-0-142-6 kubenswrapper[2564]: I0730 00:18:11.303332 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-thhcs" event={"ID":"6a73d10b-1baa-42ac-a83a-2336014e3600","Type":"ContainerStarted","Data":"7af5db3988d6d07bd899df68d4ead7711ac204a5b2e68e3a36a6160195912e2d"} Jul 30 00:18:11.304985 ip-10-0-142-6 kubenswrapper[2564]: I0730 00:18:11.304948 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-jc8px" event={"ID":"7ff24a6e-78df-4c03-830a-453386386626","Type":"ContainerStarted","Data":"15f564e036c719e6bb121ccdcd289e44e7f295ff2742d6fdf4092470c1f91553"} Jul 30 00:18:11.306555 ip-10-0-142-6 kubenswrapper[2564]: I0730 00:18:11.306526 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-tphm4" event={"ID":"d2866704-b7d7-4515-9b9d-4906b1508f68","Type":"ContainerStarted","Data":"38414c903da8774b518ed9da242902d1992c8777072b94c301d50c09bf558b8d"} Jul 30 00:18:11.309542 ip-10-0-142-6 kubenswrapper[2564]: I0730 00:18:11.309519 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-w8grm" event={"ID":"e7215f9f-4c75-4ab5-85c7-e10ab231e916","Type":"ContainerStarted","Data":"265fc1a68eac7c858000fa2a6cf415c8134247441d6fb6a52f8537b0cc081064"} Jul 30 00:18:11.309665 ip-10-0-142-6 kubenswrapper[2564]: I0730 00:18:11.309593 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-w8grm" event={"ID":"e7215f9f-4c75-4ab5-85c7-e10ab231e916","Type":"ContainerStarted","Data":"38af832f71059ff4cb7daa198f3f6956a45e98c50e37b2334d0597aa74aae982"} Jul 30 00:18:11.309665 ip-10-0-142-6 kubenswrapper[2564]: I0730 00:18:11.309641 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-w8grm" event={"ID":"e7215f9f-4c75-4ab5-85c7-e10ab231e916","Type":"ContainerStarted","Data":"f14adfd9275ddea85a8e8156e40da6e2a7b47af7c9b3d705ce4c5c94146ee134"} Jul 30 00:18:11.309770 ip-10-0-142-6 kubenswrapper[2564]: I0730 00:18:11.309686 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-w8grm" event={"ID":"e7215f9f-4c75-4ab5-85c7-e10ab231e916","Type":"ContainerStarted","Data":"a8700057bf0487885523152969b11080b56fd1f63a78dcd850517631656241f2"} Jul 30 00:18:11.309770 ip-10-0-142-6 kubenswrapper[2564]: I0730 00:18:11.309734 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-w8grm" event={"ID":"e7215f9f-4c75-4ab5-85c7-e10ab231e916","Type":"ContainerStarted","Data":"a1da32f57a8b39d54768e36ea8fdc772255e30dd92bbce439c964cf84ce789f2"} Jul 30 00:18:11.309868 ip-10-0-142-6 kubenswrapper[2564]: I0730 00:18:11.309784 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-w8grm" event={"ID":"e7215f9f-4c75-4ab5-85c7-e10ab231e916","Type":"ContainerStarted","Data":"ce400ac0d546d96c9ee2c6499ecae2754c5eb38809a54ccf9cb0044f53caa78c"} Jul 30 00:18:11.311087 ip-10-0-142-6 kubenswrapper[2564]: I0730 00:18:11.311047 2564 generic.go:356] "Generic (PLEG): container finished" podID="b1186f7d-220d-4210-a7f8-6f3f0cf554e8" containerID="cdaa992df2cda42415ad00648e718df82b86a7a3c9bbb8c4fb4bedfadd026eb4" exitCode=0 Jul 30 00:18:11.311192 ip-10-0-142-6 kubenswrapper[2564]: I0730 00:18:11.311112 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-c6sdp" event={"ID":"b1186f7d-220d-4210-a7f8-6f3f0cf554e8","Type":"ContainerDied","Data":"cdaa992df2cda42415ad00648e718df82b86a7a3c9bbb8c4fb4bedfadd026eb4"} Jul 30 00:18:11.312604 ip-10-0-142-6 kubenswrapper[2564]: I0730 00:18:11.312562 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-6ssnm" event={"ID":"99deae04-e4b4-4491-a31f-945927bd1e72","Type":"ContainerStarted","Data":"450d916861aad5df8a7efa128b37401f0481ee7e180dd2abc8be3bab08fbff5c"} Jul 30 00:18:11.314130 ip-10-0-142-6 kubenswrapper[2564]: I0730 00:18:11.314099 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6khjg" event={"ID":"3a12258d-f063-4ae8-8fec-9eb3b012df06","Type":"ContainerStarted","Data":"3601df0adacb66071f619ddd72b671f770c5e7751ed9a0ed22a129ec6fde27ee"} Jul 30 00:18:11.315593 ip-10-0-142-6 kubenswrapper[2564]: I0730 00:18:11.315571 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-t69rb" event={"ID":"853c653e-d075-4220-ae9c-86b76caa2229","Type":"ContainerStarted","Data":"bf46b3980de77f4c9070f151dff2720522c6ded1f0ce8134020d88ce0bb45980"} Jul 30 00:18:11.318154 ip-10-0-142-6 kubenswrapper[2564]: I0730 00:18:11.318058 2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-142-6.ec2.internal" podStartSLOduration=21.318045463 podStartE2EDuration="21.318045463s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-30 00:17:50 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-30 00:17:54.282695623 +0000 UTC m=+5.697435557" watchObservedRunningTime="2026-07-30 00:18:11.318045463 +0000 UTC m=+22.732785395" Jul 30 00:18:11.334733 ip-10-0-142-6 kubenswrapper[2564]: I0730 00:18:11.334647 2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-tphm4" podStartSLOduration=3.871088909 podStartE2EDuration="22.334629134s" totalImagesPullingTime="18.463540225s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-30 00:17:49 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-30 00:18:11.334602628 +0000 UTC m=+22.749342561" watchObservedRunningTime="2026-07-30 00:18:11.334629134 +0000 UTC m=+22.749369045" Jul 30 00:18:11.335170 ip-10-0-142-6 kubenswrapper[2564]: I0730 00:18:11.335143 2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/konnectivity-agent-thhcs" podStartSLOduration=8.057715843 podStartE2EDuration="22.335135517s" totalImagesPullingTime="14.277419674s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-30 00:17:49 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-30 00:18:11.318035275 +0000 UTC m=+22.732775207" watchObservedRunningTime="2026-07-30 00:18:11.335135517 +0000 UTC m=+22.749875509" Jul 30 00:18:11.353162 ip-10-0-142-6 kubenswrapper[2564]: I0730 00:18:11.353109 2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-node-tuning-operator/tuned-t69rb" podStartSLOduration=3.88738634 podStartE2EDuration="22.353090963s" totalImagesPullingTime="18.465704623s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-30 00:17:49 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-30 00:18:11.352813139 +0000 UTC m=+22.767553072" watchObservedRunningTime="2026-07-30 00:18:11.353090963 +0000 UTC m=+22.767830889" Jul 30 00:18:11.390630 ip-10-0-142-6 kubenswrapper[2564]: I0730 00:18:11.390570 2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/node-resolver-6ssnm" podStartSLOduration=3.97828521 podStartE2EDuration="22.390550552s" totalImagesPullingTime="18.412265342s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-30 00:17:49 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-30 00:18:11.389812495 +0000 UTC m=+22.804552431" watchObservedRunningTime="2026-07-30 00:18:11.390550552 +0000 UTC m=+22.805290485" Jul 30 00:18:11.403881 ip-10-0-142-6 kubenswrapper[2564]: I0730 00:18:11.403829 2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/node-ca-jc8px" podStartSLOduration=11.788683005 podStartE2EDuration="22.403816496s" totalImagesPullingTime="10.615133491s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-30 00:17:49 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-30 00:18:11.403596285 +0000 UTC m=+22.818336219" watchObservedRunningTime="2026-07-30 00:18:11.403816496 +0000 UTC m=+22.818556450" Jul 30 00:18:12.133367 ip-10-0-142-6 kubenswrapper[2564]: I0730 00:18:12.133343 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-j4ckt" Jul 30 00:18:12.133552 ip-10-0-142-6 kubenswrapper[2564]: E0730 00:18:12.133449 2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-j4ckt" podUID="a0bef212-99cc-47c3-b1da-316f26f776b9" Jul 30 00:18:12.141590 ip-10-0-142-6 kubenswrapper[2564]: I0730 00:18:12.141565 2564 plugin_watcher.go:194] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock" Jul 30 00:18:12.316936 ip-10-0-142-6 kubenswrapper[2564]: I0730 00:18:12.316848 2564 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kube-system/konnectivity-agent-thhcs" Jul 30 00:18:12.317964 ip-10-0-142-6 kubenswrapper[2564]: I0730 00:18:12.317944 2564 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kube-system/konnectivity-agent-thhcs" Jul 30 00:18:12.320198 ip-10-0-142-6 kubenswrapper[2564]: I0730 00:18:12.320167 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-k9l5m" event={"ID":"dcc57a98-213c-4ed7-8b59-c0dd84a39b36","Type":"ContainerStarted","Data":"cb2be6f1d26f87741f5ec8ed136a68640fcc6d52076d6a2300f77872815f4e25"} Jul 30 00:18:12.322173 ip-10-0-142-6 kubenswrapper[2564]: I0730 00:18:12.322139 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6khjg" event={"ID":"3a12258d-f063-4ae8-8fec-9eb3b012df06","Type":"ContainerStarted","Data":"eabffc1b75d852032725df011c117e2489cb3ce86e125f91c1ebb3e19e744eb6"} Jul 30 00:18:12.322633 ip-10-0-142-6 kubenswrapper[2564]: I0730 00:18:12.322604 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kube-system/konnectivity-agent-thhcs" Jul 30 00:18:12.322797 ip-10-0-142-6 kubenswrapper[2564]: I0730 00:18:12.322780 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-thhcs" Jul 30 00:18:12.363558 ip-10-0-142-6 kubenswrapper[2564]: I0730 00:18:12.363497 2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-operator/iptables-alerter-k9l5m" podStartSLOduration=4.9526843320000005 podStartE2EDuration="23.363474996s" totalImagesPullingTime="18.410790664s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-30 00:17:49 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-30 00:18:12.363425981 +0000 UTC m=+23.778165916" watchObservedRunningTime="2026-07-30 00:18:12.363474996 +0000 UTC m=+23.778214921" Jul 30 00:18:13.019090 ip-10-0-142-6 kubenswrapper[2564]: I0730 00:18:13.018962 2564 reconciler.go:161] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-07-30T00:18:12.141584865Z","UUID":"5453759d-e8b4-48d9-bbd7-f385b6879bbe","Handler":null,"Name":"","Endpoint":""} Jul 30 00:18:13.020847 ip-10-0-142-6 kubenswrapper[2564]: I0730 00:18:13.020758 2564 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0 Jul 30 00:18:13.020847 ip-10-0-142-6 kubenswrapper[2564]: I0730 00:18:13.020802 2564 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock Jul 30 00:18:13.133529 ip-10-0-142-6 kubenswrapper[2564]: I0730 00:18:13.133476 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-d6phj" Jul 30 00:18:13.133708 ip-10-0-142-6 kubenswrapper[2564]: I0730 00:18:13.133476 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-5skn2" Jul 30 00:18:13.133708 ip-10-0-142-6 kubenswrapper[2564]: E0730 00:18:13.133630 2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-d6phj" podUID="81450593-06b9-4f5a-81da-6e5fe67679f1" Jul 30 00:18:13.133809 ip-10-0-142-6 kubenswrapper[2564]: E0730 00:18:13.133732 2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-5skn2" podUID="f82506d3-a3ca-415e-b6c0-4de2f2bf35f5" Jul 30 00:18:14.134265 ip-10-0-142-6 kubenswrapper[2564]: I0730 00:18:14.134236 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-j4ckt" Jul 30 00:18:14.134787 ip-10-0-142-6 kubenswrapper[2564]: E0730 00:18:14.134348 2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-j4ckt" podUID="a0bef212-99cc-47c3-b1da-316f26f776b9" Jul 30 00:18:14.329664 ip-10-0-142-6 kubenswrapper[2564]: I0730 00:18:14.329631 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-w8grm" event={"ID":"e7215f9f-4c75-4ab5-85c7-e10ab231e916","Type":"ContainerStarted","Data":"d2ae9762f31815e0e6d2ed68172e32e8ee5140a720ef95957ae94f4246c34f19"} Jul 30 00:18:14.331335 ip-10-0-142-6 kubenswrapper[2564]: I0730 00:18:14.331309 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6khjg" event={"ID":"3a12258d-f063-4ae8-8fec-9eb3b012df06","Type":"ContainerStarted","Data":"2aed22f110ade7ebf36c083366deb5fc4b6088e0b3ec5cdad3dd94b80a487183"} Jul 30 00:18:14.347606 ip-10-0-142-6 kubenswrapper[2564]: I0730 00:18:14.347567 2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6khjg" podStartSLOduration=4.181654378 podStartE2EDuration="25.347556039s" totalImagesPullingTime="21.165901661s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-30 00:17:49 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-30 00:18:14.347152768 +0000 UTC m=+25.761892698" watchObservedRunningTime="2026-07-30 00:18:14.347556039 +0000 UTC m=+25.762295971" Jul 30 00:18:15.134559 ip-10-0-142-6 kubenswrapper[2564]: I0730 00:18:15.134303 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-5skn2" Jul 30 00:18:15.135011 ip-10-0-142-6 kubenswrapper[2564]: I0730 00:18:15.134356 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-d6phj" Jul 30 00:18:15.135011 ip-10-0-142-6 kubenswrapper[2564]: E0730 00:18:15.134670 2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-5skn2" podUID="f82506d3-a3ca-415e-b6c0-4de2f2bf35f5" Jul 30 00:18:15.135011 ip-10-0-142-6 kubenswrapper[2564]: E0730 00:18:15.134734 2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-d6phj" podUID="81450593-06b9-4f5a-81da-6e5fe67679f1" Jul 30 00:18:16.133688 ip-10-0-142-6 kubenswrapper[2564]: I0730 00:18:16.133652 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-j4ckt" Jul 30 00:18:16.133887 ip-10-0-142-6 kubenswrapper[2564]: E0730 00:18:16.133778 2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-j4ckt" podUID="a0bef212-99cc-47c3-b1da-316f26f776b9" Jul 30 00:18:16.340409 ip-10-0-142-6 kubenswrapper[2564]: I0730 00:18:16.340150 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-w8grm" event={"ID":"e7215f9f-4c75-4ab5-85c7-e10ab231e916","Type":"ContainerStarted","Data":"45dda3a06a6f009176c53dd9e8976541ca64a2cf8d5570844f59b6d709bfc143"} Jul 30 00:18:16.341351 ip-10-0-142-6 kubenswrapper[2564]: I0730 00:18:16.340763 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-w8grm" Jul 30 00:18:16.341351 ip-10-0-142-6 kubenswrapper[2564]: I0730 00:18:16.340875 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-w8grm" Jul 30 00:18:16.341351 ip-10-0-142-6 kubenswrapper[2564]: I0730 00:18:16.341004 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-w8grm" Jul 30 00:18:16.343506 ip-10-0-142-6 kubenswrapper[2564]: I0730 00:18:16.343478 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-c6sdp" event={"ID":"b1186f7d-220d-4210-a7f8-6f3f0cf554e8","Type":"ContainerStarted","Data":"9f61a036bd78baa4c8bc1d9092796982d13b6bcaa80e786b0f7d09675b1c78d4"} Jul 30 00:18:16.357038 ip-10-0-142-6 kubenswrapper[2564]: I0730 00:18:16.357014 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-w8grm" Jul 30 00:18:16.360531 ip-10-0-142-6 kubenswrapper[2564]: I0730 00:18:16.360509 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-w8grm" Jul 30 00:18:16.366719 ip-10-0-142-6 kubenswrapper[2564]: I0730 00:18:16.366679 2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ovn-kubernetes/ovnkube-node-w8grm" podStartSLOduration=8.87930592 podStartE2EDuration="27.366666205s" totalImagesPullingTime="18.487360285s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-30 00:17:49 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-30 00:18:16.365240085 +0000 UTC m=+27.779980018" watchObservedRunningTime="2026-07-30 00:18:16.366666205 +0000 UTC m=+27.781406138" Jul 30 00:18:17.133882 ip-10-0-142-6 kubenswrapper[2564]: I0730 00:18:17.133842 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-5skn2" Jul 30 00:18:17.134105 ip-10-0-142-6 kubenswrapper[2564]: E0730 00:18:17.134003 2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-5skn2" podUID="f82506d3-a3ca-415e-b6c0-4de2f2bf35f5" Jul 30 00:18:17.134105 ip-10-0-142-6 kubenswrapper[2564]: I0730 00:18:17.134062 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-d6phj" Jul 30 00:18:17.134197 ip-10-0-142-6 kubenswrapper[2564]: E0730 00:18:17.134170 2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-d6phj" podUID="81450593-06b9-4f5a-81da-6e5fe67679f1" Jul 30 00:18:17.347408 ip-10-0-142-6 kubenswrapper[2564]: I0730 00:18:17.347330 2564 generic.go:356] "Generic (PLEG): container finished" podID="b1186f7d-220d-4210-a7f8-6f3f0cf554e8" containerID="9f61a036bd78baa4c8bc1d9092796982d13b6bcaa80e786b0f7d09675b1c78d4" exitCode=0 Jul 30 00:18:17.347756 ip-10-0-142-6 kubenswrapper[2564]: I0730 00:18:17.347414 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-c6sdp" event={"ID":"b1186f7d-220d-4210-a7f8-6f3f0cf554e8","Type":"ContainerDied","Data":"9f61a036bd78baa4c8bc1d9092796982d13b6bcaa80e786b0f7d09675b1c78d4"} Jul 30 00:18:18.133414 ip-10-0-142-6 kubenswrapper[2564]: I0730 00:18:18.133376 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-j4ckt" Jul 30 00:18:18.133635 ip-10-0-142-6 kubenswrapper[2564]: E0730 00:18:18.133484 2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-j4ckt" podUID="a0bef212-99cc-47c3-b1da-316f26f776b9" Jul 30 00:18:19.134842 ip-10-0-142-6 kubenswrapper[2564]: I0730 00:18:19.134807 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-d6phj" Jul 30 00:18:19.135258 ip-10-0-142-6 kubenswrapper[2564]: I0730 00:18:19.134876 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-5skn2" Jul 30 00:18:19.135258 ip-10-0-142-6 kubenswrapper[2564]: E0730 00:18:19.134891 2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-d6phj" podUID="81450593-06b9-4f5a-81da-6e5fe67679f1" Jul 30 00:18:19.135258 ip-10-0-142-6 kubenswrapper[2564]: E0730 00:18:19.135000 2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-5skn2" podUID="f82506d3-a3ca-415e-b6c0-4de2f2bf35f5" Jul 30 00:18:19.352835 ip-10-0-142-6 kubenswrapper[2564]: I0730 00:18:19.352801 2564 generic.go:356] "Generic (PLEG): container finished" podID="b1186f7d-220d-4210-a7f8-6f3f0cf554e8" containerID="c7b3b0999bc9c3318b546fff05e647290d4c3a63c69f21fa086b1f28ae7ac944" exitCode=0 Jul 30 00:18:19.352993 ip-10-0-142-6 kubenswrapper[2564]: I0730 00:18:19.352844 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-c6sdp" event={"ID":"b1186f7d-220d-4210-a7f8-6f3f0cf554e8","Type":"ContainerDied","Data":"c7b3b0999bc9c3318b546fff05e647290d4c3a63c69f21fa086b1f28ae7ac944"} Jul 30 00:18:20.133694 ip-10-0-142-6 kubenswrapper[2564]: I0730 00:18:20.133654 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-j4ckt" Jul 30 00:18:20.133862 ip-10-0-142-6 kubenswrapper[2564]: E0730 00:18:20.133760 2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-j4ckt" podUID="a0bef212-99cc-47c3-b1da-316f26f776b9" Jul 30 00:18:21.133797 ip-10-0-142-6 kubenswrapper[2564]: I0730 00:18:21.133765 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-d6phj" Jul 30 00:18:21.134230 ip-10-0-142-6 kubenswrapper[2564]: I0730 00:18:21.133858 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-5skn2" Jul 30 00:18:21.134230 ip-10-0-142-6 kubenswrapper[2564]: E0730 00:18:21.133870 2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-d6phj" podUID="81450593-06b9-4f5a-81da-6e5fe67679f1" Jul 30 00:18:21.134230 ip-10-0-142-6 kubenswrapper[2564]: E0730 00:18:21.133964 2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-5skn2" podUID="f82506d3-a3ca-415e-b6c0-4de2f2bf35f5" Jul 30 00:18:21.361169 ip-10-0-142-6 kubenswrapper[2564]: I0730 00:18:21.361132 2564 generic.go:356] "Generic (PLEG): container finished" podID="b1186f7d-220d-4210-a7f8-6f3f0cf554e8" containerID="f60bb73ad14a4c81d3a7cd2971cb516168a7bb2f873511247d00c51663fc3bbd" exitCode=0 Jul 30 00:18:21.361317 ip-10-0-142-6 kubenswrapper[2564]: I0730 00:18:21.361199 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-c6sdp" event={"ID":"b1186f7d-220d-4210-a7f8-6f3f0cf554e8","Type":"ContainerDied","Data":"f60bb73ad14a4c81d3a7cd2971cb516168a7bb2f873511247d00c51663fc3bbd"} Jul 30 00:18:22.133335 ip-10-0-142-6 kubenswrapper[2564]: I0730 00:18:22.133304 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-j4ckt" Jul 30 00:18:22.133523 ip-10-0-142-6 kubenswrapper[2564]: E0730 00:18:22.133429 2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-j4ckt" podUID="a0bef212-99cc-47c3-b1da-316f26f776b9" Jul 30 00:18:22.709282 ip-10-0-142-6 kubenswrapper[2564]: I0730 00:18:22.709252 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5v98n\" (UniqueName: \"kubernetes.io/projected/a0bef212-99cc-47c3-b1da-316f26f776b9-kube-api-access-5v98n\") pod \"network-check-target-j4ckt\" (UID: \"a0bef212-99cc-47c3-b1da-316f26f776b9\") " pod="openshift-network-diagnostics/network-check-target-j4ckt" Jul 30 00:18:22.709771 ip-10-0-142-6 kubenswrapper[2564]: I0730 00:18:22.709294 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/81450593-06b9-4f5a-81da-6e5fe67679f1-original-pull-secret\") pod \"global-pull-secret-syncer-d6phj\" (UID: \"81450593-06b9-4f5a-81da-6e5fe67679f1\") " pod="kube-system/global-pull-secret-syncer-d6phj" Jul 30 00:18:22.709771 ip-10-0-142-6 kubenswrapper[2564]: E0730 00:18:22.709411 2564 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 30 00:18:22.709771 ip-10-0-142-6 kubenswrapper[2564]: E0730 00:18:22.709431 2564 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 30 00:18:22.709771 ip-10-0-142-6 kubenswrapper[2564]: E0730 00:18:22.709457 2564 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 30 00:18:22.709771 ip-10-0-142-6 kubenswrapper[2564]: E0730 00:18:22.709471 2564 projected.go:196] Error preparing data for projected volume kube-api-access-5v98n for pod openshift-network-diagnostics/network-check-target-j4ckt: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 30 00:18:22.709771 ip-10-0-142-6 kubenswrapper[2564]: E0730 00:18:22.709481 2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/81450593-06b9-4f5a-81da-6e5fe67679f1-original-pull-secret podName:81450593-06b9-4f5a-81da-6e5fe67679f1 nodeName:}" failed. No retries permitted until 2026-07-30 00:18:54.709459369 +0000 UTC m=+66.124199283 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/81450593-06b9-4f5a-81da-6e5fe67679f1-original-pull-secret") pod "global-pull-secret-syncer-d6phj" (UID: "81450593-06b9-4f5a-81da-6e5fe67679f1") : object "kube-system"/"original-pull-secret" not registered Jul 30 00:18:22.709771 ip-10-0-142-6 kubenswrapper[2564]: E0730 00:18:22.709522 2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/a0bef212-99cc-47c3-b1da-316f26f776b9-kube-api-access-5v98n podName:a0bef212-99cc-47c3-b1da-316f26f776b9 nodeName:}" failed. No retries permitted until 2026-07-30 00:18:54.709506401 +0000 UTC m=+66.124246328 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "kube-api-access-5v98n" (UniqueName: "kubernetes.io/projected/a0bef212-99cc-47c3-b1da-316f26f776b9-kube-api-access-5v98n") pod "network-check-target-j4ckt" (UID: "a0bef212-99cc-47c3-b1da-316f26f776b9") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 30 00:18:22.810593 ip-10-0-142-6 kubenswrapper[2564]: I0730 00:18:22.810556 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f82506d3-a3ca-415e-b6c0-4de2f2bf35f5-metrics-certs\") pod \"network-metrics-daemon-5skn2\" (UID: \"f82506d3-a3ca-415e-b6c0-4de2f2bf35f5\") " pod="openshift-multus/network-metrics-daemon-5skn2" Jul 30 00:18:22.810792 ip-10-0-142-6 kubenswrapper[2564]: E0730 00:18:22.810688 2564 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 30 00:18:22.810792 ip-10-0-142-6 kubenswrapper[2564]: E0730 00:18:22.810758 2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f82506d3-a3ca-415e-b6c0-4de2f2bf35f5-metrics-certs podName:f82506d3-a3ca-415e-b6c0-4de2f2bf35f5 nodeName:}" failed. No retries permitted until 2026-07-30 00:18:54.810738586 +0000 UTC m=+66.225478511 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/f82506d3-a3ca-415e-b6c0-4de2f2bf35f5-metrics-certs") pod "network-metrics-daemon-5skn2" (UID: "f82506d3-a3ca-415e-b6c0-4de2f2bf35f5") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 30 00:18:23.134031 ip-10-0-142-6 kubenswrapper[2564]: I0730 00:18:23.133994 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-5skn2" Jul 30 00:18:23.134226 ip-10-0-142-6 kubenswrapper[2564]: E0730 00:18:23.134156 2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-5skn2" podUID="f82506d3-a3ca-415e-b6c0-4de2f2bf35f5" Jul 30 00:18:23.134226 ip-10-0-142-6 kubenswrapper[2564]: I0730 00:18:23.134196 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-d6phj" Jul 30 00:18:23.134349 ip-10-0-142-6 kubenswrapper[2564]: E0730 00:18:23.134307 2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-d6phj" podUID="81450593-06b9-4f5a-81da-6e5fe67679f1" Jul 30 00:18:24.133518 ip-10-0-142-6 kubenswrapper[2564]: I0730 00:18:24.133483 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-j4ckt" Jul 30 00:18:24.133956 ip-10-0-142-6 kubenswrapper[2564]: E0730 00:18:24.133616 2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-j4ckt" podUID="a0bef212-99cc-47c3-b1da-316f26f776b9" Jul 30 00:18:25.133890 ip-10-0-142-6 kubenswrapper[2564]: I0730 00:18:25.133848 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-d6phj" Jul 30 00:18:25.134349 ip-10-0-142-6 kubenswrapper[2564]: I0730 00:18:25.133861 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-5skn2" Jul 30 00:18:25.134349 ip-10-0-142-6 kubenswrapper[2564]: E0730 00:18:25.133979 2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-d6phj" podUID="81450593-06b9-4f5a-81da-6e5fe67679f1" Jul 30 00:18:25.134349 ip-10-0-142-6 kubenswrapper[2564]: E0730 00:18:25.134065 2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-5skn2" podUID="f82506d3-a3ca-415e-b6c0-4de2f2bf35f5" Jul 30 00:18:25.970203 ip-10-0-142-6 kubenswrapper[2564]: I0730 00:18:25.969868 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-d6phj"] Jul 30 00:18:25.970203 ip-10-0-142-6 kubenswrapper[2564]: I0730 00:18:25.970188 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-j4ckt"] Jul 30 00:18:25.970441 ip-10-0-142-6 kubenswrapper[2564]: I0730 00:18:25.970286 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-j4ckt" Jul 30 00:18:25.970441 ip-10-0-142-6 kubenswrapper[2564]: I0730 00:18:25.970289 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-d6phj" Jul 30 00:18:25.970441 ip-10-0-142-6 kubenswrapper[2564]: I0730 00:18:25.970389 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-5skn2"] Jul 30 00:18:25.970441 ip-10-0-142-6 kubenswrapper[2564]: E0730 00:18:25.970399 2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-j4ckt" podUID="a0bef212-99cc-47c3-b1da-316f26f776b9" Jul 30 00:18:25.970634 ip-10-0-142-6 kubenswrapper[2564]: E0730 00:18:25.970536 2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-d6phj" podUID="81450593-06b9-4f5a-81da-6e5fe67679f1" Jul 30 00:18:25.970634 ip-10-0-142-6 kubenswrapper[2564]: I0730 00:18:25.970575 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-5skn2" Jul 30 00:18:25.970732 ip-10-0-142-6 kubenswrapper[2564]: E0730 00:18:25.970671 2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-5skn2" podUID="f82506d3-a3ca-415e-b6c0-4de2f2bf35f5" Jul 30 00:18:26.377263 ip-10-0-142-6 kubenswrapper[2564]: I0730 00:18:26.377233 2564 generic.go:356] "Generic (PLEG): container finished" podID="b1186f7d-220d-4210-a7f8-6f3f0cf554e8" containerID="c88e5fcd61a07d741c5da581e2a4a61f8f253b49db18ed142b5a0c3b47d525b8" exitCode=0 Jul 30 00:18:26.377753 ip-10-0-142-6 kubenswrapper[2564]: I0730 00:18:26.377319 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-c6sdp" event={"ID":"b1186f7d-220d-4210-a7f8-6f3f0cf554e8","Type":"ContainerDied","Data":"c88e5fcd61a07d741c5da581e2a4a61f8f253b49db18ed142b5a0c3b47d525b8"} Jul 30 00:18:27.133749 ip-10-0-142-6 kubenswrapper[2564]: I0730 00:18:27.133724 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-j4ckt" Jul 30 00:18:27.133749 ip-10-0-142-6 kubenswrapper[2564]: I0730 00:18:27.133736 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-5skn2" Jul 30 00:18:27.133960 ip-10-0-142-6 kubenswrapper[2564]: E0730 00:18:27.133824 2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-j4ckt" podUID="a0bef212-99cc-47c3-b1da-316f26f776b9" Jul 30 00:18:27.133960 ip-10-0-142-6 kubenswrapper[2564]: E0730 00:18:27.133889 2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-5skn2" podUID="f82506d3-a3ca-415e-b6c0-4de2f2bf35f5" Jul 30 00:18:27.382089 ip-10-0-142-6 kubenswrapper[2564]: I0730 00:18:27.382040 2564 generic.go:356] "Generic (PLEG): container finished" podID="b1186f7d-220d-4210-a7f8-6f3f0cf554e8" containerID="a49fe5f821e16311646e57c28ee1a643558cf27ef6e9f438d9a5bc59dec07920" exitCode=0 Jul 30 00:18:27.382656 ip-10-0-142-6 kubenswrapper[2564]: I0730 00:18:27.382106 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-c6sdp" event={"ID":"b1186f7d-220d-4210-a7f8-6f3f0cf554e8","Type":"ContainerDied","Data":"a49fe5f821e16311646e57c28ee1a643558cf27ef6e9f438d9a5bc59dec07920"} Jul 30 00:18:28.133701 ip-10-0-142-6 kubenswrapper[2564]: I0730 00:18:28.133667 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-d6phj" Jul 30 00:18:28.133862 ip-10-0-142-6 kubenswrapper[2564]: E0730 00:18:28.133778 2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-d6phj" podUID="81450593-06b9-4f5a-81da-6e5fe67679f1" Jul 30 00:18:28.387266 ip-10-0-142-6 kubenswrapper[2564]: I0730 00:18:28.387229 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-c6sdp" event={"ID":"b1186f7d-220d-4210-a7f8-6f3f0cf554e8","Type":"ContainerStarted","Data":"b594ef2acd978b0442a6330d69a1a29afea54244b6d0ff347991252e2bff1fb6"} Jul 30 00:18:28.409247 ip-10-0-142-6 kubenswrapper[2564]: I0730 00:18:28.409197 2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-additional-cni-plugins-c6sdp" podStartSLOduration=9.563063671 podStartE2EDuration="39.409182319s" totalImagesPullingTime="29.846118648s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-30 00:17:49 +0000 UTC" imagePullSessionsCount=5 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-30 00:18:28.408729052 +0000 UTC m=+39.823468986" watchObservedRunningTime="2026-07-30 00:18:28.409182319 +0000 UTC m=+39.823922288" Jul 30 00:18:29.134653 ip-10-0-142-6 kubenswrapper[2564]: I0730 00:18:29.134622 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-j4ckt" Jul 30 00:18:29.134745 ip-10-0-142-6 kubenswrapper[2564]: E0730 00:18:29.134727 2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-j4ckt" podUID="a0bef212-99cc-47c3-b1da-316f26f776b9" Jul 30 00:18:29.134801 ip-10-0-142-6 kubenswrapper[2564]: I0730 00:18:29.134776 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-5skn2" Jul 30 00:18:29.134892 ip-10-0-142-6 kubenswrapper[2564]: E0730 00:18:29.134875 2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-5skn2" podUID="f82506d3-a3ca-415e-b6c0-4de2f2bf35f5" Jul 30 00:18:29.454130 ip-10-0-142-6 kubenswrapper[2564]: I0730 00:18:29.454102 2564 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-142-6.ec2.internal" event="NodeReady" Jul 30 00:18:29.454512 ip-10-0-142-6 kubenswrapper[2564]: I0730 00:18:29.454240 2564 kubelet_node_status.go:489] "Fast updating node status as it just became ready" Jul 30 00:18:29.485535 ip-10-0-142-6 kubenswrapper[2564]: I0730 00:18:29.485505 2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-image-registry/image-registry-6477b9596b-4thrq"] Jul 30 00:18:29.506237 ip-10-0-142-6 kubenswrapper[2564]: I0730 00:18:29.506209 2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-runtime-extractor-cpgjx"] Jul 30 00:18:29.506381 ip-10-0-142-6 kubenswrapper[2564]: I0730 00:18:29.506336 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-6477b9596b-4thrq" Jul 30 00:18:29.510922 ip-10-0-142-6 kubenswrapper[2564]: I0730 00:18:29.510783 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-private-configuration\"" Jul 30 00:18:29.511069 ip-10-0-142-6 kubenswrapper[2564]: I0730 00:18:29.510937 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"installation-pull-secrets\"" Jul 30 00:18:29.511069 ip-10-0-142-6 kubenswrapper[2564]: I0730 00:18:29.510986 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-tls\"" Jul 30 00:18:29.511211 ip-10-0-142-6 kubenswrapper[2564]: I0730 00:18:29.511112 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"registry-dockercfg-s7c9v\"" Jul 30 00:18:29.515986 ip-10-0-142-6 kubenswrapper[2564]: I0730 00:18:29.515967 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"trusted-ca\"" Jul 30 00:18:29.524675 ip-10-0-142-6 kubenswrapper[2564]: I0730 00:18:29.524651 2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-dns/dns-default-8lwzf"] Jul 30 00:18:29.524800 ip-10-0-142-6 kubenswrapper[2564]: I0730 00:18:29.524763 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-cpgjx" Jul 30 00:18:29.527492 ip-10-0-142-6 kubenswrapper[2564]: I0730 00:18:29.527474 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-mbp9q\"" Jul 30 00:18:29.527606 ip-10-0-142-6 kubenswrapper[2564]: I0730 00:18:29.527509 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\"" Jul 30 00:18:29.527606 ip-10-0-142-6 kubenswrapper[2564]: I0730 00:18:29.527538 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\"" Jul 30 00:18:29.527718 ip-10-0-142-6 kubenswrapper[2564]: I0730 00:18:29.527609 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\"" Jul 30 00:18:29.527718 ip-10-0-142-6 kubenswrapper[2564]: I0730 00:18:29.527679 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\"" Jul 30 00:18:29.542328 ip-10-0-142-6 kubenswrapper[2564]: I0730 00:18:29.542289 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-6477b9596b-4thrq"] Jul 30 00:18:29.542469 ip-10-0-142-6 kubenswrapper[2564]: I0730 00:18:29.542337 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-cpgjx"] Jul 30 00:18:29.542469 ip-10-0-142-6 kubenswrapper[2564]: I0730 00:18:29.542370 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-8lwzf"] Jul 30 00:18:29.542559 ip-10-0-142-6 kubenswrapper[2564]: I0730 00:18:29.542480 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-8lwzf" Jul 30 00:18:29.544825 ip-10-0-142-6 kubenswrapper[2564]: I0730 00:18:29.544806 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\"" Jul 30 00:18:29.544956 ip-10-0-142-6 kubenswrapper[2564]: I0730 00:18:29.544941 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-twxtz\"" Jul 30 00:18:29.545007 ip-10-0-142-6 kubenswrapper[2564]: I0730 00:18:29.544941 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\"" Jul 30 00:18:29.599293 ip-10-0-142-6 kubenswrapper[2564]: I0730 00:18:29.599261 2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress-canary/ingress-canary-lh7nk"] Jul 30 00:18:29.618294 ip-10-0-142-6 kubenswrapper[2564]: I0730 00:18:29.618270 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-lh7nk"] Jul 30 00:18:29.618435 ip-10-0-142-6 kubenswrapper[2564]: I0730 00:18:29.618360 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-lh7nk" Jul 30 00:18:29.620604 ip-10-0-142-6 kubenswrapper[2564]: I0730 00:18:29.620583 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\"" Jul 30 00:18:29.620722 ip-10-0-142-6 kubenswrapper[2564]: I0730 00:18:29.620658 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"default-dockercfg-qvszd\"" Jul 30 00:18:29.620776 ip-10-0-142-6 kubenswrapper[2564]: I0730 00:18:29.620754 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\"" Jul 30 00:18:29.620919 ip-10-0-142-6 kubenswrapper[2564]: I0730 00:18:29.620904 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\"" Jul 30 00:18:29.661381 ip-10-0-142-6 kubenswrapper[2564]: I0730 00:18:29.661349 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vzjwz\" (UniqueName: \"kubernetes.io/projected/45648249-3d11-4938-8d3b-a28fbf301f7e-kube-api-access-vzjwz\") pod \"dns-default-8lwzf\" (UID: \"45648249-3d11-4938-8d3b-a28fbf301f7e\") " pod="openshift-dns/dns-default-8lwzf" Jul 30 00:18:29.661381 ip-10-0-142-6 kubenswrapper[2564]: I0730 00:18:29.661386 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qc2fh\" (UniqueName: \"kubernetes.io/projected/7b14c865-5a91-41ac-81a5-869011355781-kube-api-access-qc2fh\") pod \"insights-runtime-extractor-cpgjx\" (UID: \"7b14c865-5a91-41ac-81a5-869011355781\") " pod="openshift-insights/insights-runtime-extractor-cpgjx" Jul 30 00:18:29.661575 ip-10-0-142-6 kubenswrapper[2564]: I0730 00:18:29.661409 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/7b14c865-5a91-41ac-81a5-869011355781-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-cpgjx\" (UID: \"7b14c865-5a91-41ac-81a5-869011355781\") " pod="openshift-insights/insights-runtime-extractor-cpgjx" Jul 30 00:18:29.661575 ip-10-0-142-6 kubenswrapper[2564]: I0730 00:18:29.661459 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/7b14c865-5a91-41ac-81a5-869011355781-crio-socket\") pod \"insights-runtime-extractor-cpgjx\" (UID: \"7b14c865-5a91-41ac-81a5-869011355781\") " pod="openshift-insights/insights-runtime-extractor-cpgjx" Jul 30 00:18:29.661575 ip-10-0-142-6 kubenswrapper[2564]: I0730 00:18:29.661494 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/7b14c865-5a91-41ac-81a5-869011355781-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-cpgjx\" (UID: \"7b14c865-5a91-41ac-81a5-869011355781\") " pod="openshift-insights/insights-runtime-extractor-cpgjx" Jul 30 00:18:29.661575 ip-10-0-142-6 kubenswrapper[2564]: I0730 00:18:29.661516 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/2af09576-ffda-4b11-8ddf-ce24de77a2d9-registry-certificates\") pod \"image-registry-6477b9596b-4thrq\" (UID: \"2af09576-ffda-4b11-8ddf-ce24de77a2d9\") " pod="openshift-image-registry/image-registry-6477b9596b-4thrq" Jul 30 00:18:29.661575 ip-10-0-142-6 kubenswrapper[2564]: I0730 00:18:29.661537 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-j5jgm\" (UniqueName: \"kubernetes.io/projected/2af09576-ffda-4b11-8ddf-ce24de77a2d9-kube-api-access-j5jgm\") pod \"image-registry-6477b9596b-4thrq\" (UID: \"2af09576-ffda-4b11-8ddf-ce24de77a2d9\") " pod="openshift-image-registry/image-registry-6477b9596b-4thrq" Jul 30 00:18:29.661575 ip-10-0-142-6 kubenswrapper[2564]: I0730 00:18:29.661556 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/45648249-3d11-4938-8d3b-a28fbf301f7e-tmp-dir\") pod \"dns-default-8lwzf\" (UID: \"45648249-3d11-4938-8d3b-a28fbf301f7e\") " pod="openshift-dns/dns-default-8lwzf" Jul 30 00:18:29.661750 ip-10-0-142-6 kubenswrapper[2564]: I0730 00:18:29.661605 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/2af09576-ffda-4b11-8ddf-ce24de77a2d9-registry-tls\") pod \"image-registry-6477b9596b-4thrq\" (UID: \"2af09576-ffda-4b11-8ddf-ce24de77a2d9\") " pod="openshift-image-registry/image-registry-6477b9596b-4thrq" Jul 30 00:18:29.661750 ip-10-0-142-6 kubenswrapper[2564]: I0730 00:18:29.661622 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/45648249-3d11-4938-8d3b-a28fbf301f7e-metrics-tls\") pod \"dns-default-8lwzf\" (UID: \"45648249-3d11-4938-8d3b-a28fbf301f7e\") " pod="openshift-dns/dns-default-8lwzf" Jul 30 00:18:29.661750 ip-10-0-142-6 kubenswrapper[2564]: I0730 00:18:29.661636 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/7b14c865-5a91-41ac-81a5-869011355781-data-volume\") pod \"insights-runtime-extractor-cpgjx\" (UID: \"7b14c865-5a91-41ac-81a5-869011355781\") " pod="openshift-insights/insights-runtime-extractor-cpgjx" Jul 30 00:18:29.661750 ip-10-0-142-6 kubenswrapper[2564]: I0730 00:18:29.661677 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/45648249-3d11-4938-8d3b-a28fbf301f7e-config-volume\") pod \"dns-default-8lwzf\" (UID: \"45648249-3d11-4938-8d3b-a28fbf301f7e\") " pod="openshift-dns/dns-default-8lwzf" Jul 30 00:18:29.661750 ip-10-0-142-6 kubenswrapper[2564]: I0730 00:18:29.661706 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/2af09576-ffda-4b11-8ddf-ce24de77a2d9-ca-trust-extracted\") pod \"image-registry-6477b9596b-4thrq\" (UID: \"2af09576-ffda-4b11-8ddf-ce24de77a2d9\") " pod="openshift-image-registry/image-registry-6477b9596b-4thrq" Jul 30 00:18:29.661750 ip-10-0-142-6 kubenswrapper[2564]: I0730 00:18:29.661723 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/2af09576-ffda-4b11-8ddf-ce24de77a2d9-trusted-ca\") pod \"image-registry-6477b9596b-4thrq\" (UID: \"2af09576-ffda-4b11-8ddf-ce24de77a2d9\") " pod="openshift-image-registry/image-registry-6477b9596b-4thrq" Jul 30 00:18:29.661916 ip-10-0-142-6 kubenswrapper[2564]: I0730 00:18:29.661751 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/2af09576-ffda-4b11-8ddf-ce24de77a2d9-image-registry-private-configuration\") pod \"image-registry-6477b9596b-4thrq\" (UID: \"2af09576-ffda-4b11-8ddf-ce24de77a2d9\") " pod="openshift-image-registry/image-registry-6477b9596b-4thrq" Jul 30 00:18:29.661916 ip-10-0-142-6 kubenswrapper[2564]: I0730 00:18:29.661767 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/2af09576-ffda-4b11-8ddf-ce24de77a2d9-bound-sa-token\") pod \"image-registry-6477b9596b-4thrq\" (UID: \"2af09576-ffda-4b11-8ddf-ce24de77a2d9\") " pod="openshift-image-registry/image-registry-6477b9596b-4thrq" Jul 30 00:18:29.661916 ip-10-0-142-6 kubenswrapper[2564]: I0730 00:18:29.661792 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/2af09576-ffda-4b11-8ddf-ce24de77a2d9-installation-pull-secrets\") pod \"image-registry-6477b9596b-4thrq\" (UID: \"2af09576-ffda-4b11-8ddf-ce24de77a2d9\") " pod="openshift-image-registry/image-registry-6477b9596b-4thrq" Jul 30 00:18:29.762741 ip-10-0-142-6 kubenswrapper[2564]: I0730 00:18:29.762659 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qc2fh\" (UniqueName: \"kubernetes.io/projected/7b14c865-5a91-41ac-81a5-869011355781-kube-api-access-qc2fh\") pod \"insights-runtime-extractor-cpgjx\" (UID: \"7b14c865-5a91-41ac-81a5-869011355781\") " pod="openshift-insights/insights-runtime-extractor-cpgjx" Jul 30 00:18:29.762741 ip-10-0-142-6 kubenswrapper[2564]: I0730 00:18:29.762696 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/7b14c865-5a91-41ac-81a5-869011355781-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-cpgjx\" (UID: \"7b14c865-5a91-41ac-81a5-869011355781\") " pod="openshift-insights/insights-runtime-extractor-cpgjx" Jul 30 00:18:29.762741 ip-10-0-142-6 kubenswrapper[2564]: I0730 00:18:29.762714 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/7b14c865-5a91-41ac-81a5-869011355781-crio-socket\") pod \"insights-runtime-extractor-cpgjx\" (UID: \"7b14c865-5a91-41ac-81a5-869011355781\") " pod="openshift-insights/insights-runtime-extractor-cpgjx" Jul 30 00:18:29.762741 ip-10-0-142-6 kubenswrapper[2564]: I0730 00:18:29.762731 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/7b14c865-5a91-41ac-81a5-869011355781-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-cpgjx\" (UID: \"7b14c865-5a91-41ac-81a5-869011355781\") " pod="openshift-insights/insights-runtime-extractor-cpgjx" Jul 30 00:18:29.762741 ip-10-0-142-6 kubenswrapper[2564]: I0730 00:18:29.762748 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/2af09576-ffda-4b11-8ddf-ce24de77a2d9-registry-certificates\") pod \"image-registry-6477b9596b-4thrq\" (UID: \"2af09576-ffda-4b11-8ddf-ce24de77a2d9\") " pod="openshift-image-registry/image-registry-6477b9596b-4thrq" Jul 30 00:18:29.763204 ip-10-0-142-6 kubenswrapper[2564]: I0730 00:18:29.762765 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j5jgm\" (UniqueName: \"kubernetes.io/projected/2af09576-ffda-4b11-8ddf-ce24de77a2d9-kube-api-access-j5jgm\") pod \"image-registry-6477b9596b-4thrq\" (UID: \"2af09576-ffda-4b11-8ddf-ce24de77a2d9\") " pod="openshift-image-registry/image-registry-6477b9596b-4thrq" Jul 30 00:18:29.763204 ip-10-0-142-6 kubenswrapper[2564]: I0730 00:18:29.762791 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/45648249-3d11-4938-8d3b-a28fbf301f7e-tmp-dir\") pod \"dns-default-8lwzf\" (UID: \"45648249-3d11-4938-8d3b-a28fbf301f7e\") " pod="openshift-dns/dns-default-8lwzf" Jul 30 00:18:29.763204 ip-10-0-142-6 kubenswrapper[2564]: I0730 00:18:29.762884 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/2af09576-ffda-4b11-8ddf-ce24de77a2d9-registry-tls\") pod \"image-registry-6477b9596b-4thrq\" (UID: \"2af09576-ffda-4b11-8ddf-ce24de77a2d9\") " pod="openshift-image-registry/image-registry-6477b9596b-4thrq" Jul 30 00:18:29.763204 ip-10-0-142-6 kubenswrapper[2564]: I0730 00:18:29.762920 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-94556\" (UniqueName: \"kubernetes.io/projected/4d21b8fa-df3a-4e77-8b66-bf45c15dbe56-kube-api-access-94556\") pod \"ingress-canary-lh7nk\" (UID: \"4d21b8fa-df3a-4e77-8b66-bf45c15dbe56\") " pod="openshift-ingress-canary/ingress-canary-lh7nk" Jul 30 00:18:29.763204 ip-10-0-142-6 kubenswrapper[2564]: I0730 00:18:29.762942 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/7b14c865-5a91-41ac-81a5-869011355781-crio-socket\") pod \"insights-runtime-extractor-cpgjx\" (UID: \"7b14c865-5a91-41ac-81a5-869011355781\") " pod="openshift-insights/insights-runtime-extractor-cpgjx" Jul 30 00:18:29.763204 ip-10-0-142-6 kubenswrapper[2564]: I0730 00:18:29.762956 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/45648249-3d11-4938-8d3b-a28fbf301f7e-metrics-tls\") pod \"dns-default-8lwzf\" (UID: \"45648249-3d11-4938-8d3b-a28fbf301f7e\") " pod="openshift-dns/dns-default-8lwzf" Jul 30 00:18:29.763204 ip-10-0-142-6 kubenswrapper[2564]: I0730 00:18:29.762987 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/7b14c865-5a91-41ac-81a5-869011355781-data-volume\") pod \"insights-runtime-extractor-cpgjx\" (UID: \"7b14c865-5a91-41ac-81a5-869011355781\") " pod="openshift-insights/insights-runtime-extractor-cpgjx" Jul 30 00:18:29.763204 ip-10-0-142-6 kubenswrapper[2564]: I0730 00:18:29.763027 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/45648249-3d11-4938-8d3b-a28fbf301f7e-config-volume\") pod \"dns-default-8lwzf\" (UID: \"45648249-3d11-4938-8d3b-a28fbf301f7e\") " pod="openshift-dns/dns-default-8lwzf" Jul 30 00:18:29.763204 ip-10-0-142-6 kubenswrapper[2564]: I0730 00:18:29.763059 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/2af09576-ffda-4b11-8ddf-ce24de77a2d9-ca-trust-extracted\") pod \"image-registry-6477b9596b-4thrq\" (UID: \"2af09576-ffda-4b11-8ddf-ce24de77a2d9\") " pod="openshift-image-registry/image-registry-6477b9596b-4thrq" Jul 30 00:18:29.763204 ip-10-0-142-6 kubenswrapper[2564]: I0730 00:18:29.763111 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/2af09576-ffda-4b11-8ddf-ce24de77a2d9-trusted-ca\") pod \"image-registry-6477b9596b-4thrq\" (UID: \"2af09576-ffda-4b11-8ddf-ce24de77a2d9\") " pod="openshift-image-registry/image-registry-6477b9596b-4thrq" Jul 30 00:18:29.763204 ip-10-0-142-6 kubenswrapper[2564]: I0730 00:18:29.763164 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/2af09576-ffda-4b11-8ddf-ce24de77a2d9-image-registry-private-configuration\") pod \"image-registry-6477b9596b-4thrq\" (UID: \"2af09576-ffda-4b11-8ddf-ce24de77a2d9\") " pod="openshift-image-registry/image-registry-6477b9596b-4thrq" Jul 30 00:18:29.763204 ip-10-0-142-6 kubenswrapper[2564]: I0730 00:18:29.763193 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/2af09576-ffda-4b11-8ddf-ce24de77a2d9-bound-sa-token\") pod \"image-registry-6477b9596b-4thrq\" (UID: \"2af09576-ffda-4b11-8ddf-ce24de77a2d9\") " pod="openshift-image-registry/image-registry-6477b9596b-4thrq" Jul 30 00:18:29.763820 ip-10-0-142-6 kubenswrapper[2564]: I0730 00:18:29.763222 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/4d21b8fa-df3a-4e77-8b66-bf45c15dbe56-cert\") pod \"ingress-canary-lh7nk\" (UID: \"4d21b8fa-df3a-4e77-8b66-bf45c15dbe56\") " pod="openshift-ingress-canary/ingress-canary-lh7nk" Jul 30 00:18:29.763820 ip-10-0-142-6 kubenswrapper[2564]: I0730 00:18:29.763252 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/2af09576-ffda-4b11-8ddf-ce24de77a2d9-installation-pull-secrets\") pod \"image-registry-6477b9596b-4thrq\" (UID: \"2af09576-ffda-4b11-8ddf-ce24de77a2d9\") " pod="openshift-image-registry/image-registry-6477b9596b-4thrq" Jul 30 00:18:29.763820 ip-10-0-142-6 kubenswrapper[2564]: I0730 00:18:29.763302 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vzjwz\" (UniqueName: \"kubernetes.io/projected/45648249-3d11-4938-8d3b-a28fbf301f7e-kube-api-access-vzjwz\") pod \"dns-default-8lwzf\" (UID: \"45648249-3d11-4938-8d3b-a28fbf301f7e\") " pod="openshift-dns/dns-default-8lwzf" Jul 30 00:18:29.763820 ip-10-0-142-6 kubenswrapper[2564]: I0730 00:18:29.763443 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/7b14c865-5a91-41ac-81a5-869011355781-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-cpgjx\" (UID: \"7b14c865-5a91-41ac-81a5-869011355781\") " pod="openshift-insights/insights-runtime-extractor-cpgjx" Jul 30 00:18:29.763820 ip-10-0-142-6 kubenswrapper[2564]: I0730 00:18:29.763223 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/45648249-3d11-4938-8d3b-a28fbf301f7e-tmp-dir\") pod \"dns-default-8lwzf\" (UID: \"45648249-3d11-4938-8d3b-a28fbf301f7e\") " pod="openshift-dns/dns-default-8lwzf" Jul 30 00:18:29.763820 ip-10-0-142-6 kubenswrapper[2564]: I0730 00:18:29.763706 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/2af09576-ffda-4b11-8ddf-ce24de77a2d9-registry-certificates\") pod \"image-registry-6477b9596b-4thrq\" (UID: \"2af09576-ffda-4b11-8ddf-ce24de77a2d9\") " pod="openshift-image-registry/image-registry-6477b9596b-4thrq" Jul 30 00:18:29.763820 ip-10-0-142-6 kubenswrapper[2564]: I0730 00:18:29.763748 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/7b14c865-5a91-41ac-81a5-869011355781-data-volume\") pod \"insights-runtime-extractor-cpgjx\" (UID: \"7b14c865-5a91-41ac-81a5-869011355781\") " pod="openshift-insights/insights-runtime-extractor-cpgjx" Jul 30 00:18:29.764108 ip-10-0-142-6 kubenswrapper[2564]: I0730 00:18:29.763848 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/2af09576-ffda-4b11-8ddf-ce24de77a2d9-ca-trust-extracted\") pod \"image-registry-6477b9596b-4thrq\" (UID: \"2af09576-ffda-4b11-8ddf-ce24de77a2d9\") " pod="openshift-image-registry/image-registry-6477b9596b-4thrq" Jul 30 00:18:29.764386 ip-10-0-142-6 kubenswrapper[2564]: I0730 00:18:29.764361 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/45648249-3d11-4938-8d3b-a28fbf301f7e-config-volume\") pod \"dns-default-8lwzf\" (UID: \"45648249-3d11-4938-8d3b-a28fbf301f7e\") " pod="openshift-dns/dns-default-8lwzf" Jul 30 00:18:29.764495 ip-10-0-142-6 kubenswrapper[2564]: I0730 00:18:29.764417 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/2af09576-ffda-4b11-8ddf-ce24de77a2d9-trusted-ca\") pod \"image-registry-6477b9596b-4thrq\" (UID: \"2af09576-ffda-4b11-8ddf-ce24de77a2d9\") " pod="openshift-image-registry/image-registry-6477b9596b-4thrq" Jul 30 00:18:29.767354 ip-10-0-142-6 kubenswrapper[2564]: I0730 00:18:29.767326 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/2af09576-ffda-4b11-8ddf-ce24de77a2d9-registry-tls\") pod \"image-registry-6477b9596b-4thrq\" (UID: \"2af09576-ffda-4b11-8ddf-ce24de77a2d9\") " pod="openshift-image-registry/image-registry-6477b9596b-4thrq" Jul 30 00:18:29.767354 ip-10-0-142-6 kubenswrapper[2564]: I0730 00:18:29.767335 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/45648249-3d11-4938-8d3b-a28fbf301f7e-metrics-tls\") pod \"dns-default-8lwzf\" (UID: \"45648249-3d11-4938-8d3b-a28fbf301f7e\") " pod="openshift-dns/dns-default-8lwzf" Jul 30 00:18:29.767354 ip-10-0-142-6 kubenswrapper[2564]: I0730 00:18:29.767326 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/2af09576-ffda-4b11-8ddf-ce24de77a2d9-installation-pull-secrets\") pod \"image-registry-6477b9596b-4thrq\" (UID: \"2af09576-ffda-4b11-8ddf-ce24de77a2d9\") " pod="openshift-image-registry/image-registry-6477b9596b-4thrq" Jul 30 00:18:29.767519 ip-10-0-142-6 kubenswrapper[2564]: I0730 00:18:29.767333 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/7b14c865-5a91-41ac-81a5-869011355781-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-cpgjx\" (UID: \"7b14c865-5a91-41ac-81a5-869011355781\") " pod="openshift-insights/insights-runtime-extractor-cpgjx" Jul 30 00:18:29.767519 ip-10-0-142-6 kubenswrapper[2564]: I0730 00:18:29.767338 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/2af09576-ffda-4b11-8ddf-ce24de77a2d9-image-registry-private-configuration\") pod \"image-registry-6477b9596b-4thrq\" (UID: \"2af09576-ffda-4b11-8ddf-ce24de77a2d9\") " pod="openshift-image-registry/image-registry-6477b9596b-4thrq" Jul 30 00:18:29.770740 ip-10-0-142-6 kubenswrapper[2564]: I0730 00:18:29.770575 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vzjwz\" (UniqueName: \"kubernetes.io/projected/45648249-3d11-4938-8d3b-a28fbf301f7e-kube-api-access-vzjwz\") pod \"dns-default-8lwzf\" (UID: \"45648249-3d11-4938-8d3b-a28fbf301f7e\") " pod="openshift-dns/dns-default-8lwzf" Jul 30 00:18:29.770838 ip-10-0-142-6 kubenswrapper[2564]: I0730 00:18:29.770637 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/2af09576-ffda-4b11-8ddf-ce24de77a2d9-bound-sa-token\") pod \"image-registry-6477b9596b-4thrq\" (UID: \"2af09576-ffda-4b11-8ddf-ce24de77a2d9\") " pod="openshift-image-registry/image-registry-6477b9596b-4thrq" Jul 30 00:18:29.771164 ip-10-0-142-6 kubenswrapper[2564]: I0730 00:18:29.771145 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qc2fh\" (UniqueName: \"kubernetes.io/projected/7b14c865-5a91-41ac-81a5-869011355781-kube-api-access-qc2fh\") pod \"insights-runtime-extractor-cpgjx\" (UID: \"7b14c865-5a91-41ac-81a5-869011355781\") " pod="openshift-insights/insights-runtime-extractor-cpgjx" Jul 30 00:18:29.771432 ip-10-0-142-6 kubenswrapper[2564]: I0730 00:18:29.771411 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-j5jgm\" (UniqueName: \"kubernetes.io/projected/2af09576-ffda-4b11-8ddf-ce24de77a2d9-kube-api-access-j5jgm\") pod \"image-registry-6477b9596b-4thrq\" (UID: \"2af09576-ffda-4b11-8ddf-ce24de77a2d9\") " pod="openshift-image-registry/image-registry-6477b9596b-4thrq" Jul 30 00:18:29.819173 ip-10-0-142-6 kubenswrapper[2564]: I0730 00:18:29.819138 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-6477b9596b-4thrq" Jul 30 00:18:29.833290 ip-10-0-142-6 kubenswrapper[2564]: I0730 00:18:29.833259 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-cpgjx" Jul 30 00:18:29.850185 ip-10-0-142-6 kubenswrapper[2564]: I0730 00:18:29.850154 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-8lwzf" Jul 30 00:18:29.864584 ip-10-0-142-6 kubenswrapper[2564]: I0730 00:18:29.864560 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-94556\" (UniqueName: \"kubernetes.io/projected/4d21b8fa-df3a-4e77-8b66-bf45c15dbe56-kube-api-access-94556\") pod \"ingress-canary-lh7nk\" (UID: \"4d21b8fa-df3a-4e77-8b66-bf45c15dbe56\") " pod="openshift-ingress-canary/ingress-canary-lh7nk" Jul 30 00:18:29.864713 ip-10-0-142-6 kubenswrapper[2564]: I0730 00:18:29.864601 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/4d21b8fa-df3a-4e77-8b66-bf45c15dbe56-cert\") pod \"ingress-canary-lh7nk\" (UID: \"4d21b8fa-df3a-4e77-8b66-bf45c15dbe56\") " pod="openshift-ingress-canary/ingress-canary-lh7nk" Jul 30 00:18:29.867311 ip-10-0-142-6 kubenswrapper[2564]: I0730 00:18:29.867282 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/4d21b8fa-df3a-4e77-8b66-bf45c15dbe56-cert\") pod \"ingress-canary-lh7nk\" (UID: \"4d21b8fa-df3a-4e77-8b66-bf45c15dbe56\") " pod="openshift-ingress-canary/ingress-canary-lh7nk" Jul 30 00:18:29.883559 ip-10-0-142-6 kubenswrapper[2564]: I0730 00:18:29.883527 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-94556\" (UniqueName: \"kubernetes.io/projected/4d21b8fa-df3a-4e77-8b66-bf45c15dbe56-kube-api-access-94556\") pod \"ingress-canary-lh7nk\" (UID: \"4d21b8fa-df3a-4e77-8b66-bf45c15dbe56\") " pod="openshift-ingress-canary/ingress-canary-lh7nk" Jul 30 00:18:29.926916 ip-10-0-142-6 kubenswrapper[2564]: I0730 00:18:29.926880 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-lh7nk" Jul 30 00:18:30.069687 ip-10-0-142-6 kubenswrapper[2564]: I0730 00:18:30.069564 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-6477b9596b-4thrq"] Jul 30 00:18:30.074583 ip-10-0-142-6 kubenswrapper[2564]: W0730 00:18:30.074558 2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2af09576_ffda_4b11_8ddf_ce24de77a2d9.slice/crio-52c38fd617474a86c6e402e6f93ea12905aa62cc8b5f8b16a672e8c8ecf7c24f WatchSource:0}: Error finding container 52c38fd617474a86c6e402e6f93ea12905aa62cc8b5f8b16a672e8c8ecf7c24f: Status 404 returned error can't find the container with id 52c38fd617474a86c6e402e6f93ea12905aa62cc8b5f8b16a672e8c8ecf7c24f Jul 30 00:18:30.085721 ip-10-0-142-6 kubenswrapper[2564]: I0730 00:18:30.085691 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-8lwzf"] Jul 30 00:18:30.089507 ip-10-0-142-6 kubenswrapper[2564]: I0730 00:18:30.089474 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-cpgjx"] Jul 30 00:18:30.091588 ip-10-0-142-6 kubenswrapper[2564]: W0730 00:18:30.091559 2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod45648249_3d11_4938_8d3b_a28fbf301f7e.slice/crio-4446759e811895301670e5200db6f559c46c4b15954b4c5370062eb2c19afc02 WatchSource:0}: Error finding container 4446759e811895301670e5200db6f559c46c4b15954b4c5370062eb2c19afc02: Status 404 returned error can't find the container with id 4446759e811895301670e5200db6f559c46c4b15954b4c5370062eb2c19afc02 Jul 30 00:18:30.092200 ip-10-0-142-6 kubenswrapper[2564]: W0730 00:18:30.092140 2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7b14c865_5a91_41ac_81a5_869011355781.slice/crio-a6fb584a1c2d9beb9a434cf37154cf6a2e3175695c89e48edae7e48c8832d852 WatchSource:0}: Error finding container a6fb584a1c2d9beb9a434cf37154cf6a2e3175695c89e48edae7e48c8832d852: Status 404 returned error can't find the container with id a6fb584a1c2d9beb9a434cf37154cf6a2e3175695c89e48edae7e48c8832d852 Jul 30 00:18:30.127171 ip-10-0-142-6 kubenswrapper[2564]: I0730 00:18:30.127146 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-lh7nk"] Jul 30 00:18:30.133501 ip-10-0-142-6 kubenswrapper[2564]: I0730 00:18:30.133479 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-d6phj" Jul 30 00:18:30.135153 ip-10-0-142-6 kubenswrapper[2564]: W0730 00:18:30.135126 2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4d21b8fa_df3a_4e77_8b66_bf45c15dbe56.slice/crio-c1a7807f92db24bfde92feadda3a48f4c7cbf4d48adc77153e36a5e7070475b6 WatchSource:0}: Error finding container c1a7807f92db24bfde92feadda3a48f4c7cbf4d48adc77153e36a5e7070475b6: Status 404 returned error can't find the container with id c1a7807f92db24bfde92feadda3a48f4c7cbf4d48adc77153e36a5e7070475b6 Jul 30 00:18:30.135725 ip-10-0-142-6 kubenswrapper[2564]: I0730 00:18:30.135700 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\"" Jul 30 00:18:30.393657 ip-10-0-142-6 kubenswrapper[2564]: I0730 00:18:30.393416 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-lh7nk" event={"ID":"4d21b8fa-df3a-4e77-8b66-bf45c15dbe56","Type":"ContainerStarted","Data":"c1a7807f92db24bfde92feadda3a48f4c7cbf4d48adc77153e36a5e7070475b6"} Jul 30 00:18:30.394587 ip-10-0-142-6 kubenswrapper[2564]: I0730 00:18:30.394562 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-8lwzf" event={"ID":"45648249-3d11-4938-8d3b-a28fbf301f7e","Type":"ContainerStarted","Data":"4446759e811895301670e5200db6f559c46c4b15954b4c5370062eb2c19afc02"} Jul 30 00:18:30.395968 ip-10-0-142-6 kubenswrapper[2564]: I0730 00:18:30.395946 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-6477b9596b-4thrq" event={"ID":"2af09576-ffda-4b11-8ddf-ce24de77a2d9","Type":"ContainerStarted","Data":"2da09541c93185dd433b9f4869f1e1d94d59538afbd495d2d1588e64276faee6"} Jul 30 00:18:30.396100 ip-10-0-142-6 kubenswrapper[2564]: I0730 00:18:30.395988 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-6477b9596b-4thrq" event={"ID":"2af09576-ffda-4b11-8ddf-ce24de77a2d9","Type":"ContainerStarted","Data":"52c38fd617474a86c6e402e6f93ea12905aa62cc8b5f8b16a672e8c8ecf7c24f"} Jul 30 00:18:30.396100 ip-10-0-142-6 kubenswrapper[2564]: I0730 00:18:30.396014 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-image-registry/image-registry-6477b9596b-4thrq" Jul 30 00:18:30.396989 ip-10-0-142-6 kubenswrapper[2564]: I0730 00:18:30.396968 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-cpgjx" event={"ID":"7b14c865-5a91-41ac-81a5-869011355781","Type":"ContainerStarted","Data":"a6fb584a1c2d9beb9a434cf37154cf6a2e3175695c89e48edae7e48c8832d852"} Jul 30 00:18:30.414205 ip-10-0-142-6 kubenswrapper[2564]: I0730 00:18:30.414156 2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/image-registry-6477b9596b-4thrq" podStartSLOduration=6.414135143 podStartE2EDuration="6.414135143s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-30 00:18:24 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-30 00:18:30.413679292 +0000 UTC m=+41.828419225" watchObservedRunningTime="2026-07-30 00:18:30.414135143 +0000 UTC m=+41.828875076" Jul 30 00:18:31.137103 ip-10-0-142-6 kubenswrapper[2564]: I0730 00:18:31.137063 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-5skn2" Jul 30 00:18:31.137625 ip-10-0-142-6 kubenswrapper[2564]: I0730 00:18:31.137607 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-j4ckt" Jul 30 00:18:31.142130 ip-10-0-142-6 kubenswrapper[2564]: I0730 00:18:31.141584 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\"" Jul 30 00:18:31.142130 ip-10-0-142-6 kubenswrapper[2564]: I0730 00:18:31.141701 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\"" Jul 30 00:18:31.142130 ip-10-0-142-6 kubenswrapper[2564]: I0730 00:18:31.141833 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-qvw7x\"" Jul 30 00:18:31.142130 ip-10-0-142-6 kubenswrapper[2564]: I0730 00:18:31.141952 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-f7pzr\"" Jul 30 00:18:31.142130 ip-10-0-142-6 kubenswrapper[2564]: I0730 00:18:31.142011 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\"" Jul 30 00:18:32.185313 ip-10-0-142-6 kubenswrapper[2564]: I0730 00:18:32.183840 2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-operator-admission-webhook-7f8d78f54b-p46zv"] Jul 30 00:18:32.204725 ip-10-0-142-6 kubenswrapper[2564]: I0730 00:18:32.204681 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-admission-webhook-7f8d78f54b-p46zv"] Jul 30 00:18:32.204887 ip-10-0-142-6 kubenswrapper[2564]: I0730 00:18:32.204805 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-admission-webhook-7f8d78f54b-p46zv" Jul 30 00:18:32.207592 ip-10-0-142-6 kubenswrapper[2564]: I0730 00:18:32.207566 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-admission-webhook-tls\"" Jul 30 00:18:32.207592 ip-10-0-142-6 kubenswrapper[2564]: I0730 00:18:32.207587 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-admission-webhook-dockercfg-l2vnw\"" Jul 30 00:18:32.389454 ip-10-0-142-6 kubenswrapper[2564]: I0730 00:18:32.389409 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/1f32de58-81d4-4989-b8c2-2ae7d765bfa4-tls-certificates\") pod \"prometheus-operator-admission-webhook-7f8d78f54b-p46zv\" (UID: \"1f32de58-81d4-4989-b8c2-2ae7d765bfa4\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-7f8d78f54b-p46zv" Jul 30 00:18:32.490119 ip-10-0-142-6 kubenswrapper[2564]: I0730 00:18:32.490023 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/1f32de58-81d4-4989-b8c2-2ae7d765bfa4-tls-certificates\") pod \"prometheus-operator-admission-webhook-7f8d78f54b-p46zv\" (UID: \"1f32de58-81d4-4989-b8c2-2ae7d765bfa4\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-7f8d78f54b-p46zv" Jul 30 00:18:32.490274 ip-10-0-142-6 kubenswrapper[2564]: E0730 00:18:32.490203 2564 secret.go:189] Couldn't get secret openshift-monitoring/prometheus-operator-admission-webhook-tls: secret "prometheus-operator-admission-webhook-tls" not found Jul 30 00:18:32.490313 ip-10-0-142-6 kubenswrapper[2564]: E0730 00:18:32.490288 2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1f32de58-81d4-4989-b8c2-2ae7d765bfa4-tls-certificates podName:1f32de58-81d4-4989-b8c2-2ae7d765bfa4 nodeName:}" failed. No retries permitted until 2026-07-30 00:18:32.990264867 +0000 UTC m=+44.405004797 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "tls-certificates" (UniqueName: "kubernetes.io/secret/1f32de58-81d4-4989-b8c2-2ae7d765bfa4-tls-certificates") pod "prometheus-operator-admission-webhook-7f8d78f54b-p46zv" (UID: "1f32de58-81d4-4989-b8c2-2ae7d765bfa4") : secret "prometheus-operator-admission-webhook-tls" not found Jul 30 00:18:32.994651 ip-10-0-142-6 kubenswrapper[2564]: I0730 00:18:32.994616 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/1f32de58-81d4-4989-b8c2-2ae7d765bfa4-tls-certificates\") pod \"prometheus-operator-admission-webhook-7f8d78f54b-p46zv\" (UID: \"1f32de58-81d4-4989-b8c2-2ae7d765bfa4\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-7f8d78f54b-p46zv" Jul 30 00:18:32.998856 ip-10-0-142-6 kubenswrapper[2564]: I0730 00:18:32.998792 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/1f32de58-81d4-4989-b8c2-2ae7d765bfa4-tls-certificates\") pod \"prometheus-operator-admission-webhook-7f8d78f54b-p46zv\" (UID: \"1f32de58-81d4-4989-b8c2-2ae7d765bfa4\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-7f8d78f54b-p46zv" Jul 30 00:18:33.116999 ip-10-0-142-6 kubenswrapper[2564]: I0730 00:18:33.116960 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-admission-webhook-7f8d78f54b-p46zv" Jul 30 00:18:33.408795 ip-10-0-142-6 kubenswrapper[2564]: I0730 00:18:33.408753 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-8lwzf" event={"ID":"45648249-3d11-4938-8d3b-a28fbf301f7e","Type":"ContainerStarted","Data":"b22f5209ed430536d2a129f45e2861806433663cf22245093373497070814c66"} Jul 30 00:18:37.138903 ip-10-0-142-6 kubenswrapper[2564]: I0730 00:18:37.138708 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-admission-webhook-7f8d78f54b-p46zv"] Jul 30 00:18:37.142259 ip-10-0-142-6 kubenswrapper[2564]: W0730 00:18:37.142213 2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1f32de58_81d4_4989_b8c2_2ae7d765bfa4.slice/crio-5b5d3f87092258db1d028ba13c3bbd6f81a481de990e89b84e3cd21ce934462c WatchSource:0}: Error finding container 5b5d3f87092258db1d028ba13c3bbd6f81a481de990e89b84e3cd21ce934462c: Status 404 returned error can't find the container with id 5b5d3f87092258db1d028ba13c3bbd6f81a481de990e89b84e3cd21ce934462c Jul 30 00:18:37.419229 ip-10-0-142-6 kubenswrapper[2564]: I0730 00:18:37.419200 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-admission-webhook-7f8d78f54b-p46zv" event={"ID":"1f32de58-81d4-4989-b8c2-2ae7d765bfa4","Type":"ContainerStarted","Data":"5b5d3f87092258db1d028ba13c3bbd6f81a481de990e89b84e3cd21ce934462c"} Jul 30 00:18:37.420579 ip-10-0-142-6 kubenswrapper[2564]: I0730 00:18:37.420554 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-lh7nk" event={"ID":"4d21b8fa-df3a-4e77-8b66-bf45c15dbe56","Type":"ContainerStarted","Data":"746c039f42a38d4e9e418c2da78ebc5e2715164c904e099d6d5f1f4c968d30fd"} Jul 30 00:18:37.422153 ip-10-0-142-6 kubenswrapper[2564]: I0730 00:18:37.422127 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-8lwzf" event={"ID":"45648249-3d11-4938-8d3b-a28fbf301f7e","Type":"ContainerStarted","Data":"f74d6f74b958952ec8d76e70d35c3fac4ec79e18cc62870d590c8fa4e9718114"} Jul 30 00:18:37.436296 ip-10-0-142-6 kubenswrapper[2564]: I0730 00:18:37.436269 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-dns/dns-default-8lwzf" Jul 30 00:18:37.451495 ip-10-0-142-6 kubenswrapper[2564]: I0730 00:18:37.451450 2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress-canary/ingress-canary-lh7nk" podStartSLOduration=1.586912362 podStartE2EDuration="8.45143339s" totalImagesPullingTime="6.864521028s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-30 00:18:29 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-30 00:18:37.450752004 +0000 UTC m=+48.865491940" watchObservedRunningTime="2026-07-30 00:18:37.45143339 +0000 UTC m=+48.866173323" Jul 30 00:18:37.467555 ip-10-0-142-6 kubenswrapper[2564]: I0730 00:18:37.467491 2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/dns-default-8lwzf" podStartSLOduration=5.948362003 podStartE2EDuration="8.46746851s" totalImagesPullingTime="2.519106507s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-30 00:18:29 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-30 00:18:37.466418219 +0000 UTC m=+48.881158151" watchObservedRunningTime="2026-07-30 00:18:37.46746851 +0000 UTC m=+48.882208442" Jul 30 00:18:38.427232 ip-10-0-142-6 kubenswrapper[2564]: I0730 00:18:38.427194 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-cpgjx" event={"ID":"7b14c865-5a91-41ac-81a5-869011355781","Type":"ContainerStarted","Data":"eb78df79de42c68abc26cf1a2e2f23f461209283aa571bec76d58dc245cc942b"} Jul 30 00:18:39.432675 ip-10-0-142-6 kubenswrapper[2564]: I0730 00:18:39.432465 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-admission-webhook-7f8d78f54b-p46zv" event={"ID":"1f32de58-81d4-4989-b8c2-2ae7d765bfa4","Type":"ContainerStarted","Data":"67ea3cab3757469b05805eacb562aedc6a15adb77a3bd2fec4a5d0fdc3800805"} Jul 30 00:18:39.433102 ip-10-0-142-6 kubenswrapper[2564]: I0730 00:18:39.432699 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/prometheus-operator-admission-webhook-7f8d78f54b-p46zv" Jul 30 00:18:39.438361 ip-10-0-142-6 kubenswrapper[2564]: I0730 00:18:39.438331 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/prometheus-operator-admission-webhook-7f8d78f54b-p46zv" Jul 30 00:18:39.447486 ip-10-0-142-6 kubenswrapper[2564]: I0730 00:18:39.447443 2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/prometheus-operator-admission-webhook-7f8d78f54b-p46zv" podStartSLOduration=5.672092221 podStartE2EDuration="7.447425644s" totalImagesPullingTime="1.775333423s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-30 00:18:32 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-30 00:18:39.446423094 +0000 UTC m=+50.861163028" watchObservedRunningTime="2026-07-30 00:18:39.447425644 +0000 UTC m=+50.862165568" Jul 30 00:18:42.445848 ip-10-0-142-6 kubenswrapper[2564]: I0730 00:18:42.445807 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-cpgjx" event={"ID":"7b14c865-5a91-41ac-81a5-869011355781","Type":"ContainerStarted","Data":"f0bfc9d356cc6fa62c887900e089053f7c191d2aa965e8330a527c00a7d45f14"} Jul 30 00:18:42.489595 ip-10-0-142-6 kubenswrapper[2564]: I0730 00:18:42.489568 2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-56d468748c-tfk4j"] Jul 30 00:18:42.496244 ip-10-0-142-6 kubenswrapper[2564]: I0730 00:18:42.496222 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-56d468748c-tfk4j" Jul 30 00:18:42.499168 ip-10-0-142-6 kubenswrapper[2564]: I0730 00:18:42.499138 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"service-ca\"" Jul 30 00:18:42.500166 ip-10-0-142-6 kubenswrapper[2564]: I0730 00:18:42.500147 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"openshift-service-ca.crt\"" Jul 30 00:18:42.500457 ip-10-0-142-6 kubenswrapper[2564]: I0730 00:18:42.500427 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"oauth-serving-cert\"" Jul 30 00:18:42.500546 ip-10-0-142-6 kubenswrapper[2564]: I0730 00:18:42.500456 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"kube-root-ca.crt\"" Jul 30 00:18:42.500546 ip-10-0-142-6 kubenswrapper[2564]: I0730 00:18:42.500531 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-dockercfg-pfswq\"" Jul 30 00:18:42.500649 ip-10-0-142-6 kubenswrapper[2564]: I0730 00:18:42.500568 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-oauth-config\"" Jul 30 00:18:42.500754 ip-10-0-142-6 kubenswrapper[2564]: I0730 00:18:42.500737 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-serving-cert\"" Jul 30 00:18:42.501702 ip-10-0-142-6 kubenswrapper[2564]: I0730 00:18:42.501686 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"console-config\"" Jul 30 00:18:42.504800 ip-10-0-142-6 kubenswrapper[2564]: I0730 00:18:42.504780 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"trusted-ca-bundle\"" Jul 30 00:18:42.507722 ip-10-0-142-6 kubenswrapper[2564]: I0730 00:18:42.507701 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-56d468748c-tfk4j"] Jul 30 00:18:42.568343 ip-10-0-142-6 kubenswrapper[2564]: I0730 00:18:42.568256 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/e3ac38c6-9513-43cf-9af0-8e49aceca4ed-console-serving-cert\") pod \"console-56d468748c-tfk4j\" (UID: \"e3ac38c6-9513-43cf-9af0-8e49aceca4ed\") " pod="openshift-console/console-56d468748c-tfk4j" Jul 30 00:18:42.568343 ip-10-0-142-6 kubenswrapper[2564]: I0730 00:18:42.568296 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ks5rb\" (UniqueName: \"kubernetes.io/projected/e3ac38c6-9513-43cf-9af0-8e49aceca4ed-kube-api-access-ks5rb\") pod \"console-56d468748c-tfk4j\" (UID: \"e3ac38c6-9513-43cf-9af0-8e49aceca4ed\") " pod="openshift-console/console-56d468748c-tfk4j" Jul 30 00:18:42.568343 ip-10-0-142-6 kubenswrapper[2564]: I0730 00:18:42.568319 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/e3ac38c6-9513-43cf-9af0-8e49aceca4ed-trusted-ca-bundle\") pod \"console-56d468748c-tfk4j\" (UID: \"e3ac38c6-9513-43cf-9af0-8e49aceca4ed\") " pod="openshift-console/console-56d468748c-tfk4j" Jul 30 00:18:42.568554 ip-10-0-142-6 kubenswrapper[2564]: I0730 00:18:42.568452 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/e3ac38c6-9513-43cf-9af0-8e49aceca4ed-console-oauth-config\") pod \"console-56d468748c-tfk4j\" (UID: \"e3ac38c6-9513-43cf-9af0-8e49aceca4ed\") " pod="openshift-console/console-56d468748c-tfk4j" Jul 30 00:18:42.568554 ip-10-0-142-6 kubenswrapper[2564]: I0730 00:18:42.568501 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/e3ac38c6-9513-43cf-9af0-8e49aceca4ed-console-config\") pod \"console-56d468748c-tfk4j\" (UID: \"e3ac38c6-9513-43cf-9af0-8e49aceca4ed\") " pod="openshift-console/console-56d468748c-tfk4j" Jul 30 00:18:42.568554 ip-10-0-142-6 kubenswrapper[2564]: I0730 00:18:42.568526 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/e3ac38c6-9513-43cf-9af0-8e49aceca4ed-oauth-serving-cert\") pod \"console-56d468748c-tfk4j\" (UID: \"e3ac38c6-9513-43cf-9af0-8e49aceca4ed\") " pod="openshift-console/console-56d468748c-tfk4j" Jul 30 00:18:42.568653 ip-10-0-142-6 kubenswrapper[2564]: I0730 00:18:42.568562 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/e3ac38c6-9513-43cf-9af0-8e49aceca4ed-service-ca\") pod \"console-56d468748c-tfk4j\" (UID: \"e3ac38c6-9513-43cf-9af0-8e49aceca4ed\") " pod="openshift-console/console-56d468748c-tfk4j" Jul 30 00:18:42.669798 ip-10-0-142-6 kubenswrapper[2564]: I0730 00:18:42.669762 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/e3ac38c6-9513-43cf-9af0-8e49aceca4ed-console-config\") pod \"console-56d468748c-tfk4j\" (UID: \"e3ac38c6-9513-43cf-9af0-8e49aceca4ed\") " pod="openshift-console/console-56d468748c-tfk4j" Jul 30 00:18:42.669798 ip-10-0-142-6 kubenswrapper[2564]: I0730 00:18:42.669801 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/e3ac38c6-9513-43cf-9af0-8e49aceca4ed-oauth-serving-cert\") pod \"console-56d468748c-tfk4j\" (UID: \"e3ac38c6-9513-43cf-9af0-8e49aceca4ed\") " pod="openshift-console/console-56d468748c-tfk4j" Jul 30 00:18:42.670027 ip-10-0-142-6 kubenswrapper[2564]: I0730 00:18:42.669818 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/e3ac38c6-9513-43cf-9af0-8e49aceca4ed-service-ca\") pod \"console-56d468748c-tfk4j\" (UID: \"e3ac38c6-9513-43cf-9af0-8e49aceca4ed\") " pod="openshift-console/console-56d468748c-tfk4j" Jul 30 00:18:42.670027 ip-10-0-142-6 kubenswrapper[2564]: I0730 00:18:42.669841 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/e3ac38c6-9513-43cf-9af0-8e49aceca4ed-console-serving-cert\") pod \"console-56d468748c-tfk4j\" (UID: \"e3ac38c6-9513-43cf-9af0-8e49aceca4ed\") " pod="openshift-console/console-56d468748c-tfk4j" Jul 30 00:18:42.670027 ip-10-0-142-6 kubenswrapper[2564]: I0730 00:18:42.669913 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ks5rb\" (UniqueName: \"kubernetes.io/projected/e3ac38c6-9513-43cf-9af0-8e49aceca4ed-kube-api-access-ks5rb\") pod \"console-56d468748c-tfk4j\" (UID: \"e3ac38c6-9513-43cf-9af0-8e49aceca4ed\") " pod="openshift-console/console-56d468748c-tfk4j" Jul 30 00:18:42.670027 ip-10-0-142-6 kubenswrapper[2564]: I0730 00:18:42.669942 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/e3ac38c6-9513-43cf-9af0-8e49aceca4ed-trusted-ca-bundle\") pod \"console-56d468748c-tfk4j\" (UID: \"e3ac38c6-9513-43cf-9af0-8e49aceca4ed\") " pod="openshift-console/console-56d468748c-tfk4j" Jul 30 00:18:42.670272 ip-10-0-142-6 kubenswrapper[2564]: I0730 00:18:42.670248 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/e3ac38c6-9513-43cf-9af0-8e49aceca4ed-console-oauth-config\") pod \"console-56d468748c-tfk4j\" (UID: \"e3ac38c6-9513-43cf-9af0-8e49aceca4ed\") " pod="openshift-console/console-56d468748c-tfk4j" Jul 30 00:18:42.670606 ip-10-0-142-6 kubenswrapper[2564]: I0730 00:18:42.670584 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/e3ac38c6-9513-43cf-9af0-8e49aceca4ed-service-ca\") pod \"console-56d468748c-tfk4j\" (UID: \"e3ac38c6-9513-43cf-9af0-8e49aceca4ed\") " pod="openshift-console/console-56d468748c-tfk4j" Jul 30 00:18:42.670692 ip-10-0-142-6 kubenswrapper[2564]: I0730 00:18:42.670608 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/e3ac38c6-9513-43cf-9af0-8e49aceca4ed-console-config\") pod \"console-56d468748c-tfk4j\" (UID: \"e3ac38c6-9513-43cf-9af0-8e49aceca4ed\") " pod="openshift-console/console-56d468748c-tfk4j" Jul 30 00:18:42.670692 ip-10-0-142-6 kubenswrapper[2564]: I0730 00:18:42.670657 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/e3ac38c6-9513-43cf-9af0-8e49aceca4ed-oauth-serving-cert\") pod \"console-56d468748c-tfk4j\" (UID: \"e3ac38c6-9513-43cf-9af0-8e49aceca4ed\") " pod="openshift-console/console-56d468748c-tfk4j" Jul 30 00:18:42.670913 ip-10-0-142-6 kubenswrapper[2564]: I0730 00:18:42.670896 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/e3ac38c6-9513-43cf-9af0-8e49aceca4ed-trusted-ca-bundle\") pod \"console-56d468748c-tfk4j\" (UID: \"e3ac38c6-9513-43cf-9af0-8e49aceca4ed\") " pod="openshift-console/console-56d468748c-tfk4j" Jul 30 00:18:42.673978 ip-10-0-142-6 kubenswrapper[2564]: I0730 00:18:42.673956 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/e3ac38c6-9513-43cf-9af0-8e49aceca4ed-console-oauth-config\") pod \"console-56d468748c-tfk4j\" (UID: \"e3ac38c6-9513-43cf-9af0-8e49aceca4ed\") " pod="openshift-console/console-56d468748c-tfk4j" Jul 30 00:18:42.674127 ip-10-0-142-6 kubenswrapper[2564]: I0730 00:18:42.674111 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/e3ac38c6-9513-43cf-9af0-8e49aceca4ed-console-serving-cert\") pod \"console-56d468748c-tfk4j\" (UID: \"e3ac38c6-9513-43cf-9af0-8e49aceca4ed\") " pod="openshift-console/console-56d468748c-tfk4j" Jul 30 00:18:42.677662 ip-10-0-142-6 kubenswrapper[2564]: I0730 00:18:42.677638 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ks5rb\" (UniqueName: \"kubernetes.io/projected/e3ac38c6-9513-43cf-9af0-8e49aceca4ed-kube-api-access-ks5rb\") pod \"console-56d468748c-tfk4j\" (UID: \"e3ac38c6-9513-43cf-9af0-8e49aceca4ed\") " pod="openshift-console/console-56d468748c-tfk4j" Jul 30 00:18:42.805996 ip-10-0-142-6 kubenswrapper[2564]: I0730 00:18:42.805959 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-56d468748c-tfk4j" Jul 30 00:18:42.943139 ip-10-0-142-6 kubenswrapper[2564]: I0730 00:18:42.942878 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-56d468748c-tfk4j"] Jul 30 00:18:42.947220 ip-10-0-142-6 kubenswrapper[2564]: W0730 00:18:42.947183 2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode3ac38c6_9513_43cf_9af0_8e49aceca4ed.slice/crio-453a93f544934590033f99ae23664366715c663e57476f888027bdb2b2b55fac WatchSource:0}: Error finding container 453a93f544934590033f99ae23664366715c663e57476f888027bdb2b2b55fac: Status 404 returned error can't find the container with id 453a93f544934590033f99ae23664366715c663e57476f888027bdb2b2b55fac Jul 30 00:18:43.451632 ip-10-0-142-6 kubenswrapper[2564]: I0730 00:18:43.451552 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-56d468748c-tfk4j" event={"ID":"e3ac38c6-9513-43cf-9af0-8e49aceca4ed","Type":"ContainerStarted","Data":"453a93f544934590033f99ae23664366715c663e57476f888027bdb2b2b55fac"} Jul 30 00:18:44.431828 ip-10-0-142-6 kubenswrapper[2564]: I0730 00:18:44.431796 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-dns/dns-default-8lwzf" Jul 30 00:18:47.470611 ip-10-0-142-6 kubenswrapper[2564]: I0730 00:18:47.470572 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-cpgjx" event={"ID":"7b14c865-5a91-41ac-81a5-869011355781","Type":"ContainerStarted","Data":"a58047fb0efba62f464711681ef2fb2d75912d99d53b62595bf1fcb72f973c46"} Jul 30 00:18:47.471994 ip-10-0-142-6 kubenswrapper[2564]: I0730 00:18:47.471969 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-56d468748c-tfk4j" event={"ID":"e3ac38c6-9513-43cf-9af0-8e49aceca4ed","Type":"ContainerStarted","Data":"ab843fb3b471f2ee3ffa98a73a9b109dbe32702c74d9348f4927e709b51aef82"} Jul 30 00:18:47.491230 ip-10-0-142-6 kubenswrapper[2564]: I0730 00:18:47.491186 2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-insights/insights-runtime-extractor-cpgjx" podStartSLOduration=1.582079853 podStartE2EDuration="18.491172118s" totalImagesPullingTime="16.909092265s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-30 00:18:29 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-30 00:18:47.48964424 +0000 UTC m=+58.904384194" watchObservedRunningTime="2026-07-30 00:18:47.491172118 +0000 UTC m=+58.905912054" Jul 30 00:18:47.507832 ip-10-0-142-6 kubenswrapper[2564]: I0730 00:18:47.507732 2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-56d468748c-tfk4j" podStartSLOduration=1.196769263 podStartE2EDuration="5.507719789s" totalImagesPullingTime="4.310950526s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-30 00:18:42 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-30 00:18:47.506764346 +0000 UTC m=+58.921504281" watchObservedRunningTime="2026-07-30 00:18:47.507719789 +0000 UTC m=+58.922459722" Jul 30 00:18:47.574989 ip-10-0-142-6 kubenswrapper[2564]: I0730 00:18:47.574955 2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/openshift-state-metrics-b8f49f5c6-kbc7k"] Jul 30 00:18:47.577058 ip-10-0-142-6 kubenswrapper[2564]: I0730 00:18:47.577037 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/openshift-state-metrics-b8f49f5c6-kbc7k" Jul 30 00:18:47.579690 ip-10-0-142-6 kubenswrapper[2564]: I0730 00:18:47.579622 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\"" Jul 30 00:18:47.580028 ip-10-0-142-6 kubenswrapper[2564]: I0730 00:18:47.579869 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-kube-rbac-proxy-config\"" Jul 30 00:18:47.580028 ip-10-0-142-6 kubenswrapper[2564]: I0730 00:18:47.579926 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\"" Jul 30 00:18:47.580028 ip-10-0-142-6 kubenswrapper[2564]: I0730 00:18:47.580014 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-tls\"" Jul 30 00:18:47.580279 ip-10-0-142-6 kubenswrapper[2564]: I0730 00:18:47.580268 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-dockercfg-ddrz4\"" Jul 30 00:18:47.580391 ip-10-0-142-6 kubenswrapper[2564]: I0730 00:18:47.580375 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\"" Jul 30 00:18:47.585646 ip-10-0-142-6 kubenswrapper[2564]: I0730 00:18:47.585623 2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/kube-state-metrics-56d9f944b4-q5l9w"] Jul 30 00:18:47.588102 ip-10-0-142-6 kubenswrapper[2564]: I0730 00:18:47.588067 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/kube-state-metrics-56d9f944b4-q5l9w" Jul 30 00:18:47.588619 ip-10-0-142-6 kubenswrapper[2564]: I0730 00:18:47.588601 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/openshift-state-metrics-b8f49f5c6-kbc7k"] Jul 30 00:18:47.590661 ip-10-0-142-6 kubenswrapper[2564]: I0730 00:18:47.590640 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-tls\"" Jul 30 00:18:47.590777 ip-10-0-142-6 kubenswrapper[2564]: I0730 00:18:47.590725 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-custom-resource-state-configmap\"" Jul 30 00:18:47.590884 ip-10-0-142-6 kubenswrapper[2564]: I0730 00:18:47.590868 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-kube-rbac-proxy-config\"" Jul 30 00:18:47.590961 ip-10-0-142-6 kubenswrapper[2564]: I0730 00:18:47.590941 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-dockercfg-fd2jq\"" Jul 30 00:18:47.604086 ip-10-0-142-6 kubenswrapper[2564]: I0730 00:18:47.604052 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/kube-state-metrics-56d9f944b4-q5l9w"] Jul 30 00:18:47.604796 ip-10-0-142-6 kubenswrapper[2564]: I0730 00:18:47.604774 2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/node-exporter-5kvcx"] Jul 30 00:18:47.607157 ip-10-0-142-6 kubenswrapper[2564]: I0730 00:18:47.607136 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-5kvcx" Jul 30 00:18:47.609493 ip-10-0-142-6 kubenswrapper[2564]: I0730 00:18:47.609473 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\"" Jul 30 00:18:47.609493 ip-10-0-142-6 kubenswrapper[2564]: I0730 00:18:47.609486 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-b9dj7\"" Jul 30 00:18:47.609653 ip-10-0-142-6 kubenswrapper[2564]: I0730 00:18:47.609520 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\"" Jul 30 00:18:47.609653 ip-10-0-142-6 kubenswrapper[2564]: I0730 00:18:47.609542 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\"" Jul 30 00:18:47.612417 ip-10-0-142-6 kubenswrapper[2564]: I0730 00:18:47.612393 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/80c96de1-7df7-4fdf-afc4-75fb6e1c23ca-openshift-state-metrics-tls\") pod \"openshift-state-metrics-b8f49f5c6-kbc7k\" (UID: \"80c96de1-7df7-4fdf-afc4-75fb6e1c23ca\") " pod="openshift-monitoring/openshift-state-metrics-b8f49f5c6-kbc7k" Jul 30 00:18:47.612510 ip-10-0-142-6 kubenswrapper[2564]: I0730 00:18:47.612432 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/80c96de1-7df7-4fdf-afc4-75fb6e1c23ca-metrics-client-ca\") pod \"openshift-state-metrics-b8f49f5c6-kbc7k\" (UID: \"80c96de1-7df7-4fdf-afc4-75fb6e1c23ca\") " pod="openshift-monitoring/openshift-state-metrics-b8f49f5c6-kbc7k" Jul 30 00:18:47.612510 ip-10-0-142-6 kubenswrapper[2564]: I0730 00:18:47.612471 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bj227\" (UniqueName: \"kubernetes.io/projected/80c96de1-7df7-4fdf-afc4-75fb6e1c23ca-kube-api-access-bj227\") pod \"openshift-state-metrics-b8f49f5c6-kbc7k\" (UID: \"80c96de1-7df7-4fdf-afc4-75fb6e1c23ca\") " pod="openshift-monitoring/openshift-state-metrics-b8f49f5c6-kbc7k" Jul 30 00:18:47.612589 ip-10-0-142-6 kubenswrapper[2564]: I0730 00:18:47.612513 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/80c96de1-7df7-4fdf-afc4-75fb6e1c23ca-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-b8f49f5c6-kbc7k\" (UID: \"80c96de1-7df7-4fdf-afc4-75fb6e1c23ca\") " pod="openshift-monitoring/openshift-state-metrics-b8f49f5c6-kbc7k" Jul 30 00:18:47.713317 ip-10-0-142-6 kubenswrapper[2564]: I0730 00:18:47.713278 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-89hm9\" (UniqueName: \"kubernetes.io/projected/03d300be-130c-46de-9d44-dd302016de4a-kube-api-access-89hm9\") pod \"kube-state-metrics-56d9f944b4-q5l9w\" (UID: \"03d300be-130c-46de-9d44-dd302016de4a\") " pod="openshift-monitoring/kube-state-metrics-56d9f944b4-q5l9w" Jul 30 00:18:47.713317 ip-10-0-142-6 kubenswrapper[2564]: I0730 00:18:47.713311 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/5608ae36-4348-427e-bc57-dd15c468728e-metrics-client-ca\") pod \"node-exporter-5kvcx\" (UID: \"5608ae36-4348-427e-bc57-dd15c468728e\") " pod="openshift-monitoring/node-exporter-5kvcx" Jul 30 00:18:47.713553 ip-10-0-142-6 kubenswrapper[2564]: I0730 00:18:47.713334 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/03d300be-130c-46de-9d44-dd302016de4a-metrics-client-ca\") pod \"kube-state-metrics-56d9f944b4-q5l9w\" (UID: \"03d300be-130c-46de-9d44-dd302016de4a\") " pod="openshift-monitoring/kube-state-metrics-56d9f944b4-q5l9w" Jul 30 00:18:47.713553 ip-10-0-142-6 kubenswrapper[2564]: I0730 00:18:47.713356 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/80c96de1-7df7-4fdf-afc4-75fb6e1c23ca-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-b8f49f5c6-kbc7k\" (UID: \"80c96de1-7df7-4fdf-afc4-75fb6e1c23ca\") " pod="openshift-monitoring/openshift-state-metrics-b8f49f5c6-kbc7k" Jul 30 00:18:47.713553 ip-10-0-142-6 kubenswrapper[2564]: I0730 00:18:47.713402 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/5608ae36-4348-427e-bc57-dd15c468728e-node-exporter-wtmp\") pod \"node-exporter-5kvcx\" (UID: \"5608ae36-4348-427e-bc57-dd15c468728e\") " pod="openshift-monitoring/node-exporter-5kvcx" Jul 30 00:18:47.713553 ip-10-0-142-6 kubenswrapper[2564]: I0730 00:18:47.713420 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/03d300be-130c-46de-9d44-dd302016de4a-volume-directive-shadow\") pod \"kube-state-metrics-56d9f944b4-q5l9w\" (UID: \"03d300be-130c-46de-9d44-dd302016de4a\") " pod="openshift-monitoring/kube-state-metrics-56d9f944b4-q5l9w" Jul 30 00:18:47.713553 ip-10-0-142-6 kubenswrapper[2564]: I0730 00:18:47.713436 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mql8k\" (UniqueName: \"kubernetes.io/projected/5608ae36-4348-427e-bc57-dd15c468728e-kube-api-access-mql8k\") pod \"node-exporter-5kvcx\" (UID: \"5608ae36-4348-427e-bc57-dd15c468728e\") " pod="openshift-monitoring/node-exporter-5kvcx" Jul 30 00:18:47.713553 ip-10-0-142-6 kubenswrapper[2564]: I0730 00:18:47.713485 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/5608ae36-4348-427e-bc57-dd15c468728e-node-exporter-textfile\") pod \"node-exporter-5kvcx\" (UID: \"5608ae36-4348-427e-bc57-dd15c468728e\") " pod="openshift-monitoring/node-exporter-5kvcx" Jul 30 00:18:47.713553 ip-10-0-142-6 kubenswrapper[2564]: I0730 00:18:47.713539 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/03d300be-130c-46de-9d44-dd302016de4a-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-56d9f944b4-q5l9w\" (UID: \"03d300be-130c-46de-9d44-dd302016de4a\") " pod="openshift-monitoring/kube-state-metrics-56d9f944b4-q5l9w" Jul 30 00:18:47.713864 ip-10-0-142-6 kubenswrapper[2564]: I0730 00:18:47.713578 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/80c96de1-7df7-4fdf-afc4-75fb6e1c23ca-openshift-state-metrics-tls\") pod \"openshift-state-metrics-b8f49f5c6-kbc7k\" (UID: \"80c96de1-7df7-4fdf-afc4-75fb6e1c23ca\") " pod="openshift-monitoring/openshift-state-metrics-b8f49f5c6-kbc7k" Jul 30 00:18:47.713864 ip-10-0-142-6 kubenswrapper[2564]: I0730 00:18:47.713602 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/5608ae36-4348-427e-bc57-dd15c468728e-sys\") pod \"node-exporter-5kvcx\" (UID: \"5608ae36-4348-427e-bc57-dd15c468728e\") " pod="openshift-monitoring/node-exporter-5kvcx" Jul 30 00:18:47.713864 ip-10-0-142-6 kubenswrapper[2564]: I0730 00:18:47.713625 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/5608ae36-4348-427e-bc57-dd15c468728e-node-exporter-tls\") pod \"node-exporter-5kvcx\" (UID: \"5608ae36-4348-427e-bc57-dd15c468728e\") " pod="openshift-monitoring/node-exporter-5kvcx" Jul 30 00:18:47.713864 ip-10-0-142-6 kubenswrapper[2564]: I0730 00:18:47.713643 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/5608ae36-4348-427e-bc57-dd15c468728e-root\") pod \"node-exporter-5kvcx\" (UID: \"5608ae36-4348-427e-bc57-dd15c468728e\") " pod="openshift-monitoring/node-exporter-5kvcx" Jul 30 00:18:47.713864 ip-10-0-142-6 kubenswrapper[2564]: I0730 00:18:47.713668 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/03d300be-130c-46de-9d44-dd302016de4a-kube-state-metrics-tls\") pod \"kube-state-metrics-56d9f944b4-q5l9w\" (UID: \"03d300be-130c-46de-9d44-dd302016de4a\") " pod="openshift-monitoring/kube-state-metrics-56d9f944b4-q5l9w" Jul 30 00:18:47.713864 ip-10-0-142-6 kubenswrapper[2564]: E0730 00:18:47.713682 2564 secret.go:189] Couldn't get secret openshift-monitoring/openshift-state-metrics-tls: secret "openshift-state-metrics-tls" not found Jul 30 00:18:47.713864 ip-10-0-142-6 kubenswrapper[2564]: I0730 00:18:47.713688 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/80c96de1-7df7-4fdf-afc4-75fb6e1c23ca-metrics-client-ca\") pod \"openshift-state-metrics-b8f49f5c6-kbc7k\" (UID: \"80c96de1-7df7-4fdf-afc4-75fb6e1c23ca\") " pod="openshift-monitoring/openshift-state-metrics-b8f49f5c6-kbc7k" Jul 30 00:18:47.713864 ip-10-0-142-6 kubenswrapper[2564]: E0730 00:18:47.713760 2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/80c96de1-7df7-4fdf-afc4-75fb6e1c23ca-openshift-state-metrics-tls podName:80c96de1-7df7-4fdf-afc4-75fb6e1c23ca nodeName:}" failed. No retries permitted until 2026-07-30 00:18:48.21373514 +0000 UTC m=+59.628475066 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "openshift-state-metrics-tls" (UniqueName: "kubernetes.io/secret/80c96de1-7df7-4fdf-afc4-75fb6e1c23ca-openshift-state-metrics-tls") pod "openshift-state-metrics-b8f49f5c6-kbc7k" (UID: "80c96de1-7df7-4fdf-afc4-75fb6e1c23ca") : secret "openshift-state-metrics-tls" not found Jul 30 00:18:47.713864 ip-10-0-142-6 kubenswrapper[2564]: I0730 00:18:47.713798 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/03d300be-130c-46de-9d44-dd302016de4a-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-56d9f944b4-q5l9w\" (UID: \"03d300be-130c-46de-9d44-dd302016de4a\") " pod="openshift-monitoring/kube-state-metrics-56d9f944b4-q5l9w" Jul 30 00:18:47.713864 ip-10-0-142-6 kubenswrapper[2564]: I0730 00:18:47.713858 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bj227\" (UniqueName: \"kubernetes.io/projected/80c96de1-7df7-4fdf-afc4-75fb6e1c23ca-kube-api-access-bj227\") pod \"openshift-state-metrics-b8f49f5c6-kbc7k\" (UID: \"80c96de1-7df7-4fdf-afc4-75fb6e1c23ca\") " pod="openshift-monitoring/openshift-state-metrics-b8f49f5c6-kbc7k" Jul 30 00:18:47.714258 ip-10-0-142-6 kubenswrapper[2564]: I0730 00:18:47.713884 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/5608ae36-4348-427e-bc57-dd15c468728e-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-5kvcx\" (UID: \"5608ae36-4348-427e-bc57-dd15c468728e\") " pod="openshift-monitoring/node-exporter-5kvcx" Jul 30 00:18:47.714258 ip-10-0-142-6 kubenswrapper[2564]: I0730 00:18:47.713915 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/5608ae36-4348-427e-bc57-dd15c468728e-node-exporter-accelerators-collector-config\") pod \"node-exporter-5kvcx\" (UID: \"5608ae36-4348-427e-bc57-dd15c468728e\") " pod="openshift-monitoring/node-exporter-5kvcx" Jul 30 00:18:47.714324 ip-10-0-142-6 kubenswrapper[2564]: I0730 00:18:47.714271 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/80c96de1-7df7-4fdf-afc4-75fb6e1c23ca-metrics-client-ca\") pod \"openshift-state-metrics-b8f49f5c6-kbc7k\" (UID: \"80c96de1-7df7-4fdf-afc4-75fb6e1c23ca\") " pod="openshift-monitoring/openshift-state-metrics-b8f49f5c6-kbc7k" Jul 30 00:18:47.716819 ip-10-0-142-6 kubenswrapper[2564]: I0730 00:18:47.716791 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/80c96de1-7df7-4fdf-afc4-75fb6e1c23ca-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-b8f49f5c6-kbc7k\" (UID: \"80c96de1-7df7-4fdf-afc4-75fb6e1c23ca\") " pod="openshift-monitoring/openshift-state-metrics-b8f49f5c6-kbc7k" Jul 30 00:18:47.721754 ip-10-0-142-6 kubenswrapper[2564]: I0730 00:18:47.721736 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-bj227\" (UniqueName: \"kubernetes.io/projected/80c96de1-7df7-4fdf-afc4-75fb6e1c23ca-kube-api-access-bj227\") pod \"openshift-state-metrics-b8f49f5c6-kbc7k\" (UID: \"80c96de1-7df7-4fdf-afc4-75fb6e1c23ca\") " pod="openshift-monitoring/openshift-state-metrics-b8f49f5c6-kbc7k" Jul 30 00:18:47.814543 ip-10-0-142-6 kubenswrapper[2564]: I0730 00:18:47.814453 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/03d300be-130c-46de-9d44-dd302016de4a-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-56d9f944b4-q5l9w\" (UID: \"03d300be-130c-46de-9d44-dd302016de4a\") " pod="openshift-monitoring/kube-state-metrics-56d9f944b4-q5l9w" Jul 30 00:18:47.814543 ip-10-0-142-6 kubenswrapper[2564]: I0730 00:18:47.814509 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/5608ae36-4348-427e-bc57-dd15c468728e-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-5kvcx\" (UID: \"5608ae36-4348-427e-bc57-dd15c468728e\") " pod="openshift-monitoring/node-exporter-5kvcx" Jul 30 00:18:47.814543 ip-10-0-142-6 kubenswrapper[2564]: I0730 00:18:47.814532 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/5608ae36-4348-427e-bc57-dd15c468728e-node-exporter-accelerators-collector-config\") pod \"node-exporter-5kvcx\" (UID: \"5608ae36-4348-427e-bc57-dd15c468728e\") " pod="openshift-monitoring/node-exporter-5kvcx" Jul 30 00:18:47.814801 ip-10-0-142-6 kubenswrapper[2564]: I0730 00:18:47.814580 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-89hm9\" (UniqueName: \"kubernetes.io/projected/03d300be-130c-46de-9d44-dd302016de4a-kube-api-access-89hm9\") pod \"kube-state-metrics-56d9f944b4-q5l9w\" (UID: \"03d300be-130c-46de-9d44-dd302016de4a\") " pod="openshift-monitoring/kube-state-metrics-56d9f944b4-q5l9w" Jul 30 00:18:47.814801 ip-10-0-142-6 kubenswrapper[2564]: I0730 00:18:47.814607 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/5608ae36-4348-427e-bc57-dd15c468728e-metrics-client-ca\") pod \"node-exporter-5kvcx\" (UID: \"5608ae36-4348-427e-bc57-dd15c468728e\") " pod="openshift-monitoring/node-exporter-5kvcx" Jul 30 00:18:47.814801 ip-10-0-142-6 kubenswrapper[2564]: I0730 00:18:47.814637 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/03d300be-130c-46de-9d44-dd302016de4a-metrics-client-ca\") pod \"kube-state-metrics-56d9f944b4-q5l9w\" (UID: \"03d300be-130c-46de-9d44-dd302016de4a\") " pod="openshift-monitoring/kube-state-metrics-56d9f944b4-q5l9w" Jul 30 00:18:47.814801 ip-10-0-142-6 kubenswrapper[2564]: I0730 00:18:47.814668 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/5608ae36-4348-427e-bc57-dd15c468728e-node-exporter-wtmp\") pod \"node-exporter-5kvcx\" (UID: \"5608ae36-4348-427e-bc57-dd15c468728e\") " pod="openshift-monitoring/node-exporter-5kvcx" Jul 30 00:18:47.814801 ip-10-0-142-6 kubenswrapper[2564]: I0730 00:18:47.814698 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/03d300be-130c-46de-9d44-dd302016de4a-volume-directive-shadow\") pod \"kube-state-metrics-56d9f944b4-q5l9w\" (UID: \"03d300be-130c-46de-9d44-dd302016de4a\") " pod="openshift-monitoring/kube-state-metrics-56d9f944b4-q5l9w" Jul 30 00:18:47.814801 ip-10-0-142-6 kubenswrapper[2564]: I0730 00:18:47.814724 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mql8k\" (UniqueName: \"kubernetes.io/projected/5608ae36-4348-427e-bc57-dd15c468728e-kube-api-access-mql8k\") pod \"node-exporter-5kvcx\" (UID: \"5608ae36-4348-427e-bc57-dd15c468728e\") " pod="openshift-monitoring/node-exporter-5kvcx" Jul 30 00:18:47.814801 ip-10-0-142-6 kubenswrapper[2564]: I0730 00:18:47.814756 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/5608ae36-4348-427e-bc57-dd15c468728e-node-exporter-textfile\") pod \"node-exporter-5kvcx\" (UID: \"5608ae36-4348-427e-bc57-dd15c468728e\") " pod="openshift-monitoring/node-exporter-5kvcx" Jul 30 00:18:47.815150 ip-10-0-142-6 kubenswrapper[2564]: I0730 00:18:47.814818 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/03d300be-130c-46de-9d44-dd302016de4a-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-56d9f944b4-q5l9w\" (UID: \"03d300be-130c-46de-9d44-dd302016de4a\") " pod="openshift-monitoring/kube-state-metrics-56d9f944b4-q5l9w" Jul 30 00:18:47.815150 ip-10-0-142-6 kubenswrapper[2564]: I0730 00:18:47.814877 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/5608ae36-4348-427e-bc57-dd15c468728e-sys\") pod \"node-exporter-5kvcx\" (UID: \"5608ae36-4348-427e-bc57-dd15c468728e\") " pod="openshift-monitoring/node-exporter-5kvcx" Jul 30 00:18:47.815150 ip-10-0-142-6 kubenswrapper[2564]: I0730 00:18:47.814902 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/5608ae36-4348-427e-bc57-dd15c468728e-node-exporter-tls\") pod \"node-exporter-5kvcx\" (UID: \"5608ae36-4348-427e-bc57-dd15c468728e\") " pod="openshift-monitoring/node-exporter-5kvcx" Jul 30 00:18:47.815150 ip-10-0-142-6 kubenswrapper[2564]: I0730 00:18:47.814927 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/5608ae36-4348-427e-bc57-dd15c468728e-root\") pod \"node-exporter-5kvcx\" (UID: \"5608ae36-4348-427e-bc57-dd15c468728e\") " pod="openshift-monitoring/node-exporter-5kvcx" Jul 30 00:18:47.815150 ip-10-0-142-6 kubenswrapper[2564]: I0730 00:18:47.814957 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/03d300be-130c-46de-9d44-dd302016de4a-kube-state-metrics-tls\") pod \"kube-state-metrics-56d9f944b4-q5l9w\" (UID: \"03d300be-130c-46de-9d44-dd302016de4a\") " pod="openshift-monitoring/kube-state-metrics-56d9f944b4-q5l9w" Jul 30 00:18:47.815150 ip-10-0-142-6 kubenswrapper[2564]: I0730 00:18:47.815125 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/03d300be-130c-46de-9d44-dd302016de4a-volume-directive-shadow\") pod \"kube-state-metrics-56d9f944b4-q5l9w\" (UID: \"03d300be-130c-46de-9d44-dd302016de4a\") " pod="openshift-monitoring/kube-state-metrics-56d9f944b4-q5l9w" Jul 30 00:18:47.815463 ip-10-0-142-6 kubenswrapper[2564]: I0730 00:18:47.815205 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/5608ae36-4348-427e-bc57-dd15c468728e-node-exporter-accelerators-collector-config\") pod \"node-exporter-5kvcx\" (UID: \"5608ae36-4348-427e-bc57-dd15c468728e\") " pod="openshift-monitoring/node-exporter-5kvcx" Jul 30 00:18:47.815463 ip-10-0-142-6 kubenswrapper[2564]: I0730 00:18:47.814876 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/5608ae36-4348-427e-bc57-dd15c468728e-node-exporter-wtmp\") pod \"node-exporter-5kvcx\" (UID: \"5608ae36-4348-427e-bc57-dd15c468728e\") " pod="openshift-monitoring/node-exporter-5kvcx" Jul 30 00:18:47.815463 ip-10-0-142-6 kubenswrapper[2564]: I0730 00:18:47.815283 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/5608ae36-4348-427e-bc57-dd15c468728e-sys\") pod \"node-exporter-5kvcx\" (UID: \"5608ae36-4348-427e-bc57-dd15c468728e\") " pod="openshift-monitoring/node-exporter-5kvcx" Jul 30 00:18:47.815463 ip-10-0-142-6 kubenswrapper[2564]: I0730 00:18:47.815425 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/03d300be-130c-46de-9d44-dd302016de4a-metrics-client-ca\") pod \"kube-state-metrics-56d9f944b4-q5l9w\" (UID: \"03d300be-130c-46de-9d44-dd302016de4a\") " pod="openshift-monitoring/kube-state-metrics-56d9f944b4-q5l9w" Jul 30 00:18:47.815666 ip-10-0-142-6 kubenswrapper[2564]: E0730 00:18:47.815466 2564 secret.go:189] Couldn't get secret openshift-monitoring/node-exporter-tls: secret "node-exporter-tls" not found Jul 30 00:18:47.815666 ip-10-0-142-6 kubenswrapper[2564]: I0730 00:18:47.815487 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"root\" (UniqueName: \"kubernetes.io/host-path/5608ae36-4348-427e-bc57-dd15c468728e-root\") pod \"node-exporter-5kvcx\" (UID: \"5608ae36-4348-427e-bc57-dd15c468728e\") " pod="openshift-monitoring/node-exporter-5kvcx" Jul 30 00:18:47.815666 ip-10-0-142-6 kubenswrapper[2564]: E0730 00:18:47.815521 2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5608ae36-4348-427e-bc57-dd15c468728e-node-exporter-tls podName:5608ae36-4348-427e-bc57-dd15c468728e nodeName:}" failed. No retries permitted until 2026-07-30 00:18:48.315500417 +0000 UTC m=+59.730240343 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "node-exporter-tls" (UniqueName: "kubernetes.io/secret/5608ae36-4348-427e-bc57-dd15c468728e-node-exporter-tls") pod "node-exporter-5kvcx" (UID: "5608ae36-4348-427e-bc57-dd15c468728e") : secret "node-exporter-tls" not found Jul 30 00:18:47.815666 ip-10-0-142-6 kubenswrapper[2564]: I0730 00:18:47.815520 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/5608ae36-4348-427e-bc57-dd15c468728e-node-exporter-textfile\") pod \"node-exporter-5kvcx\" (UID: \"5608ae36-4348-427e-bc57-dd15c468728e\") " pod="openshift-monitoring/node-exporter-5kvcx" Jul 30 00:18:47.815847 ip-10-0-142-6 kubenswrapper[2564]: I0730 00:18:47.815699 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/03d300be-130c-46de-9d44-dd302016de4a-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-56d9f944b4-q5l9w\" (UID: \"03d300be-130c-46de-9d44-dd302016de4a\") " pod="openshift-monitoring/kube-state-metrics-56d9f944b4-q5l9w" Jul 30 00:18:47.815984 ip-10-0-142-6 kubenswrapper[2564]: I0730 00:18:47.815957 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/5608ae36-4348-427e-bc57-dd15c468728e-metrics-client-ca\") pod \"node-exporter-5kvcx\" (UID: \"5608ae36-4348-427e-bc57-dd15c468728e\") " pod="openshift-monitoring/node-exporter-5kvcx" Jul 30 00:18:47.817317 ip-10-0-142-6 kubenswrapper[2564]: I0730 00:18:47.817292 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/5608ae36-4348-427e-bc57-dd15c468728e-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-5kvcx\" (UID: \"5608ae36-4348-427e-bc57-dd15c468728e\") " pod="openshift-monitoring/node-exporter-5kvcx" Jul 30 00:18:47.817493 ip-10-0-142-6 kubenswrapper[2564]: I0730 00:18:47.817471 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/03d300be-130c-46de-9d44-dd302016de4a-kube-state-metrics-tls\") pod \"kube-state-metrics-56d9f944b4-q5l9w\" (UID: \"03d300be-130c-46de-9d44-dd302016de4a\") " pod="openshift-monitoring/kube-state-metrics-56d9f944b4-q5l9w" Jul 30 00:18:47.817654 ip-10-0-142-6 kubenswrapper[2564]: I0730 00:18:47.817638 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/03d300be-130c-46de-9d44-dd302016de4a-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-56d9f944b4-q5l9w\" (UID: \"03d300be-130c-46de-9d44-dd302016de4a\") " pod="openshift-monitoring/kube-state-metrics-56d9f944b4-q5l9w" Jul 30 00:18:47.823989 ip-10-0-142-6 kubenswrapper[2564]: I0730 00:18:47.823970 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-89hm9\" (UniqueName: \"kubernetes.io/projected/03d300be-130c-46de-9d44-dd302016de4a-kube-api-access-89hm9\") pod \"kube-state-metrics-56d9f944b4-q5l9w\" (UID: \"03d300be-130c-46de-9d44-dd302016de4a\") " pod="openshift-monitoring/kube-state-metrics-56d9f944b4-q5l9w" Jul 30 00:18:47.826488 ip-10-0-142-6 kubenswrapper[2564]: I0730 00:18:47.826472 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-mql8k\" (UniqueName: \"kubernetes.io/projected/5608ae36-4348-427e-bc57-dd15c468728e-kube-api-access-mql8k\") pod \"node-exporter-5kvcx\" (UID: \"5608ae36-4348-427e-bc57-dd15c468728e\") " pod="openshift-monitoring/node-exporter-5kvcx" Jul 30 00:18:47.897354 ip-10-0-142-6 kubenswrapper[2564]: I0730 00:18:47.897316 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/kube-state-metrics-56d9f944b4-q5l9w" Jul 30 00:18:48.028397 ip-10-0-142-6 kubenswrapper[2564]: I0730 00:18:48.028252 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/kube-state-metrics-56d9f944b4-q5l9w"] Jul 30 00:18:48.030735 ip-10-0-142-6 kubenswrapper[2564]: W0730 00:18:48.030701 2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod03d300be_130c_46de_9d44_dd302016de4a.slice/crio-7bc809fe5a3e23b0593c1557d0451c70a744a2ad6a91b58f4985eaef63d07474 WatchSource:0}: Error finding container 7bc809fe5a3e23b0593c1557d0451c70a744a2ad6a91b58f4985eaef63d07474: Status 404 returned error can't find the container with id 7bc809fe5a3e23b0593c1557d0451c70a744a2ad6a91b58f4985eaef63d07474 Jul 30 00:18:48.218084 ip-10-0-142-6 kubenswrapper[2564]: I0730 00:18:48.218049 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/80c96de1-7df7-4fdf-afc4-75fb6e1c23ca-openshift-state-metrics-tls\") pod \"openshift-state-metrics-b8f49f5c6-kbc7k\" (UID: \"80c96de1-7df7-4fdf-afc4-75fb6e1c23ca\") " pod="openshift-monitoring/openshift-state-metrics-b8f49f5c6-kbc7k" Jul 30 00:18:48.220364 ip-10-0-142-6 kubenswrapper[2564]: I0730 00:18:48.220341 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/80c96de1-7df7-4fdf-afc4-75fb6e1c23ca-openshift-state-metrics-tls\") pod \"openshift-state-metrics-b8f49f5c6-kbc7k\" (UID: \"80c96de1-7df7-4fdf-afc4-75fb6e1c23ca\") " pod="openshift-monitoring/openshift-state-metrics-b8f49f5c6-kbc7k" Jul 30 00:18:48.319495 ip-10-0-142-6 kubenswrapper[2564]: I0730 00:18:48.319461 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/5608ae36-4348-427e-bc57-dd15c468728e-node-exporter-tls\") pod \"node-exporter-5kvcx\" (UID: \"5608ae36-4348-427e-bc57-dd15c468728e\") " pod="openshift-monitoring/node-exporter-5kvcx" Jul 30 00:18:48.321712 ip-10-0-142-6 kubenswrapper[2564]: I0730 00:18:48.321684 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/5608ae36-4348-427e-bc57-dd15c468728e-node-exporter-tls\") pod \"node-exporter-5kvcx\" (UID: \"5608ae36-4348-427e-bc57-dd15c468728e\") " pod="openshift-monitoring/node-exporter-5kvcx" Jul 30 00:18:48.359337 ip-10-0-142-6 kubenswrapper[2564]: I0730 00:18:48.359306 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-w8grm" Jul 30 00:18:48.476372 ip-10-0-142-6 kubenswrapper[2564]: I0730 00:18:48.476287 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-56d9f944b4-q5l9w" event={"ID":"03d300be-130c-46de-9d44-dd302016de4a","Type":"ContainerStarted","Data":"7bc809fe5a3e23b0593c1557d0451c70a744a2ad6a91b58f4985eaef63d07474"} Jul 30 00:18:48.486859 ip-10-0-142-6 kubenswrapper[2564]: I0730 00:18:48.486836 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/openshift-state-metrics-b8f49f5c6-kbc7k" Jul 30 00:18:48.516306 ip-10-0-142-6 kubenswrapper[2564]: I0730 00:18:48.516276 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-5kvcx" Jul 30 00:18:48.530381 ip-10-0-142-6 kubenswrapper[2564]: W0730 00:18:48.530349 2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5608ae36_4348_427e_bc57_dd15c468728e.slice/crio-83275cb1d02aa4cf1210ad0e9fdd04f6049e06a1912f8bd99668208f259d1dc0 WatchSource:0}: Error finding container 83275cb1d02aa4cf1210ad0e9fdd04f6049e06a1912f8bd99668208f259d1dc0: Status 404 returned error can't find the container with id 83275cb1d02aa4cf1210ad0e9fdd04f6049e06a1912f8bd99668208f259d1dc0 Jul 30 00:18:48.618173 ip-10-0-142-6 kubenswrapper[2564]: I0730 00:18:48.618138 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/openshift-state-metrics-b8f49f5c6-kbc7k"] Jul 30 00:18:48.620961 ip-10-0-142-6 kubenswrapper[2564]: W0730 00:18:48.620935 2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod80c96de1_7df7_4fdf_afc4_75fb6e1c23ca.slice/crio-ba66bc32b0581f0f99c0b913522709d20c70198411488de500147feb0187a442 WatchSource:0}: Error finding container ba66bc32b0581f0f99c0b913522709d20c70198411488de500147feb0187a442: Status 404 returned error can't find the container with id ba66bc32b0581f0f99c0b913522709d20c70198411488de500147feb0187a442 Jul 30 00:18:49.481510 ip-10-0-142-6 kubenswrapper[2564]: I0730 00:18:49.481473 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-5kvcx" event={"ID":"5608ae36-4348-427e-bc57-dd15c468728e","Type":"ContainerStarted","Data":"83275cb1d02aa4cf1210ad0e9fdd04f6049e06a1912f8bd99668208f259d1dc0"} Jul 30 00:18:49.483604 ip-10-0-142-6 kubenswrapper[2564]: I0730 00:18:49.483577 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-b8f49f5c6-kbc7k" event={"ID":"80c96de1-7df7-4fdf-afc4-75fb6e1c23ca","Type":"ContainerStarted","Data":"983e81aeb967e0b7f3a130f4b00adea8284ea4c943e2559d656eb89fe9e69d6d"} Jul 30 00:18:49.483736 ip-10-0-142-6 kubenswrapper[2564]: I0730 00:18:49.483636 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-b8f49f5c6-kbc7k" event={"ID":"80c96de1-7df7-4fdf-afc4-75fb6e1c23ca","Type":"ContainerStarted","Data":"5d60ad96e3ce141aa2abef14d0882c200a82c935c2cc590f3c8fd9c2333d7340"} Jul 30 00:18:49.483736 ip-10-0-142-6 kubenswrapper[2564]: I0730 00:18:49.483662 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-b8f49f5c6-kbc7k" event={"ID":"80c96de1-7df7-4fdf-afc4-75fb6e1c23ca","Type":"ContainerStarted","Data":"ba66bc32b0581f0f99c0b913522709d20c70198411488de500147feb0187a442"} Jul 30 00:18:49.823361 ip-10-0-142-6 kubenswrapper[2564]: I0730 00:18:49.823284 2564 patch_prober.go:28] interesting pod/image-registry-6477b9596b-4thrq container/registry namespace/openshift-image-registry: Liveness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body={"errors":[{"code":"UNAVAILABLE","message":"service unavailable","detail":"health check failed: please see /debug/health"}]} Jul 30 00:18:49.823361 ip-10-0-142-6 kubenswrapper[2564]: I0730 00:18:49.823338 2564 prober.go:120] "Probe failed" probeType="Liveness" pod="openshift-image-registry/image-registry-6477b9596b-4thrq" podUID="2af09576-ffda-4b11-8ddf-ce24de77a2d9" containerName="registry" probeResult="failure" output="HTTP probe failed with statuscode: 503" Jul 30 00:18:50.490028 ip-10-0-142-6 kubenswrapper[2564]: I0730 00:18:50.489995 2564 generic.go:356] "Generic (PLEG): container finished" podID="5608ae36-4348-427e-bc57-dd15c468728e" containerID="f33e732b11cbb4e8150483b5c10646584ac83fa8e03e7f614268577a6d3c3645" exitCode=0 Jul 30 00:18:50.490539 ip-10-0-142-6 kubenswrapper[2564]: I0730 00:18:50.490086 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-5kvcx" event={"ID":"5608ae36-4348-427e-bc57-dd15c468728e","Type":"ContainerDied","Data":"f33e732b11cbb4e8150483b5c10646584ac83fa8e03e7f614268577a6d3c3645"} Jul 30 00:18:51.405558 ip-10-0-142-6 kubenswrapper[2564]: I0730 00:18:51.405531 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-image-registry/image-registry-6477b9596b-4thrq" Jul 30 00:18:51.495787 ip-10-0-142-6 kubenswrapper[2564]: I0730 00:18:51.495750 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-b8f49f5c6-kbc7k" event={"ID":"80c96de1-7df7-4fdf-afc4-75fb6e1c23ca","Type":"ContainerStarted","Data":"9adb7ebc062a76a3ca77d71c93b86436791e9663924694cfeea83c128727b4e4"} Jul 30 00:18:51.497842 ip-10-0-142-6 kubenswrapper[2564]: I0730 00:18:51.497813 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-56d9f944b4-q5l9w" event={"ID":"03d300be-130c-46de-9d44-dd302016de4a","Type":"ContainerStarted","Data":"5cbcdad89653a4f6d3e47d66b607c36fd31f583703d9ed5329639073f9528ebc"} Jul 30 00:18:51.497986 ip-10-0-142-6 kubenswrapper[2564]: I0730 00:18:51.497876 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-56d9f944b4-q5l9w" event={"ID":"03d300be-130c-46de-9d44-dd302016de4a","Type":"ContainerStarted","Data":"18741a2c0b8b7cb12034371cd9bb98456d3a4575212c957237684b52c6cd2814"} Jul 30 00:18:51.497986 ip-10-0-142-6 kubenswrapper[2564]: I0730 00:18:51.497904 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-56d9f944b4-q5l9w" event={"ID":"03d300be-130c-46de-9d44-dd302016de4a","Type":"ContainerStarted","Data":"4c58d10d6aee3499ef56d647475a54109d55af26b656a7090292a706e7d512e6"} Jul 30 00:18:51.499905 ip-10-0-142-6 kubenswrapper[2564]: I0730 00:18:51.499885 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-5kvcx" event={"ID":"5608ae36-4348-427e-bc57-dd15c468728e","Type":"ContainerStarted","Data":"2c16a2a8c8253eeb1b1b1058c69ad88dd7dfe965fdc7cbccfa4db47b67856e0a"} Jul 30 00:18:51.499984 ip-10-0-142-6 kubenswrapper[2564]: I0730 00:18:51.499928 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-5kvcx" event={"ID":"5608ae36-4348-427e-bc57-dd15c468728e","Type":"ContainerStarted","Data":"f5ef61eccbb55a6de9c9bcc0d5e6286a8566365fde52c05fb9cb703a7aff1c37"} Jul 30 00:18:51.511200 ip-10-0-142-6 kubenswrapper[2564]: I0730 00:18:51.511161 2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/openshift-state-metrics-b8f49f5c6-kbc7k" podStartSLOduration=2.507199818 podStartE2EDuration="4.51114738s" totalImagesPullingTime="2.003947562s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-30 00:18:47 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-30 00:18:51.510588492 +0000 UTC m=+62.925328428" watchObservedRunningTime="2026-07-30 00:18:51.51114738 +0000 UTC m=+62.925887312" Jul 30 00:18:51.527473 ip-10-0-142-6 kubenswrapper[2564]: I0730 00:18:51.527427 2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/node-exporter-5kvcx" podStartSLOduration=2.927750977 podStartE2EDuration="4.527412195s" totalImagesPullingTime="1.4769496s" totalInitContainerRuntime="122.711618ms" isStatefulPod=true podCreationTimestamp="2026-07-30 00:18:47 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-30 00:18:51.526504876 +0000 UTC m=+62.941244811" watchObservedRunningTime="2026-07-30 00:18:51.527412195 +0000 UTC m=+62.942152181" Jul 30 00:18:51.543805 ip-10-0-142-6 kubenswrapper[2564]: I0730 00:18:51.543760 2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/kube-state-metrics-56d9f944b4-q5l9w" podStartSLOduration=1.7197360800000001 podStartE2EDuration="4.543746885s" totalImagesPullingTime="2.824010805s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-30 00:18:47 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-30 00:18:51.543180848 +0000 UTC m=+62.957920794" watchObservedRunningTime="2026-07-30 00:18:51.543746885 +0000 UTC m=+62.958486819" Jul 30 00:18:52.806813 ip-10-0-142-6 kubenswrapper[2564]: I0730 00:18:52.806777 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-56d468748c-tfk4j" Jul 30 00:18:52.807208 ip-10-0-142-6 kubenswrapper[2564]: I0730 00:18:52.806861 2564 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-56d468748c-tfk4j" Jul 30 00:18:52.812753 ip-10-0-142-6 kubenswrapper[2564]: I0730 00:18:52.812730 2564 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-56d468748c-tfk4j" Jul 30 00:18:53.510693 ip-10-0-142-6 kubenswrapper[2564]: I0730 00:18:53.510665 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-56d468748c-tfk4j" Jul 30 00:18:54.090624 ip-10-0-142-6 kubenswrapper[2564]: I0730 00:18:54.090591 2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-7594cdd77f-4n25d"] Jul 30 00:18:54.092754 ip-10-0-142-6 kubenswrapper[2564]: I0730 00:18:54.092734 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-7594cdd77f-4n25d" Jul 30 00:18:54.103184 ip-10-0-142-6 kubenswrapper[2564]: I0730 00:18:54.103160 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-7594cdd77f-4n25d"] Jul 30 00:18:54.170442 ip-10-0-142-6 kubenswrapper[2564]: I0730 00:18:54.170407 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/07979fe1-67ca-4210-9aa8-86cf58d3642b-console-serving-cert\") pod \"console-7594cdd77f-4n25d\" (UID: \"07979fe1-67ca-4210-9aa8-86cf58d3642b\") " pod="openshift-console/console-7594cdd77f-4n25d" Jul 30 00:18:54.170442 ip-10-0-142-6 kubenswrapper[2564]: I0730 00:18:54.170445 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/07979fe1-67ca-4210-9aa8-86cf58d3642b-console-config\") pod \"console-7594cdd77f-4n25d\" (UID: \"07979fe1-67ca-4210-9aa8-86cf58d3642b\") " pod="openshift-console/console-7594cdd77f-4n25d" Jul 30 00:18:54.170610 ip-10-0-142-6 kubenswrapper[2564]: I0730 00:18:54.170485 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/07979fe1-67ca-4210-9aa8-86cf58d3642b-trusted-ca-bundle\") pod \"console-7594cdd77f-4n25d\" (UID: \"07979fe1-67ca-4210-9aa8-86cf58d3642b\") " pod="openshift-console/console-7594cdd77f-4n25d" Jul 30 00:18:54.170610 ip-10-0-142-6 kubenswrapper[2564]: I0730 00:18:54.170521 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/07979fe1-67ca-4210-9aa8-86cf58d3642b-console-oauth-config\") pod \"console-7594cdd77f-4n25d\" (UID: \"07979fe1-67ca-4210-9aa8-86cf58d3642b\") " pod="openshift-console/console-7594cdd77f-4n25d" Jul 30 00:18:54.170610 ip-10-0-142-6 kubenswrapper[2564]: I0730 00:18:54.170543 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/07979fe1-67ca-4210-9aa8-86cf58d3642b-service-ca\") pod \"console-7594cdd77f-4n25d\" (UID: \"07979fe1-67ca-4210-9aa8-86cf58d3642b\") " pod="openshift-console/console-7594cdd77f-4n25d" Jul 30 00:18:54.170610 ip-10-0-142-6 kubenswrapper[2564]: I0730 00:18:54.170573 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/07979fe1-67ca-4210-9aa8-86cf58d3642b-oauth-serving-cert\") pod \"console-7594cdd77f-4n25d\" (UID: \"07979fe1-67ca-4210-9aa8-86cf58d3642b\") " pod="openshift-console/console-7594cdd77f-4n25d" Jul 30 00:18:54.170749 ip-10-0-142-6 kubenswrapper[2564]: I0730 00:18:54.170661 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ghzd9\" (UniqueName: \"kubernetes.io/projected/07979fe1-67ca-4210-9aa8-86cf58d3642b-kube-api-access-ghzd9\") pod \"console-7594cdd77f-4n25d\" (UID: \"07979fe1-67ca-4210-9aa8-86cf58d3642b\") " pod="openshift-console/console-7594cdd77f-4n25d" Jul 30 00:18:54.272069 ip-10-0-142-6 kubenswrapper[2564]: I0730 00:18:54.272030 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/07979fe1-67ca-4210-9aa8-86cf58d3642b-console-serving-cert\") pod \"console-7594cdd77f-4n25d\" (UID: \"07979fe1-67ca-4210-9aa8-86cf58d3642b\") " pod="openshift-console/console-7594cdd77f-4n25d" Jul 30 00:18:54.272232 ip-10-0-142-6 kubenswrapper[2564]: I0730 00:18:54.272069 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/07979fe1-67ca-4210-9aa8-86cf58d3642b-console-config\") pod \"console-7594cdd77f-4n25d\" (UID: \"07979fe1-67ca-4210-9aa8-86cf58d3642b\") " pod="openshift-console/console-7594cdd77f-4n25d" Jul 30 00:18:54.272232 ip-10-0-142-6 kubenswrapper[2564]: I0730 00:18:54.272148 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/07979fe1-67ca-4210-9aa8-86cf58d3642b-trusted-ca-bundle\") pod \"console-7594cdd77f-4n25d\" (UID: \"07979fe1-67ca-4210-9aa8-86cf58d3642b\") " pod="openshift-console/console-7594cdd77f-4n25d" Jul 30 00:18:54.272232 ip-10-0-142-6 kubenswrapper[2564]: I0730 00:18:54.272166 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/07979fe1-67ca-4210-9aa8-86cf58d3642b-console-oauth-config\") pod \"console-7594cdd77f-4n25d\" (UID: \"07979fe1-67ca-4210-9aa8-86cf58d3642b\") " pod="openshift-console/console-7594cdd77f-4n25d" Jul 30 00:18:54.272344 ip-10-0-142-6 kubenswrapper[2564]: I0730 00:18:54.272303 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/07979fe1-67ca-4210-9aa8-86cf58d3642b-service-ca\") pod \"console-7594cdd77f-4n25d\" (UID: \"07979fe1-67ca-4210-9aa8-86cf58d3642b\") " pod="openshift-console/console-7594cdd77f-4n25d" Jul 30 00:18:54.272399 ip-10-0-142-6 kubenswrapper[2564]: I0730 00:18:54.272341 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/07979fe1-67ca-4210-9aa8-86cf58d3642b-oauth-serving-cert\") pod \"console-7594cdd77f-4n25d\" (UID: \"07979fe1-67ca-4210-9aa8-86cf58d3642b\") " pod="openshift-console/console-7594cdd77f-4n25d" Jul 30 00:18:54.272449 ip-10-0-142-6 kubenswrapper[2564]: I0730 00:18:54.272427 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ghzd9\" (UniqueName: \"kubernetes.io/projected/07979fe1-67ca-4210-9aa8-86cf58d3642b-kube-api-access-ghzd9\") pod \"console-7594cdd77f-4n25d\" (UID: \"07979fe1-67ca-4210-9aa8-86cf58d3642b\") " pod="openshift-console/console-7594cdd77f-4n25d" Jul 30 00:18:54.273011 ip-10-0-142-6 kubenswrapper[2564]: I0730 00:18:54.272978 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/07979fe1-67ca-4210-9aa8-86cf58d3642b-console-config\") pod \"console-7594cdd77f-4n25d\" (UID: \"07979fe1-67ca-4210-9aa8-86cf58d3642b\") " pod="openshift-console/console-7594cdd77f-4n25d" Jul 30 00:18:54.273166 ip-10-0-142-6 kubenswrapper[2564]: I0730 00:18:54.273018 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/07979fe1-67ca-4210-9aa8-86cf58d3642b-service-ca\") pod \"console-7594cdd77f-4n25d\" (UID: \"07979fe1-67ca-4210-9aa8-86cf58d3642b\") " pod="openshift-console/console-7594cdd77f-4n25d" Jul 30 00:18:54.273166 ip-10-0-142-6 kubenswrapper[2564]: I0730 00:18:54.273066 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/07979fe1-67ca-4210-9aa8-86cf58d3642b-oauth-serving-cert\") pod \"console-7594cdd77f-4n25d\" (UID: \"07979fe1-67ca-4210-9aa8-86cf58d3642b\") " pod="openshift-console/console-7594cdd77f-4n25d" Jul 30 00:18:54.273297 ip-10-0-142-6 kubenswrapper[2564]: I0730 00:18:54.273226 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/07979fe1-67ca-4210-9aa8-86cf58d3642b-trusted-ca-bundle\") pod \"console-7594cdd77f-4n25d\" (UID: \"07979fe1-67ca-4210-9aa8-86cf58d3642b\") " pod="openshift-console/console-7594cdd77f-4n25d" Jul 30 00:18:54.274472 ip-10-0-142-6 kubenswrapper[2564]: I0730 00:18:54.274450 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/07979fe1-67ca-4210-9aa8-86cf58d3642b-console-oauth-config\") pod \"console-7594cdd77f-4n25d\" (UID: \"07979fe1-67ca-4210-9aa8-86cf58d3642b\") " pod="openshift-console/console-7594cdd77f-4n25d" Jul 30 00:18:54.274659 ip-10-0-142-6 kubenswrapper[2564]: I0730 00:18:54.274640 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/07979fe1-67ca-4210-9aa8-86cf58d3642b-console-serving-cert\") pod \"console-7594cdd77f-4n25d\" (UID: \"07979fe1-67ca-4210-9aa8-86cf58d3642b\") " pod="openshift-console/console-7594cdd77f-4n25d" Jul 30 00:18:54.279845 ip-10-0-142-6 kubenswrapper[2564]: I0730 00:18:54.279825 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ghzd9\" (UniqueName: \"kubernetes.io/projected/07979fe1-67ca-4210-9aa8-86cf58d3642b-kube-api-access-ghzd9\") pod \"console-7594cdd77f-4n25d\" (UID: \"07979fe1-67ca-4210-9aa8-86cf58d3642b\") " pod="openshift-console/console-7594cdd77f-4n25d" Jul 30 00:18:54.401684 ip-10-0-142-6 kubenswrapper[2564]: I0730 00:18:54.401648 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-7594cdd77f-4n25d" Jul 30 00:18:54.527675 ip-10-0-142-6 kubenswrapper[2564]: I0730 00:18:54.527637 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-7594cdd77f-4n25d"] Jul 30 00:18:54.531032 ip-10-0-142-6 kubenswrapper[2564]: W0730 00:18:54.531008 2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod07979fe1_67ca_4210_9aa8_86cf58d3642b.slice/crio-e93762af6ece6f3e89d9429bec22bbedd3a13c370a9579b38529e1cefc8654a9 WatchSource:0}: Error finding container e93762af6ece6f3e89d9429bec22bbedd3a13c370a9579b38529e1cefc8654a9: Status 404 returned error can't find the container with id e93762af6ece6f3e89d9429bec22bbedd3a13c370a9579b38529e1cefc8654a9 Jul 30 00:18:54.776547 ip-10-0-142-6 kubenswrapper[2564]: I0730 00:18:54.776439 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5v98n\" (UniqueName: \"kubernetes.io/projected/a0bef212-99cc-47c3-b1da-316f26f776b9-kube-api-access-5v98n\") pod \"network-check-target-j4ckt\" (UID: \"a0bef212-99cc-47c3-b1da-316f26f776b9\") " pod="openshift-network-diagnostics/network-check-target-j4ckt" Jul 30 00:18:54.776547 ip-10-0-142-6 kubenswrapper[2564]: I0730 00:18:54.776492 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/81450593-06b9-4f5a-81da-6e5fe67679f1-original-pull-secret\") pod \"global-pull-secret-syncer-d6phj\" (UID: \"81450593-06b9-4f5a-81da-6e5fe67679f1\") " pod="kube-system/global-pull-secret-syncer-d6phj" Jul 30 00:18:54.779277 ip-10-0-142-6 kubenswrapper[2564]: I0730 00:18:54.779256 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\"" Jul 30 00:18:54.779389 ip-10-0-142-6 kubenswrapper[2564]: I0730 00:18:54.779257 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\"" Jul 30 00:18:54.789495 ip-10-0-142-6 kubenswrapper[2564]: I0730 00:18:54.789477 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\"" Jul 30 00:18:54.789656 ip-10-0-142-6 kubenswrapper[2564]: I0730 00:18:54.789639 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/81450593-06b9-4f5a-81da-6e5fe67679f1-original-pull-secret\") pod \"global-pull-secret-syncer-d6phj\" (UID: \"81450593-06b9-4f5a-81da-6e5fe67679f1\") " pod="kube-system/global-pull-secret-syncer-d6phj" Jul 30 00:18:54.800635 ip-10-0-142-6 kubenswrapper[2564]: I0730 00:18:54.800609 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5v98n\" (UniqueName: \"kubernetes.io/projected/a0bef212-99cc-47c3-b1da-316f26f776b9-kube-api-access-5v98n\") pod \"network-check-target-j4ckt\" (UID: \"a0bef212-99cc-47c3-b1da-316f26f776b9\") " pod="openshift-network-diagnostics/network-check-target-j4ckt" Jul 30 00:18:54.863184 ip-10-0-142-6 kubenswrapper[2564]: I0730 00:18:54.863154 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-qvw7x\"" Jul 30 00:18:54.871048 ip-10-0-142-6 kubenswrapper[2564]: I0730 00:18:54.871023 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-j4ckt" Jul 30 00:18:54.877392 ip-10-0-142-6 kubenswrapper[2564]: I0730 00:18:54.876976 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f82506d3-a3ca-415e-b6c0-4de2f2bf35f5-metrics-certs\") pod \"network-metrics-daemon-5skn2\" (UID: \"f82506d3-a3ca-415e-b6c0-4de2f2bf35f5\") " pod="openshift-multus/network-metrics-daemon-5skn2" Jul 30 00:18:54.879451 ip-10-0-142-6 kubenswrapper[2564]: I0730 00:18:54.879430 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\"" Jul 30 00:18:54.889392 ip-10-0-142-6 kubenswrapper[2564]: I0730 00:18:54.889371 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f82506d3-a3ca-415e-b6c0-4de2f2bf35f5-metrics-certs\") pod \"network-metrics-daemon-5skn2\" (UID: \"f82506d3-a3ca-415e-b6c0-4de2f2bf35f5\") " pod="openshift-multus/network-metrics-daemon-5skn2" Jul 30 00:18:54.993631 ip-10-0-142-6 kubenswrapper[2564]: I0730 00:18:54.993549 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-j4ckt"] Jul 30 00:18:54.996010 ip-10-0-142-6 kubenswrapper[2564]: W0730 00:18:54.995985 2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda0bef212_99cc_47c3_b1da_316f26f776b9.slice/crio-7c653dc7d2e4020a6361b49725b4d77bd2cc3b6c01ef4828e3d7dc4277758b81 WatchSource:0}: Error finding container 7c653dc7d2e4020a6361b49725b4d77bd2cc3b6c01ef4828e3d7dc4277758b81: Status 404 returned error can't find the container with id 7c653dc7d2e4020a6361b49725b4d77bd2cc3b6c01ef4828e3d7dc4277758b81 Jul 30 00:18:55.043226 ip-10-0-142-6 kubenswrapper[2564]: I0730 00:18:55.043146 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-d6phj" Jul 30 00:18:55.156367 ip-10-0-142-6 kubenswrapper[2564]: I0730 00:18:55.156340 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-f7pzr\"" Jul 30 00:18:55.164110 ip-10-0-142-6 kubenswrapper[2564]: I0730 00:18:55.164022 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-5skn2" Jul 30 00:18:55.165261 ip-10-0-142-6 kubenswrapper[2564]: I0730 00:18:55.165241 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-d6phj"] Jul 30 00:18:55.168411 ip-10-0-142-6 kubenswrapper[2564]: W0730 00:18:55.168386 2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod81450593_06b9_4f5a_81da_6e5fe67679f1.slice/crio-93b30653a884ad097596b419d7bfa0500f9a60f4fd065563a33d5031af6c40e4 WatchSource:0}: Error finding container 93b30653a884ad097596b419d7bfa0500f9a60f4fd065563a33d5031af6c40e4: Status 404 returned error can't find the container with id 93b30653a884ad097596b419d7bfa0500f9a60f4fd065563a33d5031af6c40e4 Jul 30 00:18:55.307729 ip-10-0-142-6 kubenswrapper[2564]: I0730 00:18:55.307657 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-5skn2"] Jul 30 00:18:55.310794 ip-10-0-142-6 kubenswrapper[2564]: W0730 00:18:55.310768 2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf82506d3_a3ca_415e_b6c0_4de2f2bf35f5.slice/crio-47d504d5c9339de137f2124d688844ec91be664af00400363938de2fee55c691 WatchSource:0}: Error finding container 47d504d5c9339de137f2124d688844ec91be664af00400363938de2fee55c691: Status 404 returned error can't find the container with id 47d504d5c9339de137f2124d688844ec91be664af00400363938de2fee55c691 Jul 30 00:18:55.516533 ip-10-0-142-6 kubenswrapper[2564]: I0730 00:18:55.516490 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-d6phj" event={"ID":"81450593-06b9-4f5a-81da-6e5fe67679f1","Type":"ContainerStarted","Data":"93b30653a884ad097596b419d7bfa0500f9a60f4fd065563a33d5031af6c40e4"} Jul 30 00:18:55.517798 ip-10-0-142-6 kubenswrapper[2564]: I0730 00:18:55.517767 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-j4ckt" event={"ID":"a0bef212-99cc-47c3-b1da-316f26f776b9","Type":"ContainerStarted","Data":"7c653dc7d2e4020a6361b49725b4d77bd2cc3b6c01ef4828e3d7dc4277758b81"} Jul 30 00:18:55.519545 ip-10-0-142-6 kubenswrapper[2564]: I0730 00:18:55.519518 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-7594cdd77f-4n25d" event={"ID":"07979fe1-67ca-4210-9aa8-86cf58d3642b","Type":"ContainerStarted","Data":"cd003bf956eb3c6219ce631a55c395e27a16afcd60322691b2aad62860b3f430"} Jul 30 00:18:55.519660 ip-10-0-142-6 kubenswrapper[2564]: I0730 00:18:55.519563 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-7594cdd77f-4n25d" event={"ID":"07979fe1-67ca-4210-9aa8-86cf58d3642b","Type":"ContainerStarted","Data":"e93762af6ece6f3e89d9429bec22bbedd3a13c370a9579b38529e1cefc8654a9"} Jul 30 00:18:55.520791 ip-10-0-142-6 kubenswrapper[2564]: I0730 00:18:55.520770 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-5skn2" event={"ID":"f82506d3-a3ca-415e-b6c0-4de2f2bf35f5","Type":"ContainerStarted","Data":"47d504d5c9339de137f2124d688844ec91be664af00400363938de2fee55c691"} Jul 30 00:18:55.537788 ip-10-0-142-6 kubenswrapper[2564]: I0730 00:18:55.537737 2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-7594cdd77f-4n25d" podStartSLOduration=1.5377207899999998 podStartE2EDuration="1.53772079s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-30 00:18:54 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-30 00:18:55.536622998 +0000 UTC m=+66.951362967" watchObservedRunningTime="2026-07-30 00:18:55.53772079 +0000 UTC m=+66.952460723" Jul 30 00:18:57.531984 ip-10-0-142-6 kubenswrapper[2564]: I0730 00:18:57.531942 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-5skn2" event={"ID":"f82506d3-a3ca-415e-b6c0-4de2f2bf35f5","Type":"ContainerStarted","Data":"ad7d57a0448b69330efdd0ab7a43da289bb6b2aa3b55f2113600ff9a61270904"} Jul 30 00:18:57.532458 ip-10-0-142-6 kubenswrapper[2564]: I0730 00:18:57.532006 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-5skn2" event={"ID":"f82506d3-a3ca-415e-b6c0-4de2f2bf35f5","Type":"ContainerStarted","Data":"00500d3271417d573baff4ea2fc5f80c7037110d9e3039ab8cec0a086647b092"} Jul 30 00:18:58.538237 ip-10-0-142-6 kubenswrapper[2564]: I0730 00:18:58.538197 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-j4ckt" event={"ID":"a0bef212-99cc-47c3-b1da-316f26f776b9","Type":"ContainerStarted","Data":"7e28488ce30f7edc4746dec2a000afbf5e99f0054487b3b04a20798d79391f26"} Jul 30 00:18:58.538652 ip-10-0-142-6 kubenswrapper[2564]: I0730 00:18:58.538385 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-j4ckt" Jul 30 00:18:58.554030 ip-10-0-142-6 kubenswrapper[2564]: I0730 00:18:58.553979 2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/network-metrics-daemon-5skn2" podStartSLOduration=68.30204852 podStartE2EDuration="1m9.553963975s" totalImagesPullingTime="1.251915455s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-30 00:17:49 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-30 00:18:57.547132956 +0000 UTC m=+68.961872890" watchObservedRunningTime="2026-07-30 00:18:58.553963975 +0000 UTC m=+69.968703908" Jul 30 00:18:58.554649 ip-10-0-142-6 kubenswrapper[2564]: I0730 00:18:58.554609 2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-target-j4ckt" podStartSLOduration=66.549728083 podStartE2EDuration="1m9.554596367s" totalImagesPullingTime="3.004868284s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-30 00:17:49 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-30 00:18:58.55364522 +0000 UTC m=+69.968385363" watchObservedRunningTime="2026-07-30 00:18:58.554596367 +0000 UTC m=+69.969336300" Jul 30 00:19:00.546971 ip-10-0-142-6 kubenswrapper[2564]: I0730 00:19:00.546937 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-d6phj" event={"ID":"81450593-06b9-4f5a-81da-6e5fe67679f1","Type":"ContainerStarted","Data":"d27396a636422aa69e0deaf6499c43316870bee9a932b30af5526da9f91c5ab5"} Jul 30 00:19:00.560517 ip-10-0-142-6 kubenswrapper[2564]: I0730 00:19:00.560459 2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/global-pull-secret-syncer-d6phj" podStartSLOduration=67.143298337 podStartE2EDuration="1m11.560444007s" totalImagesPullingTime="4.41714567s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-30 00:17:49 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-30 00:19:00.560054204 +0000 UTC m=+71.974794149" watchObservedRunningTime="2026-07-30 00:19:00.560444007 +0000 UTC m=+71.975183939" Jul 30 00:19:04.402312 ip-10-0-142-6 kubenswrapper[2564]: I0730 00:19:04.402273 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-7594cdd77f-4n25d" Jul 30 00:19:04.402710 ip-10-0-142-6 kubenswrapper[2564]: I0730 00:19:04.402340 2564 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-7594cdd77f-4n25d" Jul 30 00:19:04.407182 ip-10-0-142-6 kubenswrapper[2564]: I0730 00:19:04.407157 2564 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-7594cdd77f-4n25d" Jul 30 00:19:04.565133 ip-10-0-142-6 kubenswrapper[2564]: I0730 00:19:04.565107 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-7594cdd77f-4n25d" Jul 30 00:19:04.612977 ip-10-0-142-6 kubenswrapper[2564]: I0730 00:19:04.612941 2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-56d468748c-tfk4j"] Jul 30 00:19:29.545841 ip-10-0-142-6 kubenswrapper[2564]: I0730 00:19:29.545720 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-network-diagnostics/network-check-target-j4ckt" Jul 30 00:19:29.631924 ip-10-0-142-6 kubenswrapper[2564]: I0730 00:19:29.631852 2564 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-console/console-56d468748c-tfk4j" podUID="e3ac38c6-9513-43cf-9af0-8e49aceca4ed" containerName="console" containerID="cri-o://ab843fb3b471f2ee3ffa98a73a9b109dbe32702c74d9348f4927e709b51aef82" gracePeriod=15 Jul 30 00:19:30.391035 ip-10-0-142-6 kubenswrapper[2564]: I0730 00:19:30.390998 2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-56d468748c-tfk4j" Jul 30 00:19:30.559226 ip-10-0-142-6 kubenswrapper[2564]: I0730 00:19:30.559134 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/e3ac38c6-9513-43cf-9af0-8e49aceca4ed-trusted-ca-bundle\") pod \"e3ac38c6-9513-43cf-9af0-8e49aceca4ed\" (UID: \"e3ac38c6-9513-43cf-9af0-8e49aceca4ed\") " Jul 30 00:19:30.559226 ip-10-0-142-6 kubenswrapper[2564]: I0730 00:19:30.559186 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/e3ac38c6-9513-43cf-9af0-8e49aceca4ed-service-ca\") pod \"e3ac38c6-9513-43cf-9af0-8e49aceca4ed\" (UID: \"e3ac38c6-9513-43cf-9af0-8e49aceca4ed\") " Jul 30 00:19:30.559226 ip-10-0-142-6 kubenswrapper[2564]: I0730 00:19:30.559217 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/e3ac38c6-9513-43cf-9af0-8e49aceca4ed-console-serving-cert\") pod \"e3ac38c6-9513-43cf-9af0-8e49aceca4ed\" (UID: \"e3ac38c6-9513-43cf-9af0-8e49aceca4ed\") " Jul 30 00:19:30.559758 ip-10-0-142-6 kubenswrapper[2564]: I0730 00:19:30.559271 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/e3ac38c6-9513-43cf-9af0-8e49aceca4ed-console-config\") pod \"e3ac38c6-9513-43cf-9af0-8e49aceca4ed\" (UID: \"e3ac38c6-9513-43cf-9af0-8e49aceca4ed\") " Jul 30 00:19:30.559758 ip-10-0-142-6 kubenswrapper[2564]: I0730 00:19:30.559302 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/e3ac38c6-9513-43cf-9af0-8e49aceca4ed-oauth-serving-cert\") pod \"e3ac38c6-9513-43cf-9af0-8e49aceca4ed\" (UID: \"e3ac38c6-9513-43cf-9af0-8e49aceca4ed\") " Jul 30 00:19:30.559758 ip-10-0-142-6 kubenswrapper[2564]: I0730 00:19:30.559334 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-ks5rb\" (UniqueName: \"kubernetes.io/projected/e3ac38c6-9513-43cf-9af0-8e49aceca4ed-kube-api-access-ks5rb\") pod \"e3ac38c6-9513-43cf-9af0-8e49aceca4ed\" (UID: \"e3ac38c6-9513-43cf-9af0-8e49aceca4ed\") " Jul 30 00:19:30.559758 ip-10-0-142-6 kubenswrapper[2564]: I0730 00:19:30.559418 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/e3ac38c6-9513-43cf-9af0-8e49aceca4ed-console-oauth-config\") pod \"e3ac38c6-9513-43cf-9af0-8e49aceca4ed\" (UID: \"e3ac38c6-9513-43cf-9af0-8e49aceca4ed\") " Jul 30 00:19:30.559758 ip-10-0-142-6 kubenswrapper[2564]: I0730 00:19:30.559536 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e3ac38c6-9513-43cf-9af0-8e49aceca4ed-trusted-ca-bundle" (OuterVolumeSpecName: "trusted-ca-bundle") pod "e3ac38c6-9513-43cf-9af0-8e49aceca4ed" (UID: "e3ac38c6-9513-43cf-9af0-8e49aceca4ed"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 30 00:19:30.559758 ip-10-0-142-6 kubenswrapper[2564]: I0730 00:19:30.559683 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e3ac38c6-9513-43cf-9af0-8e49aceca4ed-console-config" (OuterVolumeSpecName: "console-config") pod "e3ac38c6-9513-43cf-9af0-8e49aceca4ed" (UID: "e3ac38c6-9513-43cf-9af0-8e49aceca4ed"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 30 00:19:30.559758 ip-10-0-142-6 kubenswrapper[2564]: I0730 00:19:30.559709 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e3ac38c6-9513-43cf-9af0-8e49aceca4ed-service-ca" (OuterVolumeSpecName: "service-ca") pod "e3ac38c6-9513-43cf-9af0-8e49aceca4ed" (UID: "e3ac38c6-9513-43cf-9af0-8e49aceca4ed"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 30 00:19:30.560126 ip-10-0-142-6 kubenswrapper[2564]: I0730 00:19:30.559752 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e3ac38c6-9513-43cf-9af0-8e49aceca4ed-oauth-serving-cert" (OuterVolumeSpecName: "oauth-serving-cert") pod "e3ac38c6-9513-43cf-9af0-8e49aceca4ed" (UID: "e3ac38c6-9513-43cf-9af0-8e49aceca4ed"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 30 00:19:30.560126 ip-10-0-142-6 kubenswrapper[2564]: I0730 00:19:30.559918 2564 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/e3ac38c6-9513-43cf-9af0-8e49aceca4ed-console-config\") on node \"ip-10-0-142-6.ec2.internal\" DevicePath \"\"" Jul 30 00:19:30.560126 ip-10-0-142-6 kubenswrapper[2564]: I0730 00:19:30.559938 2564 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/e3ac38c6-9513-43cf-9af0-8e49aceca4ed-oauth-serving-cert\") on node \"ip-10-0-142-6.ec2.internal\" DevicePath \"\"" Jul 30 00:19:30.560126 ip-10-0-142-6 kubenswrapper[2564]: I0730 00:19:30.559952 2564 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/e3ac38c6-9513-43cf-9af0-8e49aceca4ed-trusted-ca-bundle\") on node \"ip-10-0-142-6.ec2.internal\" DevicePath \"\"" Jul 30 00:19:30.560126 ip-10-0-142-6 kubenswrapper[2564]: I0730 00:19:30.559964 2564 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/e3ac38c6-9513-43cf-9af0-8e49aceca4ed-service-ca\") on node \"ip-10-0-142-6.ec2.internal\" DevicePath \"\"" Jul 30 00:19:30.562171 ip-10-0-142-6 kubenswrapper[2564]: I0730 00:19:30.562147 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e3ac38c6-9513-43cf-9af0-8e49aceca4ed-console-oauth-config" (OuterVolumeSpecName: "console-oauth-config") pod "e3ac38c6-9513-43cf-9af0-8e49aceca4ed" (UID: "e3ac38c6-9513-43cf-9af0-8e49aceca4ed"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 30 00:19:30.562286 ip-10-0-142-6 kubenswrapper[2564]: I0730 00:19:30.562227 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e3ac38c6-9513-43cf-9af0-8e49aceca4ed-console-serving-cert" (OuterVolumeSpecName: "console-serving-cert") pod "e3ac38c6-9513-43cf-9af0-8e49aceca4ed" (UID: "e3ac38c6-9513-43cf-9af0-8e49aceca4ed"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 30 00:19:30.562286 ip-10-0-142-6 kubenswrapper[2564]: I0730 00:19:30.562255 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e3ac38c6-9513-43cf-9af0-8e49aceca4ed-kube-api-access-ks5rb" (OuterVolumeSpecName: "kube-api-access-ks5rb") pod "e3ac38c6-9513-43cf-9af0-8e49aceca4ed" (UID: "e3ac38c6-9513-43cf-9af0-8e49aceca4ed"). InnerVolumeSpecName "kube-api-access-ks5rb". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 30 00:19:30.650962 ip-10-0-142-6 kubenswrapper[2564]: I0730 00:19:30.650924 2564 generic.go:356] "Generic (PLEG): container finished" podID="e3ac38c6-9513-43cf-9af0-8e49aceca4ed" containerID="ab843fb3b471f2ee3ffa98a73a9b109dbe32702c74d9348f4927e709b51aef82" exitCode=0 Jul 30 00:19:30.651121 ip-10-0-142-6 kubenswrapper[2564]: I0730 00:19:30.650976 2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-56d468748c-tfk4j" Jul 30 00:19:30.651121 ip-10-0-142-6 kubenswrapper[2564]: I0730 00:19:30.651004 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-56d468748c-tfk4j" event={"ID":"e3ac38c6-9513-43cf-9af0-8e49aceca4ed","Type":"ContainerDied","Data":"ab843fb3b471f2ee3ffa98a73a9b109dbe32702c74d9348f4927e709b51aef82"} Jul 30 00:19:30.651121 ip-10-0-142-6 kubenswrapper[2564]: I0730 00:19:30.651093 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-56d468748c-tfk4j" event={"ID":"e3ac38c6-9513-43cf-9af0-8e49aceca4ed","Type":"ContainerDied","Data":"453a93f544934590033f99ae23664366715c663e57476f888027bdb2b2b55fac"} Jul 30 00:19:30.651121 ip-10-0-142-6 kubenswrapper[2564]: I0730 00:19:30.651115 2564 scope.go:117] "RemoveContainer" containerID="ab843fb3b471f2ee3ffa98a73a9b109dbe32702c74d9348f4927e709b51aef82" Jul 30 00:19:30.659374 ip-10-0-142-6 kubenswrapper[2564]: I0730 00:19:30.659357 2564 scope.go:117] "RemoveContainer" containerID="ab843fb3b471f2ee3ffa98a73a9b109dbe32702c74d9348f4927e709b51aef82" Jul 30 00:19:30.659735 ip-10-0-142-6 kubenswrapper[2564]: E0730 00:19:30.659701 2564 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ab843fb3b471f2ee3ffa98a73a9b109dbe32702c74d9348f4927e709b51aef82\": container with ID starting with ab843fb3b471f2ee3ffa98a73a9b109dbe32702c74d9348f4927e709b51aef82 not found: ID does not exist" containerID="ab843fb3b471f2ee3ffa98a73a9b109dbe32702c74d9348f4927e709b51aef82" Jul 30 00:19:30.659794 ip-10-0-142-6 kubenswrapper[2564]: I0730 00:19:30.659744 2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ab843fb3b471f2ee3ffa98a73a9b109dbe32702c74d9348f4927e709b51aef82"} err="failed to get container status \"ab843fb3b471f2ee3ffa98a73a9b109dbe32702c74d9348f4927e709b51aef82\": rpc error: code = NotFound desc = could not find container \"ab843fb3b471f2ee3ffa98a73a9b109dbe32702c74d9348f4927e709b51aef82\": container with ID starting with ab843fb3b471f2ee3ffa98a73a9b109dbe32702c74d9348f4927e709b51aef82 not found: ID does not exist" Jul 30 00:19:30.661009 ip-10-0-142-6 kubenswrapper[2564]: I0730 00:19:30.660993 2564 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/e3ac38c6-9513-43cf-9af0-8e49aceca4ed-console-serving-cert\") on node \"ip-10-0-142-6.ec2.internal\" DevicePath \"\"" Jul 30 00:19:30.661058 ip-10-0-142-6 kubenswrapper[2564]: I0730 00:19:30.661011 2564 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ks5rb\" (UniqueName: \"kubernetes.io/projected/e3ac38c6-9513-43cf-9af0-8e49aceca4ed-kube-api-access-ks5rb\") on node \"ip-10-0-142-6.ec2.internal\" DevicePath \"\"" Jul 30 00:19:30.661058 ip-10-0-142-6 kubenswrapper[2564]: I0730 00:19:30.661020 2564 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/e3ac38c6-9513-43cf-9af0-8e49aceca4ed-console-oauth-config\") on node \"ip-10-0-142-6.ec2.internal\" DevicePath \"\"" Jul 30 00:19:30.670972 ip-10-0-142-6 kubenswrapper[2564]: I0730 00:19:30.670949 2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-56d468748c-tfk4j"] Jul 30 00:19:30.673088 ip-10-0-142-6 kubenswrapper[2564]: I0730 00:19:30.673050 2564 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-56d468748c-tfk4j"] Jul 30 00:19:31.138458 ip-10-0-142-6 kubenswrapper[2564]: I0730 00:19:31.138427 2564 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="e3ac38c6-9513-43cf-9af0-8e49aceca4ed" path="/var/lib/kubelet/pods/e3ac38c6-9513-43cf-9af0-8e49aceca4ed/volumes" Jul 30 00:19:41.263530 ip-10-0-142-6 kubenswrapper[2564]: I0730 00:19:41.263495 2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/telemeter-client-7d6b996d65-fvmqd"] Jul 30 00:19:41.263943 ip-10-0-142-6 kubenswrapper[2564]: I0730 00:19:41.263738 2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e3ac38c6-9513-43cf-9af0-8e49aceca4ed" containerName="console" Jul 30 00:19:41.263943 ip-10-0-142-6 kubenswrapper[2564]: I0730 00:19:41.263748 2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="e3ac38c6-9513-43cf-9af0-8e49aceca4ed" containerName="console" Jul 30 00:19:41.263943 ip-10-0-142-6 kubenswrapper[2564]: I0730 00:19:41.263832 2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="e3ac38c6-9513-43cf-9af0-8e49aceca4ed" containerName="console" Jul 30 00:19:41.265783 ip-10-0-142-6 kubenswrapper[2564]: I0730 00:19:41.265766 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/telemeter-client-7d6b996d65-fvmqd" Jul 30 00:19:41.268242 ip-10-0-142-6 kubenswrapper[2564]: I0730 00:19:41.268213 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"federate-client-certs\"" Jul 30 00:19:41.268373 ip-10-0-142-6 kubenswrapper[2564]: I0730 00:19:41.268248 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client\"" Jul 30 00:19:41.268373 ip-10-0-142-6 kubenswrapper[2564]: I0730 00:19:41.268248 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client-kube-rbac-proxy-config\"" Jul 30 00:19:41.268537 ip-10-0-142-6 kubenswrapper[2564]: I0730 00:19:41.268519 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client-tls\"" Jul 30 00:19:41.268731 ip-10-0-142-6 kubenswrapper[2564]: I0730 00:19:41.268711 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"telemeter-client-serving-certs-ca-bundle\"" Jul 30 00:19:41.268841 ip-10-0-142-6 kubenswrapper[2564]: I0730 00:19:41.268715 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client-dockercfg-pq6hl\"" Jul 30 00:19:41.273389 ip-10-0-142-6 kubenswrapper[2564]: I0730 00:19:41.273369 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"telemeter-trusted-ca-bundle-8i12ta5c71j38\"" Jul 30 00:19:41.280917 ip-10-0-142-6 kubenswrapper[2564]: I0730 00:19:41.280891 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/telemeter-client-7d6b996d65-fvmqd"] Jul 30 00:19:41.337249 ip-10-0-142-6 kubenswrapper[2564]: I0730 00:19:41.337217 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/de6ee145-7f06-4d65-84f7-e679fff20407-metrics-client-ca\") pod \"telemeter-client-7d6b996d65-fvmqd\" (UID: \"de6ee145-7f06-4d65-84f7-e679fff20407\") " pod="openshift-monitoring/telemeter-client-7d6b996d65-fvmqd" Jul 30 00:19:41.337430 ip-10-0-142-6 kubenswrapper[2564]: I0730 00:19:41.337273 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"federate-client-tls\" (UniqueName: \"kubernetes.io/secret/de6ee145-7f06-4d65-84f7-e679fff20407-federate-client-tls\") pod \"telemeter-client-7d6b996d65-fvmqd\" (UID: \"de6ee145-7f06-4d65-84f7-e679fff20407\") " pod="openshift-monitoring/telemeter-client-7d6b996d65-fvmqd" Jul 30 00:19:41.337430 ip-10-0-142-6 kubenswrapper[2564]: I0730 00:19:41.337337 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9hvjh\" (UniqueName: \"kubernetes.io/projected/de6ee145-7f06-4d65-84f7-e679fff20407-kube-api-access-9hvjh\") pod \"telemeter-client-7d6b996d65-fvmqd\" (UID: \"de6ee145-7f06-4d65-84f7-e679fff20407\") " pod="openshift-monitoring/telemeter-client-7d6b996d65-fvmqd" Jul 30 00:19:41.337430 ip-10-0-142-6 kubenswrapper[2564]: I0730 00:19:41.337373 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-telemeter-client-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/de6ee145-7f06-4d65-84f7-e679fff20407-secret-telemeter-client-kube-rbac-proxy-config\") pod \"telemeter-client-7d6b996d65-fvmqd\" (UID: \"de6ee145-7f06-4d65-84f7-e679fff20407\") " pod="openshift-monitoring/telemeter-client-7d6b996d65-fvmqd" Jul 30 00:19:41.337430 ip-10-0-142-6 kubenswrapper[2564]: I0730 00:19:41.337399 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"telemeter-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/de6ee145-7f06-4d65-84f7-e679fff20407-telemeter-trusted-ca-bundle\") pod \"telemeter-client-7d6b996d65-fvmqd\" (UID: \"de6ee145-7f06-4d65-84f7-e679fff20407\") " pod="openshift-monitoring/telemeter-client-7d6b996d65-fvmqd" Jul 30 00:19:41.337430 ip-10-0-142-6 kubenswrapper[2564]: I0730 00:19:41.337424 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/de6ee145-7f06-4d65-84f7-e679fff20407-serving-certs-ca-bundle\") pod \"telemeter-client-7d6b996d65-fvmqd\" (UID: \"de6ee145-7f06-4d65-84f7-e679fff20407\") " pod="openshift-monitoring/telemeter-client-7d6b996d65-fvmqd" Jul 30 00:19:41.337614 ip-10-0-142-6 kubenswrapper[2564]: I0730 00:19:41.337449 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"telemeter-client-tls\" (UniqueName: \"kubernetes.io/secret/de6ee145-7f06-4d65-84f7-e679fff20407-telemeter-client-tls\") pod \"telemeter-client-7d6b996d65-fvmqd\" (UID: \"de6ee145-7f06-4d65-84f7-e679fff20407\") " pod="openshift-monitoring/telemeter-client-7d6b996d65-fvmqd" Jul 30 00:19:41.337614 ip-10-0-142-6 kubenswrapper[2564]: I0730 00:19:41.337465 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-telemeter-client\" (UniqueName: \"kubernetes.io/secret/de6ee145-7f06-4d65-84f7-e679fff20407-secret-telemeter-client\") pod \"telemeter-client-7d6b996d65-fvmqd\" (UID: \"de6ee145-7f06-4d65-84f7-e679fff20407\") " pod="openshift-monitoring/telemeter-client-7d6b996d65-fvmqd" Jul 30 00:19:41.437819 ip-10-0-142-6 kubenswrapper[2564]: I0730 00:19:41.437788 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"telemeter-client-tls\" (UniqueName: \"kubernetes.io/secret/de6ee145-7f06-4d65-84f7-e679fff20407-telemeter-client-tls\") pod \"telemeter-client-7d6b996d65-fvmqd\" (UID: \"de6ee145-7f06-4d65-84f7-e679fff20407\") " pod="openshift-monitoring/telemeter-client-7d6b996d65-fvmqd" Jul 30 00:19:41.437819 ip-10-0-142-6 kubenswrapper[2564]: I0730 00:19:41.437826 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-telemeter-client\" (UniqueName: \"kubernetes.io/secret/de6ee145-7f06-4d65-84f7-e679fff20407-secret-telemeter-client\") pod \"telemeter-client-7d6b996d65-fvmqd\" (UID: \"de6ee145-7f06-4d65-84f7-e679fff20407\") " pod="openshift-monitoring/telemeter-client-7d6b996d65-fvmqd" Jul 30 00:19:41.438124 ip-10-0-142-6 kubenswrapper[2564]: I0730 00:19:41.437873 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/de6ee145-7f06-4d65-84f7-e679fff20407-metrics-client-ca\") pod \"telemeter-client-7d6b996d65-fvmqd\" (UID: \"de6ee145-7f06-4d65-84f7-e679fff20407\") " pod="openshift-monitoring/telemeter-client-7d6b996d65-fvmqd" Jul 30 00:19:41.438124 ip-10-0-142-6 kubenswrapper[2564]: I0730 00:19:41.437919 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"federate-client-tls\" (UniqueName: \"kubernetes.io/secret/de6ee145-7f06-4d65-84f7-e679fff20407-federate-client-tls\") pod \"telemeter-client-7d6b996d65-fvmqd\" (UID: \"de6ee145-7f06-4d65-84f7-e679fff20407\") " pod="openshift-monitoring/telemeter-client-7d6b996d65-fvmqd" Jul 30 00:19:41.438124 ip-10-0-142-6 kubenswrapper[2564]: I0730 00:19:41.437945 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9hvjh\" (UniqueName: \"kubernetes.io/projected/de6ee145-7f06-4d65-84f7-e679fff20407-kube-api-access-9hvjh\") pod \"telemeter-client-7d6b996d65-fvmqd\" (UID: \"de6ee145-7f06-4d65-84f7-e679fff20407\") " pod="openshift-monitoring/telemeter-client-7d6b996d65-fvmqd" Jul 30 00:19:41.438124 ip-10-0-142-6 kubenswrapper[2564]: I0730 00:19:41.437973 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-telemeter-client-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/de6ee145-7f06-4d65-84f7-e679fff20407-secret-telemeter-client-kube-rbac-proxy-config\") pod \"telemeter-client-7d6b996d65-fvmqd\" (UID: \"de6ee145-7f06-4d65-84f7-e679fff20407\") " pod="openshift-monitoring/telemeter-client-7d6b996d65-fvmqd" Jul 30 00:19:41.438124 ip-10-0-142-6 kubenswrapper[2564]: I0730 00:19:41.438000 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"telemeter-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/de6ee145-7f06-4d65-84f7-e679fff20407-telemeter-trusted-ca-bundle\") pod \"telemeter-client-7d6b996d65-fvmqd\" (UID: \"de6ee145-7f06-4d65-84f7-e679fff20407\") " pod="openshift-monitoring/telemeter-client-7d6b996d65-fvmqd" Jul 30 00:19:41.438124 ip-10-0-142-6 kubenswrapper[2564]: I0730 00:19:41.438025 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/de6ee145-7f06-4d65-84f7-e679fff20407-serving-certs-ca-bundle\") pod \"telemeter-client-7d6b996d65-fvmqd\" (UID: \"de6ee145-7f06-4d65-84f7-e679fff20407\") " pod="openshift-monitoring/telemeter-client-7d6b996d65-fvmqd" Jul 30 00:19:41.438759 ip-10-0-142-6 kubenswrapper[2564]: I0730 00:19:41.438731 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/de6ee145-7f06-4d65-84f7-e679fff20407-serving-certs-ca-bundle\") pod \"telemeter-client-7d6b996d65-fvmqd\" (UID: \"de6ee145-7f06-4d65-84f7-e679fff20407\") " pod="openshift-monitoring/telemeter-client-7d6b996d65-fvmqd" Jul 30 00:19:41.438863 ip-10-0-142-6 kubenswrapper[2564]: I0730 00:19:41.438781 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/de6ee145-7f06-4d65-84f7-e679fff20407-metrics-client-ca\") pod \"telemeter-client-7d6b996d65-fvmqd\" (UID: \"de6ee145-7f06-4d65-84f7-e679fff20407\") " pod="openshift-monitoring/telemeter-client-7d6b996d65-fvmqd" Jul 30 00:19:41.439132 ip-10-0-142-6 kubenswrapper[2564]: I0730 00:19:41.439111 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"telemeter-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/de6ee145-7f06-4d65-84f7-e679fff20407-telemeter-trusted-ca-bundle\") pod \"telemeter-client-7d6b996d65-fvmqd\" (UID: \"de6ee145-7f06-4d65-84f7-e679fff20407\") " pod="openshift-monitoring/telemeter-client-7d6b996d65-fvmqd" Jul 30 00:19:41.440475 ip-10-0-142-6 kubenswrapper[2564]: I0730 00:19:41.440455 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-telemeter-client\" (UniqueName: \"kubernetes.io/secret/de6ee145-7f06-4d65-84f7-e679fff20407-secret-telemeter-client\") pod \"telemeter-client-7d6b996d65-fvmqd\" (UID: \"de6ee145-7f06-4d65-84f7-e679fff20407\") " pod="openshift-monitoring/telemeter-client-7d6b996d65-fvmqd" Jul 30 00:19:41.440608 ip-10-0-142-6 kubenswrapper[2564]: I0730 00:19:41.440588 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-telemeter-client-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/de6ee145-7f06-4d65-84f7-e679fff20407-secret-telemeter-client-kube-rbac-proxy-config\") pod \"telemeter-client-7d6b996d65-fvmqd\" (UID: \"de6ee145-7f06-4d65-84f7-e679fff20407\") " pod="openshift-monitoring/telemeter-client-7d6b996d65-fvmqd" Jul 30 00:19:41.440768 ip-10-0-142-6 kubenswrapper[2564]: I0730 00:19:41.440752 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"federate-client-tls\" (UniqueName: \"kubernetes.io/secret/de6ee145-7f06-4d65-84f7-e679fff20407-federate-client-tls\") pod \"telemeter-client-7d6b996d65-fvmqd\" (UID: \"de6ee145-7f06-4d65-84f7-e679fff20407\") " pod="openshift-monitoring/telemeter-client-7d6b996d65-fvmqd" Jul 30 00:19:41.440826 ip-10-0-142-6 kubenswrapper[2564]: I0730 00:19:41.440813 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"telemeter-client-tls\" (UniqueName: \"kubernetes.io/secret/de6ee145-7f06-4d65-84f7-e679fff20407-telemeter-client-tls\") pod \"telemeter-client-7d6b996d65-fvmqd\" (UID: \"de6ee145-7f06-4d65-84f7-e679fff20407\") " pod="openshift-monitoring/telemeter-client-7d6b996d65-fvmqd" Jul 30 00:19:41.445728 ip-10-0-142-6 kubenswrapper[2564]: I0730 00:19:41.445703 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9hvjh\" (UniqueName: \"kubernetes.io/projected/de6ee145-7f06-4d65-84f7-e679fff20407-kube-api-access-9hvjh\") pod \"telemeter-client-7d6b996d65-fvmqd\" (UID: \"de6ee145-7f06-4d65-84f7-e679fff20407\") " pod="openshift-monitoring/telemeter-client-7d6b996d65-fvmqd" Jul 30 00:19:41.575266 ip-10-0-142-6 kubenswrapper[2564]: I0730 00:19:41.575176 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/telemeter-client-7d6b996d65-fvmqd" Jul 30 00:19:41.704762 ip-10-0-142-6 kubenswrapper[2564]: I0730 00:19:41.704718 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/telemeter-client-7d6b996d65-fvmqd"] Jul 30 00:19:41.708779 ip-10-0-142-6 kubenswrapper[2564]: W0730 00:19:41.708750 2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podde6ee145_7f06_4d65_84f7_e679fff20407.slice/crio-e0bbd1e9f982a92de6f7ae4042de5dafb32daaa04c16cae5037f4c9468f26490 WatchSource:0}: Error finding container e0bbd1e9f982a92de6f7ae4042de5dafb32daaa04c16cae5037f4c9468f26490: Status 404 returned error can't find the container with id e0bbd1e9f982a92de6f7ae4042de5dafb32daaa04c16cae5037f4c9468f26490 Jul 30 00:19:42.694688 ip-10-0-142-6 kubenswrapper[2564]: I0730 00:19:42.694652 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-7d6b996d65-fvmqd" event={"ID":"de6ee145-7f06-4d65-84f7-e679fff20407","Type":"ContainerStarted","Data":"e0bbd1e9f982a92de6f7ae4042de5dafb32daaa04c16cae5037f4c9468f26490"} Jul 30 00:19:44.703394 ip-10-0-142-6 kubenswrapper[2564]: I0730 00:19:44.703312 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-7d6b996d65-fvmqd" event={"ID":"de6ee145-7f06-4d65-84f7-e679fff20407","Type":"ContainerStarted","Data":"581178c3d03a98924f24781c5d532e839e518438c238fb55ee0bf32d784b61b2"} Jul 30 00:19:45.709246 ip-10-0-142-6 kubenswrapper[2564]: I0730 00:19:45.709207 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-7d6b996d65-fvmqd" event={"ID":"de6ee145-7f06-4d65-84f7-e679fff20407","Type":"ContainerStarted","Data":"5eb4a9cc05c613c396fdae1abdb4bf90f9bfcf2573b146b06d0cd6382933f34c"} Jul 30 00:19:45.709625 ip-10-0-142-6 kubenswrapper[2564]: I0730 00:19:45.709261 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-7d6b996d65-fvmqd" event={"ID":"de6ee145-7f06-4d65-84f7-e679fff20407","Type":"ContainerStarted","Data":"8d533227a55ff97d99adc0f13fdff72653e35e5ebf37359663ea49239789944d"} Jul 30 00:19:45.731177 ip-10-0-142-6 kubenswrapper[2564]: I0730 00:19:45.731065 2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/telemeter-client-7d6b996d65-fvmqd" podStartSLOduration=1.036722966 podStartE2EDuration="4.731044818s" totalImagesPullingTime="3.694321852s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-30 00:19:41 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-30 00:19:45.730009188 +0000 UTC m=+117.144749121" watchObservedRunningTime="2026-07-30 00:19:45.731044818 +0000 UTC m=+117.145784774" Jul 30 00:19:46.666741 ip-10-0-142-6 kubenswrapper[2564]: I0730 00:19:46.666707 2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-66bc4cd48c-57dmw"] Jul 30 00:19:46.669513 ip-10-0-142-6 kubenswrapper[2564]: I0730 00:19:46.669493 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-66bc4cd48c-57dmw" Jul 30 00:19:46.682008 ip-10-0-142-6 kubenswrapper[2564]: I0730 00:19:46.681979 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-66bc4cd48c-57dmw"] Jul 30 00:19:46.688976 ip-10-0-142-6 kubenswrapper[2564]: I0730 00:19:46.688952 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/79e0b7f8-d0ec-47d6-85fb-d79aeebcfc17-console-serving-cert\") pod \"console-66bc4cd48c-57dmw\" (UID: \"79e0b7f8-d0ec-47d6-85fb-d79aeebcfc17\") " pod="openshift-console/console-66bc4cd48c-57dmw" Jul 30 00:19:46.689127 ip-10-0-142-6 kubenswrapper[2564]: I0730 00:19:46.688990 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/79e0b7f8-d0ec-47d6-85fb-d79aeebcfc17-trusted-ca-bundle\") pod \"console-66bc4cd48c-57dmw\" (UID: \"79e0b7f8-d0ec-47d6-85fb-d79aeebcfc17\") " pod="openshift-console/console-66bc4cd48c-57dmw" Jul 30 00:19:46.689127 ip-10-0-142-6 kubenswrapper[2564]: I0730 00:19:46.689025 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/79e0b7f8-d0ec-47d6-85fb-d79aeebcfc17-console-oauth-config\") pod \"console-66bc4cd48c-57dmw\" (UID: \"79e0b7f8-d0ec-47d6-85fb-d79aeebcfc17\") " pod="openshift-console/console-66bc4cd48c-57dmw" Jul 30 00:19:46.689127 ip-10-0-142-6 kubenswrapper[2564]: I0730 00:19:46.689055 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/79e0b7f8-d0ec-47d6-85fb-d79aeebcfc17-console-config\") pod \"console-66bc4cd48c-57dmw\" (UID: \"79e0b7f8-d0ec-47d6-85fb-d79aeebcfc17\") " pod="openshift-console/console-66bc4cd48c-57dmw" Jul 30 00:19:46.689127 ip-10-0-142-6 kubenswrapper[2564]: I0730 00:19:46.689119 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-csrw2\" (UniqueName: \"kubernetes.io/projected/79e0b7f8-d0ec-47d6-85fb-d79aeebcfc17-kube-api-access-csrw2\") pod \"console-66bc4cd48c-57dmw\" (UID: \"79e0b7f8-d0ec-47d6-85fb-d79aeebcfc17\") " pod="openshift-console/console-66bc4cd48c-57dmw" Jul 30 00:19:46.689279 ip-10-0-142-6 kubenswrapper[2564]: I0730 00:19:46.689150 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/79e0b7f8-d0ec-47d6-85fb-d79aeebcfc17-oauth-serving-cert\") pod \"console-66bc4cd48c-57dmw\" (UID: \"79e0b7f8-d0ec-47d6-85fb-d79aeebcfc17\") " pod="openshift-console/console-66bc4cd48c-57dmw" Jul 30 00:19:46.689279 ip-10-0-142-6 kubenswrapper[2564]: I0730 00:19:46.689216 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/79e0b7f8-d0ec-47d6-85fb-d79aeebcfc17-service-ca\") pod \"console-66bc4cd48c-57dmw\" (UID: \"79e0b7f8-d0ec-47d6-85fb-d79aeebcfc17\") " pod="openshift-console/console-66bc4cd48c-57dmw" Jul 30 00:19:46.790684 ip-10-0-142-6 kubenswrapper[2564]: I0730 00:19:46.790642 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/79e0b7f8-d0ec-47d6-85fb-d79aeebcfc17-console-serving-cert\") pod \"console-66bc4cd48c-57dmw\" (UID: \"79e0b7f8-d0ec-47d6-85fb-d79aeebcfc17\") " pod="openshift-console/console-66bc4cd48c-57dmw" Jul 30 00:19:46.791226 ip-10-0-142-6 kubenswrapper[2564]: I0730 00:19:46.790690 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/79e0b7f8-d0ec-47d6-85fb-d79aeebcfc17-trusted-ca-bundle\") pod \"console-66bc4cd48c-57dmw\" (UID: \"79e0b7f8-d0ec-47d6-85fb-d79aeebcfc17\") " pod="openshift-console/console-66bc4cd48c-57dmw" Jul 30 00:19:46.791226 ip-10-0-142-6 kubenswrapper[2564]: I0730 00:19:46.790719 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/79e0b7f8-d0ec-47d6-85fb-d79aeebcfc17-console-oauth-config\") pod \"console-66bc4cd48c-57dmw\" (UID: \"79e0b7f8-d0ec-47d6-85fb-d79aeebcfc17\") " pod="openshift-console/console-66bc4cd48c-57dmw" Jul 30 00:19:46.791226 ip-10-0-142-6 kubenswrapper[2564]: I0730 00:19:46.790748 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/79e0b7f8-d0ec-47d6-85fb-d79aeebcfc17-console-config\") pod \"console-66bc4cd48c-57dmw\" (UID: \"79e0b7f8-d0ec-47d6-85fb-d79aeebcfc17\") " pod="openshift-console/console-66bc4cd48c-57dmw" Jul 30 00:19:46.791226 ip-10-0-142-6 kubenswrapper[2564]: I0730 00:19:46.790796 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-csrw2\" (UniqueName: \"kubernetes.io/projected/79e0b7f8-d0ec-47d6-85fb-d79aeebcfc17-kube-api-access-csrw2\") pod \"console-66bc4cd48c-57dmw\" (UID: \"79e0b7f8-d0ec-47d6-85fb-d79aeebcfc17\") " pod="openshift-console/console-66bc4cd48c-57dmw" Jul 30 00:19:46.791226 ip-10-0-142-6 kubenswrapper[2564]: I0730 00:19:46.790848 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/79e0b7f8-d0ec-47d6-85fb-d79aeebcfc17-oauth-serving-cert\") pod \"console-66bc4cd48c-57dmw\" (UID: \"79e0b7f8-d0ec-47d6-85fb-d79aeebcfc17\") " pod="openshift-console/console-66bc4cd48c-57dmw" Jul 30 00:19:46.791226 ip-10-0-142-6 kubenswrapper[2564]: I0730 00:19:46.791136 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/79e0b7f8-d0ec-47d6-85fb-d79aeebcfc17-service-ca\") pod \"console-66bc4cd48c-57dmw\" (UID: \"79e0b7f8-d0ec-47d6-85fb-d79aeebcfc17\") " pod="openshift-console/console-66bc4cd48c-57dmw" Jul 30 00:19:46.791551 ip-10-0-142-6 kubenswrapper[2564]: I0730 00:19:46.791534 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/79e0b7f8-d0ec-47d6-85fb-d79aeebcfc17-oauth-serving-cert\") pod \"console-66bc4cd48c-57dmw\" (UID: \"79e0b7f8-d0ec-47d6-85fb-d79aeebcfc17\") " pod="openshift-console/console-66bc4cd48c-57dmw" Jul 30 00:19:46.791794 ip-10-0-142-6 kubenswrapper[2564]: I0730 00:19:46.791766 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/79e0b7f8-d0ec-47d6-85fb-d79aeebcfc17-service-ca\") pod \"console-66bc4cd48c-57dmw\" (UID: \"79e0b7f8-d0ec-47d6-85fb-d79aeebcfc17\") " pod="openshift-console/console-66bc4cd48c-57dmw" Jul 30 00:19:46.791924 ip-10-0-142-6 kubenswrapper[2564]: I0730 00:19:46.791774 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/79e0b7f8-d0ec-47d6-85fb-d79aeebcfc17-console-config\") pod \"console-66bc4cd48c-57dmw\" (UID: \"79e0b7f8-d0ec-47d6-85fb-d79aeebcfc17\") " pod="openshift-console/console-66bc4cd48c-57dmw" Jul 30 00:19:46.791983 ip-10-0-142-6 kubenswrapper[2564]: I0730 00:19:46.791939 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/79e0b7f8-d0ec-47d6-85fb-d79aeebcfc17-trusted-ca-bundle\") pod \"console-66bc4cd48c-57dmw\" (UID: \"79e0b7f8-d0ec-47d6-85fb-d79aeebcfc17\") " pod="openshift-console/console-66bc4cd48c-57dmw" Jul 30 00:19:46.793349 ip-10-0-142-6 kubenswrapper[2564]: I0730 00:19:46.793324 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/79e0b7f8-d0ec-47d6-85fb-d79aeebcfc17-console-oauth-config\") pod \"console-66bc4cd48c-57dmw\" (UID: \"79e0b7f8-d0ec-47d6-85fb-d79aeebcfc17\") " pod="openshift-console/console-66bc4cd48c-57dmw" Jul 30 00:19:46.793513 ip-10-0-142-6 kubenswrapper[2564]: I0730 00:19:46.793493 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/79e0b7f8-d0ec-47d6-85fb-d79aeebcfc17-console-serving-cert\") pod \"console-66bc4cd48c-57dmw\" (UID: \"79e0b7f8-d0ec-47d6-85fb-d79aeebcfc17\") " pod="openshift-console/console-66bc4cd48c-57dmw" Jul 30 00:19:46.798743 ip-10-0-142-6 kubenswrapper[2564]: I0730 00:19:46.798721 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-csrw2\" (UniqueName: \"kubernetes.io/projected/79e0b7f8-d0ec-47d6-85fb-d79aeebcfc17-kube-api-access-csrw2\") pod \"console-66bc4cd48c-57dmw\" (UID: \"79e0b7f8-d0ec-47d6-85fb-d79aeebcfc17\") " pod="openshift-console/console-66bc4cd48c-57dmw" Jul 30 00:19:46.977706 ip-10-0-142-6 kubenswrapper[2564]: I0730 00:19:46.977624 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-66bc4cd48c-57dmw" Jul 30 00:19:47.107303 ip-10-0-142-6 kubenswrapper[2564]: I0730 00:19:47.107274 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-66bc4cd48c-57dmw"] Jul 30 00:19:47.109612 ip-10-0-142-6 kubenswrapper[2564]: W0730 00:19:47.109582 2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod79e0b7f8_d0ec_47d6_85fb_d79aeebcfc17.slice/crio-648ee5696a029294789f3d1a1d26f396247264fe5a9c35455900bd1084af5d4d WatchSource:0}: Error finding container 648ee5696a029294789f3d1a1d26f396247264fe5a9c35455900bd1084af5d4d: Status 404 returned error can't find the container with id 648ee5696a029294789f3d1a1d26f396247264fe5a9c35455900bd1084af5d4d Jul 30 00:19:47.718047 ip-10-0-142-6 kubenswrapper[2564]: I0730 00:19:47.718011 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-66bc4cd48c-57dmw" event={"ID":"79e0b7f8-d0ec-47d6-85fb-d79aeebcfc17","Type":"ContainerStarted","Data":"2d7c8fc950da762585bca3881a3aa2d4c92d8e58f9f956e9d27c7601e5d726e1"} Jul 30 00:19:47.718237 ip-10-0-142-6 kubenswrapper[2564]: I0730 00:19:47.718068 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-66bc4cd48c-57dmw" event={"ID":"79e0b7f8-d0ec-47d6-85fb-d79aeebcfc17","Type":"ContainerStarted","Data":"648ee5696a029294789f3d1a1d26f396247264fe5a9c35455900bd1084af5d4d"} Jul 30 00:19:47.734953 ip-10-0-142-6 kubenswrapper[2564]: I0730 00:19:47.734903 2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-66bc4cd48c-57dmw" podStartSLOduration=1.7348865770000002 podStartE2EDuration="1.734886577s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-30 00:19:46 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-30 00:19:47.733701347 +0000 UTC m=+119.148441284" watchObservedRunningTime="2026-07-30 00:19:47.734886577 +0000 UTC m=+119.149626510" Jul 30 00:19:56.977975 ip-10-0-142-6 kubenswrapper[2564]: I0730 00:19:56.977919 2564 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-66bc4cd48c-57dmw" Jul 30 00:19:56.978402 ip-10-0-142-6 kubenswrapper[2564]: I0730 00:19:56.977999 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-66bc4cd48c-57dmw" Jul 30 00:19:56.982678 ip-10-0-142-6 kubenswrapper[2564]: I0730 00:19:56.982651 2564 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-66bc4cd48c-57dmw" Jul 30 00:19:57.757832 ip-10-0-142-6 kubenswrapper[2564]: I0730 00:19:57.757805 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-66bc4cd48c-57dmw" Jul 30 00:19:57.803029 ip-10-0-142-6 kubenswrapper[2564]: I0730 00:19:57.802996 2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-7594cdd77f-4n25d"] Jul 30 00:20:22.821879 ip-10-0-142-6 kubenswrapper[2564]: I0730 00:20:22.821830 2564 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-console/console-7594cdd77f-4n25d" podUID="07979fe1-67ca-4210-9aa8-86cf58d3642b" containerName="console" containerID="cri-o://cd003bf956eb3c6219ce631a55c395e27a16afcd60322691b2aad62860b3f430" gracePeriod=15 Jul 30 00:20:23.054507 ip-10-0-142-6 kubenswrapper[2564]: I0730 00:20:23.054484 2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-7594cdd77f-4n25d" Jul 30 00:20:23.170431 ip-10-0-142-6 kubenswrapper[2564]: I0730 00:20:23.170398 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/07979fe1-67ca-4210-9aa8-86cf58d3642b-oauth-serving-cert\") pod \"07979fe1-67ca-4210-9aa8-86cf58d3642b\" (UID: \"07979fe1-67ca-4210-9aa8-86cf58d3642b\") " Jul 30 00:20:23.170610 ip-10-0-142-6 kubenswrapper[2564]: I0730 00:20:23.170448 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/07979fe1-67ca-4210-9aa8-86cf58d3642b-trusted-ca-bundle\") pod \"07979fe1-67ca-4210-9aa8-86cf58d3642b\" (UID: \"07979fe1-67ca-4210-9aa8-86cf58d3642b\") " Jul 30 00:20:23.170610 ip-10-0-142-6 kubenswrapper[2564]: I0730 00:20:23.170473 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/07979fe1-67ca-4210-9aa8-86cf58d3642b-console-oauth-config\") pod \"07979fe1-67ca-4210-9aa8-86cf58d3642b\" (UID: \"07979fe1-67ca-4210-9aa8-86cf58d3642b\") " Jul 30 00:20:23.170610 ip-10-0-142-6 kubenswrapper[2564]: I0730 00:20:23.170498 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/07979fe1-67ca-4210-9aa8-86cf58d3642b-console-serving-cert\") pod \"07979fe1-67ca-4210-9aa8-86cf58d3642b\" (UID: \"07979fe1-67ca-4210-9aa8-86cf58d3642b\") " Jul 30 00:20:23.170610 ip-10-0-142-6 kubenswrapper[2564]: I0730 00:20:23.170552 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-ghzd9\" (UniqueName: \"kubernetes.io/projected/07979fe1-67ca-4210-9aa8-86cf58d3642b-kube-api-access-ghzd9\") pod \"07979fe1-67ca-4210-9aa8-86cf58d3642b\" (UID: \"07979fe1-67ca-4210-9aa8-86cf58d3642b\") " Jul 30 00:20:23.170610 ip-10-0-142-6 kubenswrapper[2564]: I0730 00:20:23.170586 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/07979fe1-67ca-4210-9aa8-86cf58d3642b-console-config\") pod \"07979fe1-67ca-4210-9aa8-86cf58d3642b\" (UID: \"07979fe1-67ca-4210-9aa8-86cf58d3642b\") " Jul 30 00:20:23.170865 ip-10-0-142-6 kubenswrapper[2564]: I0730 00:20:23.170614 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/07979fe1-67ca-4210-9aa8-86cf58d3642b-service-ca\") pod \"07979fe1-67ca-4210-9aa8-86cf58d3642b\" (UID: \"07979fe1-67ca-4210-9aa8-86cf58d3642b\") " Jul 30 00:20:23.170956 ip-10-0-142-6 kubenswrapper[2564]: I0730 00:20:23.170925 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/07979fe1-67ca-4210-9aa8-86cf58d3642b-trusted-ca-bundle" (OuterVolumeSpecName: "trusted-ca-bundle") pod "07979fe1-67ca-4210-9aa8-86cf58d3642b" (UID: "07979fe1-67ca-4210-9aa8-86cf58d3642b"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 30 00:20:23.171119 ip-10-0-142-6 kubenswrapper[2564]: I0730 00:20:23.171067 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/07979fe1-67ca-4210-9aa8-86cf58d3642b-oauth-serving-cert" (OuterVolumeSpecName: "oauth-serving-cert") pod "07979fe1-67ca-4210-9aa8-86cf58d3642b" (UID: "07979fe1-67ca-4210-9aa8-86cf58d3642b"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 30 00:20:23.171214 ip-10-0-142-6 kubenswrapper[2564]: I0730 00:20:23.171096 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/07979fe1-67ca-4210-9aa8-86cf58d3642b-console-config" (OuterVolumeSpecName: "console-config") pod "07979fe1-67ca-4210-9aa8-86cf58d3642b" (UID: "07979fe1-67ca-4210-9aa8-86cf58d3642b"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 30 00:20:23.171214 ip-10-0-142-6 kubenswrapper[2564]: I0730 00:20:23.171119 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/07979fe1-67ca-4210-9aa8-86cf58d3642b-service-ca" (OuterVolumeSpecName: "service-ca") pod "07979fe1-67ca-4210-9aa8-86cf58d3642b" (UID: "07979fe1-67ca-4210-9aa8-86cf58d3642b"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 30 00:20:23.172715 ip-10-0-142-6 kubenswrapper[2564]: I0730 00:20:23.172691 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/07979fe1-67ca-4210-9aa8-86cf58d3642b-console-serving-cert" (OuterVolumeSpecName: "console-serving-cert") pod "07979fe1-67ca-4210-9aa8-86cf58d3642b" (UID: "07979fe1-67ca-4210-9aa8-86cf58d3642b"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 30 00:20:23.172817 ip-10-0-142-6 kubenswrapper[2564]: I0730 00:20:23.172733 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/07979fe1-67ca-4210-9aa8-86cf58d3642b-kube-api-access-ghzd9" (OuterVolumeSpecName: "kube-api-access-ghzd9") pod "07979fe1-67ca-4210-9aa8-86cf58d3642b" (UID: "07979fe1-67ca-4210-9aa8-86cf58d3642b"). InnerVolumeSpecName "kube-api-access-ghzd9". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 30 00:20:23.172817 ip-10-0-142-6 kubenswrapper[2564]: I0730 00:20:23.172748 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/07979fe1-67ca-4210-9aa8-86cf58d3642b-console-oauth-config" (OuterVolumeSpecName: "console-oauth-config") pod "07979fe1-67ca-4210-9aa8-86cf58d3642b" (UID: "07979fe1-67ca-4210-9aa8-86cf58d3642b"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 30 00:20:23.272205 ip-10-0-142-6 kubenswrapper[2564]: I0730 00:20:23.272169 2564 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/07979fe1-67ca-4210-9aa8-86cf58d3642b-oauth-serving-cert\") on node \"ip-10-0-142-6.ec2.internal\" DevicePath \"\"" Jul 30 00:20:23.272205 ip-10-0-142-6 kubenswrapper[2564]: I0730 00:20:23.272199 2564 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/07979fe1-67ca-4210-9aa8-86cf58d3642b-trusted-ca-bundle\") on node \"ip-10-0-142-6.ec2.internal\" DevicePath \"\"" Jul 30 00:20:23.272205 ip-10-0-142-6 kubenswrapper[2564]: I0730 00:20:23.272212 2564 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/07979fe1-67ca-4210-9aa8-86cf58d3642b-console-oauth-config\") on node \"ip-10-0-142-6.ec2.internal\" DevicePath \"\"" Jul 30 00:20:23.272426 ip-10-0-142-6 kubenswrapper[2564]: I0730 00:20:23.272224 2564 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/07979fe1-67ca-4210-9aa8-86cf58d3642b-console-serving-cert\") on node \"ip-10-0-142-6.ec2.internal\" DevicePath \"\"" Jul 30 00:20:23.272426 ip-10-0-142-6 kubenswrapper[2564]: I0730 00:20:23.272236 2564 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ghzd9\" (UniqueName: \"kubernetes.io/projected/07979fe1-67ca-4210-9aa8-86cf58d3642b-kube-api-access-ghzd9\") on node \"ip-10-0-142-6.ec2.internal\" DevicePath \"\"" Jul 30 00:20:23.272426 ip-10-0-142-6 kubenswrapper[2564]: I0730 00:20:23.272248 2564 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/07979fe1-67ca-4210-9aa8-86cf58d3642b-console-config\") on node \"ip-10-0-142-6.ec2.internal\" DevicePath \"\"" Jul 30 00:20:23.272426 ip-10-0-142-6 kubenswrapper[2564]: I0730 00:20:23.272259 2564 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/07979fe1-67ca-4210-9aa8-86cf58d3642b-service-ca\") on node \"ip-10-0-142-6.ec2.internal\" DevicePath \"\"" Jul 30 00:20:23.842763 ip-10-0-142-6 kubenswrapper[2564]: I0730 00:20:23.842730 2564 generic.go:356] "Generic (PLEG): container finished" podID="07979fe1-67ca-4210-9aa8-86cf58d3642b" containerID="cd003bf956eb3c6219ce631a55c395e27a16afcd60322691b2aad62860b3f430" exitCode=0 Jul 30 00:20:23.843160 ip-10-0-142-6 kubenswrapper[2564]: I0730 00:20:23.842774 2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-7594cdd77f-4n25d" Jul 30 00:20:23.843160 ip-10-0-142-6 kubenswrapper[2564]: I0730 00:20:23.842815 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-7594cdd77f-4n25d" event={"ID":"07979fe1-67ca-4210-9aa8-86cf58d3642b","Type":"ContainerDied","Data":"cd003bf956eb3c6219ce631a55c395e27a16afcd60322691b2aad62860b3f430"} Jul 30 00:20:23.843160 ip-10-0-142-6 kubenswrapper[2564]: I0730 00:20:23.842867 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-7594cdd77f-4n25d" event={"ID":"07979fe1-67ca-4210-9aa8-86cf58d3642b","Type":"ContainerDied","Data":"e93762af6ece6f3e89d9429bec22bbedd3a13c370a9579b38529e1cefc8654a9"} Jul 30 00:20:23.843160 ip-10-0-142-6 kubenswrapper[2564]: I0730 00:20:23.842889 2564 scope.go:117] "RemoveContainer" containerID="cd003bf956eb3c6219ce631a55c395e27a16afcd60322691b2aad62860b3f430" Jul 30 00:20:23.851512 ip-10-0-142-6 kubenswrapper[2564]: I0730 00:20:23.851497 2564 scope.go:117] "RemoveContainer" containerID="cd003bf956eb3c6219ce631a55c395e27a16afcd60322691b2aad62860b3f430" Jul 30 00:20:23.851808 ip-10-0-142-6 kubenswrapper[2564]: E0730 00:20:23.851789 2564 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"cd003bf956eb3c6219ce631a55c395e27a16afcd60322691b2aad62860b3f430\": container with ID starting with cd003bf956eb3c6219ce631a55c395e27a16afcd60322691b2aad62860b3f430 not found: ID does not exist" containerID="cd003bf956eb3c6219ce631a55c395e27a16afcd60322691b2aad62860b3f430" Jul 30 00:20:23.851853 ip-10-0-142-6 kubenswrapper[2564]: I0730 00:20:23.851817 2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cd003bf956eb3c6219ce631a55c395e27a16afcd60322691b2aad62860b3f430"} err="failed to get container status \"cd003bf956eb3c6219ce631a55c395e27a16afcd60322691b2aad62860b3f430\": rpc error: code = NotFound desc = could not find container \"cd003bf956eb3c6219ce631a55c395e27a16afcd60322691b2aad62860b3f430\": container with ID starting with cd003bf956eb3c6219ce631a55c395e27a16afcd60322691b2aad62860b3f430 not found: ID does not exist" Jul 30 00:20:23.861850 ip-10-0-142-6 kubenswrapper[2564]: I0730 00:20:23.861827 2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-7594cdd77f-4n25d"] Jul 30 00:20:23.865367 ip-10-0-142-6 kubenswrapper[2564]: I0730 00:20:23.865345 2564 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-7594cdd77f-4n25d"] Jul 30 00:20:25.138373 ip-10-0-142-6 kubenswrapper[2564]: I0730 00:20:25.138332 2564 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="07979fe1-67ca-4210-9aa8-86cf58d3642b" path="/var/lib/kubelet/pods/07979fe1-67ca-4210-9aa8-86cf58d3642b/volumes" Jul 30 00:20:58.860947 ip-10-0-142-6 kubenswrapper[2564]: I0730 00:20:58.860910 2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-59f8c89778-p7jds"] Jul 30 00:20:58.861476 ip-10-0-142-6 kubenswrapper[2564]: I0730 00:20:58.861200 2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="07979fe1-67ca-4210-9aa8-86cf58d3642b" containerName="console" Jul 30 00:20:58.861476 ip-10-0-142-6 kubenswrapper[2564]: I0730 00:20:58.861211 2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="07979fe1-67ca-4210-9aa8-86cf58d3642b" containerName="console" Jul 30 00:20:58.861476 ip-10-0-142-6 kubenswrapper[2564]: I0730 00:20:58.861256 2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="07979fe1-67ca-4210-9aa8-86cf58d3642b" containerName="console" Jul 30 00:20:58.865183 ip-10-0-142-6 kubenswrapper[2564]: I0730 00:20:58.865167 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-59f8c89778-p7jds" Jul 30 00:20:58.876481 ip-10-0-142-6 kubenswrapper[2564]: I0730 00:20:58.876454 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-59f8c89778-p7jds"] Jul 30 00:20:58.948788 ip-10-0-142-6 kubenswrapper[2564]: I0730 00:20:58.948751 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/8a7dcdc5-5d0c-4188-a64c-3813a16ba4c5-service-ca\") pod \"console-59f8c89778-p7jds\" (UID: \"8a7dcdc5-5d0c-4188-a64c-3813a16ba4c5\") " pod="openshift-console/console-59f8c89778-p7jds" Jul 30 00:20:58.948964 ip-10-0-142-6 kubenswrapper[2564]: I0730 00:20:58.948798 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/8a7dcdc5-5d0c-4188-a64c-3813a16ba4c5-console-oauth-config\") pod \"console-59f8c89778-p7jds\" (UID: \"8a7dcdc5-5d0c-4188-a64c-3813a16ba4c5\") " pod="openshift-console/console-59f8c89778-p7jds" Jul 30 00:20:58.948964 ip-10-0-142-6 kubenswrapper[2564]: I0730 00:20:58.948848 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/8a7dcdc5-5d0c-4188-a64c-3813a16ba4c5-console-serving-cert\") pod \"console-59f8c89778-p7jds\" (UID: \"8a7dcdc5-5d0c-4188-a64c-3813a16ba4c5\") " pod="openshift-console/console-59f8c89778-p7jds" Jul 30 00:20:58.948964 ip-10-0-142-6 kubenswrapper[2564]: I0730 00:20:58.948873 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/8a7dcdc5-5d0c-4188-a64c-3813a16ba4c5-console-config\") pod \"console-59f8c89778-p7jds\" (UID: \"8a7dcdc5-5d0c-4188-a64c-3813a16ba4c5\") " pod="openshift-console/console-59f8c89778-p7jds" Jul 30 00:20:58.948964 ip-10-0-142-6 kubenswrapper[2564]: I0730 00:20:58.948891 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/8a7dcdc5-5d0c-4188-a64c-3813a16ba4c5-trusted-ca-bundle\") pod \"console-59f8c89778-p7jds\" (UID: \"8a7dcdc5-5d0c-4188-a64c-3813a16ba4c5\") " pod="openshift-console/console-59f8c89778-p7jds" Jul 30 00:20:58.948964 ip-10-0-142-6 kubenswrapper[2564]: I0730 00:20:58.948940 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/8a7dcdc5-5d0c-4188-a64c-3813a16ba4c5-oauth-serving-cert\") pod \"console-59f8c89778-p7jds\" (UID: \"8a7dcdc5-5d0c-4188-a64c-3813a16ba4c5\") " pod="openshift-console/console-59f8c89778-p7jds" Jul 30 00:20:58.949182 ip-10-0-142-6 kubenswrapper[2564]: I0730 00:20:58.948972 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6qvs7\" (UniqueName: \"kubernetes.io/projected/8a7dcdc5-5d0c-4188-a64c-3813a16ba4c5-kube-api-access-6qvs7\") pod \"console-59f8c89778-p7jds\" (UID: \"8a7dcdc5-5d0c-4188-a64c-3813a16ba4c5\") " pod="openshift-console/console-59f8c89778-p7jds" Jul 30 00:20:59.049397 ip-10-0-142-6 kubenswrapper[2564]: I0730 00:20:59.049359 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/8a7dcdc5-5d0c-4188-a64c-3813a16ba4c5-console-oauth-config\") pod \"console-59f8c89778-p7jds\" (UID: \"8a7dcdc5-5d0c-4188-a64c-3813a16ba4c5\") " pod="openshift-console/console-59f8c89778-p7jds" Jul 30 00:20:59.049591 ip-10-0-142-6 kubenswrapper[2564]: I0730 00:20:59.049419 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/8a7dcdc5-5d0c-4188-a64c-3813a16ba4c5-console-serving-cert\") pod \"console-59f8c89778-p7jds\" (UID: \"8a7dcdc5-5d0c-4188-a64c-3813a16ba4c5\") " pod="openshift-console/console-59f8c89778-p7jds" Jul 30 00:20:59.049591 ip-10-0-142-6 kubenswrapper[2564]: I0730 00:20:59.049461 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/8a7dcdc5-5d0c-4188-a64c-3813a16ba4c5-console-config\") pod \"console-59f8c89778-p7jds\" (UID: \"8a7dcdc5-5d0c-4188-a64c-3813a16ba4c5\") " pod="openshift-console/console-59f8c89778-p7jds" Jul 30 00:20:59.049591 ip-10-0-142-6 kubenswrapper[2564]: I0730 00:20:59.049486 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/8a7dcdc5-5d0c-4188-a64c-3813a16ba4c5-trusted-ca-bundle\") pod \"console-59f8c89778-p7jds\" (UID: \"8a7dcdc5-5d0c-4188-a64c-3813a16ba4c5\") " pod="openshift-console/console-59f8c89778-p7jds" Jul 30 00:20:59.049591 ip-10-0-142-6 kubenswrapper[2564]: I0730 00:20:59.049505 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/8a7dcdc5-5d0c-4188-a64c-3813a16ba4c5-oauth-serving-cert\") pod \"console-59f8c89778-p7jds\" (UID: \"8a7dcdc5-5d0c-4188-a64c-3813a16ba4c5\") " pod="openshift-console/console-59f8c89778-p7jds" Jul 30 00:20:59.049591 ip-10-0-142-6 kubenswrapper[2564]: I0730 00:20:59.049526 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6qvs7\" (UniqueName: \"kubernetes.io/projected/8a7dcdc5-5d0c-4188-a64c-3813a16ba4c5-kube-api-access-6qvs7\") pod \"console-59f8c89778-p7jds\" (UID: \"8a7dcdc5-5d0c-4188-a64c-3813a16ba4c5\") " pod="openshift-console/console-59f8c89778-p7jds" Jul 30 00:20:59.049591 ip-10-0-142-6 kubenswrapper[2564]: I0730 00:20:59.049556 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/8a7dcdc5-5d0c-4188-a64c-3813a16ba4c5-service-ca\") pod \"console-59f8c89778-p7jds\" (UID: \"8a7dcdc5-5d0c-4188-a64c-3813a16ba4c5\") " pod="openshift-console/console-59f8c89778-p7jds" Jul 30 00:20:59.050410 ip-10-0-142-6 kubenswrapper[2564]: I0730 00:20:59.050379 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/8a7dcdc5-5d0c-4188-a64c-3813a16ba4c5-service-ca\") pod \"console-59f8c89778-p7jds\" (UID: \"8a7dcdc5-5d0c-4188-a64c-3813a16ba4c5\") " pod="openshift-console/console-59f8c89778-p7jds" Jul 30 00:20:59.050534 ip-10-0-142-6 kubenswrapper[2564]: I0730 00:20:59.050423 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/8a7dcdc5-5d0c-4188-a64c-3813a16ba4c5-trusted-ca-bundle\") pod \"console-59f8c89778-p7jds\" (UID: \"8a7dcdc5-5d0c-4188-a64c-3813a16ba4c5\") " pod="openshift-console/console-59f8c89778-p7jds" Jul 30 00:20:59.050534 ip-10-0-142-6 kubenswrapper[2564]: I0730 00:20:59.050459 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/8a7dcdc5-5d0c-4188-a64c-3813a16ba4c5-oauth-serving-cert\") pod \"console-59f8c89778-p7jds\" (UID: \"8a7dcdc5-5d0c-4188-a64c-3813a16ba4c5\") " pod="openshift-console/console-59f8c89778-p7jds" Jul 30 00:20:59.050632 ip-10-0-142-6 kubenswrapper[2564]: I0730 00:20:59.050540 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/8a7dcdc5-5d0c-4188-a64c-3813a16ba4c5-console-config\") pod \"console-59f8c89778-p7jds\" (UID: \"8a7dcdc5-5d0c-4188-a64c-3813a16ba4c5\") " pod="openshift-console/console-59f8c89778-p7jds" Jul 30 00:20:59.052276 ip-10-0-142-6 kubenswrapper[2564]: I0730 00:20:59.052252 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/8a7dcdc5-5d0c-4188-a64c-3813a16ba4c5-console-serving-cert\") pod \"console-59f8c89778-p7jds\" (UID: \"8a7dcdc5-5d0c-4188-a64c-3813a16ba4c5\") " pod="openshift-console/console-59f8c89778-p7jds" Jul 30 00:20:59.052276 ip-10-0-142-6 kubenswrapper[2564]: I0730 00:20:59.052273 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/8a7dcdc5-5d0c-4188-a64c-3813a16ba4c5-console-oauth-config\") pod \"console-59f8c89778-p7jds\" (UID: \"8a7dcdc5-5d0c-4188-a64c-3813a16ba4c5\") " pod="openshift-console/console-59f8c89778-p7jds" Jul 30 00:20:59.057342 ip-10-0-142-6 kubenswrapper[2564]: I0730 00:20:59.057322 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6qvs7\" (UniqueName: \"kubernetes.io/projected/8a7dcdc5-5d0c-4188-a64c-3813a16ba4c5-kube-api-access-6qvs7\") pod \"console-59f8c89778-p7jds\" (UID: \"8a7dcdc5-5d0c-4188-a64c-3813a16ba4c5\") " pod="openshift-console/console-59f8c89778-p7jds" Jul 30 00:20:59.174372 ip-10-0-142-6 kubenswrapper[2564]: I0730 00:20:59.174337 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-59f8c89778-p7jds" Jul 30 00:20:59.303993 ip-10-0-142-6 kubenswrapper[2564]: I0730 00:20:59.303962 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-59f8c89778-p7jds"] Jul 30 00:20:59.307170 ip-10-0-142-6 kubenswrapper[2564]: W0730 00:20:59.307143 2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8a7dcdc5_5d0c_4188_a64c_3813a16ba4c5.slice/crio-2729715a01565cad922d986309a4cff4553599f0afefe131f2d6ee0d2f8a041f WatchSource:0}: Error finding container 2729715a01565cad922d986309a4cff4553599f0afefe131f2d6ee0d2f8a041f: Status 404 returned error can't find the container with id 2729715a01565cad922d986309a4cff4553599f0afefe131f2d6ee0d2f8a041f Jul 30 00:20:59.962609 ip-10-0-142-6 kubenswrapper[2564]: I0730 00:20:59.962570 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-59f8c89778-p7jds" event={"ID":"8a7dcdc5-5d0c-4188-a64c-3813a16ba4c5","Type":"ContainerStarted","Data":"90cf487b20b88e3c992f464610b963c4d10106dcdbfe73ad069c88d23d396d60"} Jul 30 00:20:59.963092 ip-10-0-142-6 kubenswrapper[2564]: I0730 00:20:59.962620 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-59f8c89778-p7jds" event={"ID":"8a7dcdc5-5d0c-4188-a64c-3813a16ba4c5","Type":"ContainerStarted","Data":"2729715a01565cad922d986309a4cff4553599f0afefe131f2d6ee0d2f8a041f"} Jul 30 00:20:59.979714 ip-10-0-142-6 kubenswrapper[2564]: I0730 00:20:59.979634 2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-59f8c89778-p7jds" podStartSLOduration=1.9796173609999999 podStartE2EDuration="1.979617361s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-30 00:20:58 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-30 00:20:59.977976825 +0000 UTC m=+191.392716759" watchObservedRunningTime="2026-07-30 00:20:59.979617361 +0000 UTC m=+191.394357294" Jul 30 00:21:09.174719 ip-10-0-142-6 kubenswrapper[2564]: I0730 00:21:09.174679 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-59f8c89778-p7jds" Jul 30 00:21:09.175137 ip-10-0-142-6 kubenswrapper[2564]: I0730 00:21:09.174743 2564 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-59f8c89778-p7jds" Jul 30 00:21:09.179617 ip-10-0-142-6 kubenswrapper[2564]: I0730 00:21:09.179597 2564 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-59f8c89778-p7jds" Jul 30 00:21:10.001452 ip-10-0-142-6 kubenswrapper[2564]: I0730 00:21:10.001426 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-59f8c89778-p7jds" Jul 30 00:21:10.046798 ip-10-0-142-6 kubenswrapper[2564]: I0730 00:21:10.046764 2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-66bc4cd48c-57dmw"] Jul 30 00:21:35.065646 ip-10-0-142-6 kubenswrapper[2564]: I0730 00:21:35.065583 2564 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-console/console-66bc4cd48c-57dmw" podUID="79e0b7f8-d0ec-47d6-85fb-d79aeebcfc17" containerName="console" containerID="cri-o://2d7c8fc950da762585bca3881a3aa2d4c92d8e58f9f956e9d27c7601e5d726e1" gracePeriod=15 Jul 30 00:21:35.311778 ip-10-0-142-6 kubenswrapper[2564]: I0730 00:21:35.311754 2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-66bc4cd48c-57dmw" Jul 30 00:21:35.432941 ip-10-0-142-6 kubenswrapper[2564]: I0730 00:21:35.432914 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/79e0b7f8-d0ec-47d6-85fb-d79aeebcfc17-console-oauth-config\") pod \"79e0b7f8-d0ec-47d6-85fb-d79aeebcfc17\" (UID: \"79e0b7f8-d0ec-47d6-85fb-d79aeebcfc17\") " Jul 30 00:21:35.432941 ip-10-0-142-6 kubenswrapper[2564]: I0730 00:21:35.432950 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/79e0b7f8-d0ec-47d6-85fb-d79aeebcfc17-console-serving-cert\") pod \"79e0b7f8-d0ec-47d6-85fb-d79aeebcfc17\" (UID: \"79e0b7f8-d0ec-47d6-85fb-d79aeebcfc17\") " Jul 30 00:21:35.433205 ip-10-0-142-6 kubenswrapper[2564]: I0730 00:21:35.432997 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/79e0b7f8-d0ec-47d6-85fb-d79aeebcfc17-oauth-serving-cert\") pod \"79e0b7f8-d0ec-47d6-85fb-d79aeebcfc17\" (UID: \"79e0b7f8-d0ec-47d6-85fb-d79aeebcfc17\") " Jul 30 00:21:35.433205 ip-10-0-142-6 kubenswrapper[2564]: I0730 00:21:35.433038 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/79e0b7f8-d0ec-47d6-85fb-d79aeebcfc17-service-ca\") pod \"79e0b7f8-d0ec-47d6-85fb-d79aeebcfc17\" (UID: \"79e0b7f8-d0ec-47d6-85fb-d79aeebcfc17\") " Jul 30 00:21:35.433205 ip-10-0-142-6 kubenswrapper[2564]: I0730 00:21:35.433068 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/79e0b7f8-d0ec-47d6-85fb-d79aeebcfc17-console-config\") pod \"79e0b7f8-d0ec-47d6-85fb-d79aeebcfc17\" (UID: \"79e0b7f8-d0ec-47d6-85fb-d79aeebcfc17\") " Jul 30 00:21:35.433205 ip-10-0-142-6 kubenswrapper[2564]: I0730 00:21:35.433118 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-csrw2\" (UniqueName: \"kubernetes.io/projected/79e0b7f8-d0ec-47d6-85fb-d79aeebcfc17-kube-api-access-csrw2\") pod \"79e0b7f8-d0ec-47d6-85fb-d79aeebcfc17\" (UID: \"79e0b7f8-d0ec-47d6-85fb-d79aeebcfc17\") " Jul 30 00:21:35.433205 ip-10-0-142-6 kubenswrapper[2564]: I0730 00:21:35.433148 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/79e0b7f8-d0ec-47d6-85fb-d79aeebcfc17-trusted-ca-bundle\") pod \"79e0b7f8-d0ec-47d6-85fb-d79aeebcfc17\" (UID: \"79e0b7f8-d0ec-47d6-85fb-d79aeebcfc17\") " Jul 30 00:21:35.433485 ip-10-0-142-6 kubenswrapper[2564]: I0730 00:21:35.433460 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/79e0b7f8-d0ec-47d6-85fb-d79aeebcfc17-oauth-serving-cert" (OuterVolumeSpecName: "oauth-serving-cert") pod "79e0b7f8-d0ec-47d6-85fb-d79aeebcfc17" (UID: "79e0b7f8-d0ec-47d6-85fb-d79aeebcfc17"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 30 00:21:35.433544 ip-10-0-142-6 kubenswrapper[2564]: I0730 00:21:35.433520 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/79e0b7f8-d0ec-47d6-85fb-d79aeebcfc17-console-config" (OuterVolumeSpecName: "console-config") pod "79e0b7f8-d0ec-47d6-85fb-d79aeebcfc17" (UID: "79e0b7f8-d0ec-47d6-85fb-d79aeebcfc17"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 30 00:21:35.433600 ip-10-0-142-6 kubenswrapper[2564]: I0730 00:21:35.433561 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/79e0b7f8-d0ec-47d6-85fb-d79aeebcfc17-service-ca" (OuterVolumeSpecName: "service-ca") pod "79e0b7f8-d0ec-47d6-85fb-d79aeebcfc17" (UID: "79e0b7f8-d0ec-47d6-85fb-d79aeebcfc17"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 30 00:21:35.433681 ip-10-0-142-6 kubenswrapper[2564]: I0730 00:21:35.433657 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/79e0b7f8-d0ec-47d6-85fb-d79aeebcfc17-trusted-ca-bundle" (OuterVolumeSpecName: "trusted-ca-bundle") pod "79e0b7f8-d0ec-47d6-85fb-d79aeebcfc17" (UID: "79e0b7f8-d0ec-47d6-85fb-d79aeebcfc17"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 30 00:21:35.435271 ip-10-0-142-6 kubenswrapper[2564]: I0730 00:21:35.435246 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/79e0b7f8-d0ec-47d6-85fb-d79aeebcfc17-kube-api-access-csrw2" (OuterVolumeSpecName: "kube-api-access-csrw2") pod "79e0b7f8-d0ec-47d6-85fb-d79aeebcfc17" (UID: "79e0b7f8-d0ec-47d6-85fb-d79aeebcfc17"). InnerVolumeSpecName "kube-api-access-csrw2". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 30 00:21:35.435271 ip-10-0-142-6 kubenswrapper[2564]: I0730 00:21:35.435262 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/79e0b7f8-d0ec-47d6-85fb-d79aeebcfc17-console-serving-cert" (OuterVolumeSpecName: "console-serving-cert") pod "79e0b7f8-d0ec-47d6-85fb-d79aeebcfc17" (UID: "79e0b7f8-d0ec-47d6-85fb-d79aeebcfc17"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 30 00:21:35.435380 ip-10-0-142-6 kubenswrapper[2564]: I0730 00:21:35.435249 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/79e0b7f8-d0ec-47d6-85fb-d79aeebcfc17-console-oauth-config" (OuterVolumeSpecName: "console-oauth-config") pod "79e0b7f8-d0ec-47d6-85fb-d79aeebcfc17" (UID: "79e0b7f8-d0ec-47d6-85fb-d79aeebcfc17"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 30 00:21:35.533797 ip-10-0-142-6 kubenswrapper[2564]: I0730 00:21:35.533768 2564 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/79e0b7f8-d0ec-47d6-85fb-d79aeebcfc17-console-oauth-config\") on node \"ip-10-0-142-6.ec2.internal\" DevicePath \"\"" Jul 30 00:21:35.533797 ip-10-0-142-6 kubenswrapper[2564]: I0730 00:21:35.533795 2564 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/79e0b7f8-d0ec-47d6-85fb-d79aeebcfc17-console-serving-cert\") on node \"ip-10-0-142-6.ec2.internal\" DevicePath \"\"" Jul 30 00:21:35.533933 ip-10-0-142-6 kubenswrapper[2564]: I0730 00:21:35.533805 2564 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/79e0b7f8-d0ec-47d6-85fb-d79aeebcfc17-oauth-serving-cert\") on node \"ip-10-0-142-6.ec2.internal\" DevicePath \"\"" Jul 30 00:21:35.533933 ip-10-0-142-6 kubenswrapper[2564]: I0730 00:21:35.533814 2564 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/79e0b7f8-d0ec-47d6-85fb-d79aeebcfc17-service-ca\") on node \"ip-10-0-142-6.ec2.internal\" DevicePath \"\"" Jul 30 00:21:35.533933 ip-10-0-142-6 kubenswrapper[2564]: I0730 00:21:35.533822 2564 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/79e0b7f8-d0ec-47d6-85fb-d79aeebcfc17-console-config\") on node \"ip-10-0-142-6.ec2.internal\" DevicePath \"\"" Jul 30 00:21:35.533933 ip-10-0-142-6 kubenswrapper[2564]: I0730 00:21:35.533830 2564 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-csrw2\" (UniqueName: \"kubernetes.io/projected/79e0b7f8-d0ec-47d6-85fb-d79aeebcfc17-kube-api-access-csrw2\") on node \"ip-10-0-142-6.ec2.internal\" DevicePath \"\"" Jul 30 00:21:35.533933 ip-10-0-142-6 kubenswrapper[2564]: I0730 00:21:35.533838 2564 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/79e0b7f8-d0ec-47d6-85fb-d79aeebcfc17-trusted-ca-bundle\") on node \"ip-10-0-142-6.ec2.internal\" DevicePath \"\"" Jul 30 00:21:36.087153 ip-10-0-142-6 kubenswrapper[2564]: I0730 00:21:36.087115 2564 generic.go:356] "Generic (PLEG): container finished" podID="79e0b7f8-d0ec-47d6-85fb-d79aeebcfc17" containerID="2d7c8fc950da762585bca3881a3aa2d4c92d8e58f9f956e9d27c7601e5d726e1" exitCode=0 Jul 30 00:21:36.087537 ip-10-0-142-6 kubenswrapper[2564]: I0730 00:21:36.087163 2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-66bc4cd48c-57dmw" Jul 30 00:21:36.087537 ip-10-0-142-6 kubenswrapper[2564]: I0730 00:21:36.087200 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-66bc4cd48c-57dmw" event={"ID":"79e0b7f8-d0ec-47d6-85fb-d79aeebcfc17","Type":"ContainerDied","Data":"2d7c8fc950da762585bca3881a3aa2d4c92d8e58f9f956e9d27c7601e5d726e1"} Jul 30 00:21:36.087537 ip-10-0-142-6 kubenswrapper[2564]: I0730 00:21:36.087252 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-66bc4cd48c-57dmw" event={"ID":"79e0b7f8-d0ec-47d6-85fb-d79aeebcfc17","Type":"ContainerDied","Data":"648ee5696a029294789f3d1a1d26f396247264fe5a9c35455900bd1084af5d4d"} Jul 30 00:21:36.087537 ip-10-0-142-6 kubenswrapper[2564]: I0730 00:21:36.087272 2564 scope.go:117] "RemoveContainer" containerID="2d7c8fc950da762585bca3881a3aa2d4c92d8e58f9f956e9d27c7601e5d726e1" Jul 30 00:21:36.095535 ip-10-0-142-6 kubenswrapper[2564]: I0730 00:21:36.095520 2564 scope.go:117] "RemoveContainer" containerID="2d7c8fc950da762585bca3881a3aa2d4c92d8e58f9f956e9d27c7601e5d726e1" Jul 30 00:21:36.095779 ip-10-0-142-6 kubenswrapper[2564]: E0730 00:21:36.095763 2564 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2d7c8fc950da762585bca3881a3aa2d4c92d8e58f9f956e9d27c7601e5d726e1\": container with ID starting with 2d7c8fc950da762585bca3881a3aa2d4c92d8e58f9f956e9d27c7601e5d726e1 not found: ID does not exist" containerID="2d7c8fc950da762585bca3881a3aa2d4c92d8e58f9f956e9d27c7601e5d726e1" Jul 30 00:21:36.095822 ip-10-0-142-6 kubenswrapper[2564]: I0730 00:21:36.095787 2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2d7c8fc950da762585bca3881a3aa2d4c92d8e58f9f956e9d27c7601e5d726e1"} err="failed to get container status \"2d7c8fc950da762585bca3881a3aa2d4c92d8e58f9f956e9d27c7601e5d726e1\": rpc error: code = NotFound desc = could not find container \"2d7c8fc950da762585bca3881a3aa2d4c92d8e58f9f956e9d27c7601e5d726e1\": container with ID starting with 2d7c8fc950da762585bca3881a3aa2d4c92d8e58f9f956e9d27c7601e5d726e1 not found: ID does not exist" Jul 30 00:21:36.107351 ip-10-0-142-6 kubenswrapper[2564]: I0730 00:21:36.107320 2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-66bc4cd48c-57dmw"] Jul 30 00:21:36.111032 ip-10-0-142-6 kubenswrapper[2564]: I0730 00:21:36.111012 2564 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-66bc4cd48c-57dmw"] Jul 30 00:21:37.137821 ip-10-0-142-6 kubenswrapper[2564]: I0730 00:21:37.137791 2564 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="79e0b7f8-d0ec-47d6-85fb-d79aeebcfc17" path="/var/lib/kubelet/pods/79e0b7f8-d0ec-47d6-85fb-d79aeebcfc17/volumes" Jul 30 00:22:23.689164 ip-10-0-142-6 kubenswrapper[2564]: I0730 00:22:23.689088 2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540nblc9"] Jul 30 00:22:23.689649 ip-10-0-142-6 kubenswrapper[2564]: I0730 00:22:23.689383 2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="79e0b7f8-d0ec-47d6-85fb-d79aeebcfc17" containerName="console" Jul 30 00:22:23.689649 ip-10-0-142-6 kubenswrapper[2564]: I0730 00:22:23.689395 2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="79e0b7f8-d0ec-47d6-85fb-d79aeebcfc17" containerName="console" Jul 30 00:22:23.689649 ip-10-0-142-6 kubenswrapper[2564]: I0730 00:22:23.689434 2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="79e0b7f8-d0ec-47d6-85fb-d79aeebcfc17" containerName="console" Jul 30 00:22:23.692479 ip-10-0-142-6 kubenswrapper[2564]: I0730 00:22:23.692462 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540nblc9" Jul 30 00:22:23.694805 ip-10-0-142-6 kubenswrapper[2564]: I0730 00:22:23.694788 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"kube-root-ca.crt\"" Jul 30 00:22:23.695762 ip-10-0-142-6 kubenswrapper[2564]: I0730 00:22:23.695740 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"openshift-service-ca.crt\"" Jul 30 00:22:23.695857 ip-10-0-142-6 kubenswrapper[2564]: I0730 00:22:23.695749 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"default-dockercfg-wdqwh\"" Jul 30 00:22:23.700888 ip-10-0-142-6 kubenswrapper[2564]: I0730 00:22:23.700863 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540nblc9"] Jul 30 00:22:23.793195 ip-10-0-142-6 kubenswrapper[2564]: I0730 00:22:23.793161 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-w4dhm\" (UniqueName: \"kubernetes.io/projected/af12f3d1-fe25-4876-9080-4ee92d46073d-kube-api-access-w4dhm\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540nblc9\" (UID: \"af12f3d1-fe25-4876-9080-4ee92d46073d\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540nblc9" Jul 30 00:22:23.793367 ip-10-0-142-6 kubenswrapper[2564]: I0730 00:22:23.793219 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/af12f3d1-fe25-4876-9080-4ee92d46073d-util\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540nblc9\" (UID: \"af12f3d1-fe25-4876-9080-4ee92d46073d\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540nblc9" Jul 30 00:22:23.793367 ip-10-0-142-6 kubenswrapper[2564]: I0730 00:22:23.793268 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/af12f3d1-fe25-4876-9080-4ee92d46073d-bundle\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540nblc9\" (UID: \"af12f3d1-fe25-4876-9080-4ee92d46073d\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540nblc9" Jul 30 00:22:23.893794 ip-10-0-142-6 kubenswrapper[2564]: I0730 00:22:23.893741 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/af12f3d1-fe25-4876-9080-4ee92d46073d-util\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540nblc9\" (UID: \"af12f3d1-fe25-4876-9080-4ee92d46073d\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540nblc9" Jul 30 00:22:23.893794 ip-10-0-142-6 kubenswrapper[2564]: I0730 00:22:23.893803 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/af12f3d1-fe25-4876-9080-4ee92d46073d-bundle\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540nblc9\" (UID: \"af12f3d1-fe25-4876-9080-4ee92d46073d\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540nblc9" Jul 30 00:22:23.894012 ip-10-0-142-6 kubenswrapper[2564]: I0730 00:22:23.893857 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w4dhm\" (UniqueName: \"kubernetes.io/projected/af12f3d1-fe25-4876-9080-4ee92d46073d-kube-api-access-w4dhm\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540nblc9\" (UID: \"af12f3d1-fe25-4876-9080-4ee92d46073d\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540nblc9" Jul 30 00:22:23.894242 ip-10-0-142-6 kubenswrapper[2564]: I0730 00:22:23.894219 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/af12f3d1-fe25-4876-9080-4ee92d46073d-bundle\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540nblc9\" (UID: \"af12f3d1-fe25-4876-9080-4ee92d46073d\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540nblc9" Jul 30 00:22:23.894313 ip-10-0-142-6 kubenswrapper[2564]: I0730 00:22:23.894231 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/af12f3d1-fe25-4876-9080-4ee92d46073d-util\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540nblc9\" (UID: \"af12f3d1-fe25-4876-9080-4ee92d46073d\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540nblc9" Jul 30 00:22:23.901632 ip-10-0-142-6 kubenswrapper[2564]: I0730 00:22:23.901609 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-w4dhm\" (UniqueName: \"kubernetes.io/projected/af12f3d1-fe25-4876-9080-4ee92d46073d-kube-api-access-w4dhm\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540nblc9\" (UID: \"af12f3d1-fe25-4876-9080-4ee92d46073d\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540nblc9" Jul 30 00:22:24.002236 ip-10-0-142-6 kubenswrapper[2564]: I0730 00:22:24.002131 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540nblc9" Jul 30 00:22:24.129350 ip-10-0-142-6 kubenswrapper[2564]: I0730 00:22:24.129318 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540nblc9"] Jul 30 00:22:24.133037 ip-10-0-142-6 kubenswrapper[2564]: W0730 00:22:24.133004 2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podaf12f3d1_fe25_4876_9080_4ee92d46073d.slice/crio-d38ca297e2a671ae931aecaddf8e69d0349e1a44aadd3876b6e788d40468ae0c WatchSource:0}: Error finding container d38ca297e2a671ae931aecaddf8e69d0349e1a44aadd3876b6e788d40468ae0c: Status 404 returned error can't find the container with id d38ca297e2a671ae931aecaddf8e69d0349e1a44aadd3876b6e788d40468ae0c Jul 30 00:22:24.246721 ip-10-0-142-6 kubenswrapper[2564]: I0730 00:22:24.246687 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540nblc9" event={"ID":"af12f3d1-fe25-4876-9080-4ee92d46073d","Type":"ContainerStarted","Data":"d38ca297e2a671ae931aecaddf8e69d0349e1a44aadd3876b6e788d40468ae0c"} Jul 30 00:22:33.280724 ip-10-0-142-6 kubenswrapper[2564]: I0730 00:22:33.280684 2564 generic.go:356] "Generic (PLEG): container finished" podID="af12f3d1-fe25-4876-9080-4ee92d46073d" containerID="44495a823848652fd1096ebddcfb30ac3b763882b2f677cfd2b016389b2c09cd" exitCode=0 Jul 30 00:22:33.281171 ip-10-0-142-6 kubenswrapper[2564]: I0730 00:22:33.280724 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540nblc9" event={"ID":"af12f3d1-fe25-4876-9080-4ee92d46073d","Type":"ContainerDied","Data":"44495a823848652fd1096ebddcfb30ac3b763882b2f677cfd2b016389b2c09cd"} Jul 30 00:22:38.300263 ip-10-0-142-6 kubenswrapper[2564]: I0730 00:22:38.300233 2564 generic.go:356] "Generic (PLEG): container finished" podID="af12f3d1-fe25-4876-9080-4ee92d46073d" containerID="e3e855c33b4ef9e87acbea72066dc4969aa779b9a32875872bdd0c9160e18753" exitCode=0 Jul 30 00:22:38.300613 ip-10-0-142-6 kubenswrapper[2564]: I0730 00:22:38.300324 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540nblc9" event={"ID":"af12f3d1-fe25-4876-9080-4ee92d46073d","Type":"ContainerDied","Data":"e3e855c33b4ef9e87acbea72066dc4969aa779b9a32875872bdd0c9160e18753"} Jul 30 00:22:46.338261 ip-10-0-142-6 kubenswrapper[2564]: I0730 00:22:46.338235 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540nblc9" event={"ID":"af12f3d1-fe25-4876-9080-4ee92d46073d","Type":"ContainerStarted","Data":"37281d8ee201e052dbdb1d27e5323ab3ae6e83838a1fa500cd77eed6188772bc"} Jul 30 00:22:47.343659 ip-10-0-142-6 kubenswrapper[2564]: I0730 00:22:47.343625 2564 generic.go:356] "Generic (PLEG): container finished" podID="af12f3d1-fe25-4876-9080-4ee92d46073d" containerID="37281d8ee201e052dbdb1d27e5323ab3ae6e83838a1fa500cd77eed6188772bc" exitCode=0 Jul 30 00:22:47.344091 ip-10-0-142-6 kubenswrapper[2564]: I0730 00:22:47.343713 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540nblc9" event={"ID":"af12f3d1-fe25-4876-9080-4ee92d46073d","Type":"ContainerDied","Data":"37281d8ee201e052dbdb1d27e5323ab3ae6e83838a1fa500cd77eed6188772bc"} Jul 30 00:22:48.468463 ip-10-0-142-6 kubenswrapper[2564]: I0730 00:22:48.468440 2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540nblc9" Jul 30 00:22:48.594275 ip-10-0-142-6 kubenswrapper[2564]: I0730 00:22:48.594234 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/af12f3d1-fe25-4876-9080-4ee92d46073d-util\") pod \"af12f3d1-fe25-4876-9080-4ee92d46073d\" (UID: \"af12f3d1-fe25-4876-9080-4ee92d46073d\") " Jul 30 00:22:48.594418 ip-10-0-142-6 kubenswrapper[2564]: I0730 00:22:48.594298 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-w4dhm\" (UniqueName: \"kubernetes.io/projected/af12f3d1-fe25-4876-9080-4ee92d46073d-kube-api-access-w4dhm\") pod \"af12f3d1-fe25-4876-9080-4ee92d46073d\" (UID: \"af12f3d1-fe25-4876-9080-4ee92d46073d\") " Jul 30 00:22:48.594418 ip-10-0-142-6 kubenswrapper[2564]: I0730 00:22:48.594338 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/af12f3d1-fe25-4876-9080-4ee92d46073d-bundle\") pod \"af12f3d1-fe25-4876-9080-4ee92d46073d\" (UID: \"af12f3d1-fe25-4876-9080-4ee92d46073d\") " Jul 30 00:22:48.595025 ip-10-0-142-6 kubenswrapper[2564]: I0730 00:22:48.595000 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/af12f3d1-fe25-4876-9080-4ee92d46073d-bundle" (OuterVolumeSpecName: "bundle") pod "af12f3d1-fe25-4876-9080-4ee92d46073d" (UID: "af12f3d1-fe25-4876-9080-4ee92d46073d"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 30 00:22:48.596498 ip-10-0-142-6 kubenswrapper[2564]: I0730 00:22:48.596466 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/af12f3d1-fe25-4876-9080-4ee92d46073d-kube-api-access-w4dhm" (OuterVolumeSpecName: "kube-api-access-w4dhm") pod "af12f3d1-fe25-4876-9080-4ee92d46073d" (UID: "af12f3d1-fe25-4876-9080-4ee92d46073d"). InnerVolumeSpecName "kube-api-access-w4dhm". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 30 00:22:48.599097 ip-10-0-142-6 kubenswrapper[2564]: I0730 00:22:48.599041 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/af12f3d1-fe25-4876-9080-4ee92d46073d-util" (OuterVolumeSpecName: "util") pod "af12f3d1-fe25-4876-9080-4ee92d46073d" (UID: "af12f3d1-fe25-4876-9080-4ee92d46073d"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 30 00:22:48.695110 ip-10-0-142-6 kubenswrapper[2564]: I0730 00:22:48.695063 2564 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-w4dhm\" (UniqueName: \"kubernetes.io/projected/af12f3d1-fe25-4876-9080-4ee92d46073d-kube-api-access-w4dhm\") on node \"ip-10-0-142-6.ec2.internal\" DevicePath \"\"" Jul 30 00:22:48.695110 ip-10-0-142-6 kubenswrapper[2564]: I0730 00:22:48.695105 2564 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/af12f3d1-fe25-4876-9080-4ee92d46073d-bundle\") on node \"ip-10-0-142-6.ec2.internal\" DevicePath \"\"" Jul 30 00:22:48.695110 ip-10-0-142-6 kubenswrapper[2564]: I0730 00:22:48.695115 2564 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/af12f3d1-fe25-4876-9080-4ee92d46073d-util\") on node \"ip-10-0-142-6.ec2.internal\" DevicePath \"\"" Jul 30 00:22:48.960735 ip-10-0-142-6 kubenswrapper[2564]: I0730 00:22:48.960711 2564 kubelet.go:1648] "Image garbage collection succeeded" Jul 30 00:22:49.352345 ip-10-0-142-6 kubenswrapper[2564]: I0730 00:22:49.352265 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540nblc9" event={"ID":"af12f3d1-fe25-4876-9080-4ee92d46073d","Type":"ContainerDied","Data":"d38ca297e2a671ae931aecaddf8e69d0349e1a44aadd3876b6e788d40468ae0c"} Jul 30 00:22:49.352345 ip-10-0-142-6 kubenswrapper[2564]: I0730 00:22:49.352313 2564 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="d38ca297e2a671ae931aecaddf8e69d0349e1a44aadd3876b6e788d40468ae0c" Jul 30 00:22:49.352345 ip-10-0-142-6 kubenswrapper[2564]: I0730 00:22:49.352271 2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540nblc9" Jul 30 00:22:56.888500 ip-10-0-142-6 kubenswrapper[2564]: I0730 00:22:56.888466 2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-v44q5"] Jul 30 00:22:56.888892 ip-10-0-142-6 kubenswrapper[2564]: I0730 00:22:56.888712 2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="af12f3d1-fe25-4876-9080-4ee92d46073d" containerName="util" Jul 30 00:22:56.888892 ip-10-0-142-6 kubenswrapper[2564]: I0730 00:22:56.888722 2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="af12f3d1-fe25-4876-9080-4ee92d46073d" containerName="util" Jul 30 00:22:56.888892 ip-10-0-142-6 kubenswrapper[2564]: I0730 00:22:56.888730 2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="af12f3d1-fe25-4876-9080-4ee92d46073d" containerName="extract" Jul 30 00:22:56.888892 ip-10-0-142-6 kubenswrapper[2564]: I0730 00:22:56.888735 2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="af12f3d1-fe25-4876-9080-4ee92d46073d" containerName="extract" Jul 30 00:22:56.888892 ip-10-0-142-6 kubenswrapper[2564]: I0730 00:22:56.888749 2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="af12f3d1-fe25-4876-9080-4ee92d46073d" containerName="pull" Jul 30 00:22:56.888892 ip-10-0-142-6 kubenswrapper[2564]: I0730 00:22:56.888754 2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="af12f3d1-fe25-4876-9080-4ee92d46073d" containerName="pull" Jul 30 00:22:56.888892 ip-10-0-142-6 kubenswrapper[2564]: I0730 00:22:56.888791 2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="af12f3d1-fe25-4876-9080-4ee92d46073d" containerName="extract" Jul 30 00:22:56.890469 ip-10-0-142-6 kubenswrapper[2564]: I0730 00:22:56.890453 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-v44q5" Jul 30 00:22:56.893048 ip-10-0-142-6 kubenswrapper[2564]: I0730 00:22:56.893023 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager-operator\"/\"kube-root-ca.crt\"" Jul 30 00:22:56.893153 ip-10-0-142-6 kubenswrapper[2564]: I0730 00:22:56.893095 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager-operator\"/\"openshift-service-ca.crt\"" Jul 30 00:22:56.893153 ip-10-0-142-6 kubenswrapper[2564]: I0730 00:22:56.893115 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"cert-manager-operator\"/\"cert-manager-operator-controller-manager-dockercfg-kmpxl\"" Jul 30 00:22:56.904845 ip-10-0-142-6 kubenswrapper[2564]: I0730 00:22:56.904808 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-v44q5"] Jul 30 00:22:57.061445 ip-10-0-142-6 kubenswrapper[2564]: I0730 00:22:57.061405 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dtdgk\" (UniqueName: \"kubernetes.io/projected/69c9b100-95c4-47af-b2a8-c1845480d589-kube-api-access-dtdgk\") pod \"cert-manager-operator-controller-manager-55c444bf86-v44q5\" (UID: \"69c9b100-95c4-47af-b2a8-c1845480d589\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-v44q5" Jul 30 00:22:57.061613 ip-10-0-142-6 kubenswrapper[2564]: I0730 00:22:57.061486 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/69c9b100-95c4-47af-b2a8-c1845480d589-tmp\") pod \"cert-manager-operator-controller-manager-55c444bf86-v44q5\" (UID: \"69c9b100-95c4-47af-b2a8-c1845480d589\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-v44q5" Jul 30 00:22:57.162787 ip-10-0-142-6 kubenswrapper[2564]: I0730 00:22:57.162700 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/69c9b100-95c4-47af-b2a8-c1845480d589-tmp\") pod \"cert-manager-operator-controller-manager-55c444bf86-v44q5\" (UID: \"69c9b100-95c4-47af-b2a8-c1845480d589\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-v44q5" Jul 30 00:22:57.162787 ip-10-0-142-6 kubenswrapper[2564]: I0730 00:22:57.162752 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dtdgk\" (UniqueName: \"kubernetes.io/projected/69c9b100-95c4-47af-b2a8-c1845480d589-kube-api-access-dtdgk\") pod \"cert-manager-operator-controller-manager-55c444bf86-v44q5\" (UID: \"69c9b100-95c4-47af-b2a8-c1845480d589\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-v44q5" Jul 30 00:22:57.163117 ip-10-0-142-6 kubenswrapper[2564]: I0730 00:22:57.163031 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/69c9b100-95c4-47af-b2a8-c1845480d589-tmp\") pod \"cert-manager-operator-controller-manager-55c444bf86-v44q5\" (UID: \"69c9b100-95c4-47af-b2a8-c1845480d589\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-v44q5" Jul 30 00:22:57.171147 ip-10-0-142-6 kubenswrapper[2564]: I0730 00:22:57.171122 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-dtdgk\" (UniqueName: \"kubernetes.io/projected/69c9b100-95c4-47af-b2a8-c1845480d589-kube-api-access-dtdgk\") pod \"cert-manager-operator-controller-manager-55c444bf86-v44q5\" (UID: \"69c9b100-95c4-47af-b2a8-c1845480d589\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-v44q5" Jul 30 00:22:57.199245 ip-10-0-142-6 kubenswrapper[2564]: I0730 00:22:57.199218 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-v44q5" Jul 30 00:22:57.328049 ip-10-0-142-6 kubenswrapper[2564]: I0730 00:22:57.328019 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-v44q5"] Jul 30 00:22:57.331023 ip-10-0-142-6 kubenswrapper[2564]: W0730 00:22:57.330997 2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod69c9b100_95c4_47af_b2a8_c1845480d589.slice/crio-227ec71ea31fae3ea4c0c7ed3aa44d3419d7a238e26644e97eefd63038516f7c WatchSource:0}: Error finding container 227ec71ea31fae3ea4c0c7ed3aa44d3419d7a238e26644e97eefd63038516f7c: Status 404 returned error can't find the container with id 227ec71ea31fae3ea4c0c7ed3aa44d3419d7a238e26644e97eefd63038516f7c Jul 30 00:22:57.333631 ip-10-0-142-6 kubenswrapper[2564]: I0730 00:22:57.333617 2564 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Jul 30 00:22:57.382006 ip-10-0-142-6 kubenswrapper[2564]: I0730 00:22:57.381975 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-v44q5" event={"ID":"69c9b100-95c4-47af-b2a8-c1845480d589","Type":"ContainerStarted","Data":"227ec71ea31fae3ea4c0c7ed3aa44d3419d7a238e26644e97eefd63038516f7c"} Jul 30 00:22:59.391038 ip-10-0-142-6 kubenswrapper[2564]: I0730 00:22:59.391005 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-v44q5" event={"ID":"69c9b100-95c4-47af-b2a8-c1845480d589","Type":"ContainerStarted","Data":"54945089bec4bf4f86077c2f1281194cae2ba7f3a949a6947d2b66db1109f54a"} Jul 30 00:22:59.413595 ip-10-0-142-6 kubenswrapper[2564]: I0730 00:22:59.413506 2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-v44q5" podStartSLOduration=1.602658333 podStartE2EDuration="3.413489266s" totalImagesPullingTime="1.810830933s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-30 00:22:56 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-30 00:22:59.411576175 +0000 UTC m=+310.826316120" watchObservedRunningTime="2026-07-30 00:22:59.413489266 +0000 UTC m=+310.828229199" Jul 30 00:23:00.376771 ip-10-0-142-6 kubenswrapper[2564]: I0730 00:23:00.376739 2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fk757h"] Jul 30 00:23:00.378806 ip-10-0-142-6 kubenswrapper[2564]: I0730 00:23:00.378790 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fk757h" Jul 30 00:23:00.381514 ip-10-0-142-6 kubenswrapper[2564]: I0730 00:23:00.381490 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"openshift-service-ca.crt\"" Jul 30 00:23:00.382298 ip-10-0-142-6 kubenswrapper[2564]: I0730 00:23:00.382280 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"default-dockercfg-wdqwh\"" Jul 30 00:23:00.382403 ip-10-0-142-6 kubenswrapper[2564]: I0730 00:23:00.382283 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"kube-root-ca.crt\"" Jul 30 00:23:00.387660 ip-10-0-142-6 kubenswrapper[2564]: I0730 00:23:00.387632 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fk757h"] Jul 30 00:23:00.489325 ip-10-0-142-6 kubenswrapper[2564]: I0730 00:23:00.489294 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/cb25ceb5-a070-4f20-9486-c8aad93678a0-bundle\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fk757h\" (UID: \"cb25ceb5-a070-4f20-9486-c8aad93678a0\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fk757h" Jul 30 00:23:00.489325 ip-10-0-142-6 kubenswrapper[2564]: I0730 00:23:00.489328 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/cb25ceb5-a070-4f20-9486-c8aad93678a0-util\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fk757h\" (UID: \"cb25ceb5-a070-4f20-9486-c8aad93678a0\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fk757h" Jul 30 00:23:00.489834 ip-10-0-142-6 kubenswrapper[2564]: I0730 00:23:00.489417 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6d5jc\" (UniqueName: \"kubernetes.io/projected/cb25ceb5-a070-4f20-9486-c8aad93678a0-kube-api-access-6d5jc\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fk757h\" (UID: \"cb25ceb5-a070-4f20-9486-c8aad93678a0\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fk757h" Jul 30 00:23:00.590846 ip-10-0-142-6 kubenswrapper[2564]: I0730 00:23:00.590804 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/cb25ceb5-a070-4f20-9486-c8aad93678a0-bundle\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fk757h\" (UID: \"cb25ceb5-a070-4f20-9486-c8aad93678a0\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fk757h" Jul 30 00:23:00.590846 ip-10-0-142-6 kubenswrapper[2564]: I0730 00:23:00.590846 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/cb25ceb5-a070-4f20-9486-c8aad93678a0-util\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fk757h\" (UID: \"cb25ceb5-a070-4f20-9486-c8aad93678a0\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fk757h" Jul 30 00:23:00.591111 ip-10-0-142-6 kubenswrapper[2564]: I0730 00:23:00.590903 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6d5jc\" (UniqueName: \"kubernetes.io/projected/cb25ceb5-a070-4f20-9486-c8aad93678a0-kube-api-access-6d5jc\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fk757h\" (UID: \"cb25ceb5-a070-4f20-9486-c8aad93678a0\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fk757h" Jul 30 00:23:00.591230 ip-10-0-142-6 kubenswrapper[2564]: I0730 00:23:00.591208 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/cb25ceb5-a070-4f20-9486-c8aad93678a0-bundle\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fk757h\" (UID: \"cb25ceb5-a070-4f20-9486-c8aad93678a0\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fk757h" Jul 30 00:23:00.591292 ip-10-0-142-6 kubenswrapper[2564]: I0730 00:23:00.591233 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/cb25ceb5-a070-4f20-9486-c8aad93678a0-util\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fk757h\" (UID: \"cb25ceb5-a070-4f20-9486-c8aad93678a0\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fk757h" Jul 30 00:23:00.599246 ip-10-0-142-6 kubenswrapper[2564]: I0730 00:23:00.599226 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6d5jc\" (UniqueName: \"kubernetes.io/projected/cb25ceb5-a070-4f20-9486-c8aad93678a0-kube-api-access-6d5jc\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fk757h\" (UID: \"cb25ceb5-a070-4f20-9486-c8aad93678a0\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fk757h" Jul 30 00:23:00.688419 ip-10-0-142-6 kubenswrapper[2564]: I0730 00:23:00.688337 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fk757h" Jul 30 00:23:00.821843 ip-10-0-142-6 kubenswrapper[2564]: I0730 00:23:00.821803 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fk757h"] Jul 30 00:23:00.825036 ip-10-0-142-6 kubenswrapper[2564]: W0730 00:23:00.825010 2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcb25ceb5_a070_4f20_9486_c8aad93678a0.slice/crio-5fa3ef4abbd071b4721e4640d40d12056f9e272905c03431009cfa97d441e059 WatchSource:0}: Error finding container 5fa3ef4abbd071b4721e4640d40d12056f9e272905c03431009cfa97d441e059: Status 404 returned error can't find the container with id 5fa3ef4abbd071b4721e4640d40d12056f9e272905c03431009cfa97d441e059 Jul 30 00:23:00.991150 ip-10-0-142-6 kubenswrapper[2564]: I0730 00:23:00.991059 2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["cert-manager/cert-manager-webhook-6486ffd997-57l94"] Jul 30 00:23:00.993128 ip-10-0-142-6 kubenswrapper[2564]: I0730 00:23:00.993111 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-webhook-6486ffd997-57l94" Jul 30 00:23:00.995813 ip-10-0-142-6 kubenswrapper[2564]: I0730 00:23:00.995794 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"cert-manager\"/\"cert-manager-webhook-dockercfg-695f9\"" Jul 30 00:23:00.995904 ip-10-0-142-6 kubenswrapper[2564]: I0730 00:23:00.995817 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager\"/\"kube-root-ca.crt\"" Jul 30 00:23:00.995904 ip-10-0-142-6 kubenswrapper[2564]: I0730 00:23:00.995795 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager\"/\"openshift-service-ca.crt\"" Jul 30 00:23:01.003639 ip-10-0-142-6 kubenswrapper[2564]: I0730 00:23:01.003616 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-webhook-6486ffd997-57l94"] Jul 30 00:23:01.094180 ip-10-0-142-6 kubenswrapper[2564]: I0730 00:23:01.094145 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xw4pt\" (UniqueName: \"kubernetes.io/projected/77188928-1fb1-430a-9411-2d405e5026fb-kube-api-access-xw4pt\") pod \"cert-manager-webhook-6486ffd997-57l94\" (UID: \"77188928-1fb1-430a-9411-2d405e5026fb\") " pod="cert-manager/cert-manager-webhook-6486ffd997-57l94" Jul 30 00:23:01.094337 ip-10-0-142-6 kubenswrapper[2564]: I0730 00:23:01.094189 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/77188928-1fb1-430a-9411-2d405e5026fb-bound-sa-token\") pod \"cert-manager-webhook-6486ffd997-57l94\" (UID: \"77188928-1fb1-430a-9411-2d405e5026fb\") " pod="cert-manager/cert-manager-webhook-6486ffd997-57l94" Jul 30 00:23:01.194774 ip-10-0-142-6 kubenswrapper[2564]: I0730 00:23:01.194730 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xw4pt\" (UniqueName: \"kubernetes.io/projected/77188928-1fb1-430a-9411-2d405e5026fb-kube-api-access-xw4pt\") pod \"cert-manager-webhook-6486ffd997-57l94\" (UID: \"77188928-1fb1-430a-9411-2d405e5026fb\") " pod="cert-manager/cert-manager-webhook-6486ffd997-57l94" Jul 30 00:23:01.194969 ip-10-0-142-6 kubenswrapper[2564]: I0730 00:23:01.194791 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/77188928-1fb1-430a-9411-2d405e5026fb-bound-sa-token\") pod \"cert-manager-webhook-6486ffd997-57l94\" (UID: \"77188928-1fb1-430a-9411-2d405e5026fb\") " pod="cert-manager/cert-manager-webhook-6486ffd997-57l94" Jul 30 00:23:01.202937 ip-10-0-142-6 kubenswrapper[2564]: I0730 00:23:01.202902 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/77188928-1fb1-430a-9411-2d405e5026fb-bound-sa-token\") pod \"cert-manager-webhook-6486ffd997-57l94\" (UID: \"77188928-1fb1-430a-9411-2d405e5026fb\") " pod="cert-manager/cert-manager-webhook-6486ffd997-57l94" Jul 30 00:23:01.203033 ip-10-0-142-6 kubenswrapper[2564]: I0730 00:23:01.202995 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xw4pt\" (UniqueName: \"kubernetes.io/projected/77188928-1fb1-430a-9411-2d405e5026fb-kube-api-access-xw4pt\") pod \"cert-manager-webhook-6486ffd997-57l94\" (UID: \"77188928-1fb1-430a-9411-2d405e5026fb\") " pod="cert-manager/cert-manager-webhook-6486ffd997-57l94" Jul 30 00:23:01.317005 ip-10-0-142-6 kubenswrapper[2564]: I0730 00:23:01.316928 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-webhook-6486ffd997-57l94" Jul 30 00:23:01.401773 ip-10-0-142-6 kubenswrapper[2564]: I0730 00:23:01.401738 2564 generic.go:356] "Generic (PLEG): container finished" podID="cb25ceb5-a070-4f20-9486-c8aad93678a0" containerID="cbad67f30e0b55c71de8cdab7cdc1f7dc96606bb973d9b9a38cc7b9bce346222" exitCode=0 Jul 30 00:23:01.402018 ip-10-0-142-6 kubenswrapper[2564]: I0730 00:23:01.401779 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fk757h" event={"ID":"cb25ceb5-a070-4f20-9486-c8aad93678a0","Type":"ContainerDied","Data":"cbad67f30e0b55c71de8cdab7cdc1f7dc96606bb973d9b9a38cc7b9bce346222"} Jul 30 00:23:01.402170 ip-10-0-142-6 kubenswrapper[2564]: I0730 00:23:01.402050 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fk757h" event={"ID":"cb25ceb5-a070-4f20-9486-c8aad93678a0","Type":"ContainerStarted","Data":"5fa3ef4abbd071b4721e4640d40d12056f9e272905c03431009cfa97d441e059"} Jul 30 00:23:01.438126 ip-10-0-142-6 kubenswrapper[2564]: I0730 00:23:01.438101 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-webhook-6486ffd997-57l94"] Jul 30 00:23:01.440363 ip-10-0-142-6 kubenswrapper[2564]: W0730 00:23:01.440341 2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod77188928_1fb1_430a_9411_2d405e5026fb.slice/crio-02bec8352dc8dfab77a8b63fecd8320e77973a6015775589858d6e6f42f80def WatchSource:0}: Error finding container 02bec8352dc8dfab77a8b63fecd8320e77973a6015775589858d6e6f42f80def: Status 404 returned error can't find the container with id 02bec8352dc8dfab77a8b63fecd8320e77973a6015775589858d6e6f42f80def Jul 30 00:23:02.408187 ip-10-0-142-6 kubenswrapper[2564]: I0730 00:23:02.408153 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-webhook-6486ffd997-57l94" event={"ID":"77188928-1fb1-430a-9411-2d405e5026fb","Type":"ContainerStarted","Data":"02bec8352dc8dfab77a8b63fecd8320e77973a6015775589858d6e6f42f80def"} Jul 30 00:23:05.423811 ip-10-0-142-6 kubenswrapper[2564]: I0730 00:23:05.423778 2564 generic.go:356] "Generic (PLEG): container finished" podID="cb25ceb5-a070-4f20-9486-c8aad93678a0" containerID="26bef18830278e66ff4866977aba2b5d14f1e888f1801a08213abd24a41e96cf" exitCode=0 Jul 30 00:23:05.424264 ip-10-0-142-6 kubenswrapper[2564]: I0730 00:23:05.423849 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fk757h" event={"ID":"cb25ceb5-a070-4f20-9486-c8aad93678a0","Type":"ContainerDied","Data":"26bef18830278e66ff4866977aba2b5d14f1e888f1801a08213abd24a41e96cf"} Jul 30 00:23:05.425473 ip-10-0-142-6 kubenswrapper[2564]: I0730 00:23:05.425453 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-webhook-6486ffd997-57l94" event={"ID":"77188928-1fb1-430a-9411-2d405e5026fb","Type":"ContainerStarted","Data":"9e602e0cd04c37bf52a73f8dfebedde12fdb35c2cd1f782dfc98ac5c8a4e3c70"} Jul 30 00:23:05.425556 ip-10-0-142-6 kubenswrapper[2564]: I0730 00:23:05.425506 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="cert-manager/cert-manager-webhook-6486ffd997-57l94" Jul 30 00:23:05.455384 ip-10-0-142-6 kubenswrapper[2564]: I0730 00:23:05.455344 2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="cert-manager/cert-manager-webhook-6486ffd997-57l94" podStartSLOduration=1.812686931 podStartE2EDuration="5.455328854s" totalImagesPullingTime="3.642641923s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-30 00:23:00 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-30 00:23:05.454650695 +0000 UTC m=+316.869390640" watchObservedRunningTime="2026-07-30 00:23:05.455328854 +0000 UTC m=+316.870068786" Jul 30 00:23:06.431304 ip-10-0-142-6 kubenswrapper[2564]: I0730 00:23:06.431276 2564 generic.go:356] "Generic (PLEG): container finished" podID="cb25ceb5-a070-4f20-9486-c8aad93678a0" containerID="410cbebf571e93b88b118f0a6a7da8bd25cc1bddf945ce38f8c4ac318e282499" exitCode=0 Jul 30 00:23:06.431698 ip-10-0-142-6 kubenswrapper[2564]: I0730 00:23:06.431364 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fk757h" event={"ID":"cb25ceb5-a070-4f20-9486-c8aad93678a0","Type":"ContainerDied","Data":"410cbebf571e93b88b118f0a6a7da8bd25cc1bddf945ce38f8c4ac318e282499"} Jul 30 00:23:07.510068 ip-10-0-142-6 kubenswrapper[2564]: I0730 00:23:07.510036 2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["cert-manager/cert-manager-cainjector-559cdd8c87-z874n"] Jul 30 00:23:07.514400 ip-10-0-142-6 kubenswrapper[2564]: I0730 00:23:07.514382 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-cainjector-559cdd8c87-z874n" Jul 30 00:23:07.516936 ip-10-0-142-6 kubenswrapper[2564]: I0730 00:23:07.516914 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"cert-manager\"/\"cert-manager-cainjector-dockercfg-8mvkv\"" Jul 30 00:23:07.521370 ip-10-0-142-6 kubenswrapper[2564]: I0730 00:23:07.521348 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-cainjector-559cdd8c87-z874n"] Jul 30 00:23:07.557306 ip-10-0-142-6 kubenswrapper[2564]: I0730 00:23:07.557284 2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fk757h" Jul 30 00:23:07.645392 ip-10-0-142-6 kubenswrapper[2564]: I0730 00:23:07.645365 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-6d5jc\" (UniqueName: \"kubernetes.io/projected/cb25ceb5-a070-4f20-9486-c8aad93678a0-kube-api-access-6d5jc\") pod \"cb25ceb5-a070-4f20-9486-c8aad93678a0\" (UID: \"cb25ceb5-a070-4f20-9486-c8aad93678a0\") " Jul 30 00:23:07.645549 ip-10-0-142-6 kubenswrapper[2564]: I0730 00:23:07.645456 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/cb25ceb5-a070-4f20-9486-c8aad93678a0-bundle\") pod \"cb25ceb5-a070-4f20-9486-c8aad93678a0\" (UID: \"cb25ceb5-a070-4f20-9486-c8aad93678a0\") " Jul 30 00:23:07.645549 ip-10-0-142-6 kubenswrapper[2564]: I0730 00:23:07.645514 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/cb25ceb5-a070-4f20-9486-c8aad93678a0-util\") pod \"cb25ceb5-a070-4f20-9486-c8aad93678a0\" (UID: \"cb25ceb5-a070-4f20-9486-c8aad93678a0\") " Jul 30 00:23:07.645702 ip-10-0-142-6 kubenswrapper[2564]: I0730 00:23:07.645681 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/8b2dc2b3-2da2-4bf0-8b27-0466d03afd0f-bound-sa-token\") pod \"cert-manager-cainjector-559cdd8c87-z874n\" (UID: \"8b2dc2b3-2da2-4bf0-8b27-0466d03afd0f\") " pod="cert-manager/cert-manager-cainjector-559cdd8c87-z874n" Jul 30 00:23:07.645748 ip-10-0-142-6 kubenswrapper[2564]: I0730 00:23:07.645730 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zpxmw\" (UniqueName: \"kubernetes.io/projected/8b2dc2b3-2da2-4bf0-8b27-0466d03afd0f-kube-api-access-zpxmw\") pod \"cert-manager-cainjector-559cdd8c87-z874n\" (UID: \"8b2dc2b3-2da2-4bf0-8b27-0466d03afd0f\") " pod="cert-manager/cert-manager-cainjector-559cdd8c87-z874n" Jul 30 00:23:07.645819 ip-10-0-142-6 kubenswrapper[2564]: I0730 00:23:07.645795 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cb25ceb5-a070-4f20-9486-c8aad93678a0-bundle" (OuterVolumeSpecName: "bundle") pod "cb25ceb5-a070-4f20-9486-c8aad93678a0" (UID: "cb25ceb5-a070-4f20-9486-c8aad93678a0"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 30 00:23:07.647460 ip-10-0-142-6 kubenswrapper[2564]: I0730 00:23:07.647440 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cb25ceb5-a070-4f20-9486-c8aad93678a0-kube-api-access-6d5jc" (OuterVolumeSpecName: "kube-api-access-6d5jc") pod "cb25ceb5-a070-4f20-9486-c8aad93678a0" (UID: "cb25ceb5-a070-4f20-9486-c8aad93678a0"). InnerVolumeSpecName "kube-api-access-6d5jc". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 30 00:23:07.650462 ip-10-0-142-6 kubenswrapper[2564]: I0730 00:23:07.650431 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cb25ceb5-a070-4f20-9486-c8aad93678a0-util" (OuterVolumeSpecName: "util") pod "cb25ceb5-a070-4f20-9486-c8aad93678a0" (UID: "cb25ceb5-a070-4f20-9486-c8aad93678a0"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 30 00:23:07.746461 ip-10-0-142-6 kubenswrapper[2564]: I0730 00:23:07.746377 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/8b2dc2b3-2da2-4bf0-8b27-0466d03afd0f-bound-sa-token\") pod \"cert-manager-cainjector-559cdd8c87-z874n\" (UID: \"8b2dc2b3-2da2-4bf0-8b27-0466d03afd0f\") " pod="cert-manager/cert-manager-cainjector-559cdd8c87-z874n" Jul 30 00:23:07.746461 ip-10-0-142-6 kubenswrapper[2564]: I0730 00:23:07.746413 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zpxmw\" (UniqueName: \"kubernetes.io/projected/8b2dc2b3-2da2-4bf0-8b27-0466d03afd0f-kube-api-access-zpxmw\") pod \"cert-manager-cainjector-559cdd8c87-z874n\" (UID: \"8b2dc2b3-2da2-4bf0-8b27-0466d03afd0f\") " pod="cert-manager/cert-manager-cainjector-559cdd8c87-z874n" Jul 30 00:23:07.746461 ip-10-0-142-6 kubenswrapper[2564]: I0730 00:23:07.746461 2564 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6d5jc\" (UniqueName: \"kubernetes.io/projected/cb25ceb5-a070-4f20-9486-c8aad93678a0-kube-api-access-6d5jc\") on node \"ip-10-0-142-6.ec2.internal\" DevicePath \"\"" Jul 30 00:23:07.746661 ip-10-0-142-6 kubenswrapper[2564]: I0730 00:23:07.746470 2564 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/cb25ceb5-a070-4f20-9486-c8aad93678a0-bundle\") on node \"ip-10-0-142-6.ec2.internal\" DevicePath \"\"" Jul 30 00:23:07.746661 ip-10-0-142-6 kubenswrapper[2564]: I0730 00:23:07.746479 2564 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/cb25ceb5-a070-4f20-9486-c8aad93678a0-util\") on node \"ip-10-0-142-6.ec2.internal\" DevicePath \"\"" Jul 30 00:23:07.754536 ip-10-0-142-6 kubenswrapper[2564]: I0730 00:23:07.754513 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/8b2dc2b3-2da2-4bf0-8b27-0466d03afd0f-bound-sa-token\") pod \"cert-manager-cainjector-559cdd8c87-z874n\" (UID: \"8b2dc2b3-2da2-4bf0-8b27-0466d03afd0f\") " pod="cert-manager/cert-manager-cainjector-559cdd8c87-z874n" Jul 30 00:23:07.754668 ip-10-0-142-6 kubenswrapper[2564]: I0730 00:23:07.754647 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-zpxmw\" (UniqueName: \"kubernetes.io/projected/8b2dc2b3-2da2-4bf0-8b27-0466d03afd0f-kube-api-access-zpxmw\") pod \"cert-manager-cainjector-559cdd8c87-z874n\" (UID: \"8b2dc2b3-2da2-4bf0-8b27-0466d03afd0f\") " pod="cert-manager/cert-manager-cainjector-559cdd8c87-z874n" Jul 30 00:23:07.823768 ip-10-0-142-6 kubenswrapper[2564]: I0730 00:23:07.823730 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-cainjector-559cdd8c87-z874n" Jul 30 00:23:07.946138 ip-10-0-142-6 kubenswrapper[2564]: I0730 00:23:07.946050 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-cainjector-559cdd8c87-z874n"] Jul 30 00:23:07.948466 ip-10-0-142-6 kubenswrapper[2564]: W0730 00:23:07.948434 2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod8b2dc2b3_2da2_4bf0_8b27_0466d03afd0f.slice/crio-5e81f31e4c144b104ea2ee7e8b575f27b3876e7eb860f88753667afc2afe547f WatchSource:0}: Error finding container 5e81f31e4c144b104ea2ee7e8b575f27b3876e7eb860f88753667afc2afe547f: Status 404 returned error can't find the container with id 5e81f31e4c144b104ea2ee7e8b575f27b3876e7eb860f88753667afc2afe547f Jul 30 00:23:08.442416 ip-10-0-142-6 kubenswrapper[2564]: I0730 00:23:08.442378 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fk757h" event={"ID":"cb25ceb5-a070-4f20-9486-c8aad93678a0","Type":"ContainerDied","Data":"5fa3ef4abbd071b4721e4640d40d12056f9e272905c03431009cfa97d441e059"} Jul 30 00:23:08.442612 ip-10-0-142-6 kubenswrapper[2564]: I0730 00:23:08.442432 2564 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="5fa3ef4abbd071b4721e4640d40d12056f9e272905c03431009cfa97d441e059" Jul 30 00:23:08.442612 ip-10-0-142-6 kubenswrapper[2564]: I0730 00:23:08.442388 2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fk757h" Jul 30 00:23:08.443922 ip-10-0-142-6 kubenswrapper[2564]: I0730 00:23:08.443899 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-cainjector-559cdd8c87-z874n" event={"ID":"8b2dc2b3-2da2-4bf0-8b27-0466d03afd0f","Type":"ContainerStarted","Data":"11bf096f63b653ad2e449879f1e346c7eaa19946192a6c1bc95f9be3c8a6fb89"} Jul 30 00:23:08.444021 ip-10-0-142-6 kubenswrapper[2564]: I0730 00:23:08.443934 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-cainjector-559cdd8c87-z874n" event={"ID":"8b2dc2b3-2da2-4bf0-8b27-0466d03afd0f","Type":"ContainerStarted","Data":"5e81f31e4c144b104ea2ee7e8b575f27b3876e7eb860f88753667afc2afe547f"} Jul 30 00:23:08.459838 ip-10-0-142-6 kubenswrapper[2564]: I0730 00:23:08.459797 2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="cert-manager/cert-manager-cainjector-559cdd8c87-z874n" podStartSLOduration=1.459783976 podStartE2EDuration="1.459783976s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-30 00:23:07 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-30 00:23:08.458637814 +0000 UTC m=+319.873377751" watchObservedRunningTime="2026-07-30 00:23:08.459783976 +0000 UTC m=+319.874523908" Jul 30 00:23:11.433743 ip-10-0-142-6 kubenswrapper[2564]: I0730 00:23:11.433712 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="cert-manager/cert-manager-webhook-6486ffd997-57l94" Jul 30 00:23:22.056353 ip-10-0-142-6 kubenswrapper[2564]: I0730 00:23:22.056306 2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afkxlgw"] Jul 30 00:23:22.056804 ip-10-0-142-6 kubenswrapper[2564]: I0730 00:23:22.056574 2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="cb25ceb5-a070-4f20-9486-c8aad93678a0" containerName="pull" Jul 30 00:23:22.056804 ip-10-0-142-6 kubenswrapper[2564]: I0730 00:23:22.056584 2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="cb25ceb5-a070-4f20-9486-c8aad93678a0" containerName="pull" Jul 30 00:23:22.056804 ip-10-0-142-6 kubenswrapper[2564]: I0730 00:23:22.056595 2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="cb25ceb5-a070-4f20-9486-c8aad93678a0" containerName="extract" Jul 30 00:23:22.056804 ip-10-0-142-6 kubenswrapper[2564]: I0730 00:23:22.056601 2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="cb25ceb5-a070-4f20-9486-c8aad93678a0" containerName="extract" Jul 30 00:23:22.056804 ip-10-0-142-6 kubenswrapper[2564]: I0730 00:23:22.056616 2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="cb25ceb5-a070-4f20-9486-c8aad93678a0" containerName="util" Jul 30 00:23:22.056804 ip-10-0-142-6 kubenswrapper[2564]: I0730 00:23:22.056621 2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="cb25ceb5-a070-4f20-9486-c8aad93678a0" containerName="util" Jul 30 00:23:22.056804 ip-10-0-142-6 kubenswrapper[2564]: I0730 00:23:22.056677 2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="cb25ceb5-a070-4f20-9486-c8aad93678a0" containerName="extract" Jul 30 00:23:22.060903 ip-10-0-142-6 kubenswrapper[2564]: I0730 00:23:22.060886 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afkxlgw" Jul 30 00:23:22.063281 ip-10-0-142-6 kubenswrapper[2564]: I0730 00:23:22.063261 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"kube-root-ca.crt\"" Jul 30 00:23:22.064114 ip-10-0-142-6 kubenswrapper[2564]: I0730 00:23:22.064096 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"default-dockercfg-wdqwh\"" Jul 30 00:23:22.064114 ip-10-0-142-6 kubenswrapper[2564]: I0730 00:23:22.064107 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"openshift-service-ca.crt\"" Jul 30 00:23:22.066365 ip-10-0-142-6 kubenswrapper[2564]: I0730 00:23:22.066334 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afkxlgw"] Jul 30 00:23:22.160404 ip-10-0-142-6 kubenswrapper[2564]: I0730 00:23:22.160370 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8rjqx\" (UniqueName: \"kubernetes.io/projected/7afba4f5-99c1-4365-b000-11fdca797e89-kube-api-access-8rjqx\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afkxlgw\" (UID: \"7afba4f5-99c1-4365-b000-11fdca797e89\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afkxlgw" Jul 30 00:23:22.160576 ip-10-0-142-6 kubenswrapper[2564]: I0730 00:23:22.160442 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/7afba4f5-99c1-4365-b000-11fdca797e89-bundle\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afkxlgw\" (UID: \"7afba4f5-99c1-4365-b000-11fdca797e89\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afkxlgw" Jul 30 00:23:22.160576 ip-10-0-142-6 kubenswrapper[2564]: I0730 00:23:22.160486 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/7afba4f5-99c1-4365-b000-11fdca797e89-util\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afkxlgw\" (UID: \"7afba4f5-99c1-4365-b000-11fdca797e89\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afkxlgw" Jul 30 00:23:22.261734 ip-10-0-142-6 kubenswrapper[2564]: I0730 00:23:22.261699 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/7afba4f5-99c1-4365-b000-11fdca797e89-util\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afkxlgw\" (UID: \"7afba4f5-99c1-4365-b000-11fdca797e89\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afkxlgw" Jul 30 00:23:22.261918 ip-10-0-142-6 kubenswrapper[2564]: I0730 00:23:22.261744 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8rjqx\" (UniqueName: \"kubernetes.io/projected/7afba4f5-99c1-4365-b000-11fdca797e89-kube-api-access-8rjqx\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afkxlgw\" (UID: \"7afba4f5-99c1-4365-b000-11fdca797e89\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afkxlgw" Jul 30 00:23:22.261918 ip-10-0-142-6 kubenswrapper[2564]: I0730 00:23:22.261785 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/7afba4f5-99c1-4365-b000-11fdca797e89-bundle\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afkxlgw\" (UID: \"7afba4f5-99c1-4365-b000-11fdca797e89\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afkxlgw" Jul 30 00:23:22.262225 ip-10-0-142-6 kubenswrapper[2564]: I0730 00:23:22.262209 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/7afba4f5-99c1-4365-b000-11fdca797e89-bundle\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afkxlgw\" (UID: \"7afba4f5-99c1-4365-b000-11fdca797e89\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afkxlgw" Jul 30 00:23:22.262276 ip-10-0-142-6 kubenswrapper[2564]: I0730 00:23:22.262205 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/7afba4f5-99c1-4365-b000-11fdca797e89-util\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afkxlgw\" (UID: \"7afba4f5-99c1-4365-b000-11fdca797e89\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afkxlgw" Jul 30 00:23:22.270180 ip-10-0-142-6 kubenswrapper[2564]: I0730 00:23:22.270148 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8rjqx\" (UniqueName: \"kubernetes.io/projected/7afba4f5-99c1-4365-b000-11fdca797e89-kube-api-access-8rjqx\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afkxlgw\" (UID: \"7afba4f5-99c1-4365-b000-11fdca797e89\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afkxlgw" Jul 30 00:23:22.370757 ip-10-0-142-6 kubenswrapper[2564]: I0730 00:23:22.370729 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afkxlgw" Jul 30 00:23:22.491873 ip-10-0-142-6 kubenswrapper[2564]: I0730 00:23:22.491841 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afkxlgw"] Jul 30 00:23:22.495137 ip-10-0-142-6 kubenswrapper[2564]: W0730 00:23:22.495103 2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7afba4f5_99c1_4365_b000_11fdca797e89.slice/crio-838feda56090f96888b9106331a45a404f7d055ec59ec12a9bdb35b4c2f41a0d WatchSource:0}: Error finding container 838feda56090f96888b9106331a45a404f7d055ec59ec12a9bdb35b4c2f41a0d: Status 404 returned error can't find the container with id 838feda56090f96888b9106331a45a404f7d055ec59ec12a9bdb35b4c2f41a0d Jul 30 00:23:23.504149 ip-10-0-142-6 kubenswrapper[2564]: I0730 00:23:23.504112 2564 generic.go:356] "Generic (PLEG): container finished" podID="7afba4f5-99c1-4365-b000-11fdca797e89" containerID="62cab7b0da2e40735642725dbde8c4496fdf89c14898d778bf38dc19c124e265" exitCode=0 Jul 30 00:23:23.504557 ip-10-0-142-6 kubenswrapper[2564]: I0730 00:23:23.504197 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afkxlgw" event={"ID":"7afba4f5-99c1-4365-b000-11fdca797e89","Type":"ContainerDied","Data":"62cab7b0da2e40735642725dbde8c4496fdf89c14898d778bf38dc19c124e265"} Jul 30 00:23:23.504557 ip-10-0-142-6 kubenswrapper[2564]: I0730 00:23:23.504250 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afkxlgw" event={"ID":"7afba4f5-99c1-4365-b000-11fdca797e89","Type":"ContainerStarted","Data":"838feda56090f96888b9106331a45a404f7d055ec59ec12a9bdb35b4c2f41a0d"} Jul 30 00:23:24.509989 ip-10-0-142-6 kubenswrapper[2564]: I0730 00:23:24.509950 2564 generic.go:356] "Generic (PLEG): container finished" podID="7afba4f5-99c1-4365-b000-11fdca797e89" containerID="216efd051853063c69379457fd8155dbeaca08b24432fa4298a201d597bbd37f" exitCode=0 Jul 30 00:23:24.510373 ip-10-0-142-6 kubenswrapper[2564]: I0730 00:23:24.510038 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afkxlgw" event={"ID":"7afba4f5-99c1-4365-b000-11fdca797e89","Type":"ContainerDied","Data":"216efd051853063c69379457fd8155dbeaca08b24432fa4298a201d597bbd37f"} Jul 30 00:23:25.516317 ip-10-0-142-6 kubenswrapper[2564]: I0730 00:23:25.516284 2564 generic.go:356] "Generic (PLEG): container finished" podID="7afba4f5-99c1-4365-b000-11fdca797e89" containerID="15f1ff8b612bca4c32f89888b51191f614aae3662fea0cc9ee5065668a5cd940" exitCode=0 Jul 30 00:23:25.516708 ip-10-0-142-6 kubenswrapper[2564]: I0730 00:23:25.516372 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afkxlgw" event={"ID":"7afba4f5-99c1-4365-b000-11fdca797e89","Type":"ContainerDied","Data":"15f1ff8b612bca4c32f89888b51191f614aae3662fea0cc9ee5065668a5cd940"} Jul 30 00:23:26.637137 ip-10-0-142-6 kubenswrapper[2564]: I0730 00:23:26.637114 2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afkxlgw" Jul 30 00:23:26.695625 ip-10-0-142-6 kubenswrapper[2564]: I0730 00:23:26.695600 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/7afba4f5-99c1-4365-b000-11fdca797e89-util\") pod \"7afba4f5-99c1-4365-b000-11fdca797e89\" (UID: \"7afba4f5-99c1-4365-b000-11fdca797e89\") " Jul 30 00:23:26.695754 ip-10-0-142-6 kubenswrapper[2564]: I0730 00:23:26.695698 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/7afba4f5-99c1-4365-b000-11fdca797e89-bundle\") pod \"7afba4f5-99c1-4365-b000-11fdca797e89\" (UID: \"7afba4f5-99c1-4365-b000-11fdca797e89\") " Jul 30 00:23:26.695754 ip-10-0-142-6 kubenswrapper[2564]: I0730 00:23:26.695727 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-8rjqx\" (UniqueName: \"kubernetes.io/projected/7afba4f5-99c1-4365-b000-11fdca797e89-kube-api-access-8rjqx\") pod \"7afba4f5-99c1-4365-b000-11fdca797e89\" (UID: \"7afba4f5-99c1-4365-b000-11fdca797e89\") " Jul 30 00:23:26.696409 ip-10-0-142-6 kubenswrapper[2564]: I0730 00:23:26.696383 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7afba4f5-99c1-4365-b000-11fdca797e89-bundle" (OuterVolumeSpecName: "bundle") pod "7afba4f5-99c1-4365-b000-11fdca797e89" (UID: "7afba4f5-99c1-4365-b000-11fdca797e89"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 30 00:23:26.697769 ip-10-0-142-6 kubenswrapper[2564]: I0730 00:23:26.697744 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7afba4f5-99c1-4365-b000-11fdca797e89-kube-api-access-8rjqx" (OuterVolumeSpecName: "kube-api-access-8rjqx") pod "7afba4f5-99c1-4365-b000-11fdca797e89" (UID: "7afba4f5-99c1-4365-b000-11fdca797e89"). InnerVolumeSpecName "kube-api-access-8rjqx". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 30 00:23:26.701451 ip-10-0-142-6 kubenswrapper[2564]: I0730 00:23:26.701356 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7afba4f5-99c1-4365-b000-11fdca797e89-util" (OuterVolumeSpecName: "util") pod "7afba4f5-99c1-4365-b000-11fdca797e89" (UID: "7afba4f5-99c1-4365-b000-11fdca797e89"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 30 00:23:26.796248 ip-10-0-142-6 kubenswrapper[2564]: I0730 00:23:26.796180 2564 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/7afba4f5-99c1-4365-b000-11fdca797e89-util\") on node \"ip-10-0-142-6.ec2.internal\" DevicePath \"\"" Jul 30 00:23:26.796248 ip-10-0-142-6 kubenswrapper[2564]: I0730 00:23:26.796204 2564 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/7afba4f5-99c1-4365-b000-11fdca797e89-bundle\") on node \"ip-10-0-142-6.ec2.internal\" DevicePath \"\"" Jul 30 00:23:26.796248 ip-10-0-142-6 kubenswrapper[2564]: I0730 00:23:26.796212 2564 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8rjqx\" (UniqueName: \"kubernetes.io/projected/7afba4f5-99c1-4365-b000-11fdca797e89-kube-api-access-8rjqx\") on node \"ip-10-0-142-6.ec2.internal\" DevicePath \"\"" Jul 30 00:23:27.526193 ip-10-0-142-6 kubenswrapper[2564]: I0730 00:23:27.526115 2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afkxlgw" Jul 30 00:23:27.526193 ip-10-0-142-6 kubenswrapper[2564]: I0730 00:23:27.526118 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afkxlgw" event={"ID":"7afba4f5-99c1-4365-b000-11fdca797e89","Type":"ContainerDied","Data":"838feda56090f96888b9106331a45a404f7d055ec59ec12a9bdb35b4c2f41a0d"} Jul 30 00:23:27.526193 ip-10-0-142-6 kubenswrapper[2564]: I0730 00:23:27.526161 2564 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="838feda56090f96888b9106331a45a404f7d055ec59ec12a9bdb35b4c2f41a0d" Jul 30 00:23:36.539661 ip-10-0-142-6 kubenswrapper[2564]: I0730 00:23:36.539627 2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-lws-operator/lws-controller-manager-685c7855bf-xzzzz"] Jul 30 00:23:36.540054 ip-10-0-142-6 kubenswrapper[2564]: I0730 00:23:36.539899 2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7afba4f5-99c1-4365-b000-11fdca797e89" containerName="util" Jul 30 00:23:36.540054 ip-10-0-142-6 kubenswrapper[2564]: I0730 00:23:36.539910 2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="7afba4f5-99c1-4365-b000-11fdca797e89" containerName="util" Jul 30 00:23:36.540054 ip-10-0-142-6 kubenswrapper[2564]: I0730 00:23:36.539917 2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7afba4f5-99c1-4365-b000-11fdca797e89" containerName="extract" Jul 30 00:23:36.540054 ip-10-0-142-6 kubenswrapper[2564]: I0730 00:23:36.539923 2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="7afba4f5-99c1-4365-b000-11fdca797e89" containerName="extract" Jul 30 00:23:36.540054 ip-10-0-142-6 kubenswrapper[2564]: I0730 00:23:36.539943 2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7afba4f5-99c1-4365-b000-11fdca797e89" containerName="pull" Jul 30 00:23:36.540054 ip-10-0-142-6 kubenswrapper[2564]: I0730 00:23:36.539949 2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="7afba4f5-99c1-4365-b000-11fdca797e89" containerName="pull" Jul 30 00:23:36.540054 ip-10-0-142-6 kubenswrapper[2564]: I0730 00:23:36.539989 2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="7afba4f5-99c1-4365-b000-11fdca797e89" containerName="extract" Jul 30 00:23:36.542184 ip-10-0-142-6 kubenswrapper[2564]: I0730 00:23:36.542169 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/lws-controller-manager-685c7855bf-xzzzz" Jul 30 00:23:36.544730 ip-10-0-142-6 kubenswrapper[2564]: I0730 00:23:36.544708 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"webhook-server-cert\"" Jul 30 00:23:36.544845 ip-10-0-142-6 kubenswrapper[2564]: I0730 00:23:36.544708 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"lws-manager-config\"" Jul 30 00:23:36.545909 ip-10-0-142-6 kubenswrapper[2564]: I0730 00:23:36.545883 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"metrics-server-cert\"" Jul 30 00:23:36.545909 ip-10-0-142-6 kubenswrapper[2564]: I0730 00:23:36.545899 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"openshift-service-ca.crt\"" Jul 30 00:23:36.546059 ip-10-0-142-6 kubenswrapper[2564]: I0730 00:23:36.545995 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"lws-controller-manager-dockercfg-tjgkc\"" Jul 30 00:23:36.546301 ip-10-0-142-6 kubenswrapper[2564]: I0730 00:23:36.546214 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"kube-root-ca.crt\"" Jul 30 00:23:36.554792 ip-10-0-142-6 kubenswrapper[2564]: I0730 00:23:36.554770 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/lws-controller-manager-685c7855bf-xzzzz"] Jul 30 00:23:36.570340 ip-10-0-142-6 kubenswrapper[2564]: I0730 00:23:36.570298 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ddw9x\" (UniqueName: \"kubernetes.io/projected/8c9f42c1-c9d8-4a60-b2d5-97150a0a9ef7-kube-api-access-ddw9x\") pod \"lws-controller-manager-685c7855bf-xzzzz\" (UID: \"8c9f42c1-c9d8-4a60-b2d5-97150a0a9ef7\") " pod="openshift-lws-operator/lws-controller-manager-685c7855bf-xzzzz" Jul 30 00:23:36.570340 ip-10-0-142-6 kubenswrapper[2564]: I0730 00:23:36.570349 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/8c9f42c1-c9d8-4a60-b2d5-97150a0a9ef7-cert\") pod \"lws-controller-manager-685c7855bf-xzzzz\" (UID: \"8c9f42c1-c9d8-4a60-b2d5-97150a0a9ef7\") " pod="openshift-lws-operator/lws-controller-manager-685c7855bf-xzzzz" Jul 30 00:23:36.570512 ip-10-0-142-6 kubenswrapper[2564]: I0730 00:23:36.570467 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/8c9f42c1-c9d8-4a60-b2d5-97150a0a9ef7-manager-config\") pod \"lws-controller-manager-685c7855bf-xzzzz\" (UID: \"8c9f42c1-c9d8-4a60-b2d5-97150a0a9ef7\") " pod="openshift-lws-operator/lws-controller-manager-685c7855bf-xzzzz" Jul 30 00:23:36.570512 ip-10-0-142-6 kubenswrapper[2564]: I0730 00:23:36.570497 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/8c9f42c1-c9d8-4a60-b2d5-97150a0a9ef7-metrics-cert\") pod \"lws-controller-manager-685c7855bf-xzzzz\" (UID: \"8c9f42c1-c9d8-4a60-b2d5-97150a0a9ef7\") " pod="openshift-lws-operator/lws-controller-manager-685c7855bf-xzzzz" Jul 30 00:23:36.671826 ip-10-0-142-6 kubenswrapper[2564]: I0730 00:23:36.671794 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/8c9f42c1-c9d8-4a60-b2d5-97150a0a9ef7-manager-config\") pod \"lws-controller-manager-685c7855bf-xzzzz\" (UID: \"8c9f42c1-c9d8-4a60-b2d5-97150a0a9ef7\") " pod="openshift-lws-operator/lws-controller-manager-685c7855bf-xzzzz" Jul 30 00:23:36.671826 ip-10-0-142-6 kubenswrapper[2564]: I0730 00:23:36.671830 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/8c9f42c1-c9d8-4a60-b2d5-97150a0a9ef7-metrics-cert\") pod \"lws-controller-manager-685c7855bf-xzzzz\" (UID: \"8c9f42c1-c9d8-4a60-b2d5-97150a0a9ef7\") " pod="openshift-lws-operator/lws-controller-manager-685c7855bf-xzzzz" Jul 30 00:23:36.672049 ip-10-0-142-6 kubenswrapper[2564]: I0730 00:23:36.671853 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ddw9x\" (UniqueName: \"kubernetes.io/projected/8c9f42c1-c9d8-4a60-b2d5-97150a0a9ef7-kube-api-access-ddw9x\") pod \"lws-controller-manager-685c7855bf-xzzzz\" (UID: \"8c9f42c1-c9d8-4a60-b2d5-97150a0a9ef7\") " pod="openshift-lws-operator/lws-controller-manager-685c7855bf-xzzzz" Jul 30 00:23:36.672049 ip-10-0-142-6 kubenswrapper[2564]: I0730 00:23:36.671873 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/8c9f42c1-c9d8-4a60-b2d5-97150a0a9ef7-cert\") pod \"lws-controller-manager-685c7855bf-xzzzz\" (UID: \"8c9f42c1-c9d8-4a60-b2d5-97150a0a9ef7\") " pod="openshift-lws-operator/lws-controller-manager-685c7855bf-xzzzz" Jul 30 00:23:36.672551 ip-10-0-142-6 kubenswrapper[2564]: I0730 00:23:36.672532 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/8c9f42c1-c9d8-4a60-b2d5-97150a0a9ef7-manager-config\") pod \"lws-controller-manager-685c7855bf-xzzzz\" (UID: \"8c9f42c1-c9d8-4a60-b2d5-97150a0a9ef7\") " pod="openshift-lws-operator/lws-controller-manager-685c7855bf-xzzzz" Jul 30 00:23:36.674371 ip-10-0-142-6 kubenswrapper[2564]: I0730 00:23:36.674351 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/8c9f42c1-c9d8-4a60-b2d5-97150a0a9ef7-cert\") pod \"lws-controller-manager-685c7855bf-xzzzz\" (UID: \"8c9f42c1-c9d8-4a60-b2d5-97150a0a9ef7\") " pod="openshift-lws-operator/lws-controller-manager-685c7855bf-xzzzz" Jul 30 00:23:36.674460 ip-10-0-142-6 kubenswrapper[2564]: I0730 00:23:36.674385 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/8c9f42c1-c9d8-4a60-b2d5-97150a0a9ef7-metrics-cert\") pod \"lws-controller-manager-685c7855bf-xzzzz\" (UID: \"8c9f42c1-c9d8-4a60-b2d5-97150a0a9ef7\") " pod="openshift-lws-operator/lws-controller-manager-685c7855bf-xzzzz" Jul 30 00:23:36.684406 ip-10-0-142-6 kubenswrapper[2564]: I0730 00:23:36.684387 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ddw9x\" (UniqueName: \"kubernetes.io/projected/8c9f42c1-c9d8-4a60-b2d5-97150a0a9ef7-kube-api-access-ddw9x\") pod \"lws-controller-manager-685c7855bf-xzzzz\" (UID: \"8c9f42c1-c9d8-4a60-b2d5-97150a0a9ef7\") " pod="openshift-lws-operator/lws-controller-manager-685c7855bf-xzzzz" Jul 30 00:23:36.852900 ip-10-0-142-6 kubenswrapper[2564]: I0730 00:23:36.852810 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/lws-controller-manager-685c7855bf-xzzzz" Jul 30 00:23:36.982207 ip-10-0-142-6 kubenswrapper[2564]: I0730 00:23:36.982185 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/lws-controller-manager-685c7855bf-xzzzz"] Jul 30 00:23:36.984310 ip-10-0-142-6 kubenswrapper[2564]: W0730 00:23:36.984287 2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8c9f42c1_c9d8_4a60_b2d5_97150a0a9ef7.slice/crio-4fab1f58af330ec3cfdd542bf1d0a0555cf90481b678ce464822d7fb4b918897 WatchSource:0}: Error finding container 4fab1f58af330ec3cfdd542bf1d0a0555cf90481b678ce464822d7fb4b918897: Status 404 returned error can't find the container with id 4fab1f58af330ec3cfdd542bf1d0a0555cf90481b678ce464822d7fb4b918897 Jul 30 00:23:37.568647 ip-10-0-142-6 kubenswrapper[2564]: I0730 00:23:37.568615 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/lws-controller-manager-685c7855bf-xzzzz" event={"ID":"8c9f42c1-c9d8-4a60-b2d5-97150a0a9ef7","Type":"ContainerStarted","Data":"4fab1f58af330ec3cfdd542bf1d0a0555cf90481b678ce464822d7fb4b918897"} Jul 30 00:23:38.998406 ip-10-0-142-6 kubenswrapper[2564]: I0730 00:23:38.998372 2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-5b74bc5486-thrgp"] Jul 30 00:23:39.000968 ip-10-0-142-6 kubenswrapper[2564]: I0730 00:23:39.000949 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/opendatahub-operator-controller-manager-5b74bc5486-thrgp" Jul 30 00:23:39.004192 ip-10-0-142-6 kubenswrapper[2564]: I0730 00:23:39.004170 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-manager-service-cert\"" Jul 30 00:23:39.004561 ip-10-0-142-6 kubenswrapper[2564]: I0730 00:23:39.004535 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"openshift-service-ca.crt\"" Jul 30 00:23:39.004883 ip-10-0-142-6 kubenswrapper[2564]: I0730 00:23:39.004848 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-webhook-cert\"" Jul 30 00:23:39.005164 ip-10-0-142-6 kubenswrapper[2564]: I0730 00:23:39.004992 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-manager-dockercfg-kwj6t\"" Jul 30 00:23:39.005262 ip-10-0-142-6 kubenswrapper[2564]: I0730 00:23:39.005188 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"kube-root-ca.crt\"" Jul 30 00:23:39.027383 ip-10-0-142-6 kubenswrapper[2564]: I0730 00:23:39.027358 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-5b74bc5486-thrgp"] Jul 30 00:23:39.089551 ip-10-0-142-6 kubenswrapper[2564]: I0730 00:23:39.089517 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/db0f2405-58a7-4112-af82-65d1d07bb5a9-tmp\") pod \"opendatahub-operator-controller-manager-5b74bc5486-thrgp\" (UID: \"db0f2405-58a7-4112-af82-65d1d07bb5a9\") " pod="opendatahub/opendatahub-operator-controller-manager-5b74bc5486-thrgp" Jul 30 00:23:39.089765 ip-10-0-142-6 kubenswrapper[2564]: I0730 00:23:39.089563 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/db0f2405-58a7-4112-af82-65d1d07bb5a9-manifests\") pod \"opendatahub-operator-controller-manager-5b74bc5486-thrgp\" (UID: \"db0f2405-58a7-4112-af82-65d1d07bb5a9\") " pod="opendatahub/opendatahub-operator-controller-manager-5b74bc5486-thrgp" Jul 30 00:23:39.089765 ip-10-0-142-6 kubenswrapper[2564]: I0730 00:23:39.089593 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/db0f2405-58a7-4112-af82-65d1d07bb5a9-apiservice-cert\") pod \"opendatahub-operator-controller-manager-5b74bc5486-thrgp\" (UID: \"db0f2405-58a7-4112-af82-65d1d07bb5a9\") " pod="opendatahub/opendatahub-operator-controller-manager-5b74bc5486-thrgp" Jul 30 00:23:39.089765 ip-10-0-142-6 kubenswrapper[2564]: I0730 00:23:39.089667 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qcwn4\" (UniqueName: \"kubernetes.io/projected/db0f2405-58a7-4112-af82-65d1d07bb5a9-kube-api-access-qcwn4\") pod \"opendatahub-operator-controller-manager-5b74bc5486-thrgp\" (UID: \"db0f2405-58a7-4112-af82-65d1d07bb5a9\") " pod="opendatahub/opendatahub-operator-controller-manager-5b74bc5486-thrgp" Jul 30 00:23:39.089765 ip-10-0-142-6 kubenswrapper[2564]: I0730 00:23:39.089707 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/db0f2405-58a7-4112-af82-65d1d07bb5a9-webhook-cert\") pod \"opendatahub-operator-controller-manager-5b74bc5486-thrgp\" (UID: \"db0f2405-58a7-4112-af82-65d1d07bb5a9\") " pod="opendatahub/opendatahub-operator-controller-manager-5b74bc5486-thrgp" Jul 30 00:23:39.191166 ip-10-0-142-6 kubenswrapper[2564]: I0730 00:23:39.191134 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/db0f2405-58a7-4112-af82-65d1d07bb5a9-apiservice-cert\") pod \"opendatahub-operator-controller-manager-5b74bc5486-thrgp\" (UID: \"db0f2405-58a7-4112-af82-65d1d07bb5a9\") " pod="opendatahub/opendatahub-operator-controller-manager-5b74bc5486-thrgp" Jul 30 00:23:39.191359 ip-10-0-142-6 kubenswrapper[2564]: I0730 00:23:39.191202 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qcwn4\" (UniqueName: \"kubernetes.io/projected/db0f2405-58a7-4112-af82-65d1d07bb5a9-kube-api-access-qcwn4\") pod \"opendatahub-operator-controller-manager-5b74bc5486-thrgp\" (UID: \"db0f2405-58a7-4112-af82-65d1d07bb5a9\") " pod="opendatahub/opendatahub-operator-controller-manager-5b74bc5486-thrgp" Jul 30 00:23:39.191359 ip-10-0-142-6 kubenswrapper[2564]: I0730 00:23:39.191227 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/db0f2405-58a7-4112-af82-65d1d07bb5a9-webhook-cert\") pod \"opendatahub-operator-controller-manager-5b74bc5486-thrgp\" (UID: \"db0f2405-58a7-4112-af82-65d1d07bb5a9\") " pod="opendatahub/opendatahub-operator-controller-manager-5b74bc5486-thrgp" Jul 30 00:23:39.191359 ip-10-0-142-6 kubenswrapper[2564]: I0730 00:23:39.191308 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/db0f2405-58a7-4112-af82-65d1d07bb5a9-tmp\") pod \"opendatahub-operator-controller-manager-5b74bc5486-thrgp\" (UID: \"db0f2405-58a7-4112-af82-65d1d07bb5a9\") " pod="opendatahub/opendatahub-operator-controller-manager-5b74bc5486-thrgp" Jul 30 00:23:39.191359 ip-10-0-142-6 kubenswrapper[2564]: I0730 00:23:39.191334 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/db0f2405-58a7-4112-af82-65d1d07bb5a9-manifests\") pod \"opendatahub-operator-controller-manager-5b74bc5486-thrgp\" (UID: \"db0f2405-58a7-4112-af82-65d1d07bb5a9\") " pod="opendatahub/opendatahub-operator-controller-manager-5b74bc5486-thrgp" Jul 30 00:23:39.191722 ip-10-0-142-6 kubenswrapper[2564]: I0730 00:23:39.191699 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/db0f2405-58a7-4112-af82-65d1d07bb5a9-manifests\") pod \"opendatahub-operator-controller-manager-5b74bc5486-thrgp\" (UID: \"db0f2405-58a7-4112-af82-65d1d07bb5a9\") " pod="opendatahub/opendatahub-operator-controller-manager-5b74bc5486-thrgp" Jul 30 00:23:39.193644 ip-10-0-142-6 kubenswrapper[2564]: I0730 00:23:39.193610 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/db0f2405-58a7-4112-af82-65d1d07bb5a9-tmp\") pod \"opendatahub-operator-controller-manager-5b74bc5486-thrgp\" (UID: \"db0f2405-58a7-4112-af82-65d1d07bb5a9\") " pod="opendatahub/opendatahub-operator-controller-manager-5b74bc5486-thrgp" Jul 30 00:23:39.193813 ip-10-0-142-6 kubenswrapper[2564]: I0730 00:23:39.193795 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/db0f2405-58a7-4112-af82-65d1d07bb5a9-webhook-cert\") pod \"opendatahub-operator-controller-manager-5b74bc5486-thrgp\" (UID: \"db0f2405-58a7-4112-af82-65d1d07bb5a9\") " pod="opendatahub/opendatahub-operator-controller-manager-5b74bc5486-thrgp" Jul 30 00:23:39.194059 ip-10-0-142-6 kubenswrapper[2564]: I0730 00:23:39.194041 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/db0f2405-58a7-4112-af82-65d1d07bb5a9-apiservice-cert\") pod \"opendatahub-operator-controller-manager-5b74bc5486-thrgp\" (UID: \"db0f2405-58a7-4112-af82-65d1d07bb5a9\") " pod="opendatahub/opendatahub-operator-controller-manager-5b74bc5486-thrgp" Jul 30 00:23:39.198680 ip-10-0-142-6 kubenswrapper[2564]: I0730 00:23:39.198661 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qcwn4\" (UniqueName: \"kubernetes.io/projected/db0f2405-58a7-4112-af82-65d1d07bb5a9-kube-api-access-qcwn4\") pod \"opendatahub-operator-controller-manager-5b74bc5486-thrgp\" (UID: \"db0f2405-58a7-4112-af82-65d1d07bb5a9\") " pod="opendatahub/opendatahub-operator-controller-manager-5b74bc5486-thrgp" Jul 30 00:23:39.314067 ip-10-0-142-6 kubenswrapper[2564]: I0730 00:23:39.313989 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/opendatahub-operator-controller-manager-5b74bc5486-thrgp" Jul 30 00:23:39.451404 ip-10-0-142-6 kubenswrapper[2564]: I0730 00:23:39.451345 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-5b74bc5486-thrgp"] Jul 30 00:23:39.454361 ip-10-0-142-6 kubenswrapper[2564]: W0730 00:23:39.454335 2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddb0f2405_58a7_4112_af82_65d1d07bb5a9.slice/crio-90e4949d31436aa1c93846b6c714c7d11e81801ffc859c40eec45252f987453c WatchSource:0}: Error finding container 90e4949d31436aa1c93846b6c714c7d11e81801ffc859c40eec45252f987453c: Status 404 returned error can't find the container with id 90e4949d31436aa1c93846b6c714c7d11e81801ffc859c40eec45252f987453c Jul 30 00:23:39.579047 ip-10-0-142-6 kubenswrapper[2564]: I0730 00:23:39.578965 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-5b74bc5486-thrgp" event={"ID":"db0f2405-58a7-4112-af82-65d1d07bb5a9","Type":"ContainerStarted","Data":"90e4949d31436aa1c93846b6c714c7d11e81801ffc859c40eec45252f987453c"} Jul 30 00:23:40.590536 ip-10-0-142-6 kubenswrapper[2564]: I0730 00:23:40.590406 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/lws-controller-manager-685c7855bf-xzzzz" event={"ID":"8c9f42c1-c9d8-4a60-b2d5-97150a0a9ef7","Type":"ContainerStarted","Data":"e853551fbb2a448c12528e1641c0caeb421872bbd7816a46cca2cdbfc120f809"} Jul 30 00:23:40.590536 ip-10-0-142-6 kubenswrapper[2564]: I0730 00:23:40.590515 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-lws-operator/lws-controller-manager-685c7855bf-xzzzz" Jul 30 00:23:40.617620 ip-10-0-142-6 kubenswrapper[2564]: I0730 00:23:40.617573 2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-lws-operator/lws-controller-manager-685c7855bf-xzzzz" podStartSLOduration=1.4612181180000001 podStartE2EDuration="4.617558599s" totalImagesPullingTime="3.156340481s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-30 00:23:36 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-30 00:23:40.617130094 +0000 UTC m=+352.031870027" watchObservedRunningTime="2026-07-30 00:23:40.617558599 +0000 UTC m=+352.032298532" Jul 30 00:23:43.606087 ip-10-0-142-6 kubenswrapper[2564]: I0730 00:23:43.606048 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-5b74bc5486-thrgp" event={"ID":"db0f2405-58a7-4112-af82-65d1d07bb5a9","Type":"ContainerStarted","Data":"0e711aadac525202e1558e804b89a96e2c4b4f56c353640f32276ed3a6c6ab0c"} Jul 30 00:23:44.611360 ip-10-0-142-6 kubenswrapper[2564]: I0730 00:23:44.611329 2564 generic.go:356] "Generic (PLEG): container finished" podID="db0f2405-58a7-4112-af82-65d1d07bb5a9" containerID="0e711aadac525202e1558e804b89a96e2c4b4f56c353640f32276ed3a6c6ab0c" exitCode=0 Jul 30 00:23:44.611737 ip-10-0-142-6 kubenswrapper[2564]: I0730 00:23:44.611400 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-5b74bc5486-thrgp" event={"ID":"db0f2405-58a7-4112-af82-65d1d07bb5a9","Type":"ContainerDied","Data":"0e711aadac525202e1558e804b89a96e2c4b4f56c353640f32276ed3a6c6ab0c"} Jul 30 00:23:45.617201 ip-10-0-142-6 kubenswrapper[2564]: I0730 00:23:45.617172 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-5b74bc5486-thrgp" event={"ID":"db0f2405-58a7-4112-af82-65d1d07bb5a9","Type":"ContainerStarted","Data":"4521f05ab8a88074ac798cfaba56df6f97531a20dcbf1e4355d8b04e9cc35d80"} Jul 30 00:23:45.617580 ip-10-0-142-6 kubenswrapper[2564]: I0730 00:23:45.617420 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/opendatahub-operator-controller-manager-5b74bc5486-thrgp" Jul 30 00:23:45.637784 ip-10-0-142-6 kubenswrapper[2564]: I0730 00:23:45.637738 2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/opendatahub-operator-controller-manager-5b74bc5486-thrgp" podStartSLOduration=2.899865884 podStartE2EDuration="7.637723623s" totalImagesPullingTime="3.72031264s" totalInitContainerRuntime="1.017545099s" isStatefulPod=false podCreationTimestamp="2026-07-30 00:23:38 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-30 00:23:45.635688394 +0000 UTC m=+357.050428327" watchObservedRunningTime="2026-07-30 00:23:45.637723623 +0000 UTC m=+357.052463625" Jul 30 00:23:51.597458 ip-10-0-142-6 kubenswrapper[2564]: I0730 00:23:51.597431 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-lws-operator/lws-controller-manager-685c7855bf-xzzzz" Jul 30 00:23:56.624542 ip-10-0-142-6 kubenswrapper[2564]: I0730 00:23:56.624509 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/opendatahub-operator-controller-manager-5b74bc5486-thrgp" Jul 30 00:24:06.821894 ip-10-0-142-6 kubenswrapper[2564]: I0730 00:24:06.821856 2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/istiod-openshift-gateway-5677ccb49b-ww47t"] Jul 30 00:24:06.834864 ip-10-0-142-6 kubenswrapper[2564]: I0730 00:24:06.834834 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-ww47t" Jul 30 00:24:06.837741 ip-10-0-142-6 kubenswrapper[2564]: I0730 00:24:06.837716 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-gw-ca-root-cert\"" Jul 30 00:24:06.837906 ip-10-0-142-6 kubenswrapper[2564]: I0730 00:24:06.837827 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istiod-tls\"" Jul 30 00:24:06.837906 ip-10-0-142-6 kubenswrapper[2564]: I0730 00:24:06.837855 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-service-ca.crt\"" Jul 30 00:24:06.838011 ip-10-0-142-6 kubenswrapper[2564]: I0730 00:24:06.837943 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istiod-openshift-gateway-dockercfg-x2thh\"" Jul 30 00:24:06.838011 ip-10-0-142-6 kubenswrapper[2564]: I0730 00:24:06.837999 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"kube-root-ca.crt\"" Jul 30 00:24:06.838222 ip-10-0-142-6 kubenswrapper[2564]: I0730 00:24:06.838198 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"cacerts\"" Jul 30 00:24:06.838336 ip-10-0-142-6 kubenswrapper[2564]: I0730 00:24:06.838260 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istio-kubeconfig\"" Jul 30 00:24:06.843786 ip-10-0-142-6 kubenswrapper[2564]: I0730 00:24:06.843764 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-5677ccb49b-ww47t"] Jul 30 00:24:06.926851 ip-10-0-142-6 kubenswrapper[2564]: I0730 00:24:06.926812 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/3e278986-434b-4fd2-bfd5-2f055bc33629-local-certs\") pod \"istiod-openshift-gateway-5677ccb49b-ww47t\" (UID: \"3e278986-434b-4fd2-bfd5-2f055bc33629\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-ww47t" Jul 30 00:24:06.927032 ip-10-0-142-6 kubenswrapper[2564]: I0730 00:24:06.926871 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/3e278986-434b-4fd2-bfd5-2f055bc33629-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-5677ccb49b-ww47t\" (UID: \"3e278986-434b-4fd2-bfd5-2f055bc33629\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-ww47t" Jul 30 00:24:06.927032 ip-10-0-142-6 kubenswrapper[2564]: I0730 00:24:06.926945 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3e278986-434b-4fd2-bfd5-2f055bc33629-istio-token\") pod \"istiod-openshift-gateway-5677ccb49b-ww47t\" (UID: \"3e278986-434b-4fd2-bfd5-2f055bc33629\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-ww47t" Jul 30 00:24:06.927032 ip-10-0-142-6 kubenswrapper[2564]: I0730 00:24:06.926977 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/3e278986-434b-4fd2-bfd5-2f055bc33629-cacerts\") pod \"istiod-openshift-gateway-5677ccb49b-ww47t\" (UID: \"3e278986-434b-4fd2-bfd5-2f055bc33629\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-ww47t" Jul 30 00:24:06.927188 ip-10-0-142-6 kubenswrapper[2564]: I0730 00:24:06.927038 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/3e278986-434b-4fd2-bfd5-2f055bc33629-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-5677ccb49b-ww47t\" (UID: \"3e278986-434b-4fd2-bfd5-2f055bc33629\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-ww47t" Jul 30 00:24:06.927188 ip-10-0-142-6 kubenswrapper[2564]: I0730 00:24:06.927057 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qxtqw\" (UniqueName: \"kubernetes.io/projected/3e278986-434b-4fd2-bfd5-2f055bc33629-kube-api-access-qxtqw\") pod \"istiod-openshift-gateway-5677ccb49b-ww47t\" (UID: \"3e278986-434b-4fd2-bfd5-2f055bc33629\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-ww47t" Jul 30 00:24:06.927188 ip-10-0-142-6 kubenswrapper[2564]: I0730 00:24:06.927104 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/3e278986-434b-4fd2-bfd5-2f055bc33629-istio-kubeconfig\") pod \"istiod-openshift-gateway-5677ccb49b-ww47t\" (UID: \"3e278986-434b-4fd2-bfd5-2f055bc33629\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-ww47t" Jul 30 00:24:07.028485 ip-10-0-142-6 kubenswrapper[2564]: I0730 00:24:07.028452 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/3e278986-434b-4fd2-bfd5-2f055bc33629-local-certs\") pod \"istiod-openshift-gateway-5677ccb49b-ww47t\" (UID: \"3e278986-434b-4fd2-bfd5-2f055bc33629\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-ww47t" Jul 30 00:24:07.028674 ip-10-0-142-6 kubenswrapper[2564]: I0730 00:24:07.028499 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/3e278986-434b-4fd2-bfd5-2f055bc33629-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-5677ccb49b-ww47t\" (UID: \"3e278986-434b-4fd2-bfd5-2f055bc33629\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-ww47t" Jul 30 00:24:07.028674 ip-10-0-142-6 kubenswrapper[2564]: I0730 00:24:07.028522 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3e278986-434b-4fd2-bfd5-2f055bc33629-istio-token\") pod \"istiod-openshift-gateway-5677ccb49b-ww47t\" (UID: \"3e278986-434b-4fd2-bfd5-2f055bc33629\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-ww47t" Jul 30 00:24:07.028674 ip-10-0-142-6 kubenswrapper[2564]: I0730 00:24:07.028539 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/3e278986-434b-4fd2-bfd5-2f055bc33629-cacerts\") pod \"istiod-openshift-gateway-5677ccb49b-ww47t\" (UID: \"3e278986-434b-4fd2-bfd5-2f055bc33629\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-ww47t" Jul 30 00:24:07.028844 ip-10-0-142-6 kubenswrapper[2564]: I0730 00:24:07.028687 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/3e278986-434b-4fd2-bfd5-2f055bc33629-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-5677ccb49b-ww47t\" (UID: \"3e278986-434b-4fd2-bfd5-2f055bc33629\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-ww47t" Jul 30 00:24:07.028844 ip-10-0-142-6 kubenswrapper[2564]: I0730 00:24:07.028743 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qxtqw\" (UniqueName: \"kubernetes.io/projected/3e278986-434b-4fd2-bfd5-2f055bc33629-kube-api-access-qxtqw\") pod \"istiod-openshift-gateway-5677ccb49b-ww47t\" (UID: \"3e278986-434b-4fd2-bfd5-2f055bc33629\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-ww47t" Jul 30 00:24:07.028844 ip-10-0-142-6 kubenswrapper[2564]: I0730 00:24:07.028774 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/3e278986-434b-4fd2-bfd5-2f055bc33629-istio-kubeconfig\") pod \"istiod-openshift-gateway-5677ccb49b-ww47t\" (UID: \"3e278986-434b-4fd2-bfd5-2f055bc33629\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-ww47t" Jul 30 00:24:07.029435 ip-10-0-142-6 kubenswrapper[2564]: I0730 00:24:07.029408 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/3e278986-434b-4fd2-bfd5-2f055bc33629-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-5677ccb49b-ww47t\" (UID: \"3e278986-434b-4fd2-bfd5-2f055bc33629\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-ww47t" Jul 30 00:24:07.030875 ip-10-0-142-6 kubenswrapper[2564]: I0730 00:24:07.030853 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/3e278986-434b-4fd2-bfd5-2f055bc33629-local-certs\") pod \"istiod-openshift-gateway-5677ccb49b-ww47t\" (UID: \"3e278986-434b-4fd2-bfd5-2f055bc33629\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-ww47t" Jul 30 00:24:07.031058 ip-10-0-142-6 kubenswrapper[2564]: I0730 00:24:07.031040 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/3e278986-434b-4fd2-bfd5-2f055bc33629-cacerts\") pod \"istiod-openshift-gateway-5677ccb49b-ww47t\" (UID: \"3e278986-434b-4fd2-bfd5-2f055bc33629\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-ww47t" Jul 30 00:24:07.031172 ip-10-0-142-6 kubenswrapper[2564]: I0730 00:24:07.031152 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/3e278986-434b-4fd2-bfd5-2f055bc33629-istio-kubeconfig\") pod \"istiod-openshift-gateway-5677ccb49b-ww47t\" (UID: \"3e278986-434b-4fd2-bfd5-2f055bc33629\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-ww47t" Jul 30 00:24:07.031233 ip-10-0-142-6 kubenswrapper[2564]: I0730 00:24:07.031170 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/3e278986-434b-4fd2-bfd5-2f055bc33629-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-5677ccb49b-ww47t\" (UID: \"3e278986-434b-4fd2-bfd5-2f055bc33629\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-ww47t" Jul 30 00:24:07.036145 ip-10-0-142-6 kubenswrapper[2564]: I0730 00:24:07.036093 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3e278986-434b-4fd2-bfd5-2f055bc33629-istio-token\") pod \"istiod-openshift-gateway-5677ccb49b-ww47t\" (UID: \"3e278986-434b-4fd2-bfd5-2f055bc33629\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-ww47t" Jul 30 00:24:07.039889 ip-10-0-142-6 kubenswrapper[2564]: I0730 00:24:07.039870 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qxtqw\" (UniqueName: \"kubernetes.io/projected/3e278986-434b-4fd2-bfd5-2f055bc33629-kube-api-access-qxtqw\") pod \"istiod-openshift-gateway-5677ccb49b-ww47t\" (UID: \"3e278986-434b-4fd2-bfd5-2f055bc33629\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-ww47t" Jul 30 00:24:07.146572 ip-10-0-142-6 kubenswrapper[2564]: I0730 00:24:07.146538 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-ww47t" Jul 30 00:24:07.302261 ip-10-0-142-6 kubenswrapper[2564]: I0730 00:24:07.302235 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-5677ccb49b-ww47t"] Jul 30 00:24:07.304088 ip-10-0-142-6 kubenswrapper[2564]: W0730 00:24:07.304046 2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3e278986_434b_4fd2_bfd5_2f055bc33629.slice/crio-18e01d7d3f4e27c4a0af3c84108aa1b53607114b9c6ce8612fe8368f51bb51aa WatchSource:0}: Error finding container 18e01d7d3f4e27c4a0af3c84108aa1b53607114b9c6ce8612fe8368f51bb51aa: Status 404 returned error can't find the container with id 18e01d7d3f4e27c4a0af3c84108aa1b53607114b9c6ce8612fe8368f51bb51aa Jul 30 00:24:07.717042 ip-10-0-142-6 kubenswrapper[2564]: I0730 00:24:07.716998 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-ww47t" event={"ID":"3e278986-434b-4fd2-bfd5-2f055bc33629","Type":"ContainerStarted","Data":"18e01d7d3f4e27c4a0af3c84108aa1b53607114b9c6ce8612fe8368f51bb51aa"} Jul 30 00:24:09.484591 ip-10-0-142-6 kubenswrapper[2564]: I0730 00:24:09.484556 2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/kserve-controller-manager-d6f4b65b8-bwrb2"] Jul 30 00:24:09.492362 ip-10-0-142-6 kubenswrapper[2564]: I0730 00:24:09.492338 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/kserve-controller-manager-d6f4b65b8-bwrb2" Jul 30 00:24:09.495053 ip-10-0-142-6 kubenswrapper[2564]: I0730 00:24:09.495028 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"kserve-webhook-server-cert\"" Jul 30 00:24:09.496150 ip-10-0-142-6 kubenswrapper[2564]: I0730 00:24:09.496129 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"kserve-controller-manager-dockercfg-ms4zs\"" Jul 30 00:24:09.501413 ip-10-0-142-6 kubenswrapper[2564]: I0730 00:24:09.501158 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/kserve-controller-manager-d6f4b65b8-bwrb2"] Jul 30 00:24:09.650885 ip-10-0-142-6 kubenswrapper[2564]: I0730 00:24:09.650853 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gh9tt\" (UniqueName: \"kubernetes.io/projected/c998e5e8-aa41-4da6-a8a8-94b49e8f99fb-kube-api-access-gh9tt\") pod \"kserve-controller-manager-d6f4b65b8-bwrb2\" (UID: \"c998e5e8-aa41-4da6-a8a8-94b49e8f99fb\") " pod="opendatahub/kserve-controller-manager-d6f4b65b8-bwrb2" Jul 30 00:24:09.651063 ip-10-0-142-6 kubenswrapper[2564]: I0730 00:24:09.650919 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/c998e5e8-aa41-4da6-a8a8-94b49e8f99fb-cert\") pod \"kserve-controller-manager-d6f4b65b8-bwrb2\" (UID: \"c998e5e8-aa41-4da6-a8a8-94b49e8f99fb\") " pod="opendatahub/kserve-controller-manager-d6f4b65b8-bwrb2" Jul 30 00:24:09.751556 ip-10-0-142-6 kubenswrapper[2564]: I0730 00:24:09.751425 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/c998e5e8-aa41-4da6-a8a8-94b49e8f99fb-cert\") pod \"kserve-controller-manager-d6f4b65b8-bwrb2\" (UID: \"c998e5e8-aa41-4da6-a8a8-94b49e8f99fb\") " pod="opendatahub/kserve-controller-manager-d6f4b65b8-bwrb2" Jul 30 00:24:09.751556 ip-10-0-142-6 kubenswrapper[2564]: I0730 00:24:09.751506 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gh9tt\" (UniqueName: \"kubernetes.io/projected/c998e5e8-aa41-4da6-a8a8-94b49e8f99fb-kube-api-access-gh9tt\") pod \"kserve-controller-manager-d6f4b65b8-bwrb2\" (UID: \"c998e5e8-aa41-4da6-a8a8-94b49e8f99fb\") " pod="opendatahub/kserve-controller-manager-d6f4b65b8-bwrb2" Jul 30 00:24:09.754290 ip-10-0-142-6 kubenswrapper[2564]: I0730 00:24:09.754264 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/c998e5e8-aa41-4da6-a8a8-94b49e8f99fb-cert\") pod \"kserve-controller-manager-d6f4b65b8-bwrb2\" (UID: \"c998e5e8-aa41-4da6-a8a8-94b49e8f99fb\") " pod="opendatahub/kserve-controller-manager-d6f4b65b8-bwrb2" Jul 30 00:24:09.777489 ip-10-0-142-6 kubenswrapper[2564]: I0730 00:24:09.777460 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-gh9tt\" (UniqueName: \"kubernetes.io/projected/c998e5e8-aa41-4da6-a8a8-94b49e8f99fb-kube-api-access-gh9tt\") pod \"kserve-controller-manager-d6f4b65b8-bwrb2\" (UID: \"c998e5e8-aa41-4da6-a8a8-94b49e8f99fb\") " pod="opendatahub/kserve-controller-manager-d6f4b65b8-bwrb2" Jul 30 00:24:09.807142 ip-10-0-142-6 kubenswrapper[2564]: I0730 00:24:09.807111 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/kserve-controller-manager-d6f4b65b8-bwrb2" Jul 30 00:24:10.674578 ip-10-0-142-6 kubenswrapper[2564]: I0730 00:24:10.674522 2564 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236220Ki","pods":"250"} Jul 30 00:24:10.674952 ip-10-0-142-6 kubenswrapper[2564]: I0730 00:24:10.674612 2564 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236220Ki","pods":"250"} Jul 30 00:24:10.760857 ip-10-0-142-6 kubenswrapper[2564]: I0730 00:24:10.760827 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/kserve-controller-manager-d6f4b65b8-bwrb2"] Jul 30 00:24:10.763137 ip-10-0-142-6 kubenswrapper[2564]: W0730 00:24:10.763106 2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc998e5e8_aa41_4da6_a8a8_94b49e8f99fb.slice/crio-6a9e7e14c4df1453681538ab694ed28ef7f70008676824fa72edbbf4ef6d6bd0 WatchSource:0}: Error finding container 6a9e7e14c4df1453681538ab694ed28ef7f70008676824fa72edbbf4ef6d6bd0: Status 404 returned error can't find the container with id 6a9e7e14c4df1453681538ab694ed28ef7f70008676824fa72edbbf4ef6d6bd0 Jul 30 00:24:11.740094 ip-10-0-142-6 kubenswrapper[2564]: I0730 00:24:11.740045 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-ww47t" event={"ID":"3e278986-434b-4fd2-bfd5-2f055bc33629","Type":"ContainerStarted","Data":"529e50579a242ce94565f4b0e3dcc7bef7209106e7842555eefd8e647c967337"} Jul 30 00:24:11.740564 ip-10-0-142-6 kubenswrapper[2564]: I0730 00:24:11.740120 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-ww47t" Jul 30 00:24:11.741451 ip-10-0-142-6 kubenswrapper[2564]: I0730 00:24:11.741421 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/kserve-controller-manager-d6f4b65b8-bwrb2" event={"ID":"c998e5e8-aa41-4da6-a8a8-94b49e8f99fb","Type":"ContainerStarted","Data":"6a9e7e14c4df1453681538ab694ed28ef7f70008676824fa72edbbf4ef6d6bd0"} Jul 30 00:24:11.741950 ip-10-0-142-6 kubenswrapper[2564]: I0730 00:24:11.741924 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-ww47t" Jul 30 00:24:11.761792 ip-10-0-142-6 kubenswrapper[2564]: I0730 00:24:11.761726 2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-ww47t" podStartSLOduration=2.393423125 podStartE2EDuration="5.761705745s" totalImagesPullingTime="3.36828262s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-30 00:24:06 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-30 00:24:11.760139362 +0000 UTC m=+383.174879295" watchObservedRunningTime="2026-07-30 00:24:11.761705745 +0000 UTC m=+383.176445679" Jul 30 00:24:13.751767 ip-10-0-142-6 kubenswrapper[2564]: I0730 00:24:13.751726 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/kserve-controller-manager-d6f4b65b8-bwrb2" event={"ID":"c998e5e8-aa41-4da6-a8a8-94b49e8f99fb","Type":"ContainerStarted","Data":"afeeab55a5e05e98bff18bf9bc8328713135b9b9a5e49a4af15d28d04380242e"} Jul 30 00:24:13.752329 ip-10-0-142-6 kubenswrapper[2564]: I0730 00:24:13.751791 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/kserve-controller-manager-d6f4b65b8-bwrb2" Jul 30 00:24:13.768335 ip-10-0-142-6 kubenswrapper[2564]: I0730 00:24:13.768290 2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/kserve-controller-manager-d6f4b65b8-bwrb2" podStartSLOduration=2.438903405 podStartE2EDuration="4.768275137s" totalImagesPullingTime="2.329371732s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-30 00:24:09 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-30 00:24:13.767422105 +0000 UTC m=+385.182162049" watchObservedRunningTime="2026-07-30 00:24:13.768275137 +0000 UTC m=+385.183015069" Jul 30 00:24:44.761260 ip-10-0-142-6 kubenswrapper[2564]: I0730 00:24:44.761221 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/kserve-controller-manager-d6f4b65b8-bwrb2" Jul 30 00:25:15.467811 ip-10-0-142-6 kubenswrapper[2564]: I0730 00:25:15.467729 2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73k9924"] Jul 30 00:25:15.470112 ip-10-0-142-6 kubenswrapper[2564]: I0730 00:25:15.470096 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73k9924" Jul 30 00:25:15.472720 ip-10-0-142-6 kubenswrapper[2564]: I0730 00:25:15.472697 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"openshift-service-ca.crt\"" Jul 30 00:25:15.472846 ip-10-0-142-6 kubenswrapper[2564]: I0730 00:25:15.472723 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kube-root-ca.crt\"" Jul 30 00:25:15.472846 ip-10-0-142-6 kubenswrapper[2564]: I0730 00:25:15.472736 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"default-dockercfg-4v4jb\"" Jul 30 00:25:15.479124 ip-10-0-142-6 kubenswrapper[2564]: I0730 00:25:15.479101 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73k9924"] Jul 30 00:25:15.592405 ip-10-0-142-6 kubenswrapper[2564]: I0730 00:25:15.592378 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/f3647c9a-1ae5-4ec6-8d69-c8db736e6fad-bundle\") pod \"5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73k9924\" (UID: \"f3647c9a-1ae5-4ec6-8d69-c8db736e6fad\") " pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73k9924" Jul 30 00:25:15.592405 ip-10-0-142-6 kubenswrapper[2564]: I0730 00:25:15.592409 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/f3647c9a-1ae5-4ec6-8d69-c8db736e6fad-util\") pod \"5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73k9924\" (UID: \"f3647c9a-1ae5-4ec6-8d69-c8db736e6fad\") " pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73k9924" Jul 30 00:25:15.592689 ip-10-0-142-6 kubenswrapper[2564]: I0730 00:25:15.592456 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cvqsm\" (UniqueName: \"kubernetes.io/projected/f3647c9a-1ae5-4ec6-8d69-c8db736e6fad-kube-api-access-cvqsm\") pod \"5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73k9924\" (UID: \"f3647c9a-1ae5-4ec6-8d69-c8db736e6fad\") " pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73k9924" Jul 30 00:25:15.693731 ip-10-0-142-6 kubenswrapper[2564]: I0730 00:25:15.693693 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cvqsm\" (UniqueName: \"kubernetes.io/projected/f3647c9a-1ae5-4ec6-8d69-c8db736e6fad-kube-api-access-cvqsm\") pod \"5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73k9924\" (UID: \"f3647c9a-1ae5-4ec6-8d69-c8db736e6fad\") " pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73k9924" Jul 30 00:25:15.693935 ip-10-0-142-6 kubenswrapper[2564]: I0730 00:25:15.693770 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/f3647c9a-1ae5-4ec6-8d69-c8db736e6fad-bundle\") pod \"5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73k9924\" (UID: \"f3647c9a-1ae5-4ec6-8d69-c8db736e6fad\") " pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73k9924" Jul 30 00:25:15.693935 ip-10-0-142-6 kubenswrapper[2564]: I0730 00:25:15.693798 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/f3647c9a-1ae5-4ec6-8d69-c8db736e6fad-util\") pod \"5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73k9924\" (UID: \"f3647c9a-1ae5-4ec6-8d69-c8db736e6fad\") " pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73k9924" Jul 30 00:25:15.694179 ip-10-0-142-6 kubenswrapper[2564]: I0730 00:25:15.694158 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/f3647c9a-1ae5-4ec6-8d69-c8db736e6fad-bundle\") pod \"5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73k9924\" (UID: \"f3647c9a-1ae5-4ec6-8d69-c8db736e6fad\") " pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73k9924" Jul 30 00:25:15.694262 ip-10-0-142-6 kubenswrapper[2564]: I0730 00:25:15.694217 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/f3647c9a-1ae5-4ec6-8d69-c8db736e6fad-util\") pod \"5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73k9924\" (UID: \"f3647c9a-1ae5-4ec6-8d69-c8db736e6fad\") " pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73k9924" Jul 30 00:25:15.702482 ip-10-0-142-6 kubenswrapper[2564]: I0730 00:25:15.702457 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-cvqsm\" (UniqueName: \"kubernetes.io/projected/f3647c9a-1ae5-4ec6-8d69-c8db736e6fad-kube-api-access-cvqsm\") pod \"5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73k9924\" (UID: \"f3647c9a-1ae5-4ec6-8d69-c8db736e6fad\") " pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73k9924" Jul 30 00:25:15.780266 ip-10-0-142-6 kubenswrapper[2564]: I0730 00:25:15.780188 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73k9924" Jul 30 00:25:15.905811 ip-10-0-142-6 kubenswrapper[2564]: I0730 00:25:15.905781 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73k9924"] Jul 30 00:25:15.909655 ip-10-0-142-6 kubenswrapper[2564]: W0730 00:25:15.909629 2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf3647c9a_1ae5_4ec6_8d69_c8db736e6fad.slice/crio-2436c4af27c8f663ffa0c709808c9d1f27abfc3de559896165eb4bc95b8e9744 WatchSource:0}: Error finding container 2436c4af27c8f663ffa0c709808c9d1f27abfc3de559896165eb4bc95b8e9744: Status 404 returned error can't find the container with id 2436c4af27c8f663ffa0c709808c9d1f27abfc3de559896165eb4bc95b8e9744 Jul 30 00:25:16.026842 ip-10-0-142-6 kubenswrapper[2564]: I0730 00:25:16.026815 2564 generic.go:356] "Generic (PLEG): container finished" podID="f3647c9a-1ae5-4ec6-8d69-c8db736e6fad" containerID="31bcade9cb586ffe69fae75d40020f387fed6c0ee153b29d84119b3481a5a154" exitCode=0 Jul 30 00:25:16.026975 ip-10-0-142-6 kubenswrapper[2564]: I0730 00:25:16.026902 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73k9924" event={"ID":"f3647c9a-1ae5-4ec6-8d69-c8db736e6fad","Type":"ContainerDied","Data":"31bcade9cb586ffe69fae75d40020f387fed6c0ee153b29d84119b3481a5a154"} Jul 30 00:25:16.026975 ip-10-0-142-6 kubenswrapper[2564]: I0730 00:25:16.026948 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73k9924" event={"ID":"f3647c9a-1ae5-4ec6-8d69-c8db736e6fad","Type":"ContainerStarted","Data":"2436c4af27c8f663ffa0c709808c9d1f27abfc3de559896165eb4bc95b8e9744"} Jul 30 00:25:16.266019 ip-10-0-142-6 kubenswrapper[2564]: I0730 00:25:16.265982 2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef187fz8"] Jul 30 00:25:16.269193 ip-10-0-142-6 kubenswrapper[2564]: I0730 00:25:16.269171 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef187fz8" Jul 30 00:25:16.277027 ip-10-0-142-6 kubenswrapper[2564]: I0730 00:25:16.277003 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef187fz8"] Jul 30 00:25:16.401323 ip-10-0-142-6 kubenswrapper[2564]: I0730 00:25:16.401294 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/fbd780fd-defb-402c-98ff-2cca1c03d7b0-util\") pod \"9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef187fz8\" (UID: \"fbd780fd-defb-402c-98ff-2cca1c03d7b0\") " pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef187fz8" Jul 30 00:25:16.401490 ip-10-0-142-6 kubenswrapper[2564]: I0730 00:25:16.401337 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/fbd780fd-defb-402c-98ff-2cca1c03d7b0-bundle\") pod \"9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef187fz8\" (UID: \"fbd780fd-defb-402c-98ff-2cca1c03d7b0\") " pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef187fz8" Jul 30 00:25:16.401490 ip-10-0-142-6 kubenswrapper[2564]: I0730 00:25:16.401360 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8rq2q\" (UniqueName: \"kubernetes.io/projected/fbd780fd-defb-402c-98ff-2cca1c03d7b0-kube-api-access-8rq2q\") pod \"9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef187fz8\" (UID: \"fbd780fd-defb-402c-98ff-2cca1c03d7b0\") " pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef187fz8" Jul 30 00:25:16.501964 ip-10-0-142-6 kubenswrapper[2564]: I0730 00:25:16.501928 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/fbd780fd-defb-402c-98ff-2cca1c03d7b0-util\") pod \"9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef187fz8\" (UID: \"fbd780fd-defb-402c-98ff-2cca1c03d7b0\") " pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef187fz8" Jul 30 00:25:16.502445 ip-10-0-142-6 kubenswrapper[2564]: I0730 00:25:16.501976 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/fbd780fd-defb-402c-98ff-2cca1c03d7b0-bundle\") pod \"9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef187fz8\" (UID: \"fbd780fd-defb-402c-98ff-2cca1c03d7b0\") " pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef187fz8" Jul 30 00:25:16.502445 ip-10-0-142-6 kubenswrapper[2564]: I0730 00:25:16.502000 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8rq2q\" (UniqueName: \"kubernetes.io/projected/fbd780fd-defb-402c-98ff-2cca1c03d7b0-kube-api-access-8rq2q\") pod \"9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef187fz8\" (UID: \"fbd780fd-defb-402c-98ff-2cca1c03d7b0\") " pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef187fz8" Jul 30 00:25:16.502445 ip-10-0-142-6 kubenswrapper[2564]: I0730 00:25:16.502350 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/fbd780fd-defb-402c-98ff-2cca1c03d7b0-bundle\") pod \"9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef187fz8\" (UID: \"fbd780fd-defb-402c-98ff-2cca1c03d7b0\") " pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef187fz8" Jul 30 00:25:16.502569 ip-10-0-142-6 kubenswrapper[2564]: I0730 00:25:16.502458 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/fbd780fd-defb-402c-98ff-2cca1c03d7b0-util\") pod \"9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef187fz8\" (UID: \"fbd780fd-defb-402c-98ff-2cca1c03d7b0\") " pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef187fz8" Jul 30 00:25:16.509624 ip-10-0-142-6 kubenswrapper[2564]: I0730 00:25:16.509605 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8rq2q\" (UniqueName: \"kubernetes.io/projected/fbd780fd-defb-402c-98ff-2cca1c03d7b0-kube-api-access-8rq2q\") pod \"9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef187fz8\" (UID: \"fbd780fd-defb-402c-98ff-2cca1c03d7b0\") " pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef187fz8" Jul 30 00:25:16.578995 ip-10-0-142-6 kubenswrapper[2564]: I0730 00:25:16.578925 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef187fz8" Jul 30 00:25:16.671016 ip-10-0-142-6 kubenswrapper[2564]: I0730 00:25:16.670981 2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759t2h7b"] Jul 30 00:25:16.674426 ip-10-0-142-6 kubenswrapper[2564]: I0730 00:25:16.674401 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759t2h7b" Jul 30 00:25:16.682051 ip-10-0-142-6 kubenswrapper[2564]: I0730 00:25:16.682022 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759t2h7b"] Jul 30 00:25:16.712115 ip-10-0-142-6 kubenswrapper[2564]: I0730 00:25:16.708183 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef187fz8"] Jul 30 00:25:16.805409 ip-10-0-142-6 kubenswrapper[2564]: I0730 00:25:16.805376 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hhnx4\" (UniqueName: \"kubernetes.io/projected/f546aac1-87f6-499c-98c8-02afb9ae43f6-kube-api-access-hhnx4\") pod \"0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759t2h7b\" (UID: \"f546aac1-87f6-499c-98c8-02afb9ae43f6\") " pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759t2h7b" Jul 30 00:25:16.805576 ip-10-0-142-6 kubenswrapper[2564]: I0730 00:25:16.805455 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/f546aac1-87f6-499c-98c8-02afb9ae43f6-bundle\") pod \"0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759t2h7b\" (UID: \"f546aac1-87f6-499c-98c8-02afb9ae43f6\") " pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759t2h7b" Jul 30 00:25:16.805576 ip-10-0-142-6 kubenswrapper[2564]: I0730 00:25:16.805490 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/f546aac1-87f6-499c-98c8-02afb9ae43f6-util\") pod \"0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759t2h7b\" (UID: \"f546aac1-87f6-499c-98c8-02afb9ae43f6\") " pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759t2h7b" Jul 30 00:25:16.906764 ip-10-0-142-6 kubenswrapper[2564]: I0730 00:25:16.906737 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/f546aac1-87f6-499c-98c8-02afb9ae43f6-bundle\") pod \"0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759t2h7b\" (UID: \"f546aac1-87f6-499c-98c8-02afb9ae43f6\") " pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759t2h7b" Jul 30 00:25:16.906898 ip-10-0-142-6 kubenswrapper[2564]: I0730 00:25:16.906774 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/f546aac1-87f6-499c-98c8-02afb9ae43f6-util\") pod \"0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759t2h7b\" (UID: \"f546aac1-87f6-499c-98c8-02afb9ae43f6\") " pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759t2h7b" Jul 30 00:25:16.906898 ip-10-0-142-6 kubenswrapper[2564]: I0730 00:25:16.906824 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hhnx4\" (UniqueName: \"kubernetes.io/projected/f546aac1-87f6-499c-98c8-02afb9ae43f6-kube-api-access-hhnx4\") pod \"0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759t2h7b\" (UID: \"f546aac1-87f6-499c-98c8-02afb9ae43f6\") " pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759t2h7b" Jul 30 00:25:16.907124 ip-10-0-142-6 kubenswrapper[2564]: I0730 00:25:16.907107 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/f546aac1-87f6-499c-98c8-02afb9ae43f6-bundle\") pod \"0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759t2h7b\" (UID: \"f546aac1-87f6-499c-98c8-02afb9ae43f6\") " pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759t2h7b" Jul 30 00:25:16.907202 ip-10-0-142-6 kubenswrapper[2564]: I0730 00:25:16.907182 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/f546aac1-87f6-499c-98c8-02afb9ae43f6-util\") pod \"0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759t2h7b\" (UID: \"f546aac1-87f6-499c-98c8-02afb9ae43f6\") " pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759t2h7b" Jul 30 00:25:16.914456 ip-10-0-142-6 kubenswrapper[2564]: I0730 00:25:16.914307 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-hhnx4\" (UniqueName: \"kubernetes.io/projected/f546aac1-87f6-499c-98c8-02afb9ae43f6-kube-api-access-hhnx4\") pod \"0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759t2h7b\" (UID: \"f546aac1-87f6-499c-98c8-02afb9ae43f6\") " pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759t2h7b" Jul 30 00:25:16.986235 ip-10-0-142-6 kubenswrapper[2564]: I0730 00:25:16.986207 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759t2h7b" Jul 30 00:25:17.034286 ip-10-0-142-6 kubenswrapper[2564]: I0730 00:25:17.034024 2564 generic.go:356] "Generic (PLEG): container finished" podID="f3647c9a-1ae5-4ec6-8d69-c8db736e6fad" containerID="690dd40ed7c1698cf56f977540a75d5be639fdb7667486d7c526e47ad55e86ed" exitCode=0 Jul 30 00:25:17.034286 ip-10-0-142-6 kubenswrapper[2564]: I0730 00:25:17.034154 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73k9924" event={"ID":"f3647c9a-1ae5-4ec6-8d69-c8db736e6fad","Type":"ContainerDied","Data":"690dd40ed7c1698cf56f977540a75d5be639fdb7667486d7c526e47ad55e86ed"} Jul 30 00:25:17.036639 ip-10-0-142-6 kubenswrapper[2564]: I0730 00:25:17.036584 2564 generic.go:356] "Generic (PLEG): container finished" podID="fbd780fd-defb-402c-98ff-2cca1c03d7b0" containerID="7f065c3b8c4649845e3a1d0f58b4d76ea0114f7c2272348ba03f2135b2137227" exitCode=0 Jul 30 00:25:17.036757 ip-10-0-142-6 kubenswrapper[2564]: I0730 00:25:17.036675 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef187fz8" event={"ID":"fbd780fd-defb-402c-98ff-2cca1c03d7b0","Type":"ContainerDied","Data":"7f065c3b8c4649845e3a1d0f58b4d76ea0114f7c2272348ba03f2135b2137227"} Jul 30 00:25:17.036757 ip-10-0-142-6 kubenswrapper[2564]: I0730 00:25:17.036719 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef187fz8" event={"ID":"fbd780fd-defb-402c-98ff-2cca1c03d7b0","Type":"ContainerStarted","Data":"55c93145e98070660b1e9de7d32de42c5dfa9302ab8b1dd6b9684968ea3a3a77"} Jul 30 00:25:17.078560 ip-10-0-142-6 kubenswrapper[2564]: I0730 00:25:17.078512 2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0w74kw"] Jul 30 00:25:17.083210 ip-10-0-142-6 kubenswrapper[2564]: I0730 00:25:17.083191 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0w74kw" Jul 30 00:25:17.090948 ip-10-0-142-6 kubenswrapper[2564]: I0730 00:25:17.090897 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0w74kw"] Jul 30 00:25:17.123100 ip-10-0-142-6 kubenswrapper[2564]: I0730 00:25:17.123059 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759t2h7b"] Jul 30 00:25:17.124601 ip-10-0-142-6 kubenswrapper[2564]: W0730 00:25:17.124579 2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf546aac1_87f6_499c_98c8_02afb9ae43f6.slice/crio-5b64dd5c87de2a74f4e5a7af6d155248af780d6a8e589a604bc3b35533839183 WatchSource:0}: Error finding container 5b64dd5c87de2a74f4e5a7af6d155248af780d6a8e589a604bc3b35533839183: Status 404 returned error can't find the container with id 5b64dd5c87de2a74f4e5a7af6d155248af780d6a8e589a604bc3b35533839183 Jul 30 00:25:17.208940 ip-10-0-142-6 kubenswrapper[2564]: I0730 00:25:17.208914 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/7069b0f9-9545-48ce-885e-1ec788c65ea5-bundle\") pod \"19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0w74kw\" (UID: \"7069b0f9-9545-48ce-885e-1ec788c65ea5\") " pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0w74kw" Jul 30 00:25:17.209120 ip-10-0-142-6 kubenswrapper[2564]: I0730 00:25:17.209089 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/7069b0f9-9545-48ce-885e-1ec788c65ea5-util\") pod \"19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0w74kw\" (UID: \"7069b0f9-9545-48ce-885e-1ec788c65ea5\") " pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0w74kw" Jul 30 00:25:17.209256 ip-10-0-142-6 kubenswrapper[2564]: I0730 00:25:17.209149 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tvrws\" (UniqueName: \"kubernetes.io/projected/7069b0f9-9545-48ce-885e-1ec788c65ea5-kube-api-access-tvrws\") pod \"19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0w74kw\" (UID: \"7069b0f9-9545-48ce-885e-1ec788c65ea5\") " pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0w74kw" Jul 30 00:25:17.310314 ip-10-0-142-6 kubenswrapper[2564]: I0730 00:25:17.310267 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/7069b0f9-9545-48ce-885e-1ec788c65ea5-util\") pod \"19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0w74kw\" (UID: \"7069b0f9-9545-48ce-885e-1ec788c65ea5\") " pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0w74kw" Jul 30 00:25:17.310314 ip-10-0-142-6 kubenswrapper[2564]: I0730 00:25:17.310317 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tvrws\" (UniqueName: \"kubernetes.io/projected/7069b0f9-9545-48ce-885e-1ec788c65ea5-kube-api-access-tvrws\") pod \"19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0w74kw\" (UID: \"7069b0f9-9545-48ce-885e-1ec788c65ea5\") " pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0w74kw" Jul 30 00:25:17.310558 ip-10-0-142-6 kubenswrapper[2564]: I0730 00:25:17.310383 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/7069b0f9-9545-48ce-885e-1ec788c65ea5-bundle\") pod \"19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0w74kw\" (UID: \"7069b0f9-9545-48ce-885e-1ec788c65ea5\") " pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0w74kw" Jul 30 00:25:17.310679 ip-10-0-142-6 kubenswrapper[2564]: I0730 00:25:17.310659 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/7069b0f9-9545-48ce-885e-1ec788c65ea5-util\") pod \"19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0w74kw\" (UID: \"7069b0f9-9545-48ce-885e-1ec788c65ea5\") " pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0w74kw" Jul 30 00:25:17.310679 ip-10-0-142-6 kubenswrapper[2564]: I0730 00:25:17.310674 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/7069b0f9-9545-48ce-885e-1ec788c65ea5-bundle\") pod \"19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0w74kw\" (UID: \"7069b0f9-9545-48ce-885e-1ec788c65ea5\") " pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0w74kw" Jul 30 00:25:17.318301 ip-10-0-142-6 kubenswrapper[2564]: I0730 00:25:17.318275 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-tvrws\" (UniqueName: \"kubernetes.io/projected/7069b0f9-9545-48ce-885e-1ec788c65ea5-kube-api-access-tvrws\") pod \"19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0w74kw\" (UID: \"7069b0f9-9545-48ce-885e-1ec788c65ea5\") " pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0w74kw" Jul 30 00:25:17.400957 ip-10-0-142-6 kubenswrapper[2564]: I0730 00:25:17.400926 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0w74kw" Jul 30 00:25:17.543919 ip-10-0-142-6 kubenswrapper[2564]: I0730 00:25:17.543894 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0w74kw"] Jul 30 00:25:17.545479 ip-10-0-142-6 kubenswrapper[2564]: W0730 00:25:17.545447 2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7069b0f9_9545_48ce_885e_1ec788c65ea5.slice/crio-cd3d31ef9cafe994921c1f14d6d236ebd75cf045ba5e0de3604cdfd5bc3103b7 WatchSource:0}: Error finding container cd3d31ef9cafe994921c1f14d6d236ebd75cf045ba5e0de3604cdfd5bc3103b7: Status 404 returned error can't find the container with id cd3d31ef9cafe994921c1f14d6d236ebd75cf045ba5e0de3604cdfd5bc3103b7 Jul 30 00:25:18.043816 ip-10-0-142-6 kubenswrapper[2564]: I0730 00:25:18.043735 2564 generic.go:356] "Generic (PLEG): container finished" podID="f546aac1-87f6-499c-98c8-02afb9ae43f6" containerID="e5d53d7e23ea1ab0f3fb901b8ebe135ea2bd7bcfff9ee79fcf486698412e3b25" exitCode=0 Jul 30 00:25:18.043971 ip-10-0-142-6 kubenswrapper[2564]: I0730 00:25:18.043827 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759t2h7b" event={"ID":"f546aac1-87f6-499c-98c8-02afb9ae43f6","Type":"ContainerDied","Data":"e5d53d7e23ea1ab0f3fb901b8ebe135ea2bd7bcfff9ee79fcf486698412e3b25"} Jul 30 00:25:18.043971 ip-10-0-142-6 kubenswrapper[2564]: I0730 00:25:18.043885 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759t2h7b" event={"ID":"f546aac1-87f6-499c-98c8-02afb9ae43f6","Type":"ContainerStarted","Data":"5b64dd5c87de2a74f4e5a7af6d155248af780d6a8e589a604bc3b35533839183"} Jul 30 00:25:18.045379 ip-10-0-142-6 kubenswrapper[2564]: I0730 00:25:18.045356 2564 generic.go:356] "Generic (PLEG): container finished" podID="7069b0f9-9545-48ce-885e-1ec788c65ea5" containerID="dbe52c7cbc5a5aa8f377de920f6d1103485a50385d23786e64bdd061a3f93c25" exitCode=0 Jul 30 00:25:18.045464 ip-10-0-142-6 kubenswrapper[2564]: I0730 00:25:18.045435 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0w74kw" event={"ID":"7069b0f9-9545-48ce-885e-1ec788c65ea5","Type":"ContainerDied","Data":"dbe52c7cbc5a5aa8f377de920f6d1103485a50385d23786e64bdd061a3f93c25"} Jul 30 00:25:18.045516 ip-10-0-142-6 kubenswrapper[2564]: I0730 00:25:18.045481 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0w74kw" event={"ID":"7069b0f9-9545-48ce-885e-1ec788c65ea5","Type":"ContainerStarted","Data":"cd3d31ef9cafe994921c1f14d6d236ebd75cf045ba5e0de3604cdfd5bc3103b7"} Jul 30 00:25:18.047737 ip-10-0-142-6 kubenswrapper[2564]: I0730 00:25:18.047713 2564 generic.go:356] "Generic (PLEG): container finished" podID="f3647c9a-1ae5-4ec6-8d69-c8db736e6fad" containerID="257d68074f6fe79f31f61b72af6f59ec9590083cdc3ec3b20a4b2b7cbdddee5b" exitCode=0 Jul 30 00:25:18.047822 ip-10-0-142-6 kubenswrapper[2564]: I0730 00:25:18.047775 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73k9924" event={"ID":"f3647c9a-1ae5-4ec6-8d69-c8db736e6fad","Type":"ContainerDied","Data":"257d68074f6fe79f31f61b72af6f59ec9590083cdc3ec3b20a4b2b7cbdddee5b"} Jul 30 00:25:18.049576 ip-10-0-142-6 kubenswrapper[2564]: I0730 00:25:18.049557 2564 generic.go:356] "Generic (PLEG): container finished" podID="fbd780fd-defb-402c-98ff-2cca1c03d7b0" containerID="04039b5765b9bade7088b9b0b71363c0b7bbebb3f669ad8db06828846ad34900" exitCode=0 Jul 30 00:25:18.049656 ip-10-0-142-6 kubenswrapper[2564]: I0730 00:25:18.049588 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef187fz8" event={"ID":"fbd780fd-defb-402c-98ff-2cca1c03d7b0","Type":"ContainerDied","Data":"04039b5765b9bade7088b9b0b71363c0b7bbebb3f669ad8db06828846ad34900"} Jul 30 00:25:19.058793 ip-10-0-142-6 kubenswrapper[2564]: I0730 00:25:19.058759 2564 generic.go:356] "Generic (PLEG): container finished" podID="fbd780fd-defb-402c-98ff-2cca1c03d7b0" containerID="b4df52ddbc6addc4dacf23a7ad536f04d86cc4c075032c86a277779ff305913b" exitCode=0 Jul 30 00:25:19.059160 ip-10-0-142-6 kubenswrapper[2564]: I0730 00:25:19.058838 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef187fz8" event={"ID":"fbd780fd-defb-402c-98ff-2cca1c03d7b0","Type":"ContainerDied","Data":"b4df52ddbc6addc4dacf23a7ad536f04d86cc4c075032c86a277779ff305913b"} Jul 30 00:25:19.251659 ip-10-0-142-6 kubenswrapper[2564]: I0730 00:25:19.251632 2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73k9924" Jul 30 00:25:19.326993 ip-10-0-142-6 kubenswrapper[2564]: I0730 00:25:19.326917 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-cvqsm\" (UniqueName: \"kubernetes.io/projected/f3647c9a-1ae5-4ec6-8d69-c8db736e6fad-kube-api-access-cvqsm\") pod \"f3647c9a-1ae5-4ec6-8d69-c8db736e6fad\" (UID: \"f3647c9a-1ae5-4ec6-8d69-c8db736e6fad\") " Jul 30 00:25:19.326993 ip-10-0-142-6 kubenswrapper[2564]: I0730 00:25:19.326987 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/f3647c9a-1ae5-4ec6-8d69-c8db736e6fad-util\") pod \"f3647c9a-1ae5-4ec6-8d69-c8db736e6fad\" (UID: \"f3647c9a-1ae5-4ec6-8d69-c8db736e6fad\") " Jul 30 00:25:19.327247 ip-10-0-142-6 kubenswrapper[2564]: I0730 00:25:19.327014 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/f3647c9a-1ae5-4ec6-8d69-c8db736e6fad-bundle\") pod \"f3647c9a-1ae5-4ec6-8d69-c8db736e6fad\" (UID: \"f3647c9a-1ae5-4ec6-8d69-c8db736e6fad\") " Jul 30 00:25:19.327641 ip-10-0-142-6 kubenswrapper[2564]: I0730 00:25:19.327618 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f3647c9a-1ae5-4ec6-8d69-c8db736e6fad-bundle" (OuterVolumeSpecName: "bundle") pod "f3647c9a-1ae5-4ec6-8d69-c8db736e6fad" (UID: "f3647c9a-1ae5-4ec6-8d69-c8db736e6fad"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 30 00:25:19.329096 ip-10-0-142-6 kubenswrapper[2564]: I0730 00:25:19.329063 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f3647c9a-1ae5-4ec6-8d69-c8db736e6fad-kube-api-access-cvqsm" (OuterVolumeSpecName: "kube-api-access-cvqsm") pod "f3647c9a-1ae5-4ec6-8d69-c8db736e6fad" (UID: "f3647c9a-1ae5-4ec6-8d69-c8db736e6fad"). InnerVolumeSpecName "kube-api-access-cvqsm". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 30 00:25:19.334469 ip-10-0-142-6 kubenswrapper[2564]: I0730 00:25:19.334446 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f3647c9a-1ae5-4ec6-8d69-c8db736e6fad-util" (OuterVolumeSpecName: "util") pod "f3647c9a-1ae5-4ec6-8d69-c8db736e6fad" (UID: "f3647c9a-1ae5-4ec6-8d69-c8db736e6fad"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 30 00:25:19.427861 ip-10-0-142-6 kubenswrapper[2564]: I0730 00:25:19.427826 2564 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/f3647c9a-1ae5-4ec6-8d69-c8db736e6fad-util\") on node \"ip-10-0-142-6.ec2.internal\" DevicePath \"\"" Jul 30 00:25:19.427861 ip-10-0-142-6 kubenswrapper[2564]: I0730 00:25:19.427852 2564 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/f3647c9a-1ae5-4ec6-8d69-c8db736e6fad-bundle\") on node \"ip-10-0-142-6.ec2.internal\" DevicePath \"\"" Jul 30 00:25:19.427861 ip-10-0-142-6 kubenswrapper[2564]: I0730 00:25:19.427862 2564 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cvqsm\" (UniqueName: \"kubernetes.io/projected/f3647c9a-1ae5-4ec6-8d69-c8db736e6fad-kube-api-access-cvqsm\") on node \"ip-10-0-142-6.ec2.internal\" DevicePath \"\"" Jul 30 00:25:20.065477 ip-10-0-142-6 kubenswrapper[2564]: I0730 00:25:20.065445 2564 generic.go:356] "Generic (PLEG): container finished" podID="7069b0f9-9545-48ce-885e-1ec788c65ea5" containerID="22c4c7bd184e0287b8fbab1a55106c26d1d184162ed14fd242b19550f46404ff" exitCode=0 Jul 30 00:25:20.065925 ip-10-0-142-6 kubenswrapper[2564]: I0730 00:25:20.065525 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0w74kw" event={"ID":"7069b0f9-9545-48ce-885e-1ec788c65ea5","Type":"ContainerDied","Data":"22c4c7bd184e0287b8fbab1a55106c26d1d184162ed14fd242b19550f46404ff"} Jul 30 00:25:20.067603 ip-10-0-142-6 kubenswrapper[2564]: I0730 00:25:20.067583 2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73k9924" Jul 30 00:25:20.067673 ip-10-0-142-6 kubenswrapper[2564]: I0730 00:25:20.067614 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73k9924" event={"ID":"f3647c9a-1ae5-4ec6-8d69-c8db736e6fad","Type":"ContainerDied","Data":"2436c4af27c8f663ffa0c709808c9d1f27abfc3de559896165eb4bc95b8e9744"} Jul 30 00:25:20.067707 ip-10-0-142-6 kubenswrapper[2564]: I0730 00:25:20.067672 2564 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="2436c4af27c8f663ffa0c709808c9d1f27abfc3de559896165eb4bc95b8e9744" Jul 30 00:25:20.098172 ip-10-0-142-6 kubenswrapper[2564]: E0730 00:25:20.098148 2564 cadvisor_stats_provider.go:567] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf3647c9a_1ae5_4ec6_8d69_c8db736e6fad.slice\": RecentStats: unable to find data in memory cache]" Jul 30 00:25:20.201229 ip-10-0-142-6 kubenswrapper[2564]: I0730 00:25:20.201207 2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef187fz8" Jul 30 00:25:20.234655 ip-10-0-142-6 kubenswrapper[2564]: I0730 00:25:20.234631 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-8rq2q\" (UniqueName: \"kubernetes.io/projected/fbd780fd-defb-402c-98ff-2cca1c03d7b0-kube-api-access-8rq2q\") pod \"fbd780fd-defb-402c-98ff-2cca1c03d7b0\" (UID: \"fbd780fd-defb-402c-98ff-2cca1c03d7b0\") " Jul 30 00:25:20.234764 ip-10-0-142-6 kubenswrapper[2564]: I0730 00:25:20.234671 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/fbd780fd-defb-402c-98ff-2cca1c03d7b0-util\") pod \"fbd780fd-defb-402c-98ff-2cca1c03d7b0\" (UID: \"fbd780fd-defb-402c-98ff-2cca1c03d7b0\") " Jul 30 00:25:20.234764 ip-10-0-142-6 kubenswrapper[2564]: I0730 00:25:20.234695 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/fbd780fd-defb-402c-98ff-2cca1c03d7b0-bundle\") pod \"fbd780fd-defb-402c-98ff-2cca1c03d7b0\" (UID: \"fbd780fd-defb-402c-98ff-2cca1c03d7b0\") " Jul 30 00:25:20.235349 ip-10-0-142-6 kubenswrapper[2564]: I0730 00:25:20.235317 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fbd780fd-defb-402c-98ff-2cca1c03d7b0-bundle" (OuterVolumeSpecName: "bundle") pod "fbd780fd-defb-402c-98ff-2cca1c03d7b0" (UID: "fbd780fd-defb-402c-98ff-2cca1c03d7b0"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 30 00:25:20.236581 ip-10-0-142-6 kubenswrapper[2564]: I0730 00:25:20.236561 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fbd780fd-defb-402c-98ff-2cca1c03d7b0-kube-api-access-8rq2q" (OuterVolumeSpecName: "kube-api-access-8rq2q") pod "fbd780fd-defb-402c-98ff-2cca1c03d7b0" (UID: "fbd780fd-defb-402c-98ff-2cca1c03d7b0"). InnerVolumeSpecName "kube-api-access-8rq2q". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 30 00:25:20.239721 ip-10-0-142-6 kubenswrapper[2564]: I0730 00:25:20.239689 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fbd780fd-defb-402c-98ff-2cca1c03d7b0-util" (OuterVolumeSpecName: "util") pod "fbd780fd-defb-402c-98ff-2cca1c03d7b0" (UID: "fbd780fd-defb-402c-98ff-2cca1c03d7b0"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 30 00:25:20.335287 ip-10-0-142-6 kubenswrapper[2564]: I0730 00:25:20.335196 2564 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8rq2q\" (UniqueName: \"kubernetes.io/projected/fbd780fd-defb-402c-98ff-2cca1c03d7b0-kube-api-access-8rq2q\") on node \"ip-10-0-142-6.ec2.internal\" DevicePath \"\"" Jul 30 00:25:20.335287 ip-10-0-142-6 kubenswrapper[2564]: I0730 00:25:20.335228 2564 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/fbd780fd-defb-402c-98ff-2cca1c03d7b0-util\") on node \"ip-10-0-142-6.ec2.internal\" DevicePath \"\"" Jul 30 00:25:20.335287 ip-10-0-142-6 kubenswrapper[2564]: I0730 00:25:20.335240 2564 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/fbd780fd-defb-402c-98ff-2cca1c03d7b0-bundle\") on node \"ip-10-0-142-6.ec2.internal\" DevicePath \"\"" Jul 30 00:25:21.073952 ip-10-0-142-6 kubenswrapper[2564]: I0730 00:25:21.073916 2564 generic.go:356] "Generic (PLEG): container finished" podID="7069b0f9-9545-48ce-885e-1ec788c65ea5" containerID="135d78e51307008dd673cac8ee3cb2c323bd07fd21dfac5cf92254c0424d1a13" exitCode=0 Jul 30 00:25:21.074323 ip-10-0-142-6 kubenswrapper[2564]: I0730 00:25:21.073979 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0w74kw" event={"ID":"7069b0f9-9545-48ce-885e-1ec788c65ea5","Type":"ContainerDied","Data":"135d78e51307008dd673cac8ee3cb2c323bd07fd21dfac5cf92254c0424d1a13"} Jul 30 00:25:21.075625 ip-10-0-142-6 kubenswrapper[2564]: I0730 00:25:21.075607 2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef187fz8" Jul 30 00:25:21.075727 ip-10-0-142-6 kubenswrapper[2564]: I0730 00:25:21.075645 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef187fz8" event={"ID":"fbd780fd-defb-402c-98ff-2cca1c03d7b0","Type":"ContainerDied","Data":"55c93145e98070660b1e9de7d32de42c5dfa9302ab8b1dd6b9684968ea3a3a77"} Jul 30 00:25:21.075727 ip-10-0-142-6 kubenswrapper[2564]: I0730 00:25:21.075680 2564 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="55c93145e98070660b1e9de7d32de42c5dfa9302ab8b1dd6b9684968ea3a3a77" Jul 30 00:25:21.077309 ip-10-0-142-6 kubenswrapper[2564]: I0730 00:25:21.077289 2564 generic.go:356] "Generic (PLEG): container finished" podID="f546aac1-87f6-499c-98c8-02afb9ae43f6" containerID="4518d14936c56a360609f537ace098dc350a6e2631278986fc5d0a5b9a4c6b5e" exitCode=0 Jul 30 00:25:21.077420 ip-10-0-142-6 kubenswrapper[2564]: I0730 00:25:21.077320 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759t2h7b" event={"ID":"f546aac1-87f6-499c-98c8-02afb9ae43f6","Type":"ContainerDied","Data":"4518d14936c56a360609f537ace098dc350a6e2631278986fc5d0a5b9a4c6b5e"} Jul 30 00:25:22.083638 ip-10-0-142-6 kubenswrapper[2564]: I0730 00:25:22.083608 2564 generic.go:356] "Generic (PLEG): container finished" podID="f546aac1-87f6-499c-98c8-02afb9ae43f6" containerID="29426be183409faca3dab6ac81608e1f4c13b2e9267aabcfb5a99b4de699550a" exitCode=0 Jul 30 00:25:22.084009 ip-10-0-142-6 kubenswrapper[2564]: I0730 00:25:22.083644 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759t2h7b" event={"ID":"f546aac1-87f6-499c-98c8-02afb9ae43f6","Type":"ContainerDied","Data":"29426be183409faca3dab6ac81608e1f4c13b2e9267aabcfb5a99b4de699550a"} Jul 30 00:25:22.211898 ip-10-0-142-6 kubenswrapper[2564]: I0730 00:25:22.211877 2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0w74kw" Jul 30 00:25:22.247953 ip-10-0-142-6 kubenswrapper[2564]: I0730 00:25:22.247923 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/7069b0f9-9545-48ce-885e-1ec788c65ea5-bundle\") pod \"7069b0f9-9545-48ce-885e-1ec788c65ea5\" (UID: \"7069b0f9-9545-48ce-885e-1ec788c65ea5\") " Jul 30 00:25:22.248102 ip-10-0-142-6 kubenswrapper[2564]: I0730 00:25:22.247995 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/7069b0f9-9545-48ce-885e-1ec788c65ea5-util\") pod \"7069b0f9-9545-48ce-885e-1ec788c65ea5\" (UID: \"7069b0f9-9545-48ce-885e-1ec788c65ea5\") " Jul 30 00:25:22.248102 ip-10-0-142-6 kubenswrapper[2564]: I0730 00:25:22.248023 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-tvrws\" (UniqueName: \"kubernetes.io/projected/7069b0f9-9545-48ce-885e-1ec788c65ea5-kube-api-access-tvrws\") pod \"7069b0f9-9545-48ce-885e-1ec788c65ea5\" (UID: \"7069b0f9-9545-48ce-885e-1ec788c65ea5\") " Jul 30 00:25:22.248376 ip-10-0-142-6 kubenswrapper[2564]: I0730 00:25:22.248351 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7069b0f9-9545-48ce-885e-1ec788c65ea5-bundle" (OuterVolumeSpecName: "bundle") pod "7069b0f9-9545-48ce-885e-1ec788c65ea5" (UID: "7069b0f9-9545-48ce-885e-1ec788c65ea5"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 30 00:25:22.250152 ip-10-0-142-6 kubenswrapper[2564]: I0730 00:25:22.250123 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7069b0f9-9545-48ce-885e-1ec788c65ea5-kube-api-access-tvrws" (OuterVolumeSpecName: "kube-api-access-tvrws") pod "7069b0f9-9545-48ce-885e-1ec788c65ea5" (UID: "7069b0f9-9545-48ce-885e-1ec788c65ea5"). InnerVolumeSpecName "kube-api-access-tvrws". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 30 00:25:22.254996 ip-10-0-142-6 kubenswrapper[2564]: I0730 00:25:22.254968 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7069b0f9-9545-48ce-885e-1ec788c65ea5-util" (OuterVolumeSpecName: "util") pod "7069b0f9-9545-48ce-885e-1ec788c65ea5" (UID: "7069b0f9-9545-48ce-885e-1ec788c65ea5"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 30 00:25:22.348763 ip-10-0-142-6 kubenswrapper[2564]: I0730 00:25:22.348660 2564 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/7069b0f9-9545-48ce-885e-1ec788c65ea5-util\") on node \"ip-10-0-142-6.ec2.internal\" DevicePath \"\"" Jul 30 00:25:22.348763 ip-10-0-142-6 kubenswrapper[2564]: I0730 00:25:22.348701 2564 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tvrws\" (UniqueName: \"kubernetes.io/projected/7069b0f9-9545-48ce-885e-1ec788c65ea5-kube-api-access-tvrws\") on node \"ip-10-0-142-6.ec2.internal\" DevicePath \"\"" Jul 30 00:25:22.348763 ip-10-0-142-6 kubenswrapper[2564]: I0730 00:25:22.348714 2564 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/7069b0f9-9545-48ce-885e-1ec788c65ea5-bundle\") on node \"ip-10-0-142-6.ec2.internal\" DevicePath \"\"" Jul 30 00:25:23.094922 ip-10-0-142-6 kubenswrapper[2564]: I0730 00:25:23.094883 2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0w74kw" Jul 30 00:25:23.094922 ip-10-0-142-6 kubenswrapper[2564]: I0730 00:25:23.094904 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0w74kw" event={"ID":"7069b0f9-9545-48ce-885e-1ec788c65ea5","Type":"ContainerDied","Data":"cd3d31ef9cafe994921c1f14d6d236ebd75cf045ba5e0de3604cdfd5bc3103b7"} Jul 30 00:25:23.095351 ip-10-0-142-6 kubenswrapper[2564]: I0730 00:25:23.094972 2564 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="cd3d31ef9cafe994921c1f14d6d236ebd75cf045ba5e0de3604cdfd5bc3103b7" Jul 30 00:25:23.220304 ip-10-0-142-6 kubenswrapper[2564]: I0730 00:25:23.220280 2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759t2h7b" Jul 30 00:25:23.254341 ip-10-0-142-6 kubenswrapper[2564]: I0730 00:25:23.254310 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/f546aac1-87f6-499c-98c8-02afb9ae43f6-bundle\") pod \"f546aac1-87f6-499c-98c8-02afb9ae43f6\" (UID: \"f546aac1-87f6-499c-98c8-02afb9ae43f6\") " Jul 30 00:25:23.254518 ip-10-0-142-6 kubenswrapper[2564]: I0730 00:25:23.254360 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/f546aac1-87f6-499c-98c8-02afb9ae43f6-util\") pod \"f546aac1-87f6-499c-98c8-02afb9ae43f6\" (UID: \"f546aac1-87f6-499c-98c8-02afb9ae43f6\") " Jul 30 00:25:23.254518 ip-10-0-142-6 kubenswrapper[2564]: I0730 00:25:23.254415 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-hhnx4\" (UniqueName: \"kubernetes.io/projected/f546aac1-87f6-499c-98c8-02afb9ae43f6-kube-api-access-hhnx4\") pod \"f546aac1-87f6-499c-98c8-02afb9ae43f6\" (UID: \"f546aac1-87f6-499c-98c8-02afb9ae43f6\") " Jul 30 00:25:23.254788 ip-10-0-142-6 kubenswrapper[2564]: I0730 00:25:23.254765 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f546aac1-87f6-499c-98c8-02afb9ae43f6-bundle" (OuterVolumeSpecName: "bundle") pod "f546aac1-87f6-499c-98c8-02afb9ae43f6" (UID: "f546aac1-87f6-499c-98c8-02afb9ae43f6"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 30 00:25:23.256684 ip-10-0-142-6 kubenswrapper[2564]: I0730 00:25:23.256648 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f546aac1-87f6-499c-98c8-02afb9ae43f6-kube-api-access-hhnx4" (OuterVolumeSpecName: "kube-api-access-hhnx4") pod "f546aac1-87f6-499c-98c8-02afb9ae43f6" (UID: "f546aac1-87f6-499c-98c8-02afb9ae43f6"). InnerVolumeSpecName "kube-api-access-hhnx4". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 30 00:25:23.259493 ip-10-0-142-6 kubenswrapper[2564]: I0730 00:25:23.259470 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f546aac1-87f6-499c-98c8-02afb9ae43f6-util" (OuterVolumeSpecName: "util") pod "f546aac1-87f6-499c-98c8-02afb9ae43f6" (UID: "f546aac1-87f6-499c-98c8-02afb9ae43f6"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 30 00:25:23.355389 ip-10-0-142-6 kubenswrapper[2564]: I0730 00:25:23.355308 2564 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/f546aac1-87f6-499c-98c8-02afb9ae43f6-bundle\") on node \"ip-10-0-142-6.ec2.internal\" DevicePath \"\"" Jul 30 00:25:23.355389 ip-10-0-142-6 kubenswrapper[2564]: I0730 00:25:23.355338 2564 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/f546aac1-87f6-499c-98c8-02afb9ae43f6-util\") on node \"ip-10-0-142-6.ec2.internal\" DevicePath \"\"" Jul 30 00:25:23.355389 ip-10-0-142-6 kubenswrapper[2564]: I0730 00:25:23.355351 2564 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hhnx4\" (UniqueName: \"kubernetes.io/projected/f546aac1-87f6-499c-98c8-02afb9ae43f6-kube-api-access-hhnx4\") on node \"ip-10-0-142-6.ec2.internal\" DevicePath \"\"" Jul 30 00:25:24.100892 ip-10-0-142-6 kubenswrapper[2564]: I0730 00:25:24.100863 2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759t2h7b" Jul 30 00:25:24.101289 ip-10-0-142-6 kubenswrapper[2564]: I0730 00:25:24.100892 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759t2h7b" event={"ID":"f546aac1-87f6-499c-98c8-02afb9ae43f6","Type":"ContainerDied","Data":"5b64dd5c87de2a74f4e5a7af6d155248af780d6a8e589a604bc3b35533839183"} Jul 30 00:25:24.101289 ip-10-0-142-6 kubenswrapper[2564]: I0730 00:25:24.100934 2564 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="5b64dd5c87de2a74f4e5a7af6d155248af780d6a8e589a604bc3b35533839183" Jul 30 00:25:36.389712 ip-10-0-142-6 kubenswrapper[2564]: I0730 00:25:36.389674 2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-59f8c89778-p7jds"] Jul 30 00:26:01.408730 ip-10-0-142-6 kubenswrapper[2564]: I0730 00:26:01.408671 2564 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-console/console-59f8c89778-p7jds" podUID="8a7dcdc5-5d0c-4188-a64c-3813a16ba4c5" containerName="console" containerID="cri-o://90cf487b20b88e3c992f464610b963c4d10106dcdbfe73ad069c88d23d396d60" gracePeriod=15 Jul 30 00:26:01.650332 ip-10-0-142-6 kubenswrapper[2564]: I0730 00:26:01.650310 2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-59f8c89778-p7jds" Jul 30 00:26:01.772782 ip-10-0-142-6 kubenswrapper[2564]: I0730 00:26:01.772702 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/8a7dcdc5-5d0c-4188-a64c-3813a16ba4c5-trusted-ca-bundle\") pod \"8a7dcdc5-5d0c-4188-a64c-3813a16ba4c5\" (UID: \"8a7dcdc5-5d0c-4188-a64c-3813a16ba4c5\") " Jul 30 00:26:01.772782 ip-10-0-142-6 kubenswrapper[2564]: I0730 00:26:01.772737 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/8a7dcdc5-5d0c-4188-a64c-3813a16ba4c5-console-serving-cert\") pod \"8a7dcdc5-5d0c-4188-a64c-3813a16ba4c5\" (UID: \"8a7dcdc5-5d0c-4188-a64c-3813a16ba4c5\") " Jul 30 00:26:01.772782 ip-10-0-142-6 kubenswrapper[2564]: I0730 00:26:01.772774 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/8a7dcdc5-5d0c-4188-a64c-3813a16ba4c5-console-oauth-config\") pod \"8a7dcdc5-5d0c-4188-a64c-3813a16ba4c5\" (UID: \"8a7dcdc5-5d0c-4188-a64c-3813a16ba4c5\") " Jul 30 00:26:01.773041 ip-10-0-142-6 kubenswrapper[2564]: I0730 00:26:01.772790 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/8a7dcdc5-5d0c-4188-a64c-3813a16ba4c5-oauth-serving-cert\") pod \"8a7dcdc5-5d0c-4188-a64c-3813a16ba4c5\" (UID: \"8a7dcdc5-5d0c-4188-a64c-3813a16ba4c5\") " Jul 30 00:26:01.773041 ip-10-0-142-6 kubenswrapper[2564]: I0730 00:26:01.772822 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-6qvs7\" (UniqueName: \"kubernetes.io/projected/8a7dcdc5-5d0c-4188-a64c-3813a16ba4c5-kube-api-access-6qvs7\") pod \"8a7dcdc5-5d0c-4188-a64c-3813a16ba4c5\" (UID: \"8a7dcdc5-5d0c-4188-a64c-3813a16ba4c5\") " Jul 30 00:26:01.773041 ip-10-0-142-6 kubenswrapper[2564]: I0730 00:26:01.772850 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/8a7dcdc5-5d0c-4188-a64c-3813a16ba4c5-console-config\") pod \"8a7dcdc5-5d0c-4188-a64c-3813a16ba4c5\" (UID: \"8a7dcdc5-5d0c-4188-a64c-3813a16ba4c5\") " Jul 30 00:26:01.773041 ip-10-0-142-6 kubenswrapper[2564]: I0730 00:26:01.772889 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/8a7dcdc5-5d0c-4188-a64c-3813a16ba4c5-service-ca\") pod \"8a7dcdc5-5d0c-4188-a64c-3813a16ba4c5\" (UID: \"8a7dcdc5-5d0c-4188-a64c-3813a16ba4c5\") " Jul 30 00:26:01.773236 ip-10-0-142-6 kubenswrapper[2564]: I0730 00:26:01.773151 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8a7dcdc5-5d0c-4188-a64c-3813a16ba4c5-trusted-ca-bundle" (OuterVolumeSpecName: "trusted-ca-bundle") pod "8a7dcdc5-5d0c-4188-a64c-3813a16ba4c5" (UID: "8a7dcdc5-5d0c-4188-a64c-3813a16ba4c5"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 30 00:26:01.773295 ip-10-0-142-6 kubenswrapper[2564]: I0730 00:26:01.773247 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8a7dcdc5-5d0c-4188-a64c-3813a16ba4c5-oauth-serving-cert" (OuterVolumeSpecName: "oauth-serving-cert") pod "8a7dcdc5-5d0c-4188-a64c-3813a16ba4c5" (UID: "8a7dcdc5-5d0c-4188-a64c-3813a16ba4c5"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 30 00:26:01.773567 ip-10-0-142-6 kubenswrapper[2564]: I0730 00:26:01.773540 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8a7dcdc5-5d0c-4188-a64c-3813a16ba4c5-console-config" (OuterVolumeSpecName: "console-config") pod "8a7dcdc5-5d0c-4188-a64c-3813a16ba4c5" (UID: "8a7dcdc5-5d0c-4188-a64c-3813a16ba4c5"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 30 00:26:01.773677 ip-10-0-142-6 kubenswrapper[2564]: I0730 00:26:01.773544 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8a7dcdc5-5d0c-4188-a64c-3813a16ba4c5-service-ca" (OuterVolumeSpecName: "service-ca") pod "8a7dcdc5-5d0c-4188-a64c-3813a16ba4c5" (UID: "8a7dcdc5-5d0c-4188-a64c-3813a16ba4c5"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 30 00:26:01.775030 ip-10-0-142-6 kubenswrapper[2564]: I0730 00:26:01.775008 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8a7dcdc5-5d0c-4188-a64c-3813a16ba4c5-console-oauth-config" (OuterVolumeSpecName: "console-oauth-config") pod "8a7dcdc5-5d0c-4188-a64c-3813a16ba4c5" (UID: "8a7dcdc5-5d0c-4188-a64c-3813a16ba4c5"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 30 00:26:01.775127 ip-10-0-142-6 kubenswrapper[2564]: I0730 00:26:01.775049 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8a7dcdc5-5d0c-4188-a64c-3813a16ba4c5-console-serving-cert" (OuterVolumeSpecName: "console-serving-cert") pod "8a7dcdc5-5d0c-4188-a64c-3813a16ba4c5" (UID: "8a7dcdc5-5d0c-4188-a64c-3813a16ba4c5"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 30 00:26:01.775182 ip-10-0-142-6 kubenswrapper[2564]: I0730 00:26:01.775143 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8a7dcdc5-5d0c-4188-a64c-3813a16ba4c5-kube-api-access-6qvs7" (OuterVolumeSpecName: "kube-api-access-6qvs7") pod "8a7dcdc5-5d0c-4188-a64c-3813a16ba4c5" (UID: "8a7dcdc5-5d0c-4188-a64c-3813a16ba4c5"). InnerVolumeSpecName "kube-api-access-6qvs7". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 30 00:26:01.874189 ip-10-0-142-6 kubenswrapper[2564]: I0730 00:26:01.874155 2564 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/8a7dcdc5-5d0c-4188-a64c-3813a16ba4c5-console-oauth-config\") on node \"ip-10-0-142-6.ec2.internal\" DevicePath \"\"" Jul 30 00:26:01.874189 ip-10-0-142-6 kubenswrapper[2564]: I0730 00:26:01.874183 2564 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/8a7dcdc5-5d0c-4188-a64c-3813a16ba4c5-oauth-serving-cert\") on node \"ip-10-0-142-6.ec2.internal\" DevicePath \"\"" Jul 30 00:26:01.874189 ip-10-0-142-6 kubenswrapper[2564]: I0730 00:26:01.874195 2564 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6qvs7\" (UniqueName: \"kubernetes.io/projected/8a7dcdc5-5d0c-4188-a64c-3813a16ba4c5-kube-api-access-6qvs7\") on node \"ip-10-0-142-6.ec2.internal\" DevicePath \"\"" Jul 30 00:26:01.874422 ip-10-0-142-6 kubenswrapper[2564]: I0730 00:26:01.874207 2564 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/8a7dcdc5-5d0c-4188-a64c-3813a16ba4c5-console-config\") on node \"ip-10-0-142-6.ec2.internal\" DevicePath \"\"" Jul 30 00:26:01.874422 ip-10-0-142-6 kubenswrapper[2564]: I0730 00:26:01.874218 2564 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/8a7dcdc5-5d0c-4188-a64c-3813a16ba4c5-service-ca\") on node \"ip-10-0-142-6.ec2.internal\" DevicePath \"\"" Jul 30 00:26:01.874422 ip-10-0-142-6 kubenswrapper[2564]: I0730 00:26:01.874228 2564 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/8a7dcdc5-5d0c-4188-a64c-3813a16ba4c5-trusted-ca-bundle\") on node \"ip-10-0-142-6.ec2.internal\" DevicePath \"\"" Jul 30 00:26:01.874422 ip-10-0-142-6 kubenswrapper[2564]: I0730 00:26:01.874241 2564 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/8a7dcdc5-5d0c-4188-a64c-3813a16ba4c5-console-serving-cert\") on node \"ip-10-0-142-6.ec2.internal\" DevicePath \"\"" Jul 30 00:26:02.281104 ip-10-0-142-6 kubenswrapper[2564]: I0730 00:26:02.281047 2564 generic.go:356] "Generic (PLEG): container finished" podID="8a7dcdc5-5d0c-4188-a64c-3813a16ba4c5" containerID="90cf487b20b88e3c992f464610b963c4d10106dcdbfe73ad069c88d23d396d60" exitCode=0 Jul 30 00:26:02.281273 ip-10-0-142-6 kubenswrapper[2564]: I0730 00:26:02.281112 2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-59f8c89778-p7jds" Jul 30 00:26:02.281273 ip-10-0-142-6 kubenswrapper[2564]: I0730 00:26:02.281111 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-59f8c89778-p7jds" event={"ID":"8a7dcdc5-5d0c-4188-a64c-3813a16ba4c5","Type":"ContainerDied","Data":"90cf487b20b88e3c992f464610b963c4d10106dcdbfe73ad069c88d23d396d60"} Jul 30 00:26:02.281273 ip-10-0-142-6 kubenswrapper[2564]: I0730 00:26:02.281162 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-59f8c89778-p7jds" event={"ID":"8a7dcdc5-5d0c-4188-a64c-3813a16ba4c5","Type":"ContainerDied","Data":"2729715a01565cad922d986309a4cff4553599f0afefe131f2d6ee0d2f8a041f"} Jul 30 00:26:02.281273 ip-10-0-142-6 kubenswrapper[2564]: I0730 00:26:02.281182 2564 scope.go:117] "RemoveContainer" containerID="90cf487b20b88e3c992f464610b963c4d10106dcdbfe73ad069c88d23d396d60" Jul 30 00:26:02.289888 ip-10-0-142-6 kubenswrapper[2564]: I0730 00:26:02.289873 2564 scope.go:117] "RemoveContainer" containerID="90cf487b20b88e3c992f464610b963c4d10106dcdbfe73ad069c88d23d396d60" Jul 30 00:26:02.290144 ip-10-0-142-6 kubenswrapper[2564]: E0730 00:26:02.290125 2564 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"90cf487b20b88e3c992f464610b963c4d10106dcdbfe73ad069c88d23d396d60\": container with ID starting with 90cf487b20b88e3c992f464610b963c4d10106dcdbfe73ad069c88d23d396d60 not found: ID does not exist" containerID="90cf487b20b88e3c992f464610b963c4d10106dcdbfe73ad069c88d23d396d60" Jul 30 00:26:02.290219 ip-10-0-142-6 kubenswrapper[2564]: I0730 00:26:02.290154 2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"90cf487b20b88e3c992f464610b963c4d10106dcdbfe73ad069c88d23d396d60"} err="failed to get container status \"90cf487b20b88e3c992f464610b963c4d10106dcdbfe73ad069c88d23d396d60\": rpc error: code = NotFound desc = could not find container \"90cf487b20b88e3c992f464610b963c4d10106dcdbfe73ad069c88d23d396d60\": container with ID starting with 90cf487b20b88e3c992f464610b963c4d10106dcdbfe73ad069c88d23d396d60 not found: ID does not exist" Jul 30 00:26:02.305334 ip-10-0-142-6 kubenswrapper[2564]: I0730 00:26:02.305309 2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-59f8c89778-p7jds"] Jul 30 00:26:02.307178 ip-10-0-142-6 kubenswrapper[2564]: I0730 00:26:02.307157 2564 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-59f8c89778-p7jds"] Jul 30 00:26:03.139772 ip-10-0-142-6 kubenswrapper[2564]: I0730 00:26:03.139740 2564 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="8a7dcdc5-5d0c-4188-a64c-3813a16ba4c5" path="/var/lib/kubelet/pods/8a7dcdc5-5d0c-4188-a64c-3813a16ba4c5/volumes" Jul 30 00:26:48.460815 ip-10-0-142-6 kubenswrapper[2564]: I0730 00:26:48.460725 2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-nrv2p"] Jul 30 00:26:48.461390 ip-10-0-142-6 kubenswrapper[2564]: I0730 00:26:48.461349 2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f3647c9a-1ae5-4ec6-8d69-c8db736e6fad" containerName="pull" Jul 30 00:26:48.461390 ip-10-0-142-6 kubenswrapper[2564]: I0730 00:26:48.461365 2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="f3647c9a-1ae5-4ec6-8d69-c8db736e6fad" containerName="pull" Jul 30 00:26:48.461390 ip-10-0-142-6 kubenswrapper[2564]: I0730 00:26:48.461377 2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7069b0f9-9545-48ce-885e-1ec788c65ea5" containerName="pull" Jul 30 00:26:48.461390 ip-10-0-142-6 kubenswrapper[2564]: I0730 00:26:48.461386 2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="7069b0f9-9545-48ce-885e-1ec788c65ea5" containerName="pull" Jul 30 00:26:48.461530 ip-10-0-142-6 kubenswrapper[2564]: I0730 00:26:48.461396 2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f546aac1-87f6-499c-98c8-02afb9ae43f6" containerName="pull" Jul 30 00:26:48.461530 ip-10-0-142-6 kubenswrapper[2564]: I0730 00:26:48.461401 2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="f546aac1-87f6-499c-98c8-02afb9ae43f6" containerName="pull" Jul 30 00:26:48.461530 ip-10-0-142-6 kubenswrapper[2564]: I0730 00:26:48.461410 2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f3647c9a-1ae5-4ec6-8d69-c8db736e6fad" containerName="extract" Jul 30 00:26:48.461530 ip-10-0-142-6 kubenswrapper[2564]: I0730 00:26:48.461416 2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="f3647c9a-1ae5-4ec6-8d69-c8db736e6fad" containerName="extract" Jul 30 00:26:48.461530 ip-10-0-142-6 kubenswrapper[2564]: I0730 00:26:48.461425 2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f546aac1-87f6-499c-98c8-02afb9ae43f6" containerName="extract" Jul 30 00:26:48.461530 ip-10-0-142-6 kubenswrapper[2564]: I0730 00:26:48.461430 2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="f546aac1-87f6-499c-98c8-02afb9ae43f6" containerName="extract" Jul 30 00:26:48.461530 ip-10-0-142-6 kubenswrapper[2564]: I0730 00:26:48.461437 2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="fbd780fd-defb-402c-98ff-2cca1c03d7b0" containerName="extract" Jul 30 00:26:48.461530 ip-10-0-142-6 kubenswrapper[2564]: I0730 00:26:48.461442 2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="fbd780fd-defb-402c-98ff-2cca1c03d7b0" containerName="extract" Jul 30 00:26:48.461530 ip-10-0-142-6 kubenswrapper[2564]: I0730 00:26:48.461449 2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="fbd780fd-defb-402c-98ff-2cca1c03d7b0" containerName="pull" Jul 30 00:26:48.461530 ip-10-0-142-6 kubenswrapper[2564]: I0730 00:26:48.461454 2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="fbd780fd-defb-402c-98ff-2cca1c03d7b0" containerName="pull" Jul 30 00:26:48.461530 ip-10-0-142-6 kubenswrapper[2564]: I0730 00:26:48.461466 2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f546aac1-87f6-499c-98c8-02afb9ae43f6" containerName="util" Jul 30 00:26:48.461530 ip-10-0-142-6 kubenswrapper[2564]: I0730 00:26:48.461473 2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="f546aac1-87f6-499c-98c8-02afb9ae43f6" containerName="util" Jul 30 00:26:48.461530 ip-10-0-142-6 kubenswrapper[2564]: I0730 00:26:48.461479 2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7069b0f9-9545-48ce-885e-1ec788c65ea5" containerName="extract" Jul 30 00:26:48.461530 ip-10-0-142-6 kubenswrapper[2564]: I0730 00:26:48.461484 2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="7069b0f9-9545-48ce-885e-1ec788c65ea5" containerName="extract" Jul 30 00:26:48.461530 ip-10-0-142-6 kubenswrapper[2564]: I0730 00:26:48.461491 2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="fbd780fd-defb-402c-98ff-2cca1c03d7b0" containerName="util" Jul 30 00:26:48.461530 ip-10-0-142-6 kubenswrapper[2564]: I0730 00:26:48.461495 2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="fbd780fd-defb-402c-98ff-2cca1c03d7b0" containerName="util" Jul 30 00:26:48.461530 ip-10-0-142-6 kubenswrapper[2564]: I0730 00:26:48.461500 2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8a7dcdc5-5d0c-4188-a64c-3813a16ba4c5" containerName="console" Jul 30 00:26:48.461530 ip-10-0-142-6 kubenswrapper[2564]: I0730 00:26:48.461505 2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="8a7dcdc5-5d0c-4188-a64c-3813a16ba4c5" containerName="console" Jul 30 00:26:48.461530 ip-10-0-142-6 kubenswrapper[2564]: I0730 00:26:48.461513 2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7069b0f9-9545-48ce-885e-1ec788c65ea5" containerName="util" Jul 30 00:26:48.461530 ip-10-0-142-6 kubenswrapper[2564]: I0730 00:26:48.461519 2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="7069b0f9-9545-48ce-885e-1ec788c65ea5" containerName="util" Jul 30 00:26:48.461530 ip-10-0-142-6 kubenswrapper[2564]: I0730 00:26:48.461525 2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f3647c9a-1ae5-4ec6-8d69-c8db736e6fad" containerName="util" Jul 30 00:26:48.461530 ip-10-0-142-6 kubenswrapper[2564]: I0730 00:26:48.461531 2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="f3647c9a-1ae5-4ec6-8d69-c8db736e6fad" containerName="util" Jul 30 00:26:48.462197 ip-10-0-142-6 kubenswrapper[2564]: I0730 00:26:48.461588 2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="f546aac1-87f6-499c-98c8-02afb9ae43f6" containerName="extract" Jul 30 00:26:48.462197 ip-10-0-142-6 kubenswrapper[2564]: I0730 00:26:48.461596 2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="7069b0f9-9545-48ce-885e-1ec788c65ea5" containerName="extract" Jul 30 00:26:48.462197 ip-10-0-142-6 kubenswrapper[2564]: I0730 00:26:48.461604 2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="f3647c9a-1ae5-4ec6-8d69-c8db736e6fad" containerName="extract" Jul 30 00:26:48.462197 ip-10-0-142-6 kubenswrapper[2564]: I0730 00:26:48.461610 2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="8a7dcdc5-5d0c-4188-a64c-3813a16ba4c5" containerName="console" Jul 30 00:26:48.462197 ip-10-0-142-6 kubenswrapper[2564]: I0730 00:26:48.461621 2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="fbd780fd-defb-402c-98ff-2cca1c03d7b0" containerName="extract" Jul 30 00:26:48.464477 ip-10-0-142-6 kubenswrapper[2564]: I0730 00:26:48.464463 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-54d6d988d5-nrv2p" Jul 30 00:26:48.466992 ip-10-0-142-6 kubenswrapper[2564]: I0730 00:26:48.466973 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kube-root-ca.crt\"" Jul 30 00:26:48.467149 ip-10-0-142-6 kubenswrapper[2564]: I0730 00:26:48.467131 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"limitador-limits-config-limitador\"" Jul 30 00:26:48.467208 ip-10-0-142-6 kubenswrapper[2564]: I0730 00:26:48.467164 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"openshift-service-ca.crt\"" Jul 30 00:26:48.467893 ip-10-0-142-6 kubenswrapper[2564]: I0730 00:26:48.467878 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"default-dockercfg-4v4jb\"" Jul 30 00:26:48.475112 ip-10-0-142-6 kubenswrapper[2564]: I0730 00:26:48.475086 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-nrv2p"] Jul 30 00:26:48.507514 ip-10-0-142-6 kubenswrapper[2564]: I0730 00:26:48.507487 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-nrv2p"] Jul 30 00:26:48.550715 ip-10-0-142-6 kubenswrapper[2564]: I0730 00:26:48.550690 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-79z65\" (UniqueName: \"kubernetes.io/projected/4197b50a-70ea-4d6d-b57d-d648866eca86-kube-api-access-79z65\") pod \"limitador-limitador-54d6d988d5-nrv2p\" (UID: \"4197b50a-70ea-4d6d-b57d-d648866eca86\") " pod="kuadrant-system/limitador-limitador-54d6d988d5-nrv2p" Jul 30 00:26:48.550829 ip-10-0-142-6 kubenswrapper[2564]: I0730 00:26:48.550745 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/4197b50a-70ea-4d6d-b57d-d648866eca86-config-file\") pod \"limitador-limitador-54d6d988d5-nrv2p\" (UID: \"4197b50a-70ea-4d6d-b57d-d648866eca86\") " pod="kuadrant-system/limitador-limitador-54d6d988d5-nrv2p" Jul 30 00:26:48.651551 ip-10-0-142-6 kubenswrapper[2564]: I0730 00:26:48.651518 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-79z65\" (UniqueName: \"kubernetes.io/projected/4197b50a-70ea-4d6d-b57d-d648866eca86-kube-api-access-79z65\") pod \"limitador-limitador-54d6d988d5-nrv2p\" (UID: \"4197b50a-70ea-4d6d-b57d-d648866eca86\") " pod="kuadrant-system/limitador-limitador-54d6d988d5-nrv2p" Jul 30 00:26:48.651702 ip-10-0-142-6 kubenswrapper[2564]: I0730 00:26:48.651569 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/4197b50a-70ea-4d6d-b57d-d648866eca86-config-file\") pod \"limitador-limitador-54d6d988d5-nrv2p\" (UID: \"4197b50a-70ea-4d6d-b57d-d648866eca86\") " pod="kuadrant-system/limitador-limitador-54d6d988d5-nrv2p" Jul 30 00:26:48.652146 ip-10-0-142-6 kubenswrapper[2564]: I0730 00:26:48.652130 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/4197b50a-70ea-4d6d-b57d-d648866eca86-config-file\") pod \"limitador-limitador-54d6d988d5-nrv2p\" (UID: \"4197b50a-70ea-4d6d-b57d-d648866eca86\") " pod="kuadrant-system/limitador-limitador-54d6d988d5-nrv2p" Jul 30 00:26:48.660172 ip-10-0-142-6 kubenswrapper[2564]: I0730 00:26:48.660150 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-79z65\" (UniqueName: \"kubernetes.io/projected/4197b50a-70ea-4d6d-b57d-d648866eca86-kube-api-access-79z65\") pod \"limitador-limitador-54d6d988d5-nrv2p\" (UID: \"4197b50a-70ea-4d6d-b57d-d648866eca86\") " pod="kuadrant-system/limitador-limitador-54d6d988d5-nrv2p" Jul 30 00:26:48.774633 ip-10-0-142-6 kubenswrapper[2564]: I0730 00:26:48.774545 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-54d6d988d5-nrv2p" Jul 30 00:26:48.902524 ip-10-0-142-6 kubenswrapper[2564]: I0730 00:26:48.902495 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-nrv2p"] Jul 30 00:26:48.904353 ip-10-0-142-6 kubenswrapper[2564]: W0730 00:26:48.904325 2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4197b50a_70ea_4d6d_b57d_d648866eca86.slice/crio-79e9f1e27ef41d5806331d3767e95c869af0e23b93c37481130ffa41dbe744ef WatchSource:0}: Error finding container 79e9f1e27ef41d5806331d3767e95c869af0e23b93c37481130ffa41dbe744ef: Status 404 returned error can't find the container with id 79e9f1e27ef41d5806331d3767e95c869af0e23b93c37481130ffa41dbe744ef Jul 30 00:26:49.502877 ip-10-0-142-6 kubenswrapper[2564]: I0730 00:26:49.502793 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-54d6d988d5-nrv2p" event={"ID":"4197b50a-70ea-4d6d-b57d-d648866eca86","Type":"ContainerStarted","Data":"79e9f1e27ef41d5806331d3767e95c869af0e23b93c37481130ffa41dbe744ef"} Jul 30 00:26:51.515331 ip-10-0-142-6 kubenswrapper[2564]: I0730 00:26:51.515301 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-54d6d988d5-nrv2p" event={"ID":"4197b50a-70ea-4d6d-b57d-d648866eca86","Type":"ContainerStarted","Data":"b79b6925e2cc75e181bb4badf5ae25e374d923b46ad2c2147f94e2c17f990b72"} Jul 30 00:26:51.515690 ip-10-0-142-6 kubenswrapper[2564]: I0730 00:26:51.515374 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/limitador-limitador-54d6d988d5-nrv2p" Jul 30 00:26:51.532090 ip-10-0-142-6 kubenswrapper[2564]: I0730 00:26:51.532029 2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/limitador-limitador-54d6d988d5-nrv2p" podStartSLOduration=1.062881669 podStartE2EDuration="3.53201523s" totalImagesPullingTime="2.469133561s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-30 00:26:48 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-30 00:26:51.530236599 +0000 UTC m=+542.944976527" watchObservedRunningTime="2026-07-30 00:26:51.53201523 +0000 UTC m=+542.946755167" Jul 30 00:27:02.520930 ip-10-0-142-6 kubenswrapper[2564]: I0730 00:27:02.520901 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/limitador-limitador-54d6d988d5-nrv2p" Jul 30 00:27:59.213497 ip-10-0-142-6 kubenswrapper[2564]: I0730 00:27:59.213462 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-nrv2p"] Jul 30 00:28:24.808692 ip-10-0-142-6 kubenswrapper[2564]: I0730 00:28:24.808615 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-nrv2p"] Jul 30 00:28:34.411448 ip-10-0-142-6 kubenswrapper[2564]: I0730 00:28:34.411411 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-nrv2p"] Jul 30 00:28:45.015469 ip-10-0-142-6 kubenswrapper[2564]: I0730 00:28:45.015435 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-nrv2p"] Jul 30 00:28:49.006251 ip-10-0-142-6 kubenswrapper[2564]: I0730 00:28:49.006214 2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fgdcgh"] Jul 30 00:28:49.009762 ip-10-0-142-6 kubenswrapper[2564]: I0730 00:28:49.009740 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fgdcgh" Jul 30 00:28:49.013121 ip-10-0-142-6 kubenswrapper[2564]: I0730 00:28:49.013100 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"openshift-service-ca.crt\"" Jul 30 00:28:49.013255 ip-10-0-142-6 kubenswrapper[2564]: I0730 00:28:49.013100 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"kube-root-ca.crt\"" Jul 30 00:28:49.013255 ip-10-0-142-6 kubenswrapper[2564]: I0730 00:28:49.013156 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"default-dockercfg-fwqxf\"" Jul 30 00:28:49.013255 ip-10-0-142-6 kubenswrapper[2564]: I0730 00:28:49.013207 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-unab60ef4d3a239b5143b412cab04acac3-kserve-self-signed-certs\"" Jul 30 00:28:49.019064 ip-10-0-142-6 kubenswrapper[2564]: I0730 00:28:49.019042 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fgdcgh"] Jul 30 00:28:49.117804 ip-10-0-142-6 kubenswrapper[2564]: I0730 00:28:49.117772 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/eebf6898-fe64-4cde-945c-f7226f4cd960-tls-certs\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fgdcgh\" (UID: \"eebf6898-fe64-4cde-945c-f7226f4cd960\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fgdcgh" Jul 30 00:28:49.117961 ip-10-0-142-6 kubenswrapper[2564]: I0730 00:28:49.117819 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nxrt5\" (UniqueName: \"kubernetes.io/projected/eebf6898-fe64-4cde-945c-f7226f4cd960-kube-api-access-nxrt5\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fgdcgh\" (UID: \"eebf6898-fe64-4cde-945c-f7226f4cd960\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fgdcgh" Jul 30 00:28:49.117961 ip-10-0-142-6 kubenswrapper[2564]: I0730 00:28:49.117875 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/eebf6898-fe64-4cde-945c-f7226f4cd960-dshm\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fgdcgh\" (UID: \"eebf6898-fe64-4cde-945c-f7226f4cd960\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fgdcgh" Jul 30 00:28:49.117961 ip-10-0-142-6 kubenswrapper[2564]: I0730 00:28:49.117953 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/eebf6898-fe64-4cde-945c-f7226f4cd960-tmp-dir\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fgdcgh\" (UID: \"eebf6898-fe64-4cde-945c-f7226f4cd960\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fgdcgh" Jul 30 00:28:49.118058 ip-10-0-142-6 kubenswrapper[2564]: I0730 00:28:49.117993 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/eebf6898-fe64-4cde-945c-f7226f4cd960-home\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fgdcgh\" (UID: \"eebf6898-fe64-4cde-945c-f7226f4cd960\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fgdcgh" Jul 30 00:28:49.118058 ip-10-0-142-6 kubenswrapper[2564]: I0730 00:28:49.118015 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/eebf6898-fe64-4cde-945c-f7226f4cd960-model-cache\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fgdcgh\" (UID: \"eebf6898-fe64-4cde-945c-f7226f4cd960\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fgdcgh" Jul 30 00:28:49.219371 ip-10-0-142-6 kubenswrapper[2564]: I0730 00:28:49.219335 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/eebf6898-fe64-4cde-945c-f7226f4cd960-home\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fgdcgh\" (UID: \"eebf6898-fe64-4cde-945c-f7226f4cd960\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fgdcgh" Jul 30 00:28:49.219371 ip-10-0-142-6 kubenswrapper[2564]: I0730 00:28:49.219377 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/eebf6898-fe64-4cde-945c-f7226f4cd960-model-cache\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fgdcgh\" (UID: \"eebf6898-fe64-4cde-945c-f7226f4cd960\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fgdcgh" Jul 30 00:28:49.219601 ip-10-0-142-6 kubenswrapper[2564]: I0730 00:28:49.219501 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/eebf6898-fe64-4cde-945c-f7226f4cd960-tls-certs\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fgdcgh\" (UID: \"eebf6898-fe64-4cde-945c-f7226f4cd960\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fgdcgh" Jul 30 00:28:49.219671 ip-10-0-142-6 kubenswrapper[2564]: I0730 00:28:49.219647 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nxrt5\" (UniqueName: \"kubernetes.io/projected/eebf6898-fe64-4cde-945c-f7226f4cd960-kube-api-access-nxrt5\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fgdcgh\" (UID: \"eebf6898-fe64-4cde-945c-f7226f4cd960\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fgdcgh" Jul 30 00:28:49.219719 ip-10-0-142-6 kubenswrapper[2564]: I0730 00:28:49.219673 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/eebf6898-fe64-4cde-945c-f7226f4cd960-home\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fgdcgh\" (UID: \"eebf6898-fe64-4cde-945c-f7226f4cd960\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fgdcgh" Jul 30 00:28:49.219719 ip-10-0-142-6 kubenswrapper[2564]: I0730 00:28:49.219700 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/eebf6898-fe64-4cde-945c-f7226f4cd960-dshm\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fgdcgh\" (UID: \"eebf6898-fe64-4cde-945c-f7226f4cd960\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fgdcgh" Jul 30 00:28:49.219719 ip-10-0-142-6 kubenswrapper[2564]: I0730 00:28:49.219707 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/eebf6898-fe64-4cde-945c-f7226f4cd960-model-cache\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fgdcgh\" (UID: \"eebf6898-fe64-4cde-945c-f7226f4cd960\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fgdcgh" Jul 30 00:28:49.219825 ip-10-0-142-6 kubenswrapper[2564]: I0730 00:28:49.219756 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/eebf6898-fe64-4cde-945c-f7226f4cd960-tmp-dir\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fgdcgh\" (UID: \"eebf6898-fe64-4cde-945c-f7226f4cd960\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fgdcgh" Jul 30 00:28:49.219977 ip-10-0-142-6 kubenswrapper[2564]: I0730 00:28:49.219952 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/eebf6898-fe64-4cde-945c-f7226f4cd960-tmp-dir\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fgdcgh\" (UID: \"eebf6898-fe64-4cde-945c-f7226f4cd960\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fgdcgh" Jul 30 00:28:49.221729 ip-10-0-142-6 kubenswrapper[2564]: I0730 00:28:49.221709 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/eebf6898-fe64-4cde-945c-f7226f4cd960-dshm\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fgdcgh\" (UID: \"eebf6898-fe64-4cde-945c-f7226f4cd960\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fgdcgh" Jul 30 00:28:49.221991 ip-10-0-142-6 kubenswrapper[2564]: I0730 00:28:49.221974 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/eebf6898-fe64-4cde-945c-f7226f4cd960-tls-certs\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fgdcgh\" (UID: \"eebf6898-fe64-4cde-945c-f7226f4cd960\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fgdcgh" Jul 30 00:28:49.228129 ip-10-0-142-6 kubenswrapper[2564]: I0730 00:28:49.228104 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-nxrt5\" (UniqueName: \"kubernetes.io/projected/eebf6898-fe64-4cde-945c-f7226f4cd960-kube-api-access-nxrt5\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fgdcgh\" (UID: \"eebf6898-fe64-4cde-945c-f7226f4cd960\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fgdcgh" Jul 30 00:28:49.321718 ip-10-0-142-6 kubenswrapper[2564]: I0730 00:28:49.321637 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fgdcgh" Jul 30 00:28:49.450446 ip-10-0-142-6 kubenswrapper[2564]: I0730 00:28:49.450425 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fgdcgh"] Jul 30 00:28:49.452248 ip-10-0-142-6 kubenswrapper[2564]: W0730 00:28:49.452213 2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podeebf6898_fe64_4cde_945c_f7226f4cd960.slice/crio-c3463c60e54ddeda80c2f824d5022a8eed562161fcc27afce7fcfa857581c4a4 WatchSource:0}: Error finding container c3463c60e54ddeda80c2f824d5022a8eed562161fcc27afce7fcfa857581c4a4: Status 404 returned error can't find the container with id c3463c60e54ddeda80c2f824d5022a8eed562161fcc27afce7fcfa857581c4a4 Jul 30 00:28:49.453581 ip-10-0-142-6 kubenswrapper[2564]: I0730 00:28:49.453567 2564 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Jul 30 00:28:50.067604 ip-10-0-142-6 kubenswrapper[2564]: I0730 00:28:50.067568 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fgdcgh" event={"ID":"eebf6898-fe64-4cde-945c-f7226f4cd960","Type":"ContainerStarted","Data":"c3463c60e54ddeda80c2f824d5022a8eed562161fcc27afce7fcfa857581c4a4"} Jul 30 00:28:51.007912 ip-10-0-142-6 kubenswrapper[2564]: I0730 00:28:51.007878 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-nrv2p"] Jul 30 00:28:51.074003 ip-10-0-142-6 kubenswrapper[2564]: I0730 00:28:51.073968 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fgdcgh" event={"ID":"eebf6898-fe64-4cde-945c-f7226f4cd960","Type":"ContainerStarted","Data":"8feebdd6269da0513e4bc4f93a953e895546bdd2dcd5d5f34c4083386a6ec0cd"} Jul 30 00:28:51.091059 ip-10-0-142-6 kubenswrapper[2564]: I0730 00:28:51.091004 2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fgdcgh" podStartSLOduration=2.150150685 podStartE2EDuration="3.090987518s" totalImagesPullingTime="940.836833ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-30 00:28:48 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-30 00:28:51.089115143 +0000 UTC m=+662.503855076" watchObservedRunningTime="2026-07-30 00:28:51.090987518 +0000 UTC m=+662.505727455" Jul 30 00:28:54.102692 ip-10-0-142-6 kubenswrapper[2564]: I0730 00:28:54.102657 2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/facebook-opt-125m-simulated-kserve-74b68447df-czxkz"] Jul 30 00:28:54.106219 ip-10-0-142-6 kubenswrapper[2564]: I0730 00:28:54.106200 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-czxkz" Jul 30 00:28:54.110402 ip-10-0-142-6 kubenswrapper[2564]: I0730 00:28:54.110384 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"facebook-opt-125m-simulated-kserve-self-signed-certs\"" Jul 30 00:28:54.117397 ip-10-0-142-6 kubenswrapper[2564]: I0730 00:28:54.117377 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/facebook-opt-125m-simulated-kserve-74b68447df-czxkz"] Jul 30 00:28:54.161992 ip-10-0-142-6 kubenswrapper[2564]: I0730 00:28:54.161968 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4300d011-90a2-4b87-a736-91f40a42f8c3-model-cache\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-czxkz\" (UID: \"4300d011-90a2-4b87-a736-91f40a42f8c3\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-czxkz" Jul 30 00:28:54.162145 ip-10-0-142-6 kubenswrapper[2564]: I0730 00:28:54.162004 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4300d011-90a2-4b87-a736-91f40a42f8c3-home\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-czxkz\" (UID: \"4300d011-90a2-4b87-a736-91f40a42f8c3\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-czxkz" Jul 30 00:28:54.162145 ip-10-0-142-6 kubenswrapper[2564]: I0730 00:28:54.162020 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4300d011-90a2-4b87-a736-91f40a42f8c3-tls-certs\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-czxkz\" (UID: \"4300d011-90a2-4b87-a736-91f40a42f8c3\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-czxkz" Jul 30 00:28:54.162145 ip-10-0-142-6 kubenswrapper[2564]: I0730 00:28:54.162039 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4300d011-90a2-4b87-a736-91f40a42f8c3-tmp-dir\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-czxkz\" (UID: \"4300d011-90a2-4b87-a736-91f40a42f8c3\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-czxkz" Jul 30 00:28:54.162251 ip-10-0-142-6 kubenswrapper[2564]: I0730 00:28:54.162176 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4300d011-90a2-4b87-a736-91f40a42f8c3-dshm\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-czxkz\" (UID: \"4300d011-90a2-4b87-a736-91f40a42f8c3\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-czxkz" Jul 30 00:28:54.162251 ip-10-0-142-6 kubenswrapper[2564]: I0730 00:28:54.162200 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7ctpp\" (UniqueName: \"kubernetes.io/projected/4300d011-90a2-4b87-a736-91f40a42f8c3-kube-api-access-7ctpp\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-czxkz\" (UID: \"4300d011-90a2-4b87-a736-91f40a42f8c3\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-czxkz" Jul 30 00:28:54.263011 ip-10-0-142-6 kubenswrapper[2564]: I0730 00:28:54.262974 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4300d011-90a2-4b87-a736-91f40a42f8c3-dshm\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-czxkz\" (UID: \"4300d011-90a2-4b87-a736-91f40a42f8c3\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-czxkz" Jul 30 00:28:54.263011 ip-10-0-142-6 kubenswrapper[2564]: I0730 00:28:54.263013 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7ctpp\" (UniqueName: \"kubernetes.io/projected/4300d011-90a2-4b87-a736-91f40a42f8c3-kube-api-access-7ctpp\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-czxkz\" (UID: \"4300d011-90a2-4b87-a736-91f40a42f8c3\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-czxkz" Jul 30 00:28:54.263290 ip-10-0-142-6 kubenswrapper[2564]: I0730 00:28:54.263041 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4300d011-90a2-4b87-a736-91f40a42f8c3-model-cache\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-czxkz\" (UID: \"4300d011-90a2-4b87-a736-91f40a42f8c3\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-czxkz" Jul 30 00:28:54.263290 ip-10-0-142-6 kubenswrapper[2564]: I0730 00:28:54.263101 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4300d011-90a2-4b87-a736-91f40a42f8c3-home\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-czxkz\" (UID: \"4300d011-90a2-4b87-a736-91f40a42f8c3\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-czxkz" Jul 30 00:28:54.263290 ip-10-0-142-6 kubenswrapper[2564]: I0730 00:28:54.263123 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4300d011-90a2-4b87-a736-91f40a42f8c3-tls-certs\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-czxkz\" (UID: \"4300d011-90a2-4b87-a736-91f40a42f8c3\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-czxkz" Jul 30 00:28:54.263290 ip-10-0-142-6 kubenswrapper[2564]: I0730 00:28:54.263141 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4300d011-90a2-4b87-a736-91f40a42f8c3-tmp-dir\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-czxkz\" (UID: \"4300d011-90a2-4b87-a736-91f40a42f8c3\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-czxkz" Jul 30 00:28:54.263532 ip-10-0-142-6 kubenswrapper[2564]: I0730 00:28:54.263503 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4300d011-90a2-4b87-a736-91f40a42f8c3-model-cache\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-czxkz\" (UID: \"4300d011-90a2-4b87-a736-91f40a42f8c3\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-czxkz" Jul 30 00:28:54.263609 ip-10-0-142-6 kubenswrapper[2564]: I0730 00:28:54.263538 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4300d011-90a2-4b87-a736-91f40a42f8c3-home\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-czxkz\" (UID: \"4300d011-90a2-4b87-a736-91f40a42f8c3\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-czxkz" Jul 30 00:28:54.263666 ip-10-0-142-6 kubenswrapper[2564]: I0730 00:28:54.263604 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4300d011-90a2-4b87-a736-91f40a42f8c3-tmp-dir\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-czxkz\" (UID: \"4300d011-90a2-4b87-a736-91f40a42f8c3\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-czxkz" Jul 30 00:28:54.265506 ip-10-0-142-6 kubenswrapper[2564]: I0730 00:28:54.265487 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4300d011-90a2-4b87-a736-91f40a42f8c3-dshm\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-czxkz\" (UID: \"4300d011-90a2-4b87-a736-91f40a42f8c3\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-czxkz" Jul 30 00:28:54.265613 ip-10-0-142-6 kubenswrapper[2564]: I0730 00:28:54.265565 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4300d011-90a2-4b87-a736-91f40a42f8c3-tls-certs\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-czxkz\" (UID: \"4300d011-90a2-4b87-a736-91f40a42f8c3\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-czxkz" Jul 30 00:28:54.270512 ip-10-0-142-6 kubenswrapper[2564]: I0730 00:28:54.270494 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7ctpp\" (UniqueName: \"kubernetes.io/projected/4300d011-90a2-4b87-a736-91f40a42f8c3-kube-api-access-7ctpp\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-czxkz\" (UID: \"4300d011-90a2-4b87-a736-91f40a42f8c3\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-czxkz" Jul 30 00:28:54.416130 ip-10-0-142-6 kubenswrapper[2564]: I0730 00:28:54.416100 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-czxkz" Jul 30 00:28:54.542975 ip-10-0-142-6 kubenswrapper[2564]: I0730 00:28:54.542947 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/facebook-opt-125m-simulated-kserve-74b68447df-czxkz"] Jul 30 00:28:54.544329 ip-10-0-142-6 kubenswrapper[2564]: W0730 00:28:54.544304 2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4300d011_90a2_4b87_a736_91f40a42f8c3.slice/crio-ce1198e56ee0a718cc0285e010e4998a6ece7c13c326c885deef3c7a64d36bf0 WatchSource:0}: Error finding container ce1198e56ee0a718cc0285e010e4998a6ece7c13c326c885deef3c7a64d36bf0: Status 404 returned error can't find the container with id ce1198e56ee0a718cc0285e010e4998a6ece7c13c326c885deef3c7a64d36bf0 Jul 30 00:28:55.098875 ip-10-0-142-6 kubenswrapper[2564]: I0730 00:28:55.098781 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-czxkz" event={"ID":"4300d011-90a2-4b87-a736-91f40a42f8c3","Type":"ContainerStarted","Data":"acbb5a43d7b312cfd3ee986a47f8d5dbc9aa0dcd097ac4cb54fe97e5f8028e64"} Jul 30 00:28:55.099034 ip-10-0-142-6 kubenswrapper[2564]: I0730 00:28:55.098875 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-czxkz" event={"ID":"4300d011-90a2-4b87-a736-91f40a42f8c3","Type":"ContainerStarted","Data":"ce1198e56ee0a718cc0285e010e4998a6ece7c13c326c885deef3c7a64d36bf0"} Jul 30 00:28:55.117472 ip-10-0-142-6 kubenswrapper[2564]: I0730 00:28:55.117422 2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-czxkz" podStartSLOduration=0.947344065 podStartE2EDuration="1.117406796s" totalImagesPullingTime="170.062731ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-30 00:28:54 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-30 00:28:55.115634553 +0000 UTC m=+666.530374487" watchObservedRunningTime="2026-07-30 00:28:55.117406796 +0000 UTC m=+666.532146729" Jul 30 00:28:56.604949 ip-10-0-142-6 kubenswrapper[2564]: I0730 00:28:56.604912 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-nrv2p"] Jul 30 00:28:59.322457 ip-10-0-142-6 kubenswrapper[2564]: I0730 00:28:59.322427 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fgdcgh" Jul 30 00:28:59.322829 ip-10-0-142-6 kubenswrapper[2564]: I0730 00:28:59.322483 2564 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fgdcgh" Jul 30 00:28:59.334647 ip-10-0-142-6 kubenswrapper[2564]: I0730 00:28:59.334621 2564 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fgdcgh" Jul 30 00:29:00.133148 ip-10-0-142-6 kubenswrapper[2564]: I0730 00:29:00.133122 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fgdcgh" Jul 30 00:29:04.417341 ip-10-0-142-6 kubenswrapper[2564]: I0730 00:29:04.417300 2564 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-czxkz" Jul 30 00:29:04.417807 ip-10-0-142-6 kubenswrapper[2564]: I0730 00:29:04.417366 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-czxkz" Jul 30 00:29:04.429842 ip-10-0-142-6 kubenswrapper[2564]: I0730 00:29:04.429815 2564 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-czxkz" Jul 30 00:29:05.158789 ip-10-0-142-6 kubenswrapper[2564]: I0730 00:29:05.158758 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-czxkz" Jul 30 00:29:06.402674 ip-10-0-142-6 kubenswrapper[2564]: I0730 00:29:06.402637 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-nrv2p"] Jul 30 00:30:48.814101 ip-10-0-142-6 kubenswrapper[2564]: I0730 00:30:48.813582 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-nrv2p"] Jul 30 00:30:59.013568 ip-10-0-142-6 kubenswrapper[2564]: I0730 00:30:59.013531 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-nrv2p"] Jul 30 00:31:09.907532 ip-10-0-142-6 kubenswrapper[2564]: I0730 00:31:09.907496 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-nrv2p"] Jul 30 00:32:56.713122 ip-10-0-142-6 kubenswrapper[2564]: I0730 00:32:56.713084 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-nrv2p"] Jul 30 00:33:43.409463 ip-10-0-142-6 kubenswrapper[2564]: I0730 00:33:43.409427 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-nrv2p"] Jul 30 00:33:52.404158 ip-10-0-142-6 kubenswrapper[2564]: I0730 00:33:52.404126 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-nrv2p"] Jul 30 00:34:01.509965 ip-10-0-142-6 kubenswrapper[2564]: I0730 00:34:01.509928 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-nrv2p"] Jul 30 00:34:12.307590 ip-10-0-142-6 kubenswrapper[2564]: I0730 00:34:12.307515 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-nrv2p"] Jul 30 00:34:22.606291 ip-10-0-142-6 kubenswrapper[2564]: I0730 00:34:22.606252 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-nrv2p"] Jul 30 00:34:39.001867 ip-10-0-142-6 kubenswrapper[2564]: I0730 00:34:39.001833 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-nrv2p"] Jul 30 00:34:49.001734 ip-10-0-142-6 kubenswrapper[2564]: I0730 00:34:49.001701 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-nrv2p"] Jul 30 00:34:56.707235 ip-10-0-142-6 kubenswrapper[2564]: I0730 00:34:56.707199 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-nrv2p"] Jul 30 00:35:05.806943 ip-10-0-142-6 kubenswrapper[2564]: I0730 00:35:05.806906 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-nrv2p"] Jul 30 00:35:17.912430 ip-10-0-142-6 kubenswrapper[2564]: I0730 00:35:17.912399 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-nrv2p"] Jul 30 00:35:27.621773 ip-10-0-142-6 kubenswrapper[2564]: I0730 00:35:27.621738 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-nrv2p"] Jul 30 00:35:49.807173 ip-10-0-142-6 kubenswrapper[2564]: I0730 00:35:49.807087 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-nrv2p"] Jul 30 00:35:59.310555 ip-10-0-142-6 kubenswrapper[2564]: I0730 00:35:59.310523 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-nrv2p"] Jul 30 00:36:10.303856 ip-10-0-142-6 kubenswrapper[2564]: I0730 00:36:10.303813 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-nrv2p"] Jul 30 00:36:27.813293 ip-10-0-142-6 kubenswrapper[2564]: I0730 00:36:27.813258 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-nrv2p"] Jul 30 00:36:37.009816 ip-10-0-142-6 kubenswrapper[2564]: I0730 00:36:37.009781 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-nrv2p"] Jul 30 00:36:47.408952 ip-10-0-142-6 kubenswrapper[2564]: I0730 00:36:47.408911 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-nrv2p"] Jul 30 00:36:56.808977 ip-10-0-142-6 kubenswrapper[2564]: I0730 00:36:56.808933 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-nrv2p"] Jul 30 00:37:06.406475 ip-10-0-142-6 kubenswrapper[2564]: I0730 00:37:06.406442 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-nrv2p"] Jul 30 00:38:54.909906 ip-10-0-142-6 kubenswrapper[2564]: I0730 00:38:54.909869 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-nrv2p"] Jul 30 00:39:05.805097 ip-10-0-142-6 kubenswrapper[2564]: I0730 00:39:05.805041 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-nrv2p"] Jul 30 00:39:21.109847 ip-10-0-142-6 kubenswrapper[2564]: I0730 00:39:21.109816 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-nrv2p"] Jul 30 00:39:36.805744 ip-10-0-142-6 kubenswrapper[2564]: I0730 00:39:36.805707 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-nrv2p"] Jul 30 00:39:47.403809 ip-10-0-142-6 kubenswrapper[2564]: I0730 00:39:47.403771 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-nrv2p"] Jul 30 00:39:57.925153 ip-10-0-142-6 kubenswrapper[2564]: I0730 00:39:57.925112 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-nrv2p"] Jul 30 00:40:08.305867 ip-10-0-142-6 kubenswrapper[2564]: I0730 00:40:08.305830 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-nrv2p"] Jul 30 00:40:19.201855 ip-10-0-142-6 kubenswrapper[2564]: I0730 00:40:19.201780 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-nrv2p"] Jul 30 00:40:27.714885 ip-10-0-142-6 kubenswrapper[2564]: I0730 00:40:27.714851 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-nrv2p"] Jul 30 00:40:38.511184 ip-10-0-142-6 kubenswrapper[2564]: I0730 00:40:38.511142 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-nrv2p"] Jul 30 00:40:50.306909 ip-10-0-142-6 kubenswrapper[2564]: I0730 00:40:50.306865 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-nrv2p"] Jul 30 00:40:57.814193 ip-10-0-142-6 kubenswrapper[2564]: I0730 00:40:57.814156 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-nrv2p"] Jul 30 00:41:15.320280 ip-10-0-142-6 kubenswrapper[2564]: I0730 00:41:15.320244 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-nrv2p"] Jul 30 00:41:25.412233 ip-10-0-142-6 kubenswrapper[2564]: I0730 00:41:25.412197 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-nrv2p"] Jul 30 00:41:38.713193 ip-10-0-142-6 kubenswrapper[2564]: I0730 00:41:38.713160 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-nrv2p"] Jul 30 00:41:59.103266 ip-10-0-142-6 kubenswrapper[2564]: I0730 00:41:59.103229 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-nrv2p"] Jul 30 00:42:12.908457 ip-10-0-142-6 kubenswrapper[2564]: I0730 00:42:12.908421 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-nrv2p"] Jul 30 00:42:21.911691 ip-10-0-142-6 kubenswrapper[2564]: I0730 00:42:21.911650 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-nrv2p"] Jul 30 00:42:34.504373 ip-10-0-142-6 kubenswrapper[2564]: I0730 00:42:34.504331 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-nrv2p"] Jul 30 00:42:48.705337 ip-10-0-142-6 kubenswrapper[2564]: I0730 00:42:48.705299 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-nrv2p"] Jul 30 00:42:57.915160 ip-10-0-142-6 kubenswrapper[2564]: I0730 00:42:57.915118 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-nrv2p"] Jul 30 00:43:11.403922 ip-10-0-142-6 kubenswrapper[2564]: I0730 00:43:11.403847 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-nrv2p"] Jul 30 00:43:20.405134 ip-10-0-142-6 kubenswrapper[2564]: I0730 00:43:20.405099 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-nrv2p"] Jul 30 00:43:33.914200 ip-10-0-142-6 kubenswrapper[2564]: I0730 00:43:33.914165 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-nrv2p"] Jul 30 00:43:43.705882 ip-10-0-142-6 kubenswrapper[2564]: I0730 00:43:43.705836 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-nrv2p"] Jul 30 00:43:53.212600 ip-10-0-142-6 kubenswrapper[2564]: I0730 00:43:53.212564 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-nrv2p"] Jul 30 00:44:02.816264 ip-10-0-142-6 kubenswrapper[2564]: I0730 00:44:02.816230 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-nrv2p"] Jul 30 00:44:15.907461 ip-10-0-142-6 kubenswrapper[2564]: I0730 00:44:15.907430 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-nrv2p"] Jul 30 00:44:24.213426 ip-10-0-142-6 kubenswrapper[2564]: I0730 00:44:24.213390 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-nrv2p"] Jul 30 00:44:36.009339 ip-10-0-142-6 kubenswrapper[2564]: I0730 00:44:36.009302 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-nrv2p"] Jul 30 00:44:48.406129 ip-10-0-142-6 kubenswrapper[2564]: I0730 00:44:48.406013 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-nrv2p"] Jul 30 00:44:57.607153 ip-10-0-142-6 kubenswrapper[2564]: I0730 00:44:57.607120 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-nrv2p"] Jul 30 00:45:05.707980 ip-10-0-142-6 kubenswrapper[2564]: I0730 00:45:05.707943 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-nrv2p"] Jul 30 00:45:18.011303 ip-10-0-142-6 kubenswrapper[2564]: I0730 00:45:18.011266 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-nrv2p"] Jul 30 00:45:31.708899 ip-10-0-142-6 kubenswrapper[2564]: I0730 00:45:31.708863 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-nrv2p"] Jul 30 00:45:44.108340 ip-10-0-142-6 kubenswrapper[2564]: I0730 00:45:44.108305 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-nrv2p"] Jul 30 00:46:14.109952 ip-10-0-142-6 kubenswrapper[2564]: I0730 00:46:14.109917 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-nrv2p"] Jul 30 00:46:25.905164 ip-10-0-142-6 kubenswrapper[2564]: I0730 00:46:25.905126 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-nrv2p"] Jul 30 00:46:48.916137 ip-10-0-142-6 kubenswrapper[2564]: I0730 00:46:48.916099 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-nrv2p"] Jul 30 00:47:00.601838 ip-10-0-142-6 kubenswrapper[2564]: I0730 00:47:00.601800 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-nrv2p"] Jul 30 00:47:24.208271 ip-10-0-142-6 kubenswrapper[2564]: I0730 00:47:24.208236 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-nrv2p"] Jul 30 00:47:35.708879 ip-10-0-142-6 kubenswrapper[2564]: I0730 00:47:35.708845 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-nrv2p"] Jul 30 00:48:00.005547 ip-10-0-142-6 kubenswrapper[2564]: I0730 00:48:00.005508 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-nrv2p"] Jul 30 00:48:11.808097 ip-10-0-142-6 kubenswrapper[2564]: I0730 00:48:11.808048 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-nrv2p"] Jul 30 00:49:20.610927 ip-10-0-142-6 kubenswrapper[2564]: I0730 00:49:20.610833 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-nrv2p"] Jul 30 00:49:31.813182 ip-10-0-142-6 kubenswrapper[2564]: I0730 00:49:31.813143 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-nrv2p"] Jul 30 00:49:41.808312 ip-10-0-142-6 kubenswrapper[2564]: I0730 00:49:41.808276 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-nrv2p"] Jul 30 00:49:52.702450 ip-10-0-142-6 kubenswrapper[2564]: I0730 00:49:52.702413 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-nrv2p"] Jul 30 00:50:32.803793 ip-10-0-142-6 kubenswrapper[2564]: I0730 00:50:32.803762 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-nrv2p"] Jul 30 00:51:04.900157 ip-10-0-142-6 kubenswrapper[2564]: I0730 00:51:04.900109 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-nrv2p"] Jul 30 00:51:29.013341 ip-10-0-142-6 kubenswrapper[2564]: I0730 00:51:29.013306 2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-mt-80d5102b/e2e-lifecycle-model-80d5102b-kserve-86b658789f-j6xvr"] Jul 30 00:51:29.016964 ip-10-0-142-6 kubenswrapper[2564]: I0730 00:51:29.016943 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-mt-80d5102b/e2e-lifecycle-model-80d5102b-kserve-86b658789f-j6xvr" Jul 30 00:51:29.020261 ip-10-0-142-6 kubenswrapper[2564]: I0730 00:51:29.020236 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-mt-80d5102b\"/\"default-dockercfg-dqzg4\"" Jul 30 00:51:29.020261 ip-10-0-142-6 kubenswrapper[2564]: I0730 00:51:29.020249 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-mt-80d5102b\"/\"kube-root-ca.crt\"" Jul 30 00:51:29.020452 ip-10-0-142-6 kubenswrapper[2564]: I0730 00:51:29.020236 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-mt-80d5102b\"/\"e2e-lifecycle-model-80d5102b-kserve-self-signed-certs\"" Jul 30 00:51:29.020452 ip-10-0-142-6 kubenswrapper[2564]: I0730 00:51:29.020364 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-mt-80d5102b\"/\"openshift-service-ca.crt\"" Jul 30 00:51:29.025632 ip-10-0-142-6 kubenswrapper[2564]: I0730 00:51:29.025611 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-mt-80d5102b/e2e-lifecycle-model-80d5102b-kserve-86b658789f-j6xvr"] Jul 30 00:51:29.166771 ip-10-0-142-6 kubenswrapper[2564]: I0730 00:51:29.166733 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1e162c1d-ee12-4111-8b21-5ba59b9e0388-dshm\") pod \"e2e-lifecycle-model-80d5102b-kserve-86b658789f-j6xvr\" (UID: \"1e162c1d-ee12-4111-8b21-5ba59b9e0388\") " pod="ai-tenant-e2e-mt-80d5102b/e2e-lifecycle-model-80d5102b-kserve-86b658789f-j6xvr" Jul 30 00:51:29.166954 ip-10-0-142-6 kubenswrapper[2564]: I0730 00:51:29.166777 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1e162c1d-ee12-4111-8b21-5ba59b9e0388-model-cache\") pod \"e2e-lifecycle-model-80d5102b-kserve-86b658789f-j6xvr\" (UID: \"1e162c1d-ee12-4111-8b21-5ba59b9e0388\") " pod="ai-tenant-e2e-mt-80d5102b/e2e-lifecycle-model-80d5102b-kserve-86b658789f-j6xvr" Jul 30 00:51:29.166954 ip-10-0-142-6 kubenswrapper[2564]: I0730 00:51:29.166857 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1e162c1d-ee12-4111-8b21-5ba59b9e0388-tmp-dir\") pod \"e2e-lifecycle-model-80d5102b-kserve-86b658789f-j6xvr\" (UID: \"1e162c1d-ee12-4111-8b21-5ba59b9e0388\") " pod="ai-tenant-e2e-mt-80d5102b/e2e-lifecycle-model-80d5102b-kserve-86b658789f-j6xvr" Jul 30 00:51:29.166954 ip-10-0-142-6 kubenswrapper[2564]: I0730 00:51:29.166915 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ljsrx\" (UniqueName: \"kubernetes.io/projected/1e162c1d-ee12-4111-8b21-5ba59b9e0388-kube-api-access-ljsrx\") pod \"e2e-lifecycle-model-80d5102b-kserve-86b658789f-j6xvr\" (UID: \"1e162c1d-ee12-4111-8b21-5ba59b9e0388\") " pod="ai-tenant-e2e-mt-80d5102b/e2e-lifecycle-model-80d5102b-kserve-86b658789f-j6xvr" Jul 30 00:51:29.166954 ip-10-0-142-6 kubenswrapper[2564]: I0730 00:51:29.166950 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1e162c1d-ee12-4111-8b21-5ba59b9e0388-tls-certs\") pod \"e2e-lifecycle-model-80d5102b-kserve-86b658789f-j6xvr\" (UID: \"1e162c1d-ee12-4111-8b21-5ba59b9e0388\") " pod="ai-tenant-e2e-mt-80d5102b/e2e-lifecycle-model-80d5102b-kserve-86b658789f-j6xvr" Jul 30 00:51:29.167137 ip-10-0-142-6 kubenswrapper[2564]: I0730 00:51:29.166982 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1e162c1d-ee12-4111-8b21-5ba59b9e0388-home\") pod \"e2e-lifecycle-model-80d5102b-kserve-86b658789f-j6xvr\" (UID: \"1e162c1d-ee12-4111-8b21-5ba59b9e0388\") " pod="ai-tenant-e2e-mt-80d5102b/e2e-lifecycle-model-80d5102b-kserve-86b658789f-j6xvr" Jul 30 00:51:29.268009 ip-10-0-142-6 kubenswrapper[2564]: I0730 00:51:29.267910 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1e162c1d-ee12-4111-8b21-5ba59b9e0388-tls-certs\") pod \"e2e-lifecycle-model-80d5102b-kserve-86b658789f-j6xvr\" (UID: \"1e162c1d-ee12-4111-8b21-5ba59b9e0388\") " pod="ai-tenant-e2e-mt-80d5102b/e2e-lifecycle-model-80d5102b-kserve-86b658789f-j6xvr" Jul 30 00:51:29.268009 ip-10-0-142-6 kubenswrapper[2564]: I0730 00:51:29.267955 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1e162c1d-ee12-4111-8b21-5ba59b9e0388-home\") pod \"e2e-lifecycle-model-80d5102b-kserve-86b658789f-j6xvr\" (UID: \"1e162c1d-ee12-4111-8b21-5ba59b9e0388\") " pod="ai-tenant-e2e-mt-80d5102b/e2e-lifecycle-model-80d5102b-kserve-86b658789f-j6xvr" Jul 30 00:51:29.268009 ip-10-0-142-6 kubenswrapper[2564]: I0730 00:51:29.267990 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1e162c1d-ee12-4111-8b21-5ba59b9e0388-dshm\") pod \"e2e-lifecycle-model-80d5102b-kserve-86b658789f-j6xvr\" (UID: \"1e162c1d-ee12-4111-8b21-5ba59b9e0388\") " pod="ai-tenant-e2e-mt-80d5102b/e2e-lifecycle-model-80d5102b-kserve-86b658789f-j6xvr" Jul 30 00:51:29.268329 ip-10-0-142-6 kubenswrapper[2564]: I0730 00:51:29.268015 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1e162c1d-ee12-4111-8b21-5ba59b9e0388-model-cache\") pod \"e2e-lifecycle-model-80d5102b-kserve-86b658789f-j6xvr\" (UID: \"1e162c1d-ee12-4111-8b21-5ba59b9e0388\") " pod="ai-tenant-e2e-mt-80d5102b/e2e-lifecycle-model-80d5102b-kserve-86b658789f-j6xvr" Jul 30 00:51:29.268329 ip-10-0-142-6 kubenswrapper[2564]: I0730 00:51:29.268047 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1e162c1d-ee12-4111-8b21-5ba59b9e0388-tmp-dir\") pod \"e2e-lifecycle-model-80d5102b-kserve-86b658789f-j6xvr\" (UID: \"1e162c1d-ee12-4111-8b21-5ba59b9e0388\") " pod="ai-tenant-e2e-mt-80d5102b/e2e-lifecycle-model-80d5102b-kserve-86b658789f-j6xvr" Jul 30 00:51:29.268329 ip-10-0-142-6 kubenswrapper[2564]: I0730 00:51:29.268110 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ljsrx\" (UniqueName: \"kubernetes.io/projected/1e162c1d-ee12-4111-8b21-5ba59b9e0388-kube-api-access-ljsrx\") pod \"e2e-lifecycle-model-80d5102b-kserve-86b658789f-j6xvr\" (UID: \"1e162c1d-ee12-4111-8b21-5ba59b9e0388\") " pod="ai-tenant-e2e-mt-80d5102b/e2e-lifecycle-model-80d5102b-kserve-86b658789f-j6xvr" Jul 30 00:51:29.268481 ip-10-0-142-6 kubenswrapper[2564]: I0730 00:51:29.268419 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1e162c1d-ee12-4111-8b21-5ba59b9e0388-model-cache\") pod \"e2e-lifecycle-model-80d5102b-kserve-86b658789f-j6xvr\" (UID: \"1e162c1d-ee12-4111-8b21-5ba59b9e0388\") " pod="ai-tenant-e2e-mt-80d5102b/e2e-lifecycle-model-80d5102b-kserve-86b658789f-j6xvr" Jul 30 00:51:29.268481 ip-10-0-142-6 kubenswrapper[2564]: I0730 00:51:29.268430 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1e162c1d-ee12-4111-8b21-5ba59b9e0388-home\") pod \"e2e-lifecycle-model-80d5102b-kserve-86b658789f-j6xvr\" (UID: \"1e162c1d-ee12-4111-8b21-5ba59b9e0388\") " pod="ai-tenant-e2e-mt-80d5102b/e2e-lifecycle-model-80d5102b-kserve-86b658789f-j6xvr" Jul 30 00:51:29.268552 ip-10-0-142-6 kubenswrapper[2564]: I0730 00:51:29.268490 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1e162c1d-ee12-4111-8b21-5ba59b9e0388-tmp-dir\") pod \"e2e-lifecycle-model-80d5102b-kserve-86b658789f-j6xvr\" (UID: \"1e162c1d-ee12-4111-8b21-5ba59b9e0388\") " pod="ai-tenant-e2e-mt-80d5102b/e2e-lifecycle-model-80d5102b-kserve-86b658789f-j6xvr" Jul 30 00:51:29.270331 ip-10-0-142-6 kubenswrapper[2564]: I0730 00:51:29.270310 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1e162c1d-ee12-4111-8b21-5ba59b9e0388-dshm\") pod \"e2e-lifecycle-model-80d5102b-kserve-86b658789f-j6xvr\" (UID: \"1e162c1d-ee12-4111-8b21-5ba59b9e0388\") " pod="ai-tenant-e2e-mt-80d5102b/e2e-lifecycle-model-80d5102b-kserve-86b658789f-j6xvr" Jul 30 00:51:29.270678 ip-10-0-142-6 kubenswrapper[2564]: I0730 00:51:29.270661 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1e162c1d-ee12-4111-8b21-5ba59b9e0388-tls-certs\") pod \"e2e-lifecycle-model-80d5102b-kserve-86b658789f-j6xvr\" (UID: \"1e162c1d-ee12-4111-8b21-5ba59b9e0388\") " pod="ai-tenant-e2e-mt-80d5102b/e2e-lifecycle-model-80d5102b-kserve-86b658789f-j6xvr" Jul 30 00:51:29.275233 ip-10-0-142-6 kubenswrapper[2564]: I0730 00:51:29.275208 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ljsrx\" (UniqueName: \"kubernetes.io/projected/1e162c1d-ee12-4111-8b21-5ba59b9e0388-kube-api-access-ljsrx\") pod \"e2e-lifecycle-model-80d5102b-kserve-86b658789f-j6xvr\" (UID: \"1e162c1d-ee12-4111-8b21-5ba59b9e0388\") " pod="ai-tenant-e2e-mt-80d5102b/e2e-lifecycle-model-80d5102b-kserve-86b658789f-j6xvr" Jul 30 00:51:29.327300 ip-10-0-142-6 kubenswrapper[2564]: I0730 00:51:29.327274 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-mt-80d5102b/e2e-lifecycle-model-80d5102b-kserve-86b658789f-j6xvr" Jul 30 00:51:29.459683 ip-10-0-142-6 kubenswrapper[2564]: I0730 00:51:29.459606 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-mt-80d5102b/e2e-lifecycle-model-80d5102b-kserve-86b658789f-j6xvr"] Jul 30 00:51:29.462466 ip-10-0-142-6 kubenswrapper[2564]: W0730 00:51:29.462437 2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod1e162c1d_ee12_4111_8b21_5ba59b9e0388.slice/crio-ec669ededfdd32efc812671021ef9d1101d929db4e4a9e16d8efea5efa7cc840 WatchSource:0}: Error finding container ec669ededfdd32efc812671021ef9d1101d929db4e4a9e16d8efea5efa7cc840: Status 404 returned error can't find the container with id ec669ededfdd32efc812671021ef9d1101d929db4e4a9e16d8efea5efa7cc840 Jul 30 00:51:29.480020 ip-10-0-142-6 kubenswrapper[2564]: I0730 00:51:29.479991 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-mt-80d5102b/e2e-lifecycle-model-80d5102b-kserve-86b658789f-j6xvr" event={"ID":"1e162c1d-ee12-4111-8b21-5ba59b9e0388","Type":"ContainerStarted","Data":"ec669ededfdd32efc812671021ef9d1101d929db4e4a9e16d8efea5efa7cc840"} Jul 30 00:51:30.486138 ip-10-0-142-6 kubenswrapper[2564]: I0730 00:51:30.486099 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-mt-80d5102b/e2e-lifecycle-model-80d5102b-kserve-86b658789f-j6xvr" event={"ID":"1e162c1d-ee12-4111-8b21-5ba59b9e0388","Type":"ContainerStarted","Data":"50d194430d18ebb6c734753f23fad021960303f4698ed4ab6c7f5ce927e80855"} Jul 30 00:51:30.504127 ip-10-0-142-6 kubenswrapper[2564]: I0730 00:51:30.504061 2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-mt-80d5102b/e2e-lifecycle-model-80d5102b-kserve-86b658789f-j6xvr" podStartSLOduration=2.5040434339999997 podStartE2EDuration="2.504043434s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-30 00:51:28 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-30 00:51:30.50201287 +0000 UTC m=+2021.916752794" watchObservedRunningTime="2026-07-30 00:51:30.504043434 +0000 UTC m=+2021.918783366" Jul 30 00:51:32.514176 ip-10-0-142-6 kubenswrapper[2564]: I0730 00:51:32.514140 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-nrv2p"] Jul 30 00:51:39.328130 ip-10-0-142-6 kubenswrapper[2564]: I0730 00:51:39.328093 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-mt-80d5102b/e2e-lifecycle-model-80d5102b-kserve-86b658789f-j6xvr" Jul 30 00:51:39.328733 ip-10-0-142-6 kubenswrapper[2564]: I0730 00:51:39.328230 2564 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-mt-80d5102b/e2e-lifecycle-model-80d5102b-kserve-86b658789f-j6xvr" Jul 30 00:51:39.340665 ip-10-0-142-6 kubenswrapper[2564]: I0730 00:51:39.340639 2564 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-mt-80d5102b/e2e-lifecycle-model-80d5102b-kserve-86b658789f-j6xvr" Jul 30 00:51:39.547090 ip-10-0-142-6 kubenswrapper[2564]: I0730 00:51:39.547050 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-mt-80d5102b/e2e-lifecycle-model-80d5102b-kserve-86b658789f-j6xvr" Jul 30 00:51:47.909979 ip-10-0-142-6 kubenswrapper[2564]: I0730 00:51:47.909942 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-nrv2p"] Jul 30 00:52:06.311243 ip-10-0-142-6 kubenswrapper[2564]: I0730 00:52:06.311204 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-nrv2p"] Jul 30 00:52:16.074330 ip-10-0-142-6 kubenswrapper[2564]: I0730 00:52:16.074236 2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-mt-80d5102b/e2e-lifecycle-model-80d5102b-kserve-86b658789f-j6xvr"] Jul 30 00:52:17.008737 ip-10-0-142-6 kubenswrapper[2564]: I0730 00:52:17.008702 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-nrv2p"] Jul 30 00:52:31.093464 ip-10-0-142-6 kubenswrapper[2564]: I0730 00:52:31.093396 2564 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-mt-80d5102b/e2e-lifecycle-model-80d5102b-kserve-86b658789f-j6xvr" podUID="1e162c1d-ee12-4111-8b21-5ba59b9e0388" containerName="main" containerID="cri-o://50d194430d18ebb6c734753f23fad021960303f4698ed4ab6c7f5ce927e80855" gracePeriod=45 Jul 30 00:52:31.345593 ip-10-0-142-6 kubenswrapper[2564]: I0730 00:52:31.345539 2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-mt-80d5102b/e2e-lifecycle-model-80d5102b-kserve-86b658789f-j6xvr" Jul 30 00:52:31.495945 ip-10-0-142-6 kubenswrapper[2564]: I0730 00:52:31.495906 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1e162c1d-ee12-4111-8b21-5ba59b9e0388-tmp-dir\") pod \"1e162c1d-ee12-4111-8b21-5ba59b9e0388\" (UID: \"1e162c1d-ee12-4111-8b21-5ba59b9e0388\") " Jul 30 00:52:31.495945 ip-10-0-142-6 kubenswrapper[2564]: I0730 00:52:31.495956 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1e162c1d-ee12-4111-8b21-5ba59b9e0388-tls-certs\") pod \"1e162c1d-ee12-4111-8b21-5ba59b9e0388\" (UID: \"1e162c1d-ee12-4111-8b21-5ba59b9e0388\") " Jul 30 00:52:31.496233 ip-10-0-142-6 kubenswrapper[2564]: I0730 00:52:31.495990 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1e162c1d-ee12-4111-8b21-5ba59b9e0388-home\") pod \"1e162c1d-ee12-4111-8b21-5ba59b9e0388\" (UID: \"1e162c1d-ee12-4111-8b21-5ba59b9e0388\") " Jul 30 00:52:31.496233 ip-10-0-142-6 kubenswrapper[2564]: I0730 00:52:31.496046 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1e162c1d-ee12-4111-8b21-5ba59b9e0388-model-cache\") pod \"1e162c1d-ee12-4111-8b21-5ba59b9e0388\" (UID: \"1e162c1d-ee12-4111-8b21-5ba59b9e0388\") " Jul 30 00:52:31.496233 ip-10-0-142-6 kubenswrapper[2564]: I0730 00:52:31.496133 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1e162c1d-ee12-4111-8b21-5ba59b9e0388-dshm\") pod \"1e162c1d-ee12-4111-8b21-5ba59b9e0388\" (UID: \"1e162c1d-ee12-4111-8b21-5ba59b9e0388\") " Jul 30 00:52:31.496233 ip-10-0-142-6 kubenswrapper[2564]: I0730 00:52:31.496166 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-ljsrx\" (UniqueName: \"kubernetes.io/projected/1e162c1d-ee12-4111-8b21-5ba59b9e0388-kube-api-access-ljsrx\") pod \"1e162c1d-ee12-4111-8b21-5ba59b9e0388\" (UID: \"1e162c1d-ee12-4111-8b21-5ba59b9e0388\") " Jul 30 00:52:31.496413 ip-10-0-142-6 kubenswrapper[2564]: I0730 00:52:31.496263 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1e162c1d-ee12-4111-8b21-5ba59b9e0388-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "1e162c1d-ee12-4111-8b21-5ba59b9e0388" (UID: "1e162c1d-ee12-4111-8b21-5ba59b9e0388"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 30 00:52:31.496413 ip-10-0-142-6 kubenswrapper[2564]: I0730 00:52:31.496342 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1e162c1d-ee12-4111-8b21-5ba59b9e0388-home" (OuterVolumeSpecName: "home") pod "1e162c1d-ee12-4111-8b21-5ba59b9e0388" (UID: "1e162c1d-ee12-4111-8b21-5ba59b9e0388"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 30 00:52:31.496531 ip-10-0-142-6 kubenswrapper[2564]: I0730 00:52:31.496507 2564 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1e162c1d-ee12-4111-8b21-5ba59b9e0388-tmp-dir\") on node \"ip-10-0-142-6.ec2.internal\" DevicePath \"\"" Jul 30 00:52:31.496596 ip-10-0-142-6 kubenswrapper[2564]: I0730 00:52:31.496539 2564 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1e162c1d-ee12-4111-8b21-5ba59b9e0388-home\") on node \"ip-10-0-142-6.ec2.internal\" DevicePath \"\"" Jul 30 00:52:31.496646 ip-10-0-142-6 kubenswrapper[2564]: I0730 00:52:31.496584 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1e162c1d-ee12-4111-8b21-5ba59b9e0388-model-cache" (OuterVolumeSpecName: "model-cache") pod "1e162c1d-ee12-4111-8b21-5ba59b9e0388" (UID: "1e162c1d-ee12-4111-8b21-5ba59b9e0388"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 30 00:52:31.498319 ip-10-0-142-6 kubenswrapper[2564]: I0730 00:52:31.498289 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1e162c1d-ee12-4111-8b21-5ba59b9e0388-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "1e162c1d-ee12-4111-8b21-5ba59b9e0388" (UID: "1e162c1d-ee12-4111-8b21-5ba59b9e0388"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 30 00:52:31.498319 ip-10-0-142-6 kubenswrapper[2564]: I0730 00:52:31.498301 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1e162c1d-ee12-4111-8b21-5ba59b9e0388-dshm" (OuterVolumeSpecName: "dshm") pod "1e162c1d-ee12-4111-8b21-5ba59b9e0388" (UID: "1e162c1d-ee12-4111-8b21-5ba59b9e0388"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 30 00:52:31.498472 ip-10-0-142-6 kubenswrapper[2564]: I0730 00:52:31.498396 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1e162c1d-ee12-4111-8b21-5ba59b9e0388-kube-api-access-ljsrx" (OuterVolumeSpecName: "kube-api-access-ljsrx") pod "1e162c1d-ee12-4111-8b21-5ba59b9e0388" (UID: "1e162c1d-ee12-4111-8b21-5ba59b9e0388"). InnerVolumeSpecName "kube-api-access-ljsrx". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 30 00:52:31.597799 ip-10-0-142-6 kubenswrapper[2564]: I0730 00:52:31.597709 2564 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1e162c1d-ee12-4111-8b21-5ba59b9e0388-model-cache\") on node \"ip-10-0-142-6.ec2.internal\" DevicePath \"\"" Jul 30 00:52:31.597799 ip-10-0-142-6 kubenswrapper[2564]: I0730 00:52:31.597736 2564 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1e162c1d-ee12-4111-8b21-5ba59b9e0388-dshm\") on node \"ip-10-0-142-6.ec2.internal\" DevicePath \"\"" Jul 30 00:52:31.597799 ip-10-0-142-6 kubenswrapper[2564]: I0730 00:52:31.597750 2564 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ljsrx\" (UniqueName: \"kubernetes.io/projected/1e162c1d-ee12-4111-8b21-5ba59b9e0388-kube-api-access-ljsrx\") on node \"ip-10-0-142-6.ec2.internal\" DevicePath \"\"" Jul 30 00:52:31.597799 ip-10-0-142-6 kubenswrapper[2564]: I0730 00:52:31.597760 2564 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1e162c1d-ee12-4111-8b21-5ba59b9e0388-tls-certs\") on node \"ip-10-0-142-6.ec2.internal\" DevicePath \"\"" Jul 30 00:52:31.787845 ip-10-0-142-6 kubenswrapper[2564]: I0730 00:52:31.787807 2564 generic.go:356] "Generic (PLEG): container finished" podID="1e162c1d-ee12-4111-8b21-5ba59b9e0388" containerID="50d194430d18ebb6c734753f23fad021960303f4698ed4ab6c7f5ce927e80855" exitCode=0 Jul 30 00:52:31.787845 ip-10-0-142-6 kubenswrapper[2564]: I0730 00:52:31.787848 2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-mt-80d5102b/e2e-lifecycle-model-80d5102b-kserve-86b658789f-j6xvr" Jul 30 00:52:31.788086 ip-10-0-142-6 kubenswrapper[2564]: I0730 00:52:31.787889 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-mt-80d5102b/e2e-lifecycle-model-80d5102b-kserve-86b658789f-j6xvr" event={"ID":"1e162c1d-ee12-4111-8b21-5ba59b9e0388","Type":"ContainerDied","Data":"50d194430d18ebb6c734753f23fad021960303f4698ed4ab6c7f5ce927e80855"} Jul 30 00:52:31.788086 ip-10-0-142-6 kubenswrapper[2564]: I0730 00:52:31.787931 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-mt-80d5102b/e2e-lifecycle-model-80d5102b-kserve-86b658789f-j6xvr" event={"ID":"1e162c1d-ee12-4111-8b21-5ba59b9e0388","Type":"ContainerDied","Data":"ec669ededfdd32efc812671021ef9d1101d929db4e4a9e16d8efea5efa7cc840"} Jul 30 00:52:31.788086 ip-10-0-142-6 kubenswrapper[2564]: I0730 00:52:31.787948 2564 scope.go:117] "RemoveContainer" containerID="50d194430d18ebb6c734753f23fad021960303f4698ed4ab6c7f5ce927e80855" Jul 30 00:52:31.797627 ip-10-0-142-6 kubenswrapper[2564]: I0730 00:52:31.797607 2564 scope.go:117] "RemoveContainer" containerID="50d194430d18ebb6c734753f23fad021960303f4698ed4ab6c7f5ce927e80855" Jul 30 00:52:31.797884 ip-10-0-142-6 kubenswrapper[2564]: E0730 00:52:31.797863 2564 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"50d194430d18ebb6c734753f23fad021960303f4698ed4ab6c7f5ce927e80855\": container with ID starting with 50d194430d18ebb6c734753f23fad021960303f4698ed4ab6c7f5ce927e80855 not found: ID does not exist" containerID="50d194430d18ebb6c734753f23fad021960303f4698ed4ab6c7f5ce927e80855" Jul 30 00:52:31.797952 ip-10-0-142-6 kubenswrapper[2564]: I0730 00:52:31.797899 2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"50d194430d18ebb6c734753f23fad021960303f4698ed4ab6c7f5ce927e80855"} err="failed to get container status \"50d194430d18ebb6c734753f23fad021960303f4698ed4ab6c7f5ce927e80855\": rpc error: code = NotFound desc = could not find container \"50d194430d18ebb6c734753f23fad021960303f4698ed4ab6c7f5ce927e80855\": container with ID starting with 50d194430d18ebb6c734753f23fad021960303f4698ed4ab6c7f5ce927e80855 not found: ID does not exist" Jul 30 00:52:31.809193 ip-10-0-142-6 kubenswrapper[2564]: I0730 00:52:31.809167 2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-mt-80d5102b/e2e-lifecycle-model-80d5102b-kserve-86b658789f-j6xvr"] Jul 30 00:52:31.811175 ip-10-0-142-6 kubenswrapper[2564]: I0730 00:52:31.811153 2564 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-mt-80d5102b/e2e-lifecycle-model-80d5102b-kserve-86b658789f-j6xvr"] Jul 30 00:52:33.141933 ip-10-0-142-6 kubenswrapper[2564]: I0730 00:52:33.141890 2564 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="1e162c1d-ee12-4111-8b21-5ba59b9e0388" path="/var/lib/kubelet/pods/1e162c1d-ee12-4111-8b21-5ba59b9e0388/volumes" Jul 30 00:53:08.319683 ip-10-0-142-6 kubenswrapper[2564]: I0730 00:53:08.319652 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-nrv2p"] Jul 30 00:53:21.309999 ip-10-0-142-6 kubenswrapper[2564]: I0730 00:53:21.309961 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-nrv2p"] Jul 30 00:54:08.610675 ip-10-0-142-6 kubenswrapper[2564]: I0730 00:54:08.610636 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-nrv2p"] Jul 30 00:54:37.806698 ip-10-0-142-6 kubenswrapper[2564]: I0730 00:54:37.806662 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-nrv2p"] Jul 30 00:55:01.620504 ip-10-0-142-6 kubenswrapper[2564]: I0730 00:55:01.620467 2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-inf-a-2239f6/test-model-2239f6-kserve-5588cfd8df-ccsnd"] Jul 30 00:55:01.620910 ip-10-0-142-6 kubenswrapper[2564]: I0730 00:55:01.620793 2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1e162c1d-ee12-4111-8b21-5ba59b9e0388" containerName="main" Jul 30 00:55:01.620910 ip-10-0-142-6 kubenswrapper[2564]: I0730 00:55:01.620805 2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="1e162c1d-ee12-4111-8b21-5ba59b9e0388" containerName="main" Jul 30 00:55:01.620910 ip-10-0-142-6 kubenswrapper[2564]: I0730 00:55:01.620860 2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="1e162c1d-ee12-4111-8b21-5ba59b9e0388" containerName="main" Jul 30 00:55:01.623688 ip-10-0-142-6 kubenswrapper[2564]: I0730 00:55:01.623674 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-2239f6/test-model-2239f6-kserve-5588cfd8df-ccsnd" Jul 30 00:55:01.627042 ip-10-0-142-6 kubenswrapper[2564]: I0730 00:55:01.627015 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-a-2239f6\"/\"default-dockercfg-t8dnb\"" Jul 30 00:55:01.627202 ip-10-0-142-6 kubenswrapper[2564]: I0730 00:55:01.627060 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-a-2239f6\"/\"openshift-service-ca.crt\"" Jul 30 00:55:01.627202 ip-10-0-142-6 kubenswrapper[2564]: I0730 00:55:01.627063 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-a-2239f6\"/\"kube-root-ca.crt\"" Jul 30 00:55:01.627202 ip-10-0-142-6 kubenswrapper[2564]: I0730 00:55:01.627063 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-a-2239f6\"/\"test-model-2239f6-kserve-self-signed-certs\"" Jul 30 00:55:01.631863 ip-10-0-142-6 kubenswrapper[2564]: I0730 00:55:01.631842 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-a-2239f6/test-model-2239f6-kserve-5588cfd8df-ccsnd"] Jul 30 00:55:01.650735 ip-10-0-142-6 kubenswrapper[2564]: I0730 00:55:01.650697 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c4da3216-9857-497f-a5ca-0e193ba681a5-home\") pod \"test-model-2239f6-kserve-5588cfd8df-ccsnd\" (UID: \"c4da3216-9857-497f-a5ca-0e193ba681a5\") " pod="ai-tenant-e2e-inf-a-2239f6/test-model-2239f6-kserve-5588cfd8df-ccsnd" Jul 30 00:55:01.650735 ip-10-0-142-6 kubenswrapper[2564]: I0730 00:55:01.650732 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c4da3216-9857-497f-a5ca-0e193ba681a5-dshm\") pod \"test-model-2239f6-kserve-5588cfd8df-ccsnd\" (UID: \"c4da3216-9857-497f-a5ca-0e193ba681a5\") " pod="ai-tenant-e2e-inf-a-2239f6/test-model-2239f6-kserve-5588cfd8df-ccsnd" Jul 30 00:55:01.650983 ip-10-0-142-6 kubenswrapper[2564]: I0730 00:55:01.650760 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c4da3216-9857-497f-a5ca-0e193ba681a5-model-cache\") pod \"test-model-2239f6-kserve-5588cfd8df-ccsnd\" (UID: \"c4da3216-9857-497f-a5ca-0e193ba681a5\") " pod="ai-tenant-e2e-inf-a-2239f6/test-model-2239f6-kserve-5588cfd8df-ccsnd" Jul 30 00:55:01.650983 ip-10-0-142-6 kubenswrapper[2564]: I0730 00:55:01.650785 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c4da3216-9857-497f-a5ca-0e193ba681a5-tls-certs\") pod \"test-model-2239f6-kserve-5588cfd8df-ccsnd\" (UID: \"c4da3216-9857-497f-a5ca-0e193ba681a5\") " pod="ai-tenant-e2e-inf-a-2239f6/test-model-2239f6-kserve-5588cfd8df-ccsnd" Jul 30 00:55:01.650983 ip-10-0-142-6 kubenswrapper[2564]: I0730 00:55:01.650900 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c4da3216-9857-497f-a5ca-0e193ba681a5-tmp-dir\") pod \"test-model-2239f6-kserve-5588cfd8df-ccsnd\" (UID: \"c4da3216-9857-497f-a5ca-0e193ba681a5\") " pod="ai-tenant-e2e-inf-a-2239f6/test-model-2239f6-kserve-5588cfd8df-ccsnd" Jul 30 00:55:01.651132 ip-10-0-142-6 kubenswrapper[2564]: I0730 00:55:01.650993 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9zhrr\" (UniqueName: \"kubernetes.io/projected/c4da3216-9857-497f-a5ca-0e193ba681a5-kube-api-access-9zhrr\") pod \"test-model-2239f6-kserve-5588cfd8df-ccsnd\" (UID: \"c4da3216-9857-497f-a5ca-0e193ba681a5\") " pod="ai-tenant-e2e-inf-a-2239f6/test-model-2239f6-kserve-5588cfd8df-ccsnd" Jul 30 00:55:01.752142 ip-10-0-142-6 kubenswrapper[2564]: I0730 00:55:01.752110 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c4da3216-9857-497f-a5ca-0e193ba681a5-tmp-dir\") pod \"test-model-2239f6-kserve-5588cfd8df-ccsnd\" (UID: \"c4da3216-9857-497f-a5ca-0e193ba681a5\") " pod="ai-tenant-e2e-inf-a-2239f6/test-model-2239f6-kserve-5588cfd8df-ccsnd" Jul 30 00:55:01.752349 ip-10-0-142-6 kubenswrapper[2564]: I0730 00:55:01.752184 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9zhrr\" (UniqueName: \"kubernetes.io/projected/c4da3216-9857-497f-a5ca-0e193ba681a5-kube-api-access-9zhrr\") pod \"test-model-2239f6-kserve-5588cfd8df-ccsnd\" (UID: \"c4da3216-9857-497f-a5ca-0e193ba681a5\") " pod="ai-tenant-e2e-inf-a-2239f6/test-model-2239f6-kserve-5588cfd8df-ccsnd" Jul 30 00:55:01.752349 ip-10-0-142-6 kubenswrapper[2564]: I0730 00:55:01.752213 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c4da3216-9857-497f-a5ca-0e193ba681a5-home\") pod \"test-model-2239f6-kserve-5588cfd8df-ccsnd\" (UID: \"c4da3216-9857-497f-a5ca-0e193ba681a5\") " pod="ai-tenant-e2e-inf-a-2239f6/test-model-2239f6-kserve-5588cfd8df-ccsnd" Jul 30 00:55:01.752349 ip-10-0-142-6 kubenswrapper[2564]: I0730 00:55:01.752242 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c4da3216-9857-497f-a5ca-0e193ba681a5-dshm\") pod \"test-model-2239f6-kserve-5588cfd8df-ccsnd\" (UID: \"c4da3216-9857-497f-a5ca-0e193ba681a5\") " pod="ai-tenant-e2e-inf-a-2239f6/test-model-2239f6-kserve-5588cfd8df-ccsnd" Jul 30 00:55:01.752349 ip-10-0-142-6 kubenswrapper[2564]: I0730 00:55:01.752269 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c4da3216-9857-497f-a5ca-0e193ba681a5-model-cache\") pod \"test-model-2239f6-kserve-5588cfd8df-ccsnd\" (UID: \"c4da3216-9857-497f-a5ca-0e193ba681a5\") " pod="ai-tenant-e2e-inf-a-2239f6/test-model-2239f6-kserve-5588cfd8df-ccsnd" Jul 30 00:55:01.752349 ip-10-0-142-6 kubenswrapper[2564]: I0730 00:55:01.752295 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c4da3216-9857-497f-a5ca-0e193ba681a5-tls-certs\") pod \"test-model-2239f6-kserve-5588cfd8df-ccsnd\" (UID: \"c4da3216-9857-497f-a5ca-0e193ba681a5\") " pod="ai-tenant-e2e-inf-a-2239f6/test-model-2239f6-kserve-5588cfd8df-ccsnd" Jul 30 00:55:01.753190 ip-10-0-142-6 kubenswrapper[2564]: I0730 00:55:01.753013 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c4da3216-9857-497f-a5ca-0e193ba681a5-tmp-dir\") pod \"test-model-2239f6-kserve-5588cfd8df-ccsnd\" (UID: \"c4da3216-9857-497f-a5ca-0e193ba681a5\") " pod="ai-tenant-e2e-inf-a-2239f6/test-model-2239f6-kserve-5588cfd8df-ccsnd" Jul 30 00:55:01.753190 ip-10-0-142-6 kubenswrapper[2564]: I0730 00:55:01.753042 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c4da3216-9857-497f-a5ca-0e193ba681a5-home\") pod \"test-model-2239f6-kserve-5588cfd8df-ccsnd\" (UID: \"c4da3216-9857-497f-a5ca-0e193ba681a5\") " pod="ai-tenant-e2e-inf-a-2239f6/test-model-2239f6-kserve-5588cfd8df-ccsnd" Jul 30 00:55:01.753190 ip-10-0-142-6 kubenswrapper[2564]: I0730 00:55:01.753090 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c4da3216-9857-497f-a5ca-0e193ba681a5-model-cache\") pod \"test-model-2239f6-kserve-5588cfd8df-ccsnd\" (UID: \"c4da3216-9857-497f-a5ca-0e193ba681a5\") " pod="ai-tenant-e2e-inf-a-2239f6/test-model-2239f6-kserve-5588cfd8df-ccsnd" Jul 30 00:55:01.758456 ip-10-0-142-6 kubenswrapper[2564]: I0730 00:55:01.755788 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c4da3216-9857-497f-a5ca-0e193ba681a5-tls-certs\") pod \"test-model-2239f6-kserve-5588cfd8df-ccsnd\" (UID: \"c4da3216-9857-497f-a5ca-0e193ba681a5\") " pod="ai-tenant-e2e-inf-a-2239f6/test-model-2239f6-kserve-5588cfd8df-ccsnd" Jul 30 00:55:01.758931 ip-10-0-142-6 kubenswrapper[2564]: I0730 00:55:01.758907 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c4da3216-9857-497f-a5ca-0e193ba681a5-dshm\") pod \"test-model-2239f6-kserve-5588cfd8df-ccsnd\" (UID: \"c4da3216-9857-497f-a5ca-0e193ba681a5\") " pod="ai-tenant-e2e-inf-a-2239f6/test-model-2239f6-kserve-5588cfd8df-ccsnd" Jul 30 00:55:01.761675 ip-10-0-142-6 kubenswrapper[2564]: I0730 00:55:01.761651 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9zhrr\" (UniqueName: \"kubernetes.io/projected/c4da3216-9857-497f-a5ca-0e193ba681a5-kube-api-access-9zhrr\") pod \"test-model-2239f6-kserve-5588cfd8df-ccsnd\" (UID: \"c4da3216-9857-497f-a5ca-0e193ba681a5\") " pod="ai-tenant-e2e-inf-a-2239f6/test-model-2239f6-kserve-5588cfd8df-ccsnd" Jul 30 00:55:01.933995 ip-10-0-142-6 kubenswrapper[2564]: I0730 00:55:01.933906 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-2239f6/test-model-2239f6-kserve-5588cfd8df-ccsnd" Jul 30 00:55:02.067964 ip-10-0-142-6 kubenswrapper[2564]: I0730 00:55:02.067903 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-a-2239f6/test-model-2239f6-kserve-5588cfd8df-ccsnd"] Jul 30 00:55:02.070587 ip-10-0-142-6 kubenswrapper[2564]: W0730 00:55:02.070558 2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podc4da3216_9857_497f_a5ca_0e193ba681a5.slice/crio-56b30df8d0c72aac8024ba477225e19410ed0cb5fa5813066b30826ed8b6679b WatchSource:0}: Error finding container 56b30df8d0c72aac8024ba477225e19410ed0cb5fa5813066b30826ed8b6679b: Status 404 returned error can't find the container with id 56b30df8d0c72aac8024ba477225e19410ed0cb5fa5813066b30826ed8b6679b Jul 30 00:55:02.520021 ip-10-0-142-6 kubenswrapper[2564]: I0730 00:55:02.519986 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-2239f6/test-model-2239f6-kserve-5588cfd8df-ccsnd" event={"ID":"c4da3216-9857-497f-a5ca-0e193ba681a5","Type":"ContainerStarted","Data":"b91aae439bc1ef1dda8fec07f64b7379c64715ff51ecaf5785d49987598d9f1b"} Jul 30 00:55:02.520021 ip-10-0-142-6 kubenswrapper[2564]: I0730 00:55:02.520027 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-2239f6/test-model-2239f6-kserve-5588cfd8df-ccsnd" event={"ID":"c4da3216-9857-497f-a5ca-0e193ba681a5","Type":"ContainerStarted","Data":"56b30df8d0c72aac8024ba477225e19410ed0cb5fa5813066b30826ed8b6679b"} Jul 30 00:55:02.537858 ip-10-0-142-6 kubenswrapper[2564]: I0730 00:55:02.537802 2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-inf-a-2239f6/test-model-2239f6-kserve-5588cfd8df-ccsnd" podStartSLOduration=1.5377856680000002 podStartE2EDuration="1.537785668s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-30 00:55:01 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-30 00:55:02.53739698 +0000 UTC m=+2233.952136928" watchObservedRunningTime="2026-07-30 00:55:02.537785668 +0000 UTC m=+2233.952525600" Jul 30 00:55:05.803648 ip-10-0-142-6 kubenswrapper[2564]: I0730 00:55:05.803610 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-nrv2p"] Jul 30 00:55:11.935112 ip-10-0-142-6 kubenswrapper[2564]: I0730 00:55:11.934974 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-inf-a-2239f6/test-model-2239f6-kserve-5588cfd8df-ccsnd" Jul 30 00:55:11.935112 ip-10-0-142-6 kubenswrapper[2564]: I0730 00:55:11.935042 2564 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-inf-a-2239f6/test-model-2239f6-kserve-5588cfd8df-ccsnd" Jul 30 00:55:11.947566 ip-10-0-142-6 kubenswrapper[2564]: I0730 00:55:11.947541 2564 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-inf-a-2239f6/test-model-2239f6-kserve-5588cfd8df-ccsnd" Jul 30 00:55:12.586265 ip-10-0-142-6 kubenswrapper[2564]: I0730 00:55:12.586236 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-inf-a-2239f6/test-model-2239f6-kserve-5588cfd8df-ccsnd" Jul 30 00:55:17.014476 ip-10-0-142-6 kubenswrapper[2564]: I0730 00:55:17.014443 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-nrv2p"] Jul 30 00:56:01.498277 ip-10-0-142-6 kubenswrapper[2564]: I0730 00:56:01.498238 2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-inf-b-6898ea/test-model-6898ea-kserve-5c99d8f67f-hwtzd"] Jul 30 00:56:01.501921 ip-10-0-142-6 kubenswrapper[2564]: I0730 00:56:01.501901 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-6898ea/test-model-6898ea-kserve-5c99d8f67f-hwtzd" Jul 30 00:56:01.504423 ip-10-0-142-6 kubenswrapper[2564]: I0730 00:56:01.504391 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-b-6898ea\"/\"default-dockercfg-tm7n6\"" Jul 30 00:56:01.505435 ip-10-0-142-6 kubenswrapper[2564]: I0730 00:56:01.505415 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-b-6898ea\"/\"openshift-service-ca.crt\"" Jul 30 00:56:01.505610 ip-10-0-142-6 kubenswrapper[2564]: I0730 00:56:01.505585 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-b-6898ea\"/\"kube-root-ca.crt\"" Jul 30 00:56:01.505702 ip-10-0-142-6 kubenswrapper[2564]: I0730 00:56:01.505653 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-b-6898ea\"/\"test-model-6898ea-kserve-self-signed-certs\"" Jul 30 00:56:01.509167 ip-10-0-142-6 kubenswrapper[2564]: I0730 00:56:01.509132 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-b-6898ea/test-model-6898ea-kserve-5c99d8f67f-hwtzd"] Jul 30 00:56:01.538555 ip-10-0-142-6 kubenswrapper[2564]: I0730 00:56:01.538522 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/831c807c-c28e-43e5-a73c-e79a2eacc198-dshm\") pod \"test-model-6898ea-kserve-5c99d8f67f-hwtzd\" (UID: \"831c807c-c28e-43e5-a73c-e79a2eacc198\") " pod="ai-tenant-e2e-inf-b-6898ea/test-model-6898ea-kserve-5c99d8f67f-hwtzd" Jul 30 00:56:01.538697 ip-10-0-142-6 kubenswrapper[2564]: I0730 00:56:01.538561 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6vdr6\" (UniqueName: \"kubernetes.io/projected/831c807c-c28e-43e5-a73c-e79a2eacc198-kube-api-access-6vdr6\") pod \"test-model-6898ea-kserve-5c99d8f67f-hwtzd\" (UID: \"831c807c-c28e-43e5-a73c-e79a2eacc198\") " pod="ai-tenant-e2e-inf-b-6898ea/test-model-6898ea-kserve-5c99d8f67f-hwtzd" Jul 30 00:56:01.538697 ip-10-0-142-6 kubenswrapper[2564]: I0730 00:56:01.538589 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/831c807c-c28e-43e5-a73c-e79a2eacc198-tmp-dir\") pod \"test-model-6898ea-kserve-5c99d8f67f-hwtzd\" (UID: \"831c807c-c28e-43e5-a73c-e79a2eacc198\") " pod="ai-tenant-e2e-inf-b-6898ea/test-model-6898ea-kserve-5c99d8f67f-hwtzd" Jul 30 00:56:01.538697 ip-10-0-142-6 kubenswrapper[2564]: I0730 00:56:01.538630 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/831c807c-c28e-43e5-a73c-e79a2eacc198-model-cache\") pod \"test-model-6898ea-kserve-5c99d8f67f-hwtzd\" (UID: \"831c807c-c28e-43e5-a73c-e79a2eacc198\") " pod="ai-tenant-e2e-inf-b-6898ea/test-model-6898ea-kserve-5c99d8f67f-hwtzd" Jul 30 00:56:01.538697 ip-10-0-142-6 kubenswrapper[2564]: I0730 00:56:01.538679 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/831c807c-c28e-43e5-a73c-e79a2eacc198-home\") pod \"test-model-6898ea-kserve-5c99d8f67f-hwtzd\" (UID: \"831c807c-c28e-43e5-a73c-e79a2eacc198\") " pod="ai-tenant-e2e-inf-b-6898ea/test-model-6898ea-kserve-5c99d8f67f-hwtzd" Jul 30 00:56:01.538828 ip-10-0-142-6 kubenswrapper[2564]: I0730 00:56:01.538714 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/831c807c-c28e-43e5-a73c-e79a2eacc198-tls-certs\") pod \"test-model-6898ea-kserve-5c99d8f67f-hwtzd\" (UID: \"831c807c-c28e-43e5-a73c-e79a2eacc198\") " pod="ai-tenant-e2e-inf-b-6898ea/test-model-6898ea-kserve-5c99d8f67f-hwtzd" Jul 30 00:56:01.639185 ip-10-0-142-6 kubenswrapper[2564]: I0730 00:56:01.639148 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/831c807c-c28e-43e5-a73c-e79a2eacc198-dshm\") pod \"test-model-6898ea-kserve-5c99d8f67f-hwtzd\" (UID: \"831c807c-c28e-43e5-a73c-e79a2eacc198\") " pod="ai-tenant-e2e-inf-b-6898ea/test-model-6898ea-kserve-5c99d8f67f-hwtzd" Jul 30 00:56:01.639185 ip-10-0-142-6 kubenswrapper[2564]: I0730 00:56:01.639191 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6vdr6\" (UniqueName: \"kubernetes.io/projected/831c807c-c28e-43e5-a73c-e79a2eacc198-kube-api-access-6vdr6\") pod \"test-model-6898ea-kserve-5c99d8f67f-hwtzd\" (UID: \"831c807c-c28e-43e5-a73c-e79a2eacc198\") " pod="ai-tenant-e2e-inf-b-6898ea/test-model-6898ea-kserve-5c99d8f67f-hwtzd" Jul 30 00:56:01.639439 ip-10-0-142-6 kubenswrapper[2564]: I0730 00:56:01.639218 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/831c807c-c28e-43e5-a73c-e79a2eacc198-tmp-dir\") pod \"test-model-6898ea-kserve-5c99d8f67f-hwtzd\" (UID: \"831c807c-c28e-43e5-a73c-e79a2eacc198\") " pod="ai-tenant-e2e-inf-b-6898ea/test-model-6898ea-kserve-5c99d8f67f-hwtzd" Jul 30 00:56:01.639439 ip-10-0-142-6 kubenswrapper[2564]: I0730 00:56:01.639236 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/831c807c-c28e-43e5-a73c-e79a2eacc198-model-cache\") pod \"test-model-6898ea-kserve-5c99d8f67f-hwtzd\" (UID: \"831c807c-c28e-43e5-a73c-e79a2eacc198\") " pod="ai-tenant-e2e-inf-b-6898ea/test-model-6898ea-kserve-5c99d8f67f-hwtzd" Jul 30 00:56:01.639439 ip-10-0-142-6 kubenswrapper[2564]: I0730 00:56:01.639350 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/831c807c-c28e-43e5-a73c-e79a2eacc198-home\") pod \"test-model-6898ea-kserve-5c99d8f67f-hwtzd\" (UID: \"831c807c-c28e-43e5-a73c-e79a2eacc198\") " pod="ai-tenant-e2e-inf-b-6898ea/test-model-6898ea-kserve-5c99d8f67f-hwtzd" Jul 30 00:56:01.639439 ip-10-0-142-6 kubenswrapper[2564]: I0730 00:56:01.639427 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/831c807c-c28e-43e5-a73c-e79a2eacc198-tls-certs\") pod \"test-model-6898ea-kserve-5c99d8f67f-hwtzd\" (UID: \"831c807c-c28e-43e5-a73c-e79a2eacc198\") " pod="ai-tenant-e2e-inf-b-6898ea/test-model-6898ea-kserve-5c99d8f67f-hwtzd" Jul 30 00:56:01.639607 ip-10-0-142-6 kubenswrapper[2564]: I0730 00:56:01.639562 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/831c807c-c28e-43e5-a73c-e79a2eacc198-model-cache\") pod \"test-model-6898ea-kserve-5c99d8f67f-hwtzd\" (UID: \"831c807c-c28e-43e5-a73c-e79a2eacc198\") " pod="ai-tenant-e2e-inf-b-6898ea/test-model-6898ea-kserve-5c99d8f67f-hwtzd" Jul 30 00:56:01.639661 ip-10-0-142-6 kubenswrapper[2564]: I0730 00:56:01.639642 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/831c807c-c28e-43e5-a73c-e79a2eacc198-tmp-dir\") pod \"test-model-6898ea-kserve-5c99d8f67f-hwtzd\" (UID: \"831c807c-c28e-43e5-a73c-e79a2eacc198\") " pod="ai-tenant-e2e-inf-b-6898ea/test-model-6898ea-kserve-5c99d8f67f-hwtzd" Jul 30 00:56:01.639770 ip-10-0-142-6 kubenswrapper[2564]: I0730 00:56:01.639753 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/831c807c-c28e-43e5-a73c-e79a2eacc198-home\") pod \"test-model-6898ea-kserve-5c99d8f67f-hwtzd\" (UID: \"831c807c-c28e-43e5-a73c-e79a2eacc198\") " pod="ai-tenant-e2e-inf-b-6898ea/test-model-6898ea-kserve-5c99d8f67f-hwtzd" Jul 30 00:56:01.641653 ip-10-0-142-6 kubenswrapper[2564]: I0730 00:56:01.641629 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/831c807c-c28e-43e5-a73c-e79a2eacc198-dshm\") pod \"test-model-6898ea-kserve-5c99d8f67f-hwtzd\" (UID: \"831c807c-c28e-43e5-a73c-e79a2eacc198\") " pod="ai-tenant-e2e-inf-b-6898ea/test-model-6898ea-kserve-5c99d8f67f-hwtzd" Jul 30 00:56:01.641896 ip-10-0-142-6 kubenswrapper[2564]: I0730 00:56:01.641881 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/831c807c-c28e-43e5-a73c-e79a2eacc198-tls-certs\") pod \"test-model-6898ea-kserve-5c99d8f67f-hwtzd\" (UID: \"831c807c-c28e-43e5-a73c-e79a2eacc198\") " pod="ai-tenant-e2e-inf-b-6898ea/test-model-6898ea-kserve-5c99d8f67f-hwtzd" Jul 30 00:56:01.654634 ip-10-0-142-6 kubenswrapper[2564]: I0730 00:56:01.654609 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6vdr6\" (UniqueName: \"kubernetes.io/projected/831c807c-c28e-43e5-a73c-e79a2eacc198-kube-api-access-6vdr6\") pod \"test-model-6898ea-kserve-5c99d8f67f-hwtzd\" (UID: \"831c807c-c28e-43e5-a73c-e79a2eacc198\") " pod="ai-tenant-e2e-inf-b-6898ea/test-model-6898ea-kserve-5c99d8f67f-hwtzd" Jul 30 00:56:01.814373 ip-10-0-142-6 kubenswrapper[2564]: I0730 00:56:01.814284 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-6898ea/test-model-6898ea-kserve-5c99d8f67f-hwtzd" Jul 30 00:56:01.945714 ip-10-0-142-6 kubenswrapper[2564]: I0730 00:56:01.945685 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-b-6898ea/test-model-6898ea-kserve-5c99d8f67f-hwtzd"] Jul 30 00:56:01.946830 ip-10-0-142-6 kubenswrapper[2564]: W0730 00:56:01.946806 2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod831c807c_c28e_43e5_a73c_e79a2eacc198.slice/crio-8eddadc12efbbfc1f6cd3608efecb0b76defc17d2764434674434cc0fda5ad49 WatchSource:0}: Error finding container 8eddadc12efbbfc1f6cd3608efecb0b76defc17d2764434674434cc0fda5ad49: Status 404 returned error can't find the container with id 8eddadc12efbbfc1f6cd3608efecb0b76defc17d2764434674434cc0fda5ad49 Jul 30 00:56:02.814418 ip-10-0-142-6 kubenswrapper[2564]: I0730 00:56:02.814385 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-6898ea/test-model-6898ea-kserve-5c99d8f67f-hwtzd" event={"ID":"831c807c-c28e-43e5-a73c-e79a2eacc198","Type":"ContainerStarted","Data":"3fb9df1886f42bc355bc4998d13bac9e8189d7dffc347a00a564bbb068afe526"} Jul 30 00:56:02.814808 ip-10-0-142-6 kubenswrapper[2564]: I0730 00:56:02.814427 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-6898ea/test-model-6898ea-kserve-5c99d8f67f-hwtzd" event={"ID":"831c807c-c28e-43e5-a73c-e79a2eacc198","Type":"ContainerStarted","Data":"8eddadc12efbbfc1f6cd3608efecb0b76defc17d2764434674434cc0fda5ad49"} Jul 30 00:56:02.830314 ip-10-0-142-6 kubenswrapper[2564]: I0730 00:56:02.830265 2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-inf-b-6898ea/test-model-6898ea-kserve-5c99d8f67f-hwtzd" podStartSLOduration=1.830248338 podStartE2EDuration="1.830248338s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-30 00:56:01 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-30 00:56:02.829389375 +0000 UTC m=+2294.244129315" watchObservedRunningTime="2026-07-30 00:56:02.830248338 +0000 UTC m=+2294.244988272" Jul 30 00:56:06.907748 ip-10-0-142-6 kubenswrapper[2564]: I0730 00:56:06.907710 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-nrv2p"] Jul 30 00:56:11.814658 ip-10-0-142-6 kubenswrapper[2564]: I0730 00:56:11.814622 2564 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-inf-b-6898ea/test-model-6898ea-kserve-5c99d8f67f-hwtzd" Jul 30 00:56:11.814658 ip-10-0-142-6 kubenswrapper[2564]: I0730 00:56:11.814665 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-inf-b-6898ea/test-model-6898ea-kserve-5c99d8f67f-hwtzd" Jul 30 00:56:11.827238 ip-10-0-142-6 kubenswrapper[2564]: I0730 00:56:11.827213 2564 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-inf-b-6898ea/test-model-6898ea-kserve-5c99d8f67f-hwtzd" Jul 30 00:56:11.867716 ip-10-0-142-6 kubenswrapper[2564]: I0730 00:56:11.867690 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-inf-b-6898ea/test-model-6898ea-kserve-5c99d8f67f-hwtzd" Jul 30 00:56:40.508860 ip-10-0-142-6 kubenswrapper[2564]: I0730 00:56:40.508783 2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-a-2239f6/test-model-2239f6-kserve-5588cfd8df-ccsnd"] Jul 30 00:56:41.392835 ip-10-0-142-6 kubenswrapper[2564]: I0730 00:56:41.392803 2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-b-6898ea/test-model-6898ea-kserve-5c99d8f67f-hwtzd"] Jul 30 00:56:45.003426 ip-10-0-142-6 kubenswrapper[2564]: I0730 00:56:45.003389 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-nrv2p"] Jul 30 00:56:55.528263 ip-10-0-142-6 kubenswrapper[2564]: I0730 00:56:55.528193 2564 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-inf-a-2239f6/test-model-2239f6-kserve-5588cfd8df-ccsnd" podUID="c4da3216-9857-497f-a5ca-0e193ba681a5" containerName="main" containerID="cri-o://b91aae439bc1ef1dda8fec07f64b7379c64715ff51ecaf5785d49987598d9f1b" gracePeriod=45 Jul 30 00:56:55.779452 ip-10-0-142-6 kubenswrapper[2564]: I0730 00:56:55.779388 2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-2239f6/test-model-2239f6-kserve-5588cfd8df-ccsnd" Jul 30 00:56:55.800258 ip-10-0-142-6 kubenswrapper[2564]: I0730 00:56:55.800227 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c4da3216-9857-497f-a5ca-0e193ba681a5-model-cache\") pod \"c4da3216-9857-497f-a5ca-0e193ba681a5\" (UID: \"c4da3216-9857-497f-a5ca-0e193ba681a5\") " Jul 30 00:56:55.800412 ip-10-0-142-6 kubenswrapper[2564]: I0730 00:56:55.800285 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c4da3216-9857-497f-a5ca-0e193ba681a5-home\") pod \"c4da3216-9857-497f-a5ca-0e193ba681a5\" (UID: \"c4da3216-9857-497f-a5ca-0e193ba681a5\") " Jul 30 00:56:55.800412 ip-10-0-142-6 kubenswrapper[2564]: I0730 00:56:55.800315 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-9zhrr\" (UniqueName: \"kubernetes.io/projected/c4da3216-9857-497f-a5ca-0e193ba681a5-kube-api-access-9zhrr\") pod \"c4da3216-9857-497f-a5ca-0e193ba681a5\" (UID: \"c4da3216-9857-497f-a5ca-0e193ba681a5\") " Jul 30 00:56:55.800412 ip-10-0-142-6 kubenswrapper[2564]: I0730 00:56:55.800384 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c4da3216-9857-497f-a5ca-0e193ba681a5-tls-certs\") pod \"c4da3216-9857-497f-a5ca-0e193ba681a5\" (UID: \"c4da3216-9857-497f-a5ca-0e193ba681a5\") " Jul 30 00:56:55.800587 ip-10-0-142-6 kubenswrapper[2564]: I0730 00:56:55.800418 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c4da3216-9857-497f-a5ca-0e193ba681a5-tmp-dir\") pod \"c4da3216-9857-497f-a5ca-0e193ba681a5\" (UID: \"c4da3216-9857-497f-a5ca-0e193ba681a5\") " Jul 30 00:56:55.800587 ip-10-0-142-6 kubenswrapper[2564]: I0730 00:56:55.800458 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c4da3216-9857-497f-a5ca-0e193ba681a5-dshm\") pod \"c4da3216-9857-497f-a5ca-0e193ba681a5\" (UID: \"c4da3216-9857-497f-a5ca-0e193ba681a5\") " Jul 30 00:56:55.800587 ip-10-0-142-6 kubenswrapper[2564]: I0730 00:56:55.800478 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c4da3216-9857-497f-a5ca-0e193ba681a5-model-cache" (OuterVolumeSpecName: "model-cache") pod "c4da3216-9857-497f-a5ca-0e193ba681a5" (UID: "c4da3216-9857-497f-a5ca-0e193ba681a5"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 30 00:56:55.800587 ip-10-0-142-6 kubenswrapper[2564]: I0730 00:56:55.800564 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c4da3216-9857-497f-a5ca-0e193ba681a5-home" (OuterVolumeSpecName: "home") pod "c4da3216-9857-497f-a5ca-0e193ba681a5" (UID: "c4da3216-9857-497f-a5ca-0e193ba681a5"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 30 00:56:55.800765 ip-10-0-142-6 kubenswrapper[2564]: I0730 00:56:55.800678 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c4da3216-9857-497f-a5ca-0e193ba681a5-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "c4da3216-9857-497f-a5ca-0e193ba681a5" (UID: "c4da3216-9857-497f-a5ca-0e193ba681a5"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 30 00:56:55.800829 ip-10-0-142-6 kubenswrapper[2564]: I0730 00:56:55.800770 2564 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c4da3216-9857-497f-a5ca-0e193ba681a5-model-cache\") on node \"ip-10-0-142-6.ec2.internal\" DevicePath \"\"" Jul 30 00:56:55.800829 ip-10-0-142-6 kubenswrapper[2564]: I0730 00:56:55.800787 2564 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c4da3216-9857-497f-a5ca-0e193ba681a5-home\") on node \"ip-10-0-142-6.ec2.internal\" DevicePath \"\"" Jul 30 00:56:55.800829 ip-10-0-142-6 kubenswrapper[2564]: I0730 00:56:55.800801 2564 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c4da3216-9857-497f-a5ca-0e193ba681a5-tmp-dir\") on node \"ip-10-0-142-6.ec2.internal\" DevicePath \"\"" Jul 30 00:56:55.802589 ip-10-0-142-6 kubenswrapper[2564]: I0730 00:56:55.802562 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c4da3216-9857-497f-a5ca-0e193ba681a5-dshm" (OuterVolumeSpecName: "dshm") pod "c4da3216-9857-497f-a5ca-0e193ba681a5" (UID: "c4da3216-9857-497f-a5ca-0e193ba681a5"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 30 00:56:55.802693 ip-10-0-142-6 kubenswrapper[2564]: I0730 00:56:55.802621 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c4da3216-9857-497f-a5ca-0e193ba681a5-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "c4da3216-9857-497f-a5ca-0e193ba681a5" (UID: "c4da3216-9857-497f-a5ca-0e193ba681a5"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 30 00:56:55.802737 ip-10-0-142-6 kubenswrapper[2564]: I0730 00:56:55.802697 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c4da3216-9857-497f-a5ca-0e193ba681a5-kube-api-access-9zhrr" (OuterVolumeSpecName: "kube-api-access-9zhrr") pod "c4da3216-9857-497f-a5ca-0e193ba681a5" (UID: "c4da3216-9857-497f-a5ca-0e193ba681a5"). InnerVolumeSpecName "kube-api-access-9zhrr". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 30 00:56:55.901444 ip-10-0-142-6 kubenswrapper[2564]: I0730 00:56:55.901402 2564 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c4da3216-9857-497f-a5ca-0e193ba681a5-tls-certs\") on node \"ip-10-0-142-6.ec2.internal\" DevicePath \"\"" Jul 30 00:56:55.901444 ip-10-0-142-6 kubenswrapper[2564]: I0730 00:56:55.901434 2564 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c4da3216-9857-497f-a5ca-0e193ba681a5-dshm\") on node \"ip-10-0-142-6.ec2.internal\" DevicePath \"\"" Jul 30 00:56:55.901444 ip-10-0-142-6 kubenswrapper[2564]: I0730 00:56:55.901447 2564 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9zhrr\" (UniqueName: \"kubernetes.io/projected/c4da3216-9857-497f-a5ca-0e193ba681a5-kube-api-access-9zhrr\") on node \"ip-10-0-142-6.ec2.internal\" DevicePath \"\"" Jul 30 00:56:56.079270 ip-10-0-142-6 kubenswrapper[2564]: I0730 00:56:56.079184 2564 generic.go:356] "Generic (PLEG): container finished" podID="c4da3216-9857-497f-a5ca-0e193ba681a5" containerID="b91aae439bc1ef1dda8fec07f64b7379c64715ff51ecaf5785d49987598d9f1b" exitCode=0 Jul 30 00:56:56.079270 ip-10-0-142-6 kubenswrapper[2564]: I0730 00:56:56.079229 2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-2239f6/test-model-2239f6-kserve-5588cfd8df-ccsnd" Jul 30 00:56:56.079270 ip-10-0-142-6 kubenswrapper[2564]: I0730 00:56:56.079249 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-2239f6/test-model-2239f6-kserve-5588cfd8df-ccsnd" event={"ID":"c4da3216-9857-497f-a5ca-0e193ba681a5","Type":"ContainerDied","Data":"b91aae439bc1ef1dda8fec07f64b7379c64715ff51ecaf5785d49987598d9f1b"} Jul 30 00:56:56.079506 ip-10-0-142-6 kubenswrapper[2564]: I0730 00:56:56.079279 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-2239f6/test-model-2239f6-kserve-5588cfd8df-ccsnd" event={"ID":"c4da3216-9857-497f-a5ca-0e193ba681a5","Type":"ContainerDied","Data":"56b30df8d0c72aac8024ba477225e19410ed0cb5fa5813066b30826ed8b6679b"} Jul 30 00:56:56.079506 ip-10-0-142-6 kubenswrapper[2564]: I0730 00:56:56.079296 2564 scope.go:117] "RemoveContainer" containerID="b91aae439bc1ef1dda8fec07f64b7379c64715ff51ecaf5785d49987598d9f1b" Jul 30 00:56:56.089164 ip-10-0-142-6 kubenswrapper[2564]: I0730 00:56:56.089142 2564 scope.go:117] "RemoveContainer" containerID="b91aae439bc1ef1dda8fec07f64b7379c64715ff51ecaf5785d49987598d9f1b" Jul 30 00:56:56.089436 ip-10-0-142-6 kubenswrapper[2564]: E0730 00:56:56.089416 2564 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b91aae439bc1ef1dda8fec07f64b7379c64715ff51ecaf5785d49987598d9f1b\": container with ID starting with b91aae439bc1ef1dda8fec07f64b7379c64715ff51ecaf5785d49987598d9f1b not found: ID does not exist" containerID="b91aae439bc1ef1dda8fec07f64b7379c64715ff51ecaf5785d49987598d9f1b" Jul 30 00:56:56.089507 ip-10-0-142-6 kubenswrapper[2564]: I0730 00:56:56.089444 2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b91aae439bc1ef1dda8fec07f64b7379c64715ff51ecaf5785d49987598d9f1b"} err="failed to get container status \"b91aae439bc1ef1dda8fec07f64b7379c64715ff51ecaf5785d49987598d9f1b\": rpc error: code = NotFound desc = could not find container \"b91aae439bc1ef1dda8fec07f64b7379c64715ff51ecaf5785d49987598d9f1b\": container with ID starting with b91aae439bc1ef1dda8fec07f64b7379c64715ff51ecaf5785d49987598d9f1b not found: ID does not exist" Jul 30 00:56:56.100623 ip-10-0-142-6 kubenswrapper[2564]: I0730 00:56:56.100595 2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-a-2239f6/test-model-2239f6-kserve-5588cfd8df-ccsnd"] Jul 30 00:56:56.104582 ip-10-0-142-6 kubenswrapper[2564]: I0730 00:56:56.104559 2564 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-inf-a-2239f6/test-model-2239f6-kserve-5588cfd8df-ccsnd"] Jul 30 00:56:56.411744 ip-10-0-142-6 kubenswrapper[2564]: I0730 00:56:56.411680 2564 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-inf-b-6898ea/test-model-6898ea-kserve-5c99d8f67f-hwtzd" podUID="831c807c-c28e-43e5-a73c-e79a2eacc198" containerName="main" containerID="cri-o://3fb9df1886f42bc355bc4998d13bac9e8189d7dffc347a00a564bbb068afe526" gracePeriod=45 Jul 30 00:56:56.662233 ip-10-0-142-6 kubenswrapper[2564]: I0730 00:56:56.662173 2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-6898ea/test-model-6898ea-kserve-5c99d8f67f-hwtzd" Jul 30 00:56:56.707403 ip-10-0-142-6 kubenswrapper[2564]: I0730 00:56:56.707370 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/831c807c-c28e-43e5-a73c-e79a2eacc198-model-cache\") pod \"831c807c-c28e-43e5-a73c-e79a2eacc198\" (UID: \"831c807c-c28e-43e5-a73c-e79a2eacc198\") " Jul 30 00:56:56.707576 ip-10-0-142-6 kubenswrapper[2564]: I0730 00:56:56.707417 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/831c807c-c28e-43e5-a73c-e79a2eacc198-tls-certs\") pod \"831c807c-c28e-43e5-a73c-e79a2eacc198\" (UID: \"831c807c-c28e-43e5-a73c-e79a2eacc198\") " Jul 30 00:56:56.707576 ip-10-0-142-6 kubenswrapper[2564]: I0730 00:56:56.707456 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/831c807c-c28e-43e5-a73c-e79a2eacc198-home\") pod \"831c807c-c28e-43e5-a73c-e79a2eacc198\" (UID: \"831c807c-c28e-43e5-a73c-e79a2eacc198\") " Jul 30 00:56:56.707576 ip-10-0-142-6 kubenswrapper[2564]: I0730 00:56:56.707523 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-6vdr6\" (UniqueName: \"kubernetes.io/projected/831c807c-c28e-43e5-a73c-e79a2eacc198-kube-api-access-6vdr6\") pod \"831c807c-c28e-43e5-a73c-e79a2eacc198\" (UID: \"831c807c-c28e-43e5-a73c-e79a2eacc198\") " Jul 30 00:56:56.707576 ip-10-0-142-6 kubenswrapper[2564]: I0730 00:56:56.707549 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/831c807c-c28e-43e5-a73c-e79a2eacc198-dshm\") pod \"831c807c-c28e-43e5-a73c-e79a2eacc198\" (UID: \"831c807c-c28e-43e5-a73c-e79a2eacc198\") " Jul 30 00:56:56.708044 ip-10-0-142-6 kubenswrapper[2564]: I0730 00:56:56.707986 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/831c807c-c28e-43e5-a73c-e79a2eacc198-model-cache" (OuterVolumeSpecName: "model-cache") pod "831c807c-c28e-43e5-a73c-e79a2eacc198" (UID: "831c807c-c28e-43e5-a73c-e79a2eacc198"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 30 00:56:56.708275 ip-10-0-142-6 kubenswrapper[2564]: I0730 00:56:56.708247 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/831c807c-c28e-43e5-a73c-e79a2eacc198-tmp-dir\") pod \"831c807c-c28e-43e5-a73c-e79a2eacc198\" (UID: \"831c807c-c28e-43e5-a73c-e79a2eacc198\") " Jul 30 00:56:56.708275 ip-10-0-142-6 kubenswrapper[2564]: I0730 00:56:56.708266 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/831c807c-c28e-43e5-a73c-e79a2eacc198-home" (OuterVolumeSpecName: "home") pod "831c807c-c28e-43e5-a73c-e79a2eacc198" (UID: "831c807c-c28e-43e5-a73c-e79a2eacc198"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 30 00:56:56.708604 ip-10-0-142-6 kubenswrapper[2564]: I0730 00:56:56.708563 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/831c807c-c28e-43e5-a73c-e79a2eacc198-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "831c807c-c28e-43e5-a73c-e79a2eacc198" (UID: "831c807c-c28e-43e5-a73c-e79a2eacc198"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 30 00:56:56.708815 ip-10-0-142-6 kubenswrapper[2564]: I0730 00:56:56.708797 2564 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/831c807c-c28e-43e5-a73c-e79a2eacc198-tmp-dir\") on node \"ip-10-0-142-6.ec2.internal\" DevicePath \"\"" Jul 30 00:56:56.708892 ip-10-0-142-6 kubenswrapper[2564]: I0730 00:56:56.708822 2564 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/831c807c-c28e-43e5-a73c-e79a2eacc198-model-cache\") on node \"ip-10-0-142-6.ec2.internal\" DevicePath \"\"" Jul 30 00:56:56.708892 ip-10-0-142-6 kubenswrapper[2564]: I0730 00:56:56.708845 2564 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/831c807c-c28e-43e5-a73c-e79a2eacc198-home\") on node \"ip-10-0-142-6.ec2.internal\" DevicePath \"\"" Jul 30 00:56:56.715557 ip-10-0-142-6 kubenswrapper[2564]: I0730 00:56:56.715528 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/831c807c-c28e-43e5-a73c-e79a2eacc198-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "831c807c-c28e-43e5-a73c-e79a2eacc198" (UID: "831c807c-c28e-43e5-a73c-e79a2eacc198"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 30 00:56:56.715682 ip-10-0-142-6 kubenswrapper[2564]: I0730 00:56:56.715629 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/831c807c-c28e-43e5-a73c-e79a2eacc198-dshm" (OuterVolumeSpecName: "dshm") pod "831c807c-c28e-43e5-a73c-e79a2eacc198" (UID: "831c807c-c28e-43e5-a73c-e79a2eacc198"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 30 00:56:56.715883 ip-10-0-142-6 kubenswrapper[2564]: I0730 00:56:56.715859 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/831c807c-c28e-43e5-a73c-e79a2eacc198-kube-api-access-6vdr6" (OuterVolumeSpecName: "kube-api-access-6vdr6") pod "831c807c-c28e-43e5-a73c-e79a2eacc198" (UID: "831c807c-c28e-43e5-a73c-e79a2eacc198"). InnerVolumeSpecName "kube-api-access-6vdr6". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 30 00:56:56.809576 ip-10-0-142-6 kubenswrapper[2564]: I0730 00:56:56.809537 2564 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/831c807c-c28e-43e5-a73c-e79a2eacc198-tls-certs\") on node \"ip-10-0-142-6.ec2.internal\" DevicePath \"\"" Jul 30 00:56:56.809576 ip-10-0-142-6 kubenswrapper[2564]: I0730 00:56:56.809568 2564 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6vdr6\" (UniqueName: \"kubernetes.io/projected/831c807c-c28e-43e5-a73c-e79a2eacc198-kube-api-access-6vdr6\") on node \"ip-10-0-142-6.ec2.internal\" DevicePath \"\"" Jul 30 00:56:56.809576 ip-10-0-142-6 kubenswrapper[2564]: I0730 00:56:56.809580 2564 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/831c807c-c28e-43e5-a73c-e79a2eacc198-dshm\") on node \"ip-10-0-142-6.ec2.internal\" DevicePath \"\"" Jul 30 00:56:57.085428 ip-10-0-142-6 kubenswrapper[2564]: I0730 00:56:57.085343 2564 generic.go:356] "Generic (PLEG): container finished" podID="831c807c-c28e-43e5-a73c-e79a2eacc198" containerID="3fb9df1886f42bc355bc4998d13bac9e8189d7dffc347a00a564bbb068afe526" exitCode=0 Jul 30 00:56:57.085428 ip-10-0-142-6 kubenswrapper[2564]: I0730 00:56:57.085409 2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-6898ea/test-model-6898ea-kserve-5c99d8f67f-hwtzd" Jul 30 00:56:57.085649 ip-10-0-142-6 kubenswrapper[2564]: I0730 00:56:57.085431 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-6898ea/test-model-6898ea-kserve-5c99d8f67f-hwtzd" event={"ID":"831c807c-c28e-43e5-a73c-e79a2eacc198","Type":"ContainerDied","Data":"3fb9df1886f42bc355bc4998d13bac9e8189d7dffc347a00a564bbb068afe526"} Jul 30 00:56:57.085649 ip-10-0-142-6 kubenswrapper[2564]: I0730 00:56:57.085481 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-6898ea/test-model-6898ea-kserve-5c99d8f67f-hwtzd" event={"ID":"831c807c-c28e-43e5-a73c-e79a2eacc198","Type":"ContainerDied","Data":"8eddadc12efbbfc1f6cd3608efecb0b76defc17d2764434674434cc0fda5ad49"} Jul 30 00:56:57.085649 ip-10-0-142-6 kubenswrapper[2564]: I0730 00:56:57.085503 2564 scope.go:117] "RemoveContainer" containerID="3fb9df1886f42bc355bc4998d13bac9e8189d7dffc347a00a564bbb068afe526" Jul 30 00:56:57.093980 ip-10-0-142-6 kubenswrapper[2564]: I0730 00:56:57.093963 2564 scope.go:117] "RemoveContainer" containerID="3fb9df1886f42bc355bc4998d13bac9e8189d7dffc347a00a564bbb068afe526" Jul 30 00:56:57.094282 ip-10-0-142-6 kubenswrapper[2564]: E0730 00:56:57.094263 2564 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3fb9df1886f42bc355bc4998d13bac9e8189d7dffc347a00a564bbb068afe526\": container with ID starting with 3fb9df1886f42bc355bc4998d13bac9e8189d7dffc347a00a564bbb068afe526 not found: ID does not exist" containerID="3fb9df1886f42bc355bc4998d13bac9e8189d7dffc347a00a564bbb068afe526" Jul 30 00:56:57.094329 ip-10-0-142-6 kubenswrapper[2564]: I0730 00:56:57.094291 2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3fb9df1886f42bc355bc4998d13bac9e8189d7dffc347a00a564bbb068afe526"} err="failed to get container status \"3fb9df1886f42bc355bc4998d13bac9e8189d7dffc347a00a564bbb068afe526\": rpc error: code = NotFound desc = could not find container \"3fb9df1886f42bc355bc4998d13bac9e8189d7dffc347a00a564bbb068afe526\": container with ID starting with 3fb9df1886f42bc355bc4998d13bac9e8189d7dffc347a00a564bbb068afe526 not found: ID does not exist" Jul 30 00:56:57.106379 ip-10-0-142-6 kubenswrapper[2564]: I0730 00:56:57.106349 2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-b-6898ea/test-model-6898ea-kserve-5c99d8f67f-hwtzd"] Jul 30 00:56:57.109667 ip-10-0-142-6 kubenswrapper[2564]: I0730 00:56:57.109638 2564 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-inf-b-6898ea/test-model-6898ea-kserve-5c99d8f67f-hwtzd"] Jul 30 00:56:57.139383 ip-10-0-142-6 kubenswrapper[2564]: I0730 00:56:57.139354 2564 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="831c807c-c28e-43e5-a73c-e79a2eacc198" path="/var/lib/kubelet/pods/831c807c-c28e-43e5-a73c-e79a2eacc198/volumes" Jul 30 00:56:57.139734 ip-10-0-142-6 kubenswrapper[2564]: I0730 00:56:57.139721 2564 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="c4da3216-9857-497f-a5ca-0e193ba681a5" path="/var/lib/kubelet/pods/c4da3216-9857-497f-a5ca-0e193ba681a5/volumes" Jul 30 00:57:01.108473 ip-10-0-142-6 kubenswrapper[2564]: I0730 00:57:01.108437 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-nrv2p"] Jul 30 00:57:12.407614 ip-10-0-142-6 kubenswrapper[2564]: I0730 00:57:12.407571 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-nrv2p"] Jul 30 00:57:25.509727 ip-10-0-142-6 kubenswrapper[2564]: I0730 00:57:25.509692 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-nrv2p"] Jul 30 00:57:55.313211 ip-10-0-142-6 kubenswrapper[2564]: I0730 00:57:55.313177 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-nrv2p"] Jul 30 00:58:15.435641 ip-10-0-142-6 kubenswrapper[2564]: I0730 00:58:15.435563 2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-772c30-5wnr8"] Jul 30 00:58:15.436011 ip-10-0-142-6 kubenswrapper[2564]: I0730 00:58:15.435879 2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="831c807c-c28e-43e5-a73c-e79a2eacc198" containerName="main" Jul 30 00:58:15.436011 ip-10-0-142-6 kubenswrapper[2564]: I0730 00:58:15.435889 2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="831c807c-c28e-43e5-a73c-e79a2eacc198" containerName="main" Jul 30 00:58:15.436011 ip-10-0-142-6 kubenswrapper[2564]: I0730 00:58:15.435898 2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c4da3216-9857-497f-a5ca-0e193ba681a5" containerName="main" Jul 30 00:58:15.436011 ip-10-0-142-6 kubenswrapper[2564]: I0730 00:58:15.435903 2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="c4da3216-9857-497f-a5ca-0e193ba681a5" containerName="main" Jul 30 00:58:15.436011 ip-10-0-142-6 kubenswrapper[2564]: I0730 00:58:15.435957 2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="c4da3216-9857-497f-a5ca-0e193ba681a5" containerName="main" Jul 30 00:58:15.436011 ip-10-0-142-6 kubenswrapper[2564]: I0730 00:58:15.435965 2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="831c807c-c28e-43e5-a73c-e79a2eacc198" containerName="main" Jul 30 00:58:15.438666 ip-10-0-142-6 kubenswrapper[2564]: I0730 00:58:15.438650 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-772c30-5wnr8" Jul 30 00:58:15.441092 ip-10-0-142-6 kubenswrapper[2564]: I0730 00:58:15.441050 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Jul 30 00:58:15.441210 ip-10-0-142-6 kubenswrapper[2564]: I0730 00:58:15.441131 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-v8q7j\"" Jul 30 00:58:15.447779 ip-10-0-142-6 kubenswrapper[2564]: I0730 00:58:15.447754 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-772c30-5wnr8"] Jul 30 00:58:15.547209 ip-10-0-142-6 kubenswrapper[2564]: I0730 00:58:15.547173 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/2ec1f1d5-774c-4574-bb7a-a4490190f1c9-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-api-a-772c30-5wnr8\" (UID: \"2ec1f1d5-774c-4574-bb7a-a4490190f1c9\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-772c30-5wnr8" Jul 30 00:58:15.648039 ip-10-0-142-6 kubenswrapper[2564]: I0730 00:58:15.647999 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/2ec1f1d5-774c-4574-bb7a-a4490190f1c9-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-api-a-772c30-5wnr8\" (UID: \"2ec1f1d5-774c-4574-bb7a-a4490190f1c9\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-772c30-5wnr8" Jul 30 00:58:15.648675 ip-10-0-142-6 kubenswrapper[2564]: I0730 00:58:15.648654 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/2ec1f1d5-774c-4574-bb7a-a4490190f1c9-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-api-a-772c30-5wnr8\" (UID: \"2ec1f1d5-774c-4574-bb7a-a4490190f1c9\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-772c30-5wnr8" Jul 30 00:58:15.749803 ip-10-0-142-6 kubenswrapper[2564]: I0730 00:58:15.749715 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-772c30-5wnr8" Jul 30 00:58:15.878245 ip-10-0-142-6 kubenswrapper[2564]: I0730 00:58:15.878215 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-772c30-5wnr8"] Jul 30 00:58:15.880128 ip-10-0-142-6 kubenswrapper[2564]: W0730 00:58:15.880099 2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2ec1f1d5_774c_4574_bb7a_a4490190f1c9.slice/crio-fd294eb6701a11f5364a0e6faf2ac2dc53871973b4accb36d43f116783841800 WatchSource:0}: Error finding container fd294eb6701a11f5364a0e6faf2ac2dc53871973b4accb36d43f116783841800: Status 404 returned error can't find the container with id fd294eb6701a11f5364a0e6faf2ac2dc53871973b4accb36d43f116783841800 Jul 30 00:58:15.881930 ip-10-0-142-6 kubenswrapper[2564]: I0730 00:58:15.881913 2564 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Jul 30 00:58:16.461431 ip-10-0-142-6 kubenswrapper[2564]: I0730 00:58:16.461400 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-772c30-5wnr8" event={"ID":"2ec1f1d5-774c-4574-bb7a-a4490190f1c9","Type":"ContainerStarted","Data":"fd294eb6701a11f5364a0e6faf2ac2dc53871973b4accb36d43f116783841800"} Jul 30 00:58:19.478892 ip-10-0-142-6 kubenswrapper[2564]: I0730 00:58:19.478862 2564 generic.go:356] "Generic (PLEG): container finished" podID="2ec1f1d5-774c-4574-bb7a-a4490190f1c9" containerID="5d8cad04efb57fe9da07e9d6b0a648ccf43320db8d6a1e0ec2166e248f42ae2c" exitCode=0 Jul 30 00:58:19.479319 ip-10-0-142-6 kubenswrapper[2564]: I0730 00:58:19.478950 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-772c30-5wnr8" event={"ID":"2ec1f1d5-774c-4574-bb7a-a4490190f1c9","Type":"ContainerDied","Data":"5d8cad04efb57fe9da07e9d6b0a648ccf43320db8d6a1e0ec2166e248f42ae2c"} Jul 30 00:58:20.613540 ip-10-0-142-6 kubenswrapper[2564]: I0730 00:58:20.613512 2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-772c30-5wnr8" Jul 30 00:58:20.694563 ip-10-0-142-6 kubenswrapper[2564]: I0730 00:58:20.694527 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/2ec1f1d5-774c-4574-bb7a-a4490190f1c9-maas-api-service-ca\") pod \"2ec1f1d5-774c-4574-bb7a-a4490190f1c9\" (UID: \"2ec1f1d5-774c-4574-bb7a-a4490190f1c9\") " Jul 30 00:58:20.694893 ip-10-0-142-6 kubenswrapper[2564]: I0730 00:58:20.694868 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2ec1f1d5-774c-4574-bb7a-a4490190f1c9-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "2ec1f1d5-774c-4574-bb7a-a4490190f1c9" (UID: "2ec1f1d5-774c-4574-bb7a-a4490190f1c9"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 30 00:58:20.795532 ip-10-0-142-6 kubenswrapper[2564]: I0730 00:58:20.795443 2564 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/2ec1f1d5-774c-4574-bb7a-a4490190f1c9-maas-api-service-ca\") on node \"ip-10-0-142-6.ec2.internal\" DevicePath \"\"" Jul 30 00:58:21.490201 ip-10-0-142-6 kubenswrapper[2564]: I0730 00:58:21.490096 2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-772c30-5wnr8" Jul 30 00:58:21.490201 ip-10-0-142-6 kubenswrapper[2564]: I0730 00:58:21.490118 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-772c30-5wnr8" event={"ID":"2ec1f1d5-774c-4574-bb7a-a4490190f1c9","Type":"ContainerDied","Data":"fd294eb6701a11f5364a0e6faf2ac2dc53871973b4accb36d43f116783841800"} Jul 30 00:58:21.490201 ip-10-0-142-6 kubenswrapper[2564]: I0730 00:58:21.490168 2564 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="fd294eb6701a11f5364a0e6faf2ac2dc53871973b4accb36d43f116783841800" Jul 30 00:58:31.200510 ip-10-0-142-6 kubenswrapper[2564]: I0730 00:58:31.200426 2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-772c30-5wnr8"] Jul 30 00:58:31.202518 ip-10-0-142-6 kubenswrapper[2564]: I0730 00:58:31.202495 2564 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-772c30-5wnr8"] Jul 30 00:58:33.140173 ip-10-0-142-6 kubenswrapper[2564]: I0730 00:58:33.140133 2564 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="2ec1f1d5-774c-4574-bb7a-a4490190f1c9" path="/var/lib/kubelet/pods/2ec1f1d5-774c-4574-bb7a-a4490190f1c9/volumes" Jul 30 00:58:37.315843 ip-10-0-142-6 kubenswrapper[2564]: I0730 00:58:37.315805 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-nrv2p"] Jul 30 00:58:57.711423 ip-10-0-142-6 kubenswrapper[2564]: I0730 00:58:57.711389 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-nrv2p"] Jul 30 00:59:09.607417 ip-10-0-142-6 kubenswrapper[2564]: I0730 00:59:09.607380 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-nrv2p"] Jul 30 00:59:38.607354 ip-10-0-142-6 kubenswrapper[2564]: I0730 00:59:38.607316 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-nrv2p"] Jul 30 01:00:00.151900 ip-10-0-142-6 kubenswrapper[2564]: I0730 01:00:00.151866 2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29756220-xz969"] Jul 30 01:00:00.152403 ip-10-0-142-6 kubenswrapper[2564]: I0730 01:00:00.152195 2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2ec1f1d5-774c-4574-bb7a-a4490190f1c9" containerName="revoke-keys" Jul 30 01:00:00.152403 ip-10-0-142-6 kubenswrapper[2564]: I0730 01:00:00.152208 2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="2ec1f1d5-774c-4574-bb7a-a4490190f1c9" containerName="revoke-keys" Jul 30 01:00:00.152403 ip-10-0-142-6 kubenswrapper[2564]: I0730 01:00:00.152257 2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="2ec1f1d5-774c-4574-bb7a-a4490190f1c9" containerName="revoke-keys" Jul 30 01:00:00.154975 ip-10-0-142-6 kubenswrapper[2564]: I0730 01:00:00.154960 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29756220-xz969" Jul 30 01:00:00.157866 ip-10-0-142-6 kubenswrapper[2564]: I0730 01:00:00.157846 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-v8q7j\"" Jul 30 01:00:00.164020 ip-10-0-142-6 kubenswrapper[2564]: I0730 01:00:00.163995 2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-a47fe8-29756220-nn56f"] Jul 30 01:00:00.164679 ip-10-0-142-6 kubenswrapper[2564]: I0730 01:00:00.164663 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29756220-xz969" Jul 30 01:00:00.167514 ip-10-0-142-6 kubenswrapper[2564]: I0730 01:00:00.167475 2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-c4c410-29756220-g8hvx"] Jul 30 01:00:00.167706 ip-10-0-142-6 kubenswrapper[2564]: I0730 01:00:00.167686 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-a47fe8-29756220-nn56f" Jul 30 01:00:00.171205 ip-10-0-142-6 kubenswrapper[2564]: I0730 01:00:00.171185 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29756220-xz969"] Jul 30 01:00:00.171316 ip-10-0-142-6 kubenswrapper[2564]: I0730 01:00:00.171295 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-c4c410-29756220-g8hvx" Jul 30 01:00:00.182251 ip-10-0-142-6 kubenswrapper[2564]: I0730 01:00:00.182177 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-c4c410-29756220-g8hvx"] Jul 30 01:00:00.187177 ip-10-0-142-6 kubenswrapper[2564]: I0730 01:00:00.186795 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-a47fe8-29756220-nn56f"] Jul 30 01:00:00.198133 ip-10-0-142-6 kubenswrapper[2564]: I0730 01:00:00.198114 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-a47fe8-29756220-nn56f" Jul 30 01:00:00.210609 ip-10-0-142-6 kubenswrapper[2564]: I0730 01:00:00.210579 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-c4c410-29756220-g8hvx" Jul 30 01:00:00.361563 ip-10-0-142-6 kubenswrapper[2564]: I0730 01:00:00.361530 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-a47fe8-29756220-nn56f"] Jul 30 01:00:00.366185 ip-10-0-142-6 kubenswrapper[2564]: W0730 01:00:00.366141 2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6972ad1f_97e5_496b_ac48_b451868cc732.slice/crio-586fca9d0a25250521920efc6f2a34c179ffa1a503e857c0b2e550706f09a5db WatchSource:0}: Error finding container 586fca9d0a25250521920efc6f2a34c179ffa1a503e857c0b2e550706f09a5db: Status 404 returned error can't find the container with id 586fca9d0a25250521920efc6f2a34c179ffa1a503e857c0b2e550706f09a5db Jul 30 01:00:00.381609 ip-10-0-142-6 kubenswrapper[2564]: I0730 01:00:00.381582 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-c4c410-29756220-g8hvx"] Jul 30 01:00:00.382839 ip-10-0-142-6 kubenswrapper[2564]: W0730 01:00:00.382811 2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc5191b01_4f68_4c40_b828_341dbb28b89d.slice/crio-6d66f92676b4034c4b6de5aa6eec65ddab3448e12b440162575be9a0b2d36e8a WatchSource:0}: Error finding container 6d66f92676b4034c4b6de5aa6eec65ddab3448e12b440162575be9a0b2d36e8a: Status 404 returned error can't find the container with id 6d66f92676b4034c4b6de5aa6eec65ddab3448e12b440162575be9a0b2d36e8a Jul 30 01:00:00.532048 ip-10-0-142-6 kubenswrapper[2564]: I0730 01:00:00.532019 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29756220-xz969"] Jul 30 01:00:00.535296 ip-10-0-142-6 kubenswrapper[2564]: W0730 01:00:00.535274 2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf0fea26f_8c9f_4c0a_bd82_4231140ea823.slice/crio-f4feec5faefac52da0dd5300a1a06df8381f079a1a632a1c15af4de10f8b44ad WatchSource:0}: Error finding container f4feec5faefac52da0dd5300a1a06df8381f079a1a632a1c15af4de10f8b44ad: Status 404 returned error can't find the container with id f4feec5faefac52da0dd5300a1a06df8381f079a1a632a1c15af4de10f8b44ad Jul 30 01:00:00.971966 ip-10-0-142-6 kubenswrapper[2564]: I0730 01:00:00.971933 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-a47fe8-29756220-nn56f" event={"ID":"6972ad1f-97e5-496b-ac48-b451868cc732","Type":"ContainerStarted","Data":"f51b119f10564aae38ef8f697c53cfb624c586d58bd7017fff11c7fbdfd18bbc"} Jul 30 01:00:00.972187 ip-10-0-142-6 kubenswrapper[2564]: I0730 01:00:00.971991 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-a47fe8-29756220-nn56f" event={"ID":"6972ad1f-97e5-496b-ac48-b451868cc732","Type":"ContainerStarted","Data":"586fca9d0a25250521920efc6f2a34c179ffa1a503e857c0b2e550706f09a5db"} Jul 30 01:00:00.973542 ip-10-0-142-6 kubenswrapper[2564]: I0730 01:00:00.973512 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-c4c410-29756220-g8hvx" event={"ID":"c5191b01-4f68-4c40-b828-341dbb28b89d","Type":"ContainerStarted","Data":"09db2cf8be7f10d2ff62c1dca2c9aa6127d8ac3fba95df11f9080a35e2ac4457"} Jul 30 01:00:00.973639 ip-10-0-142-6 kubenswrapper[2564]: I0730 01:00:00.973556 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-c4c410-29756220-g8hvx" event={"ID":"c5191b01-4f68-4c40-b828-341dbb28b89d","Type":"ContainerStarted","Data":"6d66f92676b4034c4b6de5aa6eec65ddab3448e12b440162575be9a0b2d36e8a"} Jul 30 01:00:00.975004 ip-10-0-142-6 kubenswrapper[2564]: I0730 01:00:00.974979 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29756220-xz969" event={"ID":"f0fea26f-8c9f-4c0a-bd82-4231140ea823","Type":"ContainerStarted","Data":"87e4f753416e60dbdfa73412d22c8ff761c0b118d4dbc9d7581295f691f3a192"} Jul 30 01:00:00.975119 ip-10-0-142-6 kubenswrapper[2564]: I0730 01:00:00.975026 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29756220-xz969" event={"ID":"f0fea26f-8c9f-4c0a-bd82-4231140ea823","Type":"ContainerStarted","Data":"f4feec5faefac52da0dd5300a1a06df8381f079a1a632a1c15af4de10f8b44ad"} Jul 30 01:00:00.987860 ip-10-0-142-6 kubenswrapper[2564]: I0730 01:00:00.987776 2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-a47fe8-29756220-nn56f" podStartSLOduration=0.987761579 podStartE2EDuration="987.761579ms" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-30 01:00:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-30 01:00:00.984880614 +0000 UTC m=+2532.399620540" watchObservedRunningTime="2026-07-30 01:00:00.987761579 +0000 UTC m=+2532.402501520" Jul 30 01:00:00.999474 ip-10-0-142-6 kubenswrapper[2564]: I0730 01:00:00.999432 2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29756220-xz969" podStartSLOduration=0.999416579 podStartE2EDuration="999.416579ms" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-30 01:00:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-30 01:00:00.997988409 +0000 UTC m=+2532.412728341" watchObservedRunningTime="2026-07-30 01:00:00.999416579 +0000 UTC m=+2532.414156503" Jul 30 01:00:01.015612 ip-10-0-142-6 kubenswrapper[2564]: I0730 01:00:01.015562 2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-c4c410-29756220-g8hvx" podStartSLOduration=1.015545915 podStartE2EDuration="1.015545915s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-30 01:00:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-30 01:00:01.013393886 +0000 UTC m=+2532.428158273" watchObservedRunningTime="2026-07-30 01:00:01.015545915 +0000 UTC m=+2532.430285848" Jul 30 01:00:01.980936 ip-10-0-142-6 kubenswrapper[2564]: I0730 01:00:01.980894 2564 generic.go:356] "Generic (PLEG): container finished" podID="f0fea26f-8c9f-4c0a-bd82-4231140ea823" containerID="87e4f753416e60dbdfa73412d22c8ff761c0b118d4dbc9d7581295f691f3a192" exitCode=0 Jul 30 01:00:01.981350 ip-10-0-142-6 kubenswrapper[2564]: I0730 01:00:01.980975 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29756220-xz969" event={"ID":"f0fea26f-8c9f-4c0a-bd82-4231140ea823","Type":"ContainerDied","Data":"87e4f753416e60dbdfa73412d22c8ff761c0b118d4dbc9d7581295f691f3a192"} Jul 30 01:00:01.982421 ip-10-0-142-6 kubenswrapper[2564]: I0730 01:00:01.982394 2564 generic.go:356] "Generic (PLEG): container finished" podID="6972ad1f-97e5-496b-ac48-b451868cc732" containerID="f51b119f10564aae38ef8f697c53cfb624c586d58bd7017fff11c7fbdfd18bbc" exitCode=0 Jul 30 01:00:01.982527 ip-10-0-142-6 kubenswrapper[2564]: I0730 01:00:01.982467 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-a47fe8-29756220-nn56f" event={"ID":"6972ad1f-97e5-496b-ac48-b451868cc732","Type":"ContainerDied","Data":"f51b119f10564aae38ef8f697c53cfb624c586d58bd7017fff11c7fbdfd18bbc"} Jul 30 01:00:01.983827 ip-10-0-142-6 kubenswrapper[2564]: I0730 01:00:01.983805 2564 generic.go:356] "Generic (PLEG): container finished" podID="c5191b01-4f68-4c40-b828-341dbb28b89d" containerID="09db2cf8be7f10d2ff62c1dca2c9aa6127d8ac3fba95df11f9080a35e2ac4457" exitCode=0 Jul 30 01:00:01.983895 ip-10-0-142-6 kubenswrapper[2564]: I0730 01:00:01.983840 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-c4c410-29756220-g8hvx" event={"ID":"c5191b01-4f68-4c40-b828-341dbb28b89d","Type":"ContainerDied","Data":"09db2cf8be7f10d2ff62c1dca2c9aa6127d8ac3fba95df11f9080a35e2ac4457"} Jul 30 01:00:03.129524 ip-10-0-142-6 kubenswrapper[2564]: I0730 01:00:03.129488 2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29756220-xz969" Jul 30 01:00:03.170229 ip-10-0-142-6 kubenswrapper[2564]: I0730 01:00:03.170208 2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-a47fe8-29756220-nn56f" Jul 30 01:00:03.173787 ip-10-0-142-6 kubenswrapper[2564]: I0730 01:00:03.173767 2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-c4c410-29756220-g8hvx" Jul 30 01:00:03.994794 ip-10-0-142-6 kubenswrapper[2564]: I0730 01:00:03.994764 2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-c4c410-29756220-g8hvx" Jul 30 01:00:03.994794 ip-10-0-142-6 kubenswrapper[2564]: I0730 01:00:03.994789 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-c4c410-29756220-g8hvx" event={"ID":"c5191b01-4f68-4c40-b828-341dbb28b89d","Type":"ContainerDied","Data":"6d66f92676b4034c4b6de5aa6eec65ddab3448e12b440162575be9a0b2d36e8a"} Jul 30 01:00:03.995047 ip-10-0-142-6 kubenswrapper[2564]: I0730 01:00:03.994841 2564 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="6d66f92676b4034c4b6de5aa6eec65ddab3448e12b440162575be9a0b2d36e8a" Jul 30 01:00:03.996137 ip-10-0-142-6 kubenswrapper[2564]: I0730 01:00:03.996121 2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29756220-xz969" Jul 30 01:00:03.996376 ip-10-0-142-6 kubenswrapper[2564]: I0730 01:00:03.996157 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29756220-xz969" event={"ID":"f0fea26f-8c9f-4c0a-bd82-4231140ea823","Type":"ContainerDied","Data":"f4feec5faefac52da0dd5300a1a06df8381f079a1a632a1c15af4de10f8b44ad"} Jul 30 01:00:03.996376 ip-10-0-142-6 kubenswrapper[2564]: I0730 01:00:03.996190 2564 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="f4feec5faefac52da0dd5300a1a06df8381f079a1a632a1c15af4de10f8b44ad" Jul 30 01:00:03.997475 ip-10-0-142-6 kubenswrapper[2564]: I0730 01:00:03.997459 2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-a47fe8-29756220-nn56f" Jul 30 01:00:03.997475 ip-10-0-142-6 kubenswrapper[2564]: I0730 01:00:03.997460 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-a47fe8-29756220-nn56f" event={"ID":"6972ad1f-97e5-496b-ac48-b451868cc732","Type":"ContainerDied","Data":"586fca9d0a25250521920efc6f2a34c179ffa1a503e857c0b2e550706f09a5db"} Jul 30 01:00:03.997642 ip-10-0-142-6 kubenswrapper[2564]: I0730 01:00:03.997495 2564 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="586fca9d0a25250521920efc6f2a34c179ffa1a503e857c0b2e550706f09a5db" Jul 30 01:00:11.893207 ip-10-0-142-6 kubenswrapper[2564]: I0730 01:00:11.893172 2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-a-a47fe8/auth-test-model-a47fe8-kserve-696b8fb547-49wkh"] Jul 30 01:00:11.893712 ip-10-0-142-6 kubenswrapper[2564]: I0730 01:00:11.893554 2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f0fea26f-8c9f-4c0a-bd82-4231140ea823" containerName="cleanup" Jul 30 01:00:11.893712 ip-10-0-142-6 kubenswrapper[2564]: I0730 01:00:11.893568 2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="f0fea26f-8c9f-4c0a-bd82-4231140ea823" containerName="cleanup" Jul 30 01:00:11.893712 ip-10-0-142-6 kubenswrapper[2564]: I0730 01:00:11.893594 2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c5191b01-4f68-4c40-b828-341dbb28b89d" containerName="cleanup" Jul 30 01:00:11.893712 ip-10-0-142-6 kubenswrapper[2564]: I0730 01:00:11.893604 2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="c5191b01-4f68-4c40-b828-341dbb28b89d" containerName="cleanup" Jul 30 01:00:11.893712 ip-10-0-142-6 kubenswrapper[2564]: I0730 01:00:11.893625 2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6972ad1f-97e5-496b-ac48-b451868cc732" containerName="cleanup" Jul 30 01:00:11.893712 ip-10-0-142-6 kubenswrapper[2564]: I0730 01:00:11.893632 2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="6972ad1f-97e5-496b-ac48-b451868cc732" containerName="cleanup" Jul 30 01:00:11.893712 ip-10-0-142-6 kubenswrapper[2564]: I0730 01:00:11.893712 2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="c5191b01-4f68-4c40-b828-341dbb28b89d" containerName="cleanup" Jul 30 01:00:11.893962 ip-10-0-142-6 kubenswrapper[2564]: I0730 01:00:11.893726 2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="6972ad1f-97e5-496b-ac48-b451868cc732" containerName="cleanup" Jul 30 01:00:11.893962 ip-10-0-142-6 kubenswrapper[2564]: I0730 01:00:11.893738 2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="f0fea26f-8c9f-4c0a-bd82-4231140ea823" containerName="cleanup" Jul 30 01:00:11.896687 ip-10-0-142-6 kubenswrapper[2564]: I0730 01:00:11.896673 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-a47fe8/auth-test-model-a47fe8-kserve-696b8fb547-49wkh" Jul 30 01:00:11.899041 ip-10-0-142-6 kubenswrapper[2564]: I0730 01:00:11.899022 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-a47fe8\"/\"kube-root-ca.crt\"" Jul 30 01:00:11.900103 ip-10-0-142-6 kubenswrapper[2564]: I0730 01:00:11.900059 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-a47fe8\"/\"openshift-service-ca.crt\"" Jul 30 01:00:11.900103 ip-10-0-142-6 kubenswrapper[2564]: I0730 01:00:11.900089 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-a47fe8\"/\"default-dockercfg-skjjk\"" Jul 30 01:00:11.900103 ip-10-0-142-6 kubenswrapper[2564]: I0730 01:00:11.900094 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-a47fe8\"/\"auth-test-model-a47fe8-kserve-self-signed-certs\"" Jul 30 01:00:11.906706 ip-10-0-142-6 kubenswrapper[2564]: I0730 01:00:11.906683 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-a47fe8/auth-test-model-a47fe8-kserve-696b8fb547-49wkh"] Jul 30 01:00:11.926667 ip-10-0-142-6 kubenswrapper[2564]: I0730 01:00:11.926637 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1294cdac-3833-47ea-bf38-0d415391e281-home\") pod \"auth-test-model-a47fe8-kserve-696b8fb547-49wkh\" (UID: \"1294cdac-3833-47ea-bf38-0d415391e281\") " pod="ai-tenant-e2e-shared-a-a47fe8/auth-test-model-a47fe8-kserve-696b8fb547-49wkh" Jul 30 01:00:11.926788 ip-10-0-142-6 kubenswrapper[2564]: I0730 01:00:11.926671 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-brzv4\" (UniqueName: \"kubernetes.io/projected/1294cdac-3833-47ea-bf38-0d415391e281-kube-api-access-brzv4\") pod \"auth-test-model-a47fe8-kserve-696b8fb547-49wkh\" (UID: \"1294cdac-3833-47ea-bf38-0d415391e281\") " pod="ai-tenant-e2e-shared-a-a47fe8/auth-test-model-a47fe8-kserve-696b8fb547-49wkh" Jul 30 01:00:11.926788 ip-10-0-142-6 kubenswrapper[2564]: I0730 01:00:11.926697 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1294cdac-3833-47ea-bf38-0d415391e281-model-cache\") pod \"auth-test-model-a47fe8-kserve-696b8fb547-49wkh\" (UID: \"1294cdac-3833-47ea-bf38-0d415391e281\") " pod="ai-tenant-e2e-shared-a-a47fe8/auth-test-model-a47fe8-kserve-696b8fb547-49wkh" Jul 30 01:00:11.926788 ip-10-0-142-6 kubenswrapper[2564]: I0730 01:00:11.926750 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1294cdac-3833-47ea-bf38-0d415391e281-tls-certs\") pod \"auth-test-model-a47fe8-kserve-696b8fb547-49wkh\" (UID: \"1294cdac-3833-47ea-bf38-0d415391e281\") " pod="ai-tenant-e2e-shared-a-a47fe8/auth-test-model-a47fe8-kserve-696b8fb547-49wkh" Jul 30 01:00:11.926885 ip-10-0-142-6 kubenswrapper[2564]: I0730 01:00:11.926822 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1294cdac-3833-47ea-bf38-0d415391e281-dshm\") pod \"auth-test-model-a47fe8-kserve-696b8fb547-49wkh\" (UID: \"1294cdac-3833-47ea-bf38-0d415391e281\") " pod="ai-tenant-e2e-shared-a-a47fe8/auth-test-model-a47fe8-kserve-696b8fb547-49wkh" Jul 30 01:00:11.926885 ip-10-0-142-6 kubenswrapper[2564]: I0730 01:00:11.926856 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1294cdac-3833-47ea-bf38-0d415391e281-tmp-dir\") pod \"auth-test-model-a47fe8-kserve-696b8fb547-49wkh\" (UID: \"1294cdac-3833-47ea-bf38-0d415391e281\") " pod="ai-tenant-e2e-shared-a-a47fe8/auth-test-model-a47fe8-kserve-696b8fb547-49wkh" Jul 30 01:00:12.028240 ip-10-0-142-6 kubenswrapper[2564]: I0730 01:00:12.028200 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1294cdac-3833-47ea-bf38-0d415391e281-dshm\") pod \"auth-test-model-a47fe8-kserve-696b8fb547-49wkh\" (UID: \"1294cdac-3833-47ea-bf38-0d415391e281\") " pod="ai-tenant-e2e-shared-a-a47fe8/auth-test-model-a47fe8-kserve-696b8fb547-49wkh" Jul 30 01:00:12.028240 ip-10-0-142-6 kubenswrapper[2564]: I0730 01:00:12.028247 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1294cdac-3833-47ea-bf38-0d415391e281-tmp-dir\") pod \"auth-test-model-a47fe8-kserve-696b8fb547-49wkh\" (UID: \"1294cdac-3833-47ea-bf38-0d415391e281\") " pod="ai-tenant-e2e-shared-a-a47fe8/auth-test-model-a47fe8-kserve-696b8fb547-49wkh" Jul 30 01:00:12.028483 ip-10-0-142-6 kubenswrapper[2564]: I0730 01:00:12.028288 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1294cdac-3833-47ea-bf38-0d415391e281-home\") pod \"auth-test-model-a47fe8-kserve-696b8fb547-49wkh\" (UID: \"1294cdac-3833-47ea-bf38-0d415391e281\") " pod="ai-tenant-e2e-shared-a-a47fe8/auth-test-model-a47fe8-kserve-696b8fb547-49wkh" Jul 30 01:00:12.028483 ip-10-0-142-6 kubenswrapper[2564]: I0730 01:00:12.028306 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-brzv4\" (UniqueName: \"kubernetes.io/projected/1294cdac-3833-47ea-bf38-0d415391e281-kube-api-access-brzv4\") pod \"auth-test-model-a47fe8-kserve-696b8fb547-49wkh\" (UID: \"1294cdac-3833-47ea-bf38-0d415391e281\") " pod="ai-tenant-e2e-shared-a-a47fe8/auth-test-model-a47fe8-kserve-696b8fb547-49wkh" Jul 30 01:00:12.028483 ip-10-0-142-6 kubenswrapper[2564]: I0730 01:00:12.028324 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1294cdac-3833-47ea-bf38-0d415391e281-model-cache\") pod \"auth-test-model-a47fe8-kserve-696b8fb547-49wkh\" (UID: \"1294cdac-3833-47ea-bf38-0d415391e281\") " pod="ai-tenant-e2e-shared-a-a47fe8/auth-test-model-a47fe8-kserve-696b8fb547-49wkh" Jul 30 01:00:12.028483 ip-10-0-142-6 kubenswrapper[2564]: I0730 01:00:12.028339 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1294cdac-3833-47ea-bf38-0d415391e281-tls-certs\") pod \"auth-test-model-a47fe8-kserve-696b8fb547-49wkh\" (UID: \"1294cdac-3833-47ea-bf38-0d415391e281\") " pod="ai-tenant-e2e-shared-a-a47fe8/auth-test-model-a47fe8-kserve-696b8fb547-49wkh" Jul 30 01:00:12.028743 ip-10-0-142-6 kubenswrapper[2564]: I0730 01:00:12.028721 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1294cdac-3833-47ea-bf38-0d415391e281-home\") pod \"auth-test-model-a47fe8-kserve-696b8fb547-49wkh\" (UID: \"1294cdac-3833-47ea-bf38-0d415391e281\") " pod="ai-tenant-e2e-shared-a-a47fe8/auth-test-model-a47fe8-kserve-696b8fb547-49wkh" Jul 30 01:00:12.028813 ip-10-0-142-6 kubenswrapper[2564]: I0730 01:00:12.028748 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1294cdac-3833-47ea-bf38-0d415391e281-model-cache\") pod \"auth-test-model-a47fe8-kserve-696b8fb547-49wkh\" (UID: \"1294cdac-3833-47ea-bf38-0d415391e281\") " pod="ai-tenant-e2e-shared-a-a47fe8/auth-test-model-a47fe8-kserve-696b8fb547-49wkh" Jul 30 01:00:12.028849 ip-10-0-142-6 kubenswrapper[2564]: I0730 01:00:12.028804 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1294cdac-3833-47ea-bf38-0d415391e281-tmp-dir\") pod \"auth-test-model-a47fe8-kserve-696b8fb547-49wkh\" (UID: \"1294cdac-3833-47ea-bf38-0d415391e281\") " pod="ai-tenant-e2e-shared-a-a47fe8/auth-test-model-a47fe8-kserve-696b8fb547-49wkh" Jul 30 01:00:12.030453 ip-10-0-142-6 kubenswrapper[2564]: I0730 01:00:12.030435 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1294cdac-3833-47ea-bf38-0d415391e281-dshm\") pod \"auth-test-model-a47fe8-kserve-696b8fb547-49wkh\" (UID: \"1294cdac-3833-47ea-bf38-0d415391e281\") " pod="ai-tenant-e2e-shared-a-a47fe8/auth-test-model-a47fe8-kserve-696b8fb547-49wkh" Jul 30 01:00:12.030719 ip-10-0-142-6 kubenswrapper[2564]: I0730 01:00:12.030702 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1294cdac-3833-47ea-bf38-0d415391e281-tls-certs\") pod \"auth-test-model-a47fe8-kserve-696b8fb547-49wkh\" (UID: \"1294cdac-3833-47ea-bf38-0d415391e281\") " pod="ai-tenant-e2e-shared-a-a47fe8/auth-test-model-a47fe8-kserve-696b8fb547-49wkh" Jul 30 01:00:12.042236 ip-10-0-142-6 kubenswrapper[2564]: I0730 01:00:12.042203 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-brzv4\" (UniqueName: \"kubernetes.io/projected/1294cdac-3833-47ea-bf38-0d415391e281-kube-api-access-brzv4\") pod \"auth-test-model-a47fe8-kserve-696b8fb547-49wkh\" (UID: \"1294cdac-3833-47ea-bf38-0d415391e281\") " pod="ai-tenant-e2e-shared-a-a47fe8/auth-test-model-a47fe8-kserve-696b8fb547-49wkh" Jul 30 01:00:12.207761 ip-10-0-142-6 kubenswrapper[2564]: I0730 01:00:12.207671 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-a47fe8/auth-test-model-a47fe8-kserve-696b8fb547-49wkh" Jul 30 01:00:12.351490 ip-10-0-142-6 kubenswrapper[2564]: I0730 01:00:12.351466 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-a47fe8/auth-test-model-a47fe8-kserve-696b8fb547-49wkh"] Jul 30 01:00:12.354040 ip-10-0-142-6 kubenswrapper[2564]: W0730 01:00:12.354013 2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod1294cdac_3833_47ea_bf38_0d415391e281.slice/crio-73267502de47a1767bb7c2b6dabf5c4afdfa9d21a2c8767d7fbc09f8d57be358 WatchSource:0}: Error finding container 73267502de47a1767bb7c2b6dabf5c4afdfa9d21a2c8767d7fbc09f8d57be358: Status 404 returned error can't find the container with id 73267502de47a1767bb7c2b6dabf5c4afdfa9d21a2c8767d7fbc09f8d57be358 Jul 30 01:00:13.049795 ip-10-0-142-6 kubenswrapper[2564]: I0730 01:00:13.049761 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-a47fe8/auth-test-model-a47fe8-kserve-696b8fb547-49wkh" event={"ID":"1294cdac-3833-47ea-bf38-0d415391e281","Type":"ContainerStarted","Data":"fcba08c58989f7318acb6fe07d0d751a5b782523c652e1a43535954b9ec507f2"} Jul 30 01:00:13.049795 ip-10-0-142-6 kubenswrapper[2564]: I0730 01:00:13.049801 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-a47fe8/auth-test-model-a47fe8-kserve-696b8fb547-49wkh" event={"ID":"1294cdac-3833-47ea-bf38-0d415391e281","Type":"ContainerStarted","Data":"73267502de47a1767bb7c2b6dabf5c4afdfa9d21a2c8767d7fbc09f8d57be358"} Jul 30 01:00:13.067526 ip-10-0-142-6 kubenswrapper[2564]: I0730 01:00:13.067478 2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-a-a47fe8/auth-test-model-a47fe8-kserve-696b8fb547-49wkh" podStartSLOduration=2.06746262 podStartE2EDuration="2.06746262s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-30 01:00:11 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-30 01:00:13.066295289 +0000 UTC m=+2544.481035224" watchObservedRunningTime="2026-07-30 01:00:13.06746262 +0000 UTC m=+2544.482202572" Jul 30 01:00:15.600692 ip-10-0-142-6 kubenswrapper[2564]: I0730 01:00:15.600657 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-nrv2p"] Jul 30 01:00:22.208646 ip-10-0-142-6 kubenswrapper[2564]: I0730 01:00:22.208601 2564 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-a-a47fe8/auth-test-model-a47fe8-kserve-696b8fb547-49wkh" Jul 30 01:00:22.208646 ip-10-0-142-6 kubenswrapper[2564]: I0730 01:00:22.208660 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-a-a47fe8/auth-test-model-a47fe8-kserve-696b8fb547-49wkh" Jul 30 01:00:22.221580 ip-10-0-142-6 kubenswrapper[2564]: I0730 01:00:22.221556 2564 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-a-a47fe8/auth-test-model-a47fe8-kserve-696b8fb547-49wkh" Jul 30 01:00:23.109141 ip-10-0-142-6 kubenswrapper[2564]: I0730 01:00:23.109111 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-a-a47fe8/auth-test-model-a47fe8-kserve-696b8fb547-49wkh" Jul 30 01:00:38.022742 ip-10-0-142-6 kubenswrapper[2564]: I0730 01:00:38.022708 2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-b-c4c410/auth-test-model-c4c410-kserve-5cb6c845dc-mvjj7"] Jul 30 01:00:38.026194 ip-10-0-142-6 kubenswrapper[2564]: I0730 01:00:38.026172 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-c4c410/auth-test-model-c4c410-kserve-5cb6c845dc-mvjj7" Jul 30 01:00:38.029621 ip-10-0-142-6 kubenswrapper[2564]: I0730 01:00:38.029589 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-c4c410\"/\"kube-root-ca.crt\"" Jul 30 01:00:38.029745 ip-10-0-142-6 kubenswrapper[2564]: I0730 01:00:38.029589 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-c4c410\"/\"openshift-service-ca.crt\"" Jul 30 01:00:38.029745 ip-10-0-142-6 kubenswrapper[2564]: I0730 01:00:38.029589 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-c4c410\"/\"default-dockercfg-62jh9\"" Jul 30 01:00:38.029745 ip-10-0-142-6 kubenswrapper[2564]: I0730 01:00:38.029589 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-c4c410\"/\"auth-test-model-c4c410-kserve-self-signed-certs\"" Jul 30 01:00:38.037369 ip-10-0-142-6 kubenswrapper[2564]: I0730 01:00:38.037345 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-c4c410/auth-test-model-c4c410-kserve-5cb6c845dc-mvjj7"] Jul 30 01:00:38.154237 ip-10-0-142-6 kubenswrapper[2564]: I0730 01:00:38.154198 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/24aa6ed0-aa50-4b8f-8180-a1a12b909666-dshm\") pod \"auth-test-model-c4c410-kserve-5cb6c845dc-mvjj7\" (UID: \"24aa6ed0-aa50-4b8f-8180-a1a12b909666\") " pod="ai-tenant-e2e-shared-b-c4c410/auth-test-model-c4c410-kserve-5cb6c845dc-mvjj7" Jul 30 01:00:38.154237 ip-10-0-142-6 kubenswrapper[2564]: I0730 01:00:38.154240 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-f4c4d\" (UniqueName: \"kubernetes.io/projected/24aa6ed0-aa50-4b8f-8180-a1a12b909666-kube-api-access-f4c4d\") pod \"auth-test-model-c4c410-kserve-5cb6c845dc-mvjj7\" (UID: \"24aa6ed0-aa50-4b8f-8180-a1a12b909666\") " pod="ai-tenant-e2e-shared-b-c4c410/auth-test-model-c4c410-kserve-5cb6c845dc-mvjj7" Jul 30 01:00:38.154456 ip-10-0-142-6 kubenswrapper[2564]: I0730 01:00:38.154336 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/24aa6ed0-aa50-4b8f-8180-a1a12b909666-home\") pod \"auth-test-model-c4c410-kserve-5cb6c845dc-mvjj7\" (UID: \"24aa6ed0-aa50-4b8f-8180-a1a12b909666\") " pod="ai-tenant-e2e-shared-b-c4c410/auth-test-model-c4c410-kserve-5cb6c845dc-mvjj7" Jul 30 01:00:38.154456 ip-10-0-142-6 kubenswrapper[2564]: I0730 01:00:38.154379 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/24aa6ed0-aa50-4b8f-8180-a1a12b909666-model-cache\") pod \"auth-test-model-c4c410-kserve-5cb6c845dc-mvjj7\" (UID: \"24aa6ed0-aa50-4b8f-8180-a1a12b909666\") " pod="ai-tenant-e2e-shared-b-c4c410/auth-test-model-c4c410-kserve-5cb6c845dc-mvjj7" Jul 30 01:00:38.154456 ip-10-0-142-6 kubenswrapper[2564]: I0730 01:00:38.154418 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/24aa6ed0-aa50-4b8f-8180-a1a12b909666-tls-certs\") pod \"auth-test-model-c4c410-kserve-5cb6c845dc-mvjj7\" (UID: \"24aa6ed0-aa50-4b8f-8180-a1a12b909666\") " pod="ai-tenant-e2e-shared-b-c4c410/auth-test-model-c4c410-kserve-5cb6c845dc-mvjj7" Jul 30 01:00:38.154456 ip-10-0-142-6 kubenswrapper[2564]: I0730 01:00:38.154444 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/24aa6ed0-aa50-4b8f-8180-a1a12b909666-tmp-dir\") pod \"auth-test-model-c4c410-kserve-5cb6c845dc-mvjj7\" (UID: \"24aa6ed0-aa50-4b8f-8180-a1a12b909666\") " pod="ai-tenant-e2e-shared-b-c4c410/auth-test-model-c4c410-kserve-5cb6c845dc-mvjj7" Jul 30 01:00:38.255323 ip-10-0-142-6 kubenswrapper[2564]: I0730 01:00:38.255281 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/24aa6ed0-aa50-4b8f-8180-a1a12b909666-dshm\") pod \"auth-test-model-c4c410-kserve-5cb6c845dc-mvjj7\" (UID: \"24aa6ed0-aa50-4b8f-8180-a1a12b909666\") " pod="ai-tenant-e2e-shared-b-c4c410/auth-test-model-c4c410-kserve-5cb6c845dc-mvjj7" Jul 30 01:00:38.255323 ip-10-0-142-6 kubenswrapper[2564]: I0730 01:00:38.255324 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f4c4d\" (UniqueName: \"kubernetes.io/projected/24aa6ed0-aa50-4b8f-8180-a1a12b909666-kube-api-access-f4c4d\") pod \"auth-test-model-c4c410-kserve-5cb6c845dc-mvjj7\" (UID: \"24aa6ed0-aa50-4b8f-8180-a1a12b909666\") " pod="ai-tenant-e2e-shared-b-c4c410/auth-test-model-c4c410-kserve-5cb6c845dc-mvjj7" Jul 30 01:00:38.255583 ip-10-0-142-6 kubenswrapper[2564]: I0730 01:00:38.255364 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/24aa6ed0-aa50-4b8f-8180-a1a12b909666-home\") pod \"auth-test-model-c4c410-kserve-5cb6c845dc-mvjj7\" (UID: \"24aa6ed0-aa50-4b8f-8180-a1a12b909666\") " pod="ai-tenant-e2e-shared-b-c4c410/auth-test-model-c4c410-kserve-5cb6c845dc-mvjj7" Jul 30 01:00:38.255583 ip-10-0-142-6 kubenswrapper[2564]: I0730 01:00:38.255394 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/24aa6ed0-aa50-4b8f-8180-a1a12b909666-model-cache\") pod \"auth-test-model-c4c410-kserve-5cb6c845dc-mvjj7\" (UID: \"24aa6ed0-aa50-4b8f-8180-a1a12b909666\") " pod="ai-tenant-e2e-shared-b-c4c410/auth-test-model-c4c410-kserve-5cb6c845dc-mvjj7" Jul 30 01:00:38.255583 ip-10-0-142-6 kubenswrapper[2564]: I0730 01:00:38.255441 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/24aa6ed0-aa50-4b8f-8180-a1a12b909666-tls-certs\") pod \"auth-test-model-c4c410-kserve-5cb6c845dc-mvjj7\" (UID: \"24aa6ed0-aa50-4b8f-8180-a1a12b909666\") " pod="ai-tenant-e2e-shared-b-c4c410/auth-test-model-c4c410-kserve-5cb6c845dc-mvjj7" Jul 30 01:00:38.255583 ip-10-0-142-6 kubenswrapper[2564]: I0730 01:00:38.255470 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/24aa6ed0-aa50-4b8f-8180-a1a12b909666-tmp-dir\") pod \"auth-test-model-c4c410-kserve-5cb6c845dc-mvjj7\" (UID: \"24aa6ed0-aa50-4b8f-8180-a1a12b909666\") " pod="ai-tenant-e2e-shared-b-c4c410/auth-test-model-c4c410-kserve-5cb6c845dc-mvjj7" Jul 30 01:00:38.255828 ip-10-0-142-6 kubenswrapper[2564]: I0730 01:00:38.255803 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/24aa6ed0-aa50-4b8f-8180-a1a12b909666-home\") pod \"auth-test-model-c4c410-kserve-5cb6c845dc-mvjj7\" (UID: \"24aa6ed0-aa50-4b8f-8180-a1a12b909666\") " pod="ai-tenant-e2e-shared-b-c4c410/auth-test-model-c4c410-kserve-5cb6c845dc-mvjj7" Jul 30 01:00:38.255897 ip-10-0-142-6 kubenswrapper[2564]: I0730 01:00:38.255837 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/24aa6ed0-aa50-4b8f-8180-a1a12b909666-model-cache\") pod \"auth-test-model-c4c410-kserve-5cb6c845dc-mvjj7\" (UID: \"24aa6ed0-aa50-4b8f-8180-a1a12b909666\") " pod="ai-tenant-e2e-shared-b-c4c410/auth-test-model-c4c410-kserve-5cb6c845dc-mvjj7" Jul 30 01:00:38.255897 ip-10-0-142-6 kubenswrapper[2564]: I0730 01:00:38.255879 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/24aa6ed0-aa50-4b8f-8180-a1a12b909666-tmp-dir\") pod \"auth-test-model-c4c410-kserve-5cb6c845dc-mvjj7\" (UID: \"24aa6ed0-aa50-4b8f-8180-a1a12b909666\") " pod="ai-tenant-e2e-shared-b-c4c410/auth-test-model-c4c410-kserve-5cb6c845dc-mvjj7" Jul 30 01:00:38.257687 ip-10-0-142-6 kubenswrapper[2564]: I0730 01:00:38.257661 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/24aa6ed0-aa50-4b8f-8180-a1a12b909666-dshm\") pod \"auth-test-model-c4c410-kserve-5cb6c845dc-mvjj7\" (UID: \"24aa6ed0-aa50-4b8f-8180-a1a12b909666\") " pod="ai-tenant-e2e-shared-b-c4c410/auth-test-model-c4c410-kserve-5cb6c845dc-mvjj7" Jul 30 01:00:38.257862 ip-10-0-142-6 kubenswrapper[2564]: I0730 01:00:38.257846 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/24aa6ed0-aa50-4b8f-8180-a1a12b909666-tls-certs\") pod \"auth-test-model-c4c410-kserve-5cb6c845dc-mvjj7\" (UID: \"24aa6ed0-aa50-4b8f-8180-a1a12b909666\") " pod="ai-tenant-e2e-shared-b-c4c410/auth-test-model-c4c410-kserve-5cb6c845dc-mvjj7" Jul 30 01:00:38.264448 ip-10-0-142-6 kubenswrapper[2564]: I0730 01:00:38.264423 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-f4c4d\" (UniqueName: \"kubernetes.io/projected/24aa6ed0-aa50-4b8f-8180-a1a12b909666-kube-api-access-f4c4d\") pod \"auth-test-model-c4c410-kserve-5cb6c845dc-mvjj7\" (UID: \"24aa6ed0-aa50-4b8f-8180-a1a12b909666\") " pod="ai-tenant-e2e-shared-b-c4c410/auth-test-model-c4c410-kserve-5cb6c845dc-mvjj7" Jul 30 01:00:38.338354 ip-10-0-142-6 kubenswrapper[2564]: I0730 01:00:38.338265 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-c4c410/auth-test-model-c4c410-kserve-5cb6c845dc-mvjj7" Jul 30 01:00:38.470852 ip-10-0-142-6 kubenswrapper[2564]: I0730 01:00:38.470823 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-c4c410/auth-test-model-c4c410-kserve-5cb6c845dc-mvjj7"] Jul 30 01:00:38.472523 ip-10-0-142-6 kubenswrapper[2564]: W0730 01:00:38.472500 2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod24aa6ed0_aa50_4b8f_8180_a1a12b909666.slice/crio-96130526d55674a099193ce5f31c40514896e1ebcb8d744de93c61cbd0cb0ef7 WatchSource:0}: Error finding container 96130526d55674a099193ce5f31c40514896e1ebcb8d744de93c61cbd0cb0ef7: Status 404 returned error can't find the container with id 96130526d55674a099193ce5f31c40514896e1ebcb8d744de93c61cbd0cb0ef7 Jul 30 01:00:39.178741 ip-10-0-142-6 kubenswrapper[2564]: I0730 01:00:39.178698 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-c4c410/auth-test-model-c4c410-kserve-5cb6c845dc-mvjj7" event={"ID":"24aa6ed0-aa50-4b8f-8180-a1a12b909666","Type":"ContainerStarted","Data":"0478a502f50c2a0895bf3ad79498d3a77c051bd77720ebf96b78656916af0705"} Jul 30 01:00:39.179190 ip-10-0-142-6 kubenswrapper[2564]: I0730 01:00:39.178749 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-c4c410/auth-test-model-c4c410-kserve-5cb6c845dc-mvjj7" event={"ID":"24aa6ed0-aa50-4b8f-8180-a1a12b909666","Type":"ContainerStarted","Data":"96130526d55674a099193ce5f31c40514896e1ebcb8d744de93c61cbd0cb0ef7"} Jul 30 01:00:39.196130 ip-10-0-142-6 kubenswrapper[2564]: I0730 01:00:39.196052 2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-b-c4c410/auth-test-model-c4c410-kserve-5cb6c845dc-mvjj7" podStartSLOduration=1.1960342800000001 podStartE2EDuration="1.19603428s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-30 01:00:38 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-30 01:00:39.193691625 +0000 UTC m=+2570.608431593" watchObservedRunningTime="2026-07-30 01:00:39.19603428 +0000 UTC m=+2570.610774212" Jul 30 01:00:41.711287 ip-10-0-142-6 kubenswrapper[2564]: I0730 01:00:41.711253 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-nrv2p"] Jul 30 01:00:48.339235 ip-10-0-142-6 kubenswrapper[2564]: I0730 01:00:48.339195 2564 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-b-c4c410/auth-test-model-c4c410-kserve-5cb6c845dc-mvjj7" Jul 30 01:00:48.339666 ip-10-0-142-6 kubenswrapper[2564]: I0730 01:00:48.339254 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-b-c4c410/auth-test-model-c4c410-kserve-5cb6c845dc-mvjj7" Jul 30 01:00:48.351786 ip-10-0-142-6 kubenswrapper[2564]: I0730 01:00:48.351755 2564 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-b-c4c410/auth-test-model-c4c410-kserve-5cb6c845dc-mvjj7" Jul 30 01:00:49.239692 ip-10-0-142-6 kubenswrapper[2564]: I0730 01:00:49.239659 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-b-c4c410/auth-test-model-c4c410-kserve-5cb6c845dc-mvjj7" Jul 30 01:01:07.310265 ip-10-0-142-6 kubenswrapper[2564]: I0730 01:01:07.310230 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-nrv2p"] Jul 30 01:01:28.306804 ip-10-0-142-6 kubenswrapper[2564]: I0730 01:01:28.306722 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-nrv2p"] Jul 30 01:01:41.909346 ip-10-0-142-6 kubenswrapper[2564]: I0730 01:01:41.909309 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-nrv2p"] Jul 30 01:01:54.016423 ip-10-0-142-6 kubenswrapper[2564]: I0730 01:01:54.016383 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-nrv2p"] Jul 30 01:02:11.411812 ip-10-0-142-6 kubenswrapper[2564]: I0730 01:02:11.411777 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-nrv2p"] Jul 30 01:02:30.311212 ip-10-0-142-6 kubenswrapper[2564]: I0730 01:02:30.311175 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-nrv2p"] Jul 30 01:02:53.203486 ip-10-0-142-6 kubenswrapper[2564]: I0730 01:02:53.203406 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-nrv2p"] Jul 30 01:03:10.015245 ip-10-0-142-6 kubenswrapper[2564]: I0730 01:03:10.015209 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-nrv2p"] Jul 30 01:03:31.327262 ip-10-0-142-6 kubenswrapper[2564]: I0730 01:03:31.327231 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-nrv2p"] Jul 30 01:03:48.909651 ip-10-0-142-6 kubenswrapper[2564]: I0730 01:03:48.909612 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-nrv2p"] Jul 30 01:04:09.111519 ip-10-0-142-6 kubenswrapper[2564]: I0730 01:04:09.111481 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-nrv2p"] Jul 30 01:04:26.517547 ip-10-0-142-6 kubenswrapper[2564]: I0730 01:04:26.517469 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-nrv2p"] Jul 30 01:04:47.615933 ip-10-0-142-6 kubenswrapper[2564]: I0730 01:04:47.615897 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-nrv2p"] Jul 30 01:04:49.297552 ip-10-0-142-6 kubenswrapper[2564]: I0730 01:04:49.297520 2564 scope.go:117] "RemoveContainer" containerID="5d8cad04efb57fe9da07e9d6b0a648ccf43320db8d6a1e0ec2166e248f42ae2c" Jul 30 01:04:52.123316 ip-10-0-142-6 kubenswrapper[2564]: I0730 01:04:52.123281 2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-a47fe8/auth-test-model-a47fe8-kserve-696b8fb547-49wkh"] Jul 30 01:04:53.020435 ip-10-0-142-6 kubenswrapper[2564]: I0730 01:04:53.020401 2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-c4c410/auth-test-model-c4c410-kserve-5cb6c845dc-mvjj7"] Jul 30 01:05:03.915890 ip-10-0-142-6 kubenswrapper[2564]: I0730 01:05:03.915849 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-nrv2p"] Jul 30 01:05:07.141840 ip-10-0-142-6 kubenswrapper[2564]: I0730 01:05:07.141785 2564 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-shared-a-a47fe8/auth-test-model-a47fe8-kserve-696b8fb547-49wkh" podUID="1294cdac-3833-47ea-bf38-0d415391e281" containerName="main" containerID="cri-o://fcba08c58989f7318acb6fe07d0d751a5b782523c652e1a43535954b9ec507f2" gracePeriod=45 Jul 30 01:05:07.394511 ip-10-0-142-6 kubenswrapper[2564]: I0730 01:05:07.394454 2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-a47fe8/auth-test-model-a47fe8-kserve-696b8fb547-49wkh" Jul 30 01:05:07.457562 ip-10-0-142-6 kubenswrapper[2564]: I0730 01:05:07.457523 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1294cdac-3833-47ea-bf38-0d415391e281-tmp-dir\") pod \"1294cdac-3833-47ea-bf38-0d415391e281\" (UID: \"1294cdac-3833-47ea-bf38-0d415391e281\") " Jul 30 01:05:07.457727 ip-10-0-142-6 kubenswrapper[2564]: I0730 01:05:07.457572 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1294cdac-3833-47ea-bf38-0d415391e281-dshm\") pod \"1294cdac-3833-47ea-bf38-0d415391e281\" (UID: \"1294cdac-3833-47ea-bf38-0d415391e281\") " Jul 30 01:05:07.457727 ip-10-0-142-6 kubenswrapper[2564]: I0730 01:05:07.457663 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1294cdac-3833-47ea-bf38-0d415391e281-model-cache\") pod \"1294cdac-3833-47ea-bf38-0d415391e281\" (UID: \"1294cdac-3833-47ea-bf38-0d415391e281\") " Jul 30 01:05:07.457727 ip-10-0-142-6 kubenswrapper[2564]: I0730 01:05:07.457690 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1294cdac-3833-47ea-bf38-0d415391e281-tls-certs\") pod \"1294cdac-3833-47ea-bf38-0d415391e281\" (UID: \"1294cdac-3833-47ea-bf38-0d415391e281\") " Jul 30 01:05:07.457877 ip-10-0-142-6 kubenswrapper[2564]: I0730 01:05:07.457734 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-brzv4\" (UniqueName: \"kubernetes.io/projected/1294cdac-3833-47ea-bf38-0d415391e281-kube-api-access-brzv4\") pod \"1294cdac-3833-47ea-bf38-0d415391e281\" (UID: \"1294cdac-3833-47ea-bf38-0d415391e281\") " Jul 30 01:05:07.457877 ip-10-0-142-6 kubenswrapper[2564]: I0730 01:05:07.457768 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1294cdac-3833-47ea-bf38-0d415391e281-home\") pod \"1294cdac-3833-47ea-bf38-0d415391e281\" (UID: \"1294cdac-3833-47ea-bf38-0d415391e281\") " Jul 30 01:05:07.457877 ip-10-0-142-6 kubenswrapper[2564]: I0730 01:05:07.457800 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1294cdac-3833-47ea-bf38-0d415391e281-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "1294cdac-3833-47ea-bf38-0d415391e281" (UID: "1294cdac-3833-47ea-bf38-0d415391e281"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 30 01:05:07.458043 ip-10-0-142-6 kubenswrapper[2564]: I0730 01:05:07.457912 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1294cdac-3833-47ea-bf38-0d415391e281-model-cache" (OuterVolumeSpecName: "model-cache") pod "1294cdac-3833-47ea-bf38-0d415391e281" (UID: "1294cdac-3833-47ea-bf38-0d415391e281"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 30 01:05:07.458043 ip-10-0-142-6 kubenswrapper[2564]: I0730 01:05:07.458031 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1294cdac-3833-47ea-bf38-0d415391e281-home" (OuterVolumeSpecName: "home") pod "1294cdac-3833-47ea-bf38-0d415391e281" (UID: "1294cdac-3833-47ea-bf38-0d415391e281"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 30 01:05:07.458190 ip-10-0-142-6 kubenswrapper[2564]: I0730 01:05:07.458101 2564 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1294cdac-3833-47ea-bf38-0d415391e281-model-cache\") on node \"ip-10-0-142-6.ec2.internal\" DevicePath \"\"" Jul 30 01:05:07.458190 ip-10-0-142-6 kubenswrapper[2564]: I0730 01:05:07.458119 2564 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1294cdac-3833-47ea-bf38-0d415391e281-tmp-dir\") on node \"ip-10-0-142-6.ec2.internal\" DevicePath \"\"" Jul 30 01:05:07.459928 ip-10-0-142-6 kubenswrapper[2564]: I0730 01:05:07.459897 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1294cdac-3833-47ea-bf38-0d415391e281-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "1294cdac-3833-47ea-bf38-0d415391e281" (UID: "1294cdac-3833-47ea-bf38-0d415391e281"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 30 01:05:07.460037 ip-10-0-142-6 kubenswrapper[2564]: I0730 01:05:07.459926 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1294cdac-3833-47ea-bf38-0d415391e281-dshm" (OuterVolumeSpecName: "dshm") pod "1294cdac-3833-47ea-bf38-0d415391e281" (UID: "1294cdac-3833-47ea-bf38-0d415391e281"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 30 01:05:07.460037 ip-10-0-142-6 kubenswrapper[2564]: I0730 01:05:07.459959 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1294cdac-3833-47ea-bf38-0d415391e281-kube-api-access-brzv4" (OuterVolumeSpecName: "kube-api-access-brzv4") pod "1294cdac-3833-47ea-bf38-0d415391e281" (UID: "1294cdac-3833-47ea-bf38-0d415391e281"). InnerVolumeSpecName "kube-api-access-brzv4". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 30 01:05:07.537486 ip-10-0-142-6 kubenswrapper[2564]: I0730 01:05:07.537455 2564 generic.go:356] "Generic (PLEG): container finished" podID="1294cdac-3833-47ea-bf38-0d415391e281" containerID="fcba08c58989f7318acb6fe07d0d751a5b782523c652e1a43535954b9ec507f2" exitCode=0 Jul 30 01:05:07.537664 ip-10-0-142-6 kubenswrapper[2564]: I0730 01:05:07.537494 2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-a47fe8/auth-test-model-a47fe8-kserve-696b8fb547-49wkh" Jul 30 01:05:07.537664 ip-10-0-142-6 kubenswrapper[2564]: I0730 01:05:07.537500 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-a47fe8/auth-test-model-a47fe8-kserve-696b8fb547-49wkh" event={"ID":"1294cdac-3833-47ea-bf38-0d415391e281","Type":"ContainerDied","Data":"fcba08c58989f7318acb6fe07d0d751a5b782523c652e1a43535954b9ec507f2"} Jul 30 01:05:07.537664 ip-10-0-142-6 kubenswrapper[2564]: I0730 01:05:07.537533 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-a47fe8/auth-test-model-a47fe8-kserve-696b8fb547-49wkh" event={"ID":"1294cdac-3833-47ea-bf38-0d415391e281","Type":"ContainerDied","Data":"73267502de47a1767bb7c2b6dabf5c4afdfa9d21a2c8767d7fbc09f8d57be358"} Jul 30 01:05:07.537664 ip-10-0-142-6 kubenswrapper[2564]: I0730 01:05:07.537551 2564 scope.go:117] "RemoveContainer" containerID="fcba08c58989f7318acb6fe07d0d751a5b782523c652e1a43535954b9ec507f2" Jul 30 01:05:07.546322 ip-10-0-142-6 kubenswrapper[2564]: I0730 01:05:07.546303 2564 scope.go:117] "RemoveContainer" containerID="fcba08c58989f7318acb6fe07d0d751a5b782523c652e1a43535954b9ec507f2" Jul 30 01:05:07.546608 ip-10-0-142-6 kubenswrapper[2564]: E0730 01:05:07.546586 2564 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fcba08c58989f7318acb6fe07d0d751a5b782523c652e1a43535954b9ec507f2\": container with ID starting with fcba08c58989f7318acb6fe07d0d751a5b782523c652e1a43535954b9ec507f2 not found: ID does not exist" containerID="fcba08c58989f7318acb6fe07d0d751a5b782523c652e1a43535954b9ec507f2" Jul 30 01:05:07.546683 ip-10-0-142-6 kubenswrapper[2564]: I0730 01:05:07.546621 2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fcba08c58989f7318acb6fe07d0d751a5b782523c652e1a43535954b9ec507f2"} err="failed to get container status \"fcba08c58989f7318acb6fe07d0d751a5b782523c652e1a43535954b9ec507f2\": rpc error: code = NotFound desc = could not find container \"fcba08c58989f7318acb6fe07d0d751a5b782523c652e1a43535954b9ec507f2\": container with ID starting with fcba08c58989f7318acb6fe07d0d751a5b782523c652e1a43535954b9ec507f2 not found: ID does not exist" Jul 30 01:05:07.558667 ip-10-0-142-6 kubenswrapper[2564]: I0730 01:05:07.558636 2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-a47fe8/auth-test-model-a47fe8-kserve-696b8fb547-49wkh"] Jul 30 01:05:07.559016 ip-10-0-142-6 kubenswrapper[2564]: I0730 01:05:07.558993 2564 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1294cdac-3833-47ea-bf38-0d415391e281-tls-certs\") on node \"ip-10-0-142-6.ec2.internal\" DevicePath \"\"" Jul 30 01:05:07.559131 ip-10-0-142-6 kubenswrapper[2564]: I0730 01:05:07.559023 2564 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-brzv4\" (UniqueName: \"kubernetes.io/projected/1294cdac-3833-47ea-bf38-0d415391e281-kube-api-access-brzv4\") on node \"ip-10-0-142-6.ec2.internal\" DevicePath \"\"" Jul 30 01:05:07.559131 ip-10-0-142-6 kubenswrapper[2564]: I0730 01:05:07.559045 2564 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1294cdac-3833-47ea-bf38-0d415391e281-home\") on node \"ip-10-0-142-6.ec2.internal\" DevicePath \"\"" Jul 30 01:05:07.559131 ip-10-0-142-6 kubenswrapper[2564]: I0730 01:05:07.559060 2564 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1294cdac-3833-47ea-bf38-0d415391e281-dshm\") on node \"ip-10-0-142-6.ec2.internal\" DevicePath \"\"" Jul 30 01:05:07.561951 ip-10-0-142-6 kubenswrapper[2564]: I0730 01:05:07.561925 2564 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-a-a47fe8/auth-test-model-a47fe8-kserve-696b8fb547-49wkh"] Jul 30 01:05:08.039470 ip-10-0-142-6 kubenswrapper[2564]: I0730 01:05:08.039421 2564 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-shared-b-c4c410/auth-test-model-c4c410-kserve-5cb6c845dc-mvjj7" podUID="24aa6ed0-aa50-4b8f-8180-a1a12b909666" containerName="main" containerID="cri-o://0478a502f50c2a0895bf3ad79498d3a77c051bd77720ebf96b78656916af0705" gracePeriod=45 Jul 30 01:05:08.288497 ip-10-0-142-6 kubenswrapper[2564]: I0730 01:05:08.288475 2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-c4c410/auth-test-model-c4c410-kserve-5cb6c845dc-mvjj7" Jul 30 01:05:08.365621 ip-10-0-142-6 kubenswrapper[2564]: I0730 01:05:08.365536 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/24aa6ed0-aa50-4b8f-8180-a1a12b909666-dshm\") pod \"24aa6ed0-aa50-4b8f-8180-a1a12b909666\" (UID: \"24aa6ed0-aa50-4b8f-8180-a1a12b909666\") " Jul 30 01:05:08.365621 ip-10-0-142-6 kubenswrapper[2564]: I0730 01:05:08.365601 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-f4c4d\" (UniqueName: \"kubernetes.io/projected/24aa6ed0-aa50-4b8f-8180-a1a12b909666-kube-api-access-f4c4d\") pod \"24aa6ed0-aa50-4b8f-8180-a1a12b909666\" (UID: \"24aa6ed0-aa50-4b8f-8180-a1a12b909666\") " Jul 30 01:05:08.365867 ip-10-0-142-6 kubenswrapper[2564]: I0730 01:05:08.365658 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/24aa6ed0-aa50-4b8f-8180-a1a12b909666-home\") pod \"24aa6ed0-aa50-4b8f-8180-a1a12b909666\" (UID: \"24aa6ed0-aa50-4b8f-8180-a1a12b909666\") " Jul 30 01:05:08.365867 ip-10-0-142-6 kubenswrapper[2564]: I0730 01:05:08.365683 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/24aa6ed0-aa50-4b8f-8180-a1a12b909666-tmp-dir\") pod \"24aa6ed0-aa50-4b8f-8180-a1a12b909666\" (UID: \"24aa6ed0-aa50-4b8f-8180-a1a12b909666\") " Jul 30 01:05:08.365867 ip-10-0-142-6 kubenswrapper[2564]: I0730 01:05:08.365832 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/24aa6ed0-aa50-4b8f-8180-a1a12b909666-tls-certs\") pod \"24aa6ed0-aa50-4b8f-8180-a1a12b909666\" (UID: \"24aa6ed0-aa50-4b8f-8180-a1a12b909666\") " Jul 30 01:05:08.366041 ip-10-0-142-6 kubenswrapper[2564]: I0730 01:05:08.365901 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/24aa6ed0-aa50-4b8f-8180-a1a12b909666-model-cache\") pod \"24aa6ed0-aa50-4b8f-8180-a1a12b909666\" (UID: \"24aa6ed0-aa50-4b8f-8180-a1a12b909666\") " Jul 30 01:05:08.366041 ip-10-0-142-6 kubenswrapper[2564]: I0730 01:05:08.365924 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/24aa6ed0-aa50-4b8f-8180-a1a12b909666-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "24aa6ed0-aa50-4b8f-8180-a1a12b909666" (UID: "24aa6ed0-aa50-4b8f-8180-a1a12b909666"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 30 01:05:08.366041 ip-10-0-142-6 kubenswrapper[2564]: I0730 01:05:08.365942 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/24aa6ed0-aa50-4b8f-8180-a1a12b909666-home" (OuterVolumeSpecName: "home") pod "24aa6ed0-aa50-4b8f-8180-a1a12b909666" (UID: "24aa6ed0-aa50-4b8f-8180-a1a12b909666"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 30 01:05:08.366193 ip-10-0-142-6 kubenswrapper[2564]: I0730 01:05:08.366173 2564 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/24aa6ed0-aa50-4b8f-8180-a1a12b909666-home\") on node \"ip-10-0-142-6.ec2.internal\" DevicePath \"\"" Jul 30 01:05:08.366193 ip-10-0-142-6 kubenswrapper[2564]: I0730 01:05:08.366186 2564 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/24aa6ed0-aa50-4b8f-8180-a1a12b909666-tmp-dir\") on node \"ip-10-0-142-6.ec2.internal\" DevicePath \"\"" Jul 30 01:05:08.366265 ip-10-0-142-6 kubenswrapper[2564]: I0730 01:05:08.366205 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/24aa6ed0-aa50-4b8f-8180-a1a12b909666-model-cache" (OuterVolumeSpecName: "model-cache") pod "24aa6ed0-aa50-4b8f-8180-a1a12b909666" (UID: "24aa6ed0-aa50-4b8f-8180-a1a12b909666"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 30 01:05:08.367885 ip-10-0-142-6 kubenswrapper[2564]: I0730 01:05:08.367860 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/24aa6ed0-aa50-4b8f-8180-a1a12b909666-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "24aa6ed0-aa50-4b8f-8180-a1a12b909666" (UID: "24aa6ed0-aa50-4b8f-8180-a1a12b909666"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 30 01:05:08.367993 ip-10-0-142-6 kubenswrapper[2564]: I0730 01:05:08.367924 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/24aa6ed0-aa50-4b8f-8180-a1a12b909666-dshm" (OuterVolumeSpecName: "dshm") pod "24aa6ed0-aa50-4b8f-8180-a1a12b909666" (UID: "24aa6ed0-aa50-4b8f-8180-a1a12b909666"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 30 01:05:08.368043 ip-10-0-142-6 kubenswrapper[2564]: I0730 01:05:08.367986 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/24aa6ed0-aa50-4b8f-8180-a1a12b909666-kube-api-access-f4c4d" (OuterVolumeSpecName: "kube-api-access-f4c4d") pod "24aa6ed0-aa50-4b8f-8180-a1a12b909666" (UID: "24aa6ed0-aa50-4b8f-8180-a1a12b909666"). InnerVolumeSpecName "kube-api-access-f4c4d". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 30 01:05:08.467557 ip-10-0-142-6 kubenswrapper[2564]: I0730 01:05:08.467512 2564 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-f4c4d\" (UniqueName: \"kubernetes.io/projected/24aa6ed0-aa50-4b8f-8180-a1a12b909666-kube-api-access-f4c4d\") on node \"ip-10-0-142-6.ec2.internal\" DevicePath \"\"" Jul 30 01:05:08.467557 ip-10-0-142-6 kubenswrapper[2564]: I0730 01:05:08.467550 2564 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/24aa6ed0-aa50-4b8f-8180-a1a12b909666-tls-certs\") on node \"ip-10-0-142-6.ec2.internal\" DevicePath \"\"" Jul 30 01:05:08.467557 ip-10-0-142-6 kubenswrapper[2564]: I0730 01:05:08.467561 2564 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/24aa6ed0-aa50-4b8f-8180-a1a12b909666-model-cache\") on node \"ip-10-0-142-6.ec2.internal\" DevicePath \"\"" Jul 30 01:05:08.467557 ip-10-0-142-6 kubenswrapper[2564]: I0730 01:05:08.467571 2564 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/24aa6ed0-aa50-4b8f-8180-a1a12b909666-dshm\") on node \"ip-10-0-142-6.ec2.internal\" DevicePath \"\"" Jul 30 01:05:08.544290 ip-10-0-142-6 kubenswrapper[2564]: I0730 01:05:08.544248 2564 generic.go:356] "Generic (PLEG): container finished" podID="24aa6ed0-aa50-4b8f-8180-a1a12b909666" containerID="0478a502f50c2a0895bf3ad79498d3a77c051bd77720ebf96b78656916af0705" exitCode=0 Jul 30 01:05:08.544290 ip-10-0-142-6 kubenswrapper[2564]: I0730 01:05:08.544296 2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-c4c410/auth-test-model-c4c410-kserve-5cb6c845dc-mvjj7" Jul 30 01:05:08.544511 ip-10-0-142-6 kubenswrapper[2564]: I0730 01:05:08.544324 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-c4c410/auth-test-model-c4c410-kserve-5cb6c845dc-mvjj7" event={"ID":"24aa6ed0-aa50-4b8f-8180-a1a12b909666","Type":"ContainerDied","Data":"0478a502f50c2a0895bf3ad79498d3a77c051bd77720ebf96b78656916af0705"} Jul 30 01:05:08.544511 ip-10-0-142-6 kubenswrapper[2564]: I0730 01:05:08.544359 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-c4c410/auth-test-model-c4c410-kserve-5cb6c845dc-mvjj7" event={"ID":"24aa6ed0-aa50-4b8f-8180-a1a12b909666","Type":"ContainerDied","Data":"96130526d55674a099193ce5f31c40514896e1ebcb8d744de93c61cbd0cb0ef7"} Jul 30 01:05:08.544511 ip-10-0-142-6 kubenswrapper[2564]: I0730 01:05:08.544381 2564 scope.go:117] "RemoveContainer" containerID="0478a502f50c2a0895bf3ad79498d3a77c051bd77720ebf96b78656916af0705" Jul 30 01:05:08.553462 ip-10-0-142-6 kubenswrapper[2564]: I0730 01:05:08.553441 2564 scope.go:117] "RemoveContainer" containerID="0478a502f50c2a0895bf3ad79498d3a77c051bd77720ebf96b78656916af0705" Jul 30 01:05:08.553719 ip-10-0-142-6 kubenswrapper[2564]: E0730 01:05:08.553700 2564 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0478a502f50c2a0895bf3ad79498d3a77c051bd77720ebf96b78656916af0705\": container with ID starting with 0478a502f50c2a0895bf3ad79498d3a77c051bd77720ebf96b78656916af0705 not found: ID does not exist" containerID="0478a502f50c2a0895bf3ad79498d3a77c051bd77720ebf96b78656916af0705" Jul 30 01:05:08.553764 ip-10-0-142-6 kubenswrapper[2564]: I0730 01:05:08.553729 2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0478a502f50c2a0895bf3ad79498d3a77c051bd77720ebf96b78656916af0705"} err="failed to get container status \"0478a502f50c2a0895bf3ad79498d3a77c051bd77720ebf96b78656916af0705\": rpc error: code = NotFound desc = could not find container \"0478a502f50c2a0895bf3ad79498d3a77c051bd77720ebf96b78656916af0705\": container with ID starting with 0478a502f50c2a0895bf3ad79498d3a77c051bd77720ebf96b78656916af0705 not found: ID does not exist" Jul 30 01:05:08.566917 ip-10-0-142-6 kubenswrapper[2564]: I0730 01:05:08.566886 2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-c4c410/auth-test-model-c4c410-kserve-5cb6c845dc-mvjj7"] Jul 30 01:05:08.570532 ip-10-0-142-6 kubenswrapper[2564]: I0730 01:05:08.570510 2564 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-b-c4c410/auth-test-model-c4c410-kserve-5cb6c845dc-mvjj7"] Jul 30 01:05:09.139530 ip-10-0-142-6 kubenswrapper[2564]: I0730 01:05:09.139498 2564 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="1294cdac-3833-47ea-bf38-0d415391e281" path="/var/lib/kubelet/pods/1294cdac-3833-47ea-bf38-0d415391e281/volumes" Jul 30 01:05:09.139878 ip-10-0-142-6 kubenswrapper[2564]: I0730 01:05:09.139865 2564 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="24aa6ed0-aa50-4b8f-8180-a1a12b909666" path="/var/lib/kubelet/pods/24aa6ed0-aa50-4b8f-8180-a1a12b909666/volumes" Jul 30 01:05:13.518829 ip-10-0-142-6 kubenswrapper[2564]: I0730 01:05:13.518798 2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-a-a47fe8/sub-test-model-a47fe8-kserve-68bd694f85-nltsn"] Jul 30 01:05:13.519207 ip-10-0-142-6 kubenswrapper[2564]: I0730 01:05:13.519117 2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="24aa6ed0-aa50-4b8f-8180-a1a12b909666" containerName="main" Jul 30 01:05:13.519207 ip-10-0-142-6 kubenswrapper[2564]: I0730 01:05:13.519128 2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="24aa6ed0-aa50-4b8f-8180-a1a12b909666" containerName="main" Jul 30 01:05:13.519207 ip-10-0-142-6 kubenswrapper[2564]: I0730 01:05:13.519138 2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1294cdac-3833-47ea-bf38-0d415391e281" containerName="main" Jul 30 01:05:13.519207 ip-10-0-142-6 kubenswrapper[2564]: I0730 01:05:13.519144 2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="1294cdac-3833-47ea-bf38-0d415391e281" containerName="main" Jul 30 01:05:13.519338 ip-10-0-142-6 kubenswrapper[2564]: I0730 01:05:13.519217 2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="1294cdac-3833-47ea-bf38-0d415391e281" containerName="main" Jul 30 01:05:13.519338 ip-10-0-142-6 kubenswrapper[2564]: I0730 01:05:13.519226 2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="24aa6ed0-aa50-4b8f-8180-a1a12b909666" containerName="main" Jul 30 01:05:13.523519 ip-10-0-142-6 kubenswrapper[2564]: I0730 01:05:13.523502 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-a47fe8/sub-test-model-a47fe8-kserve-68bd694f85-nltsn" Jul 30 01:05:13.527144 ip-10-0-142-6 kubenswrapper[2564]: I0730 01:05:13.527111 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-a47fe8\"/\"openshift-service-ca.crt\"" Jul 30 01:05:13.527337 ip-10-0-142-6 kubenswrapper[2564]: I0730 01:05:13.527172 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-a47fe8\"/\"default-dockercfg-skjjk\"" Jul 30 01:05:13.527337 ip-10-0-142-6 kubenswrapper[2564]: I0730 01:05:13.527111 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-a47fe8\"/\"sub-test-model-a47fe8-kserve-self-signed-certs\"" Jul 30 01:05:13.527337 ip-10-0-142-6 kubenswrapper[2564]: I0730 01:05:13.527124 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-a47fe8\"/\"kube-root-ca.crt\"" Jul 30 01:05:13.530378 ip-10-0-142-6 kubenswrapper[2564]: I0730 01:05:13.530353 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-a47fe8/sub-test-model-a47fe8-kserve-68bd694f85-nltsn"] Jul 30 01:05:13.609868 ip-10-0-142-6 kubenswrapper[2564]: I0730 01:05:13.609832 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d0fea680-43b6-4c1d-90e2-7409dcb4f577-home\") pod \"sub-test-model-a47fe8-kserve-68bd694f85-nltsn\" (UID: \"d0fea680-43b6-4c1d-90e2-7409dcb4f577\") " pod="ai-tenant-e2e-shared-a-a47fe8/sub-test-model-a47fe8-kserve-68bd694f85-nltsn" Jul 30 01:05:13.610038 ip-10-0-142-6 kubenswrapper[2564]: I0730 01:05:13.609879 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d0fea680-43b6-4c1d-90e2-7409dcb4f577-tmp-dir\") pod \"sub-test-model-a47fe8-kserve-68bd694f85-nltsn\" (UID: \"d0fea680-43b6-4c1d-90e2-7409dcb4f577\") " pod="ai-tenant-e2e-shared-a-a47fe8/sub-test-model-a47fe8-kserve-68bd694f85-nltsn" Jul 30 01:05:13.610038 ip-10-0-142-6 kubenswrapper[2564]: I0730 01:05:13.609969 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d0fea680-43b6-4c1d-90e2-7409dcb4f577-dshm\") pod \"sub-test-model-a47fe8-kserve-68bd694f85-nltsn\" (UID: \"d0fea680-43b6-4c1d-90e2-7409dcb4f577\") " pod="ai-tenant-e2e-shared-a-a47fe8/sub-test-model-a47fe8-kserve-68bd694f85-nltsn" Jul 30 01:05:13.610038 ip-10-0-142-6 kubenswrapper[2564]: I0730 01:05:13.610010 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d0fea680-43b6-4c1d-90e2-7409dcb4f577-model-cache\") pod \"sub-test-model-a47fe8-kserve-68bd694f85-nltsn\" (UID: \"d0fea680-43b6-4c1d-90e2-7409dcb4f577\") " pod="ai-tenant-e2e-shared-a-a47fe8/sub-test-model-a47fe8-kserve-68bd694f85-nltsn" Jul 30 01:05:13.610173 ip-10-0-142-6 kubenswrapper[2564]: I0730 01:05:13.610044 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lx8vb\" (UniqueName: \"kubernetes.io/projected/d0fea680-43b6-4c1d-90e2-7409dcb4f577-kube-api-access-lx8vb\") pod \"sub-test-model-a47fe8-kserve-68bd694f85-nltsn\" (UID: \"d0fea680-43b6-4c1d-90e2-7409dcb4f577\") " pod="ai-tenant-e2e-shared-a-a47fe8/sub-test-model-a47fe8-kserve-68bd694f85-nltsn" Jul 30 01:05:13.610173 ip-10-0-142-6 kubenswrapper[2564]: I0730 01:05:13.610069 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d0fea680-43b6-4c1d-90e2-7409dcb4f577-tls-certs\") pod \"sub-test-model-a47fe8-kserve-68bd694f85-nltsn\" (UID: \"d0fea680-43b6-4c1d-90e2-7409dcb4f577\") " pod="ai-tenant-e2e-shared-a-a47fe8/sub-test-model-a47fe8-kserve-68bd694f85-nltsn" Jul 30 01:05:13.711319 ip-10-0-142-6 kubenswrapper[2564]: I0730 01:05:13.711283 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d0fea680-43b6-4c1d-90e2-7409dcb4f577-model-cache\") pod \"sub-test-model-a47fe8-kserve-68bd694f85-nltsn\" (UID: \"d0fea680-43b6-4c1d-90e2-7409dcb4f577\") " pod="ai-tenant-e2e-shared-a-a47fe8/sub-test-model-a47fe8-kserve-68bd694f85-nltsn" Jul 30 01:05:13.711492 ip-10-0-142-6 kubenswrapper[2564]: I0730 01:05:13.711330 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lx8vb\" (UniqueName: \"kubernetes.io/projected/d0fea680-43b6-4c1d-90e2-7409dcb4f577-kube-api-access-lx8vb\") pod \"sub-test-model-a47fe8-kserve-68bd694f85-nltsn\" (UID: \"d0fea680-43b6-4c1d-90e2-7409dcb4f577\") " pod="ai-tenant-e2e-shared-a-a47fe8/sub-test-model-a47fe8-kserve-68bd694f85-nltsn" Jul 30 01:05:13.711492 ip-10-0-142-6 kubenswrapper[2564]: I0730 01:05:13.711352 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d0fea680-43b6-4c1d-90e2-7409dcb4f577-tls-certs\") pod \"sub-test-model-a47fe8-kserve-68bd694f85-nltsn\" (UID: \"d0fea680-43b6-4c1d-90e2-7409dcb4f577\") " pod="ai-tenant-e2e-shared-a-a47fe8/sub-test-model-a47fe8-kserve-68bd694f85-nltsn" Jul 30 01:05:13.711492 ip-10-0-142-6 kubenswrapper[2564]: I0730 01:05:13.711381 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d0fea680-43b6-4c1d-90e2-7409dcb4f577-home\") pod \"sub-test-model-a47fe8-kserve-68bd694f85-nltsn\" (UID: \"d0fea680-43b6-4c1d-90e2-7409dcb4f577\") " pod="ai-tenant-e2e-shared-a-a47fe8/sub-test-model-a47fe8-kserve-68bd694f85-nltsn" Jul 30 01:05:13.711492 ip-10-0-142-6 kubenswrapper[2564]: I0730 01:05:13.711405 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d0fea680-43b6-4c1d-90e2-7409dcb4f577-tmp-dir\") pod \"sub-test-model-a47fe8-kserve-68bd694f85-nltsn\" (UID: \"d0fea680-43b6-4c1d-90e2-7409dcb4f577\") " pod="ai-tenant-e2e-shared-a-a47fe8/sub-test-model-a47fe8-kserve-68bd694f85-nltsn" Jul 30 01:05:13.711492 ip-10-0-142-6 kubenswrapper[2564]: I0730 01:05:13.711446 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d0fea680-43b6-4c1d-90e2-7409dcb4f577-dshm\") pod \"sub-test-model-a47fe8-kserve-68bd694f85-nltsn\" (UID: \"d0fea680-43b6-4c1d-90e2-7409dcb4f577\") " pod="ai-tenant-e2e-shared-a-a47fe8/sub-test-model-a47fe8-kserve-68bd694f85-nltsn" Jul 30 01:05:13.711785 ip-10-0-142-6 kubenswrapper[2564]: I0730 01:05:13.711765 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d0fea680-43b6-4c1d-90e2-7409dcb4f577-model-cache\") pod \"sub-test-model-a47fe8-kserve-68bd694f85-nltsn\" (UID: \"d0fea680-43b6-4c1d-90e2-7409dcb4f577\") " pod="ai-tenant-e2e-shared-a-a47fe8/sub-test-model-a47fe8-kserve-68bd694f85-nltsn" Jul 30 01:05:13.711849 ip-10-0-142-6 kubenswrapper[2564]: I0730 01:05:13.711815 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d0fea680-43b6-4c1d-90e2-7409dcb4f577-home\") pod \"sub-test-model-a47fe8-kserve-68bd694f85-nltsn\" (UID: \"d0fea680-43b6-4c1d-90e2-7409dcb4f577\") " pod="ai-tenant-e2e-shared-a-a47fe8/sub-test-model-a47fe8-kserve-68bd694f85-nltsn" Jul 30 01:05:13.711849 ip-10-0-142-6 kubenswrapper[2564]: I0730 01:05:13.711841 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d0fea680-43b6-4c1d-90e2-7409dcb4f577-tmp-dir\") pod \"sub-test-model-a47fe8-kserve-68bd694f85-nltsn\" (UID: \"d0fea680-43b6-4c1d-90e2-7409dcb4f577\") " pod="ai-tenant-e2e-shared-a-a47fe8/sub-test-model-a47fe8-kserve-68bd694f85-nltsn" Jul 30 01:05:13.713812 ip-10-0-142-6 kubenswrapper[2564]: I0730 01:05:13.713787 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d0fea680-43b6-4c1d-90e2-7409dcb4f577-dshm\") pod \"sub-test-model-a47fe8-kserve-68bd694f85-nltsn\" (UID: \"d0fea680-43b6-4c1d-90e2-7409dcb4f577\") " pod="ai-tenant-e2e-shared-a-a47fe8/sub-test-model-a47fe8-kserve-68bd694f85-nltsn" Jul 30 01:05:13.714306 ip-10-0-142-6 kubenswrapper[2564]: I0730 01:05:13.714288 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d0fea680-43b6-4c1d-90e2-7409dcb4f577-tls-certs\") pod \"sub-test-model-a47fe8-kserve-68bd694f85-nltsn\" (UID: \"d0fea680-43b6-4c1d-90e2-7409dcb4f577\") " pod="ai-tenant-e2e-shared-a-a47fe8/sub-test-model-a47fe8-kserve-68bd694f85-nltsn" Jul 30 01:05:13.719707 ip-10-0-142-6 kubenswrapper[2564]: I0730 01:05:13.719684 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-lx8vb\" (UniqueName: \"kubernetes.io/projected/d0fea680-43b6-4c1d-90e2-7409dcb4f577-kube-api-access-lx8vb\") pod \"sub-test-model-a47fe8-kserve-68bd694f85-nltsn\" (UID: \"d0fea680-43b6-4c1d-90e2-7409dcb4f577\") " pod="ai-tenant-e2e-shared-a-a47fe8/sub-test-model-a47fe8-kserve-68bd694f85-nltsn" Jul 30 01:05:13.833785 ip-10-0-142-6 kubenswrapper[2564]: I0730 01:05:13.833688 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-a47fe8/sub-test-model-a47fe8-kserve-68bd694f85-nltsn" Jul 30 01:05:13.965025 ip-10-0-142-6 kubenswrapper[2564]: I0730 01:05:13.964998 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-a47fe8/sub-test-model-a47fe8-kserve-68bd694f85-nltsn"] Jul 30 01:05:13.966663 ip-10-0-142-6 kubenswrapper[2564]: W0730 01:05:13.966632 2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podd0fea680_43b6_4c1d_90e2_7409dcb4f577.slice/crio-24f5550bf33a8534c386926578114424c66d06e18bd480c71238d0ad0a5315f2 WatchSource:0}: Error finding container 24f5550bf33a8534c386926578114424c66d06e18bd480c71238d0ad0a5315f2: Status 404 returned error can't find the container with id 24f5550bf33a8534c386926578114424c66d06e18bd480c71238d0ad0a5315f2 Jul 30 01:05:14.574959 ip-10-0-142-6 kubenswrapper[2564]: I0730 01:05:14.574928 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-a47fe8/sub-test-model-a47fe8-kserve-68bd694f85-nltsn" event={"ID":"d0fea680-43b6-4c1d-90e2-7409dcb4f577","Type":"ContainerStarted","Data":"0025ba76977a983a07c629bf68ab927e4329e4c26b26b5c8782919c2cb725882"} Jul 30 01:05:14.575421 ip-10-0-142-6 kubenswrapper[2564]: I0730 01:05:14.574970 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-a47fe8/sub-test-model-a47fe8-kserve-68bd694f85-nltsn" event={"ID":"d0fea680-43b6-4c1d-90e2-7409dcb4f577","Type":"ContainerStarted","Data":"24f5550bf33a8534c386926578114424c66d06e18bd480c71238d0ad0a5315f2"} Jul 30 01:05:14.592673 ip-10-0-142-6 kubenswrapper[2564]: I0730 01:05:14.592613 2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-a-a47fe8/sub-test-model-a47fe8-kserve-68bd694f85-nltsn" podStartSLOduration=1.5925966 podStartE2EDuration="1.5925966s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-30 01:05:13 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-30 01:05:14.59122374 +0000 UTC m=+2846.005963674" watchObservedRunningTime="2026-07-30 01:05:14.5925966 +0000 UTC m=+2846.007336533" Jul 30 01:05:17.018684 ip-10-0-142-6 kubenswrapper[2564]: I0730 01:05:17.018646 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-nrv2p"] Jul 30 01:05:23.834807 ip-10-0-142-6 kubenswrapper[2564]: I0730 01:05:23.834756 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-a-a47fe8/sub-test-model-a47fe8-kserve-68bd694f85-nltsn" Jul 30 01:05:23.835316 ip-10-0-142-6 kubenswrapper[2564]: I0730 01:05:23.834826 2564 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-a-a47fe8/sub-test-model-a47fe8-kserve-68bd694f85-nltsn" Jul 30 01:05:23.847326 ip-10-0-142-6 kubenswrapper[2564]: I0730 01:05:23.847300 2564 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-a-a47fe8/sub-test-model-a47fe8-kserve-68bd694f85-nltsn" Jul 30 01:05:24.639377 ip-10-0-142-6 kubenswrapper[2564]: I0730 01:05:24.639349 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-a-a47fe8/sub-test-model-a47fe8-kserve-68bd694f85-nltsn" Jul 30 01:06:03.404272 ip-10-0-142-6 kubenswrapper[2564]: I0730 01:06:03.404237 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-nrv2p"] Jul 30 01:06:31.930322 ip-10-0-142-6 kubenswrapper[2564]: I0730 01:06:31.930280 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-nrv2p"] Jul 30 01:06:51.105856 ip-10-0-142-6 kubenswrapper[2564]: I0730 01:06:51.105808 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-nrv2p"] Jul 30 01:06:56.693327 ip-10-0-142-6 kubenswrapper[2564]: I0730 01:06:56.693290 2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-a47fe8/sub-test-model-a47fe8-kserve-68bd694f85-nltsn"] Jul 30 01:07:08.405968 ip-10-0-142-6 kubenswrapper[2564]: I0730 01:07:08.405930 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-nrv2p"] Jul 30 01:07:11.712056 ip-10-0-142-6 kubenswrapper[2564]: I0730 01:07:11.711947 2564 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-shared-a-a47fe8/sub-test-model-a47fe8-kserve-68bd694f85-nltsn" podUID="d0fea680-43b6-4c1d-90e2-7409dcb4f577" containerName="main" containerID="cri-o://0025ba76977a983a07c629bf68ab927e4329e4c26b26b5c8782919c2cb725882" gracePeriod=45 Jul 30 01:07:11.962864 ip-10-0-142-6 kubenswrapper[2564]: I0730 01:07:11.962801 2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-a47fe8/sub-test-model-a47fe8-kserve-68bd694f85-nltsn" Jul 30 01:07:12.110637 ip-10-0-142-6 kubenswrapper[2564]: I0730 01:07:12.110604 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d0fea680-43b6-4c1d-90e2-7409dcb4f577-tmp-dir\") pod \"d0fea680-43b6-4c1d-90e2-7409dcb4f577\" (UID: \"d0fea680-43b6-4c1d-90e2-7409dcb4f577\") " Jul 30 01:07:12.110637 ip-10-0-142-6 kubenswrapper[2564]: I0730 01:07:12.110642 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d0fea680-43b6-4c1d-90e2-7409dcb4f577-home\") pod \"d0fea680-43b6-4c1d-90e2-7409dcb4f577\" (UID: \"d0fea680-43b6-4c1d-90e2-7409dcb4f577\") " Jul 30 01:07:12.110852 ip-10-0-142-6 kubenswrapper[2564]: I0730 01:07:12.110658 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d0fea680-43b6-4c1d-90e2-7409dcb4f577-dshm\") pod \"d0fea680-43b6-4c1d-90e2-7409dcb4f577\" (UID: \"d0fea680-43b6-4c1d-90e2-7409dcb4f577\") " Jul 30 01:07:12.110852 ip-10-0-142-6 kubenswrapper[2564]: I0730 01:07:12.110675 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d0fea680-43b6-4c1d-90e2-7409dcb4f577-tls-certs\") pod \"d0fea680-43b6-4c1d-90e2-7409dcb4f577\" (UID: \"d0fea680-43b6-4c1d-90e2-7409dcb4f577\") " Jul 30 01:07:12.110852 ip-10-0-142-6 kubenswrapper[2564]: I0730 01:07:12.110729 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d0fea680-43b6-4c1d-90e2-7409dcb4f577-model-cache\") pod \"d0fea680-43b6-4c1d-90e2-7409dcb4f577\" (UID: \"d0fea680-43b6-4c1d-90e2-7409dcb4f577\") " Jul 30 01:07:12.110852 ip-10-0-142-6 kubenswrapper[2564]: I0730 01:07:12.110815 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-lx8vb\" (UniqueName: \"kubernetes.io/projected/d0fea680-43b6-4c1d-90e2-7409dcb4f577-kube-api-access-lx8vb\") pod \"d0fea680-43b6-4c1d-90e2-7409dcb4f577\" (UID: \"d0fea680-43b6-4c1d-90e2-7409dcb4f577\") " Jul 30 01:07:12.111068 ip-10-0-142-6 kubenswrapper[2564]: I0730 01:07:12.110923 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d0fea680-43b6-4c1d-90e2-7409dcb4f577-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "d0fea680-43b6-4c1d-90e2-7409dcb4f577" (UID: "d0fea680-43b6-4c1d-90e2-7409dcb4f577"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 30 01:07:12.111068 ip-10-0-142-6 kubenswrapper[2564]: I0730 01:07:12.110940 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d0fea680-43b6-4c1d-90e2-7409dcb4f577-home" (OuterVolumeSpecName: "home") pod "d0fea680-43b6-4c1d-90e2-7409dcb4f577" (UID: "d0fea680-43b6-4c1d-90e2-7409dcb4f577"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 30 01:07:12.111210 ip-10-0-142-6 kubenswrapper[2564]: I0730 01:07:12.111063 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d0fea680-43b6-4c1d-90e2-7409dcb4f577-model-cache" (OuterVolumeSpecName: "model-cache") pod "d0fea680-43b6-4c1d-90e2-7409dcb4f577" (UID: "d0fea680-43b6-4c1d-90e2-7409dcb4f577"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 30 01:07:12.111210 ip-10-0-142-6 kubenswrapper[2564]: I0730 01:07:12.111175 2564 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d0fea680-43b6-4c1d-90e2-7409dcb4f577-tmp-dir\") on node \"ip-10-0-142-6.ec2.internal\" DevicePath \"\"" Jul 30 01:07:12.111210 ip-10-0-142-6 kubenswrapper[2564]: I0730 01:07:12.111192 2564 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d0fea680-43b6-4c1d-90e2-7409dcb4f577-home\") on node \"ip-10-0-142-6.ec2.internal\" DevicePath \"\"" Jul 30 01:07:12.112877 ip-10-0-142-6 kubenswrapper[2564]: I0730 01:07:12.112845 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d0fea680-43b6-4c1d-90e2-7409dcb4f577-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "d0fea680-43b6-4c1d-90e2-7409dcb4f577" (UID: "d0fea680-43b6-4c1d-90e2-7409dcb4f577"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 30 01:07:12.112994 ip-10-0-142-6 kubenswrapper[2564]: I0730 01:07:12.112890 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d0fea680-43b6-4c1d-90e2-7409dcb4f577-dshm" (OuterVolumeSpecName: "dshm") pod "d0fea680-43b6-4c1d-90e2-7409dcb4f577" (UID: "d0fea680-43b6-4c1d-90e2-7409dcb4f577"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 30 01:07:12.112994 ip-10-0-142-6 kubenswrapper[2564]: I0730 01:07:12.112894 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d0fea680-43b6-4c1d-90e2-7409dcb4f577-kube-api-access-lx8vb" (OuterVolumeSpecName: "kube-api-access-lx8vb") pod "d0fea680-43b6-4c1d-90e2-7409dcb4f577" (UID: "d0fea680-43b6-4c1d-90e2-7409dcb4f577"). InnerVolumeSpecName "kube-api-access-lx8vb". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 30 01:07:12.157054 ip-10-0-142-6 kubenswrapper[2564]: I0730 01:07:12.157019 2564 generic.go:356] "Generic (PLEG): container finished" podID="d0fea680-43b6-4c1d-90e2-7409dcb4f577" containerID="0025ba76977a983a07c629bf68ab927e4329e4c26b26b5c8782919c2cb725882" exitCode=0 Jul 30 01:07:12.157256 ip-10-0-142-6 kubenswrapper[2564]: I0730 01:07:12.157092 2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-a47fe8/sub-test-model-a47fe8-kserve-68bd694f85-nltsn" Jul 30 01:07:12.157256 ip-10-0-142-6 kubenswrapper[2564]: I0730 01:07:12.157103 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-a47fe8/sub-test-model-a47fe8-kserve-68bd694f85-nltsn" event={"ID":"d0fea680-43b6-4c1d-90e2-7409dcb4f577","Type":"ContainerDied","Data":"0025ba76977a983a07c629bf68ab927e4329e4c26b26b5c8782919c2cb725882"} Jul 30 01:07:12.157256 ip-10-0-142-6 kubenswrapper[2564]: I0730 01:07:12.157147 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-a47fe8/sub-test-model-a47fe8-kserve-68bd694f85-nltsn" event={"ID":"d0fea680-43b6-4c1d-90e2-7409dcb4f577","Type":"ContainerDied","Data":"24f5550bf33a8534c386926578114424c66d06e18bd480c71238d0ad0a5315f2"} Jul 30 01:07:12.157256 ip-10-0-142-6 kubenswrapper[2564]: I0730 01:07:12.157165 2564 scope.go:117] "RemoveContainer" containerID="0025ba76977a983a07c629bf68ab927e4329e4c26b26b5c8782919c2cb725882" Jul 30 01:07:12.166126 ip-10-0-142-6 kubenswrapper[2564]: I0730 01:07:12.166105 2564 scope.go:117] "RemoveContainer" containerID="0025ba76977a983a07c629bf68ab927e4329e4c26b26b5c8782919c2cb725882" Jul 30 01:07:12.166370 ip-10-0-142-6 kubenswrapper[2564]: E0730 01:07:12.166353 2564 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0025ba76977a983a07c629bf68ab927e4329e4c26b26b5c8782919c2cb725882\": container with ID starting with 0025ba76977a983a07c629bf68ab927e4329e4c26b26b5c8782919c2cb725882 not found: ID does not exist" containerID="0025ba76977a983a07c629bf68ab927e4329e4c26b26b5c8782919c2cb725882" Jul 30 01:07:12.166424 ip-10-0-142-6 kubenswrapper[2564]: I0730 01:07:12.166379 2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0025ba76977a983a07c629bf68ab927e4329e4c26b26b5c8782919c2cb725882"} err="failed to get container status \"0025ba76977a983a07c629bf68ab927e4329e4c26b26b5c8782919c2cb725882\": rpc error: code = NotFound desc = could not find container \"0025ba76977a983a07c629bf68ab927e4329e4c26b26b5c8782919c2cb725882\": container with ID starting with 0025ba76977a983a07c629bf68ab927e4329e4c26b26b5c8782919c2cb725882 not found: ID does not exist" Jul 30 01:07:12.180490 ip-10-0-142-6 kubenswrapper[2564]: I0730 01:07:12.180463 2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-a47fe8/sub-test-model-a47fe8-kserve-68bd694f85-nltsn"] Jul 30 01:07:12.184126 ip-10-0-142-6 kubenswrapper[2564]: I0730 01:07:12.184103 2564 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-a-a47fe8/sub-test-model-a47fe8-kserve-68bd694f85-nltsn"] Jul 30 01:07:12.212069 ip-10-0-142-6 kubenswrapper[2564]: I0730 01:07:12.212044 2564 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lx8vb\" (UniqueName: \"kubernetes.io/projected/d0fea680-43b6-4c1d-90e2-7409dcb4f577-kube-api-access-lx8vb\") on node \"ip-10-0-142-6.ec2.internal\" DevicePath \"\"" Jul 30 01:07:12.212069 ip-10-0-142-6 kubenswrapper[2564]: I0730 01:07:12.212067 2564 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d0fea680-43b6-4c1d-90e2-7409dcb4f577-dshm\") on node \"ip-10-0-142-6.ec2.internal\" DevicePath \"\"" Jul 30 01:07:12.212218 ip-10-0-142-6 kubenswrapper[2564]: I0730 01:07:12.212105 2564 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d0fea680-43b6-4c1d-90e2-7409dcb4f577-tls-certs\") on node \"ip-10-0-142-6.ec2.internal\" DevicePath \"\"" Jul 30 01:07:12.212218 ip-10-0-142-6 kubenswrapper[2564]: I0730 01:07:12.212114 2564 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d0fea680-43b6-4c1d-90e2-7409dcb4f577-model-cache\") on node \"ip-10-0-142-6.ec2.internal\" DevicePath \"\"" Jul 30 01:07:13.139680 ip-10-0-142-6 kubenswrapper[2564]: I0730 01:07:13.139645 2564 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="d0fea680-43b6-4c1d-90e2-7409dcb4f577" path="/var/lib/kubelet/pods/d0fea680-43b6-4c1d-90e2-7409dcb4f577/volumes" Jul 30 01:07:14.628144 ip-10-0-142-6 kubenswrapper[2564]: I0730 01:07:14.628115 2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-a-a47fe8/rate-test-model-a47fe8-kserve-7cc787b7f-q8bhj"] Jul 30 01:07:14.628512 ip-10-0-142-6 kubenswrapper[2564]: I0730 01:07:14.628420 2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d0fea680-43b6-4c1d-90e2-7409dcb4f577" containerName="main" Jul 30 01:07:14.628512 ip-10-0-142-6 kubenswrapper[2564]: I0730 01:07:14.628431 2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="d0fea680-43b6-4c1d-90e2-7409dcb4f577" containerName="main" Jul 30 01:07:14.628512 ip-10-0-142-6 kubenswrapper[2564]: I0730 01:07:14.628493 2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="d0fea680-43b6-4c1d-90e2-7409dcb4f577" containerName="main" Jul 30 01:07:14.632965 ip-10-0-142-6 kubenswrapper[2564]: I0730 01:07:14.632947 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-a47fe8/rate-test-model-a47fe8-kserve-7cc787b7f-q8bhj" Jul 30 01:07:14.635703 ip-10-0-142-6 kubenswrapper[2564]: I0730 01:07:14.635682 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-a47fe8\"/\"kube-root-ca.crt\"" Jul 30 01:07:14.635816 ip-10-0-142-6 kubenswrapper[2564]: I0730 01:07:14.635705 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-a47fe8\"/\"openshift-service-ca.crt\"" Jul 30 01:07:14.636685 ip-10-0-142-6 kubenswrapper[2564]: I0730 01:07:14.636637 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-a47fe8\"/\"default-dockercfg-skjjk\"" Jul 30 01:07:14.636685 ip-10-0-142-6 kubenswrapper[2564]: I0730 01:07:14.636668 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-a47fe8\"/\"rate-test-model-a47fe8-kserve-self-signed-certs\"" Jul 30 01:07:14.640257 ip-10-0-142-6 kubenswrapper[2564]: I0730 01:07:14.640043 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-a47fe8/rate-test-model-a47fe8-kserve-7cc787b7f-q8bhj"] Jul 30 01:07:14.730950 ip-10-0-142-6 kubenswrapper[2564]: I0730 01:07:14.730913 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5146083b-485d-406b-ae56-54eaf05efef8-tmp-dir\") pod \"rate-test-model-a47fe8-kserve-7cc787b7f-q8bhj\" (UID: \"5146083b-485d-406b-ae56-54eaf05efef8\") " pod="ai-tenant-e2e-shared-a-a47fe8/rate-test-model-a47fe8-kserve-7cc787b7f-q8bhj" Jul 30 01:07:14.730950 ip-10-0-142-6 kubenswrapper[2564]: I0730 01:07:14.730953 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5146083b-485d-406b-ae56-54eaf05efef8-dshm\") pod \"rate-test-model-a47fe8-kserve-7cc787b7f-q8bhj\" (UID: \"5146083b-485d-406b-ae56-54eaf05efef8\") " pod="ai-tenant-e2e-shared-a-a47fe8/rate-test-model-a47fe8-kserve-7cc787b7f-q8bhj" Jul 30 01:07:14.731199 ip-10-0-142-6 kubenswrapper[2564]: I0730 01:07:14.730982 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lvqn5\" (UniqueName: \"kubernetes.io/projected/5146083b-485d-406b-ae56-54eaf05efef8-kube-api-access-lvqn5\") pod \"rate-test-model-a47fe8-kserve-7cc787b7f-q8bhj\" (UID: \"5146083b-485d-406b-ae56-54eaf05efef8\") " pod="ai-tenant-e2e-shared-a-a47fe8/rate-test-model-a47fe8-kserve-7cc787b7f-q8bhj" Jul 30 01:07:14.731199 ip-10-0-142-6 kubenswrapper[2564]: I0730 01:07:14.731009 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5146083b-485d-406b-ae56-54eaf05efef8-tls-certs\") pod \"rate-test-model-a47fe8-kserve-7cc787b7f-q8bhj\" (UID: \"5146083b-485d-406b-ae56-54eaf05efef8\") " pod="ai-tenant-e2e-shared-a-a47fe8/rate-test-model-a47fe8-kserve-7cc787b7f-q8bhj" Jul 30 01:07:14.731199 ip-10-0-142-6 kubenswrapper[2564]: I0730 01:07:14.731041 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5146083b-485d-406b-ae56-54eaf05efef8-home\") pod \"rate-test-model-a47fe8-kserve-7cc787b7f-q8bhj\" (UID: \"5146083b-485d-406b-ae56-54eaf05efef8\") " pod="ai-tenant-e2e-shared-a-a47fe8/rate-test-model-a47fe8-kserve-7cc787b7f-q8bhj" Jul 30 01:07:14.731199 ip-10-0-142-6 kubenswrapper[2564]: I0730 01:07:14.731127 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5146083b-485d-406b-ae56-54eaf05efef8-model-cache\") pod \"rate-test-model-a47fe8-kserve-7cc787b7f-q8bhj\" (UID: \"5146083b-485d-406b-ae56-54eaf05efef8\") " pod="ai-tenant-e2e-shared-a-a47fe8/rate-test-model-a47fe8-kserve-7cc787b7f-q8bhj" Jul 30 01:07:14.832578 ip-10-0-142-6 kubenswrapper[2564]: I0730 01:07:14.832536 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5146083b-485d-406b-ae56-54eaf05efef8-home\") pod \"rate-test-model-a47fe8-kserve-7cc787b7f-q8bhj\" (UID: \"5146083b-485d-406b-ae56-54eaf05efef8\") " pod="ai-tenant-e2e-shared-a-a47fe8/rate-test-model-a47fe8-kserve-7cc787b7f-q8bhj" Jul 30 01:07:14.832578 ip-10-0-142-6 kubenswrapper[2564]: I0730 01:07:14.832583 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5146083b-485d-406b-ae56-54eaf05efef8-model-cache\") pod \"rate-test-model-a47fe8-kserve-7cc787b7f-q8bhj\" (UID: \"5146083b-485d-406b-ae56-54eaf05efef8\") " pod="ai-tenant-e2e-shared-a-a47fe8/rate-test-model-a47fe8-kserve-7cc787b7f-q8bhj" Jul 30 01:07:14.832818 ip-10-0-142-6 kubenswrapper[2564]: I0730 01:07:14.832629 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5146083b-485d-406b-ae56-54eaf05efef8-tmp-dir\") pod \"rate-test-model-a47fe8-kserve-7cc787b7f-q8bhj\" (UID: \"5146083b-485d-406b-ae56-54eaf05efef8\") " pod="ai-tenant-e2e-shared-a-a47fe8/rate-test-model-a47fe8-kserve-7cc787b7f-q8bhj" Jul 30 01:07:14.832818 ip-10-0-142-6 kubenswrapper[2564]: I0730 01:07:14.832650 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5146083b-485d-406b-ae56-54eaf05efef8-dshm\") pod \"rate-test-model-a47fe8-kserve-7cc787b7f-q8bhj\" (UID: \"5146083b-485d-406b-ae56-54eaf05efef8\") " pod="ai-tenant-e2e-shared-a-a47fe8/rate-test-model-a47fe8-kserve-7cc787b7f-q8bhj" Jul 30 01:07:14.832818 ip-10-0-142-6 kubenswrapper[2564]: I0730 01:07:14.832668 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lvqn5\" (UniqueName: \"kubernetes.io/projected/5146083b-485d-406b-ae56-54eaf05efef8-kube-api-access-lvqn5\") pod \"rate-test-model-a47fe8-kserve-7cc787b7f-q8bhj\" (UID: \"5146083b-485d-406b-ae56-54eaf05efef8\") " pod="ai-tenant-e2e-shared-a-a47fe8/rate-test-model-a47fe8-kserve-7cc787b7f-q8bhj" Jul 30 01:07:14.832818 ip-10-0-142-6 kubenswrapper[2564]: I0730 01:07:14.832803 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5146083b-485d-406b-ae56-54eaf05efef8-tls-certs\") pod \"rate-test-model-a47fe8-kserve-7cc787b7f-q8bhj\" (UID: \"5146083b-485d-406b-ae56-54eaf05efef8\") " pod="ai-tenant-e2e-shared-a-a47fe8/rate-test-model-a47fe8-kserve-7cc787b7f-q8bhj" Jul 30 01:07:14.832996 ip-10-0-142-6 kubenswrapper[2564]: I0730 01:07:14.832977 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5146083b-485d-406b-ae56-54eaf05efef8-home\") pod \"rate-test-model-a47fe8-kserve-7cc787b7f-q8bhj\" (UID: \"5146083b-485d-406b-ae56-54eaf05efef8\") " pod="ai-tenant-e2e-shared-a-a47fe8/rate-test-model-a47fe8-kserve-7cc787b7f-q8bhj" Jul 30 01:07:14.833069 ip-10-0-142-6 kubenswrapper[2564]: I0730 01:07:14.833040 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5146083b-485d-406b-ae56-54eaf05efef8-model-cache\") pod \"rate-test-model-a47fe8-kserve-7cc787b7f-q8bhj\" (UID: \"5146083b-485d-406b-ae56-54eaf05efef8\") " pod="ai-tenant-e2e-shared-a-a47fe8/rate-test-model-a47fe8-kserve-7cc787b7f-q8bhj" Jul 30 01:07:14.833069 ip-10-0-142-6 kubenswrapper[2564]: I0730 01:07:14.833068 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5146083b-485d-406b-ae56-54eaf05efef8-tmp-dir\") pod \"rate-test-model-a47fe8-kserve-7cc787b7f-q8bhj\" (UID: \"5146083b-485d-406b-ae56-54eaf05efef8\") " pod="ai-tenant-e2e-shared-a-a47fe8/rate-test-model-a47fe8-kserve-7cc787b7f-q8bhj" Jul 30 01:07:14.834921 ip-10-0-142-6 kubenswrapper[2564]: I0730 01:07:14.834901 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5146083b-485d-406b-ae56-54eaf05efef8-dshm\") pod \"rate-test-model-a47fe8-kserve-7cc787b7f-q8bhj\" (UID: \"5146083b-485d-406b-ae56-54eaf05efef8\") " pod="ai-tenant-e2e-shared-a-a47fe8/rate-test-model-a47fe8-kserve-7cc787b7f-q8bhj" Jul 30 01:07:14.835160 ip-10-0-142-6 kubenswrapper[2564]: I0730 01:07:14.835144 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5146083b-485d-406b-ae56-54eaf05efef8-tls-certs\") pod \"rate-test-model-a47fe8-kserve-7cc787b7f-q8bhj\" (UID: \"5146083b-485d-406b-ae56-54eaf05efef8\") " pod="ai-tenant-e2e-shared-a-a47fe8/rate-test-model-a47fe8-kserve-7cc787b7f-q8bhj" Jul 30 01:07:14.840826 ip-10-0-142-6 kubenswrapper[2564]: I0730 01:07:14.840803 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-lvqn5\" (UniqueName: \"kubernetes.io/projected/5146083b-485d-406b-ae56-54eaf05efef8-kube-api-access-lvqn5\") pod \"rate-test-model-a47fe8-kserve-7cc787b7f-q8bhj\" (UID: \"5146083b-485d-406b-ae56-54eaf05efef8\") " pod="ai-tenant-e2e-shared-a-a47fe8/rate-test-model-a47fe8-kserve-7cc787b7f-q8bhj" Jul 30 01:07:14.942867 ip-10-0-142-6 kubenswrapper[2564]: I0730 01:07:14.942782 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-a47fe8/rate-test-model-a47fe8-kserve-7cc787b7f-q8bhj" Jul 30 01:07:15.077358 ip-10-0-142-6 kubenswrapper[2564]: I0730 01:07:15.077321 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-a47fe8/rate-test-model-a47fe8-kserve-7cc787b7f-q8bhj"] Jul 30 01:07:15.082841 ip-10-0-142-6 kubenswrapper[2564]: W0730 01:07:15.082816 2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod5146083b_485d_406b_ae56_54eaf05efef8.slice/crio-53fb23264156efc982caa410f93d4d212f40c85a287925d0f980aa8939489281 WatchSource:0}: Error finding container 53fb23264156efc982caa410f93d4d212f40c85a287925d0f980aa8939489281: Status 404 returned error can't find the container with id 53fb23264156efc982caa410f93d4d212f40c85a287925d0f980aa8939489281 Jul 30 01:07:15.174603 ip-10-0-142-6 kubenswrapper[2564]: I0730 01:07:15.174565 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-a47fe8/rate-test-model-a47fe8-kserve-7cc787b7f-q8bhj" event={"ID":"5146083b-485d-406b-ae56-54eaf05efef8","Type":"ContainerStarted","Data":"d13b789d83ea29a47fab817465c2dda95550fcb2176f6df9398fa59617533520"} Jul 30 01:07:15.174773 ip-10-0-142-6 kubenswrapper[2564]: I0730 01:07:15.174615 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-a47fe8/rate-test-model-a47fe8-kserve-7cc787b7f-q8bhj" event={"ID":"5146083b-485d-406b-ae56-54eaf05efef8","Type":"ContainerStarted","Data":"53fb23264156efc982caa410f93d4d212f40c85a287925d0f980aa8939489281"} Jul 30 01:07:15.192372 ip-10-0-142-6 kubenswrapper[2564]: I0730 01:07:15.192320 2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-a-a47fe8/rate-test-model-a47fe8-kserve-7cc787b7f-q8bhj" podStartSLOduration=1.192306062 podStartE2EDuration="1.192306062s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-30 01:07:14 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-30 01:07:15.189992333 +0000 UTC m=+2966.604732290" watchObservedRunningTime="2026-07-30 01:07:15.192306062 +0000 UTC m=+2966.607045995" Jul 30 01:07:18.211446 ip-10-0-142-6 kubenswrapper[2564]: I0730 01:07:18.211408 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-nrv2p"] Jul 30 01:07:24.943345 ip-10-0-142-6 kubenswrapper[2564]: I0730 01:07:24.943308 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-a-a47fe8/rate-test-model-a47fe8-kserve-7cc787b7f-q8bhj" Jul 30 01:07:24.943766 ip-10-0-142-6 kubenswrapper[2564]: I0730 01:07:24.943366 2564 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-a-a47fe8/rate-test-model-a47fe8-kserve-7cc787b7f-q8bhj" Jul 30 01:07:24.955857 ip-10-0-142-6 kubenswrapper[2564]: I0730 01:07:24.955824 2564 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-a-a47fe8/rate-test-model-a47fe8-kserve-7cc787b7f-q8bhj" Jul 30 01:07:25.233516 ip-10-0-142-6 kubenswrapper[2564]: I0730 01:07:25.233438 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-a-a47fe8/rate-test-model-a47fe8-kserve-7cc787b7f-q8bhj" Jul 30 01:07:36.805797 ip-10-0-142-6 kubenswrapper[2564]: I0730 01:07:36.805758 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-nrv2p"] Jul 30 01:08:10.605009 ip-10-0-142-6 kubenswrapper[2564]: I0730 01:08:10.604971 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-nrv2p"] Jul 30 01:08:30.404839 ip-10-0-142-6 kubenswrapper[2564]: I0730 01:08:30.404798 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-nrv2p"] Jul 30 01:08:37.429062 ip-10-0-142-6 kubenswrapper[2564]: I0730 01:08:37.429024 2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-a47fe8/rate-test-model-a47fe8-kserve-7cc787b7f-q8bhj"] Jul 30 01:08:39.245596 ip-10-0-142-6 kubenswrapper[2564]: I0730 01:08:39.245556 2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7574-140267593049904"] Jul 30 01:08:39.248897 ip-10-0-142-6 kubenswrapper[2564]: I0730 01:08:39.248879 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7574-140267593049904" Jul 30 01:08:39.251185 ip-10-0-142-6 kubenswrapper[2564]: I0730 01:08:39.251166 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"default-dockercfg-6jwfc\"" Jul 30 01:08:39.255339 ip-10-0-142-6 kubenswrapper[2564]: I0730 01:08:39.255315 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7574-140267593049904"] Jul 30 01:08:39.424140 ip-10-0-142-6 kubenswrapper[2564]: I0730 01:08:39.424093 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xldpj\" (UniqueName: \"kubernetes.io/projected/8955c9ae-e1e4-4002-b247-24c330f4c5c8-kube-api-access-xldpj\") pod \"test-curl-7574-140267593049904\" (UID: \"8955c9ae-e1e4-4002-b247-24c330f4c5c8\") " pod="opendatahub/test-curl-7574-140267593049904" Jul 30 01:08:39.524729 ip-10-0-142-6 kubenswrapper[2564]: I0730 01:08:39.524644 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xldpj\" (UniqueName: \"kubernetes.io/projected/8955c9ae-e1e4-4002-b247-24c330f4c5c8-kube-api-access-xldpj\") pod \"test-curl-7574-140267593049904\" (UID: \"8955c9ae-e1e4-4002-b247-24c330f4c5c8\") " pod="opendatahub/test-curl-7574-140267593049904" Jul 30 01:08:39.532878 ip-10-0-142-6 kubenswrapper[2564]: I0730 01:08:39.532856 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xldpj\" (UniqueName: \"kubernetes.io/projected/8955c9ae-e1e4-4002-b247-24c330f4c5c8-kube-api-access-xldpj\") pod \"test-curl-7574-140267593049904\" (UID: \"8955c9ae-e1e4-4002-b247-24c330f4c5c8\") " pod="opendatahub/test-curl-7574-140267593049904" Jul 30 01:08:39.559403 ip-10-0-142-6 kubenswrapper[2564]: I0730 01:08:39.559376 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7574-140267593049904" Jul 30 01:08:39.686578 ip-10-0-142-6 kubenswrapper[2564]: I0730 01:08:39.686520 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7574-140267593049904"] Jul 30 01:08:39.688750 ip-10-0-142-6 kubenswrapper[2564]: W0730 01:08:39.688722 2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod8955c9ae_e1e4_4002_b247_24c330f4c5c8.slice/crio-383f93c341b4b5891a9e8474c1dfa2137885f7c3fd6db41cb848e5333d6110c4 WatchSource:0}: Error finding container 383f93c341b4b5891a9e8474c1dfa2137885f7c3fd6db41cb848e5333d6110c4: Status 404 returned error can't find the container with id 383f93c341b4b5891a9e8474c1dfa2137885f7c3fd6db41cb848e5333d6110c4 Jul 30 01:08:39.690127 ip-10-0-142-6 kubenswrapper[2564]: I0730 01:08:39.690107 2564 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Jul 30 01:08:40.600505 ip-10-0-142-6 kubenswrapper[2564]: I0730 01:08:40.600476 2564 generic.go:356] "Generic (PLEG): container finished" podID="8955c9ae-e1e4-4002-b247-24c330f4c5c8" containerID="233313db699975e6699944621a15ca1fa1639d098edf639f52c9fe38ae35424f" exitCode=0 Jul 30 01:08:40.600889 ip-10-0-142-6 kubenswrapper[2564]: I0730 01:08:40.600522 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7574-140267593049904" event={"ID":"8955c9ae-e1e4-4002-b247-24c330f4c5c8","Type":"ContainerDied","Data":"233313db699975e6699944621a15ca1fa1639d098edf639f52c9fe38ae35424f"} Jul 30 01:08:40.600889 ip-10-0-142-6 kubenswrapper[2564]: I0730 01:08:40.600551 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7574-140267593049904" event={"ID":"8955c9ae-e1e4-4002-b247-24c330f4c5c8","Type":"ContainerStarted","Data":"383f93c341b4b5891a9e8474c1dfa2137885f7c3fd6db41cb848e5333d6110c4"} Jul 30 01:08:41.737151 ip-10-0-142-6 kubenswrapper[2564]: I0730 01:08:41.737128 2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7574-140267593049904" Jul 30 01:08:41.738531 ip-10-0-142-6 kubenswrapper[2564]: I0730 01:08:41.738508 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-xldpj\" (UniqueName: \"kubernetes.io/projected/8955c9ae-e1e4-4002-b247-24c330f4c5c8-kube-api-access-xldpj\") pod \"8955c9ae-e1e4-4002-b247-24c330f4c5c8\" (UID: \"8955c9ae-e1e4-4002-b247-24c330f4c5c8\") " Jul 30 01:08:41.740605 ip-10-0-142-6 kubenswrapper[2564]: I0730 01:08:41.740580 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8955c9ae-e1e4-4002-b247-24c330f4c5c8-kube-api-access-xldpj" (OuterVolumeSpecName: "kube-api-access-xldpj") pod "8955c9ae-e1e4-4002-b247-24c330f4c5c8" (UID: "8955c9ae-e1e4-4002-b247-24c330f4c5c8"). InnerVolumeSpecName "kube-api-access-xldpj". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 30 01:08:41.771668 ip-10-0-142-6 kubenswrapper[2564]: I0730 01:08:41.771638 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_test-curl-7574-140267593049904_8955c9ae-e1e4-4002-b247-24c330f4c5c8/test-curl-7574-140267593049904/0.log" Jul 30 01:08:41.796340 ip-10-0-142-6 kubenswrapper[2564]: I0730 01:08:41.796302 2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7574-140267593049904"] Jul 30 01:08:41.799481 ip-10-0-142-6 kubenswrapper[2564]: I0730 01:08:41.799452 2564 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7574-140267593049904"] Jul 30 01:08:41.839840 ip-10-0-142-6 kubenswrapper[2564]: I0730 01:08:41.839809 2564 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xldpj\" (UniqueName: \"kubernetes.io/projected/8955c9ae-e1e4-4002-b247-24c330f4c5c8-kube-api-access-xldpj\") on node \"ip-10-0-142-6.ec2.internal\" DevicePath \"\"" Jul 30 01:08:41.910902 ip-10-0-142-6 kubenswrapper[2564]: I0730 01:08:41.910869 2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7574-140267593051952"] Jul 30 01:08:41.911232 ip-10-0-142-6 kubenswrapper[2564]: I0730 01:08:41.911218 2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8955c9ae-e1e4-4002-b247-24c330f4c5c8" containerName="test-curl-7574-140267593049904" Jul 30 01:08:41.911283 ip-10-0-142-6 kubenswrapper[2564]: I0730 01:08:41.911234 2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="8955c9ae-e1e4-4002-b247-24c330f4c5c8" containerName="test-curl-7574-140267593049904" Jul 30 01:08:41.911318 ip-10-0-142-6 kubenswrapper[2564]: I0730 01:08:41.911291 2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="8955c9ae-e1e4-4002-b247-24c330f4c5c8" containerName="test-curl-7574-140267593049904" Jul 30 01:08:41.914209 ip-10-0-142-6 kubenswrapper[2564]: I0730 01:08:41.914193 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7574-140267593051952" Jul 30 01:08:41.920845 ip-10-0-142-6 kubenswrapper[2564]: I0730 01:08:41.920820 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7574-140267593051952"] Jul 30 01:08:41.941186 ip-10-0-142-6 kubenswrapper[2564]: I0730 01:08:41.941153 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8wkbl\" (UniqueName: \"kubernetes.io/projected/20227bca-9676-4666-9e81-8f235c65327b-kube-api-access-8wkbl\") pod \"test-curl-7574-140267593051952\" (UID: \"20227bca-9676-4666-9e81-8f235c65327b\") " pod="opendatahub/test-curl-7574-140267593051952" Jul 30 01:08:42.041774 ip-10-0-142-6 kubenswrapper[2564]: I0730 01:08:42.041744 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8wkbl\" (UniqueName: \"kubernetes.io/projected/20227bca-9676-4666-9e81-8f235c65327b-kube-api-access-8wkbl\") pod \"test-curl-7574-140267593051952\" (UID: \"20227bca-9676-4666-9e81-8f235c65327b\") " pod="opendatahub/test-curl-7574-140267593051952" Jul 30 01:08:42.049979 ip-10-0-142-6 kubenswrapper[2564]: I0730 01:08:42.049948 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8wkbl\" (UniqueName: \"kubernetes.io/projected/20227bca-9676-4666-9e81-8f235c65327b-kube-api-access-8wkbl\") pod \"test-curl-7574-140267593051952\" (UID: \"20227bca-9676-4666-9e81-8f235c65327b\") " pod="opendatahub/test-curl-7574-140267593051952" Jul 30 01:08:42.224892 ip-10-0-142-6 kubenswrapper[2564]: I0730 01:08:42.224806 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7574-140267593051952" Jul 30 01:08:42.353451 ip-10-0-142-6 kubenswrapper[2564]: I0730 01:08:42.353428 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7574-140267593051952"] Jul 30 01:08:42.355098 ip-10-0-142-6 kubenswrapper[2564]: W0730 01:08:42.355054 2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod20227bca_9676_4666_9e81_8f235c65327b.slice/crio-dd803ffd6733410fa4821e9dd3642a86869d46469800ed789b7738059b319e6a WatchSource:0}: Error finding container dd803ffd6733410fa4821e9dd3642a86869d46469800ed789b7738059b319e6a: Status 404 returned error can't find the container with id dd803ffd6733410fa4821e9dd3642a86869d46469800ed789b7738059b319e6a Jul 30 01:08:42.611577 ip-10-0-142-6 kubenswrapper[2564]: I0730 01:08:42.611485 2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7574-140267593049904" Jul 30 01:08:42.611577 ip-10-0-142-6 kubenswrapper[2564]: I0730 01:08:42.611509 2564 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="383f93c341b4b5891a9e8474c1dfa2137885f7c3fd6db41cb848e5333d6110c4" Jul 30 01:08:42.612682 ip-10-0-142-6 kubenswrapper[2564]: I0730 01:08:42.612660 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7574-140267593051952" event={"ID":"20227bca-9676-4666-9e81-8f235c65327b","Type":"ContainerStarted","Data":"dd803ffd6733410fa4821e9dd3642a86869d46469800ed789b7738059b319e6a"} Jul 30 01:08:42.624657 ip-10-0-142-6 kubenswrapper[2564]: E0730 01:08:42.624626 2564 status_manager.go:1018] "Failed to get status for pod" err="pods \"test-curl-7574-140267593049904\" is forbidden: User \"system:node:ip-10-0-142-6.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-142-6.ec2.internal' and this object" podUID="8955c9ae-e1e4-4002-b247-24c330f4c5c8" pod="opendatahub/test-curl-7574-140267593049904" Jul 30 01:08:43.139395 ip-10-0-142-6 kubenswrapper[2564]: I0730 01:08:43.139361 2564 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="8955c9ae-e1e4-4002-b247-24c330f4c5c8" path="/var/lib/kubelet/pods/8955c9ae-e1e4-4002-b247-24c330f4c5c8/volumes" Jul 30 01:08:43.619151 ip-10-0-142-6 kubenswrapper[2564]: I0730 01:08:43.619107 2564 generic.go:356] "Generic (PLEG): container finished" podID="20227bca-9676-4666-9e81-8f235c65327b" containerID="fd64266e3feb01f33f8ee2d272836c6980bfa1e6c25a133863951df117b0c56f" exitCode=0 Jul 30 01:08:43.619318 ip-10-0-142-6 kubenswrapper[2564]: I0730 01:08:43.619200 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7574-140267593051952" event={"ID":"20227bca-9676-4666-9e81-8f235c65327b","Type":"ContainerDied","Data":"fd64266e3feb01f33f8ee2d272836c6980bfa1e6c25a133863951df117b0c56f"} Jul 30 01:08:44.757345 ip-10-0-142-6 kubenswrapper[2564]: I0730 01:08:44.757314 2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7574-140267593051952" Jul 30 01:08:44.759676 ip-10-0-142-6 kubenswrapper[2564]: I0730 01:08:44.759656 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-8wkbl\" (UniqueName: \"kubernetes.io/projected/20227bca-9676-4666-9e81-8f235c65327b-kube-api-access-8wkbl\") pod \"20227bca-9676-4666-9e81-8f235c65327b\" (UID: \"20227bca-9676-4666-9e81-8f235c65327b\") " Jul 30 01:08:44.761652 ip-10-0-142-6 kubenswrapper[2564]: I0730 01:08:44.761623 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/20227bca-9676-4666-9e81-8f235c65327b-kube-api-access-8wkbl" (OuterVolumeSpecName: "kube-api-access-8wkbl") pod "20227bca-9676-4666-9e81-8f235c65327b" (UID: "20227bca-9676-4666-9e81-8f235c65327b"). InnerVolumeSpecName "kube-api-access-8wkbl". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 30 01:08:44.779976 ip-10-0-142-6 kubenswrapper[2564]: I0730 01:08:44.779945 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_test-curl-7574-140267593051952_20227bca-9676-4666-9e81-8f235c65327b/test-curl-7574-140267593051952/0.log" Jul 30 01:08:44.800979 ip-10-0-142-6 kubenswrapper[2564]: I0730 01:08:44.800947 2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7574-140267593051952"] Jul 30 01:08:44.807046 ip-10-0-142-6 kubenswrapper[2564]: I0730 01:08:44.807014 2564 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7574-140267593051952"] Jul 30 01:08:44.860212 ip-10-0-142-6 kubenswrapper[2564]: I0730 01:08:44.860185 2564 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8wkbl\" (UniqueName: \"kubernetes.io/projected/20227bca-9676-4666-9e81-8f235c65327b-kube-api-access-8wkbl\") on node \"ip-10-0-142-6.ec2.internal\" DevicePath \"\"" Jul 30 01:08:44.953197 ip-10-0-142-6 kubenswrapper[2564]: I0730 01:08:44.953111 2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7574-140267593033744"] Jul 30 01:08:44.953445 ip-10-0-142-6 kubenswrapper[2564]: I0730 01:08:44.953432 2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="20227bca-9676-4666-9e81-8f235c65327b" containerName="test-curl-7574-140267593051952" Jul 30 01:08:44.953488 ip-10-0-142-6 kubenswrapper[2564]: I0730 01:08:44.953447 2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="20227bca-9676-4666-9e81-8f235c65327b" containerName="test-curl-7574-140267593051952" Jul 30 01:08:44.953570 ip-10-0-142-6 kubenswrapper[2564]: I0730 01:08:44.953512 2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="20227bca-9676-4666-9e81-8f235c65327b" containerName="test-curl-7574-140267593051952" Jul 30 01:08:44.956346 ip-10-0-142-6 kubenswrapper[2564]: I0730 01:08:44.956331 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7574-140267593033744" Jul 30 01:08:44.960787 ip-10-0-142-6 kubenswrapper[2564]: I0730 01:08:44.960750 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dx64c\" (UniqueName: \"kubernetes.io/projected/a87510c7-9f56-43b6-9b9a-07081b62a26f-kube-api-access-dx64c\") pod \"test-curl-7574-140267593033744\" (UID: \"a87510c7-9f56-43b6-9b9a-07081b62a26f\") " pod="opendatahub/test-curl-7574-140267593033744" Jul 30 01:08:44.962525 ip-10-0-142-6 kubenswrapper[2564]: I0730 01:08:44.962499 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7574-140267593033744"] Jul 30 01:08:45.061548 ip-10-0-142-6 kubenswrapper[2564]: I0730 01:08:45.061516 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dx64c\" (UniqueName: \"kubernetes.io/projected/a87510c7-9f56-43b6-9b9a-07081b62a26f-kube-api-access-dx64c\") pod \"test-curl-7574-140267593033744\" (UID: \"a87510c7-9f56-43b6-9b9a-07081b62a26f\") " pod="opendatahub/test-curl-7574-140267593033744" Jul 30 01:08:45.072224 ip-10-0-142-6 kubenswrapper[2564]: I0730 01:08:45.072194 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-dx64c\" (UniqueName: \"kubernetes.io/projected/a87510c7-9f56-43b6-9b9a-07081b62a26f-kube-api-access-dx64c\") pod \"test-curl-7574-140267593033744\" (UID: \"a87510c7-9f56-43b6-9b9a-07081b62a26f\") " pod="opendatahub/test-curl-7574-140267593033744" Jul 30 01:08:45.139592 ip-10-0-142-6 kubenswrapper[2564]: I0730 01:08:45.139559 2564 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="20227bca-9676-4666-9e81-8f235c65327b" path="/var/lib/kubelet/pods/20227bca-9676-4666-9e81-8f235c65327b/volumes" Jul 30 01:08:45.208466 ip-10-0-142-6 kubenswrapper[2564]: I0730 01:08:45.208386 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-nrv2p"] Jul 30 01:08:45.269330 ip-10-0-142-6 kubenswrapper[2564]: I0730 01:08:45.269297 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7574-140267593033744" Jul 30 01:08:45.399427 ip-10-0-142-6 kubenswrapper[2564]: I0730 01:08:45.399403 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7574-140267593033744"] Jul 30 01:08:45.401318 ip-10-0-142-6 kubenswrapper[2564]: W0730 01:08:45.401284 2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-poda87510c7_9f56_43b6_9b9a_07081b62a26f.slice/crio-e1368dbfc71f876e1de01bc6a4c07ad51726ca715956e053e055f23d16253b70 WatchSource:0}: Error finding container e1368dbfc71f876e1de01bc6a4c07ad51726ca715956e053e055f23d16253b70: Status 404 returned error can't find the container with id e1368dbfc71f876e1de01bc6a4c07ad51726ca715956e053e055f23d16253b70 Jul 30 01:08:45.630996 ip-10-0-142-6 kubenswrapper[2564]: I0730 01:08:45.630961 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7574-140267593033744" event={"ID":"a87510c7-9f56-43b6-9b9a-07081b62a26f","Type":"ContainerStarted","Data":"e1368dbfc71f876e1de01bc6a4c07ad51726ca715956e053e055f23d16253b70"} Jul 30 01:08:45.632164 ip-10-0-142-6 kubenswrapper[2564]: I0730 01:08:45.632136 2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7574-140267593051952" Jul 30 01:08:45.632164 ip-10-0-142-6 kubenswrapper[2564]: I0730 01:08:45.632159 2564 scope.go:117] "RemoveContainer" containerID="fd64266e3feb01f33f8ee2d272836c6980bfa1e6c25a133863951df117b0c56f" Jul 30 01:08:46.638327 ip-10-0-142-6 kubenswrapper[2564]: I0730 01:08:46.638297 2564 generic.go:356] "Generic (PLEG): container finished" podID="a87510c7-9f56-43b6-9b9a-07081b62a26f" containerID="12f3f85339edf29accc365129610bc4ea008394da5fc090244c459a9e0085a0f" exitCode=0 Jul 30 01:08:46.638746 ip-10-0-142-6 kubenswrapper[2564]: I0730 01:08:46.638390 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7574-140267593033744" event={"ID":"a87510c7-9f56-43b6-9b9a-07081b62a26f","Type":"ContainerDied","Data":"12f3f85339edf29accc365129610bc4ea008394da5fc090244c459a9e0085a0f"} Jul 30 01:08:47.773161 ip-10-0-142-6 kubenswrapper[2564]: I0730 01:08:47.773140 2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7574-140267593033744" Jul 30 01:08:47.779273 ip-10-0-142-6 kubenswrapper[2564]: I0730 01:08:47.779243 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-dx64c\" (UniqueName: \"kubernetes.io/projected/a87510c7-9f56-43b6-9b9a-07081b62a26f-kube-api-access-dx64c\") pod \"a87510c7-9f56-43b6-9b9a-07081b62a26f\" (UID: \"a87510c7-9f56-43b6-9b9a-07081b62a26f\") " Jul 30 01:08:47.781194 ip-10-0-142-6 kubenswrapper[2564]: I0730 01:08:47.781168 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a87510c7-9f56-43b6-9b9a-07081b62a26f-kube-api-access-dx64c" (OuterVolumeSpecName: "kube-api-access-dx64c") pod "a87510c7-9f56-43b6-9b9a-07081b62a26f" (UID: "a87510c7-9f56-43b6-9b9a-07081b62a26f"). InnerVolumeSpecName "kube-api-access-dx64c". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 30 01:08:47.790222 ip-10-0-142-6 kubenswrapper[2564]: I0730 01:08:47.790196 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_test-curl-7574-140267593033744_a87510c7-9f56-43b6-9b9a-07081b62a26f/test-curl-7574-140267593033744/0.log" Jul 30 01:08:47.810154 ip-10-0-142-6 kubenswrapper[2564]: I0730 01:08:47.810125 2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7574-140267593033744"] Jul 30 01:08:47.815867 ip-10-0-142-6 kubenswrapper[2564]: I0730 01:08:47.815846 2564 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7574-140267593033744"] Jul 30 01:08:47.879771 ip-10-0-142-6 kubenswrapper[2564]: I0730 01:08:47.879742 2564 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dx64c\" (UniqueName: \"kubernetes.io/projected/a87510c7-9f56-43b6-9b9a-07081b62a26f-kube-api-access-dx64c\") on node \"ip-10-0-142-6.ec2.internal\" DevicePath \"\"" Jul 30 01:08:47.925168 ip-10-0-142-6 kubenswrapper[2564]: I0730 01:08:47.925089 2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7574-140267593033744"] Jul 30 01:08:47.925422 ip-10-0-142-6 kubenswrapper[2564]: I0730 01:08:47.925410 2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a87510c7-9f56-43b6-9b9a-07081b62a26f" containerName="test-curl-7574-140267593033744" Jul 30 01:08:47.925460 ip-10-0-142-6 kubenswrapper[2564]: I0730 01:08:47.925424 2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="a87510c7-9f56-43b6-9b9a-07081b62a26f" containerName="test-curl-7574-140267593033744" Jul 30 01:08:47.925504 ip-10-0-142-6 kubenswrapper[2564]: I0730 01:08:47.925472 2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="a87510c7-9f56-43b6-9b9a-07081b62a26f" containerName="test-curl-7574-140267593033744" Jul 30 01:08:47.928289 ip-10-0-142-6 kubenswrapper[2564]: I0730 01:08:47.928274 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7574-140267593033744" Jul 30 01:08:47.934587 ip-10-0-142-6 kubenswrapper[2564]: I0730 01:08:47.934559 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7574-140267593033744"] Jul 30 01:08:47.980212 ip-10-0-142-6 kubenswrapper[2564]: I0730 01:08:47.980183 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8vm2r\" (UniqueName: \"kubernetes.io/projected/39b1e0e5-d8b5-4a01-b0e9-6eee968e5ef6-kube-api-access-8vm2r\") pod \"test-curl-7574-140267593033744\" (UID: \"39b1e0e5-d8b5-4a01-b0e9-6eee968e5ef6\") " pod="opendatahub/test-curl-7574-140267593033744" Jul 30 01:08:48.080745 ip-10-0-142-6 kubenswrapper[2564]: I0730 01:08:48.080711 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8vm2r\" (UniqueName: \"kubernetes.io/projected/39b1e0e5-d8b5-4a01-b0e9-6eee968e5ef6-kube-api-access-8vm2r\") pod \"test-curl-7574-140267593033744\" (UID: \"39b1e0e5-d8b5-4a01-b0e9-6eee968e5ef6\") " pod="opendatahub/test-curl-7574-140267593033744" Jul 30 01:08:48.088742 ip-10-0-142-6 kubenswrapper[2564]: I0730 01:08:48.088711 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8vm2r\" (UniqueName: \"kubernetes.io/projected/39b1e0e5-d8b5-4a01-b0e9-6eee968e5ef6-kube-api-access-8vm2r\") pod \"test-curl-7574-140267593033744\" (UID: \"39b1e0e5-d8b5-4a01-b0e9-6eee968e5ef6\") " pod="opendatahub/test-curl-7574-140267593033744" Jul 30 01:08:48.239210 ip-10-0-142-6 kubenswrapper[2564]: I0730 01:08:48.239130 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7574-140267593033744" Jul 30 01:08:48.367376 ip-10-0-142-6 kubenswrapper[2564]: I0730 01:08:48.367350 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7574-140267593033744"] Jul 30 01:08:48.369285 ip-10-0-142-6 kubenswrapper[2564]: W0730 01:08:48.369253 2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod39b1e0e5_d8b5_4a01_b0e9_6eee968e5ef6.slice/crio-03ec2afd17b9b574700e1f30312c3a3204b608d5070ae32f8ecda79ec477a022 WatchSource:0}: Error finding container 03ec2afd17b9b574700e1f30312c3a3204b608d5070ae32f8ecda79ec477a022: Status 404 returned error can't find the container with id 03ec2afd17b9b574700e1f30312c3a3204b608d5070ae32f8ecda79ec477a022 Jul 30 01:08:48.651336 ip-10-0-142-6 kubenswrapper[2564]: I0730 01:08:48.651270 2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7574-140267593033744" Jul 30 01:08:48.651336 ip-10-0-142-6 kubenswrapper[2564]: I0730 01:08:48.651287 2564 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="e1368dbfc71f876e1de01bc6a4c07ad51726ca715956e053e055f23d16253b70" Jul 30 01:08:48.652515 ip-10-0-142-6 kubenswrapper[2564]: I0730 01:08:48.652491 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7574-140267593033744" event={"ID":"39b1e0e5-d8b5-4a01-b0e9-6eee968e5ef6","Type":"ContainerStarted","Data":"03ec2afd17b9b574700e1f30312c3a3204b608d5070ae32f8ecda79ec477a022"} Jul 30 01:08:48.662781 ip-10-0-142-6 kubenswrapper[2564]: I0730 01:08:48.662750 2564 status_manager.go:1027] "Pod was deleted and then recreated, skipping status update" pod="opendatahub/test-curl-7574-140267593033744" oldPodUID="a87510c7-9f56-43b6-9b9a-07081b62a26f" podUID="39b1e0e5-d8b5-4a01-b0e9-6eee968e5ef6" Jul 30 01:08:49.139856 ip-10-0-142-6 kubenswrapper[2564]: I0730 01:08:49.139826 2564 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="a87510c7-9f56-43b6-9b9a-07081b62a26f" path="/var/lib/kubelet/pods/a87510c7-9f56-43b6-9b9a-07081b62a26f/volumes" Jul 30 01:08:49.659217 ip-10-0-142-6 kubenswrapper[2564]: I0730 01:08:49.659183 2564 generic.go:356] "Generic (PLEG): container finished" podID="39b1e0e5-d8b5-4a01-b0e9-6eee968e5ef6" containerID="ead27e1b2eb8834d9e65caa5370ec0d6f9da2e41fca9b3a810cf42fa31b0a8dd" exitCode=0 Jul 30 01:08:49.659389 ip-10-0-142-6 kubenswrapper[2564]: I0730 01:08:49.659235 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7574-140267593033744" event={"ID":"39b1e0e5-d8b5-4a01-b0e9-6eee968e5ef6","Type":"ContainerDied","Data":"ead27e1b2eb8834d9e65caa5370ec0d6f9da2e41fca9b3a810cf42fa31b0a8dd"} Jul 30 01:08:50.797639 ip-10-0-142-6 kubenswrapper[2564]: I0730 01:08:50.797616 2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7574-140267593033744" Jul 30 01:08:50.799543 ip-10-0-142-6 kubenswrapper[2564]: I0730 01:08:50.799526 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-8vm2r\" (UniqueName: \"kubernetes.io/projected/39b1e0e5-d8b5-4a01-b0e9-6eee968e5ef6-kube-api-access-8vm2r\") pod \"39b1e0e5-d8b5-4a01-b0e9-6eee968e5ef6\" (UID: \"39b1e0e5-d8b5-4a01-b0e9-6eee968e5ef6\") " Jul 30 01:08:50.801547 ip-10-0-142-6 kubenswrapper[2564]: I0730 01:08:50.801525 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/39b1e0e5-d8b5-4a01-b0e9-6eee968e5ef6-kube-api-access-8vm2r" (OuterVolumeSpecName: "kube-api-access-8vm2r") pod "39b1e0e5-d8b5-4a01-b0e9-6eee968e5ef6" (UID: "39b1e0e5-d8b5-4a01-b0e9-6eee968e5ef6"). InnerVolumeSpecName "kube-api-access-8vm2r". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 30 01:08:50.816627 ip-10-0-142-6 kubenswrapper[2564]: I0730 01:08:50.816602 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_test-curl-7574-140267593033744_39b1e0e5-d8b5-4a01-b0e9-6eee968e5ef6/test-curl-7574-140267593033744/0.log" Jul 30 01:08:50.838277 ip-10-0-142-6 kubenswrapper[2564]: I0730 01:08:50.838248 2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7574-140267593033744"] Jul 30 01:08:50.841692 ip-10-0-142-6 kubenswrapper[2564]: I0730 01:08:50.841669 2564 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7574-140267593033744"] Jul 30 01:08:50.900753 ip-10-0-142-6 kubenswrapper[2564]: I0730 01:08:50.900726 2564 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8vm2r\" (UniqueName: \"kubernetes.io/projected/39b1e0e5-d8b5-4a01-b0e9-6eee968e5ef6-kube-api-access-8vm2r\") on node \"ip-10-0-142-6.ec2.internal\" DevicePath \"\"" Jul 30 01:08:50.955543 ip-10-0-142-6 kubenswrapper[2564]: I0730 01:08:50.955471 2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7574-140267593034752"] Jul 30 01:08:50.955796 ip-10-0-142-6 kubenswrapper[2564]: I0730 01:08:50.955783 2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="39b1e0e5-d8b5-4a01-b0e9-6eee968e5ef6" containerName="test-curl-7574-140267593033744" Jul 30 01:08:50.955848 ip-10-0-142-6 kubenswrapper[2564]: I0730 01:08:50.955798 2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="39b1e0e5-d8b5-4a01-b0e9-6eee968e5ef6" containerName="test-curl-7574-140267593033744" Jul 30 01:08:50.955887 ip-10-0-142-6 kubenswrapper[2564]: I0730 01:08:50.955854 2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="39b1e0e5-d8b5-4a01-b0e9-6eee968e5ef6" containerName="test-curl-7574-140267593033744" Jul 30 01:08:50.958700 ip-10-0-142-6 kubenswrapper[2564]: I0730 01:08:50.958685 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7574-140267593034752" Jul 30 01:08:50.964005 ip-10-0-142-6 kubenswrapper[2564]: I0730 01:08:50.963975 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7574-140267593034752"] Jul 30 01:08:51.001092 ip-10-0-142-6 kubenswrapper[2564]: I0730 01:08:51.001045 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8kb5c\" (UniqueName: \"kubernetes.io/projected/dc4c5a5d-2596-403e-9883-91953c4646ba-kube-api-access-8kb5c\") pod \"test-curl-7574-140267593034752\" (UID: \"dc4c5a5d-2596-403e-9883-91953c4646ba\") " pod="opendatahub/test-curl-7574-140267593034752" Jul 30 01:08:51.101598 ip-10-0-142-6 kubenswrapper[2564]: I0730 01:08:51.101561 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8kb5c\" (UniqueName: \"kubernetes.io/projected/dc4c5a5d-2596-403e-9883-91953c4646ba-kube-api-access-8kb5c\") pod \"test-curl-7574-140267593034752\" (UID: \"dc4c5a5d-2596-403e-9883-91953c4646ba\") " pod="opendatahub/test-curl-7574-140267593034752" Jul 30 01:08:51.109459 ip-10-0-142-6 kubenswrapper[2564]: I0730 01:08:51.109429 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8kb5c\" (UniqueName: \"kubernetes.io/projected/dc4c5a5d-2596-403e-9883-91953c4646ba-kube-api-access-8kb5c\") pod \"test-curl-7574-140267593034752\" (UID: \"dc4c5a5d-2596-403e-9883-91953c4646ba\") " pod="opendatahub/test-curl-7574-140267593034752" Jul 30 01:08:51.139312 ip-10-0-142-6 kubenswrapper[2564]: I0730 01:08:51.139276 2564 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="39b1e0e5-d8b5-4a01-b0e9-6eee968e5ef6" path="/var/lib/kubelet/pods/39b1e0e5-d8b5-4a01-b0e9-6eee968e5ef6/volumes" Jul 30 01:08:51.277235 ip-10-0-142-6 kubenswrapper[2564]: I0730 01:08:51.277128 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7574-140267593034752" Jul 30 01:08:51.405298 ip-10-0-142-6 kubenswrapper[2564]: I0730 01:08:51.405272 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7574-140267593034752"] Jul 30 01:08:51.407160 ip-10-0-142-6 kubenswrapper[2564]: W0730 01:08:51.407133 2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-poddc4c5a5d_2596_403e_9883_91953c4646ba.slice/crio-d9860862eb94b8afe8968db16f2d75eb9f5a77cf4808b1ad454a94b051f59181 WatchSource:0}: Error finding container d9860862eb94b8afe8968db16f2d75eb9f5a77cf4808b1ad454a94b051f59181: Status 404 returned error can't find the container with id d9860862eb94b8afe8968db16f2d75eb9f5a77cf4808b1ad454a94b051f59181 Jul 30 01:08:51.671359 ip-10-0-142-6 kubenswrapper[2564]: I0730 01:08:51.671320 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7574-140267593034752" event={"ID":"dc4c5a5d-2596-403e-9883-91953c4646ba","Type":"ContainerStarted","Data":"d9860862eb94b8afe8968db16f2d75eb9f5a77cf4808b1ad454a94b051f59181"} Jul 30 01:08:51.672549 ip-10-0-142-6 kubenswrapper[2564]: I0730 01:08:51.672533 2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7574-140267593033744" Jul 30 01:08:51.672609 ip-10-0-142-6 kubenswrapper[2564]: I0730 01:08:51.672563 2564 scope.go:117] "RemoveContainer" containerID="ead27e1b2eb8834d9e65caa5370ec0d6f9da2e41fca9b3a810cf42fa31b0a8dd" Jul 30 01:08:52.449910 ip-10-0-142-6 kubenswrapper[2564]: I0730 01:08:52.449875 2564 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-shared-a-a47fe8/rate-test-model-a47fe8-kserve-7cc787b7f-q8bhj" podUID="5146083b-485d-406b-ae56-54eaf05efef8" containerName="main" containerID="cri-o://d13b789d83ea29a47fab817465c2dda95550fcb2176f6df9398fa59617533520" gracePeriod=45 Jul 30 01:08:52.680258 ip-10-0-142-6 kubenswrapper[2564]: I0730 01:08:52.680229 2564 generic.go:356] "Generic (PLEG): container finished" podID="5146083b-485d-406b-ae56-54eaf05efef8" containerID="d13b789d83ea29a47fab817465c2dda95550fcb2176f6df9398fa59617533520" exitCode=0 Jul 30 01:08:52.680442 ip-10-0-142-6 kubenswrapper[2564]: I0730 01:08:52.680303 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-a47fe8/rate-test-model-a47fe8-kserve-7cc787b7f-q8bhj" event={"ID":"5146083b-485d-406b-ae56-54eaf05efef8","Type":"ContainerDied","Data":"d13b789d83ea29a47fab817465c2dda95550fcb2176f6df9398fa59617533520"} Jul 30 01:08:52.681735 ip-10-0-142-6 kubenswrapper[2564]: I0730 01:08:52.681707 2564 generic.go:356] "Generic (PLEG): container finished" podID="dc4c5a5d-2596-403e-9883-91953c4646ba" containerID="7d3f5fa3558983c40d179781d3aa634e86f7fd927ec97f85fc9b0d2d22dece7b" exitCode=0 Jul 30 01:08:52.681860 ip-10-0-142-6 kubenswrapper[2564]: I0730 01:08:52.681777 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7574-140267593034752" event={"ID":"dc4c5a5d-2596-403e-9883-91953c4646ba","Type":"ContainerDied","Data":"7d3f5fa3558983c40d179781d3aa634e86f7fd927ec97f85fc9b0d2d22dece7b"} Jul 30 01:08:52.715908 ip-10-0-142-6 kubenswrapper[2564]: I0730 01:08:52.715886 2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-a47fe8/rate-test-model-a47fe8-kserve-7cc787b7f-q8bhj" Jul 30 01:08:52.813292 ip-10-0-142-6 kubenswrapper[2564]: I0730 01:08:52.813264 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5146083b-485d-406b-ae56-54eaf05efef8-home\") pod \"5146083b-485d-406b-ae56-54eaf05efef8\" (UID: \"5146083b-485d-406b-ae56-54eaf05efef8\") " Jul 30 01:08:52.813472 ip-10-0-142-6 kubenswrapper[2564]: I0730 01:08:52.813326 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-lvqn5\" (UniqueName: \"kubernetes.io/projected/5146083b-485d-406b-ae56-54eaf05efef8-kube-api-access-lvqn5\") pod \"5146083b-485d-406b-ae56-54eaf05efef8\" (UID: \"5146083b-485d-406b-ae56-54eaf05efef8\") " Jul 30 01:08:52.813472 ip-10-0-142-6 kubenswrapper[2564]: I0730 01:08:52.813366 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5146083b-485d-406b-ae56-54eaf05efef8-tmp-dir\") pod \"5146083b-485d-406b-ae56-54eaf05efef8\" (UID: \"5146083b-485d-406b-ae56-54eaf05efef8\") " Jul 30 01:08:52.813472 ip-10-0-142-6 kubenswrapper[2564]: I0730 01:08:52.813382 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5146083b-485d-406b-ae56-54eaf05efef8-dshm\") pod \"5146083b-485d-406b-ae56-54eaf05efef8\" (UID: \"5146083b-485d-406b-ae56-54eaf05efef8\") " Jul 30 01:08:52.813472 ip-10-0-142-6 kubenswrapper[2564]: I0730 01:08:52.813396 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5146083b-485d-406b-ae56-54eaf05efef8-model-cache\") pod \"5146083b-485d-406b-ae56-54eaf05efef8\" (UID: \"5146083b-485d-406b-ae56-54eaf05efef8\") " Jul 30 01:08:52.813472 ip-10-0-142-6 kubenswrapper[2564]: I0730 01:08:52.813429 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5146083b-485d-406b-ae56-54eaf05efef8-tls-certs\") pod \"5146083b-485d-406b-ae56-54eaf05efef8\" (UID: \"5146083b-485d-406b-ae56-54eaf05efef8\") " Jul 30 01:08:52.813722 ip-10-0-142-6 kubenswrapper[2564]: I0730 01:08:52.813566 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5146083b-485d-406b-ae56-54eaf05efef8-home" (OuterVolumeSpecName: "home") pod "5146083b-485d-406b-ae56-54eaf05efef8" (UID: "5146083b-485d-406b-ae56-54eaf05efef8"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 30 01:08:52.813722 ip-10-0-142-6 kubenswrapper[2564]: I0730 01:08:52.813672 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5146083b-485d-406b-ae56-54eaf05efef8-model-cache" (OuterVolumeSpecName: "model-cache") pod "5146083b-485d-406b-ae56-54eaf05efef8" (UID: "5146083b-485d-406b-ae56-54eaf05efef8"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 30 01:08:52.813722 ip-10-0-142-6 kubenswrapper[2564]: I0730 01:08:52.813680 2564 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5146083b-485d-406b-ae56-54eaf05efef8-home\") on node \"ip-10-0-142-6.ec2.internal\" DevicePath \"\"" Jul 30 01:08:52.813722 ip-10-0-142-6 kubenswrapper[2564]: I0730 01:08:52.813687 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5146083b-485d-406b-ae56-54eaf05efef8-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "5146083b-485d-406b-ae56-54eaf05efef8" (UID: "5146083b-485d-406b-ae56-54eaf05efef8"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 30 01:08:52.815501 ip-10-0-142-6 kubenswrapper[2564]: I0730 01:08:52.815476 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5146083b-485d-406b-ae56-54eaf05efef8-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "5146083b-485d-406b-ae56-54eaf05efef8" (UID: "5146083b-485d-406b-ae56-54eaf05efef8"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 30 01:08:52.815937 ip-10-0-142-6 kubenswrapper[2564]: I0730 01:08:52.815916 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5146083b-485d-406b-ae56-54eaf05efef8-dshm" (OuterVolumeSpecName: "dshm") pod "5146083b-485d-406b-ae56-54eaf05efef8" (UID: "5146083b-485d-406b-ae56-54eaf05efef8"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 30 01:08:52.815980 ip-10-0-142-6 kubenswrapper[2564]: I0730 01:08:52.815934 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5146083b-485d-406b-ae56-54eaf05efef8-kube-api-access-lvqn5" (OuterVolumeSpecName: "kube-api-access-lvqn5") pod "5146083b-485d-406b-ae56-54eaf05efef8" (UID: "5146083b-485d-406b-ae56-54eaf05efef8"). InnerVolumeSpecName "kube-api-access-lvqn5". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 30 01:08:52.914178 ip-10-0-142-6 kubenswrapper[2564]: I0730 01:08:52.914144 2564 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lvqn5\" (UniqueName: \"kubernetes.io/projected/5146083b-485d-406b-ae56-54eaf05efef8-kube-api-access-lvqn5\") on node \"ip-10-0-142-6.ec2.internal\" DevicePath \"\"" Jul 30 01:08:52.914178 ip-10-0-142-6 kubenswrapper[2564]: I0730 01:08:52.914173 2564 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5146083b-485d-406b-ae56-54eaf05efef8-tmp-dir\") on node \"ip-10-0-142-6.ec2.internal\" DevicePath \"\"" Jul 30 01:08:52.914178 ip-10-0-142-6 kubenswrapper[2564]: I0730 01:08:52.914182 2564 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5146083b-485d-406b-ae56-54eaf05efef8-dshm\") on node \"ip-10-0-142-6.ec2.internal\" DevicePath \"\"" Jul 30 01:08:52.914178 ip-10-0-142-6 kubenswrapper[2564]: I0730 01:08:52.914190 2564 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5146083b-485d-406b-ae56-54eaf05efef8-model-cache\") on node \"ip-10-0-142-6.ec2.internal\" DevicePath \"\"" Jul 30 01:08:52.914437 ip-10-0-142-6 kubenswrapper[2564]: I0730 01:08:52.914199 2564 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5146083b-485d-406b-ae56-54eaf05efef8-tls-certs\") on node \"ip-10-0-142-6.ec2.internal\" DevicePath \"\"" Jul 30 01:08:53.688591 ip-10-0-142-6 kubenswrapper[2564]: I0730 01:08:53.688550 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-a47fe8/rate-test-model-a47fe8-kserve-7cc787b7f-q8bhj" event={"ID":"5146083b-485d-406b-ae56-54eaf05efef8","Type":"ContainerDied","Data":"53fb23264156efc982caa410f93d4d212f40c85a287925d0f980aa8939489281"} Jul 30 01:08:53.689010 ip-10-0-142-6 kubenswrapper[2564]: I0730 01:08:53.688606 2564 scope.go:117] "RemoveContainer" containerID="d13b789d83ea29a47fab817465c2dda95550fcb2176f6df9398fa59617533520" Jul 30 01:08:53.689010 ip-10-0-142-6 kubenswrapper[2564]: I0730 01:08:53.688563 2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-a47fe8/rate-test-model-a47fe8-kserve-7cc787b7f-q8bhj" Jul 30 01:08:53.705851 ip-10-0-142-6 kubenswrapper[2564]: I0730 01:08:53.705813 2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-a47fe8/rate-test-model-a47fe8-kserve-7cc787b7f-q8bhj"] Jul 30 01:08:53.707971 ip-10-0-142-6 kubenswrapper[2564]: I0730 01:08:53.707941 2564 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-a-a47fe8/rate-test-model-a47fe8-kserve-7cc787b7f-q8bhj"] Jul 30 01:08:53.820846 ip-10-0-142-6 kubenswrapper[2564]: I0730 01:08:53.820821 2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7574-140267593034752" Jul 30 01:08:53.841538 ip-10-0-142-6 kubenswrapper[2564]: I0730 01:08:53.841512 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_test-curl-7574-140267593034752_dc4c5a5d-2596-403e-9883-91953c4646ba/test-curl-7574-140267593034752/0.log" Jul 30 01:08:53.863351 ip-10-0-142-6 kubenswrapper[2564]: I0730 01:08:53.863318 2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7574-140267593034752"] Jul 30 01:08:53.866452 ip-10-0-142-6 kubenswrapper[2564]: I0730 01:08:53.866426 2564 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7574-140267593034752"] Jul 30 01:08:53.923775 ip-10-0-142-6 kubenswrapper[2564]: I0730 01:08:53.923742 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-8kb5c\" (UniqueName: \"kubernetes.io/projected/dc4c5a5d-2596-403e-9883-91953c4646ba-kube-api-access-8kb5c\") pod \"dc4c5a5d-2596-403e-9883-91953c4646ba\" (UID: \"dc4c5a5d-2596-403e-9883-91953c4646ba\") " Jul 30 01:08:53.925883 ip-10-0-142-6 kubenswrapper[2564]: I0730 01:08:53.925850 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/dc4c5a5d-2596-403e-9883-91953c4646ba-kube-api-access-8kb5c" (OuterVolumeSpecName: "kube-api-access-8kb5c") pod "dc4c5a5d-2596-403e-9883-91953c4646ba" (UID: "dc4c5a5d-2596-403e-9883-91953c4646ba"). InnerVolumeSpecName "kube-api-access-8kb5c". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 30 01:08:53.980996 ip-10-0-142-6 kubenswrapper[2564]: I0730 01:08:53.980911 2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7574-140267593034752"] Jul 30 01:08:53.981270 ip-10-0-142-6 kubenswrapper[2564]: I0730 01:08:53.981256 2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="dc4c5a5d-2596-403e-9883-91953c4646ba" containerName="test-curl-7574-140267593034752" Jul 30 01:08:53.981325 ip-10-0-142-6 kubenswrapper[2564]: I0730 01:08:53.981271 2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="dc4c5a5d-2596-403e-9883-91953c4646ba" containerName="test-curl-7574-140267593034752" Jul 30 01:08:53.981325 ip-10-0-142-6 kubenswrapper[2564]: I0730 01:08:53.981286 2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5146083b-485d-406b-ae56-54eaf05efef8" containerName="main" Jul 30 01:08:53.981325 ip-10-0-142-6 kubenswrapper[2564]: I0730 01:08:53.981292 2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="5146083b-485d-406b-ae56-54eaf05efef8" containerName="main" Jul 30 01:08:53.981414 ip-10-0-142-6 kubenswrapper[2564]: I0730 01:08:53.981348 2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="dc4c5a5d-2596-403e-9883-91953c4646ba" containerName="test-curl-7574-140267593034752" Jul 30 01:08:53.981414 ip-10-0-142-6 kubenswrapper[2564]: I0730 01:08:53.981356 2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="5146083b-485d-406b-ae56-54eaf05efef8" containerName="main" Jul 30 01:08:53.984246 ip-10-0-142-6 kubenswrapper[2564]: I0730 01:08:53.984231 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7574-140267593034752" Jul 30 01:08:53.990999 ip-10-0-142-6 kubenswrapper[2564]: I0730 01:08:53.990972 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7574-140267593034752"] Jul 30 01:08:54.025170 ip-10-0-142-6 kubenswrapper[2564]: I0730 01:08:54.025136 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zl4w9\" (UniqueName: \"kubernetes.io/projected/821ea192-dd9a-45c5-94d7-8f7b420b80d5-kube-api-access-zl4w9\") pod \"test-curl-7574-140267593034752\" (UID: \"821ea192-dd9a-45c5-94d7-8f7b420b80d5\") " pod="opendatahub/test-curl-7574-140267593034752" Jul 30 01:08:54.025301 ip-10-0-142-6 kubenswrapper[2564]: I0730 01:08:54.025190 2564 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8kb5c\" (UniqueName: \"kubernetes.io/projected/dc4c5a5d-2596-403e-9883-91953c4646ba-kube-api-access-8kb5c\") on node \"ip-10-0-142-6.ec2.internal\" DevicePath \"\"" Jul 30 01:08:54.125880 ip-10-0-142-6 kubenswrapper[2564]: I0730 01:08:54.125836 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zl4w9\" (UniqueName: \"kubernetes.io/projected/821ea192-dd9a-45c5-94d7-8f7b420b80d5-kube-api-access-zl4w9\") pod \"test-curl-7574-140267593034752\" (UID: \"821ea192-dd9a-45c5-94d7-8f7b420b80d5\") " pod="opendatahub/test-curl-7574-140267593034752" Jul 30 01:08:54.133799 ip-10-0-142-6 kubenswrapper[2564]: I0730 01:08:54.133778 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-zl4w9\" (UniqueName: \"kubernetes.io/projected/821ea192-dd9a-45c5-94d7-8f7b420b80d5-kube-api-access-zl4w9\") pod \"test-curl-7574-140267593034752\" (UID: \"821ea192-dd9a-45c5-94d7-8f7b420b80d5\") " pod="opendatahub/test-curl-7574-140267593034752" Jul 30 01:08:54.294211 ip-10-0-142-6 kubenswrapper[2564]: I0730 01:08:54.294123 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7574-140267593034752" Jul 30 01:08:54.422523 ip-10-0-142-6 kubenswrapper[2564]: I0730 01:08:54.422488 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7574-140267593034752"] Jul 30 01:08:54.423336 ip-10-0-142-6 kubenswrapper[2564]: W0730 01:08:54.423308 2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod821ea192_dd9a_45c5_94d7_8f7b420b80d5.slice/crio-0d1c31540ff66e534cccd7aea9967de456db002c92907e93c6b2b6881569e040 WatchSource:0}: Error finding container 0d1c31540ff66e534cccd7aea9967de456db002c92907e93c6b2b6881569e040: Status 404 returned error can't find the container with id 0d1c31540ff66e534cccd7aea9967de456db002c92907e93c6b2b6881569e040 Jul 30 01:08:54.695373 ip-10-0-142-6 kubenswrapper[2564]: I0730 01:08:54.695348 2564 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="d9860862eb94b8afe8968db16f2d75eb9f5a77cf4808b1ad454a94b051f59181" Jul 30 01:08:54.695373 ip-10-0-142-6 kubenswrapper[2564]: I0730 01:08:54.695364 2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7574-140267593034752" Jul 30 01:08:54.696568 ip-10-0-142-6 kubenswrapper[2564]: I0730 01:08:54.696541 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7574-140267593034752" event={"ID":"821ea192-dd9a-45c5-94d7-8f7b420b80d5","Type":"ContainerStarted","Data":"0d1c31540ff66e534cccd7aea9967de456db002c92907e93c6b2b6881569e040"} Jul 30 01:08:54.706858 ip-10-0-142-6 kubenswrapper[2564]: I0730 01:08:54.706834 2564 status_manager.go:1027] "Pod was deleted and then recreated, skipping status update" pod="opendatahub/test-curl-7574-140267593034752" oldPodUID="dc4c5a5d-2596-403e-9883-91953c4646ba" podUID="821ea192-dd9a-45c5-94d7-8f7b420b80d5" Jul 30 01:08:55.138980 ip-10-0-142-6 kubenswrapper[2564]: I0730 01:08:55.138947 2564 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="5146083b-485d-406b-ae56-54eaf05efef8" path="/var/lib/kubelet/pods/5146083b-485d-406b-ae56-54eaf05efef8/volumes" Jul 30 01:08:55.139282 ip-10-0-142-6 kubenswrapper[2564]: I0730 01:08:55.139270 2564 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="dc4c5a5d-2596-403e-9883-91953c4646ba" path="/var/lib/kubelet/pods/dc4c5a5d-2596-403e-9883-91953c4646ba/volumes" Jul 30 01:08:55.703620 ip-10-0-142-6 kubenswrapper[2564]: I0730 01:08:55.703586 2564 generic.go:356] "Generic (PLEG): container finished" podID="821ea192-dd9a-45c5-94d7-8f7b420b80d5" containerID="3976e8b810c138e38dc5a8919fd686911af3eba28ef0193c7b9115ec23bf9083" exitCode=0 Jul 30 01:08:55.704000 ip-10-0-142-6 kubenswrapper[2564]: I0730 01:08:55.703666 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7574-140267593034752" event={"ID":"821ea192-dd9a-45c5-94d7-8f7b420b80d5","Type":"ContainerDied","Data":"3976e8b810c138e38dc5a8919fd686911af3eba28ef0193c7b9115ec23bf9083"} Jul 30 01:08:56.841172 ip-10-0-142-6 kubenswrapper[2564]: I0730 01:08:56.841150 2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7574-140267593034752" Jul 30 01:08:56.859572 ip-10-0-142-6 kubenswrapper[2564]: I0730 01:08:56.859546 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_test-curl-7574-140267593034752_821ea192-dd9a-45c5-94d7-8f7b420b80d5/test-curl-7574-140267593034752/0.log" Jul 30 01:08:56.882991 ip-10-0-142-6 kubenswrapper[2564]: I0730 01:08:56.882932 2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7574-140267593034752"] Jul 30 01:08:56.885128 ip-10-0-142-6 kubenswrapper[2564]: I0730 01:08:56.885105 2564 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7574-140267593034752"] Jul 30 01:08:56.948593 ip-10-0-142-6 kubenswrapper[2564]: I0730 01:08:56.948569 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-zl4w9\" (UniqueName: \"kubernetes.io/projected/821ea192-dd9a-45c5-94d7-8f7b420b80d5-kube-api-access-zl4w9\") pod \"821ea192-dd9a-45c5-94d7-8f7b420b80d5\" (UID: \"821ea192-dd9a-45c5-94d7-8f7b420b80d5\") " Jul 30 01:08:56.950602 ip-10-0-142-6 kubenswrapper[2564]: I0730 01:08:56.950578 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/821ea192-dd9a-45c5-94d7-8f7b420b80d5-kube-api-access-zl4w9" (OuterVolumeSpecName: "kube-api-access-zl4w9") pod "821ea192-dd9a-45c5-94d7-8f7b420b80d5" (UID: "821ea192-dd9a-45c5-94d7-8f7b420b80d5"). InnerVolumeSpecName "kube-api-access-zl4w9". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 30 01:08:57.049833 ip-10-0-142-6 kubenswrapper[2564]: I0730 01:08:57.049750 2564 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zl4w9\" (UniqueName: \"kubernetes.io/projected/821ea192-dd9a-45c5-94d7-8f7b420b80d5-kube-api-access-zl4w9\") on node \"ip-10-0-142-6.ec2.internal\" DevicePath \"\"" Jul 30 01:08:57.139400 ip-10-0-142-6 kubenswrapper[2564]: I0730 01:08:57.139370 2564 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="821ea192-dd9a-45c5-94d7-8f7b420b80d5" path="/var/lib/kubelet/pods/821ea192-dd9a-45c5-94d7-8f7b420b80d5/volumes" Jul 30 01:08:57.636898 ip-10-0-142-6 kubenswrapper[2564]: I0730 01:08:57.636869 2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ipp-a-5912cb-openshift-default-65b755b7cd-x7mnv"] Jul 30 01:08:57.637239 ip-10-0-142-6 kubenswrapper[2564]: I0730 01:08:57.637224 2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="821ea192-dd9a-45c5-94d7-8f7b420b80d5" containerName="test-curl-7574-140267593034752" Jul 30 01:08:57.637326 ip-10-0-142-6 kubenswrapper[2564]: I0730 01:08:57.637240 2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="821ea192-dd9a-45c5-94d7-8f7b420b80d5" containerName="test-curl-7574-140267593034752" Jul 30 01:08:57.637326 ip-10-0-142-6 kubenswrapper[2564]: I0730 01:08:57.637310 2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="821ea192-dd9a-45c5-94d7-8f7b420b80d5" containerName="test-curl-7574-140267593034752" Jul 30 01:08:57.640489 ip-10-0-142-6 kubenswrapper[2564]: I0730 01:08:57.640472 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-5912cb-openshift-default-65b755b7cd-x7mnv" Jul 30 01:08:57.642926 ip-10-0-142-6 kubenswrapper[2564]: I0730 01:08:57.642906 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ipp-a-5912cb-openshift-default-dockercfg-q7l6z\"" Jul 30 01:08:57.651882 ip-10-0-142-6 kubenswrapper[2564]: I0730 01:08:57.651854 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-a-5912cb-openshift-default-65b755b7cd-x7mnv"] Jul 30 01:08:57.653639 ip-10-0-142-6 kubenswrapper[2564]: I0730 01:08:57.653615 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/eb89fc81-7fdb-44f0-ab34-749e1d98c776-istio-token\") pod \"e2e-ipp-a-5912cb-openshift-default-65b755b7cd-x7mnv\" (UID: \"eb89fc81-7fdb-44f0-ab34-749e1d98c776\") " pod="openshift-ingress/e2e-ipp-a-5912cb-openshift-default-65b755b7cd-x7mnv" Jul 30 01:08:57.653760 ip-10-0-142-6 kubenswrapper[2564]: I0730 01:08:57.653646 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/eb89fc81-7fdb-44f0-ab34-749e1d98c776-workload-socket\") pod \"e2e-ipp-a-5912cb-openshift-default-65b755b7cd-x7mnv\" (UID: \"eb89fc81-7fdb-44f0-ab34-749e1d98c776\") " pod="openshift-ingress/e2e-ipp-a-5912cb-openshift-default-65b755b7cd-x7mnv" Jul 30 01:08:57.653760 ip-10-0-142-6 kubenswrapper[2564]: I0730 01:08:57.653669 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/eb89fc81-7fdb-44f0-ab34-749e1d98c776-istiod-ca-cert\") pod \"e2e-ipp-a-5912cb-openshift-default-65b755b7cd-x7mnv\" (UID: \"eb89fc81-7fdb-44f0-ab34-749e1d98c776\") " pod="openshift-ingress/e2e-ipp-a-5912cb-openshift-default-65b755b7cd-x7mnv" Jul 30 01:08:57.653878 ip-10-0-142-6 kubenswrapper[2564]: I0730 01:08:57.653780 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/eb89fc81-7fdb-44f0-ab34-749e1d98c776-istio-envoy\") pod \"e2e-ipp-a-5912cb-openshift-default-65b755b7cd-x7mnv\" (UID: \"eb89fc81-7fdb-44f0-ab34-749e1d98c776\") " pod="openshift-ingress/e2e-ipp-a-5912cb-openshift-default-65b755b7cd-x7mnv" Jul 30 01:08:57.653878 ip-10-0-142-6 kubenswrapper[2564]: I0730 01:08:57.653816 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/eb89fc81-7fdb-44f0-ab34-749e1d98c776-istio-podinfo\") pod \"e2e-ipp-a-5912cb-openshift-default-65b755b7cd-x7mnv\" (UID: \"eb89fc81-7fdb-44f0-ab34-749e1d98c776\") " pod="openshift-ingress/e2e-ipp-a-5912cb-openshift-default-65b755b7cd-x7mnv" Jul 30 01:08:57.653878 ip-10-0-142-6 kubenswrapper[2564]: I0730 01:08:57.653847 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wrln8\" (UniqueName: \"kubernetes.io/projected/eb89fc81-7fdb-44f0-ab34-749e1d98c776-kube-api-access-wrln8\") pod \"e2e-ipp-a-5912cb-openshift-default-65b755b7cd-x7mnv\" (UID: \"eb89fc81-7fdb-44f0-ab34-749e1d98c776\") " pod="openshift-ingress/e2e-ipp-a-5912cb-openshift-default-65b755b7cd-x7mnv" Jul 30 01:08:57.654033 ip-10-0-142-6 kubenswrapper[2564]: I0730 01:08:57.653876 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/eb89fc81-7fdb-44f0-ab34-749e1d98c776-workload-certs\") pod \"e2e-ipp-a-5912cb-openshift-default-65b755b7cd-x7mnv\" (UID: \"eb89fc81-7fdb-44f0-ab34-749e1d98c776\") " pod="openshift-ingress/e2e-ipp-a-5912cb-openshift-default-65b755b7cd-x7mnv" Jul 30 01:08:57.654033 ip-10-0-142-6 kubenswrapper[2564]: I0730 01:08:57.653898 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/eb89fc81-7fdb-44f0-ab34-749e1d98c776-istio-data\") pod \"e2e-ipp-a-5912cb-openshift-default-65b755b7cd-x7mnv\" (UID: \"eb89fc81-7fdb-44f0-ab34-749e1d98c776\") " pod="openshift-ingress/e2e-ipp-a-5912cb-openshift-default-65b755b7cd-x7mnv" Jul 30 01:08:57.654033 ip-10-0-142-6 kubenswrapper[2564]: I0730 01:08:57.654023 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/eb89fc81-7fdb-44f0-ab34-749e1d98c776-credential-socket\") pod \"e2e-ipp-a-5912cb-openshift-default-65b755b7cd-x7mnv\" (UID: \"eb89fc81-7fdb-44f0-ab34-749e1d98c776\") " pod="openshift-ingress/e2e-ipp-a-5912cb-openshift-default-65b755b7cd-x7mnv" Jul 30 01:08:57.714864 ip-10-0-142-6 kubenswrapper[2564]: I0730 01:08:57.714836 2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7574-140267593034752" Jul 30 01:08:57.715728 ip-10-0-142-6 kubenswrapper[2564]: I0730 01:08:57.714861 2564 scope.go:117] "RemoveContainer" containerID="3976e8b810c138e38dc5a8919fd686911af3eba28ef0193c7b9115ec23bf9083" Jul 30 01:08:57.754515 ip-10-0-142-6 kubenswrapper[2564]: I0730 01:08:57.754484 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/eb89fc81-7fdb-44f0-ab34-749e1d98c776-credential-socket\") pod \"e2e-ipp-a-5912cb-openshift-default-65b755b7cd-x7mnv\" (UID: \"eb89fc81-7fdb-44f0-ab34-749e1d98c776\") " pod="openshift-ingress/e2e-ipp-a-5912cb-openshift-default-65b755b7cd-x7mnv" Jul 30 01:08:57.754696 ip-10-0-142-6 kubenswrapper[2564]: I0730 01:08:57.754536 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/eb89fc81-7fdb-44f0-ab34-749e1d98c776-istio-token\") pod \"e2e-ipp-a-5912cb-openshift-default-65b755b7cd-x7mnv\" (UID: \"eb89fc81-7fdb-44f0-ab34-749e1d98c776\") " pod="openshift-ingress/e2e-ipp-a-5912cb-openshift-default-65b755b7cd-x7mnv" Jul 30 01:08:57.754696 ip-10-0-142-6 kubenswrapper[2564]: I0730 01:08:57.754559 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/eb89fc81-7fdb-44f0-ab34-749e1d98c776-workload-socket\") pod \"e2e-ipp-a-5912cb-openshift-default-65b755b7cd-x7mnv\" (UID: \"eb89fc81-7fdb-44f0-ab34-749e1d98c776\") " pod="openshift-ingress/e2e-ipp-a-5912cb-openshift-default-65b755b7cd-x7mnv" Jul 30 01:08:57.754696 ip-10-0-142-6 kubenswrapper[2564]: I0730 01:08:57.754576 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/eb89fc81-7fdb-44f0-ab34-749e1d98c776-istiod-ca-cert\") pod \"e2e-ipp-a-5912cb-openshift-default-65b755b7cd-x7mnv\" (UID: \"eb89fc81-7fdb-44f0-ab34-749e1d98c776\") " pod="openshift-ingress/e2e-ipp-a-5912cb-openshift-default-65b755b7cd-x7mnv" Jul 30 01:08:57.754696 ip-10-0-142-6 kubenswrapper[2564]: I0730 01:08:57.754602 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/eb89fc81-7fdb-44f0-ab34-749e1d98c776-istio-envoy\") pod \"e2e-ipp-a-5912cb-openshift-default-65b755b7cd-x7mnv\" (UID: \"eb89fc81-7fdb-44f0-ab34-749e1d98c776\") " pod="openshift-ingress/e2e-ipp-a-5912cb-openshift-default-65b755b7cd-x7mnv" Jul 30 01:08:57.754922 ip-10-0-142-6 kubenswrapper[2564]: I0730 01:08:57.754729 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/eb89fc81-7fdb-44f0-ab34-749e1d98c776-istio-podinfo\") pod \"e2e-ipp-a-5912cb-openshift-default-65b755b7cd-x7mnv\" (UID: \"eb89fc81-7fdb-44f0-ab34-749e1d98c776\") " pod="openshift-ingress/e2e-ipp-a-5912cb-openshift-default-65b755b7cd-x7mnv" Jul 30 01:08:57.754922 ip-10-0-142-6 kubenswrapper[2564]: I0730 01:08:57.754793 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wrln8\" (UniqueName: \"kubernetes.io/projected/eb89fc81-7fdb-44f0-ab34-749e1d98c776-kube-api-access-wrln8\") pod \"e2e-ipp-a-5912cb-openshift-default-65b755b7cd-x7mnv\" (UID: \"eb89fc81-7fdb-44f0-ab34-749e1d98c776\") " pod="openshift-ingress/e2e-ipp-a-5912cb-openshift-default-65b755b7cd-x7mnv" Jul 30 01:08:57.754922 ip-10-0-142-6 kubenswrapper[2564]: I0730 01:08:57.754824 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/eb89fc81-7fdb-44f0-ab34-749e1d98c776-workload-certs\") pod \"e2e-ipp-a-5912cb-openshift-default-65b755b7cd-x7mnv\" (UID: \"eb89fc81-7fdb-44f0-ab34-749e1d98c776\") " pod="openshift-ingress/e2e-ipp-a-5912cb-openshift-default-65b755b7cd-x7mnv" Jul 30 01:08:57.754922 ip-10-0-142-6 kubenswrapper[2564]: I0730 01:08:57.754861 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/eb89fc81-7fdb-44f0-ab34-749e1d98c776-istio-data\") pod \"e2e-ipp-a-5912cb-openshift-default-65b755b7cd-x7mnv\" (UID: \"eb89fc81-7fdb-44f0-ab34-749e1d98c776\") " pod="openshift-ingress/e2e-ipp-a-5912cb-openshift-default-65b755b7cd-x7mnv" Jul 30 01:08:57.754922 ip-10-0-142-6 kubenswrapper[2564]: I0730 01:08:57.754907 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/eb89fc81-7fdb-44f0-ab34-749e1d98c776-workload-socket\") pod \"e2e-ipp-a-5912cb-openshift-default-65b755b7cd-x7mnv\" (UID: \"eb89fc81-7fdb-44f0-ab34-749e1d98c776\") " pod="openshift-ingress/e2e-ipp-a-5912cb-openshift-default-65b755b7cd-x7mnv" Jul 30 01:08:57.755252 ip-10-0-142-6 kubenswrapper[2564]: I0730 01:08:57.755225 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/eb89fc81-7fdb-44f0-ab34-749e1d98c776-istio-data\") pod \"e2e-ipp-a-5912cb-openshift-default-65b755b7cd-x7mnv\" (UID: \"eb89fc81-7fdb-44f0-ab34-749e1d98c776\") " pod="openshift-ingress/e2e-ipp-a-5912cb-openshift-default-65b755b7cd-x7mnv" Jul 30 01:08:57.755384 ip-10-0-142-6 kubenswrapper[2564]: I0730 01:08:57.755308 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/eb89fc81-7fdb-44f0-ab34-749e1d98c776-workload-certs\") pod \"e2e-ipp-a-5912cb-openshift-default-65b755b7cd-x7mnv\" (UID: \"eb89fc81-7fdb-44f0-ab34-749e1d98c776\") " pod="openshift-ingress/e2e-ipp-a-5912cb-openshift-default-65b755b7cd-x7mnv" Jul 30 01:08:57.755384 ip-10-0-142-6 kubenswrapper[2564]: I0730 01:08:57.755356 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/eb89fc81-7fdb-44f0-ab34-749e1d98c776-istiod-ca-cert\") pod \"e2e-ipp-a-5912cb-openshift-default-65b755b7cd-x7mnv\" (UID: \"eb89fc81-7fdb-44f0-ab34-749e1d98c776\") " pod="openshift-ingress/e2e-ipp-a-5912cb-openshift-default-65b755b7cd-x7mnv" Jul 30 01:08:57.755512 ip-10-0-142-6 kubenswrapper[2564]: I0730 01:08:57.755424 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/eb89fc81-7fdb-44f0-ab34-749e1d98c776-credential-socket\") pod \"e2e-ipp-a-5912cb-openshift-default-65b755b7cd-x7mnv\" (UID: \"eb89fc81-7fdb-44f0-ab34-749e1d98c776\") " pod="openshift-ingress/e2e-ipp-a-5912cb-openshift-default-65b755b7cd-x7mnv" Jul 30 01:08:57.756978 ip-10-0-142-6 kubenswrapper[2564]: I0730 01:08:57.756959 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/eb89fc81-7fdb-44f0-ab34-749e1d98c776-istio-envoy\") pod \"e2e-ipp-a-5912cb-openshift-default-65b755b7cd-x7mnv\" (UID: \"eb89fc81-7fdb-44f0-ab34-749e1d98c776\") " pod="openshift-ingress/e2e-ipp-a-5912cb-openshift-default-65b755b7cd-x7mnv" Jul 30 01:08:57.757293 ip-10-0-142-6 kubenswrapper[2564]: I0730 01:08:57.757277 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/eb89fc81-7fdb-44f0-ab34-749e1d98c776-istio-podinfo\") pod \"e2e-ipp-a-5912cb-openshift-default-65b755b7cd-x7mnv\" (UID: \"eb89fc81-7fdb-44f0-ab34-749e1d98c776\") " pod="openshift-ingress/e2e-ipp-a-5912cb-openshift-default-65b755b7cd-x7mnv" Jul 30 01:08:57.763215 ip-10-0-142-6 kubenswrapper[2564]: I0730 01:08:57.763188 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/eb89fc81-7fdb-44f0-ab34-749e1d98c776-istio-token\") pod \"e2e-ipp-a-5912cb-openshift-default-65b755b7cd-x7mnv\" (UID: \"eb89fc81-7fdb-44f0-ab34-749e1d98c776\") " pod="openshift-ingress/e2e-ipp-a-5912cb-openshift-default-65b755b7cd-x7mnv" Jul 30 01:08:57.763403 ip-10-0-142-6 kubenswrapper[2564]: I0730 01:08:57.763381 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-wrln8\" (UniqueName: \"kubernetes.io/projected/eb89fc81-7fdb-44f0-ab34-749e1d98c776-kube-api-access-wrln8\") pod \"e2e-ipp-a-5912cb-openshift-default-65b755b7cd-x7mnv\" (UID: \"eb89fc81-7fdb-44f0-ab34-749e1d98c776\") " pod="openshift-ingress/e2e-ipp-a-5912cb-openshift-default-65b755b7cd-x7mnv" Jul 30 01:08:57.952372 ip-10-0-142-6 kubenswrapper[2564]: I0730 01:08:57.952283 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-5912cb-openshift-default-65b755b7cd-x7mnv" Jul 30 01:08:58.088445 ip-10-0-142-6 kubenswrapper[2564]: I0730 01:08:58.088415 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-a-5912cb-openshift-default-65b755b7cd-x7mnv"] Jul 30 01:08:58.089962 ip-10-0-142-6 kubenswrapper[2564]: W0730 01:08:58.089938 2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podeb89fc81_7fdb_44f0_ab34_749e1d98c776.slice/crio-115df141208f9488745878d195bf4afb9b780d1a56a767672e17a3396dba6ceb WatchSource:0}: Error finding container 115df141208f9488745878d195bf4afb9b780d1a56a767672e17a3396dba6ceb: Status 404 returned error can't find the container with id 115df141208f9488745878d195bf4afb9b780d1a56a767672e17a3396dba6ceb Jul 30 01:08:58.721238 ip-10-0-142-6 kubenswrapper[2564]: I0730 01:08:58.721196 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-5912cb-openshift-default-65b755b7cd-x7mnv" event={"ID":"eb89fc81-7fdb-44f0-ab34-749e1d98c776","Type":"ContainerStarted","Data":"115df141208f9488745878d195bf4afb9b780d1a56a767672e17a3396dba6ceb"} Jul 30 01:09:01.715305 ip-10-0-142-6 kubenswrapper[2564]: I0730 01:09:01.715267 2564 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236220Ki","pods":"250"} Jul 30 01:09:01.715731 ip-10-0-142-6 kubenswrapper[2564]: I0730 01:09:01.715359 2564 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236220Ki","pods":"250"} Jul 30 01:09:01.715731 ip-10-0-142-6 kubenswrapper[2564]: I0730 01:09:01.715408 2564 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236220Ki","pods":"250"} Jul 30 01:09:02.746776 ip-10-0-142-6 kubenswrapper[2564]: I0730 01:09:02.746737 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-5912cb-openshift-default-65b755b7cd-x7mnv" event={"ID":"eb89fc81-7fdb-44f0-ab34-749e1d98c776","Type":"ContainerStarted","Data":"9e3f6bb3c5ca8e9c1580faecc8be839aad45a6ee0e3edddae72afeb93205223e"} Jul 30 01:09:02.768389 ip-10-0-142-6 kubenswrapper[2564]: I0730 01:09:02.768334 2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-ipp-a-5912cb-openshift-default-65b755b7cd-x7mnv" podStartSLOduration=2.145820675 podStartE2EDuration="5.768315163s" totalImagesPullingTime="3.622494488s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-30 01:08:57 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-30 01:09:02.766942068 +0000 UTC m=+3074.181682002" watchObservedRunningTime="2026-07-30 01:09:02.768315163 +0000 UTC m=+3074.183055096" Jul 30 01:09:02.952812 ip-10-0-142-6 kubenswrapper[2564]: I0730 01:09:02.952776 2564 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-a-5912cb-openshift-default-65b755b7cd-x7mnv" Jul 30 01:09:02.957935 ip-10-0-142-6 kubenswrapper[2564]: I0730 01:09:02.957910 2564 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-a-5912cb-openshift-default-65b755b7cd-x7mnv" Jul 30 01:09:03.752542 ip-10-0-142-6 kubenswrapper[2564]: I0730 01:09:03.752513 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-a-5912cb-openshift-default-65b755b7cd-x7mnv" Jul 30 01:09:03.753712 ip-10-0-142-6 kubenswrapper[2564]: I0730 01:09:03.753687 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-a-5912cb-openshift-default-65b755b7cd-x7mnv" Jul 30 01:09:06.414197 ip-10-0-142-6 kubenswrapper[2564]: I0730 01:09:06.414164 2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-5912cb-6fc86f47bb-zv4lr"] Jul 30 01:09:06.417861 ip-10-0-142-6 kubenswrapper[2564]: I0730 01:09:06.417840 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-5912cb-6fc86f47bb-zv4lr" Jul 30 01:09:06.420555 ip-10-0-142-6 kubenswrapper[2564]: I0730 01:09:06.420534 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-a-5912cb\"" Jul 30 01:09:06.420659 ip-10-0-142-6 kubenswrapper[2564]: I0730 01:09:06.420640 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-a-5912cb-dockercfg-r4qkw\"" Jul 30 01:09:06.425631 ip-10-0-142-6 kubenswrapper[2564]: I0730 01:09:06.425606 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-5912cb-6fc86f47bb-zv4lr"] Jul 30 01:09:06.529524 ip-10-0-142-6 kubenswrapper[2564]: I0730 01:09:06.529494 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5e58f698-a89d-48bd-86fb-673fe182dcfa-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-5912cb-6fc86f47bb-zv4lr\" (UID: \"5e58f698-a89d-48bd-86fb-673fe182dcfa\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-5912cb-6fc86f47bb-zv4lr" Jul 30 01:09:06.529708 ip-10-0-142-6 kubenswrapper[2564]: I0730 01:09:06.529533 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-l95r8\" (UniqueName: \"kubernetes.io/projected/5e58f698-a89d-48bd-86fb-673fe182dcfa-kube-api-access-l95r8\") pod \"payload-processing-e2e-ipp-a-5912cb-6fc86f47bb-zv4lr\" (UID: \"5e58f698-a89d-48bd-86fb-673fe182dcfa\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-5912cb-6fc86f47bb-zv4lr" Jul 30 01:09:06.630567 ip-10-0-142-6 kubenswrapper[2564]: I0730 01:09:06.630534 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5e58f698-a89d-48bd-86fb-673fe182dcfa-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-5912cb-6fc86f47bb-zv4lr\" (UID: \"5e58f698-a89d-48bd-86fb-673fe182dcfa\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-5912cb-6fc86f47bb-zv4lr" Jul 30 01:09:06.630735 ip-10-0-142-6 kubenswrapper[2564]: I0730 01:09:06.630574 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l95r8\" (UniqueName: \"kubernetes.io/projected/5e58f698-a89d-48bd-86fb-673fe182dcfa-kube-api-access-l95r8\") pod \"payload-processing-e2e-ipp-a-5912cb-6fc86f47bb-zv4lr\" (UID: \"5e58f698-a89d-48bd-86fb-673fe182dcfa\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-5912cb-6fc86f47bb-zv4lr" Jul 30 01:09:06.631234 ip-10-0-142-6 kubenswrapper[2564]: I0730 01:09:06.631214 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5e58f698-a89d-48bd-86fb-673fe182dcfa-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-5912cb-6fc86f47bb-zv4lr\" (UID: \"5e58f698-a89d-48bd-86fb-673fe182dcfa\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-5912cb-6fc86f47bb-zv4lr" Jul 30 01:09:06.644166 ip-10-0-142-6 kubenswrapper[2564]: I0730 01:09:06.644139 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-l95r8\" (UniqueName: \"kubernetes.io/projected/5e58f698-a89d-48bd-86fb-673fe182dcfa-kube-api-access-l95r8\") pod \"payload-processing-e2e-ipp-a-5912cb-6fc86f47bb-zv4lr\" (UID: \"5e58f698-a89d-48bd-86fb-673fe182dcfa\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-5912cb-6fc86f47bb-zv4lr" Jul 30 01:09:06.729319 ip-10-0-142-6 kubenswrapper[2564]: I0730 01:09:06.729233 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-5912cb-6fc86f47bb-zv4lr" Jul 30 01:09:06.860340 ip-10-0-142-6 kubenswrapper[2564]: I0730 01:09:06.860309 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-5912cb-6fc86f47bb-zv4lr"] Jul 30 01:09:06.863066 ip-10-0-142-6 kubenswrapper[2564]: W0730 01:09:06.863032 2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5e58f698_a89d_48bd_86fb_673fe182dcfa.slice/crio-07fb428895fefc58e5a9e274dd53617f8bd64b5c9dd56ae9a39d92f8869ad4e7 WatchSource:0}: Error finding container 07fb428895fefc58e5a9e274dd53617f8bd64b5c9dd56ae9a39d92f8869ad4e7: Status 404 returned error can't find the container with id 07fb428895fefc58e5a9e274dd53617f8bd64b5c9dd56ae9a39d92f8869ad4e7 Jul 30 01:09:07.778345 ip-10-0-142-6 kubenswrapper[2564]: I0730 01:09:07.778300 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-5912cb-6fc86f47bb-zv4lr" event={"ID":"5e58f698-a89d-48bd-86fb-673fe182dcfa","Type":"ContainerStarted","Data":"07fb428895fefc58e5a9e274dd53617f8bd64b5c9dd56ae9a39d92f8869ad4e7"} Jul 30 01:09:08.785174 ip-10-0-142-6 kubenswrapper[2564]: I0730 01:09:08.785084 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-5912cb-6fc86f47bb-zv4lr" event={"ID":"5e58f698-a89d-48bd-86fb-673fe182dcfa","Type":"ContainerStarted","Data":"48c7d53afef979fbff86c25312c59523bf69c4c91980dd1d8252e41315b21bee"} Jul 30 01:09:08.785174 ip-10-0-142-6 kubenswrapper[2564]: I0730 01:09:08.785144 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ipp-a-5912cb-6fc86f47bb-zv4lr" Jul 30 01:09:08.803226 ip-10-0-142-6 kubenswrapper[2564]: I0730 01:09:08.803179 2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ipp-a-5912cb-6fc86f47bb-zv4lr" podStartSLOduration=1.148535841 podStartE2EDuration="2.803163212s" totalImagesPullingTime="1.654627371s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-30 01:09:06 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-30 01:09:08.801218962 +0000 UTC m=+3080.215958896" watchObservedRunningTime="2026-07-30 01:09:08.803163212 +0000 UTC m=+3080.217903145" Jul 30 01:09:11.811474 ip-10-0-142-6 kubenswrapper[2564]: I0730 01:09:11.811437 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-nrv2p"] Jul 30 01:09:19.579463 ip-10-0-142-6 kubenswrapper[2564]: I0730 01:09:19.579041 2564 patch_prober.go:28] interesting pod/e2e-ipp-a-5912cb-openshift-default-65b755b7cd-x7mnv container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.52:15021/healthz/ready\": read tcp 10.133.0.2:55158->10.133.0.52:15021: read: connection reset by peer" start-of-body= Jul 30 01:09:19.579463 ip-10-0-142-6 kubenswrapper[2564]: I0730 01:09:19.579138 2564 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-ipp-a-5912cb-openshift-default-65b755b7cd-x7mnv" podUID="eb89fc81-7fdb-44f0-ab34-749e1d98c776" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.52:15021/healthz/ready\": read tcp 10.133.0.2:55158->10.133.0.52:15021: read: connection reset by peer" Jul 30 01:09:19.793259 ip-10-0-142-6 kubenswrapper[2564]: I0730 01:09:19.793182 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ipp-a-5912cb-6fc86f47bb-zv4lr" Jul 30 01:09:19.849207 ip-10-0-142-6 kubenswrapper[2564]: I0730 01:09:19.849111 2564 generic.go:356] "Generic (PLEG): container finished" podID="eb89fc81-7fdb-44f0-ab34-749e1d98c776" containerID="9e3f6bb3c5ca8e9c1580faecc8be839aad45a6ee0e3edddae72afeb93205223e" exitCode=0 Jul 30 01:09:19.849345 ip-10-0-142-6 kubenswrapper[2564]: I0730 01:09:19.849197 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-5912cb-openshift-default-65b755b7cd-x7mnv" event={"ID":"eb89fc81-7fdb-44f0-ab34-749e1d98c776","Type":"ContainerDied","Data":"9e3f6bb3c5ca8e9c1580faecc8be839aad45a6ee0e3edddae72afeb93205223e"} Jul 30 01:09:19.849691 ip-10-0-142-6 kubenswrapper[2564]: I0730 01:09:19.849665 2564 scope.go:117] "RemoveContainer" containerID="9e3f6bb3c5ca8e9c1580faecc8be839aad45a6ee0e3edddae72afeb93205223e" Jul 30 01:09:19.851048 ip-10-0-142-6 kubenswrapper[2564]: I0730 01:09:19.851012 2564 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236220Ki","pods":"250"} Jul 30 01:09:19.851181 ip-10-0-142-6 kubenswrapper[2564]: I0730 01:09:19.851119 2564 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236220Ki","pods":"250"} Jul 30 01:09:19.851181 ip-10-0-142-6 kubenswrapper[2564]: I0730 01:09:19.851159 2564 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236220Ki","pods":"250"} Jul 30 01:09:20.856399 ip-10-0-142-6 kubenswrapper[2564]: I0730 01:09:20.856368 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-5912cb-openshift-default-65b755b7cd-x7mnv" event={"ID":"eb89fc81-7fdb-44f0-ab34-749e1d98c776","Type":"ContainerStarted","Data":"0425ef3ee24cb53bec5eba4df141ab9247fad103eb63d70d6ee89609dbb72ecd"} Jul 30 01:09:20.952999 ip-10-0-142-6 kubenswrapper[2564]: I0730 01:09:20.952962 2564 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-a-5912cb-openshift-default-65b755b7cd-x7mnv" Jul 30 01:09:20.953253 ip-10-0-142-6 kubenswrapper[2564]: I0730 01:09:20.953232 2564 patch_prober.go:28] interesting pod/e2e-ipp-a-5912cb-openshift-default-65b755b7cd-x7mnv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.52:15021/healthz/ready\": dial tcp 10.133.0.52:15021: connect: connection refused" start-of-body= Jul 30 01:09:20.953354 ip-10-0-142-6 kubenswrapper[2564]: I0730 01:09:20.953284 2564 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-5912cb-openshift-default-65b755b7cd-x7mnv" podUID="eb89fc81-7fdb-44f0-ab34-749e1d98c776" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.52:15021/healthz/ready\": dial tcp 10.133.0.52:15021: connect: connection refused" Jul 30 01:09:21.952689 ip-10-0-142-6 kubenswrapper[2564]: I0730 01:09:21.952653 2564 patch_prober.go:28] interesting pod/e2e-ipp-a-5912cb-openshift-default-65b755b7cd-x7mnv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.52:15021/healthz/ready\": dial tcp 10.133.0.52:15021: connect: connection refused" start-of-body= Jul 30 01:09:21.953101 ip-10-0-142-6 kubenswrapper[2564]: I0730 01:09:21.952708 2564 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-5912cb-openshift-default-65b755b7cd-x7mnv" podUID="eb89fc81-7fdb-44f0-ab34-749e1d98c776" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.52:15021/healthz/ready\": dial tcp 10.133.0.52:15021: connect: connection refused" Jul 30 01:09:22.952793 ip-10-0-142-6 kubenswrapper[2564]: I0730 01:09:22.952748 2564 patch_prober.go:28] interesting pod/e2e-ipp-a-5912cb-openshift-default-65b755b7cd-x7mnv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.52:15021/healthz/ready\": dial tcp 10.133.0.52:15021: connect: connection refused" start-of-body= Jul 30 01:09:22.953223 ip-10-0-142-6 kubenswrapper[2564]: I0730 01:09:22.952816 2564 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-5912cb-openshift-default-65b755b7cd-x7mnv" podUID="eb89fc81-7fdb-44f0-ab34-749e1d98c776" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.52:15021/healthz/ready\": dial tcp 10.133.0.52:15021: connect: connection refused" Jul 30 01:09:23.952858 ip-10-0-142-6 kubenswrapper[2564]: I0730 01:09:23.952821 2564 patch_prober.go:28] interesting pod/e2e-ipp-a-5912cb-openshift-default-65b755b7cd-x7mnv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.52:15021/healthz/ready\": dial tcp 10.133.0.52:15021: connect: connection refused" start-of-body= Jul 30 01:09:23.953280 ip-10-0-142-6 kubenswrapper[2564]: I0730 01:09:23.952881 2564 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-5912cb-openshift-default-65b755b7cd-x7mnv" podUID="eb89fc81-7fdb-44f0-ab34-749e1d98c776" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.52:15021/healthz/ready\": dial tcp 10.133.0.52:15021: connect: connection refused" Jul 30 01:09:24.956380 ip-10-0-142-6 kubenswrapper[2564]: I0730 01:09:24.956347 2564 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-a-5912cb-openshift-default-65b755b7cd-x7mnv" Jul 30 01:09:24.956819 ip-10-0-142-6 kubenswrapper[2564]: I0730 01:09:24.956760 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-a-5912cb-openshift-default-65b755b7cd-x7mnv" Jul 30 01:09:24.957403 ip-10-0-142-6 kubenswrapper[2564]: I0730 01:09:24.957383 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-a-5912cb-openshift-default-65b755b7cd-x7mnv" Jul 30 01:09:30.485701 ip-10-0-142-6 kubenswrapper[2564]: I0730 01:09:30.485670 2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ipp-b-47d629-openshift-default-746894d964-wplq8"] Jul 30 01:09:30.488789 ip-10-0-142-6 kubenswrapper[2564]: I0730 01:09:30.488772 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-47d629-openshift-default-746894d964-wplq8" Jul 30 01:09:30.491315 ip-10-0-142-6 kubenswrapper[2564]: I0730 01:09:30.491294 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ipp-b-47d629-openshift-default-dockercfg-xt8t9\"" Jul 30 01:09:30.501155 ip-10-0-142-6 kubenswrapper[2564]: I0730 01:09:30.501132 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-b-47d629-openshift-default-746894d964-wplq8"] Jul 30 01:09:30.657223 ip-10-0-142-6 kubenswrapper[2564]: I0730 01:09:30.657183 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e9cd27ab-1777-4242-afb7-31edf86b4fa0-istio-token\") pod \"e2e-ipp-b-47d629-openshift-default-746894d964-wplq8\" (UID: \"e9cd27ab-1777-4242-afb7-31edf86b4fa0\") " pod="openshift-ingress/e2e-ipp-b-47d629-openshift-default-746894d964-wplq8" Jul 30 01:09:30.657411 ip-10-0-142-6 kubenswrapper[2564]: I0730 01:09:30.657230 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cnbht\" (UniqueName: \"kubernetes.io/projected/e9cd27ab-1777-4242-afb7-31edf86b4fa0-kube-api-access-cnbht\") pod \"e2e-ipp-b-47d629-openshift-default-746894d964-wplq8\" (UID: \"e9cd27ab-1777-4242-afb7-31edf86b4fa0\") " pod="openshift-ingress/e2e-ipp-b-47d629-openshift-default-746894d964-wplq8" Jul 30 01:09:30.657411 ip-10-0-142-6 kubenswrapper[2564]: I0730 01:09:30.657277 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e9cd27ab-1777-4242-afb7-31edf86b4fa0-istio-podinfo\") pod \"e2e-ipp-b-47d629-openshift-default-746894d964-wplq8\" (UID: \"e9cd27ab-1777-4242-afb7-31edf86b4fa0\") " pod="openshift-ingress/e2e-ipp-b-47d629-openshift-default-746894d964-wplq8" Jul 30 01:09:30.657411 ip-10-0-142-6 kubenswrapper[2564]: I0730 01:09:30.657308 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e9cd27ab-1777-4242-afb7-31edf86b4fa0-istiod-ca-cert\") pod \"e2e-ipp-b-47d629-openshift-default-746894d964-wplq8\" (UID: \"e9cd27ab-1777-4242-afb7-31edf86b4fa0\") " pod="openshift-ingress/e2e-ipp-b-47d629-openshift-default-746894d964-wplq8" Jul 30 01:09:30.657411 ip-10-0-142-6 kubenswrapper[2564]: I0730 01:09:30.657333 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e9cd27ab-1777-4242-afb7-31edf86b4fa0-workload-socket\") pod \"e2e-ipp-b-47d629-openshift-default-746894d964-wplq8\" (UID: \"e9cd27ab-1777-4242-afb7-31edf86b4fa0\") " pod="openshift-ingress/e2e-ipp-b-47d629-openshift-default-746894d964-wplq8" Jul 30 01:09:30.657411 ip-10-0-142-6 kubenswrapper[2564]: I0730 01:09:30.657363 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e9cd27ab-1777-4242-afb7-31edf86b4fa0-istio-envoy\") pod \"e2e-ipp-b-47d629-openshift-default-746894d964-wplq8\" (UID: \"e9cd27ab-1777-4242-afb7-31edf86b4fa0\") " pod="openshift-ingress/e2e-ipp-b-47d629-openshift-default-746894d964-wplq8" Jul 30 01:09:30.657411 ip-10-0-142-6 kubenswrapper[2564]: I0730 01:09:30.657383 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e9cd27ab-1777-4242-afb7-31edf86b4fa0-istio-data\") pod \"e2e-ipp-b-47d629-openshift-default-746894d964-wplq8\" (UID: \"e9cd27ab-1777-4242-afb7-31edf86b4fa0\") " pod="openshift-ingress/e2e-ipp-b-47d629-openshift-default-746894d964-wplq8" Jul 30 01:09:30.657622 ip-10-0-142-6 kubenswrapper[2564]: I0730 01:09:30.657417 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e9cd27ab-1777-4242-afb7-31edf86b4fa0-credential-socket\") pod \"e2e-ipp-b-47d629-openshift-default-746894d964-wplq8\" (UID: \"e9cd27ab-1777-4242-afb7-31edf86b4fa0\") " pod="openshift-ingress/e2e-ipp-b-47d629-openshift-default-746894d964-wplq8" Jul 30 01:09:30.657622 ip-10-0-142-6 kubenswrapper[2564]: I0730 01:09:30.657461 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e9cd27ab-1777-4242-afb7-31edf86b4fa0-workload-certs\") pod \"e2e-ipp-b-47d629-openshift-default-746894d964-wplq8\" (UID: \"e9cd27ab-1777-4242-afb7-31edf86b4fa0\") " pod="openshift-ingress/e2e-ipp-b-47d629-openshift-default-746894d964-wplq8" Jul 30 01:09:30.758492 ip-10-0-142-6 kubenswrapper[2564]: I0730 01:09:30.758402 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e9cd27ab-1777-4242-afb7-31edf86b4fa0-istio-podinfo\") pod \"e2e-ipp-b-47d629-openshift-default-746894d964-wplq8\" (UID: \"e9cd27ab-1777-4242-afb7-31edf86b4fa0\") " pod="openshift-ingress/e2e-ipp-b-47d629-openshift-default-746894d964-wplq8" Jul 30 01:09:30.758492 ip-10-0-142-6 kubenswrapper[2564]: I0730 01:09:30.758440 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e9cd27ab-1777-4242-afb7-31edf86b4fa0-istiod-ca-cert\") pod \"e2e-ipp-b-47d629-openshift-default-746894d964-wplq8\" (UID: \"e9cd27ab-1777-4242-afb7-31edf86b4fa0\") " pod="openshift-ingress/e2e-ipp-b-47d629-openshift-default-746894d964-wplq8" Jul 30 01:09:30.758492 ip-10-0-142-6 kubenswrapper[2564]: I0730 01:09:30.758467 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e9cd27ab-1777-4242-afb7-31edf86b4fa0-workload-socket\") pod \"e2e-ipp-b-47d629-openshift-default-746894d964-wplq8\" (UID: \"e9cd27ab-1777-4242-afb7-31edf86b4fa0\") " pod="openshift-ingress/e2e-ipp-b-47d629-openshift-default-746894d964-wplq8" Jul 30 01:09:30.758492 ip-10-0-142-6 kubenswrapper[2564]: I0730 01:09:30.758486 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e9cd27ab-1777-4242-afb7-31edf86b4fa0-istio-envoy\") pod \"e2e-ipp-b-47d629-openshift-default-746894d964-wplq8\" (UID: \"e9cd27ab-1777-4242-afb7-31edf86b4fa0\") " pod="openshift-ingress/e2e-ipp-b-47d629-openshift-default-746894d964-wplq8" Jul 30 01:09:30.758839 ip-10-0-142-6 kubenswrapper[2564]: I0730 01:09:30.758502 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e9cd27ab-1777-4242-afb7-31edf86b4fa0-istio-data\") pod \"e2e-ipp-b-47d629-openshift-default-746894d964-wplq8\" (UID: \"e9cd27ab-1777-4242-afb7-31edf86b4fa0\") " pod="openshift-ingress/e2e-ipp-b-47d629-openshift-default-746894d964-wplq8" Jul 30 01:09:30.758839 ip-10-0-142-6 kubenswrapper[2564]: I0730 01:09:30.758641 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e9cd27ab-1777-4242-afb7-31edf86b4fa0-credential-socket\") pod \"e2e-ipp-b-47d629-openshift-default-746894d964-wplq8\" (UID: \"e9cd27ab-1777-4242-afb7-31edf86b4fa0\") " pod="openshift-ingress/e2e-ipp-b-47d629-openshift-default-746894d964-wplq8" Jul 30 01:09:30.758839 ip-10-0-142-6 kubenswrapper[2564]: I0730 01:09:30.758684 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e9cd27ab-1777-4242-afb7-31edf86b4fa0-workload-certs\") pod \"e2e-ipp-b-47d629-openshift-default-746894d964-wplq8\" (UID: \"e9cd27ab-1777-4242-afb7-31edf86b4fa0\") " pod="openshift-ingress/e2e-ipp-b-47d629-openshift-default-746894d964-wplq8" Jul 30 01:09:30.758839 ip-10-0-142-6 kubenswrapper[2564]: I0730 01:09:30.758784 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e9cd27ab-1777-4242-afb7-31edf86b4fa0-istio-token\") pod \"e2e-ipp-b-47d629-openshift-default-746894d964-wplq8\" (UID: \"e9cd27ab-1777-4242-afb7-31edf86b4fa0\") " pod="openshift-ingress/e2e-ipp-b-47d629-openshift-default-746894d964-wplq8" Jul 30 01:09:30.758839 ip-10-0-142-6 kubenswrapper[2564]: I0730 01:09:30.758826 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cnbht\" (UniqueName: \"kubernetes.io/projected/e9cd27ab-1777-4242-afb7-31edf86b4fa0-kube-api-access-cnbht\") pod \"e2e-ipp-b-47d629-openshift-default-746894d964-wplq8\" (UID: \"e9cd27ab-1777-4242-afb7-31edf86b4fa0\") " pod="openshift-ingress/e2e-ipp-b-47d629-openshift-default-746894d964-wplq8" Jul 30 01:09:30.759127 ip-10-0-142-6 kubenswrapper[2564]: I0730 01:09:30.758886 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e9cd27ab-1777-4242-afb7-31edf86b4fa0-istio-data\") pod \"e2e-ipp-b-47d629-openshift-default-746894d964-wplq8\" (UID: \"e9cd27ab-1777-4242-afb7-31edf86b4fa0\") " pod="openshift-ingress/e2e-ipp-b-47d629-openshift-default-746894d964-wplq8" Jul 30 01:09:30.759195 ip-10-0-142-6 kubenswrapper[2564]: I0730 01:09:30.759145 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e9cd27ab-1777-4242-afb7-31edf86b4fa0-credential-socket\") pod \"e2e-ipp-b-47d629-openshift-default-746894d964-wplq8\" (UID: \"e9cd27ab-1777-4242-afb7-31edf86b4fa0\") " pod="openshift-ingress/e2e-ipp-b-47d629-openshift-default-746894d964-wplq8" Jul 30 01:09:30.759252 ip-10-0-142-6 kubenswrapper[2564]: I0730 01:09:30.759216 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e9cd27ab-1777-4242-afb7-31edf86b4fa0-workload-socket\") pod \"e2e-ipp-b-47d629-openshift-default-746894d964-wplq8\" (UID: \"e9cd27ab-1777-4242-afb7-31edf86b4fa0\") " pod="openshift-ingress/e2e-ipp-b-47d629-openshift-default-746894d964-wplq8" Jul 30 01:09:30.759309 ip-10-0-142-6 kubenswrapper[2564]: I0730 01:09:30.759262 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e9cd27ab-1777-4242-afb7-31edf86b4fa0-workload-certs\") pod \"e2e-ipp-b-47d629-openshift-default-746894d964-wplq8\" (UID: \"e9cd27ab-1777-4242-afb7-31edf86b4fa0\") " pod="openshift-ingress/e2e-ipp-b-47d629-openshift-default-746894d964-wplq8" Jul 30 01:09:30.759309 ip-10-0-142-6 kubenswrapper[2564]: I0730 01:09:30.759297 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e9cd27ab-1777-4242-afb7-31edf86b4fa0-istiod-ca-cert\") pod \"e2e-ipp-b-47d629-openshift-default-746894d964-wplq8\" (UID: \"e9cd27ab-1777-4242-afb7-31edf86b4fa0\") " pod="openshift-ingress/e2e-ipp-b-47d629-openshift-default-746894d964-wplq8" Jul 30 01:09:30.760974 ip-10-0-142-6 kubenswrapper[2564]: I0730 01:09:30.760946 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e9cd27ab-1777-4242-afb7-31edf86b4fa0-istio-envoy\") pod \"e2e-ipp-b-47d629-openshift-default-746894d964-wplq8\" (UID: \"e9cd27ab-1777-4242-afb7-31edf86b4fa0\") " pod="openshift-ingress/e2e-ipp-b-47d629-openshift-default-746894d964-wplq8" Jul 30 01:09:30.761145 ip-10-0-142-6 kubenswrapper[2564]: I0730 01:09:30.761127 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e9cd27ab-1777-4242-afb7-31edf86b4fa0-istio-podinfo\") pod \"e2e-ipp-b-47d629-openshift-default-746894d964-wplq8\" (UID: \"e9cd27ab-1777-4242-afb7-31edf86b4fa0\") " pod="openshift-ingress/e2e-ipp-b-47d629-openshift-default-746894d964-wplq8" Jul 30 01:09:30.769431 ip-10-0-142-6 kubenswrapper[2564]: I0730 01:09:30.769404 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-cnbht\" (UniqueName: \"kubernetes.io/projected/e9cd27ab-1777-4242-afb7-31edf86b4fa0-kube-api-access-cnbht\") pod \"e2e-ipp-b-47d629-openshift-default-746894d964-wplq8\" (UID: \"e9cd27ab-1777-4242-afb7-31edf86b4fa0\") " pod="openshift-ingress/e2e-ipp-b-47d629-openshift-default-746894d964-wplq8" Jul 30 01:09:30.769580 ip-10-0-142-6 kubenswrapper[2564]: I0730 01:09:30.769563 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e9cd27ab-1777-4242-afb7-31edf86b4fa0-istio-token\") pod \"e2e-ipp-b-47d629-openshift-default-746894d964-wplq8\" (UID: \"e9cd27ab-1777-4242-afb7-31edf86b4fa0\") " pod="openshift-ingress/e2e-ipp-b-47d629-openshift-default-746894d964-wplq8" Jul 30 01:09:30.799715 ip-10-0-142-6 kubenswrapper[2564]: I0730 01:09:30.799688 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-47d629-openshift-default-746894d964-wplq8" Jul 30 01:09:30.939892 ip-10-0-142-6 kubenswrapper[2564]: I0730 01:09:30.939868 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-b-47d629-openshift-default-746894d964-wplq8"] Jul 30 01:09:30.940934 ip-10-0-142-6 kubenswrapper[2564]: W0730 01:09:30.940903 2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode9cd27ab_1777_4242_afb7_31edf86b4fa0.slice/crio-6c321708ea15bbc1d160b068d883038a94fbcf422625ee80b4fc9289aa38b5dc WatchSource:0}: Error finding container 6c321708ea15bbc1d160b068d883038a94fbcf422625ee80b4fc9289aa38b5dc: Status 404 returned error can't find the container with id 6c321708ea15bbc1d160b068d883038a94fbcf422625ee80b4fc9289aa38b5dc Jul 30 01:09:30.943095 ip-10-0-142-6 kubenswrapper[2564]: I0730 01:09:30.943049 2564 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236220Ki","pods":"250"} Jul 30 01:09:30.943158 ip-10-0-142-6 kubenswrapper[2564]: I0730 01:09:30.943126 2564 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236220Ki","pods":"250"} Jul 30 01:09:30.943158 ip-10-0-142-6 kubenswrapper[2564]: I0730 01:09:30.943155 2564 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236220Ki","pods":"250"} Jul 30 01:09:31.917386 ip-10-0-142-6 kubenswrapper[2564]: I0730 01:09:31.917347 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-47d629-openshift-default-746894d964-wplq8" event={"ID":"e9cd27ab-1777-4242-afb7-31edf86b4fa0","Type":"ContainerStarted","Data":"0da6b45169c01b8a0b7d8dd44ec273acfc576c271e02e4ad1ea3177125f86007"} Jul 30 01:09:31.917787 ip-10-0-142-6 kubenswrapper[2564]: I0730 01:09:31.917416 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-47d629-openshift-default-746894d964-wplq8" event={"ID":"e9cd27ab-1777-4242-afb7-31edf86b4fa0","Type":"ContainerStarted","Data":"6c321708ea15bbc1d160b068d883038a94fbcf422625ee80b4fc9289aa38b5dc"} Jul 30 01:09:31.938486 ip-10-0-142-6 kubenswrapper[2564]: I0730 01:09:31.938436 2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-ipp-b-47d629-openshift-default-746894d964-wplq8" podStartSLOduration=1.938419248 podStartE2EDuration="1.938419248s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-30 01:09:30 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-30 01:09:31.936487454 +0000 UTC m=+3103.351227388" watchObservedRunningTime="2026-07-30 01:09:31.938419248 +0000 UTC m=+3103.353159180" Jul 30 01:09:32.800406 ip-10-0-142-6 kubenswrapper[2564]: I0730 01:09:32.800374 2564 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-b-47d629-openshift-default-746894d964-wplq8" Jul 30 01:09:32.805753 ip-10-0-142-6 kubenswrapper[2564]: I0730 01:09:32.805729 2564 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-b-47d629-openshift-default-746894d964-wplq8" Jul 30 01:09:32.922297 ip-10-0-142-6 kubenswrapper[2564]: I0730 01:09:32.922271 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-b-47d629-openshift-default-746894d964-wplq8" Jul 30 01:09:32.923148 ip-10-0-142-6 kubenswrapper[2564]: I0730 01:09:32.923132 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-b-47d629-openshift-default-746894d964-wplq8" Jul 30 01:09:36.309106 ip-10-0-142-6 kubenswrapper[2564]: I0730 01:09:36.309051 2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-47d629-76d8595b56-sf2ms"] Jul 30 01:09:36.312630 ip-10-0-142-6 kubenswrapper[2564]: I0730 01:09:36.312612 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-47d629-76d8595b56-sf2ms" Jul 30 01:09:36.315136 ip-10-0-142-6 kubenswrapper[2564]: I0730 01:09:36.315115 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-b-47d629\"" Jul 30 01:09:36.315244 ip-10-0-142-6 kubenswrapper[2564]: I0730 01:09:36.315222 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-b-47d629-dockercfg-btc5h\"" Jul 30 01:09:36.321524 ip-10-0-142-6 kubenswrapper[2564]: I0730 01:09:36.321495 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-47d629-76d8595b56-sf2ms"] Jul 30 01:09:36.343046 ip-10-0-142-6 kubenswrapper[2564]: I0730 01:09:36.343017 2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-47d629-58fcfd8d64-drzlv"] Jul 30 01:09:36.346525 ip-10-0-142-6 kubenswrapper[2564]: I0730 01:09:36.346503 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-47d629-58fcfd8d64-drzlv" Jul 30 01:09:36.356479 ip-10-0-142-6 kubenswrapper[2564]: I0730 01:09:36.356452 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-47d629-58fcfd8d64-drzlv"] Jul 30 01:09:36.414411 ip-10-0-142-6 kubenswrapper[2564]: I0730 01:09:36.414374 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2jzjr\" (UniqueName: \"kubernetes.io/projected/1235e5ac-6ee9-4ae3-b5ca-fa824edb6227-kube-api-access-2jzjr\") pod \"payload-pre-processing-e2e-ipp-b-47d629-76d8595b56-sf2ms\" (UID: \"1235e5ac-6ee9-4ae3-b5ca-fa824edb6227\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-47d629-76d8595b56-sf2ms" Jul 30 01:09:36.414588 ip-10-0-142-6 kubenswrapper[2564]: I0730 01:09:36.414454 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1235e5ac-6ee9-4ae3-b5ca-fa824edb6227-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-47d629-76d8595b56-sf2ms\" (UID: \"1235e5ac-6ee9-4ae3-b5ca-fa824edb6227\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-47d629-76d8595b56-sf2ms" Jul 30 01:09:36.515526 ip-10-0-142-6 kubenswrapper[2564]: I0730 01:09:36.515491 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1235e5ac-6ee9-4ae3-b5ca-fa824edb6227-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-47d629-76d8595b56-sf2ms\" (UID: \"1235e5ac-6ee9-4ae3-b5ca-fa824edb6227\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-47d629-76d8595b56-sf2ms" Jul 30 01:09:36.515526 ip-10-0-142-6 kubenswrapper[2564]: I0730 01:09:36.515528 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/65fecc99-42fa-4ab1-8215-d800172c1b0d-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-47d629-58fcfd8d64-drzlv\" (UID: \"65fecc99-42fa-4ab1-8215-d800172c1b0d\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-47d629-58fcfd8d64-drzlv" Jul 30 01:09:36.515762 ip-10-0-142-6 kubenswrapper[2564]: I0730 01:09:36.515607 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2jzjr\" (UniqueName: \"kubernetes.io/projected/1235e5ac-6ee9-4ae3-b5ca-fa824edb6227-kube-api-access-2jzjr\") pod \"payload-pre-processing-e2e-ipp-b-47d629-76d8595b56-sf2ms\" (UID: \"1235e5ac-6ee9-4ae3-b5ca-fa824edb6227\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-47d629-76d8595b56-sf2ms" Jul 30 01:09:36.515762 ip-10-0-142-6 kubenswrapper[2564]: I0730 01:09:36.515626 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6whh6\" (UniqueName: \"kubernetes.io/projected/65fecc99-42fa-4ab1-8215-d800172c1b0d-kube-api-access-6whh6\") pod \"payload-processing-e2e-ipp-b-47d629-58fcfd8d64-drzlv\" (UID: \"65fecc99-42fa-4ab1-8215-d800172c1b0d\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-47d629-58fcfd8d64-drzlv" Jul 30 01:09:36.516249 ip-10-0-142-6 kubenswrapper[2564]: I0730 01:09:36.516229 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1235e5ac-6ee9-4ae3-b5ca-fa824edb6227-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-47d629-76d8595b56-sf2ms\" (UID: \"1235e5ac-6ee9-4ae3-b5ca-fa824edb6227\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-47d629-76d8595b56-sf2ms" Jul 30 01:09:36.523443 ip-10-0-142-6 kubenswrapper[2564]: I0730 01:09:36.523414 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2jzjr\" (UniqueName: \"kubernetes.io/projected/1235e5ac-6ee9-4ae3-b5ca-fa824edb6227-kube-api-access-2jzjr\") pod \"payload-pre-processing-e2e-ipp-b-47d629-76d8595b56-sf2ms\" (UID: \"1235e5ac-6ee9-4ae3-b5ca-fa824edb6227\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-47d629-76d8595b56-sf2ms" Jul 30 01:09:36.616311 ip-10-0-142-6 kubenswrapper[2564]: I0730 01:09:36.616220 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6whh6\" (UniqueName: \"kubernetes.io/projected/65fecc99-42fa-4ab1-8215-d800172c1b0d-kube-api-access-6whh6\") pod \"payload-processing-e2e-ipp-b-47d629-58fcfd8d64-drzlv\" (UID: \"65fecc99-42fa-4ab1-8215-d800172c1b0d\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-47d629-58fcfd8d64-drzlv" Jul 30 01:09:36.616311 ip-10-0-142-6 kubenswrapper[2564]: I0730 01:09:36.616280 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/65fecc99-42fa-4ab1-8215-d800172c1b0d-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-47d629-58fcfd8d64-drzlv\" (UID: \"65fecc99-42fa-4ab1-8215-d800172c1b0d\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-47d629-58fcfd8d64-drzlv" Jul 30 01:09:36.617094 ip-10-0-142-6 kubenswrapper[2564]: I0730 01:09:36.617051 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/65fecc99-42fa-4ab1-8215-d800172c1b0d-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-47d629-58fcfd8d64-drzlv\" (UID: \"65fecc99-42fa-4ab1-8215-d800172c1b0d\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-47d629-58fcfd8d64-drzlv" Jul 30 01:09:36.624561 ip-10-0-142-6 kubenswrapper[2564]: I0730 01:09:36.624530 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6whh6\" (UniqueName: \"kubernetes.io/projected/65fecc99-42fa-4ab1-8215-d800172c1b0d-kube-api-access-6whh6\") pod \"payload-processing-e2e-ipp-b-47d629-58fcfd8d64-drzlv\" (UID: \"65fecc99-42fa-4ab1-8215-d800172c1b0d\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-47d629-58fcfd8d64-drzlv" Jul 30 01:09:36.624673 ip-10-0-142-6 kubenswrapper[2564]: I0730 01:09:36.624657 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-47d629-76d8595b56-sf2ms" Jul 30 01:09:36.659518 ip-10-0-142-6 kubenswrapper[2564]: I0730 01:09:36.659487 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-47d629-58fcfd8d64-drzlv" Jul 30 01:09:36.773458 ip-10-0-142-6 kubenswrapper[2564]: I0730 01:09:36.773432 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-47d629-76d8595b56-sf2ms"] Jul 30 01:09:36.775919 ip-10-0-142-6 kubenswrapper[2564]: W0730 01:09:36.775884 2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1235e5ac_6ee9_4ae3_b5ca_fa824edb6227.slice/crio-680838ab5b29067c7bc6f450bd2d865c49b7ea2f49a7b771883a7460917b86f7 WatchSource:0}: Error finding container 680838ab5b29067c7bc6f450bd2d865c49b7ea2f49a7b771883a7460917b86f7: Status 404 returned error can't find the container with id 680838ab5b29067c7bc6f450bd2d865c49b7ea2f49a7b771883a7460917b86f7 Jul 30 01:09:36.807289 ip-10-0-142-6 kubenswrapper[2564]: I0730 01:09:36.807265 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-47d629-58fcfd8d64-drzlv"] Jul 30 01:09:36.808823 ip-10-0-142-6 kubenswrapper[2564]: W0730 01:09:36.808799 2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod65fecc99_42fa_4ab1_8215_d800172c1b0d.slice/crio-4273001edabcc9b09b4b0f099aa717589fc6e9af0fbc65be39a4c908e1fe0347 WatchSource:0}: Error finding container 4273001edabcc9b09b4b0f099aa717589fc6e9af0fbc65be39a4c908e1fe0347: Status 404 returned error can't find the container with id 4273001edabcc9b09b4b0f099aa717589fc6e9af0fbc65be39a4c908e1fe0347 Jul 30 01:09:36.945791 ip-10-0-142-6 kubenswrapper[2564]: I0730 01:09:36.945760 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-47d629-76d8595b56-sf2ms" event={"ID":"1235e5ac-6ee9-4ae3-b5ca-fa824edb6227","Type":"ContainerStarted","Data":"0ad0373533c170d1b3927f2d401f45e8f0525c8678eb823b8d4f0a43bd7e5b8c"} Jul 30 01:09:36.945987 ip-10-0-142-6 kubenswrapper[2564]: I0730 01:09:36.945816 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-47d629-76d8595b56-sf2ms" event={"ID":"1235e5ac-6ee9-4ae3-b5ca-fa824edb6227","Type":"ContainerStarted","Data":"680838ab5b29067c7bc6f450bd2d865c49b7ea2f49a7b771883a7460917b86f7"} Jul 30 01:09:36.945987 ip-10-0-142-6 kubenswrapper[2564]: I0730 01:09:36.945843 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-47d629-76d8595b56-sf2ms" Jul 30 01:09:36.947596 ip-10-0-142-6 kubenswrapper[2564]: I0730 01:09:36.947569 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-47d629-58fcfd8d64-drzlv" event={"ID":"65fecc99-42fa-4ab1-8215-d800172c1b0d","Type":"ContainerStarted","Data":"9cc66e3954463bba33cc2a02c6713fe917b7cc6aaf4433eec5facb4a402e0966"} Jul 30 01:09:36.947733 ip-10-0-142-6 kubenswrapper[2564]: I0730 01:09:36.947615 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-47d629-58fcfd8d64-drzlv" event={"ID":"65fecc99-42fa-4ab1-8215-d800172c1b0d","Type":"ContainerStarted","Data":"4273001edabcc9b09b4b0f099aa717589fc6e9af0fbc65be39a4c908e1fe0347"} Jul 30 01:09:36.947733 ip-10-0-142-6 kubenswrapper[2564]: I0730 01:09:36.947647 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ipp-b-47d629-58fcfd8d64-drzlv" Jul 30 01:09:36.962454 ip-10-0-142-6 kubenswrapper[2564]: I0730 01:09:36.962404 2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-47d629-76d8595b56-sf2ms" podStartSLOduration=0.962385624 podStartE2EDuration="962.385624ms" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-30 01:09:36 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-30 01:09:36.961640315 +0000 UTC m=+3108.376380260" watchObservedRunningTime="2026-07-30 01:09:36.962385624 +0000 UTC m=+3108.377125557" Jul 30 01:09:36.976254 ip-10-0-142-6 kubenswrapper[2564]: I0730 01:09:36.976205 2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ipp-b-47d629-58fcfd8d64-drzlv" podStartSLOduration=0.976185111 podStartE2EDuration="976.185111ms" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-30 01:09:36 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-30 01:09:36.975410532 +0000 UTC m=+3108.390150467" watchObservedRunningTime="2026-07-30 01:09:36.976185111 +0000 UTC m=+3108.390925045" Jul 30 01:09:43.513630 ip-10-0-142-6 kubenswrapper[2564]: I0730 01:09:43.513591 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-nrv2p"] Jul 30 01:09:47.955739 ip-10-0-142-6 kubenswrapper[2564]: I0730 01:09:47.955698 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ipp-b-47d629-58fcfd8d64-drzlv" Jul 30 01:09:47.956130 ip-10-0-142-6 kubenswrapper[2564]: I0730 01:09:47.955777 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-47d629-76d8595b56-sf2ms" Jul 30 01:09:52.036831 ip-10-0-142-6 kubenswrapper[2564]: I0730 01:09:52.036797 2564 generic.go:356] "Generic (PLEG): container finished" podID="e9cd27ab-1777-4242-afb7-31edf86b4fa0" containerID="0da6b45169c01b8a0b7d8dd44ec273acfc576c271e02e4ad1ea3177125f86007" exitCode=0 Jul 30 01:09:52.037297 ip-10-0-142-6 kubenswrapper[2564]: I0730 01:09:52.036874 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-47d629-openshift-default-746894d964-wplq8" event={"ID":"e9cd27ab-1777-4242-afb7-31edf86b4fa0","Type":"ContainerDied","Data":"0da6b45169c01b8a0b7d8dd44ec273acfc576c271e02e4ad1ea3177125f86007"} Jul 30 01:09:52.037344 ip-10-0-142-6 kubenswrapper[2564]: I0730 01:09:52.037321 2564 scope.go:117] "RemoveContainer" containerID="0da6b45169c01b8a0b7d8dd44ec273acfc576c271e02e4ad1ea3177125f86007" Jul 30 01:09:52.038489 ip-10-0-142-6 kubenswrapper[2564]: I0730 01:09:52.038459 2564 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236220Ki","pods":"250"} Jul 30 01:09:52.038568 ip-10-0-142-6 kubenswrapper[2564]: I0730 01:09:52.038540 2564 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236220Ki","pods":"250"} Jul 30 01:09:52.038606 ip-10-0-142-6 kubenswrapper[2564]: I0730 01:09:52.038575 2564 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236220Ki","pods":"250"} Jul 30 01:09:53.047012 ip-10-0-142-6 kubenswrapper[2564]: I0730 01:09:53.046975 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-47d629-openshift-default-746894d964-wplq8" event={"ID":"e9cd27ab-1777-4242-afb7-31edf86b4fa0","Type":"ContainerStarted","Data":"7b4e5fc7168ca87e194e9db93217d936e26992cfebc0327c2993ffca9c493d9c"} Jul 30 01:09:53.799975 ip-10-0-142-6 kubenswrapper[2564]: I0730 01:09:53.799947 2564 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-b-47d629-openshift-default-746894d964-wplq8" Jul 30 01:09:53.800258 ip-10-0-142-6 kubenswrapper[2564]: I0730 01:09:53.800236 2564 patch_prober.go:28] interesting pod/e2e-ipp-b-47d629-openshift-default-746894d964-wplq8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.54:15021/healthz/ready\": dial tcp 10.133.0.54:15021: connect: connection refused" start-of-body= Jul 30 01:09:53.800318 ip-10-0-142-6 kubenswrapper[2564]: I0730 01:09:53.800299 2564 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-47d629-openshift-default-746894d964-wplq8" podUID="e9cd27ab-1777-4242-afb7-31edf86b4fa0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.54:15021/healthz/ready\": dial tcp 10.133.0.54:15021: connect: connection refused" Jul 30 01:09:54.800485 ip-10-0-142-6 kubenswrapper[2564]: I0730 01:09:54.800452 2564 patch_prober.go:28] interesting pod/e2e-ipp-b-47d629-openshift-default-746894d964-wplq8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.54:15021/healthz/ready\": dial tcp 10.133.0.54:15021: connect: connection refused" start-of-body= Jul 30 01:09:54.800874 ip-10-0-142-6 kubenswrapper[2564]: I0730 01:09:54.800505 2564 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-47d629-openshift-default-746894d964-wplq8" podUID="e9cd27ab-1777-4242-afb7-31edf86b4fa0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.54:15021/healthz/ready\": dial tcp 10.133.0.54:15021: connect: connection refused" Jul 30 01:09:55.435498 ip-10-0-142-6 kubenswrapper[2564]: I0730 01:09:55.435455 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-e2e-ipp-b-47d629-58fcfd8d64-drzlv_65fecc99-42fa-4ab1-8215-d800172c1b0d/payload-processing/0.log" Jul 30 01:09:55.800540 ip-10-0-142-6 kubenswrapper[2564]: I0730 01:09:55.800452 2564 patch_prober.go:28] interesting pod/e2e-ipp-b-47d629-openshift-default-746894d964-wplq8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.54:15021/healthz/ready\": dial tcp 10.133.0.54:15021: connect: connection refused" start-of-body= Jul 30 01:09:55.800540 ip-10-0-142-6 kubenswrapper[2564]: I0730 01:09:55.800510 2564 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-47d629-openshift-default-746894d964-wplq8" podUID="e9cd27ab-1777-4242-afb7-31edf86b4fa0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.54:15021/healthz/ready\": dial tcp 10.133.0.54:15021: connect: connection refused" Jul 30 01:09:56.800290 ip-10-0-142-6 kubenswrapper[2564]: I0730 01:09:56.800247 2564 patch_prober.go:28] interesting pod/e2e-ipp-b-47d629-openshift-default-746894d964-wplq8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.54:15021/healthz/ready\": dial tcp 10.133.0.54:15021: connect: connection refused" start-of-body= Jul 30 01:09:56.800495 ip-10-0-142-6 kubenswrapper[2564]: I0730 01:09:56.800323 2564 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-47d629-openshift-default-746894d964-wplq8" podUID="e9cd27ab-1777-4242-afb7-31edf86b4fa0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.54:15021/healthz/ready\": dial tcp 10.133.0.54:15021: connect: connection refused" Jul 30 01:09:57.804031 ip-10-0-142-6 kubenswrapper[2564]: I0730 01:09:57.804002 2564 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-b-47d629-openshift-default-746894d964-wplq8" Jul 30 01:09:57.804511 ip-10-0-142-6 kubenswrapper[2564]: I0730 01:09:57.804432 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-b-47d629-openshift-default-746894d964-wplq8" Jul 30 01:09:57.805138 ip-10-0-142-6 kubenswrapper[2564]: I0730 01:09:57.805118 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-b-47d629-openshift-default-746894d964-wplq8" Jul 30 01:10:07.813519 ip-10-0-142-6 kubenswrapper[2564]: I0730 01:10:07.813484 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-nrv2p"] Jul 30 01:10:45.524752 ip-10-0-142-6 kubenswrapper[2564]: I0730 01:10:45.524672 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-nrv2p"] Jul 30 01:10:55.365371 ip-10-0-142-6 kubenswrapper[2564]: I0730 01:10:55.365339 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-e2e-ipp-a-5912cb-6fc86f47bb-zv4lr_5e58f698-a89d-48bd-86fb-673fe182dcfa/payload-processing/0.log" Jul 30 01:10:55.590099 ip-10-0-142-6 kubenswrapper[2564]: I0730 01:10:55.590047 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-e2e-ipp-b-47d629-58fcfd8d64-drzlv_65fecc99-42fa-4ab1-8215-d800172c1b0d/payload-processing/0.log" Jul 30 01:10:56.116593 ip-10-0-142-6 kubenswrapper[2564]: I0730 01:10:56.116560 2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ipp-cleanup-ff67f5-openshift-default-69dfcf4d7b-64kkj"] Jul 30 01:10:56.123637 ip-10-0-142-6 kubenswrapper[2564]: I0730 01:10:56.123611 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-cleanup-ff67f5-openshift-default-69dfcf4d7b-64kkj" Jul 30 01:10:56.127276 ip-10-0-142-6 kubenswrapper[2564]: I0730 01:10:56.127247 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ipp-cleanup-ff67f5-openshift-default-dockercfg-92dw4\"" Jul 30 01:10:56.134632 ip-10-0-142-6 kubenswrapper[2564]: I0730 01:10:56.134600 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-cleanup-ff67f5-openshift-default-69dfcf4d7b-64kkj"] Jul 30 01:10:56.236247 ip-10-0-142-6 kubenswrapper[2564]: I0730 01:10:56.236214 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/754f6d7c-9685-42de-b0df-bb2d658a6903-istio-token\") pod \"e2e-ipp-cleanup-ff67f5-openshift-default-69dfcf4d7b-64kkj\" (UID: \"754f6d7c-9685-42de-b0df-bb2d658a6903\") " pod="openshift-ingress/e2e-ipp-cleanup-ff67f5-openshift-default-69dfcf4d7b-64kkj" Jul 30 01:10:56.236455 ip-10-0-142-6 kubenswrapper[2564]: I0730 01:10:56.236336 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/754f6d7c-9685-42de-b0df-bb2d658a6903-istio-data\") pod \"e2e-ipp-cleanup-ff67f5-openshift-default-69dfcf4d7b-64kkj\" (UID: \"754f6d7c-9685-42de-b0df-bb2d658a6903\") " pod="openshift-ingress/e2e-ipp-cleanup-ff67f5-openshift-default-69dfcf4d7b-64kkj" Jul 30 01:10:56.236455 ip-10-0-142-6 kubenswrapper[2564]: I0730 01:10:56.236362 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/754f6d7c-9685-42de-b0df-bb2d658a6903-istio-podinfo\") pod \"e2e-ipp-cleanup-ff67f5-openshift-default-69dfcf4d7b-64kkj\" (UID: \"754f6d7c-9685-42de-b0df-bb2d658a6903\") " pod="openshift-ingress/e2e-ipp-cleanup-ff67f5-openshift-default-69dfcf4d7b-64kkj" Jul 30 01:10:56.236455 ip-10-0-142-6 kubenswrapper[2564]: I0730 01:10:56.236402 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/754f6d7c-9685-42de-b0df-bb2d658a6903-istiod-ca-cert\") pod \"e2e-ipp-cleanup-ff67f5-openshift-default-69dfcf4d7b-64kkj\" (UID: \"754f6d7c-9685-42de-b0df-bb2d658a6903\") " pod="openshift-ingress/e2e-ipp-cleanup-ff67f5-openshift-default-69dfcf4d7b-64kkj" Jul 30 01:10:56.236716 ip-10-0-142-6 kubenswrapper[2564]: I0730 01:10:56.236690 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/754f6d7c-9685-42de-b0df-bb2d658a6903-workload-socket\") pod \"e2e-ipp-cleanup-ff67f5-openshift-default-69dfcf4d7b-64kkj\" (UID: \"754f6d7c-9685-42de-b0df-bb2d658a6903\") " pod="openshift-ingress/e2e-ipp-cleanup-ff67f5-openshift-default-69dfcf4d7b-64kkj" Jul 30 01:10:56.237108 ip-10-0-142-6 kubenswrapper[2564]: I0730 01:10:56.236811 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ztqk9\" (UniqueName: \"kubernetes.io/projected/754f6d7c-9685-42de-b0df-bb2d658a6903-kube-api-access-ztqk9\") pod \"e2e-ipp-cleanup-ff67f5-openshift-default-69dfcf4d7b-64kkj\" (UID: \"754f6d7c-9685-42de-b0df-bb2d658a6903\") " pod="openshift-ingress/e2e-ipp-cleanup-ff67f5-openshift-default-69dfcf4d7b-64kkj" Jul 30 01:10:56.237108 ip-10-0-142-6 kubenswrapper[2564]: I0730 01:10:56.236869 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/754f6d7c-9685-42de-b0df-bb2d658a6903-istio-envoy\") pod \"e2e-ipp-cleanup-ff67f5-openshift-default-69dfcf4d7b-64kkj\" (UID: \"754f6d7c-9685-42de-b0df-bb2d658a6903\") " pod="openshift-ingress/e2e-ipp-cleanup-ff67f5-openshift-default-69dfcf4d7b-64kkj" Jul 30 01:10:56.237108 ip-10-0-142-6 kubenswrapper[2564]: I0730 01:10:56.236904 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/754f6d7c-9685-42de-b0df-bb2d658a6903-workload-certs\") pod \"e2e-ipp-cleanup-ff67f5-openshift-default-69dfcf4d7b-64kkj\" (UID: \"754f6d7c-9685-42de-b0df-bb2d658a6903\") " pod="openshift-ingress/e2e-ipp-cleanup-ff67f5-openshift-default-69dfcf4d7b-64kkj" Jul 30 01:10:56.237108 ip-10-0-142-6 kubenswrapper[2564]: I0730 01:10:56.237100 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/754f6d7c-9685-42de-b0df-bb2d658a6903-credential-socket\") pod \"e2e-ipp-cleanup-ff67f5-openshift-default-69dfcf4d7b-64kkj\" (UID: \"754f6d7c-9685-42de-b0df-bb2d658a6903\") " pod="openshift-ingress/e2e-ipp-cleanup-ff67f5-openshift-default-69dfcf4d7b-64kkj" Jul 30 01:10:56.338245 ip-10-0-142-6 kubenswrapper[2564]: I0730 01:10:56.338211 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/754f6d7c-9685-42de-b0df-bb2d658a6903-istio-token\") pod \"e2e-ipp-cleanup-ff67f5-openshift-default-69dfcf4d7b-64kkj\" (UID: \"754f6d7c-9685-42de-b0df-bb2d658a6903\") " pod="openshift-ingress/e2e-ipp-cleanup-ff67f5-openshift-default-69dfcf4d7b-64kkj" Jul 30 01:10:56.338459 ip-10-0-142-6 kubenswrapper[2564]: I0730 01:10:56.338277 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/754f6d7c-9685-42de-b0df-bb2d658a6903-istio-data\") pod \"e2e-ipp-cleanup-ff67f5-openshift-default-69dfcf4d7b-64kkj\" (UID: \"754f6d7c-9685-42de-b0df-bb2d658a6903\") " pod="openshift-ingress/e2e-ipp-cleanup-ff67f5-openshift-default-69dfcf4d7b-64kkj" Jul 30 01:10:56.338459 ip-10-0-142-6 kubenswrapper[2564]: I0730 01:10:56.338394 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/754f6d7c-9685-42de-b0df-bb2d658a6903-istio-podinfo\") pod \"e2e-ipp-cleanup-ff67f5-openshift-default-69dfcf4d7b-64kkj\" (UID: \"754f6d7c-9685-42de-b0df-bb2d658a6903\") " pod="openshift-ingress/e2e-ipp-cleanup-ff67f5-openshift-default-69dfcf4d7b-64kkj" Jul 30 01:10:56.338459 ip-10-0-142-6 kubenswrapper[2564]: I0730 01:10:56.338443 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/754f6d7c-9685-42de-b0df-bb2d658a6903-istiod-ca-cert\") pod \"e2e-ipp-cleanup-ff67f5-openshift-default-69dfcf4d7b-64kkj\" (UID: \"754f6d7c-9685-42de-b0df-bb2d658a6903\") " pod="openshift-ingress/e2e-ipp-cleanup-ff67f5-openshift-default-69dfcf4d7b-64kkj" Jul 30 01:10:56.338636 ip-10-0-142-6 kubenswrapper[2564]: I0730 01:10:56.338482 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/754f6d7c-9685-42de-b0df-bb2d658a6903-workload-socket\") pod \"e2e-ipp-cleanup-ff67f5-openshift-default-69dfcf4d7b-64kkj\" (UID: \"754f6d7c-9685-42de-b0df-bb2d658a6903\") " pod="openshift-ingress/e2e-ipp-cleanup-ff67f5-openshift-default-69dfcf4d7b-64kkj" Jul 30 01:10:56.338636 ip-10-0-142-6 kubenswrapper[2564]: I0730 01:10:56.338519 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ztqk9\" (UniqueName: \"kubernetes.io/projected/754f6d7c-9685-42de-b0df-bb2d658a6903-kube-api-access-ztqk9\") pod \"e2e-ipp-cleanup-ff67f5-openshift-default-69dfcf4d7b-64kkj\" (UID: \"754f6d7c-9685-42de-b0df-bb2d658a6903\") " pod="openshift-ingress/e2e-ipp-cleanup-ff67f5-openshift-default-69dfcf4d7b-64kkj" Jul 30 01:10:56.338744 ip-10-0-142-6 kubenswrapper[2564]: I0730 01:10:56.338646 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/754f6d7c-9685-42de-b0df-bb2d658a6903-istio-envoy\") pod \"e2e-ipp-cleanup-ff67f5-openshift-default-69dfcf4d7b-64kkj\" (UID: \"754f6d7c-9685-42de-b0df-bb2d658a6903\") " pod="openshift-ingress/e2e-ipp-cleanup-ff67f5-openshift-default-69dfcf4d7b-64kkj" Jul 30 01:10:56.338744 ip-10-0-142-6 kubenswrapper[2564]: I0730 01:10:56.338679 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/754f6d7c-9685-42de-b0df-bb2d658a6903-istio-data\") pod \"e2e-ipp-cleanup-ff67f5-openshift-default-69dfcf4d7b-64kkj\" (UID: \"754f6d7c-9685-42de-b0df-bb2d658a6903\") " pod="openshift-ingress/e2e-ipp-cleanup-ff67f5-openshift-default-69dfcf4d7b-64kkj" Jul 30 01:10:56.338744 ip-10-0-142-6 kubenswrapper[2564]: I0730 01:10:56.338699 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/754f6d7c-9685-42de-b0df-bb2d658a6903-workload-certs\") pod \"e2e-ipp-cleanup-ff67f5-openshift-default-69dfcf4d7b-64kkj\" (UID: \"754f6d7c-9685-42de-b0df-bb2d658a6903\") " pod="openshift-ingress/e2e-ipp-cleanup-ff67f5-openshift-default-69dfcf4d7b-64kkj" Jul 30 01:10:56.338896 ip-10-0-142-6 kubenswrapper[2564]: I0730 01:10:56.338753 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/754f6d7c-9685-42de-b0df-bb2d658a6903-credential-socket\") pod \"e2e-ipp-cleanup-ff67f5-openshift-default-69dfcf4d7b-64kkj\" (UID: \"754f6d7c-9685-42de-b0df-bb2d658a6903\") " pod="openshift-ingress/e2e-ipp-cleanup-ff67f5-openshift-default-69dfcf4d7b-64kkj" Jul 30 01:10:56.338896 ip-10-0-142-6 kubenswrapper[2564]: I0730 01:10:56.338853 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/754f6d7c-9685-42de-b0df-bb2d658a6903-workload-socket\") pod \"e2e-ipp-cleanup-ff67f5-openshift-default-69dfcf4d7b-64kkj\" (UID: \"754f6d7c-9685-42de-b0df-bb2d658a6903\") " pod="openshift-ingress/e2e-ipp-cleanup-ff67f5-openshift-default-69dfcf4d7b-64kkj" Jul 30 01:10:56.339122 ip-10-0-142-6 kubenswrapper[2564]: I0730 01:10:56.339099 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/754f6d7c-9685-42de-b0df-bb2d658a6903-workload-certs\") pod \"e2e-ipp-cleanup-ff67f5-openshift-default-69dfcf4d7b-64kkj\" (UID: \"754f6d7c-9685-42de-b0df-bb2d658a6903\") " pod="openshift-ingress/e2e-ipp-cleanup-ff67f5-openshift-default-69dfcf4d7b-64kkj" Jul 30 01:10:56.339211 ip-10-0-142-6 kubenswrapper[2564]: I0730 01:10:56.339110 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/754f6d7c-9685-42de-b0df-bb2d658a6903-credential-socket\") pod \"e2e-ipp-cleanup-ff67f5-openshift-default-69dfcf4d7b-64kkj\" (UID: \"754f6d7c-9685-42de-b0df-bb2d658a6903\") " pod="openshift-ingress/e2e-ipp-cleanup-ff67f5-openshift-default-69dfcf4d7b-64kkj" Jul 30 01:10:56.339277 ip-10-0-142-6 kubenswrapper[2564]: I0730 01:10:56.339230 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/754f6d7c-9685-42de-b0df-bb2d658a6903-istiod-ca-cert\") pod \"e2e-ipp-cleanup-ff67f5-openshift-default-69dfcf4d7b-64kkj\" (UID: \"754f6d7c-9685-42de-b0df-bb2d658a6903\") " pod="openshift-ingress/e2e-ipp-cleanup-ff67f5-openshift-default-69dfcf4d7b-64kkj" Jul 30 01:10:56.340819 ip-10-0-142-6 kubenswrapper[2564]: I0730 01:10:56.340791 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/754f6d7c-9685-42de-b0df-bb2d658a6903-istio-envoy\") pod \"e2e-ipp-cleanup-ff67f5-openshift-default-69dfcf4d7b-64kkj\" (UID: \"754f6d7c-9685-42de-b0df-bb2d658a6903\") " pod="openshift-ingress/e2e-ipp-cleanup-ff67f5-openshift-default-69dfcf4d7b-64kkj" Jul 30 01:10:56.341140 ip-10-0-142-6 kubenswrapper[2564]: I0730 01:10:56.341119 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/754f6d7c-9685-42de-b0df-bb2d658a6903-istio-podinfo\") pod \"e2e-ipp-cleanup-ff67f5-openshift-default-69dfcf4d7b-64kkj\" (UID: \"754f6d7c-9685-42de-b0df-bb2d658a6903\") " pod="openshift-ingress/e2e-ipp-cleanup-ff67f5-openshift-default-69dfcf4d7b-64kkj" Jul 30 01:10:56.347422 ip-10-0-142-6 kubenswrapper[2564]: I0730 01:10:56.347390 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ztqk9\" (UniqueName: \"kubernetes.io/projected/754f6d7c-9685-42de-b0df-bb2d658a6903-kube-api-access-ztqk9\") pod \"e2e-ipp-cleanup-ff67f5-openshift-default-69dfcf4d7b-64kkj\" (UID: \"754f6d7c-9685-42de-b0df-bb2d658a6903\") " pod="openshift-ingress/e2e-ipp-cleanup-ff67f5-openshift-default-69dfcf4d7b-64kkj" Jul 30 01:10:56.347674 ip-10-0-142-6 kubenswrapper[2564]: I0730 01:10:56.347656 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/754f6d7c-9685-42de-b0df-bb2d658a6903-istio-token\") pod \"e2e-ipp-cleanup-ff67f5-openshift-default-69dfcf4d7b-64kkj\" (UID: \"754f6d7c-9685-42de-b0df-bb2d658a6903\") " pod="openshift-ingress/e2e-ipp-cleanup-ff67f5-openshift-default-69dfcf4d7b-64kkj" Jul 30 01:10:56.435719 ip-10-0-142-6 kubenswrapper[2564]: I0730 01:10:56.435691 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-cleanup-ff67f5-openshift-default-69dfcf4d7b-64kkj" Jul 30 01:10:56.574376 ip-10-0-142-6 kubenswrapper[2564]: I0730 01:10:56.574347 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-cleanup-ff67f5-openshift-default-69dfcf4d7b-64kkj"] Jul 30 01:10:56.576009 ip-10-0-142-6 kubenswrapper[2564]: W0730 01:10:56.575983 2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod754f6d7c_9685_42de_b0df_bb2d658a6903.slice/crio-79df781de79ed8843355e9ec2e7e2d4c0fe4179abcb32cf1090d76e86a714982 WatchSource:0}: Error finding container 79df781de79ed8843355e9ec2e7e2d4c0fe4179abcb32cf1090d76e86a714982: Status 404 returned error can't find the container with id 79df781de79ed8843355e9ec2e7e2d4c0fe4179abcb32cf1090d76e86a714982 Jul 30 01:10:56.581295 ip-10-0-142-6 kubenswrapper[2564]: I0730 01:10:56.581263 2564 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236220Ki","pods":"250"} Jul 30 01:10:56.581390 ip-10-0-142-6 kubenswrapper[2564]: I0730 01:10:56.581353 2564 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236220Ki","pods":"250"} Jul 30 01:10:56.581390 ip-10-0-142-6 kubenswrapper[2564]: I0730 01:10:56.581385 2564 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236220Ki","pods":"250"} Jul 30 01:10:57.389683 ip-10-0-142-6 kubenswrapper[2564]: I0730 01:10:57.389649 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-cleanup-ff67f5-openshift-default-69dfcf4d7b-64kkj" event={"ID":"754f6d7c-9685-42de-b0df-bb2d658a6903","Type":"ContainerStarted","Data":"5ac4b2cdcd2dff2ee7ce4911eea82d67173b9c7d19c7b366ba0fde7519694d4e"} Jul 30 01:10:57.389874 ip-10-0-142-6 kubenswrapper[2564]: I0730 01:10:57.389702 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-cleanup-ff67f5-openshift-default-69dfcf4d7b-64kkj" event={"ID":"754f6d7c-9685-42de-b0df-bb2d658a6903","Type":"ContainerStarted","Data":"79df781de79ed8843355e9ec2e7e2d4c0fe4179abcb32cf1090d76e86a714982"} Jul 30 01:10:57.411318 ip-10-0-142-6 kubenswrapper[2564]: I0730 01:10:57.411267 2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-ipp-cleanup-ff67f5-openshift-default-69dfcf4d7b-64kkj" podStartSLOduration=1.411250875 podStartE2EDuration="1.411250875s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-30 01:10:56 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-30 01:10:57.408893212 +0000 UTC m=+3188.823633145" watchObservedRunningTime="2026-07-30 01:10:57.411250875 +0000 UTC m=+3188.825990808" Jul 30 01:10:57.436431 ip-10-0-142-6 kubenswrapper[2564]: I0730 01:10:57.436400 2564 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-cleanup-ff67f5-openshift-default-69dfcf4d7b-64kkj" Jul 30 01:10:57.441612 ip-10-0-142-6 kubenswrapper[2564]: I0730 01:10:57.441589 2564 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-cleanup-ff67f5-openshift-default-69dfcf4d7b-64kkj" Jul 30 01:10:58.395385 ip-10-0-142-6 kubenswrapper[2564]: I0730 01:10:58.395353 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-cleanup-ff67f5-openshift-default-69dfcf4d7b-64kkj" Jul 30 01:10:58.396400 ip-10-0-142-6 kubenswrapper[2564]: I0730 01:10:58.396374 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-cleanup-ff67f5-openshift-default-69dfcf4d7b-64kkj" Jul 30 01:11:10.814088 ip-10-0-142-6 kubenswrapper[2564]: I0730 01:11:10.814034 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-nrv2p"] Jul 30 01:11:19.507030 ip-10-0-142-6 kubenswrapper[2564]: I0730 01:11:19.506997 2564 generic.go:356] "Generic (PLEG): container finished" podID="754f6d7c-9685-42de-b0df-bb2d658a6903" containerID="5ac4b2cdcd2dff2ee7ce4911eea82d67173b9c7d19c7b366ba0fde7519694d4e" exitCode=0 Jul 30 01:11:19.507504 ip-10-0-142-6 kubenswrapper[2564]: I0730 01:11:19.507085 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-cleanup-ff67f5-openshift-default-69dfcf4d7b-64kkj" event={"ID":"754f6d7c-9685-42de-b0df-bb2d658a6903","Type":"ContainerDied","Data":"5ac4b2cdcd2dff2ee7ce4911eea82d67173b9c7d19c7b366ba0fde7519694d4e"} Jul 30 01:11:19.507556 ip-10-0-142-6 kubenswrapper[2564]: I0730 01:11:19.507521 2564 scope.go:117] "RemoveContainer" containerID="5ac4b2cdcd2dff2ee7ce4911eea82d67173b9c7d19c7b366ba0fde7519694d4e" Jul 30 01:11:19.508746 ip-10-0-142-6 kubenswrapper[2564]: I0730 01:11:19.508714 2564 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236220Ki","pods":"250"} Jul 30 01:11:19.508809 ip-10-0-142-6 kubenswrapper[2564]: I0730 01:11:19.508789 2564 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236220Ki","pods":"250"} Jul 30 01:11:19.508869 ip-10-0-142-6 kubenswrapper[2564]: I0730 01:11:19.508829 2564 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236220Ki","pods":"250"} Jul 30 01:11:19.787105 ip-10-0-142-6 kubenswrapper[2564]: I0730 01:11:19.787014 2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-ff67f5-ndvmz"] Jul 30 01:11:19.789289 ip-10-0-142-6 kubenswrapper[2564]: I0730 01:11:19.789271 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-ff67f5-ndvmz" Jul 30 01:11:19.791959 ip-10-0-142-6 kubenswrapper[2564]: I0730 01:11:19.791935 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-v8q7j\"" Jul 30 01:11:19.792149 ip-10-0-142-6 kubenswrapper[2564]: I0730 01:11:19.791957 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Jul 30 01:11:19.798831 ip-10-0-142-6 kubenswrapper[2564]: I0730 01:11:19.798805 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-ff67f5-ndvmz"] Jul 30 01:11:19.863189 ip-10-0-142-6 kubenswrapper[2564]: I0730 01:11:19.863157 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/74425d91-a03b-44a2-a447-a20d426d665d-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ipp-cleanup-ff67f5-ndvmz\" (UID: \"74425d91-a03b-44a2-a447-a20d426d665d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-ff67f5-ndvmz" Jul 30 01:11:19.964382 ip-10-0-142-6 kubenswrapper[2564]: I0730 01:11:19.964339 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/74425d91-a03b-44a2-a447-a20d426d665d-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ipp-cleanup-ff67f5-ndvmz\" (UID: \"74425d91-a03b-44a2-a447-a20d426d665d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-ff67f5-ndvmz" Jul 30 01:11:19.965014 ip-10-0-142-6 kubenswrapper[2564]: I0730 01:11:19.964992 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/74425d91-a03b-44a2-a447-a20d426d665d-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ipp-cleanup-ff67f5-ndvmz\" (UID: \"74425d91-a03b-44a2-a447-a20d426d665d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-ff67f5-ndvmz" Jul 30 01:11:20.101514 ip-10-0-142-6 kubenswrapper[2564]: I0730 01:11:20.101425 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-ff67f5-ndvmz" Jul 30 01:11:20.250673 ip-10-0-142-6 kubenswrapper[2564]: W0730 01:11:20.250642 2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod74425d91_a03b_44a2_a447_a20d426d665d.slice/crio-ec8d146425a819e60e74ef24257fb3281f977bc304c759994e1f325baee3c9e0 WatchSource:0}: Error finding container ec8d146425a819e60e74ef24257fb3281f977bc304c759994e1f325baee3c9e0: Status 404 returned error can't find the container with id ec8d146425a819e60e74ef24257fb3281f977bc304c759994e1f325baee3c9e0 Jul 30 01:11:20.250834 ip-10-0-142-6 kubenswrapper[2564]: I0730 01:11:20.250665 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-ff67f5-ndvmz"] Jul 30 01:11:20.514652 ip-10-0-142-6 kubenswrapper[2564]: I0730 01:11:20.514618 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-cleanup-ff67f5-openshift-default-69dfcf4d7b-64kkj" event={"ID":"754f6d7c-9685-42de-b0df-bb2d658a6903","Type":"ContainerStarted","Data":"1cf8c0688ea89c86532071bd9b0c835998ccd0e6bb7cfee76b50010af0bcef09"} Jul 30 01:11:20.515857 ip-10-0-142-6 kubenswrapper[2564]: I0730 01:11:20.515837 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-ff67f5-ndvmz" event={"ID":"74425d91-a03b-44a2-a447-a20d426d665d","Type":"ContainerStarted","Data":"ec8d146425a819e60e74ef24257fb3281f977bc304c759994e1f325baee3c9e0"} Jul 30 01:11:21.436698 ip-10-0-142-6 kubenswrapper[2564]: I0730 01:11:21.436594 2564 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-cleanup-ff67f5-openshift-default-69dfcf4d7b-64kkj" Jul 30 01:11:21.436698 ip-10-0-142-6 kubenswrapper[2564]: I0730 01:11:21.436663 2564 patch_prober.go:28] interesting pod/e2e-ipp-cleanup-ff67f5-openshift-default-69dfcf4d7b-64kkj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.57:15021/healthz/ready\": dial tcp 10.133.0.57:15021: connect: connection refused" start-of-body= Jul 30 01:11:21.436960 ip-10-0-142-6 kubenswrapper[2564]: I0730 01:11:21.436715 2564 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-cleanup-ff67f5-openshift-default-69dfcf4d7b-64kkj" podUID="754f6d7c-9685-42de-b0df-bb2d658a6903" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.57:15021/healthz/ready\": dial tcp 10.133.0.57:15021: connect: connection refused" Jul 30 01:11:21.541185 ip-10-0-142-6 kubenswrapper[2564]: I0730 01:11:21.541142 2564 generic.go:356] "Generic (PLEG): container finished" podID="74425d91-a03b-44a2-a447-a20d426d665d" containerID="8c8cba681322631db2e1edb032da8f88e13506578b54af4dc0db4eff04353962" exitCode=0 Jul 30 01:11:21.545094 ip-10-0-142-6 kubenswrapper[2564]: I0730 01:11:21.542650 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-ff67f5-ndvmz" event={"ID":"74425d91-a03b-44a2-a447-a20d426d665d","Type":"ContainerDied","Data":"8c8cba681322631db2e1edb032da8f88e13506578b54af4dc0db4eff04353962"} Jul 30 01:11:22.436890 ip-10-0-142-6 kubenswrapper[2564]: I0730 01:11:22.436851 2564 patch_prober.go:28] interesting pod/e2e-ipp-cleanup-ff67f5-openshift-default-69dfcf4d7b-64kkj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.57:15021/healthz/ready\": dial tcp 10.133.0.57:15021: connect: connection refused" start-of-body= Jul 30 01:11:22.437231 ip-10-0-142-6 kubenswrapper[2564]: I0730 01:11:22.437198 2564 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-cleanup-ff67f5-openshift-default-69dfcf4d7b-64kkj" podUID="754f6d7c-9685-42de-b0df-bb2d658a6903" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.57:15021/healthz/ready\": dial tcp 10.133.0.57:15021: connect: connection refused" Jul 30 01:11:22.695705 ip-10-0-142-6 kubenswrapper[2564]: I0730 01:11:22.695642 2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-ff67f5-ndvmz" Jul 30 01:11:22.795046 ip-10-0-142-6 kubenswrapper[2564]: I0730 01:11:22.795016 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/74425d91-a03b-44a2-a447-a20d426d665d-maas-api-service-ca\") pod \"74425d91-a03b-44a2-a447-a20d426d665d\" (UID: \"74425d91-a03b-44a2-a447-a20d426d665d\") " Jul 30 01:11:22.795372 ip-10-0-142-6 kubenswrapper[2564]: I0730 01:11:22.795352 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/74425d91-a03b-44a2-a447-a20d426d665d-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "74425d91-a03b-44a2-a447-a20d426d665d" (UID: "74425d91-a03b-44a2-a447-a20d426d665d"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 30 01:11:22.896714 ip-10-0-142-6 kubenswrapper[2564]: I0730 01:11:22.896678 2564 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/74425d91-a03b-44a2-a447-a20d426d665d-maas-api-service-ca\") on node \"ip-10-0-142-6.ec2.internal\" DevicePath \"\"" Jul 30 01:11:23.437224 ip-10-0-142-6 kubenswrapper[2564]: I0730 01:11:23.437120 2564 patch_prober.go:28] interesting pod/e2e-ipp-cleanup-ff67f5-openshift-default-69dfcf4d7b-64kkj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.57:15021/healthz/ready\": dial tcp 10.133.0.57:15021: connect: connection refused" start-of-body= Jul 30 01:11:23.437224 ip-10-0-142-6 kubenswrapper[2564]: I0730 01:11:23.437187 2564 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-cleanup-ff67f5-openshift-default-69dfcf4d7b-64kkj" podUID="754f6d7c-9685-42de-b0df-bb2d658a6903" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.57:15021/healthz/ready\": dial tcp 10.133.0.57:15021: connect: connection refused" Jul 30 01:11:23.553738 ip-10-0-142-6 kubenswrapper[2564]: I0730 01:11:23.553711 2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-ff67f5-ndvmz" Jul 30 01:11:23.553919 ip-10-0-142-6 kubenswrapper[2564]: I0730 01:11:23.553709 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-ff67f5-ndvmz" event={"ID":"74425d91-a03b-44a2-a447-a20d426d665d","Type":"ContainerDied","Data":"ec8d146425a819e60e74ef24257fb3281f977bc304c759994e1f325baee3c9e0"} Jul 30 01:11:23.553919 ip-10-0-142-6 kubenswrapper[2564]: I0730 01:11:23.553830 2564 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="ec8d146425a819e60e74ef24257fb3281f977bc304c759994e1f325baee3c9e0" Jul 30 01:11:24.436391 ip-10-0-142-6 kubenswrapper[2564]: I0730 01:11:24.436359 2564 patch_prober.go:28] interesting pod/e2e-ipp-cleanup-ff67f5-openshift-default-69dfcf4d7b-64kkj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.57:15021/healthz/ready\": dial tcp 10.133.0.57:15021: connect: connection refused" start-of-body= Jul 30 01:11:24.436799 ip-10-0-142-6 kubenswrapper[2564]: I0730 01:11:24.436415 2564 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-cleanup-ff67f5-openshift-default-69dfcf4d7b-64kkj" podUID="754f6d7c-9685-42de-b0df-bb2d658a6903" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.57:15021/healthz/ready\": dial tcp 10.133.0.57:15021: connect: connection refused" Jul 30 01:11:25.440338 ip-10-0-142-6 kubenswrapper[2564]: I0730 01:11:25.440308 2564 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-cleanup-ff67f5-openshift-default-69dfcf4d7b-64kkj" Jul 30 01:11:25.440743 ip-10-0-142-6 kubenswrapper[2564]: I0730 01:11:25.440677 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-cleanup-ff67f5-openshift-default-69dfcf4d7b-64kkj" Jul 30 01:11:25.441485 ip-10-0-142-6 kubenswrapper[2564]: I0730 01:11:25.441470 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-cleanup-ff67f5-openshift-default-69dfcf4d7b-64kkj" Jul 30 01:11:27.558190 ip-10-0-142-6 kubenswrapper[2564]: I0730 01:11:27.558148 2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-ff67f5-ndvmz"] Jul 30 01:11:27.561664 ip-10-0-142-6 kubenswrapper[2564]: I0730 01:11:27.561638 2564 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-ff67f5-ndvmz"] Jul 30 01:11:29.139600 ip-10-0-142-6 kubenswrapper[2564]: I0730 01:11:29.139559 2564 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="74425d91-a03b-44a2-a447-a20d426d665d" path="/var/lib/kubelet/pods/74425d91-a03b-44a2-a447-a20d426d665d/volumes" Jul 30 01:11:31.817885 ip-10-0-142-6 kubenswrapper[2564]: I0730 01:11:31.817851 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-nrv2p"] Jul 30 01:11:34.104809 ip-10-0-142-6 kubenswrapper[2564]: I0730 01:11:34.104766 2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-cleanup-ff67f5-openshift-default-69dfcf4d7b-64kkj"] Jul 30 01:11:34.105239 ip-10-0-142-6 kubenswrapper[2564]: I0730 01:11:34.105025 2564 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-ipp-cleanup-ff67f5-openshift-default-69dfcf4d7b-64kkj" podUID="754f6d7c-9685-42de-b0df-bb2d658a6903" containerName="istio-proxy" containerID="cri-o://1cf8c0688ea89c86532071bd9b0c835998ccd0e6bb7cfee76b50010af0bcef09" gracePeriod=30 Jul 30 01:11:39.371703 ip-10-0-142-6 kubenswrapper[2564]: I0730 01:11:39.371642 2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-cleanup-ff67f5-openshift-default-69dfcf4d7b-64kkj" Jul 30 01:11:39.547728 ip-10-0-142-6 kubenswrapper[2564]: I0730 01:11:39.547696 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/754f6d7c-9685-42de-b0df-bb2d658a6903-workload-certs\") pod \"754f6d7c-9685-42de-b0df-bb2d658a6903\" (UID: \"754f6d7c-9685-42de-b0df-bb2d658a6903\") " Jul 30 01:11:39.547914 ip-10-0-142-6 kubenswrapper[2564]: I0730 01:11:39.547753 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/754f6d7c-9685-42de-b0df-bb2d658a6903-istio-token\") pod \"754f6d7c-9685-42de-b0df-bb2d658a6903\" (UID: \"754f6d7c-9685-42de-b0df-bb2d658a6903\") " Jul 30 01:11:39.547914 ip-10-0-142-6 kubenswrapper[2564]: I0730 01:11:39.547778 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-ztqk9\" (UniqueName: \"kubernetes.io/projected/754f6d7c-9685-42de-b0df-bb2d658a6903-kube-api-access-ztqk9\") pod \"754f6d7c-9685-42de-b0df-bb2d658a6903\" (UID: \"754f6d7c-9685-42de-b0df-bb2d658a6903\") " Jul 30 01:11:39.547914 ip-10-0-142-6 kubenswrapper[2564]: I0730 01:11:39.547875 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/754f6d7c-9685-42de-b0df-bb2d658a6903-credential-socket\") pod \"754f6d7c-9685-42de-b0df-bb2d658a6903\" (UID: \"754f6d7c-9685-42de-b0df-bb2d658a6903\") " Jul 30 01:11:39.547914 ip-10-0-142-6 kubenswrapper[2564]: I0730 01:11:39.547903 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/754f6d7c-9685-42de-b0df-bb2d658a6903-istio-podinfo\") pod \"754f6d7c-9685-42de-b0df-bb2d658a6903\" (UID: \"754f6d7c-9685-42de-b0df-bb2d658a6903\") " Jul 30 01:11:39.548567 ip-10-0-142-6 kubenswrapper[2564]: I0730 01:11:39.547927 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/754f6d7c-9685-42de-b0df-bb2d658a6903-istiod-ca-cert\") pod \"754f6d7c-9685-42de-b0df-bb2d658a6903\" (UID: \"754f6d7c-9685-42de-b0df-bb2d658a6903\") " Jul 30 01:11:39.548567 ip-10-0-142-6 kubenswrapper[2564]: I0730 01:11:39.547950 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/754f6d7c-9685-42de-b0df-bb2d658a6903-istio-data\") pod \"754f6d7c-9685-42de-b0df-bb2d658a6903\" (UID: \"754f6d7c-9685-42de-b0df-bb2d658a6903\") " Jul 30 01:11:39.548567 ip-10-0-142-6 kubenswrapper[2564]: I0730 01:11:39.547976 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/754f6d7c-9685-42de-b0df-bb2d658a6903-istio-envoy\") pod \"754f6d7c-9685-42de-b0df-bb2d658a6903\" (UID: \"754f6d7c-9685-42de-b0df-bb2d658a6903\") " Jul 30 01:11:39.548567 ip-10-0-142-6 kubenswrapper[2564]: I0730 01:11:39.547983 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/754f6d7c-9685-42de-b0df-bb2d658a6903-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "754f6d7c-9685-42de-b0df-bb2d658a6903" (UID: "754f6d7c-9685-42de-b0df-bb2d658a6903"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 30 01:11:39.548567 ip-10-0-142-6 kubenswrapper[2564]: I0730 01:11:39.548031 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/754f6d7c-9685-42de-b0df-bb2d658a6903-workload-socket\") pod \"754f6d7c-9685-42de-b0df-bb2d658a6903\" (UID: \"754f6d7c-9685-42de-b0df-bb2d658a6903\") " Jul 30 01:11:39.548567 ip-10-0-142-6 kubenswrapper[2564]: I0730 01:11:39.548355 2564 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/754f6d7c-9685-42de-b0df-bb2d658a6903-workload-certs\") on node \"ip-10-0-142-6.ec2.internal\" DevicePath \"\"" Jul 30 01:11:39.548567 ip-10-0-142-6 kubenswrapper[2564]: I0730 01:11:39.548361 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/754f6d7c-9685-42de-b0df-bb2d658a6903-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "754f6d7c-9685-42de-b0df-bb2d658a6903" (UID: "754f6d7c-9685-42de-b0df-bb2d658a6903"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 30 01:11:39.548567 ip-10-0-142-6 kubenswrapper[2564]: I0730 01:11:39.548476 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/754f6d7c-9685-42de-b0df-bb2d658a6903-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "754f6d7c-9685-42de-b0df-bb2d658a6903" (UID: "754f6d7c-9685-42de-b0df-bb2d658a6903"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 30 01:11:39.548851 ip-10-0-142-6 kubenswrapper[2564]: I0730 01:11:39.548584 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/754f6d7c-9685-42de-b0df-bb2d658a6903-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "754f6d7c-9685-42de-b0df-bb2d658a6903" (UID: "754f6d7c-9685-42de-b0df-bb2d658a6903"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 30 01:11:39.548970 ip-10-0-142-6 kubenswrapper[2564]: I0730 01:11:39.548946 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/754f6d7c-9685-42de-b0df-bb2d658a6903-istio-data" (OuterVolumeSpecName: "istio-data") pod "754f6d7c-9685-42de-b0df-bb2d658a6903" (UID: "754f6d7c-9685-42de-b0df-bb2d658a6903"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 30 01:11:39.550324 ip-10-0-142-6 kubenswrapper[2564]: I0730 01:11:39.550300 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/754f6d7c-9685-42de-b0df-bb2d658a6903-kube-api-access-ztqk9" (OuterVolumeSpecName: "kube-api-access-ztqk9") pod "754f6d7c-9685-42de-b0df-bb2d658a6903" (UID: "754f6d7c-9685-42de-b0df-bb2d658a6903"). InnerVolumeSpecName "kube-api-access-ztqk9". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 30 01:11:39.550425 ip-10-0-142-6 kubenswrapper[2564]: I0730 01:11:39.550344 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/754f6d7c-9685-42de-b0df-bb2d658a6903-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "754f6d7c-9685-42de-b0df-bb2d658a6903" (UID: "754f6d7c-9685-42de-b0df-bb2d658a6903"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Jul 30 01:11:39.550483 ip-10-0-142-6 kubenswrapper[2564]: I0730 01:11:39.550474 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/754f6d7c-9685-42de-b0df-bb2d658a6903-istio-token" (OuterVolumeSpecName: "istio-token") pod "754f6d7c-9685-42de-b0df-bb2d658a6903" (UID: "754f6d7c-9685-42de-b0df-bb2d658a6903"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 30 01:11:39.550526 ip-10-0-142-6 kubenswrapper[2564]: I0730 01:11:39.550509 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/754f6d7c-9685-42de-b0df-bb2d658a6903-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "754f6d7c-9685-42de-b0df-bb2d658a6903" (UID: "754f6d7c-9685-42de-b0df-bb2d658a6903"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 30 01:11:39.640043 ip-10-0-142-6 kubenswrapper[2564]: I0730 01:11:39.639950 2564 generic.go:356] "Generic (PLEG): container finished" podID="754f6d7c-9685-42de-b0df-bb2d658a6903" containerID="1cf8c0688ea89c86532071bd9b0c835998ccd0e6bb7cfee76b50010af0bcef09" exitCode=0 Jul 30 01:11:39.640043 ip-10-0-142-6 kubenswrapper[2564]: I0730 01:11:39.639997 2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-cleanup-ff67f5-openshift-default-69dfcf4d7b-64kkj" Jul 30 01:11:39.640043 ip-10-0-142-6 kubenswrapper[2564]: I0730 01:11:39.640035 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-cleanup-ff67f5-openshift-default-69dfcf4d7b-64kkj" event={"ID":"754f6d7c-9685-42de-b0df-bb2d658a6903","Type":"ContainerDied","Data":"1cf8c0688ea89c86532071bd9b0c835998ccd0e6bb7cfee76b50010af0bcef09"} Jul 30 01:11:39.640285 ip-10-0-142-6 kubenswrapper[2564]: I0730 01:11:39.640117 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-cleanup-ff67f5-openshift-default-69dfcf4d7b-64kkj" event={"ID":"754f6d7c-9685-42de-b0df-bb2d658a6903","Type":"ContainerDied","Data":"79df781de79ed8843355e9ec2e7e2d4c0fe4179abcb32cf1090d76e86a714982"} Jul 30 01:11:39.640285 ip-10-0-142-6 kubenswrapper[2564]: I0730 01:11:39.640142 2564 scope.go:117] "RemoveContainer" containerID="1cf8c0688ea89c86532071bd9b0c835998ccd0e6bb7cfee76b50010af0bcef09" Jul 30 01:11:39.650880 ip-10-0-142-6 kubenswrapper[2564]: I0730 01:11:39.650854 2564 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/754f6d7c-9685-42de-b0df-bb2d658a6903-istio-token\") on node \"ip-10-0-142-6.ec2.internal\" DevicePath \"\"" Jul 30 01:11:39.651122 ip-10-0-142-6 kubenswrapper[2564]: I0730 01:11:39.650894 2564 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ztqk9\" (UniqueName: \"kubernetes.io/projected/754f6d7c-9685-42de-b0df-bb2d658a6903-kube-api-access-ztqk9\") on node \"ip-10-0-142-6.ec2.internal\" DevicePath \"\"" Jul 30 01:11:39.651243 ip-10-0-142-6 kubenswrapper[2564]: I0730 01:11:39.651141 2564 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/754f6d7c-9685-42de-b0df-bb2d658a6903-credential-socket\") on node \"ip-10-0-142-6.ec2.internal\" DevicePath \"\"" Jul 30 01:11:39.651243 ip-10-0-142-6 kubenswrapper[2564]: I0730 01:11:39.651153 2564 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/754f6d7c-9685-42de-b0df-bb2d658a6903-istio-podinfo\") on node \"ip-10-0-142-6.ec2.internal\" DevicePath \"\"" Jul 30 01:11:39.651243 ip-10-0-142-6 kubenswrapper[2564]: I0730 01:11:39.651163 2564 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/754f6d7c-9685-42de-b0df-bb2d658a6903-istiod-ca-cert\") on node \"ip-10-0-142-6.ec2.internal\" DevicePath \"\"" Jul 30 01:11:39.651243 ip-10-0-142-6 kubenswrapper[2564]: I0730 01:11:39.651171 2564 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/754f6d7c-9685-42de-b0df-bb2d658a6903-istio-data\") on node \"ip-10-0-142-6.ec2.internal\" DevicePath \"\"" Jul 30 01:11:39.651243 ip-10-0-142-6 kubenswrapper[2564]: I0730 01:11:39.651178 2564 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/754f6d7c-9685-42de-b0df-bb2d658a6903-istio-envoy\") on node \"ip-10-0-142-6.ec2.internal\" DevicePath \"\"" Jul 30 01:11:39.651243 ip-10-0-142-6 kubenswrapper[2564]: I0730 01:11:39.651186 2564 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/754f6d7c-9685-42de-b0df-bb2d658a6903-workload-socket\") on node \"ip-10-0-142-6.ec2.internal\" DevicePath \"\"" Jul 30 01:11:39.656839 ip-10-0-142-6 kubenswrapper[2564]: I0730 01:11:39.656822 2564 scope.go:117] "RemoveContainer" containerID="5ac4b2cdcd2dff2ee7ce4911eea82d67173b9c7d19c7b366ba0fde7519694d4e" Jul 30 01:11:39.665331 ip-10-0-142-6 kubenswrapper[2564]: I0730 01:11:39.665309 2564 scope.go:117] "RemoveContainer" containerID="1cf8c0688ea89c86532071bd9b0c835998ccd0e6bb7cfee76b50010af0bcef09" Jul 30 01:11:39.665618 ip-10-0-142-6 kubenswrapper[2564]: E0730 01:11:39.665599 2564 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1cf8c0688ea89c86532071bd9b0c835998ccd0e6bb7cfee76b50010af0bcef09\": container with ID starting with 1cf8c0688ea89c86532071bd9b0c835998ccd0e6bb7cfee76b50010af0bcef09 not found: ID does not exist" containerID="1cf8c0688ea89c86532071bd9b0c835998ccd0e6bb7cfee76b50010af0bcef09" Jul 30 01:11:39.665659 ip-10-0-142-6 kubenswrapper[2564]: I0730 01:11:39.665629 2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1cf8c0688ea89c86532071bd9b0c835998ccd0e6bb7cfee76b50010af0bcef09"} err="failed to get container status \"1cf8c0688ea89c86532071bd9b0c835998ccd0e6bb7cfee76b50010af0bcef09\": rpc error: code = NotFound desc = could not find container \"1cf8c0688ea89c86532071bd9b0c835998ccd0e6bb7cfee76b50010af0bcef09\": container with ID starting with 1cf8c0688ea89c86532071bd9b0c835998ccd0e6bb7cfee76b50010af0bcef09 not found: ID does not exist" Jul 30 01:11:39.665659 ip-10-0-142-6 kubenswrapper[2564]: I0730 01:11:39.665648 2564 scope.go:117] "RemoveContainer" containerID="5ac4b2cdcd2dff2ee7ce4911eea82d67173b9c7d19c7b366ba0fde7519694d4e" Jul 30 01:11:39.665911 ip-10-0-142-6 kubenswrapper[2564]: E0730 01:11:39.665893 2564 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5ac4b2cdcd2dff2ee7ce4911eea82d67173b9c7d19c7b366ba0fde7519694d4e\": container with ID starting with 5ac4b2cdcd2dff2ee7ce4911eea82d67173b9c7d19c7b366ba0fde7519694d4e not found: ID does not exist" containerID="5ac4b2cdcd2dff2ee7ce4911eea82d67173b9c7d19c7b366ba0fde7519694d4e" Jul 30 01:11:39.665959 ip-10-0-142-6 kubenswrapper[2564]: I0730 01:11:39.665917 2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5ac4b2cdcd2dff2ee7ce4911eea82d67173b9c7d19c7b366ba0fde7519694d4e"} err="failed to get container status \"5ac4b2cdcd2dff2ee7ce4911eea82d67173b9c7d19c7b366ba0fde7519694d4e\": rpc error: code = NotFound desc = could not find container \"5ac4b2cdcd2dff2ee7ce4911eea82d67173b9c7d19c7b366ba0fde7519694d4e\": container with ID starting with 5ac4b2cdcd2dff2ee7ce4911eea82d67173b9c7d19c7b366ba0fde7519694d4e not found: ID does not exist" Jul 30 01:11:39.675694 ip-10-0-142-6 kubenswrapper[2564]: I0730 01:11:39.675668 2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-cleanup-ff67f5-openshift-default-69dfcf4d7b-64kkj"] Jul 30 01:11:39.677714 ip-10-0-142-6 kubenswrapper[2564]: I0730 01:11:39.677694 2564 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ipp-cleanup-ff67f5-openshift-default-69dfcf4d7b-64kkj"] Jul 30 01:11:41.139365 ip-10-0-142-6 kubenswrapper[2564]: I0730 01:11:41.139289 2564 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="754f6d7c-9685-42de-b0df-bb2d658a6903" path="/var/lib/kubelet/pods/754f6d7c-9685-42de-b0df-bb2d658a6903/volumes" Jul 30 01:11:49.312892 ip-10-0-142-6 kubenswrapper[2564]: I0730 01:11:49.312850 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-nrv2p"] Jul 30 01:11:51.039729 ip-10-0-142-6 kubenswrapper[2564]: I0730 01:11:51.039694 2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-5912cb-6fc86f47bb-zv4lr"] Jul 30 01:11:51.040367 ip-10-0-142-6 kubenswrapper[2564]: I0730 01:11:51.039951 2564 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ipp-a-5912cb-6fc86f47bb-zv4lr" podUID="5e58f698-a89d-48bd-86fb-673fe182dcfa" containerName="payload-processing" containerID="cri-o://48c7d53afef979fbff86c25312c59523bf69c4c91980dd1d8252e41315b21bee" gracePeriod=30 Jul 30 01:11:51.302223 ip-10-0-142-6 kubenswrapper[2564]: I0730 01:11:51.302159 2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-5912cb-6fc86f47bb-zv4lr" Jul 30 01:11:51.350830 ip-10-0-142-6 kubenswrapper[2564]: I0730 01:11:51.350795 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-l95r8\" (UniqueName: \"kubernetes.io/projected/5e58f698-a89d-48bd-86fb-673fe182dcfa-kube-api-access-l95r8\") pod \"5e58f698-a89d-48bd-86fb-673fe182dcfa\" (UID: \"5e58f698-a89d-48bd-86fb-673fe182dcfa\") " Jul 30 01:11:51.351033 ip-10-0-142-6 kubenswrapper[2564]: I0730 01:11:51.350974 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5e58f698-a89d-48bd-86fb-673fe182dcfa-plugins-config-volume\") pod \"5e58f698-a89d-48bd-86fb-673fe182dcfa\" (UID: \"5e58f698-a89d-48bd-86fb-673fe182dcfa\") " Jul 30 01:11:51.351558 ip-10-0-142-6 kubenswrapper[2564]: I0730 01:11:51.351528 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5e58f698-a89d-48bd-86fb-673fe182dcfa-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "5e58f698-a89d-48bd-86fb-673fe182dcfa" (UID: "5e58f698-a89d-48bd-86fb-673fe182dcfa"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 30 01:11:51.353266 ip-10-0-142-6 kubenswrapper[2564]: I0730 01:11:51.353240 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5e58f698-a89d-48bd-86fb-673fe182dcfa-kube-api-access-l95r8" (OuterVolumeSpecName: "kube-api-access-l95r8") pod "5e58f698-a89d-48bd-86fb-673fe182dcfa" (UID: "5e58f698-a89d-48bd-86fb-673fe182dcfa"). InnerVolumeSpecName "kube-api-access-l95r8". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 30 01:11:51.452211 ip-10-0-142-6 kubenswrapper[2564]: I0730 01:11:51.452171 2564 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-l95r8\" (UniqueName: \"kubernetes.io/projected/5e58f698-a89d-48bd-86fb-673fe182dcfa-kube-api-access-l95r8\") on node \"ip-10-0-142-6.ec2.internal\" DevicePath \"\"" Jul 30 01:11:51.452211 ip-10-0-142-6 kubenswrapper[2564]: I0730 01:11:51.452205 2564 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5e58f698-a89d-48bd-86fb-673fe182dcfa-plugins-config-volume\") on node \"ip-10-0-142-6.ec2.internal\" DevicePath \"\"" Jul 30 01:11:51.713974 ip-10-0-142-6 kubenswrapper[2564]: I0730 01:11:51.713933 2564 generic.go:356] "Generic (PLEG): container finished" podID="5e58f698-a89d-48bd-86fb-673fe182dcfa" containerID="48c7d53afef979fbff86c25312c59523bf69c4c91980dd1d8252e41315b21bee" exitCode=0 Jul 30 01:11:51.714185 ip-10-0-142-6 kubenswrapper[2564]: I0730 01:11:51.713983 2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-5912cb-6fc86f47bb-zv4lr" Jul 30 01:11:51.714185 ip-10-0-142-6 kubenswrapper[2564]: I0730 01:11:51.713985 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-5912cb-6fc86f47bb-zv4lr" event={"ID":"5e58f698-a89d-48bd-86fb-673fe182dcfa","Type":"ContainerDied","Data":"48c7d53afef979fbff86c25312c59523bf69c4c91980dd1d8252e41315b21bee"} Jul 30 01:11:51.714185 ip-10-0-142-6 kubenswrapper[2564]: I0730 01:11:51.714161 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-5912cb-6fc86f47bb-zv4lr" event={"ID":"5e58f698-a89d-48bd-86fb-673fe182dcfa","Type":"ContainerDied","Data":"07fb428895fefc58e5a9e274dd53617f8bd64b5c9dd56ae9a39d92f8869ad4e7"} Jul 30 01:11:51.714298 ip-10-0-142-6 kubenswrapper[2564]: I0730 01:11:51.714196 2564 scope.go:117] "RemoveContainer" containerID="48c7d53afef979fbff86c25312c59523bf69c4c91980dd1d8252e41315b21bee" Jul 30 01:11:51.724009 ip-10-0-142-6 kubenswrapper[2564]: I0730 01:11:51.723989 2564 scope.go:117] "RemoveContainer" containerID="48c7d53afef979fbff86c25312c59523bf69c4c91980dd1d8252e41315b21bee" Jul 30 01:11:51.724321 ip-10-0-142-6 kubenswrapper[2564]: E0730 01:11:51.724299 2564 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"48c7d53afef979fbff86c25312c59523bf69c4c91980dd1d8252e41315b21bee\": container with ID starting with 48c7d53afef979fbff86c25312c59523bf69c4c91980dd1d8252e41315b21bee not found: ID does not exist" containerID="48c7d53afef979fbff86c25312c59523bf69c4c91980dd1d8252e41315b21bee" Jul 30 01:11:51.724379 ip-10-0-142-6 kubenswrapper[2564]: I0730 01:11:51.724330 2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"48c7d53afef979fbff86c25312c59523bf69c4c91980dd1d8252e41315b21bee"} err="failed to get container status \"48c7d53afef979fbff86c25312c59523bf69c4c91980dd1d8252e41315b21bee\": rpc error: code = NotFound desc = could not find container \"48c7d53afef979fbff86c25312c59523bf69c4c91980dd1d8252e41315b21bee\": container with ID starting with 48c7d53afef979fbff86c25312c59523bf69c4c91980dd1d8252e41315b21bee not found: ID does not exist" Jul 30 01:11:51.735979 ip-10-0-142-6 kubenswrapper[2564]: I0730 01:11:51.735948 2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-5912cb-6fc86f47bb-zv4lr"] Jul 30 01:11:51.741826 ip-10-0-142-6 kubenswrapper[2564]: I0730 01:11:51.741800 2564 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-5912cb-6fc86f47bb-zv4lr"] Jul 30 01:11:53.139761 ip-10-0-142-6 kubenswrapper[2564]: I0730 01:11:53.139724 2564 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="5e58f698-a89d-48bd-86fb-673fe182dcfa" path="/var/lib/kubelet/pods/5e58f698-a89d-48bd-86fb-673fe182dcfa/volumes" Jul 30 01:12:00.105658 ip-10-0-142-6 kubenswrapper[2564]: I0730 01:12:00.105619 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-nrv2p"] Jul 30 01:12:46.597428 ip-10-0-142-6 kubenswrapper[2564]: I0730 01:12:46.597391 2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-a-5912cb-openshift-default-65b755b7cd-x7mnv"] Jul 30 01:12:46.597933 ip-10-0-142-6 kubenswrapper[2564]: I0730 01:12:46.597719 2564 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-ipp-a-5912cb-openshift-default-65b755b7cd-x7mnv" podUID="eb89fc81-7fdb-44f0-ab34-749e1d98c776" containerName="istio-proxy" containerID="cri-o://0425ef3ee24cb53bec5eba4df141ab9247fad103eb63d70d6ee89609dbb72ecd" gracePeriod=30 Jul 30 01:12:51.853865 ip-10-0-142-6 kubenswrapper[2564]: I0730 01:12:51.853839 2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-5912cb-openshift-default-65b755b7cd-x7mnv" Jul 30 01:12:51.897424 ip-10-0-142-6 kubenswrapper[2564]: I0730 01:12:51.897397 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/eb89fc81-7fdb-44f0-ab34-749e1d98c776-istio-podinfo\") pod \"eb89fc81-7fdb-44f0-ab34-749e1d98c776\" (UID: \"eb89fc81-7fdb-44f0-ab34-749e1d98c776\") " Jul 30 01:12:51.897613 ip-10-0-142-6 kubenswrapper[2564]: I0730 01:12:51.897437 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-wrln8\" (UniqueName: \"kubernetes.io/projected/eb89fc81-7fdb-44f0-ab34-749e1d98c776-kube-api-access-wrln8\") pod \"eb89fc81-7fdb-44f0-ab34-749e1d98c776\" (UID: \"eb89fc81-7fdb-44f0-ab34-749e1d98c776\") " Jul 30 01:12:51.897613 ip-10-0-142-6 kubenswrapper[2564]: I0730 01:12:51.897474 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/eb89fc81-7fdb-44f0-ab34-749e1d98c776-istio-data\") pod \"eb89fc81-7fdb-44f0-ab34-749e1d98c776\" (UID: \"eb89fc81-7fdb-44f0-ab34-749e1d98c776\") " Jul 30 01:12:51.897613 ip-10-0-142-6 kubenswrapper[2564]: I0730 01:12:51.897519 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/eb89fc81-7fdb-44f0-ab34-749e1d98c776-istio-token\") pod \"eb89fc81-7fdb-44f0-ab34-749e1d98c776\" (UID: \"eb89fc81-7fdb-44f0-ab34-749e1d98c776\") " Jul 30 01:12:51.897613 ip-10-0-142-6 kubenswrapper[2564]: I0730 01:12:51.897554 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/eb89fc81-7fdb-44f0-ab34-749e1d98c776-istio-envoy\") pod \"eb89fc81-7fdb-44f0-ab34-749e1d98c776\" (UID: \"eb89fc81-7fdb-44f0-ab34-749e1d98c776\") " Jul 30 01:12:51.897613 ip-10-0-142-6 kubenswrapper[2564]: I0730 01:12:51.897578 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/eb89fc81-7fdb-44f0-ab34-749e1d98c776-credential-socket\") pod \"eb89fc81-7fdb-44f0-ab34-749e1d98c776\" (UID: \"eb89fc81-7fdb-44f0-ab34-749e1d98c776\") " Jul 30 01:12:51.897613 ip-10-0-142-6 kubenswrapper[2564]: I0730 01:12:51.897613 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/eb89fc81-7fdb-44f0-ab34-749e1d98c776-istiod-ca-cert\") pod \"eb89fc81-7fdb-44f0-ab34-749e1d98c776\" (UID: \"eb89fc81-7fdb-44f0-ab34-749e1d98c776\") " Jul 30 01:12:51.897968 ip-10-0-142-6 kubenswrapper[2564]: I0730 01:12:51.897671 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/eb89fc81-7fdb-44f0-ab34-749e1d98c776-workload-socket\") pod \"eb89fc81-7fdb-44f0-ab34-749e1d98c776\" (UID: \"eb89fc81-7fdb-44f0-ab34-749e1d98c776\") " Jul 30 01:12:51.897968 ip-10-0-142-6 kubenswrapper[2564]: I0730 01:12:51.897694 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/eb89fc81-7fdb-44f0-ab34-749e1d98c776-workload-certs\") pod \"eb89fc81-7fdb-44f0-ab34-749e1d98c776\" (UID: \"eb89fc81-7fdb-44f0-ab34-749e1d98c776\") " Jul 30 01:12:51.898493 ip-10-0-142-6 kubenswrapper[2564]: I0730 01:12:51.898162 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/eb89fc81-7fdb-44f0-ab34-749e1d98c776-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "eb89fc81-7fdb-44f0-ab34-749e1d98c776" (UID: "eb89fc81-7fdb-44f0-ab34-749e1d98c776"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 30 01:12:51.898493 ip-10-0-142-6 kubenswrapper[2564]: I0730 01:12:51.898360 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/eb89fc81-7fdb-44f0-ab34-749e1d98c776-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "eb89fc81-7fdb-44f0-ab34-749e1d98c776" (UID: "eb89fc81-7fdb-44f0-ab34-749e1d98c776"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 30 01:12:51.898493 ip-10-0-142-6 kubenswrapper[2564]: I0730 01:12:51.898352 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/eb89fc81-7fdb-44f0-ab34-749e1d98c776-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "eb89fc81-7fdb-44f0-ab34-749e1d98c776" (UID: "eb89fc81-7fdb-44f0-ab34-749e1d98c776"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 30 01:12:51.898493 ip-10-0-142-6 kubenswrapper[2564]: I0730 01:12:51.898452 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/eb89fc81-7fdb-44f0-ab34-749e1d98c776-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "eb89fc81-7fdb-44f0-ab34-749e1d98c776" (UID: "eb89fc81-7fdb-44f0-ab34-749e1d98c776"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 30 01:12:51.898782 ip-10-0-142-6 kubenswrapper[2564]: I0730 01:12:51.898504 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/eb89fc81-7fdb-44f0-ab34-749e1d98c776-istio-data" (OuterVolumeSpecName: "istio-data") pod "eb89fc81-7fdb-44f0-ab34-749e1d98c776" (UID: "eb89fc81-7fdb-44f0-ab34-749e1d98c776"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 30 01:12:51.900026 ip-10-0-142-6 kubenswrapper[2564]: I0730 01:12:51.900001 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/eb89fc81-7fdb-44f0-ab34-749e1d98c776-kube-api-access-wrln8" (OuterVolumeSpecName: "kube-api-access-wrln8") pod "eb89fc81-7fdb-44f0-ab34-749e1d98c776" (UID: "eb89fc81-7fdb-44f0-ab34-749e1d98c776"). InnerVolumeSpecName "kube-api-access-wrln8". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 30 01:12:51.900158 ip-10-0-142-6 kubenswrapper[2564]: I0730 01:12:51.900023 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/eb89fc81-7fdb-44f0-ab34-749e1d98c776-istio-token" (OuterVolumeSpecName: "istio-token") pod "eb89fc81-7fdb-44f0-ab34-749e1d98c776" (UID: "eb89fc81-7fdb-44f0-ab34-749e1d98c776"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 30 01:12:51.900158 ip-10-0-142-6 kubenswrapper[2564]: I0730 01:12:51.900100 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/eb89fc81-7fdb-44f0-ab34-749e1d98c776-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "eb89fc81-7fdb-44f0-ab34-749e1d98c776" (UID: "eb89fc81-7fdb-44f0-ab34-749e1d98c776"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Jul 30 01:12:51.900416 ip-10-0-142-6 kubenswrapper[2564]: I0730 01:12:51.900390 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/eb89fc81-7fdb-44f0-ab34-749e1d98c776-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "eb89fc81-7fdb-44f0-ab34-749e1d98c776" (UID: "eb89fc81-7fdb-44f0-ab34-749e1d98c776"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 30 01:12:51.998759 ip-10-0-142-6 kubenswrapper[2564]: I0730 01:12:51.998721 2564 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/eb89fc81-7fdb-44f0-ab34-749e1d98c776-workload-socket\") on node \"ip-10-0-142-6.ec2.internal\" DevicePath \"\"" Jul 30 01:12:51.998759 ip-10-0-142-6 kubenswrapper[2564]: I0730 01:12:51.998750 2564 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/eb89fc81-7fdb-44f0-ab34-749e1d98c776-workload-certs\") on node \"ip-10-0-142-6.ec2.internal\" DevicePath \"\"" Jul 30 01:12:51.998759 ip-10-0-142-6 kubenswrapper[2564]: I0730 01:12:51.998758 2564 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/eb89fc81-7fdb-44f0-ab34-749e1d98c776-istio-podinfo\") on node \"ip-10-0-142-6.ec2.internal\" DevicePath \"\"" Jul 30 01:12:51.998759 ip-10-0-142-6 kubenswrapper[2564]: I0730 01:12:51.998766 2564 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wrln8\" (UniqueName: \"kubernetes.io/projected/eb89fc81-7fdb-44f0-ab34-749e1d98c776-kube-api-access-wrln8\") on node \"ip-10-0-142-6.ec2.internal\" DevicePath \"\"" Jul 30 01:12:51.999032 ip-10-0-142-6 kubenswrapper[2564]: I0730 01:12:51.998776 2564 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/eb89fc81-7fdb-44f0-ab34-749e1d98c776-istio-data\") on node \"ip-10-0-142-6.ec2.internal\" DevicePath \"\"" Jul 30 01:12:51.999032 ip-10-0-142-6 kubenswrapper[2564]: I0730 01:12:51.998783 2564 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/eb89fc81-7fdb-44f0-ab34-749e1d98c776-istio-token\") on node \"ip-10-0-142-6.ec2.internal\" DevicePath \"\"" Jul 30 01:12:51.999032 ip-10-0-142-6 kubenswrapper[2564]: I0730 01:12:51.998791 2564 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/eb89fc81-7fdb-44f0-ab34-749e1d98c776-istio-envoy\") on node \"ip-10-0-142-6.ec2.internal\" DevicePath \"\"" Jul 30 01:12:51.999032 ip-10-0-142-6 kubenswrapper[2564]: I0730 01:12:51.998798 2564 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/eb89fc81-7fdb-44f0-ab34-749e1d98c776-credential-socket\") on node \"ip-10-0-142-6.ec2.internal\" DevicePath \"\"" Jul 30 01:12:51.999032 ip-10-0-142-6 kubenswrapper[2564]: I0730 01:12:51.998807 2564 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/eb89fc81-7fdb-44f0-ab34-749e1d98c776-istiod-ca-cert\") on node \"ip-10-0-142-6.ec2.internal\" DevicePath \"\"" Jul 30 01:12:52.046531 ip-10-0-142-6 kubenswrapper[2564]: I0730 01:12:52.046502 2564 generic.go:356] "Generic (PLEG): container finished" podID="eb89fc81-7fdb-44f0-ab34-749e1d98c776" containerID="0425ef3ee24cb53bec5eba4df141ab9247fad103eb63d70d6ee89609dbb72ecd" exitCode=0 Jul 30 01:12:52.046690 ip-10-0-142-6 kubenswrapper[2564]: I0730 01:12:52.046559 2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-5912cb-openshift-default-65b755b7cd-x7mnv" Jul 30 01:12:52.046690 ip-10-0-142-6 kubenswrapper[2564]: I0730 01:12:52.046587 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-5912cb-openshift-default-65b755b7cd-x7mnv" event={"ID":"eb89fc81-7fdb-44f0-ab34-749e1d98c776","Type":"ContainerDied","Data":"0425ef3ee24cb53bec5eba4df141ab9247fad103eb63d70d6ee89609dbb72ecd"} Jul 30 01:12:52.046690 ip-10-0-142-6 kubenswrapper[2564]: I0730 01:12:52.046640 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-5912cb-openshift-default-65b755b7cd-x7mnv" event={"ID":"eb89fc81-7fdb-44f0-ab34-749e1d98c776","Type":"ContainerDied","Data":"115df141208f9488745878d195bf4afb9b780d1a56a767672e17a3396dba6ceb"} Jul 30 01:12:52.046690 ip-10-0-142-6 kubenswrapper[2564]: I0730 01:12:52.046661 2564 scope.go:117] "RemoveContainer" containerID="0425ef3ee24cb53bec5eba4df141ab9247fad103eb63d70d6ee89609dbb72ecd" Jul 30 01:12:52.057806 ip-10-0-142-6 kubenswrapper[2564]: I0730 01:12:52.057785 2564 scope.go:117] "RemoveContainer" containerID="9e3f6bb3c5ca8e9c1580faecc8be839aad45a6ee0e3edddae72afeb93205223e" Jul 30 01:12:52.067316 ip-10-0-142-6 kubenswrapper[2564]: I0730 01:12:52.067273 2564 scope.go:117] "RemoveContainer" containerID="0425ef3ee24cb53bec5eba4df141ab9247fad103eb63d70d6ee89609dbb72ecd" Jul 30 01:12:52.067593 ip-10-0-142-6 kubenswrapper[2564]: E0730 01:12:52.067574 2564 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0425ef3ee24cb53bec5eba4df141ab9247fad103eb63d70d6ee89609dbb72ecd\": container with ID starting with 0425ef3ee24cb53bec5eba4df141ab9247fad103eb63d70d6ee89609dbb72ecd not found: ID does not exist" containerID="0425ef3ee24cb53bec5eba4df141ab9247fad103eb63d70d6ee89609dbb72ecd" Jul 30 01:12:52.067639 ip-10-0-142-6 kubenswrapper[2564]: I0730 01:12:52.067601 2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0425ef3ee24cb53bec5eba4df141ab9247fad103eb63d70d6ee89609dbb72ecd"} err="failed to get container status \"0425ef3ee24cb53bec5eba4df141ab9247fad103eb63d70d6ee89609dbb72ecd\": rpc error: code = NotFound desc = could not find container \"0425ef3ee24cb53bec5eba4df141ab9247fad103eb63d70d6ee89609dbb72ecd\": container with ID starting with 0425ef3ee24cb53bec5eba4df141ab9247fad103eb63d70d6ee89609dbb72ecd not found: ID does not exist" Jul 30 01:12:52.067639 ip-10-0-142-6 kubenswrapper[2564]: I0730 01:12:52.067620 2564 scope.go:117] "RemoveContainer" containerID="9e3f6bb3c5ca8e9c1580faecc8be839aad45a6ee0e3edddae72afeb93205223e" Jul 30 01:12:52.067877 ip-10-0-142-6 kubenswrapper[2564]: E0730 01:12:52.067838 2564 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9e3f6bb3c5ca8e9c1580faecc8be839aad45a6ee0e3edddae72afeb93205223e\": container with ID starting with 9e3f6bb3c5ca8e9c1580faecc8be839aad45a6ee0e3edddae72afeb93205223e not found: ID does not exist" containerID="9e3f6bb3c5ca8e9c1580faecc8be839aad45a6ee0e3edddae72afeb93205223e" Jul 30 01:12:52.067877 ip-10-0-142-6 kubenswrapper[2564]: I0730 01:12:52.067858 2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9e3f6bb3c5ca8e9c1580faecc8be839aad45a6ee0e3edddae72afeb93205223e"} err="failed to get container status \"9e3f6bb3c5ca8e9c1580faecc8be839aad45a6ee0e3edddae72afeb93205223e\": rpc error: code = NotFound desc = could not find container \"9e3f6bb3c5ca8e9c1580faecc8be839aad45a6ee0e3edddae72afeb93205223e\": container with ID starting with 9e3f6bb3c5ca8e9c1580faecc8be839aad45a6ee0e3edddae72afeb93205223e not found: ID does not exist" Jul 30 01:12:52.069323 ip-10-0-142-6 kubenswrapper[2564]: I0730 01:12:52.069299 2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-a-5912cb-openshift-default-65b755b7cd-x7mnv"] Jul 30 01:12:52.072147 ip-10-0-142-6 kubenswrapper[2564]: I0730 01:12:52.072128 2564 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ipp-a-5912cb-openshift-default-65b755b7cd-x7mnv"] Jul 30 01:12:53.138972 ip-10-0-142-6 kubenswrapper[2564]: I0730 01:12:53.138941 2564 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="eb89fc81-7fdb-44f0-ab34-749e1d98c776" path="/var/lib/kubelet/pods/eb89fc81-7fdb-44f0-ab34-749e1d98c776/volumes" Jul 30 01:12:53.181436 ip-10-0-142-6 kubenswrapper[2564]: I0730 01:12:53.181402 2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-47d629-58fcfd8d64-drzlv"] Jul 30 01:12:53.181699 ip-10-0-142-6 kubenswrapper[2564]: I0730 01:12:53.181670 2564 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ipp-b-47d629-58fcfd8d64-drzlv" podUID="65fecc99-42fa-4ab1-8215-d800172c1b0d" containerName="payload-processing" containerID="cri-o://9cc66e3954463bba33cc2a02c6713fe917b7cc6aaf4433eec5facb4a402e0966" gracePeriod=30 Jul 30 01:12:53.188093 ip-10-0-142-6 kubenswrapper[2564]: I0730 01:12:53.188052 2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-47d629-76d8595b56-sf2ms"] Jul 30 01:12:53.188322 ip-10-0-142-6 kubenswrapper[2564]: I0730 01:12:53.188293 2564 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-47d629-76d8595b56-sf2ms" podUID="1235e5ac-6ee9-4ae3-b5ca-fa824edb6227" containerName="payload-pre-processing" containerID="cri-o://0ad0373533c170d1b3927f2d401f45e8f0525c8678eb823b8d4f0a43bd7e5b8c" gracePeriod=30 Jul 30 01:12:53.432797 ip-10-0-142-6 kubenswrapper[2564]: I0730 01:12:53.432770 2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-47d629-58fcfd8d64-drzlv" Jul 30 01:12:53.466295 ip-10-0-142-6 kubenswrapper[2564]: I0730 01:12:53.466268 2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-47d629-76d8595b56-sf2ms" Jul 30 01:12:53.512774 ip-10-0-142-6 kubenswrapper[2564]: I0730 01:12:53.512741 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-2jzjr\" (UniqueName: \"kubernetes.io/projected/1235e5ac-6ee9-4ae3-b5ca-fa824edb6227-kube-api-access-2jzjr\") pod \"1235e5ac-6ee9-4ae3-b5ca-fa824edb6227\" (UID: \"1235e5ac-6ee9-4ae3-b5ca-fa824edb6227\") " Jul 30 01:12:53.512774 ip-10-0-142-6 kubenswrapper[2564]: I0730 01:12:53.512772 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-6whh6\" (UniqueName: \"kubernetes.io/projected/65fecc99-42fa-4ab1-8215-d800172c1b0d-kube-api-access-6whh6\") pod \"65fecc99-42fa-4ab1-8215-d800172c1b0d\" (UID: \"65fecc99-42fa-4ab1-8215-d800172c1b0d\") " Jul 30 01:12:53.513031 ip-10-0-142-6 kubenswrapper[2564]: I0730 01:12:53.512793 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/65fecc99-42fa-4ab1-8215-d800172c1b0d-plugins-config-volume\") pod \"65fecc99-42fa-4ab1-8215-d800172c1b0d\" (UID: \"65fecc99-42fa-4ab1-8215-d800172c1b0d\") " Jul 30 01:12:53.513031 ip-10-0-142-6 kubenswrapper[2564]: I0730 01:12:53.512889 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1235e5ac-6ee9-4ae3-b5ca-fa824edb6227-plugins-config-volume\") pod \"1235e5ac-6ee9-4ae3-b5ca-fa824edb6227\" (UID: \"1235e5ac-6ee9-4ae3-b5ca-fa824edb6227\") " Jul 30 01:12:53.513259 ip-10-0-142-6 kubenswrapper[2564]: I0730 01:12:53.513225 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/65fecc99-42fa-4ab1-8215-d800172c1b0d-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "65fecc99-42fa-4ab1-8215-d800172c1b0d" (UID: "65fecc99-42fa-4ab1-8215-d800172c1b0d"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 30 01:12:53.513259 ip-10-0-142-6 kubenswrapper[2564]: I0730 01:12:53.513257 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1235e5ac-6ee9-4ae3-b5ca-fa824edb6227-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "1235e5ac-6ee9-4ae3-b5ca-fa824edb6227" (UID: "1235e5ac-6ee9-4ae3-b5ca-fa824edb6227"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 30 01:12:53.514960 ip-10-0-142-6 kubenswrapper[2564]: I0730 01:12:53.514938 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/65fecc99-42fa-4ab1-8215-d800172c1b0d-kube-api-access-6whh6" (OuterVolumeSpecName: "kube-api-access-6whh6") pod "65fecc99-42fa-4ab1-8215-d800172c1b0d" (UID: "65fecc99-42fa-4ab1-8215-d800172c1b0d"). InnerVolumeSpecName "kube-api-access-6whh6". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 30 01:12:53.514960 ip-10-0-142-6 kubenswrapper[2564]: I0730 01:12:53.514946 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1235e5ac-6ee9-4ae3-b5ca-fa824edb6227-kube-api-access-2jzjr" (OuterVolumeSpecName: "kube-api-access-2jzjr") pod "1235e5ac-6ee9-4ae3-b5ca-fa824edb6227" (UID: "1235e5ac-6ee9-4ae3-b5ca-fa824edb6227"). InnerVolumeSpecName "kube-api-access-2jzjr". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 30 01:12:53.614384 ip-10-0-142-6 kubenswrapper[2564]: I0730 01:12:53.614349 2564 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1235e5ac-6ee9-4ae3-b5ca-fa824edb6227-plugins-config-volume\") on node \"ip-10-0-142-6.ec2.internal\" DevicePath \"\"" Jul 30 01:12:53.614384 ip-10-0-142-6 kubenswrapper[2564]: I0730 01:12:53.614377 2564 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2jzjr\" (UniqueName: \"kubernetes.io/projected/1235e5ac-6ee9-4ae3-b5ca-fa824edb6227-kube-api-access-2jzjr\") on node \"ip-10-0-142-6.ec2.internal\" DevicePath \"\"" Jul 30 01:12:53.614384 ip-10-0-142-6 kubenswrapper[2564]: I0730 01:12:53.614387 2564 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6whh6\" (UniqueName: \"kubernetes.io/projected/65fecc99-42fa-4ab1-8215-d800172c1b0d-kube-api-access-6whh6\") on node \"ip-10-0-142-6.ec2.internal\" DevicePath \"\"" Jul 30 01:12:53.614384 ip-10-0-142-6 kubenswrapper[2564]: I0730 01:12:53.614398 2564 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/65fecc99-42fa-4ab1-8215-d800172c1b0d-plugins-config-volume\") on node \"ip-10-0-142-6.ec2.internal\" DevicePath \"\"" Jul 30 01:12:54.059709 ip-10-0-142-6 kubenswrapper[2564]: I0730 01:12:54.059677 2564 generic.go:356] "Generic (PLEG): container finished" podID="65fecc99-42fa-4ab1-8215-d800172c1b0d" containerID="9cc66e3954463bba33cc2a02c6713fe917b7cc6aaf4433eec5facb4a402e0966" exitCode=0 Jul 30 01:12:54.059863 ip-10-0-142-6 kubenswrapper[2564]: I0730 01:12:54.059725 2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-47d629-58fcfd8d64-drzlv" Jul 30 01:12:54.059863 ip-10-0-142-6 kubenswrapper[2564]: I0730 01:12:54.059759 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-47d629-58fcfd8d64-drzlv" event={"ID":"65fecc99-42fa-4ab1-8215-d800172c1b0d","Type":"ContainerDied","Data":"9cc66e3954463bba33cc2a02c6713fe917b7cc6aaf4433eec5facb4a402e0966"} Jul 30 01:12:54.059863 ip-10-0-142-6 kubenswrapper[2564]: I0730 01:12:54.059810 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-47d629-58fcfd8d64-drzlv" event={"ID":"65fecc99-42fa-4ab1-8215-d800172c1b0d","Type":"ContainerDied","Data":"4273001edabcc9b09b4b0f099aa717589fc6e9af0fbc65be39a4c908e1fe0347"} Jul 30 01:12:54.059863 ip-10-0-142-6 kubenswrapper[2564]: I0730 01:12:54.059841 2564 scope.go:117] "RemoveContainer" containerID="9cc66e3954463bba33cc2a02c6713fe917b7cc6aaf4433eec5facb4a402e0966" Jul 30 01:12:54.061235 ip-10-0-142-6 kubenswrapper[2564]: I0730 01:12:54.061208 2564 generic.go:356] "Generic (PLEG): container finished" podID="1235e5ac-6ee9-4ae3-b5ca-fa824edb6227" containerID="0ad0373533c170d1b3927f2d401f45e8f0525c8678eb823b8d4f0a43bd7e5b8c" exitCode=0 Jul 30 01:12:54.061374 ip-10-0-142-6 kubenswrapper[2564]: I0730 01:12:54.061264 2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-47d629-76d8595b56-sf2ms" Jul 30 01:12:54.061374 ip-10-0-142-6 kubenswrapper[2564]: I0730 01:12:54.061281 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-47d629-76d8595b56-sf2ms" event={"ID":"1235e5ac-6ee9-4ae3-b5ca-fa824edb6227","Type":"ContainerDied","Data":"0ad0373533c170d1b3927f2d401f45e8f0525c8678eb823b8d4f0a43bd7e5b8c"} Jul 30 01:12:54.061374 ip-10-0-142-6 kubenswrapper[2564]: I0730 01:12:54.061321 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-47d629-76d8595b56-sf2ms" event={"ID":"1235e5ac-6ee9-4ae3-b5ca-fa824edb6227","Type":"ContainerDied","Data":"680838ab5b29067c7bc6f450bd2d865c49b7ea2f49a7b771883a7460917b86f7"} Jul 30 01:12:54.068861 ip-10-0-142-6 kubenswrapper[2564]: I0730 01:12:54.068842 2564 scope.go:117] "RemoveContainer" containerID="9cc66e3954463bba33cc2a02c6713fe917b7cc6aaf4433eec5facb4a402e0966" Jul 30 01:12:54.069161 ip-10-0-142-6 kubenswrapper[2564]: E0730 01:12:54.069140 2564 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9cc66e3954463bba33cc2a02c6713fe917b7cc6aaf4433eec5facb4a402e0966\": container with ID starting with 9cc66e3954463bba33cc2a02c6713fe917b7cc6aaf4433eec5facb4a402e0966 not found: ID does not exist" containerID="9cc66e3954463bba33cc2a02c6713fe917b7cc6aaf4433eec5facb4a402e0966" Jul 30 01:12:54.069268 ip-10-0-142-6 kubenswrapper[2564]: I0730 01:12:54.069166 2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9cc66e3954463bba33cc2a02c6713fe917b7cc6aaf4433eec5facb4a402e0966"} err="failed to get container status \"9cc66e3954463bba33cc2a02c6713fe917b7cc6aaf4433eec5facb4a402e0966\": rpc error: code = NotFound desc = could not find container \"9cc66e3954463bba33cc2a02c6713fe917b7cc6aaf4433eec5facb4a402e0966\": container with ID starting with 9cc66e3954463bba33cc2a02c6713fe917b7cc6aaf4433eec5facb4a402e0966 not found: ID does not exist" Jul 30 01:12:54.069268 ip-10-0-142-6 kubenswrapper[2564]: I0730 01:12:54.069183 2564 scope.go:117] "RemoveContainer" containerID="0ad0373533c170d1b3927f2d401f45e8f0525c8678eb823b8d4f0a43bd7e5b8c" Jul 30 01:12:54.077300 ip-10-0-142-6 kubenswrapper[2564]: I0730 01:12:54.077283 2564 scope.go:117] "RemoveContainer" containerID="0ad0373533c170d1b3927f2d401f45e8f0525c8678eb823b8d4f0a43bd7e5b8c" Jul 30 01:12:54.077545 ip-10-0-142-6 kubenswrapper[2564]: E0730 01:12:54.077527 2564 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0ad0373533c170d1b3927f2d401f45e8f0525c8678eb823b8d4f0a43bd7e5b8c\": container with ID starting with 0ad0373533c170d1b3927f2d401f45e8f0525c8678eb823b8d4f0a43bd7e5b8c not found: ID does not exist" containerID="0ad0373533c170d1b3927f2d401f45e8f0525c8678eb823b8d4f0a43bd7e5b8c" Jul 30 01:12:54.077589 ip-10-0-142-6 kubenswrapper[2564]: I0730 01:12:54.077553 2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0ad0373533c170d1b3927f2d401f45e8f0525c8678eb823b8d4f0a43bd7e5b8c"} err="failed to get container status \"0ad0373533c170d1b3927f2d401f45e8f0525c8678eb823b8d4f0a43bd7e5b8c\": rpc error: code = NotFound desc = could not find container \"0ad0373533c170d1b3927f2d401f45e8f0525c8678eb823b8d4f0a43bd7e5b8c\": container with ID starting with 0ad0373533c170d1b3927f2d401f45e8f0525c8678eb823b8d4f0a43bd7e5b8c not found: ID does not exist" Jul 30 01:12:54.083597 ip-10-0-142-6 kubenswrapper[2564]: I0730 01:12:54.083571 2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-47d629-76d8595b56-sf2ms"] Jul 30 01:12:54.085642 ip-10-0-142-6 kubenswrapper[2564]: I0730 01:12:54.085621 2564 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-47d629-76d8595b56-sf2ms"] Jul 30 01:12:54.097600 ip-10-0-142-6 kubenswrapper[2564]: I0730 01:12:54.097580 2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-47d629-58fcfd8d64-drzlv"] Jul 30 01:12:54.108130 ip-10-0-142-6 kubenswrapper[2564]: I0730 01:12:54.105065 2564 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-47d629-58fcfd8d64-drzlv"] Jul 30 01:12:55.139218 ip-10-0-142-6 kubenswrapper[2564]: I0730 01:12:55.139173 2564 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="1235e5ac-6ee9-4ae3-b5ca-fa824edb6227" path="/var/lib/kubelet/pods/1235e5ac-6ee9-4ae3-b5ca-fa824edb6227/volumes" Jul 30 01:12:55.139701 ip-10-0-142-6 kubenswrapper[2564]: I0730 01:12:55.139673 2564 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="65fecc99-42fa-4ab1-8215-d800172c1b0d" path="/var/lib/kubelet/pods/65fecc99-42fa-4ab1-8215-d800172c1b0d/volumes" Jul 30 01:12:56.909935 ip-10-0-142-6 kubenswrapper[2564]: I0730 01:12:56.909895 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-nrv2p"] Jul 30 01:13:00.056392 ip-10-0-142-6 kubenswrapper[2564]: I0730 01:13:00.050715 2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-b-47d629-openshift-default-746894d964-wplq8"] Jul 30 01:13:00.056392 ip-10-0-142-6 kubenswrapper[2564]: I0730 01:13:00.051116 2564 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-ipp-b-47d629-openshift-default-746894d964-wplq8" podUID="e9cd27ab-1777-4242-afb7-31edf86b4fa0" containerName="istio-proxy" containerID="cri-o://7b4e5fc7168ca87e194e9db93217d936e26992cfebc0327c2993ffca9c493d9c" gracePeriod=30 Jul 30 01:13:00.394203 ip-10-0-142-6 kubenswrapper[2564]: I0730 01:13:00.394156 2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-a47fe8-jv7bn"] Jul 30 01:13:00.394499 ip-10-0-142-6 kubenswrapper[2564]: I0730 01:13:00.394485 2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1235e5ac-6ee9-4ae3-b5ca-fa824edb6227" containerName="payload-pre-processing" Jul 30 01:13:00.394548 ip-10-0-142-6 kubenswrapper[2564]: I0730 01:13:00.394500 2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="1235e5ac-6ee9-4ae3-b5ca-fa824edb6227" containerName="payload-pre-processing" Jul 30 01:13:00.394548 ip-10-0-142-6 kubenswrapper[2564]: I0730 01:13:00.394513 2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="eb89fc81-7fdb-44f0-ab34-749e1d98c776" containerName="istio-proxy" Jul 30 01:13:00.394548 ip-10-0-142-6 kubenswrapper[2564]: I0730 01:13:00.394518 2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="eb89fc81-7fdb-44f0-ab34-749e1d98c776" containerName="istio-proxy" Jul 30 01:13:00.394548 ip-10-0-142-6 kubenswrapper[2564]: I0730 01:13:00.394526 2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="65fecc99-42fa-4ab1-8215-d800172c1b0d" containerName="payload-processing" Jul 30 01:13:00.394548 ip-10-0-142-6 kubenswrapper[2564]: I0730 01:13:00.394532 2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="65fecc99-42fa-4ab1-8215-d800172c1b0d" containerName="payload-processing" Jul 30 01:13:00.394548 ip-10-0-142-6 kubenswrapper[2564]: I0730 01:13:00.394540 2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="754f6d7c-9685-42de-b0df-bb2d658a6903" containerName="istio-proxy" Jul 30 01:13:00.394548 ip-10-0-142-6 kubenswrapper[2564]: I0730 01:13:00.394545 2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="754f6d7c-9685-42de-b0df-bb2d658a6903" containerName="istio-proxy" Jul 30 01:13:00.394750 ip-10-0-142-6 kubenswrapper[2564]: I0730 01:13:00.394553 2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="754f6d7c-9685-42de-b0df-bb2d658a6903" containerName="istio-proxy" Jul 30 01:13:00.394750 ip-10-0-142-6 kubenswrapper[2564]: I0730 01:13:00.394559 2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="754f6d7c-9685-42de-b0df-bb2d658a6903" containerName="istio-proxy" Jul 30 01:13:00.394750 ip-10-0-142-6 kubenswrapper[2564]: I0730 01:13:00.394568 2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="eb89fc81-7fdb-44f0-ab34-749e1d98c776" containerName="istio-proxy" Jul 30 01:13:00.394750 ip-10-0-142-6 kubenswrapper[2564]: I0730 01:13:00.394573 2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="eb89fc81-7fdb-44f0-ab34-749e1d98c776" containerName="istio-proxy" Jul 30 01:13:00.394750 ip-10-0-142-6 kubenswrapper[2564]: I0730 01:13:00.394584 2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5e58f698-a89d-48bd-86fb-673fe182dcfa" containerName="payload-processing" Jul 30 01:13:00.394750 ip-10-0-142-6 kubenswrapper[2564]: I0730 01:13:00.394590 2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="5e58f698-a89d-48bd-86fb-673fe182dcfa" containerName="payload-processing" Jul 30 01:13:00.394750 ip-10-0-142-6 kubenswrapper[2564]: I0730 01:13:00.394596 2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="74425d91-a03b-44a2-a447-a20d426d665d" containerName="revoke-keys" Jul 30 01:13:00.394750 ip-10-0-142-6 kubenswrapper[2564]: I0730 01:13:00.394602 2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="74425d91-a03b-44a2-a447-a20d426d665d" containerName="revoke-keys" Jul 30 01:13:00.394750 ip-10-0-142-6 kubenswrapper[2564]: I0730 01:13:00.394651 2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="1235e5ac-6ee9-4ae3-b5ca-fa824edb6227" containerName="payload-pre-processing" Jul 30 01:13:00.394750 ip-10-0-142-6 kubenswrapper[2564]: I0730 01:13:00.394658 2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="5e58f698-a89d-48bd-86fb-673fe182dcfa" containerName="payload-processing" Jul 30 01:13:00.394750 ip-10-0-142-6 kubenswrapper[2564]: I0730 01:13:00.394665 2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="74425d91-a03b-44a2-a447-a20d426d665d" containerName="revoke-keys" Jul 30 01:13:00.394750 ip-10-0-142-6 kubenswrapper[2564]: I0730 01:13:00.394670 2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="eb89fc81-7fdb-44f0-ab34-749e1d98c776" containerName="istio-proxy" Jul 30 01:13:00.394750 ip-10-0-142-6 kubenswrapper[2564]: I0730 01:13:00.394674 2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="eb89fc81-7fdb-44f0-ab34-749e1d98c776" containerName="istio-proxy" Jul 30 01:13:00.394750 ip-10-0-142-6 kubenswrapper[2564]: I0730 01:13:00.394679 2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="65fecc99-42fa-4ab1-8215-d800172c1b0d" containerName="payload-processing" Jul 30 01:13:00.394750 ip-10-0-142-6 kubenswrapper[2564]: I0730 01:13:00.394684 2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="754f6d7c-9685-42de-b0df-bb2d658a6903" containerName="istio-proxy" Jul 30 01:13:00.394750 ip-10-0-142-6 kubenswrapper[2564]: I0730 01:13:00.394689 2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="754f6d7c-9685-42de-b0df-bb2d658a6903" containerName="istio-proxy" Jul 30 01:13:00.399234 ip-10-0-142-6 kubenswrapper[2564]: I0730 01:13:00.399216 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-a47fe8-jv7bn" Jul 30 01:13:00.401806 ip-10-0-142-6 kubenswrapper[2564]: I0730 01:13:00.401782 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Jul 30 01:13:00.401972 ip-10-0-142-6 kubenswrapper[2564]: I0730 01:13:00.401941 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-v8q7j\"" Jul 30 01:13:00.405335 ip-10-0-142-6 kubenswrapper[2564]: I0730 01:13:00.405312 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-a47fe8-jv7bn"] Jul 30 01:13:00.473715 ip-10-0-142-6 kubenswrapper[2564]: I0730 01:13:00.473680 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d07a5d88-d062-4aa7-a572-e567963d84ce-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-a47fe8-jv7bn\" (UID: \"d07a5d88-d062-4aa7-a572-e567963d84ce\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-a47fe8-jv7bn" Jul 30 01:13:00.575033 ip-10-0-142-6 kubenswrapper[2564]: I0730 01:13:00.574989 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d07a5d88-d062-4aa7-a572-e567963d84ce-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-a47fe8-jv7bn\" (UID: \"d07a5d88-d062-4aa7-a572-e567963d84ce\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-a47fe8-jv7bn" Jul 30 01:13:00.575687 ip-10-0-142-6 kubenswrapper[2564]: I0730 01:13:00.575663 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d07a5d88-d062-4aa7-a572-e567963d84ce-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-a47fe8-jv7bn\" (UID: \"d07a5d88-d062-4aa7-a572-e567963d84ce\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-a47fe8-jv7bn" Jul 30 01:13:00.710390 ip-10-0-142-6 kubenswrapper[2564]: I0730 01:13:00.710302 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-a47fe8-jv7bn" Jul 30 01:13:00.841607 ip-10-0-142-6 kubenswrapper[2564]: I0730 01:13:00.841582 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-a47fe8-jv7bn"] Jul 30 01:13:00.843253 ip-10-0-142-6 kubenswrapper[2564]: W0730 01:13:00.843224 2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd07a5d88_d062_4aa7_a572_e567963d84ce.slice/crio-2bee7f4e5290840294a200e7568da4f5066d8c22e05e6ec9cd55ee1ac140dd6e WatchSource:0}: Error finding container 2bee7f4e5290840294a200e7568da4f5066d8c22e05e6ec9cd55ee1ac140dd6e: Status 404 returned error can't find the container with id 2bee7f4e5290840294a200e7568da4f5066d8c22e05e6ec9cd55ee1ac140dd6e Jul 30 01:13:01.100009 ip-10-0-142-6 kubenswrapper[2564]: I0730 01:13:01.099974 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-a47fe8-jv7bn" event={"ID":"d07a5d88-d062-4aa7-a572-e567963d84ce","Type":"ContainerStarted","Data":"9b07123293bb8f57d3fd6cc63717f6549b4c894e41ceb417ca0a5dde84a800bf"} Jul 30 01:13:01.100414 ip-10-0-142-6 kubenswrapper[2564]: I0730 01:13:01.100029 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-a47fe8-jv7bn" event={"ID":"d07a5d88-d062-4aa7-a572-e567963d84ce","Type":"ContainerStarted","Data":"2bee7f4e5290840294a200e7568da4f5066d8c22e05e6ec9cd55ee1ac140dd6e"} Jul 30 01:13:01.115046 ip-10-0-142-6 kubenswrapper[2564]: I0730 01:13:01.114993 2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-a47fe8-jv7bn" podStartSLOduration=1.1149759 podStartE2EDuration="1.1149759s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-30 01:13:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-30 01:13:01.113556031 +0000 UTC m=+3312.528295966" watchObservedRunningTime="2026-07-30 01:13:01.1149759 +0000 UTC m=+3312.529715829" Jul 30 01:13:02.106060 ip-10-0-142-6 kubenswrapper[2564]: I0730 01:13:02.106028 2564 generic.go:356] "Generic (PLEG): container finished" podID="d07a5d88-d062-4aa7-a572-e567963d84ce" containerID="9b07123293bb8f57d3fd6cc63717f6549b4c894e41ceb417ca0a5dde84a800bf" exitCode=0 Jul 30 01:13:02.106411 ip-10-0-142-6 kubenswrapper[2564]: I0730 01:13:02.106118 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-a47fe8-jv7bn" event={"ID":"d07a5d88-d062-4aa7-a572-e567963d84ce","Type":"ContainerDied","Data":"9b07123293bb8f57d3fd6cc63717f6549b4c894e41ceb417ca0a5dde84a800bf"} Jul 30 01:13:03.245666 ip-10-0-142-6 kubenswrapper[2564]: I0730 01:13:03.245639 2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-a47fe8-jv7bn" Jul 30 01:13:03.398706 ip-10-0-142-6 kubenswrapper[2564]: I0730 01:13:03.398671 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d07a5d88-d062-4aa7-a572-e567963d84ce-maas-api-service-ca\") pod \"d07a5d88-d062-4aa7-a572-e567963d84ce\" (UID: \"d07a5d88-d062-4aa7-a572-e567963d84ce\") " Jul 30 01:13:03.399105 ip-10-0-142-6 kubenswrapper[2564]: I0730 01:13:03.399046 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d07a5d88-d062-4aa7-a572-e567963d84ce-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "d07a5d88-d062-4aa7-a572-e567963d84ce" (UID: "d07a5d88-d062-4aa7-a572-e567963d84ce"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 30 01:13:03.499726 ip-10-0-142-6 kubenswrapper[2564]: I0730 01:13:03.499685 2564 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d07a5d88-d062-4aa7-a572-e567963d84ce-maas-api-service-ca\") on node \"ip-10-0-142-6.ec2.internal\" DevicePath \"\"" Jul 30 01:13:04.117884 ip-10-0-142-6 kubenswrapper[2564]: I0730 01:13:04.117848 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-a47fe8-jv7bn" event={"ID":"d07a5d88-d062-4aa7-a572-e567963d84ce","Type":"ContainerDied","Data":"2bee7f4e5290840294a200e7568da4f5066d8c22e05e6ec9cd55ee1ac140dd6e"} Jul 30 01:13:04.118040 ip-10-0-142-6 kubenswrapper[2564]: I0730 01:13:04.117899 2564 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="2bee7f4e5290840294a200e7568da4f5066d8c22e05e6ec9cd55ee1ac140dd6e" Jul 30 01:13:04.118040 ip-10-0-142-6 kubenswrapper[2564]: I0730 01:13:04.117858 2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-a47fe8-jv7bn" Jul 30 01:13:04.862657 ip-10-0-142-6 kubenswrapper[2564]: I0730 01:13:04.862604 2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-a47fe8-29756220-nn56f"] Jul 30 01:13:04.864942 ip-10-0-142-6 kubenswrapper[2564]: I0730 01:13:04.864906 2564 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-a47fe8-29756220-nn56f"] Jul 30 01:13:05.138808 ip-10-0-142-6 kubenswrapper[2564]: I0730 01:13:05.138780 2564 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="6972ad1f-97e5-496b-ac48-b451868cc732" path="/var/lib/kubelet/pods/6972ad1f-97e5-496b-ac48-b451868cc732/volumes" Jul 30 01:13:05.167007 ip-10-0-142-6 kubenswrapper[2564]: I0730 01:13:05.166970 2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-a47fe8-jv7bn"] Jul 30 01:13:05.169469 ip-10-0-142-6 kubenswrapper[2564]: I0730 01:13:05.169443 2564 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-a47fe8-jv7bn"] Jul 30 01:13:05.310986 ip-10-0-142-6 kubenswrapper[2564]: I0730 01:13:05.310959 2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-47d629-openshift-default-746894d964-wplq8" Jul 30 01:13:05.420348 ip-10-0-142-6 kubenswrapper[2564]: I0730 01:13:05.420266 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e9cd27ab-1777-4242-afb7-31edf86b4fa0-istio-data\") pod \"e9cd27ab-1777-4242-afb7-31edf86b4fa0\" (UID: \"e9cd27ab-1777-4242-afb7-31edf86b4fa0\") " Jul 30 01:13:05.420348 ip-10-0-142-6 kubenswrapper[2564]: I0730 01:13:05.420310 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e9cd27ab-1777-4242-afb7-31edf86b4fa0-istio-envoy\") pod \"e9cd27ab-1777-4242-afb7-31edf86b4fa0\" (UID: \"e9cd27ab-1777-4242-afb7-31edf86b4fa0\") " Jul 30 01:13:05.420348 ip-10-0-142-6 kubenswrapper[2564]: I0730 01:13:05.420329 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e9cd27ab-1777-4242-afb7-31edf86b4fa0-istiod-ca-cert\") pod \"e9cd27ab-1777-4242-afb7-31edf86b4fa0\" (UID: \"e9cd27ab-1777-4242-afb7-31edf86b4fa0\") " Jul 30 01:13:05.420632 ip-10-0-142-6 kubenswrapper[2564]: I0730 01:13:05.420388 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e9cd27ab-1777-4242-afb7-31edf86b4fa0-istio-podinfo\") pod \"e9cd27ab-1777-4242-afb7-31edf86b4fa0\" (UID: \"e9cd27ab-1777-4242-afb7-31edf86b4fa0\") " Jul 30 01:13:05.420632 ip-10-0-142-6 kubenswrapper[2564]: I0730 01:13:05.420404 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e9cd27ab-1777-4242-afb7-31edf86b4fa0-credential-socket\") pod \"e9cd27ab-1777-4242-afb7-31edf86b4fa0\" (UID: \"e9cd27ab-1777-4242-afb7-31edf86b4fa0\") " Jul 30 01:13:05.420632 ip-10-0-142-6 kubenswrapper[2564]: I0730 01:13:05.420420 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e9cd27ab-1777-4242-afb7-31edf86b4fa0-workload-certs\") pod \"e9cd27ab-1777-4242-afb7-31edf86b4fa0\" (UID: \"e9cd27ab-1777-4242-afb7-31edf86b4fa0\") " Jul 30 01:13:05.420632 ip-10-0-142-6 kubenswrapper[2564]: I0730 01:13:05.420447 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-cnbht\" (UniqueName: \"kubernetes.io/projected/e9cd27ab-1777-4242-afb7-31edf86b4fa0-kube-api-access-cnbht\") pod \"e9cd27ab-1777-4242-afb7-31edf86b4fa0\" (UID: \"e9cd27ab-1777-4242-afb7-31edf86b4fa0\") " Jul 30 01:13:05.420632 ip-10-0-142-6 kubenswrapper[2564]: I0730 01:13:05.420489 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e9cd27ab-1777-4242-afb7-31edf86b4fa0-istio-token\") pod \"e9cd27ab-1777-4242-afb7-31edf86b4fa0\" (UID: \"e9cd27ab-1777-4242-afb7-31edf86b4fa0\") " Jul 30 01:13:05.420632 ip-10-0-142-6 kubenswrapper[2564]: I0730 01:13:05.420542 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e9cd27ab-1777-4242-afb7-31edf86b4fa0-workload-socket\") pod \"e9cd27ab-1777-4242-afb7-31edf86b4fa0\" (UID: \"e9cd27ab-1777-4242-afb7-31edf86b4fa0\") " Jul 30 01:13:05.420999 ip-10-0-142-6 kubenswrapper[2564]: I0730 01:13:05.420749 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e9cd27ab-1777-4242-afb7-31edf86b4fa0-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "e9cd27ab-1777-4242-afb7-31edf86b4fa0" (UID: "e9cd27ab-1777-4242-afb7-31edf86b4fa0"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 30 01:13:05.420999 ip-10-0-142-6 kubenswrapper[2564]: I0730 01:13:05.420834 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e9cd27ab-1777-4242-afb7-31edf86b4fa0-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "e9cd27ab-1777-4242-afb7-31edf86b4fa0" (UID: "e9cd27ab-1777-4242-afb7-31edf86b4fa0"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 30 01:13:05.420999 ip-10-0-142-6 kubenswrapper[2564]: I0730 01:13:05.420855 2564 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e9cd27ab-1777-4242-afb7-31edf86b4fa0-workload-certs\") on node \"ip-10-0-142-6.ec2.internal\" DevicePath \"\"" Jul 30 01:13:05.420999 ip-10-0-142-6 kubenswrapper[2564]: I0730 01:13:05.420974 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e9cd27ab-1777-4242-afb7-31edf86b4fa0-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "e9cd27ab-1777-4242-afb7-31edf86b4fa0" (UID: "e9cd27ab-1777-4242-afb7-31edf86b4fa0"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 30 01:13:05.420999 ip-10-0-142-6 kubenswrapper[2564]: I0730 01:13:05.420984 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e9cd27ab-1777-4242-afb7-31edf86b4fa0-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "e9cd27ab-1777-4242-afb7-31edf86b4fa0" (UID: "e9cd27ab-1777-4242-afb7-31edf86b4fa0"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 30 01:13:05.428400 ip-10-0-142-6 kubenswrapper[2564]: I0730 01:13:05.421641 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e9cd27ab-1777-4242-afb7-31edf86b4fa0-istio-data" (OuterVolumeSpecName: "istio-data") pod "e9cd27ab-1777-4242-afb7-31edf86b4fa0" (UID: "e9cd27ab-1777-4242-afb7-31edf86b4fa0"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 30 01:13:05.428400 ip-10-0-142-6 kubenswrapper[2564]: I0730 01:13:05.422983 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e9cd27ab-1777-4242-afb7-31edf86b4fa0-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "e9cd27ab-1777-4242-afb7-31edf86b4fa0" (UID: "e9cd27ab-1777-4242-afb7-31edf86b4fa0"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 30 01:13:05.428400 ip-10-0-142-6 kubenswrapper[2564]: I0730 01:13:05.423093 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e9cd27ab-1777-4242-afb7-31edf86b4fa0-kube-api-access-cnbht" (OuterVolumeSpecName: "kube-api-access-cnbht") pod "e9cd27ab-1777-4242-afb7-31edf86b4fa0" (UID: "e9cd27ab-1777-4242-afb7-31edf86b4fa0"). InnerVolumeSpecName "kube-api-access-cnbht". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 30 01:13:05.428400 ip-10-0-142-6 kubenswrapper[2564]: I0730 01:13:05.423409 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/e9cd27ab-1777-4242-afb7-31edf86b4fa0-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "e9cd27ab-1777-4242-afb7-31edf86b4fa0" (UID: "e9cd27ab-1777-4242-afb7-31edf86b4fa0"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Jul 30 01:13:05.428400 ip-10-0-142-6 kubenswrapper[2564]: I0730 01:13:05.423525 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e9cd27ab-1777-4242-afb7-31edf86b4fa0-istio-token" (OuterVolumeSpecName: "istio-token") pod "e9cd27ab-1777-4242-afb7-31edf86b4fa0" (UID: "e9cd27ab-1777-4242-afb7-31edf86b4fa0"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 30 01:13:05.521350 ip-10-0-142-6 kubenswrapper[2564]: I0730 01:13:05.521309 2564 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e9cd27ab-1777-4242-afb7-31edf86b4fa0-workload-socket\") on node \"ip-10-0-142-6.ec2.internal\" DevicePath \"\"" Jul 30 01:13:05.521350 ip-10-0-142-6 kubenswrapper[2564]: I0730 01:13:05.521343 2564 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e9cd27ab-1777-4242-afb7-31edf86b4fa0-istio-data\") on node \"ip-10-0-142-6.ec2.internal\" DevicePath \"\"" Jul 30 01:13:05.521350 ip-10-0-142-6 kubenswrapper[2564]: I0730 01:13:05.521351 2564 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e9cd27ab-1777-4242-afb7-31edf86b4fa0-istio-envoy\") on node \"ip-10-0-142-6.ec2.internal\" DevicePath \"\"" Jul 30 01:13:05.521350 ip-10-0-142-6 kubenswrapper[2564]: I0730 01:13:05.521359 2564 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e9cd27ab-1777-4242-afb7-31edf86b4fa0-istiod-ca-cert\") on node \"ip-10-0-142-6.ec2.internal\" DevicePath \"\"" Jul 30 01:13:05.521618 ip-10-0-142-6 kubenswrapper[2564]: I0730 01:13:05.521368 2564 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e9cd27ab-1777-4242-afb7-31edf86b4fa0-istio-podinfo\") on node \"ip-10-0-142-6.ec2.internal\" DevicePath \"\"" Jul 30 01:13:05.521618 ip-10-0-142-6 kubenswrapper[2564]: I0730 01:13:05.521376 2564 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e9cd27ab-1777-4242-afb7-31edf86b4fa0-credential-socket\") on node \"ip-10-0-142-6.ec2.internal\" DevicePath \"\"" Jul 30 01:13:05.521618 ip-10-0-142-6 kubenswrapper[2564]: I0730 01:13:05.521386 2564 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cnbht\" (UniqueName: \"kubernetes.io/projected/e9cd27ab-1777-4242-afb7-31edf86b4fa0-kube-api-access-cnbht\") on node \"ip-10-0-142-6.ec2.internal\" DevicePath \"\"" Jul 30 01:13:05.521618 ip-10-0-142-6 kubenswrapper[2564]: I0730 01:13:05.521395 2564 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e9cd27ab-1777-4242-afb7-31edf86b4fa0-istio-token\") on node \"ip-10-0-142-6.ec2.internal\" DevicePath \"\"" Jul 30 01:13:06.129574 ip-10-0-142-6 kubenswrapper[2564]: I0730 01:13:06.129537 2564 generic.go:356] "Generic (PLEG): container finished" podID="e9cd27ab-1777-4242-afb7-31edf86b4fa0" containerID="7b4e5fc7168ca87e194e9db93217d936e26992cfebc0327c2993ffca9c493d9c" exitCode=0 Jul 30 01:13:06.130057 ip-10-0-142-6 kubenswrapper[2564]: I0730 01:13:06.129581 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-47d629-openshift-default-746894d964-wplq8" event={"ID":"e9cd27ab-1777-4242-afb7-31edf86b4fa0","Type":"ContainerDied","Data":"7b4e5fc7168ca87e194e9db93217d936e26992cfebc0327c2993ffca9c493d9c"} Jul 30 01:13:06.130057 ip-10-0-142-6 kubenswrapper[2564]: I0730 01:13:06.129603 2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-47d629-openshift-default-746894d964-wplq8" Jul 30 01:13:06.130057 ip-10-0-142-6 kubenswrapper[2564]: I0730 01:13:06.129668 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-47d629-openshift-default-746894d964-wplq8" event={"ID":"e9cd27ab-1777-4242-afb7-31edf86b4fa0","Type":"ContainerDied","Data":"6c321708ea15bbc1d160b068d883038a94fbcf422625ee80b4fc9289aa38b5dc"} Jul 30 01:13:06.130057 ip-10-0-142-6 kubenswrapper[2564]: I0730 01:13:06.129693 2564 scope.go:117] "RemoveContainer" containerID="7b4e5fc7168ca87e194e9db93217d936e26992cfebc0327c2993ffca9c493d9c" Jul 30 01:13:06.141297 ip-10-0-142-6 kubenswrapper[2564]: I0730 01:13:06.141272 2564 scope.go:117] "RemoveContainer" containerID="0da6b45169c01b8a0b7d8dd44ec273acfc576c271e02e4ad1ea3177125f86007" Jul 30 01:13:06.149804 ip-10-0-142-6 kubenswrapper[2564]: I0730 01:13:06.149788 2564 scope.go:117] "RemoveContainer" containerID="7b4e5fc7168ca87e194e9db93217d936e26992cfebc0327c2993ffca9c493d9c" Jul 30 01:13:06.150041 ip-10-0-142-6 kubenswrapper[2564]: E0730 01:13:06.150022 2564 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7b4e5fc7168ca87e194e9db93217d936e26992cfebc0327c2993ffca9c493d9c\": container with ID starting with 7b4e5fc7168ca87e194e9db93217d936e26992cfebc0327c2993ffca9c493d9c not found: ID does not exist" containerID="7b4e5fc7168ca87e194e9db93217d936e26992cfebc0327c2993ffca9c493d9c" Jul 30 01:13:06.150100 ip-10-0-142-6 kubenswrapper[2564]: I0730 01:13:06.150048 2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7b4e5fc7168ca87e194e9db93217d936e26992cfebc0327c2993ffca9c493d9c"} err="failed to get container status \"7b4e5fc7168ca87e194e9db93217d936e26992cfebc0327c2993ffca9c493d9c\": rpc error: code = NotFound desc = could not find container \"7b4e5fc7168ca87e194e9db93217d936e26992cfebc0327c2993ffca9c493d9c\": container with ID starting with 7b4e5fc7168ca87e194e9db93217d936e26992cfebc0327c2993ffca9c493d9c not found: ID does not exist" Jul 30 01:13:06.150100 ip-10-0-142-6 kubenswrapper[2564]: I0730 01:13:06.150063 2564 scope.go:117] "RemoveContainer" containerID="0da6b45169c01b8a0b7d8dd44ec273acfc576c271e02e4ad1ea3177125f86007" Jul 30 01:13:06.150297 ip-10-0-142-6 kubenswrapper[2564]: E0730 01:13:06.150277 2564 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0da6b45169c01b8a0b7d8dd44ec273acfc576c271e02e4ad1ea3177125f86007\": container with ID starting with 0da6b45169c01b8a0b7d8dd44ec273acfc576c271e02e4ad1ea3177125f86007 not found: ID does not exist" containerID="0da6b45169c01b8a0b7d8dd44ec273acfc576c271e02e4ad1ea3177125f86007" Jul 30 01:13:06.150363 ip-10-0-142-6 kubenswrapper[2564]: I0730 01:13:06.150306 2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0da6b45169c01b8a0b7d8dd44ec273acfc576c271e02e4ad1ea3177125f86007"} err="failed to get container status \"0da6b45169c01b8a0b7d8dd44ec273acfc576c271e02e4ad1ea3177125f86007\": rpc error: code = NotFound desc = could not find container \"0da6b45169c01b8a0b7d8dd44ec273acfc576c271e02e4ad1ea3177125f86007\": container with ID starting with 0da6b45169c01b8a0b7d8dd44ec273acfc576c271e02e4ad1ea3177125f86007 not found: ID does not exist" Jul 30 01:13:06.153505 ip-10-0-142-6 kubenswrapper[2564]: I0730 01:13:06.153481 2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-b-47d629-openshift-default-746894d964-wplq8"] Jul 30 01:13:06.159465 ip-10-0-142-6 kubenswrapper[2564]: I0730 01:13:06.159444 2564 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ipp-b-47d629-openshift-default-746894d964-wplq8"] Jul 30 01:13:07.139803 ip-10-0-142-6 kubenswrapper[2564]: I0730 01:13:07.139763 2564 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="d07a5d88-d062-4aa7-a572-e567963d84ce" path="/var/lib/kubelet/pods/d07a5d88-d062-4aa7-a572-e567963d84ce/volumes" Jul 30 01:13:07.140373 ip-10-0-142-6 kubenswrapper[2564]: I0730 01:13:07.140353 2564 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="e9cd27ab-1777-4242-afb7-31edf86b4fa0" path="/var/lib/kubelet/pods/e9cd27ab-1777-4242-afb7-31edf86b4fa0/volumes" Jul 30 01:13:10.616381 ip-10-0-142-6 kubenswrapper[2564]: I0730 01:13:10.616299 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-nrv2p"] Jul 30 01:13:12.278582 ip-10-0-142-6 kubenswrapper[2564]: I0730 01:13:12.278537 2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-c4c410-d4kkh"] Jul 30 01:13:12.278984 ip-10-0-142-6 kubenswrapper[2564]: I0730 01:13:12.278891 2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e9cd27ab-1777-4242-afb7-31edf86b4fa0" containerName="istio-proxy" Jul 30 01:13:12.278984 ip-10-0-142-6 kubenswrapper[2564]: I0730 01:13:12.278907 2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="e9cd27ab-1777-4242-afb7-31edf86b4fa0" containerName="istio-proxy" Jul 30 01:13:12.278984 ip-10-0-142-6 kubenswrapper[2564]: I0730 01:13:12.278926 2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d07a5d88-d062-4aa7-a572-e567963d84ce" containerName="revoke-keys" Jul 30 01:13:12.278984 ip-10-0-142-6 kubenswrapper[2564]: I0730 01:13:12.278932 2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="d07a5d88-d062-4aa7-a572-e567963d84ce" containerName="revoke-keys" Jul 30 01:13:12.279144 ip-10-0-142-6 kubenswrapper[2564]: I0730 01:13:12.278989 2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="d07a5d88-d062-4aa7-a572-e567963d84ce" containerName="revoke-keys" Jul 30 01:13:12.279144 ip-10-0-142-6 kubenswrapper[2564]: I0730 01:13:12.278997 2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="e9cd27ab-1777-4242-afb7-31edf86b4fa0" containerName="istio-proxy" Jul 30 01:13:12.279144 ip-10-0-142-6 kubenswrapper[2564]: I0730 01:13:12.279003 2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="e9cd27ab-1777-4242-afb7-31edf86b4fa0" containerName="istio-proxy" Jul 30 01:13:12.283603 ip-10-0-142-6 kubenswrapper[2564]: I0730 01:13:12.283588 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-c4c410-d4kkh" Jul 30 01:13:12.286044 ip-10-0-142-6 kubenswrapper[2564]: I0730 01:13:12.286013 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-v8q7j\"" Jul 30 01:13:12.286207 ip-10-0-142-6 kubenswrapper[2564]: I0730 01:13:12.286099 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Jul 30 01:13:12.290110 ip-10-0-142-6 kubenswrapper[2564]: I0730 01:13:12.290068 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-c4c410-d4kkh"] Jul 30 01:13:12.382174 ip-10-0-142-6 kubenswrapper[2564]: I0730 01:13:12.382141 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d8a2c8f9-1aea-4356-bf0d-ad50e738f9b7-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-c4c410-d4kkh\" (UID: \"d8a2c8f9-1aea-4356-bf0d-ad50e738f9b7\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-c4c410-d4kkh" Jul 30 01:13:12.483286 ip-10-0-142-6 kubenswrapper[2564]: I0730 01:13:12.483254 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d8a2c8f9-1aea-4356-bf0d-ad50e738f9b7-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-c4c410-d4kkh\" (UID: \"d8a2c8f9-1aea-4356-bf0d-ad50e738f9b7\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-c4c410-d4kkh" Jul 30 01:13:12.483873 ip-10-0-142-6 kubenswrapper[2564]: I0730 01:13:12.483854 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d8a2c8f9-1aea-4356-bf0d-ad50e738f9b7-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-c4c410-d4kkh\" (UID: \"d8a2c8f9-1aea-4356-bf0d-ad50e738f9b7\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-c4c410-d4kkh" Jul 30 01:13:12.595120 ip-10-0-142-6 kubenswrapper[2564]: I0730 01:13:12.595023 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-c4c410-d4kkh" Jul 30 01:13:12.727966 ip-10-0-142-6 kubenswrapper[2564]: I0730 01:13:12.727842 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-c4c410-d4kkh"] Jul 30 01:13:12.730165 ip-10-0-142-6 kubenswrapper[2564]: W0730 01:13:12.730111 2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd8a2c8f9_1aea_4356_bf0d_ad50e738f9b7.slice/crio-6a51663ed13882b9f6cb9b8926a5bc2fe9c16d58dc9c50c0281e1232f790026e WatchSource:0}: Error finding container 6a51663ed13882b9f6cb9b8926a5bc2fe9c16d58dc9c50c0281e1232f790026e: Status 404 returned error can't find the container with id 6a51663ed13882b9f6cb9b8926a5bc2fe9c16d58dc9c50c0281e1232f790026e Jul 30 01:13:13.166233 ip-10-0-142-6 kubenswrapper[2564]: I0730 01:13:13.166197 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-c4c410-d4kkh" event={"ID":"d8a2c8f9-1aea-4356-bf0d-ad50e738f9b7","Type":"ContainerStarted","Data":"76155e1a4d5a09dbdccdfcd78a449df7e9059181d86da72904d8ab2a40a60f88"} Jul 30 01:13:13.166437 ip-10-0-142-6 kubenswrapper[2564]: I0730 01:13:13.166248 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-c4c410-d4kkh" event={"ID":"d8a2c8f9-1aea-4356-bf0d-ad50e738f9b7","Type":"ContainerStarted","Data":"6a51663ed13882b9f6cb9b8926a5bc2fe9c16d58dc9c50c0281e1232f790026e"} Jul 30 01:13:14.173129 ip-10-0-142-6 kubenswrapper[2564]: I0730 01:13:14.173066 2564 generic.go:356] "Generic (PLEG): container finished" podID="d8a2c8f9-1aea-4356-bf0d-ad50e738f9b7" containerID="76155e1a4d5a09dbdccdfcd78a449df7e9059181d86da72904d8ab2a40a60f88" exitCode=0 Jul 30 01:13:14.173129 ip-10-0-142-6 kubenswrapper[2564]: I0730 01:13:14.173109 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-c4c410-d4kkh" event={"ID":"d8a2c8f9-1aea-4356-bf0d-ad50e738f9b7","Type":"ContainerDied","Data":"76155e1a4d5a09dbdccdfcd78a449df7e9059181d86da72904d8ab2a40a60f88"} Jul 30 01:13:15.308839 ip-10-0-142-6 kubenswrapper[2564]: I0730 01:13:15.308811 2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-c4c410-d4kkh" Jul 30 01:13:15.409020 ip-10-0-142-6 kubenswrapper[2564]: I0730 01:13:15.408985 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d8a2c8f9-1aea-4356-bf0d-ad50e738f9b7-maas-api-service-ca\") pod \"d8a2c8f9-1aea-4356-bf0d-ad50e738f9b7\" (UID: \"d8a2c8f9-1aea-4356-bf0d-ad50e738f9b7\") " Jul 30 01:13:15.409387 ip-10-0-142-6 kubenswrapper[2564]: I0730 01:13:15.409366 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d8a2c8f9-1aea-4356-bf0d-ad50e738f9b7-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "d8a2c8f9-1aea-4356-bf0d-ad50e738f9b7" (UID: "d8a2c8f9-1aea-4356-bf0d-ad50e738f9b7"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 30 01:13:15.510392 ip-10-0-142-6 kubenswrapper[2564]: I0730 01:13:15.510300 2564 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d8a2c8f9-1aea-4356-bf0d-ad50e738f9b7-maas-api-service-ca\") on node \"ip-10-0-142-6.ec2.internal\" DevicePath \"\"" Jul 30 01:13:16.185732 ip-10-0-142-6 kubenswrapper[2564]: I0730 01:13:16.185700 2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-c4c410-d4kkh" Jul 30 01:13:16.185938 ip-10-0-142-6 kubenswrapper[2564]: I0730 01:13:16.185742 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-c4c410-d4kkh" event={"ID":"d8a2c8f9-1aea-4356-bf0d-ad50e738f9b7","Type":"ContainerDied","Data":"6a51663ed13882b9f6cb9b8926a5bc2fe9c16d58dc9c50c0281e1232f790026e"} Jul 30 01:13:16.185938 ip-10-0-142-6 kubenswrapper[2564]: I0730 01:13:16.185793 2564 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="6a51663ed13882b9f6cb9b8926a5bc2fe9c16d58dc9c50c0281e1232f790026e" Jul 30 01:13:17.533203 ip-10-0-142-6 kubenswrapper[2564]: I0730 01:13:17.533167 2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-c4c410-29756220-g8hvx"] Jul 30 01:13:17.535197 ip-10-0-142-6 kubenswrapper[2564]: I0730 01:13:17.535176 2564 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-c4c410-29756220-g8hvx"] Jul 30 01:13:17.752508 ip-10-0-142-6 kubenswrapper[2564]: I0730 01:13:17.752476 2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-c4c410-d4kkh"] Jul 30 01:13:17.755182 ip-10-0-142-6 kubenswrapper[2564]: I0730 01:13:17.755157 2564 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-c4c410-d4kkh"] Jul 30 01:13:19.139371 ip-10-0-142-6 kubenswrapper[2564]: I0730 01:13:19.139341 2564 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="c5191b01-4f68-4c40-b828-341dbb28b89d" path="/var/lib/kubelet/pods/c5191b01-4f68-4c40-b828-341dbb28b89d/volumes" Jul 30 01:13:19.139751 ip-10-0-142-6 kubenswrapper[2564]: I0730 01:13:19.139627 2564 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="d8a2c8f9-1aea-4356-bf0d-ad50e738f9b7" path="/var/lib/kubelet/pods/d8a2c8f9-1aea-4356-bf0d-ad50e738f9b7/volumes" Jul 30 01:13:21.630137 ip-10-0-142-6 kubenswrapper[2564]: I0730 01:13:21.630100 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-nrv2p"] Jul 30 01:13:30.252527 ip-10-0-142-6 kubenswrapper[2564]: I0730 01:13:30.252499 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_kserve-controller-manager-d6f4b65b8-bwrb2_c998e5e8-aa41-4da6-a8a8-94b49e8f99fb/manager/0.log" Jul 30 01:13:31.001387 ip-10-0-142-6 kubenswrapper[2564]: I0730 01:13:31.001351 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-5b74bc5486-thrgp_db0f2405-58a7-4112-af82-65d1d07bb5a9/copy-manifests/0.log" Jul 30 01:13:31.008507 ip-10-0-142-6 kubenswrapper[2564]: I0730 01:13:31.008477 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-5b74bc5486-thrgp_db0f2405-58a7-4112-af82-65d1d07bb5a9/manager/0.log" Jul 30 01:13:31.771170 ip-10-0-142-6 kubenswrapper[2564]: I0730 01:13:31.771139 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-key-cleanup-29756220-xz969_f0fea26f-8c9f-4c0a-bd82-4231140ea823/cleanup/0.log" Jul 30 01:13:32.599864 ip-10-0-142-6 kubenswrapper[2564]: I0730 01:13:32.599839 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759t2h7b_f546aac1-87f6-499c-98c8-02afb9ae43f6/util/0.log" Jul 30 01:13:32.605798 ip-10-0-142-6 kubenswrapper[2564]: I0730 01:13:32.605779 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759t2h7b_f546aac1-87f6-499c-98c8-02afb9ae43f6/pull/0.log" Jul 30 01:13:32.611724 ip-10-0-142-6 kubenswrapper[2564]: I0730 01:13:32.611703 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759t2h7b_f546aac1-87f6-499c-98c8-02afb9ae43f6/extract/0.log" Jul 30 01:13:32.718733 ip-10-0-142-6 kubenswrapper[2564]: I0730 01:13:32.718709 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0w74kw_7069b0f9-9545-48ce-885e-1ec788c65ea5/util/0.log" Jul 30 01:13:32.724604 ip-10-0-142-6 kubenswrapper[2564]: I0730 01:13:32.724583 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0w74kw_7069b0f9-9545-48ce-885e-1ec788c65ea5/pull/0.log" Jul 30 01:13:32.730661 ip-10-0-142-6 kubenswrapper[2564]: I0730 01:13:32.730645 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0w74kw_7069b0f9-9545-48ce-885e-1ec788c65ea5/extract/0.log" Jul 30 01:13:32.836287 ip-10-0-142-6 kubenswrapper[2564]: I0730 01:13:32.836256 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73k9924_f3647c9a-1ae5-4ec6-8d69-c8db736e6fad/util/0.log" Jul 30 01:13:32.842242 ip-10-0-142-6 kubenswrapper[2564]: I0730 01:13:32.842222 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73k9924_f3647c9a-1ae5-4ec6-8d69-c8db736e6fad/pull/0.log" Jul 30 01:13:32.850982 ip-10-0-142-6 kubenswrapper[2564]: I0730 01:13:32.850902 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73k9924_f3647c9a-1ae5-4ec6-8d69-c8db736e6fad/extract/0.log" Jul 30 01:13:32.954403 ip-10-0-142-6 kubenswrapper[2564]: I0730 01:13:32.954363 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef187fz8_fbd780fd-defb-402c-98ff-2cca1c03d7b0/util/0.log" Jul 30 01:13:32.960575 ip-10-0-142-6 kubenswrapper[2564]: I0730 01:13:32.960552 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef187fz8_fbd780fd-defb-402c-98ff-2cca1c03d7b0/pull/0.log" Jul 30 01:13:32.967170 ip-10-0-142-6 kubenswrapper[2564]: I0730 01:13:32.967121 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef187fz8_fbd780fd-defb-402c-98ff-2cca1c03d7b0/extract/0.log" Jul 30 01:13:33.731467 ip-10-0-142-6 kubenswrapper[2564]: I0730 01:13:33.731439 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-limitador-54d6d988d5-nrv2p_4197b50a-70ea-4d6d-b57d-d648866eca86/limitador/0.log" Jul 30 01:13:34.279938 ip-10-0-142-6 kubenswrapper[2564]: I0730 01:13:34.279908 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_istiod-openshift-gateway-5677ccb49b-ww47t_3e278986-434b-4fd2-bfd5-2f055bc33629/discovery/0.log" Jul 30 01:13:35.569380 ip-10-0-142-6 kubenswrapper[2564]: I0730 01:13:35.569332 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fgdcgh_eebf6898-fe64-4cde-945c-f7226f4cd960/main/0.log" Jul 30 01:13:35.679306 ip-10-0-142-6 kubenswrapper[2564]: I0730 01:13:35.679273 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_facebook-opt-125m-simulated-kserve-74b68447df-czxkz_4300d011-90a2-4b87-a736-91f40a42f8c3/main/0.log" Jul 30 01:13:44.528344 ip-10-0-142-6 kubenswrapper[2564]: I0730 01:13:44.528316 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_global-pull-secret-syncer-d6phj_81450593-06b9-4f5a-81da-6e5fe67679f1/global-pull-secret-syncer/0.log" Jul 30 01:13:44.730551 ip-10-0-142-6 kubenswrapper[2564]: I0730 01:13:44.730520 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_konnectivity-agent-thhcs_6a73d10b-1baa-42ac-a83a-2336014e3600/konnectivity-agent/0.log" Jul 30 01:13:44.809782 ip-10-0-142-6 kubenswrapper[2564]: I0730 01:13:44.809681 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kube-apiserver-proxy-ip-10-0-142-6.ec2.internal_514977950851753a7cc9d5e8c3264eee/haproxy/0.log" Jul 30 01:13:48.508708 ip-10-0-142-6 kubenswrapper[2564]: I0730 01:13:48.508677 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759t2h7b_f546aac1-87f6-499c-98c8-02afb9ae43f6/extract/0.log" Jul 30 01:13:48.538527 ip-10-0-142-6 kubenswrapper[2564]: I0730 01:13:48.538507 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759t2h7b_f546aac1-87f6-499c-98c8-02afb9ae43f6/util/0.log" Jul 30 01:13:48.568873 ip-10-0-142-6 kubenswrapper[2564]: I0730 01:13:48.568851 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759t2h7b_f546aac1-87f6-499c-98c8-02afb9ae43f6/pull/0.log" Jul 30 01:13:48.631528 ip-10-0-142-6 kubenswrapper[2564]: I0730 01:13:48.631508 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0w74kw_7069b0f9-9545-48ce-885e-1ec788c65ea5/extract/0.log" Jul 30 01:13:48.666299 ip-10-0-142-6 kubenswrapper[2564]: I0730 01:13:48.666266 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0w74kw_7069b0f9-9545-48ce-885e-1ec788c65ea5/util/0.log" Jul 30 01:13:48.703026 ip-10-0-142-6 kubenswrapper[2564]: I0730 01:13:48.703000 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0w74kw_7069b0f9-9545-48ce-885e-1ec788c65ea5/pull/0.log" Jul 30 01:13:48.745038 ip-10-0-142-6 kubenswrapper[2564]: I0730 01:13:48.745014 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73k9924_f3647c9a-1ae5-4ec6-8d69-c8db736e6fad/extract/0.log" Jul 30 01:13:48.778610 ip-10-0-142-6 kubenswrapper[2564]: I0730 01:13:48.778548 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73k9924_f3647c9a-1ae5-4ec6-8d69-c8db736e6fad/util/0.log" Jul 30 01:13:48.810642 ip-10-0-142-6 kubenswrapper[2564]: I0730 01:13:48.810613 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73k9924_f3647c9a-1ae5-4ec6-8d69-c8db736e6fad/pull/0.log" Jul 30 01:13:48.844759 ip-10-0-142-6 kubenswrapper[2564]: I0730 01:13:48.844727 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef187fz8_fbd780fd-defb-402c-98ff-2cca1c03d7b0/extract/0.log" Jul 30 01:13:48.875923 ip-10-0-142-6 kubenswrapper[2564]: I0730 01:13:48.875900 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef187fz8_fbd780fd-defb-402c-98ff-2cca1c03d7b0/util/0.log" Jul 30 01:13:48.901449 ip-10-0-142-6 kubenswrapper[2564]: I0730 01:13:48.901427 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef187fz8_fbd780fd-defb-402c-98ff-2cca1c03d7b0/pull/0.log" Jul 30 01:13:49.425577 ip-10-0-142-6 kubenswrapper[2564]: I0730 01:13:49.425544 2564 scope.go:117] "RemoveContainer" containerID="f51b119f10564aae38ef8f697c53cfb624c586d58bd7017fff11c7fbdfd18bbc" Jul 30 01:13:49.434058 ip-10-0-142-6 kubenswrapper[2564]: I0730 01:13:49.434039 2564 scope.go:117] "RemoveContainer" containerID="09db2cf8be7f10d2ff62c1dca2c9aa6127d8ac3fba95df11f9080a35e2ac4457" Jul 30 01:13:49.621897 ip-10-0-142-6 kubenswrapper[2564]: I0730 01:13:49.621861 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-limitador-54d6d988d5-nrv2p_4197b50a-70ea-4d6d-b57d-d648866eca86/limitador/0.log" Jul 30 01:13:51.762899 ip-10-0-142-6 kubenswrapper[2564]: I0730 01:13:51.762821 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_kube-state-metrics-56d9f944b4-q5l9w_03d300be-130c-46de-9d44-dd302016de4a/kube-state-metrics/0.log" Jul 30 01:13:51.785453 ip-10-0-142-6 kubenswrapper[2564]: I0730 01:13:51.785428 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_kube-state-metrics-56d9f944b4-q5l9w_03d300be-130c-46de-9d44-dd302016de4a/kube-rbac-proxy-main/0.log" Jul 30 01:13:51.814003 ip-10-0-142-6 kubenswrapper[2564]: I0730 01:13:51.813974 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_kube-state-metrics-56d9f944b4-q5l9w_03d300be-130c-46de-9d44-dd302016de4a/kube-rbac-proxy-self/0.log" Jul 30 01:13:51.915994 ip-10-0-142-6 kubenswrapper[2564]: I0730 01:13:51.915964 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-5kvcx_5608ae36-4348-427e-bc57-dd15c468728e/node-exporter/0.log" Jul 30 01:13:51.947097 ip-10-0-142-6 kubenswrapper[2564]: I0730 01:13:51.947062 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-5kvcx_5608ae36-4348-427e-bc57-dd15c468728e/kube-rbac-proxy/0.log" Jul 30 01:13:51.973013 ip-10-0-142-6 kubenswrapper[2564]: I0730 01:13:51.972988 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-5kvcx_5608ae36-4348-427e-bc57-dd15c468728e/init-textfile/0.log" Jul 30 01:13:52.183504 ip-10-0-142-6 kubenswrapper[2564]: I0730 01:13:52.183472 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_openshift-state-metrics-b8f49f5c6-kbc7k_80c96de1-7df7-4fdf-afc4-75fb6e1c23ca/kube-rbac-proxy-main/0.log" Jul 30 01:13:52.210067 ip-10-0-142-6 kubenswrapper[2564]: I0730 01:13:52.210037 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_openshift-state-metrics-b8f49f5c6-kbc7k_80c96de1-7df7-4fdf-afc4-75fb6e1c23ca/kube-rbac-proxy-self/0.log" Jul 30 01:13:52.256581 ip-10-0-142-6 kubenswrapper[2564]: I0730 01:13:52.256555 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_openshift-state-metrics-b8f49f5c6-kbc7k_80c96de1-7df7-4fdf-afc4-75fb6e1c23ca/openshift-state-metrics/0.log" Jul 30 01:13:52.590624 ip-10-0-142-6 kubenswrapper[2564]: I0730 01:13:52.590543 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-operator-admission-webhook-7f8d78f54b-p46zv_1f32de58-81d4-4989-b8c2-2ae7d765bfa4/prometheus-operator-admission-webhook/0.log" Jul 30 01:13:52.622790 ip-10-0-142-6 kubenswrapper[2564]: I0730 01:13:52.622767 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_telemeter-client-7d6b996d65-fvmqd_de6ee145-7f06-4d65-84f7-e679fff20407/telemeter-client/0.log" Jul 30 01:13:52.648867 ip-10-0-142-6 kubenswrapper[2564]: I0730 01:13:52.648842 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_telemeter-client-7d6b996d65-fvmqd_de6ee145-7f06-4d65-84f7-e679fff20407/reload/0.log" Jul 30 01:13:52.672188 ip-10-0-142-6 kubenswrapper[2564]: I0730 01:13:52.672158 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_telemeter-client-7d6b996d65-fvmqd_de6ee145-7f06-4d65-84f7-e679fff20407/kube-rbac-proxy/0.log" Jul 30 01:13:54.682534 ip-10-0-142-6 kubenswrapper[2564]: I0730 01:13:54.682485 2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-must-gather-p9qsq/perf-node-gather-daemonset-j4h68"] Jul 30 01:13:54.682955 ip-10-0-142-6 kubenswrapper[2564]: I0730 01:13:54.682797 2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d8a2c8f9-1aea-4356-bf0d-ad50e738f9b7" containerName="revoke-keys" Jul 30 01:13:54.682955 ip-10-0-142-6 kubenswrapper[2564]: I0730 01:13:54.682808 2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="d8a2c8f9-1aea-4356-bf0d-ad50e738f9b7" containerName="revoke-keys" Jul 30 01:13:54.682955 ip-10-0-142-6 kubenswrapper[2564]: I0730 01:13:54.682818 2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e9cd27ab-1777-4242-afb7-31edf86b4fa0" containerName="istio-proxy" Jul 30 01:13:54.682955 ip-10-0-142-6 kubenswrapper[2564]: I0730 01:13:54.682824 2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="e9cd27ab-1777-4242-afb7-31edf86b4fa0" containerName="istio-proxy" Jul 30 01:13:54.682955 ip-10-0-142-6 kubenswrapper[2564]: I0730 01:13:54.682886 2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="d8a2c8f9-1aea-4356-bf0d-ad50e738f9b7" containerName="revoke-keys" Jul 30 01:13:54.687937 ip-10-0-142-6 kubenswrapper[2564]: I0730 01:13:54.687915 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-p9qsq/perf-node-gather-daemonset-j4h68" Jul 30 01:13:54.691684 ip-10-0-142-6 kubenswrapper[2564]: I0730 01:13:54.691662 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-p9qsq\"/\"openshift-service-ca.crt\"" Jul 30 01:13:54.691684 ip-10-0-142-6 kubenswrapper[2564]: I0730 01:13:54.691674 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-p9qsq\"/\"default-dockercfg-6smbp\"" Jul 30 01:13:54.692294 ip-10-0-142-6 kubenswrapper[2564]: I0730 01:13:54.692280 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-p9qsq\"/\"kube-root-ca.crt\"" Jul 30 01:13:54.710819 ip-10-0-142-6 kubenswrapper[2564]: I0730 01:13:54.710788 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-p9qsq/perf-node-gather-daemonset-j4h68"] Jul 30 01:13:54.836986 ip-10-0-142-6 kubenswrapper[2564]: I0730 01:13:54.836943 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-b9ddj\" (UniqueName: \"kubernetes.io/projected/1605395a-2e60-44cc-8ac8-18878e0ebea3-kube-api-access-b9ddj\") pod \"perf-node-gather-daemonset-j4h68\" (UID: \"1605395a-2e60-44cc-8ac8-18878e0ebea3\") " pod="openshift-must-gather-p9qsq/perf-node-gather-daemonset-j4h68" Jul 30 01:13:54.837203 ip-10-0-142-6 kubenswrapper[2564]: I0730 01:13:54.837056 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/1605395a-2e60-44cc-8ac8-18878e0ebea3-lib-modules\") pod \"perf-node-gather-daemonset-j4h68\" (UID: \"1605395a-2e60-44cc-8ac8-18878e0ebea3\") " pod="openshift-must-gather-p9qsq/perf-node-gather-daemonset-j4h68" Jul 30 01:13:54.837203 ip-10-0-142-6 kubenswrapper[2564]: I0730 01:13:54.837113 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/1605395a-2e60-44cc-8ac8-18878e0ebea3-podres\") pod \"perf-node-gather-daemonset-j4h68\" (UID: \"1605395a-2e60-44cc-8ac8-18878e0ebea3\") " pod="openshift-must-gather-p9qsq/perf-node-gather-daemonset-j4h68" Jul 30 01:13:54.837203 ip-10-0-142-6 kubenswrapper[2564]: I0730 01:13:54.837143 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/1605395a-2e60-44cc-8ac8-18878e0ebea3-proc\") pod \"perf-node-gather-daemonset-j4h68\" (UID: \"1605395a-2e60-44cc-8ac8-18878e0ebea3\") " pod="openshift-must-gather-p9qsq/perf-node-gather-daemonset-j4h68" Jul 30 01:13:54.837203 ip-10-0-142-6 kubenswrapper[2564]: I0730 01:13:54.837171 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/1605395a-2e60-44cc-8ac8-18878e0ebea3-sys\") pod \"perf-node-gather-daemonset-j4h68\" (UID: \"1605395a-2e60-44cc-8ac8-18878e0ebea3\") " pod="openshift-must-gather-p9qsq/perf-node-gather-daemonset-j4h68" Jul 30 01:13:54.938215 ip-10-0-142-6 kubenswrapper[2564]: I0730 01:13:54.938138 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/1605395a-2e60-44cc-8ac8-18878e0ebea3-lib-modules\") pod \"perf-node-gather-daemonset-j4h68\" (UID: \"1605395a-2e60-44cc-8ac8-18878e0ebea3\") " pod="openshift-must-gather-p9qsq/perf-node-gather-daemonset-j4h68" Jul 30 01:13:54.938215 ip-10-0-142-6 kubenswrapper[2564]: I0730 01:13:54.938174 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/1605395a-2e60-44cc-8ac8-18878e0ebea3-podres\") pod \"perf-node-gather-daemonset-j4h68\" (UID: \"1605395a-2e60-44cc-8ac8-18878e0ebea3\") " pod="openshift-must-gather-p9qsq/perf-node-gather-daemonset-j4h68" Jul 30 01:13:54.938215 ip-10-0-142-6 kubenswrapper[2564]: I0730 01:13:54.938198 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/1605395a-2e60-44cc-8ac8-18878e0ebea3-proc\") pod \"perf-node-gather-daemonset-j4h68\" (UID: \"1605395a-2e60-44cc-8ac8-18878e0ebea3\") " pod="openshift-must-gather-p9qsq/perf-node-gather-daemonset-j4h68" Jul 30 01:13:54.938390 ip-10-0-142-6 kubenswrapper[2564]: I0730 01:13:54.938219 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/1605395a-2e60-44cc-8ac8-18878e0ebea3-sys\") pod \"perf-node-gather-daemonset-j4h68\" (UID: \"1605395a-2e60-44cc-8ac8-18878e0ebea3\") " pod="openshift-must-gather-p9qsq/perf-node-gather-daemonset-j4h68" Jul 30 01:13:54.938390 ip-10-0-142-6 kubenswrapper[2564]: I0730 01:13:54.938253 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b9ddj\" (UniqueName: \"kubernetes.io/projected/1605395a-2e60-44cc-8ac8-18878e0ebea3-kube-api-access-b9ddj\") pod \"perf-node-gather-daemonset-j4h68\" (UID: \"1605395a-2e60-44cc-8ac8-18878e0ebea3\") " pod="openshift-must-gather-p9qsq/perf-node-gather-daemonset-j4h68" Jul 30 01:13:54.938390 ip-10-0-142-6 kubenswrapper[2564]: I0730 01:13:54.938303 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/1605395a-2e60-44cc-8ac8-18878e0ebea3-proc\") pod \"perf-node-gather-daemonset-j4h68\" (UID: \"1605395a-2e60-44cc-8ac8-18878e0ebea3\") " pod="openshift-must-gather-p9qsq/perf-node-gather-daemonset-j4h68" Jul 30 01:13:54.938390 ip-10-0-142-6 kubenswrapper[2564]: I0730 01:13:54.938332 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/1605395a-2e60-44cc-8ac8-18878e0ebea3-lib-modules\") pod \"perf-node-gather-daemonset-j4h68\" (UID: \"1605395a-2e60-44cc-8ac8-18878e0ebea3\") " pod="openshift-must-gather-p9qsq/perf-node-gather-daemonset-j4h68" Jul 30 01:13:54.938390 ip-10-0-142-6 kubenswrapper[2564]: I0730 01:13:54.938332 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/1605395a-2e60-44cc-8ac8-18878e0ebea3-podres\") pod \"perf-node-gather-daemonset-j4h68\" (UID: \"1605395a-2e60-44cc-8ac8-18878e0ebea3\") " pod="openshift-must-gather-p9qsq/perf-node-gather-daemonset-j4h68" Jul 30 01:13:54.938390 ip-10-0-142-6 kubenswrapper[2564]: I0730 01:13:54.938363 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/1605395a-2e60-44cc-8ac8-18878e0ebea3-sys\") pod \"perf-node-gather-daemonset-j4h68\" (UID: \"1605395a-2e60-44cc-8ac8-18878e0ebea3\") " pod="openshift-must-gather-p9qsq/perf-node-gather-daemonset-j4h68" Jul 30 01:13:54.946774 ip-10-0-142-6 kubenswrapper[2564]: I0730 01:13:54.946751 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-b9ddj\" (UniqueName: \"kubernetes.io/projected/1605395a-2e60-44cc-8ac8-18878e0ebea3-kube-api-access-b9ddj\") pod \"perf-node-gather-daemonset-j4h68\" (UID: \"1605395a-2e60-44cc-8ac8-18878e0ebea3\") " pod="openshift-must-gather-p9qsq/perf-node-gather-daemonset-j4h68" Jul 30 01:13:54.997579 ip-10-0-142-6 kubenswrapper[2564]: I0730 01:13:54.997546 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-p9qsq/perf-node-gather-daemonset-j4h68" Jul 30 01:13:55.130282 ip-10-0-142-6 kubenswrapper[2564]: I0730 01:13:55.130257 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-p9qsq/perf-node-gather-daemonset-j4h68"] Jul 30 01:13:55.132136 ip-10-0-142-6 kubenswrapper[2564]: W0730 01:13:55.132110 2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-pod1605395a_2e60_44cc_8ac8_18878e0ebea3.slice/crio-955b96c8efeb1b4a4e8b5e3a99845c4443653bd3da5c78205467467da84346d5 WatchSource:0}: Error finding container 955b96c8efeb1b4a4e8b5e3a99845c4443653bd3da5c78205467467da84346d5: Status 404 returned error can't find the container with id 955b96c8efeb1b4a4e8b5e3a99845c4443653bd3da5c78205467467da84346d5 Jul 30 01:13:55.386378 ip-10-0-142-6 kubenswrapper[2564]: I0730 01:13:55.386335 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-p9qsq/perf-node-gather-daemonset-j4h68" event={"ID":"1605395a-2e60-44cc-8ac8-18878e0ebea3","Type":"ContainerStarted","Data":"ff276f74b3a99208fb69dab98cc1129e3b3013351b39aa3f9716e5dec5d756ed"} Jul 30 01:13:55.386586 ip-10-0-142-6 kubenswrapper[2564]: I0730 01:13:55.386396 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-p9qsq/perf-node-gather-daemonset-j4h68" event={"ID":"1605395a-2e60-44cc-8ac8-18878e0ebea3","Type":"ContainerStarted","Data":"955b96c8efeb1b4a4e8b5e3a99845c4443653bd3da5c78205467467da84346d5"} Jul 30 01:13:55.386586 ip-10-0-142-6 kubenswrapper[2564]: I0730 01:13:55.386536 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-must-gather-p9qsq/perf-node-gather-daemonset-j4h68" Jul 30 01:13:55.407954 ip-10-0-142-6 kubenswrapper[2564]: I0730 01:13:55.407903 2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-must-gather-p9qsq/perf-node-gather-daemonset-j4h68" podStartSLOduration=1.407888699 podStartE2EDuration="1.407888699s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-30 01:13:54 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-30 01:13:55.405432895 +0000 UTC m=+3366.820172831" watchObservedRunningTime="2026-07-30 01:13:55.407888699 +0000 UTC m=+3366.822628631" Jul 30 01:13:57.205226 ip-10-0-142-6 kubenswrapper[2564]: I0730 01:13:57.205194 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-8lwzf_45648249-3d11-4938-8d3b-a28fbf301f7e/dns/0.log" Jul 30 01:13:57.230739 ip-10-0-142-6 kubenswrapper[2564]: I0730 01:13:57.230715 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-8lwzf_45648249-3d11-4938-8d3b-a28fbf301f7e/kube-rbac-proxy/0.log" Jul 30 01:13:57.336502 ip-10-0-142-6 kubenswrapper[2564]: I0730 01:13:57.336469 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-6ssnm_99deae04-e4b4-4491-a31f-945927bd1e72/dns-node-resolver/0.log" Jul 30 01:13:57.910441 ip-10-0-142-6 kubenswrapper[2564]: I0730 01:13:57.910416 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_image-registry-6477b9596b-4thrq_2af09576-ffda-4b11-8ddf-ce24de77a2d9/registry/0.log" Jul 30 01:13:57.962032 ip-10-0-142-6 kubenswrapper[2564]: I0730 01:13:57.962006 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-jc8px_7ff24a6e-78df-4c03-830a-453386386626/node-ca/0.log" Jul 30 01:13:59.133990 ip-10-0-142-6 kubenswrapper[2564]: I0730 01:13:59.133964 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_istiod-openshift-gateway-5677ccb49b-ww47t_3e278986-434b-4fd2-bfd5-2f055bc33629/discovery/0.log" Jul 30 01:14:00.238283 ip-10-0-142-6 kubenswrapper[2564]: I0730 01:14:00.238259 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-lh7nk_4d21b8fa-df3a-4e77-8b66-bf45c15dbe56/serve-healthcheck-canary/0.log" Jul 30 01:14:00.961112 ip-10-0-142-6 kubenswrapper[2564]: I0730 01:14:00.961063 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-cpgjx_7b14c865-5a91-41ac-81a5-869011355781/kube-rbac-proxy/0.log" Jul 30 01:14:00.990116 ip-10-0-142-6 kubenswrapper[2564]: I0730 01:14:00.990044 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-cpgjx_7b14c865-5a91-41ac-81a5-869011355781/exporter/0.log" Jul 30 01:14:01.018816 ip-10-0-142-6 kubenswrapper[2564]: I0730 01:14:01.018791 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-cpgjx_7b14c865-5a91-41ac-81a5-869011355781/extractor/0.log" Jul 30 01:14:01.400784 ip-10-0-142-6 kubenswrapper[2564]: I0730 01:14:01.400758 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-must-gather-p9qsq/perf-node-gather-daemonset-j4h68" Jul 30 01:14:03.391618 ip-10-0-142-6 kubenswrapper[2564]: I0730 01:14:03.391594 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_kserve-controller-manager-d6f4b65b8-bwrb2_c998e5e8-aa41-4da6-a8a8-94b49e8f99fb/manager/0.log" Jul 30 01:14:04.076710 ip-10-0-142-6 kubenswrapper[2564]: I0730 01:14:04.076623 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-5b74bc5486-thrgp_db0f2405-58a7-4112-af82-65d1d07bb5a9/manager/0.log" Jul 30 01:14:04.101065 ip-10-0-142-6 kubenswrapper[2564]: I0730 01:14:04.101036 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-5b74bc5486-thrgp_db0f2405-58a7-4112-af82-65d1d07bb5a9/copy-manifests/0.log" Jul 30 01:14:05.251412 ip-10-0-142-6 kubenswrapper[2564]: I0730 01:14:05.251324 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_istiod-openshift-gateway-5677ccb49b-ww47t_3e278986-434b-4fd2-bfd5-2f055bc33629/discovery/0.log" Jul 30 01:14:05.665774 ip-10-0-142-6 kubenswrapper[2564]: I0730 01:14:05.665748 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-lws-operator_lws-controller-manager-685c7855bf-xzzzz_8c9f42c1-c9d8-4a60-b2d5-97150a0a9ef7/manager/0.log" Jul 30 01:14:07.011815 ip-10-0-142-6 kubenswrapper[2564]: I0730 01:14:07.011784 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_istiod-openshift-gateway-5677ccb49b-ww47t_3e278986-434b-4fd2-bfd5-2f055bc33629/discovery/0.log" Jul 30 01:14:12.249320 ip-10-0-142-6 kubenswrapper[2564]: I0730 01:14:12.249288 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-c6sdp_b1186f7d-220d-4210-a7f8-6f3f0cf554e8/kube-multus-additional-cni-plugins/0.log" Jul 30 01:14:12.278918 ip-10-0-142-6 kubenswrapper[2564]: I0730 01:14:12.278893 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-c6sdp_b1186f7d-220d-4210-a7f8-6f3f0cf554e8/egress-router-binary-copy/0.log" Jul 30 01:14:12.306311 ip-10-0-142-6 kubenswrapper[2564]: I0730 01:14:12.306287 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-c6sdp_b1186f7d-220d-4210-a7f8-6f3f0cf554e8/cni-plugins/0.log" Jul 30 01:14:12.338058 ip-10-0-142-6 kubenswrapper[2564]: I0730 01:14:12.338033 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-c6sdp_b1186f7d-220d-4210-a7f8-6f3f0cf554e8/bond-cni-plugin/0.log" Jul 30 01:14:12.365119 ip-10-0-142-6 kubenswrapper[2564]: I0730 01:14:12.365095 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-c6sdp_b1186f7d-220d-4210-a7f8-6f3f0cf554e8/routeoverride-cni/0.log" Jul 30 01:14:12.393022 ip-10-0-142-6 kubenswrapper[2564]: I0730 01:14:12.393003 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-c6sdp_b1186f7d-220d-4210-a7f8-6f3f0cf554e8/whereabouts-cni-bincopy/0.log" Jul 30 01:14:12.421059 ip-10-0-142-6 kubenswrapper[2564]: I0730 01:14:12.421031 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-c6sdp_b1186f7d-220d-4210-a7f8-6f3f0cf554e8/whereabouts-cni/0.log"