Sep 10 13:26:32.268623 ip-10-0-140-8 systemd[1]: kubelet.service: Failed to load environment files: No such file or directory Sep 10 13:26:32.268637 ip-10-0-140-8 systemd[1]: kubelet.service: Failed to run 'start-pre' task: No such file or directory Sep 10 13:26:32.268646 ip-10-0-140-8 systemd[1]: kubelet.service: Failed with result 'resources'. Sep 10 13:26:32.268979 ip-10-0-140-8 systemd[1]: Failed to start Kubernetes Kubelet. Sep 10 13:26:42.272291 ip-10-0-140-8 systemd[1]: kubelet.service: Failed to schedule restart job: Unit crio.service not found. Sep 10 13:26:42.272314 ip-10-0-140-8 systemd[1]: kubelet.service: Failed with result 'resources'. -- Boot 818b2245626246318e963d7b0adba730 -- Sep 10 13:28:33.734569 ip-10-0-140-8 systemd[1]: Starting Kubernetes Kubelet... Sep 10 13:28:34.175355 ip-10-0-140-8 kubenswrapper[2585]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Sep 10 13:28:34.175355 ip-10-0-140-8 kubenswrapper[2585]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version. Sep 10 13:28:34.175355 ip-10-0-140-8 kubenswrapper[2585]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Sep 10 13:28:34.175355 ip-10-0-140-8 kubenswrapper[2585]: Flag --pod-infra-container-image has been deprecated, will be removed in 1.35. Image garbage collector will get sandbox image information from CRI. Sep 10 13:28:34.175355 ip-10-0-140-8 kubenswrapper[2585]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Sep 10 13:28:34.178401 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:28:34.178304 2585 server.go:213] "--pod-infra-container-image will not be pruned by the image garbage collector in kubelet and should also be set in the remote runtime" Sep 10 13:28:34.182621 ip-10-0-140-8 kubenswrapper[2585]: W0910 13:28:34.182604 2585 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Sep 10 13:28:34.182621 ip-10-0-140-8 kubenswrapper[2585]: W0910 13:28:34.182620 2585 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Sep 10 13:28:34.182688 ip-10-0-140-8 kubenswrapper[2585]: W0910 13:28:34.182624 2585 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Sep 10 13:28:34.182688 ip-10-0-140-8 kubenswrapper[2585]: W0910 13:28:34.182627 2585 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Sep 10 13:28:34.182688 ip-10-0-140-8 kubenswrapper[2585]: W0910 13:28:34.182631 2585 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Sep 10 13:28:34.182688 ip-10-0-140-8 kubenswrapper[2585]: W0910 13:28:34.182635 2585 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Sep 10 13:28:34.182688 ip-10-0-140-8 kubenswrapper[2585]: W0910 13:28:34.182638 2585 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Sep 10 13:28:34.182688 ip-10-0-140-8 kubenswrapper[2585]: W0910 13:28:34.182640 2585 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Sep 10 13:28:34.182688 ip-10-0-140-8 kubenswrapper[2585]: W0910 13:28:34.182644 2585 feature_gate.go:328] unrecognized feature gate: InsightsConfig Sep 10 13:28:34.182688 ip-10-0-140-8 kubenswrapper[2585]: W0910 13:28:34.182646 2585 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Sep 10 13:28:34.182688 ip-10-0-140-8 kubenswrapper[2585]: W0910 13:28:34.182649 2585 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Sep 10 13:28:34.182688 ip-10-0-140-8 kubenswrapper[2585]: W0910 13:28:34.182653 2585 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Sep 10 13:28:34.182688 ip-10-0-140-8 kubenswrapper[2585]: W0910 13:28:34.182657 2585 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Sep 10 13:28:34.182688 ip-10-0-140-8 kubenswrapper[2585]: W0910 13:28:34.182661 2585 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Sep 10 13:28:34.182688 ip-10-0-140-8 kubenswrapper[2585]: W0910 13:28:34.182669 2585 feature_gate.go:328] unrecognized feature gate: GatewayAPI Sep 10 13:28:34.182688 ip-10-0-140-8 kubenswrapper[2585]: W0910 13:28:34.182672 2585 feature_gate.go:328] unrecognized feature gate: SignatureStores Sep 10 13:28:34.182688 ip-10-0-140-8 kubenswrapper[2585]: W0910 13:28:34.182675 2585 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Sep 10 13:28:34.182688 ip-10-0-140-8 kubenswrapper[2585]: W0910 13:28:34.182678 2585 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Sep 10 13:28:34.182688 ip-10-0-140-8 kubenswrapper[2585]: W0910 13:28:34.182681 2585 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Sep 10 13:28:34.182688 ip-10-0-140-8 kubenswrapper[2585]: W0910 13:28:34.182684 2585 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Sep 10 13:28:34.182688 ip-10-0-140-8 kubenswrapper[2585]: W0910 13:28:34.182686 2585 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Sep 10 13:28:34.183177 ip-10-0-140-8 kubenswrapper[2585]: W0910 13:28:34.182689 2585 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Sep 10 13:28:34.183177 ip-10-0-140-8 kubenswrapper[2585]: W0910 13:28:34.182692 2585 feature_gate.go:328] unrecognized feature gate: EventTTL Sep 10 13:28:34.183177 ip-10-0-140-8 kubenswrapper[2585]: W0910 13:28:34.182695 2585 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Sep 10 13:28:34.183177 ip-10-0-140-8 kubenswrapper[2585]: W0910 13:28:34.182698 2585 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Sep 10 13:28:34.183177 ip-10-0-140-8 kubenswrapper[2585]: W0910 13:28:34.182701 2585 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Sep 10 13:28:34.183177 ip-10-0-140-8 kubenswrapper[2585]: W0910 13:28:34.182704 2585 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Sep 10 13:28:34.183177 ip-10-0-140-8 kubenswrapper[2585]: W0910 13:28:34.182706 2585 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Sep 10 13:28:34.183177 ip-10-0-140-8 kubenswrapper[2585]: W0910 13:28:34.182709 2585 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Sep 10 13:28:34.183177 ip-10-0-140-8 kubenswrapper[2585]: W0910 13:28:34.182711 2585 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Sep 10 13:28:34.183177 ip-10-0-140-8 kubenswrapper[2585]: W0910 13:28:34.182714 2585 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Sep 10 13:28:34.183177 ip-10-0-140-8 kubenswrapper[2585]: W0910 13:28:34.182717 2585 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Sep 10 13:28:34.183177 ip-10-0-140-8 kubenswrapper[2585]: W0910 13:28:34.182719 2585 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Sep 10 13:28:34.183177 ip-10-0-140-8 kubenswrapper[2585]: W0910 13:28:34.182722 2585 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Sep 10 13:28:34.183177 ip-10-0-140-8 kubenswrapper[2585]: W0910 13:28:34.182724 2585 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Sep 10 13:28:34.183177 ip-10-0-140-8 kubenswrapper[2585]: W0910 13:28:34.182727 2585 feature_gate.go:328] unrecognized feature gate: Example Sep 10 13:28:34.183177 ip-10-0-140-8 kubenswrapper[2585]: W0910 13:28:34.182729 2585 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Sep 10 13:28:34.183177 ip-10-0-140-8 kubenswrapper[2585]: W0910 13:28:34.182731 2585 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Sep 10 13:28:34.183177 ip-10-0-140-8 kubenswrapper[2585]: W0910 13:28:34.182734 2585 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Sep 10 13:28:34.183177 ip-10-0-140-8 kubenswrapper[2585]: W0910 13:28:34.182736 2585 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Sep 10 13:28:34.183177 ip-10-0-140-8 kubenswrapper[2585]: W0910 13:28:34.182739 2585 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Sep 10 13:28:34.183695 ip-10-0-140-8 kubenswrapper[2585]: W0910 13:28:34.182741 2585 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Sep 10 13:28:34.183695 ip-10-0-140-8 kubenswrapper[2585]: W0910 13:28:34.182743 2585 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Sep 10 13:28:34.183695 ip-10-0-140-8 kubenswrapper[2585]: W0910 13:28:34.182746 2585 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Sep 10 13:28:34.183695 ip-10-0-140-8 kubenswrapper[2585]: W0910 13:28:34.182748 2585 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Sep 10 13:28:34.183695 ip-10-0-140-8 kubenswrapper[2585]: W0910 13:28:34.182751 2585 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Sep 10 13:28:34.183695 ip-10-0-140-8 kubenswrapper[2585]: W0910 13:28:34.182754 2585 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Sep 10 13:28:34.183695 ip-10-0-140-8 kubenswrapper[2585]: W0910 13:28:34.182757 2585 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Sep 10 13:28:34.183695 ip-10-0-140-8 kubenswrapper[2585]: W0910 13:28:34.182759 2585 feature_gate.go:328] unrecognized feature gate: Example2 Sep 10 13:28:34.183695 ip-10-0-140-8 kubenswrapper[2585]: W0910 13:28:34.182762 2585 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Sep 10 13:28:34.183695 ip-10-0-140-8 kubenswrapper[2585]: W0910 13:28:34.182765 2585 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Sep 10 13:28:34.183695 ip-10-0-140-8 kubenswrapper[2585]: W0910 13:28:34.182767 2585 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Sep 10 13:28:34.183695 ip-10-0-140-8 kubenswrapper[2585]: W0910 13:28:34.182770 2585 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Sep 10 13:28:34.183695 ip-10-0-140-8 kubenswrapper[2585]: W0910 13:28:34.182773 2585 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Sep 10 13:28:34.183695 ip-10-0-140-8 kubenswrapper[2585]: W0910 13:28:34.182776 2585 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Sep 10 13:28:34.183695 ip-10-0-140-8 kubenswrapper[2585]: W0910 13:28:34.182778 2585 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Sep 10 13:28:34.183695 ip-10-0-140-8 kubenswrapper[2585]: W0910 13:28:34.182781 2585 feature_gate.go:328] unrecognized feature gate: OVNObservability Sep 10 13:28:34.183695 ip-10-0-140-8 kubenswrapper[2585]: W0910 13:28:34.182783 2585 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Sep 10 13:28:34.183695 ip-10-0-140-8 kubenswrapper[2585]: W0910 13:28:34.182786 2585 feature_gate.go:328] unrecognized feature gate: NewOLM Sep 10 13:28:34.183695 ip-10-0-140-8 kubenswrapper[2585]: W0910 13:28:34.182789 2585 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Sep 10 13:28:34.183695 ip-10-0-140-8 kubenswrapper[2585]: W0910 13:28:34.182791 2585 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Sep 10 13:28:34.184176 ip-10-0-140-8 kubenswrapper[2585]: W0910 13:28:34.182794 2585 feature_gate.go:328] unrecognized feature gate: OSStreams Sep 10 13:28:34.184176 ip-10-0-140-8 kubenswrapper[2585]: W0910 13:28:34.182796 2585 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Sep 10 13:28:34.184176 ip-10-0-140-8 kubenswrapper[2585]: W0910 13:28:34.182800 2585 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Sep 10 13:28:34.184176 ip-10-0-140-8 kubenswrapper[2585]: W0910 13:28:34.182803 2585 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Sep 10 13:28:34.184176 ip-10-0-140-8 kubenswrapper[2585]: W0910 13:28:34.182805 2585 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Sep 10 13:28:34.184176 ip-10-0-140-8 kubenswrapper[2585]: W0910 13:28:34.182808 2585 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Sep 10 13:28:34.184176 ip-10-0-140-8 kubenswrapper[2585]: W0910 13:28:34.182810 2585 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Sep 10 13:28:34.184176 ip-10-0-140-8 kubenswrapper[2585]: W0910 13:28:34.182814 2585 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Sep 10 13:28:34.184176 ip-10-0-140-8 kubenswrapper[2585]: W0910 13:28:34.182818 2585 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Sep 10 13:28:34.184176 ip-10-0-140-8 kubenswrapper[2585]: W0910 13:28:34.182821 2585 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Sep 10 13:28:34.184176 ip-10-0-140-8 kubenswrapper[2585]: W0910 13:28:34.182824 2585 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Sep 10 13:28:34.184176 ip-10-0-140-8 kubenswrapper[2585]: W0910 13:28:34.182826 2585 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Sep 10 13:28:34.184176 ip-10-0-140-8 kubenswrapper[2585]: W0910 13:28:34.182829 2585 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Sep 10 13:28:34.184176 ip-10-0-140-8 kubenswrapper[2585]: W0910 13:28:34.182832 2585 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Sep 10 13:28:34.184176 ip-10-0-140-8 kubenswrapper[2585]: W0910 13:28:34.182834 2585 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Sep 10 13:28:34.184176 ip-10-0-140-8 kubenswrapper[2585]: W0910 13:28:34.182837 2585 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Sep 10 13:28:34.184176 ip-10-0-140-8 kubenswrapper[2585]: W0910 13:28:34.182845 2585 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Sep 10 13:28:34.184176 ip-10-0-140-8 kubenswrapper[2585]: W0910 13:28:34.182848 2585 feature_gate.go:328] unrecognized feature gate: DualReplica Sep 10 13:28:34.184625 ip-10-0-140-8 kubenswrapper[2585]: W0910 13:28:34.182851 2585 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Sep 10 13:28:34.184625 ip-10-0-140-8 kubenswrapper[2585]: W0910 13:28:34.182854 2585 feature_gate.go:328] unrecognized feature gate: KMSEncryption Sep 10 13:28:34.184625 ip-10-0-140-8 kubenswrapper[2585]: W0910 13:28:34.182858 2585 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Sep 10 13:28:34.184625 ip-10-0-140-8 kubenswrapper[2585]: W0910 13:28:34.182861 2585 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Sep 10 13:28:34.184625 ip-10-0-140-8 kubenswrapper[2585]: W0910 13:28:34.182863 2585 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Sep 10 13:28:34.184625 ip-10-0-140-8 kubenswrapper[2585]: W0910 13:28:34.182866 2585 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Sep 10 13:28:34.184625 ip-10-0-140-8 kubenswrapper[2585]: W0910 13:28:34.182869 2585 feature_gate.go:328] unrecognized feature gate: PinnedImages Sep 10 13:28:34.184625 ip-10-0-140-8 kubenswrapper[2585]: W0910 13:28:34.182872 2585 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Sep 10 13:28:34.184625 ip-10-0-140-8 kubenswrapper[2585]: W0910 13:28:34.182875 2585 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Sep 10 13:28:34.184625 ip-10-0-140-8 kubenswrapper[2585]: W0910 13:28:34.182878 2585 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Sep 10 13:28:34.184625 ip-10-0-140-8 kubenswrapper[2585]: W0910 13:28:34.182880 2585 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Sep 10 13:28:34.184625 ip-10-0-140-8 kubenswrapper[2585]: W0910 13:28:34.182883 2585 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Sep 10 13:28:34.184625 ip-10-0-140-8 kubenswrapper[2585]: W0910 13:28:34.182885 2585 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Sep 10 13:28:34.184625 ip-10-0-140-8 kubenswrapper[2585]: W0910 13:28:34.182888 2585 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Sep 10 13:28:34.184625 ip-10-0-140-8 kubenswrapper[2585]: W0910 13:28:34.182890 2585 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Sep 10 13:28:34.184625 ip-10-0-140-8 kubenswrapper[2585]: W0910 13:28:34.182893 2585 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Sep 10 13:28:34.184625 ip-10-0-140-8 kubenswrapper[2585]: W0910 13:28:34.182896 2585 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Sep 10 13:28:34.184625 ip-10-0-140-8 kubenswrapper[2585]: W0910 13:28:34.182898 2585 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Sep 10 13:28:34.184625 ip-10-0-140-8 kubenswrapper[2585]: W0910 13:28:34.182900 2585 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Sep 10 13:28:34.184625 ip-10-0-140-8 kubenswrapper[2585]: W0910 13:28:34.182903 2585 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Sep 10 13:28:34.185110 ip-10-0-140-8 kubenswrapper[2585]: W0910 13:28:34.183649 2585 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Sep 10 13:28:34.185110 ip-10-0-140-8 kubenswrapper[2585]: W0910 13:28:34.183657 2585 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Sep 10 13:28:34.185110 ip-10-0-140-8 kubenswrapper[2585]: W0910 13:28:34.183661 2585 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Sep 10 13:28:34.185110 ip-10-0-140-8 kubenswrapper[2585]: W0910 13:28:34.183665 2585 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Sep 10 13:28:34.185110 ip-10-0-140-8 kubenswrapper[2585]: W0910 13:28:34.183668 2585 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Sep 10 13:28:34.185110 ip-10-0-140-8 kubenswrapper[2585]: W0910 13:28:34.183672 2585 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Sep 10 13:28:34.185110 ip-10-0-140-8 kubenswrapper[2585]: W0910 13:28:34.183675 2585 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Sep 10 13:28:34.185110 ip-10-0-140-8 kubenswrapper[2585]: W0910 13:28:34.183678 2585 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Sep 10 13:28:34.185110 ip-10-0-140-8 kubenswrapper[2585]: W0910 13:28:34.183680 2585 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Sep 10 13:28:34.185110 ip-10-0-140-8 kubenswrapper[2585]: W0910 13:28:34.183683 2585 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Sep 10 13:28:34.185110 ip-10-0-140-8 kubenswrapper[2585]: W0910 13:28:34.183686 2585 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Sep 10 13:28:34.185110 ip-10-0-140-8 kubenswrapper[2585]: W0910 13:28:34.183689 2585 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Sep 10 13:28:34.185110 ip-10-0-140-8 kubenswrapper[2585]: W0910 13:28:34.183692 2585 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Sep 10 13:28:34.185110 ip-10-0-140-8 kubenswrapper[2585]: W0910 13:28:34.183695 2585 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Sep 10 13:28:34.185110 ip-10-0-140-8 kubenswrapper[2585]: W0910 13:28:34.183697 2585 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Sep 10 13:28:34.185110 ip-10-0-140-8 kubenswrapper[2585]: W0910 13:28:34.183700 2585 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Sep 10 13:28:34.185110 ip-10-0-140-8 kubenswrapper[2585]: W0910 13:28:34.183703 2585 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Sep 10 13:28:34.185110 ip-10-0-140-8 kubenswrapper[2585]: W0910 13:28:34.183706 2585 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Sep 10 13:28:34.185110 ip-10-0-140-8 kubenswrapper[2585]: W0910 13:28:34.183709 2585 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Sep 10 13:28:34.185597 ip-10-0-140-8 kubenswrapper[2585]: W0910 13:28:34.183712 2585 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Sep 10 13:28:34.185597 ip-10-0-140-8 kubenswrapper[2585]: W0910 13:28:34.183715 2585 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Sep 10 13:28:34.185597 ip-10-0-140-8 kubenswrapper[2585]: W0910 13:28:34.183718 2585 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Sep 10 13:28:34.185597 ip-10-0-140-8 kubenswrapper[2585]: W0910 13:28:34.183720 2585 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Sep 10 13:28:34.185597 ip-10-0-140-8 kubenswrapper[2585]: W0910 13:28:34.183723 2585 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Sep 10 13:28:34.185597 ip-10-0-140-8 kubenswrapper[2585]: W0910 13:28:34.183725 2585 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Sep 10 13:28:34.185597 ip-10-0-140-8 kubenswrapper[2585]: W0910 13:28:34.183728 2585 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Sep 10 13:28:34.185597 ip-10-0-140-8 kubenswrapper[2585]: W0910 13:28:34.183730 2585 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Sep 10 13:28:34.185597 ip-10-0-140-8 kubenswrapper[2585]: W0910 13:28:34.183733 2585 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Sep 10 13:28:34.185597 ip-10-0-140-8 kubenswrapper[2585]: W0910 13:28:34.183736 2585 feature_gate.go:328] unrecognized feature gate: DualReplica Sep 10 13:28:34.185597 ip-10-0-140-8 kubenswrapper[2585]: W0910 13:28:34.183738 2585 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Sep 10 13:28:34.185597 ip-10-0-140-8 kubenswrapper[2585]: W0910 13:28:34.183741 2585 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Sep 10 13:28:34.185597 ip-10-0-140-8 kubenswrapper[2585]: W0910 13:28:34.183743 2585 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Sep 10 13:28:34.185597 ip-10-0-140-8 kubenswrapper[2585]: W0910 13:28:34.183747 2585 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Sep 10 13:28:34.185597 ip-10-0-140-8 kubenswrapper[2585]: W0910 13:28:34.183750 2585 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Sep 10 13:28:34.185597 ip-10-0-140-8 kubenswrapper[2585]: W0910 13:28:34.183752 2585 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Sep 10 13:28:34.185597 ip-10-0-140-8 kubenswrapper[2585]: W0910 13:28:34.183755 2585 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Sep 10 13:28:34.185597 ip-10-0-140-8 kubenswrapper[2585]: W0910 13:28:34.183758 2585 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Sep 10 13:28:34.185597 ip-10-0-140-8 kubenswrapper[2585]: W0910 13:28:34.183760 2585 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Sep 10 13:28:34.185597 ip-10-0-140-8 kubenswrapper[2585]: W0910 13:28:34.183763 2585 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Sep 10 13:28:34.186090 ip-10-0-140-8 kubenswrapper[2585]: W0910 13:28:34.183766 2585 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Sep 10 13:28:34.186090 ip-10-0-140-8 kubenswrapper[2585]: W0910 13:28:34.183768 2585 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Sep 10 13:28:34.186090 ip-10-0-140-8 kubenswrapper[2585]: W0910 13:28:34.183771 2585 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Sep 10 13:28:34.186090 ip-10-0-140-8 kubenswrapper[2585]: W0910 13:28:34.183774 2585 feature_gate.go:328] unrecognized feature gate: Example2 Sep 10 13:28:34.186090 ip-10-0-140-8 kubenswrapper[2585]: W0910 13:28:34.183776 2585 feature_gate.go:328] unrecognized feature gate: Example Sep 10 13:28:34.186090 ip-10-0-140-8 kubenswrapper[2585]: W0910 13:28:34.183778 2585 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Sep 10 13:28:34.186090 ip-10-0-140-8 kubenswrapper[2585]: W0910 13:28:34.183781 2585 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Sep 10 13:28:34.186090 ip-10-0-140-8 kubenswrapper[2585]: W0910 13:28:34.183783 2585 feature_gate.go:328] unrecognized feature gate: EventTTL Sep 10 13:28:34.186090 ip-10-0-140-8 kubenswrapper[2585]: W0910 13:28:34.183786 2585 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Sep 10 13:28:34.186090 ip-10-0-140-8 kubenswrapper[2585]: W0910 13:28:34.183788 2585 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Sep 10 13:28:34.186090 ip-10-0-140-8 kubenswrapper[2585]: W0910 13:28:34.183793 2585 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Sep 10 13:28:34.186090 ip-10-0-140-8 kubenswrapper[2585]: W0910 13:28:34.183796 2585 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Sep 10 13:28:34.186090 ip-10-0-140-8 kubenswrapper[2585]: W0910 13:28:34.183799 2585 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Sep 10 13:28:34.186090 ip-10-0-140-8 kubenswrapper[2585]: W0910 13:28:34.183802 2585 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Sep 10 13:28:34.186090 ip-10-0-140-8 kubenswrapper[2585]: W0910 13:28:34.183805 2585 feature_gate.go:328] unrecognized feature gate: OSStreams Sep 10 13:28:34.186090 ip-10-0-140-8 kubenswrapper[2585]: W0910 13:28:34.183808 2585 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Sep 10 13:28:34.186090 ip-10-0-140-8 kubenswrapper[2585]: W0910 13:28:34.183810 2585 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Sep 10 13:28:34.186090 ip-10-0-140-8 kubenswrapper[2585]: W0910 13:28:34.183813 2585 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Sep 10 13:28:34.186090 ip-10-0-140-8 kubenswrapper[2585]: W0910 13:28:34.183816 2585 feature_gate.go:328] unrecognized feature gate: SignatureStores Sep 10 13:28:34.186587 ip-10-0-140-8 kubenswrapper[2585]: W0910 13:28:34.183819 2585 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Sep 10 13:28:34.186587 ip-10-0-140-8 kubenswrapper[2585]: W0910 13:28:34.183821 2585 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Sep 10 13:28:34.186587 ip-10-0-140-8 kubenswrapper[2585]: W0910 13:28:34.183824 2585 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Sep 10 13:28:34.186587 ip-10-0-140-8 kubenswrapper[2585]: W0910 13:28:34.183827 2585 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Sep 10 13:28:34.186587 ip-10-0-140-8 kubenswrapper[2585]: W0910 13:28:34.183829 2585 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Sep 10 13:28:34.186587 ip-10-0-140-8 kubenswrapper[2585]: W0910 13:28:34.183832 2585 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Sep 10 13:28:34.186587 ip-10-0-140-8 kubenswrapper[2585]: W0910 13:28:34.183835 2585 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Sep 10 13:28:34.186587 ip-10-0-140-8 kubenswrapper[2585]: W0910 13:28:34.183839 2585 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Sep 10 13:28:34.186587 ip-10-0-140-8 kubenswrapper[2585]: W0910 13:28:34.183841 2585 feature_gate.go:328] unrecognized feature gate: InsightsConfig Sep 10 13:28:34.186587 ip-10-0-140-8 kubenswrapper[2585]: W0910 13:28:34.183844 2585 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Sep 10 13:28:34.186587 ip-10-0-140-8 kubenswrapper[2585]: W0910 13:28:34.183846 2585 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Sep 10 13:28:34.186587 ip-10-0-140-8 kubenswrapper[2585]: W0910 13:28:34.183849 2585 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Sep 10 13:28:34.186587 ip-10-0-140-8 kubenswrapper[2585]: W0910 13:28:34.183853 2585 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Sep 10 13:28:34.186587 ip-10-0-140-8 kubenswrapper[2585]: W0910 13:28:34.183856 2585 feature_gate.go:328] unrecognized feature gate: NewOLM Sep 10 13:28:34.186587 ip-10-0-140-8 kubenswrapper[2585]: W0910 13:28:34.183859 2585 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Sep 10 13:28:34.186587 ip-10-0-140-8 kubenswrapper[2585]: W0910 13:28:34.183863 2585 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Sep 10 13:28:34.186587 ip-10-0-140-8 kubenswrapper[2585]: W0910 13:28:34.183865 2585 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Sep 10 13:28:34.186587 ip-10-0-140-8 kubenswrapper[2585]: W0910 13:28:34.183868 2585 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Sep 10 13:28:34.186587 ip-10-0-140-8 kubenswrapper[2585]: W0910 13:28:34.183870 2585 feature_gate.go:328] unrecognized feature gate: GatewayAPI Sep 10 13:28:34.186587 ip-10-0-140-8 kubenswrapper[2585]: W0910 13:28:34.183873 2585 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Sep 10 13:28:34.187186 ip-10-0-140-8 kubenswrapper[2585]: W0910 13:28:34.183875 2585 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Sep 10 13:28:34.187186 ip-10-0-140-8 kubenswrapper[2585]: W0910 13:28:34.183878 2585 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Sep 10 13:28:34.187186 ip-10-0-140-8 kubenswrapper[2585]: W0910 13:28:34.183881 2585 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Sep 10 13:28:34.187186 ip-10-0-140-8 kubenswrapper[2585]: W0910 13:28:34.183883 2585 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Sep 10 13:28:34.187186 ip-10-0-140-8 kubenswrapper[2585]: W0910 13:28:34.183885 2585 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Sep 10 13:28:34.187186 ip-10-0-140-8 kubenswrapper[2585]: W0910 13:28:34.183888 2585 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Sep 10 13:28:34.187186 ip-10-0-140-8 kubenswrapper[2585]: W0910 13:28:34.183890 2585 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Sep 10 13:28:34.187186 ip-10-0-140-8 kubenswrapper[2585]: W0910 13:28:34.183893 2585 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Sep 10 13:28:34.187186 ip-10-0-140-8 kubenswrapper[2585]: W0910 13:28:34.183896 2585 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Sep 10 13:28:34.187186 ip-10-0-140-8 kubenswrapper[2585]: W0910 13:28:34.183898 2585 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Sep 10 13:28:34.187186 ip-10-0-140-8 kubenswrapper[2585]: W0910 13:28:34.183901 2585 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Sep 10 13:28:34.187186 ip-10-0-140-8 kubenswrapper[2585]: W0910 13:28:34.183903 2585 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Sep 10 13:28:34.187186 ip-10-0-140-8 kubenswrapper[2585]: W0910 13:28:34.183906 2585 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Sep 10 13:28:34.187186 ip-10-0-140-8 kubenswrapper[2585]: W0910 13:28:34.183908 2585 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Sep 10 13:28:34.187186 ip-10-0-140-8 kubenswrapper[2585]: W0910 13:28:34.183911 2585 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Sep 10 13:28:34.187186 ip-10-0-140-8 kubenswrapper[2585]: W0910 13:28:34.183913 2585 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Sep 10 13:28:34.187186 ip-10-0-140-8 kubenswrapper[2585]: W0910 13:28:34.183916 2585 feature_gate.go:328] unrecognized feature gate: KMSEncryption Sep 10 13:28:34.187186 ip-10-0-140-8 kubenswrapper[2585]: W0910 13:28:34.183918 2585 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Sep 10 13:28:34.187186 ip-10-0-140-8 kubenswrapper[2585]: W0910 13:28:34.183921 2585 feature_gate.go:328] unrecognized feature gate: PinnedImages Sep 10 13:28:34.187186 ip-10-0-140-8 kubenswrapper[2585]: W0910 13:28:34.183924 2585 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Sep 10 13:28:34.187712 ip-10-0-140-8 kubenswrapper[2585]: W0910 13:28:34.183926 2585 feature_gate.go:328] unrecognized feature gate: OVNObservability Sep 10 13:28:34.187712 ip-10-0-140-8 kubenswrapper[2585]: W0910 13:28:34.184333 2585 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Sep 10 13:28:34.187712 ip-10-0-140-8 kubenswrapper[2585]: W0910 13:28:34.184339 2585 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Sep 10 13:28:34.187712 ip-10-0-140-8 kubenswrapper[2585]: W0910 13:28:34.184342 2585 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Sep 10 13:28:34.187712 ip-10-0-140-8 kubenswrapper[2585]: W0910 13:28:34.184344 2585 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Sep 10 13:28:34.187712 ip-10-0-140-8 kubenswrapper[2585]: W0910 13:28:34.184348 2585 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Sep 10 13:28:34.187712 ip-10-0-140-8 kubenswrapper[2585]: W0910 13:28:34.184351 2585 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Sep 10 13:28:34.187712 ip-10-0-140-8 kubenswrapper[2585]: W0910 13:28:34.184354 2585 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Sep 10 13:28:34.187712 ip-10-0-140-8 kubenswrapper[2585]: W0910 13:28:34.184357 2585 feature_gate.go:328] unrecognized feature gate: OSStreams Sep 10 13:28:34.187712 ip-10-0-140-8 kubenswrapper[2585]: W0910 13:28:34.184359 2585 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Sep 10 13:28:34.187712 ip-10-0-140-8 kubenswrapper[2585]: W0910 13:28:34.184362 2585 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Sep 10 13:28:34.187712 ip-10-0-140-8 kubenswrapper[2585]: W0910 13:28:34.184364 2585 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Sep 10 13:28:34.187712 ip-10-0-140-8 kubenswrapper[2585]: W0910 13:28:34.184367 2585 feature_gate.go:328] unrecognized feature gate: SignatureStores Sep 10 13:28:34.187712 ip-10-0-140-8 kubenswrapper[2585]: W0910 13:28:34.184369 2585 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Sep 10 13:28:34.187712 ip-10-0-140-8 kubenswrapper[2585]: W0910 13:28:34.184372 2585 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Sep 10 13:28:34.187712 ip-10-0-140-8 kubenswrapper[2585]: W0910 13:28:34.184374 2585 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Sep 10 13:28:34.187712 ip-10-0-140-8 kubenswrapper[2585]: W0910 13:28:34.184377 2585 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Sep 10 13:28:34.187712 ip-10-0-140-8 kubenswrapper[2585]: W0910 13:28:34.184379 2585 feature_gate.go:328] unrecognized feature gate: EventTTL Sep 10 13:28:34.187712 ip-10-0-140-8 kubenswrapper[2585]: W0910 13:28:34.184382 2585 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Sep 10 13:28:34.187712 ip-10-0-140-8 kubenswrapper[2585]: W0910 13:28:34.184385 2585 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Sep 10 13:28:34.188201 ip-10-0-140-8 kubenswrapper[2585]: W0910 13:28:34.184387 2585 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Sep 10 13:28:34.188201 ip-10-0-140-8 kubenswrapper[2585]: W0910 13:28:34.184390 2585 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Sep 10 13:28:34.188201 ip-10-0-140-8 kubenswrapper[2585]: W0910 13:28:34.184393 2585 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Sep 10 13:28:34.188201 ip-10-0-140-8 kubenswrapper[2585]: W0910 13:28:34.184395 2585 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Sep 10 13:28:34.188201 ip-10-0-140-8 kubenswrapper[2585]: W0910 13:28:34.184398 2585 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Sep 10 13:28:34.188201 ip-10-0-140-8 kubenswrapper[2585]: W0910 13:28:34.184400 2585 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Sep 10 13:28:34.188201 ip-10-0-140-8 kubenswrapper[2585]: W0910 13:28:34.184403 2585 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Sep 10 13:28:34.188201 ip-10-0-140-8 kubenswrapper[2585]: W0910 13:28:34.184405 2585 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Sep 10 13:28:34.188201 ip-10-0-140-8 kubenswrapper[2585]: W0910 13:28:34.184408 2585 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Sep 10 13:28:34.188201 ip-10-0-140-8 kubenswrapper[2585]: W0910 13:28:34.184410 2585 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Sep 10 13:28:34.188201 ip-10-0-140-8 kubenswrapper[2585]: W0910 13:28:34.184413 2585 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Sep 10 13:28:34.188201 ip-10-0-140-8 kubenswrapper[2585]: W0910 13:28:34.184416 2585 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Sep 10 13:28:34.188201 ip-10-0-140-8 kubenswrapper[2585]: W0910 13:28:34.184418 2585 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Sep 10 13:28:34.188201 ip-10-0-140-8 kubenswrapper[2585]: W0910 13:28:34.184421 2585 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Sep 10 13:28:34.188201 ip-10-0-140-8 kubenswrapper[2585]: W0910 13:28:34.184423 2585 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Sep 10 13:28:34.188201 ip-10-0-140-8 kubenswrapper[2585]: W0910 13:28:34.184426 2585 feature_gate.go:328] unrecognized feature gate: NewOLM Sep 10 13:28:34.188201 ip-10-0-140-8 kubenswrapper[2585]: W0910 13:28:34.184429 2585 feature_gate.go:328] unrecognized feature gate: DualReplica Sep 10 13:28:34.188201 ip-10-0-140-8 kubenswrapper[2585]: W0910 13:28:34.184431 2585 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Sep 10 13:28:34.188201 ip-10-0-140-8 kubenswrapper[2585]: W0910 13:28:34.184434 2585 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Sep 10 13:28:34.188708 ip-10-0-140-8 kubenswrapper[2585]: W0910 13:28:34.184437 2585 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Sep 10 13:28:34.188708 ip-10-0-140-8 kubenswrapper[2585]: W0910 13:28:34.184439 2585 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Sep 10 13:28:34.188708 ip-10-0-140-8 kubenswrapper[2585]: W0910 13:28:34.184442 2585 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Sep 10 13:28:34.188708 ip-10-0-140-8 kubenswrapper[2585]: W0910 13:28:34.184444 2585 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Sep 10 13:28:34.188708 ip-10-0-140-8 kubenswrapper[2585]: W0910 13:28:34.184447 2585 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Sep 10 13:28:34.188708 ip-10-0-140-8 kubenswrapper[2585]: W0910 13:28:34.184449 2585 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Sep 10 13:28:34.188708 ip-10-0-140-8 kubenswrapper[2585]: W0910 13:28:34.184452 2585 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Sep 10 13:28:34.188708 ip-10-0-140-8 kubenswrapper[2585]: W0910 13:28:34.184454 2585 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Sep 10 13:28:34.188708 ip-10-0-140-8 kubenswrapper[2585]: W0910 13:28:34.184457 2585 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Sep 10 13:28:34.188708 ip-10-0-140-8 kubenswrapper[2585]: W0910 13:28:34.184459 2585 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Sep 10 13:28:34.188708 ip-10-0-140-8 kubenswrapper[2585]: W0910 13:28:34.184462 2585 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Sep 10 13:28:34.188708 ip-10-0-140-8 kubenswrapper[2585]: W0910 13:28:34.184466 2585 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Sep 10 13:28:34.188708 ip-10-0-140-8 kubenswrapper[2585]: W0910 13:28:34.184469 2585 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Sep 10 13:28:34.188708 ip-10-0-140-8 kubenswrapper[2585]: W0910 13:28:34.184472 2585 feature_gate.go:328] unrecognized feature gate: OVNObservability Sep 10 13:28:34.188708 ip-10-0-140-8 kubenswrapper[2585]: W0910 13:28:34.184475 2585 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Sep 10 13:28:34.188708 ip-10-0-140-8 kubenswrapper[2585]: W0910 13:28:34.184478 2585 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Sep 10 13:28:34.188708 ip-10-0-140-8 kubenswrapper[2585]: W0910 13:28:34.184480 2585 feature_gate.go:328] unrecognized feature gate: InsightsConfig Sep 10 13:28:34.188708 ip-10-0-140-8 kubenswrapper[2585]: W0910 13:28:34.184483 2585 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Sep 10 13:28:34.188708 ip-10-0-140-8 kubenswrapper[2585]: W0910 13:28:34.184488 2585 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Sep 10 13:28:34.189161 ip-10-0-140-8 kubenswrapper[2585]: W0910 13:28:34.184491 2585 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Sep 10 13:28:34.189161 ip-10-0-140-8 kubenswrapper[2585]: W0910 13:28:34.184494 2585 feature_gate.go:328] unrecognized feature gate: Example Sep 10 13:28:34.189161 ip-10-0-140-8 kubenswrapper[2585]: W0910 13:28:34.184497 2585 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Sep 10 13:28:34.189161 ip-10-0-140-8 kubenswrapper[2585]: W0910 13:28:34.184499 2585 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Sep 10 13:28:34.189161 ip-10-0-140-8 kubenswrapper[2585]: W0910 13:28:34.184505 2585 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Sep 10 13:28:34.189161 ip-10-0-140-8 kubenswrapper[2585]: W0910 13:28:34.184507 2585 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Sep 10 13:28:34.189161 ip-10-0-140-8 kubenswrapper[2585]: W0910 13:28:34.184510 2585 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Sep 10 13:28:34.189161 ip-10-0-140-8 kubenswrapper[2585]: W0910 13:28:34.184513 2585 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Sep 10 13:28:34.189161 ip-10-0-140-8 kubenswrapper[2585]: W0910 13:28:34.184516 2585 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Sep 10 13:28:34.189161 ip-10-0-140-8 kubenswrapper[2585]: W0910 13:28:34.184518 2585 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Sep 10 13:28:34.189161 ip-10-0-140-8 kubenswrapper[2585]: W0910 13:28:34.184521 2585 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Sep 10 13:28:34.189161 ip-10-0-140-8 kubenswrapper[2585]: W0910 13:28:34.184524 2585 feature_gate.go:328] unrecognized feature gate: GatewayAPI Sep 10 13:28:34.189161 ip-10-0-140-8 kubenswrapper[2585]: W0910 13:28:34.184526 2585 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Sep 10 13:28:34.189161 ip-10-0-140-8 kubenswrapper[2585]: W0910 13:28:34.184529 2585 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Sep 10 13:28:34.189161 ip-10-0-140-8 kubenswrapper[2585]: W0910 13:28:34.184532 2585 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Sep 10 13:28:34.189161 ip-10-0-140-8 kubenswrapper[2585]: W0910 13:28:34.184534 2585 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Sep 10 13:28:34.189161 ip-10-0-140-8 kubenswrapper[2585]: W0910 13:28:34.184537 2585 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Sep 10 13:28:34.189161 ip-10-0-140-8 kubenswrapper[2585]: W0910 13:28:34.184539 2585 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Sep 10 13:28:34.189161 ip-10-0-140-8 kubenswrapper[2585]: W0910 13:28:34.184542 2585 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Sep 10 13:28:34.189161 ip-10-0-140-8 kubenswrapper[2585]: W0910 13:28:34.184545 2585 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Sep 10 13:28:34.189657 ip-10-0-140-8 kubenswrapper[2585]: W0910 13:28:34.184547 2585 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Sep 10 13:28:34.189657 ip-10-0-140-8 kubenswrapper[2585]: W0910 13:28:34.184550 2585 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Sep 10 13:28:34.189657 ip-10-0-140-8 kubenswrapper[2585]: W0910 13:28:34.184552 2585 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Sep 10 13:28:34.189657 ip-10-0-140-8 kubenswrapper[2585]: W0910 13:28:34.184555 2585 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Sep 10 13:28:34.189657 ip-10-0-140-8 kubenswrapper[2585]: W0910 13:28:34.184559 2585 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Sep 10 13:28:34.189657 ip-10-0-140-8 kubenswrapper[2585]: W0910 13:28:34.184562 2585 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Sep 10 13:28:34.189657 ip-10-0-140-8 kubenswrapper[2585]: W0910 13:28:34.184565 2585 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Sep 10 13:28:34.189657 ip-10-0-140-8 kubenswrapper[2585]: W0910 13:28:34.184567 2585 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Sep 10 13:28:34.189657 ip-10-0-140-8 kubenswrapper[2585]: W0910 13:28:34.184570 2585 feature_gate.go:328] unrecognized feature gate: PinnedImages Sep 10 13:28:34.189657 ip-10-0-140-8 kubenswrapper[2585]: W0910 13:28:34.184572 2585 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Sep 10 13:28:34.189657 ip-10-0-140-8 kubenswrapper[2585]: W0910 13:28:34.184575 2585 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Sep 10 13:28:34.189657 ip-10-0-140-8 kubenswrapper[2585]: W0910 13:28:34.184577 2585 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Sep 10 13:28:34.189657 ip-10-0-140-8 kubenswrapper[2585]: W0910 13:28:34.184580 2585 feature_gate.go:328] unrecognized feature gate: KMSEncryption Sep 10 13:28:34.189657 ip-10-0-140-8 kubenswrapper[2585]: W0910 13:28:34.184583 2585 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Sep 10 13:28:34.189657 ip-10-0-140-8 kubenswrapper[2585]: W0910 13:28:34.184585 2585 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Sep 10 13:28:34.189657 ip-10-0-140-8 kubenswrapper[2585]: W0910 13:28:34.184588 2585 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Sep 10 13:28:34.189657 ip-10-0-140-8 kubenswrapper[2585]: W0910 13:28:34.184591 2585 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Sep 10 13:28:34.189657 ip-10-0-140-8 kubenswrapper[2585]: W0910 13:28:34.184594 2585 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Sep 10 13:28:34.189657 ip-10-0-140-8 kubenswrapper[2585]: W0910 13:28:34.184596 2585 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Sep 10 13:28:34.189657 ip-10-0-140-8 kubenswrapper[2585]: W0910 13:28:34.184599 2585 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Sep 10 13:28:34.190180 ip-10-0-140-8 kubenswrapper[2585]: W0910 13:28:34.184602 2585 feature_gate.go:328] unrecognized feature gate: Example2 Sep 10 13:28:34.190180 ip-10-0-140-8 kubenswrapper[2585]: W0910 13:28:34.184604 2585 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Sep 10 13:28:34.190180 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:28:34.184688 2585 flags.go:64] FLAG: --address="0.0.0.0" Sep 10 13:28:34.190180 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:28:34.184730 2585 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]" Sep 10 13:28:34.190180 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:28:34.184737 2585 flags.go:64] FLAG: --anonymous-auth="true" Sep 10 13:28:34.190180 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:28:34.184742 2585 flags.go:64] FLAG: --application-metrics-count-limit="100" Sep 10 13:28:34.190180 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:28:34.184748 2585 flags.go:64] FLAG: --authentication-token-webhook="false" Sep 10 13:28:34.190180 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:28:34.184751 2585 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s" Sep 10 13:28:34.190180 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:28:34.184756 2585 flags.go:64] FLAG: --authorization-mode="AlwaysAllow" Sep 10 13:28:34.190180 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:28:34.184760 2585 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s" Sep 10 13:28:34.190180 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:28:34.184764 2585 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s" Sep 10 13:28:34.190180 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:28:34.184767 2585 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id" Sep 10 13:28:34.190180 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:28:34.184770 2585 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig" Sep 10 13:28:34.190180 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:28:34.184773 2585 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki" Sep 10 13:28:34.190180 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:28:34.184777 2585 flags.go:64] FLAG: --cgroup-driver="cgroupfs" Sep 10 13:28:34.190180 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:28:34.184780 2585 flags.go:64] FLAG: --cgroup-root="" Sep 10 13:28:34.190180 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:28:34.184783 2585 flags.go:64] FLAG: --cgroups-per-qos="true" Sep 10 13:28:34.190180 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:28:34.184786 2585 flags.go:64] FLAG: --client-ca-file="" Sep 10 13:28:34.190180 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:28:34.184788 2585 flags.go:64] FLAG: --cloud-provider="external" Sep 10 13:28:34.190180 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:28:34.184791 2585 flags.go:64] FLAG: --cluster-dns="[]" Sep 10 13:28:34.190180 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:28:34.184796 2585 flags.go:64] FLAG: --cluster-domain="" Sep 10 13:28:34.190180 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:28:34.184799 2585 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf" Sep 10 13:28:34.190180 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:28:34.184802 2585 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d" Sep 10 13:28:34.190180 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:28:34.184805 2585 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json" Sep 10 13:28:34.190773 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:28:34.184810 2585 flags.go:64] FLAG: --container-log-max-files="5" Sep 10 13:28:34.190773 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:28:34.184814 2585 flags.go:64] FLAG: --container-log-max-size="10Mi" Sep 10 13:28:34.190773 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:28:34.184817 2585 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock" Sep 10 13:28:34.190773 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:28:34.184821 2585 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock" Sep 10 13:28:34.190773 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:28:34.184825 2585 flags.go:64] FLAG: --containerd-namespace="k8s.io" Sep 10 13:28:34.190773 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:28:34.184828 2585 flags.go:64] FLAG: --contention-profiling="false" Sep 10 13:28:34.190773 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:28:34.184831 2585 flags.go:64] FLAG: --cpu-cfs-quota="true" Sep 10 13:28:34.190773 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:28:34.184834 2585 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms" Sep 10 13:28:34.190773 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:28:34.184837 2585 flags.go:64] FLAG: --cpu-manager-policy="none" Sep 10 13:28:34.190773 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:28:34.184840 2585 flags.go:64] FLAG: --cpu-manager-policy-options="" Sep 10 13:28:34.190773 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:28:34.184844 2585 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s" Sep 10 13:28:34.190773 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:28:34.184847 2585 flags.go:64] FLAG: --enable-controller-attach-detach="true" Sep 10 13:28:34.190773 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:28:34.184850 2585 flags.go:64] FLAG: --enable-debugging-handlers="true" Sep 10 13:28:34.190773 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:28:34.184853 2585 flags.go:64] FLAG: --enable-load-reader="false" Sep 10 13:28:34.190773 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:28:34.184856 2585 flags.go:64] FLAG: --enable-server="true" Sep 10 13:28:34.190773 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:28:34.184860 2585 flags.go:64] FLAG: --enforce-node-allocatable="[pods]" Sep 10 13:28:34.190773 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:28:34.184864 2585 flags.go:64] FLAG: --event-burst="100" Sep 10 13:28:34.190773 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:28:34.184867 2585 flags.go:64] FLAG: --event-qps="50" Sep 10 13:28:34.190773 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:28:34.184870 2585 flags.go:64] FLAG: --event-storage-age-limit="default=0" Sep 10 13:28:34.190773 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:28:34.184873 2585 flags.go:64] FLAG: --event-storage-event-limit="default=0" Sep 10 13:28:34.190773 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:28:34.184876 2585 flags.go:64] FLAG: --eviction-hard="" Sep 10 13:28:34.190773 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:28:34.184879 2585 flags.go:64] FLAG: --eviction-max-pod-grace-period="0" Sep 10 13:28:34.190773 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:28:34.184883 2585 flags.go:64] FLAG: --eviction-minimum-reclaim="" Sep 10 13:28:34.190773 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:28:34.184886 2585 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s" Sep 10 13:28:34.190773 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:28:34.184889 2585 flags.go:64] FLAG: --eviction-soft="" Sep 10 13:28:34.191398 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:28:34.184892 2585 flags.go:64] FLAG: --eviction-soft-grace-period="" Sep 10 13:28:34.191398 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:28:34.184895 2585 flags.go:64] FLAG: --exit-on-lock-contention="false" Sep 10 13:28:34.191398 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:28:34.184897 2585 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false" Sep 10 13:28:34.191398 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:28:34.184900 2585 flags.go:64] FLAG: --experimental-mounter-path="" Sep 10 13:28:34.191398 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:28:34.184903 2585 flags.go:64] FLAG: --fail-cgroupv1="false" Sep 10 13:28:34.191398 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:28:34.184906 2585 flags.go:64] FLAG: --fail-swap-on="true" Sep 10 13:28:34.191398 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:28:34.184909 2585 flags.go:64] FLAG: --feature-gates="" Sep 10 13:28:34.191398 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:28:34.184914 2585 flags.go:64] FLAG: --file-check-frequency="20s" Sep 10 13:28:34.191398 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:28:34.184918 2585 flags.go:64] FLAG: --global-housekeeping-interval="1m0s" Sep 10 13:28:34.191398 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:28:34.184921 2585 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge" Sep 10 13:28:34.191398 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:28:34.184924 2585 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1" Sep 10 13:28:34.191398 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:28:34.184929 2585 flags.go:64] FLAG: --healthz-port="10248" Sep 10 13:28:34.191398 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:28:34.184932 2585 flags.go:64] FLAG: --help="false" Sep 10 13:28:34.191398 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:28:34.184934 2585 flags.go:64] FLAG: --hostname-override="ip-10-0-140-8.ec2.internal" Sep 10 13:28:34.191398 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:28:34.184938 2585 flags.go:64] FLAG: --housekeeping-interval="10s" Sep 10 13:28:34.191398 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:28:34.184941 2585 flags.go:64] FLAG: --http-check-frequency="20s" Sep 10 13:28:34.191398 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:28:34.184944 2585 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins" Sep 10 13:28:34.191398 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:28:34.184947 2585 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml" Sep 10 13:28:34.191398 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:28:34.184950 2585 flags.go:64] FLAG: --image-gc-high-threshold="85" Sep 10 13:28:34.191398 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:28:34.184953 2585 flags.go:64] FLAG: --image-gc-low-threshold="80" Sep 10 13:28:34.191398 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:28:34.184956 2585 flags.go:64] FLAG: --image-service-endpoint="" Sep 10 13:28:34.191398 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:28:34.184959 2585 flags.go:64] FLAG: --kernel-memcg-notification="false" Sep 10 13:28:34.191398 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:28:34.184962 2585 flags.go:64] FLAG: --kube-api-burst="100" Sep 10 13:28:34.191988 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:28:34.184965 2585 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf" Sep 10 13:28:34.191988 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:28:34.184969 2585 flags.go:64] FLAG: --kube-api-qps="50" Sep 10 13:28:34.191988 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:28:34.184972 2585 flags.go:64] FLAG: --kube-reserved="" Sep 10 13:28:34.191988 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:28:34.184975 2585 flags.go:64] FLAG: --kube-reserved-cgroup="" Sep 10 13:28:34.191988 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:28:34.184978 2585 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig" Sep 10 13:28:34.191988 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:28:34.184981 2585 flags.go:64] FLAG: --kubelet-cgroups="" Sep 10 13:28:34.191988 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:28:34.184984 2585 flags.go:64] FLAG: --local-storage-capacity-isolation="true" Sep 10 13:28:34.191988 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:28:34.184987 2585 flags.go:64] FLAG: --lock-file="" Sep 10 13:28:34.191988 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:28:34.184990 2585 flags.go:64] FLAG: --log-cadvisor-usage="false" Sep 10 13:28:34.191988 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:28:34.184993 2585 flags.go:64] FLAG: --log-flush-frequency="5s" Sep 10 13:28:34.191988 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:28:34.184996 2585 flags.go:64] FLAG: --log-json-info-buffer-size="0" Sep 10 13:28:34.191988 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:28:34.185002 2585 flags.go:64] FLAG: --log-json-split-stream="false" Sep 10 13:28:34.191988 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:28:34.185005 2585 flags.go:64] FLAG: --log-text-info-buffer-size="0" Sep 10 13:28:34.191988 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:28:34.185008 2585 flags.go:64] FLAG: --log-text-split-stream="false" Sep 10 13:28:34.191988 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:28:34.185010 2585 flags.go:64] FLAG: --logging-format="text" Sep 10 13:28:34.191988 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:28:34.185013 2585 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id" Sep 10 13:28:34.191988 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:28:34.185018 2585 flags.go:64] FLAG: --make-iptables-util-chains="true" Sep 10 13:28:34.191988 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:28:34.185021 2585 flags.go:64] FLAG: --manifest-url="" Sep 10 13:28:34.191988 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:28:34.185023 2585 flags.go:64] FLAG: --manifest-url-header="" Sep 10 13:28:34.191988 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:28:34.185028 2585 flags.go:64] FLAG: --max-housekeeping-interval="15s" Sep 10 13:28:34.191988 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:28:34.185033 2585 flags.go:64] FLAG: --max-open-files="1000000" Sep 10 13:28:34.191988 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:28:34.185037 2585 flags.go:64] FLAG: --max-pods="110" Sep 10 13:28:34.191988 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:28:34.185040 2585 flags.go:64] FLAG: --maximum-dead-containers="-1" Sep 10 13:28:34.191988 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:28:34.185043 2585 flags.go:64] FLAG: --maximum-dead-containers-per-container="1" Sep 10 13:28:34.191988 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:28:34.185046 2585 flags.go:64] FLAG: --memory-manager-policy="None" Sep 10 13:28:34.192594 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:28:34.185049 2585 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s" Sep 10 13:28:34.192594 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:28:34.185052 2585 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s" Sep 10 13:28:34.192594 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:28:34.185055 2585 flags.go:64] FLAG: --node-ip="0.0.0.0" Sep 10 13:28:34.192594 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:28:34.185058 2585 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel" Sep 10 13:28:34.192594 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:28:34.185065 2585 flags.go:64] FLAG: --node-status-max-images="50" Sep 10 13:28:34.192594 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:28:34.185068 2585 flags.go:64] FLAG: --node-status-update-frequency="10s" Sep 10 13:28:34.192594 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:28:34.185071 2585 flags.go:64] FLAG: --oom-score-adj="-999" Sep 10 13:28:34.192594 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:28:34.185074 2585 flags.go:64] FLAG: --pod-cidr="" Sep 10 13:28:34.192594 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:28:34.185077 2585 flags.go:64] FLAG: --pod-infra-container-image="quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:8ccbacc999dbf9abad8f4009240188312b683f36ced4763eed166c1500c164b1" Sep 10 13:28:34.192594 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:28:34.185083 2585 flags.go:64] FLAG: --pod-manifest-path="" Sep 10 13:28:34.192594 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:28:34.185086 2585 flags.go:64] FLAG: --pod-max-pids="-1" Sep 10 13:28:34.192594 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:28:34.185089 2585 flags.go:64] FLAG: --pods-per-core="0" Sep 10 13:28:34.192594 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:28:34.185091 2585 flags.go:64] FLAG: --port="10250" Sep 10 13:28:34.192594 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:28:34.185094 2585 flags.go:64] FLAG: --protect-kernel-defaults="false" Sep 10 13:28:34.192594 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:28:34.185097 2585 flags.go:64] FLAG: --provider-id="aws:///us-east-1a/i-0ab762a9837fac5c8" Sep 10 13:28:34.192594 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:28:34.185100 2585 flags.go:64] FLAG: --qos-reserved="" Sep 10 13:28:34.192594 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:28:34.185103 2585 flags.go:64] FLAG: --read-only-port="10255" Sep 10 13:28:34.192594 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:28:34.185107 2585 flags.go:64] FLAG: --register-node="true" Sep 10 13:28:34.192594 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:28:34.185109 2585 flags.go:64] FLAG: --register-with-taints="" Sep 10 13:28:34.192594 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:28:34.185113 2585 flags.go:64] FLAG: --registry-burst="10" Sep 10 13:28:34.192594 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:28:34.185116 2585 flags.go:64] FLAG: --registry-qps="5" Sep 10 13:28:34.192594 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:28:34.185119 2585 flags.go:64] FLAG: --reserved-cpus="" Sep 10 13:28:34.192594 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:28:34.185122 2585 flags.go:64] FLAG: --reserved-memory="" Sep 10 13:28:34.192594 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:28:34.185127 2585 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf" Sep 10 13:28:34.192594 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:28:34.185130 2585 flags.go:64] FLAG: --root-dir="/var/lib/kubelet" Sep 10 13:28:34.193205 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:28:34.185133 2585 flags.go:64] FLAG: --rotate-certificates="false" Sep 10 13:28:34.193205 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:28:34.185136 2585 flags.go:64] FLAG: --rotate-server-certificates="false" Sep 10 13:28:34.193205 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:28:34.185139 2585 flags.go:64] FLAG: --runonce="false" Sep 10 13:28:34.193205 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:28:34.185142 2585 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service" Sep 10 13:28:34.193205 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:28:34.185145 2585 flags.go:64] FLAG: --runtime-request-timeout="2m0s" Sep 10 13:28:34.193205 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:28:34.185149 2585 flags.go:64] FLAG: --seccomp-default="false" Sep 10 13:28:34.193205 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:28:34.185151 2585 flags.go:64] FLAG: --serialize-image-pulls="true" Sep 10 13:28:34.193205 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:28:34.185154 2585 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s" Sep 10 13:28:34.193205 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:28:34.185158 2585 flags.go:64] FLAG: --storage-driver-db="cadvisor" Sep 10 13:28:34.193205 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:28:34.185160 2585 flags.go:64] FLAG: --storage-driver-host="localhost:8086" Sep 10 13:28:34.193205 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:28:34.185164 2585 flags.go:64] FLAG: --storage-driver-password="root" Sep 10 13:28:34.193205 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:28:34.185166 2585 flags.go:64] FLAG: --storage-driver-secure="false" Sep 10 13:28:34.193205 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:28:34.185169 2585 flags.go:64] FLAG: --storage-driver-table="stats" Sep 10 13:28:34.193205 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:28:34.185172 2585 flags.go:64] FLAG: --storage-driver-user="root" Sep 10 13:28:34.193205 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:28:34.185175 2585 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s" Sep 10 13:28:34.193205 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:28:34.185178 2585 flags.go:64] FLAG: --sync-frequency="1m0s" Sep 10 13:28:34.193205 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:28:34.185182 2585 flags.go:64] FLAG: --system-cgroups="" Sep 10 13:28:34.193205 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:28:34.185198 2585 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi" Sep 10 13:28:34.193205 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:28:34.185205 2585 flags.go:64] FLAG: --system-reserved-cgroup="" Sep 10 13:28:34.193205 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:28:34.185208 2585 flags.go:64] FLAG: --tls-cert-file="" Sep 10 13:28:34.193205 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:28:34.185212 2585 flags.go:64] FLAG: --tls-cipher-suites="[]" Sep 10 13:28:34.193205 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:28:34.185216 2585 flags.go:64] FLAG: --tls-min-version="" Sep 10 13:28:34.193205 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:28:34.185219 2585 flags.go:64] FLAG: --tls-private-key-file="" Sep 10 13:28:34.193205 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:28:34.185222 2585 flags.go:64] FLAG: --topology-manager-policy="none" Sep 10 13:28:34.193205 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:28:34.185225 2585 flags.go:64] FLAG: --topology-manager-policy-options="" Sep 10 13:28:34.193871 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:28:34.185228 2585 flags.go:64] FLAG: --topology-manager-scope="container" Sep 10 13:28:34.193871 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:28:34.185231 2585 flags.go:64] FLAG: --v="2" Sep 10 13:28:34.193871 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:28:34.185235 2585 flags.go:64] FLAG: --version="false" Sep 10 13:28:34.193871 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:28:34.185239 2585 flags.go:64] FLAG: --vmodule="" Sep 10 13:28:34.193871 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:28:34.185243 2585 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec" Sep 10 13:28:34.193871 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:28:34.185246 2585 flags.go:64] FLAG: --volume-stats-agg-period="1m0s" Sep 10 13:28:34.193871 ip-10-0-140-8 kubenswrapper[2585]: W0910 13:28:34.185397 2585 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Sep 10 13:28:34.193871 ip-10-0-140-8 kubenswrapper[2585]: W0910 13:28:34.185401 2585 feature_gate.go:328] unrecognized feature gate: InsightsConfig Sep 10 13:28:34.193871 ip-10-0-140-8 kubenswrapper[2585]: W0910 13:28:34.185405 2585 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Sep 10 13:28:34.193871 ip-10-0-140-8 kubenswrapper[2585]: W0910 13:28:34.185408 2585 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Sep 10 13:28:34.193871 ip-10-0-140-8 kubenswrapper[2585]: W0910 13:28:34.185410 2585 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Sep 10 13:28:34.193871 ip-10-0-140-8 kubenswrapper[2585]: W0910 13:28:34.185413 2585 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Sep 10 13:28:34.193871 ip-10-0-140-8 kubenswrapper[2585]: W0910 13:28:34.185416 2585 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Sep 10 13:28:34.193871 ip-10-0-140-8 kubenswrapper[2585]: W0910 13:28:34.185418 2585 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Sep 10 13:28:34.193871 ip-10-0-140-8 kubenswrapper[2585]: W0910 13:28:34.185421 2585 feature_gate.go:328] unrecognized feature gate: KMSEncryption Sep 10 13:28:34.193871 ip-10-0-140-8 kubenswrapper[2585]: W0910 13:28:34.185424 2585 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Sep 10 13:28:34.193871 ip-10-0-140-8 kubenswrapper[2585]: W0910 13:28:34.185426 2585 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Sep 10 13:28:34.193871 ip-10-0-140-8 kubenswrapper[2585]: W0910 13:28:34.185429 2585 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Sep 10 13:28:34.193871 ip-10-0-140-8 kubenswrapper[2585]: W0910 13:28:34.185432 2585 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Sep 10 13:28:34.193871 ip-10-0-140-8 kubenswrapper[2585]: W0910 13:28:34.185434 2585 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Sep 10 13:28:34.193871 ip-10-0-140-8 kubenswrapper[2585]: W0910 13:28:34.185437 2585 feature_gate.go:328] unrecognized feature gate: SignatureStores Sep 10 13:28:34.193871 ip-10-0-140-8 kubenswrapper[2585]: W0910 13:28:34.185439 2585 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Sep 10 13:28:34.194430 ip-10-0-140-8 kubenswrapper[2585]: W0910 13:28:34.185442 2585 feature_gate.go:328] unrecognized feature gate: NewOLM Sep 10 13:28:34.194430 ip-10-0-140-8 kubenswrapper[2585]: W0910 13:28:34.185444 2585 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Sep 10 13:28:34.194430 ip-10-0-140-8 kubenswrapper[2585]: W0910 13:28:34.185448 2585 feature_gate.go:328] unrecognized feature gate: GatewayAPI Sep 10 13:28:34.194430 ip-10-0-140-8 kubenswrapper[2585]: W0910 13:28:34.185450 2585 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Sep 10 13:28:34.194430 ip-10-0-140-8 kubenswrapper[2585]: W0910 13:28:34.185453 2585 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Sep 10 13:28:34.194430 ip-10-0-140-8 kubenswrapper[2585]: W0910 13:28:34.185456 2585 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Sep 10 13:28:34.194430 ip-10-0-140-8 kubenswrapper[2585]: W0910 13:28:34.185458 2585 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Sep 10 13:28:34.194430 ip-10-0-140-8 kubenswrapper[2585]: W0910 13:28:34.185461 2585 feature_gate.go:328] unrecognized feature gate: OSStreams Sep 10 13:28:34.194430 ip-10-0-140-8 kubenswrapper[2585]: W0910 13:28:34.185463 2585 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Sep 10 13:28:34.194430 ip-10-0-140-8 kubenswrapper[2585]: W0910 13:28:34.185466 2585 feature_gate.go:328] unrecognized feature gate: DualReplica Sep 10 13:28:34.194430 ip-10-0-140-8 kubenswrapper[2585]: W0910 13:28:34.185468 2585 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Sep 10 13:28:34.194430 ip-10-0-140-8 kubenswrapper[2585]: W0910 13:28:34.185471 2585 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Sep 10 13:28:34.194430 ip-10-0-140-8 kubenswrapper[2585]: W0910 13:28:34.185473 2585 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Sep 10 13:28:34.194430 ip-10-0-140-8 kubenswrapper[2585]: W0910 13:28:34.185476 2585 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Sep 10 13:28:34.194430 ip-10-0-140-8 kubenswrapper[2585]: W0910 13:28:34.185478 2585 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Sep 10 13:28:34.194430 ip-10-0-140-8 kubenswrapper[2585]: W0910 13:28:34.185481 2585 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Sep 10 13:28:34.194430 ip-10-0-140-8 kubenswrapper[2585]: W0910 13:28:34.185483 2585 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Sep 10 13:28:34.194430 ip-10-0-140-8 kubenswrapper[2585]: W0910 13:28:34.185486 2585 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Sep 10 13:28:34.194430 ip-10-0-140-8 kubenswrapper[2585]: W0910 13:28:34.185489 2585 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Sep 10 13:28:34.194430 ip-10-0-140-8 kubenswrapper[2585]: W0910 13:28:34.185492 2585 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Sep 10 13:28:34.194946 ip-10-0-140-8 kubenswrapper[2585]: W0910 13:28:34.185494 2585 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Sep 10 13:28:34.194946 ip-10-0-140-8 kubenswrapper[2585]: W0910 13:28:34.185497 2585 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Sep 10 13:28:34.194946 ip-10-0-140-8 kubenswrapper[2585]: W0910 13:28:34.185499 2585 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Sep 10 13:28:34.194946 ip-10-0-140-8 kubenswrapper[2585]: W0910 13:28:34.185502 2585 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Sep 10 13:28:34.194946 ip-10-0-140-8 kubenswrapper[2585]: W0910 13:28:34.185504 2585 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Sep 10 13:28:34.194946 ip-10-0-140-8 kubenswrapper[2585]: W0910 13:28:34.185507 2585 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Sep 10 13:28:34.194946 ip-10-0-140-8 kubenswrapper[2585]: W0910 13:28:34.185509 2585 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Sep 10 13:28:34.194946 ip-10-0-140-8 kubenswrapper[2585]: W0910 13:28:34.185511 2585 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Sep 10 13:28:34.194946 ip-10-0-140-8 kubenswrapper[2585]: W0910 13:28:34.185514 2585 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Sep 10 13:28:34.194946 ip-10-0-140-8 kubenswrapper[2585]: W0910 13:28:34.185516 2585 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Sep 10 13:28:34.194946 ip-10-0-140-8 kubenswrapper[2585]: W0910 13:28:34.185519 2585 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Sep 10 13:28:34.194946 ip-10-0-140-8 kubenswrapper[2585]: W0910 13:28:34.185521 2585 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Sep 10 13:28:34.194946 ip-10-0-140-8 kubenswrapper[2585]: W0910 13:28:34.185524 2585 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Sep 10 13:28:34.194946 ip-10-0-140-8 kubenswrapper[2585]: W0910 13:28:34.185526 2585 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Sep 10 13:28:34.194946 ip-10-0-140-8 kubenswrapper[2585]: W0910 13:28:34.185528 2585 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Sep 10 13:28:34.194946 ip-10-0-140-8 kubenswrapper[2585]: W0910 13:28:34.185531 2585 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Sep 10 13:28:34.194946 ip-10-0-140-8 kubenswrapper[2585]: W0910 13:28:34.185534 2585 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Sep 10 13:28:34.194946 ip-10-0-140-8 kubenswrapper[2585]: W0910 13:28:34.185536 2585 feature_gate.go:328] unrecognized feature gate: PinnedImages Sep 10 13:28:34.194946 ip-10-0-140-8 kubenswrapper[2585]: W0910 13:28:34.185538 2585 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Sep 10 13:28:34.194946 ip-10-0-140-8 kubenswrapper[2585]: W0910 13:28:34.185541 2585 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Sep 10 13:28:34.195443 ip-10-0-140-8 kubenswrapper[2585]: W0910 13:28:34.185543 2585 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Sep 10 13:28:34.195443 ip-10-0-140-8 kubenswrapper[2585]: W0910 13:28:34.185546 2585 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Sep 10 13:28:34.195443 ip-10-0-140-8 kubenswrapper[2585]: W0910 13:28:34.185548 2585 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Sep 10 13:28:34.195443 ip-10-0-140-8 kubenswrapper[2585]: W0910 13:28:34.185551 2585 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Sep 10 13:28:34.195443 ip-10-0-140-8 kubenswrapper[2585]: W0910 13:28:34.185553 2585 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Sep 10 13:28:34.195443 ip-10-0-140-8 kubenswrapper[2585]: W0910 13:28:34.185556 2585 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Sep 10 13:28:34.195443 ip-10-0-140-8 kubenswrapper[2585]: W0910 13:28:34.185559 2585 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Sep 10 13:28:34.195443 ip-10-0-140-8 kubenswrapper[2585]: W0910 13:28:34.185561 2585 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Sep 10 13:28:34.195443 ip-10-0-140-8 kubenswrapper[2585]: W0910 13:28:34.185564 2585 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Sep 10 13:28:34.195443 ip-10-0-140-8 kubenswrapper[2585]: W0910 13:28:34.185566 2585 feature_gate.go:328] unrecognized feature gate: OVNObservability Sep 10 13:28:34.195443 ip-10-0-140-8 kubenswrapper[2585]: W0910 13:28:34.185569 2585 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Sep 10 13:28:34.195443 ip-10-0-140-8 kubenswrapper[2585]: W0910 13:28:34.185571 2585 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Sep 10 13:28:34.195443 ip-10-0-140-8 kubenswrapper[2585]: W0910 13:28:34.185573 2585 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Sep 10 13:28:34.195443 ip-10-0-140-8 kubenswrapper[2585]: W0910 13:28:34.185576 2585 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Sep 10 13:28:34.195443 ip-10-0-140-8 kubenswrapper[2585]: W0910 13:28:34.185578 2585 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Sep 10 13:28:34.195443 ip-10-0-140-8 kubenswrapper[2585]: W0910 13:28:34.185581 2585 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Sep 10 13:28:34.195443 ip-10-0-140-8 kubenswrapper[2585]: W0910 13:28:34.185583 2585 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Sep 10 13:28:34.195443 ip-10-0-140-8 kubenswrapper[2585]: W0910 13:28:34.185586 2585 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Sep 10 13:28:34.195443 ip-10-0-140-8 kubenswrapper[2585]: W0910 13:28:34.185588 2585 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Sep 10 13:28:34.196147 ip-10-0-140-8 kubenswrapper[2585]: W0910 13:28:34.185591 2585 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Sep 10 13:28:34.196147 ip-10-0-140-8 kubenswrapper[2585]: W0910 13:28:34.185593 2585 feature_gate.go:328] unrecognized feature gate: Example Sep 10 13:28:34.196147 ip-10-0-140-8 kubenswrapper[2585]: W0910 13:28:34.185596 2585 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Sep 10 13:28:34.196147 ip-10-0-140-8 kubenswrapper[2585]: W0910 13:28:34.185598 2585 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Sep 10 13:28:34.196147 ip-10-0-140-8 kubenswrapper[2585]: W0910 13:28:34.185601 2585 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Sep 10 13:28:34.196147 ip-10-0-140-8 kubenswrapper[2585]: W0910 13:28:34.185604 2585 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Sep 10 13:28:34.196147 ip-10-0-140-8 kubenswrapper[2585]: W0910 13:28:34.185606 2585 feature_gate.go:328] unrecognized feature gate: Example2 Sep 10 13:28:34.196147 ip-10-0-140-8 kubenswrapper[2585]: W0910 13:28:34.185609 2585 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Sep 10 13:28:34.196147 ip-10-0-140-8 kubenswrapper[2585]: W0910 13:28:34.185611 2585 feature_gate.go:328] unrecognized feature gate: EventTTL Sep 10 13:28:34.196147 ip-10-0-140-8 kubenswrapper[2585]: W0910 13:28:34.185615 2585 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Sep 10 13:28:34.196147 ip-10-0-140-8 kubenswrapper[2585]: W0910 13:28:34.185617 2585 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Sep 10 13:28:34.196147 ip-10-0-140-8 kubenswrapper[2585]: W0910 13:28:34.185621 2585 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Sep 10 13:28:34.196147 ip-10-0-140-8 kubenswrapper[2585]: W0910 13:28:34.185624 2585 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Sep 10 13:28:34.196147 ip-10-0-140-8 kubenswrapper[2585]: W0910 13:28:34.185627 2585 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Sep 10 13:28:34.196147 ip-10-0-140-8 kubenswrapper[2585]: W0910 13:28:34.185631 2585 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Sep 10 13:28:34.196147 ip-10-0-140-8 kubenswrapper[2585]: W0910 13:28:34.185634 2585 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Sep 10 13:28:34.196147 ip-10-0-140-8 kubenswrapper[2585]: W0910 13:28:34.185638 2585 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Sep 10 13:28:34.196147 ip-10-0-140-8 kubenswrapper[2585]: W0910 13:28:34.185643 2585 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Sep 10 13:28:34.196147 ip-10-0-140-8 kubenswrapper[2585]: W0910 13:28:34.185648 2585 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Sep 10 13:28:34.196742 ip-10-0-140-8 kubenswrapper[2585]: W0910 13:28:34.185651 2585 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Sep 10 13:28:34.196742 ip-10-0-140-8 kubenswrapper[2585]: W0910 13:28:34.185654 2585 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Sep 10 13:28:34.196742 ip-10-0-140-8 kubenswrapper[2585]: W0910 13:28:34.185657 2585 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Sep 10 13:28:34.196742 ip-10-0-140-8 kubenswrapper[2585]: W0910 13:28:34.185659 2585 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Sep 10 13:28:34.196742 ip-10-0-140-8 kubenswrapper[2585]: W0910 13:28:34.185662 2585 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Sep 10 13:28:34.196742 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:28:34.186301 2585 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]} Sep 10 13:28:34.196742 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:28:34.193705 2585 server.go:529] "Kubelet version" kubeletVersion="v1.34.9" Sep 10 13:28:34.196742 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:28:34.193720 2585 server.go:531] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK="" Sep 10 13:28:34.196742 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:28:34.194540 2585 watchdog_linux.go:95] "Systemd watchdog is not enabled" Sep 10 13:28:34.196742 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:28:34.194548 2585 watchdog_linux.go:137] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started." Sep 10 13:28:34.196742 ip-10-0-140-8 kubenswrapper[2585]: W0910 13:28:34.194608 2585 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Sep 10 13:28:34.196742 ip-10-0-140-8 kubenswrapper[2585]: W0910 13:28:34.194613 2585 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Sep 10 13:28:34.196742 ip-10-0-140-8 kubenswrapper[2585]: W0910 13:28:34.194617 2585 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Sep 10 13:28:34.196742 ip-10-0-140-8 kubenswrapper[2585]: W0910 13:28:34.194621 2585 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Sep 10 13:28:34.197094 ip-10-0-140-8 kubenswrapper[2585]: W0910 13:28:34.194624 2585 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Sep 10 13:28:34.197094 ip-10-0-140-8 kubenswrapper[2585]: W0910 13:28:34.194627 2585 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Sep 10 13:28:34.197094 ip-10-0-140-8 kubenswrapper[2585]: W0910 13:28:34.194630 2585 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Sep 10 13:28:34.197094 ip-10-0-140-8 kubenswrapper[2585]: W0910 13:28:34.194633 2585 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Sep 10 13:28:34.197094 ip-10-0-140-8 kubenswrapper[2585]: W0910 13:28:34.194636 2585 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Sep 10 13:28:34.197094 ip-10-0-140-8 kubenswrapper[2585]: W0910 13:28:34.194639 2585 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Sep 10 13:28:34.197094 ip-10-0-140-8 kubenswrapper[2585]: W0910 13:28:34.194642 2585 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Sep 10 13:28:34.197094 ip-10-0-140-8 kubenswrapper[2585]: W0910 13:28:34.194644 2585 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Sep 10 13:28:34.197094 ip-10-0-140-8 kubenswrapper[2585]: W0910 13:28:34.194647 2585 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Sep 10 13:28:34.197094 ip-10-0-140-8 kubenswrapper[2585]: W0910 13:28:34.194649 2585 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Sep 10 13:28:34.197094 ip-10-0-140-8 kubenswrapper[2585]: W0910 13:28:34.194653 2585 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Sep 10 13:28:34.197094 ip-10-0-140-8 kubenswrapper[2585]: W0910 13:28:34.194656 2585 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Sep 10 13:28:34.197094 ip-10-0-140-8 kubenswrapper[2585]: W0910 13:28:34.194658 2585 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Sep 10 13:28:34.197094 ip-10-0-140-8 kubenswrapper[2585]: W0910 13:28:34.194661 2585 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Sep 10 13:28:34.197094 ip-10-0-140-8 kubenswrapper[2585]: W0910 13:28:34.194663 2585 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Sep 10 13:28:34.197094 ip-10-0-140-8 kubenswrapper[2585]: W0910 13:28:34.194666 2585 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Sep 10 13:28:34.197094 ip-10-0-140-8 kubenswrapper[2585]: W0910 13:28:34.194669 2585 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Sep 10 13:28:34.197094 ip-10-0-140-8 kubenswrapper[2585]: W0910 13:28:34.194673 2585 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Sep 10 13:28:34.197094 ip-10-0-140-8 kubenswrapper[2585]: W0910 13:28:34.194675 2585 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Sep 10 13:28:34.197660 ip-10-0-140-8 kubenswrapper[2585]: W0910 13:28:34.194678 2585 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Sep 10 13:28:34.197660 ip-10-0-140-8 kubenswrapper[2585]: W0910 13:28:34.194680 2585 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Sep 10 13:28:34.197660 ip-10-0-140-8 kubenswrapper[2585]: W0910 13:28:34.194683 2585 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Sep 10 13:28:34.197660 ip-10-0-140-8 kubenswrapper[2585]: W0910 13:28:34.194686 2585 feature_gate.go:328] unrecognized feature gate: PinnedImages Sep 10 13:28:34.197660 ip-10-0-140-8 kubenswrapper[2585]: W0910 13:28:34.194688 2585 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Sep 10 13:28:34.197660 ip-10-0-140-8 kubenswrapper[2585]: W0910 13:28:34.194691 2585 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Sep 10 13:28:34.197660 ip-10-0-140-8 kubenswrapper[2585]: W0910 13:28:34.194695 2585 feature_gate.go:328] unrecognized feature gate: EventTTL Sep 10 13:28:34.197660 ip-10-0-140-8 kubenswrapper[2585]: W0910 13:28:34.194697 2585 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Sep 10 13:28:34.197660 ip-10-0-140-8 kubenswrapper[2585]: W0910 13:28:34.194701 2585 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Sep 10 13:28:34.197660 ip-10-0-140-8 kubenswrapper[2585]: W0910 13:28:34.194705 2585 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Sep 10 13:28:34.197660 ip-10-0-140-8 kubenswrapper[2585]: W0910 13:28:34.194708 2585 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Sep 10 13:28:34.197660 ip-10-0-140-8 kubenswrapper[2585]: W0910 13:28:34.194711 2585 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Sep 10 13:28:34.197660 ip-10-0-140-8 kubenswrapper[2585]: W0910 13:28:34.194714 2585 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Sep 10 13:28:34.197660 ip-10-0-140-8 kubenswrapper[2585]: W0910 13:28:34.194717 2585 feature_gate.go:328] unrecognized feature gate: GatewayAPI Sep 10 13:28:34.197660 ip-10-0-140-8 kubenswrapper[2585]: W0910 13:28:34.194720 2585 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Sep 10 13:28:34.197660 ip-10-0-140-8 kubenswrapper[2585]: W0910 13:28:34.194722 2585 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Sep 10 13:28:34.197660 ip-10-0-140-8 kubenswrapper[2585]: W0910 13:28:34.194725 2585 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Sep 10 13:28:34.197660 ip-10-0-140-8 kubenswrapper[2585]: W0910 13:28:34.194727 2585 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Sep 10 13:28:34.197660 ip-10-0-140-8 kubenswrapper[2585]: W0910 13:28:34.194730 2585 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Sep 10 13:28:34.197660 ip-10-0-140-8 kubenswrapper[2585]: W0910 13:28:34.194733 2585 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Sep 10 13:28:34.198202 ip-10-0-140-8 kubenswrapper[2585]: W0910 13:28:34.194735 2585 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Sep 10 13:28:34.198202 ip-10-0-140-8 kubenswrapper[2585]: W0910 13:28:34.194738 2585 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Sep 10 13:28:34.198202 ip-10-0-140-8 kubenswrapper[2585]: W0910 13:28:34.194740 2585 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Sep 10 13:28:34.198202 ip-10-0-140-8 kubenswrapper[2585]: W0910 13:28:34.194743 2585 feature_gate.go:328] unrecognized feature gate: SignatureStores Sep 10 13:28:34.198202 ip-10-0-140-8 kubenswrapper[2585]: W0910 13:28:34.194745 2585 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Sep 10 13:28:34.198202 ip-10-0-140-8 kubenswrapper[2585]: W0910 13:28:34.194748 2585 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Sep 10 13:28:34.198202 ip-10-0-140-8 kubenswrapper[2585]: W0910 13:28:34.194750 2585 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Sep 10 13:28:34.198202 ip-10-0-140-8 kubenswrapper[2585]: W0910 13:28:34.194753 2585 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Sep 10 13:28:34.198202 ip-10-0-140-8 kubenswrapper[2585]: W0910 13:28:34.194755 2585 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Sep 10 13:28:34.198202 ip-10-0-140-8 kubenswrapper[2585]: W0910 13:28:34.194758 2585 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Sep 10 13:28:34.198202 ip-10-0-140-8 kubenswrapper[2585]: W0910 13:28:34.194761 2585 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Sep 10 13:28:34.198202 ip-10-0-140-8 kubenswrapper[2585]: W0910 13:28:34.194764 2585 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Sep 10 13:28:34.198202 ip-10-0-140-8 kubenswrapper[2585]: W0910 13:28:34.194766 2585 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Sep 10 13:28:34.198202 ip-10-0-140-8 kubenswrapper[2585]: W0910 13:28:34.194769 2585 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Sep 10 13:28:34.198202 ip-10-0-140-8 kubenswrapper[2585]: W0910 13:28:34.194772 2585 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Sep 10 13:28:34.198202 ip-10-0-140-8 kubenswrapper[2585]: W0910 13:28:34.194775 2585 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Sep 10 13:28:34.198202 ip-10-0-140-8 kubenswrapper[2585]: W0910 13:28:34.194777 2585 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Sep 10 13:28:34.198202 ip-10-0-140-8 kubenswrapper[2585]: W0910 13:28:34.194780 2585 feature_gate.go:328] unrecognized feature gate: InsightsConfig Sep 10 13:28:34.198202 ip-10-0-140-8 kubenswrapper[2585]: W0910 13:28:34.194782 2585 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Sep 10 13:28:34.198202 ip-10-0-140-8 kubenswrapper[2585]: W0910 13:28:34.194785 2585 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Sep 10 13:28:34.198987 ip-10-0-140-8 kubenswrapper[2585]: W0910 13:28:34.194788 2585 feature_gate.go:328] unrecognized feature gate: Example2 Sep 10 13:28:34.198987 ip-10-0-140-8 kubenswrapper[2585]: W0910 13:28:34.194792 2585 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Sep 10 13:28:34.198987 ip-10-0-140-8 kubenswrapper[2585]: W0910 13:28:34.194796 2585 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Sep 10 13:28:34.198987 ip-10-0-140-8 kubenswrapper[2585]: W0910 13:28:34.194798 2585 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Sep 10 13:28:34.198987 ip-10-0-140-8 kubenswrapper[2585]: W0910 13:28:34.194801 2585 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Sep 10 13:28:34.198987 ip-10-0-140-8 kubenswrapper[2585]: W0910 13:28:34.194803 2585 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Sep 10 13:28:34.198987 ip-10-0-140-8 kubenswrapper[2585]: W0910 13:28:34.194807 2585 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Sep 10 13:28:34.198987 ip-10-0-140-8 kubenswrapper[2585]: W0910 13:28:34.194810 2585 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Sep 10 13:28:34.198987 ip-10-0-140-8 kubenswrapper[2585]: W0910 13:28:34.194813 2585 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Sep 10 13:28:34.198987 ip-10-0-140-8 kubenswrapper[2585]: W0910 13:28:34.194816 2585 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Sep 10 13:28:34.198987 ip-10-0-140-8 kubenswrapper[2585]: W0910 13:28:34.194819 2585 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Sep 10 13:28:34.198987 ip-10-0-140-8 kubenswrapper[2585]: W0910 13:28:34.194822 2585 feature_gate.go:328] unrecognized feature gate: DualReplica Sep 10 13:28:34.198987 ip-10-0-140-8 kubenswrapper[2585]: W0910 13:28:34.194825 2585 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Sep 10 13:28:34.198987 ip-10-0-140-8 kubenswrapper[2585]: W0910 13:28:34.194827 2585 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Sep 10 13:28:34.198987 ip-10-0-140-8 kubenswrapper[2585]: W0910 13:28:34.194830 2585 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Sep 10 13:28:34.198987 ip-10-0-140-8 kubenswrapper[2585]: W0910 13:28:34.194832 2585 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Sep 10 13:28:34.198987 ip-10-0-140-8 kubenswrapper[2585]: W0910 13:28:34.194835 2585 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Sep 10 13:28:34.198987 ip-10-0-140-8 kubenswrapper[2585]: W0910 13:28:34.194837 2585 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Sep 10 13:28:34.199742 ip-10-0-140-8 kubenswrapper[2585]: W0910 13:28:34.194840 2585 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Sep 10 13:28:34.199742 ip-10-0-140-8 kubenswrapper[2585]: W0910 13:28:34.194842 2585 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Sep 10 13:28:34.199742 ip-10-0-140-8 kubenswrapper[2585]: W0910 13:28:34.194845 2585 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Sep 10 13:28:34.199742 ip-10-0-140-8 kubenswrapper[2585]: W0910 13:28:34.194847 2585 feature_gate.go:328] unrecognized feature gate: OVNObservability Sep 10 13:28:34.199742 ip-10-0-140-8 kubenswrapper[2585]: W0910 13:28:34.194850 2585 feature_gate.go:328] unrecognized feature gate: OSStreams Sep 10 13:28:34.199742 ip-10-0-140-8 kubenswrapper[2585]: W0910 13:28:34.194853 2585 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Sep 10 13:28:34.199742 ip-10-0-140-8 kubenswrapper[2585]: W0910 13:28:34.194856 2585 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Sep 10 13:28:34.199742 ip-10-0-140-8 kubenswrapper[2585]: W0910 13:28:34.194858 2585 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Sep 10 13:28:34.199742 ip-10-0-140-8 kubenswrapper[2585]: W0910 13:28:34.194861 2585 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Sep 10 13:28:34.199742 ip-10-0-140-8 kubenswrapper[2585]: W0910 13:28:34.194863 2585 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Sep 10 13:28:34.199742 ip-10-0-140-8 kubenswrapper[2585]: W0910 13:28:34.194866 2585 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Sep 10 13:28:34.199742 ip-10-0-140-8 kubenswrapper[2585]: W0910 13:28:34.194868 2585 feature_gate.go:328] unrecognized feature gate: Example Sep 10 13:28:34.199742 ip-10-0-140-8 kubenswrapper[2585]: W0910 13:28:34.194871 2585 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Sep 10 13:28:34.199742 ip-10-0-140-8 kubenswrapper[2585]: W0910 13:28:34.194873 2585 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Sep 10 13:28:34.199742 ip-10-0-140-8 kubenswrapper[2585]: W0910 13:28:34.194876 2585 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Sep 10 13:28:34.199742 ip-10-0-140-8 kubenswrapper[2585]: W0910 13:28:34.194879 2585 feature_gate.go:328] unrecognized feature gate: KMSEncryption Sep 10 13:28:34.199742 ip-10-0-140-8 kubenswrapper[2585]: W0910 13:28:34.194882 2585 feature_gate.go:328] unrecognized feature gate: NewOLM Sep 10 13:28:34.199742 ip-10-0-140-8 kubenswrapper[2585]: W0910 13:28:34.194884 2585 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Sep 10 13:28:34.200303 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:28:34.194889 2585 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]} Sep 10 13:28:34.200303 ip-10-0-140-8 kubenswrapper[2585]: W0910 13:28:34.195004 2585 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Sep 10 13:28:34.200303 ip-10-0-140-8 kubenswrapper[2585]: W0910 13:28:34.195010 2585 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Sep 10 13:28:34.200303 ip-10-0-140-8 kubenswrapper[2585]: W0910 13:28:34.195014 2585 feature_gate.go:328] unrecognized feature gate: PinnedImages Sep 10 13:28:34.200303 ip-10-0-140-8 kubenswrapper[2585]: W0910 13:28:34.195016 2585 feature_gate.go:328] unrecognized feature gate: OVNObservability Sep 10 13:28:34.200303 ip-10-0-140-8 kubenswrapper[2585]: W0910 13:28:34.195019 2585 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Sep 10 13:28:34.200303 ip-10-0-140-8 kubenswrapper[2585]: W0910 13:28:34.195022 2585 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Sep 10 13:28:34.200303 ip-10-0-140-8 kubenswrapper[2585]: W0910 13:28:34.195025 2585 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Sep 10 13:28:34.200303 ip-10-0-140-8 kubenswrapper[2585]: W0910 13:28:34.195028 2585 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Sep 10 13:28:34.200303 ip-10-0-140-8 kubenswrapper[2585]: W0910 13:28:34.195031 2585 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Sep 10 13:28:34.200303 ip-10-0-140-8 kubenswrapper[2585]: W0910 13:28:34.195034 2585 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Sep 10 13:28:34.200303 ip-10-0-140-8 kubenswrapper[2585]: W0910 13:28:34.195037 2585 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Sep 10 13:28:34.200303 ip-10-0-140-8 kubenswrapper[2585]: W0910 13:28:34.195039 2585 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Sep 10 13:28:34.200303 ip-10-0-140-8 kubenswrapper[2585]: W0910 13:28:34.195042 2585 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Sep 10 13:28:34.200658 ip-10-0-140-8 kubenswrapper[2585]: W0910 13:28:34.195045 2585 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Sep 10 13:28:34.200658 ip-10-0-140-8 kubenswrapper[2585]: W0910 13:28:34.195047 2585 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Sep 10 13:28:34.200658 ip-10-0-140-8 kubenswrapper[2585]: W0910 13:28:34.195050 2585 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Sep 10 13:28:34.200658 ip-10-0-140-8 kubenswrapper[2585]: W0910 13:28:34.195052 2585 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Sep 10 13:28:34.200658 ip-10-0-140-8 kubenswrapper[2585]: W0910 13:28:34.195055 2585 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Sep 10 13:28:34.200658 ip-10-0-140-8 kubenswrapper[2585]: W0910 13:28:34.195057 2585 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Sep 10 13:28:34.200658 ip-10-0-140-8 kubenswrapper[2585]: W0910 13:28:34.195060 2585 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Sep 10 13:28:34.200658 ip-10-0-140-8 kubenswrapper[2585]: W0910 13:28:34.195062 2585 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Sep 10 13:28:34.200658 ip-10-0-140-8 kubenswrapper[2585]: W0910 13:28:34.195064 2585 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Sep 10 13:28:34.200658 ip-10-0-140-8 kubenswrapper[2585]: W0910 13:28:34.195067 2585 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Sep 10 13:28:34.200658 ip-10-0-140-8 kubenswrapper[2585]: W0910 13:28:34.195069 2585 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Sep 10 13:28:34.200658 ip-10-0-140-8 kubenswrapper[2585]: W0910 13:28:34.195072 2585 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Sep 10 13:28:34.200658 ip-10-0-140-8 kubenswrapper[2585]: W0910 13:28:34.195074 2585 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Sep 10 13:28:34.200658 ip-10-0-140-8 kubenswrapper[2585]: W0910 13:28:34.195077 2585 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Sep 10 13:28:34.200658 ip-10-0-140-8 kubenswrapper[2585]: W0910 13:28:34.195080 2585 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Sep 10 13:28:34.200658 ip-10-0-140-8 kubenswrapper[2585]: W0910 13:28:34.195084 2585 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Sep 10 13:28:34.200658 ip-10-0-140-8 kubenswrapper[2585]: W0910 13:28:34.195088 2585 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Sep 10 13:28:34.200658 ip-10-0-140-8 kubenswrapper[2585]: W0910 13:28:34.195090 2585 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Sep 10 13:28:34.200658 ip-10-0-140-8 kubenswrapper[2585]: W0910 13:28:34.195093 2585 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Sep 10 13:28:34.201197 ip-10-0-140-8 kubenswrapper[2585]: W0910 13:28:34.195096 2585 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Sep 10 13:28:34.201197 ip-10-0-140-8 kubenswrapper[2585]: W0910 13:28:34.195099 2585 feature_gate.go:328] unrecognized feature gate: GatewayAPI Sep 10 13:28:34.201197 ip-10-0-140-8 kubenswrapper[2585]: W0910 13:28:34.195102 2585 feature_gate.go:328] unrecognized feature gate: KMSEncryption Sep 10 13:28:34.201197 ip-10-0-140-8 kubenswrapper[2585]: W0910 13:28:34.195105 2585 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Sep 10 13:28:34.201197 ip-10-0-140-8 kubenswrapper[2585]: W0910 13:28:34.195107 2585 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Sep 10 13:28:34.201197 ip-10-0-140-8 kubenswrapper[2585]: W0910 13:28:34.195110 2585 feature_gate.go:328] unrecognized feature gate: EventTTL Sep 10 13:28:34.201197 ip-10-0-140-8 kubenswrapper[2585]: W0910 13:28:34.195113 2585 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Sep 10 13:28:34.201197 ip-10-0-140-8 kubenswrapper[2585]: W0910 13:28:34.195116 2585 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Sep 10 13:28:34.201197 ip-10-0-140-8 kubenswrapper[2585]: W0910 13:28:34.195119 2585 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Sep 10 13:28:34.201197 ip-10-0-140-8 kubenswrapper[2585]: W0910 13:28:34.195121 2585 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Sep 10 13:28:34.201197 ip-10-0-140-8 kubenswrapper[2585]: W0910 13:28:34.195124 2585 feature_gate.go:328] unrecognized feature gate: SignatureStores Sep 10 13:28:34.201197 ip-10-0-140-8 kubenswrapper[2585]: W0910 13:28:34.195126 2585 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Sep 10 13:28:34.201197 ip-10-0-140-8 kubenswrapper[2585]: W0910 13:28:34.195129 2585 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Sep 10 13:28:34.201197 ip-10-0-140-8 kubenswrapper[2585]: W0910 13:28:34.195131 2585 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Sep 10 13:28:34.201197 ip-10-0-140-8 kubenswrapper[2585]: W0910 13:28:34.195133 2585 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Sep 10 13:28:34.201197 ip-10-0-140-8 kubenswrapper[2585]: W0910 13:28:34.195136 2585 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Sep 10 13:28:34.201197 ip-10-0-140-8 kubenswrapper[2585]: W0910 13:28:34.195139 2585 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Sep 10 13:28:34.201197 ip-10-0-140-8 kubenswrapper[2585]: W0910 13:28:34.195141 2585 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Sep 10 13:28:34.201197 ip-10-0-140-8 kubenswrapper[2585]: W0910 13:28:34.195144 2585 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Sep 10 13:28:34.201197 ip-10-0-140-8 kubenswrapper[2585]: W0910 13:28:34.195146 2585 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Sep 10 13:28:34.201928 ip-10-0-140-8 kubenswrapper[2585]: W0910 13:28:34.195148 2585 feature_gate.go:328] unrecognized feature gate: InsightsConfig Sep 10 13:28:34.201928 ip-10-0-140-8 kubenswrapper[2585]: W0910 13:28:34.195151 2585 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Sep 10 13:28:34.201928 ip-10-0-140-8 kubenswrapper[2585]: W0910 13:28:34.195153 2585 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Sep 10 13:28:34.201928 ip-10-0-140-8 kubenswrapper[2585]: W0910 13:28:34.195156 2585 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Sep 10 13:28:34.201928 ip-10-0-140-8 kubenswrapper[2585]: W0910 13:28:34.195158 2585 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Sep 10 13:28:34.201928 ip-10-0-140-8 kubenswrapper[2585]: W0910 13:28:34.195160 2585 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Sep 10 13:28:34.201928 ip-10-0-140-8 kubenswrapper[2585]: W0910 13:28:34.195163 2585 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Sep 10 13:28:34.201928 ip-10-0-140-8 kubenswrapper[2585]: W0910 13:28:34.195165 2585 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Sep 10 13:28:34.201928 ip-10-0-140-8 kubenswrapper[2585]: W0910 13:28:34.195168 2585 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Sep 10 13:28:34.201928 ip-10-0-140-8 kubenswrapper[2585]: W0910 13:28:34.195171 2585 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Sep 10 13:28:34.201928 ip-10-0-140-8 kubenswrapper[2585]: W0910 13:28:34.195173 2585 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Sep 10 13:28:34.201928 ip-10-0-140-8 kubenswrapper[2585]: W0910 13:28:34.195175 2585 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Sep 10 13:28:34.201928 ip-10-0-140-8 kubenswrapper[2585]: W0910 13:28:34.195178 2585 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Sep 10 13:28:34.201928 ip-10-0-140-8 kubenswrapper[2585]: W0910 13:28:34.195182 2585 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Sep 10 13:28:34.201928 ip-10-0-140-8 kubenswrapper[2585]: W0910 13:28:34.195185 2585 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Sep 10 13:28:34.201928 ip-10-0-140-8 kubenswrapper[2585]: W0910 13:28:34.195188 2585 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Sep 10 13:28:34.201928 ip-10-0-140-8 kubenswrapper[2585]: W0910 13:28:34.195191 2585 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Sep 10 13:28:34.201928 ip-10-0-140-8 kubenswrapper[2585]: W0910 13:28:34.195193 2585 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Sep 10 13:28:34.201928 ip-10-0-140-8 kubenswrapper[2585]: W0910 13:28:34.195196 2585 feature_gate.go:328] unrecognized feature gate: DualReplica Sep 10 13:28:34.202410 ip-10-0-140-8 kubenswrapper[2585]: W0910 13:28:34.195199 2585 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Sep 10 13:28:34.202410 ip-10-0-140-8 kubenswrapper[2585]: W0910 13:28:34.195201 2585 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Sep 10 13:28:34.202410 ip-10-0-140-8 kubenswrapper[2585]: W0910 13:28:34.195204 2585 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Sep 10 13:28:34.202410 ip-10-0-140-8 kubenswrapper[2585]: W0910 13:28:34.195206 2585 feature_gate.go:328] unrecognized feature gate: Example2 Sep 10 13:28:34.202410 ip-10-0-140-8 kubenswrapper[2585]: W0910 13:28:34.195208 2585 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Sep 10 13:28:34.202410 ip-10-0-140-8 kubenswrapper[2585]: W0910 13:28:34.195211 2585 feature_gate.go:328] unrecognized feature gate: OSStreams Sep 10 13:28:34.202410 ip-10-0-140-8 kubenswrapper[2585]: W0910 13:28:34.195214 2585 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Sep 10 13:28:34.202410 ip-10-0-140-8 kubenswrapper[2585]: W0910 13:28:34.195216 2585 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Sep 10 13:28:34.202410 ip-10-0-140-8 kubenswrapper[2585]: W0910 13:28:34.195219 2585 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Sep 10 13:28:34.202410 ip-10-0-140-8 kubenswrapper[2585]: W0910 13:28:34.195221 2585 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Sep 10 13:28:34.202410 ip-10-0-140-8 kubenswrapper[2585]: W0910 13:28:34.195224 2585 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Sep 10 13:28:34.202410 ip-10-0-140-8 kubenswrapper[2585]: W0910 13:28:34.195226 2585 feature_gate.go:328] unrecognized feature gate: Example Sep 10 13:28:34.202410 ip-10-0-140-8 kubenswrapper[2585]: W0910 13:28:34.195229 2585 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Sep 10 13:28:34.202410 ip-10-0-140-8 kubenswrapper[2585]: W0910 13:28:34.195231 2585 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Sep 10 13:28:34.202410 ip-10-0-140-8 kubenswrapper[2585]: W0910 13:28:34.195234 2585 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Sep 10 13:28:34.202410 ip-10-0-140-8 kubenswrapper[2585]: W0910 13:28:34.195236 2585 feature_gate.go:328] unrecognized feature gate: NewOLM Sep 10 13:28:34.202410 ip-10-0-140-8 kubenswrapper[2585]: W0910 13:28:34.195238 2585 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Sep 10 13:28:34.202410 ip-10-0-140-8 kubenswrapper[2585]: W0910 13:28:34.195241 2585 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Sep 10 13:28:34.202410 ip-10-0-140-8 kubenswrapper[2585]: W0910 13:28:34.195243 2585 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Sep 10 13:28:34.202410 ip-10-0-140-8 kubenswrapper[2585]: W0910 13:28:34.195246 2585 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Sep 10 13:28:34.202971 ip-10-0-140-8 kubenswrapper[2585]: W0910 13:28:34.195248 2585 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Sep 10 13:28:34.202971 ip-10-0-140-8 kubenswrapper[2585]: W0910 13:28:34.195251 2585 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Sep 10 13:28:34.202971 ip-10-0-140-8 kubenswrapper[2585]: W0910 13:28:34.195253 2585 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Sep 10 13:28:34.202971 ip-10-0-140-8 kubenswrapper[2585]: W0910 13:28:34.195270 2585 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Sep 10 13:28:34.202971 ip-10-0-140-8 kubenswrapper[2585]: W0910 13:28:34.195274 2585 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Sep 10 13:28:34.202971 ip-10-0-140-8 kubenswrapper[2585]: W0910 13:28:34.195277 2585 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Sep 10 13:28:34.202971 ip-10-0-140-8 kubenswrapper[2585]: W0910 13:28:34.195280 2585 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Sep 10 13:28:34.202971 ip-10-0-140-8 kubenswrapper[2585]: W0910 13:28:34.195282 2585 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Sep 10 13:28:34.202971 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:28:34.195287 2585 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]} Sep 10 13:28:34.202971 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:28:34.197586 2585 server.go:962] "Client rotation is on, will bootstrap in background" Sep 10 13:28:34.202971 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:28:34.201612 2585 bootstrap.go:101] "Use the bootstrap credentials to request a cert, and set kubeconfig to point to the certificate dir" Sep 10 13:28:34.202971 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:28:34.202740 2585 server.go:1019] "Starting client certificate rotation" Sep 10 13:28:34.202971 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:28:34.202838 2585 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet" Sep 10 13:28:34.202971 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:28:34.202877 2585 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kube-apiserver-client-kubelet" Sep 10 13:28:34.205522 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:28:34.205504 2585 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Sep 10 13:28:34.207825 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:28:34.207802 2585 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Sep 10 13:28:34.219818 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:28:34.219797 2585 log.go:25] "Validated CRI v1 runtime API" Sep 10 13:28:34.224141 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:28:34.224126 2585 log.go:25] "Validated CRI v1 image API" Sep 10 13:28:34.224998 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:28:34.224985 2585 server.go:1429] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd" Sep 10 13:28:34.228406 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:28:34.228384 2585 fs.go:135] Filesystem UUIDs: map[7B77-95E7:/dev/nvme0n1p2 88f71200-fb1b-46f0-8bd4-a90673f1a437:/dev/nvme0n1p4 d5660a0a-c166-41f1-b585-fd88f3b542fe:/dev/nvme0n1p3] Sep 10 13:28:34.228468 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:28:34.228404 2585 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:22 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:24 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:31 fsType:tmpfs blockSize:0} composefs_0-33:{mountpoint:/ major:0 minor:33 fsType:overlay blockSize:0}] Sep 10 13:28:34.234595 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:28:34.234489 2585 manager.go:216] Machine: {Timestamp:2026-09-10 13:28:34.233050808 +0000 UTC m=+0.377740741 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3099118 MemoryCapacity:33164480512 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec287a04fe123c7351c45113b82d7b62 SystemUUID:ec287a04-fe12-3c73-51c4-5113b82d7b62 BootID:818b2245-6262-4631-8e96-3d7b0adba730 Filesystems:[{Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:composefs_0-33 DeviceMajor:0 DeviceMinor:33 Capacity:6111232 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:31 Capacity:16582242304 Type:vfs Inodes:1048576 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true} {Device:/dev/shm DeviceMajor:0 DeviceMinor:22 Capacity:16582238208 Type:vfs Inodes:4048398 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:24 Capacity:6632898560 Type:vfs Inodes:819200 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:br-ex MacAddress:02:09:0a:c8:2b:49 Speed:0 Mtu:9001} {Name:ens5 MacAddress:02:09:0a:c8:2b:49 Speed:0 Mtu:9001} {Name:ovs-system MacAddress:62:bf:f5:b5:39:a3 Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:33164480512 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:37486592 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None} Sep 10 13:28:34.234595 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:28:34.234588 2585 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available. Sep 10 13:28:34.234693 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:28:34.234669 2585 manager.go:232] Version: {KernelVersion:5.14.0-570.138.1.el9_6.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.6.20260901-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:} Sep 10 13:28:34.238217 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:28:34.238185 2585 container_manager_linux.go:273] "Container manager verified user specified cgroup-root exists" cgroupRoot=[] Sep 10 13:28:34.238369 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:28:34.238213 2585 container_manager_linux.go:278] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-140-8.ec2.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2} Sep 10 13:28:34.239295 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:28:34.239283 2585 topology_manager.go:138] "Creating topology manager with none policy" Sep 10 13:28:34.239335 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:28:34.239299 2585 container_manager_linux.go:309] "Creating device plugin manager" Sep 10 13:28:34.239335 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:28:34.239308 2585 manager.go:141] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock" Sep 10 13:28:34.240135 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:28:34.240125 2585 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock" Sep 10 13:28:34.241916 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:28:34.241906 2585 container_manager_linux.go:318] "Creating Dynamic Resource Allocation (DRA) manager" Sep 10 13:28:34.246347 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:28:34.246334 2585 state_mem.go:36] "Initialized new in-memory state store" Sep 10 13:28:34.246454 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:28:34.246445 2585 server.go:1240] "Using root directory" path="/var/lib/kubelet" Sep 10 13:28:34.246501 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:28:34.246486 2585 kubelet.go:486] "Attempting to sync node with API server" Sep 10 13:28:34.246529 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:28:34.246505 2585 kubelet.go:387] "Adding static pod path" path="/etc/kubernetes/manifests" Sep 10 13:28:34.246529 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:28:34.246516 2585 file.go:69] "Watching path" path="/etc/kubernetes/manifests" Sep 10 13:28:34.246529 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:28:34.246525 2585 kubelet.go:398] "Adding apiserver pod source" Sep 10 13:28:34.246616 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:28:34.246533 2585 apiserver.go:42] "Waiting for node sync before watching apiserver pods" Sep 10 13:28:34.247091 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:28:34.247050 2585 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Sep 10 13:28:34.247179 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:28:34.247097 2585 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Sep 10 13:28:34.249104 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:28:34.248993 2585 reflector.go:439] "Caches populated" logger="kubernetes.io/kube-apiserver-client-kubelet" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" Sep 10 13:28:34.250909 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:28:34.250890 2585 kuberuntime_manager.go:301] "Container runtime initialized" containerRuntime="cri-o" version="1.34.12-5.rhaos4.21.gitd1ec63c.el9" apiVersion="v1" Sep 10 13:28:34.253894 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:28:34.253879 2585 kubelet.go:960] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled" Sep 10 13:28:34.253955 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:28:34.253909 2585 kubelet.go:984] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled" Sep 10 13:28:34.254074 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:28:34.254064 2585 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume" Sep 10 13:28:34.254108 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:28:34.254079 2585 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir" Sep 10 13:28:34.254108 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:28:34.254085 2585 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo" Sep 10 13:28:34.254108 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:28:34.254090 2585 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path" Sep 10 13:28:34.254108 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:28:34.254098 2585 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs" Sep 10 13:28:34.254108 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:28:34.254103 2585 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret" Sep 10 13:28:34.254755 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:28:34.254744 2585 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi" Sep 10 13:28:34.254755 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:28:34.254754 2585 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api" Sep 10 13:28:34.254810 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:28:34.254761 2585 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc" Sep 10 13:28:34.254810 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:28:34.254767 2585 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap" Sep 10 13:28:34.254810 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:28:34.254776 2585 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected" Sep 10 13:28:34.254810 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:28:34.254784 2585 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume" Sep 10 13:28:34.255689 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:28:34.255679 2585 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi" Sep 10 13:28:34.255689 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:28:34.255688 2585 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image" Sep 10 13:28:34.258505 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:28:34.258488 2585 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-140-8.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope Sep 10 13:28:34.258579 ip-10-0-140-8 kubenswrapper[2585]: E0910 13:28:34.258505 2585 reflector.go:205] "Failed to watch" err="failed to list *v1.Node: nodes \"ip-10-0-140-8.ec2.internal\" is forbidden: User \"system:anonymous\" cannot list resource \"nodes\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Node" Sep 10 13:28:34.258579 ip-10-0-140-8 kubenswrapper[2585]: E0910 13:28:34.258566 2585 reflector.go:205] "Failed to watch" err="failed to list *v1.Service: services is forbidden: User \"system:anonymous\" cannot list resource \"services\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Service" Sep 10 13:28:34.259681 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:28:34.259668 2585 server.go:1268] "Started kubelet" Sep 10 13:28:34.259827 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:28:34.259779 2585 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10 Sep 10 13:28:34.259827 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:28:34.259789 2585 server.go:180] "Starting to listen" address="0.0.0.0" port=10250 Sep 10 13:28:34.260007 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:28:34.259849 2585 server_v1.go:49] "podresources" method="list" useActivePods=true Sep 10 13:28:34.260492 ip-10-0-140-8 systemd[1]: Started Kubernetes Kubelet. Sep 10 13:28:34.261253 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:28:34.261234 2585 server.go:249] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock" Sep 10 13:28:34.262306 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:28:34.262291 2585 server.go:310] "Adding debug handlers to kubelet server" Sep 10 13:28:34.266535 ip-10-0-140-8 kubenswrapper[2585]: E0910 13:28:34.266503 2585 kubelet.go:1638] "Image garbage collection failed once. Stats initialization may not have completed yet" err="failed to get imageFs info: unable to find data in memory cache" Sep 10 13:28:34.268410 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:28:34.268391 2585 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving" Sep 10 13:28:34.268972 ip-10-0-140-8 kubenswrapper[2585]: E0910 13:28:34.267833 2585 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-140-8.ec2.internal.18d3f8b092f464b9 default 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-140-8.ec2.internal,UID:ip-10-0-140-8.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:Starting,Message:Starting kubelet.,Source:EventSource{Component:kubelet,Host:ip-10-0-140-8.ec2.internal,},FirstTimestamp:2026-09-10 13:28:34.259641529 +0000 UTC m=+0.404331463,LastTimestamp:2026-09-10 13:28:34.259641529 +0000 UTC m=+0.404331463,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-140-8.ec2.internal,}" Sep 10 13:28:34.269048 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:28:34.269032 2585 fs_resource_analyzer.go:67] "Starting FS ResourceAnalyzer" Sep 10 13:28:34.269157 ip-10-0-140-8 kubenswrapper[2585]: E0910 13:28:34.269138 2585 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-140-8.ec2.internal\" not found" Sep 10 13:28:34.269837 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:28:34.269819 2585 factory.go:55] Registering systemd factory Sep 10 13:28:34.269910 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:28:34.269840 2585 factory.go:223] Registration of the systemd container factory successfully Sep 10 13:28:34.269958 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:28:34.269941 2585 desired_state_of_world_populator.go:146] "Desired state populator starts to run" Sep 10 13:28:34.269958 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:28:34.269941 2585 volume_manager.go:311] "The desired_state_of_world populator starts" Sep 10 13:28:34.270049 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:28:34.269966 2585 volume_manager.go:313] "Starting Kubelet Volume Manager" Sep 10 13:28:34.270089 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:28:34.270069 2585 factory.go:153] Registering CRI-O factory Sep 10 13:28:34.270089 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:28:34.270081 2585 factory.go:223] Registration of the crio container factory successfully Sep 10 13:28:34.270089 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:28:34.270085 2585 reconstruct.go:97] "Volume reconstruction finished" Sep 10 13:28:34.270230 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:28:34.270095 2585 reconciler.go:29] "Reconciler: start to sync state" Sep 10 13:28:34.270230 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:28:34.270130 2585 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory Sep 10 13:28:34.270230 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:28:34.270144 2585 factory.go:103] Registering Raw factory Sep 10 13:28:34.270230 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:28:34.270154 2585 manager.go:1196] Started watching for new ooms in manager Sep 10 13:28:34.270490 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:28:34.270479 2585 manager.go:349] Starting recovery of all containers Sep 10 13:28:34.278969 ip-10-0-140-8 kubenswrapper[2585]: E0910 13:28:34.278764 2585 reflector.go:205] "Failed to watch" err="failed to list *v1.CSIDriver: csidrivers.storage.k8s.io is forbidden: User \"system:anonymous\" cannot list resource \"csidrivers\" in API group \"storage.k8s.io\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.CSIDriver" Sep 10 13:28:34.278969 ip-10-0-140-8 kubenswrapper[2585]: E0910 13:28:34.278780 2585 controller.go:145] "Failed to ensure lease exists, will retry" err="leases.coordination.k8s.io \"ip-10-0-140-8.ec2.internal\" is forbidden: User \"system:anonymous\" cannot get resource \"leases\" in API group \"coordination.k8s.io\" in the namespace \"kube-node-lease\"" interval="200ms" Sep 10 13:28:34.281806 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:28:34.281779 2585 manager.go:354] Recovery completed Sep 10 13:28:34.287789 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:28:34.287776 2585 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Sep 10 13:28:34.289480 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:28:34.289462 2585 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-8.ec2.internal" event="NodeHasSufficientMemory" Sep 10 13:28:34.289594 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:28:34.289497 2585 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-8.ec2.internal" event="NodeHasNoDiskPressure" Sep 10 13:28:34.289594 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:28:34.289511 2585 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-8.ec2.internal" event="NodeHasSufficientPID" Sep 10 13:28:34.289942 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:28:34.289925 2585 cpu_manager.go:222] "Starting CPU manager" policy="none" Sep 10 13:28:34.289942 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:28:34.289942 2585 cpu_manager.go:223] "Reconciling" reconcilePeriod="10s" Sep 10 13:28:34.290056 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:28:34.289948 2585 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Sep 10 13:28:34.290056 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:28:34.289958 2585 state_mem.go:36] "Initialized new in-memory state store" Sep 10 13:28:34.290698 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:28:34.290684 2585 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-8.ec2.internal" event="NodeHasSufficientMemory" Sep 10 13:28:34.290772 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:28:34.290712 2585 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-8.ec2.internal" event="NodeHasNoDiskPressure" Sep 10 13:28:34.290772 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:28:34.290726 2585 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-8.ec2.internal" event="NodeHasSufficientPID" Sep 10 13:28:34.292108 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:28:34.292095 2585 policy_none.go:49] "None policy: Start" Sep 10 13:28:34.292164 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:28:34.292112 2585 memory_manager.go:187] "Starting memorymanager" policy="None" Sep 10 13:28:34.292164 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:28:34.292121 2585 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint" Sep 10 13:28:34.292529 ip-10-0-140-8 kubenswrapper[2585]: E0910 13:28:34.292464 2585 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-140-8.ec2.internal.18d3f8b094bbb2e6 default 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-140-8.ec2.internal,UID:ip-10-0-140-8.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeHasSufficientMemory,Message:Node ip-10-0-140-8.ec2.internal status is now: NodeHasSufficientMemory,Source:EventSource{Component:kubelet,Host:ip-10-0-140-8.ec2.internal,},FirstTimestamp:2026-09-10 13:28:34.289480422 +0000 UTC m=+0.434170365,LastTimestamp:2026-09-10 13:28:34.289480422 +0000 UTC m=+0.434170365,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-140-8.ec2.internal,}" Sep 10 13:28:34.294197 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:28:34.294181 2585 policy_none.go:47] "Start" Sep 10 13:28:34.306654 ip-10-0-140-8 kubenswrapper[2585]: E0910 13:28:34.306580 2585 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-140-8.ec2.internal.18d3f8b094bc1284 default 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-140-8.ec2.internal,UID:ip-10-0-140-8.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeHasNoDiskPressure,Message:Node ip-10-0-140-8.ec2.internal status is now: NodeHasNoDiskPressure,Source:EventSource{Component:kubelet,Host:ip-10-0-140-8.ec2.internal,},FirstTimestamp:2026-09-10 13:28:34.2895049 +0000 UTC m=+0.434194833,LastTimestamp:2026-09-10 13:28:34.2895049 +0000 UTC m=+0.434194833,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-140-8.ec2.internal,}" Sep 10 13:28:34.309829 ip-10-0-140-8 kubenswrapper[2585]: E0910 13:28:34.309769 2585 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-140-8.ec2.internal.18d3f8b094bc386e default 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-140-8.ec2.internal,UID:ip-10-0-140-8.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeHasSufficientPID,Message:Node ip-10-0-140-8.ec2.internal status is now: NodeHasSufficientPID,Source:EventSource{Component:kubelet,Host:ip-10-0-140-8.ec2.internal,},FirstTimestamp:2026-09-10 13:28:34.289514606 +0000 UTC m=+0.434204539,LastTimestamp:2026-09-10 13:28:34.289514606 +0000 UTC m=+0.434204539,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-140-8.ec2.internal,}" Sep 10 13:28:34.313513 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:28:34.313496 2585 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-p6frn" Sep 10 13:28:34.321562 ip-10-0-140-8 kubenswrapper[2585]: E0910 13:28:34.321494 2585 event.go:359] "Server rejected event (will not retry!)" err="events \"ip-10-0-140-8.ec2.internal.18d3f8b094bbb2e6\" is forbidden: User \"system:anonymous\" cannot patch resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-140-8.ec2.internal.18d3f8b094bbb2e6 default 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-140-8.ec2.internal,UID:ip-10-0-140-8.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeHasSufficientMemory,Message:Node ip-10-0-140-8.ec2.internal status is now: NodeHasSufficientMemory,Source:EventSource{Component:kubelet,Host:ip-10-0-140-8.ec2.internal,},FirstTimestamp:2026-09-10 13:28:34.289480422 +0000 UTC m=+0.434170365,LastTimestamp:2026-09-10 13:28:34.29069678 +0000 UTC m=+0.435386714,Count:2,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-140-8.ec2.internal,}" Sep 10 13:28:34.348099 ip-10-0-140-8 kubenswrapper[2585]: E0910 13:28:34.334077 2585 event.go:359] "Server rejected event (will not retry!)" err="events \"ip-10-0-140-8.ec2.internal.18d3f8b094bc1284\" is forbidden: User \"system:anonymous\" cannot patch resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-140-8.ec2.internal.18d3f8b094bc1284 default 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-140-8.ec2.internal,UID:ip-10-0-140-8.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeHasNoDiskPressure,Message:Node ip-10-0-140-8.ec2.internal status is now: NodeHasNoDiskPressure,Source:EventSource{Component:kubelet,Host:ip-10-0-140-8.ec2.internal,},FirstTimestamp:2026-09-10 13:28:34.2895049 +0000 UTC m=+0.434194833,LastTimestamp:2026-09-10 13:28:34.290718173 +0000 UTC m=+0.435408114,Count:2,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-140-8.ec2.internal,}" Sep 10 13:28:34.348099 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:28:34.337911 2585 manager.go:337] "Starting Device Plugin manager" Sep 10 13:28:34.348099 ip-10-0-140-8 kubenswrapper[2585]: E0910 13:28:34.337948 2585 manager.go:513] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint" Sep 10 13:28:34.348099 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:28:34.337957 2585 server.go:87] "Starting device plugin registration server" Sep 10 13:28:34.348099 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:28:34.338246 2585 eviction_manager.go:189] "Eviction manager: starting control loop" Sep 10 13:28:34.348099 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:28:34.338269 2585 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s" Sep 10 13:28:34.348099 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:28:34.338364 2585 plugin_watcher.go:51] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry" Sep 10 13:28:34.348099 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:28:34.338480 2585 plugin_manager.go:116] "The desired_state_of_world populator (plugin watcher) starts" Sep 10 13:28:34.348099 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:28:34.338489 2585 plugin_manager.go:118] "Starting Kubelet Plugin Manager" Sep 10 13:28:34.348099 ip-10-0-140-8 kubenswrapper[2585]: E0910 13:28:34.338927 2585 eviction_manager.go:267] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\"" Sep 10 13:28:34.348099 ip-10-0-140-8 kubenswrapper[2585]: E0910 13:28:34.338970 2585 eviction_manager.go:292] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-140-8.ec2.internal\" not found" Sep 10 13:28:34.348099 ip-10-0-140-8 kubenswrapper[2585]: E0910 13:28:34.347838 2585 event.go:359] "Server rejected event (will not retry!)" err="events \"ip-10-0-140-8.ec2.internal.18d3f8b094bc386e\" is forbidden: User \"system:anonymous\" cannot patch resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-140-8.ec2.internal.18d3f8b094bc386e default 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-140-8.ec2.internal,UID:ip-10-0-140-8.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeHasSufficientPID,Message:Node ip-10-0-140-8.ec2.internal status is now: NodeHasSufficientPID,Source:EventSource{Component:kubelet,Host:ip-10-0-140-8.ec2.internal,},FirstTimestamp:2026-09-10 13:28:34.289514606 +0000 UTC m=+0.434204539,LastTimestamp:2026-09-10 13:28:34.290731651 +0000 UTC m=+0.435421585,Count:2,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-140-8.ec2.internal,}" Sep 10 13:28:34.358751 ip-10-0-140-8 kubenswrapper[2585]: E0910 13:28:34.358687 2585 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-140-8.ec2.internal.18d3f8b097d97226 default 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-140-8.ec2.internal,UID:ip-10-0-140-8.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeAllocatableEnforced,Message:Updated Node Allocatable limit across pods,Source:EventSource{Component:kubelet,Host:ip-10-0-140-8.ec2.internal,},FirstTimestamp:2026-09-10 13:28:34.341761574 +0000 UTC m=+0.486451494,LastTimestamp:2026-09-10 13:28:34.341761574 +0000 UTC m=+0.486451494,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-140-8.ec2.internal,}" Sep 10 13:28:34.367229 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:28:34.367188 2585 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4" Sep 10 13:28:34.368393 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:28:34.368375 2585 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6" Sep 10 13:28:34.368489 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:28:34.368417 2585 status_manager.go:244] "Starting to sync pod status with apiserver" Sep 10 13:28:34.368489 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:28:34.368438 2585 kubelet.go:2448] "Starting kubelet main sync loop" Sep 10 13:28:34.368489 ip-10-0-140-8 kubenswrapper[2585]: E0910 13:28:34.368480 2585 kubelet.go:2472] "Skipping pod synchronization" err="PLEG is not healthy: pleg has yet to be successful" Sep 10 13:28:34.378091 ip-10-0-140-8 kubenswrapper[2585]: E0910 13:28:34.378064 2585 reflector.go:205] "Failed to watch" err="failed to list *v1.RuntimeClass: runtimeclasses.node.k8s.io is forbidden: User \"system:anonymous\" cannot list resource \"runtimeclasses\" in API group \"node.k8s.io\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.RuntimeClass" Sep 10 13:28:34.439328 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:28:34.439222 2585 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Sep 10 13:28:34.440271 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:28:34.440239 2585 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-8.ec2.internal" event="NodeHasSufficientMemory" Sep 10 13:28:34.440336 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:28:34.440286 2585 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-8.ec2.internal" event="NodeHasNoDiskPressure" Sep 10 13:28:34.440336 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:28:34.440301 2585 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-8.ec2.internal" event="NodeHasSufficientPID" Sep 10 13:28:34.440336 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:28:34.440328 2585 kubelet_node_status.go:78] "Attempting to register node" node="ip-10-0-140-8.ec2.internal" Sep 10 13:28:34.450114 ip-10-0-140-8 kubenswrapper[2585]: E0910 13:28:34.450018 2585 event.go:359] "Server rejected event (will not retry!)" err="events \"ip-10-0-140-8.ec2.internal.18d3f8b094bbb2e6\" is forbidden: User \"system:anonymous\" cannot patch resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-140-8.ec2.internal.18d3f8b094bbb2e6 default 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-140-8.ec2.internal,UID:ip-10-0-140-8.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeHasSufficientMemory,Message:Node ip-10-0-140-8.ec2.internal status is now: NodeHasSufficientMemory,Source:EventSource{Component:kubelet,Host:ip-10-0-140-8.ec2.internal,},FirstTimestamp:2026-09-10 13:28:34.289480422 +0000 UTC m=+0.434170365,LastTimestamp:2026-09-10 13:28:34.440270014 +0000 UTC m=+0.584959947,Count:3,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-140-8.ec2.internal,}" Sep 10 13:28:34.459316 ip-10-0-140-8 kubenswrapper[2585]: E0910 13:28:34.459291 2585 kubelet_node_status.go:116] "Unable to register node with API server, error getting existing node" err="nodes \"ip-10-0-140-8.ec2.internal\" is forbidden: User \"system:anonymous\" cannot get resource \"nodes\" in API group \"\" at the cluster scope" node="ip-10-0-140-8.ec2.internal" Sep 10 13:28:34.459393 ip-10-0-140-8 kubenswrapper[2585]: E0910 13:28:34.459280 2585 event.go:359] "Server rejected event (will not retry!)" err="events \"ip-10-0-140-8.ec2.internal.18d3f8b094bc1284\" is forbidden: User \"system:anonymous\" cannot patch resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-140-8.ec2.internal.18d3f8b094bc1284 default 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-140-8.ec2.internal,UID:ip-10-0-140-8.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeHasNoDiskPressure,Message:Node ip-10-0-140-8.ec2.internal status is now: NodeHasNoDiskPressure,Source:EventSource{Component:kubelet,Host:ip-10-0-140-8.ec2.internal,},FirstTimestamp:2026-09-10 13:28:34.2895049 +0000 UTC m=+0.434194833,LastTimestamp:2026-09-10 13:28:34.440294257 +0000 UTC m=+0.584984191,Count:3,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-140-8.ec2.internal,}" Sep 10 13:28:34.468920 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:28:34.468891 2585 kubelet.go:2535] "SyncLoop ADD" source="file" pods=["kube-system/kube-apiserver-proxy-ip-10-0-140-8.ec2.internal","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-8.ec2.internal"] Sep 10 13:28:34.469031 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:28:34.468974 2585 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Sep 10 13:28:34.471064 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:28:34.471044 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/e9d508517cba13afe9410d7b1b4171d3-config\") pod \"kube-apiserver-proxy-ip-10-0-140-8.ec2.internal\" (UID: \"e9d508517cba13afe9410d7b1b4171d3\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-140-8.ec2.internal" Sep 10 13:28:34.471356 ip-10-0-140-8 kubenswrapper[2585]: E0910 13:28:34.471292 2585 event.go:359] "Server rejected event (will not retry!)" err="events \"ip-10-0-140-8.ec2.internal.18d3f8b094bc386e\" is forbidden: User \"system:anonymous\" cannot patch resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-140-8.ec2.internal.18d3f8b094bc386e default 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-140-8.ec2.internal,UID:ip-10-0-140-8.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeHasSufficientPID,Message:Node ip-10-0-140-8.ec2.internal status is now: NodeHasSufficientPID,Source:EventSource{Component:kubelet,Host:ip-10-0-140-8.ec2.internal,},FirstTimestamp:2026-09-10 13:28:34.289514606 +0000 UTC m=+0.434204539,LastTimestamp:2026-09-10 13:28:34.440306987 +0000 UTC m=+0.584996922,Count:3,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-140-8.ec2.internal,}" Sep 10 13:28:34.471643 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:28:34.471626 2585 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-8.ec2.internal" event="NodeHasSufficientMemory" Sep 10 13:28:34.471709 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:28:34.471664 2585 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-8.ec2.internal" event="NodeHasNoDiskPressure" Sep 10 13:28:34.471709 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:28:34.471675 2585 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-8.ec2.internal" event="NodeHasSufficientPID" Sep 10 13:28:34.473974 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:28:34.473960 2585 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Sep 10 13:28:34.474078 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:28:34.474061 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-140-8.ec2.internal" Sep 10 13:28:34.474137 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:28:34.474092 2585 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Sep 10 13:28:34.474797 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:28:34.474783 2585 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-8.ec2.internal" event="NodeHasSufficientMemory" Sep 10 13:28:34.474876 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:28:34.474786 2585 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-8.ec2.internal" event="NodeHasSufficientMemory" Sep 10 13:28:34.474876 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:28:34.474829 2585 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-8.ec2.internal" event="NodeHasNoDiskPressure" Sep 10 13:28:34.474876 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:28:34.474839 2585 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-8.ec2.internal" event="NodeHasSufficientPID" Sep 10 13:28:34.474876 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:28:34.474808 2585 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-8.ec2.internal" event="NodeHasNoDiskPressure" Sep 10 13:28:34.475043 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:28:34.474883 2585 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-8.ec2.internal" event="NodeHasSufficientPID" Sep 10 13:28:34.477147 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:28:34.477133 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-8.ec2.internal" Sep 10 13:28:34.477211 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:28:34.477158 2585 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Sep 10 13:28:34.478122 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:28:34.478104 2585 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-8.ec2.internal" event="NodeHasSufficientMemory" Sep 10 13:28:34.478220 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:28:34.478137 2585 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-8.ec2.internal" event="NodeHasNoDiskPressure" Sep 10 13:28:34.478220 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:28:34.478153 2585 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-8.ec2.internal" event="NodeHasSufficientPID" Sep 10 13:28:34.481771 ip-10-0-140-8 kubenswrapper[2585]: E0910 13:28:34.481751 2585 controller.go:145] "Failed to ensure lease exists, will retry" err="leases.coordination.k8s.io \"ip-10-0-140-8.ec2.internal\" is forbidden: User \"system:anonymous\" cannot get resource \"leases\" in API group \"coordination.k8s.io\" in the namespace \"kube-node-lease\"" interval="400ms" Sep 10 13:28:34.481843 ip-10-0-140-8 kubenswrapper[2585]: E0910 13:28:34.481754 2585 event.go:359] "Server rejected event (will not retry!)" err="events \"ip-10-0-140-8.ec2.internal.18d3f8b094bbb2e6\" is forbidden: User \"system:anonymous\" cannot patch resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-140-8.ec2.internal.18d3f8b094bbb2e6 default 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-140-8.ec2.internal,UID:ip-10-0-140-8.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeHasSufficientMemory,Message:Node ip-10-0-140-8.ec2.internal status is now: NodeHasSufficientMemory,Source:EventSource{Component:kubelet,Host:ip-10-0-140-8.ec2.internal,},FirstTimestamp:2026-09-10 13:28:34.289480422 +0000 UTC m=+0.434170365,LastTimestamp:2026-09-10 13:28:34.471645097 +0000 UTC m=+0.616335034,Count:4,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-140-8.ec2.internal,}" Sep 10 13:28:34.496109 ip-10-0-140-8 kubenswrapper[2585]: E0910 13:28:34.496040 2585 event.go:359] "Server rejected event (will not retry!)" err="events \"ip-10-0-140-8.ec2.internal.18d3f8b094bc1284\" is forbidden: User \"system:anonymous\" cannot patch resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-140-8.ec2.internal.18d3f8b094bc1284 default 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-140-8.ec2.internal,UID:ip-10-0-140-8.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeHasNoDiskPressure,Message:Node ip-10-0-140-8.ec2.internal status is now: NodeHasNoDiskPressure,Source:EventSource{Component:kubelet,Host:ip-10-0-140-8.ec2.internal,},FirstTimestamp:2026-09-10 13:28:34.2895049 +0000 UTC m=+0.434194833,LastTimestamp:2026-09-10 13:28:34.471670026 +0000 UTC m=+0.616359958,Count:4,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-140-8.ec2.internal,}" Sep 10 13:28:34.500936 ip-10-0-140-8 kubenswrapper[2585]: E0910 13:28:34.500919 2585 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-140-8.ec2.internal\" not found" node="ip-10-0-140-8.ec2.internal" Sep 10 13:28:34.505502 ip-10-0-140-8 kubenswrapper[2585]: E0910 13:28:34.505488 2585 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-140-8.ec2.internal\" not found" node="ip-10-0-140-8.ec2.internal" Sep 10 13:28:34.507232 ip-10-0-140-8 kubenswrapper[2585]: E0910 13:28:34.507172 2585 event.go:359] "Server rejected event (will not retry!)" err="events \"ip-10-0-140-8.ec2.internal.18d3f8b094bc386e\" is forbidden: User \"system:anonymous\" cannot patch resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-140-8.ec2.internal.18d3f8b094bc386e default 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-140-8.ec2.internal,UID:ip-10-0-140-8.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeHasSufficientPID,Message:Node ip-10-0-140-8.ec2.internal status is now: NodeHasSufficientPID,Source:EventSource{Component:kubelet,Host:ip-10-0-140-8.ec2.internal,},FirstTimestamp:2026-09-10 13:28:34.289514606 +0000 UTC m=+0.434204539,LastTimestamp:2026-09-10 13:28:34.471679335 +0000 UTC m=+0.616369268,Count:4,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-140-8.ec2.internal,}" Sep 10 13:28:34.517210 ip-10-0-140-8 kubenswrapper[2585]: E0910 13:28:34.517150 2585 event.go:359] "Server rejected event (will not retry!)" err="events \"ip-10-0-140-8.ec2.internal.18d3f8b094bbb2e6\" is forbidden: User \"system:anonymous\" cannot patch resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-140-8.ec2.internal.18d3f8b094bbb2e6 default 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-140-8.ec2.internal,UID:ip-10-0-140-8.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeHasSufficientMemory,Message:Node ip-10-0-140-8.ec2.internal status is now: NodeHasSufficientMemory,Source:EventSource{Component:kubelet,Host:ip-10-0-140-8.ec2.internal,},FirstTimestamp:2026-09-10 13:28:34.289480422 +0000 UTC m=+0.434170365,LastTimestamp:2026-09-10 13:28:34.47480047 +0000 UTC m=+0.619490403,Count:5,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-140-8.ec2.internal,}" Sep 10 13:28:34.526478 ip-10-0-140-8 kubenswrapper[2585]: E0910 13:28:34.526409 2585 event.go:359] "Server rejected event (will not retry!)" err="events \"ip-10-0-140-8.ec2.internal.18d3f8b094bbb2e6\" is forbidden: User \"system:anonymous\" cannot patch resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-140-8.ec2.internal.18d3f8b094bbb2e6 default 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-140-8.ec2.internal,UID:ip-10-0-140-8.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeHasSufficientMemory,Message:Node ip-10-0-140-8.ec2.internal status is now: NodeHasSufficientMemory,Source:EventSource{Component:kubelet,Host:ip-10-0-140-8.ec2.internal,},FirstTimestamp:2026-09-10 13:28:34.289480422 +0000 UTC m=+0.434170365,LastTimestamp:2026-09-10 13:28:34.47481866 +0000 UTC m=+0.619508592,Count:6,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-140-8.ec2.internal,}" Sep 10 13:28:34.537306 ip-10-0-140-8 kubenswrapper[2585]: E0910 13:28:34.537217 2585 event.go:359] "Server rejected event (will not retry!)" err="events \"ip-10-0-140-8.ec2.internal.18d3f8b094bc1284\" is forbidden: User \"system:anonymous\" cannot patch resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-140-8.ec2.internal.18d3f8b094bc1284 default 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-140-8.ec2.internal,UID:ip-10-0-140-8.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeHasNoDiskPressure,Message:Node ip-10-0-140-8.ec2.internal status is now: NodeHasNoDiskPressure,Source:EventSource{Component:kubelet,Host:ip-10-0-140-8.ec2.internal,},FirstTimestamp:2026-09-10 13:28:34.2895049 +0000 UTC m=+0.434194833,LastTimestamp:2026-09-10 13:28:34.474834005 +0000 UTC m=+0.619523938,Count:5,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-140-8.ec2.internal,}" Sep 10 13:28:34.548953 ip-10-0-140-8 kubenswrapper[2585]: E0910 13:28:34.548896 2585 event.go:359] "Server rejected event (will not retry!)" err="events \"ip-10-0-140-8.ec2.internal.18d3f8b094bc386e\" is forbidden: User \"system:anonymous\" cannot patch resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-140-8.ec2.internal.18d3f8b094bc386e default 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-140-8.ec2.internal,UID:ip-10-0-140-8.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeHasSufficientPID,Message:Node ip-10-0-140-8.ec2.internal status is now: NodeHasSufficientPID,Source:EventSource{Component:kubelet,Host:ip-10-0-140-8.ec2.internal,},FirstTimestamp:2026-09-10 13:28:34.289514606 +0000 UTC m=+0.434204539,LastTimestamp:2026-09-10 13:28:34.474842973 +0000 UTC m=+0.619532905,Count:5,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-140-8.ec2.internal,}" Sep 10 13:28:34.559012 ip-10-0-140-8 kubenswrapper[2585]: E0910 13:28:34.558953 2585 event.go:359] "Server rejected event (will not retry!)" err="events \"ip-10-0-140-8.ec2.internal.18d3f8b094bc1284\" is forbidden: User \"system:anonymous\" cannot patch resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-140-8.ec2.internal.18d3f8b094bc1284 default 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-140-8.ec2.internal,UID:ip-10-0-140-8.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeHasNoDiskPressure,Message:Node ip-10-0-140-8.ec2.internal status is now: NodeHasNoDiskPressure,Source:EventSource{Component:kubelet,Host:ip-10-0-140-8.ec2.internal,},FirstTimestamp:2026-09-10 13:28:34.2895049 +0000 UTC m=+0.434194833,LastTimestamp:2026-09-10 13:28:34.47487484 +0000 UTC m=+0.619564775,Count:6,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-140-8.ec2.internal,}" Sep 10 13:28:34.570298 ip-10-0-140-8 kubenswrapper[2585]: E0910 13:28:34.570223 2585 event.go:359] "Server rejected event (will not retry!)" err="events \"ip-10-0-140-8.ec2.internal.18d3f8b094bc386e\" is forbidden: User \"system:anonymous\" cannot patch resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-140-8.ec2.internal.18d3f8b094bc386e default 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-140-8.ec2.internal,UID:ip-10-0-140-8.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeHasSufficientPID,Message:Node ip-10-0-140-8.ec2.internal status is now: NodeHasSufficientPID,Source:EventSource{Component:kubelet,Host:ip-10-0-140-8.ec2.internal,},FirstTimestamp:2026-09-10 13:28:34.289514606 +0000 UTC m=+0.434204539,LastTimestamp:2026-09-10 13:28:34.474889805 +0000 UTC m=+0.619579740,Count:6,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-140-8.ec2.internal,}" Sep 10 13:28:34.571918 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:28:34.571900 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/13b304e90198285e2c639c5158a21371-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-140-8.ec2.internal\" (UID: \"13b304e90198285e2c639c5158a21371\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-8.ec2.internal" Sep 10 13:28:34.571965 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:28:34.571946 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/e9d508517cba13afe9410d7b1b4171d3-config\") pod \"kube-apiserver-proxy-ip-10-0-140-8.ec2.internal\" (UID: \"e9d508517cba13afe9410d7b1b4171d3\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-140-8.ec2.internal" Sep 10 13:28:34.572006 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:28:34.571963 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/13b304e90198285e2c639c5158a21371-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-140-8.ec2.internal\" (UID: \"13b304e90198285e2c639c5158a21371\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-8.ec2.internal" Sep 10 13:28:34.572006 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:28:34.571996 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/e9d508517cba13afe9410d7b1b4171d3-config\") pod \"kube-apiserver-proxy-ip-10-0-140-8.ec2.internal\" (UID: \"e9d508517cba13afe9410d7b1b4171d3\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-140-8.ec2.internal" Sep 10 13:28:34.579416 ip-10-0-140-8 kubenswrapper[2585]: E0910 13:28:34.579350 2585 event.go:359] "Server rejected event (will not retry!)" err="events \"ip-10-0-140-8.ec2.internal.18d3f8b094bbb2e6\" is forbidden: User \"system:anonymous\" cannot patch resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-140-8.ec2.internal.18d3f8b094bbb2e6 default 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-140-8.ec2.internal,UID:ip-10-0-140-8.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeHasSufficientMemory,Message:Node ip-10-0-140-8.ec2.internal status is now: NodeHasSufficientMemory,Source:EventSource{Component:kubelet,Host:ip-10-0-140-8.ec2.internal,},FirstTimestamp:2026-09-10 13:28:34.289480422 +0000 UTC m=+0.434170365,LastTimestamp:2026-09-10 13:28:34.478120961 +0000 UTC m=+0.622810899,Count:7,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-140-8.ec2.internal,}" Sep 10 13:28:34.594174 ip-10-0-140-8 kubenswrapper[2585]: E0910 13:28:34.594104 2585 event.go:359] "Server rejected event (will not retry!)" err="events \"ip-10-0-140-8.ec2.internal.18d3f8b094bc1284\" is forbidden: User \"system:anonymous\" cannot patch resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-140-8.ec2.internal.18d3f8b094bc1284 default 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-140-8.ec2.internal,UID:ip-10-0-140-8.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeHasNoDiskPressure,Message:Node ip-10-0-140-8.ec2.internal status is now: NodeHasNoDiskPressure,Source:EventSource{Component:kubelet,Host:ip-10-0-140-8.ec2.internal,},FirstTimestamp:2026-09-10 13:28:34.2895049 +0000 UTC m=+0.434194833,LastTimestamp:2026-09-10 13:28:34.478144248 +0000 UTC m=+0.622834182,Count:7,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-140-8.ec2.internal,}" Sep 10 13:28:34.608108 ip-10-0-140-8 kubenswrapper[2585]: E0910 13:28:34.608045 2585 event.go:359] "Server rejected event (will not retry!)" err="events \"ip-10-0-140-8.ec2.internal.18d3f8b094bc386e\" is forbidden: User \"system:anonymous\" cannot patch resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-140-8.ec2.internal.18d3f8b094bc386e default 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-140-8.ec2.internal,UID:ip-10-0-140-8.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeHasSufficientPID,Message:Node ip-10-0-140-8.ec2.internal status is now: NodeHasSufficientPID,Source:EventSource{Component:kubelet,Host:ip-10-0-140-8.ec2.internal,},FirstTimestamp:2026-09-10 13:28:34.289514606 +0000 UTC m=+0.434204539,LastTimestamp:2026-09-10 13:28:34.478159632 +0000 UTC m=+0.622849566,Count:7,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-140-8.ec2.internal,}" Sep 10 13:28:34.660185 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:28:34.660149 2585 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Sep 10 13:28:34.661117 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:28:34.661097 2585 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-8.ec2.internal" event="NodeHasSufficientMemory" Sep 10 13:28:34.661218 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:28:34.661128 2585 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-8.ec2.internal" event="NodeHasNoDiskPressure" Sep 10 13:28:34.661218 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:28:34.661139 2585 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-8.ec2.internal" event="NodeHasSufficientPID" Sep 10 13:28:34.661218 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:28:34.661166 2585 kubelet_node_status.go:78] "Attempting to register node" node="ip-10-0-140-8.ec2.internal" Sep 10 13:28:34.672171 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:28:34.672150 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/13b304e90198285e2c639c5158a21371-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-140-8.ec2.internal\" (UID: \"13b304e90198285e2c639c5158a21371\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-8.ec2.internal" Sep 10 13:28:34.672287 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:28:34.672174 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/13b304e90198285e2c639c5158a21371-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-140-8.ec2.internal\" (UID: \"13b304e90198285e2c639c5158a21371\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-8.ec2.internal" Sep 10 13:28:34.672287 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:28:34.672204 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/13b304e90198285e2c639c5158a21371-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-140-8.ec2.internal\" (UID: \"13b304e90198285e2c639c5158a21371\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-8.ec2.internal" Sep 10 13:28:34.672287 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:28:34.672227 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/13b304e90198285e2c639c5158a21371-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-140-8.ec2.internal\" (UID: \"13b304e90198285e2c639c5158a21371\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-8.ec2.internal" Sep 10 13:28:34.672895 ip-10-0-140-8 kubenswrapper[2585]: E0910 13:28:34.672831 2585 event.go:359] "Server rejected event (will not retry!)" err="events \"ip-10-0-140-8.ec2.internal.18d3f8b094bbb2e6\" is forbidden: User \"system:anonymous\" cannot patch resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-140-8.ec2.internal.18d3f8b094bbb2e6 default 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-140-8.ec2.internal,UID:ip-10-0-140-8.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeHasSufficientMemory,Message:Node ip-10-0-140-8.ec2.internal status is now: NodeHasSufficientMemory,Source:EventSource{Component:kubelet,Host:ip-10-0-140-8.ec2.internal,},FirstTimestamp:2026-09-10 13:28:34.289480422 +0000 UTC m=+0.434170365,LastTimestamp:2026-09-10 13:28:34.661114438 +0000 UTC m=+0.805804371,Count:8,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-140-8.ec2.internal,}" Sep 10 13:28:34.682195 ip-10-0-140-8 kubenswrapper[2585]: E0910 13:28:34.682124 2585 event.go:359] "Server rejected event (will not retry!)" err="events \"ip-10-0-140-8.ec2.internal.18d3f8b094bc1284\" is forbidden: User \"system:anonymous\" cannot patch resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-140-8.ec2.internal.18d3f8b094bc1284 default 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-140-8.ec2.internal,UID:ip-10-0-140-8.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeHasNoDiskPressure,Message:Node ip-10-0-140-8.ec2.internal status is now: NodeHasNoDiskPressure,Source:EventSource{Component:kubelet,Host:ip-10-0-140-8.ec2.internal,},FirstTimestamp:2026-09-10 13:28:34.2895049 +0000 UTC m=+0.434194833,LastTimestamp:2026-09-10 13:28:34.66113379 +0000 UTC m=+0.805823724,Count:8,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-140-8.ec2.internal,}" Sep 10 13:28:34.682311 ip-10-0-140-8 kubenswrapper[2585]: E0910 13:28:34.682213 2585 kubelet_node_status.go:116] "Unable to register node with API server, error getting existing node" err="nodes \"ip-10-0-140-8.ec2.internal\" is forbidden: User \"system:anonymous\" cannot get resource \"nodes\" in API group \"\" at the cluster scope" node="ip-10-0-140-8.ec2.internal" Sep 10 13:28:34.802601 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:28:34.802574 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-140-8.ec2.internal" Sep 10 13:28:34.808285 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:28:34.808252 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-8.ec2.internal" Sep 10 13:28:34.891896 ip-10-0-140-8 kubenswrapper[2585]: E0910 13:28:34.891852 2585 controller.go:145] "Failed to ensure lease exists, will retry" err="leases.coordination.k8s.io \"ip-10-0-140-8.ec2.internal\" is forbidden: User \"system:anonymous\" cannot get resource \"leases\" in API group \"coordination.k8s.io\" in the namespace \"kube-node-lease\"" interval="800ms" Sep 10 13:28:36.354739 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:28:36.354648 2585 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Sep 10 13:28:36.356604 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:28:36.356585 2585 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-8.ec2.internal" event="NodeHasSufficientMemory" Sep 10 13:28:36.356722 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:28:36.356620 2585 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-8.ec2.internal" event="NodeHasNoDiskPressure" Sep 10 13:28:36.356722 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:28:36.356634 2585 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-8.ec2.internal" event="NodeHasSufficientPID" Sep 10 13:28:36.356722 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:28:36.356666 2585 kubelet_node_status.go:78] "Attempting to register node" node="ip-10-0-140-8.ec2.internal" Sep 10 13:28:36.375173 ip-10-0-140-8 kubenswrapper[2585]: E0910 13:28:36.375141 2585 kubelet_node_status.go:116] "Unable to register node with API server, error getting existing node" err="nodes \"ip-10-0-140-8.ec2.internal\" is forbidden: User \"system:anonymous\" cannot get resource \"nodes\" in API group \"\" at the cluster scope" node="ip-10-0-140-8.ec2.internal" Sep 10 13:28:36.399414 ip-10-0-140-8 kubenswrapper[2585]: E0910 13:28:36.399389 2585 reflector.go:205] "Failed to watch" err="failed to list *v1.Service: services is forbidden: User \"system:anonymous\" cannot list resource \"services\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Service" Sep 10 13:28:36.433352 ip-10-0-140-8 kubenswrapper[2585]: E0910 13:28:36.433323 2585 reflector.go:205] "Failed to watch" err="failed to list *v1.Node: nodes \"ip-10-0-140-8.ec2.internal\" is forbidden: User \"system:anonymous\" cannot list resource \"nodes\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Node" Sep 10 13:28:36.541434 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:28:36.541409 2585 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-140-8.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope Sep 10 13:28:36.648076 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:28:36.648008 2585 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Sep 10 13:28:36.648894 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:28:36.648874 2585 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-8.ec2.internal" event="NodeHasSufficientMemory" Sep 10 13:28:36.648996 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:28:36.648904 2585 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-8.ec2.internal" event="NodeHasNoDiskPressure" Sep 10 13:28:36.648996 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:28:36.648917 2585 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-8.ec2.internal" event="NodeHasSufficientPID" Sep 10 13:28:36.688924 ip-10-0-140-8 kubenswrapper[2585]: E0910 13:28:36.688889 2585 reflector.go:205] "Failed to watch" err="failed to list *v1.CSIDriver: csidrivers.storage.k8s.io is forbidden: User \"system:anonymous\" cannot list resource \"csidrivers\" in API group \"storage.k8s.io\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.CSIDriver" Sep 10 13:28:36.974220 ip-10-0-140-8 kubenswrapper[2585]: E0910 13:28:36.974191 2585 controller.go:145] "Failed to ensure lease exists, will retry" err="leases.coordination.k8s.io \"ip-10-0-140-8.ec2.internal\" is forbidden: User \"system:anonymous\" cannot get resource \"leases\" in API group \"coordination.k8s.io\" in the namespace \"kube-node-lease\"" interval="1.6s" Sep 10 13:28:36.979906 ip-10-0-140-8 kubenswrapper[2585]: W0910 13:28:36.979877 2585 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode9d508517cba13afe9410d7b1b4171d3.slice/crio-f2b8c1e81ffb3f03c05fbcdd45ebbc6b4fd7f335a7de1b08a076c6b61ca46882 WatchSource:0}: Error finding container f2b8c1e81ffb3f03c05fbcdd45ebbc6b4fd7f335a7de1b08a076c6b61ca46882: Status 404 returned error can't find the container with id f2b8c1e81ffb3f03c05fbcdd45ebbc6b4fd7f335a7de1b08a076c6b61ca46882 Sep 10 13:28:36.980218 ip-10-0-140-8 kubenswrapper[2585]: W0910 13:28:36.980195 2585 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod13b304e90198285e2c639c5158a21371.slice/crio-7559703836be33fb7b5d7eb362b0bbf8c63d5fac9424c9db66ee5bc3059c6898 WatchSource:0}: Error finding container 7559703836be33fb7b5d7eb362b0bbf8c63d5fac9424c9db66ee5bc3059c6898: Status 404 returned error can't find the container with id 7559703836be33fb7b5d7eb362b0bbf8c63d5fac9424c9db66ee5bc3059c6898 Sep 10 13:28:36.985570 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:28:36.985551 2585 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Sep 10 13:28:36.995769 ip-10-0-140-8 kubenswrapper[2585]: E0910 13:28:36.995697 2585 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"kube-system\"" event="&Event{ObjectMeta:{kube-apiserver-proxy-ip-10-0-140-8.ec2.internal.18d3f8b1357119a6 kube-system 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Pod,Namespace:kube-system,Name:kube-apiserver-proxy-ip-10-0-140-8.ec2.internal,UID:e9d508517cba13afe9410d7b1b4171d3,APIVersion:v1,ResourceVersion:,FieldPath:spec.containers{haproxy},},Reason:Pulling,Message:Pulling image \"quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:da130d90f0bffd5c5876c94544c91b2a1be0d23ce2747ddfeafe88e03ee7d8d1\",Source:EventSource{Component:kubelet,Host:ip-10-0-140-8.ec2.internal,},FirstTimestamp:2026-09-10 13:28:36.985723302 +0000 UTC m=+1.858427480,LastTimestamp:2026-09-10 13:28:36.985723302 +0000 UTC m=+1.858427480,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-140-8.ec2.internal,}" Sep 10 13:28:37.005848 ip-10-0-140-8 kubenswrapper[2585]: E0910 13:28:37.005782 2585 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"openshift-machine-config-operator\"" event="&Event{ObjectMeta:{kube-rbac-proxy-crio-ip-10-0-140-8.ec2.internal.18d3f8b1357202c1 openshift-machine-config-operator 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Pod,Namespace:openshift-machine-config-operator,Name:kube-rbac-proxy-crio-ip-10-0-140-8.ec2.internal,UID:13b304e90198285e2c639c5158a21371,APIVersion:v1,ResourceVersion:,FieldPath:spec.initContainers{setup},},Reason:Pulling,Message:Pulling image \"quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:cec847b1f08cae0315bd44fe477d725a0180b78ab0ad844de3b218bf2dde0bb2\",Source:EventSource{Component:kubelet,Host:ip-10-0-140-8.ec2.internal,},FirstTimestamp:2026-09-10 13:28:36.985782977 +0000 UTC m=+1.858487157,LastTimestamp:2026-09-10 13:28:36.985782977 +0000 UTC m=+1.858487157,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-140-8.ec2.internal,}" Sep 10 13:28:37.156223 ip-10-0-140-8 kubenswrapper[2585]: E0910 13:28:37.156181 2585 reflector.go:205] "Failed to watch" err="failed to list *v1.RuntimeClass: runtimeclasses.node.k8s.io is forbidden: User \"system:anonymous\" cannot list resource \"runtimeclasses\" in API group \"node.k8s.io\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.RuntimeClass" Sep 10 13:28:37.175284 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:28:37.175258 2585 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Sep 10 13:28:37.176324 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:28:37.176308 2585 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-8.ec2.internal" event="NodeHasSufficientMemory" Sep 10 13:28:37.176424 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:28:37.176342 2585 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-8.ec2.internal" event="NodeHasNoDiskPressure" Sep 10 13:28:37.176424 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:28:37.176356 2585 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-8.ec2.internal" event="NodeHasSufficientPID" Sep 10 13:28:37.176424 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:28:37.176394 2585 kubelet_node_status.go:78] "Attempting to register node" node="ip-10-0-140-8.ec2.internal" Sep 10 13:28:37.194926 ip-10-0-140-8 kubenswrapper[2585]: E0910 13:28:37.194903 2585 kubelet_node_status.go:116] "Unable to register node with API server, error getting existing node" err="nodes \"ip-10-0-140-8.ec2.internal\" is forbidden: User \"system:anonymous\" cannot get resource \"nodes\" in API group \"\" at the cluster scope" node="ip-10-0-140-8.ec2.internal" Sep 10 13:28:37.542911 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:28:37.542872 2585 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-140-8.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope Sep 10 13:28:37.647159 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:28:37.647100 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-8.ec2.internal" event={"ID":"13b304e90198285e2c639c5158a21371","Type":"ContainerStarted","Data":"7559703836be33fb7b5d7eb362b0bbf8c63d5fac9424c9db66ee5bc3059c6898"} Sep 10 13:28:37.649189 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:28:37.649155 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-140-8.ec2.internal" event={"ID":"e9d508517cba13afe9410d7b1b4171d3","Type":"ContainerStarted","Data":"f2b8c1e81ffb3f03c05fbcdd45ebbc6b4fd7f335a7de1b08a076c6b61ca46882"} Sep 10 13:28:38.458605 ip-10-0-140-8 kubenswrapper[2585]: E0910 13:28:38.458564 2585 reflector.go:205] "Failed to watch" err="failed to list *v1.Service: services is forbidden: User \"system:anonymous\" cannot list resource \"services\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Service" Sep 10 13:28:38.539813 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:28:38.539782 2585 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-140-8.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope Sep 10 13:28:38.585533 ip-10-0-140-8 kubenswrapper[2585]: E0910 13:28:38.585492 2585 controller.go:145] "Failed to ensure lease exists, will retry" err="leases.coordination.k8s.io \"ip-10-0-140-8.ec2.internal\" is forbidden: User \"system:anonymous\" cannot get resource \"leases\" in API group \"coordination.k8s.io\" in the namespace \"kube-node-lease\"" interval="3.2s" Sep 10 13:28:38.795799 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:28:38.795704 2585 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Sep 10 13:28:38.796937 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:28:38.796914 2585 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-8.ec2.internal" event="NodeHasSufficientMemory" Sep 10 13:28:38.797061 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:28:38.796949 2585 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-8.ec2.internal" event="NodeHasNoDiskPressure" Sep 10 13:28:38.797061 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:28:38.796964 2585 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-8.ec2.internal" event="NodeHasSufficientPID" Sep 10 13:28:38.797061 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:28:38.796999 2585 kubelet_node_status.go:78] "Attempting to register node" node="ip-10-0-140-8.ec2.internal" Sep 10 13:28:38.817152 ip-10-0-140-8 kubenswrapper[2585]: E0910 13:28:38.817109 2585 kubelet_node_status.go:116] "Unable to register node with API server, error getting existing node" err="nodes \"ip-10-0-140-8.ec2.internal\" is forbidden: User \"system:anonymous\" cannot get resource \"nodes\" in API group \"\" at the cluster scope" node="ip-10-0-140-8.ec2.internal" Sep 10 13:28:38.858965 ip-10-0-140-8 kubenswrapper[2585]: E0910 13:28:38.858928 2585 reflector.go:205] "Failed to watch" err="failed to list *v1.Node: nodes \"ip-10-0-140-8.ec2.internal\" is forbidden: User \"system:anonymous\" cannot list resource \"nodes\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Node" Sep 10 13:28:38.973475 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:28:38.973435 2585 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Sep 10 13:28:38.974485 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:28:38.974464 2585 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-8.ec2.internal" event="NodeHasSufficientMemory" Sep 10 13:28:38.974602 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:28:38.974500 2585 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-8.ec2.internal" event="NodeHasNoDiskPressure" Sep 10 13:28:38.974602 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:28:38.974514 2585 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-8.ec2.internal" event="NodeHasSufficientPID" Sep 10 13:28:39.006993 ip-10-0-140-8 kubenswrapper[2585]: E0910 13:28:39.006922 2585 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"kube-system\"" event="&Event{ObjectMeta:{kube-apiserver-proxy-ip-10-0-140-8.ec2.internal.18d3f8b1ad64ad0d kube-system 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Pod,Namespace:kube-system,Name:kube-apiserver-proxy-ip-10-0-140-8.ec2.internal,UID:e9d508517cba13afe9410d7b1b4171d3,APIVersion:v1,ResourceVersion:,FieldPath:spec.containers{haproxy},},Reason:Pulled,Message:Successfully pulled image \"quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:da130d90f0bffd5c5876c94544c91b2a1be0d23ce2747ddfeafe88e03ee7d8d1\" in 2.012s (2.012s including waiting). Image size: 321671430 bytes.,Source:EventSource{Component:kubelet,Host:ip-10-0-140-8.ec2.internal,},FirstTimestamp:2026-09-10 13:28:38.998174989 +0000 UTC m=+3.870879181,LastTimestamp:2026-09-10 13:28:38.998174989 +0000 UTC m=+3.870879181,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-140-8.ec2.internal,}" Sep 10 13:28:39.016009 ip-10-0-140-8 kubenswrapper[2585]: E0910 13:28:39.015930 2585 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"openshift-machine-config-operator\"" event="&Event{ObjectMeta:{kube-rbac-proxy-crio-ip-10-0-140-8.ec2.internal.18d3f8b1ad8ba34d openshift-machine-config-operator 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Pod,Namespace:openshift-machine-config-operator,Name:kube-rbac-proxy-crio-ip-10-0-140-8.ec2.internal,UID:13b304e90198285e2c639c5158a21371,APIVersion:v1,ResourceVersion:,FieldPath:spec.initContainers{setup},},Reason:Pulled,Message:Successfully pulled image \"quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:cec847b1f08cae0315bd44fe477d725a0180b78ab0ad844de3b218bf2dde0bb2\" in 2.014s (2.014s including waiting). Image size: 282837706 bytes.,Source:EventSource{Component:kubelet,Host:ip-10-0-140-8.ec2.internal,},FirstTimestamp:2026-09-10 13:28:39.000728397 +0000 UTC m=+3.873432617,LastTimestamp:2026-09-10 13:28:39.000728397 +0000 UTC m=+3.873432617,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-140-8.ec2.internal,}" Sep 10 13:28:39.069120 ip-10-0-140-8 kubenswrapper[2585]: E0910 13:28:39.068833 2585 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"kube-system\"" event="&Event{ObjectMeta:{kube-apiserver-proxy-ip-10-0-140-8.ec2.internal.18d3f8b1b1059627 kube-system 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Pod,Namespace:kube-system,Name:kube-apiserver-proxy-ip-10-0-140-8.ec2.internal,UID:e9d508517cba13afe9410d7b1b4171d3,APIVersion:v1,ResourceVersion:,FieldPath:spec.containers{haproxy},},Reason:Created,Message:Created container: haproxy,Source:EventSource{Component:kubelet,Host:ip-10-0-140-8.ec2.internal,},FirstTimestamp:2026-09-10 13:28:39.059052071 +0000 UTC m=+3.931756263,LastTimestamp:2026-09-10 13:28:39.059052071 +0000 UTC m=+3.931756263,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-140-8.ec2.internal,}" Sep 10 13:28:39.078459 ip-10-0-140-8 kubenswrapper[2585]: E0910 13:28:39.078356 2585 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"kube-system\"" event="&Event{ObjectMeta:{kube-apiserver-proxy-ip-10-0-140-8.ec2.internal.18d3f8b1b17bc3ef kube-system 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Pod,Namespace:kube-system,Name:kube-apiserver-proxy-ip-10-0-140-8.ec2.internal,UID:e9d508517cba13afe9410d7b1b4171d3,APIVersion:v1,ResourceVersion:,FieldPath:spec.containers{haproxy},},Reason:Started,Message:Started container haproxy,Source:EventSource{Component:kubelet,Host:ip-10-0-140-8.ec2.internal,},FirstTimestamp:2026-09-10 13:28:39.066797039 +0000 UTC m=+3.939501217,LastTimestamp:2026-09-10 13:28:39.066797039 +0000 UTC m=+3.939501217,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-140-8.ec2.internal,}" Sep 10 13:28:39.280848 ip-10-0-140-8 kubenswrapper[2585]: E0910 13:28:39.280763 2585 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"openshift-machine-config-operator\"" event="&Event{ObjectMeta:{kube-rbac-proxy-crio-ip-10-0-140-8.ec2.internal.18d3f8b1bd9324c8 openshift-machine-config-operator 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Pod,Namespace:openshift-machine-config-operator,Name:kube-rbac-proxy-crio-ip-10-0-140-8.ec2.internal,UID:13b304e90198285e2c639c5158a21371,APIVersion:v1,ResourceVersion:,FieldPath:spec.initContainers{setup},},Reason:Created,Message:Created container: setup,Source:EventSource{Component:kubelet,Host:ip-10-0-140-8.ec2.internal,},FirstTimestamp:2026-09-10 13:28:39.269655752 +0000 UTC m=+4.142359945,LastTimestamp:2026-09-10 13:28:39.269655752 +0000 UTC m=+4.142359945,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-140-8.ec2.internal,}" Sep 10 13:28:39.289660 ip-10-0-140-8 kubenswrapper[2585]: E0910 13:28:39.289591 2585 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"openshift-machine-config-operator\"" event="&Event{ObjectMeta:{kube-rbac-proxy-crio-ip-10-0-140-8.ec2.internal.18d3f8b1be1ebafe openshift-machine-config-operator 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Pod,Namespace:openshift-machine-config-operator,Name:kube-rbac-proxy-crio-ip-10-0-140-8.ec2.internal,UID:13b304e90198285e2c639c5158a21371,APIVersion:v1,ResourceVersion:,FieldPath:spec.initContainers{setup},},Reason:Started,Message:Started container setup,Source:EventSource{Component:kubelet,Host:ip-10-0-140-8.ec2.internal,},FirstTimestamp:2026-09-10 13:28:39.27880371 +0000 UTC m=+4.151507910,LastTimestamp:2026-09-10 13:28:39.27880371 +0000 UTC m=+4.151507910,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-140-8.ec2.internal,}" Sep 10 13:28:39.308029 ip-10-0-140-8 kubenswrapper[2585]: E0910 13:28:39.307970 2585 reflector.go:205] "Failed to watch" err="failed to list *v1.CSIDriver: csidrivers.storage.k8s.io is forbidden: User \"system:anonymous\" cannot list resource \"csidrivers\" in API group \"storage.k8s.io\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.CSIDriver" Sep 10 13:28:39.541058 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:28:39.541019 2585 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-140-8.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope Sep 10 13:28:39.581235 ip-10-0-140-8 kubenswrapper[2585]: E0910 13:28:39.581198 2585 reflector.go:205] "Failed to watch" err="failed to list *v1.RuntimeClass: runtimeclasses.node.k8s.io is forbidden: User \"system:anonymous\" cannot list resource \"runtimeclasses\" in API group \"node.k8s.io\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.RuntimeClass" Sep 10 13:28:39.654809 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:28:39.654777 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-8.ec2.internal" event={"ID":"13b304e90198285e2c639c5158a21371","Type":"ContainerStarted","Data":"2bf530a7f40750d6e2f4653eade1223a580aa818515170971108be91ef4f5a74"} Sep 10 13:28:39.654809 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:28:39.654797 2585 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Sep 10 13:28:39.657324 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:28:39.657300 2585 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-8.ec2.internal" event="NodeHasSufficientMemory" Sep 10 13:28:39.657453 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:28:39.657340 2585 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-8.ec2.internal" event="NodeHasNoDiskPressure" Sep 10 13:28:39.657453 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:28:39.657355 2585 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-8.ec2.internal" event="NodeHasSufficientPID" Sep 10 13:28:39.657559 ip-10-0-140-8 kubenswrapper[2585]: E0910 13:28:39.657543 2585 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-140-8.ec2.internal\" not found" node="ip-10-0-140-8.ec2.internal" Sep 10 13:28:39.658627 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:28:39.658604 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-140-8.ec2.internal" event={"ID":"e9d508517cba13afe9410d7b1b4171d3","Type":"ContainerStarted","Data":"5b87abbfb9f46d9071a4c2516283e35bdbff856d53670047bfe4800aae4ade7b"} Sep 10 13:28:39.658708 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:28:39.658659 2585 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Sep 10 13:28:39.659345 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:28:39.659332 2585 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-8.ec2.internal" event="NodeHasSufficientMemory" Sep 10 13:28:39.659389 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:28:39.659356 2585 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-8.ec2.internal" event="NodeHasNoDiskPressure" Sep 10 13:28:39.659389 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:28:39.659366 2585 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-8.ec2.internal" event="NodeHasSufficientPID" Sep 10 13:28:39.659513 ip-10-0-140-8 kubenswrapper[2585]: E0910 13:28:39.659500 2585 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-140-8.ec2.internal\" not found" node="ip-10-0-140-8.ec2.internal" Sep 10 13:28:40.553410 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:28:40.553380 2585 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-140-8.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope Sep 10 13:28:40.660556 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:28:40.660532 2585 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Sep 10 13:28:40.661047 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:28:40.660532 2585 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Sep 10 13:28:40.661644 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:28:40.661619 2585 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-8.ec2.internal" event="NodeHasSufficientMemory" Sep 10 13:28:40.661644 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:28:40.661632 2585 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-8.ec2.internal" event="NodeHasSufficientMemory" Sep 10 13:28:40.661644 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:28:40.661648 2585 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-8.ec2.internal" event="NodeHasNoDiskPressure" Sep 10 13:28:40.661872 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:28:40.661659 2585 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-8.ec2.internal" event="NodeHasSufficientPID" Sep 10 13:28:40.661872 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:28:40.661686 2585 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-8.ec2.internal" event="NodeHasNoDiskPressure" Sep 10 13:28:40.661872 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:28:40.661697 2585 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-8.ec2.internal" event="NodeHasSufficientPID" Sep 10 13:28:40.661872 ip-10-0-140-8 kubenswrapper[2585]: E0910 13:28:40.661846 2585 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-140-8.ec2.internal\" not found" node="ip-10-0-140-8.ec2.internal" Sep 10 13:28:40.662062 ip-10-0-140-8 kubenswrapper[2585]: E0910 13:28:40.661997 2585 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-140-8.ec2.internal\" not found" node="ip-10-0-140-8.ec2.internal" Sep 10 13:28:41.545357 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:28:41.545327 2585 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-140-8.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope Sep 10 13:28:41.796793 ip-10-0-140-8 kubenswrapper[2585]: E0910 13:28:41.796713 2585 controller.go:145] "Failed to ensure lease exists, will retry" err="leases.coordination.k8s.io \"ip-10-0-140-8.ec2.internal\" is forbidden: User \"system:anonymous\" cannot get resource \"leases\" in API group \"coordination.k8s.io\" in the namespace \"kube-node-lease\"" interval="6.4s" Sep 10 13:28:42.017681 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:28:42.017651 2585 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Sep 10 13:28:42.018699 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:28:42.018681 2585 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-8.ec2.internal" event="NodeHasSufficientMemory" Sep 10 13:28:42.018758 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:28:42.018719 2585 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-8.ec2.internal" event="NodeHasNoDiskPressure" Sep 10 13:28:42.018758 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:28:42.018730 2585 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-8.ec2.internal" event="NodeHasSufficientPID" Sep 10 13:28:42.018758 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:28:42.018757 2585 kubelet_node_status.go:78] "Attempting to register node" node="ip-10-0-140-8.ec2.internal" Sep 10 13:28:42.037698 ip-10-0-140-8 kubenswrapper[2585]: E0910 13:28:42.037671 2585 kubelet_node_status.go:116] "Unable to register node with API server, error getting existing node" err="nodes \"ip-10-0-140-8.ec2.internal\" is forbidden: User \"system:anonymous\" cannot get resource \"nodes\" in API group \"\" at the cluster scope" node="ip-10-0-140-8.ec2.internal" Sep 10 13:28:42.116067 ip-10-0-140-8 kubenswrapper[2585]: E0910 13:28:42.116038 2585 reflector.go:205] "Failed to watch" err="failed to list *v1.Node: nodes \"ip-10-0-140-8.ec2.internal\" is forbidden: User \"system:anonymous\" cannot list resource \"nodes\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Node" Sep 10 13:28:42.545566 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:28:42.545502 2585 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-140-8.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope Sep 10 13:28:42.863752 ip-10-0-140-8 kubenswrapper[2585]: E0910 13:28:42.863719 2585 reflector.go:205] "Failed to watch" err="failed to list *v1.Service: services is forbidden: User \"system:anonymous\" cannot list resource \"services\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Service" Sep 10 13:28:43.050932 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:28:43.050880 2585 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Sep 10 13:28:43.052848 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:28:43.052831 2585 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-8.ec2.internal" event="NodeHasSufficientMemory" Sep 10 13:28:43.052921 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:28:43.052865 2585 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-8.ec2.internal" event="NodeHasNoDiskPressure" Sep 10 13:28:43.052921 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:28:43.052878 2585 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-8.ec2.internal" event="NodeHasSufficientPID" Sep 10 13:28:43.164473 ip-10-0-140-8 kubenswrapper[2585]: E0910 13:28:43.164390 2585 reflector.go:205] "Failed to watch" err="failed to list *v1.CSIDriver: csidrivers.storage.k8s.io is forbidden: User \"system:anonymous\" cannot list resource \"csidrivers\" in API group \"storage.k8s.io\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.CSIDriver" Sep 10 13:28:43.543844 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:28:43.543761 2585 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-140-8.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope Sep 10 13:28:44.503435 ip-10-0-140-8 kubenswrapper[2585]: E0910 13:28:44.503352 2585 reflector.go:205] "Failed to watch" err="failed to list *v1.RuntimeClass: runtimeclasses.node.k8s.io is forbidden: User \"system:anonymous\" cannot list resource \"runtimeclasses\" in API group \"node.k8s.io\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.RuntimeClass" Sep 10 13:28:44.539424 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:28:44.539396 2585 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-140-8.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope Sep 10 13:28:45.540263 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:28:45.540214 2585 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-140-8.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope Sep 10 13:28:45.611890 ip-10-0-140-8 kubenswrapper[2585]: E0910 13:28:45.611842 2585 eviction_manager.go:292] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-140-8.ec2.internal\" not found" Sep 10 13:28:46.541354 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:28:46.541319 2585 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-140-8.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope Sep 10 13:28:47.540589 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:28:47.540557 2585 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-140-8.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope Sep 10 13:28:48.205851 ip-10-0-140-8 kubenswrapper[2585]: E0910 13:28:48.205812 2585 controller.go:145] "Failed to ensure lease exists, will retry" err="leases.coordination.k8s.io \"ip-10-0-140-8.ec2.internal\" is forbidden: User \"system:anonymous\" cannot get resource \"leases\" in API group \"coordination.k8s.io\" in the namespace \"kube-node-lease\"" interval="7s" Sep 10 13:28:48.437793 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:28:48.437759 2585 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Sep 10 13:28:48.438830 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:28:48.438814 2585 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-8.ec2.internal" event="NodeHasSufficientMemory" Sep 10 13:28:48.438910 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:28:48.438843 2585 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-8.ec2.internal" event="NodeHasNoDiskPressure" Sep 10 13:28:48.438910 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:28:48.438853 2585 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-8.ec2.internal" event="NodeHasSufficientPID" Sep 10 13:28:48.438910 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:28:48.438880 2585 kubelet_node_status.go:78] "Attempting to register node" node="ip-10-0-140-8.ec2.internal" Sep 10 13:28:48.455212 ip-10-0-140-8 kubenswrapper[2585]: E0910 13:28:48.455178 2585 kubelet_node_status.go:116] "Unable to register node with API server, error getting existing node" err="nodes \"ip-10-0-140-8.ec2.internal\" is forbidden: User \"system:anonymous\" cannot get resource \"nodes\" in API group \"\" at the cluster scope" node="ip-10-0-140-8.ec2.internal" Sep 10 13:28:48.541149 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:28:48.541073 2585 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-140-8.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope Sep 10 13:28:49.540265 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:28:49.540225 2585 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-140-8.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope Sep 10 13:28:50.547874 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:28:50.547839 2585 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-140-8.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope Sep 10 13:28:50.653817 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:28:50.653730 2585 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-p6frn" Sep 10 13:28:50.677581 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:28:50.677549 2585 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-machine-config-operator_kube-rbac-proxy-crio-ip-10-0-140-8.ec2.internal_13b304e90198285e2c639c5158a21371/setup/0.log" Sep 10 13:28:50.677749 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:28:50.677587 2585 generic.go:356] "Generic (PLEG): container finished" podID="13b304e90198285e2c639c5158a21371" containerID="2bf530a7f40750d6e2f4653eade1223a580aa818515170971108be91ef4f5a74" exitCode=1 Sep 10 13:28:50.677749 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:28:50.677623 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-8.ec2.internal" event={"ID":"13b304e90198285e2c639c5158a21371","Type":"ContainerDied","Data":"2bf530a7f40750d6e2f4653eade1223a580aa818515170971108be91ef4f5a74"} Sep 10 13:28:50.677859 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:28:50.677749 2585 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Sep 10 13:28:50.678595 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:28:50.678578 2585 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-8.ec2.internal" event="NodeHasSufficientMemory" Sep 10 13:28:50.678781 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:28:50.678613 2585 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-8.ec2.internal" event="NodeHasNoDiskPressure" Sep 10 13:28:50.678781 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:28:50.678627 2585 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-8.ec2.internal" event="NodeHasSufficientPID" Sep 10 13:28:50.678899 ip-10-0-140-8 kubenswrapper[2585]: E0910 13:28:50.678862 2585 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-140-8.ec2.internal\" not found" node="ip-10-0-140-8.ec2.internal" Sep 10 13:28:51.433500 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:28:51.433473 2585 reflector.go:439] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:160" Sep 10 13:28:51.474773 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:28:51.474747 2585 transport.go:147] "Certificate rotation detected, shutting down client connections to start using new credentials" Sep 10 13:28:51.474919 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:28:51.474889 2585 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Node" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received" Sep 10 13:28:51.528354 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:28:51.528309 2585 apiserver.go:52] "Watching apiserver" Sep 10 13:28:51.555065 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:28:51.555032 2585 reflector.go:439] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66" Sep 10 13:28:51.555532 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:28:51.555110 2585 kubelet.go:2535] "SyncLoop ADD" source="api" pods=[] Sep 10 13:28:51.570358 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:28:51.570330 2585 nodeinfomanager.go:417] Failed to publish CSINode: nodes "ip-10-0-140-8.ec2.internal" not found Sep 10 13:28:51.599736 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:28:51.599711 2585 nodeinfomanager.go:417] Failed to publish CSINode: nodes "ip-10-0-140-8.ec2.internal" not found Sep 10 13:28:51.642036 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:28:51.642012 2585 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world" Sep 10 13:28:51.657954 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:28:51.657913 2585 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-09-09 13:23:50 +0000 UTC" deadline="2028-05-17 11:28:46.496422206 +0000 UTC" Sep 10 13:28:51.657954 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:28:51.657948 2585 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="14757h59m54.838477008s" Sep 10 13:28:51.661826 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:28:51.661812 2585 nodeinfomanager.go:417] Failed to publish CSINode: nodes "ip-10-0-140-8.ec2.internal" not found Sep 10 13:28:51.680578 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:28:51.680555 2585 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-machine-config-operator_kube-rbac-proxy-crio-ip-10-0-140-8.ec2.internal_13b304e90198285e2c639c5158a21371/setup/0.log" Sep 10 13:28:51.680678 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:28:51.680601 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-8.ec2.internal" event={"ID":"13b304e90198285e2c639c5158a21371","Type":"ContainerStarted","Data":"315d5848780927b4716cdf165cfb4e5552db7a32d37b279ceeb0eb32267f0dbf"} Sep 10 13:28:51.680729 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:28:51.680701 2585 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Sep 10 13:28:51.682358 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:28:51.682335 2585 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-8.ec2.internal" event="NodeHasSufficientMemory" Sep 10 13:28:51.682474 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:28:51.682367 2585 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-8.ec2.internal" event="NodeHasNoDiskPressure" Sep 10 13:28:51.682474 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:28:51.682385 2585 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-8.ec2.internal" event="NodeHasSufficientPID" Sep 10 13:28:51.682601 ip-10-0-140-8 kubenswrapper[2585]: E0910 13:28:51.682586 2585 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-140-8.ec2.internal\" not found" node="ip-10-0-140-8.ec2.internal" Sep 10 13:28:51.932068 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:28:51.932042 2585 nodeinfomanager.go:417] Failed to publish CSINode: nodes "ip-10-0-140-8.ec2.internal" not found Sep 10 13:28:51.932068 ip-10-0-140-8 kubenswrapper[2585]: E0910 13:28:51.932068 2585 csi_plugin.go:399] Failed to initialize CSINode: error updating CSINode annotation: timed out waiting for the condition; caused by: nodes "ip-10-0-140-8.ec2.internal" not found Sep 10 13:28:51.975935 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:28:51.975909 2585 nodeinfomanager.go:417] Failed to publish CSINode: nodes "ip-10-0-140-8.ec2.internal" not found Sep 10 13:28:51.995966 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:28:51.995945 2585 nodeinfomanager.go:417] Failed to publish CSINode: nodes "ip-10-0-140-8.ec2.internal" not found Sep 10 13:28:52.061750 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:28:52.061727 2585 nodeinfomanager.go:417] Failed to publish CSINode: nodes "ip-10-0-140-8.ec2.internal" not found Sep 10 13:28:52.241600 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:28:52.241524 2585 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Sep 10 13:28:52.242574 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:28:52.242555 2585 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-8.ec2.internal" event="NodeHasSufficientMemory" Sep 10 13:28:52.242683 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:28:52.242590 2585 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-8.ec2.internal" event="NodeHasNoDiskPressure" Sep 10 13:28:52.242683 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:28:52.242605 2585 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-8.ec2.internal" event="NodeHasSufficientPID" Sep 10 13:28:52.312588 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:28:52.312559 2585 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160" Sep 10 13:28:52.369189 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:28:52.369162 2585 nodeinfomanager.go:417] Failed to publish CSINode: nodes "ip-10-0-140-8.ec2.internal" not found Sep 10 13:28:52.369189 ip-10-0-140-8 kubenswrapper[2585]: E0910 13:28:52.369188 2585 csi_plugin.go:399] Failed to initialize CSINode: error updating CSINode annotation: timed out waiting for the condition; caused by: nodes "ip-10-0-140-8.ec2.internal" not found Sep 10 13:28:52.630313 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:28:52.630282 2585 nodeinfomanager.go:417] Failed to publish CSINode: nodes "ip-10-0-140-8.ec2.internal" not found Sep 10 13:28:52.648010 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:28:52.647992 2585 nodeinfomanager.go:417] Failed to publish CSINode: nodes "ip-10-0-140-8.ec2.internal" not found Sep 10 13:28:52.708105 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:28:52.708078 2585 nodeinfomanager.go:417] Failed to publish CSINode: nodes "ip-10-0-140-8.ec2.internal" not found Sep 10 13:28:52.983976 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:28:52.983900 2585 nodeinfomanager.go:417] Failed to publish CSINode: nodes "ip-10-0-140-8.ec2.internal" not found Sep 10 13:28:52.983976 ip-10-0-140-8 kubenswrapper[2585]: E0910 13:28:52.983926 2585 csi_plugin.go:399] Failed to initialize CSINode: error updating CSINode annotation: timed out waiting for the condition; caused by: nodes "ip-10-0-140-8.ec2.internal" not found Sep 10 13:28:53.878746 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:28:53.878715 2585 reflector.go:439] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:160" Sep 10 13:28:54.777023 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:28:54.776995 2585 reflector.go:439] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:160" Sep 10 13:28:54.963743 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:28:54.963716 2585 nodeinfomanager.go:417] Failed to publish CSINode: nodes "ip-10-0-140-8.ec2.internal" not found Sep 10 13:28:54.982075 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:28:54.982058 2585 nodeinfomanager.go:417] Failed to publish CSINode: nodes "ip-10-0-140-8.ec2.internal" not found Sep 10 13:28:55.043156 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:28:55.043095 2585 nodeinfomanager.go:417] Failed to publish CSINode: nodes "ip-10-0-140-8.ec2.internal" not found Sep 10 13:28:55.218130 ip-10-0-140-8 kubenswrapper[2585]: E0910 13:28:55.218102 2585 nodelease.go:49] "Failed to get node when trying to set owner ref to the node lease" err="nodes \"ip-10-0-140-8.ec2.internal\" not found" node="ip-10-0-140-8.ec2.internal" Sep 10 13:28:55.317791 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:28:55.317769 2585 nodeinfomanager.go:417] Failed to publish CSINode: nodes "ip-10-0-140-8.ec2.internal" not found Sep 10 13:28:55.317888 ip-10-0-140-8 kubenswrapper[2585]: E0910 13:28:55.317795 2585 csi_plugin.go:399] Failed to initialize CSINode: error updating CSINode annotation: timed out waiting for the condition; caused by: nodes "ip-10-0-140-8.ec2.internal" not found Sep 10 13:28:55.456062 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:28:55.456030 2585 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Sep 10 13:28:55.457418 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:28:55.457399 2585 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-8.ec2.internal" event="NodeHasSufficientMemory" Sep 10 13:28:55.457503 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:28:55.457432 2585 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-8.ec2.internal" event="NodeHasNoDiskPressure" Sep 10 13:28:55.457503 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:28:55.457444 2585 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-8.ec2.internal" event="NodeHasSufficientPID" Sep 10 13:28:55.457503 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:28:55.457471 2585 kubelet_node_status.go:78] "Attempting to register node" node="ip-10-0-140-8.ec2.internal" Sep 10 13:28:55.465395 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:28:55.465379 2585 kubelet_node_status.go:81] "Successfully registered node" node="ip-10-0-140-8.ec2.internal" Sep 10 13:28:55.465450 ip-10-0-140-8 kubenswrapper[2585]: E0910 13:28:55.465401 2585 kubelet_node_status.go:536] "Error updating node status, will retry" err="error getting node \"ip-10-0-140-8.ec2.internal\": node \"ip-10-0-140-8.ec2.internal\" not found" Sep 10 13:28:55.480862 ip-10-0-140-8 kubenswrapper[2585]: E0910 13:28:55.480838 2585 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-140-8.ec2.internal\" not found" Sep 10 13:28:55.492234 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:28:55.492213 2585 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kube-system/konnectivity-agent-fxgmp"] Sep 10 13:28:55.492335 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:28:55.492301 2585 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Sep 10 13:28:55.493326 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:28:55.493311 2585 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-8.ec2.internal" event="NodeHasSufficientMemory" Sep 10 13:28:55.493410 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:28:55.493343 2585 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-8.ec2.internal" event="NodeHasNoDiskPressure" Sep 10 13:28:55.493410 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:28:55.493358 2585 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-8.ec2.internal" event="NodeHasSufficientPID" Sep 10 13:28:55.499598 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:28:55.499581 2585 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-network-diagnostics/network-check-target-7262t"] Sep 10 13:28:55.499672 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:28:55.499660 2585 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Sep 10 13:28:55.499733 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:28:55.499692 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-fxgmp" Sep 10 13:28:55.499733 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:28:55.499724 2585 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Sep 10 13:28:55.500438 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:28:55.500423 2585 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-8.ec2.internal" event="NodeHasSufficientMemory" Sep 10 13:28:55.500496 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:28:55.500452 2585 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-8.ec2.internal" event="NodeHasNoDiskPressure" Sep 10 13:28:55.500496 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:28:55.500463 2585 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-8.ec2.internal" event="NodeHasSufficientPID" Sep 10 13:28:55.500560 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:28:55.500425 2585 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-8.ec2.internal" event="NodeHasSufficientMemory" Sep 10 13:28:55.500560 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:28:55.500516 2585 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-8.ec2.internal" event="NodeHasNoDiskPressure" Sep 10 13:28:55.500560 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:28:55.500525 2585 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-8.ec2.internal" event="NodeHasSufficientPID" Sep 10 13:28:55.503187 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:28:55.503172 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-7262t" Sep 10 13:28:55.503266 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:28:55.503197 2585 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Sep 10 13:28:55.503468 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:28:55.503451 2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\"" Sep 10 13:28:55.503848 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:28:55.503835 2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\"" Sep 10 13:28:55.503888 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:28:55.503855 2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"default-dockercfg-b2hqk\"" Sep 10 13:28:55.506268 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:28:55.506236 2585 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-8.ec2.internal" event="NodeHasSufficientMemory" Sep 10 13:28:55.506343 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:28:55.506279 2585 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-8.ec2.internal" event="NodeHasNoDiskPressure" Sep 10 13:28:55.506343 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:28:55.506289 2585 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-8.ec2.internal" event="NodeHasSufficientPID" Sep 10 13:28:55.506343 ip-10-0-140-8 kubenswrapper[2585]: E0910 13:28:55.506336 2585 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-7262t" podUID="082a7577-a14f-49fb-a9b1-3cf250e6735a" Sep 10 13:28:55.524734 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:28:55.524708 2585 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-multus/multus-l9l8q"] Sep 10 13:28:55.524830 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:28:55.524787 2585 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Sep 10 13:28:55.525613 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:28:55.525593 2585 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-8.ec2.internal" event="NodeHasSufficientMemory" Sep 10 13:28:55.525714 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:28:55.525626 2585 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-8.ec2.internal" event="NodeHasNoDiskPressure" Sep 10 13:28:55.525714 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:28:55.525639 2585 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-8.ec2.internal" event="NodeHasSufficientPID" Sep 10 13:28:55.528385 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:28:55.528366 2585 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-image-registry/node-ca-zhn9x"] Sep 10 13:28:55.528483 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:28:55.528453 2585 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Sep 10 13:28:55.528483 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:28:55.528466 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-l9l8q" Sep 10 13:28:55.528602 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:28:55.528489 2585 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Sep 10 13:28:55.529258 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:28:55.529228 2585 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-8.ec2.internal" event="NodeHasSufficientMemory" Sep 10 13:28:55.529258 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:28:55.529239 2585 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-8.ec2.internal" event="NodeHasSufficientMemory" Sep 10 13:28:55.529386 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:28:55.529272 2585 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-8.ec2.internal" event="NodeHasNoDiskPressure" Sep 10 13:28:55.529386 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:28:55.529277 2585 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-8.ec2.internal" event="NodeHasNoDiskPressure" Sep 10 13:28:55.529386 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:28:55.529287 2585 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-8.ec2.internal" event="NodeHasSufficientPID" Sep 10 13:28:55.529386 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:28:55.529288 2585 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-8.ec2.internal" event="NodeHasSufficientPID" Sep 10 13:28:55.531669 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:28:55.531651 2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\"" Sep 10 13:28:55.531778 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:28:55.531761 2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\"" Sep 10 13:28:55.531851 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:28:55.531824 2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\"" Sep 10 13:28:55.531909 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:28:55.531869 2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\"" Sep 10 13:28:55.531954 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:28:55.531907 2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"default-dockercfg-67hsw\"" Sep 10 13:28:55.531990 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:28:55.531970 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-zhn9x" Sep 10 13:28:55.532031 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:28:55.532000 2585 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Sep 10 13:28:55.532748 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:28:55.532734 2585 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-8.ec2.internal" event="NodeHasSufficientMemory" Sep 10 13:28:55.532833 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:28:55.532762 2585 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-8.ec2.internal" event="NodeHasNoDiskPressure" Sep 10 13:28:55.532833 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:28:55.532775 2585 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-8.ec2.internal" event="NodeHasSufficientPID" Sep 10 13:28:55.535626 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:28:55.535607 2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\"" Sep 10 13:28:55.535732 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:28:55.535716 2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\"" Sep 10 13:28:55.535791 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:28:55.535746 2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\"" Sep 10 13:28:55.535961 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:28:55.535947 2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-497bg\"" Sep 10 13:28:55.555038 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:28:55.555013 2585 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-cluster-node-tuning-operator/tuned-s9hkb"] Sep 10 13:28:55.555165 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:28:55.555149 2585 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Sep 10 13:28:55.556037 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:28:55.556021 2585 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-8.ec2.internal" event="NodeHasSufficientMemory" Sep 10 13:28:55.556119 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:28:55.556047 2585 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-8.ec2.internal" event="NodeHasNoDiskPressure" Sep 10 13:28:55.556119 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:28:55.556057 2585 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-8.ec2.internal" event="NodeHasSufficientPID" Sep 10 13:28:55.558770 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:28:55.558755 2585 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-multus/network-metrics-daemon-xp5dk"] Sep 10 13:28:55.558851 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:28:55.558842 2585 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Sep 10 13:28:55.558889 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:28:55.558854 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-s9hkb" Sep 10 13:28:55.558889 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:28:55.558877 2585 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Sep 10 13:28:55.559607 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:28:55.559584 2585 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-8.ec2.internal" event="NodeHasSufficientMemory" Sep 10 13:28:55.559607 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:28:55.559605 2585 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-8.ec2.internal" event="NodeHasNoDiskPressure" Sep 10 13:28:55.559735 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:28:55.559615 2585 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-8.ec2.internal" event="NodeHasSufficientPID" Sep 10 13:28:55.559735 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:28:55.559628 2585 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-8.ec2.internal" event="NodeHasSufficientMemory" Sep 10 13:28:55.559735 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:28:55.559651 2585 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-8.ec2.internal" event="NodeHasNoDiskPressure" Sep 10 13:28:55.559735 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:28:55.559665 2585 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-8.ec2.internal" event="NodeHasSufficientPID" Sep 10 13:28:55.561814 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:28:55.561797 2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-4xxgx\"" Sep 10 13:28:55.562095 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:28:55.562078 2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\"" Sep 10 13:28:55.562176 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:28:55.562160 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-xp5dk" Sep 10 13:28:55.562228 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:28:55.562196 2585 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Sep 10 13:28:55.562683 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:28:55.562663 2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\"" Sep 10 13:28:55.562967 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:28:55.562954 2585 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-8.ec2.internal" event="NodeHasSufficientMemory" Sep 10 13:28:55.563037 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:28:55.562982 2585 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-8.ec2.internal" event="NodeHasNoDiskPressure" Sep 10 13:28:55.563037 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:28:55.562993 2585 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-8.ec2.internal" event="NodeHasSufficientPID" Sep 10 13:28:55.563113 ip-10-0-140-8 kubenswrapper[2585]: E0910 13:28:55.563036 2585 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-xp5dk" podUID="d552154f-1d5b-4c5d-99bf-7aeef22ce032" Sep 10 13:28:55.576585 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:28:55.576544 2585 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-multus/multus-additional-cni-plugins-vlmmc"] Sep 10 13:28:55.576801 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:28:55.576791 2585 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Sep 10 13:28:55.577518 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:28:55.577497 2585 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-8.ec2.internal" event="NodeHasSufficientMemory" Sep 10 13:28:55.577591 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:28:55.577521 2585 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-8.ec2.internal" event="NodeHasNoDiskPressure" Sep 10 13:28:55.577591 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:28:55.577531 2585 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-8.ec2.internal" event="NodeHasSufficientPID" Sep 10 13:28:55.580102 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:28:55.580088 2585 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ovn-kubernetes/ovnkube-node-5h2q6"] Sep 10 13:28:55.580217 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:28:55.580202 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-vlmmc" Sep 10 13:28:55.580278 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:28:55.580230 2585 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Sep 10 13:28:55.580401 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:28:55.580389 2585 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Sep 10 13:28:55.580890 ip-10-0-140-8 kubenswrapper[2585]: E0910 13:28:55.580875 2585 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-140-8.ec2.internal\" not found" Sep 10 13:28:55.580980 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:28:55.580947 2585 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-8.ec2.internal" event="NodeHasSufficientMemory" Sep 10 13:28:55.580980 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:28:55.580976 2585 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-8.ec2.internal" event="NodeHasNoDiskPressure" Sep 10 13:28:55.581075 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:28:55.580990 2585 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-8.ec2.internal" event="NodeHasSufficientPID" Sep 10 13:28:55.581075 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:28:55.580976 2585 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-8.ec2.internal" event="NodeHasSufficientMemory" Sep 10 13:28:55.581075 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:28:55.581046 2585 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-8.ec2.internal" event="NodeHasNoDiskPressure" Sep 10 13:28:55.581075 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:28:55.581073 2585 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-8.ec2.internal" event="NodeHasSufficientPID" Sep 10 13:28:55.583736 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:28:55.583708 2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\"" Sep 10 13:28:55.583736 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:28:55.583727 2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-628mh\"" Sep 10 13:28:55.583893 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:28:55.583728 2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\"" Sep 10 13:28:55.583893 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:28:55.583823 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-5h2q6" Sep 10 13:28:55.583893 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:28:55.583852 2585 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Sep 10 13:28:55.584594 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:28:55.584577 2585 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-8.ec2.internal" event="NodeHasSufficientMemory" Sep 10 13:28:55.584683 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:28:55.584604 2585 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-8.ec2.internal" event="NodeHasNoDiskPressure" Sep 10 13:28:55.584683 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:28:55.584614 2585 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-8.ec2.internal" event="NodeHasSufficientPID" Sep 10 13:28:55.587330 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:28:55.587309 2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\"" Sep 10 13:28:55.587330 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:28:55.587323 2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\"" Sep 10 13:28:55.587484 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:28:55.587333 2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\"" Sep 10 13:28:55.587524 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:28:55.587483 2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\"" Sep 10 13:28:55.587734 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:28:55.587719 2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-kqtrd\"" Sep 10 13:28:55.588759 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:28:55.588745 2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\"" Sep 10 13:28:55.588816 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:28:55.588770 2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\"" Sep 10 13:28:55.596605 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:28:55.596589 2585 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-network-operator/iptables-alerter-x2wmd"] Sep 10 13:28:55.596732 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:28:55.596723 2585 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Sep 10 13:28:55.597573 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:28:55.597555 2585 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-8.ec2.internal" event="NodeHasSufficientMemory" Sep 10 13:28:55.597662 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:28:55.597582 2585 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-8.ec2.internal" event="NodeHasNoDiskPressure" Sep 10 13:28:55.597662 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:28:55.597597 2585 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-8.ec2.internal" event="NodeHasSufficientPID" Sep 10 13:28:55.600305 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:28:55.600291 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-x2wmd" Sep 10 13:28:55.600351 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:28:55.600317 2585 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Sep 10 13:28:55.601031 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:28:55.601016 2585 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-8.ec2.internal" event="NodeHasSufficientMemory" Sep 10 13:28:55.601106 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:28:55.601040 2585 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-8.ec2.internal" event="NodeHasNoDiskPressure" Sep 10 13:28:55.601106 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:28:55.601050 2585 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-8.ec2.internal" event="NodeHasSufficientPID" Sep 10 13:28:55.603859 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:28:55.603831 2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\"" Sep 10 13:28:55.603859 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:28:55.603849 2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\"" Sep 10 13:28:55.604092 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:28:55.604078 2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\"" Sep 10 13:28:55.604165 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:28:55.604089 2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-bpkml\"" Sep 10 13:28:55.613067 ip-10-0-140-8 kubenswrapper[2585]: E0910 13:28:55.613047 2585 eviction_manager.go:292] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-140-8.ec2.internal\" not found" Sep 10 13:28:55.651902 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:28:55.651872 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/ad555c6a-4385-4f8e-93bb-a174cfaa8e74-etc-kubernetes\") pod \"multus-l9l8q\" (UID: \"ad555c6a-4385-4f8e-93bb-a174cfaa8e74\") " pod="openshift-multus/multus-l9l8q" Sep 10 13:28:55.651902 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:28:55.651900 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/e5fa823c-4cfc-48cf-ac04-865b3769417c-run\") pod \"tuned-s9hkb\" (UID: \"e5fa823c-4cfc-48cf-ac04-865b3769417c\") " pod="openshift-cluster-node-tuning-operator/tuned-s9hkb" Sep 10 13:28:55.652076 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:28:55.651917 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-72phb\" (UniqueName: \"kubernetes.io/projected/e5fa823c-4cfc-48cf-ac04-865b3769417c-kube-api-access-72phb\") pod \"tuned-s9hkb\" (UID: \"e5fa823c-4cfc-48cf-ac04-865b3769417c\") " pod="openshift-cluster-node-tuning-operator/tuned-s9hkb" Sep 10 13:28:55.652076 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:28:55.651933 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d552154f-1d5b-4c5d-99bf-7aeef22ce032-metrics-certs\") pod \"network-metrics-daemon-xp5dk\" (UID: \"d552154f-1d5b-4c5d-99bf-7aeef22ce032\") " pod="openshift-multus/network-metrics-daemon-xp5dk" Sep 10 13:28:55.652076 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:28:55.651948 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/fa53c783-d40a-4024-9f7b-1d0d2b57f6dc-konnectivity-ca\") pod \"konnectivity-agent-fxgmp\" (UID: \"fa53c783-d40a-4024-9f7b-1d0d2b57f6dc\") " pod="kube-system/konnectivity-agent-fxgmp" Sep 10 13:28:55.652076 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:28:55.651994 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/ad555c6a-4385-4f8e-93bb-a174cfaa8e74-host-run-netns\") pod \"multus-l9l8q\" (UID: \"ad555c6a-4385-4f8e-93bb-a174cfaa8e74\") " pod="openshift-multus/multus-l9l8q" Sep 10 13:28:55.652076 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:28:55.652020 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/ad555c6a-4385-4f8e-93bb-a174cfaa8e74-hostroot\") pod \"multus-l9l8q\" (UID: \"ad555c6a-4385-4f8e-93bb-a174cfaa8e74\") " pod="openshift-multus/multus-l9l8q" Sep 10 13:28:55.652076 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:28:55.652039 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/e5fa823c-4cfc-48cf-ac04-865b3769417c-etc-sysctl-d\") pod \"tuned-s9hkb\" (UID: \"e5fa823c-4cfc-48cf-ac04-865b3769417c\") " pod="openshift-cluster-node-tuning-operator/tuned-s9hkb" Sep 10 13:28:55.652076 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:28:55.652058 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/e5fa823c-4cfc-48cf-ac04-865b3769417c-etc-systemd\") pod \"tuned-s9hkb\" (UID: \"e5fa823c-4cfc-48cf-ac04-865b3769417c\") " pod="openshift-cluster-node-tuning-operator/tuned-s9hkb" Sep 10 13:28:55.652076 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:28:55.652079 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/e5fa823c-4cfc-48cf-ac04-865b3769417c-lib-modules\") pod \"tuned-s9hkb\" (UID: \"e5fa823c-4cfc-48cf-ac04-865b3769417c\") " pod="openshift-cluster-node-tuning-operator/tuned-s9hkb" Sep 10 13:28:55.652352 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:28:55.652095 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-22z4g\" (UniqueName: \"kubernetes.io/projected/d552154f-1d5b-4c5d-99bf-7aeef22ce032-kube-api-access-22z4g\") pod \"network-metrics-daemon-xp5dk\" (UID: \"d552154f-1d5b-4c5d-99bf-7aeef22ce032\") " pod="openshift-multus/network-metrics-daemon-xp5dk" Sep 10 13:28:55.652352 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:28:55.652124 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/ad555c6a-4385-4f8e-93bb-a174cfaa8e74-host-run-k8s-cni-cncf-io\") pod \"multus-l9l8q\" (UID: \"ad555c6a-4385-4f8e-93bb-a174cfaa8e74\") " pod="openshift-multus/multus-l9l8q" Sep 10 13:28:55.652352 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:28:55.652144 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/ad555c6a-4385-4f8e-93bb-a174cfaa8e74-multus-daemon-config\") pod \"multus-l9l8q\" (UID: \"ad555c6a-4385-4f8e-93bb-a174cfaa8e74\") " pod="openshift-multus/multus-l9l8q" Sep 10 13:28:55.652352 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:28:55.652159 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/1748fe6b-cce2-42ce-b8a7-10d12ffef95a-host\") pod \"node-ca-zhn9x\" (UID: \"1748fe6b-cce2-42ce-b8a7-10d12ffef95a\") " pod="openshift-image-registry/node-ca-zhn9x" Sep 10 13:28:55.652352 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:28:55.652172 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/e5fa823c-4cfc-48cf-ac04-865b3769417c-etc-sysconfig\") pod \"tuned-s9hkb\" (UID: \"e5fa823c-4cfc-48cf-ac04-865b3769417c\") " pod="openshift-cluster-node-tuning-operator/tuned-s9hkb" Sep 10 13:28:55.652352 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:28:55.652186 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/ad555c6a-4385-4f8e-93bb-a174cfaa8e74-multus-cni-dir\") pod \"multus-l9l8q\" (UID: \"ad555c6a-4385-4f8e-93bb-a174cfaa8e74\") " pod="openshift-multus/multus-l9l8q" Sep 10 13:28:55.652352 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:28:55.652221 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/e5fa823c-4cfc-48cf-ac04-865b3769417c-etc-modprobe-d\") pod \"tuned-s9hkb\" (UID: \"e5fa823c-4cfc-48cf-ac04-865b3769417c\") " pod="openshift-cluster-node-tuning-operator/tuned-s9hkb" Sep 10 13:28:55.652352 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:28:55.652239 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/e5fa823c-4cfc-48cf-ac04-865b3769417c-tmp\") pod \"tuned-s9hkb\" (UID: \"e5fa823c-4cfc-48cf-ac04-865b3769417c\") " pod="openshift-cluster-node-tuning-operator/tuned-s9hkb" Sep 10 13:28:55.652352 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:28:55.652279 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-x2vv5\" (UniqueName: \"kubernetes.io/projected/082a7577-a14f-49fb-a9b1-3cf250e6735a-kube-api-access-x2vv5\") pod \"network-check-target-7262t\" (UID: \"082a7577-a14f-49fb-a9b1-3cf250e6735a\") " pod="openshift-network-diagnostics/network-check-target-7262t" Sep 10 13:28:55.652352 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:28:55.652295 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/ad555c6a-4385-4f8e-93bb-a174cfaa8e74-multus-conf-dir\") pod \"multus-l9l8q\" (UID: \"ad555c6a-4385-4f8e-93bb-a174cfaa8e74\") " pod="openshift-multus/multus-l9l8q" Sep 10 13:28:55.652352 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:28:55.652310 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/fa53c783-d40a-4024-9f7b-1d0d2b57f6dc-agent-certs\") pod \"konnectivity-agent-fxgmp\" (UID: \"fa53c783-d40a-4024-9f7b-1d0d2b57f6dc\") " pod="kube-system/konnectivity-agent-fxgmp" Sep 10 13:28:55.652352 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:28:55.652323 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/ad555c6a-4385-4f8e-93bb-a174cfaa8e74-system-cni-dir\") pod \"multus-l9l8q\" (UID: \"ad555c6a-4385-4f8e-93bb-a174cfaa8e74\") " pod="openshift-multus/multus-l9l8q" Sep 10 13:28:55.652352 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:28:55.652336 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/ad555c6a-4385-4f8e-93bb-a174cfaa8e74-cni-binary-copy\") pod \"multus-l9l8q\" (UID: \"ad555c6a-4385-4f8e-93bb-a174cfaa8e74\") " pod="openshift-multus/multus-l9l8q" Sep 10 13:28:55.652352 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:28:55.652349 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/ad555c6a-4385-4f8e-93bb-a174cfaa8e74-host-var-lib-kubelet\") pod \"multus-l9l8q\" (UID: \"ad555c6a-4385-4f8e-93bb-a174cfaa8e74\") " pod="openshift-multus/multus-l9l8q" Sep 10 13:28:55.652872 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:28:55.652367 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/e5fa823c-4cfc-48cf-ac04-865b3769417c-etc-kubernetes\") pod \"tuned-s9hkb\" (UID: \"e5fa823c-4cfc-48cf-ac04-865b3769417c\") " pod="openshift-cluster-node-tuning-operator/tuned-s9hkb" Sep 10 13:28:55.652872 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:28:55.652388 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/e5fa823c-4cfc-48cf-ac04-865b3769417c-etc-tuned\") pod \"tuned-s9hkb\" (UID: \"e5fa823c-4cfc-48cf-ac04-865b3769417c\") " pod="openshift-cluster-node-tuning-operator/tuned-s9hkb" Sep 10 13:28:55.652872 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:28:55.652440 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/ad555c6a-4385-4f8e-93bb-a174cfaa8e74-host-var-lib-cni-bin\") pod \"multus-l9l8q\" (UID: \"ad555c6a-4385-4f8e-93bb-a174cfaa8e74\") " pod="openshift-multus/multus-l9l8q" Sep 10 13:28:55.652872 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:28:55.652473 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/ad555c6a-4385-4f8e-93bb-a174cfaa8e74-host-run-multus-certs\") pod \"multus-l9l8q\" (UID: \"ad555c6a-4385-4f8e-93bb-a174cfaa8e74\") " pod="openshift-multus/multus-l9l8q" Sep 10 13:28:55.652872 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:28:55.652497 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/e5fa823c-4cfc-48cf-ac04-865b3769417c-etc-sysctl-conf\") pod \"tuned-s9hkb\" (UID: \"e5fa823c-4cfc-48cf-ac04-865b3769417c\") " pod="openshift-cluster-node-tuning-operator/tuned-s9hkb" Sep 10 13:28:55.652872 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:28:55.652523 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/ad555c6a-4385-4f8e-93bb-a174cfaa8e74-host-var-lib-cni-multus\") pod \"multus-l9l8q\" (UID: \"ad555c6a-4385-4f8e-93bb-a174cfaa8e74\") " pod="openshift-multus/multus-l9l8q" Sep 10 13:28:55.652872 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:28:55.652546 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kd4fk\" (UniqueName: \"kubernetes.io/projected/ad555c6a-4385-4f8e-93bb-a174cfaa8e74-kube-api-access-kd4fk\") pod \"multus-l9l8q\" (UID: \"ad555c6a-4385-4f8e-93bb-a174cfaa8e74\") " pod="openshift-multus/multus-l9l8q" Sep 10 13:28:55.652872 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:28:55.652571 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/1748fe6b-cce2-42ce-b8a7-10d12ffef95a-serviceca\") pod \"node-ca-zhn9x\" (UID: \"1748fe6b-cce2-42ce-b8a7-10d12ffef95a\") " pod="openshift-image-registry/node-ca-zhn9x" Sep 10 13:28:55.652872 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:28:55.652593 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bz2b8\" (UniqueName: \"kubernetes.io/projected/1748fe6b-cce2-42ce-b8a7-10d12ffef95a-kube-api-access-bz2b8\") pod \"node-ca-zhn9x\" (UID: \"1748fe6b-cce2-42ce-b8a7-10d12ffef95a\") " pod="openshift-image-registry/node-ca-zhn9x" Sep 10 13:28:55.652872 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:28:55.652614 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/e5fa823c-4cfc-48cf-ac04-865b3769417c-sys\") pod \"tuned-s9hkb\" (UID: \"e5fa823c-4cfc-48cf-ac04-865b3769417c\") " pod="openshift-cluster-node-tuning-operator/tuned-s9hkb" Sep 10 13:28:55.652872 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:28:55.652635 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/e5fa823c-4cfc-48cf-ac04-865b3769417c-var-lib-kubelet\") pod \"tuned-s9hkb\" (UID: \"e5fa823c-4cfc-48cf-ac04-865b3769417c\") " pod="openshift-cluster-node-tuning-operator/tuned-s9hkb" Sep 10 13:28:55.652872 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:28:55.652665 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/e5fa823c-4cfc-48cf-ac04-865b3769417c-host\") pod \"tuned-s9hkb\" (UID: \"e5fa823c-4cfc-48cf-ac04-865b3769417c\") " pod="openshift-cluster-node-tuning-operator/tuned-s9hkb" Sep 10 13:28:55.652872 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:28:55.652693 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/ad555c6a-4385-4f8e-93bb-a174cfaa8e74-cnibin\") pod \"multus-l9l8q\" (UID: \"ad555c6a-4385-4f8e-93bb-a174cfaa8e74\") " pod="openshift-multus/multus-l9l8q" Sep 10 13:28:55.652872 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:28:55.652720 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/ad555c6a-4385-4f8e-93bb-a174cfaa8e74-multus-socket-dir-parent\") pod \"multus-l9l8q\" (UID: \"ad555c6a-4385-4f8e-93bb-a174cfaa8e74\") " pod="openshift-multus/multus-l9l8q" Sep 10 13:28:55.681407 ip-10-0-140-8 kubenswrapper[2585]: E0910 13:28:55.681377 2585 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-140-8.ec2.internal\" not found" Sep 10 13:28:55.753658 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:28:55.753631 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-72phb\" (UniqueName: \"kubernetes.io/projected/e5fa823c-4cfc-48cf-ac04-865b3769417c-kube-api-access-72phb\") pod \"tuned-s9hkb\" (UID: \"e5fa823c-4cfc-48cf-ac04-865b3769417c\") " pod="openshift-cluster-node-tuning-operator/tuned-s9hkb" Sep 10 13:28:55.753796 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:28:55.753673 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d552154f-1d5b-4c5d-99bf-7aeef22ce032-metrics-certs\") pod \"network-metrics-daemon-xp5dk\" (UID: \"d552154f-1d5b-4c5d-99bf-7aeef22ce032\") " pod="openshift-multus/network-metrics-daemon-xp5dk" Sep 10 13:28:55.753796 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:28:55.753691 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-t78qb\" (UniqueName: \"kubernetes.io/projected/9913a6bd-8e01-4034-9da7-f7d0b65a7eee-kube-api-access-t78qb\") pod \"multus-additional-cni-plugins-vlmmc\" (UID: \"9913a6bd-8e01-4034-9da7-f7d0b65a7eee\") " pod="openshift-multus/multus-additional-cni-plugins-vlmmc" Sep 10 13:28:55.753796 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:28:55.753743 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/ad555c6a-4385-4f8e-93bb-a174cfaa8e74-host-var-lib-cni-multus\") pod \"multus-l9l8q\" (UID: \"ad555c6a-4385-4f8e-93bb-a174cfaa8e74\") " pod="openshift-multus/multus-l9l8q" Sep 10 13:28:55.753796 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:28:55.753766 2585 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Sep 10 13:28:55.754093 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:28:55.753836 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/ad555c6a-4385-4f8e-93bb-a174cfaa8e74-host-var-lib-cni-multus\") pod \"multus-l9l8q\" (UID: \"ad555c6a-4385-4f8e-93bb-a174cfaa8e74\") " pod="openshift-multus/multus-l9l8q" Sep 10 13:28:55.754093 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:28:55.753840 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kd4fk\" (UniqueName: \"kubernetes.io/projected/ad555c6a-4385-4f8e-93bb-a174cfaa8e74-kube-api-access-kd4fk\") pod \"multus-l9l8q\" (UID: \"ad555c6a-4385-4f8e-93bb-a174cfaa8e74\") " pod="openshift-multus/multus-l9l8q" Sep 10 13:28:55.754093 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:28:55.753848 2585 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Sep 10 13:28:55.754093 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:28:55.753902 2585 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Sep 10 13:28:55.754093 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:28:55.753926 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/e5fa823c-4cfc-48cf-ac04-865b3769417c-sys\") pod \"tuned-s9hkb\" (UID: \"e5fa823c-4cfc-48cf-ac04-865b3769417c\") " pod="openshift-cluster-node-tuning-operator/tuned-s9hkb" Sep 10 13:28:55.754093 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:28:55.753956 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/2dac1f99-4132-43bd-8241-687afc22e3d5-ovnkube-script-lib\") pod \"ovnkube-node-5h2q6\" (UID: \"2dac1f99-4132-43bd-8241-687afc22e3d5\") " pod="openshift-ovn-kubernetes/ovnkube-node-5h2q6" Sep 10 13:28:55.754093 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:28:55.754036 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/e5fa823c-4cfc-48cf-ac04-865b3769417c-sys\") pod \"tuned-s9hkb\" (UID: \"e5fa823c-4cfc-48cf-ac04-865b3769417c\") " pod="openshift-cluster-node-tuning-operator/tuned-s9hkb" Sep 10 13:28:55.754093 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:28:55.754070 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/ad555c6a-4385-4f8e-93bb-a174cfaa8e74-host-run-netns\") pod \"multus-l9l8q\" (UID: \"ad555c6a-4385-4f8e-93bb-a174cfaa8e74\") " pod="openshift-multus/multus-l9l8q" Sep 10 13:28:55.754093 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:28:55.754092 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/ad555c6a-4385-4f8e-93bb-a174cfaa8e74-hostroot\") pod \"multus-l9l8q\" (UID: \"ad555c6a-4385-4f8e-93bb-a174cfaa8e74\") " pod="openshift-multus/multus-l9l8q" Sep 10 13:28:55.754531 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:28:55.754104 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/ad555c6a-4385-4f8e-93bb-a174cfaa8e74-host-run-netns\") pod \"multus-l9l8q\" (UID: \"ad555c6a-4385-4f8e-93bb-a174cfaa8e74\") " pod="openshift-multus/multus-l9l8q" Sep 10 13:28:55.754531 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:28:55.754108 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/e5fa823c-4cfc-48cf-ac04-865b3769417c-etc-systemd\") pod \"tuned-s9hkb\" (UID: \"e5fa823c-4cfc-48cf-ac04-865b3769417c\") " pod="openshift-cluster-node-tuning-operator/tuned-s9hkb" Sep 10 13:28:55.754531 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:28:55.754145 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/e5fa823c-4cfc-48cf-ac04-865b3769417c-lib-modules\") pod \"tuned-s9hkb\" (UID: \"e5fa823c-4cfc-48cf-ac04-865b3769417c\") " pod="openshift-cluster-node-tuning-operator/tuned-s9hkb" Sep 10 13:28:55.754531 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:28:55.754154 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/e5fa823c-4cfc-48cf-ac04-865b3769417c-etc-systemd\") pod \"tuned-s9hkb\" (UID: \"e5fa823c-4cfc-48cf-ac04-865b3769417c\") " pod="openshift-cluster-node-tuning-operator/tuned-s9hkb" Sep 10 13:28:55.754531 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:28:55.754161 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/ad555c6a-4385-4f8e-93bb-a174cfaa8e74-hostroot\") pod \"multus-l9l8q\" (UID: \"ad555c6a-4385-4f8e-93bb-a174cfaa8e74\") " pod="openshift-multus/multus-l9l8q" Sep 10 13:28:55.754531 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:28:55.754175 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/9913a6bd-8e01-4034-9da7-f7d0b65a7eee-system-cni-dir\") pod \"multus-additional-cni-plugins-vlmmc\" (UID: \"9913a6bd-8e01-4034-9da7-f7d0b65a7eee\") " pod="openshift-multus/multus-additional-cni-plugins-vlmmc" Sep 10 13:28:55.754531 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:28:55.754208 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/9913a6bd-8e01-4034-9da7-f7d0b65a7eee-cni-binary-copy\") pod \"multus-additional-cni-plugins-vlmmc\" (UID: \"9913a6bd-8e01-4034-9da7-f7d0b65a7eee\") " pod="openshift-multus/multus-additional-cni-plugins-vlmmc" Sep 10 13:28:55.754531 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:28:55.754236 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/9913a6bd-8e01-4034-9da7-f7d0b65a7eee-tuning-conf-dir\") pod \"multus-additional-cni-plugins-vlmmc\" (UID: \"9913a6bd-8e01-4034-9da7-f7d0b65a7eee\") " pod="openshift-multus/multus-additional-cni-plugins-vlmmc" Sep 10 13:28:55.754531 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:28:55.754284 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/ad555c6a-4385-4f8e-93bb-a174cfaa8e74-multus-daemon-config\") pod \"multus-l9l8q\" (UID: \"ad555c6a-4385-4f8e-93bb-a174cfaa8e74\") " pod="openshift-multus/multus-l9l8q" Sep 10 13:28:55.754531 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:28:55.754309 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/1748fe6b-cce2-42ce-b8a7-10d12ffef95a-host\") pod \"node-ca-zhn9x\" (UID: \"1748fe6b-cce2-42ce-b8a7-10d12ffef95a\") " pod="openshift-image-registry/node-ca-zhn9x" Sep 10 13:28:55.754531 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:28:55.754332 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/e5fa823c-4cfc-48cf-ac04-865b3769417c-etc-sysconfig\") pod \"tuned-s9hkb\" (UID: \"e5fa823c-4cfc-48cf-ac04-865b3769417c\") " pod="openshift-cluster-node-tuning-operator/tuned-s9hkb" Sep 10 13:28:55.754531 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:28:55.754358 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/2dac1f99-4132-43bd-8241-687afc22e3d5-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-5h2q6\" (UID: \"2dac1f99-4132-43bd-8241-687afc22e3d5\") " pod="openshift-ovn-kubernetes/ovnkube-node-5h2q6" Sep 10 13:28:55.754531 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:28:55.754362 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/e5fa823c-4cfc-48cf-ac04-865b3769417c-lib-modules\") pod \"tuned-s9hkb\" (UID: \"e5fa823c-4cfc-48cf-ac04-865b3769417c\") " pod="openshift-cluster-node-tuning-operator/tuned-s9hkb" Sep 10 13:28:55.754531 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:28:55.754407 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/e5fa823c-4cfc-48cf-ac04-865b3769417c-etc-sysconfig\") pod \"tuned-s9hkb\" (UID: \"e5fa823c-4cfc-48cf-ac04-865b3769417c\") " pod="openshift-cluster-node-tuning-operator/tuned-s9hkb" Sep 10 13:28:55.754531 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:28:55.754420 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/1748fe6b-cce2-42ce-b8a7-10d12ffef95a-host\") pod \"node-ca-zhn9x\" (UID: \"1748fe6b-cce2-42ce-b8a7-10d12ffef95a\") " pod="openshift-image-registry/node-ca-zhn9x" Sep 10 13:28:55.754531 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:28:55.754445 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/e5fa823c-4cfc-48cf-ac04-865b3769417c-etc-modprobe-d\") pod \"tuned-s9hkb\" (UID: \"e5fa823c-4cfc-48cf-ac04-865b3769417c\") " pod="openshift-cluster-node-tuning-operator/tuned-s9hkb" Sep 10 13:28:55.754531 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:28:55.754464 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/e5fa823c-4cfc-48cf-ac04-865b3769417c-tmp\") pod \"tuned-s9hkb\" (UID: \"e5fa823c-4cfc-48cf-ac04-865b3769417c\") " pod="openshift-cluster-node-tuning-operator/tuned-s9hkb" Sep 10 13:28:55.755302 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:28:55.754488 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/2dac1f99-4132-43bd-8241-687afc22e3d5-host-cni-bin\") pod \"ovnkube-node-5h2q6\" (UID: \"2dac1f99-4132-43bd-8241-687afc22e3d5\") " pod="openshift-ovn-kubernetes/ovnkube-node-5h2q6" Sep 10 13:28:55.755302 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:28:55.754512 2585 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Sep 10 13:28:55.755302 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:28:55.754513 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x2vv5\" (UniqueName: \"kubernetes.io/projected/082a7577-a14f-49fb-a9b1-3cf250e6735a-kube-api-access-x2vv5\") pod \"network-check-target-7262t\" (UID: \"082a7577-a14f-49fb-a9b1-3cf250e6735a\") " pod="openshift-network-diagnostics/network-check-target-7262t" Sep 10 13:28:55.755302 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:28:55.754560 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/e5fa823c-4cfc-48cf-ac04-865b3769417c-etc-modprobe-d\") pod \"tuned-s9hkb\" (UID: \"e5fa823c-4cfc-48cf-ac04-865b3769417c\") " pod="openshift-cluster-node-tuning-operator/tuned-s9hkb" Sep 10 13:28:55.755302 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:28:55.754575 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/ad555c6a-4385-4f8e-93bb-a174cfaa8e74-multus-conf-dir\") pod \"multus-l9l8q\" (UID: \"ad555c6a-4385-4f8e-93bb-a174cfaa8e74\") " pod="openshift-multus/multus-l9l8q" Sep 10 13:28:55.755302 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:28:55.754608 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/fa53c783-d40a-4024-9f7b-1d0d2b57f6dc-agent-certs\") pod \"konnectivity-agent-fxgmp\" (UID: \"fa53c783-d40a-4024-9f7b-1d0d2b57f6dc\") " pod="kube-system/konnectivity-agent-fxgmp" Sep 10 13:28:55.755302 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:28:55.754647 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/ad555c6a-4385-4f8e-93bb-a174cfaa8e74-multus-conf-dir\") pod \"multus-l9l8q\" (UID: \"ad555c6a-4385-4f8e-93bb-a174cfaa8e74\") " pod="openshift-multus/multus-l9l8q" Sep 10 13:28:55.755302 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:28:55.754657 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/ad555c6a-4385-4f8e-93bb-a174cfaa8e74-system-cni-dir\") pod \"multus-l9l8q\" (UID: \"ad555c6a-4385-4f8e-93bb-a174cfaa8e74\") " pod="openshift-multus/multus-l9l8q" Sep 10 13:28:55.755302 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:28:55.754664 2585 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Sep 10 13:28:55.755302 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:28:55.754680 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/ad555c6a-4385-4f8e-93bb-a174cfaa8e74-host-var-lib-kubelet\") pod \"multus-l9l8q\" (UID: \"ad555c6a-4385-4f8e-93bb-a174cfaa8e74\") " pod="openshift-multus/multus-l9l8q" Sep 10 13:28:55.755302 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:28:55.754722 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/e5fa823c-4cfc-48cf-ac04-865b3769417c-etc-kubernetes\") pod \"tuned-s9hkb\" (UID: \"e5fa823c-4cfc-48cf-ac04-865b3769417c\") " pod="openshift-cluster-node-tuning-operator/tuned-s9hkb" Sep 10 13:28:55.755302 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:28:55.754750 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/2dac1f99-4132-43bd-8241-687afc22e3d5-etc-openvswitch\") pod \"ovnkube-node-5h2q6\" (UID: \"2dac1f99-4132-43bd-8241-687afc22e3d5\") " pod="openshift-ovn-kubernetes/ovnkube-node-5h2q6" Sep 10 13:28:55.755302 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:28:55.754765 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/ad555c6a-4385-4f8e-93bb-a174cfaa8e74-system-cni-dir\") pod \"multus-l9l8q\" (UID: \"ad555c6a-4385-4f8e-93bb-a174cfaa8e74\") " pod="openshift-multus/multus-l9l8q" Sep 10 13:28:55.755302 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:28:55.754770 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/ad555c6a-4385-4f8e-93bb-a174cfaa8e74-host-var-lib-kubelet\") pod \"multus-l9l8q\" (UID: \"ad555c6a-4385-4f8e-93bb-a174cfaa8e74\") " pod="openshift-multus/multus-l9l8q" Sep 10 13:28:55.755302 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:28:55.754791 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/2dac1f99-4132-43bd-8241-687afc22e3d5-env-overrides\") pod \"ovnkube-node-5h2q6\" (UID: \"2dac1f99-4132-43bd-8241-687afc22e3d5\") " pod="openshift-ovn-kubernetes/ovnkube-node-5h2q6" Sep 10 13:28:55.755302 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:28:55.754802 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/e5fa823c-4cfc-48cf-ac04-865b3769417c-etc-kubernetes\") pod \"tuned-s9hkb\" (UID: \"e5fa823c-4cfc-48cf-ac04-865b3769417c\") " pod="openshift-cluster-node-tuning-operator/tuned-s9hkb" Sep 10 13:28:55.755302 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:28:55.754817 2585 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Sep 10 13:28:55.755302 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:28:55.754832 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/2dac1f99-4132-43bd-8241-687afc22e3d5-ovn-node-metrics-cert\") pod \"ovnkube-node-5h2q6\" (UID: \"2dac1f99-4132-43bd-8241-687afc22e3d5\") " pod="openshift-ovn-kubernetes/ovnkube-node-5h2q6" Sep 10 13:28:55.755302 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:28:55.754863 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/ad555c6a-4385-4f8e-93bb-a174cfaa8e74-host-var-lib-cni-bin\") pod \"multus-l9l8q\" (UID: \"ad555c6a-4385-4f8e-93bb-a174cfaa8e74\") " pod="openshift-multus/multus-l9l8q" Sep 10 13:28:55.756157 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:28:55.754889 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/2dac1f99-4132-43bd-8241-687afc22e3d5-systemd-units\") pod \"ovnkube-node-5h2q6\" (UID: \"2dac1f99-4132-43bd-8241-687afc22e3d5\") " pod="openshift-ovn-kubernetes/ovnkube-node-5h2q6" Sep 10 13:28:55.756157 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:28:55.754916 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/2dac1f99-4132-43bd-8241-687afc22e3d5-host-run-ovn-kubernetes\") pod \"ovnkube-node-5h2q6\" (UID: \"2dac1f99-4132-43bd-8241-687afc22e3d5\") " pod="openshift-ovn-kubernetes/ovnkube-node-5h2q6" Sep 10 13:28:55.756157 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:28:55.754944 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/2dac1f99-4132-43bd-8241-687afc22e3d5-host-cni-netd\") pod \"ovnkube-node-5h2q6\" (UID: \"2dac1f99-4132-43bd-8241-687afc22e3d5\") " pod="openshift-ovn-kubernetes/ovnkube-node-5h2q6" Sep 10 13:28:55.756157 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:28:55.754955 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/ad555c6a-4385-4f8e-93bb-a174cfaa8e74-host-var-lib-cni-bin\") pod \"multus-l9l8q\" (UID: \"ad555c6a-4385-4f8e-93bb-a174cfaa8e74\") " pod="openshift-multus/multus-l9l8q" Sep 10 13:28:55.756157 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:28:55.754969 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/9913a6bd-8e01-4034-9da7-f7d0b65a7eee-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-vlmmc\" (UID: \"9913a6bd-8e01-4034-9da7-f7d0b65a7eee\") " pod="openshift-multus/multus-additional-cni-plugins-vlmmc" Sep 10 13:28:55.756157 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:28:55.754983 2585 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-8.ec2.internal" event="NodeHasSufficientMemory" Sep 10 13:28:55.756157 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:28:55.754996 2585 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-8.ec2.internal" event="NodeHasSufficientMemory" Sep 10 13:28:55.756157 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:28:55.755004 2585 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-8.ec2.internal" event="NodeHasSufficientMemory" Sep 10 13:28:55.756157 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:28:55.755012 2585 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-8.ec2.internal" event="NodeHasNoDiskPressure" Sep 10 13:28:55.756157 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:28:55.755019 2585 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-8.ec2.internal" event="NodeHasNoDiskPressure" Sep 10 13:28:55.756157 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:28:55.755023 2585 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-8.ec2.internal" event="NodeHasNoDiskPressure" Sep 10 13:28:55.756157 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:28:55.755029 2585 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-8.ec2.internal" event="NodeHasSufficientPID" Sep 10 13:28:55.756157 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:28:55.755034 2585 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-8.ec2.internal" event="NodeHasSufficientPID" Sep 10 13:28:55.756157 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:28:55.755038 2585 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-8.ec2.internal" event="NodeHasSufficientPID" Sep 10 13:28:55.756157 ip-10-0-140-8 kubenswrapper[2585]: E0910 13:28:55.755136 2585 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Sep 10 13:28:55.756157 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:28:55.755012 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/2dac1f99-4132-43bd-8241-687afc22e3d5-host-slash\") pod \"ovnkube-node-5h2q6\" (UID: \"2dac1f99-4132-43bd-8241-687afc22e3d5\") " pod="openshift-ovn-kubernetes/ovnkube-node-5h2q6" Sep 10 13:28:55.756157 ip-10-0-140-8 kubenswrapper[2585]: E0910 13:28:55.755195 2585 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d552154f-1d5b-4c5d-99bf-7aeef22ce032-metrics-certs podName:d552154f-1d5b-4c5d-99bf-7aeef22ce032 nodeName:}" failed. No retries permitted until 2026-09-10 13:28:56.255170916 +0000 UTC m=+21.127875111 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/d552154f-1d5b-4c5d-99bf-7aeef22ce032-metrics-certs") pod "network-metrics-daemon-xp5dk" (UID: "d552154f-1d5b-4c5d-99bf-7aeef22ce032") : object "openshift-multus"/"metrics-daemon-secret" not registered Sep 10 13:28:55.756157 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:28:55.755223 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/1748fe6b-cce2-42ce-b8a7-10d12ffef95a-serviceca\") pod \"node-ca-zhn9x\" (UID: \"1748fe6b-cce2-42ce-b8a7-10d12ffef95a\") " pod="openshift-image-registry/node-ca-zhn9x" Sep 10 13:28:55.756157 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:28:55.755264 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bz2b8\" (UniqueName: \"kubernetes.io/projected/1748fe6b-cce2-42ce-b8a7-10d12ffef95a-kube-api-access-bz2b8\") pod \"node-ca-zhn9x\" (UID: \"1748fe6b-cce2-42ce-b8a7-10d12ffef95a\") " pod="openshift-image-registry/node-ca-zhn9x" Sep 10 13:28:55.756157 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:28:55.755291 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/e5fa823c-4cfc-48cf-ac04-865b3769417c-var-lib-kubelet\") pod \"tuned-s9hkb\" (UID: \"e5fa823c-4cfc-48cf-ac04-865b3769417c\") " pod="openshift-cluster-node-tuning-operator/tuned-s9hkb" Sep 10 13:28:55.756157 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:28:55.755310 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/e5fa823c-4cfc-48cf-ac04-865b3769417c-host\") pod \"tuned-s9hkb\" (UID: \"e5fa823c-4cfc-48cf-ac04-865b3769417c\") " pod="openshift-cluster-node-tuning-operator/tuned-s9hkb" Sep 10 13:28:55.756157 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:28:55.755331 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/ad555c6a-4385-4f8e-93bb-a174cfaa8e74-cnibin\") pod \"multus-l9l8q\" (UID: \"ad555c6a-4385-4f8e-93bb-a174cfaa8e74\") " pod="openshift-multus/multus-l9l8q" Sep 10 13:28:55.757081 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:28:55.755359 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/ad555c6a-4385-4f8e-93bb-a174cfaa8e74-multus-socket-dir-parent\") pod \"multus-l9l8q\" (UID: \"ad555c6a-4385-4f8e-93bb-a174cfaa8e74\") " pod="openshift-multus/multus-l9l8q" Sep 10 13:28:55.757081 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:28:55.755376 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/e5fa823c-4cfc-48cf-ac04-865b3769417c-host\") pod \"tuned-s9hkb\" (UID: \"e5fa823c-4cfc-48cf-ac04-865b3769417c\") " pod="openshift-cluster-node-tuning-operator/tuned-s9hkb" Sep 10 13:28:55.757081 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:28:55.755382 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/ad555c6a-4385-4f8e-93bb-a174cfaa8e74-etc-kubernetes\") pod \"multus-l9l8q\" (UID: \"ad555c6a-4385-4f8e-93bb-a174cfaa8e74\") " pod="openshift-multus/multus-l9l8q" Sep 10 13:28:55.757081 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:28:55.755406 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/e5fa823c-4cfc-48cf-ac04-865b3769417c-run\") pod \"tuned-s9hkb\" (UID: \"e5fa823c-4cfc-48cf-ac04-865b3769417c\") " pod="openshift-cluster-node-tuning-operator/tuned-s9hkb" Sep 10 13:28:55.757081 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:28:55.755415 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/e5fa823c-4cfc-48cf-ac04-865b3769417c-var-lib-kubelet\") pod \"tuned-s9hkb\" (UID: \"e5fa823c-4cfc-48cf-ac04-865b3769417c\") " pod="openshift-cluster-node-tuning-operator/tuned-s9hkb" Sep 10 13:28:55.757081 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:28:55.755422 2585 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Sep 10 13:28:55.757081 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:28:55.755436 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/9913a6bd-8e01-4034-9da7-f7d0b65a7eee-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-vlmmc\" (UID: \"9913a6bd-8e01-4034-9da7-f7d0b65a7eee\") " pod="openshift-multus/multus-additional-cni-plugins-vlmmc" Sep 10 13:28:55.757081 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:28:55.755447 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/ad555c6a-4385-4f8e-93bb-a174cfaa8e74-multus-socket-dir-parent\") pod \"multus-l9l8q\" (UID: \"ad555c6a-4385-4f8e-93bb-a174cfaa8e74\") " pod="openshift-multus/multus-l9l8q" Sep 10 13:28:55.757081 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:28:55.755459 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/ad555c6a-4385-4f8e-93bb-a174cfaa8e74-etc-kubernetes\") pod \"multus-l9l8q\" (UID: \"ad555c6a-4385-4f8e-93bb-a174cfaa8e74\") " pod="openshift-multus/multus-l9l8q" Sep 10 13:28:55.757081 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:28:55.755461 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/2dac1f99-4132-43bd-8241-687afc22e3d5-host-run-netns\") pod \"ovnkube-node-5h2q6\" (UID: \"2dac1f99-4132-43bd-8241-687afc22e3d5\") " pod="openshift-ovn-kubernetes/ovnkube-node-5h2q6" Sep 10 13:28:55.757081 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:28:55.755434 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/ad555c6a-4385-4f8e-93bb-a174cfaa8e74-cnibin\") pod \"multus-l9l8q\" (UID: \"ad555c6a-4385-4f8e-93bb-a174cfaa8e74\") " pod="openshift-multus/multus-l9l8q" Sep 10 13:28:55.757081 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:28:55.755476 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/e5fa823c-4cfc-48cf-ac04-865b3769417c-run\") pod \"tuned-s9hkb\" (UID: \"e5fa823c-4cfc-48cf-ac04-865b3769417c\") " pod="openshift-cluster-node-tuning-operator/tuned-s9hkb" Sep 10 13:28:55.757081 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:28:55.755503 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/2dac1f99-4132-43bd-8241-687afc22e3d5-log-socket\") pod \"ovnkube-node-5h2q6\" (UID: \"2dac1f99-4132-43bd-8241-687afc22e3d5\") " pod="openshift-ovn-kubernetes/ovnkube-node-5h2q6" Sep 10 13:28:55.757081 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:28:55.755534 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lvjqt\" (UniqueName: \"kubernetes.io/projected/2dac1f99-4132-43bd-8241-687afc22e3d5-kube-api-access-lvjqt\") pod \"ovnkube-node-5h2q6\" (UID: \"2dac1f99-4132-43bd-8241-687afc22e3d5\") " pod="openshift-ovn-kubernetes/ovnkube-node-5h2q6" Sep 10 13:28:55.757081 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:28:55.755561 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/fa53c783-d40a-4024-9f7b-1d0d2b57f6dc-konnectivity-ca\") pod \"konnectivity-agent-fxgmp\" (UID: \"fa53c783-d40a-4024-9f7b-1d0d2b57f6dc\") " pod="kube-system/konnectivity-agent-fxgmp" Sep 10 13:28:55.757081 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:28:55.755611 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/e5fa823c-4cfc-48cf-ac04-865b3769417c-etc-sysctl-d\") pod \"tuned-s9hkb\" (UID: \"e5fa823c-4cfc-48cf-ac04-865b3769417c\") " pod="openshift-cluster-node-tuning-operator/tuned-s9hkb" Sep 10 13:28:55.757081 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:28:55.755641 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-22z4g\" (UniqueName: \"kubernetes.io/projected/d552154f-1d5b-4c5d-99bf-7aeef22ce032-kube-api-access-22z4g\") pod \"network-metrics-daemon-xp5dk\" (UID: \"d552154f-1d5b-4c5d-99bf-7aeef22ce032\") " pod="openshift-multus/network-metrics-daemon-xp5dk" Sep 10 13:28:55.757081 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:28:55.755675 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/c366889d-cdc5-4a43-af92-806d84e99e45-iptables-alerter-script\") pod \"iptables-alerter-x2wmd\" (UID: \"c366889d-cdc5-4a43-af92-806d84e99e45\") " pod="openshift-network-operator/iptables-alerter-x2wmd" Sep 10 13:28:55.757687 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:28:55.755701 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/2dac1f99-4132-43bd-8241-687afc22e3d5-host-kubelet\") pod \"ovnkube-node-5h2q6\" (UID: \"2dac1f99-4132-43bd-8241-687afc22e3d5\") " pod="openshift-ovn-kubernetes/ovnkube-node-5h2q6" Sep 10 13:28:55.757687 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:28:55.755718 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/1748fe6b-cce2-42ce-b8a7-10d12ffef95a-serviceca\") pod \"node-ca-zhn9x\" (UID: \"1748fe6b-cce2-42ce-b8a7-10d12ffef95a\") " pod="openshift-image-registry/node-ca-zhn9x" Sep 10 13:28:55.757687 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:28:55.755724 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/2dac1f99-4132-43bd-8241-687afc22e3d5-var-lib-openvswitch\") pod \"ovnkube-node-5h2q6\" (UID: \"2dac1f99-4132-43bd-8241-687afc22e3d5\") " pod="openshift-ovn-kubernetes/ovnkube-node-5h2q6" Sep 10 13:28:55.757687 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:28:55.755750 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/ad555c6a-4385-4f8e-93bb-a174cfaa8e74-host-run-k8s-cni-cncf-io\") pod \"multus-l9l8q\" (UID: \"ad555c6a-4385-4f8e-93bb-a174cfaa8e74\") " pod="openshift-multus/multus-l9l8q" Sep 10 13:28:55.757687 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:28:55.755763 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/e5fa823c-4cfc-48cf-ac04-865b3769417c-etc-sysctl-d\") pod \"tuned-s9hkb\" (UID: \"e5fa823c-4cfc-48cf-ac04-865b3769417c\") " pod="openshift-cluster-node-tuning-operator/tuned-s9hkb" Sep 10 13:28:55.757687 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:28:55.755777 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/2dac1f99-4132-43bd-8241-687afc22e3d5-node-log\") pod \"ovnkube-node-5h2q6\" (UID: \"2dac1f99-4132-43bd-8241-687afc22e3d5\") " pod="openshift-ovn-kubernetes/ovnkube-node-5h2q6" Sep 10 13:28:55.757687 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:28:55.755802 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/ad555c6a-4385-4f8e-93bb-a174cfaa8e74-multus-cni-dir\") pod \"multus-l9l8q\" (UID: \"ad555c6a-4385-4f8e-93bb-a174cfaa8e74\") " pod="openshift-multus/multus-l9l8q" Sep 10 13:28:55.757687 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:28:55.755812 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/ad555c6a-4385-4f8e-93bb-a174cfaa8e74-host-run-k8s-cni-cncf-io\") pod \"multus-l9l8q\" (UID: \"ad555c6a-4385-4f8e-93bb-a174cfaa8e74\") " pod="openshift-multus/multus-l9l8q" Sep 10 13:28:55.757687 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:28:55.755835 2585 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Sep 10 13:28:55.757687 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:28:55.755837 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/2dac1f99-4132-43bd-8241-687afc22e3d5-run-openvswitch\") pod \"ovnkube-node-5h2q6\" (UID: \"2dac1f99-4132-43bd-8241-687afc22e3d5\") " pod="openshift-ovn-kubernetes/ovnkube-node-5h2q6" Sep 10 13:28:55.757687 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:28:55.755860 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/ad555c6a-4385-4f8e-93bb-a174cfaa8e74-multus-cni-dir\") pod \"multus-l9l8q\" (UID: \"ad555c6a-4385-4f8e-93bb-a174cfaa8e74\") " pod="openshift-multus/multus-l9l8q" Sep 10 13:28:55.757687 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:28:55.755876 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/c366889d-cdc5-4a43-af92-806d84e99e45-host-slash\") pod \"iptables-alerter-x2wmd\" (UID: \"c366889d-cdc5-4a43-af92-806d84e99e45\") " pod="openshift-network-operator/iptables-alerter-x2wmd" Sep 10 13:28:55.757687 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:28:55.755895 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/ad555c6a-4385-4f8e-93bb-a174cfaa8e74-cni-binary-copy\") pod \"multus-l9l8q\" (UID: \"ad555c6a-4385-4f8e-93bb-a174cfaa8e74\") " pod="openshift-multus/multus-l9l8q" Sep 10 13:28:55.757687 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:28:55.755910 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/e5fa823c-4cfc-48cf-ac04-865b3769417c-etc-tuned\") pod \"tuned-s9hkb\" (UID: \"e5fa823c-4cfc-48cf-ac04-865b3769417c\") " pod="openshift-cluster-node-tuning-operator/tuned-s9hkb" Sep 10 13:28:55.757687 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:28:55.755936 2585 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Sep 10 13:28:55.757687 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:28:55.755942 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wqwnw\" (UniqueName: \"kubernetes.io/projected/c366889d-cdc5-4a43-af92-806d84e99e45-kube-api-access-wqwnw\") pod \"iptables-alerter-x2wmd\" (UID: \"c366889d-cdc5-4a43-af92-806d84e99e45\") " pod="openshift-network-operator/iptables-alerter-x2wmd" Sep 10 13:28:55.757687 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:28:55.755954 2585 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-8.ec2.internal" event="NodeHasSufficientMemory" Sep 10 13:28:55.757687 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:28:55.755978 2585 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-8.ec2.internal" event="NodeHasNoDiskPressure" Sep 10 13:28:55.757687 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:28:55.755992 2585 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-8.ec2.internal" event="NodeHasSufficientPID" Sep 10 13:28:55.757687 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:28:55.755976 2585 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-8.ec2.internal" event="NodeHasSufficientMemory" Sep 10 13:28:55.757687 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:28:55.756048 2585 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-8.ec2.internal" event="NodeHasNoDiskPressure" Sep 10 13:28:55.757687 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:28:55.756063 2585 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-8.ec2.internal" event="NodeHasSufficientPID" Sep 10 13:28:55.758376 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:28:55.756089 2585 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-8.ec2.internal" event="NodeHasSufficientMemory" Sep 10 13:28:55.758376 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:28:55.756104 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/fa53c783-d40a-4024-9f7b-1d0d2b57f6dc-konnectivity-ca\") pod \"konnectivity-agent-fxgmp\" (UID: \"fa53c783-d40a-4024-9f7b-1d0d2b57f6dc\") " pod="kube-system/konnectivity-agent-fxgmp" Sep 10 13:28:55.758376 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:28:55.756110 2585 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-8.ec2.internal" event="NodeHasNoDiskPressure" Sep 10 13:28:55.758376 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:28:55.756121 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/ad555c6a-4385-4f8e-93bb-a174cfaa8e74-host-run-multus-certs\") pod \"multus-l9l8q\" (UID: \"ad555c6a-4385-4f8e-93bb-a174cfaa8e74\") " pod="openshift-multus/multus-l9l8q" Sep 10 13:28:55.758376 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:28:55.756138 2585 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-8.ec2.internal" event="NodeHasSufficientPID" Sep 10 13:28:55.758376 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:28:55.756404 2585 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-8.ec2.internal" event="NodeHasSufficientMemory" Sep 10 13:28:55.758376 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:28:55.756425 2585 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-8.ec2.internal" event="NodeHasNoDiskPressure" Sep 10 13:28:55.758376 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:28:55.756438 2585 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-8.ec2.internal" event="NodeHasSufficientPID" Sep 10 13:28:55.758376 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:28:55.756488 2585 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-8.ec2.internal" event="NodeHasSufficientMemory" Sep 10 13:28:55.758376 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:28:55.756512 2585 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-8.ec2.internal" event="NodeHasNoDiskPressure" Sep 10 13:28:55.758376 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:28:55.756513 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/ad555c6a-4385-4f8e-93bb-a174cfaa8e74-multus-daemon-config\") pod \"multus-l9l8q\" (UID: \"ad555c6a-4385-4f8e-93bb-a174cfaa8e74\") " pod="openshift-multus/multus-l9l8q" Sep 10 13:28:55.758376 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:28:55.756526 2585 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-8.ec2.internal" event="NodeHasSufficientPID" Sep 10 13:28:55.758376 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:28:55.756532 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/ad555c6a-4385-4f8e-93bb-a174cfaa8e74-cni-binary-copy\") pod \"multus-l9l8q\" (UID: \"ad555c6a-4385-4f8e-93bb-a174cfaa8e74\") " pod="openshift-multus/multus-l9l8q" Sep 10 13:28:55.758376 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:28:55.756565 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/ad555c6a-4385-4f8e-93bb-a174cfaa8e74-host-run-multus-certs\") pod \"multus-l9l8q\" (UID: \"ad555c6a-4385-4f8e-93bb-a174cfaa8e74\") " pod="openshift-multus/multus-l9l8q" Sep 10 13:28:55.758376 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:28:55.756608 2585 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-8.ec2.internal" event="NodeHasSufficientMemory" Sep 10 13:28:55.758376 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:28:55.756630 2585 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-8.ec2.internal" event="NodeHasNoDiskPressure" Sep 10 13:28:55.758376 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:28:55.756643 2585 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-8.ec2.internal" event="NodeHasSufficientPID" Sep 10 13:28:55.758376 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:28:55.756600 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/e5fa823c-4cfc-48cf-ac04-865b3769417c-etc-sysctl-conf\") pod \"tuned-s9hkb\" (UID: \"e5fa823c-4cfc-48cf-ac04-865b3769417c\") " pod="openshift-cluster-node-tuning-operator/tuned-s9hkb" Sep 10 13:28:55.758376 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:28:55.756745 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/9913a6bd-8e01-4034-9da7-f7d0b65a7eee-cnibin\") pod \"multus-additional-cni-plugins-vlmmc\" (UID: \"9913a6bd-8e01-4034-9da7-f7d0b65a7eee\") " pod="openshift-multus/multus-additional-cni-plugins-vlmmc" Sep 10 13:28:55.758376 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:28:55.756780 2585 swap_util.go:74] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory" Sep 10 13:28:55.758376 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:28:55.756808 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/e5fa823c-4cfc-48cf-ac04-865b3769417c-etc-sysctl-conf\") pod \"tuned-s9hkb\" (UID: \"e5fa823c-4cfc-48cf-ac04-865b3769417c\") " pod="openshift-cluster-node-tuning-operator/tuned-s9hkb" Sep 10 13:28:55.758376 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:28:55.756846 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/2dac1f99-4132-43bd-8241-687afc22e3d5-run-systemd\") pod \"ovnkube-node-5h2q6\" (UID: \"2dac1f99-4132-43bd-8241-687afc22e3d5\") " pod="openshift-ovn-kubernetes/ovnkube-node-5h2q6" Sep 10 13:28:55.758376 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:28:55.756876 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/2dac1f99-4132-43bd-8241-687afc22e3d5-run-ovn\") pod \"ovnkube-node-5h2q6\" (UID: \"2dac1f99-4132-43bd-8241-687afc22e3d5\") " pod="openshift-ovn-kubernetes/ovnkube-node-5h2q6" Sep 10 13:28:55.758376 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:28:55.756899 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/2dac1f99-4132-43bd-8241-687afc22e3d5-ovnkube-config\") pod \"ovnkube-node-5h2q6\" (UID: \"2dac1f99-4132-43bd-8241-687afc22e3d5\") " pod="openshift-ovn-kubernetes/ovnkube-node-5h2q6" Sep 10 13:28:55.760238 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:28:55.760219 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/e5fa823c-4cfc-48cf-ac04-865b3769417c-etc-tuned\") pod \"tuned-s9hkb\" (UID: \"e5fa823c-4cfc-48cf-ac04-865b3769417c\") " pod="openshift-cluster-node-tuning-operator/tuned-s9hkb" Sep 10 13:28:55.760323 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:28:55.760267 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/e5fa823c-4cfc-48cf-ac04-865b3769417c-tmp\") pod \"tuned-s9hkb\" (UID: \"e5fa823c-4cfc-48cf-ac04-865b3769417c\") " pod="openshift-cluster-node-tuning-operator/tuned-s9hkb" Sep 10 13:28:55.761545 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:28:55.761525 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/fa53c783-d40a-4024-9f7b-1d0d2b57f6dc-agent-certs\") pod \"konnectivity-agent-fxgmp\" (UID: \"fa53c783-d40a-4024-9f7b-1d0d2b57f6dc\") " pod="kube-system/konnectivity-agent-fxgmp" Sep 10 13:28:55.769289 ip-10-0-140-8 kubenswrapper[2585]: E0910 13:28:55.769271 2585 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Sep 10 13:28:55.769289 ip-10-0-140-8 kubenswrapper[2585]: E0910 13:28:55.769291 2585 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Sep 10 13:28:55.769405 ip-10-0-140-8 kubenswrapper[2585]: E0910 13:28:55.769300 2585 projected.go:196] Error preparing data for projected volume kube-api-access-x2vv5 for pod openshift-network-diagnostics/network-check-target-7262t: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Sep 10 13:28:55.769405 ip-10-0-140-8 kubenswrapper[2585]: E0910 13:28:55.769348 2585 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/082a7577-a14f-49fb-a9b1-3cf250e6735a-kube-api-access-x2vv5 podName:082a7577-a14f-49fb-a9b1-3cf250e6735a nodeName:}" failed. No retries permitted until 2026-09-10 13:28:56.269335415 +0000 UTC m=+21.142039613 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-api-access-x2vv5" (UniqueName: "kubernetes.io/projected/082a7577-a14f-49fb-a9b1-3cf250e6735a-kube-api-access-x2vv5") pod "network-check-target-7262t" (UID: "082a7577-a14f-49fb-a9b1-3cf250e6735a") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Sep 10 13:28:55.771368 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:28:55.771348 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-72phb\" (UniqueName: \"kubernetes.io/projected/e5fa823c-4cfc-48cf-ac04-865b3769417c-kube-api-access-72phb\") pod \"tuned-s9hkb\" (UID: \"e5fa823c-4cfc-48cf-ac04-865b3769417c\") " pod="openshift-cluster-node-tuning-operator/tuned-s9hkb" Sep 10 13:28:55.771967 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:28:55.771943 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-22z4g\" (UniqueName: \"kubernetes.io/projected/d552154f-1d5b-4c5d-99bf-7aeef22ce032-kube-api-access-22z4g\") pod \"network-metrics-daemon-xp5dk\" (UID: \"d552154f-1d5b-4c5d-99bf-7aeef22ce032\") " pod="openshift-multus/network-metrics-daemon-xp5dk" Sep 10 13:28:55.772047 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:28:55.771992 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-kd4fk\" (UniqueName: \"kubernetes.io/projected/ad555c6a-4385-4f8e-93bb-a174cfaa8e74-kube-api-access-kd4fk\") pod \"multus-l9l8q\" (UID: \"ad555c6a-4385-4f8e-93bb-a174cfaa8e74\") " pod="openshift-multus/multus-l9l8q" Sep 10 13:28:55.772103 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:28:55.772085 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-bz2b8\" (UniqueName: \"kubernetes.io/projected/1748fe6b-cce2-42ce-b8a7-10d12ffef95a-kube-api-access-bz2b8\") pod \"node-ca-zhn9x\" (UID: \"1748fe6b-cce2-42ce-b8a7-10d12ffef95a\") " pod="openshift-image-registry/node-ca-zhn9x" Sep 10 13:28:55.781777 ip-10-0-140-8 kubenswrapper[2585]: E0910 13:28:55.781760 2585 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-140-8.ec2.internal\" not found" Sep 10 13:28:55.809009 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:28:55.808974 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-fxgmp" Sep 10 13:28:55.837874 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:28:55.837821 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-l9l8q" Sep 10 13:28:55.842787 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:28:55.842773 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-zhn9x" Sep 10 13:28:55.845575 ip-10-0-140-8 kubenswrapper[2585]: W0910 13:28:55.845551 2585 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podad555c6a_4385_4f8e_93bb_a174cfaa8e74.slice/crio-1e277d736dc2f622dc21811ba1abf679156f9e45c5a2561759f4754b41a82dc6 WatchSource:0}: Error finding container 1e277d736dc2f622dc21811ba1abf679156f9e45c5a2561759f4754b41a82dc6: Status 404 returned error can't find the container with id 1e277d736dc2f622dc21811ba1abf679156f9e45c5a2561759f4754b41a82dc6 Sep 10 13:28:55.851448 ip-10-0-140-8 kubenswrapper[2585]: W0910 13:28:55.851299 2585 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1748fe6b_cce2_42ce_b8a7_10d12ffef95a.slice/crio-bfb2c5587e406a6e67ef73464bca555d3e703bf5b0ecd47daec021d7387cd875 WatchSource:0}: Error finding container bfb2c5587e406a6e67ef73464bca555d3e703bf5b0ecd47daec021d7387cd875: Status 404 returned error can't find the container with id bfb2c5587e406a6e67ef73464bca555d3e703bf5b0ecd47daec021d7387cd875 Sep 10 13:28:55.858079 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:28:55.858061 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/c366889d-cdc5-4a43-af92-806d84e99e45-iptables-alerter-script\") pod \"iptables-alerter-x2wmd\" (UID: \"c366889d-cdc5-4a43-af92-806d84e99e45\") " pod="openshift-network-operator/iptables-alerter-x2wmd" Sep 10 13:28:55.858153 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:28:55.858085 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/2dac1f99-4132-43bd-8241-687afc22e3d5-host-kubelet\") pod \"ovnkube-node-5h2q6\" (UID: \"2dac1f99-4132-43bd-8241-687afc22e3d5\") " pod="openshift-ovn-kubernetes/ovnkube-node-5h2q6" Sep 10 13:28:55.858153 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:28:55.858100 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/2dac1f99-4132-43bd-8241-687afc22e3d5-var-lib-openvswitch\") pod \"ovnkube-node-5h2q6\" (UID: \"2dac1f99-4132-43bd-8241-687afc22e3d5\") " pod="openshift-ovn-kubernetes/ovnkube-node-5h2q6" Sep 10 13:28:55.858153 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:28:55.858115 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/2dac1f99-4132-43bd-8241-687afc22e3d5-node-log\") pod \"ovnkube-node-5h2q6\" (UID: \"2dac1f99-4132-43bd-8241-687afc22e3d5\") " pod="openshift-ovn-kubernetes/ovnkube-node-5h2q6" Sep 10 13:28:55.858153 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:28:55.858131 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/2dac1f99-4132-43bd-8241-687afc22e3d5-run-openvswitch\") pod \"ovnkube-node-5h2q6\" (UID: \"2dac1f99-4132-43bd-8241-687afc22e3d5\") " pod="openshift-ovn-kubernetes/ovnkube-node-5h2q6" Sep 10 13:28:55.858153 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:28:55.858145 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/c366889d-cdc5-4a43-af92-806d84e99e45-host-slash\") pod \"iptables-alerter-x2wmd\" (UID: \"c366889d-cdc5-4a43-af92-806d84e99e45\") " pod="openshift-network-operator/iptables-alerter-x2wmd" Sep 10 13:28:55.858457 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:28:55.858161 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wqwnw\" (UniqueName: \"kubernetes.io/projected/c366889d-cdc5-4a43-af92-806d84e99e45-kube-api-access-wqwnw\") pod \"iptables-alerter-x2wmd\" (UID: \"c366889d-cdc5-4a43-af92-806d84e99e45\") " pod="openshift-network-operator/iptables-alerter-x2wmd" Sep 10 13:28:55.858457 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:28:55.858169 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/2dac1f99-4132-43bd-8241-687afc22e3d5-host-kubelet\") pod \"ovnkube-node-5h2q6\" (UID: \"2dac1f99-4132-43bd-8241-687afc22e3d5\") " pod="openshift-ovn-kubernetes/ovnkube-node-5h2q6" Sep 10 13:28:55.858457 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:28:55.858184 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/2dac1f99-4132-43bd-8241-687afc22e3d5-node-log\") pod \"ovnkube-node-5h2q6\" (UID: \"2dac1f99-4132-43bd-8241-687afc22e3d5\") " pod="openshift-ovn-kubernetes/ovnkube-node-5h2q6" Sep 10 13:28:55.858457 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:28:55.858184 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/2dac1f99-4132-43bd-8241-687afc22e3d5-run-openvswitch\") pod \"ovnkube-node-5h2q6\" (UID: \"2dac1f99-4132-43bd-8241-687afc22e3d5\") " pod="openshift-ovn-kubernetes/ovnkube-node-5h2q6" Sep 10 13:28:55.858457 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:28:55.858207 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/c366889d-cdc5-4a43-af92-806d84e99e45-host-slash\") pod \"iptables-alerter-x2wmd\" (UID: \"c366889d-cdc5-4a43-af92-806d84e99e45\") " pod="openshift-network-operator/iptables-alerter-x2wmd" Sep 10 13:28:55.858457 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:28:55.858221 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/2dac1f99-4132-43bd-8241-687afc22e3d5-var-lib-openvswitch\") pod \"ovnkube-node-5h2q6\" (UID: \"2dac1f99-4132-43bd-8241-687afc22e3d5\") " pod="openshift-ovn-kubernetes/ovnkube-node-5h2q6" Sep 10 13:28:55.858457 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:28:55.858231 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/9913a6bd-8e01-4034-9da7-f7d0b65a7eee-cnibin\") pod \"multus-additional-cni-plugins-vlmmc\" (UID: \"9913a6bd-8e01-4034-9da7-f7d0b65a7eee\") " pod="openshift-multus/multus-additional-cni-plugins-vlmmc" Sep 10 13:28:55.858457 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:28:55.858266 2585 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Sep 10 13:28:55.858457 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:28:55.858272 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/2dac1f99-4132-43bd-8241-687afc22e3d5-run-systemd\") pod \"ovnkube-node-5h2q6\" (UID: \"2dac1f99-4132-43bd-8241-687afc22e3d5\") " pod="openshift-ovn-kubernetes/ovnkube-node-5h2q6" Sep 10 13:28:55.858457 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:28:55.858283 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/9913a6bd-8e01-4034-9da7-f7d0b65a7eee-cnibin\") pod \"multus-additional-cni-plugins-vlmmc\" (UID: \"9913a6bd-8e01-4034-9da7-f7d0b65a7eee\") " pod="openshift-multus/multus-additional-cni-plugins-vlmmc" Sep 10 13:28:55.858457 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:28:55.858311 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/2dac1f99-4132-43bd-8241-687afc22e3d5-run-systemd\") pod \"ovnkube-node-5h2q6\" (UID: \"2dac1f99-4132-43bd-8241-687afc22e3d5\") " pod="openshift-ovn-kubernetes/ovnkube-node-5h2q6" Sep 10 13:28:55.858457 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:28:55.858319 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/2dac1f99-4132-43bd-8241-687afc22e3d5-run-ovn\") pod \"ovnkube-node-5h2q6\" (UID: \"2dac1f99-4132-43bd-8241-687afc22e3d5\") " pod="openshift-ovn-kubernetes/ovnkube-node-5h2q6" Sep 10 13:28:55.858457 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:28:55.858287 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/2dac1f99-4132-43bd-8241-687afc22e3d5-run-ovn\") pod \"ovnkube-node-5h2q6\" (UID: \"2dac1f99-4132-43bd-8241-687afc22e3d5\") " pod="openshift-ovn-kubernetes/ovnkube-node-5h2q6" Sep 10 13:28:55.858457 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:28:55.858349 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/2dac1f99-4132-43bd-8241-687afc22e3d5-ovnkube-config\") pod \"ovnkube-node-5h2q6\" (UID: \"2dac1f99-4132-43bd-8241-687afc22e3d5\") " pod="openshift-ovn-kubernetes/ovnkube-node-5h2q6" Sep 10 13:28:55.858457 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:28:55.858374 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t78qb\" (UniqueName: \"kubernetes.io/projected/9913a6bd-8e01-4034-9da7-f7d0b65a7eee-kube-api-access-t78qb\") pod \"multus-additional-cni-plugins-vlmmc\" (UID: \"9913a6bd-8e01-4034-9da7-f7d0b65a7eee\") " pod="openshift-multus/multus-additional-cni-plugins-vlmmc" Sep 10 13:28:55.858457 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:28:55.858392 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/2dac1f99-4132-43bd-8241-687afc22e3d5-ovnkube-script-lib\") pod \"ovnkube-node-5h2q6\" (UID: \"2dac1f99-4132-43bd-8241-687afc22e3d5\") " pod="openshift-ovn-kubernetes/ovnkube-node-5h2q6" Sep 10 13:28:55.858457 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:28:55.858409 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/9913a6bd-8e01-4034-9da7-f7d0b65a7eee-system-cni-dir\") pod \"multus-additional-cni-plugins-vlmmc\" (UID: \"9913a6bd-8e01-4034-9da7-f7d0b65a7eee\") " pod="openshift-multus/multus-additional-cni-plugins-vlmmc" Sep 10 13:28:55.858457 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:28:55.858423 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/9913a6bd-8e01-4034-9da7-f7d0b65a7eee-cni-binary-copy\") pod \"multus-additional-cni-plugins-vlmmc\" (UID: \"9913a6bd-8e01-4034-9da7-f7d0b65a7eee\") " pod="openshift-multus/multus-additional-cni-plugins-vlmmc" Sep 10 13:28:55.859206 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:28:55.858450 2585 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Sep 10 13:28:55.859206 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:28:55.858453 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/9913a6bd-8e01-4034-9da7-f7d0b65a7eee-tuning-conf-dir\") pod \"multus-additional-cni-plugins-vlmmc\" (UID: \"9913a6bd-8e01-4034-9da7-f7d0b65a7eee\") " pod="openshift-multus/multus-additional-cni-plugins-vlmmc" Sep 10 13:28:55.859206 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:28:55.858497 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/2dac1f99-4132-43bd-8241-687afc22e3d5-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-5h2q6\" (UID: \"2dac1f99-4132-43bd-8241-687afc22e3d5\") " pod="openshift-ovn-kubernetes/ovnkube-node-5h2q6" Sep 10 13:28:55.859206 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:28:55.858512 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/9913a6bd-8e01-4034-9da7-f7d0b65a7eee-system-cni-dir\") pod \"multus-additional-cni-plugins-vlmmc\" (UID: \"9913a6bd-8e01-4034-9da7-f7d0b65a7eee\") " pod="openshift-multus/multus-additional-cni-plugins-vlmmc" Sep 10 13:28:55.859206 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:28:55.858525 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/2dac1f99-4132-43bd-8241-687afc22e3d5-host-cni-bin\") pod \"ovnkube-node-5h2q6\" (UID: \"2dac1f99-4132-43bd-8241-687afc22e3d5\") " pod="openshift-ovn-kubernetes/ovnkube-node-5h2q6" Sep 10 13:28:55.859206 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:28:55.858568 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/2dac1f99-4132-43bd-8241-687afc22e3d5-etc-openvswitch\") pod \"ovnkube-node-5h2q6\" (UID: \"2dac1f99-4132-43bd-8241-687afc22e3d5\") " pod="openshift-ovn-kubernetes/ovnkube-node-5h2q6" Sep 10 13:28:55.859206 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:28:55.858598 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/2dac1f99-4132-43bd-8241-687afc22e3d5-env-overrides\") pod \"ovnkube-node-5h2q6\" (UID: \"2dac1f99-4132-43bd-8241-687afc22e3d5\") " pod="openshift-ovn-kubernetes/ovnkube-node-5h2q6" Sep 10 13:28:55.859206 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:28:55.858621 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/2dac1f99-4132-43bd-8241-687afc22e3d5-ovn-node-metrics-cert\") pod \"ovnkube-node-5h2q6\" (UID: \"2dac1f99-4132-43bd-8241-687afc22e3d5\") " pod="openshift-ovn-kubernetes/ovnkube-node-5h2q6" Sep 10 13:28:55.859206 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:28:55.858647 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/2dac1f99-4132-43bd-8241-687afc22e3d5-systemd-units\") pod \"ovnkube-node-5h2q6\" (UID: \"2dac1f99-4132-43bd-8241-687afc22e3d5\") " pod="openshift-ovn-kubernetes/ovnkube-node-5h2q6" Sep 10 13:28:55.859206 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:28:55.858672 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/2dac1f99-4132-43bd-8241-687afc22e3d5-host-run-ovn-kubernetes\") pod \"ovnkube-node-5h2q6\" (UID: \"2dac1f99-4132-43bd-8241-687afc22e3d5\") " pod="openshift-ovn-kubernetes/ovnkube-node-5h2q6" Sep 10 13:28:55.859206 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:28:55.858679 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/9913a6bd-8e01-4034-9da7-f7d0b65a7eee-tuning-conf-dir\") pod \"multus-additional-cni-plugins-vlmmc\" (UID: \"9913a6bd-8e01-4034-9da7-f7d0b65a7eee\") " pod="openshift-multus/multus-additional-cni-plugins-vlmmc" Sep 10 13:28:55.859206 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:28:55.858697 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/2dac1f99-4132-43bd-8241-687afc22e3d5-host-cni-netd\") pod \"ovnkube-node-5h2q6\" (UID: \"2dac1f99-4132-43bd-8241-687afc22e3d5\") " pod="openshift-ovn-kubernetes/ovnkube-node-5h2q6" Sep 10 13:28:55.859206 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:28:55.858717 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/c366889d-cdc5-4a43-af92-806d84e99e45-iptables-alerter-script\") pod \"iptables-alerter-x2wmd\" (UID: \"c366889d-cdc5-4a43-af92-806d84e99e45\") " pod="openshift-network-operator/iptables-alerter-x2wmd" Sep 10 13:28:55.859206 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:28:55.858724 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/9913a6bd-8e01-4034-9da7-f7d0b65a7eee-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-vlmmc\" (UID: \"9913a6bd-8e01-4034-9da7-f7d0b65a7eee\") " pod="openshift-multus/multus-additional-cni-plugins-vlmmc" Sep 10 13:28:55.859206 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:28:55.858741 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/2dac1f99-4132-43bd-8241-687afc22e3d5-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-5h2q6\" (UID: \"2dac1f99-4132-43bd-8241-687afc22e3d5\") " pod="openshift-ovn-kubernetes/ovnkube-node-5h2q6" Sep 10 13:28:55.859206 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:28:55.858748 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/2dac1f99-4132-43bd-8241-687afc22e3d5-host-slash\") pod \"ovnkube-node-5h2q6\" (UID: \"2dac1f99-4132-43bd-8241-687afc22e3d5\") " pod="openshift-ovn-kubernetes/ovnkube-node-5h2q6" Sep 10 13:28:55.859206 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:28:55.858785 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/9913a6bd-8e01-4034-9da7-f7d0b65a7eee-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-vlmmc\" (UID: \"9913a6bd-8e01-4034-9da7-f7d0b65a7eee\") " pod="openshift-multus/multus-additional-cni-plugins-vlmmc" Sep 10 13:28:55.859894 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:28:55.858809 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/2dac1f99-4132-43bd-8241-687afc22e3d5-host-run-netns\") pod \"ovnkube-node-5h2q6\" (UID: \"2dac1f99-4132-43bd-8241-687afc22e3d5\") " pod="openshift-ovn-kubernetes/ovnkube-node-5h2q6" Sep 10 13:28:55.859894 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:28:55.858834 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/2dac1f99-4132-43bd-8241-687afc22e3d5-log-socket\") pod \"ovnkube-node-5h2q6\" (UID: \"2dac1f99-4132-43bd-8241-687afc22e3d5\") " pod="openshift-ovn-kubernetes/ovnkube-node-5h2q6" Sep 10 13:28:55.859894 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:28:55.858859 2585 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Sep 10 13:28:55.859894 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:28:55.858915 2585 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Sep 10 13:28:55.859894 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:28:55.858859 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lvjqt\" (UniqueName: \"kubernetes.io/projected/2dac1f99-4132-43bd-8241-687afc22e3d5-kube-api-access-lvjqt\") pod \"ovnkube-node-5h2q6\" (UID: \"2dac1f99-4132-43bd-8241-687afc22e3d5\") " pod="openshift-ovn-kubernetes/ovnkube-node-5h2q6" Sep 10 13:28:55.859894 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:28:55.858980 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/2dac1f99-4132-43bd-8241-687afc22e3d5-ovnkube-config\") pod \"ovnkube-node-5h2q6\" (UID: \"2dac1f99-4132-43bd-8241-687afc22e3d5\") " pod="openshift-ovn-kubernetes/ovnkube-node-5h2q6" Sep 10 13:28:55.859894 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:28:55.858960 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/2dac1f99-4132-43bd-8241-687afc22e3d5-host-cni-netd\") pod \"ovnkube-node-5h2q6\" (UID: \"2dac1f99-4132-43bd-8241-687afc22e3d5\") " pod="openshift-ovn-kubernetes/ovnkube-node-5h2q6" Sep 10 13:28:55.859894 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:28:55.858987 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/2dac1f99-4132-43bd-8241-687afc22e3d5-etc-openvswitch\") pod \"ovnkube-node-5h2q6\" (UID: \"2dac1f99-4132-43bd-8241-687afc22e3d5\") " pod="openshift-ovn-kubernetes/ovnkube-node-5h2q6" Sep 10 13:28:55.859894 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:28:55.859006 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/9913a6bd-8e01-4034-9da7-f7d0b65a7eee-cni-binary-copy\") pod \"multus-additional-cni-plugins-vlmmc\" (UID: \"9913a6bd-8e01-4034-9da7-f7d0b65a7eee\") " pod="openshift-multus/multus-additional-cni-plugins-vlmmc" Sep 10 13:28:55.859894 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:28:55.858984 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/2dac1f99-4132-43bd-8241-687afc22e3d5-host-run-ovn-kubernetes\") pod \"ovnkube-node-5h2q6\" (UID: \"2dac1f99-4132-43bd-8241-687afc22e3d5\") " pod="openshift-ovn-kubernetes/ovnkube-node-5h2q6" Sep 10 13:28:55.859894 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:28:55.858787 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/2dac1f99-4132-43bd-8241-687afc22e3d5-host-cni-bin\") pod \"ovnkube-node-5h2q6\" (UID: \"2dac1f99-4132-43bd-8241-687afc22e3d5\") " pod="openshift-ovn-kubernetes/ovnkube-node-5h2q6" Sep 10 13:28:55.859894 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:28:55.859035 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/2dac1f99-4132-43bd-8241-687afc22e3d5-ovnkube-script-lib\") pod \"ovnkube-node-5h2q6\" (UID: \"2dac1f99-4132-43bd-8241-687afc22e3d5\") " pod="openshift-ovn-kubernetes/ovnkube-node-5h2q6" Sep 10 13:28:55.859894 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:28:55.859072 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/2dac1f99-4132-43bd-8241-687afc22e3d5-systemd-units\") pod \"ovnkube-node-5h2q6\" (UID: \"2dac1f99-4132-43bd-8241-687afc22e3d5\") " pod="openshift-ovn-kubernetes/ovnkube-node-5h2q6" Sep 10 13:28:55.859894 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:28:55.859144 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/2dac1f99-4132-43bd-8241-687afc22e3d5-host-slash\") pod \"ovnkube-node-5h2q6\" (UID: \"2dac1f99-4132-43bd-8241-687afc22e3d5\") " pod="openshift-ovn-kubernetes/ovnkube-node-5h2q6" Sep 10 13:28:55.859894 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:28:55.859187 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/2dac1f99-4132-43bd-8241-687afc22e3d5-host-run-netns\") pod \"ovnkube-node-5h2q6\" (UID: \"2dac1f99-4132-43bd-8241-687afc22e3d5\") " pod="openshift-ovn-kubernetes/ovnkube-node-5h2q6" Sep 10 13:28:55.859894 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:28:55.859224 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/2dac1f99-4132-43bd-8241-687afc22e3d5-log-socket\") pod \"ovnkube-node-5h2q6\" (UID: \"2dac1f99-4132-43bd-8241-687afc22e3d5\") " pod="openshift-ovn-kubernetes/ovnkube-node-5h2q6" Sep 10 13:28:55.859894 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:28:55.859294 2585 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-8.ec2.internal" event="NodeHasSufficientMemory" Sep 10 13:28:55.859894 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:28:55.859321 2585 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-8.ec2.internal" event="NodeHasNoDiskPressure" Sep 10 13:28:55.859894 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:28:55.859335 2585 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-8.ec2.internal" event="NodeHasSufficientPID" Sep 10 13:28:55.859894 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:28:55.859357 2585 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-8.ec2.internal" event="NodeHasSufficientMemory" Sep 10 13:28:55.859894 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:28:55.859375 2585 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-8.ec2.internal" event="NodeHasNoDiskPressure" Sep 10 13:28:55.859894 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:28:55.859388 2585 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-8.ec2.internal" event="NodeHasSufficientPID" Sep 10 13:28:55.860490 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:28:55.859580 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/9913a6bd-8e01-4034-9da7-f7d0b65a7eee-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-vlmmc\" (UID: \"9913a6bd-8e01-4034-9da7-f7d0b65a7eee\") " pod="openshift-multus/multus-additional-cni-plugins-vlmmc" Sep 10 13:28:55.860490 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:28:55.859583 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/9913a6bd-8e01-4034-9da7-f7d0b65a7eee-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-vlmmc\" (UID: \"9913a6bd-8e01-4034-9da7-f7d0b65a7eee\") " pod="openshift-multus/multus-additional-cni-plugins-vlmmc" Sep 10 13:28:55.860490 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:28:55.859625 2585 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-8.ec2.internal" event="NodeHasSufficientMemory" Sep 10 13:28:55.860490 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:28:55.859650 2585 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-8.ec2.internal" event="NodeHasNoDiskPressure" Sep 10 13:28:55.860490 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:28:55.859660 2585 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-8.ec2.internal" event="NodeHasSufficientMemory" Sep 10 13:28:55.860490 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:28:55.859681 2585 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-8.ec2.internal" event="NodeHasNoDiskPressure" Sep 10 13:28:55.860490 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:28:55.859691 2585 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-8.ec2.internal" event="NodeHasSufficientPID" Sep 10 13:28:55.860490 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:28:55.859665 2585 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-8.ec2.internal" event="NodeHasSufficientPID" Sep 10 13:28:55.860490 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:28:55.859746 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/2dac1f99-4132-43bd-8241-687afc22e3d5-env-overrides\") pod \"ovnkube-node-5h2q6\" (UID: \"2dac1f99-4132-43bd-8241-687afc22e3d5\") " pod="openshift-ovn-kubernetes/ovnkube-node-5h2q6" Sep 10 13:28:55.861656 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:28:55.861641 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/2dac1f99-4132-43bd-8241-687afc22e3d5-ovn-node-metrics-cert\") pod \"ovnkube-node-5h2q6\" (UID: \"2dac1f99-4132-43bd-8241-687afc22e3d5\") " pod="openshift-ovn-kubernetes/ovnkube-node-5h2q6" Sep 10 13:28:55.866725 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:28:55.866707 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-s9hkb" Sep 10 13:28:55.871585 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:28:55.871568 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-wqwnw\" (UniqueName: \"kubernetes.io/projected/c366889d-cdc5-4a43-af92-806d84e99e45-kube-api-access-wqwnw\") pod \"iptables-alerter-x2wmd\" (UID: \"c366889d-cdc5-4a43-af92-806d84e99e45\") " pod="openshift-network-operator/iptables-alerter-x2wmd" Sep 10 13:28:55.871893 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:28:55.871879 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-t78qb\" (UniqueName: \"kubernetes.io/projected/9913a6bd-8e01-4034-9da7-f7d0b65a7eee-kube-api-access-t78qb\") pod \"multus-additional-cni-plugins-vlmmc\" (UID: \"9913a6bd-8e01-4034-9da7-f7d0b65a7eee\") " pod="openshift-multus/multus-additional-cni-plugins-vlmmc" Sep 10 13:28:55.872200 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:28:55.872186 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-lvjqt\" (UniqueName: \"kubernetes.io/projected/2dac1f99-4132-43bd-8241-687afc22e3d5-kube-api-access-lvjqt\") pod \"ovnkube-node-5h2q6\" (UID: \"2dac1f99-4132-43bd-8241-687afc22e3d5\") " pod="openshift-ovn-kubernetes/ovnkube-node-5h2q6" Sep 10 13:28:55.875698 ip-10-0-140-8 kubenswrapper[2585]: W0910 13:28:55.875681 2585 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode5fa823c_4cfc_48cf_ac04_865b3769417c.slice/crio-3c620d83fa84a429777e9cfdd75e51e335cf632c3cbf15e57e275a381762b021 WatchSource:0}: Error finding container 3c620d83fa84a429777e9cfdd75e51e335cf632c3cbf15e57e275a381762b021: Status 404 returned error can't find the container with id 3c620d83fa84a429777e9cfdd75e51e335cf632c3cbf15e57e275a381762b021 Sep 10 13:28:55.882029 ip-10-0-140-8 kubenswrapper[2585]: E0910 13:28:55.882010 2585 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-140-8.ec2.internal\" not found" Sep 10 13:28:55.890200 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:28:55.890183 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-vlmmc" Sep 10 13:28:55.895154 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:28:55.895139 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-5h2q6" Sep 10 13:28:55.899375 ip-10-0-140-8 kubenswrapper[2585]: W0910 13:28:55.899349 2585 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9913a6bd_8e01_4034_9da7_f7d0b65a7eee.slice/crio-95fb6678c12e0647c302939f8396211b215644008ca7ef6ee8e661ad21f2f866 WatchSource:0}: Error finding container 95fb6678c12e0647c302939f8396211b215644008ca7ef6ee8e661ad21f2f866: Status 404 returned error can't find the container with id 95fb6678c12e0647c302939f8396211b215644008ca7ef6ee8e661ad21f2f866 Sep 10 13:28:55.903412 ip-10-0-140-8 kubenswrapper[2585]: W0910 13:28:55.903394 2585 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2dac1f99_4132_43bd_8241_687afc22e3d5.slice/crio-7ea5e1ed0662fddce1d31abbb204e24bd3e52206ae0f4efdbc675bf009af397c WatchSource:0}: Error finding container 7ea5e1ed0662fddce1d31abbb204e24bd3e52206ae0f4efdbc675bf009af397c: Status 404 returned error can't find the container with id 7ea5e1ed0662fddce1d31abbb204e24bd3e52206ae0f4efdbc675bf009af397c Sep 10 13:28:55.908276 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:28:55.908259 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-x2wmd" Sep 10 13:28:55.917714 ip-10-0-140-8 kubenswrapper[2585]: W0910 13:28:55.917691 2585 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc366889d_cdc5_4a43_af92_806d84e99e45.slice/crio-fb47e6a230aa634672539a418c083f3947dcd3fb0f5e6fb563de559ffc49b66f WatchSource:0}: Error finding container fb47e6a230aa634672539a418c083f3947dcd3fb0f5e6fb563de559ffc49b66f: Status 404 returned error can't find the container with id fb47e6a230aa634672539a418c083f3947dcd3fb0f5e6fb563de559ffc49b66f Sep 10 13:28:55.982487 ip-10-0-140-8 kubenswrapper[2585]: E0910 13:28:55.982453 2585 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-140-8.ec2.internal\" not found" Sep 10 13:28:56.083227 ip-10-0-140-8 kubenswrapper[2585]: E0910 13:28:56.083186 2585 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-140-8.ec2.internal\" not found" Sep 10 13:28:56.183844 ip-10-0-140-8 kubenswrapper[2585]: E0910 13:28:56.183741 2585 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-140-8.ec2.internal\" not found" Sep 10 13:28:56.262192 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:28:56.262163 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d552154f-1d5b-4c5d-99bf-7aeef22ce032-metrics-certs\") pod \"network-metrics-daemon-xp5dk\" (UID: \"d552154f-1d5b-4c5d-99bf-7aeef22ce032\") " pod="openshift-multus/network-metrics-daemon-xp5dk" Sep 10 13:28:56.262372 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:28:56.262315 2585 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Sep 10 13:28:56.263337 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:28:56.263321 2585 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-8.ec2.internal" event="NodeHasSufficientMemory" Sep 10 13:28:56.263407 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:28:56.263350 2585 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-8.ec2.internal" event="NodeHasNoDiskPressure" Sep 10 13:28:56.263407 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:28:56.263360 2585 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-8.ec2.internal" event="NodeHasSufficientPID" Sep 10 13:28:56.263483 ip-10-0-140-8 kubenswrapper[2585]: E0910 13:28:56.263471 2585 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Sep 10 13:28:56.263528 ip-10-0-140-8 kubenswrapper[2585]: E0910 13:28:56.263518 2585 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d552154f-1d5b-4c5d-99bf-7aeef22ce032-metrics-certs podName:d552154f-1d5b-4c5d-99bf-7aeef22ce032 nodeName:}" failed. No retries permitted until 2026-09-10 13:28:57.263503401 +0000 UTC m=+22.136207580 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/d552154f-1d5b-4c5d-99bf-7aeef22ce032-metrics-certs") pod "network-metrics-daemon-xp5dk" (UID: "d552154f-1d5b-4c5d-99bf-7aeef22ce032") : object "openshift-multus"/"metrics-daemon-secret" not registered Sep 10 13:28:56.283971 ip-10-0-140-8 kubenswrapper[2585]: E0910 13:28:56.283944 2585 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-140-8.ec2.internal\" not found" Sep 10 13:28:56.363544 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:28:56.363508 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x2vv5\" (UniqueName: \"kubernetes.io/projected/082a7577-a14f-49fb-a9b1-3cf250e6735a-kube-api-access-x2vv5\") pod \"network-check-target-7262t\" (UID: \"082a7577-a14f-49fb-a9b1-3cf250e6735a\") " pod="openshift-network-diagnostics/network-check-target-7262t" Sep 10 13:28:56.363705 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:28:56.363625 2585 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Sep 10 13:28:56.365600 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:28:56.365581 2585 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-8.ec2.internal" event="NodeHasSufficientMemory" Sep 10 13:28:56.365681 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:28:56.365614 2585 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-8.ec2.internal" event="NodeHasNoDiskPressure" Sep 10 13:28:56.365681 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:28:56.365628 2585 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-8.ec2.internal" event="NodeHasSufficientPID" Sep 10 13:28:56.365788 ip-10-0-140-8 kubenswrapper[2585]: E0910 13:28:56.365762 2585 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Sep 10 13:28:56.365788 ip-10-0-140-8 kubenswrapper[2585]: E0910 13:28:56.365784 2585 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Sep 10 13:28:56.365896 ip-10-0-140-8 kubenswrapper[2585]: E0910 13:28:56.365794 2585 projected.go:196] Error preparing data for projected volume kube-api-access-x2vv5 for pod openshift-network-diagnostics/network-check-target-7262t: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Sep 10 13:28:56.365896 ip-10-0-140-8 kubenswrapper[2585]: E0910 13:28:56.365846 2585 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/082a7577-a14f-49fb-a9b1-3cf250e6735a-kube-api-access-x2vv5 podName:082a7577-a14f-49fb-a9b1-3cf250e6735a nodeName:}" failed. No retries permitted until 2026-09-10 13:28:57.365828123 +0000 UTC m=+22.238532316 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-api-access-x2vv5" (UniqueName: "kubernetes.io/projected/082a7577-a14f-49fb-a9b1-3cf250e6735a-kube-api-access-x2vv5") pod "network-check-target-7262t" (UID: "082a7577-a14f-49fb-a9b1-3cf250e6735a") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Sep 10 13:28:56.384849 ip-10-0-140-8 kubenswrapper[2585]: E0910 13:28:56.384817 2585 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-140-8.ec2.internal\" not found" Sep 10 13:28:56.485454 ip-10-0-140-8 kubenswrapper[2585]: E0910 13:28:56.485373 2585 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-140-8.ec2.internal\" not found" Sep 10 13:28:56.585897 ip-10-0-140-8 kubenswrapper[2585]: E0910 13:28:56.585862 2585 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-140-8.ec2.internal\" not found" Sep 10 13:28:56.686270 ip-10-0-140-8 kubenswrapper[2585]: E0910 13:28:56.686224 2585 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-140-8.ec2.internal\" not found" Sep 10 13:28:56.709727 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:28:56.709652 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-5h2q6" event={"ID":"2dac1f99-4132-43bd-8241-687afc22e3d5","Type":"ContainerStarted","Data":"7ea5e1ed0662fddce1d31abbb204e24bd3e52206ae0f4efdbc675bf009af397c"} Sep 10 13:28:56.720817 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:28:56.720782 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-vlmmc" event={"ID":"9913a6bd-8e01-4034-9da7-f7d0b65a7eee","Type":"ContainerStarted","Data":"95fb6678c12e0647c302939f8396211b215644008ca7ef6ee8e661ad21f2f866"} Sep 10 13:28:56.729215 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:28:56.729185 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-s9hkb" event={"ID":"e5fa823c-4cfc-48cf-ac04-865b3769417c","Type":"ContainerStarted","Data":"3c620d83fa84a429777e9cfdd75e51e335cf632c3cbf15e57e275a381762b021"} Sep 10 13:28:56.735047 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:28:56.734618 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-zhn9x" event={"ID":"1748fe6b-cce2-42ce-b8a7-10d12ffef95a","Type":"ContainerStarted","Data":"bfb2c5587e406a6e67ef73464bca555d3e703bf5b0ecd47daec021d7387cd875"} Sep 10 13:28:56.737967 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:28:56.737901 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-x2wmd" event={"ID":"c366889d-cdc5-4a43-af92-806d84e99e45","Type":"ContainerStarted","Data":"fb47e6a230aa634672539a418c083f3947dcd3fb0f5e6fb563de559ffc49b66f"} Sep 10 13:28:56.741392 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:28:56.741366 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-l9l8q" event={"ID":"ad555c6a-4385-4f8e-93bb-a174cfaa8e74","Type":"ContainerStarted","Data":"1e277d736dc2f622dc21811ba1abf679156f9e45c5a2561759f4754b41a82dc6"} Sep 10 13:28:56.752593 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:28:56.752524 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-fxgmp" event={"ID":"fa53c783-d40a-4024-9f7b-1d0d2b57f6dc","Type":"ContainerStarted","Data":"3a29b84f76951f021c939ed2035b0d90080d058c7c26a3d378c5c5969ce38789"} Sep 10 13:28:56.786345 ip-10-0-140-8 kubenswrapper[2585]: E0910 13:28:56.786307 2585 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-140-8.ec2.internal\" not found" Sep 10 13:28:56.887409 ip-10-0-140-8 kubenswrapper[2585]: E0910 13:28:56.887376 2585 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-140-8.ec2.internal\" not found" Sep 10 13:28:56.987539 ip-10-0-140-8 kubenswrapper[2585]: E0910 13:28:56.987501 2585 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-140-8.ec2.internal\" not found" Sep 10 13:28:57.087657 ip-10-0-140-8 kubenswrapper[2585]: E0910 13:28:57.087624 2585 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-140-8.ec2.internal\" not found" Sep 10 13:28:57.188331 ip-10-0-140-8 kubenswrapper[2585]: E0910 13:28:57.188297 2585 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-140-8.ec2.internal\" not found" Sep 10 13:28:57.272948 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:28:57.272828 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d552154f-1d5b-4c5d-99bf-7aeef22ce032-metrics-certs\") pod \"network-metrics-daemon-xp5dk\" (UID: \"d552154f-1d5b-4c5d-99bf-7aeef22ce032\") " pod="openshift-multus/network-metrics-daemon-xp5dk" Sep 10 13:28:57.272948 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:28:57.272942 2585 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Sep 10 13:28:57.285317 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:28:57.285222 2585 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-8.ec2.internal" event="NodeHasSufficientMemory" Sep 10 13:28:57.285317 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:28:57.285271 2585 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-8.ec2.internal" event="NodeHasNoDiskPressure" Sep 10 13:28:57.285317 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:28:57.285288 2585 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-8.ec2.internal" event="NodeHasSufficientPID" Sep 10 13:28:57.285576 ip-10-0-140-8 kubenswrapper[2585]: E0910 13:28:57.285407 2585 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Sep 10 13:28:57.285576 ip-10-0-140-8 kubenswrapper[2585]: E0910 13:28:57.285461 2585 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d552154f-1d5b-4c5d-99bf-7aeef22ce032-metrics-certs podName:d552154f-1d5b-4c5d-99bf-7aeef22ce032 nodeName:}" failed. No retries permitted until 2026-09-10 13:28:59.285440045 +0000 UTC m=+24.158144228 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/d552154f-1d5b-4c5d-99bf-7aeef22ce032-metrics-certs") pod "network-metrics-daemon-xp5dk" (UID: "d552154f-1d5b-4c5d-99bf-7aeef22ce032") : object "openshift-multus"/"metrics-daemon-secret" not registered Sep 10 13:28:57.289066 ip-10-0-140-8 kubenswrapper[2585]: E0910 13:28:57.288994 2585 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-140-8.ec2.internal\" not found" Sep 10 13:28:57.374237 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:28:57.374148 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x2vv5\" (UniqueName: \"kubernetes.io/projected/082a7577-a14f-49fb-a9b1-3cf250e6735a-kube-api-access-x2vv5\") pod \"network-check-target-7262t\" (UID: \"082a7577-a14f-49fb-a9b1-3cf250e6735a\") " pod="openshift-network-diagnostics/network-check-target-7262t" Sep 10 13:28:57.374439 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:28:57.374302 2585 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Sep 10 13:28:57.378476 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:28:57.378448 2585 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-8.ec2.internal" event="NodeHasSufficientMemory" Sep 10 13:28:57.378609 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:28:57.378489 2585 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-8.ec2.internal" event="NodeHasNoDiskPressure" Sep 10 13:28:57.378609 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:28:57.378504 2585 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-8.ec2.internal" event="NodeHasSufficientPID" Sep 10 13:28:57.378710 ip-10-0-140-8 kubenswrapper[2585]: E0910 13:28:57.378646 2585 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Sep 10 13:28:57.378710 ip-10-0-140-8 kubenswrapper[2585]: E0910 13:28:57.378665 2585 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Sep 10 13:28:57.378710 ip-10-0-140-8 kubenswrapper[2585]: E0910 13:28:57.378677 2585 projected.go:196] Error preparing data for projected volume kube-api-access-x2vv5 for pod openshift-network-diagnostics/network-check-target-7262t: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Sep 10 13:28:57.378872 ip-10-0-140-8 kubenswrapper[2585]: E0910 13:28:57.378727 2585 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/082a7577-a14f-49fb-a9b1-3cf250e6735a-kube-api-access-x2vv5 podName:082a7577-a14f-49fb-a9b1-3cf250e6735a nodeName:}" failed. No retries permitted until 2026-09-10 13:28:59.378707921 +0000 UTC m=+24.251412101 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-api-access-x2vv5" (UniqueName: "kubernetes.io/projected/082a7577-a14f-49fb-a9b1-3cf250e6735a-kube-api-access-x2vv5") pod "network-check-target-7262t" (UID: "082a7577-a14f-49fb-a9b1-3cf250e6735a") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Sep 10 13:28:57.389319 ip-10-0-140-8 kubenswrapper[2585]: E0910 13:28:57.389291 2585 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-140-8.ec2.internal\" not found" Sep 10 13:28:57.490643 ip-10-0-140-8 kubenswrapper[2585]: E0910 13:28:57.490609 2585 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-140-8.ec2.internal\" not found" Sep 10 13:28:57.591197 ip-10-0-140-8 kubenswrapper[2585]: E0910 13:28:57.591164 2585 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-140-8.ec2.internal\" not found" Sep 10 13:28:57.643050 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:28:57.642955 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-7262t" Sep 10 13:28:57.643050 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:28:57.643002 2585 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Sep 10 13:28:57.643278 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:28:57.643209 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-xp5dk" Sep 10 13:28:57.643278 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:28:57.643234 2585 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Sep 10 13:28:57.647964 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:28:57.647941 2585 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-8.ec2.internal" event="NodeHasSufficientMemory" Sep 10 13:28:57.648089 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:28:57.647977 2585 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-8.ec2.internal" event="NodeHasNoDiskPressure" Sep 10 13:28:57.648089 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:28:57.647992 2585 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-8.ec2.internal" event="NodeHasSufficientPID" Sep 10 13:28:57.648230 ip-10-0-140-8 kubenswrapper[2585]: E0910 13:28:57.648205 2585 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-7262t" podUID="082a7577-a14f-49fb-a9b1-3cf250e6735a" Sep 10 13:28:57.650680 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:28:57.650660 2585 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-8.ec2.internal" event="NodeHasSufficientMemory" Sep 10 13:28:57.650798 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:28:57.650693 2585 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-8.ec2.internal" event="NodeHasNoDiskPressure" Sep 10 13:28:57.650798 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:28:57.650707 2585 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-8.ec2.internal" event="NodeHasSufficientPID" Sep 10 13:28:57.650899 ip-10-0-140-8 kubenswrapper[2585]: E0910 13:28:57.650875 2585 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-xp5dk" podUID="d552154f-1d5b-4c5d-99bf-7aeef22ce032" Sep 10 13:28:57.691982 ip-10-0-140-8 kubenswrapper[2585]: E0910 13:28:57.691924 2585 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-140-8.ec2.internal\" not found" Sep 10 13:28:57.792080 ip-10-0-140-8 kubenswrapper[2585]: E0910 13:28:57.792017 2585 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-140-8.ec2.internal\" not found" Sep 10 13:28:57.892441 ip-10-0-140-8 kubenswrapper[2585]: E0910 13:28:57.892393 2585 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-140-8.ec2.internal\" not found" Sep 10 13:28:57.995859 ip-10-0-140-8 kubenswrapper[2585]: E0910 13:28:57.995746 2585 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-140-8.ec2.internal\" not found" Sep 10 13:28:58.096540 ip-10-0-140-8 kubenswrapper[2585]: E0910 13:28:58.096452 2585 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-140-8.ec2.internal\" not found" Sep 10 13:28:58.197521 ip-10-0-140-8 kubenswrapper[2585]: E0910 13:28:58.197471 2585 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-140-8.ec2.internal\" not found" Sep 10 13:28:58.298576 ip-10-0-140-8 kubenswrapper[2585]: E0910 13:28:58.298481 2585 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-140-8.ec2.internal\" not found" Sep 10 13:28:58.399508 ip-10-0-140-8 kubenswrapper[2585]: E0910 13:28:58.399462 2585 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-140-8.ec2.internal\" not found" Sep 10 13:28:58.500195 ip-10-0-140-8 kubenswrapper[2585]: E0910 13:28:58.500156 2585 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-140-8.ec2.internal\" not found" Sep 10 13:28:58.600669 ip-10-0-140-8 kubenswrapper[2585]: E0910 13:28:58.600616 2585 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-140-8.ec2.internal\" not found" Sep 10 13:28:58.700969 ip-10-0-140-8 kubenswrapper[2585]: E0910 13:28:58.700918 2585 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-140-8.ec2.internal\" not found" Sep 10 13:28:58.801839 ip-10-0-140-8 kubenswrapper[2585]: E0910 13:28:58.801804 2585 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-140-8.ec2.internal\" not found" Sep 10 13:28:58.902541 ip-10-0-140-8 kubenswrapper[2585]: E0910 13:28:58.902442 2585 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-140-8.ec2.internal\" not found" Sep 10 13:28:59.002963 ip-10-0-140-8 kubenswrapper[2585]: E0910 13:28:59.002926 2585 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-140-8.ec2.internal\" not found" Sep 10 13:28:59.103664 ip-10-0-140-8 kubenswrapper[2585]: E0910 13:28:59.103631 2585 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-140-8.ec2.internal\" not found" Sep 10 13:28:59.204820 ip-10-0-140-8 kubenswrapper[2585]: E0910 13:28:59.204733 2585 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-140-8.ec2.internal\" not found" Sep 10 13:28:59.291540 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:28:59.291060 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d552154f-1d5b-4c5d-99bf-7aeef22ce032-metrics-certs\") pod \"network-metrics-daemon-xp5dk\" (UID: \"d552154f-1d5b-4c5d-99bf-7aeef22ce032\") " pod="openshift-multus/network-metrics-daemon-xp5dk" Sep 10 13:28:59.291540 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:28:59.291188 2585 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Sep 10 13:28:59.292380 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:28:59.292354 2585 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-8.ec2.internal" event="NodeHasSufficientMemory" Sep 10 13:28:59.292488 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:28:59.292392 2585 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-8.ec2.internal" event="NodeHasNoDiskPressure" Sep 10 13:28:59.292488 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:28:59.292407 2585 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-8.ec2.internal" event="NodeHasSufficientPID" Sep 10 13:28:59.292605 ip-10-0-140-8 kubenswrapper[2585]: E0910 13:28:59.292535 2585 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Sep 10 13:28:59.292605 ip-10-0-140-8 kubenswrapper[2585]: E0910 13:28:59.292588 2585 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d552154f-1d5b-4c5d-99bf-7aeef22ce032-metrics-certs podName:d552154f-1d5b-4c5d-99bf-7aeef22ce032 nodeName:}" failed. No retries permitted until 2026-09-10 13:29:03.292568708 +0000 UTC m=+28.165272902 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/d552154f-1d5b-4c5d-99bf-7aeef22ce032-metrics-certs") pod "network-metrics-daemon-xp5dk" (UID: "d552154f-1d5b-4c5d-99bf-7aeef22ce032") : object "openshift-multus"/"metrics-daemon-secret" not registered Sep 10 13:28:59.305858 ip-10-0-140-8 kubenswrapper[2585]: E0910 13:28:59.305805 2585 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-140-8.ec2.internal\" not found" Sep 10 13:28:59.391628 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:28:59.391583 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x2vv5\" (UniqueName: \"kubernetes.io/projected/082a7577-a14f-49fb-a9b1-3cf250e6735a-kube-api-access-x2vv5\") pod \"network-check-target-7262t\" (UID: \"082a7577-a14f-49fb-a9b1-3cf250e6735a\") " pod="openshift-network-diagnostics/network-check-target-7262t" Sep 10 13:28:59.391809 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:28:59.391793 2585 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Sep 10 13:28:59.393221 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:28:59.393173 2585 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-8.ec2.internal" event="NodeHasSufficientMemory" Sep 10 13:28:59.393335 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:28:59.393235 2585 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-8.ec2.internal" event="NodeHasNoDiskPressure" Sep 10 13:28:59.393335 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:28:59.393268 2585 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-8.ec2.internal" event="NodeHasSufficientPID" Sep 10 13:28:59.393430 ip-10-0-140-8 kubenswrapper[2585]: E0910 13:28:59.393396 2585 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Sep 10 13:28:59.393430 ip-10-0-140-8 kubenswrapper[2585]: E0910 13:28:59.393413 2585 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Sep 10 13:28:59.393430 ip-10-0-140-8 kubenswrapper[2585]: E0910 13:28:59.393424 2585 projected.go:196] Error preparing data for projected volume kube-api-access-x2vv5 for pod openshift-network-diagnostics/network-check-target-7262t: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Sep 10 13:28:59.393585 ip-10-0-140-8 kubenswrapper[2585]: E0910 13:28:59.393473 2585 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/082a7577-a14f-49fb-a9b1-3cf250e6735a-kube-api-access-x2vv5 podName:082a7577-a14f-49fb-a9b1-3cf250e6735a nodeName:}" failed. No retries permitted until 2026-09-10 13:29:03.393454468 +0000 UTC m=+28.266158660 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-api-access-x2vv5" (UniqueName: "kubernetes.io/projected/082a7577-a14f-49fb-a9b1-3cf250e6735a-kube-api-access-x2vv5") pod "network-check-target-7262t" (UID: "082a7577-a14f-49fb-a9b1-3cf250e6735a") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Sep 10 13:28:59.406164 ip-10-0-140-8 kubenswrapper[2585]: E0910 13:28:59.406139 2585 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-140-8.ec2.internal\" not found" Sep 10 13:28:59.506378 ip-10-0-140-8 kubenswrapper[2585]: E0910 13:28:59.506299 2585 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-140-8.ec2.internal\" not found" Sep 10 13:28:59.606672 ip-10-0-140-8 kubenswrapper[2585]: E0910 13:28:59.606622 2585 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-140-8.ec2.internal\" not found" Sep 10 13:28:59.641671 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:28:59.641354 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-7262t" Sep 10 13:28:59.641671 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:28:59.641399 2585 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Sep 10 13:28:59.642197 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:28:59.642041 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-xp5dk" Sep 10 13:28:59.642197 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:28:59.642080 2585 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Sep 10 13:28:59.644723 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:28:59.644329 2585 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-8.ec2.internal" event="NodeHasSufficientMemory" Sep 10 13:28:59.644723 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:28:59.644359 2585 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-8.ec2.internal" event="NodeHasNoDiskPressure" Sep 10 13:28:59.644723 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:28:59.644375 2585 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-8.ec2.internal" event="NodeHasSufficientPID" Sep 10 13:28:59.644723 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:28:59.644520 2585 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-8.ec2.internal" event="NodeHasSufficientMemory" Sep 10 13:28:59.644723 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:28:59.644537 2585 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-8.ec2.internal" event="NodeHasNoDiskPressure" Sep 10 13:28:59.644723 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:28:59.644550 2585 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-8.ec2.internal" event="NodeHasSufficientPID" Sep 10 13:28:59.644723 ip-10-0-140-8 kubenswrapper[2585]: E0910 13:28:59.644550 2585 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-7262t" podUID="082a7577-a14f-49fb-a9b1-3cf250e6735a" Sep 10 13:28:59.644723 ip-10-0-140-8 kubenswrapper[2585]: E0910 13:28:59.644679 2585 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-xp5dk" podUID="d552154f-1d5b-4c5d-99bf-7aeef22ce032" Sep 10 13:28:59.707337 ip-10-0-140-8 kubenswrapper[2585]: E0910 13:28:59.707296 2585 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-140-8.ec2.internal\" not found" Sep 10 13:28:59.808295 ip-10-0-140-8 kubenswrapper[2585]: E0910 13:28:59.808194 2585 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-140-8.ec2.internal\" not found" Sep 10 13:28:59.909128 ip-10-0-140-8 kubenswrapper[2585]: E0910 13:28:59.909078 2585 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-140-8.ec2.internal\" not found" Sep 10 13:29:00.009340 ip-10-0-140-8 kubenswrapper[2585]: E0910 13:29:00.009302 2585 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-140-8.ec2.internal\" not found" Sep 10 13:29:00.109882 ip-10-0-140-8 kubenswrapper[2585]: E0910 13:29:00.109816 2585 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-140-8.ec2.internal\" not found" Sep 10 13:29:00.210592 ip-10-0-140-8 kubenswrapper[2585]: E0910 13:29:00.210549 2585 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-140-8.ec2.internal\" not found" Sep 10 13:29:00.311270 ip-10-0-140-8 kubenswrapper[2585]: E0910 13:29:00.311218 2585 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-140-8.ec2.internal\" not found" Sep 10 13:29:00.412385 ip-10-0-140-8 kubenswrapper[2585]: E0910 13:29:00.412305 2585 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-140-8.ec2.internal\" not found" Sep 10 13:29:00.512662 ip-10-0-140-8 kubenswrapper[2585]: E0910 13:29:00.512627 2585 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-140-8.ec2.internal\" not found" Sep 10 13:29:00.582003 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:29:00.581956 2585 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ovn-kubernetes/ovnkube-node-5h2q6"] Sep 10 13:29:00.613571 ip-10-0-140-8 kubenswrapper[2585]: E0910 13:29:00.613514 2585 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-140-8.ec2.internal\" not found" Sep 10 13:29:00.714084 ip-10-0-140-8 kubenswrapper[2585]: E0910 13:29:00.714011 2585 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-140-8.ec2.internal\" not found" Sep 10 13:29:00.815078 ip-10-0-140-8 kubenswrapper[2585]: E0910 13:29:00.814992 2585 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-140-8.ec2.internal\" not found" Sep 10 13:29:00.916021 ip-10-0-140-8 kubenswrapper[2585]: E0910 13:29:00.915985 2585 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-140-8.ec2.internal\" not found" Sep 10 13:29:01.016704 ip-10-0-140-8 kubenswrapper[2585]: E0910 13:29:01.016623 2585 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-140-8.ec2.internal\" not found" Sep 10 13:29:01.117564 ip-10-0-140-8 kubenswrapper[2585]: E0910 13:29:01.117521 2585 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-140-8.ec2.internal\" not found" Sep 10 13:29:01.218204 ip-10-0-140-8 kubenswrapper[2585]: E0910 13:29:01.218172 2585 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-140-8.ec2.internal\" not found" Sep 10 13:29:01.318909 ip-10-0-140-8 kubenswrapper[2585]: E0910 13:29:01.318854 2585 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-140-8.ec2.internal\" not found" Sep 10 13:29:01.419677 ip-10-0-140-8 kubenswrapper[2585]: E0910 13:29:01.419644 2585 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-140-8.ec2.internal\" not found" Sep 10 13:29:01.520578 ip-10-0-140-8 kubenswrapper[2585]: E0910 13:29:01.520539 2585 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-140-8.ec2.internal\" not found" Sep 10 13:29:01.621743 ip-10-0-140-8 kubenswrapper[2585]: E0910 13:29:01.621661 2585 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-140-8.ec2.internal\" not found" Sep 10 13:29:01.641566 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:29:01.641534 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-xp5dk" Sep 10 13:29:01.641731 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:29:01.641605 2585 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Sep 10 13:29:01.641975 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:29:01.641953 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-7262t" Sep 10 13:29:01.642130 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:29:01.642006 2585 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Sep 10 13:29:01.643633 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:29:01.642981 2585 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-8.ec2.internal" event="NodeHasSufficientMemory" Sep 10 13:29:01.643633 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:29:01.643014 2585 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-8.ec2.internal" event="NodeHasNoDiskPressure" Sep 10 13:29:01.643633 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:29:01.643030 2585 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-8.ec2.internal" event="NodeHasSufficientPID" Sep 10 13:29:01.643633 ip-10-0-140-8 kubenswrapper[2585]: E0910 13:29:01.643266 2585 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-xp5dk" podUID="d552154f-1d5b-4c5d-99bf-7aeef22ce032" Sep 10 13:29:01.643633 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:29:01.643281 2585 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-8.ec2.internal" event="NodeHasSufficientMemory" Sep 10 13:29:01.643633 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:29:01.643301 2585 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-8.ec2.internal" event="NodeHasNoDiskPressure" Sep 10 13:29:01.643633 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:29:01.643313 2585 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-8.ec2.internal" event="NodeHasSufficientPID" Sep 10 13:29:01.643633 ip-10-0-140-8 kubenswrapper[2585]: E0910 13:29:01.643574 2585 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-7262t" podUID="082a7577-a14f-49fb-a9b1-3cf250e6735a" Sep 10 13:29:01.722635 ip-10-0-140-8 kubenswrapper[2585]: E0910 13:29:01.722599 2585 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-140-8.ec2.internal\" not found" Sep 10 13:29:01.822745 ip-10-0-140-8 kubenswrapper[2585]: E0910 13:29:01.822697 2585 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-140-8.ec2.internal\" not found" Sep 10 13:29:01.923688 ip-10-0-140-8 kubenswrapper[2585]: E0910 13:29:01.923609 2585 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-140-8.ec2.internal\" not found" Sep 10 13:29:02.024592 ip-10-0-140-8 kubenswrapper[2585]: E0910 13:29:02.024555 2585 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-140-8.ec2.internal\" not found" Sep 10 13:29:02.124952 ip-10-0-140-8 kubenswrapper[2585]: E0910 13:29:02.124922 2585 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-140-8.ec2.internal\" not found" Sep 10 13:29:02.225754 ip-10-0-140-8 kubenswrapper[2585]: E0910 13:29:02.225681 2585 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-140-8.ec2.internal\" not found" Sep 10 13:29:02.326460 ip-10-0-140-8 kubenswrapper[2585]: E0910 13:29:02.326419 2585 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-140-8.ec2.internal\" not found" Sep 10 13:29:02.427108 ip-10-0-140-8 kubenswrapper[2585]: E0910 13:29:02.427070 2585 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-140-8.ec2.internal\" not found" Sep 10 13:29:02.527954 ip-10-0-140-8 kubenswrapper[2585]: E0910 13:29:02.527873 2585 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-140-8.ec2.internal\" not found" Sep 10 13:29:02.629028 ip-10-0-140-8 kubenswrapper[2585]: E0910 13:29:02.628984 2585 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-140-8.ec2.internal\" not found" Sep 10 13:29:02.729783 ip-10-0-140-8 kubenswrapper[2585]: E0910 13:29:02.729744 2585 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-140-8.ec2.internal\" not found" Sep 10 13:29:02.830101 ip-10-0-140-8 kubenswrapper[2585]: E0910 13:29:02.830065 2585 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-140-8.ec2.internal\" not found" Sep 10 13:29:02.930837 ip-10-0-140-8 kubenswrapper[2585]: E0910 13:29:02.930797 2585 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-140-8.ec2.internal\" not found" Sep 10 13:29:03.031885 ip-10-0-140-8 kubenswrapper[2585]: E0910 13:29:03.031827 2585 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-140-8.ec2.internal\" not found" Sep 10 13:29:03.132981 ip-10-0-140-8 kubenswrapper[2585]: E0910 13:29:03.132901 2585 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-140-8.ec2.internal\" not found" Sep 10 13:29:03.233075 ip-10-0-140-8 kubenswrapper[2585]: E0910 13:29:03.233042 2585 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-140-8.ec2.internal\" not found" Sep 10 13:29:03.323093 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:29:03.322736 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d552154f-1d5b-4c5d-99bf-7aeef22ce032-metrics-certs\") pod \"network-metrics-daemon-xp5dk\" (UID: \"d552154f-1d5b-4c5d-99bf-7aeef22ce032\") " pod="openshift-multus/network-metrics-daemon-xp5dk" Sep 10 13:29:03.323093 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:29:03.322871 2585 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Sep 10 13:29:03.324360 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:29:03.324335 2585 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-8.ec2.internal" event="NodeHasSufficientMemory" Sep 10 13:29:03.324476 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:29:03.324378 2585 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-8.ec2.internal" event="NodeHasNoDiskPressure" Sep 10 13:29:03.324476 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:29:03.324394 2585 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-8.ec2.internal" event="NodeHasSufficientPID" Sep 10 13:29:03.324575 ip-10-0-140-8 kubenswrapper[2585]: E0910 13:29:03.324526 2585 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Sep 10 13:29:03.324626 ip-10-0-140-8 kubenswrapper[2585]: E0910 13:29:03.324580 2585 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d552154f-1d5b-4c5d-99bf-7aeef22ce032-metrics-certs podName:d552154f-1d5b-4c5d-99bf-7aeef22ce032 nodeName:}" failed. No retries permitted until 2026-09-10 13:29:11.324560746 +0000 UTC m=+36.197264928 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/d552154f-1d5b-4c5d-99bf-7aeef22ce032-metrics-certs") pod "network-metrics-daemon-xp5dk" (UID: "d552154f-1d5b-4c5d-99bf-7aeef22ce032") : object "openshift-multus"/"metrics-daemon-secret" not registered Sep 10 13:29:03.333601 ip-10-0-140-8 kubenswrapper[2585]: E0910 13:29:03.333577 2585 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-140-8.ec2.internal\" not found" Sep 10 13:29:03.423513 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:29:03.423442 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x2vv5\" (UniqueName: \"kubernetes.io/projected/082a7577-a14f-49fb-a9b1-3cf250e6735a-kube-api-access-x2vv5\") pod \"network-check-target-7262t\" (UID: \"082a7577-a14f-49fb-a9b1-3cf250e6735a\") " pod="openshift-network-diagnostics/network-check-target-7262t" Sep 10 13:29:03.423644 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:29:03.423556 2585 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Sep 10 13:29:03.424713 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:29:03.424693 2585 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-8.ec2.internal" event="NodeHasSufficientMemory" Sep 10 13:29:03.424794 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:29:03.424729 2585 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-8.ec2.internal" event="NodeHasNoDiskPressure" Sep 10 13:29:03.424794 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:29:03.424746 2585 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-8.ec2.internal" event="NodeHasSufficientPID" Sep 10 13:29:03.424929 ip-10-0-140-8 kubenswrapper[2585]: E0910 13:29:03.424914 2585 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Sep 10 13:29:03.424985 ip-10-0-140-8 kubenswrapper[2585]: E0910 13:29:03.424935 2585 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Sep 10 13:29:03.424985 ip-10-0-140-8 kubenswrapper[2585]: E0910 13:29:03.424946 2585 projected.go:196] Error preparing data for projected volume kube-api-access-x2vv5 for pod openshift-network-diagnostics/network-check-target-7262t: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Sep 10 13:29:03.425080 ip-10-0-140-8 kubenswrapper[2585]: E0910 13:29:03.425005 2585 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/082a7577-a14f-49fb-a9b1-3cf250e6735a-kube-api-access-x2vv5 podName:082a7577-a14f-49fb-a9b1-3cf250e6735a nodeName:}" failed. No retries permitted until 2026-09-10 13:29:11.424985944 +0000 UTC m=+36.297690127 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-api-access-x2vv5" (UniqueName: "kubernetes.io/projected/082a7577-a14f-49fb-a9b1-3cf250e6735a-kube-api-access-x2vv5") pod "network-check-target-7262t" (UID: "082a7577-a14f-49fb-a9b1-3cf250e6735a") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Sep 10 13:29:03.433965 ip-10-0-140-8 kubenswrapper[2585]: E0910 13:29:03.433940 2585 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-140-8.ec2.internal\" not found" Sep 10 13:29:03.534089 ip-10-0-140-8 kubenswrapper[2585]: E0910 13:29:03.534058 2585 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-140-8.ec2.internal\" not found" Sep 10 13:29:03.635129 ip-10-0-140-8 kubenswrapper[2585]: E0910 13:29:03.635086 2585 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-140-8.ec2.internal\" not found" Sep 10 13:29:03.641661 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:29:03.641444 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-7262t" Sep 10 13:29:03.641661 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:29:03.641488 2585 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Sep 10 13:29:03.641661 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:29:03.641502 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-xp5dk" Sep 10 13:29:03.641661 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:29:03.641544 2585 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Sep 10 13:29:03.643159 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:29:03.642801 2585 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-8.ec2.internal" event="NodeHasSufficientMemory" Sep 10 13:29:03.643159 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:29:03.642833 2585 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-8.ec2.internal" event="NodeHasNoDiskPressure" Sep 10 13:29:03.643159 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:29:03.642848 2585 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-8.ec2.internal" event="NodeHasSufficientPID" Sep 10 13:29:03.643159 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:29:03.642889 2585 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-8.ec2.internal" event="NodeHasSufficientMemory" Sep 10 13:29:03.643159 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:29:03.642905 2585 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-8.ec2.internal" event="NodeHasNoDiskPressure" Sep 10 13:29:03.643159 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:29:03.642916 2585 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-8.ec2.internal" event="NodeHasSufficientPID" Sep 10 13:29:03.643159 ip-10-0-140-8 kubenswrapper[2585]: E0910 13:29:03.643035 2585 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-7262t" podUID="082a7577-a14f-49fb-a9b1-3cf250e6735a" Sep 10 13:29:03.643159 ip-10-0-140-8 kubenswrapper[2585]: E0910 13:29:03.643120 2585 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-xp5dk" podUID="d552154f-1d5b-4c5d-99bf-7aeef22ce032" Sep 10 13:29:03.735306 ip-10-0-140-8 kubenswrapper[2585]: E0910 13:29:03.735207 2585 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-140-8.ec2.internal\" not found" Sep 10 13:29:03.835462 ip-10-0-140-8 kubenswrapper[2585]: E0910 13:29:03.835436 2585 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-140-8.ec2.internal\" not found" Sep 10 13:29:03.935545 ip-10-0-140-8 kubenswrapper[2585]: E0910 13:29:03.935512 2585 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-140-8.ec2.internal\" not found" Sep 10 13:29:04.036136 ip-10-0-140-8 kubenswrapper[2585]: E0910 13:29:04.036062 2585 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-140-8.ec2.internal\" not found" Sep 10 13:29:04.136176 ip-10-0-140-8 kubenswrapper[2585]: E0910 13:29:04.136139 2585 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-140-8.ec2.internal\" not found" Sep 10 13:29:04.236722 ip-10-0-140-8 kubenswrapper[2585]: E0910 13:29:04.236691 2585 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-140-8.ec2.internal\" not found" Sep 10 13:29:04.337611 ip-10-0-140-8 kubenswrapper[2585]: E0910 13:29:04.337540 2585 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-140-8.ec2.internal\" not found" Sep 10 13:29:04.438323 ip-10-0-140-8 kubenswrapper[2585]: E0910 13:29:04.438285 2585 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-140-8.ec2.internal\" not found" Sep 10 13:29:04.538595 ip-10-0-140-8 kubenswrapper[2585]: E0910 13:29:04.538571 2585 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-140-8.ec2.internal\" not found" Sep 10 13:29:04.639379 ip-10-0-140-8 kubenswrapper[2585]: E0910 13:29:04.639303 2585 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-140-8.ec2.internal\" not found" Sep 10 13:29:04.739976 ip-10-0-140-8 kubenswrapper[2585]: E0910 13:29:04.739937 2585 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-140-8.ec2.internal\" not found" Sep 10 13:29:04.772143 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:29:04.772110 2585 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-machine-config-operator_kube-rbac-proxy-crio-ip-10-0-140-8.ec2.internal_13b304e90198285e2c639c5158a21371/setup/1.log" Sep 10 13:29:04.772602 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:29:04.772586 2585 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-machine-config-operator_kube-rbac-proxy-crio-ip-10-0-140-8.ec2.internal_13b304e90198285e2c639c5158a21371/setup/0.log" Sep 10 13:29:04.772671 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:29:04.772621 2585 generic.go:356] "Generic (PLEG): container finished" podID="13b304e90198285e2c639c5158a21371" containerID="315d5848780927b4716cdf165cfb4e5552db7a32d37b279ceeb0eb32267f0dbf" exitCode=1 Sep 10 13:29:04.772671 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:29:04.772655 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-8.ec2.internal" event={"ID":"13b304e90198285e2c639c5158a21371","Type":"ContainerDied","Data":"315d5848780927b4716cdf165cfb4e5552db7a32d37b279ceeb0eb32267f0dbf"} Sep 10 13:29:04.772771 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:29:04.772707 2585 scope.go:117] "RemoveContainer" containerID="2bf530a7f40750d6e2f4653eade1223a580aa818515170971108be91ef4f5a74" Sep 10 13:29:04.772847 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:29:04.772833 2585 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Sep 10 13:29:04.774705 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:29:04.774681 2585 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-8.ec2.internal" event="NodeHasSufficientMemory" Sep 10 13:29:04.774793 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:29:04.774720 2585 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-8.ec2.internal" event="NodeHasNoDiskPressure" Sep 10 13:29:04.774793 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:29:04.774734 2585 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-8.ec2.internal" event="NodeHasSufficientPID" Sep 10 13:29:04.775053 ip-10-0-140-8 kubenswrapper[2585]: E0910 13:29:04.775031 2585 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-140-8.ec2.internal\" not found" node="ip-10-0-140-8.ec2.internal" Sep 10 13:29:04.775148 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:29:04.775112 2585 scope.go:117] "RemoveContainer" containerID="2bf530a7f40750d6e2f4653eade1223a580aa818515170971108be91ef4f5a74" Sep 10 13:29:04.840406 ip-10-0-140-8 kubenswrapper[2585]: E0910 13:29:04.840364 2585 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-140-8.ec2.internal\" not found" Sep 10 13:29:04.941013 ip-10-0-140-8 kubenswrapper[2585]: E0910 13:29:04.940930 2585 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-140-8.ec2.internal\" not found" Sep 10 13:29:05.041835 ip-10-0-140-8 kubenswrapper[2585]: E0910 13:29:05.041799 2585 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-140-8.ec2.internal\" not found" Sep 10 13:29:05.142385 ip-10-0-140-8 kubenswrapper[2585]: E0910 13:29:05.142354 2585 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-140-8.ec2.internal\" not found" Sep 10 13:29:05.243001 ip-10-0-140-8 kubenswrapper[2585]: E0910 13:29:05.242929 2585 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-140-8.ec2.internal\" not found" Sep 10 13:29:05.343596 ip-10-0-140-8 kubenswrapper[2585]: E0910 13:29:05.343561 2585 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-140-8.ec2.internal\" not found" Sep 10 13:29:05.443706 ip-10-0-140-8 kubenswrapper[2585]: E0910 13:29:05.443667 2585 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-140-8.ec2.internal\" not found" Sep 10 13:29:05.483288 ip-10-0-140-8 kubenswrapper[2585]: E0910 13:29:05.483239 2585 kubelet_node_status.go:536] "Error updating node status, will retry" err="error getting node \"ip-10-0-140-8.ec2.internal\": node \"ip-10-0-140-8.ec2.internal\" not found" Sep 10 13:29:05.544351 ip-10-0-140-8 kubenswrapper[2585]: E0910 13:29:05.544285 2585 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-140-8.ec2.internal\" not found" Sep 10 13:29:05.613639 ip-10-0-140-8 kubenswrapper[2585]: E0910 13:29:05.613606 2585 eviction_manager.go:292] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-140-8.ec2.internal\" not found" Sep 10 13:29:05.641647 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:29:05.641620 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-7262t" Sep 10 13:29:05.641647 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:29:05.641654 2585 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Sep 10 13:29:05.641850 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:29:05.641703 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-xp5dk" Sep 10 13:29:05.641850 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:29:05.641736 2585 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Sep 10 13:29:05.642845 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:29:05.642826 2585 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-8.ec2.internal" event="NodeHasSufficientMemory" Sep 10 13:29:05.642924 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:29:05.642855 2585 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-8.ec2.internal" event="NodeHasSufficientMemory" Sep 10 13:29:05.642924 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:29:05.642887 2585 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-8.ec2.internal" event="NodeHasNoDiskPressure" Sep 10 13:29:05.642924 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:29:05.642904 2585 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-8.ec2.internal" event="NodeHasSufficientPID" Sep 10 13:29:05.643065 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:29:05.642864 2585 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-8.ec2.internal" event="NodeHasNoDiskPressure" Sep 10 13:29:05.643065 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:29:05.643015 2585 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-8.ec2.internal" event="NodeHasSufficientPID" Sep 10 13:29:05.643223 ip-10-0-140-8 kubenswrapper[2585]: E0910 13:29:05.643198 2585 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-xp5dk" podUID="d552154f-1d5b-4c5d-99bf-7aeef22ce032" Sep 10 13:29:05.643344 ip-10-0-140-8 kubenswrapper[2585]: E0910 13:29:05.643317 2585 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-7262t" podUID="082a7577-a14f-49fb-a9b1-3cf250e6735a" Sep 10 13:29:05.645062 ip-10-0-140-8 kubenswrapper[2585]: E0910 13:29:05.645043 2585 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-140-8.ec2.internal\" not found" Sep 10 13:29:05.745670 ip-10-0-140-8 kubenswrapper[2585]: E0910 13:29:05.745633 2585 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-140-8.ec2.internal\" not found" Sep 10 13:29:05.846000 ip-10-0-140-8 kubenswrapper[2585]: E0910 13:29:05.845968 2585 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-140-8.ec2.internal\" not found" Sep 10 13:29:05.946315 ip-10-0-140-8 kubenswrapper[2585]: E0910 13:29:05.946276 2585 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-140-8.ec2.internal\" not found" Sep 10 13:29:06.047218 ip-10-0-140-8 kubenswrapper[2585]: E0910 13:29:06.047180 2585 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-140-8.ec2.internal\" not found" Sep 10 13:29:06.147740 ip-10-0-140-8 kubenswrapper[2585]: E0910 13:29:06.147658 2585 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-140-8.ec2.internal\" not found" Sep 10 13:29:06.248376 ip-10-0-140-8 kubenswrapper[2585]: E0910 13:29:06.248346 2585 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-140-8.ec2.internal\" not found" Sep 10 13:29:06.349027 ip-10-0-140-8 kubenswrapper[2585]: E0910 13:29:06.348996 2585 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-140-8.ec2.internal\" not found" Sep 10 13:29:06.449918 ip-10-0-140-8 kubenswrapper[2585]: E0910 13:29:06.449831 2585 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-140-8.ec2.internal\" not found" Sep 10 13:29:06.550760 ip-10-0-140-8 kubenswrapper[2585]: E0910 13:29:06.550723 2585 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-140-8.ec2.internal\" not found" Sep 10 13:29:06.651399 ip-10-0-140-8 kubenswrapper[2585]: E0910 13:29:06.651356 2585 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-140-8.ec2.internal\" not found" Sep 10 13:29:06.751747 ip-10-0-140-8 kubenswrapper[2585]: E0910 13:29:06.751653 2585 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-140-8.ec2.internal\" not found" Sep 10 13:29:06.851832 ip-10-0-140-8 kubenswrapper[2585]: E0910 13:29:06.851794 2585 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-140-8.ec2.internal\" not found" Sep 10 13:29:06.952301 ip-10-0-140-8 kubenswrapper[2585]: E0910 13:29:06.952270 2585 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-140-8.ec2.internal\" not found" Sep 10 13:29:07.053055 ip-10-0-140-8 kubenswrapper[2585]: E0910 13:29:07.052976 2585 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-140-8.ec2.internal\" not found" Sep 10 13:29:07.153625 ip-10-0-140-8 kubenswrapper[2585]: E0910 13:29:07.153589 2585 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-140-8.ec2.internal\" not found" Sep 10 13:29:07.254391 ip-10-0-140-8 kubenswrapper[2585]: E0910 13:29:07.254362 2585 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-140-8.ec2.internal\" not found" Sep 10 13:29:07.355007 ip-10-0-140-8 kubenswrapper[2585]: E0910 13:29:07.354966 2585 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-140-8.ec2.internal\" not found" Sep 10 13:29:07.455577 ip-10-0-140-8 kubenswrapper[2585]: E0910 13:29:07.455544 2585 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-140-8.ec2.internal\" not found" Sep 10 13:29:07.556573 ip-10-0-140-8 kubenswrapper[2585]: E0910 13:29:07.556535 2585 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-140-8.ec2.internal\" not found" Sep 10 13:29:07.641235 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:29:07.641157 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-7262t" Sep 10 13:29:07.641235 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:29:07.641203 2585 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Sep 10 13:29:07.641235 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:29:07.641208 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-xp5dk" Sep 10 13:29:07.641235 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:29:07.641239 2585 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Sep 10 13:29:07.642560 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:29:07.642539 2585 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-8.ec2.internal" event="NodeHasSufficientMemory" Sep 10 13:29:07.642668 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:29:07.642570 2585 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-8.ec2.internal" event="NodeHasNoDiskPressure" Sep 10 13:29:07.642668 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:29:07.642586 2585 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-8.ec2.internal" event="NodeHasSufficientPID" Sep 10 13:29:07.642668 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:29:07.642616 2585 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-8.ec2.internal" event="NodeHasSufficientMemory" Sep 10 13:29:07.642668 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:29:07.642646 2585 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-8.ec2.internal" event="NodeHasNoDiskPressure" Sep 10 13:29:07.642668 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:29:07.642663 2585 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-8.ec2.internal" event="NodeHasSufficientPID" Sep 10 13:29:07.642905 ip-10-0-140-8 kubenswrapper[2585]: E0910 13:29:07.642787 2585 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-xp5dk" podUID="d552154f-1d5b-4c5d-99bf-7aeef22ce032" Sep 10 13:29:07.642905 ip-10-0-140-8 kubenswrapper[2585]: E0910 13:29:07.642869 2585 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-7262t" podUID="082a7577-a14f-49fb-a9b1-3cf250e6735a" Sep 10 13:29:07.657396 ip-10-0-140-8 kubenswrapper[2585]: E0910 13:29:07.657368 2585 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-140-8.ec2.internal\" not found" Sep 10 13:29:07.758497 ip-10-0-140-8 kubenswrapper[2585]: E0910 13:29:07.758449 2585 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-140-8.ec2.internal\" not found" Sep 10 13:29:07.859514 ip-10-0-140-8 kubenswrapper[2585]: E0910 13:29:07.859470 2585 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-140-8.ec2.internal\" not found" Sep 10 13:29:07.960636 ip-10-0-140-8 kubenswrapper[2585]: E0910 13:29:07.960557 2585 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-140-8.ec2.internal\" not found" Sep 10 13:29:08.061531 ip-10-0-140-8 kubenswrapper[2585]: E0910 13:29:08.061497 2585 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-140-8.ec2.internal\" not found" Sep 10 13:29:08.162304 ip-10-0-140-8 kubenswrapper[2585]: E0910 13:29:08.162271 2585 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-140-8.ec2.internal\" not found" Sep 10 13:29:08.262926 ip-10-0-140-8 kubenswrapper[2585]: E0910 13:29:08.262846 2585 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-140-8.ec2.internal\" not found" Sep 10 13:29:08.363601 ip-10-0-140-8 kubenswrapper[2585]: E0910 13:29:08.363567 2585 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-140-8.ec2.internal\" not found" Sep 10 13:29:08.464393 ip-10-0-140-8 kubenswrapper[2585]: E0910 13:29:08.464359 2585 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-140-8.ec2.internal\" not found" Sep 10 13:29:08.564954 ip-10-0-140-8 kubenswrapper[2585]: E0910 13:29:08.564924 2585 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-140-8.ec2.internal\" not found" Sep 10 13:29:08.665865 ip-10-0-140-8 kubenswrapper[2585]: E0910 13:29:08.665831 2585 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-140-8.ec2.internal\" not found" Sep 10 13:29:08.762168 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:29:08.762129 2585 reflector.go:439] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:160" Sep 10 13:29:08.841922 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:29:08.841844 2585 kubelet.go:3248] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-140-8.ec2.internal" Sep 10 13:29:08.854136 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:29:08.854109 2585 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Sep 10 13:29:08.854324 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:29:08.854202 2585 kubelet.go:3248] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-8.ec2.internal" Sep 10 13:29:08.854619 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:29:08.854596 2585 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kube-system/kube-apiserver-proxy-ip-10-0-140-8.ec2.internal"] Sep 10 13:29:08.865303 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:29:08.865276 2585 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Sep 10 13:29:08.865684 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:29:08.865663 2585 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-8.ec2.internal"] Sep 10 13:29:09.561045 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:29:09.561010 2585 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kubelet-serving" Sep 10 13:29:09.574960 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:29:09.574929 2585 reflector.go:439] "Caches populated" logger="kubernetes.io/kubelet-serving" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" Sep 10 13:29:09.610505 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:29:09.610473 2585 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kubelet-serving" csr="csr-xl9xq" Sep 10 13:29:09.619910 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:29:09.619883 2585 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kubelet-serving" csr="csr-xl9xq" Sep 10 13:29:09.641676 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:29:09.641647 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-7262t" Sep 10 13:29:09.641830 ip-10-0-140-8 kubenswrapper[2585]: E0910 13:29:09.641765 2585 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-7262t" podUID="082a7577-a14f-49fb-a9b1-3cf250e6735a" Sep 10 13:29:09.641830 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:29:09.641795 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-xp5dk" Sep 10 13:29:09.641938 ip-10-0-140-8 kubenswrapper[2585]: E0910 13:29:09.641918 2585 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-xp5dk" podUID="d552154f-1d5b-4c5d-99bf-7aeef22ce032" Sep 10 13:29:10.625989 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:29:10.625944 2585 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-09-09 13:24:09 +0000 UTC" deadline="2028-02-25 06:34:28.991709614 +0000 UTC" Sep 10 13:29:10.625989 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:29:10.625985 2585 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="12785h5m18.365729249s" Sep 10 13:29:11.380259 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:29:11.380211 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d552154f-1d5b-4c5d-99bf-7aeef22ce032-metrics-certs\") pod \"network-metrics-daemon-xp5dk\" (UID: \"d552154f-1d5b-4c5d-99bf-7aeef22ce032\") " pod="openshift-multus/network-metrics-daemon-xp5dk" Sep 10 13:29:11.380421 ip-10-0-140-8 kubenswrapper[2585]: E0910 13:29:11.380347 2585 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Sep 10 13:29:11.380421 ip-10-0-140-8 kubenswrapper[2585]: E0910 13:29:11.380408 2585 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d552154f-1d5b-4c5d-99bf-7aeef22ce032-metrics-certs podName:d552154f-1d5b-4c5d-99bf-7aeef22ce032 nodeName:}" failed. No retries permitted until 2026-09-10 13:29:27.380389616 +0000 UTC m=+52.253093812 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/d552154f-1d5b-4c5d-99bf-7aeef22ce032-metrics-certs") pod "network-metrics-daemon-xp5dk" (UID: "d552154f-1d5b-4c5d-99bf-7aeef22ce032") : object "openshift-multus"/"metrics-daemon-secret" not registered Sep 10 13:29:11.480646 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:29:11.480615 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x2vv5\" (UniqueName: \"kubernetes.io/projected/082a7577-a14f-49fb-a9b1-3cf250e6735a-kube-api-access-x2vv5\") pod \"network-check-target-7262t\" (UID: \"082a7577-a14f-49fb-a9b1-3cf250e6735a\") " pod="openshift-network-diagnostics/network-check-target-7262t" Sep 10 13:29:11.480805 ip-10-0-140-8 kubenswrapper[2585]: E0910 13:29:11.480723 2585 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Sep 10 13:29:11.480805 ip-10-0-140-8 kubenswrapper[2585]: E0910 13:29:11.480737 2585 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Sep 10 13:29:11.480805 ip-10-0-140-8 kubenswrapper[2585]: E0910 13:29:11.480746 2585 projected.go:196] Error preparing data for projected volume kube-api-access-x2vv5 for pod openshift-network-diagnostics/network-check-target-7262t: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Sep 10 13:29:11.480898 ip-10-0-140-8 kubenswrapper[2585]: E0910 13:29:11.480811 2585 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/082a7577-a14f-49fb-a9b1-3cf250e6735a-kube-api-access-x2vv5 podName:082a7577-a14f-49fb-a9b1-3cf250e6735a nodeName:}" failed. No retries permitted until 2026-09-10 13:29:27.480800102 +0000 UTC m=+52.353504281 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "kube-api-access-x2vv5" (UniqueName: "kubernetes.io/projected/082a7577-a14f-49fb-a9b1-3cf250e6735a-kube-api-access-x2vv5") pod "network-check-target-7262t" (UID: "082a7577-a14f-49fb-a9b1-3cf250e6735a") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Sep 10 13:29:11.626895 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:29:11.626857 2585 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-09-09 13:24:09 +0000 UTC" deadline="2028-06-06 07:06:43.029500216 +0000 UTC" Sep 10 13:29:11.626895 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:29:11.626883 2585 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="15233h37m31.402619512s" Sep 10 13:29:11.641368 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:29:11.641309 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-7262t" Sep 10 13:29:11.641368 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:29:11.641330 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-xp5dk" Sep 10 13:29:11.641552 ip-10-0-140-8 kubenswrapper[2585]: E0910 13:29:11.641425 2585 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-7262t" podUID="082a7577-a14f-49fb-a9b1-3cf250e6735a" Sep 10 13:29:11.641552 ip-10-0-140-8 kubenswrapper[2585]: E0910 13:29:11.641492 2585 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-xp5dk" podUID="d552154f-1d5b-4c5d-99bf-7aeef22ce032" Sep 10 13:29:11.722620 ip-10-0-140-8 kubenswrapper[2585]: E0910 13:29:11.722568 2585 log.go:32] "RemoveContainer from runtime service failed" err="rpc error: code = Unknown desc = failed to delete container k8s_setup_kube-rbac-proxy-crio-ip-10-0-140-8.ec2.internal_openshift-machine-config-operator_13b304e90198285e2c639c5158a21371_0 in pod sandbox 7559703836be33fb7b5d7eb362b0bbf8c63d5fac9424c9db66ee5bc3059c6898: identifier is not a container" containerID="2bf530a7f40750d6e2f4653eade1223a580aa818515170971108be91ef4f5a74" Sep 10 13:29:11.722755 ip-10-0-140-8 kubenswrapper[2585]: E0910 13:29:11.722639 2585 kuberuntime_container.go:962] "Unhandled Error" err="failed to remove pod init container \"setup\": rpc error: code = Unknown desc = failed to delete container k8s_setup_kube-rbac-proxy-crio-ip-10-0-140-8.ec2.internal_openshift-machine-config-operator_13b304e90198285e2c639c5158a21371_0 in pod sandbox 7559703836be33fb7b5d7eb362b0bbf8c63d5fac9424c9db66ee5bc3059c6898: identifier is not a container; Skipping pod \"kube-rbac-proxy-crio-ip-10-0-140-8.ec2.internal_openshift-machine-config-operator(13b304e90198285e2c639c5158a21371)\"" logger="UnhandledError" Sep 10 13:29:11.722827 ip-10-0-140-8 kubenswrapper[2585]: E0910 13:29:11.722812 2585 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"setup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=setup pod=kube-rbac-proxy-crio-ip-10-0-140-8.ec2.internal_openshift-machine-config-operator(13b304e90198285e2c639c5158a21371)\"" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-8.ec2.internal" podUID="13b304e90198285e2c639c5158a21371" Sep 10 13:29:11.790295 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:29:11.790277 2585 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-machine-config-operator_kube-rbac-proxy-crio-ip-10-0-140-8.ec2.internal_13b304e90198285e2c639c5158a21371/setup/1.log" Sep 10 13:29:11.819469 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:29:11.819413 2585 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/kube-apiserver-proxy-ip-10-0-140-8.ec2.internal" podStartSLOduration=3.8193924729999997 podStartE2EDuration="3.819392473s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-10 13:29:08 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 13:29:11.818789663 +0000 UTC m=+36.691493891" watchObservedRunningTime="2026-09-10 13:29:11.819392473 +0000 UTC m=+36.692096674" Sep 10 13:29:12.793783 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:29:12.793541 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-l9l8q" event={"ID":"ad555c6a-4385-4f8e-93bb-a174cfaa8e74","Type":"ContainerStarted","Data":"9c82c71b605c1b995ae44d736d4e81b9fb104099b45514b83cb72952bcbfc47f"} Sep 10 13:29:12.795032 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:29:12.795008 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-fxgmp" event={"ID":"fa53c783-d40a-4024-9f7b-1d0d2b57f6dc","Type":"ContainerStarted","Data":"089d7eacfa14c5b8d2f06e032ff5e68ccd0465bc4e88eb503a02297ff7f5d2ee"} Sep 10 13:29:12.796888 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:29:12.796866 2585 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-machine-config-operator_kube-rbac-proxy-crio-ip-10-0-140-8.ec2.internal_13b304e90198285e2c639c5158a21371/setup/1.log" Sep 10 13:29:12.796973 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:29:12.796899 2585 generic.go:356] "Generic (PLEG): container finished" podID="13b304e90198285e2c639c5158a21371" containerID="935b591bd5d3aeb15c750954b5c0df3c2d8415441ffcef8cf019ac4d45f4ba4b" exitCode=0 Sep 10 13:29:12.796973 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:29:12.796963 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-8.ec2.internal" event={"ID":"13b304e90198285e2c639c5158a21371","Type":"ContainerDied","Data":"935b591bd5d3aeb15c750954b5c0df3c2d8415441ffcef8cf019ac4d45f4ba4b"} Sep 10 13:29:12.797047 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:29:12.797005 2585 scope.go:117] "RemoveContainer" containerID="315d5848780927b4716cdf165cfb4e5552db7a32d37b279ceeb0eb32267f0dbf" Sep 10 13:29:12.797238 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:29:12.797220 2585 scope.go:117] "RemoveContainer" containerID="315d5848780927b4716cdf165cfb4e5552db7a32d37b279ceeb0eb32267f0dbf" Sep 10 13:29:12.800278 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:29:12.800239 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-5h2q6" event={"ID":"2dac1f99-4132-43bd-8241-687afc22e3d5","Type":"ContainerStarted","Data":"b3d1394536749287777f28bad78fbcf1374a445f7ba2642616829fe616010c39"} Sep 10 13:29:12.800375 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:29:12.800327 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-5h2q6" event={"ID":"2dac1f99-4132-43bd-8241-687afc22e3d5","Type":"ContainerStarted","Data":"bd0cccb67dfce305af8bdbda97df16424208d8203d6d87b9ee6d403ff1a4746a"} Sep 10 13:29:12.800375 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:29:12.800367 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-5h2q6" event={"ID":"2dac1f99-4132-43bd-8241-687afc22e3d5","Type":"ContainerStarted","Data":"8e502b5a84a7273f20a4e2a3b2eada301addbd0a3fc1adb7e64f97b6e0486899"} Sep 10 13:29:12.800449 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:29:12.800406 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-5h2q6" event={"ID":"2dac1f99-4132-43bd-8241-687afc22e3d5","Type":"ContainerStarted","Data":"6bba35d62cccbf1a59ad491e45c756ef03a933c89e9d9598e4f112ece97084e1"} Sep 10 13:29:12.800449 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:29:12.800442 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-5h2q6" event={"ID":"2dac1f99-4132-43bd-8241-687afc22e3d5","Type":"ContainerStarted","Data":"ac85c413de3c568c5a255d54c91df74202333b9a023eab23788cf5c1d72d2144"} Sep 10 13:29:12.800543 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:29:12.800477 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-5h2q6" event={"ID":"2dac1f99-4132-43bd-8241-687afc22e3d5","Type":"ContainerStarted","Data":"118b0da0ab94fb196925adea90b54a345529eaca4e0a0d8bc14acc25b96cc5a5"} Sep 10 13:29:12.801816 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:29:12.801795 2585 generic.go:356] "Generic (PLEG): container finished" podID="9913a6bd-8e01-4034-9da7-f7d0b65a7eee" containerID="fa0c8d35353fbc048606a16acb3ee289e797520f636f4e87b36434ce530b865f" exitCode=0 Sep 10 13:29:12.801996 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:29:12.801864 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-vlmmc" event={"ID":"9913a6bd-8e01-4034-9da7-f7d0b65a7eee","Type":"ContainerDied","Data":"fa0c8d35353fbc048606a16acb3ee289e797520f636f4e87b36434ce530b865f"} Sep 10 13:29:12.803512 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:29:12.803477 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-s9hkb" event={"ID":"e5fa823c-4cfc-48cf-ac04-865b3769417c","Type":"ContainerStarted","Data":"59bafefa8ba32c98a6815261dd62486d28f2bc7694ff46ae094381739f734d03"} Sep 10 13:29:12.804950 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:29:12.804924 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-zhn9x" event={"ID":"1748fe6b-cce2-42ce-b8a7-10d12ffef95a","Type":"ContainerStarted","Data":"1debe134a9209f627abe3d30fd337bb1e8d3eae892b9e3d08b79ea75d3b97cfa"} Sep 10 13:29:12.807660 ip-10-0-140-8 kubenswrapper[2585]: E0910 13:29:12.807637 2585 log.go:32] "RemoveContainer from runtime service failed" err="rpc error: code = Unknown desc = failed to delete container k8s_setup_kube-rbac-proxy-crio-ip-10-0-140-8.ec2.internal_openshift-machine-config-operator_13b304e90198285e2c639c5158a21371_1 in pod sandbox 7559703836be33fb7b5d7eb362b0bbf8c63d5fac9424c9db66ee5bc3059c6898 from index: no such id: '315d5848780927b4716cdf165cfb4e5552db7a32d37b279ceeb0eb32267f0dbf'" containerID="315d5848780927b4716cdf165cfb4e5552db7a32d37b279ceeb0eb32267f0dbf" Sep 10 13:29:12.807737 ip-10-0-140-8 kubenswrapper[2585]: E0910 13:29:12.807673 2585 kuberuntime_container.go:962] "Unhandled Error" err="failed to remove pod init container \"setup\": rpc error: code = Unknown desc = failed to delete container k8s_setup_kube-rbac-proxy-crio-ip-10-0-140-8.ec2.internal_openshift-machine-config-operator_13b304e90198285e2c639c5158a21371_1 in pod sandbox 7559703836be33fb7b5d7eb362b0bbf8c63d5fac9424c9db66ee5bc3059c6898 from index: no such id: '315d5848780927b4716cdf165cfb4e5552db7a32d37b279ceeb0eb32267f0dbf'; Skipping pod \"kube-rbac-proxy-crio-ip-10-0-140-8.ec2.internal_openshift-machine-config-operator(13b304e90198285e2c639c5158a21371)\"" logger="UnhandledError" Sep 10 13:29:12.810185 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:29:12.810148 2585 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-l9l8q" podStartSLOduration=1.87786147 podStartE2EDuration="17.810135928s" totalImagesPullingTime="15.932274458s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-10 13:28:55 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 13:29:12.809939459 +0000 UTC m=+37.682643659" watchObservedRunningTime="2026-09-10 13:29:12.810135928 +0000 UTC m=+37.682840125" Sep 10 13:29:12.837119 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:29:12.837068 2585 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/konnectivity-agent-fxgmp" podStartSLOduration=2.149155091 podStartE2EDuration="17.837049945s" totalImagesPullingTime="15.687894854s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 13:28:55 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 13:29:12.836522608 +0000 UTC m=+37.709226810" watchObservedRunningTime="2026-09-10 13:29:12.837049945 +0000 UTC m=+37.709754147" Sep 10 13:29:12.850841 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:29:12.850696 2585 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/node-ca-zhn9x" podStartSLOduration=2.194481524 podStartE2EDuration="17.850678284s" totalImagesPullingTime="15.65619676s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-10 13:28:55 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 13:29:12.850155856 +0000 UTC m=+37.722860058" watchObservedRunningTime="2026-09-10 13:29:12.850678284 +0000 UTC m=+37.723382486" Sep 10 13:29:12.865212 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:29:12.865169 2585 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-node-tuning-operator/tuned-s9hkb" podStartSLOduration=2.012056778 podStartE2EDuration="17.865151569s" totalImagesPullingTime="15.853094791s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-10 13:28:55 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 13:29:12.864814563 +0000 UTC m=+37.737518765" watchObservedRunningTime="2026-09-10 13:29:12.865151569 +0000 UTC m=+37.737855773" Sep 10 13:29:13.641263 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:29:13.641220 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-xp5dk" Sep 10 13:29:13.641408 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:29:13.641294 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-7262t" Sep 10 13:29:13.641468 ip-10-0-140-8 kubenswrapper[2585]: E0910 13:29:13.641395 2585 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-xp5dk" podUID="d552154f-1d5b-4c5d-99bf-7aeef22ce032" Sep 10 13:29:13.641511 ip-10-0-140-8 kubenswrapper[2585]: E0910 13:29:13.641492 2585 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-7262t" podUID="082a7577-a14f-49fb-a9b1-3cf250e6735a" Sep 10 13:29:13.807876 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:29:13.807839 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-x2wmd" event={"ID":"c366889d-cdc5-4a43-af92-806d84e99e45","Type":"ContainerStarted","Data":"c606110503ea6f6f47644e578c0044cbc28a6dc803a0d2df1786bc37398d7b5e"} Sep 10 13:29:13.809703 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:29:13.809675 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-8.ec2.internal" event={"ID":"13b304e90198285e2c639c5158a21371","Type":"ContainerStarted","Data":"75a1794f6f3fad39f2bdde6a54e84caff8be17b5cbbd8a6da15197802eea0650"} Sep 10 13:29:13.856043 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:29:13.855996 2585 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-8.ec2.internal" podStartSLOduration=5.8559811029999995 podStartE2EDuration="5.855981103s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-10 13:29:08 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 13:29:13.855970114 +0000 UTC m=+38.728674315" watchObservedRunningTime="2026-09-10 13:29:13.855981103 +0000 UTC m=+38.728685303" Sep 10 13:29:13.856220 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:29:13.856202 2585 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-operator/iptables-alerter-x2wmd" podStartSLOduration=3.266313216 podStartE2EDuration="18.856197061s" totalImagesPullingTime="15.589883845s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-10 13:28:55 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 13:29:13.82643479 +0000 UTC m=+38.699138989" watchObservedRunningTime="2026-09-10 13:29:13.856197061 +0000 UTC m=+38.728901262" Sep 10 13:29:14.814905 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:29:14.814649 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-5h2q6" event={"ID":"2dac1f99-4132-43bd-8241-687afc22e3d5","Type":"ContainerStarted","Data":"8c3fecf46432f5f9b54b76df43d6a5864f171a8f3d13575cf2efb89aadc1727f"} Sep 10 13:29:15.642151 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:29:15.642119 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-7262t" Sep 10 13:29:15.642151 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:29:15.642151 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-xp5dk" Sep 10 13:29:15.642441 ip-10-0-140-8 kubenswrapper[2585]: E0910 13:29:15.642224 2585 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-7262t" podUID="082a7577-a14f-49fb-a9b1-3cf250e6735a" Sep 10 13:29:15.642441 ip-10-0-140-8 kubenswrapper[2585]: E0910 13:29:15.642364 2585 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-xp5dk" podUID="d552154f-1d5b-4c5d-99bf-7aeef22ce032" Sep 10 13:29:15.809729 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:29:15.809697 2585 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kube-system/konnectivity-agent-fxgmp" Sep 10 13:29:15.810301 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:29:15.810278 2585 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kube-system/konnectivity-agent-fxgmp" Sep 10 13:29:16.823123 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:29:16.822648 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-5h2q6" event={"ID":"2dac1f99-4132-43bd-8241-687afc22e3d5","Type":"ContainerStarted","Data":"598d801d63dd69008d0ed8a7f14e45bf62ace20c108b3a2ef18981cfc9e7f468"} Sep 10 13:29:16.823123 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:29:16.822859 2585 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ovn-kubernetes/ovnkube-node-5h2q6" podUID="2dac1f99-4132-43bd-8241-687afc22e3d5" containerName="ovn-controller" containerID="cri-o://118b0da0ab94fb196925adea90b54a345529eaca4e0a0d8bc14acc25b96cc5a5" gracePeriod=30 Sep 10 13:29:16.823123 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:29:16.822950 2585 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ovn-kubernetes/ovnkube-node-5h2q6" podUID="2dac1f99-4132-43bd-8241-687afc22e3d5" containerName="northd" containerID="cri-o://bd0cccb67dfce305af8bdbda97df16424208d8203d6d87b9ee6d403ff1a4746a" gracePeriod=30 Sep 10 13:29:16.823123 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:29:16.822966 2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-5h2q6" Sep 10 13:29:16.823123 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:29:16.822993 2585 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ovn-kubernetes/ovnkube-node-5h2q6" podUID="2dac1f99-4132-43bd-8241-687afc22e3d5" containerName="ovn-acl-logging" containerID="cri-o://ac85c413de3c568c5a255d54c91df74202333b9a023eab23788cf5c1d72d2144" gracePeriod=30 Sep 10 13:29:16.823123 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:29:16.823028 2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-5h2q6" Sep 10 13:29:16.823123 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:29:16.822925 2585 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ovn-kubernetes/ovnkube-node-5h2q6" podUID="2dac1f99-4132-43bd-8241-687afc22e3d5" containerName="kube-rbac-proxy-ovn-metrics" containerID="cri-o://8e502b5a84a7273f20a4e2a3b2eada301addbd0a3fc1adb7e64f97b6e0486899" gracePeriod=30 Sep 10 13:29:16.823123 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:29:16.823049 2585 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ovn-kubernetes/ovnkube-node-5h2q6" podUID="2dac1f99-4132-43bd-8241-687afc22e3d5" containerName="kube-rbac-proxy-node" containerID="cri-o://6bba35d62cccbf1a59ad491e45c756ef03a933c89e9d9598e4f112ece97084e1" gracePeriod=30 Sep 10 13:29:16.823123 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:29:16.822873 2585 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ovn-kubernetes/ovnkube-node-5h2q6" podUID="2dac1f99-4132-43bd-8241-687afc22e3d5" containerName="nbdb" containerID="cri-o://b3d1394536749287777f28bad78fbcf1374a445f7ba2642616829fe616010c39" gracePeriod=30 Sep 10 13:29:16.823123 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:29:16.822956 2585 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ovn-kubernetes/ovnkube-node-5h2q6" podUID="2dac1f99-4132-43bd-8241-687afc22e3d5" containerName="sbdb" containerID="cri-o://8c3fecf46432f5f9b54b76df43d6a5864f171a8f3d13575cf2efb89aadc1727f" gracePeriod=30 Sep 10 13:29:16.825602 ip-10-0-140-8 kubenswrapper[2585]: E0910 13:29:16.825490 2585 log.go:32] "ExecSync cmd from runtime service failed" err="rpc error: code = Unknown desc = command error: cannot register an exec PID: container is stopping, stdout: , stderr: , exit code -1" containerID="8c3fecf46432f5f9b54b76df43d6a5864f171a8f3d13575cf2efb89aadc1727f" cmd=["/bin/bash","-c","set -xeo pipefail\n. /ovnkube-lib/ovnkube-lib.sh || exit 1\novndb-readiness-probe \"sb\"\n"] Sep 10 13:29:16.827290 ip-10-0-140-8 kubenswrapper[2585]: E0910 13:29:16.827068 2585 log.go:32] "ExecSync cmd from runtime service failed" err="rpc error: code = Unknown desc = command error: cannot register an exec PID: container is stopping, stdout: , stderr: , exit code -1" containerID="8c3fecf46432f5f9b54b76df43d6a5864f171a8f3d13575cf2efb89aadc1727f" cmd=["/bin/bash","-c","set -xeo pipefail\n. /ovnkube-lib/ovnkube-lib.sh || exit 1\novndb-readiness-probe \"sb\"\n"] Sep 10 13:29:16.829136 ip-10-0-140-8 kubenswrapper[2585]: E0910 13:29:16.829105 2585 log.go:32] "ExecSync cmd from runtime service failed" err="rpc error: code = Unknown desc = command error: cannot register an exec PID: container is stopping, stdout: , stderr: , exit code -1" containerID="8c3fecf46432f5f9b54b76df43d6a5864f171a8f3d13575cf2efb89aadc1727f" cmd=["/bin/bash","-c","set -xeo pipefail\n. /ovnkube-lib/ovnkube-lib.sh || exit 1\novndb-readiness-probe \"sb\"\n"] Sep 10 13:29:16.829218 ip-10-0-140-8 kubenswrapper[2585]: E0910 13:29:16.829153 2585 prober.go:104] "Probe errored" err="rpc error: code = Unknown desc = command error: cannot register an exec PID: container is stopping, stdout: , stderr: , exit code -1" probeType="Readiness" pod="openshift-ovn-kubernetes/ovnkube-node-5h2q6" podUID="2dac1f99-4132-43bd-8241-687afc22e3d5" containerName="sbdb" probeResult="unknown" Sep 10 13:29:16.838445 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:29:16.838401 2585 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ovn-kubernetes/ovnkube-node-5h2q6" podUID="2dac1f99-4132-43bd-8241-687afc22e3d5" containerName="ovnkube-controller" containerID="cri-o://598d801d63dd69008d0ed8a7f14e45bf62ace20c108b3a2ef18981cfc9e7f468" gracePeriod=30 Sep 10 13:29:17.641203 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:29:17.641118 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-7262t" Sep 10 13:29:17.641452 ip-10-0-140-8 kubenswrapper[2585]: E0910 13:29:17.641278 2585 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-7262t" podUID="082a7577-a14f-49fb-a9b1-3cf250e6735a" Sep 10 13:29:17.641452 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:29:17.641361 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-xp5dk" Sep 10 13:29:17.641540 ip-10-0-140-8 kubenswrapper[2585]: E0910 13:29:17.641520 2585 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-xp5dk" podUID="d552154f-1d5b-4c5d-99bf-7aeef22ce032" Sep 10 13:29:17.829617 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:29:17.829593 2585 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-5h2q6_2dac1f99-4132-43bd-8241-687afc22e3d5/kube-rbac-proxy-ovn-metrics/0.log" Sep 10 13:29:17.830114 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:29:17.830096 2585 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-5h2q6_2dac1f99-4132-43bd-8241-687afc22e3d5/kube-rbac-proxy-node/0.log" Sep 10 13:29:17.830593 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:29:17.830569 2585 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-5h2q6_2dac1f99-4132-43bd-8241-687afc22e3d5/ovn-acl-logging/0.log" Sep 10 13:29:17.831088 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:29:17.831069 2585 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-5h2q6_2dac1f99-4132-43bd-8241-687afc22e3d5/ovn-controller/0.log" Sep 10 13:29:17.831180 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:29:17.831112 2585 generic.go:356] "Generic (PLEG): container finished" podID="2dac1f99-4132-43bd-8241-687afc22e3d5" containerID="8c3fecf46432f5f9b54b76df43d6a5864f171a8f3d13575cf2efb89aadc1727f" exitCode=0 Sep 10 13:29:17.831180 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:29:17.831133 2585 generic.go:356] "Generic (PLEG): container finished" podID="2dac1f99-4132-43bd-8241-687afc22e3d5" containerID="b3d1394536749287777f28bad78fbcf1374a445f7ba2642616829fe616010c39" exitCode=0 Sep 10 13:29:17.831180 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:29:17.831141 2585 generic.go:356] "Generic (PLEG): container finished" podID="2dac1f99-4132-43bd-8241-687afc22e3d5" containerID="bd0cccb67dfce305af8bdbda97df16424208d8203d6d87b9ee6d403ff1a4746a" exitCode=0 Sep 10 13:29:17.831180 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:29:17.831149 2585 generic.go:356] "Generic (PLEG): container finished" podID="2dac1f99-4132-43bd-8241-687afc22e3d5" containerID="8e502b5a84a7273f20a4e2a3b2eada301addbd0a3fc1adb7e64f97b6e0486899" exitCode=143 Sep 10 13:29:17.831180 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:29:17.831158 2585 generic.go:356] "Generic (PLEG): container finished" podID="2dac1f99-4132-43bd-8241-687afc22e3d5" containerID="6bba35d62cccbf1a59ad491e45c756ef03a933c89e9d9598e4f112ece97084e1" exitCode=143 Sep 10 13:29:17.831180 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:29:17.831166 2585 generic.go:356] "Generic (PLEG): container finished" podID="2dac1f99-4132-43bd-8241-687afc22e3d5" containerID="ac85c413de3c568c5a255d54c91df74202333b9a023eab23788cf5c1d72d2144" exitCode=143 Sep 10 13:29:17.831180 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:29:17.831174 2585 generic.go:356] "Generic (PLEG): container finished" podID="2dac1f99-4132-43bd-8241-687afc22e3d5" containerID="118b0da0ab94fb196925adea90b54a345529eaca4e0a0d8bc14acc25b96cc5a5" exitCode=143 Sep 10 13:29:17.831464 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:29:17.831187 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-5h2q6" event={"ID":"2dac1f99-4132-43bd-8241-687afc22e3d5","Type":"ContainerDied","Data":"8c3fecf46432f5f9b54b76df43d6a5864f171a8f3d13575cf2efb89aadc1727f"} Sep 10 13:29:17.831464 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:29:17.831269 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-5h2q6" event={"ID":"2dac1f99-4132-43bd-8241-687afc22e3d5","Type":"ContainerDied","Data":"b3d1394536749287777f28bad78fbcf1374a445f7ba2642616829fe616010c39"} Sep 10 13:29:17.831464 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:29:17.831310 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-5h2q6" event={"ID":"2dac1f99-4132-43bd-8241-687afc22e3d5","Type":"ContainerDied","Data":"bd0cccb67dfce305af8bdbda97df16424208d8203d6d87b9ee6d403ff1a4746a"} Sep 10 13:29:17.831464 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:29:17.831353 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-5h2q6" event={"ID":"2dac1f99-4132-43bd-8241-687afc22e3d5","Type":"ContainerDied","Data":"8e502b5a84a7273f20a4e2a3b2eada301addbd0a3fc1adb7e64f97b6e0486899"} Sep 10 13:29:17.831464 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:29:17.831394 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-5h2q6" event={"ID":"2dac1f99-4132-43bd-8241-687afc22e3d5","Type":"ContainerDied","Data":"6bba35d62cccbf1a59ad491e45c756ef03a933c89e9d9598e4f112ece97084e1"} Sep 10 13:29:17.831464 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:29:17.831422 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-5h2q6" event={"ID":"2dac1f99-4132-43bd-8241-687afc22e3d5","Type":"ContainerDied","Data":"ac85c413de3c568c5a255d54c91df74202333b9a023eab23788cf5c1d72d2144"} Sep 10 13:29:17.831464 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:29:17.831451 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-5h2q6" event={"ID":"2dac1f99-4132-43bd-8241-687afc22e3d5","Type":"ContainerDied","Data":"118b0da0ab94fb196925adea90b54a345529eaca4e0a0d8bc14acc25b96cc5a5"} Sep 10 13:29:19.641109 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:29:19.641078 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-xp5dk" Sep 10 13:29:19.641671 ip-10-0-140-8 kubenswrapper[2585]: E0910 13:29:19.641183 2585 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-xp5dk" podUID="d552154f-1d5b-4c5d-99bf-7aeef22ce032" Sep 10 13:29:19.641671 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:29:19.641235 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-7262t" Sep 10 13:29:19.641671 ip-10-0-140-8 kubenswrapper[2585]: E0910 13:29:19.641355 2585 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-7262t" podUID="082a7577-a14f-49fb-a9b1-3cf250e6735a" Sep 10 13:29:19.836951 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:29:19.836918 2585 generic.go:356] "Generic (PLEG): container finished" podID="9913a6bd-8e01-4034-9da7-f7d0b65a7eee" containerID="b48167f15e257e97cb703b844f83453e4387fd1f5ca6cee8b55eccbb1b0dbb0e" exitCode=0 Sep 10 13:29:19.837109 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:29:19.836979 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-vlmmc" event={"ID":"9913a6bd-8e01-4034-9da7-f7d0b65a7eee","Type":"ContainerDied","Data":"b48167f15e257e97cb703b844f83453e4387fd1f5ca6cee8b55eccbb1b0dbb0e"} Sep 10 13:29:19.858712 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:29:19.858667 2585 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ovn-kubernetes/ovnkube-node-5h2q6" podStartSLOduration=8.985252152 podStartE2EDuration="24.858652293s" totalImagesPullingTime="15.873400141s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-10 13:28:55 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 13:29:16.848535846 +0000 UTC m=+41.721240049" watchObservedRunningTime="2026-09-10 13:29:19.858652293 +0000 UTC m=+44.731356530" Sep 10 13:29:21.641107 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:29:21.641072 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-7262t" Sep 10 13:29:21.641574 ip-10-0-140-8 kubenswrapper[2585]: E0910 13:29:21.641172 2585 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-7262t" podUID="082a7577-a14f-49fb-a9b1-3cf250e6735a" Sep 10 13:29:21.641574 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:29:21.641265 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-xp5dk" Sep 10 13:29:21.641574 ip-10-0-140-8 kubenswrapper[2585]: E0910 13:29:21.641368 2585 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-xp5dk" podUID="d552154f-1d5b-4c5d-99bf-7aeef22ce032" Sep 10 13:29:21.843090 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:29:21.843061 2585 generic.go:356] "Generic (PLEG): container finished" podID="9913a6bd-8e01-4034-9da7-f7d0b65a7eee" containerID="8862f9eec0601ce991cf634cfdae9f602783535f3fc2df1107fd825ba872cbe3" exitCode=0 Sep 10 13:29:21.843269 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:29:21.843104 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-vlmmc" event={"ID":"9913a6bd-8e01-4034-9da7-f7d0b65a7eee","Type":"ContainerDied","Data":"8862f9eec0601ce991cf634cfdae9f602783535f3fc2df1107fd825ba872cbe3"} Sep 10 13:29:22.686999 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:29:22.686967 2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kube-system/konnectivity-agent-fxgmp" Sep 10 13:29:22.687424 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:29:22.687142 2585 prober_manager.go:312] "Failed to trigger a manual run" probe="Readiness" Sep 10 13:29:22.687756 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:29:22.687737 2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-fxgmp" Sep 10 13:29:23.640736 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:29:23.640702 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-xp5dk" Sep 10 13:29:23.640906 ip-10-0-140-8 kubenswrapper[2585]: E0910 13:29:23.640812 2585 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-xp5dk" podUID="d552154f-1d5b-4c5d-99bf-7aeef22ce032" Sep 10 13:29:23.640906 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:29:23.640842 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-7262t" Sep 10 13:29:23.640906 ip-10-0-140-8 kubenswrapper[2585]: E0910 13:29:23.640900 2585 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-7262t" podUID="082a7577-a14f-49fb-a9b1-3cf250e6735a" Sep 10 13:29:23.849351 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:29:23.849318 2585 generic.go:356] "Generic (PLEG): container finished" podID="9913a6bd-8e01-4034-9da7-f7d0b65a7eee" containerID="fd3235c3a5d72f14db8a41dda8d8d34d6f5f2e9adacdc1b7e1ef93d2c34335e1" exitCode=0 Sep 10 13:29:23.849705 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:29:23.849362 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-vlmmc" event={"ID":"9913a6bd-8e01-4034-9da7-f7d0b65a7eee","Type":"ContainerDied","Data":"fd3235c3a5d72f14db8a41dda8d8d34d6f5f2e9adacdc1b7e1ef93d2c34335e1"} Sep 10 13:29:25.641477 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:29:25.641451 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-7262t" Sep 10 13:29:25.641909 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:29:25.641486 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-xp5dk" Sep 10 13:29:25.641909 ip-10-0-140-8 kubenswrapper[2585]: E0910 13:29:25.641554 2585 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-7262t" podUID="082a7577-a14f-49fb-a9b1-3cf250e6735a" Sep 10 13:29:25.641909 ip-10-0-140-8 kubenswrapper[2585]: E0910 13:29:25.641725 2585 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-xp5dk" podUID="d552154f-1d5b-4c5d-99bf-7aeef22ce032" Sep 10 13:29:26.823418 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:29:26.823368 2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-5h2q6" Sep 10 13:29:26.823884 ip-10-0-140-8 kubenswrapper[2585]: E0910 13:29:26.823766 2585 log.go:32] "ExecSync cmd from runtime service failed" err="rpc error: code = NotFound desc = container is not created or running: checking if PID of 8c3fecf46432f5f9b54b76df43d6a5864f171a8f3d13575cf2efb89aadc1727f is running failed: container process not found" containerID="8c3fecf46432f5f9b54b76df43d6a5864f171a8f3d13575cf2efb89aadc1727f" cmd=["/bin/bash","-c","set -xeo pipefail\n. /ovnkube-lib/ovnkube-lib.sh || exit 1\novndb-readiness-probe \"sb\"\n"] Sep 10 13:29:26.823884 ip-10-0-140-8 kubenswrapper[2585]: E0910 13:29:26.823777 2585 log.go:32] "ExecSync cmd from runtime service failed" err="rpc error: code = NotFound desc = container is not created or running: checking if PID of b3d1394536749287777f28bad78fbcf1374a445f7ba2642616829fe616010c39 is running failed: container process not found" containerID="b3d1394536749287777f28bad78fbcf1374a445f7ba2642616829fe616010c39" cmd=["/bin/bash","-c","set -xeo pipefail\n. /ovnkube-lib/ovnkube-lib.sh || exit 1\novndb-readiness-probe \"nb\"\n"] Sep 10 13:29:26.824267 ip-10-0-140-8 kubenswrapper[2585]: E0910 13:29:26.824201 2585 log.go:32] "ExecSync cmd from runtime service failed" err="rpc error: code = NotFound desc = container is not created or running: checking if PID of 8c3fecf46432f5f9b54b76df43d6a5864f171a8f3d13575cf2efb89aadc1727f is running failed: container process not found" containerID="8c3fecf46432f5f9b54b76df43d6a5864f171a8f3d13575cf2efb89aadc1727f" cmd=["/bin/bash","-c","set -xeo pipefail\n. /ovnkube-lib/ovnkube-lib.sh || exit 1\novndb-readiness-probe \"sb\"\n"] Sep 10 13:29:26.824267 ip-10-0-140-8 kubenswrapper[2585]: E0910 13:29:26.824225 2585 log.go:32] "ExecSync cmd from runtime service failed" err="rpc error: code = NotFound desc = container is not created or running: checking if PID of b3d1394536749287777f28bad78fbcf1374a445f7ba2642616829fe616010c39 is running failed: container process not found" containerID="b3d1394536749287777f28bad78fbcf1374a445f7ba2642616829fe616010c39" cmd=["/bin/bash","-c","set -xeo pipefail\n. /ovnkube-lib/ovnkube-lib.sh || exit 1\novndb-readiness-probe \"nb\"\n"] Sep 10 13:29:26.824593 ip-10-0-140-8 kubenswrapper[2585]: E0910 13:29:26.824567 2585 log.go:32] "ExecSync cmd from runtime service failed" err="rpc error: code = NotFound desc = container is not created or running: checking if PID of 8c3fecf46432f5f9b54b76df43d6a5864f171a8f3d13575cf2efb89aadc1727f is running failed: container process not found" containerID="8c3fecf46432f5f9b54b76df43d6a5864f171a8f3d13575cf2efb89aadc1727f" cmd=["/bin/bash","-c","set -xeo pipefail\n. /ovnkube-lib/ovnkube-lib.sh || exit 1\novndb-readiness-probe \"sb\"\n"] Sep 10 13:29:26.824668 ip-10-0-140-8 kubenswrapper[2585]: E0910 13:29:26.824609 2585 prober.go:104] "Probe errored" err="rpc error: code = NotFound desc = container is not created or running: checking if PID of 8c3fecf46432f5f9b54b76df43d6a5864f171a8f3d13575cf2efb89aadc1727f is running failed: container process not found" probeType="Readiness" pod="openshift-ovn-kubernetes/ovnkube-node-5h2q6" podUID="2dac1f99-4132-43bd-8241-687afc22e3d5" containerName="sbdb" probeResult="unknown" Sep 10 13:29:26.824668 ip-10-0-140-8 kubenswrapper[2585]: E0910 13:29:26.824637 2585 log.go:32] "ExecSync cmd from runtime service failed" err="rpc error: code = NotFound desc = container is not created or running: checking if PID of b3d1394536749287777f28bad78fbcf1374a445f7ba2642616829fe616010c39 is running failed: container process not found" containerID="b3d1394536749287777f28bad78fbcf1374a445f7ba2642616829fe616010c39" cmd=["/bin/bash","-c","set -xeo pipefail\n. /ovnkube-lib/ovnkube-lib.sh || exit 1\novndb-readiness-probe \"nb\"\n"] Sep 10 13:29:26.824776 ip-10-0-140-8 kubenswrapper[2585]: E0910 13:29:26.824674 2585 prober.go:104] "Probe errored" err="rpc error: code = NotFound desc = container is not created or running: checking if PID of b3d1394536749287777f28bad78fbcf1374a445f7ba2642616829fe616010c39 is running failed: container process not found" probeType="Readiness" pod="openshift-ovn-kubernetes/ovnkube-node-5h2q6" podUID="2dac1f99-4132-43bd-8241-687afc22e3d5" containerName="nbdb" probeResult="unknown" Sep 10 13:29:27.480587 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:29:27.480549 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d552154f-1d5b-4c5d-99bf-7aeef22ce032-metrics-certs\") pod \"network-metrics-daemon-xp5dk\" (UID: \"d552154f-1d5b-4c5d-99bf-7aeef22ce032\") " pod="openshift-multus/network-metrics-daemon-xp5dk" Sep 10 13:29:27.480772 ip-10-0-140-8 kubenswrapper[2585]: E0910 13:29:27.480694 2585 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Sep 10 13:29:27.480772 ip-10-0-140-8 kubenswrapper[2585]: E0910 13:29:27.480765 2585 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d552154f-1d5b-4c5d-99bf-7aeef22ce032-metrics-certs podName:d552154f-1d5b-4c5d-99bf-7aeef22ce032 nodeName:}" failed. No retries permitted until 2026-09-10 13:29:59.480746917 +0000 UTC m=+84.353451099 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/d552154f-1d5b-4c5d-99bf-7aeef22ce032-metrics-certs") pod "network-metrics-daemon-xp5dk" (UID: "d552154f-1d5b-4c5d-99bf-7aeef22ce032") : object "openshift-multus"/"metrics-daemon-secret" not registered Sep 10 13:29:27.581521 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:29:27.581489 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x2vv5\" (UniqueName: \"kubernetes.io/projected/082a7577-a14f-49fb-a9b1-3cf250e6735a-kube-api-access-x2vv5\") pod \"network-check-target-7262t\" (UID: \"082a7577-a14f-49fb-a9b1-3cf250e6735a\") " pod="openshift-network-diagnostics/network-check-target-7262t" Sep 10 13:29:27.581713 ip-10-0-140-8 kubenswrapper[2585]: E0910 13:29:27.581661 2585 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Sep 10 13:29:27.581713 ip-10-0-140-8 kubenswrapper[2585]: E0910 13:29:27.581683 2585 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Sep 10 13:29:27.581713 ip-10-0-140-8 kubenswrapper[2585]: E0910 13:29:27.581694 2585 projected.go:196] Error preparing data for projected volume kube-api-access-x2vv5 for pod openshift-network-diagnostics/network-check-target-7262t: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Sep 10 13:29:27.581867 ip-10-0-140-8 kubenswrapper[2585]: E0910 13:29:27.581758 2585 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/082a7577-a14f-49fb-a9b1-3cf250e6735a-kube-api-access-x2vv5 podName:082a7577-a14f-49fb-a9b1-3cf250e6735a nodeName:}" failed. No retries permitted until 2026-09-10 13:29:59.581739385 +0000 UTC m=+84.454443578 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "kube-api-access-x2vv5" (UniqueName: "kubernetes.io/projected/082a7577-a14f-49fb-a9b1-3cf250e6735a-kube-api-access-x2vv5") pod "network-check-target-7262t" (UID: "082a7577-a14f-49fb-a9b1-3cf250e6735a") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Sep 10 13:29:27.641150 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:29:27.641121 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-xp5dk" Sep 10 13:29:27.641324 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:29:27.641281 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-7262t" Sep 10 13:29:27.641378 ip-10-0-140-8 kubenswrapper[2585]: E0910 13:29:27.641279 2585 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-xp5dk" podUID="d552154f-1d5b-4c5d-99bf-7aeef22ce032" Sep 10 13:29:27.641378 ip-10-0-140-8 kubenswrapper[2585]: E0910 13:29:27.641364 2585 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-7262t" podUID="082a7577-a14f-49fb-a9b1-3cf250e6735a" Sep 10 13:29:29.640706 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:29:29.640674 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-7262t" Sep 10 13:29:29.641317 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:29:29.640685 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-xp5dk" Sep 10 13:29:29.641317 ip-10-0-140-8 kubenswrapper[2585]: E0910 13:29:29.640817 2585 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-7262t" podUID="082a7577-a14f-49fb-a9b1-3cf250e6735a" Sep 10 13:29:29.641317 ip-10-0-140-8 kubenswrapper[2585]: E0910 13:29:29.640875 2585 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-xp5dk" podUID="d552154f-1d5b-4c5d-99bf-7aeef22ce032" Sep 10 13:29:29.864819 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:29:29.864735 2585 generic.go:356] "Generic (PLEG): container finished" podID="9913a6bd-8e01-4034-9da7-f7d0b65a7eee" containerID="18e5ab53726a7afd9317e354230af48c10110dc19bd9762134f0850446b8cb3d" exitCode=0 Sep 10 13:29:29.864819 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:29:29.864783 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-vlmmc" event={"ID":"9913a6bd-8e01-4034-9da7-f7d0b65a7eee","Type":"ContainerDied","Data":"18e5ab53726a7afd9317e354230af48c10110dc19bd9762134f0850446b8cb3d"} Sep 10 13:29:30.869593 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:29:30.869562 2585 generic.go:356] "Generic (PLEG): container finished" podID="9913a6bd-8e01-4034-9da7-f7d0b65a7eee" containerID="ff3e6296adacb6cb20bea9b4a8ded92281ef370b1d6de3963aefdd59608521bc" exitCode=0 Sep 10 13:29:30.869957 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:29:30.869603 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-vlmmc" event={"ID":"9913a6bd-8e01-4034-9da7-f7d0b65a7eee","Type":"ContainerDied","Data":"ff3e6296adacb6cb20bea9b4a8ded92281ef370b1d6de3963aefdd59608521bc"} Sep 10 13:29:31.643774 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:29:31.643750 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-xp5dk" Sep 10 13:29:31.643929 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:29:31.643750 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-7262t" Sep 10 13:29:31.643929 ip-10-0-140-8 kubenswrapper[2585]: E0910 13:29:31.643842 2585 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-xp5dk" podUID="d552154f-1d5b-4c5d-99bf-7aeef22ce032" Sep 10 13:29:31.643929 ip-10-0-140-8 kubenswrapper[2585]: E0910 13:29:31.643904 2585 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-7262t" podUID="082a7577-a14f-49fb-a9b1-3cf250e6735a" Sep 10 13:29:31.875195 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:29:31.875160 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-vlmmc" event={"ID":"9913a6bd-8e01-4034-9da7-f7d0b65a7eee","Type":"ContainerStarted","Data":"d61cd85715b4f97f2beeaa0cad20ebb3ddd1427cca8b139342a4dc470488de31"} Sep 10 13:29:31.907142 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:29:31.907052 2585 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-additional-cni-plugins-vlmmc" podStartSLOduration=6.284757849 podStartE2EDuration="36.907035123s" totalImagesPullingTime="30.622277274s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-10 13:28:55 +0000 UTC" imagePullSessionsCount=5 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 13:29:31.905804048 +0000 UTC m=+56.778508250" watchObservedRunningTime="2026-09-10 13:29:31.907035123 +0000 UTC m=+56.779739301" Sep 10 13:29:33.641369 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:29:33.641324 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-7262t" Sep 10 13:29:33.641780 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:29:33.641324 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-xp5dk" Sep 10 13:29:33.641780 ip-10-0-140-8 kubenswrapper[2585]: E0910 13:29:33.641429 2585 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-7262t" podUID="082a7577-a14f-49fb-a9b1-3cf250e6735a" Sep 10 13:29:33.641780 ip-10-0-140-8 kubenswrapper[2585]: E0910 13:29:33.641517 2585 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-xp5dk" podUID="d552154f-1d5b-4c5d-99bf-7aeef22ce032" Sep 10 13:29:35.641901 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:29:35.641876 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-xp5dk" Sep 10 13:29:35.642336 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:29:35.641949 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-7262t" Sep 10 13:29:35.642336 ip-10-0-140-8 kubenswrapper[2585]: E0910 13:29:35.641969 2585 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-xp5dk" podUID="d552154f-1d5b-4c5d-99bf-7aeef22ce032" Sep 10 13:29:35.642336 ip-10-0-140-8 kubenswrapper[2585]: E0910 13:29:35.642052 2585 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-7262t" podUID="082a7577-a14f-49fb-a9b1-3cf250e6735a" Sep 10 13:29:36.823623 ip-10-0-140-8 kubenswrapper[2585]: E0910 13:29:36.823430 2585 log.go:32] "ExecSync cmd from runtime service failed" err="rpc error: code = NotFound desc = container is not created or running: checking if PID of 8c3fecf46432f5f9b54b76df43d6a5864f171a8f3d13575cf2efb89aadc1727f is running failed: container process not found" containerID="8c3fecf46432f5f9b54b76df43d6a5864f171a8f3d13575cf2efb89aadc1727f" cmd=["/bin/bash","-c","set -xeo pipefail\n. /ovnkube-lib/ovnkube-lib.sh || exit 1\novndb-readiness-probe \"sb\"\n"] Sep 10 13:29:36.824044 ip-10-0-140-8 kubenswrapper[2585]: E0910 13:29:36.823437 2585 log.go:32] "ExecSync cmd from runtime service failed" err="rpc error: code = NotFound desc = container is not created or running: checking if PID of b3d1394536749287777f28bad78fbcf1374a445f7ba2642616829fe616010c39 is running failed: container process not found" containerID="b3d1394536749287777f28bad78fbcf1374a445f7ba2642616829fe616010c39" cmd=["/bin/bash","-c","set -xeo pipefail\n. /ovnkube-lib/ovnkube-lib.sh || exit 1\novndb-readiness-probe \"nb\"\n"] Sep 10 13:29:36.824044 ip-10-0-140-8 kubenswrapper[2585]: E0910 13:29:36.823908 2585 log.go:32] "ExecSync cmd from runtime service failed" err="rpc error: code = NotFound desc = container is not created or running: checking if PID of 8c3fecf46432f5f9b54b76df43d6a5864f171a8f3d13575cf2efb89aadc1727f is running failed: container process not found" containerID="8c3fecf46432f5f9b54b76df43d6a5864f171a8f3d13575cf2efb89aadc1727f" cmd=["/bin/bash","-c","set -xeo pipefail\n. /ovnkube-lib/ovnkube-lib.sh || exit 1\novndb-readiness-probe \"sb\"\n"] Sep 10 13:29:36.824044 ip-10-0-140-8 kubenswrapper[2585]: E0910 13:29:36.823916 2585 log.go:32] "ExecSync cmd from runtime service failed" err="rpc error: code = NotFound desc = container is not created or running: checking if PID of b3d1394536749287777f28bad78fbcf1374a445f7ba2642616829fe616010c39 is running failed: container process not found" containerID="b3d1394536749287777f28bad78fbcf1374a445f7ba2642616829fe616010c39" cmd=["/bin/bash","-c","set -xeo pipefail\n. /ovnkube-lib/ovnkube-lib.sh || exit 1\novndb-readiness-probe \"nb\"\n"] Sep 10 13:29:36.824203 ip-10-0-140-8 kubenswrapper[2585]: E0910 13:29:36.824183 2585 log.go:32] "ExecSync cmd from runtime service failed" err="rpc error: code = NotFound desc = container is not created or running: checking if PID of b3d1394536749287777f28bad78fbcf1374a445f7ba2642616829fe616010c39 is running failed: container process not found" containerID="b3d1394536749287777f28bad78fbcf1374a445f7ba2642616829fe616010c39" cmd=["/bin/bash","-c","set -xeo pipefail\n. /ovnkube-lib/ovnkube-lib.sh || exit 1\novndb-readiness-probe \"nb\"\n"] Sep 10 13:29:36.824258 ip-10-0-140-8 kubenswrapper[2585]: E0910 13:29:36.824199 2585 log.go:32] "ExecSync cmd from runtime service failed" err="rpc error: code = NotFound desc = container is not created or running: checking if PID of 8c3fecf46432f5f9b54b76df43d6a5864f171a8f3d13575cf2efb89aadc1727f is running failed: container process not found" containerID="8c3fecf46432f5f9b54b76df43d6a5864f171a8f3d13575cf2efb89aadc1727f" cmd=["/bin/bash","-c","set -xeo pipefail\n. /ovnkube-lib/ovnkube-lib.sh || exit 1\novndb-readiness-probe \"sb\"\n"] Sep 10 13:29:36.824258 ip-10-0-140-8 kubenswrapper[2585]: E0910 13:29:36.824212 2585 prober.go:104] "Probe errored" err="rpc error: code = NotFound desc = container is not created or running: checking if PID of b3d1394536749287777f28bad78fbcf1374a445f7ba2642616829fe616010c39 is running failed: container process not found" probeType="Readiness" pod="openshift-ovn-kubernetes/ovnkube-node-5h2q6" podUID="2dac1f99-4132-43bd-8241-687afc22e3d5" containerName="nbdb" probeResult="unknown" Sep 10 13:29:36.824258 ip-10-0-140-8 kubenswrapper[2585]: E0910 13:29:36.824231 2585 prober.go:104] "Probe errored" err="rpc error: code = NotFound desc = container is not created or running: checking if PID of 8c3fecf46432f5f9b54b76df43d6a5864f171a8f3d13575cf2efb89aadc1727f is running failed: container process not found" probeType="Readiness" pod="openshift-ovn-kubernetes/ovnkube-node-5h2q6" podUID="2dac1f99-4132-43bd-8241-687afc22e3d5" containerName="sbdb" probeResult="unknown" Sep 10 13:29:37.641021 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:29:37.640988 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-xp5dk" Sep 10 13:29:37.641225 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:29:37.640991 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-7262t" Sep 10 13:29:37.641225 ip-10-0-140-8 kubenswrapper[2585]: E0910 13:29:37.641095 2585 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-xp5dk" podUID="d552154f-1d5b-4c5d-99bf-7aeef22ce032" Sep 10 13:29:37.641225 ip-10-0-140-8 kubenswrapper[2585]: E0910 13:29:37.641157 2585 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-7262t" podUID="082a7577-a14f-49fb-a9b1-3cf250e6735a" Sep 10 13:29:39.641695 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:29:39.641664 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-7262t" Sep 10 13:29:39.642189 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:29:39.641670 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-xp5dk" Sep 10 13:29:39.642189 ip-10-0-140-8 kubenswrapper[2585]: E0910 13:29:39.641757 2585 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-7262t" podUID="082a7577-a14f-49fb-a9b1-3cf250e6735a" Sep 10 13:29:39.642189 ip-10-0-140-8 kubenswrapper[2585]: E0910 13:29:39.641866 2585 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-xp5dk" podUID="d552154f-1d5b-4c5d-99bf-7aeef22ce032" Sep 10 13:29:41.641346 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:29:41.641305 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-7262t" Sep 10 13:29:41.641346 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:29:41.641347 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-xp5dk" Sep 10 13:29:41.641743 ip-10-0-140-8 kubenswrapper[2585]: E0910 13:29:41.641442 2585 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-7262t" podUID="082a7577-a14f-49fb-a9b1-3cf250e6735a" Sep 10 13:29:41.641743 ip-10-0-140-8 kubenswrapper[2585]: E0910 13:29:41.641542 2585 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-xp5dk" podUID="d552154f-1d5b-4c5d-99bf-7aeef22ce032" Sep 10 13:29:43.641056 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:29:43.641020 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-xp5dk" Sep 10 13:29:43.641550 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:29:43.641024 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-7262t" Sep 10 13:29:43.641550 ip-10-0-140-8 kubenswrapper[2585]: E0910 13:29:43.641163 2585 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-xp5dk" podUID="d552154f-1d5b-4c5d-99bf-7aeef22ce032" Sep 10 13:29:43.641550 ip-10-0-140-8 kubenswrapper[2585]: E0910 13:29:43.641186 2585 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-7262t" podUID="082a7577-a14f-49fb-a9b1-3cf250e6735a" Sep 10 13:29:45.641747 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:29:45.641719 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-7262t" Sep 10 13:29:45.642117 ip-10-0-140-8 kubenswrapper[2585]: E0910 13:29:45.641798 2585 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-7262t" podUID="082a7577-a14f-49fb-a9b1-3cf250e6735a" Sep 10 13:29:45.642117 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:29:45.641908 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-xp5dk" Sep 10 13:29:45.642117 ip-10-0-140-8 kubenswrapper[2585]: E0910 13:29:45.642037 2585 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-xp5dk" podUID="d552154f-1d5b-4c5d-99bf-7aeef22ce032" Sep 10 13:29:46.824090 ip-10-0-140-8 kubenswrapper[2585]: E0910 13:29:46.824041 2585 log.go:32] "ExecSync cmd from runtime service failed" err="rpc error: code = NotFound desc = container is not created or running: checking if PID of 8c3fecf46432f5f9b54b76df43d6a5864f171a8f3d13575cf2efb89aadc1727f is running failed: container process not found" containerID="8c3fecf46432f5f9b54b76df43d6a5864f171a8f3d13575cf2efb89aadc1727f" cmd=["/bin/bash","-c","set -xeo pipefail\n. /ovnkube-lib/ovnkube-lib.sh || exit 1\novndb-readiness-probe \"sb\"\n"] Sep 10 13:29:46.824561 ip-10-0-140-8 kubenswrapper[2585]: E0910 13:29:46.824049 2585 log.go:32] "ExecSync cmd from runtime service failed" err="rpc error: code = NotFound desc = container is not created or running: checking if PID of b3d1394536749287777f28bad78fbcf1374a445f7ba2642616829fe616010c39 is running failed: container process not found" containerID="b3d1394536749287777f28bad78fbcf1374a445f7ba2642616829fe616010c39" cmd=["/bin/bash","-c","set -xeo pipefail\n. /ovnkube-lib/ovnkube-lib.sh || exit 1\novndb-readiness-probe \"nb\"\n"] Sep 10 13:29:46.824561 ip-10-0-140-8 kubenswrapper[2585]: E0910 13:29:46.824387 2585 log.go:32] "ExecSync cmd from runtime service failed" err="rpc error: code = NotFound desc = container is not created or running: checking if PID of b3d1394536749287777f28bad78fbcf1374a445f7ba2642616829fe616010c39 is running failed: container process not found" containerID="b3d1394536749287777f28bad78fbcf1374a445f7ba2642616829fe616010c39" cmd=["/bin/bash","-c","set -xeo pipefail\n. /ovnkube-lib/ovnkube-lib.sh || exit 1\novndb-readiness-probe \"nb\"\n"] Sep 10 13:29:46.824561 ip-10-0-140-8 kubenswrapper[2585]: E0910 13:29:46.824392 2585 log.go:32] "ExecSync cmd from runtime service failed" err="rpc error: code = NotFound desc = container is not created or running: checking if PID of 8c3fecf46432f5f9b54b76df43d6a5864f171a8f3d13575cf2efb89aadc1727f is running failed: container process not found" containerID="8c3fecf46432f5f9b54b76df43d6a5864f171a8f3d13575cf2efb89aadc1727f" cmd=["/bin/bash","-c","set -xeo pipefail\n. /ovnkube-lib/ovnkube-lib.sh || exit 1\novndb-readiness-probe \"sb\"\n"] Sep 10 13:29:46.824758 ip-10-0-140-8 kubenswrapper[2585]: E0910 13:29:46.824725 2585 log.go:32] "ExecSync cmd from runtime service failed" err="rpc error: code = NotFound desc = container is not created or running: checking if PID of b3d1394536749287777f28bad78fbcf1374a445f7ba2642616829fe616010c39 is running failed: container process not found" containerID="b3d1394536749287777f28bad78fbcf1374a445f7ba2642616829fe616010c39" cmd=["/bin/bash","-c","set -xeo pipefail\n. /ovnkube-lib/ovnkube-lib.sh || exit 1\novndb-readiness-probe \"nb\"\n"] Sep 10 13:29:46.824885 ip-10-0-140-8 kubenswrapper[2585]: E0910 13:29:46.824769 2585 prober.go:104] "Probe errored" err="rpc error: code = NotFound desc = container is not created or running: checking if PID of b3d1394536749287777f28bad78fbcf1374a445f7ba2642616829fe616010c39 is running failed: container process not found" probeType="Readiness" pod="openshift-ovn-kubernetes/ovnkube-node-5h2q6" podUID="2dac1f99-4132-43bd-8241-687afc22e3d5" containerName="nbdb" probeResult="unknown" Sep 10 13:29:46.824885 ip-10-0-140-8 kubenswrapper[2585]: E0910 13:29:46.824727 2585 log.go:32] "ExecSync cmd from runtime service failed" err="rpc error: code = NotFound desc = container is not created or running: checking if PID of 8c3fecf46432f5f9b54b76df43d6a5864f171a8f3d13575cf2efb89aadc1727f is running failed: container process not found" containerID="8c3fecf46432f5f9b54b76df43d6a5864f171a8f3d13575cf2efb89aadc1727f" cmd=["/bin/bash","-c","set -xeo pipefail\n. /ovnkube-lib/ovnkube-lib.sh || exit 1\novndb-readiness-probe \"sb\"\n"] Sep 10 13:29:46.824885 ip-10-0-140-8 kubenswrapper[2585]: E0910 13:29:46.824838 2585 prober.go:104] "Probe errored" err="rpc error: code = NotFound desc = container is not created or running: checking if PID of 8c3fecf46432f5f9b54b76df43d6a5864f171a8f3d13575cf2efb89aadc1727f is running failed: container process not found" probeType="Readiness" pod="openshift-ovn-kubernetes/ovnkube-node-5h2q6" podUID="2dac1f99-4132-43bd-8241-687afc22e3d5" containerName="sbdb" probeResult="unknown" Sep 10 13:29:46.825210 ip-10-0-140-8 kubenswrapper[2585]: E0910 13:29:46.825187 2585 log.go:32] "ExecSync cmd from runtime service failed" err="rpc error: code = Unknown desc = command error: cannot register an exec PID: container is stopping, stdout: , stderr: , exit code -1" containerID="598d801d63dd69008d0ed8a7f14e45bf62ace20c108b3a2ef18981cfc9e7f468" cmd=["test","-f","/etc/cni/net.d/10-ovn-kubernetes.conf"] Sep 10 13:29:46.826092 ip-10-0-140-8 kubenswrapper[2585]: E0910 13:29:46.826059 2585 log.go:32] "ExecSync cmd from runtime service failed" err="rpc error: code = Unknown desc = command error: cannot register an exec PID: container is stopping, stdout: , stderr: , exit code -1" containerID="598d801d63dd69008d0ed8a7f14e45bf62ace20c108b3a2ef18981cfc9e7f468" cmd=["test","-f","/etc/cni/net.d/10-ovn-kubernetes.conf"] Sep 10 13:29:46.826917 ip-10-0-140-8 kubenswrapper[2585]: E0910 13:29:46.826894 2585 log.go:32] "ExecSync cmd from runtime service failed" err="rpc error: code = Unknown desc = command error: cannot register an exec PID: container is stopping, stdout: , stderr: , exit code -1" containerID="598d801d63dd69008d0ed8a7f14e45bf62ace20c108b3a2ef18981cfc9e7f468" cmd=["test","-f","/etc/cni/net.d/10-ovn-kubernetes.conf"] Sep 10 13:29:46.826996 ip-10-0-140-8 kubenswrapper[2585]: E0910 13:29:46.826924 2585 prober.go:104] "Probe errored" err="rpc error: code = Unknown desc = command error: cannot register an exec PID: container is stopping, stdout: , stderr: , exit code -1" probeType="Readiness" pod="openshift-ovn-kubernetes/ovnkube-node-5h2q6" podUID="2dac1f99-4132-43bd-8241-687afc22e3d5" containerName="ovnkube-controller" probeResult="unknown" Sep 10 13:29:46.908745 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:29:46.908722 2585 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-5h2q6_2dac1f99-4132-43bd-8241-687afc22e3d5/ovnkube-controller/0.log" Sep 10 13:29:46.910112 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:29:46.910094 2585 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-5h2q6_2dac1f99-4132-43bd-8241-687afc22e3d5/kube-rbac-proxy-ovn-metrics/0.log" Sep 10 13:29:46.910547 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:29:46.910533 2585 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-5h2q6_2dac1f99-4132-43bd-8241-687afc22e3d5/kube-rbac-proxy-node/0.log" Sep 10 13:29:46.910906 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:29:46.910892 2585 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-5h2q6_2dac1f99-4132-43bd-8241-687afc22e3d5/ovn-acl-logging/0.log" Sep 10 13:29:46.911305 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:29:46.911292 2585 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-5h2q6_2dac1f99-4132-43bd-8241-687afc22e3d5/ovn-controller/0.log" Sep 10 13:29:46.911370 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:29:46.911321 2585 generic.go:356] "Generic (PLEG): container finished" podID="2dac1f99-4132-43bd-8241-687afc22e3d5" containerID="598d801d63dd69008d0ed8a7f14e45bf62ace20c108b3a2ef18981cfc9e7f468" exitCode=137 Sep 10 13:29:46.911370 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:29:46.911348 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-5h2q6" event={"ID":"2dac1f99-4132-43bd-8241-687afc22e3d5","Type":"ContainerDied","Data":"598d801d63dd69008d0ed8a7f14e45bf62ace20c108b3a2ef18981cfc9e7f468"} Sep 10 13:29:47.386409 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:29:47.386378 2585 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-5h2q6_2dac1f99-4132-43bd-8241-687afc22e3d5/ovnkube-controller/0.log" Sep 10 13:29:47.387756 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:29:47.387737 2585 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-5h2q6_2dac1f99-4132-43bd-8241-687afc22e3d5/kube-rbac-proxy-ovn-metrics/0.log" Sep 10 13:29:47.388138 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:29:47.388125 2585 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-5h2q6_2dac1f99-4132-43bd-8241-687afc22e3d5/kube-rbac-proxy-node/0.log" Sep 10 13:29:47.388465 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:29:47.388452 2585 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-5h2q6_2dac1f99-4132-43bd-8241-687afc22e3d5/ovn-acl-logging/0.log" Sep 10 13:29:47.388865 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:29:47.388852 2585 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-5h2q6_2dac1f99-4132-43bd-8241-687afc22e3d5/ovn-controller/0.log" Sep 10 13:29:47.405941 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:29:47.405914 2585 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-5h2q6" Sep 10 13:29:47.458097 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:29:47.458063 2585 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ovn-kubernetes/ovnkube-node-qz8x4"] Sep 10 13:29:47.458277 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:29:47.458229 2585 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2dac1f99-4132-43bd-8241-687afc22e3d5" containerName="ovnkube-controller" Sep 10 13:29:47.458352 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:29:47.458266 2585 state_mem.go:107] "Deleted CPUSet assignment" podUID="2dac1f99-4132-43bd-8241-687afc22e3d5" containerName="ovnkube-controller" Sep 10 13:29:47.458352 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:29:47.458292 2585 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2dac1f99-4132-43bd-8241-687afc22e3d5" containerName="kube-rbac-proxy-node" Sep 10 13:29:47.458352 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:29:47.458300 2585 state_mem.go:107] "Deleted CPUSet assignment" podUID="2dac1f99-4132-43bd-8241-687afc22e3d5" containerName="kube-rbac-proxy-node" Sep 10 13:29:47.458352 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:29:47.458311 2585 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2dac1f99-4132-43bd-8241-687afc22e3d5" containerName="kube-rbac-proxy-ovn-metrics" Sep 10 13:29:47.458352 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:29:47.458320 2585 state_mem.go:107] "Deleted CPUSet assignment" podUID="2dac1f99-4132-43bd-8241-687afc22e3d5" containerName="kube-rbac-proxy-ovn-metrics" Sep 10 13:29:47.458352 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:29:47.458329 2585 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2dac1f99-4132-43bd-8241-687afc22e3d5" containerName="northd" Sep 10 13:29:47.458352 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:29:47.458337 2585 state_mem.go:107] "Deleted CPUSet assignment" podUID="2dac1f99-4132-43bd-8241-687afc22e3d5" containerName="northd" Sep 10 13:29:47.458352 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:29:47.458346 2585 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2dac1f99-4132-43bd-8241-687afc22e3d5" containerName="nbdb" Sep 10 13:29:47.458352 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:29:47.458356 2585 state_mem.go:107] "Deleted CPUSet assignment" podUID="2dac1f99-4132-43bd-8241-687afc22e3d5" containerName="nbdb" Sep 10 13:29:47.458782 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:29:47.458369 2585 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2dac1f99-4132-43bd-8241-687afc22e3d5" containerName="ovn-controller" Sep 10 13:29:47.458782 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:29:47.458377 2585 state_mem.go:107] "Deleted CPUSet assignment" podUID="2dac1f99-4132-43bd-8241-687afc22e3d5" containerName="ovn-controller" Sep 10 13:29:47.458782 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:29:47.458387 2585 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2dac1f99-4132-43bd-8241-687afc22e3d5" containerName="ovn-acl-logging" Sep 10 13:29:47.458782 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:29:47.458402 2585 state_mem.go:107] "Deleted CPUSet assignment" podUID="2dac1f99-4132-43bd-8241-687afc22e3d5" containerName="ovn-acl-logging" Sep 10 13:29:47.458782 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:29:47.458415 2585 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2dac1f99-4132-43bd-8241-687afc22e3d5" containerName="sbdb" Sep 10 13:29:47.458782 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:29:47.458423 2585 state_mem.go:107] "Deleted CPUSet assignment" podUID="2dac1f99-4132-43bd-8241-687afc22e3d5" containerName="sbdb" Sep 10 13:29:47.458782 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:29:47.458459 2585 memory_manager.go:368] "RemoveStaleState removing state" podUID="2dac1f99-4132-43bd-8241-687afc22e3d5" containerName="ovnkube-controller" Sep 10 13:29:47.458782 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:29:47.458470 2585 memory_manager.go:368] "RemoveStaleState removing state" podUID="2dac1f99-4132-43bd-8241-687afc22e3d5" containerName="ovn-acl-logging" Sep 10 13:29:47.458782 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:29:47.458479 2585 memory_manager.go:368] "RemoveStaleState removing state" podUID="2dac1f99-4132-43bd-8241-687afc22e3d5" containerName="kube-rbac-proxy-ovn-metrics" Sep 10 13:29:47.458782 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:29:47.458488 2585 memory_manager.go:368] "RemoveStaleState removing state" podUID="2dac1f99-4132-43bd-8241-687afc22e3d5" containerName="nbdb" Sep 10 13:29:47.458782 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:29:47.458496 2585 memory_manager.go:368] "RemoveStaleState removing state" podUID="2dac1f99-4132-43bd-8241-687afc22e3d5" containerName="ovn-controller" Sep 10 13:29:47.458782 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:29:47.458503 2585 memory_manager.go:368] "RemoveStaleState removing state" podUID="2dac1f99-4132-43bd-8241-687afc22e3d5" containerName="kube-rbac-proxy-node" Sep 10 13:29:47.458782 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:29:47.458510 2585 memory_manager.go:368] "RemoveStaleState removing state" podUID="2dac1f99-4132-43bd-8241-687afc22e3d5" containerName="northd" Sep 10 13:29:47.458782 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:29:47.458517 2585 memory_manager.go:368] "RemoveStaleState removing state" podUID="2dac1f99-4132-43bd-8241-687afc22e3d5" containerName="sbdb" Sep 10 13:29:47.472801 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:29:47.472775 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-qz8x4" Sep 10 13:29:47.509257 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:29:47.509214 2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/2dac1f99-4132-43bd-8241-687afc22e3d5-ovnkube-config\") pod \"2dac1f99-4132-43bd-8241-687afc22e3d5\" (UID: \"2dac1f99-4132-43bd-8241-687afc22e3d5\") " Sep 10 13:29:47.509257 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:29:47.509257 2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/2dac1f99-4132-43bd-8241-687afc22e3d5-node-log\") pod \"2dac1f99-4132-43bd-8241-687afc22e3d5\" (UID: \"2dac1f99-4132-43bd-8241-687afc22e3d5\") " Sep 10 13:29:47.509446 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:29:47.509277 2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/2dac1f99-4132-43bd-8241-687afc22e3d5-host-run-netns\") pod \"2dac1f99-4132-43bd-8241-687afc22e3d5\" (UID: \"2dac1f99-4132-43bd-8241-687afc22e3d5\") " Sep 10 13:29:47.509446 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:29:47.509290 2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/2dac1f99-4132-43bd-8241-687afc22e3d5-host-kubelet\") pod \"2dac1f99-4132-43bd-8241-687afc22e3d5\" (UID: \"2dac1f99-4132-43bd-8241-687afc22e3d5\") " Sep 10 13:29:47.509446 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:29:47.509310 2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/2dac1f99-4132-43bd-8241-687afc22e3d5-host-cni-netd\") pod \"2dac1f99-4132-43bd-8241-687afc22e3d5\" (UID: \"2dac1f99-4132-43bd-8241-687afc22e3d5\") " Sep 10 13:29:47.509446 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:29:47.509330 2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/2dac1f99-4132-43bd-8241-687afc22e3d5-etc-openvswitch\") pod \"2dac1f99-4132-43bd-8241-687afc22e3d5\" (UID: \"2dac1f99-4132-43bd-8241-687afc22e3d5\") " Sep 10 13:29:47.509446 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:29:47.509325 2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/host-path/2dac1f99-4132-43bd-8241-687afc22e3d5-node-log" (OuterVolumeSpecName: "node-log") pod "2dac1f99-4132-43bd-8241-687afc22e3d5" (UID: "2dac1f99-4132-43bd-8241-687afc22e3d5"). InnerVolumeSpecName "node-log". PluginName "kubernetes.io/host-path", VolumeGIDValue "" Sep 10 13:29:47.509446 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:29:47.509336 2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/host-path/2dac1f99-4132-43bd-8241-687afc22e3d5-host-run-netns" (OuterVolumeSpecName: "host-run-netns") pod "2dac1f99-4132-43bd-8241-687afc22e3d5" (UID: "2dac1f99-4132-43bd-8241-687afc22e3d5"). InnerVolumeSpecName "host-run-netns". PluginName "kubernetes.io/host-path", VolumeGIDValue "" Sep 10 13:29:47.509446 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:29:47.509352 2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/2dac1f99-4132-43bd-8241-687afc22e3d5-env-overrides\") pod \"2dac1f99-4132-43bd-8241-687afc22e3d5\" (UID: \"2dac1f99-4132-43bd-8241-687afc22e3d5\") " Sep 10 13:29:47.509446 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:29:47.509353 2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/host-path/2dac1f99-4132-43bd-8241-687afc22e3d5-host-kubelet" (OuterVolumeSpecName: "host-kubelet") pod "2dac1f99-4132-43bd-8241-687afc22e3d5" (UID: "2dac1f99-4132-43bd-8241-687afc22e3d5"). InnerVolumeSpecName "host-kubelet". PluginName "kubernetes.io/host-path", VolumeGIDValue "" Sep 10 13:29:47.509446 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:29:47.509377 2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/2dac1f99-4132-43bd-8241-687afc22e3d5-host-var-lib-cni-networks-ovn-kubernetes\") pod \"2dac1f99-4132-43bd-8241-687afc22e3d5\" (UID: \"2dac1f99-4132-43bd-8241-687afc22e3d5\") " Sep 10 13:29:47.509446 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:29:47.509371 2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/host-path/2dac1f99-4132-43bd-8241-687afc22e3d5-host-cni-netd" (OuterVolumeSpecName: "host-cni-netd") pod "2dac1f99-4132-43bd-8241-687afc22e3d5" (UID: "2dac1f99-4132-43bd-8241-687afc22e3d5"). InnerVolumeSpecName "host-cni-netd". PluginName "kubernetes.io/host-path", VolumeGIDValue "" Sep 10 13:29:47.509446 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:29:47.509386 2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/host-path/2dac1f99-4132-43bd-8241-687afc22e3d5-etc-openvswitch" (OuterVolumeSpecName: "etc-openvswitch") pod "2dac1f99-4132-43bd-8241-687afc22e3d5" (UID: "2dac1f99-4132-43bd-8241-687afc22e3d5"). InnerVolumeSpecName "etc-openvswitch". PluginName "kubernetes.io/host-path", VolumeGIDValue "" Sep 10 13:29:47.509446 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:29:47.509401 2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/2dac1f99-4132-43bd-8241-687afc22e3d5-var-lib-openvswitch\") pod \"2dac1f99-4132-43bd-8241-687afc22e3d5\" (UID: \"2dac1f99-4132-43bd-8241-687afc22e3d5\") " Sep 10 13:29:47.509446 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:29:47.509414 2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/host-path/2dac1f99-4132-43bd-8241-687afc22e3d5-host-var-lib-cni-networks-ovn-kubernetes" (OuterVolumeSpecName: "host-var-lib-cni-networks-ovn-kubernetes") pod "2dac1f99-4132-43bd-8241-687afc22e3d5" (UID: "2dac1f99-4132-43bd-8241-687afc22e3d5"). InnerVolumeSpecName "host-var-lib-cni-networks-ovn-kubernetes". PluginName "kubernetes.io/host-path", VolumeGIDValue "" Sep 10 13:29:47.509446 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:29:47.509424 2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/host-path/2dac1f99-4132-43bd-8241-687afc22e3d5-var-lib-openvswitch" (OuterVolumeSpecName: "var-lib-openvswitch") pod "2dac1f99-4132-43bd-8241-687afc22e3d5" (UID: "2dac1f99-4132-43bd-8241-687afc22e3d5"). InnerVolumeSpecName "var-lib-openvswitch". PluginName "kubernetes.io/host-path", VolumeGIDValue "" Sep 10 13:29:47.509446 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:29:47.509446 2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/2dac1f99-4132-43bd-8241-687afc22e3d5-run-systemd\") pod \"2dac1f99-4132-43bd-8241-687afc22e3d5\" (UID: \"2dac1f99-4132-43bd-8241-687afc22e3d5\") " Sep 10 13:29:47.510060 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:29:47.509473 2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/2dac1f99-4132-43bd-8241-687afc22e3d5-systemd-units\") pod \"2dac1f99-4132-43bd-8241-687afc22e3d5\" (UID: \"2dac1f99-4132-43bd-8241-687afc22e3d5\") " Sep 10 13:29:47.510060 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:29:47.509508 2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/2dac1f99-4132-43bd-8241-687afc22e3d5-ovn-node-metrics-cert\") pod \"2dac1f99-4132-43bd-8241-687afc22e3d5\" (UID: \"2dac1f99-4132-43bd-8241-687afc22e3d5\") " Sep 10 13:29:47.510060 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:29:47.509531 2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/host-path/2dac1f99-4132-43bd-8241-687afc22e3d5-systemd-units" (OuterVolumeSpecName: "systemd-units") pod "2dac1f99-4132-43bd-8241-687afc22e3d5" (UID: "2dac1f99-4132-43bd-8241-687afc22e3d5"). InnerVolumeSpecName "systemd-units". PluginName "kubernetes.io/host-path", VolumeGIDValue "" Sep 10 13:29:47.510060 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:29:47.509560 2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/2dac1f99-4132-43bd-8241-687afc22e3d5-run-ovn\") pod \"2dac1f99-4132-43bd-8241-687afc22e3d5\" (UID: \"2dac1f99-4132-43bd-8241-687afc22e3d5\") " Sep 10 13:29:47.510060 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:29:47.509587 2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-lvjqt\" (UniqueName: \"kubernetes.io/projected/2dac1f99-4132-43bd-8241-687afc22e3d5-kube-api-access-lvjqt\") pod \"2dac1f99-4132-43bd-8241-687afc22e3d5\" (UID: \"2dac1f99-4132-43bd-8241-687afc22e3d5\") " Sep 10 13:29:47.510060 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:29:47.509607 2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/2dac1f99-4132-43bd-8241-687afc22e3d5-run-openvswitch\") pod \"2dac1f99-4132-43bd-8241-687afc22e3d5\" (UID: \"2dac1f99-4132-43bd-8241-687afc22e3d5\") " Sep 10 13:29:47.510060 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:29:47.509626 2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/2dac1f99-4132-43bd-8241-687afc22e3d5-host-slash\") pod \"2dac1f99-4132-43bd-8241-687afc22e3d5\" (UID: \"2dac1f99-4132-43bd-8241-687afc22e3d5\") " Sep 10 13:29:47.510060 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:29:47.509630 2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/host-path/2dac1f99-4132-43bd-8241-687afc22e3d5-run-ovn" (OuterVolumeSpecName: "run-ovn") pod "2dac1f99-4132-43bd-8241-687afc22e3d5" (UID: "2dac1f99-4132-43bd-8241-687afc22e3d5"). InnerVolumeSpecName "run-ovn". PluginName "kubernetes.io/host-path", VolumeGIDValue "" Sep 10 13:29:47.510060 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:29:47.509648 2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/2dac1f99-4132-43bd-8241-687afc22e3d5-ovnkube-script-lib\") pod \"2dac1f99-4132-43bd-8241-687afc22e3d5\" (UID: \"2dac1f99-4132-43bd-8241-687afc22e3d5\") " Sep 10 13:29:47.510060 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:29:47.509663 2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/host-path/2dac1f99-4132-43bd-8241-687afc22e3d5-run-openvswitch" (OuterVolumeSpecName: "run-openvswitch") pod "2dac1f99-4132-43bd-8241-687afc22e3d5" (UID: "2dac1f99-4132-43bd-8241-687afc22e3d5"). InnerVolumeSpecName "run-openvswitch". PluginName "kubernetes.io/host-path", VolumeGIDValue "" Sep 10 13:29:47.510060 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:29:47.509667 2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/2dac1f99-4132-43bd-8241-687afc22e3d5-log-socket\") pod \"2dac1f99-4132-43bd-8241-687afc22e3d5\" (UID: \"2dac1f99-4132-43bd-8241-687afc22e3d5\") " Sep 10 13:29:47.510060 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:29:47.509692 2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/host-path/2dac1f99-4132-43bd-8241-687afc22e3d5-log-socket" (OuterVolumeSpecName: "log-socket") pod "2dac1f99-4132-43bd-8241-687afc22e3d5" (UID: "2dac1f99-4132-43bd-8241-687afc22e3d5"). InnerVolumeSpecName "log-socket". PluginName "kubernetes.io/host-path", VolumeGIDValue "" Sep 10 13:29:47.510060 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:29:47.509706 2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/2dac1f99-4132-43bd-8241-687afc22e3d5-host-cni-bin\") pod \"2dac1f99-4132-43bd-8241-687afc22e3d5\" (UID: \"2dac1f99-4132-43bd-8241-687afc22e3d5\") " Sep 10 13:29:47.510060 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:29:47.509722 2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/host-path/2dac1f99-4132-43bd-8241-687afc22e3d5-host-slash" (OuterVolumeSpecName: "host-slash") pod "2dac1f99-4132-43bd-8241-687afc22e3d5" (UID: "2dac1f99-4132-43bd-8241-687afc22e3d5"). InnerVolumeSpecName "host-slash". PluginName "kubernetes.io/host-path", VolumeGIDValue "" Sep 10 13:29:47.510060 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:29:47.509724 2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2dac1f99-4132-43bd-8241-687afc22e3d5-env-overrides" (OuterVolumeSpecName: "env-overrides") pod "2dac1f99-4132-43bd-8241-687afc22e3d5" (UID: "2dac1f99-4132-43bd-8241-687afc22e3d5"). InnerVolumeSpecName "env-overrides". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 10 13:29:47.510060 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:29:47.509733 2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2dac1f99-4132-43bd-8241-687afc22e3d5-ovnkube-config" (OuterVolumeSpecName: "ovnkube-config") pod "2dac1f99-4132-43bd-8241-687afc22e3d5" (UID: "2dac1f99-4132-43bd-8241-687afc22e3d5"). InnerVolumeSpecName "ovnkube-config". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 10 13:29:47.510060 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:29:47.509730 2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/2dac1f99-4132-43bd-8241-687afc22e3d5-host-run-ovn-kubernetes\") pod \"2dac1f99-4132-43bd-8241-687afc22e3d5\" (UID: \"2dac1f99-4132-43bd-8241-687afc22e3d5\") " Sep 10 13:29:47.510702 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:29:47.509751 2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/host-path/2dac1f99-4132-43bd-8241-687afc22e3d5-host-run-ovn-kubernetes" (OuterVolumeSpecName: "host-run-ovn-kubernetes") pod "2dac1f99-4132-43bd-8241-687afc22e3d5" (UID: "2dac1f99-4132-43bd-8241-687afc22e3d5"). InnerVolumeSpecName "host-run-ovn-kubernetes". PluginName "kubernetes.io/host-path", VolumeGIDValue "" Sep 10 13:29:47.510702 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:29:47.509767 2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/host-path/2dac1f99-4132-43bd-8241-687afc22e3d5-host-cni-bin" (OuterVolumeSpecName: "host-cni-bin") pod "2dac1f99-4132-43bd-8241-687afc22e3d5" (UID: "2dac1f99-4132-43bd-8241-687afc22e3d5"). InnerVolumeSpecName "host-cni-bin". PluginName "kubernetes.io/host-path", VolumeGIDValue "" Sep 10 13:29:47.510702 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:29:47.509896 2585 reconciler_common.go:299] "Volume detached for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/2dac1f99-4132-43bd-8241-687afc22e3d5-host-slash\") on node \"ip-10-0-140-8.ec2.internal\" DevicePath \"\"" Sep 10 13:29:47.510702 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:29:47.509908 2585 reconciler_common.go:299] "Volume detached for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/2dac1f99-4132-43bd-8241-687afc22e3d5-log-socket\") on node \"ip-10-0-140-8.ec2.internal\" DevicePath \"\"" Sep 10 13:29:47.510702 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:29:47.509917 2585 reconciler_common.go:299] "Volume detached for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/2dac1f99-4132-43bd-8241-687afc22e3d5-host-cni-bin\") on node \"ip-10-0-140-8.ec2.internal\" DevicePath \"\"" Sep 10 13:29:47.510702 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:29:47.509932 2585 reconciler_common.go:299] "Volume detached for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/2dac1f99-4132-43bd-8241-687afc22e3d5-host-run-ovn-kubernetes\") on node \"ip-10-0-140-8.ec2.internal\" DevicePath \"\"" Sep 10 13:29:47.510702 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:29:47.509951 2585 reconciler_common.go:299] "Volume detached for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/2dac1f99-4132-43bd-8241-687afc22e3d5-ovnkube-config\") on node \"ip-10-0-140-8.ec2.internal\" DevicePath \"\"" Sep 10 13:29:47.510702 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:29:47.509966 2585 reconciler_common.go:299] "Volume detached for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/2dac1f99-4132-43bd-8241-687afc22e3d5-node-log\") on node \"ip-10-0-140-8.ec2.internal\" DevicePath \"\"" Sep 10 13:29:47.510702 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:29:47.509976 2585 reconciler_common.go:299] "Volume detached for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/2dac1f99-4132-43bd-8241-687afc22e3d5-host-run-netns\") on node \"ip-10-0-140-8.ec2.internal\" DevicePath \"\"" Sep 10 13:29:47.510702 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:29:47.509987 2585 reconciler_common.go:299] "Volume detached for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/2dac1f99-4132-43bd-8241-687afc22e3d5-host-kubelet\") on node \"ip-10-0-140-8.ec2.internal\" DevicePath \"\"" Sep 10 13:29:47.510702 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:29:47.509999 2585 reconciler_common.go:299] "Volume detached for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/2dac1f99-4132-43bd-8241-687afc22e3d5-host-cni-netd\") on node \"ip-10-0-140-8.ec2.internal\" DevicePath \"\"" Sep 10 13:29:47.510702 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:29:47.510019 2585 reconciler_common.go:299] "Volume detached for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/2dac1f99-4132-43bd-8241-687afc22e3d5-etc-openvswitch\") on node \"ip-10-0-140-8.ec2.internal\" DevicePath \"\"" Sep 10 13:29:47.510702 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:29:47.510029 2585 reconciler_common.go:299] "Volume detached for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/2dac1f99-4132-43bd-8241-687afc22e3d5-env-overrides\") on node \"ip-10-0-140-8.ec2.internal\" DevicePath \"\"" Sep 10 13:29:47.510702 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:29:47.510040 2585 reconciler_common.go:299] "Volume detached for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/2dac1f99-4132-43bd-8241-687afc22e3d5-host-var-lib-cni-networks-ovn-kubernetes\") on node \"ip-10-0-140-8.ec2.internal\" DevicePath \"\"" Sep 10 13:29:47.510702 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:29:47.510052 2585 reconciler_common.go:299] "Volume detached for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/2dac1f99-4132-43bd-8241-687afc22e3d5-var-lib-openvswitch\") on node \"ip-10-0-140-8.ec2.internal\" DevicePath \"\"" Sep 10 13:29:47.510702 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:29:47.510062 2585 reconciler_common.go:299] "Volume detached for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/2dac1f99-4132-43bd-8241-687afc22e3d5-systemd-units\") on node \"ip-10-0-140-8.ec2.internal\" DevicePath \"\"" Sep 10 13:29:47.510702 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:29:47.510073 2585 reconciler_common.go:299] "Volume detached for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/2dac1f99-4132-43bd-8241-687afc22e3d5-run-ovn\") on node \"ip-10-0-140-8.ec2.internal\" DevicePath \"\"" Sep 10 13:29:47.510702 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:29:47.509996 2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2dac1f99-4132-43bd-8241-687afc22e3d5-ovnkube-script-lib" (OuterVolumeSpecName: "ovnkube-script-lib") pod "2dac1f99-4132-43bd-8241-687afc22e3d5" (UID: "2dac1f99-4132-43bd-8241-687afc22e3d5"). InnerVolumeSpecName "ovnkube-script-lib". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 10 13:29:47.510702 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:29:47.510085 2585 reconciler_common.go:299] "Volume detached for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/2dac1f99-4132-43bd-8241-687afc22e3d5-run-openvswitch\") on node \"ip-10-0-140-8.ec2.internal\" DevicePath \"\"" Sep 10 13:29:47.512772 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:29:47.512750 2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2dac1f99-4132-43bd-8241-687afc22e3d5-ovn-node-metrics-cert" (OuterVolumeSpecName: "ovn-node-metrics-cert") pod "2dac1f99-4132-43bd-8241-687afc22e3d5" (UID: "2dac1f99-4132-43bd-8241-687afc22e3d5"). InnerVolumeSpecName "ovn-node-metrics-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 10 13:29:47.512961 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:29:47.512940 2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2dac1f99-4132-43bd-8241-687afc22e3d5-kube-api-access-lvjqt" (OuterVolumeSpecName: "kube-api-access-lvjqt") pod "2dac1f99-4132-43bd-8241-687afc22e3d5" (UID: "2dac1f99-4132-43bd-8241-687afc22e3d5"). InnerVolumeSpecName "kube-api-access-lvjqt". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 10 13:29:47.513821 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:29:47.513805 2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/host-path/2dac1f99-4132-43bd-8241-687afc22e3d5-run-systemd" (OuterVolumeSpecName: "run-systemd") pod "2dac1f99-4132-43bd-8241-687afc22e3d5" (UID: "2dac1f99-4132-43bd-8241-687afc22e3d5"). InnerVolumeSpecName "run-systemd". PluginName "kubernetes.io/host-path", VolumeGIDValue "" Sep 10 13:29:47.611344 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:29:47.611302 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/6c90873d-f3b0-42a4-adc8-6a644a1f2d11-host-slash\") pod \"ovnkube-node-qz8x4\" (UID: \"6c90873d-f3b0-42a4-adc8-6a644a1f2d11\") " pod="openshift-ovn-kubernetes/ovnkube-node-qz8x4" Sep 10 13:29:47.611344 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:29:47.611347 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/6c90873d-f3b0-42a4-adc8-6a644a1f2d11-host-run-netns\") pod \"ovnkube-node-qz8x4\" (UID: \"6c90873d-f3b0-42a4-adc8-6a644a1f2d11\") " pod="openshift-ovn-kubernetes/ovnkube-node-qz8x4" Sep 10 13:29:47.611555 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:29:47.611371 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/6c90873d-f3b0-42a4-adc8-6a644a1f2d11-log-socket\") pod \"ovnkube-node-qz8x4\" (UID: \"6c90873d-f3b0-42a4-adc8-6a644a1f2d11\") " pod="openshift-ovn-kubernetes/ovnkube-node-qz8x4" Sep 10 13:29:47.611555 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:29:47.611417 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/6c90873d-f3b0-42a4-adc8-6a644a1f2d11-ovnkube-script-lib\") pod \"ovnkube-node-qz8x4\" (UID: \"6c90873d-f3b0-42a4-adc8-6a644a1f2d11\") " pod="openshift-ovn-kubernetes/ovnkube-node-qz8x4" Sep 10 13:29:47.611555 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:29:47.611439 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/6c90873d-f3b0-42a4-adc8-6a644a1f2d11-node-log\") pod \"ovnkube-node-qz8x4\" (UID: \"6c90873d-f3b0-42a4-adc8-6a644a1f2d11\") " pod="openshift-ovn-kubernetes/ovnkube-node-qz8x4" Sep 10 13:29:47.611555 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:29:47.611459 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/6c90873d-f3b0-42a4-adc8-6a644a1f2d11-host-cni-netd\") pod \"ovnkube-node-qz8x4\" (UID: \"6c90873d-f3b0-42a4-adc8-6a644a1f2d11\") " pod="openshift-ovn-kubernetes/ovnkube-node-qz8x4" Sep 10 13:29:47.611555 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:29:47.611494 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/6c90873d-f3b0-42a4-adc8-6a644a1f2d11-systemd-units\") pod \"ovnkube-node-qz8x4\" (UID: \"6c90873d-f3b0-42a4-adc8-6a644a1f2d11\") " pod="openshift-ovn-kubernetes/ovnkube-node-qz8x4" Sep 10 13:29:47.611555 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:29:47.611518 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/6c90873d-f3b0-42a4-adc8-6a644a1f2d11-run-openvswitch\") pod \"ovnkube-node-qz8x4\" (UID: \"6c90873d-f3b0-42a4-adc8-6a644a1f2d11\") " pod="openshift-ovn-kubernetes/ovnkube-node-qz8x4" Sep 10 13:29:47.611555 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:29:47.611547 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/6c90873d-f3b0-42a4-adc8-6a644a1f2d11-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-qz8x4\" (UID: \"6c90873d-f3b0-42a4-adc8-6a644a1f2d11\") " pod="openshift-ovn-kubernetes/ovnkube-node-qz8x4" Sep 10 13:29:47.611914 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:29:47.611578 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/6c90873d-f3b0-42a4-adc8-6a644a1f2d11-host-kubelet\") pod \"ovnkube-node-qz8x4\" (UID: \"6c90873d-f3b0-42a4-adc8-6a644a1f2d11\") " pod="openshift-ovn-kubernetes/ovnkube-node-qz8x4" Sep 10 13:29:47.611914 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:29:47.611600 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/6c90873d-f3b0-42a4-adc8-6a644a1f2d11-run-systemd\") pod \"ovnkube-node-qz8x4\" (UID: \"6c90873d-f3b0-42a4-adc8-6a644a1f2d11\") " pod="openshift-ovn-kubernetes/ovnkube-node-qz8x4" Sep 10 13:29:47.611914 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:29:47.611623 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/6c90873d-f3b0-42a4-adc8-6a644a1f2d11-var-lib-openvswitch\") pod \"ovnkube-node-qz8x4\" (UID: \"6c90873d-f3b0-42a4-adc8-6a644a1f2d11\") " pod="openshift-ovn-kubernetes/ovnkube-node-qz8x4" Sep 10 13:29:47.611914 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:29:47.611645 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/6c90873d-f3b0-42a4-adc8-6a644a1f2d11-run-ovn\") pod \"ovnkube-node-qz8x4\" (UID: \"6c90873d-f3b0-42a4-adc8-6a644a1f2d11\") " pod="openshift-ovn-kubernetes/ovnkube-node-qz8x4" Sep 10 13:29:47.611914 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:29:47.611674 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/6c90873d-f3b0-42a4-adc8-6a644a1f2d11-host-run-ovn-kubernetes\") pod \"ovnkube-node-qz8x4\" (UID: \"6c90873d-f3b0-42a4-adc8-6a644a1f2d11\") " pod="openshift-ovn-kubernetes/ovnkube-node-qz8x4" Sep 10 13:29:47.611914 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:29:47.611697 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/6c90873d-f3b0-42a4-adc8-6a644a1f2d11-host-cni-bin\") pod \"ovnkube-node-qz8x4\" (UID: \"6c90873d-f3b0-42a4-adc8-6a644a1f2d11\") " pod="openshift-ovn-kubernetes/ovnkube-node-qz8x4" Sep 10 13:29:47.611914 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:29:47.611721 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/6c90873d-f3b0-42a4-adc8-6a644a1f2d11-env-overrides\") pod \"ovnkube-node-qz8x4\" (UID: \"6c90873d-f3b0-42a4-adc8-6a644a1f2d11\") " pod="openshift-ovn-kubernetes/ovnkube-node-qz8x4" Sep 10 13:29:47.611914 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:29:47.611745 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bkt4p\" (UniqueName: \"kubernetes.io/projected/6c90873d-f3b0-42a4-adc8-6a644a1f2d11-kube-api-access-bkt4p\") pod \"ovnkube-node-qz8x4\" (UID: \"6c90873d-f3b0-42a4-adc8-6a644a1f2d11\") " pod="openshift-ovn-kubernetes/ovnkube-node-qz8x4" Sep 10 13:29:47.611914 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:29:47.611773 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/6c90873d-f3b0-42a4-adc8-6a644a1f2d11-etc-openvswitch\") pod \"ovnkube-node-qz8x4\" (UID: \"6c90873d-f3b0-42a4-adc8-6a644a1f2d11\") " pod="openshift-ovn-kubernetes/ovnkube-node-qz8x4" Sep 10 13:29:47.611914 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:29:47.611796 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/6c90873d-f3b0-42a4-adc8-6a644a1f2d11-ovnkube-config\") pod \"ovnkube-node-qz8x4\" (UID: \"6c90873d-f3b0-42a4-adc8-6a644a1f2d11\") " pod="openshift-ovn-kubernetes/ovnkube-node-qz8x4" Sep 10 13:29:47.611914 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:29:47.611821 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/6c90873d-f3b0-42a4-adc8-6a644a1f2d11-ovn-node-metrics-cert\") pod \"ovnkube-node-qz8x4\" (UID: \"6c90873d-f3b0-42a4-adc8-6a644a1f2d11\") " pod="openshift-ovn-kubernetes/ovnkube-node-qz8x4" Sep 10 13:29:47.611914 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:29:47.611873 2585 reconciler_common.go:299] "Volume detached for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/2dac1f99-4132-43bd-8241-687afc22e3d5-run-systemd\") on node \"ip-10-0-140-8.ec2.internal\" DevicePath \"\"" Sep 10 13:29:47.611914 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:29:47.611891 2585 reconciler_common.go:299] "Volume detached for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/2dac1f99-4132-43bd-8241-687afc22e3d5-ovn-node-metrics-cert\") on node \"ip-10-0-140-8.ec2.internal\" DevicePath \"\"" Sep 10 13:29:47.611914 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:29:47.611906 2585 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lvjqt\" (UniqueName: \"kubernetes.io/projected/2dac1f99-4132-43bd-8241-687afc22e3d5-kube-api-access-lvjqt\") on node \"ip-10-0-140-8.ec2.internal\" DevicePath \"\"" Sep 10 13:29:47.611914 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:29:47.611921 2585 reconciler_common.go:299] "Volume detached for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/2dac1f99-4132-43bd-8241-687afc22e3d5-ovnkube-script-lib\") on node \"ip-10-0-140-8.ec2.internal\" DevicePath \"\"" Sep 10 13:29:47.641596 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:29:47.641534 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-xp5dk" Sep 10 13:29:47.641685 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:29:47.641535 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-7262t" Sep 10 13:29:47.641685 ip-10-0-140-8 kubenswrapper[2585]: E0910 13:29:47.641648 2585 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-xp5dk" podUID="d552154f-1d5b-4c5d-99bf-7aeef22ce032" Sep 10 13:29:47.641760 ip-10-0-140-8 kubenswrapper[2585]: E0910 13:29:47.641701 2585 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-7262t" podUID="082a7577-a14f-49fb-a9b1-3cf250e6735a" Sep 10 13:29:47.712814 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:29:47.712778 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/6c90873d-f3b0-42a4-adc8-6a644a1f2d11-host-run-netns\") pod \"ovnkube-node-qz8x4\" (UID: \"6c90873d-f3b0-42a4-adc8-6a644a1f2d11\") " pod="openshift-ovn-kubernetes/ovnkube-node-qz8x4" Sep 10 13:29:47.712814 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:29:47.712813 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/6c90873d-f3b0-42a4-adc8-6a644a1f2d11-log-socket\") pod \"ovnkube-node-qz8x4\" (UID: \"6c90873d-f3b0-42a4-adc8-6a644a1f2d11\") " pod="openshift-ovn-kubernetes/ovnkube-node-qz8x4" Sep 10 13:29:47.713009 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:29:47.712831 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/6c90873d-f3b0-42a4-adc8-6a644a1f2d11-ovnkube-script-lib\") pod \"ovnkube-node-qz8x4\" (UID: \"6c90873d-f3b0-42a4-adc8-6a644a1f2d11\") " pod="openshift-ovn-kubernetes/ovnkube-node-qz8x4" Sep 10 13:29:47.713009 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:29:47.712854 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/6c90873d-f3b0-42a4-adc8-6a644a1f2d11-node-log\") pod \"ovnkube-node-qz8x4\" (UID: \"6c90873d-f3b0-42a4-adc8-6a644a1f2d11\") " pod="openshift-ovn-kubernetes/ovnkube-node-qz8x4" Sep 10 13:29:47.713009 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:29:47.712874 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/6c90873d-f3b0-42a4-adc8-6a644a1f2d11-host-cni-netd\") pod \"ovnkube-node-qz8x4\" (UID: \"6c90873d-f3b0-42a4-adc8-6a644a1f2d11\") " pod="openshift-ovn-kubernetes/ovnkube-node-qz8x4" Sep 10 13:29:47.713009 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:29:47.712879 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/6c90873d-f3b0-42a4-adc8-6a644a1f2d11-host-run-netns\") pod \"ovnkube-node-qz8x4\" (UID: \"6c90873d-f3b0-42a4-adc8-6a644a1f2d11\") " pod="openshift-ovn-kubernetes/ovnkube-node-qz8x4" Sep 10 13:29:47.713009 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:29:47.712890 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/6c90873d-f3b0-42a4-adc8-6a644a1f2d11-log-socket\") pod \"ovnkube-node-qz8x4\" (UID: \"6c90873d-f3b0-42a4-adc8-6a644a1f2d11\") " pod="openshift-ovn-kubernetes/ovnkube-node-qz8x4" Sep 10 13:29:47.713009 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:29:47.712912 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/6c90873d-f3b0-42a4-adc8-6a644a1f2d11-systemd-units\") pod \"ovnkube-node-qz8x4\" (UID: \"6c90873d-f3b0-42a4-adc8-6a644a1f2d11\") " pod="openshift-ovn-kubernetes/ovnkube-node-qz8x4" Sep 10 13:29:47.713009 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:29:47.712919 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/6c90873d-f3b0-42a4-adc8-6a644a1f2d11-node-log\") pod \"ovnkube-node-qz8x4\" (UID: \"6c90873d-f3b0-42a4-adc8-6a644a1f2d11\") " pod="openshift-ovn-kubernetes/ovnkube-node-qz8x4" Sep 10 13:29:47.713009 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:29:47.712938 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/6c90873d-f3b0-42a4-adc8-6a644a1f2d11-run-openvswitch\") pod \"ovnkube-node-qz8x4\" (UID: \"6c90873d-f3b0-42a4-adc8-6a644a1f2d11\") " pod="openshift-ovn-kubernetes/ovnkube-node-qz8x4" Sep 10 13:29:47.713009 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:29:47.712941 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/6c90873d-f3b0-42a4-adc8-6a644a1f2d11-host-cni-netd\") pod \"ovnkube-node-qz8x4\" (UID: \"6c90873d-f3b0-42a4-adc8-6a644a1f2d11\") " pod="openshift-ovn-kubernetes/ovnkube-node-qz8x4" Sep 10 13:29:47.713009 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:29:47.712968 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/6c90873d-f3b0-42a4-adc8-6a644a1f2d11-systemd-units\") pod \"ovnkube-node-qz8x4\" (UID: \"6c90873d-f3b0-42a4-adc8-6a644a1f2d11\") " pod="openshift-ovn-kubernetes/ovnkube-node-qz8x4" Sep 10 13:29:47.713009 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:29:47.712984 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/6c90873d-f3b0-42a4-adc8-6a644a1f2d11-run-openvswitch\") pod \"ovnkube-node-qz8x4\" (UID: \"6c90873d-f3b0-42a4-adc8-6a644a1f2d11\") " pod="openshift-ovn-kubernetes/ovnkube-node-qz8x4" Sep 10 13:29:47.713009 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:29:47.712964 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/6c90873d-f3b0-42a4-adc8-6a644a1f2d11-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-qz8x4\" (UID: \"6c90873d-f3b0-42a4-adc8-6a644a1f2d11\") " pod="openshift-ovn-kubernetes/ovnkube-node-qz8x4" Sep 10 13:29:47.713484 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:29:47.713026 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/6c90873d-f3b0-42a4-adc8-6a644a1f2d11-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-qz8x4\" (UID: \"6c90873d-f3b0-42a4-adc8-6a644a1f2d11\") " pod="openshift-ovn-kubernetes/ovnkube-node-qz8x4" Sep 10 13:29:47.713484 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:29:47.713051 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/6c90873d-f3b0-42a4-adc8-6a644a1f2d11-host-kubelet\") pod \"ovnkube-node-qz8x4\" (UID: \"6c90873d-f3b0-42a4-adc8-6a644a1f2d11\") " pod="openshift-ovn-kubernetes/ovnkube-node-qz8x4" Sep 10 13:29:47.713484 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:29:47.713082 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/6c90873d-f3b0-42a4-adc8-6a644a1f2d11-run-systemd\") pod \"ovnkube-node-qz8x4\" (UID: \"6c90873d-f3b0-42a4-adc8-6a644a1f2d11\") " pod="openshift-ovn-kubernetes/ovnkube-node-qz8x4" Sep 10 13:29:47.713484 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:29:47.713105 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/6c90873d-f3b0-42a4-adc8-6a644a1f2d11-var-lib-openvswitch\") pod \"ovnkube-node-qz8x4\" (UID: \"6c90873d-f3b0-42a4-adc8-6a644a1f2d11\") " pod="openshift-ovn-kubernetes/ovnkube-node-qz8x4" Sep 10 13:29:47.713484 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:29:47.713133 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/6c90873d-f3b0-42a4-adc8-6a644a1f2d11-host-kubelet\") pod \"ovnkube-node-qz8x4\" (UID: \"6c90873d-f3b0-42a4-adc8-6a644a1f2d11\") " pod="openshift-ovn-kubernetes/ovnkube-node-qz8x4" Sep 10 13:29:47.713484 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:29:47.713147 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/6c90873d-f3b0-42a4-adc8-6a644a1f2d11-run-systemd\") pod \"ovnkube-node-qz8x4\" (UID: \"6c90873d-f3b0-42a4-adc8-6a644a1f2d11\") " pod="openshift-ovn-kubernetes/ovnkube-node-qz8x4" Sep 10 13:29:47.713484 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:29:47.713148 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/6c90873d-f3b0-42a4-adc8-6a644a1f2d11-var-lib-openvswitch\") pod \"ovnkube-node-qz8x4\" (UID: \"6c90873d-f3b0-42a4-adc8-6a644a1f2d11\") " pod="openshift-ovn-kubernetes/ovnkube-node-qz8x4" Sep 10 13:29:47.713484 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:29:47.713167 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/6c90873d-f3b0-42a4-adc8-6a644a1f2d11-run-ovn\") pod \"ovnkube-node-qz8x4\" (UID: \"6c90873d-f3b0-42a4-adc8-6a644a1f2d11\") " pod="openshift-ovn-kubernetes/ovnkube-node-qz8x4" Sep 10 13:29:47.713484 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:29:47.713199 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/6c90873d-f3b0-42a4-adc8-6a644a1f2d11-host-run-ovn-kubernetes\") pod \"ovnkube-node-qz8x4\" (UID: \"6c90873d-f3b0-42a4-adc8-6a644a1f2d11\") " pod="openshift-ovn-kubernetes/ovnkube-node-qz8x4" Sep 10 13:29:47.713484 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:29:47.713223 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/6c90873d-f3b0-42a4-adc8-6a644a1f2d11-host-cni-bin\") pod \"ovnkube-node-qz8x4\" (UID: \"6c90873d-f3b0-42a4-adc8-6a644a1f2d11\") " pod="openshift-ovn-kubernetes/ovnkube-node-qz8x4" Sep 10 13:29:47.713484 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:29:47.713253 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/6c90873d-f3b0-42a4-adc8-6a644a1f2d11-run-ovn\") pod \"ovnkube-node-qz8x4\" (UID: \"6c90873d-f3b0-42a4-adc8-6a644a1f2d11\") " pod="openshift-ovn-kubernetes/ovnkube-node-qz8x4" Sep 10 13:29:47.713484 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:29:47.713262 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/6c90873d-f3b0-42a4-adc8-6a644a1f2d11-env-overrides\") pod \"ovnkube-node-qz8x4\" (UID: \"6c90873d-f3b0-42a4-adc8-6a644a1f2d11\") " pod="openshift-ovn-kubernetes/ovnkube-node-qz8x4" Sep 10 13:29:47.713484 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:29:47.713281 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/6c90873d-f3b0-42a4-adc8-6a644a1f2d11-host-run-ovn-kubernetes\") pod \"ovnkube-node-qz8x4\" (UID: \"6c90873d-f3b0-42a4-adc8-6a644a1f2d11\") " pod="openshift-ovn-kubernetes/ovnkube-node-qz8x4" Sep 10 13:29:47.713484 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:29:47.713288 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bkt4p\" (UniqueName: \"kubernetes.io/projected/6c90873d-f3b0-42a4-adc8-6a644a1f2d11-kube-api-access-bkt4p\") pod \"ovnkube-node-qz8x4\" (UID: \"6c90873d-f3b0-42a4-adc8-6a644a1f2d11\") " pod="openshift-ovn-kubernetes/ovnkube-node-qz8x4" Sep 10 13:29:47.713484 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:29:47.713292 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/6c90873d-f3b0-42a4-adc8-6a644a1f2d11-host-cni-bin\") pod \"ovnkube-node-qz8x4\" (UID: \"6c90873d-f3b0-42a4-adc8-6a644a1f2d11\") " pod="openshift-ovn-kubernetes/ovnkube-node-qz8x4" Sep 10 13:29:47.713484 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:29:47.713315 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/6c90873d-f3b0-42a4-adc8-6a644a1f2d11-etc-openvswitch\") pod \"ovnkube-node-qz8x4\" (UID: \"6c90873d-f3b0-42a4-adc8-6a644a1f2d11\") " pod="openshift-ovn-kubernetes/ovnkube-node-qz8x4" Sep 10 13:29:47.713484 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:29:47.713342 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/6c90873d-f3b0-42a4-adc8-6a644a1f2d11-ovnkube-config\") pod \"ovnkube-node-qz8x4\" (UID: \"6c90873d-f3b0-42a4-adc8-6a644a1f2d11\") " pod="openshift-ovn-kubernetes/ovnkube-node-qz8x4" Sep 10 13:29:47.714021 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:29:47.713365 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/6c90873d-f3b0-42a4-adc8-6a644a1f2d11-ovn-node-metrics-cert\") pod \"ovnkube-node-qz8x4\" (UID: \"6c90873d-f3b0-42a4-adc8-6a644a1f2d11\") " pod="openshift-ovn-kubernetes/ovnkube-node-qz8x4" Sep 10 13:29:47.714021 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:29:47.713403 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/6c90873d-f3b0-42a4-adc8-6a644a1f2d11-etc-openvswitch\") pod \"ovnkube-node-qz8x4\" (UID: \"6c90873d-f3b0-42a4-adc8-6a644a1f2d11\") " pod="openshift-ovn-kubernetes/ovnkube-node-qz8x4" Sep 10 13:29:47.714021 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:29:47.713409 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/6c90873d-f3b0-42a4-adc8-6a644a1f2d11-host-slash\") pod \"ovnkube-node-qz8x4\" (UID: \"6c90873d-f3b0-42a4-adc8-6a644a1f2d11\") " pod="openshift-ovn-kubernetes/ovnkube-node-qz8x4" Sep 10 13:29:47.714021 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:29:47.713447 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/6c90873d-f3b0-42a4-adc8-6a644a1f2d11-host-slash\") pod \"ovnkube-node-qz8x4\" (UID: \"6c90873d-f3b0-42a4-adc8-6a644a1f2d11\") " pod="openshift-ovn-kubernetes/ovnkube-node-qz8x4" Sep 10 13:29:47.714021 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:29:47.713523 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/6c90873d-f3b0-42a4-adc8-6a644a1f2d11-ovnkube-script-lib\") pod \"ovnkube-node-qz8x4\" (UID: \"6c90873d-f3b0-42a4-adc8-6a644a1f2d11\") " pod="openshift-ovn-kubernetes/ovnkube-node-qz8x4" Sep 10 13:29:47.714021 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:29:47.713685 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/6c90873d-f3b0-42a4-adc8-6a644a1f2d11-env-overrides\") pod \"ovnkube-node-qz8x4\" (UID: \"6c90873d-f3b0-42a4-adc8-6a644a1f2d11\") " pod="openshift-ovn-kubernetes/ovnkube-node-qz8x4" Sep 10 13:29:47.714021 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:29:47.713769 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/6c90873d-f3b0-42a4-adc8-6a644a1f2d11-ovnkube-config\") pod \"ovnkube-node-qz8x4\" (UID: \"6c90873d-f3b0-42a4-adc8-6a644a1f2d11\") " pod="openshift-ovn-kubernetes/ovnkube-node-qz8x4" Sep 10 13:29:47.715577 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:29:47.715560 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/6c90873d-f3b0-42a4-adc8-6a644a1f2d11-ovn-node-metrics-cert\") pod \"ovnkube-node-qz8x4\" (UID: \"6c90873d-f3b0-42a4-adc8-6a644a1f2d11\") " pod="openshift-ovn-kubernetes/ovnkube-node-qz8x4" Sep 10 13:29:47.723316 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:29:47.723301 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-bkt4p\" (UniqueName: \"kubernetes.io/projected/6c90873d-f3b0-42a4-adc8-6a644a1f2d11-kube-api-access-bkt4p\") pod \"ovnkube-node-qz8x4\" (UID: \"6c90873d-f3b0-42a4-adc8-6a644a1f2d11\") " pod="openshift-ovn-kubernetes/ovnkube-node-qz8x4" Sep 10 13:29:47.780539 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:29:47.780506 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-qz8x4" Sep 10 13:29:47.812533 ip-10-0-140-8 kubenswrapper[2585]: W0910 13:29:47.812510 2585 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6c90873d_f3b0_42a4_adc8_6a644a1f2d11.slice/crio-12cd691a519d688e5916449a210cd1f4a472f5dc466b666e97984cb5742cd47c WatchSource:0}: Error finding container 12cd691a519d688e5916449a210cd1f4a472f5dc466b666e97984cb5742cd47c: Status 404 returned error can't find the container with id 12cd691a519d688e5916449a210cd1f4a472f5dc466b666e97984cb5742cd47c Sep 10 13:29:47.915357 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:29:47.915337 2585 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-5h2q6_2dac1f99-4132-43bd-8241-687afc22e3d5/ovnkube-controller/0.log" Sep 10 13:29:47.916844 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:29:47.916823 2585 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-5h2q6_2dac1f99-4132-43bd-8241-687afc22e3d5/kube-rbac-proxy-ovn-metrics/0.log" Sep 10 13:29:47.917258 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:29:47.917230 2585 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-5h2q6_2dac1f99-4132-43bd-8241-687afc22e3d5/kube-rbac-proxy-node/0.log" Sep 10 13:29:47.917672 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:29:47.917653 2585 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-5h2q6_2dac1f99-4132-43bd-8241-687afc22e3d5/ovn-acl-logging/0.log" Sep 10 13:29:47.918070 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:29:47.918056 2585 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-5h2q6_2dac1f99-4132-43bd-8241-687afc22e3d5/ovn-controller/0.log" Sep 10 13:29:47.918175 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:29:47.918149 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-5h2q6" event={"ID":"2dac1f99-4132-43bd-8241-687afc22e3d5","Type":"ContainerDied","Data":"7ea5e1ed0662fddce1d31abbb204e24bd3e52206ae0f4efdbc675bf009af397c"} Sep 10 13:29:47.918298 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:29:47.918198 2585 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-5h2q6" Sep 10 13:29:47.918298 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:29:47.918277 2585 scope.go:117] "RemoveContainer" containerID="598d801d63dd69008d0ed8a7f14e45bf62ace20c108b3a2ef18981cfc9e7f468" Sep 10 13:29:47.919901 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:29:47.919876 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-qz8x4" event={"ID":"6c90873d-f3b0-42a4-adc8-6a644a1f2d11","Type":"ContainerStarted","Data":"12cd691a519d688e5916449a210cd1f4a472f5dc466b666e97984cb5742cd47c"} Sep 10 13:29:47.928686 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:29:47.928666 2585 scope.go:117] "RemoveContainer" containerID="8c3fecf46432f5f9b54b76df43d6a5864f171a8f3d13575cf2efb89aadc1727f" Sep 10 13:29:47.942313 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:29:47.942292 2585 scope.go:117] "RemoveContainer" containerID="b3d1394536749287777f28bad78fbcf1374a445f7ba2642616829fe616010c39" Sep 10 13:29:47.944463 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:29:47.944446 2585 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ovn-kubernetes/ovnkube-node-5h2q6"] Sep 10 13:29:47.947849 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:29:47.947831 2585 scope.go:117] "RemoveContainer" containerID="bd0cccb67dfce305af8bdbda97df16424208d8203d6d87b9ee6d403ff1a4746a" Sep 10 13:29:47.950973 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:29:47.950950 2585 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ovn-kubernetes/ovnkube-node-5h2q6"] Sep 10 13:29:47.953701 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:29:47.953659 2585 scope.go:117] "RemoveContainer" containerID="8e502b5a84a7273f20a4e2a3b2eada301addbd0a3fc1adb7e64f97b6e0486899" Sep 10 13:29:47.959234 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:29:47.959219 2585 scope.go:117] "RemoveContainer" containerID="6bba35d62cccbf1a59ad491e45c756ef03a933c89e9d9598e4f112ece97084e1" Sep 10 13:29:47.964725 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:29:47.964705 2585 scope.go:117] "RemoveContainer" containerID="ac85c413de3c568c5a255d54c91df74202333b9a023eab23788cf5c1d72d2144" Sep 10 13:29:47.970151 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:29:47.970136 2585 scope.go:117] "RemoveContainer" containerID="118b0da0ab94fb196925adea90b54a345529eaca4e0a0d8bc14acc25b96cc5a5" Sep 10 13:29:48.925149 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:29:48.925108 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-qz8x4" event={"ID":"6c90873d-f3b0-42a4-adc8-6a644a1f2d11","Type":"ContainerStarted","Data":"8cddc01813b870b6b8e109ef597cdffca5be4688abdee9f76bb5d474047f941d"} Sep 10 13:29:48.925639 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:29:48.925224 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-qz8x4" event={"ID":"6c90873d-f3b0-42a4-adc8-6a644a1f2d11","Type":"ContainerStarted","Data":"cfe47d108cd4ecfa51d7251a193a86d9923500ddd1146b90abf118a86c181d36"} Sep 10 13:29:48.925639 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:29:48.925296 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-qz8x4" event={"ID":"6c90873d-f3b0-42a4-adc8-6a644a1f2d11","Type":"ContainerStarted","Data":"6fecce83a7d3c5daf0b7d39e01316657f6b73722418cd07052b4d11216a8d7f4"} Sep 10 13:29:48.925639 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:29:48.925348 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-qz8x4" event={"ID":"6c90873d-f3b0-42a4-adc8-6a644a1f2d11","Type":"ContainerStarted","Data":"00b70c40fe32c4846207ef52abb63f9b734fe2ecbae19933bd6f5d558af27168"} Sep 10 13:29:48.925639 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:29:48.925395 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-qz8x4" event={"ID":"6c90873d-f3b0-42a4-adc8-6a644a1f2d11","Type":"ContainerStarted","Data":"ff7bd71833f7ef0c0130ad64404d20c6ca00f36951c32482c21e238433466597"} Sep 10 13:29:48.925639 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:29:48.925439 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-qz8x4" event={"ID":"6c90873d-f3b0-42a4-adc8-6a644a1f2d11","Type":"ContainerStarted","Data":"f220f253896586639e68df2bd65beaf2db8d8529f1de8ee8cef607b39e6f318a"} Sep 10 13:29:49.641371 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:29:49.641335 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-xp5dk" Sep 10 13:29:49.641553 ip-10-0-140-8 kubenswrapper[2585]: E0910 13:29:49.641447 2585 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-xp5dk" podUID="d552154f-1d5b-4c5d-99bf-7aeef22ce032" Sep 10 13:29:49.641553 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:29:49.641505 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-7262t" Sep 10 13:29:49.641679 ip-10-0-140-8 kubenswrapper[2585]: E0910 13:29:49.641615 2585 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-7262t" podUID="082a7577-a14f-49fb-a9b1-3cf250e6735a" Sep 10 13:29:49.644273 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:29:49.644232 2585 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="2dac1f99-4132-43bd-8241-687afc22e3d5" path="/var/lib/kubelet/pods/2dac1f99-4132-43bd-8241-687afc22e3d5/volumes" Sep 10 13:29:50.933622 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:29:50.933587 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-qz8x4" event={"ID":"6c90873d-f3b0-42a4-adc8-6a644a1f2d11","Type":"ContainerStarted","Data":"1bc3572bfb30ef26a0658fb292d9bcada48f24744d6e4a60bf210f3615740484"} Sep 10 13:29:51.640745 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:29:51.640712 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-xp5dk" Sep 10 13:29:51.640745 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:29:51.640749 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-7262t" Sep 10 13:29:51.640949 ip-10-0-140-8 kubenswrapper[2585]: E0910 13:29:51.640834 2585 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-xp5dk" podUID="d552154f-1d5b-4c5d-99bf-7aeef22ce032" Sep 10 13:29:51.640986 ip-10-0-140-8 kubenswrapper[2585]: E0910 13:29:51.640950 2585 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-7262t" podUID="082a7577-a14f-49fb-a9b1-3cf250e6735a" Sep 10 13:29:52.944557 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:29:52.944523 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-qz8x4" event={"ID":"6c90873d-f3b0-42a4-adc8-6a644a1f2d11","Type":"ContainerStarted","Data":"bd8e0dcb9c125c60434e581289162c0af543f3b548c5989d3f1869211004fbf7"} Sep 10 13:29:52.945003 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:29:52.944790 2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-qz8x4" Sep 10 13:29:52.959661 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:29:52.959635 2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-qz8x4" Sep 10 13:29:52.969554 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:29:52.969474 2585 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ovn-kubernetes/ovnkube-node-qz8x4" podStartSLOduration=5.969459141 podStartE2EDuration="5.969459141s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-10 13:29:47 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 13:29:52.96921415 +0000 UTC m=+77.841918351" watchObservedRunningTime="2026-09-10 13:29:52.969459141 +0000 UTC m=+77.842163341" Sep 10 13:29:53.641656 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:29:53.641624 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-xp5dk" Sep 10 13:29:53.641801 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:29:53.641624 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-7262t" Sep 10 13:29:53.641801 ip-10-0-140-8 kubenswrapper[2585]: E0910 13:29:53.641730 2585 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-xp5dk" podUID="d552154f-1d5b-4c5d-99bf-7aeef22ce032" Sep 10 13:29:53.641891 ip-10-0-140-8 kubenswrapper[2585]: E0910 13:29:53.641799 2585 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-7262t" podUID="082a7577-a14f-49fb-a9b1-3cf250e6735a" Sep 10 13:29:53.946847 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:29:53.946755 2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-qz8x4" Sep 10 13:29:53.946847 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:29:53.946829 2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-qz8x4" Sep 10 13:29:53.960804 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:29:53.960781 2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-qz8x4" Sep 10 13:29:55.642129 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:29:55.642099 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-xp5dk" Sep 10 13:29:55.642570 ip-10-0-140-8 kubenswrapper[2585]: E0910 13:29:55.642188 2585 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-xp5dk" podUID="d552154f-1d5b-4c5d-99bf-7aeef22ce032" Sep 10 13:29:55.642570 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:29:55.642290 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-7262t" Sep 10 13:29:55.642570 ip-10-0-140-8 kubenswrapper[2585]: E0910 13:29:55.642388 2585 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-7262t" podUID="082a7577-a14f-49fb-a9b1-3cf250e6735a" Sep 10 13:29:57.193063 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:29:57.193027 2585 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-dns/node-resolver-84qhc"] Sep 10 13:29:57.195662 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:29:57.195641 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-84qhc" Sep 10 13:29:57.198025 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:29:57.198002 2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\"" Sep 10 13:29:57.198123 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:29:57.198077 2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\"" Sep 10 13:29:57.199049 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:29:57.199031 2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-glzgh\"" Sep 10 13:29:57.272480 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:29:57.272449 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/57474dbb-5547-4132-9d67-dacfb20ea367-tmp-dir\") pod \"node-resolver-84qhc\" (UID: \"57474dbb-5547-4132-9d67-dacfb20ea367\") " pod="openshift-dns/node-resolver-84qhc" Sep 10 13:29:57.272645 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:29:57.272493 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/57474dbb-5547-4132-9d67-dacfb20ea367-hosts-file\") pod \"node-resolver-84qhc\" (UID: \"57474dbb-5547-4132-9d67-dacfb20ea367\") " pod="openshift-dns/node-resolver-84qhc" Sep 10 13:29:57.272645 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:29:57.272524 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2rkrp\" (UniqueName: \"kubernetes.io/projected/57474dbb-5547-4132-9d67-dacfb20ea367-kube-api-access-2rkrp\") pod \"node-resolver-84qhc\" (UID: \"57474dbb-5547-4132-9d67-dacfb20ea367\") " pod="openshift-dns/node-resolver-84qhc" Sep 10 13:29:57.373822 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:29:57.373725 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/57474dbb-5547-4132-9d67-dacfb20ea367-hosts-file\") pod \"node-resolver-84qhc\" (UID: \"57474dbb-5547-4132-9d67-dacfb20ea367\") " pod="openshift-dns/node-resolver-84qhc" Sep 10 13:29:57.373822 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:29:57.373782 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2rkrp\" (UniqueName: \"kubernetes.io/projected/57474dbb-5547-4132-9d67-dacfb20ea367-kube-api-access-2rkrp\") pod \"node-resolver-84qhc\" (UID: \"57474dbb-5547-4132-9d67-dacfb20ea367\") " pod="openshift-dns/node-resolver-84qhc" Sep 10 13:29:57.374101 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:29:57.373876 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/57474dbb-5547-4132-9d67-dacfb20ea367-hosts-file\") pod \"node-resolver-84qhc\" (UID: \"57474dbb-5547-4132-9d67-dacfb20ea367\") " pod="openshift-dns/node-resolver-84qhc" Sep 10 13:29:57.374101 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:29:57.373941 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/57474dbb-5547-4132-9d67-dacfb20ea367-tmp-dir\") pod \"node-resolver-84qhc\" (UID: \"57474dbb-5547-4132-9d67-dacfb20ea367\") " pod="openshift-dns/node-resolver-84qhc" Sep 10 13:29:57.374334 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:29:57.374312 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/57474dbb-5547-4132-9d67-dacfb20ea367-tmp-dir\") pod \"node-resolver-84qhc\" (UID: \"57474dbb-5547-4132-9d67-dacfb20ea367\") " pod="openshift-dns/node-resolver-84qhc" Sep 10 13:29:57.383203 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:29:57.383176 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2rkrp\" (UniqueName: \"kubernetes.io/projected/57474dbb-5547-4132-9d67-dacfb20ea367-kube-api-access-2rkrp\") pod \"node-resolver-84qhc\" (UID: \"57474dbb-5547-4132-9d67-dacfb20ea367\") " pod="openshift-dns/node-resolver-84qhc" Sep 10 13:29:57.504563 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:29:57.504537 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-84qhc" Sep 10 13:29:57.514061 ip-10-0-140-8 kubenswrapper[2585]: W0910 13:29:57.514035 2585 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod57474dbb_5547_4132_9d67_dacfb20ea367.slice/crio-83a4dee907126d4729a4858b50ddf304132a6de2738895d097a04f921038a47a WatchSource:0}: Error finding container 83a4dee907126d4729a4858b50ddf304132a6de2738895d097a04f921038a47a: Status 404 returned error can't find the container with id 83a4dee907126d4729a4858b50ddf304132a6de2738895d097a04f921038a47a Sep 10 13:29:57.641304 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:29:57.641046 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-xp5dk" Sep 10 13:29:57.641304 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:29:57.641058 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-7262t" Sep 10 13:29:57.641508 ip-10-0-140-8 kubenswrapper[2585]: E0910 13:29:57.641368 2585 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-xp5dk" podUID="d552154f-1d5b-4c5d-99bf-7aeef22ce032" Sep 10 13:29:57.641508 ip-10-0-140-8 kubenswrapper[2585]: E0910 13:29:57.641459 2585 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-7262t" podUID="082a7577-a14f-49fb-a9b1-3cf250e6735a" Sep 10 13:29:57.957211 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:29:57.957080 2585 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-l9l8q_ad555c6a-4385-4f8e-93bb-a174cfaa8e74/kube-multus/0.log" Sep 10 13:29:57.957211 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:29:57.957115 2585 generic.go:356] "Generic (PLEG): container finished" podID="ad555c6a-4385-4f8e-93bb-a174cfaa8e74" containerID="9c82c71b605c1b995ae44d736d4e81b9fb104099b45514b83cb72952bcbfc47f" exitCode=1 Sep 10 13:29:57.957211 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:29:57.957189 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-l9l8q" event={"ID":"ad555c6a-4385-4f8e-93bb-a174cfaa8e74","Type":"ContainerDied","Data":"9c82c71b605c1b995ae44d736d4e81b9fb104099b45514b83cb72952bcbfc47f"} Sep 10 13:29:57.957593 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:29:57.957574 2585 scope.go:117] "RemoveContainer" containerID="9c82c71b605c1b995ae44d736d4e81b9fb104099b45514b83cb72952bcbfc47f" Sep 10 13:29:57.958833 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:29:57.958802 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-84qhc" event={"ID":"57474dbb-5547-4132-9d67-dacfb20ea367","Type":"ContainerStarted","Data":"dae56df4c033130566d5ce81b620303e823b112533f8942cb93eedbc530c18b3"} Sep 10 13:29:57.958884 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:29:57.958848 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-84qhc" event={"ID":"57474dbb-5547-4132-9d67-dacfb20ea367","Type":"ContainerStarted","Data":"83a4dee907126d4729a4858b50ddf304132a6de2738895d097a04f921038a47a"} Sep 10 13:29:57.985016 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:29:57.984972 2585 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/node-resolver-84qhc" podStartSLOduration=0.984957812 podStartE2EDuration="984.957812ms" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-10 13:29:57 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 13:29:57.984751646 +0000 UTC m=+82.857455846" watchObservedRunningTime="2026-09-10 13:29:57.984957812 +0000 UTC m=+82.857662010" Sep 10 13:29:58.962668 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:29:58.962640 2585 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-l9l8q_ad555c6a-4385-4f8e-93bb-a174cfaa8e74/kube-multus/0.log" Sep 10 13:29:58.963040 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:29:58.962710 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-l9l8q" event={"ID":"ad555c6a-4385-4f8e-93bb-a174cfaa8e74","Type":"ContainerStarted","Data":"24d6d79549538f1d536272aee5a52d89c1116efcd876fcb7e0148947fd754dd3"} Sep 10 13:29:59.486423 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:29:59.486382 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d552154f-1d5b-4c5d-99bf-7aeef22ce032-metrics-certs\") pod \"network-metrics-daemon-xp5dk\" (UID: \"d552154f-1d5b-4c5d-99bf-7aeef22ce032\") " pod="openshift-multus/network-metrics-daemon-xp5dk" Sep 10 13:29:59.486576 ip-10-0-140-8 kubenswrapper[2585]: E0910 13:29:59.486493 2585 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Sep 10 13:29:59.486576 ip-10-0-140-8 kubenswrapper[2585]: E0910 13:29:59.486541 2585 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d552154f-1d5b-4c5d-99bf-7aeef22ce032-metrics-certs podName:d552154f-1d5b-4c5d-99bf-7aeef22ce032 nodeName:}" failed. No retries permitted until 2026-09-10 13:31:03.486528383 +0000 UTC m=+148.359232561 (durationBeforeRetry 1m4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/d552154f-1d5b-4c5d-99bf-7aeef22ce032-metrics-certs") pod "network-metrics-daemon-xp5dk" (UID: "d552154f-1d5b-4c5d-99bf-7aeef22ce032") : object "openshift-multus"/"metrics-daemon-secret" not registered Sep 10 13:29:59.586679 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:29:59.586641 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x2vv5\" (UniqueName: \"kubernetes.io/projected/082a7577-a14f-49fb-a9b1-3cf250e6735a-kube-api-access-x2vv5\") pod \"network-check-target-7262t\" (UID: \"082a7577-a14f-49fb-a9b1-3cf250e6735a\") " pod="openshift-network-diagnostics/network-check-target-7262t" Sep 10 13:29:59.586880 ip-10-0-140-8 kubenswrapper[2585]: E0910 13:29:59.586778 2585 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Sep 10 13:29:59.586880 ip-10-0-140-8 kubenswrapper[2585]: E0910 13:29:59.586792 2585 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Sep 10 13:29:59.586880 ip-10-0-140-8 kubenswrapper[2585]: E0910 13:29:59.586803 2585 projected.go:196] Error preparing data for projected volume kube-api-access-x2vv5 for pod openshift-network-diagnostics/network-check-target-7262t: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Sep 10 13:29:59.586880 ip-10-0-140-8 kubenswrapper[2585]: E0910 13:29:59.586847 2585 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/082a7577-a14f-49fb-a9b1-3cf250e6735a-kube-api-access-x2vv5 podName:082a7577-a14f-49fb-a9b1-3cf250e6735a nodeName:}" failed. No retries permitted until 2026-09-10 13:31:03.586834685 +0000 UTC m=+148.459538864 (durationBeforeRetry 1m4s). Error: MountVolume.SetUp failed for volume "kube-api-access-x2vv5" (UniqueName: "kubernetes.io/projected/082a7577-a14f-49fb-a9b1-3cf250e6735a-kube-api-access-x2vv5") pod "network-check-target-7262t" (UID: "082a7577-a14f-49fb-a9b1-3cf250e6735a") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Sep 10 13:29:59.640889 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:29:59.640858 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-xp5dk" Sep 10 13:29:59.640889 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:29:59.640866 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-7262t" Sep 10 13:29:59.641053 ip-10-0-140-8 kubenswrapper[2585]: E0910 13:29:59.640961 2585 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-xp5dk" podUID="d552154f-1d5b-4c5d-99bf-7aeef22ce032" Sep 10 13:29:59.641151 ip-10-0-140-8 kubenswrapper[2585]: E0910 13:29:59.641086 2585 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-7262t" podUID="082a7577-a14f-49fb-a9b1-3cf250e6735a" Sep 10 13:30:01.641593 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:30:01.641554 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-xp5dk" Sep 10 13:30:01.642050 ip-10-0-140-8 kubenswrapper[2585]: E0910 13:30:01.641708 2585 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-xp5dk" podUID="d552154f-1d5b-4c5d-99bf-7aeef22ce032" Sep 10 13:30:01.642050 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:30:01.641757 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-7262t" Sep 10 13:30:01.642050 ip-10-0-140-8 kubenswrapper[2585]: E0910 13:30:01.641844 2585 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-7262t" podUID="082a7577-a14f-49fb-a9b1-3cf250e6735a" Sep 10 13:30:03.641443 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:30:03.641404 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-7262t" Sep 10 13:30:03.641839 ip-10-0-140-8 kubenswrapper[2585]: E0910 13:30:03.641516 2585 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-7262t" podUID="082a7577-a14f-49fb-a9b1-3cf250e6735a" Sep 10 13:30:03.641839 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:30:03.641571 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-xp5dk" Sep 10 13:30:03.641839 ip-10-0-140-8 kubenswrapper[2585]: E0910 13:30:03.641673 2585 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-xp5dk" podUID="d552154f-1d5b-4c5d-99bf-7aeef22ce032" Sep 10 13:30:05.642259 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:30:05.642222 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-7262t" Sep 10 13:30:05.642725 ip-10-0-140-8 kubenswrapper[2585]: E0910 13:30:05.642385 2585 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-7262t" podUID="082a7577-a14f-49fb-a9b1-3cf250e6735a" Sep 10 13:30:05.642725 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:30:05.642423 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-xp5dk" Sep 10 13:30:05.642725 ip-10-0-140-8 kubenswrapper[2585]: E0910 13:30:05.642520 2585 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-xp5dk" podUID="d552154f-1d5b-4c5d-99bf-7aeef22ce032" Sep 10 13:30:07.640673 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:30:07.640644 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-xp5dk" Sep 10 13:30:07.641138 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:30:07.640644 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-7262t" Sep 10 13:30:07.641138 ip-10-0-140-8 kubenswrapper[2585]: E0910 13:30:07.640744 2585 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-xp5dk" podUID="d552154f-1d5b-4c5d-99bf-7aeef22ce032" Sep 10 13:30:07.641138 ip-10-0-140-8 kubenswrapper[2585]: E0910 13:30:07.640808 2585 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-7262t" podUID="082a7577-a14f-49fb-a9b1-3cf250e6735a" Sep 10 13:30:09.644106 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:30:09.644077 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-7262t" Sep 10 13:30:09.644496 ip-10-0-140-8 kubenswrapper[2585]: E0910 13:30:09.644167 2585 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-7262t" podUID="082a7577-a14f-49fb-a9b1-3cf250e6735a" Sep 10 13:30:09.644496 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:30:09.644213 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-xp5dk" Sep 10 13:30:09.644496 ip-10-0-140-8 kubenswrapper[2585]: E0910 13:30:09.644290 2585 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-xp5dk" podUID="d552154f-1d5b-4c5d-99bf-7aeef22ce032" Sep 10 13:30:11.641613 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:30:11.641576 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-7262t" Sep 10 13:30:11.641613 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:30:11.641597 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-xp5dk" Sep 10 13:30:11.642101 ip-10-0-140-8 kubenswrapper[2585]: E0910 13:30:11.641671 2585 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-7262t" podUID="082a7577-a14f-49fb-a9b1-3cf250e6735a" Sep 10 13:30:11.642101 ip-10-0-140-8 kubenswrapper[2585]: E0910 13:30:11.641821 2585 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-xp5dk" podUID="d552154f-1d5b-4c5d-99bf-7aeef22ce032" Sep 10 13:30:13.640945 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:30:13.640911 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-7262t" Sep 10 13:30:13.641510 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:30:13.640912 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-xp5dk" Sep 10 13:30:13.641510 ip-10-0-140-8 kubenswrapper[2585]: E0910 13:30:13.641034 2585 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-7262t" podUID="082a7577-a14f-49fb-a9b1-3cf250e6735a" Sep 10 13:30:13.641510 ip-10-0-140-8 kubenswrapper[2585]: E0910 13:30:13.641084 2585 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-xp5dk" podUID="d552154f-1d5b-4c5d-99bf-7aeef22ce032" Sep 10 13:30:15.641826 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:30:15.641796 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-7262t" Sep 10 13:30:15.642182 ip-10-0-140-8 kubenswrapper[2585]: E0910 13:30:15.641905 2585 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-7262t" podUID="082a7577-a14f-49fb-a9b1-3cf250e6735a" Sep 10 13:30:15.642182 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:30:15.641955 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-xp5dk" Sep 10 13:30:15.642182 ip-10-0-140-8 kubenswrapper[2585]: E0910 13:30:15.642052 2585 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-xp5dk" podUID="d552154f-1d5b-4c5d-99bf-7aeef22ce032" Sep 10 13:30:17.641343 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:30:17.641313 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-7262t" Sep 10 13:30:17.641700 ip-10-0-140-8 kubenswrapper[2585]: E0910 13:30:17.641414 2585 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-7262t" podUID="082a7577-a14f-49fb-a9b1-3cf250e6735a" Sep 10 13:30:17.641700 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:30:17.641513 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-xp5dk" Sep 10 13:30:17.641700 ip-10-0-140-8 kubenswrapper[2585]: E0910 13:30:17.641635 2585 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-xp5dk" podUID="d552154f-1d5b-4c5d-99bf-7aeef22ce032" Sep 10 13:30:19.641089 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:30:19.641048 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-xp5dk" Sep 10 13:30:19.641089 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:30:19.641048 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-7262t" Sep 10 13:30:19.641591 ip-10-0-140-8 kubenswrapper[2585]: E0910 13:30:19.641367 2585 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-xp5dk" podUID="d552154f-1d5b-4c5d-99bf-7aeef22ce032" Sep 10 13:30:19.641591 ip-10-0-140-8 kubenswrapper[2585]: E0910 13:30:19.641490 2585 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-7262t" podUID="082a7577-a14f-49fb-a9b1-3cf250e6735a" Sep 10 13:30:21.641321 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:30:21.641291 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-7262t" Sep 10 13:30:21.641321 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:30:21.641317 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-xp5dk" Sep 10 13:30:21.641789 ip-10-0-140-8 kubenswrapper[2585]: E0910 13:30:21.641382 2585 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-7262t" podUID="082a7577-a14f-49fb-a9b1-3cf250e6735a" Sep 10 13:30:21.641789 ip-10-0-140-8 kubenswrapper[2585]: E0910 13:30:21.641519 2585 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-xp5dk" podUID="d552154f-1d5b-4c5d-99bf-7aeef22ce032" Sep 10 13:30:23.641393 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:30:23.641357 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-xp5dk" Sep 10 13:30:23.641784 ip-10-0-140-8 kubenswrapper[2585]: E0910 13:30:23.641509 2585 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-xp5dk" podUID="d552154f-1d5b-4c5d-99bf-7aeef22ce032" Sep 10 13:30:23.641784 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:30:23.641553 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-7262t" Sep 10 13:30:23.641784 ip-10-0-140-8 kubenswrapper[2585]: E0910 13:30:23.641637 2585 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-7262t" podUID="082a7577-a14f-49fb-a9b1-3cf250e6735a" Sep 10 13:30:25.641370 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:30:25.641336 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-7262t" Sep 10 13:30:25.641769 ip-10-0-140-8 kubenswrapper[2585]: E0910 13:30:25.641441 2585 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-7262t" podUID="082a7577-a14f-49fb-a9b1-3cf250e6735a" Sep 10 13:30:25.641769 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:30:25.641479 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-xp5dk" Sep 10 13:30:25.641769 ip-10-0-140-8 kubenswrapper[2585]: E0910 13:30:25.641556 2585 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-xp5dk" podUID="d552154f-1d5b-4c5d-99bf-7aeef22ce032" Sep 10 13:30:25.961239 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:30:25.961167 2585 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ovn-kubernetes/ovnkube-node-qz8x4" podUID="6c90873d-f3b0-42a4-adc8-6a644a1f2d11" containerName="ovnkube-controller" probeResult="failure" output="" Sep 10 13:30:27.640918 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:30:27.640887 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-7262t" Sep 10 13:30:27.641404 ip-10-0-140-8 kubenswrapper[2585]: E0910 13:30:27.640984 2585 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-7262t" podUID="082a7577-a14f-49fb-a9b1-3cf250e6735a" Sep 10 13:30:27.641404 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:30:27.641024 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-xp5dk" Sep 10 13:30:27.641404 ip-10-0-140-8 kubenswrapper[2585]: E0910 13:30:27.641085 2585 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-xp5dk" podUID="d552154f-1d5b-4c5d-99bf-7aeef22ce032" Sep 10 13:30:29.640842 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:30:29.640810 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-7262t" Sep 10 13:30:29.641276 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:30:29.640817 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-xp5dk" Sep 10 13:30:29.641276 ip-10-0-140-8 kubenswrapper[2585]: E0910 13:30:29.640920 2585 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-7262t" podUID="082a7577-a14f-49fb-a9b1-3cf250e6735a" Sep 10 13:30:29.641276 ip-10-0-140-8 kubenswrapper[2585]: E0910 13:30:29.641007 2585 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-xp5dk" podUID="d552154f-1d5b-4c5d-99bf-7aeef22ce032" Sep 10 13:30:31.641170 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:30:31.641132 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-xp5dk" Sep 10 13:30:31.641575 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:30:31.641132 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-7262t" Sep 10 13:30:31.641575 ip-10-0-140-8 kubenswrapper[2585]: E0910 13:30:31.641255 2585 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-xp5dk" podUID="d552154f-1d5b-4c5d-99bf-7aeef22ce032" Sep 10 13:30:31.641575 ip-10-0-140-8 kubenswrapper[2585]: E0910 13:30:31.641318 2585 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-7262t" podUID="082a7577-a14f-49fb-a9b1-3cf250e6735a" Sep 10 13:30:33.641445 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:30:33.641405 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-7262t" Sep 10 13:30:33.641830 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:30:33.641416 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-xp5dk" Sep 10 13:30:33.641830 ip-10-0-140-8 kubenswrapper[2585]: E0910 13:30:33.641518 2585 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-7262t" podUID="082a7577-a14f-49fb-a9b1-3cf250e6735a" Sep 10 13:30:33.641830 ip-10-0-140-8 kubenswrapper[2585]: E0910 13:30:33.641615 2585 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-xp5dk" podUID="d552154f-1d5b-4c5d-99bf-7aeef22ce032" Sep 10 13:30:35.637839 ip-10-0-140-8 kubenswrapper[2585]: E0910 13:30:35.637808 2585 kubelet_node_status.go:448] "Node not becoming ready in time after startup" Sep 10 13:30:35.641807 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:30:35.641779 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-xp5dk" Sep 10 13:30:35.641940 ip-10-0-140-8 kubenswrapper[2585]: E0910 13:30:35.641891 2585 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-xp5dk" podUID="d552154f-1d5b-4c5d-99bf-7aeef22ce032" Sep 10 13:30:35.642002 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:30:35.641936 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-7262t" Sep 10 13:30:35.642057 ip-10-0-140-8 kubenswrapper[2585]: E0910 13:30:35.642027 2585 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-7262t" podUID="082a7577-a14f-49fb-a9b1-3cf250e6735a" Sep 10 13:30:37.643616 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:30:37.643583 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-xp5dk" Sep 10 13:30:37.643993 ip-10-0-140-8 kubenswrapper[2585]: E0910 13:30:37.643682 2585 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-xp5dk" podUID="d552154f-1d5b-4c5d-99bf-7aeef22ce032" Sep 10 13:30:37.643993 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:30:37.643758 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-7262t" Sep 10 13:30:37.643993 ip-10-0-140-8 kubenswrapper[2585]: E0910 13:30:37.643835 2585 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-7262t" podUID="082a7577-a14f-49fb-a9b1-3cf250e6735a" Sep 10 13:30:39.643521 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:30:39.643486 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-xp5dk" Sep 10 13:30:39.643963 ip-10-0-140-8 kubenswrapper[2585]: E0910 13:30:39.643612 2585 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-xp5dk" podUID="d552154f-1d5b-4c5d-99bf-7aeef22ce032" Sep 10 13:30:39.643963 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:30:39.643652 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-7262t" Sep 10 13:30:39.643963 ip-10-0-140-8 kubenswrapper[2585]: E0910 13:30:39.643750 2585 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-7262t" podUID="082a7577-a14f-49fb-a9b1-3cf250e6735a" Sep 10 13:30:40.631728 ip-10-0-140-8 kubenswrapper[2585]: E0910 13:30:40.631683 2585 kubelet.go:3032] "Container runtime network not ready" networkReady="NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" Sep 10 13:30:41.641053 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:30:41.641010 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-7262t" Sep 10 13:30:41.641486 ip-10-0-140-8 kubenswrapper[2585]: E0910 13:30:41.641150 2585 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-7262t" podUID="082a7577-a14f-49fb-a9b1-3cf250e6735a" Sep 10 13:30:41.641486 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:30:41.641193 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-xp5dk" Sep 10 13:30:41.641486 ip-10-0-140-8 kubenswrapper[2585]: E0910 13:30:41.641315 2585 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-xp5dk" podUID="d552154f-1d5b-4c5d-99bf-7aeef22ce032" Sep 10 13:30:43.640980 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:30:43.640946 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-xp5dk" Sep 10 13:30:43.641462 ip-10-0-140-8 kubenswrapper[2585]: E0910 13:30:43.641061 2585 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-xp5dk" podUID="d552154f-1d5b-4c5d-99bf-7aeef22ce032" Sep 10 13:30:43.641462 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:30:43.641113 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-7262t" Sep 10 13:30:43.641462 ip-10-0-140-8 kubenswrapper[2585]: E0910 13:30:43.641204 2585 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-7262t" podUID="082a7577-a14f-49fb-a9b1-3cf250e6735a" Sep 10 13:30:44.062146 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:30:44.062120 2585 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-l9l8q_ad555c6a-4385-4f8e-93bb-a174cfaa8e74/kube-multus/1.log" Sep 10 13:30:44.062598 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:30:44.062582 2585 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-l9l8q_ad555c6a-4385-4f8e-93bb-a174cfaa8e74/kube-multus/0.log" Sep 10 13:30:44.062659 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:30:44.062615 2585 generic.go:356] "Generic (PLEG): container finished" podID="ad555c6a-4385-4f8e-93bb-a174cfaa8e74" containerID="24d6d79549538f1d536272aee5a52d89c1116efcd876fcb7e0148947fd754dd3" exitCode=1 Sep 10 13:30:44.062699 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:30:44.062653 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-l9l8q" event={"ID":"ad555c6a-4385-4f8e-93bb-a174cfaa8e74","Type":"ContainerDied","Data":"24d6d79549538f1d536272aee5a52d89c1116efcd876fcb7e0148947fd754dd3"} Sep 10 13:30:44.062733 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:30:44.062704 2585 scope.go:117] "RemoveContainer" containerID="9c82c71b605c1b995ae44d736d4e81b9fb104099b45514b83cb72952bcbfc47f" Sep 10 13:30:44.063073 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:30:44.063053 2585 scope.go:117] "RemoveContainer" containerID="24d6d79549538f1d536272aee5a52d89c1116efcd876fcb7e0148947fd754dd3" Sep 10 13:30:44.063279 ip-10-0-140-8 kubenswrapper[2585]: E0910 13:30:44.063238 2585 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"kube-multus\" with CrashLoopBackOff: \"back-off 10s restarting failed container=kube-multus pod=multus-l9l8q_openshift-multus(ad555c6a-4385-4f8e-93bb-a174cfaa8e74)\"" pod="openshift-multus/multus-l9l8q" podUID="ad555c6a-4385-4f8e-93bb-a174cfaa8e74" Sep 10 13:30:45.066532 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:30:45.066507 2585 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-l9l8q_ad555c6a-4385-4f8e-93bb-a174cfaa8e74/kube-multus/1.log" Sep 10 13:30:45.631988 ip-10-0-140-8 kubenswrapper[2585]: E0910 13:30:45.631957 2585 kubelet.go:3032] "Container runtime network not ready" networkReady="NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" Sep 10 13:30:45.641818 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:30:45.641791 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-7262t" Sep 10 13:30:45.641967 ip-10-0-140-8 kubenswrapper[2585]: E0910 13:30:45.641883 2585 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-7262t" podUID="082a7577-a14f-49fb-a9b1-3cf250e6735a" Sep 10 13:30:45.641967 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:30:45.641954 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-xp5dk" Sep 10 13:30:45.642086 ip-10-0-140-8 kubenswrapper[2585]: E0910 13:30:45.642040 2585 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-xp5dk" podUID="d552154f-1d5b-4c5d-99bf-7aeef22ce032" Sep 10 13:30:47.641715 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:30:47.641675 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-7262t" Sep 10 13:30:47.642162 ip-10-0-140-8 kubenswrapper[2585]: E0910 13:30:47.641807 2585 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-7262t" podUID="082a7577-a14f-49fb-a9b1-3cf250e6735a" Sep 10 13:30:47.642162 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:30:47.641863 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-xp5dk" Sep 10 13:30:47.642162 ip-10-0-140-8 kubenswrapper[2585]: E0910 13:30:47.641992 2585 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-xp5dk" podUID="d552154f-1d5b-4c5d-99bf-7aeef22ce032" Sep 10 13:30:49.641327 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:30:49.641280 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-7262t" Sep 10 13:30:49.641715 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:30:49.641290 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-xp5dk" Sep 10 13:30:49.641715 ip-10-0-140-8 kubenswrapper[2585]: E0910 13:30:49.641435 2585 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-7262t" podUID="082a7577-a14f-49fb-a9b1-3cf250e6735a" Sep 10 13:30:49.641715 ip-10-0-140-8 kubenswrapper[2585]: E0910 13:30:49.641486 2585 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-xp5dk" podUID="d552154f-1d5b-4c5d-99bf-7aeef22ce032" Sep 10 13:30:50.632842 ip-10-0-140-8 kubenswrapper[2585]: E0910 13:30:50.632809 2585 kubelet.go:3032] "Container runtime network not ready" networkReady="NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" Sep 10 13:30:51.641357 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:30:51.641316 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-xp5dk" Sep 10 13:30:51.641745 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:30:51.641316 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-7262t" Sep 10 13:30:51.641745 ip-10-0-140-8 kubenswrapper[2585]: E0910 13:30:51.641466 2585 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-xp5dk" podUID="d552154f-1d5b-4c5d-99bf-7aeef22ce032" Sep 10 13:30:51.641745 ip-10-0-140-8 kubenswrapper[2585]: E0910 13:30:51.641497 2585 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-7262t" podUID="082a7577-a14f-49fb-a9b1-3cf250e6735a" Sep 10 13:30:53.641079 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:30:53.641044 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-xp5dk" Sep 10 13:30:53.641567 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:30:53.641046 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-7262t" Sep 10 13:30:53.641567 ip-10-0-140-8 kubenswrapper[2585]: E0910 13:30:53.641159 2585 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-xp5dk" podUID="d552154f-1d5b-4c5d-99bf-7aeef22ce032" Sep 10 13:30:53.641567 ip-10-0-140-8 kubenswrapper[2585]: E0910 13:30:53.641239 2585 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-7262t" podUID="082a7577-a14f-49fb-a9b1-3cf250e6735a" Sep 10 13:30:55.633146 ip-10-0-140-8 kubenswrapper[2585]: E0910 13:30:55.633107 2585 kubelet.go:3032] "Container runtime network not ready" networkReady="NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" Sep 10 13:30:55.642320 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:30:55.642296 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-7262t" Sep 10 13:30:55.642466 ip-10-0-140-8 kubenswrapper[2585]: E0910 13:30:55.642385 2585 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-7262t" podUID="082a7577-a14f-49fb-a9b1-3cf250e6735a" Sep 10 13:30:55.642534 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:30:55.642519 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-xp5dk" Sep 10 13:30:55.642663 ip-10-0-140-8 kubenswrapper[2585]: E0910 13:30:55.642643 2585 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-xp5dk" podUID="d552154f-1d5b-4c5d-99bf-7aeef22ce032" Sep 10 13:30:55.642728 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:30:55.642678 2585 scope.go:117] "RemoveContainer" containerID="24d6d79549538f1d536272aee5a52d89c1116efcd876fcb7e0148947fd754dd3" Sep 10 13:30:55.960882 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:30:55.960711 2585 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ovn-kubernetes/ovnkube-node-qz8x4" podUID="6c90873d-f3b0-42a4-adc8-6a644a1f2d11" containerName="ovnkube-controller" probeResult="failure" output="" Sep 10 13:30:56.092411 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:30:56.092385 2585 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-l9l8q_ad555c6a-4385-4f8e-93bb-a174cfaa8e74/kube-multus/1.log" Sep 10 13:30:56.092563 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:30:56.092443 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-l9l8q" event={"ID":"ad555c6a-4385-4f8e-93bb-a174cfaa8e74","Type":"ContainerStarted","Data":"035f3abf7ba714be912ecde7271228a45f8e9be8c2ddd63c8e35aa88ba549e17"} Sep 10 13:30:56.144926 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:30:56.144850 2585 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kube-system/global-pull-secret-syncer-lhr9l"] Sep 10 13:30:56.147530 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:30:56.147513 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-lhr9l" Sep 10 13:30:56.147586 ip-10-0-140-8 kubenswrapper[2585]: E0910 13:30:56.147574 2585 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-lhr9l" podUID="b37f8a5b-3444-4775-8751-b60c859154d9" Sep 10 13:30:56.245048 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:30:56.245005 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/b37f8a5b-3444-4775-8751-b60c859154d9-dbus\") pod \"global-pull-secret-syncer-lhr9l\" (UID: \"b37f8a5b-3444-4775-8751-b60c859154d9\") " pod="kube-system/global-pull-secret-syncer-lhr9l" Sep 10 13:30:56.245229 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:30:56.245067 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/b37f8a5b-3444-4775-8751-b60c859154d9-original-pull-secret\") pod \"global-pull-secret-syncer-lhr9l\" (UID: \"b37f8a5b-3444-4775-8751-b60c859154d9\") " pod="kube-system/global-pull-secret-syncer-lhr9l" Sep 10 13:30:56.245229 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:30:56.245106 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/b37f8a5b-3444-4775-8751-b60c859154d9-kubelet-config\") pod \"global-pull-secret-syncer-lhr9l\" (UID: \"b37f8a5b-3444-4775-8751-b60c859154d9\") " pod="kube-system/global-pull-secret-syncer-lhr9l" Sep 10 13:30:56.346338 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:30:56.346313 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/b37f8a5b-3444-4775-8751-b60c859154d9-kubelet-config\") pod \"global-pull-secret-syncer-lhr9l\" (UID: \"b37f8a5b-3444-4775-8751-b60c859154d9\") " pod="kube-system/global-pull-secret-syncer-lhr9l" Sep 10 13:30:56.346440 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:30:56.346346 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/b37f8a5b-3444-4775-8751-b60c859154d9-dbus\") pod \"global-pull-secret-syncer-lhr9l\" (UID: \"b37f8a5b-3444-4775-8751-b60c859154d9\") " pod="kube-system/global-pull-secret-syncer-lhr9l" Sep 10 13:30:56.346440 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:30:56.346384 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/b37f8a5b-3444-4775-8751-b60c859154d9-original-pull-secret\") pod \"global-pull-secret-syncer-lhr9l\" (UID: \"b37f8a5b-3444-4775-8751-b60c859154d9\") " pod="kube-system/global-pull-secret-syncer-lhr9l" Sep 10 13:30:56.346544 ip-10-0-140-8 kubenswrapper[2585]: E0910 13:30:56.346475 2585 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Sep 10 13:30:56.346544 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:30:56.346512 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/b37f8a5b-3444-4775-8751-b60c859154d9-dbus\") pod \"global-pull-secret-syncer-lhr9l\" (UID: \"b37f8a5b-3444-4775-8751-b60c859154d9\") " pod="kube-system/global-pull-secret-syncer-lhr9l" Sep 10 13:30:56.346544 ip-10-0-140-8 kubenswrapper[2585]: E0910 13:30:56.346533 2585 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b37f8a5b-3444-4775-8751-b60c859154d9-original-pull-secret podName:b37f8a5b-3444-4775-8751-b60c859154d9 nodeName:}" failed. No retries permitted until 2026-09-10 13:30:56.846520269 +0000 UTC m=+141.719224448 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/b37f8a5b-3444-4775-8751-b60c859154d9-original-pull-secret") pod "global-pull-secret-syncer-lhr9l" (UID: "b37f8a5b-3444-4775-8751-b60c859154d9") : object "kube-system"/"original-pull-secret" not registered Sep 10 13:30:56.346656 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:30:56.346469 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/b37f8a5b-3444-4775-8751-b60c859154d9-kubelet-config\") pod \"global-pull-secret-syncer-lhr9l\" (UID: \"b37f8a5b-3444-4775-8751-b60c859154d9\") " pod="kube-system/global-pull-secret-syncer-lhr9l" Sep 10 13:30:56.849496 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:30:56.849447 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/b37f8a5b-3444-4775-8751-b60c859154d9-original-pull-secret\") pod \"global-pull-secret-syncer-lhr9l\" (UID: \"b37f8a5b-3444-4775-8751-b60c859154d9\") " pod="kube-system/global-pull-secret-syncer-lhr9l" Sep 10 13:30:56.849999 ip-10-0-140-8 kubenswrapper[2585]: E0910 13:30:56.849586 2585 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Sep 10 13:30:56.849999 ip-10-0-140-8 kubenswrapper[2585]: E0910 13:30:56.849651 2585 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b37f8a5b-3444-4775-8751-b60c859154d9-original-pull-secret podName:b37f8a5b-3444-4775-8751-b60c859154d9 nodeName:}" failed. No retries permitted until 2026-09-10 13:30:57.849632994 +0000 UTC m=+142.722337176 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/b37f8a5b-3444-4775-8751-b60c859154d9-original-pull-secret") pod "global-pull-secret-syncer-lhr9l" (UID: "b37f8a5b-3444-4775-8751-b60c859154d9") : object "kube-system"/"original-pull-secret" not registered Sep 10 13:30:57.640841 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:30:57.640780 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-7262t" Sep 10 13:30:57.640841 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:30:57.640822 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-xp5dk" Sep 10 13:30:57.640841 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:30:57.640822 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-lhr9l" Sep 10 13:30:57.641121 ip-10-0-140-8 kubenswrapper[2585]: E0910 13:30:57.640915 2585 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-7262t" podUID="082a7577-a14f-49fb-a9b1-3cf250e6735a" Sep 10 13:30:57.641121 ip-10-0-140-8 kubenswrapper[2585]: E0910 13:30:57.641030 2585 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-xp5dk" podUID="d552154f-1d5b-4c5d-99bf-7aeef22ce032" Sep 10 13:30:57.641194 ip-10-0-140-8 kubenswrapper[2585]: E0910 13:30:57.641124 2585 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-lhr9l" podUID="b37f8a5b-3444-4775-8751-b60c859154d9" Sep 10 13:30:57.856783 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:30:57.856740 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/b37f8a5b-3444-4775-8751-b60c859154d9-original-pull-secret\") pod \"global-pull-secret-syncer-lhr9l\" (UID: \"b37f8a5b-3444-4775-8751-b60c859154d9\") " pod="kube-system/global-pull-secret-syncer-lhr9l" Sep 10 13:30:57.857167 ip-10-0-140-8 kubenswrapper[2585]: E0910 13:30:57.856856 2585 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Sep 10 13:30:57.857167 ip-10-0-140-8 kubenswrapper[2585]: E0910 13:30:57.856906 2585 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b37f8a5b-3444-4775-8751-b60c859154d9-original-pull-secret podName:b37f8a5b-3444-4775-8751-b60c859154d9 nodeName:}" failed. No retries permitted until 2026-09-10 13:30:59.856893306 +0000 UTC m=+144.729597485 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/b37f8a5b-3444-4775-8751-b60c859154d9-original-pull-secret") pod "global-pull-secret-syncer-lhr9l" (UID: "b37f8a5b-3444-4775-8751-b60c859154d9") : object "kube-system"/"original-pull-secret" not registered Sep 10 13:30:58.650999 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:30:58.650964 2585 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ovn-kubernetes/ovnkube-node-qz8x4" podUID="6c90873d-f3b0-42a4-adc8-6a644a1f2d11" containerName="ovnkube-controller" probeResult="failure" output="" Sep 10 13:30:59.640828 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:30:59.640787 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-7262t" Sep 10 13:30:59.641374 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:30:59.640802 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-lhr9l" Sep 10 13:30:59.641374 ip-10-0-140-8 kubenswrapper[2585]: E0910 13:30:59.640924 2585 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-7262t" podUID="082a7577-a14f-49fb-a9b1-3cf250e6735a" Sep 10 13:30:59.641374 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:30:59.640802 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-xp5dk" Sep 10 13:30:59.641374 ip-10-0-140-8 kubenswrapper[2585]: E0910 13:30:59.640979 2585 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-lhr9l" podUID="b37f8a5b-3444-4775-8751-b60c859154d9" Sep 10 13:30:59.641374 ip-10-0-140-8 kubenswrapper[2585]: E0910 13:30:59.641048 2585 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-xp5dk" podUID="d552154f-1d5b-4c5d-99bf-7aeef22ce032" Sep 10 13:30:59.872490 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:30:59.872444 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/b37f8a5b-3444-4775-8751-b60c859154d9-original-pull-secret\") pod \"global-pull-secret-syncer-lhr9l\" (UID: \"b37f8a5b-3444-4775-8751-b60c859154d9\") " pod="kube-system/global-pull-secret-syncer-lhr9l" Sep 10 13:30:59.872675 ip-10-0-140-8 kubenswrapper[2585]: E0910 13:30:59.872573 2585 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Sep 10 13:30:59.872675 ip-10-0-140-8 kubenswrapper[2585]: E0910 13:30:59.872632 2585 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b37f8a5b-3444-4775-8751-b60c859154d9-original-pull-secret podName:b37f8a5b-3444-4775-8751-b60c859154d9 nodeName:}" failed. No retries permitted until 2026-09-10 13:31:03.872614721 +0000 UTC m=+148.745318900 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/b37f8a5b-3444-4775-8751-b60c859154d9-original-pull-secret") pod "global-pull-secret-syncer-lhr9l" (UID: "b37f8a5b-3444-4775-8751-b60c859154d9") : object "kube-system"/"original-pull-secret" not registered Sep 10 13:31:00.634699 ip-10-0-140-8 kubenswrapper[2585]: E0910 13:31:00.634663 2585 kubelet.go:3032] "Container runtime network not ready" networkReady="NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" Sep 10 13:31:01.641094 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:31:01.641051 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-7262t" Sep 10 13:31:01.641513 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:31:01.641071 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-xp5dk" Sep 10 13:31:01.641513 ip-10-0-140-8 kubenswrapper[2585]: E0910 13:31:01.641180 2585 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-7262t" podUID="082a7577-a14f-49fb-a9b1-3cf250e6735a" Sep 10 13:31:01.641513 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:31:01.641071 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-lhr9l" Sep 10 13:31:01.641513 ip-10-0-140-8 kubenswrapper[2585]: E0910 13:31:01.641289 2585 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-xp5dk" podUID="d552154f-1d5b-4c5d-99bf-7aeef22ce032" Sep 10 13:31:01.641513 ip-10-0-140-8 kubenswrapper[2585]: E0910 13:31:01.641343 2585 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-lhr9l" podUID="b37f8a5b-3444-4775-8751-b60c859154d9" Sep 10 13:31:03.494014 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:31:03.493958 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d552154f-1d5b-4c5d-99bf-7aeef22ce032-metrics-certs\") pod \"network-metrics-daemon-xp5dk\" (UID: \"d552154f-1d5b-4c5d-99bf-7aeef22ce032\") " pod="openshift-multus/network-metrics-daemon-xp5dk" Sep 10 13:31:03.494417 ip-10-0-140-8 kubenswrapper[2585]: E0910 13:31:03.494120 2585 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Sep 10 13:31:03.494417 ip-10-0-140-8 kubenswrapper[2585]: E0910 13:31:03.494200 2585 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d552154f-1d5b-4c5d-99bf-7aeef22ce032-metrics-certs podName:d552154f-1d5b-4c5d-99bf-7aeef22ce032 nodeName:}" failed. No retries permitted until 2026-09-10 13:33:05.494184414 +0000 UTC m=+270.366888599 (durationBeforeRetry 2m2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/d552154f-1d5b-4c5d-99bf-7aeef22ce032-metrics-certs") pod "network-metrics-daemon-xp5dk" (UID: "d552154f-1d5b-4c5d-99bf-7aeef22ce032") : object "openshift-multus"/"metrics-daemon-secret" not registered Sep 10 13:31:03.594541 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:31:03.594498 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x2vv5\" (UniqueName: \"kubernetes.io/projected/082a7577-a14f-49fb-a9b1-3cf250e6735a-kube-api-access-x2vv5\") pod \"network-check-target-7262t\" (UID: \"082a7577-a14f-49fb-a9b1-3cf250e6735a\") " pod="openshift-network-diagnostics/network-check-target-7262t" Sep 10 13:31:03.594655 ip-10-0-140-8 kubenswrapper[2585]: E0910 13:31:03.594632 2585 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Sep 10 13:31:03.594655 ip-10-0-140-8 kubenswrapper[2585]: E0910 13:31:03.594647 2585 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Sep 10 13:31:03.594655 ip-10-0-140-8 kubenswrapper[2585]: E0910 13:31:03.594655 2585 projected.go:196] Error preparing data for projected volume kube-api-access-x2vv5 for pod openshift-network-diagnostics/network-check-target-7262t: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Sep 10 13:31:03.594763 ip-10-0-140-8 kubenswrapper[2585]: E0910 13:31:03.594698 2585 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/082a7577-a14f-49fb-a9b1-3cf250e6735a-kube-api-access-x2vv5 podName:082a7577-a14f-49fb-a9b1-3cf250e6735a nodeName:}" failed. No retries permitted until 2026-09-10 13:33:05.594686192 +0000 UTC m=+270.467390376 (durationBeforeRetry 2m2s). Error: MountVolume.SetUp failed for volume "kube-api-access-x2vv5" (UniqueName: "kubernetes.io/projected/082a7577-a14f-49fb-a9b1-3cf250e6735a-kube-api-access-x2vv5") pod "network-check-target-7262t" (UID: "082a7577-a14f-49fb-a9b1-3cf250e6735a") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Sep 10 13:31:03.641618 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:31:03.641585 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-lhr9l" Sep 10 13:31:03.641751 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:31:03.641587 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-xp5dk" Sep 10 13:31:03.641751 ip-10-0-140-8 kubenswrapper[2585]: E0910 13:31:03.641688 2585 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-lhr9l" podUID="b37f8a5b-3444-4775-8751-b60c859154d9" Sep 10 13:31:03.641862 ip-10-0-140-8 kubenswrapper[2585]: E0910 13:31:03.641762 2585 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-xp5dk" podUID="d552154f-1d5b-4c5d-99bf-7aeef22ce032" Sep 10 13:31:03.641862 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:31:03.641589 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-7262t" Sep 10 13:31:03.641862 ip-10-0-140-8 kubenswrapper[2585]: E0910 13:31:03.641823 2585 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-7262t" podUID="082a7577-a14f-49fb-a9b1-3cf250e6735a" Sep 10 13:31:03.895850 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:31:03.895820 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/b37f8a5b-3444-4775-8751-b60c859154d9-original-pull-secret\") pod \"global-pull-secret-syncer-lhr9l\" (UID: \"b37f8a5b-3444-4775-8751-b60c859154d9\") " pod="kube-system/global-pull-secret-syncer-lhr9l" Sep 10 13:31:03.896011 ip-10-0-140-8 kubenswrapper[2585]: E0910 13:31:03.895935 2585 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Sep 10 13:31:03.896011 ip-10-0-140-8 kubenswrapper[2585]: E0910 13:31:03.895986 2585 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b37f8a5b-3444-4775-8751-b60c859154d9-original-pull-secret podName:b37f8a5b-3444-4775-8751-b60c859154d9 nodeName:}" failed. No retries permitted until 2026-09-10 13:31:11.895974424 +0000 UTC m=+156.768678603 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/b37f8a5b-3444-4775-8751-b60c859154d9-original-pull-secret") pod "global-pull-secret-syncer-lhr9l" (UID: "b37f8a5b-3444-4775-8751-b60c859154d9") : object "kube-system"/"original-pull-secret" not registered Sep 10 13:31:05.635026 ip-10-0-140-8 kubenswrapper[2585]: E0910 13:31:05.634989 2585 kubelet.go:3032] "Container runtime network not ready" networkReady="NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" Sep 10 13:31:05.641399 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:31:05.641380 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-7262t" Sep 10 13:31:05.641501 ip-10-0-140-8 kubenswrapper[2585]: E0910 13:31:05.641483 2585 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-7262t" podUID="082a7577-a14f-49fb-a9b1-3cf250e6735a" Sep 10 13:31:05.641589 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:31:05.641572 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-xp5dk" Sep 10 13:31:05.641752 ip-10-0-140-8 kubenswrapper[2585]: E0910 13:31:05.641732 2585 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-xp5dk" podUID="d552154f-1d5b-4c5d-99bf-7aeef22ce032" Sep 10 13:31:05.641825 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:31:05.641791 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-lhr9l" Sep 10 13:31:05.641879 ip-10-0-140-8 kubenswrapper[2585]: E0910 13:31:05.641846 2585 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-lhr9l" podUID="b37f8a5b-3444-4775-8751-b60c859154d9" Sep 10 13:31:07.640771 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:31:07.640726 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-7262t" Sep 10 13:31:07.641172 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:31:07.640726 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-xp5dk" Sep 10 13:31:07.641172 ip-10-0-140-8 kubenswrapper[2585]: E0910 13:31:07.640836 2585 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-7262t" podUID="082a7577-a14f-49fb-a9b1-3cf250e6735a" Sep 10 13:31:07.641172 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:31:07.640867 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-lhr9l" Sep 10 13:31:07.641172 ip-10-0-140-8 kubenswrapper[2585]: E0910 13:31:07.640980 2585 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-xp5dk" podUID="d552154f-1d5b-4c5d-99bf-7aeef22ce032" Sep 10 13:31:07.641172 ip-10-0-140-8 kubenswrapper[2585]: E0910 13:31:07.641006 2585 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-lhr9l" podUID="b37f8a5b-3444-4775-8751-b60c859154d9" Sep 10 13:31:09.640865 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:31:09.640825 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-7262t" Sep 10 13:31:09.641491 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:31:09.640825 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-lhr9l" Sep 10 13:31:09.641491 ip-10-0-140-8 kubenswrapper[2585]: E0910 13:31:09.640934 2585 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-7262t" podUID="082a7577-a14f-49fb-a9b1-3cf250e6735a" Sep 10 13:31:09.641491 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:31:09.640831 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-xp5dk" Sep 10 13:31:09.641491 ip-10-0-140-8 kubenswrapper[2585]: E0910 13:31:09.641009 2585 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-lhr9l" podUID="b37f8a5b-3444-4775-8751-b60c859154d9" Sep 10 13:31:09.641491 ip-10-0-140-8 kubenswrapper[2585]: E0910 13:31:09.641072 2585 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-xp5dk" podUID="d552154f-1d5b-4c5d-99bf-7aeef22ce032" Sep 10 13:31:10.636345 ip-10-0-140-8 kubenswrapper[2585]: E0910 13:31:10.636307 2585 kubelet.go:3032] "Container runtime network not ready" networkReady="NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" Sep 10 13:31:11.641341 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:31:11.641302 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-lhr9l" Sep 10 13:31:11.641828 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:31:11.641302 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-7262t" Sep 10 13:31:11.641828 ip-10-0-140-8 kubenswrapper[2585]: E0910 13:31:11.641411 2585 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-lhr9l" podUID="b37f8a5b-3444-4775-8751-b60c859154d9" Sep 10 13:31:11.641828 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:31:11.641302 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-xp5dk" Sep 10 13:31:11.641828 ip-10-0-140-8 kubenswrapper[2585]: E0910 13:31:11.641475 2585 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-7262t" podUID="082a7577-a14f-49fb-a9b1-3cf250e6735a" Sep 10 13:31:11.641828 ip-10-0-140-8 kubenswrapper[2585]: E0910 13:31:11.641550 2585 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-xp5dk" podUID="d552154f-1d5b-4c5d-99bf-7aeef22ce032" Sep 10 13:31:11.946391 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:31:11.946300 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/b37f8a5b-3444-4775-8751-b60c859154d9-original-pull-secret\") pod \"global-pull-secret-syncer-lhr9l\" (UID: \"b37f8a5b-3444-4775-8751-b60c859154d9\") " pod="kube-system/global-pull-secret-syncer-lhr9l" Sep 10 13:31:11.946561 ip-10-0-140-8 kubenswrapper[2585]: E0910 13:31:11.946430 2585 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Sep 10 13:31:11.946561 ip-10-0-140-8 kubenswrapper[2585]: E0910 13:31:11.946494 2585 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b37f8a5b-3444-4775-8751-b60c859154d9-original-pull-secret podName:b37f8a5b-3444-4775-8751-b60c859154d9 nodeName:}" failed. No retries permitted until 2026-09-10 13:31:27.94647715 +0000 UTC m=+172.819181343 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/b37f8a5b-3444-4775-8751-b60c859154d9-original-pull-secret") pod "global-pull-secret-syncer-lhr9l" (UID: "b37f8a5b-3444-4775-8751-b60c859154d9") : object "kube-system"/"original-pull-secret" not registered Sep 10 13:31:13.641611 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:31:13.641576 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-7262t" Sep 10 13:31:13.642074 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:31:13.641590 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-lhr9l" Sep 10 13:31:13.642074 ip-10-0-140-8 kubenswrapper[2585]: E0910 13:31:13.641698 2585 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-7262t" podUID="082a7577-a14f-49fb-a9b1-3cf250e6735a" Sep 10 13:31:13.642074 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:31:13.641590 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-xp5dk" Sep 10 13:31:13.642074 ip-10-0-140-8 kubenswrapper[2585]: E0910 13:31:13.641749 2585 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-lhr9l" podUID="b37f8a5b-3444-4775-8751-b60c859154d9" Sep 10 13:31:13.642074 ip-10-0-140-8 kubenswrapper[2585]: E0910 13:31:13.641821 2585 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-xp5dk" podUID="d552154f-1d5b-4c5d-99bf-7aeef22ce032" Sep 10 13:31:14.758416 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:31:14.758381 2585 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-cj6zv"] Sep 10 13:31:14.761267 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:31:14.761233 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-cj6zv" Sep 10 13:31:14.764542 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:31:14.764206 2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-vqf84\"" Sep 10 13:31:14.765321 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:31:14.765296 2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\"" Sep 10 13:31:14.801505 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:31:14.801479 2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\"" Sep 10 13:31:14.801831 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:31:14.801818 2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\"" Sep 10 13:31:14.865737 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:31:14.865696 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/aea5c7dc-36e2-460e-8299-febd48063f65-device-dir\") pod \"aws-ebs-csi-driver-node-cj6zv\" (UID: \"aea5c7dc-36e2-460e-8299-febd48063f65\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-cj6zv" Sep 10 13:31:14.865737 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:31:14.865737 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sq8xr\" (UniqueName: \"kubernetes.io/projected/aea5c7dc-36e2-460e-8299-febd48063f65-kube-api-access-sq8xr\") pod \"aws-ebs-csi-driver-node-cj6zv\" (UID: \"aea5c7dc-36e2-460e-8299-febd48063f65\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-cj6zv" Sep 10 13:31:14.865935 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:31:14.865798 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/aea5c7dc-36e2-460e-8299-febd48063f65-kubelet-dir\") pod \"aws-ebs-csi-driver-node-cj6zv\" (UID: \"aea5c7dc-36e2-460e-8299-febd48063f65\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-cj6zv" Sep 10 13:31:14.865935 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:31:14.865873 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/aea5c7dc-36e2-460e-8299-febd48063f65-registration-dir\") pod \"aws-ebs-csi-driver-node-cj6zv\" (UID: \"aea5c7dc-36e2-460e-8299-febd48063f65\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-cj6zv" Sep 10 13:31:14.865935 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:31:14.865915 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/aea5c7dc-36e2-460e-8299-febd48063f65-socket-dir\") pod \"aws-ebs-csi-driver-node-cj6zv\" (UID: \"aea5c7dc-36e2-460e-8299-febd48063f65\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-cj6zv" Sep 10 13:31:14.866028 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:31:14.865937 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/aea5c7dc-36e2-460e-8299-febd48063f65-sys-fs\") pod \"aws-ebs-csi-driver-node-cj6zv\" (UID: \"aea5c7dc-36e2-460e-8299-febd48063f65\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-cj6zv" Sep 10 13:31:14.866028 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:31:14.865955 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/aea5c7dc-36e2-460e-8299-febd48063f65-etc-selinux\") pod \"aws-ebs-csi-driver-node-cj6zv\" (UID: \"aea5c7dc-36e2-460e-8299-febd48063f65\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-cj6zv" Sep 10 13:31:14.967140 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:31:14.967106 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/aea5c7dc-36e2-460e-8299-febd48063f65-registration-dir\") pod \"aws-ebs-csi-driver-node-cj6zv\" (UID: \"aea5c7dc-36e2-460e-8299-febd48063f65\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-cj6zv" Sep 10 13:31:14.967279 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:31:14.967158 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/aea5c7dc-36e2-460e-8299-febd48063f65-socket-dir\") pod \"aws-ebs-csi-driver-node-cj6zv\" (UID: \"aea5c7dc-36e2-460e-8299-febd48063f65\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-cj6zv" Sep 10 13:31:14.967279 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:31:14.967179 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/aea5c7dc-36e2-460e-8299-febd48063f65-sys-fs\") pod \"aws-ebs-csi-driver-node-cj6zv\" (UID: \"aea5c7dc-36e2-460e-8299-febd48063f65\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-cj6zv" Sep 10 13:31:14.967279 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:31:14.967202 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/aea5c7dc-36e2-460e-8299-febd48063f65-etc-selinux\") pod \"aws-ebs-csi-driver-node-cj6zv\" (UID: \"aea5c7dc-36e2-460e-8299-febd48063f65\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-cj6zv" Sep 10 13:31:14.967279 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:31:14.967236 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/aea5c7dc-36e2-460e-8299-febd48063f65-device-dir\") pod \"aws-ebs-csi-driver-node-cj6zv\" (UID: \"aea5c7dc-36e2-460e-8299-febd48063f65\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-cj6zv" Sep 10 13:31:14.967463 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:31:14.967273 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/aea5c7dc-36e2-460e-8299-febd48063f65-sys-fs\") pod \"aws-ebs-csi-driver-node-cj6zv\" (UID: \"aea5c7dc-36e2-460e-8299-febd48063f65\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-cj6zv" Sep 10 13:31:14.967463 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:31:14.967279 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sq8xr\" (UniqueName: \"kubernetes.io/projected/aea5c7dc-36e2-460e-8299-febd48063f65-kube-api-access-sq8xr\") pod \"aws-ebs-csi-driver-node-cj6zv\" (UID: \"aea5c7dc-36e2-460e-8299-febd48063f65\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-cj6zv" Sep 10 13:31:14.967463 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:31:14.967347 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/aea5c7dc-36e2-460e-8299-febd48063f65-device-dir\") pod \"aws-ebs-csi-driver-node-cj6zv\" (UID: \"aea5c7dc-36e2-460e-8299-febd48063f65\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-cj6zv" Sep 10 13:31:14.967463 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:31:14.967358 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/aea5c7dc-36e2-460e-8299-febd48063f65-kubelet-dir\") pod \"aws-ebs-csi-driver-node-cj6zv\" (UID: \"aea5c7dc-36e2-460e-8299-febd48063f65\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-cj6zv" Sep 10 13:31:14.967463 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:31:14.967374 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/aea5c7dc-36e2-460e-8299-febd48063f65-registration-dir\") pod \"aws-ebs-csi-driver-node-cj6zv\" (UID: \"aea5c7dc-36e2-460e-8299-febd48063f65\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-cj6zv" Sep 10 13:31:14.967463 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:31:14.967384 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/aea5c7dc-36e2-460e-8299-febd48063f65-etc-selinux\") pod \"aws-ebs-csi-driver-node-cj6zv\" (UID: \"aea5c7dc-36e2-460e-8299-febd48063f65\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-cj6zv" Sep 10 13:31:14.967463 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:31:14.967402 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/aea5c7dc-36e2-460e-8299-febd48063f65-kubelet-dir\") pod \"aws-ebs-csi-driver-node-cj6zv\" (UID: \"aea5c7dc-36e2-460e-8299-febd48063f65\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-cj6zv" Sep 10 13:31:14.967463 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:31:14.967455 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/aea5c7dc-36e2-460e-8299-febd48063f65-socket-dir\") pod \"aws-ebs-csi-driver-node-cj6zv\" (UID: \"aea5c7dc-36e2-460e-8299-febd48063f65\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-cj6zv" Sep 10 13:31:14.978185 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:31:14.978161 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-sq8xr\" (UniqueName: \"kubernetes.io/projected/aea5c7dc-36e2-460e-8299-febd48063f65-kube-api-access-sq8xr\") pod \"aws-ebs-csi-driver-node-cj6zv\" (UID: \"aea5c7dc-36e2-460e-8299-febd48063f65\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-cj6zv" Sep 10 13:31:15.070362 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:31:15.070333 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-cj6zv" Sep 10 13:31:15.079282 ip-10-0-140-8 kubenswrapper[2585]: W0910 13:31:15.079253 2585 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podaea5c7dc_36e2_460e_8299_febd48063f65.slice/crio-a78dc384e6e539346d09f4095ee50b0781fda1e0d6b7bdcf2b11f019926fd693 WatchSource:0}: Error finding container a78dc384e6e539346d09f4095ee50b0781fda1e0d6b7bdcf2b11f019926fd693: Status 404 returned error can't find the container with id a78dc384e6e539346d09f4095ee50b0781fda1e0d6b7bdcf2b11f019926fd693 Sep 10 13:31:15.136130 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:31:15.136099 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-cj6zv" event={"ID":"aea5c7dc-36e2-460e-8299-febd48063f65","Type":"ContainerStarted","Data":"a78dc384e6e539346d09f4095ee50b0781fda1e0d6b7bdcf2b11f019926fd693"} Sep 10 13:31:15.636712 ip-10-0-140-8 kubenswrapper[2585]: E0910 13:31:15.636673 2585 kubelet.go:3032] "Container runtime network not ready" networkReady="NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" Sep 10 13:31:15.644719 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:31:15.644690 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-lhr9l" Sep 10 13:31:15.644858 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:31:15.644732 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-7262t" Sep 10 13:31:15.644858 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:31:15.644797 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-xp5dk" Sep 10 13:31:15.644858 ip-10-0-140-8 kubenswrapper[2585]: E0910 13:31:15.644810 2585 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-lhr9l" podUID="b37f8a5b-3444-4775-8751-b60c859154d9" Sep 10 13:31:15.645026 ip-10-0-140-8 kubenswrapper[2585]: E0910 13:31:15.644932 2585 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-xp5dk" podUID="d552154f-1d5b-4c5d-99bf-7aeef22ce032" Sep 10 13:31:15.645026 ip-10-0-140-8 kubenswrapper[2585]: E0910 13:31:15.644995 2585 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-7262t" podUID="082a7577-a14f-49fb-a9b1-3cf250e6735a" Sep 10 13:31:17.143093 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:31:17.143010 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-cj6zv" event={"ID":"aea5c7dc-36e2-460e-8299-febd48063f65","Type":"ContainerStarted","Data":"55b69c25b776ad0e972100a41e372cf81c5a89fac9514ce39d22fb6bc78cbd54"} Sep 10 13:31:17.641665 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:31:17.641630 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-7262t" Sep 10 13:31:17.641831 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:31:17.641679 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-lhr9l" Sep 10 13:31:17.641831 ip-10-0-140-8 kubenswrapper[2585]: E0910 13:31:17.641760 2585 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-7262t" podUID="082a7577-a14f-49fb-a9b1-3cf250e6735a" Sep 10 13:31:17.641831 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:31:17.641638 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-xp5dk" Sep 10 13:31:17.642016 ip-10-0-140-8 kubenswrapper[2585]: E0910 13:31:17.641853 2585 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-lhr9l" podUID="b37f8a5b-3444-4775-8751-b60c859154d9" Sep 10 13:31:17.642016 ip-10-0-140-8 kubenswrapper[2585]: E0910 13:31:17.641942 2585 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-xp5dk" podUID="d552154f-1d5b-4c5d-99bf-7aeef22ce032" Sep 10 13:31:17.912774 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:31:17.912746 2585 plugin_watcher.go:194] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock" Sep 10 13:31:18.147567 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:31:18.147533 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-cj6zv" event={"ID":"aea5c7dc-36e2-460e-8299-febd48063f65","Type":"ContainerStarted","Data":"4324c7ac2a742b2d076fbb6c2ba2892c1ade9d174b8704914e741e373c118797"} Sep 10 13:31:18.699683 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:31:18.699592 2585 reconciler.go:161] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-09-10T13:31:17.91276794Z","UUID":"f9cf12e7-fd83-4c27-9106-a679ddc52052","Handler":null,"Name":"","Endpoint":""} Sep 10 13:31:18.701150 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:31:18.701132 2585 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0 Sep 10 13:31:18.701272 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:31:18.701165 2585 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock Sep 10 13:31:19.641786 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:31:19.641749 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-lhr9l" Sep 10 13:31:19.642227 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:31:19.641749 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-7262t" Sep 10 13:31:19.642227 ip-10-0-140-8 kubenswrapper[2585]: E0910 13:31:19.641890 2585 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-lhr9l" podUID="b37f8a5b-3444-4775-8751-b60c859154d9" Sep 10 13:31:19.642227 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:31:19.641941 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-xp5dk" Sep 10 13:31:19.642227 ip-10-0-140-8 kubenswrapper[2585]: E0910 13:31:19.642046 2585 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-7262t" podUID="082a7577-a14f-49fb-a9b1-3cf250e6735a" Sep 10 13:31:19.642227 ip-10-0-140-8 kubenswrapper[2585]: E0910 13:31:19.642143 2585 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-xp5dk" podUID="d552154f-1d5b-4c5d-99bf-7aeef22ce032" Sep 10 13:31:20.154200 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:31:20.154165 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-cj6zv" event={"ID":"aea5c7dc-36e2-460e-8299-febd48063f65","Type":"ContainerStarted","Data":"b2683d807a681d0b83a8322a7ad306dfef5e7337447eca33a0156cd4412014b0"} Sep 10 13:31:20.170503 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:31:20.170465 2585 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-cj6zv" podStartSLOduration=1.6348188160000001 podStartE2EDuration="6.170451484s" totalImagesPullingTime="4.535632668s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-10 13:31:14 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 13:31:20.170260615 +0000 UTC m=+165.042964807" watchObservedRunningTime="2026-09-10 13:31:20.170451484 +0000 UTC m=+165.043155685" Sep 10 13:31:20.637516 ip-10-0-140-8 kubenswrapper[2585]: E0910 13:31:20.637481 2585 kubelet.go:3032] "Container runtime network not ready" networkReady="NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" Sep 10 13:31:21.640682 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:31:21.640640 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-7262t" Sep 10 13:31:21.640682 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:31:21.640664 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-lhr9l" Sep 10 13:31:21.641195 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:31:21.640642 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-xp5dk" Sep 10 13:31:21.641195 ip-10-0-140-8 kubenswrapper[2585]: E0910 13:31:21.640755 2585 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-7262t" podUID="082a7577-a14f-49fb-a9b1-3cf250e6735a" Sep 10 13:31:21.641195 ip-10-0-140-8 kubenswrapper[2585]: E0910 13:31:21.640852 2585 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-xp5dk" podUID="d552154f-1d5b-4c5d-99bf-7aeef22ce032" Sep 10 13:31:21.641195 ip-10-0-140-8 kubenswrapper[2585]: E0910 13:31:21.640940 2585 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-lhr9l" podUID="b37f8a5b-3444-4775-8751-b60c859154d9" Sep 10 13:31:23.640681 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:31:23.640642 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-xp5dk" Sep 10 13:31:23.641114 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:31:23.640642 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-7262t" Sep 10 13:31:23.641114 ip-10-0-140-8 kubenswrapper[2585]: E0910 13:31:23.640765 2585 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-xp5dk" podUID="d552154f-1d5b-4c5d-99bf-7aeef22ce032" Sep 10 13:31:23.641114 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:31:23.640796 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-lhr9l" Sep 10 13:31:23.641114 ip-10-0-140-8 kubenswrapper[2585]: E0910 13:31:23.640900 2585 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-7262t" podUID="082a7577-a14f-49fb-a9b1-3cf250e6735a" Sep 10 13:31:23.641114 ip-10-0-140-8 kubenswrapper[2585]: E0910 13:31:23.640931 2585 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-lhr9l" podUID="b37f8a5b-3444-4775-8751-b60c859154d9" Sep 10 13:31:25.637832 ip-10-0-140-8 kubenswrapper[2585]: E0910 13:31:25.637792 2585 kubelet.go:3032] "Container runtime network not ready" networkReady="NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" Sep 10 13:31:25.643885 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:31:25.643856 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-xp5dk" Sep 10 13:31:25.644236 ip-10-0-140-8 kubenswrapper[2585]: E0910 13:31:25.644208 2585 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-xp5dk" podUID="d552154f-1d5b-4c5d-99bf-7aeef22ce032" Sep 10 13:31:25.645158 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:31:25.645137 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-lhr9l" Sep 10 13:31:25.645410 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:31:25.645235 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-7262t" Sep 10 13:31:25.645532 ip-10-0-140-8 kubenswrapper[2585]: E0910 13:31:25.645393 2585 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-lhr9l" podUID="b37f8a5b-3444-4775-8751-b60c859154d9" Sep 10 13:31:25.645655 ip-10-0-140-8 kubenswrapper[2585]: E0910 13:31:25.645633 2585 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-7262t" podUID="082a7577-a14f-49fb-a9b1-3cf250e6735a" Sep 10 13:31:27.644204 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:31:27.644171 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-xp5dk" Sep 10 13:31:27.644696 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:31:27.644301 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-7262t" Sep 10 13:31:27.644696 ip-10-0-140-8 kubenswrapper[2585]: E0910 13:31:27.644305 2585 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-xp5dk" podUID="d552154f-1d5b-4c5d-99bf-7aeef22ce032" Sep 10 13:31:27.644696 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:31:27.644333 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-lhr9l" Sep 10 13:31:27.644696 ip-10-0-140-8 kubenswrapper[2585]: E0910 13:31:27.644440 2585 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-7262t" podUID="082a7577-a14f-49fb-a9b1-3cf250e6735a" Sep 10 13:31:27.644696 ip-10-0-140-8 kubenswrapper[2585]: E0910 13:31:27.644538 2585 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-lhr9l" podUID="b37f8a5b-3444-4775-8751-b60c859154d9" Sep 10 13:31:28.046087 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:31:28.045915 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/b37f8a5b-3444-4775-8751-b60c859154d9-original-pull-secret\") pod \"global-pull-secret-syncer-lhr9l\" (UID: \"b37f8a5b-3444-4775-8751-b60c859154d9\") " pod="kube-system/global-pull-secret-syncer-lhr9l" Sep 10 13:31:28.046087 ip-10-0-140-8 kubenswrapper[2585]: E0910 13:31:28.046071 2585 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Sep 10 13:31:28.046326 ip-10-0-140-8 kubenswrapper[2585]: E0910 13:31:28.046133 2585 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b37f8a5b-3444-4775-8751-b60c859154d9-original-pull-secret podName:b37f8a5b-3444-4775-8751-b60c859154d9 nodeName:}" failed. No retries permitted until 2026-09-10 13:32:00.046118125 +0000 UTC m=+204.918822308 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/b37f8a5b-3444-4775-8751-b60c859154d9-original-pull-secret") pod "global-pull-secret-syncer-lhr9l" (UID: "b37f8a5b-3444-4775-8751-b60c859154d9") : object "kube-system"/"original-pull-secret" not registered Sep 10 13:31:28.652041 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:31:28.652000 2585 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ovn-kubernetes/ovnkube-node-qz8x4" podUID="6c90873d-f3b0-42a4-adc8-6a644a1f2d11" containerName="ovnkube-controller" probeResult="failure" output="" Sep 10 13:31:29.644498 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:31:29.644469 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-lhr9l" Sep 10 13:31:29.644724 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:31:29.644563 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-7262t" Sep 10 13:31:29.644724 ip-10-0-140-8 kubenswrapper[2585]: E0910 13:31:29.644665 2585 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-7262t" podUID="082a7577-a14f-49fb-a9b1-3cf250e6735a" Sep 10 13:31:29.644724 ip-10-0-140-8 kubenswrapper[2585]: E0910 13:31:29.644565 2585 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-lhr9l" podUID="b37f8a5b-3444-4775-8751-b60c859154d9" Sep 10 13:31:29.644724 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:31:29.644720 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-xp5dk" Sep 10 13:31:29.644884 ip-10-0-140-8 kubenswrapper[2585]: E0910 13:31:29.644796 2585 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-xp5dk" podUID="d552154f-1d5b-4c5d-99bf-7aeef22ce032" Sep 10 13:31:30.638847 ip-10-0-140-8 kubenswrapper[2585]: E0910 13:31:30.638812 2585 kubelet.go:3032] "Container runtime network not ready" networkReady="NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" Sep 10 13:31:31.641120 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:31:31.641082 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-7262t" Sep 10 13:31:31.641610 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:31:31.641082 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-xp5dk" Sep 10 13:31:31.641610 ip-10-0-140-8 kubenswrapper[2585]: E0910 13:31:31.641202 2585 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-7262t" podUID="082a7577-a14f-49fb-a9b1-3cf250e6735a" Sep 10 13:31:31.641610 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:31:31.641082 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-lhr9l" Sep 10 13:31:31.641610 ip-10-0-140-8 kubenswrapper[2585]: E0910 13:31:31.641288 2585 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-xp5dk" podUID="d552154f-1d5b-4c5d-99bf-7aeef22ce032" Sep 10 13:31:31.641610 ip-10-0-140-8 kubenswrapper[2585]: E0910 13:31:31.641397 2585 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-lhr9l" podUID="b37f8a5b-3444-4775-8751-b60c859154d9" Sep 10 13:31:33.640791 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:31:33.640752 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-xp5dk" Sep 10 13:31:33.641307 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:31:33.640752 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-lhr9l" Sep 10 13:31:33.641307 ip-10-0-140-8 kubenswrapper[2585]: E0910 13:31:33.640901 2585 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-xp5dk" podUID="d552154f-1d5b-4c5d-99bf-7aeef22ce032" Sep 10 13:31:33.641307 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:31:33.640752 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-7262t" Sep 10 13:31:33.641307 ip-10-0-140-8 kubenswrapper[2585]: E0910 13:31:33.640938 2585 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-lhr9l" podUID="b37f8a5b-3444-4775-8751-b60c859154d9" Sep 10 13:31:33.641307 ip-10-0-140-8 kubenswrapper[2585]: E0910 13:31:33.641008 2585 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-7262t" podUID="082a7577-a14f-49fb-a9b1-3cf250e6735a" Sep 10 13:31:35.639174 ip-10-0-140-8 kubenswrapper[2585]: E0910 13:31:35.639136 2585 kubelet.go:3032] "Container runtime network not ready" networkReady="NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" Sep 10 13:31:35.641927 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:31:35.641903 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-xp5dk" Sep 10 13:31:35.642039 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:31:35.641993 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-7262t" Sep 10 13:31:35.642039 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:31:35.642023 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-lhr9l" Sep 10 13:31:35.642154 ip-10-0-140-8 kubenswrapper[2585]: E0910 13:31:35.642041 2585 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-xp5dk" podUID="d552154f-1d5b-4c5d-99bf-7aeef22ce032" Sep 10 13:31:35.642154 ip-10-0-140-8 kubenswrapper[2585]: E0910 13:31:35.642110 2585 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-lhr9l" podUID="b37f8a5b-3444-4775-8751-b60c859154d9" Sep 10 13:31:35.642219 ip-10-0-140-8 kubenswrapper[2585]: E0910 13:31:35.642194 2585 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-7262t" podUID="082a7577-a14f-49fb-a9b1-3cf250e6735a" Sep 10 13:31:37.641093 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:31:37.641055 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-xp5dk" Sep 10 13:31:37.641585 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:31:37.641055 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-7262t" Sep 10 13:31:37.641585 ip-10-0-140-8 kubenswrapper[2585]: E0910 13:31:37.641175 2585 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-xp5dk" podUID="d552154f-1d5b-4c5d-99bf-7aeef22ce032" Sep 10 13:31:37.641585 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:31:37.641058 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-lhr9l" Sep 10 13:31:37.641585 ip-10-0-140-8 kubenswrapper[2585]: E0910 13:31:37.641330 2585 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-7262t" podUID="082a7577-a14f-49fb-a9b1-3cf250e6735a" Sep 10 13:31:37.641585 ip-10-0-140-8 kubenswrapper[2585]: E0910 13:31:37.641410 2585 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-lhr9l" podUID="b37f8a5b-3444-4775-8751-b60c859154d9" Sep 10 13:31:39.641275 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:31:39.641225 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-7262t" Sep 10 13:31:39.641275 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:31:39.641256 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-xp5dk" Sep 10 13:31:39.641666 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:31:39.641256 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-lhr9l" Sep 10 13:31:39.641666 ip-10-0-140-8 kubenswrapper[2585]: E0910 13:31:39.641352 2585 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-7262t" podUID="082a7577-a14f-49fb-a9b1-3cf250e6735a" Sep 10 13:31:39.641666 ip-10-0-140-8 kubenswrapper[2585]: E0910 13:31:39.641432 2585 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-xp5dk" podUID="d552154f-1d5b-4c5d-99bf-7aeef22ce032" Sep 10 13:31:39.641666 ip-10-0-140-8 kubenswrapper[2585]: E0910 13:31:39.641486 2585 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-lhr9l" podUID="b37f8a5b-3444-4775-8751-b60c859154d9" Sep 10 13:31:40.640262 ip-10-0-140-8 kubenswrapper[2585]: E0910 13:31:40.640221 2585 kubelet.go:3032] "Container runtime network not ready" networkReady="NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" Sep 10 13:31:41.203100 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:31:41.203025 2585 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-l9l8q_ad555c6a-4385-4f8e-93bb-a174cfaa8e74/kube-multus/2.log" Sep 10 13:31:41.203498 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:31:41.203411 2585 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-l9l8q_ad555c6a-4385-4f8e-93bb-a174cfaa8e74/kube-multus/1.log" Sep 10 13:31:41.203498 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:31:41.203443 2585 generic.go:356] "Generic (PLEG): container finished" podID="ad555c6a-4385-4f8e-93bb-a174cfaa8e74" containerID="035f3abf7ba714be912ecde7271228a45f8e9be8c2ddd63c8e35aa88ba549e17" exitCode=1 Sep 10 13:31:41.203567 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:31:41.203498 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-l9l8q" event={"ID":"ad555c6a-4385-4f8e-93bb-a174cfaa8e74","Type":"ContainerDied","Data":"035f3abf7ba714be912ecde7271228a45f8e9be8c2ddd63c8e35aa88ba549e17"} Sep 10 13:31:41.203567 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:31:41.203545 2585 scope.go:117] "RemoveContainer" containerID="24d6d79549538f1d536272aee5a52d89c1116efcd876fcb7e0148947fd754dd3" Sep 10 13:31:41.203913 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:31:41.203892 2585 scope.go:117] "RemoveContainer" containerID="035f3abf7ba714be912ecde7271228a45f8e9be8c2ddd63c8e35aa88ba549e17" Sep 10 13:31:41.204089 ip-10-0-140-8 kubenswrapper[2585]: E0910 13:31:41.204063 2585 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"kube-multus\" with CrashLoopBackOff: \"back-off 20s restarting failed container=kube-multus pod=multus-l9l8q_openshift-multus(ad555c6a-4385-4f8e-93bb-a174cfaa8e74)\"" pod="openshift-multus/multus-l9l8q" podUID="ad555c6a-4385-4f8e-93bb-a174cfaa8e74" Sep 10 13:31:41.641030 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:31:41.640991 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-7262t" Sep 10 13:31:41.641030 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:31:41.641015 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-xp5dk" Sep 10 13:31:41.641030 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:31:41.640991 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-lhr9l" Sep 10 13:31:41.641347 ip-10-0-140-8 kubenswrapper[2585]: E0910 13:31:41.641173 2585 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-7262t" podUID="082a7577-a14f-49fb-a9b1-3cf250e6735a" Sep 10 13:31:41.641347 ip-10-0-140-8 kubenswrapper[2585]: E0910 13:31:41.641269 2585 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-xp5dk" podUID="d552154f-1d5b-4c5d-99bf-7aeef22ce032" Sep 10 13:31:41.641347 ip-10-0-140-8 kubenswrapper[2585]: E0910 13:31:41.641325 2585 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-lhr9l" podUID="b37f8a5b-3444-4775-8751-b60c859154d9" Sep 10 13:31:42.207067 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:31:42.207043 2585 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-l9l8q_ad555c6a-4385-4f8e-93bb-a174cfaa8e74/kube-multus/2.log" Sep 10 13:31:43.641370 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:31:43.641341 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-lhr9l" Sep 10 13:31:43.641370 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:31:43.641357 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-xp5dk" Sep 10 13:31:43.641769 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:31:43.641348 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-7262t" Sep 10 13:31:43.641769 ip-10-0-140-8 kubenswrapper[2585]: E0910 13:31:43.641444 2585 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-lhr9l" podUID="b37f8a5b-3444-4775-8751-b60c859154d9" Sep 10 13:31:43.641769 ip-10-0-140-8 kubenswrapper[2585]: E0910 13:31:43.641508 2585 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-7262t" podUID="082a7577-a14f-49fb-a9b1-3cf250e6735a" Sep 10 13:31:43.641769 ip-10-0-140-8 kubenswrapper[2585]: E0910 13:31:43.641601 2585 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-xp5dk" podUID="d552154f-1d5b-4c5d-99bf-7aeef22ce032" Sep 10 13:31:45.640791 ip-10-0-140-8 kubenswrapper[2585]: E0910 13:31:45.640729 2585 kubelet.go:3032] "Container runtime network not ready" networkReady="NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" Sep 10 13:31:45.641899 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:31:45.641871 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-7262t" Sep 10 13:31:45.642034 ip-10-0-140-8 kubenswrapper[2585]: E0910 13:31:45.641976 2585 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-7262t" podUID="082a7577-a14f-49fb-a9b1-3cf250e6735a" Sep 10 13:31:45.642150 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:31:45.642114 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-lhr9l" Sep 10 13:31:45.642292 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:31:45.642236 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-xp5dk" Sep 10 13:31:45.642292 ip-10-0-140-8 kubenswrapper[2585]: E0910 13:31:45.642279 2585 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-lhr9l" podUID="b37f8a5b-3444-4775-8751-b60c859154d9" Sep 10 13:31:45.642437 ip-10-0-140-8 kubenswrapper[2585]: E0910 13:31:45.642370 2585 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-xp5dk" podUID="d552154f-1d5b-4c5d-99bf-7aeef22ce032" Sep 10 13:31:45.782341 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:31:45.782309 2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-lhr9l"] Sep 10 13:31:45.792191 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:31:45.792126 2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-xp5dk"] Sep 10 13:31:45.792942 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:31:45.792910 2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-7262t"] Sep 10 13:31:46.217450 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:31:46.217425 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-lhr9l" Sep 10 13:31:46.217629 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:31:46.217425 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-xp5dk" Sep 10 13:31:46.217629 ip-10-0-140-8 kubenswrapper[2585]: E0910 13:31:46.217538 2585 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-lhr9l" podUID="b37f8a5b-3444-4775-8751-b60c859154d9" Sep 10 13:31:46.217629 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:31:46.217425 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-7262t" Sep 10 13:31:46.217629 ip-10-0-140-8 kubenswrapper[2585]: E0910 13:31:46.217608 2585 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-xp5dk" podUID="d552154f-1d5b-4c5d-99bf-7aeef22ce032" Sep 10 13:31:46.217803 ip-10-0-140-8 kubenswrapper[2585]: E0910 13:31:46.217681 2585 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-7262t" podUID="082a7577-a14f-49fb-a9b1-3cf250e6735a" Sep 10 13:31:47.641519 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:31:47.641479 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-lhr9l" Sep 10 13:31:47.641962 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:31:47.641479 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-7262t" Sep 10 13:31:47.641962 ip-10-0-140-8 kubenswrapper[2585]: E0910 13:31:47.641592 2585 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-lhr9l" podUID="b37f8a5b-3444-4775-8751-b60c859154d9" Sep 10 13:31:47.641962 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:31:47.641479 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-xp5dk" Sep 10 13:31:47.641962 ip-10-0-140-8 kubenswrapper[2585]: E0910 13:31:47.641662 2585 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-7262t" podUID="082a7577-a14f-49fb-a9b1-3cf250e6735a" Sep 10 13:31:47.641962 ip-10-0-140-8 kubenswrapper[2585]: E0910 13:31:47.641778 2585 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-xp5dk" podUID="d552154f-1d5b-4c5d-99bf-7aeef22ce032" Sep 10 13:31:49.641439 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:31:49.641399 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-xp5dk" Sep 10 13:31:49.641439 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:31:49.641429 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-lhr9l" Sep 10 13:31:49.641814 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:31:49.641399 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-7262t" Sep 10 13:31:49.641814 ip-10-0-140-8 kubenswrapper[2585]: E0910 13:31:49.641522 2585 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-xp5dk" podUID="d552154f-1d5b-4c5d-99bf-7aeef22ce032" Sep 10 13:31:49.641814 ip-10-0-140-8 kubenswrapper[2585]: E0910 13:31:49.641580 2585 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-7262t" podUID="082a7577-a14f-49fb-a9b1-3cf250e6735a" Sep 10 13:31:49.641814 ip-10-0-140-8 kubenswrapper[2585]: E0910 13:31:49.641670 2585 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-lhr9l" podUID="b37f8a5b-3444-4775-8751-b60c859154d9" Sep 10 13:31:50.641604 ip-10-0-140-8 kubenswrapper[2585]: E0910 13:31:50.641567 2585 kubelet.go:3032] "Container runtime network not ready" networkReady="NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" Sep 10 13:31:51.643714 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:31:51.643678 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-lhr9l" Sep 10 13:31:51.644213 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:31:51.643772 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-7262t" Sep 10 13:31:51.644213 ip-10-0-140-8 kubenswrapper[2585]: E0910 13:31:51.643786 2585 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-lhr9l" podUID="b37f8a5b-3444-4775-8751-b60c859154d9" Sep 10 13:31:51.644213 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:31:51.643803 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-xp5dk" Sep 10 13:31:51.644213 ip-10-0-140-8 kubenswrapper[2585]: E0910 13:31:51.643907 2585 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-7262t" podUID="082a7577-a14f-49fb-a9b1-3cf250e6735a" Sep 10 13:31:51.644213 ip-10-0-140-8 kubenswrapper[2585]: E0910 13:31:51.643964 2585 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-xp5dk" podUID="d552154f-1d5b-4c5d-99bf-7aeef22ce032" Sep 10 13:31:53.641338 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:31:53.641304 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-lhr9l" Sep 10 13:31:53.641338 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:31:53.641326 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-xp5dk" Sep 10 13:31:53.641732 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:31:53.641326 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-7262t" Sep 10 13:31:53.641732 ip-10-0-140-8 kubenswrapper[2585]: E0910 13:31:53.641454 2585 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-lhr9l" podUID="b37f8a5b-3444-4775-8751-b60c859154d9" Sep 10 13:31:53.641732 ip-10-0-140-8 kubenswrapper[2585]: E0910 13:31:53.641482 2585 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-xp5dk" podUID="d552154f-1d5b-4c5d-99bf-7aeef22ce032" Sep 10 13:31:53.641732 ip-10-0-140-8 kubenswrapper[2585]: E0910 13:31:53.641541 2585 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-7262t" podUID="082a7577-a14f-49fb-a9b1-3cf250e6735a" Sep 10 13:31:55.641925 ip-10-0-140-8 kubenswrapper[2585]: E0910 13:31:55.641888 2585 kubelet.go:3032] "Container runtime network not ready" networkReady="NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" Sep 10 13:31:55.641925 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:31:55.641919 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-lhr9l" Sep 10 13:31:55.642428 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:31:55.641995 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-xp5dk" Sep 10 13:31:55.642428 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:31:55.642027 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-7262t" Sep 10 13:31:55.642428 ip-10-0-140-8 kubenswrapper[2585]: E0910 13:31:55.642090 2585 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-lhr9l" podUID="b37f8a5b-3444-4775-8751-b60c859154d9" Sep 10 13:31:55.642428 ip-10-0-140-8 kubenswrapper[2585]: E0910 13:31:55.642146 2585 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-7262t" podUID="082a7577-a14f-49fb-a9b1-3cf250e6735a" Sep 10 13:31:55.642428 ip-10-0-140-8 kubenswrapper[2585]: E0910 13:31:55.642263 2585 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-xp5dk" podUID="d552154f-1d5b-4c5d-99bf-7aeef22ce032" Sep 10 13:31:56.640921 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:31:56.640888 2585 scope.go:117] "RemoveContainer" containerID="035f3abf7ba714be912ecde7271228a45f8e9be8c2ddd63c8e35aa88ba549e17" Sep 10 13:31:56.641094 ip-10-0-140-8 kubenswrapper[2585]: E0910 13:31:56.641027 2585 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"kube-multus\" with CrashLoopBackOff: \"back-off 20s restarting failed container=kube-multus pod=multus-l9l8q_openshift-multus(ad555c6a-4385-4f8e-93bb-a174cfaa8e74)\"" pod="openshift-multus/multus-l9l8q" podUID="ad555c6a-4385-4f8e-93bb-a174cfaa8e74" Sep 10 13:31:57.641216 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:31:57.641173 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-lhr9l" Sep 10 13:31:57.641643 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:31:57.641173 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-7262t" Sep 10 13:31:57.641643 ip-10-0-140-8 kubenswrapper[2585]: E0910 13:31:57.641331 2585 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-lhr9l" podUID="b37f8a5b-3444-4775-8751-b60c859154d9" Sep 10 13:31:57.641643 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:31:57.641173 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-xp5dk" Sep 10 13:31:57.641643 ip-10-0-140-8 kubenswrapper[2585]: E0910 13:31:57.641374 2585 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-7262t" podUID="082a7577-a14f-49fb-a9b1-3cf250e6735a" Sep 10 13:31:57.641643 ip-10-0-140-8 kubenswrapper[2585]: E0910 13:31:57.641474 2585 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-xp5dk" podUID="d552154f-1d5b-4c5d-99bf-7aeef22ce032" Sep 10 13:31:58.651194 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:31:58.651167 2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-qz8x4" Sep 10 13:31:59.641117 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:31:59.641082 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-xp5dk" Sep 10 13:31:59.641349 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:31:59.641083 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-lhr9l" Sep 10 13:31:59.641349 ip-10-0-140-8 kubenswrapper[2585]: E0910 13:31:59.641200 2585 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-xp5dk" podUID="d552154f-1d5b-4c5d-99bf-7aeef22ce032" Sep 10 13:31:59.641349 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:31:59.641082 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-7262t" Sep 10 13:31:59.641349 ip-10-0-140-8 kubenswrapper[2585]: E0910 13:31:59.641281 2585 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-lhr9l" podUID="b37f8a5b-3444-4775-8751-b60c859154d9" Sep 10 13:31:59.641494 ip-10-0-140-8 kubenswrapper[2585]: E0910 13:31:59.641358 2585 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-7262t" podUID="082a7577-a14f-49fb-a9b1-3cf250e6735a" Sep 10 13:32:00.049667 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:32:00.049579 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/b37f8a5b-3444-4775-8751-b60c859154d9-original-pull-secret\") pod \"global-pull-secret-syncer-lhr9l\" (UID: \"b37f8a5b-3444-4775-8751-b60c859154d9\") " pod="kube-system/global-pull-secret-syncer-lhr9l" Sep 10 13:32:00.050070 ip-10-0-140-8 kubenswrapper[2585]: E0910 13:32:00.049706 2585 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Sep 10 13:32:00.050070 ip-10-0-140-8 kubenswrapper[2585]: E0910 13:32:00.049758 2585 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b37f8a5b-3444-4775-8751-b60c859154d9-original-pull-secret podName:b37f8a5b-3444-4775-8751-b60c859154d9 nodeName:}" failed. No retries permitted until 2026-09-10 13:33:04.049743654 +0000 UTC m=+268.922447834 (durationBeforeRetry 1m4s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/b37f8a5b-3444-4775-8751-b60c859154d9-original-pull-secret") pod "global-pull-secret-syncer-lhr9l" (UID: "b37f8a5b-3444-4775-8751-b60c859154d9") : object "kube-system"/"original-pull-secret" not registered Sep 10 13:32:00.576899 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:32:00.576868 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fxgmp_fa53c783-d40a-4024-9f7b-1d0d2b57f6dc/konnectivity-agent/0.log" limit=1024 Sep 10 13:32:00.642736 ip-10-0-140-8 kubenswrapper[2585]: E0910 13:32:00.642707 2585 kubelet.go:3032] "Container runtime network not ready" networkReady="NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" Sep 10 13:32:00.842081 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:32:00.842005 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fxgmp_fa53c783-d40a-4024-9f7b-1d0d2b57f6dc/konnectivity-agent/0.log" limit=1024 Sep 10 13:32:01.641523 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:32:01.641485 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-7262t" Sep 10 13:32:01.641826 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:32:01.641533 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-xp5dk" Sep 10 13:32:01.641826 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:32:01.641505 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-lhr9l" Sep 10 13:32:01.641826 ip-10-0-140-8 kubenswrapper[2585]: E0910 13:32:01.641612 2585 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-7262t" podUID="082a7577-a14f-49fb-a9b1-3cf250e6735a" Sep 10 13:32:01.641826 ip-10-0-140-8 kubenswrapper[2585]: E0910 13:32:01.641663 2585 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-lhr9l" podUID="b37f8a5b-3444-4775-8751-b60c859154d9" Sep 10 13:32:01.641826 ip-10-0-140-8 kubenswrapper[2585]: E0910 13:32:01.641726 2585 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-xp5dk" podUID="d552154f-1d5b-4c5d-99bf-7aeef22ce032" Sep 10 13:32:01.960487 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:32:01.960404 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fxgmp_fa53c783-d40a-4024-9f7b-1d0d2b57f6dc/konnectivity-agent/0.log" limit=1024 Sep 10 13:32:02.272316 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:32:02.272223 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fxgmp_fa53c783-d40a-4024-9f7b-1d0d2b57f6dc/konnectivity-agent/0.log" limit=1024 Sep 10 13:32:02.400957 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:32:02.400927 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fxgmp_fa53c783-d40a-4024-9f7b-1d0d2b57f6dc/konnectivity-agent/0.log" limit=1024 Sep 10 13:32:02.494610 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:32:02.494584 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fxgmp_fa53c783-d40a-4024-9f7b-1d0d2b57f6dc/konnectivity-agent/0.log" limit=1024 Sep 10 13:32:02.716167 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:32:02.716129 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fxgmp_fa53c783-d40a-4024-9f7b-1d0d2b57f6dc/konnectivity-agent/0.log" limit=1024 Sep 10 13:32:03.640699 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:32:03.640663 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-7262t" Sep 10 13:32:03.640869 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:32:03.640663 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-lhr9l" Sep 10 13:32:03.640869 ip-10-0-140-8 kubenswrapper[2585]: E0910 13:32:03.640806 2585 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-7262t" podUID="082a7577-a14f-49fb-a9b1-3cf250e6735a" Sep 10 13:32:03.640869 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:32:03.640665 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-xp5dk" Sep 10 13:32:03.640974 ip-10-0-140-8 kubenswrapper[2585]: E0910 13:32:03.640861 2585 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-lhr9l" podUID="b37f8a5b-3444-4775-8751-b60c859154d9" Sep 10 13:32:03.640974 ip-10-0-140-8 kubenswrapper[2585]: E0910 13:32:03.640931 2585 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-xp5dk" podUID="d552154f-1d5b-4c5d-99bf-7aeef22ce032" Sep 10 13:32:04.365410 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:32:04.365379 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fxgmp_fa53c783-d40a-4024-9f7b-1d0d2b57f6dc/konnectivity-agent/0.log" limit=1024 Sep 10 13:32:04.470039 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:32:04.470011 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fxgmp_fa53c783-d40a-4024-9f7b-1d0d2b57f6dc/konnectivity-agent/0.log" limit=1024 Sep 10 13:32:05.641474 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:32:05.641448 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-xp5dk" Sep 10 13:32:05.641899 ip-10-0-140-8 kubenswrapper[2585]: E0910 13:32:05.641549 2585 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-xp5dk" podUID="d552154f-1d5b-4c5d-99bf-7aeef22ce032" Sep 10 13:32:05.641899 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:32:05.641583 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-lhr9l" Sep 10 13:32:05.641899 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:32:05.641691 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-7262t" Sep 10 13:32:05.641899 ip-10-0-140-8 kubenswrapper[2585]: E0910 13:32:05.641695 2585 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-lhr9l" podUID="b37f8a5b-3444-4775-8751-b60c859154d9" Sep 10 13:32:05.641899 ip-10-0-140-8 kubenswrapper[2585]: E0910 13:32:05.641760 2585 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-7262t" podUID="082a7577-a14f-49fb-a9b1-3cf250e6735a" Sep 10 13:32:05.643176 ip-10-0-140-8 kubenswrapper[2585]: E0910 13:32:05.643153 2585 kubelet.go:3032] "Container runtime network not ready" networkReady="NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" Sep 10 13:32:06.889036 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:32:06.889003 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fxgmp_fa53c783-d40a-4024-9f7b-1d0d2b57f6dc/konnectivity-agent/0.log" limit=1024 Sep 10 13:32:07.039028 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:32:07.038996 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fxgmp_fa53c783-d40a-4024-9f7b-1d0d2b57f6dc/konnectivity-agent/0.log" limit=1024 Sep 10 13:32:07.164276 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:32:07.164165 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fxgmp_fa53c783-d40a-4024-9f7b-1d0d2b57f6dc/konnectivity-agent/0.log" limit=1024 Sep 10 13:32:07.591606 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:32:07.591569 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fxgmp_fa53c783-d40a-4024-9f7b-1d0d2b57f6dc/konnectivity-agent/0.log" limit=1024 Sep 10 13:32:07.640872 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:32:07.640846 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-7262t" Sep 10 13:32:07.640999 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:32:07.640856 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-xp5dk" Sep 10 13:32:07.640999 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:32:07.640969 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-lhr9l" Sep 10 13:32:07.640999 ip-10-0-140-8 kubenswrapper[2585]: E0910 13:32:07.640976 2585 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-7262t" podUID="082a7577-a14f-49fb-a9b1-3cf250e6735a" Sep 10 13:32:07.641154 ip-10-0-140-8 kubenswrapper[2585]: E0910 13:32:07.641045 2585 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-xp5dk" podUID="d552154f-1d5b-4c5d-99bf-7aeef22ce032" Sep 10 13:32:07.641154 ip-10-0-140-8 kubenswrapper[2585]: E0910 13:32:07.641132 2585 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-lhr9l" podUID="b37f8a5b-3444-4775-8751-b60c859154d9" Sep 10 13:32:07.713205 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:32:07.713177 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fxgmp_fa53c783-d40a-4024-9f7b-1d0d2b57f6dc/konnectivity-agent/0.log" limit=1024 Sep 10 13:32:08.641758 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:32:08.641727 2585 scope.go:117] "RemoveContainer" containerID="035f3abf7ba714be912ecde7271228a45f8e9be8c2ddd63c8e35aa88ba549e17" Sep 10 13:32:09.273356 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:32:09.273286 2585 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-l9l8q_ad555c6a-4385-4f8e-93bb-a174cfaa8e74/kube-multus/2.log" Sep 10 13:32:09.273490 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:32:09.273362 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-l9l8q" event={"ID":"ad555c6a-4385-4f8e-93bb-a174cfaa8e74","Type":"ContainerStarted","Data":"efb8bd45c072e51bfec4e79109095ea9300bb57f7bd2301eb5ba04005a6bfb1a"} Sep 10 13:32:09.492060 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:32:09.492026 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fxgmp_fa53c783-d40a-4024-9f7b-1d0d2b57f6dc/konnectivity-agent/0.log" limit=1024 Sep 10 13:32:09.614092 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:32:09.614058 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fxgmp_fa53c783-d40a-4024-9f7b-1d0d2b57f6dc/konnectivity-agent/0.log" limit=1024 Sep 10 13:32:09.641532 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:32:09.641511 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-xp5dk" Sep 10 13:32:09.641648 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:32:09.641511 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-lhr9l" Sep 10 13:32:09.641648 ip-10-0-140-8 kubenswrapper[2585]: E0910 13:32:09.641612 2585 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-xp5dk" podUID="d552154f-1d5b-4c5d-99bf-7aeef22ce032" Sep 10 13:32:09.641735 ip-10-0-140-8 kubenswrapper[2585]: E0910 13:32:09.641670 2585 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-lhr9l" podUID="b37f8a5b-3444-4775-8751-b60c859154d9" Sep 10 13:32:09.641735 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:32:09.641517 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-7262t" Sep 10 13:32:09.641735 ip-10-0-140-8 kubenswrapper[2585]: E0910 13:32:09.641731 2585 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-7262t" podUID="082a7577-a14f-49fb-a9b1-3cf250e6735a" Sep 10 13:32:09.829414 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:32:09.829380 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fxgmp_fa53c783-d40a-4024-9f7b-1d0d2b57f6dc/konnectivity-agent/0.log" limit=1024 Sep 10 13:32:10.021161 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:32:10.021084 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fxgmp_fa53c783-d40a-4024-9f7b-1d0d2b57f6dc/konnectivity-agent/0.log" limit=1024 Sep 10 13:32:11.640862 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:32:11.640824 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-lhr9l" Sep 10 13:32:11.641281 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:32:11.640824 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-xp5dk" Sep 10 13:32:11.641281 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:32:11.640834 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-7262t" Sep 10 13:32:11.643959 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:32:11.643935 2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\"" Sep 10 13:32:11.644082 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:32:11.643947 2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-4vqdt\"" Sep 10 13:32:11.644895 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:32:11.644874 2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\"" Sep 10 13:32:11.645003 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:32:11.644929 2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\"" Sep 10 13:32:11.645070 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:32:11.645024 2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\"" Sep 10 13:32:11.645070 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:32:11.645024 2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-cdbpg\"" Sep 10 13:32:11.773208 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:32:11.773173 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fxgmp_fa53c783-d40a-4024-9f7b-1d0d2b57f6dc/konnectivity-agent/0.log" limit=1024 Sep 10 13:32:11.916964 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:32:11.916881 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fxgmp_fa53c783-d40a-4024-9f7b-1d0d2b57f6dc/konnectivity-agent/0.log" limit=1024 Sep 10 13:32:12.820810 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:32:12.820780 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fxgmp_fa53c783-d40a-4024-9f7b-1d0d2b57f6dc/konnectivity-agent/0.log" limit=1024 Sep 10 13:32:12.932833 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:32:12.932795 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fxgmp_fa53c783-d40a-4024-9f7b-1d0d2b57f6dc/konnectivity-agent/0.log" limit=1024 Sep 10 13:32:13.276106 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:32:13.276011 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fxgmp_fa53c783-d40a-4024-9f7b-1d0d2b57f6dc/konnectivity-agent/0.log" limit=1024 Sep 10 13:32:13.454756 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:32:13.454709 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fxgmp_fa53c783-d40a-4024-9f7b-1d0d2b57f6dc/konnectivity-agent/0.log" limit=1024 Sep 10 13:32:13.909492 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:32:13.909336 2585 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-84qhc_57474dbb-5547-4132-9d67-dacfb20ea367/dns-node-resolver/0.log" Sep 10 13:32:14.349863 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:32:14.349819 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fxgmp_fa53c783-d40a-4024-9f7b-1d0d2b57f6dc/konnectivity-agent/0.log" limit=1024 Sep 10 13:32:14.503651 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:32:14.503594 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fxgmp_fa53c783-d40a-4024-9f7b-1d0d2b57f6dc/konnectivity-agent/0.log" limit=1024 Sep 10 13:32:14.647651 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:32:14.647559 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fxgmp_fa53c783-d40a-4024-9f7b-1d0d2b57f6dc/konnectivity-agent/0.log" limit=1024 Sep 10 13:32:14.752041 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:32:14.752011 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fxgmp_fa53c783-d40a-4024-9f7b-1d0d2b57f6dc/konnectivity-agent/0.log" limit=1024 Sep 10 13:32:14.980504 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:32:14.980424 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fxgmp_fa53c783-d40a-4024-9f7b-1d0d2b57f6dc/konnectivity-agent/0.log" limit=1024 Sep 10 13:32:15.115025 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:32:15.114992 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fxgmp_fa53c783-d40a-4024-9f7b-1d0d2b57f6dc/konnectivity-agent/0.log" limit=1024 Sep 10 13:32:15.225065 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:32:15.225032 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fxgmp_fa53c783-d40a-4024-9f7b-1d0d2b57f6dc/konnectivity-agent/0.log" limit=1024 Sep 10 13:32:15.309683 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:32:15.309660 2585 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-zhn9x_1748fe6b-cce2-42ce-b8a7-10d12ffef95a/node-ca/0.log" Sep 10 13:32:15.338890 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:32:15.338865 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fxgmp_fa53c783-d40a-4024-9f7b-1d0d2b57f6dc/konnectivity-agent/0.log" limit=1024 Sep 10 13:32:15.753079 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:32:15.752993 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fxgmp_fa53c783-d40a-4024-9f7b-1d0d2b57f6dc/konnectivity-agent/0.log" limit=1024 Sep 10 13:32:16.383196 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:32:16.383164 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fxgmp_fa53c783-d40a-4024-9f7b-1d0d2b57f6dc/konnectivity-agent/0.log" limit=1024 Sep 10 13:32:16.957891 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:32:16.957855 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fxgmp_fa53c783-d40a-4024-9f7b-1d0d2b57f6dc/konnectivity-agent/0.log" limit=1024 Sep 10 13:32:17.153265 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:32:17.153210 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fxgmp_fa53c783-d40a-4024-9f7b-1d0d2b57f6dc/konnectivity-agent/0.log" limit=1024 Sep 10 13:32:17.403196 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:32:17.403157 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fxgmp_fa53c783-d40a-4024-9f7b-1d0d2b57f6dc/konnectivity-agent/0.log" limit=1024 Sep 10 13:32:17.794104 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:32:17.794017 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fxgmp_fa53c783-d40a-4024-9f7b-1d0d2b57f6dc/konnectivity-agent/0.log" limit=1024 Sep 10 13:32:17.956649 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:32:17.956608 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fxgmp_fa53c783-d40a-4024-9f7b-1d0d2b57f6dc/konnectivity-agent/0.log" limit=1024 Sep 10 13:32:18.206299 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:32:18.206269 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fxgmp_fa53c783-d40a-4024-9f7b-1d0d2b57f6dc/konnectivity-agent/0.log" limit=1024 Sep 10 13:32:18.590999 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:32:18.590973 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fxgmp_fa53c783-d40a-4024-9f7b-1d0d2b57f6dc/konnectivity-agent/0.log" limit=1024 Sep 10 13:32:18.709218 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:32:18.709192 2585 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-vlmmc_9913a6bd-8e01-4034-9da7-f7d0b65a7eee/egress-router-binary-copy/0.log" Sep 10 13:32:18.742320 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:32:18.742297 2585 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-8.ec2.internal" event="NodeReady" Sep 10 13:32:18.745679 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:32:18.745655 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fxgmp_fa53c783-d40a-4024-9f7b-1d0d2b57f6dc/konnectivity-agent/0.log" limit=1024 Sep 10 13:32:18.909112 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:32:18.909036 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fxgmp_fa53c783-d40a-4024-9f7b-1d0d2b57f6dc/konnectivity-agent/0.log" limit=1024 Sep 10 13:32:18.912331 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:32:18.912314 2585 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-vlmmc_9913a6bd-8e01-4034-9da7-f7d0b65a7eee/cni-plugins/0.log" Sep 10 13:32:19.111473 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:32:19.111441 2585 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-vlmmc_9913a6bd-8e01-4034-9da7-f7d0b65a7eee/bond-cni-plugin/0.log" Sep 10 13:32:19.151040 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:32:19.151003 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fxgmp_fa53c783-d40a-4024-9f7b-1d0d2b57f6dc/konnectivity-agent/0.log" limit=1024 Sep 10 13:32:19.262158 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:32:19.262072 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fxgmp_fa53c783-d40a-4024-9f7b-1d0d2b57f6dc/konnectivity-agent/0.log" limit=1024 Sep 10 13:32:19.322398 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:32:19.322364 2585 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-vlmmc_9913a6bd-8e01-4034-9da7-f7d0b65a7eee/routeoverride-cni/0.log" Sep 10 13:32:19.508620 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:32:19.508581 2585 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-vlmmc_9913a6bd-8e01-4034-9da7-f7d0b65a7eee/whereabouts-cni-bincopy/0.log" Sep 10 13:32:19.565912 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:32:19.565883 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fxgmp_fa53c783-d40a-4024-9f7b-1d0d2b57f6dc/konnectivity-agent/0.log" limit=1024 Sep 10 13:32:19.711266 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:32:19.711214 2585 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-vlmmc_9913a6bd-8e01-4034-9da7-f7d0b65a7eee/whereabouts-cni/0.log" Sep 10 13:32:19.745464 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:32:19.745434 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fxgmp_fa53c783-d40a-4024-9f7b-1d0d2b57f6dc/konnectivity-agent/0.log" limit=1024 Sep 10 13:32:19.908818 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:32:19.908744 2585 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-vlmmc_9913a6bd-8e01-4034-9da7-f7d0b65a7eee/kube-multus-additional-cni-plugins/0.log" Sep 10 13:32:20.433437 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:32:20.433405 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fxgmp_fa53c783-d40a-4024-9f7b-1d0d2b57f6dc/konnectivity-agent/0.log" limit=1024 Sep 10 13:32:20.573864 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:32:20.573830 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fxgmp_fa53c783-d40a-4024-9f7b-1d0d2b57f6dc/konnectivity-agent/0.log" limit=1024 Sep 10 13:32:20.711110 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:32:20.711020 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fxgmp_fa53c783-d40a-4024-9f7b-1d0d2b57f6dc/konnectivity-agent/0.log" limit=1024 Sep 10 13:32:20.910547 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:32:20.910515 2585 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-l9l8q_ad555c6a-4385-4f8e-93bb-a174cfaa8e74/kube-multus/2.log" Sep 10 13:32:21.109686 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:32:21.109652 2585 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-l9l8q_ad555c6a-4385-4f8e-93bb-a174cfaa8e74/kube-multus/3.log" Sep 10 13:32:21.154850 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:32:21.154813 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fxgmp_fa53c783-d40a-4024-9f7b-1d0d2b57f6dc/konnectivity-agent/0.log" limit=1024 Sep 10 13:32:21.367947 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:32:21.367859 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fxgmp_fa53c783-d40a-4024-9f7b-1d0d2b57f6dc/konnectivity-agent/0.log" limit=1024 Sep 10 13:32:21.957980 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:32:21.957945 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fxgmp_fa53c783-d40a-4024-9f7b-1d0d2b57f6dc/konnectivity-agent/0.log" limit=1024 Sep 10 13:32:22.355996 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:32:22.355967 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fxgmp_fa53c783-d40a-4024-9f7b-1d0d2b57f6dc/konnectivity-agent/0.log" limit=1024 Sep 10 13:32:22.577255 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:32:22.577209 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fxgmp_fa53c783-d40a-4024-9f7b-1d0d2b57f6dc/konnectivity-agent/0.log" limit=1024 Sep 10 13:32:22.944889 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:32:22.944857 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fxgmp_fa53c783-d40a-4024-9f7b-1d0d2b57f6dc/konnectivity-agent/0.log" limit=1024 Sep 10 13:32:24.073194 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:32:24.073156 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fxgmp_fa53c783-d40a-4024-9f7b-1d0d2b57f6dc/konnectivity-agent/0.log" limit=1024 Sep 10 13:32:24.112954 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:32:24.112926 2585 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-qz8x4_6c90873d-f3b0-42a4-adc8-6a644a1f2d11/ovn-controller/0.log" Sep 10 13:32:24.310053 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:32:24.310024 2585 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-qz8x4_6c90873d-f3b0-42a4-adc8-6a644a1f2d11/ovn-acl-logging/0.log" Sep 10 13:32:24.330275 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:32:24.330203 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fxgmp_fa53c783-d40a-4024-9f7b-1d0d2b57f6dc/konnectivity-agent/0.log" limit=1024 Sep 10 13:32:24.511334 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:32:24.511304 2585 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-qz8x4_6c90873d-f3b0-42a4-adc8-6a644a1f2d11/kube-rbac-proxy-node/0.log" Sep 10 13:32:24.513974 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:32:24.513951 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fxgmp_fa53c783-d40a-4024-9f7b-1d0d2b57f6dc/konnectivity-agent/0.log" limit=1024 Sep 10 13:32:24.659222 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:32:24.659138 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fxgmp_fa53c783-d40a-4024-9f7b-1d0d2b57f6dc/konnectivity-agent/0.log" limit=1024 Sep 10 13:32:24.711048 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:32:24.711017 2585 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-qz8x4_6c90873d-f3b0-42a4-adc8-6a644a1f2d11/kube-rbac-proxy-ovn-metrics/0.log" Sep 10 13:32:24.799165 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:32:24.799134 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fxgmp_fa53c783-d40a-4024-9f7b-1d0d2b57f6dc/konnectivity-agent/0.log" limit=1024 Sep 10 13:32:24.911364 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:32:24.911289 2585 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-qz8x4_6c90873d-f3b0-42a4-adc8-6a644a1f2d11/northd/0.log" Sep 10 13:32:25.113409 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:32:25.113383 2585 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-qz8x4_6c90873d-f3b0-42a4-adc8-6a644a1f2d11/nbdb/0.log" Sep 10 13:32:25.282195 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:32:25.282116 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fxgmp_fa53c783-d40a-4024-9f7b-1d0d2b57f6dc/konnectivity-agent/0.log" limit=1024 Sep 10 13:32:25.311159 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:32:25.311134 2585 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-qz8x4_6c90873d-f3b0-42a4-adc8-6a644a1f2d11/sbdb/0.log" Sep 10 13:32:25.511638 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:32:25.511610 2585 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-qz8x4_6c90873d-f3b0-42a4-adc8-6a644a1f2d11/ovnkube-controller/0.log" Sep 10 13:32:25.842674 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:32:25.842637 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fxgmp_fa53c783-d40a-4024-9f7b-1d0d2b57f6dc/konnectivity-agent/0.log" limit=1024 Sep 10 13:32:26.367091 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:32:26.367060 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fxgmp_fa53c783-d40a-4024-9f7b-1d0d2b57f6dc/konnectivity-agent/0.log" limit=1024 Sep 10 13:32:28.713208 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:32:28.713184 2585 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-network-operator_iptables-alerter-x2wmd_c366889d-cdc5-4a43-af92-806d84e99e45/iptables-alerter/0.log" Sep 10 13:32:29.686306 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:32:29.686275 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fxgmp_fa53c783-d40a-4024-9f7b-1d0d2b57f6dc/konnectivity-agent/0.log" limit=1024 Sep 10 13:32:32.319353 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:32:32.319322 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fxgmp_fa53c783-d40a-4024-9f7b-1d0d2b57f6dc/konnectivity-agent/0.log" limit=1024 Sep 10 13:32:32.428872 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:32:32.428824 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fxgmp_fa53c783-d40a-4024-9f7b-1d0d2b57f6dc/konnectivity-agent/0.log" limit=1024 Sep 10 13:32:34.053547 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:32:34.053515 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fxgmp_fa53c783-d40a-4024-9f7b-1d0d2b57f6dc/konnectivity-agent/0.log" limit=1024 Sep 10 13:32:34.152070 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:32:34.152042 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fxgmp_fa53c783-d40a-4024-9f7b-1d0d2b57f6dc/konnectivity-agent/0.log" limit=1024 Sep 10 13:32:35.805310 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:32:35.805239 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fxgmp_fa53c783-d40a-4024-9f7b-1d0d2b57f6dc/konnectivity-agent/0.log" limit=1024 Sep 10 13:32:36.027645 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:32:36.027598 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fxgmp_fa53c783-d40a-4024-9f7b-1d0d2b57f6dc/konnectivity-agent/0.log" limit=1024 Sep 10 13:32:36.327545 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:32:36.327500 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fxgmp_fa53c783-d40a-4024-9f7b-1d0d2b57f6dc/konnectivity-agent/0.log" limit=1024 Sep 10 13:32:37.142358 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:32:37.142316 2585 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-runtime-extractor-5ftzf"] Sep 10 13:32:37.145293 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:32:37.145276 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-5ftzf" Sep 10 13:32:37.148315 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:32:37.148288 2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\"" Sep 10 13:32:37.148470 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:32:37.148348 2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\"" Sep 10 13:32:37.148537 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:32:37.148488 2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-n9gkj\"" Sep 10 13:32:37.148628 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:32:37.148425 2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\"" Sep 10 13:32:37.149460 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:32:37.149435 2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\"" Sep 10 13:32:37.159398 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:32:37.159378 2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-5ftzf"] Sep 10 13:32:37.207345 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:32:37.207308 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/abeadb08-451a-477c-bff5-0cb019997ba4-crio-socket\") pod \"insights-runtime-extractor-5ftzf\" (UID: \"abeadb08-451a-477c-bff5-0cb019997ba4\") " pod="openshift-insights/insights-runtime-extractor-5ftzf" Sep 10 13:32:37.207345 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:32:37.207343 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/abeadb08-451a-477c-bff5-0cb019997ba4-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-5ftzf\" (UID: \"abeadb08-451a-477c-bff5-0cb019997ba4\") " pod="openshift-insights/insights-runtime-extractor-5ftzf" Sep 10 13:32:37.207553 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:32:37.207368 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-clqlh\" (UniqueName: \"kubernetes.io/projected/abeadb08-451a-477c-bff5-0cb019997ba4-kube-api-access-clqlh\") pod \"insights-runtime-extractor-5ftzf\" (UID: \"abeadb08-451a-477c-bff5-0cb019997ba4\") " pod="openshift-insights/insights-runtime-extractor-5ftzf" Sep 10 13:32:37.207553 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:32:37.207445 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/abeadb08-451a-477c-bff5-0cb019997ba4-data-volume\") pod \"insights-runtime-extractor-5ftzf\" (UID: \"abeadb08-451a-477c-bff5-0cb019997ba4\") " pod="openshift-insights/insights-runtime-extractor-5ftzf" Sep 10 13:32:37.207553 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:32:37.207520 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/abeadb08-451a-477c-bff5-0cb019997ba4-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-5ftzf\" (UID: \"abeadb08-451a-477c-bff5-0cb019997ba4\") " pod="openshift-insights/insights-runtime-extractor-5ftzf" Sep 10 13:32:37.217943 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:32:37.217917 2585 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress-canary/ingress-canary-vpwqf"] Sep 10 13:32:37.220649 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:32:37.220635 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-vpwqf" Sep 10 13:32:37.223150 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:32:37.223126 2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\"" Sep 10 13:32:37.223292 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:32:37.223274 2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\"" Sep 10 13:32:37.223357 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:32:37.223311 2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"default-dockercfg-xxdkm\"" Sep 10 13:32:37.223357 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:32:37.223315 2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\"" Sep 10 13:32:37.229231 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:32:37.229209 2585 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-dns/dns-default-l2ks7"] Sep 10 13:32:37.232035 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:32:37.232018 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-l2ks7" Sep 10 13:32:37.234442 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:32:37.234410 2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-m68qf\"" Sep 10 13:32:37.234546 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:32:37.234448 2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\"" Sep 10 13:32:37.234609 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:32:37.234591 2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\"" Sep 10 13:32:37.237444 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:32:37.237424 2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-vpwqf"] Sep 10 13:32:37.252440 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:32:37.252417 2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-l2ks7"] Sep 10 13:32:37.308436 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:32:37.308405 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/abeadb08-451a-477c-bff5-0cb019997ba4-crio-socket\") pod \"insights-runtime-extractor-5ftzf\" (UID: \"abeadb08-451a-477c-bff5-0cb019997ba4\") " pod="openshift-insights/insights-runtime-extractor-5ftzf" Sep 10 13:32:37.308436 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:32:37.308437 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/abeadb08-451a-477c-bff5-0cb019997ba4-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-5ftzf\" (UID: \"abeadb08-451a-477c-bff5-0cb019997ba4\") " pod="openshift-insights/insights-runtime-extractor-5ftzf" Sep 10 13:32:37.308655 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:32:37.308454 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-clqlh\" (UniqueName: \"kubernetes.io/projected/abeadb08-451a-477c-bff5-0cb019997ba4-kube-api-access-clqlh\") pod \"insights-runtime-extractor-5ftzf\" (UID: \"abeadb08-451a-477c-bff5-0cb019997ba4\") " pod="openshift-insights/insights-runtime-extractor-5ftzf" Sep 10 13:32:37.308655 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:32:37.308474 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4wvk8\" (UniqueName: \"kubernetes.io/projected/b308bee9-e8b3-4e7b-98b1-c2f6d833b7a3-kube-api-access-4wvk8\") pod \"ingress-canary-vpwqf\" (UID: \"b308bee9-e8b3-4e7b-98b1-c2f6d833b7a3\") " pod="openshift-ingress-canary/ingress-canary-vpwqf" Sep 10 13:32:37.308655 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:32:37.308500 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/a572632c-303d-4828-a58a-e1c230b6e301-metrics-tls\") pod \"dns-default-l2ks7\" (UID: \"a572632c-303d-4828-a58a-e1c230b6e301\") " pod="openshift-dns/dns-default-l2ks7" Sep 10 13:32:37.308655 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:32:37.308516 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/abeadb08-451a-477c-bff5-0cb019997ba4-data-volume\") pod \"insights-runtime-extractor-5ftzf\" (UID: \"abeadb08-451a-477c-bff5-0cb019997ba4\") " pod="openshift-insights/insights-runtime-extractor-5ftzf" Sep 10 13:32:37.308655 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:32:37.308535 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/abeadb08-451a-477c-bff5-0cb019997ba4-crio-socket\") pod \"insights-runtime-extractor-5ftzf\" (UID: \"abeadb08-451a-477c-bff5-0cb019997ba4\") " pod="openshift-insights/insights-runtime-extractor-5ftzf" Sep 10 13:32:37.308655 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:32:37.308549 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/b308bee9-e8b3-4e7b-98b1-c2f6d833b7a3-cert\") pod \"ingress-canary-vpwqf\" (UID: \"b308bee9-e8b3-4e7b-98b1-c2f6d833b7a3\") " pod="openshift-ingress-canary/ingress-canary-vpwqf" Sep 10 13:32:37.308943 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:32:37.308682 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8hn9q\" (UniqueName: \"kubernetes.io/projected/a572632c-303d-4828-a58a-e1c230b6e301-kube-api-access-8hn9q\") pod \"dns-default-l2ks7\" (UID: \"a572632c-303d-4828-a58a-e1c230b6e301\") " pod="openshift-dns/dns-default-l2ks7" Sep 10 13:32:37.308943 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:32:37.308726 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/abeadb08-451a-477c-bff5-0cb019997ba4-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-5ftzf\" (UID: \"abeadb08-451a-477c-bff5-0cb019997ba4\") " pod="openshift-insights/insights-runtime-extractor-5ftzf" Sep 10 13:32:37.308943 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:32:37.308762 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/a572632c-303d-4828-a58a-e1c230b6e301-config-volume\") pod \"dns-default-l2ks7\" (UID: \"a572632c-303d-4828-a58a-e1c230b6e301\") " pod="openshift-dns/dns-default-l2ks7" Sep 10 13:32:37.308943 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:32:37.308827 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a572632c-303d-4828-a58a-e1c230b6e301-tmp-dir\") pod \"dns-default-l2ks7\" (UID: \"a572632c-303d-4828-a58a-e1c230b6e301\") " pod="openshift-dns/dns-default-l2ks7" Sep 10 13:32:37.308943 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:32:37.308872 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/abeadb08-451a-477c-bff5-0cb019997ba4-data-volume\") pod \"insights-runtime-extractor-5ftzf\" (UID: \"abeadb08-451a-477c-bff5-0cb019997ba4\") " pod="openshift-insights/insights-runtime-extractor-5ftzf" Sep 10 13:32:37.309095 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:32:37.308984 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/abeadb08-451a-477c-bff5-0cb019997ba4-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-5ftzf\" (UID: \"abeadb08-451a-477c-bff5-0cb019997ba4\") " pod="openshift-insights/insights-runtime-extractor-5ftzf" Sep 10 13:32:37.312351 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:32:37.312330 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/abeadb08-451a-477c-bff5-0cb019997ba4-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-5ftzf\" (UID: \"abeadb08-451a-477c-bff5-0cb019997ba4\") " pod="openshift-insights/insights-runtime-extractor-5ftzf" Sep 10 13:32:37.339008 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:32:37.338974 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-clqlh\" (UniqueName: \"kubernetes.io/projected/abeadb08-451a-477c-bff5-0cb019997ba4-kube-api-access-clqlh\") pod \"insights-runtime-extractor-5ftzf\" (UID: \"abeadb08-451a-477c-bff5-0cb019997ba4\") " pod="openshift-insights/insights-runtime-extractor-5ftzf" Sep 10 13:32:37.410166 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:32:37.410077 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4wvk8\" (UniqueName: \"kubernetes.io/projected/b308bee9-e8b3-4e7b-98b1-c2f6d833b7a3-kube-api-access-4wvk8\") pod \"ingress-canary-vpwqf\" (UID: \"b308bee9-e8b3-4e7b-98b1-c2f6d833b7a3\") " pod="openshift-ingress-canary/ingress-canary-vpwqf" Sep 10 13:32:37.410166 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:32:37.410121 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/a572632c-303d-4828-a58a-e1c230b6e301-metrics-tls\") pod \"dns-default-l2ks7\" (UID: \"a572632c-303d-4828-a58a-e1c230b6e301\") " pod="openshift-dns/dns-default-l2ks7" Sep 10 13:32:37.410166 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:32:37.410143 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/b308bee9-e8b3-4e7b-98b1-c2f6d833b7a3-cert\") pod \"ingress-canary-vpwqf\" (UID: \"b308bee9-e8b3-4e7b-98b1-c2f6d833b7a3\") " pod="openshift-ingress-canary/ingress-canary-vpwqf" Sep 10 13:32:37.410493 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:32:37.410211 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8hn9q\" (UniqueName: \"kubernetes.io/projected/a572632c-303d-4828-a58a-e1c230b6e301-kube-api-access-8hn9q\") pod \"dns-default-l2ks7\" (UID: \"a572632c-303d-4828-a58a-e1c230b6e301\") " pod="openshift-dns/dns-default-l2ks7" Sep 10 13:32:37.410493 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:32:37.410268 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/a572632c-303d-4828-a58a-e1c230b6e301-config-volume\") pod \"dns-default-l2ks7\" (UID: \"a572632c-303d-4828-a58a-e1c230b6e301\") " pod="openshift-dns/dns-default-l2ks7" Sep 10 13:32:37.410493 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:32:37.410293 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a572632c-303d-4828-a58a-e1c230b6e301-tmp-dir\") pod \"dns-default-l2ks7\" (UID: \"a572632c-303d-4828-a58a-e1c230b6e301\") " pod="openshift-dns/dns-default-l2ks7" Sep 10 13:32:37.410667 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:32:37.410634 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a572632c-303d-4828-a58a-e1c230b6e301-tmp-dir\") pod \"dns-default-l2ks7\" (UID: \"a572632c-303d-4828-a58a-e1c230b6e301\") " pod="openshift-dns/dns-default-l2ks7" Sep 10 13:32:37.410882 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:32:37.410863 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/a572632c-303d-4828-a58a-e1c230b6e301-config-volume\") pod \"dns-default-l2ks7\" (UID: \"a572632c-303d-4828-a58a-e1c230b6e301\") " pod="openshift-dns/dns-default-l2ks7" Sep 10 13:32:37.412434 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:32:37.412416 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/a572632c-303d-4828-a58a-e1c230b6e301-metrics-tls\") pod \"dns-default-l2ks7\" (UID: \"a572632c-303d-4828-a58a-e1c230b6e301\") " pod="openshift-dns/dns-default-l2ks7" Sep 10 13:32:37.412638 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:32:37.412620 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/b308bee9-e8b3-4e7b-98b1-c2f6d833b7a3-cert\") pod \"ingress-canary-vpwqf\" (UID: \"b308bee9-e8b3-4e7b-98b1-c2f6d833b7a3\") " pod="openshift-ingress-canary/ingress-canary-vpwqf" Sep 10 13:32:37.425102 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:32:37.425075 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4wvk8\" (UniqueName: \"kubernetes.io/projected/b308bee9-e8b3-4e7b-98b1-c2f6d833b7a3-kube-api-access-4wvk8\") pod \"ingress-canary-vpwqf\" (UID: \"b308bee9-e8b3-4e7b-98b1-c2f6d833b7a3\") " pod="openshift-ingress-canary/ingress-canary-vpwqf" Sep 10 13:32:37.425314 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:32:37.425298 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8hn9q\" (UniqueName: \"kubernetes.io/projected/a572632c-303d-4828-a58a-e1c230b6e301-kube-api-access-8hn9q\") pod \"dns-default-l2ks7\" (UID: \"a572632c-303d-4828-a58a-e1c230b6e301\") " pod="openshift-dns/dns-default-l2ks7" Sep 10 13:32:37.455444 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:32:37.455416 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-5ftzf" Sep 10 13:32:37.529670 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:32:37.529640 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-vpwqf" Sep 10 13:32:37.540694 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:32:37.540672 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-l2ks7" Sep 10 13:32:37.737615 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:32:37.737589 2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-5ftzf"] Sep 10 13:32:37.740128 ip-10-0-140-8 kubenswrapper[2585]: W0910 13:32:37.740103 2585 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podabeadb08_451a_477c_bff5_0cb019997ba4.slice/crio-3086a8d38389842d8cabd2b54e6989412d257c65b4e26da4ed31e9db51e10f96 WatchSource:0}: Error finding container 3086a8d38389842d8cabd2b54e6989412d257c65b4e26da4ed31e9db51e10f96: Status 404 returned error can't find the container with id 3086a8d38389842d8cabd2b54e6989412d257c65b4e26da4ed31e9db51e10f96 Sep 10 13:32:37.801715 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:32:37.801653 2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-vpwqf"] Sep 10 13:32:37.803871 ip-10-0-140-8 kubenswrapper[2585]: W0910 13:32:37.803845 2585 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb308bee9_e8b3_4e7b_98b1_c2f6d833b7a3.slice/crio-50a9d04f06efe219f7e09ec125aaa8a98f582bc7a85fd2a0b83eb5bd95b1f3ee WatchSource:0}: Error finding container 50a9d04f06efe219f7e09ec125aaa8a98f582bc7a85fd2a0b83eb5bd95b1f3ee: Status 404 returned error can't find the container with id 50a9d04f06efe219f7e09ec125aaa8a98f582bc7a85fd2a0b83eb5bd95b1f3ee Sep 10 13:32:37.834469 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:32:37.834438 2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-l2ks7"] Sep 10 13:32:37.837565 ip-10-0-140-8 kubenswrapper[2585]: W0910 13:32:37.837542 2585 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda572632c_303d_4828_a58a_e1c230b6e301.slice/crio-189119285b075affb4fa0d4a5ce60b911228912f7256b5a67911d6756e5ed2d3 WatchSource:0}: Error finding container 189119285b075affb4fa0d4a5ce60b911228912f7256b5a67911d6756e5ed2d3: Status 404 returned error can't find the container with id 189119285b075affb4fa0d4a5ce60b911228912f7256b5a67911d6756e5ed2d3 Sep 10 13:32:38.340207 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:32:38.340166 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-l2ks7" event={"ID":"a572632c-303d-4828-a58a-e1c230b6e301","Type":"ContainerStarted","Data":"189119285b075affb4fa0d4a5ce60b911228912f7256b5a67911d6756e5ed2d3"} Sep 10 13:32:38.341455 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:32:38.341427 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-vpwqf" event={"ID":"b308bee9-e8b3-4e7b-98b1-c2f6d833b7a3","Type":"ContainerStarted","Data":"50a9d04f06efe219f7e09ec125aaa8a98f582bc7a85fd2a0b83eb5bd95b1f3ee"} Sep 10 13:32:38.343043 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:32:38.343017 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-5ftzf" event={"ID":"abeadb08-451a-477c-bff5-0cb019997ba4","Type":"ContainerStarted","Data":"ed442e749fea8b6efe50cdaf4745a0c74d432d2602ebe4041395070881507ba3"} Sep 10 13:32:38.343132 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:32:38.343076 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-5ftzf" event={"ID":"abeadb08-451a-477c-bff5-0cb019997ba4","Type":"ContainerStarted","Data":"3086a8d38389842d8cabd2b54e6989412d257c65b4e26da4ed31e9db51e10f96"} Sep 10 13:32:38.795475 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:32:38.795398 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fxgmp_fa53c783-d40a-4024-9f7b-1d0d2b57f6dc/konnectivity-agent/0.log" limit=1024 Sep 10 13:32:38.917118 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:32:38.917083 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fxgmp_fa53c783-d40a-4024-9f7b-1d0d2b57f6dc/konnectivity-agent/0.log" limit=1024 Sep 10 13:32:40.350652 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:32:40.350609 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-l2ks7" event={"ID":"a572632c-303d-4828-a58a-e1c230b6e301","Type":"ContainerStarted","Data":"29d1a238a73be53c7cc6f74accd6e71fdcbf8ec1a1627ce78144c52e4e83a2e9"} Sep 10 13:32:41.024920 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:32:41.024886 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fxgmp_fa53c783-d40a-4024-9f7b-1d0d2b57f6dc/konnectivity-agent/0.log" limit=1024 Sep 10 13:32:41.075141 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:32:41.075111 2585 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-operator-6f4946db56-8j7lz"] Sep 10 13:32:41.078298 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:32:41.078275 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-6f4946db56-8j7lz" Sep 10 13:32:41.082189 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:32:41.082147 2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\"" Sep 10 13:32:41.082189 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:32:41.082166 2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\"" Sep 10 13:32:41.082189 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:32:41.082181 2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\"" Sep 10 13:32:41.082448 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:32:41.082221 2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-tls\"" Sep 10 13:32:41.082448 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:32:41.082180 2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-dockercfg-pq962\"" Sep 10 13:32:41.082574 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:32:41.082504 2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-kube-rbac-proxy-config\"" Sep 10 13:32:41.093172 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:32:41.093146 2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-6f4946db56-8j7lz"] Sep 10 13:32:41.138091 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:32:41.138052 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-operator-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/c46642aa-e014-4a5d-a2a7-47bdf9bd0fe3-prometheus-operator-kube-rbac-proxy-config\") pod \"prometheus-operator-6f4946db56-8j7lz\" (UID: \"c46642aa-e014-4a5d-a2a7-47bdf9bd0fe3\") " pod="openshift-monitoring/prometheus-operator-6f4946db56-8j7lz" Sep 10 13:32:41.138311 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:32:41.138124 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/c46642aa-e014-4a5d-a2a7-47bdf9bd0fe3-metrics-client-ca\") pod \"prometheus-operator-6f4946db56-8j7lz\" (UID: \"c46642aa-e014-4a5d-a2a7-47bdf9bd0fe3\") " pod="openshift-monitoring/prometheus-operator-6f4946db56-8j7lz" Sep 10 13:32:41.138311 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:32:41.138158 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/c46642aa-e014-4a5d-a2a7-47bdf9bd0fe3-prometheus-operator-tls\") pod \"prometheus-operator-6f4946db56-8j7lz\" (UID: \"c46642aa-e014-4a5d-a2a7-47bdf9bd0fe3\") " pod="openshift-monitoring/prometheus-operator-6f4946db56-8j7lz" Sep 10 13:32:41.138311 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:32:41.138210 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2j4jk\" (UniqueName: \"kubernetes.io/projected/c46642aa-e014-4a5d-a2a7-47bdf9bd0fe3-kube-api-access-2j4jk\") pod \"prometheus-operator-6f4946db56-8j7lz\" (UID: \"c46642aa-e014-4a5d-a2a7-47bdf9bd0fe3\") " pod="openshift-monitoring/prometheus-operator-6f4946db56-8j7lz" Sep 10 13:32:41.154159 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:32:41.154124 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fxgmp_fa53c783-d40a-4024-9f7b-1d0d2b57f6dc/konnectivity-agent/0.log" limit=1024 Sep 10 13:32:41.239372 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:32:41.239334 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/c46642aa-e014-4a5d-a2a7-47bdf9bd0fe3-metrics-client-ca\") pod \"prometheus-operator-6f4946db56-8j7lz\" (UID: \"c46642aa-e014-4a5d-a2a7-47bdf9bd0fe3\") " pod="openshift-monitoring/prometheus-operator-6f4946db56-8j7lz" Sep 10 13:32:41.239372 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:32:41.239374 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/c46642aa-e014-4a5d-a2a7-47bdf9bd0fe3-prometheus-operator-tls\") pod \"prometheus-operator-6f4946db56-8j7lz\" (UID: \"c46642aa-e014-4a5d-a2a7-47bdf9bd0fe3\") " pod="openshift-monitoring/prometheus-operator-6f4946db56-8j7lz" Sep 10 13:32:41.239618 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:32:41.239403 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2j4jk\" (UniqueName: \"kubernetes.io/projected/c46642aa-e014-4a5d-a2a7-47bdf9bd0fe3-kube-api-access-2j4jk\") pod \"prometheus-operator-6f4946db56-8j7lz\" (UID: \"c46642aa-e014-4a5d-a2a7-47bdf9bd0fe3\") " pod="openshift-monitoring/prometheus-operator-6f4946db56-8j7lz" Sep 10 13:32:41.239618 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:32:41.239467 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-operator-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/c46642aa-e014-4a5d-a2a7-47bdf9bd0fe3-prometheus-operator-kube-rbac-proxy-config\") pod \"prometheus-operator-6f4946db56-8j7lz\" (UID: \"c46642aa-e014-4a5d-a2a7-47bdf9bd0fe3\") " pod="openshift-monitoring/prometheus-operator-6f4946db56-8j7lz" Sep 10 13:32:41.239618 ip-10-0-140-8 kubenswrapper[2585]: E0910 13:32:41.239518 2585 secret.go:189] Couldn't get secret openshift-monitoring/prometheus-operator-tls: secret "prometheus-operator-tls" not found Sep 10 13:32:41.239618 ip-10-0-140-8 kubenswrapper[2585]: E0910 13:32:41.239595 2585 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c46642aa-e014-4a5d-a2a7-47bdf9bd0fe3-prometheus-operator-tls podName:c46642aa-e014-4a5d-a2a7-47bdf9bd0fe3 nodeName:}" failed. No retries permitted until 2026-09-10 13:32:41.739573501 +0000 UTC m=+246.612277680 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "prometheus-operator-tls" (UniqueName: "kubernetes.io/secret/c46642aa-e014-4a5d-a2a7-47bdf9bd0fe3-prometheus-operator-tls") pod "prometheus-operator-6f4946db56-8j7lz" (UID: "c46642aa-e014-4a5d-a2a7-47bdf9bd0fe3") : secret "prometheus-operator-tls" not found Sep 10 13:32:41.240175 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:32:41.240150 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/c46642aa-e014-4a5d-a2a7-47bdf9bd0fe3-metrics-client-ca\") pod \"prometheus-operator-6f4946db56-8j7lz\" (UID: \"c46642aa-e014-4a5d-a2a7-47bdf9bd0fe3\") " pod="openshift-monitoring/prometheus-operator-6f4946db56-8j7lz" Sep 10 13:32:41.242044 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:32:41.242020 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-operator-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/c46642aa-e014-4a5d-a2a7-47bdf9bd0fe3-prometheus-operator-kube-rbac-proxy-config\") pod \"prometheus-operator-6f4946db56-8j7lz\" (UID: \"c46642aa-e014-4a5d-a2a7-47bdf9bd0fe3\") " pod="openshift-monitoring/prometheus-operator-6f4946db56-8j7lz" Sep 10 13:32:41.249159 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:32:41.249136 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2j4jk\" (UniqueName: \"kubernetes.io/projected/c46642aa-e014-4a5d-a2a7-47bdf9bd0fe3-kube-api-access-2j4jk\") pod \"prometheus-operator-6f4946db56-8j7lz\" (UID: \"c46642aa-e014-4a5d-a2a7-47bdf9bd0fe3\") " pod="openshift-monitoring/prometheus-operator-6f4946db56-8j7lz" Sep 10 13:32:41.355528 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:32:41.355490 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-l2ks7" event={"ID":"a572632c-303d-4828-a58a-e1c230b6e301","Type":"ContainerStarted","Data":"8ba3828a49133b4ce9526f359f2e41e39492be6115c4e1ba8188bdea9fa0f930"} Sep 10 13:32:41.355979 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:32:41.355595 2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-dns/dns-default-l2ks7" Sep 10 13:32:41.357307 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:32:41.357276 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-vpwqf" event={"ID":"b308bee9-e8b3-4e7b-98b1-c2f6d833b7a3","Type":"ContainerStarted","Data":"b99d64c9bdddd94fe4169408181b8233b274cc71a9e32a270677db0181bac94d"} Sep 10 13:32:41.359097 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:32:41.359073 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-5ftzf" event={"ID":"abeadb08-451a-477c-bff5-0cb019997ba4","Type":"ContainerStarted","Data":"90378e6089ba8c2b53ff4f809b73b2e2f33e5ee3d4ae0892ba7648e89e1018eb"} Sep 10 13:32:41.377325 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:32:41.377280 2585 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/dns-default-l2ks7" podStartSLOduration=2.703249024 podStartE2EDuration="4.377266916s" totalImagesPullingTime="1.674017892s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 13:32:37 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 13:32:41.376166406 +0000 UTC m=+246.248870607" watchObservedRunningTime="2026-09-10 13:32:41.377266916 +0000 UTC m=+246.249971111" Sep 10 13:32:41.588690 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:32:41.588652 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fxgmp_fa53c783-d40a-4024-9f7b-1d0d2b57f6dc/konnectivity-agent/0.log" limit=1024 Sep 10 13:32:41.744282 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:32:41.744177 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/c46642aa-e014-4a5d-a2a7-47bdf9bd0fe3-prometheus-operator-tls\") pod \"prometheus-operator-6f4946db56-8j7lz\" (UID: \"c46642aa-e014-4a5d-a2a7-47bdf9bd0fe3\") " pod="openshift-monitoring/prometheus-operator-6f4946db56-8j7lz" Sep 10 13:32:41.746774 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:32:41.746748 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/c46642aa-e014-4a5d-a2a7-47bdf9bd0fe3-prometheus-operator-tls\") pod \"prometheus-operator-6f4946db56-8j7lz\" (UID: \"c46642aa-e014-4a5d-a2a7-47bdf9bd0fe3\") " pod="openshift-monitoring/prometheus-operator-6f4946db56-8j7lz" Sep 10 13:32:41.989058 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:32:41.989030 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-6f4946db56-8j7lz" Sep 10 13:32:42.261026 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:32:42.260935 2585 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress-canary/ingress-canary-vpwqf" podStartSLOduration=2.684894426 podStartE2EDuration="5.260916672s" totalImagesPullingTime="2.576022246s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 13:32:37 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 13:32:41.396474256 +0000 UTC m=+246.269178486" watchObservedRunningTime="2026-09-10 13:32:42.260916672 +0000 UTC m=+247.133620873" Sep 10 13:32:42.261443 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:32:42.261385 2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-6f4946db56-8j7lz"] Sep 10 13:32:42.263771 ip-10-0-140-8 kubenswrapper[2585]: W0910 13:32:42.263750 2585 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc46642aa_e014_4a5d_a2a7_47bdf9bd0fe3.slice/crio-befb6982a73e06aefe1cb72ec22089cb14272a1b88a82bd0258c7ac4d4dd6986 WatchSource:0}: Error finding container befb6982a73e06aefe1cb72ec22089cb14272a1b88a82bd0258c7ac4d4dd6986: Status 404 returned error can't find the container with id befb6982a73e06aefe1cb72ec22089cb14272a1b88a82bd0258c7ac4d4dd6986 Sep 10 13:32:42.364072 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:32:42.364038 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-6f4946db56-8j7lz" event={"ID":"c46642aa-e014-4a5d-a2a7-47bdf9bd0fe3","Type":"ContainerStarted","Data":"befb6982a73e06aefe1cb72ec22089cb14272a1b88a82bd0258c7ac4d4dd6986"} Sep 10 13:32:42.366026 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:32:42.365998 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-5ftzf" event={"ID":"abeadb08-451a-477c-bff5-0cb019997ba4","Type":"ContainerStarted","Data":"88db8b213480fe889bad33161642190f9a761f4074ca46d0cb223912c687da2f"} Sep 10 13:32:42.391024 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:32:42.390979 2585 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-insights/insights-runtime-extractor-5ftzf" podStartSLOduration=1.396414741 podStartE2EDuration="5.390963212s" totalImagesPullingTime="3.994548471s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-10 13:32:37 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 13:32:42.390640945 +0000 UTC m=+247.263345147" watchObservedRunningTime="2026-09-10 13:32:42.390963212 +0000 UTC m=+247.263667412" Sep 10 13:32:43.881323 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:32:43.881231 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fxgmp_fa53c783-d40a-4024-9f7b-1d0d2b57f6dc/konnectivity-agent/0.log" limit=1024 Sep 10 13:32:44.373664 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:32:44.373626 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-6f4946db56-8j7lz" event={"ID":"c46642aa-e014-4a5d-a2a7-47bdf9bd0fe3","Type":"ContainerStarted","Data":"fd5eb4dc32726574f4b393a1c1a7f8b08f1663ae3995b2d39d4a944a8f343aa8"} Sep 10 13:32:44.373833 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:32:44.373682 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-6f4946db56-8j7lz" event={"ID":"c46642aa-e014-4a5d-a2a7-47bdf9bd0fe3","Type":"ContainerStarted","Data":"e1c7cc4b839b8dc9b65e5616f2d19ecb7c3dec70f650b2740bbe3da955aa51bc"} Sep 10 13:32:44.394087 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:32:44.394042 2585 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/prometheus-operator-6f4946db56-8j7lz" podStartSLOduration=2.06631284 podStartE2EDuration="3.394027256s" totalImagesPullingTime="1.327714416s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 13:32:41 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 13:32:44.393178456 +0000 UTC m=+249.265882657" watchObservedRunningTime="2026-09-10 13:32:44.394027256 +0000 UTC m=+249.266731456" Sep 10 13:32:46.251889 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:32:46.251858 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fxgmp_fa53c783-d40a-4024-9f7b-1d0d2b57f6dc/konnectivity-agent/0.log" limit=1024 Sep 10 13:32:46.422663 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:32:46.422632 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fxgmp_fa53c783-d40a-4024-9f7b-1d0d2b57f6dc/konnectivity-agent/0.log" limit=1024 Sep 10 13:32:46.624861 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:32:46.624829 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fxgmp_fa53c783-d40a-4024-9f7b-1d0d2b57f6dc/konnectivity-agent/0.log" limit=1024 Sep 10 13:32:46.639110 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:32:46.639081 2585 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/openshift-state-metrics-657fd6c54-mthpx"] Sep 10 13:32:46.642154 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:32:46.642139 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/openshift-state-metrics-657fd6c54-mthpx" Sep 10 13:32:46.644717 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:32:46.644696 2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-tls\"" Sep 10 13:32:46.644815 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:32:46.644701 2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-kube-rbac-proxy-config\"" Sep 10 13:32:46.644946 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:32:46.644932 2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-dockercfg-wl57l\"" Sep 10 13:32:46.660789 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:32:46.660761 2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/openshift-state-metrics-657fd6c54-mthpx"] Sep 10 13:32:46.680212 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:32:46.680186 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/1e07789a-9850-45b5-958a-c098e9744d8e-openshift-state-metrics-tls\") pod \"openshift-state-metrics-657fd6c54-mthpx\" (UID: \"1e07789a-9850-45b5-958a-c098e9744d8e\") " pod="openshift-monitoring/openshift-state-metrics-657fd6c54-mthpx" Sep 10 13:32:46.680344 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:32:46.680216 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/1e07789a-9850-45b5-958a-c098e9744d8e-metrics-client-ca\") pod \"openshift-state-metrics-657fd6c54-mthpx\" (UID: \"1e07789a-9850-45b5-958a-c098e9744d8e\") " pod="openshift-monitoring/openshift-state-metrics-657fd6c54-mthpx" Sep 10 13:32:46.680344 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:32:46.680234 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/1e07789a-9850-45b5-958a-c098e9744d8e-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-657fd6c54-mthpx\" (UID: \"1e07789a-9850-45b5-958a-c098e9744d8e\") " pod="openshift-monitoring/openshift-state-metrics-657fd6c54-mthpx" Sep 10 13:32:46.680426 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:32:46.680386 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hznrm\" (UniqueName: \"kubernetes.io/projected/1e07789a-9850-45b5-958a-c098e9744d8e-kube-api-access-hznrm\") pod \"openshift-state-metrics-657fd6c54-mthpx\" (UID: \"1e07789a-9850-45b5-958a-c098e9744d8e\") " pod="openshift-monitoring/openshift-state-metrics-657fd6c54-mthpx" Sep 10 13:32:46.697151 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:32:46.697124 2585 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/node-exporter-m4khk"] Sep 10 13:32:46.702321 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:32:46.700989 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-m4khk" Sep 10 13:32:46.707446 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:32:46.707422 2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\"" Sep 10 13:32:46.707564 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:32:46.707452 2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-s6fhr\"" Sep 10 13:32:46.707564 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:32:46.707422 2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\"" Sep 10 13:32:46.710226 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:32:46.710207 2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\"" Sep 10 13:32:46.765490 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:32:46.765459 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fxgmp_fa53c783-d40a-4024-9f7b-1d0d2b57f6dc/konnectivity-agent/0.log" limit=1024 Sep 10 13:32:46.781504 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:32:46.781475 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/1e07789a-9850-45b5-958a-c098e9744d8e-openshift-state-metrics-tls\") pod \"openshift-state-metrics-657fd6c54-mthpx\" (UID: \"1e07789a-9850-45b5-958a-c098e9744d8e\") " pod="openshift-monitoring/openshift-state-metrics-657fd6c54-mthpx" Sep 10 13:32:46.781613 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:32:46.781507 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/1e07789a-9850-45b5-958a-c098e9744d8e-metrics-client-ca\") pod \"openshift-state-metrics-657fd6c54-mthpx\" (UID: \"1e07789a-9850-45b5-958a-c098e9744d8e\") " pod="openshift-monitoring/openshift-state-metrics-657fd6c54-mthpx" Sep 10 13:32:46.781613 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:32:46.781527 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/1e07789a-9850-45b5-958a-c098e9744d8e-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-657fd6c54-mthpx\" (UID: \"1e07789a-9850-45b5-958a-c098e9744d8e\") " pod="openshift-monitoring/openshift-state-metrics-657fd6c54-mthpx" Sep 10 13:32:46.781613 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:32:46.781548 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/4e671dec-efdd-4004-9c16-c0c2692371d6-node-exporter-tls\") pod \"node-exporter-m4khk\" (UID: \"4e671dec-efdd-4004-9c16-c0c2692371d6\") " pod="openshift-monitoring/node-exporter-m4khk" Sep 10 13:32:46.781613 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:32:46.781568 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gfz8g\" (UniqueName: \"kubernetes.io/projected/4e671dec-efdd-4004-9c16-c0c2692371d6-kube-api-access-gfz8g\") pod \"node-exporter-m4khk\" (UID: \"4e671dec-efdd-4004-9c16-c0c2692371d6\") " pod="openshift-monitoring/node-exporter-m4khk" Sep 10 13:32:46.781613 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:32:46.781593 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/4e671dec-efdd-4004-9c16-c0c2692371d6-sys\") pod \"node-exporter-m4khk\" (UID: \"4e671dec-efdd-4004-9c16-c0c2692371d6\") " pod="openshift-monitoring/node-exporter-m4khk" Sep 10 13:32:46.781828 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:32:46.781629 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/4e671dec-efdd-4004-9c16-c0c2692371d6-node-exporter-wtmp\") pod \"node-exporter-m4khk\" (UID: \"4e671dec-efdd-4004-9c16-c0c2692371d6\") " pod="openshift-monitoring/node-exporter-m4khk" Sep 10 13:32:46.781828 ip-10-0-140-8 kubenswrapper[2585]: E0910 13:32:46.781637 2585 secret.go:189] Couldn't get secret openshift-monitoring/openshift-state-metrics-tls: secret "openshift-state-metrics-tls" not found Sep 10 13:32:46.781828 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:32:46.781694 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/4e671dec-efdd-4004-9c16-c0c2692371d6-node-exporter-accelerators-collector-config\") pod \"node-exporter-m4khk\" (UID: \"4e671dec-efdd-4004-9c16-c0c2692371d6\") " pod="openshift-monitoring/node-exporter-m4khk" Sep 10 13:32:46.781828 ip-10-0-140-8 kubenswrapper[2585]: E0910 13:32:46.781765 2585 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1e07789a-9850-45b5-958a-c098e9744d8e-openshift-state-metrics-tls podName:1e07789a-9850-45b5-958a-c098e9744d8e nodeName:}" failed. No retries permitted until 2026-09-10 13:32:47.281744985 +0000 UTC m=+252.154449186 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "openshift-state-metrics-tls" (UniqueName: "kubernetes.io/secret/1e07789a-9850-45b5-958a-c098e9744d8e-openshift-state-metrics-tls") pod "openshift-state-metrics-657fd6c54-mthpx" (UID: "1e07789a-9850-45b5-958a-c098e9744d8e") : secret "openshift-state-metrics-tls" not found Sep 10 13:32:46.781828 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:32:46.781817 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/4e671dec-efdd-4004-9c16-c0c2692371d6-metrics-client-ca\") pod \"node-exporter-m4khk\" (UID: \"4e671dec-efdd-4004-9c16-c0c2692371d6\") " pod="openshift-monitoring/node-exporter-m4khk" Sep 10 13:32:46.782011 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:32:46.781840 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/4e671dec-efdd-4004-9c16-c0c2692371d6-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-m4khk\" (UID: \"4e671dec-efdd-4004-9c16-c0c2692371d6\") " pod="openshift-monitoring/node-exporter-m4khk" Sep 10 13:32:46.782011 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:32:46.781928 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hznrm\" (UniqueName: \"kubernetes.io/projected/1e07789a-9850-45b5-958a-c098e9744d8e-kube-api-access-hznrm\") pod \"openshift-state-metrics-657fd6c54-mthpx\" (UID: \"1e07789a-9850-45b5-958a-c098e9744d8e\") " pod="openshift-monitoring/openshift-state-metrics-657fd6c54-mthpx" Sep 10 13:32:46.782011 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:32:46.781958 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/4e671dec-efdd-4004-9c16-c0c2692371d6-node-exporter-textfile\") pod \"node-exporter-m4khk\" (UID: \"4e671dec-efdd-4004-9c16-c0c2692371d6\") " pod="openshift-monitoring/node-exporter-m4khk" Sep 10 13:32:46.782011 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:32:46.781985 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/4e671dec-efdd-4004-9c16-c0c2692371d6-root\") pod \"node-exporter-m4khk\" (UID: \"4e671dec-efdd-4004-9c16-c0c2692371d6\") " pod="openshift-monitoring/node-exporter-m4khk" Sep 10 13:32:46.782211 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:32:46.782185 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/1e07789a-9850-45b5-958a-c098e9744d8e-metrics-client-ca\") pod \"openshift-state-metrics-657fd6c54-mthpx\" (UID: \"1e07789a-9850-45b5-958a-c098e9744d8e\") " pod="openshift-monitoring/openshift-state-metrics-657fd6c54-mthpx" Sep 10 13:32:46.784026 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:32:46.784005 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/1e07789a-9850-45b5-958a-c098e9744d8e-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-657fd6c54-mthpx\" (UID: \"1e07789a-9850-45b5-958a-c098e9744d8e\") " pod="openshift-monitoring/openshift-state-metrics-657fd6c54-mthpx" Sep 10 13:32:46.796061 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:32:46.796034 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-hznrm\" (UniqueName: \"kubernetes.io/projected/1e07789a-9850-45b5-958a-c098e9744d8e-kube-api-access-hznrm\") pod \"openshift-state-metrics-657fd6c54-mthpx\" (UID: \"1e07789a-9850-45b5-958a-c098e9744d8e\") " pod="openshift-monitoring/openshift-state-metrics-657fd6c54-mthpx" Sep 10 13:32:46.882644 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:32:46.882563 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/4e671dec-efdd-4004-9c16-c0c2692371d6-node-exporter-tls\") pod \"node-exporter-m4khk\" (UID: \"4e671dec-efdd-4004-9c16-c0c2692371d6\") " pod="openshift-monitoring/node-exporter-m4khk" Sep 10 13:32:46.882644 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:32:46.882596 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gfz8g\" (UniqueName: \"kubernetes.io/projected/4e671dec-efdd-4004-9c16-c0c2692371d6-kube-api-access-gfz8g\") pod \"node-exporter-m4khk\" (UID: \"4e671dec-efdd-4004-9c16-c0c2692371d6\") " pod="openshift-monitoring/node-exporter-m4khk" Sep 10 13:32:46.882644 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:32:46.882616 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/4e671dec-efdd-4004-9c16-c0c2692371d6-sys\") pod \"node-exporter-m4khk\" (UID: \"4e671dec-efdd-4004-9c16-c0c2692371d6\") " pod="openshift-monitoring/node-exporter-m4khk" Sep 10 13:32:46.882644 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:32:46.882631 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/4e671dec-efdd-4004-9c16-c0c2692371d6-node-exporter-wtmp\") pod \"node-exporter-m4khk\" (UID: \"4e671dec-efdd-4004-9c16-c0c2692371d6\") " pod="openshift-monitoring/node-exporter-m4khk" Sep 10 13:32:46.882950 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:32:46.882649 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/4e671dec-efdd-4004-9c16-c0c2692371d6-node-exporter-accelerators-collector-config\") pod \"node-exporter-m4khk\" (UID: \"4e671dec-efdd-4004-9c16-c0c2692371d6\") " pod="openshift-monitoring/node-exporter-m4khk" Sep 10 13:32:46.882950 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:32:46.882683 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/4e671dec-efdd-4004-9c16-c0c2692371d6-metrics-client-ca\") pod \"node-exporter-m4khk\" (UID: \"4e671dec-efdd-4004-9c16-c0c2692371d6\") " pod="openshift-monitoring/node-exporter-m4khk" Sep 10 13:32:46.882950 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:32:46.882703 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/4e671dec-efdd-4004-9c16-c0c2692371d6-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-m4khk\" (UID: \"4e671dec-efdd-4004-9c16-c0c2692371d6\") " pod="openshift-monitoring/node-exporter-m4khk" Sep 10 13:32:46.882950 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:32:46.882718 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/4e671dec-efdd-4004-9c16-c0c2692371d6-sys\") pod \"node-exporter-m4khk\" (UID: \"4e671dec-efdd-4004-9c16-c0c2692371d6\") " pod="openshift-monitoring/node-exporter-m4khk" Sep 10 13:32:46.882950 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:32:46.882794 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/4e671dec-efdd-4004-9c16-c0c2692371d6-node-exporter-wtmp\") pod \"node-exporter-m4khk\" (UID: \"4e671dec-efdd-4004-9c16-c0c2692371d6\") " pod="openshift-monitoring/node-exporter-m4khk" Sep 10 13:32:46.882950 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:32:46.882855 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/4e671dec-efdd-4004-9c16-c0c2692371d6-node-exporter-textfile\") pod \"node-exporter-m4khk\" (UID: \"4e671dec-efdd-4004-9c16-c0c2692371d6\") " pod="openshift-monitoring/node-exporter-m4khk" Sep 10 13:32:46.882950 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:32:46.882896 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/4e671dec-efdd-4004-9c16-c0c2692371d6-root\") pod \"node-exporter-m4khk\" (UID: \"4e671dec-efdd-4004-9c16-c0c2692371d6\") " pod="openshift-monitoring/node-exporter-m4khk" Sep 10 13:32:46.883287 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:32:46.882982 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"root\" (UniqueName: \"kubernetes.io/host-path/4e671dec-efdd-4004-9c16-c0c2692371d6-root\") pod \"node-exporter-m4khk\" (UID: \"4e671dec-efdd-4004-9c16-c0c2692371d6\") " pod="openshift-monitoring/node-exporter-m4khk" Sep 10 13:32:46.883287 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:32:46.883167 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/4e671dec-efdd-4004-9c16-c0c2692371d6-node-exporter-textfile\") pod \"node-exporter-m4khk\" (UID: \"4e671dec-efdd-4004-9c16-c0c2692371d6\") " pod="openshift-monitoring/node-exporter-m4khk" Sep 10 13:32:46.883400 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:32:46.883380 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/4e671dec-efdd-4004-9c16-c0c2692371d6-metrics-client-ca\") pod \"node-exporter-m4khk\" (UID: \"4e671dec-efdd-4004-9c16-c0c2692371d6\") " pod="openshift-monitoring/node-exporter-m4khk" Sep 10 13:32:46.883457 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:32:46.883380 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/4e671dec-efdd-4004-9c16-c0c2692371d6-node-exporter-accelerators-collector-config\") pod \"node-exporter-m4khk\" (UID: \"4e671dec-efdd-4004-9c16-c0c2692371d6\") " pod="openshift-monitoring/node-exporter-m4khk" Sep 10 13:32:46.884932 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:32:46.884911 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/4e671dec-efdd-4004-9c16-c0c2692371d6-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-m4khk\" (UID: \"4e671dec-efdd-4004-9c16-c0c2692371d6\") " pod="openshift-monitoring/node-exporter-m4khk" Sep 10 13:32:46.885030 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:32:46.884934 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/4e671dec-efdd-4004-9c16-c0c2692371d6-node-exporter-tls\") pod \"node-exporter-m4khk\" (UID: \"4e671dec-efdd-4004-9c16-c0c2692371d6\") " pod="openshift-monitoring/node-exporter-m4khk" Sep 10 13:32:46.894611 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:32:46.894587 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-gfz8g\" (UniqueName: \"kubernetes.io/projected/4e671dec-efdd-4004-9c16-c0c2692371d6-kube-api-access-gfz8g\") pod \"node-exporter-m4khk\" (UID: \"4e671dec-efdd-4004-9c16-c0c2692371d6\") " pod="openshift-monitoring/node-exporter-m4khk" Sep 10 13:32:46.908486 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:32:46.908462 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fxgmp_fa53c783-d40a-4024-9f7b-1d0d2b57f6dc/konnectivity-agent/0.log" limit=1024 Sep 10 13:32:47.010705 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:32:47.010670 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-m4khk" Sep 10 13:32:47.021387 ip-10-0-140-8 kubenswrapper[2585]: W0910 13:32:47.021358 2585 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4e671dec_efdd_4004_9c16_c0c2692371d6.slice/crio-007b1c651643be587253fa0538fc6f23b7d89cf5fba99fd306235f3227a99783 WatchSource:0}: Error finding container 007b1c651643be587253fa0538fc6f23b7d89cf5fba99fd306235f3227a99783: Status 404 returned error can't find the container with id 007b1c651643be587253fa0538fc6f23b7d89cf5fba99fd306235f3227a99783 Sep 10 13:32:47.094786 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:32:47.094750 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fxgmp_fa53c783-d40a-4024-9f7b-1d0d2b57f6dc/konnectivity-agent/0.log" limit=1024 Sep 10 13:32:47.285983 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:32:47.285897 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/1e07789a-9850-45b5-958a-c098e9744d8e-openshift-state-metrics-tls\") pod \"openshift-state-metrics-657fd6c54-mthpx\" (UID: \"1e07789a-9850-45b5-958a-c098e9744d8e\") " pod="openshift-monitoring/openshift-state-metrics-657fd6c54-mthpx" Sep 10 13:32:47.288218 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:32:47.288199 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/1e07789a-9850-45b5-958a-c098e9744d8e-openshift-state-metrics-tls\") pod \"openshift-state-metrics-657fd6c54-mthpx\" (UID: \"1e07789a-9850-45b5-958a-c098e9744d8e\") " pod="openshift-monitoring/openshift-state-metrics-657fd6c54-mthpx" Sep 10 13:32:47.316102 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:32:47.316070 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fxgmp_fa53c783-d40a-4024-9f7b-1d0d2b57f6dc/konnectivity-agent/0.log" limit=1024 Sep 10 13:32:47.382505 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:32:47.382473 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-m4khk" event={"ID":"4e671dec-efdd-4004-9c16-c0c2692371d6","Type":"ContainerStarted","Data":"007b1c651643be587253fa0538fc6f23b7d89cf5fba99fd306235f3227a99783"} Sep 10 13:32:47.551003 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:32:47.550930 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/openshift-state-metrics-657fd6c54-mthpx" Sep 10 13:32:47.558308 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:32:47.558279 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fxgmp_fa53c783-d40a-4024-9f7b-1d0d2b57f6dc/konnectivity-agent/0.log" limit=1024 Sep 10 13:32:47.712358 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:32:47.712312 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fxgmp_fa53c783-d40a-4024-9f7b-1d0d2b57f6dc/konnectivity-agent/0.log" limit=1024 Sep 10 13:32:47.866196 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:32:47.866161 2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/openshift-state-metrics-657fd6c54-mthpx"] Sep 10 13:32:47.899907 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:32:47.899583 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fxgmp_fa53c783-d40a-4024-9f7b-1d0d2b57f6dc/konnectivity-agent/0.log" limit=1024 Sep 10 13:32:47.977735 ip-10-0-140-8 kubenswrapper[2585]: W0910 13:32:47.977695 2585 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1e07789a_9850_45b5_958a_c098e9744d8e.slice/crio-f28c977ed632f03444f02940a72acb5d67011608dd4631b81fe90ec6bc5d4c31 WatchSource:0}: Error finding container f28c977ed632f03444f02940a72acb5d67011608dd4631b81fe90ec6bc5d4c31: Status 404 returned error can't find the container with id f28c977ed632f03444f02940a72acb5d67011608dd4631b81fe90ec6bc5d4c31 Sep 10 13:32:48.154091 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:32:48.153997 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fxgmp_fa53c783-d40a-4024-9f7b-1d0d2b57f6dc/konnectivity-agent/0.log" limit=1024 Sep 10 13:32:48.387556 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:32:48.387523 2585 generic.go:356] "Generic (PLEG): container finished" podID="4e671dec-efdd-4004-9c16-c0c2692371d6" containerID="f3a32e4ae061cb0d8bc8a11160c7aaf24a119260479bf7e468900c3d7a376fa8" exitCode=0 Sep 10 13:32:48.387966 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:32:48.387607 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-m4khk" event={"ID":"4e671dec-efdd-4004-9c16-c0c2692371d6","Type":"ContainerDied","Data":"f3a32e4ae061cb0d8bc8a11160c7aaf24a119260479bf7e468900c3d7a376fa8"} Sep 10 13:32:48.389601 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:32:48.389577 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-657fd6c54-mthpx" event={"ID":"1e07789a-9850-45b5-958a-c098e9744d8e","Type":"ContainerStarted","Data":"50d6c40d58be7c593b64ec1e4e43ec6f096dfc702d4741f5b043188cfcaf3ecb"} Sep 10 13:32:48.389694 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:32:48.389630 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-657fd6c54-mthpx" event={"ID":"1e07789a-9850-45b5-958a-c098e9744d8e","Type":"ContainerStarted","Data":"d08cb363aa8e3a887d60774c441c11737d0c15d4ada98fec67e2361ad1a5bf27"} Sep 10 13:32:48.389694 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:32:48.389662 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-657fd6c54-mthpx" event={"ID":"1e07789a-9850-45b5-958a-c098e9744d8e","Type":"ContainerStarted","Data":"f28c977ed632f03444f02940a72acb5d67011608dd4631b81fe90ec6bc5d4c31"} Sep 10 13:32:48.654056 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:32:48.654018 2585 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/thanos-querier-6b46bcc57-874m7"] Sep 10 13:32:48.657540 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:32:48.657523 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/thanos-querier-6b46bcc57-874m7" Sep 10 13:32:48.660301 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:32:48.660281 2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy\"" Sep 10 13:32:48.660525 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:32:48.660507 2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-rules\"" Sep 10 13:32:48.660805 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:32:48.660758 2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-metrics\"" Sep 10 13:32:48.660993 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:32:48.660974 2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-dockercfg-58bps\"" Sep 10 13:32:48.661101 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:32:48.660998 2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-tls\"" Sep 10 13:32:48.661204 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:32:48.661191 2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-grpc-tls-12qj3n67qp6q0\"" Sep 10 13:32:48.661375 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:32:48.661364 2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-web\"" Sep 10 13:32:48.673625 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:32:48.673573 2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/thanos-querier-6b46bcc57-874m7"] Sep 10 13:32:48.695797 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:32:48.695768 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/796c05e9-33a7-476b-bc85-c189e991629e-metrics-client-ca\") pod \"thanos-querier-6b46bcc57-874m7\" (UID: \"796c05e9-33a7-476b-bc85-c189e991629e\") " pod="openshift-monitoring/thanos-querier-6b46bcc57-874m7" Sep 10 13:32:48.695938 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:32:48.695800 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/796c05e9-33a7-476b-bc85-c189e991629e-secret-grpc-tls\") pod \"thanos-querier-6b46bcc57-874m7\" (UID: \"796c05e9-33a7-476b-bc85-c189e991629e\") " pod="openshift-monitoring/thanos-querier-6b46bcc57-874m7" Sep 10 13:32:48.695938 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:32:48.695818 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rm879\" (UniqueName: \"kubernetes.io/projected/796c05e9-33a7-476b-bc85-c189e991629e-kube-api-access-rm879\") pod \"thanos-querier-6b46bcc57-874m7\" (UID: \"796c05e9-33a7-476b-bc85-c189e991629e\") " pod="openshift-monitoring/thanos-querier-6b46bcc57-874m7" Sep 10 13:32:48.695938 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:32:48.695860 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-tls\" (UniqueName: \"kubernetes.io/secret/796c05e9-33a7-476b-bc85-c189e991629e-secret-thanos-querier-tls\") pod \"thanos-querier-6b46bcc57-874m7\" (UID: \"796c05e9-33a7-476b-bc85-c189e991629e\") " pod="openshift-monitoring/thanos-querier-6b46bcc57-874m7" Sep 10 13:32:48.695938 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:32:48.695921 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/796c05e9-33a7-476b-bc85-c189e991629e-secret-thanos-querier-kube-rbac-proxy\") pod \"thanos-querier-6b46bcc57-874m7\" (UID: \"796c05e9-33a7-476b-bc85-c189e991629e\") " pod="openshift-monitoring/thanos-querier-6b46bcc57-874m7" Sep 10 13:32:48.696093 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:32:48.695968 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/796c05e9-33a7-476b-bc85-c189e991629e-secret-thanos-querier-kube-rbac-proxy-web\") pod \"thanos-querier-6b46bcc57-874m7\" (UID: \"796c05e9-33a7-476b-bc85-c189e991629e\") " pod="openshift-monitoring/thanos-querier-6b46bcc57-874m7" Sep 10 13:32:48.696093 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:32:48.695993 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-rules\" (UniqueName: \"kubernetes.io/secret/796c05e9-33a7-476b-bc85-c189e991629e-secret-thanos-querier-kube-rbac-proxy-rules\") pod \"thanos-querier-6b46bcc57-874m7\" (UID: \"796c05e9-33a7-476b-bc85-c189e991629e\") " pod="openshift-monitoring/thanos-querier-6b46bcc57-874m7" Sep 10 13:32:48.696093 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:32:48.696022 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-metrics\" (UniqueName: \"kubernetes.io/secret/796c05e9-33a7-476b-bc85-c189e991629e-secret-thanos-querier-kube-rbac-proxy-metrics\") pod \"thanos-querier-6b46bcc57-874m7\" (UID: \"796c05e9-33a7-476b-bc85-c189e991629e\") " pod="openshift-monitoring/thanos-querier-6b46bcc57-874m7" Sep 10 13:32:48.796853 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:32:48.796825 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-tls\" (UniqueName: \"kubernetes.io/secret/796c05e9-33a7-476b-bc85-c189e991629e-secret-thanos-querier-tls\") pod \"thanos-querier-6b46bcc57-874m7\" (UID: \"796c05e9-33a7-476b-bc85-c189e991629e\") " pod="openshift-monitoring/thanos-querier-6b46bcc57-874m7" Sep 10 13:32:48.797000 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:32:48.796859 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/796c05e9-33a7-476b-bc85-c189e991629e-secret-thanos-querier-kube-rbac-proxy\") pod \"thanos-querier-6b46bcc57-874m7\" (UID: \"796c05e9-33a7-476b-bc85-c189e991629e\") " pod="openshift-monitoring/thanos-querier-6b46bcc57-874m7" Sep 10 13:32:48.797000 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:32:48.796880 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/796c05e9-33a7-476b-bc85-c189e991629e-secret-thanos-querier-kube-rbac-proxy-web\") pod \"thanos-querier-6b46bcc57-874m7\" (UID: \"796c05e9-33a7-476b-bc85-c189e991629e\") " pod="openshift-monitoring/thanos-querier-6b46bcc57-874m7" Sep 10 13:32:48.797000 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:32:48.796897 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-rules\" (UniqueName: \"kubernetes.io/secret/796c05e9-33a7-476b-bc85-c189e991629e-secret-thanos-querier-kube-rbac-proxy-rules\") pod \"thanos-querier-6b46bcc57-874m7\" (UID: \"796c05e9-33a7-476b-bc85-c189e991629e\") " pod="openshift-monitoring/thanos-querier-6b46bcc57-874m7" Sep 10 13:32:48.797000 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:32:48.796921 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-metrics\" (UniqueName: \"kubernetes.io/secret/796c05e9-33a7-476b-bc85-c189e991629e-secret-thanos-querier-kube-rbac-proxy-metrics\") pod \"thanos-querier-6b46bcc57-874m7\" (UID: \"796c05e9-33a7-476b-bc85-c189e991629e\") " pod="openshift-monitoring/thanos-querier-6b46bcc57-874m7" Sep 10 13:32:48.797192 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:32:48.797010 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/796c05e9-33a7-476b-bc85-c189e991629e-metrics-client-ca\") pod \"thanos-querier-6b46bcc57-874m7\" (UID: \"796c05e9-33a7-476b-bc85-c189e991629e\") " pod="openshift-monitoring/thanos-querier-6b46bcc57-874m7" Sep 10 13:32:48.797192 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:32:48.797043 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/796c05e9-33a7-476b-bc85-c189e991629e-secret-grpc-tls\") pod \"thanos-querier-6b46bcc57-874m7\" (UID: \"796c05e9-33a7-476b-bc85-c189e991629e\") " pod="openshift-monitoring/thanos-querier-6b46bcc57-874m7" Sep 10 13:32:48.797192 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:32:48.797064 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rm879\" (UniqueName: \"kubernetes.io/projected/796c05e9-33a7-476b-bc85-c189e991629e-kube-api-access-rm879\") pod \"thanos-querier-6b46bcc57-874m7\" (UID: \"796c05e9-33a7-476b-bc85-c189e991629e\") " pod="openshift-monitoring/thanos-querier-6b46bcc57-874m7" Sep 10 13:32:48.797829 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:32:48.797801 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/796c05e9-33a7-476b-bc85-c189e991629e-metrics-client-ca\") pod \"thanos-querier-6b46bcc57-874m7\" (UID: \"796c05e9-33a7-476b-bc85-c189e991629e\") " pod="openshift-monitoring/thanos-querier-6b46bcc57-874m7" Sep 10 13:32:48.799437 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:32:48.799413 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-rules\" (UniqueName: \"kubernetes.io/secret/796c05e9-33a7-476b-bc85-c189e991629e-secret-thanos-querier-kube-rbac-proxy-rules\") pod \"thanos-querier-6b46bcc57-874m7\" (UID: \"796c05e9-33a7-476b-bc85-c189e991629e\") " pod="openshift-monitoring/thanos-querier-6b46bcc57-874m7" Sep 10 13:32:48.799678 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:32:48.799656 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-tls\" (UniqueName: \"kubernetes.io/secret/796c05e9-33a7-476b-bc85-c189e991629e-secret-thanos-querier-tls\") pod \"thanos-querier-6b46bcc57-874m7\" (UID: \"796c05e9-33a7-476b-bc85-c189e991629e\") " pod="openshift-monitoring/thanos-querier-6b46bcc57-874m7" Sep 10 13:32:48.800143 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:32:48.800113 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-metrics\" (UniqueName: \"kubernetes.io/secret/796c05e9-33a7-476b-bc85-c189e991629e-secret-thanos-querier-kube-rbac-proxy-metrics\") pod \"thanos-querier-6b46bcc57-874m7\" (UID: \"796c05e9-33a7-476b-bc85-c189e991629e\") " pod="openshift-monitoring/thanos-querier-6b46bcc57-874m7" Sep 10 13:32:48.800224 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:32:48.800162 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/796c05e9-33a7-476b-bc85-c189e991629e-secret-thanos-querier-kube-rbac-proxy-web\") pod \"thanos-querier-6b46bcc57-874m7\" (UID: \"796c05e9-33a7-476b-bc85-c189e991629e\") " pod="openshift-monitoring/thanos-querier-6b46bcc57-874m7" Sep 10 13:32:48.800224 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:32:48.800171 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/796c05e9-33a7-476b-bc85-c189e991629e-secret-thanos-querier-kube-rbac-proxy\") pod \"thanos-querier-6b46bcc57-874m7\" (UID: \"796c05e9-33a7-476b-bc85-c189e991629e\") " pod="openshift-monitoring/thanos-querier-6b46bcc57-874m7" Sep 10 13:32:48.800433 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:32:48.800416 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/796c05e9-33a7-476b-bc85-c189e991629e-secret-grpc-tls\") pod \"thanos-querier-6b46bcc57-874m7\" (UID: \"796c05e9-33a7-476b-bc85-c189e991629e\") " pod="openshift-monitoring/thanos-querier-6b46bcc57-874m7" Sep 10 13:32:48.806998 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:32:48.806973 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rm879\" (UniqueName: \"kubernetes.io/projected/796c05e9-33a7-476b-bc85-c189e991629e-kube-api-access-rm879\") pod \"thanos-querier-6b46bcc57-874m7\" (UID: \"796c05e9-33a7-476b-bc85-c189e991629e\") " pod="openshift-monitoring/thanos-querier-6b46bcc57-874m7" Sep 10 13:32:48.966607 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:32:48.966511 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/thanos-querier-6b46bcc57-874m7" Sep 10 13:32:49.258597 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:32:49.258512 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fxgmp_fa53c783-d40a-4024-9f7b-1d0d2b57f6dc/konnectivity-agent/0.log" limit=1024 Sep 10 13:32:49.284368 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:32:49.284314 2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/thanos-querier-6b46bcc57-874m7"] Sep 10 13:32:49.286287 ip-10-0-140-8 kubenswrapper[2585]: W0910 13:32:49.286263 2585 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod796c05e9_33a7_476b_bc85_c189e991629e.slice/crio-fc928b749b237d7031ec49a5cf2a21f0cab8ce544062dba8b75ab1d659130e35 WatchSource:0}: Error finding container fc928b749b237d7031ec49a5cf2a21f0cab8ce544062dba8b75ab1d659130e35: Status 404 returned error can't find the container with id fc928b749b237d7031ec49a5cf2a21f0cab8ce544062dba8b75ab1d659130e35 Sep 10 13:32:49.391855 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:32:49.391822 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fxgmp_fa53c783-d40a-4024-9f7b-1d0d2b57f6dc/konnectivity-agent/0.log" limit=1024 Sep 10 13:32:49.397409 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:32:49.397373 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-m4khk" event={"ID":"4e671dec-efdd-4004-9c16-c0c2692371d6","Type":"ContainerStarted","Data":"8847fc8348eee7d97107414fff35185eeb09849c90144a945a48fd060f136eab"} Sep 10 13:32:49.397586 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:32:49.397439 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-m4khk" event={"ID":"4e671dec-efdd-4004-9c16-c0c2692371d6","Type":"ContainerStarted","Data":"3b18793bc4071d39100de3d3631443e067608c9c6abd8c9644c3462511600537"} Sep 10 13:32:49.398560 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:32:49.398538 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-6b46bcc57-874m7" event={"ID":"796c05e9-33a7-476b-bc85-c189e991629e","Type":"ContainerStarted","Data":"fc928b749b237d7031ec49a5cf2a21f0cab8ce544062dba8b75ab1d659130e35"} Sep 10 13:32:49.431391 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:32:49.431337 2585 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/node-exporter-m4khk" podStartSLOduration=2.3658300199999998 podStartE2EDuration="3.431321788s" totalImagesPullingTime="1.001416005s" totalInitContainerRuntime="64.075763ms" isStatefulPod=true podCreationTimestamp="2026-09-10 13:32:46 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 13:32:49.430287306 +0000 UTC m=+254.302991508" watchObservedRunningTime="2026-09-10 13:32:49.431321788 +0000 UTC m=+254.304025999" Sep 10 13:32:49.843436 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:32:49.843403 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fxgmp_fa53c783-d40a-4024-9f7b-1d0d2b57f6dc/konnectivity-agent/0.log" limit=1024 Sep 10 13:32:49.957075 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:32:49.957044 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fxgmp_fa53c783-d40a-4024-9f7b-1d0d2b57f6dc/konnectivity-agent/0.log" limit=1024 Sep 10 13:32:50.064434 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:32:50.064401 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fxgmp_fa53c783-d40a-4024-9f7b-1d0d2b57f6dc/konnectivity-agent/0.log" limit=1024 Sep 10 13:32:50.198407 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:32:50.198318 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fxgmp_fa53c783-d40a-4024-9f7b-1d0d2b57f6dc/konnectivity-agent/0.log" limit=1024 Sep 10 13:32:50.405091 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:32:50.405045 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-657fd6c54-mthpx" event={"ID":"1e07789a-9850-45b5-958a-c098e9744d8e","Type":"ContainerStarted","Data":"04020f700b357ed2bd92215f31ff22600d2a43c4a66d80a04de1da4a608f7e7e"} Sep 10 13:32:50.433590 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:32:50.433524 2585 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/openshift-state-metrics-657fd6c54-mthpx" podStartSLOduration=2.245324077 podStartE2EDuration="4.433483175s" totalImagesPullingTime="2.188159098s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 13:32:46 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 13:32:50.432622853 +0000 UTC m=+255.305327058" watchObservedRunningTime="2026-09-10 13:32:50.433483175 +0000 UTC m=+255.306187376" Sep 10 13:32:51.238529 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:32:51.238502 2585 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/metrics-server-6ffbdddb-wjr2c"] Sep 10 13:32:51.241499 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:32:51.241482 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/metrics-server-6ffbdddb-wjr2c" Sep 10 13:32:51.244555 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:32:51.244535 2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-client-certs\"" Sep 10 13:32:51.245731 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:32:51.245709 2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-server-audit-profiles\"" Sep 10 13:32:51.245820 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:32:51.245744 2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-tls\"" Sep 10 13:32:51.245871 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:32:51.245829 2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-dockercfg-mpdgq\"" Sep 10 13:32:51.245979 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:32:51.245964 2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-7md1gch2b2pul\"" Sep 10 13:32:51.246031 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:32:51.246022 2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kubelet-serving-ca-bundle\"" Sep 10 13:32:51.257921 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:32:51.257899 2585 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/monitoring-plugin-85bbbbbf4-hvgfb"] Sep 10 13:32:51.262657 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:32:51.262634 2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/metrics-server-6ffbdddb-wjr2c"] Sep 10 13:32:51.262768 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:32:51.262749 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/monitoring-plugin-85bbbbbf4-hvgfb" Sep 10 13:32:51.265484 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:32:51.265467 2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"monitoring-plugin-dockercfg-hbcpr\"" Sep 10 13:32:51.265885 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:32:51.265867 2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"monitoring-plugin-cert\"" Sep 10 13:32:51.282343 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:32:51.282309 2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/monitoring-plugin-85bbbbbf4-hvgfb"] Sep 10 13:32:51.289585 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:32:51.289553 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fxgmp_fa53c783-d40a-4024-9f7b-1d0d2b57f6dc/konnectivity-agent/0.log" limit=1024 Sep 10 13:32:51.327566 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:32:51.327542 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-server-audit-profiles\" (UniqueName: \"kubernetes.io/configmap/9f2ee68b-c2cd-4d70-b1c9-81ceb68670c2-metrics-server-audit-profiles\") pod \"metrics-server-6ffbdddb-wjr2c\" (UID: \"9f2ee68b-c2cd-4d70-b1c9-81ceb68670c2\") " pod="openshift-monitoring/metrics-server-6ffbdddb-wjr2c" Sep 10 13:32:51.327662 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:32:51.327578 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-server-tls\" (UniqueName: \"kubernetes.io/secret/9f2ee68b-c2cd-4d70-b1c9-81ceb68670c2-secret-metrics-server-tls\") pod \"metrics-server-6ffbdddb-wjr2c\" (UID: \"9f2ee68b-c2cd-4d70-b1c9-81ceb68670c2\") " pod="openshift-monitoring/metrics-server-6ffbdddb-wjr2c" Sep 10 13:32:51.327662 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:32:51.327601 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lgm8t\" (UniqueName: \"kubernetes.io/projected/71f4a37f-99f5-4cd3-9d7e-d2a011d07ee0-kube-api-access-lgm8t\") pod \"monitoring-plugin-85bbbbbf4-hvgfb\" (UID: \"71f4a37f-99f5-4cd3-9d7e-d2a011d07ee0\") " pod="openshift-monitoring/monitoring-plugin-85bbbbbf4-hvgfb" Sep 10 13:32:51.327662 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:32:51.327637 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-x42qz\" (UniqueName: \"kubernetes.io/projected/9f2ee68b-c2cd-4d70-b1c9-81ceb68670c2-kube-api-access-x42qz\") pod \"metrics-server-6ffbdddb-wjr2c\" (UID: \"9f2ee68b-c2cd-4d70-b1c9-81ceb68670c2\") " pod="openshift-monitoring/metrics-server-6ffbdddb-wjr2c" Sep 10 13:32:51.327783 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:32:51.327685 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"client-ca-bundle\" (UniqueName: \"kubernetes.io/secret/9f2ee68b-c2cd-4d70-b1c9-81ceb68670c2-client-ca-bundle\") pod \"metrics-server-6ffbdddb-wjr2c\" (UID: \"9f2ee68b-c2cd-4d70-b1c9-81ceb68670c2\") " pod="openshift-monitoring/metrics-server-6ffbdddb-wjr2c" Sep 10 13:32:51.327783 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:32:51.327708 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-server-client-certs\" (UniqueName: \"kubernetes.io/secret/9f2ee68b-c2cd-4d70-b1c9-81ceb68670c2-secret-metrics-server-client-certs\") pod \"metrics-server-6ffbdddb-wjr2c\" (UID: \"9f2ee68b-c2cd-4d70-b1c9-81ceb68670c2\") " pod="openshift-monitoring/metrics-server-6ffbdddb-wjr2c" Sep 10 13:32:51.327783 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:32:51.327765 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/9f2ee68b-c2cd-4d70-b1c9-81ceb68670c2-configmap-kubelet-serving-ca-bundle\") pod \"metrics-server-6ffbdddb-wjr2c\" (UID: \"9f2ee68b-c2cd-4d70-b1c9-81ceb68670c2\") " pod="openshift-monitoring/metrics-server-6ffbdddb-wjr2c" Sep 10 13:32:51.327876 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:32:51.327805 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/71f4a37f-99f5-4cd3-9d7e-d2a011d07ee0-monitoring-plugin-cert\") pod \"monitoring-plugin-85bbbbbf4-hvgfb\" (UID: \"71f4a37f-99f5-4cd3-9d7e-d2a011d07ee0\") " pod="openshift-monitoring/monitoring-plugin-85bbbbbf4-hvgfb" Sep 10 13:32:51.327876 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:32:51.327867 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"audit-log\" (UniqueName: \"kubernetes.io/empty-dir/9f2ee68b-c2cd-4d70-b1c9-81ceb68670c2-audit-log\") pod \"metrics-server-6ffbdddb-wjr2c\" (UID: \"9f2ee68b-c2cd-4d70-b1c9-81ceb68670c2\") " pod="openshift-monitoring/metrics-server-6ffbdddb-wjr2c" Sep 10 13:32:51.368205 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:32:51.368178 2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-dns/dns-default-l2ks7" Sep 10 13:32:51.411418 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:32:51.411253 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-6b46bcc57-874m7" event={"ID":"796c05e9-33a7-476b-bc85-c189e991629e","Type":"ContainerStarted","Data":"04f0836bc37e2f69851d330ae9e1861be0b2970802f1b674aba4d22eafd08267"} Sep 10 13:32:51.411418 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:32:51.411347 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-6b46bcc57-874m7" event={"ID":"796c05e9-33a7-476b-bc85-c189e991629e","Type":"ContainerStarted","Data":"77d162588280671683143fd4e24cbda0b8b6a7bc955c40922ced6b7852e53e5c"} Sep 10 13:32:51.411885 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:32:51.411422 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-6b46bcc57-874m7" event={"ID":"796c05e9-33a7-476b-bc85-c189e991629e","Type":"ContainerStarted","Data":"c93a45177aeb7c14a5713a9ba3105fac6862cc6ffec9b663668374664a9b9b39"} Sep 10 13:32:51.428989 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:32:51.428961 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"audit-log\" (UniqueName: \"kubernetes.io/empty-dir/9f2ee68b-c2cd-4d70-b1c9-81ceb68670c2-audit-log\") pod \"metrics-server-6ffbdddb-wjr2c\" (UID: \"9f2ee68b-c2cd-4d70-b1c9-81ceb68670c2\") " pod="openshift-monitoring/metrics-server-6ffbdddb-wjr2c" Sep 10 13:32:51.429131 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:32:51.429006 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-server-audit-profiles\" (UniqueName: \"kubernetes.io/configmap/9f2ee68b-c2cd-4d70-b1c9-81ceb68670c2-metrics-server-audit-profiles\") pod \"metrics-server-6ffbdddb-wjr2c\" (UID: \"9f2ee68b-c2cd-4d70-b1c9-81ceb68670c2\") " pod="openshift-monitoring/metrics-server-6ffbdddb-wjr2c" Sep 10 13:32:51.429131 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:32:51.429035 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-server-tls\" (UniqueName: \"kubernetes.io/secret/9f2ee68b-c2cd-4d70-b1c9-81ceb68670c2-secret-metrics-server-tls\") pod \"metrics-server-6ffbdddb-wjr2c\" (UID: \"9f2ee68b-c2cd-4d70-b1c9-81ceb68670c2\") " pod="openshift-monitoring/metrics-server-6ffbdddb-wjr2c" Sep 10 13:32:51.429131 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:32:51.429061 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lgm8t\" (UniqueName: \"kubernetes.io/projected/71f4a37f-99f5-4cd3-9d7e-d2a011d07ee0-kube-api-access-lgm8t\") pod \"monitoring-plugin-85bbbbbf4-hvgfb\" (UID: \"71f4a37f-99f5-4cd3-9d7e-d2a011d07ee0\") " pod="openshift-monitoring/monitoring-plugin-85bbbbbf4-hvgfb" Sep 10 13:32:51.429316 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:32:51.429263 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x42qz\" (UniqueName: \"kubernetes.io/projected/9f2ee68b-c2cd-4d70-b1c9-81ceb68670c2-kube-api-access-x42qz\") pod \"metrics-server-6ffbdddb-wjr2c\" (UID: \"9f2ee68b-c2cd-4d70-b1c9-81ceb68670c2\") " pod="openshift-monitoring/metrics-server-6ffbdddb-wjr2c" Sep 10 13:32:51.429435 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:32:51.429340 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"client-ca-bundle\" (UniqueName: \"kubernetes.io/secret/9f2ee68b-c2cd-4d70-b1c9-81ceb68670c2-client-ca-bundle\") pod \"metrics-server-6ffbdddb-wjr2c\" (UID: \"9f2ee68b-c2cd-4d70-b1c9-81ceb68670c2\") " pod="openshift-monitoring/metrics-server-6ffbdddb-wjr2c" Sep 10 13:32:51.429435 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:32:51.429385 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-server-client-certs\" (UniqueName: \"kubernetes.io/secret/9f2ee68b-c2cd-4d70-b1c9-81ceb68670c2-secret-metrics-server-client-certs\") pod \"metrics-server-6ffbdddb-wjr2c\" (UID: \"9f2ee68b-c2cd-4d70-b1c9-81ceb68670c2\") " pod="openshift-monitoring/metrics-server-6ffbdddb-wjr2c" Sep 10 13:32:51.429435 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:32:51.429426 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/9f2ee68b-c2cd-4d70-b1c9-81ceb68670c2-configmap-kubelet-serving-ca-bundle\") pod \"metrics-server-6ffbdddb-wjr2c\" (UID: \"9f2ee68b-c2cd-4d70-b1c9-81ceb68670c2\") " pod="openshift-monitoring/metrics-server-6ffbdddb-wjr2c" Sep 10 13:32:51.429598 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:32:51.429453 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"audit-log\" (UniqueName: \"kubernetes.io/empty-dir/9f2ee68b-c2cd-4d70-b1c9-81ceb68670c2-audit-log\") pod \"metrics-server-6ffbdddb-wjr2c\" (UID: \"9f2ee68b-c2cd-4d70-b1c9-81ceb68670c2\") " pod="openshift-monitoring/metrics-server-6ffbdddb-wjr2c" Sep 10 13:32:51.429598 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:32:51.429456 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/71f4a37f-99f5-4cd3-9d7e-d2a011d07ee0-monitoring-plugin-cert\") pod \"monitoring-plugin-85bbbbbf4-hvgfb\" (UID: \"71f4a37f-99f5-4cd3-9d7e-d2a011d07ee0\") " pod="openshift-monitoring/monitoring-plugin-85bbbbbf4-hvgfb" Sep 10 13:32:51.429598 ip-10-0-140-8 kubenswrapper[2585]: E0910 13:32:51.429525 2585 secret.go:189] Couldn't get secret openshift-monitoring/monitoring-plugin-cert: secret "monitoring-plugin-cert" not found Sep 10 13:32:51.429598 ip-10-0-140-8 kubenswrapper[2585]: E0910 13:32:51.429578 2585 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/71f4a37f-99f5-4cd3-9d7e-d2a011d07ee0-monitoring-plugin-cert podName:71f4a37f-99f5-4cd3-9d7e-d2a011d07ee0 nodeName:}" failed. No retries permitted until 2026-09-10 13:32:51.929561841 +0000 UTC m=+256.802266035 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "monitoring-plugin-cert" (UniqueName: "kubernetes.io/secret/71f4a37f-99f5-4cd3-9d7e-d2a011d07ee0-monitoring-plugin-cert") pod "monitoring-plugin-85bbbbbf4-hvgfb" (UID: "71f4a37f-99f5-4cd3-9d7e-d2a011d07ee0") : secret "monitoring-plugin-cert" not found Sep 10 13:32:51.430106 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:32:51.430076 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-server-audit-profiles\" (UniqueName: \"kubernetes.io/configmap/9f2ee68b-c2cd-4d70-b1c9-81ceb68670c2-metrics-server-audit-profiles\") pod \"metrics-server-6ffbdddb-wjr2c\" (UID: \"9f2ee68b-c2cd-4d70-b1c9-81ceb68670c2\") " pod="openshift-monitoring/metrics-server-6ffbdddb-wjr2c" Sep 10 13:32:51.430274 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:32:51.430197 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/9f2ee68b-c2cd-4d70-b1c9-81ceb68670c2-configmap-kubelet-serving-ca-bundle\") pod \"metrics-server-6ffbdddb-wjr2c\" (UID: \"9f2ee68b-c2cd-4d70-b1c9-81ceb68670c2\") " pod="openshift-monitoring/metrics-server-6ffbdddb-wjr2c" Sep 10 13:32:51.432010 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:32:51.431988 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-metrics-server-tls\" (UniqueName: \"kubernetes.io/secret/9f2ee68b-c2cd-4d70-b1c9-81ceb68670c2-secret-metrics-server-tls\") pod \"metrics-server-6ffbdddb-wjr2c\" (UID: \"9f2ee68b-c2cd-4d70-b1c9-81ceb68670c2\") " pod="openshift-monitoring/metrics-server-6ffbdddb-wjr2c" Sep 10 13:32:51.432110 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:32:51.432034 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"client-ca-bundle\" (UniqueName: \"kubernetes.io/secret/9f2ee68b-c2cd-4d70-b1c9-81ceb68670c2-client-ca-bundle\") pod \"metrics-server-6ffbdddb-wjr2c\" (UID: \"9f2ee68b-c2cd-4d70-b1c9-81ceb68670c2\") " pod="openshift-monitoring/metrics-server-6ffbdddb-wjr2c" Sep 10 13:32:51.432150 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:32:51.432130 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-metrics-server-client-certs\" (UniqueName: \"kubernetes.io/secret/9f2ee68b-c2cd-4d70-b1c9-81ceb68670c2-secret-metrics-server-client-certs\") pod \"metrics-server-6ffbdddb-wjr2c\" (UID: \"9f2ee68b-c2cd-4d70-b1c9-81ceb68670c2\") " pod="openshift-monitoring/metrics-server-6ffbdddb-wjr2c" Sep 10 13:32:51.448996 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:32:51.448971 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-x42qz\" (UniqueName: \"kubernetes.io/projected/9f2ee68b-c2cd-4d70-b1c9-81ceb68670c2-kube-api-access-x42qz\") pod \"metrics-server-6ffbdddb-wjr2c\" (UID: \"9f2ee68b-c2cd-4d70-b1c9-81ceb68670c2\") " pod="openshift-monitoring/metrics-server-6ffbdddb-wjr2c" Sep 10 13:32:51.449117 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:32:51.449100 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-lgm8t\" (UniqueName: \"kubernetes.io/projected/71f4a37f-99f5-4cd3-9d7e-d2a011d07ee0-kube-api-access-lgm8t\") pod \"monitoring-plugin-85bbbbbf4-hvgfb\" (UID: \"71f4a37f-99f5-4cd3-9d7e-d2a011d07ee0\") " pod="openshift-monitoring/monitoring-plugin-85bbbbbf4-hvgfb" Sep 10 13:32:51.552660 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:32:51.552630 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/metrics-server-6ffbdddb-wjr2c" Sep 10 13:32:51.829500 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:32:51.829466 2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/metrics-server-6ffbdddb-wjr2c"] Sep 10 13:32:51.832113 ip-10-0-140-8 kubenswrapper[2585]: W0910 13:32:51.832091 2585 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9f2ee68b_c2cd_4d70_b1c9_81ceb68670c2.slice/crio-a436a50c1a26fd0725e4438e548dd4ad0d3532d4d3c720587c53ad5c91279bdf WatchSource:0}: Error finding container a436a50c1a26fd0725e4438e548dd4ad0d3532d4d3c720587c53ad5c91279bdf: Status 404 returned error can't find the container with id a436a50c1a26fd0725e4438e548dd4ad0d3532d4d3c720587c53ad5c91279bdf Sep 10 13:32:51.934048 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:32:51.934010 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/71f4a37f-99f5-4cd3-9d7e-d2a011d07ee0-monitoring-plugin-cert\") pod \"monitoring-plugin-85bbbbbf4-hvgfb\" (UID: \"71f4a37f-99f5-4cd3-9d7e-d2a011d07ee0\") " pod="openshift-monitoring/monitoring-plugin-85bbbbbf4-hvgfb" Sep 10 13:32:51.934210 ip-10-0-140-8 kubenswrapper[2585]: E0910 13:32:51.934172 2585 secret.go:189] Couldn't get secret openshift-monitoring/monitoring-plugin-cert: secret "monitoring-plugin-cert" not found Sep 10 13:32:51.934296 ip-10-0-140-8 kubenswrapper[2585]: E0910 13:32:51.934262 2585 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/71f4a37f-99f5-4cd3-9d7e-d2a011d07ee0-monitoring-plugin-cert podName:71f4a37f-99f5-4cd3-9d7e-d2a011d07ee0 nodeName:}" failed. No retries permitted until 2026-09-10 13:32:52.934222921 +0000 UTC m=+257.806927109 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "monitoring-plugin-cert" (UniqueName: "kubernetes.io/secret/71f4a37f-99f5-4cd3-9d7e-d2a011d07ee0-monitoring-plugin-cert") pod "monitoring-plugin-85bbbbbf4-hvgfb" (UID: "71f4a37f-99f5-4cd3-9d7e-d2a011d07ee0") : secret "monitoring-plugin-cert" not found Sep 10 13:32:52.161222 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:32:52.161190 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fxgmp_fa53c783-d40a-4024-9f7b-1d0d2b57f6dc/konnectivity-agent/0.log" limit=1024 Sep 10 13:32:52.416407 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:32:52.416312 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/metrics-server-6ffbdddb-wjr2c" event={"ID":"9f2ee68b-c2cd-4d70-b1c9-81ceb68670c2","Type":"ContainerStarted","Data":"a436a50c1a26fd0725e4438e548dd4ad0d3532d4d3c720587c53ad5c91279bdf"} Sep 10 13:32:52.419082 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:32:52.419058 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-6b46bcc57-874m7" event={"ID":"796c05e9-33a7-476b-bc85-c189e991629e","Type":"ContainerStarted","Data":"42f99f149efbf0d6ed7a3471aedbda7117da688f193e6b77a56474522f5978eb"} Sep 10 13:32:52.419184 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:32:52.419111 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-6b46bcc57-874m7" event={"ID":"796c05e9-33a7-476b-bc85-c189e991629e","Type":"ContainerStarted","Data":"06ea988a8123950bb4b8d949657fdb24f8a4f985b6c3c538017a1926d7294f3c"} Sep 10 13:32:52.419184 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:32:52.419139 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-6b46bcc57-874m7" event={"ID":"796c05e9-33a7-476b-bc85-c189e991629e","Type":"ContainerStarted","Data":"86963f1521fa84c32d2e7b62479a490a4016b501efaa3aca826a6a2a038aff40"} Sep 10 13:32:52.419295 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:32:52.419216 2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/thanos-querier-6b46bcc57-874m7" Sep 10 13:32:52.450753 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:32:52.450707 2585 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/thanos-querier-6b46bcc57-874m7" podStartSLOduration=1.932777849 podStartE2EDuration="4.450690498s" totalImagesPullingTime="2.517912649s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 13:32:48 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 13:32:52.44880206 +0000 UTC m=+257.321506262" watchObservedRunningTime="2026-09-10 13:32:52.450690498 +0000 UTC m=+257.323394699" Sep 10 13:32:52.945114 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:32:52.945083 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/71f4a37f-99f5-4cd3-9d7e-d2a011d07ee0-monitoring-plugin-cert\") pod \"monitoring-plugin-85bbbbbf4-hvgfb\" (UID: \"71f4a37f-99f5-4cd3-9d7e-d2a011d07ee0\") " pod="openshift-monitoring/monitoring-plugin-85bbbbbf4-hvgfb" Sep 10 13:32:52.948092 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:32:52.948063 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/71f4a37f-99f5-4cd3-9d7e-d2a011d07ee0-monitoring-plugin-cert\") pod \"monitoring-plugin-85bbbbbf4-hvgfb\" (UID: \"71f4a37f-99f5-4cd3-9d7e-d2a011d07ee0\") " pod="openshift-monitoring/monitoring-plugin-85bbbbbf4-hvgfb" Sep 10 13:32:53.110310 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:32:53.110271 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/monitoring-plugin-85bbbbbf4-hvgfb" Sep 10 13:32:53.682216 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:32:53.682143 2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/monitoring-plugin-85bbbbbf4-hvgfb"] Sep 10 13:32:53.685112 ip-10-0-140-8 kubenswrapper[2585]: W0910 13:32:53.685076 2585 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod71f4a37f_99f5_4cd3_9d7e_d2a011d07ee0.slice/crio-8bdc09e1c276b8e87f332391b98b07cbf26a0df28a095ef75ab648dbb200ab90 WatchSource:0}: Error finding container 8bdc09e1c276b8e87f332391b98b07cbf26a0df28a095ef75ab648dbb200ab90: Status 404 returned error can't find the container with id 8bdc09e1c276b8e87f332391b98b07cbf26a0df28a095ef75ab648dbb200ab90 Sep 10 13:32:54.235817 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:32:54.235782 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fxgmp_fa53c783-d40a-4024-9f7b-1d0d2b57f6dc/konnectivity-agent/0.log" limit=1024 Sep 10 13:32:54.427095 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:32:54.427057 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/monitoring-plugin-85bbbbbf4-hvgfb" event={"ID":"71f4a37f-99f5-4cd3-9d7e-d2a011d07ee0","Type":"ContainerStarted","Data":"8bdc09e1c276b8e87f332391b98b07cbf26a0df28a095ef75ab648dbb200ab90"} Sep 10 13:32:54.428497 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:32:54.428469 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/metrics-server-6ffbdddb-wjr2c" event={"ID":"9f2ee68b-c2cd-4d70-b1c9-81ceb68670c2","Type":"ContainerStarted","Data":"45cb8d82834ba0e7d20598d0c172d8aa9a63e8f8325bb12eccd0c547d66ef965"} Sep 10 13:32:54.462924 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:32:54.462868 2585 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/metrics-server-6ffbdddb-wjr2c" podStartSLOduration=1.8385227739999999 podStartE2EDuration="3.46284527s" totalImagesPullingTime="1.624322496s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 13:32:51 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 13:32:54.461467434 +0000 UTC m=+259.334171634" watchObservedRunningTime="2026-09-10 13:32:54.46284527 +0000 UTC m=+259.335549475" Sep 10 13:32:54.549441 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:32:54.549360 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fxgmp_fa53c783-d40a-4024-9f7b-1d0d2b57f6dc/konnectivity-agent/0.log" limit=1024 Sep 10 13:32:54.693568 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:32:54.693530 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fxgmp_fa53c783-d40a-4024-9f7b-1d0d2b57f6dc/konnectivity-agent/0.log" limit=1024 Sep 10 13:32:54.977116 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:32:54.977071 2585 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/downloads-85d98c998f-cjph8"] Sep 10 13:32:54.983469 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:32:54.983445 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/downloads-85d98c998f-cjph8" Sep 10 13:32:54.986471 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:32:54.986437 2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"openshift-service-ca.crt\"" Sep 10 13:32:54.986613 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:32:54.986516 2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"kube-root-ca.crt\"" Sep 10 13:32:55.002639 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:32:55.002614 2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"default-dockercfg-49m6g\"" Sep 10 13:32:55.029260 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:32:55.029209 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fxgmp_fa53c783-d40a-4024-9f7b-1d0d2b57f6dc/konnectivity-agent/0.log" limit=1024 Sep 10 13:32:55.051466 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:32:55.051436 2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/downloads-85d98c998f-cjph8"] Sep 10 13:32:55.063957 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:32:55.063935 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9hlrv\" (UniqueName: \"kubernetes.io/projected/f9a6f8e4-4890-463b-b583-c37100f359e5-kube-api-access-9hlrv\") pod \"downloads-85d98c998f-cjph8\" (UID: \"f9a6f8e4-4890-463b-b583-c37100f359e5\") " pod="openshift-console/downloads-85d98c998f-cjph8" Sep 10 13:32:55.165016 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:32:55.164980 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9hlrv\" (UniqueName: \"kubernetes.io/projected/f9a6f8e4-4890-463b-b583-c37100f359e5-kube-api-access-9hlrv\") pod \"downloads-85d98c998f-cjph8\" (UID: \"f9a6f8e4-4890-463b-b583-c37100f359e5\") " pod="openshift-console/downloads-85d98c998f-cjph8" Sep 10 13:32:55.177115 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:32:55.177081 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9hlrv\" (UniqueName: \"kubernetes.io/projected/f9a6f8e4-4890-463b-b583-c37100f359e5-kube-api-access-9hlrv\") pod \"downloads-85d98c998f-cjph8\" (UID: \"f9a6f8e4-4890-463b-b583-c37100f359e5\") " pod="openshift-console/downloads-85d98c998f-cjph8" Sep 10 13:32:55.202810 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:32:55.202777 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fxgmp_fa53c783-d40a-4024-9f7b-1d0d2b57f6dc/konnectivity-agent/0.log" limit=1024 Sep 10 13:32:55.293492 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:32:55.293416 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/downloads-85d98c998f-cjph8" Sep 10 13:32:55.406471 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:32:55.406444 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fxgmp_fa53c783-d40a-4024-9f7b-1d0d2b57f6dc/konnectivity-agent/0.log" limit=1024 Sep 10 13:32:55.435778 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:32:55.435745 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/monitoring-plugin-85bbbbbf4-hvgfb" event={"ID":"71f4a37f-99f5-4cd3-9d7e-d2a011d07ee0","Type":"ContainerStarted","Data":"e358396132a221df582febc83c36f805dc58489b9fb83739b488a7d228fe5246"} Sep 10 13:32:55.463224 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:32:55.463178 2585 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/monitoring-plugin-85bbbbbf4-hvgfb" podStartSLOduration=3.08863055 podStartE2EDuration="4.463162255s" totalImagesPullingTime="1.374531705s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 13:32:51 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 13:32:55.461585009 +0000 UTC m=+260.334289209" watchObservedRunningTime="2026-09-10 13:32:55.463162255 +0000 UTC m=+260.335866455" Sep 10 13:32:55.524696 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:32:55.524667 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fxgmp_fa53c783-d40a-4024-9f7b-1d0d2b57f6dc/konnectivity-agent/0.log" limit=1024 Sep 10 13:32:55.569052 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:32:55.569022 2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/downloads-85d98c998f-cjph8"] Sep 10 13:32:55.572009 ip-10-0-140-8 kubenswrapper[2585]: W0910 13:32:55.571984 2585 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf9a6f8e4_4890_463b_b583_c37100f359e5.slice/crio-57727a2ca91975e4f2cb5e53c17346724fa4382e5f1f383bb7a0f71c8ee9d6fc WatchSource:0}: Error finding container 57727a2ca91975e4f2cb5e53c17346724fa4382e5f1f383bb7a0f71c8ee9d6fc: Status 404 returned error can't find the container with id 57727a2ca91975e4f2cb5e53c17346724fa4382e5f1f383bb7a0f71c8ee9d6fc Sep 10 13:32:55.669013 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:32:55.668983 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fxgmp_fa53c783-d40a-4024-9f7b-1d0d2b57f6dc/konnectivity-agent/0.log" limit=1024 Sep 10 13:32:56.115359 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:32:56.115313 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fxgmp_fa53c783-d40a-4024-9f7b-1d0d2b57f6dc/konnectivity-agent/0.log" limit=1024 Sep 10 13:32:56.439930 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:32:56.439846 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/downloads-85d98c998f-cjph8" event={"ID":"f9a6f8e4-4890-463b-b583-c37100f359e5","Type":"ContainerStarted","Data":"57727a2ca91975e4f2cb5e53c17346724fa4382e5f1f383bb7a0f71c8ee9d6fc"} Sep 10 13:32:56.440094 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:32:56.439985 2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/monitoring-plugin-85bbbbbf4-hvgfb" Sep 10 13:32:56.444802 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:32:56.444780 2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/monitoring-plugin-85bbbbbf4-hvgfb" Sep 10 13:32:56.469014 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:32:56.468989 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fxgmp_fa53c783-d40a-4024-9f7b-1d0d2b57f6dc/konnectivity-agent/0.log" limit=1024 Sep 10 13:32:56.671592 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:32:56.671558 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fxgmp_fa53c783-d40a-4024-9f7b-1d0d2b57f6dc/konnectivity-agent/0.log" limit=1024 Sep 10 13:32:56.780510 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:32:56.780433 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fxgmp_fa53c783-d40a-4024-9f7b-1d0d2b57f6dc/konnectivity-agent/0.log" limit=1024 Sep 10 13:32:56.893173 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:32:56.893141 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fxgmp_fa53c783-d40a-4024-9f7b-1d0d2b57f6dc/konnectivity-agent/0.log" limit=1024 Sep 10 13:32:58.429666 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:32:58.429554 2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/thanos-querier-6b46bcc57-874m7" Sep 10 13:33:00.016776 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:33:00.016735 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fxgmp_fa53c783-d40a-4024-9f7b-1d0d2b57f6dc/konnectivity-agent/0.log" limit=1024 Sep 10 13:33:00.145638 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:33:00.145585 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fxgmp_fa53c783-d40a-4024-9f7b-1d0d2b57f6dc/konnectivity-agent/0.log" limit=1024 Sep 10 13:33:00.778000 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:33:00.777960 2585 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-75ff89b47-w9lfj"] Sep 10 13:33:00.781525 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:33:00.781502 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-75ff89b47-w9lfj" Sep 10 13:33:00.784392 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:33:00.784369 2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"console-config\"" Sep 10 13:33:00.784506 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:33:00.784462 2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"oauth-serving-cert\"" Sep 10 13:33:00.784506 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:33:00.784481 2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"service-ca\"" Sep 10 13:33:00.784630 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:33:00.784604 2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-oauth-config\"" Sep 10 13:33:00.785678 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:33:00.785657 2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-serving-cert\"" Sep 10 13:33:00.785783 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:33:00.785660 2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-dockercfg-4vtfq\"" Sep 10 13:33:00.804923 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:33:00.804899 2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-75ff89b47-w9lfj"] Sep 10 13:33:00.919275 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:33:00.919224 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/1faeed93-e4d2-434b-ab30-6bc7b9881a0b-console-config\") pod \"console-75ff89b47-w9lfj\" (UID: \"1faeed93-e4d2-434b-ab30-6bc7b9881a0b\") " pod="openshift-console/console-75ff89b47-w9lfj" Sep 10 13:33:00.919475 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:33:00.919283 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qfcvr\" (UniqueName: \"kubernetes.io/projected/1faeed93-e4d2-434b-ab30-6bc7b9881a0b-kube-api-access-qfcvr\") pod \"console-75ff89b47-w9lfj\" (UID: \"1faeed93-e4d2-434b-ab30-6bc7b9881a0b\") " pod="openshift-console/console-75ff89b47-w9lfj" Sep 10 13:33:00.919475 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:33:00.919313 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/1faeed93-e4d2-434b-ab30-6bc7b9881a0b-service-ca\") pod \"console-75ff89b47-w9lfj\" (UID: \"1faeed93-e4d2-434b-ab30-6bc7b9881a0b\") " pod="openshift-console/console-75ff89b47-w9lfj" Sep 10 13:33:00.919475 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:33:00.919335 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/1faeed93-e4d2-434b-ab30-6bc7b9881a0b-console-oauth-config\") pod \"console-75ff89b47-w9lfj\" (UID: \"1faeed93-e4d2-434b-ab30-6bc7b9881a0b\") " pod="openshift-console/console-75ff89b47-w9lfj" Sep 10 13:33:00.919475 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:33:00.919396 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/1faeed93-e4d2-434b-ab30-6bc7b9881a0b-console-serving-cert\") pod \"console-75ff89b47-w9lfj\" (UID: \"1faeed93-e4d2-434b-ab30-6bc7b9881a0b\") " pod="openshift-console/console-75ff89b47-w9lfj" Sep 10 13:33:00.919475 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:33:00.919434 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/1faeed93-e4d2-434b-ab30-6bc7b9881a0b-oauth-serving-cert\") pod \"console-75ff89b47-w9lfj\" (UID: \"1faeed93-e4d2-434b-ab30-6bc7b9881a0b\") " pod="openshift-console/console-75ff89b47-w9lfj" Sep 10 13:33:01.020803 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:33:01.020765 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/1faeed93-e4d2-434b-ab30-6bc7b9881a0b-console-config\") pod \"console-75ff89b47-w9lfj\" (UID: \"1faeed93-e4d2-434b-ab30-6bc7b9881a0b\") " pod="openshift-console/console-75ff89b47-w9lfj" Sep 10 13:33:01.020803 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:33:01.020809 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qfcvr\" (UniqueName: \"kubernetes.io/projected/1faeed93-e4d2-434b-ab30-6bc7b9881a0b-kube-api-access-qfcvr\") pod \"console-75ff89b47-w9lfj\" (UID: \"1faeed93-e4d2-434b-ab30-6bc7b9881a0b\") " pod="openshift-console/console-75ff89b47-w9lfj" Sep 10 13:33:01.021372 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:33:01.020947 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/1faeed93-e4d2-434b-ab30-6bc7b9881a0b-service-ca\") pod \"console-75ff89b47-w9lfj\" (UID: \"1faeed93-e4d2-434b-ab30-6bc7b9881a0b\") " pod="openshift-console/console-75ff89b47-w9lfj" Sep 10 13:33:01.021372 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:33:01.020996 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/1faeed93-e4d2-434b-ab30-6bc7b9881a0b-console-oauth-config\") pod \"console-75ff89b47-w9lfj\" (UID: \"1faeed93-e4d2-434b-ab30-6bc7b9881a0b\") " pod="openshift-console/console-75ff89b47-w9lfj" Sep 10 13:33:01.021372 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:33:01.021041 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/1faeed93-e4d2-434b-ab30-6bc7b9881a0b-console-serving-cert\") pod \"console-75ff89b47-w9lfj\" (UID: \"1faeed93-e4d2-434b-ab30-6bc7b9881a0b\") " pod="openshift-console/console-75ff89b47-w9lfj" Sep 10 13:33:01.021372 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:33:01.021096 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/1faeed93-e4d2-434b-ab30-6bc7b9881a0b-oauth-serving-cert\") pod \"console-75ff89b47-w9lfj\" (UID: \"1faeed93-e4d2-434b-ab30-6bc7b9881a0b\") " pod="openshift-console/console-75ff89b47-w9lfj" Sep 10 13:33:01.021632 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:33:01.021591 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/1faeed93-e4d2-434b-ab30-6bc7b9881a0b-console-config\") pod \"console-75ff89b47-w9lfj\" (UID: \"1faeed93-e4d2-434b-ab30-6bc7b9881a0b\") " pod="openshift-console/console-75ff89b47-w9lfj" Sep 10 13:33:01.021721 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:33:01.021697 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/1faeed93-e4d2-434b-ab30-6bc7b9881a0b-service-ca\") pod \"console-75ff89b47-w9lfj\" (UID: \"1faeed93-e4d2-434b-ab30-6bc7b9881a0b\") " pod="openshift-console/console-75ff89b47-w9lfj" Sep 10 13:33:01.021806 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:33:01.021760 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/1faeed93-e4d2-434b-ab30-6bc7b9881a0b-oauth-serving-cert\") pod \"console-75ff89b47-w9lfj\" (UID: \"1faeed93-e4d2-434b-ab30-6bc7b9881a0b\") " pod="openshift-console/console-75ff89b47-w9lfj" Sep 10 13:33:01.023706 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:33:01.023676 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/1faeed93-e4d2-434b-ab30-6bc7b9881a0b-console-oauth-config\") pod \"console-75ff89b47-w9lfj\" (UID: \"1faeed93-e4d2-434b-ab30-6bc7b9881a0b\") " pod="openshift-console/console-75ff89b47-w9lfj" Sep 10 13:33:01.024317 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:33:01.024295 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/1faeed93-e4d2-434b-ab30-6bc7b9881a0b-console-serving-cert\") pod \"console-75ff89b47-w9lfj\" (UID: \"1faeed93-e4d2-434b-ab30-6bc7b9881a0b\") " pod="openshift-console/console-75ff89b47-w9lfj" Sep 10 13:33:01.029936 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:33:01.029864 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qfcvr\" (UniqueName: \"kubernetes.io/projected/1faeed93-e4d2-434b-ab30-6bc7b9881a0b-kube-api-access-qfcvr\") pod \"console-75ff89b47-w9lfj\" (UID: \"1faeed93-e4d2-434b-ab30-6bc7b9881a0b\") " pod="openshift-console/console-75ff89b47-w9lfj" Sep 10 13:33:01.092050 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:33:01.091996 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-75ff89b47-w9lfj" Sep 10 13:33:01.373208 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:33:01.373183 2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-75ff89b47-w9lfj"] Sep 10 13:33:01.376316 ip-10-0-140-8 kubenswrapper[2585]: W0910 13:33:01.376287 2585 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1faeed93_e4d2_434b_ab30_6bc7b9881a0b.slice/crio-353caed72129be6b38164b1c7d613151b596fa7e95a4e4377d0295d6f695fc2c WatchSource:0}: Error finding container 353caed72129be6b38164b1c7d613151b596fa7e95a4e4377d0295d6f695fc2c: Status 404 returned error can't find the container with id 353caed72129be6b38164b1c7d613151b596fa7e95a4e4377d0295d6f695fc2c Sep 10 13:33:01.459690 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:33:01.459652 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-75ff89b47-w9lfj" event={"ID":"1faeed93-e4d2-434b-ab30-6bc7b9881a0b","Type":"ContainerStarted","Data":"353caed72129be6b38164b1c7d613151b596fa7e95a4e4377d0295d6f695fc2c"} Sep 10 13:33:01.713540 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:33:01.713403 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fxgmp_fa53c783-d40a-4024-9f7b-1d0d2b57f6dc/konnectivity-agent/0.log" limit=1024 Sep 10 13:33:01.921859 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:33:01.921816 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fxgmp_fa53c783-d40a-4024-9f7b-1d0d2b57f6dc/konnectivity-agent/0.log" limit=1024 Sep 10 13:33:04.148196 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:33:04.148151 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/b37f8a5b-3444-4775-8751-b60c859154d9-original-pull-secret\") pod \"global-pull-secret-syncer-lhr9l\" (UID: \"b37f8a5b-3444-4775-8751-b60c859154d9\") " pod="kube-system/global-pull-secret-syncer-lhr9l" Sep 10 13:33:04.150909 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:33:04.150796 2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\"" Sep 10 13:33:04.161280 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:33:04.161231 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/b37f8a5b-3444-4775-8751-b60c859154d9-original-pull-secret\") pod \"global-pull-secret-syncer-lhr9l\" (UID: \"b37f8a5b-3444-4775-8751-b60c859154d9\") " pod="kube-system/global-pull-secret-syncer-lhr9l" Sep 10 13:33:04.209970 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:33:04.209935 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fxgmp_fa53c783-d40a-4024-9f7b-1d0d2b57f6dc/konnectivity-agent/0.log" limit=1024 Sep 10 13:33:04.317705 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:33:04.317651 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fxgmp_fa53c783-d40a-4024-9f7b-1d0d2b57f6dc/konnectivity-agent/0.log" limit=1024 Sep 10 13:33:04.451030 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:33:04.450945 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-lhr9l" Sep 10 13:33:04.564648 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:33:04.564618 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fxgmp_fa53c783-d40a-4024-9f7b-1d0d2b57f6dc/konnectivity-agent/0.log" limit=1024 Sep 10 13:33:04.648715 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:33:04.648643 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fxgmp_fa53c783-d40a-4024-9f7b-1d0d2b57f6dc/konnectivity-agent/0.log" limit=1024 Sep 10 13:33:04.988653 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:33:04.988572 2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-lhr9l"] Sep 10 13:33:05.478960 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:33:05.478923 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-75ff89b47-w9lfj" event={"ID":"1faeed93-e4d2-434b-ab30-6bc7b9881a0b","Type":"ContainerStarted","Data":"08409cb3d6867e9f17694afd26253d4ba52db7cfc6c04752967c2aae0cf2d93b"} Sep 10 13:33:05.480233 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:33:05.480208 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-lhr9l" event={"ID":"b37f8a5b-3444-4775-8751-b60c859154d9","Type":"ContainerStarted","Data":"0766769f2195844d0889e0ff013be37a8968656e29c0c768847b2712bd339960"} Sep 10 13:33:05.505398 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:33:05.505349 2585 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-75ff89b47-w9lfj" podStartSLOduration=2.112234092 podStartE2EDuration="5.505331783s" totalImagesPullingTime="3.393097691s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 13:33:00 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 13:33:05.503672692 +0000 UTC m=+270.376376920" watchObservedRunningTime="2026-09-10 13:33:05.505331783 +0000 UTC m=+270.378035984" Sep 10 13:33:05.562939 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:33:05.562903 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d552154f-1d5b-4c5d-99bf-7aeef22ce032-metrics-certs\") pod \"network-metrics-daemon-xp5dk\" (UID: \"d552154f-1d5b-4c5d-99bf-7aeef22ce032\") " pod="openshift-multus/network-metrics-daemon-xp5dk" Sep 10 13:33:05.566333 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:33:05.566306 2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\"" Sep 10 13:33:05.575677 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:33:05.575650 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d552154f-1d5b-4c5d-99bf-7aeef22ce032-metrics-certs\") pod \"network-metrics-daemon-xp5dk\" (UID: \"d552154f-1d5b-4c5d-99bf-7aeef22ce032\") " pod="openshift-multus/network-metrics-daemon-xp5dk" Sep 10 13:33:05.659828 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:33:05.659797 2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-4vqdt\"" Sep 10 13:33:05.663692 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:33:05.663659 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x2vv5\" (UniqueName: \"kubernetes.io/projected/082a7577-a14f-49fb-a9b1-3cf250e6735a-kube-api-access-x2vv5\") pod \"network-check-target-7262t\" (UID: \"082a7577-a14f-49fb-a9b1-3cf250e6735a\") " pod="openshift-network-diagnostics/network-check-target-7262t" Sep 10 13:33:05.666478 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:33:05.666446 2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\"" Sep 10 13:33:05.667385 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:33:05.667368 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-xp5dk" Sep 10 13:33:05.677001 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:33:05.676983 2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\"" Sep 10 13:33:05.688091 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:33:05.688067 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-x2vv5\" (UniqueName: \"kubernetes.io/projected/082a7577-a14f-49fb-a9b1-3cf250e6735a-kube-api-access-x2vv5\") pod \"network-check-target-7262t\" (UID: \"082a7577-a14f-49fb-a9b1-3cf250e6735a\") " pod="openshift-network-diagnostics/network-check-target-7262t" Sep 10 13:33:05.964141 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:33:05.964115 2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-cdbpg\"" Sep 10 13:33:05.972573 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:33:05.972541 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-7262t" Sep 10 13:33:06.312497 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:33:06.312460 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fxgmp_fa53c783-d40a-4024-9f7b-1d0d2b57f6dc/konnectivity-agent/0.log" limit=1024 Sep 10 13:33:06.406274 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:33:06.406204 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fxgmp_fa53c783-d40a-4024-9f7b-1d0d2b57f6dc/konnectivity-agent/0.log" limit=1024 Sep 10 13:33:08.121092 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:33:08.121050 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fxgmp_fa53c783-d40a-4024-9f7b-1d0d2b57f6dc/konnectivity-agent/0.log" limit=1024 Sep 10 13:33:08.220639 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:33:08.220584 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fxgmp_fa53c783-d40a-4024-9f7b-1d0d2b57f6dc/konnectivity-agent/0.log" limit=1024 Sep 10 13:33:09.311460 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:33:09.311425 2585 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-778b57cc48-jwmrp"] Sep 10 13:33:09.317551 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:33:09.317525 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-778b57cc48-jwmrp" Sep 10 13:33:09.322465 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:33:09.322426 2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-778b57cc48-jwmrp"] Sep 10 13:33:09.329908 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:33:09.329880 2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"trusted-ca-bundle\"" Sep 10 13:33:09.493770 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:33:09.493738 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/d079b23f-a7c7-43c3-b1a9-c213a7edd0ea-console-oauth-config\") pod \"console-778b57cc48-jwmrp\" (UID: \"d079b23f-a7c7-43c3-b1a9-c213a7edd0ea\") " pod="openshift-console/console-778b57cc48-jwmrp" Sep 10 13:33:09.493949 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:33:09.493787 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/d079b23f-a7c7-43c3-b1a9-c213a7edd0ea-console-config\") pod \"console-778b57cc48-jwmrp\" (UID: \"d079b23f-a7c7-43c3-b1a9-c213a7edd0ea\") " pod="openshift-console/console-778b57cc48-jwmrp" Sep 10 13:33:09.493949 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:33:09.493814 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/d079b23f-a7c7-43c3-b1a9-c213a7edd0ea-oauth-serving-cert\") pod \"console-778b57cc48-jwmrp\" (UID: \"d079b23f-a7c7-43c3-b1a9-c213a7edd0ea\") " pod="openshift-console/console-778b57cc48-jwmrp" Sep 10 13:33:09.493949 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:33:09.493858 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/d079b23f-a7c7-43c3-b1a9-c213a7edd0ea-service-ca\") pod \"console-778b57cc48-jwmrp\" (UID: \"d079b23f-a7c7-43c3-b1a9-c213a7edd0ea\") " pod="openshift-console/console-778b57cc48-jwmrp" Sep 10 13:33:09.493949 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:33:09.493884 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/d079b23f-a7c7-43c3-b1a9-c213a7edd0ea-console-serving-cert\") pod \"console-778b57cc48-jwmrp\" (UID: \"d079b23f-a7c7-43c3-b1a9-c213a7edd0ea\") " pod="openshift-console/console-778b57cc48-jwmrp" Sep 10 13:33:09.494173 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:33:09.493961 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ht9xd\" (UniqueName: \"kubernetes.io/projected/d079b23f-a7c7-43c3-b1a9-c213a7edd0ea-kube-api-access-ht9xd\") pod \"console-778b57cc48-jwmrp\" (UID: \"d079b23f-a7c7-43c3-b1a9-c213a7edd0ea\") " pod="openshift-console/console-778b57cc48-jwmrp" Sep 10 13:33:09.494173 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:33:09.493997 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d079b23f-a7c7-43c3-b1a9-c213a7edd0ea-trusted-ca-bundle\") pod \"console-778b57cc48-jwmrp\" (UID: \"d079b23f-a7c7-43c3-b1a9-c213a7edd0ea\") " pod="openshift-console/console-778b57cc48-jwmrp" Sep 10 13:33:09.595958 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:33:09.595198 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ht9xd\" (UniqueName: \"kubernetes.io/projected/d079b23f-a7c7-43c3-b1a9-c213a7edd0ea-kube-api-access-ht9xd\") pod \"console-778b57cc48-jwmrp\" (UID: \"d079b23f-a7c7-43c3-b1a9-c213a7edd0ea\") " pod="openshift-console/console-778b57cc48-jwmrp" Sep 10 13:33:09.595958 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:33:09.595269 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d079b23f-a7c7-43c3-b1a9-c213a7edd0ea-trusted-ca-bundle\") pod \"console-778b57cc48-jwmrp\" (UID: \"d079b23f-a7c7-43c3-b1a9-c213a7edd0ea\") " pod="openshift-console/console-778b57cc48-jwmrp" Sep 10 13:33:09.595958 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:33:09.595332 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/d079b23f-a7c7-43c3-b1a9-c213a7edd0ea-console-oauth-config\") pod \"console-778b57cc48-jwmrp\" (UID: \"d079b23f-a7c7-43c3-b1a9-c213a7edd0ea\") " pod="openshift-console/console-778b57cc48-jwmrp" Sep 10 13:33:09.595958 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:33:09.595375 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/d079b23f-a7c7-43c3-b1a9-c213a7edd0ea-console-config\") pod \"console-778b57cc48-jwmrp\" (UID: \"d079b23f-a7c7-43c3-b1a9-c213a7edd0ea\") " pod="openshift-console/console-778b57cc48-jwmrp" Sep 10 13:33:09.595958 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:33:09.595402 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/d079b23f-a7c7-43c3-b1a9-c213a7edd0ea-oauth-serving-cert\") pod \"console-778b57cc48-jwmrp\" (UID: \"d079b23f-a7c7-43c3-b1a9-c213a7edd0ea\") " pod="openshift-console/console-778b57cc48-jwmrp" Sep 10 13:33:09.595958 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:33:09.595430 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/d079b23f-a7c7-43c3-b1a9-c213a7edd0ea-service-ca\") pod \"console-778b57cc48-jwmrp\" (UID: \"d079b23f-a7c7-43c3-b1a9-c213a7edd0ea\") " pod="openshift-console/console-778b57cc48-jwmrp" Sep 10 13:33:09.595958 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:33:09.595458 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/d079b23f-a7c7-43c3-b1a9-c213a7edd0ea-console-serving-cert\") pod \"console-778b57cc48-jwmrp\" (UID: \"d079b23f-a7c7-43c3-b1a9-c213a7edd0ea\") " pod="openshift-console/console-778b57cc48-jwmrp" Sep 10 13:33:09.596767 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:33:09.596701 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/d079b23f-a7c7-43c3-b1a9-c213a7edd0ea-oauth-serving-cert\") pod \"console-778b57cc48-jwmrp\" (UID: \"d079b23f-a7c7-43c3-b1a9-c213a7edd0ea\") " pod="openshift-console/console-778b57cc48-jwmrp" Sep 10 13:33:09.596965 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:33:09.596782 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/d079b23f-a7c7-43c3-b1a9-c213a7edd0ea-service-ca\") pod \"console-778b57cc48-jwmrp\" (UID: \"d079b23f-a7c7-43c3-b1a9-c213a7edd0ea\") " pod="openshift-console/console-778b57cc48-jwmrp" Sep 10 13:33:09.596965 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:33:09.596934 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d079b23f-a7c7-43c3-b1a9-c213a7edd0ea-trusted-ca-bundle\") pod \"console-778b57cc48-jwmrp\" (UID: \"d079b23f-a7c7-43c3-b1a9-c213a7edd0ea\") " pod="openshift-console/console-778b57cc48-jwmrp" Sep 10 13:33:09.596965 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:33:09.596941 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/d079b23f-a7c7-43c3-b1a9-c213a7edd0ea-console-config\") pod \"console-778b57cc48-jwmrp\" (UID: \"d079b23f-a7c7-43c3-b1a9-c213a7edd0ea\") " pod="openshift-console/console-778b57cc48-jwmrp" Sep 10 13:33:09.598985 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:33:09.598945 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/d079b23f-a7c7-43c3-b1a9-c213a7edd0ea-console-serving-cert\") pod \"console-778b57cc48-jwmrp\" (UID: \"d079b23f-a7c7-43c3-b1a9-c213a7edd0ea\") " pod="openshift-console/console-778b57cc48-jwmrp" Sep 10 13:33:09.599075 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:33:09.599018 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/d079b23f-a7c7-43c3-b1a9-c213a7edd0ea-console-oauth-config\") pod \"console-778b57cc48-jwmrp\" (UID: \"d079b23f-a7c7-43c3-b1a9-c213a7edd0ea\") " pod="openshift-console/console-778b57cc48-jwmrp" Sep 10 13:33:09.607864 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:33:09.607835 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ht9xd\" (UniqueName: \"kubernetes.io/projected/d079b23f-a7c7-43c3-b1a9-c213a7edd0ea-kube-api-access-ht9xd\") pod \"console-778b57cc48-jwmrp\" (UID: \"d079b23f-a7c7-43c3-b1a9-c213a7edd0ea\") " pod="openshift-console/console-778b57cc48-jwmrp" Sep 10 13:33:09.630786 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:33:09.630746 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-778b57cc48-jwmrp" Sep 10 13:33:10.850902 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:33:10.850859 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fxgmp_fa53c783-d40a-4024-9f7b-1d0d2b57f6dc/konnectivity-agent/0.log" limit=1024 Sep 10 13:33:10.930268 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:33:10.930224 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fxgmp_fa53c783-d40a-4024-9f7b-1d0d2b57f6dc/konnectivity-agent/0.log" limit=1024 Sep 10 13:33:11.093106 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:33:11.093072 2585 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-75ff89b47-w9lfj" Sep 10 13:33:11.093316 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:33:11.093169 2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-75ff89b47-w9lfj" Sep 10 13:33:11.098612 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:33:11.098585 2585 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-75ff89b47-w9lfj" Sep 10 13:33:11.147675 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:33:11.147591 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fxgmp_fa53c783-d40a-4024-9f7b-1d0d2b57f6dc/konnectivity-agent/0.log" limit=1024 Sep 10 13:33:11.255543 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:33:11.255507 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fxgmp_fa53c783-d40a-4024-9f7b-1d0d2b57f6dc/konnectivity-agent/0.log" limit=1024 Sep 10 13:33:11.506378 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:33:11.506288 2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-75ff89b47-w9lfj" Sep 10 13:33:11.552876 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:33:11.552844 2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/metrics-server-6ffbdddb-wjr2c" Sep 10 13:33:11.553046 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:33:11.552907 2585 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-monitoring/metrics-server-6ffbdddb-wjr2c" Sep 10 13:33:12.679695 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:33:12.679612 2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-7262t"] Sep 10 13:33:12.680560 ip-10-0-140-8 kubenswrapper[2585]: W0910 13:33:12.680514 2585 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod082a7577_a14f_49fb_a9b1_3cf250e6735a.slice/crio-07d104576f54189fa758b82066de3c365e01f6f83740bd9e123cc84d1f362b24 WatchSource:0}: Error finding container 07d104576f54189fa758b82066de3c365e01f6f83740bd9e123cc84d1f362b24: Status 404 returned error can't find the container with id 07d104576f54189fa758b82066de3c365e01f6f83740bd9e123cc84d1f362b24 Sep 10 13:33:12.689194 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:33:12.689170 2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-xp5dk"] Sep 10 13:33:12.691177 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:33:12.691153 2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-778b57cc48-jwmrp"] Sep 10 13:33:12.693630 ip-10-0-140-8 kubenswrapper[2585]: W0910 13:33:12.693516 2585 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd552154f_1d5b_4c5d_99bf_7aeef22ce032.slice/crio-c57ee64eb21921a6d5dbb61b7d282a707c4a2b675db1c00a9b809af1fe414beb WatchSource:0}: Error finding container c57ee64eb21921a6d5dbb61b7d282a707c4a2b675db1c00a9b809af1fe414beb: Status 404 returned error can't find the container with id c57ee64eb21921a6d5dbb61b7d282a707c4a2b675db1c00a9b809af1fe414beb Sep 10 13:33:12.694794 ip-10-0-140-8 kubenswrapper[2585]: W0910 13:33:12.694767 2585 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd079b23f_a7c7_43c3_b1a9_c213a7edd0ea.slice/crio-3cff41dd5dd5620d929acfa2190f96f87fd74f5592b84ba5e848c64a621f392e WatchSource:0}: Error finding container 3cff41dd5dd5620d929acfa2190f96f87fd74f5592b84ba5e848c64a621f392e: Status 404 returned error can't find the container with id 3cff41dd5dd5620d929acfa2190f96f87fd74f5592b84ba5e848c64a621f392e Sep 10 13:33:12.730321 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:33:12.730218 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fxgmp_fa53c783-d40a-4024-9f7b-1d0d2b57f6dc/konnectivity-agent/0.log" limit=1024 Sep 10 13:33:12.831686 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:33:12.831653 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fxgmp_fa53c783-d40a-4024-9f7b-1d0d2b57f6dc/konnectivity-agent/0.log" limit=1024 Sep 10 13:33:13.512209 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:33:13.512173 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-7262t" event={"ID":"082a7577-a14f-49fb-a9b1-3cf250e6735a","Type":"ContainerStarted","Data":"07d104576f54189fa758b82066de3c365e01f6f83740bd9e123cc84d1f362b24"} Sep 10 13:33:13.514315 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:33:13.514285 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/downloads-85d98c998f-cjph8" event={"ID":"f9a6f8e4-4890-463b-b583-c37100f359e5","Type":"ContainerStarted","Data":"88799b06395daccd6484f57236db2ce87602ca89b7fa0ed5922bfb3fdf6fec50"} Sep 10 13:33:13.515799 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:33:13.515766 2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/downloads-85d98c998f-cjph8" Sep 10 13:33:13.520138 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:33:13.519974 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-xp5dk" event={"ID":"d552154f-1d5b-4c5d-99bf-7aeef22ce032","Type":"ContainerStarted","Data":"c57ee64eb21921a6d5dbb61b7d282a707c4a2b675db1c00a9b809af1fe414beb"} Sep 10 13:33:13.521490 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:33:13.521459 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-778b57cc48-jwmrp" event={"ID":"d079b23f-a7c7-43c3-b1a9-c213a7edd0ea","Type":"ContainerStarted","Data":"0f6f5c8889419889bd8039a9243efef7cf85a4b02f1e48c4406778c127de3e51"} Sep 10 13:33:13.521607 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:33:13.521515 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-778b57cc48-jwmrp" event={"ID":"d079b23f-a7c7-43c3-b1a9-c213a7edd0ea","Type":"ContainerStarted","Data":"3cff41dd5dd5620d929acfa2190f96f87fd74f5592b84ba5e848c64a621f392e"} Sep 10 13:33:13.532919 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:33:13.532764 2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/downloads-85d98c998f-cjph8" Sep 10 13:33:13.542826 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:33:13.542657 2585 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/downloads-85d98c998f-cjph8" podStartSLOduration=2.631929713 podStartE2EDuration="19.54263772s" totalImagesPullingTime="16.910708007s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 13:32:54 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 13:33:13.54089425 +0000 UTC m=+278.413598454" watchObservedRunningTime="2026-09-10 13:33:13.54263772 +0000 UTC m=+278.415341932" Sep 10 13:33:13.569851 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:33:13.569797 2585 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-778b57cc48-jwmrp" podStartSLOduration=4.569774492 podStartE2EDuration="4.569774492s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 13:33:09 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 13:33:13.568582299 +0000 UTC m=+278.441286520" watchObservedRunningTime="2026-09-10 13:33:13.569774492 +0000 UTC m=+278.442478694" Sep 10 13:33:15.534530 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:33:15.534479 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-xp5dk" event={"ID":"d552154f-1d5b-4c5d-99bf-7aeef22ce032","Type":"ContainerStarted","Data":"4a8e866abb8e2f5dd5f62735f049e2b8c6ae492bcbb2acb32999daa721678507"} Sep 10 13:33:15.535025 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:33:15.534591 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-xp5dk" event={"ID":"d552154f-1d5b-4c5d-99bf-7aeef22ce032","Type":"ContainerStarted","Data":"bf65752140364bbd40d11e011f70d5125ace55c39e1e267538c44720af8a6c62"} Sep 10 13:33:15.562944 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:33:15.559770 2585 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/network-metrics-daemon-xp5dk" podStartSLOduration=258.718206019 podStartE2EDuration="4m20.559732572s" totalImagesPullingTime="1.841526553s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 13:28:55 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 13:33:15.557711914 +0000 UTC m=+280.430416140" watchObservedRunningTime="2026-09-10 13:33:15.559732572 +0000 UTC m=+280.432436771" Sep 10 13:33:15.717483 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:33:15.717446 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fxgmp_fa53c783-d40a-4024-9f7b-1d0d2b57f6dc/konnectivity-agent/0.log" limit=1024 Sep 10 13:33:15.819644 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:33:15.819612 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fxgmp_fa53c783-d40a-4024-9f7b-1d0d2b57f6dc/konnectivity-agent/0.log" limit=1024 Sep 10 13:33:15.920960 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:33:15.920888 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fxgmp_fa53c783-d40a-4024-9f7b-1d0d2b57f6dc/konnectivity-agent/0.log" limit=1024 Sep 10 13:33:17.614620 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:33:17.614515 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fxgmp_fa53c783-d40a-4024-9f7b-1d0d2b57f6dc/konnectivity-agent/0.log" limit=1024 Sep 10 13:33:17.718428 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:33:17.718354 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fxgmp_fa53c783-d40a-4024-9f7b-1d0d2b57f6dc/konnectivity-agent/0.log" limit=1024 Sep 10 13:33:17.790527 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:33:17.790485 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fxgmp_fa53c783-d40a-4024-9f7b-1d0d2b57f6dc/konnectivity-agent/0.log" limit=1024 Sep 10 13:33:19.027075 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:33:19.027034 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fxgmp_fa53c783-d40a-4024-9f7b-1d0d2b57f6dc/konnectivity-agent/0.log" limit=1024 Sep 10 13:33:19.127497 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:33:19.127451 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fxgmp_fa53c783-d40a-4024-9f7b-1d0d2b57f6dc/konnectivity-agent/0.log" limit=1024 Sep 10 13:33:19.553800 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:33:19.553760 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-lhr9l" event={"ID":"b37f8a5b-3444-4775-8751-b60c859154d9","Type":"ContainerStarted","Data":"e141150ffed406832ded98d3a0745b8c2d2e9773f9bc197d5321c10a562cf945"} Sep 10 13:33:19.555547 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:33:19.555520 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-7262t" event={"ID":"082a7577-a14f-49fb-a9b1-3cf250e6735a","Type":"ContainerStarted","Data":"fd524650734317691df2c9e864071fd87f26404b264581f824ee6719ce3e31c1"} Sep 10 13:33:19.555674 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:33:19.555577 2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-7262t" Sep 10 13:33:19.574830 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:33:19.574770 2585 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/global-pull-secret-syncer-lhr9l" podStartSLOduration=129.504264075 podStartE2EDuration="2m23.574747803s" totalImagesPullingTime="14.070483728s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-10 13:30:56 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 13:33:19.573195277 +0000 UTC m=+284.445899478" watchObservedRunningTime="2026-09-10 13:33:19.574747803 +0000 UTC m=+284.447452020" Sep 10 13:33:19.631502 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:33:19.631461 2585 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-778b57cc48-jwmrp" Sep 10 13:33:19.631688 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:33:19.631562 2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-778b57cc48-jwmrp" Sep 10 13:33:19.636631 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:33:19.636605 2585 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-778b57cc48-jwmrp" Sep 10 13:33:19.659997 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:33:19.659947 2585 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-target-7262t" podStartSLOduration=258.289484037 podStartE2EDuration="4m24.659925581s" totalImagesPullingTime="6.370441544s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 13:28:55 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 13:33:19.594122629 +0000 UTC m=+284.466826831" watchObservedRunningTime="2026-09-10 13:33:19.659925581 +0000 UTC m=+284.532629798" Sep 10 13:33:20.563992 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:33:20.563953 2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-778b57cc48-jwmrp" Sep 10 13:33:20.661565 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:33:20.661533 2585 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-75ff89b47-w9lfj"] Sep 10 13:33:21.285688 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:33:21.285651 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fxgmp_fa53c783-d40a-4024-9f7b-1d0d2b57f6dc/konnectivity-agent/0.log" limit=1024 Sep 10 13:33:21.384736 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:33:21.384702 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fxgmp_fa53c783-d40a-4024-9f7b-1d0d2b57f6dc/konnectivity-agent/0.log" limit=1024 Sep 10 13:33:25.086691 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:33:25.086660 2585 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-5b86cf75d6-hccrk"] Sep 10 13:33:25.095148 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:33:25.095119 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fxgmp_fa53c783-d40a-4024-9f7b-1d0d2b57f6dc/konnectivity-agent/0.log" limit=1024 Sep 10 13:33:25.107805 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:33:25.107775 2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-5b86cf75d6-hccrk"] Sep 10 13:33:25.108201 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:33:25.107886 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-5b86cf75d6-hccrk" Sep 10 13:33:25.110446 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:33:25.110425 2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"open-cluster-management-agent-addon\"/\"kube-root-ca.crt\"" Sep 10 13:33:25.110554 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:33:25.110491 2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"open-cluster-management-image-pull-credentials\"" Sep 10 13:33:25.111165 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:33:25.111146 2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"open-cluster-management-agent-addon\"/\"openshift-service-ca.crt\"" Sep 10 13:33:25.111712 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:33:25.111694 2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"managed-serviceaccount-hub-kubeconfig\"" Sep 10 13:33:25.112290 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:33:25.112273 2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"managed-serviceaccount-dockercfg-hd5vd\"" Sep 10 13:33:25.132211 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:33:25.132184 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/5e2904d0-d884-470e-8197-3b7c0cf6b7a4-hub-kubeconfig\") pod \"managed-serviceaccount-addon-agent-5b86cf75d6-hccrk\" (UID: \"5e2904d0-d884-470e-8197-3b7c0cf6b7a4\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-5b86cf75d6-hccrk" Sep 10 13:33:25.132358 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:33:25.132266 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jf7wr\" (UniqueName: \"kubernetes.io/projected/5e2904d0-d884-470e-8197-3b7c0cf6b7a4-kube-api-access-jf7wr\") pod \"managed-serviceaccount-addon-agent-5b86cf75d6-hccrk\" (UID: \"5e2904d0-d884-470e-8197-3b7c0cf6b7a4\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-5b86cf75d6-hccrk" Sep 10 13:33:25.201509 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:33:25.201477 2585 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["open-cluster-management-agent-addon/klusterlet-addon-workmgr-5d567556cf-zbqdn"] Sep 10 13:33:25.203577 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:33:25.203548 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fxgmp_fa53c783-d40a-4024-9f7b-1d0d2b57f6dc/konnectivity-agent/0.log" limit=1024 Sep 10 13:33:25.216603 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:33:25.216579 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-5d567556cf-zbqdn" Sep 10 13:33:25.219314 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:33:25.219294 2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"work-manager-hub-kubeconfig\"" Sep 10 13:33:25.225069 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:33:25.225044 2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/klusterlet-addon-workmgr-5d567556cf-zbqdn"] Sep 10 13:33:25.233342 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:33:25.233317 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"klusterlet-config\" (UniqueName: \"kubernetes.io/secret/c0de58ce-49b1-4c71-84ff-83e0e6dd1c7e-klusterlet-config\") pod \"klusterlet-addon-workmgr-5d567556cf-zbqdn\" (UID: \"c0de58ce-49b1-4c71-84ff-83e0e6dd1c7e\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-5d567556cf-zbqdn" Sep 10 13:33:25.233470 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:33:25.233354 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/c0de58ce-49b1-4c71-84ff-83e0e6dd1c7e-tmp\") pod \"klusterlet-addon-workmgr-5d567556cf-zbqdn\" (UID: \"c0de58ce-49b1-4c71-84ff-83e0e6dd1c7e\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-5d567556cf-zbqdn" Sep 10 13:33:25.233470 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:33:25.233393 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/5e2904d0-d884-470e-8197-3b7c0cf6b7a4-hub-kubeconfig\") pod \"managed-serviceaccount-addon-agent-5b86cf75d6-hccrk\" (UID: \"5e2904d0-d884-470e-8197-3b7c0cf6b7a4\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-5b86cf75d6-hccrk" Sep 10 13:33:25.233470 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:33:25.233451 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rh984\" (UniqueName: \"kubernetes.io/projected/c0de58ce-49b1-4c71-84ff-83e0e6dd1c7e-kube-api-access-rh984\") pod \"klusterlet-addon-workmgr-5d567556cf-zbqdn\" (UID: \"c0de58ce-49b1-4c71-84ff-83e0e6dd1c7e\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-5d567556cf-zbqdn" Sep 10 13:33:25.233598 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:33:25.233504 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jf7wr\" (UniqueName: \"kubernetes.io/projected/5e2904d0-d884-470e-8197-3b7c0cf6b7a4-kube-api-access-jf7wr\") pod \"managed-serviceaccount-addon-agent-5b86cf75d6-hccrk\" (UID: \"5e2904d0-d884-470e-8197-3b7c0cf6b7a4\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-5b86cf75d6-hccrk" Sep 10 13:33:25.236056 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:33:25.236025 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/5e2904d0-d884-470e-8197-3b7c0cf6b7a4-hub-kubeconfig\") pod \"managed-serviceaccount-addon-agent-5b86cf75d6-hccrk\" (UID: \"5e2904d0-d884-470e-8197-3b7c0cf6b7a4\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-5b86cf75d6-hccrk" Sep 10 13:33:25.248901 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:33:25.248870 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jf7wr\" (UniqueName: \"kubernetes.io/projected/5e2904d0-d884-470e-8197-3b7c0cf6b7a4-kube-api-access-jf7wr\") pod \"managed-serviceaccount-addon-agent-5b86cf75d6-hccrk\" (UID: \"5e2904d0-d884-470e-8197-3b7c0cf6b7a4\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-5b86cf75d6-hccrk" Sep 10 13:33:25.283892 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:33:25.283858 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fxgmp_fa53c783-d40a-4024-9f7b-1d0d2b57f6dc/konnectivity-agent/0.log" limit=1024 Sep 10 13:33:25.333987 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:33:25.333952 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rh984\" (UniqueName: \"kubernetes.io/projected/c0de58ce-49b1-4c71-84ff-83e0e6dd1c7e-kube-api-access-rh984\") pod \"klusterlet-addon-workmgr-5d567556cf-zbqdn\" (UID: \"c0de58ce-49b1-4c71-84ff-83e0e6dd1c7e\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-5d567556cf-zbqdn" Sep 10 13:33:25.334146 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:33:25.334015 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"klusterlet-config\" (UniqueName: \"kubernetes.io/secret/c0de58ce-49b1-4c71-84ff-83e0e6dd1c7e-klusterlet-config\") pod \"klusterlet-addon-workmgr-5d567556cf-zbqdn\" (UID: \"c0de58ce-49b1-4c71-84ff-83e0e6dd1c7e\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-5d567556cf-zbqdn" Sep 10 13:33:25.334146 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:33:25.334042 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/c0de58ce-49b1-4c71-84ff-83e0e6dd1c7e-tmp\") pod \"klusterlet-addon-workmgr-5d567556cf-zbqdn\" (UID: \"c0de58ce-49b1-4c71-84ff-83e0e6dd1c7e\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-5d567556cf-zbqdn" Sep 10 13:33:25.334453 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:33:25.334437 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/c0de58ce-49b1-4c71-84ff-83e0e6dd1c7e-tmp\") pod \"klusterlet-addon-workmgr-5d567556cf-zbqdn\" (UID: \"c0de58ce-49b1-4c71-84ff-83e0e6dd1c7e\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-5d567556cf-zbqdn" Sep 10 13:33:25.336478 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:33:25.336459 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"klusterlet-config\" (UniqueName: \"kubernetes.io/secret/c0de58ce-49b1-4c71-84ff-83e0e6dd1c7e-klusterlet-config\") pod \"klusterlet-addon-workmgr-5d567556cf-zbqdn\" (UID: \"c0de58ce-49b1-4c71-84ff-83e0e6dd1c7e\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-5d567556cf-zbqdn" Sep 10 13:33:25.344196 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:33:25.344142 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rh984\" (UniqueName: \"kubernetes.io/projected/c0de58ce-49b1-4c71-84ff-83e0e6dd1c7e-kube-api-access-rh984\") pod \"klusterlet-addon-workmgr-5d567556cf-zbqdn\" (UID: \"c0de58ce-49b1-4c71-84ff-83e0e6dd1c7e\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-5d567556cf-zbqdn" Sep 10 13:33:25.427608 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:33:25.427575 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-5b86cf75d6-hccrk" Sep 10 13:33:25.526794 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:33:25.526763 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-5d567556cf-zbqdn" Sep 10 13:33:25.711530 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:33:25.711499 2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-5b86cf75d6-hccrk"] Sep 10 13:33:25.714432 ip-10-0-140-8 kubenswrapper[2585]: W0910 13:33:25.714406 2585 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod5e2904d0_d884_470e_8197_3b7c0cf6b7a4.slice/crio-7da2e57bd299210514a2684c659a21889863ddbdea2c93ebcba4f3eaaf79842f WatchSource:0}: Error finding container 7da2e57bd299210514a2684c659a21889863ddbdea2c93ebcba4f3eaaf79842f: Status 404 returned error can't find the container with id 7da2e57bd299210514a2684c659a21889863ddbdea2c93ebcba4f3eaaf79842f Sep 10 13:33:25.818652 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:33:25.818464 2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/klusterlet-addon-workmgr-5d567556cf-zbqdn"] Sep 10 13:33:25.821023 ip-10-0-140-8 kubenswrapper[2585]: W0910 13:33:25.820996 2585 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc0de58ce_49b1_4c71_84ff_83e0e6dd1c7e.slice/crio-9aace6bc00418c344ac1010f6a37443a9daea89bd2281dd8c5868cf99cf39556 WatchSource:0}: Error finding container 9aace6bc00418c344ac1010f6a37443a9daea89bd2281dd8c5868cf99cf39556: Status 404 returned error can't find the container with id 9aace6bc00418c344ac1010f6a37443a9daea89bd2281dd8c5868cf99cf39556 Sep 10 13:33:26.586414 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:33:26.586378 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-5d567556cf-zbqdn" event={"ID":"c0de58ce-49b1-4c71-84ff-83e0e6dd1c7e","Type":"ContainerStarted","Data":"9aace6bc00418c344ac1010f6a37443a9daea89bd2281dd8c5868cf99cf39556"} Sep 10 13:33:26.588191 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:33:26.588159 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-5b86cf75d6-hccrk" event={"ID":"5e2904d0-d884-470e-8197-3b7c0cf6b7a4","Type":"ContainerStarted","Data":"7da2e57bd299210514a2684c659a21889863ddbdea2c93ebcba4f3eaaf79842f"} Sep 10 13:33:31.558927 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:33:31.558899 2585 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-monitoring/metrics-server-6ffbdddb-wjr2c" Sep 10 13:33:31.563122 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:33:31.563101 2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/metrics-server-6ffbdddb-wjr2c" Sep 10 13:33:31.613200 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:33:31.613173 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-5d567556cf-zbqdn" event={"ID":"c0de58ce-49b1-4c71-84ff-83e0e6dd1c7e","Type":"ContainerStarted","Data":"da57501b00926bb2284d927c2d98885ef1a5c892b1109d40fa4ab3fbc12c914c"} Sep 10 13:33:31.614753 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:33:31.614720 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-5b86cf75d6-hccrk" event={"ID":"5e2904d0-d884-470e-8197-3b7c0cf6b7a4","Type":"ContainerStarted","Data":"5d1534f407e677ed3ab9981c91a160e35496f829c9a5e61945f758afc17b3e27"} Sep 10 13:33:31.631798 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:33:31.631672 2585 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-5b86cf75d6-hccrk" podStartSLOduration=1.283906065 podStartE2EDuration="6.63165322s" totalImagesPullingTime="5.347747155s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 13:33:25 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 13:33:31.63106971 +0000 UTC m=+296.503773938" watchObservedRunningTime="2026-09-10 13:33:31.63165322 +0000 UTC m=+296.504357422" Sep 10 13:33:32.638204 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:33:32.638151 2585 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-5d567556cf-zbqdn" podStartSLOduration=1.969577514 podStartE2EDuration="7.63813412s" totalImagesPullingTime="5.668556606s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 13:33:25 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 13:33:32.636617703 +0000 UTC m=+297.509321903" watchObservedRunningTime="2026-09-10 13:33:32.63813412 +0000 UTC m=+297.510838321" Sep 10 13:33:34.003766 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:33:34.003716 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fxgmp_fa53c783-d40a-4024-9f7b-1d0d2b57f6dc/konnectivity-agent/0.log" limit=1024 Sep 10 13:33:34.064391 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:33:34.064360 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fxgmp_fa53c783-d40a-4024-9f7b-1d0d2b57f6dc/konnectivity-agent/0.log" limit=1024 Sep 10 13:33:35.569843 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:33:35.569814 2585 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-l9l8q_ad555c6a-4385-4f8e-93bb-a174cfaa8e74/kube-multus/2.log" Sep 10 13:33:35.570301 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:33:35.569973 2585 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-l9l8q_ad555c6a-4385-4f8e-93bb-a174cfaa8e74/kube-multus/2.log" Sep 10 13:33:35.572877 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:33:35.572856 2585 kubelet.go:1648] "Image garbage collection succeeded" Sep 10 13:33:42.618543 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:33:42.618502 2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-5d567556cf-zbqdn" Sep 10 13:33:42.645461 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:33:42.619280 2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-5d567556cf-zbqdn" Sep 10 13:33:45.685898 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:33:45.685863 2585 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-console/console-75ff89b47-w9lfj" podUID="1faeed93-e4d2-434b-ab30-6bc7b9881a0b" containerName="console" containerID="cri-o://08409cb3d6867e9f17694afd26253d4ba52db7cfc6c04752967c2aae0cf2d93b" gracePeriod=15 Sep 10 13:33:45.944762 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:33:45.944699 2585 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-75ff89b47-w9lfj" Sep 10 13:33:45.979360 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:33:45.979335 2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/1faeed93-e4d2-434b-ab30-6bc7b9881a0b-oauth-serving-cert\") pod \"1faeed93-e4d2-434b-ab30-6bc7b9881a0b\" (UID: \"1faeed93-e4d2-434b-ab30-6bc7b9881a0b\") " Sep 10 13:33:45.979526 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:33:45.979400 2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/1faeed93-e4d2-434b-ab30-6bc7b9881a0b-service-ca\") pod \"1faeed93-e4d2-434b-ab30-6bc7b9881a0b\" (UID: \"1faeed93-e4d2-434b-ab30-6bc7b9881a0b\") " Sep 10 13:33:45.979526 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:33:45.979426 2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/1faeed93-e4d2-434b-ab30-6bc7b9881a0b-console-config\") pod \"1faeed93-e4d2-434b-ab30-6bc7b9881a0b\" (UID: \"1faeed93-e4d2-434b-ab30-6bc7b9881a0b\") " Sep 10 13:33:45.979526 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:33:45.979482 2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-qfcvr\" (UniqueName: \"kubernetes.io/projected/1faeed93-e4d2-434b-ab30-6bc7b9881a0b-kube-api-access-qfcvr\") pod \"1faeed93-e4d2-434b-ab30-6bc7b9881a0b\" (UID: \"1faeed93-e4d2-434b-ab30-6bc7b9881a0b\") " Sep 10 13:33:45.979526 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:33:45.979506 2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/1faeed93-e4d2-434b-ab30-6bc7b9881a0b-console-serving-cert\") pod \"1faeed93-e4d2-434b-ab30-6bc7b9881a0b\" (UID: \"1faeed93-e4d2-434b-ab30-6bc7b9881a0b\") " Sep 10 13:33:45.979744 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:33:45.979532 2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/1faeed93-e4d2-434b-ab30-6bc7b9881a0b-console-oauth-config\") pod \"1faeed93-e4d2-434b-ab30-6bc7b9881a0b\" (UID: \"1faeed93-e4d2-434b-ab30-6bc7b9881a0b\") " Sep 10 13:33:45.979839 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:33:45.979811 2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1faeed93-e4d2-434b-ab30-6bc7b9881a0b-oauth-serving-cert" (OuterVolumeSpecName: "oauth-serving-cert") pod "1faeed93-e4d2-434b-ab30-6bc7b9881a0b" (UID: "1faeed93-e4d2-434b-ab30-6bc7b9881a0b"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 10 13:33:45.979937 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:33:45.979904 2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1faeed93-e4d2-434b-ab30-6bc7b9881a0b-service-ca" (OuterVolumeSpecName: "service-ca") pod "1faeed93-e4d2-434b-ab30-6bc7b9881a0b" (UID: "1faeed93-e4d2-434b-ab30-6bc7b9881a0b"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 10 13:33:45.980005 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:33:45.979913 2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1faeed93-e4d2-434b-ab30-6bc7b9881a0b-console-config" (OuterVolumeSpecName: "console-config") pod "1faeed93-e4d2-434b-ab30-6bc7b9881a0b" (UID: "1faeed93-e4d2-434b-ab30-6bc7b9881a0b"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 10 13:33:45.981932 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:33:45.981905 2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1faeed93-e4d2-434b-ab30-6bc7b9881a0b-kube-api-access-qfcvr" (OuterVolumeSpecName: "kube-api-access-qfcvr") pod "1faeed93-e4d2-434b-ab30-6bc7b9881a0b" (UID: "1faeed93-e4d2-434b-ab30-6bc7b9881a0b"). InnerVolumeSpecName "kube-api-access-qfcvr". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 10 13:33:45.982052 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:33:45.981909 2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1faeed93-e4d2-434b-ab30-6bc7b9881a0b-console-serving-cert" (OuterVolumeSpecName: "console-serving-cert") pod "1faeed93-e4d2-434b-ab30-6bc7b9881a0b" (UID: "1faeed93-e4d2-434b-ab30-6bc7b9881a0b"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 10 13:33:45.982052 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:33:45.981976 2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1faeed93-e4d2-434b-ab30-6bc7b9881a0b-console-oauth-config" (OuterVolumeSpecName: "console-oauth-config") pod "1faeed93-e4d2-434b-ab30-6bc7b9881a0b" (UID: "1faeed93-e4d2-434b-ab30-6bc7b9881a0b"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 10 13:33:46.080215 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:33:46.080177 2585 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qfcvr\" (UniqueName: \"kubernetes.io/projected/1faeed93-e4d2-434b-ab30-6bc7b9881a0b-kube-api-access-qfcvr\") on node \"ip-10-0-140-8.ec2.internal\" DevicePath \"\"" Sep 10 13:33:46.080215 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:33:46.080205 2585 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/1faeed93-e4d2-434b-ab30-6bc7b9881a0b-console-serving-cert\") on node \"ip-10-0-140-8.ec2.internal\" DevicePath \"\"" Sep 10 13:33:46.080215 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:33:46.080215 2585 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/1faeed93-e4d2-434b-ab30-6bc7b9881a0b-console-oauth-config\") on node \"ip-10-0-140-8.ec2.internal\" DevicePath \"\"" Sep 10 13:33:46.080215 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:33:46.080224 2585 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/1faeed93-e4d2-434b-ab30-6bc7b9881a0b-oauth-serving-cert\") on node \"ip-10-0-140-8.ec2.internal\" DevicePath \"\"" Sep 10 13:33:46.080501 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:33:46.080235 2585 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/1faeed93-e4d2-434b-ab30-6bc7b9881a0b-service-ca\") on node \"ip-10-0-140-8.ec2.internal\" DevicePath \"\"" Sep 10 13:33:46.080501 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:33:46.080266 2585 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/1faeed93-e4d2-434b-ab30-6bc7b9881a0b-console-config\") on node \"ip-10-0-140-8.ec2.internal\" DevicePath \"\"" Sep 10 13:33:46.677188 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:33:46.677148 2585 generic.go:356] "Generic (PLEG): container finished" podID="1faeed93-e4d2-434b-ab30-6bc7b9881a0b" containerID="08409cb3d6867e9f17694afd26253d4ba52db7cfc6c04752967c2aae0cf2d93b" exitCode=0 Sep 10 13:33:46.677387 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:33:46.677203 2585 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-75ff89b47-w9lfj" Sep 10 13:33:46.677387 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:33:46.677235 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-75ff89b47-w9lfj" event={"ID":"1faeed93-e4d2-434b-ab30-6bc7b9881a0b","Type":"ContainerDied","Data":"08409cb3d6867e9f17694afd26253d4ba52db7cfc6c04752967c2aae0cf2d93b"} Sep 10 13:33:46.677387 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:33:46.677339 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-75ff89b47-w9lfj" event={"ID":"1faeed93-e4d2-434b-ab30-6bc7b9881a0b","Type":"ContainerDied","Data":"353caed72129be6b38164b1c7d613151b596fa7e95a4e4377d0295d6f695fc2c"} Sep 10 13:33:46.677387 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:33:46.677361 2585 scope.go:117] "RemoveContainer" containerID="08409cb3d6867e9f17694afd26253d4ba52db7cfc6c04752967c2aae0cf2d93b" Sep 10 13:33:46.686152 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:33:46.686136 2585 scope.go:117] "RemoveContainer" containerID="08409cb3d6867e9f17694afd26253d4ba52db7cfc6c04752967c2aae0cf2d93b" Sep 10 13:33:46.686470 ip-10-0-140-8 kubenswrapper[2585]: E0910 13:33:46.686434 2585 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"08409cb3d6867e9f17694afd26253d4ba52db7cfc6c04752967c2aae0cf2d93b\": container with ID starting with 08409cb3d6867e9f17694afd26253d4ba52db7cfc6c04752967c2aae0cf2d93b not found: ID does not exist" containerID="08409cb3d6867e9f17694afd26253d4ba52db7cfc6c04752967c2aae0cf2d93b" Sep 10 13:33:46.686518 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:33:46.686467 2585 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"08409cb3d6867e9f17694afd26253d4ba52db7cfc6c04752967c2aae0cf2d93b"} err="failed to get container status \"08409cb3d6867e9f17694afd26253d4ba52db7cfc6c04752967c2aae0cf2d93b\": rpc error: code = NotFound desc = could not find container \"08409cb3d6867e9f17694afd26253d4ba52db7cfc6c04752967c2aae0cf2d93b\": container with ID starting with 08409cb3d6867e9f17694afd26253d4ba52db7cfc6c04752967c2aae0cf2d93b not found: ID does not exist" Sep 10 13:33:46.702279 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:33:46.702235 2585 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-75ff89b47-w9lfj"] Sep 10 13:33:46.707191 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:33:46.707168 2585 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-75ff89b47-w9lfj"] Sep 10 13:33:47.646323 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:33:47.646292 2585 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="1faeed93-e4d2-434b-ab30-6bc7b9881a0b" path="/var/lib/kubelet/pods/1faeed93-e4d2-434b-ab30-6bc7b9881a0b/volumes" Sep 10 13:33:50.563013 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:33:50.562985 2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-network-diagnostics/network-check-target-7262t" Sep 10 13:34:06.141358 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:34:06.141296 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fxgmp_fa53c783-d40a-4024-9f7b-1d0d2b57f6dc/konnectivity-agent/0.log" limit=1024 Sep 10 13:34:06.342169 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:34:06.342133 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fxgmp_fa53c783-d40a-4024-9f7b-1d0d2b57f6dc/konnectivity-agent/0.log" limit=1024 Sep 10 13:34:09.269361 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:34:09.269326 2585 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Sep 10 13:34:09.269717 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:34:09.269599 2585 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1faeed93-e4d2-434b-ab30-6bc7b9881a0b" containerName="console" Sep 10 13:34:09.269717 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:34:09.269610 2585 state_mem.go:107] "Deleted CPUSet assignment" podUID="1faeed93-e4d2-434b-ab30-6bc7b9881a0b" containerName="console" Sep 10 13:34:09.269717 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:34:09.269662 2585 memory_manager.go:368] "RemoveStaleState removing state" podUID="1faeed93-e4d2-434b-ab30-6bc7b9881a0b" containerName="console" Sep 10 13:34:09.302434 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:34:09.302404 2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Sep 10 13:34:09.302592 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:34:09.302544 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0" Sep 10 13:34:09.305527 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:34:09.305503 2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls\"" Sep 10 13:34:09.305668 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:34:09.305506 2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy\"" Sep 10 13:34:09.305668 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:34:09.305541 2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-web\"" Sep 10 13:34:09.305820 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:34:09.305794 2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-metric\"" Sep 10 13:34:09.305868 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:34:09.305842 2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls-assets-0\"" Sep 10 13:34:09.305868 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:34:09.305852 2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-generated\"" Sep 10 13:34:09.305948 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:34:09.305927 2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-web-config\"" Sep 10 13:34:09.306217 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:34:09.306196 2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-dockercfg-glqq7\"" Sep 10 13:34:09.306217 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:34:09.306197 2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-cluster-tls-config\"" Sep 10 13:34:09.311859 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:34:09.311841 2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"alertmanager-trusted-ca-bundle\"" Sep 10 13:34:09.366225 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:34:09.366186 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/41d71fbc-b878-481d-85ae-78beaea2515d-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"41d71fbc-b878-481d-85ae-78beaea2515d\") " pod="openshift-monitoring/alertmanager-main-0" Sep 10 13:34:09.366225 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:34:09.366229 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/41d71fbc-b878-481d-85ae-78beaea2515d-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"41d71fbc-b878-481d-85ae-78beaea2515d\") " pod="openshift-monitoring/alertmanager-main-0" Sep 10 13:34:09.366475 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:34:09.366350 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/41d71fbc-b878-481d-85ae-78beaea2515d-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"41d71fbc-b878-481d-85ae-78beaea2515d\") " pod="openshift-monitoring/alertmanager-main-0" Sep 10 13:34:09.366475 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:34:09.366387 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/41d71fbc-b878-481d-85ae-78beaea2515d-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"41d71fbc-b878-481d-85ae-78beaea2515d\") " pod="openshift-monitoring/alertmanager-main-0" Sep 10 13:34:09.366475 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:34:09.366421 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/41d71fbc-b878-481d-85ae-78beaea2515d-config-volume\") pod \"alertmanager-main-0\" (UID: \"41d71fbc-b878-481d-85ae-78beaea2515d\") " pod="openshift-monitoring/alertmanager-main-0" Sep 10 13:34:09.366654 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:34:09.366488 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/41d71fbc-b878-481d-85ae-78beaea2515d-web-config\") pod \"alertmanager-main-0\" (UID: \"41d71fbc-b878-481d-85ae-78beaea2515d\") " pod="openshift-monitoring/alertmanager-main-0" Sep 10 13:34:09.366654 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:34:09.366530 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/41d71fbc-b878-481d-85ae-78beaea2515d-tls-assets\") pod \"alertmanager-main-0\" (UID: \"41d71fbc-b878-481d-85ae-78beaea2515d\") " pod="openshift-monitoring/alertmanager-main-0" Sep 10 13:34:09.366654 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:34:09.366555 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/41d71fbc-b878-481d-85ae-78beaea2515d-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"41d71fbc-b878-481d-85ae-78beaea2515d\") " pod="openshift-monitoring/alertmanager-main-0" Sep 10 13:34:09.366654 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:34:09.366586 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/41d71fbc-b878-481d-85ae-78beaea2515d-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"41d71fbc-b878-481d-85ae-78beaea2515d\") " pod="openshift-monitoring/alertmanager-main-0" Sep 10 13:34:09.366654 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:34:09.366617 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/41d71fbc-b878-481d-85ae-78beaea2515d-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"41d71fbc-b878-481d-85ae-78beaea2515d\") " pod="openshift-monitoring/alertmanager-main-0" Sep 10 13:34:09.366843 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:34:09.366683 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-g9sg9\" (UniqueName: \"kubernetes.io/projected/41d71fbc-b878-481d-85ae-78beaea2515d-kube-api-access-g9sg9\") pod \"alertmanager-main-0\" (UID: \"41d71fbc-b878-481d-85ae-78beaea2515d\") " pod="openshift-monitoring/alertmanager-main-0" Sep 10 13:34:09.366843 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:34:09.366729 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/41d71fbc-b878-481d-85ae-78beaea2515d-config-out\") pod \"alertmanager-main-0\" (UID: \"41d71fbc-b878-481d-85ae-78beaea2515d\") " pod="openshift-monitoring/alertmanager-main-0" Sep 10 13:34:09.366843 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:34:09.366743 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/41d71fbc-b878-481d-85ae-78beaea2515d-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"41d71fbc-b878-481d-85ae-78beaea2515d\") " pod="openshift-monitoring/alertmanager-main-0" Sep 10 13:34:09.467173 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:34:09.467135 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/41d71fbc-b878-481d-85ae-78beaea2515d-web-config\") pod \"alertmanager-main-0\" (UID: \"41d71fbc-b878-481d-85ae-78beaea2515d\") " pod="openshift-monitoring/alertmanager-main-0" Sep 10 13:34:09.467173 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:34:09.467172 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/41d71fbc-b878-481d-85ae-78beaea2515d-tls-assets\") pod \"alertmanager-main-0\" (UID: \"41d71fbc-b878-481d-85ae-78beaea2515d\") " pod="openshift-monitoring/alertmanager-main-0" Sep 10 13:34:09.467418 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:34:09.467188 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/41d71fbc-b878-481d-85ae-78beaea2515d-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"41d71fbc-b878-481d-85ae-78beaea2515d\") " pod="openshift-monitoring/alertmanager-main-0" Sep 10 13:34:09.467418 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:34:09.467232 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/41d71fbc-b878-481d-85ae-78beaea2515d-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"41d71fbc-b878-481d-85ae-78beaea2515d\") " pod="openshift-monitoring/alertmanager-main-0" Sep 10 13:34:09.467418 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:34:09.467286 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/41d71fbc-b878-481d-85ae-78beaea2515d-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"41d71fbc-b878-481d-85ae-78beaea2515d\") " pod="openshift-monitoring/alertmanager-main-0" Sep 10 13:34:09.467418 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:34:09.467318 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g9sg9\" (UniqueName: \"kubernetes.io/projected/41d71fbc-b878-481d-85ae-78beaea2515d-kube-api-access-g9sg9\") pod \"alertmanager-main-0\" (UID: \"41d71fbc-b878-481d-85ae-78beaea2515d\") " pod="openshift-monitoring/alertmanager-main-0" Sep 10 13:34:09.467418 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:34:09.467348 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/41d71fbc-b878-481d-85ae-78beaea2515d-config-out\") pod \"alertmanager-main-0\" (UID: \"41d71fbc-b878-481d-85ae-78beaea2515d\") " pod="openshift-monitoring/alertmanager-main-0" Sep 10 13:34:09.467418 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:34:09.467364 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/41d71fbc-b878-481d-85ae-78beaea2515d-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"41d71fbc-b878-481d-85ae-78beaea2515d\") " pod="openshift-monitoring/alertmanager-main-0" Sep 10 13:34:09.467721 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:34:09.467700 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/41d71fbc-b878-481d-85ae-78beaea2515d-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"41d71fbc-b878-481d-85ae-78beaea2515d\") " pod="openshift-monitoring/alertmanager-main-0" Sep 10 13:34:09.467782 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:34:09.467735 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/41d71fbc-b878-481d-85ae-78beaea2515d-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"41d71fbc-b878-481d-85ae-78beaea2515d\") " pod="openshift-monitoring/alertmanager-main-0" Sep 10 13:34:09.467782 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:34:09.467736 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/41d71fbc-b878-481d-85ae-78beaea2515d-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"41d71fbc-b878-481d-85ae-78beaea2515d\") " pod="openshift-monitoring/alertmanager-main-0" Sep 10 13:34:09.467895 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:34:09.467824 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/41d71fbc-b878-481d-85ae-78beaea2515d-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"41d71fbc-b878-481d-85ae-78beaea2515d\") " pod="openshift-monitoring/alertmanager-main-0" Sep 10 13:34:09.467895 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:34:09.467854 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/41d71fbc-b878-481d-85ae-78beaea2515d-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"41d71fbc-b878-481d-85ae-78beaea2515d\") " pod="openshift-monitoring/alertmanager-main-0" Sep 10 13:34:09.467895 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:34:09.467893 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/41d71fbc-b878-481d-85ae-78beaea2515d-config-volume\") pod \"alertmanager-main-0\" (UID: \"41d71fbc-b878-481d-85ae-78beaea2515d\") " pod="openshift-monitoring/alertmanager-main-0" Sep 10 13:34:09.468339 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:34:09.468316 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/41d71fbc-b878-481d-85ae-78beaea2515d-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"41d71fbc-b878-481d-85ae-78beaea2515d\") " pod="openshift-monitoring/alertmanager-main-0" Sep 10 13:34:09.469713 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:34:09.469632 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/41d71fbc-b878-481d-85ae-78beaea2515d-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"41d71fbc-b878-481d-85ae-78beaea2515d\") " pod="openshift-monitoring/alertmanager-main-0" Sep 10 13:34:09.470319 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:34:09.470278 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/41d71fbc-b878-481d-85ae-78beaea2515d-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"41d71fbc-b878-481d-85ae-78beaea2515d\") " pod="openshift-monitoring/alertmanager-main-0" Sep 10 13:34:09.470453 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:34:09.470430 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/41d71fbc-b878-481d-85ae-78beaea2515d-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"41d71fbc-b878-481d-85ae-78beaea2515d\") " pod="openshift-monitoring/alertmanager-main-0" Sep 10 13:34:09.470887 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:34:09.470840 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/41d71fbc-b878-481d-85ae-78beaea2515d-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"41d71fbc-b878-481d-85ae-78beaea2515d\") " pod="openshift-monitoring/alertmanager-main-0" Sep 10 13:34:09.471042 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:34:09.471018 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/41d71fbc-b878-481d-85ae-78beaea2515d-web-config\") pod \"alertmanager-main-0\" (UID: \"41d71fbc-b878-481d-85ae-78beaea2515d\") " pod="openshift-monitoring/alertmanager-main-0" Sep 10 13:34:09.471146 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:34:09.471044 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/41d71fbc-b878-481d-85ae-78beaea2515d-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"41d71fbc-b878-481d-85ae-78beaea2515d\") " pod="openshift-monitoring/alertmanager-main-0" Sep 10 13:34:09.471407 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:34:09.471387 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/41d71fbc-b878-481d-85ae-78beaea2515d-config-out\") pod \"alertmanager-main-0\" (UID: \"41d71fbc-b878-481d-85ae-78beaea2515d\") " pod="openshift-monitoring/alertmanager-main-0" Sep 10 13:34:09.471467 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:34:09.471397 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/41d71fbc-b878-481d-85ae-78beaea2515d-config-volume\") pod \"alertmanager-main-0\" (UID: \"41d71fbc-b878-481d-85ae-78beaea2515d\") " pod="openshift-monitoring/alertmanager-main-0" Sep 10 13:34:09.471614 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:34:09.471596 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/41d71fbc-b878-481d-85ae-78beaea2515d-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"41d71fbc-b878-481d-85ae-78beaea2515d\") " pod="openshift-monitoring/alertmanager-main-0" Sep 10 13:34:09.471920 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:34:09.471903 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/41d71fbc-b878-481d-85ae-78beaea2515d-tls-assets\") pod \"alertmanager-main-0\" (UID: \"41d71fbc-b878-481d-85ae-78beaea2515d\") " pod="openshift-monitoring/alertmanager-main-0" Sep 10 13:34:09.476978 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:34:09.476951 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-g9sg9\" (UniqueName: \"kubernetes.io/projected/41d71fbc-b878-481d-85ae-78beaea2515d-kube-api-access-g9sg9\") pod \"alertmanager-main-0\" (UID: \"41d71fbc-b878-481d-85ae-78beaea2515d\") " pod="openshift-monitoring/alertmanager-main-0" Sep 10 13:34:09.612826 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:34:09.612796 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0" Sep 10 13:34:09.931339 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:34:09.931313 2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Sep 10 13:34:09.932505 ip-10-0-140-8 kubenswrapper[2585]: W0910 13:34:09.932473 2585 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod41d71fbc_b878_481d_85ae_78beaea2515d.slice/crio-bf0046f80eaa01f545b2246d5c8d7301ee44618919d76606e159841ad16e2248 WatchSource:0}: Error finding container bf0046f80eaa01f545b2246d5c8d7301ee44618919d76606e159841ad16e2248: Status 404 returned error can't find the container with id bf0046f80eaa01f545b2246d5c8d7301ee44618919d76606e159841ad16e2248 Sep 10 13:34:09.934499 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:34:09.934482 2585 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Sep 10 13:34:10.769026 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:34:10.768980 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"41d71fbc-b878-481d-85ae-78beaea2515d","Type":"ContainerStarted","Data":"bf0046f80eaa01f545b2246d5c8d7301ee44618919d76606e159841ad16e2248"} Sep 10 13:34:11.234055 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:34:11.234016 2585 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/telemeter-client-5d8cdf878c-fkvrj"] Sep 10 13:34:11.248630 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:34:11.248589 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/telemeter-client-5d8cdf878c-fkvrj" Sep 10 13:34:11.251160 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:34:11.250988 2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/telemeter-client-5d8cdf878c-fkvrj"] Sep 10 13:34:11.251955 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:34:11.251608 2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client-kube-rbac-proxy-config\"" Sep 10 13:34:11.251955 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:34:11.251753 2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"telemeter-client-serving-certs-ca-bundle\"" Sep 10 13:34:11.251955 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:34:11.251842 2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client-dockercfg-wq9nv\"" Sep 10 13:34:11.252176 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:34:11.252015 2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client\"" Sep 10 13:34:11.252867 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:34:11.252661 2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"federate-client-certs\"" Sep 10 13:34:11.252867 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:34:11.252858 2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client-tls\"" Sep 10 13:34:11.256368 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:34:11.256346 2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"telemeter-trusted-ca-bundle-8i12ta5c71j38\"" Sep 10 13:34:11.385320 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:34:11.385281 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-telemeter-client\" (UniqueName: \"kubernetes.io/secret/e1faf521-1963-4ae2-a92e-49f9a7bc15d2-secret-telemeter-client\") pod \"telemeter-client-5d8cdf878c-fkvrj\" (UID: \"e1faf521-1963-4ae2-a92e-49f9a7bc15d2\") " pod="openshift-monitoring/telemeter-client-5d8cdf878c-fkvrj" Sep 10 13:34:11.385486 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:34:11.385325 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"federate-client-tls\" (UniqueName: \"kubernetes.io/secret/e1faf521-1963-4ae2-a92e-49f9a7bc15d2-federate-client-tls\") pod \"telemeter-client-5d8cdf878c-fkvrj\" (UID: \"e1faf521-1963-4ae2-a92e-49f9a7bc15d2\") " pod="openshift-monitoring/telemeter-client-5d8cdf878c-fkvrj" Sep 10 13:34:11.385486 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:34:11.385471 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"telemeter-client-tls\" (UniqueName: \"kubernetes.io/secret/e1faf521-1963-4ae2-a92e-49f9a7bc15d2-telemeter-client-tls\") pod \"telemeter-client-5d8cdf878c-fkvrj\" (UID: \"e1faf521-1963-4ae2-a92e-49f9a7bc15d2\") " pod="openshift-monitoring/telemeter-client-5d8cdf878c-fkvrj" Sep 10 13:34:11.385573 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:34:11.385511 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-telemeter-client-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/e1faf521-1963-4ae2-a92e-49f9a7bc15d2-secret-telemeter-client-kube-rbac-proxy-config\") pod \"telemeter-client-5d8cdf878c-fkvrj\" (UID: \"e1faf521-1963-4ae2-a92e-49f9a7bc15d2\") " pod="openshift-monitoring/telemeter-client-5d8cdf878c-fkvrj" Sep 10 13:34:11.385612 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:34:11.385587 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6bz68\" (UniqueName: \"kubernetes.io/projected/e1faf521-1963-4ae2-a92e-49f9a7bc15d2-kube-api-access-6bz68\") pod \"telemeter-client-5d8cdf878c-fkvrj\" (UID: \"e1faf521-1963-4ae2-a92e-49f9a7bc15d2\") " pod="openshift-monitoring/telemeter-client-5d8cdf878c-fkvrj" Sep 10 13:34:11.385659 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:34:11.385626 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"telemeter-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/e1faf521-1963-4ae2-a92e-49f9a7bc15d2-telemeter-trusted-ca-bundle\") pod \"telemeter-client-5d8cdf878c-fkvrj\" (UID: \"e1faf521-1963-4ae2-a92e-49f9a7bc15d2\") " pod="openshift-monitoring/telemeter-client-5d8cdf878c-fkvrj" Sep 10 13:34:11.385697 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:34:11.385662 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/e1faf521-1963-4ae2-a92e-49f9a7bc15d2-serving-certs-ca-bundle\") pod \"telemeter-client-5d8cdf878c-fkvrj\" (UID: \"e1faf521-1963-4ae2-a92e-49f9a7bc15d2\") " pod="openshift-monitoring/telemeter-client-5d8cdf878c-fkvrj" Sep 10 13:34:11.385697 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:34:11.385687 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/e1faf521-1963-4ae2-a92e-49f9a7bc15d2-metrics-client-ca\") pod \"telemeter-client-5d8cdf878c-fkvrj\" (UID: \"e1faf521-1963-4ae2-a92e-49f9a7bc15d2\") " pod="openshift-monitoring/telemeter-client-5d8cdf878c-fkvrj" Sep 10 13:34:11.486309 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:34:11.486205 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"telemeter-client-tls\" (UniqueName: \"kubernetes.io/secret/e1faf521-1963-4ae2-a92e-49f9a7bc15d2-telemeter-client-tls\") pod \"telemeter-client-5d8cdf878c-fkvrj\" (UID: \"e1faf521-1963-4ae2-a92e-49f9a7bc15d2\") " pod="openshift-monitoring/telemeter-client-5d8cdf878c-fkvrj" Sep 10 13:34:11.486309 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:34:11.486263 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-telemeter-client-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/e1faf521-1963-4ae2-a92e-49f9a7bc15d2-secret-telemeter-client-kube-rbac-proxy-config\") pod \"telemeter-client-5d8cdf878c-fkvrj\" (UID: \"e1faf521-1963-4ae2-a92e-49f9a7bc15d2\") " pod="openshift-monitoring/telemeter-client-5d8cdf878c-fkvrj" Sep 10 13:34:11.486491 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:34:11.486315 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6bz68\" (UniqueName: \"kubernetes.io/projected/e1faf521-1963-4ae2-a92e-49f9a7bc15d2-kube-api-access-6bz68\") pod \"telemeter-client-5d8cdf878c-fkvrj\" (UID: \"e1faf521-1963-4ae2-a92e-49f9a7bc15d2\") " pod="openshift-monitoring/telemeter-client-5d8cdf878c-fkvrj" Sep 10 13:34:11.486491 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:34:11.486342 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"telemeter-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/e1faf521-1963-4ae2-a92e-49f9a7bc15d2-telemeter-trusted-ca-bundle\") pod \"telemeter-client-5d8cdf878c-fkvrj\" (UID: \"e1faf521-1963-4ae2-a92e-49f9a7bc15d2\") " pod="openshift-monitoring/telemeter-client-5d8cdf878c-fkvrj" Sep 10 13:34:11.486592 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:34:11.486565 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/e1faf521-1963-4ae2-a92e-49f9a7bc15d2-serving-certs-ca-bundle\") pod \"telemeter-client-5d8cdf878c-fkvrj\" (UID: \"e1faf521-1963-4ae2-a92e-49f9a7bc15d2\") " pod="openshift-monitoring/telemeter-client-5d8cdf878c-fkvrj" Sep 10 13:34:11.486643 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:34:11.486610 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/e1faf521-1963-4ae2-a92e-49f9a7bc15d2-metrics-client-ca\") pod \"telemeter-client-5d8cdf878c-fkvrj\" (UID: \"e1faf521-1963-4ae2-a92e-49f9a7bc15d2\") " pod="openshift-monitoring/telemeter-client-5d8cdf878c-fkvrj" Sep 10 13:34:11.486698 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:34:11.486641 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-telemeter-client\" (UniqueName: \"kubernetes.io/secret/e1faf521-1963-4ae2-a92e-49f9a7bc15d2-secret-telemeter-client\") pod \"telemeter-client-5d8cdf878c-fkvrj\" (UID: \"e1faf521-1963-4ae2-a92e-49f9a7bc15d2\") " pod="openshift-monitoring/telemeter-client-5d8cdf878c-fkvrj" Sep 10 13:34:11.486698 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:34:11.486669 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"federate-client-tls\" (UniqueName: \"kubernetes.io/secret/e1faf521-1963-4ae2-a92e-49f9a7bc15d2-federate-client-tls\") pod \"telemeter-client-5d8cdf878c-fkvrj\" (UID: \"e1faf521-1963-4ae2-a92e-49f9a7bc15d2\") " pod="openshift-monitoring/telemeter-client-5d8cdf878c-fkvrj" Sep 10 13:34:11.487297 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:34:11.487229 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"telemeter-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/e1faf521-1963-4ae2-a92e-49f9a7bc15d2-telemeter-trusted-ca-bundle\") pod \"telemeter-client-5d8cdf878c-fkvrj\" (UID: \"e1faf521-1963-4ae2-a92e-49f9a7bc15d2\") " pod="openshift-monitoring/telemeter-client-5d8cdf878c-fkvrj" Sep 10 13:34:11.487462 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:34:11.487337 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/e1faf521-1963-4ae2-a92e-49f9a7bc15d2-serving-certs-ca-bundle\") pod \"telemeter-client-5d8cdf878c-fkvrj\" (UID: \"e1faf521-1963-4ae2-a92e-49f9a7bc15d2\") " pod="openshift-monitoring/telemeter-client-5d8cdf878c-fkvrj" Sep 10 13:34:11.487462 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:34:11.487437 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/e1faf521-1963-4ae2-a92e-49f9a7bc15d2-metrics-client-ca\") pod \"telemeter-client-5d8cdf878c-fkvrj\" (UID: \"e1faf521-1963-4ae2-a92e-49f9a7bc15d2\") " pod="openshift-monitoring/telemeter-client-5d8cdf878c-fkvrj" Sep 10 13:34:11.489089 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:34:11.489064 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"telemeter-client-tls\" (UniqueName: \"kubernetes.io/secret/e1faf521-1963-4ae2-a92e-49f9a7bc15d2-telemeter-client-tls\") pod \"telemeter-client-5d8cdf878c-fkvrj\" (UID: \"e1faf521-1963-4ae2-a92e-49f9a7bc15d2\") " pod="openshift-monitoring/telemeter-client-5d8cdf878c-fkvrj" Sep 10 13:34:11.489279 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:34:11.489262 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-telemeter-client-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/e1faf521-1963-4ae2-a92e-49f9a7bc15d2-secret-telemeter-client-kube-rbac-proxy-config\") pod \"telemeter-client-5d8cdf878c-fkvrj\" (UID: \"e1faf521-1963-4ae2-a92e-49f9a7bc15d2\") " pod="openshift-monitoring/telemeter-client-5d8cdf878c-fkvrj" Sep 10 13:34:11.489399 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:34:11.489374 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-telemeter-client\" (UniqueName: \"kubernetes.io/secret/e1faf521-1963-4ae2-a92e-49f9a7bc15d2-secret-telemeter-client\") pod \"telemeter-client-5d8cdf878c-fkvrj\" (UID: \"e1faf521-1963-4ae2-a92e-49f9a7bc15d2\") " pod="openshift-monitoring/telemeter-client-5d8cdf878c-fkvrj" Sep 10 13:34:11.489494 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:34:11.489477 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"federate-client-tls\" (UniqueName: \"kubernetes.io/secret/e1faf521-1963-4ae2-a92e-49f9a7bc15d2-federate-client-tls\") pod \"telemeter-client-5d8cdf878c-fkvrj\" (UID: \"e1faf521-1963-4ae2-a92e-49f9a7bc15d2\") " pod="openshift-monitoring/telemeter-client-5d8cdf878c-fkvrj" Sep 10 13:34:11.498135 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:34:11.498112 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6bz68\" (UniqueName: \"kubernetes.io/projected/e1faf521-1963-4ae2-a92e-49f9a7bc15d2-kube-api-access-6bz68\") pod \"telemeter-client-5d8cdf878c-fkvrj\" (UID: \"e1faf521-1963-4ae2-a92e-49f9a7bc15d2\") " pod="openshift-monitoring/telemeter-client-5d8cdf878c-fkvrj" Sep 10 13:34:11.561131 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:34:11.560891 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/telemeter-client-5d8cdf878c-fkvrj" Sep 10 13:34:11.776123 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:34:11.776033 2585 generic.go:356] "Generic (PLEG): container finished" podID="41d71fbc-b878-481d-85ae-78beaea2515d" containerID="0210e05751f8cdafaaadbac8e4af70d67ee85e65c7348f22200a2c2bd5c9591d" exitCode=0 Sep 10 13:34:11.776123 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:34:11.776095 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"41d71fbc-b878-481d-85ae-78beaea2515d","Type":"ContainerDied","Data":"0210e05751f8cdafaaadbac8e4af70d67ee85e65c7348f22200a2c2bd5c9591d"} Sep 10 13:34:11.872859 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:34:11.872717 2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/telemeter-client-5d8cdf878c-fkvrj"] Sep 10 13:34:11.875004 ip-10-0-140-8 kubenswrapper[2585]: W0910 13:34:11.874979 2585 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode1faf521_1963_4ae2_a92e_49f9a7bc15d2.slice/crio-0ebbe7fdd6e921594d711227a085c5e6b77e683a2d762e743f58ceade07c2f20 WatchSource:0}: Error finding container 0ebbe7fdd6e921594d711227a085c5e6b77e683a2d762e743f58ceade07c2f20: Status 404 returned error can't find the container with id 0ebbe7fdd6e921594d711227a085c5e6b77e683a2d762e743f58ceade07c2f20 Sep 10 13:34:12.782623 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:34:12.782576 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-5d8cdf878c-fkvrj" event={"ID":"e1faf521-1963-4ae2-a92e-49f9a7bc15d2","Type":"ContainerStarted","Data":"0ebbe7fdd6e921594d711227a085c5e6b77e683a2d762e743f58ceade07c2f20"} Sep 10 13:34:14.792963 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:34:14.792927 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-5d8cdf878c-fkvrj" event={"ID":"e1faf521-1963-4ae2-a92e-49f9a7bc15d2","Type":"ContainerStarted","Data":"35380b58104cad9f77447b9e6903c67b554fb1133c410b58d3e5398bbadd7d1b"} Sep 10 13:34:14.793384 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:34:14.793029 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-5d8cdf878c-fkvrj" event={"ID":"e1faf521-1963-4ae2-a92e-49f9a7bc15d2","Type":"ContainerStarted","Data":"2bad950ebe0ebaa7114aca01bdeee8819dacf7baf0d0d3c224896f12b26c7705"} Sep 10 13:34:14.793384 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:34:14.793061 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-5d8cdf878c-fkvrj" event={"ID":"e1faf521-1963-4ae2-a92e-49f9a7bc15d2","Type":"ContainerStarted","Data":"50a86b632bbe6632848ce6f786a231665b266a7547bc648bc8017ba4b83447ce"} Sep 10 13:34:14.796403 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:34:14.796374 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"41d71fbc-b878-481d-85ae-78beaea2515d","Type":"ContainerStarted","Data":"13066228466c085e7b2b777b8cf65a760068ba32a88dd98c5179f2f4e2018af5"} Sep 10 13:34:14.796528 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:34:14.796444 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"41d71fbc-b878-481d-85ae-78beaea2515d","Type":"ContainerStarted","Data":"d05ef82f9608c8d691a4e4504565f5382ff14274370643c4b8b5a553a65df73a"} Sep 10 13:34:14.796528 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:34:14.796489 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"41d71fbc-b878-481d-85ae-78beaea2515d","Type":"ContainerStarted","Data":"551699d6ce737ec468a50201361cf9a9f6d7f7ec3f3367736acff2261e02f8b7"} Sep 10 13:34:14.796528 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:34:14.796518 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"41d71fbc-b878-481d-85ae-78beaea2515d","Type":"ContainerStarted","Data":"68050ea1bd54acde6bb589b3c7854fc5e682fa7a6760270e3a568ece9234ee91"} Sep 10 13:34:14.796642 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:34:14.796542 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"41d71fbc-b878-481d-85ae-78beaea2515d","Type":"ContainerStarted","Data":"30a0fcff2863a806ddfac764a8f5df8d4c17660172ba05c043f840b44282d106"} Sep 10 13:34:14.796642 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:34:14.796578 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"41d71fbc-b878-481d-85ae-78beaea2515d","Type":"ContainerStarted","Data":"a8af7394f57bbf556fe67dd376daf494fe9f00224ae0cb26949e2072b278298c"} Sep 10 13:34:14.820292 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:34:14.820222 2585 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/telemeter-client-5d8cdf878c-fkvrj" podStartSLOduration=1.910452464 podStartE2EDuration="3.820208094s" totalImagesPullingTime="1.90975563s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 13:34:11 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 13:34:14.819113531 +0000 UTC m=+339.691817746" watchObservedRunningTime="2026-09-10 13:34:14.820208094 +0000 UTC m=+339.692912305" Sep 10 13:34:14.850944 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:34:14.850901 2585 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/alertmanager-main-0" podStartSLOduration=2.189177656 podStartE2EDuration="5.850885321s" totalImagesPullingTime="3.645933289s" totalInitContainerRuntime="15.774376ms" isStatefulPod=false podCreationTimestamp="2026-09-10 13:34:09 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 13:34:14.848542302 +0000 UTC m=+339.721246502" watchObservedRunningTime="2026-09-10 13:34:14.850885321 +0000 UTC m=+339.723589532" Sep 10 13:34:14.938909 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:34:14.938875 2585 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-77d4d5dcdc-cfx2b"] Sep 10 13:34:14.940943 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:34:14.940927 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-77d4d5dcdc-cfx2b" Sep 10 13:34:14.960328 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:34:14.960305 2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-77d4d5dcdc-cfx2b"] Sep 10 13:34:15.018661 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:34:15.018622 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c2247a60-77ff-4b26-ab61-be378f21c54d-trusted-ca-bundle\") pod \"console-77d4d5dcdc-cfx2b\" (UID: \"c2247a60-77ff-4b26-ab61-be378f21c54d\") " pod="openshift-console/console-77d4d5dcdc-cfx2b" Sep 10 13:34:15.018661 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:34:15.018661 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dq86s\" (UniqueName: \"kubernetes.io/projected/c2247a60-77ff-4b26-ab61-be378f21c54d-kube-api-access-dq86s\") pod \"console-77d4d5dcdc-cfx2b\" (UID: \"c2247a60-77ff-4b26-ab61-be378f21c54d\") " pod="openshift-console/console-77d4d5dcdc-cfx2b" Sep 10 13:34:15.018843 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:34:15.018683 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/c2247a60-77ff-4b26-ab61-be378f21c54d-console-serving-cert\") pod \"console-77d4d5dcdc-cfx2b\" (UID: \"c2247a60-77ff-4b26-ab61-be378f21c54d\") " pod="openshift-console/console-77d4d5dcdc-cfx2b" Sep 10 13:34:15.018843 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:34:15.018754 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/c2247a60-77ff-4b26-ab61-be378f21c54d-console-config\") pod \"console-77d4d5dcdc-cfx2b\" (UID: \"c2247a60-77ff-4b26-ab61-be378f21c54d\") " pod="openshift-console/console-77d4d5dcdc-cfx2b" Sep 10 13:34:15.018843 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:34:15.018797 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/c2247a60-77ff-4b26-ab61-be378f21c54d-console-oauth-config\") pod \"console-77d4d5dcdc-cfx2b\" (UID: \"c2247a60-77ff-4b26-ab61-be378f21c54d\") " pod="openshift-console/console-77d4d5dcdc-cfx2b" Sep 10 13:34:15.018942 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:34:15.018855 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/c2247a60-77ff-4b26-ab61-be378f21c54d-service-ca\") pod \"console-77d4d5dcdc-cfx2b\" (UID: \"c2247a60-77ff-4b26-ab61-be378f21c54d\") " pod="openshift-console/console-77d4d5dcdc-cfx2b" Sep 10 13:34:15.018942 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:34:15.018873 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/c2247a60-77ff-4b26-ab61-be378f21c54d-oauth-serving-cert\") pod \"console-77d4d5dcdc-cfx2b\" (UID: \"c2247a60-77ff-4b26-ab61-be378f21c54d\") " pod="openshift-console/console-77d4d5dcdc-cfx2b" Sep 10 13:34:15.022705 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:34:15.022682 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fxgmp_fa53c783-d40a-4024-9f7b-1d0d2b57f6dc/konnectivity-agent/0.log" limit=1024 Sep 10 13:34:15.119310 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:34:15.119274 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/c2247a60-77ff-4b26-ab61-be378f21c54d-service-ca\") pod \"console-77d4d5dcdc-cfx2b\" (UID: \"c2247a60-77ff-4b26-ab61-be378f21c54d\") " pod="openshift-console/console-77d4d5dcdc-cfx2b" Sep 10 13:34:15.119310 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:34:15.119310 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/c2247a60-77ff-4b26-ab61-be378f21c54d-oauth-serving-cert\") pod \"console-77d4d5dcdc-cfx2b\" (UID: \"c2247a60-77ff-4b26-ab61-be378f21c54d\") " pod="openshift-console/console-77d4d5dcdc-cfx2b" Sep 10 13:34:15.119526 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:34:15.119366 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c2247a60-77ff-4b26-ab61-be378f21c54d-trusted-ca-bundle\") pod \"console-77d4d5dcdc-cfx2b\" (UID: \"c2247a60-77ff-4b26-ab61-be378f21c54d\") " pod="openshift-console/console-77d4d5dcdc-cfx2b" Sep 10 13:34:15.119526 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:34:15.119396 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dq86s\" (UniqueName: \"kubernetes.io/projected/c2247a60-77ff-4b26-ab61-be378f21c54d-kube-api-access-dq86s\") pod \"console-77d4d5dcdc-cfx2b\" (UID: \"c2247a60-77ff-4b26-ab61-be378f21c54d\") " pod="openshift-console/console-77d4d5dcdc-cfx2b" Sep 10 13:34:15.119526 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:34:15.119429 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/c2247a60-77ff-4b26-ab61-be378f21c54d-console-serving-cert\") pod \"console-77d4d5dcdc-cfx2b\" (UID: \"c2247a60-77ff-4b26-ab61-be378f21c54d\") " pod="openshift-console/console-77d4d5dcdc-cfx2b" Sep 10 13:34:15.119526 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:34:15.119456 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/c2247a60-77ff-4b26-ab61-be378f21c54d-console-config\") pod \"console-77d4d5dcdc-cfx2b\" (UID: \"c2247a60-77ff-4b26-ab61-be378f21c54d\") " pod="openshift-console/console-77d4d5dcdc-cfx2b" Sep 10 13:34:15.119526 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:34:15.119490 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/c2247a60-77ff-4b26-ab61-be378f21c54d-console-oauth-config\") pod \"console-77d4d5dcdc-cfx2b\" (UID: \"c2247a60-77ff-4b26-ab61-be378f21c54d\") " pod="openshift-console/console-77d4d5dcdc-cfx2b" Sep 10 13:34:15.120082 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:34:15.120061 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/c2247a60-77ff-4b26-ab61-be378f21c54d-service-ca\") pod \"console-77d4d5dcdc-cfx2b\" (UID: \"c2247a60-77ff-4b26-ab61-be378f21c54d\") " pod="openshift-console/console-77d4d5dcdc-cfx2b" Sep 10 13:34:15.120186 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:34:15.120130 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/c2247a60-77ff-4b26-ab61-be378f21c54d-oauth-serving-cert\") pod \"console-77d4d5dcdc-cfx2b\" (UID: \"c2247a60-77ff-4b26-ab61-be378f21c54d\") " pod="openshift-console/console-77d4d5dcdc-cfx2b" Sep 10 13:34:15.120186 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:34:15.120148 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/c2247a60-77ff-4b26-ab61-be378f21c54d-console-config\") pod \"console-77d4d5dcdc-cfx2b\" (UID: \"c2247a60-77ff-4b26-ab61-be378f21c54d\") " pod="openshift-console/console-77d4d5dcdc-cfx2b" Sep 10 13:34:15.120332 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:34:15.120311 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c2247a60-77ff-4b26-ab61-be378f21c54d-trusted-ca-bundle\") pod \"console-77d4d5dcdc-cfx2b\" (UID: \"c2247a60-77ff-4b26-ab61-be378f21c54d\") " pod="openshift-console/console-77d4d5dcdc-cfx2b" Sep 10 13:34:15.121911 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:34:15.121889 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/c2247a60-77ff-4b26-ab61-be378f21c54d-console-oauth-config\") pod \"console-77d4d5dcdc-cfx2b\" (UID: \"c2247a60-77ff-4b26-ab61-be378f21c54d\") " pod="openshift-console/console-77d4d5dcdc-cfx2b" Sep 10 13:34:15.122062 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:34:15.122042 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/c2247a60-77ff-4b26-ab61-be378f21c54d-console-serving-cert\") pod \"console-77d4d5dcdc-cfx2b\" (UID: \"c2247a60-77ff-4b26-ab61-be378f21c54d\") " pod="openshift-console/console-77d4d5dcdc-cfx2b" Sep 10 13:34:15.125820 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:34:15.125795 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fxgmp_fa53c783-d40a-4024-9f7b-1d0d2b57f6dc/konnectivity-agent/0.log" limit=1024 Sep 10 13:34:15.129491 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:34:15.129469 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-dq86s\" (UniqueName: \"kubernetes.io/projected/c2247a60-77ff-4b26-ab61-be378f21c54d-kube-api-access-dq86s\") pod \"console-77d4d5dcdc-cfx2b\" (UID: \"c2247a60-77ff-4b26-ab61-be378f21c54d\") " pod="openshift-console/console-77d4d5dcdc-cfx2b" Sep 10 13:34:15.211733 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:34:15.211698 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fxgmp_fa53c783-d40a-4024-9f7b-1d0d2b57f6dc/konnectivity-agent/0.log" limit=1024 Sep 10 13:34:15.249815 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:34:15.249775 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-77d4d5dcdc-cfx2b" Sep 10 13:34:15.547589 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:34:15.547502 2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-77d4d5dcdc-cfx2b"] Sep 10 13:34:15.550781 ip-10-0-140-8 kubenswrapper[2585]: W0910 13:34:15.550745 2585 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc2247a60_77ff_4b26_ab61_be378f21c54d.slice/crio-5d25095106b543bf3f30fe222767bc5358ca2cf4bb1aa002e4721670e7b8c8e7 WatchSource:0}: Error finding container 5d25095106b543bf3f30fe222767bc5358ca2cf4bb1aa002e4721670e7b8c8e7: Status 404 returned error can't find the container with id 5d25095106b543bf3f30fe222767bc5358ca2cf4bb1aa002e4721670e7b8c8e7 Sep 10 13:34:15.779414 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:34:15.779381 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fxgmp_fa53c783-d40a-4024-9f7b-1d0d2b57f6dc/konnectivity-agent/0.log" limit=1024 Sep 10 13:34:15.802853 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:34:15.802762 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-77d4d5dcdc-cfx2b" event={"ID":"c2247a60-77ff-4b26-ab61-be378f21c54d","Type":"ContainerStarted","Data":"e3fcbb730d62547e32fa9d129371bec0b84de7ee405554aa100170654176dedf"} Sep 10 13:34:15.802853 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:34:15.802814 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-77d4d5dcdc-cfx2b" event={"ID":"c2247a60-77ff-4b26-ab61-be378f21c54d","Type":"ContainerStarted","Data":"5d25095106b543bf3f30fe222767bc5358ca2cf4bb1aa002e4721670e7b8c8e7"} Sep 10 13:34:15.832052 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:34:15.832004 2585 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-77d4d5dcdc-cfx2b" podStartSLOduration=1.8319894589999999 podStartE2EDuration="1.831989459s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 13:34:14 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 13:34:15.830133807 +0000 UTC m=+340.702838008" watchObservedRunningTime="2026-09-10 13:34:15.831989459 +0000 UTC m=+340.704693702" Sep 10 13:34:15.877162 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:34:15.877134 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fxgmp_fa53c783-d40a-4024-9f7b-1d0d2b57f6dc/konnectivity-agent/0.log" limit=1024 Sep 10 13:34:19.267358 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:34:19.267323 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fxgmp_fa53c783-d40a-4024-9f7b-1d0d2b57f6dc/konnectivity-agent/0.log" limit=1024 Sep 10 13:34:25.250536 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:34:25.250494 2585 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-77d4d5dcdc-cfx2b" Sep 10 13:34:25.251066 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:34:25.250573 2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-77d4d5dcdc-cfx2b" Sep 10 13:34:25.255350 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:34:25.255324 2585 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-77d4d5dcdc-cfx2b" Sep 10 13:34:25.844364 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:34:25.844335 2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-77d4d5dcdc-cfx2b" Sep 10 13:34:25.903569 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:34:25.903530 2585 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-778b57cc48-jwmrp"] Sep 10 13:34:26.065167 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:34:26.065135 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fxgmp_fa53c783-d40a-4024-9f7b-1d0d2b57f6dc/konnectivity-agent/0.log" limit=1024 Sep 10 13:34:26.165083 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:34:26.165004 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fxgmp_fa53c783-d40a-4024-9f7b-1d0d2b57f6dc/konnectivity-agent/0.log" limit=1024 Sep 10 13:34:26.233010 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:34:26.232979 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fxgmp_fa53c783-d40a-4024-9f7b-1d0d2b57f6dc/konnectivity-agent/0.log" limit=1024 Sep 10 13:34:50.923503 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:34:50.923432 2585 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-console/console-778b57cc48-jwmrp" podUID="d079b23f-a7c7-43c3-b1a9-c213a7edd0ea" containerName="console" containerID="cri-o://0f6f5c8889419889bd8039a9243efef7cf85a4b02f1e48c4406778c127de3e51" gracePeriod=15 Sep 10 13:34:51.168062 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:34:51.168034 2585 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-778b57cc48-jwmrp" Sep 10 13:34:51.222924 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:34:51.222833 2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/d079b23f-a7c7-43c3-b1a9-c213a7edd0ea-console-config\") pod \"d079b23f-a7c7-43c3-b1a9-c213a7edd0ea\" (UID: \"d079b23f-a7c7-43c3-b1a9-c213a7edd0ea\") " Sep 10 13:34:51.222924 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:34:51.222889 2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/d079b23f-a7c7-43c3-b1a9-c213a7edd0ea-console-oauth-config\") pod \"d079b23f-a7c7-43c3-b1a9-c213a7edd0ea\" (UID: \"d079b23f-a7c7-43c3-b1a9-c213a7edd0ea\") " Sep 10 13:34:51.222924 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:34:51.222916 2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-ht9xd\" (UniqueName: \"kubernetes.io/projected/d079b23f-a7c7-43c3-b1a9-c213a7edd0ea-kube-api-access-ht9xd\") pod \"d079b23f-a7c7-43c3-b1a9-c213a7edd0ea\" (UID: \"d079b23f-a7c7-43c3-b1a9-c213a7edd0ea\") " Sep 10 13:34:51.223215 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:34:51.222947 2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/d079b23f-a7c7-43c3-b1a9-c213a7edd0ea-console-serving-cert\") pod \"d079b23f-a7c7-43c3-b1a9-c213a7edd0ea\" (UID: \"d079b23f-a7c7-43c3-b1a9-c213a7edd0ea\") " Sep 10 13:34:51.223215 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:34:51.222978 2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/d079b23f-a7c7-43c3-b1a9-c213a7edd0ea-oauth-serving-cert\") pod \"d079b23f-a7c7-43c3-b1a9-c213a7edd0ea\" (UID: \"d079b23f-a7c7-43c3-b1a9-c213a7edd0ea\") " Sep 10 13:34:51.223484 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:34:51.223410 2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d079b23f-a7c7-43c3-b1a9-c213a7edd0ea-oauth-serving-cert" (OuterVolumeSpecName: "oauth-serving-cert") pod "d079b23f-a7c7-43c3-b1a9-c213a7edd0ea" (UID: "d079b23f-a7c7-43c3-b1a9-c213a7edd0ea"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 10 13:34:51.223624 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:34:51.223544 2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/d079b23f-a7c7-43c3-b1a9-c213a7edd0ea-service-ca\") pod \"d079b23f-a7c7-43c3-b1a9-c213a7edd0ea\" (UID: \"d079b23f-a7c7-43c3-b1a9-c213a7edd0ea\") " Sep 10 13:34:51.223624 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:34:51.223589 2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d079b23f-a7c7-43c3-b1a9-c213a7edd0ea-console-config" (OuterVolumeSpecName: "console-config") pod "d079b23f-a7c7-43c3-b1a9-c213a7edd0ea" (UID: "d079b23f-a7c7-43c3-b1a9-c213a7edd0ea"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 10 13:34:51.223723 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:34:51.223633 2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d079b23f-a7c7-43c3-b1a9-c213a7edd0ea-trusted-ca-bundle\") pod \"d079b23f-a7c7-43c3-b1a9-c213a7edd0ea\" (UID: \"d079b23f-a7c7-43c3-b1a9-c213a7edd0ea\") " Sep 10 13:34:51.224433 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:34:51.223900 2585 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/d079b23f-a7c7-43c3-b1a9-c213a7edd0ea-oauth-serving-cert\") on node \"ip-10-0-140-8.ec2.internal\" DevicePath \"\"" Sep 10 13:34:51.224433 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:34:51.223921 2585 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/d079b23f-a7c7-43c3-b1a9-c213a7edd0ea-console-config\") on node \"ip-10-0-140-8.ec2.internal\" DevicePath \"\"" Sep 10 13:34:51.224433 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:34:51.224213 2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d079b23f-a7c7-43c3-b1a9-c213a7edd0ea-service-ca" (OuterVolumeSpecName: "service-ca") pod "d079b23f-a7c7-43c3-b1a9-c213a7edd0ea" (UID: "d079b23f-a7c7-43c3-b1a9-c213a7edd0ea"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 10 13:34:51.224433 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:34:51.224402 2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d079b23f-a7c7-43c3-b1a9-c213a7edd0ea-trusted-ca-bundle" (OuterVolumeSpecName: "trusted-ca-bundle") pod "d079b23f-a7c7-43c3-b1a9-c213a7edd0ea" (UID: "d079b23f-a7c7-43c3-b1a9-c213a7edd0ea"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 10 13:34:51.225637 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:34:51.225602 2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d079b23f-a7c7-43c3-b1a9-c213a7edd0ea-console-oauth-config" (OuterVolumeSpecName: "console-oauth-config") pod "d079b23f-a7c7-43c3-b1a9-c213a7edd0ea" (UID: "d079b23f-a7c7-43c3-b1a9-c213a7edd0ea"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 10 13:34:51.225827 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:34:51.225799 2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d079b23f-a7c7-43c3-b1a9-c213a7edd0ea-console-serving-cert" (OuterVolumeSpecName: "console-serving-cert") pod "d079b23f-a7c7-43c3-b1a9-c213a7edd0ea" (UID: "d079b23f-a7c7-43c3-b1a9-c213a7edd0ea"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 10 13:34:51.225945 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:34:51.225910 2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d079b23f-a7c7-43c3-b1a9-c213a7edd0ea-kube-api-access-ht9xd" (OuterVolumeSpecName: "kube-api-access-ht9xd") pod "d079b23f-a7c7-43c3-b1a9-c213a7edd0ea" (UID: "d079b23f-a7c7-43c3-b1a9-c213a7edd0ea"). InnerVolumeSpecName "kube-api-access-ht9xd". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 10 13:34:51.325319 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:34:51.325279 2585 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/d079b23f-a7c7-43c3-b1a9-c213a7edd0ea-service-ca\") on node \"ip-10-0-140-8.ec2.internal\" DevicePath \"\"" Sep 10 13:34:51.325319 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:34:51.325311 2585 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d079b23f-a7c7-43c3-b1a9-c213a7edd0ea-trusted-ca-bundle\") on node \"ip-10-0-140-8.ec2.internal\" DevicePath \"\"" Sep 10 13:34:51.325319 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:34:51.325320 2585 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/d079b23f-a7c7-43c3-b1a9-c213a7edd0ea-console-oauth-config\") on node \"ip-10-0-140-8.ec2.internal\" DevicePath \"\"" Sep 10 13:34:51.325319 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:34:51.325330 2585 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ht9xd\" (UniqueName: \"kubernetes.io/projected/d079b23f-a7c7-43c3-b1a9-c213a7edd0ea-kube-api-access-ht9xd\") on node \"ip-10-0-140-8.ec2.internal\" DevicePath \"\"" Sep 10 13:34:51.325585 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:34:51.325339 2585 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/d079b23f-a7c7-43c3-b1a9-c213a7edd0ea-console-serving-cert\") on node \"ip-10-0-140-8.ec2.internal\" DevicePath \"\"" Sep 10 13:34:51.948108 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:34:51.948020 2585 generic.go:356] "Generic (PLEG): container finished" podID="d079b23f-a7c7-43c3-b1a9-c213a7edd0ea" containerID="0f6f5c8889419889bd8039a9243efef7cf85a4b02f1e48c4406778c127de3e51" exitCode=0 Sep 10 13:34:51.948108 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:34:51.948067 2585 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-778b57cc48-jwmrp" Sep 10 13:34:51.948617 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:34:51.948105 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-778b57cc48-jwmrp" event={"ID":"d079b23f-a7c7-43c3-b1a9-c213a7edd0ea","Type":"ContainerDied","Data":"0f6f5c8889419889bd8039a9243efef7cf85a4b02f1e48c4406778c127de3e51"} Sep 10 13:34:51.948617 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:34:51.948150 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-778b57cc48-jwmrp" event={"ID":"d079b23f-a7c7-43c3-b1a9-c213a7edd0ea","Type":"ContainerDied","Data":"3cff41dd5dd5620d929acfa2190f96f87fd74f5592b84ba5e848c64a621f392e"} Sep 10 13:34:51.948617 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:34:51.948170 2585 scope.go:117] "RemoveContainer" containerID="0f6f5c8889419889bd8039a9243efef7cf85a4b02f1e48c4406778c127de3e51" Sep 10 13:34:51.956370 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:34:51.956348 2585 scope.go:117] "RemoveContainer" containerID="0f6f5c8889419889bd8039a9243efef7cf85a4b02f1e48c4406778c127de3e51" Sep 10 13:34:51.956677 ip-10-0-140-8 kubenswrapper[2585]: E0910 13:34:51.956657 2585 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0f6f5c8889419889bd8039a9243efef7cf85a4b02f1e48c4406778c127de3e51\": container with ID starting with 0f6f5c8889419889bd8039a9243efef7cf85a4b02f1e48c4406778c127de3e51 not found: ID does not exist" containerID="0f6f5c8889419889bd8039a9243efef7cf85a4b02f1e48c4406778c127de3e51" Sep 10 13:34:51.956751 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:34:51.956686 2585 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0f6f5c8889419889bd8039a9243efef7cf85a4b02f1e48c4406778c127de3e51"} err="failed to get container status \"0f6f5c8889419889bd8039a9243efef7cf85a4b02f1e48c4406778c127de3e51\": rpc error: code = NotFound desc = could not find container \"0f6f5c8889419889bd8039a9243efef7cf85a4b02f1e48c4406778c127de3e51\": container with ID starting with 0f6f5c8889419889bd8039a9243efef7cf85a4b02f1e48c4406778c127de3e51 not found: ID does not exist" Sep 10 13:34:51.976130 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:34:51.976099 2585 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-778b57cc48-jwmrp"] Sep 10 13:34:51.981233 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:34:51.981208 2585 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-778b57cc48-jwmrp"] Sep 10 13:34:53.646765 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:34:53.646730 2585 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="d079b23f-a7c7-43c3-b1a9-c213a7edd0ea" path="/var/lib/kubelet/pods/d079b23f-a7c7-43c3-b1a9-c213a7edd0ea/volumes" Sep 10 13:35:10.411102 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:35:10.411068 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fxgmp_fa53c783-d40a-4024-9f7b-1d0d2b57f6dc/konnectivity-agent/0.log" limit=1024 Sep 10 13:35:10.517568 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:35:10.517532 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fxgmp_fa53c783-d40a-4024-9f7b-1d0d2b57f6dc/konnectivity-agent/0.log" limit=1024 Sep 10 13:35:21.400187 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:35:21.400150 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fxgmp_fa53c783-d40a-4024-9f7b-1d0d2b57f6dc/konnectivity-agent/0.log" limit=1024 Sep 10 13:35:21.589768 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:35:21.589734 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fxgmp_fa53c783-d40a-4024-9f7b-1d0d2b57f6dc/konnectivity-agent/0.log" limit=1024 Sep 10 13:35:23.574949 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:35:23.574911 2585 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-6b9cc86549-4t9vq"] Sep 10 13:35:23.575424 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:35:23.575390 2585 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d079b23f-a7c7-43c3-b1a9-c213a7edd0ea" containerName="console" Sep 10 13:35:23.575424 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:35:23.575409 2585 state_mem.go:107] "Deleted CPUSet assignment" podUID="d079b23f-a7c7-43c3-b1a9-c213a7edd0ea" containerName="console" Sep 10 13:35:23.575544 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:35:23.575479 2585 memory_manager.go:368] "RemoveStaleState removing state" podUID="d079b23f-a7c7-43c3-b1a9-c213a7edd0ea" containerName="console" Sep 10 13:35:23.578795 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:35:23.578775 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-6b9cc86549-4t9vq" Sep 10 13:35:23.596661 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:35:23.596640 2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-6b9cc86549-4t9vq"] Sep 10 13:35:23.649972 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:35:23.649937 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fxgmp_fa53c783-d40a-4024-9f7b-1d0d2b57f6dc/konnectivity-agent/0.log" limit=1024 Sep 10 13:35:23.700762 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:35:23.700734 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/75f58905-a4db-4513-a597-3148034fcfd4-console-serving-cert\") pod \"console-6b9cc86549-4t9vq\" (UID: \"75f58905-a4db-4513-a597-3148034fcfd4\") " pod="openshift-console/console-6b9cc86549-4t9vq" Sep 10 13:35:23.700922 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:35:23.700786 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nm4hw\" (UniqueName: \"kubernetes.io/projected/75f58905-a4db-4513-a597-3148034fcfd4-kube-api-access-nm4hw\") pod \"console-6b9cc86549-4t9vq\" (UID: \"75f58905-a4db-4513-a597-3148034fcfd4\") " pod="openshift-console/console-6b9cc86549-4t9vq" Sep 10 13:35:23.700922 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:35:23.700846 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/75f58905-a4db-4513-a597-3148034fcfd4-console-config\") pod \"console-6b9cc86549-4t9vq\" (UID: \"75f58905-a4db-4513-a597-3148034fcfd4\") " pod="openshift-console/console-6b9cc86549-4t9vq" Sep 10 13:35:23.700922 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:35:23.700895 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/75f58905-a4db-4513-a597-3148034fcfd4-console-oauth-config\") pod \"console-6b9cc86549-4t9vq\" (UID: \"75f58905-a4db-4513-a597-3148034fcfd4\") " pod="openshift-console/console-6b9cc86549-4t9vq" Sep 10 13:35:23.701054 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:35:23.700922 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/75f58905-a4db-4513-a597-3148034fcfd4-oauth-serving-cert\") pod \"console-6b9cc86549-4t9vq\" (UID: \"75f58905-a4db-4513-a597-3148034fcfd4\") " pod="openshift-console/console-6b9cc86549-4t9vq" Sep 10 13:35:23.701054 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:35:23.700977 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/75f58905-a4db-4513-a597-3148034fcfd4-service-ca\") pod \"console-6b9cc86549-4t9vq\" (UID: \"75f58905-a4db-4513-a597-3148034fcfd4\") " pod="openshift-console/console-6b9cc86549-4t9vq" Sep 10 13:35:23.701054 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:35:23.700999 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/75f58905-a4db-4513-a597-3148034fcfd4-trusted-ca-bundle\") pod \"console-6b9cc86549-4t9vq\" (UID: \"75f58905-a4db-4513-a597-3148034fcfd4\") " pod="openshift-console/console-6b9cc86549-4t9vq" Sep 10 13:35:23.721422 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:35:23.721393 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fxgmp_fa53c783-d40a-4024-9f7b-1d0d2b57f6dc/konnectivity-agent/0.log" limit=1024 Sep 10 13:35:23.793662 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:35:23.793636 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fxgmp_fa53c783-d40a-4024-9f7b-1d0d2b57f6dc/konnectivity-agent/0.log" limit=1024 Sep 10 13:35:23.801791 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:35:23.801758 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/75f58905-a4db-4513-a597-3148034fcfd4-console-oauth-config\") pod \"console-6b9cc86549-4t9vq\" (UID: \"75f58905-a4db-4513-a597-3148034fcfd4\") " pod="openshift-console/console-6b9cc86549-4t9vq" Sep 10 13:35:23.801877 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:35:23.801805 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/75f58905-a4db-4513-a597-3148034fcfd4-oauth-serving-cert\") pod \"console-6b9cc86549-4t9vq\" (UID: \"75f58905-a4db-4513-a597-3148034fcfd4\") " pod="openshift-console/console-6b9cc86549-4t9vq" Sep 10 13:35:23.801877 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:35:23.801837 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/75f58905-a4db-4513-a597-3148034fcfd4-service-ca\") pod \"console-6b9cc86549-4t9vq\" (UID: \"75f58905-a4db-4513-a597-3148034fcfd4\") " pod="openshift-console/console-6b9cc86549-4t9vq" Sep 10 13:35:23.801877 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:35:23.801852 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/75f58905-a4db-4513-a597-3148034fcfd4-trusted-ca-bundle\") pod \"console-6b9cc86549-4t9vq\" (UID: \"75f58905-a4db-4513-a597-3148034fcfd4\") " pod="openshift-console/console-6b9cc86549-4t9vq" Sep 10 13:35:23.802032 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:35:23.801878 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/75f58905-a4db-4513-a597-3148034fcfd4-console-serving-cert\") pod \"console-6b9cc86549-4t9vq\" (UID: \"75f58905-a4db-4513-a597-3148034fcfd4\") " pod="openshift-console/console-6b9cc86549-4t9vq" Sep 10 13:35:23.802032 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:35:23.801962 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nm4hw\" (UniqueName: \"kubernetes.io/projected/75f58905-a4db-4513-a597-3148034fcfd4-kube-api-access-nm4hw\") pod \"console-6b9cc86549-4t9vq\" (UID: \"75f58905-a4db-4513-a597-3148034fcfd4\") " pod="openshift-console/console-6b9cc86549-4t9vq" Sep 10 13:35:23.802032 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:35:23.802006 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/75f58905-a4db-4513-a597-3148034fcfd4-console-config\") pod \"console-6b9cc86549-4t9vq\" (UID: \"75f58905-a4db-4513-a597-3148034fcfd4\") " pod="openshift-console/console-6b9cc86549-4t9vq" Sep 10 13:35:23.802751 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:35:23.802726 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/75f58905-a4db-4513-a597-3148034fcfd4-service-ca\") pod \"console-6b9cc86549-4t9vq\" (UID: \"75f58905-a4db-4513-a597-3148034fcfd4\") " pod="openshift-console/console-6b9cc86549-4t9vq" Sep 10 13:35:23.802880 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:35:23.802773 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/75f58905-a4db-4513-a597-3148034fcfd4-console-config\") pod \"console-6b9cc86549-4t9vq\" (UID: \"75f58905-a4db-4513-a597-3148034fcfd4\") " pod="openshift-console/console-6b9cc86549-4t9vq" Sep 10 13:35:23.802880 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:35:23.802773 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/75f58905-a4db-4513-a597-3148034fcfd4-trusted-ca-bundle\") pod \"console-6b9cc86549-4t9vq\" (UID: \"75f58905-a4db-4513-a597-3148034fcfd4\") " pod="openshift-console/console-6b9cc86549-4t9vq" Sep 10 13:35:23.803100 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:35:23.803072 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/75f58905-a4db-4513-a597-3148034fcfd4-oauth-serving-cert\") pod \"console-6b9cc86549-4t9vq\" (UID: \"75f58905-a4db-4513-a597-3148034fcfd4\") " pod="openshift-console/console-6b9cc86549-4t9vq" Sep 10 13:35:23.804480 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:35:23.804451 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/75f58905-a4db-4513-a597-3148034fcfd4-console-oauth-config\") pod \"console-6b9cc86549-4t9vq\" (UID: \"75f58905-a4db-4513-a597-3148034fcfd4\") " pod="openshift-console/console-6b9cc86549-4t9vq" Sep 10 13:35:23.804480 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:35:23.804460 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/75f58905-a4db-4513-a597-3148034fcfd4-console-serving-cert\") pod \"console-6b9cc86549-4t9vq\" (UID: \"75f58905-a4db-4513-a597-3148034fcfd4\") " pod="openshift-console/console-6b9cc86549-4t9vq" Sep 10 13:35:23.812753 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:35:23.812733 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-nm4hw\" (UniqueName: \"kubernetes.io/projected/75f58905-a4db-4513-a597-3148034fcfd4-kube-api-access-nm4hw\") pod \"console-6b9cc86549-4t9vq\" (UID: \"75f58905-a4db-4513-a597-3148034fcfd4\") " pod="openshift-console/console-6b9cc86549-4t9vq" Sep 10 13:35:23.888012 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:35:23.887946 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-6b9cc86549-4t9vq" Sep 10 13:35:24.172652 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:35:24.171109 2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-6b9cc86549-4t9vq"] Sep 10 13:35:24.175131 ip-10-0-140-8 kubenswrapper[2585]: W0910 13:35:24.175088 2585 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod75f58905_a4db_4513_a597_3148034fcfd4.slice/crio-8d639dea40c4ce5a50324090737db4c62897b356e8552dec396eeddd0ab6b2ca WatchSource:0}: Error finding container 8d639dea40c4ce5a50324090737db4c62897b356e8552dec396eeddd0ab6b2ca: Status 404 returned error can't find the container with id 8d639dea40c4ce5a50324090737db4c62897b356e8552dec396eeddd0ab6b2ca Sep 10 13:35:24.372029 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:35:24.371992 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fxgmp_fa53c783-d40a-4024-9f7b-1d0d2b57f6dc/konnectivity-agent/0.log" limit=1024 Sep 10 13:35:24.452995 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:35:24.452923 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fxgmp_fa53c783-d40a-4024-9f7b-1d0d2b57f6dc/konnectivity-agent/0.log" limit=1024 Sep 10 13:35:25.078563 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:35:25.078527 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-6b9cc86549-4t9vq" event={"ID":"75f58905-a4db-4513-a597-3148034fcfd4","Type":"ContainerStarted","Data":"4e045292d59bf40c25a5f2d414d63b4ab7e5ed70c08f9df5f882fb1f0e2568d1"} Sep 10 13:35:25.078917 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:35:25.078584 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-6b9cc86549-4t9vq" event={"ID":"75f58905-a4db-4513-a597-3148034fcfd4","Type":"ContainerStarted","Data":"8d639dea40c4ce5a50324090737db4c62897b356e8552dec396eeddd0ab6b2ca"} Sep 10 13:35:25.102546 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:35:25.102505 2585 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-6b9cc86549-4t9vq" podStartSLOduration=2.102483633 podStartE2EDuration="2.102483633s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 13:35:23 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 13:35:25.101423336 +0000 UTC m=+409.974127540" watchObservedRunningTime="2026-09-10 13:35:25.102483633 +0000 UTC m=+409.975187834" Sep 10 13:35:33.384915 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:35:33.384879 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fxgmp_fa53c783-d40a-4024-9f7b-1d0d2b57f6dc/konnectivity-agent/0.log" limit=1024 Sep 10 13:35:33.447105 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:35:33.447074 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fxgmp_fa53c783-d40a-4024-9f7b-1d0d2b57f6dc/konnectivity-agent/0.log" limit=1024 Sep 10 13:35:33.889025 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:35:33.888983 2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-6b9cc86549-4t9vq" Sep 10 13:35:33.889196 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:35:33.889043 2585 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-6b9cc86549-4t9vq" Sep 10 13:35:33.893796 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:35:33.893775 2585 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-6b9cc86549-4t9vq" Sep 10 13:35:34.118849 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:35:34.118820 2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-6b9cc86549-4t9vq" Sep 10 13:35:34.194626 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:35:34.194534 2585 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-77d4d5dcdc-cfx2b"] Sep 10 13:35:34.420915 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:35:34.420880 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fxgmp_fa53c783-d40a-4024-9f7b-1d0d2b57f6dc/konnectivity-agent/0.log" limit=1024 Sep 10 13:35:34.514412 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:35:34.514336 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fxgmp_fa53c783-d40a-4024-9f7b-1d0d2b57f6dc/konnectivity-agent/0.log" limit=1024 Sep 10 13:35:34.579473 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:35:34.579445 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fxgmp_fa53c783-d40a-4024-9f7b-1d0d2b57f6dc/konnectivity-agent/0.log" limit=1024 Sep 10 13:35:49.013395 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:35:49.013358 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fxgmp_fa53c783-d40a-4024-9f7b-1d0d2b57f6dc/konnectivity-agent/0.log" limit=1024 Sep 10 13:35:49.113443 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:35:49.113415 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fxgmp_fa53c783-d40a-4024-9f7b-1d0d2b57f6dc/konnectivity-agent/0.log" limit=1024 Sep 10 13:35:49.199051 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:35:49.199007 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fxgmp_fa53c783-d40a-4024-9f7b-1d0d2b57f6dc/konnectivity-agent/0.log" limit=1024 Sep 10 13:35:49.311842 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:35:49.311813 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fxgmp_fa53c783-d40a-4024-9f7b-1d0d2b57f6dc/konnectivity-agent/0.log" limit=1024 Sep 10 13:35:53.716853 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:35:53.716819 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fxgmp_fa53c783-d40a-4024-9f7b-1d0d2b57f6dc/konnectivity-agent/0.log" limit=1024 Sep 10 13:35:59.214287 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:35:59.214176 2585 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-console/console-77d4d5dcdc-cfx2b" podUID="c2247a60-77ff-4b26-ab61-be378f21c54d" containerName="console" containerID="cri-o://e3fcbb730d62547e32fa9d129371bec0b84de7ee405554aa100170654176dedf" gracePeriod=15 Sep 10 13:35:59.460794 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:35:59.460768 2585 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-77d4d5dcdc-cfx2b" Sep 10 13:35:59.595081 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:35:59.595044 2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/c2247a60-77ff-4b26-ab61-be378f21c54d-console-oauth-config\") pod \"c2247a60-77ff-4b26-ab61-be378f21c54d\" (UID: \"c2247a60-77ff-4b26-ab61-be378f21c54d\") " Sep 10 13:35:59.595286 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:35:59.595100 2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/c2247a60-77ff-4b26-ab61-be378f21c54d-console-serving-cert\") pod \"c2247a60-77ff-4b26-ab61-be378f21c54d\" (UID: \"c2247a60-77ff-4b26-ab61-be378f21c54d\") " Sep 10 13:35:59.595286 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:35:59.595139 2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/c2247a60-77ff-4b26-ab61-be378f21c54d-oauth-serving-cert\") pod \"c2247a60-77ff-4b26-ab61-be378f21c54d\" (UID: \"c2247a60-77ff-4b26-ab61-be378f21c54d\") " Sep 10 13:35:59.595286 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:35:59.595192 2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/c2247a60-77ff-4b26-ab61-be378f21c54d-service-ca\") pod \"c2247a60-77ff-4b26-ab61-be378f21c54d\" (UID: \"c2247a60-77ff-4b26-ab61-be378f21c54d\") " Sep 10 13:35:59.595286 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:35:59.595232 2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c2247a60-77ff-4b26-ab61-be378f21c54d-trusted-ca-bundle\") pod \"c2247a60-77ff-4b26-ab61-be378f21c54d\" (UID: \"c2247a60-77ff-4b26-ab61-be378f21c54d\") " Sep 10 13:35:59.595286 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:35:59.595277 2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/c2247a60-77ff-4b26-ab61-be378f21c54d-console-config\") pod \"c2247a60-77ff-4b26-ab61-be378f21c54d\" (UID: \"c2247a60-77ff-4b26-ab61-be378f21c54d\") " Sep 10 13:35:59.595554 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:35:59.595311 2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-dq86s\" (UniqueName: \"kubernetes.io/projected/c2247a60-77ff-4b26-ab61-be378f21c54d-kube-api-access-dq86s\") pod \"c2247a60-77ff-4b26-ab61-be378f21c54d\" (UID: \"c2247a60-77ff-4b26-ab61-be378f21c54d\") " Sep 10 13:35:59.595658 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:35:59.595630 2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c2247a60-77ff-4b26-ab61-be378f21c54d-oauth-serving-cert" (OuterVolumeSpecName: "oauth-serving-cert") pod "c2247a60-77ff-4b26-ab61-be378f21c54d" (UID: "c2247a60-77ff-4b26-ab61-be378f21c54d"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 10 13:35:59.595726 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:35:59.595681 2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c2247a60-77ff-4b26-ab61-be378f21c54d-service-ca" (OuterVolumeSpecName: "service-ca") pod "c2247a60-77ff-4b26-ab61-be378f21c54d" (UID: "c2247a60-77ff-4b26-ab61-be378f21c54d"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 10 13:35:59.595726 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:35:59.595693 2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c2247a60-77ff-4b26-ab61-be378f21c54d-console-config" (OuterVolumeSpecName: "console-config") pod "c2247a60-77ff-4b26-ab61-be378f21c54d" (UID: "c2247a60-77ff-4b26-ab61-be378f21c54d"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 10 13:35:59.595726 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:35:59.595686 2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c2247a60-77ff-4b26-ab61-be378f21c54d-trusted-ca-bundle" (OuterVolumeSpecName: "trusted-ca-bundle") pod "c2247a60-77ff-4b26-ab61-be378f21c54d" (UID: "c2247a60-77ff-4b26-ab61-be378f21c54d"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 10 13:35:59.597210 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:35:59.597187 2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c2247a60-77ff-4b26-ab61-be378f21c54d-console-serving-cert" (OuterVolumeSpecName: "console-serving-cert") pod "c2247a60-77ff-4b26-ab61-be378f21c54d" (UID: "c2247a60-77ff-4b26-ab61-be378f21c54d"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 10 13:35:59.597322 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:35:59.597206 2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c2247a60-77ff-4b26-ab61-be378f21c54d-console-oauth-config" (OuterVolumeSpecName: "console-oauth-config") pod "c2247a60-77ff-4b26-ab61-be378f21c54d" (UID: "c2247a60-77ff-4b26-ab61-be378f21c54d"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 10 13:35:59.597383 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:35:59.597343 2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c2247a60-77ff-4b26-ab61-be378f21c54d-kube-api-access-dq86s" (OuterVolumeSpecName: "kube-api-access-dq86s") pod "c2247a60-77ff-4b26-ab61-be378f21c54d" (UID: "c2247a60-77ff-4b26-ab61-be378f21c54d"). InnerVolumeSpecName "kube-api-access-dq86s". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 10 13:35:59.696626 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:35:59.696599 2585 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c2247a60-77ff-4b26-ab61-be378f21c54d-trusted-ca-bundle\") on node \"ip-10-0-140-8.ec2.internal\" DevicePath \"\"" Sep 10 13:35:59.696626 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:35:59.696620 2585 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/c2247a60-77ff-4b26-ab61-be378f21c54d-console-config\") on node \"ip-10-0-140-8.ec2.internal\" DevicePath \"\"" Sep 10 13:35:59.696626 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:35:59.696629 2585 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dq86s\" (UniqueName: \"kubernetes.io/projected/c2247a60-77ff-4b26-ab61-be378f21c54d-kube-api-access-dq86s\") on node \"ip-10-0-140-8.ec2.internal\" DevicePath \"\"" Sep 10 13:35:59.696806 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:35:59.696639 2585 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/c2247a60-77ff-4b26-ab61-be378f21c54d-console-oauth-config\") on node \"ip-10-0-140-8.ec2.internal\" DevicePath \"\"" Sep 10 13:35:59.696806 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:35:59.696648 2585 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/c2247a60-77ff-4b26-ab61-be378f21c54d-console-serving-cert\") on node \"ip-10-0-140-8.ec2.internal\" DevicePath \"\"" Sep 10 13:35:59.696806 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:35:59.696657 2585 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/c2247a60-77ff-4b26-ab61-be378f21c54d-oauth-serving-cert\") on node \"ip-10-0-140-8.ec2.internal\" DevicePath \"\"" Sep 10 13:35:59.696806 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:35:59.696666 2585 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/c2247a60-77ff-4b26-ab61-be378f21c54d-service-ca\") on node \"ip-10-0-140-8.ec2.internal\" DevicePath \"\"" Sep 10 13:36:00.214118 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:36:00.214083 2585 generic.go:356] "Generic (PLEG): container finished" podID="c2247a60-77ff-4b26-ab61-be378f21c54d" containerID="e3fcbb730d62547e32fa9d129371bec0b84de7ee405554aa100170654176dedf" exitCode=0 Sep 10 13:36:00.214332 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:36:00.214139 2585 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-77d4d5dcdc-cfx2b" Sep 10 13:36:00.214332 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:36:00.214174 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-77d4d5dcdc-cfx2b" event={"ID":"c2247a60-77ff-4b26-ab61-be378f21c54d","Type":"ContainerDied","Data":"e3fcbb730d62547e32fa9d129371bec0b84de7ee405554aa100170654176dedf"} Sep 10 13:36:00.214332 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:36:00.214277 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-77d4d5dcdc-cfx2b" event={"ID":"c2247a60-77ff-4b26-ab61-be378f21c54d","Type":"ContainerDied","Data":"5d25095106b543bf3f30fe222767bc5358ca2cf4bb1aa002e4721670e7b8c8e7"} Sep 10 13:36:00.214332 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:36:00.214305 2585 scope.go:117] "RemoveContainer" containerID="e3fcbb730d62547e32fa9d129371bec0b84de7ee405554aa100170654176dedf" Sep 10 13:36:00.222327 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:36:00.222310 2585 scope.go:117] "RemoveContainer" containerID="e3fcbb730d62547e32fa9d129371bec0b84de7ee405554aa100170654176dedf" Sep 10 13:36:00.222581 ip-10-0-140-8 kubenswrapper[2585]: E0910 13:36:00.222563 2585 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e3fcbb730d62547e32fa9d129371bec0b84de7ee405554aa100170654176dedf\": container with ID starting with e3fcbb730d62547e32fa9d129371bec0b84de7ee405554aa100170654176dedf not found: ID does not exist" containerID="e3fcbb730d62547e32fa9d129371bec0b84de7ee405554aa100170654176dedf" Sep 10 13:36:00.222640 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:36:00.222589 2585 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e3fcbb730d62547e32fa9d129371bec0b84de7ee405554aa100170654176dedf"} err="failed to get container status \"e3fcbb730d62547e32fa9d129371bec0b84de7ee405554aa100170654176dedf\": rpc error: code = NotFound desc = could not find container \"e3fcbb730d62547e32fa9d129371bec0b84de7ee405554aa100170654176dedf\": container with ID starting with e3fcbb730d62547e32fa9d129371bec0b84de7ee405554aa100170654176dedf not found: ID does not exist" Sep 10 13:36:00.246593 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:36:00.246570 2585 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-77d4d5dcdc-cfx2b"] Sep 10 13:36:00.250402 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:36:00.250381 2585 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-77d4d5dcdc-cfx2b"] Sep 10 13:36:01.646297 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:36:01.646265 2585 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="c2247a60-77ff-4b26-ab61-be378f21c54d" path="/var/lib/kubelet/pods/c2247a60-77ff-4b26-ab61-be378f21c54d/volumes" Sep 10 13:36:37.784950 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:36:37.784912 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fxgmp_fa53c783-d40a-4024-9f7b-1d0d2b57f6dc/konnectivity-agent/0.log" limit=1024 Sep 10 13:36:40.581327 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:36:40.581292 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fxgmp_fa53c783-d40a-4024-9f7b-1d0d2b57f6dc/konnectivity-agent/0.log" limit=1024 Sep 10 13:37:04.679397 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:37:04.679364 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fxgmp_fa53c783-d40a-4024-9f7b-1d0d2b57f6dc/konnectivity-agent/0.log" limit=1024 Sep 10 13:37:05.522975 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:37:05.522940 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fxgmp_fa53c783-d40a-4024-9f7b-1d0d2b57f6dc/konnectivity-agent/0.log" limit=1024 Sep 10 13:37:08.513458 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:37:08.513429 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fxgmp_fa53c783-d40a-4024-9f7b-1d0d2b57f6dc/konnectivity-agent/0.log" limit=1024 Sep 10 13:37:08.621604 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:37:08.621570 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fxgmp_fa53c783-d40a-4024-9f7b-1d0d2b57f6dc/konnectivity-agent/0.log" limit=1024 Sep 10 13:37:09.653120 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:37:09.653059 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fxgmp_fa53c783-d40a-4024-9f7b-1d0d2b57f6dc/konnectivity-agent/0.log" limit=1024 Sep 10 13:37:09.782167 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:37:09.782117 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fxgmp_fa53c783-d40a-4024-9f7b-1d0d2b57f6dc/konnectivity-agent/0.log" limit=1024 Sep 10 13:37:10.539529 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:37:10.539497 2585 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f6kfp7"] Sep 10 13:37:10.539801 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:37:10.539789 2585 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c2247a60-77ff-4b26-ab61-be378f21c54d" containerName="console" Sep 10 13:37:10.539846 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:37:10.539802 2585 state_mem.go:107] "Deleted CPUSet assignment" podUID="c2247a60-77ff-4b26-ab61-be378f21c54d" containerName="console" Sep 10 13:37:10.539881 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:37:10.539861 2585 memory_manager.go:368] "RemoveStaleState removing state" podUID="c2247a60-77ff-4b26-ab61-be378f21c54d" containerName="console" Sep 10 13:37:10.542912 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:37:10.542897 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f6kfp7" Sep 10 13:37:10.545623 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:37:10.545601 2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"default-dockercfg-m87wf\"" Sep 10 13:37:10.545818 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:37:10.545802 2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"kube-root-ca.crt\"" Sep 10 13:37:10.546670 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:37:10.546650 2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"openshift-service-ca.crt\"" Sep 10 13:37:10.553765 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:37:10.553742 2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f6kfp7"] Sep 10 13:37:10.667439 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:37:10.667408 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-prvhb\" (UniqueName: \"kubernetes.io/projected/ef282945-09c0-4f65-9040-e1b7c849665e-kube-api-access-prvhb\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f6kfp7\" (UID: \"ef282945-09c0-4f65-9040-e1b7c849665e\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f6kfp7" Sep 10 13:37:10.667785 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:37:10.667455 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/ef282945-09c0-4f65-9040-e1b7c849665e-bundle\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f6kfp7\" (UID: \"ef282945-09c0-4f65-9040-e1b7c849665e\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f6kfp7" Sep 10 13:37:10.667785 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:37:10.667485 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/ef282945-09c0-4f65-9040-e1b7c849665e-util\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f6kfp7\" (UID: \"ef282945-09c0-4f65-9040-e1b7c849665e\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f6kfp7" Sep 10 13:37:10.768290 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:37:10.768231 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-prvhb\" (UniqueName: \"kubernetes.io/projected/ef282945-09c0-4f65-9040-e1b7c849665e-kube-api-access-prvhb\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f6kfp7\" (UID: \"ef282945-09c0-4f65-9040-e1b7c849665e\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f6kfp7" Sep 10 13:37:10.768461 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:37:10.768302 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/ef282945-09c0-4f65-9040-e1b7c849665e-bundle\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f6kfp7\" (UID: \"ef282945-09c0-4f65-9040-e1b7c849665e\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f6kfp7" Sep 10 13:37:10.768461 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:37:10.768323 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/ef282945-09c0-4f65-9040-e1b7c849665e-util\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f6kfp7\" (UID: \"ef282945-09c0-4f65-9040-e1b7c849665e\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f6kfp7" Sep 10 13:37:10.768661 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:37:10.768645 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/ef282945-09c0-4f65-9040-e1b7c849665e-util\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f6kfp7\" (UID: \"ef282945-09c0-4f65-9040-e1b7c849665e\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f6kfp7" Sep 10 13:37:10.768700 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:37:10.768680 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/ef282945-09c0-4f65-9040-e1b7c849665e-bundle\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f6kfp7\" (UID: \"ef282945-09c0-4f65-9040-e1b7c849665e\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f6kfp7" Sep 10 13:37:10.780467 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:37:10.780437 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-prvhb\" (UniqueName: \"kubernetes.io/projected/ef282945-09c0-4f65-9040-e1b7c849665e-kube-api-access-prvhb\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f6kfp7\" (UID: \"ef282945-09c0-4f65-9040-e1b7c849665e\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f6kfp7" Sep 10 13:37:10.851655 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:37:10.851634 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f6kfp7" Sep 10 13:37:11.141235 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:37:11.141210 2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f6kfp7"] Sep 10 13:37:11.143766 ip-10-0-140-8 kubenswrapper[2585]: W0910 13:37:11.143740 2585 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podef282945_09c0_4f65_9040_e1b7c849665e.slice/crio-cfd8f3d89fdefe889c44e888a047a3e1b3aa4af91d175b071ba63940c43666e5 WatchSource:0}: Error finding container cfd8f3d89fdefe889c44e888a047a3e1b3aa4af91d175b071ba63940c43666e5: Status 404 returned error can't find the container with id cfd8f3d89fdefe889c44e888a047a3e1b3aa4af91d175b071ba63940c43666e5 Sep 10 13:37:11.489571 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:37:11.489484 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f6kfp7" event={"ID":"ef282945-09c0-4f65-9040-e1b7c849665e","Type":"ContainerStarted","Data":"cfd8f3d89fdefe889c44e888a047a3e1b3aa4af91d175b071ba63940c43666e5"} Sep 10 13:37:12.070817 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:37:12.070765 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fxgmp_fa53c783-d40a-4024-9f7b-1d0d2b57f6dc/konnectivity-agent/0.log" limit=1024 Sep 10 13:37:12.494887 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:37:12.494780 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fxgmp_fa53c783-d40a-4024-9f7b-1d0d2b57f6dc/konnectivity-agent/0.log" limit=1024 Sep 10 13:37:13.426654 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:37:13.426616 2585 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["cert-manager/cert-manager-cainjector-559cdd8c87-zlffs"] Sep 10 13:37:13.430369 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:37:13.430343 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-cainjector-559cdd8c87-zlffs" Sep 10 13:37:13.433187 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:37:13.433163 2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager\"/\"kube-root-ca.crt\"" Sep 10 13:37:13.433328 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:37:13.433163 2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"cert-manager\"/\"cert-manager-cainjector-dockercfg-9dgth\"" Sep 10 13:37:13.433328 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:37:13.433171 2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager\"/\"openshift-service-ca.crt\"" Sep 10 13:37:13.440899 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:37:13.440876 2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-cainjector-559cdd8c87-zlffs"] Sep 10 13:37:13.596103 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:37:13.596063 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-blfrk\" (UniqueName: \"kubernetes.io/projected/3a7a55cb-f13a-418f-937d-6217007895c6-kube-api-access-blfrk\") pod \"cert-manager-cainjector-559cdd8c87-zlffs\" (UID: \"3a7a55cb-f13a-418f-937d-6217007895c6\") " pod="cert-manager/cert-manager-cainjector-559cdd8c87-zlffs" Sep 10 13:37:13.596312 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:37:13.596149 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/3a7a55cb-f13a-418f-937d-6217007895c6-bound-sa-token\") pod \"cert-manager-cainjector-559cdd8c87-zlffs\" (UID: \"3a7a55cb-f13a-418f-937d-6217007895c6\") " pod="cert-manager/cert-manager-cainjector-559cdd8c87-zlffs" Sep 10 13:37:13.697026 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:37:13.696946 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-blfrk\" (UniqueName: \"kubernetes.io/projected/3a7a55cb-f13a-418f-937d-6217007895c6-kube-api-access-blfrk\") pod \"cert-manager-cainjector-559cdd8c87-zlffs\" (UID: \"3a7a55cb-f13a-418f-937d-6217007895c6\") " pod="cert-manager/cert-manager-cainjector-559cdd8c87-zlffs" Sep 10 13:37:13.697026 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:37:13.696999 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/3a7a55cb-f13a-418f-937d-6217007895c6-bound-sa-token\") pod \"cert-manager-cainjector-559cdd8c87-zlffs\" (UID: \"3a7a55cb-f13a-418f-937d-6217007895c6\") " pod="cert-manager/cert-manager-cainjector-559cdd8c87-zlffs" Sep 10 13:37:13.706964 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:37:13.706933 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/3a7a55cb-f13a-418f-937d-6217007895c6-bound-sa-token\") pod \"cert-manager-cainjector-559cdd8c87-zlffs\" (UID: \"3a7a55cb-f13a-418f-937d-6217007895c6\") " pod="cert-manager/cert-manager-cainjector-559cdd8c87-zlffs" Sep 10 13:37:13.707160 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:37:13.707143 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-blfrk\" (UniqueName: \"kubernetes.io/projected/3a7a55cb-f13a-418f-937d-6217007895c6-kube-api-access-blfrk\") pod \"cert-manager-cainjector-559cdd8c87-zlffs\" (UID: \"3a7a55cb-f13a-418f-937d-6217007895c6\") " pod="cert-manager/cert-manager-cainjector-559cdd8c87-zlffs" Sep 10 13:37:13.742548 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:37:13.742525 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-cainjector-559cdd8c87-zlffs" Sep 10 13:37:14.026400 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:37:14.026340 2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-cainjector-559cdd8c87-zlffs"] Sep 10 13:37:14.029275 ip-10-0-140-8 kubenswrapper[2585]: W0910 13:37:14.029232 2585 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod3a7a55cb_f13a_418f_937d_6217007895c6.slice/crio-4030bf0db488590ea0417789b1203aafbc9a9f80a5abb5f7e2405e608eadbe08 WatchSource:0}: Error finding container 4030bf0db488590ea0417789b1203aafbc9a9f80a5abb5f7e2405e608eadbe08: Status 404 returned error can't find the container with id 4030bf0db488590ea0417789b1203aafbc9a9f80a5abb5f7e2405e608eadbe08 Sep 10 13:37:14.504648 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:37:14.504613 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-cainjector-559cdd8c87-zlffs" event={"ID":"3a7a55cb-f13a-418f-937d-6217007895c6","Type":"ContainerStarted","Data":"4030bf0db488590ea0417789b1203aafbc9a9f80a5abb5f7e2405e608eadbe08"} Sep 10 13:37:15.081104 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:37:15.080997 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fxgmp_fa53c783-d40a-4024-9f7b-1d0d2b57f6dc/konnectivity-agent/0.log" limit=1024 Sep 10 13:37:15.147559 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:37:15.147467 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fxgmp_fa53c783-d40a-4024-9f7b-1d0d2b57f6dc/konnectivity-agent/0.log" limit=1024 Sep 10 13:37:19.388895 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:37:19.388842 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fxgmp_fa53c783-d40a-4024-9f7b-1d0d2b57f6dc/konnectivity-agent/0.log" limit=1024 Sep 10 13:37:19.529556 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:37:19.529523 2585 generic.go:356] "Generic (PLEG): container finished" podID="ef282945-09c0-4f65-9040-e1b7c849665e" containerID="4c6111b8402a9486dec1456adc1c7a2490326b1fb466f48ff68b08422b2a7e4a" exitCode=0 Sep 10 13:37:19.529743 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:37:19.529602 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f6kfp7" event={"ID":"ef282945-09c0-4f65-9040-e1b7c849665e","Type":"ContainerDied","Data":"4c6111b8402a9486dec1456adc1c7a2490326b1fb466f48ff68b08422b2a7e4a"} Sep 10 13:37:19.531428 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:37:19.531401 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-cainjector-559cdd8c87-zlffs" event={"ID":"3a7a55cb-f13a-418f-937d-6217007895c6","Type":"ContainerStarted","Data":"d505fb57e5b510c63f650d67ad05fc96f50c31d84f795d77256c132b2b00f294"} Sep 10 13:37:19.626560 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:37:19.626505 2585 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="cert-manager/cert-manager-cainjector-559cdd8c87-zlffs" podStartSLOduration=1.593341841 podStartE2EDuration="6.626477139s" totalImagesPullingTime="5.033135298s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 13:37:13 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 13:37:19.624188054 +0000 UTC m=+524.496892256" watchObservedRunningTime="2026-09-10 13:37:19.626477139 +0000 UTC m=+524.499181340" Sep 10 13:37:19.997019 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:37:19.996990 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fxgmp_fa53c783-d40a-4024-9f7b-1d0d2b57f6dc/konnectivity-agent/0.log" limit=1024 Sep 10 13:37:20.085769 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:37:20.085737 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fxgmp_fa53c783-d40a-4024-9f7b-1d0d2b57f6dc/konnectivity-agent/0.log" limit=1024 Sep 10 13:37:22.547180 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:37:22.547141 2585 generic.go:356] "Generic (PLEG): container finished" podID="ef282945-09c0-4f65-9040-e1b7c849665e" containerID="538bf2a2e460ae5100f55e419024f84e33c4e24d5fd46c080c75580404e7edd5" exitCode=0 Sep 10 13:37:22.547570 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:37:22.547223 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f6kfp7" event={"ID":"ef282945-09c0-4f65-9040-e1b7c849665e","Type":"ContainerDied","Data":"538bf2a2e460ae5100f55e419024f84e33c4e24d5fd46c080c75580404e7edd5"} Sep 10 13:37:29.580195 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:37:29.580155 2585 generic.go:356] "Generic (PLEG): container finished" podID="ef282945-09c0-4f65-9040-e1b7c849665e" containerID="1677df97e775aa8d00a116569bdaeaa108871d169acc8d039199cf1ff510675a" exitCode=0 Sep 10 13:37:29.580575 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:37:29.580232 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f6kfp7" event={"ID":"ef282945-09c0-4f65-9040-e1b7c849665e","Type":"ContainerDied","Data":"1677df97e775aa8d00a116569bdaeaa108871d169acc8d039199cf1ff510675a"} Sep 10 13:37:30.709031 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:37:30.709007 2585 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f6kfp7" Sep 10 13:37:30.740471 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:37:30.740443 2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/ef282945-09c0-4f65-9040-e1b7c849665e-bundle\") pod \"ef282945-09c0-4f65-9040-e1b7c849665e\" (UID: \"ef282945-09c0-4f65-9040-e1b7c849665e\") " Sep 10 13:37:30.740592 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:37:30.740503 2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-prvhb\" (UniqueName: \"kubernetes.io/projected/ef282945-09c0-4f65-9040-e1b7c849665e-kube-api-access-prvhb\") pod \"ef282945-09c0-4f65-9040-e1b7c849665e\" (UID: \"ef282945-09c0-4f65-9040-e1b7c849665e\") " Sep 10 13:37:30.740592 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:37:30.740541 2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/ef282945-09c0-4f65-9040-e1b7c849665e-util\") pod \"ef282945-09c0-4f65-9040-e1b7c849665e\" (UID: \"ef282945-09c0-4f65-9040-e1b7c849665e\") " Sep 10 13:37:30.740904 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:37:30.740877 2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ef282945-09c0-4f65-9040-e1b7c849665e-bundle" (OuterVolumeSpecName: "bundle") pod "ef282945-09c0-4f65-9040-e1b7c849665e" (UID: "ef282945-09c0-4f65-9040-e1b7c849665e"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 10 13:37:30.742798 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:37:30.742765 2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ef282945-09c0-4f65-9040-e1b7c849665e-kube-api-access-prvhb" (OuterVolumeSpecName: "kube-api-access-prvhb") pod "ef282945-09c0-4f65-9040-e1b7c849665e" (UID: "ef282945-09c0-4f65-9040-e1b7c849665e"). InnerVolumeSpecName "kube-api-access-prvhb". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 10 13:37:30.745443 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:37:30.745421 2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ef282945-09c0-4f65-9040-e1b7c849665e-util" (OuterVolumeSpecName: "util") pod "ef282945-09c0-4f65-9040-e1b7c849665e" (UID: "ef282945-09c0-4f65-9040-e1b7c849665e"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 10 13:37:30.841638 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:37:30.841562 2585 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/ef282945-09c0-4f65-9040-e1b7c849665e-bundle\") on node \"ip-10-0-140-8.ec2.internal\" DevicePath \"\"" Sep 10 13:37:30.841638 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:37:30.841588 2585 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-prvhb\" (UniqueName: \"kubernetes.io/projected/ef282945-09c0-4f65-9040-e1b7c849665e-kube-api-access-prvhb\") on node \"ip-10-0-140-8.ec2.internal\" DevicePath \"\"" Sep 10 13:37:30.841638 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:37:30.841599 2585 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/ef282945-09c0-4f65-9040-e1b7c849665e-util\") on node \"ip-10-0-140-8.ec2.internal\" DevicePath \"\"" Sep 10 13:37:31.006504 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:37:31.006472 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fxgmp_fa53c783-d40a-4024-9f7b-1d0d2b57f6dc/konnectivity-agent/0.log" limit=1024 Sep 10 13:37:31.590660 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:37:31.590626 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f6kfp7" event={"ID":"ef282945-09c0-4f65-9040-e1b7c849665e","Type":"ContainerDied","Data":"cfd8f3d89fdefe889c44e888a047a3e1b3aa4af91d175b071ba63940c43666e5"} Sep 10 13:37:31.590660 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:37:31.590649 2585 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f6kfp7" Sep 10 13:37:31.590877 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:37:31.590679 2585 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="cfd8f3d89fdefe889c44e888a047a3e1b3aa4af91d175b071ba63940c43666e5" Sep 10 13:37:33.679902 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:37:33.679868 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fxgmp_fa53c783-d40a-4024-9f7b-1d0d2b57f6dc/konnectivity-agent/0.log" limit=1024 Sep 10 13:37:33.854835 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:37:33.854802 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fxgmp_fa53c783-d40a-4024-9f7b-1d0d2b57f6dc/konnectivity-agent/0.log" limit=1024 Sep 10 13:37:34.871112 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:37:34.871079 2585 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-lws-operator/openshift-lws-operator-fd8ccff4c-w7vrb"] Sep 10 13:37:34.871709 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:37:34.871546 2585 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ef282945-09c0-4f65-9040-e1b7c849665e" containerName="extract" Sep 10 13:37:34.871709 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:37:34.871566 2585 state_mem.go:107] "Deleted CPUSet assignment" podUID="ef282945-09c0-4f65-9040-e1b7c849665e" containerName="extract" Sep 10 13:37:34.871709 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:37:34.871601 2585 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ef282945-09c0-4f65-9040-e1b7c849665e" containerName="util" Sep 10 13:37:34.871709 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:37:34.871610 2585 state_mem.go:107] "Deleted CPUSet assignment" podUID="ef282945-09c0-4f65-9040-e1b7c849665e" containerName="util" Sep 10 13:37:34.871709 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:37:34.871625 2585 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ef282945-09c0-4f65-9040-e1b7c849665e" containerName="pull" Sep 10 13:37:34.871709 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:37:34.871633 2585 state_mem.go:107] "Deleted CPUSet assignment" podUID="ef282945-09c0-4f65-9040-e1b7c849665e" containerName="pull" Sep 10 13:37:34.871709 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:37:34.871710 2585 memory_manager.go:368] "RemoveStaleState removing state" podUID="ef282945-09c0-4f65-9040-e1b7c849665e" containerName="extract" Sep 10 13:37:34.938466 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:37:34.938429 2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/openshift-lws-operator-fd8ccff4c-w7vrb"] Sep 10 13:37:34.938669 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:37:34.938557 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-w7vrb" Sep 10 13:37:34.943124 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:37:34.943101 2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"kube-root-ca.crt\"" Sep 10 13:37:34.944184 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:37:34.944163 2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"openshift-lws-operator-dockercfg-dt7cb\"" Sep 10 13:37:34.944318 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:37:34.944198 2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"openshift-service-ca.crt\"" Sep 10 13:37:34.970571 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:37:34.970535 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/37ec0b8f-049f-4318-bec4-bf1fa159c313-tmp\") pod \"openshift-lws-operator-fd8ccff4c-w7vrb\" (UID: \"37ec0b8f-049f-4318-bec4-bf1fa159c313\") " pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-w7vrb" Sep 10 13:37:34.970732 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:37:34.970614 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-58pl4\" (UniqueName: \"kubernetes.io/projected/37ec0b8f-049f-4318-bec4-bf1fa159c313-kube-api-access-58pl4\") pod \"openshift-lws-operator-fd8ccff4c-w7vrb\" (UID: \"37ec0b8f-049f-4318-bec4-bf1fa159c313\") " pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-w7vrb" Sep 10 13:37:35.071645 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:37:35.071602 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-58pl4\" (UniqueName: \"kubernetes.io/projected/37ec0b8f-049f-4318-bec4-bf1fa159c313-kube-api-access-58pl4\") pod \"openshift-lws-operator-fd8ccff4c-w7vrb\" (UID: \"37ec0b8f-049f-4318-bec4-bf1fa159c313\") " pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-w7vrb" Sep 10 13:37:35.071851 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:37:35.071718 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/37ec0b8f-049f-4318-bec4-bf1fa159c313-tmp\") pod \"openshift-lws-operator-fd8ccff4c-w7vrb\" (UID: \"37ec0b8f-049f-4318-bec4-bf1fa159c313\") " pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-w7vrb" Sep 10 13:37:35.072142 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:37:35.072113 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/37ec0b8f-049f-4318-bec4-bf1fa159c313-tmp\") pod \"openshift-lws-operator-fd8ccff4c-w7vrb\" (UID: \"37ec0b8f-049f-4318-bec4-bf1fa159c313\") " pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-w7vrb" Sep 10 13:37:35.086642 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:37:35.086616 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-58pl4\" (UniqueName: \"kubernetes.io/projected/37ec0b8f-049f-4318-bec4-bf1fa159c313-kube-api-access-58pl4\") pod \"openshift-lws-operator-fd8ccff4c-w7vrb\" (UID: \"37ec0b8f-049f-4318-bec4-bf1fa159c313\") " pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-w7vrb" Sep 10 13:37:35.247672 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:37:35.247592 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-w7vrb" Sep 10 13:37:35.555394 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:37:35.555316 2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/openshift-lws-operator-fd8ccff4c-w7vrb"] Sep 10 13:37:35.558058 ip-10-0-140-8 kubenswrapper[2585]: W0910 13:37:35.558031 2585 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod37ec0b8f_049f_4318_bec4_bf1fa159c313.slice/crio-c621e399fb266d8883ddf3ab21cfc9c918746d84fd9eb453ed92970e59ca4a70 WatchSource:0}: Error finding container c621e399fb266d8883ddf3ab21cfc9c918746d84fd9eb453ed92970e59ca4a70: Status 404 returned error can't find the container with id c621e399fb266d8883ddf3ab21cfc9c918746d84fd9eb453ed92970e59ca4a70 Sep 10 13:37:35.607855 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:37:35.607824 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-w7vrb" event={"ID":"37ec0b8f-049f-4318-bec4-bf1fa159c313","Type":"ContainerStarted","Data":"c621e399fb266d8883ddf3ab21cfc9c918746d84fd9eb453ed92970e59ca4a70"} Sep 10 13:37:38.622688 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:37:38.622651 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-w7vrb" event={"ID":"37ec0b8f-049f-4318-bec4-bf1fa159c313","Type":"ContainerStarted","Data":"148e55fc4e48c0989364a059d71fcdb91cb10959ab34aa505c1b6cd016637d7b"} Sep 10 13:37:39.598313 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:37:39.598276 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fxgmp_fa53c783-d40a-4024-9f7b-1d0d2b57f6dc/konnectivity-agent/0.log" limit=1024 Sep 10 13:37:39.712917 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:37:39.712880 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fxgmp_fa53c783-d40a-4024-9f7b-1d0d2b57f6dc/konnectivity-agent/0.log" limit=1024 Sep 10 13:37:41.351265 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:37:41.351214 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fxgmp_fa53c783-d40a-4024-9f7b-1d0d2b57f6dc/konnectivity-agent/0.log" limit=1024 Sep 10 13:37:45.564487 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:37:45.564457 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fxgmp_fa53c783-d40a-4024-9f7b-1d0d2b57f6dc/konnectivity-agent/0.log" limit=1024 Sep 10 13:37:45.653914 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:37:45.653860 2585 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-w7vrb" podStartSLOduration=9.326835248 podStartE2EDuration="11.653843323s" totalImagesPullingTime="2.327008075s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 13:37:34 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 13:37:38.651647539 +0000 UTC m=+543.524351740" watchObservedRunningTime="2026-09-10 13:37:45.653843323 +0000 UTC m=+550.526547528" Sep 10 13:37:45.654134 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:37:45.654117 2585 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-lws-operator/lws-controller-manager-7c4cb59d47-45xnl"] Sep 10 13:37:45.656318 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:37:45.656295 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/lws-controller-manager-7c4cb59d47-45xnl" Sep 10 13:37:45.660694 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:37:45.660672 2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"lws-controller-manager-dockercfg-mjg6w\"" Sep 10 13:37:45.660813 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:37:45.660740 2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"metrics-server-cert\"" Sep 10 13:37:45.660813 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:37:45.660785 2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"lws-manager-config\"" Sep 10 13:37:45.660937 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:37:45.660826 2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"webhook-server-cert\"" Sep 10 13:37:45.678681 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:37:45.678650 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fxgmp_fa53c783-d40a-4024-9f7b-1d0d2b57f6dc/konnectivity-agent/0.log" limit=1024 Sep 10 13:37:45.699853 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:37:45.699819 2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/lws-controller-manager-7c4cb59d47-45xnl"] Sep 10 13:37:45.759974 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:37:45.759943 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/ff6d9ee9-0cdd-4cb9-be20-42a90c11918c-manager-config\") pod \"lws-controller-manager-7c4cb59d47-45xnl\" (UID: \"ff6d9ee9-0cdd-4cb9-be20-42a90c11918c\") " pod="openshift-lws-operator/lws-controller-manager-7c4cb59d47-45xnl" Sep 10 13:37:45.760167 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:37:45.759982 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9h84w\" (UniqueName: \"kubernetes.io/projected/ff6d9ee9-0cdd-4cb9-be20-42a90c11918c-kube-api-access-9h84w\") pod \"lws-controller-manager-7c4cb59d47-45xnl\" (UID: \"ff6d9ee9-0cdd-4cb9-be20-42a90c11918c\") " pod="openshift-lws-operator/lws-controller-manager-7c4cb59d47-45xnl" Sep 10 13:37:45.760167 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:37:45.760017 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/ff6d9ee9-0cdd-4cb9-be20-42a90c11918c-metrics-cert\") pod \"lws-controller-manager-7c4cb59d47-45xnl\" (UID: \"ff6d9ee9-0cdd-4cb9-be20-42a90c11918c\") " pod="openshift-lws-operator/lws-controller-manager-7c4cb59d47-45xnl" Sep 10 13:37:45.760167 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:37:45.760135 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/ff6d9ee9-0cdd-4cb9-be20-42a90c11918c-cert\") pod \"lws-controller-manager-7c4cb59d47-45xnl\" (UID: \"ff6d9ee9-0cdd-4cb9-be20-42a90c11918c\") " pod="openshift-lws-operator/lws-controller-manager-7c4cb59d47-45xnl" Sep 10 13:37:45.861497 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:37:45.861411 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9h84w\" (UniqueName: \"kubernetes.io/projected/ff6d9ee9-0cdd-4cb9-be20-42a90c11918c-kube-api-access-9h84w\") pod \"lws-controller-manager-7c4cb59d47-45xnl\" (UID: \"ff6d9ee9-0cdd-4cb9-be20-42a90c11918c\") " pod="openshift-lws-operator/lws-controller-manager-7c4cb59d47-45xnl" Sep 10 13:37:45.861497 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:37:45.861461 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/ff6d9ee9-0cdd-4cb9-be20-42a90c11918c-metrics-cert\") pod \"lws-controller-manager-7c4cb59d47-45xnl\" (UID: \"ff6d9ee9-0cdd-4cb9-be20-42a90c11918c\") " pod="openshift-lws-operator/lws-controller-manager-7c4cb59d47-45xnl" Sep 10 13:37:45.861675 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:37:45.861590 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/ff6d9ee9-0cdd-4cb9-be20-42a90c11918c-cert\") pod \"lws-controller-manager-7c4cb59d47-45xnl\" (UID: \"ff6d9ee9-0cdd-4cb9-be20-42a90c11918c\") " pod="openshift-lws-operator/lws-controller-manager-7c4cb59d47-45xnl" Sep 10 13:37:45.861675 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:37:45.861658 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/ff6d9ee9-0cdd-4cb9-be20-42a90c11918c-manager-config\") pod \"lws-controller-manager-7c4cb59d47-45xnl\" (UID: \"ff6d9ee9-0cdd-4cb9-be20-42a90c11918c\") " pod="openshift-lws-operator/lws-controller-manager-7c4cb59d47-45xnl" Sep 10 13:37:45.862290 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:37:45.862269 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/ff6d9ee9-0cdd-4cb9-be20-42a90c11918c-manager-config\") pod \"lws-controller-manager-7c4cb59d47-45xnl\" (UID: \"ff6d9ee9-0cdd-4cb9-be20-42a90c11918c\") " pod="openshift-lws-operator/lws-controller-manager-7c4cb59d47-45xnl" Sep 10 13:37:45.863951 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:37:45.863930 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/ff6d9ee9-0cdd-4cb9-be20-42a90c11918c-cert\") pod \"lws-controller-manager-7c4cb59d47-45xnl\" (UID: \"ff6d9ee9-0cdd-4cb9-be20-42a90c11918c\") " pod="openshift-lws-operator/lws-controller-manager-7c4cb59d47-45xnl" Sep 10 13:37:45.864031 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:37:45.863932 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/ff6d9ee9-0cdd-4cb9-be20-42a90c11918c-metrics-cert\") pod \"lws-controller-manager-7c4cb59d47-45xnl\" (UID: \"ff6d9ee9-0cdd-4cb9-be20-42a90c11918c\") " pod="openshift-lws-operator/lws-controller-manager-7c4cb59d47-45xnl" Sep 10 13:37:45.872932 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:37:45.872903 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9h84w\" (UniqueName: \"kubernetes.io/projected/ff6d9ee9-0cdd-4cb9-be20-42a90c11918c-kube-api-access-9h84w\") pod \"lws-controller-manager-7c4cb59d47-45xnl\" (UID: \"ff6d9ee9-0cdd-4cb9-be20-42a90c11918c\") " pod="openshift-lws-operator/lws-controller-manager-7c4cb59d47-45xnl" Sep 10 13:37:45.966018 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:37:45.965970 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/lws-controller-manager-7c4cb59d47-45xnl" Sep 10 13:37:46.241043 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:37:46.241019 2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/lws-controller-manager-7c4cb59d47-45xnl"] Sep 10 13:37:46.243229 ip-10-0-140-8 kubenswrapper[2585]: W0910 13:37:46.243203 2585 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podff6d9ee9_0cdd_4cb9_be20_42a90c11918c.slice/crio-4d8cb6667598803d9f621b74d4ac0168410a257f2664fd92c28567ada4118bb2 WatchSource:0}: Error finding container 4d8cb6667598803d9f621b74d4ac0168410a257f2664fd92c28567ada4118bb2: Status 404 returned error can't find the container with id 4d8cb6667598803d9f621b74d4ac0168410a257f2664fd92c28567ada4118bb2 Sep 10 13:37:46.655163 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:37:46.655129 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/lws-controller-manager-7c4cb59d47-45xnl" event={"ID":"ff6d9ee9-0cdd-4cb9-be20-42a90c11918c","Type":"ContainerStarted","Data":"4d8cb6667598803d9f621b74d4ac0168410a257f2664fd92c28567ada4118bb2"} Sep 10 13:37:48.665632 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:37:48.665594 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/lws-controller-manager-7c4cb59d47-45xnl" event={"ID":"ff6d9ee9-0cdd-4cb9-be20-42a90c11918c","Type":"ContainerStarted","Data":"92f2f51ec7a1c891a6ea2b6ef5b36c77f43dd4cb30d193049ca41b68784eab17"} Sep 10 13:37:48.666019 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:37:48.665666 2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-lws-operator/lws-controller-manager-7c4cb59d47-45xnl" Sep 10 13:37:48.685423 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:37:48.685366 2585 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-lws-operator/lws-controller-manager-7c4cb59d47-45xnl" podStartSLOduration=1.712249012 podStartE2EDuration="3.685349599s" totalImagesPullingTime="1.973100587s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 13:37:45 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 13:37:48.683833201 +0000 UTC m=+553.556537403" watchObservedRunningTime="2026-09-10 13:37:48.685349599 +0000 UTC m=+553.558053850" Sep 10 13:37:57.371219 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:37:57.371187 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fxgmp_fa53c783-d40a-4024-9f7b-1d0d2b57f6dc/konnectivity-agent/0.log" limit=1024 Sep 10 13:37:57.504132 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:37:57.504096 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fxgmp_fa53c783-d40a-4024-9f7b-1d0d2b57f6dc/konnectivity-agent/0.log" limit=1024 Sep 10 13:37:58.181460 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:37:58.181420 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fxgmp_fa53c783-d40a-4024-9f7b-1d0d2b57f6dc/konnectivity-agent/0.log" limit=1024 Sep 10 13:37:58.667290 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:37:58.667258 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fxgmp_fa53c783-d40a-4024-9f7b-1d0d2b57f6dc/konnectivity-agent/0.log" limit=1024 Sep 10 13:37:58.767761 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:37:58.767728 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fxgmp_fa53c783-d40a-4024-9f7b-1d0d2b57f6dc/konnectivity-agent/0.log" limit=1024 Sep 10 13:37:58.854305 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:37:58.854266 2585 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-579d56558b-2bdlh"] Sep 10 13:37:58.856909 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:37:58.856894 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/opendatahub-operator-controller-manager-579d56558b-2bdlh" Sep 10 13:37:58.861945 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:37:58.861921 2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-webhook-cert\"" Sep 10 13:37:58.862503 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:37:58.862486 2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-manager-service-cert\"" Sep 10 13:37:58.862596 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:37:58.862554 2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"kube-root-ca.crt\"" Sep 10 13:37:58.862743 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:37:58.862727 2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"openshift-service-ca.crt\"" Sep 10 13:37:58.862961 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:37:58.862943 2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-manager-dockercfg-6c66s\"" Sep 10 13:37:58.872363 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:37:58.872342 2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-579d56558b-2bdlh"] Sep 10 13:37:58.977136 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:37:58.977060 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/0673b863-b95e-427e-a6b6-c7ad063b27c9-tmp\") pod \"opendatahub-operator-controller-manager-579d56558b-2bdlh\" (UID: \"0673b863-b95e-427e-a6b6-c7ad063b27c9\") " pod="opendatahub/opendatahub-operator-controller-manager-579d56558b-2bdlh" Sep 10 13:37:58.977136 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:37:58.977096 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/0673b863-b95e-427e-a6b6-c7ad063b27c9-manifests\") pod \"opendatahub-operator-controller-manager-579d56558b-2bdlh\" (UID: \"0673b863-b95e-427e-a6b6-c7ad063b27c9\") " pod="opendatahub/opendatahub-operator-controller-manager-579d56558b-2bdlh" Sep 10 13:37:58.977354 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:37:58.977164 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wqj45\" (UniqueName: \"kubernetes.io/projected/0673b863-b95e-427e-a6b6-c7ad063b27c9-kube-api-access-wqj45\") pod \"opendatahub-operator-controller-manager-579d56558b-2bdlh\" (UID: \"0673b863-b95e-427e-a6b6-c7ad063b27c9\") " pod="opendatahub/opendatahub-operator-controller-manager-579d56558b-2bdlh" Sep 10 13:37:58.977354 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:37:58.977220 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/0673b863-b95e-427e-a6b6-c7ad063b27c9-webhook-cert\") pod \"opendatahub-operator-controller-manager-579d56558b-2bdlh\" (UID: \"0673b863-b95e-427e-a6b6-c7ad063b27c9\") " pod="opendatahub/opendatahub-operator-controller-manager-579d56558b-2bdlh" Sep 10 13:37:58.977354 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:37:58.977289 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/0673b863-b95e-427e-a6b6-c7ad063b27c9-apiservice-cert\") pod \"opendatahub-operator-controller-manager-579d56558b-2bdlh\" (UID: \"0673b863-b95e-427e-a6b6-c7ad063b27c9\") " pod="opendatahub/opendatahub-operator-controller-manager-579d56558b-2bdlh" Sep 10 13:37:59.078435 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:37:59.078402 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/0673b863-b95e-427e-a6b6-c7ad063b27c9-apiservice-cert\") pod \"opendatahub-operator-controller-manager-579d56558b-2bdlh\" (UID: \"0673b863-b95e-427e-a6b6-c7ad063b27c9\") " pod="opendatahub/opendatahub-operator-controller-manager-579d56558b-2bdlh" Sep 10 13:37:59.078642 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:37:59.078462 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/0673b863-b95e-427e-a6b6-c7ad063b27c9-tmp\") pod \"opendatahub-operator-controller-manager-579d56558b-2bdlh\" (UID: \"0673b863-b95e-427e-a6b6-c7ad063b27c9\") " pod="opendatahub/opendatahub-operator-controller-manager-579d56558b-2bdlh" Sep 10 13:37:59.078642 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:37:59.078494 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/0673b863-b95e-427e-a6b6-c7ad063b27c9-manifests\") pod \"opendatahub-operator-controller-manager-579d56558b-2bdlh\" (UID: \"0673b863-b95e-427e-a6b6-c7ad063b27c9\") " pod="opendatahub/opendatahub-operator-controller-manager-579d56558b-2bdlh" Sep 10 13:37:59.078642 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:37:59.078546 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wqj45\" (UniqueName: \"kubernetes.io/projected/0673b863-b95e-427e-a6b6-c7ad063b27c9-kube-api-access-wqj45\") pod \"opendatahub-operator-controller-manager-579d56558b-2bdlh\" (UID: \"0673b863-b95e-427e-a6b6-c7ad063b27c9\") " pod="opendatahub/opendatahub-operator-controller-manager-579d56558b-2bdlh" Sep 10 13:37:59.078642 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:37:59.078599 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/0673b863-b95e-427e-a6b6-c7ad063b27c9-webhook-cert\") pod \"opendatahub-operator-controller-manager-579d56558b-2bdlh\" (UID: \"0673b863-b95e-427e-a6b6-c7ad063b27c9\") " pod="opendatahub/opendatahub-operator-controller-manager-579d56558b-2bdlh" Sep 10 13:37:59.078900 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:37:59.078878 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/0673b863-b95e-427e-a6b6-c7ad063b27c9-manifests\") pod \"opendatahub-operator-controller-manager-579d56558b-2bdlh\" (UID: \"0673b863-b95e-427e-a6b6-c7ad063b27c9\") " pod="opendatahub/opendatahub-operator-controller-manager-579d56558b-2bdlh" Sep 10 13:37:59.080867 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:37:59.080837 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/0673b863-b95e-427e-a6b6-c7ad063b27c9-tmp\") pod \"opendatahub-operator-controller-manager-579d56558b-2bdlh\" (UID: \"0673b863-b95e-427e-a6b6-c7ad063b27c9\") " pod="opendatahub/opendatahub-operator-controller-manager-579d56558b-2bdlh" Sep 10 13:37:59.081164 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:37:59.081142 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/0673b863-b95e-427e-a6b6-c7ad063b27c9-apiservice-cert\") pod \"opendatahub-operator-controller-manager-579d56558b-2bdlh\" (UID: \"0673b863-b95e-427e-a6b6-c7ad063b27c9\") " pod="opendatahub/opendatahub-operator-controller-manager-579d56558b-2bdlh" Sep 10 13:37:59.081229 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:37:59.081176 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/0673b863-b95e-427e-a6b6-c7ad063b27c9-webhook-cert\") pod \"opendatahub-operator-controller-manager-579d56558b-2bdlh\" (UID: \"0673b863-b95e-427e-a6b6-c7ad063b27c9\") " pod="opendatahub/opendatahub-operator-controller-manager-579d56558b-2bdlh" Sep 10 13:37:59.102318 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:37:59.102294 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-wqj45\" (UniqueName: \"kubernetes.io/projected/0673b863-b95e-427e-a6b6-c7ad063b27c9-kube-api-access-wqj45\") pod \"opendatahub-operator-controller-manager-579d56558b-2bdlh\" (UID: \"0673b863-b95e-427e-a6b6-c7ad063b27c9\") " pod="opendatahub/opendatahub-operator-controller-manager-579d56558b-2bdlh" Sep 10 13:37:59.150317 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:37:59.150263 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fxgmp_fa53c783-d40a-4024-9f7b-1d0d2b57f6dc/konnectivity-agent/0.log" limit=1024 Sep 10 13:37:59.166679 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:37:59.166657 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/opendatahub-operator-controller-manager-579d56558b-2bdlh" Sep 10 13:37:59.353079 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:37:59.353043 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fxgmp_fa53c783-d40a-4024-9f7b-1d0d2b57f6dc/konnectivity-agent/0.log" limit=1024 Sep 10 13:37:59.452447 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:37:59.452412 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fxgmp_fa53c783-d40a-4024-9f7b-1d0d2b57f6dc/konnectivity-agent/0.log" limit=1024 Sep 10 13:37:59.473655 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:37:59.473629 2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-579d56558b-2bdlh"] Sep 10 13:37:59.475494 ip-10-0-140-8 kubenswrapper[2585]: W0910 13:37:59.475470 2585 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0673b863_b95e_427e_a6b6_c7ad063b27c9.slice/crio-4a447686543dcc32484c44d380d5d649218cc43bea3cb7db53097dd473f98b1a WatchSource:0}: Error finding container 4a447686543dcc32484c44d380d5d649218cc43bea3cb7db53097dd473f98b1a: Status 404 returned error can't find the container with id 4a447686543dcc32484c44d380d5d649218cc43bea3cb7db53097dd473f98b1a Sep 10 13:37:59.564057 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:37:59.564030 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fxgmp_fa53c783-d40a-4024-9f7b-1d0d2b57f6dc/konnectivity-agent/0.log" limit=1024 Sep 10 13:37:59.671770 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:37:59.671695 2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-lws-operator/lws-controller-manager-7c4cb59d47-45xnl" Sep 10 13:37:59.675101 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:37:59.675055 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fxgmp_fa53c783-d40a-4024-9f7b-1d0d2b57f6dc/konnectivity-agent/0.log" limit=1024 Sep 10 13:37:59.717180 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:37:59.717146 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-579d56558b-2bdlh" event={"ID":"0673b863-b95e-427e-a6b6-c7ad063b27c9","Type":"ContainerStarted","Data":"4a447686543dcc32484c44d380d5d649218cc43bea3cb7db53097dd473f98b1a"} Sep 10 13:37:59.768513 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:37:59.768480 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fxgmp_fa53c783-d40a-4024-9f7b-1d0d2b57f6dc/konnectivity-agent/0.log" limit=1024 Sep 10 13:37:59.868346 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:37:59.868318 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fxgmp_fa53c783-d40a-4024-9f7b-1d0d2b57f6dc/konnectivity-agent/0.log" limit=1024 Sep 10 13:38:00.412505 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:38:00.412472 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fxgmp_fa53c783-d40a-4024-9f7b-1d0d2b57f6dc/konnectivity-agent/0.log" limit=1024 Sep 10 13:38:02.734530 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:38:02.734497 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-579d56558b-2bdlh" event={"ID":"0673b863-b95e-427e-a6b6-c7ad063b27c9","Type":"ContainerStarted","Data":"d54b5b618ea83437f604fb6354bb97c6bdba05d3113870aaa661a85b6ef4dfca"} Sep 10 13:38:03.741092 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:38:03.741059 2585 generic.go:356] "Generic (PLEG): container finished" podID="0673b863-b95e-427e-a6b6-c7ad063b27c9" containerID="d54b5b618ea83437f604fb6354bb97c6bdba05d3113870aaa661a85b6ef4dfca" exitCode=0 Sep 10 13:38:03.741496 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:38:03.741143 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-579d56558b-2bdlh" event={"ID":"0673b863-b95e-427e-a6b6-c7ad063b27c9","Type":"ContainerDied","Data":"d54b5b618ea83437f604fb6354bb97c6bdba05d3113870aaa661a85b6ef4dfca"} Sep 10 13:38:04.748418 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:38:04.748331 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-579d56558b-2bdlh" event={"ID":"0673b863-b95e-427e-a6b6-c7ad063b27c9","Type":"ContainerStarted","Data":"9dc29ba51b09e1718732c9d90360dae128ab58967c132d3f15a67db2d3d06471"} Sep 10 13:38:04.748794 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:38:04.748464 2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/opendatahub-operator-controller-manager-579d56558b-2bdlh" Sep 10 13:38:10.426236 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:38:10.426198 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fxgmp_fa53c783-d40a-4024-9f7b-1d0d2b57f6dc/konnectivity-agent/0.log" limit=1024 Sep 10 13:38:10.524426 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:38:10.524389 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fxgmp_fa53c783-d40a-4024-9f7b-1d0d2b57f6dc/konnectivity-agent/0.log" limit=1024 Sep 10 13:38:15.755576 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:38:15.755547 2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/opendatahub-operator-controller-manager-579d56558b-2bdlh" Sep 10 13:38:15.791119 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:38:15.791062 2585 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/opendatahub-operator-controller-manager-579d56558b-2bdlh" podStartSLOduration=13.15742421 podStartE2EDuration="17.791047005s" totalImagesPullingTime="3.630705839s" totalInitContainerRuntime="1.002916956s" isStatefulPod=false podCreationTimestamp="2026-09-10 13:37:58 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 13:38:04.791012932 +0000 UTC m=+569.663717152" watchObservedRunningTime="2026-09-10 13:38:15.791047005 +0000 UTC m=+580.663751206" Sep 10 13:38:18.497403 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:38:18.497370 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fxgmp_fa53c783-d40a-4024-9f7b-1d0d2b57f6dc/konnectivity-agent/0.log" limit=1024 Sep 10 13:38:18.595606 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:38:18.595571 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fxgmp_fa53c783-d40a-4024-9f7b-1d0d2b57f6dc/konnectivity-agent/0.log" limit=1024 Sep 10 13:38:18.844359 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:38:18.844325 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fxgmp_fa53c783-d40a-4024-9f7b-1d0d2b57f6dc/konnectivity-agent/0.log" limit=1024 Sep 10 13:38:18.925224 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:38:18.925184 2585 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/kube-auth-proxy-69ddc5c8c5-5dr6s"] Sep 10 13:38:18.935126 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:38:18.935100 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/kube-auth-proxy-69ddc5c8c5-5dr6s" Sep 10 13:38:18.937917 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:38:18.937888 2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-creds\"" Sep 10 13:38:18.938050 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:38:18.937924 2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-dockercfg-9grd5\"" Sep 10 13:38:18.938050 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:38:18.938026 2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-tls\"" Sep 10 13:38:18.939473 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:38:18.939265 2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"kube-root-ca.crt\"" Sep 10 13:38:18.939473 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:38:18.939315 2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-service-ca.crt\"" Sep 10 13:38:18.941307 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:38:18.941281 2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/kube-auth-proxy-69ddc5c8c5-5dr6s"] Sep 10 13:38:18.951027 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:38:18.951005 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lwn96\" (UniqueName: \"kubernetes.io/projected/3fcb014d-e04c-4ffc-98e9-76502822202b-kube-api-access-lwn96\") pod \"kube-auth-proxy-69ddc5c8c5-5dr6s\" (UID: \"3fcb014d-e04c-4ffc-98e9-76502822202b\") " pod="openshift-ingress/kube-auth-proxy-69ddc5c8c5-5dr6s" Sep 10 13:38:18.951111 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:38:18.951041 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3fcb014d-e04c-4ffc-98e9-76502822202b-tls-certs\") pod \"kube-auth-proxy-69ddc5c8c5-5dr6s\" (UID: \"3fcb014d-e04c-4ffc-98e9-76502822202b\") " pod="openshift-ingress/kube-auth-proxy-69ddc5c8c5-5dr6s" Sep 10 13:38:18.951165 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:38:18.951118 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/3fcb014d-e04c-4ffc-98e9-76502822202b-tmp\") pod \"kube-auth-proxy-69ddc5c8c5-5dr6s\" (UID: \"3fcb014d-e04c-4ffc-98e9-76502822202b\") " pod="openshift-ingress/kube-auth-proxy-69ddc5c8c5-5dr6s" Sep 10 13:38:19.001376 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:38:19.001342 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fxgmp_fa53c783-d40a-4024-9f7b-1d0d2b57f6dc/konnectivity-agent/0.log" limit=1024 Sep 10 13:38:19.051648 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:38:19.051609 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/3fcb014d-e04c-4ffc-98e9-76502822202b-tmp\") pod \"kube-auth-proxy-69ddc5c8c5-5dr6s\" (UID: \"3fcb014d-e04c-4ffc-98e9-76502822202b\") " pod="openshift-ingress/kube-auth-proxy-69ddc5c8c5-5dr6s" Sep 10 13:38:19.051826 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:38:19.051656 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lwn96\" (UniqueName: \"kubernetes.io/projected/3fcb014d-e04c-4ffc-98e9-76502822202b-kube-api-access-lwn96\") pod \"kube-auth-proxy-69ddc5c8c5-5dr6s\" (UID: \"3fcb014d-e04c-4ffc-98e9-76502822202b\") " pod="openshift-ingress/kube-auth-proxy-69ddc5c8c5-5dr6s" Sep 10 13:38:19.051826 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:38:19.051701 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3fcb014d-e04c-4ffc-98e9-76502822202b-tls-certs\") pod \"kube-auth-proxy-69ddc5c8c5-5dr6s\" (UID: \"3fcb014d-e04c-4ffc-98e9-76502822202b\") " pod="openshift-ingress/kube-auth-proxy-69ddc5c8c5-5dr6s" Sep 10 13:38:19.054096 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:38:19.054069 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/3fcb014d-e04c-4ffc-98e9-76502822202b-tmp\") pod \"kube-auth-proxy-69ddc5c8c5-5dr6s\" (UID: \"3fcb014d-e04c-4ffc-98e9-76502822202b\") " pod="openshift-ingress/kube-auth-proxy-69ddc5c8c5-5dr6s" Sep 10 13:38:19.054313 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:38:19.054295 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3fcb014d-e04c-4ffc-98e9-76502822202b-tls-certs\") pod \"kube-auth-proxy-69ddc5c8c5-5dr6s\" (UID: \"3fcb014d-e04c-4ffc-98e9-76502822202b\") " pod="openshift-ingress/kube-auth-proxy-69ddc5c8c5-5dr6s" Sep 10 13:38:19.063164 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:38:19.063137 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-lwn96\" (UniqueName: \"kubernetes.io/projected/3fcb014d-e04c-4ffc-98e9-76502822202b-kube-api-access-lwn96\") pod \"kube-auth-proxy-69ddc5c8c5-5dr6s\" (UID: \"3fcb014d-e04c-4ffc-98e9-76502822202b\") " pod="openshift-ingress/kube-auth-proxy-69ddc5c8c5-5dr6s" Sep 10 13:38:19.246338 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:38:19.246260 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/kube-auth-proxy-69ddc5c8c5-5dr6s" Sep 10 13:38:19.520716 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:38:19.520689 2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/kube-auth-proxy-69ddc5c8c5-5dr6s"] Sep 10 13:38:19.523281 ip-10-0-140-8 kubenswrapper[2585]: W0910 13:38:19.523240 2585 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3fcb014d_e04c_4ffc_98e9_76502822202b.slice/crio-4024dbccff001e84c04125b8697e26e91693f29e42706f0eb602df0d8ae97ffd WatchSource:0}: Error finding container 4024dbccff001e84c04125b8697e26e91693f29e42706f0eb602df0d8ae97ffd: Status 404 returned error can't find the container with id 4024dbccff001e84c04125b8697e26e91693f29e42706f0eb602df0d8ae97ffd Sep 10 13:38:19.815359 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:38:19.815328 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/kube-auth-proxy-69ddc5c8c5-5dr6s" event={"ID":"3fcb014d-e04c-4ffc-98e9-76502822202b","Type":"ContainerStarted","Data":"4024dbccff001e84c04125b8697e26e91693f29e42706f0eb602df0d8ae97ffd"} Sep 10 13:38:20.186360 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:38:20.186260 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fxgmp_fa53c783-d40a-4024-9f7b-1d0d2b57f6dc/konnectivity-agent/0.log" limit=1024 Sep 10 13:38:20.286304 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:38:20.286256 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fxgmp_fa53c783-d40a-4024-9f7b-1d0d2b57f6dc/konnectivity-agent/0.log" limit=1024 Sep 10 13:38:21.530956 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:38:21.530909 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fxgmp_fa53c783-d40a-4024-9f7b-1d0d2b57f6dc/konnectivity-agent/0.log" limit=1024 Sep 10 13:38:22.833011 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:38:22.832980 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/kube-auth-proxy-69ddc5c8c5-5dr6s" event={"ID":"3fcb014d-e04c-4ffc-98e9-76502822202b","Type":"ContainerStarted","Data":"cf0da5811271025d9c48cf95a7254c7c8c77d58f3b391afda850353e78ddc495"} Sep 10 13:38:22.854058 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:38:22.854006 2585 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/kube-auth-proxy-69ddc5c8c5-5dr6s" podStartSLOduration=2.036621912 podStartE2EDuration="4.853989644s" totalImagesPullingTime="2.817367732s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 13:38:18 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 13:38:22.853383294 +0000 UTC m=+587.726087520" watchObservedRunningTime="2026-09-10 13:38:22.853989644 +0000 UTC m=+587.726693845" Sep 10 13:38:24.971547 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:38:24.971505 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fxgmp_fa53c783-d40a-4024-9f7b-1d0d2b57f6dc/konnectivity-agent/0.log" limit=1024 Sep 10 13:38:25.091879 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:38:25.091847 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fxgmp_fa53c783-d40a-4024-9f7b-1d0d2b57f6dc/konnectivity-agent/0.log" limit=1024 Sep 10 13:38:26.323122 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:38:26.323088 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fxgmp_fa53c783-d40a-4024-9f7b-1d0d2b57f6dc/konnectivity-agent/0.log" limit=1024 Sep 10 13:38:26.642074 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:38:26.641989 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fxgmp_fa53c783-d40a-4024-9f7b-1d0d2b57f6dc/konnectivity-agent/0.log" limit=1024 Sep 10 13:38:26.900474 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:38:26.900390 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fxgmp_fa53c783-d40a-4024-9f7b-1d0d2b57f6dc/konnectivity-agent/0.log" limit=1024 Sep 10 13:38:29.478448 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:38:29.478415 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fxgmp_fa53c783-d40a-4024-9f7b-1d0d2b57f6dc/konnectivity-agent/0.log" limit=1024 Sep 10 13:38:29.556436 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:38:29.556401 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fxgmp_fa53c783-d40a-4024-9f7b-1d0d2b57f6dc/konnectivity-agent/0.log" limit=1024 Sep 10 13:38:29.731088 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:38:29.731014 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fxgmp_fa53c783-d40a-4024-9f7b-1d0d2b57f6dc/konnectivity-agent/0.log" limit=1024 Sep 10 13:38:29.828536 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:38:29.828498 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fxgmp_fa53c783-d40a-4024-9f7b-1d0d2b57f6dc/konnectivity-agent/0.log" limit=1024 Sep 10 13:38:30.279654 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:38:30.279625 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fxgmp_fa53c783-d40a-4024-9f7b-1d0d2b57f6dc/konnectivity-agent/0.log" limit=1024 Sep 10 13:38:30.543200 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:38:30.543121 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fxgmp_fa53c783-d40a-4024-9f7b-1d0d2b57f6dc/konnectivity-agent/0.log" limit=1024 Sep 10 13:38:35.599368 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:38:35.599337 2585 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-l9l8q_ad555c6a-4385-4f8e-93bb-a174cfaa8e74/kube-multus/2.log" Sep 10 13:38:35.600499 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:38:35.600470 2585 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-l9l8q_ad555c6a-4385-4f8e-93bb-a174cfaa8e74/kube-multus/2.log" Sep 10 13:38:44.237299 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:38:44.237260 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fxgmp_fa53c783-d40a-4024-9f7b-1d0d2b57f6dc/konnectivity-agent/0.log" limit=1024 Sep 10 13:38:44.380485 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:38:44.380450 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fxgmp_fa53c783-d40a-4024-9f7b-1d0d2b57f6dc/konnectivity-agent/0.log" limit=1024 Sep 10 13:38:44.549327 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:38:44.549222 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fxgmp_fa53c783-d40a-4024-9f7b-1d0d2b57f6dc/konnectivity-agent/0.log" limit=1024 Sep 10 13:38:44.649210 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:38:44.649181 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fxgmp_fa53c783-d40a-4024-9f7b-1d0d2b57f6dc/konnectivity-agent/0.log" limit=1024 Sep 10 13:38:44.802586 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:38:44.802415 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fxgmp_fa53c783-d40a-4024-9f7b-1d0d2b57f6dc/konnectivity-agent/0.log" limit=1024 Sep 10 13:38:44.940802 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:38:44.940774 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fxgmp_fa53c783-d40a-4024-9f7b-1d0d2b57f6dc/konnectivity-agent/0.log" limit=1024 Sep 10 13:38:59.517125 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:38:59.517044 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fxgmp_fa53c783-d40a-4024-9f7b-1d0d2b57f6dc/konnectivity-agent/0.log" limit=1024 Sep 10 13:39:12.506946 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:39:12.506911 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fxgmp_fa53c783-d40a-4024-9f7b-1d0d2b57f6dc/konnectivity-agent/0.log" limit=1024 Sep 10 13:39:30.656308 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:39:30.656272 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fxgmp_fa53c783-d40a-4024-9f7b-1d0d2b57f6dc/konnectivity-agent/0.log" limit=1024 Sep 10 13:39:30.755777 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:39:30.755746 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fxgmp_fa53c783-d40a-4024-9f7b-1d0d2b57f6dc/konnectivity-agent/0.log" limit=1024 Sep 10 13:39:35.009931 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:39:35.009898 2585 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847ak6wkh"] Sep 10 13:39:35.013578 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:39:35.013557 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847ak6wkh" Sep 10 13:39:35.016320 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:39:35.016300 2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"openshift-service-ca.crt\"" Sep 10 13:39:35.017325 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:39:35.017302 2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"default-dockercfg-m87wf\"" Sep 10 13:39:35.017423 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:39:35.017309 2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"kube-root-ca.crt\"" Sep 10 13:39:35.023534 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:39:35.023513 2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847ak6wkh"] Sep 10 13:39:35.097699 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:39:35.097668 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/ae377d6b-2529-422f-8618-fb5c4064ea16-util\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847ak6wkh\" (UID: \"ae377d6b-2529-422f-8618-fb5c4064ea16\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847ak6wkh" Sep 10 13:39:35.097879 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:39:35.097721 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/ae377d6b-2529-422f-8618-fb5c4064ea16-bundle\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847ak6wkh\" (UID: \"ae377d6b-2529-422f-8618-fb5c4064ea16\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847ak6wkh" Sep 10 13:39:35.097879 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:39:35.097782 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qxlzb\" (UniqueName: \"kubernetes.io/projected/ae377d6b-2529-422f-8618-fb5c4064ea16-kube-api-access-qxlzb\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847ak6wkh\" (UID: \"ae377d6b-2529-422f-8618-fb5c4064ea16\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847ak6wkh" Sep 10 13:39:35.160861 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:39:35.160823 2585 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cqg6br"] Sep 10 13:39:35.164494 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:39:35.164473 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cqg6br" Sep 10 13:39:35.174676 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:39:35.174650 2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cqg6br"] Sep 10 13:39:35.202397 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:39:35.199523 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qxlzb\" (UniqueName: \"kubernetes.io/projected/ae377d6b-2529-422f-8618-fb5c4064ea16-kube-api-access-qxlzb\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847ak6wkh\" (UID: \"ae377d6b-2529-422f-8618-fb5c4064ea16\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847ak6wkh" Sep 10 13:39:35.202397 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:39:35.199632 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/ae377d6b-2529-422f-8618-fb5c4064ea16-util\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847ak6wkh\" (UID: \"ae377d6b-2529-422f-8618-fb5c4064ea16\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847ak6wkh" Sep 10 13:39:35.202397 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:39:35.199698 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/ae377d6b-2529-422f-8618-fb5c4064ea16-bundle\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847ak6wkh\" (UID: \"ae377d6b-2529-422f-8618-fb5c4064ea16\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847ak6wkh" Sep 10 13:39:35.202397 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:39:35.200107 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/ae377d6b-2529-422f-8618-fb5c4064ea16-bundle\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847ak6wkh\" (UID: \"ae377d6b-2529-422f-8618-fb5c4064ea16\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847ak6wkh" Sep 10 13:39:35.202397 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:39:35.200698 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/ae377d6b-2529-422f-8618-fb5c4064ea16-util\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847ak6wkh\" (UID: \"ae377d6b-2529-422f-8618-fb5c4064ea16\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847ak6wkh" Sep 10 13:39:35.212400 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:39:35.212372 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qxlzb\" (UniqueName: \"kubernetes.io/projected/ae377d6b-2529-422f-8618-fb5c4064ea16-kube-api-access-qxlzb\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847ak6wkh\" (UID: \"ae377d6b-2529-422f-8618-fb5c4064ea16\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847ak6wkh" Sep 10 13:39:35.225854 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:39:35.225826 2585 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c62xnwc"] Sep 10 13:39:35.229605 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:39:35.229588 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c62xnwc" Sep 10 13:39:35.239134 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:39:35.239108 2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c62xnwc"] Sep 10 13:39:35.300516 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:39:35.300430 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/1caeb857-b271-4ee5-af15-3e098e17d134-util\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cqg6br\" (UID: \"1caeb857-b271-4ee5-af15-3e098e17d134\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cqg6br" Sep 10 13:39:35.300516 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:39:35.300488 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pd2z5\" (UniqueName: \"kubernetes.io/projected/d24a1606-266b-4849-876b-ad73b9263515-kube-api-access-pd2z5\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c62xnwc\" (UID: \"d24a1606-266b-4849-876b-ad73b9263515\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c62xnwc" Sep 10 13:39:35.300695 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:39:35.300543 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/1caeb857-b271-4ee5-af15-3e098e17d134-bundle\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cqg6br\" (UID: \"1caeb857-b271-4ee5-af15-3e098e17d134\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cqg6br" Sep 10 13:39:35.300695 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:39:35.300576 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/d24a1606-266b-4849-876b-ad73b9263515-bundle\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c62xnwc\" (UID: \"d24a1606-266b-4849-876b-ad73b9263515\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c62xnwc" Sep 10 13:39:35.300695 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:39:35.300596 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/d24a1606-266b-4849-876b-ad73b9263515-util\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c62xnwc\" (UID: \"d24a1606-266b-4849-876b-ad73b9263515\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c62xnwc" Sep 10 13:39:35.300695 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:39:35.300614 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nrbks\" (UniqueName: \"kubernetes.io/projected/1caeb857-b271-4ee5-af15-3e098e17d134-kube-api-access-nrbks\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cqg6br\" (UID: \"1caeb857-b271-4ee5-af15-3e098e17d134\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cqg6br" Sep 10 13:39:35.323384 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:39:35.323350 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847ak6wkh" Sep 10 13:39:35.326904 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:39:35.326881 2585 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7jbf9j"] Sep 10 13:39:35.330536 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:39:35.330514 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7jbf9j" Sep 10 13:39:35.342890 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:39:35.342861 2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7jbf9j"] Sep 10 13:39:35.401870 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:39:35.401841 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pd2z5\" (UniqueName: \"kubernetes.io/projected/d24a1606-266b-4849-876b-ad73b9263515-kube-api-access-pd2z5\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c62xnwc\" (UID: \"d24a1606-266b-4849-876b-ad73b9263515\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c62xnwc" Sep 10 13:39:35.401998 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:39:35.401881 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/507ca360-5b3b-4cba-85a3-20ca526f88a7-util\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7jbf9j\" (UID: \"507ca360-5b3b-4cba-85a3-20ca526f88a7\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7jbf9j" Sep 10 13:39:35.401998 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:39:35.401906 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-h4czg\" (UniqueName: \"kubernetes.io/projected/507ca360-5b3b-4cba-85a3-20ca526f88a7-kube-api-access-h4czg\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7jbf9j\" (UID: \"507ca360-5b3b-4cba-85a3-20ca526f88a7\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7jbf9j" Sep 10 13:39:35.401998 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:39:35.401931 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/1caeb857-b271-4ee5-af15-3e098e17d134-bundle\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cqg6br\" (UID: \"1caeb857-b271-4ee5-af15-3e098e17d134\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cqg6br" Sep 10 13:39:35.401998 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:39:35.401954 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/d24a1606-266b-4849-876b-ad73b9263515-bundle\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c62xnwc\" (UID: \"d24a1606-266b-4849-876b-ad73b9263515\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c62xnwc" Sep 10 13:39:35.401998 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:39:35.401980 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/d24a1606-266b-4849-876b-ad73b9263515-util\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c62xnwc\" (UID: \"d24a1606-266b-4849-876b-ad73b9263515\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c62xnwc" Sep 10 13:39:35.402310 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:39:35.402006 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nrbks\" (UniqueName: \"kubernetes.io/projected/1caeb857-b271-4ee5-af15-3e098e17d134-kube-api-access-nrbks\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cqg6br\" (UID: \"1caeb857-b271-4ee5-af15-3e098e17d134\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cqg6br" Sep 10 13:39:35.402310 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:39:35.402054 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/507ca360-5b3b-4cba-85a3-20ca526f88a7-bundle\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7jbf9j\" (UID: \"507ca360-5b3b-4cba-85a3-20ca526f88a7\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7jbf9j" Sep 10 13:39:35.402310 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:39:35.402100 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/1caeb857-b271-4ee5-af15-3e098e17d134-util\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cqg6br\" (UID: \"1caeb857-b271-4ee5-af15-3e098e17d134\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cqg6br" Sep 10 13:39:35.402495 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:39:35.402414 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/d24a1606-266b-4849-876b-ad73b9263515-bundle\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c62xnwc\" (UID: \"d24a1606-266b-4849-876b-ad73b9263515\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c62xnwc" Sep 10 13:39:35.402495 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:39:35.402416 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/1caeb857-b271-4ee5-af15-3e098e17d134-bundle\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cqg6br\" (UID: \"1caeb857-b271-4ee5-af15-3e098e17d134\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cqg6br" Sep 10 13:39:35.402607 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:39:35.402555 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/d24a1606-266b-4849-876b-ad73b9263515-util\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c62xnwc\" (UID: \"d24a1606-266b-4849-876b-ad73b9263515\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c62xnwc" Sep 10 13:39:35.402607 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:39:35.402590 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/1caeb857-b271-4ee5-af15-3e098e17d134-util\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cqg6br\" (UID: \"1caeb857-b271-4ee5-af15-3e098e17d134\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cqg6br" Sep 10 13:39:35.412226 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:39:35.412186 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-pd2z5\" (UniqueName: \"kubernetes.io/projected/d24a1606-266b-4849-876b-ad73b9263515-kube-api-access-pd2z5\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c62xnwc\" (UID: \"d24a1606-266b-4849-876b-ad73b9263515\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c62xnwc" Sep 10 13:39:35.412391 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:39:35.412313 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-nrbks\" (UniqueName: \"kubernetes.io/projected/1caeb857-b271-4ee5-af15-3e098e17d134-kube-api-access-nrbks\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cqg6br\" (UID: \"1caeb857-b271-4ee5-af15-3e098e17d134\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cqg6br" Sep 10 13:39:35.474416 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:39:35.474365 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cqg6br" Sep 10 13:39:35.503624 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:39:35.503586 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h4czg\" (UniqueName: \"kubernetes.io/projected/507ca360-5b3b-4cba-85a3-20ca526f88a7-kube-api-access-h4czg\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7jbf9j\" (UID: \"507ca360-5b3b-4cba-85a3-20ca526f88a7\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7jbf9j" Sep 10 13:39:35.503786 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:39:35.503682 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/507ca360-5b3b-4cba-85a3-20ca526f88a7-bundle\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7jbf9j\" (UID: \"507ca360-5b3b-4cba-85a3-20ca526f88a7\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7jbf9j" Sep 10 13:39:35.503786 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:39:35.503765 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/507ca360-5b3b-4cba-85a3-20ca526f88a7-util\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7jbf9j\" (UID: \"507ca360-5b3b-4cba-85a3-20ca526f88a7\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7jbf9j" Sep 10 13:39:35.504066 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:39:35.504042 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/507ca360-5b3b-4cba-85a3-20ca526f88a7-bundle\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7jbf9j\" (UID: \"507ca360-5b3b-4cba-85a3-20ca526f88a7\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7jbf9j" Sep 10 13:39:35.504170 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:39:35.504148 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/507ca360-5b3b-4cba-85a3-20ca526f88a7-util\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7jbf9j\" (UID: \"507ca360-5b3b-4cba-85a3-20ca526f88a7\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7jbf9j" Sep 10 13:39:35.514994 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:39:35.514968 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-h4czg\" (UniqueName: \"kubernetes.io/projected/507ca360-5b3b-4cba-85a3-20ca526f88a7-kube-api-access-h4czg\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7jbf9j\" (UID: \"507ca360-5b3b-4cba-85a3-20ca526f88a7\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7jbf9j" Sep 10 13:39:35.539811 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:39:35.539780 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c62xnwc" Sep 10 13:39:35.606289 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:39:35.606212 2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847ak6wkh"] Sep 10 13:39:35.611763 ip-10-0-140-8 kubenswrapper[2585]: W0910 13:39:35.611725 2585 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podae377d6b_2529_422f_8618_fb5c4064ea16.slice/crio-00166e62a98cceea6bfa56c4e0a3f86465e00cd9adf548808d6eb1ab08e206f2 WatchSource:0}: Error finding container 00166e62a98cceea6bfa56c4e0a3f86465e00cd9adf548808d6eb1ab08e206f2: Status 404 returned error can't find the container with id 00166e62a98cceea6bfa56c4e0a3f86465e00cd9adf548808d6eb1ab08e206f2 Sep 10 13:39:35.659663 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:39:35.659640 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7jbf9j" Sep 10 13:39:35.753353 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:39:35.753324 2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cqg6br"] Sep 10 13:39:35.779064 ip-10-0-140-8 kubenswrapper[2585]: W0910 13:39:35.779036 2585 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1caeb857_b271_4ee5_af15_3e098e17d134.slice/crio-f8533aed8ec587f714200998b1867f65d7f404eb110ce93cb21d3c1ba88a0922 WatchSource:0}: Error finding container f8533aed8ec587f714200998b1867f65d7f404eb110ce93cb21d3c1ba88a0922: Status 404 returned error can't find the container with id f8533aed8ec587f714200998b1867f65d7f404eb110ce93cb21d3c1ba88a0922 Sep 10 13:39:35.820296 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:39:35.820271 2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c62xnwc"] Sep 10 13:39:35.837315 ip-10-0-140-8 kubenswrapper[2585]: W0910 13:39:35.837291 2585 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd24a1606_266b_4849_876b_ad73b9263515.slice/crio-cbe07e52df59f288f53a7d2db0c78134d510b60c6fca995dbb17fa352d78c18b WatchSource:0}: Error finding container cbe07e52df59f288f53a7d2db0c78134d510b60c6fca995dbb17fa352d78c18b: Status 404 returned error can't find the container with id cbe07e52df59f288f53a7d2db0c78134d510b60c6fca995dbb17fa352d78c18b Sep 10 13:39:35.909377 ip-10-0-140-8 kubenswrapper[2585]: E0910 13:39:35.909345 2585 cadvisor_stats_provider.go:567] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1caeb857_b271_4ee5_af15_3e098e17d134.slice/crio-conmon-34bd69fe9bab6e44c1315124471754c7ac1617b63d0bd27b1886e592a76254c5.scope\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1caeb857_b271_4ee5_af15_3e098e17d134.slice/crio-34bd69fe9bab6e44c1315124471754c7ac1617b63d0bd27b1886e592a76254c5.scope\": RecentStats: unable to find data in memory cache]" Sep 10 13:39:35.909510 ip-10-0-140-8 kubenswrapper[2585]: E0910 13:39:35.909411 2585 cadvisor_stats_provider.go:567] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1caeb857_b271_4ee5_af15_3e098e17d134.slice/crio-34bd69fe9bab6e44c1315124471754c7ac1617b63d0bd27b1886e592a76254c5.scope\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1caeb857_b271_4ee5_af15_3e098e17d134.slice/crio-conmon-34bd69fe9bab6e44c1315124471754c7ac1617b63d0bd27b1886e592a76254c5.scope\": RecentStats: unable to find data in memory cache]" Sep 10 13:39:35.937861 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:39:35.937838 2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7jbf9j"] Sep 10 13:39:35.990106 ip-10-0-140-8 kubenswrapper[2585]: W0910 13:39:35.990072 2585 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod507ca360_5b3b_4cba_85a3_20ca526f88a7.slice/crio-5517a9a5d6091aae884a3925c645a888793fdfbd1d19624ddbe865495aab412e WatchSource:0}: Error finding container 5517a9a5d6091aae884a3925c645a888793fdfbd1d19624ddbe865495aab412e: Status 404 returned error can't find the container with id 5517a9a5d6091aae884a3925c645a888793fdfbd1d19624ddbe865495aab412e Sep 10 13:39:36.170837 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:39:36.170807 2585 generic.go:356] "Generic (PLEG): container finished" podID="507ca360-5b3b-4cba-85a3-20ca526f88a7" containerID="762cf05359ca8ebfc1b39a08323cdabe2125882b94bbb353aa84fbcc58ae56f3" exitCode=0 Sep 10 13:39:36.171411 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:39:36.170898 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7jbf9j" event={"ID":"507ca360-5b3b-4cba-85a3-20ca526f88a7","Type":"ContainerDied","Data":"762cf05359ca8ebfc1b39a08323cdabe2125882b94bbb353aa84fbcc58ae56f3"} Sep 10 13:39:36.171411 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:39:36.170949 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7jbf9j" event={"ID":"507ca360-5b3b-4cba-85a3-20ca526f88a7","Type":"ContainerStarted","Data":"5517a9a5d6091aae884a3925c645a888793fdfbd1d19624ddbe865495aab412e"} Sep 10 13:39:36.172076 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:39:36.172056 2585 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Sep 10 13:39:36.172649 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:39:36.172587 2585 generic.go:356] "Generic (PLEG): container finished" podID="ae377d6b-2529-422f-8618-fb5c4064ea16" containerID="0886f239b3bdcac9b094215995c3427a78917e44b680665cf539b8a175382564" exitCode=0 Sep 10 13:39:36.172734 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:39:36.172683 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847ak6wkh" event={"ID":"ae377d6b-2529-422f-8618-fb5c4064ea16","Type":"ContainerDied","Data":"0886f239b3bdcac9b094215995c3427a78917e44b680665cf539b8a175382564"} Sep 10 13:39:36.172798 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:39:36.172743 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847ak6wkh" event={"ID":"ae377d6b-2529-422f-8618-fb5c4064ea16","Type":"ContainerStarted","Data":"00166e62a98cceea6bfa56c4e0a3f86465e00cd9adf548808d6eb1ab08e206f2"} Sep 10 13:39:36.174297 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:39:36.174276 2585 generic.go:356] "Generic (PLEG): container finished" podID="d24a1606-266b-4849-876b-ad73b9263515" containerID="0328659dae2206e774142cbe1cfe764d1b9dc644ebb84d62ea46078083159ce1" exitCode=0 Sep 10 13:39:36.174383 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:39:36.174353 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c62xnwc" event={"ID":"d24a1606-266b-4849-876b-ad73b9263515","Type":"ContainerDied","Data":"0328659dae2206e774142cbe1cfe764d1b9dc644ebb84d62ea46078083159ce1"} Sep 10 13:39:36.174447 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:39:36.174390 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c62xnwc" event={"ID":"d24a1606-266b-4849-876b-ad73b9263515","Type":"ContainerStarted","Data":"cbe07e52df59f288f53a7d2db0c78134d510b60c6fca995dbb17fa352d78c18b"} Sep 10 13:39:36.176144 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:39:36.176124 2585 generic.go:356] "Generic (PLEG): container finished" podID="1caeb857-b271-4ee5-af15-3e098e17d134" containerID="34bd69fe9bab6e44c1315124471754c7ac1617b63d0bd27b1886e592a76254c5" exitCode=0 Sep 10 13:39:36.176276 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:39:36.176150 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cqg6br" event={"ID":"1caeb857-b271-4ee5-af15-3e098e17d134","Type":"ContainerDied","Data":"34bd69fe9bab6e44c1315124471754c7ac1617b63d0bd27b1886e592a76254c5"} Sep 10 13:39:36.176276 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:39:36.176181 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cqg6br" event={"ID":"1caeb857-b271-4ee5-af15-3e098e17d134","Type":"ContainerStarted","Data":"f8533aed8ec587f714200998b1867f65d7f404eb110ce93cb21d3c1ba88a0922"} Sep 10 13:39:37.182788 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:39:37.182756 2585 generic.go:356] "Generic (PLEG): container finished" podID="1caeb857-b271-4ee5-af15-3e098e17d134" containerID="c7898b74135dbd93858d2b31df5a09249213bf273f0fee5497b56050a9add900" exitCode=0 Sep 10 13:39:37.183185 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:39:37.182869 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cqg6br" event={"ID":"1caeb857-b271-4ee5-af15-3e098e17d134","Type":"ContainerDied","Data":"c7898b74135dbd93858d2b31df5a09249213bf273f0fee5497b56050a9add900"} Sep 10 13:39:38.190537 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:39:38.190502 2585 generic.go:356] "Generic (PLEG): container finished" podID="ae377d6b-2529-422f-8618-fb5c4064ea16" containerID="9dba238ab43ac21e3d70f2ea09650037359edcc821f147b6854a708a9d65f324" exitCode=0 Sep 10 13:39:38.190952 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:39:38.190588 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847ak6wkh" event={"ID":"ae377d6b-2529-422f-8618-fb5c4064ea16","Type":"ContainerDied","Data":"9dba238ab43ac21e3d70f2ea09650037359edcc821f147b6854a708a9d65f324"} Sep 10 13:39:38.192574 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:39:38.192552 2585 generic.go:356] "Generic (PLEG): container finished" podID="d24a1606-266b-4849-876b-ad73b9263515" containerID="582211c46161bf3e0555345ce63a30c9417856ed16f0b50190f6eb4bdf98d5d9" exitCode=0 Sep 10 13:39:38.192661 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:39:38.192622 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c62xnwc" event={"ID":"d24a1606-266b-4849-876b-ad73b9263515","Type":"ContainerDied","Data":"582211c46161bf3e0555345ce63a30c9417856ed16f0b50190f6eb4bdf98d5d9"} Sep 10 13:39:38.194848 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:39:38.194825 2585 generic.go:356] "Generic (PLEG): container finished" podID="1caeb857-b271-4ee5-af15-3e098e17d134" containerID="0226b72fdd555b858b58476d6cea5725783f949692ed1d9f30a44b0d0c2947a6" exitCode=0 Sep 10 13:39:38.194977 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:39:38.194954 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cqg6br" event={"ID":"1caeb857-b271-4ee5-af15-3e098e17d134","Type":"ContainerDied","Data":"0226b72fdd555b858b58476d6cea5725783f949692ed1d9f30a44b0d0c2947a6"} Sep 10 13:39:38.196792 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:39:38.196771 2585 generic.go:356] "Generic (PLEG): container finished" podID="507ca360-5b3b-4cba-85a3-20ca526f88a7" containerID="21455dc249cd50c9b89ddfc5de659d5e23cc110615a65a44bb49a1c9f058b128" exitCode=0 Sep 10 13:39:38.196922 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:39:38.196807 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7jbf9j" event={"ID":"507ca360-5b3b-4cba-85a3-20ca526f88a7","Type":"ContainerDied","Data":"21455dc249cd50c9b89ddfc5de659d5e23cc110615a65a44bb49a1c9f058b128"} Sep 10 13:39:39.203488 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:39:39.203456 2585 generic.go:356] "Generic (PLEG): container finished" podID="d24a1606-266b-4849-876b-ad73b9263515" containerID="52f851fc193f7f21e44acec235d0388f776c39afe9e458d7b36acee63f4b48d5" exitCode=0 Sep 10 13:39:39.203949 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:39:39.203545 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c62xnwc" event={"ID":"d24a1606-266b-4849-876b-ad73b9263515","Type":"ContainerDied","Data":"52f851fc193f7f21e44acec235d0388f776c39afe9e458d7b36acee63f4b48d5"} Sep 10 13:39:39.205655 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:39:39.205632 2585 generic.go:356] "Generic (PLEG): container finished" podID="507ca360-5b3b-4cba-85a3-20ca526f88a7" containerID="cf77cfce3b59a71bce229b87d33f7cdd11652f5ab61fbe01a0d5b1b793211896" exitCode=0 Sep 10 13:39:39.205765 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:39:39.205710 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7jbf9j" event={"ID":"507ca360-5b3b-4cba-85a3-20ca526f88a7","Type":"ContainerDied","Data":"cf77cfce3b59a71bce229b87d33f7cdd11652f5ab61fbe01a0d5b1b793211896"} Sep 10 13:39:39.207757 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:39:39.207738 2585 generic.go:356] "Generic (PLEG): container finished" podID="ae377d6b-2529-422f-8618-fb5c4064ea16" containerID="44132a5878ffb0fc618e7d61bc06d1bc176b3fd455b4caac1d2b016e862be63b" exitCode=0 Sep 10 13:39:39.207829 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:39:39.207763 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847ak6wkh" event={"ID":"ae377d6b-2529-422f-8618-fb5c4064ea16","Type":"ContainerDied","Data":"44132a5878ffb0fc618e7d61bc06d1bc176b3fd455b4caac1d2b016e862be63b"} Sep 10 13:39:39.340956 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:39:39.340926 2585 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cqg6br" Sep 10 13:39:39.443925 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:39:39.443888 2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-nrbks\" (UniqueName: \"kubernetes.io/projected/1caeb857-b271-4ee5-af15-3e098e17d134-kube-api-access-nrbks\") pod \"1caeb857-b271-4ee5-af15-3e098e17d134\" (UID: \"1caeb857-b271-4ee5-af15-3e098e17d134\") " Sep 10 13:39:39.444095 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:39:39.443948 2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/1caeb857-b271-4ee5-af15-3e098e17d134-util\") pod \"1caeb857-b271-4ee5-af15-3e098e17d134\" (UID: \"1caeb857-b271-4ee5-af15-3e098e17d134\") " Sep 10 13:39:39.444095 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:39:39.443982 2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/1caeb857-b271-4ee5-af15-3e098e17d134-bundle\") pod \"1caeb857-b271-4ee5-af15-3e098e17d134\" (UID: \"1caeb857-b271-4ee5-af15-3e098e17d134\") " Sep 10 13:39:39.444589 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:39:39.444564 2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1caeb857-b271-4ee5-af15-3e098e17d134-bundle" (OuterVolumeSpecName: "bundle") pod "1caeb857-b271-4ee5-af15-3e098e17d134" (UID: "1caeb857-b271-4ee5-af15-3e098e17d134"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 10 13:39:39.445973 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:39:39.445948 2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1caeb857-b271-4ee5-af15-3e098e17d134-kube-api-access-nrbks" (OuterVolumeSpecName: "kube-api-access-nrbks") pod "1caeb857-b271-4ee5-af15-3e098e17d134" (UID: "1caeb857-b271-4ee5-af15-3e098e17d134"). InnerVolumeSpecName "kube-api-access-nrbks". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 10 13:39:39.449679 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:39:39.449653 2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1caeb857-b271-4ee5-af15-3e098e17d134-util" (OuterVolumeSpecName: "util") pod "1caeb857-b271-4ee5-af15-3e098e17d134" (UID: "1caeb857-b271-4ee5-af15-3e098e17d134"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 10 13:39:39.545573 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:39:39.545468 2585 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nrbks\" (UniqueName: \"kubernetes.io/projected/1caeb857-b271-4ee5-af15-3e098e17d134-kube-api-access-nrbks\") on node \"ip-10-0-140-8.ec2.internal\" DevicePath \"\"" Sep 10 13:39:39.545573 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:39:39.545512 2585 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/1caeb857-b271-4ee5-af15-3e098e17d134-util\") on node \"ip-10-0-140-8.ec2.internal\" DevicePath \"\"" Sep 10 13:39:39.545573 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:39:39.545522 2585 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/1caeb857-b271-4ee5-af15-3e098e17d134-bundle\") on node \"ip-10-0-140-8.ec2.internal\" DevicePath \"\"" Sep 10 13:39:40.214328 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:39:40.214297 2585 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cqg6br" Sep 10 13:39:40.214328 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:39:40.214309 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cqg6br" event={"ID":"1caeb857-b271-4ee5-af15-3e098e17d134","Type":"ContainerDied","Data":"f8533aed8ec587f714200998b1867f65d7f404eb110ce93cb21d3c1ba88a0922"} Sep 10 13:39:40.214793 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:39:40.214400 2585 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="f8533aed8ec587f714200998b1867f65d7f404eb110ce93cb21d3c1ba88a0922" Sep 10 13:39:40.385768 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:39:40.385740 2585 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c62xnwc" Sep 10 13:39:40.387974 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:39:40.387952 2585 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847ak6wkh" Sep 10 13:39:40.409487 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:39:40.409464 2585 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7jbf9j" Sep 10 13:39:40.453862 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:39:40.453830 2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-qxlzb\" (UniqueName: \"kubernetes.io/projected/ae377d6b-2529-422f-8618-fb5c4064ea16-kube-api-access-qxlzb\") pod \"ae377d6b-2529-422f-8618-fb5c4064ea16\" (UID: \"ae377d6b-2529-422f-8618-fb5c4064ea16\") " Sep 10 13:39:40.453996 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:39:40.453882 2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-pd2z5\" (UniqueName: \"kubernetes.io/projected/d24a1606-266b-4849-876b-ad73b9263515-kube-api-access-pd2z5\") pod \"d24a1606-266b-4849-876b-ad73b9263515\" (UID: \"d24a1606-266b-4849-876b-ad73b9263515\") " Sep 10 13:39:40.453996 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:39:40.453951 2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/ae377d6b-2529-422f-8618-fb5c4064ea16-bundle\") pod \"ae377d6b-2529-422f-8618-fb5c4064ea16\" (UID: \"ae377d6b-2529-422f-8618-fb5c4064ea16\") " Sep 10 13:39:40.454085 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:39:40.454048 2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/ae377d6b-2529-422f-8618-fb5c4064ea16-util\") pod \"ae377d6b-2529-422f-8618-fb5c4064ea16\" (UID: \"ae377d6b-2529-422f-8618-fb5c4064ea16\") " Sep 10 13:39:40.454085 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:39:40.454073 2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/d24a1606-266b-4849-876b-ad73b9263515-bundle\") pod \"d24a1606-266b-4849-876b-ad73b9263515\" (UID: \"d24a1606-266b-4849-876b-ad73b9263515\") " Sep 10 13:39:40.454206 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:39:40.454094 2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/d24a1606-266b-4849-876b-ad73b9263515-util\") pod \"d24a1606-266b-4849-876b-ad73b9263515\" (UID: \"d24a1606-266b-4849-876b-ad73b9263515\") " Sep 10 13:39:40.454899 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:39:40.454810 2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d24a1606-266b-4849-876b-ad73b9263515-bundle" (OuterVolumeSpecName: "bundle") pod "d24a1606-266b-4849-876b-ad73b9263515" (UID: "d24a1606-266b-4849-876b-ad73b9263515"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 10 13:39:40.455078 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:39:40.454912 2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ae377d6b-2529-422f-8618-fb5c4064ea16-bundle" (OuterVolumeSpecName: "bundle") pod "ae377d6b-2529-422f-8618-fb5c4064ea16" (UID: "ae377d6b-2529-422f-8618-fb5c4064ea16"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 10 13:39:40.456309 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:39:40.456273 2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ae377d6b-2529-422f-8618-fb5c4064ea16-kube-api-access-qxlzb" (OuterVolumeSpecName: "kube-api-access-qxlzb") pod "ae377d6b-2529-422f-8618-fb5c4064ea16" (UID: "ae377d6b-2529-422f-8618-fb5c4064ea16"). InnerVolumeSpecName "kube-api-access-qxlzb". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 10 13:39:40.456437 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:39:40.456416 2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d24a1606-266b-4849-876b-ad73b9263515-kube-api-access-pd2z5" (OuterVolumeSpecName: "kube-api-access-pd2z5") pod "d24a1606-266b-4849-876b-ad73b9263515" (UID: "d24a1606-266b-4849-876b-ad73b9263515"). InnerVolumeSpecName "kube-api-access-pd2z5". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 10 13:39:40.461029 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:39:40.461005 2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ae377d6b-2529-422f-8618-fb5c4064ea16-util" (OuterVolumeSpecName: "util") pod "ae377d6b-2529-422f-8618-fb5c4064ea16" (UID: "ae377d6b-2529-422f-8618-fb5c4064ea16"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 10 13:39:40.462586 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:39:40.462558 2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d24a1606-266b-4849-876b-ad73b9263515-util" (OuterVolumeSpecName: "util") pod "d24a1606-266b-4849-876b-ad73b9263515" (UID: "d24a1606-266b-4849-876b-ad73b9263515"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 10 13:39:40.554692 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:39:40.554594 2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-h4czg\" (UniqueName: \"kubernetes.io/projected/507ca360-5b3b-4cba-85a3-20ca526f88a7-kube-api-access-h4czg\") pod \"507ca360-5b3b-4cba-85a3-20ca526f88a7\" (UID: \"507ca360-5b3b-4cba-85a3-20ca526f88a7\") " Sep 10 13:39:40.554692 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:39:40.554658 2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/507ca360-5b3b-4cba-85a3-20ca526f88a7-bundle\") pod \"507ca360-5b3b-4cba-85a3-20ca526f88a7\" (UID: \"507ca360-5b3b-4cba-85a3-20ca526f88a7\") " Sep 10 13:39:40.554910 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:39:40.554716 2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/507ca360-5b3b-4cba-85a3-20ca526f88a7-util\") pod \"507ca360-5b3b-4cba-85a3-20ca526f88a7\" (UID: \"507ca360-5b3b-4cba-85a3-20ca526f88a7\") " Sep 10 13:39:40.554995 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:39:40.554979 2585 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/ae377d6b-2529-422f-8618-fb5c4064ea16-bundle\") on node \"ip-10-0-140-8.ec2.internal\" DevicePath \"\"" Sep 10 13:39:40.555053 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:39:40.555000 2585 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/ae377d6b-2529-422f-8618-fb5c4064ea16-util\") on node \"ip-10-0-140-8.ec2.internal\" DevicePath \"\"" Sep 10 13:39:40.555053 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:39:40.555013 2585 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/d24a1606-266b-4849-876b-ad73b9263515-bundle\") on node \"ip-10-0-140-8.ec2.internal\" DevicePath \"\"" Sep 10 13:39:40.555053 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:39:40.555025 2585 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/d24a1606-266b-4849-876b-ad73b9263515-util\") on node \"ip-10-0-140-8.ec2.internal\" DevicePath \"\"" Sep 10 13:39:40.555053 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:39:40.555037 2585 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qxlzb\" (UniqueName: \"kubernetes.io/projected/ae377d6b-2529-422f-8618-fb5c4064ea16-kube-api-access-qxlzb\") on node \"ip-10-0-140-8.ec2.internal\" DevicePath \"\"" Sep 10 13:39:40.555185 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:39:40.555051 2585 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pd2z5\" (UniqueName: \"kubernetes.io/projected/d24a1606-266b-4849-876b-ad73b9263515-kube-api-access-pd2z5\") on node \"ip-10-0-140-8.ec2.internal\" DevicePath \"\"" Sep 10 13:39:40.555185 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:39:40.555122 2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/507ca360-5b3b-4cba-85a3-20ca526f88a7-bundle" (OuterVolumeSpecName: "bundle") pod "507ca360-5b3b-4cba-85a3-20ca526f88a7" (UID: "507ca360-5b3b-4cba-85a3-20ca526f88a7"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 10 13:39:40.556721 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:39:40.556694 2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/507ca360-5b3b-4cba-85a3-20ca526f88a7-kube-api-access-h4czg" (OuterVolumeSpecName: "kube-api-access-h4czg") pod "507ca360-5b3b-4cba-85a3-20ca526f88a7" (UID: "507ca360-5b3b-4cba-85a3-20ca526f88a7"). InnerVolumeSpecName "kube-api-access-h4czg". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 10 13:39:40.560450 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:39:40.560428 2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/507ca360-5b3b-4cba-85a3-20ca526f88a7-util" (OuterVolumeSpecName: "util") pod "507ca360-5b3b-4cba-85a3-20ca526f88a7" (UID: "507ca360-5b3b-4cba-85a3-20ca526f88a7"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 10 13:39:40.656348 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:39:40.656310 2585 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-h4czg\" (UniqueName: \"kubernetes.io/projected/507ca360-5b3b-4cba-85a3-20ca526f88a7-kube-api-access-h4czg\") on node \"ip-10-0-140-8.ec2.internal\" DevicePath \"\"" Sep 10 13:39:40.656348 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:39:40.656342 2585 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/507ca360-5b3b-4cba-85a3-20ca526f88a7-bundle\") on node \"ip-10-0-140-8.ec2.internal\" DevicePath \"\"" Sep 10 13:39:40.656348 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:39:40.656352 2585 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/507ca360-5b3b-4cba-85a3-20ca526f88a7-util\") on node \"ip-10-0-140-8.ec2.internal\" DevicePath \"\"" Sep 10 13:39:41.220692 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:39:41.220662 2585 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847ak6wkh" Sep 10 13:39:41.221041 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:39:41.220690 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847ak6wkh" event={"ID":"ae377d6b-2529-422f-8618-fb5c4064ea16","Type":"ContainerDied","Data":"00166e62a98cceea6bfa56c4e0a3f86465e00cd9adf548808d6eb1ab08e206f2"} Sep 10 13:39:41.221041 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:39:41.220737 2585 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="00166e62a98cceea6bfa56c4e0a3f86465e00cd9adf548808d6eb1ab08e206f2" Sep 10 13:39:41.222787 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:39:41.222766 2585 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c62xnwc" Sep 10 13:39:41.222787 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:39:41.222780 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c62xnwc" event={"ID":"d24a1606-266b-4849-876b-ad73b9263515","Type":"ContainerDied","Data":"cbe07e52df59f288f53a7d2db0c78134d510b60c6fca995dbb17fa352d78c18b"} Sep 10 13:39:41.222977 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:39:41.222824 2585 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="cbe07e52df59f288f53a7d2db0c78134d510b60c6fca995dbb17fa352d78c18b" Sep 10 13:39:41.224749 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:39:41.224730 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7jbf9j" event={"ID":"507ca360-5b3b-4cba-85a3-20ca526f88a7","Type":"ContainerDied","Data":"5517a9a5d6091aae884a3925c645a888793fdfbd1d19624ddbe865495aab412e"} Sep 10 13:39:41.224854 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:39:41.224765 2585 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="5517a9a5d6091aae884a3925c645a888793fdfbd1d19624ddbe865495aab412e" Sep 10 13:39:41.224854 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:39:41.224825 2585 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7jbf9j" Sep 10 13:39:43.758911 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:39:43.758875 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fxgmp_fa53c783-d40a-4024-9f7b-1d0d2b57f6dc/konnectivity-agent/0.log" limit=1024 Sep 10 13:39:44.586778 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:39:44.586742 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fxgmp_fa53c783-d40a-4024-9f7b-1d0d2b57f6dc/konnectivity-agent/0.log" limit=1024 Sep 10 13:39:45.344600 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:39:45.344572 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fxgmp_fa53c783-d40a-4024-9f7b-1d0d2b57f6dc/konnectivity-agent/0.log" limit=1024 Sep 10 13:39:47.608008 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:39:47.607969 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fxgmp_fa53c783-d40a-4024-9f7b-1d0d2b57f6dc/konnectivity-agent/0.log" limit=1024 Sep 10 13:39:47.699695 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:39:47.699655 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fxgmp_fa53c783-d40a-4024-9f7b-1d0d2b57f6dc/konnectivity-agent/0.log" limit=1024 Sep 10 13:39:48.575133 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:39:48.575102 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fxgmp_fa53c783-d40a-4024-9f7b-1d0d2b57f6dc/konnectivity-agent/0.log" limit=1024 Sep 10 13:39:48.965785 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:39:48.965706 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fxgmp_fa53c783-d40a-4024-9f7b-1d0d2b57f6dc/konnectivity-agent/0.log" limit=1024 Sep 10 13:39:49.204040 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:39:49.203994 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fxgmp_fa53c783-d40a-4024-9f7b-1d0d2b57f6dc/konnectivity-agent/0.log" limit=1024 Sep 10 13:39:49.304396 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:39:49.304315 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fxgmp_fa53c783-d40a-4024-9f7b-1d0d2b57f6dc/konnectivity-agent/0.log" limit=1024 Sep 10 13:39:51.557633 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:39:51.557589 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fxgmp_fa53c783-d40a-4024-9f7b-1d0d2b57f6dc/konnectivity-agent/0.log" limit=1024 Sep 10 13:39:51.692945 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:39:51.692910 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fxgmp_fa53c783-d40a-4024-9f7b-1d0d2b57f6dc/konnectivity-agent/0.log" limit=1024 Sep 10 13:39:53.784539 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:39:53.784506 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fxgmp_fa53c783-d40a-4024-9f7b-1d0d2b57f6dc/konnectivity-agent/0.log" limit=1024 Sep 10 13:39:53.989459 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:39:53.989429 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fxgmp_fa53c783-d40a-4024-9f7b-1d0d2b57f6dc/konnectivity-agent/0.log" limit=1024 Sep 10 13:39:56.333475 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:39:56.333412 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fxgmp_fa53c783-d40a-4024-9f7b-1d0d2b57f6dc/konnectivity-agent/0.log" limit=1024 Sep 10 13:39:56.615560 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:39:56.615478 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fxgmp_fa53c783-d40a-4024-9f7b-1d0d2b57f6dc/konnectivity-agent/0.log" limit=1024 Sep 10 13:39:58.753217 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:39:58.753185 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fxgmp_fa53c783-d40a-4024-9f7b-1d0d2b57f6dc/konnectivity-agent/0.log" limit=1024 Sep 10 13:39:58.822266 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:39:58.822214 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fxgmp_fa53c783-d40a-4024-9f7b-1d0d2b57f6dc/konnectivity-agent/0.log" limit=1024 Sep 10 13:39:59.140290 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:39:59.140231 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fxgmp_fa53c783-d40a-4024-9f7b-1d0d2b57f6dc/konnectivity-agent/0.log" limit=1024 Sep 10 13:39:59.251436 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:39:59.251402 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fxgmp_fa53c783-d40a-4024-9f7b-1d0d2b57f6dc/konnectivity-agent/0.log" limit=1024 Sep 10 13:39:59.532409 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:39:59.532334 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fxgmp_fa53c783-d40a-4024-9f7b-1d0d2b57f6dc/konnectivity-agent/0.log" limit=1024 Sep 10 13:39:59.640887 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:39:59.640851 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fxgmp_fa53c783-d40a-4024-9f7b-1d0d2b57f6dc/konnectivity-agent/0.log" limit=1024 Sep 10 13:40:07.375649 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:40:07.375603 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fxgmp_fa53c783-d40a-4024-9f7b-1d0d2b57f6dc/konnectivity-agent/0.log" limit=1024 Sep 10 13:40:07.673827 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:40:07.673742 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fxgmp_fa53c783-d40a-4024-9f7b-1d0d2b57f6dc/konnectivity-agent/0.log" limit=1024 Sep 10 13:40:07.780524 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:40:07.780492 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fxgmp_fa53c783-d40a-4024-9f7b-1d0d2b57f6dc/konnectivity-agent/0.log" limit=1024 Sep 10 13:40:08.891634 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:40:08.891599 2585 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-qz6k6"] Sep 10 13:40:08.892004 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:40:08.891932 2585 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ae377d6b-2529-422f-8618-fb5c4064ea16" containerName="pull" Sep 10 13:40:08.892004 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:40:08.891943 2585 state_mem.go:107] "Deleted CPUSet assignment" podUID="ae377d6b-2529-422f-8618-fb5c4064ea16" containerName="pull" Sep 10 13:40:08.892004 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:40:08.891951 2585 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="507ca360-5b3b-4cba-85a3-20ca526f88a7" containerName="pull" Sep 10 13:40:08.892004 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:40:08.891956 2585 state_mem.go:107] "Deleted CPUSet assignment" podUID="507ca360-5b3b-4cba-85a3-20ca526f88a7" containerName="pull" Sep 10 13:40:08.892004 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:40:08.891963 2585 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d24a1606-266b-4849-876b-ad73b9263515" containerName="extract" Sep 10 13:40:08.892004 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:40:08.891969 2585 state_mem.go:107] "Deleted CPUSet assignment" podUID="d24a1606-266b-4849-876b-ad73b9263515" containerName="extract" Sep 10 13:40:08.892004 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:40:08.891976 2585 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="507ca360-5b3b-4cba-85a3-20ca526f88a7" containerName="extract" Sep 10 13:40:08.892004 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:40:08.891981 2585 state_mem.go:107] "Deleted CPUSet assignment" podUID="507ca360-5b3b-4cba-85a3-20ca526f88a7" containerName="extract" Sep 10 13:40:08.892004 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:40:08.891992 2585 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1caeb857-b271-4ee5-af15-3e098e17d134" containerName="pull" Sep 10 13:40:08.892004 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:40:08.891996 2585 state_mem.go:107] "Deleted CPUSet assignment" podUID="1caeb857-b271-4ee5-af15-3e098e17d134" containerName="pull" Sep 10 13:40:08.892004 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:40:08.892003 2585 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1caeb857-b271-4ee5-af15-3e098e17d134" containerName="extract" Sep 10 13:40:08.892004 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:40:08.892008 2585 state_mem.go:107] "Deleted CPUSet assignment" podUID="1caeb857-b271-4ee5-af15-3e098e17d134" containerName="extract" Sep 10 13:40:08.892391 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:40:08.892017 2585 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d24a1606-266b-4849-876b-ad73b9263515" containerName="pull" Sep 10 13:40:08.892391 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:40:08.892021 2585 state_mem.go:107] "Deleted CPUSet assignment" podUID="d24a1606-266b-4849-876b-ad73b9263515" containerName="pull" Sep 10 13:40:08.892391 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:40:08.892028 2585 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="507ca360-5b3b-4cba-85a3-20ca526f88a7" containerName="util" Sep 10 13:40:08.892391 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:40:08.892033 2585 state_mem.go:107] "Deleted CPUSet assignment" podUID="507ca360-5b3b-4cba-85a3-20ca526f88a7" containerName="util" Sep 10 13:40:08.892391 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:40:08.892039 2585 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1caeb857-b271-4ee5-af15-3e098e17d134" containerName="util" Sep 10 13:40:08.892391 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:40:08.892044 2585 state_mem.go:107] "Deleted CPUSet assignment" podUID="1caeb857-b271-4ee5-af15-3e098e17d134" containerName="util" Sep 10 13:40:08.892391 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:40:08.892050 2585 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d24a1606-266b-4849-876b-ad73b9263515" containerName="util" Sep 10 13:40:08.892391 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:40:08.892054 2585 state_mem.go:107] "Deleted CPUSet assignment" podUID="d24a1606-266b-4849-876b-ad73b9263515" containerName="util" Sep 10 13:40:08.892391 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:40:08.892059 2585 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ae377d6b-2529-422f-8618-fb5c4064ea16" containerName="extract" Sep 10 13:40:08.892391 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:40:08.892064 2585 state_mem.go:107] "Deleted CPUSet assignment" podUID="ae377d6b-2529-422f-8618-fb5c4064ea16" containerName="extract" Sep 10 13:40:08.892391 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:40:08.892077 2585 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ae377d6b-2529-422f-8618-fb5c4064ea16" containerName="util" Sep 10 13:40:08.892391 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:40:08.892082 2585 state_mem.go:107] "Deleted CPUSet assignment" podUID="ae377d6b-2529-422f-8618-fb5c4064ea16" containerName="util" Sep 10 13:40:08.892391 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:40:08.892144 2585 memory_manager.go:368] "RemoveStaleState removing state" podUID="1caeb857-b271-4ee5-af15-3e098e17d134" containerName="extract" Sep 10 13:40:08.892391 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:40:08.892151 2585 memory_manager.go:368] "RemoveStaleState removing state" podUID="507ca360-5b3b-4cba-85a3-20ca526f88a7" containerName="extract" Sep 10 13:40:08.892391 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:40:08.892159 2585 memory_manager.go:368] "RemoveStaleState removing state" podUID="ae377d6b-2529-422f-8618-fb5c4064ea16" containerName="extract" Sep 10 13:40:08.892391 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:40:08.892165 2585 memory_manager.go:368] "RemoveStaleState removing state" podUID="d24a1606-266b-4849-876b-ad73b9263515" containerName="extract" Sep 10 13:40:08.895504 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:40:08.895487 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-qz6k6" Sep 10 13:40:08.898990 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:40:08.898968 2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"kuadrant-operator-controller-manager-dockercfg-lsk9v\"" Sep 10 13:40:08.899259 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:40:08.899227 2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"openshift-service-ca.crt\"" Sep 10 13:40:08.899559 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:40:08.899543 2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kube-root-ca.crt\"" Sep 10 13:40:08.913307 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:40:08.913281 2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-qz6k6"] Sep 10 13:40:09.008994 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:40:09.008962 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4wl7x\" (UniqueName: \"kubernetes.io/projected/d6fb0c5a-5f04-414f-b011-6704f8243b7a-kube-api-access-4wl7x\") pod \"kuadrant-operator-controller-manager-5769f99d9f-qz6k6\" (UID: \"d6fb0c5a-5f04-414f-b011-6704f8243b7a\") " pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-qz6k6" Sep 10 13:40:09.009162 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:40:09.009015 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/d6fb0c5a-5f04-414f-b011-6704f8243b7a-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-5769f99d9f-qz6k6\" (UID: \"d6fb0c5a-5f04-414f-b011-6704f8243b7a\") " pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-qz6k6" Sep 10 13:40:09.109855 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:40:09.109813 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4wl7x\" (UniqueName: \"kubernetes.io/projected/d6fb0c5a-5f04-414f-b011-6704f8243b7a-kube-api-access-4wl7x\") pod \"kuadrant-operator-controller-manager-5769f99d9f-qz6k6\" (UID: \"d6fb0c5a-5f04-414f-b011-6704f8243b7a\") " pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-qz6k6" Sep 10 13:40:09.110002 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:40:09.109868 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/d6fb0c5a-5f04-414f-b011-6704f8243b7a-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-5769f99d9f-qz6k6\" (UID: \"d6fb0c5a-5f04-414f-b011-6704f8243b7a\") " pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-qz6k6" Sep 10 13:40:09.110301 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:40:09.110278 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/d6fb0c5a-5f04-414f-b011-6704f8243b7a-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-5769f99d9f-qz6k6\" (UID: \"d6fb0c5a-5f04-414f-b011-6704f8243b7a\") " pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-qz6k6" Sep 10 13:40:09.138472 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:40:09.138443 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4wl7x\" (UniqueName: \"kubernetes.io/projected/d6fb0c5a-5f04-414f-b011-6704f8243b7a-kube-api-access-4wl7x\") pod \"kuadrant-operator-controller-manager-5769f99d9f-qz6k6\" (UID: \"d6fb0c5a-5f04-414f-b011-6704f8243b7a\") " pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-qz6k6" Sep 10 13:40:09.205664 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:40:09.205584 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-qz6k6" Sep 10 13:40:09.493012 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:40:09.492983 2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-qz6k6"] Sep 10 13:40:09.494674 ip-10-0-140-8 kubenswrapper[2585]: W0910 13:40:09.494647 2585 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd6fb0c5a_5f04_414f_b011_6704f8243b7a.slice/crio-5d22fbe7e2d6d03fc4be3f198ff681a7cf9724d6ca5ce8acd28e8b9a6e083f86 WatchSource:0}: Error finding container 5d22fbe7e2d6d03fc4be3f198ff681a7cf9724d6ca5ce8acd28e8b9a6e083f86: Status 404 returned error can't find the container with id 5d22fbe7e2d6d03fc4be3f198ff681a7cf9724d6ca5ce8acd28e8b9a6e083f86 Sep 10 13:40:10.367489 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:40:10.367450 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-qz6k6" event={"ID":"d6fb0c5a-5f04-414f-b011-6704f8243b7a","Type":"ContainerStarted","Data":"5d22fbe7e2d6d03fc4be3f198ff681a7cf9724d6ca5ce8acd28e8b9a6e083f86"} Sep 10 13:40:10.545211 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:40:10.545150 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fxgmp_fa53c783-d40a-4024-9f7b-1d0d2b57f6dc/konnectivity-agent/0.log" limit=1024 Sep 10 13:40:10.663547 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:40:10.663462 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fxgmp_fa53c783-d40a-4024-9f7b-1d0d2b57f6dc/konnectivity-agent/0.log" limit=1024 Sep 10 13:40:12.809724 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:40:12.809689 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fxgmp_fa53c783-d40a-4024-9f7b-1d0d2b57f6dc/konnectivity-agent/0.log" limit=1024 Sep 10 13:40:12.884782 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:40:12.884749 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fxgmp_fa53c783-d40a-4024-9f7b-1d0d2b57f6dc/konnectivity-agent/0.log" limit=1024 Sep 10 13:40:13.388326 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:40:13.388293 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-qz6k6" event={"ID":"d6fb0c5a-5f04-414f-b011-6704f8243b7a","Type":"ContainerStarted","Data":"a27718ec89d8a14dc1093a95b52be441ac7990ca28de0fec82350eee1d852558"} Sep 10 13:40:13.388516 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:40:13.388442 2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-qz6k6" Sep 10 13:40:13.416552 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:40:13.416459 2585 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-qz6k6" podStartSLOduration=1.748480815 podStartE2EDuration="5.416445546s" totalImagesPullingTime="3.667964731s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 13:40:08 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 13:40:13.413855098 +0000 UTC m=+698.286559342" watchObservedRunningTime="2026-09-10 13:40:13.416445546 +0000 UTC m=+698.289149746" Sep 10 13:40:19.751761 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:40:19.751726 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fxgmp_fa53c783-d40a-4024-9f7b-1d0d2b57f6dc/konnectivity-agent/0.log" limit=1024 Sep 10 13:40:19.852566 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:40:19.852535 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fxgmp_fa53c783-d40a-4024-9f7b-1d0d2b57f6dc/konnectivity-agent/0.log" limit=1024 Sep 10 13:40:20.172146 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:40:20.172110 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fxgmp_fa53c783-d40a-4024-9f7b-1d0d2b57f6dc/konnectivity-agent/0.log" limit=1024 Sep 10 13:40:20.273746 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:40:20.273708 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fxgmp_fa53c783-d40a-4024-9f7b-1d0d2b57f6dc/konnectivity-agent/0.log" limit=1024 Sep 10 13:40:20.398175 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:40:20.398122 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fxgmp_fa53c783-d40a-4024-9f7b-1d0d2b57f6dc/konnectivity-agent/0.log" limit=1024 Sep 10 13:40:20.497938 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:40:20.497849 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fxgmp_fa53c783-d40a-4024-9f7b-1d0d2b57f6dc/konnectivity-agent/0.log" limit=1024 Sep 10 13:40:24.394962 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:40:24.394931 2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-qz6k6" Sep 10 13:40:27.864470 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:40:27.864384 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fxgmp_fa53c783-d40a-4024-9f7b-1d0d2b57f6dc/konnectivity-agent/0.log" limit=1024 Sep 10 13:40:29.170803 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:40:29.170766 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fxgmp_fa53c783-d40a-4024-9f7b-1d0d2b57f6dc/konnectivity-agent/0.log" limit=1024 Sep 10 13:40:29.271459 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:40:29.271427 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fxgmp_fa53c783-d40a-4024-9f7b-1d0d2b57f6dc/konnectivity-agent/0.log" limit=1024 Sep 10 13:40:32.161265 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:40:32.161213 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fxgmp_fa53c783-d40a-4024-9f7b-1d0d2b57f6dc/konnectivity-agent/0.log" limit=1024 Sep 10 13:40:32.264502 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:40:32.264472 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fxgmp_fa53c783-d40a-4024-9f7b-1d0d2b57f6dc/konnectivity-agent/0.log" limit=1024 Sep 10 13:40:34.662226 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:40:34.662194 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fxgmp_fa53c783-d40a-4024-9f7b-1d0d2b57f6dc/konnectivity-agent/0.log" limit=1024 Sep 10 13:40:34.774424 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:40:34.774392 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fxgmp_fa53c783-d40a-4024-9f7b-1d0d2b57f6dc/konnectivity-agent/0.log" limit=1024 Sep 10 13:40:41.178010 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:40:41.177979 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fxgmp_fa53c783-d40a-4024-9f7b-1d0d2b57f6dc/konnectivity-agent/0.log" limit=1024 Sep 10 13:40:41.412364 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:40:41.412330 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fxgmp_fa53c783-d40a-4024-9f7b-1d0d2b57f6dc/konnectivity-agent/0.log" limit=1024 Sep 10 13:40:45.819890 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:40:45.819857 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fxgmp_fa53c783-d40a-4024-9f7b-1d0d2b57f6dc/konnectivity-agent/0.log" limit=1024 Sep 10 13:40:45.926869 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:40:45.926830 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fxgmp_fa53c783-d40a-4024-9f7b-1d0d2b57f6dc/konnectivity-agent/0.log" limit=1024 Sep 10 13:40:46.028396 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:40:46.028363 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fxgmp_fa53c783-d40a-4024-9f7b-1d0d2b57f6dc/konnectivity-agent/0.log" limit=1024 Sep 10 13:40:46.711209 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:40:46.711175 2585 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-7bcfd8f9cd-md5jb"] Sep 10 13:40:46.715743 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:40:46.715721 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-7bcfd8f9cd-md5jb" Sep 10 13:40:46.718956 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:40:46.718935 2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-authorino-dockercfg-6r98w\"" Sep 10 13:40:46.728487 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:40:46.728465 2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-7bcfd8f9cd-md5jb"] Sep 10 13:40:46.730855 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:40:46.730824 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mth8p\" (UniqueName: \"kubernetes.io/projected/16f4182e-a88d-4c9d-84bc-1f07dcc9db18-kube-api-access-mth8p\") pod \"authorino-7bcfd8f9cd-md5jb\" (UID: \"16f4182e-a88d-4c9d-84bc-1f07dcc9db18\") " pod="kuadrant-system/authorino-7bcfd8f9cd-md5jb" Sep 10 13:40:46.769820 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:40:46.769753 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fxgmp_fa53c783-d40a-4024-9f7b-1d0d2b57f6dc/konnectivity-agent/0.log" limit=1024 Sep 10 13:40:46.831609 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:40:46.831560 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mth8p\" (UniqueName: \"kubernetes.io/projected/16f4182e-a88d-4c9d-84bc-1f07dcc9db18-kube-api-access-mth8p\") pod \"authorino-7bcfd8f9cd-md5jb\" (UID: \"16f4182e-a88d-4c9d-84bc-1f07dcc9db18\") " pod="kuadrant-system/authorino-7bcfd8f9cd-md5jb" Sep 10 13:40:46.862049 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:40:46.862020 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-mth8p\" (UniqueName: \"kubernetes.io/projected/16f4182e-a88d-4c9d-84bc-1f07dcc9db18-kube-api-access-mth8p\") pod \"authorino-7bcfd8f9cd-md5jb\" (UID: \"16f4182e-a88d-4c9d-84bc-1f07dcc9db18\") " pod="kuadrant-system/authorino-7bcfd8f9cd-md5jb" Sep 10 13:40:46.892369 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:40:46.892337 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fxgmp_fa53c783-d40a-4024-9f7b-1d0d2b57f6dc/konnectivity-agent/0.log" limit=1024 Sep 10 13:40:47.025001 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:40:47.024914 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-7bcfd8f9cd-md5jb" Sep 10 13:40:47.064193 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:40:47.064157 2585 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-59c974bd48-hpbjm"] Sep 10 13:40:47.066610 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:40:47.066583 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-59c974bd48-hpbjm" Sep 10 13:40:47.140585 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:40:47.140549 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7jpmk\" (UniqueName: \"kubernetes.io/projected/be9446b9-962c-491f-ab18-0ea4d7a999d6-kube-api-access-7jpmk\") pod \"authorino-59c974bd48-hpbjm\" (UID: \"be9446b9-962c-491f-ab18-0ea4d7a999d6\") " pod="kuadrant-system/authorino-59c974bd48-hpbjm" Sep 10 13:40:47.159590 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:40:47.159553 2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-59c974bd48-hpbjm"] Sep 10 13:40:47.241507 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:40:47.241454 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7jpmk\" (UniqueName: \"kubernetes.io/projected/be9446b9-962c-491f-ab18-0ea4d7a999d6-kube-api-access-7jpmk\") pod \"authorino-59c974bd48-hpbjm\" (UID: \"be9446b9-962c-491f-ab18-0ea4d7a999d6\") " pod="kuadrant-system/authorino-59c974bd48-hpbjm" Sep 10 13:40:47.257966 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:40:47.257936 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7jpmk\" (UniqueName: \"kubernetes.io/projected/be9446b9-962c-491f-ab18-0ea4d7a999d6-kube-api-access-7jpmk\") pod \"authorino-59c974bd48-hpbjm\" (UID: \"be9446b9-962c-491f-ab18-0ea4d7a999d6\") " pod="kuadrant-system/authorino-59c974bd48-hpbjm" Sep 10 13:40:47.305068 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:40:47.305044 2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-7bcfd8f9cd-md5jb"] Sep 10 13:40:47.306774 ip-10-0-140-8 kubenswrapper[2585]: W0910 13:40:47.306741 2585 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod16f4182e_a88d_4c9d_84bc_1f07dcc9db18.slice/crio-bc51eae4538e0dabe07bbf502e59746cedc94dd03521aa0dfdae87f26a91a6af WatchSource:0}: Error finding container bc51eae4538e0dabe07bbf502e59746cedc94dd03521aa0dfdae87f26a91a6af: Status 404 returned error can't find the container with id bc51eae4538e0dabe07bbf502e59746cedc94dd03521aa0dfdae87f26a91a6af Sep 10 13:40:47.379134 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:40:47.379102 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-59c974bd48-hpbjm" Sep 10 13:40:47.553785 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:40:47.553705 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-7bcfd8f9cd-md5jb" event={"ID":"16f4182e-a88d-4c9d-84bc-1f07dcc9db18","Type":"ContainerStarted","Data":"bc51eae4538e0dabe07bbf502e59746cedc94dd03521aa0dfdae87f26a91a6af"} Sep 10 13:40:47.648621 ip-10-0-140-8 kubenswrapper[2585]: W0910 13:40:47.648590 2585 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podbe9446b9_962c_491f_ab18_0ea4d7a999d6.slice/crio-f1007ca9046f68057ff080bd1710a55ed2c0871e4d6bac3f5d84c77b547a1af7 WatchSource:0}: Error finding container f1007ca9046f68057ff080bd1710a55ed2c0871e4d6bac3f5d84c77b547a1af7: Status 404 returned error can't find the container with id f1007ca9046f68057ff080bd1710a55ed2c0871e4d6bac3f5d84c77b547a1af7 Sep 10 13:40:47.649029 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:40:47.649007 2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-59c974bd48-hpbjm"] Sep 10 13:40:48.561798 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:40:48.561762 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-59c974bd48-hpbjm" event={"ID":"be9446b9-962c-491f-ab18-0ea4d7a999d6","Type":"ContainerStarted","Data":"f1007ca9046f68057ff080bd1710a55ed2c0871e4d6bac3f5d84c77b547a1af7"} Sep 10 13:40:49.568704 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:40:49.568668 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-59c974bd48-hpbjm" event={"ID":"be9446b9-962c-491f-ab18-0ea4d7a999d6","Type":"ContainerStarted","Data":"2854107aab2ef7d5589326ee1fbaa2b19c2de62b5e23e844ce8bb17d0ec7f35d"} Sep 10 13:40:49.570166 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:40:49.570144 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-7bcfd8f9cd-md5jb" event={"ID":"16f4182e-a88d-4c9d-84bc-1f07dcc9db18","Type":"ContainerStarted","Data":"56acb1b2427080458e81296ed66721a81187d4155e5a9bbcd55e4a8da4b2cd4e"} Sep 10 13:40:49.590077 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:40:49.590031 2585 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/authorino-59c974bd48-hpbjm" podStartSLOduration=0.937069258 podStartE2EDuration="2.590016436s" totalImagesPullingTime="1.652947178s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 13:40:47 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 13:40:49.588291078 +0000 UTC m=+734.460995279" watchObservedRunningTime="2026-09-10 13:40:49.590016436 +0000 UTC m=+734.462720683" Sep 10 13:40:49.606748 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:40:49.606705 2585 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/authorino-7bcfd8f9cd-md5jb" podStartSLOduration=1.607757431 podStartE2EDuration="3.606689657s" totalImagesPullingTime="1.998932226s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 13:40:46 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 13:40:49.606283851 +0000 UTC m=+734.478988078" watchObservedRunningTime="2026-09-10 13:40:49.606689657 +0000 UTC m=+734.479393895" Sep 10 13:40:49.646857 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:40:49.646826 2585 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-7bcfd8f9cd-md5jb"] Sep 10 13:40:51.580194 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:40:51.580152 2585 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/authorino-7bcfd8f9cd-md5jb" podUID="16f4182e-a88d-4c9d-84bc-1f07dcc9db18" containerName="authorino" containerID="cri-o://56acb1b2427080458e81296ed66721a81187d4155e5a9bbcd55e4a8da4b2cd4e" gracePeriod=30 Sep 10 13:40:51.818859 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:40:51.818837 2585 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-7bcfd8f9cd-md5jb" Sep 10 13:40:51.881731 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:40:51.881652 2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-mth8p\" (UniqueName: \"kubernetes.io/projected/16f4182e-a88d-4c9d-84bc-1f07dcc9db18-kube-api-access-mth8p\") pod \"16f4182e-a88d-4c9d-84bc-1f07dcc9db18\" (UID: \"16f4182e-a88d-4c9d-84bc-1f07dcc9db18\") " Sep 10 13:40:51.883754 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:40:51.883732 2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/16f4182e-a88d-4c9d-84bc-1f07dcc9db18-kube-api-access-mth8p" (OuterVolumeSpecName: "kube-api-access-mth8p") pod "16f4182e-a88d-4c9d-84bc-1f07dcc9db18" (UID: "16f4182e-a88d-4c9d-84bc-1f07dcc9db18"). InnerVolumeSpecName "kube-api-access-mth8p". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 10 13:40:51.983165 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:40:51.983131 2585 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mth8p\" (UniqueName: \"kubernetes.io/projected/16f4182e-a88d-4c9d-84bc-1f07dcc9db18-kube-api-access-mth8p\") on node \"ip-10-0-140-8.ec2.internal\" DevicePath \"\"" Sep 10 13:40:52.586417 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:40:52.586378 2585 generic.go:356] "Generic (PLEG): container finished" podID="16f4182e-a88d-4c9d-84bc-1f07dcc9db18" containerID="56acb1b2427080458e81296ed66721a81187d4155e5a9bbcd55e4a8da4b2cd4e" exitCode=0 Sep 10 13:40:52.586417 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:40:52.586421 2585 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-7bcfd8f9cd-md5jb" Sep 10 13:40:52.586891 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:40:52.586441 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-7bcfd8f9cd-md5jb" event={"ID":"16f4182e-a88d-4c9d-84bc-1f07dcc9db18","Type":"ContainerDied","Data":"56acb1b2427080458e81296ed66721a81187d4155e5a9bbcd55e4a8da4b2cd4e"} Sep 10 13:40:52.586891 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:40:52.586478 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-7bcfd8f9cd-md5jb" event={"ID":"16f4182e-a88d-4c9d-84bc-1f07dcc9db18","Type":"ContainerDied","Data":"bc51eae4538e0dabe07bbf502e59746cedc94dd03521aa0dfdae87f26a91a6af"} Sep 10 13:40:52.586891 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:40:52.586500 2585 scope.go:117] "RemoveContainer" containerID="56acb1b2427080458e81296ed66721a81187d4155e5a9bbcd55e4a8da4b2cd4e" Sep 10 13:40:52.595115 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:40:52.595099 2585 scope.go:117] "RemoveContainer" containerID="56acb1b2427080458e81296ed66721a81187d4155e5a9bbcd55e4a8da4b2cd4e" Sep 10 13:40:52.595404 ip-10-0-140-8 kubenswrapper[2585]: E0910 13:40:52.595382 2585 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"56acb1b2427080458e81296ed66721a81187d4155e5a9bbcd55e4a8da4b2cd4e\": container with ID starting with 56acb1b2427080458e81296ed66721a81187d4155e5a9bbcd55e4a8da4b2cd4e not found: ID does not exist" containerID="56acb1b2427080458e81296ed66721a81187d4155e5a9bbcd55e4a8da4b2cd4e" Sep 10 13:40:52.595484 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:40:52.595411 2585 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"56acb1b2427080458e81296ed66721a81187d4155e5a9bbcd55e4a8da4b2cd4e"} err="failed to get container status \"56acb1b2427080458e81296ed66721a81187d4155e5a9bbcd55e4a8da4b2cd4e\": rpc error: code = NotFound desc = could not find container \"56acb1b2427080458e81296ed66721a81187d4155e5a9bbcd55e4a8da4b2cd4e\": container with ID starting with 56acb1b2427080458e81296ed66721a81187d4155e5a9bbcd55e4a8da4b2cd4e not found: ID does not exist" Sep 10 13:40:52.608779 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:40:52.608759 2585 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-7bcfd8f9cd-md5jb"] Sep 10 13:40:52.613388 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:40:52.613370 2585 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-7bcfd8f9cd-md5jb"] Sep 10 13:40:53.647614 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:40:53.647585 2585 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="16f4182e-a88d-4c9d-84bc-1f07dcc9db18" path="/var/lib/kubelet/pods/16f4182e-a88d-4c9d-84bc-1f07dcc9db18/volumes" Sep 10 13:40:54.959691 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:40:54.959659 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fxgmp_fa53c783-d40a-4024-9f7b-1d0d2b57f6dc/konnectivity-agent/0.log" limit=1024 Sep 10 13:40:55.027307 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:40:55.027278 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fxgmp_fa53c783-d40a-4024-9f7b-1d0d2b57f6dc/konnectivity-agent/0.log" limit=1024 Sep 10 13:41:02.068858 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:41:02.068829 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fxgmp_fa53c783-d40a-4024-9f7b-1d0d2b57f6dc/konnectivity-agent/0.log" limit=1024 Sep 10 13:41:02.150359 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:41:02.150322 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fxgmp_fa53c783-d40a-4024-9f7b-1d0d2b57f6dc/konnectivity-agent/0.log" limit=1024 Sep 10 13:41:02.630488 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:41:02.630455 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fxgmp_fa53c783-d40a-4024-9f7b-1d0d2b57f6dc/konnectivity-agent/0.log" limit=1024 Sep 10 13:41:12.197546 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:41:12.197494 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fxgmp_fa53c783-d40a-4024-9f7b-1d0d2b57f6dc/konnectivity-agent/0.log" limit=1024 Sep 10 13:41:12.432949 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:41:12.432916 2585 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-557f876f69-drlpt"] Sep 10 13:41:12.433291 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:41:12.433278 2585 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="16f4182e-a88d-4c9d-84bc-1f07dcc9db18" containerName="authorino" Sep 10 13:41:12.433347 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:41:12.433293 2585 state_mem.go:107] "Deleted CPUSet assignment" podUID="16f4182e-a88d-4c9d-84bc-1f07dcc9db18" containerName="authorino" Sep 10 13:41:12.433387 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:41:12.433368 2585 memory_manager.go:368] "RemoveStaleState removing state" podUID="16f4182e-a88d-4c9d-84bc-1f07dcc9db18" containerName="authorino" Sep 10 13:41:12.436019 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:41:12.436002 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-557f876f69-drlpt" Sep 10 13:41:12.445942 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:41:12.445912 2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-557f876f69-drlpt"] Sep 10 13:41:12.548664 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:41:12.548584 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fxgmp_fa53c783-d40a-4024-9f7b-1d0d2b57f6dc/konnectivity-agent/0.log" limit=1024 Sep 10 13:41:12.566765 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:41:12.566735 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gdv58\" (UniqueName: \"kubernetes.io/projected/8759eee7-20fb-4535-852e-a66dccb1f675-kube-api-access-gdv58\") pod \"authorino-557f876f69-drlpt\" (UID: \"8759eee7-20fb-4535-852e-a66dccb1f675\") " pod="kuadrant-system/authorino-557f876f69-drlpt" Sep 10 13:41:12.587769 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:41:12.587743 2585 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-557f876f69-drlpt"] Sep 10 13:41:12.588008 ip-10-0-140-8 kubenswrapper[2585]: E0910 13:41:12.587990 2585 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[kube-api-access-gdv58], unattached volumes=[], failed to process volumes=[]: context canceled" pod="kuadrant-system/authorino-557f876f69-drlpt" podUID="8759eee7-20fb-4535-852e-a66dccb1f675" Sep 10 13:41:12.668024 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:41:12.667988 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gdv58\" (UniqueName: \"kubernetes.io/projected/8759eee7-20fb-4535-852e-a66dccb1f675-kube-api-access-gdv58\") pod \"authorino-557f876f69-drlpt\" (UID: \"8759eee7-20fb-4535-852e-a66dccb1f675\") " pod="kuadrant-system/authorino-557f876f69-drlpt" Sep 10 13:41:12.678371 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:41:12.678341 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-gdv58\" (UniqueName: \"kubernetes.io/projected/8759eee7-20fb-4535-852e-a66dccb1f675-kube-api-access-gdv58\") pod \"authorino-557f876f69-drlpt\" (UID: \"8759eee7-20fb-4535-852e-a66dccb1f675\") " pod="kuadrant-system/authorino-557f876f69-drlpt" Sep 10 13:41:12.685306 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:41:12.685280 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-557f876f69-drlpt" Sep 10 13:41:12.692188 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:41:12.692168 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-557f876f69-drlpt" Sep 10 13:41:12.752153 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:41:12.752110 2585 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-6b7cb4c856-crbnk"] Sep 10 13:41:12.755569 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:41:12.755545 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-6b7cb4c856-crbnk" Sep 10 13:41:12.765706 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:41:12.765681 2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-server-cert\"" Sep 10 13:41:12.768998 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:41:12.768981 2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-gdv58\" (UniqueName: \"kubernetes.io/projected/8759eee7-20fb-4535-852e-a66dccb1f675-kube-api-access-gdv58\") pod \"8759eee7-20fb-4535-852e-a66dccb1f675\" (UID: \"8759eee7-20fb-4535-852e-a66dccb1f675\") " Sep 10 13:41:12.770991 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:41:12.770969 2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8759eee7-20fb-4535-852e-a66dccb1f675-kube-api-access-gdv58" (OuterVolumeSpecName: "kube-api-access-gdv58") pod "8759eee7-20fb-4535-852e-a66dccb1f675" (UID: "8759eee7-20fb-4535-852e-a66dccb1f675"). InnerVolumeSpecName "kube-api-access-gdv58". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 10 13:41:12.782453 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:41:12.782429 2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-6b7cb4c856-crbnk"] Sep 10 13:41:12.870568 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:41:12.870527 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/261a0443-b54f-4416-ab5e-7337bbcf0d6c-tls-cert\") pod \"authorino-6b7cb4c856-crbnk\" (UID: \"261a0443-b54f-4416-ab5e-7337bbcf0d6c\") " pod="kuadrant-system/authorino-6b7cb4c856-crbnk" Sep 10 13:41:12.870743 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:41:12.870641 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4sbw6\" (UniqueName: \"kubernetes.io/projected/261a0443-b54f-4416-ab5e-7337bbcf0d6c-kube-api-access-4sbw6\") pod \"authorino-6b7cb4c856-crbnk\" (UID: \"261a0443-b54f-4416-ab5e-7337bbcf0d6c\") " pod="kuadrant-system/authorino-6b7cb4c856-crbnk" Sep 10 13:41:12.870743 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:41:12.870722 2585 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gdv58\" (UniqueName: \"kubernetes.io/projected/8759eee7-20fb-4535-852e-a66dccb1f675-kube-api-access-gdv58\") on node \"ip-10-0-140-8.ec2.internal\" DevicePath \"\"" Sep 10 13:41:12.971864 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:41:12.971829 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4sbw6\" (UniqueName: \"kubernetes.io/projected/261a0443-b54f-4416-ab5e-7337bbcf0d6c-kube-api-access-4sbw6\") pod \"authorino-6b7cb4c856-crbnk\" (UID: \"261a0443-b54f-4416-ab5e-7337bbcf0d6c\") " pod="kuadrant-system/authorino-6b7cb4c856-crbnk" Sep 10 13:41:12.972050 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:41:12.971973 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/261a0443-b54f-4416-ab5e-7337bbcf0d6c-tls-cert\") pod \"authorino-6b7cb4c856-crbnk\" (UID: \"261a0443-b54f-4416-ab5e-7337bbcf0d6c\") " pod="kuadrant-system/authorino-6b7cb4c856-crbnk" Sep 10 13:41:12.974112 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:41:12.974093 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/261a0443-b54f-4416-ab5e-7337bbcf0d6c-tls-cert\") pod \"authorino-6b7cb4c856-crbnk\" (UID: \"261a0443-b54f-4416-ab5e-7337bbcf0d6c\") " pod="kuadrant-system/authorino-6b7cb4c856-crbnk" Sep 10 13:41:12.982757 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:41:12.982731 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4sbw6\" (UniqueName: \"kubernetes.io/projected/261a0443-b54f-4416-ab5e-7337bbcf0d6c-kube-api-access-4sbw6\") pod \"authorino-6b7cb4c856-crbnk\" (UID: \"261a0443-b54f-4416-ab5e-7337bbcf0d6c\") " pod="kuadrant-system/authorino-6b7cb4c856-crbnk" Sep 10 13:41:13.064077 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:41:13.064044 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-6b7cb4c856-crbnk" Sep 10 13:41:13.349405 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:41:13.349377 2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-6b7cb4c856-crbnk"] Sep 10 13:41:13.351220 ip-10-0-140-8 kubenswrapper[2585]: W0910 13:41:13.351195 2585 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod261a0443_b54f_4416_ab5e_7337bbcf0d6c.slice/crio-cf80936647756f8f2c89d39227d06d0cf6460d48f2834348054a2e8b464cec8b WatchSource:0}: Error finding container cf80936647756f8f2c89d39227d06d0cf6460d48f2834348054a2e8b464cec8b: Status 404 returned error can't find the container with id cf80936647756f8f2c89d39227d06d0cf6460d48f2834348054a2e8b464cec8b Sep 10 13:41:13.691590 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:41:13.691555 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-6b7cb4c856-crbnk" event={"ID":"261a0443-b54f-4416-ab5e-7337bbcf0d6c","Type":"ContainerStarted","Data":"cf80936647756f8f2c89d39227d06d0cf6460d48f2834348054a2e8b464cec8b"} Sep 10 13:41:13.691590 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:41:13.691590 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-557f876f69-drlpt" Sep 10 13:41:13.735635 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:41:13.735602 2585 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-557f876f69-drlpt"] Sep 10 13:41:13.738467 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:41:13.738443 2585 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-557f876f69-drlpt"] Sep 10 13:41:14.698737 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:41:14.698702 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-6b7cb4c856-crbnk" event={"ID":"261a0443-b54f-4416-ab5e-7337bbcf0d6c","Type":"ContainerStarted","Data":"a3984b53af511f3b82559cc835b51a6d5c1f657cd36598e5d957b6060e10111a"} Sep 10 13:41:14.717996 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:41:14.717954 2585 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/authorino-6b7cb4c856-crbnk" podStartSLOduration=2.219201648 podStartE2EDuration="2.717938863s" totalImagesPullingTime="498.737215ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 13:41:12 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 13:41:14.716153588 +0000 UTC m=+759.588857793" watchObservedRunningTime="2026-09-10 13:41:14.717938863 +0000 UTC m=+759.590643064" Sep 10 13:41:14.749523 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:41:14.749491 2585 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-59c974bd48-hpbjm"] Sep 10 13:41:14.749685 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:41:14.749667 2585 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/authorino-59c974bd48-hpbjm" podUID="be9446b9-962c-491f-ab18-0ea4d7a999d6" containerName="authorino" containerID="cri-o://2854107aab2ef7d5589326ee1fbaa2b19c2de62b5e23e844ce8bb17d0ec7f35d" gracePeriod=30 Sep 10 13:41:15.004161 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:41:15.004134 2585 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-59c974bd48-hpbjm" Sep 10 13:41:15.090168 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:41:15.090137 2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-7jpmk\" (UniqueName: \"kubernetes.io/projected/be9446b9-962c-491f-ab18-0ea4d7a999d6-kube-api-access-7jpmk\") pod \"be9446b9-962c-491f-ab18-0ea4d7a999d6\" (UID: \"be9446b9-962c-491f-ab18-0ea4d7a999d6\") " Sep 10 13:41:15.092156 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:41:15.092131 2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/be9446b9-962c-491f-ab18-0ea4d7a999d6-kube-api-access-7jpmk" (OuterVolumeSpecName: "kube-api-access-7jpmk") pod "be9446b9-962c-491f-ab18-0ea4d7a999d6" (UID: "be9446b9-962c-491f-ab18-0ea4d7a999d6"). InnerVolumeSpecName "kube-api-access-7jpmk". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 10 13:41:15.191185 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:41:15.191153 2585 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7jpmk\" (UniqueName: \"kubernetes.io/projected/be9446b9-962c-491f-ab18-0ea4d7a999d6-kube-api-access-7jpmk\") on node \"ip-10-0-140-8.ec2.internal\" DevicePath \"\"" Sep 10 13:41:15.647507 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:41:15.647475 2585 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="8759eee7-20fb-4535-852e-a66dccb1f675" path="/var/lib/kubelet/pods/8759eee7-20fb-4535-852e-a66dccb1f675/volumes" Sep 10 13:41:15.705193 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:41:15.705160 2585 generic.go:356] "Generic (PLEG): container finished" podID="be9446b9-962c-491f-ab18-0ea4d7a999d6" containerID="2854107aab2ef7d5589326ee1fbaa2b19c2de62b5e23e844ce8bb17d0ec7f35d" exitCode=0 Sep 10 13:41:15.705574 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:41:15.705213 2585 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-59c974bd48-hpbjm" Sep 10 13:41:15.705574 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:41:15.705258 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-59c974bd48-hpbjm" event={"ID":"be9446b9-962c-491f-ab18-0ea4d7a999d6","Type":"ContainerDied","Data":"2854107aab2ef7d5589326ee1fbaa2b19c2de62b5e23e844ce8bb17d0ec7f35d"} Sep 10 13:41:15.705574 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:41:15.705303 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-59c974bd48-hpbjm" event={"ID":"be9446b9-962c-491f-ab18-0ea4d7a999d6","Type":"ContainerDied","Data":"f1007ca9046f68057ff080bd1710a55ed2c0871e4d6bac3f5d84c77b547a1af7"} Sep 10 13:41:15.705574 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:41:15.705319 2585 scope.go:117] "RemoveContainer" containerID="2854107aab2ef7d5589326ee1fbaa2b19c2de62b5e23e844ce8bb17d0ec7f35d" Sep 10 13:41:15.713906 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:41:15.713887 2585 scope.go:117] "RemoveContainer" containerID="2854107aab2ef7d5589326ee1fbaa2b19c2de62b5e23e844ce8bb17d0ec7f35d" Sep 10 13:41:15.714186 ip-10-0-140-8 kubenswrapper[2585]: E0910 13:41:15.714166 2585 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2854107aab2ef7d5589326ee1fbaa2b19c2de62b5e23e844ce8bb17d0ec7f35d\": container with ID starting with 2854107aab2ef7d5589326ee1fbaa2b19c2de62b5e23e844ce8bb17d0ec7f35d not found: ID does not exist" containerID="2854107aab2ef7d5589326ee1fbaa2b19c2de62b5e23e844ce8bb17d0ec7f35d" Sep 10 13:41:15.714236 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:41:15.714194 2585 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2854107aab2ef7d5589326ee1fbaa2b19c2de62b5e23e844ce8bb17d0ec7f35d"} err="failed to get container status \"2854107aab2ef7d5589326ee1fbaa2b19c2de62b5e23e844ce8bb17d0ec7f35d\": rpc error: code = NotFound desc = could not find container \"2854107aab2ef7d5589326ee1fbaa2b19c2de62b5e23e844ce8bb17d0ec7f35d\": container with ID starting with 2854107aab2ef7d5589326ee1fbaa2b19c2de62b5e23e844ce8bb17d0ec7f35d not found: ID does not exist" Sep 10 13:41:15.726411 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:41:15.726382 2585 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-59c974bd48-hpbjm"] Sep 10 13:41:15.729296 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:41:15.729275 2585 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-59c974bd48-hpbjm"] Sep 10 13:41:17.646810 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:41:17.646780 2585 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="be9446b9-962c-491f-ab18-0ea4d7a999d6" path="/var/lib/kubelet/pods/be9446b9-962c-491f-ab18-0ea4d7a999d6/volumes" Sep 10 13:41:20.140513 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:41:20.140474 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fxgmp_fa53c783-d40a-4024-9f7b-1d0d2b57f6dc/konnectivity-agent/0.log" limit=1024 Sep 10 13:41:20.264960 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:41:20.264931 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fxgmp_fa53c783-d40a-4024-9f7b-1d0d2b57f6dc/konnectivity-agent/0.log" limit=1024 Sep 10 13:41:20.364403 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:41:20.364362 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fxgmp_fa53c783-d40a-4024-9f7b-1d0d2b57f6dc/konnectivity-agent/0.log" limit=1024 Sep 10 13:41:20.452108 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:41:20.452021 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fxgmp_fa53c783-d40a-4024-9f7b-1d0d2b57f6dc/konnectivity-agent/0.log" limit=1024 Sep 10 13:41:20.453042 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:41:20.453022 2585 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/maas-controller-69bcf98bbb-vbmx7"] Sep 10 13:41:20.453443 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:41:20.453426 2585 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="be9446b9-962c-491f-ab18-0ea4d7a999d6" containerName="authorino" Sep 10 13:41:20.453520 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:41:20.453445 2585 state_mem.go:107] "Deleted CPUSet assignment" podUID="be9446b9-962c-491f-ab18-0ea4d7a999d6" containerName="authorino" Sep 10 13:41:20.453588 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:41:20.453564 2585 memory_manager.go:368] "RemoveStaleState removing state" podUID="be9446b9-962c-491f-ab18-0ea4d7a999d6" containerName="authorino" Sep 10 13:41:20.457750 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:41:20.457729 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-69bcf98bbb-vbmx7" Sep 10 13:41:20.463578 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:41:20.463555 2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-dockercfg-nn6pf\"" Sep 10 13:41:20.463699 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:41:20.463555 2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-metrics-tls\"" Sep 10 13:41:20.463792 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:41:20.463774 2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"maas-parameters\"" Sep 10 13:41:20.464013 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:41:20.463992 2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-webhook-cert\"" Sep 10 13:41:20.476239 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:41:20.476213 2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-69bcf98bbb-vbmx7"] Sep 10 13:41:20.572961 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:41:20.572932 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fxgmp_fa53c783-d40a-4024-9f7b-1d0d2b57f6dc/konnectivity-agent/0.log" limit=1024 Sep 10 13:41:20.639622 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:41:20.639594 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/111c86ef-d8e5-4460-b7c9-e7d3cf1a044f-webhook-cert\") pod \"maas-controller-69bcf98bbb-vbmx7\" (UID: \"111c86ef-d8e5-4460-b7c9-e7d3cf1a044f\") " pod="opendatahub/maas-controller-69bcf98bbb-vbmx7" Sep 10 13:41:20.639768 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:41:20.639655 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-t8ckn\" (UniqueName: \"kubernetes.io/projected/111c86ef-d8e5-4460-b7c9-e7d3cf1a044f-kube-api-access-t8ckn\") pod \"maas-controller-69bcf98bbb-vbmx7\" (UID: \"111c86ef-d8e5-4460-b7c9-e7d3cf1a044f\") " pod="opendatahub/maas-controller-69bcf98bbb-vbmx7" Sep 10 13:41:20.639768 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:41:20.639713 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/111c86ef-d8e5-4460-b7c9-e7d3cf1a044f-metrics-certs\") pod \"maas-controller-69bcf98bbb-vbmx7\" (UID: \"111c86ef-d8e5-4460-b7c9-e7d3cf1a044f\") " pod="opendatahub/maas-controller-69bcf98bbb-vbmx7" Sep 10 13:41:20.741103 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:41:20.741019 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/111c86ef-d8e5-4460-b7c9-e7d3cf1a044f-webhook-cert\") pod \"maas-controller-69bcf98bbb-vbmx7\" (UID: \"111c86ef-d8e5-4460-b7c9-e7d3cf1a044f\") " pod="opendatahub/maas-controller-69bcf98bbb-vbmx7" Sep 10 13:41:20.741103 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:41:20.741080 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t8ckn\" (UniqueName: \"kubernetes.io/projected/111c86ef-d8e5-4460-b7c9-e7d3cf1a044f-kube-api-access-t8ckn\") pod \"maas-controller-69bcf98bbb-vbmx7\" (UID: \"111c86ef-d8e5-4460-b7c9-e7d3cf1a044f\") " pod="opendatahub/maas-controller-69bcf98bbb-vbmx7" Sep 10 13:41:20.741331 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:41:20.741190 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/111c86ef-d8e5-4460-b7c9-e7d3cf1a044f-metrics-certs\") pod \"maas-controller-69bcf98bbb-vbmx7\" (UID: \"111c86ef-d8e5-4460-b7c9-e7d3cf1a044f\") " pod="opendatahub/maas-controller-69bcf98bbb-vbmx7" Sep 10 13:41:20.743455 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:41:20.743431 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/111c86ef-d8e5-4460-b7c9-e7d3cf1a044f-webhook-cert\") pod \"maas-controller-69bcf98bbb-vbmx7\" (UID: \"111c86ef-d8e5-4460-b7c9-e7d3cf1a044f\") " pod="opendatahub/maas-controller-69bcf98bbb-vbmx7" Sep 10 13:41:20.743570 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:41:20.743457 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/111c86ef-d8e5-4460-b7c9-e7d3cf1a044f-metrics-certs\") pod \"maas-controller-69bcf98bbb-vbmx7\" (UID: \"111c86ef-d8e5-4460-b7c9-e7d3cf1a044f\") " pod="opendatahub/maas-controller-69bcf98bbb-vbmx7" Sep 10 13:41:20.752579 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:41:20.752553 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-t8ckn\" (UniqueName: \"kubernetes.io/projected/111c86ef-d8e5-4460-b7c9-e7d3cf1a044f-kube-api-access-t8ckn\") pod \"maas-controller-69bcf98bbb-vbmx7\" (UID: \"111c86ef-d8e5-4460-b7c9-e7d3cf1a044f\") " pod="opendatahub/maas-controller-69bcf98bbb-vbmx7" Sep 10 13:41:20.767847 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:41:20.767824 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-69bcf98bbb-vbmx7" Sep 10 13:41:21.046307 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:41:21.046283 2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-69bcf98bbb-vbmx7"] Sep 10 13:41:21.047904 ip-10-0-140-8 kubenswrapper[2585]: W0910 13:41:21.047875 2585 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod111c86ef_d8e5_4460_b7c9_e7d3cf1a044f.slice/crio-9eb38234827af8dc003fdc55954ee87a77af2bf80a1c58df2cf6953429e3a9cb WatchSource:0}: Error finding container 9eb38234827af8dc003fdc55954ee87a77af2bf80a1c58df2cf6953429e3a9cb: Status 404 returned error can't find the container with id 9eb38234827af8dc003fdc55954ee87a77af2bf80a1c58df2cf6953429e3a9cb Sep 10 13:41:21.736883 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:41:21.736849 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-69bcf98bbb-vbmx7" event={"ID":"111c86ef-d8e5-4460-b7c9-e7d3cf1a044f","Type":"ContainerStarted","Data":"9eb38234827af8dc003fdc55954ee87a77af2bf80a1c58df2cf6953429e3a9cb"} Sep 10 13:41:23.751428 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:41:23.751391 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-69bcf98bbb-vbmx7" event={"ID":"111c86ef-d8e5-4460-b7c9-e7d3cf1a044f","Type":"ContainerStarted","Data":"90c0af958ba5134805b7b10bba452e96f388561ace902d86e5471790ec9d916d"} Sep 10 13:41:23.751816 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:41:23.751463 2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-controller-69bcf98bbb-vbmx7" Sep 10 13:41:23.825543 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:41:23.825507 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fxgmp_fa53c783-d40a-4024-9f7b-1d0d2b57f6dc/konnectivity-agent/0.log" limit=1024 Sep 10 13:41:23.919779 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:41:23.919697 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fxgmp_fa53c783-d40a-4024-9f7b-1d0d2b57f6dc/konnectivity-agent/0.log" limit=1024 Sep 10 13:41:34.758635 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:41:34.758606 2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/maas-controller-69bcf98bbb-vbmx7" Sep 10 13:41:34.780930 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:41:34.780883 2585 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/maas-controller-69bcf98bbb-vbmx7" podStartSLOduration=12.240831831 podStartE2EDuration="14.78086905s" totalImagesPullingTime="2.540037219s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 13:41:20 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 13:41:23.802001982 +0000 UTC m=+768.674706196" watchObservedRunningTime="2026-09-10 13:41:34.78086905 +0000 UTC m=+779.653573276" Sep 10 13:41:54.679416 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:41:54.679371 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fxgmp_fa53c783-d40a-4024-9f7b-1d0d2b57f6dc/konnectivity-agent/0.log" limit=1024 Sep 10 13:41:55.800226 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:41:55.800181 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fxgmp_fa53c783-d40a-4024-9f7b-1d0d2b57f6dc/konnectivity-agent/0.log" limit=1024 Sep 10 13:41:56.180446 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:41:56.180413 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fxgmp_fa53c783-d40a-4024-9f7b-1d0d2b57f6dc/konnectivity-agent/0.log" limit=1024 Sep 10 13:41:56.274768 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:41:56.274735 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fxgmp_fa53c783-d40a-4024-9f7b-1d0d2b57f6dc/konnectivity-agent/0.log" limit=1024 Sep 10 13:41:56.375782 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:41:56.375748 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fxgmp_fa53c783-d40a-4024-9f7b-1d0d2b57f6dc/konnectivity-agent/0.log" limit=1024 Sep 10 13:41:56.583581 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:41:56.583545 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fxgmp_fa53c783-d40a-4024-9f7b-1d0d2b57f6dc/konnectivity-agent/0.log" limit=1024 Sep 10 13:41:56.954385 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:41:56.954310 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fxgmp_fa53c783-d40a-4024-9f7b-1d0d2b57f6dc/konnectivity-agent/0.log" limit=1024 Sep 10 13:41:57.057354 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:41:57.057322 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fxgmp_fa53c783-d40a-4024-9f7b-1d0d2b57f6dc/konnectivity-agent/0.log" limit=1024 Sep 10 13:41:57.126435 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:41:57.126400 2585 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-59fbc47bf8-tklgb"] Sep 10 13:41:57.131670 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:41:57.131649 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-59fbc47bf8-tklgb" Sep 10 13:41:57.134351 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:41:57.134326 2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-dockercfg-8b77f\"" Sep 10 13:41:57.134775 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:41:57.134751 2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins\"" Sep 10 13:41:57.147456 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:41:57.147428 2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-59fbc47bf8-tklgb"] Sep 10 13:41:57.248147 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:41:57.248068 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b99a6276-397c-41d5-ac7f-3a4d380e2834-plugins-config-volume\") pod \"payload-pre-processing-59fbc47bf8-tklgb\" (UID: \"b99a6276-397c-41d5-ac7f-3a4d380e2834\") " pod="openshift-ingress/payload-pre-processing-59fbc47bf8-tklgb" Sep 10 13:41:57.248147 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:41:57.248113 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2bwzj\" (UniqueName: \"kubernetes.io/projected/b99a6276-397c-41d5-ac7f-3a4d380e2834-kube-api-access-2bwzj\") pod \"payload-pre-processing-59fbc47bf8-tklgb\" (UID: \"b99a6276-397c-41d5-ac7f-3a4d380e2834\") " pod="openshift-ingress/payload-pre-processing-59fbc47bf8-tklgb" Sep 10 13:41:57.348848 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:41:57.348809 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2bwzj\" (UniqueName: \"kubernetes.io/projected/b99a6276-397c-41d5-ac7f-3a4d380e2834-kube-api-access-2bwzj\") pod \"payload-pre-processing-59fbc47bf8-tklgb\" (UID: \"b99a6276-397c-41d5-ac7f-3a4d380e2834\") " pod="openshift-ingress/payload-pre-processing-59fbc47bf8-tklgb" Sep 10 13:41:57.349045 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:41:57.348902 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b99a6276-397c-41d5-ac7f-3a4d380e2834-plugins-config-volume\") pod \"payload-pre-processing-59fbc47bf8-tklgb\" (UID: \"b99a6276-397c-41d5-ac7f-3a4d380e2834\") " pod="openshift-ingress/payload-pre-processing-59fbc47bf8-tklgb" Sep 10 13:41:57.349580 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:41:57.349559 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b99a6276-397c-41d5-ac7f-3a4d380e2834-plugins-config-volume\") pod \"payload-pre-processing-59fbc47bf8-tklgb\" (UID: \"b99a6276-397c-41d5-ac7f-3a4d380e2834\") " pod="openshift-ingress/payload-pre-processing-59fbc47bf8-tklgb" Sep 10 13:41:57.361023 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:41:57.360994 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2bwzj\" (UniqueName: \"kubernetes.io/projected/b99a6276-397c-41d5-ac7f-3a4d380e2834-kube-api-access-2bwzj\") pod \"payload-pre-processing-59fbc47bf8-tklgb\" (UID: \"b99a6276-397c-41d5-ac7f-3a4d380e2834\") " pod="openshift-ingress/payload-pre-processing-59fbc47bf8-tklgb" Sep 10 13:41:57.443812 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:41:57.443771 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-59fbc47bf8-tklgb" Sep 10 13:41:57.724120 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:41:57.724000 2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-59fbc47bf8-tklgb"] Sep 10 13:41:57.726480 ip-10-0-140-8 kubenswrapper[2585]: W0910 13:41:57.726454 2585 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb99a6276_397c_41d5_ac7f_3a4d380e2834.slice/crio-46876502a75deef9307c81ffb6696ef2bbf1edfe607e8087942d22ce245815dd WatchSource:0}: Error finding container 46876502a75deef9307c81ffb6696ef2bbf1edfe607e8087942d22ce245815dd: Status 404 returned error can't find the container with id 46876502a75deef9307c81ffb6696ef2bbf1edfe607e8087942d22ce245815dd Sep 10 13:41:57.921337 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:41:57.921268 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-59fbc47bf8-tklgb" event={"ID":"b99a6276-397c-41d5-ac7f-3a4d380e2834","Type":"ContainerStarted","Data":"46876502a75deef9307c81ffb6696ef2bbf1edfe607e8087942d22ce245815dd"} Sep 10 13:41:59.932954 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:41:59.932917 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-59fbc47bf8-tklgb" event={"ID":"b99a6276-397c-41d5-ac7f-3a4d380e2834","Type":"ContainerStarted","Data":"930a1faafde64fdeed7900374b86578ab8852641ab89321852da6d61bd288d23"} Sep 10 13:41:59.933340 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:41:59.933039 2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-59fbc47bf8-tklgb" Sep 10 13:41:59.952819 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:41:59.952606 2585 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-59fbc47bf8-tklgb" podStartSLOduration=1.418882975 podStartE2EDuration="2.952585065s" totalImagesPullingTime="1.53370209s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 13:41:57 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 13:41:59.952582346 +0000 UTC m=+804.825286548" watchObservedRunningTime="2026-09-10 13:41:59.952585065 +0000 UTC m=+804.825289260" Sep 10 13:42:08.150800 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:42:08.150769 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fxgmp_fa53c783-d40a-4024-9f7b-1d0d2b57f6dc/konnectivity-agent/0.log" limit=1024 Sep 10 13:42:08.244009 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:42:08.243973 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fxgmp_fa53c783-d40a-4024-9f7b-1d0d2b57f6dc/konnectivity-agent/0.log" limit=1024 Sep 10 13:42:10.946707 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:42:10.946677 2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-59fbc47bf8-tklgb" Sep 10 13:42:12.727260 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:42:12.727216 2585 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-kn59r"] Sep 10 13:42:12.730995 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:42:12.730962 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-kn59r" Sep 10 13:42:12.738465 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:42:12.738441 2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"kube-root-ca.crt\"" Sep 10 13:42:12.739648 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:42:12.739630 2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"default-dockercfg-vzhxx\"" Sep 10 13:42:12.739712 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:42:12.739649 2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"openshift-service-ca.crt\"" Sep 10 13:42:12.739758 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:42:12.739630 2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-distinct-2-simulated-kserve-self-signed-certs\"" Sep 10 13:42:12.745892 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:42:12.745869 2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-kn59r"] Sep 10 13:42:12.765739 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:42:12.765709 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fxgmp_fa53c783-d40a-4024-9f7b-1d0d2b57f6dc/konnectivity-agent/0.log" limit=1024 Sep 10 13:42:12.890501 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:42:12.890463 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7a036868-f373-4ef0-92c6-c2c60b7ff160-tls-certs\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-kn59r\" (UID: \"7a036868-f373-4ef0-92c6-c2c60b7ff160\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-kn59r" Sep 10 13:42:12.890696 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:42:12.890513 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7a036868-f373-4ef0-92c6-c2c60b7ff160-home\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-kn59r\" (UID: \"7a036868-f373-4ef0-92c6-c2c60b7ff160\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-kn59r" Sep 10 13:42:12.890696 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:42:12.890563 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7a036868-f373-4ef0-92c6-c2c60b7ff160-tmp-dir\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-kn59r\" (UID: \"7a036868-f373-4ef0-92c6-c2c60b7ff160\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-kn59r" Sep 10 13:42:12.890696 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:42:12.890587 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-t5xwm\" (UniqueName: \"kubernetes.io/projected/7a036868-f373-4ef0-92c6-c2c60b7ff160-kube-api-access-t5xwm\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-kn59r\" (UID: \"7a036868-f373-4ef0-92c6-c2c60b7ff160\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-kn59r" Sep 10 13:42:12.890696 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:42:12.890608 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7a036868-f373-4ef0-92c6-c2c60b7ff160-model-cache\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-kn59r\" (UID: \"7a036868-f373-4ef0-92c6-c2c60b7ff160\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-kn59r" Sep 10 13:42:12.890696 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:42:12.890654 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7a036868-f373-4ef0-92c6-c2c60b7ff160-dshm\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-kn59r\" (UID: \"7a036868-f373-4ef0-92c6-c2c60b7ff160\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-kn59r" Sep 10 13:42:12.991348 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:42:12.991260 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7a036868-f373-4ef0-92c6-c2c60b7ff160-dshm\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-kn59r\" (UID: \"7a036868-f373-4ef0-92c6-c2c60b7ff160\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-kn59r" Sep 10 13:42:12.991348 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:42:12.991332 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7a036868-f373-4ef0-92c6-c2c60b7ff160-tls-certs\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-kn59r\" (UID: \"7a036868-f373-4ef0-92c6-c2c60b7ff160\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-kn59r" Sep 10 13:42:12.991562 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:42:12.991374 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7a036868-f373-4ef0-92c6-c2c60b7ff160-home\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-kn59r\" (UID: \"7a036868-f373-4ef0-92c6-c2c60b7ff160\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-kn59r" Sep 10 13:42:12.991562 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:42:12.991435 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7a036868-f373-4ef0-92c6-c2c60b7ff160-tmp-dir\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-kn59r\" (UID: \"7a036868-f373-4ef0-92c6-c2c60b7ff160\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-kn59r" Sep 10 13:42:12.991562 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:42:12.991480 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t5xwm\" (UniqueName: \"kubernetes.io/projected/7a036868-f373-4ef0-92c6-c2c60b7ff160-kube-api-access-t5xwm\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-kn59r\" (UID: \"7a036868-f373-4ef0-92c6-c2c60b7ff160\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-kn59r" Sep 10 13:42:12.991562 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:42:12.991505 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7a036868-f373-4ef0-92c6-c2c60b7ff160-model-cache\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-kn59r\" (UID: \"7a036868-f373-4ef0-92c6-c2c60b7ff160\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-kn59r" Sep 10 13:42:12.991858 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:42:12.991834 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7a036868-f373-4ef0-92c6-c2c60b7ff160-home\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-kn59r\" (UID: \"7a036868-f373-4ef0-92c6-c2c60b7ff160\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-kn59r" Sep 10 13:42:12.991947 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:42:12.991872 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7a036868-f373-4ef0-92c6-c2c60b7ff160-model-cache\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-kn59r\" (UID: \"7a036868-f373-4ef0-92c6-c2c60b7ff160\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-kn59r" Sep 10 13:42:12.992149 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:42:12.992130 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7a036868-f373-4ef0-92c6-c2c60b7ff160-tmp-dir\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-kn59r\" (UID: \"7a036868-f373-4ef0-92c6-c2c60b7ff160\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-kn59r" Sep 10 13:42:12.993836 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:42:12.993808 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7a036868-f373-4ef0-92c6-c2c60b7ff160-dshm\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-kn59r\" (UID: \"7a036868-f373-4ef0-92c6-c2c60b7ff160\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-kn59r" Sep 10 13:42:12.993921 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:42:12.993843 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7a036868-f373-4ef0-92c6-c2c60b7ff160-tls-certs\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-kn59r\" (UID: \"7a036868-f373-4ef0-92c6-c2c60b7ff160\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-kn59r" Sep 10 13:42:13.005325 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:42:13.005304 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-t5xwm\" (UniqueName: \"kubernetes.io/projected/7a036868-f373-4ef0-92c6-c2c60b7ff160-kube-api-access-t5xwm\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-kn59r\" (UID: \"7a036868-f373-4ef0-92c6-c2c60b7ff160\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-kn59r" Sep 10 13:42:13.041716 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:42:13.041687 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-kn59r" Sep 10 13:42:13.319134 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:42:13.319097 2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-kn59r"] Sep 10 13:42:13.319985 ip-10-0-140-8 kubenswrapper[2585]: W0910 13:42:13.319959 2585 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7a036868_f373_4ef0_92c6_c2c60b7ff160.slice/crio-2e8d8248ec518cd0f487e7dfcf11c03e53c74247372e58aa08b2d40961a5c7a2 WatchSource:0}: Error finding container 2e8d8248ec518cd0f487e7dfcf11c03e53c74247372e58aa08b2d40961a5c7a2: Status 404 returned error can't find the container with id 2e8d8248ec518cd0f487e7dfcf11c03e53c74247372e58aa08b2d40961a5c7a2 Sep 10 13:42:14.015971 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:42:14.015914 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-kn59r" event={"ID":"7a036868-f373-4ef0-92c6-c2c60b7ff160","Type":"ContainerStarted","Data":"2e8d8248ec518cd0f487e7dfcf11c03e53c74247372e58aa08b2d40961a5c7a2"} Sep 10 13:42:15.022493 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:42:15.022462 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-kn59r" event={"ID":"7a036868-f373-4ef0-92c6-c2c60b7ff160","Type":"ContainerStarted","Data":"9357b640e9d599a78b495c71df3d7820d4707038356fb94ceba0d580d3dca92d"} Sep 10 13:42:15.045108 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:42:15.045056 2585 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-kn59r" podStartSLOduration=1.918387831 podStartE2EDuration="3.045034104s" totalImagesPullingTime="1.126646273s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 13:42:12 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 13:42:15.043261553 +0000 UTC m=+819.915965748" watchObservedRunningTime="2026-09-10 13:42:15.045034104 +0000 UTC m=+819.917738306" Sep 10 13:42:23.042727 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:42:23.042688 2585 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-kn59r" Sep 10 13:42:23.043105 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:42:23.042755 2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-kn59r" Sep 10 13:42:23.055310 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:42:23.055284 2585 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-kn59r" Sep 10 13:42:23.075233 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:42:23.075203 2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-kn59r" Sep 10 13:42:29.042675 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:42:29.042642 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fxgmp_fa53c783-d40a-4024-9f7b-1d0d2b57f6dc/konnectivity-agent/0.log" limit=1024 Sep 10 13:42:56.808132 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:42:56.808095 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fxgmp_fa53c783-d40a-4024-9f7b-1d0d2b57f6dc/konnectivity-agent/0.log" limit=1024 Sep 10 13:43:09.708014 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:43:09.707979 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fxgmp_fa53c783-d40a-4024-9f7b-1d0d2b57f6dc/konnectivity-agent/0.log" limit=1024 Sep 10 13:43:14.916749 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:43:14.916709 2585 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fc46dg"] Sep 10 13:43:14.920070 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:43:14.920053 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fc46dg" Sep 10 13:43:14.926625 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:43:14.926396 2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-unab60ef4d3a239b5143b412cab04acac3-kserve-self-signed-certs\"" Sep 10 13:43:14.934659 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:43:14.934634 2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fc46dg"] Sep 10 13:43:14.939753 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:43:14.939725 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/92c99139-db0f-43e4-a9d8-dd0b791fe1ee-dshm\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fc46dg\" (UID: \"92c99139-db0f-43e4-a9d8-dd0b791fe1ee\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fc46dg" Sep 10 13:43:14.939850 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:43:14.939770 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-45rwr\" (UniqueName: \"kubernetes.io/projected/92c99139-db0f-43e4-a9d8-dd0b791fe1ee-kube-api-access-45rwr\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fc46dg\" (UID: \"92c99139-db0f-43e4-a9d8-dd0b791fe1ee\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fc46dg" Sep 10 13:43:14.939850 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:43:14.939831 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/92c99139-db0f-43e4-a9d8-dd0b791fe1ee-home\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fc46dg\" (UID: \"92c99139-db0f-43e4-a9d8-dd0b791fe1ee\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fc46dg" Sep 10 13:43:14.939924 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:43:14.939905 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/92c99139-db0f-43e4-a9d8-dd0b791fe1ee-tmp-dir\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fc46dg\" (UID: \"92c99139-db0f-43e4-a9d8-dd0b791fe1ee\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fc46dg" Sep 10 13:43:14.940016 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:43:14.939994 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/92c99139-db0f-43e4-a9d8-dd0b791fe1ee-model-cache\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fc46dg\" (UID: \"92c99139-db0f-43e4-a9d8-dd0b791fe1ee\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fc46dg" Sep 10 13:43:14.940060 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:43:14.940039 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/92c99139-db0f-43e4-a9d8-dd0b791fe1ee-tls-certs\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fc46dg\" (UID: \"92c99139-db0f-43e4-a9d8-dd0b791fe1ee\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fc46dg" Sep 10 13:43:15.040803 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:43:15.040771 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/92c99139-db0f-43e4-a9d8-dd0b791fe1ee-model-cache\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fc46dg\" (UID: \"92c99139-db0f-43e4-a9d8-dd0b791fe1ee\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fc46dg" Sep 10 13:43:15.040987 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:43:15.040815 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/92c99139-db0f-43e4-a9d8-dd0b791fe1ee-tls-certs\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fc46dg\" (UID: \"92c99139-db0f-43e4-a9d8-dd0b791fe1ee\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fc46dg" Sep 10 13:43:15.040987 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:43:15.040848 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/92c99139-db0f-43e4-a9d8-dd0b791fe1ee-dshm\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fc46dg\" (UID: \"92c99139-db0f-43e4-a9d8-dd0b791fe1ee\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fc46dg" Sep 10 13:43:15.040987 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:43:15.040869 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-45rwr\" (UniqueName: \"kubernetes.io/projected/92c99139-db0f-43e4-a9d8-dd0b791fe1ee-kube-api-access-45rwr\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fc46dg\" (UID: \"92c99139-db0f-43e4-a9d8-dd0b791fe1ee\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fc46dg" Sep 10 13:43:15.040987 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:43:15.040886 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/92c99139-db0f-43e4-a9d8-dd0b791fe1ee-home\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fc46dg\" (UID: \"92c99139-db0f-43e4-a9d8-dd0b791fe1ee\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fc46dg" Sep 10 13:43:15.040987 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:43:15.040940 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/92c99139-db0f-43e4-a9d8-dd0b791fe1ee-tmp-dir\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fc46dg\" (UID: \"92c99139-db0f-43e4-a9d8-dd0b791fe1ee\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fc46dg" Sep 10 13:43:15.041285 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:43:15.041179 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/92c99139-db0f-43e4-a9d8-dd0b791fe1ee-model-cache\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fc46dg\" (UID: \"92c99139-db0f-43e4-a9d8-dd0b791fe1ee\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fc46dg" Sep 10 13:43:15.041285 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:43:15.041209 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/92c99139-db0f-43e4-a9d8-dd0b791fe1ee-tmp-dir\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fc46dg\" (UID: \"92c99139-db0f-43e4-a9d8-dd0b791fe1ee\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fc46dg" Sep 10 13:43:15.041411 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:43:15.041311 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/92c99139-db0f-43e4-a9d8-dd0b791fe1ee-home\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fc46dg\" (UID: \"92c99139-db0f-43e4-a9d8-dd0b791fe1ee\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fc46dg" Sep 10 13:43:15.043281 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:43:15.043240 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/92c99139-db0f-43e4-a9d8-dd0b791fe1ee-dshm\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fc46dg\" (UID: \"92c99139-db0f-43e4-a9d8-dd0b791fe1ee\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fc46dg" Sep 10 13:43:15.043560 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:43:15.043539 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/92c99139-db0f-43e4-a9d8-dd0b791fe1ee-tls-certs\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fc46dg\" (UID: \"92c99139-db0f-43e4-a9d8-dd0b791fe1ee\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fc46dg" Sep 10 13:43:15.067073 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:43:15.067051 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-45rwr\" (UniqueName: \"kubernetes.io/projected/92c99139-db0f-43e4-a9d8-dd0b791fe1ee-kube-api-access-45rwr\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fc46dg\" (UID: \"92c99139-db0f-43e4-a9d8-dd0b791fe1ee\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fc46dg" Sep 10 13:43:15.067904 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:43:15.067880 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fxgmp_fa53c783-d40a-4024-9f7b-1d0d2b57f6dc/konnectivity-agent/0.log" limit=1024 Sep 10 13:43:15.230429 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:43:15.230341 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fc46dg" Sep 10 13:43:15.504160 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:43:15.504138 2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fc46dg"] Sep 10 13:43:15.505997 ip-10-0-140-8 kubenswrapper[2585]: W0910 13:43:15.505969 2585 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod92c99139_db0f_43e4_a9d8_dd0b791fe1ee.slice/crio-0b2ee807372c815b661cbd911cac4895b5691ecc5614b30968ec9c06c8e71900 WatchSource:0}: Error finding container 0b2ee807372c815b661cbd911cac4895b5691ecc5614b30968ec9c06c8e71900: Status 404 returned error can't find the container with id 0b2ee807372c815b661cbd911cac4895b5691ecc5614b30968ec9c06c8e71900 Sep 10 13:43:16.337004 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:43:16.336970 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fc46dg" event={"ID":"92c99139-db0f-43e4-a9d8-dd0b791fe1ee","Type":"ContainerStarted","Data":"4a486260563735130261345dab9e684aee681f615ff3c7603ce52e7cdf8484f0"} Sep 10 13:43:16.337406 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:43:16.337028 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fc46dg" event={"ID":"92c99139-db0f-43e4-a9d8-dd0b791fe1ee","Type":"ContainerStarted","Data":"0b2ee807372c815b661cbd911cac4895b5691ecc5614b30968ec9c06c8e71900"} Sep 10 13:43:16.360418 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:43:16.360374 2585 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fc46dg" podStartSLOduration=2.170869275 podStartE2EDuration="2.360357354s" totalImagesPullingTime="189.488079ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 13:43:14 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 13:43:16.35882564 +0000 UTC m=+881.231529843" watchObservedRunningTime="2026-09-10 13:43:16.360357354 +0000 UTC m=+881.233061554" Sep 10 13:43:18.312847 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:43:18.312811 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fxgmp_fa53c783-d40a-4024-9f7b-1d0d2b57f6dc/konnectivity-agent/0.log" limit=1024 Sep 10 13:43:19.039137 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:43:19.039100 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fxgmp_fa53c783-d40a-4024-9f7b-1d0d2b57f6dc/konnectivity-agent/0.log" limit=1024 Sep 10 13:43:24.738114 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:43:24.738083 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fxgmp_fa53c783-d40a-4024-9f7b-1d0d2b57f6dc/konnectivity-agent/0.log" limit=1024 Sep 10 13:43:25.230596 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:43:25.230552 2585 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fc46dg" Sep 10 13:43:25.230767 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:43:25.230628 2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fc46dg" Sep 10 13:43:25.243060 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:43:25.243026 2585 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fc46dg" Sep 10 13:43:25.394365 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:43:25.394335 2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fc46dg" Sep 10 13:43:28.017238 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:43:28.017161 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fxgmp_fa53c783-d40a-4024-9f7b-1d0d2b57f6dc/konnectivity-agent/0.log" limit=1024 Sep 10 13:43:35.634798 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:43:35.634767 2585 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-l9l8q_ad555c6a-4385-4f8e-93bb-a174cfaa8e74/kube-multus/2.log" Sep 10 13:43:35.636591 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:43:35.636570 2585 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-l9l8q_ad555c6a-4385-4f8e-93bb-a174cfaa8e74/kube-multus/2.log" Sep 10 13:43:49.932544 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:43:49.932506 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fxgmp_fa53c783-d40a-4024-9f7b-1d0d2b57f6dc/konnectivity-agent/0.log" limit=1024 Sep 10 13:43:56.277962 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:43:56.277920 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fxgmp_fa53c783-d40a-4024-9f7b-1d0d2b57f6dc/konnectivity-agent/0.log" limit=1024 Sep 10 13:43:56.378379 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:43:56.378344 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fxgmp_fa53c783-d40a-4024-9f7b-1d0d2b57f6dc/konnectivity-agent/0.log" limit=1024 Sep 10 13:45:10.810447 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:45:10.810372 2585 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-7477759fbf-zp9pr"] Sep 10 13:45:10.813682 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:45:10.813666 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-7477759fbf-zp9pr" Sep 10 13:45:10.827440 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:45:10.827411 2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-7477759fbf-zp9pr"] Sep 10 13:45:10.887577 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:45:10.887538 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/38d8ecc2-8168-406b-ab33-b14b0b6c272a-tls-cert\") pod \"authorino-7477759fbf-zp9pr\" (UID: \"38d8ecc2-8168-406b-ab33-b14b0b6c272a\") " pod="kuadrant-system/authorino-7477759fbf-zp9pr" Sep 10 13:45:10.887754 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:45:10.887606 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5vqpq\" (UniqueName: \"kubernetes.io/projected/38d8ecc2-8168-406b-ab33-b14b0b6c272a-kube-api-access-5vqpq\") pod \"authorino-7477759fbf-zp9pr\" (UID: \"38d8ecc2-8168-406b-ab33-b14b0b6c272a\") " pod="kuadrant-system/authorino-7477759fbf-zp9pr" Sep 10 13:45:10.988409 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:45:10.988370 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/38d8ecc2-8168-406b-ab33-b14b0b6c272a-tls-cert\") pod \"authorino-7477759fbf-zp9pr\" (UID: \"38d8ecc2-8168-406b-ab33-b14b0b6c272a\") " pod="kuadrant-system/authorino-7477759fbf-zp9pr" Sep 10 13:45:10.988603 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:45:10.988450 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5vqpq\" (UniqueName: \"kubernetes.io/projected/38d8ecc2-8168-406b-ab33-b14b0b6c272a-kube-api-access-5vqpq\") pod \"authorino-7477759fbf-zp9pr\" (UID: \"38d8ecc2-8168-406b-ab33-b14b0b6c272a\") " pod="kuadrant-system/authorino-7477759fbf-zp9pr" Sep 10 13:45:10.991084 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:45:10.991061 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/38d8ecc2-8168-406b-ab33-b14b0b6c272a-tls-cert\") pod \"authorino-7477759fbf-zp9pr\" (UID: \"38d8ecc2-8168-406b-ab33-b14b0b6c272a\") " pod="kuadrant-system/authorino-7477759fbf-zp9pr" Sep 10 13:45:10.998968 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:45:10.998944 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5vqpq\" (UniqueName: \"kubernetes.io/projected/38d8ecc2-8168-406b-ab33-b14b0b6c272a-kube-api-access-5vqpq\") pod \"authorino-7477759fbf-zp9pr\" (UID: \"38d8ecc2-8168-406b-ab33-b14b0b6c272a\") " pod="kuadrant-system/authorino-7477759fbf-zp9pr" Sep 10 13:45:11.122912 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:45:11.122824 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-7477759fbf-zp9pr" Sep 10 13:45:11.405919 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:45:11.405890 2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-7477759fbf-zp9pr"] Sep 10 13:45:11.406715 ip-10-0-140-8 kubenswrapper[2585]: W0910 13:45:11.406686 2585 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod38d8ecc2_8168_406b_ab33_b14b0b6c272a.slice/crio-29a231ba649f79fe214a4f75dc494f3d2d78ed12adaa3263a83e23917ae91e00 WatchSource:0}: Error finding container 29a231ba649f79fe214a4f75dc494f3d2d78ed12adaa3263a83e23917ae91e00: Status 404 returned error can't find the container with id 29a231ba649f79fe214a4f75dc494f3d2d78ed12adaa3263a83e23917ae91e00 Sep 10 13:45:11.407999 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:45:11.407981 2585 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Sep 10 13:45:11.930178 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:45:11.930145 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-7477759fbf-zp9pr" event={"ID":"38d8ecc2-8168-406b-ab33-b14b0b6c272a","Type":"ContainerStarted","Data":"29a231ba649f79fe214a4f75dc494f3d2d78ed12adaa3263a83e23917ae91e00"} Sep 10 13:45:12.937997 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:45:12.937962 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-7477759fbf-zp9pr" event={"ID":"38d8ecc2-8168-406b-ab33-b14b0b6c272a","Type":"ContainerStarted","Data":"65710c991b0ee629e7ea90127362fc7c7ffff08fe41b167d8743aa18f0433cfa"} Sep 10 13:45:12.962365 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:45:12.962299 2585 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/authorino-7477759fbf-zp9pr" podStartSLOduration=2.129498184 podStartE2EDuration="2.96227207s" totalImagesPullingTime="832.773886ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 13:45:10 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 13:45:12.958855105 +0000 UTC m=+997.831559310" watchObservedRunningTime="2026-09-10 13:45:12.96227207 +0000 UTC m=+997.834976272" Sep 10 13:45:12.996300 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:45:12.995627 2585 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-6b7cb4c856-crbnk"] Sep 10 13:45:12.996300 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:45:12.995888 2585 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/authorino-6b7cb4c856-crbnk" podUID="261a0443-b54f-4416-ab5e-7337bbcf0d6c" containerName="authorino" containerID="cri-o://a3984b53af511f3b82559cc835b51a6d5c1f657cd36598e5d957b6060e10111a" gracePeriod=30 Sep 10 13:45:13.274889 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:45:13.274860 2585 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-6b7cb4c856-crbnk" Sep 10 13:45:13.414004 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:45:13.413960 2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/261a0443-b54f-4416-ab5e-7337bbcf0d6c-tls-cert\") pod \"261a0443-b54f-4416-ab5e-7337bbcf0d6c\" (UID: \"261a0443-b54f-4416-ab5e-7337bbcf0d6c\") " Sep 10 13:45:13.414192 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:45:13.414039 2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-4sbw6\" (UniqueName: \"kubernetes.io/projected/261a0443-b54f-4416-ab5e-7337bbcf0d6c-kube-api-access-4sbw6\") pod \"261a0443-b54f-4416-ab5e-7337bbcf0d6c\" (UID: \"261a0443-b54f-4416-ab5e-7337bbcf0d6c\") " Sep 10 13:45:13.416055 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:45:13.416022 2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/261a0443-b54f-4416-ab5e-7337bbcf0d6c-kube-api-access-4sbw6" (OuterVolumeSpecName: "kube-api-access-4sbw6") pod "261a0443-b54f-4416-ab5e-7337bbcf0d6c" (UID: "261a0443-b54f-4416-ab5e-7337bbcf0d6c"). InnerVolumeSpecName "kube-api-access-4sbw6". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 10 13:45:13.424419 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:45:13.424391 2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/261a0443-b54f-4416-ab5e-7337bbcf0d6c-tls-cert" (OuterVolumeSpecName: "tls-cert") pod "261a0443-b54f-4416-ab5e-7337bbcf0d6c" (UID: "261a0443-b54f-4416-ab5e-7337bbcf0d6c"). InnerVolumeSpecName "tls-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 10 13:45:13.515107 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:45:13.515018 2585 reconciler_common.go:299] "Volume detached for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/261a0443-b54f-4416-ab5e-7337bbcf0d6c-tls-cert\") on node \"ip-10-0-140-8.ec2.internal\" DevicePath \"\"" Sep 10 13:45:13.515107 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:45:13.515053 2585 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4sbw6\" (UniqueName: \"kubernetes.io/projected/261a0443-b54f-4416-ab5e-7337bbcf0d6c-kube-api-access-4sbw6\") on node \"ip-10-0-140-8.ec2.internal\" DevicePath \"\"" Sep 10 13:45:13.607983 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:45:13.607838 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fxgmp_fa53c783-d40a-4024-9f7b-1d0d2b57f6dc/konnectivity-agent/0.log" limit=1024 Sep 10 13:45:13.690378 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:45:13.690336 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fxgmp_fa53c783-d40a-4024-9f7b-1d0d2b57f6dc/konnectivity-agent/0.log" limit=1024 Sep 10 13:45:13.945179 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:45:13.945140 2585 generic.go:356] "Generic (PLEG): container finished" podID="261a0443-b54f-4416-ab5e-7337bbcf0d6c" containerID="a3984b53af511f3b82559cc835b51a6d5c1f657cd36598e5d957b6060e10111a" exitCode=0 Sep 10 13:45:13.945608 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:45:13.945228 2585 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-6b7cb4c856-crbnk" Sep 10 13:45:13.945608 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:45:13.945235 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-6b7cb4c856-crbnk" event={"ID":"261a0443-b54f-4416-ab5e-7337bbcf0d6c","Type":"ContainerDied","Data":"a3984b53af511f3b82559cc835b51a6d5c1f657cd36598e5d957b6060e10111a"} Sep 10 13:45:13.945608 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:45:13.945298 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-6b7cb4c856-crbnk" event={"ID":"261a0443-b54f-4416-ab5e-7337bbcf0d6c","Type":"ContainerDied","Data":"cf80936647756f8f2c89d39227d06d0cf6460d48f2834348054a2e8b464cec8b"} Sep 10 13:45:13.945608 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:45:13.945320 2585 scope.go:117] "RemoveContainer" containerID="a3984b53af511f3b82559cc835b51a6d5c1f657cd36598e5d957b6060e10111a" Sep 10 13:45:13.954976 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:45:13.954958 2585 scope.go:117] "RemoveContainer" containerID="a3984b53af511f3b82559cc835b51a6d5c1f657cd36598e5d957b6060e10111a" Sep 10 13:45:13.955324 ip-10-0-140-8 kubenswrapper[2585]: E0910 13:45:13.955291 2585 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a3984b53af511f3b82559cc835b51a6d5c1f657cd36598e5d957b6060e10111a\": container with ID starting with a3984b53af511f3b82559cc835b51a6d5c1f657cd36598e5d957b6060e10111a not found: ID does not exist" containerID="a3984b53af511f3b82559cc835b51a6d5c1f657cd36598e5d957b6060e10111a" Sep 10 13:45:13.955450 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:45:13.955335 2585 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a3984b53af511f3b82559cc835b51a6d5c1f657cd36598e5d957b6060e10111a"} err="failed to get container status \"a3984b53af511f3b82559cc835b51a6d5c1f657cd36598e5d957b6060e10111a\": rpc error: code = NotFound desc = could not find container \"a3984b53af511f3b82559cc835b51a6d5c1f657cd36598e5d957b6060e10111a\": container with ID starting with a3984b53af511f3b82559cc835b51a6d5c1f657cd36598e5d957b6060e10111a not found: ID does not exist" Sep 10 13:45:13.973338 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:45:13.973306 2585 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-6b7cb4c856-crbnk"] Sep 10 13:45:13.978888 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:45:13.978864 2585 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-6b7cb4c856-crbnk"] Sep 10 13:45:14.860177 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:45:14.860139 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fxgmp_fa53c783-d40a-4024-9f7b-1d0d2b57f6dc/konnectivity-agent/0.log" limit=1024 Sep 10 13:45:14.966422 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:45:14.966389 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fxgmp_fa53c783-d40a-4024-9f7b-1d0d2b57f6dc/konnectivity-agent/0.log" limit=1024 Sep 10 13:45:15.647980 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:45:15.647949 2585 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="261a0443-b54f-4416-ab5e-7337bbcf0d6c" path="/var/lib/kubelet/pods/261a0443-b54f-4416-ab5e-7337bbcf0d6c/volumes" Sep 10 13:45:32.748064 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:45:32.748028 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fxgmp_fa53c783-d40a-4024-9f7b-1d0d2b57f6dc/konnectivity-agent/0.log" limit=1024 Sep 10 13:45:32.912057 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:45:32.912021 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fxgmp_fa53c783-d40a-4024-9f7b-1d0d2b57f6dc/konnectivity-agent/0.log" limit=1024 Sep 10 13:45:33.122742 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:45:33.122700 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fxgmp_fa53c783-d40a-4024-9f7b-1d0d2b57f6dc/konnectivity-agent/0.log" limit=1024 Sep 10 13:45:33.195388 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:45:33.195348 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fxgmp_fa53c783-d40a-4024-9f7b-1d0d2b57f6dc/konnectivity-agent/0.log" limit=1024 Sep 10 13:45:33.274995 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:45:33.274954 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fxgmp_fa53c783-d40a-4024-9f7b-1d0d2b57f6dc/konnectivity-agent/0.log" limit=1024 Sep 10 13:45:34.498168 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:45:34.498129 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fxgmp_fa53c783-d40a-4024-9f7b-1d0d2b57f6dc/konnectivity-agent/0.log" limit=1024 Sep 10 13:45:34.595263 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:45:34.595204 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fxgmp_fa53c783-d40a-4024-9f7b-1d0d2b57f6dc/konnectivity-agent/0.log" limit=1024 Sep 10 13:45:35.127384 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:45:35.127345 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fxgmp_fa53c783-d40a-4024-9f7b-1d0d2b57f6dc/konnectivity-agent/0.log" limit=1024 Sep 10 13:45:35.198289 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:45:35.198255 2585 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7528-92f832"] Sep 10 13:45:35.198666 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:45:35.198652 2585 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="261a0443-b54f-4416-ab5e-7337bbcf0d6c" containerName="authorino" Sep 10 13:45:35.198712 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:45:35.198668 2585 state_mem.go:107] "Deleted CPUSet assignment" podUID="261a0443-b54f-4416-ab5e-7337bbcf0d6c" containerName="authorino" Sep 10 13:45:35.198752 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:45:35.198737 2585 memory_manager.go:368] "RemoveStaleState removing state" podUID="261a0443-b54f-4416-ab5e-7337bbcf0d6c" containerName="authorino" Sep 10 13:45:35.201681 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:45:35.201667 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7528-92f832" Sep 10 13:45:35.205036 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:45:35.205016 2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"default-dockercfg-zn6pb\"" Sep 10 13:45:35.213563 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:45:35.213542 2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7528-92f832"] Sep 10 13:45:35.307867 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:45:35.307834 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2jsft\" (UniqueName: \"kubernetes.io/projected/2867f1d9-8213-4663-8f82-0593e1a3d875-kube-api-access-2jsft\") pod \"test-curl-7528-92f832\" (UID: \"2867f1d9-8213-4663-8f82-0593e1a3d875\") " pod="opendatahub/test-curl-7528-92f832" Sep 10 13:45:35.408525 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:45:35.408425 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2jsft\" (UniqueName: \"kubernetes.io/projected/2867f1d9-8213-4663-8f82-0593e1a3d875-kube-api-access-2jsft\") pod \"test-curl-7528-92f832\" (UID: \"2867f1d9-8213-4663-8f82-0593e1a3d875\") " pod="opendatahub/test-curl-7528-92f832" Sep 10 13:45:35.422799 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:45:35.422774 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2jsft\" (UniqueName: \"kubernetes.io/projected/2867f1d9-8213-4663-8f82-0593e1a3d875-kube-api-access-2jsft\") pod \"test-curl-7528-92f832\" (UID: \"2867f1d9-8213-4663-8f82-0593e1a3d875\") " pod="opendatahub/test-curl-7528-92f832" Sep 10 13:45:35.511118 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:45:35.511081 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7528-92f832" Sep 10 13:45:35.788971 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:45:35.788947 2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7528-92f832"] Sep 10 13:45:35.792511 ip-10-0-140-8 kubenswrapper[2585]: W0910 13:45:35.792473 2585 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod2867f1d9_8213_4663_8f82_0593e1a3d875.slice/crio-eea8d1733a0817acbfd51c8a038f635021fef5238ead408336a6d2135618570f WatchSource:0}: Error finding container eea8d1733a0817acbfd51c8a038f635021fef5238ead408336a6d2135618570f: Status 404 returned error can't find the container with id eea8d1733a0817acbfd51c8a038f635021fef5238ead408336a6d2135618570f Sep 10 13:45:36.070822 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:45:36.070786 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7528-92f832" event={"ID":"2867f1d9-8213-4663-8f82-0593e1a3d875","Type":"ContainerStarted","Data":"eea8d1733a0817acbfd51c8a038f635021fef5238ead408336a6d2135618570f"} Sep 10 13:45:36.829800 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:45:36.829758 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fxgmp_fa53c783-d40a-4024-9f7b-1d0d2b57f6dc/konnectivity-agent/0.log" limit=1024 Sep 10 13:45:36.942566 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:45:36.942531 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fxgmp_fa53c783-d40a-4024-9f7b-1d0d2b57f6dc/konnectivity-agent/0.log" limit=1024 Sep 10 13:45:38.086737 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:45:38.086695 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7528-92f832" event={"ID":"2867f1d9-8213-4663-8f82-0593e1a3d875","Type":"ContainerStarted","Data":"9b2b81b780657c84ea8cdfb20068504f1bac70eb369c128f0ac89776da4f157c"} Sep 10 13:45:38.104600 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:45:38.104548 2585 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/test-curl-7528-92f832" podStartSLOduration=0.979223128 podStartE2EDuration="3.104531734s" totalImagesPullingTime="2.125308606s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 13:45:35 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 13:45:38.103713305 +0000 UTC m=+1022.976417505" watchObservedRunningTime="2026-09-10 13:45:38.104531734 +0000 UTC m=+1022.977235934" Sep 10 13:45:38.174724 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:45:38.174628 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fxgmp_fa53c783-d40a-4024-9f7b-1d0d2b57f6dc/konnectivity-agent/0.log" limit=1024 Sep 10 13:45:38.276730 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:45:38.276690 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fxgmp_fa53c783-d40a-4024-9f7b-1d0d2b57f6dc/konnectivity-agent/0.log" limit=1024 Sep 10 13:45:38.399197 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:45:38.399162 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fxgmp_fa53c783-d40a-4024-9f7b-1d0d2b57f6dc/konnectivity-agent/0.log" limit=1024 Sep 10 13:45:38.772778 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:45:38.772741 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fxgmp_fa53c783-d40a-4024-9f7b-1d0d2b57f6dc/konnectivity-agent/0.log" limit=1024 Sep 10 13:45:38.794632 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:45:38.794594 2585 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7528-92f832"] Sep 10 13:45:38.800040 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:45:38.800009 2585 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7528-92f832"] Sep 10 13:45:38.872099 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:45:38.872054 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fxgmp_fa53c783-d40a-4024-9f7b-1d0d2b57f6dc/konnectivity-agent/0.log" limit=1024 Sep 10 13:45:38.882444 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:45:38.882413 2585 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-86fdcf-6cb885c959-6vx6m"] Sep 10 13:45:38.882845 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:45:38.882829 2585 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2867f1d9-8213-4663-8f82-0593e1a3d875" containerName="test-curl-7528-92f832" Sep 10 13:45:38.882900 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:45:38.882847 2585 state_mem.go:107] "Deleted CPUSet assignment" podUID="2867f1d9-8213-4663-8f82-0593e1a3d875" containerName="test-curl-7528-92f832" Sep 10 13:45:38.882936 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:45:38.882919 2585 memory_manager.go:368] "RemoveStaleState removing state" podUID="2867f1d9-8213-4663-8f82-0593e1a3d875" containerName="test-curl-7528-92f832" Sep 10 13:45:38.886125 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:45:38.886108 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-86fdcf-6cb885c959-6vx6m" Sep 10 13:45:38.888472 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:45:38.888448 2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-inf-a-86fdcf\"" Sep 10 13:45:38.897299 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:45:38.897278 2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-86fdcf-6cb885c959-6vx6m"] Sep 10 13:45:38.925517 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:45:38.925491 2585 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7528-5078da"] Sep 10 13:45:38.929051 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:45:38.929032 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7528-5078da" Sep 10 13:45:38.937133 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:45:38.937109 2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7528-5078da"] Sep 10 13:45:38.940476 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:45:38.940452 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vsbck\" (UniqueName: \"kubernetes.io/projected/2ad48c0c-9065-4dd8-91d2-053505c1078d-kube-api-access-vsbck\") pod \"payload-processing-e2e-inf-a-86fdcf-6cb885c959-6vx6m\" (UID: \"2ad48c0c-9065-4dd8-91d2-053505c1078d\") " pod="openshift-ingress/payload-processing-e2e-inf-a-86fdcf-6cb885c959-6vx6m" Sep 10 13:45:38.940576 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:45:38.940549 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2ad48c0c-9065-4dd8-91d2-053505c1078d-plugins-config-volume\") pod \"payload-processing-e2e-inf-a-86fdcf-6cb885c959-6vx6m\" (UID: \"2ad48c0c-9065-4dd8-91d2-053505c1078d\") " pod="openshift-ingress/payload-processing-e2e-inf-a-86fdcf-6cb885c959-6vx6m" Sep 10 13:45:39.041625 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:45:39.041591 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2ad48c0c-9065-4dd8-91d2-053505c1078d-plugins-config-volume\") pod \"payload-processing-e2e-inf-a-86fdcf-6cb885c959-6vx6m\" (UID: \"2ad48c0c-9065-4dd8-91d2-053505c1078d\") " pod="openshift-ingress/payload-processing-e2e-inf-a-86fdcf-6cb885c959-6vx6m" Sep 10 13:45:39.041815 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:45:39.041659 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vsbck\" (UniqueName: \"kubernetes.io/projected/2ad48c0c-9065-4dd8-91d2-053505c1078d-kube-api-access-vsbck\") pod \"payload-processing-e2e-inf-a-86fdcf-6cb885c959-6vx6m\" (UID: \"2ad48c0c-9065-4dd8-91d2-053505c1078d\") " pod="openshift-ingress/payload-processing-e2e-inf-a-86fdcf-6cb885c959-6vx6m" Sep 10 13:45:39.041815 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:45:39.041720 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2vxq5\" (UniqueName: \"kubernetes.io/projected/83dd0325-e802-4f32-8ef8-cc5b95f59e7c-kube-api-access-2vxq5\") pod \"test-curl-7528-5078da\" (UID: \"83dd0325-e802-4f32-8ef8-cc5b95f59e7c\") " pod="opendatahub/test-curl-7528-5078da" Sep 10 13:45:39.042300 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:45:39.042279 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2ad48c0c-9065-4dd8-91d2-053505c1078d-plugins-config-volume\") pod \"payload-processing-e2e-inf-a-86fdcf-6cb885c959-6vx6m\" (UID: \"2ad48c0c-9065-4dd8-91d2-053505c1078d\") " pod="openshift-ingress/payload-processing-e2e-inf-a-86fdcf-6cb885c959-6vx6m" Sep 10 13:45:39.052875 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:45:39.052844 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vsbck\" (UniqueName: \"kubernetes.io/projected/2ad48c0c-9065-4dd8-91d2-053505c1078d-kube-api-access-vsbck\") pod \"payload-processing-e2e-inf-a-86fdcf-6cb885c959-6vx6m\" (UID: \"2ad48c0c-9065-4dd8-91d2-053505c1078d\") " pod="openshift-ingress/payload-processing-e2e-inf-a-86fdcf-6cb885c959-6vx6m" Sep 10 13:45:39.092848 ip-10-0-140-8 kubenswrapper[2585]: E0910 13:45:39.092817 2585 kuberuntime_manager.go:649] "Missing actuated resource record" pod="opendatahub/test-curl-7528-92f832" container="test-curl-7528-92f832" Sep 10 13:45:39.143006 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:45:39.142929 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2vxq5\" (UniqueName: \"kubernetes.io/projected/83dd0325-e802-4f32-8ef8-cc5b95f59e7c-kube-api-access-2vxq5\") pod \"test-curl-7528-5078da\" (UID: \"83dd0325-e802-4f32-8ef8-cc5b95f59e7c\") " pod="opendatahub/test-curl-7528-5078da" Sep 10 13:45:39.151878 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:45:39.151852 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2vxq5\" (UniqueName: \"kubernetes.io/projected/83dd0325-e802-4f32-8ef8-cc5b95f59e7c-kube-api-access-2vxq5\") pod \"test-curl-7528-5078da\" (UID: \"83dd0325-e802-4f32-8ef8-cc5b95f59e7c\") " pod="opendatahub/test-curl-7528-5078da" Sep 10 13:45:39.198385 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:45:39.198350 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-86fdcf-6cb885c959-6vx6m" Sep 10 13:45:39.238907 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:45:39.238876 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7528-5078da" Sep 10 13:45:39.483044 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:45:39.483015 2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-86fdcf-6cb885c959-6vx6m"] Sep 10 13:45:39.485346 ip-10-0-140-8 kubenswrapper[2585]: W0910 13:45:39.485314 2585 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2ad48c0c_9065_4dd8_91d2_053505c1078d.slice/crio-49cb1f8bbe2d27d195af90904b051561ce62905b85333c009ef2b0521664b523 WatchSource:0}: Error finding container 49cb1f8bbe2d27d195af90904b051561ce62905b85333c009ef2b0521664b523: Status 404 returned error can't find the container with id 49cb1f8bbe2d27d195af90904b051561ce62905b85333c009ef2b0521664b523 Sep 10 13:45:39.530726 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:45:39.530700 2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7528-5078da"] Sep 10 13:45:39.531775 ip-10-0-140-8 kubenswrapper[2585]: W0910 13:45:39.531748 2585 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod83dd0325_e802_4f32_8ef8_cc5b95f59e7c.slice/crio-502cd8dcc8aced48fbb79a2717abe73692f17d1db80b7c182ffba52669685f53 WatchSource:0}: Error finding container 502cd8dcc8aced48fbb79a2717abe73692f17d1db80b7c182ffba52669685f53: Status 404 returned error can't find the container with id 502cd8dcc8aced48fbb79a2717abe73692f17d1db80b7c182ffba52669685f53 Sep 10 13:45:40.100630 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:45:40.100593 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-86fdcf-6cb885c959-6vx6m" event={"ID":"2ad48c0c-9065-4dd8-91d2-053505c1078d","Type":"ContainerStarted","Data":"19a0d58e563bd71c676e35dea76e745dc50edfe25a2a6639fb12777e6163f2ee"} Sep 10 13:45:40.101237 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:45:40.100684 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-86fdcf-6cb885c959-6vx6m" event={"ID":"2ad48c0c-9065-4dd8-91d2-053505c1078d","Type":"ContainerStarted","Data":"49cb1f8bbe2d27d195af90904b051561ce62905b85333c009ef2b0521664b523"} Sep 10 13:45:40.101237 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:45:40.100838 2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-inf-a-86fdcf-6cb885c959-6vx6m" Sep 10 13:45:40.102315 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:45:40.102292 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7528-5078da" event={"ID":"83dd0325-e802-4f32-8ef8-cc5b95f59e7c","Type":"ContainerStarted","Data":"484cee4234bd2de64b114d24d0c29b5538aa9f3f48b7567f80dfbc826ce102f0"} Sep 10 13:45:40.102436 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:45:40.102326 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7528-5078da" event={"ID":"83dd0325-e802-4f32-8ef8-cc5b95f59e7c","Type":"ContainerStarted","Data":"502cd8dcc8aced48fbb79a2717abe73692f17d1db80b7c182ffba52669685f53"} Sep 10 13:45:40.102436 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:45:40.102347 2585 kuberuntime_container.go:873] "Killing container with a grace period" pod="opendatahub/test-curl-7528-92f832" podUID="2867f1d9-8213-4663-8f82-0593e1a3d875" containerName="test-curl-7528-92f832" containerID="cri-o://9b2b81b780657c84ea8cdfb20068504f1bac70eb369c128f0ac89776da4f157c" gracePeriod=2 Sep 10 13:45:40.127721 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:45:40.127668 2585 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-inf-a-86fdcf-6cb885c959-6vx6m" podStartSLOduration=2.127652 podStartE2EDuration="2.127652s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 13:45:38 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 13:45:40.124866804 +0000 UTC m=+1024.997571007" watchObservedRunningTime="2026-09-10 13:45:40.127652 +0000 UTC m=+1025.000356201" Sep 10 13:45:40.148087 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:45:40.148030 2585 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/test-curl-7528-5078da" podStartSLOduration=2.148008703 podStartE2EDuration="2.148008703s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 13:45:38 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 13:45:40.145370725 +0000 UTC m=+1025.018074952" watchObservedRunningTime="2026-09-10 13:45:40.148008703 +0000 UTC m=+1025.020712905" Sep 10 13:45:40.865748 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:45:40.865653 2585 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7528-5078da"] Sep 10 13:45:40.869663 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:45:40.869629 2585 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7528-5078da"] Sep 10 13:45:41.107762 ip-10-0-140-8 kubenswrapper[2585]: E0910 13:45:41.107736 2585 kuberuntime_manager.go:649] "Missing actuated resource record" pod="opendatahub/test-curl-7528-5078da" container="test-curl-7528-5078da" Sep 10 13:45:41.452777 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:45:41.452744 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fxgmp_fa53c783-d40a-4024-9f7b-1d0d2b57f6dc/konnectivity-agent/0.log" limit=1024 Sep 10 13:45:41.933776 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:45:41.933748 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fxgmp_fa53c783-d40a-4024-9f7b-1d0d2b57f6dc/konnectivity-agent/0.log" limit=1024 Sep 10 13:45:42.113372 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:45:42.113336 2585 kuberuntime_container.go:873] "Killing container with a grace period" pod="opendatahub/test-curl-7528-5078da" podUID="83dd0325-e802-4f32-8ef8-cc5b95f59e7c" containerName="test-curl-7528-5078da" containerID="cri-o://484cee4234bd2de64b114d24d0c29b5538aa9f3f48b7567f80dfbc826ce102f0" gracePeriod=2 Sep 10 13:45:42.289923 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:45:42.289895 2585 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7528-92f832" Sep 10 13:45:42.373953 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:45:42.373917 2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-2jsft\" (UniqueName: \"kubernetes.io/projected/2867f1d9-8213-4663-8f82-0593e1a3d875-kube-api-access-2jsft\") pod \"2867f1d9-8213-4663-8f82-0593e1a3d875\" (UID: \"2867f1d9-8213-4663-8f82-0593e1a3d875\") " Sep 10 13:45:42.376144 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:45:42.376120 2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2867f1d9-8213-4663-8f82-0593e1a3d875-kube-api-access-2jsft" (OuterVolumeSpecName: "kube-api-access-2jsft") pod "2867f1d9-8213-4663-8f82-0593e1a3d875" (UID: "2867f1d9-8213-4663-8f82-0593e1a3d875"). InnerVolumeSpecName "kube-api-access-2jsft". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 10 13:45:42.432471 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:45:42.432431 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fxgmp_fa53c783-d40a-4024-9f7b-1d0d2b57f6dc/konnectivity-agent/0.log" limit=1024 Sep 10 13:45:42.474942 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:45:42.474860 2585 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2jsft\" (UniqueName: \"kubernetes.io/projected/2867f1d9-8213-4663-8f82-0593e1a3d875-kube-api-access-2jsft\") on node \"ip-10-0-140-8.ec2.internal\" DevicePath \"\"" Sep 10 13:45:43.120072 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:45:43.120030 2585 generic.go:356] "Generic (PLEG): container finished" podID="2867f1d9-8213-4663-8f82-0593e1a3d875" containerID="9b2b81b780657c84ea8cdfb20068504f1bac70eb369c128f0ac89776da4f157c" exitCode=137 Sep 10 13:45:43.120072 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:45:43.120076 2585 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7528-92f832" Sep 10 13:45:43.120546 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:45:43.120094 2585 scope.go:117] "RemoveContainer" containerID="9b2b81b780657c84ea8cdfb20068504f1bac70eb369c128f0ac89776da4f157c" Sep 10 13:45:43.131548 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:45:43.131528 2585 scope.go:117] "RemoveContainer" containerID="9b2b81b780657c84ea8cdfb20068504f1bac70eb369c128f0ac89776da4f157c" Sep 10 13:45:43.131834 ip-10-0-140-8 kubenswrapper[2585]: E0910 13:45:43.131811 2585 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9b2b81b780657c84ea8cdfb20068504f1bac70eb369c128f0ac89776da4f157c\": container with ID starting with 9b2b81b780657c84ea8cdfb20068504f1bac70eb369c128f0ac89776da4f157c not found: ID does not exist" containerID="9b2b81b780657c84ea8cdfb20068504f1bac70eb369c128f0ac89776da4f157c" Sep 10 13:45:43.131878 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:45:43.131845 2585 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9b2b81b780657c84ea8cdfb20068504f1bac70eb369c128f0ac89776da4f157c"} err="failed to get container status \"9b2b81b780657c84ea8cdfb20068504f1bac70eb369c128f0ac89776da4f157c\": rpc error: code = NotFound desc = could not find container \"9b2b81b780657c84ea8cdfb20068504f1bac70eb369c128f0ac89776da4f157c\": container with ID starting with 9b2b81b780657c84ea8cdfb20068504f1bac70eb369c128f0ac89776da4f157c not found: ID does not exist" Sep 10 13:45:43.133897 ip-10-0-140-8 kubenswrapper[2585]: E0910 13:45:43.133873 2585 status_manager.go:1018] "Failed to get status for pod" err="pods \"test-curl-7528-92f832\" is forbidden: User \"system:node:ip-10-0-140-8.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-140-8.ec2.internal' and this object" podUID="2867f1d9-8213-4663-8f82-0593e1a3d875" pod="opendatahub/test-curl-7528-92f832" Sep 10 13:45:43.218266 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:45:43.218211 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fxgmp_fa53c783-d40a-4024-9f7b-1d0d2b57f6dc/konnectivity-agent/0.log" limit=1024 Sep 10 13:45:43.283405 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:45:43.283368 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fxgmp_fa53c783-d40a-4024-9f7b-1d0d2b57f6dc/konnectivity-agent/0.log" limit=1024 Sep 10 13:45:43.499533 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:45:43.499451 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fxgmp_fa53c783-d40a-4024-9f7b-1d0d2b57f6dc/konnectivity-agent/0.log" limit=1024 Sep 10 13:45:43.648793 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:45:43.648757 2585 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="2867f1d9-8213-4663-8f82-0593e1a3d875" path="/var/lib/kubelet/pods/2867f1d9-8213-4663-8f82-0593e1a3d875/volumes" Sep 10 13:45:43.908143 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:45:43.908114 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fxgmp_fa53c783-d40a-4024-9f7b-1d0d2b57f6dc/konnectivity-agent/0.log" limit=1024 Sep 10 13:45:44.076749 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:45:44.076685 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fxgmp_fa53c783-d40a-4024-9f7b-1d0d2b57f6dc/konnectivity-agent/0.log" limit=1024 Sep 10 13:45:44.320193 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:45:44.320165 2585 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7528-5078da" Sep 10 13:45:44.393987 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:45:44.393942 2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-2vxq5\" (UniqueName: \"kubernetes.io/projected/83dd0325-e802-4f32-8ef8-cc5b95f59e7c-kube-api-access-2vxq5\") pod \"83dd0325-e802-4f32-8ef8-cc5b95f59e7c\" (UID: \"83dd0325-e802-4f32-8ef8-cc5b95f59e7c\") " Sep 10 13:45:44.396006 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:45:44.395977 2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/83dd0325-e802-4f32-8ef8-cc5b95f59e7c-kube-api-access-2vxq5" (OuterVolumeSpecName: "kube-api-access-2vxq5") pod "83dd0325-e802-4f32-8ef8-cc5b95f59e7c" (UID: "83dd0325-e802-4f32-8ef8-cc5b95f59e7c"). InnerVolumeSpecName "kube-api-access-2vxq5". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 10 13:45:44.495570 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:45:44.495490 2585 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2vxq5\" (UniqueName: \"kubernetes.io/projected/83dd0325-e802-4f32-8ef8-cc5b95f59e7c-kube-api-access-2vxq5\") on node \"ip-10-0-140-8.ec2.internal\" DevicePath \"\"" Sep 10 13:45:44.941009 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:45:44.940972 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fxgmp_fa53c783-d40a-4024-9f7b-1d0d2b57f6dc/konnectivity-agent/0.log" limit=1024 Sep 10 13:45:45.036718 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:45:45.036685 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fxgmp_fa53c783-d40a-4024-9f7b-1d0d2b57f6dc/konnectivity-agent/0.log" limit=1024 Sep 10 13:45:45.133659 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:45:45.133624 2585 generic.go:356] "Generic (PLEG): container finished" podID="83dd0325-e802-4f32-8ef8-cc5b95f59e7c" containerID="484cee4234bd2de64b114d24d0c29b5538aa9f3f48b7567f80dfbc826ce102f0" exitCode=137 Sep 10 13:45:45.133659 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:45:45.133666 2585 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7528-5078da" Sep 10 13:45:45.133871 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:45:45.133711 2585 scope.go:117] "RemoveContainer" containerID="484cee4234bd2de64b114d24d0c29b5538aa9f3f48b7567f80dfbc826ce102f0" Sep 10 13:45:45.142806 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:45:45.142788 2585 scope.go:117] "RemoveContainer" containerID="484cee4234bd2de64b114d24d0c29b5538aa9f3f48b7567f80dfbc826ce102f0" Sep 10 13:45:45.143085 ip-10-0-140-8 kubenswrapper[2585]: E0910 13:45:45.143066 2585 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"484cee4234bd2de64b114d24d0c29b5538aa9f3f48b7567f80dfbc826ce102f0\": container with ID starting with 484cee4234bd2de64b114d24d0c29b5538aa9f3f48b7567f80dfbc826ce102f0 not found: ID does not exist" containerID="484cee4234bd2de64b114d24d0c29b5538aa9f3f48b7567f80dfbc826ce102f0" Sep 10 13:45:45.143124 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:45:45.143097 2585 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"484cee4234bd2de64b114d24d0c29b5538aa9f3f48b7567f80dfbc826ce102f0"} err="failed to get container status \"484cee4234bd2de64b114d24d0c29b5538aa9f3f48b7567f80dfbc826ce102f0\": rpc error: code = NotFound desc = could not find container \"484cee4234bd2de64b114d24d0c29b5538aa9f3f48b7567f80dfbc826ce102f0\": container with ID starting with 484cee4234bd2de64b114d24d0c29b5538aa9f3f48b7567f80dfbc826ce102f0 not found: ID does not exist" Sep 10 13:45:45.147529 ip-10-0-140-8 kubenswrapper[2585]: E0910 13:45:45.147508 2585 status_manager.go:1018] "Failed to get status for pod" err="pods \"test-curl-7528-5078da\" is forbidden: User \"system:node:ip-10-0-140-8.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-140-8.ec2.internal' and this object" podUID="83dd0325-e802-4f32-8ef8-cc5b95f59e7c" pod="opendatahub/test-curl-7528-5078da" Sep 10 13:45:45.646117 ip-10-0-140-8 kubenswrapper[2585]: E0910 13:45:45.646078 2585 status_manager.go:1018] "Failed to get status for pod" err="pods \"test-curl-7528-5078da\" is forbidden: User \"system:node:ip-10-0-140-8.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-140-8.ec2.internal' and this object" podUID="83dd0325-e802-4f32-8ef8-cc5b95f59e7c" pod="opendatahub/test-curl-7528-5078da" Sep 10 13:45:45.647341 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:45:45.647320 2585 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="83dd0325-e802-4f32-8ef8-cc5b95f59e7c" path="/var/lib/kubelet/pods/83dd0325-e802-4f32-8ef8-cc5b95f59e7c/volumes" Sep 10 13:45:46.140693 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:45:46.140663 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fxgmp_fa53c783-d40a-4024-9f7b-1d0d2b57f6dc/konnectivity-agent/0.log" limit=1024 Sep 10 13:45:46.246052 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:45:46.246011 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fxgmp_fa53c783-d40a-4024-9f7b-1d0d2b57f6dc/konnectivity-agent/0.log" limit=1024 Sep 10 13:45:46.551536 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:45:46.551449 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fxgmp_fa53c783-d40a-4024-9f7b-1d0d2b57f6dc/konnectivity-agent/0.log" limit=1024 Sep 10 13:45:46.664977 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:45:46.664936 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fxgmp_fa53c783-d40a-4024-9f7b-1d0d2b57f6dc/konnectivity-agent/0.log" limit=1024 Sep 10 13:45:46.762626 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:45:46.762595 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fxgmp_fa53c783-d40a-4024-9f7b-1d0d2b57f6dc/konnectivity-agent/0.log" limit=1024 Sep 10 13:45:46.866058 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:45:46.866022 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fxgmp_fa53c783-d40a-4024-9f7b-1d0d2b57f6dc/konnectivity-agent/0.log" limit=1024 Sep 10 13:45:46.920653 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:45:46.920622 2585 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-autores-b1f490-5f54877b85-4d97t"] Sep 10 13:45:46.921024 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:45:46.921012 2585 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="83dd0325-e802-4f32-8ef8-cc5b95f59e7c" containerName="test-curl-7528-5078da" Sep 10 13:45:46.921068 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:45:46.921026 2585 state_mem.go:107] "Deleted CPUSet assignment" podUID="83dd0325-e802-4f32-8ef8-cc5b95f59e7c" containerName="test-curl-7528-5078da" Sep 10 13:45:46.921103 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:45:46.921098 2585 memory_manager.go:368] "RemoveStaleState removing state" podUID="83dd0325-e802-4f32-8ef8-cc5b95f59e7c" containerName="test-curl-7528-5078da" Sep 10 13:45:46.925700 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:45:46.925683 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-autores-b1f490-5f54877b85-4d97t" Sep 10 13:45:46.932529 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:45:46.932504 2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-autores-b1f490\"" Sep 10 13:45:46.937856 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:45:46.937827 2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-autores-b1f490-5f54877b85-4d97t"] Sep 10 13:45:46.957115 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:45:46.957083 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fxgmp_fa53c783-d40a-4024-9f7b-1d0d2b57f6dc/konnectivity-agent/0.log" limit=1024 Sep 10 13:45:46.974481 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:45:46.974455 2585 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-autores-b1f490-77f869c5bb-458gp"] Sep 10 13:45:46.977878 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:45:46.977862 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-autores-b1f490-77f869c5bb-458gp" Sep 10 13:45:46.983106 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:45:46.983080 2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-autores-b1f490-dockercfg-wqlh9\"" Sep 10 13:45:46.992784 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:45:46.992758 2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-autores-b1f490-77f869c5bb-458gp"] Sep 10 13:45:47.019836 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:45:47.019799 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gw2tk\" (UniqueName: \"kubernetes.io/projected/c7db93d0-3329-45c0-99c8-18690e72647c-kube-api-access-gw2tk\") pod \"payload-processing-e2e-autores-b1f490-5f54877b85-4d97t\" (UID: \"c7db93d0-3329-45c0-99c8-18690e72647c\") " pod="openshift-ingress/payload-processing-e2e-autores-b1f490-5f54877b85-4d97t" Sep 10 13:45:47.019999 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:45:47.019846 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c7db93d0-3329-45c0-99c8-18690e72647c-plugins-config-volume\") pod \"payload-processing-e2e-autores-b1f490-5f54877b85-4d97t\" (UID: \"c7db93d0-3329-45c0-99c8-18690e72647c\") " pod="openshift-ingress/payload-processing-e2e-autores-b1f490-5f54877b85-4d97t" Sep 10 13:45:47.121157 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:45:47.121068 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-72vx9\" (UniqueName: \"kubernetes.io/projected/ae3341b0-646b-4447-aeac-4a49bcdac1ce-kube-api-access-72vx9\") pod \"payload-pre-processing-e2e-autores-b1f490-77f869c5bb-458gp\" (UID: \"ae3341b0-646b-4447-aeac-4a49bcdac1ce\") " pod="openshift-ingress/payload-pre-processing-e2e-autores-b1f490-77f869c5bb-458gp" Sep 10 13:45:47.121157 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:45:47.121119 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gw2tk\" (UniqueName: \"kubernetes.io/projected/c7db93d0-3329-45c0-99c8-18690e72647c-kube-api-access-gw2tk\") pod \"payload-processing-e2e-autores-b1f490-5f54877b85-4d97t\" (UID: \"c7db93d0-3329-45c0-99c8-18690e72647c\") " pod="openshift-ingress/payload-processing-e2e-autores-b1f490-5f54877b85-4d97t" Sep 10 13:45:47.121382 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:45:47.121177 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c7db93d0-3329-45c0-99c8-18690e72647c-plugins-config-volume\") pod \"payload-processing-e2e-autores-b1f490-5f54877b85-4d97t\" (UID: \"c7db93d0-3329-45c0-99c8-18690e72647c\") " pod="openshift-ingress/payload-processing-e2e-autores-b1f490-5f54877b85-4d97t" Sep 10 13:45:47.121382 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:45:47.121229 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ae3341b0-646b-4447-aeac-4a49bcdac1ce-plugins-config-volume\") pod \"payload-pre-processing-e2e-autores-b1f490-77f869c5bb-458gp\" (UID: \"ae3341b0-646b-4447-aeac-4a49bcdac1ce\") " pod="openshift-ingress/payload-pre-processing-e2e-autores-b1f490-77f869c5bb-458gp" Sep 10 13:45:47.122044 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:45:47.122024 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c7db93d0-3329-45c0-99c8-18690e72647c-plugins-config-volume\") pod \"payload-processing-e2e-autores-b1f490-5f54877b85-4d97t\" (UID: \"c7db93d0-3329-45c0-99c8-18690e72647c\") " pod="openshift-ingress/payload-processing-e2e-autores-b1f490-5f54877b85-4d97t" Sep 10 13:45:47.145297 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:45:47.145267 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-gw2tk\" (UniqueName: \"kubernetes.io/projected/c7db93d0-3329-45c0-99c8-18690e72647c-kube-api-access-gw2tk\") pod \"payload-processing-e2e-autores-b1f490-5f54877b85-4d97t\" (UID: \"c7db93d0-3329-45c0-99c8-18690e72647c\") " pod="openshift-ingress/payload-processing-e2e-autores-b1f490-5f54877b85-4d97t" Sep 10 13:45:47.222008 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:45:47.221969 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ae3341b0-646b-4447-aeac-4a49bcdac1ce-plugins-config-volume\") pod \"payload-pre-processing-e2e-autores-b1f490-77f869c5bb-458gp\" (UID: \"ae3341b0-646b-4447-aeac-4a49bcdac1ce\") " pod="openshift-ingress/payload-pre-processing-e2e-autores-b1f490-77f869c5bb-458gp" Sep 10 13:45:47.222153 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:45:47.222041 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-72vx9\" (UniqueName: \"kubernetes.io/projected/ae3341b0-646b-4447-aeac-4a49bcdac1ce-kube-api-access-72vx9\") pod \"payload-pre-processing-e2e-autores-b1f490-77f869c5bb-458gp\" (UID: \"ae3341b0-646b-4447-aeac-4a49bcdac1ce\") " pod="openshift-ingress/payload-pre-processing-e2e-autores-b1f490-77f869c5bb-458gp" Sep 10 13:45:47.222794 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:45:47.222774 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ae3341b0-646b-4447-aeac-4a49bcdac1ce-plugins-config-volume\") pod \"payload-pre-processing-e2e-autores-b1f490-77f869c5bb-458gp\" (UID: \"ae3341b0-646b-4447-aeac-4a49bcdac1ce\") " pod="openshift-ingress/payload-pre-processing-e2e-autores-b1f490-77f869c5bb-458gp" Sep 10 13:45:47.243449 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:45:47.243412 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-72vx9\" (UniqueName: \"kubernetes.io/projected/ae3341b0-646b-4447-aeac-4a49bcdac1ce-kube-api-access-72vx9\") pod \"payload-pre-processing-e2e-autores-b1f490-77f869c5bb-458gp\" (UID: \"ae3341b0-646b-4447-aeac-4a49bcdac1ce\") " pod="openshift-ingress/payload-pre-processing-e2e-autores-b1f490-77f869c5bb-458gp" Sep 10 13:45:47.243583 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:45:47.243535 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-autores-b1f490-5f54877b85-4d97t" Sep 10 13:45:47.288617 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:45:47.288584 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-autores-b1f490-77f869c5bb-458gp" Sep 10 13:45:47.523620 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:45:47.523591 2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-autores-b1f490-5f54877b85-4d97t"] Sep 10 13:45:47.525319 ip-10-0-140-8 kubenswrapper[2585]: W0910 13:45:47.525295 2585 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc7db93d0_3329_45c0_99c8_18690e72647c.slice/crio-aaaad59c880fcb9daed251da7871ff6c768ab25c182fce04feefaebf336b54ec WatchSource:0}: Error finding container aaaad59c880fcb9daed251da7871ff6c768ab25c182fce04feefaebf336b54ec: Status 404 returned error can't find the container with id aaaad59c880fcb9daed251da7871ff6c768ab25c182fce04feefaebf336b54ec Sep 10 13:45:47.601110 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:45:47.601082 2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-autores-b1f490-77f869c5bb-458gp"] Sep 10 13:45:47.601876 ip-10-0-140-8 kubenswrapper[2585]: W0910 13:45:47.601855 2585 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podae3341b0_646b_4447_aeac_4a49bcdac1ce.slice/crio-15054aa446a63305eee7def37b85aff081f3757381f0baf9e9f4f8e8a1efe4f0 WatchSource:0}: Error finding container 15054aa446a63305eee7def37b85aff081f3757381f0baf9e9f4f8e8a1efe4f0: Status 404 returned error can't find the container with id 15054aa446a63305eee7def37b85aff081f3757381f0baf9e9f4f8e8a1efe4f0 Sep 10 13:45:48.158656 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:45:48.158613 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-autores-b1f490-77f869c5bb-458gp" event={"ID":"ae3341b0-646b-4447-aeac-4a49bcdac1ce","Type":"ContainerStarted","Data":"f7f612e5f72d41b2661cc57a9b027b1d95e17f1cdd387bce36c4253d4103c945"} Sep 10 13:45:48.159112 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:45:48.158684 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-autores-b1f490-77f869c5bb-458gp" event={"ID":"ae3341b0-646b-4447-aeac-4a49bcdac1ce","Type":"ContainerStarted","Data":"15054aa446a63305eee7def37b85aff081f3757381f0baf9e9f4f8e8a1efe4f0"} Sep 10 13:45:48.159112 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:45:48.158827 2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-autores-b1f490-77f869c5bb-458gp" Sep 10 13:45:48.160310 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:45:48.160286 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-autores-b1f490-5f54877b85-4d97t" event={"ID":"c7db93d0-3329-45c0-99c8-18690e72647c","Type":"ContainerStarted","Data":"5d9e69ea4932fa6cc6e62697791471d3a2f2994f8f281248907ce62ed925f305"} Sep 10 13:45:48.160409 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:45:48.160328 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-autores-b1f490-5f54877b85-4d97t" event={"ID":"c7db93d0-3329-45c0-99c8-18690e72647c","Type":"ContainerStarted","Data":"aaaad59c880fcb9daed251da7871ff6c768ab25c182fce04feefaebf336b54ec"} Sep 10 13:45:48.160409 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:45:48.160356 2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-autores-b1f490-5f54877b85-4d97t" Sep 10 13:45:48.182503 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:45:48.182139 2585 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-autores-b1f490-77f869c5bb-458gp" podStartSLOduration=2.182119705 podStartE2EDuration="2.182119705s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 13:45:46 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 13:45:48.181800253 +0000 UTC m=+1033.054504456" watchObservedRunningTime="2026-09-10 13:45:48.182119705 +0000 UTC m=+1033.054823913" Sep 10 13:45:48.215330 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:45:48.215276 2585 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-autores-b1f490-5f54877b85-4d97t" podStartSLOduration=2.215258244 podStartE2EDuration="2.215258244s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 13:45:46 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 13:45:48.213000546 +0000 UTC m=+1033.085704757" watchObservedRunningTime="2026-09-10 13:45:48.215258244 +0000 UTC m=+1033.087962444" Sep 10 13:45:48.579142 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:45:48.579114 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fxgmp_fa53c783-d40a-4024-9f7b-1d0d2b57f6dc/konnectivity-agent/0.log" limit=1024 Sep 10 13:45:48.686074 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:45:48.686039 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fxgmp_fa53c783-d40a-4024-9f7b-1d0d2b57f6dc/konnectivity-agent/0.log" limit=1024 Sep 10 13:45:49.616319 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:45:49.616278 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fxgmp_fa53c783-d40a-4024-9f7b-1d0d2b57f6dc/konnectivity-agent/0.log" limit=1024 Sep 10 13:45:49.827063 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:45:49.827031 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fxgmp_fa53c783-d40a-4024-9f7b-1d0d2b57f6dc/konnectivity-agent/0.log" limit=1024 Sep 10 13:45:51.109536 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:45:51.109504 2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-inf-a-86fdcf-6cb885c959-6vx6m" Sep 10 13:45:54.014292 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:45:54.014234 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fxgmp_fa53c783-d40a-4024-9f7b-1d0d2b57f6dc/konnectivity-agent/0.log" limit=1024 Sep 10 13:45:54.197013 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:45:54.196966 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fxgmp_fa53c783-d40a-4024-9f7b-1d0d2b57f6dc/konnectivity-agent/0.log" limit=1024 Sep 10 13:45:54.328395 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:45:54.328367 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fxgmp_fa53c783-d40a-4024-9f7b-1d0d2b57f6dc/konnectivity-agent/0.log" limit=1024 Sep 10 13:45:54.558955 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:45:54.558919 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fxgmp_fa53c783-d40a-4024-9f7b-1d0d2b57f6dc/konnectivity-agent/0.log" limit=1024 Sep 10 13:45:55.595462 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:45:55.595431 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fxgmp_fa53c783-d40a-4024-9f7b-1d0d2b57f6dc/konnectivity-agent/0.log" limit=1024 Sep 10 13:45:55.749069 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:45:55.749034 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fxgmp_fa53c783-d40a-4024-9f7b-1d0d2b57f6dc/konnectivity-agent/0.log" limit=1024 Sep 10 13:45:57.331945 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:45:57.331916 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fxgmp_fa53c783-d40a-4024-9f7b-1d0d2b57f6dc/konnectivity-agent/0.log" limit=1024 Sep 10 13:45:59.167781 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:45:59.167743 2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-autores-b1f490-5f54877b85-4d97t" Sep 10 13:45:59.168161 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:45:59.167918 2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-autores-b1f490-77f869c5bb-458gp" Sep 10 13:45:59.291914 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:45:59.291875 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fxgmp_fa53c783-d40a-4024-9f7b-1d0d2b57f6dc/konnectivity-agent/0.log" limit=1024 Sep 10 13:45:59.647126 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:45:59.647092 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fxgmp_fa53c783-d40a-4024-9f7b-1d0d2b57f6dc/konnectivity-agent/0.log" limit=1024 Sep 10 13:46:00.968377 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:46:00.968344 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fxgmp_fa53c783-d40a-4024-9f7b-1d0d2b57f6dc/konnectivity-agent/0.log" limit=1024 Sep 10 13:46:01.107125 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:46:01.107081 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fxgmp_fa53c783-d40a-4024-9f7b-1d0d2b57f6dc/konnectivity-agent/0.log" limit=1024 Sep 10 13:46:01.391359 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:46:01.391326 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fxgmp_fa53c783-d40a-4024-9f7b-1d0d2b57f6dc/konnectivity-agent/0.log" limit=1024 Sep 10 13:46:02.057868 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:46:02.057834 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fxgmp_fa53c783-d40a-4024-9f7b-1d0d2b57f6dc/konnectivity-agent/0.log" limit=1024 Sep 10 13:46:02.425422 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:46:02.425393 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fxgmp_fa53c783-d40a-4024-9f7b-1d0d2b57f6dc/konnectivity-agent/0.log" limit=1024 Sep 10 13:46:02.826059 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:46:02.826029 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fxgmp_fa53c783-d40a-4024-9f7b-1d0d2b57f6dc/konnectivity-agent/0.log" limit=1024 Sep 10 13:46:03.101921 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:46:03.101836 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fxgmp_fa53c783-d40a-4024-9f7b-1d0d2b57f6dc/konnectivity-agent/0.log" limit=1024 Sep 10 13:46:06.263350 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:46:06.263316 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fxgmp_fa53c783-d40a-4024-9f7b-1d0d2b57f6dc/konnectivity-agent/0.log" limit=1024 Sep 10 13:46:06.379295 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:46:06.379264 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fxgmp_fa53c783-d40a-4024-9f7b-1d0d2b57f6dc/konnectivity-agent/0.log" limit=1024 Sep 10 13:46:06.611613 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:46:06.611580 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fxgmp_fa53c783-d40a-4024-9f7b-1d0d2b57f6dc/konnectivity-agent/0.log" limit=1024 Sep 10 13:46:07.547858 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:46:07.547828 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fxgmp_fa53c783-d40a-4024-9f7b-1d0d2b57f6dc/konnectivity-agent/0.log" limit=1024 Sep 10 13:46:08.884050 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:46:08.884013 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fxgmp_fa53c783-d40a-4024-9f7b-1d0d2b57f6dc/konnectivity-agent/0.log" limit=1024 Sep 10 13:46:08.989067 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:46:08.989034 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fxgmp_fa53c783-d40a-4024-9f7b-1d0d2b57f6dc/konnectivity-agent/0.log" limit=1024 Sep 10 13:46:09.387281 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:46:09.387222 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fxgmp_fa53c783-d40a-4024-9f7b-1d0d2b57f6dc/konnectivity-agent/0.log" limit=1024 Sep 10 13:46:09.776322 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:46:09.776224 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fxgmp_fa53c783-d40a-4024-9f7b-1d0d2b57f6dc/konnectivity-agent/0.log" limit=1024 Sep 10 13:46:09.890857 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:46:09.890827 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fxgmp_fa53c783-d40a-4024-9f7b-1d0d2b57f6dc/konnectivity-agent/0.log" limit=1024 Sep 10 13:46:09.992945 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:46:09.992913 2585 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-02fc2e-84df4b4db-trkgz"] Sep 10 13:46:10.001387 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:46:10.001363 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-02fc2e-84df4b4db-trkgz" Sep 10 13:46:10.004087 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:46:10.004063 2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-inf-b-02fc2e\"" Sep 10 13:46:10.004186 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:46:10.004065 2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-inf-b-02fc2e-dockercfg-59jgf\"" Sep 10 13:46:10.009990 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:46:10.009966 2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-02fc2e-84df4b4db-trkgz"] Sep 10 13:46:10.062107 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:46:10.062076 2585 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-02fc2e-c6594bd4c-bj5bk"] Sep 10 13:46:10.066102 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:46:10.066087 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-02fc2e-c6594bd4c-bj5bk" Sep 10 13:46:10.082065 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:46:10.082036 2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-02fc2e-c6594bd4c-bj5bk"] Sep 10 13:46:10.130828 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:46:10.130795 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gxgg6\" (UniqueName: \"kubernetes.io/projected/ea137a2a-3a6e-47dd-835c-fb1e1958e262-kube-api-access-gxgg6\") pod \"payload-pre-processing-e2e-inf-b-02fc2e-84df4b4db-trkgz\" (UID: \"ea137a2a-3a6e-47dd-835c-fb1e1958e262\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-02fc2e-84df4b4db-trkgz" Sep 10 13:46:10.130992 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:46:10.130860 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ea137a2a-3a6e-47dd-835c-fb1e1958e262-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-b-02fc2e-84df4b4db-trkgz\" (UID: \"ea137a2a-3a6e-47dd-835c-fb1e1958e262\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-02fc2e-84df4b4db-trkgz" Sep 10 13:46:10.150862 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:46:10.150837 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fxgmp_fa53c783-d40a-4024-9f7b-1d0d2b57f6dc/konnectivity-agent/0.log" limit=1024 Sep 10 13:46:10.231422 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:46:10.231393 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gxgg6\" (UniqueName: \"kubernetes.io/projected/ea137a2a-3a6e-47dd-835c-fb1e1958e262-kube-api-access-gxgg6\") pod \"payload-pre-processing-e2e-inf-b-02fc2e-84df4b4db-trkgz\" (UID: \"ea137a2a-3a6e-47dd-835c-fb1e1958e262\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-02fc2e-84df4b4db-trkgz" Sep 10 13:46:10.231601 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:46:10.231476 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f3447c97-b582-4158-8a31-90c3a207f0f4-plugins-config-volume\") pod \"payload-processing-e2e-inf-b-02fc2e-c6594bd4c-bj5bk\" (UID: \"f3447c97-b582-4158-8a31-90c3a207f0f4\") " pod="openshift-ingress/payload-processing-e2e-inf-b-02fc2e-c6594bd4c-bj5bk" Sep 10 13:46:10.231601 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:46:10.231502 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ea137a2a-3a6e-47dd-835c-fb1e1958e262-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-b-02fc2e-84df4b4db-trkgz\" (UID: \"ea137a2a-3a6e-47dd-835c-fb1e1958e262\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-02fc2e-84df4b4db-trkgz" Sep 10 13:46:10.231601 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:46:10.231537 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dvns6\" (UniqueName: \"kubernetes.io/projected/f3447c97-b582-4158-8a31-90c3a207f0f4-kube-api-access-dvns6\") pod \"payload-processing-e2e-inf-b-02fc2e-c6594bd4c-bj5bk\" (UID: \"f3447c97-b582-4158-8a31-90c3a207f0f4\") " pod="openshift-ingress/payload-processing-e2e-inf-b-02fc2e-c6594bd4c-bj5bk" Sep 10 13:46:10.232297 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:46:10.232268 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ea137a2a-3a6e-47dd-835c-fb1e1958e262-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-b-02fc2e-84df4b4db-trkgz\" (UID: \"ea137a2a-3a6e-47dd-835c-fb1e1958e262\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-02fc2e-84df4b4db-trkgz" Sep 10 13:46:10.245542 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:46:10.245514 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-gxgg6\" (UniqueName: \"kubernetes.io/projected/ea137a2a-3a6e-47dd-835c-fb1e1958e262-kube-api-access-gxgg6\") pod \"payload-pre-processing-e2e-inf-b-02fc2e-84df4b4db-trkgz\" (UID: \"ea137a2a-3a6e-47dd-835c-fb1e1958e262\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-02fc2e-84df4b4db-trkgz" Sep 10 13:46:10.313167 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:46:10.313083 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-02fc2e-84df4b4db-trkgz" Sep 10 13:46:10.332222 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:46:10.332191 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f3447c97-b582-4158-8a31-90c3a207f0f4-plugins-config-volume\") pod \"payload-processing-e2e-inf-b-02fc2e-c6594bd4c-bj5bk\" (UID: \"f3447c97-b582-4158-8a31-90c3a207f0f4\") " pod="openshift-ingress/payload-processing-e2e-inf-b-02fc2e-c6594bd4c-bj5bk" Sep 10 13:46:10.332381 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:46:10.332287 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dvns6\" (UniqueName: \"kubernetes.io/projected/f3447c97-b582-4158-8a31-90c3a207f0f4-kube-api-access-dvns6\") pod \"payload-processing-e2e-inf-b-02fc2e-c6594bd4c-bj5bk\" (UID: \"f3447c97-b582-4158-8a31-90c3a207f0f4\") " pod="openshift-ingress/payload-processing-e2e-inf-b-02fc2e-c6594bd4c-bj5bk" Sep 10 13:46:10.332846 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:46:10.332820 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f3447c97-b582-4158-8a31-90c3a207f0f4-plugins-config-volume\") pod \"payload-processing-e2e-inf-b-02fc2e-c6594bd4c-bj5bk\" (UID: \"f3447c97-b582-4158-8a31-90c3a207f0f4\") " pod="openshift-ingress/payload-processing-e2e-inf-b-02fc2e-c6594bd4c-bj5bk" Sep 10 13:46:10.344495 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:46:10.344468 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-dvns6\" (UniqueName: \"kubernetes.io/projected/f3447c97-b582-4158-8a31-90c3a207f0f4-kube-api-access-dvns6\") pod \"payload-processing-e2e-inf-b-02fc2e-c6594bd4c-bj5bk\" (UID: \"f3447c97-b582-4158-8a31-90c3a207f0f4\") " pod="openshift-ingress/payload-processing-e2e-inf-b-02fc2e-c6594bd4c-bj5bk" Sep 10 13:46:10.376736 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:46:10.376597 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-02fc2e-c6594bd4c-bj5bk" Sep 10 13:46:10.476113 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:46:10.476082 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fxgmp_fa53c783-d40a-4024-9f7b-1d0d2b57f6dc/konnectivity-agent/0.log" limit=1024 Sep 10 13:46:10.605395 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:46:10.605369 2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-02fc2e-84df4b4db-trkgz"] Sep 10 13:46:10.606377 ip-10-0-140-8 kubenswrapper[2585]: W0910 13:46:10.606354 2585 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podea137a2a_3a6e_47dd_835c_fb1e1958e262.slice/crio-ef897824ea311365c73010486366f319c8ebc2a28c6cf876714c6fd165461232 WatchSource:0}: Error finding container ef897824ea311365c73010486366f319c8ebc2a28c6cf876714c6fd165461232: Status 404 returned error can't find the container with id ef897824ea311365c73010486366f319c8ebc2a28c6cf876714c6fd165461232 Sep 10 13:46:10.677210 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:46:10.677178 2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-02fc2e-c6594bd4c-bj5bk"] Sep 10 13:46:10.678147 ip-10-0-140-8 kubenswrapper[2585]: W0910 13:46:10.678121 2585 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf3447c97_b582_4158_8a31_90c3a207f0f4.slice/crio-15fadacce6f16eb4b389a24f35e9337abd378ca32aeda001ecd591ec1efd6bd2 WatchSource:0}: Error finding container 15fadacce6f16eb4b389a24f35e9337abd378ca32aeda001ecd591ec1efd6bd2: Status 404 returned error can't find the container with id 15fadacce6f16eb4b389a24f35e9337abd378ca32aeda001ecd591ec1efd6bd2 Sep 10 13:46:11.288846 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:46:11.288788 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-02fc2e-84df4b4db-trkgz" event={"ID":"ea137a2a-3a6e-47dd-835c-fb1e1958e262","Type":"ContainerStarted","Data":"2e2cf1249869fcbfb12e625547c16c13241de951c8cf170eef922640e613b3ce"} Sep 10 13:46:11.289349 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:46:11.288851 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-02fc2e-84df4b4db-trkgz" event={"ID":"ea137a2a-3a6e-47dd-835c-fb1e1958e262","Type":"ContainerStarted","Data":"ef897824ea311365c73010486366f319c8ebc2a28c6cf876714c6fd165461232"} Sep 10 13:46:11.289349 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:46:11.288960 2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-02fc2e-84df4b4db-trkgz" Sep 10 13:46:11.290631 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:46:11.290608 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-02fc2e-c6594bd4c-bj5bk" event={"ID":"f3447c97-b582-4158-8a31-90c3a207f0f4","Type":"ContainerStarted","Data":"230c4643c328de6f3997b015365c8701928136566cb5997b889f3a4a82906d7f"} Sep 10 13:46:11.290758 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:46:11.290644 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-02fc2e-c6594bd4c-bj5bk" event={"ID":"f3447c97-b582-4158-8a31-90c3a207f0f4","Type":"ContainerStarted","Data":"15fadacce6f16eb4b389a24f35e9337abd378ca32aeda001ecd591ec1efd6bd2"} Sep 10 13:46:11.290758 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:46:11.290723 2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-inf-b-02fc2e-c6594bd4c-bj5bk" Sep 10 13:46:11.325285 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:46:11.325213 2585 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-02fc2e-84df4b4db-trkgz" podStartSLOduration=2.325196046 podStartE2EDuration="2.325196046s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 13:46:09 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 13:46:11.323518568 +0000 UTC m=+1056.196222768" watchObservedRunningTime="2026-09-10 13:46:11.325196046 +0000 UTC m=+1056.197900247" Sep 10 13:46:11.632257 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:46:11.632220 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fxgmp_fa53c783-d40a-4024-9f7b-1d0d2b57f6dc/konnectivity-agent/0.log" limit=1024 Sep 10 13:46:12.525253 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:46:12.525198 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fxgmp_fa53c783-d40a-4024-9f7b-1d0d2b57f6dc/konnectivity-agent/0.log" limit=1024 Sep 10 13:46:12.699501 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:46:12.699445 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fxgmp_fa53c783-d40a-4024-9f7b-1d0d2b57f6dc/konnectivity-agent/0.log" limit=1024 Sep 10 13:46:12.797114 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:46:12.797008 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fxgmp_fa53c783-d40a-4024-9f7b-1d0d2b57f6dc/konnectivity-agent/0.log" limit=1024 Sep 10 13:46:12.975982 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:46:12.975934 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fxgmp_fa53c783-d40a-4024-9f7b-1d0d2b57f6dc/konnectivity-agent/0.log" limit=1024 Sep 10 13:46:13.080494 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:46:13.080457 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fxgmp_fa53c783-d40a-4024-9f7b-1d0d2b57f6dc/konnectivity-agent/0.log" limit=1024 Sep 10 13:46:19.966170 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:46:19.966134 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fxgmp_fa53c783-d40a-4024-9f7b-1d0d2b57f6dc/konnectivity-agent/0.log" limit=1024 Sep 10 13:46:20.191914 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:46:20.191882 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fxgmp_fa53c783-d40a-4024-9f7b-1d0d2b57f6dc/konnectivity-agent/0.log" limit=1024 Sep 10 13:46:22.298272 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:46:22.298229 2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-02fc2e-84df4b4db-trkgz" Sep 10 13:46:22.298649 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:46:22.298322 2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-inf-b-02fc2e-c6594bd4c-bj5bk" Sep 10 13:46:22.351512 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:46:22.351454 2585 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-inf-b-02fc2e-c6594bd4c-bj5bk" podStartSLOduration=12.35143948 podStartE2EDuration="12.35143948s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 13:46:10 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 13:46:11.354556035 +0000 UTC m=+1056.227260217" watchObservedRunningTime="2026-09-10 13:46:22.35143948 +0000 UTC m=+1067.224143716" Sep 10 13:46:28.839233 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:46:28.839149 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fxgmp_fa53c783-d40a-4024-9f7b-1d0d2b57f6dc/konnectivity-agent/0.log" limit=1024 Sep 10 13:46:28.931273 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:46:28.931198 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fxgmp_fa53c783-d40a-4024-9f7b-1d0d2b57f6dc/konnectivity-agent/0.log" limit=1024 Sep 10 13:46:29.081494 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:46:29.081454 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fxgmp_fa53c783-d40a-4024-9f7b-1d0d2b57f6dc/konnectivity-agent/0.log" limit=1024 Sep 10 13:46:29.182765 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:46:29.182653 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fxgmp_fa53c783-d40a-4024-9f7b-1d0d2b57f6dc/konnectivity-agent/0.log" limit=1024 Sep 10 13:46:29.263300 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:46:29.263236 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fxgmp_fa53c783-d40a-4024-9f7b-1d0d2b57f6dc/konnectivity-agent/0.log" limit=1024 Sep 10 13:46:29.389654 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:46:29.389619 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fxgmp_fa53c783-d40a-4024-9f7b-1d0d2b57f6dc/konnectivity-agent/0.log" limit=1024 Sep 10 13:46:29.871125 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:46:29.871079 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fxgmp_fa53c783-d40a-4024-9f7b-1d0d2b57f6dc/konnectivity-agent/0.log" limit=1024 Sep 10 13:46:29.972453 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:46:29.972416 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fxgmp_fa53c783-d40a-4024-9f7b-1d0d2b57f6dc/konnectivity-agent/0.log" limit=1024 Sep 10 13:46:30.128055 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:46:30.127951 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fxgmp_fa53c783-d40a-4024-9f7b-1d0d2b57f6dc/konnectivity-agent/0.log" limit=1024 Sep 10 13:46:30.393464 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:46:30.393369 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fxgmp_fa53c783-d40a-4024-9f7b-1d0d2b57f6dc/konnectivity-agent/0.log" limit=1024 Sep 10 13:46:30.495464 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:46:30.495435 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fxgmp_fa53c783-d40a-4024-9f7b-1d0d2b57f6dc/konnectivity-agent/0.log" limit=1024 Sep 10 13:46:31.182455 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:46:31.182393 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fxgmp_fa53c783-d40a-4024-9f7b-1d0d2b57f6dc/konnectivity-agent/0.log" limit=1024 Sep 10 13:46:33.715989 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:46:33.715956 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fxgmp_fa53c783-d40a-4024-9f7b-1d0d2b57f6dc/konnectivity-agent/0.log" limit=1024 Sep 10 13:46:33.817729 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:46:33.817692 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fxgmp_fa53c783-d40a-4024-9f7b-1d0d2b57f6dc/konnectivity-agent/0.log" limit=1024 Sep 10 13:46:33.916729 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:46:33.916689 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fxgmp_fa53c783-d40a-4024-9f7b-1d0d2b57f6dc/konnectivity-agent/0.log" limit=1024 Sep 10 13:46:34.025456 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:46:34.025380 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fxgmp_fa53c783-d40a-4024-9f7b-1d0d2b57f6dc/konnectivity-agent/0.log" limit=1024 Sep 10 13:46:34.121326 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:46:34.121294 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fxgmp_fa53c783-d40a-4024-9f7b-1d0d2b57f6dc/konnectivity-agent/0.log" limit=1024 Sep 10 13:46:34.513961 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:46:34.513934 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fxgmp_fa53c783-d40a-4024-9f7b-1d0d2b57f6dc/konnectivity-agent/0.log" limit=1024 Sep 10 13:46:34.612105 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:46:34.612070 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fxgmp_fa53c783-d40a-4024-9f7b-1d0d2b57f6dc/konnectivity-agent/0.log" limit=1024 Sep 10 13:46:34.645037 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:46:34.645005 2585 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-migrate-0e325272-55b8d48bcd-fc72c"] Sep 10 13:46:34.648703 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:46:34.648674 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-migrate-0e325272-55b8d48bcd-fc72c" Sep 10 13:46:34.652896 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:46:34.652877 2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-migrate-0e325272\"" Sep 10 13:46:34.661054 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:46:34.661030 2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-migrate-0e325272-55b8d48bcd-fc72c"] Sep 10 13:46:34.760974 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:46:34.760941 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/64d5b4ca-e477-409c-8962-f967a6abfe95-plugins-config-volume\") pod \"payload-processing-e2e-migrate-0e325272-55b8d48bcd-fc72c\" (UID: \"64d5b4ca-e477-409c-8962-f967a6abfe95\") " pod="openshift-ingress/payload-processing-e2e-migrate-0e325272-55b8d48bcd-fc72c" Sep 10 13:46:34.761474 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:46:34.761111 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kdtc5\" (UniqueName: \"kubernetes.io/projected/64d5b4ca-e477-409c-8962-f967a6abfe95-kube-api-access-kdtc5\") pod \"payload-processing-e2e-migrate-0e325272-55b8d48bcd-fc72c\" (UID: \"64d5b4ca-e477-409c-8962-f967a6abfe95\") " pod="openshift-ingress/payload-processing-e2e-migrate-0e325272-55b8d48bcd-fc72c" Sep 10 13:46:34.861968 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:46:34.861930 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kdtc5\" (UniqueName: \"kubernetes.io/projected/64d5b4ca-e477-409c-8962-f967a6abfe95-kube-api-access-kdtc5\") pod \"payload-processing-e2e-migrate-0e325272-55b8d48bcd-fc72c\" (UID: \"64d5b4ca-e477-409c-8962-f967a6abfe95\") " pod="openshift-ingress/payload-processing-e2e-migrate-0e325272-55b8d48bcd-fc72c" Sep 10 13:46:34.862141 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:46:34.861991 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/64d5b4ca-e477-409c-8962-f967a6abfe95-plugins-config-volume\") pod \"payload-processing-e2e-migrate-0e325272-55b8d48bcd-fc72c\" (UID: \"64d5b4ca-e477-409c-8962-f967a6abfe95\") " pod="openshift-ingress/payload-processing-e2e-migrate-0e325272-55b8d48bcd-fc72c" Sep 10 13:46:34.862670 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:46:34.862651 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/64d5b4ca-e477-409c-8962-f967a6abfe95-plugins-config-volume\") pod \"payload-processing-e2e-migrate-0e325272-55b8d48bcd-fc72c\" (UID: \"64d5b4ca-e477-409c-8962-f967a6abfe95\") " pod="openshift-ingress/payload-processing-e2e-migrate-0e325272-55b8d48bcd-fc72c" Sep 10 13:46:34.873857 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:46:34.873831 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-kdtc5\" (UniqueName: \"kubernetes.io/projected/64d5b4ca-e477-409c-8962-f967a6abfe95-kube-api-access-kdtc5\") pod \"payload-processing-e2e-migrate-0e325272-55b8d48bcd-fc72c\" (UID: \"64d5b4ca-e477-409c-8962-f967a6abfe95\") " pod="openshift-ingress/payload-processing-e2e-migrate-0e325272-55b8d48bcd-fc72c" Sep 10 13:46:34.960845 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:46:34.960804 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-migrate-0e325272-55b8d48bcd-fc72c" Sep 10 13:46:35.241514 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:46:35.241482 2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-migrate-0e325272-55b8d48bcd-fc72c"] Sep 10 13:46:35.242853 ip-10-0-140-8 kubenswrapper[2585]: W0910 13:46:35.242822 2585 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod64d5b4ca_e477_409c_8962_f967a6abfe95.slice/crio-9d893a9233d128de67cc4c388a6720fdbd62953c394b5b00549879dffe3e6bd5 WatchSource:0}: Error finding container 9d893a9233d128de67cc4c388a6720fdbd62953c394b5b00549879dffe3e6bd5: Status 404 returned error can't find the container with id 9d893a9233d128de67cc4c388a6720fdbd62953c394b5b00549879dffe3e6bd5 Sep 10 13:46:35.421854 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:46:35.421816 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-migrate-0e325272-55b8d48bcd-fc72c" event={"ID":"64d5b4ca-e477-409c-8962-f967a6abfe95","Type":"ContainerStarted","Data":"01a391123cdcf904a01194f929dd1372ec7f4880c0132760ea33d004687c583c"} Sep 10 13:46:35.422016 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:46:35.421879 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-migrate-0e325272-55b8d48bcd-fc72c" event={"ID":"64d5b4ca-e477-409c-8962-f967a6abfe95","Type":"ContainerStarted","Data":"9d893a9233d128de67cc4c388a6720fdbd62953c394b5b00549879dffe3e6bd5"} Sep 10 13:46:35.422016 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:46:35.421919 2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-migrate-0e325272-55b8d48bcd-fc72c" Sep 10 13:46:35.450337 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:46:35.450283 2585 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-migrate-0e325272-55b8d48bcd-fc72c" podStartSLOduration=1.450264169 podStartE2EDuration="1.450264169s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 13:46:34 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 13:46:35.447432703 +0000 UTC m=+1080.320136904" watchObservedRunningTime="2026-09-10 13:46:35.450264169 +0000 UTC m=+1080.322968373" Sep 10 13:46:36.533026 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:46:36.532993 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fxgmp_fa53c783-d40a-4024-9f7b-1d0d2b57f6dc/konnectivity-agent/0.log" limit=1024 Sep 10 13:46:37.784861 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:46:37.784827 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fxgmp_fa53c783-d40a-4024-9f7b-1d0d2b57f6dc/konnectivity-agent/0.log" limit=1024 Sep 10 13:46:38.294565 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:46:38.294536 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fxgmp_fa53c783-d40a-4024-9f7b-1d0d2b57f6dc/konnectivity-agent/0.log" limit=1024 Sep 10 13:46:46.429661 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:46:46.429629 2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-migrate-0e325272-55b8d48bcd-fc72c" Sep 10 13:46:53.252101 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:46:53.252062 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fxgmp_fa53c783-d40a-4024-9f7b-1d0d2b57f6dc/konnectivity-agent/0.log" limit=1024 Sep 10 13:47:05.387440 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:47:05.387385 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fxgmp_fa53c783-d40a-4024-9f7b-1d0d2b57f6dc/konnectivity-agent/0.log" limit=1024 Sep 10 13:47:05.434939 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:47:05.434908 2585 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-migrate-0e325272-55b8d48bcd-fc72c"] Sep 10 13:47:05.435459 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:47:05.435424 2585 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-migrate-0e325272-55b8d48bcd-fc72c" podUID="64d5b4ca-e477-409c-8962-f967a6abfe95" containerName="payload-processing" containerID="cri-o://01a391123cdcf904a01194f929dd1372ec7f4880c0132760ea33d004687c583c" gracePeriod=30 Sep 10 13:47:05.588667 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:47:05.588635 2585 generic.go:356] "Generic (PLEG): container finished" podID="64d5b4ca-e477-409c-8962-f967a6abfe95" containerID="01a391123cdcf904a01194f929dd1372ec7f4880c0132760ea33d004687c583c" exitCode=0 Sep 10 13:47:05.588825 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:47:05.588711 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-migrate-0e325272-55b8d48bcd-fc72c" event={"ID":"64d5b4ca-e477-409c-8962-f967a6abfe95","Type":"ContainerDied","Data":"01a391123cdcf904a01194f929dd1372ec7f4880c0132760ea33d004687c583c"} Sep 10 13:47:05.695959 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:47:05.695935 2585 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-migrate-0e325272-55b8d48bcd-fc72c" Sep 10 13:47:05.757808 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:47:05.757767 2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-kdtc5\" (UniqueName: \"kubernetes.io/projected/64d5b4ca-e477-409c-8962-f967a6abfe95-kube-api-access-kdtc5\") pod \"64d5b4ca-e477-409c-8962-f967a6abfe95\" (UID: \"64d5b4ca-e477-409c-8962-f967a6abfe95\") " Sep 10 13:47:05.757989 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:47:05.757856 2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/64d5b4ca-e477-409c-8962-f967a6abfe95-plugins-config-volume\") pod \"64d5b4ca-e477-409c-8962-f967a6abfe95\" (UID: \"64d5b4ca-e477-409c-8962-f967a6abfe95\") " Sep 10 13:47:05.758293 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:47:05.758260 2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/64d5b4ca-e477-409c-8962-f967a6abfe95-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "64d5b4ca-e477-409c-8962-f967a6abfe95" (UID: "64d5b4ca-e477-409c-8962-f967a6abfe95"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 10 13:47:05.761131 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:47:05.761094 2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/64d5b4ca-e477-409c-8962-f967a6abfe95-kube-api-access-kdtc5" (OuterVolumeSpecName: "kube-api-access-kdtc5") pod "64d5b4ca-e477-409c-8962-f967a6abfe95" (UID: "64d5b4ca-e477-409c-8962-f967a6abfe95"). InnerVolumeSpecName "kube-api-access-kdtc5". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 10 13:47:05.858760 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:47:05.858728 2585 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/64d5b4ca-e477-409c-8962-f967a6abfe95-plugins-config-volume\") on node \"ip-10-0-140-8.ec2.internal\" DevicePath \"\"" Sep 10 13:47:05.858760 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:47:05.858754 2585 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kdtc5\" (UniqueName: \"kubernetes.io/projected/64d5b4ca-e477-409c-8962-f967a6abfe95-kube-api-access-kdtc5\") on node \"ip-10-0-140-8.ec2.internal\" DevicePath \"\"" Sep 10 13:47:06.595658 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:47:06.595631 2585 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-migrate-0e325272-55b8d48bcd-fc72c" Sep 10 13:47:06.596079 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:47:06.595631 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-migrate-0e325272-55b8d48bcd-fc72c" event={"ID":"64d5b4ca-e477-409c-8962-f967a6abfe95","Type":"ContainerDied","Data":"9d893a9233d128de67cc4c388a6720fdbd62953c394b5b00549879dffe3e6bd5"} Sep 10 13:47:06.596079 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:47:06.595765 2585 scope.go:117] "RemoveContainer" containerID="01a391123cdcf904a01194f929dd1372ec7f4880c0132760ea33d004687c583c" Sep 10 13:47:06.650633 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:47:06.650606 2585 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-migrate-0e325272-55b8d48bcd-fc72c"] Sep 10 13:47:06.662199 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:47:06.662169 2585 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-migrate-0e325272-55b8d48bcd-fc72c"] Sep 10 13:47:06.827821 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:47:06.827786 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fxgmp_fa53c783-d40a-4024-9f7b-1d0d2b57f6dc/konnectivity-agent/0.log" limit=1024 Sep 10 13:47:06.922531 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:47:06.922440 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fxgmp_fa53c783-d40a-4024-9f7b-1d0d2b57f6dc/konnectivity-agent/0.log" limit=1024 Sep 10 13:47:07.647333 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:47:07.647296 2585 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="64d5b4ca-e477-409c-8962-f967a6abfe95" path="/var/lib/kubelet/pods/64d5b4ca-e477-409c-8962-f967a6abfe95/volumes" Sep 10 13:47:09.816840 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:47:09.816801 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fxgmp_fa53c783-d40a-4024-9f7b-1d0d2b57f6dc/konnectivity-agent/0.log" limit=1024 Sep 10 13:47:09.904562 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:47:09.904526 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fxgmp_fa53c783-d40a-4024-9f7b-1d0d2b57f6dc/konnectivity-agent/0.log" limit=1024 Sep 10 13:47:11.073185 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:47:11.073142 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fxgmp_fa53c783-d40a-4024-9f7b-1d0d2b57f6dc/konnectivity-agent/0.log" limit=1024 Sep 10 13:47:11.152841 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:47:11.152798 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fxgmp_fa53c783-d40a-4024-9f7b-1d0d2b57f6dc/konnectivity-agent/0.log" limit=1024 Sep 10 13:47:11.261039 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:47:11.261004 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fxgmp_fa53c783-d40a-4024-9f7b-1d0d2b57f6dc/konnectivity-agent/0.log" limit=1024 Sep 10 13:47:11.494576 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:47:11.494471 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fxgmp_fa53c783-d40a-4024-9f7b-1d0d2b57f6dc/konnectivity-agent/0.log" limit=1024 Sep 10 13:47:12.127166 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:47:12.127137 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fxgmp_fa53c783-d40a-4024-9f7b-1d0d2b57f6dc/konnectivity-agent/0.log" limit=1024 Sep 10 13:47:12.264526 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:47:12.264494 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fxgmp_fa53c783-d40a-4024-9f7b-1d0d2b57f6dc/konnectivity-agent/0.log" limit=1024 Sep 10 13:47:12.397586 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:47:12.397511 2585 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-10e9e4e8-7494999d6f-8xkpn"] Sep 10 13:47:12.397919 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:47:12.397907 2585 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="64d5b4ca-e477-409c-8962-f967a6abfe95" containerName="payload-processing" Sep 10 13:47:12.397965 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:47:12.397921 2585 state_mem.go:107] "Deleted CPUSet assignment" podUID="64d5b4ca-e477-409c-8962-f967a6abfe95" containerName="payload-processing" Sep 10 13:47:12.398008 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:47:12.397999 2585 memory_manager.go:368] "RemoveStaleState removing state" podUID="64d5b4ca-e477-409c-8962-f967a6abfe95" containerName="payload-processing" Sep 10 13:47:12.402122 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:47:12.402101 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-10e9e4e8-7494999d6f-8xkpn" Sep 10 13:47:12.406687 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:47:12.406668 2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ait-10e9e4e8-dockercfg-bn7h2\"" Sep 10 13:47:12.406687 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:47:12.406681 2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ait-10e9e4e8\"" Sep 10 13:47:12.415938 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:47:12.415913 2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-10e9e4e8-7494999d6f-8xkpn"] Sep 10 13:47:12.520120 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:47:12.520084 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9t9zs\" (UniqueName: \"kubernetes.io/projected/7a21fdae-ffad-4907-9865-1cc4b3e1edb8-kube-api-access-9t9zs\") pod \"payload-processing-e2e-ait-10e9e4e8-7494999d6f-8xkpn\" (UID: \"7a21fdae-ffad-4907-9865-1cc4b3e1edb8\") " pod="openshift-ingress/payload-processing-e2e-ait-10e9e4e8-7494999d6f-8xkpn" Sep 10 13:47:12.520371 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:47:12.520145 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7a21fdae-ffad-4907-9865-1cc4b3e1edb8-plugins-config-volume\") pod \"payload-processing-e2e-ait-10e9e4e8-7494999d6f-8xkpn\" (UID: \"7a21fdae-ffad-4907-9865-1cc4b3e1edb8\") " pod="openshift-ingress/payload-processing-e2e-ait-10e9e4e8-7494999d6f-8xkpn" Sep 10 13:47:12.620833 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:47:12.620801 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7a21fdae-ffad-4907-9865-1cc4b3e1edb8-plugins-config-volume\") pod \"payload-processing-e2e-ait-10e9e4e8-7494999d6f-8xkpn\" (UID: \"7a21fdae-ffad-4907-9865-1cc4b3e1edb8\") " pod="openshift-ingress/payload-processing-e2e-ait-10e9e4e8-7494999d6f-8xkpn" Sep 10 13:47:12.620984 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:47:12.620894 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9t9zs\" (UniqueName: \"kubernetes.io/projected/7a21fdae-ffad-4907-9865-1cc4b3e1edb8-kube-api-access-9t9zs\") pod \"payload-processing-e2e-ait-10e9e4e8-7494999d6f-8xkpn\" (UID: \"7a21fdae-ffad-4907-9865-1cc4b3e1edb8\") " pod="openshift-ingress/payload-processing-e2e-ait-10e9e4e8-7494999d6f-8xkpn" Sep 10 13:47:12.621491 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:47:12.621471 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7a21fdae-ffad-4907-9865-1cc4b3e1edb8-plugins-config-volume\") pod \"payload-processing-e2e-ait-10e9e4e8-7494999d6f-8xkpn\" (UID: \"7a21fdae-ffad-4907-9865-1cc4b3e1edb8\") " pod="openshift-ingress/payload-processing-e2e-ait-10e9e4e8-7494999d6f-8xkpn" Sep 10 13:47:12.633683 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:47:12.633657 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9t9zs\" (UniqueName: \"kubernetes.io/projected/7a21fdae-ffad-4907-9865-1cc4b3e1edb8-kube-api-access-9t9zs\") pod \"payload-processing-e2e-ait-10e9e4e8-7494999d6f-8xkpn\" (UID: \"7a21fdae-ffad-4907-9865-1cc4b3e1edb8\") " pod="openshift-ingress/payload-processing-e2e-ait-10e9e4e8-7494999d6f-8xkpn" Sep 10 13:47:12.712827 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:47:12.712738 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-10e9e4e8-7494999d6f-8xkpn" Sep 10 13:47:12.996398 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:47:12.996368 2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-10e9e4e8-7494999d6f-8xkpn"] Sep 10 13:47:12.997565 ip-10-0-140-8 kubenswrapper[2585]: W0910 13:47:12.997539 2585 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7a21fdae_ffad_4907_9865_1cc4b3e1edb8.slice/crio-9cceff5ff1b28e11c7d6042532a1efac76a2ae2ff9e76c0ced2bca50ee98a5bb WatchSource:0}: Error finding container 9cceff5ff1b28e11c7d6042532a1efac76a2ae2ff9e76c0ced2bca50ee98a5bb: Status 404 returned error can't find the container with id 9cceff5ff1b28e11c7d6042532a1efac76a2ae2ff9e76c0ced2bca50ee98a5bb Sep 10 13:47:13.638911 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:47:13.638871 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-10e9e4e8-7494999d6f-8xkpn" event={"ID":"7a21fdae-ffad-4907-9865-1cc4b3e1edb8","Type":"ContainerStarted","Data":"fe1c492305d54da2af10e1bd547b50f1aee68da61a46e86c3040ecb58b3b3c7c"} Sep 10 13:47:13.639319 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:47:13.638976 2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ait-10e9e4e8-7494999d6f-8xkpn" Sep 10 13:47:13.639319 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:47:13.639007 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-10e9e4e8-7494999d6f-8xkpn" event={"ID":"7a21fdae-ffad-4907-9865-1cc4b3e1edb8","Type":"ContainerStarted","Data":"9cceff5ff1b28e11c7d6042532a1efac76a2ae2ff9e76c0ced2bca50ee98a5bb"} Sep 10 13:47:13.696293 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:47:13.696212 2585 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ait-10e9e4e8-7494999d6f-8xkpn" podStartSLOduration=1.6961963519999999 podStartE2EDuration="1.696196352s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 13:47:12 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 13:47:13.692483381 +0000 UTC m=+1118.565187654" watchObservedRunningTime="2026-09-10 13:47:13.696196352 +0000 UTC m=+1118.568900552" Sep 10 13:47:18.180841 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:47:18.180800 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fxgmp_fa53c783-d40a-4024-9f7b-1d0d2b57f6dc/konnectivity-agent/0.log" limit=1024 Sep 10 13:47:24.646220 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:47:24.646188 2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ait-10e9e4e8-7494999d6f-8xkpn" Sep 10 13:47:28.592656 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:47:28.592620 2585 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-b1f490-fzb9w"] Sep 10 13:47:28.596478 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:47:28.596461 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-b1f490-fzb9w" Sep 10 13:47:28.607470 ip-10-0-140-8 kubenswrapper[2585]: E0910 13:47:28.607443 2585 status_manager.go:1018] "Failed to get status for pod" err="pods \"maas-api-revoke-keys-e2e-autores-b1f490-fzb9w\" is forbidden: User \"system:node:ip-10-0-140-8.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"odh-ai-gateway-infra\": no relationship found between node 'ip-10-0-140-8.ec2.internal' and this object" podUID="278aa0ba-7bf8-4c2c-bebb-2ae9ee2946b6" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-b1f490-fzb9w" Sep 10 13:47:28.607701 ip-10-0-140-8 kubenswrapper[2585]: E0910 13:47:28.607682 2585 reflector.go:205] "Failed to watch" err="failed to list *v1.ConfigMap: configmaps \"openshift-service-ca.crt\" is forbidden: User \"system:node:ip-10-0-140-8.ec2.internal\" cannot list resource \"configmaps\" in API group \"\" in the namespace \"odh-ai-gateway-infra\": no relationship found between node 'ip-10-0-140-8.ec2.internal' and this object" logger="UnhandledError" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" type="*v1.ConfigMap" Sep 10 13:47:28.607920 ip-10-0-140-8 kubenswrapper[2585]: E0910 13:47:28.607903 2585 reflector.go:205] "Failed to watch" err="failed to list *v1.Secret: secrets \"maas-api-cleanup-dockercfg-7gcbr\" is forbidden: User \"system:node:ip-10-0-140-8.ec2.internal\" cannot list resource \"secrets\" in API group \"\" in the namespace \"odh-ai-gateway-infra\": no relationship found between node 'ip-10-0-140-8.ec2.internal' and this object" logger="UnhandledError" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-7gcbr\"" type="*v1.Secret" Sep 10 13:47:28.621751 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:47:28.621719 2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-b1f490-fzb9w"] Sep 10 13:47:28.668627 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:47:28.668596 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/278aa0ba-7bf8-4c2c-bebb-2ae9ee2946b6-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-autores-b1f490-fzb9w\" (UID: \"278aa0ba-7bf8-4c2c-bebb-2ae9ee2946b6\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-b1f490-fzb9w" Sep 10 13:47:28.769490 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:47:28.769454 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/278aa0ba-7bf8-4c2c-bebb-2ae9ee2946b6-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-autores-b1f490-fzb9w\" (UID: \"278aa0ba-7bf8-4c2c-bebb-2ae9ee2946b6\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-b1f490-fzb9w" Sep 10 13:47:28.825674 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:47:28.825601 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fxgmp_fa53c783-d40a-4024-9f7b-1d0d2b57f6dc/konnectivity-agent/0.log" limit=1024 Sep 10 13:47:29.515979 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:47:29.515946 2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Sep 10 13:47:29.520604 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:47:29.520581 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/278aa0ba-7bf8-4c2c-bebb-2ae9ee2946b6-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-autores-b1f490-fzb9w\" (UID: \"278aa0ba-7bf8-4c2c-bebb-2ae9ee2946b6\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-b1f490-fzb9w" Sep 10 13:47:30.011666 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:47:30.011631 2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-7gcbr\"" Sep 10 13:47:30.016660 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:47:30.016639 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-b1f490-fzb9w" Sep 10 13:47:30.299074 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:47:30.299045 2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-b1f490-fzb9w"] Sep 10 13:47:30.300698 ip-10-0-140-8 kubenswrapper[2585]: W0910 13:47:30.300671 2585 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod278aa0ba_7bf8_4c2c_bebb_2ae9ee2946b6.slice/crio-1798aa57154910407b35cdc54fc45b67fb44e9036e0968a0c7189566b82e931d WatchSource:0}: Error finding container 1798aa57154910407b35cdc54fc45b67fb44e9036e0968a0c7189566b82e931d: Status 404 returned error can't find the container with id 1798aa57154910407b35cdc54fc45b67fb44e9036e0968a0c7189566b82e931d Sep 10 13:47:30.742918 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:47:30.742880 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-b1f490-fzb9w" event={"ID":"278aa0ba-7bf8-4c2c-bebb-2ae9ee2946b6","Type":"ContainerStarted","Data":"1798aa57154910407b35cdc54fc45b67fb44e9036e0968a0c7189566b82e931d"} Sep 10 13:47:32.756587 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:47:32.756501 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-b1f490-fzb9w" event={"ID":"278aa0ba-7bf8-4c2c-bebb-2ae9ee2946b6","Type":"ContainerStarted","Data":"9bbe617cc779e7c87eeea60f42c051c83f52ac5cf21451bc0c8efb5a1cddf646"} Sep 10 13:47:32.786564 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:47:32.786515 2585 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-b1f490-fzb9w" podStartSLOduration=2.602072903 podStartE2EDuration="4.786490838s" totalImagesPullingTime="2.184417935s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 13:47:28 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 13:47:32.785111623 +0000 UTC m=+1137.657815823" watchObservedRunningTime="2026-09-10 13:47:32.786490838 +0000 UTC m=+1137.659195078" Sep 10 13:47:32.883625 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:47:32.883590 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fxgmp_fa53c783-d40a-4024-9f7b-1d0d2b57f6dc/konnectivity-agent/0.log" limit=1024 Sep 10 13:47:33.763952 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:47:33.763917 2585 generic.go:356] "Generic (PLEG): container finished" podID="278aa0ba-7bf8-4c2c-bebb-2ae9ee2946b6" containerID="9bbe617cc779e7c87eeea60f42c051c83f52ac5cf21451bc0c8efb5a1cddf646" exitCode=0 Sep 10 13:47:33.764420 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:47:33.764002 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-b1f490-fzb9w" event={"ID":"278aa0ba-7bf8-4c2c-bebb-2ae9ee2946b6","Type":"ContainerDied","Data":"9bbe617cc779e7c87eeea60f42c051c83f52ac5cf21451bc0c8efb5a1cddf646"} Sep 10 13:47:34.904725 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:47:34.904702 2585 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-b1f490-fzb9w" Sep 10 13:47:35.029407 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:47:35.029311 2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/278aa0ba-7bf8-4c2c-bebb-2ae9ee2946b6-maas-api-service-ca\") pod \"278aa0ba-7bf8-4c2c-bebb-2ae9ee2946b6\" (UID: \"278aa0ba-7bf8-4c2c-bebb-2ae9ee2946b6\") " Sep 10 13:47:35.029733 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:47:35.029704 2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/278aa0ba-7bf8-4c2c-bebb-2ae9ee2946b6-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "278aa0ba-7bf8-4c2c-bebb-2ae9ee2946b6" (UID: "278aa0ba-7bf8-4c2c-bebb-2ae9ee2946b6"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 10 13:47:35.130508 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:47:35.130476 2585 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/278aa0ba-7bf8-4c2c-bebb-2ae9ee2946b6-maas-api-service-ca\") on node \"ip-10-0-140-8.ec2.internal\" DevicePath \"\"" Sep 10 13:47:35.777098 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:47:35.777058 2585 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-b1f490-fzb9w" Sep 10 13:47:35.777296 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:47:35.777104 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-b1f490-fzb9w" event={"ID":"278aa0ba-7bf8-4c2c-bebb-2ae9ee2946b6","Type":"ContainerDied","Data":"1798aa57154910407b35cdc54fc45b67fb44e9036e0968a0c7189566b82e931d"} Sep 10 13:47:35.777296 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:47:35.777157 2585 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="1798aa57154910407b35cdc54fc45b67fb44e9036e0968a0c7189566b82e931d" Sep 10 13:47:35.779163 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:47:35.779139 2585 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-10e9e4e8-gj4q9"] Sep 10 13:47:35.779677 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:47:35.779657 2585 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="278aa0ba-7bf8-4c2c-bebb-2ae9ee2946b6" containerName="revoke-keys" Sep 10 13:47:35.779786 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:47:35.779679 2585 state_mem.go:107] "Deleted CPUSet assignment" podUID="278aa0ba-7bf8-4c2c-bebb-2ae9ee2946b6" containerName="revoke-keys" Sep 10 13:47:35.779841 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:47:35.779815 2585 memory_manager.go:368] "RemoveStaleState removing state" podUID="278aa0ba-7bf8-4c2c-bebb-2ae9ee2946b6" containerName="revoke-keys" Sep 10 13:47:35.783810 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:47:35.783787 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-10e9e4e8-gj4q9" Sep 10 13:47:35.789550 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:47:35.789530 2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Sep 10 13:47:35.789794 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:47:35.789700 2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-7gcbr\"" Sep 10 13:47:35.801144 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:47:35.801119 2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-10e9e4e8-gj4q9"] Sep 10 13:47:35.937012 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:47:35.936939 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a0094743-a9a8-4636-afc1-f27aa01b4717-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-10e9e4e8-gj4q9\" (UID: \"a0094743-a9a8-4636-afc1-f27aa01b4717\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-10e9e4e8-gj4q9" Sep 10 13:47:36.037566 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:47:36.037528 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a0094743-a9a8-4636-afc1-f27aa01b4717-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-10e9e4e8-gj4q9\" (UID: \"a0094743-a9a8-4636-afc1-f27aa01b4717\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-10e9e4e8-gj4q9" Sep 10 13:47:36.038159 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:47:36.038137 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a0094743-a9a8-4636-afc1-f27aa01b4717-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-10e9e4e8-gj4q9\" (UID: \"a0094743-a9a8-4636-afc1-f27aa01b4717\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-10e9e4e8-gj4q9" Sep 10 13:47:36.094569 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:47:36.094535 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-10e9e4e8-gj4q9" Sep 10 13:47:36.374617 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:47:36.374591 2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-10e9e4e8-gj4q9"] Sep 10 13:47:36.376876 ip-10-0-140-8 kubenswrapper[2585]: W0910 13:47:36.376845 2585 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda0094743_a9a8_4636_afc1_f27aa01b4717.slice/crio-58605c0573529e0d65b886393aeb8fff0fb72f0696aaf408a6c965b8f7f8df5f WatchSource:0}: Error finding container 58605c0573529e0d65b886393aeb8fff0fb72f0696aaf408a6c965b8f7f8df5f: Status 404 returned error can't find the container with id 58605c0573529e0d65b886393aeb8fff0fb72f0696aaf408a6c965b8f7f8df5f Sep 10 13:47:36.785476 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:47:36.785445 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-10e9e4e8-gj4q9" event={"ID":"a0094743-a9a8-4636-afc1-f27aa01b4717","Type":"ContainerStarted","Data":"8ca39d8469b882fc7f9bd894e69fa7e0631ec669481ef591d03f03ecab54051e"} Sep 10 13:47:36.785616 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:47:36.785497 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-10e9e4e8-gj4q9" event={"ID":"a0094743-a9a8-4636-afc1-f27aa01b4717","Type":"ContainerStarted","Data":"58605c0573529e0d65b886393aeb8fff0fb72f0696aaf408a6c965b8f7f8df5f"} Sep 10 13:47:36.815867 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:47:36.815817 2585 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-10e9e4e8-gj4q9" podStartSLOduration=1.8158003950000001 podStartE2EDuration="1.815800395s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 13:47:35 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 13:47:36.813087304 +0000 UTC m=+1141.685791506" watchObservedRunningTime="2026-09-10 13:47:36.815800395 +0000 UTC m=+1141.688504597" Sep 10 13:47:37.792742 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:47:37.792710 2585 generic.go:356] "Generic (PLEG): container finished" podID="a0094743-a9a8-4636-afc1-f27aa01b4717" containerID="8ca39d8469b882fc7f9bd894e69fa7e0631ec669481ef591d03f03ecab54051e" exitCode=0 Sep 10 13:47:37.793121 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:47:37.792806 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-10e9e4e8-gj4q9" event={"ID":"a0094743-a9a8-4636-afc1-f27aa01b4717","Type":"ContainerDied","Data":"8ca39d8469b882fc7f9bd894e69fa7e0631ec669481ef591d03f03ecab54051e"} Sep 10 13:47:38.726270 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:47:38.726204 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fxgmp_fa53c783-d40a-4024-9f7b-1d0d2b57f6dc/konnectivity-agent/0.log" limit=1024 Sep 10 13:47:38.931723 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:47:38.931701 2585 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-10e9e4e8-gj4q9" Sep 10 13:47:39.065811 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:47:39.065779 2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a0094743-a9a8-4636-afc1-f27aa01b4717-maas-api-service-ca\") pod \"a0094743-a9a8-4636-afc1-f27aa01b4717\" (UID: \"a0094743-a9a8-4636-afc1-f27aa01b4717\") " Sep 10 13:47:39.066172 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:47:39.066141 2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a0094743-a9a8-4636-afc1-f27aa01b4717-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "a0094743-a9a8-4636-afc1-f27aa01b4717" (UID: "a0094743-a9a8-4636-afc1-f27aa01b4717"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 10 13:47:39.166674 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:47:39.166645 2585 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a0094743-a9a8-4636-afc1-f27aa01b4717-maas-api-service-ca\") on node \"ip-10-0-140-8.ec2.internal\" DevicePath \"\"" Sep 10 13:47:39.610172 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:47:39.610089 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fxgmp_fa53c783-d40a-4024-9f7b-1d0d2b57f6dc/konnectivity-agent/0.log" limit=1024 Sep 10 13:47:39.649408 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:47:39.649379 2585 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-autores-b1f490-5f54877b85-4d97t"] Sep 10 13:47:39.649551 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:47:39.649443 2585 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-autores-b1f490-77f869c5bb-458gp"] Sep 10 13:47:39.649675 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:47:39.649643 2585 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-autores-b1f490-77f869c5bb-458gp" podUID="ae3341b0-646b-4447-aeac-4a49bcdac1ce" containerName="payload-pre-processing" containerID="cri-o://f7f612e5f72d41b2661cc57a9b027b1d95e17f1cdd387bce36c4253d4103c945" gracePeriod=30 Sep 10 13:47:39.649895 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:47:39.649848 2585 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-autores-b1f490-5f54877b85-4d97t" podUID="c7db93d0-3329-45c0-99c8-18690e72647c" containerName="payload-processing" containerID="cri-o://5d9e69ea4932fa6cc6e62697791471d3a2f2994f8f281248907ce62ed925f305" gracePeriod=30 Sep 10 13:47:39.741665 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:47:39.741629 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fxgmp_fa53c783-d40a-4024-9f7b-1d0d2b57f6dc/konnectivity-agent/0.log" limit=1024 Sep 10 13:47:39.807891 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:47:39.807790 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-10e9e4e8-gj4q9" event={"ID":"a0094743-a9a8-4636-afc1-f27aa01b4717","Type":"ContainerDied","Data":"58605c0573529e0d65b886393aeb8fff0fb72f0696aaf408a6c965b8f7f8df5f"} Sep 10 13:47:39.807891 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:47:39.807873 2585 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="58605c0573529e0d65b886393aeb8fff0fb72f0696aaf408a6c965b8f7f8df5f" Sep 10 13:47:39.808131 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:47:39.808008 2585 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-10e9e4e8-gj4q9" Sep 10 13:47:39.809979 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:47:39.809951 2585 generic.go:356] "Generic (PLEG): container finished" podID="ae3341b0-646b-4447-aeac-4a49bcdac1ce" containerID="f7f612e5f72d41b2661cc57a9b027b1d95e17f1cdd387bce36c4253d4103c945" exitCode=0 Sep 10 13:47:39.810104 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:47:39.810049 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-autores-b1f490-77f869c5bb-458gp" event={"ID":"ae3341b0-646b-4447-aeac-4a49bcdac1ce","Type":"ContainerDied","Data":"f7f612e5f72d41b2661cc57a9b027b1d95e17f1cdd387bce36c4253d4103c945"} Sep 10 13:47:39.812376 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:47:39.812356 2585 generic.go:356] "Generic (PLEG): container finished" podID="c7db93d0-3329-45c0-99c8-18690e72647c" containerID="5d9e69ea4932fa6cc6e62697791471d3a2f2994f8f281248907ce62ed925f305" exitCode=0 Sep 10 13:47:39.812492 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:47:39.812421 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-autores-b1f490-5f54877b85-4d97t" event={"ID":"c7db93d0-3329-45c0-99c8-18690e72647c","Type":"ContainerDied","Data":"5d9e69ea4932fa6cc6e62697791471d3a2f2994f8f281248907ce62ed925f305"} Sep 10 13:47:39.837836 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:47:39.837799 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fxgmp_fa53c783-d40a-4024-9f7b-1d0d2b57f6dc/konnectivity-agent/0.log" limit=1024 Sep 10 13:47:39.889873 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:47:39.889827 2585 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-b1f490-fzb9w"] Sep 10 13:47:39.896420 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:47:39.896385 2585 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-b1f490-fzb9w"] Sep 10 13:47:39.946474 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:47:39.946441 2585 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-autores-b1f490-77f869c5bb-458gp" Sep 10 13:47:39.949372 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:47:39.949351 2585 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-autores-b1f490-5f54877b85-4d97t" Sep 10 13:47:39.997681 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:47:39.997650 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fxgmp_fa53c783-d40a-4024-9f7b-1d0d2b57f6dc/konnectivity-agent/0.log" limit=1024 Sep 10 13:47:40.074876 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:47:40.074836 2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c7db93d0-3329-45c0-99c8-18690e72647c-plugins-config-volume\") pod \"c7db93d0-3329-45c0-99c8-18690e72647c\" (UID: \"c7db93d0-3329-45c0-99c8-18690e72647c\") " Sep 10 13:47:40.075045 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:47:40.074905 2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-gw2tk\" (UniqueName: \"kubernetes.io/projected/c7db93d0-3329-45c0-99c8-18690e72647c-kube-api-access-gw2tk\") pod \"c7db93d0-3329-45c0-99c8-18690e72647c\" (UID: \"c7db93d0-3329-45c0-99c8-18690e72647c\") " Sep 10 13:47:40.075045 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:47:40.075004 2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-72vx9\" (UniqueName: \"kubernetes.io/projected/ae3341b0-646b-4447-aeac-4a49bcdac1ce-kube-api-access-72vx9\") pod \"ae3341b0-646b-4447-aeac-4a49bcdac1ce\" (UID: \"ae3341b0-646b-4447-aeac-4a49bcdac1ce\") " Sep 10 13:47:40.075191 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:47:40.075062 2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ae3341b0-646b-4447-aeac-4a49bcdac1ce-plugins-config-volume\") pod \"ae3341b0-646b-4447-aeac-4a49bcdac1ce\" (UID: \"ae3341b0-646b-4447-aeac-4a49bcdac1ce\") " Sep 10 13:47:40.075309 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:47:40.075279 2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c7db93d0-3329-45c0-99c8-18690e72647c-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "c7db93d0-3329-45c0-99c8-18690e72647c" (UID: "c7db93d0-3329-45c0-99c8-18690e72647c"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 10 13:47:40.075398 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:47:40.075357 2585 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c7db93d0-3329-45c0-99c8-18690e72647c-plugins-config-volume\") on node \"ip-10-0-140-8.ec2.internal\" DevicePath \"\"" Sep 10 13:47:40.075565 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:47:40.075535 2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ae3341b0-646b-4447-aeac-4a49bcdac1ce-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "ae3341b0-646b-4447-aeac-4a49bcdac1ce" (UID: "ae3341b0-646b-4447-aeac-4a49bcdac1ce"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 10 13:47:40.077107 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:47:40.077085 2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c7db93d0-3329-45c0-99c8-18690e72647c-kube-api-access-gw2tk" (OuterVolumeSpecName: "kube-api-access-gw2tk") pod "c7db93d0-3329-45c0-99c8-18690e72647c" (UID: "c7db93d0-3329-45c0-99c8-18690e72647c"). InnerVolumeSpecName "kube-api-access-gw2tk". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 10 13:47:40.077511 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:47:40.077495 2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ae3341b0-646b-4447-aeac-4a49bcdac1ce-kube-api-access-72vx9" (OuterVolumeSpecName: "kube-api-access-72vx9") pod "ae3341b0-646b-4447-aeac-4a49bcdac1ce" (UID: "ae3341b0-646b-4447-aeac-4a49bcdac1ce"). InnerVolumeSpecName "kube-api-access-72vx9". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 10 13:47:40.176871 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:47:40.176776 2585 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gw2tk\" (UniqueName: \"kubernetes.io/projected/c7db93d0-3329-45c0-99c8-18690e72647c-kube-api-access-gw2tk\") on node \"ip-10-0-140-8.ec2.internal\" DevicePath \"\"" Sep 10 13:47:40.176871 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:47:40.176812 2585 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-72vx9\" (UniqueName: \"kubernetes.io/projected/ae3341b0-646b-4447-aeac-4a49bcdac1ce-kube-api-access-72vx9\") on node \"ip-10-0-140-8.ec2.internal\" DevicePath \"\"" Sep 10 13:47:40.176871 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:47:40.176825 2585 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ae3341b0-646b-4447-aeac-4a49bcdac1ce-plugins-config-volume\") on node \"ip-10-0-140-8.ec2.internal\" DevicePath \"\"" Sep 10 13:47:40.278683 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:47:40.278649 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fxgmp_fa53c783-d40a-4024-9f7b-1d0d2b57f6dc/konnectivity-agent/0.log" limit=1024 Sep 10 13:47:40.819696 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:47:40.819664 2585 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-autores-b1f490-5f54877b85-4d97t" Sep 10 13:47:40.820003 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:47:40.819707 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-autores-b1f490-5f54877b85-4d97t" event={"ID":"c7db93d0-3329-45c0-99c8-18690e72647c","Type":"ContainerDied","Data":"aaaad59c880fcb9daed251da7871ff6c768ab25c182fce04feefaebf336b54ec"} Sep 10 13:47:40.820003 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:47:40.819763 2585 scope.go:117] "RemoveContainer" containerID="5d9e69ea4932fa6cc6e62697791471d3a2f2994f8f281248907ce62ed925f305" Sep 10 13:47:40.821223 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:47:40.821195 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-autores-b1f490-77f869c5bb-458gp" event={"ID":"ae3341b0-646b-4447-aeac-4a49bcdac1ce","Type":"ContainerDied","Data":"15054aa446a63305eee7def37b85aff081f3757381f0baf9e9f4f8e8a1efe4f0"} Sep 10 13:47:40.821330 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:47:40.821235 2585 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-autores-b1f490-77f869c5bb-458gp" Sep 10 13:47:40.831909 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:47:40.831890 2585 scope.go:117] "RemoveContainer" containerID="f7f612e5f72d41b2661cc57a9b027b1d95e17f1cdd387bce36c4253d4103c945" Sep 10 13:47:40.846255 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:47:40.846220 2585 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-autores-b1f490-5f54877b85-4d97t"] Sep 10 13:47:40.851377 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:47:40.851356 2585 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-autores-b1f490-5f54877b85-4d97t"] Sep 10 13:47:40.865105 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:47:40.865082 2585 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-autores-b1f490-77f869c5bb-458gp"] Sep 10 13:47:40.872114 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:47:40.872093 2585 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-autores-b1f490-77f869c5bb-458gp"] Sep 10 13:47:41.648453 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:47:41.648423 2585 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="278aa0ba-7bf8-4c2c-bebb-2ae9ee2946b6" path="/var/lib/kubelet/pods/278aa0ba-7bf8-4c2c-bebb-2ae9ee2946b6/volumes" Sep 10 13:47:41.648851 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:47:41.648587 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fxgmp_fa53c783-d40a-4024-9f7b-1d0d2b57f6dc/konnectivity-agent/0.log" limit=1024 Sep 10 13:47:41.648851 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:47:41.648799 2585 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="ae3341b0-646b-4447-aeac-4a49bcdac1ce" path="/var/lib/kubelet/pods/ae3341b0-646b-4447-aeac-4a49bcdac1ce/volumes" Sep 10 13:47:41.649117 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:47:41.649104 2585 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="c7db93d0-3329-45c0-99c8-18690e72647c" path="/var/lib/kubelet/pods/c7db93d0-3329-45c0-99c8-18690e72647c/volumes" Sep 10 13:47:41.759102 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:47:41.759066 2585 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/e2e-conflict-b-92ef2340-kserve-76ddfcb8c8-rlgxc"] Sep 10 13:47:41.759518 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:47:41.759503 2585 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c7db93d0-3329-45c0-99c8-18690e72647c" containerName="payload-processing" Sep 10 13:47:41.759577 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:47:41.759520 2585 state_mem.go:107] "Deleted CPUSet assignment" podUID="c7db93d0-3329-45c0-99c8-18690e72647c" containerName="payload-processing" Sep 10 13:47:41.759577 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:47:41.759540 2585 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ae3341b0-646b-4447-aeac-4a49bcdac1ce" containerName="payload-pre-processing" Sep 10 13:47:41.759577 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:47:41.759546 2585 state_mem.go:107] "Deleted CPUSet assignment" podUID="ae3341b0-646b-4447-aeac-4a49bcdac1ce" containerName="payload-pre-processing" Sep 10 13:47:41.759577 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:47:41.759560 2585 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a0094743-a9a8-4636-afc1-f27aa01b4717" containerName="revoke-keys" Sep 10 13:47:41.759577 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:47:41.759567 2585 state_mem.go:107] "Deleted CPUSet assignment" podUID="a0094743-a9a8-4636-afc1-f27aa01b4717" containerName="revoke-keys" Sep 10 13:47:41.759750 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:47:41.759621 2585 memory_manager.go:368] "RemoveStaleState removing state" podUID="a0094743-a9a8-4636-afc1-f27aa01b4717" containerName="revoke-keys" Sep 10 13:47:41.759750 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:47:41.759629 2585 memory_manager.go:368] "RemoveStaleState removing state" podUID="ae3341b0-646b-4447-aeac-4a49bcdac1ce" containerName="payload-pre-processing" Sep 10 13:47:41.759750 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:47:41.759639 2585 memory_manager.go:368] "RemoveStaleState removing state" podUID="c7db93d0-3329-45c0-99c8-18690e72647c" containerName="payload-processing" Sep 10 13:47:41.764148 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:47:41.764129 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-conflict-b-92ef2340-kserve-76ddfcb8c8-rlgxc" Sep 10 13:47:41.767212 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:47:41.767191 2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-conflict-b-92ef2340-kserve-self-signed-certs\"" Sep 10 13:47:41.779074 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:47:41.779045 2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-conflict-b-92ef2340-kserve-76ddfcb8c8-rlgxc"] Sep 10 13:47:41.890950 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:47:41.890920 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5c431918-5de0-4f81-b96a-5653bd8c67ec-dshm\") pod \"e2e-conflict-b-92ef2340-kserve-76ddfcb8c8-rlgxc\" (UID: \"5c431918-5de0-4f81-b96a-5653bd8c67ec\") " pod="llm/e2e-conflict-b-92ef2340-kserve-76ddfcb8c8-rlgxc" Sep 10 13:47:41.891106 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:47:41.890970 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5c431918-5de0-4f81-b96a-5653bd8c67ec-model-cache\") pod \"e2e-conflict-b-92ef2340-kserve-76ddfcb8c8-rlgxc\" (UID: \"5c431918-5de0-4f81-b96a-5653bd8c67ec\") " pod="llm/e2e-conflict-b-92ef2340-kserve-76ddfcb8c8-rlgxc" Sep 10 13:47:41.891106 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:47:41.890999 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8hpgc\" (UniqueName: \"kubernetes.io/projected/5c431918-5de0-4f81-b96a-5653bd8c67ec-kube-api-access-8hpgc\") pod \"e2e-conflict-b-92ef2340-kserve-76ddfcb8c8-rlgxc\" (UID: \"5c431918-5de0-4f81-b96a-5653bd8c67ec\") " pod="llm/e2e-conflict-b-92ef2340-kserve-76ddfcb8c8-rlgxc" Sep 10 13:47:41.891106 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:47:41.891031 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5c431918-5de0-4f81-b96a-5653bd8c67ec-home\") pod \"e2e-conflict-b-92ef2340-kserve-76ddfcb8c8-rlgxc\" (UID: \"5c431918-5de0-4f81-b96a-5653bd8c67ec\") " pod="llm/e2e-conflict-b-92ef2340-kserve-76ddfcb8c8-rlgxc" Sep 10 13:47:41.891106 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:47:41.891050 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5c431918-5de0-4f81-b96a-5653bd8c67ec-tls-certs\") pod \"e2e-conflict-b-92ef2340-kserve-76ddfcb8c8-rlgxc\" (UID: \"5c431918-5de0-4f81-b96a-5653bd8c67ec\") " pod="llm/e2e-conflict-b-92ef2340-kserve-76ddfcb8c8-rlgxc" Sep 10 13:47:41.891106 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:47:41.891098 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5c431918-5de0-4f81-b96a-5653bd8c67ec-tmp-dir\") pod \"e2e-conflict-b-92ef2340-kserve-76ddfcb8c8-rlgxc\" (UID: \"5c431918-5de0-4f81-b96a-5653bd8c67ec\") " pod="llm/e2e-conflict-b-92ef2340-kserve-76ddfcb8c8-rlgxc" Sep 10 13:47:41.992631 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:47:41.992524 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5c431918-5de0-4f81-b96a-5653bd8c67ec-dshm\") pod \"e2e-conflict-b-92ef2340-kserve-76ddfcb8c8-rlgxc\" (UID: \"5c431918-5de0-4f81-b96a-5653bd8c67ec\") " pod="llm/e2e-conflict-b-92ef2340-kserve-76ddfcb8c8-rlgxc" Sep 10 13:47:41.992631 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:47:41.992595 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5c431918-5de0-4f81-b96a-5653bd8c67ec-model-cache\") pod \"e2e-conflict-b-92ef2340-kserve-76ddfcb8c8-rlgxc\" (UID: \"5c431918-5de0-4f81-b96a-5653bd8c67ec\") " pod="llm/e2e-conflict-b-92ef2340-kserve-76ddfcb8c8-rlgxc" Sep 10 13:47:41.992631 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:47:41.992623 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8hpgc\" (UniqueName: \"kubernetes.io/projected/5c431918-5de0-4f81-b96a-5653bd8c67ec-kube-api-access-8hpgc\") pod \"e2e-conflict-b-92ef2340-kserve-76ddfcb8c8-rlgxc\" (UID: \"5c431918-5de0-4f81-b96a-5653bd8c67ec\") " pod="llm/e2e-conflict-b-92ef2340-kserve-76ddfcb8c8-rlgxc" Sep 10 13:47:41.992925 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:47:41.992660 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5c431918-5de0-4f81-b96a-5653bd8c67ec-home\") pod \"e2e-conflict-b-92ef2340-kserve-76ddfcb8c8-rlgxc\" (UID: \"5c431918-5de0-4f81-b96a-5653bd8c67ec\") " pod="llm/e2e-conflict-b-92ef2340-kserve-76ddfcb8c8-rlgxc" Sep 10 13:47:41.992925 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:47:41.992696 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5c431918-5de0-4f81-b96a-5653bd8c67ec-tls-certs\") pod \"e2e-conflict-b-92ef2340-kserve-76ddfcb8c8-rlgxc\" (UID: \"5c431918-5de0-4f81-b96a-5653bd8c67ec\") " pod="llm/e2e-conflict-b-92ef2340-kserve-76ddfcb8c8-rlgxc" Sep 10 13:47:41.992925 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:47:41.992725 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5c431918-5de0-4f81-b96a-5653bd8c67ec-tmp-dir\") pod \"e2e-conflict-b-92ef2340-kserve-76ddfcb8c8-rlgxc\" (UID: \"5c431918-5de0-4f81-b96a-5653bd8c67ec\") " pod="llm/e2e-conflict-b-92ef2340-kserve-76ddfcb8c8-rlgxc" Sep 10 13:47:41.993074 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:47:41.993032 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5c431918-5de0-4f81-b96a-5653bd8c67ec-model-cache\") pod \"e2e-conflict-b-92ef2340-kserve-76ddfcb8c8-rlgxc\" (UID: \"5c431918-5de0-4f81-b96a-5653bd8c67ec\") " pod="llm/e2e-conflict-b-92ef2340-kserve-76ddfcb8c8-rlgxc" Sep 10 13:47:41.993135 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:47:41.993067 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5c431918-5de0-4f81-b96a-5653bd8c67ec-home\") pod \"e2e-conflict-b-92ef2340-kserve-76ddfcb8c8-rlgxc\" (UID: \"5c431918-5de0-4f81-b96a-5653bd8c67ec\") " pod="llm/e2e-conflict-b-92ef2340-kserve-76ddfcb8c8-rlgxc" Sep 10 13:47:41.993280 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:47:41.993225 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5c431918-5de0-4f81-b96a-5653bd8c67ec-tmp-dir\") pod \"e2e-conflict-b-92ef2340-kserve-76ddfcb8c8-rlgxc\" (UID: \"5c431918-5de0-4f81-b96a-5653bd8c67ec\") " pod="llm/e2e-conflict-b-92ef2340-kserve-76ddfcb8c8-rlgxc" Sep 10 13:47:41.994906 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:47:41.994878 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5c431918-5de0-4f81-b96a-5653bd8c67ec-dshm\") pod \"e2e-conflict-b-92ef2340-kserve-76ddfcb8c8-rlgxc\" (UID: \"5c431918-5de0-4f81-b96a-5653bd8c67ec\") " pod="llm/e2e-conflict-b-92ef2340-kserve-76ddfcb8c8-rlgxc" Sep 10 13:47:41.995267 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:47:41.995228 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5c431918-5de0-4f81-b96a-5653bd8c67ec-tls-certs\") pod \"e2e-conflict-b-92ef2340-kserve-76ddfcb8c8-rlgxc\" (UID: \"5c431918-5de0-4f81-b96a-5653bd8c67ec\") " pod="llm/e2e-conflict-b-92ef2340-kserve-76ddfcb8c8-rlgxc" Sep 10 13:47:42.016718 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:47:42.016691 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8hpgc\" (UniqueName: \"kubernetes.io/projected/5c431918-5de0-4f81-b96a-5653bd8c67ec-kube-api-access-8hpgc\") pod \"e2e-conflict-b-92ef2340-kserve-76ddfcb8c8-rlgxc\" (UID: \"5c431918-5de0-4f81-b96a-5653bd8c67ec\") " pod="llm/e2e-conflict-b-92ef2340-kserve-76ddfcb8c8-rlgxc" Sep 10 13:47:42.044228 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:47:42.044197 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fxgmp_fa53c783-d40a-4024-9f7b-1d0d2b57f6dc/konnectivity-agent/0.log" limit=1024 Sep 10 13:47:42.074723 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:47:42.074697 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-conflict-b-92ef2340-kserve-76ddfcb8c8-rlgxc" Sep 10 13:47:42.368747 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:47:42.368719 2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-conflict-b-92ef2340-kserve-76ddfcb8c8-rlgxc"] Sep 10 13:47:42.370118 ip-10-0-140-8 kubenswrapper[2585]: W0910 13:47:42.370093 2585 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod5c431918_5de0_4f81_b96a_5653bd8c67ec.slice/crio-73f54f90ad503b26f28e563972bf867f0178d1a70984a2e1c8d751fff9e1830c WatchSource:0}: Error finding container 73f54f90ad503b26f28e563972bf867f0178d1a70984a2e1c8d751fff9e1830c: Status 404 returned error can't find the container with id 73f54f90ad503b26f28e563972bf867f0178d1a70984a2e1c8d751fff9e1830c Sep 10 13:47:42.836319 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:47:42.836276 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-conflict-b-92ef2340-kserve-76ddfcb8c8-rlgxc" event={"ID":"5c431918-5de0-4f81-b96a-5653bd8c67ec","Type":"ContainerStarted","Data":"d99447558e64b6cf82147e7f4c75c0a962b08b4539c8163ff4639555ad7b2869"} Sep 10 13:47:42.836319 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:47:42.836320 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-conflict-b-92ef2340-kserve-76ddfcb8c8-rlgxc" event={"ID":"5c431918-5de0-4f81-b96a-5653bd8c67ec","Type":"ContainerStarted","Data":"73f54f90ad503b26f28e563972bf867f0178d1a70984a2e1c8d751fff9e1830c"} Sep 10 13:47:42.858069 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:47:42.858008 2585 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/e2e-conflict-b-92ef2340-kserve-76ddfcb8c8-rlgxc" podStartSLOduration=1.857984648 podStartE2EDuration="1.857984648s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 13:47:41 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 13:47:42.857497833 +0000 UTC m=+1147.730202035" watchObservedRunningTime="2026-09-10 13:47:42.857984648 +0000 UTC m=+1147.730688851" Sep 10 13:47:42.863133 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:47:42.863100 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fxgmp_fa53c783-d40a-4024-9f7b-1d0d2b57f6dc/konnectivity-agent/0.log" limit=1024 Sep 10 13:47:42.916914 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:47:42.916884 2585 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-10e9e4e8-7494999d6f-8xkpn"] Sep 10 13:47:42.917141 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:47:42.917113 2585 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ait-10e9e4e8-7494999d6f-8xkpn" podUID="7a21fdae-ffad-4907-9865-1cc4b3e1edb8" containerName="payload-processing" containerID="cri-o://fe1c492305d54da2af10e1bd547b50f1aee68da61a46e86c3040ecb58b3b3c7c" gracePeriod=30 Sep 10 13:47:42.952430 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:47:42.952403 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fxgmp_fa53c783-d40a-4024-9f7b-1d0d2b57f6dc/konnectivity-agent/0.log" limit=1024 Sep 10 13:47:43.024741 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:47:43.024714 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fxgmp_fa53c783-d40a-4024-9f7b-1d0d2b57f6dc/konnectivity-agent/0.log" limit=1024 Sep 10 13:47:43.118946 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:47:43.118916 2585 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-10e9e4e8-gj4q9"] Sep 10 13:47:43.122788 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:47:43.122763 2585 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-10e9e4e8-gj4q9"] Sep 10 13:47:43.170047 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:47:43.170021 2585 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-10e9e4e8-7494999d6f-8xkpn" Sep 10 13:47:43.227623 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:47:43.227590 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fxgmp_fa53c783-d40a-4024-9f7b-1d0d2b57f6dc/konnectivity-agent/0.log" limit=1024 Sep 10 13:47:43.304741 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:47:43.304702 2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7a21fdae-ffad-4907-9865-1cc4b3e1edb8-plugins-config-volume\") pod \"7a21fdae-ffad-4907-9865-1cc4b3e1edb8\" (UID: \"7a21fdae-ffad-4907-9865-1cc4b3e1edb8\") " Sep 10 13:47:43.304895 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:47:43.304755 2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-9t9zs\" (UniqueName: \"kubernetes.io/projected/7a21fdae-ffad-4907-9865-1cc4b3e1edb8-kube-api-access-9t9zs\") pod \"7a21fdae-ffad-4907-9865-1cc4b3e1edb8\" (UID: \"7a21fdae-ffad-4907-9865-1cc4b3e1edb8\") " Sep 10 13:47:43.305133 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:47:43.305107 2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7a21fdae-ffad-4907-9865-1cc4b3e1edb8-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "7a21fdae-ffad-4907-9865-1cc4b3e1edb8" (UID: "7a21fdae-ffad-4907-9865-1cc4b3e1edb8"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 10 13:47:43.306851 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:47:43.306826 2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7a21fdae-ffad-4907-9865-1cc4b3e1edb8-kube-api-access-9t9zs" (OuterVolumeSpecName: "kube-api-access-9t9zs") pod "7a21fdae-ffad-4907-9865-1cc4b3e1edb8" (UID: "7a21fdae-ffad-4907-9865-1cc4b3e1edb8"). InnerVolumeSpecName "kube-api-access-9t9zs". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 10 13:47:43.310885 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:47:43.310860 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fxgmp_fa53c783-d40a-4024-9f7b-1d0d2b57f6dc/konnectivity-agent/0.log" limit=1024 Sep 10 13:47:43.405843 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:47:43.405766 2585 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9t9zs\" (UniqueName: \"kubernetes.io/projected/7a21fdae-ffad-4907-9865-1cc4b3e1edb8-kube-api-access-9t9zs\") on node \"ip-10-0-140-8.ec2.internal\" DevicePath \"\"" Sep 10 13:47:43.405843 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:47:43.405791 2585 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7a21fdae-ffad-4907-9865-1cc4b3e1edb8-plugins-config-volume\") on node \"ip-10-0-140-8.ec2.internal\" DevicePath \"\"" Sep 10 13:47:43.647540 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:47:43.647506 2585 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="a0094743-a9a8-4636-afc1-f27aa01b4717" path="/var/lib/kubelet/pods/a0094743-a9a8-4636-afc1-f27aa01b4717/volumes" Sep 10 13:47:43.843363 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:47:43.843329 2585 generic.go:356] "Generic (PLEG): container finished" podID="7a21fdae-ffad-4907-9865-1cc4b3e1edb8" containerID="fe1c492305d54da2af10e1bd547b50f1aee68da61a46e86c3040ecb58b3b3c7c" exitCode=0 Sep 10 13:47:43.843832 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:47:43.843374 2585 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-10e9e4e8-7494999d6f-8xkpn" Sep 10 13:47:43.843832 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:47:43.843392 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-10e9e4e8-7494999d6f-8xkpn" event={"ID":"7a21fdae-ffad-4907-9865-1cc4b3e1edb8","Type":"ContainerDied","Data":"fe1c492305d54da2af10e1bd547b50f1aee68da61a46e86c3040ecb58b3b3c7c"} Sep 10 13:47:43.843832 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:47:43.843457 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-10e9e4e8-7494999d6f-8xkpn" event={"ID":"7a21fdae-ffad-4907-9865-1cc4b3e1edb8","Type":"ContainerDied","Data":"9cceff5ff1b28e11c7d6042532a1efac76a2ae2ff9e76c0ced2bca50ee98a5bb"} Sep 10 13:47:43.843832 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:47:43.843484 2585 scope.go:117] "RemoveContainer" containerID="fe1c492305d54da2af10e1bd547b50f1aee68da61a46e86c3040ecb58b3b3c7c" Sep 10 13:47:43.852525 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:47:43.852511 2585 scope.go:117] "RemoveContainer" containerID="fe1c492305d54da2af10e1bd547b50f1aee68da61a46e86c3040ecb58b3b3c7c" Sep 10 13:47:43.852772 ip-10-0-140-8 kubenswrapper[2585]: E0910 13:47:43.852752 2585 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fe1c492305d54da2af10e1bd547b50f1aee68da61a46e86c3040ecb58b3b3c7c\": container with ID starting with fe1c492305d54da2af10e1bd547b50f1aee68da61a46e86c3040ecb58b3b3c7c not found: ID does not exist" containerID="fe1c492305d54da2af10e1bd547b50f1aee68da61a46e86c3040ecb58b3b3c7c" Sep 10 13:47:43.852824 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:47:43.852781 2585 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fe1c492305d54da2af10e1bd547b50f1aee68da61a46e86c3040ecb58b3b3c7c"} err="failed to get container status \"fe1c492305d54da2af10e1bd547b50f1aee68da61a46e86c3040ecb58b3b3c7c\": rpc error: code = NotFound desc = could not find container \"fe1c492305d54da2af10e1bd547b50f1aee68da61a46e86c3040ecb58b3b3c7c\": container with ID starting with fe1c492305d54da2af10e1bd547b50f1aee68da61a46e86c3040ecb58b3b3c7c not found: ID does not exist" Sep 10 13:47:43.877117 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:47:43.877085 2585 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-10e9e4e8-7494999d6f-8xkpn"] Sep 10 13:47:43.883179 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:47:43.883150 2585 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-10e9e4e8-7494999d6f-8xkpn"] Sep 10 13:47:44.380851 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:47:44.380822 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fxgmp_fa53c783-d40a-4024-9f7b-1d0d2b57f6dc/konnectivity-agent/0.log" limit=1024 Sep 10 13:47:44.798566 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:47:44.798454 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fxgmp_fa53c783-d40a-4024-9f7b-1d0d2b57f6dc/konnectivity-agent/0.log" limit=1024 Sep 10 13:47:45.120238 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:47:45.120206 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fxgmp_fa53c783-d40a-4024-9f7b-1d0d2b57f6dc/konnectivity-agent/0.log" limit=1024 Sep 10 13:47:45.316642 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:47:45.316608 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fxgmp_fa53c783-d40a-4024-9f7b-1d0d2b57f6dc/konnectivity-agent/0.log" limit=1024 Sep 10 13:47:45.593146 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:47:45.593115 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fxgmp_fa53c783-d40a-4024-9f7b-1d0d2b57f6dc/konnectivity-agent/0.log" limit=1024 Sep 10 13:47:45.649393 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:47:45.649354 2585 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="7a21fdae-ffad-4907-9865-1cc4b3e1edb8" path="/var/lib/kubelet/pods/7a21fdae-ffad-4907-9865-1cc4b3e1edb8/volumes" Sep 10 13:47:47.243997 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:47:47.243955 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fxgmp_fa53c783-d40a-4024-9f7b-1d0d2b57f6dc/konnectivity-agent/0.log" limit=1024 Sep 10 13:47:47.360720 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:47:47.360687 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fxgmp_fa53c783-d40a-4024-9f7b-1d0d2b57f6dc/konnectivity-agent/0.log" limit=1024 Sep 10 13:47:48.660273 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:47:48.660225 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fxgmp_fa53c783-d40a-4024-9f7b-1d0d2b57f6dc/konnectivity-agent/0.log" limit=1024 Sep 10 13:47:48.755006 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:47:48.754974 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fxgmp_fa53c783-d40a-4024-9f7b-1d0d2b57f6dc/konnectivity-agent/0.log" limit=1024 Sep 10 13:47:48.866268 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:47:48.866194 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fxgmp_fa53c783-d40a-4024-9f7b-1d0d2b57f6dc/konnectivity-agent/0.log" limit=1024 Sep 10 13:47:49.608057 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:47:49.608014 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fxgmp_fa53c783-d40a-4024-9f7b-1d0d2b57f6dc/konnectivity-agent/0.log" limit=1024 Sep 10 13:47:49.721677 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:47:49.717421 2585 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-derive-bc2301a5-85f4c659cd-86fts"] Sep 10 13:47:49.721677 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:47:49.718336 2585 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7a21fdae-ffad-4907-9865-1cc4b3e1edb8" containerName="payload-processing" Sep 10 13:47:49.721677 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:47:49.718358 2585 state_mem.go:107] "Deleted CPUSet assignment" podUID="7a21fdae-ffad-4907-9865-1cc4b3e1edb8" containerName="payload-processing" Sep 10 13:47:49.721677 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:47:49.718632 2585 memory_manager.go:368] "RemoveStaleState removing state" podUID="7a21fdae-ffad-4907-9865-1cc4b3e1edb8" containerName="payload-processing" Sep 10 13:47:49.722368 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:47:49.722141 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fxgmp_fa53c783-d40a-4024-9f7b-1d0d2b57f6dc/konnectivity-agent/0.log" limit=1024 Sep 10 13:47:49.724078 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:47:49.724021 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-derive-bc2301a5-85f4c659cd-86fts" Sep 10 13:47:49.727931 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:47:49.727904 2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-derive-bc2301a5-dockercfg-qrxwq\"" Sep 10 13:47:49.728065 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:47:49.727908 2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-derive-bc2301a5\"" Sep 10 13:47:49.728065 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:47:49.727988 2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-derive-bc2301a5-85f4c659cd-86fts"] Sep 10 13:47:49.766810 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:47:49.766773 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d7656741-4c49-4e97-b2cb-47e6b2a89a5f-plugins-config-volume\") pod \"payload-pre-processing-e2e-derive-bc2301a5-85f4c659cd-86fts\" (UID: \"d7656741-4c49-4e97-b2cb-47e6b2a89a5f\") " pod="openshift-ingress/payload-pre-processing-e2e-derive-bc2301a5-85f4c659cd-86fts" Sep 10 13:47:49.766971 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:47:49.766820 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5rxxd\" (UniqueName: \"kubernetes.io/projected/d7656741-4c49-4e97-b2cb-47e6b2a89a5f-kube-api-access-5rxxd\") pod \"payload-pre-processing-e2e-derive-bc2301a5-85f4c659cd-86fts\" (UID: \"d7656741-4c49-4e97-b2cb-47e6b2a89a5f\") " pod="openshift-ingress/payload-pre-processing-e2e-derive-bc2301a5-85f4c659cd-86fts" Sep 10 13:47:49.793162 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:47:49.793124 2585 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-derive-bc2301a5-c68784649-mkjkf"] Sep 10 13:47:49.796761 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:47:49.796743 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-derive-bc2301a5-c68784649-mkjkf" Sep 10 13:47:49.815986 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:47:49.815951 2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-derive-bc2301a5-c68784649-mkjkf"] Sep 10 13:47:49.867598 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:47:49.867517 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8zbs6\" (UniqueName: \"kubernetes.io/projected/857b920c-4d70-4dd5-acdc-e35cc79d59c9-kube-api-access-8zbs6\") pod \"payload-processing-e2e-derive-bc2301a5-c68784649-mkjkf\" (UID: \"857b920c-4d70-4dd5-acdc-e35cc79d59c9\") " pod="openshift-ingress/payload-processing-e2e-derive-bc2301a5-c68784649-mkjkf" Sep 10 13:47:49.867598 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:47:49.867548 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/857b920c-4d70-4dd5-acdc-e35cc79d59c9-plugins-config-volume\") pod \"payload-processing-e2e-derive-bc2301a5-c68784649-mkjkf\" (UID: \"857b920c-4d70-4dd5-acdc-e35cc79d59c9\") " pod="openshift-ingress/payload-processing-e2e-derive-bc2301a5-c68784649-mkjkf" Sep 10 13:47:49.867598 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:47:49.867577 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d7656741-4c49-4e97-b2cb-47e6b2a89a5f-plugins-config-volume\") pod \"payload-pre-processing-e2e-derive-bc2301a5-85f4c659cd-86fts\" (UID: \"d7656741-4c49-4e97-b2cb-47e6b2a89a5f\") " pod="openshift-ingress/payload-pre-processing-e2e-derive-bc2301a5-85f4c659cd-86fts" Sep 10 13:47:49.867812 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:47:49.867650 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5rxxd\" (UniqueName: \"kubernetes.io/projected/d7656741-4c49-4e97-b2cb-47e6b2a89a5f-kube-api-access-5rxxd\") pod \"payload-pre-processing-e2e-derive-bc2301a5-85f4c659cd-86fts\" (UID: \"d7656741-4c49-4e97-b2cb-47e6b2a89a5f\") " pod="openshift-ingress/payload-pre-processing-e2e-derive-bc2301a5-85f4c659cd-86fts" Sep 10 13:47:49.868304 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:47:49.868287 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d7656741-4c49-4e97-b2cb-47e6b2a89a5f-plugins-config-volume\") pod \"payload-pre-processing-e2e-derive-bc2301a5-85f4c659cd-86fts\" (UID: \"d7656741-4c49-4e97-b2cb-47e6b2a89a5f\") " pod="openshift-ingress/payload-pre-processing-e2e-derive-bc2301a5-85f4c659cd-86fts" Sep 10 13:47:49.880528 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:47:49.880497 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5rxxd\" (UniqueName: \"kubernetes.io/projected/d7656741-4c49-4e97-b2cb-47e6b2a89a5f-kube-api-access-5rxxd\") pod \"payload-pre-processing-e2e-derive-bc2301a5-85f4c659cd-86fts\" (UID: \"d7656741-4c49-4e97-b2cb-47e6b2a89a5f\") " pod="openshift-ingress/payload-pre-processing-e2e-derive-bc2301a5-85f4c659cd-86fts" Sep 10 13:47:49.968302 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:47:49.968231 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8zbs6\" (UniqueName: \"kubernetes.io/projected/857b920c-4d70-4dd5-acdc-e35cc79d59c9-kube-api-access-8zbs6\") pod \"payload-processing-e2e-derive-bc2301a5-c68784649-mkjkf\" (UID: \"857b920c-4d70-4dd5-acdc-e35cc79d59c9\") " pod="openshift-ingress/payload-processing-e2e-derive-bc2301a5-c68784649-mkjkf" Sep 10 13:47:49.968302 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:47:49.968304 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/857b920c-4d70-4dd5-acdc-e35cc79d59c9-plugins-config-volume\") pod \"payload-processing-e2e-derive-bc2301a5-c68784649-mkjkf\" (UID: \"857b920c-4d70-4dd5-acdc-e35cc79d59c9\") " pod="openshift-ingress/payload-processing-e2e-derive-bc2301a5-c68784649-mkjkf" Sep 10 13:47:49.969258 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:47:49.969219 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/857b920c-4d70-4dd5-acdc-e35cc79d59c9-plugins-config-volume\") pod \"payload-processing-e2e-derive-bc2301a5-c68784649-mkjkf\" (UID: \"857b920c-4d70-4dd5-acdc-e35cc79d59c9\") " pod="openshift-ingress/payload-processing-e2e-derive-bc2301a5-c68784649-mkjkf" Sep 10 13:47:49.988533 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:47:49.988498 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8zbs6\" (UniqueName: \"kubernetes.io/projected/857b920c-4d70-4dd5-acdc-e35cc79d59c9-kube-api-access-8zbs6\") pod \"payload-processing-e2e-derive-bc2301a5-c68784649-mkjkf\" (UID: \"857b920c-4d70-4dd5-acdc-e35cc79d59c9\") " pod="openshift-ingress/payload-processing-e2e-derive-bc2301a5-c68784649-mkjkf" Sep 10 13:47:50.037571 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:47:50.037533 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-derive-bc2301a5-85f4c659cd-86fts" Sep 10 13:47:50.109595 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:47:50.109565 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-derive-bc2301a5-c68784649-mkjkf" Sep 10 13:47:50.321436 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:47:50.321401 2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-derive-bc2301a5-85f4c659cd-86fts"] Sep 10 13:47:50.322529 ip-10-0-140-8 kubenswrapper[2585]: W0910 13:47:50.322500 2585 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd7656741_4c49_4e97_b2cb_47e6b2a89a5f.slice/crio-ed70d79d9877d3b1a198fdabce0575c81e222164528ccb22c25deb197e66da85 WatchSource:0}: Error finding container ed70d79d9877d3b1a198fdabce0575c81e222164528ccb22c25deb197e66da85: Status 404 returned error can't find the container with id ed70d79d9877d3b1a198fdabce0575c81e222164528ccb22c25deb197e66da85 Sep 10 13:47:50.424034 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:47:50.424007 2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-derive-bc2301a5-c68784649-mkjkf"] Sep 10 13:47:50.425800 ip-10-0-140-8 kubenswrapper[2585]: W0910 13:47:50.425765 2585 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod857b920c_4d70_4dd5_acdc_e35cc79d59c9.slice/crio-2dda47b99bf9efc8ac651a4a8e5b584e185276291740dbac83a9c9dadb2948c3 WatchSource:0}: Error finding container 2dda47b99bf9efc8ac651a4a8e5b584e185276291740dbac83a9c9dadb2948c3: Status 404 returned error can't find the container with id 2dda47b99bf9efc8ac651a4a8e5b584e185276291740dbac83a9c9dadb2948c3 Sep 10 13:47:50.887208 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:47:50.886897 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-derive-bc2301a5-c68784649-mkjkf" event={"ID":"857b920c-4d70-4dd5-acdc-e35cc79d59c9","Type":"ContainerStarted","Data":"288fa3203dace207295848348e324cccf161fc26028a104039833ff616eef10c"} Sep 10 13:47:50.887208 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:47:50.886959 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-derive-bc2301a5-c68784649-mkjkf" event={"ID":"857b920c-4d70-4dd5-acdc-e35cc79d59c9","Type":"ContainerStarted","Data":"2dda47b99bf9efc8ac651a4a8e5b584e185276291740dbac83a9c9dadb2948c3"} Sep 10 13:47:50.887208 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:47:50.887130 2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-derive-bc2301a5-c68784649-mkjkf" Sep 10 13:47:50.889582 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:47:50.889554 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-derive-bc2301a5-85f4c659cd-86fts" event={"ID":"d7656741-4c49-4e97-b2cb-47e6b2a89a5f","Type":"ContainerStarted","Data":"b6732fd8768bdf55c171c406f66de5f236101331ac6a4265d7dc290d725a967d"} Sep 10 13:47:50.889721 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:47:50.889604 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-derive-bc2301a5-85f4c659cd-86fts" event={"ID":"d7656741-4c49-4e97-b2cb-47e6b2a89a5f","Type":"ContainerStarted","Data":"ed70d79d9877d3b1a198fdabce0575c81e222164528ccb22c25deb197e66da85"} Sep 10 13:47:50.889721 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:47:50.889633 2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-derive-bc2301a5-85f4c659cd-86fts" Sep 10 13:47:50.908840 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:47:50.908788 2585 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-derive-bc2301a5-c68784649-mkjkf" podStartSLOduration=1.908771899 podStartE2EDuration="1.908771899s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 13:47:49 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 13:47:50.906457333 +0000 UTC m=+1155.779161537" watchObservedRunningTime="2026-09-10 13:47:50.908771899 +0000 UTC m=+1155.781476163" Sep 10 13:47:50.925113 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:47:50.925060 2585 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-derive-bc2301a5-85f4c659cd-86fts" podStartSLOduration=1.925041753 podStartE2EDuration="1.925041753s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 13:47:49 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 13:47:50.924561573 +0000 UTC m=+1155.797265785" watchObservedRunningTime="2026-09-10 13:47:50.925041753 +0000 UTC m=+1155.797745954" Sep 10 13:47:52.075470 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:47:52.075418 2585 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-conflict-b-92ef2340-kserve-76ddfcb8c8-rlgxc" Sep 10 13:47:52.075470 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:47:52.075476 2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-conflict-b-92ef2340-kserve-76ddfcb8c8-rlgxc" Sep 10 13:47:52.089204 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:47:52.089170 2585 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-conflict-b-92ef2340-kserve-76ddfcb8c8-rlgxc" Sep 10 13:47:52.911334 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:47:52.911302 2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-conflict-b-92ef2340-kserve-76ddfcb8c8-rlgxc" Sep 10 13:47:54.116959 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:47:54.116924 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fxgmp_fa53c783-d40a-4024-9f7b-1d0d2b57f6dc/konnectivity-agent/0.log" limit=1024 Sep 10 13:48:01.896912 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:48:01.896837 2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-derive-bc2301a5-c68784649-mkjkf" Sep 10 13:48:01.897291 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:48:01.897021 2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-derive-bc2301a5-85f4c659cd-86fts" Sep 10 13:48:06.575534 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:48:06.575498 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fxgmp_fa53c783-d40a-4024-9f7b-1d0d2b57f6dc/konnectivity-agent/0.log" limit=1024 Sep 10 13:48:06.675634 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:48:06.675601 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fxgmp_fa53c783-d40a-4024-9f7b-1d0d2b57f6dc/konnectivity-agent/0.log" limit=1024 Sep 10 13:48:19.396096 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:48:19.396062 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fxgmp_fa53c783-d40a-4024-9f7b-1d0d2b57f6dc/konnectivity-agent/0.log" limit=1024 Sep 10 13:48:19.442422 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:48:19.442386 2585 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-derive-bc2301a5-c68784649-mkjkf"] Sep 10 13:48:19.442913 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:48:19.442860 2585 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-derive-bc2301a5-c68784649-mkjkf" podUID="857b920c-4d70-4dd5-acdc-e35cc79d59c9" containerName="payload-processing" containerID="cri-o://288fa3203dace207295848348e324cccf161fc26028a104039833ff616eef10c" gracePeriod=30 Sep 10 13:48:19.449568 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:48:19.449533 2585 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-derive-bc2301a5-85f4c659cd-86fts"] Sep 10 13:48:19.449843 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:48:19.449813 2585 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-derive-bc2301a5-85f4c659cd-86fts" podUID="d7656741-4c49-4e97-b2cb-47e6b2a89a5f" containerName="payload-pre-processing" containerID="cri-o://b6732fd8768bdf55c171c406f66de5f236101331ac6a4265d7dc290d725a967d" gracePeriod=30 Sep 10 13:48:19.474074 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:48:19.473218 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fxgmp_fa53c783-d40a-4024-9f7b-1d0d2b57f6dc/konnectivity-agent/0.log" limit=1024 Sep 10 13:48:19.541002 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:48:19.540978 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fxgmp_fa53c783-d40a-4024-9f7b-1d0d2b57f6dc/konnectivity-agent/0.log" limit=1024 Sep 10 13:48:19.633585 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:48:19.633546 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fxgmp_fa53c783-d40a-4024-9f7b-1d0d2b57f6dc/konnectivity-agent/0.log" limit=1024 Sep 10 13:48:19.719083 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:48:19.719050 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fxgmp_fa53c783-d40a-4024-9f7b-1d0d2b57f6dc/konnectivity-agent/0.log" limit=1024 Sep 10 13:48:19.723314 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:48:19.723292 2585 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-derive-bc2301a5-c68784649-mkjkf" Sep 10 13:48:19.726291 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:48:19.726272 2585 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-derive-bc2301a5-85f4c659cd-86fts" Sep 10 13:48:19.840108 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:48:19.840076 2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/857b920c-4d70-4dd5-acdc-e35cc79d59c9-plugins-config-volume\") pod \"857b920c-4d70-4dd5-acdc-e35cc79d59c9\" (UID: \"857b920c-4d70-4dd5-acdc-e35cc79d59c9\") " Sep 10 13:48:19.840318 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:48:19.840162 2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-5rxxd\" (UniqueName: \"kubernetes.io/projected/d7656741-4c49-4e97-b2cb-47e6b2a89a5f-kube-api-access-5rxxd\") pod \"d7656741-4c49-4e97-b2cb-47e6b2a89a5f\" (UID: \"d7656741-4c49-4e97-b2cb-47e6b2a89a5f\") " Sep 10 13:48:19.840318 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:48:19.840196 2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-8zbs6\" (UniqueName: \"kubernetes.io/projected/857b920c-4d70-4dd5-acdc-e35cc79d59c9-kube-api-access-8zbs6\") pod \"857b920c-4d70-4dd5-acdc-e35cc79d59c9\" (UID: \"857b920c-4d70-4dd5-acdc-e35cc79d59c9\") " Sep 10 13:48:19.840318 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:48:19.840264 2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d7656741-4c49-4e97-b2cb-47e6b2a89a5f-plugins-config-volume\") pod \"d7656741-4c49-4e97-b2cb-47e6b2a89a5f\" (UID: \"d7656741-4c49-4e97-b2cb-47e6b2a89a5f\") " Sep 10 13:48:19.840580 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:48:19.840553 2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/857b920c-4d70-4dd5-acdc-e35cc79d59c9-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "857b920c-4d70-4dd5-acdc-e35cc79d59c9" (UID: "857b920c-4d70-4dd5-acdc-e35cc79d59c9"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 10 13:48:19.840766 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:48:19.840733 2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d7656741-4c49-4e97-b2cb-47e6b2a89a5f-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "d7656741-4c49-4e97-b2cb-47e6b2a89a5f" (UID: "d7656741-4c49-4e97-b2cb-47e6b2a89a5f"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 10 13:48:19.842452 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:48:19.842428 2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/857b920c-4d70-4dd5-acdc-e35cc79d59c9-kube-api-access-8zbs6" (OuterVolumeSpecName: "kube-api-access-8zbs6") pod "857b920c-4d70-4dd5-acdc-e35cc79d59c9" (UID: "857b920c-4d70-4dd5-acdc-e35cc79d59c9"). InnerVolumeSpecName "kube-api-access-8zbs6". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 10 13:48:19.842533 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:48:19.842446 2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d7656741-4c49-4e97-b2cb-47e6b2a89a5f-kube-api-access-5rxxd" (OuterVolumeSpecName: "kube-api-access-5rxxd") pod "d7656741-4c49-4e97-b2cb-47e6b2a89a5f" (UID: "d7656741-4c49-4e97-b2cb-47e6b2a89a5f"). InnerVolumeSpecName "kube-api-access-5rxxd". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 10 13:48:19.848110 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:48:19.848082 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fxgmp_fa53c783-d40a-4024-9f7b-1d0d2b57f6dc/konnectivity-agent/0.log" limit=1024 Sep 10 13:48:19.941619 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:48:19.941588 2585 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5rxxd\" (UniqueName: \"kubernetes.io/projected/d7656741-4c49-4e97-b2cb-47e6b2a89a5f-kube-api-access-5rxxd\") on node \"ip-10-0-140-8.ec2.internal\" DevicePath \"\"" Sep 10 13:48:19.941619 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:48:19.941619 2585 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8zbs6\" (UniqueName: \"kubernetes.io/projected/857b920c-4d70-4dd5-acdc-e35cc79d59c9-kube-api-access-8zbs6\") on node \"ip-10-0-140-8.ec2.internal\" DevicePath \"\"" Sep 10 13:48:19.941814 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:48:19.941631 2585 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d7656741-4c49-4e97-b2cb-47e6b2a89a5f-plugins-config-volume\") on node \"ip-10-0-140-8.ec2.internal\" DevicePath \"\"" Sep 10 13:48:19.941814 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:48:19.941642 2585 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/857b920c-4d70-4dd5-acdc-e35cc79d59c9-plugins-config-volume\") on node \"ip-10-0-140-8.ec2.internal\" DevicePath \"\"" Sep 10 13:48:20.052717 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:48:20.052624 2585 generic.go:356] "Generic (PLEG): container finished" podID="857b920c-4d70-4dd5-acdc-e35cc79d59c9" containerID="288fa3203dace207295848348e324cccf161fc26028a104039833ff616eef10c" exitCode=0 Sep 10 13:48:20.052717 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:48:20.052672 2585 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-derive-bc2301a5-c68784649-mkjkf" Sep 10 13:48:20.052948 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:48:20.052717 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-derive-bc2301a5-c68784649-mkjkf" event={"ID":"857b920c-4d70-4dd5-acdc-e35cc79d59c9","Type":"ContainerDied","Data":"288fa3203dace207295848348e324cccf161fc26028a104039833ff616eef10c"} Sep 10 13:48:20.052948 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:48:20.052774 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-derive-bc2301a5-c68784649-mkjkf" event={"ID":"857b920c-4d70-4dd5-acdc-e35cc79d59c9","Type":"ContainerDied","Data":"2dda47b99bf9efc8ac651a4a8e5b584e185276291740dbac83a9c9dadb2948c3"} Sep 10 13:48:20.052948 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:48:20.052802 2585 scope.go:117] "RemoveContainer" containerID="288fa3203dace207295848348e324cccf161fc26028a104039833ff616eef10c" Sep 10 13:48:20.054202 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:48:20.054184 2585 generic.go:356] "Generic (PLEG): container finished" podID="d7656741-4c49-4e97-b2cb-47e6b2a89a5f" containerID="b6732fd8768bdf55c171c406f66de5f236101331ac6a4265d7dc290d725a967d" exitCode=0 Sep 10 13:48:20.054310 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:48:20.054238 2585 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-derive-bc2301a5-85f4c659cd-86fts" Sep 10 13:48:20.054383 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:48:20.054311 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-derive-bc2301a5-85f4c659cd-86fts" event={"ID":"d7656741-4c49-4e97-b2cb-47e6b2a89a5f","Type":"ContainerDied","Data":"b6732fd8768bdf55c171c406f66de5f236101331ac6a4265d7dc290d725a967d"} Sep 10 13:48:20.054383 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:48:20.054363 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-derive-bc2301a5-85f4c659cd-86fts" event={"ID":"d7656741-4c49-4e97-b2cb-47e6b2a89a5f","Type":"ContainerDied","Data":"ed70d79d9877d3b1a198fdabce0575c81e222164528ccb22c25deb197e66da85"} Sep 10 13:48:20.062767 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:48:20.062691 2585 scope.go:117] "RemoveContainer" containerID="288fa3203dace207295848348e324cccf161fc26028a104039833ff616eef10c" Sep 10 13:48:20.063545 ip-10-0-140-8 kubenswrapper[2585]: E0910 13:48:20.063360 2585 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"288fa3203dace207295848348e324cccf161fc26028a104039833ff616eef10c\": container with ID starting with 288fa3203dace207295848348e324cccf161fc26028a104039833ff616eef10c not found: ID does not exist" containerID="288fa3203dace207295848348e324cccf161fc26028a104039833ff616eef10c" Sep 10 13:48:20.063545 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:48:20.063399 2585 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"288fa3203dace207295848348e324cccf161fc26028a104039833ff616eef10c"} err="failed to get container status \"288fa3203dace207295848348e324cccf161fc26028a104039833ff616eef10c\": rpc error: code = NotFound desc = could not find container \"288fa3203dace207295848348e324cccf161fc26028a104039833ff616eef10c\": container with ID starting with 288fa3203dace207295848348e324cccf161fc26028a104039833ff616eef10c not found: ID does not exist" Sep 10 13:48:20.063545 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:48:20.063422 2585 scope.go:117] "RemoveContainer" containerID="b6732fd8768bdf55c171c406f66de5f236101331ac6a4265d7dc290d725a967d" Sep 10 13:48:20.073426 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:48:20.073407 2585 scope.go:117] "RemoveContainer" containerID="b6732fd8768bdf55c171c406f66de5f236101331ac6a4265d7dc290d725a967d" Sep 10 13:48:20.073710 ip-10-0-140-8 kubenswrapper[2585]: E0910 13:48:20.073689 2585 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b6732fd8768bdf55c171c406f66de5f236101331ac6a4265d7dc290d725a967d\": container with ID starting with b6732fd8768bdf55c171c406f66de5f236101331ac6a4265d7dc290d725a967d not found: ID does not exist" containerID="b6732fd8768bdf55c171c406f66de5f236101331ac6a4265d7dc290d725a967d" Sep 10 13:48:20.073773 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:48:20.073717 2585 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b6732fd8768bdf55c171c406f66de5f236101331ac6a4265d7dc290d725a967d"} err="failed to get container status \"b6732fd8768bdf55c171c406f66de5f236101331ac6a4265d7dc290d725a967d\": rpc error: code = NotFound desc = could not find container \"b6732fd8768bdf55c171c406f66de5f236101331ac6a4265d7dc290d725a967d\": container with ID starting with b6732fd8768bdf55c171c406f66de5f236101331ac6a4265d7dc290d725a967d not found: ID does not exist" Sep 10 13:48:20.089946 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:48:20.089922 2585 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-derive-bc2301a5-85f4c659cd-86fts"] Sep 10 13:48:20.096403 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:48:20.096380 2585 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-derive-bc2301a5-85f4c659cd-86fts"] Sep 10 13:48:20.108558 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:48:20.108533 2585 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-derive-bc2301a5-c68784649-mkjkf"] Sep 10 13:48:20.113119 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:48:20.113094 2585 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-derive-bc2301a5-c68784649-mkjkf"] Sep 10 13:48:20.559348 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:48:20.559308 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fxgmp_fa53c783-d40a-4024-9f7b-1d0d2b57f6dc/konnectivity-agent/0.log" limit=1024 Sep 10 13:48:20.666092 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:48:20.666054 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fxgmp_fa53c783-d40a-4024-9f7b-1d0d2b57f6dc/konnectivity-agent/0.log" limit=1024 Sep 10 13:48:20.760556 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:48:20.760524 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fxgmp_fa53c783-d40a-4024-9f7b-1d0d2b57f6dc/konnectivity-agent/0.log" limit=1024 Sep 10 13:48:21.647285 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:48:21.647231 2585 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="857b920c-4d70-4dd5-acdc-e35cc79d59c9" path="/var/lib/kubelet/pods/857b920c-4d70-4dd5-acdc-e35cc79d59c9/volumes" Sep 10 13:48:21.647656 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:48:21.647642 2585 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="d7656741-4c49-4e97-b2cb-47e6b2a89a5f" path="/var/lib/kubelet/pods/d7656741-4c49-4e97-b2cb-47e6b2a89a5f/volumes" Sep 10 13:48:22.339513 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:48:22.339486 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fxgmp_fa53c783-d40a-4024-9f7b-1d0d2b57f6dc/konnectivity-agent/0.log" limit=1024 Sep 10 13:48:24.186451 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:48:24.186413 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fxgmp_fa53c783-d40a-4024-9f7b-1d0d2b57f6dc/konnectivity-agent/0.log" limit=1024 Sep 10 13:48:24.521639 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:48:24.521555 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fxgmp_fa53c783-d40a-4024-9f7b-1d0d2b57f6dc/konnectivity-agent/0.log" limit=1024 Sep 10 13:48:24.964440 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:48:24.964408 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fxgmp_fa53c783-d40a-4024-9f7b-1d0d2b57f6dc/konnectivity-agent/0.log" limit=1024 Sep 10 13:48:25.323406 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:48:25.323373 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fxgmp_fa53c783-d40a-4024-9f7b-1d0d2b57f6dc/konnectivity-agent/0.log" limit=1024 Sep 10 13:48:25.591564 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:48:25.591476 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fxgmp_fa53c783-d40a-4024-9f7b-1d0d2b57f6dc/konnectivity-agent/0.log" limit=1024 Sep 10 13:48:28.132722 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:48:28.132675 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fxgmp_fa53c783-d40a-4024-9f7b-1d0d2b57f6dc/konnectivity-agent/0.log" limit=1024 Sep 10 13:48:28.230153 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:48:28.230119 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fxgmp_fa53c783-d40a-4024-9f7b-1d0d2b57f6dc/konnectivity-agent/0.log" limit=1024 Sep 10 13:48:29.420216 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:48:29.420179 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fxgmp_fa53c783-d40a-4024-9f7b-1d0d2b57f6dc/konnectivity-agent/0.log" limit=1024 Sep 10 13:48:29.536288 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:48:29.536232 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fxgmp_fa53c783-d40a-4024-9f7b-1d0d2b57f6dc/konnectivity-agent/0.log" limit=1024 Sep 10 13:48:29.615715 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:48:29.615684 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fxgmp_fa53c783-d40a-4024-9f7b-1d0d2b57f6dc/konnectivity-agent/0.log" limit=1024 Sep 10 13:48:30.010804 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:48:30.010771 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fxgmp_fa53c783-d40a-4024-9f7b-1d0d2b57f6dc/konnectivity-agent/0.log" limit=1024 Sep 10 13:48:30.113482 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:48:30.113453 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fxgmp_fa53c783-d40a-4024-9f7b-1d0d2b57f6dc/konnectivity-agent/0.log" limit=1024 Sep 10 13:48:30.189114 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:48:30.189083 2585 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-explicit-5d0c07-66f7795546-nltcg"] Sep 10 13:48:30.189508 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:48:30.189495 2585 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d7656741-4c49-4e97-b2cb-47e6b2a89a5f" containerName="payload-pre-processing" Sep 10 13:48:30.189560 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:48:30.189510 2585 state_mem.go:107] "Deleted CPUSet assignment" podUID="d7656741-4c49-4e97-b2cb-47e6b2a89a5f" containerName="payload-pre-processing" Sep 10 13:48:30.189560 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:48:30.189524 2585 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="857b920c-4d70-4dd5-acdc-e35cc79d59c9" containerName="payload-processing" Sep 10 13:48:30.189560 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:48:30.189531 2585 state_mem.go:107] "Deleted CPUSet assignment" podUID="857b920c-4d70-4dd5-acdc-e35cc79d59c9" containerName="payload-processing" Sep 10 13:48:30.189651 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:48:30.189603 2585 memory_manager.go:368] "RemoveStaleState removing state" podUID="857b920c-4d70-4dd5-acdc-e35cc79d59c9" containerName="payload-processing" Sep 10 13:48:30.189651 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:48:30.189612 2585 memory_manager.go:368] "RemoveStaleState removing state" podUID="d7656741-4c49-4e97-b2cb-47e6b2a89a5f" containerName="payload-pre-processing" Sep 10 13:48:30.194331 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:48:30.194314 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-explicit-5d0c07-66f7795546-nltcg" Sep 10 13:48:30.196942 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:48:30.196912 2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-explicit-5d0c07\"" Sep 10 13:48:30.197064 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:48:30.196969 2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-explicit-5d0c07-dockercfg-pqmdr\"" Sep 10 13:48:30.205153 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:48:30.205130 2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-explicit-5d0c07-66f7795546-nltcg"] Sep 10 13:48:30.344356 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:48:30.344318 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c80ca4a8-acf3-4d86-9cd5-f202b124c17d-plugins-config-volume\") pod \"payload-pre-processing-e2e-explicit-5d0c07-66f7795546-nltcg\" (UID: \"c80ca4a8-acf3-4d86-9cd5-f202b124c17d\") " pod="openshift-ingress/payload-pre-processing-e2e-explicit-5d0c07-66f7795546-nltcg" Sep 10 13:48:30.344590 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:48:30.344383 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6j6vk\" (UniqueName: \"kubernetes.io/projected/c80ca4a8-acf3-4d86-9cd5-f202b124c17d-kube-api-access-6j6vk\") pod \"payload-pre-processing-e2e-explicit-5d0c07-66f7795546-nltcg\" (UID: \"c80ca4a8-acf3-4d86-9cd5-f202b124c17d\") " pod="openshift-ingress/payload-pre-processing-e2e-explicit-5d0c07-66f7795546-nltcg" Sep 10 13:48:30.445732 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:48:30.445697 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c80ca4a8-acf3-4d86-9cd5-f202b124c17d-plugins-config-volume\") pod \"payload-pre-processing-e2e-explicit-5d0c07-66f7795546-nltcg\" (UID: \"c80ca4a8-acf3-4d86-9cd5-f202b124c17d\") " pod="openshift-ingress/payload-pre-processing-e2e-explicit-5d0c07-66f7795546-nltcg" Sep 10 13:48:30.446123 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:48:30.445756 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6j6vk\" (UniqueName: \"kubernetes.io/projected/c80ca4a8-acf3-4d86-9cd5-f202b124c17d-kube-api-access-6j6vk\") pod \"payload-pre-processing-e2e-explicit-5d0c07-66f7795546-nltcg\" (UID: \"c80ca4a8-acf3-4d86-9cd5-f202b124c17d\") " pod="openshift-ingress/payload-pre-processing-e2e-explicit-5d0c07-66f7795546-nltcg" Sep 10 13:48:30.446419 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:48:30.446402 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c80ca4a8-acf3-4d86-9cd5-f202b124c17d-plugins-config-volume\") pod \"payload-pre-processing-e2e-explicit-5d0c07-66f7795546-nltcg\" (UID: \"c80ca4a8-acf3-4d86-9cd5-f202b124c17d\") " pod="openshift-ingress/payload-pre-processing-e2e-explicit-5d0c07-66f7795546-nltcg" Sep 10 13:48:30.458290 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:48:30.458236 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6j6vk\" (UniqueName: \"kubernetes.io/projected/c80ca4a8-acf3-4d86-9cd5-f202b124c17d-kube-api-access-6j6vk\") pod \"payload-pre-processing-e2e-explicit-5d0c07-66f7795546-nltcg\" (UID: \"c80ca4a8-acf3-4d86-9cd5-f202b124c17d\") " pod="openshift-ingress/payload-pre-processing-e2e-explicit-5d0c07-66f7795546-nltcg" Sep 10 13:48:30.505178 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:48:30.505134 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-explicit-5d0c07-66f7795546-nltcg" Sep 10 13:48:30.806054 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:48:30.806020 2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-explicit-5d0c07-66f7795546-nltcg"] Sep 10 13:48:30.808044 ip-10-0-140-8 kubenswrapper[2585]: W0910 13:48:30.808013 2585 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc80ca4a8_acf3_4d86_9cd5_f202b124c17d.slice/crio-2039dc7429dcc2deb8d3d0df906aa3572a05a789d29f67e2bf25abff6c1fcded WatchSource:0}: Error finding container 2039dc7429dcc2deb8d3d0df906aa3572a05a789d29f67e2bf25abff6c1fcded: Status 404 returned error can't find the container with id 2039dc7429dcc2deb8d3d0df906aa3572a05a789d29f67e2bf25abff6c1fcded Sep 10 13:48:31.121206 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:48:31.121172 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-explicit-5d0c07-66f7795546-nltcg" event={"ID":"c80ca4a8-acf3-4d86-9cd5-f202b124c17d","Type":"ContainerStarted","Data":"342207babb4408ebb0193d6ca571b693fc768eadfd7a3ac04428c1e85616cc37"} Sep 10 13:48:31.121391 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:48:31.121229 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-explicit-5d0c07-66f7795546-nltcg" event={"ID":"c80ca4a8-acf3-4d86-9cd5-f202b124c17d","Type":"ContainerStarted","Data":"2039dc7429dcc2deb8d3d0df906aa3572a05a789d29f67e2bf25abff6c1fcded"} Sep 10 13:48:31.121391 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:48:31.121368 2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-explicit-5d0c07-66f7795546-nltcg" Sep 10 13:48:31.143125 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:48:31.143071 2585 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-explicit-5d0c07-66f7795546-nltcg" podStartSLOduration=1.143052978 podStartE2EDuration="1.143052978s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 13:48:30 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 13:48:31.142338379 +0000 UTC m=+1196.015042607" watchObservedRunningTime="2026-09-10 13:48:31.143052978 +0000 UTC m=+1196.015757180" Sep 10 13:48:31.409326 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:48:31.409232 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fxgmp_fa53c783-d40a-4024-9f7b-1d0d2b57f6dc/konnectivity-agent/0.log" limit=1024 Sep 10 13:48:31.539214 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:48:31.539172 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fxgmp_fa53c783-d40a-4024-9f7b-1d0d2b57f6dc/konnectivity-agent/0.log" limit=1024 Sep 10 13:48:31.697062 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:48:31.696985 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fxgmp_fa53c783-d40a-4024-9f7b-1d0d2b57f6dc/konnectivity-agent/0.log" limit=1024 Sep 10 13:48:31.804519 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:48:31.804484 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fxgmp_fa53c783-d40a-4024-9f7b-1d0d2b57f6dc/konnectivity-agent/0.log" limit=1024 Sep 10 13:48:31.913380 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:48:31.913346 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fxgmp_fa53c783-d40a-4024-9f7b-1d0d2b57f6dc/konnectivity-agent/0.log" limit=1024 Sep 10 13:48:32.376833 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:48:32.376798 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fxgmp_fa53c783-d40a-4024-9f7b-1d0d2b57f6dc/konnectivity-agent/0.log" limit=1024 Sep 10 13:48:35.676003 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:48:35.675974 2585 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-l9l8q_ad555c6a-4385-4f8e-93bb-a174cfaa8e74/kube-multus/2.log" Sep 10 13:48:35.680397 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:48:35.680376 2585 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-l9l8q_ad555c6a-4385-4f8e-93bb-a174cfaa8e74/kube-multus/2.log" Sep 10 13:48:36.829182 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:48:36.829149 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fxgmp_fa53c783-d40a-4024-9f7b-1d0d2b57f6dc/konnectivity-agent/0.log" limit=1024 Sep 10 13:48:38.090157 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:48:38.090122 2585 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["llm/e2e-conflict-b-92ef2340-kserve-76ddfcb8c8-rlgxc"] Sep 10 13:48:38.113968 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:48:38.113936 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fxgmp_fa53c783-d40a-4024-9f7b-1d0d2b57f6dc/konnectivity-agent/0.log" limit=1024 Sep 10 13:48:42.128989 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:48:42.128957 2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-explicit-5d0c07-66f7795546-nltcg" Sep 10 13:48:53.109778 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:48:53.109717 2585 kuberuntime_container.go:873] "Killing container with a grace period" pod="llm/e2e-conflict-b-92ef2340-kserve-76ddfcb8c8-rlgxc" podUID="5c431918-5de0-4f81-b96a-5653bd8c67ec" containerName="main" containerID="cri-o://d99447558e64b6cf82147e7f4c75c0a962b08b4539c8163ff4639555ad7b2869" gracePeriod=45 Sep 10 13:48:53.129848 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:48:53.129823 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fxgmp_fa53c783-d40a-4024-9f7b-1d0d2b57f6dc/konnectivity-agent/0.log" limit=1024 Sep 10 13:48:53.256865 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:48:53.256828 2585 generic.go:356] "Generic (PLEG): container finished" podID="5c431918-5de0-4f81-b96a-5653bd8c67ec" containerID="d99447558e64b6cf82147e7f4c75c0a962b08b4539c8163ff4639555ad7b2869" exitCode=0 Sep 10 13:48:53.257020 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:48:53.256892 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-conflict-b-92ef2340-kserve-76ddfcb8c8-rlgxc" event={"ID":"5c431918-5de0-4f81-b96a-5653bd8c67ec","Type":"ContainerDied","Data":"d99447558e64b6cf82147e7f4c75c0a962b08b4539c8163ff4639555ad7b2869"} Sep 10 13:48:53.361139 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:48:53.361086 2585 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="llm/e2e-conflict-b-92ef2340-kserve-76ddfcb8c8-rlgxc" Sep 10 13:48:53.464136 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:48:53.464108 2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5c431918-5de0-4f81-b96a-5653bd8c67ec-home\") pod \"5c431918-5de0-4f81-b96a-5653bd8c67ec\" (UID: \"5c431918-5de0-4f81-b96a-5653bd8c67ec\") " Sep 10 13:48:53.464316 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:48:53.464177 2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5c431918-5de0-4f81-b96a-5653bd8c67ec-model-cache\") pod \"5c431918-5de0-4f81-b96a-5653bd8c67ec\" (UID: \"5c431918-5de0-4f81-b96a-5653bd8c67ec\") " Sep 10 13:48:53.464316 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:48:53.464201 2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-8hpgc\" (UniqueName: \"kubernetes.io/projected/5c431918-5de0-4f81-b96a-5653bd8c67ec-kube-api-access-8hpgc\") pod \"5c431918-5de0-4f81-b96a-5653bd8c67ec\" (UID: \"5c431918-5de0-4f81-b96a-5653bd8c67ec\") " Sep 10 13:48:53.464316 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:48:53.464266 2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5c431918-5de0-4f81-b96a-5653bd8c67ec-dshm\") pod \"5c431918-5de0-4f81-b96a-5653bd8c67ec\" (UID: \"5c431918-5de0-4f81-b96a-5653bd8c67ec\") " Sep 10 13:48:53.464444 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:48:53.464318 2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5c431918-5de0-4f81-b96a-5653bd8c67ec-tls-certs\") pod \"5c431918-5de0-4f81-b96a-5653bd8c67ec\" (UID: \"5c431918-5de0-4f81-b96a-5653bd8c67ec\") " Sep 10 13:48:53.464444 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:48:53.464351 2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5c431918-5de0-4f81-b96a-5653bd8c67ec-tmp-dir\") pod \"5c431918-5de0-4f81-b96a-5653bd8c67ec\" (UID: \"5c431918-5de0-4f81-b96a-5653bd8c67ec\") " Sep 10 13:48:53.464538 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:48:53.464449 2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5c431918-5de0-4f81-b96a-5653bd8c67ec-home" (OuterVolumeSpecName: "home") pod "5c431918-5de0-4f81-b96a-5653bd8c67ec" (UID: "5c431918-5de0-4f81-b96a-5653bd8c67ec"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 10 13:48:53.464538 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:48:53.464485 2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5c431918-5de0-4f81-b96a-5653bd8c67ec-model-cache" (OuterVolumeSpecName: "model-cache") pod "5c431918-5de0-4f81-b96a-5653bd8c67ec" (UID: "5c431918-5de0-4f81-b96a-5653bd8c67ec"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 10 13:48:53.464678 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:48:53.464658 2585 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5c431918-5de0-4f81-b96a-5653bd8c67ec-home\") on node \"ip-10-0-140-8.ec2.internal\" DevicePath \"\"" Sep 10 13:48:53.464744 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:48:53.464690 2585 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5c431918-5de0-4f81-b96a-5653bd8c67ec-model-cache\") on node \"ip-10-0-140-8.ec2.internal\" DevicePath \"\"" Sep 10 13:48:53.464744 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:48:53.464662 2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5c431918-5de0-4f81-b96a-5653bd8c67ec-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "5c431918-5de0-4f81-b96a-5653bd8c67ec" (UID: "5c431918-5de0-4f81-b96a-5653bd8c67ec"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 10 13:48:53.466505 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:48:53.466479 2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5c431918-5de0-4f81-b96a-5653bd8c67ec-dshm" (OuterVolumeSpecName: "dshm") pod "5c431918-5de0-4f81-b96a-5653bd8c67ec" (UID: "5c431918-5de0-4f81-b96a-5653bd8c67ec"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 10 13:48:53.466619 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:48:53.466524 2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5c431918-5de0-4f81-b96a-5653bd8c67ec-kube-api-access-8hpgc" (OuterVolumeSpecName: "kube-api-access-8hpgc") pod "5c431918-5de0-4f81-b96a-5653bd8c67ec" (UID: "5c431918-5de0-4f81-b96a-5653bd8c67ec"). InnerVolumeSpecName "kube-api-access-8hpgc". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 10 13:48:53.466619 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:48:53.466553 2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5c431918-5de0-4f81-b96a-5653bd8c67ec-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "5c431918-5de0-4f81-b96a-5653bd8c67ec" (UID: "5c431918-5de0-4f81-b96a-5653bd8c67ec"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 10 13:48:53.565303 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:48:53.565272 2585 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8hpgc\" (UniqueName: \"kubernetes.io/projected/5c431918-5de0-4f81-b96a-5653bd8c67ec-kube-api-access-8hpgc\") on node \"ip-10-0-140-8.ec2.internal\" DevicePath \"\"" Sep 10 13:48:53.565303 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:48:53.565300 2585 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5c431918-5de0-4f81-b96a-5653bd8c67ec-dshm\") on node \"ip-10-0-140-8.ec2.internal\" DevicePath \"\"" Sep 10 13:48:53.565487 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:48:53.565314 2585 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5c431918-5de0-4f81-b96a-5653bd8c67ec-tls-certs\") on node \"ip-10-0-140-8.ec2.internal\" DevicePath \"\"" Sep 10 13:48:53.565487 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:48:53.565349 2585 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5c431918-5de0-4f81-b96a-5653bd8c67ec-tmp-dir\") on node \"ip-10-0-140-8.ec2.internal\" DevicePath \"\"" Sep 10 13:48:54.264289 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:48:54.264260 2585 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="llm/e2e-conflict-b-92ef2340-kserve-76ddfcb8c8-rlgxc" Sep 10 13:48:54.264289 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:48:54.264286 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-conflict-b-92ef2340-kserve-76ddfcb8c8-rlgxc" event={"ID":"5c431918-5de0-4f81-b96a-5653bd8c67ec","Type":"ContainerDied","Data":"73f54f90ad503b26f28e563972bf867f0178d1a70984a2e1c8d751fff9e1830c"} Sep 10 13:48:54.264751 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:48:54.264334 2585 scope.go:117] "RemoveContainer" containerID="d99447558e64b6cf82147e7f4c75c0a962b08b4539c8163ff4639555ad7b2869" Sep 10 13:48:54.284854 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:48:54.284823 2585 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["llm/e2e-conflict-b-92ef2340-kserve-76ddfcb8c8-rlgxc"] Sep 10 13:48:54.288568 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:48:54.288547 2585 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["llm/e2e-conflict-b-92ef2340-kserve-76ddfcb8c8-rlgxc"] Sep 10 13:48:55.648149 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:48:55.648114 2585 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="5c431918-5de0-4f81-b96a-5653bd8c67ec" path="/var/lib/kubelet/pods/5c431918-5de0-4f81-b96a-5653bd8c67ec/volumes" Sep 10 13:48:56.612096 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:48:56.612058 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fxgmp_fa53c783-d40a-4024-9f7b-1d0d2b57f6dc/konnectivity-agent/0.log" limit=1024 Sep 10 13:48:58.407045 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:48:58.407012 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fxgmp_fa53c783-d40a-4024-9f7b-1d0d2b57f6dc/konnectivity-agent/0.log" limit=1024 Sep 10 13:49:07.105116 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:49:07.105073 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fxgmp_fa53c783-d40a-4024-9f7b-1d0d2b57f6dc/konnectivity-agent/0.log" limit=1024 Sep 10 13:49:07.212334 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:49:07.212295 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fxgmp_fa53c783-d40a-4024-9f7b-1d0d2b57f6dc/konnectivity-agent/0.log" limit=1024 Sep 10 13:49:07.324685 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:49:07.324650 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fxgmp_fa53c783-d40a-4024-9f7b-1d0d2b57f6dc/konnectivity-agent/0.log" limit=1024 Sep 10 13:49:17.370334 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:49:17.370295 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fxgmp_fa53c783-d40a-4024-9f7b-1d0d2b57f6dc/konnectivity-agent/0.log" limit=1024 Sep 10 13:49:23.334745 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:49:23.334714 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fxgmp_fa53c783-d40a-4024-9f7b-1d0d2b57f6dc/konnectivity-agent/0.log" limit=1024 Sep 10 13:49:30.138655 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:49:30.138618 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fxgmp_fa53c783-d40a-4024-9f7b-1d0d2b57f6dc/konnectivity-agent/0.log" limit=1024 Sep 10 13:49:30.256500 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:49:30.256463 2585 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-5d0c07-t8kfb"] Sep 10 13:49:30.256909 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:49:30.256895 2585 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5c431918-5de0-4f81-b96a-5653bd8c67ec" containerName="main" Sep 10 13:49:30.256953 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:49:30.256910 2585 state_mem.go:107] "Deleted CPUSet assignment" podUID="5c431918-5de0-4f81-b96a-5653bd8c67ec" containerName="main" Sep 10 13:49:30.257006 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:49:30.256997 2585 memory_manager.go:368] "RemoveStaleState removing state" podUID="5c431918-5de0-4f81-b96a-5653bd8c67ec" containerName="main" Sep 10 13:49:30.260403 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:49:30.260384 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-5d0c07-t8kfb" Sep 10 13:49:30.263394 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:49:30.263364 2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Sep 10 13:49:30.263499 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:49:30.263428 2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-7gcbr\"" Sep 10 13:49:30.269420 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:49:30.269393 2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-5d0c07-t8kfb"] Sep 10 13:49:30.297971 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:49:30.297934 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/335f6774-8f56-488a-881c-ae95ace1144d-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-explicit-5d0c07-t8kfb\" (UID: \"335f6774-8f56-488a-881c-ae95ace1144d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-5d0c07-t8kfb" Sep 10 13:49:30.399080 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:49:30.398996 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/335f6774-8f56-488a-881c-ae95ace1144d-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-explicit-5d0c07-t8kfb\" (UID: \"335f6774-8f56-488a-881c-ae95ace1144d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-5d0c07-t8kfb" Sep 10 13:49:30.399623 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:49:30.399601 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/335f6774-8f56-488a-881c-ae95ace1144d-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-explicit-5d0c07-t8kfb\" (UID: \"335f6774-8f56-488a-881c-ae95ace1144d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-5d0c07-t8kfb" Sep 10 13:49:30.421720 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:49:30.421682 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fxgmp_fa53c783-d40a-4024-9f7b-1d0d2b57f6dc/konnectivity-agent/0.log" limit=1024 Sep 10 13:49:30.504663 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:49:30.504633 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fxgmp_fa53c783-d40a-4024-9f7b-1d0d2b57f6dc/konnectivity-agent/0.log" limit=1024 Sep 10 13:49:30.572266 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:49:30.572215 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-5d0c07-t8kfb" Sep 10 13:49:30.616897 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:49:30.616862 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fxgmp_fa53c783-d40a-4024-9f7b-1d0d2b57f6dc/konnectivity-agent/0.log" limit=1024 Sep 10 13:49:30.849030 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:49:30.849001 2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-5d0c07-t8kfb"] Sep 10 13:49:30.850574 ip-10-0-140-8 kubenswrapper[2585]: W0910 13:49:30.850543 2585 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod335f6774_8f56_488a_881c_ae95ace1144d.slice/crio-ebb888d16e07c426e81c0f20fab5f0b5095a989dd33a913254be5073a3ea3127 WatchSource:0}: Error finding container ebb888d16e07c426e81c0f20fab5f0b5095a989dd33a913254be5073a3ea3127: Status 404 returned error can't find the container with id ebb888d16e07c426e81c0f20fab5f0b5095a989dd33a913254be5073a3ea3127 Sep 10 13:49:31.470768 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:49:31.470675 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-5d0c07-t8kfb" event={"ID":"335f6774-8f56-488a-881c-ae95ace1144d","Type":"ContainerStarted","Data":"d3ab127fa4ab61c3d341ccabb80b690cde2f4e3ea3b889842691d3dda78bd347"} Sep 10 13:49:31.470768 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:49:31.470744 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-5d0c07-t8kfb" event={"ID":"335f6774-8f56-488a-881c-ae95ace1144d","Type":"ContainerStarted","Data":"ebb888d16e07c426e81c0f20fab5f0b5095a989dd33a913254be5073a3ea3127"} Sep 10 13:49:32.478193 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:49:32.478153 2585 generic.go:356] "Generic (PLEG): container finished" podID="335f6774-8f56-488a-881c-ae95ace1144d" containerID="d3ab127fa4ab61c3d341ccabb80b690cde2f4e3ea3b889842691d3dda78bd347" exitCode=0 Sep 10 13:49:32.478585 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:49:32.478233 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-5d0c07-t8kfb" event={"ID":"335f6774-8f56-488a-881c-ae95ace1144d","Type":"ContainerDied","Data":"d3ab127fa4ab61c3d341ccabb80b690cde2f4e3ea3b889842691d3dda78bd347"} Sep 10 13:49:33.623690 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:49:33.623662 2585 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-5d0c07-t8kfb" Sep 10 13:49:33.729468 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:49:33.729428 2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/335f6774-8f56-488a-881c-ae95ace1144d-maas-api-service-ca\") pod \"335f6774-8f56-488a-881c-ae95ace1144d\" (UID: \"335f6774-8f56-488a-881c-ae95ace1144d\") " Sep 10 13:49:33.729792 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:49:33.729768 2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/335f6774-8f56-488a-881c-ae95ace1144d-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "335f6774-8f56-488a-881c-ae95ace1144d" (UID: "335f6774-8f56-488a-881c-ae95ace1144d"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 10 13:49:33.830467 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:49:33.830438 2585 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/335f6774-8f56-488a-881c-ae95ace1144d-maas-api-service-ca\") on node \"ip-10-0-140-8.ec2.internal\" DevicePath \"\"" Sep 10 13:49:34.491630 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:49:34.491594 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-5d0c07-t8kfb" event={"ID":"335f6774-8f56-488a-881c-ae95ace1144d","Type":"ContainerDied","Data":"ebb888d16e07c426e81c0f20fab5f0b5095a989dd33a913254be5073a3ea3127"} Sep 10 13:49:34.491630 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:49:34.491618 2585 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-5d0c07-t8kfb" Sep 10 13:49:34.491838 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:49:34.491672 2585 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="ebb888d16e07c426e81c0f20fab5f0b5095a989dd33a913254be5073a3ea3127" Sep 10 13:49:41.268166 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:49:41.268120 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fxgmp_fa53c783-d40a-4024-9f7b-1d0d2b57f6dc/konnectivity-agent/0.log" limit=1024 Sep 10 13:49:41.695209 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:49:41.695180 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fxgmp_fa53c783-d40a-4024-9f7b-1d0d2b57f6dc/konnectivity-agent/0.log" limit=1024 Sep 10 13:49:41.851473 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:49:41.851430 2585 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-explicit-5d0c07-66f7795546-nltcg"] Sep 10 13:49:41.851700 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:49:41.851671 2585 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-explicit-5d0c07-66f7795546-nltcg" podUID="c80ca4a8-acf3-4d86-9cd5-f202b124c17d" containerName="payload-pre-processing" containerID="cri-o://342207babb4408ebb0193d6ca571b693fc768eadfd7a3ac04428c1e85616cc37" gracePeriod=30 Sep 10 13:49:41.906041 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:49:41.906009 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fxgmp_fa53c783-d40a-4024-9f7b-1d0d2b57f6dc/konnectivity-agent/0.log" limit=1024 Sep 10 13:49:42.020850 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:49:42.020807 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fxgmp_fa53c783-d40a-4024-9f7b-1d0d2b57f6dc/konnectivity-agent/0.log" limit=1024 Sep 10 13:49:42.105214 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:49:42.105190 2585 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-explicit-5d0c07-66f7795546-nltcg" Sep 10 13:49:42.183968 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:49:42.183932 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fxgmp_fa53c783-d40a-4024-9f7b-1d0d2b57f6dc/konnectivity-agent/0.log" limit=1024 Sep 10 13:49:42.215737 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:49:42.215705 2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c80ca4a8-acf3-4d86-9cd5-f202b124c17d-plugins-config-volume\") pod \"c80ca4a8-acf3-4d86-9cd5-f202b124c17d\" (UID: \"c80ca4a8-acf3-4d86-9cd5-f202b124c17d\") " Sep 10 13:49:42.215911 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:49:42.215801 2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-6j6vk\" (UniqueName: \"kubernetes.io/projected/c80ca4a8-acf3-4d86-9cd5-f202b124c17d-kube-api-access-6j6vk\") pod \"c80ca4a8-acf3-4d86-9cd5-f202b124c17d\" (UID: \"c80ca4a8-acf3-4d86-9cd5-f202b124c17d\") " Sep 10 13:49:42.216150 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:49:42.216118 2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c80ca4a8-acf3-4d86-9cd5-f202b124c17d-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "c80ca4a8-acf3-4d86-9cd5-f202b124c17d" (UID: "c80ca4a8-acf3-4d86-9cd5-f202b124c17d"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 10 13:49:42.218304 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:49:42.218272 2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c80ca4a8-acf3-4d86-9cd5-f202b124c17d-kube-api-access-6j6vk" (OuterVolumeSpecName: "kube-api-access-6j6vk") pod "c80ca4a8-acf3-4d86-9cd5-f202b124c17d" (UID: "c80ca4a8-acf3-4d86-9cd5-f202b124c17d"). InnerVolumeSpecName "kube-api-access-6j6vk". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 10 13:49:42.276587 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:49:42.276550 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fxgmp_fa53c783-d40a-4024-9f7b-1d0d2b57f6dc/konnectivity-agent/0.log" limit=1024 Sep 10 13:49:42.317207 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:49:42.317171 2585 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c80ca4a8-acf3-4d86-9cd5-f202b124c17d-plugins-config-volume\") on node \"ip-10-0-140-8.ec2.internal\" DevicePath \"\"" Sep 10 13:49:42.317207 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:49:42.317200 2585 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6j6vk\" (UniqueName: \"kubernetes.io/projected/c80ca4a8-acf3-4d86-9cd5-f202b124c17d-kube-api-access-6j6vk\") on node \"ip-10-0-140-8.ec2.internal\" DevicePath \"\"" Sep 10 13:49:42.345225 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:49:42.345193 2585 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-5d0c07-t8kfb"] Sep 10 13:49:42.350463 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:49:42.350440 2585 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-5d0c07-t8kfb"] Sep 10 13:49:42.370422 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:49:42.370386 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fxgmp_fa53c783-d40a-4024-9f7b-1d0d2b57f6dc/konnectivity-agent/0.log" limit=1024 Sep 10 13:49:42.535592 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:49:42.535493 2585 generic.go:356] "Generic (PLEG): container finished" podID="c80ca4a8-acf3-4d86-9cd5-f202b124c17d" containerID="342207babb4408ebb0193d6ca571b693fc768eadfd7a3ac04428c1e85616cc37" exitCode=0 Sep 10 13:49:42.535592 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:49:42.535538 2585 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-explicit-5d0c07-66f7795546-nltcg" Sep 10 13:49:42.535592 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:49:42.535580 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-explicit-5d0c07-66f7795546-nltcg" event={"ID":"c80ca4a8-acf3-4d86-9cd5-f202b124c17d","Type":"ContainerDied","Data":"342207babb4408ebb0193d6ca571b693fc768eadfd7a3ac04428c1e85616cc37"} Sep 10 13:49:42.535867 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:49:42.535633 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-explicit-5d0c07-66f7795546-nltcg" event={"ID":"c80ca4a8-acf3-4d86-9cd5-f202b124c17d","Type":"ContainerDied","Data":"2039dc7429dcc2deb8d3d0df906aa3572a05a789d29f67e2bf25abff6c1fcded"} Sep 10 13:49:42.535867 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:49:42.535658 2585 scope.go:117] "RemoveContainer" containerID="342207babb4408ebb0193d6ca571b693fc768eadfd7a3ac04428c1e85616cc37" Sep 10 13:49:42.545064 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:49:42.545042 2585 scope.go:117] "RemoveContainer" containerID="342207babb4408ebb0193d6ca571b693fc768eadfd7a3ac04428c1e85616cc37" Sep 10 13:49:42.545362 ip-10-0-140-8 kubenswrapper[2585]: E0910 13:49:42.545341 2585 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"342207babb4408ebb0193d6ca571b693fc768eadfd7a3ac04428c1e85616cc37\": container with ID starting with 342207babb4408ebb0193d6ca571b693fc768eadfd7a3ac04428c1e85616cc37 not found: ID does not exist" containerID="342207babb4408ebb0193d6ca571b693fc768eadfd7a3ac04428c1e85616cc37" Sep 10 13:49:42.545444 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:49:42.545402 2585 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"342207babb4408ebb0193d6ca571b693fc768eadfd7a3ac04428c1e85616cc37"} err="failed to get container status \"342207babb4408ebb0193d6ca571b693fc768eadfd7a3ac04428c1e85616cc37\": rpc error: code = NotFound desc = could not find container \"342207babb4408ebb0193d6ca571b693fc768eadfd7a3ac04428c1e85616cc37\": container with ID starting with 342207babb4408ebb0193d6ca571b693fc768eadfd7a3ac04428c1e85616cc37 not found: ID does not exist" Sep 10 13:49:42.570930 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:49:42.570900 2585 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-explicit-5d0c07-66f7795546-nltcg"] Sep 10 13:49:42.573564 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:49:42.573540 2585 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-explicit-5d0c07-66f7795546-nltcg"] Sep 10 13:49:42.744722 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:49:42.744689 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fxgmp_fa53c783-d40a-4024-9f7b-1d0d2b57f6dc/konnectivity-agent/0.log" limit=1024 Sep 10 13:49:43.647798 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:49:43.647769 2585 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="335f6774-8f56-488a-881c-ae95ace1144d" path="/var/lib/kubelet/pods/335f6774-8f56-488a-881c-ae95ace1144d/volumes" Sep 10 13:49:43.648163 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:49:43.648097 2585 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="c80ca4a8-acf3-4d86-9cd5-f202b124c17d" path="/var/lib/kubelet/pods/c80ca4a8-acf3-4d86-9cd5-f202b124c17d/volumes" Sep 10 13:49:44.788181 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:49:44.788103 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fxgmp_fa53c783-d40a-4024-9f7b-1d0d2b57f6dc/konnectivity-agent/0.log" limit=1024 Sep 10 13:49:52.121865 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:49:52.121813 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fxgmp_fa53c783-d40a-4024-9f7b-1d0d2b57f6dc/konnectivity-agent/0.log" limit=1024 Sep 10 13:49:52.253346 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:49:52.253315 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fxgmp_fa53c783-d40a-4024-9f7b-1d0d2b57f6dc/konnectivity-agent/0.log" limit=1024 Sep 10 13:49:52.417764 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:49:52.417652 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fxgmp_fa53c783-d40a-4024-9f7b-1d0d2b57f6dc/konnectivity-agent/0.log" limit=1024 Sep 10 13:49:52.515676 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:49:52.515640 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fxgmp_fa53c783-d40a-4024-9f7b-1d0d2b57f6dc/konnectivity-agent/0.log" limit=1024 Sep 10 13:49:52.909169 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:49:52.909139 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fxgmp_fa53c783-d40a-4024-9f7b-1d0d2b57f6dc/konnectivity-agent/0.log" limit=1024 Sep 10 13:49:53.017362 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:49:53.017320 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fxgmp_fa53c783-d40a-4024-9f7b-1d0d2b57f6dc/konnectivity-agent/0.log" limit=1024 Sep 10 13:49:53.760595 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:49:53.760562 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fxgmp_fa53c783-d40a-4024-9f7b-1d0d2b57f6dc/konnectivity-agent/0.log" limit=1024 Sep 10 13:49:53.895283 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:49:53.895253 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fxgmp_fa53c783-d40a-4024-9f7b-1d0d2b57f6dc/konnectivity-agent/0.log" limit=1024 Sep 10 13:49:54.139452 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:49:54.139393 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fxgmp_fa53c783-d40a-4024-9f7b-1d0d2b57f6dc/konnectivity-agent/0.log" limit=1024 Sep 10 13:49:54.500711 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:49:54.500618 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fxgmp_fa53c783-d40a-4024-9f7b-1d0d2b57f6dc/konnectivity-agent/0.log" limit=1024 Sep 10 13:49:55.073294 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:49:55.073264 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fxgmp_fa53c783-d40a-4024-9f7b-1d0d2b57f6dc/konnectivity-agent/0.log" limit=1024 Sep 10 13:49:56.067768 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:49:56.067729 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fxgmp_fa53c783-d40a-4024-9f7b-1d0d2b57f6dc/konnectivity-agent/0.log" limit=1024 Sep 10 13:49:56.174678 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:49:56.174647 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fxgmp_fa53c783-d40a-4024-9f7b-1d0d2b57f6dc/konnectivity-agent/0.log" limit=1024 Sep 10 13:50:02.184040 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:50:02.184004 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fxgmp_fa53c783-d40a-4024-9f7b-1d0d2b57f6dc/konnectivity-agent/0.log" limit=1024 Sep 10 13:50:02.377106 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:50:02.377070 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fxgmp_fa53c783-d40a-4024-9f7b-1d0d2b57f6dc/konnectivity-agent/0.log" limit=1024 Sep 10 13:50:02.951650 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:50:02.951619 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fxgmp_fa53c783-d40a-4024-9f7b-1d0d2b57f6dc/konnectivity-agent/0.log" limit=1024 Sep 10 13:50:10.231119 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:50:10.231087 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fxgmp_fa53c783-d40a-4024-9f7b-1d0d2b57f6dc/konnectivity-agent/0.log" limit=1024 Sep 10 13:50:10.329195 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:50:10.329151 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fxgmp_fa53c783-d40a-4024-9f7b-1d0d2b57f6dc/konnectivity-agent/0.log" limit=1024 Sep 10 13:50:10.542599 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:50:10.542506 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fxgmp_fa53c783-d40a-4024-9f7b-1d0d2b57f6dc/konnectivity-agent/0.log" limit=1024 Sep 10 13:50:10.926947 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:50:10.926918 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fxgmp_fa53c783-d40a-4024-9f7b-1d0d2b57f6dc/konnectivity-agent/0.log" limit=1024 Sep 10 13:50:11.202886 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:50:11.202806 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fxgmp_fa53c783-d40a-4024-9f7b-1d0d2b57f6dc/konnectivity-agent/0.log" limit=1024 Sep 10 13:50:13.988628 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:50:13.988592 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fxgmp_fa53c783-d40a-4024-9f7b-1d0d2b57f6dc/konnectivity-agent/0.log" limit=1024 Sep 10 13:50:14.109727 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:50:14.109690 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fxgmp_fa53c783-d40a-4024-9f7b-1d0d2b57f6dc/konnectivity-agent/0.log" limit=1024 Sep 10 13:50:15.416370 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:50:15.416336 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fxgmp_fa53c783-d40a-4024-9f7b-1d0d2b57f6dc/konnectivity-agent/0.log" limit=1024 Sep 10 13:50:15.525122 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:50:15.525089 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fxgmp_fa53c783-d40a-4024-9f7b-1d0d2b57f6dc/konnectivity-agent/0.log" limit=1024 Sep 10 13:50:15.792964 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:50:15.792877 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fxgmp_fa53c783-d40a-4024-9f7b-1d0d2b57f6dc/konnectivity-agent/0.log" limit=1024 Sep 10 13:50:16.370019 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:50:16.369987 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fxgmp_fa53c783-d40a-4024-9f7b-1d0d2b57f6dc/konnectivity-agent/0.log" limit=1024 Sep 10 13:50:16.473860 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:50:16.473826 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fxgmp_fa53c783-d40a-4024-9f7b-1d0d2b57f6dc/konnectivity-agent/0.log" limit=1024 Sep 10 13:50:16.605865 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:50:16.605832 2585 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-3fe902b6-756fd75898-85tdm"] Sep 10 13:50:16.606239 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:50:16.606227 2585 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="335f6774-8f56-488a-881c-ae95ace1144d" containerName="revoke-keys" Sep 10 13:50:16.606239 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:50:16.606254 2585 state_mem.go:107] "Deleted CPUSet assignment" podUID="335f6774-8f56-488a-881c-ae95ace1144d" containerName="revoke-keys" Sep 10 13:50:16.606372 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:50:16.606276 2585 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c80ca4a8-acf3-4d86-9cd5-f202b124c17d" containerName="payload-pre-processing" Sep 10 13:50:16.606372 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:50:16.606284 2585 state_mem.go:107] "Deleted CPUSet assignment" podUID="c80ca4a8-acf3-4d86-9cd5-f202b124c17d" containerName="payload-pre-processing" Sep 10 13:50:16.606372 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:50:16.606355 2585 memory_manager.go:368] "RemoveStaleState removing state" podUID="c80ca4a8-acf3-4d86-9cd5-f202b124c17d" containerName="payload-pre-processing" Sep 10 13:50:16.606372 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:50:16.606366 2585 memory_manager.go:368] "RemoveStaleState removing state" podUID="335f6774-8f56-488a-881c-ae95ace1144d" containerName="revoke-keys" Sep 10 13:50:16.609395 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:50:16.609376 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-3fe902b6-756fd75898-85tdm" Sep 10 13:50:16.614316 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:50:16.614294 2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-mt-3fe902b6\"" Sep 10 13:50:16.614452 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:50:16.614350 2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-mt-3fe902b6-dockercfg-p4hgb\"" Sep 10 13:50:16.636093 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:50:16.636019 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0dc33333-2406-434b-a0fb-a8e31e5b9482-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-3fe902b6-756fd75898-85tdm\" (UID: \"0dc33333-2406-434b-a0fb-a8e31e5b9482\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-3fe902b6-756fd75898-85tdm" Sep 10 13:50:16.636093 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:50:16.636081 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5hjhs\" (UniqueName: \"kubernetes.io/projected/0dc33333-2406-434b-a0fb-a8e31e5b9482-kube-api-access-5hjhs\") pod \"payload-pre-processing-e2e-mt-3fe902b6-756fd75898-85tdm\" (UID: \"0dc33333-2406-434b-a0fb-a8e31e5b9482\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-3fe902b6-756fd75898-85tdm" Sep 10 13:50:16.637232 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:50:16.637207 2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-3fe902b6-756fd75898-85tdm"] Sep 10 13:50:16.724260 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:50:16.724222 2585 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-3fe902b6-64b64df47c-wl5k7"] Sep 10 13:50:16.727421 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:50:16.727398 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-3fe902b6-64b64df47c-wl5k7" Sep 10 13:50:16.736977 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:50:16.736959 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0dc33333-2406-434b-a0fb-a8e31e5b9482-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-3fe902b6-756fd75898-85tdm\" (UID: \"0dc33333-2406-434b-a0fb-a8e31e5b9482\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-3fe902b6-756fd75898-85tdm" Sep 10 13:50:16.737071 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:50:16.736999 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5hjhs\" (UniqueName: \"kubernetes.io/projected/0dc33333-2406-434b-a0fb-a8e31e5b9482-kube-api-access-5hjhs\") pod \"payload-pre-processing-e2e-mt-3fe902b6-756fd75898-85tdm\" (UID: \"0dc33333-2406-434b-a0fb-a8e31e5b9482\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-3fe902b6-756fd75898-85tdm" Sep 10 13:50:16.737595 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:50:16.737577 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0dc33333-2406-434b-a0fb-a8e31e5b9482-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-3fe902b6-756fd75898-85tdm\" (UID: \"0dc33333-2406-434b-a0fb-a8e31e5b9482\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-3fe902b6-756fd75898-85tdm" Sep 10 13:50:16.770685 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:50:16.770660 2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-3fe902b6-64b64df47c-wl5k7"] Sep 10 13:50:16.775698 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:50:16.775671 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5hjhs\" (UniqueName: \"kubernetes.io/projected/0dc33333-2406-434b-a0fb-a8e31e5b9482-kube-api-access-5hjhs\") pod \"payload-pre-processing-e2e-mt-3fe902b6-756fd75898-85tdm\" (UID: \"0dc33333-2406-434b-a0fb-a8e31e5b9482\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-3fe902b6-756fd75898-85tdm" Sep 10 13:50:16.838287 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:50:16.838225 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4rm2m\" (UniqueName: \"kubernetes.io/projected/2203a706-a39d-433e-9439-7ff84c02d7e5-kube-api-access-4rm2m\") pod \"payload-processing-e2e-mt-3fe902b6-64b64df47c-wl5k7\" (UID: \"2203a706-a39d-433e-9439-7ff84c02d7e5\") " pod="openshift-ingress/payload-processing-e2e-mt-3fe902b6-64b64df47c-wl5k7" Sep 10 13:50:16.838287 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:50:16.838285 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2203a706-a39d-433e-9439-7ff84c02d7e5-plugins-config-volume\") pod \"payload-processing-e2e-mt-3fe902b6-64b64df47c-wl5k7\" (UID: \"2203a706-a39d-433e-9439-7ff84c02d7e5\") " pod="openshift-ingress/payload-processing-e2e-mt-3fe902b6-64b64df47c-wl5k7" Sep 10 13:50:16.919339 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:50:16.919229 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-3fe902b6-756fd75898-85tdm" Sep 10 13:50:16.939503 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:50:16.939471 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2203a706-a39d-433e-9439-7ff84c02d7e5-plugins-config-volume\") pod \"payload-processing-e2e-mt-3fe902b6-64b64df47c-wl5k7\" (UID: \"2203a706-a39d-433e-9439-7ff84c02d7e5\") " pod="openshift-ingress/payload-processing-e2e-mt-3fe902b6-64b64df47c-wl5k7" Sep 10 13:50:16.939644 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:50:16.939601 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4rm2m\" (UniqueName: \"kubernetes.io/projected/2203a706-a39d-433e-9439-7ff84c02d7e5-kube-api-access-4rm2m\") pod \"payload-processing-e2e-mt-3fe902b6-64b64df47c-wl5k7\" (UID: \"2203a706-a39d-433e-9439-7ff84c02d7e5\") " pod="openshift-ingress/payload-processing-e2e-mt-3fe902b6-64b64df47c-wl5k7" Sep 10 13:50:16.940137 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:50:16.940115 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2203a706-a39d-433e-9439-7ff84c02d7e5-plugins-config-volume\") pod \"payload-processing-e2e-mt-3fe902b6-64b64df47c-wl5k7\" (UID: \"2203a706-a39d-433e-9439-7ff84c02d7e5\") " pod="openshift-ingress/payload-processing-e2e-mt-3fe902b6-64b64df47c-wl5k7" Sep 10 13:50:16.965511 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:50:16.965480 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4rm2m\" (UniqueName: \"kubernetes.io/projected/2203a706-a39d-433e-9439-7ff84c02d7e5-kube-api-access-4rm2m\") pod \"payload-processing-e2e-mt-3fe902b6-64b64df47c-wl5k7\" (UID: \"2203a706-a39d-433e-9439-7ff84c02d7e5\") " pod="openshift-ingress/payload-processing-e2e-mt-3fe902b6-64b64df47c-wl5k7" Sep 10 13:50:17.037942 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:50:17.037909 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-3fe902b6-64b64df47c-wl5k7" Sep 10 13:50:17.220593 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:50:17.220569 2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-3fe902b6-756fd75898-85tdm"] Sep 10 13:50:17.222412 ip-10-0-140-8 kubenswrapper[2585]: W0910 13:50:17.222384 2585 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0dc33333_2406_434b_a0fb_a8e31e5b9482.slice/crio-5540f4f3ca7ff2707ed33bfdb6a253ca8b0204d1ef2d9fed276f5269497b9c54 WatchSource:0}: Error finding container 5540f4f3ca7ff2707ed33bfdb6a253ca8b0204d1ef2d9fed276f5269497b9c54: Status 404 returned error can't find the container with id 5540f4f3ca7ff2707ed33bfdb6a253ca8b0204d1ef2d9fed276f5269497b9c54 Sep 10 13:50:17.348092 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:50:17.348062 2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-3fe902b6-64b64df47c-wl5k7"] Sep 10 13:50:17.348092 ip-10-0-140-8 kubenswrapper[2585]: W0910 13:50:17.348078 2585 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2203a706_a39d_433e_9439_7ff84c02d7e5.slice/crio-dd56c64de53618485c41e73750a28ad4d868a881177a1b1c73d1040a74c04b30 WatchSource:0}: Error finding container dd56c64de53618485c41e73750a28ad4d868a881177a1b1c73d1040a74c04b30: Status 404 returned error can't find the container with id dd56c64de53618485c41e73750a28ad4d868a881177a1b1c73d1040a74c04b30 Sep 10 13:50:17.731021 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:50:17.730981 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-3fe902b6-756fd75898-85tdm" event={"ID":"0dc33333-2406-434b-a0fb-a8e31e5b9482","Type":"ContainerStarted","Data":"50396ca7a22016b4febc4f9b702b07f16019791391ecafdc341d3f428491a713"} Sep 10 13:50:17.731520 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:50:17.731046 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-3fe902b6-756fd75898-85tdm" event={"ID":"0dc33333-2406-434b-a0fb-a8e31e5b9482","Type":"ContainerStarted","Data":"5540f4f3ca7ff2707ed33bfdb6a253ca8b0204d1ef2d9fed276f5269497b9c54"} Sep 10 13:50:17.731520 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:50:17.731079 2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-mt-3fe902b6-756fd75898-85tdm" Sep 10 13:50:17.732872 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:50:17.732848 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-3fe902b6-64b64df47c-wl5k7" event={"ID":"2203a706-a39d-433e-9439-7ff84c02d7e5","Type":"ContainerStarted","Data":"bc34d6974ab2987be4731b0828ef4a734470ec0173d4dd055cee5e7624398635"} Sep 10 13:50:17.732990 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:50:17.732893 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-3fe902b6-64b64df47c-wl5k7" event={"ID":"2203a706-a39d-433e-9439-7ff84c02d7e5","Type":"ContainerStarted","Data":"dd56c64de53618485c41e73750a28ad4d868a881177a1b1c73d1040a74c04b30"} Sep 10 13:50:17.733031 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:50:17.732984 2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-mt-3fe902b6-64b64df47c-wl5k7" Sep 10 13:50:17.754730 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:50:17.754686 2585 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-mt-3fe902b6-756fd75898-85tdm" podStartSLOduration=1.754670924 podStartE2EDuration="1.754670924s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 13:50:16 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 13:50:17.752419482 +0000 UTC m=+1302.625123706" watchObservedRunningTime="2026-09-10 13:50:17.754670924 +0000 UTC m=+1302.627375179" Sep 10 13:50:17.782369 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:50:17.782311 2585 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-mt-3fe902b6-64b64df47c-wl5k7" podStartSLOduration=1.782291426 podStartE2EDuration="1.782291426s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 13:50:16 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 13:50:17.780348478 +0000 UTC m=+1302.653052673" watchObservedRunningTime="2026-09-10 13:50:17.782291426 +0000 UTC m=+1302.654995638" Sep 10 13:50:23.761515 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:50:23.761482 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fxgmp_fa53c783-d40a-4024-9f7b-1d0d2b57f6dc/konnectivity-agent/0.log" limit=1024 Sep 10 13:50:28.741060 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:50:28.741031 2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-mt-3fe902b6-64b64df47c-wl5k7" Sep 10 13:50:28.741462 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:50:28.741158 2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-mt-3fe902b6-756fd75898-85tdm" Sep 10 13:50:29.112875 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:50:29.112843 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fxgmp_fa53c783-d40a-4024-9f7b-1d0d2b57f6dc/konnectivity-agent/0.log" limit=1024 Sep 10 13:50:30.474505 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:50:30.474471 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fxgmp_fa53c783-d40a-4024-9f7b-1d0d2b57f6dc/konnectivity-agent/0.log" limit=1024 Sep 10 13:50:30.591037 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:50:30.590993 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fxgmp_fa53c783-d40a-4024-9f7b-1d0d2b57f6dc/konnectivity-agent/0.log" limit=1024 Sep 10 13:50:30.822699 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:50:30.822664 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fxgmp_fa53c783-d40a-4024-9f7b-1d0d2b57f6dc/konnectivity-agent/0.log" limit=1024 Sep 10 13:50:30.922305 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:50:30.922263 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fxgmp_fa53c783-d40a-4024-9f7b-1d0d2b57f6dc/konnectivity-agent/0.log" limit=1024 Sep 10 13:50:31.555988 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:50:31.555960 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fxgmp_fa53c783-d40a-4024-9f7b-1d0d2b57f6dc/konnectivity-agent/0.log" limit=1024 Sep 10 13:50:31.926397 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:50:31.926363 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fxgmp_fa53c783-d40a-4024-9f7b-1d0d2b57f6dc/konnectivity-agent/0.log" limit=1024 Sep 10 13:50:35.992063 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:50:35.992031 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fxgmp_fa53c783-d40a-4024-9f7b-1d0d2b57f6dc/konnectivity-agent/0.log" limit=1024 Sep 10 13:50:37.314458 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:50:37.314425 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fxgmp_fa53c783-d40a-4024-9f7b-1d0d2b57f6dc/konnectivity-agent/0.log" limit=1024 Sep 10 13:50:38.959114 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:50:38.959078 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fxgmp_fa53c783-d40a-4024-9f7b-1d0d2b57f6dc/konnectivity-agent/0.log" limit=1024 Sep 10 13:50:39.582941 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:50:39.582899 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fxgmp_fa53c783-d40a-4024-9f7b-1d0d2b57f6dc/konnectivity-agent/0.log" limit=1024 Sep 10 13:50:39.683815 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:50:39.683778 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fxgmp_fa53c783-d40a-4024-9f7b-1d0d2b57f6dc/konnectivity-agent/0.log" limit=1024 Sep 10 13:50:40.041698 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:50:40.041607 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fxgmp_fa53c783-d40a-4024-9f7b-1d0d2b57f6dc/konnectivity-agent/0.log" limit=1024 Sep 10 13:50:41.882821 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:50:41.882790 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fxgmp_fa53c783-d40a-4024-9f7b-1d0d2b57f6dc/konnectivity-agent/0.log" limit=1024 Sep 10 13:50:42.616437 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:50:42.616395 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fxgmp_fa53c783-d40a-4024-9f7b-1d0d2b57f6dc/konnectivity-agent/0.log" limit=1024 Sep 10 13:50:42.648832 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:50:42.648796 2585 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-86fdcf-6cb885c959-6vx6m"] Sep 10 13:50:42.649095 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:50:42.649063 2585 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-inf-a-86fdcf-6cb885c959-6vx6m" podUID="2ad48c0c-9065-4dd8-91d2-053505c1078d" containerName="payload-processing" containerID="cri-o://19a0d58e563bd71c676e35dea76e745dc50edfe25a2a6639fb12777e6163f2ee" gracePeriod=30 Sep 10 13:50:42.875111 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:50:42.875023 2585 generic.go:356] "Generic (PLEG): container finished" podID="2ad48c0c-9065-4dd8-91d2-053505c1078d" containerID="19a0d58e563bd71c676e35dea76e745dc50edfe25a2a6639fb12777e6163f2ee" exitCode=0 Sep 10 13:50:42.875286 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:50:42.875179 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-86fdcf-6cb885c959-6vx6m" event={"ID":"2ad48c0c-9065-4dd8-91d2-053505c1078d","Type":"ContainerDied","Data":"19a0d58e563bd71c676e35dea76e745dc50edfe25a2a6639fb12777e6163f2ee"} Sep 10 13:50:42.907780 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:50:42.907753 2585 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-86fdcf-6cb885c959-6vx6m" Sep 10 13:50:43.088004 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:50:43.087969 2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-vsbck\" (UniqueName: \"kubernetes.io/projected/2ad48c0c-9065-4dd8-91d2-053505c1078d-kube-api-access-vsbck\") pod \"2ad48c0c-9065-4dd8-91d2-053505c1078d\" (UID: \"2ad48c0c-9065-4dd8-91d2-053505c1078d\") " Sep 10 13:50:43.088206 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:50:43.088111 2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2ad48c0c-9065-4dd8-91d2-053505c1078d-plugins-config-volume\") pod \"2ad48c0c-9065-4dd8-91d2-053505c1078d\" (UID: \"2ad48c0c-9065-4dd8-91d2-053505c1078d\") " Sep 10 13:50:43.088490 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:50:43.088464 2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2ad48c0c-9065-4dd8-91d2-053505c1078d-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "2ad48c0c-9065-4dd8-91d2-053505c1078d" (UID: "2ad48c0c-9065-4dd8-91d2-053505c1078d"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 10 13:50:43.090153 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:50:43.090130 2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2ad48c0c-9065-4dd8-91d2-053505c1078d-kube-api-access-vsbck" (OuterVolumeSpecName: "kube-api-access-vsbck") pod "2ad48c0c-9065-4dd8-91d2-053505c1078d" (UID: "2ad48c0c-9065-4dd8-91d2-053505c1078d"). InnerVolumeSpecName "kube-api-access-vsbck". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 10 13:50:43.189889 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:50:43.189798 2585 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vsbck\" (UniqueName: \"kubernetes.io/projected/2ad48c0c-9065-4dd8-91d2-053505c1078d-kube-api-access-vsbck\") on node \"ip-10-0-140-8.ec2.internal\" DevicePath \"\"" Sep 10 13:50:43.189889 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:50:43.189828 2585 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2ad48c0c-9065-4dd8-91d2-053505c1078d-plugins-config-volume\") on node \"ip-10-0-140-8.ec2.internal\" DevicePath \"\"" Sep 10 13:50:43.370668 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:50:43.370636 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fxgmp_fa53c783-d40a-4024-9f7b-1d0d2b57f6dc/konnectivity-agent/0.log" limit=1024 Sep 10 13:50:43.471348 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:50:43.471261 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fxgmp_fa53c783-d40a-4024-9f7b-1d0d2b57f6dc/konnectivity-agent/0.log" limit=1024 Sep 10 13:50:43.883213 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:50:43.883182 2585 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-86fdcf-6cb885c959-6vx6m" Sep 10 13:50:43.883213 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:50:43.883192 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-86fdcf-6cb885c959-6vx6m" event={"ID":"2ad48c0c-9065-4dd8-91d2-053505c1078d","Type":"ContainerDied","Data":"49cb1f8bbe2d27d195af90904b051561ce62905b85333c009ef2b0521664b523"} Sep 10 13:50:43.883456 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:50:43.883271 2585 scope.go:117] "RemoveContainer" containerID="19a0d58e563bd71c676e35dea76e745dc50edfe25a2a6639fb12777e6163f2ee" Sep 10 13:50:43.902180 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:50:43.902143 2585 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-86fdcf-6cb885c959-6vx6m"] Sep 10 13:50:43.907709 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:50:43.907685 2585 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-86fdcf-6cb885c959-6vx6m"] Sep 10 13:50:45.647738 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:50:45.647705 2585 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="2ad48c0c-9065-4dd8-91d2-053505c1078d" path="/var/lib/kubelet/pods/2ad48c0c-9065-4dd8-91d2-053505c1078d/volumes" Sep 10 13:50:46.874263 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:50:46.874216 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fxgmp_fa53c783-d40a-4024-9f7b-1d0d2b57f6dc/konnectivity-agent/0.log" limit=1024 Sep 10 13:50:46.974589 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:50:46.974557 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fxgmp_fa53c783-d40a-4024-9f7b-1d0d2b57f6dc/konnectivity-agent/0.log" limit=1024 Sep 10 13:50:47.591156 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:50:47.591109 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fxgmp_fa53c783-d40a-4024-9f7b-1d0d2b57f6dc/konnectivity-agent/0.log" limit=1024 Sep 10 13:50:50.862227 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:50:50.862193 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fxgmp_fa53c783-d40a-4024-9f7b-1d0d2b57f6dc/konnectivity-agent/0.log" limit=1024 Sep 10 13:50:50.946123 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:50:50.946091 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fxgmp_fa53c783-d40a-4024-9f7b-1d0d2b57f6dc/konnectivity-agent/0.log" limit=1024 Sep 10 13:50:51.175540 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:50:51.175452 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fxgmp_fa53c783-d40a-4024-9f7b-1d0d2b57f6dc/konnectivity-agent/0.log" limit=1024 Sep 10 13:50:52.652457 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:50:52.652425 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fxgmp_fa53c783-d40a-4024-9f7b-1d0d2b57f6dc/konnectivity-agent/0.log" limit=1024 Sep 10 13:50:57.489645 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:50:57.489609 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fxgmp_fa53c783-d40a-4024-9f7b-1d0d2b57f6dc/konnectivity-agent/0.log" limit=1024 Sep 10 13:50:58.371094 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:50:58.371012 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fxgmp_fa53c783-d40a-4024-9f7b-1d0d2b57f6dc/konnectivity-agent/0.log" limit=1024 Sep 10 13:50:58.484029 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:50:58.483998 2585 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-3fe902b6-64b64df47c-wl5k7"] Sep 10 13:50:58.484266 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:50:58.484219 2585 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-mt-3fe902b6-64b64df47c-wl5k7" podUID="2203a706-a39d-433e-9439-7ff84c02d7e5" containerName="payload-processing" containerID="cri-o://bc34d6974ab2987be4731b0828ef4a734470ec0173d4dd055cee5e7624398635" gracePeriod=30 Sep 10 13:50:58.522347 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:50:58.522315 2585 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-3fe902b6-756fd75898-85tdm"] Sep 10 13:50:58.522778 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:50:58.522580 2585 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-mt-3fe902b6-756fd75898-85tdm" podUID="0dc33333-2406-434b-a0fb-a8e31e5b9482" containerName="payload-pre-processing" containerID="cri-o://50396ca7a22016b4febc4f9b702b07f16019791391ecafdc341d3f428491a713" gracePeriod=30 Sep 10 13:50:58.548893 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:50:58.548864 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fxgmp_fa53c783-d40a-4024-9f7b-1d0d2b57f6dc/konnectivity-agent/0.log" limit=1024 Sep 10 13:50:58.673201 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:50:58.673157 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fxgmp_fa53c783-d40a-4024-9f7b-1d0d2b57f6dc/konnectivity-agent/0.log" limit=1024 Sep 10 13:50:58.785499 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:50:58.785475 2585 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-3fe902b6-64b64df47c-wl5k7" Sep 10 13:50:58.806890 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:50:58.806865 2585 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-3fe902b6-756fd75898-85tdm" Sep 10 13:50:58.824741 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:50:58.824716 2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2203a706-a39d-433e-9439-7ff84c02d7e5-plugins-config-volume\") pod \"2203a706-a39d-433e-9439-7ff84c02d7e5\" (UID: \"2203a706-a39d-433e-9439-7ff84c02d7e5\") " Sep 10 13:50:58.824741 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:50:58.824762 2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-4rm2m\" (UniqueName: \"kubernetes.io/projected/2203a706-a39d-433e-9439-7ff84c02d7e5-kube-api-access-4rm2m\") pod \"2203a706-a39d-433e-9439-7ff84c02d7e5\" (UID: \"2203a706-a39d-433e-9439-7ff84c02d7e5\") " Sep 10 13:50:58.825134 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:50:58.825112 2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2203a706-a39d-433e-9439-7ff84c02d7e5-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "2203a706-a39d-433e-9439-7ff84c02d7e5" (UID: "2203a706-a39d-433e-9439-7ff84c02d7e5"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 10 13:50:58.826801 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:50:58.826778 2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2203a706-a39d-433e-9439-7ff84c02d7e5-kube-api-access-4rm2m" (OuterVolumeSpecName: "kube-api-access-4rm2m") pod "2203a706-a39d-433e-9439-7ff84c02d7e5" (UID: "2203a706-a39d-433e-9439-7ff84c02d7e5"). InnerVolumeSpecName "kube-api-access-4rm2m". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 10 13:50:58.880346 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:50:58.880236 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fxgmp_fa53c783-d40a-4024-9f7b-1d0d2b57f6dc/konnectivity-agent/0.log" limit=1024 Sep 10 13:50:58.925686 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:50:58.925656 2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0dc33333-2406-434b-a0fb-a8e31e5b9482-plugins-config-volume\") pod \"0dc33333-2406-434b-a0fb-a8e31e5b9482\" (UID: \"0dc33333-2406-434b-a0fb-a8e31e5b9482\") " Sep 10 13:50:58.925856 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:50:58.925692 2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-5hjhs\" (UniqueName: \"kubernetes.io/projected/0dc33333-2406-434b-a0fb-a8e31e5b9482-kube-api-access-5hjhs\") pod \"0dc33333-2406-434b-a0fb-a8e31e5b9482\" (UID: \"0dc33333-2406-434b-a0fb-a8e31e5b9482\") " Sep 10 13:50:58.925907 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:50:58.925890 2585 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2203a706-a39d-433e-9439-7ff84c02d7e5-plugins-config-volume\") on node \"ip-10-0-140-8.ec2.internal\" DevicePath \"\"" Sep 10 13:50:58.925944 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:50:58.925912 2585 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4rm2m\" (UniqueName: \"kubernetes.io/projected/2203a706-a39d-433e-9439-7ff84c02d7e5-kube-api-access-4rm2m\") on node \"ip-10-0-140-8.ec2.internal\" DevicePath \"\"" Sep 10 13:50:58.926022 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:50:58.925999 2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0dc33333-2406-434b-a0fb-a8e31e5b9482-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "0dc33333-2406-434b-a0fb-a8e31e5b9482" (UID: "0dc33333-2406-434b-a0fb-a8e31e5b9482"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 10 13:50:58.927801 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:50:58.927778 2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0dc33333-2406-434b-a0fb-a8e31e5b9482-kube-api-access-5hjhs" (OuterVolumeSpecName: "kube-api-access-5hjhs") pod "0dc33333-2406-434b-a0fb-a8e31e5b9482" (UID: "0dc33333-2406-434b-a0fb-a8e31e5b9482"). InnerVolumeSpecName "kube-api-access-5hjhs". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 10 13:50:58.969646 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:50:58.969611 2585 generic.go:356] "Generic (PLEG): container finished" podID="0dc33333-2406-434b-a0fb-a8e31e5b9482" containerID="50396ca7a22016b4febc4f9b702b07f16019791391ecafdc341d3f428491a713" exitCode=0 Sep 10 13:50:58.969841 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:50:58.969663 2585 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-3fe902b6-756fd75898-85tdm" Sep 10 13:50:58.969841 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:50:58.969705 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-3fe902b6-756fd75898-85tdm" event={"ID":"0dc33333-2406-434b-a0fb-a8e31e5b9482","Type":"ContainerDied","Data":"50396ca7a22016b4febc4f9b702b07f16019791391ecafdc341d3f428491a713"} Sep 10 13:50:58.969841 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:50:58.969770 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-3fe902b6-756fd75898-85tdm" event={"ID":"0dc33333-2406-434b-a0fb-a8e31e5b9482","Type":"ContainerDied","Data":"5540f4f3ca7ff2707ed33bfdb6a253ca8b0204d1ef2d9fed276f5269497b9c54"} Sep 10 13:50:58.969841 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:50:58.969802 2585 scope.go:117] "RemoveContainer" containerID="50396ca7a22016b4febc4f9b702b07f16019791391ecafdc341d3f428491a713" Sep 10 13:50:58.971234 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:50:58.971207 2585 generic.go:356] "Generic (PLEG): container finished" podID="2203a706-a39d-433e-9439-7ff84c02d7e5" containerID="bc34d6974ab2987be4731b0828ef4a734470ec0173d4dd055cee5e7624398635" exitCode=0 Sep 10 13:50:58.971392 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:50:58.971267 2585 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-3fe902b6-64b64df47c-wl5k7" Sep 10 13:50:58.971392 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:50:58.971288 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-3fe902b6-64b64df47c-wl5k7" event={"ID":"2203a706-a39d-433e-9439-7ff84c02d7e5","Type":"ContainerDied","Data":"bc34d6974ab2987be4731b0828ef4a734470ec0173d4dd055cee5e7624398635"} Sep 10 13:50:58.971392 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:50:58.971352 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-3fe902b6-64b64df47c-wl5k7" event={"ID":"2203a706-a39d-433e-9439-7ff84c02d7e5","Type":"ContainerDied","Data":"dd56c64de53618485c41e73750a28ad4d868a881177a1b1c73d1040a74c04b30"} Sep 10 13:50:58.981967 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:50:58.981941 2585 scope.go:117] "RemoveContainer" containerID="50396ca7a22016b4febc4f9b702b07f16019791391ecafdc341d3f428491a713" Sep 10 13:50:58.982566 ip-10-0-140-8 kubenswrapper[2585]: E0910 13:50:58.982526 2585 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"50396ca7a22016b4febc4f9b702b07f16019791391ecafdc341d3f428491a713\": container with ID starting with 50396ca7a22016b4febc4f9b702b07f16019791391ecafdc341d3f428491a713 not found: ID does not exist" containerID="50396ca7a22016b4febc4f9b702b07f16019791391ecafdc341d3f428491a713" Sep 10 13:50:58.982675 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:50:58.982577 2585 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"50396ca7a22016b4febc4f9b702b07f16019791391ecafdc341d3f428491a713"} err="failed to get container status \"50396ca7a22016b4febc4f9b702b07f16019791391ecafdc341d3f428491a713\": rpc error: code = NotFound desc = could not find container \"50396ca7a22016b4febc4f9b702b07f16019791391ecafdc341d3f428491a713\": container with ID starting with 50396ca7a22016b4febc4f9b702b07f16019791391ecafdc341d3f428491a713 not found: ID does not exist" Sep 10 13:50:58.982675 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:50:58.982603 2585 scope.go:117] "RemoveContainer" containerID="bc34d6974ab2987be4731b0828ef4a734470ec0173d4dd055cee5e7624398635" Sep 10 13:50:58.989499 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:50:58.989461 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fxgmp_fa53c783-d40a-4024-9f7b-1d0d2b57f6dc/konnectivity-agent/0.log" limit=1024 Sep 10 13:50:58.992712 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:50:58.992696 2585 scope.go:117] "RemoveContainer" containerID="bc34d6974ab2987be4731b0828ef4a734470ec0173d4dd055cee5e7624398635" Sep 10 13:50:58.992998 ip-10-0-140-8 kubenswrapper[2585]: E0910 13:50:58.992959 2585 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"bc34d6974ab2987be4731b0828ef4a734470ec0173d4dd055cee5e7624398635\": container with ID starting with bc34d6974ab2987be4731b0828ef4a734470ec0173d4dd055cee5e7624398635 not found: ID does not exist" containerID="bc34d6974ab2987be4731b0828ef4a734470ec0173d4dd055cee5e7624398635" Sep 10 13:50:58.993046 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:50:58.993004 2585 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bc34d6974ab2987be4731b0828ef4a734470ec0173d4dd055cee5e7624398635"} err="failed to get container status \"bc34d6974ab2987be4731b0828ef4a734470ec0173d4dd055cee5e7624398635\": rpc error: code = NotFound desc = could not find container \"bc34d6974ab2987be4731b0828ef4a734470ec0173d4dd055cee5e7624398635\": container with ID starting with bc34d6974ab2987be4731b0828ef4a734470ec0173d4dd055cee5e7624398635 not found: ID does not exist" Sep 10 13:50:59.015095 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:50:59.015068 2585 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-3fe902b6-756fd75898-85tdm"] Sep 10 13:50:59.022353 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:50:59.022331 2585 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-3fe902b6-756fd75898-85tdm"] Sep 10 13:50:59.026676 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:50:59.026652 2585 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0dc33333-2406-434b-a0fb-a8e31e5b9482-plugins-config-volume\") on node \"ip-10-0-140-8.ec2.internal\" DevicePath \"\"" Sep 10 13:50:59.026676 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:50:59.026675 2585 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5hjhs\" (UniqueName: \"kubernetes.io/projected/0dc33333-2406-434b-a0fb-a8e31e5b9482-kube-api-access-5hjhs\") on node \"ip-10-0-140-8.ec2.internal\" DevicePath \"\"" Sep 10 13:50:59.044797 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:50:59.044764 2585 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-3fe902b6-64b64df47c-wl5k7"] Sep 10 13:50:59.053049 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:50:59.053024 2585 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-3fe902b6-64b64df47c-wl5k7"] Sep 10 13:50:59.335520 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:50:59.335490 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fxgmp_fa53c783-d40a-4024-9f7b-1d0d2b57f6dc/konnectivity-agent/0.log" limit=1024 Sep 10 13:50:59.647170 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:50:59.647091 2585 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="0dc33333-2406-434b-a0fb-a8e31e5b9482" path="/var/lib/kubelet/pods/0dc33333-2406-434b-a0fb-a8e31e5b9482/volumes" Sep 10 13:50:59.647541 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:50:59.647487 2585 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="2203a706-a39d-433e-9439-7ff84c02d7e5" path="/var/lib/kubelet/pods/2203a706-a39d-433e-9439-7ff84c02d7e5/volumes" Sep 10 13:50:59.739708 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:50:59.739664 2585 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-mt-3fe902b6-756fd75898-85tdm" podUID="0dc33333-2406-434b-a0fb-a8e31e5b9482" containerName="payload-pre-processing" probeResult="failure" output="dial tcp 10.133.0.59:9004: i/o timeout" Sep 10 13:50:59.739887 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:50:59.739677 2585 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-mt-3fe902b6-64b64df47c-wl5k7" podUID="2203a706-a39d-433e-9439-7ff84c02d7e5" containerName="payload-processing" probeResult="failure" output="dial tcp 10.133.0.60:9004: i/o timeout" Sep 10 13:51:00.782418 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:51:00.782384 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fxgmp_fa53c783-d40a-4024-9f7b-1d0d2b57f6dc/konnectivity-agent/0.log" limit=1024 Sep 10 13:51:00.883265 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:51:00.883211 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fxgmp_fa53c783-d40a-4024-9f7b-1d0d2b57f6dc/konnectivity-agent/0.log" limit=1024 Sep 10 13:51:01.931752 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:51:01.931717 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fxgmp_fa53c783-d40a-4024-9f7b-1d0d2b57f6dc/konnectivity-agent/0.log" limit=1024 Sep 10 13:51:02.834413 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:51:02.834371 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fxgmp_fa53c783-d40a-4024-9f7b-1d0d2b57f6dc/konnectivity-agent/0.log" limit=1024 Sep 10 13:51:02.870729 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:51:02.870696 2585 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-02fc2e-c6594bd4c-bj5bk"] Sep 10 13:51:02.870958 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:51:02.870931 2585 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-inf-b-02fc2e-c6594bd4c-bj5bk" podUID="f3447c97-b582-4158-8a31-90c3a207f0f4" containerName="payload-processing" containerID="cri-o://230c4643c328de6f3997b015365c8701928136566cb5997b889f3a4a82906d7f" gracePeriod=30 Sep 10 13:51:02.890898 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:51:02.890868 2585 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-02fc2e-84df4b4db-trkgz"] Sep 10 13:51:02.891184 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:51:02.891125 2585 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-02fc2e-84df4b4db-trkgz" podUID="ea137a2a-3a6e-47dd-835c-fb1e1958e262" containerName="payload-pre-processing" containerID="cri-o://2e2cf1249869fcbfb12e625547c16c13241de951c8cf170eef922640e613b3ce" gracePeriod=30 Sep 10 13:51:02.924821 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:51:02.924794 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fxgmp_fa53c783-d40a-4024-9f7b-1d0d2b57f6dc/konnectivity-agent/0.log" limit=1024 Sep 10 13:51:02.998215 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:51:02.998180 2585 generic.go:356] "Generic (PLEG): container finished" podID="f3447c97-b582-4158-8a31-90c3a207f0f4" containerID="230c4643c328de6f3997b015365c8701928136566cb5997b889f3a4a82906d7f" exitCode=0 Sep 10 13:51:02.998782 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:51:02.998227 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-02fc2e-c6594bd4c-bj5bk" event={"ID":"f3447c97-b582-4158-8a31-90c3a207f0f4","Type":"ContainerDied","Data":"230c4643c328de6f3997b015365c8701928136566cb5997b889f3a4a82906d7f"} Sep 10 13:51:03.171436 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:51:03.171411 2585 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-02fc2e-c6594bd4c-bj5bk" Sep 10 13:51:03.174637 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:51:03.174616 2585 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-02fc2e-84df4b4db-trkgz" Sep 10 13:51:03.268603 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:51:03.268571 2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ea137a2a-3a6e-47dd-835c-fb1e1958e262-plugins-config-volume\") pod \"ea137a2a-3a6e-47dd-835c-fb1e1958e262\" (UID: \"ea137a2a-3a6e-47dd-835c-fb1e1958e262\") " Sep 10 13:51:03.268603 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:51:03.268610 2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f3447c97-b582-4158-8a31-90c3a207f0f4-plugins-config-volume\") pod \"f3447c97-b582-4158-8a31-90c3a207f0f4\" (UID: \"f3447c97-b582-4158-8a31-90c3a207f0f4\") " Sep 10 13:51:03.268862 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:51:03.268633 2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-dvns6\" (UniqueName: \"kubernetes.io/projected/f3447c97-b582-4158-8a31-90c3a207f0f4-kube-api-access-dvns6\") pod \"f3447c97-b582-4158-8a31-90c3a207f0f4\" (UID: \"f3447c97-b582-4158-8a31-90c3a207f0f4\") " Sep 10 13:51:03.268862 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:51:03.268733 2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-gxgg6\" (UniqueName: \"kubernetes.io/projected/ea137a2a-3a6e-47dd-835c-fb1e1958e262-kube-api-access-gxgg6\") pod \"ea137a2a-3a6e-47dd-835c-fb1e1958e262\" (UID: \"ea137a2a-3a6e-47dd-835c-fb1e1958e262\") " Sep 10 13:51:03.268981 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:51:03.268928 2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ea137a2a-3a6e-47dd-835c-fb1e1958e262-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "ea137a2a-3a6e-47dd-835c-fb1e1958e262" (UID: "ea137a2a-3a6e-47dd-835c-fb1e1958e262"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 10 13:51:03.269039 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:51:03.268993 2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f3447c97-b582-4158-8a31-90c3a207f0f4-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "f3447c97-b582-4158-8a31-90c3a207f0f4" (UID: "f3447c97-b582-4158-8a31-90c3a207f0f4"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 10 13:51:03.269039 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:51:03.269014 2585 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ea137a2a-3a6e-47dd-835c-fb1e1958e262-plugins-config-volume\") on node \"ip-10-0-140-8.ec2.internal\" DevicePath \"\"" Sep 10 13:51:03.270771 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:51:03.270746 2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ea137a2a-3a6e-47dd-835c-fb1e1958e262-kube-api-access-gxgg6" (OuterVolumeSpecName: "kube-api-access-gxgg6") pod "ea137a2a-3a6e-47dd-835c-fb1e1958e262" (UID: "ea137a2a-3a6e-47dd-835c-fb1e1958e262"). InnerVolumeSpecName "kube-api-access-gxgg6". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 10 13:51:03.270908 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:51:03.270873 2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f3447c97-b582-4158-8a31-90c3a207f0f4-kube-api-access-dvns6" (OuterVolumeSpecName: "kube-api-access-dvns6") pod "f3447c97-b582-4158-8a31-90c3a207f0f4" (UID: "f3447c97-b582-4158-8a31-90c3a207f0f4"). InnerVolumeSpecName "kube-api-access-dvns6". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 10 13:51:03.369485 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:51:03.369413 2585 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gxgg6\" (UniqueName: \"kubernetes.io/projected/ea137a2a-3a6e-47dd-835c-fb1e1958e262-kube-api-access-gxgg6\") on node \"ip-10-0-140-8.ec2.internal\" DevicePath \"\"" Sep 10 13:51:03.369485 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:51:03.369436 2585 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f3447c97-b582-4158-8a31-90c3a207f0f4-plugins-config-volume\") on node \"ip-10-0-140-8.ec2.internal\" DevicePath \"\"" Sep 10 13:51:03.369485 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:51:03.369446 2585 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dvns6\" (UniqueName: \"kubernetes.io/projected/f3447c97-b582-4158-8a31-90c3a207f0f4-kube-api-access-dvns6\") on node \"ip-10-0-140-8.ec2.internal\" DevicePath \"\"" Sep 10 13:51:03.874942 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:51:03.874909 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fxgmp_fa53c783-d40a-4024-9f7b-1d0d2b57f6dc/konnectivity-agent/0.log" limit=1024 Sep 10 13:51:04.014740 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:51:04.014650 2585 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-02fc2e-c6594bd4c-bj5bk" Sep 10 13:51:04.014740 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:51:04.014669 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-02fc2e-c6594bd4c-bj5bk" event={"ID":"f3447c97-b582-4158-8a31-90c3a207f0f4","Type":"ContainerDied","Data":"15fadacce6f16eb4b389a24f35e9337abd378ca32aeda001ecd591ec1efd6bd2"} Sep 10 13:51:04.015228 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:51:04.014755 2585 scope.go:117] "RemoveContainer" containerID="230c4643c328de6f3997b015365c8701928136566cb5997b889f3a4a82906d7f" Sep 10 13:51:04.016318 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:51:04.016290 2585 generic.go:356] "Generic (PLEG): container finished" podID="ea137a2a-3a6e-47dd-835c-fb1e1958e262" containerID="2e2cf1249869fcbfb12e625547c16c13241de951c8cf170eef922640e613b3ce" exitCode=0 Sep 10 13:51:04.016455 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:51:04.016348 2585 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-02fc2e-84df4b4db-trkgz" Sep 10 13:51:04.016455 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:51:04.016362 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-02fc2e-84df4b4db-trkgz" event={"ID":"ea137a2a-3a6e-47dd-835c-fb1e1958e262","Type":"ContainerDied","Data":"2e2cf1249869fcbfb12e625547c16c13241de951c8cf170eef922640e613b3ce"} Sep 10 13:51:04.016455 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:51:04.016393 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-02fc2e-84df4b4db-trkgz" event={"ID":"ea137a2a-3a6e-47dd-835c-fb1e1958e262","Type":"ContainerDied","Data":"ef897824ea311365c73010486366f319c8ebc2a28c6cf876714c6fd165461232"} Sep 10 13:51:04.023870 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:51:04.023852 2585 scope.go:117] "RemoveContainer" containerID="2e2cf1249869fcbfb12e625547c16c13241de951c8cf170eef922640e613b3ce" Sep 10 13:51:04.031793 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:51:04.031770 2585 scope.go:117] "RemoveContainer" containerID="2e2cf1249869fcbfb12e625547c16c13241de951c8cf170eef922640e613b3ce" Sep 10 13:51:04.032064 ip-10-0-140-8 kubenswrapper[2585]: E0910 13:51:04.032042 2585 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2e2cf1249869fcbfb12e625547c16c13241de951c8cf170eef922640e613b3ce\": container with ID starting with 2e2cf1249869fcbfb12e625547c16c13241de951c8cf170eef922640e613b3ce not found: ID does not exist" containerID="2e2cf1249869fcbfb12e625547c16c13241de951c8cf170eef922640e613b3ce" Sep 10 13:51:04.032149 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:51:04.032070 2585 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2e2cf1249869fcbfb12e625547c16c13241de951c8cf170eef922640e613b3ce"} err="failed to get container status \"2e2cf1249869fcbfb12e625547c16c13241de951c8cf170eef922640e613b3ce\": rpc error: code = NotFound desc = could not find container \"2e2cf1249869fcbfb12e625547c16c13241de951c8cf170eef922640e613b3ce\": container with ID starting with 2e2cf1249869fcbfb12e625547c16c13241de951c8cf170eef922640e613b3ce not found: ID does not exist" Sep 10 13:51:04.034018 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:51:04.033995 2585 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-02fc2e-c6594bd4c-bj5bk"] Sep 10 13:51:04.038081 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:51:04.038060 2585 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-02fc2e-c6594bd4c-bj5bk"] Sep 10 13:51:04.050414 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:51:04.050393 2585 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-02fc2e-84df4b4db-trkgz"] Sep 10 13:51:04.056330 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:51:04.056305 2585 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-02fc2e-84df4b4db-trkgz"] Sep 10 13:51:04.289871 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:51:04.289779 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fxgmp_fa53c783-d40a-4024-9f7b-1d0d2b57f6dc/konnectivity-agent/0.log" limit=1024 Sep 10 13:51:05.647220 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:51:05.647190 2585 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="ea137a2a-3a6e-47dd-835c-fb1e1958e262" path="/var/lib/kubelet/pods/ea137a2a-3a6e-47dd-835c-fb1e1958e262/volumes" Sep 10 13:51:05.647624 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:51:05.647554 2585 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="f3447c97-b582-4158-8a31-90c3a207f0f4" path="/var/lib/kubelet/pods/f3447c97-b582-4158-8a31-90c3a207f0f4/volumes" Sep 10 13:51:07.082447 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:51:07.082412 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fxgmp_fa53c783-d40a-4024-9f7b-1d0d2b57f6dc/konnectivity-agent/0.log" limit=1024 Sep 10 13:51:07.152391 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:51:07.152360 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fxgmp_fa53c783-d40a-4024-9f7b-1d0d2b57f6dc/konnectivity-agent/0.log" limit=1024 Sep 10 13:51:07.418122 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:51:07.418038 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fxgmp_fa53c783-d40a-4024-9f7b-1d0d2b57f6dc/konnectivity-agent/0.log" limit=1024 Sep 10 13:51:07.802210 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:51:07.802120 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fxgmp_fa53c783-d40a-4024-9f7b-1d0d2b57f6dc/konnectivity-agent/0.log" limit=1024 Sep 10 13:51:07.996168 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:51:07.996135 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fxgmp_fa53c783-d40a-4024-9f7b-1d0d2b57f6dc/konnectivity-agent/0.log" limit=1024 Sep 10 13:51:08.090466 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:51:08.090426 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fxgmp_fa53c783-d40a-4024-9f7b-1d0d2b57f6dc/konnectivity-agent/0.log" limit=1024 Sep 10 13:51:10.930002 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:51:10.929961 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fxgmp_fa53c783-d40a-4024-9f7b-1d0d2b57f6dc/konnectivity-agent/0.log" limit=1024 Sep 10 13:51:11.034629 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:51:11.034597 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fxgmp_fa53c783-d40a-4024-9f7b-1d0d2b57f6dc/konnectivity-agent/0.log" limit=1024 Sep 10 13:51:11.181045 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:51:11.180954 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fxgmp_fa53c783-d40a-4024-9f7b-1d0d2b57f6dc/konnectivity-agent/0.log" limit=1024 Sep 10 13:51:11.779323 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:51:11.779291 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fxgmp_fa53c783-d40a-4024-9f7b-1d0d2b57f6dc/konnectivity-agent/0.log" limit=1024 Sep 10 13:51:12.105343 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:51:12.105308 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fxgmp_fa53c783-d40a-4024-9f7b-1d0d2b57f6dc/konnectivity-agent/0.log" limit=1024 Sep 10 13:51:12.516224 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:51:12.516145 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fxgmp_fa53c783-d40a-4024-9f7b-1d0d2b57f6dc/konnectivity-agent/0.log" limit=1024 Sep 10 13:51:14.256440 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:51:14.256407 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fxgmp_fa53c783-d40a-4024-9f7b-1d0d2b57f6dc/konnectivity-agent/0.log" limit=1024 Sep 10 13:51:14.750052 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:51:14.750018 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fxgmp_fa53c783-d40a-4024-9f7b-1d0d2b57f6dc/konnectivity-agent/0.log" limit=1024 Sep 10 13:51:14.850471 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:51:14.850438 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fxgmp_fa53c783-d40a-4024-9f7b-1d0d2b57f6dc/konnectivity-agent/0.log" limit=1024 Sep 10 13:51:15.443775 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:51:15.443732 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fxgmp_fa53c783-d40a-4024-9f7b-1d0d2b57f6dc/konnectivity-agent/0.log" limit=1024 Sep 10 13:51:16.165139 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:51:16.165098 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fxgmp_fa53c783-d40a-4024-9f7b-1d0d2b57f6dc/konnectivity-agent/0.log" limit=1024 Sep 10 13:51:16.278973 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:51:16.278944 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fxgmp_fa53c783-d40a-4024-9f7b-1d0d2b57f6dc/konnectivity-agent/0.log" limit=1024 Sep 10 13:51:16.362186 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:51:16.362154 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fxgmp_fa53c783-d40a-4024-9f7b-1d0d2b57f6dc/konnectivity-agent/0.log" limit=1024 Sep 10 13:51:16.875046 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:51:16.875014 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fxgmp_fa53c783-d40a-4024-9f7b-1d0d2b57f6dc/konnectivity-agent/0.log" limit=1024 Sep 10 13:51:16.962757 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:51:16.962718 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fxgmp_fa53c783-d40a-4024-9f7b-1d0d2b57f6dc/konnectivity-agent/0.log" limit=1024 Sep 10 13:51:17.137199 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:51:17.137118 2585 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w3-7a11e9-6864c5c8cd-x6gvq"] Sep 10 13:51:17.137517 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:51:17.137504 2585 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f3447c97-b582-4158-8a31-90c3a207f0f4" containerName="payload-processing" Sep 10 13:51:17.137562 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:51:17.137518 2585 state_mem.go:107] "Deleted CPUSet assignment" podUID="f3447c97-b582-4158-8a31-90c3a207f0f4" containerName="payload-processing" Sep 10 13:51:17.137562 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:51:17.137528 2585 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2ad48c0c-9065-4dd8-91d2-053505c1078d" containerName="payload-processing" Sep 10 13:51:17.137562 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:51:17.137535 2585 state_mem.go:107] "Deleted CPUSet assignment" podUID="2ad48c0c-9065-4dd8-91d2-053505c1078d" containerName="payload-processing" Sep 10 13:51:17.137562 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:51:17.137542 2585 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0dc33333-2406-434b-a0fb-a8e31e5b9482" containerName="payload-pre-processing" Sep 10 13:51:17.137562 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:51:17.137548 2585 state_mem.go:107] "Deleted CPUSet assignment" podUID="0dc33333-2406-434b-a0fb-a8e31e5b9482" containerName="payload-pre-processing" Sep 10 13:51:17.137562 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:51:17.137560 2585 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ea137a2a-3a6e-47dd-835c-fb1e1958e262" containerName="payload-pre-processing" Sep 10 13:51:17.137790 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:51:17.137565 2585 state_mem.go:107] "Deleted CPUSet assignment" podUID="ea137a2a-3a6e-47dd-835c-fb1e1958e262" containerName="payload-pre-processing" Sep 10 13:51:17.137790 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:51:17.137575 2585 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2203a706-a39d-433e-9439-7ff84c02d7e5" containerName="payload-processing" Sep 10 13:51:17.137790 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:51:17.137579 2585 state_mem.go:107] "Deleted CPUSet assignment" podUID="2203a706-a39d-433e-9439-7ff84c02d7e5" containerName="payload-processing" Sep 10 13:51:17.137790 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:51:17.137643 2585 memory_manager.go:368] "RemoveStaleState removing state" podUID="f3447c97-b582-4158-8a31-90c3a207f0f4" containerName="payload-processing" Sep 10 13:51:17.137790 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:51:17.137650 2585 memory_manager.go:368] "RemoveStaleState removing state" podUID="2203a706-a39d-433e-9439-7ff84c02d7e5" containerName="payload-processing" Sep 10 13:51:17.137790 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:51:17.137659 2585 memory_manager.go:368] "RemoveStaleState removing state" podUID="ea137a2a-3a6e-47dd-835c-fb1e1958e262" containerName="payload-pre-processing" Sep 10 13:51:17.137790 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:51:17.137667 2585 memory_manager.go:368] "RemoveStaleState removing state" podUID="0dc33333-2406-434b-a0fb-a8e31e5b9482" containerName="payload-pre-processing" Sep 10 13:51:17.137790 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:51:17.137673 2585 memory_manager.go:368] "RemoveStaleState removing state" podUID="2ad48c0c-9065-4dd8-91d2-053505c1078d" containerName="payload-processing" Sep 10 13:51:17.140577 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:51:17.140554 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-7a11e9-6864c5c8cd-x6gvq" Sep 10 13:51:17.143751 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:51:17.143728 2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-a-w3-7a11e9\"" Sep 10 13:51:17.143950 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:51:17.143931 2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-a-w3-7a11e9-dockercfg-kjc59\"" Sep 10 13:51:17.164807 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:51:17.164780 2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w3-7a11e9-6864c5c8cd-x6gvq"] Sep 10 13:51:17.302335 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:51:17.302296 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tdq8v\" (UniqueName: \"kubernetes.io/projected/bf622551-e014-4e8b-8df2-6e3723f1df26-kube-api-access-tdq8v\") pod \"payload-pre-processing-e2e-shared-a-w3-7a11e9-6864c5c8cd-x6gvq\" (UID: \"bf622551-e014-4e8b-8df2-6e3723f1df26\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-7a11e9-6864c5c8cd-x6gvq" Sep 10 13:51:17.302514 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:51:17.302431 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bf622551-e014-4e8b-8df2-6e3723f1df26-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w3-7a11e9-6864c5c8cd-x6gvq\" (UID: \"bf622551-e014-4e8b-8df2-6e3723f1df26\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-7a11e9-6864c5c8cd-x6gvq" Sep 10 13:51:17.403728 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:51:17.403633 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bf622551-e014-4e8b-8df2-6e3723f1df26-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w3-7a11e9-6864c5c8cd-x6gvq\" (UID: \"bf622551-e014-4e8b-8df2-6e3723f1df26\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-7a11e9-6864c5c8cd-x6gvq" Sep 10 13:51:17.403728 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:51:17.403704 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tdq8v\" (UniqueName: \"kubernetes.io/projected/bf622551-e014-4e8b-8df2-6e3723f1df26-kube-api-access-tdq8v\") pod \"payload-pre-processing-e2e-shared-a-w3-7a11e9-6864c5c8cd-x6gvq\" (UID: \"bf622551-e014-4e8b-8df2-6e3723f1df26\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-7a11e9-6864c5c8cd-x6gvq" Sep 10 13:51:17.404373 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:51:17.404352 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bf622551-e014-4e8b-8df2-6e3723f1df26-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w3-7a11e9-6864c5c8cd-x6gvq\" (UID: \"bf622551-e014-4e8b-8df2-6e3723f1df26\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-7a11e9-6864c5c8cd-x6gvq" Sep 10 13:51:17.427028 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:51:17.426995 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-tdq8v\" (UniqueName: \"kubernetes.io/projected/bf622551-e014-4e8b-8df2-6e3723f1df26-kube-api-access-tdq8v\") pod \"payload-pre-processing-e2e-shared-a-w3-7a11e9-6864c5c8cd-x6gvq\" (UID: \"bf622551-e014-4e8b-8df2-6e3723f1df26\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-7a11e9-6864c5c8cd-x6gvq" Sep 10 13:51:17.451997 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:51:17.451966 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-7a11e9-6864c5c8cd-x6gvq" Sep 10 13:51:17.726682 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:51:17.726648 2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w3-7a11e9-6864c5c8cd-x6gvq"] Sep 10 13:51:17.728100 ip-10-0-140-8 kubenswrapper[2585]: W0910 13:51:17.728072 2585 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbf622551_e014_4e8b_8df2_6e3723f1df26.slice/crio-b32253d06e8a79782f1959449482e730206db21d795cd1084a12edbb88ef73d4 WatchSource:0}: Error finding container b32253d06e8a79782f1959449482e730206db21d795cd1084a12edbb88ef73d4: Status 404 returned error can't find the container with id b32253d06e8a79782f1959449482e730206db21d795cd1084a12edbb88ef73d4 Sep 10 13:51:18.107439 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:51:18.107402 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-7a11e9-6864c5c8cd-x6gvq" event={"ID":"bf622551-e014-4e8b-8df2-6e3723f1df26","Type":"ContainerStarted","Data":"37c6018fefadafe3a14e52a267ff667fa9e63185d82a4e21dc44c0c6c7bf6a29"} Sep 10 13:51:18.107912 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:51:18.107460 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-7a11e9-6864c5c8cd-x6gvq" event={"ID":"bf622551-e014-4e8b-8df2-6e3723f1df26","Type":"ContainerStarted","Data":"b32253d06e8a79782f1959449482e730206db21d795cd1084a12edbb88ef73d4"} Sep 10 13:51:18.107912 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:51:18.107542 2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-7a11e9-6864c5c8cd-x6gvq" Sep 10 13:51:18.127824 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:51:18.127778 2585 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-7a11e9-6864c5c8cd-x6gvq" podStartSLOduration=1.127762844 podStartE2EDuration="1.127762844s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 13:51:17 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 13:51:18.127022474 +0000 UTC m=+1362.999726675" watchObservedRunningTime="2026-09-10 13:51:18.127762844 +0000 UTC m=+1363.000467044" Sep 10 13:51:22.889257 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:51:22.889195 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fxgmp_fa53c783-d40a-4024-9f7b-1d0d2b57f6dc/konnectivity-agent/0.log" limit=1024 Sep 10 13:51:22.968729 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:51:22.968701 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fxgmp_fa53c783-d40a-4024-9f7b-1d0d2b57f6dc/konnectivity-agent/0.log" limit=1024 Sep 10 13:51:24.022480 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:51:24.022449 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fxgmp_fa53c783-d40a-4024-9f7b-1d0d2b57f6dc/konnectivity-agent/0.log" limit=1024 Sep 10 13:51:24.350305 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:51:24.350237 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fxgmp_fa53c783-d40a-4024-9f7b-1d0d2b57f6dc/konnectivity-agent/0.log" limit=1024 Sep 10 13:51:24.593347 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:51:24.593314 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fxgmp_fa53c783-d40a-4024-9f7b-1d0d2b57f6dc/konnectivity-agent/0.log" limit=1024 Sep 10 13:51:27.034845 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:51:27.034817 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fxgmp_fa53c783-d40a-4024-9f7b-1d0d2b57f6dc/konnectivity-agent/0.log" limit=1024 Sep 10 13:51:27.132111 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:51:27.132079 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fxgmp_fa53c783-d40a-4024-9f7b-1d0d2b57f6dc/konnectivity-agent/0.log" limit=1024 Sep 10 13:51:28.282038 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:51:28.281987 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fxgmp_fa53c783-d40a-4024-9f7b-1d0d2b57f6dc/konnectivity-agent/0.log" limit=1024 Sep 10 13:51:28.379595 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:51:28.379560 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fxgmp_fa53c783-d40a-4024-9f7b-1d0d2b57f6dc/konnectivity-agent/0.log" limit=1024 Sep 10 13:51:28.976206 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:51:28.976175 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fxgmp_fa53c783-d40a-4024-9f7b-1d0d2b57f6dc/konnectivity-agent/0.log" limit=1024 Sep 10 13:51:29.074576 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:51:29.074545 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fxgmp_fa53c783-d40a-4024-9f7b-1d0d2b57f6dc/konnectivity-agent/0.log" limit=1024 Sep 10 13:51:29.096885 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:51:29.096852 2585 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w2-83c318-8c7bcb8db-9f552"] Sep 10 13:51:29.100504 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:51:29.100488 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-83c318-8c7bcb8db-9f552" Sep 10 13:51:29.103317 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:51:29.103297 2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-a-w2-83c318\"" Sep 10 13:51:29.113335 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:51:29.113312 2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w2-83c318-8c7bcb8db-9f552"] Sep 10 13:51:29.115054 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:51:29.115034 2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-7a11e9-6864c5c8cd-x6gvq" Sep 10 13:51:29.120592 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:51:29.120573 2585 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w2-83c318-5d96b6dbdf-r7lms"] Sep 10 13:51:29.124185 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:51:29.124168 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-83c318-5d96b6dbdf-r7lms" Sep 10 13:51:29.127043 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:51:29.127016 2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-a-w2-83c318-dockercfg-nr558\"" Sep 10 13:51:29.133597 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:51:29.133574 2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w2-83c318-5d96b6dbdf-r7lms"] Sep 10 13:51:29.213477 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:51:29.213447 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/be2e5ae8-d29d-46ad-b6ce-826118e204a7-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w2-83c318-8c7bcb8db-9f552\" (UID: \"be2e5ae8-d29d-46ad-b6ce-826118e204a7\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-83c318-8c7bcb8db-9f552" Sep 10 13:51:29.213654 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:51:29.213504 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7zjff\" (UniqueName: \"kubernetes.io/projected/be2e5ae8-d29d-46ad-b6ce-826118e204a7-kube-api-access-7zjff\") pod \"payload-pre-processing-e2e-shared-a-w2-83c318-8c7bcb8db-9f552\" (UID: \"be2e5ae8-d29d-46ad-b6ce-826118e204a7\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-83c318-8c7bcb8db-9f552" Sep 10 13:51:29.314812 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:51:29.314777 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7zjff\" (UniqueName: \"kubernetes.io/projected/be2e5ae8-d29d-46ad-b6ce-826118e204a7-kube-api-access-7zjff\") pod \"payload-pre-processing-e2e-shared-a-w2-83c318-8c7bcb8db-9f552\" (UID: \"be2e5ae8-d29d-46ad-b6ce-826118e204a7\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-83c318-8c7bcb8db-9f552" Sep 10 13:51:29.315197 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:51:29.314958 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sv5x5\" (UniqueName: \"kubernetes.io/projected/9299822d-564e-408d-955f-0bc46c91bf93-kube-api-access-sv5x5\") pod \"payload-processing-e2e-shared-a-w2-83c318-5d96b6dbdf-r7lms\" (UID: \"9299822d-564e-408d-955f-0bc46c91bf93\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w2-83c318-5d96b6dbdf-r7lms" Sep 10 13:51:29.315197 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:51:29.315008 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9299822d-564e-408d-955f-0bc46c91bf93-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w2-83c318-5d96b6dbdf-r7lms\" (UID: \"9299822d-564e-408d-955f-0bc46c91bf93\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w2-83c318-5d96b6dbdf-r7lms" Sep 10 13:51:29.315197 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:51:29.315072 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/be2e5ae8-d29d-46ad-b6ce-826118e204a7-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w2-83c318-8c7bcb8db-9f552\" (UID: \"be2e5ae8-d29d-46ad-b6ce-826118e204a7\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-83c318-8c7bcb8db-9f552" Sep 10 13:51:29.315743 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:51:29.315726 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/be2e5ae8-d29d-46ad-b6ce-826118e204a7-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w2-83c318-8c7bcb8db-9f552\" (UID: \"be2e5ae8-d29d-46ad-b6ce-826118e204a7\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-83c318-8c7bcb8db-9f552" Sep 10 13:51:29.324197 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:51:29.324172 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7zjff\" (UniqueName: \"kubernetes.io/projected/be2e5ae8-d29d-46ad-b6ce-826118e204a7-kube-api-access-7zjff\") pod \"payload-pre-processing-e2e-shared-a-w2-83c318-8c7bcb8db-9f552\" (UID: \"be2e5ae8-d29d-46ad-b6ce-826118e204a7\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-83c318-8c7bcb8db-9f552" Sep 10 13:51:29.411000 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:51:29.410970 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-83c318-8c7bcb8db-9f552" Sep 10 13:51:29.416344 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:51:29.416316 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sv5x5\" (UniqueName: \"kubernetes.io/projected/9299822d-564e-408d-955f-0bc46c91bf93-kube-api-access-sv5x5\") pod \"payload-processing-e2e-shared-a-w2-83c318-5d96b6dbdf-r7lms\" (UID: \"9299822d-564e-408d-955f-0bc46c91bf93\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w2-83c318-5d96b6dbdf-r7lms" Sep 10 13:51:29.416472 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:51:29.416357 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9299822d-564e-408d-955f-0bc46c91bf93-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w2-83c318-5d96b6dbdf-r7lms\" (UID: \"9299822d-564e-408d-955f-0bc46c91bf93\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w2-83c318-5d96b6dbdf-r7lms" Sep 10 13:51:29.416991 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:51:29.416967 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9299822d-564e-408d-955f-0bc46c91bf93-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w2-83c318-5d96b6dbdf-r7lms\" (UID: \"9299822d-564e-408d-955f-0bc46c91bf93\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w2-83c318-5d96b6dbdf-r7lms" Sep 10 13:51:29.425631 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:51:29.425604 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-sv5x5\" (UniqueName: \"kubernetes.io/projected/9299822d-564e-408d-955f-0bc46c91bf93-kube-api-access-sv5x5\") pod \"payload-processing-e2e-shared-a-w2-83c318-5d96b6dbdf-r7lms\" (UID: \"9299822d-564e-408d-955f-0bc46c91bf93\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w2-83c318-5d96b6dbdf-r7lms" Sep 10 13:51:29.437443 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:51:29.437416 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-83c318-5d96b6dbdf-r7lms" Sep 10 13:51:29.699279 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:51:29.699229 2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w2-83c318-8c7bcb8db-9f552"] Sep 10 13:51:29.701345 ip-10-0-140-8 kubenswrapper[2585]: W0910 13:51:29.701301 2585 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbe2e5ae8_d29d_46ad_b6ce_826118e204a7.slice/crio-a8caa053c5db3388eeed7a67dddaf081cb740a6ddd010288812c64d6d631249f WatchSource:0}: Error finding container a8caa053c5db3388eeed7a67dddaf081cb740a6ddd010288812c64d6d631249f: Status 404 returned error can't find the container with id a8caa053c5db3388eeed7a67dddaf081cb740a6ddd010288812c64d6d631249f Sep 10 13:51:29.740873 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:51:29.740846 2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w2-83c318-5d96b6dbdf-r7lms"] Sep 10 13:51:29.748449 ip-10-0-140-8 kubenswrapper[2585]: W0910 13:51:29.748423 2585 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9299822d_564e_408d_955f_0bc46c91bf93.slice/crio-befa65b74b9abd583c5175bf0d93752b0fc424f75591993153b56ff67b240e14 WatchSource:0}: Error finding container befa65b74b9abd583c5175bf0d93752b0fc424f75591993153b56ff67b240e14: Status 404 returned error can't find the container with id befa65b74b9abd583c5175bf0d93752b0fc424f75591993153b56ff67b240e14 Sep 10 13:51:30.178411 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:51:30.178372 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-83c318-8c7bcb8db-9f552" event={"ID":"be2e5ae8-d29d-46ad-b6ce-826118e204a7","Type":"ContainerStarted","Data":"6d16dae5670b1bc5e33cc6a2d013e284deb85cc12250724ef6880730aa1160f1"} Sep 10 13:51:30.178635 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:51:30.178440 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-83c318-8c7bcb8db-9f552" event={"ID":"be2e5ae8-d29d-46ad-b6ce-826118e204a7","Type":"ContainerStarted","Data":"a8caa053c5db3388eeed7a67dddaf081cb740a6ddd010288812c64d6d631249f"} Sep 10 13:51:30.178635 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:51:30.178470 2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-83c318-8c7bcb8db-9f552" Sep 10 13:51:30.180353 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:51:30.180322 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-83c318-5d96b6dbdf-r7lms" event={"ID":"9299822d-564e-408d-955f-0bc46c91bf93","Type":"ContainerStarted","Data":"07dbf626706644e9e0cca5fe277d69e73bf7075aff7b2552436e8f10bffc6c8a"} Sep 10 13:51:30.180695 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:51:30.180380 2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-83c318-5d96b6dbdf-r7lms" Sep 10 13:51:30.180695 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:51:30.180404 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-83c318-5d96b6dbdf-r7lms" event={"ID":"9299822d-564e-408d-955f-0bc46c91bf93","Type":"ContainerStarted","Data":"befa65b74b9abd583c5175bf0d93752b0fc424f75591993153b56ff67b240e14"} Sep 10 13:51:30.199346 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:51:30.199285 2585 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-83c318-8c7bcb8db-9f552" podStartSLOduration=1.199260333 podStartE2EDuration="1.199260333s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 13:51:29 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 13:51:30.196945925 +0000 UTC m=+1375.069650129" watchObservedRunningTime="2026-09-10 13:51:30.199260333 +0000 UTC m=+1375.071964535" Sep 10 13:51:30.216729 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:51:30.216674 2585 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-83c318-5d96b6dbdf-r7lms" podStartSLOduration=1.21665346 podStartE2EDuration="1.21665346s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 13:51:29 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 13:51:30.214521796 +0000 UTC m=+1375.087225998" watchObservedRunningTime="2026-09-10 13:51:30.21665346 +0000 UTC m=+1375.089357664" Sep 10 13:51:39.531578 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:51:39.531545 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fxgmp_fa53c783-d40a-4024-9f7b-1d0d2b57f6dc/konnectivity-agent/0.log" limit=1024 Sep 10 13:51:41.187000 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:51:41.186965 2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-83c318-8c7bcb8db-9f552" Sep 10 13:51:41.187458 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:51:41.187138 2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-83c318-5d96b6dbdf-r7lms" Sep 10 13:51:44.962123 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:51:44.962082 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fxgmp_fa53c783-d40a-4024-9f7b-1d0d2b57f6dc/konnectivity-agent/0.log" limit=1024 Sep 10 13:51:45.344960 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:51:45.344931 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fxgmp_fa53c783-d40a-4024-9f7b-1d0d2b57f6dc/konnectivity-agent/0.log" limit=1024 Sep 10 13:51:45.722402 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:51:45.722316 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fxgmp_fa53c783-d40a-4024-9f7b-1d0d2b57f6dc/konnectivity-agent/0.log" limit=1024 Sep 10 13:51:46.028239 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:51:46.028156 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fxgmp_fa53c783-d40a-4024-9f7b-1d0d2b57f6dc/konnectivity-agent/0.log" limit=1024 Sep 10 13:51:49.029516 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:51:49.029482 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fxgmp_fa53c783-d40a-4024-9f7b-1d0d2b57f6dc/konnectivity-agent/0.log" limit=1024 Sep 10 13:51:49.124416 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:51:49.124376 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fxgmp_fa53c783-d40a-4024-9f7b-1d0d2b57f6dc/konnectivity-agent/0.log" limit=1024 Sep 10 13:51:49.427384 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:51:49.427338 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fxgmp_fa53c783-d40a-4024-9f7b-1d0d2b57f6dc/konnectivity-agent/0.log" limit=1024 Sep 10 13:51:51.048329 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:51:51.048297 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fxgmp_fa53c783-d40a-4024-9f7b-1d0d2b57f6dc/konnectivity-agent/0.log" limit=1024 Sep 10 13:51:51.147085 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:51:51.147033 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fxgmp_fa53c783-d40a-4024-9f7b-1d0d2b57f6dc/konnectivity-agent/0.log" limit=1024 Sep 10 13:51:51.598906 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:51:51.598866 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fxgmp_fa53c783-d40a-4024-9f7b-1d0d2b57f6dc/konnectivity-agent/0.log" limit=1024 Sep 10 13:51:51.695101 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:51:51.695069 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fxgmp_fa53c783-d40a-4024-9f7b-1d0d2b57f6dc/konnectivity-agent/0.log" limit=1024 Sep 10 13:51:51.715584 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:51:51.715554 2585 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w3-a00003-66c855d86-wvhm5"] Sep 10 13:51:51.720409 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:51:51.720386 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-a00003-66c855d86-wvhm5" Sep 10 13:51:51.723391 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:51:51.723372 2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-b-w3-a00003\"" Sep 10 13:51:51.732490 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:51:51.732465 2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w3-a00003-66c855d86-wvhm5"] Sep 10 13:51:51.797425 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:51:51.797392 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fxgmp_fa53c783-d40a-4024-9f7b-1d0d2b57f6dc/konnectivity-agent/0.log" limit=1024 Sep 10 13:51:51.807444 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:51:51.807414 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-n6h4f\" (UniqueName: \"kubernetes.io/projected/f6f81da6-a652-4cdc-90ff-24e5ba9c179c-kube-api-access-n6h4f\") pod \"payload-pre-processing-e2e-shared-b-w3-a00003-66c855d86-wvhm5\" (UID: \"f6f81da6-a652-4cdc-90ff-24e5ba9c179c\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-a00003-66c855d86-wvhm5" Sep 10 13:51:51.807579 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:51:51.807555 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f6f81da6-a652-4cdc-90ff-24e5ba9c179c-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w3-a00003-66c855d86-wvhm5\" (UID: \"f6f81da6-a652-4cdc-90ff-24e5ba9c179c\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-a00003-66c855d86-wvhm5" Sep 10 13:51:51.908643 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:51:51.908561 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n6h4f\" (UniqueName: \"kubernetes.io/projected/f6f81da6-a652-4cdc-90ff-24e5ba9c179c-kube-api-access-n6h4f\") pod \"payload-pre-processing-e2e-shared-b-w3-a00003-66c855d86-wvhm5\" (UID: \"f6f81da6-a652-4cdc-90ff-24e5ba9c179c\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-a00003-66c855d86-wvhm5" Sep 10 13:51:51.908643 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:51:51.908643 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f6f81da6-a652-4cdc-90ff-24e5ba9c179c-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w3-a00003-66c855d86-wvhm5\" (UID: \"f6f81da6-a652-4cdc-90ff-24e5ba9c179c\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-a00003-66c855d86-wvhm5" Sep 10 13:51:51.909358 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:51:51.909338 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f6f81da6-a652-4cdc-90ff-24e5ba9c179c-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w3-a00003-66c855d86-wvhm5\" (UID: \"f6f81da6-a652-4cdc-90ff-24e5ba9c179c\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-a00003-66c855d86-wvhm5" Sep 10 13:51:51.921552 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:51:51.921524 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-n6h4f\" (UniqueName: \"kubernetes.io/projected/f6f81da6-a652-4cdc-90ff-24e5ba9c179c-kube-api-access-n6h4f\") pod \"payload-pre-processing-e2e-shared-b-w3-a00003-66c855d86-wvhm5\" (UID: \"f6f81da6-a652-4cdc-90ff-24e5ba9c179c\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-a00003-66c855d86-wvhm5" Sep 10 13:51:52.031272 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:51:52.031208 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-a00003-66c855d86-wvhm5" Sep 10 13:51:52.066183 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:51:52.066132 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fxgmp_fa53c783-d40a-4024-9f7b-1d0d2b57f6dc/konnectivity-agent/0.log" limit=1024 Sep 10 13:51:52.167550 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:51:52.167414 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fxgmp_fa53c783-d40a-4024-9f7b-1d0d2b57f6dc/konnectivity-agent/0.log" limit=1024 Sep 10 13:51:52.328518 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:51:52.328495 2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w3-a00003-66c855d86-wvhm5"] Sep 10 13:51:52.331402 ip-10-0-140-8 kubenswrapper[2585]: W0910 13:51:52.331367 2585 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf6f81da6_a652_4cdc_90ff_24e5ba9c179c.slice/crio-bcb5353dda29370aad474073a4fc8f3be8a8d51184c348c32d2a184d1542f42b WatchSource:0}: Error finding container bcb5353dda29370aad474073a4fc8f3be8a8d51184c348c32d2a184d1542f42b: Status 404 returned error can't find the container with id bcb5353dda29370aad474073a4fc8f3be8a8d51184c348c32d2a184d1542f42b Sep 10 13:51:52.494188 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:51:52.494101 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fxgmp_fa53c783-d40a-4024-9f7b-1d0d2b57f6dc/konnectivity-agent/0.log" limit=1024 Sep 10 13:51:52.793427 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:51:52.793335 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fxgmp_fa53c783-d40a-4024-9f7b-1d0d2b57f6dc/konnectivity-agent/0.log" limit=1024 Sep 10 13:51:53.092771 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:51:53.092731 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fxgmp_fa53c783-d40a-4024-9f7b-1d0d2b57f6dc/konnectivity-agent/0.log" limit=1024 Sep 10 13:51:53.268167 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:51:53.268135 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fxgmp_fa53c783-d40a-4024-9f7b-1d0d2b57f6dc/konnectivity-agent/0.log" limit=1024 Sep 10 13:51:53.307536 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:51:53.307497 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-a00003-66c855d86-wvhm5" event={"ID":"f6f81da6-a652-4cdc-90ff-24e5ba9c179c","Type":"ContainerStarted","Data":"8d4537d7c1f9c72ab42384dee6cd2f8e3824824cd33f07121bf07b953b180198"} Sep 10 13:51:53.307729 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:51:53.307556 2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-a00003-66c855d86-wvhm5" Sep 10 13:51:53.307729 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:51:53.307573 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-a00003-66c855d86-wvhm5" event={"ID":"f6f81da6-a652-4cdc-90ff-24e5ba9c179c","Type":"ContainerStarted","Data":"bcb5353dda29370aad474073a4fc8f3be8a8d51184c348c32d2a184d1542f42b"} Sep 10 13:51:53.327189 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:51:53.327137 2585 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-a00003-66c855d86-wvhm5" podStartSLOduration=2.327122311 podStartE2EDuration="2.327122311s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 13:51:51 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 13:51:53.325474724 +0000 UTC m=+1398.198178926" watchObservedRunningTime="2026-09-10 13:51:53.327122311 +0000 UTC m=+1398.199826512" Sep 10 13:51:56.043820 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:51:56.043785 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fxgmp_fa53c783-d40a-4024-9f7b-1d0d2b57f6dc/konnectivity-agent/0.log" limit=1024 Sep 10 13:51:56.150412 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:51:56.150375 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fxgmp_fa53c783-d40a-4024-9f7b-1d0d2b57f6dc/konnectivity-agent/0.log" limit=1024 Sep 10 13:51:57.285887 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:51:57.285848 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fxgmp_fa53c783-d40a-4024-9f7b-1d0d2b57f6dc/konnectivity-agent/0.log" limit=1024 Sep 10 13:51:57.395528 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:51:57.395490 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fxgmp_fa53c783-d40a-4024-9f7b-1d0d2b57f6dc/konnectivity-agent/0.log" limit=1024 Sep 10 13:51:57.856594 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:51:57.856561 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fxgmp_fa53c783-d40a-4024-9f7b-1d0d2b57f6dc/konnectivity-agent/0.log" limit=1024 Sep 10 13:51:57.981268 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:51:57.981210 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fxgmp_fa53c783-d40a-4024-9f7b-1d0d2b57f6dc/konnectivity-agent/0.log" limit=1024 Sep 10 13:51:58.083585 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:51:58.083534 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fxgmp_fa53c783-d40a-4024-9f7b-1d0d2b57f6dc/konnectivity-agent/0.log" limit=1024 Sep 10 13:51:58.189097 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:51:58.189005 2585 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w2-a310dd-6466864494-wfq7f"] Sep 10 13:51:58.192758 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:51:58.192735 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-a310dd-6466864494-wfq7f" Sep 10 13:51:58.196658 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:51:58.196637 2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-b-w2-a310dd\"" Sep 10 13:51:58.196658 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:51:58.196637 2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-b-w2-a310dd-dockercfg-zprmb\"" Sep 10 13:51:58.204099 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:51:58.204078 2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w2-a310dd-6466864494-wfq7f"] Sep 10 13:51:58.233597 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:51:58.233563 2585 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w2-a310dd-8457c575c-7fwnk"] Sep 10 13:51:58.238869 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:51:58.238844 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-a310dd-8457c575c-7fwnk" Sep 10 13:51:58.257787 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:51:58.257761 2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w2-a310dd-8457c575c-7fwnk"] Sep 10 13:51:58.274031 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:51:58.273990 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gqwhz\" (UniqueName: \"kubernetes.io/projected/4beb2e3e-5cc7-4dca-b4d0-5d4bfdce437e-kube-api-access-gqwhz\") pod \"payload-pre-processing-e2e-shared-b-w2-a310dd-6466864494-wfq7f\" (UID: \"4beb2e3e-5cc7-4dca-b4d0-5d4bfdce437e\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-a310dd-6466864494-wfq7f" Sep 10 13:51:58.274165 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:51:58.274113 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4beb2e3e-5cc7-4dca-b4d0-5d4bfdce437e-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w2-a310dd-6466864494-wfq7f\" (UID: \"4beb2e3e-5cc7-4dca-b4d0-5d4bfdce437e\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-a310dd-6466864494-wfq7f" Sep 10 13:51:58.375764 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:51:58.375727 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gqwhz\" (UniqueName: \"kubernetes.io/projected/4beb2e3e-5cc7-4dca-b4d0-5d4bfdce437e-kube-api-access-gqwhz\") pod \"payload-pre-processing-e2e-shared-b-w2-a310dd-6466864494-wfq7f\" (UID: \"4beb2e3e-5cc7-4dca-b4d0-5d4bfdce437e\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-a310dd-6466864494-wfq7f" Sep 10 13:51:58.376287 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:51:58.375779 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c13d1924-9f38-4994-88ba-5f381e8c3355-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w2-a310dd-8457c575c-7fwnk\" (UID: \"c13d1924-9f38-4994-88ba-5f381e8c3355\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w2-a310dd-8457c575c-7fwnk" Sep 10 13:51:58.376287 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:51:58.375860 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4beb2e3e-5cc7-4dca-b4d0-5d4bfdce437e-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w2-a310dd-6466864494-wfq7f\" (UID: \"4beb2e3e-5cc7-4dca-b4d0-5d4bfdce437e\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-a310dd-6466864494-wfq7f" Sep 10 13:51:58.376287 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:51:58.375916 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-q65qh\" (UniqueName: \"kubernetes.io/projected/c13d1924-9f38-4994-88ba-5f381e8c3355-kube-api-access-q65qh\") pod \"payload-processing-e2e-shared-b-w2-a310dd-8457c575c-7fwnk\" (UID: \"c13d1924-9f38-4994-88ba-5f381e8c3355\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w2-a310dd-8457c575c-7fwnk" Sep 10 13:51:58.376850 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:51:58.376823 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4beb2e3e-5cc7-4dca-b4d0-5d4bfdce437e-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w2-a310dd-6466864494-wfq7f\" (UID: \"4beb2e3e-5cc7-4dca-b4d0-5d4bfdce437e\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-a310dd-6466864494-wfq7f" Sep 10 13:51:58.386390 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:51:58.386365 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-gqwhz\" (UniqueName: \"kubernetes.io/projected/4beb2e3e-5cc7-4dca-b4d0-5d4bfdce437e-kube-api-access-gqwhz\") pod \"payload-pre-processing-e2e-shared-b-w2-a310dd-6466864494-wfq7f\" (UID: \"4beb2e3e-5cc7-4dca-b4d0-5d4bfdce437e\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-a310dd-6466864494-wfq7f" Sep 10 13:51:58.476581 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:51:58.476473 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c13d1924-9f38-4994-88ba-5f381e8c3355-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w2-a310dd-8457c575c-7fwnk\" (UID: \"c13d1924-9f38-4994-88ba-5f381e8c3355\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w2-a310dd-8457c575c-7fwnk" Sep 10 13:51:58.476777 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:51:58.476584 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q65qh\" (UniqueName: \"kubernetes.io/projected/c13d1924-9f38-4994-88ba-5f381e8c3355-kube-api-access-q65qh\") pod \"payload-processing-e2e-shared-b-w2-a310dd-8457c575c-7fwnk\" (UID: \"c13d1924-9f38-4994-88ba-5f381e8c3355\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w2-a310dd-8457c575c-7fwnk" Sep 10 13:51:58.477719 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:51:58.477684 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c13d1924-9f38-4994-88ba-5f381e8c3355-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w2-a310dd-8457c575c-7fwnk\" (UID: \"c13d1924-9f38-4994-88ba-5f381e8c3355\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w2-a310dd-8457c575c-7fwnk" Sep 10 13:51:58.492373 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:51:58.492349 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-q65qh\" (UniqueName: \"kubernetes.io/projected/c13d1924-9f38-4994-88ba-5f381e8c3355-kube-api-access-q65qh\") pod \"payload-processing-e2e-shared-b-w2-a310dd-8457c575c-7fwnk\" (UID: \"c13d1924-9f38-4994-88ba-5f381e8c3355\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w2-a310dd-8457c575c-7fwnk" Sep 10 13:51:58.503120 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:51:58.503098 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-a310dd-6466864494-wfq7f" Sep 10 13:51:58.549105 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:51:58.549075 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-a310dd-8457c575c-7fwnk" Sep 10 13:51:58.780912 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:51:58.780885 2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w2-a310dd-6466864494-wfq7f"] Sep 10 13:51:58.783788 ip-10-0-140-8 kubenswrapper[2585]: W0910 13:51:58.783764 2585 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4beb2e3e_5cc7_4dca_b4d0_5d4bfdce437e.slice/crio-88ef24f3758e32ab970930b4b6b5ff7f1c7f141d0fb120d8a88d3a457929081c WatchSource:0}: Error finding container 88ef24f3758e32ab970930b4b6b5ff7f1c7f141d0fb120d8a88d3a457929081c: Status 404 returned error can't find the container with id 88ef24f3758e32ab970930b4b6b5ff7f1c7f141d0fb120d8a88d3a457929081c Sep 10 13:51:58.836213 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:51:58.836191 2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w2-a310dd-8457c575c-7fwnk"] Sep 10 13:51:58.837746 ip-10-0-140-8 kubenswrapper[2585]: W0910 13:51:58.837720 2585 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc13d1924_9f38_4994_88ba_5f381e8c3355.slice/crio-972be6152c7872b2ce2f72b3d60ae6746fb30d75232e1a29f5aaa28367b053bf WatchSource:0}: Error finding container 972be6152c7872b2ce2f72b3d60ae6746fb30d75232e1a29f5aaa28367b053bf: Status 404 returned error can't find the container with id 972be6152c7872b2ce2f72b3d60ae6746fb30d75232e1a29f5aaa28367b053bf Sep 10 13:51:59.344799 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:51:59.344762 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-a310dd-6466864494-wfq7f" event={"ID":"4beb2e3e-5cc7-4dca-b4d0-5d4bfdce437e","Type":"ContainerStarted","Data":"61abf5783773209b63c65359cb933e5a3a3bdf362930c103da601b263e1b7b98"} Sep 10 13:51:59.345006 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:51:59.344823 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-a310dd-6466864494-wfq7f" event={"ID":"4beb2e3e-5cc7-4dca-b4d0-5d4bfdce437e","Type":"ContainerStarted","Data":"88ef24f3758e32ab970930b4b6b5ff7f1c7f141d0fb120d8a88d3a457929081c"} Sep 10 13:51:59.345006 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:51:59.344952 2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-a310dd-6466864494-wfq7f" Sep 10 13:51:59.346503 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:51:59.346466 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-a310dd-8457c575c-7fwnk" event={"ID":"c13d1924-9f38-4994-88ba-5f381e8c3355","Type":"ContainerStarted","Data":"75df9492db4aca6c957efa9e0654c677f88982710eda0125dfb87de8d9fad557"} Sep 10 13:51:59.346651 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:51:59.346514 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-a310dd-8457c575c-7fwnk" event={"ID":"c13d1924-9f38-4994-88ba-5f381e8c3355","Type":"ContainerStarted","Data":"972be6152c7872b2ce2f72b3d60ae6746fb30d75232e1a29f5aaa28367b053bf"} Sep 10 13:51:59.346651 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:51:59.346594 2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-a310dd-8457c575c-7fwnk" Sep 10 13:51:59.366989 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:51:59.366944 2585 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-a310dd-6466864494-wfq7f" podStartSLOduration=1.366928248 podStartE2EDuration="1.366928248s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 13:51:58 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 13:51:59.36531686 +0000 UTC m=+1404.238021061" watchObservedRunningTime="2026-09-10 13:51:59.366928248 +0000 UTC m=+1404.239632449" Sep 10 13:51:59.386654 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:51:59.386594 2585 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-a310dd-8457c575c-7fwnk" podStartSLOduration=1.386574817 podStartE2EDuration="1.386574817s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 13:51:58 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 13:51:59.386159811 +0000 UTC m=+1404.258864013" watchObservedRunningTime="2026-09-10 13:51:59.386574817 +0000 UTC m=+1404.259279029" Sep 10 13:52:04.316142 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:52:04.316113 2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-a00003-66c855d86-wvhm5" Sep 10 13:52:10.355135 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:52:10.355102 2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-a310dd-8457c575c-7fwnk" Sep 10 13:52:10.355525 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:52:10.355173 2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-a310dd-6466864494-wfq7f" Sep 10 13:52:12.189441 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:52:12.189408 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fxgmp_fa53c783-d40a-4024-9f7b-1d0d2b57f6dc/konnectivity-agent/0.log" limit=1024 Sep 10 13:52:15.076486 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:52:15.076456 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fxgmp_fa53c783-d40a-4024-9f7b-1d0d2b57f6dc/konnectivity-agent/0.log" limit=1024 Sep 10 13:52:18.573367 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:52:18.573336 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fxgmp_fa53c783-d40a-4024-9f7b-1d0d2b57f6dc/konnectivity-agent/0.log" limit=1024 Sep 10 13:52:19.484395 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:52:19.484360 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fxgmp_fa53c783-d40a-4024-9f7b-1d0d2b57f6dc/konnectivity-agent/0.log" limit=1024 Sep 10 13:52:20.264333 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:52:20.264301 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fxgmp_fa53c783-d40a-4024-9f7b-1d0d2b57f6dc/konnectivity-agent/0.log" limit=1024 Sep 10 13:52:21.101382 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:52:21.101353 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fxgmp_fa53c783-d40a-4024-9f7b-1d0d2b57f6dc/konnectivity-agent/0.log" limit=1024 Sep 10 13:52:22.011342 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:52:22.011306 2585 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7516-50e653"] Sep 10 13:52:22.014932 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:52:22.014913 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7516-50e653" Sep 10 13:52:22.019103 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:52:22.019081 2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"default-dockercfg-zn6pb\"" Sep 10 13:52:22.024457 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:52:22.024432 2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7516-50e653"] Sep 10 13:52:22.092213 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:52:22.092177 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-47z5z\" (UniqueName: \"kubernetes.io/projected/308bc622-0581-470e-a08a-eff7e13a57e8-kube-api-access-47z5z\") pod \"test-curl-7516-50e653\" (UID: \"308bc622-0581-470e-a08a-eff7e13a57e8\") " pod="opendatahub/test-curl-7516-50e653" Sep 10 13:52:22.193615 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:52:22.193574 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-47z5z\" (UniqueName: \"kubernetes.io/projected/308bc622-0581-470e-a08a-eff7e13a57e8-kube-api-access-47z5z\") pod \"test-curl-7516-50e653\" (UID: \"308bc622-0581-470e-a08a-eff7e13a57e8\") " pod="opendatahub/test-curl-7516-50e653" Sep 10 13:52:22.203632 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:52:22.203601 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-47z5z\" (UniqueName: \"kubernetes.io/projected/308bc622-0581-470e-a08a-eff7e13a57e8-kube-api-access-47z5z\") pod \"test-curl-7516-50e653\" (UID: \"308bc622-0581-470e-a08a-eff7e13a57e8\") " pod="opendatahub/test-curl-7516-50e653" Sep 10 13:52:22.324403 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:52:22.324376 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7516-50e653" Sep 10 13:52:22.620561 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:52:22.620531 2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7516-50e653"] Sep 10 13:52:22.622232 ip-10-0-140-8 kubenswrapper[2585]: W0910 13:52:22.622206 2585 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod308bc622_0581_470e_a08a_eff7e13a57e8.slice/crio-d5723febe34c9702fe2de7798e79633e43b8a049af980aa70e274a9a38e4122a WatchSource:0}: Error finding container d5723febe34c9702fe2de7798e79633e43b8a049af980aa70e274a9a38e4122a: Status 404 returned error can't find the container with id d5723febe34c9702fe2de7798e79633e43b8a049af980aa70e274a9a38e4122a Sep 10 13:52:23.488436 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:52:23.488396 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7516-50e653" event={"ID":"308bc622-0581-470e-a08a-eff7e13a57e8","Type":"ContainerStarted","Data":"b558df3b5c523d629a9ccde6912603e0148258add3dfa86c05c6b0a09888fd47"} Sep 10 13:52:23.488815 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:52:23.488446 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7516-50e653" event={"ID":"308bc622-0581-470e-a08a-eff7e13a57e8","Type":"ContainerStarted","Data":"d5723febe34c9702fe2de7798e79633e43b8a049af980aa70e274a9a38e4122a"} Sep 10 13:52:23.507147 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:52:23.507101 2585 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/test-curl-7516-50e653" podStartSLOduration=2.5070858449999998 podStartE2EDuration="2.507085845s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 13:52:21 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 13:52:23.504609396 +0000 UTC m=+1428.377313598" watchObservedRunningTime="2026-09-10 13:52:23.507085845 +0000 UTC m=+1428.379790045" Sep 10 13:52:24.127364 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:52:24.127329 2585 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7516-50e653"] Sep 10 13:52:24.130916 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:52:24.130891 2585 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7516-50e653"] Sep 10 13:52:24.287066 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:52:24.286983 2585 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7516-47709d"] Sep 10 13:52:24.287452 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:52:24.287437 2585 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="308bc622-0581-470e-a08a-eff7e13a57e8" containerName="test-curl-7516-50e653" Sep 10 13:52:24.287503 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:52:24.287453 2585 state_mem.go:107] "Deleted CPUSet assignment" podUID="308bc622-0581-470e-a08a-eff7e13a57e8" containerName="test-curl-7516-50e653" Sep 10 13:52:24.287540 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:52:24.287513 2585 memory_manager.go:368] "RemoveStaleState removing state" podUID="308bc622-0581-470e-a08a-eff7e13a57e8" containerName="test-curl-7516-50e653" Sep 10 13:52:24.290697 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:52:24.290680 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7516-47709d" Sep 10 13:52:24.307698 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:52:24.307669 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5bc2p\" (UniqueName: \"kubernetes.io/projected/5b2541cf-bd15-4153-b01f-99d6ec13f477-kube-api-access-5bc2p\") pod \"test-curl-7516-47709d\" (UID: \"5b2541cf-bd15-4153-b01f-99d6ec13f477\") " pod="opendatahub/test-curl-7516-47709d" Sep 10 13:52:24.309821 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:52:24.309797 2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7516-47709d"] Sep 10 13:52:24.408663 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:52:24.408631 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5bc2p\" (UniqueName: \"kubernetes.io/projected/5b2541cf-bd15-4153-b01f-99d6ec13f477-kube-api-access-5bc2p\") pod \"test-curl-7516-47709d\" (UID: \"5b2541cf-bd15-4153-b01f-99d6ec13f477\") " pod="opendatahub/test-curl-7516-47709d" Sep 10 13:52:24.420065 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:52:24.420043 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5bc2p\" (UniqueName: \"kubernetes.io/projected/5b2541cf-bd15-4153-b01f-99d6ec13f477-kube-api-access-5bc2p\") pod \"test-curl-7516-47709d\" (UID: \"5b2541cf-bd15-4153-b01f-99d6ec13f477\") " pod="opendatahub/test-curl-7516-47709d" Sep 10 13:52:24.493494 ip-10-0-140-8 kubenswrapper[2585]: E0910 13:52:24.493467 2585 kuberuntime_manager.go:649] "Missing actuated resource record" pod="opendatahub/test-curl-7516-50e653" container="test-curl-7516-50e653" Sep 10 13:52:24.599586 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:52:24.599555 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7516-47709d" Sep 10 13:52:24.722694 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:52:24.722657 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fxgmp_fa53c783-d40a-4024-9f7b-1d0d2b57f6dc/konnectivity-agent/0.log" limit=1024 Sep 10 13:52:24.878208 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:52:24.878182 2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7516-47709d"] Sep 10 13:52:24.880127 ip-10-0-140-8 kubenswrapper[2585]: W0910 13:52:24.880098 2585 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod5b2541cf_bd15_4153_b01f_99d6ec13f477.slice/crio-6eafaa7edac7c6c95304adb1eb1edb6edf3156cdf3b6437d3390fad118d5cc5e WatchSource:0}: Error finding container 6eafaa7edac7c6c95304adb1eb1edb6edf3156cdf3b6437d3390fad118d5cc5e: Status 404 returned error can't find the container with id 6eafaa7edac7c6c95304adb1eb1edb6edf3156cdf3b6437d3390fad118d5cc5e Sep 10 13:52:25.500481 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:52:25.500442 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7516-47709d" event={"ID":"5b2541cf-bd15-4153-b01f-99d6ec13f477","Type":"ContainerStarted","Data":"1b493a6057292e2aaf80e0feb8d1d27fda2d913a65e4201f90d2c4020605ae62"} Sep 10 13:52:25.500943 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:52:25.500489 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7516-47709d" event={"ID":"5b2541cf-bd15-4153-b01f-99d6ec13f477","Type":"ContainerStarted","Data":"6eafaa7edac7c6c95304adb1eb1edb6edf3156cdf3b6437d3390fad118d5cc5e"} Sep 10 13:52:25.500943 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:52:25.500604 2585 kuberuntime_container.go:873] "Killing container with a grace period" pod="opendatahub/test-curl-7516-50e653" podUID="308bc622-0581-470e-a08a-eff7e13a57e8" containerName="test-curl-7516-50e653" containerID="cri-o://b558df3b5c523d629a9ccde6912603e0148258add3dfa86c05c6b0a09888fd47" gracePeriod=2 Sep 10 13:52:25.521422 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:52:25.521369 2585 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/test-curl-7516-47709d" podStartSLOduration=1.521352459 podStartE2EDuration="1.521352459s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 13:52:24 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 13:52:25.519889 +0000 UTC m=+1430.392593200" watchObservedRunningTime="2026-09-10 13:52:25.521352459 +0000 UTC m=+1430.394056661" Sep 10 13:52:25.522205 ip-10-0-140-8 kubenswrapper[2585]: E0910 13:52:25.522180 2585 status_manager.go:1018] "Failed to get status for pod" err="pods \"test-curl-7516-50e653\" is forbidden: User \"system:node:ip-10-0-140-8.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-140-8.ec2.internal' and this object" podUID="308bc622-0581-470e-a08a-eff7e13a57e8" pod="opendatahub/test-curl-7516-50e653" Sep 10 13:52:25.647075 ip-10-0-140-8 kubenswrapper[2585]: E0910 13:52:25.647033 2585 status_manager.go:1018] "Failed to get status for pod" err="pods \"test-curl-7516-50e653\" is forbidden: User \"system:node:ip-10-0-140-8.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-140-8.ec2.internal' and this object" podUID="308bc622-0581-470e-a08a-eff7e13a57e8" pod="opendatahub/test-curl-7516-50e653" Sep 10 13:52:26.257920 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:52:26.257821 2585 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7516-47709d"] Sep 10 13:52:26.263478 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:52:26.263440 2585 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7516-47709d"] Sep 10 13:52:26.387993 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:52:26.387949 2585 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7516-957c22"] Sep 10 13:52:26.388551 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:52:26.388524 2585 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5b2541cf-bd15-4153-b01f-99d6ec13f477" containerName="test-curl-7516-47709d" Sep 10 13:52:26.388551 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:52:26.388548 2585 state_mem.go:107] "Deleted CPUSet assignment" podUID="5b2541cf-bd15-4153-b01f-99d6ec13f477" containerName="test-curl-7516-47709d" Sep 10 13:52:26.388726 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:52:26.388658 2585 memory_manager.go:368] "RemoveStaleState removing state" podUID="5b2541cf-bd15-4153-b01f-99d6ec13f477" containerName="test-curl-7516-47709d" Sep 10 13:52:26.391952 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:52:26.391934 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7516-957c22" Sep 10 13:52:26.399119 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:52:26.399092 2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7516-957c22"] Sep 10 13:52:26.423491 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:52:26.423458 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xmfb9\" (UniqueName: \"kubernetes.io/projected/34f491eb-1e0f-48ef-acbd-40feb1cf1f7b-kube-api-access-xmfb9\") pod \"test-curl-7516-957c22\" (UID: \"34f491eb-1e0f-48ef-acbd-40feb1cf1f7b\") " pod="opendatahub/test-curl-7516-957c22" Sep 10 13:52:26.505952 ip-10-0-140-8 kubenswrapper[2585]: E0910 13:52:26.505917 2585 kuberuntime_manager.go:649] "Missing actuated resource record" pod="opendatahub/test-curl-7516-47709d" container="test-curl-7516-47709d" Sep 10 13:52:26.524118 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:52:26.524042 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xmfb9\" (UniqueName: \"kubernetes.io/projected/34f491eb-1e0f-48ef-acbd-40feb1cf1f7b-kube-api-access-xmfb9\") pod \"test-curl-7516-957c22\" (UID: \"34f491eb-1e0f-48ef-acbd-40feb1cf1f7b\") " pod="opendatahub/test-curl-7516-957c22" Sep 10 13:52:26.534915 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:52:26.534885 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xmfb9\" (UniqueName: \"kubernetes.io/projected/34f491eb-1e0f-48ef-acbd-40feb1cf1f7b-kube-api-access-xmfb9\") pod \"test-curl-7516-957c22\" (UID: \"34f491eb-1e0f-48ef-acbd-40feb1cf1f7b\") " pod="opendatahub/test-curl-7516-957c22" Sep 10 13:52:26.702020 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:52:26.701986 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7516-957c22" Sep 10 13:52:26.985487 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:52:26.985465 2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7516-957c22"] Sep 10 13:52:26.987157 ip-10-0-140-8 kubenswrapper[2585]: W0910 13:52:26.987129 2585 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod34f491eb_1e0f_48ef_acbd_40feb1cf1f7b.slice/crio-8de54a60d995014ab64ba9cf894ce514dc6b5ab3973bf7ba46fca38f0d14b90e WatchSource:0}: Error finding container 8de54a60d995014ab64ba9cf894ce514dc6b5ab3973bf7ba46fca38f0d14b90e: Status 404 returned error can't find the container with id 8de54a60d995014ab64ba9cf894ce514dc6b5ab3973bf7ba46fca38f0d14b90e Sep 10 13:52:27.513218 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:52:27.513190 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7516-957c22" event={"ID":"34f491eb-1e0f-48ef-acbd-40feb1cf1f7b","Type":"ContainerStarted","Data":"cea61a31c0dbbc2948f4107288abb3f849e1ab2aecdb737d54d854cf7644e748"} Sep 10 13:52:27.513727 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:52:27.513235 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7516-957c22" event={"ID":"34f491eb-1e0f-48ef-acbd-40feb1cf1f7b","Type":"ContainerStarted","Data":"8de54a60d995014ab64ba9cf894ce514dc6b5ab3973bf7ba46fca38f0d14b90e"} Sep 10 13:52:27.513727 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:52:27.513488 2585 kuberuntime_container.go:873] "Killing container with a grace period" pod="opendatahub/test-curl-7516-47709d" podUID="5b2541cf-bd15-4153-b01f-99d6ec13f477" containerName="test-curl-7516-47709d" containerID="cri-o://1b493a6057292e2aaf80e0feb8d1d27fda2d913a65e4201f90d2c4020605ae62" gracePeriod=2 Sep 10 13:52:27.530880 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:52:27.530826 2585 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/test-curl-7516-957c22" podStartSLOduration=1.530809533 podStartE2EDuration="1.530809533s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 13:52:26 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 13:52:27.529062502 +0000 UTC m=+1432.401766760" watchObservedRunningTime="2026-09-10 13:52:27.530809533 +0000 UTC m=+1432.403513734" Sep 10 13:52:27.531134 ip-10-0-140-8 kubenswrapper[2585]: E0910 13:52:27.531102 2585 status_manager.go:1018] "Failed to get status for pod" err="pods \"test-curl-7516-47709d\" is forbidden: User \"system:node:ip-10-0-140-8.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-140-8.ec2.internal' and this object" podUID="5b2541cf-bd15-4153-b01f-99d6ec13f477" pod="opendatahub/test-curl-7516-47709d" Sep 10 13:52:27.690865 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:52:27.690839 2585 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7516-50e653" Sep 10 13:52:27.733460 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:52:27.733431 2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-47z5z\" (UniqueName: \"kubernetes.io/projected/308bc622-0581-470e-a08a-eff7e13a57e8-kube-api-access-47z5z\") pod \"308bc622-0581-470e-a08a-eff7e13a57e8\" (UID: \"308bc622-0581-470e-a08a-eff7e13a57e8\") " Sep 10 13:52:27.735623 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:52:27.735591 2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/308bc622-0581-470e-a08a-eff7e13a57e8-kube-api-access-47z5z" (OuterVolumeSpecName: "kube-api-access-47z5z") pod "308bc622-0581-470e-a08a-eff7e13a57e8" (UID: "308bc622-0581-470e-a08a-eff7e13a57e8"). InnerVolumeSpecName "kube-api-access-47z5z". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 10 13:52:27.835141 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:52:27.835071 2585 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-47z5z\" (UniqueName: \"kubernetes.io/projected/308bc622-0581-470e-a08a-eff7e13a57e8-kube-api-access-47z5z\") on node \"ip-10-0-140-8.ec2.internal\" DevicePath \"\"" Sep 10 13:52:28.159916 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:52:28.159828 2585 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7516-957c22"] Sep 10 13:52:28.173830 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:52:28.173798 2585 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7516-957c22"] Sep 10 13:52:28.293477 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:52:28.293441 2585 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7516-a45dff"] Sep 10 13:52:28.293868 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:52:28.293855 2585 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="34f491eb-1e0f-48ef-acbd-40feb1cf1f7b" containerName="test-curl-7516-957c22" Sep 10 13:52:28.293914 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:52:28.293870 2585 state_mem.go:107] "Deleted CPUSet assignment" podUID="34f491eb-1e0f-48ef-acbd-40feb1cf1f7b" containerName="test-curl-7516-957c22" Sep 10 13:52:28.293956 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:52:28.293946 2585 memory_manager.go:368] "RemoveStaleState removing state" podUID="34f491eb-1e0f-48ef-acbd-40feb1cf1f7b" containerName="test-curl-7516-957c22" Sep 10 13:52:28.297019 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:52:28.297001 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7516-a45dff" Sep 10 13:52:28.309264 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:52:28.305137 2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7516-a45dff"] Sep 10 13:52:28.339148 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:52:28.339116 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dkvkl\" (UniqueName: \"kubernetes.io/projected/22d58af2-fa22-4af0-9239-a8ae99e3d5ca-kube-api-access-dkvkl\") pod \"test-curl-7516-a45dff\" (UID: \"22d58af2-fa22-4af0-9239-a8ae99e3d5ca\") " pod="opendatahub/test-curl-7516-a45dff" Sep 10 13:52:28.440009 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:52:28.439916 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dkvkl\" (UniqueName: \"kubernetes.io/projected/22d58af2-fa22-4af0-9239-a8ae99e3d5ca-kube-api-access-dkvkl\") pod \"test-curl-7516-a45dff\" (UID: \"22d58af2-fa22-4af0-9239-a8ae99e3d5ca\") " pod="opendatahub/test-curl-7516-a45dff" Sep 10 13:52:28.449594 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:52:28.449571 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-dkvkl\" (UniqueName: \"kubernetes.io/projected/22d58af2-fa22-4af0-9239-a8ae99e3d5ca-kube-api-access-dkvkl\") pod \"test-curl-7516-a45dff\" (UID: \"22d58af2-fa22-4af0-9239-a8ae99e3d5ca\") " pod="opendatahub/test-curl-7516-a45dff" Sep 10 13:52:28.519783 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:52:28.519744 2585 generic.go:356] "Generic (PLEG): container finished" podID="308bc622-0581-470e-a08a-eff7e13a57e8" containerID="b558df3b5c523d629a9ccde6912603e0148258add3dfa86c05c6b0a09888fd47" exitCode=137 Sep 10 13:52:28.520173 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:52:28.519809 2585 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7516-50e653" Sep 10 13:52:28.520173 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:52:28.519841 2585 scope.go:117] "RemoveContainer" containerID="b558df3b5c523d629a9ccde6912603e0148258add3dfa86c05c6b0a09888fd47" Sep 10 13:52:28.520173 ip-10-0-140-8 kubenswrapper[2585]: E0910 13:52:28.520106 2585 kuberuntime_manager.go:649] "Missing actuated resource record" pod="opendatahub/test-curl-7516-957c22" container="test-curl-7516-957c22" Sep 10 13:52:28.529791 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:52:28.529774 2585 scope.go:117] "RemoveContainer" containerID="b558df3b5c523d629a9ccde6912603e0148258add3dfa86c05c6b0a09888fd47" Sep 10 13:52:28.530058 ip-10-0-140-8 kubenswrapper[2585]: E0910 13:52:28.530037 2585 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b558df3b5c523d629a9ccde6912603e0148258add3dfa86c05c6b0a09888fd47\": container with ID starting with b558df3b5c523d629a9ccde6912603e0148258add3dfa86c05c6b0a09888fd47 not found: ID does not exist" containerID="b558df3b5c523d629a9ccde6912603e0148258add3dfa86c05c6b0a09888fd47" Sep 10 13:52:28.530128 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:52:28.530065 2585 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b558df3b5c523d629a9ccde6912603e0148258add3dfa86c05c6b0a09888fd47"} err="failed to get container status \"b558df3b5c523d629a9ccde6912603e0148258add3dfa86c05c6b0a09888fd47\": rpc error: code = NotFound desc = could not find container \"b558df3b5c523d629a9ccde6912603e0148258add3dfa86c05c6b0a09888fd47\": container with ID starting with b558df3b5c523d629a9ccde6912603e0148258add3dfa86c05c6b0a09888fd47 not found: ID does not exist" Sep 10 13:52:28.532257 ip-10-0-140-8 kubenswrapper[2585]: E0910 13:52:28.532220 2585 status_manager.go:1018] "Failed to get status for pod" err="pods \"test-curl-7516-50e653\" is forbidden: User \"system:node:ip-10-0-140-8.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-140-8.ec2.internal' and this object" podUID="308bc622-0581-470e-a08a-eff7e13a57e8" pod="opendatahub/test-curl-7516-50e653" Sep 10 13:52:28.611841 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:52:28.611804 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7516-a45dff" Sep 10 13:52:28.888349 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:52:28.888320 2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7516-a45dff"] Sep 10 13:52:28.889483 ip-10-0-140-8 kubenswrapper[2585]: W0910 13:52:28.889448 2585 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod22d58af2_fa22_4af0_9239_a8ae99e3d5ca.slice/crio-3ed36a5341245d9843bee791e1d71381d78e857cf13ea59e610573f631e1c8f0 WatchSource:0}: Error finding container 3ed36a5341245d9843bee791e1d71381d78e857cf13ea59e610573f631e1c8f0: Status 404 returned error can't find the container with id 3ed36a5341245d9843bee791e1d71381d78e857cf13ea59e610573f631e1c8f0 Sep 10 13:52:29.532738 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:52:29.532708 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7516-a45dff" event={"ID":"22d58af2-fa22-4af0-9239-a8ae99e3d5ca","Type":"ContainerStarted","Data":"3f83aeaf206377675b209ac289f5a60c4fab4f00bc3a5688dbce6d9633a182c9"} Sep 10 13:52:29.533024 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:52:29.532753 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7516-a45dff" event={"ID":"22d58af2-fa22-4af0-9239-a8ae99e3d5ca","Type":"ContainerStarted","Data":"3ed36a5341245d9843bee791e1d71381d78e857cf13ea59e610573f631e1c8f0"} Sep 10 13:52:29.533024 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:52:29.533001 2585 kuberuntime_container.go:873] "Killing container with a grace period" pod="opendatahub/test-curl-7516-957c22" podUID="34f491eb-1e0f-48ef-acbd-40feb1cf1f7b" containerName="test-curl-7516-957c22" containerID="cri-o://cea61a31c0dbbc2948f4107288abb3f849e1ab2aecdb737d54d854cf7644e748" gracePeriod=2 Sep 10 13:52:29.536641 ip-10-0-140-8 kubenswrapper[2585]: E0910 13:52:29.536613 2585 status_manager.go:1018] "Failed to get status for pod" err="pods \"test-curl-7516-50e653\" is forbidden: User \"system:node:ip-10-0-140-8.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-140-8.ec2.internal' and this object" podUID="308bc622-0581-470e-a08a-eff7e13a57e8" pod="opendatahub/test-curl-7516-50e653" Sep 10 13:52:29.562261 ip-10-0-140-8 kubenswrapper[2585]: E0910 13:52:29.562213 2585 status_manager.go:1018] "Failed to get status for pod" err="pods \"test-curl-7516-957c22\" is forbidden: User \"system:node:ip-10-0-140-8.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-140-8.ec2.internal' and this object" podUID="34f491eb-1e0f-48ef-acbd-40feb1cf1f7b" pod="opendatahub/test-curl-7516-957c22" Sep 10 13:52:29.563545 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:52:29.563496 2585 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/test-curl-7516-a45dff" podStartSLOduration=1.5634779669999999 podStartE2EDuration="1.563477967s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 13:52:28 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 13:52:29.560105093 +0000 UTC m=+1434.432809295" watchObservedRunningTime="2026-09-10 13:52:29.563477967 +0000 UTC m=+1434.436182171" Sep 10 13:52:29.564060 ip-10-0-140-8 kubenswrapper[2585]: E0910 13:52:29.564035 2585 status_manager.go:1018] "Failed to get status for pod" err="pods \"test-curl-7516-50e653\" is forbidden: User \"system:node:ip-10-0-140-8.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-140-8.ec2.internal' and this object" podUID="308bc622-0581-470e-a08a-eff7e13a57e8" pod="opendatahub/test-curl-7516-50e653" Sep 10 13:52:29.649947 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:52:29.649913 2585 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="308bc622-0581-470e-a08a-eff7e13a57e8" path="/var/lib/kubelet/pods/308bc622-0581-470e-a08a-eff7e13a57e8/volumes" Sep 10 13:52:29.698413 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:52:29.698380 2585 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7516-47709d" Sep 10 13:52:29.751165 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:52:29.751139 2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-5bc2p\" (UniqueName: \"kubernetes.io/projected/5b2541cf-bd15-4153-b01f-99d6ec13f477-kube-api-access-5bc2p\") pod \"5b2541cf-bd15-4153-b01f-99d6ec13f477\" (UID: \"5b2541cf-bd15-4153-b01f-99d6ec13f477\") " Sep 10 13:52:29.753159 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:52:29.753133 2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5b2541cf-bd15-4153-b01f-99d6ec13f477-kube-api-access-5bc2p" (OuterVolumeSpecName: "kube-api-access-5bc2p") pod "5b2541cf-bd15-4153-b01f-99d6ec13f477" (UID: "5b2541cf-bd15-4153-b01f-99d6ec13f477"). InnerVolumeSpecName "kube-api-access-5bc2p". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 10 13:52:29.852556 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:52:29.852523 2585 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5bc2p\" (UniqueName: \"kubernetes.io/projected/5b2541cf-bd15-4153-b01f-99d6ec13f477-kube-api-access-5bc2p\") on node \"ip-10-0-140-8.ec2.internal\" DevicePath \"\"" Sep 10 13:52:30.214385 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:52:30.214294 2585 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7516-a45dff"] Sep 10 13:52:30.221779 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:52:30.221753 2585 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7516-a45dff"] Sep 10 13:52:30.539672 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:52:30.539582 2585 generic.go:356] "Generic (PLEG): container finished" podID="5b2541cf-bd15-4153-b01f-99d6ec13f477" containerID="1b493a6057292e2aaf80e0feb8d1d27fda2d913a65e4201f90d2c4020605ae62" exitCode=137 Sep 10 13:52:30.539672 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:52:30.539627 2585 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7516-47709d" Sep 10 13:52:30.540125 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:52:30.539702 2585 scope.go:117] "RemoveContainer" containerID="1b493a6057292e2aaf80e0feb8d1d27fda2d913a65e4201f90d2c4020605ae62" Sep 10 13:52:30.540125 ip-10-0-140-8 kubenswrapper[2585]: E0910 13:52:30.539892 2585 kuberuntime_manager.go:649] "Missing actuated resource record" pod="opendatahub/test-curl-7516-a45dff" container="test-curl-7516-a45dff" Sep 10 13:52:30.549470 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:52:30.549452 2585 scope.go:117] "RemoveContainer" containerID="1b493a6057292e2aaf80e0feb8d1d27fda2d913a65e4201f90d2c4020605ae62" Sep 10 13:52:30.549747 ip-10-0-140-8 kubenswrapper[2585]: E0910 13:52:30.549729 2585 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1b493a6057292e2aaf80e0feb8d1d27fda2d913a65e4201f90d2c4020605ae62\": container with ID starting with 1b493a6057292e2aaf80e0feb8d1d27fda2d913a65e4201f90d2c4020605ae62 not found: ID does not exist" containerID="1b493a6057292e2aaf80e0feb8d1d27fda2d913a65e4201f90d2c4020605ae62" Sep 10 13:52:30.549805 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:52:30.549756 2585 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1b493a6057292e2aaf80e0feb8d1d27fda2d913a65e4201f90d2c4020605ae62"} err="failed to get container status \"1b493a6057292e2aaf80e0feb8d1d27fda2d913a65e4201f90d2c4020605ae62\": rpc error: code = NotFound desc = could not find container \"1b493a6057292e2aaf80e0feb8d1d27fda2d913a65e4201f90d2c4020605ae62\": container with ID starting with 1b493a6057292e2aaf80e0feb8d1d27fda2d913a65e4201f90d2c4020605ae62 not found: ID does not exist" Sep 10 13:52:30.550541 ip-10-0-140-8 kubenswrapper[2585]: E0910 13:52:30.550516 2585 status_manager.go:1018] "Failed to get status for pod" err="pods \"test-curl-7516-47709d\" is forbidden: User \"system:node:ip-10-0-140-8.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-140-8.ec2.internal' and this object" podUID="5b2541cf-bd15-4153-b01f-99d6ec13f477" pod="opendatahub/test-curl-7516-47709d" Sep 10 13:52:31.057061 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:52:31.057025 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fxgmp_fa53c783-d40a-4024-9f7b-1d0d2b57f6dc/konnectivity-agent/0.log" limit=1024 Sep 10 13:52:31.200273 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:52:31.200223 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fxgmp_fa53c783-d40a-4024-9f7b-1d0d2b57f6dc/konnectivity-agent/0.log" limit=1024 Sep 10 13:52:31.424204 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:52:31.424170 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fxgmp_fa53c783-d40a-4024-9f7b-1d0d2b57f6dc/konnectivity-agent/0.log" limit=1024 Sep 10 13:52:31.547464 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:52:31.547433 2585 kuberuntime_container.go:873] "Killing container with a grace period" pod="opendatahub/test-curl-7516-a45dff" podUID="22d58af2-fa22-4af0-9239-a8ae99e3d5ca" containerName="test-curl-7516-a45dff" containerID="cri-o://3f83aeaf206377675b209ac289f5a60c4fab4f00bc3a5688dbce6d9633a182c9" gracePeriod=2 Sep 10 13:52:31.551704 ip-10-0-140-8 kubenswrapper[2585]: E0910 13:52:31.551676 2585 status_manager.go:1018] "Failed to get status for pod" err="pods \"test-curl-7516-47709d\" is forbidden: User \"system:node:ip-10-0-140-8.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-140-8.ec2.internal' and this object" podUID="5b2541cf-bd15-4153-b01f-99d6ec13f477" pod="opendatahub/test-curl-7516-47709d" Sep 10 13:52:31.555479 ip-10-0-140-8 kubenswrapper[2585]: E0910 13:52:31.555448 2585 status_manager.go:1018] "Failed to get status for pod" err="pods \"test-curl-7516-a45dff\" is forbidden: User \"system:node:ip-10-0-140-8.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-140-8.ec2.internal' and this object" podUID="22d58af2-fa22-4af0-9239-a8ae99e3d5ca" pod="opendatahub/test-curl-7516-a45dff" Sep 10 13:52:31.648386 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:52:31.648352 2585 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="5b2541cf-bd15-4153-b01f-99d6ec13f477" path="/var/lib/kubelet/pods/5b2541cf-bd15-4153-b01f-99d6ec13f477/volumes" Sep 10 13:52:31.683319 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:52:31.683263 2585 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7516-957c22" Sep 10 13:52:31.769897 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:52:31.769865 2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-xmfb9\" (UniqueName: \"kubernetes.io/projected/34f491eb-1e0f-48ef-acbd-40feb1cf1f7b-kube-api-access-xmfb9\") pod \"34f491eb-1e0f-48ef-acbd-40feb1cf1f7b\" (UID: \"34f491eb-1e0f-48ef-acbd-40feb1cf1f7b\") " Sep 10 13:52:31.772004 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:52:31.771970 2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/34f491eb-1e0f-48ef-acbd-40feb1cf1f7b-kube-api-access-xmfb9" (OuterVolumeSpecName: "kube-api-access-xmfb9") pod "34f491eb-1e0f-48ef-acbd-40feb1cf1f7b" (UID: "34f491eb-1e0f-48ef-acbd-40feb1cf1f7b"). InnerVolumeSpecName "kube-api-access-xmfb9". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 10 13:52:31.871552 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:52:31.871519 2585 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xmfb9\" (UniqueName: \"kubernetes.io/projected/34f491eb-1e0f-48ef-acbd-40feb1cf1f7b-kube-api-access-xmfb9\") on node \"ip-10-0-140-8.ec2.internal\" DevicePath \"\"" Sep 10 13:52:32.554112 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:52:32.554076 2585 generic.go:356] "Generic (PLEG): container finished" podID="34f491eb-1e0f-48ef-acbd-40feb1cf1f7b" containerID="cea61a31c0dbbc2948f4107288abb3f849e1ab2aecdb737d54d854cf7644e748" exitCode=137 Sep 10 13:52:32.554607 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:52:32.554125 2585 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7516-957c22" Sep 10 13:52:32.554607 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:52:32.554180 2585 scope.go:117] "RemoveContainer" containerID="cea61a31c0dbbc2948f4107288abb3f849e1ab2aecdb737d54d854cf7644e748" Sep 10 13:52:32.563118 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:52:32.563102 2585 scope.go:117] "RemoveContainer" containerID="cea61a31c0dbbc2948f4107288abb3f849e1ab2aecdb737d54d854cf7644e748" Sep 10 13:52:32.563484 ip-10-0-140-8 kubenswrapper[2585]: E0910 13:52:32.563464 2585 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"cea61a31c0dbbc2948f4107288abb3f849e1ab2aecdb737d54d854cf7644e748\": container with ID starting with cea61a31c0dbbc2948f4107288abb3f849e1ab2aecdb737d54d854cf7644e748 not found: ID does not exist" containerID="cea61a31c0dbbc2948f4107288abb3f849e1ab2aecdb737d54d854cf7644e748" Sep 10 13:52:32.563543 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:52:32.563492 2585 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cea61a31c0dbbc2948f4107288abb3f849e1ab2aecdb737d54d854cf7644e748"} err="failed to get container status \"cea61a31c0dbbc2948f4107288abb3f849e1ab2aecdb737d54d854cf7644e748\": rpc error: code = NotFound desc = could not find container \"cea61a31c0dbbc2948f4107288abb3f849e1ab2aecdb737d54d854cf7644e748\": container with ID starting with cea61a31c0dbbc2948f4107288abb3f849e1ab2aecdb737d54d854cf7644e748 not found: ID does not exist" Sep 10 13:52:32.564471 ip-10-0-140-8 kubenswrapper[2585]: E0910 13:52:32.564455 2585 status_manager.go:1018] "Failed to get status for pod" err="pods \"test-curl-7516-957c22\" is forbidden: User \"system:node:ip-10-0-140-8.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-140-8.ec2.internal' and this object" podUID="34f491eb-1e0f-48ef-acbd-40feb1cf1f7b" pod="opendatahub/test-curl-7516-957c22" Sep 10 13:52:33.649643 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:52:33.649608 2585 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="34f491eb-1e0f-48ef-acbd-40feb1cf1f7b" path="/var/lib/kubelet/pods/34f491eb-1e0f-48ef-acbd-40feb1cf1f7b/volumes" Sep 10 13:52:33.727925 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:52:33.727902 2585 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7516-a45dff" Sep 10 13:52:33.790082 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:52:33.790047 2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-dkvkl\" (UniqueName: \"kubernetes.io/projected/22d58af2-fa22-4af0-9239-a8ae99e3d5ca-kube-api-access-dkvkl\") pod \"22d58af2-fa22-4af0-9239-a8ae99e3d5ca\" (UID: \"22d58af2-fa22-4af0-9239-a8ae99e3d5ca\") " Sep 10 13:52:33.792170 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:52:33.792141 2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/22d58af2-fa22-4af0-9239-a8ae99e3d5ca-kube-api-access-dkvkl" (OuterVolumeSpecName: "kube-api-access-dkvkl") pod "22d58af2-fa22-4af0-9239-a8ae99e3d5ca" (UID: "22d58af2-fa22-4af0-9239-a8ae99e3d5ca"). InnerVolumeSpecName "kube-api-access-dkvkl". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 10 13:52:33.891107 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:52:33.891075 2585 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dkvkl\" (UniqueName: \"kubernetes.io/projected/22d58af2-fa22-4af0-9239-a8ae99e3d5ca-kube-api-access-dkvkl\") on node \"ip-10-0-140-8.ec2.internal\" DevicePath \"\"" Sep 10 13:52:34.328358 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:52:34.328322 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fxgmp_fa53c783-d40a-4024-9f7b-1d0d2b57f6dc/konnectivity-agent/0.log" limit=1024 Sep 10 13:52:34.433130 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:52:34.433096 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fxgmp_fa53c783-d40a-4024-9f7b-1d0d2b57f6dc/konnectivity-agent/0.log" limit=1024 Sep 10 13:52:34.568296 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:52:34.568260 2585 generic.go:356] "Generic (PLEG): container finished" podID="22d58af2-fa22-4af0-9239-a8ae99e3d5ca" containerID="3f83aeaf206377675b209ac289f5a60c4fab4f00bc3a5688dbce6d9633a182c9" exitCode=137 Sep 10 13:52:34.568296 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:52:34.568292 2585 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7516-a45dff" Sep 10 13:52:34.568531 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:52:34.568354 2585 scope.go:117] "RemoveContainer" containerID="3f83aeaf206377675b209ac289f5a60c4fab4f00bc3a5688dbce6d9633a182c9" Sep 10 13:52:34.583116 ip-10-0-140-8 kubenswrapper[2585]: E0910 13:52:34.583080 2585 status_manager.go:1018] "Failed to get status for pod" err="pods \"test-curl-7516-a45dff\" is forbidden: User \"system:node:ip-10-0-140-8.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-140-8.ec2.internal' and this object" podUID="22d58af2-fa22-4af0-9239-a8ae99e3d5ca" pod="opendatahub/test-curl-7516-a45dff" Sep 10 13:52:34.585132 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:52:34.585114 2585 scope.go:117] "RemoveContainer" containerID="3f83aeaf206377675b209ac289f5a60c4fab4f00bc3a5688dbce6d9633a182c9" Sep 10 13:52:34.585464 ip-10-0-140-8 kubenswrapper[2585]: E0910 13:52:34.585441 2585 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3f83aeaf206377675b209ac289f5a60c4fab4f00bc3a5688dbce6d9633a182c9\": container with ID starting with 3f83aeaf206377675b209ac289f5a60c4fab4f00bc3a5688dbce6d9633a182c9 not found: ID does not exist" containerID="3f83aeaf206377675b209ac289f5a60c4fab4f00bc3a5688dbce6d9633a182c9" Sep 10 13:52:34.585523 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:52:34.585471 2585 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3f83aeaf206377675b209ac289f5a60c4fab4f00bc3a5688dbce6d9633a182c9"} err="failed to get container status \"3f83aeaf206377675b209ac289f5a60c4fab4f00bc3a5688dbce6d9633a182c9\": rpc error: code = NotFound desc = could not find container \"3f83aeaf206377675b209ac289f5a60c4fab4f00bc3a5688dbce6d9633a182c9\": container with ID starting with 3f83aeaf206377675b209ac289f5a60c4fab4f00bc3a5688dbce6d9633a182c9 not found: ID does not exist" Sep 10 13:52:35.353703 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:52:35.353668 2585 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-713ee90e-59584c8f96-55vtb"] Sep 10 13:52:35.354402 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:52:35.354383 2585 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="22d58af2-fa22-4af0-9239-a8ae99e3d5ca" containerName="test-curl-7516-a45dff" Sep 10 13:52:35.354450 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:52:35.354405 2585 state_mem.go:107] "Deleted CPUSet assignment" podUID="22d58af2-fa22-4af0-9239-a8ae99e3d5ca" containerName="test-curl-7516-a45dff" Sep 10 13:52:35.354557 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:52:35.354533 2585 memory_manager.go:368] "RemoveStaleState removing state" podUID="22d58af2-fa22-4af0-9239-a8ae99e3d5ca" containerName="test-curl-7516-a45dff" Sep 10 13:52:35.359371 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:52:35.359348 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-713ee90e-59584c8f96-55vtb" Sep 10 13:52:35.361733 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:52:35.361703 2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Sep 10 13:52:35.363419 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:52:35.363395 2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-q8qnd\"" Sep 10 13:52:35.363748 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:52:35.363425 2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Sep 10 13:52:35.363748 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:52:35.363511 2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-mt-713ee90e-serving-cert\"" Sep 10 13:52:35.363748 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:52:35.363517 2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-metrics-tls\"" Sep 10 13:52:35.366907 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:52:35.366877 2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-713ee90e-59584c8f96-55vtb"] Sep 10 13:52:35.388016 ip-10-0-140-8 kubenswrapper[2585]: E0910 13:52:35.387970 2585 status_manager.go:1018] "Failed to get status for pod" err="pods \"test-curl-7516-a45dff\" is forbidden: User \"system:node:ip-10-0-140-8.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-140-8.ec2.internal' and this object" podUID="22d58af2-fa22-4af0-9239-a8ae99e3d5ca" pod="opendatahub/test-curl-7516-a45dff" Sep 10 13:52:35.403756 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:52:35.403729 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-t2dq9\" (UniqueName: \"kubernetes.io/projected/85733206-46ff-4ed1-824e-fe0beec5c828-kube-api-access-t2dq9\") pod \"maas-api-e2e-mt-713ee90e-59584c8f96-55vtb\" (UID: \"85733206-46ff-4ed1-824e-fe0beec5c828\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-713ee90e-59584c8f96-55vtb" Sep 10 13:52:35.403890 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:52:35.403789 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/85733206-46ff-4ed1-824e-fe0beec5c828-openshift-service-ca\") pod \"maas-api-e2e-mt-713ee90e-59584c8f96-55vtb\" (UID: \"85733206-46ff-4ed1-824e-fe0beec5c828\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-713ee90e-59584c8f96-55vtb" Sep 10 13:52:35.403890 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:52:35.403835 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/85733206-46ff-4ed1-824e-fe0beec5c828-maas-api-tls\") pod \"maas-api-e2e-mt-713ee90e-59584c8f96-55vtb\" (UID: \"85733206-46ff-4ed1-824e-fe0beec5c828\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-713ee90e-59584c8f96-55vtb" Sep 10 13:52:35.403890 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:52:35.403859 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/85733206-46ff-4ed1-824e-fe0beec5c828-metrics-certs\") pod \"maas-api-e2e-mt-713ee90e-59584c8f96-55vtb\" (UID: \"85733206-46ff-4ed1-824e-fe0beec5c828\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-713ee90e-59584c8f96-55vtb" Sep 10 13:52:35.420674 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:52:35.420643 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fxgmp_fa53c783-d40a-4024-9f7b-1d0d2b57f6dc/konnectivity-agent/0.log" limit=1024 Sep 10 13:52:35.505456 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:52:35.505423 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t2dq9\" (UniqueName: \"kubernetes.io/projected/85733206-46ff-4ed1-824e-fe0beec5c828-kube-api-access-t2dq9\") pod \"maas-api-e2e-mt-713ee90e-59584c8f96-55vtb\" (UID: \"85733206-46ff-4ed1-824e-fe0beec5c828\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-713ee90e-59584c8f96-55vtb" Sep 10 13:52:35.505745 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:52:35.505503 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/85733206-46ff-4ed1-824e-fe0beec5c828-openshift-service-ca\") pod \"maas-api-e2e-mt-713ee90e-59584c8f96-55vtb\" (UID: \"85733206-46ff-4ed1-824e-fe0beec5c828\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-713ee90e-59584c8f96-55vtb" Sep 10 13:52:35.505745 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:52:35.505528 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/85733206-46ff-4ed1-824e-fe0beec5c828-maas-api-tls\") pod \"maas-api-e2e-mt-713ee90e-59584c8f96-55vtb\" (UID: \"85733206-46ff-4ed1-824e-fe0beec5c828\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-713ee90e-59584c8f96-55vtb" Sep 10 13:52:35.505745 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:52:35.505557 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/85733206-46ff-4ed1-824e-fe0beec5c828-metrics-certs\") pod \"maas-api-e2e-mt-713ee90e-59584c8f96-55vtb\" (UID: \"85733206-46ff-4ed1-824e-fe0beec5c828\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-713ee90e-59584c8f96-55vtb" Sep 10 13:52:35.506307 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:52:35.506232 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/85733206-46ff-4ed1-824e-fe0beec5c828-openshift-service-ca\") pod \"maas-api-e2e-mt-713ee90e-59584c8f96-55vtb\" (UID: \"85733206-46ff-4ed1-824e-fe0beec5c828\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-713ee90e-59584c8f96-55vtb" Sep 10 13:52:35.508020 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:52:35.508003 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/85733206-46ff-4ed1-824e-fe0beec5c828-metrics-certs\") pod \"maas-api-e2e-mt-713ee90e-59584c8f96-55vtb\" (UID: \"85733206-46ff-4ed1-824e-fe0beec5c828\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-713ee90e-59584c8f96-55vtb" Sep 10 13:52:35.508095 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:52:35.508050 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/85733206-46ff-4ed1-824e-fe0beec5c828-maas-api-tls\") pod \"maas-api-e2e-mt-713ee90e-59584c8f96-55vtb\" (UID: \"85733206-46ff-4ed1-824e-fe0beec5c828\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-713ee90e-59584c8f96-55vtb" Sep 10 13:52:35.513815 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:52:35.513796 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-t2dq9\" (UniqueName: \"kubernetes.io/projected/85733206-46ff-4ed1-824e-fe0beec5c828-kube-api-access-t2dq9\") pod \"maas-api-e2e-mt-713ee90e-59584c8f96-55vtb\" (UID: \"85733206-46ff-4ed1-824e-fe0beec5c828\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-713ee90e-59584c8f96-55vtb" Sep 10 13:52:35.632597 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:52:35.632513 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fxgmp_fa53c783-d40a-4024-9f7b-1d0d2b57f6dc/konnectivity-agent/0.log" limit=1024 Sep 10 13:52:35.647885 ip-10-0-140-8 kubenswrapper[2585]: E0910 13:52:35.647847 2585 status_manager.go:1018] "Failed to get status for pod" err="pods \"test-curl-7516-a45dff\" is forbidden: User \"system:node:ip-10-0-140-8.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-140-8.ec2.internal' and this object" podUID="22d58af2-fa22-4af0-9239-a8ae99e3d5ca" pod="opendatahub/test-curl-7516-a45dff" Sep 10 13:52:35.649528 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:52:35.649506 2585 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="22d58af2-fa22-4af0-9239-a8ae99e3d5ca" path="/var/lib/kubelet/pods/22d58af2-fa22-4af0-9239-a8ae99e3d5ca/volumes" Sep 10 13:52:35.674630 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:52:35.674606 2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-q8qnd\"" Sep 10 13:52:35.682754 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:52:35.682727 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-713ee90e-59584c8f96-55vtb" Sep 10 13:52:35.994032 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:52:35.994002 2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-713ee90e-59584c8f96-55vtb"] Sep 10 13:52:35.995954 ip-10-0-140-8 kubenswrapper[2585]: W0910 13:52:35.995919 2585 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod85733206_46ff_4ed1_824e_fe0beec5c828.slice/crio-33a16d013797d2c21e3ba6ce77a845072972a123d71d99d49e546ad08905ffb7 WatchSource:0}: Error finding container 33a16d013797d2c21e3ba6ce77a845072972a123d71d99d49e546ad08905ffb7: Status 404 returned error can't find the container with id 33a16d013797d2c21e3ba6ce77a845072972a123d71d99d49e546ad08905ffb7 Sep 10 13:52:35.997623 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:52:35.997586 2585 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Sep 10 13:52:36.186981 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:52:36.186938 2585 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-713ee90e-86c97fff6b-xmc2n"] Sep 10 13:52:36.190563 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:52:36.190538 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-713ee90e-86c97fff6b-xmc2n" Sep 10 13:52:36.194056 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:52:36.194034 2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-mt-713ee90e\"" Sep 10 13:52:36.194169 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:52:36.194077 2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-mt-713ee90e-dockercfg-xj82n\"" Sep 10 13:52:36.209234 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:52:36.209208 2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-713ee90e-86c97fff6b-xmc2n"] Sep 10 13:52:36.211395 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:52:36.211374 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0163bdf8-1e1c-4c6d-8072-b6416c913fe5-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-713ee90e-86c97fff6b-xmc2n\" (UID: \"0163bdf8-1e1c-4c6d-8072-b6416c913fe5\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-713ee90e-86c97fff6b-xmc2n" Sep 10 13:52:36.211481 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:52:36.211446 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-b6k9m\" (UniqueName: \"kubernetes.io/projected/0163bdf8-1e1c-4c6d-8072-b6416c913fe5-kube-api-access-b6k9m\") pod \"payload-pre-processing-e2e-mt-713ee90e-86c97fff6b-xmc2n\" (UID: \"0163bdf8-1e1c-4c6d-8072-b6416c913fe5\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-713ee90e-86c97fff6b-xmc2n" Sep 10 13:52:36.233300 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:52:36.233274 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fxgmp_fa53c783-d40a-4024-9f7b-1d0d2b57f6dc/konnectivity-agent/0.log" limit=1024 Sep 10 13:52:36.312754 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:52:36.312722 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0163bdf8-1e1c-4c6d-8072-b6416c913fe5-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-713ee90e-86c97fff6b-xmc2n\" (UID: \"0163bdf8-1e1c-4c6d-8072-b6416c913fe5\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-713ee90e-86c97fff6b-xmc2n" Sep 10 13:52:36.312937 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:52:36.312779 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b6k9m\" (UniqueName: \"kubernetes.io/projected/0163bdf8-1e1c-4c6d-8072-b6416c913fe5-kube-api-access-b6k9m\") pod \"payload-pre-processing-e2e-mt-713ee90e-86c97fff6b-xmc2n\" (UID: \"0163bdf8-1e1c-4c6d-8072-b6416c913fe5\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-713ee90e-86c97fff6b-xmc2n" Sep 10 13:52:36.313486 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:52:36.313460 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0163bdf8-1e1c-4c6d-8072-b6416c913fe5-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-713ee90e-86c97fff6b-xmc2n\" (UID: \"0163bdf8-1e1c-4c6d-8072-b6416c913fe5\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-713ee90e-86c97fff6b-xmc2n" Sep 10 13:52:36.321854 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:52:36.321831 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-b6k9m\" (UniqueName: \"kubernetes.io/projected/0163bdf8-1e1c-4c6d-8072-b6416c913fe5-kube-api-access-b6k9m\") pod \"payload-pre-processing-e2e-mt-713ee90e-86c97fff6b-xmc2n\" (UID: \"0163bdf8-1e1c-4c6d-8072-b6416c913fe5\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-713ee90e-86c97fff6b-xmc2n" Sep 10 13:52:36.328185 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:52:36.328157 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fxgmp_fa53c783-d40a-4024-9f7b-1d0d2b57f6dc/konnectivity-agent/0.log" limit=1024 Sep 10 13:52:36.501449 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:52:36.501416 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-713ee90e-86c97fff6b-xmc2n" Sep 10 13:52:36.595792 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:52:36.595700 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-713ee90e-59584c8f96-55vtb" event={"ID":"85733206-46ff-4ed1-824e-fe0beec5c828","Type":"ContainerStarted","Data":"33a16d013797d2c21e3ba6ce77a845072972a123d71d99d49e546ad08905ffb7"} Sep 10 13:52:36.781627 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:52:36.781591 2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-713ee90e-86c97fff6b-xmc2n"] Sep 10 13:52:36.783034 ip-10-0-140-8 kubenswrapper[2585]: W0910 13:52:36.783001 2585 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0163bdf8_1e1c_4c6d_8072_b6416c913fe5.slice/crio-2c8c0f3f2c61f17b611e4809c885608eeef91f7d5b72e56880a3ab4f369f0503 WatchSource:0}: Error finding container 2c8c0f3f2c61f17b611e4809c885608eeef91f7d5b72e56880a3ab4f369f0503: Status 404 returned error can't find the container with id 2c8c0f3f2c61f17b611e4809c885608eeef91f7d5b72e56880a3ab4f369f0503 Sep 10 13:52:37.604868 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:52:37.604816 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-713ee90e-86c97fff6b-xmc2n" event={"ID":"0163bdf8-1e1c-4c6d-8072-b6416c913fe5","Type":"ContainerStarted","Data":"067b13f33243a60d85e5a3249a4ae11332921ae2671a1655b62d5d46ccc62271"} Sep 10 13:52:37.605345 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:52:37.604882 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-713ee90e-86c97fff6b-xmc2n" event={"ID":"0163bdf8-1e1c-4c6d-8072-b6416c913fe5","Type":"ContainerStarted","Data":"2c8c0f3f2c61f17b611e4809c885608eeef91f7d5b72e56880a3ab4f369f0503"} Sep 10 13:52:37.605345 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:52:37.604903 2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-mt-713ee90e-86c97fff6b-xmc2n" Sep 10 13:52:37.631825 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:52:37.631770 2585 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-mt-713ee90e-86c97fff6b-xmc2n" podStartSLOduration=1.631753405 podStartE2EDuration="1.631753405s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 13:52:36 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 13:52:37.628373309 +0000 UTC m=+1442.501077511" watchObservedRunningTime="2026-09-10 13:52:37.631753405 +0000 UTC m=+1442.504457597" Sep 10 13:52:38.612982 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:52:38.612948 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-713ee90e-59584c8f96-55vtb" event={"ID":"85733206-46ff-4ed1-824e-fe0beec5c828","Type":"ContainerStarted","Data":"779c4abfed0190c186816da821c086a6ebb671161324bf19ec75000417ba507e"} Sep 10 13:52:38.633823 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:52:38.633778 2585 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-mt-713ee90e-59584c8f96-55vtb" podStartSLOduration=1.9001005709999998 podStartE2EDuration="3.633761346s" totalImagesPullingTime="1.733660775s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 13:52:35 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 13:52:38.632420897 +0000 UTC m=+1443.505125128" watchObservedRunningTime="2026-09-10 13:52:38.633761346 +0000 UTC m=+1443.506465614" Sep 10 13:52:39.619583 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:52:39.619547 2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-mt-713ee90e-59584c8f96-55vtb" Sep 10 13:52:45.631056 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:52:45.631023 2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-mt-713ee90e-59584c8f96-55vtb" Sep 10 13:52:48.615990 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:52:48.615960 2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-mt-713ee90e-86c97fff6b-xmc2n" Sep 10 13:52:59.279941 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:52:59.279904 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fxgmp_fa53c783-d40a-4024-9f7b-1d0d2b57f6dc/konnectivity-agent/0.log" limit=1024 Sep 10 13:53:24.273471 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:53:24.273417 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fxgmp_fa53c783-d40a-4024-9f7b-1d0d2b57f6dc/konnectivity-agent/0.log" limit=1024 Sep 10 13:53:35.713194 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:53:35.713165 2585 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-l9l8q_ad555c6a-4385-4f8e-93bb-a174cfaa8e74/kube-multus/2.log" Sep 10 13:53:35.718568 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:53:35.718544 2585 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-l9l8q_ad555c6a-4385-4f8e-93bb-a174cfaa8e74/kube-multus/2.log" Sep 10 13:53:35.827031 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:53:35.826998 2585 scope.go:117] "RemoveContainer" containerID="9bbe617cc779e7c87eeea60f42c051c83f52ac5cf21451bc0c8efb5a1cddf646" Sep 10 13:53:44.248916 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:53:44.248883 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fxgmp_fa53c783-d40a-4024-9f7b-1d0d2b57f6dc/konnectivity-agent/0.log" limit=1024 Sep 10 13:54:04.040871 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:54:04.040783 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fxgmp_fa53c783-d40a-4024-9f7b-1d0d2b57f6dc/konnectivity-agent/0.log" limit=1024 Sep 10 13:54:12.465930 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:54:12.465896 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fxgmp_fa53c783-d40a-4024-9f7b-1d0d2b57f6dc/konnectivity-agent/0.log" limit=1024 Sep 10 13:54:12.524101 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:54:12.524066 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fxgmp_fa53c783-d40a-4024-9f7b-1d0d2b57f6dc/konnectivity-agent/0.log" limit=1024 Sep 10 13:54:20.500816 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:54:20.500784 2585 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-713ee90e-59584c8f96-55vtb"] Sep 10 13:54:20.501391 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:54:20.500985 2585 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-mt-713ee90e-59584c8f96-55vtb" podUID="85733206-46ff-4ed1-824e-fe0beec5c828" containerName="maas-api" containerID="cri-o://779c4abfed0190c186816da821c086a6ebb671161324bf19ec75000417ba507e" gracePeriod=30 Sep 10 13:54:20.532136 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:54:20.532103 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fxgmp_fa53c783-d40a-4024-9f7b-1d0d2b57f6dc/konnectivity-agent/0.log" limit=1024 Sep 10 13:54:20.568024 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:54:20.567988 2585 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-713ee90e-86c97fff6b-xmc2n"] Sep 10 13:54:20.568331 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:54:20.568297 2585 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-mt-713ee90e-86c97fff6b-xmc2n" podUID="0163bdf8-1e1c-4c6d-8072-b6416c913fe5" containerName="payload-pre-processing" containerID="cri-o://067b13f33243a60d85e5a3249a4ae11332921ae2671a1655b62d5d46ccc62271" gracePeriod=30 Sep 10 13:54:20.625417 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:54:20.625377 2585 prober.go:120] "Probe failed" probeType="Readiness" pod="odh-ai-gateway-infra/maas-api-e2e-mt-713ee90e-59584c8f96-55vtb" podUID="85733206-46ff-4ed1-824e-fe0beec5c828" containerName="maas-api" probeResult="failure" output="Get \"https://10.133.0.71:8443/health\": dial tcp 10.133.0.71:8443: connect: connection refused" Sep 10 13:54:20.633399 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:54:20.633367 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fxgmp_fa53c783-d40a-4024-9f7b-1d0d2b57f6dc/konnectivity-agent/0.log" limit=1024 Sep 10 13:54:20.732711 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:54:20.732668 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fxgmp_fa53c783-d40a-4024-9f7b-1d0d2b57f6dc/konnectivity-agent/0.log" limit=1024 Sep 10 13:54:20.788551 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:54:20.788518 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fxgmp_fa53c783-d40a-4024-9f7b-1d0d2b57f6dc/konnectivity-agent/0.log" limit=1024 Sep 10 13:54:20.818932 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:54:20.818907 2585 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-713ee90e-59584c8f96-55vtb" Sep 10 13:54:20.848937 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:54:20.848910 2585 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-713ee90e-86c97fff6b-xmc2n" Sep 10 13:54:20.933958 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:54:20.933931 2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/85733206-46ff-4ed1-824e-fe0beec5c828-metrics-certs\") pod \"85733206-46ff-4ed1-824e-fe0beec5c828\" (UID: \"85733206-46ff-4ed1-824e-fe0beec5c828\") " Sep 10 13:54:20.934162 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:54:20.933994 2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/85733206-46ff-4ed1-824e-fe0beec5c828-openshift-service-ca\") pod \"85733206-46ff-4ed1-824e-fe0beec5c828\" (UID: \"85733206-46ff-4ed1-824e-fe0beec5c828\") " Sep 10 13:54:20.934162 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:54:20.934025 2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-t2dq9\" (UniqueName: \"kubernetes.io/projected/85733206-46ff-4ed1-824e-fe0beec5c828-kube-api-access-t2dq9\") pod \"85733206-46ff-4ed1-824e-fe0beec5c828\" (UID: \"85733206-46ff-4ed1-824e-fe0beec5c828\") " Sep 10 13:54:20.934162 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:54:20.934076 2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/85733206-46ff-4ed1-824e-fe0beec5c828-maas-api-tls\") pod \"85733206-46ff-4ed1-824e-fe0beec5c828\" (UID: \"85733206-46ff-4ed1-824e-fe0beec5c828\") " Sep 10 13:54:20.934162 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:54:20.934096 2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-b6k9m\" (UniqueName: \"kubernetes.io/projected/0163bdf8-1e1c-4c6d-8072-b6416c913fe5-kube-api-access-b6k9m\") pod \"0163bdf8-1e1c-4c6d-8072-b6416c913fe5\" (UID: \"0163bdf8-1e1c-4c6d-8072-b6416c913fe5\") " Sep 10 13:54:20.934162 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:54:20.934132 2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0163bdf8-1e1c-4c6d-8072-b6416c913fe5-plugins-config-volume\") pod \"0163bdf8-1e1c-4c6d-8072-b6416c913fe5\" (UID: \"0163bdf8-1e1c-4c6d-8072-b6416c913fe5\") " Sep 10 13:54:20.934556 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:54:20.934525 2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/85733206-46ff-4ed1-824e-fe0beec5c828-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "85733206-46ff-4ed1-824e-fe0beec5c828" (UID: "85733206-46ff-4ed1-824e-fe0beec5c828"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 10 13:54:20.934666 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:54:20.934606 2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0163bdf8-1e1c-4c6d-8072-b6416c913fe5-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "0163bdf8-1e1c-4c6d-8072-b6416c913fe5" (UID: "0163bdf8-1e1c-4c6d-8072-b6416c913fe5"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 10 13:54:20.936673 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:54:20.936646 2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0163bdf8-1e1c-4c6d-8072-b6416c913fe5-kube-api-access-b6k9m" (OuterVolumeSpecName: "kube-api-access-b6k9m") pod "0163bdf8-1e1c-4c6d-8072-b6416c913fe5" (UID: "0163bdf8-1e1c-4c6d-8072-b6416c913fe5"). InnerVolumeSpecName "kube-api-access-b6k9m". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 10 13:54:20.936759 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:54:20.936693 2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/85733206-46ff-4ed1-824e-fe0beec5c828-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "85733206-46ff-4ed1-824e-fe0beec5c828" (UID: "85733206-46ff-4ed1-824e-fe0beec5c828"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 10 13:54:20.936830 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:54:20.936779 2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/85733206-46ff-4ed1-824e-fe0beec5c828-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "85733206-46ff-4ed1-824e-fe0beec5c828" (UID: "85733206-46ff-4ed1-824e-fe0beec5c828"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 10 13:54:20.936830 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:54:20.936818 2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/85733206-46ff-4ed1-824e-fe0beec5c828-kube-api-access-t2dq9" (OuterVolumeSpecName: "kube-api-access-t2dq9") pod "85733206-46ff-4ed1-824e-fe0beec5c828" (UID: "85733206-46ff-4ed1-824e-fe0beec5c828"). InnerVolumeSpecName "kube-api-access-t2dq9". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 10 13:54:21.035876 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:54:21.035779 2585 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-t2dq9\" (UniqueName: \"kubernetes.io/projected/85733206-46ff-4ed1-824e-fe0beec5c828-kube-api-access-t2dq9\") on node \"ip-10-0-140-8.ec2.internal\" DevicePath \"\"" Sep 10 13:54:21.035876 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:54:21.035822 2585 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/85733206-46ff-4ed1-824e-fe0beec5c828-maas-api-tls\") on node \"ip-10-0-140-8.ec2.internal\" DevicePath \"\"" Sep 10 13:54:21.035876 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:54:21.035836 2585 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-b6k9m\" (UniqueName: \"kubernetes.io/projected/0163bdf8-1e1c-4c6d-8072-b6416c913fe5-kube-api-access-b6k9m\") on node \"ip-10-0-140-8.ec2.internal\" DevicePath \"\"" Sep 10 13:54:21.035876 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:54:21.035846 2585 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0163bdf8-1e1c-4c6d-8072-b6416c913fe5-plugins-config-volume\") on node \"ip-10-0-140-8.ec2.internal\" DevicePath \"\"" Sep 10 13:54:21.035876 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:54:21.035855 2585 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/85733206-46ff-4ed1-824e-fe0beec5c828-metrics-certs\") on node \"ip-10-0-140-8.ec2.internal\" DevicePath \"\"" Sep 10 13:54:21.035876 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:54:21.035865 2585 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/85733206-46ff-4ed1-824e-fe0beec5c828-openshift-service-ca\") on node \"ip-10-0-140-8.ec2.internal\" DevicePath \"\"" Sep 10 13:54:21.199377 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:54:21.199336 2585 generic.go:356] "Generic (PLEG): container finished" podID="0163bdf8-1e1c-4c6d-8072-b6416c913fe5" containerID="067b13f33243a60d85e5a3249a4ae11332921ae2671a1655b62d5d46ccc62271" exitCode=0 Sep 10 13:54:21.199572 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:54:21.199397 2585 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-713ee90e-86c97fff6b-xmc2n" Sep 10 13:54:21.199572 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:54:21.199424 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-713ee90e-86c97fff6b-xmc2n" event={"ID":"0163bdf8-1e1c-4c6d-8072-b6416c913fe5","Type":"ContainerDied","Data":"067b13f33243a60d85e5a3249a4ae11332921ae2671a1655b62d5d46ccc62271"} Sep 10 13:54:21.199572 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:54:21.199513 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-713ee90e-86c97fff6b-xmc2n" event={"ID":"0163bdf8-1e1c-4c6d-8072-b6416c913fe5","Type":"ContainerDied","Data":"2c8c0f3f2c61f17b611e4809c885608eeef91f7d5b72e56880a3ab4f369f0503"} Sep 10 13:54:21.199572 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:54:21.199544 2585 scope.go:117] "RemoveContainer" containerID="067b13f33243a60d85e5a3249a4ae11332921ae2671a1655b62d5d46ccc62271" Sep 10 13:54:21.201085 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:54:21.201061 2585 generic.go:356] "Generic (PLEG): container finished" podID="85733206-46ff-4ed1-824e-fe0beec5c828" containerID="779c4abfed0190c186816da821c086a6ebb671161324bf19ec75000417ba507e" exitCode=0 Sep 10 13:54:21.201202 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:54:21.201123 2585 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-713ee90e-59584c8f96-55vtb" Sep 10 13:54:21.201202 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:54:21.201123 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-713ee90e-59584c8f96-55vtb" event={"ID":"85733206-46ff-4ed1-824e-fe0beec5c828","Type":"ContainerDied","Data":"779c4abfed0190c186816da821c086a6ebb671161324bf19ec75000417ba507e"} Sep 10 13:54:21.201320 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:54:21.201231 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-713ee90e-59584c8f96-55vtb" event={"ID":"85733206-46ff-4ed1-824e-fe0beec5c828","Type":"ContainerDied","Data":"33a16d013797d2c21e3ba6ce77a845072972a123d71d99d49e546ad08905ffb7"} Sep 10 13:54:21.210770 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:54:21.210744 2585 scope.go:117] "RemoveContainer" containerID="067b13f33243a60d85e5a3249a4ae11332921ae2671a1655b62d5d46ccc62271" Sep 10 13:54:21.211384 ip-10-0-140-8 kubenswrapper[2585]: E0910 13:54:21.211271 2585 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"067b13f33243a60d85e5a3249a4ae11332921ae2671a1655b62d5d46ccc62271\": container with ID starting with 067b13f33243a60d85e5a3249a4ae11332921ae2671a1655b62d5d46ccc62271 not found: ID does not exist" containerID="067b13f33243a60d85e5a3249a4ae11332921ae2671a1655b62d5d46ccc62271" Sep 10 13:54:21.211384 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:54:21.211323 2585 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"067b13f33243a60d85e5a3249a4ae11332921ae2671a1655b62d5d46ccc62271"} err="failed to get container status \"067b13f33243a60d85e5a3249a4ae11332921ae2671a1655b62d5d46ccc62271\": rpc error: code = NotFound desc = could not find container \"067b13f33243a60d85e5a3249a4ae11332921ae2671a1655b62d5d46ccc62271\": container with ID starting with 067b13f33243a60d85e5a3249a4ae11332921ae2671a1655b62d5d46ccc62271 not found: ID does not exist" Sep 10 13:54:21.211384 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:54:21.211352 2585 scope.go:117] "RemoveContainer" containerID="779c4abfed0190c186816da821c086a6ebb671161324bf19ec75000417ba507e" Sep 10 13:54:21.219904 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:54:21.219886 2585 scope.go:117] "RemoveContainer" containerID="779c4abfed0190c186816da821c086a6ebb671161324bf19ec75000417ba507e" Sep 10 13:54:21.220167 ip-10-0-140-8 kubenswrapper[2585]: E0910 13:54:21.220147 2585 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"779c4abfed0190c186816da821c086a6ebb671161324bf19ec75000417ba507e\": container with ID starting with 779c4abfed0190c186816da821c086a6ebb671161324bf19ec75000417ba507e not found: ID does not exist" containerID="779c4abfed0190c186816da821c086a6ebb671161324bf19ec75000417ba507e" Sep 10 13:54:21.220230 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:54:21.220177 2585 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"779c4abfed0190c186816da821c086a6ebb671161324bf19ec75000417ba507e"} err="failed to get container status \"779c4abfed0190c186816da821c086a6ebb671161324bf19ec75000417ba507e\": rpc error: code = NotFound desc = could not find container \"779c4abfed0190c186816da821c086a6ebb671161324bf19ec75000417ba507e\": container with ID starting with 779c4abfed0190c186816da821c086a6ebb671161324bf19ec75000417ba507e not found: ID does not exist" Sep 10 13:54:21.226780 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:54:21.226757 2585 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-713ee90e-86c97fff6b-xmc2n"] Sep 10 13:54:21.230889 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:54:21.230865 2585 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-713ee90e-86c97fff6b-xmc2n"] Sep 10 13:54:21.243615 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:54:21.243582 2585 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-713ee90e-59584c8f96-55vtb"] Sep 10 13:54:21.248024 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:54:21.248002 2585 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-713ee90e-59584c8f96-55vtb"] Sep 10 13:54:21.647696 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:54:21.647668 2585 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="0163bdf8-1e1c-4c6d-8072-b6416c913fe5" path="/var/lib/kubelet/pods/0163bdf8-1e1c-4c6d-8072-b6416c913fe5/volumes" Sep 10 13:54:21.648101 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:54:21.648085 2585 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="85733206-46ff-4ed1-824e-fe0beec5c828" path="/var/lib/kubelet/pods/85733206-46ff-4ed1-824e-fe0beec5c828/volumes" Sep 10 13:54:24.002212 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:54:24.002183 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fxgmp_fa53c783-d40a-4024-9f7b-1d0d2b57f6dc/konnectivity-agent/0.log" limit=1024 Sep 10 13:54:29.037034 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:54:29.036972 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fxgmp_fa53c783-d40a-4024-9f7b-1d0d2b57f6dc/konnectivity-agent/0.log" limit=1024 Sep 10 13:54:35.848449 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:54:35.848418 2585 scope.go:117] "RemoveContainer" containerID="8ca39d8469b882fc7f9bd894e69fa7e0631ec669481ef591d03f03ecab54051e" Sep 10 13:55:05.297671 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:55:05.297633 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fxgmp_fa53c783-d40a-4024-9f7b-1d0d2b57f6dc/konnectivity-agent/0.log" limit=1024 Sep 10 13:55:07.692501 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:55:07.692465 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fxgmp_fa53c783-d40a-4024-9f7b-1d0d2b57f6dc/konnectivity-agent/0.log" limit=1024 Sep 10 13:55:14.468023 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:55:14.467991 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fxgmp_fa53c783-d40a-4024-9f7b-1d0d2b57f6dc/konnectivity-agent/0.log" limit=1024 Sep 10 13:55:21.034694 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:55:21.034657 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fxgmp_fa53c783-d40a-4024-9f7b-1d0d2b57f6dc/konnectivity-agent/0.log" limit=1024 Sep 10 13:55:21.140489 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:55:21.140454 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fxgmp_fa53c783-d40a-4024-9f7b-1d0d2b57f6dc/konnectivity-agent/0.log" limit=1024 Sep 10 13:55:21.294035 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:55:21.293940 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fxgmp_fa53c783-d40a-4024-9f7b-1d0d2b57f6dc/konnectivity-agent/0.log" limit=1024 Sep 10 13:55:24.897878 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:55:24.897844 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fxgmp_fa53c783-d40a-4024-9f7b-1d0d2b57f6dc/konnectivity-agent/0.log" limit=1024 Sep 10 13:55:25.012654 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:55:25.012614 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fxgmp_fa53c783-d40a-4024-9f7b-1d0d2b57f6dc/konnectivity-agent/0.log" limit=1024 Sep 10 13:55:25.096078 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:55:25.096047 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fxgmp_fa53c783-d40a-4024-9f7b-1d0d2b57f6dc/konnectivity-agent/0.log" limit=1024 Sep 10 13:55:26.166227 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:55:26.166189 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fxgmp_fa53c783-d40a-4024-9f7b-1d0d2b57f6dc/konnectivity-agent/0.log" limit=1024 Sep 10 13:55:32.099518 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:55:32.099485 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fxgmp_fa53c783-d40a-4024-9f7b-1d0d2b57f6dc/konnectivity-agent/0.log" limit=1024 Sep 10 13:55:32.178349 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:55:32.178300 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fxgmp_fa53c783-d40a-4024-9f7b-1d0d2b57f6dc/konnectivity-agent/0.log" limit=1024 Sep 10 13:55:32.793585 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:55:32.793554 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fxgmp_fa53c783-d40a-4024-9f7b-1d0d2b57f6dc/konnectivity-agent/0.log" limit=1024 Sep 10 13:55:33.762815 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:55:33.762778 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fxgmp_fa53c783-d40a-4024-9f7b-1d0d2b57f6dc/konnectivity-agent/0.log" limit=1024 Sep 10 13:55:33.865671 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:55:33.865633 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fxgmp_fa53c783-d40a-4024-9f7b-1d0d2b57f6dc/konnectivity-agent/0.log" limit=1024 Sep 10 13:55:35.878697 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:55:35.878664 2585 scope.go:117] "RemoveContainer" containerID="d3ab127fa4ab61c3d341ccabb80b690cde2f4e3ea3b889842691d3dda78bd347" Sep 10 13:55:38.575804 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:55:38.575766 2585 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7519-475754"] Sep 10 13:55:38.576206 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:55:38.576192 2585 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="85733206-46ff-4ed1-824e-fe0beec5c828" containerName="maas-api" Sep 10 13:55:38.576280 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:55:38.576208 2585 state_mem.go:107] "Deleted CPUSet assignment" podUID="85733206-46ff-4ed1-824e-fe0beec5c828" containerName="maas-api" Sep 10 13:55:38.576280 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:55:38.576225 2585 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0163bdf8-1e1c-4c6d-8072-b6416c913fe5" containerName="payload-pre-processing" Sep 10 13:55:38.576280 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:55:38.576231 2585 state_mem.go:107] "Deleted CPUSet assignment" podUID="0163bdf8-1e1c-4c6d-8072-b6416c913fe5" containerName="payload-pre-processing" Sep 10 13:55:38.576385 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:55:38.576324 2585 memory_manager.go:368] "RemoveStaleState removing state" podUID="0163bdf8-1e1c-4c6d-8072-b6416c913fe5" containerName="payload-pre-processing" Sep 10 13:55:38.576385 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:55:38.576331 2585 memory_manager.go:368] "RemoveStaleState removing state" podUID="85733206-46ff-4ed1-824e-fe0beec5c828" containerName="maas-api" Sep 10 13:55:38.579463 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:55:38.579445 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7519-475754" Sep 10 13:55:38.582136 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:55:38.582114 2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"default-dockercfg-zn6pb\"" Sep 10 13:55:38.588501 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:55:38.588479 2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7519-475754"] Sep 10 13:55:38.659583 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:55:38.659550 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-n6zh7\" (UniqueName: \"kubernetes.io/projected/69ea9233-7093-4601-8ce4-ce0648231598-kube-api-access-n6zh7\") pod \"test-curl-7519-475754\" (UID: \"69ea9233-7093-4601-8ce4-ce0648231598\") " pod="opendatahub/test-curl-7519-475754" Sep 10 13:55:38.760429 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:55:38.760382 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n6zh7\" (UniqueName: \"kubernetes.io/projected/69ea9233-7093-4601-8ce4-ce0648231598-kube-api-access-n6zh7\") pod \"test-curl-7519-475754\" (UID: \"69ea9233-7093-4601-8ce4-ce0648231598\") " pod="opendatahub/test-curl-7519-475754" Sep 10 13:55:38.771086 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:55:38.771056 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-n6zh7\" (UniqueName: \"kubernetes.io/projected/69ea9233-7093-4601-8ce4-ce0648231598-kube-api-access-n6zh7\") pod \"test-curl-7519-475754\" (UID: \"69ea9233-7093-4601-8ce4-ce0648231598\") " pod="opendatahub/test-curl-7519-475754" Sep 10 13:55:38.889116 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:55:38.889034 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7519-475754" Sep 10 13:55:39.159561 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:55:39.159535 2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7519-475754"] Sep 10 13:55:39.161988 ip-10-0-140-8 kubenswrapper[2585]: W0910 13:55:39.161960 2585 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod69ea9233_7093_4601_8ce4_ce0648231598.slice/crio-22126761c0c4153fb30b0f85e95b455ca22e4763b4bd42b39609860eaaa96bee WatchSource:0}: Error finding container 22126761c0c4153fb30b0f85e95b455ca22e4763b4bd42b39609860eaaa96bee: Status 404 returned error can't find the container with id 22126761c0c4153fb30b0f85e95b455ca22e4763b4bd42b39609860eaaa96bee Sep 10 13:55:39.640451 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:55:39.640416 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7519-475754" event={"ID":"69ea9233-7093-4601-8ce4-ce0648231598","Type":"ContainerStarted","Data":"4961df1fc56d250ca73fa927dc12ab328dded4119d5fc9b4238782377f687e66"} Sep 10 13:55:39.640825 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:55:39.640461 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7519-475754" event={"ID":"69ea9233-7093-4601-8ce4-ce0648231598","Type":"ContainerStarted","Data":"22126761c0c4153fb30b0f85e95b455ca22e4763b4bd42b39609860eaaa96bee"} Sep 10 13:55:39.657579 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:55:39.657529 2585 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/test-curl-7519-475754" podStartSLOduration=1.657513351 podStartE2EDuration="1.657513351s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 13:55:38 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 13:55:39.656606695 +0000 UTC m=+1624.529310909" watchObservedRunningTime="2026-09-10 13:55:39.657513351 +0000 UTC m=+1624.530217552" Sep 10 13:55:39.722821 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:55:39.722777 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fxgmp_fa53c783-d40a-4024-9f7b-1d0d2b57f6dc/konnectivity-agent/0.log" limit=1024 Sep 10 13:55:39.788214 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:55:39.788176 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fxgmp_fa53c783-d40a-4024-9f7b-1d0d2b57f6dc/konnectivity-agent/0.log" limit=1024 Sep 10 13:55:39.866424 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:55:39.866388 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fxgmp_fa53c783-d40a-4024-9f7b-1d0d2b57f6dc/konnectivity-agent/0.log" limit=1024 Sep 10 13:55:39.958396 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:55:39.958318 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fxgmp_fa53c783-d40a-4024-9f7b-1d0d2b57f6dc/konnectivity-agent/0.log" limit=1024 Sep 10 13:55:40.058471 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:55:40.058437 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fxgmp_fa53c783-d40a-4024-9f7b-1d0d2b57f6dc/konnectivity-agent/0.log" limit=1024 Sep 10 13:55:40.398181 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:55:40.398147 2585 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7519-475754"] Sep 10 13:55:40.401564 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:55:40.401533 2585 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7519-475754"] Sep 10 13:55:40.646085 ip-10-0-140-8 kubenswrapper[2585]: E0910 13:55:40.646056 2585 kuberuntime_manager.go:649] "Missing actuated resource record" pod="opendatahub/test-curl-7519-475754" container="test-curl-7519-475754" Sep 10 13:55:41.651079 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:55:41.651043 2585 kuberuntime_container.go:873] "Killing container with a grace period" pod="opendatahub/test-curl-7519-475754" podUID="69ea9233-7093-4601-8ce4-ce0648231598" containerName="test-curl-7519-475754" containerID="cri-o://4961df1fc56d250ca73fa927dc12ab328dded4119d5fc9b4238782377f687e66" gracePeriod=2 Sep 10 13:55:42.059082 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:55:42.059054 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fxgmp_fa53c783-d40a-4024-9f7b-1d0d2b57f6dc/konnectivity-agent/0.log" limit=1024 Sep 10 13:55:42.933608 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:55:42.933575 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fxgmp_fa53c783-d40a-4024-9f7b-1d0d2b57f6dc/konnectivity-agent/0.log" limit=1024 Sep 10 13:55:43.835898 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:55:43.835873 2585 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7519-475754" Sep 10 13:55:43.903887 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:55:43.903858 2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-n6zh7\" (UniqueName: \"kubernetes.io/projected/69ea9233-7093-4601-8ce4-ce0648231598-kube-api-access-n6zh7\") pod \"69ea9233-7093-4601-8ce4-ce0648231598\" (UID: \"69ea9233-7093-4601-8ce4-ce0648231598\") " Sep 10 13:55:43.905932 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:55:43.905867 2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/69ea9233-7093-4601-8ce4-ce0648231598-kube-api-access-n6zh7" (OuterVolumeSpecName: "kube-api-access-n6zh7") pod "69ea9233-7093-4601-8ce4-ce0648231598" (UID: "69ea9233-7093-4601-8ce4-ce0648231598"). InnerVolumeSpecName "kube-api-access-n6zh7". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 10 13:55:44.004618 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:55:44.004579 2585 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-n6zh7\" (UniqueName: \"kubernetes.io/projected/69ea9233-7093-4601-8ce4-ce0648231598-kube-api-access-n6zh7\") on node \"ip-10-0-140-8.ec2.internal\" DevicePath \"\"" Sep 10 13:55:44.668338 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:55:44.668303 2585 generic.go:356] "Generic (PLEG): container finished" podID="69ea9233-7093-4601-8ce4-ce0648231598" containerID="4961df1fc56d250ca73fa927dc12ab328dded4119d5fc9b4238782377f687e66" exitCode=137 Sep 10 13:55:44.668522 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:55:44.668342 2585 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7519-475754" Sep 10 13:55:44.668522 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:55:44.668388 2585 scope.go:117] "RemoveContainer" containerID="4961df1fc56d250ca73fa927dc12ab328dded4119d5fc9b4238782377f687e66" Sep 10 13:55:44.677427 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:55:44.677401 2585 scope.go:117] "RemoveContainer" containerID="4961df1fc56d250ca73fa927dc12ab328dded4119d5fc9b4238782377f687e66" Sep 10 13:55:44.677701 ip-10-0-140-8 kubenswrapper[2585]: E0910 13:55:44.677680 2585 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4961df1fc56d250ca73fa927dc12ab328dded4119d5fc9b4238782377f687e66\": container with ID starting with 4961df1fc56d250ca73fa927dc12ab328dded4119d5fc9b4238782377f687e66 not found: ID does not exist" containerID="4961df1fc56d250ca73fa927dc12ab328dded4119d5fc9b4238782377f687e66" Sep 10 13:55:44.677760 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:55:44.677710 2585 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4961df1fc56d250ca73fa927dc12ab328dded4119d5fc9b4238782377f687e66"} err="failed to get container status \"4961df1fc56d250ca73fa927dc12ab328dded4119d5fc9b4238782377f687e66\": rpc error: code = NotFound desc = could not find container \"4961df1fc56d250ca73fa927dc12ab328dded4119d5fc9b4238782377f687e66\": container with ID starting with 4961df1fc56d250ca73fa927dc12ab328dded4119d5fc9b4238782377f687e66 not found: ID does not exist" Sep 10 13:55:44.678887 ip-10-0-140-8 kubenswrapper[2585]: E0910 13:55:44.678861 2585 status_manager.go:1018] "Failed to get status for pod" err="pods \"test-curl-7519-475754\" is forbidden: User \"system:node:ip-10-0-140-8.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-140-8.ec2.internal' and this object" podUID="69ea9233-7093-4601-8ce4-ce0648231598" pod="opendatahub/test-curl-7519-475754" Sep 10 13:55:45.646053 ip-10-0-140-8 kubenswrapper[2585]: E0910 13:55:45.646010 2585 status_manager.go:1018] "Failed to get status for pod" err="pods \"test-curl-7519-475754\" is forbidden: User \"system:node:ip-10-0-140-8.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-140-8.ec2.internal' and this object" podUID="69ea9233-7093-4601-8ce4-ce0648231598" pod="opendatahub/test-curl-7519-475754" Sep 10 13:55:45.646994 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:55:45.646971 2585 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="69ea9233-7093-4601-8ce4-ce0648231598" path="/var/lib/kubelet/pods/69ea9233-7093-4601-8ce4-ce0648231598/volumes" Sep 10 13:55:50.416638 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:55:50.416596 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fxgmp_fa53c783-d40a-4024-9f7b-1d0d2b57f6dc/konnectivity-agent/0.log" limit=1024 Sep 10 13:55:50.668301 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:55:50.668209 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fxgmp_fa53c783-d40a-4024-9f7b-1d0d2b57f6dc/konnectivity-agent/0.log" limit=1024 Sep 10 13:55:54.122744 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:55:54.122679 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fxgmp_fa53c783-d40a-4024-9f7b-1d0d2b57f6dc/konnectivity-agent/0.log" limit=1024 Sep 10 13:56:11.683943 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:56:11.683902 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fxgmp_fa53c783-d40a-4024-9f7b-1d0d2b57f6dc/konnectivity-agent/0.log" limit=1024 Sep 10 13:56:19.066029 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:56:19.065998 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fxgmp_fa53c783-d40a-4024-9f7b-1d0d2b57f6dc/konnectivity-agent/0.log" limit=1024 Sep 10 13:56:19.165193 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:56:19.165162 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fxgmp_fa53c783-d40a-4024-9f7b-1d0d2b57f6dc/konnectivity-agent/0.log" limit=1024 Sep 10 13:56:19.371464 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:56:19.371383 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fxgmp_fa53c783-d40a-4024-9f7b-1d0d2b57f6dc/konnectivity-agent/0.log" limit=1024 Sep 10 13:56:19.709299 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:56:19.709178 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fxgmp_fa53c783-d40a-4024-9f7b-1d0d2b57f6dc/konnectivity-agent/0.log" limit=1024 Sep 10 13:56:20.181746 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:56:20.181686 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fxgmp_fa53c783-d40a-4024-9f7b-1d0d2b57f6dc/konnectivity-agent/0.log" limit=1024 Sep 10 13:56:22.638072 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:56:22.638039 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fxgmp_fa53c783-d40a-4024-9f7b-1d0d2b57f6dc/konnectivity-agent/0.log" limit=1024 Sep 10 13:56:22.733321 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:56:22.733289 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fxgmp_fa53c783-d40a-4024-9f7b-1d0d2b57f6dc/konnectivity-agent/0.log" limit=1024 Sep 10 13:56:23.976216 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:56:23.976182 2585 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-ab74b3-5d985f8f7f-llgx2"] Sep 10 13:56:23.976779 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:56:23.976760 2585 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="69ea9233-7093-4601-8ce4-ce0648231598" containerName="test-curl-7519-475754" Sep 10 13:56:23.976779 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:56:23.976780 2585 state_mem.go:107] "Deleted CPUSet assignment" podUID="69ea9233-7093-4601-8ce4-ce0648231598" containerName="test-curl-7519-475754" Sep 10 13:56:23.976900 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:56:23.976870 2585 memory_manager.go:368] "RemoveStaleState removing state" podUID="69ea9233-7093-4601-8ce4-ce0648231598" containerName="test-curl-7519-475754" Sep 10 13:56:23.980204 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:56:23.980181 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-ab74b3-5d985f8f7f-llgx2" Sep 10 13:56:23.981499 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:56:23.981476 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0a9b5f2f-c6eb-4915-9c8f-731e7ca6c3a4-openshift-service-ca\") pod \"maas-api-e2e-api-a-ab74b3-5d985f8f7f-llgx2\" (UID: \"0a9b5f2f-c6eb-4915-9c8f-731e7ca6c3a4\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-ab74b3-5d985f8f7f-llgx2" Sep 10 13:56:23.981614 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:56:23.981543 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6nnhp\" (UniqueName: \"kubernetes.io/projected/0a9b5f2f-c6eb-4915-9c8f-731e7ca6c3a4-kube-api-access-6nnhp\") pod \"maas-api-e2e-api-a-ab74b3-5d985f8f7f-llgx2\" (UID: \"0a9b5f2f-c6eb-4915-9c8f-731e7ca6c3a4\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-ab74b3-5d985f8f7f-llgx2" Sep 10 13:56:23.981614 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:56:23.981586 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0a9b5f2f-c6eb-4915-9c8f-731e7ca6c3a4-metrics-certs\") pod \"maas-api-e2e-api-a-ab74b3-5d985f8f7f-llgx2\" (UID: \"0a9b5f2f-c6eb-4915-9c8f-731e7ca6c3a4\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-ab74b3-5d985f8f7f-llgx2" Sep 10 13:56:23.981754 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:56:23.981616 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0a9b5f2f-c6eb-4915-9c8f-731e7ca6c3a4-maas-api-tls\") pod \"maas-api-e2e-api-a-ab74b3-5d985f8f7f-llgx2\" (UID: \"0a9b5f2f-c6eb-4915-9c8f-731e7ca6c3a4\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-ab74b3-5d985f8f7f-llgx2" Sep 10 13:56:23.985327 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:56:23.985306 2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-metrics-tls\"" Sep 10 13:56:23.985519 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:56:23.985384 2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Sep 10 13:56:23.985598 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:56:23.985395 2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-q8qnd\"" Sep 10 13:56:23.985598 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:56:23.985400 2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-api-a-ab74b3-serving-cert\"" Sep 10 13:56:23.985598 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:56:23.985419 2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Sep 10 13:56:24.000984 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:56:24.000957 2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-ab74b3-5d985f8f7f-llgx2"] Sep 10 13:56:24.025658 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:56:24.025625 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fxgmp_fa53c783-d40a-4024-9f7b-1d0d2b57f6dc/konnectivity-agent/0.log" limit=1024 Sep 10 13:56:24.082454 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:56:24.082411 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0a9b5f2f-c6eb-4915-9c8f-731e7ca6c3a4-openshift-service-ca\") pod \"maas-api-e2e-api-a-ab74b3-5d985f8f7f-llgx2\" (UID: \"0a9b5f2f-c6eb-4915-9c8f-731e7ca6c3a4\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-ab74b3-5d985f8f7f-llgx2" Sep 10 13:56:24.082664 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:56:24.082506 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6nnhp\" (UniqueName: \"kubernetes.io/projected/0a9b5f2f-c6eb-4915-9c8f-731e7ca6c3a4-kube-api-access-6nnhp\") pod \"maas-api-e2e-api-a-ab74b3-5d985f8f7f-llgx2\" (UID: \"0a9b5f2f-c6eb-4915-9c8f-731e7ca6c3a4\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-ab74b3-5d985f8f7f-llgx2" Sep 10 13:56:24.082664 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:56:24.082576 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0a9b5f2f-c6eb-4915-9c8f-731e7ca6c3a4-metrics-certs\") pod \"maas-api-e2e-api-a-ab74b3-5d985f8f7f-llgx2\" (UID: \"0a9b5f2f-c6eb-4915-9c8f-731e7ca6c3a4\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-ab74b3-5d985f8f7f-llgx2" Sep 10 13:56:24.082664 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:56:24.082614 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0a9b5f2f-c6eb-4915-9c8f-731e7ca6c3a4-maas-api-tls\") pod \"maas-api-e2e-api-a-ab74b3-5d985f8f7f-llgx2\" (UID: \"0a9b5f2f-c6eb-4915-9c8f-731e7ca6c3a4\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-ab74b3-5d985f8f7f-llgx2" Sep 10 13:56:24.083255 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:56:24.083213 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0a9b5f2f-c6eb-4915-9c8f-731e7ca6c3a4-openshift-service-ca\") pod \"maas-api-e2e-api-a-ab74b3-5d985f8f7f-llgx2\" (UID: \"0a9b5f2f-c6eb-4915-9c8f-731e7ca6c3a4\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-ab74b3-5d985f8f7f-llgx2" Sep 10 13:56:24.085674 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:56:24.085647 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0a9b5f2f-c6eb-4915-9c8f-731e7ca6c3a4-metrics-certs\") pod \"maas-api-e2e-api-a-ab74b3-5d985f8f7f-llgx2\" (UID: \"0a9b5f2f-c6eb-4915-9c8f-731e7ca6c3a4\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-ab74b3-5d985f8f7f-llgx2" Sep 10 13:56:24.085762 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:56:24.085726 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0a9b5f2f-c6eb-4915-9c8f-731e7ca6c3a4-maas-api-tls\") pod \"maas-api-e2e-api-a-ab74b3-5d985f8f7f-llgx2\" (UID: \"0a9b5f2f-c6eb-4915-9c8f-731e7ca6c3a4\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-ab74b3-5d985f8f7f-llgx2" Sep 10 13:56:24.093034 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:56:24.093010 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6nnhp\" (UniqueName: \"kubernetes.io/projected/0a9b5f2f-c6eb-4915-9c8f-731e7ca6c3a4-kube-api-access-6nnhp\") pod \"maas-api-e2e-api-a-ab74b3-5d985f8f7f-llgx2\" (UID: \"0a9b5f2f-c6eb-4915-9c8f-731e7ca6c3a4\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-ab74b3-5d985f8f7f-llgx2" Sep 10 13:56:24.127700 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:56:24.127671 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fxgmp_fa53c783-d40a-4024-9f7b-1d0d2b57f6dc/konnectivity-agent/0.log" limit=1024 Sep 10 13:56:24.226325 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:56:24.226217 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fxgmp_fa53c783-d40a-4024-9f7b-1d0d2b57f6dc/konnectivity-agent/0.log" limit=1024 Sep 10 13:56:24.296025 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:56:24.295994 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-ab74b3-5d985f8f7f-llgx2" Sep 10 13:56:24.323653 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:56:24.323613 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fxgmp_fa53c783-d40a-4024-9f7b-1d0d2b57f6dc/konnectivity-agent/0.log" limit=1024 Sep 10 13:56:24.423637 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:56:24.423581 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fxgmp_fa53c783-d40a-4024-9f7b-1d0d2b57f6dc/konnectivity-agent/0.log" limit=1024 Sep 10 13:56:24.571931 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:56:24.571900 2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-ab74b3-5d985f8f7f-llgx2"] Sep 10 13:56:24.573638 ip-10-0-140-8 kubenswrapper[2585]: W0910 13:56:24.573606 2585 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0a9b5f2f_c6eb_4915_9c8f_731e7ca6c3a4.slice/crio-3ed982e1352d73a4a22f8c8080f524992ad5663c140ee1e8419b9196014e2cfb WatchSource:0}: Error finding container 3ed982e1352d73a4a22f8c8080f524992ad5663c140ee1e8419b9196014e2cfb: Status 404 returned error can't find the container with id 3ed982e1352d73a4a22f8c8080f524992ad5663c140ee1e8419b9196014e2cfb Sep 10 13:56:24.797911 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:56:24.797878 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fxgmp_fa53c783-d40a-4024-9f7b-1d0d2b57f6dc/konnectivity-agent/0.log" limit=1024 Sep 10 13:56:24.888513 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:56:24.888431 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-ab74b3-5d985f8f7f-llgx2" event={"ID":"0a9b5f2f-c6eb-4915-9c8f-731e7ca6c3a4","Type":"ContainerStarted","Data":"3ed982e1352d73a4a22f8c8080f524992ad5663c140ee1e8419b9196014e2cfb"} Sep 10 13:56:24.906044 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:56:24.906008 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fxgmp_fa53c783-d40a-4024-9f7b-1d0d2b57f6dc/konnectivity-agent/0.log" limit=1024 Sep 10 13:56:25.895377 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:56:25.895336 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-ab74b3-5d985f8f7f-llgx2" event={"ID":"0a9b5f2f-c6eb-4915-9c8f-731e7ca6c3a4","Type":"ContainerStarted","Data":"fbf2937833dda7d9fda6e2ff0c02f841c67f9cb4fbab7e89524eca88cb2c7c30"} Sep 10 13:56:25.895764 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:56:25.895450 2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-ab74b3-5d985f8f7f-llgx2" Sep 10 13:56:25.918384 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:56:25.918319 2585 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-ab74b3-5d985f8f7f-llgx2" podStartSLOduration=2.561339735 podStartE2EDuration="2.918302886s" totalImagesPullingTime="356.963151ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 13:56:23 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 13:56:25.915755686 +0000 UTC m=+1670.788459878" watchObservedRunningTime="2026-09-10 13:56:25.918302886 +0000 UTC m=+1670.791007086" Sep 10 13:56:31.906570 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:56:31.906535 2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-ab74b3-5d985f8f7f-llgx2" Sep 10 13:56:36.844449 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:56:36.844408 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fxgmp_fa53c783-d40a-4024-9f7b-1d0d2b57f6dc/konnectivity-agent/0.log" limit=1024 Sep 10 13:56:39.751976 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:56:39.751935 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fxgmp_fa53c783-d40a-4024-9f7b-1d0d2b57f6dc/konnectivity-agent/0.log" limit=1024 Sep 10 13:56:46.899609 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:56:46.899572 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fxgmp_fa53c783-d40a-4024-9f7b-1d0d2b57f6dc/konnectivity-agent/0.log" limit=1024 Sep 10 13:56:50.454714 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:56:50.454678 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fxgmp_fa53c783-d40a-4024-9f7b-1d0d2b57f6dc/konnectivity-agent/0.log" limit=1024 Sep 10 13:56:52.704101 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:56:52.704066 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fxgmp_fa53c783-d40a-4024-9f7b-1d0d2b57f6dc/konnectivity-agent/0.log" limit=1024 Sep 10 13:56:53.202572 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:56:53.202534 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fxgmp_fa53c783-d40a-4024-9f7b-1d0d2b57f6dc/konnectivity-agent/0.log" limit=1024 Sep 10 13:56:53.588209 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:56:53.588145 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fxgmp_fa53c783-d40a-4024-9f7b-1d0d2b57f6dc/konnectivity-agent/0.log" limit=1024 Sep 10 13:56:54.206401 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:56:54.206369 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fxgmp_fa53c783-d40a-4024-9f7b-1d0d2b57f6dc/konnectivity-agent/0.log" limit=1024 Sep 10 13:56:56.782411 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:56:56.782378 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fxgmp_fa53c783-d40a-4024-9f7b-1d0d2b57f6dc/konnectivity-agent/0.log" limit=1024 Sep 10 13:56:56.890043 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:56:56.890010 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fxgmp_fa53c783-d40a-4024-9f7b-1d0d2b57f6dc/konnectivity-agent/0.log" limit=1024 Sep 10 13:56:57.975045 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:56:57.974965 2585 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-8aa282-b7f965bb5-ltrdc"] Sep 10 13:56:57.979451 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:56:57.979430 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-8aa282-b7f965bb5-ltrdc" Sep 10 13:56:57.982838 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:56:57.982818 2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-api-b-8aa282-serving-cert\"" Sep 10 13:56:57.993005 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:56:57.992967 2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-8aa282-b7f965bb5-ltrdc"] Sep 10 13:56:58.032532 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:56:58.032507 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fxgmp_fa53c783-d40a-4024-9f7b-1d0d2b57f6dc/konnectivity-agent/0.log" limit=1024 Sep 10 13:56:58.094429 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:56:58.094394 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e2a26243-f93a-4d2c-9d10-a383d79c63a7-maas-api-tls\") pod \"maas-api-e2e-api-b-8aa282-b7f965bb5-ltrdc\" (UID: \"e2a26243-f93a-4d2c-9d10-a383d79c63a7\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-8aa282-b7f965bb5-ltrdc" Sep 10 13:56:58.094616 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:56:58.094449 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4tgxj\" (UniqueName: \"kubernetes.io/projected/e2a26243-f93a-4d2c-9d10-a383d79c63a7-kube-api-access-4tgxj\") pod \"maas-api-e2e-api-b-8aa282-b7f965bb5-ltrdc\" (UID: \"e2a26243-f93a-4d2c-9d10-a383d79c63a7\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-8aa282-b7f965bb5-ltrdc" Sep 10 13:56:58.094616 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:56:58.094485 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e2a26243-f93a-4d2c-9d10-a383d79c63a7-metrics-certs\") pod \"maas-api-e2e-api-b-8aa282-b7f965bb5-ltrdc\" (UID: \"e2a26243-f93a-4d2c-9d10-a383d79c63a7\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-8aa282-b7f965bb5-ltrdc" Sep 10 13:56:58.094616 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:56:58.094582 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e2a26243-f93a-4d2c-9d10-a383d79c63a7-openshift-service-ca\") pod \"maas-api-e2e-api-b-8aa282-b7f965bb5-ltrdc\" (UID: \"e2a26243-f93a-4d2c-9d10-a383d79c63a7\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-8aa282-b7f965bb5-ltrdc" Sep 10 13:56:58.135214 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:56:58.135181 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fxgmp_fa53c783-d40a-4024-9f7b-1d0d2b57f6dc/konnectivity-agent/0.log" limit=1024 Sep 10 13:56:58.195492 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:56:58.195455 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e2a26243-f93a-4d2c-9d10-a383d79c63a7-maas-api-tls\") pod \"maas-api-e2e-api-b-8aa282-b7f965bb5-ltrdc\" (UID: \"e2a26243-f93a-4d2c-9d10-a383d79c63a7\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-8aa282-b7f965bb5-ltrdc" Sep 10 13:56:58.195658 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:56:58.195512 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4tgxj\" (UniqueName: \"kubernetes.io/projected/e2a26243-f93a-4d2c-9d10-a383d79c63a7-kube-api-access-4tgxj\") pod \"maas-api-e2e-api-b-8aa282-b7f965bb5-ltrdc\" (UID: \"e2a26243-f93a-4d2c-9d10-a383d79c63a7\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-8aa282-b7f965bb5-ltrdc" Sep 10 13:56:58.195658 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:56:58.195554 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e2a26243-f93a-4d2c-9d10-a383d79c63a7-metrics-certs\") pod \"maas-api-e2e-api-b-8aa282-b7f965bb5-ltrdc\" (UID: \"e2a26243-f93a-4d2c-9d10-a383d79c63a7\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-8aa282-b7f965bb5-ltrdc" Sep 10 13:56:58.195658 ip-10-0-140-8 kubenswrapper[2585]: E0910 13:56:58.195601 2585 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-api-b-8aa282-serving-cert: secret "maas-api-e2e-api-b-8aa282-serving-cert" not found Sep 10 13:56:58.195658 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:56:58.195625 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e2a26243-f93a-4d2c-9d10-a383d79c63a7-openshift-service-ca\") pod \"maas-api-e2e-api-b-8aa282-b7f965bb5-ltrdc\" (UID: \"e2a26243-f93a-4d2c-9d10-a383d79c63a7\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-8aa282-b7f965bb5-ltrdc" Sep 10 13:56:58.195879 ip-10-0-140-8 kubenswrapper[2585]: E0910 13:56:58.195667 2585 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e2a26243-f93a-4d2c-9d10-a383d79c63a7-maas-api-tls podName:e2a26243-f93a-4d2c-9d10-a383d79c63a7 nodeName:}" failed. No retries permitted until 2026-09-10 13:56:58.695651395 +0000 UTC m=+1703.568355579 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/e2a26243-f93a-4d2c-9d10-a383d79c63a7-maas-api-tls") pod "maas-api-e2e-api-b-8aa282-b7f965bb5-ltrdc" (UID: "e2a26243-f93a-4d2c-9d10-a383d79c63a7") : secret "maas-api-e2e-api-b-8aa282-serving-cert" not found Sep 10 13:56:58.196293 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:56:58.196275 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e2a26243-f93a-4d2c-9d10-a383d79c63a7-openshift-service-ca\") pod \"maas-api-e2e-api-b-8aa282-b7f965bb5-ltrdc\" (UID: \"e2a26243-f93a-4d2c-9d10-a383d79c63a7\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-8aa282-b7f965bb5-ltrdc" Sep 10 13:56:58.198030 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:56:58.198007 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e2a26243-f93a-4d2c-9d10-a383d79c63a7-metrics-certs\") pod \"maas-api-e2e-api-b-8aa282-b7f965bb5-ltrdc\" (UID: \"e2a26243-f93a-4d2c-9d10-a383d79c63a7\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-8aa282-b7f965bb5-ltrdc" Sep 10 13:56:58.210700 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:56:58.210680 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4tgxj\" (UniqueName: \"kubernetes.io/projected/e2a26243-f93a-4d2c-9d10-a383d79c63a7-kube-api-access-4tgxj\") pod \"maas-api-e2e-api-b-8aa282-b7f965bb5-ltrdc\" (UID: \"e2a26243-f93a-4d2c-9d10-a383d79c63a7\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-8aa282-b7f965bb5-ltrdc" Sep 10 13:56:58.305697 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:56:58.305615 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fxgmp_fa53c783-d40a-4024-9f7b-1d0d2b57f6dc/konnectivity-agent/0.log" limit=1024 Sep 10 13:56:58.691897 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:56:58.691864 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fxgmp_fa53c783-d40a-4024-9f7b-1d0d2b57f6dc/konnectivity-agent/0.log" limit=1024 Sep 10 13:56:58.701280 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:56:58.701230 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e2a26243-f93a-4d2c-9d10-a383d79c63a7-maas-api-tls\") pod \"maas-api-e2e-api-b-8aa282-b7f965bb5-ltrdc\" (UID: \"e2a26243-f93a-4d2c-9d10-a383d79c63a7\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-8aa282-b7f965bb5-ltrdc" Sep 10 13:56:58.703678 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:56:58.703652 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e2a26243-f93a-4d2c-9d10-a383d79c63a7-maas-api-tls\") pod \"maas-api-e2e-api-b-8aa282-b7f965bb5-ltrdc\" (UID: \"e2a26243-f93a-4d2c-9d10-a383d79c63a7\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-8aa282-b7f965bb5-ltrdc" Sep 10 13:56:58.793281 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:56:58.793205 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fxgmp_fa53c783-d40a-4024-9f7b-1d0d2b57f6dc/konnectivity-agent/0.log" limit=1024 Sep 10 13:56:58.890082 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:56:58.890044 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-8aa282-b7f965bb5-ltrdc" Sep 10 13:56:59.179879 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:56:59.179849 2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-8aa282-b7f965bb5-ltrdc"] Sep 10 13:56:59.182408 ip-10-0-140-8 kubenswrapper[2585]: W0910 13:56:59.182379 2585 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode2a26243_f93a_4d2c_9d10_a383d79c63a7.slice/crio-58bf5f690da704f9e04289dab75fde75b1959d79333334caa1713a6fbae68a0a WatchSource:0}: Error finding container 58bf5f690da704f9e04289dab75fde75b1959d79333334caa1713a6fbae68a0a: Status 404 returned error can't find the container with id 58bf5f690da704f9e04289dab75fde75b1959d79333334caa1713a6fbae68a0a Sep 10 13:57:00.093312 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:57:00.093279 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-8aa282-b7f965bb5-ltrdc" event={"ID":"e2a26243-f93a-4d2c-9d10-a383d79c63a7","Type":"ContainerStarted","Data":"8457ba804662b31889abc7f308fd9dcb5f4e254cb4e528497fcbcbc272b9b50a"} Sep 10 13:57:00.093497 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:57:00.093331 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-8aa282-b7f965bb5-ltrdc" event={"ID":"e2a26243-f93a-4d2c-9d10-a383d79c63a7","Type":"ContainerStarted","Data":"58bf5f690da704f9e04289dab75fde75b1959d79333334caa1713a6fbae68a0a"} Sep 10 13:57:00.093497 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:57:00.093392 2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-8aa282-b7f965bb5-ltrdc" Sep 10 13:57:00.153133 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:57:00.153081 2585 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-8aa282-b7f965bb5-ltrdc" podStartSLOduration=2.830084929 podStartE2EDuration="3.153064718s" totalImagesPullingTime="322.979789ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 13:56:57 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 13:57:00.152661148 +0000 UTC m=+1705.025365350" watchObservedRunningTime="2026-09-10 13:57:00.153064718 +0000 UTC m=+1705.025768918" Sep 10 13:57:06.106290 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:57:06.106257 2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-8aa282-b7f965bb5-ltrdc" Sep 10 13:57:13.367773 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:57:13.367730 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fxgmp_fa53c783-d40a-4024-9f7b-1d0d2b57f6dc/konnectivity-agent/0.log" limit=1024 Sep 10 13:57:19.453931 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:57:19.453890 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fxgmp_fa53c783-d40a-4024-9f7b-1d0d2b57f6dc/konnectivity-agent/0.log" limit=1024 Sep 10 13:57:20.799153 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:57:20.799123 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fxgmp_fa53c783-d40a-4024-9f7b-1d0d2b57f6dc/konnectivity-agent/0.log" limit=1024 Sep 10 13:57:21.441056 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:57:21.441022 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fxgmp_fa53c783-d40a-4024-9f7b-1d0d2b57f6dc/konnectivity-agent/0.log" limit=1024 Sep 10 13:57:26.851471 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:57:26.851427 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fxgmp_fa53c783-d40a-4024-9f7b-1d0d2b57f6dc/konnectivity-agent/0.log" limit=1024 Sep 10 13:57:29.971560 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:57:29.971521 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fxgmp_fa53c783-d40a-4024-9f7b-1d0d2b57f6dc/konnectivity-agent/0.log" limit=1024 Sep 10 13:57:45.676186 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:57:45.676151 2585 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-ab74b3-5d985f8f7f-llgx2"] Sep 10 13:57:45.676654 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:57:45.676429 2585 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-ab74b3-5d985f8f7f-llgx2" podUID="0a9b5f2f-c6eb-4915-9c8f-731e7ca6c3a4" containerName="maas-api" containerID="cri-o://fbf2937833dda7d9fda6e2ff0c02f841c67f9cb4fbab7e89524eca88cb2c7c30" gracePeriod=30 Sep 10 13:57:45.706925 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:57:45.706895 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fxgmp_fa53c783-d40a-4024-9f7b-1d0d2b57f6dc/konnectivity-agent/0.log" limit=1024 Sep 10 13:57:45.785768 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:57:45.785734 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fxgmp_fa53c783-d40a-4024-9f7b-1d0d2b57f6dc/konnectivity-agent/0.log" limit=1024 Sep 10 13:57:45.788508 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:57:45.788479 2585 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7519-bdd2fc"] Sep 10 13:57:45.793273 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:57:45.793237 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7519-bdd2fc" Sep 10 13:57:45.796096 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:57:45.796076 2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"default-dockercfg-zn6pb\"" Sep 10 13:57:45.796706 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:57:45.796680 2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7519-bdd2fc"] Sep 10 13:57:45.867607 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:57:45.867578 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fxgmp_fa53c783-d40a-4024-9f7b-1d0d2b57f6dc/konnectivity-agent/0.log" limit=1024 Sep 10 13:57:45.942944 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:57:45.942923 2585 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-ab74b3-5d985f8f7f-llgx2" Sep 10 13:57:45.950901 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:57:45.950876 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xk595\" (UniqueName: \"kubernetes.io/projected/63cf7c55-959c-435d-8d7b-cde24c6b3716-kube-api-access-xk595\") pod \"test-curl-7519-bdd2fc\" (UID: \"63cf7c55-959c-435d-8d7b-cde24c6b3716\") " pod="opendatahub/test-curl-7519-bdd2fc" Sep 10 13:57:45.984784 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:57:45.984750 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fxgmp_fa53c783-d40a-4024-9f7b-1d0d2b57f6dc/konnectivity-agent/0.log" limit=1024 Sep 10 13:57:46.051715 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:57:46.051685 2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0a9b5f2f-c6eb-4915-9c8f-731e7ca6c3a4-maas-api-tls\") pod \"0a9b5f2f-c6eb-4915-9c8f-731e7ca6c3a4\" (UID: \"0a9b5f2f-c6eb-4915-9c8f-731e7ca6c3a4\") " Sep 10 13:57:46.051875 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:57:46.051754 2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0a9b5f2f-c6eb-4915-9c8f-731e7ca6c3a4-metrics-certs\") pod \"0a9b5f2f-c6eb-4915-9c8f-731e7ca6c3a4\" (UID: \"0a9b5f2f-c6eb-4915-9c8f-731e7ca6c3a4\") " Sep 10 13:57:46.051875 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:57:46.051838 2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0a9b5f2f-c6eb-4915-9c8f-731e7ca6c3a4-openshift-service-ca\") pod \"0a9b5f2f-c6eb-4915-9c8f-731e7ca6c3a4\" (UID: \"0a9b5f2f-c6eb-4915-9c8f-731e7ca6c3a4\") " Sep 10 13:57:46.052021 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:57:46.051886 2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-6nnhp\" (UniqueName: \"kubernetes.io/projected/0a9b5f2f-c6eb-4915-9c8f-731e7ca6c3a4-kube-api-access-6nnhp\") pod \"0a9b5f2f-c6eb-4915-9c8f-731e7ca6c3a4\" (UID: \"0a9b5f2f-c6eb-4915-9c8f-731e7ca6c3a4\") " Sep 10 13:57:46.052084 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:57:46.052030 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xk595\" (UniqueName: \"kubernetes.io/projected/63cf7c55-959c-435d-8d7b-cde24c6b3716-kube-api-access-xk595\") pod \"test-curl-7519-bdd2fc\" (UID: \"63cf7c55-959c-435d-8d7b-cde24c6b3716\") " pod="opendatahub/test-curl-7519-bdd2fc" Sep 10 13:57:46.052156 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:57:46.052133 2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0a9b5f2f-c6eb-4915-9c8f-731e7ca6c3a4-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "0a9b5f2f-c6eb-4915-9c8f-731e7ca6c3a4" (UID: "0a9b5f2f-c6eb-4915-9c8f-731e7ca6c3a4"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 10 13:57:46.053926 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:57:46.053903 2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0a9b5f2f-c6eb-4915-9c8f-731e7ca6c3a4-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "0a9b5f2f-c6eb-4915-9c8f-731e7ca6c3a4" (UID: "0a9b5f2f-c6eb-4915-9c8f-731e7ca6c3a4"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 10 13:57:46.054017 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:57:46.053996 2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0a9b5f2f-c6eb-4915-9c8f-731e7ca6c3a4-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "0a9b5f2f-c6eb-4915-9c8f-731e7ca6c3a4" (UID: "0a9b5f2f-c6eb-4915-9c8f-731e7ca6c3a4"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 10 13:57:46.054082 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:57:46.054062 2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0a9b5f2f-c6eb-4915-9c8f-731e7ca6c3a4-kube-api-access-6nnhp" (OuterVolumeSpecName: "kube-api-access-6nnhp") pod "0a9b5f2f-c6eb-4915-9c8f-731e7ca6c3a4" (UID: "0a9b5f2f-c6eb-4915-9c8f-731e7ca6c3a4"). InnerVolumeSpecName "kube-api-access-6nnhp". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 10 13:57:46.064143 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:57:46.064097 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fxgmp_fa53c783-d40a-4024-9f7b-1d0d2b57f6dc/konnectivity-agent/0.log" limit=1024 Sep 10 13:57:46.065654 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:57:46.065629 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xk595\" (UniqueName: \"kubernetes.io/projected/63cf7c55-959c-435d-8d7b-cde24c6b3716-kube-api-access-xk595\") pod \"test-curl-7519-bdd2fc\" (UID: \"63cf7c55-959c-435d-8d7b-cde24c6b3716\") " pod="opendatahub/test-curl-7519-bdd2fc" Sep 10 13:57:46.105495 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:57:46.105470 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7519-bdd2fc" Sep 10 13:57:46.153419 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:57:46.153386 2585 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0a9b5f2f-c6eb-4915-9c8f-731e7ca6c3a4-metrics-certs\") on node \"ip-10-0-140-8.ec2.internal\" DevicePath \"\"" Sep 10 13:57:46.153419 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:57:46.153415 2585 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0a9b5f2f-c6eb-4915-9c8f-731e7ca6c3a4-openshift-service-ca\") on node \"ip-10-0-140-8.ec2.internal\" DevicePath \"\"" Sep 10 13:57:46.153419 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:57:46.153426 2585 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6nnhp\" (UniqueName: \"kubernetes.io/projected/0a9b5f2f-c6eb-4915-9c8f-731e7ca6c3a4-kube-api-access-6nnhp\") on node \"ip-10-0-140-8.ec2.internal\" DevicePath \"\"" Sep 10 13:57:46.153730 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:57:46.153435 2585 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0a9b5f2f-c6eb-4915-9c8f-731e7ca6c3a4-maas-api-tls\") on node \"ip-10-0-140-8.ec2.internal\" DevicePath \"\"" Sep 10 13:57:46.349003 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:57:46.348971 2585 generic.go:356] "Generic (PLEG): container finished" podID="0a9b5f2f-c6eb-4915-9c8f-731e7ca6c3a4" containerID="fbf2937833dda7d9fda6e2ff0c02f841c67f9cb4fbab7e89524eca88cb2c7c30" exitCode=0 Sep 10 13:57:46.349193 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:57:46.349012 2585 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-ab74b3-5d985f8f7f-llgx2" Sep 10 13:57:46.349193 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:57:46.349052 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-ab74b3-5d985f8f7f-llgx2" event={"ID":"0a9b5f2f-c6eb-4915-9c8f-731e7ca6c3a4","Type":"ContainerDied","Data":"fbf2937833dda7d9fda6e2ff0c02f841c67f9cb4fbab7e89524eca88cb2c7c30"} Sep 10 13:57:46.349193 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:57:46.349115 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-ab74b3-5d985f8f7f-llgx2" event={"ID":"0a9b5f2f-c6eb-4915-9c8f-731e7ca6c3a4","Type":"ContainerDied","Data":"3ed982e1352d73a4a22f8c8080f524992ad5663c140ee1e8419b9196014e2cfb"} Sep 10 13:57:46.349193 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:57:46.349145 2585 scope.go:117] "RemoveContainer" containerID="fbf2937833dda7d9fda6e2ff0c02f841c67f9cb4fbab7e89524eca88cb2c7c30" Sep 10 13:57:46.358936 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:57:46.358917 2585 scope.go:117] "RemoveContainer" containerID="fbf2937833dda7d9fda6e2ff0c02f841c67f9cb4fbab7e89524eca88cb2c7c30" Sep 10 13:57:46.359221 ip-10-0-140-8 kubenswrapper[2585]: E0910 13:57:46.359203 2585 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fbf2937833dda7d9fda6e2ff0c02f841c67f9cb4fbab7e89524eca88cb2c7c30\": container with ID starting with fbf2937833dda7d9fda6e2ff0c02f841c67f9cb4fbab7e89524eca88cb2c7c30 not found: ID does not exist" containerID="fbf2937833dda7d9fda6e2ff0c02f841c67f9cb4fbab7e89524eca88cb2c7c30" Sep 10 13:57:46.359334 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:57:46.359229 2585 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fbf2937833dda7d9fda6e2ff0c02f841c67f9cb4fbab7e89524eca88cb2c7c30"} err="failed to get container status \"fbf2937833dda7d9fda6e2ff0c02f841c67f9cb4fbab7e89524eca88cb2c7c30\": rpc error: code = NotFound desc = could not find container \"fbf2937833dda7d9fda6e2ff0c02f841c67f9cb4fbab7e89524eca88cb2c7c30\": container with ID starting with fbf2937833dda7d9fda6e2ff0c02f841c67f9cb4fbab7e89524eca88cb2c7c30 not found: ID does not exist" Sep 10 13:57:46.385454 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:57:46.385428 2585 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-ab74b3-5d985f8f7f-llgx2"] Sep 10 13:57:46.390521 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:57:46.390493 2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7519-bdd2fc"] Sep 10 13:57:46.392010 ip-10-0-140-8 kubenswrapper[2585]: W0910 13:57:46.391987 2585 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod63cf7c55_959c_435d_8d7b_cde24c6b3716.slice/crio-34956a229ea955cdb10dcecd114b1c74eb8fcd788d9243fd19511dd34d292312 WatchSource:0}: Error finding container 34956a229ea955cdb10dcecd114b1c74eb8fcd788d9243fd19511dd34d292312: Status 404 returned error can't find the container with id 34956a229ea955cdb10dcecd114b1c74eb8fcd788d9243fd19511dd34d292312 Sep 10 13:57:46.395384 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:57:46.395358 2585 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-ab74b3-5d985f8f7f-llgx2"] Sep 10 13:57:46.403855 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:57:46.403833 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fxgmp_fa53c783-d40a-4024-9f7b-1d0d2b57f6dc/konnectivity-agent/0.log" limit=1024 Sep 10 13:57:47.357337 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:57:47.357299 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7519-bdd2fc" event={"ID":"63cf7c55-959c-435d-8d7b-cde24c6b3716","Type":"ContainerStarted","Data":"964d89c28e3207222b3e9f58772730ef84cfec013ad1a5f512218753343bde69"} Sep 10 13:57:47.357337 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:57:47.357342 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7519-bdd2fc" event={"ID":"63cf7c55-959c-435d-8d7b-cde24c6b3716","Type":"ContainerStarted","Data":"34956a229ea955cdb10dcecd114b1c74eb8fcd788d9243fd19511dd34d292312"} Sep 10 13:57:47.388435 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:57:47.388386 2585 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/test-curl-7519-bdd2fc" podStartSLOduration=2.388370057 podStartE2EDuration="2.388370057s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 13:57:45 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 13:57:47.385610754 +0000 UTC m=+1752.258314959" watchObservedRunningTime="2026-09-10 13:57:47.388370057 +0000 UTC m=+1752.261074269" Sep 10 13:57:47.647396 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:57:47.647308 2585 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="0a9b5f2f-c6eb-4915-9c8f-731e7ca6c3a4" path="/var/lib/kubelet/pods/0a9b5f2f-c6eb-4915-9c8f-731e7ca6c3a4/volumes" Sep 10 13:57:48.072160 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:57:48.072121 2585 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7519-bdd2fc"] Sep 10 13:57:48.080119 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:57:48.080088 2585 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7519-bdd2fc"] Sep 10 13:57:48.199375 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:57:48.199337 2585 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7519-57fb2b"] Sep 10 13:57:48.199801 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:57:48.199787 2585 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0a9b5f2f-c6eb-4915-9c8f-731e7ca6c3a4" containerName="maas-api" Sep 10 13:57:48.199856 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:57:48.199803 2585 state_mem.go:107] "Deleted CPUSet assignment" podUID="0a9b5f2f-c6eb-4915-9c8f-731e7ca6c3a4" containerName="maas-api" Sep 10 13:57:48.199856 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:57:48.199816 2585 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="63cf7c55-959c-435d-8d7b-cde24c6b3716" containerName="test-curl-7519-bdd2fc" Sep 10 13:57:48.199856 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:57:48.199822 2585 state_mem.go:107] "Deleted CPUSet assignment" podUID="63cf7c55-959c-435d-8d7b-cde24c6b3716" containerName="test-curl-7519-bdd2fc" Sep 10 13:57:48.199962 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:57:48.199885 2585 memory_manager.go:368] "RemoveStaleState removing state" podUID="0a9b5f2f-c6eb-4915-9c8f-731e7ca6c3a4" containerName="maas-api" Sep 10 13:57:48.199962 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:57:48.199895 2585 memory_manager.go:368] "RemoveStaleState removing state" podUID="63cf7c55-959c-435d-8d7b-cde24c6b3716" containerName="test-curl-7519-bdd2fc" Sep 10 13:57:48.203276 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:57:48.203261 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7519-57fb2b" Sep 10 13:57:48.217432 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:57:48.217406 2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7519-57fb2b"] Sep 10 13:57:48.363218 ip-10-0-140-8 kubenswrapper[2585]: E0910 13:57:48.363138 2585 kuberuntime_manager.go:649] "Missing actuated resource record" pod="opendatahub/test-curl-7519-bdd2fc" container="test-curl-7519-bdd2fc" Sep 10 13:57:48.376826 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:57:48.376786 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zkmg9\" (UniqueName: \"kubernetes.io/projected/7c269a6a-c2f7-4076-94af-aa59429278fc-kube-api-access-zkmg9\") pod \"test-curl-7519-57fb2b\" (UID: \"7c269a6a-c2f7-4076-94af-aa59429278fc\") " pod="opendatahub/test-curl-7519-57fb2b" Sep 10 13:57:48.478263 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:57:48.478213 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zkmg9\" (UniqueName: \"kubernetes.io/projected/7c269a6a-c2f7-4076-94af-aa59429278fc-kube-api-access-zkmg9\") pod \"test-curl-7519-57fb2b\" (UID: \"7c269a6a-c2f7-4076-94af-aa59429278fc\") " pod="opendatahub/test-curl-7519-57fb2b" Sep 10 13:57:48.488561 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:57:48.488536 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-zkmg9\" (UniqueName: \"kubernetes.io/projected/7c269a6a-c2f7-4076-94af-aa59429278fc-kube-api-access-zkmg9\") pod \"test-curl-7519-57fb2b\" (UID: \"7c269a6a-c2f7-4076-94af-aa59429278fc\") " pod="opendatahub/test-curl-7519-57fb2b" Sep 10 13:57:48.512520 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:57:48.512496 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7519-57fb2b" Sep 10 13:57:48.785664 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:57:48.785640 2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7519-57fb2b"] Sep 10 13:57:48.787506 ip-10-0-140-8 kubenswrapper[2585]: W0910 13:57:48.787476 2585 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod7c269a6a_c2f7_4076_94af_aa59429278fc.slice/crio-d0316e0ecc5f5efbc1bef41174bd9689d2c82a6e8d5318ee0068c083441e24fd WatchSource:0}: Error finding container d0316e0ecc5f5efbc1bef41174bd9689d2c82a6e8d5318ee0068c083441e24fd: Status 404 returned error can't find the container with id d0316e0ecc5f5efbc1bef41174bd9689d2c82a6e8d5318ee0068c083441e24fd Sep 10 13:57:49.370457 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:57:49.370420 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7519-57fb2b" event={"ID":"7c269a6a-c2f7-4076-94af-aa59429278fc","Type":"ContainerStarted","Data":"340b8f40f989d37854a0122dd5f0df022a61be08ce5f59034fe6f369eea6fdb8"} Sep 10 13:57:49.370957 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:57:49.370472 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7519-57fb2b" event={"ID":"7c269a6a-c2f7-4076-94af-aa59429278fc","Type":"ContainerStarted","Data":"d0316e0ecc5f5efbc1bef41174bd9689d2c82a6e8d5318ee0068c083441e24fd"} Sep 10 13:57:49.370957 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:57:49.370531 2585 kuberuntime_container.go:873] "Killing container with a grace period" pod="opendatahub/test-curl-7519-bdd2fc" podUID="63cf7c55-959c-435d-8d7b-cde24c6b3716" containerName="test-curl-7519-bdd2fc" containerID="cri-o://964d89c28e3207222b3e9f58772730ef84cfec013ad1a5f512218753343bde69" gracePeriod=2 Sep 10 13:57:49.389880 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:57:49.389826 2585 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/test-curl-7519-57fb2b" podStartSLOduration=1.3898100310000001 podStartE2EDuration="1.389810031s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 13:57:48 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 13:57:49.388847757 +0000 UTC m=+1754.261551960" watchObservedRunningTime="2026-09-10 13:57:49.389810031 +0000 UTC m=+1754.262514231" Sep 10 13:57:49.395503 ip-10-0-140-8 kubenswrapper[2585]: E0910 13:57:49.395471 2585 status_manager.go:1018] "Failed to get status for pod" err="pods \"test-curl-7519-bdd2fc\" is forbidden: User \"system:node:ip-10-0-140-8.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-140-8.ec2.internal' and this object" podUID="63cf7c55-959c-435d-8d7b-cde24c6b3716" pod="opendatahub/test-curl-7519-bdd2fc" Sep 10 13:57:49.399447 ip-10-0-140-8 kubenswrapper[2585]: E0910 13:57:49.399415 2585 status_manager.go:1018] "Failed to get status for pod" err="pods \"test-curl-7519-bdd2fc\" is forbidden: User \"system:node:ip-10-0-140-8.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-140-8.ec2.internal' and this object" podUID="63cf7c55-959c-435d-8d7b-cde24c6b3716" pod="opendatahub/test-curl-7519-bdd2fc" Sep 10 13:57:50.087838 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:57:50.087801 2585 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7519-57fb2b"] Sep 10 13:57:50.092773 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:57:50.092744 2585 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7519-57fb2b"] Sep 10 13:57:50.376219 ip-10-0-140-8 kubenswrapper[2585]: E0910 13:57:50.376135 2585 kuberuntime_manager.go:649] "Missing actuated resource record" pod="opendatahub/test-curl-7519-57fb2b" container="test-curl-7519-57fb2b" Sep 10 13:57:51.382566 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:57:51.382535 2585 kuberuntime_container.go:873] "Killing container with a grace period" pod="opendatahub/test-curl-7519-57fb2b" podUID="7c269a6a-c2f7-4076-94af-aa59429278fc" containerName="test-curl-7519-57fb2b" containerID="cri-o://340b8f40f989d37854a0122dd5f0df022a61be08ce5f59034fe6f369eea6fdb8" gracePeriod=2 Sep 10 13:57:51.386899 ip-10-0-140-8 kubenswrapper[2585]: E0910 13:57:51.386865 2585 status_manager.go:1018] "Failed to get status for pod" err="pods \"test-curl-7519-57fb2b\" is forbidden: User \"system:node:ip-10-0-140-8.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-140-8.ec2.internal' and this object" podUID="7c269a6a-c2f7-4076-94af-aa59429278fc" pod="opendatahub/test-curl-7519-57fb2b" Sep 10 13:57:51.564361 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:57:51.564339 2585 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7519-bdd2fc" Sep 10 13:57:51.568561 ip-10-0-140-8 kubenswrapper[2585]: E0910 13:57:51.568534 2585 status_manager.go:1018] "Failed to get status for pod" err="pods \"test-curl-7519-57fb2b\" is forbidden: User \"system:node:ip-10-0-140-8.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-140-8.ec2.internal' and this object" podUID="7c269a6a-c2f7-4076-94af-aa59429278fc" pod="opendatahub/test-curl-7519-57fb2b" Sep 10 13:57:51.604939 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:57:51.604909 2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-xk595\" (UniqueName: \"kubernetes.io/projected/63cf7c55-959c-435d-8d7b-cde24c6b3716-kube-api-access-xk595\") pod \"63cf7c55-959c-435d-8d7b-cde24c6b3716\" (UID: \"63cf7c55-959c-435d-8d7b-cde24c6b3716\") " Sep 10 13:57:51.606871 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:57:51.606849 2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/63cf7c55-959c-435d-8d7b-cde24c6b3716-kube-api-access-xk595" (OuterVolumeSpecName: "kube-api-access-xk595") pod "63cf7c55-959c-435d-8d7b-cde24c6b3716" (UID: "63cf7c55-959c-435d-8d7b-cde24c6b3716"). InnerVolumeSpecName "kube-api-access-xk595". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 10 13:57:51.647913 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:57:51.647841 2585 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="63cf7c55-959c-435d-8d7b-cde24c6b3716" path="/var/lib/kubelet/pods/63cf7c55-959c-435d-8d7b-cde24c6b3716/volumes" Sep 10 13:57:51.705725 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:57:51.705697 2585 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xk595\" (UniqueName: \"kubernetes.io/projected/63cf7c55-959c-435d-8d7b-cde24c6b3716-kube-api-access-xk595\") on node \"ip-10-0-140-8.ec2.internal\" DevicePath \"\"" Sep 10 13:57:52.395694 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:57:52.395650 2585 generic.go:356] "Generic (PLEG): container finished" podID="63cf7c55-959c-435d-8d7b-cde24c6b3716" containerID="964d89c28e3207222b3e9f58772730ef84cfec013ad1a5f512218753343bde69" exitCode=137 Sep 10 13:57:52.396195 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:57:52.395706 2585 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7519-bdd2fc" Sep 10 13:57:52.396195 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:57:52.395749 2585 scope.go:117] "RemoveContainer" containerID="964d89c28e3207222b3e9f58772730ef84cfec013ad1a5f512218753343bde69" Sep 10 13:57:52.404662 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:57:52.404632 2585 scope.go:117] "RemoveContainer" containerID="964d89c28e3207222b3e9f58772730ef84cfec013ad1a5f512218753343bde69" Sep 10 13:57:52.404934 ip-10-0-140-8 kubenswrapper[2585]: E0910 13:57:52.404918 2585 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"964d89c28e3207222b3e9f58772730ef84cfec013ad1a5f512218753343bde69\": container with ID starting with 964d89c28e3207222b3e9f58772730ef84cfec013ad1a5f512218753343bde69 not found: ID does not exist" containerID="964d89c28e3207222b3e9f58772730ef84cfec013ad1a5f512218753343bde69" Sep 10 13:57:52.404975 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:57:52.404947 2585 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"964d89c28e3207222b3e9f58772730ef84cfec013ad1a5f512218753343bde69"} err="failed to get container status \"964d89c28e3207222b3e9f58772730ef84cfec013ad1a5f512218753343bde69\": rpc error: code = NotFound desc = could not find container \"964d89c28e3207222b3e9f58772730ef84cfec013ad1a5f512218753343bde69\": container with ID starting with 964d89c28e3207222b3e9f58772730ef84cfec013ad1a5f512218753343bde69 not found: ID does not exist" Sep 10 13:57:52.467619 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:57:52.467585 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fxgmp_fa53c783-d40a-4024-9f7b-1d0d2b57f6dc/konnectivity-agent/0.log" limit=1024 Sep 10 13:57:53.344746 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:57:53.344714 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fxgmp_fa53c783-d40a-4024-9f7b-1d0d2b57f6dc/konnectivity-agent/0.log" limit=1024 Sep 10 13:57:53.582867 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:57:53.582842 2585 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7519-57fb2b" Sep 10 13:57:53.622701 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:57:53.622613 2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-zkmg9\" (UniqueName: \"kubernetes.io/projected/7c269a6a-c2f7-4076-94af-aa59429278fc-kube-api-access-zkmg9\") pod \"7c269a6a-c2f7-4076-94af-aa59429278fc\" (UID: \"7c269a6a-c2f7-4076-94af-aa59429278fc\") " Sep 10 13:57:53.624591 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:57:53.624567 2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7c269a6a-c2f7-4076-94af-aa59429278fc-kube-api-access-zkmg9" (OuterVolumeSpecName: "kube-api-access-zkmg9") pod "7c269a6a-c2f7-4076-94af-aa59429278fc" (UID: "7c269a6a-c2f7-4076-94af-aa59429278fc"). InnerVolumeSpecName "kube-api-access-zkmg9". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 10 13:57:53.647514 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:57:53.647485 2585 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="7c269a6a-c2f7-4076-94af-aa59429278fc" path="/var/lib/kubelet/pods/7c269a6a-c2f7-4076-94af-aa59429278fc/volumes" Sep 10 13:57:53.724060 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:57:53.724023 2585 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zkmg9\" (UniqueName: \"kubernetes.io/projected/7c269a6a-c2f7-4076-94af-aa59429278fc-kube-api-access-zkmg9\") on node \"ip-10-0-140-8.ec2.internal\" DevicePath \"\"" Sep 10 13:57:54.409316 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:57:54.409278 2585 generic.go:356] "Generic (PLEG): container finished" podID="7c269a6a-c2f7-4076-94af-aa59429278fc" containerID="340b8f40f989d37854a0122dd5f0df022a61be08ce5f59034fe6f369eea6fdb8" exitCode=137 Sep 10 13:57:54.409502 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:57:54.409324 2585 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7519-57fb2b" Sep 10 13:57:54.409502 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:57:54.409333 2585 scope.go:117] "RemoveContainer" containerID="340b8f40f989d37854a0122dd5f0df022a61be08ce5f59034fe6f369eea6fdb8" Sep 10 13:57:54.418128 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:57:54.418096 2585 scope.go:117] "RemoveContainer" containerID="340b8f40f989d37854a0122dd5f0df022a61be08ce5f59034fe6f369eea6fdb8" Sep 10 13:57:54.418406 ip-10-0-140-8 kubenswrapper[2585]: E0910 13:57:54.418387 2585 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"340b8f40f989d37854a0122dd5f0df022a61be08ce5f59034fe6f369eea6fdb8\": container with ID starting with 340b8f40f989d37854a0122dd5f0df022a61be08ce5f59034fe6f369eea6fdb8 not found: ID does not exist" containerID="340b8f40f989d37854a0122dd5f0df022a61be08ce5f59034fe6f369eea6fdb8" Sep 10 13:57:54.418466 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:57:54.418415 2585 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"340b8f40f989d37854a0122dd5f0df022a61be08ce5f59034fe6f369eea6fdb8"} err="failed to get container status \"340b8f40f989d37854a0122dd5f0df022a61be08ce5f59034fe6f369eea6fdb8\": rpc error: code = NotFound desc = could not find container \"340b8f40f989d37854a0122dd5f0df022a61be08ce5f59034fe6f369eea6fdb8\": container with ID starting with 340b8f40f989d37854a0122dd5f0df022a61be08ce5f59034fe6f369eea6fdb8 not found: ID does not exist" Sep 10 13:57:54.473137 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:57:54.473105 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fxgmp_fa53c783-d40a-4024-9f7b-1d0d2b57f6dc/konnectivity-agent/0.log" limit=1024 Sep 10 13:57:54.572359 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:57:54.572332 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fxgmp_fa53c783-d40a-4024-9f7b-1d0d2b57f6dc/konnectivity-agent/0.log" limit=1024 Sep 10 13:57:54.716842 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:57:54.716754 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fxgmp_fa53c783-d40a-4024-9f7b-1d0d2b57f6dc/konnectivity-agent/0.log" limit=1024 Sep 10 13:57:55.391289 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:57:55.391239 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fxgmp_fa53c783-d40a-4024-9f7b-1d0d2b57f6dc/konnectivity-agent/0.log" limit=1024 Sep 10 13:58:10.561021 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:58:10.560980 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fxgmp_fa53c783-d40a-4024-9f7b-1d0d2b57f6dc/konnectivity-agent/0.log" limit=1024 Sep 10 13:58:10.959182 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:58:10.959106 2585 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-8aa282-b7f965bb5-ltrdc"] Sep 10 13:58:10.959403 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:58:10.959368 2585 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-8aa282-b7f965bb5-ltrdc" podUID="e2a26243-f93a-4d2c-9d10-a383d79c63a7" containerName="maas-api" containerID="cri-o://8457ba804662b31889abc7f308fd9dcb5f4e254cb4e528497fcbcbc272b9b50a" gracePeriod=30 Sep 10 13:58:10.996983 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:58:10.996946 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fxgmp_fa53c783-d40a-4024-9f7b-1d0d2b57f6dc/konnectivity-agent/0.log" limit=1024 Sep 10 13:58:11.090360 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:58:11.090322 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fxgmp_fa53c783-d40a-4024-9f7b-1d0d2b57f6dc/konnectivity-agent/0.log" limit=1024 Sep 10 13:58:11.101735 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:58:11.101701 2585 prober.go:120] "Probe failed" probeType="Readiness" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-8aa282-b7f965bb5-ltrdc" podUID="e2a26243-f93a-4d2c-9d10-a383d79c63a7" containerName="maas-api" probeResult="failure" output="Get \"https://10.133.0.75:8443/health\": dial tcp 10.133.0.75:8443: connect: connection refused" Sep 10 13:58:11.191664 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:58:11.191639 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fxgmp_fa53c783-d40a-4024-9f7b-1d0d2b57f6dc/konnectivity-agent/0.log" limit=1024 Sep 10 13:58:11.222636 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:58:11.222569 2585 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-8aa282-b7f965bb5-ltrdc" Sep 10 13:58:11.285290 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:58:11.285230 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fxgmp_fa53c783-d40a-4024-9f7b-1d0d2b57f6dc/konnectivity-agent/0.log" limit=1024 Sep 10 13:58:11.299323 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:58:11.299296 2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-4tgxj\" (UniqueName: \"kubernetes.io/projected/e2a26243-f93a-4d2c-9d10-a383d79c63a7-kube-api-access-4tgxj\") pod \"e2a26243-f93a-4d2c-9d10-a383d79c63a7\" (UID: \"e2a26243-f93a-4d2c-9d10-a383d79c63a7\") " Sep 10 13:58:11.299433 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:58:11.299354 2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e2a26243-f93a-4d2c-9d10-a383d79c63a7-maas-api-tls\") pod \"e2a26243-f93a-4d2c-9d10-a383d79c63a7\" (UID: \"e2a26243-f93a-4d2c-9d10-a383d79c63a7\") " Sep 10 13:58:11.299433 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:58:11.299406 2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e2a26243-f93a-4d2c-9d10-a383d79c63a7-metrics-certs\") pod \"e2a26243-f93a-4d2c-9d10-a383d79c63a7\" (UID: \"e2a26243-f93a-4d2c-9d10-a383d79c63a7\") " Sep 10 13:58:11.299521 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:58:11.299434 2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e2a26243-f93a-4d2c-9d10-a383d79c63a7-openshift-service-ca\") pod \"e2a26243-f93a-4d2c-9d10-a383d79c63a7\" (UID: \"e2a26243-f93a-4d2c-9d10-a383d79c63a7\") " Sep 10 13:58:11.299847 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:58:11.299824 2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e2a26243-f93a-4d2c-9d10-a383d79c63a7-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "e2a26243-f93a-4d2c-9d10-a383d79c63a7" (UID: "e2a26243-f93a-4d2c-9d10-a383d79c63a7"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 10 13:58:11.301611 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:58:11.301585 2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e2a26243-f93a-4d2c-9d10-a383d79c63a7-kube-api-access-4tgxj" (OuterVolumeSpecName: "kube-api-access-4tgxj") pod "e2a26243-f93a-4d2c-9d10-a383d79c63a7" (UID: "e2a26243-f93a-4d2c-9d10-a383d79c63a7"). InnerVolumeSpecName "kube-api-access-4tgxj". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 10 13:58:11.301698 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:58:11.301644 2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e2a26243-f93a-4d2c-9d10-a383d79c63a7-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "e2a26243-f93a-4d2c-9d10-a383d79c63a7" (UID: "e2a26243-f93a-4d2c-9d10-a383d79c63a7"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 10 13:58:11.301698 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:58:11.301664 2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e2a26243-f93a-4d2c-9d10-a383d79c63a7-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "e2a26243-f93a-4d2c-9d10-a383d79c63a7" (UID: "e2a26243-f93a-4d2c-9d10-a383d79c63a7"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 10 13:58:11.400472 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:58:11.400436 2585 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4tgxj\" (UniqueName: \"kubernetes.io/projected/e2a26243-f93a-4d2c-9d10-a383d79c63a7-kube-api-access-4tgxj\") on node \"ip-10-0-140-8.ec2.internal\" DevicePath \"\"" Sep 10 13:58:11.400472 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:58:11.400467 2585 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e2a26243-f93a-4d2c-9d10-a383d79c63a7-maas-api-tls\") on node \"ip-10-0-140-8.ec2.internal\" DevicePath \"\"" Sep 10 13:58:11.400472 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:58:11.400476 2585 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e2a26243-f93a-4d2c-9d10-a383d79c63a7-metrics-certs\") on node \"ip-10-0-140-8.ec2.internal\" DevicePath \"\"" Sep 10 13:58:11.400472 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:58:11.400484 2585 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e2a26243-f93a-4d2c-9d10-a383d79c63a7-openshift-service-ca\") on node \"ip-10-0-140-8.ec2.internal\" DevicePath \"\"" Sep 10 13:58:11.507704 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:58:11.507616 2585 generic.go:356] "Generic (PLEG): container finished" podID="e2a26243-f93a-4d2c-9d10-a383d79c63a7" containerID="8457ba804662b31889abc7f308fd9dcb5f4e254cb4e528497fcbcbc272b9b50a" exitCode=0 Sep 10 13:58:11.507704 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:58:11.507659 2585 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-8aa282-b7f965bb5-ltrdc" Sep 10 13:58:11.507941 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:58:11.507707 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-8aa282-b7f965bb5-ltrdc" event={"ID":"e2a26243-f93a-4d2c-9d10-a383d79c63a7","Type":"ContainerDied","Data":"8457ba804662b31889abc7f308fd9dcb5f4e254cb4e528497fcbcbc272b9b50a"} Sep 10 13:58:11.507941 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:58:11.507770 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-8aa282-b7f965bb5-ltrdc" event={"ID":"e2a26243-f93a-4d2c-9d10-a383d79c63a7","Type":"ContainerDied","Data":"58bf5f690da704f9e04289dab75fde75b1959d79333334caa1713a6fbae68a0a"} Sep 10 13:58:11.507941 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:58:11.507799 2585 scope.go:117] "RemoveContainer" containerID="8457ba804662b31889abc7f308fd9dcb5f4e254cb4e528497fcbcbc272b9b50a" Sep 10 13:58:11.517225 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:58:11.517208 2585 scope.go:117] "RemoveContainer" containerID="8457ba804662b31889abc7f308fd9dcb5f4e254cb4e528497fcbcbc272b9b50a" Sep 10 13:58:11.517574 ip-10-0-140-8 kubenswrapper[2585]: E0910 13:58:11.517552 2585 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8457ba804662b31889abc7f308fd9dcb5f4e254cb4e528497fcbcbc272b9b50a\": container with ID starting with 8457ba804662b31889abc7f308fd9dcb5f4e254cb4e528497fcbcbc272b9b50a not found: ID does not exist" containerID="8457ba804662b31889abc7f308fd9dcb5f4e254cb4e528497fcbcbc272b9b50a" Sep 10 13:58:11.517655 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:58:11.517581 2585 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8457ba804662b31889abc7f308fd9dcb5f4e254cb4e528497fcbcbc272b9b50a"} err="failed to get container status \"8457ba804662b31889abc7f308fd9dcb5f4e254cb4e528497fcbcbc272b9b50a\": rpc error: code = NotFound desc = could not find container \"8457ba804662b31889abc7f308fd9dcb5f4e254cb4e528497fcbcbc272b9b50a\": container with ID starting with 8457ba804662b31889abc7f308fd9dcb5f4e254cb4e528497fcbcbc272b9b50a not found: ID does not exist" Sep 10 13:58:11.543068 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:58:11.543038 2585 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-8aa282-b7f965bb5-ltrdc"] Sep 10 13:58:11.562555 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:58:11.562534 2585 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-8aa282-b7f965bb5-ltrdc"] Sep 10 13:58:11.647099 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:58:11.647067 2585 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="e2a26243-f93a-4d2c-9d10-a383d79c63a7" path="/var/lib/kubelet/pods/e2a26243-f93a-4d2c-9d10-a383d79c63a7/volumes" Sep 10 13:58:11.707748 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:58:11.707715 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fxgmp_fa53c783-d40a-4024-9f7b-1d0d2b57f6dc/konnectivity-agent/0.log" limit=1024 Sep 10 13:58:18.770188 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:58:18.770150 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fxgmp_fa53c783-d40a-4024-9f7b-1d0d2b57f6dc/konnectivity-agent/0.log" limit=1024 Sep 10 13:58:18.871035 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:58:18.870987 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fxgmp_fa53c783-d40a-4024-9f7b-1d0d2b57f6dc/konnectivity-agent/0.log" limit=1024 Sep 10 13:58:19.097913 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:58:19.097882 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fxgmp_fa53c783-d40a-4024-9f7b-1d0d2b57f6dc/konnectivity-agent/0.log" limit=1024 Sep 10 13:58:19.406548 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:58:19.406467 2585 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-83c318-tsscb"] Sep 10 13:58:19.406903 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:58:19.406890 2585 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7c269a6a-c2f7-4076-94af-aa59429278fc" containerName="test-curl-7519-57fb2b" Sep 10 13:58:19.406945 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:58:19.406904 2585 state_mem.go:107] "Deleted CPUSet assignment" podUID="7c269a6a-c2f7-4076-94af-aa59429278fc" containerName="test-curl-7519-57fb2b" Sep 10 13:58:19.406945 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:58:19.406917 2585 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e2a26243-f93a-4d2c-9d10-a383d79c63a7" containerName="maas-api" Sep 10 13:58:19.406945 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:58:19.406923 2585 state_mem.go:107] "Deleted CPUSet assignment" podUID="e2a26243-f93a-4d2c-9d10-a383d79c63a7" containerName="maas-api" Sep 10 13:58:19.407043 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:58:19.406989 2585 memory_manager.go:368] "RemoveStaleState removing state" podUID="7c269a6a-c2f7-4076-94af-aa59429278fc" containerName="test-curl-7519-57fb2b" Sep 10 13:58:19.407043 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:58:19.407000 2585 memory_manager.go:368] "RemoveStaleState removing state" podUID="e2a26243-f93a-4d2c-9d10-a383d79c63a7" containerName="maas-api" Sep 10 13:58:19.411152 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:58:19.411136 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-83c318-tsscb" Sep 10 13:58:19.414189 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:58:19.414165 2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-7gcbr\"" Sep 10 13:58:19.414189 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:58:19.414180 2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Sep 10 13:58:19.419568 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:58:19.419541 2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-83c318-tsscb"] Sep 10 13:58:19.557901 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:58:19.557871 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fxgmp_fa53c783-d40a-4024-9f7b-1d0d2b57f6dc/konnectivity-agent/0.log" limit=1024 Sep 10 13:58:19.584081 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:58:19.584050 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/5714cced-05e6-4862-851e-c9d3db13bfc0-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w2-83c318-tsscb\" (UID: \"5714cced-05e6-4862-851e-c9d3db13bfc0\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-83c318-tsscb" Sep 10 13:58:19.685292 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:58:19.685192 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/5714cced-05e6-4862-851e-c9d3db13bfc0-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w2-83c318-tsscb\" (UID: \"5714cced-05e6-4862-851e-c9d3db13bfc0\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-83c318-tsscb" Sep 10 13:58:19.685911 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:58:19.685893 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/5714cced-05e6-4862-851e-c9d3db13bfc0-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w2-83c318-tsscb\" (UID: \"5714cced-05e6-4862-851e-c9d3db13bfc0\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-83c318-tsscb" Sep 10 13:58:19.727187 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:58:19.727147 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-83c318-tsscb" Sep 10 13:58:20.012281 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:58:20.012235 2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-83c318-tsscb"] Sep 10 13:58:20.013784 ip-10-0-140-8 kubenswrapper[2585]: W0910 13:58:20.013751 2585 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5714cced_05e6_4862_851e_c9d3db13bfc0.slice/crio-5cd3de39cf7fac8b016834f8105c8f3c6b22022f85a02f4f84224c9d62abdc39 WatchSource:0}: Error finding container 5cd3de39cf7fac8b016834f8105c8f3c6b22022f85a02f4f84224c9d62abdc39: Status 404 returned error can't find the container with id 5cd3de39cf7fac8b016834f8105c8f3c6b22022f85a02f4f84224c9d62abdc39 Sep 10 13:58:20.560389 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:58:20.560347 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-83c318-tsscb" event={"ID":"5714cced-05e6-4862-851e-c9d3db13bfc0","Type":"ContainerStarted","Data":"8f8db4f2a01e38c271fab73383b262367d5613b679f1cc7a98b045b56446fe8b"} Sep 10 13:58:20.560588 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:58:20.560409 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-83c318-tsscb" event={"ID":"5714cced-05e6-4862-851e-c9d3db13bfc0","Type":"ContainerStarted","Data":"5cd3de39cf7fac8b016834f8105c8f3c6b22022f85a02f4f84224c9d62abdc39"} Sep 10 13:58:20.578008 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:58:20.577944 2585 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-83c318-tsscb" podStartSLOduration=1.577928033 podStartE2EDuration="1.577928033s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 13:58:19 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 13:58:20.57745043 +0000 UTC m=+1785.450154632" watchObservedRunningTime="2026-09-10 13:58:20.577928033 +0000 UTC m=+1785.450632234" Sep 10 13:58:21.567164 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:58:21.567130 2585 generic.go:356] "Generic (PLEG): container finished" podID="5714cced-05e6-4862-851e-c9d3db13bfc0" containerID="8f8db4f2a01e38c271fab73383b262367d5613b679f1cc7a98b045b56446fe8b" exitCode=0 Sep 10 13:58:21.567619 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:58:21.567211 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-83c318-tsscb" event={"ID":"5714cced-05e6-4862-851e-c9d3db13bfc0","Type":"ContainerDied","Data":"8f8db4f2a01e38c271fab73383b262367d5613b679f1cc7a98b045b56446fe8b"} Sep 10 13:58:22.707319 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:58:22.707296 2585 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-83c318-tsscb" Sep 10 13:58:22.817573 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:58:22.817542 2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/5714cced-05e6-4862-851e-c9d3db13bfc0-maas-api-service-ca\") pod \"5714cced-05e6-4862-851e-c9d3db13bfc0\" (UID: \"5714cced-05e6-4862-851e-c9d3db13bfc0\") " Sep 10 13:58:22.817940 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:58:22.817917 2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5714cced-05e6-4862-851e-c9d3db13bfc0-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "5714cced-05e6-4862-851e-c9d3db13bfc0" (UID: "5714cced-05e6-4862-851e-c9d3db13bfc0"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 10 13:58:22.919403 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:58:22.919315 2585 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/5714cced-05e6-4862-851e-c9d3db13bfc0-maas-api-service-ca\") on node \"ip-10-0-140-8.ec2.internal\" DevicePath \"\"" Sep 10 13:58:23.580703 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:58:23.580673 2585 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-83c318-tsscb" Sep 10 13:58:23.580879 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:58:23.580712 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-83c318-tsscb" event={"ID":"5714cced-05e6-4862-851e-c9d3db13bfc0","Type":"ContainerDied","Data":"5cd3de39cf7fac8b016834f8105c8f3c6b22022f85a02f4f84224c9d62abdc39"} Sep 10 13:58:23.580879 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:58:23.580797 2585 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="5cd3de39cf7fac8b016834f8105c8f3c6b22022f85a02f4f84224c9d62abdc39" Sep 10 13:58:28.048984 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:58:28.048909 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fxgmp_fa53c783-d40a-4024-9f7b-1d0d2b57f6dc/konnectivity-agent/0.log" limit=1024 Sep 10 13:58:28.175167 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:58:28.175134 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fxgmp_fa53c783-d40a-4024-9f7b-1d0d2b57f6dc/konnectivity-agent/0.log" limit=1024 Sep 10 13:58:30.042361 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:58:30.042328 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fxgmp_fa53c783-d40a-4024-9f7b-1d0d2b57f6dc/konnectivity-agent/0.log" limit=1024 Sep 10 13:58:30.084005 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:58:30.083964 2585 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w2-83c318-5d96b6dbdf-r7lms"] Sep 10 13:58:30.084231 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:58:30.084203 2585 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-83c318-5d96b6dbdf-r7lms" podUID="9299822d-564e-408d-955f-0bc46c91bf93" containerName="payload-processing" containerID="cri-o://07dbf626706644e9e0cca5fe277d69e73bf7075aff7b2552436e8f10bffc6c8a" gracePeriod=30 Sep 10 13:58:30.094230 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:58:30.094198 2585 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w2-83c318-8c7bcb8db-9f552"] Sep 10 13:58:30.094670 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:58:30.094590 2585 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-83c318-8c7bcb8db-9f552" podUID="be2e5ae8-d29d-46ad-b6ce-826118e204a7" containerName="payload-pre-processing" containerID="cri-o://6d16dae5670b1bc5e33cc6a2d013e284deb85cc12250724ef6880730aa1160f1" gracePeriod=30 Sep 10 13:58:30.129541 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:58:30.129512 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fxgmp_fa53c783-d40a-4024-9f7b-1d0d2b57f6dc/konnectivity-agent/0.log" limit=1024 Sep 10 13:58:30.197453 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:58:30.197424 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fxgmp_fa53c783-d40a-4024-9f7b-1d0d2b57f6dc/konnectivity-agent/0.log" limit=1024 Sep 10 13:58:30.298882 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:58:30.298771 2585 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-83c318-tsscb"] Sep 10 13:58:30.300107 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:58:30.300077 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fxgmp_fa53c783-d40a-4024-9f7b-1d0d2b57f6dc/konnectivity-agent/0.log" limit=1024 Sep 10 13:58:30.301904 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:58:30.301878 2585 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-83c318-tsscb"] Sep 10 13:58:30.342708 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:58:30.342682 2585 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-83c318-5d96b6dbdf-r7lms" Sep 10 13:58:30.367662 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:58:30.367576 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fxgmp_fa53c783-d40a-4024-9f7b-1d0d2b57f6dc/konnectivity-agent/0.log" limit=1024 Sep 10 13:58:30.379466 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:58:30.379444 2585 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-83c318-8c7bcb8db-9f552" Sep 10 13:58:30.383377 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:58:30.383352 2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/be2e5ae8-d29d-46ad-b6ce-826118e204a7-plugins-config-volume\") pod \"be2e5ae8-d29d-46ad-b6ce-826118e204a7\" (UID: \"be2e5ae8-d29d-46ad-b6ce-826118e204a7\") " Sep 10 13:58:30.383475 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:58:30.383423 2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-sv5x5\" (UniqueName: \"kubernetes.io/projected/9299822d-564e-408d-955f-0bc46c91bf93-kube-api-access-sv5x5\") pod \"9299822d-564e-408d-955f-0bc46c91bf93\" (UID: \"9299822d-564e-408d-955f-0bc46c91bf93\") " Sep 10 13:58:30.383539 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:58:30.383494 2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9299822d-564e-408d-955f-0bc46c91bf93-plugins-config-volume\") pod \"9299822d-564e-408d-955f-0bc46c91bf93\" (UID: \"9299822d-564e-408d-955f-0bc46c91bf93\") " Sep 10 13:58:30.383600 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:58:30.383539 2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-7zjff\" (UniqueName: \"kubernetes.io/projected/be2e5ae8-d29d-46ad-b6ce-826118e204a7-kube-api-access-7zjff\") pod \"be2e5ae8-d29d-46ad-b6ce-826118e204a7\" (UID: \"be2e5ae8-d29d-46ad-b6ce-826118e204a7\") " Sep 10 13:58:30.384064 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:58:30.383728 2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/be2e5ae8-d29d-46ad-b6ce-826118e204a7-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "be2e5ae8-d29d-46ad-b6ce-826118e204a7" (UID: "be2e5ae8-d29d-46ad-b6ce-826118e204a7"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 10 13:58:30.384064 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:58:30.383823 2585 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/be2e5ae8-d29d-46ad-b6ce-826118e204a7-plugins-config-volume\") on node \"ip-10-0-140-8.ec2.internal\" DevicePath \"\"" Sep 10 13:58:30.384064 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:58:30.384015 2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9299822d-564e-408d-955f-0bc46c91bf93-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "9299822d-564e-408d-955f-0bc46c91bf93" (UID: "9299822d-564e-408d-955f-0bc46c91bf93"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 10 13:58:30.385751 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:58:30.385726 2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9299822d-564e-408d-955f-0bc46c91bf93-kube-api-access-sv5x5" (OuterVolumeSpecName: "kube-api-access-sv5x5") pod "9299822d-564e-408d-955f-0bc46c91bf93" (UID: "9299822d-564e-408d-955f-0bc46c91bf93"). InnerVolumeSpecName "kube-api-access-sv5x5". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 10 13:58:30.385825 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:58:30.385794 2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/be2e5ae8-d29d-46ad-b6ce-826118e204a7-kube-api-access-7zjff" (OuterVolumeSpecName: "kube-api-access-7zjff") pod "be2e5ae8-d29d-46ad-b6ce-826118e204a7" (UID: "be2e5ae8-d29d-46ad-b6ce-826118e204a7"). InnerVolumeSpecName "kube-api-access-7zjff". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 10 13:58:30.485125 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:58:30.485085 2585 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sv5x5\" (UniqueName: \"kubernetes.io/projected/9299822d-564e-408d-955f-0bc46c91bf93-kube-api-access-sv5x5\") on node \"ip-10-0-140-8.ec2.internal\" DevicePath \"\"" Sep 10 13:58:30.485125 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:58:30.485116 2585 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9299822d-564e-408d-955f-0bc46c91bf93-plugins-config-volume\") on node \"ip-10-0-140-8.ec2.internal\" DevicePath \"\"" Sep 10 13:58:30.485125 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:58:30.485130 2585 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7zjff\" (UniqueName: \"kubernetes.io/projected/be2e5ae8-d29d-46ad-b6ce-826118e204a7-kube-api-access-7zjff\") on node \"ip-10-0-140-8.ec2.internal\" DevicePath \"\"" Sep 10 13:58:30.626423 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:58:30.626379 2585 generic.go:356] "Generic (PLEG): container finished" podID="9299822d-564e-408d-955f-0bc46c91bf93" containerID="07dbf626706644e9e0cca5fe277d69e73bf7075aff7b2552436e8f10bffc6c8a" exitCode=0 Sep 10 13:58:30.626604 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:58:30.626431 2585 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-83c318-5d96b6dbdf-r7lms" Sep 10 13:58:30.626604 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:58:30.626466 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-83c318-5d96b6dbdf-r7lms" event={"ID":"9299822d-564e-408d-955f-0bc46c91bf93","Type":"ContainerDied","Data":"07dbf626706644e9e0cca5fe277d69e73bf7075aff7b2552436e8f10bffc6c8a"} Sep 10 13:58:30.626604 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:58:30.626525 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-83c318-5d96b6dbdf-r7lms" event={"ID":"9299822d-564e-408d-955f-0bc46c91bf93","Type":"ContainerDied","Data":"befa65b74b9abd583c5175bf0d93752b0fc424f75591993153b56ff67b240e14"} Sep 10 13:58:30.626604 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:58:30.626540 2585 scope.go:117] "RemoveContainer" containerID="07dbf626706644e9e0cca5fe277d69e73bf7075aff7b2552436e8f10bffc6c8a" Sep 10 13:58:30.627917 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:58:30.627880 2585 generic.go:356] "Generic (PLEG): container finished" podID="be2e5ae8-d29d-46ad-b6ce-826118e204a7" containerID="6d16dae5670b1bc5e33cc6a2d013e284deb85cc12250724ef6880730aa1160f1" exitCode=0 Sep 10 13:58:30.628036 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:58:30.627914 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-83c318-8c7bcb8db-9f552" event={"ID":"be2e5ae8-d29d-46ad-b6ce-826118e204a7","Type":"ContainerDied","Data":"6d16dae5670b1bc5e33cc6a2d013e284deb85cc12250724ef6880730aa1160f1"} Sep 10 13:58:30.628036 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:58:30.627964 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-83c318-8c7bcb8db-9f552" event={"ID":"be2e5ae8-d29d-46ad-b6ce-826118e204a7","Type":"ContainerDied","Data":"a8caa053c5db3388eeed7a67dddaf081cb740a6ddd010288812c64d6d631249f"} Sep 10 13:58:30.628036 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:58:30.627919 2585 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-83c318-8c7bcb8db-9f552" Sep 10 13:58:30.636011 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:58:30.635994 2585 scope.go:117] "RemoveContainer" containerID="07dbf626706644e9e0cca5fe277d69e73bf7075aff7b2552436e8f10bffc6c8a" Sep 10 13:58:30.636444 ip-10-0-140-8 kubenswrapper[2585]: E0910 13:58:30.636313 2585 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"07dbf626706644e9e0cca5fe277d69e73bf7075aff7b2552436e8f10bffc6c8a\": container with ID starting with 07dbf626706644e9e0cca5fe277d69e73bf7075aff7b2552436e8f10bffc6c8a not found: ID does not exist" containerID="07dbf626706644e9e0cca5fe277d69e73bf7075aff7b2552436e8f10bffc6c8a" Sep 10 13:58:30.636444 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:58:30.636346 2585 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"07dbf626706644e9e0cca5fe277d69e73bf7075aff7b2552436e8f10bffc6c8a"} err="failed to get container status \"07dbf626706644e9e0cca5fe277d69e73bf7075aff7b2552436e8f10bffc6c8a\": rpc error: code = NotFound desc = could not find container \"07dbf626706644e9e0cca5fe277d69e73bf7075aff7b2552436e8f10bffc6c8a\": container with ID starting with 07dbf626706644e9e0cca5fe277d69e73bf7075aff7b2552436e8f10bffc6c8a not found: ID does not exist" Sep 10 13:58:30.636444 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:58:30.636371 2585 scope.go:117] "RemoveContainer" containerID="6d16dae5670b1bc5e33cc6a2d013e284deb85cc12250724ef6880730aa1160f1" Sep 10 13:58:30.644750 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:58:30.644733 2585 scope.go:117] "RemoveContainer" containerID="6d16dae5670b1bc5e33cc6a2d013e284deb85cc12250724ef6880730aa1160f1" Sep 10 13:58:30.645023 ip-10-0-140-8 kubenswrapper[2585]: E0910 13:58:30.645006 2585 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6d16dae5670b1bc5e33cc6a2d013e284deb85cc12250724ef6880730aa1160f1\": container with ID starting with 6d16dae5670b1bc5e33cc6a2d013e284deb85cc12250724ef6880730aa1160f1 not found: ID does not exist" containerID="6d16dae5670b1bc5e33cc6a2d013e284deb85cc12250724ef6880730aa1160f1" Sep 10 13:58:30.645066 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:58:30.645033 2585 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6d16dae5670b1bc5e33cc6a2d013e284deb85cc12250724ef6880730aa1160f1"} err="failed to get container status \"6d16dae5670b1bc5e33cc6a2d013e284deb85cc12250724ef6880730aa1160f1\": rpc error: code = NotFound desc = could not find container \"6d16dae5670b1bc5e33cc6a2d013e284deb85cc12250724ef6880730aa1160f1\": container with ID starting with 6d16dae5670b1bc5e33cc6a2d013e284deb85cc12250724ef6880730aa1160f1 not found: ID does not exist" Sep 10 13:58:30.667869 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:58:30.667845 2585 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w2-83c318-8c7bcb8db-9f552"] Sep 10 13:58:30.670188 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:58:30.670092 2585 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w2-83c318-8c7bcb8db-9f552"] Sep 10 13:58:30.682815 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:58:30.682789 2585 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w2-83c318-5d96b6dbdf-r7lms"] Sep 10 13:58:30.684390 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:58:30.684368 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fxgmp_fa53c783-d40a-4024-9f7b-1d0d2b57f6dc/konnectivity-agent/0.log" limit=1024 Sep 10 13:58:30.684930 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:58:30.684910 2585 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w2-83c318-5d96b6dbdf-r7lms"] Sep 10 13:58:31.647174 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:58:31.647142 2585 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="5714cced-05e6-4862-851e-c9d3db13bfc0" path="/var/lib/kubelet/pods/5714cced-05e6-4862-851e-c9d3db13bfc0/volumes" Sep 10 13:58:31.647570 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:58:31.647524 2585 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="9299822d-564e-408d-955f-0bc46c91bf93" path="/var/lib/kubelet/pods/9299822d-564e-408d-955f-0bc46c91bf93/volumes" Sep 10 13:58:31.647836 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:58:31.647823 2585 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="be2e5ae8-d29d-46ad-b6ce-826118e204a7" path="/var/lib/kubelet/pods/be2e5ae8-d29d-46ad-b6ce-826118e204a7/volumes" Sep 10 13:58:32.001187 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:58:32.001112 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fxgmp_fa53c783-d40a-4024-9f7b-1d0d2b57f6dc/konnectivity-agent/0.log" limit=1024 Sep 10 13:58:35.761957 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:58:35.761928 2585 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-l9l8q_ad555c6a-4385-4f8e-93bb-a174cfaa8e74/kube-multus/2.log" Sep 10 13:58:35.764495 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:58:35.764475 2585 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-l9l8q_ad555c6a-4385-4f8e-93bb-a174cfaa8e74/kube-multus/2.log" Sep 10 13:58:38.123671 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:58:38.123637 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fxgmp_fa53c783-d40a-4024-9f7b-1d0d2b57f6dc/konnectivity-agent/0.log" limit=1024 Sep 10 13:58:38.185459 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:58:38.185419 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fxgmp_fa53c783-d40a-4024-9f7b-1d0d2b57f6dc/konnectivity-agent/0.log" limit=1024 Sep 10 13:58:38.402610 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:58:38.402529 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fxgmp_fa53c783-d40a-4024-9f7b-1d0d2b57f6dc/konnectivity-agent/0.log" limit=1024 Sep 10 13:58:38.748775 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:58:38.748684 2585 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-a310dd-zv7r7"] Sep 10 13:58:38.749235 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:58:38.749206 2585 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5714cced-05e6-4862-851e-c9d3db13bfc0" containerName="revoke-keys" Sep 10 13:58:38.749235 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:58:38.749233 2585 state_mem.go:107] "Deleted CPUSet assignment" podUID="5714cced-05e6-4862-851e-c9d3db13bfc0" containerName="revoke-keys" Sep 10 13:58:38.749417 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:58:38.749283 2585 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9299822d-564e-408d-955f-0bc46c91bf93" containerName="payload-processing" Sep 10 13:58:38.749417 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:58:38.749293 2585 state_mem.go:107] "Deleted CPUSet assignment" podUID="9299822d-564e-408d-955f-0bc46c91bf93" containerName="payload-processing" Sep 10 13:58:38.749417 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:58:38.749326 2585 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="be2e5ae8-d29d-46ad-b6ce-826118e204a7" containerName="payload-pre-processing" Sep 10 13:58:38.749417 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:58:38.749333 2585 state_mem.go:107] "Deleted CPUSet assignment" podUID="be2e5ae8-d29d-46ad-b6ce-826118e204a7" containerName="payload-pre-processing" Sep 10 13:58:38.749417 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:58:38.749408 2585 memory_manager.go:368] "RemoveStaleState removing state" podUID="5714cced-05e6-4862-851e-c9d3db13bfc0" containerName="revoke-keys" Sep 10 13:58:38.749417 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:58:38.749420 2585 memory_manager.go:368] "RemoveStaleState removing state" podUID="be2e5ae8-d29d-46ad-b6ce-826118e204a7" containerName="payload-pre-processing" Sep 10 13:58:38.749601 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:58:38.749426 2585 memory_manager.go:368] "RemoveStaleState removing state" podUID="9299822d-564e-408d-955f-0bc46c91bf93" containerName="payload-processing" Sep 10 13:58:38.752534 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:58:38.752517 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-a310dd-zv7r7" Sep 10 13:58:38.755453 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:58:38.755430 2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-7gcbr\"" Sep 10 13:58:38.755602 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:58:38.755580 2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Sep 10 13:58:38.761883 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:58:38.761860 2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-a310dd-zv7r7"] Sep 10 13:58:38.866389 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:58:38.866349 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/95c870ea-9594-4976-90ec-b41892167a03-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w2-a310dd-zv7r7\" (UID: \"95c870ea-9594-4976-90ec-b41892167a03\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-a310dd-zv7r7" Sep 10 13:58:38.967740 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:58:38.967702 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/95c870ea-9594-4976-90ec-b41892167a03-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w2-a310dd-zv7r7\" (UID: \"95c870ea-9594-4976-90ec-b41892167a03\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-a310dd-zv7r7" Sep 10 13:58:38.968327 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:58:38.968308 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/95c870ea-9594-4976-90ec-b41892167a03-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w2-a310dd-zv7r7\" (UID: \"95c870ea-9594-4976-90ec-b41892167a03\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-a310dd-zv7r7" Sep 10 13:58:38.971499 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:58:38.971475 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fxgmp_fa53c783-d40a-4024-9f7b-1d0d2b57f6dc/konnectivity-agent/0.log" limit=1024 Sep 10 13:58:39.064863 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:58:39.064839 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-a310dd-zv7r7" Sep 10 13:58:39.372002 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:58:39.371978 2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-a310dd-zv7r7"] Sep 10 13:58:39.374485 ip-10-0-140-8 kubenswrapper[2585]: W0910 13:58:39.374458 2585 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod95c870ea_9594_4976_90ec_b41892167a03.slice/crio-7e1c7976f38123006ff19a3dfdd40ed27c4f18d06d07e652347d1ee5203c47f4 WatchSource:0}: Error finding container 7e1c7976f38123006ff19a3dfdd40ed27c4f18d06d07e652347d1ee5203c47f4: Status 404 returned error can't find the container with id 7e1c7976f38123006ff19a3dfdd40ed27c4f18d06d07e652347d1ee5203c47f4 Sep 10 13:58:39.681495 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:58:39.681460 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-a310dd-zv7r7" event={"ID":"95c870ea-9594-4976-90ec-b41892167a03","Type":"ContainerStarted","Data":"7e1c7976f38123006ff19a3dfdd40ed27c4f18d06d07e652347d1ee5203c47f4"} Sep 10 13:58:40.688861 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:58:40.688827 2585 generic.go:356] "Generic (PLEG): container finished" podID="95c870ea-9594-4976-90ec-b41892167a03" containerID="466628d15fa2e47dc77a429e9868871f11628e3b3ca0aa2129f6c4ab11b6516c" exitCode=0 Sep 10 13:58:40.689206 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:58:40.688907 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-a310dd-zv7r7" event={"ID":"95c870ea-9594-4976-90ec-b41892167a03","Type":"ContainerDied","Data":"466628d15fa2e47dc77a429e9868871f11628e3b3ca0aa2129f6c4ab11b6516c"} Sep 10 13:58:41.829844 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:58:41.829822 2585 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-a310dd-zv7r7" Sep 10 13:58:41.898457 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:58:41.898422 2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/95c870ea-9594-4976-90ec-b41892167a03-maas-api-service-ca\") pod \"95c870ea-9594-4976-90ec-b41892167a03\" (UID: \"95c870ea-9594-4976-90ec-b41892167a03\") " Sep 10 13:58:41.898776 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:58:41.898754 2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/95c870ea-9594-4976-90ec-b41892167a03-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "95c870ea-9594-4976-90ec-b41892167a03" (UID: "95c870ea-9594-4976-90ec-b41892167a03"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 10 13:58:41.999609 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:58:41.999532 2585 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/95c870ea-9594-4976-90ec-b41892167a03-maas-api-service-ca\") on node \"ip-10-0-140-8.ec2.internal\" DevicePath \"\"" Sep 10 13:58:42.700323 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:58:42.700292 2585 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-a310dd-zv7r7" Sep 10 13:58:42.700495 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:58:42.700331 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-a310dd-zv7r7" event={"ID":"95c870ea-9594-4976-90ec-b41892167a03","Type":"ContainerDied","Data":"7e1c7976f38123006ff19a3dfdd40ed27c4f18d06d07e652347d1ee5203c47f4"} Sep 10 13:58:42.700495 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:58:42.700376 2585 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="7e1c7976f38123006ff19a3dfdd40ed27c4f18d06d07e652347d1ee5203c47f4" Sep 10 13:58:49.476450 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:58:49.476406 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fxgmp_fa53c783-d40a-4024-9f7b-1d0d2b57f6dc/konnectivity-agent/0.log" limit=1024 Sep 10 13:58:49.654632 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:58:49.654601 2585 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w2-a310dd-8457c575c-7fwnk"] Sep 10 13:58:49.654851 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:58:49.654827 2585 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-a310dd-8457c575c-7fwnk" podUID="c13d1924-9f38-4994-88ba-5f381e8c3355" containerName="payload-processing" containerID="cri-o://75df9492db4aca6c957efa9e0654c677f88982710eda0125dfb87de8d9fad557" gracePeriod=30 Sep 10 13:58:49.671560 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:58:49.671536 2585 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w2-a310dd-6466864494-wfq7f"] Sep 10 13:58:49.671795 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:58:49.671771 2585 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-a310dd-6466864494-wfq7f" podUID="4beb2e3e-5cc7-4dca-b4d0-5d4bfdce437e" containerName="payload-pre-processing" containerID="cri-o://61abf5783773209b63c65359cb933e5a3a3bdf362930c103da601b263e1b7b98" gracePeriod=30 Sep 10 13:58:49.728772 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:58:49.728705 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fxgmp_fa53c783-d40a-4024-9f7b-1d0d2b57f6dc/konnectivity-agent/0.log" limit=1024 Sep 10 13:58:49.852603 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:58:49.852533 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fxgmp_fa53c783-d40a-4024-9f7b-1d0d2b57f6dc/konnectivity-agent/0.log" limit=1024 Sep 10 13:58:49.956001 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:58:49.955979 2585 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-a310dd-8457c575c-7fwnk" Sep 10 13:58:49.958842 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:58:49.958824 2585 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-a310dd-6466864494-wfq7f" Sep 10 13:58:49.989876 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:58:49.989803 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fxgmp_fa53c783-d40a-4024-9f7b-1d0d2b57f6dc/konnectivity-agent/0.log" limit=1024 Sep 10 13:58:50.053842 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:58:50.053805 2585 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-a310dd-zv7r7"] Sep 10 13:58:50.061567 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:58:50.061539 2585 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-a310dd-zv7r7"] Sep 10 13:58:50.077536 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:58:50.077507 2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-gqwhz\" (UniqueName: \"kubernetes.io/projected/4beb2e3e-5cc7-4dca-b4d0-5d4bfdce437e-kube-api-access-gqwhz\") pod \"4beb2e3e-5cc7-4dca-b4d0-5d4bfdce437e\" (UID: \"4beb2e3e-5cc7-4dca-b4d0-5d4bfdce437e\") " Sep 10 13:58:50.077660 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:58:50.077555 2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4beb2e3e-5cc7-4dca-b4d0-5d4bfdce437e-plugins-config-volume\") pod \"4beb2e3e-5cc7-4dca-b4d0-5d4bfdce437e\" (UID: \"4beb2e3e-5cc7-4dca-b4d0-5d4bfdce437e\") " Sep 10 13:58:50.077660 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:58:50.077600 2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-q65qh\" (UniqueName: \"kubernetes.io/projected/c13d1924-9f38-4994-88ba-5f381e8c3355-kube-api-access-q65qh\") pod \"c13d1924-9f38-4994-88ba-5f381e8c3355\" (UID: \"c13d1924-9f38-4994-88ba-5f381e8c3355\") " Sep 10 13:58:50.077772 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:58:50.077659 2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c13d1924-9f38-4994-88ba-5f381e8c3355-plugins-config-volume\") pod \"c13d1924-9f38-4994-88ba-5f381e8c3355\" (UID: \"c13d1924-9f38-4994-88ba-5f381e8c3355\") " Sep 10 13:58:50.078038 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:58:50.078015 2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4beb2e3e-5cc7-4dca-b4d0-5d4bfdce437e-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "4beb2e3e-5cc7-4dca-b4d0-5d4bfdce437e" (UID: "4beb2e3e-5cc7-4dca-b4d0-5d4bfdce437e"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 10 13:58:50.078126 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:58:50.078047 2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c13d1924-9f38-4994-88ba-5f381e8c3355-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "c13d1924-9f38-4994-88ba-5f381e8c3355" (UID: "c13d1924-9f38-4994-88ba-5f381e8c3355"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 10 13:58:50.079877 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:58:50.079851 2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c13d1924-9f38-4994-88ba-5f381e8c3355-kube-api-access-q65qh" (OuterVolumeSpecName: "kube-api-access-q65qh") pod "c13d1924-9f38-4994-88ba-5f381e8c3355" (UID: "c13d1924-9f38-4994-88ba-5f381e8c3355"). InnerVolumeSpecName "kube-api-access-q65qh". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 10 13:58:50.079970 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:58:50.079930 2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4beb2e3e-5cc7-4dca-b4d0-5d4bfdce437e-kube-api-access-gqwhz" (OuterVolumeSpecName: "kube-api-access-gqwhz") pod "4beb2e3e-5cc7-4dca-b4d0-5d4bfdce437e" (UID: "4beb2e3e-5cc7-4dca-b4d0-5d4bfdce437e"). InnerVolumeSpecName "kube-api-access-gqwhz". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 10 13:58:50.179031 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:58:50.178999 2585 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-q65qh\" (UniqueName: \"kubernetes.io/projected/c13d1924-9f38-4994-88ba-5f381e8c3355-kube-api-access-q65qh\") on node \"ip-10-0-140-8.ec2.internal\" DevicePath \"\"" Sep 10 13:58:50.179031 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:58:50.179026 2585 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c13d1924-9f38-4994-88ba-5f381e8c3355-plugins-config-volume\") on node \"ip-10-0-140-8.ec2.internal\" DevicePath \"\"" Sep 10 13:58:50.179031 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:58:50.179036 2585 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gqwhz\" (UniqueName: \"kubernetes.io/projected/4beb2e3e-5cc7-4dca-b4d0-5d4bfdce437e-kube-api-access-gqwhz\") on node \"ip-10-0-140-8.ec2.internal\" DevicePath \"\"" Sep 10 13:58:50.179261 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:58:50.179045 2585 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4beb2e3e-5cc7-4dca-b4d0-5d4bfdce437e-plugins-config-volume\") on node \"ip-10-0-140-8.ec2.internal\" DevicePath \"\"" Sep 10 13:58:50.749410 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:58:50.749376 2585 generic.go:356] "Generic (PLEG): container finished" podID="c13d1924-9f38-4994-88ba-5f381e8c3355" containerID="75df9492db4aca6c957efa9e0654c677f88982710eda0125dfb87de8d9fad557" exitCode=0 Sep 10 13:58:50.749873 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:58:50.749421 2585 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-a310dd-8457c575c-7fwnk" Sep 10 13:58:50.749873 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:58:50.749471 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-a310dd-8457c575c-7fwnk" event={"ID":"c13d1924-9f38-4994-88ba-5f381e8c3355","Type":"ContainerDied","Data":"75df9492db4aca6c957efa9e0654c677f88982710eda0125dfb87de8d9fad557"} Sep 10 13:58:50.749873 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:58:50.749533 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-a310dd-8457c575c-7fwnk" event={"ID":"c13d1924-9f38-4994-88ba-5f381e8c3355","Type":"ContainerDied","Data":"972be6152c7872b2ce2f72b3d60ae6746fb30d75232e1a29f5aaa28367b053bf"} Sep 10 13:58:50.749873 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:58:50.749565 2585 scope.go:117] "RemoveContainer" containerID="75df9492db4aca6c957efa9e0654c677f88982710eda0125dfb87de8d9fad557" Sep 10 13:58:50.751135 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:58:50.750940 2585 generic.go:356] "Generic (PLEG): container finished" podID="4beb2e3e-5cc7-4dca-b4d0-5d4bfdce437e" containerID="61abf5783773209b63c65359cb933e5a3a3bdf362930c103da601b263e1b7b98" exitCode=0 Sep 10 13:58:50.751135 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:58:50.750973 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-a310dd-6466864494-wfq7f" event={"ID":"4beb2e3e-5cc7-4dca-b4d0-5d4bfdce437e","Type":"ContainerDied","Data":"61abf5783773209b63c65359cb933e5a3a3bdf362930c103da601b263e1b7b98"} Sep 10 13:58:50.751135 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:58:50.751000 2585 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-a310dd-6466864494-wfq7f" Sep 10 13:58:50.751135 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:58:50.751009 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-a310dd-6466864494-wfq7f" event={"ID":"4beb2e3e-5cc7-4dca-b4d0-5d4bfdce437e","Type":"ContainerDied","Data":"88ef24f3758e32ab970930b4b6b5ff7f1c7f141d0fb120d8a88d3a457929081c"} Sep 10 13:58:50.762415 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:58:50.761946 2585 scope.go:117] "RemoveContainer" containerID="75df9492db4aca6c957efa9e0654c677f88982710eda0125dfb87de8d9fad557" Sep 10 13:58:50.762565 ip-10-0-140-8 kubenswrapper[2585]: E0910 13:58:50.762439 2585 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"75df9492db4aca6c957efa9e0654c677f88982710eda0125dfb87de8d9fad557\": container with ID starting with 75df9492db4aca6c957efa9e0654c677f88982710eda0125dfb87de8d9fad557 not found: ID does not exist" containerID="75df9492db4aca6c957efa9e0654c677f88982710eda0125dfb87de8d9fad557" Sep 10 13:58:50.762565 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:58:50.762505 2585 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"75df9492db4aca6c957efa9e0654c677f88982710eda0125dfb87de8d9fad557"} err="failed to get container status \"75df9492db4aca6c957efa9e0654c677f88982710eda0125dfb87de8d9fad557\": rpc error: code = NotFound desc = could not find container \"75df9492db4aca6c957efa9e0654c677f88982710eda0125dfb87de8d9fad557\": container with ID starting with 75df9492db4aca6c957efa9e0654c677f88982710eda0125dfb87de8d9fad557 not found: ID does not exist" Sep 10 13:58:50.762565 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:58:50.762534 2585 scope.go:117] "RemoveContainer" containerID="61abf5783773209b63c65359cb933e5a3a3bdf362930c103da601b263e1b7b98" Sep 10 13:58:50.773578 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:58:50.773557 2585 scope.go:117] "RemoveContainer" containerID="61abf5783773209b63c65359cb933e5a3a3bdf362930c103da601b263e1b7b98" Sep 10 13:58:50.773906 ip-10-0-140-8 kubenswrapper[2585]: E0910 13:58:50.773882 2585 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"61abf5783773209b63c65359cb933e5a3a3bdf362930c103da601b263e1b7b98\": container with ID starting with 61abf5783773209b63c65359cb933e5a3a3bdf362930c103da601b263e1b7b98 not found: ID does not exist" containerID="61abf5783773209b63c65359cb933e5a3a3bdf362930c103da601b263e1b7b98" Sep 10 13:58:50.773971 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:58:50.773916 2585 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"61abf5783773209b63c65359cb933e5a3a3bdf362930c103da601b263e1b7b98"} err="failed to get container status \"61abf5783773209b63c65359cb933e5a3a3bdf362930c103da601b263e1b7b98\": rpc error: code = NotFound desc = could not find container \"61abf5783773209b63c65359cb933e5a3a3bdf362930c103da601b263e1b7b98\": container with ID starting with 61abf5783773209b63c65359cb933e5a3a3bdf362930c103da601b263e1b7b98 not found: ID does not exist" Sep 10 13:58:50.781889 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:58:50.781866 2585 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w2-a310dd-6466864494-wfq7f"] Sep 10 13:58:50.787176 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:58:50.787153 2585 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w2-a310dd-6466864494-wfq7f"] Sep 10 13:58:50.799666 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:58:50.799638 2585 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w2-a310dd-8457c575c-7fwnk"] Sep 10 13:58:50.808107 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:58:50.808085 2585 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w2-a310dd-8457c575c-7fwnk"] Sep 10 13:58:51.470810 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:58:51.470770 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fxgmp_fa53c783-d40a-4024-9f7b-1d0d2b57f6dc/konnectivity-agent/0.log" limit=1024 Sep 10 13:58:51.647224 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:58:51.647188 2585 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="4beb2e3e-5cc7-4dca-b4d0-5d4bfdce437e" path="/var/lib/kubelet/pods/4beb2e3e-5cc7-4dca-b4d0-5d4bfdce437e/volumes" Sep 10 13:58:51.647568 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:58:51.647554 2585 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="95c870ea-9594-4976-90ec-b41892167a03" path="/var/lib/kubelet/pods/95c870ea-9594-4976-90ec-b41892167a03/volumes" Sep 10 13:58:51.647870 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:58:51.647859 2585 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="c13d1924-9f38-4994-88ba-5f381e8c3355" path="/var/lib/kubelet/pods/c13d1924-9f38-4994-88ba-5f381e8c3355/volumes" Sep 10 13:58:52.878657 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:58:52.878624 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fxgmp_fa53c783-d40a-4024-9f7b-1d0d2b57f6dc/konnectivity-agent/0.log" limit=1024 Sep 10 13:58:57.875897 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:58:57.875859 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fxgmp_fa53c783-d40a-4024-9f7b-1d0d2b57f6dc/konnectivity-agent/0.log" limit=1024 Sep 10 13:58:57.989514 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:58:57.989479 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fxgmp_fa53c783-d40a-4024-9f7b-1d0d2b57f6dc/konnectivity-agent/0.log" limit=1024 Sep 10 13:58:58.175711 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:58:58.175615 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fxgmp_fa53c783-d40a-4024-9f7b-1d0d2b57f6dc/konnectivity-agent/0.log" limit=1024 Sep 10 13:59:18.446472 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:59:18.446442 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fxgmp_fa53c783-d40a-4024-9f7b-1d0d2b57f6dc/konnectivity-agent/0.log" limit=1024 Sep 10 13:59:19.441924 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:59:19.441874 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fxgmp_fa53c783-d40a-4024-9f7b-1d0d2b57f6dc/konnectivity-agent/0.log" limit=1024 Sep 10 13:59:19.910087 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:59:19.910048 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fxgmp_fa53c783-d40a-4024-9f7b-1d0d2b57f6dc/konnectivity-agent/0.log" limit=1024 Sep 10 13:59:20.276269 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:59:20.276129 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fxgmp_fa53c783-d40a-4024-9f7b-1d0d2b57f6dc/konnectivity-agent/0.log" limit=1024 Sep 10 13:59:20.534555 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:59:20.534433 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fxgmp_fa53c783-d40a-4024-9f7b-1d0d2b57f6dc/konnectivity-agent/0.log" limit=1024 Sep 10 13:59:23.774600 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:59:23.774559 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fxgmp_fa53c783-d40a-4024-9f7b-1d0d2b57f6dc/konnectivity-agent/0.log" limit=1024 Sep 10 13:59:23.874415 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:59:23.874378 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fxgmp_fa53c783-d40a-4024-9f7b-1d0d2b57f6dc/konnectivity-agent/0.log" limit=1024 Sep 10 13:59:25.152539 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:59:25.152498 2585 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-3f01c7-cbdfbc885-6f9lk"] Sep 10 13:59:25.153173 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:59:25.153148 2585 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4beb2e3e-5cc7-4dca-b4d0-5d4bfdce437e" containerName="payload-pre-processing" Sep 10 13:59:25.153173 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:59:25.153175 2585 state_mem.go:107] "Deleted CPUSet assignment" podUID="4beb2e3e-5cc7-4dca-b4d0-5d4bfdce437e" containerName="payload-pre-processing" Sep 10 13:59:25.153454 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:59:25.153187 2585 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c13d1924-9f38-4994-88ba-5f381e8c3355" containerName="payload-processing" Sep 10 13:59:25.153454 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:59:25.153195 2585 state_mem.go:107] "Deleted CPUSet assignment" podUID="c13d1924-9f38-4994-88ba-5f381e8c3355" containerName="payload-processing" Sep 10 13:59:25.153454 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:59:25.153211 2585 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="95c870ea-9594-4976-90ec-b41892167a03" containerName="revoke-keys" Sep 10 13:59:25.153454 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:59:25.153220 2585 state_mem.go:107] "Deleted CPUSet assignment" podUID="95c870ea-9594-4976-90ec-b41892167a03" containerName="revoke-keys" Sep 10 13:59:25.153454 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:59:25.153382 2585 memory_manager.go:368] "RemoveStaleState removing state" podUID="4beb2e3e-5cc7-4dca-b4d0-5d4bfdce437e" containerName="payload-pre-processing" Sep 10 13:59:25.153454 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:59:25.153395 2585 memory_manager.go:368] "RemoveStaleState removing state" podUID="c13d1924-9f38-4994-88ba-5f381e8c3355" containerName="payload-processing" Sep 10 13:59:25.153454 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:59:25.153405 2585 memory_manager.go:368] "RemoveStaleState removing state" podUID="95c870ea-9594-4976-90ec-b41892167a03" containerName="revoke-keys" Sep 10 13:59:25.156914 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:59:25.156896 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-3f01c7-cbdfbc885-6f9lk" Sep 10 13:59:25.159870 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:59:25.159846 2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-metrics-tls\"" Sep 10 13:59:25.159980 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:59:25.159846 2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ipp-a-3f01c7-serving-cert\"" Sep 10 13:59:25.160999 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:59:25.160979 2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Sep 10 13:59:25.161103 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:59:25.160998 2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-q8qnd\"" Sep 10 13:59:25.161103 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:59:25.161025 2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Sep 10 13:59:25.168530 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:59:25.168507 2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-3f01c7-cbdfbc885-6f9lk"] Sep 10 13:59:25.185222 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:59:25.185190 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fxgmp_fa53c783-d40a-4024-9f7b-1d0d2b57f6dc/konnectivity-agent/0.log" limit=1024 Sep 10 13:59:25.212560 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:59:25.212529 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/af6ff637-3d66-4a99-bccf-dc9cff523629-maas-api-tls\") pod \"maas-api-e2e-ipp-a-3f01c7-cbdfbc885-6f9lk\" (UID: \"af6ff637-3d66-4a99-bccf-dc9cff523629\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-3f01c7-cbdfbc885-6f9lk" Sep 10 13:59:25.212721 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:59:25.212586 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-88mmc\" (UniqueName: \"kubernetes.io/projected/af6ff637-3d66-4a99-bccf-dc9cff523629-kube-api-access-88mmc\") pod \"maas-api-e2e-ipp-a-3f01c7-cbdfbc885-6f9lk\" (UID: \"af6ff637-3d66-4a99-bccf-dc9cff523629\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-3f01c7-cbdfbc885-6f9lk" Sep 10 13:59:25.212721 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:59:25.212690 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/af6ff637-3d66-4a99-bccf-dc9cff523629-openshift-service-ca\") pod \"maas-api-e2e-ipp-a-3f01c7-cbdfbc885-6f9lk\" (UID: \"af6ff637-3d66-4a99-bccf-dc9cff523629\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-3f01c7-cbdfbc885-6f9lk" Sep 10 13:59:25.212832 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:59:25.212728 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/af6ff637-3d66-4a99-bccf-dc9cff523629-metrics-certs\") pod \"maas-api-e2e-ipp-a-3f01c7-cbdfbc885-6f9lk\" (UID: \"af6ff637-3d66-4a99-bccf-dc9cff523629\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-3f01c7-cbdfbc885-6f9lk" Sep 10 13:59:25.287724 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:59:25.287668 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fxgmp_fa53c783-d40a-4024-9f7b-1d0d2b57f6dc/konnectivity-agent/0.log" limit=1024 Sep 10 13:59:25.313553 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:59:25.313520 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/af6ff637-3d66-4a99-bccf-dc9cff523629-openshift-service-ca\") pod \"maas-api-e2e-ipp-a-3f01c7-cbdfbc885-6f9lk\" (UID: \"af6ff637-3d66-4a99-bccf-dc9cff523629\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-3f01c7-cbdfbc885-6f9lk" Sep 10 13:59:25.313715 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:59:25.313560 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/af6ff637-3d66-4a99-bccf-dc9cff523629-metrics-certs\") pod \"maas-api-e2e-ipp-a-3f01c7-cbdfbc885-6f9lk\" (UID: \"af6ff637-3d66-4a99-bccf-dc9cff523629\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-3f01c7-cbdfbc885-6f9lk" Sep 10 13:59:25.313715 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:59:25.313630 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/af6ff637-3d66-4a99-bccf-dc9cff523629-maas-api-tls\") pod \"maas-api-e2e-ipp-a-3f01c7-cbdfbc885-6f9lk\" (UID: \"af6ff637-3d66-4a99-bccf-dc9cff523629\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-3f01c7-cbdfbc885-6f9lk" Sep 10 13:59:25.313715 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:59:25.313694 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-88mmc\" (UniqueName: \"kubernetes.io/projected/af6ff637-3d66-4a99-bccf-dc9cff523629-kube-api-access-88mmc\") pod \"maas-api-e2e-ipp-a-3f01c7-cbdfbc885-6f9lk\" (UID: \"af6ff637-3d66-4a99-bccf-dc9cff523629\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-3f01c7-cbdfbc885-6f9lk" Sep 10 13:59:25.313854 ip-10-0-140-8 kubenswrapper[2585]: E0910 13:59:25.313765 2585 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-ipp-a-3f01c7-serving-cert: secret "maas-api-e2e-ipp-a-3f01c7-serving-cert" not found Sep 10 13:59:25.313854 ip-10-0-140-8 kubenswrapper[2585]: E0910 13:59:25.313833 2585 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/af6ff637-3d66-4a99-bccf-dc9cff523629-maas-api-tls podName:af6ff637-3d66-4a99-bccf-dc9cff523629 nodeName:}" failed. No retries permitted until 2026-09-10 13:59:25.813814609 +0000 UTC m=+1850.686518801 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/af6ff637-3d66-4a99-bccf-dc9cff523629-maas-api-tls") pod "maas-api-e2e-ipp-a-3f01c7-cbdfbc885-6f9lk" (UID: "af6ff637-3d66-4a99-bccf-dc9cff523629") : secret "maas-api-e2e-ipp-a-3f01c7-serving-cert" not found Sep 10 13:59:25.314269 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:59:25.314227 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/af6ff637-3d66-4a99-bccf-dc9cff523629-openshift-service-ca\") pod \"maas-api-e2e-ipp-a-3f01c7-cbdfbc885-6f9lk\" (UID: \"af6ff637-3d66-4a99-bccf-dc9cff523629\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-3f01c7-cbdfbc885-6f9lk" Sep 10 13:59:25.316056 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:59:25.316038 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/af6ff637-3d66-4a99-bccf-dc9cff523629-metrics-certs\") pod \"maas-api-e2e-ipp-a-3f01c7-cbdfbc885-6f9lk\" (UID: \"af6ff637-3d66-4a99-bccf-dc9cff523629\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-3f01c7-cbdfbc885-6f9lk" Sep 10 13:59:25.338788 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:59:25.338767 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-88mmc\" (UniqueName: \"kubernetes.io/projected/af6ff637-3d66-4a99-bccf-dc9cff523629-kube-api-access-88mmc\") pod \"maas-api-e2e-ipp-a-3f01c7-cbdfbc885-6f9lk\" (UID: \"af6ff637-3d66-4a99-bccf-dc9cff523629\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-3f01c7-cbdfbc885-6f9lk" Sep 10 13:59:25.603046 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:59:25.603015 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fxgmp_fa53c783-d40a-4024-9f7b-1d0d2b57f6dc/konnectivity-agent/0.log" limit=1024 Sep 10 13:59:25.791395 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:59:25.791354 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fxgmp_fa53c783-d40a-4024-9f7b-1d0d2b57f6dc/konnectivity-agent/0.log" limit=1024 Sep 10 13:59:25.818803 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:59:25.818771 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/af6ff637-3d66-4a99-bccf-dc9cff523629-maas-api-tls\") pod \"maas-api-e2e-ipp-a-3f01c7-cbdfbc885-6f9lk\" (UID: \"af6ff637-3d66-4a99-bccf-dc9cff523629\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-3f01c7-cbdfbc885-6f9lk" Sep 10 13:59:25.821107 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:59:25.821088 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/af6ff637-3d66-4a99-bccf-dc9cff523629-maas-api-tls\") pod \"maas-api-e2e-ipp-a-3f01c7-cbdfbc885-6f9lk\" (UID: \"af6ff637-3d66-4a99-bccf-dc9cff523629\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-3f01c7-cbdfbc885-6f9lk" Sep 10 13:59:25.892383 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:59:25.892303 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fxgmp_fa53c783-d40a-4024-9f7b-1d0d2b57f6dc/konnectivity-agent/0.log" limit=1024 Sep 10 13:59:26.069284 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:59:26.069227 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-3f01c7-cbdfbc885-6f9lk" Sep 10 13:59:26.377065 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:59:26.377037 2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-3f01c7-cbdfbc885-6f9lk"] Sep 10 13:59:26.379223 ip-10-0-140-8 kubenswrapper[2585]: W0910 13:59:26.379195 2585 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podaf6ff637_3d66_4a99_bccf_dc9cff523629.slice/crio-64411bbc596de7dbdabc56ba3037dbcd119c125c944060e09a94fbb29ac1e2f5 WatchSource:0}: Error finding container 64411bbc596de7dbdabc56ba3037dbcd119c125c944060e09a94fbb29ac1e2f5: Status 404 returned error can't find the container with id 64411bbc596de7dbdabc56ba3037dbcd119c125c944060e09a94fbb29ac1e2f5 Sep 10 13:59:26.380687 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:59:26.380663 2585 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Sep 10 13:59:26.961985 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:59:26.961945 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-3f01c7-cbdfbc885-6f9lk" event={"ID":"af6ff637-3d66-4a99-bccf-dc9cff523629","Type":"ContainerStarted","Data":"b9a989515d21da6518fe8847ea3b60f7aa34d56ec62ae88071225101f1c0ed9e"} Sep 10 13:59:26.962176 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:59:26.962045 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-3f01c7-cbdfbc885-6f9lk" event={"ID":"af6ff637-3d66-4a99-bccf-dc9cff523629","Type":"ContainerStarted","Data":"64411bbc596de7dbdabc56ba3037dbcd119c125c944060e09a94fbb29ac1e2f5"} Sep 10 13:59:26.962176 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:59:26.962075 2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-3f01c7-cbdfbc885-6f9lk" Sep 10 13:59:27.001680 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:59:27.001581 2585 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-3f01c7-cbdfbc885-6f9lk" podStartSLOduration=1.630499918 podStartE2EDuration="2.001564853s" totalImagesPullingTime="371.064935ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 13:59:25 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 13:59:26.999138883 +0000 UTC m=+1851.871843085" watchObservedRunningTime="2026-09-10 13:59:27.001564853 +0000 UTC m=+1851.874269054" Sep 10 13:59:32.973927 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:59:32.973898 2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-3f01c7-cbdfbc885-6f9lk" Sep 10 13:59:48.859512 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:59:48.859477 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fxgmp_fa53c783-d40a-4024-9f7b-1d0d2b57f6dc/konnectivity-agent/0.log" limit=1024 Sep 10 13:59:49.578371 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:59:49.578239 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fxgmp_fa53c783-d40a-4024-9f7b-1d0d2b57f6dc/konnectivity-agent/0.log" limit=1024 Sep 10 13:59:49.932795 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:59:49.932705 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fxgmp_fa53c783-d40a-4024-9f7b-1d0d2b57f6dc/konnectivity-agent/0.log" limit=1024 Sep 10 13:59:50.356428 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:59:50.356399 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fxgmp_fa53c783-d40a-4024-9f7b-1d0d2b57f6dc/konnectivity-agent/0.log" limit=1024 Sep 10 13:59:52.774914 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:59:52.774878 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fxgmp_fa53c783-d40a-4024-9f7b-1d0d2b57f6dc/konnectivity-agent/0.log" limit=1024 Sep 10 13:59:52.875210 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:59:52.875174 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fxgmp_fa53c783-d40a-4024-9f7b-1d0d2b57f6dc/konnectivity-agent/0.log" limit=1024 Sep 10 13:59:54.049376 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:59:54.049346 2585 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-b0741a-686c995db9-kx2bd"] Sep 10 13:59:54.053056 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:59:54.052989 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-b0741a-686c995db9-kx2bd" Sep 10 13:59:54.055641 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:59:54.055625 2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ipp-b-b0741a-serving-cert\"" Sep 10 13:59:54.063308 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:59:54.063276 2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-b0741a-686c995db9-kx2bd"] Sep 10 13:59:54.085310 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:59:54.085277 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fxgmp_fa53c783-d40a-4024-9f7b-1d0d2b57f6dc/konnectivity-agent/0.log" limit=1024 Sep 10 13:59:54.094369 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:59:54.094341 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-flnll\" (UniqueName: \"kubernetes.io/projected/bc1998d8-6acd-4867-af9a-807dcaa131c4-kube-api-access-flnll\") pod \"maas-api-e2e-ipp-b-b0741a-686c995db9-kx2bd\" (UID: \"bc1998d8-6acd-4867-af9a-807dcaa131c4\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-b0741a-686c995db9-kx2bd" Sep 10 13:59:54.094519 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:59:54.094401 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/bc1998d8-6acd-4867-af9a-807dcaa131c4-maas-api-tls\") pod \"maas-api-e2e-ipp-b-b0741a-686c995db9-kx2bd\" (UID: \"bc1998d8-6acd-4867-af9a-807dcaa131c4\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-b0741a-686c995db9-kx2bd" Sep 10 13:59:54.094519 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:59:54.094431 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/bc1998d8-6acd-4867-af9a-807dcaa131c4-metrics-certs\") pod \"maas-api-e2e-ipp-b-b0741a-686c995db9-kx2bd\" (UID: \"bc1998d8-6acd-4867-af9a-807dcaa131c4\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-b0741a-686c995db9-kx2bd" Sep 10 13:59:54.094611 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:59:54.094533 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/bc1998d8-6acd-4867-af9a-807dcaa131c4-openshift-service-ca\") pod \"maas-api-e2e-ipp-b-b0741a-686c995db9-kx2bd\" (UID: \"bc1998d8-6acd-4867-af9a-807dcaa131c4\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-b0741a-686c995db9-kx2bd" Sep 10 13:59:54.188043 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:59:54.188002 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fxgmp_fa53c783-d40a-4024-9f7b-1d0d2b57f6dc/konnectivity-agent/0.log" limit=1024 Sep 10 13:59:54.195638 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:59:54.195612 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/bc1998d8-6acd-4867-af9a-807dcaa131c4-metrics-certs\") pod \"maas-api-e2e-ipp-b-b0741a-686c995db9-kx2bd\" (UID: \"bc1998d8-6acd-4867-af9a-807dcaa131c4\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-b0741a-686c995db9-kx2bd" Sep 10 13:59:54.195731 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:59:54.195689 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/bc1998d8-6acd-4867-af9a-807dcaa131c4-openshift-service-ca\") pod \"maas-api-e2e-ipp-b-b0741a-686c995db9-kx2bd\" (UID: \"bc1998d8-6acd-4867-af9a-807dcaa131c4\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-b0741a-686c995db9-kx2bd" Sep 10 13:59:54.195786 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:59:54.195757 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-flnll\" (UniqueName: \"kubernetes.io/projected/bc1998d8-6acd-4867-af9a-807dcaa131c4-kube-api-access-flnll\") pod \"maas-api-e2e-ipp-b-b0741a-686c995db9-kx2bd\" (UID: \"bc1998d8-6acd-4867-af9a-807dcaa131c4\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-b0741a-686c995db9-kx2bd" Sep 10 13:59:54.195833 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:59:54.195808 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/bc1998d8-6acd-4867-af9a-807dcaa131c4-maas-api-tls\") pod \"maas-api-e2e-ipp-b-b0741a-686c995db9-kx2bd\" (UID: \"bc1998d8-6acd-4867-af9a-807dcaa131c4\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-b0741a-686c995db9-kx2bd" Sep 10 13:59:54.196371 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:59:54.196343 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/bc1998d8-6acd-4867-af9a-807dcaa131c4-openshift-service-ca\") pod \"maas-api-e2e-ipp-b-b0741a-686c995db9-kx2bd\" (UID: \"bc1998d8-6acd-4867-af9a-807dcaa131c4\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-b0741a-686c995db9-kx2bd" Sep 10 13:59:54.198102 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:59:54.198082 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/bc1998d8-6acd-4867-af9a-807dcaa131c4-maas-api-tls\") pod \"maas-api-e2e-ipp-b-b0741a-686c995db9-kx2bd\" (UID: \"bc1998d8-6acd-4867-af9a-807dcaa131c4\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-b0741a-686c995db9-kx2bd" Sep 10 13:59:54.198202 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:59:54.198148 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/bc1998d8-6acd-4867-af9a-807dcaa131c4-metrics-certs\") pod \"maas-api-e2e-ipp-b-b0741a-686c995db9-kx2bd\" (UID: \"bc1998d8-6acd-4867-af9a-807dcaa131c4\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-b0741a-686c995db9-kx2bd" Sep 10 13:59:54.205410 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:59:54.205385 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-flnll\" (UniqueName: \"kubernetes.io/projected/bc1998d8-6acd-4867-af9a-807dcaa131c4-kube-api-access-flnll\") pod \"maas-api-e2e-ipp-b-b0741a-686c995db9-kx2bd\" (UID: \"bc1998d8-6acd-4867-af9a-807dcaa131c4\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-b0741a-686c995db9-kx2bd" Sep 10 13:59:54.287341 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:59:54.287309 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fxgmp_fa53c783-d40a-4024-9f7b-1d0d2b57f6dc/konnectivity-agent/0.log" limit=1024 Sep 10 13:59:54.366297 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:59:54.366240 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-b0741a-686c995db9-kx2bd" Sep 10 13:59:54.654581 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:59:54.654554 2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-b0741a-686c995db9-kx2bd"] Sep 10 13:59:54.656167 ip-10-0-140-8 kubenswrapper[2585]: W0910 13:59:54.656140 2585 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbc1998d8_6acd_4867_af9a_807dcaa131c4.slice/crio-a885cd67150f018e79a86486a4c38c8d5af205b77aba20d2ec561a179a74799f WatchSource:0}: Error finding container a885cd67150f018e79a86486a4c38c8d5af205b77aba20d2ec561a179a74799f: Status 404 returned error can't find the container with id a885cd67150f018e79a86486a4c38c8d5af205b77aba20d2ec561a179a74799f Sep 10 13:59:54.803015 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:59:54.802976 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fxgmp_fa53c783-d40a-4024-9f7b-1d0d2b57f6dc/konnectivity-agent/0.log" limit=1024 Sep 10 13:59:54.960754 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:59:54.960680 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fxgmp_fa53c783-d40a-4024-9f7b-1d0d2b57f6dc/konnectivity-agent/0.log" limit=1024 Sep 10 13:59:55.123350 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:59:55.123316 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-b0741a-686c995db9-kx2bd" event={"ID":"bc1998d8-6acd-4867-af9a-807dcaa131c4","Type":"ContainerStarted","Data":"89d4f703c89c34c4538ea43da72187742a9a1e7ec67839b9e43e074810b502e7"} Sep 10 13:59:55.123772 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:59:55.123371 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-b0741a-686c995db9-kx2bd" event={"ID":"bc1998d8-6acd-4867-af9a-807dcaa131c4","Type":"ContainerStarted","Data":"a885cd67150f018e79a86486a4c38c8d5af205b77aba20d2ec561a179a74799f"} Sep 10 13:59:55.123772 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:59:55.123392 2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-b0741a-686c995db9-kx2bd" Sep 10 13:59:55.165879 ip-10-0-140-8 kubenswrapper[2585]: I0910 13:59:55.165832 2585 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-b0741a-686c995db9-kx2bd" podStartSLOduration=0.854985906 podStartE2EDuration="1.165815402s" totalImagesPullingTime="310.829496ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 13:59:54 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 13:59:55.163984393 +0000 UTC m=+1880.036688585" watchObservedRunningTime="2026-09-10 13:59:55.165815402 +0000 UTC m=+1880.038519659" Sep 10 14:00:00.148163 ip-10-0-140-8 kubenswrapper[2585]: I0910 14:00:00.148064 2585 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29817480-j7vlj"] Sep 10 14:00:00.152558 ip-10-0-140-8 kubenswrapper[2585]: I0910 14:00:00.152534 2585 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-3f01c7-29817480-scs69"] Sep 10 14:00:00.152702 ip-10-0-140-8 kubenswrapper[2585]: I0910 14:00:00.152638 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29817480-j7vlj" Sep 10 14:00:00.155334 ip-10-0-140-8 kubenswrapper[2585]: I0910 14:00:00.155311 2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-7gcbr\"" Sep 10 14:00:00.156015 ip-10-0-140-8 kubenswrapper[2585]: I0910 14:00:00.155996 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-3f01c7-29817480-scs69" Sep 10 14:00:00.163580 ip-10-0-140-8 kubenswrapper[2585]: I0910 14:00:00.163494 2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29817480-j7vlj"] Sep 10 14:00:00.167227 ip-10-0-140-8 kubenswrapper[2585]: I0910 14:00:00.167204 2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-3f01c7-29817480-scs69"] Sep 10 14:00:00.168650 ip-10-0-140-8 kubenswrapper[2585]: I0910 14:00:00.168625 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29817480-j7vlj" Sep 10 14:00:00.178386 ip-10-0-140-8 kubenswrapper[2585]: I0910 14:00:00.178363 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-3f01c7-29817480-scs69" Sep 10 14:00:00.464101 ip-10-0-140-8 kubenswrapper[2585]: I0910 14:00:00.464069 2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29817480-j7vlj"] Sep 10 14:00:00.465122 ip-10-0-140-8 kubenswrapper[2585]: W0910 14:00:00.465084 2585 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poded18dcb9_5e17_4e57_8ad4_56f4c8b7cb17.slice/crio-ca7ba6b92f86d0b3ef811c08fa0ca4d90b91cc6c664d82169434c7635d6b032f WatchSource:0}: Error finding container ca7ba6b92f86d0b3ef811c08fa0ca4d90b91cc6c664d82169434c7635d6b032f: Status 404 returned error can't find the container with id ca7ba6b92f86d0b3ef811c08fa0ca4d90b91cc6c664d82169434c7635d6b032f Sep 10 14:00:00.466810 ip-10-0-140-8 kubenswrapper[2585]: I0910 14:00:00.466786 2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-3f01c7-29817480-scs69"] Sep 10 14:00:00.467784 ip-10-0-140-8 kubenswrapper[2585]: W0910 14:00:00.467757 2585 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb793b1f8_dd93_455f_808b_7cf44e4bbea5.slice/crio-997a76be9e2f713d5a187facab67c2803b543e93b4fab1e5bbca2807f9683dac WatchSource:0}: Error finding container 997a76be9e2f713d5a187facab67c2803b543e93b4fab1e5bbca2807f9683dac: Status 404 returned error can't find the container with id 997a76be9e2f713d5a187facab67c2803b543e93b4fab1e5bbca2807f9683dac Sep 10 14:00:01.135734 ip-10-0-140-8 kubenswrapper[2585]: I0910 14:00:01.135704 2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-b0741a-686c995db9-kx2bd" Sep 10 14:00:01.161700 ip-10-0-140-8 kubenswrapper[2585]: I0910 14:00:01.161663 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29817480-j7vlj" event={"ID":"ed18dcb9-5e17-4e57-8ad4-56f4c8b7cb17","Type":"ContainerStarted","Data":"184048665a1b8779fb5c6977a310a0d1a742998c0b05b65fb08a318f1042cf30"} Sep 10 14:00:01.162150 ip-10-0-140-8 kubenswrapper[2585]: I0910 14:00:01.161723 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29817480-j7vlj" event={"ID":"ed18dcb9-5e17-4e57-8ad4-56f4c8b7cb17","Type":"ContainerStarted","Data":"ca7ba6b92f86d0b3ef811c08fa0ca4d90b91cc6c664d82169434c7635d6b032f"} Sep 10 14:00:01.163470 ip-10-0-140-8 kubenswrapper[2585]: I0910 14:00:01.163439 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-3f01c7-29817480-scs69" event={"ID":"b793b1f8-dd93-455f-808b-7cf44e4bbea5","Type":"ContainerStarted","Data":"ab937828b4b46d9381c0df25bcd0db7586e1ea2c5e2ab349d963ca5c72235c39"} Sep 10 14:00:01.163624 ip-10-0-140-8 kubenswrapper[2585]: I0910 14:00:01.163491 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-3f01c7-29817480-scs69" event={"ID":"b793b1f8-dd93-455f-808b-7cf44e4bbea5","Type":"ContainerStarted","Data":"997a76be9e2f713d5a187facab67c2803b543e93b4fab1e5bbca2807f9683dac"} Sep 10 14:00:01.183681 ip-10-0-140-8 kubenswrapper[2585]: I0910 14:00:01.183628 2585 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29817480-j7vlj" podStartSLOduration=1.183607166 podStartE2EDuration="1.183607166s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 14:00:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 14:00:01.182718957 +0000 UTC m=+1886.055423160" watchObservedRunningTime="2026-09-10 14:00:01.183607166 +0000 UTC m=+1886.056311368" Sep 10 14:00:01.202440 ip-10-0-140-8 kubenswrapper[2585]: I0910 14:00:01.202391 2585 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-3f01c7-29817480-scs69" podStartSLOduration=1.202374857 podStartE2EDuration="1.202374857s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 14:00:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 14:00:01.20017714 +0000 UTC m=+1886.072881342" watchObservedRunningTime="2026-09-10 14:00:01.202374857 +0000 UTC m=+1886.075079059" Sep 10 14:00:02.170416 ip-10-0-140-8 kubenswrapper[2585]: I0910 14:00:02.170379 2585 generic.go:356] "Generic (PLEG): container finished" podID="b793b1f8-dd93-455f-808b-7cf44e4bbea5" containerID="ab937828b4b46d9381c0df25bcd0db7586e1ea2c5e2ab349d963ca5c72235c39" exitCode=0 Sep 10 14:00:02.170905 ip-10-0-140-8 kubenswrapper[2585]: I0910 14:00:02.170458 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-3f01c7-29817480-scs69" event={"ID":"b793b1f8-dd93-455f-808b-7cf44e4bbea5","Type":"ContainerDied","Data":"ab937828b4b46d9381c0df25bcd0db7586e1ea2c5e2ab349d963ca5c72235c39"} Sep 10 14:00:02.172641 ip-10-0-140-8 kubenswrapper[2585]: I0910 14:00:02.172602 2585 generic.go:356] "Generic (PLEG): container finished" podID="ed18dcb9-5e17-4e57-8ad4-56f4c8b7cb17" containerID="184048665a1b8779fb5c6977a310a0d1a742998c0b05b65fb08a318f1042cf30" exitCode=0 Sep 10 14:00:02.172854 ip-10-0-140-8 kubenswrapper[2585]: I0910 14:00:02.172819 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29817480-j7vlj" event={"ID":"ed18dcb9-5e17-4e57-8ad4-56f4c8b7cb17","Type":"ContainerDied","Data":"184048665a1b8779fb5c6977a310a0d1a742998c0b05b65fb08a318f1042cf30"} Sep 10 14:00:03.329935 ip-10-0-140-8 kubenswrapper[2585]: I0910 14:00:03.329912 2585 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-3f01c7-29817480-scs69" Sep 10 14:00:03.346622 ip-10-0-140-8 kubenswrapper[2585]: I0910 14:00:03.346574 2585 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29817480-j7vlj" Sep 10 14:00:04.186309 ip-10-0-140-8 kubenswrapper[2585]: I0910 14:00:04.186282 2585 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29817480-j7vlj" Sep 10 14:00:04.186488 ip-10-0-140-8 kubenswrapper[2585]: I0910 14:00:04.186364 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29817480-j7vlj" event={"ID":"ed18dcb9-5e17-4e57-8ad4-56f4c8b7cb17","Type":"ContainerDied","Data":"ca7ba6b92f86d0b3ef811c08fa0ca4d90b91cc6c664d82169434c7635d6b032f"} Sep 10 14:00:04.186488 ip-10-0-140-8 kubenswrapper[2585]: I0910 14:00:04.186414 2585 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="ca7ba6b92f86d0b3ef811c08fa0ca4d90b91cc6c664d82169434c7635d6b032f" Sep 10 14:00:04.187868 ip-10-0-140-8 kubenswrapper[2585]: I0910 14:00:04.187846 2585 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-3f01c7-29817480-scs69" Sep 10 14:00:04.187990 ip-10-0-140-8 kubenswrapper[2585]: I0910 14:00:04.187875 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-3f01c7-29817480-scs69" event={"ID":"b793b1f8-dd93-455f-808b-7cf44e4bbea5","Type":"ContainerDied","Data":"997a76be9e2f713d5a187facab67c2803b543e93b4fab1e5bbca2807f9683dac"} Sep 10 14:00:04.187990 ip-10-0-140-8 kubenswrapper[2585]: I0910 14:00:04.187910 2585 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="997a76be9e2f713d5a187facab67c2803b543e93b4fab1e5bbca2807f9683dac" Sep 10 14:00:17.739350 ip-10-0-140-8 kubenswrapper[2585]: I0910 14:00:17.739307 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fxgmp_fa53c783-d40a-4024-9f7b-1d0d2b57f6dc/konnectivity-agent/0.log" limit=1024 Sep 10 14:00:27.817630 ip-10-0-140-8 kubenswrapper[2585]: I0910 14:00:27.817591 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fxgmp_fa53c783-d40a-4024-9f7b-1d0d2b57f6dc/konnectivity-agent/0.log" limit=1024 Sep 10 14:00:54.216463 ip-10-0-140-8 kubenswrapper[2585]: I0910 14:00:54.216428 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fxgmp_fa53c783-d40a-4024-9f7b-1d0d2b57f6dc/konnectivity-agent/0.log" limit=1024 Sep 10 14:00:55.389048 ip-10-0-140-8 kubenswrapper[2585]: I0910 14:00:55.389011 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fxgmp_fa53c783-d40a-4024-9f7b-1d0d2b57f6dc/konnectivity-agent/0.log" limit=1024 Sep 10 14:00:55.492954 ip-10-0-140-8 kubenswrapper[2585]: I0910 14:00:55.492872 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fxgmp_fa53c783-d40a-4024-9f7b-1d0d2b57f6dc/konnectivity-agent/0.log" limit=1024 Sep 10 14:00:58.421457 ip-10-0-140-8 kubenswrapper[2585]: I0910 14:00:58.421418 2585 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-eb3d6d-c744fb444-vvtzm"] Sep 10 14:00:58.421945 ip-10-0-140-8 kubenswrapper[2585]: I0910 14:00:58.421863 2585 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ed18dcb9-5e17-4e57-8ad4-56f4c8b7cb17" containerName="cleanup" Sep 10 14:00:58.421945 ip-10-0-140-8 kubenswrapper[2585]: I0910 14:00:58.421877 2585 state_mem.go:107] "Deleted CPUSet assignment" podUID="ed18dcb9-5e17-4e57-8ad4-56f4c8b7cb17" containerName="cleanup" Sep 10 14:00:58.421945 ip-10-0-140-8 kubenswrapper[2585]: I0910 14:00:58.421888 2585 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b793b1f8-dd93-455f-808b-7cf44e4bbea5" containerName="cleanup" Sep 10 14:00:58.421945 ip-10-0-140-8 kubenswrapper[2585]: I0910 14:00:58.421893 2585 state_mem.go:107] "Deleted CPUSet assignment" podUID="b793b1f8-dd93-455f-808b-7cf44e4bbea5" containerName="cleanup" Sep 10 14:00:58.422168 ip-10-0-140-8 kubenswrapper[2585]: I0910 14:00:58.421957 2585 memory_manager.go:368] "RemoveStaleState removing state" podUID="ed18dcb9-5e17-4e57-8ad4-56f4c8b7cb17" containerName="cleanup" Sep 10 14:00:58.422168 ip-10-0-140-8 kubenswrapper[2585]: I0910 14:00:58.421968 2585 memory_manager.go:368] "RemoveStaleState removing state" podUID="b793b1f8-dd93-455f-808b-7cf44e4bbea5" containerName="cleanup" Sep 10 14:00:58.425176 ip-10-0-140-8 kubenswrapper[2585]: I0910 14:00:58.425154 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-eb3d6d-c744fb444-vvtzm" Sep 10 14:00:58.428076 ip-10-0-140-8 kubenswrapper[2585]: I0910 14:00:58.428045 2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ipp-cleanup-eb3d6d-serving-cert\"" Sep 10 14:00:58.436450 ip-10-0-140-8 kubenswrapper[2585]: I0910 14:00:58.436429 2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-eb3d6d-c744fb444-vvtzm"] Sep 10 14:00:58.594880 ip-10-0-140-8 kubenswrapper[2585]: I0910 14:00:58.594842 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-t4fv9\" (UniqueName: \"kubernetes.io/projected/3e7718e6-425b-416f-8fc7-8f36603cba2c-kube-api-access-t4fv9\") pod \"maas-api-e2e-ipp-cleanup-eb3d6d-c744fb444-vvtzm\" (UID: \"3e7718e6-425b-416f-8fc7-8f36603cba2c\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-eb3d6d-c744fb444-vvtzm" Sep 10 14:00:58.594880 ip-10-0-140-8 kubenswrapper[2585]: I0910 14:00:58.594884 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3e7718e6-425b-416f-8fc7-8f36603cba2c-metrics-certs\") pod \"maas-api-e2e-ipp-cleanup-eb3d6d-c744fb444-vvtzm\" (UID: \"3e7718e6-425b-416f-8fc7-8f36603cba2c\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-eb3d6d-c744fb444-vvtzm" Sep 10 14:00:58.595084 ip-10-0-140-8 kubenswrapper[2585]: I0910 14:00:58.594959 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3e7718e6-425b-416f-8fc7-8f36603cba2c-maas-api-tls\") pod \"maas-api-e2e-ipp-cleanup-eb3d6d-c744fb444-vvtzm\" (UID: \"3e7718e6-425b-416f-8fc7-8f36603cba2c\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-eb3d6d-c744fb444-vvtzm" Sep 10 14:00:58.595084 ip-10-0-140-8 kubenswrapper[2585]: I0910 14:00:58.595014 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/3e7718e6-425b-416f-8fc7-8f36603cba2c-openshift-service-ca\") pod \"maas-api-e2e-ipp-cleanup-eb3d6d-c744fb444-vvtzm\" (UID: \"3e7718e6-425b-416f-8fc7-8f36603cba2c\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-eb3d6d-c744fb444-vvtzm" Sep 10 14:00:58.696522 ip-10-0-140-8 kubenswrapper[2585]: I0910 14:00:58.696442 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t4fv9\" (UniqueName: \"kubernetes.io/projected/3e7718e6-425b-416f-8fc7-8f36603cba2c-kube-api-access-t4fv9\") pod \"maas-api-e2e-ipp-cleanup-eb3d6d-c744fb444-vvtzm\" (UID: \"3e7718e6-425b-416f-8fc7-8f36603cba2c\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-eb3d6d-c744fb444-vvtzm" Sep 10 14:00:58.696522 ip-10-0-140-8 kubenswrapper[2585]: I0910 14:00:58.696498 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3e7718e6-425b-416f-8fc7-8f36603cba2c-metrics-certs\") pod \"maas-api-e2e-ipp-cleanup-eb3d6d-c744fb444-vvtzm\" (UID: \"3e7718e6-425b-416f-8fc7-8f36603cba2c\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-eb3d6d-c744fb444-vvtzm" Sep 10 14:00:58.696703 ip-10-0-140-8 kubenswrapper[2585]: I0910 14:00:58.696543 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3e7718e6-425b-416f-8fc7-8f36603cba2c-maas-api-tls\") pod \"maas-api-e2e-ipp-cleanup-eb3d6d-c744fb444-vvtzm\" (UID: \"3e7718e6-425b-416f-8fc7-8f36603cba2c\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-eb3d6d-c744fb444-vvtzm" Sep 10 14:00:58.696703 ip-10-0-140-8 kubenswrapper[2585]: I0910 14:00:58.696573 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/3e7718e6-425b-416f-8fc7-8f36603cba2c-openshift-service-ca\") pod \"maas-api-e2e-ipp-cleanup-eb3d6d-c744fb444-vvtzm\" (UID: \"3e7718e6-425b-416f-8fc7-8f36603cba2c\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-eb3d6d-c744fb444-vvtzm" Sep 10 14:00:58.697267 ip-10-0-140-8 kubenswrapper[2585]: I0910 14:00:58.697215 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/3e7718e6-425b-416f-8fc7-8f36603cba2c-openshift-service-ca\") pod \"maas-api-e2e-ipp-cleanup-eb3d6d-c744fb444-vvtzm\" (UID: \"3e7718e6-425b-416f-8fc7-8f36603cba2c\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-eb3d6d-c744fb444-vvtzm" Sep 10 14:00:58.698979 ip-10-0-140-8 kubenswrapper[2585]: I0910 14:00:58.698946 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3e7718e6-425b-416f-8fc7-8f36603cba2c-metrics-certs\") pod \"maas-api-e2e-ipp-cleanup-eb3d6d-c744fb444-vvtzm\" (UID: \"3e7718e6-425b-416f-8fc7-8f36603cba2c\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-eb3d6d-c744fb444-vvtzm" Sep 10 14:00:58.699075 ip-10-0-140-8 kubenswrapper[2585]: I0910 14:00:58.699040 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3e7718e6-425b-416f-8fc7-8f36603cba2c-maas-api-tls\") pod \"maas-api-e2e-ipp-cleanup-eb3d6d-c744fb444-vvtzm\" (UID: \"3e7718e6-425b-416f-8fc7-8f36603cba2c\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-eb3d6d-c744fb444-vvtzm" Sep 10 14:00:58.722755 ip-10-0-140-8 kubenswrapper[2585]: I0910 14:00:58.722731 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-t4fv9\" (UniqueName: \"kubernetes.io/projected/3e7718e6-425b-416f-8fc7-8f36603cba2c-kube-api-access-t4fv9\") pod \"maas-api-e2e-ipp-cleanup-eb3d6d-c744fb444-vvtzm\" (UID: \"3e7718e6-425b-416f-8fc7-8f36603cba2c\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-eb3d6d-c744fb444-vvtzm" Sep 10 14:00:58.737591 ip-10-0-140-8 kubenswrapper[2585]: I0910 14:00:58.737569 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-eb3d6d-c744fb444-vvtzm" Sep 10 14:00:59.017491 ip-10-0-140-8 kubenswrapper[2585]: I0910 14:00:59.017462 2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-eb3d6d-c744fb444-vvtzm"] Sep 10 14:00:59.018814 ip-10-0-140-8 kubenswrapper[2585]: W0910 14:00:59.018784 2585 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3e7718e6_425b_416f_8fc7_8f36603cba2c.slice/crio-1834fb8bdc5f4da8e628d9c60a80f185d420ffe94b55d0efc336f5025f3356b7 WatchSource:0}: Error finding container 1834fb8bdc5f4da8e628d9c60a80f185d420ffe94b55d0efc336f5025f3356b7: Status 404 returned error can't find the container with id 1834fb8bdc5f4da8e628d9c60a80f185d420ffe94b55d0efc336f5025f3356b7 Sep 10 14:00:59.492423 ip-10-0-140-8 kubenswrapper[2585]: I0910 14:00:59.492389 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-eb3d6d-c744fb444-vvtzm" event={"ID":"3e7718e6-425b-416f-8fc7-8f36603cba2c","Type":"ContainerStarted","Data":"1834fb8bdc5f4da8e628d9c60a80f185d420ffe94b55d0efc336f5025f3356b7"} Sep 10 14:00:59.537687 ip-10-0-140-8 kubenswrapper[2585]: I0910 14:00:59.537641 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fxgmp_fa53c783-d40a-4024-9f7b-1d0d2b57f6dc/konnectivity-agent/0.log" limit=1024 Sep 10 14:00:59.640389 ip-10-0-140-8 kubenswrapper[2585]: I0910 14:00:59.640362 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fxgmp_fa53c783-d40a-4024-9f7b-1d0d2b57f6dc/konnectivity-agent/0.log" limit=1024 Sep 10 14:01:00.499028 ip-10-0-140-8 kubenswrapper[2585]: I0910 14:01:00.498992 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-eb3d6d-c744fb444-vvtzm" event={"ID":"3e7718e6-425b-416f-8fc7-8f36603cba2c","Type":"ContainerStarted","Data":"ca0f28433f0fccb04456bd43783c81d43c67361199d474321d30215e9569d718"} Sep 10 14:01:00.499517 ip-10-0-140-8 kubenswrapper[2585]: I0910 14:01:00.499168 2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-eb3d6d-c744fb444-vvtzm" Sep 10 14:01:00.521598 ip-10-0-140-8 kubenswrapper[2585]: I0910 14:01:00.521553 2585 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-eb3d6d-c744fb444-vvtzm" podStartSLOduration=1.996540669 podStartE2EDuration="2.521540032s" totalImagesPullingTime="524.999363ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 14:00:58 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 14:01:00.520898591 +0000 UTC m=+1945.393602791" watchObservedRunningTime="2026-09-10 14:01:00.521540032 +0000 UTC m=+1945.394244232" Sep 10 14:01:06.510524 ip-10-0-140-8 kubenswrapper[2585]: I0910 14:01:06.510488 2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-eb3d6d-c744fb444-vvtzm" Sep 10 14:01:18.110351 ip-10-0-140-8 kubenswrapper[2585]: I0910 14:01:18.110319 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fxgmp_fa53c783-d40a-4024-9f7b-1d0d2b57f6dc/konnectivity-agent/0.log" limit=1024 Sep 10 14:01:18.197678 ip-10-0-140-8 kubenswrapper[2585]: I0910 14:01:18.197642 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fxgmp_fa53c783-d40a-4024-9f7b-1d0d2b57f6dc/konnectivity-agent/0.log" limit=1024 Sep 10 14:01:22.376306 ip-10-0-140-8 kubenswrapper[2585]: I0910 14:01:22.376271 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fxgmp_fa53c783-d40a-4024-9f7b-1d0d2b57f6dc/konnectivity-agent/0.log" limit=1024 Sep 10 14:01:29.407922 ip-10-0-140-8 kubenswrapper[2585]: I0910 14:01:29.407842 2585 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-eb3d6d-c744fb444-vvtzm"] Sep 10 14:01:29.408306 ip-10-0-140-8 kubenswrapper[2585]: I0910 14:01:29.408094 2585 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-eb3d6d-c744fb444-vvtzm" podUID="3e7718e6-425b-416f-8fc7-8f36603cba2c" containerName="maas-api" containerID="cri-o://ca0f28433f0fccb04456bd43783c81d43c67361199d474321d30215e9569d718" gracePeriod=30 Sep 10 14:01:29.474822 ip-10-0-140-8 kubenswrapper[2585]: I0910 14:01:29.474790 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fxgmp_fa53c783-d40a-4024-9f7b-1d0d2b57f6dc/konnectivity-agent/0.log" limit=1024 Sep 10 14:01:29.624762 ip-10-0-140-8 kubenswrapper[2585]: I0910 14:01:29.624725 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fxgmp_fa53c783-d40a-4024-9f7b-1d0d2b57f6dc/konnectivity-agent/0.log" limit=1024 Sep 10 14:01:29.679481 ip-10-0-140-8 kubenswrapper[2585]: I0910 14:01:29.679445 2585 generic.go:356] "Generic (PLEG): container finished" podID="3e7718e6-425b-416f-8fc7-8f36603cba2c" containerID="ca0f28433f0fccb04456bd43783c81d43c67361199d474321d30215e9569d718" exitCode=0 Sep 10 14:01:29.679656 ip-10-0-140-8 kubenswrapper[2585]: I0910 14:01:29.679521 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-eb3d6d-c744fb444-vvtzm" event={"ID":"3e7718e6-425b-416f-8fc7-8f36603cba2c","Type":"ContainerDied","Data":"ca0f28433f0fccb04456bd43783c81d43c67361199d474321d30215e9569d718"} Sep 10 14:01:29.679656 ip-10-0-140-8 kubenswrapper[2585]: I0910 14:01:29.679583 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-eb3d6d-c744fb444-vvtzm" event={"ID":"3e7718e6-425b-416f-8fc7-8f36603cba2c","Type":"ContainerDied","Data":"1834fb8bdc5f4da8e628d9c60a80f185d420ffe94b55d0efc336f5025f3356b7"} Sep 10 14:01:29.679656 ip-10-0-140-8 kubenswrapper[2585]: I0910 14:01:29.679607 2585 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="1834fb8bdc5f4da8e628d9c60a80f185d420ffe94b55d0efc336f5025f3356b7" Sep 10 14:01:29.683130 ip-10-0-140-8 kubenswrapper[2585]: I0910 14:01:29.683107 2585 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-eb3d6d-c744fb444-vvtzm" Sep 10 14:01:29.784826 ip-10-0-140-8 kubenswrapper[2585]: I0910 14:01:29.784800 2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3e7718e6-425b-416f-8fc7-8f36603cba2c-metrics-certs\") pod \"3e7718e6-425b-416f-8fc7-8f36603cba2c\" (UID: \"3e7718e6-425b-416f-8fc7-8f36603cba2c\") " Sep 10 14:01:29.785031 ip-10-0-140-8 kubenswrapper[2585]: I0910 14:01:29.784852 2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-t4fv9\" (UniqueName: \"kubernetes.io/projected/3e7718e6-425b-416f-8fc7-8f36603cba2c-kube-api-access-t4fv9\") pod \"3e7718e6-425b-416f-8fc7-8f36603cba2c\" (UID: \"3e7718e6-425b-416f-8fc7-8f36603cba2c\") " Sep 10 14:01:29.785031 ip-10-0-140-8 kubenswrapper[2585]: I0910 14:01:29.784882 2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3e7718e6-425b-416f-8fc7-8f36603cba2c-maas-api-tls\") pod \"3e7718e6-425b-416f-8fc7-8f36603cba2c\" (UID: \"3e7718e6-425b-416f-8fc7-8f36603cba2c\") " Sep 10 14:01:29.785031 ip-10-0-140-8 kubenswrapper[2585]: I0910 14:01:29.784928 2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/3e7718e6-425b-416f-8fc7-8f36603cba2c-openshift-service-ca\") pod \"3e7718e6-425b-416f-8fc7-8f36603cba2c\" (UID: \"3e7718e6-425b-416f-8fc7-8f36603cba2c\") " Sep 10 14:01:29.785450 ip-10-0-140-8 kubenswrapper[2585]: I0910 14:01:29.785416 2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3e7718e6-425b-416f-8fc7-8f36603cba2c-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "3e7718e6-425b-416f-8fc7-8f36603cba2c" (UID: "3e7718e6-425b-416f-8fc7-8f36603cba2c"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 10 14:01:29.787035 ip-10-0-140-8 kubenswrapper[2585]: I0910 14:01:29.787000 2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3e7718e6-425b-416f-8fc7-8f36603cba2c-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "3e7718e6-425b-416f-8fc7-8f36603cba2c" (UID: "3e7718e6-425b-416f-8fc7-8f36603cba2c"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 10 14:01:29.787134 ip-10-0-140-8 kubenswrapper[2585]: I0910 14:01:29.787072 2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3e7718e6-425b-416f-8fc7-8f36603cba2c-kube-api-access-t4fv9" (OuterVolumeSpecName: "kube-api-access-t4fv9") pod "3e7718e6-425b-416f-8fc7-8f36603cba2c" (UID: "3e7718e6-425b-416f-8fc7-8f36603cba2c"). InnerVolumeSpecName "kube-api-access-t4fv9". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 10 14:01:29.787134 ip-10-0-140-8 kubenswrapper[2585]: I0910 14:01:29.787103 2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3e7718e6-425b-416f-8fc7-8f36603cba2c-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "3e7718e6-425b-416f-8fc7-8f36603cba2c" (UID: "3e7718e6-425b-416f-8fc7-8f36603cba2c"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 10 14:01:29.886346 ip-10-0-140-8 kubenswrapper[2585]: I0910 14:01:29.886310 2585 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-t4fv9\" (UniqueName: \"kubernetes.io/projected/3e7718e6-425b-416f-8fc7-8f36603cba2c-kube-api-access-t4fv9\") on node \"ip-10-0-140-8.ec2.internal\" DevicePath \"\"" Sep 10 14:01:29.886346 ip-10-0-140-8 kubenswrapper[2585]: I0910 14:01:29.886339 2585 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3e7718e6-425b-416f-8fc7-8f36603cba2c-maas-api-tls\") on node \"ip-10-0-140-8.ec2.internal\" DevicePath \"\"" Sep 10 14:01:29.886346 ip-10-0-140-8 kubenswrapper[2585]: I0910 14:01:29.886349 2585 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/3e7718e6-425b-416f-8fc7-8f36603cba2c-openshift-service-ca\") on node \"ip-10-0-140-8.ec2.internal\" DevicePath \"\"" Sep 10 14:01:29.886346 ip-10-0-140-8 kubenswrapper[2585]: I0910 14:01:29.886357 2585 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3e7718e6-425b-416f-8fc7-8f36603cba2c-metrics-certs\") on node \"ip-10-0-140-8.ec2.internal\" DevicePath \"\"" Sep 10 14:01:30.685798 ip-10-0-140-8 kubenswrapper[2585]: I0910 14:01:30.685767 2585 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-eb3d6d-c744fb444-vvtzm" Sep 10 14:01:30.713023 ip-10-0-140-8 kubenswrapper[2585]: I0910 14:01:30.712990 2585 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-eb3d6d-c744fb444-vvtzm"] Sep 10 14:01:30.719138 ip-10-0-140-8 kubenswrapper[2585]: I0910 14:01:30.719116 2585 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-eb3d6d-c744fb444-vvtzm"] Sep 10 14:01:31.647293 ip-10-0-140-8 kubenswrapper[2585]: I0910 14:01:31.647232 2585 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="3e7718e6-425b-416f-8fc7-8f36603cba2c" path="/var/lib/kubelet/pods/3e7718e6-425b-416f-8fc7-8f36603cba2c/volumes" Sep 10 14:01:31.758764 ip-10-0-140-8 kubenswrapper[2585]: I0910 14:01:31.758735 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fxgmp_fa53c783-d40a-4024-9f7b-1d0d2b57f6dc/konnectivity-agent/0.log" limit=1024 Sep 10 14:01:31.851288 ip-10-0-140-8 kubenswrapper[2585]: I0910 14:01:31.851214 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fxgmp_fa53c783-d40a-4024-9f7b-1d0d2b57f6dc/konnectivity-agent/0.log" limit=1024 Sep 10 14:01:38.033769 ip-10-0-140-8 kubenswrapper[2585]: I0910 14:01:38.033735 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fxgmp_fa53c783-d40a-4024-9f7b-1d0d2b57f6dc/konnectivity-agent/0.log" limit=1024 Sep 10 14:01:38.120509 ip-10-0-140-8 kubenswrapper[2585]: I0910 14:01:38.120479 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fxgmp_fa53c783-d40a-4024-9f7b-1d0d2b57f6dc/konnectivity-agent/0.log" limit=1024 Sep 10 14:01:38.340030 ip-10-0-140-8 kubenswrapper[2585]: I0910 14:01:38.339999 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fxgmp_fa53c783-d40a-4024-9f7b-1d0d2b57f6dc/konnectivity-agent/0.log" limit=1024 Sep 10 14:01:40.951749 ip-10-0-140-8 kubenswrapper[2585]: I0910 14:01:40.951714 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fxgmp_fa53c783-d40a-4024-9f7b-1d0d2b57f6dc/konnectivity-agent/0.log" limit=1024 Sep 10 14:01:42.225378 ip-10-0-140-8 kubenswrapper[2585]: I0910 14:01:42.225341 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fxgmp_fa53c783-d40a-4024-9f7b-1d0d2b57f6dc/konnectivity-agent/0.log" limit=1024 Sep 10 14:01:55.138009 ip-10-0-140-8 kubenswrapper[2585]: I0910 14:01:55.137967 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fxgmp_fa53c783-d40a-4024-9f7b-1d0d2b57f6dc/konnectivity-agent/0.log" limit=1024 Sep 10 14:01:57.825129 ip-10-0-140-8 kubenswrapper[2585]: I0910 14:01:57.825091 2585 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-3f01c7-cbdfbc885-6f9lk"] Sep 10 14:01:57.825606 ip-10-0-140-8 kubenswrapper[2585]: I0910 14:01:57.825337 2585 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-3f01c7-cbdfbc885-6f9lk" podUID="af6ff637-3d66-4a99-bccf-dc9cff523629" containerName="maas-api" containerID="cri-o://b9a989515d21da6518fe8847ea3b60f7aa34d56ec62ae88071225101f1c0ed9e" gracePeriod=30 Sep 10 14:01:57.855360 ip-10-0-140-8 kubenswrapper[2585]: I0910 14:01:57.855318 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fxgmp_fa53c783-d40a-4024-9f7b-1d0d2b57f6dc/konnectivity-agent/0.log" limit=1024 Sep 10 14:01:57.873948 ip-10-0-140-8 kubenswrapper[2585]: I0910 14:01:57.873919 2585 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-3f01c7-29817480-scs69"] Sep 10 14:01:57.880028 ip-10-0-140-8 kubenswrapper[2585]: I0910 14:01:57.879999 2585 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-3f01c7-29817480-scs69"] Sep 10 14:01:57.967992 ip-10-0-140-8 kubenswrapper[2585]: I0910 14:01:57.967958 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fxgmp_fa53c783-d40a-4024-9f7b-1d0d2b57f6dc/konnectivity-agent/0.log" limit=1024 Sep 10 14:01:57.968234 ip-10-0-140-8 kubenswrapper[2585]: I0910 14:01:57.968209 2585 prober.go:120] "Probe failed" probeType="Readiness" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-3f01c7-cbdfbc885-6f9lk" podUID="af6ff637-3d66-4a99-bccf-dc9cff523629" containerName="maas-api" probeResult="failure" output="Get \"https://10.133.0.80:8443/health\": dial tcp 10.133.0.80:8443: connect: connection refused" Sep 10 14:01:58.071112 ip-10-0-140-8 kubenswrapper[2585]: I0910 14:01:58.071081 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fxgmp_fa53c783-d40a-4024-9f7b-1d0d2b57f6dc/konnectivity-agent/0.log" limit=1024 Sep 10 14:01:58.077459 ip-10-0-140-8 kubenswrapper[2585]: I0910 14:01:58.077410 2585 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-3f01c7-cbdfbc885-6f9lk" Sep 10 14:01:58.146889 ip-10-0-140-8 kubenswrapper[2585]: I0910 14:01:58.146856 2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/af6ff637-3d66-4a99-bccf-dc9cff523629-metrics-certs\") pod \"af6ff637-3d66-4a99-bccf-dc9cff523629\" (UID: \"af6ff637-3d66-4a99-bccf-dc9cff523629\") " Sep 10 14:01:58.147038 ip-10-0-140-8 kubenswrapper[2585]: I0910 14:01:58.146905 2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-88mmc\" (UniqueName: \"kubernetes.io/projected/af6ff637-3d66-4a99-bccf-dc9cff523629-kube-api-access-88mmc\") pod \"af6ff637-3d66-4a99-bccf-dc9cff523629\" (UID: \"af6ff637-3d66-4a99-bccf-dc9cff523629\") " Sep 10 14:01:58.147038 ip-10-0-140-8 kubenswrapper[2585]: I0910 14:01:58.146949 2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/af6ff637-3d66-4a99-bccf-dc9cff523629-openshift-service-ca\") pod \"af6ff637-3d66-4a99-bccf-dc9cff523629\" (UID: \"af6ff637-3d66-4a99-bccf-dc9cff523629\") " Sep 10 14:01:58.147038 ip-10-0-140-8 kubenswrapper[2585]: I0910 14:01:58.146992 2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/af6ff637-3d66-4a99-bccf-dc9cff523629-maas-api-tls\") pod \"af6ff637-3d66-4a99-bccf-dc9cff523629\" (UID: \"af6ff637-3d66-4a99-bccf-dc9cff523629\") " Sep 10 14:01:58.147328 ip-10-0-140-8 kubenswrapper[2585]: I0910 14:01:58.147301 2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/af6ff637-3d66-4a99-bccf-dc9cff523629-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "af6ff637-3d66-4a99-bccf-dc9cff523629" (UID: "af6ff637-3d66-4a99-bccf-dc9cff523629"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 10 14:01:58.149400 ip-10-0-140-8 kubenswrapper[2585]: I0910 14:01:58.149368 2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/af6ff637-3d66-4a99-bccf-dc9cff523629-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "af6ff637-3d66-4a99-bccf-dc9cff523629" (UID: "af6ff637-3d66-4a99-bccf-dc9cff523629"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 10 14:01:58.149490 ip-10-0-140-8 kubenswrapper[2585]: I0910 14:01:58.149457 2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/af6ff637-3d66-4a99-bccf-dc9cff523629-kube-api-access-88mmc" (OuterVolumeSpecName: "kube-api-access-88mmc") pod "af6ff637-3d66-4a99-bccf-dc9cff523629" (UID: "af6ff637-3d66-4a99-bccf-dc9cff523629"). InnerVolumeSpecName "kube-api-access-88mmc". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 10 14:01:58.149529 ip-10-0-140-8 kubenswrapper[2585]: I0910 14:01:58.149501 2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/af6ff637-3d66-4a99-bccf-dc9cff523629-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "af6ff637-3d66-4a99-bccf-dc9cff523629" (UID: "af6ff637-3d66-4a99-bccf-dc9cff523629"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 10 14:01:58.167344 ip-10-0-140-8 kubenswrapper[2585]: I0910 14:01:58.167312 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fxgmp_fa53c783-d40a-4024-9f7b-1d0d2b57f6dc/konnectivity-agent/0.log" limit=1024 Sep 10 14:01:58.247213 ip-10-0-140-8 kubenswrapper[2585]: I0910 14:01:58.247175 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fxgmp_fa53c783-d40a-4024-9f7b-1d0d2b57f6dc/konnectivity-agent/0.log" limit=1024 Sep 10 14:01:58.248485 ip-10-0-140-8 kubenswrapper[2585]: I0910 14:01:58.248458 2585 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/af6ff637-3d66-4a99-bccf-dc9cff523629-metrics-certs\") on node \"ip-10-0-140-8.ec2.internal\" DevicePath \"\"" Sep 10 14:01:58.248556 ip-10-0-140-8 kubenswrapper[2585]: I0910 14:01:58.248493 2585 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-88mmc\" (UniqueName: \"kubernetes.io/projected/af6ff637-3d66-4a99-bccf-dc9cff523629-kube-api-access-88mmc\") on node \"ip-10-0-140-8.ec2.internal\" DevicePath \"\"" Sep 10 14:01:58.248556 ip-10-0-140-8 kubenswrapper[2585]: I0910 14:01:58.248508 2585 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/af6ff637-3d66-4a99-bccf-dc9cff523629-openshift-service-ca\") on node \"ip-10-0-140-8.ec2.internal\" DevicePath \"\"" Sep 10 14:01:58.248556 ip-10-0-140-8 kubenswrapper[2585]: I0910 14:01:58.248521 2585 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/af6ff637-3d66-4a99-bccf-dc9cff523629-maas-api-tls\") on node \"ip-10-0-140-8.ec2.internal\" DevicePath \"\"" Sep 10 14:01:58.628607 ip-10-0-140-8 kubenswrapper[2585]: I0910 14:01:58.628566 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fxgmp_fa53c783-d40a-4024-9f7b-1d0d2b57f6dc/konnectivity-agent/0.log" limit=1024 Sep 10 14:01:58.848012 ip-10-0-140-8 kubenswrapper[2585]: I0910 14:01:58.847976 2585 generic.go:356] "Generic (PLEG): container finished" podID="af6ff637-3d66-4a99-bccf-dc9cff523629" containerID="b9a989515d21da6518fe8847ea3b60f7aa34d56ec62ae88071225101f1c0ed9e" exitCode=0 Sep 10 14:01:58.848483 ip-10-0-140-8 kubenswrapper[2585]: I0910 14:01:58.848021 2585 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-3f01c7-cbdfbc885-6f9lk" Sep 10 14:01:58.848483 ip-10-0-140-8 kubenswrapper[2585]: I0910 14:01:58.848064 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-3f01c7-cbdfbc885-6f9lk" event={"ID":"af6ff637-3d66-4a99-bccf-dc9cff523629","Type":"ContainerDied","Data":"b9a989515d21da6518fe8847ea3b60f7aa34d56ec62ae88071225101f1c0ed9e"} Sep 10 14:01:58.848483 ip-10-0-140-8 kubenswrapper[2585]: I0910 14:01:58.848116 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-3f01c7-cbdfbc885-6f9lk" event={"ID":"af6ff637-3d66-4a99-bccf-dc9cff523629","Type":"ContainerDied","Data":"64411bbc596de7dbdabc56ba3037dbcd119c125c944060e09a94fbb29ac1e2f5"} Sep 10 14:01:58.848483 ip-10-0-140-8 kubenswrapper[2585]: I0910 14:01:58.848140 2585 scope.go:117] "RemoveContainer" containerID="b9a989515d21da6518fe8847ea3b60f7aa34d56ec62ae88071225101f1c0ed9e" Sep 10 14:01:58.857565 ip-10-0-140-8 kubenswrapper[2585]: I0910 14:01:58.857548 2585 scope.go:117] "RemoveContainer" containerID="b9a989515d21da6518fe8847ea3b60f7aa34d56ec62ae88071225101f1c0ed9e" Sep 10 14:01:58.857802 ip-10-0-140-8 kubenswrapper[2585]: E0910 14:01:58.857782 2585 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b9a989515d21da6518fe8847ea3b60f7aa34d56ec62ae88071225101f1c0ed9e\": container with ID starting with b9a989515d21da6518fe8847ea3b60f7aa34d56ec62ae88071225101f1c0ed9e not found: ID does not exist" containerID="b9a989515d21da6518fe8847ea3b60f7aa34d56ec62ae88071225101f1c0ed9e" Sep 10 14:01:58.857851 ip-10-0-140-8 kubenswrapper[2585]: I0910 14:01:58.857810 2585 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b9a989515d21da6518fe8847ea3b60f7aa34d56ec62ae88071225101f1c0ed9e"} err="failed to get container status \"b9a989515d21da6518fe8847ea3b60f7aa34d56ec62ae88071225101f1c0ed9e\": rpc error: code = NotFound desc = could not find container \"b9a989515d21da6518fe8847ea3b60f7aa34d56ec62ae88071225101f1c0ed9e\": container with ID starting with b9a989515d21da6518fe8847ea3b60f7aa34d56ec62ae88071225101f1c0ed9e not found: ID does not exist" Sep 10 14:01:58.921100 ip-10-0-140-8 kubenswrapper[2585]: I0910 14:01:58.921034 2585 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-3f01c7-cbdfbc885-6f9lk"] Sep 10 14:01:58.925470 ip-10-0-140-8 kubenswrapper[2585]: I0910 14:01:58.925446 2585 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-3f01c7-cbdfbc885-6f9lk"] Sep 10 14:01:59.647948 ip-10-0-140-8 kubenswrapper[2585]: I0910 14:01:59.647914 2585 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="af6ff637-3d66-4a99-bccf-dc9cff523629" path="/var/lib/kubelet/pods/af6ff637-3d66-4a99-bccf-dc9cff523629/volumes" Sep 10 14:01:59.648225 ip-10-0-140-8 kubenswrapper[2585]: I0910 14:01:59.648211 2585 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="b793b1f8-dd93-455f-808b-7cf44e4bbea5" path="/var/lib/kubelet/pods/b793b1f8-dd93-455f-808b-7cf44e4bbea5/volumes" Sep 10 14:02:04.598785 ip-10-0-140-8 kubenswrapper[2585]: I0910 14:02:04.598746 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fxgmp_fa53c783-d40a-4024-9f7b-1d0d2b57f6dc/konnectivity-agent/0.log" limit=1024 Sep 10 14:02:14.442290 ip-10-0-140-8 kubenswrapper[2585]: I0910 14:02:14.442230 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fxgmp_fa53c783-d40a-4024-9f7b-1d0d2b57f6dc/konnectivity-agent/0.log" limit=1024 Sep 10 14:02:21.491344 ip-10-0-140-8 kubenswrapper[2585]: I0910 14:02:21.491304 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fxgmp_fa53c783-d40a-4024-9f7b-1d0d2b57f6dc/konnectivity-agent/0.log" limit=1024 Sep 10 14:02:32.015188 ip-10-0-140-8 kubenswrapper[2585]: I0910 14:02:32.015153 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fxgmp_fa53c783-d40a-4024-9f7b-1d0d2b57f6dc/konnectivity-agent/0.log" limit=1024 Sep 10 14:02:32.115546 ip-10-0-140-8 kubenswrapper[2585]: I0910 14:02:32.115510 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fxgmp_fa53c783-d40a-4024-9f7b-1d0d2b57f6dc/konnectivity-agent/0.log" limit=1024 Sep 10 14:02:41.434061 ip-10-0-140-8 kubenswrapper[2585]: I0910 14:02:41.434019 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fxgmp_fa53c783-d40a-4024-9f7b-1d0d2b57f6dc/konnectivity-agent/0.log" limit=1024 Sep 10 14:02:42.595437 ip-10-0-140-8 kubenswrapper[2585]: I0910 14:02:42.595403 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fxgmp_fa53c783-d40a-4024-9f7b-1d0d2b57f6dc/konnectivity-agent/0.log" limit=1024 Sep 10 14:02:49.635477 ip-10-0-140-8 kubenswrapper[2585]: I0910 14:02:49.635442 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fxgmp_fa53c783-d40a-4024-9f7b-1d0d2b57f6dc/konnectivity-agent/0.log" limit=1024 Sep 10 14:02:49.764259 ip-10-0-140-8 kubenswrapper[2585]: I0910 14:02:49.764210 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fxgmp_fa53c783-d40a-4024-9f7b-1d0d2b57f6dc/konnectivity-agent/0.log" limit=1024 Sep 10 14:02:49.952762 ip-10-0-140-8 kubenswrapper[2585]: I0910 14:02:49.952667 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fxgmp_fa53c783-d40a-4024-9f7b-1d0d2b57f6dc/konnectivity-agent/0.log" limit=1024 Sep 10 14:02:51.027983 ip-10-0-140-8 kubenswrapper[2585]: I0910 14:02:51.027943 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fxgmp_fa53c783-d40a-4024-9f7b-1d0d2b57f6dc/konnectivity-agent/0.log" limit=1024 Sep 10 14:03:01.171994 ip-10-0-140-8 kubenswrapper[2585]: I0910 14:03:01.171919 2585 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-b0741a-686c995db9-kx2bd"] Sep 10 14:03:01.172405 ip-10-0-140-8 kubenswrapper[2585]: I0910 14:03:01.172144 2585 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-b0741a-686c995db9-kx2bd" podUID="bc1998d8-6acd-4867-af9a-807dcaa131c4" containerName="maas-api" containerID="cri-o://89d4f703c89c34c4538ea43da72187742a9a1e7ec67839b9e43e074810b502e7" gracePeriod=30 Sep 10 14:03:01.437207 ip-10-0-140-8 kubenswrapper[2585]: I0910 14:03:01.437143 2585 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-b0741a-686c995db9-kx2bd" Sep 10 14:03:01.546976 ip-10-0-140-8 kubenswrapper[2585]: I0910 14:03:01.546942 2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/bc1998d8-6acd-4867-af9a-807dcaa131c4-openshift-service-ca\") pod \"bc1998d8-6acd-4867-af9a-807dcaa131c4\" (UID: \"bc1998d8-6acd-4867-af9a-807dcaa131c4\") " Sep 10 14:03:01.547168 ip-10-0-140-8 kubenswrapper[2585]: I0910 14:03:01.546991 2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-flnll\" (UniqueName: \"kubernetes.io/projected/bc1998d8-6acd-4867-af9a-807dcaa131c4-kube-api-access-flnll\") pod \"bc1998d8-6acd-4867-af9a-807dcaa131c4\" (UID: \"bc1998d8-6acd-4867-af9a-807dcaa131c4\") " Sep 10 14:03:01.547168 ip-10-0-140-8 kubenswrapper[2585]: I0910 14:03:01.547014 2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/bc1998d8-6acd-4867-af9a-807dcaa131c4-maas-api-tls\") pod \"bc1998d8-6acd-4867-af9a-807dcaa131c4\" (UID: \"bc1998d8-6acd-4867-af9a-807dcaa131c4\") " Sep 10 14:03:01.547168 ip-10-0-140-8 kubenswrapper[2585]: I0910 14:03:01.547050 2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/bc1998d8-6acd-4867-af9a-807dcaa131c4-metrics-certs\") pod \"bc1998d8-6acd-4867-af9a-807dcaa131c4\" (UID: \"bc1998d8-6acd-4867-af9a-807dcaa131c4\") " Sep 10 14:03:01.547428 ip-10-0-140-8 kubenswrapper[2585]: I0910 14:03:01.547401 2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bc1998d8-6acd-4867-af9a-807dcaa131c4-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "bc1998d8-6acd-4867-af9a-807dcaa131c4" (UID: "bc1998d8-6acd-4867-af9a-807dcaa131c4"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 10 14:03:01.549120 ip-10-0-140-8 kubenswrapper[2585]: I0910 14:03:01.549086 2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/bc1998d8-6acd-4867-af9a-807dcaa131c4-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "bc1998d8-6acd-4867-af9a-807dcaa131c4" (UID: "bc1998d8-6acd-4867-af9a-807dcaa131c4"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 10 14:03:01.549232 ip-10-0-140-8 kubenswrapper[2585]: I0910 14:03:01.549142 2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/bc1998d8-6acd-4867-af9a-807dcaa131c4-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "bc1998d8-6acd-4867-af9a-807dcaa131c4" (UID: "bc1998d8-6acd-4867-af9a-807dcaa131c4"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 10 14:03:01.549232 ip-10-0-140-8 kubenswrapper[2585]: I0910 14:03:01.549145 2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bc1998d8-6acd-4867-af9a-807dcaa131c4-kube-api-access-flnll" (OuterVolumeSpecName: "kube-api-access-flnll") pod "bc1998d8-6acd-4867-af9a-807dcaa131c4" (UID: "bc1998d8-6acd-4867-af9a-807dcaa131c4"). InnerVolumeSpecName "kube-api-access-flnll". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 10 14:03:01.648290 ip-10-0-140-8 kubenswrapper[2585]: I0910 14:03:01.648260 2585 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/bc1998d8-6acd-4867-af9a-807dcaa131c4-openshift-service-ca\") on node \"ip-10-0-140-8.ec2.internal\" DevicePath \"\"" Sep 10 14:03:01.648290 ip-10-0-140-8 kubenswrapper[2585]: I0910 14:03:01.648285 2585 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-flnll\" (UniqueName: \"kubernetes.io/projected/bc1998d8-6acd-4867-af9a-807dcaa131c4-kube-api-access-flnll\") on node \"ip-10-0-140-8.ec2.internal\" DevicePath \"\"" Sep 10 14:03:01.648290 ip-10-0-140-8 kubenswrapper[2585]: I0910 14:03:01.648295 2585 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/bc1998d8-6acd-4867-af9a-807dcaa131c4-maas-api-tls\") on node \"ip-10-0-140-8.ec2.internal\" DevicePath \"\"" Sep 10 14:03:01.648530 ip-10-0-140-8 kubenswrapper[2585]: I0910 14:03:01.648306 2585 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/bc1998d8-6acd-4867-af9a-807dcaa131c4-metrics-certs\") on node \"ip-10-0-140-8.ec2.internal\" DevicePath \"\"" Sep 10 14:03:02.203294 ip-10-0-140-8 kubenswrapper[2585]: I0910 14:03:02.203261 2585 generic.go:356] "Generic (PLEG): container finished" podID="bc1998d8-6acd-4867-af9a-807dcaa131c4" containerID="89d4f703c89c34c4538ea43da72187742a9a1e7ec67839b9e43e074810b502e7" exitCode=0 Sep 10 14:03:02.203745 ip-10-0-140-8 kubenswrapper[2585]: I0910 14:03:02.203316 2585 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-b0741a-686c995db9-kx2bd" Sep 10 14:03:02.203745 ip-10-0-140-8 kubenswrapper[2585]: I0910 14:03:02.203342 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-b0741a-686c995db9-kx2bd" event={"ID":"bc1998d8-6acd-4867-af9a-807dcaa131c4","Type":"ContainerDied","Data":"89d4f703c89c34c4538ea43da72187742a9a1e7ec67839b9e43e074810b502e7"} Sep 10 14:03:02.203745 ip-10-0-140-8 kubenswrapper[2585]: I0910 14:03:02.203404 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-b0741a-686c995db9-kx2bd" event={"ID":"bc1998d8-6acd-4867-af9a-807dcaa131c4","Type":"ContainerDied","Data":"a885cd67150f018e79a86486a4c38c8d5af205b77aba20d2ec561a179a74799f"} Sep 10 14:03:02.203745 ip-10-0-140-8 kubenswrapper[2585]: I0910 14:03:02.203427 2585 scope.go:117] "RemoveContainer" containerID="89d4f703c89c34c4538ea43da72187742a9a1e7ec67839b9e43e074810b502e7" Sep 10 14:03:02.213183 ip-10-0-140-8 kubenswrapper[2585]: I0910 14:03:02.213167 2585 scope.go:117] "RemoveContainer" containerID="89d4f703c89c34c4538ea43da72187742a9a1e7ec67839b9e43e074810b502e7" Sep 10 14:03:02.213492 ip-10-0-140-8 kubenswrapper[2585]: E0910 14:03:02.213472 2585 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"89d4f703c89c34c4538ea43da72187742a9a1e7ec67839b9e43e074810b502e7\": container with ID starting with 89d4f703c89c34c4538ea43da72187742a9a1e7ec67839b9e43e074810b502e7 not found: ID does not exist" containerID="89d4f703c89c34c4538ea43da72187742a9a1e7ec67839b9e43e074810b502e7" Sep 10 14:03:02.213566 ip-10-0-140-8 kubenswrapper[2585]: I0910 14:03:02.213500 2585 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"89d4f703c89c34c4538ea43da72187742a9a1e7ec67839b9e43e074810b502e7"} err="failed to get container status \"89d4f703c89c34c4538ea43da72187742a9a1e7ec67839b9e43e074810b502e7\": rpc error: code = NotFound desc = could not find container \"89d4f703c89c34c4538ea43da72187742a9a1e7ec67839b9e43e074810b502e7\": container with ID starting with 89d4f703c89c34c4538ea43da72187742a9a1e7ec67839b9e43e074810b502e7 not found: ID does not exist" Sep 10 14:03:02.233999 ip-10-0-140-8 kubenswrapper[2585]: I0910 14:03:02.233969 2585 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-b0741a-686c995db9-kx2bd"] Sep 10 14:03:02.236024 ip-10-0-140-8 kubenswrapper[2585]: I0910 14:03:02.236001 2585 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-b0741a-686c995db9-kx2bd"] Sep 10 14:03:02.616424 ip-10-0-140-8 kubenswrapper[2585]: I0910 14:03:02.616391 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fxgmp_fa53c783-d40a-4024-9f7b-1d0d2b57f6dc/konnectivity-agent/0.log" limit=1024 Sep 10 14:03:02.712051 ip-10-0-140-8 kubenswrapper[2585]: I0910 14:03:02.712018 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fxgmp_fa53c783-d40a-4024-9f7b-1d0d2b57f6dc/konnectivity-agent/0.log" limit=1024 Sep 10 14:03:03.647238 ip-10-0-140-8 kubenswrapper[2585]: I0910 14:03:03.647208 2585 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="bc1998d8-6acd-4867-af9a-807dcaa131c4" path="/var/lib/kubelet/pods/bc1998d8-6acd-4867-af9a-807dcaa131c4/volumes" Sep 10 14:03:10.415677 ip-10-0-140-8 kubenswrapper[2585]: I0910 14:03:10.415637 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fxgmp_fa53c783-d40a-4024-9f7b-1d0d2b57f6dc/konnectivity-agent/0.log" limit=1024 Sep 10 14:03:10.510852 ip-10-0-140-8 kubenswrapper[2585]: I0910 14:03:10.510821 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fxgmp_fa53c783-d40a-4024-9f7b-1d0d2b57f6dc/konnectivity-agent/0.log" limit=1024 Sep 10 14:03:10.711789 ip-10-0-140-8 kubenswrapper[2585]: I0910 14:03:10.711703 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fxgmp_fa53c783-d40a-4024-9f7b-1d0d2b57f6dc/konnectivity-agent/0.log" limit=1024 Sep 10 14:03:11.062661 ip-10-0-140-8 kubenswrapper[2585]: I0910 14:03:11.062625 2585 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-7a11e9-vm8td"] Sep 10 14:03:11.063096 ip-10-0-140-8 kubenswrapper[2585]: I0910 14:03:11.063078 2585 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="af6ff637-3d66-4a99-bccf-dc9cff523629" containerName="maas-api" Sep 10 14:03:11.063187 ip-10-0-140-8 kubenswrapper[2585]: I0910 14:03:11.063101 2585 state_mem.go:107] "Deleted CPUSet assignment" podUID="af6ff637-3d66-4a99-bccf-dc9cff523629" containerName="maas-api" Sep 10 14:03:11.063187 ip-10-0-140-8 kubenswrapper[2585]: I0910 14:03:11.063123 2585 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3e7718e6-425b-416f-8fc7-8f36603cba2c" containerName="maas-api" Sep 10 14:03:11.063187 ip-10-0-140-8 kubenswrapper[2585]: I0910 14:03:11.063131 2585 state_mem.go:107] "Deleted CPUSet assignment" podUID="3e7718e6-425b-416f-8fc7-8f36603cba2c" containerName="maas-api" Sep 10 14:03:11.063187 ip-10-0-140-8 kubenswrapper[2585]: I0910 14:03:11.063164 2585 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="bc1998d8-6acd-4867-af9a-807dcaa131c4" containerName="maas-api" Sep 10 14:03:11.063187 ip-10-0-140-8 kubenswrapper[2585]: I0910 14:03:11.063175 2585 state_mem.go:107] "Deleted CPUSet assignment" podUID="bc1998d8-6acd-4867-af9a-807dcaa131c4" containerName="maas-api" Sep 10 14:03:11.063477 ip-10-0-140-8 kubenswrapper[2585]: I0910 14:03:11.063274 2585 memory_manager.go:368] "RemoveStaleState removing state" podUID="af6ff637-3d66-4a99-bccf-dc9cff523629" containerName="maas-api" Sep 10 14:03:11.063477 ip-10-0-140-8 kubenswrapper[2585]: I0910 14:03:11.063290 2585 memory_manager.go:368] "RemoveStaleState removing state" podUID="3e7718e6-425b-416f-8fc7-8f36603cba2c" containerName="maas-api" Sep 10 14:03:11.063477 ip-10-0-140-8 kubenswrapper[2585]: I0910 14:03:11.063303 2585 memory_manager.go:368] "RemoveStaleState removing state" podUID="bc1998d8-6acd-4867-af9a-807dcaa131c4" containerName="maas-api" Sep 10 14:03:11.066539 ip-10-0-140-8 kubenswrapper[2585]: I0910 14:03:11.066519 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-7a11e9-vm8td" Sep 10 14:03:11.069478 ip-10-0-140-8 kubenswrapper[2585]: I0910 14:03:11.069460 2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Sep 10 14:03:11.069744 ip-10-0-140-8 kubenswrapper[2585]: I0910 14:03:11.069727 2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-7gcbr\"" Sep 10 14:03:11.082300 ip-10-0-140-8 kubenswrapper[2585]: I0910 14:03:11.082276 2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-7a11e9-vm8td"] Sep 10 14:03:11.214555 ip-10-0-140-8 kubenswrapper[2585]: I0910 14:03:11.214519 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fxgmp_fa53c783-d40a-4024-9f7b-1d0d2b57f6dc/konnectivity-agent/0.log" limit=1024 Sep 10 14:03:11.238620 ip-10-0-140-8 kubenswrapper[2585]: I0910 14:03:11.238572 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/047277cb-ad53-4971-9e34-219c5748928f-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w3-7a11e9-vm8td\" (UID: \"047277cb-ad53-4971-9e34-219c5748928f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-7a11e9-vm8td" Sep 10 14:03:11.339146 ip-10-0-140-8 kubenswrapper[2585]: I0910 14:03:11.339058 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/047277cb-ad53-4971-9e34-219c5748928f-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w3-7a11e9-vm8td\" (UID: \"047277cb-ad53-4971-9e34-219c5748928f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-7a11e9-vm8td" Sep 10 14:03:11.339758 ip-10-0-140-8 kubenswrapper[2585]: I0910 14:03:11.339733 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/047277cb-ad53-4971-9e34-219c5748928f-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w3-7a11e9-vm8td\" (UID: \"047277cb-ad53-4971-9e34-219c5748928f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-7a11e9-vm8td" Sep 10 14:03:11.379634 ip-10-0-140-8 kubenswrapper[2585]: I0910 14:03:11.379605 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-7a11e9-vm8td" Sep 10 14:03:11.666962 ip-10-0-140-8 kubenswrapper[2585]: I0910 14:03:11.666934 2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-7a11e9-vm8td"] Sep 10 14:03:11.669111 ip-10-0-140-8 kubenswrapper[2585]: W0910 14:03:11.669075 2585 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod047277cb_ad53_4971_9e34_219c5748928f.slice/crio-6596d073471c2c6514ca862d7d77ec5b97a2311630d00330d07934c7938c0a5f WatchSource:0}: Error finding container 6596d073471c2c6514ca862d7d77ec5b97a2311630d00330d07934c7938c0a5f: Status 404 returned error can't find the container with id 6596d073471c2c6514ca862d7d77ec5b97a2311630d00330d07934c7938c0a5f Sep 10 14:03:12.261598 ip-10-0-140-8 kubenswrapper[2585]: I0910 14:03:12.261565 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-7a11e9-vm8td" event={"ID":"047277cb-ad53-4971-9e34-219c5748928f","Type":"ContainerStarted","Data":"2fc8d5ce5b4265a9496b8488b876cee8162c6aa05c8a3589a489ce3a0dc9a956"} Sep 10 14:03:12.261789 ip-10-0-140-8 kubenswrapper[2585]: I0910 14:03:12.261623 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-7a11e9-vm8td" event={"ID":"047277cb-ad53-4971-9e34-219c5748928f","Type":"ContainerStarted","Data":"6596d073471c2c6514ca862d7d77ec5b97a2311630d00330d07934c7938c0a5f"} Sep 10 14:03:12.298689 ip-10-0-140-8 kubenswrapper[2585]: I0910 14:03:12.298637 2585 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-7a11e9-vm8td" podStartSLOduration=1.298619375 podStartE2EDuration="1.298619375s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 14:03:11 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 14:03:12.296685499 +0000 UTC m=+2077.169389711" watchObservedRunningTime="2026-09-10 14:03:12.298619375 +0000 UTC m=+2077.171323575" Sep 10 14:03:13.268895 ip-10-0-140-8 kubenswrapper[2585]: I0910 14:03:13.268862 2585 generic.go:356] "Generic (PLEG): container finished" podID="047277cb-ad53-4971-9e34-219c5748928f" containerID="2fc8d5ce5b4265a9496b8488b876cee8162c6aa05c8a3589a489ce3a0dc9a956" exitCode=0 Sep 10 14:03:13.269407 ip-10-0-140-8 kubenswrapper[2585]: I0910 14:03:13.268948 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-7a11e9-vm8td" event={"ID":"047277cb-ad53-4971-9e34-219c5748928f","Type":"ContainerDied","Data":"2fc8d5ce5b4265a9496b8488b876cee8162c6aa05c8a3589a489ce3a0dc9a956"} Sep 10 14:03:14.405543 ip-10-0-140-8 kubenswrapper[2585]: I0910 14:03:14.405518 2585 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-7a11e9-vm8td" Sep 10 14:03:14.468895 ip-10-0-140-8 kubenswrapper[2585]: I0910 14:03:14.468863 2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/047277cb-ad53-4971-9e34-219c5748928f-maas-api-service-ca\") pod \"047277cb-ad53-4971-9e34-219c5748928f\" (UID: \"047277cb-ad53-4971-9e34-219c5748928f\") " Sep 10 14:03:14.469181 ip-10-0-140-8 kubenswrapper[2585]: I0910 14:03:14.469160 2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/047277cb-ad53-4971-9e34-219c5748928f-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "047277cb-ad53-4971-9e34-219c5748928f" (UID: "047277cb-ad53-4971-9e34-219c5748928f"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 10 14:03:14.569829 ip-10-0-140-8 kubenswrapper[2585]: I0910 14:03:14.569782 2585 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/047277cb-ad53-4971-9e34-219c5748928f-maas-api-service-ca\") on node \"ip-10-0-140-8.ec2.internal\" DevicePath \"\"" Sep 10 14:03:15.280850 ip-10-0-140-8 kubenswrapper[2585]: I0910 14:03:15.280821 2585 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-7a11e9-vm8td" Sep 10 14:03:15.281051 ip-10-0-140-8 kubenswrapper[2585]: I0910 14:03:15.280851 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-7a11e9-vm8td" event={"ID":"047277cb-ad53-4971-9e34-219c5748928f","Type":"ContainerDied","Data":"6596d073471c2c6514ca862d7d77ec5b97a2311630d00330d07934c7938c0a5f"} Sep 10 14:03:15.281051 ip-10-0-140-8 kubenswrapper[2585]: I0910 14:03:15.280893 2585 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="6596d073471c2c6514ca862d7d77ec5b97a2311630d00330d07934c7938c0a5f" Sep 10 14:03:22.191105 ip-10-0-140-8 kubenswrapper[2585]: I0910 14:03:22.191074 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fxgmp_fa53c783-d40a-4024-9f7b-1d0d2b57f6dc/konnectivity-agent/0.log" limit=1024 Sep 10 14:03:22.294671 ip-10-0-140-8 kubenswrapper[2585]: I0910 14:03:22.294638 2585 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w3-7a11e9-6864c5c8cd-x6gvq"] Sep 10 14:03:22.294909 ip-10-0-140-8 kubenswrapper[2585]: I0910 14:03:22.294863 2585 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-7a11e9-6864c5c8cd-x6gvq" podUID="bf622551-e014-4e8b-8df2-6e3723f1df26" containerName="payload-pre-processing" containerID="cri-o://37c6018fefadafe3a14e52a267ff667fa9e63185d82a4e21dc44c0c6c7bf6a29" gracePeriod=30 Sep 10 14:03:22.361615 ip-10-0-140-8 kubenswrapper[2585]: I0910 14:03:22.361575 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fxgmp_fa53c783-d40a-4024-9f7b-1d0d2b57f6dc/konnectivity-agent/0.log" limit=1024 Sep 10 14:03:22.464796 ip-10-0-140-8 kubenswrapper[2585]: I0910 14:03:22.464767 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fxgmp_fa53c783-d40a-4024-9f7b-1d0d2b57f6dc/konnectivity-agent/0.log" limit=1024 Sep 10 14:03:22.554645 ip-10-0-140-8 kubenswrapper[2585]: I0910 14:03:22.554615 2585 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-7a11e9-6864c5c8cd-x6gvq" Sep 10 14:03:22.586345 ip-10-0-140-8 kubenswrapper[2585]: I0910 14:03:22.586311 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fxgmp_fa53c783-d40a-4024-9f7b-1d0d2b57f6dc/konnectivity-agent/0.log" limit=1024 Sep 10 14:03:22.624353 ip-10-0-140-8 kubenswrapper[2585]: I0910 14:03:22.624315 2585 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-7a11e9-vm8td"] Sep 10 14:03:22.630119 ip-10-0-140-8 kubenswrapper[2585]: I0910 14:03:22.630090 2585 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-7a11e9-vm8td"] Sep 10 14:03:22.631811 ip-10-0-140-8 kubenswrapper[2585]: I0910 14:03:22.631791 2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bf622551-e014-4e8b-8df2-6e3723f1df26-plugins-config-volume\") pod \"bf622551-e014-4e8b-8df2-6e3723f1df26\" (UID: \"bf622551-e014-4e8b-8df2-6e3723f1df26\") " Sep 10 14:03:22.631903 ip-10-0-140-8 kubenswrapper[2585]: I0910 14:03:22.631843 2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-tdq8v\" (UniqueName: \"kubernetes.io/projected/bf622551-e014-4e8b-8df2-6e3723f1df26-kube-api-access-tdq8v\") pod \"bf622551-e014-4e8b-8df2-6e3723f1df26\" (UID: \"bf622551-e014-4e8b-8df2-6e3723f1df26\") " Sep 10 14:03:22.632195 ip-10-0-140-8 kubenswrapper[2585]: I0910 14:03:22.632171 2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bf622551-e014-4e8b-8df2-6e3723f1df26-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "bf622551-e014-4e8b-8df2-6e3723f1df26" (UID: "bf622551-e014-4e8b-8df2-6e3723f1df26"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 10 14:03:22.633976 ip-10-0-140-8 kubenswrapper[2585]: I0910 14:03:22.633947 2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bf622551-e014-4e8b-8df2-6e3723f1df26-kube-api-access-tdq8v" (OuterVolumeSpecName: "kube-api-access-tdq8v") pod "bf622551-e014-4e8b-8df2-6e3723f1df26" (UID: "bf622551-e014-4e8b-8df2-6e3723f1df26"). InnerVolumeSpecName "kube-api-access-tdq8v". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 10 14:03:22.652631 ip-10-0-140-8 kubenswrapper[2585]: I0910 14:03:22.652602 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fxgmp_fa53c783-d40a-4024-9f7b-1d0d2b57f6dc/konnectivity-agent/0.log" limit=1024 Sep 10 14:03:22.733004 ip-10-0-140-8 kubenswrapper[2585]: I0910 14:03:22.732923 2585 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bf622551-e014-4e8b-8df2-6e3723f1df26-plugins-config-volume\") on node \"ip-10-0-140-8.ec2.internal\" DevicePath \"\"" Sep 10 14:03:22.733004 ip-10-0-140-8 kubenswrapper[2585]: I0910 14:03:22.732951 2585 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tdq8v\" (UniqueName: \"kubernetes.io/projected/bf622551-e014-4e8b-8df2-6e3723f1df26-kube-api-access-tdq8v\") on node \"ip-10-0-140-8.ec2.internal\" DevicePath \"\"" Sep 10 14:03:23.049815 ip-10-0-140-8 kubenswrapper[2585]: I0910 14:03:23.049732 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fxgmp_fa53c783-d40a-4024-9f7b-1d0d2b57f6dc/konnectivity-agent/0.log" limit=1024 Sep 10 14:03:23.325316 ip-10-0-140-8 kubenswrapper[2585]: I0910 14:03:23.325283 2585 generic.go:356] "Generic (PLEG): container finished" podID="bf622551-e014-4e8b-8df2-6e3723f1df26" containerID="37c6018fefadafe3a14e52a267ff667fa9e63185d82a4e21dc44c0c6c7bf6a29" exitCode=0 Sep 10 14:03:23.325708 ip-10-0-140-8 kubenswrapper[2585]: I0910 14:03:23.325325 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-7a11e9-6864c5c8cd-x6gvq" event={"ID":"bf622551-e014-4e8b-8df2-6e3723f1df26","Type":"ContainerDied","Data":"37c6018fefadafe3a14e52a267ff667fa9e63185d82a4e21dc44c0c6c7bf6a29"} Sep 10 14:03:23.325708 ip-10-0-140-8 kubenswrapper[2585]: I0910 14:03:23.325332 2585 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-7a11e9-6864c5c8cd-x6gvq" Sep 10 14:03:23.325708 ip-10-0-140-8 kubenswrapper[2585]: I0910 14:03:23.325369 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-7a11e9-6864c5c8cd-x6gvq" event={"ID":"bf622551-e014-4e8b-8df2-6e3723f1df26","Type":"ContainerDied","Data":"b32253d06e8a79782f1959449482e730206db21d795cd1084a12edbb88ef73d4"} Sep 10 14:03:23.325708 ip-10-0-140-8 kubenswrapper[2585]: I0910 14:03:23.325391 2585 scope.go:117] "RemoveContainer" containerID="37c6018fefadafe3a14e52a267ff667fa9e63185d82a4e21dc44c0c6c7bf6a29" Sep 10 14:03:23.334545 ip-10-0-140-8 kubenswrapper[2585]: I0910 14:03:23.334530 2585 scope.go:117] "RemoveContainer" containerID="37c6018fefadafe3a14e52a267ff667fa9e63185d82a4e21dc44c0c6c7bf6a29" Sep 10 14:03:23.334789 ip-10-0-140-8 kubenswrapper[2585]: E0910 14:03:23.334770 2585 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"37c6018fefadafe3a14e52a267ff667fa9e63185d82a4e21dc44c0c6c7bf6a29\": container with ID starting with 37c6018fefadafe3a14e52a267ff667fa9e63185d82a4e21dc44c0c6c7bf6a29 not found: ID does not exist" containerID="37c6018fefadafe3a14e52a267ff667fa9e63185d82a4e21dc44c0c6c7bf6a29" Sep 10 14:03:23.334847 ip-10-0-140-8 kubenswrapper[2585]: I0910 14:03:23.334794 2585 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"37c6018fefadafe3a14e52a267ff667fa9e63185d82a4e21dc44c0c6c7bf6a29"} err="failed to get container status \"37c6018fefadafe3a14e52a267ff667fa9e63185d82a4e21dc44c0c6c7bf6a29\": rpc error: code = NotFound desc = could not find container \"37c6018fefadafe3a14e52a267ff667fa9e63185d82a4e21dc44c0c6c7bf6a29\": container with ID starting with 37c6018fefadafe3a14e52a267ff667fa9e63185d82a4e21dc44c0c6c7bf6a29 not found: ID does not exist" Sep 10 14:03:23.361616 ip-10-0-140-8 kubenswrapper[2585]: I0910 14:03:23.361595 2585 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w3-7a11e9-6864c5c8cd-x6gvq"] Sep 10 14:03:23.374495 ip-10-0-140-8 kubenswrapper[2585]: I0910 14:03:23.374476 2585 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w3-7a11e9-6864c5c8cd-x6gvq"] Sep 10 14:03:23.652727 ip-10-0-140-8 kubenswrapper[2585]: I0910 14:03:23.652647 2585 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="047277cb-ad53-4971-9e34-219c5748928f" path="/var/lib/kubelet/pods/047277cb-ad53-4971-9e34-219c5748928f/volumes" Sep 10 14:03:23.653008 ip-10-0-140-8 kubenswrapper[2585]: I0910 14:03:23.652995 2585 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="bf622551-e014-4e8b-8df2-6e3723f1df26" path="/var/lib/kubelet/pods/bf622551-e014-4e8b-8df2-6e3723f1df26/volumes" Sep 10 14:03:30.012707 ip-10-0-140-8 kubenswrapper[2585]: I0910 14:03:30.012670 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fxgmp_fa53c783-d40a-4024-9f7b-1d0d2b57f6dc/konnectivity-agent/0.log" limit=1024 Sep 10 14:03:30.079559 ip-10-0-140-8 kubenswrapper[2585]: I0910 14:03:30.079526 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fxgmp_fa53c783-d40a-4024-9f7b-1d0d2b57f6dc/konnectivity-agent/0.log" limit=1024 Sep 10 14:03:30.294729 ip-10-0-140-8 kubenswrapper[2585]: I0910 14:03:30.294642 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fxgmp_fa53c783-d40a-4024-9f7b-1d0d2b57f6dc/konnectivity-agent/0.log" limit=1024 Sep 10 14:03:30.690802 ip-10-0-140-8 kubenswrapper[2585]: I0910 14:03:30.690763 2585 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-a00003-vffn7"] Sep 10 14:03:30.691409 ip-10-0-140-8 kubenswrapper[2585]: I0910 14:03:30.691390 2585 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="bf622551-e014-4e8b-8df2-6e3723f1df26" containerName="payload-pre-processing" Sep 10 14:03:30.691472 ip-10-0-140-8 kubenswrapper[2585]: I0910 14:03:30.691413 2585 state_mem.go:107] "Deleted CPUSet assignment" podUID="bf622551-e014-4e8b-8df2-6e3723f1df26" containerName="payload-pre-processing" Sep 10 14:03:30.691472 ip-10-0-140-8 kubenswrapper[2585]: I0910 14:03:30.691455 2585 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="047277cb-ad53-4971-9e34-219c5748928f" containerName="revoke-keys" Sep 10 14:03:30.691472 ip-10-0-140-8 kubenswrapper[2585]: I0910 14:03:30.691463 2585 state_mem.go:107] "Deleted CPUSet assignment" podUID="047277cb-ad53-4971-9e34-219c5748928f" containerName="revoke-keys" Sep 10 14:03:30.691585 ip-10-0-140-8 kubenswrapper[2585]: I0910 14:03:30.691573 2585 memory_manager.go:368] "RemoveStaleState removing state" podUID="bf622551-e014-4e8b-8df2-6e3723f1df26" containerName="payload-pre-processing" Sep 10 14:03:30.691619 ip-10-0-140-8 kubenswrapper[2585]: I0910 14:03:30.691591 2585 memory_manager.go:368] "RemoveStaleState removing state" podUID="047277cb-ad53-4971-9e34-219c5748928f" containerName="revoke-keys" Sep 10 14:03:30.696158 ip-10-0-140-8 kubenswrapper[2585]: I0910 14:03:30.696139 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-a00003-vffn7" Sep 10 14:03:30.700154 ip-10-0-140-8 kubenswrapper[2585]: I0910 14:03:30.700130 2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-7gcbr\"" Sep 10 14:03:30.700878 ip-10-0-140-8 kubenswrapper[2585]: I0910 14:03:30.700861 2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Sep 10 14:03:30.709984 ip-10-0-140-8 kubenswrapper[2585]: I0910 14:03:30.709961 2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-a00003-vffn7"] Sep 10 14:03:30.807891 ip-10-0-140-8 kubenswrapper[2585]: I0910 14:03:30.807859 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/08f3ff75-e1ca-49ed-96fd-99159b27f798-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w3-a00003-vffn7\" (UID: \"08f3ff75-e1ca-49ed-96fd-99159b27f798\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-a00003-vffn7" Sep 10 14:03:30.909419 ip-10-0-140-8 kubenswrapper[2585]: I0910 14:03:30.909382 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/08f3ff75-e1ca-49ed-96fd-99159b27f798-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w3-a00003-vffn7\" (UID: \"08f3ff75-e1ca-49ed-96fd-99159b27f798\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-a00003-vffn7" Sep 10 14:03:30.910013 ip-10-0-140-8 kubenswrapper[2585]: I0910 14:03:30.909991 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/08f3ff75-e1ca-49ed-96fd-99159b27f798-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w3-a00003-vffn7\" (UID: \"08f3ff75-e1ca-49ed-96fd-99159b27f798\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-a00003-vffn7" Sep 10 14:03:30.931239 ip-10-0-140-8 kubenswrapper[2585]: I0910 14:03:30.931209 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fxgmp_fa53c783-d40a-4024-9f7b-1d0d2b57f6dc/konnectivity-agent/0.log" limit=1024 Sep 10 14:03:31.006196 ip-10-0-140-8 kubenswrapper[2585]: I0910 14:03:31.006114 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-a00003-vffn7" Sep 10 14:03:31.292087 ip-10-0-140-8 kubenswrapper[2585]: I0910 14:03:31.289580 2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-a00003-vffn7"] Sep 10 14:03:31.292934 ip-10-0-140-8 kubenswrapper[2585]: W0910 14:03:31.292902 2585 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod08f3ff75_e1ca_49ed_96fd_99159b27f798.slice/crio-bb6baa3b7b28facf1e1ab0b5e04ba207dc5a6b9fc9ecfae59b1f1a692d853add WatchSource:0}: Error finding container bb6baa3b7b28facf1e1ab0b5e04ba207dc5a6b9fc9ecfae59b1f1a692d853add: Status 404 returned error can't find the container with id bb6baa3b7b28facf1e1ab0b5e04ba207dc5a6b9fc9ecfae59b1f1a692d853add Sep 10 14:03:31.375114 ip-10-0-140-8 kubenswrapper[2585]: I0910 14:03:31.375081 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-a00003-vffn7" event={"ID":"08f3ff75-e1ca-49ed-96fd-99159b27f798","Type":"ContainerStarted","Data":"bb6baa3b7b28facf1e1ab0b5e04ba207dc5a6b9fc9ecfae59b1f1a692d853add"} Sep 10 14:03:32.382487 ip-10-0-140-8 kubenswrapper[2585]: I0910 14:03:32.382450 2585 generic.go:356] "Generic (PLEG): container finished" podID="08f3ff75-e1ca-49ed-96fd-99159b27f798" containerID="ed554c8fba92443a77c5b33664f3343eae1e9b00aa606fec30a45cf0e5b210e4" exitCode=0 Sep 10 14:03:32.382965 ip-10-0-140-8 kubenswrapper[2585]: I0910 14:03:32.382540 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-a00003-vffn7" event={"ID":"08f3ff75-e1ca-49ed-96fd-99159b27f798","Type":"ContainerDied","Data":"ed554c8fba92443a77c5b33664f3343eae1e9b00aa606fec30a45cf0e5b210e4"} Sep 10 14:03:33.522567 ip-10-0-140-8 kubenswrapper[2585]: I0910 14:03:33.522539 2585 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-a00003-vffn7" Sep 10 14:03:33.634644 ip-10-0-140-8 kubenswrapper[2585]: I0910 14:03:33.634610 2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/08f3ff75-e1ca-49ed-96fd-99159b27f798-maas-api-service-ca\") pod \"08f3ff75-e1ca-49ed-96fd-99159b27f798\" (UID: \"08f3ff75-e1ca-49ed-96fd-99159b27f798\") " Sep 10 14:03:33.634956 ip-10-0-140-8 kubenswrapper[2585]: I0910 14:03:33.634932 2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/08f3ff75-e1ca-49ed-96fd-99159b27f798-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "08f3ff75-e1ca-49ed-96fd-99159b27f798" (UID: "08f3ff75-e1ca-49ed-96fd-99159b27f798"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 10 14:03:33.736047 ip-10-0-140-8 kubenswrapper[2585]: I0910 14:03:33.736006 2585 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/08f3ff75-e1ca-49ed-96fd-99159b27f798-maas-api-service-ca\") on node \"ip-10-0-140-8.ec2.internal\" DevicePath \"\"" Sep 10 14:03:34.395076 ip-10-0-140-8 kubenswrapper[2585]: I0910 14:03:34.395046 2585 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-a00003-vffn7" Sep 10 14:03:34.395239 ip-10-0-140-8 kubenswrapper[2585]: I0910 14:03:34.395050 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-a00003-vffn7" event={"ID":"08f3ff75-e1ca-49ed-96fd-99159b27f798","Type":"ContainerDied","Data":"bb6baa3b7b28facf1e1ab0b5e04ba207dc5a6b9fc9ecfae59b1f1a692d853add"} Sep 10 14:03:34.395239 ip-10-0-140-8 kubenswrapper[2585]: I0910 14:03:34.395181 2585 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="bb6baa3b7b28facf1e1ab0b5e04ba207dc5a6b9fc9ecfae59b1f1a692d853add" Sep 10 14:03:35.802294 ip-10-0-140-8 kubenswrapper[2585]: I0910 14:03:35.802268 2585 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-l9l8q_ad555c6a-4385-4f8e-93bb-a174cfaa8e74/kube-multus/2.log" Sep 10 14:03:35.805411 ip-10-0-140-8 kubenswrapper[2585]: I0910 14:03:35.805387 2585 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-l9l8q_ad555c6a-4385-4f8e-93bb-a174cfaa8e74/kube-multus/2.log" Sep 10 14:03:41.834202 ip-10-0-140-8 kubenswrapper[2585]: I0910 14:03:41.834132 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fxgmp_fa53c783-d40a-4024-9f7b-1d0d2b57f6dc/konnectivity-agent/0.log" limit=1024 Sep 10 14:03:41.983779 ip-10-0-140-8 kubenswrapper[2585]: I0910 14:03:41.983746 2585 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w3-a00003-66c855d86-wvhm5"] Sep 10 14:03:42.083095 ip-10-0-140-8 kubenswrapper[2585]: I0910 14:03:42.083050 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fxgmp_fa53c783-d40a-4024-9f7b-1d0d2b57f6dc/konnectivity-agent/0.log" limit=1024 Sep 10 14:03:42.306670 ip-10-0-140-8 kubenswrapper[2585]: I0910 14:03:42.306583 2585 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-a00003-vffn7"] Sep 10 14:03:42.310344 ip-10-0-140-8 kubenswrapper[2585]: I0910 14:03:42.310319 2585 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-a00003-vffn7"] Sep 10 14:03:42.438932 ip-10-0-140-8 kubenswrapper[2585]: I0910 14:03:42.438894 2585 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-a00003-66c855d86-wvhm5" podUID="f6f81da6-a652-4cdc-90ff-24e5ba9c179c" containerName="payload-pre-processing" containerID="cri-o://8d4537d7c1f9c72ab42384dee6cd2f8e3824824cd33f07121bf07b953b180198" gracePeriod=30 Sep 10 14:03:42.689803 ip-10-0-140-8 kubenswrapper[2585]: I0910 14:03:42.689778 2585 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-a00003-66c855d86-wvhm5" Sep 10 14:03:42.819586 ip-10-0-140-8 kubenswrapper[2585]: I0910 14:03:42.819548 2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-n6h4f\" (UniqueName: \"kubernetes.io/projected/f6f81da6-a652-4cdc-90ff-24e5ba9c179c-kube-api-access-n6h4f\") pod \"f6f81da6-a652-4cdc-90ff-24e5ba9c179c\" (UID: \"f6f81da6-a652-4cdc-90ff-24e5ba9c179c\") " Sep 10 14:03:42.819780 ip-10-0-140-8 kubenswrapper[2585]: I0910 14:03:42.819632 2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f6f81da6-a652-4cdc-90ff-24e5ba9c179c-plugins-config-volume\") pod \"f6f81da6-a652-4cdc-90ff-24e5ba9c179c\" (UID: \"f6f81da6-a652-4cdc-90ff-24e5ba9c179c\") " Sep 10 14:03:42.819993 ip-10-0-140-8 kubenswrapper[2585]: I0910 14:03:42.819967 2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f6f81da6-a652-4cdc-90ff-24e5ba9c179c-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "f6f81da6-a652-4cdc-90ff-24e5ba9c179c" (UID: "f6f81da6-a652-4cdc-90ff-24e5ba9c179c"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 10 14:03:42.821691 ip-10-0-140-8 kubenswrapper[2585]: I0910 14:03:42.821664 2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f6f81da6-a652-4cdc-90ff-24e5ba9c179c-kube-api-access-n6h4f" (OuterVolumeSpecName: "kube-api-access-n6h4f") pod "f6f81da6-a652-4cdc-90ff-24e5ba9c179c" (UID: "f6f81da6-a652-4cdc-90ff-24e5ba9c179c"). InnerVolumeSpecName "kube-api-access-n6h4f". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 10 14:03:42.921258 ip-10-0-140-8 kubenswrapper[2585]: I0910 14:03:42.921158 2585 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-n6h4f\" (UniqueName: \"kubernetes.io/projected/f6f81da6-a652-4cdc-90ff-24e5ba9c179c-kube-api-access-n6h4f\") on node \"ip-10-0-140-8.ec2.internal\" DevicePath \"\"" Sep 10 14:03:42.921258 ip-10-0-140-8 kubenswrapper[2585]: I0910 14:03:42.921188 2585 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f6f81da6-a652-4cdc-90ff-24e5ba9c179c-plugins-config-volume\") on node \"ip-10-0-140-8.ec2.internal\" DevicePath \"\"" Sep 10 14:03:43.224603 ip-10-0-140-8 kubenswrapper[2585]: I0910 14:03:43.224480 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fxgmp_fa53c783-d40a-4024-9f7b-1d0d2b57f6dc/konnectivity-agent/0.log" limit=1024 Sep 10 14:03:43.294047 ip-10-0-140-8 kubenswrapper[2585]: I0910 14:03:43.293994 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fxgmp_fa53c783-d40a-4024-9f7b-1d0d2b57f6dc/konnectivity-agent/0.log" limit=1024 Sep 10 14:03:43.445388 ip-10-0-140-8 kubenswrapper[2585]: I0910 14:03:43.445350 2585 generic.go:356] "Generic (PLEG): container finished" podID="f6f81da6-a652-4cdc-90ff-24e5ba9c179c" containerID="8d4537d7c1f9c72ab42384dee6cd2f8e3824824cd33f07121bf07b953b180198" exitCode=0 Sep 10 14:03:43.445581 ip-10-0-140-8 kubenswrapper[2585]: I0910 14:03:43.445399 2585 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-a00003-66c855d86-wvhm5" Sep 10 14:03:43.445581 ip-10-0-140-8 kubenswrapper[2585]: I0910 14:03:43.445433 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-a00003-66c855d86-wvhm5" event={"ID":"f6f81da6-a652-4cdc-90ff-24e5ba9c179c","Type":"ContainerDied","Data":"8d4537d7c1f9c72ab42384dee6cd2f8e3824824cd33f07121bf07b953b180198"} Sep 10 14:03:43.445581 ip-10-0-140-8 kubenswrapper[2585]: I0910 14:03:43.445487 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-a00003-66c855d86-wvhm5" event={"ID":"f6f81da6-a652-4cdc-90ff-24e5ba9c179c","Type":"ContainerDied","Data":"bcb5353dda29370aad474073a4fc8f3be8a8d51184c348c32d2a184d1542f42b"} Sep 10 14:03:43.445581 ip-10-0-140-8 kubenswrapper[2585]: I0910 14:03:43.445512 2585 scope.go:117] "RemoveContainer" containerID="8d4537d7c1f9c72ab42384dee6cd2f8e3824824cd33f07121bf07b953b180198" Sep 10 14:03:43.455425 ip-10-0-140-8 kubenswrapper[2585]: I0910 14:03:43.455410 2585 scope.go:117] "RemoveContainer" containerID="8d4537d7c1f9c72ab42384dee6cd2f8e3824824cd33f07121bf07b953b180198" Sep 10 14:03:43.455665 ip-10-0-140-8 kubenswrapper[2585]: E0910 14:03:43.455645 2585 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8d4537d7c1f9c72ab42384dee6cd2f8e3824824cd33f07121bf07b953b180198\": container with ID starting with 8d4537d7c1f9c72ab42384dee6cd2f8e3824824cd33f07121bf07b953b180198 not found: ID does not exist" containerID="8d4537d7c1f9c72ab42384dee6cd2f8e3824824cd33f07121bf07b953b180198" Sep 10 14:03:43.455711 ip-10-0-140-8 kubenswrapper[2585]: I0910 14:03:43.455674 2585 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8d4537d7c1f9c72ab42384dee6cd2f8e3824824cd33f07121bf07b953b180198"} err="failed to get container status \"8d4537d7c1f9c72ab42384dee6cd2f8e3824824cd33f07121bf07b953b180198\": rpc error: code = NotFound desc = could not find container \"8d4537d7c1f9c72ab42384dee6cd2f8e3824824cd33f07121bf07b953b180198\": container with ID starting with 8d4537d7c1f9c72ab42384dee6cd2f8e3824824cd33f07121bf07b953b180198 not found: ID does not exist" Sep 10 14:03:43.472036 ip-10-0-140-8 kubenswrapper[2585]: I0910 14:03:43.472013 2585 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w3-a00003-66c855d86-wvhm5"] Sep 10 14:03:43.474479 ip-10-0-140-8 kubenswrapper[2585]: I0910 14:03:43.474444 2585 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w3-a00003-66c855d86-wvhm5"] Sep 10 14:03:43.647398 ip-10-0-140-8 kubenswrapper[2585]: I0910 14:03:43.647351 2585 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="08f3ff75-e1ca-49ed-96fd-99159b27f798" path="/var/lib/kubelet/pods/08f3ff75-e1ca-49ed-96fd-99159b27f798/volumes" Sep 10 14:03:43.647739 ip-10-0-140-8 kubenswrapper[2585]: I0910 14:03:43.647726 2585 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="f6f81da6-a652-4cdc-90ff-24e5ba9c179c" path="/var/lib/kubelet/pods/f6f81da6-a652-4cdc-90ff-24e5ba9c179c/volumes" Sep 10 14:03:50.109347 ip-10-0-140-8 kubenswrapper[2585]: I0910 14:03:50.109310 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fxgmp_fa53c783-d40a-4024-9f7b-1d0d2b57f6dc/konnectivity-agent/0.log" limit=1024 Sep 10 14:03:50.210109 ip-10-0-140-8 kubenswrapper[2585]: I0910 14:03:50.210047 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fxgmp_fa53c783-d40a-4024-9f7b-1d0d2b57f6dc/konnectivity-agent/0.log" limit=1024 Sep 10 14:03:50.394615 ip-10-0-140-8 kubenswrapper[2585]: I0910 14:03:50.394528 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fxgmp_fa53c783-d40a-4024-9f7b-1d0d2b57f6dc/konnectivity-agent/0.log" limit=1024 Sep 10 14:03:51.065424 ip-10-0-140-8 kubenswrapper[2585]: I0910 14:03:51.065385 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fxgmp_fa53c783-d40a-4024-9f7b-1d0d2b57f6dc/konnectivity-agent/0.log" limit=1024 Sep 10 14:03:53.128461 ip-10-0-140-8 kubenswrapper[2585]: I0910 14:03:53.128426 2585 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-69bcf98bbb-vbmx7"] Sep 10 14:03:53.128932 ip-10-0-140-8 kubenswrapper[2585]: I0910 14:03:53.128686 2585 kuberuntime_container.go:873] "Killing container with a grace period" pod="opendatahub/maas-controller-69bcf98bbb-vbmx7" podUID="111c86ef-d8e5-4460-b7c9-e7d3cf1a044f" containerName="manager" containerID="cri-o://90c0af958ba5134805b7b10bba452e96f388561ace902d86e5471790ec9d916d" gracePeriod=30 Sep 10 14:03:53.377616 ip-10-0-140-8 kubenswrapper[2585]: I0910 14:03:53.377591 2585 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-69bcf98bbb-vbmx7" Sep 10 14:03:53.506761 ip-10-0-140-8 kubenswrapper[2585]: I0910 14:03:53.506668 2585 generic.go:356] "Generic (PLEG): container finished" podID="111c86ef-d8e5-4460-b7c9-e7d3cf1a044f" containerID="90c0af958ba5134805b7b10bba452e96f388561ace902d86e5471790ec9d916d" exitCode=0 Sep 10 14:03:53.506761 ip-10-0-140-8 kubenswrapper[2585]: I0910 14:03:53.506719 2585 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-69bcf98bbb-vbmx7" Sep 10 14:03:53.506961 ip-10-0-140-8 kubenswrapper[2585]: I0910 14:03:53.506769 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-69bcf98bbb-vbmx7" event={"ID":"111c86ef-d8e5-4460-b7c9-e7d3cf1a044f","Type":"ContainerDied","Data":"90c0af958ba5134805b7b10bba452e96f388561ace902d86e5471790ec9d916d"} Sep 10 14:03:53.506961 ip-10-0-140-8 kubenswrapper[2585]: I0910 14:03:53.506835 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-69bcf98bbb-vbmx7" event={"ID":"111c86ef-d8e5-4460-b7c9-e7d3cf1a044f","Type":"ContainerDied","Data":"9eb38234827af8dc003fdc55954ee87a77af2bf80a1c58df2cf6953429e3a9cb"} Sep 10 14:03:53.506961 ip-10-0-140-8 kubenswrapper[2585]: I0910 14:03:53.506851 2585 scope.go:117] "RemoveContainer" containerID="90c0af958ba5134805b7b10bba452e96f388561ace902d86e5471790ec9d916d" Sep 10 14:03:53.522680 ip-10-0-140-8 kubenswrapper[2585]: I0910 14:03:53.522649 2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-t8ckn\" (UniqueName: \"kubernetes.io/projected/111c86ef-d8e5-4460-b7c9-e7d3cf1a044f-kube-api-access-t8ckn\") pod \"111c86ef-d8e5-4460-b7c9-e7d3cf1a044f\" (UID: \"111c86ef-d8e5-4460-b7c9-e7d3cf1a044f\") " Sep 10 14:03:53.522856 ip-10-0-140-8 kubenswrapper[2585]: I0910 14:03:53.522732 2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/111c86ef-d8e5-4460-b7c9-e7d3cf1a044f-webhook-cert\") pod \"111c86ef-d8e5-4460-b7c9-e7d3cf1a044f\" (UID: \"111c86ef-d8e5-4460-b7c9-e7d3cf1a044f\") " Sep 10 14:03:53.522856 ip-10-0-140-8 kubenswrapper[2585]: I0910 14:03:53.522800 2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/111c86ef-d8e5-4460-b7c9-e7d3cf1a044f-metrics-certs\") pod \"111c86ef-d8e5-4460-b7c9-e7d3cf1a044f\" (UID: \"111c86ef-d8e5-4460-b7c9-e7d3cf1a044f\") " Sep 10 14:03:53.524950 ip-10-0-140-8 kubenswrapper[2585]: I0910 14:03:53.524921 2585 scope.go:117] "RemoveContainer" containerID="90c0af958ba5134805b7b10bba452e96f388561ace902d86e5471790ec9d916d" Sep 10 14:03:53.525084 ip-10-0-140-8 kubenswrapper[2585]: I0910 14:03:53.524949 2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/111c86ef-d8e5-4460-b7c9-e7d3cf1a044f-webhook-cert" (OuterVolumeSpecName: "webhook-cert") pod "111c86ef-d8e5-4460-b7c9-e7d3cf1a044f" (UID: "111c86ef-d8e5-4460-b7c9-e7d3cf1a044f"). InnerVolumeSpecName "webhook-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 10 14:03:53.525377 ip-10-0-140-8 kubenswrapper[2585]: I0910 14:03:53.525351 2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/111c86ef-d8e5-4460-b7c9-e7d3cf1a044f-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "111c86ef-d8e5-4460-b7c9-e7d3cf1a044f" (UID: "111c86ef-d8e5-4460-b7c9-e7d3cf1a044f"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 10 14:03:53.525489 ip-10-0-140-8 kubenswrapper[2585]: E0910 14:03:53.525390 2585 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"90c0af958ba5134805b7b10bba452e96f388561ace902d86e5471790ec9d916d\": container with ID starting with 90c0af958ba5134805b7b10bba452e96f388561ace902d86e5471790ec9d916d not found: ID does not exist" containerID="90c0af958ba5134805b7b10bba452e96f388561ace902d86e5471790ec9d916d" Sep 10 14:03:53.525489 ip-10-0-140-8 kubenswrapper[2585]: I0910 14:03:53.525446 2585 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"90c0af958ba5134805b7b10bba452e96f388561ace902d86e5471790ec9d916d"} err="failed to get container status \"90c0af958ba5134805b7b10bba452e96f388561ace902d86e5471790ec9d916d\": rpc error: code = NotFound desc = could not find container \"90c0af958ba5134805b7b10bba452e96f388561ace902d86e5471790ec9d916d\": container with ID starting with 90c0af958ba5134805b7b10bba452e96f388561ace902d86e5471790ec9d916d not found: ID does not exist" Sep 10 14:03:53.525681 ip-10-0-140-8 kubenswrapper[2585]: I0910 14:03:53.525655 2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/111c86ef-d8e5-4460-b7c9-e7d3cf1a044f-kube-api-access-t8ckn" (OuterVolumeSpecName: "kube-api-access-t8ckn") pod "111c86ef-d8e5-4460-b7c9-e7d3cf1a044f" (UID: "111c86ef-d8e5-4460-b7c9-e7d3cf1a044f"). InnerVolumeSpecName "kube-api-access-t8ckn". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 10 14:03:53.623879 ip-10-0-140-8 kubenswrapper[2585]: I0910 14:03:53.623844 2585 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/111c86ef-d8e5-4460-b7c9-e7d3cf1a044f-metrics-certs\") on node \"ip-10-0-140-8.ec2.internal\" DevicePath \"\"" Sep 10 14:03:53.623879 ip-10-0-140-8 kubenswrapper[2585]: I0910 14:03:53.623875 2585 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-t8ckn\" (UniqueName: \"kubernetes.io/projected/111c86ef-d8e5-4460-b7c9-e7d3cf1a044f-kube-api-access-t8ckn\") on node \"ip-10-0-140-8.ec2.internal\" DevicePath \"\"" Sep 10 14:03:53.623879 ip-10-0-140-8 kubenswrapper[2585]: I0910 14:03:53.623887 2585 reconciler_common.go:299] "Volume detached for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/111c86ef-d8e5-4460-b7c9-e7d3cf1a044f-webhook-cert\") on node \"ip-10-0-140-8.ec2.internal\" DevicePath \"\"" Sep 10 14:03:53.827733 ip-10-0-140-8 kubenswrapper[2585]: I0910 14:03:53.827706 2585 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-69bcf98bbb-vbmx7"] Sep 10 14:03:53.832829 ip-10-0-140-8 kubenswrapper[2585]: I0910 14:03:53.832805 2585 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/maas-controller-69bcf98bbb-vbmx7"] Sep 10 14:03:55.648629 ip-10-0-140-8 kubenswrapper[2585]: I0910 14:03:55.648591 2585 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="111c86ef-d8e5-4460-b7c9-e7d3cf1a044f" path="/var/lib/kubelet/pods/111c86ef-d8e5-4460-b7c9-e7d3cf1a044f/volumes" Sep 10 14:03:55.688282 ip-10-0-140-8 kubenswrapper[2585]: I0910 14:03:55.688252 2585 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-79d8d45465-rwgx9"] Sep 10 14:03:55.688628 ip-10-0-140-8 kubenswrapper[2585]: I0910 14:03:55.688615 2585 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="111c86ef-d8e5-4460-b7c9-e7d3cf1a044f" containerName="manager" Sep 10 14:03:55.688673 ip-10-0-140-8 kubenswrapper[2585]: I0910 14:03:55.688631 2585 state_mem.go:107] "Deleted CPUSet assignment" podUID="111c86ef-d8e5-4460-b7c9-e7d3cf1a044f" containerName="manager" Sep 10 14:03:55.688673 ip-10-0-140-8 kubenswrapper[2585]: I0910 14:03:55.688645 2585 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f6f81da6-a652-4cdc-90ff-24e5ba9c179c" containerName="payload-pre-processing" Sep 10 14:03:55.688673 ip-10-0-140-8 kubenswrapper[2585]: I0910 14:03:55.688651 2585 state_mem.go:107] "Deleted CPUSet assignment" podUID="f6f81da6-a652-4cdc-90ff-24e5ba9c179c" containerName="payload-pre-processing" Sep 10 14:03:55.688673 ip-10-0-140-8 kubenswrapper[2585]: I0910 14:03:55.688661 2585 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="08f3ff75-e1ca-49ed-96fd-99159b27f798" containerName="revoke-keys" Sep 10 14:03:55.688673 ip-10-0-140-8 kubenswrapper[2585]: I0910 14:03:55.688667 2585 state_mem.go:107] "Deleted CPUSet assignment" podUID="08f3ff75-e1ca-49ed-96fd-99159b27f798" containerName="revoke-keys" Sep 10 14:03:55.688832 ip-10-0-140-8 kubenswrapper[2585]: I0910 14:03:55.688726 2585 memory_manager.go:368] "RemoveStaleState removing state" podUID="111c86ef-d8e5-4460-b7c9-e7d3cf1a044f" containerName="manager" Sep 10 14:03:55.688832 ip-10-0-140-8 kubenswrapper[2585]: I0910 14:03:55.688733 2585 memory_manager.go:368] "RemoveStaleState removing state" podUID="f6f81da6-a652-4cdc-90ff-24e5ba9c179c" containerName="payload-pre-processing" Sep 10 14:03:55.688832 ip-10-0-140-8 kubenswrapper[2585]: I0910 14:03:55.688740 2585 memory_manager.go:368] "RemoveStaleState removing state" podUID="08f3ff75-e1ca-49ed-96fd-99159b27f798" containerName="revoke-keys" Sep 10 14:03:55.691886 ip-10-0-140-8 kubenswrapper[2585]: I0910 14:03:55.691867 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-79d8d45465-rwgx9" Sep 10 14:03:55.711426 ip-10-0-140-8 kubenswrapper[2585]: I0910 14:03:55.711404 2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-79d8d45465-rwgx9"] Sep 10 14:03:55.770806 ip-10-0-140-8 kubenswrapper[2585]: I0910 14:03:55.770772 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fxgmp_fa53c783-d40a-4024-9f7b-1d0d2b57f6dc/konnectivity-agent/0.log" limit=1024 Sep 10 14:03:55.843916 ip-10-0-140-8 kubenswrapper[2585]: I0910 14:03:55.843888 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/8239a1fd-3466-432a-a253-b8575a6c2604-console-serving-cert\") pod \"console-79d8d45465-rwgx9\" (UID: \"8239a1fd-3466-432a-a253-b8575a6c2604\") " pod="openshift-console/console-79d8d45465-rwgx9" Sep 10 14:03:55.843916 ip-10-0-140-8 kubenswrapper[2585]: I0910 14:03:55.843918 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-khtk4\" (UniqueName: \"kubernetes.io/projected/8239a1fd-3466-432a-a253-b8575a6c2604-kube-api-access-khtk4\") pod \"console-79d8d45465-rwgx9\" (UID: \"8239a1fd-3466-432a-a253-b8575a6c2604\") " pod="openshift-console/console-79d8d45465-rwgx9" Sep 10 14:03:55.844120 ip-10-0-140-8 kubenswrapper[2585]: I0910 14:03:55.843987 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/8239a1fd-3466-432a-a253-b8575a6c2604-trusted-ca-bundle\") pod \"console-79d8d45465-rwgx9\" (UID: \"8239a1fd-3466-432a-a253-b8575a6c2604\") " pod="openshift-console/console-79d8d45465-rwgx9" Sep 10 14:03:55.844120 ip-10-0-140-8 kubenswrapper[2585]: I0910 14:03:55.844024 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/8239a1fd-3466-432a-a253-b8575a6c2604-service-ca\") pod \"console-79d8d45465-rwgx9\" (UID: \"8239a1fd-3466-432a-a253-b8575a6c2604\") " pod="openshift-console/console-79d8d45465-rwgx9" Sep 10 14:03:55.844120 ip-10-0-140-8 kubenswrapper[2585]: I0910 14:03:55.844068 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/8239a1fd-3466-432a-a253-b8575a6c2604-oauth-serving-cert\") pod \"console-79d8d45465-rwgx9\" (UID: \"8239a1fd-3466-432a-a253-b8575a6c2604\") " pod="openshift-console/console-79d8d45465-rwgx9" Sep 10 14:03:55.844120 ip-10-0-140-8 kubenswrapper[2585]: I0910 14:03:55.844094 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/8239a1fd-3466-432a-a253-b8575a6c2604-console-config\") pod \"console-79d8d45465-rwgx9\" (UID: \"8239a1fd-3466-432a-a253-b8575a6c2604\") " pod="openshift-console/console-79d8d45465-rwgx9" Sep 10 14:03:55.844270 ip-10-0-140-8 kubenswrapper[2585]: I0910 14:03:55.844157 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/8239a1fd-3466-432a-a253-b8575a6c2604-console-oauth-config\") pod \"console-79d8d45465-rwgx9\" (UID: \"8239a1fd-3466-432a-a253-b8575a6c2604\") " pod="openshift-console/console-79d8d45465-rwgx9" Sep 10 14:03:55.945659 ip-10-0-140-8 kubenswrapper[2585]: I0910 14:03:55.945563 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/8239a1fd-3466-432a-a253-b8575a6c2604-oauth-serving-cert\") pod \"console-79d8d45465-rwgx9\" (UID: \"8239a1fd-3466-432a-a253-b8575a6c2604\") " pod="openshift-console/console-79d8d45465-rwgx9" Sep 10 14:03:55.945659 ip-10-0-140-8 kubenswrapper[2585]: I0910 14:03:55.945606 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/8239a1fd-3466-432a-a253-b8575a6c2604-console-config\") pod \"console-79d8d45465-rwgx9\" (UID: \"8239a1fd-3466-432a-a253-b8575a6c2604\") " pod="openshift-console/console-79d8d45465-rwgx9" Sep 10 14:03:55.945659 ip-10-0-140-8 kubenswrapper[2585]: I0910 14:03:55.945658 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/8239a1fd-3466-432a-a253-b8575a6c2604-console-oauth-config\") pod \"console-79d8d45465-rwgx9\" (UID: \"8239a1fd-3466-432a-a253-b8575a6c2604\") " pod="openshift-console/console-79d8d45465-rwgx9" Sep 10 14:03:55.945944 ip-10-0-140-8 kubenswrapper[2585]: I0910 14:03:55.945715 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/8239a1fd-3466-432a-a253-b8575a6c2604-console-serving-cert\") pod \"console-79d8d45465-rwgx9\" (UID: \"8239a1fd-3466-432a-a253-b8575a6c2604\") " pod="openshift-console/console-79d8d45465-rwgx9" Sep 10 14:03:55.945944 ip-10-0-140-8 kubenswrapper[2585]: I0910 14:03:55.945738 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-khtk4\" (UniqueName: \"kubernetes.io/projected/8239a1fd-3466-432a-a253-b8575a6c2604-kube-api-access-khtk4\") pod \"console-79d8d45465-rwgx9\" (UID: \"8239a1fd-3466-432a-a253-b8575a6c2604\") " pod="openshift-console/console-79d8d45465-rwgx9" Sep 10 14:03:55.945944 ip-10-0-140-8 kubenswrapper[2585]: I0910 14:03:55.945779 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/8239a1fd-3466-432a-a253-b8575a6c2604-trusted-ca-bundle\") pod \"console-79d8d45465-rwgx9\" (UID: \"8239a1fd-3466-432a-a253-b8575a6c2604\") " pod="openshift-console/console-79d8d45465-rwgx9" Sep 10 14:03:55.945944 ip-10-0-140-8 kubenswrapper[2585]: I0910 14:03:55.945823 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/8239a1fd-3466-432a-a253-b8575a6c2604-service-ca\") pod \"console-79d8d45465-rwgx9\" (UID: \"8239a1fd-3466-432a-a253-b8575a6c2604\") " pod="openshift-console/console-79d8d45465-rwgx9" Sep 10 14:03:55.946467 ip-10-0-140-8 kubenswrapper[2585]: I0910 14:03:55.946437 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/8239a1fd-3466-432a-a253-b8575a6c2604-oauth-serving-cert\") pod \"console-79d8d45465-rwgx9\" (UID: \"8239a1fd-3466-432a-a253-b8575a6c2604\") " pod="openshift-console/console-79d8d45465-rwgx9" Sep 10 14:03:55.946582 ip-10-0-140-8 kubenswrapper[2585]: I0910 14:03:55.946437 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/8239a1fd-3466-432a-a253-b8575a6c2604-console-config\") pod \"console-79d8d45465-rwgx9\" (UID: \"8239a1fd-3466-432a-a253-b8575a6c2604\") " pod="openshift-console/console-79d8d45465-rwgx9" Sep 10 14:03:55.946582 ip-10-0-140-8 kubenswrapper[2585]: I0910 14:03:55.946493 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/8239a1fd-3466-432a-a253-b8575a6c2604-service-ca\") pod \"console-79d8d45465-rwgx9\" (UID: \"8239a1fd-3466-432a-a253-b8575a6c2604\") " pod="openshift-console/console-79d8d45465-rwgx9" Sep 10 14:03:55.946705 ip-10-0-140-8 kubenswrapper[2585]: I0910 14:03:55.946669 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/8239a1fd-3466-432a-a253-b8575a6c2604-trusted-ca-bundle\") pod \"console-79d8d45465-rwgx9\" (UID: \"8239a1fd-3466-432a-a253-b8575a6c2604\") " pod="openshift-console/console-79d8d45465-rwgx9" Sep 10 14:03:55.948662 ip-10-0-140-8 kubenswrapper[2585]: I0910 14:03:55.948642 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/8239a1fd-3466-432a-a253-b8575a6c2604-console-oauth-config\") pod \"console-79d8d45465-rwgx9\" (UID: \"8239a1fd-3466-432a-a253-b8575a6c2604\") " pod="openshift-console/console-79d8d45465-rwgx9" Sep 10 14:03:55.948782 ip-10-0-140-8 kubenswrapper[2585]: I0910 14:03:55.948737 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/8239a1fd-3466-432a-a253-b8575a6c2604-console-serving-cert\") pod \"console-79d8d45465-rwgx9\" (UID: \"8239a1fd-3466-432a-a253-b8575a6c2604\") " pod="openshift-console/console-79d8d45465-rwgx9" Sep 10 14:03:55.964518 ip-10-0-140-8 kubenswrapper[2585]: I0910 14:03:55.964487 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-khtk4\" (UniqueName: \"kubernetes.io/projected/8239a1fd-3466-432a-a253-b8575a6c2604-kube-api-access-khtk4\") pod \"console-79d8d45465-rwgx9\" (UID: \"8239a1fd-3466-432a-a253-b8575a6c2604\") " pod="openshift-console/console-79d8d45465-rwgx9" Sep 10 14:03:56.001583 ip-10-0-140-8 kubenswrapper[2585]: I0910 14:03:56.001556 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-79d8d45465-rwgx9" Sep 10 14:03:56.288407 ip-10-0-140-8 kubenswrapper[2585]: I0910 14:03:56.288195 2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-79d8d45465-rwgx9"] Sep 10 14:03:56.291116 ip-10-0-140-8 kubenswrapper[2585]: W0910 14:03:56.291087 2585 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8239a1fd_3466_432a_a253_b8575a6c2604.slice/crio-56ce641f8cd6a03f352049a028efc224a4d1dde2fc026efd4f6740efb6365359 WatchSource:0}: Error finding container 56ce641f8cd6a03f352049a028efc224a4d1dde2fc026efd4f6740efb6365359: Status 404 returned error can't find the container with id 56ce641f8cd6a03f352049a028efc224a4d1dde2fc026efd4f6740efb6365359 Sep 10 14:03:56.528662 ip-10-0-140-8 kubenswrapper[2585]: I0910 14:03:56.528628 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-79d8d45465-rwgx9" event={"ID":"8239a1fd-3466-432a-a253-b8575a6c2604","Type":"ContainerStarted","Data":"a5eef957cea8b7ac1f69cae70e97e6305208931776d18cb36b46aa5b7f472fd0"} Sep 10 14:03:56.528836 ip-10-0-140-8 kubenswrapper[2585]: I0910 14:03:56.528678 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-79d8d45465-rwgx9" event={"ID":"8239a1fd-3466-432a-a253-b8575a6c2604","Type":"ContainerStarted","Data":"56ce641f8cd6a03f352049a028efc224a4d1dde2fc026efd4f6740efb6365359"} Sep 10 14:03:56.553384 ip-10-0-140-8 kubenswrapper[2585]: I0910 14:03:56.553286 2585 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-79d8d45465-rwgx9" podStartSLOduration=1.553270818 podStartE2EDuration="1.553270818s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 14:03:55 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 14:03:56.551600662 +0000 UTC m=+2121.424304866" watchObservedRunningTime="2026-09-10 14:03:56.553270818 +0000 UTC m=+2121.425975019" Sep 10 14:04:06.001922 ip-10-0-140-8 kubenswrapper[2585]: I0910 14:04:06.001881 2585 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-79d8d45465-rwgx9" Sep 10 14:04:06.002360 ip-10-0-140-8 kubenswrapper[2585]: I0910 14:04:06.001944 2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-79d8d45465-rwgx9" Sep 10 14:04:06.006818 ip-10-0-140-8 kubenswrapper[2585]: I0910 14:04:06.006796 2585 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-79d8d45465-rwgx9" Sep 10 14:04:06.589171 ip-10-0-140-8 kubenswrapper[2585]: I0910 14:04:06.589140 2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-79d8d45465-rwgx9" Sep 10 14:04:06.645704 ip-10-0-140-8 kubenswrapper[2585]: I0910 14:04:06.645669 2585 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-6b9cc86549-4t9vq"] Sep 10 14:04:06.759063 ip-10-0-140-8 kubenswrapper[2585]: I0910 14:04:06.759035 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fxgmp_fa53c783-d40a-4024-9f7b-1d0d2b57f6dc/konnectivity-agent/0.log" limit=1024 Sep 10 14:04:06.863529 ip-10-0-140-8 kubenswrapper[2585]: I0910 14:04:06.863449 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fxgmp_fa53c783-d40a-4024-9f7b-1d0d2b57f6dc/konnectivity-agent/0.log" limit=1024 Sep 10 14:04:06.932081 ip-10-0-140-8 kubenswrapper[2585]: I0910 14:04:06.932043 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fxgmp_fa53c783-d40a-4024-9f7b-1d0d2b57f6dc/konnectivity-agent/0.log" limit=1024 Sep 10 14:04:27.909605 ip-10-0-140-8 kubenswrapper[2585]: I0910 14:04:27.909509 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fxgmp_fa53c783-d40a-4024-9f7b-1d0d2b57f6dc/konnectivity-agent/0.log" limit=1024 Sep 10 14:04:27.976056 ip-10-0-140-8 kubenswrapper[2585]: I0910 14:04:27.976018 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fxgmp_fa53c783-d40a-4024-9f7b-1d0d2b57f6dc/konnectivity-agent/0.log" limit=1024 Sep 10 14:04:31.672350 ip-10-0-140-8 kubenswrapper[2585]: I0910 14:04:31.672289 2585 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-console/console-6b9cc86549-4t9vq" podUID="75f58905-a4db-4513-a597-3148034fcfd4" containerName="console" containerID="cri-o://4e045292d59bf40c25a5f2d414d63b4ab7e5ed70c08f9df5f882fb1f0e2568d1" gracePeriod=15 Sep 10 14:04:31.916751 ip-10-0-140-8 kubenswrapper[2585]: I0910 14:04:31.916704 2585 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-6b9cc86549-4t9vq" Sep 10 14:04:32.061551 ip-10-0-140-8 kubenswrapper[2585]: I0910 14:04:32.061524 2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/75f58905-a4db-4513-a597-3148034fcfd4-oauth-serving-cert\") pod \"75f58905-a4db-4513-a597-3148034fcfd4\" (UID: \"75f58905-a4db-4513-a597-3148034fcfd4\") " Sep 10 14:04:32.061704 ip-10-0-140-8 kubenswrapper[2585]: I0910 14:04:32.061565 2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/75f58905-a4db-4513-a597-3148034fcfd4-console-oauth-config\") pod \"75f58905-a4db-4513-a597-3148034fcfd4\" (UID: \"75f58905-a4db-4513-a597-3148034fcfd4\") " Sep 10 14:04:32.061704 ip-10-0-140-8 kubenswrapper[2585]: I0910 14:04:32.061591 2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/75f58905-a4db-4513-a597-3148034fcfd4-service-ca\") pod \"75f58905-a4db-4513-a597-3148034fcfd4\" (UID: \"75f58905-a4db-4513-a597-3148034fcfd4\") " Sep 10 14:04:32.061704 ip-10-0-140-8 kubenswrapper[2585]: I0910 14:04:32.061627 2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/75f58905-a4db-4513-a597-3148034fcfd4-trusted-ca-bundle\") pod \"75f58905-a4db-4513-a597-3148034fcfd4\" (UID: \"75f58905-a4db-4513-a597-3148034fcfd4\") " Sep 10 14:04:32.061704 ip-10-0-140-8 kubenswrapper[2585]: I0910 14:04:32.061659 2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/75f58905-a4db-4513-a597-3148034fcfd4-console-config\") pod \"75f58905-a4db-4513-a597-3148034fcfd4\" (UID: \"75f58905-a4db-4513-a597-3148034fcfd4\") " Sep 10 14:04:32.061879 ip-10-0-140-8 kubenswrapper[2585]: I0910 14:04:32.061709 2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/75f58905-a4db-4513-a597-3148034fcfd4-console-serving-cert\") pod \"75f58905-a4db-4513-a597-3148034fcfd4\" (UID: \"75f58905-a4db-4513-a597-3148034fcfd4\") " Sep 10 14:04:32.061879 ip-10-0-140-8 kubenswrapper[2585]: I0910 14:04:32.061774 2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-nm4hw\" (UniqueName: \"kubernetes.io/projected/75f58905-a4db-4513-a597-3148034fcfd4-kube-api-access-nm4hw\") pod \"75f58905-a4db-4513-a597-3148034fcfd4\" (UID: \"75f58905-a4db-4513-a597-3148034fcfd4\") " Sep 10 14:04:32.062116 ip-10-0-140-8 kubenswrapper[2585]: I0910 14:04:32.062085 2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/75f58905-a4db-4513-a597-3148034fcfd4-trusted-ca-bundle" (OuterVolumeSpecName: "trusted-ca-bundle") pod "75f58905-a4db-4513-a597-3148034fcfd4" (UID: "75f58905-a4db-4513-a597-3148034fcfd4"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 10 14:04:32.062193 ip-10-0-140-8 kubenswrapper[2585]: I0910 14:04:32.062011 2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/75f58905-a4db-4513-a597-3148034fcfd4-service-ca" (OuterVolumeSpecName: "service-ca") pod "75f58905-a4db-4513-a597-3148034fcfd4" (UID: "75f58905-a4db-4513-a597-3148034fcfd4"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 10 14:04:32.062193 ip-10-0-140-8 kubenswrapper[2585]: I0910 14:04:32.062093 2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/75f58905-a4db-4513-a597-3148034fcfd4-console-config" (OuterVolumeSpecName: "console-config") pod "75f58905-a4db-4513-a597-3148034fcfd4" (UID: "75f58905-a4db-4513-a597-3148034fcfd4"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 10 14:04:32.062193 ip-10-0-140-8 kubenswrapper[2585]: I0910 14:04:32.062137 2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/75f58905-a4db-4513-a597-3148034fcfd4-oauth-serving-cert" (OuterVolumeSpecName: "oauth-serving-cert") pod "75f58905-a4db-4513-a597-3148034fcfd4" (UID: "75f58905-a4db-4513-a597-3148034fcfd4"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 10 14:04:32.063907 ip-10-0-140-8 kubenswrapper[2585]: I0910 14:04:32.063883 2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/75f58905-a4db-4513-a597-3148034fcfd4-console-oauth-config" (OuterVolumeSpecName: "console-oauth-config") pod "75f58905-a4db-4513-a597-3148034fcfd4" (UID: "75f58905-a4db-4513-a597-3148034fcfd4"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 10 14:04:32.064026 ip-10-0-140-8 kubenswrapper[2585]: I0910 14:04:32.063937 2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/75f58905-a4db-4513-a597-3148034fcfd4-kube-api-access-nm4hw" (OuterVolumeSpecName: "kube-api-access-nm4hw") pod "75f58905-a4db-4513-a597-3148034fcfd4" (UID: "75f58905-a4db-4513-a597-3148034fcfd4"). InnerVolumeSpecName "kube-api-access-nm4hw". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 10 14:04:32.064026 ip-10-0-140-8 kubenswrapper[2585]: I0910 14:04:32.063982 2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/75f58905-a4db-4513-a597-3148034fcfd4-console-serving-cert" (OuterVolumeSpecName: "console-serving-cert") pod "75f58905-a4db-4513-a597-3148034fcfd4" (UID: "75f58905-a4db-4513-a597-3148034fcfd4"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 10 14:04:32.163370 ip-10-0-140-8 kubenswrapper[2585]: I0910 14:04:32.163336 2585 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/75f58905-a4db-4513-a597-3148034fcfd4-console-serving-cert\") on node \"ip-10-0-140-8.ec2.internal\" DevicePath \"\"" Sep 10 14:04:32.163370 ip-10-0-140-8 kubenswrapper[2585]: I0910 14:04:32.163365 2585 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nm4hw\" (UniqueName: \"kubernetes.io/projected/75f58905-a4db-4513-a597-3148034fcfd4-kube-api-access-nm4hw\") on node \"ip-10-0-140-8.ec2.internal\" DevicePath \"\"" Sep 10 14:04:32.163370 ip-10-0-140-8 kubenswrapper[2585]: I0910 14:04:32.163375 2585 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/75f58905-a4db-4513-a597-3148034fcfd4-oauth-serving-cert\") on node \"ip-10-0-140-8.ec2.internal\" DevicePath \"\"" Sep 10 14:04:32.163592 ip-10-0-140-8 kubenswrapper[2585]: I0910 14:04:32.163383 2585 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/75f58905-a4db-4513-a597-3148034fcfd4-console-oauth-config\") on node \"ip-10-0-140-8.ec2.internal\" DevicePath \"\"" Sep 10 14:04:32.163592 ip-10-0-140-8 kubenswrapper[2585]: I0910 14:04:32.163392 2585 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/75f58905-a4db-4513-a597-3148034fcfd4-service-ca\") on node \"ip-10-0-140-8.ec2.internal\" DevicePath \"\"" Sep 10 14:04:32.163592 ip-10-0-140-8 kubenswrapper[2585]: I0910 14:04:32.163401 2585 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/75f58905-a4db-4513-a597-3148034fcfd4-trusted-ca-bundle\") on node \"ip-10-0-140-8.ec2.internal\" DevicePath \"\"" Sep 10 14:04:32.163592 ip-10-0-140-8 kubenswrapper[2585]: I0910 14:04:32.163408 2585 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/75f58905-a4db-4513-a597-3148034fcfd4-console-config\") on node \"ip-10-0-140-8.ec2.internal\" DevicePath \"\"" Sep 10 14:04:32.744032 ip-10-0-140-8 kubenswrapper[2585]: I0910 14:04:32.743990 2585 generic.go:356] "Generic (PLEG): container finished" podID="75f58905-a4db-4513-a597-3148034fcfd4" containerID="4e045292d59bf40c25a5f2d414d63b4ab7e5ed70c08f9df5f882fb1f0e2568d1" exitCode=0 Sep 10 14:04:32.744032 ip-10-0-140-8 kubenswrapper[2585]: I0910 14:04:32.744034 2585 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-6b9cc86549-4t9vq" Sep 10 14:04:32.744495 ip-10-0-140-8 kubenswrapper[2585]: I0910 14:04:32.744078 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-6b9cc86549-4t9vq" event={"ID":"75f58905-a4db-4513-a597-3148034fcfd4","Type":"ContainerDied","Data":"4e045292d59bf40c25a5f2d414d63b4ab7e5ed70c08f9df5f882fb1f0e2568d1"} Sep 10 14:04:32.744495 ip-10-0-140-8 kubenswrapper[2585]: I0910 14:04:32.744128 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-6b9cc86549-4t9vq" event={"ID":"75f58905-a4db-4513-a597-3148034fcfd4","Type":"ContainerDied","Data":"8d639dea40c4ce5a50324090737db4c62897b356e8552dec396eeddd0ab6b2ca"} Sep 10 14:04:32.744495 ip-10-0-140-8 kubenswrapper[2585]: I0910 14:04:32.744152 2585 scope.go:117] "RemoveContainer" containerID="4e045292d59bf40c25a5f2d414d63b4ab7e5ed70c08f9df5f882fb1f0e2568d1" Sep 10 14:04:32.754108 ip-10-0-140-8 kubenswrapper[2585]: I0910 14:04:32.754088 2585 scope.go:117] "RemoveContainer" containerID="4e045292d59bf40c25a5f2d414d63b4ab7e5ed70c08f9df5f882fb1f0e2568d1" Sep 10 14:04:32.754435 ip-10-0-140-8 kubenswrapper[2585]: E0910 14:04:32.754416 2585 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4e045292d59bf40c25a5f2d414d63b4ab7e5ed70c08f9df5f882fb1f0e2568d1\": container with ID starting with 4e045292d59bf40c25a5f2d414d63b4ab7e5ed70c08f9df5f882fb1f0e2568d1 not found: ID does not exist" containerID="4e045292d59bf40c25a5f2d414d63b4ab7e5ed70c08f9df5f882fb1f0e2568d1" Sep 10 14:04:32.754510 ip-10-0-140-8 kubenswrapper[2585]: I0910 14:04:32.754443 2585 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4e045292d59bf40c25a5f2d414d63b4ab7e5ed70c08f9df5f882fb1f0e2568d1"} err="failed to get container status \"4e045292d59bf40c25a5f2d414d63b4ab7e5ed70c08f9df5f882fb1f0e2568d1\": rpc error: code = NotFound desc = could not find container \"4e045292d59bf40c25a5f2d414d63b4ab7e5ed70c08f9df5f882fb1f0e2568d1\": container with ID starting with 4e045292d59bf40c25a5f2d414d63b4ab7e5ed70c08f9df5f882fb1f0e2568d1 not found: ID does not exist" Sep 10 14:04:32.772452 ip-10-0-140-8 kubenswrapper[2585]: I0910 14:04:32.772422 2585 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-6b9cc86549-4t9vq"] Sep 10 14:04:32.779607 ip-10-0-140-8 kubenswrapper[2585]: I0910 14:04:32.779581 2585 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-6b9cc86549-4t9vq"] Sep 10 14:04:33.646853 ip-10-0-140-8 kubenswrapper[2585]: I0910 14:04:33.646819 2585 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="75f58905-a4db-4513-a597-3148034fcfd4" path="/var/lib/kubelet/pods/75f58905-a4db-4513-a597-3148034fcfd4/volumes" Sep 10 14:04:36.029015 ip-10-0-140-8 kubenswrapper[2585]: I0910 14:04:36.028978 2585 scope.go:117] "RemoveContainer" containerID="8f8db4f2a01e38c271fab73383b262367d5613b679f1cc7a98b045b56446fe8b" Sep 10 14:04:54.666846 ip-10-0-140-8 kubenswrapper[2585]: I0910 14:04:54.666805 2585 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/maas-controller-69bcf98bbb-pfc4t"] Sep 10 14:04:54.667381 ip-10-0-140-8 kubenswrapper[2585]: I0910 14:04:54.667361 2585 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="75f58905-a4db-4513-a597-3148034fcfd4" containerName="console" Sep 10 14:04:54.667475 ip-10-0-140-8 kubenswrapper[2585]: I0910 14:04:54.667383 2585 state_mem.go:107] "Deleted CPUSet assignment" podUID="75f58905-a4db-4513-a597-3148034fcfd4" containerName="console" Sep 10 14:04:54.667532 ip-10-0-140-8 kubenswrapper[2585]: I0910 14:04:54.667504 2585 memory_manager.go:368] "RemoveStaleState removing state" podUID="75f58905-a4db-4513-a597-3148034fcfd4" containerName="console" Sep 10 14:04:54.670648 ip-10-0-140-8 kubenswrapper[2585]: I0910 14:04:54.670626 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-69bcf98bbb-pfc4t" Sep 10 14:04:54.674878 ip-10-0-140-8 kubenswrapper[2585]: I0910 14:04:54.674856 2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-dockercfg-nn6pf\"" Sep 10 14:04:54.675001 ip-10-0-140-8 kubenswrapper[2585]: I0910 14:04:54.674879 2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-metrics-tls\"" Sep 10 14:04:54.675001 ip-10-0-140-8 kubenswrapper[2585]: I0910 14:04:54.674884 2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-webhook-cert\"" Sep 10 14:04:54.675001 ip-10-0-140-8 kubenswrapper[2585]: I0910 14:04:54.674974 2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"maas-parameters\"" Sep 10 14:04:54.687681 ip-10-0-140-8 kubenswrapper[2585]: I0910 14:04:54.687657 2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-69bcf98bbb-pfc4t"] Sep 10 14:04:54.768824 ip-10-0-140-8 kubenswrapper[2585]: I0910 14:04:54.768795 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/409c04f7-d273-4e5f-a2a4-f6b0707fd21b-metrics-certs\") pod \"maas-controller-69bcf98bbb-pfc4t\" (UID: \"409c04f7-d273-4e5f-a2a4-f6b0707fd21b\") " pod="opendatahub/maas-controller-69bcf98bbb-pfc4t" Sep 10 14:04:54.768979 ip-10-0-140-8 kubenswrapper[2585]: I0910 14:04:54.768852 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/409c04f7-d273-4e5f-a2a4-f6b0707fd21b-webhook-cert\") pod \"maas-controller-69bcf98bbb-pfc4t\" (UID: \"409c04f7-d273-4e5f-a2a4-f6b0707fd21b\") " pod="opendatahub/maas-controller-69bcf98bbb-pfc4t" Sep 10 14:04:54.768979 ip-10-0-140-8 kubenswrapper[2585]: I0910 14:04:54.768888 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8vbgh\" (UniqueName: \"kubernetes.io/projected/409c04f7-d273-4e5f-a2a4-f6b0707fd21b-kube-api-access-8vbgh\") pod \"maas-controller-69bcf98bbb-pfc4t\" (UID: \"409c04f7-d273-4e5f-a2a4-f6b0707fd21b\") " pod="opendatahub/maas-controller-69bcf98bbb-pfc4t" Sep 10 14:04:54.870133 ip-10-0-140-8 kubenswrapper[2585]: I0910 14:04:54.870100 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/409c04f7-d273-4e5f-a2a4-f6b0707fd21b-metrics-certs\") pod \"maas-controller-69bcf98bbb-pfc4t\" (UID: \"409c04f7-d273-4e5f-a2a4-f6b0707fd21b\") " pod="opendatahub/maas-controller-69bcf98bbb-pfc4t" Sep 10 14:04:54.870340 ip-10-0-140-8 kubenswrapper[2585]: I0910 14:04:54.870160 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/409c04f7-d273-4e5f-a2a4-f6b0707fd21b-webhook-cert\") pod \"maas-controller-69bcf98bbb-pfc4t\" (UID: \"409c04f7-d273-4e5f-a2a4-f6b0707fd21b\") " pod="opendatahub/maas-controller-69bcf98bbb-pfc4t" Sep 10 14:04:54.870340 ip-10-0-140-8 kubenswrapper[2585]: I0910 14:04:54.870187 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8vbgh\" (UniqueName: \"kubernetes.io/projected/409c04f7-d273-4e5f-a2a4-f6b0707fd21b-kube-api-access-8vbgh\") pod \"maas-controller-69bcf98bbb-pfc4t\" (UID: \"409c04f7-d273-4e5f-a2a4-f6b0707fd21b\") " pod="opendatahub/maas-controller-69bcf98bbb-pfc4t" Sep 10 14:04:54.872692 ip-10-0-140-8 kubenswrapper[2585]: I0910 14:04:54.872669 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/409c04f7-d273-4e5f-a2a4-f6b0707fd21b-webhook-cert\") pod \"maas-controller-69bcf98bbb-pfc4t\" (UID: \"409c04f7-d273-4e5f-a2a4-f6b0707fd21b\") " pod="opendatahub/maas-controller-69bcf98bbb-pfc4t" Sep 10 14:04:54.872793 ip-10-0-140-8 kubenswrapper[2585]: I0910 14:04:54.872720 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/409c04f7-d273-4e5f-a2a4-f6b0707fd21b-metrics-certs\") pod \"maas-controller-69bcf98bbb-pfc4t\" (UID: \"409c04f7-d273-4e5f-a2a4-f6b0707fd21b\") " pod="opendatahub/maas-controller-69bcf98bbb-pfc4t" Sep 10 14:04:54.883003 ip-10-0-140-8 kubenswrapper[2585]: I0910 14:04:54.882979 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8vbgh\" (UniqueName: \"kubernetes.io/projected/409c04f7-d273-4e5f-a2a4-f6b0707fd21b-kube-api-access-8vbgh\") pod \"maas-controller-69bcf98bbb-pfc4t\" (UID: \"409c04f7-d273-4e5f-a2a4-f6b0707fd21b\") " pod="opendatahub/maas-controller-69bcf98bbb-pfc4t" Sep 10 14:04:54.980994 ip-10-0-140-8 kubenswrapper[2585]: I0910 14:04:54.980919 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-69bcf98bbb-pfc4t" Sep 10 14:04:55.261295 ip-10-0-140-8 kubenswrapper[2585]: I0910 14:04:55.260756 2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-69bcf98bbb-pfc4t"] Sep 10 14:04:55.264611 ip-10-0-140-8 kubenswrapper[2585]: I0910 14:04:55.264592 2585 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Sep 10 14:04:55.873310 ip-10-0-140-8 kubenswrapper[2585]: I0910 14:04:55.873273 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-69bcf98bbb-pfc4t" event={"ID":"409c04f7-d273-4e5f-a2a4-f6b0707fd21b","Type":"ContainerStarted","Data":"88973dcbd1dbd62f81717964a27a19461f059ab04ba7ae3a746ac4d43839a413"} Sep 10 14:04:56.880724 ip-10-0-140-8 kubenswrapper[2585]: I0910 14:04:56.880688 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-69bcf98bbb-pfc4t" event={"ID":"409c04f7-d273-4e5f-a2a4-f6b0707fd21b","Type":"ContainerStarted","Data":"960d9e3dd1f85341a4de0d280e4624ba2a3ea46c817f8d965d9b8cb172c9f6a3"} Sep 10 14:04:56.881154 ip-10-0-140-8 kubenswrapper[2585]: I0910 14:04:56.880780 2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-controller-69bcf98bbb-pfc4t" Sep 10 14:04:56.906694 ip-10-0-140-8 kubenswrapper[2585]: I0910 14:04:56.906646 2585 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/maas-controller-69bcf98bbb-pfc4t" podStartSLOduration=2.279031592 podStartE2EDuration="2.906629936s" totalImagesPullingTime="627.598344ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 14:04:54 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 14:04:56.904568187 +0000 UTC m=+2181.777272388" watchObservedRunningTime="2026-09-10 14:04:56.906629936 +0000 UTC m=+2181.779334171" Sep 10 14:05:07.888953 ip-10-0-140-8 kubenswrapper[2585]: I0910 14:05:07.888915 2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/maas-controller-69bcf98bbb-pfc4t" Sep 10 14:05:15.258071 ip-10-0-140-8 kubenswrapper[2585]: I0910 14:05:15.258033 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fxgmp_fa53c783-d40a-4024-9f7b-1d0d2b57f6dc/konnectivity-agent/0.log" limit=1024 Sep 10 14:05:15.650665 ip-10-0-140-8 kubenswrapper[2585]: I0910 14:05:15.650637 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fxgmp_fa53c783-d40a-4024-9f7b-1d0d2b57f6dc/konnectivity-agent/0.log" limit=1024 Sep 10 14:05:17.193635 ip-10-0-140-8 kubenswrapper[2585]: I0910 14:05:17.193604 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fxgmp_fa53c783-d40a-4024-9f7b-1d0d2b57f6dc/konnectivity-agent/0.log" limit=1024 Sep 10 14:05:17.309078 ip-10-0-140-8 kubenswrapper[2585]: I0910 14:05:17.309045 2585 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-69bcf98bbb-pfc4t"] Sep 10 14:05:17.309294 ip-10-0-140-8 kubenswrapper[2585]: I0910 14:05:17.309270 2585 kuberuntime_container.go:873] "Killing container with a grace period" pod="opendatahub/maas-controller-69bcf98bbb-pfc4t" podUID="409c04f7-d273-4e5f-a2a4-f6b0707fd21b" containerName="manager" containerID="cri-o://960d9e3dd1f85341a4de0d280e4624ba2a3ea46c817f8d965d9b8cb172c9f6a3" gracePeriod=30 Sep 10 14:05:17.551612 ip-10-0-140-8 kubenswrapper[2585]: I0910 14:05:17.551589 2585 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-69bcf98bbb-pfc4t" Sep 10 14:05:17.684367 ip-10-0-140-8 kubenswrapper[2585]: I0910 14:05:17.684338 2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/409c04f7-d273-4e5f-a2a4-f6b0707fd21b-metrics-certs\") pod \"409c04f7-d273-4e5f-a2a4-f6b0707fd21b\" (UID: \"409c04f7-d273-4e5f-a2a4-f6b0707fd21b\") " Sep 10 14:05:17.684565 ip-10-0-140-8 kubenswrapper[2585]: I0910 14:05:17.684437 2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-8vbgh\" (UniqueName: \"kubernetes.io/projected/409c04f7-d273-4e5f-a2a4-f6b0707fd21b-kube-api-access-8vbgh\") pod \"409c04f7-d273-4e5f-a2a4-f6b0707fd21b\" (UID: \"409c04f7-d273-4e5f-a2a4-f6b0707fd21b\") " Sep 10 14:05:17.684642 ip-10-0-140-8 kubenswrapper[2585]: I0910 14:05:17.684567 2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/409c04f7-d273-4e5f-a2a4-f6b0707fd21b-webhook-cert\") pod \"409c04f7-d273-4e5f-a2a4-f6b0707fd21b\" (UID: \"409c04f7-d273-4e5f-a2a4-f6b0707fd21b\") " Sep 10 14:05:17.686339 ip-10-0-140-8 kubenswrapper[2585]: I0910 14:05:17.686309 2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/409c04f7-d273-4e5f-a2a4-f6b0707fd21b-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "409c04f7-d273-4e5f-a2a4-f6b0707fd21b" (UID: "409c04f7-d273-4e5f-a2a4-f6b0707fd21b"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 10 14:05:17.686503 ip-10-0-140-8 kubenswrapper[2585]: I0910 14:05:17.686447 2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/409c04f7-d273-4e5f-a2a4-f6b0707fd21b-kube-api-access-8vbgh" (OuterVolumeSpecName: "kube-api-access-8vbgh") pod "409c04f7-d273-4e5f-a2a4-f6b0707fd21b" (UID: "409c04f7-d273-4e5f-a2a4-f6b0707fd21b"). InnerVolumeSpecName "kube-api-access-8vbgh". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 10 14:05:17.686590 ip-10-0-140-8 kubenswrapper[2585]: I0910 14:05:17.686515 2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/409c04f7-d273-4e5f-a2a4-f6b0707fd21b-webhook-cert" (OuterVolumeSpecName: "webhook-cert") pod "409c04f7-d273-4e5f-a2a4-f6b0707fd21b" (UID: "409c04f7-d273-4e5f-a2a4-f6b0707fd21b"). InnerVolumeSpecName "webhook-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 10 14:05:17.785610 ip-10-0-140-8 kubenswrapper[2585]: I0910 14:05:17.785525 2585 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8vbgh\" (UniqueName: \"kubernetes.io/projected/409c04f7-d273-4e5f-a2a4-f6b0707fd21b-kube-api-access-8vbgh\") on node \"ip-10-0-140-8.ec2.internal\" DevicePath \"\"" Sep 10 14:05:17.785610 ip-10-0-140-8 kubenswrapper[2585]: I0910 14:05:17.785556 2585 reconciler_common.go:299] "Volume detached for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/409c04f7-d273-4e5f-a2a4-f6b0707fd21b-webhook-cert\") on node \"ip-10-0-140-8.ec2.internal\" DevicePath \"\"" Sep 10 14:05:17.785610 ip-10-0-140-8 kubenswrapper[2585]: I0910 14:05:17.785567 2585 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/409c04f7-d273-4e5f-a2a4-f6b0707fd21b-metrics-certs\") on node \"ip-10-0-140-8.ec2.internal\" DevicePath \"\"" Sep 10 14:05:17.997255 ip-10-0-140-8 kubenswrapper[2585]: I0910 14:05:17.997207 2585 generic.go:356] "Generic (PLEG): container finished" podID="409c04f7-d273-4e5f-a2a4-f6b0707fd21b" containerID="960d9e3dd1f85341a4de0d280e4624ba2a3ea46c817f8d965d9b8cb172c9f6a3" exitCode=0 Sep 10 14:05:17.997427 ip-10-0-140-8 kubenswrapper[2585]: I0910 14:05:17.997291 2585 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-69bcf98bbb-pfc4t" Sep 10 14:05:17.997427 ip-10-0-140-8 kubenswrapper[2585]: I0910 14:05:17.997297 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-69bcf98bbb-pfc4t" event={"ID":"409c04f7-d273-4e5f-a2a4-f6b0707fd21b","Type":"ContainerDied","Data":"960d9e3dd1f85341a4de0d280e4624ba2a3ea46c817f8d965d9b8cb172c9f6a3"} Sep 10 14:05:17.997427 ip-10-0-140-8 kubenswrapper[2585]: I0910 14:05:17.997357 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-69bcf98bbb-pfc4t" event={"ID":"409c04f7-d273-4e5f-a2a4-f6b0707fd21b","Type":"ContainerDied","Data":"88973dcbd1dbd62f81717964a27a19461f059ab04ba7ae3a746ac4d43839a413"} Sep 10 14:05:17.997427 ip-10-0-140-8 kubenswrapper[2585]: I0910 14:05:17.997380 2585 scope.go:117] "RemoveContainer" containerID="960d9e3dd1f85341a4de0d280e4624ba2a3ea46c817f8d965d9b8cb172c9f6a3" Sep 10 14:05:18.006820 ip-10-0-140-8 kubenswrapper[2585]: I0910 14:05:18.006759 2585 scope.go:117] "RemoveContainer" containerID="960d9e3dd1f85341a4de0d280e4624ba2a3ea46c817f8d965d9b8cb172c9f6a3" Sep 10 14:05:18.007212 ip-10-0-140-8 kubenswrapper[2585]: E0910 14:05:18.007190 2585 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"960d9e3dd1f85341a4de0d280e4624ba2a3ea46c817f8d965d9b8cb172c9f6a3\": container with ID starting with 960d9e3dd1f85341a4de0d280e4624ba2a3ea46c817f8d965d9b8cb172c9f6a3 not found: ID does not exist" containerID="960d9e3dd1f85341a4de0d280e4624ba2a3ea46c817f8d965d9b8cb172c9f6a3" Sep 10 14:05:18.007333 ip-10-0-140-8 kubenswrapper[2585]: I0910 14:05:18.007220 2585 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"960d9e3dd1f85341a4de0d280e4624ba2a3ea46c817f8d965d9b8cb172c9f6a3"} err="failed to get container status \"960d9e3dd1f85341a4de0d280e4624ba2a3ea46c817f8d965d9b8cb172c9f6a3\": rpc error: code = NotFound desc = could not find container \"960d9e3dd1f85341a4de0d280e4624ba2a3ea46c817f8d965d9b8cb172c9f6a3\": container with ID starting with 960d9e3dd1f85341a4de0d280e4624ba2a3ea46c817f8d965d9b8cb172c9f6a3 not found: ID does not exist" Sep 10 14:05:18.026479 ip-10-0-140-8 kubenswrapper[2585]: I0910 14:05:18.026454 2585 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-69bcf98bbb-pfc4t"] Sep 10 14:05:18.034688 ip-10-0-140-8 kubenswrapper[2585]: I0910 14:05:18.034663 2585 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/maas-controller-69bcf98bbb-pfc4t"] Sep 10 14:05:18.915469 ip-10-0-140-8 kubenswrapper[2585]: I0910 14:05:18.915437 2585 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/maas-controller-69bcf98bbb-jzkg9"] Sep 10 14:05:18.915850 ip-10-0-140-8 kubenswrapper[2585]: I0910 14:05:18.915830 2585 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="409c04f7-d273-4e5f-a2a4-f6b0707fd21b" containerName="manager" Sep 10 14:05:18.915850 ip-10-0-140-8 kubenswrapper[2585]: I0910 14:05:18.915845 2585 state_mem.go:107] "Deleted CPUSet assignment" podUID="409c04f7-d273-4e5f-a2a4-f6b0707fd21b" containerName="manager" Sep 10 14:05:18.915923 ip-10-0-140-8 kubenswrapper[2585]: I0910 14:05:18.915905 2585 memory_manager.go:368] "RemoveStaleState removing state" podUID="409c04f7-d273-4e5f-a2a4-f6b0707fd21b" containerName="manager" Sep 10 14:05:18.920120 ip-10-0-140-8 kubenswrapper[2585]: I0910 14:05:18.920103 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-69bcf98bbb-jzkg9" Sep 10 14:05:18.924356 ip-10-0-140-8 kubenswrapper[2585]: I0910 14:05:18.924325 2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"maas-parameters\"" Sep 10 14:05:18.924515 ip-10-0-140-8 kubenswrapper[2585]: I0910 14:05:18.924390 2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-metrics-tls\"" Sep 10 14:05:18.924515 ip-10-0-140-8 kubenswrapper[2585]: I0910 14:05:18.924404 2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-webhook-cert\"" Sep 10 14:05:18.924515 ip-10-0-140-8 kubenswrapper[2585]: I0910 14:05:18.924387 2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-dockercfg-nn6pf\"" Sep 10 14:05:18.931423 ip-10-0-140-8 kubenswrapper[2585]: I0910 14:05:18.931364 2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-69bcf98bbb-jzkg9"] Sep 10 14:05:18.995831 ip-10-0-140-8 kubenswrapper[2585]: I0910 14:05:18.995794 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/44258a49-e44d-42ef-9fc6-6e303ad8552c-webhook-cert\") pod \"maas-controller-69bcf98bbb-jzkg9\" (UID: \"44258a49-e44d-42ef-9fc6-6e303ad8552c\") " pod="opendatahub/maas-controller-69bcf98bbb-jzkg9" Sep 10 14:05:18.995997 ip-10-0-140-8 kubenswrapper[2585]: I0910 14:05:18.995839 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/44258a49-e44d-42ef-9fc6-6e303ad8552c-metrics-certs\") pod \"maas-controller-69bcf98bbb-jzkg9\" (UID: \"44258a49-e44d-42ef-9fc6-6e303ad8552c\") " pod="opendatahub/maas-controller-69bcf98bbb-jzkg9" Sep 10 14:05:18.995997 ip-10-0-140-8 kubenswrapper[2585]: I0910 14:05:18.995871 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bjb4h\" (UniqueName: \"kubernetes.io/projected/44258a49-e44d-42ef-9fc6-6e303ad8552c-kube-api-access-bjb4h\") pod \"maas-controller-69bcf98bbb-jzkg9\" (UID: \"44258a49-e44d-42ef-9fc6-6e303ad8552c\") " pod="opendatahub/maas-controller-69bcf98bbb-jzkg9" Sep 10 14:05:19.097336 ip-10-0-140-8 kubenswrapper[2585]: I0910 14:05:19.097301 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/44258a49-e44d-42ef-9fc6-6e303ad8552c-webhook-cert\") pod \"maas-controller-69bcf98bbb-jzkg9\" (UID: \"44258a49-e44d-42ef-9fc6-6e303ad8552c\") " pod="opendatahub/maas-controller-69bcf98bbb-jzkg9" Sep 10 14:05:19.097513 ip-10-0-140-8 kubenswrapper[2585]: I0910 14:05:19.097357 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/44258a49-e44d-42ef-9fc6-6e303ad8552c-metrics-certs\") pod \"maas-controller-69bcf98bbb-jzkg9\" (UID: \"44258a49-e44d-42ef-9fc6-6e303ad8552c\") " pod="opendatahub/maas-controller-69bcf98bbb-jzkg9" Sep 10 14:05:19.097513 ip-10-0-140-8 kubenswrapper[2585]: I0910 14:05:19.097395 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bjb4h\" (UniqueName: \"kubernetes.io/projected/44258a49-e44d-42ef-9fc6-6e303ad8552c-kube-api-access-bjb4h\") pod \"maas-controller-69bcf98bbb-jzkg9\" (UID: \"44258a49-e44d-42ef-9fc6-6e303ad8552c\") " pod="opendatahub/maas-controller-69bcf98bbb-jzkg9" Sep 10 14:05:19.099851 ip-10-0-140-8 kubenswrapper[2585]: I0910 14:05:19.099816 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/44258a49-e44d-42ef-9fc6-6e303ad8552c-webhook-cert\") pod \"maas-controller-69bcf98bbb-jzkg9\" (UID: \"44258a49-e44d-42ef-9fc6-6e303ad8552c\") " pod="opendatahub/maas-controller-69bcf98bbb-jzkg9" Sep 10 14:05:19.099851 ip-10-0-140-8 kubenswrapper[2585]: I0910 14:05:19.099844 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/44258a49-e44d-42ef-9fc6-6e303ad8552c-metrics-certs\") pod \"maas-controller-69bcf98bbb-jzkg9\" (UID: \"44258a49-e44d-42ef-9fc6-6e303ad8552c\") " pod="opendatahub/maas-controller-69bcf98bbb-jzkg9" Sep 10 14:05:19.107583 ip-10-0-140-8 kubenswrapper[2585]: I0910 14:05:19.107559 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-bjb4h\" (UniqueName: \"kubernetes.io/projected/44258a49-e44d-42ef-9fc6-6e303ad8552c-kube-api-access-bjb4h\") pod \"maas-controller-69bcf98bbb-jzkg9\" (UID: \"44258a49-e44d-42ef-9fc6-6e303ad8552c\") " pod="opendatahub/maas-controller-69bcf98bbb-jzkg9" Sep 10 14:05:19.179584 ip-10-0-140-8 kubenswrapper[2585]: I0910 14:05:19.179504 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fxgmp_fa53c783-d40a-4024-9f7b-1d0d2b57f6dc/konnectivity-agent/0.log" limit=1024 Sep 10 14:05:19.231922 ip-10-0-140-8 kubenswrapper[2585]: I0910 14:05:19.231891 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-69bcf98bbb-jzkg9" Sep 10 14:05:19.505947 ip-10-0-140-8 kubenswrapper[2585]: I0910 14:05:19.505923 2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-69bcf98bbb-jzkg9"] Sep 10 14:05:19.507553 ip-10-0-140-8 kubenswrapper[2585]: W0910 14:05:19.507525 2585 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod44258a49_e44d_42ef_9fc6_6e303ad8552c.slice/crio-94a4872473fcbfc3a2eca41459afc6c214d38ab33a2277b88acb1b8ee2604ee6 WatchSource:0}: Error finding container 94a4872473fcbfc3a2eca41459afc6c214d38ab33a2277b88acb1b8ee2604ee6: Status 404 returned error can't find the container with id 94a4872473fcbfc3a2eca41459afc6c214d38ab33a2277b88acb1b8ee2604ee6 Sep 10 14:05:19.647647 ip-10-0-140-8 kubenswrapper[2585]: I0910 14:05:19.647610 2585 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="409c04f7-d273-4e5f-a2a4-f6b0707fd21b" path="/var/lib/kubelet/pods/409c04f7-d273-4e5f-a2a4-f6b0707fd21b/volumes" Sep 10 14:05:20.011950 ip-10-0-140-8 kubenswrapper[2585]: I0910 14:05:20.011919 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-69bcf98bbb-jzkg9" event={"ID":"44258a49-e44d-42ef-9fc6-6e303ad8552c","Type":"ContainerStarted","Data":"1178e0bd895b67be3e7138cbbecea7ba114fe2edc248ac30c1b39f2f67915570"} Sep 10 14:05:20.012397 ip-10-0-140-8 kubenswrapper[2585]: I0910 14:05:20.011998 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-69bcf98bbb-jzkg9" event={"ID":"44258a49-e44d-42ef-9fc6-6e303ad8552c","Type":"ContainerStarted","Data":"94a4872473fcbfc3a2eca41459afc6c214d38ab33a2277b88acb1b8ee2604ee6"} Sep 10 14:05:20.012397 ip-10-0-140-8 kubenswrapper[2585]: I0910 14:05:20.012063 2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-controller-69bcf98bbb-jzkg9" Sep 10 14:05:20.041667 ip-10-0-140-8 kubenswrapper[2585]: I0910 14:05:20.041619 2585 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/maas-controller-69bcf98bbb-jzkg9" podStartSLOduration=1.6377144000000001 podStartE2EDuration="2.041603486s" totalImagesPullingTime="403.889086ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 14:05:18 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 14:05:20.041463917 +0000 UTC m=+2204.914168119" watchObservedRunningTime="2026-09-10 14:05:20.041603486 +0000 UTC m=+2204.914307687" Sep 10 14:05:31.019942 ip-10-0-140-8 kubenswrapper[2585]: I0910 14:05:31.019908 2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/maas-controller-69bcf98bbb-jzkg9" Sep 10 14:05:36.062029 ip-10-0-140-8 kubenswrapper[2585]: I0910 14:05:36.061996 2585 scope.go:117] "RemoveContainer" containerID="466628d15fa2e47dc77a429e9868871f11628e3b3ca0aa2129f6c4ab11b6516c" Sep 10 14:06:36.086302 ip-10-0-140-8 kubenswrapper[2585]: I0910 14:06:36.086271 2585 scope.go:117] "RemoveContainer" containerID="ab937828b4b46d9381c0df25bcd0db7586e1ea2c5e2ab349d963ca5c72235c39" Sep 10 14:06:36.716188 ip-10-0-140-8 kubenswrapper[2585]: I0910 14:06:36.716153 2585 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-qz6k6"] Sep 10 14:06:36.716403 ip-10-0-140-8 kubenswrapper[2585]: I0910 14:06:36.716378 2585 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-qz6k6" podUID="d6fb0c5a-5f04-414f-b011-6704f8243b7a" containerName="manager" containerID="cri-o://a27718ec89d8a14dc1093a95b52be441ac7990ca28de0fec82350eee1d852558" gracePeriod=10 Sep 10 14:06:39.051808 ip-10-0-140-8 kubenswrapper[2585]: I0910 14:06:39.051783 2585 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-qz6k6" Sep 10 14:06:39.116971 ip-10-0-140-8 kubenswrapper[2585]: I0910 14:06:39.116942 2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/d6fb0c5a-5f04-414f-b011-6704f8243b7a-extensions-socket-volume\") pod \"d6fb0c5a-5f04-414f-b011-6704f8243b7a\" (UID: \"d6fb0c5a-5f04-414f-b011-6704f8243b7a\") " Sep 10 14:06:39.117176 ip-10-0-140-8 kubenswrapper[2585]: I0910 14:06:39.116980 2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-4wl7x\" (UniqueName: \"kubernetes.io/projected/d6fb0c5a-5f04-414f-b011-6704f8243b7a-kube-api-access-4wl7x\") pod \"d6fb0c5a-5f04-414f-b011-6704f8243b7a\" (UID: \"d6fb0c5a-5f04-414f-b011-6704f8243b7a\") " Sep 10 14:06:39.117329 ip-10-0-140-8 kubenswrapper[2585]: I0910 14:06:39.117306 2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d6fb0c5a-5f04-414f-b011-6704f8243b7a-extensions-socket-volume" (OuterVolumeSpecName: "extensions-socket-volume") pod "d6fb0c5a-5f04-414f-b011-6704f8243b7a" (UID: "d6fb0c5a-5f04-414f-b011-6704f8243b7a"). InnerVolumeSpecName "extensions-socket-volume". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 10 14:06:39.119084 ip-10-0-140-8 kubenswrapper[2585]: I0910 14:06:39.119054 2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d6fb0c5a-5f04-414f-b011-6704f8243b7a-kube-api-access-4wl7x" (OuterVolumeSpecName: "kube-api-access-4wl7x") pod "d6fb0c5a-5f04-414f-b011-6704f8243b7a" (UID: "d6fb0c5a-5f04-414f-b011-6704f8243b7a"). InnerVolumeSpecName "kube-api-access-4wl7x". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 10 14:06:39.217652 ip-10-0-140-8 kubenswrapper[2585]: I0910 14:06:39.217580 2585 reconciler_common.go:299] "Volume detached for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/d6fb0c5a-5f04-414f-b011-6704f8243b7a-extensions-socket-volume\") on node \"ip-10-0-140-8.ec2.internal\" DevicePath \"\"" Sep 10 14:06:39.217652 ip-10-0-140-8 kubenswrapper[2585]: I0910 14:06:39.217604 2585 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4wl7x\" (UniqueName: \"kubernetes.io/projected/d6fb0c5a-5f04-414f-b011-6704f8243b7a-kube-api-access-4wl7x\") on node \"ip-10-0-140-8.ec2.internal\" DevicePath \"\"" Sep 10 14:06:39.440577 ip-10-0-140-8 kubenswrapper[2585]: I0910 14:06:39.440544 2585 generic.go:356] "Generic (PLEG): container finished" podID="d6fb0c5a-5f04-414f-b011-6704f8243b7a" containerID="a27718ec89d8a14dc1093a95b52be441ac7990ca28de0fec82350eee1d852558" exitCode=0 Sep 10 14:06:39.440577 ip-10-0-140-8 kubenswrapper[2585]: I0910 14:06:39.440582 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-qz6k6" event={"ID":"d6fb0c5a-5f04-414f-b011-6704f8243b7a","Type":"ContainerDied","Data":"a27718ec89d8a14dc1093a95b52be441ac7990ca28de0fec82350eee1d852558"} Sep 10 14:06:39.440787 ip-10-0-140-8 kubenswrapper[2585]: I0910 14:06:39.440590 2585 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-qz6k6" Sep 10 14:06:39.440787 ip-10-0-140-8 kubenswrapper[2585]: I0910 14:06:39.440625 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-qz6k6" event={"ID":"d6fb0c5a-5f04-414f-b011-6704f8243b7a","Type":"ContainerDied","Data":"5d22fbe7e2d6d03fc4be3f198ff681a7cf9724d6ca5ce8acd28e8b9a6e083f86"} Sep 10 14:06:39.440787 ip-10-0-140-8 kubenswrapper[2585]: I0910 14:06:39.440647 2585 scope.go:117] "RemoveContainer" containerID="a27718ec89d8a14dc1093a95b52be441ac7990ca28de0fec82350eee1d852558" Sep 10 14:06:39.450536 ip-10-0-140-8 kubenswrapper[2585]: I0910 14:06:39.450516 2585 scope.go:117] "RemoveContainer" containerID="a27718ec89d8a14dc1093a95b52be441ac7990ca28de0fec82350eee1d852558" Sep 10 14:06:39.450810 ip-10-0-140-8 kubenswrapper[2585]: E0910 14:06:39.450789 2585 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a27718ec89d8a14dc1093a95b52be441ac7990ca28de0fec82350eee1d852558\": container with ID starting with a27718ec89d8a14dc1093a95b52be441ac7990ca28de0fec82350eee1d852558 not found: ID does not exist" containerID="a27718ec89d8a14dc1093a95b52be441ac7990ca28de0fec82350eee1d852558" Sep 10 14:06:39.450866 ip-10-0-140-8 kubenswrapper[2585]: I0910 14:06:39.450819 2585 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a27718ec89d8a14dc1093a95b52be441ac7990ca28de0fec82350eee1d852558"} err="failed to get container status \"a27718ec89d8a14dc1093a95b52be441ac7990ca28de0fec82350eee1d852558\": rpc error: code = NotFound desc = could not find container \"a27718ec89d8a14dc1093a95b52be441ac7990ca28de0fec82350eee1d852558\": container with ID starting with a27718ec89d8a14dc1093a95b52be441ac7990ca28de0fec82350eee1d852558 not found: ID does not exist" Sep 10 14:06:39.465294 ip-10-0-140-8 kubenswrapper[2585]: I0910 14:06:39.465267 2585 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-qz6k6"] Sep 10 14:06:39.471646 ip-10-0-140-8 kubenswrapper[2585]: I0910 14:06:39.471581 2585 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-qz6k6"] Sep 10 14:06:39.647204 ip-10-0-140-8 kubenswrapper[2585]: I0910 14:06:39.647175 2585 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="d6fb0c5a-5f04-414f-b011-6704f8243b7a" path="/var/lib/kubelet/pods/d6fb0c5a-5f04-414f-b011-6704f8243b7a/volumes" Sep 10 14:07:36.107763 ip-10-0-140-8 kubenswrapper[2585]: I0910 14:07:36.107731 2585 scope.go:117] "RemoveContainer" containerID="ca0f28433f0fccb04456bd43783c81d43c67361199d474321d30215e9569d718" Sep 10 14:07:37.538732 ip-10-0-140-8 kubenswrapper[2585]: I0910 14:07:37.538697 2585 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-2m99n"] Sep 10 14:07:37.539098 ip-10-0-140-8 kubenswrapper[2585]: I0910 14:07:37.539091 2585 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d6fb0c5a-5f04-414f-b011-6704f8243b7a" containerName="manager" Sep 10 14:07:37.539141 ip-10-0-140-8 kubenswrapper[2585]: I0910 14:07:37.539102 2585 state_mem.go:107] "Deleted CPUSet assignment" podUID="d6fb0c5a-5f04-414f-b011-6704f8243b7a" containerName="manager" Sep 10 14:07:37.539184 ip-10-0-140-8 kubenswrapper[2585]: I0910 14:07:37.539174 2585 memory_manager.go:368] "RemoveStaleState removing state" podUID="d6fb0c5a-5f04-414f-b011-6704f8243b7a" containerName="manager" Sep 10 14:07:37.543577 ip-10-0-140-8 kubenswrapper[2585]: I0910 14:07:37.543558 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-2m99n" Sep 10 14:07:37.567432 ip-10-0-140-8 kubenswrapper[2585]: I0910 14:07:37.567410 2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"kuadrant-operator-controller-manager-dockercfg-lsk9v\"" Sep 10 14:07:37.582024 ip-10-0-140-8 kubenswrapper[2585]: I0910 14:07:37.581998 2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-2m99n"] Sep 10 14:07:37.611315 ip-10-0-140-8 kubenswrapper[2585]: I0910 14:07:37.611282 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-44psr\" (UniqueName: \"kubernetes.io/projected/121489c6-246f-40fd-8988-8c904befbc4c-kube-api-access-44psr\") pod \"kuadrant-operator-controller-manager-5769f99d9f-2m99n\" (UID: \"121489c6-246f-40fd-8988-8c904befbc4c\") " pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-2m99n" Sep 10 14:07:37.611465 ip-10-0-140-8 kubenswrapper[2585]: I0910 14:07:37.611358 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/121489c6-246f-40fd-8988-8c904befbc4c-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-5769f99d9f-2m99n\" (UID: \"121489c6-246f-40fd-8988-8c904befbc4c\") " pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-2m99n" Sep 10 14:07:37.712048 ip-10-0-140-8 kubenswrapper[2585]: I0910 14:07:37.712004 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-44psr\" (UniqueName: \"kubernetes.io/projected/121489c6-246f-40fd-8988-8c904befbc4c-kube-api-access-44psr\") pod \"kuadrant-operator-controller-manager-5769f99d9f-2m99n\" (UID: \"121489c6-246f-40fd-8988-8c904befbc4c\") " pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-2m99n" Sep 10 14:07:37.712418 ip-10-0-140-8 kubenswrapper[2585]: I0910 14:07:37.712391 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/121489c6-246f-40fd-8988-8c904befbc4c-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-5769f99d9f-2m99n\" (UID: \"121489c6-246f-40fd-8988-8c904befbc4c\") " pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-2m99n" Sep 10 14:07:37.712710 ip-10-0-140-8 kubenswrapper[2585]: I0910 14:07:37.712687 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/121489c6-246f-40fd-8988-8c904befbc4c-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-5769f99d9f-2m99n\" (UID: \"121489c6-246f-40fd-8988-8c904befbc4c\") " pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-2m99n" Sep 10 14:07:37.723214 ip-10-0-140-8 kubenswrapper[2585]: I0910 14:07:37.723190 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-44psr\" (UniqueName: \"kubernetes.io/projected/121489c6-246f-40fd-8988-8c904befbc4c-kube-api-access-44psr\") pod \"kuadrant-operator-controller-manager-5769f99d9f-2m99n\" (UID: \"121489c6-246f-40fd-8988-8c904befbc4c\") " pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-2m99n" Sep 10 14:07:37.852961 ip-10-0-140-8 kubenswrapper[2585]: I0910 14:07:37.852928 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-2m99n" Sep 10 14:07:38.145433 ip-10-0-140-8 kubenswrapper[2585]: I0910 14:07:38.145405 2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-2m99n"] Sep 10 14:07:38.146986 ip-10-0-140-8 kubenswrapper[2585]: W0910 14:07:38.146957 2585 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod121489c6_246f_40fd_8988_8c904befbc4c.slice/crio-a433a32efce34373e4bd26917eed55b884f3d5ee6335d2840621ef7b1da4e519 WatchSource:0}: Error finding container a433a32efce34373e4bd26917eed55b884f3d5ee6335d2840621ef7b1da4e519: Status 404 returned error can't find the container with id a433a32efce34373e4bd26917eed55b884f3d5ee6335d2840621ef7b1da4e519 Sep 10 14:07:38.754857 ip-10-0-140-8 kubenswrapper[2585]: I0910 14:07:38.754823 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-2m99n" event={"ID":"121489c6-246f-40fd-8988-8c904befbc4c","Type":"ContainerStarted","Data":"cc61d234f6c00232a7f2144fc9cc0510395bc8620b61ee5f7b8306813b196952"} Sep 10 14:07:38.755231 ip-10-0-140-8 kubenswrapper[2585]: I0910 14:07:38.754873 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-2m99n" event={"ID":"121489c6-246f-40fd-8988-8c904befbc4c","Type":"ContainerStarted","Data":"a433a32efce34373e4bd26917eed55b884f3d5ee6335d2840621ef7b1da4e519"} Sep 10 14:07:38.755231 ip-10-0-140-8 kubenswrapper[2585]: I0910 14:07:38.754916 2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-2m99n" Sep 10 14:07:38.788528 ip-10-0-140-8 kubenswrapper[2585]: I0910 14:07:38.788458 2585 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-2m99n" podStartSLOduration=1.788440391 podStartE2EDuration="1.788440391s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 14:07:37 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 14:07:38.787749662 +0000 UTC m=+2343.660453863" watchObservedRunningTime="2026-09-10 14:07:38.788440391 +0000 UTC m=+2343.661144592" Sep 10 14:07:49.762439 ip-10-0-140-8 kubenswrapper[2585]: I0910 14:07:49.762405 2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-2m99n" Sep 10 14:08:35.841661 ip-10-0-140-8 kubenswrapper[2585]: I0910 14:08:35.841633 2585 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-l9l8q_ad555c6a-4385-4f8e-93bb-a174cfaa8e74/kube-multus/2.log" Sep 10 14:08:35.847076 ip-10-0-140-8 kubenswrapper[2585]: I0910 14:08:35.847056 2585 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-l9l8q_ad555c6a-4385-4f8e-93bb-a174cfaa8e74/kube-multus/2.log" Sep 10 14:08:45.868785 ip-10-0-140-8 kubenswrapper[2585]: I0910 14:08:45.868746 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fxgmp_fa53c783-d40a-4024-9f7b-1d0d2b57f6dc/konnectivity-agent/0.log" limit=1024 Sep 10 14:09:00.485624 ip-10-0-140-8 kubenswrapper[2585]: I0910 14:09:00.485545 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fxgmp_fa53c783-d40a-4024-9f7b-1d0d2b57f6dc/konnectivity-agent/0.log" limit=1024 Sep 10 14:09:36.141139 ip-10-0-140-8 kubenswrapper[2585]: I0910 14:09:36.141104 2585 scope.go:117] "RemoveContainer" containerID="ed554c8fba92443a77c5b33664f3343eae1e9b00aa606fec30a45cf0e5b210e4" Sep 10 14:09:36.150078 ip-10-0-140-8 kubenswrapper[2585]: I0910 14:09:36.150055 2585 scope.go:117] "RemoveContainer" containerID="2fc8d5ce5b4265a9496b8488b876cee8162c6aa05c8a3589a489ce3a0dc9a956" Sep 10 14:09:48.559714 ip-10-0-140-8 kubenswrapper[2585]: I0910 14:09:48.559682 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fxgmp_fa53c783-d40a-4024-9f7b-1d0d2b57f6dc/konnectivity-agent/0.log" limit=1024 Sep 10 14:09:49.557379 ip-10-0-140-8 kubenswrapper[2585]: I0910 14:09:49.557340 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fxgmp_fa53c783-d40a-4024-9f7b-1d0d2b57f6dc/konnectivity-agent/0.log" limit=1024 Sep 10 14:10:13.035191 ip-10-0-140-8 kubenswrapper[2585]: I0910 14:10:13.035157 2585 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-cb9d756dc-nrnc7"] Sep 10 14:10:13.038871 ip-10-0-140-8 kubenswrapper[2585]: I0910 14:10:13.038851 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-cb9d756dc-nrnc7" Sep 10 14:10:13.042984 ip-10-0-140-8 kubenswrapper[2585]: I0910 14:10:13.042957 2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-q8qnd\"" Sep 10 14:10:13.042984 ip-10-0-140-8 kubenswrapper[2585]: I0910 14:10:13.042980 2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Sep 10 14:10:13.043192 ip-10-0-140-8 kubenswrapper[2585]: I0910 14:10:13.043007 2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-serving-cert\"" Sep 10 14:10:13.043192 ip-10-0-140-8 kubenswrapper[2585]: I0910 14:10:13.043050 2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-metrics-tls\"" Sep 10 14:10:13.043192 ip-10-0-140-8 kubenswrapper[2585]: I0910 14:10:13.043085 2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Sep 10 14:10:13.051022 ip-10-0-140-8 kubenswrapper[2585]: I0910 14:10:13.050998 2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-cb9d756dc-nrnc7"] Sep 10 14:10:13.138423 ip-10-0-140-8 kubenswrapper[2585]: I0910 14:10:13.138394 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ebd9aa63-c57f-44d9-884a-a4db1c5f0950-metrics-certs\") pod \"maas-api-cb9d756dc-nrnc7\" (UID: \"ebd9aa63-c57f-44d9-884a-a4db1c5f0950\") " pod="odh-ai-gateway-infra/maas-api-cb9d756dc-nrnc7" Sep 10 14:10:13.138610 ip-10-0-140-8 kubenswrapper[2585]: I0910 14:10:13.138456 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-h9sx6\" (UniqueName: \"kubernetes.io/projected/ebd9aa63-c57f-44d9-884a-a4db1c5f0950-kube-api-access-h9sx6\") pod \"maas-api-cb9d756dc-nrnc7\" (UID: \"ebd9aa63-c57f-44d9-884a-a4db1c5f0950\") " pod="odh-ai-gateway-infra/maas-api-cb9d756dc-nrnc7" Sep 10 14:10:13.138610 ip-10-0-140-8 kubenswrapper[2585]: I0910 14:10:13.138480 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ebd9aa63-c57f-44d9-884a-a4db1c5f0950-maas-api-tls\") pod \"maas-api-cb9d756dc-nrnc7\" (UID: \"ebd9aa63-c57f-44d9-884a-a4db1c5f0950\") " pod="odh-ai-gateway-infra/maas-api-cb9d756dc-nrnc7" Sep 10 14:10:13.138610 ip-10-0-140-8 kubenswrapper[2585]: I0910 14:10:13.138507 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ebd9aa63-c57f-44d9-884a-a4db1c5f0950-openshift-service-ca\") pod \"maas-api-cb9d756dc-nrnc7\" (UID: \"ebd9aa63-c57f-44d9-884a-a4db1c5f0950\") " pod="odh-ai-gateway-infra/maas-api-cb9d756dc-nrnc7" Sep 10 14:10:13.240027 ip-10-0-140-8 kubenswrapper[2585]: I0910 14:10:13.239991 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h9sx6\" (UniqueName: \"kubernetes.io/projected/ebd9aa63-c57f-44d9-884a-a4db1c5f0950-kube-api-access-h9sx6\") pod \"maas-api-cb9d756dc-nrnc7\" (UID: \"ebd9aa63-c57f-44d9-884a-a4db1c5f0950\") " pod="odh-ai-gateway-infra/maas-api-cb9d756dc-nrnc7" Sep 10 14:10:13.240234 ip-10-0-140-8 kubenswrapper[2585]: I0910 14:10:13.240040 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ebd9aa63-c57f-44d9-884a-a4db1c5f0950-maas-api-tls\") pod \"maas-api-cb9d756dc-nrnc7\" (UID: \"ebd9aa63-c57f-44d9-884a-a4db1c5f0950\") " pod="odh-ai-gateway-infra/maas-api-cb9d756dc-nrnc7" Sep 10 14:10:13.240234 ip-10-0-140-8 kubenswrapper[2585]: I0910 14:10:13.240071 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ebd9aa63-c57f-44d9-884a-a4db1c5f0950-openshift-service-ca\") pod \"maas-api-cb9d756dc-nrnc7\" (UID: \"ebd9aa63-c57f-44d9-884a-a4db1c5f0950\") " pod="odh-ai-gateway-infra/maas-api-cb9d756dc-nrnc7" Sep 10 14:10:13.240234 ip-10-0-140-8 kubenswrapper[2585]: I0910 14:10:13.240126 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ebd9aa63-c57f-44d9-884a-a4db1c5f0950-metrics-certs\") pod \"maas-api-cb9d756dc-nrnc7\" (UID: \"ebd9aa63-c57f-44d9-884a-a4db1c5f0950\") " pod="odh-ai-gateway-infra/maas-api-cb9d756dc-nrnc7" Sep 10 14:10:13.240775 ip-10-0-140-8 kubenswrapper[2585]: I0910 14:10:13.240753 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ebd9aa63-c57f-44d9-884a-a4db1c5f0950-openshift-service-ca\") pod \"maas-api-cb9d756dc-nrnc7\" (UID: \"ebd9aa63-c57f-44d9-884a-a4db1c5f0950\") " pod="odh-ai-gateway-infra/maas-api-cb9d756dc-nrnc7" Sep 10 14:10:13.243342 ip-10-0-140-8 kubenswrapper[2585]: I0910 14:10:13.243307 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ebd9aa63-c57f-44d9-884a-a4db1c5f0950-maas-api-tls\") pod \"maas-api-cb9d756dc-nrnc7\" (UID: \"ebd9aa63-c57f-44d9-884a-a4db1c5f0950\") " pod="odh-ai-gateway-infra/maas-api-cb9d756dc-nrnc7" Sep 10 14:10:13.243429 ip-10-0-140-8 kubenswrapper[2585]: I0910 14:10:13.243337 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ebd9aa63-c57f-44d9-884a-a4db1c5f0950-metrics-certs\") pod \"maas-api-cb9d756dc-nrnc7\" (UID: \"ebd9aa63-c57f-44d9-884a-a4db1c5f0950\") " pod="odh-ai-gateway-infra/maas-api-cb9d756dc-nrnc7" Sep 10 14:10:13.248798 ip-10-0-140-8 kubenswrapper[2585]: I0910 14:10:13.248768 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-h9sx6\" (UniqueName: \"kubernetes.io/projected/ebd9aa63-c57f-44d9-884a-a4db1c5f0950-kube-api-access-h9sx6\") pod \"maas-api-cb9d756dc-nrnc7\" (UID: \"ebd9aa63-c57f-44d9-884a-a4db1c5f0950\") " pod="odh-ai-gateway-infra/maas-api-cb9d756dc-nrnc7" Sep 10 14:10:13.349911 ip-10-0-140-8 kubenswrapper[2585]: I0910 14:10:13.349882 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-cb9d756dc-nrnc7" Sep 10 14:10:13.622117 ip-10-0-140-8 kubenswrapper[2585]: I0910 14:10:13.622094 2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-cb9d756dc-nrnc7"] Sep 10 14:10:13.623641 ip-10-0-140-8 kubenswrapper[2585]: W0910 14:10:13.623613 2585 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podebd9aa63_c57f_44d9_884a_a4db1c5f0950.slice/crio-4094b6569cc18cd86223174d911ac5584e54c582eea0a65153919cc5b374d56a WatchSource:0}: Error finding container 4094b6569cc18cd86223174d911ac5584e54c582eea0a65153919cc5b374d56a: Status 404 returned error can't find the container with id 4094b6569cc18cd86223174d911ac5584e54c582eea0a65153919cc5b374d56a Sep 10 14:10:13.624963 ip-10-0-140-8 kubenswrapper[2585]: I0910 14:10:13.624939 2585 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Sep 10 14:10:14.592310 ip-10-0-140-8 kubenswrapper[2585]: I0910 14:10:14.592237 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-cb9d756dc-nrnc7" event={"ID":"ebd9aa63-c57f-44d9-884a-a4db1c5f0950","Type":"ContainerStarted","Data":"8a0ef0ee1cdaa25fcdc26ff5c1749092eb3bcc1ccede2820e8dc4f6ff1e5a8eb"} Sep 10 14:10:14.592739 ip-10-0-140-8 kubenswrapper[2585]: I0910 14:10:14.592375 2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-cb9d756dc-nrnc7" Sep 10 14:10:14.592739 ip-10-0-140-8 kubenswrapper[2585]: I0910 14:10:14.592396 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-cb9d756dc-nrnc7" event={"ID":"ebd9aa63-c57f-44d9-884a-a4db1c5f0950","Type":"ContainerStarted","Data":"4094b6569cc18cd86223174d911ac5584e54c582eea0a65153919cc5b374d56a"} Sep 10 14:10:14.617529 ip-10-0-140-8 kubenswrapper[2585]: I0910 14:10:14.617462 2585 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-cb9d756dc-nrnc7" podStartSLOduration=1.180990856 podStartE2EDuration="1.617447301s" totalImagesPullingTime="436.456445ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 14:10:13 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 14:10:14.616567584 +0000 UTC m=+2499.489271797" watchObservedRunningTime="2026-09-10 14:10:14.617447301 +0000 UTC m=+2499.490151501" Sep 10 14:10:20.603223 ip-10-0-140-8 kubenswrapper[2585]: I0910 14:10:20.603195 2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-cb9d756dc-nrnc7" Sep 10 14:10:20.924682 ip-10-0-140-8 kubenswrapper[2585]: I0910 14:10:20.924591 2585 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-67ff8f498c-jn8c9"] Sep 10 14:10:20.937476 ip-10-0-140-8 kubenswrapper[2585]: I0910 14:10:20.937446 2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-67ff8f498c-jn8c9"] Sep 10 14:10:20.937616 ip-10-0-140-8 kubenswrapper[2585]: I0910 14:10:20.937542 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-67ff8f498c-jn8c9" Sep 10 14:10:21.014583 ip-10-0-140-8 kubenswrapper[2585]: I0910 14:10:21.014553 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6e419a69-d32b-41ec-b60b-a7e6e6f2ad89-metrics-certs\") pod \"maas-api-67ff8f498c-jn8c9\" (UID: \"6e419a69-d32b-41ec-b60b-a7e6e6f2ad89\") " pod="odh-ai-gateway-infra/maas-api-67ff8f498c-jn8c9" Sep 10 14:10:21.014763 ip-10-0-140-8 kubenswrapper[2585]: I0910 14:10:21.014634 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6e419a69-d32b-41ec-b60b-a7e6e6f2ad89-maas-api-tls\") pod \"maas-api-67ff8f498c-jn8c9\" (UID: \"6e419a69-d32b-41ec-b60b-a7e6e6f2ad89\") " pod="odh-ai-gateway-infra/maas-api-67ff8f498c-jn8c9" Sep 10 14:10:21.014763 ip-10-0-140-8 kubenswrapper[2585]: I0910 14:10:21.014661 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8tn5h\" (UniqueName: \"kubernetes.io/projected/6e419a69-d32b-41ec-b60b-a7e6e6f2ad89-kube-api-access-8tn5h\") pod \"maas-api-67ff8f498c-jn8c9\" (UID: \"6e419a69-d32b-41ec-b60b-a7e6e6f2ad89\") " pod="odh-ai-gateway-infra/maas-api-67ff8f498c-jn8c9" Sep 10 14:10:21.014763 ip-10-0-140-8 kubenswrapper[2585]: I0910 14:10:21.014685 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/6e419a69-d32b-41ec-b60b-a7e6e6f2ad89-openshift-service-ca\") pod \"maas-api-67ff8f498c-jn8c9\" (UID: \"6e419a69-d32b-41ec-b60b-a7e6e6f2ad89\") " pod="odh-ai-gateway-infra/maas-api-67ff8f498c-jn8c9" Sep 10 14:10:21.116171 ip-10-0-140-8 kubenswrapper[2585]: I0910 14:10:21.116140 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6e419a69-d32b-41ec-b60b-a7e6e6f2ad89-maas-api-tls\") pod \"maas-api-67ff8f498c-jn8c9\" (UID: \"6e419a69-d32b-41ec-b60b-a7e6e6f2ad89\") " pod="odh-ai-gateway-infra/maas-api-67ff8f498c-jn8c9" Sep 10 14:10:21.116390 ip-10-0-140-8 kubenswrapper[2585]: I0910 14:10:21.116181 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8tn5h\" (UniqueName: \"kubernetes.io/projected/6e419a69-d32b-41ec-b60b-a7e6e6f2ad89-kube-api-access-8tn5h\") pod \"maas-api-67ff8f498c-jn8c9\" (UID: \"6e419a69-d32b-41ec-b60b-a7e6e6f2ad89\") " pod="odh-ai-gateway-infra/maas-api-67ff8f498c-jn8c9" Sep 10 14:10:21.116390 ip-10-0-140-8 kubenswrapper[2585]: I0910 14:10:21.116204 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/6e419a69-d32b-41ec-b60b-a7e6e6f2ad89-openshift-service-ca\") pod \"maas-api-67ff8f498c-jn8c9\" (UID: \"6e419a69-d32b-41ec-b60b-a7e6e6f2ad89\") " pod="odh-ai-gateway-infra/maas-api-67ff8f498c-jn8c9" Sep 10 14:10:21.116390 ip-10-0-140-8 kubenswrapper[2585]: I0910 14:10:21.116266 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6e419a69-d32b-41ec-b60b-a7e6e6f2ad89-metrics-certs\") pod \"maas-api-67ff8f498c-jn8c9\" (UID: \"6e419a69-d32b-41ec-b60b-a7e6e6f2ad89\") " pod="odh-ai-gateway-infra/maas-api-67ff8f498c-jn8c9" Sep 10 14:10:21.116959 ip-10-0-140-8 kubenswrapper[2585]: I0910 14:10:21.116933 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/6e419a69-d32b-41ec-b60b-a7e6e6f2ad89-openshift-service-ca\") pod \"maas-api-67ff8f498c-jn8c9\" (UID: \"6e419a69-d32b-41ec-b60b-a7e6e6f2ad89\") " pod="odh-ai-gateway-infra/maas-api-67ff8f498c-jn8c9" Sep 10 14:10:21.118582 ip-10-0-140-8 kubenswrapper[2585]: I0910 14:10:21.118552 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6e419a69-d32b-41ec-b60b-a7e6e6f2ad89-maas-api-tls\") pod \"maas-api-67ff8f498c-jn8c9\" (UID: \"6e419a69-d32b-41ec-b60b-a7e6e6f2ad89\") " pod="odh-ai-gateway-infra/maas-api-67ff8f498c-jn8c9" Sep 10 14:10:21.118684 ip-10-0-140-8 kubenswrapper[2585]: I0910 14:10:21.118635 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6e419a69-d32b-41ec-b60b-a7e6e6f2ad89-metrics-certs\") pod \"maas-api-67ff8f498c-jn8c9\" (UID: \"6e419a69-d32b-41ec-b60b-a7e6e6f2ad89\") " pod="odh-ai-gateway-infra/maas-api-67ff8f498c-jn8c9" Sep 10 14:10:21.133280 ip-10-0-140-8 kubenswrapper[2585]: I0910 14:10:21.133232 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8tn5h\" (UniqueName: \"kubernetes.io/projected/6e419a69-d32b-41ec-b60b-a7e6e6f2ad89-kube-api-access-8tn5h\") pod \"maas-api-67ff8f498c-jn8c9\" (UID: \"6e419a69-d32b-41ec-b60b-a7e6e6f2ad89\") " pod="odh-ai-gateway-infra/maas-api-67ff8f498c-jn8c9" Sep 10 14:10:21.248687 ip-10-0-140-8 kubenswrapper[2585]: I0910 14:10:21.248603 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-67ff8f498c-jn8c9" Sep 10 14:10:21.522511 ip-10-0-140-8 kubenswrapper[2585]: I0910 14:10:21.522484 2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-67ff8f498c-jn8c9"] Sep 10 14:10:21.524731 ip-10-0-140-8 kubenswrapper[2585]: W0910 14:10:21.524703 2585 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6e419a69_d32b_41ec_b60b_a7e6e6f2ad89.slice/crio-725f8fb9f447b3d06419d5c957853679d91f58051820f2cfc0027dab67903f98 WatchSource:0}: Error finding container 725f8fb9f447b3d06419d5c957853679d91f58051820f2cfc0027dab67903f98: Status 404 returned error can't find the container with id 725f8fb9f447b3d06419d5c957853679d91f58051820f2cfc0027dab67903f98 Sep 10 14:10:21.630279 ip-10-0-140-8 kubenswrapper[2585]: I0910 14:10:21.630233 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-67ff8f498c-jn8c9" event={"ID":"6e419a69-d32b-41ec-b60b-a7e6e6f2ad89","Type":"ContainerStarted","Data":"725f8fb9f447b3d06419d5c957853679d91f58051820f2cfc0027dab67903f98"} Sep 10 14:10:22.637926 ip-10-0-140-8 kubenswrapper[2585]: I0910 14:10:22.637886 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-67ff8f498c-jn8c9" event={"ID":"6e419a69-d32b-41ec-b60b-a7e6e6f2ad89","Type":"ContainerStarted","Data":"f7c4a6857126434fc2ba117598e1b536f87bdaa663161bd73d7c5b9b91fc3022"} Sep 10 14:10:22.638333 ip-10-0-140-8 kubenswrapper[2585]: I0910 14:10:22.637965 2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-67ff8f498c-jn8c9" Sep 10 14:10:22.659504 ip-10-0-140-8 kubenswrapper[2585]: I0910 14:10:22.659460 2585 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-67ff8f498c-jn8c9" podStartSLOduration=2.25249819 podStartE2EDuration="2.659445313s" totalImagesPullingTime="406.947123ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 14:10:20 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 14:10:22.659073522 +0000 UTC m=+2507.531777717" watchObservedRunningTime="2026-09-10 14:10:22.659445313 +0000 UTC m=+2507.532149515" Sep 10 14:10:27.481716 ip-10-0-140-8 kubenswrapper[2585]: I0910 14:10:27.481676 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fxgmp_fa53c783-d40a-4024-9f7b-1d0d2b57f6dc/konnectivity-agent/0.log" limit=1024 Sep 10 14:10:27.539217 ip-10-0-140-8 kubenswrapper[2585]: I0910 14:10:27.539182 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fxgmp_fa53c783-d40a-4024-9f7b-1d0d2b57f6dc/konnectivity-agent/0.log" limit=1024 Sep 10 14:10:28.650443 ip-10-0-140-8 kubenswrapper[2585]: I0910 14:10:28.650364 2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-67ff8f498c-jn8c9" Sep 10 14:10:28.731513 ip-10-0-140-8 kubenswrapper[2585]: I0910 14:10:28.731481 2585 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-cb9d756dc-nrnc7"] Sep 10 14:10:28.731712 ip-10-0-140-8 kubenswrapper[2585]: I0910 14:10:28.731675 2585 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-cb9d756dc-nrnc7" podUID="ebd9aa63-c57f-44d9-884a-a4db1c5f0950" containerName="maas-api" containerID="cri-o://8a0ef0ee1cdaa25fcdc26ff5c1749092eb3bcc1ccede2820e8dc4f6ff1e5a8eb" gracePeriod=30 Sep 10 14:10:28.974251 ip-10-0-140-8 kubenswrapper[2585]: I0910 14:10:28.974214 2585 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-cb9d756dc-nrnc7" Sep 10 14:10:29.091744 ip-10-0-140-8 kubenswrapper[2585]: I0910 14:10:29.091709 2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ebd9aa63-c57f-44d9-884a-a4db1c5f0950-openshift-service-ca\") pod \"ebd9aa63-c57f-44d9-884a-a4db1c5f0950\" (UID: \"ebd9aa63-c57f-44d9-884a-a4db1c5f0950\") " Sep 10 14:10:29.091922 ip-10-0-140-8 kubenswrapper[2585]: I0910 14:10:29.091779 2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-h9sx6\" (UniqueName: \"kubernetes.io/projected/ebd9aa63-c57f-44d9-884a-a4db1c5f0950-kube-api-access-h9sx6\") pod \"ebd9aa63-c57f-44d9-884a-a4db1c5f0950\" (UID: \"ebd9aa63-c57f-44d9-884a-a4db1c5f0950\") " Sep 10 14:10:29.091922 ip-10-0-140-8 kubenswrapper[2585]: I0910 14:10:29.091803 2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ebd9aa63-c57f-44d9-884a-a4db1c5f0950-metrics-certs\") pod \"ebd9aa63-c57f-44d9-884a-a4db1c5f0950\" (UID: \"ebd9aa63-c57f-44d9-884a-a4db1c5f0950\") " Sep 10 14:10:29.091922 ip-10-0-140-8 kubenswrapper[2585]: I0910 14:10:29.091892 2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ebd9aa63-c57f-44d9-884a-a4db1c5f0950-maas-api-tls\") pod \"ebd9aa63-c57f-44d9-884a-a4db1c5f0950\" (UID: \"ebd9aa63-c57f-44d9-884a-a4db1c5f0950\") " Sep 10 14:10:29.092104 ip-10-0-140-8 kubenswrapper[2585]: I0910 14:10:29.092066 2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ebd9aa63-c57f-44d9-884a-a4db1c5f0950-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "ebd9aa63-c57f-44d9-884a-a4db1c5f0950" (UID: "ebd9aa63-c57f-44d9-884a-a4db1c5f0950"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 10 14:10:29.093958 ip-10-0-140-8 kubenswrapper[2585]: I0910 14:10:29.093924 2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ebd9aa63-c57f-44d9-884a-a4db1c5f0950-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "ebd9aa63-c57f-44d9-884a-a4db1c5f0950" (UID: "ebd9aa63-c57f-44d9-884a-a4db1c5f0950"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 10 14:10:29.093958 ip-10-0-140-8 kubenswrapper[2585]: I0910 14:10:29.093944 2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ebd9aa63-c57f-44d9-884a-a4db1c5f0950-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "ebd9aa63-c57f-44d9-884a-a4db1c5f0950" (UID: "ebd9aa63-c57f-44d9-884a-a4db1c5f0950"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 10 14:10:29.094111 ip-10-0-140-8 kubenswrapper[2585]: I0910 14:10:29.094039 2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ebd9aa63-c57f-44d9-884a-a4db1c5f0950-kube-api-access-h9sx6" (OuterVolumeSpecName: "kube-api-access-h9sx6") pod "ebd9aa63-c57f-44d9-884a-a4db1c5f0950" (UID: "ebd9aa63-c57f-44d9-884a-a4db1c5f0950"). InnerVolumeSpecName "kube-api-access-h9sx6". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 10 14:10:29.192662 ip-10-0-140-8 kubenswrapper[2585]: I0910 14:10:29.192630 2585 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ebd9aa63-c57f-44d9-884a-a4db1c5f0950-maas-api-tls\") on node \"ip-10-0-140-8.ec2.internal\" DevicePath \"\"" Sep 10 14:10:29.192662 ip-10-0-140-8 kubenswrapper[2585]: I0910 14:10:29.192658 2585 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ebd9aa63-c57f-44d9-884a-a4db1c5f0950-openshift-service-ca\") on node \"ip-10-0-140-8.ec2.internal\" DevicePath \"\"" Sep 10 14:10:29.192854 ip-10-0-140-8 kubenswrapper[2585]: I0910 14:10:29.192669 2585 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-h9sx6\" (UniqueName: \"kubernetes.io/projected/ebd9aa63-c57f-44d9-884a-a4db1c5f0950-kube-api-access-h9sx6\") on node \"ip-10-0-140-8.ec2.internal\" DevicePath \"\"" Sep 10 14:10:29.192854 ip-10-0-140-8 kubenswrapper[2585]: I0910 14:10:29.192680 2585 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ebd9aa63-c57f-44d9-884a-a4db1c5f0950-metrics-certs\") on node \"ip-10-0-140-8.ec2.internal\" DevicePath \"\"" Sep 10 14:10:29.677553 ip-10-0-140-8 kubenswrapper[2585]: I0910 14:10:29.677514 2585 generic.go:356] "Generic (PLEG): container finished" podID="ebd9aa63-c57f-44d9-884a-a4db1c5f0950" containerID="8a0ef0ee1cdaa25fcdc26ff5c1749092eb3bcc1ccede2820e8dc4f6ff1e5a8eb" exitCode=0 Sep 10 14:10:29.677959 ip-10-0-140-8 kubenswrapper[2585]: I0910 14:10:29.677558 2585 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-cb9d756dc-nrnc7" Sep 10 14:10:29.677959 ip-10-0-140-8 kubenswrapper[2585]: I0910 14:10:29.677565 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-cb9d756dc-nrnc7" event={"ID":"ebd9aa63-c57f-44d9-884a-a4db1c5f0950","Type":"ContainerDied","Data":"8a0ef0ee1cdaa25fcdc26ff5c1749092eb3bcc1ccede2820e8dc4f6ff1e5a8eb"} Sep 10 14:10:29.677959 ip-10-0-140-8 kubenswrapper[2585]: I0910 14:10:29.677626 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-cb9d756dc-nrnc7" event={"ID":"ebd9aa63-c57f-44d9-884a-a4db1c5f0950","Type":"ContainerDied","Data":"4094b6569cc18cd86223174d911ac5584e54c582eea0a65153919cc5b374d56a"} Sep 10 14:10:29.677959 ip-10-0-140-8 kubenswrapper[2585]: I0910 14:10:29.677656 2585 scope.go:117] "RemoveContainer" containerID="8a0ef0ee1cdaa25fcdc26ff5c1749092eb3bcc1ccede2820e8dc4f6ff1e5a8eb" Sep 10 14:10:29.687562 ip-10-0-140-8 kubenswrapper[2585]: I0910 14:10:29.687547 2585 scope.go:117] "RemoveContainer" containerID="8a0ef0ee1cdaa25fcdc26ff5c1749092eb3bcc1ccede2820e8dc4f6ff1e5a8eb" Sep 10 14:10:29.687834 ip-10-0-140-8 kubenswrapper[2585]: E0910 14:10:29.687808 2585 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8a0ef0ee1cdaa25fcdc26ff5c1749092eb3bcc1ccede2820e8dc4f6ff1e5a8eb\": container with ID starting with 8a0ef0ee1cdaa25fcdc26ff5c1749092eb3bcc1ccede2820e8dc4f6ff1e5a8eb not found: ID does not exist" containerID="8a0ef0ee1cdaa25fcdc26ff5c1749092eb3bcc1ccede2820e8dc4f6ff1e5a8eb" Sep 10 14:10:29.687939 ip-10-0-140-8 kubenswrapper[2585]: I0910 14:10:29.687838 2585 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8a0ef0ee1cdaa25fcdc26ff5c1749092eb3bcc1ccede2820e8dc4f6ff1e5a8eb"} err="failed to get container status \"8a0ef0ee1cdaa25fcdc26ff5c1749092eb3bcc1ccede2820e8dc4f6ff1e5a8eb\": rpc error: code = NotFound desc = could not find container \"8a0ef0ee1cdaa25fcdc26ff5c1749092eb3bcc1ccede2820e8dc4f6ff1e5a8eb\": container with ID starting with 8a0ef0ee1cdaa25fcdc26ff5c1749092eb3bcc1ccede2820e8dc4f6ff1e5a8eb not found: ID does not exist" Sep 10 14:10:29.700199 ip-10-0-140-8 kubenswrapper[2585]: I0910 14:10:29.700170 2585 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-cb9d756dc-nrnc7"] Sep 10 14:10:29.706483 ip-10-0-140-8 kubenswrapper[2585]: I0910 14:10:29.706452 2585 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-cb9d756dc-nrnc7"] Sep 10 14:10:31.647640 ip-10-0-140-8 kubenswrapper[2585]: I0910 14:10:31.647606 2585 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="ebd9aa63-c57f-44d9-884a-a4db1c5f0950" path="/var/lib/kubelet/pods/ebd9aa63-c57f-44d9-884a-a4db1c5f0950/volumes" Sep 10 14:10:42.810169 ip-10-0-140-8 kubenswrapper[2585]: I0910 14:10:42.810130 2585 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-7477759fbf-zp9pr_38d8ecc2-8168-406b-ab33-b14b0b6c272a/authorino/0.log" Sep 10 14:10:48.342600 ip-10-0-140-8 kubenswrapper[2585]: I0910 14:10:48.342562 2585 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_maas-controller-69bcf98bbb-jzkg9_44258a49-e44d-42ef-9fc6-6e303ad8552c/manager/0.log" Sep 10 14:10:48.716336 ip-10-0-140-8 kubenswrapper[2585]: I0910 14:10:48.716235 2585 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-579d56558b-2bdlh_0673b863-b95e-427e-a6b6-c7ad063b27c9/copy-manifests/0.log" Sep 10 14:10:48.742891 ip-10-0-140-8 kubenswrapper[2585]: I0910 14:10:48.742853 2585 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-579d56558b-2bdlh_0673b863-b95e-427e-a6b6-c7ad063b27c9/manager/0.log" Sep 10 14:10:49.337587 ip-10-0-140-8 kubenswrapper[2585]: I0910 14:10:49.337554 2585 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-67ff8f498c-jn8c9_6e419a69-d32b-41ec-b60b-a7e6e6f2ad89/maas-api/0.log" Sep 10 14:10:49.717165 ip-10-0-140-8 kubenswrapper[2585]: I0910 14:10:49.717084 2585 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-key-cleanup-29817480-j7vlj_ed18dcb9-5e17-4e57-8ad4-56f4c8b7cb17/cleanup/0.log" Sep 10 14:10:50.594177 ip-10-0-140-8 kubenswrapper[2585]: I0910 14:10:50.594149 2585 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-7477759fbf-zp9pr_38d8ecc2-8168-406b-ab33-b14b0b6c272a/authorino/0.log" Sep 10 14:10:51.064918 ip-10-0-140-8 kubenswrapper[2585]: I0910 14:10:51.064888 2585 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-operator-controller-manager-5769f99d9f-2m99n_121489c6-246f-40fd-8988-8c904befbc4c/manager/0.log" Sep 10 14:10:51.930356 ip-10-0-140-8 kubenswrapper[2585]: I0910 14:10:51.930326 2585 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-69ddc5c8c5-5dr6s_3fcb014d-e04c-4ffc-98e9-76502822202b/kube-auth-proxy/0.log" Sep 10 14:10:52.279748 ip-10-0-140-8 kubenswrapper[2585]: I0910 14:10:52.279651 2585 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-pre-processing-59fbc47bf8-tklgb_b99a6276-397c-41d5-ac7f-3a4d380e2834/payload-pre-processing/0.log" Sep 10 14:10:52.852419 ip-10-0-140-8 kubenswrapper[2585]: I0910 14:10:52.852393 2585 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-distinct-2-simulated-kserve-67d8d5d6d-kn59r_7a036868-f373-4ef0-92c6-c2c60b7ff160/main/0.log" Sep 10 14:10:53.314173 ip-10-0-140-8 kubenswrapper[2585]: I0910 14:10:53.314140 2585 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fc46dg_92c99139-db0f-43e4-a9d8-dd0b791fe1ee/main/0.log" Sep 10 14:10:59.055484 ip-10-0-140-8 kubenswrapper[2585]: I0910 14:10:59.055432 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fxgmp_fa53c783-d40a-4024-9f7b-1d0d2b57f6dc/konnectivity-agent/0.log" limit=1024 Sep 10 14:10:59.159959 ip-10-0-140-8 kubenswrapper[2585]: I0910 14:10:59.159924 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fxgmp_fa53c783-d40a-4024-9f7b-1d0d2b57f6dc/konnectivity-agent/0.log" limit=1024 Sep 10 14:11:01.212878 ip-10-0-140-8 kubenswrapper[2585]: I0910 14:11:01.212840 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fxgmp_fa53c783-d40a-4024-9f7b-1d0d2b57f6dc/konnectivity-agent/0.log" limit=1024 Sep 10 14:11:04.348526 ip-10-0-140-8 kubenswrapper[2585]: I0910 14:11:04.348476 2585 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_global-pull-secret-syncer-lhr9l_b37f8a5b-3444-4775-8751-b60c859154d9/global-pull-secret-syncer/0.log" Sep 10 14:11:04.407007 ip-10-0-140-8 kubenswrapper[2585]: I0910 14:11:04.406970 2585 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_konnectivity-agent-fxgmp_fa53c783-d40a-4024-9f7b-1d0d2b57f6dc/konnectivity-agent/0.log" Sep 10 14:11:04.582057 ip-10-0-140-8 kubenswrapper[2585]: I0910 14:11:04.582033 2585 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kube-apiserver-proxy-ip-10-0-140-8.ec2.internal_e9d508517cba13afe9410d7b1b4171d3/haproxy/0.log" Sep 10 14:11:08.847143 ip-10-0-140-8 kubenswrapper[2585]: I0910 14:11:08.847107 2585 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-7477759fbf-zp9pr_38d8ecc2-8168-406b-ab33-b14b0b6c272a/authorino/0.log" Sep 10 14:11:09.072758 ip-10-0-140-8 kubenswrapper[2585]: I0910 14:11:09.072710 2585 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-operator-controller-manager-5769f99d9f-2m99n_121489c6-246f-40fd-8988-8c904befbc4c/manager/0.log" Sep 10 14:11:10.792294 ip-10-0-140-8 kubenswrapper[2585]: I0910 14:11:10.792265 2585 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_41d71fbc-b878-481d-85ae-78beaea2515d/alertmanager/0.log" Sep 10 14:11:10.836520 ip-10-0-140-8 kubenswrapper[2585]: I0910 14:11:10.836494 2585 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_41d71fbc-b878-481d-85ae-78beaea2515d/config-reloader/0.log" Sep 10 14:11:10.863626 ip-10-0-140-8 kubenswrapper[2585]: I0910 14:11:10.863601 2585 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_41d71fbc-b878-481d-85ae-78beaea2515d/kube-rbac-proxy-web/0.log" Sep 10 14:11:10.897460 ip-10-0-140-8 kubenswrapper[2585]: I0910 14:11:10.897438 2585 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_41d71fbc-b878-481d-85ae-78beaea2515d/kube-rbac-proxy/0.log" Sep 10 14:11:10.947552 ip-10-0-140-8 kubenswrapper[2585]: I0910 14:11:10.947510 2585 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_41d71fbc-b878-481d-85ae-78beaea2515d/kube-rbac-proxy-metric/0.log" Sep 10 14:11:10.986674 ip-10-0-140-8 kubenswrapper[2585]: I0910 14:11:10.986648 2585 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_41d71fbc-b878-481d-85ae-78beaea2515d/prom-label-proxy/0.log" Sep 10 14:11:11.021016 ip-10-0-140-8 kubenswrapper[2585]: I0910 14:11:11.020973 2585 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_41d71fbc-b878-481d-85ae-78beaea2515d/init-config-reloader/0.log" Sep 10 14:11:11.338488 ip-10-0-140-8 kubenswrapper[2585]: I0910 14:11:11.338453 2585 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_metrics-server-6ffbdddb-wjr2c_9f2ee68b-c2cd-4d70-b1c9-81ceb68670c2/metrics-server/0.log" Sep 10 14:11:11.387789 ip-10-0-140-8 kubenswrapper[2585]: I0910 14:11:11.387758 2585 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_monitoring-plugin-85bbbbbf4-hvgfb_71f4a37f-99f5-4cd3-9d7e-d2a011d07ee0/monitoring-plugin/0.log" Sep 10 14:11:11.640207 ip-10-0-140-8 kubenswrapper[2585]: I0910 14:11:11.640178 2585 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-m4khk_4e671dec-efdd-4004-9c16-c0c2692371d6/node-exporter/0.log" Sep 10 14:11:11.698017 ip-10-0-140-8 kubenswrapper[2585]: I0910 14:11:11.697991 2585 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-m4khk_4e671dec-efdd-4004-9c16-c0c2692371d6/kube-rbac-proxy/0.log" Sep 10 14:11:11.738799 ip-10-0-140-8 kubenswrapper[2585]: I0910 14:11:11.738775 2585 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-m4khk_4e671dec-efdd-4004-9c16-c0c2692371d6/init-textfile/0.log" Sep 10 14:11:11.981316 ip-10-0-140-8 kubenswrapper[2585]: I0910 14:11:11.981211 2585 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_openshift-state-metrics-657fd6c54-mthpx_1e07789a-9850-45b5-958a-c098e9744d8e/kube-rbac-proxy-main/0.log" Sep 10 14:11:12.012479 ip-10-0-140-8 kubenswrapper[2585]: I0910 14:11:12.012452 2585 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_openshift-state-metrics-657fd6c54-mthpx_1e07789a-9850-45b5-958a-c098e9744d8e/kube-rbac-proxy-self/0.log" Sep 10 14:11:12.045721 ip-10-0-140-8 kubenswrapper[2585]: I0910 14:11:12.045691 2585 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_openshift-state-metrics-657fd6c54-mthpx_1e07789a-9850-45b5-958a-c098e9744d8e/openshift-state-metrics/0.log" Sep 10 14:11:12.466167 ip-10-0-140-8 kubenswrapper[2585]: I0910 14:11:12.466130 2585 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-operator-6f4946db56-8j7lz_c46642aa-e014-4a5d-a2a7-47bdf9bd0fe3/prometheus-operator/0.log" Sep 10 14:11:12.533823 ip-10-0-140-8 kubenswrapper[2585]: I0910 14:11:12.533764 2585 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-operator-6f4946db56-8j7lz_c46642aa-e014-4a5d-a2a7-47bdf9bd0fe3/kube-rbac-proxy/0.log" Sep 10 14:11:12.691237 ip-10-0-140-8 kubenswrapper[2585]: I0910 14:11:12.691147 2585 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_telemeter-client-5d8cdf878c-fkvrj_e1faf521-1963-4ae2-a92e-49f9a7bc15d2/telemeter-client/0.log" Sep 10 14:11:12.720956 ip-10-0-140-8 kubenswrapper[2585]: I0910 14:11:12.720926 2585 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_telemeter-client-5d8cdf878c-fkvrj_e1faf521-1963-4ae2-a92e-49f9a7bc15d2/reload/0.log" Sep 10 14:11:12.760377 ip-10-0-140-8 kubenswrapper[2585]: I0910 14:11:12.760348 2585 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_telemeter-client-5d8cdf878c-fkvrj_e1faf521-1963-4ae2-a92e-49f9a7bc15d2/kube-rbac-proxy/0.log" Sep 10 14:11:12.826988 ip-10-0-140-8 kubenswrapper[2585]: I0910 14:11:12.826955 2585 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-6b46bcc57-874m7_796c05e9-33a7-476b-bc85-c189e991629e/thanos-query/0.log" Sep 10 14:11:12.860067 ip-10-0-140-8 kubenswrapper[2585]: I0910 14:11:12.860041 2585 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-6b46bcc57-874m7_796c05e9-33a7-476b-bc85-c189e991629e/kube-rbac-proxy-web/0.log" Sep 10 14:11:12.893624 ip-10-0-140-8 kubenswrapper[2585]: I0910 14:11:12.893597 2585 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-6b46bcc57-874m7_796c05e9-33a7-476b-bc85-c189e991629e/kube-rbac-proxy/0.log" Sep 10 14:11:12.922580 ip-10-0-140-8 kubenswrapper[2585]: I0910 14:11:12.922557 2585 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-6b46bcc57-874m7_796c05e9-33a7-476b-bc85-c189e991629e/prom-label-proxy/0.log" Sep 10 14:11:12.957168 ip-10-0-140-8 kubenswrapper[2585]: I0910 14:11:12.957076 2585 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-6b46bcc57-874m7_796c05e9-33a7-476b-bc85-c189e991629e/kube-rbac-proxy-rules/0.log" Sep 10 14:11:12.992375 ip-10-0-140-8 kubenswrapper[2585]: I0910 14:11:12.992329 2585 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-6b46bcc57-874m7_796c05e9-33a7-476b-bc85-c189e991629e/kube-rbac-proxy-metrics/0.log" Sep 10 14:11:14.172995 ip-10-0-140-8 kubenswrapper[2585]: I0910 14:11:14.172959 2585 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-must-gather-5vxtc/perf-node-gather-daemonset-97gml"] Sep 10 14:11:14.173614 ip-10-0-140-8 kubenswrapper[2585]: I0910 14:11:14.173483 2585 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ebd9aa63-c57f-44d9-884a-a4db1c5f0950" containerName="maas-api" Sep 10 14:11:14.173614 ip-10-0-140-8 kubenswrapper[2585]: I0910 14:11:14.173499 2585 state_mem.go:107] "Deleted CPUSet assignment" podUID="ebd9aa63-c57f-44d9-884a-a4db1c5f0950" containerName="maas-api" Sep 10 14:11:14.173614 ip-10-0-140-8 kubenswrapper[2585]: I0910 14:11:14.173566 2585 memory_manager.go:368] "RemoveStaleState removing state" podUID="ebd9aa63-c57f-44d9-884a-a4db1c5f0950" containerName="maas-api" Sep 10 14:11:14.182687 ip-10-0-140-8 kubenswrapper[2585]: I0910 14:11:14.182664 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-5vxtc/perf-node-gather-daemonset-97gml" Sep 10 14:11:14.185576 ip-10-0-140-8 kubenswrapper[2585]: I0910 14:11:14.185556 2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-5vxtc\"/\"openshift-service-ca.crt\"" Sep 10 14:11:14.185696 ip-10-0-140-8 kubenswrapper[2585]: I0910 14:11:14.185561 2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-5vxtc\"/\"default-dockercfg-mpghg\"" Sep 10 14:11:14.186747 ip-10-0-140-8 kubenswrapper[2585]: I0910 14:11:14.186730 2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-5vxtc\"/\"kube-root-ca.crt\"" Sep 10 14:11:14.189498 ip-10-0-140-8 kubenswrapper[2585]: I0910 14:11:14.189471 2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-5vxtc/perf-node-gather-daemonset-97gml"] Sep 10 14:11:14.240921 ip-10-0-140-8 kubenswrapper[2585]: I0910 14:11:14.240887 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fxgmp_fa53c783-d40a-4024-9f7b-1d0d2b57f6dc/konnectivity-agent/0.log" limit=1024 Sep 10 14:11:14.311544 ip-10-0-140-8 kubenswrapper[2585]: I0910 14:11:14.311503 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tcpkr\" (UniqueName: \"kubernetes.io/projected/394f6505-40c5-4b3a-8e40-71dd46dfe4e2-kube-api-access-tcpkr\") pod \"perf-node-gather-daemonset-97gml\" (UID: \"394f6505-40c5-4b3a-8e40-71dd46dfe4e2\") " pod="openshift-must-gather-5vxtc/perf-node-gather-daemonset-97gml" Sep 10 14:11:14.311723 ip-10-0-140-8 kubenswrapper[2585]: I0910 14:11:14.311568 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/394f6505-40c5-4b3a-8e40-71dd46dfe4e2-proc\") pod \"perf-node-gather-daemonset-97gml\" (UID: \"394f6505-40c5-4b3a-8e40-71dd46dfe4e2\") " pod="openshift-must-gather-5vxtc/perf-node-gather-daemonset-97gml" Sep 10 14:11:14.311723 ip-10-0-140-8 kubenswrapper[2585]: I0910 14:11:14.311595 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/394f6505-40c5-4b3a-8e40-71dd46dfe4e2-lib-modules\") pod \"perf-node-gather-daemonset-97gml\" (UID: \"394f6505-40c5-4b3a-8e40-71dd46dfe4e2\") " pod="openshift-must-gather-5vxtc/perf-node-gather-daemonset-97gml" Sep 10 14:11:14.311723 ip-10-0-140-8 kubenswrapper[2585]: I0910 14:11:14.311628 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/394f6505-40c5-4b3a-8e40-71dd46dfe4e2-sys\") pod \"perf-node-gather-daemonset-97gml\" (UID: \"394f6505-40c5-4b3a-8e40-71dd46dfe4e2\") " pod="openshift-must-gather-5vxtc/perf-node-gather-daemonset-97gml" Sep 10 14:11:14.311723 ip-10-0-140-8 kubenswrapper[2585]: I0910 14:11:14.311654 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/394f6505-40c5-4b3a-8e40-71dd46dfe4e2-podres\") pod \"perf-node-gather-daemonset-97gml\" (UID: \"394f6505-40c5-4b3a-8e40-71dd46dfe4e2\") " pod="openshift-must-gather-5vxtc/perf-node-gather-daemonset-97gml" Sep 10 14:11:14.412293 ip-10-0-140-8 kubenswrapper[2585]: I0910 14:11:14.412233 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tcpkr\" (UniqueName: \"kubernetes.io/projected/394f6505-40c5-4b3a-8e40-71dd46dfe4e2-kube-api-access-tcpkr\") pod \"perf-node-gather-daemonset-97gml\" (UID: \"394f6505-40c5-4b3a-8e40-71dd46dfe4e2\") " pod="openshift-must-gather-5vxtc/perf-node-gather-daemonset-97gml" Sep 10 14:11:14.412453 ip-10-0-140-8 kubenswrapper[2585]: I0910 14:11:14.412414 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/394f6505-40c5-4b3a-8e40-71dd46dfe4e2-proc\") pod \"perf-node-gather-daemonset-97gml\" (UID: \"394f6505-40c5-4b3a-8e40-71dd46dfe4e2\") " pod="openshift-must-gather-5vxtc/perf-node-gather-daemonset-97gml" Sep 10 14:11:14.412525 ip-10-0-140-8 kubenswrapper[2585]: I0910 14:11:14.412459 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/394f6505-40c5-4b3a-8e40-71dd46dfe4e2-lib-modules\") pod \"perf-node-gather-daemonset-97gml\" (UID: \"394f6505-40c5-4b3a-8e40-71dd46dfe4e2\") " pod="openshift-must-gather-5vxtc/perf-node-gather-daemonset-97gml" Sep 10 14:11:14.412525 ip-10-0-140-8 kubenswrapper[2585]: I0910 14:11:14.412503 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/394f6505-40c5-4b3a-8e40-71dd46dfe4e2-sys\") pod \"perf-node-gather-daemonset-97gml\" (UID: \"394f6505-40c5-4b3a-8e40-71dd46dfe4e2\") " pod="openshift-must-gather-5vxtc/perf-node-gather-daemonset-97gml" Sep 10 14:11:14.412525 ip-10-0-140-8 kubenswrapper[2585]: I0910 14:11:14.412511 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/394f6505-40c5-4b3a-8e40-71dd46dfe4e2-proc\") pod \"perf-node-gather-daemonset-97gml\" (UID: \"394f6505-40c5-4b3a-8e40-71dd46dfe4e2\") " pod="openshift-must-gather-5vxtc/perf-node-gather-daemonset-97gml" Sep 10 14:11:14.412680 ip-10-0-140-8 kubenswrapper[2585]: I0910 14:11:14.412527 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/394f6505-40c5-4b3a-8e40-71dd46dfe4e2-podres\") pod \"perf-node-gather-daemonset-97gml\" (UID: \"394f6505-40c5-4b3a-8e40-71dd46dfe4e2\") " pod="openshift-must-gather-5vxtc/perf-node-gather-daemonset-97gml" Sep 10 14:11:14.412680 ip-10-0-140-8 kubenswrapper[2585]: I0910 14:11:14.412561 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/394f6505-40c5-4b3a-8e40-71dd46dfe4e2-sys\") pod \"perf-node-gather-daemonset-97gml\" (UID: \"394f6505-40c5-4b3a-8e40-71dd46dfe4e2\") " pod="openshift-must-gather-5vxtc/perf-node-gather-daemonset-97gml" Sep 10 14:11:14.412680 ip-10-0-140-8 kubenswrapper[2585]: I0910 14:11:14.412652 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/394f6505-40c5-4b3a-8e40-71dd46dfe4e2-lib-modules\") pod \"perf-node-gather-daemonset-97gml\" (UID: \"394f6505-40c5-4b3a-8e40-71dd46dfe4e2\") " pod="openshift-must-gather-5vxtc/perf-node-gather-daemonset-97gml" Sep 10 14:11:14.412680 ip-10-0-140-8 kubenswrapper[2585]: I0910 14:11:14.412657 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/394f6505-40c5-4b3a-8e40-71dd46dfe4e2-podres\") pod \"perf-node-gather-daemonset-97gml\" (UID: \"394f6505-40c5-4b3a-8e40-71dd46dfe4e2\") " pod="openshift-must-gather-5vxtc/perf-node-gather-daemonset-97gml" Sep 10 14:11:14.423343 ip-10-0-140-8 kubenswrapper[2585]: I0910 14:11:14.423287 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-tcpkr\" (UniqueName: \"kubernetes.io/projected/394f6505-40c5-4b3a-8e40-71dd46dfe4e2-kube-api-access-tcpkr\") pod \"perf-node-gather-daemonset-97gml\" (UID: \"394f6505-40c5-4b3a-8e40-71dd46dfe4e2\") " pod="openshift-must-gather-5vxtc/perf-node-gather-daemonset-97gml" Sep 10 14:11:14.494620 ip-10-0-140-8 kubenswrapper[2585]: I0910 14:11:14.494591 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-5vxtc/perf-node-gather-daemonset-97gml" Sep 10 14:11:14.765567 ip-10-0-140-8 kubenswrapper[2585]: I0910 14:11:14.765514 2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-5vxtc/perf-node-gather-daemonset-97gml"] Sep 10 14:11:14.767820 ip-10-0-140-8 kubenswrapper[2585]: W0910 14:11:14.767794 2585 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-pod394f6505_40c5_4b3a_8e40_71dd46dfe4e2.slice/crio-c0c20e8cab7a45d6a8049b40606da0370b72e43b968aeb5d99ae6803d06c6ff0 WatchSource:0}: Error finding container c0c20e8cab7a45d6a8049b40606da0370b72e43b968aeb5d99ae6803d06c6ff0: Status 404 returned error can't find the container with id c0c20e8cab7a45d6a8049b40606da0370b72e43b968aeb5d99ae6803d06c6ff0 Sep 10 14:11:14.938931 ip-10-0-140-8 kubenswrapper[2585]: I0910 14:11:14.938889 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-5vxtc/perf-node-gather-daemonset-97gml" event={"ID":"394f6505-40c5-4b3a-8e40-71dd46dfe4e2","Type":"ContainerStarted","Data":"ff5a563e67f2f3f9f1f82e86b191843b9571c9a5a8647af1a7a5ed7b7036cd80"} Sep 10 14:11:14.939102 ip-10-0-140-8 kubenswrapper[2585]: I0910 14:11:14.938952 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-5vxtc/perf-node-gather-daemonset-97gml" event={"ID":"394f6505-40c5-4b3a-8e40-71dd46dfe4e2","Type":"ContainerStarted","Data":"c0c20e8cab7a45d6a8049b40606da0370b72e43b968aeb5d99ae6803d06c6ff0"} Sep 10 14:11:14.939102 ip-10-0-140-8 kubenswrapper[2585]: I0910 14:11:14.938978 2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-must-gather-5vxtc/perf-node-gather-daemonset-97gml" Sep 10 14:11:14.960216 ip-10-0-140-8 kubenswrapper[2585]: I0910 14:11:14.960163 2585 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-must-gather-5vxtc/perf-node-gather-daemonset-97gml" podStartSLOduration=0.960146739 podStartE2EDuration="960.146739ms" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-10 14:11:14 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 14:11:14.958092762 +0000 UTC m=+2559.830796963" watchObservedRunningTime="2026-09-10 14:11:14.960146739 +0000 UTC m=+2559.832850940" Sep 10 14:11:15.554554 ip-10-0-140-8 kubenswrapper[2585]: I0910 14:11:15.554528 2585 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console_console-79d8d45465-rwgx9_8239a1fd-3466-432a-a253-b8575a6c2604/console/0.log" Sep 10 14:11:15.589968 ip-10-0-140-8 kubenswrapper[2585]: I0910 14:11:15.589922 2585 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console_downloads-85d98c998f-cjph8_f9a6f8e4-4890-463b-b583-c37100f359e5/download-server/0.log" Sep 10 14:11:17.520525 ip-10-0-140-8 kubenswrapper[2585]: I0910 14:11:17.520495 2585 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-l2ks7_a572632c-303d-4828-a58a-e1c230b6e301/dns/0.log" Sep 10 14:11:17.547278 ip-10-0-140-8 kubenswrapper[2585]: I0910 14:11:17.547236 2585 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-l2ks7_a572632c-303d-4828-a58a-e1c230b6e301/kube-rbac-proxy/0.log" Sep 10 14:11:17.641390 ip-10-0-140-8 kubenswrapper[2585]: I0910 14:11:17.641357 2585 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-84qhc_57474dbb-5547-4132-9d67-dacfb20ea367/dns-node-resolver/0.log" Sep 10 14:11:18.444449 ip-10-0-140-8 kubenswrapper[2585]: I0910 14:11:18.444422 2585 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-zhn9x_1748fe6b-cce2-42ce-b8a7-10d12ffef95a/node-ca/0.log" Sep 10 14:11:20.011397 ip-10-0-140-8 kubenswrapper[2585]: I0910 14:11:20.011367 2585 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-69ddc5c8c5-5dr6s_3fcb014d-e04c-4ffc-98e9-76502822202b/kube-auth-proxy/0.log" Sep 10 14:11:20.126198 ip-10-0-140-8 kubenswrapper[2585]: I0910 14:11:20.126172 2585 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-pre-processing-59fbc47bf8-tklgb_b99a6276-397c-41d5-ac7f-3a4d380e2834/payload-pre-processing/0.log" Sep 10 14:11:20.954264 ip-10-0-140-8 kubenswrapper[2585]: I0910 14:11:20.954216 2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-must-gather-5vxtc/perf-node-gather-daemonset-97gml" Sep 10 14:11:21.251066 ip-10-0-140-8 kubenswrapper[2585]: I0910 14:11:21.250993 2585 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-vpwqf_b308bee9-e8b3-4e7b-98b1-c2f6d833b7a3/serve-healthcheck-canary/0.log" Sep 10 14:11:21.809183 ip-10-0-140-8 kubenswrapper[2585]: I0910 14:11:21.809142 2585 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-5ftzf_abeadb08-451a-477c-bff5-0cb019997ba4/kube-rbac-proxy/0.log" Sep 10 14:11:21.834518 ip-10-0-140-8 kubenswrapper[2585]: I0910 14:11:21.834489 2585 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-5ftzf_abeadb08-451a-477c-bff5-0cb019997ba4/exporter/0.log" Sep 10 14:11:21.863735 ip-10-0-140-8 kubenswrapper[2585]: I0910 14:11:21.863706 2585 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-5ftzf_abeadb08-451a-477c-bff5-0cb019997ba4/extractor/0.log" Sep 10 14:11:24.948928 ip-10-0-140-8 kubenswrapper[2585]: I0910 14:11:24.948892 2585 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_maas-controller-69bcf98bbb-jzkg9_44258a49-e44d-42ef-9fc6-6e303ad8552c/manager/0.log" Sep 10 14:11:25.000266 ip-10-0-140-8 kubenswrapper[2585]: I0910 14:11:25.000189 2585 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-69ddc5c8c5-5dr6s_3fcb014d-e04c-4ffc-98e9-76502822202b/kube-auth-proxy/0.log" Sep 10 14:11:25.111770 ip-10-0-140-8 kubenswrapper[2585]: I0910 14:11:25.111740 2585 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-pre-processing-59fbc47bf8-tklgb_b99a6276-397c-41d5-ac7f-3a4d380e2834/payload-pre-processing/0.log" Sep 10 14:11:25.275863 ip-10-0-140-8 kubenswrapper[2585]: I0910 14:11:25.275836 2585 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-579d56558b-2bdlh_0673b863-b95e-427e-a6b6-c7ad063b27c9/manager/0.log" Sep 10 14:11:25.319583 ip-10-0-140-8 kubenswrapper[2585]: I0910 14:11:25.319554 2585 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-579d56558b-2bdlh_0673b863-b95e-427e-a6b6-c7ad063b27c9/copy-manifests/0.log" Sep 10 14:11:26.934315 ip-10-0-140-8 kubenswrapper[2585]: I0910 14:11:26.934269 2585 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-lws-operator_lws-controller-manager-7c4cb59d47-45xnl_ff6d9ee9-0cdd-4cb9-be20-42a90c11918c/manager/0.log" Sep 10 14:11:27.039437 ip-10-0-140-8 kubenswrapper[2585]: I0910 14:11:27.039391 2585 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-lws-operator_openshift-lws-operator-fd8ccff4c-w7vrb_37ec0b8f-049f-4318-bec4-bf1fa159c313/openshift-lws-operator/0.log" Sep 10 14:11:27.276966 ip-10-0-140-8 kubenswrapper[2585]: I0910 14:11:27.276897 2585 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-69ddc5c8c5-5dr6s_3fcb014d-e04c-4ffc-98e9-76502822202b/kube-auth-proxy/0.log" Sep 10 14:11:27.370230 ip-10-0-140-8 kubenswrapper[2585]: I0910 14:11:27.370195 2585 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-pre-processing-59fbc47bf8-tklgb_b99a6276-397c-41d5-ac7f-3a4d380e2834/payload-pre-processing/0.log"